| Low-fi Prototype Evaluation
Before building our system with prototyping tools or code,
it is useful to create a paper-prototype to test with potential
users. Such designs do not take much time to make and can
be easily changed, so our commitment to the design was low.
It also allowed us to test our interface early on in the process
so improvements could be made before we invested a large amount
of time coding.
We found this process so helpful, that we performed evaluations
of two Low-fi prototypes, the second was done informally.
Details regarding our first usability test are available
in the following sections:
Method
Participants
We selected our users based on our two primary personas and
their task analysis. They were representative of the users
at the Center for the Built Environment and familiar with
the current online Occupant Satisfaction Survey. The purpose
is to test their ability to use our system to explore the
survey data and not their knowledge of the survey. User #1
is was Research Specialist and User #2 was a Graduate Student
Researcher.
Tasks
| This is the first time you’re using the system.
You’ve just completed a survey on individual thermostats
and would like to look at the data for the first time.
Please login, select the survey you’re interested
in, and find the thermal comfort satisfaction chart. |
Warm-up user and test navigation |
| Now you’d like to see the data underneath one
of the charts |
Test 'select' functionality |
| Now you are interested in looking to see if there is
a relationship between having control over thermostats
and thermal comfort satisfaction levels. |
Test our logic behind finding relationships between
two single questions. |
| Now you decide you want to export the interesting graphs
and their related data for further analysis in a statistical
analysis tool. |
Test 'Library' feature and logic of exporting graphs. |
| Now you’d like to compare this survey with other
buildings in California. |
Assess dataset creation and edit functions. |
Apparatus
The testing was done on the 15” laptop
computer paper prototype described in the “Prototype”
section above. Each task built on the previous tasks so we
left the computer unchanged when we moved to the next task.
Everything that was selected or modified remained unless our
users chose to deselect it. The length of each task varied
depending on how long it took the users. We allowed 45 minutes
for all five tasks and 15 minutes at the end for questions
and feedback about the testing process and our system.
Procedure
Each team member played a different role during the testing.
One team member played the role of the “facilitator”
and is the only person who communicated with the users during
the testing. She explained our test procedure, answered questions,
and read from our “User Test Script”. She was
also responsible for prompting our users to “think aloud”
and asking questions when our users felt like they could not
proceed. Another team member played the role of the “computer”
and responded to our users’ inputs. She marked the highlighted
items with post-it notes, updated labels with ones we created
in advance, improvised by sketching graphs that were not anticipated,
and notified the users when something was “disabled”.
Another team member played the role of the “note taker”
and took notes on paper.
Test
Measures
We measures our users’ ability to complete the tasks,
create queries, combine charts, explore relationships, and
recover from going in erroneous directions. We note their
sequence of actions, mistakes, verbalized thoughts, comments,
and expectations. The goal of our test measures is to see
if our perceived interaction and workflow match that of our
users.
Focus of Observations
- Is our sequence intuitive?
We want to see if our system’s sequence maps to our
users’ workflow.
- Does it make sense to have one main panel (e.g., Exploratory)
and separate panels (e.g. Library, Stats, Dataset, Category,
and Query) that group and display common features/functions
at all times?
We display separate panels because we assume that they will
help the users with their tasks and will be easier to locate.
We hope to test our assumptions and justify dedicating permanent
screen space for the panels.
- Are we using the correct terminologies for our labels?
Terminologies are powerful and can affect the way our users
use our features. We want to make sure we are communicating
effectively with our users.
- How flexible or specific should our system be?
We hope to find a balance between flexibility and ease of
use.
- What tasks are our users able to complete with relative
ease?
We hope to keep the features that our users are successful
with.
Results
One of the first things we noticed was that there was a
distinct difference between the users' interaction styles
when faced with a new interface. The researcher completed
the simple tasks with little problem and did not take much
time. The Graduate Student Researcher, on the other hand,
was a lot more exploratory and wanted to click a lot of buttons
and read everything to see what was available before directly
completing the task. This meant that even though the GSR took
more time, he completed more tasks successfully and became
familiar with more aspects of the interface than the researcher.
Although we thought this tactic was interesting, we are aware
that it is probably not typical of our user base and we should
not depend on it.
Task #1
Both of the users were able to complete the first task without
much problem. There was some confusion regarding the labeling
of the Dataset panel, since the users were looking
for a more clearly indicated area marked Surveys. The
researcher, who works with the survey on a regular basis,
found the Thermal Comfort page very quickly, assumingly
because the navigation is similar to the Reporting Tool. The
GSR, however, tried several methods to find the page before
using the left side navigation. His first choice was to use
the Query panel, suggesting a problem with the labeling
and layout of the UI.
Task #2
Even though the users were able to complete this task fairly
easily, it was more by chance than because the interface was
clear. Both users commented that they did not expect the data
to appear when they clicked the select box, and only selected
it because it was available.
Task #3
This task was the most complicated to complete, and neither
user was fully successful. The Library panel was not
used by either user, so they were not able to display two
charts either side-by-side or combined. This is perhaps explained
by the researcher's comment that the term Library meant
something longer term than what he wanted; he suggested Workspace
or Palette as a more appropriate label. There were
several issues regarding the Query panel's functionality
(e.g., was it AND or OR if there were more than one query?)
as well as how it interacted with the Exploration panel.
It was also noted that our simple query design misrepresented
the data gathered by 'check-all-that-apply' questions, which
could cause erroneous results. There are definitely some major
interface workflow changes necessary to accommodate this important
task.
Task #4
Since neither user made use of the Library feature
in the previous task, the export function was unclear and
confusing. Both of them first chose to export the entire dataset
from the Dataset panel, instead of exporting through
the Library. The researcher did not realize that he
was exporting the entire dataset and believed he finished
the task correctly. The other user, the GSR, recognized that
the dataset export was not what he wanted and presses 'Cancel'.
Eventually, he was able to export the graph and data he wanted
to export, but only through trial and error. He was also confused
by the 'Save' option in the Query panel as well as
the interplay between objects in the Library and what
was shown in the Exploration panel.
Task #5
Both users were able to determine fairly quickly how to add
a dataset to what they were exploring, but the 'Manage Dataset'
label on the button did confuse them. Once they got past this
barrier, the rest of the interaction seemed fairly clear.
Most of the problems in this task involved the labeling of
buttons, such as 'Enable' and 'Disable'.
Debriefing
In the debriefing after the tests, both users expressed that
they would like to create crosstabs or small multiples more
easily, without having to go through the library to do so.
Interestingly, they also both commented that the query should
be more closely attached to the dataset, instead of attached
to the charts, as it was designed. They both felt that it
would be difficult to track filters applied to each saved
graph in the Library panel and may misunderstand the
results.
Appendices
Materials:
Raw Data:
|
