|
Pilot Usability Study
As part of our design process, we developed a pilot usability
study of an improved version of our second interactive prototype,
which shows real data drawn from the database. We asked three
typical users to try to accomplish seven typical tasks with
our prototype and learned that they were consistently able
to perform the tasks, they found the application to be powerful
and useful, and they had some helpful ideas for improvements
to the tool's usability.
Details regarding our final usability test are available
in the following sections:
Method
Participants
| 1 |
Male |
24 |
Ph.D. Student in Environmental Design |
Graduate Student Researcher at CBE |
| 2 |
Male |
30 |
B.S. in Physics, Ph.D. Student in Environmental Design
|
Graduate Student Researcher at CBE |
| 3 |
Female |
44 |
PhD in Mechanical Engineering |
Research Specialist at CBE and Professor of Architecture |
Apparatus
We tested all three of our participants in the CBE offices.
Tester #1 performed the tasks in her own office on her computer
running Windows ME. Testers #2 and #3 performed the test at
a workstation running Windows XP in a shared office. The latest
Java run-time environment and our prototype was loaded onto
the machines before the test.
Tasks
| Please select the POE survey and show all the responses
for the “are you near an exterior wall?” question. |
Assess the ease of navigating the left hand control
panel. |
| Now you’d like to compare this question with the
“How satisfied are you with the air quality in your
workspace?” question. |
Assess the ease of displaying a typical question comparison
and determining if the resulting view is what the user
expects. |
| This survey was a post-occupancy evaluation survey,
so you’d like to compare it with the baseline survey.
Now please select the Baseline survey and add it to the
view. |
Assess the logic of repeating the set-up of the first
survey in additional surveys. |
| Now you’d like to see the data underneath one
of the charts. |
Assess the placement of this feature in the menu. |
| Now you are interested in viewing the relationship between
air quality satisfaction and being near a window, and
want to add that question to the view. |
Assess if having many charts on screen make it difficult
to make comparisons. |
| Now you’d like to remove the exterior wall question
from the view. |
Assess the placement of the Clear feature in the menu,
and if the selection interaction is natural. |
| Now you’d like to see all the responses for the
“How satisfied are you with the air quality in your
workspace?” question. |
Assess if the Show/Hide terminology is understandable
or if it should be part of grouping. |
Procedure
- Introduce team members to participant.
- Have participant review and sign consent form.
- Explain purpose of the test.
- Introduce participant to ArchMiner and give general directions
on how it is used. Allow them to explore menus and functionality
if they like.
- Have participant attempt tasks.
- Allow participant to explore data with ArchMiner outside
the pre-determined tasks if they are interested.
- Have participant complete post-test evaluation and interview.
Test
Measures
Our test consisted of a combination of observation techniques
and a post-test questionnaire. We observed each participant
perform the tasks, encouraged them to "think aloud," and recorded
time, task completion, and error rate for our specified tasks.
Our post-test questionnaire measured satisfaction with a Likert
Scale, included open-ended questions, and provided the opportunity
to comment on the entire application as well.
Focus of Observations
During our observations we considered the following:
- Task completion time: Can the participants complete the
tasks in a reasonable time? What parts of the interface
take the longest to perform the specified action? Is that
too long?
- Error Rate: Which tasks have the highest error rates?
Are particular areas of the application difficult to use
or understand?
Focus of Questionnaire
During the post-test questionnaire, we considered the following:
- User Satisfaction: Is the user satisfied with the results
he/she obtained? Is the user satisfied with his/her experience?
What was the most satisfying interaction for the user? What
interaction dissatisfied the user the most (how can we improve
it)?
Results
Time to complete tasks
*Note average times to complete tasks are rounded to the nearest
minute because we only timed the tasks by the minute during
our testing. They are also approximate because they include
"thinking aloud" from our users.
| 1 |
Please select the POE survey and show all
the responses for the "are you near an exterior wall?"
question. |
Y,Y,Y |
2 |
| 2 |
Now you'd like to compare this question
with the "How satisfied are you with the air quality
in your workspace?" question. |
Y,N,Y |
5 |
| 3 |
This survey was a post-occupancy evaluation
survey, so you'd like to compare it with the baseline
survey. Now please select the Baseline survey and add
it to the view. |
Y,Y,Y |
2 |
| 4 |
Now you'd like to see the data underneath
one of the charts. |
Y,Y,Y |
2 |
| 5 |
Now you are interested in viewing the relationship
between air quality satisfaction and being near a window,
and want to add that question to the view. |
Y,Y,Y |
2 |
| 6 |
Now you'd like to remove the exterior wall
question from the view.
|
Y,Y,Y |
2 |
| 7 |
Now you'd like to see all the responses
for the "How satisfied are you with the air quality
in your workspace?" question. |
N,N,Y |
4 |
Post-test Likert scale satisfaction questionnaire
*Note average scores are rounded to the nearest tenths. For
questions 1-9, a lower score is preferred. For questions 10-12,
a higher score is preferred.
| 1 |
It was difficult for me to select the survey(s)
I was interested in. |
1.3 |
| 2 |
It was difficult for me to select the question(s)
I was interested in. |
2.3 |
| 3 |
It was difficult for me to navigate the
control panel on the left. |
2.7 |
| 4 |
It was difficult for me to understand the
correlation between the controls on the left and the grid
displayed on the right. |
1.7 |
| 5 |
It was difficult for me to see the relationship
between the data in the column and the row. |
2 |
| 6 |
After having generated some charts, it was
difficult to clear the results I did not want to see. |
1.3 |
| 7 |
After having generated some charts, it was
difficult to add to the results. |
2.3 |
| 8 |
Functions in the top menu bar were difficult
to access. |
2 |
| 9 |
It was difficult to navigate through the
results using the scroll bars. |
2 |
| 10 |
The resulting charts and data would be useful
for my research. |
5 |
| 11 |
The workflow was natural. |
5 |
| 12 |
I was satisfied with ArchMiner as a data
exploratory tool. |
4.7 |
Post-test comments
1) What did you like the most about the ArchMiner interface?
- Having a GUI application that focuses on chart comparison
is something that our users really like.
- Crosstabs are really powerful for the kind of research
our users do.
2) What did you like least about the ArchMiner interface?
- It is difficult to read and select questions from the
control panel because it requires scrolling back and forth
horizontally to read the entire question and then selecting
the checkbox.
- It is difficult to compare charts that are not adjacent
to each other. The current interface does not provide an
easy solution to set visibility for columns/rows or drag
and drop charts to a desired location.
- There are currently no right-click or undo functions.
3) Upon first adding a question to a row, by default all
the responses for that question are shown. Do you think this
is helpful or would you prefer to see the results of having
all the responses combined on one chart initially?
- There is no standard or default that applies to all questions.
Different question types benefit from different settings.
4) Do you think ArchMiner is a useful tool? Please elaborate
on the parts which make it useful or not.
- Our users think that ArchMiner has the potential to be
a very powerful tool. None of their current tools allow
them to generate exploratory charts quickly and easily.
Their goal is not to create presentation-quality charts
but to quickly explore the data and look for patterns.
5) You have just selected a second survey and added it to
the columns. By default, the questions from the first survey
appear in the second. You choose to select a question in the
first survey, which highlights that specific question in both
surveys. Does this multiple selection feature seem natural
to you? Please elaborate.
- In most instances, displaying multiple surveys on the
screen means that our user wants to compare the corresponding
data for those surveys. So, it is preferred that a new added
survey will, by default, be displayed in the same way as
existing surveys.
- One users stated that only survey-level actions should
affect all surveys globally. Question- and response-level
actions should only occur for the selected survey.
6) What would you expect to see in the gray square area between
the green row header and purple column header areas?
- Suggestions from users vary.
- Display options that are already in the file menu such
as views.
- Display new view options such as total column, row, or
survey information.
- Display a graphic or logo.
7) When a question is compared with another question, one
question is always broken-out by responses. The charts show
the "whole" question filtered by each of the other
question's responses. All of the charts in that comparison
add up to 100% -- does this make sense or should each chart
be its own 100%?
- This is a difficult question because some charts are better
displayed with combined responses while others broken out
by responses. One chart may reflect responses from 100%
of survey respondents, while others show a fraction of survey
respondents.
- It is important to show "n" - number of respondents the
data comes from and the total "n" from a survey.
Appendices
Materials:
Raw Data:
|
