Usability Testing was sent to everybody today March 5th, 2008 and posted in our Wiki Portal.
SCOA Site was completed and presented. Contents were updated based on the Client's and usability testing suggestions.
Prototype might be viewed using rapishare.
| |
Usability testing SCOA Site
Design and conduct a usability test on prototype with at least three people (neighbours in my building who use to go to the YMCA to use computers and do some physical exercising). Also I have to mention the great help provided by the clients that helped me to test the site (Ms. June, Dale and Pat) and provided suggestions through the Client’s Portal:
In this paper I only will comment on the process followed with three possible users (3 adults). For that, I used the following University of Saskatchewan’s chart as a guideline:
Plan developed for usability testing with 3 adults:
Pre-test preparation: design the test; evaluate the test by an expert and some subjects. Making corrections until the questionnaire is the best as possible.
Preparation:
- Schedule:
| Date | Location | Test Time | Target Group | Facilitator | NoteTaker | Test Subjects |
| 03/05 - Tues | YMCA Meeting Room | 1pm - 2pm | Adults | Nelson | Nelson | Ana, Hilda, Robert |
- Room: computer lab YMCA
- Equipment:
Computer Pentium 4 or higher - provided through the resources of the UDO-Technology Center
High speed internet connection
Speakers or head phones
- Materials:
Self-administered printed questionnaire or test (Appendix)
Instructions to participants
1. Introduction:
- Greeting/Use of script:
Welcome: Welcome the test subject and thank them for coming. (Making them feel at ease. Briefly explain the purpose of usability study. Assuring that it is fun and easy).
- Purpose stated:
The purpose of the test sessions is to gauge the usability and ease of navigation of the SCOA updated version.
Mainly I followed the normal steps for usability testing:
- Site was evaluated or assessed by 3 individuals pulled from the defined audience groups. I administered an entrance and exit survey before and after each test, and ask users to sign a release form giving their permission to me for taking notes and quoting their comments to be used for data-gathering purposes. Users were asked to complete tasks read aloud to them by the facilitator.
Test goals were to determine what was or were not working successfully on the Site (updated version) from the users’ perspective. I looked for information such as: Do you really achieve the goal of distinguishing the main themes from the Menu? Were you able to accomplish your goals within the site? Do you as users complete each task successfully? If so, how fast did you perform each task? Is that fast enough to satisfy your purposes? What paths do you take in trying? Do those paths seem efficient to you? Where do you stumble? What problems do you have? Where do you get confused? What words or paths are you looking for which aren’t in the site?
After each session, I included an open-ended general discussion period where users shared their thoughts on any aspect of the site or testing with us.
- Explanations given:
Agenda: Outlining the main things --important to organize everything. For example,
First, have the students sign a waiver that indicates their willingness to participate in this usability study and allowing me as supervisor that I'll be taking notes of their comments, but will keep all their personal information private.
Second, asked them some basic demographic questions.
Third, asked them to complete the first session tasks on the site. I will tell them: “Keep in mind that we aren't testing you, but we are testing the site. Any problems or frustrations you encounter will help me see where the design needs to be changed.”
Fourth, ask them for their general feedback on the site: their opinion. I will say: “This process should take about an hour, so let's get started. It is important to end the test on time.”
2. Conducting the test: (Appendix)
A. Checklist – Making sure everything is ready to conduct test.
B. Waiver/Entrance Questions: have each person complete the waiver. Answer any questions they could have. Have each person complete the demographic/entrance questions. (I either had them complete the form on paper, or asked them the questions out loud and fill it in for them.)
C. Task Based Questions: - The heart of hands-on usability testing is the Task Based Questionnaire. During the test, each user was sitting in front of a computer with the appropriate starting page on the browser using the link:
http://rapidshare.com/files/97142177/WebsiteUpdate.rar.html
I lead the test subject through a series of questions/tasks, encouraging the users to think out loud (not leading the subject to the answer).
I observed the subjects performing fairly simple, common tasks. I verbally guided them through the series of tasks/questions, encouraging them to think out loud and respond to what they are looking at. I asked questions about their thought processes and their decisions as they work, without being intrusive or leading. I also took in-depth notes directly on the Test or Questionnaire.
D. Think-Aloud Protocol: I employed a task-based think-aloud protocol, in which I asked users to communicate their thought processes verbally while they work. I asked them to vocalize what path they take to find information, what questions they have, and what surprises or confuses them as they go through the application. We kept questions open-ended and neutral, such as “What do you mean by that?” or “What did you expect o happen?” When users identify a problem, I will ask them how they would fix it. I will observe body language and facial expressions as well.
3. Exit Questions: When users were finished going through the set of task-based questions, we handed them the exit questions and asked them to answer them. Then I gave them a few minutes to talk informally about their usability experience with us. I asked them what they liked/disliked about the site most, if they had suggestions for improvements, etc. If they did not mention anything during the test itself, we asked further questions about to make sure we understood their perceptions about the Module’s strengths and weaknesses. In my experience, users were only too happy to talk with us when the test was over!
4. Debriefing: Summarizing testing of the content and the site.
- Provide a copy of the questionnaire given to volunteer students (friends who wanted to provide suggestions): See attach a draft of questionnaire(Appendix)
- Suggestions for improvement:
Suggestions from a couple of students here in Ottawa were the following: to highlight activities, to include post-test leading questions, to change size of letters, to change some colors, to include information regarding historical places. Use common vocabulary.
I made the appropriate changes according to their suggestions. Also, Clint one of my colleagues from the course ECMM 874 U Saskatchewan with his prototype and Jamie provided generous ideas that helped me to improve SCOA prototype.
Coding of user’s number on materials: Other adults tested the site from their homes and they sent me all the suggestions by e-mail and we talked face-to-face through messenger. They enjoyed very much the SCOA site and they are willing to work with the new final version, which ii will be much better.
- Written report upon completion: Some reflections on the overall experience of usability testing (See Instrument in Appendix)
I will comment on at least five (5) specific findings that emerge from data (things that happened and that deserved reflection or things I learned from the experience).
First, after a close review of suggestions, I had to change the Menu and put it in simplified way. They suggested the hierarchy according to their interests. I was please with this suggestion because content of the whole site was enhanced in light of priority issues. Also, page distribution was changed.
Second, the site structure was re-organized on the basis of their suggestions. The 8 parts were structured in such a way that go from the easier to the more complex. The size of the menu list was enlarged to facilitate the connexion.
Third, suggestions regarding colors (the logo on light green was not very legible at the introductory part of the site) and also regarding the size of the letters (they wanted larger letters). I eliminated illegible parts, enlarged letters and change colors to facilitate viewing the logo. Initial background was white and I changed for a light green-yellow template.
Fourth, I had to rewrite most of the site’s content in reason of users’ (and Client’s suggestions of clarity and updating information). The contents should be simple, so the test subject can keep it in their mind without too much complexity. The pictures should be realistic scenarios for target audience providing really a nice experience and a global view of the real SCOA. As a result the last version of my prototype really was improved.
Fifth, evaluation: I rewrote the usability testing, which will be used in the future to re-evaluate the site.
As a result the final prototype is different from previous version and it can be viewed by using rapidshare:
http://rapidshare.com/files/97142177/WebsiteUpdate.rar.html
USABILITY TESTING
Part A
CONSENT AND WAIVER
TO WHOM IT MAY CONCERN:
I hereby grant full permission to the University to take notes of my comments during the usability test for Site.
I understand that other students at University may review these usability notes.
I understand that I may be quoted directly in internal University documents, with or without reference to my name.
I understand that usability notes, my name, address, phone number and e-mail address will not be shared with external third parties in compliance with the University of Saskatchewan.
This consent and waiver will not be made the basis of a future claim of any kind against the University and any of its agencies.
IN WITNESS WHEREOF I have hereunto set my hand and seal this
_________________ day of ____________________ Year___.
Name: _________________________________________________
Signature :______________________________________________
Address : ___________________________________________
Witness: ____________________________________
Date: _______________________________________
PART B
Demographic
Entrance Questions - to collect demographic information that I can later use when analyzing results. Questions include name, age, gender, internet experience and target audience group.
Demographic Information Questionnaire
(Please note, your information will not be sold or given to outside entities. It is for internal use only.)
1. Name: ___________________________________________
2. Type: Teacher - Student - General - Public Adult - Older Adults -
3. Computer knowledge Level: 1 2 3 4 5
4. Years of experience in using Internet: _______
5. Age Group: 25-35 36-45 46-55 56-65 66-75 over 75
6. Gender: Female Male
7. How often do you use the internet?
Daily Weekly Monthly Occasionally Never
8. What do you usually do on the internet? (e.g., email,use of resource center, library, use reference materials such as encyclopedias and dictionaries, read news, games, entertainment etc.)
_________________________________________________________________
9. May we contact you about your input at a later date?
If so, please provide your e-mail address. _______________________________
Part C
TEST FOR EVALUATING SITE SCOA
General Instructions
The test facilitator will instruct and observe users performing fairly simple, common tasks. I (as the facilitator) verbally lead users through the series of questions, encouraging them to think out loud and respond to what they are looking at, and ask questions about their thought processes and decisions as they work, without being intrusive or leading. As facilitator I took in-depth notes.
Test Facilitator
Test Subject __________________
Test Date
Test Location
Specific Computer/Software used for this Test:
- Computer Pentium 4 or higher - provided through the resources of the UDO-Technology Center
- High speed internet connection
- Speakers or head phones
Instructions (The following steps were followed)
Facilitator asks the questions out loud. Remind the user we are testing the online Module, not them. It is important to encourage the users to think out loud. (“what words are going thru your mind?”, “what are you looking for?”)
Observe known click stream: [Facilitator checks each step user clicks]
If user does not follow the known path, what did s/he click on? [Facilitator notes different paths/dead-ends]
User’s Verbal comments: [Facilitator notes and writes main user’s verbal comments]
Any suggestions for making this (task) easier? [Facilitator asks user]
J K L M
0 1 2 3 [Facilitator indicates difficulty rate of this task for this user based on Facilitator’s opinion]
J 0 = User completed task with zero difficulty. (Zero Frustration)
K 1 = User completed task with only minor problem(s). (Little Frustration)
L 2 = User completed task, but it required more effort/time/dead-ends than the user
expected. (Medium/High Frustration)
M 3 = User did not complete task. (Point of Failure)
- Menu in eight parts:
Could you show the main contents related to each part? Did they help you to achieve the objectives? Yes___ No___ Explain:____________________________________________________________
Observe known click stream:
User’s Verbal comments: (write some notes)
_______________________________________________________________
Any suggestions for making this (task) easier?
J K L M
01 2 3
- Menu in eight parts: Specifically “Projects”
Could you show the contents and main information/issues from to part II?
Did they help you to achieve the objectives? Yes___ No___ Explain:____________________________________________________________
Observe known click stream
User’s Verbal comments: write notes
_______________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
- Menu in 8 parts:
Could you show the contents and activities related to part III?
Did they help you to achieve the objectives? Yes___ No___ Explain:____________________________________________________________
Observe known click stream
User’s Verbal comments: write comments
______________________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
- A Close Look: I want to experience the activity from the perspective of users. What will be the steps to participate at the SCOA Resource Center? Do you think that through the Resources Center you will be able to be involved in different activities? If so, which ones? Which ones do you like more? Why?
Observe known click stream
User’s Verbal comments: write comments
_____________________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
- Computer Lessons: Do you think that will be a good experience? Why?
Find the assignment and review the instructions.
Observe known click stream
User’s Verbal comments:
_______________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
- Blood Test: Can you explain how to do it at SCOA? Why is important to take advantage of this service?
Observe known click stream:
If user does not follow the known path, what did s/he click on?
User’s Verbal comments:
___________________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
- Pictures: How can you evaluate details on photos?
- Observe known click stream
User’s Verbal comments:
________________________________________________________________
________________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
Objectives:
- Did specific objectives of the site were clear to you?
- Observe known click stream
- If user does not follow the known path, what did s/he click on?
User’s Verbal comments:
______________________________________________________________________
______________________________________________________________________
Any suggestions for making this (task) easier, more motivating or more interesting?
J K L M
0 1 2 3
- Final session: What do you think about participating in one or more activities at SCOA? Please let us know which ones? Why you selected these ones?
Observe known click stream
If user does not follow the known path, what did s/he click on?
User’s Verbal comments:
________________________________________________________________________________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
- Pictures: Are music and pictures something you enjoy and benefit from? Why? Mention which are the best ones.
Observe known click stream
If user does not follow the known path, what did s/he click on?
User’s Verbal comments:
________________________________________________________________
Any suggestions for making this (task) easier?
J K L M
0 1 2 3
When users finished going through the set of task-based questions, I gave them a few minutes to talk informally about their usability experience with you. I asked them what they liked/disliked about the site most, if they have suggestions for improvements, etc.
Exit Questions - At the end of the testing session, I wanted to allow at least 10 minutes for test subject(s) to give me their opinion of the site. How easy was it to navigate the site? What did you like or dislike? What was confusing?
| Please Let Us Know What You Think… SCOA site feedback |
| |
| 1. What features of the site were vague or confusing to you, if any? |
| |
| |
| |
| 2. What is your impression about navigating the site? Does it seem easy or difficult? What does make it that way? |
| |
| |
| 3. What else should be included on the site? |
| |
| |
| 4. What did you like best about the site? |
| |
| |
| 5. What did you like the least? |
| |
| |
| 6. Do you think some people would have problems using the site? What kinds of people? What kinds of problems? |
| |
| |
| 7. Would you like to make any other comments about the site? |
| |
| |
[1] Sources: University of Saskatchewan’s instructions and the University of Texas. Online: www.usask.ca and http://www.utexas.edu/learn/usability/test.html
[2] Questions for this test were specifically formulated for our own purposes of evaluating the site (version updated March, 2008) for course 874 Online: www.usask.ca Format follows the University of Texas, model: http://www.utexas.edu/learn/usability/test.html
