Usability Evaluation Summary - University of Minnesota



469201523114000College of Education and Human Development WebsiteUsability EvaluationsSummary ReportOctober 19, 2009Prepared by:Josh CarrollUsability ConsultantOffice of Information TechnologyUniversity of MinnesotaUsability Evaluation SummaryProject: College of Education and Human Development WebsiteDate: September 28 & 29, 2009Consultant: Josh CarrollTable of ContentsUsability Team Members.............................................................2Description of CEHD Website3Purpose of the Usability Evaluations3Evaluation Methodology4Evaluator Tasks…………………………………………………6Significant Issues8Qualitative Eye-Tracking Observations10AppendicesAppendix A: College of Education and Human Development Website Usability ResultsAppendix B: College of Education and Human Development Website DesirabilityMatrix 1.Usability Team MembersThe College of Education and Human Development (CEHD) usability team was comprised of the following team members:Role on ProjectNameOrgUsability ConsultantJosh CarrollUsability Services, OITUsability Services ManagerDavid RosenUsability Services, OITParticipant Recruiting/Lab Support / Lab ReceptionistKyle DoedenChristy GronsethDesdamona RacheliSia VangUsability Services, OITTeam LeadJeff AbuzzahabCEHDDesign LeadSusan AndreCEHDInformation Architect and Video ProducerDan SagisserCEHDMarketing LeadNuria SheehanCEHDFor questions related to this report, please contact:Josh CarrollUsability ConsultantUniversity of MinnesotaPhone: 612-624-9742Email: carr0234@umn.edu2.Description of the College of Education and Human Development WebsiteThe usability evaluations tested a prototype for the College of Education and Human Development (CEHD) website. This included the core college website, but not the departmental websites. The information found on the CEHD website is intended for prospective students, current students and alumni. 3.Purpose of the Usability Evaluations This evaluation’s purpose was to examine the usability of the College of Education and Human Development website and to identify potential usability issues for its varied users.Specifically, the usability team had agreed upon the following usability goals for the CEHD website:Users can quickly and easily find the information they are seekingUsers understand the relationship between the top and side navigations4.Evaluation MethodologyA task-based usability assessment was conducted using a cross-functional evaluation team consisting of members from Usability Services and the College of Education and Human Development.At the usability kick-off meeting on September 2, 2009, the team jointly established the usability goals, determined participant recruiting criteria for evaluators, and determined representative tasks for the evaluators to use in trying out the CEHD website. Subsequent to this meeting, team members wrote the tasks and scenarios, thus providing a suitable context for the evaluators to assess the design. On September 17, 2009, there was a scenario review meeting where the team and the consultant went through the scenarios using the CEHD website. Revisions to the scenarios or the website were made as needed. Eight evaluations (up to 90 minutes each) took place on September 28 and 29, 2009 at the Usability Services Laboratory on the Minneapolis campus. The breakdown of the eight evaluators was as follows: 1 CEHD undergraduate transfer student1 CEHD undergraduate student who native language is not English1 CEHD undergraduate student3 CEHD graduate students2 CEHD prospective undergraduate studentsEach evaluation session began with an evaluator briefing and an eye-tracker calibration procedure, followed by the evaluator's completion of a number of typical tasks involving the CEHD website, and finished with a debriefing session with the evaluator. Evaluators completed 19 scenarios using the CEHD website.In each of the scenarios, the evaluators were asked to use a “thinking aloud” method to communicate their actions and thoughts to the usability team, in order to provide insight into the evaluator's intentions and perceptions about the process. Eye-tracking data was used to determine whether evaluators saw various features within the site.The usability team observed the evaluation sessions inside an observer room separated from the evaluation room by a one-way mirror. Cameras and microphones in the evaluation room, and separate computer monitors for observers, allowed the team members in the observer room to see the evaluator’s computer screen, mouse / keyboard actions, and facial expressions, and to hear the evaluator’s comments throughout each session. In addition, the usability team was able to see eye-tracking data in real time, showing the evaluators' eye fixations (where they looked for at least 100 ms.) and the paths between the fixations.After each evaluator completed his/her tasks, the usability consultant asked the evaluator a series of debriefing questions to have the evaluator summarize the experience of using the CEHD website, as well as gain additional feedback on its desired features. Evaluators were also asked to complete a desirability matrix worksheet that contained 44 adjectives that were roughly 60% negative and 40% positive. The evaluators were asked to choose the five words that they thought best described the CEHD website and then were asked to explain why they chose the words that they did. The words each evaluator chose are documented in Appendix B – CEHD Desirability Matrix Results.Among the usability team members observing each session, one team member narrated significant observations about the session while another team member recorded those observations in a usability activity log. Another team member took notes specifically about what the evaluators had seen or not seen on the computer screen, as revealed by the real time eye-tracking data. Another team member played the role of a Help Desk, for any evaluators who would have called a help desk about the use of the CEHD website. In addition, video/audio recordings were made of each session for the benefit of the usability consultant. After each evaluation session, the usability team members printed and reviewed the usability log, as well as their notes, and the usability consultant recorded issues identified by the team into a usability results spreadsheet.After all eight evaluation sessions had been completed, the team and the usability consultant met for three hours on September 30, 2009 to review the usability results spreadsheet. Using a consensus approach, the team discussed each identified issue, assigned it a severity level, determined who would take responsibility for the issue, and considered what actions might be taken to address it. These team decisions were documented in the usability results spreadsheet. A preliminary version of the spreadsheet was sent to all team members by email on September 30, 2009. The spreadsheet was revised for the final version included with this report.5.Evaluator TasksEvaluator tasks were presented in the following scenarios. Scenario 1: You have a degree in education and are interested in getting a PhD in educational administration. Find the admission requirements for such a program. Scenario 2: You were recently admitted to the Sport Management major in the College of Education and Human Development. You’ve completed coursework at a community college and would like to know if your community college coursework will transfer into your major requirements. Find out who to contact or what forms you’ll need to complete. Scenario 3: You have been working in social work and human services for nearly 30 years and have decided to go back to school to get a Masters of Social Work. You were recently admitted to the program and don’t know what kind of computer or software you’ll need. Find what kind of computer or software is required.Scenario 4: You are a high school senior and are interested in being a college-level coach. You’ve heard about the kinesiology major but you aren’t sure if this would prepare you for your intended career. Find out if this major would be a good fit for your goal.Scenario 5: You have an undergraduate degree and are thinking about becoming a school counselor. Find out how to apply to a program that will prepare you for this career.Scenario 6: You heard that Ernest Gaines will be coming to campus this fall. Find out when and where he will be speaking and if it is free.Scenario 7: You are a senior majoring in business and marketing education and would like get some career advice before you graduate. Find a professional working in your field who could give you advice.Scenario 8: You are a middle school social studies teacher and your district has mandated that all middle school teachers must get a master’s degree. Find which program would be the best for you.The district will pay 75% of all tuition costs as long as the total cost does not exceed $8,000 per school year. Find out if this program will meet this requirement.Scenario 9: You are the first person in your family to go to college and you don’t really know what to expect. Over the summer you completed the College of Education and Human Development Upward Bound program which gave you great tips on managing your first year. But now that the semester is in full swing you are struggling with a few very difficult classes. Find some help for staying on track.Scenario 10: You have an undergraduate degree and are thinking about becoming biology teacher. Find out what courses are required prior to applying to the program.Scenario 11: You have recently declared a major in Family Social Science. You want to find other students with similar interests and ways to participate in Family Social Science related activities.Scenario 12: You are a teacher and are thinking about going back to school to become a researcher in the area of child psychology and development. You want to know about the reputation and accomplishments of faculty working in child psychology.Scenario 13: You are a middle school English teacher and would like to work in your district as a reading specialist. Find how to apply to a program that will prepare you for this career.Scenario 14: You are a student in China who has recently been admitted to the College of Education and Human Development. You will begin classes in the fall but are not sure what steps you need to take before going to school in the U.S. Find out what you need to do before getting to school in the fall.Scenario 15: You were recently admitted to the Bachelor of Science in Kinesiology. You want to know what scholarships you are eligible for. Find a list of scholarships.Scenario 16: You are a student at a community college and would like to become an elementary school teacher. Find how to apply to this program.Scenario 17: You are considering getting a PhD in art education but don’t know if you’ll be able to afford the program at the U of M. Find out what kind of assistantships and funding are available.Scenario 18: You are completing your degree from the College of Education and Human Development this spring and want to make sure that you’ll walk at graduation. Find out what you have to do to participate in graduation ceremonies.Scenario 19: You want to go back to school to get a masters degree. You are interested in working either as a social worker or a therapist, but aren’t sure which program would be a better fit. Find out the differences between these programs.6.Significant IssuesIssues can be defined as significant in several ways. The primary approach is to assess the implications of the issue for a user in completing the tasks successfully, regardless of the issue’s frequency among the evaluation sessions. This is indicated by the severity level on the usability results spreadsheet, as decided by the usability team. We also note the frequency with which an issue occurred. Although we don’t use the frequency of an issue for drawing statistical conclusions, if an issue is encountered very frequently by evaluators, it becomes more significant even if there is a work-around for it.The most significant issues the team recorded are listed on the next page. See Appendix A -- CEHD Usability Results, for a comprehensive spreadsheet of issues documented in the usability evaluations (including positive comments).Evaluators made a large number of positive comments about the CEHD website during the sessions. The following comments were frequently repeated by the evaluators:Several evaluators said that CEHD website was “organized,” “helpful, “” and “easy to use” while completing the desirability matrix.Several evaluators said that they liked the top navigation.Several evaluators said that they liked the dropdown menus because they could see what was on each page.Some evaluators said that they liked that the website was organized by undergraduate, graduate and transfers student information.An evaluator said that it was easier to find department pages on the website than it is to find department pages through the University website.An evaluator said she didn’t notice when going from the college website to a departmental website.An evaluator said backtracking was easy on this website.An evaluator said it was really easy to find money and tuition information.Most significant issues:IssueNext StepsSeveral evaluators never clicked the "M.Ed" links when looking for teaching information. Always clicked "M.A." links (Issue #1).When looking for school counseling licensure, some evaluators chose the M.A. program rather than M.Ed program.An evaluator said he chose the "M.A." program because if he was already teaching he would have a M.Ed. Evaluator 2 didn't consider the M.Ed for professional studies.Referred to Nuria1. Review terminology to lead with action and outcome rather than programmatic/legal jargon2. Organize current student information by program, but future students by less formal terms/programsSome evaluators didn't think of a newly admitted graduate student as a "Current Student" (Issue #2).An evaluator said that recently admitted students are "future students."An evaluator said that often times there was a fine line between current and future student -- like what is an admitted student who hasn't begun course work?Referred to Nuria and Susan1. Change "Current Students" to "Current and New Students"2. Link on "Future Students" side navigation for "Admitted Students"3. Ambiguity in the term "future studentsInitially, some evaluators didn't see the "Find Faculty By Area" link. Took a while to click the link to find faculty areas (Issue #3).Referred to Nuria and Susan1. List out areas on the "Faculty Highlights" page2. Improve noticeability of the link3. Redesign page content and treatmentSome evaluators never clicked "Teacher Preparation" when looking for information on teaching (Issue #4)An evaluator thought that DirecTrack was for elementary education.An evaluator thought that teaching licensure was available from undergraduate programs.Referred to Nuria1. Explain what undergrad teacher preparation by user terms, not programmatic2. Be more direct that you cannot be a teacher through undergraduate programsLooked for technology information in department pages (Issue #5).Some evaluators looked for mentorship information on a department page.Some evaluators expected Fellowships and Assistantships to be listed with programs.Referred to Jeff1. In general, integrate information across departments, including admission, career services, other current student advising and resources7.Qualitative Eye-Tracking ObservationsA Tobii Technology 1750 eye-tracking monitor and the Tobii Technology ClearView 2.7.3 eye-tracking software were used to collect eye-gaze data, display the live gaze-tracking to the usability team members in the observer room, and record the eye-gaze data for subsequent analysis.The primary eye-tracking deliverable for this project was the live gaze-tracking that the observers watched as the evaluators completed their tasks. This enabled the observers to know exactly where an evaluator was looking on the computer screen during task completion. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download