Usability Test For Duolingo



7402301374141Usability Test For Duolingo final reportENC429000Usability Test For Duolingo final reportENC4290Usability Test Report for DuoLingo Smart Phone ApplicationPhone Application for English-Speaking Users Wanting to Learn Another Language(English to Spanish Tested)Completed by Group 2:Jennie LarnerArsheena MohamedBelinda ProvostMatthew Stofflet December 6, 2015Table of Contents TOC \h \z \t "Report Level 1,1,Report Level 2,2,Report Level 3,3" Executive Summary4Study4Findings PAGEREF _Toc197368151 \h 4Recommendations5Introduction6Test Objective6Purpose6Report Overview7Methodology8Overview8Goals8Procedures PAGEREF _Toc197368158 \h 8User Profiles PAGEREF _Toc197368159 \h 9Test Scenarios/ Tasks10Top Findings PAGEREF _Toc197368160 \h 12Positive Findings PAGEREF _Toc197368162 \h 12Severity Ratings PAGEREF _Toc197368165 \h 13Collated findings with Ratings PAGEREF _Toc197368166 \h 13Post Questionnaire Analyses14Quantitative Findings PAGEREF _Toc197368179 \h 14Qualitative Findings15Recommendations17Long Term17Short Term PAGEREF _Toc197368179 \h 18Urge to Action19Appendices20Appendix A: Heuristic Evaluation20Appendix B: Personas27Appendix C: Test Plan29Appendix D: Screening Questionnaire Results33 HYPERLINK \l "_Toc197368159" Appendix E: Post Questionnaire Results35Appendix F: Moderator Script39Appendix G: Session Logs/ Individual Analyses40Executive SummaryThis report contains our usability evaluation of the DuoLingo Spanish language tutorial mobile phone application. Group 2, a team of five college students enrolled in Dr. Jane Moody’s ENC 4290, Usability Testing for Technical Communication?class, conducted a usability study focused on learning whether the DuoLingo application could be successfully used by both non-Spanish speaking individuals and novice-speakers. The usability study measured how well the application helped users:Evaluate DuoLingo’s mobile application ease of useResearch requirements for successful use of the applicationLearn about the users’ experiences with application?StudyWe conducted the tests remotely, using online tools, allowing us to use participants from the local area. Participants shared their mobile phone screen with team members. That enable the team members to log participants’ comments and reactions to the DuoLingo language mobile phone application. After the participants completed the required language lesson, the moderator asked them to complete a post-test questionnaire. The questionnaire asked more detailed questions about the participants’ experience using the DuoLingo mobile application.?FindingsWhitney Quesenbery's 5E’s was the basis for the test. Quesenbery is a well-known usability consultant and created the 5E’s as an easy way to remember what is significant when testing for usability.Effectiveness: How completely and accurately the tasks, lessons, and experience are finishedEfficiency: How quickly the lessons and tasks can be completedEngaging: How well the app’s interface draws the user into the interaction and how pleasant and satisfying it is to useError Tolerance: How well the app prevents errors and can help the user recover from mistakes that do occurEasy to Use: How well the product supports both the initial orientation and continued learning throughout the complete lifetime of useParticipants consistently had a positive reaction to six aspects of the DuoLingo application:The application was easy to useNice design and layoutClear terminologyLesson buildingOverall satisfied with appWould recommend app to friendsNegative findings have been prioritized by three tiers of severity. High Priority: Lack of review before training/testingClearer instructions for each activityMedium Priority:Accent marks for the app’s keyboardClarifying difficulty evaluationThe selection of words used in application are not varied enoughLow Priority: Auto-Advancement after response givenExplain profile creationTyping on small keyboards in app in unfavorableThis report details our test methodology, findings, and recommendations for addressing usability issues discovered during the test. It includes supporting documentation, including test logs, links to test videos, and planning deliverables.Recommendations We recommend advance prompt be eliminated from application when user has chosen the correct response. Removing that advance prompt would create a more pleasant experience for user. We suggest varying the word selection and removing the redundant words from the application. Establishing a better variety of words used in the application and creating better words pairings with more diverse words make the application less mundane for the user. Imparting a voice to text into the application could alleviate typing issues on cellphones with small keyboards. Voice to text would also satisfy the users’ need to implement proper placement of accent marks. IntroductionBackgroundTest ObjectiveENC 4290, Group 2 is interested in learning whether prospective users of the DuoLingo language mobile application are able to successfully use the app to learn a new language. The group’s goals for perspective users of the app are for them to access and transition through the app with no issues with operation of the app. It is important for the users to feel as though they gain knowledge for speaking Spanish after using the application. From the group’s perspective, the ideal path for perspective users is:The DuoLingo application is comfortable for users’ operationDetermine the needs of the userApplyA heuristic that Group 2 conducted using our targeted persona indicated that the DuoLingo app has the following problems: Application requires preliminary instructions for use of applicationNavigation after correct response requires unnecessary clicking of the continue buttonWord pairing and word selection should be improvedPurposeBased on the finding from the heuristic evaluation, Group 2 created a usability test plan that focused on learning users’ experiences in noted areas. The target users for the usability test were individuals that desired to learn Spanish as a second language or for conversational purposes. The usability study measured prospective Spanish language learners’ success in:Participant’s opinion about the appeal of application’s design and layoutParticipant’s response to instructions included for use of applicationParticipant’s navigation of the application through use of lesson oneParticipant’s opinion of graphics used in appParticipant’s ability to easily learn material provided in lessonParticipant’s understanding of terminology used in applicationThe study also gathered feedback about the overall look and feel of the DuoLingo language application.The study focused on functional and structural feedback about the DuoLingo app rather than design. It focused on the broader DuoLingo app key functions, rather than the application’s design unique to a specific language.Report OverviewThis report provides the finding from the usability study of the DuoLingo language application. It explains how Group 2 shaped and conducted the test, as well as the results found. It includes the following major information:Methodology—explanation of how we selected participants and formed qualitative surveys along with test objectives and proceduresTop Findings—discussion of the major finding from each scenario, positive and negative, qualitative and quantitative, with supporting evidenceRecommendations—suggestions based on our findings on how to improve the DuoLingo app for prospective users desiring to learn a second languageThe report also includes appendices that include the information used to create this usability test and the artifacts generated from it. The appendices include:Appendix A: Heuristic EvaluationAppendix B: Personas Appendix C: Test PlanAppendix D: Screening Questionnaires Appendix E: Post-task QuestionnairesAppendix F: Post-test QuestionnairesAppendix G: Moderator ScriptAppendix H: Session Logs Methodology OverviewThis section of the report provides information on the following:Description of Test Plan with ObjectivesProceduresUser Profiles of the actual test participantsDescription of Scenarios GoalsThe following questions are formed based on the 5Es. ?Efficient: Can users use the application without assistance? Can users perform scenarios within the predetermined time frame??Effective: Can users successfully complete a lesson or create a profile??Engaging: Is the layout appealing? Do users rate their experience as satisfying or enjoyable? Do their comments (and body language) suggest their experience is positive? Do users want to continue using the application to further their learning of a foreign language??Error tolerant: Do users experience errors? If so, how many? So when they experience errors, do they recover successfully? If they receive error messages, do they understand them? Do they know how to move forward??Easy to learn: Can users get started right away? Does their ability to do tasks improve as they become familiar with the application? Does the application work the way the user expects it to work? Do the lessons adjust to the user’s learning experience?ProceduresThe focus of the usability test was task-oriented and governed toward how the user responds to the goals listed above. After finding participants with the help of the screening questionnaire created, participants were given a pre-test questionnaire to get further information about their internet use and expectations in a language tutorial application specifically for a phone. Following that, the participants were given a set of mapped out scenarios to perform finishing with post-task and post-test questionnaires to obtain wanted feedback about the DuoLingo phone application.Users were scheduled in 30-minute sessions to include the questionnaires. There are two different tracks as the app contains a learning experience for anyone with various backgrounds in the language. One track was for persons with no background in the language letting them skip the placement test. The other track was for persons with a background in the language including a placement test. Both tracks include 12 minutes of task scenarios for participants to complete as many lessons as possible in the allotted time.Each moderator conducted tests in various quiet rooms and the test was recorded via a device with a camera to film the subject being tested. The recordings of the test have been uploaded on and the links were included on UCF’s Webcourses for peer review and evaluation.From there, the group evaluated and created individual analyses of each usability test conducted. User ProfilesWe created two personas based on information learned about the DuoLingo program. See Appendix B for the full personas. Only one of the profiles were used as a basis for participant search. Both profiles had personas of busy people always on the go. We chose the college student persona because it represents the majority of the people DuoLingo attracts. The persona was of a busy 23-year-old female that works and goes to school full-time. She is tech savvy and has a background in Spanish she wants to develop.From that persona, we recruited participants with the following characteristics:Age: 18 – 25 years oldEducation: Full-time Undergraduate college studentGender: Mix of female and maleProficient with technology; uses technology on a daily basisHas not used the DuoLingo applicationWorks 40 hours/weekVarious background in SpanishWe believed these types of users would provide us with the best feedback for the app because they likely prefer convenience due to their busy schedules. These users would also be good candidates for rating the 5E’s. Test Scenarios/TasksTrack One: No Background in LanguageOverview/ Briefing (eight minutes)The moderator will explain what the application is and have the participant sign consent form. The moderator will then explain the thinking out loud process and finish the briefing by asking the participant to complete the pre-test questionnaire.Scenario 1 (one minute)You have just downloaded the application. You are not familiar with the app, and this is the first time using the application. You want to create a user profile because you can save your progress. Open the application and select Spanish. Continue with the default settings.Scenario 2 (ten minutes)You want to complete the Basics successfully. Select appropriate choice. It should be “Start with the basics.” The moderator will tell participant to go back if the selection is erroneous. The participant must try to complete the Basics in the time allotted.Scenario 3 (one minute)Review progress and lingots. Closing (ten minutes)The moderator asks the participant to complete the post-task and post- test questionnaires.Track Two: Background in LanguageOverview/ Briefing (eight minutes)The moderator will explain what the application is and have the participant sign consent form. The moderator will then explain the thinking out loud process and finish the briefing by asking the participant to complete the pre-test questionnaire.Scenario 1 (one minute)You have just downloaded the application, and this is the first time using the application and you immediately want to create a profile because you are instructed that this is how progress is saved. Open the application and select Spanish. Continue with the default settings.Scenario 2 (five minutes)You have a background in the language, so you want to try a placement test. Complete the placement test.Scenario 3 (five minutes)Select practice weak skills, or attempt to test out of next set of skills.Scenario 4 (one minute)Review progress and lingots.Closing (ten minutes)The moderator asks the participant to complete the post-task and post- test questionnaires. Top FindingsPositive Findings4051935699135005080013335001. The app is efficient. Users did not need help or assistance when it came to working their way around the app. Users completed tasks in or around predetermined time frames. 2. DuoLingo is effective in teaching. Three users agreed with the short amount of time in using the app, they walked away learning some Spanish. Creating a profile was successful and quick experience for each user. 3. The app is engaging from the layout appeal to the satisfying experience. All users said they would use the app again. 51435394335004. The app displays error tolerance with clear messages and each user understood how to move forward from their mistakes. Recovery was always successful.4058285796925005. All users deemed the app easy to use. Getting started was as simple as pushing a few buttons. The lessons build on each other and there is the option to practice weak skills so learning builds by individual. With a few exceptions, the app worked the way the user expected it to work. Some of the participants made comments on our sheets or during the testing and their reflections were quite positive. We captured their comments when we recorded their testing. Some of the positive comments include:“It’s straight forward”“The lesson structure is well formatted”“The mobile interface is unique and convenient”“It’s simple and fast”“It would take hardly any time to be proficient”Severity RatingsFindings are assigned severity ratings of high, medium, and low. Severity was established and based off frequency of occurrence, the overall impact that the problem presents, and the persistence of the problem. High: These are issues that should be solved before the app continued development. They were found in the majority of participant feedback. Medium: These issues were found in some of the participants and need to be solved but do not inhibit the overall functionality.Low: These issues were found by at least one participant these can be solved but is not necessary.Collated Findings with RatingsBased on the feedback, we identified six usability issues of major significance. The following table is a list of issues along with the severities. Usability Problem# Affected SeverityEffectiveness: There is no review before testing/training 3/4highEfficiency: There is no auto advance when correct answer entered1/4lowEngaging: The selection of words used in application are not varied enough to be challenging2/4mediumError Tolerance: Accent marks for the app’s keyboard2/4mediumEasy to Use: Unclear directions in certain activities4/4highEasy to Use: Typing on small keyboard in app is unfavorable1/4lowPost Questionnaire AnalysisQuantitative FindingsOur quantitative findings were extrapolated from the information and metrics that we collected from our participants during their testing evaluation and after the testing in the form of a Post Testing Questionnaire.Each participant was timed and despite the changes in background between the some of the participant their times came in about the same for the testing period. The average number of lessons completed was 11. Lessons in DuoLingo are completed by an awarding of points since the average of the lessons completed was 11 and only 10 are required for a perfect attempt, this means that there was rather high accuracy with our participants testing. center2416800The overall scoring of the app from the participants was 95/100 points. The scoring system was design from the questions that were taken at the end of the testing period. The guidelines for the questions were created from the basis of Quesenbery’s 5E’s these were adapted in the form of our Post Questionnaire to rate the app on a scaled scoring system.center-15905300Qualitative FindingsOur participants were filmed during the testing and provided a variety of facial expressions and exclamations that were captured to gage the qualitative feedback from our participants.The feedback from the participants was overall elated with remarks such as, “I was correct, Yay,” for whenever they completed a response. The engagement of the app definitely provided that sense of accomplishment when something is completed correctly.The participants also showed some signs of frustration or struggle when approached with a problem they didn’t know how to respond or didn’t know the solution off hand. This was exhibited through responses of “Oh, I don’t think I’m spelling this correctly” to more subtly running their hands through their hair.The graphical user interface of the app was also reviewed by our participants for overall engagement and attractiveness of the application. Our participants’ gave comments such as “I like the little bird” and “The menu is clean and simple” this shows that the app was successful at passing our participants standards. The app maintains a high quality of both engagement and attractiveness.The overall review of the app was very positive and between the questions that were asked there were a total of five points per category of the “five E’s” that the question could evaluate the app towards. There was a total of four tests completed meaning out of a total of 20 possible points per category DuoLingo scored 97/100. The only categories that were not awarded a perfect score were in Efficiency and Engagement. Reasons between the participants differed but the user interaction and the clarity of the questioning were the main faults.A word cloud of the participant’s words used during testing and what is written on the questionnaires leads to the importance of features to the participant’s and its users. A word cloud is generated based off the frequency of word usage. Our participants’ favored the following: User, Spanish, Application, word, app, question, and review. This shows that the interaction with the application and the learning of the Spanish language are the most important features for our participants.7378707302500RecommendationsThis section includes recommendations to improve the usability of the DuoLingo application. The recommendations are based on Final Project Group 2’s findings from the user testing. These recommendations are organized into four categories:InstructionsAdding a review sessionAdding a preliminary sample testDesign ChangesAll recommendations are labeled as either short-term or long-term changes.Long-term ChangesLong-term changes will involve more work and require more restructuring. Long-term changes affect major aspects of the application as well as some content.InstructionsBased on reactions from participants, the lessons need to have more specific instructions, particularly for the word pairing lesson and lessons that include typing (example below). Indicating what language the answer should be given in is needed to help the user complete the lesson. For the word pairing lesson, indicating the lesson is accomplished by pairing the English with the Spanish word is vital for the completion of the lesson. Many users had difficulty figuring out the way to complete the lesson even after getting a wrong answer notification.Having clearer instruction for lessons will allow the users to focus on the content of the application rather than guessing what they are supposed to be doing to complete a lesson successfully.?Adding a Review Session Adding a session before the lesson that reviews the content of what the user will be learning during the lessons. Having a review will allow the users to know and understand the content instead of simply memorizing the answers of the lessons. The review session should include a vocabulary review to familiarize the users.Adding a review session before the lessons will allow the users, especially beginners, to understand the language and content.?Adding a Preliminary Sample Test Adding a preliminary sample test will familiarize the users on how the lessons are set up. The users will see how to answer questions and what happens when they answer questions. They will also see how each lesson is set up. They will see each aspect of the lesson and will know what to expect during the actual lessons.Adding a preliminary sample test will better the application and make it easier to complete the lessons.?Short-term ChangesShort-term changes are simple, easy changes. The majority of the short-term changes are minor and cosmetic, but were pointed out by participants. Short-term changes do not affect the content of the application, but rather the aesthetics of the application.There are three major design changes that would make the user of the application easier and smoother. These three changes are an auto-advance, accent marks, and different color schemes for different language input.?Auto-AdvanceDuring the lessons when a user gives a correct answer, the lesson should automatically advance to the next question. The user should not have to click next after a correct answer is given. This should be a part of the application’s internal software control.This change will save time and be more efficient in the long run.Accent MarksThere are many questions that require the user to type in a Spanish word or phrase. This becomes difficult for the user because there are no accent marks on the keyboard and some questions require accent marks. Even though the user will receive a correct answer without adding an accent mark, it is still necessary for learning purposes to give the user an opportunity to answer with accent marks.Adding accent marks will show the user how the word is correctly written.Different Color SchemesBecause there are many questions that require the user to input an answer in Spanish or English, different color scheme layouts for different languages are another aspect to notify the user of what language to input the answers in. For example, when a question requires a Spanish answer, the color layout could be green. When a question requires an English answer, the color layout could be yellow. When a question requires neither, such as a multiple choice question, the color layout could be blue to go with the theme.Adding different color schemes is not only aesthetically pleasing, but it also helps the users successfully answer questions and use the application easily.?Urge to ActionThis usability test has just scratched the surface of what the DuoLingo application can do. This usability test focused on Spanish and it is necessary to complete usability tests for the other languages that are offered through the DuoLingo application. Overall, the DuoLingo application is successful. However, it is necessary to complete usability tests on all aspects of the application in order for the application to be completely successful.Appendix A: Heuristic EvaluationIntroductionThis heuristic evaluation of the DuoLingo phone application was completed on September 27, 2015. DuoLingo has a variety of options to teach a second language but Group 2 tested the application based on a user that is English speaking with a varied background in the Spanish language. The user is one that must also be able to read and write fluently in English and is comfortable using a smart phone. The group tested the usability of the app by walking through the set up and first lesson of Spanish. To accomplish the users’ goal of attempting to learn a second language, the DuoLingo application must successfully guide users through the application.PurposeThe purpose of this heuristic evaluation is to discover any potential usability weaknesses that might confuse users or sway the user from learning a second language. This report will be used to help identify goals for usability testing. The usability testing will confirm the issues reported in this report. MethodologyThe project team evaluated the website from a native English speaker’s point of view. First, the team reviewed the overall structure and content of the application to determine the scenario, which is the process of creating a profile and completing the first lesson. Then the team completed individual heuristic evaluations. After individual evaluations, the team compiled the results. The application was evaluated based on Quesenbery’s 5Es, which have been presented in Dr. Carol Barnum’s text, Usability Testing and Research.Heuristic Evaluation FindingsEffectiveness How completely and accurately the work or experience is completed or goals reachedThe DuoLingo application was tested to determine if it is effective and if there are changes that can be made to make it more effective.Making an account can be completed very easily and there are no difficulties to be found in completing this task. After creating an account, the user is brought to the screen for the first lesson and the following instruction is provided: “Beginners start at Basics. Advanced learners scroll down to test out” (see Fig. 3). The instructions help explain which lesson the user should begin first.Clicking on the first lesson, the lesson itself is self-explanatory and the accuracy of completing it relies only on the user’s knowledge of the language. There are no errors within the application to cause any inaccuracy in the user’s performance. Within the lesson, there are a few tools to assist the user, such as a speaker button that will repeat the phrase, a turtle button that will repeat the phrase slowly and yellow text which can be clicked on to display the definition of that word. These tools can help the user’s accuracy in answering the questions and to complete the first lesson. Fig. 1Fig. 2Fig. 3Efficiency How quickly the work can be completedUpon observation, the time it took for user to complete a lesson depended on how quickly the user could answer the questions and answer them correctly. The set up and format of the lesson is quite simple and straightforward. It does not take long for the user to become familiar with the application.However, setting up a profile was a little different. When the user first opens the application, the option to start a lesson is big and right in the middle of the application to draw the user’s attention (see Fig. 3). However, the option to create a profile is not. Because the option to create a profile is small and in the upper left corner, the user may easily miss it. Creating a profile may take a little more time for the users. Having the option to create a profile could be done quicker if the format was larger and closer to the lesson option. Once the users find the option, creating a profile was efficient. It was quick and easy. It required some personal information such as an email address and the user’s name (see Fig. 4). It should take the users less than 2 minutes to complete this portion. After creating an account, the user is able to input the information to sign in when the application is opened (see Fig. 2).Overall, the DuoLingo app is efficient and the tasks should not take long to complete. Fig. 4Fig. 5Fig. 6Engaging How well the interface draws the user into the interaction and how pleasant and satisfying it is to useDuoLingo grabs users’ attention from the launch with its attractive color scheme as well as an inspirational phrase (see Fig. 1). Setting up a profile is so simple; the user is over with it before they know it, allowing for lessons to start immediately after (see Fig. 4). The lessons are displayed somewhat like a game. Hearts at the top right of each lesson represent the amount of mistakes allowed before the lesson is failed. The user is also able to see how far in the lesson he/she is at any given point with a bar that fills in green at the top of the screen.The simplicity of the application contributes a lot to how engaging the application is overall. Green represents correct while red represents incorrect. There is also diversity in how questions are asked from matching vocabulary with pictures or pairing to pronunciation to translating full phrases or sentences. With this diversity, learning is never repetitive or boring. The application also has a fun mascot: Duo the Owl. He is there after each lesson to show the how far he/she has come overall and what the user has gained from completing each lesson (see Fig. 10). Fig. 7Fig. 8Fig. 9Error Tolerant How well the product prevents errors and can help the user recover from mistakes that do occurThe DuoLingo application was tested for Error Tolerance. The application, when presented with an incorrect answer in the daily practice for the user, handled it with ease. It presented the correct solution and then allowed the user to review the correct answer and find the faults behind their reasoning (see Fig. 5). After a collection of missed answers occurred during the quizzing portion of the daily training, the amount of incorrect and correct answers are analyzed. The incorrect and correct answers are then built into a little report to give to the user so they can reduce their errors. The users are given a chance to repeat the answers missed during the daily practice in general.In terms of the robustness of the application in general, the testing to break the application to make it no longer functional proved unsuccessful. The application served to be as sturdy as the device it was running on. So the error tolerance in terms of incorrect input or misuse of the application in general is not allowed due to the nature of the environment created. Overall, the DuoLingo application not only handled errors from the user but also allowed self moderation and starts a curated plan to re-mediate any misunderstood information. Fig. 10Fig. 11Fig. 12Easy to Use How well the product supports both the initial orientation and continued learning throughout the complete lifetime of useThe initial observation is that the application is easy to register in order to start the first lesson. When entering the lesson one phase of the application, it uses simplistic human-like animations as a teaching tool. The application provides step-by-step prompts on each page for each level of learning. The animated character says a word or a phrase and the user is then to attempt to replicate what the character said. If the user makes the correct selection, the application will send a green notification that notifies the user that the choice was the correct answer. However, if the user makes an incorrect selection, the application will send a red notification to notify the user that the choice was incorrect. The program advances after the successful completion of each phase of the lesson.The DuoLingo application used in this evaluation can be installed on a cell phone for convenience. The interface was very easy to understand and easy to use. The characters were easy to see and hear. Their voices were clear and precise in pronunciation of the words or phrases. There were no errors or error messages encountered while using the application. DuoLingo proved to be a useful learning tool for English speakers seeking to learn Spanish.Findings and ConclusionKey FindingsBelow is a ranking of our findings based on our evaluation of the application. We have prioritized the findings into three categories: high priority, medium priority, and low priority.High Priority IssuesWhen the user has to vocally repeat the persona's phrase or sentence, it requires multiple attempts for the application to understand and often skips over that process after a few times of not understanding the user. That can be frustrating to the user, especially if the user is pronouncing the words properly. That issue is likely to frustrate the user to the point where they leave or abandon application all together (see Fig. 7). Medium Priority IssuesThe application should include a button that will allow visually challenged users to change font size. If the user cannot read the content they may not continue to use application.Hearing impaired users may have difficulty hearing each persona without headphones, even with the phone volume turn on highest setting. The application set volume should be adjusted to higher volume to accommodate the hearing impaired users. Low Priority IssuesIt would be beneficial to the user if the application had buttons for accent marks. When the user does not input accent marks, the application notifies the user to add the accent marks. Novice cellphone users may not know how to put accent marks on the letters and will become agitated with the program.Creating a profile is necessary for saving progress. If the user did not create a profile before starting a lesson, the user will be prompted to start the second lesson or create a profile but does not urge the user to create the profile. Progress may be lost without a profile. ConclusionOverall, the Duo Lingo application is easy to use, efficient, error tolerant, engaging, and effective. The application is straightforward which prevents any confusion from occurring. The user is able to complete the lesson with ease and if the user chooses a wrong answer, the application shows the correct answer which helps prevent the user from making that mistake again. The application is formatted to keep the user engaged and focused. The time it takes to complete a lesson solely depends on the user. All in all, through the first usability tests, the Duo Lingo application proved that it is user friendly.Appendix B: PersonasPersona One508005588000Name: Benjamin RileyWhite/Male55 years oldBen is a retired widow and travels the majority of the year. ?“The primary language of Colombia is Spanish. I need quick and efficient lessons to fit my lifestyle.”Occupation: Retired Income status: Financially secureTechnical profile: Somewhat comfortable with technology, but does not use it often by preference. He goes to internet cafes or anywhere with WiFi to use his personal computer and other electronics.Internet use: Email and informationHobbies: Hiking, Snow Skiing, Racquetball, Theater, TravelGoals: To be able to speak Spanish for his trip to Colombia, without disrupting his lifestyle.Ben is a recent widow, who is the father to his two independent adult children that attend college at UCLA in California on scholarships. His free time and finances allow him to travel wherever and whenever an opportunity presents. He is currently exploring Africa and plans to tour Colombia in six months. Ben is fluent in several Asian languages, however, he does not know Spanish. For his venture to Colombia, he wants to learn conversational Spanish so that he may communicate while there. Because of his active and extensively mobile lifestyle, Ben is not able to hire a tutor or enroll in classes. He considered using programs for his laptop, but they are too time- consuming and inconvenient for Ben’s taste. Some are also confusing and difficult to understand. Ben wants to learn Spanish but more specifically he needs an option that will fit his dynamic lifestyle while effectively teaching him Spanish. Persona Two0000Name: Mary YangAsian/Female23 years oldMary Yang is a busy full-time college student. She also works full-time as a server at a local eatery. “I get a lot of customers that only speak Spanish. The language barrier makes it challenging for me to do my job.”Occupation: Full-time serverIncome status: Working StudentTechnical profile: Exceptionally proficient with technology. She stays current with her electronics and has high-speed internet wherever she goes.Internet use: Email, academic purposes, social media, blogging.Hobbies: Reading, Writing, Running, Swimming, PaintingGoals: To help brush up on her Spanish without adding the burden of extra expense and time to keep up with her busy lifestyle.Mary does both her work and schooling at the University of Miami full-time. She does not have much free time due to her busy work and school schedule. She is in her senior year and will graduate in the Spring. Mary has previously taken Spanish courses in high school but has forgotten most lessons since then. Many of the customers she interacts with are Spanish speakers, and she wants to be able to communicate more effectively with her customers. However, she is unable to do that because of the language barrier.She is interested in learning Spanish but is discouraged because of her busy schedule and the time it would take to learn a new language. She has researched different websites, such as Rosetta Stone, as well as different applications, but is discouraged by the high prices. She wants to find an application that is cheap, convenient and allows her to fit lessons in during her free time. Appendix C: Test PlanPurposeThe purpose of the DuoLingo usability test is to collect feedback about how users use the DuoLingo app, what problems they may encounter using the app, and what improvements they would like to see to make it easier to learn a second language. The usability test is an assignment for UCF students in the Usability Testing for Technical Communication class. The assignment is to conduct a usability test for something testable. After the completion of a heuristic evaluation and creating personas, the students are to construct this test plan that represents what one group of students will be testing: the DuoLingo application for a mobile phone. Goals and Test ObjectivesWhitney Quesenbery's 5Es is the basis for this test. Quesenbery is a well-known usability consultant and created the 5E’s as an easy way to remember what is significant when testing for usability. The following questions are formed based on the 5Es. Efficient: Can users use the application without assistance? Can users perform scenarios within the predetermined time frame?Effective: Can users successfully complete a lesson or create a profile?Engaging: Is the layout appealing? Do users rate their experience as satisfying or enjoyable? Do their comments (and body language) suggest their experience is positive? Do users want to continue using the application to further their learning of a foreign language?Error tolerant: Do users experience errors? If so, how many? So when they experience errors, do they recover successfully? If they receive error messages, do they understand them? Do they know how to move forward?Easy to learn: Can users get started right away? Does their ability to do tasks improve as they become familiar with the application? Does the application work the way the user expects it to work? Do the lessons adjust to the user’s learning experience?The focus of the usability test will be task-oriented and governed toward how the user responds to the goals listed above. After finding participants with the help of the screening questionnaire created, participants will be given a pre-test questionnaire to get further information about their internet use and expectations in a language tutorial application specifically for a phone. Following that, the participants will be given a set of mapped out scenarios to perform finishing with post-task and post-test questionnaires to obtain wanted feedback about the DuoLingo phone application.Users will be scheduled in 20-minute sessions to include the questionnaires. There are two different tracks as the app contains a learning experience for anyone with various backgrounds in the language. One track will be for persons with no background in the language letting them skip the placement test. The other track will be for persons with a background in the language including a placement test. Both tracks include 12 minutes of task scenarios for participants to complete as many lessons as possible in the allotted time.One moderator conducts test in various quiet rooms, the test will be recorded via a device with a camera to film the subject being tested. The recording of the test will be uploaded on and the link will be included on UCF’s Webcourses for peer review and evaluation.User ProfileOne of the personas created in a previous assignment of this group was of Mary Yang. She was created to develop the user profile for the test plan. Mary is:23 years oldFull-time college studentFull-time restaurant serverTechnical Communications major (probably)Working student – paying her tuition and expenses on her ownBased on the persona of Mary, we will be recruiting participants with the following characteristics:Age: 18 – 25 years oldEducation: Full-time Undergraduate college studentGender: Mix of female and maleProficient with technology; uses technology on a daily basisHave not used the DuoLingo applicationWorks 40 hours/weekNumber of ParticipantsWe plan to test a total of 4 participants: all college students trying to fit learning a second language into a busy schedule. Two participants have no prior knowledge of Spanish. Two participants have some background in the language. That factor may change due to what participants selected. To ensure an adequate number of participants, we plan to recruit eight users based on our screener; four out of the ten are backup participants.Length of Sessions The total length of the session will be 30 minutes including:Welcome and pre-test questions (eight minutes)Task scenarios (12 minutes)Post-task and post-test questions (10 minutes)Scope of ProjectThe test will begin with an overview of the application. The participants will then do the scenarios and complete the session with post-task questionnaires and post-test questionnaires.Participants will not be able to use outside sources or exit the application while completing tasks. That statement will not be said when briefed to avoid influencing participants' actions. Only if/when participants try to use an outside source, the moderator will tell them it is not allowed.Project TimelineWe have had one week of planning and testing creation for the features of the application. We will be utilizing quick testing. The testing conducted during one day for a 20 minute period. See the breakdown of the 20 minutes, in the Length of Sessions section. After a couple of days of reviewing the responses and creating meaningful information, the project will be complete.Responsibilities of Team MembersAttend Project Meetings before and after. We will need three members and three unique responsibilities. One member is for participant check-in, another for technology set-up including application preparation, and one for providing minimal participant support and logging observations.Some roles require certain skill sets that participant must meet.. Participant check-in role will need organizational skill to check in the participants and be able to handle questions and answer them without invalidating the tests. The Technology set-up role will need to be able to handle the product, be comfortable using the application and deal with unexpected problems. The participant support and logging observation role is necessary, for our project will be a distanced action. Very little participant support should be necessary, but a catalog of participant reactions and complications should be recorded. However, moderation of general user activity and exclusion from any outside sources will be the main goals and responsibilities for that role. Appendix D: Screening Questionnaire ResultsUser 1User 2User 3User 4Age23232225GenderMFFMOccupationSales PersonSales PersonAdmin AssistantProduction Manager1. Do you speak any other language aside from English?Yes, Which?XSome FrenchXSome SpanishNoXX2. What is your native language?EnglishXXXXOther3. Check one:I have no background in the Spanish languageXXI have little background in the Spanish languageXX4. Are you currently attending school?Yes,For what?Full-time?Bachelor Full-timeBachelorFull-timeBachelorFull-timeBachelorFull-timeNo5. How comfortable are you with using apps on your phone?ComfortableXXXXUncomfortable6. How often do you use the internet per day?Less than one hour per dayMore than one hour per dayTwo or more hours per dayXXXX7. How often do you use the Internet on your cell phone?Less than 30 minutes per dayMore than 30 minutes per dayMore than one hour per dayXXXX8. How many hours per week do you work?Less than 20 hours a weekBetween 20 -39 hours a weekAbout 40 hours a weekXXXXMore than 40 hours a week9. List all language tutorial apps you have used and for what language.NoneXXXXOtherAppendix E: Post QuestionnairesPost-Task QuestionnaireUser 1User 2User 3User 41. Finding my way around the application was:Difficult (1)Easy (5)1234X5XXX2. The instructions were:Unhelpful (1)Helpful(5)1234X5XXX3. The graphics were:Unattractive (1)Attractive (5)1234XX5XX4. Creating a profile was:Difficult (1)Easy (5)12345XXXX5. The lessons were:Unclear (1)Clear (5)12345XXXX6. When I encountered a problem, I found that solving the problem was:Difficult (1)Easy (5)12345XXXX7. What are two things about the DuoLingo application you liked?Layout and amount of languagesStraight forward, Lesson buildingSimple and fastTeaching instruction, ease of use8. What are two things about the DuoLingo application you did not like?Nothing. “I will use it again.”TypingClicking continuePairing words, needs better word choicePost-Test QuestionnaireUser 1User 2User 3User 41. Please rate how easy of difficult it was to complete a lesson:Difficult (1)Easy (5)1234X5XXX2. How helpful was the design and layout of the application?Unhelpful (1)Helpful(5)1234X5XXX3. How clear or unclear was the terminology used in the application?Unclear (1)Clear (5)12345XXXX4. How much time would it take before you felt you were proficient in using the application?10-30 minXXXX1-2 hours3 or more hoursExplanation“It would take hardly any time to be proficient since it’s a simple app.”“The design and the app was explained.”5. Please rate your overall satisfaction with the application.Unsatisfied (1)Satisfied (5)1234X5XXX6. Would you recommend this application to another person interested in learning another language?YesXXXXNoExplanation“Friends might like mobile / convenience.”7. Do you think you learned Spanish from this experience?YesXXXNoXExplanation“I only did one lesson.”“Stirred up old feelings.”“It was easy to use, I learned quickly.”“Understood the basics of Spanish”Appendix F: Moderator ScriptIntroductionHello. Thank you for participating in this usability test. Your feedback will be used to complete our final report. Videotaping PermissionWe are going to be videotaping the session. The reason for this is so the other group members that are not present will be able to also view the session if they have any questions. Please complete the permission form stating that you allow us to videotape.Introduction of the TestToday, you are going to be working with the DuoLingo smart phone application. You will be given some time to complete your tasks on the app then I am going to ask you some questions to get your feedback. Your feedback is really appreciated so telling what you are doing and why you are doing it is very helpful. I will be sitting next to you and I may ask you questions from time to time. Do you have any questions before we start?Pre-test QuestionnaireThis is a questionnaire to help my group understand your experience using phone applications and some things about your lifestyle. While you are completing it, please explain the answers you are writing and why you are writing those answers. [After questionnaire] Thank you for your feedback. That was great.During TestWe have a few scenarios for you to complete. Remember, we are not testing you. We are testing the application based on your reactions about it. Because of this, please remember to think out loud. Post-task and post-test questionnairesPlease complete this short questionnaire about the scenarios (post-task). After that, I would like you to complete this questionnaire about the app and test overall (post-test). Again, with these please explain what answers you are writing and why you are writing them. These are the last things I need from you. You are free to go after completing these. Thank you so much for your time and feedback. Appendix G: Session Logs/ Individual AnalysesJennie Larner’s Notes and AnalysisAnalysis of Findings for User 1:Using the DuoLingo application, this user wanted to learn to speak Spanish conversationally to better communicate in his work environment.This user did not need assistance accessing or using the application. He was able to open the application, create a profile, and start the lesson in less than 3 minutes. Starting the lesson was easy to find and easy to do. However, he did have some difficulty with one lesson that did not give much instruction. This lesson required the user to match words that related to one another. Some words were in Spanish and some words were in English. There was no guidance on whether the user needed to match the English word with the Spanish word. He picked a wrong answer and received an “error” notification. He was stumped for a while, trying to figure out how to answer the question with little guidance. However, he was able to figure out a solution and moved on. In conclusion, User 4 was able to successfully complete a lesson and create a profile within the time frame given.This user found the layout of the application appealing and found the application enjoyable overall. Throughout the test, this user’s body language was relaxed. There was no pressure evident and the user was at ease the entire time. This user’s body language and comments showed a positive experience with the application.This user’s ability to complete tasks improved the more the user used the application. The lessons adjusted to the user’s learning experience by repeating questions that were missed to ensure the user understood the concepts. The application proved to be easy to learn.Analysis of Findings for User 2:Using the DuoLingo application, this user wanted to learn to speak Spanish conversationally for work and travel purposes.During the usability test, this user was relaxed and focused. Throughout the test video, she was extremely focused on the lessons of the application and seemed to enjoy the application. From the test video, this user was able to work through the lessons quickly and efficiently. She did not seem to be stuck on any questions for long periods of time. Based on the post-task and post-test questionairres, this user was satisfied with the application and found it useful. She liked the simplicity of the application and how straightforward it was.One negative aspect about the application this user did not like was the typing aspect. Some questions required typing and this user did not like this.Two positive things this user liked about the application was how straightforward it was and the lesson building aspect of the application.Analysis of User 3:Using the DuoLingo application, this user wanted to learn to speak Spanish proficiently for work, school and location purposes.During this usability test, this user was relaxed and seemed to enjoy using the application. She became excited when she got a question correct. From the test video, this user did not have any issues using the application. She was able to complete each task in a timely manner. She was very satisfied with the application and found it very easy to use. She uses her phone to complete tasks regularly and liked how simple it was.One negative thing this user did not like about the application was having to click continue when an answer was correct. She would like to have the application automatically switch to the next question when a correct answer is given.Two positive things this user liked about the application was how simple and fast the application is.?Analysis of Findings for User 4:Using the DuoLingo application, this user wanted to learn to speak a few words of Spanish for work and school purposes.During the usability test, this user was relaxed and not tensed. He focused on the tasks of the application. The user did not seem to have any issues with using the application. He was able to complete the tasks successfully and in a timely manner. However, he did not realize that he created a profile until the end of the usability test. This could be an issue for further users. Overall, this user was satisfied with the application and found it very easy to use. He stated he would recommend this application to friends.One negative thing this user did not like about the application was the word pairing activity.Two positive things this user liked about the application the teaching instruction and the ease of use of the application.?Arsheena Mohamed’s Notes and AnalysisUser 1 had an overall good experience. The app was extremely easy to use?with?a favorable layout. Her participant was satisfied with the app and would use it again. He expressed a comfortable and enjoyable attitude using the app. It seems that he found the matching pairs questions to be a bit confusing at first. The lessons did not provide any review before throwing him into the lessons which were a concern.?User 2 overall thought the app was easy to use and an enjoyable experience. She has a problem with phone apps in general because she doesn't like trying to accomplish tasks on a small cell phone screen if she doesn't have to. She finds the app engaging and attractive as well as convenient. She like that users are able to share their progress with others as well and the options to invited friends. She thinks the app is efficient as far as lessons, pointing out that they build on each other.?User 3 deems the app simple and fast making it easy to use and convenient. The layout and design was favorable. She feels like she learned from the app quickly. Clicking continue was not favorable instead of grading automatically. Solving problems in the app were easy to fix because the app is clear in directions.User 4 seemed comfortable using the app. The instructions were helpful and the lessons made sense. He said the graphics were somewhat attractive. He claimed to have learned some spanish just from the short testing experience. ?Overall, his experience was satisfying. He mentions that the app would be good for practice Spanish more than actually learning at first. This is because the app did not have a review of the vocabulary before asking questions about it. Commented negatively on the word pairing needing better choice. Matching the pairs was completed through process of elimination. He said that it would have been better to have match the pairs first then check answers all at once instead of the answers being corrected right away. There was a point when he misunderstood the instructions but the app makes it easy to realize what was done wrong.?Belinda Provost’s Notes and AnalysisAnalysis of Findings of User 1:Post-Task assessment of the application:Found app very easy to useThought instructions were helpfulBelieved graphics were attractiveFelt that creating profile was easyClaimed lessons were clearParticipant identified two positive factors about the DuoLingo application was the layout and the amount of languages offered through the app. Participant thought there were no negative factors about app and would use it again.?Participant disclosed in the Post-Test Evaluation:Overall app somewhat easy to useDesign and layout of app was very helpfulApp terminology was somewhat clearDeemed user could be proficient with app within 10-30 minutesUser thought design and that app was explained helped with becoming proficientOverall was very satisfied with appWould recommend app to othersThought learning any language has its benefits to a variety of peopleParticipant did not believe he learned Spanish from the lessonStated reason was because he only did one lessonAnalysis of Findings of User 2:Participant’s Post-Task assessment of the application:Found app somewhat easy to useThought instructions were somewhat helpfulBelieved graphics were attractiveFelt that creating profile was easyClaimed lessons were clearStated that solving problems on app were easy to solveParticipant identified two positive factors about the DuoLingo application was that it was straightforward and lesson building. The the negative factor was that typing on a small mobile device was challenging.?Participant disclosed in the Post-Test Evaluation:Overall app very easy to useDesign and layout of app was somewhat helpfulApp terminology was very clearDeemed user could be proficient with app within 10-30 minutesOverall was somewhat satisfied with appWould recommend app to others/believed friends would like convenience of mobile appGained learning from the one lesson/stimulated memories from previous lessonsAnalysis of Findings of User 3:Participant’s Post-Task assessment of the application:Found app very easy to useThought instructions were helpfulBelieved graphics were attractiveFelt that creating profile was easyClaimed lessons were clearParticipant identified two positive factors about the DuoLingo application was the simple and fast. Participant thought the one negative factor about app was having to click continue when answer was correct.?Participant disclosed in the Post-Test Evaluation:Overall app very easy to useDesign and layout of app was very helpfulApp terminology was very clearDeemed user could be proficient with app within 10-30 minutesUser thought it would take hardly any time to be proficient since it’s a simple appOverall was very satisfied with appThought it was easy to use and user learned quicklyWould recommend app to othersAnalysis of Findings of User 4:Participant’s Post-Task assessment of the application:Found app very easy to useThought instructions were helpfulBelieved graphics were somewhat attractiveFelt that creating profile was easyClaimed lessons were clearStated that solving problems on app were easy to solveParticipant identified two positive factors about the DuoLingo application was teaching instructions and ease of use. The the negative factors were that he did not like word pairings and thought better word choice was needed.?Participant disclosed in the Post-Test Evaluation:Overall app very easy to useDesign and layout of app was very helpfulApp terminology was somewhat clearDeemed user could be proficient with app within 10-30 minutesUser thought design and that app was explained helped with becoming proficientOverall was very satisfied with appWould recommend app to othersUser understood the basics of Spanish after lessonLearned some Spanish from one lessonMatthew Stofflet’s Notes and AnalysisUser 1’s testing of the application went well. We got good feedback from the user since they had prior knowledge in the Spanish language. There was some complications when try to understand the little games the app has you play and didn’t get the necessity of some activities. But the participants overall reactions were positive and showed promise for the apps potential.User 2’s DuoLingo application testing went well. She really struggled with the level of difficulty and it took a while for the participant to get used to the using the interface with understanding the language. The error tolerance between typing on an English keyboard and the ability to understand the accent of the speaker in the app made the answers difficult. Overall the participant awards the app for being easy, clear, attractive, and straightforward.User 3’s testing went well. The feedback from this participant was very verbal and shared her mind about the layout and the construction of the interface. The engagement of trying to figure out the puzzle behind the activity was what I drew most from this test. The simplicity and fluidity of the app was the top feature pointed out during this test. The feedback on the ease of satisfaction from correct responses was great.User 4’s testing went well. The participant had good knowledge of the app market and availability of features on smartphones. His review went well and found the problems easy to figure out and rewarding to complete. The app he found attractive and to the point and thought it would be a good tool for learning the basics of the language. The participant found the quirks of the app appropriate for its method of teaching and the medium is was able to teach on. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download