Www





Usability test

Findings and recommendations

Intro 4

Hotmail tested at Company X 5

Reader's guide 5

General findings 7

Positive conclusions 7

Relevant content and features 7

User friendliness 7

Users liked the graphics 8

Major problems 8

Speed 8

Attachments 8

Messages difficult to save. Saved sent messages difficult to find. 9

Global navigation missing on several pages 9

"Move to" area 9

Pages and scenarios 11

1. Registration 11

2. Login 12

3. Logout 13

4. Viewing Hotmail with or without frames 14

5. Customization (under the 'Options' section, particularly "Preferences" for viewing style) 14

6. Performance issues (speed, etc.) 15

7. Compose 16

8. Password Retrieval 19

9. Attachments 20

10. Sorting messages in your InBox 23

11. InBox 23

12. Reading a message 25

13. Addresses 26

14. Folders 27

15. POP 29

16. Advertisements 30

Answers to the general questions raised by Hotmail management (in the scenario paper). 31

Hotmail and Microsoft 32

Hotmail logo vs. MSN logo 33

Other issues 34

Related links - an inactive area of navigation 34

Lack of status indication on emails 34

Email Greetings 34

Language 35

PART TWO 36

The testing 36

Questionnaire (English version) 37

Test persons 40

Technical test configuration data 41

Tested Hotmail version SCREENDUMP 42

Background 42

Who is Company X 42

General purpose of testing 42

Scenario Paper (original) 43

Intro

First, a notice to the readers of this report: The true scenario surrounding the report is that it is created as input in the CUE-2 "Comparative Evaluation of Usability Tests" in connection with the CHI99-convention (Computer Human Interaction).

Company X agreed to carry out a "life-like" test of the web-based email service free of charge. To fulfill the convention intentions of a realistic evaluation of test methods and results, we handled the approach from Rolf Molich (who is in charge of coordinating the Evaluation) the same way as we would any approach from a company wanting a usability test of their net-based service.

Here at Company X we believe that this was the best way to meet the goals of comparativity between the different usability labs addressed by CUE-2 on CHI99.

Nevertheless the test procedure deviates from the normal process in a few ways. Read a short run-through of the deviations in part two of this report.

We ended up testing Hotmail with a version of the think-aloud test. We had one pilot test and two days of actual testing with target group users. Day one with 4 inexperienced Hotmail users. Day two with 4 experienced users. Read more about the test setup in part two of this report. Here you may also read some background material on Company X and on the test method used.

Apart from the chapter "deviations", from this point on, the report is to be read as if it was a normal test report with a budget around 20,000 US Dollars.

Hotmail tested at Company X

Hotmail/MSN have wished to have their service usability tested by Company X. This step is taken so that Hotmail can further strengthen their service towards old and new users.

The test goal is to reveal the service's current possibilities and restrictions in the interaction with the user. By doing that it will be possible to optimize the future usability of the site. The test focuses on specific areas of interest as the Hotmail market managers formulated them.

The following report describes the general test findings as well as the specific findings. And in addition gives a list of operational suggestions as to how problems can be overcome and general improvements handed.

Reader's guide

We have split this report into two parts. The first part deals with the findings of the test. The second describes the premises for the test and the test method. Attachment you will find the original questionnaire and list of test assignments used during the actual testing. Attached is also a "scenario paper" dated October 8. This paper is Hotmail's marketing management's original input to the testing, explaining the specific areas of interest.

We combined this "Hotmail marketing management checklist" with a list of key areas of consideration found during our preceding heuristic evaluation of Hotmail. We used this combined list as the primary input in the process of creating the basic questionnaire and a number of assignments that worked as the backbone of the actual test.

This report however is not organized in strict accordance with this list of importance.

Here at Company X we believe that a test report should be organized according to the importance of the problems encountered rather than the areas considered interesting beforehand.

In accordance with this philosophy the first part of this report describes the main conclusions, the most important positive aspects and the major problems.

After these general findings we go through the different pages and processes of the service. Here the critique is concrete and followed by constructive recommendations for future improvement of Hotmail.

Normally we would organize this part of the report in accordance with a flowchart of the service. In this report we organize the more specific findings according to the list of importance supplied by the Hotmail management (with a few minor exceptions and with the interest areas as described in "part one" of the scenario paper, edited and treated together with the general questions afterwards). Finally we present a number of various issues that we felt important to include in this report (more general critical aspects of the service together with a list of "Nice to have" features inspired by the testing).

In the report all quotes from test persons are written in italics. All

references to pages, links or text/icons inside the Hotmail service will be written in bold letters.

The reader may experience a few repetitions through this report. The repetitions have survived the second edit because here at Company X we believe that some single paragraphs should be readable on their own.

General findings

Positive conclusions

Relevant content and features

Users found the general service on Hotmail – sending and receiving web-based email – very attractive. Hotmail's universal accessibility tested as the most valuable overall quality.

Among the features of Hotmail, users especially liked the POP possibility, the Quicklist, the Address Book and the Save Address function.

User friendliness

The basic functions on Hotmail works well for both experienced and inexperienced users.

The inexperienced user

For the inexperienced user, link-labels and narrative structures seem to be working intuitively when executing the basic functions. Exceptions to this impression will be discussed later.

Users often felt very well serviced by the many "support texts" found around the site. They seem to anticipate many relevant user problems. Even though users tend not to read the texts before they have encountered some kind of problem, the fact that the is there, its position and tone witness a user-oriented way of thinking service and communication.

The experienced user

Also the experienced users found the website easy to use. They all expressed their satisfaction with what they found was a high rate of overall functionality. Experienced users were seldom or never annoyed by the main functional routines on the website. They also appreciated the possibility of being able to tailor the site. For instance users liked the possibility of disabling the Confirmation-page that by default appears after sending an email.

Users liked the graphics

In general users found the design (default) attractive. It was described as clean and nice without too much graphical ornamentation. Only one user described the graphics as messy.

All other users appreciated the lack of disturbing "noise" in the use of graphics. Often the users connected the subject of design with the subject of downloading speed. It is our general experience that it is difficult for users to make a clear distinction between functionality and design. Experienced users seem to prefer sparse layout without graphics because it also signals pace and efficiency. And they seem to do so in the case of Hotmail too.

Major problems

Speed

After 3 AM (CET-1 – 9 AM Eastern Time) Hotmail is too slow to use, ass speed was reduced dramatically regardless of Internet provider and the speed of the connection. Here at Company X we consider this to be an extremely serious problem for Hotmail. The basic possibility of immediately being able to send and receive email is a fundamental condition for the future success of Hotmail.

Attachments

Both the function of attaching files to email and the function of extracting attachment from incoming email caused problems.

Experienced users had few problems making attachments to emails but were in general frustrated by various problems extracting attachment from incoming email.

The inexperienced users also discovered problems attaching files to emails. The whole routine of attaching was difficult to understand for people with little knowledge of this particular feature.

Messages difficult to save. Saved sent messages difficult to find.

All users found it annoying that they had to remember to activate the checkbox Save sent massage if they wanted to save an email. Some users have had the experience of loosing imported mails because they forgot to activate the checkbox. For new users this problem was emphasized by the fact that after sending an email (and not knowing for sure whether or not they had actually saved the message), they had problems relocating the exact location of the email.

Global navigation missing on several pages

Users found that the global navigation (left-side navigation) was missing in several areas. This was especially considered a problem on the compose/attachment pages, leaving users with a feeling of not knowing where they were and how to get back. All users would prefer the global navigation to be visual at all times.

Said about global navigation:

"It would've been nice if that one [global navigation] didn't disappear"

"Move to" area

The Move to function at the bottom of the InBox works as a navigation tool, bringing users the folder selected on the drop-down menu.

The main problem relates to the fact that a few users didn't understand that Move to related to the object(s) (emails) above. Secondly, users had difficulties reading Move to in the context of a sentence, as hey were not able to relate Move to to the words in the dropdown menu. However, this relation turned out stronger than the one first mentioned. Consequently users started to understand and use the Move to area as a navigation tool – and not as an easy way of moving emails to specific folders.

Users also had problems distinguishing between Trash and Delete.

Pages and scenarios

The following conclusions are loosely organized according to the original order of importance list made by Hotmail.

1. Registration

Most new users went directly to the sign up-page. Only one out of four inexperienced users read Who Should Sign Up. This user wanted to read the full content of the legal pages and would have preferred to have the possibility in my language (native language). The same user also searched the pages for quite some time looking for more explicit information on the subject of Hotmail being free of charge.

The major problems on the registration pages were found on the actual page of registration. The problems occurred in the actual user-input-process, where the users were insecure about which input-fields they were supposed to fill in and which parts they could leave out No user got through the registration pages smoothly. All users were guided to the error pages at some point during the registration. One user was sent to the error-page three times.

The most common problem with registration was input-fields being left unfilled. Especially the input-field "state/province" were awkward for the European users.

The error page worked well in the sense that users quickly understood that they had made a mistake, and were fast back on track (typing in the data they left out/typed wrong/whatever). Most users did not read the actual message of error. The red font didn't seem to attract attention. It seemed that it just made the page confused.

In general, all users managed to do the registration without encountering any severe problems. On the other hand all users experienced some minor annoying problems. We want to emphasize the critical role of the registration pages by pointing out the fact that users at this stage are not yet loyal to Hotmail.

Said about registration:

If it's out there it can't be that bad (User choosing not to read legal stuff)

It's all American. Had it been European [own nationality] I might have chosen some of them. (About subscribing to WebCourier services)

The problem is that I don't have the time to check all the information. The information I'm interested in I will find anyway. (About subscribing to WebCourier services)

Company X recommends:

- that the legal pages are translated to major languages in order to address the need for a more precise level of information.

- that there is only one registration form (for users of all nationalities). And this form should name the input-fields in a way that feels natural also for non-American users.

- that the individual input-fields on the registration form is marked with some kind of indicator signaling to the user which parts are obligatory and which parts can be left out. An example of this can be seen in the Addressbook where the nickname input-field is marked required.

2. Login

In general, users had no problems understanding the login procedure. The areas Loginname and Password seem to be well positioned on the front page.

No test users accessed their Hotmail account through MSN. During the test, users were asked to log out (for a later re-log in). Most users tried to return to Hotmail by writing the name URL in the address-box in the browser. Only one user saw the Hotmail-login area on MSN.

Said about login:

"…" (they just did it)

Company X recommends:

- that the login area is kept the way it is now.

- that users on the logout-page (the page between having activated the logout button and MSN) are informed that it is possible to login to their Hotmail account via MSN.

3. Logout

Most users "logged out" of Hotmail by closing the browser window. No users expected that they would automatically be transferred to MSN if using the Logout feature. Some users anticipated some kind of commercial after clicking Logout.

The purpose of the Logout button was not clear to all users. When asked, most users reasoned that it would prevent others from taking advantage of an open email-account. Some users (especially the inexperienced) felt misled by the Logout feature, when after clicking Logout they used the browser's back- button, and personal Hotmail pages were shown (generated from the browser cache).

Said about logout:

I would have stayed on or closed the browser. It is the first time I notice the logout button. I don't know MSN.

(Used Hotmail for 3 months)

Company X recommends:

- the logout function is configured so that attempts to "re-enter" Hotmail through the back door (using the browser's back button) is impossible. If this is not a technical possibility, users should be better informed (on logout) that they should manually clear the cache of their browser to be safe from unwanted readers.

4. Viewing Hotmail with or without frames

Neither the experienced nor the inexperienced user had any explicit preference for frames/no-frames. All users automatically used the frame version. None of the experienced users used the no-frame version at home.

One experienced user during the test changed to the non-frame version and was confused about the significant change of look. Particularly the user had difficulties with the replacement of the global navigation (The global navigation is moving up in relation to the content when the non-frame version is used.)

Company X recommends:

- that Hotmail aims at increasing similarity between the frame and the non-frame version. The more alike the better.

5. Customization (under the 'Options' section, particularly "Preferences" for viewing style)

No users had thought about changing the default looks of Hotmail by themselves. When challenged, most users found the options quite easily. Two users visited the preference-page by themselves when examining the service, but did not make any changes to the default.

On the actual preference page, users revealed some usability problems. In the Appearance area, users didn't realize that they had to click Customize before a given change (for example "Business" or "White") to come into effect. Secondly, both users left the Preference page without activating the OK-button at the bottom of the page. One of these users noticed that the background changes were not actually taking place, but came to the conclusion that he probably had to log in again to make the change active.

Said about options/preferences:

I would never use this area - as long as it works, I'm satisfied.

Company X recommends:

- that the Appearance area is changed so that it becomes clear that the Default/ Customize has to be selected before a change within the categories can take place. A way to do this could be by giving all areas (like Appearance) their own OK buttons. Alternatively, users must be noticed at the top of the Preferences-page that they have to activate the changes they make on the page.

6. Performance issues (speed, etc.)

As specified in the test configuration data (below) we tested Hotmail on a 28.8 connection. Between 9 a.m. and 3 p.m. (GMT +01:00) the connection worked without significant problems (Default-mode, frames). The impact of using the non-graphics default version as opposed to the graphics version was not tested.

After 3 p.m. (9 a.m. Eastern time) speed was reduced dramatically. From 4 p.m. it was not even possible to complete a test. At one point we started a registration routine at 6.30 p.m. (without users on a 512kb connection) and finished 1 hour and 30 minutes later.

Said about performance:

It looks as if they made it as fast as they could.

I have a Hotmail account, but I don't normally use it because it is too slow.

Company X recommends:

- that the connection problem is analyzed and dealt with. No matter where the problem occurs, we considered this a serious obstacle for the future success of Hotmail.

7. Compose

This area was typically tested task-driven. Users were given tasks like "write a letter to xxx". In general, users were quite satisfied with the looks and feel of the compose page/function. A few users mentioned that Compose was not the first word they'd associate with writing/sending mail. Two users had consequential problems understanding Compose. Both users suggested that the button be renamed "Write mail".

After hitting the Compose button, many users reacted to the fact that the global navigation disappeared. Users experienced some insecurity about how to get back to the InBox, and this gave the users in general a feeling of insecurity.

The first four input-fields were quite well received by the users. All users intuitively started entering the right data into the input-fields. Most of the users weren't sure what the bcc: input-field was used for - but this did not seem to affect the efficiency of use.

Users reacted differently to the eight buttons arranged in two rows of four surrounding the input-fields. Some users found that they were messy and should have been arranged in a more controlled way. Others happily recognized the functions from their normal mail client.

On the hole, the negative aspects seemed to be connected with the lay-out, whereas the available functionality was satisfying for almost all users.

One user missed the opportunity of telling his friends about his new email-account.

Apart from maybe the naming of Compose, Save outgoing messages was the only serious problem on the Compose page.

Many users had bad experiences with the Save outgoing messages click-box. Inexperienced Hotmail users expected that outgoing messages would be saved automatically. They overlooked the clickbox completely and lost some sent mail in the process of learning that they should click this active first. The experienced users knew the convention, but sometimes (to their frustration later) they forgot to use the click-box. Users in general understood why Hotmail probably wanted to prevent them from saving useless emails, but everybody thought that the save outgoing messages click box should at least be clicked on as default.

After having pressed the Sent button, most users expressed satisfaction with being met with the Sent message confirmation page. Some users said that they would probably find the confirmation annoying after some time (if the system proved reliable). But the help text on the Sent message confirmation page, explaining how to deselect the confirmation, was seldom read. Even by users expressing the need to deselect this service. On top of that, the users that actually read the deselect support-text thought it would be difficult to remember the steps needed to deselect the service. Users appreciated this information, but were annoyed that they weren't able to activate the bold references in the text that potentially would lead them directly to the relevant page.

The issues of sending and receiving mail with attachments are discussed in the following paragraph.

Said about Compose:

This looks like what I expected

Now I don't have the other possibilities anymore?

This is the same as my normal mail client.

This is a mess

I think "Compose" should be labeled "Write mail"

This is very easy

How boring…

Just what I want

Here I think it would be great if I could directly to the bold text-parts. (User concerning the text explaining how to get rid of the "confirmation on sent mail"- page)

The first thing I want to do [now that I registered] is to tell all my friends about my new email address.

Company X recommends:

-change the name compose to write mail.

-keep the global navigation available in the compose process.

-save the mail as default.

- Make a "click here to tell your friends of your new email address"-button on the first InBox view. This should take new users to a new address-mail-template ("Hey friends I got new email -try writing me"). Supported by intuitive input-fields for the user to submit email addresses of friends and contacts (if the user takes advantage of this service, the addresses the user provides are automatically added to the personal Addresses).

-make the selection of the sent message confirmation service a clickable option on the page itself. The convention is known from a number of other utilities: "If you do not want to see this confirmation page in the future, click here []". Generally all information on future navigation and content should be non-hypothetical, meaning that written information and help are supplied with hyperlinks or accessible on the spot.

Alternatively, some major changes in the overall way of thinking interaction could be considered. Let's discuss these perspectives at the briefing.

8. Password Retrieval

In the Password Retrieval scenario, users experienced some problems. The scenario can be divided in users without a Password Hint Question and users with a Password Hint Question. All users accessed the Password Retrieval help via the front page link: Forgot your Password?

Users without a Password Hint Question

Users without a password found the first pop-up browser page ("Forgot Your Password - type your Hotmail User Name") was irrelevant and didn't address their problem. It is obvious that this page is made strictly for users with a Password Hint Question. All users without a hint question typed their password anyway in the input-field.

The following "Password Help" gave users insufficient information about the routine taking place. Few users understood the written test i.e. the concept about activating an alternate email access to send a blank email.

Users with a Password Hint Question

These users easily complete the routine but are worried about the insecurity of the system.

Said about Password Hint Question:

One guy already registered an email address with my surname. I know that there are only about 5 people worldwide with that surname. And here I can see that his hint question is "What is your mother's maiden name?" If I wanted to, I could easily find out his password.

Company X recommends:

- that both types of users – those with and those without a Password Hint Question – are given the opportunity to identify themselves. This identification page should include a "Can't remember password" link, directing users to a page with "user with a Hint Question" opportunities.

- that it becomes possible to contact Hotmails password-service directly through the browser and not via email. Taking into account the fact that Hotmails greatest strength is probably the universal accessibility, it is very important that users can address the password service independently of an email-client/address ,for instance through a "live" help (chat/conference). This advice is also based on the assumption that many Hotmail users haven't set up their ordinary email account supported by their Internet provider.

- If the service is maintained email based. This procedure could be made more user-friendly: If the only information that Hotmail supplies in the answering letter (after receiving a blank email) is what is already specified in the help-center, users would be better off with a simple link. If Hotmail are willing to supply more information (whatever that might be?) after receiving a blank email, then, instead of users sending a blank email, Hotmail should simply ask users to fill in the email address where they wanted the help sent.

9. Attachments

In general, users had some problems attaching files to their mails.

At first, thing some users seemed a bit surprised that the attachment button transferred them to a new page. The three steps did not work intuitively for all users. And instead of reading the instructions for the three steps, users generally used the method of trial and error. Some users tried to enter the name of the file they wanted to attach in the first input-field. They did not know that they had to specify the path. Using the browse button may have been extra problematic, as we were testing Hotmail on a non-American operating system. The browse button referred to in the help text is named differently in non-American operating systems.

Step two of the attachment procedure had some users hanging as well. Maybe due to the imperative character of the step two button text Attach to Message, some users expected to be taken back to the compose process directly, since they already attached their message, ignoring step three of the procedure.

Users were generally satisfied with the way the attachments were displayed in a separate window. They praised the fact that this window automatically gave updated information about the limit in size of all attachments as well as the total size of all currently attached files.

On returning to the compose process (via the done button) a few users noted that they would have expected the attached files to be displayed in the same manner as on the previous attach file page.

Some users simply forgot that they had not actually sent the mail on returning from the attachment page. They started doing other Hotmail stuff expecting that the mail was already sent (now that done had been pressed).

Receiving attached files

Receiving attachments turned out to be a problem for all Hotmail test users. The experienced users knew this problem and said that they always tried to tell people not to send important attachments to their Hotmail address. Hotmail simply could not be trusted for handling attachments. Not even all the experienced users knew that there was a maximum size on incoming files (and attachments).

Said about Attachment:

I guess I'll have to push "Done" too?

This is simple to use. Actually it is more simple than Outlook.

Nobody I know of is able to use attachments in an efficient way. I don't know what is wrong, but I know I've got more than one mail containing important information that I'm not able to read.

I'm only trying to send a file (user not wanting to push Done)

I thought I would be coming back after hitting Attach File

Company X recommends:

- making the attachment procedure take place on top of the compose page. This solves the problem of the different conventions for showing attached files. And it makes it easier for the user to remember the main goal (sending mail). This idea of layers rather than pages for interactive sideprocedures can be discussed in depth at the briefing.

- that after pressing the attach button, you go straight to the "browse for file" window. The idea of manually writing the name of the file is lost, but you gain the flow and avoid a source of misunderstanding.

- strengthening the process of receiving attachments. Instead of trying to open the attachment, Hotmail should give the user the choice of storing the attachment or view it right away. If the viewing procedure then fails (which happens more often than not) users would have the chance of opening the file with a tool of their own choice. It is extremely important for Hotmail that users feel secure about receiving attachment.

- that if attachments are too large to store, the user gets notified that an attachment (this size) has been sent to them but has not been delivered due to size problems.

- raising the size limit for incoming files (and attachments).

10. Sorting messages in your InBox

The test we conducted did not supply a great deal of qualitative data on the users' experience of sorting messages in the InBox.

None of the experienced users used this service. And no users used the mail sorting service of their own accord during the test. The reason for this is partly the test design, but is also partly due to the lack of visual indication that this service is an option. The few users who were encouraged to try out the service (through designated assignments), seemed partly surprised that mail-sorting was possible. But they intuitively looked for the service, in the right place (clicking on the column labels). One user customized the appearance of his interface (changed from default to graphic interface) and discovered by this detour, that he actually had the possibility to rearrange his emails (due to the graphical change, with the little arrow on the headlines in the "button bar".)

Said about sorting messages in your InBox:

That's cool

I did not know this was possible. Hmm… That's pretty neat.

Company X recommends:

- that the service is kept active, but better advertised/indicated for the user. Having the orientation arrows visible from the start could probably do this, but should be considered as a trade-off on the general speed.

11. InBox

(Note: The InBox is not part of the original list from the scenario paper, but included at this point because of its natural connection with the next item: Reading a message.)

InBox - overview

All users had a good basic understanding of the InBox. Users understood all basic functions except the Move to navigation (see Major Problems). In addition to this problem, users also noticed other problems.

The text area just underneath the headline: Check for:[New Hotmail | POP Mail], tested positive, in the sense that experienced users understood it as a way of updating the InBox-area. On the other hand, some users found the links confusing because they had the impression that they were already watching an area with New Hotmail. Some of the inexperienced users activated the link, and were surprised that nothing happened.

Many users complained about the fact that it is impossible to see whether or not attachments are sent along with the email.

Some experienced users missed the opportunity to automatically forward incoming email. They wanted to keep their Hotmail address alive for travelling etc. But they did not want to check it every day when they had access to their regular email address.

Users rarely used the Related link – the MSN dropdown menu was not used at all.

Said about InBox - overview:

I don't know weather or not my mail contains attachments.

I don't care how many messages I have in my InBox - I just want to know if I have any new mail.

The worst thing is that you don't have direct access to your latest email. If I have many mails in my Inbox I always have to scroll to see the latest mail.

Company X recomends:

- that the number of new mails are displayed in a more visually distinctive way. In the tested version of Hotmail, the number new messages were displayed (in normal characters) and after the number of total emails in the InBox.

- that the latest mails are displayed at the top of the InBox (not at the bottom).

12. Reading a message

Reading a single message

In general, the test users responded positively to the experience of reading messages. Most liked the way the two "button bars" made a kind of border around the message. They liked the fact that banner ads were far away from the interesting context.

Some test persons thought that there was a bit too much "mess" at the top of the message. The data showing the way the mail had moved before reaching the InBox seemed uninteresting to most users. The different email addresses to which the mail had been sent are not hyperlinks. One user tried to "clean up" via Options/Preference by setting the Message Headers option at "None", but thought that there was too much information left out when he used this option.

URL recognition

Working with the URL recognition, most users seemed pleased that they could easily scan out and click on links in the message. One user did not like the fact that Hotmail had apparently read his mail and changed the source code (from a single email address string to a Hotmail compose link). Even though he agreed that this was smart, he felt that Hotmail had broken tacit agreement that the mail was private.

Said about reading a single message:

It is difficult to say which parts is the actual mail. (After opening mail).

What I find between these two [the button bars ed.] I expect to be what somebody sent me. Not something that Hotmail have read and edited.

I always have to find out the exact point where my email starts. (Experienced user)

If they can pick out emails from my letters I guess storing them is a natural next step, and that I don't like.

They promised in the introduction that they would not read my mail, but in some way they do.

Company X recommends:

- that Hotmail refrains from changing the source code of users' letters. The email recognition could be used better by making click'able links out of the email addresses often found at the top of the letter.

13. Addresses

All users received this feature very well. Users found it easy to add addresses both directly via the Create link and by the Save address link in received mails. For some users the Create link in the upper right corner on the Addresses was a bit out of visual focus. One user searched for an opportunity to add an address without seeing the Create-link.

Users found the Create page easy to understand and very useful. Inexperienced users were insecure whether or not they were supposed to fill in all the input-fields on the page. More experienced users understood that they weren't supposed to fill in all input-fields. All users filled in the Nickname field and the email field, but no users filled any other fields. Experienced users kept additional information (addresses etc.) in manual or other traditional software-based calendars.

Said about Addresses:

Why do they want both a nick-name and a real name?

I would like to have my other addresses here as well.

How do I add a new address?

Company X recommends:

- that users are given the opportunity to export mail to other programs.

- that users are given the opportunity to exchange calendar information between the Hotmail Addresses and more common calendar applications like MS Outlook and PalmPilot (this is currently done by among others)

- that the Create link is replaced to a more central eye-catching position (right top) on the page. We expect that the Addresses will primarily be used when users want to create a new address. Given the Quicklist feature the actual list of addresses in the Addresses becomes less important for users.

14. Folders

The majority of users thought it would facilitate their daily use of Hotmail if the sub-folder Sent massages was moved up to the global navigation. Also on this page, the Create link in the upper right corner was a bit out of visual focus. No users saw the related link Find message in the upper right corner – not even when they were given the task of finding an old email beforehand. All users used manual search to solve this task. (This way of solving the problem may also be related to the – in most cases – limited number of mails in the various sub-folders).

All users found it easy to create new folders. One user revealed some uncertainty concerning the delete option.

Experienced users also expressed the need for a general email compatibility with other applications. It seemed desirable to be able to import and export emails to and from the user mail-client. One test user had desperately tried to save her love-letters from her overseas boyfriend on a floppy disk, but had failed the attempt. She was worried about what would happen to her precious mail as the total volume of mails on her account came close to the maximum account-size limit of 2000k.

Said about folders:

I'm missing the opportunity to be able the read my emails off-line. I want to be able to save them on my local computer or on a floppy disk.

I have tried to save my mail on a floppy-disk but didn't succeed.

Company X recommends:

- lifting the sent mail folder up to global navigation level, so that the sent mail is available right underneath the InBox.

- alternative: making it possible to see saved sent mail in your InBox. Maybe a clickable option presented on the page of the InBox. "View sent mail you have saved in your InBox []"

- that the Find Message option gets a more visual status.

- that the user gets a confirmation of moved mails. "You have now moved x mails from y to z" preferably supported by an icon of the two folders (with names) seconded by an arrow, some envelopes and a number indicating the movement of a certain amount of mails.

15. POP

All users found the POP-mail feature very attractive. Some experienced users used the POP-feature already and rated it the second-most valuable feature after the universal accessibility. The inexperienced users had some difficulties understanding the feature (and the word POP), and didn't immediately understand what to write in the input-fields. After they had been given an explanation of the feature, the inexperienced users also found the feature attractive.

The experienced user didn't understand the reason for the variety of color given to mark new mail in the InBox.

Said about POP:

I don't understand Server time-out and Portsettings. (Two users).

The indications for each POP account is a good idea. The problem is that I can't remember which indications indicate the different accounts.

This thing [Hotmail] can do a lot more than I thought. (After an explanation of POP).

Company X recommends:

- that this attractive feature is given more explicit attention on the website.

- that this feature is explained and presented to the inexperienced user in a more understandable way like "Send all your other email to Hotmail". The word POP is a typical Internet term unknown to most users.

- that users are informed that they can typically find the correct information about the data needed for the POP account in the information given by their Internet provider.

- that users when adding a POP-account get more information about the individual input-fields.

16. Advertisements

Users were partly annoyed during the login procedure that they had to wait for a long list of company offers that they did not want (not a single one of the users that went through the registration procedure chose to subscribe to any of the offered email services. Some of the experienced users had originally signed up for these email- subscriptions, but after a while they found the mails were only taking up space in their InBox. This was the same fear that some of the new users expressed. They did not want their mailbox filled with non-personal mail. The users' overall focus is a smooth and effective mail service. Basically ads of all kinds are working against this target. On the other hand users respected that the ads were the reason the site could stay free. One user said he would sometimes surf some of the banner ads, just to support the service.

None of the other users ever clicked the bannerads (and not during the test either). Most users when asked said that they did not see the advertisements at all.

The reason why users did not see the banners may be related to the fact that the banners were not personalized and they were all in English and related to English/American webpages.

Some users thought that if the ads had been more relevant (by being in their own language or selling stuff closer to their interest) they might react differently.

Said about Advertisements:

I often follow banner-ads just to keep the system alive.

Ads don't bother me at all - but I would never care to click on one of them.

I don't use the ads, and I don't think they get into my mind at all.

Company X recommends:

- showing personalized ads for instance in the native language of the user. By having local companies in the banner ads, Hotmail increases the chances of users feeling that the ads are part of the service and eventually clicking them. From Hotmail's point of view, this extra effort might be worth a try as the value of each banner increases.

Answers to the general questions raised by Hotmail management (in the scenario paper).

How do users currently do email?

From the relatively little data we collected on general email behavior, it seems that there is a pattern in the target groups way of doing email. Mostly the users preferred the email address supplied by their service provider accessed via a normal mail client such as Microsoft Outlook.

Generally, how much effort do users have to expend to interact with and use Hotmail effectively?

Not a lot. If the users did not feel that the service was fast, efficient, error free and easy to use instantly, they were prepared to leave (without coming back). Read the registration chapter.

How satisfied are users with Hotmail?

Users are generally very satisfied with the idea of the service as well as with the way the service is delivered. Read the general findings.

If a first time user, do users think they will continue using Hotmail on their own?

Only if they had a good overall experience. In the actual testing the users was 50-50 when asked if they would use Hotmail in the future.

Would user recommend Hotmail to others?

Instead of asking the users this hypothetical question we directed our attention to how the test persons had heard about Hotmail. Practically everybody had at some point had the service recommended by friends. And all the experienced users (without exception) had recommended the service to their friends. Sometimes they had even taken active part in the registration procedure.

How reliable do users think Hotmail is? Would they want to use it for urgent or time sensitive information?

As mentioned, the experienced users did not trust Hotmail handling attachments. And the problem with speed (that at certain times during the day makes Hotmail practically inaccessible) kept users from generally relying on Hotmail as their only email.

Hotmail and Microsoft

As further discussed under Deviations, we have chosen to comprehend the Hotmail marketing management questions raised in part one of the original scenario paper as a more general concern about the users feelings towards the branding and corporate identity of Hotmail seen in the light of Microsoft's recent takeover.

We want to discuss the findings more refined at the briefing. Here it is only fair to state, that the powerful "Big Brother image" of Microsoft rubs of onto Hotmail in both a negative and a positive way. Giving the users one hand a sense of security and stability and on the other hand a general feeling of Microsoft becoming to big.

Said about Hotmail and Microsoft:

I learnt about MSN via Hotmail

Microsoft owns Hotmail? That is not very possitive. They got enough already.

They are so big [Hotmail] that I'm not afraid of abuse of my account. It's my opinion that you must be able to put the same kind of trust in your Internet-players as you do in your Telephone Company and your Post-office.

As a normal user it is easy to become a little afraid that they [Microsoft] will become to big. I'm keeping a eye on them.

Hotmail logo vs. MSN logo

No consistency

The main logo on Hotmail recently changed to the MSN logo. Users complained the lack of consistent use of the new logo on several pages on the website. In general the MSN-logo was found on the central pages and the old Hotmail logo was found on the more remote pages. The result being that the users was confused if Hotmail had a logo at all. This has to be seen as a problem for Hotmails own corporate identity. One experienced user mentioned that it was strange that a brand that big as Hotmail, did not have an actual logo.

Old logo tested better than MSN-logo

Users did not immediately respond to the change of Logo on different pages within the site, but after each test we asked the inexperienced users if they could remember and describe the Hotmail logo. And if they during the test had been confronted with the old Logo they only remembered the old one. When confronted with both logos all inexperienced users preferred Hotmails original when compared to the MSN logo.

Most of the experienced users had noticed the change of the logo, during the last months, and the experienced users as the inexperienced preferred Hotmails original when compared to the new the MSN logo.

Said about the logos:

That logo is different from the one on Hotmail. (About the MSN logo)

Logo? Do they have a logo?

Company X recommends:

- make a clear distinction between Microsoft and Hotmail. And keep the old Hotmail logo to show it.

Other issues

Related links - an inactive area of navigation

The area in the upper right corner with various Related Links was not used as an active way of navigation. Many users did not see the area at all, and given a specific task that could actually be solved via links the Related Links, they often preferred an alternative way – often through the global navigation.

Lack of status indication on emails

Many users complained that they had no way of knowing the history of a certain mail once they had it on screen. Drafts, Saved Sent Mails and New Mails all looked the same (in the compose window). Users wanted a visual indication of the actual status of a mail.

Email Greetings

This external hyperlink calls for some attention (even though it is not a part of the specified test scenario). The three users that selected this link were confused when they arrived to Email Greetings. They did not know whether they were in or outside Hotmail. When asked, they all thought that the link (gray, and with the same appearance as other related links) would keep them inside Hotmail, but the appearance of Email Greetings signaled otherwise.

When users wanted to get back to Hotmail from Email Greetings they were often confused and preferred to use the browsers back-button. Here users often received a "file not found"-message.

Said about Email Greetings

I have never used it and they are trying to make me use it like crazy. I just figured "and so what" and went on.

I expect that somebody else is making this, but actually I don't know. The link looks likes it is something inside Hotmail - gray and doll.

Company X recommends:

- that "Session held in background" is renamed "Hotmail sessions held in background". (On Email Greetings).

- that the present Email Greetings hyperlink from Hotmail are replaced with a more explicit hyperlink that indicates an external hyperlink and also works more redundant with the logo of Email Greetings .

Language

At no point during the test did we discuss the exact labeling of the hyperlinks on the website. As an exception to this rule we asked users directly when they expressed problems with a certain name or sentence. All users were good at reading English/American.

The names on a few hyperlinks caused minor problems to users: Compose, Browse and Draft. It is possible that a more active test leader role on this subject would have revealed more naming problems. But we also feel that our more passive approach helped the inexperienced users, that is to say that we gave the actual content behind the hyperlink a chance to teach the user the meaning of the hyperlink in question.

In general, users had more difficulties understanding explanatory texts like for instance Help texts and the introductory pages.

Said about Language:

I don't understand "Drafts".

If the text had been in my language, I might have read more intensively.

Company X recommends:

- that the introductory static pages and pages were the exact wording is important (like password retrieval information, legal-stuff attachment-help and so on) are translated into different languages.

PART TWO

The testing

The user test took place on the October 21, 22, 26, 27 at Company X's locations.

Nine representatives from the target group participated in the test. One of these was asked to participate in the pilot test that took place prior to the test itself. The purpose of the pilot test was to adjust the test method to fit the special issues concerning . For the actual test 2x4 people participated.

Group one were experienced Hotmail users. Group two had never used it. Both groups, however, did use email. By recruiting inexperienced users that did not have a Hotmail account, we were able to meet Hotmail's wish to test the registration features. At the same time, inexperienced users provided us with the opportunity to test Hotmail as an intuitive service that is easily accessible to the inexperienced user.

In all tests we used our own version of the ”think-aloud” method. During such a test, we work with one person at a time. During the test, the person is asked to explain his experience, expectations, doubts, etc. while using the service.

Each test took between 70 and 90 minutes and all tests were recorded on video. During each test, a test manager is present as well as a test rapporteur who is responsible for taking notes. Each person was given two bottles of red wine as a token of our appreciation of their participation.

It is important to underline the fact that these tests are qualitative. We do however refer to quantitative measures, ie. by referring to the number of users that have appreciate or dislike a given feature. These references, however, are based on both a subjective evaluation of each user's tone-of-voice combined with how many of the users expressed a certain attitude towards a feature. The "quantitative" references are not valid as statistical data.

Questionnaire (English version)

The questionnaire serves as a guide to the test leader and rapporteur during the test.

The questionnaire only offers guidelines and is not followed in detail in regards to chronology or its exact wording. Thus, not all questions are necessarily asked during the individual test sessions, and during all test sessions a number of questions arise, that were not part of the original questionnaire. The questionnaire contains a number of questions that are asked of both the experienced and the inexperienced user, but it also contains questions that are specific to the two different user types. For the experienced user, specific questions are indicated with an ”(E)” and for the inexperienced user, questions are marked with an ”(IE)”. Questions asked of both groups are marked with a "(C)".

(C) Introductory questions

1. What do you do?

1. work

2. leisure

2. For how long have you used the Internet?

3. How often do you use the Internet?

4. Do you use email a lot? What for? How?

3. Custom software or Internet client?

5. Do you know Hotmail?

6. From where do you know Hotmail? (E) How did you start using it?

(C) Introduction to the test session

We are not the developers of this service, but only involved testing it. Today, we are looking at Hotmail. We are not testing a prototype. The test is based on a ”think-aloud” technique. Please, try to mention everything that comes to mind. Criticize, praise, give us suggestions for improvements, etc.

(IE) Short introduction to Hotmail: "You have heard about Hotmail, a free email service on the Internet."

Getting an email account (Registration)

(C) Focus on how the users get to Hotmail

(entry of URL, bookmarks, other websites, etc.)

4. You are now familiar with the fact that you can get a free email account on the Internet. What do you do?

5. Sign up

(C) Using Hotmail

(E) Check your email.

(E) Login page (User is stopped and asked how they usually get to Hotmail)

(E) Have you ever used Hotmail via somebody else’s computer?

(E) Do you check email every time you use the Internet?

(E) Do you use the Internet every time you check mail?

(E) Do you know of any other ways to access Hotmail from the Internet? Have you used other ways?

6. Send an email to xx@xxx.xx

7. You forgot to attach an important file for xx@xx.xx. (the important file is placed on your desktop and named importantfile.dok). What do you do?

8. You need to send an email to Mr. X. You forgot his email address. What do you do?

9. Make an address book with two email addresses.

10. You are unhappy with the menu. What do you do? (Change it.)

11. You are in doubt as to whether you sent the email to Mr. X. Check if you did write the email and sent it (focus on folders. Do they use them? How?)

12. Please check your email again. Maybe Mr. X answered? (Return mail is obvious in the InBox). (Focus on whether the user notices that there are links in the copy (clickable) and whether they see the attachment)).

13. Try to sort your mail.

14. Try to make Hotmail fetch your mail (POP3).

15. No more Hotmail for today. (Focus on whether the users log out and on what they expect to find after they log out).

Finishing questions

16. What is your opinion regarding layout, graphics and colors?

17. How do you feel about the ”tone-of-voice”?

18. How do you like the navigation?

19. What do you think of the ads? (E)Have you ever clicked on them? Why/Why not?

20. What is your overall opinion of Hotmail?

21. Are you happy with Hotmail?

22. (E) Have you recommended Hotmail to friends?

23. Is it a good idea that you can check email on the Internet?

24. Do you find it easier to use Hotmail than regular mail programs?

25. Do you use Hotmail for all kinds of letters? Even if they are important, confidential or urgent?

26. How do you like Hotmail's logo?

27. Do you have a personal opinion about frames?

28. What do you think are Hotmail's strong points?

29. And the weaknesses?

30. Do you have any suggestions as to how Hotmail may improve their service?

Test persons

The main parameter for choosing the test persons was the experience/inexperience. For both groups we chose test persons with different personalities rather than just demographically dispersed.

Pilot test person

35 years old, Male, Engineer. Has used Hotmail a lot, but is now using Outlook98.

Uses the Internet every day, and is an expert user.

Experienced Hotmail Users

|Test person |Email experience |Internet experience |

|21 years old, female, receptionist, |Has used Hotmail for about 3 months, but |She uses the Internet everyday. She has |

| |uses another program in her daily use. |only a little experience. |

|30 years old, male, chemist Engineer. |Has used Hotmail a lot, when he was |Uses the Internet several times a week. |

| |travelling and studying in different |Uses the POP-mail facility. Has used the |

| |countries. |Internet for 4 years. Uses Internet for |

| | |several things. |

|24 years old, female, studying history of |Has used Hotmail for 6 months. |Used the Internet for 3 years, but for |

|art at the University. | |little else than email. |

|26 years old, male student of engineering |Has used Hotmail once he was in USA to |Uses the Internet several times a week. |

| |study. But does not use Hotmail anymore. | |

Inexperienced Hotmail users

|Testperson |Email experience |Internet experience |

|30 years old, male, now studying |He never used Hotmail, but Exchange. |Uses the Internet 3 or 4 hours each day. |

|engineering, but used to be a programmer, | |But is not an expert user. |

|who worked with database systems in DOS | | |

|25 years old, female, Studying |Does not have a Hotmail address, but uses | |

|Communication and Media at the university |email a lot. | |

|25 years old female, studying economics |Has heard about Hotmail, but never used |Mostly uses the internet for email. |

|and data. |it. | |

|45 years old, male, working as a manager |Has heard about Hotmail but has not been |Uses the Internet every day. |

|at a nightclub. |tempted. Uses email everyday. | |

Technical test configuration data

| | |

|Computer: |Zitech 166 MHz Pentium, 32 Mb Ram |

|Modem: |USRobotics 28.800 Faxmodem with v.34 and V.32 b/s |

|Browser: |Netscape 4.0 or Explorer 4.0 [European version] |

|Test time: |During 8.00 AM to 14.00 AM in GMT. |

|Test period: |20.-26. November 1998 |

| | |

Tested Hotmail version SCREENDUMP

[pic]

Background

Who is Company X

(deleted)

General purpose of testing

Our testing methods covers two aspects:

One is to ensure user-friendliness. That is, to ensure that the website is easy to use and that no unnecessary communicational "noise" disturbs the user. Testing user-friendliness thus focuses on issues such as easy access to information, the users' ability to remember where specific information is located and overview of the entire website.

Secondly, we test whether content and functionality is relevant to the user. General relevance is covered (the websites’ reason for being) as well as the relative relevance of each page or part. We examine whether the navigation is logical and whether there items in the menu items are consistent with the content that these menu items cover. Copy, and tone-of-voice is also tested. In relation to this, we take into consideration the fact that this website is in English, yet for all the test persons, it is not their native language.

Scenario Paper (original)

Dated October 8. 1998

Scenario:

You've been hired to evaluate , the world's free e-mail service you can access from any internet connected personal computer. The marketing management at Hotmail has sent you the following information, which identifies their goals and objectives for this evaluation.

Part One:

Hotmail was recently acquired by Microsoft and can now be accessed from both the following URL's:

home. and .

Management is very interested in knowing which avenue users are taking to reach their site, and whether or not users' perceptions of Hotmail (and MSN) are influenced by the avenue they choose. [Note: The original home page and the logo in the top left corner of the navigation bar within the site will most likely change branding in early October. You are required to take a screen shot of the state of the home page you evaluate for the CUE comparison so we can know if different teams evaluated different branding designs.]

If users are already familiar with Hotmail, how do they choose to access it? Is it easier for new users to locate Hotmail from one URL over the other? Is there any distinction in the kinds of users that tend to use the various avenues?

Additionally, Hotmail defaults you to the frames version of their site (if your browser is capable of interpreting frames). Management is very interested in knowing whether users prefer the non-frames version.

Hotmail's biggest competitors are: Yahoo Mail and Netscape WebMail.

Do NOT test the localized versions of HotMail, like Hotmail.dk

Part Two:

Management is also very interested in the general usability of their product and has identified some specific areas of concern. Below is a list of features that Marketing and Engineering have identified as benefiting from user feedback: (Listed in order of importance to Product Marketing)

1.Registration

2.Login

3.Logout (what do users expect after hitting the logout button?)

4.Viewing Hotmail with or without frames - do users have a preference?

5.Customization (under the 'Options' section, particularly "Preferences" for viewing style)

6.Performance issues (speed, etc.)

pose

8.Password Retrieval

9.Sending and receiving attachments

10.Reminders

11.Hotmail member directory and email lookup

12.Sorting messages in your InBox (by Subject, Date, etc.)

13.Reading a message (related issues: Screen resolution and size; URL recognition when a message has a url in it))

14.Address book

15.Folders

16.POP (under 'Options')

17.Advertisements (general attitude, do users click on them? Why,why not?)

18. When do users try to access Help (do not test help system, just identify when users feel they need help)

General questions which management is interested in addressing are:

1. How do users currently do email?

2. Generally, how much effort do users have to expend to interact with and use Hotmail effectively?

3. How satisfied are users with Hotmail?

4. If a first time user, do users think they will continue using Hotmail on their own?

5. Would user recommend Hotmail to others?

6. How reliable do users think Hotmail is? Would they want to use it for urgent or time sensitive information?

Hotmail management wants to focus only on the mail features and functions, and not the other services now offered from the site. They are *not* interested in the following areas:

Classifieds

Web Courier

News and Links

or other general portal features.

User Profile:

Hotmail is a world-wide free email service targeted towards anyone who would

like the convenience of managing their email on-line through a web browser.

Hotmail can be accessed from any computer or kiosk that provides internet

access, making it easy to access your mail from anywhere. Hotmail also allows

you to manage all your own personal email without it being recorded on your

employer's servers.

Questions?

If you have questions about product goals, user profiles, or product features, please send them to erika.kindlund@eng.. Erika will forward them on to the Hotmail contacts, as well as distribute responses from them. Erika's role as proxy is to limit the number of duplicate questions being sent to Hotmail management.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download