UNIVERSITY OF MARYLAND, COLLEGE PARK



Title:

Longitudinal Case Study Evaluation of Graphical User Interfaces

Principal Investigators

XX

Abstract

Graphical user interfaces have radically changed the way people access data on the personal computers, cell phones or the internet. Information visualization now allows users to find patterns, identify outliers and make discoveries. In our research we are developing novel information visualization techniques and interfaces.

The reports of usability studies and controlled experiments are helpful to understand the potential and limitations of our tools, but we need to consider other evaluation approaches that take into account the long exploratory nature of users’ tasks, the value of potential discoveries or the benefits of overall awareness of changes in the data. We also need reports of successful adoption and demonstrated utility.

We seek to recruit domain expert collaborators (our “study participants”) who will be involved in long-term studies to study how they utilize our interfaces. The study will mostly be qualitative. We seek to demonstrate the utility and effectiveness of our interface by collecting evidence of success due to usage from our collaborators. These collaborators will first undergo training sessions on how to use the advanced interface. The interface will then be given to our collaborators for use in their professional environments. Periodic meetings to discuss the usage, utility, satisfaction (or lack thereof) of the interfaces will be conducted. The meetings can be face-to-face, teleconferencing, or web meetings. In addition, a history of their actions using the interface will be automatically logged on their own computers. The logs will be accessed by the investigators only with the permission of our collaborators. The collaborators will fill out an exit questionnaire at the end of study to assess their subjective experience with the interface. Each participant in those studies will work with us for about 4-8 weeks.

Subject Selection

During this project, we will recruit adult volunteer professionals (e.g. analysts, journalists, professors, medical researchers, etc.). They will be expert in their domain, and have some use of our interfaces. They will recruited vis personal emails, among people who contact us in person or by email to ask information about the tools we are developing. No specific characteristics (e.g. gender, age, ethnicity, etc.) are required of the participants. It is important to note that we see our subjects are collaborators who are interested in using advanced interfaces for their professional work. We seek to observe how they use our interfaces and how our advanced interface design help them in their daily professional work. There will be no monetary compensation. Instead, we will offer co-authorship to our collaborators in the papers reporting our results in scientific publications. There will be no obligation for our collaborators to publish with identifiable information, or to publish at all. At any time, our collaborators can choose to terminate the study and collaboration with the investigators.

Because of the long-term, time-consuming, and detailed nature of this type of qualitative study, we intend to recruit only 3-6 participants per year.

Procedures

Our collaborators will first sign a consent form (sample attached), and receive a training of the interface. Collaborators will be informed of the type of data that is being collected prior to giving consent.

Once the investigators feel that collaborators are trained and are competent in using the interface, a copy of our software will be given to the collaborator for him/she to install in their professional environment. Depending on the work environment, the investigators may ask to observe collaborators in their work environments. The investigators and the collaborators will schedule regular meetings (weekly or bi-weekly depending on the participant task and availability) to discuss problems, possible discoveries, or frustrations in using the interface in collaborators’ working environment. The regular meetings may be recorded in audio. To facilitate these meetings, collaborators will be ask to write down their usage experiences in a journal. The goal is to obtain anecdotes of success and of failure when the advanced interface is put to use in a real environment.

While in use, the interface will automatically log the usage (e.g. submitted queries, browsing strategies used) of a collaborator on the collaborator’s computer. Note that no personal or identifiable information will be collected, only the actions performed within the interface are logged. The results of the logging are stored in the collaborators computer, and the investigators are only to access the log with the collaborator’s permission. The location of the log is made known to the collaborators, so they can remove or edit the log anytime they wish.

During any part of the study, the collaborators are encouraged to ask questions about the interface or the study. The collaborators may choose to stop and terminate the study.

Since the goal of the study is to collect narratives of the usage of the interface in a real-world setting, and progress in understanding data can take days or weeks, the entire study is planned for 4 to 8 weeks. Each session in the regular meeting are expected to take about 1 hour each. Each collaborator can choose to spend as much time as they wish with the interface in their work.

There will be no monetary compensation to our collaborators. Instead, summary narratives collected from the study may be published in scientific publications and the collaborators are offered to be co-authors. Collaborators are free to choose whether to be included as co-authors, or to have their portion of the study published. No information about the study or the collaborator will be published without the collaborator’s consent. There is also no obligation to publish or to be a co-author. At any time prior to any publication of a scientific paper, a collaborator may choose to withdraw from being a co-author, or to refuse the publication of certain portions of the paper.

The Informed Consent Form is attached.

Risks and Benefits

There are no risks involved in using the interface. The study ask collaborators to use their own personal computer running our novel interface. Our collaborator may rest at any time, or quit at any time. While using the interface in their work environment, the collaborators are not forced to use the interface for a minimum amount of time. If the collaborator decides not to use the tool, we will interview them to ask them why (i.e. what the tool not useful? Not reliable? Too hard to use? Other reasons?) Unless the participant expressly choose to be mentioned by name in the papers we write their name will not appear anywhere in our writings. Participants will be provided copies of the papers before they are submitted for publication if the paper mentioned the study they participated in.

Confidentiality

All results will be given to our collaborators prior to making them available or published. The collaborators can refuse the publication of certain personal or identifiable information, their narratives, their logs, or any combination of them. Since the collaborators are not receiving monetary compensation, we do not collect social security numbers or any other private information.

The logged data from collaborators’ actions using the interface will be stored only in each collaborator’s computer. The location of the log is shown to the collaborators, and they are made aware that they may choose to delete the log at any time. Furthermore, the logs are only to be accessed by the investigators with the corresponding collaborator’s permission.

E-mail addresses obtained from volunteering collaborators will be kept during the study in the investigators regular email system.

Interviews will be conducted in the setting chosen by the collaborators (i.e. their own office, or the PIs’ office) and they will choose the degree of privacy they need to report on their work with the tool.

Information and Consent Forms

Subjects will be informed about the purpose of the research. The consent form will be presented and signed before the study starts. All participants will receive a copy of the consent form for their records. The procedures will be explained before the study, and subjects can ask any questions they wish. During the length of the study, our collaborators are encouraged to ask questions and to communicate with us. At the end of the study an exit questionnaire will be provided, including an additional consent form section qith more specific questions about how we can report on their work (see Exit questionnaire). A copy of the exit questionnaire and consent form will be provided to the participants.

Conflict of Interest

No potential conflict of interest.

HIPAA Compliance

N/A

Research Outside of the United States

N/A

Research Involving Prisoners

N/A.

Longitudinal Case Study Evaluation of Graphical User Interfaces

Informed Consent Form

DESCRIPTION: You are invited to participate in a research study in data visualization graphical user interfaces. Name of the particular interface(s) to be used:

PROCEDURE: After you fill a questionnaire and participate in a training session you will use the interface in your normal tasks with your own data.  We will meet with you at regular intervals chosen by you to document your use of the tool. Your computer interactions may be logged and saved on your own computer for later data analysis. 

RISKS AND BENEFITS: There are no known risks associated with participation in this study. The expected benefits are better human computer interaction and  information visualization techniques. This research is not designed to benefit you personally.  However, you may be acknowledged in our papers and presentations, and your use of the tool describe.  You have the right to refuse being identified by name or to have the result of your analysis described in our papers.

TIME INVOLVEMENT: This study will typically take 4 – 8 weeks. The initial training session will take 2 to 4 hours. Each subsequent meeting should take about an hour. Other than these, you are asked to use the interface naturally in your own work environment during work time. The actual committed time will depend on how much you use the interface.

PAYMENTS: You will receive no monetary payments for your participation.

SUBJECT'S RIGHTS: If you have read this form and have decided to participate in this study, please understand that your participation is voluntary and you have the right to withdraw your consent or discontinue participation at any time without penalty. You have the right to refuse to answer particular questions, and you may ask any question you wish. Your individual privacy will be maintained in all published and written data resulting from the study. If you have questions about your rights as a study participant, are dissatisfied at any time with any aspect of this study, or wish to report a research-related injury, please contact - anonymously, if you wish – the Institutional Review Board Office, XXXX YOUR IRB INFO HERE XXX

I state that I am over 18 years of age and wish to participate in a program of research being conducted by XXX (xxx@email) at the University of XXX.

I understand that if the results of the study are to be published, I have the right to choose to be identified by name or to have the results of my work described:

(please initial): __ Yes __ No

I give consent to have my name and institution mentioned (as partnering in an ongoing case study) in HCIL presentations about the tool being evaluated. I am aware that can withdraw this authorization at any time by sending a written request to the PIs.

(please initial): __ Yes __ No

I give consent to be recorded by audio or video during the meeting portions of the study:

(please initial): __ Yes __ No

I give consent for my actions to be logged when using the interface for research purposes (I am told where the log is, and how to delete it if I wish to):

(please initial): __ Yes __ No

I have reviewed the entry and exit questionnaires:

(please initial): __ Yes __ No

SIGNATURE _____________________________ DATE ____________

Name (PRINT) ____________________________ Institution (Print)__________________________

Approval Date: Expiration Date:

PERSONAL EMAIL TEMPLATE (to be sent to potential participants)

Dear XX

Thank you for your interest in our research. To evaluate our work we are conducting multiple longitudinal case studies with participants who use our tools to conduct their own work (i.e. analyzing their own data, at their own pace).

During our first meeting we will give you an entry questionnaire to record your plans and expectation. We will also setup a schedule of visits at regular intervals chosen by you, during which we will collect information about your use of the tool and the results of your ongoing analysis. We will let you install our software on your own computer and provide training and assistance during the duration of the study. You will be able to choose to have your name, institution and results mentioned in our papers and presentations (or not).

Other current participants include XXX who is working on XXX and YYYY

For more detailed information about the tool please visit our website at cs.umd.edu/hcil/nameoftool

Let us know if you are interested

Catherine and Ben

ENTRY QUESTIONNAIRE

Name: ____________

Institution: ____________

Date: ____________

Tool(s) to be tested: ___________________________

Summarize the question you have or the problem you are trying to solve:

What data you will analyze:

What prior analysis you or others have already conducted with that data:

Give examples of the type of discovery, finding, insight you HOPE to make during the data analysis.

What would you consider to be a successful outcome of this analysis in your own profession work (e.g. a new drug discovered, money saved, a scientific paper submitted, a newspaper article published, a change in work practices, new hypotheses, improve confidence in data quality, better awareness, etc.)

- Moderate success:

- Significant success:

Schedule of interviews and visit

What is the estimated start and end date of the analysis you plan to conduct?

What would be an acceptable schedule of visits for HCIL researchers to come for short interviews (biweekly personal or phone interviews are preferred)

EXIT QUESTIONNAIRE

Name: ____________

Institution: ____________

Date: ____________

Start and end date of the case study: ______________________________

Tools being used: to analyze the data

HCIL tools ______________________________________

Other tools ______________________________________

Summarize in a paragraph how you used the tools HCIL provided (describe how it was used in conjunction with other tools if appropriate, and give indications of the amount of effort spent)

Summarize the discovery made or the finding/insights gained during the data analysis (provide references if necessary)

Could those discoveries/findings have taken place without the use of the tools provided?

[ ] Yes

[ ] Yes probably, but it would have been difficult

[ ] Yes possibly, but it would have been extremely difficult

[ ] Most likely No

[ ] Definitively No

Comments:

Please rate the utility of the tool in your data analysis

For this particular case study the tool was:

Not useful at all: 1 2 3 4 5 6 7 Extremely useful

In general the tool is likely to be:

Not useful at all: 1 2 3 4 5 6 7 Extremely useful

Would you be likely to use such a tool in the future if it was available?

[ ] Yes [ ] May be [ ] No

If any, what features should be added or modified for you to use it on a regular basis?

Can you give examples of other potential uses for analysis in your work?

If a discovery was made or significant insight was gained

What professional output is likely to be produced (e.g. none, a scientific paper submission, a report produced, a presentation to colleagues, a white paper, a new direction of work, etc.)

How does this compare to your original expectations before starting with the tool.

Well below my expectations 1 2 3 4 5 6 7 Well above my expectations

Longitudinal Case Study Evaluation of Graphical User Interfaces

Research being conducted by XXX (XXX email) and tel XXX

CASE STUDY EXIT Consent form

As researchers we hope to be able to report on your case study in our scientific papers and public presentations. Please specify how you want us to report on your case study:

I consent to have my case study described in generic terms that do not identify my name, institution or my discoveries and findings.

(please initial) ___ Yes ___ No

I consent to have my case study described in scientific papers or presentations, with mention of my name and institution in the credits or in the body of the paper. HCIL will use the name and institution provided below.

(please initial) ___ Yes ___ No

I consent to have a general layman description of my discoveries and findings mentioned in scientific papers or presentations

(please initial) ___ Yes ___ No

If you answered YES to any of the above questions, please answer the following question:

I request the right to review the materials to be published or presented . I will provide consent by email within a week of receiving the materials.

(please initial) ___ Yes I want to review the materials ___ No, this is not needed.

A copy of the exit questionnaire and consent form will be provided to me.

SIGNATURE _____________________________ DATE ____________

Name (PRINT) ____________________________ Institution (Print)__________________________

Approval Date: Expiration Date:

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download