Qualitative Evaluation - Administration for Children and Families

Qualitative Evaluation

Evaluation Brief Design Options for Home Visiting Evaluation

October 2020

Introduction

Qualitative research, which explores how or why something occurs, can contribute new knowledge to the understanding of home visiting. While qualitative research designs are sometimes viewed as less rigorous addons to quantitative research designs, studies utilizing qualitative research methods--whether part of a mixedmethods approach or as a standalone design--can be rigorously designed to provide reliable and trustworthy information.

The purpose of this resource is to provide awardees and their evaluators recommendations on how to implement strong qualitative evaluations and minimize bias throughout each stage of the evaluation.

Understanding Qualitative Research

Qualitative research focuses on an individual's beliefs, experiences, attitudes, behaviors, and interactions, and is often used to answer research questions focusing on how or why something occurs. Qualitative research seeks an indepth understanding of social phenomena within the everyday environment of the participant, such as in the home or place of work.1

Qualitative methods generate non-numerical data--such as written responses, interview transcripts, or videos--that are analyzed and interpreted to highlight themes, patterns, and concepts. They may be used independently or in conjunction with a quantitative evaluation to form a mixed-methods study.

Advantages of Qualitative Approaches

Qualitative data can play an important role in home visiting research. Many aspects of home visiting participation are difficult to quantify, such as the benefits of a trusting bond between a home visitor and client. But these benefits can be richly described by participants, home visitors, and supervisors, providing unique insight.

The Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program funds the development and implementation of evaluations by awardees. MIECHV is administered by the Health Resources and Services Administration in collaboration with the Administration for Children and Families.

OPRE Report #2020-136

Rigorous qualitative methods have the potential to be--

Holistic. They may support the full understanding of an issue by incorporating multiple perspectives and considering how the context or environment may influence outcomes. Flexible. They may allow researchers to adapt in real time to the information being gathered. Comprehensive. They may use unique data sources, such as program documents, observations, or video and audio recordings.

Challenges of Qualitative Approaches However, qualitative approaches have drawbacks:

Resource intensity. They may require considerable resources, including staff and participant time, which may limit sample size. Limited generalizability. Findings may not be applied to a broader population. However, methods and results may be transferable and applied to similar settings. Lack of anonymity in data collection. Although confidentiality may be maintained in reporting results, participants may be more comfortable completing anonymous surveys than participating in one-on-one interviews or focus groups, especially when asked about sensitive topics.

Considerations for Choosing a Qualitative Approach

Qualitative research can examine a phenomenon from the perspective of the people experiencing it. This approach is especially helpful for topics not previously studied, where there are major gaps in the research, or when there is a need for a new perspective. Qualitative methods are rigorous when they are credible, transferable, dependable, and confirmable.2

Consider using qualitative methods when--

The goal of the research is to understand participants' values, beliefs, behaviors, feelings, expectations, motivations, and expressions. The evaluators have developed trusting relationships with study participants, such as local implementing agency (LIA) staff. Outcomes cannot be easily quantified. There is a need to contextualize or validate quantitative data.

Avoid using qualitative methods when--

Quantitative methods, such as surveys, may be more time efficient and cost effective to answer the research questions. The analyses needed to answer the research questions require a large participant sample.

Criteria for Demonstrating Rigor in Qualitative Research

Credibility: Presenting an accurate description or interpretation of a human experience

Transferability: Transferring findings or methods from one group (or setting) to another

Dependability: Following the decision chain so other researchers can determine the credibility of the findings

Confirmability: Requiring researchers to be reflexive or self-critical about their biases

2

The responses to the research questions deal with numbers or variables that are measurable and clearly defined (e.g., duration and frequency of home visits).

Designing Qualitative Research

Whether designing a standalone qualitative evaluation or a mixed-methods evaluation with a qualitative component, thoughtfully select the methodological approach and research questions.

Methodological Approach

Evaluators should select an approach that aligns with the study's research questions and target population. Methodological approaches to qualitative research include--3,4

Grounded theory. This involves the construction of new theory through systematic data collection and analysis. Theory is typically generated from the field through observations. This approach can be used to answer process questions about experiences over time or changes that consist of discrete stages. For example, "What theory of change explains the impact of your program on parent-child interaction?" Case study. This involves an in-depth exploration of an individual, group, organization, event, or phenomenon within a setting or context. For example, "How do three local implementing agencies in the northern region of the state recruit, engage, and retain families? What similarities and differences exist between these LIAs?" Ethnography. This observational approach examines the culture of a group of people for a long period of time. The researcher collects rich descriptive data while immersed in the daily life of a group, social, or cultural setting. The researcher intensely examines, describes, and interprets practices, behaviors, values, and relationships between group members. For example, "How do families experience home visiting in your community?" Phenomenology. This examines the shared everyday life experiences of individuals within a group, describing the nature of a particular phenomenon and how people experience it. For example, "How do home visitors in your program experience working with families dealing with trauma?"

Research Questions

Write research questions with enough detail to convey their purpose and goal without additional explanation. Research questions should be--

Narrowly defined. The questions add focus to the research. They should focus on a single phenomenon or concept. Exploratory. The questions often begin with what or how and avoid directional words such as affect or impact. Answerable. The investigator can observe or assess the answers. Specific. The questions should specify the target population, align with the research design, and guide data collection and analysis.

See the Resources section for additional guidance on developing rigorous evaluation questions.

3

Planning Qualitative Data Collection

Qualitative data can be collected using multiple methods and tools designed to elicit the participant's perspective on a program, experience, or scenario. The evaluation team should consider issues related to data collection methods, and sampling and recruitment strategies to find participants with relevant and diverse perspectives.

Data Collection Methods

Qualitative evaluations may use a combination of data collection methods to examine key issues in depth, triangulate data from multiple sources, or complement quantitative data.5 Exhibit 1 describes advantages and disadvantages of common data collection methods. For example, interviews might elicit meaningful responses, but they are time consuming. Focus groups allow for more rapid and less costly data collection, but it may be difficult to collect sensitive information in group settings.

Exhibit 1. Common Qualitative Data Collection Methods

Data collection method Interviews Focus groups

Observations

Document review

Description

Advantages

Disadvantages

Structured or unstructured (conversational) one-onone or group discussions with key stakeholders Structured or unstructured small-group discussions

Video recordings, participant observations, or long-term ethnographic studies

Documents or data from agencies, programs, or service providers

Explore experiences, views, or opinions; can elicit meaningful responses

Generate additional questions or ideas; faster and less expensive than interviews

View interactions in real time and in context; observer may interact with participants if desired

Use data that were already collected and may be available to download; can use checklists or other forms to summarize data

Conducting interviews and analyzing data are time consuming and resource intensive May be difficult to gather sensitive information; data analysis may be more complex than for interviews People may alter behavior because of observation; may provide a limited view; time consuming; possible technology challenges Limited to existing data found in documents

Piloting Tools With the Target Population

Qualitative research often involves developing tools for specific situations. Pilot testing addresses the following:

Are questions clearly worded and readily interpreted as intended? How long does it take to conduct an interview or focus group? What procedural or methodological challenges may arise?

4

Conduct pilot tests with a small sample of respondents similar to the target population. The sample composition is particularly important when addressing cultural context, ability to answer questions, or sensitive topics that may be perceived as offensive. Use cognitive testing methods to assess question wording, flow, and timing. Determine whether questions capture the intended concept, use the appropriate words, and appear in a logical order. Use the pilot test process to inform the development of tool administration protocols that minimize the respondent's burden (e.g., drop or rework questions that are confusing, offensive, too long, or unnecessary).

Sampling and Recruitment Strategies

A qualitative evaluation sampling plan should include the sampling technique, inclusion and exclusion criteria, and number of participants. Use the plan to identify the following:

Who is included in the sample? What is the composition of the sample (e.g., application of inclusion and exclusion criteria)? Where and how will the sample be recruited? How and when will data be gathered? Why are the data important to the evaluation?

Qualitative sampling strategies rely on establishing homogeneity in sample groups to generate deep rather than broad findings,6 though it is important to include participants who represent diverse perspectives. To develop a sampling strategy--

Determine the sample size in the context of the evaluation design and data collection methods. The number of participants or cases should allow for multiple perspectives to be represented and for oversampling to ensure an adequate range of information is collected. Select a sampling technique (e.g., purposive, quota, snowball). See exhibit 2 for a description of common qualitative sampling techniques.

o Most qualitative research uses a form of purposive sampling and matches the specific strategy to the evaluation context and requirements.7

o Bias can skew the evaluation. Consider the potential sources of bias, including the researcher's own biases, and ways to mitigate the risk of selection bias in the sample. Biases should be discussed in the final report or presentation of findings.

Exhibit 2. Common Qualitative Evaluation Sampling Techniques

Sampling technique Purposive

Quota

Description

Participants are identified and recruited based on criteria relevant to the study goals, research questions, and researcher knowledge. Sample size depends on available resources and time, study objectives, or number of participants needed to yield theoretical saturation (i.e., when new data no longer bring additional insights). Participants are identified and recruited according to characteristics: general (e.g., demographics), related to the study (e.g., use of services), or related to insights into the research topic (e.g., prior home

Potential selection bias Due to recruitment that reinforces the researcher's beliefs, attitudes, or knowledge

Due to ease of recruitment access or desire to include/emphasize specific groups

5

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download