Running head: MIXED METHODS ARTICLE APPRAISAL



Running head: MIXED METHODS ARTICLE APPRAISAL

Mixed Methods Article Appraisal

Janice Innes

SOWK 659 (S01) – Fall 2011

Faculty of Social Work

University of Calgary

Statement of the Aims of the Research

The researchers state that the goals of the research study, in relation to developing the Child and Youth Resilience Measure – 28 (CYRM-28), are threefold:

First, we sought to create a tool for the cross-cultural study of resilience that could account for the psychosocial resources available to youth globally, making cross- cultural comparison of developmental outcomes associated with resilience possible. Second, we were seeking a way to discern which internal and external assets most influence successful developmental outcomes across all the cultural groups included in the study. And third, we wanted to identify the elements of a mixed methodology that was effective in the development of culturally sensitive psychological measures that avoided the exporting of concepts from Minority to Majority World contexts. (Ungar & Liebenberg, 2011, p.127)

Thus, the researchers sought to create a measure based upon these three primary goals. Additionally, the study of child and youth resilience across cultures is important because resilience helps to determine positive development (Ungar & Liebenberg, 2011, p.127); however, child and youth resilience differs amongst cultures, making a comparison of resilience amongst cultures difficult: “Different internal and external assets (e.g., a sense of humour, social support, and a safe community) contribute differently to positive development depending on the stressors in a child’s life (Luthar, Cicchetti, & Becker, 2000)” (Ungar & Liebenberg, 2011, p.127). Therefore, positive development of children and youth is shaped by their ability to access internal and external assets when confronted with life stressors. Furthermore, the research study is relevant because it seeks to create a tool for the cross-cultural study of resilience that can account for the psychosocial resources available to youth globally; such a tool makes the comparison of resilience across cultures possible, which in turn makes the research study relevant. As it has been shown, the researchers clearly state the aims of the research study.

Methodology

The type of methodology (i.e. mixed methods) may be considered appropriate, as it helps to ensure construct validity across cultures:

Mixed methods, however, can help address concerns related to the internal validity and generalizability of the resilience construct. According to Onwuegbuzie, Bustamante, and Nelson (2010), mixed research is particularly useful for measurement development when seeking to ensure construct validity across cultures. (Ungar & Liebenberg, 2011, p.129)

Additionally, the researchers state that little methodological innovation can be found in relation to the study of resilience: “Though an increasing number of researchers call for greater cultural relativism in studies of resilience (American Psychological Association, Task Force on Resilience and Strength in Black Children and Adolescents, 2008; Robinson, 2007), there remains little evidence of methodological innovation” (Ungar & Liebenberg, 2011, p.129). Thus, it appears as if the researchers used mixed methods in an attempt to demonstrate methodological innovation within the study of resilience. Finally, the methodology chosen by the researchers may be considered appropriate, as it helps to ensure construct validity across cultures and is representative of innovation within the study of the resilience of children and youth.

Research Design

The researchers justify the chosen research design by stating the following, “Achieving this goal meant a more reciprocal research design congruent with mixed methods as used within a transformative paradigm” (Ungar & Liebenberg, 2011, p.129). In this quotation, the researchers state that the chosen research design helped them to achieve their goal of building a more culturally sensitive measure with face and item validity; thus, the chosen research design may be considered appropriate, as the researchers created/chose a research design in an effort to achieve their goals. Therefore, the researchers did not attempt to make their study of resilience fit a research design, but rather they created a research design to fit their study of resilience. Furthermore, the researchers discuss how the chosen research design differs from the literature in terms of ambiguity:

The inclusion of qualitative methods and a design that encouraged discussions of variability in the sample resulted in tolerance for more ambiguity than is typical in the literature on measurement development (e.g., the chronological age of the youth varied, though they were matched by the developmental tasks they faced). (Ungar & Liebenberg, 2011, p.142)

Although the research design, in part, may be incongruent with the literature, the researchers justify this incongruence, thereby justifying their chosen research design.

Theoretical Framework

The researchers explicitly connect the research study to a theory base, and this theory base is the theory of resilience. Initially, the researchers describe resilience in terms of past research:

Most commonly, Minority World researchers describe resilience as a quality of individuals that reflects their capacity to engage in processes that make it likely they will overcome adversity and achieve normal or exceptional levels of psychosocial development (e.g., they will go to school, maintain a prosocial peer group, and avoid

delinquency; Garmezy, 1983; Masten, 2001; Murphy & Moriarty, 1976; Rutter, 1987). (Ungar & Liebenberg, 2011, p.142)

This definition of resilience is limited, as the researchers state that current research on resilience does not attend to cultural and contextual differences, “…the study of individual resilience and its resulting shift in focus from psychopathology to the protective processes that are associated with positive development has occurred with relatively little attention to cultural and contextual differences (Ungar, 2011)” (Ungar & Liebenberg, 2011, p.127). Thus, the researchers seek to build upon past research by broadening the theory of resilience by studying cultural and contextual differences of resilience. The researchers create the CYRM-28 in an effort examine these differences. Therefore, the researchers explicitly connect the research study to a theory base and attempt to expand this theory base through their study of cultural and contextual differences or resilience children and youth.

Literature Review

The researchers devote a section of the article to literature review, entitled “Previous research on resilience measures.” This literature review may be considered broad enough, as the researchers provide the reader with a comprehensive overview of past research pertaining to the resilience of children and youth. This literature review includes the Strengths and Difficulties Questionnaire, the Resilience and Youth Development Module, and education in relation to resilience (Ungar & Liebenberg, 2011, p.128). Although the literature review may be considered broad enough, as it addresses the main components of resilience studied by the researchers, it would have been beneficial for the researchers to discuss research that supports the creation of the CYRM-28, as opposed to solely discussing the limitations of past research which led the researchers to create the CYRM-28. Additionally, the researchers indicate that there is a gap in the literature, in regards to the effect of cultural immersion within the dominant culture and heterogeneity among ethnoracial minorities:

When cultural variation has been accounted for, most often through the study of African American or Latina/Latino youth (i.e., Parke et al., 2004), the effect of cultural immersion within the dominant culture, and heterogeneity among ethnoracial minorities themselves, has been largely overlooked. (Ungar & Liebenberg, 2011, p.127)

The researchers attempt to address this gap by developing the CTRM-28. As it has been shown, the literature review may be considered broad enough, as the literature review reveals a crucial knowledge gap in relation to the study of resilience of children and youth.

Sampling

It is difficult to determine if the sampling strategy was grounded in the literature, as this is not discussed by the researchers in the article. However, the researchers briefly outline how the participants were selected:

Within each site, 60 or more youth participated in the pilot administration of the CYRM. Final sample size at each site was determined by the local research team’s capacity to conduct the research. Participants were purposively selected by local research teams, including the LAC. (Ungar & Liebenberg, 2011, p.133)

This description of sampling fails to fully and clearly explain how the participants were selected. The researchers state the 60 participants participated in the pilot administration of the CYRM, but they fail to state how these 60 participants were selected. However, the researchers do state why the selected participants were the most appropriate, “The participants all faced at least three culturally significant risk factors based on informal assessment by members of the LACs (as a group, risks were identified and then youth identified in the community who met the selection criteria)” (Ungar & Liebenberg, 2011, p.133). Additionally, the researchers to not discuss why some people chose not to take part in the research study.

Data Collection

The researchers justify the settings for data collection by describing the reasons why participating communities were selected:

Participating communities were purposefully selected based on (a) cultural differences, (b) differences in the nature of the risks facing individual youth (all participants were sampled from one population of youth-at-risk identified locally, such as youth living in poverty, exposed to violence, or racially marginalized), and (c) the ability of the principal investigator to locate an academic partner with the capacity to supervise the research. locally. (Ungar & Liebenberg, 2011, p.131).

Therefore, the researchers adequately justify the settings for data collection. Additionally, it is clear how data were collected:

The CYRM was administered in a manner appropriate to the sample (i.e., either individually or in groups) by the local site researchers. Each question on the measure was read out loud to ensure illiteracy was not a barrier to participation. Qualitative interviews with the subsample of youth were conducted individually, using a standard interview guide (see Appendix A). (Ungar & Liebenberg, 2011, p.135).

Resultantly, the data appear to have been collected systematically and consistently; however, the chosen methods of data collection are not justified in the literature, or rather literature is not presented by the researchers to justify the chosen methods of data collection. In contrast, the researchers do make an effort to explain how the methods of data collection were used. For instance, the researchers discuss what was asked of the participants, “Youth were asked about the risks they face and the resources they had that contribute to good developmental outcomes relevant to them and their communities rather than assuming one set of homogeneous outcomes as is typical of Minority World research on resilience”(Ungar & Liebenberg, 2011, p.135). Additionally, methods of data collection were not modified during the study, which contribute to the overall clarity of the data. The form of data presented is clear: it is presented to the reader in an accessible manner in tables three and four in the article.

Reflexivity

The researchers state that there is a need for the developers of psychological tools to acknowledge their social locations and the power they hold: “Our experience demonstrates the need for developers of psychological tools to acknowledge their social locations and the power they hold. It is helpful when Minority World researchers show awareness of their role sustaining dominant world views (Chilisa, 2005; Smith, 1999)” (Ungar & Liebenberg, 2011, p.143); however, the researchers do not discuss how they attempted to acknowledge their social locations and the power they hold. Additionally, the researchers indicate that the selection of youth may have been biased, “Doing so may have biased selection of youth toward those who show patterns of resilience typical of young people in the Minority World” (Ungar & Liebenberg, 2011, p.142). Therefore, it may be said that the researchers examine sampling bias; however, this appears to be the only bias examined, as personally biases, for instance, are not examined in the article. Thus, the researchers state that it is important to critically examine roles, potential biases, and influences, but they fail to fully examine these within the article. Additionally, the researchers do not indicate that any changes were made during the study.

Ethical Issues

The researchers note that ethics approvals were obtained from the host institution of the RRC, academic institutions, government departments, and LACs (Ungar & Liebenberg, 2011, p.131); however, the researchers do not provide the names of the academic institutions and the government departments that ethics approvals were obtained from. Thus, there is evidence that ethics approvals were obtained, but the researchers’ description of ethics approvals lacks clarity. Furthermore, the ethics section within the article provides the reader with a minimal understanding of the researchers’ ethical considerations within the research study. This may be said, as there is no mention of informed consent or confidentiality within the article; however, the researchers state that, “Given the complexities of working safely with high-risk youth populations, approval was granted by the RRC’s institutional Research Ethics Board to not require parental consent if seeking this consent prevented youth from participating in the study” (Ungar & Liebenberg, 2011, p.131). This appears unethical, as the researchers appear to have greater concern for their research than they do for the high-risk youth population being studied. Additionally, the researchers do not discuss, within the article, evidence that any post-study effects on the participants were considered, nor do they provide evidence that ethical standards were maintained.

Data Analysis

The researchers provide an in-depth description of the data analysis process. Quantitative data were analyzed with the use of factor analyses:

Although the a priori assumption of the four clusters of items (what we termed the ecological model based on Bronfenbrenner’s [1979] work that described micro-, meso-, macro-, and exo-systemic factors, which match our individual, relational, community, and cultural clusters) served as a guide in the generation of items for inclusion in the CYRM, the team believed that the interpretation of items would vary the groupings across sites. (Ungar & Liebenberg, 2011, p.135).

Additionally, the researchers describe the qualitative data analysis process, “Analysis of the qualitative data were guided by Glaser and Strauss’s (1967; Strauss & Corbin, 1990) grounded theory approach, together with constructionist advances (Charmaz, 2006; Clarke, 2005)” (Ungar & Liebenberg, 2011, p.135). Furthermore, the data analysis process is grounded in the literature, as the researchers state that the data analysis process was guided by the work of: Bronfenbrenner’s, 1979; Glaser and Strauss’s, 1967; Strauss & Corbin, 1990; DeVellis, 2003; Noar, 2003; Charmaz, 2006; Clarke, 2005 (Ungar & Liebenberg, 2011, p.135).

The researchers describe how the data were transformed through data analysis. They indicate that several transformations occurred through data analysis. One example of data transformation through data analysis is, “To identify other questions that could be excluded, we used an arbitrary cutoff point where at least 45% of the variance of any item would be captured by all factors with an eigenvalue greater than one” (Ungar & Liebenberg, 2011, p.139). Additionally, there were sufficient data presented to support the findings (discussed in the data collection section of this essay). Furthermore, there is no evidence to suggest that contradictory data were taken into account; however, nonresponse rates and variance were considered. Finally, there is no evidence to suggest that the researchers critically examined their roles, potential biases, and influences during the analysis and selection of data for presentation.

Findings

The findings of the research study may be considered explicit, in terms of the development of a tool for the cross-cultural study of resilience that can account for the psychosocial resources available to youth globally (one of the goals of the research study), as the researchers state they:

…were able to identify the 58 questions of the pilot version of the CYRM that relate to resilience across all cultural groups participating in the study. However, although all questions showed relevance to each geographic subpopulation, the varying factor structures observed in response patterns indicate heterogeneity in how resilience is understood and negotiated across cultures and contexts. (Ungar & Liebenberg, 2011, p.141).

Thus, it may be said that the researchers developed a tool in accordance with their objective, or rather research aim. Additionally, it may be said that the findings are explicit, as the researchers state that they identified unique and common aspects of resilience, “By mixing qualitative and quantitative approaches to the development of a standardized measure, we have been able to identify both unique and common aspects of resilience that ensures the validity of a measure designed for use across cultures” (Ungar & Liebenberg, 2011, p.142).

The researchers do not adequately discuss evidence against their arguments: a limitations section is included in the article, but this does not outline evidence against the researchers’ arguments. In contrast, evidence for the researchers’ arguments is discussed in detail. For instance, the researchers describe the reliability of the CYMR-28 in great detail:

Using exploratory factor analysis and calculating Cronbach’s alphas for questions associated with each level of the model, the original 58-item version of the CYRM suggested reliability with Cronbach’s alpha scores for designated subsets of questions as follows: individual (24 items; .84), relational (7 items; .66), community (15 items; .79), and culture (12 items; .71). (Ungar & Liebenberg, 2011, p.136)

This is one example of how the researchers discuss evidence for their arguments. Additionally, the researchers discuss the credibility of their findings by outlining the methodological limitations of the research study. The researchers indicate that there are several methodological limitations, such as: breaking with procedures typically used for instrument design; positive wording of all CYRM questions; failure to employ CFA in the development of the CYRM (Ungar & Liebenberg, 2011, p.143). Furthermore, the findings of the research study are not entirely discussed in relation to the original research question, or rather the three goals of the research study. The researchers discuss the findings of the research study in relation to goal one and two, but they do not discuss the findings in relation to goal three, this being to determine the elements of a mixed methodology that are effective in the development of culturally sensitive psychological measures. This may be said as the researchers do not specifically state which measures in particular were most effective.

Value of the Research

The researchers discuss the contribution that the research study makes to the existing knowledge and understanding of child and youth resilience, both culturally and contextually, as the researchers discuss the capabilities of the CYRM-28:

Our results, informed by our qualitative findings, suggest that the CYRM-28 can provide a reliable representation of common factors related to resilience across all 14 research sites and a more specific understanding of which resources are associated with resilience as an outcome in different contexts. (Ungar & Liebenberg, 2011, p.144).

Therefore, the research study makes a contribution to the existing knowledge and understanding of child and youth resilience, both culturally and contextually, as the researchers create a tool for the cross-cultural study of resilience that can account for the psychosocial resources available to youth globally.

The limitations of the research study are only discussed in terms of methodological limitations (discussed in the findings section of this paper). Thus, the researchers do not fully explore and describe the limitations of the research study. Additionally, the researchers state that “Future research employing the CYRM will need to ensure samples of young people are somehow discriminated into two groups: those who LACs say are doing well and not doing well” (Ungar & Liebenberg, 2011, p.143); however, this is the only discussion/mention of future research, as new areas where research is necessary is not discussed in the article. Furthermore, the researchers do not discuss whether or how the findings of the research study can be transferred to other populations, nor do they consider other ways that the research may be used. Resultantly, the generalizability of the research study is unknown.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download