Worksheet for Critical Appraisal of Systematic Reviews ...



Critical Appraisal of a Meta-analysis

Jill Radtke

University of Pittsburgh

Worksheet for Critical Appraisal of Systematic Reviews/Meta-Analyses

Citation:

Martin, R.M., Gunnell, D., Owen, C.G., & Smith, G.D. (2005). Breast-feeding and

childhood cancer. International Journal of Cancer, 117(6), 1020-31.

What type of article is this (e.g., research/data-based, clinical paper, review, editorial)?

Research/data-based

If this is a research/data-based article, what makes it this type of article? Identify 2-3 key characteristics of the article.

1. Methodology: The article/study seeks to obtain data in a systematic fashion (e.g., the comprehensive, systematic search for articles; the choosing of articles for inclusion, based on criteria determined a priori).

2. Style: The article’s results and discussion are presented in an objective and frank manner (also discussing the limitations) in order that the reader may judge, implement, question, and/or disregard the evidence.

3. Original Findings: The study is a meta-analysis, combining results of many individual studies for new statistics that offer a more comprehensive answer as to whether breastfeeding is associated with a decreased risk of childhood cancers.

State the research question posed by the authors.

In epidemiologic studies examining the association between breastfeeding and childhood cancer, selected through systematic review for this study, what is the association between any breastfeeding and childhood cancer across studies, as compared to the never-having-been-breastfed child’s development of childhood cancer, and are these associations different depending upon type of cancer?

What is my clinical question?

In breast milk samples taken weekly in an industrialized area from postpartum women exclusively breastfeeding starting two days after birth and lasting until 26 weeks postpartum, is there any biochemical marker or element, in presence, or concentration, associated with an increased rate of childhood cancer development by 5 years of age in these women’s offspring, as compared to a similar group of breastfed children in a rural area?

Using PICO, identify the following if applicable:

P (= population): Children aged 0-19 whose cancer status/mortality was known, and whose breastfeeding status as an infant could be classified as “never breastfed” or “any breastfeeding”

I (= intervention): Breastfeeding

C (= comparison group): Children never having been breastfed

O (=outcome): Childhood cancer development

|Appraisal Guide |COMMENTS |

|I. Are the results valid/trustworthy? |

|1. Are the specific aims/research questions |The main research question, whether breast milk is associated with a decrease |

|clearly focused for this systematic |in the risk of childhood cancer, is clear both in the abstract and |

|review/meta-analysis? |introduction. Additionally, the specific aims, to determine “the consistency |

|a) What is being reviewed? |of associations [between childhood cancer development and breastfeeding] across|

|b) What is the population? |studies and to determine whether associations differed by cancer types” is |

|c) What are the independent variables (i.e., |clear in the introduction. However, the types of childhood cancer the authors |

|factors, exposures, interventions)? |are looking at and how breastfeeding is being classified (never vs. ever been) |

|d) What are the dependent variables (i.e., outcomes|is not clearly delineated. Rather, we are given this information in pieces |

|responses)? |throughout the article. This may be appropriate for a meta-analysis, as the |

|e) Were all clinically relevant (patient-important)|type of articles included should provide more direction and specificity for |

|outcomes considered? |specific aims. Nevertheless, giving this information up front would have made |

| |the article more readable. |

| | |

| |This meta-analysis reviewed the Medline database from 1966 (inception) until |

| |June 2004 for articles, papers, letters, abstracts and review articles |

| |pertaining to infant feeding and cancer. They also employed the auto-alert |

| |system and reviewed relevant reference lists (manually searched). In the end, |

| |of 1,415 “hits,” 46 articles were selected as relevant. Of these 46, 26 were |

| |selected for study inclusion (the discarded studies were discarded for a |

| |variety of reasons, including that some articles were commentary or review |

| |articles or did not focus on childhood cancer). Of these 26 articles, 2 were |

| |cohort/nested case-control studies, and the rest were case-control studies. |

| |All examined the association between childhood cancer and breastfeeding. The |

| |authors reviewed the individual studies for quality and characteristics, as |

| |shown in the tables in the article. The types of childhood cancers reviewed in|

| |the 26 studies included leukemia, acute lymphoblastic leukemia, acute |

| |nonlymphoblastic leukemia, Hodgkin’s disease, non-Hodgkin’s lymphoma, central |

| |nervous system cancers, neuroblastoma, malignant germ cell tumors, juvenile |

| |bone tumors, and other solid cancers. The meta-analysis looked at the |

| |associations of these cancers with the classification of never having been |

| |breastfed as compared to any or exclusive breastfeeding. The authors employed |

| |pooled odds ratios to determine if any breastfeeding was associated with any of|

| |the childhood cancers mentioned above, as well as childhood cancer in general. |

| |They also looked at heterogeneity for between study variation among many |

| |factors, chosen a priori, that might have impacted study results (i.e., study |

| |size, study design) using Cochran’s Q-statistic and the I2 statistic. If too |

| |much heterogeneity were found, it would not have been meaningful or possibly |

| |even valid to pool the odds ratios between the studies. The authors also |

| |looked at breastfeeding specifically for greater than 6 to 8 months as compared|

| |to never having been breastfed to determine any impact of prolonged |

| |breastfeeding on childhood cancer. Finally, the authors looked at the Egger |

| |test for small study bias, in order to determine if the small studies utilized |

| |in the review arrived at exaggerated effect sizes. |

| | |

| |The actual populations used in the individual articles included children from |

| |0-19 years whose breastfeeding status could be classified as “ever” or “never.”|

| |Additionally, their development of childhood cancer or mortality from cancer |

| |was known. |

| | |

| |The independent variables are the child’s breastfeeding classifications (e.g., |

| |never, ever, greater than 6-8 months); this includes the duration of |

| |breastfeeding. Age of the child might be considered an independent variable, |

| |but year of birth is similar, and this was considered a possible confounder in |

| |the study. |

| | |

| |The dependent variable was the rates of childhood cancer: i.e., the number of |

| |children with each type of cancer. |

| | |

| | |

| |I believe the relevant outcome is considered in the context of this study’s |

| |goal. We are interested in childhood cancer development and its association |

| |with breastfeeding in order to ascertain whether breastfeeding should be |

| |recommended to mothers for this potential benefit. The authors addressed odds |

| |ratios of childhood cancer development for each major type of childhood cancer |

| |and childhood cancer in general when any or no breastfeeding occurs. They |

| |found that breastfeeding’s effect on childhood cancer is negligible, but should|

| |be recommended anyway for its other known benefits. Truly, the most important |

| |outcome here is cancer development, in order to establish or negate the |

| |relationship between childhood cancer and breastfeeding, but other outcomes, |

| |such as mortality and age at diagnosis of cancer and its association with |

| |breastfeeding, might have been considered. |

|2. Is the search for studies comprehensive (i.e., |The Medline database was used, combining keywords for infant feeding and |

|did the authors look for the appropriate sort of |cancer. Additionally, auto-alerts were employed on Medline, and the authors |

|studies)? |manually searched the reference lists of eligible studies and previous |

|a) What bibliographic databases were used? |meta-analyses for the specific cancers. |

|b) What years were searched? | |

|c) What languages were searched? d) Were the study |Medline was searched from its inception in 1966 to June 2004. |

|inclusion criteria appropriate (i.e., study design,| |

|participants, intervention, and outcomes of |It was not mentioned what languages were searched. |

|interest)? | |

|e) Was the inclusion process discussed? |Yes, the study inclusion criteria appeared appropriate. No specifications of |

|f) Did at least 2 individuals review articles for |specific study design were mentioned under “Material and methods,” however it |

|selection? |was noted that several studies were excluded because they were reviews and |

|g) What process did they use to reach agreement on |commentaries. Thus, we see that the authors are ensuring the quality of the |

|article selection? |meta-analysis by including studies that are primary sources, and excluding |

|h) Was agreement achieved? |secondary sources. The age criterion, 0-19 years, seems appropriate, as we are|

|i) Were all relevant studies included in the |studying “childhood” cancer. Finally, the meta-analysis criterion included |

|review? |studies comparing those ever breastfed to those never breastfed, and estimates |

|j) Were any important studies missed? Explain. |of the association of having ever been breastfed to cancer outcomes had to be |

| |able to be derived or available in the study. This is appropriate, given the |

| |main objective of our study: to determine the association of breastfeeding with|

| |the development of childhood cancer. |

| | |

| |The inclusion process was discussed, as noted above, under “Material and |

| |methods,” and reasons for excluding certain studies were also discussed under |

| |“results” on page 1021. |

| | |

| |It was not mentioned how many individuals reviewed the articles for selection. |

| |(Thus, “g” and “h” are n/a.) |

| | |

| |There may be some relevant studies not included in the review. The authors |

| |mentioned that they had to exclude studies matching their inclusion criteria |

| |because they were already mentioned in their article previously. Additionally,|

| |they had to exclude some studies because estimates of the odds ratios could not|

| |be derived. This does not mean, however, that these studies did not offer |

| |valuable insight and evidence for our research question. For this particular |

| |meta-analysis, though, they did not fit. Additionally, the authors might have |

| |tried to search other databases, confer with “experts” in the area to uncover |

| |unpublished sources of information, search other languages, etc. It was not |

| |mentioned if these techniques were employed or not, but they might have |

| |contributed relevant and valuable data to our meta-analysis. |

| | |

| |Additionally, I searched OVID Medline for the same study years as the article’s|

| |authors used, using the keywords, “infant feeding” and “cancer.” I had 15 |

| |“hits”, but I did not find any relevant articles fitting the study’s inclusion |

| |criteria that the authors did not use in their systematic review. However, one|

| |study uncovered did seem to offer an important opposing viewpoint. It looked |

| |at breastmilk samples’ contamination with heavy metals and how this can |

| |contribute to cancer in later life for infants who breastfeed (Lutter, Lyengar,|

| |Barnes, Chuvakova, Kazbekova, & Sharmanov, 1998). While not meeting study |

| |inclusion criteria completely, it does make an important and valid point about |

| |environmental influence on breastmilk and subsequent cancer development in |

| |offspring. In my opinion, this should be considered in future studies looking |

| |at childhood cancer risk associated with breastfeeding. |

|3. Is the validity of the included studies |The article mentions that, unlike randomized control trials, there are no |

|adequately assessed? |widely accepted criteria used to judge the validity and quality of |

|a) What criteria were used to evaluate validity of |observational studies (which this meta-analysis used). Therefore, the study |

|individual studies (e.g., use of |employed quality criteria chosen a priori from previous meta-analyses and |

|intention-to-treat, if an intervention study; |systematic reviews examining infant feeding. They seem to be appropriate, as |

|efforts to obtain missing information; efforts to |they are chosen a priori. However, the model criteria were derived from |

|minimize publication bias; assessment of |articles whose titles implied they were looking at the association between |

|methodologic quality)? |breastfeeding and indicators of overweight. We are examining the relationship |

|b) Were these criteria appropriate? Explain. |between breastfeeding and childhood cancer. So, although the criteria may be |

|c) Did at least 2 reviewers examine individual |easily transferable to our meta-analysis, we cannot be sure. It would have |

|studies for validity? |been appropriate to also consider criteria from systematic reviews based on |

|d) What process did reviewers use to reach |childhood cancer. |

|agreement on the validity of individual studies? | |

|e) Was agreement achieved? |Some of the criteria used and accounted for by the authors included study size |

| |(less than or equal to 500, or greater than 500—cited by the authors as the |

| |number used for effect size in meta-analyses for breastfeeding), study design |

| |(cohorts considered more robust than case-control), whether breastfeeding was |

| |the main outcome measure, whether maternal recall was used to assess |

| |breastfeeding after 1 year (memory problems), early exposure to infectious |

| |disease in the child (because this has been associated with breastfeeding and |

| |cancer, according to the authors), whether effects estimates controlled for |

| |social factors in the child or adult, reproductive factors, if the study was |

| |population-based (potential for selection bias), if the control or case |

| |response was less than 80% (selection bias), whether prevalence of |

| |breastfeeding was at least 70% (authors did not justify this criteria), the |

| |region the study was done in (as different views of breastfeeding, different |

| |cancer rates in different regions), and the year of birth of the child |

| |participants (as breastfeeding is viewed differently in different generations, |

| |and formula has changed dramatically over the years). This information can be |

| |found on p.1026, Table II. |

| | |

| |Throughout the analysis/results, several studies are mentioned by the authors |

| |as having limited validity for one reason or another, and the analysis is done |

| |with and without these studies for this reason. For example, one study was |

| |only given only in abstract form and was thus considered poor quality for the |

| |meta-analysis. Removing it changed the association between Hodgkin’s disease |

| |and breastfeeding quite significantly (p.1026). |

| | |

| | |

| |It was not mentioned whether more than one reviewer arrived at the validity |

| |criteria or assessed the studies for validity. |

|4. What are the study characteristics? |26 individual studies were included. These included 2 cohort/nested |

|a) How many individual studies were included? |case-control studies, and 24 case-control studies. |

|b) What types of studies were included (e.g., | |

|randomized controlled trials, case-control studies,|The study sample sizes were in a wide range. Some studies looked at individual|

|cohort studies, cross-sectional studies)? |cancer types and gave number of cases with each type. Some studies looked one |

|c) What were the sample sizes for the studies? If |cancer type or childhood cancer in general and gave the total sample size. The|

|applicable, what are the sample sizes for the study|smallest sample was composed of 33 subjects and looked at childhood cancer in |

|groups? |general. The largest study included over 3,000 subjects, and broke down |

|d) In what countries were the studies conducted? |childhood cancer into several specific cancer types (and included number of |

|e) If an intervention was the primary independent |subjects with each type of cancer). |

|variable under investigation: What was the duration| |

|of treatment? |The studies were collected from a wide variety of countries and areas, |

|What was the adherence to the study treatment? |including among them, the United States, England, Ireland, Greece, Australia, |

|Were adverse results reported? |Italy, United Kingdom, United Arab Emirates, Shanghai, Canada, New Zealand, |

| |etc. The meta-analysis was very diverse in study location. |

| | |

| |The “intervention” was breastfeeding, and the classifications used in the |

| |meta-analysis were “ever” or “never” breastfed. Additionally, the authors did |

| |a second small analysis comparing “never” breastfed to breastfed greater than |

| |6-8 months. This was the only reference to duration of breastfeeding. We are |

| |not told how long those who “ever” breastfed nursed. Because these studies |

| |were observational in nature, breastfeeding was not classified whether it was |

| |“adhered” to. Instead, retrospectively, these children’s parents classified |

| |whether they did or did not ever breastfeed. |

| | |

| |Although some studies showed no significant decrease in risk for certain |

| |childhood cancers associated with breastfeeding, it was not noted that |

| |breastfeeding was ever positively associated with childhood cancer |

| |significantly. |

|5. What are the results of systematic |The odds ratios and 95% confidence intervals (CI’s) for the individual studies |

|review/meta-analysis? |were reported for childhood cancer (individual types and overall, depending on |

|a) Were the results in terms of effect sizes of the|study), comparing never breastfed children to breastfed children. This was |

|individual studies clearly reported? |shown in Figure 1 (p.1027) in the “Results” section, but was difficult to read.|

|b) Were these results consistent from individual |Some studies’ odds ratios were discussed individually, but most were not. |

|study to individual study? Explain. |Additionally, for the “other” cancers in childhood, such as germ cell tumors |

|c) Were the results of the different studies |(in which only one study was used), no odds ratios were provided for any |

|similar (e.g., Were tests of homogeneity conducted?|studies. We are simply told that breastfeeding and that cancer are not |

|What did they show?) |significantly associated. |

|d) Were the individual study findings combined | |

|statistically? (If results were combined, did it |For childhood cancers as a whole, it was noted that there was considerable |

|make sense to do so? Explain. |variation in effects estimates from each individual study comparing all |

|e) If results were not combined, should they have |childhood cancer (this was attributed to large between study heterogeneity, and|

|been? Explain.) |one particular study that largely contributed to the heterogeneity). Judging |

|f) Were the reasons for study variations discussed?|by Figure 1 and the article’s commentary, studies looking at acute |

| |lymphoblastic leukemia, acute nonlymphoblastic leukemia, and Hodgkin’s disease |

| |had variable odds ratios and 95% CI’s. However, it was very difficult to read |

| |Figure 1. These results in table form would have been helpful. |

| | |

| |Tests of homogeneity were not used. Instead, tests of heterogeneity were |

| |utilized to detect differences in between-study quality criteria, as mentioned |

| |previously: the Q-test for heterogeneity and the I2 value (the I2 value was |

| |used to supplement the Q-statistic, because its power to detect heterogeneity |

| |is not affected by a small number of studies, which we have here). These tests|

| |showed considerable heterogeneity among studies looking at all childhood |

| |cancers. Studies looking at all childhood leukemia showed moderate |

| |heterogeneity (I2 = 61%). Studies looking at acute lymphoblastic leukemia, and|

| |Hodgkin’s disease, and neuroblastoma showed little between study heterogeneity |

| |(I2 =16%, 0%, and 0%, respectively), however, only 3 studies were chosen |

| |looking at neuroblastoma. Non-Hodgkin’s lymphoma, other central nervous system|

| |cancers, juvenile bone tumors, and germ cell tumors failed to establish an |

| |association in this meta-analysis with breastfeeding, and this may be due to |

| |the small number of studies looking at these cancers. Therefore, estimates of |

| |heterogeneity were not as meaningful in these cases, and were classified as |

| |either 0% or N/A. |

| | |

| |The individual study findings were combined into pooled odds ratios. This made|

| |sense, as the studies were looking at the same variables (ever or never |

| |breastfed and childhood cancer). Studies that looked at the same childhood |

| |cancers were pooled together, and all noted whether the child was ever or never|

| |breastfed. Additionally, it made sense to pool to the odds ratios together, as|

| |odds ratios were available or could be derived for each study chosen for |

| |inclusion, per the authors. |

| | |

| |The reasons for the study variations (heterogeneity) are discussed. The |

| |authors note that one particular study, by Smulevitch et al., contributed to |

| |much heterogeneity for several types of childhood cancer between studies. They|

| |state that this is so, because about 1% of controls were breastfed less than a |

| |month, which contaminates the control group. By removing the study, |

| |between-study heterogeneity for these types of cancer are reduced. |

|6. What are the methodological strengths and |There are a number of methodological weaknesses. First, there were a small |

|weaknesses of this systematic review/meta-analysis?|number of studies analyzed in the meta-analysis, per each type of childhood |

| |cancer. Thus, the authors comment that significant associations between |

| |breastfeeding and rates of childhood cancer might not have been found, when in |

| |fact, they do exist. Second, the study is based on observational studies, |

| |which leads to the very real possibility of confounding. Although the authors |

| |controlled for some of these factors (and found that the results did not change|

| |when controlled for) it was impossible to rule potential masking of effects, |

| |because the etiology of breast milk’s role in childhood cancer development is |

| |poorly understood. Randomized control trials in meta-analysis are more |

| |powerful, because they nearly eliminate confounding. |

| | |

| |Another weakness is the possibility of publication bias. That is, perhaps |

| |significant associations between reduction in certain childhood cancers and |

| |breastfeeding were reflective of the fact that studies finding no effect were |

| |not published. The authors did not note whether they searched for negative |

| |findings or unpublished literature, which may have served to ease our |

| |discriminating minds. Other methodological weaknesses include the fact that |

| |control response rates were less than 80% in most studies included (possible |

| |selection bias, as participating controls are likely to be more socially |

| |affluent and breastfeed), 85% of the studies were based on maternal recall of |

| |infant feeding (possible recall bias), and most studies did not look at the |

| |timing of breastfeeding initiation, exclusivity of breastfeeding, and |

| |supplementation. Because of this last point, it was difficult to establish |

| |dose effects of breastfeeding, which would have been more methodologically |

| |sound in tying reductions in childhood cancer to breastfeeding. |

| | |

| |The methodological strengths of this article included the fact that the authors|

| |attempted to control for many possible confounders (listed in Table II), |

| |electronic literature searches were supplemented with manual searching of |

| |reference lists and auto-alerts, a wide range of time was utilized in the |

| |search, most childhood cancers were considered, as well as childhood cancer as |

| |a whole, and the authors did statistical analysis with and without studies that|

| |accounted for a large amount of variability or had questionable validity, in |

| |order to ensure that meta-analysis results were more valid and generalizable. |

|II. What are the results/findings? |

|1. What are the overall results of the systematic |A summary measure was given for rates of all childhood cancer’s association |

|review/meta-analysis? |with breastfeeding: that is, there is a 22% decrease in childhood cancer with a|

|a) If a summary measure was given, how large was |95% CI of 1-39% associated with breastfeeding. The pooled odds ratio for this |

|the overall effect (e.g., treatment effect, measure|is 0.96 (95% CI = 0.88-1.04), meaning that there is not strong evidence that |

|of association)? |breastfeeding is associated with a decrease in childhood cancer. In fact, the |

|b) Are the results clinically meaningful? Explain.|number needed to treat is given by the authors on p.1030: “breast-feeding would|

| |prevent at most 5% of these cases, if the prevalence of breast-feeding was |

|c) Based on the confidence intervals of the summary|increased from 50% to 100%.” |

|measure (if provided), how precise was the result? | |

|Explain. |The authors deny that the results are clinically meaningful, as evident in |

|d) Does the lower limit of the confidence interval |their number needed to treat statement above. They state that there is |

|include clinical relevant effects? |moderate evidence that breastfeeding is associated with a decrease in acute |

|e) Does the upper limit exclude clinically relevant|lymphoblastic leukemia, Hodgkin’s disease, and neuroblastoma (but the latter |

|effects? |two are based on one study each). The authors do recommend breastfeeding for |

| |other health benefits, but even state in the abstract, “Even if causal, the |

| |public health importance of these associations may be small.” |

| | |

| |Based on the 95% CI of the odds ratio, (0.88-1.04), the result was within a |

| |narrow window (rather precise), but precisely non-significant. A CI of an odds|

| |ratio that includes 1 (which 0.88-1.04 does) cannot claim that decreased |

| |childhood cancer rates are associated with breastfeeding, any more than not |

| |breastfeeding. The confidence interval’s upper limit probably greatly limits |

| |clinical significance, because it is so close to 1 (1 being equal odds of |

| |developing childhood cancer with or without breastfeeding). However, it does |

| |not exclude clinical significance, because it is greater than 1 (indicating |

| |that breastfeeding, as opposed to not breastfeeding, may be associated with |

| |childhood cancer). The lower limit does not include clinical relevance—it is |

| |very close to 1, in the opposite direction, indicating that not breastfeeding |

| |may be associated with decreased rates of childhood cancer to a greater degree |

| |than breastfeeding (the opposite of our hypothesis), but this was not |

| |significant. |

|III. How can I apply the results/findings? |

|1. What relevance do the findings have to nursing |As the results did not provide much clinical relevance in terms of |

|practice? |breastfeeding’s association with decreased rates of childhood cancer, nurses |

| |must use caution in advising parents to breastfeed for this reason. However, |

| |the results do show marginal significance in the association of breastfeeding |

| |with decreased rates of childhood acute lymphoblastic leukemia, Hodgkin’s |

| |disease, and neuroblastoma. Thus, as nurses we must consider the risk:benefit |

| |ratio. Breastfeeding has been shown to have many other positive outcomes in |

| |children (decreased allergies, ear infections, etc.), and this study, while not|

| |showing definitive evidence that breastfeeding decreases childhood cancer, does|

| |not show any increased risk of childhood cancer associated with breastfeeding. |

| |Thus, as nurses we should recommend breastfeeding for a myriad of reasons, and |

| |for the fact that it may possibly play a role in decreasing some childhood |

| |cancer. |

| | |

| |These findings also are relevant in the sense that we need more research, from |

| |nurses and other disciplines, in the area of breastfeeding’s association with |

| |childhood cancer. Particularly, as the authors mention, we need studies |

| |looking at the duration and exclusivity of breastfeeding and its relationship |

| |to childhood cancer. |

|2. Discuss how the findings can be applied to |As stated above, these findings suggest that nurses should continue to |

|practice. |encourage and support breastfeeding for the many health benefits that have been|

| |confirmed in both mother and child. As this meta-analysis did not show that |

| |childhood cancer is decreased for breastfed children, we should not advocate |

| |breastfeeding to families based on this reason. To do so would be unethical, |

| |as we have no sound evidence at this point to confirm the relationship. |

References

Lutter, C., Lyengar, V., Barnes R., Chuvakova, T., Kazbekova, G., & Sharmanov, T. (1998). Breast milk contamination in Kazakhstan: implications for infant feeding. Chemosphere, 37(9-12), 1761-72.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download