Ranking Literature - USFWS



Attachment 2

Ranking Evidence

A general order of preference for information sources

I. Primary Source: The information source from which evidence-based knowledge is derived. It has as a major component evidence derived directly from fully described (or referenced) formal observation, procedures or experiments performed with valid, scientifically accepted methods. In its strongest form, this material is usually (but not only) a paper in a refereed scientific publication.

A. Scientific Refereed Journal: A journal that has a mission of publicizing and storing primary scientific evidence. By convention evidence published in such a journal is subjected to anonymous review by several experts (referees) in the field prior to publication and is published only once. The methods used to acquire the evidence must be described (or a primary reference cited) with sufficient detail to allow knowledgeable person to critically appraise the study design, replicate the study, or both. Although the review processes these journals use is designed to ensure the integrity of the procedures, data, and analyses presented in a paper, a significant percentage of published papers still contain serious flaws, some which render the study invalid. The presence of these flaws is one of the primary reasons why consulting biologists must assess published literature to determine if a paper is “the best scientific information available.”

Repetition of a study by other researchers, either in whole or in part, helps support or refute the conclusions of a previous study and is essential to the long-term development and acceptance of scientific theory. A paper whose results and conclusions have been verified by independent study or studies is generally a more reliable source that a paper whose results have not been verified in this way.

B. Scientific Proceedings: A collection of current research reports, usually presented as brief abstracts, from a scientific meeting. These are a much weaker form of a primary source than is a full scientific journal article because the selection of the abstracts, which are of varying quality, is based on a much more cursory review, the reports are usually incomplete, and much of the work is in-progress. As such, these represent a form of "pre-primary" source.

C. Integrative Source (Studies): A source reporting the results of meta-analysis, which is a statistical procedure to mathematically combine the results from a number of valid studies to arrive at a stronger conclusion. An exhaustive search for all of the studies relevant to the question at hand and a critical analysis of these studies to exclude those with serious design or procedural flaws is required. Integrative studies are based on objective quantitative analysis rather than the more subjective analysis of the conventional critical review.

II. Secondary Source: An information source that does not have as a major component the description of formal observations or experiments but rather is synthesized from some combination of primary sources, experience, or authoritative belief (dogma). The primary literature used may have been selected in a biased or incomplete fashion and may have been used without comprehensive critical appraisal to establish the relative strength of evidence in each source. Examples of secondary sources are review articles in publications like Annual Reviews in Ecology and Systematics.

III. Tertiary Source: A compilation of information for application across a broad spectrum, typically represented by class notes and textbooks intended for use in core courses. The strength of the underlying evidence is not indicated and any current controversy between researchers in the area is not addressed. The bibliography is usually predominately secondary literature and is usually intended to provide the interested reader with entry points to the underlying primary literature. Much of the evidence-based information contained in textbooks is filtered sufficiently that it is accepted by most all of the experts in the field, much of it is unlikely to change in the future, and most of the changes will be minor. However, depending on the field, textbooks contain a varying amount of dogma and interpretations of facts that will change with the progress of research in the area, sometimes significantly.

IV. Derivative Service: A service that presents collections of abstracts, usually from a wide selection of primary literature, selected to meet the interests of a particular group of clinicians. Some derivative services, like Biosis, include copies of the abstract that was written by the authors of the journal article. Abstracts vary in quality and abstracters may interpret the evidence of the paper differently than intended by the original authors.

V. “Gray” Literature: This category consists of publications that are not “published” or contained in indices (like First Search or Biosis) that make it easier for other researchers to acquire and examine the results. Gray literature general includes documents issued by government agencies (federal, state, or local), private consultancies, non-governmental agencies, and private organizations. The quality of the information contained in these documents can compare to that of published, refereed scientific journals (some “gray” literature undergoes extensive peer-review before publication) or it can compare to little more than anecdotal evidence.

VI. Other, non-peer reviewed sources: This is a catch-all category.

A. Substantive News or General Interest: This category consists of mass media publications that target educated audiences. Periodicals sometime cite sources, though more often do not. A staff member, a scholar, or a free-lance writer may write articles in these publications. The main purpose of these periodicals is to provide information, in a general manner, to a broad audience. Examples include major newspapers and monthly magazines.

B. Popular Sources: This category includes mass media publications intended primarily to entertain audiences and/or to promote a viewpoint. These publications rarely, if ever, cite sources. Information in these publications is often second or third-hand; the original source is sometimes obscure. Articles are intended to meet a minimal education level and there is generally little depth to the content of these articles.

C. Sensational Sources: The main purpose of sensational publications is to cater to popular superstitions. Their language is elementary and occasionally inflammatory or sensational. They assume a certain gullibility in their audiences.

VII. Professional Judgment:

VIII. Online Sources:

IX. Anecdotal, Personal Communication:

Sources Hierarchy

Additional ways to rank evidence based on the source it comes from:

I. Type of Research of Information:

A. Controlled, experimental study (e.g., addressing cause-effect relationships)

1. Repeated with consistent results

2. Not repeated or inconsistent results

B. Statistically designed observational study (e.g., hypothesis test addressing correlations)

1. Repeated with consistent results

2. Not repeated or inconsistent results

C. Inventory and monitoring (e.g., trends information from consistent sampling protocol)

1. In area/conditions/time period of concern

2. Otherwise

D. Meta-analysis or synthesis, and modeling (level in hierarchy depends on the level of information used in the synthesis/model)

E. Expertise-based professional judgment

1. From structured process

2. Informal; uncontrolled

F. Anecdotal; all other sources of opinion

II. Information Source:

A. Established journal or monograph with national or international coverage; book from major publisher; covers primary research or data

B. Any other peer-reviewed journal or book covering primary data

C. Unpublished reports with primary data that have been peer reviewed at Level 1 (see below; e.g., some agency report series)

D. Unpublished conference proceedings or online source with primary data

E. Secondary sources: peer-reviewed journal article or book that reviews or synthesizes data, possibly including observational data or subjective opinion by an expert author

F. Unpublished reports with primary data that have been peer reviewed at level 2

G. Textbook or other broadly synthetic book or compilation, using secondary and tertiary sources

H. Abstracts by secondary services (e.g., Biosis)

I. Unpublished primary data reports that have not been peer reviewed

J. Any other secondary or tertiary source (e.g., not by the researcher/observer), or personal communication.

III. Peer Review:

A. Level 1: Anonymous, independent peer review conducted by high-credibility third party (e.g., a journal editor)

B. Level 2: Invited external review by known experts and/or public comment received from experts (e.g., status report review)

C. Internal review.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download