LibGuides: Evidence Based Practice: Study Designs & Evidence Levels

Please Note: Froedtert Health has implemented a new security protocol which has resulted in the degradation of on-campus access to MCW Libraries' electronic resources. Users on the Froedtert network must sign in via OpenAthens when attempting to access electronic library resources (including UpToDate, VisualDx, etc.). MCW Libraries is aware of this change and is working to provide a permanent access solution. If you have MCW credentials, please sign in via MCW employees and students when prompted. If you are a Froedtert employee, click here to register for an OpenAthens account or view more information.

Introduction

This section reviews some research definitions and provides commonly used evidence tables.

Levels of Evidence Johns Hopkins Nursing Evidence Based Practice

Evidence Levels	Quality Guides
Level I Experimental study, randomized controlled trial (RCT) Systematic review of RCTs, with or without meta-analysis	A High quality: Consistent, generalizable results; sufficient sample size for the study design; adequate control; definitive conclusions; consistent recommendations based on comprehensive literature review that includes thorough reference to scientific evidence
Level II Quasi-experimental study Systematic review of a combination of RCTs and quasi experimental, or quasi-experimental studies only, with or without meta-analysis	B Good quality: Reasonably consistent results; sufficient sample size for the study design; some control, fairly definitive conclusions; reasonably consistent recommendations based on fairly comprehensive literature review that includes some reference to scientific evidence
Level III Non-experimental study Systematic review of a combination of RCTs, quasi-experimental and non-experimental studies, or non-experimental studies only, with or without meta-analysis Qualitative study or systematic review with or without a meta-synthesis	C Low quality or major flaws: Little evidence with inconsistent results; insufficient sample size for the study design; conclusions cannot be drawn

Evidence Levels

Quality Guides

Level IV
Opinion of respected authorities and/or nationally recognized
expert committees/consensus panels based on scientific evidence

Includes:
Clinical practice guidelines
Consensus panels

A High quality: Material officially sponsored by a professional, public, private organization, or government agency; documentation of a systematic literature
search strategy; consistent results with sufficient numbers of well-designed studies;
criteria-based evaluation of overall scientific strength and quality of included studies
and definitive conclusions; national expertise is clearly evident; developed or
revised within the last 5 years

B Good quality: Material officially sponsored by a professional, public, private
organization, or government agency; reasonably thorough and appropriate
systematic literature search strategy; reasonably consistent results, sufficient
numbers of well-designed studies; evaluation of strengths and limitations of
included studies with fairly definitive conclusions; national expertise is clearly
evident; developed or revised within the last 5 years

C Low quality or major flaws: Material not sponsored by an official organization or agency; undefined, poorly defined, or limited literature search strategy; no evaluation of strengths and limitations of included studies, insufficient evidence with inconsistent results, conclusions cannot be drawn; not revised within the last 5 years

Evidence Levels

Organizational Experience

Level V
Based on experiential and non-research evidence

Includes:
Literature reviews
Quality improvement, program or financial evaluation
Case reports
Opinion of nationally recognized experts(s) based on experiential evidence

A High quality: Clear aims and objectives; consistent results across multiple settings; formal quality improvement, financial or program evaluation methods used; definitive conclusions; consistent recommendations with thorough reference to scientific evidence

B Good quality: Clear aims and objectives; consistent results in a single setting;
formal quality improvement or financial or program evaluation methods used;
reasonably consistent recommendations with some reference to scientific evidence

C Low quality or major flaws: Unclear or missing aims and objectives; inconsistent
results; poorly defined quality improvement, financial or program evaluation
methods; recommendations cannot be made

Literature Review, Expert Opinion, Case Report, Community
Standard, Clinician Experience, Consumer Preference:
A High quality: Expertise is clearly evident; draws definitive conclusions; provides
scientific rationale; thought leader(s) in the field

B Good quality: Expertise appears to be credible; draws fairly definitive conclusions;
provides logical argument for opinions

C Low quality or major flaws: Expertise is not discernable or is dubious; conclusions
cannot be drawn

Dang, D., & Dearholt, S. (2017). Johns Hopkins nursing evidence-based practice: model and guidelines. 3rd ed. Indianapolis, IN: Sigma Theta Tau International. www.hopkinsmedicine.org/evidence-based-practice/ijhn_2017_ebp.html

Identifying the Study Design

The type of study can generally be figured out by looking at three issues:

Q1. What was the aim of the study?

To simply describe a population (PO questions) = descriptive
To quantify the relationship between factors (PICO questions) = analytic.

Q2. If analytic, was the intervention randomly allocated?

Yes? = RCT
No? = Observational study

For an observational study, the main type will then depend on the timing of the measurement of outcome, so our third question is:

Q3. When were the outcomes determined?

Some time after the exposure or intervention? = Cohort study ('prospective study')
At the same time as the exposure or intervention? = Cross sectional study or survey
Before the exposure was determined? = Case-control study ('retrospective study' based on recall of the exposure)

Centre for Evidence-Based Medicine (CEBM)

Definitions of Study Types

Case report / Case series: A report on a series of patients with an outcome of interest. No control group is involved.

Case control study: A study which involves identifying patients who have the outcome of interest (cases) and patients without the same outcome (controls), and looking back to see if they had the exposure of interest.

Cohort study: Involves identification of two groups (cohorts) of patients, one which received the exposure of interest, and one which did not, and following these cohorts forward for the outcome of interest.

Randomized controlled clinical trial: Participants are randomly allocated into an experimental group or a control group and followed over time for the variables/outcomes of interest.

Systematic review: A summary of the medical literature that uses explicit methods to perform a comprehensive literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.

Meta-analysis: A systematic review that uses quantitative methods to synthesize and summarize the results.

Meta-synthesis: A systematic approach to the analysis of data across qualitative studies. -- EJ Erwin, MJ Brotherson, JA Summers. Understanding Qualitative Meta-synthesis. Issues and Opportunities in Early Childhood Intervention Research, 33(3) 186-200.

Cross sectional study: The observation of a defined population at a single point in time or time interval. Exposure and outcome are determined simultaneously.

Prospective, blind comparison to a gold standard: Studies that show the efficacy of a diagnostic test are also called prospective, blind comparison to a gold standard study. This is a controlled trial that looks at patients with varying degrees of an illness and administers both diagnostic tests — the test under investigation and the “gold standard” test — to all of the patients in the study group. The sensitivity and specificity of the new test are compared to that of the gold standard to determine potential usefulness.

Qualitative research: answers a wide variety of questions related to human responses to actual or potential health problems.The purpose of qualitative research is to describe, explore and explain the health-related phenomena being studied.

Retrospective cohort: follows the same direction of inquiry as a cohort study. Subjects begin with the presence or absence of an exposure or risk factor and are followed until the outcome of interest is observed. However, this study design uses information that has been collected in the past and kept in files or databases. Patients are identified for exposure or non-exposures and the data is followed forward to an effect or outcome of interest.

(Adapted from CEBM's Glossary and Duke Libraries' Intro to Evidence-Based Practice)

American Association of Critical Care Nursing-- Levels of Evidence

Level A Meta-analysis of multiple controlled studies or meta-synthesis of qualitative
studies with results that consistently support a specific action, intervention
or treatment

Level B Well designed controlled studies, both randomized and nonrandomized, with
results that consistently support a specific action, intervention, or treatment

Level C Qualitative studies, descriptive or correlational studies, integrative reviews,
systematic reviews, or randomized controlled trials with inconsistent results

Level D Peer-reviewed professional organizational standards, with clinical studies to
support recommendations

Level E Theory-based evidence from expert opinion or multiple case reports

Level M Manufacturers’ recommendations only

Armola RR, Bourgault AM, Halm MA, Board RM, Bucher L, Harrington L, Heafey CA, Lee R, Shellner PK, Medina J. (2009) AACN levels of evidence: what's new? J.Crit Care Nurse. Aug;29(4):70-3.

Flow Chart of Study Designs

Figure: Flow chart of different types of studies (Q1, 2, and 3 refer to the three questions below in "Identifying the Study Design" box.)
Centre for Evidence-Based Medicine (CEBM)

What is a "Confidence Interval (CI)"?

A confidence interval (CI) can be used to show within which interval the population's mean score will probably fall. Most researchers use a CI of 95%. By using a CI of 95%, researchers accept there is a 5% chance they have made the wrong decision in treatment. Therefore, if 0 falls within the agreed CI, it can be concluded that there is no significant difference between the two treatments. When 0 lies outside the CI, researchers will conclude that there is a statistically significant difference.

Halfens, R. G., & Meijers, J. M. (2013). Back to basics: an introduction to statistics. Journal Of Wound Care, 22(5), 248-251.

What is a "p-value?"

Categorical (nominal) tests
This category of tests can be used when the dependent, or outcome, variable is categorical (nominal), such as the difference between two wound treatments and the healing of the wound (healed versus nonhealed). One of the most used tests in this category is the chisquared test (χ2). The chisquared statistic is calculated by comparing the differences between the observed and the expected frequencies. The expected frequencies are the frequencies that would be found if there was no relationship between the two variables.

Based on the calculated χ2 statistic, a probability (p value) is given, which indicates the probability that the two means are not different from each other. Researchers are often satisfied if the probability is 5% or less, which means that the researchers would conclude that for p < 0.05, there is a significant difference. A p value ≥ 0.05 suggests that there is no significant difference between the means.

Halfens, R. G., & Meijers, J. M. (2013). Back to basics: an introduction to statistics. Journal Of Wound Care, 22(5), 248-251.