Dissecting the literature: the importance of critical appraisal — Royal College of Surgeons (2024)

This post was updatedin 2023.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.

Amanda Burls, What is Critical Appraisal?

Why is critical appraisal needed?

Literature searches using databases like Medline or EMBASE often result in an overwhelming volume of results which can vary in quality. Similarly, those who browse medical literature for the purposes of CPD or in response to a clinical query will know that there are vast amounts of content available. Critical appraisal helps to reduce the burden and allow you to focus on articles that are relevant to the research question, and that can reliably support or refute its claims with high-quality evidence, or identify high-level research relevant to your practice.

While most of us know not to believe everything we may read in a newspaper (or on Twitter), it’s also true that we cannot rely 100% on papers written in even the most prestigious academic journals. Different types of studies reported in the literature also have different strengths and weaknesses. Even if the contents of a research paper are reliable, it is sometimes difficult to find the specific information you are looking for and interpret it accurately.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognise any potential for bias.

Critical appraisal helps to separate what is significant from what is not. One way we use critical appraisal in the Library is to prioritise the most clinically relevant content for our Current Awareness Updates.

How to critically appraise a paper

There are some general rules to help you, including a range of checklists highlighted at the end of this blog. Some key questions to consider when critically appraising a paper:

• Is the study question relevant to my field?
• Does the study add anything new to the evidence in my field?
• What type of research question is being asked?
  A well-developed research question usually identifies three components: the group or population of patients, the studied parameter (e.g. a therapy or clinical intervention) and outcomes of interest.
• Was the study design appropriate for the research question?
  You can learn more about different study types and the hierarchy of evidence here.
• Did the methodology address important potential sources of bias?
  Bias can be attributed to chance (e.g. random error) or to the study methods (systematic bias).
• Was the study performed according to the original protocol?
  Deviations from the planned protocol can affect the validity or relevance of a study, e.g. a decrease in the studied population over the course of a randomised controlled trial.
• Does the study test a stated hypothesis?
  Is there a clear statement of what the investigators expect the study to find which can be tested, and confirmed or refuted.
• Were the statistical analyses performed correctly?
  The approach to dealing with missing data, and the statistical techniques that have been applied should be specified. Original data should be presented clearly so that readers can check the statistical accuracy of the paper.
• Do the data justify the conclusions?
  Watch out for definite conclusions based on statistically insignificant results, generalised findings from a small sample size, and statistically significant associations being misinterpreted to imply a cause and effect.
• Are there any conflicts of interest?
  Who has funded the study and can we trust their objectivity? Do the authors have any potential conflicts of interest, and have these been declared?

And an important consideration for surgeons:

• Will the results help me manage my patients?

At the end of the appraisal process you should have a better appreciation of how strong the evidence is, and ultimately whether or not you should apply it to your patients.

Further resources:

• How to Read a Paper by Trisha Greenhalgh
• The Doctor’s Guide to Critical Appraisal by Narinder Kaur Gosall
• CASP checklists
• CEBM Critical Appraisal Tools
• Critical Appraisal: a checklist
• Critical Appraisal of a Journal Article (PDF)
• Introduction to...Critical appraisal of literature
• Reporting guidelines for the main study types

Kirsty Morrison, Information Specialist