Statistical Significance (2024)

Introduction

In research, statistical significancemeasures the probability of the null hypothesis being true compared to the acceptable level of uncertainty regarding the true answer. We can better understand statistical significance if we break apart a study design.[1][2][3][4][5][6][7]

When creating astudy, the researcher has tostart with a hypothesis; that is, they musthave some idea of what they think the outcome may be.For example, a study is researching a new medication to lower blood pressure. The researcher hypothesizes that the new medication lowers systolic blood pressure by at least 10 mm Hg compared to not taking thenewmedication. The hypothesis can be stated: "Taking the new medication will lower systolic blood pressure by at least 10 mm Hg compared to not taking the medication." In science, researchers can never prove any statement as there are infinite alternatives as to why the outcome may haveoccurred. They can onlytry todisproveaspecific hypothesis. The researcher must then formulate a question they can disprove while concluding that the new medication lowers systolic blood pressure. The hypothesis to be disproven is the null hypothesis and typically the inverse statement of the hypothesis. Thus, the null hypothesis for our researcherwould be, "Taking the new medication will not lower systolic blood pressure by at least 10 mm Hg compared to not taking the new medication." The researcher now has the null hypothesis for the research and must specify the significance level or level of acceptable uncertainty.

Even when disproving ahypothesis, the researchercan notbe 100% certainof the outcome. The researcher must then settle for some level of confidence, or the degree of significance, for which they want to be confident their finding is correct. The significance level is given the Greek letter alpha and specified asthe probability the researcher is willing to be incorrect. Generally, a researcherwants to be correct about their outcome 95% of the time, sothe researcher is willing to be incorrect 5% of the time. Probabilities are decimals, with 1.0 being entirely positive (100%) and 0 being completely negative (0%). Thus, the researcher who wants to be 95% sure about the outcome of their study iswilling to be wrong about the result 5% of the time. The alpha is the decimal expression of how much they are ready to be incorrect. For the current example, the alpha is 0.05. Thelevel of uncertainty the researcher is willing to accept (alpha or significance level)is 0.05, or a 5% chance they are incorrect about the study's outcome.

Now, the researcher canperformthe research. In thisexample, a prospective randomized controlled study is conducted in which the researcher gives some individuals the new medication and others a placebo. The researcherthen evaluates the blood pressure of both groups after a specified time and performs a statistical analysis of the results to obtain aPvalue (probability value). Several different tests can be performed depending on the type of variable being studied and the number of subjects. The exact test is outside thescope of this review, but the output would be aPvalue. Using the correct statistical analysis tool when calculating thePvalue is imperative. If the researchersuse the wrong test, thePvalue will not be accurate, and this result can mislead the researcher. APvalue is a probability under a specified statistical model that a statistical summary of the data (eg, the sample mean difference between2 compared groups) would be equal to or more extreme than its observed value.

In this example, the researcher hypothetically found blood pressure tendedtodecrease after taking the new medication, with an average decrease of 15 mm Hg in the group taking the new medication. The researcher thenused the help of theirstatisticianto perform the correct analysis and arrived at aPvalue of 0.02for adecrease in blood pressure inthose taking the new medication versus those not taking the new medication.This researcher now has the3 required pieces of information to look at statistical significance: the null hypothesis, the significance level, and thePvalue.

The researcher can finally assess the statistical significance of the new medication. A study result is statistically significant if thePvalue of the data analysis is less than the prespecified alpha (significance level). Inthis example, the P value is 0.02, which is less than the prespecified alpha of 0.05, so the researcher rejects the null hypothesis, which has been determined within the predetermined confidence level to be disproven, and accepts the hypothesis, thus concluding there is statistical significance for the finding that the new medication lowers blood pressure.

What does this mean? The P value is not the probability of the null hypothesis itself. It is the probability that, if the study were repeated an infinite number of times, one would expect the findings to be as, or more extreme, than the one calculated inthis test. Therefore, thePvalue of 0.02 would signify that 2% of the infinite tests would find a result at least as extreme as the one in this study. Given thatthe null hypothesis states thatthereis no significant change in blood pressure if the patient is or is not taking the new medication, we can assume that this statement is false, as 98% of the infinite studies would find that there was indeed a reduction in blood pressure. However, as thePvalue implies, there is a chance that this is false, and there truly is no effect of the medication on the blood pressure. However, as the researcher prespecifiedan acceptable confidence level with an alpha of 0.05, and thePvalue is 0.02, less than the acceptable alpha of 0.05, the researcher rejects the null hypothesis. By rejecting the null hypothesis, theresearcher accepts the alternative hypothesis. The researcher rejects the idea that there is no difference in systolic blood pressure withthe new medication and accepts a difference of at least 10 mm Hg in systolic blood pressure when taking the newmedication.

If the researcher had prespecified an alpha of 0.01, implying they wanted to be 99% sure the new medication lowered the blood pressure by at least 10 mm Hg, thePvalue of 0.02 would be more significant than the prespecified alpha of 0.01. The researcher would conclude the study did not reach statistical significance as thePvalue is equal to or greater than the prespecified alpha. The research would then not be able to reject the null hypothesis.

Function

A study is statistically significant if thePvalue is less than the pre-specified alpha. Stated succinctly:

APvalue less than a predetermined alpha is considered a statisticallysignificant result
APvalue greater than or equal to alpha is not a statistically significant result.

Issues of Concern

Afew issues of concern when looking at statistical significance are evident. These issues include choosing the alpha, statistical analysis method, and clinical significance.

Many current research articles specify an alpha of 0.05 for their significancelevel. Itcannotbe statedstrongly enough that there is nothing special, mathematical, or certain about picking an alpha of 0.05. Historically, the originators concluded that for many applications, analpha of 0.05, or a one in 20chanceof being incorrect, was good enough. The researcher must consider what the confidence level should genuinely be for the research question being asked. A smaller alpha, say 0.01, may be more appropriate.

When creating a study, the alpha, or confidence level, should be specified before any intervention or collection of data. It is easy for a researcher to "see what the data shows" and then pick an alpha to give a statistically significant result. Such approaches compromise the data and resultsasthe researcher is more likely to be lax on confidence level selection to obtain a result that looks statistically significant.

Clinical Significance

A key distinction betweenstatistical significanceand clinical significance is evident. Statistical significance determines if thereismathematical significance to the analysis of the results. Clinical significance meansthe differenceisvital to the patient and the clinician. This study's statistical significance would be presentas thePvalue was less than the prespecified alpha. The clinicalsignificancewould be the 10 mmHg drop in systolic blood pressure.[6]

Two studies can have a similar statistical significance but vastly differin clinical significance. In a hypothetical example of2 new chemotherapy agents for treating cancer, Drug A increased survival by at least10 years with aPvalue of 0.01 and an alpha for the study of 0.05. Thus, this study has statistical significance (Pvalue less than alpha) andclinical significance (increased survival by10 years). A second chemotherapy agent, Drug B, increases survival by at least 10 minutes with aPvalue of 0.01 and alpha for the study of 0.05. The study for Drug B also found statistical significance (Pvalue less than alpha) but no clinical significance (a 10-minute increase in life expectancy is not clinically significant). In a separate study, those taking Drug A livedan average of 8 years after starting the medication versus living for only2 more years for those not taking Drug A, with aPvalue of 0.08 and alpha for this second study of Drug A of 0.05. In this second study of Drug A, there is no statistical significance (Pvalue greater than or equal to alpha).

Enhancing Healthcare Team Outcomes

Each healthcare team member needs a basic understanding of statistical significance. All members of the care continuum, including nurses, physicians, advanced practitioners, social workers, and pharmacists,peruse copious literature andconsiderconclusions based on statistical significance. Suppose team members do not have a cohesive and harmonious understanding of the statistical significance and its implications for research studies and findings. In that case, various members may draw opposing conclusions from the same research.

References

1.: Hayat MJ. Understanding statistical significance. Nurs Res. 2010 May-Jun;59(3):219-23. [PubMed: 20445438]
2.: Mondal H, Mondal S. Statistical Significance is Prerequisite in Study. J Clin Diagn Res. 2017 Sep;11(9):CL01. [PMC free article: PMC5713722] [PubMed: 29207700]
3.: Heston TF, King JM. Predictive power of statistical significance. World J Methodol. 2017 Dec 26;7(4):112-116. [PMC free article: PMC5746664] [PubMed: 29354483]
4.: Nakagawa S, Cuthill IC. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev Camb Philos Soc. 2007 Nov;82(4):591-605. [PubMed: 17944619]
5.: Haig BD. Tests of Statistical Significance Made Sound. Educ Psychol Meas. 2017 Jun;77(3):489-506. [PMC free article: PMC5965554] [PubMed: 29795925]
6.: Jiménez-Paneque R. The questioned p value: clinical, practical and statistical significance. Medwave. 2016 Sep 09;16(8):e6534. [PubMed: 27636600]
7.: Mariani AW, Pêgo-Fernandes PM. Statistical significance and clinical significance. Sao Paulo Med J. 2014;132(2):71-2. [PubMed: 24714985]