Statistical significance loses its relevance when “samples” get very large. A government or an education minister is not merely interested in the fact that there is a positive effect of an intervention but also in how large the effect is. However, a Z-value or p-value is not an effect size. This result, then, might be the basis of a policy decision to invest in this type of educational intervention. The statistical significance of the combined effect might now be interpreted as evidence (or “statistical proof”) that this specific intervention “significantly” improves educational outcomes. These studies have been conducted in different countries, in different contexts (e.g., in schools managed by the government and schools managed by non-governmental organisations), and with different types of students (e.g., different genders, different ages, different social backgrounds). For example, suppose that the forest plot in Figure 1 is a plot of the results of studies of the effects of a specific educational reform intervention. Because meta-analysis functions as a more powerful significance test it generates a more useful and more convincing result than a single study. This type of result of a standard practice meta-analysis is considered very valuable because it clearly is a solution to the problem of “contradictory” evidence or, more often, to the problem of “insignificant results”. The combined effect might be significantly different from zero (or not). In standard practice, meta-analysis is aimed at “solving” the problem that results of studies differ and apparently contradict each other, by generating a “ combined” effect. And there have also been some studies with effects that are statistically non-significant. Other studies have shown statistically significant negative effects. Some studies have shown statistically significant positive effects. The forest plot in Figure 1 shows that research results have been “contradictory” or “ambiguous”.

Zero in confidence interval meaning