Statistical Significance in Decision-making

Rating

Statistical Significance in Decision-making

🔥 Companies are increasingly using business analytics and are increasingly relying on data to make operational, tactical and even strategic business decisions. Obviously, when making decisions, managers should not rely on calculations and models they don't understand. That's why understanding statistical significance is becoming more important for managers.
First of all, you should realize that in statistics, "significant" does NOT mean something is big or important! Rather it means that the result we see in some sample also exists in the (total) population.

What is statistical significance?
Statistical significance is the likelihood that a relationship between two or more variables we find in our results is NOT caused by random chance. If you draw conclusions out of a lot of data, you have to use a sample. A sample is a set of individuals / objects collected or selected from a bigger statistical population by some defined procedure. But in any experiment or observation that involves drawing a sample from a larger total population, there is always the possibility that an observed effect occurred due to sampling error (coincidence). Statistical significance is a way of showing to what extent that the answers which were obtained from the data are not totally due to luck. It is the confidence level you may have in a mathematical way that some result or conclusion is reliable. The more critical your decision based on the data is going to be for your firm, the higher you will want the statistical significance to be.

Statistical hypothesis testing
Statistical hypothesis testing is the method by which an analyst determines that the results in the data are not explainable by chance alone. This test provides a p-value, which is the probability of observing results as extreme as those in the data, assuming the results are truly due to chance alone. A p-value of 5% or lower is normally considered to be statistically significant.
But when the p-value is large, then the results in the data are explainable by chance alone. So when the p-value is higher than 5% (0,05) we consider the results statistically NOT significant as they could easily be explained by chance alone.

Mistakes and errors made while working with statistical significance

• SAMPLING ERRORS (too small or non-representative). Statistical significance tells us that if something works for a smaller sample, then with how much confidence can we say that it will also work the same for a large population. But many times people make mistake of taking a sample which does not represent the total population. For example the objects in the sample might be biased towards some specific color, say red, while the total population might be a variety of colors. The statistical significance becomes low when such errors are made. Choosing a good diversified sample which represents the total population accurately is very important for getting a high statistical significance.
• NON-SAMPLING ERRORS. There are other errors that may occur that are as important as the sampling errors. Non-sampling errors include the ones where the measurement and experiment protocols were broken. Sometimes the data may get lost. Sometimes the people in the sample may have lied in the survey. Etcetera. These factors often are difficult to be controlled, but a careful analysis will help to reduce these non-sampling errors.

Applications of statistical significance
Statistical significance basically tells us whether we should trust the results of some experiment or test like for example an A/B test. Before we send a special offer to 10 million customers, we cold first send two variants to 50 thousand or 500 thousand of them to see what percentage subscribes to it. Or we might test landing page conversions, website calls to action, customer reactions to product launches, etc.
If we need to be sure about a small difference between these two variants, we would need the large sample to be sure what we find is statistically significant. And if we take only a small example, but the effect we find is very big, than these results are still statistically relevant.
Sources:
Amy Gallo, "A Refresher on Statistical Significance", Harvard Business Review, 2016, February 16.
"Statistical Significance: What is it, how to calculate it, and when to use it", Mixpanel

		Comments by date▼ Comments by rating Rating
		1 Bob R Manager, United States
	Common Mistake in not Using Statistical Significance: Type 1 Errors Good article and much needed in business decision making. The most common mistake I see is people assign significance to a random change (a "Type 1 error"). An example would be a slight change in a trend chart like in sales or scrap. Sales goes down a few ticks and there's brow beating or panic from the boss - up a few ticks and it's party time. In fact there is no identifiable cause - totally random events.

		1 Anonymous
	What is a Type 1 Error? @Bob R: Agreed. A Type 1 error (or type I error) is a statistics term used to refer to a type of error that is made in testing when a conclusive winner is declared although the test is actually inconclusive. Scientifically speaking, a type 1 error is referred to as the rejection of a true null hypothesis, as a null hypothesis is defined as the hypothesis that there is no significant difference between specified populations, any observed difference being due to sampling or experimental error. In other words, a type 1 error is like a "false positive," an incorrect belief that a variation in a test has made a statistically significant difference. You see, there is always the risk in business analytics / decision making that some effect you see is caused by sheer coincidence / randomness / luck / bad luck. The aim of using Statistical Significance is to at least know how big this chance is, and use that knowledge to reduce this chance to a level that is matching the importance of the decision. But no matter how high you make the statistical significance, the possibility that the result you see are the consequence of sheer coincidence can never be ruled out 100%.

		0 Gandhi Heryanto Management Consultant, Indonesia
	Practical versus Statistical Significance in Decision Making In decision making we must also look at the practical significance, which refers to the empirical impact that such an event has in real life. Of course the threshold for determining practical signific...