Calculating Statistical Significance in Web Analytics
by Carlin Leung
Originally published on the Quietly Blog
What is statistical significance and why does it matter for your site? If you’re wondering how to measure the success of your company’s campaigns and overall marketing efforts, read on.
Analyzing changing traffic and engagement patterns on your site is a necessity. But how do you determine which changes are significant and which are due to external, unsolicited factors? How do you understand if your campaign efforts are directly driving users to your site and increasing traffic, or if certain campaigns are performing better than others? Measuring the statistical significance of the various metrics that affect your site can tell you a lot about the success of your marketing campaigns and can help you to make strategic changes that will benefit your company.
What Is Statistical Significance?
In hypothesis testing, the result of an experiment is statistically significant when there’s less than a 5% chance that more extreme results will occur when the same test is conducted in the future. In application, this means that this result didn’t just occur by chance, and that it’s likely caused by some factor. 5% is the universal standard employed by scientists, and is widely viewed as an acceptable risk.
Here’s an example from a web analytics context:
Let’s say that in the month of July, 1,000 users visited your site, an increase compared to June’s 900 users. How do we know whether this increase is actually a “significant” change worth looking into?
If we compare this month’s data with historical data, we can calculate the probability that July would have earned 1,000 or more users organically. If the result is less than 5%, we can say that this change is statistically significant and can’t be explained by random chance, since the probability of seeing this number occur organically is so low.
So, how does statistical significance apply to your Google Analytics tracking?
Well, chances are, your web traffic (or metrics like bounce rate, average time on page, etc.) doesn’t progress in a beautiful, smooth, upwards-trending line. This data fluctuates by day, by month, and by year.
For example, maybe your pageviews are increasing slowly over the year, but there are a few months that performed better or worse than expected. Should you be concerned about the down months, or celebrate and analyze the up months? Are particular spikes or dips meaningful changes (actual outliers)? Or, do they reflect the natural ups and downs of running a website?
Statistical significance can help determine whether each variance is something out of the ordinary, and therefore worth investigating.
Why Does Statistical Significance Matter?
In short, measuring the statistical significance of metrics on your site helps you better quantify whether your efforts are actually working and know when a mysterious change is worth evaluating.
Here are some ways that calculating statistical significance can benefit your site and overall operations:
- You can ensure that you don’t take action on information that’s likely due to chance, or just part of your site’s natural rhythms.
- You can understand whether your recent campaigns were actually impactful.
- For example, if you recently implemented a Facebook campaign, and found that the campaign month received a significantly higher number of visitors from Facebook, you can conclude that your campaign was effective.
- You can investigate why spikes or downturns in various metrics occur, even if you haven’t made any changes or started a new campaign.
How Do I Find Statistical Significance for Web Analytics Metrics?
There are a number of ways that you can understand the statistical significance for metrics on your site. One option is using a tool that compiles and calculates this information for you. For example, Quietly Insights, can provide you with statistical significance information and inform you whether something is worth actioning on.
You can also use Excel to measure statistical significance.
Here are step-by-step instructions for how to organize and analyze this information in Excel:
1. Select the metric you’d like to analyze, and export a year’s worth of monthly data from your web analytics provider (most likely Google Analytics).
- You can also analyze by weekly data (in this case we recommend exporting six months instead of a year).
2. Open the export in excel.
3. Create a new tab, and add the following columns: month (or week, depending on the data you downloaded), [metric name], lower limit, average, upper limit, and standard deviation.
4. Copy the data into the [metric name] column, and fill out the corresponding months (see below).
5. In column D, calculate the average for the dataset, and copy the formula down. Don’t forget to use absolute references (the $ signs in excel formulas) so that the value is consistent for each month.
6. In cell F2 (under ‘Standard Deviation’), use the formula =STDEV.S() to calculate standard deviation for the dataset. This number tells you how different your data points are from each other, and will be used to calculate the lower and upper limits.
7. Next, we’re going to calculate the lower limit of the dataset. Essentially, if any of your data points are below this number, it represents a significant dip.
- The formula is the average-(2*standard deviation)—notice the $ in the when referencing F2, the standard deviation!
8. The next step is to repeat the above actions for the upper limit. Any data points above this value can be considered significant.
- Formula is average+(2*standard deviation). Again, notice the absolute reference ($ sign) for standard deviation. The only difference between this formula and the last is that we’re adding, rather than subtracting the standard deviations.
9. Lastly, let’s put this into a line graph to make it more visual.
- Go to insert>line chart to create a line chart.
As you can see, the only point that’s worth investigating is November, as the number of pageviews exceed the upper limit. While originally, you might have thought that August was worrisome, it’s clear that the dip is just part of the natural fluctuation
What Do I Do if Something’s Significant?
You’ve analyzed the data, and suspect that a change in your site is a result of an action you’ve recently taken. If you’ve started a new campaign, and see a change, you can likely attribute any changes to this action.
- If you started a Facebook campaign and saw a significant spike in Facebook traffic, you can conclude that the campaign made a significant impact on your traffic, and that it was effective in increasing traffic.
- If you started a Facebook campaign and it didn’t increase traffic from Facebook, but the overall site traffic increased, it’s difficult to say that the Facebook campaign was particularly effective. In this case, it’s likely that your traffic would have fluctuated even if the campaign didn’t occur.
If you are unclear as to why something’s spiked or dipped, and haven’t conducted any recent campaigns, it’s likely due to an external factor, such as:
- Seasonality: does this spike happen every year? Or just this year?
- PR/News: was your company mentioned in a news article, or featured on a site?
- Current events: did something happen that resulted in increased search or social traffic to your site?
The Bottom Line
There are many factors that can cause metrics to move up or down, and they are not always worthy of a reaction or change. Use statistical significance to know when you should take action, or when you should leave your site as is. If you’d like some help calculating statistical significance, you can always try a tool like Quietly Insights, which will take statistical significance into account and strategically inform your decisions.
Carlin Leung © 2020