What is Cohort Analysis?
Most of the time when a CRO professional speaks of testing content, they mean A/B testing. However, this is not always the case. Sometimes there are situations where cross-sectional study like A/B testing is not the right approach to analysis. In cases where time is a factor in the comparison then a longitudinal study is best.
The answer to the question, “What is cohort analysis“, is that cohort analysis is one such alternative to AB testing, a longitudinal test that can be useful to find relationships between user groups by tracking changes to a group over time.
This is highly relevant for your CRO and SEO activities when testing for the effectiveness of changes made to copy or some other element.
In many cases data displayed in analytics tools is lumped together, making it difficult to determine what caused the changes from one period to another.
In order to understand your users through your data you need to segment that data and then compare it to the same segment for previous campaigns. For example, an Easter special offer promotion, if particularly successful for some unknown reason, by comparing the results of a previous year it can help you understand the differences and potentially identify the cause of the success.
Ultimately the idea is to replicate the success and improve upon it. Using this Easter campaign scenario as an example, the following points could be examined with a cohort analysis.
Consumer behavior differences
- Conversion funnel differences year to year, consider previously encountered micro-conversions, message style, etc.
- Pricing variations year to year. Price analysis is always good to regularly test.
- How does
- Message / offer visibility and availability and interaction. Did it effect your average order value?
- Advertising type, location, exposure, and expenditure.
- Consider all touchpoints, cross channel exposure/campaigns vs previous year.
- For conversions, its useful to see how they vary according to traffic source vs a previous year
- Bounce rate and time on the page, provide a fantastic opportunity for improvement
- Cohort analysis is available for free in Google Analytics.
Feel free to suggest other uses of cohort analysis in the comments below.
How to Identify New Personas
Another great way to use cohort analysis is to use survey data to identify potential user groups and then define a persona. Cohort analysis of different groups can tell you the best way to target those groups with OptiMonk’s triggered instant messages. Cohort analysis, therefore, helps with the process of creating customer journey maps and conversion funnels for each persona.
Identify data that can be used as a comparison point for every stage of your customers journey along the conversion funnel. Typically micro-conversions tend to be a good starting point. What differentiates one persona from another is the point of interest, identified from the content the user engages with.
How to Collect Survey data effectively
One great feature of OptiMonk you can leverage using Cohort analysis is the multiple-choice feedback collection messages and the NPS nanobar. Both enable consistent data collection which can provide great insights into your website’s different visitor groups, customers or even repeat visitors. Of course the questions you ask are the critical aspect here (checkout my blog post on the topic).
Don’t forget you can also use custom fields and place them on your survey messages.
For example, to understand what industries are most interested in your products you could create a custom field for industry or business type. Once done, you have powerful data on how to improve your business, but also have a way to measure change over time. An extremely valuable tool to measure change as your product or services change over time.
Pitfalls to consider
Things change, which in part is why we are doing cohort analysis. However in order to gain a clean set of data always allow for a sufficient time period to measure.
It doesn’t need to be years, however for shorter time periods you need to consider other aspects that would vary month or season to season, for example the weather, holiday periods and business time tables of events. How does this data effect your results?
Although cross-sectional studies of data sets are extremely useful (such as A/B testing), giving you information about what needs refining, alone the results never provide the full picture.
All too often I see customers relying on A/B testing, and then making the wrong changes based on incorrect assumptions of cause. To gain a greater understanding of how to improve your offering, further, inspection is required, one great way to do this is with longitudinal studies such as cohort analysis.
I would be interested in hearing about your use cases and the insights you gain from making such investigations via the comments section below.