Cohort analysis is really fun. It’s kind of like finding a treasure map about your app usage. Here’s one of the jobs I run internally, and the script wrote to do it.
I run this analysis to count how many of our accounts (or “Orgs”) have actually tried using our service. It’s one way to measure how effective our site and docs are at helping customers take those first integration steps.
Orgs are grouped into cohorts according to the week they created an account. The green Orgs column is the number of new accounts we added in a given week, going back 18 weeks. (The first cohort is huge because it was the week we invited a large chunk of our beta signup list to create accounts).
The gray columns are the dates when I ran the cohort analysis. The # columns are the number of those organizations who, on that column date, had sent data into Keen AND performed analysis at any point in the past.
The chart tells me a few things.
- A large number of our early orgs haven’t tried our product. That makes sense because we didn’t have a whole lot of offer back in October. We didn’t even have charting at that time. They logged in, looked around, then left. This represents an opportunity for re-engagement.
- Newer cohorts are converting at a much higher rate. That means we have done a good job improving the product and or/first time experience. Of course, there is still room for improvement.
- Based on my experience working directly with customers, and this data, it’s clear that there is a time period between when a company starts thinking about analytics and when they start collecting data. An given company might create an account but not send data until weeks or months later.
- As I start a re-engagement campaign for those untouched orgs, I’ll continue to run this chart and it will show me if my emails are working or not.
In case you’d like to run something like this yourself, I’ve shared a gist of the ruby script I wrote to build the chart above. These are the prerequisites for this analysis:
- We track an event every time a new org is created, including the org ID
- We track an event every time we get an API analysis request, including the org ID of the project being queried
Given those two funnel steps and a number of weeks, the script will create a Keen IO funnel query for each week, run it, then output the results.
The output looks like this:
I’d love to make a little rails app to automatically output the chart, but for now I am manually transferring the data into the excel chart.
Update: I made another script that does retention analysis and outputs the matrix to excel. Check it out here. That code could also be used to build the cohort matrix shown in this blog post.
If you have any questions about how this all works, or how it could be applied to your business, I’m here to help!
– Michelle
