Great question. So we just had a question a couple of days ago, where someone was trying to understand their lead source - whether someone comes from social, search, or paid - are they more likely to convert from lead to sale. So later on in the funnel. So somebody comes into the website, from search, two months later are you more likely to convert to a sale?
So you’re bridging a couple of sources here. Because of our structure, our core technology that we have enables us to deliver very rich analyses that we can share amongst all our companies. So Narrator.ai, we can share analyses outside of a data modeling solution. So because you can share analyses, we can spend a lot of time designing these to be very thorough and detailed, because we’re solving it ONCE for everyone.
So when we deliver that analyses for that person, in the case we’re talking about, we do a LOT of stuff. We clean up and figure out consistent times, to make sure time variants and those various behaviors don’t interrupt the data. If you’re trying to figure out if something converts better or worse, if some months you had a really good conversion rate, while others you have a bad conversion rate, we have to remove those errors and variances from our data so that we can make better decisions if it matters or not.
Then we have to check not only if it’s statistically significant that search is better than social. And in this case, search did convert significantly better than social. An average analyst would stop there. But Narrator.ai is able to go deeper.
So now not only is it statistically significant, but has it ALWAYS been better? Is it possible at one point it spiked, or over time has search always been significantly better than social?
So then in this customer’s case, it was. 93% of the time search was better than social. A senior analyst would stop there. But we go even further.
So now when you get to this level, it could be the case of something we call “correlation vs. causation:” Is it something about the search and social, or is it that there’s something else that happens to be driving this impact? So we dive even deeper, because we want to give you an actionable recommendation.
So the actionable recommendation, if we stopped there, would say “Take people from social, and move them to search. Spend more on search.” But we’re going to check now in the past when we shifted the people to come from social or search, was there a causal effect for the conversion rate to move in that direction as well?
And it turns out that there WASN’T. So in all of the history of this company, every time you shifted people to social, it made no impact on the overall conversion rate. So this whole thing was a correlation. So the action was “don’t do anything.” This feature doesn’t affect this conversion rate at all. It happened to overly it, but it wasn’t a driver, so changing it didn’t matter.
In the case of this client, we recommended that they NOT shift their distribution. There’s something else driving the impact. Don’t start shifting your money, because it will have no impact. Instead, we’re developing a new hypothesis. And the fact we can iterate through hypothesis and analyses so quickly - that would have taken a senior person a LONG time to run. And our system is smart enough to generate it with the reasoning.