Einstein once said that insanity is doing the same thing over and over and expecting a different result.
This is a mindset that most companies are now incorporating as a part of their business ethos. Trying new ways to achieve business objectives helps businesses stay at a competitive advantage and there is no dispute that companies which are not adopting this approach are likely to be left back in the game.
So while experimentation is being recognised as the answer – it is the approach to experimentation that is ultimately making this dynamic transformation feasible and sustainable in the long run. As is the case with all new things – they must be measured if they hope to stick.
Introducing Cost Per Experiment or CPE.
Admittedly, at first glance, this can seem like a strange metric. But knowing how this affects your decisions helps clear up a lot of the fog around it.
Let’s start by defining cost per experiment.
CPE can be defined as the net cost of developing, executing and measuring an experiment (usually as hourly resource cost, cost of tools and acquiring and storing data) minus net variation performance (positive or negative lift) of your key metric (e.g e-commerce conversion rate).
This measurement is best observed per experiment – but in the long term it is best to divide the implicit cost of each experiment by the number of experiments conducted to get an average CPE.
Total Cost Of Experimentation
{(Cost of resource) + (Cost of tools and associated data costs) – (Revenue from Uplift achieved (+/-)}
Revenue from Uplift can be positive or negative, depending on your results. So a reduction in conversion rate would add to the cost of the experiment, while an increase in the conversion rate would lower the cost of experimentation.
Average cost of experiment (ACPE) = TCE/N where N is the number of experiments conducted.
This metric is a high level reliable indicator of a company’s approach to growth. It provides an answer to the question – “Does the company innovate?” and “Is the innovation at that point, where it is making money for the company?”
Knowing the CPE alone is unfortunately not enough. It must be accompanied with the knowledge of 2 other sets of data.
a. The revenue generated as a result of experimentation.
b. The existing proficiencies being used for purposes of experimentation.
Here is why each of these two datasets matter:
The revenue generated from experimentation is a measure of how much additional profits a company makes in light of a known CPE. This helps show the value and advantage of experimentation, propagating the culture for growth and in most cases is accompanied by a monetary uplift.
The proficiencies required for experimentation are indicators of a company’s ability to grow based on measurable insights. If your experiments are being performed on hunches and ‘human intuition’ they are most likely going to cost you in the long run (sometimes even in the short run).
So then what are the proficience that you need to perform experimentation in an efficient way?
While there isn’t a one size fits all seeing as how every company has a varied set of proficiencies, the list mentioned below sketches a good overview of the key requirements.
They are as follows:
a. Understanding of competitive environment and consumer behaviour: Truly understanding the market requires deep research and understanding of consumer behaviour. This case study showcases a great example of how a large scale company can properly utilise tech and proficiency to gain a better understanding of their consumer base.
b. Costing and how it affects consumer decision making: Costing has a monumental effect on how consumers perceive and ultimately purchase products – running experiments along these lines allows for you to gain insights into what consumers think of your product/services – and the chance to introduce newer products along every vertical.
c. Tech Integration and Compatibility: With experimentation, comes the need to implement tools and software and often – the experiments that you hope to run, require them. Do you have the capability to implement this? And if something goes wrong, can you put out the fire?
d. Data Management and Documentation: Collecting, storing, accessing and disseminating large volumes of data and insights is now a basic need of experimentation. A company must understand and account for these needs in cost and time.
e. Hypothesis and Research: Left to their own devices, a company would test every single idea – this would greatly increase the cost of experimentation. However, utilising scientific methodologies via hypothesis and consumer behaviour studies can help magnify your issue areas and prioritise your experimentation efforts accordingly. Not only does this reduce cost, it also derives the most effect, the fastest.
Without the above proficiencies, your machine is missing crucial parts, making it hard for it operate smoothly and without the need for constant repairs.
As a result, experiments being conducted on the basis of cutting edge methodologies and closely aligned resource proficiency indicate an accurate CPE which is essential to understanding your true position.
Once you have a complete understanding of your exact revenue and proficiencies – you can create targets on the basis of the profit maximisation theory.
The demand and supply of experimentations within a company is something that is of vital importance – let’s assume that your company is willing to spend about $2,000 on each experiment, on average (on setup, resource etc..) and on over a period of time, if the successful variant from the experiment is making your company $3,000, you are experimenting profitably. It is important to note that the same experiment may have cost you money on a particular variant – usually chalked up to unavoidable opportunity cost. But since, the measure of the cost and result are figures calculated after averaging the total cost and result over a period of time (e.g a quarter) – you obtain a ratio. Let this be your base ratio.
Now let’s observe the possible scenarios:
- You start spending more but the result remains the same – this indicates that your proficiency has been altered.
- You gain more profit but at the same cost – while this scenario is rare, this is an indicator that there is more scope for experiments (bring up the cost, and reap the benefits of more results)
In both cases, you attempt to maintain your base ratio. Since your base ratio was decided on what your company is comfortable spending in terms of experimentation, maintaining this ratio is not only profitable but also an indicator of whether you are ready to scale efforts.
As long as your Revenue per experiment is higher than your cost per experiment – your company is experimenting profitably.
In order to conduct experiments in a profitable way, you need the following :
a. Culture of experimentation : Generally cultivating the mindset of growth within the organisation and not accepting the status quo. A company can work together to create better experiences for users, and the first step is to accept that it can be made better through experimentation.
b. Resources with specialisations : In order to pull off experimentation at scale, retraining employees for the process can be expensive and often counterproductive. It is generally more advantageous to engage with resources that have a specific skill set that can streamline the process of efficient testing.
c. Constant rate of experiments : Being intermittent with testing can throw off the final readings, making high level decisions very hard to make.
Typically, sporadic experimentation raises the cost of experiments over a set period, as the number of resources (fixed cost) remains the same. Additionally, since most experiments require a duration of time to attest a seal of significance – you are essentially performing less experiments overall ultimately getting a lower bang for your buck and gleaning less insights as well.
Maintaining a steady state of experimentation and analysis helps create an exponential system – whereas stopping the momentum often brings it back to square one, where you are playing catch-up with the competitors in your field.
Understanding how CPE as a metric can be used by management as a solid indicator of whether a company is exploring every logically sound way of increasing revenue will be a new and important focus for 2019. While a lot of companies understand the need to grow in this direction – it is often execution where they find roadblocks.
Starting with an audit and objective assessment of your capabilities and standing is a great way to create a robust roadmap to higher revenue. Start now, and test hard!