As Anchorman’s Brian Fantana once said: “60% of the time, it works every time." Perhaps he was onto something?
Harvard Business Review shared that at Booking.com, a mere 10% of experiments generate positive results. Let’s flip that – 90% of experiments fail at Booking.com. But does that mean those experiments were unsuccessful? Uninformative? A waste of time?
Failure is your friend
It’s not a concept many of us feel comfortable with, but behind true innovation lies countless failures. To truly optimise and evolve your digital products you have to be comfortable with setbacks, unpredictability and failure.
In the 1970s, before founding Microsoft, Bill Gates owned a big data company called Traf-O-Data. The company’s market changing innovation was a data tracker that was able to read raw data from roadway traffic counters and pass this information on to traffic engineers. However, when the time came to launch, the tracker didn’t work as planned and the project failed miserably.
But not all was lost – the lessons Gates learnt along the way were instrumental in the successes that would follow at Microsoft. A simple test run by an engineer at Microsoft-owned Bing led to uplifts in excess of $100 million and remains one of the most cited digital experimentation stories to this day.
Don't fail fast, fail well
Experimentation is a core principle of design thinking, and is often accompanied by the ‘fail fast’ philosophy. While it’s crucial to rapidly learn from what doesn’t work, it’s dangerous to imply that sitting with failure is an inherently negative thing. Fast reaction shouldn’t always trump considerate action and deeper review.
Toto Wolff, Team Principal and CEO of the Mercedes-AMG Petronas F1 Team, shared at Web Summit how Mercedes have adopted a remote-first mindset for all testing, production and maintenance activities. Despite the breakneck speed of F1 races, Mercedes slowed the pace as they invested time into creating a digital twin for their cars for their network of engineers to deliberate over. This enabled them to carefully consider the impact of an abundance of data available on the performance of their cars, nuanced details into every inch of racetrack and numerous learnings from other industries to shave milliseconds off their lap times.
Likewise, digital experimentation alone cannot substitute the value of direct conversations and validating ideas with customers and users. For example, we worked with Mars to deliver a new petcare app. We created prototypes to test the new app with a group of customers to gain more valuable, qualitative feedback as to what was most important to pet parents than we would have done from a simple A/B test alone.
In certain occasions, it may be appropriate to override certain ‘winning’ tests in favour of stakeholder or business needs. Whether it’s a conscious or ethical decision or not, it’s no secret that some businesses may make the process of cancelling a subscription or claiming compensation difficult in the hope that customers will reconsider. Whether it’s a successful tactic or not is certainly up for debate.
Instead, consider Dominic Price’s concept of “failing well” and how businesses with a growth mindset apply a combination of data alongside accumulated experience and intuition to keep trying new things. One of the key steps is to study what didn’t work and why, and then share these insights with peers and colleagues.
‘Failing well’ was also one of the Mercedes F1 team’s tactics throughout the final races of 2022. Following a season of poor performance, Wolff shared that the team’s focus would pivot to experimentation – using the final few races of the season to test new ideas before fully analysing what went wrong in advance of the 2023 season.
Innovation starts with a shift in culture
In order to realise the full benefits of experimentation, companies need to undergo a shift in culture that encourages curiosity and an open forum to try (and fail at) new, untested ideas. After all, if you run an experiment already knowing what the outcome will be, then what’s the point in conducting the experiment in the first place? Rebecca Parsons, CTO at Thoughtworks, shared at Web Summit one of the most effective ways to do this is by “building an inclusive culture… You can’t experiment in a culture of fear”
Leading UK broadcaster Sky, are actively looking to lower the success rate of their A/B tests. Simon Elsworth, former Senior Experimentation Manager, shared in an interview with Contentsquare that, “[Unsuccessful tests] means we’re testing bigger, scarier things. It means we’re testing things that we don’t know the answer to. If everything we do is winning, then we’re being too safe. We’re not challenging anything.”
So if we build an inclusive culture that incites curiosity and places less emphasis on ‘getting things right’ the first time, we can then reframe experimentation to be an everyday, business-as-usual activity that fosters innovation and supports business growth.
Failing well among other topics was discussed at our recent trip to WebSummit. Follow our blog for further insights from the summit and for our take on the top trends to keep an eye on throughout 2023.
