Presented by Optimizely
Experimentation is vital in turning out flawless products — but it can go terribly wrong in a single iteration. Learn how to choose the right metrics and avoid wasting time, opportunities, and your project testing budget when you join this VB Live event!
Access on demand for free right here.
In product development, teams often feel like they’re wandering around in the dark, trying to figure out what users really want and whether their product is going to deliver for them, says Jon Noronha, director of product management at Optimizely.
“What many product teams often feel is this real lack of confidence, because we don’t know, until the very end of this process, how our users will react to the things we build,” Noronha says. “We can sink months or even years into something that might have a negative impact.”
Take, for example, his first-hand experience as a product manager at Microsoft in the wake of the disastrous Windows Vista waterfall launch. Microsoft spent years and years developing — mostly in secret, mostly internally, without a lot of external feedback. Once it got into users’ hands and problems emerged, a PR nightmare ensued and the company’s stock dropped drastically.
It was a wake-up call for the company. It realized that this waterfall approach to development, where everything is done in sequential stages and feedback only comes at the end, just couldn’t work in this modern software development era. Especially because other companies in Silicon Valley were pioneering a new way of working, one that was a lot more agile, a lot more gradual and iterative. That was exemplified most of all by Google, which was deploying code every single day and A/B testing everything.
That’s why so many product development teams have started to change how they operate, moving away from a linear process of design, build, launch, and pray, to a more gradual, iterative process where they’re constantly experimenting on ideas in small pieces and tweaking them as they go based on the feedback that they get.
For every single thing you do in product development, how can you actually quantify the effect that it has? How can you validate if a feature does what you said it would? How can you see what kind of adoption it gets? Does it change some core business metrics for you, like retention and engagement of your audience? And then using that information to feed back into the beginning of the cycle, and develop new ideas or refine older ones.
“When you start to embrace this way of doing product development, it changes your mindset,” Noronha says. “We don’t just have a product road map and a couple of experiment ideas on the side. Soon we start to think of every idea we have for improving our products, whether we’re a designer or an engineer or a product manager. It just becomes another hypothesis. They’re all on equal footing, ready to be tested.”
Now, this describes a kind of ideal, this end state of a culture of experimentation. But the truth is that for most companies it’s quite a journey to get there, to climb this mountain of experimentation culture, and go step by step from that launch-pray mindset to the more cyclical experimentation mindset. It is not an easy thing to do, Noronha warns. It requires some real culture change at a lot of companies, in how you work.
There’s no silver bullet, he says, pointing to his work on Microsoft’s Bing product. In 2011, Bing was managing to lose almost $500,000 per hour, which reflects how hard it is to build something as complex and big as a search engine. Thousands of people, thousands of machines and servers were all working toward this goal, but until that search engine was really good and high quality and worked for users, it wasn’t getting to benefit.
By 2015, however, Bing had actually quietly become quite profitable, to the tune of over a billion dollars a quarter — a pretty drastic turnaround.
“There was no magic silver bullet in that process, or some single thing we changed that suddenly made the company profitable or grew our market share,” he explains. “Instead, it was a gradual process of continuous experimentation, where we were constantly trying out new ideas and trying to build a culture that allowed hundreds of teams to all make those incremental improvements and measure the impact of every change they made.”
The usage of A/B testing and these techniques was directly responsible for tripling Bing’s market share in the U.S. and the rest of the world. Experimentation steadily increased, from just a handful of experiments running every week to hundreds of experiments running weekly on all different parts of the service. The user experience, the search algorithm, the advertising, the mobile apps, everything was grounds for experimentation, continuously. It meant that every product team had multiple experiments in play at any given time.
“It really showed me the power that A/B testing could have, but what it also showed me was some of the challenges,” he says. “We had big advances, but also big setbacks and big challenges in embracing this mindset of experimentation, especially in a company that didn’t always have a mindset that was suited to that constant experimentation.”
To learn how companies succeed and where they get stuck when they implement a culture of experimentation, the most common pitfalls and challenges a company can face, and how to accelerate your own progress through this journey, access this free VB Live event on demand now!
Access on demand for free right here.
You’ll learn about:
- The most common mistakes product teams make when running experiments
- Which metrics correlate best with your business’s success
- Strategies to scale experimentation across multiple teams and squads
- How the world’s top technology companies are able to experiment on all product decisions
Speakers:
- Jon Noronha, Director of Product Management at Optimizely
){kind=link}
){kind=link}