Have you ever wondered why your friend got a new feature in their Monzo app before you? Maybe you were setting up targets in Trends before your housemates could even see the option. Perhaps you’ve noticed that your Monzo home screen looks totally different to your neighbour’s. Or you might have the eagle eyes to spot that the way we describe Flex is ever so slightly different in your partner’s app.
If that rings a bell, you’ve probably spotted a phased rollout of a new feature!
As we’ve expanded the things you can do with Monzo, we’ve also developed different ways to measure that what we build is actually meeting your needs. In this post, I’ll give you an overview of the different approaches we take to launching new products and features. We’ll look at when we might choose an experiment instead of Monzo Labs, and vice versa. And I’ll share some examples of when it’s worked well, along with a time when we got it wrong.
What are we trying to learn?
We’re here to make money work for everyone, so we always start with our customers. Before we get to work building a new product or feature, we make sure we understand the customer needs we’re trying to solve. We identify these through research: usually detailed interviews with both Monzo users and people who don’t use Monzo.
For example, we recently launched a feature to allow customers to see their mortgage in the Monzo app. Before we started work, we built an understanding of all the parts of buying and owning a home that people had trouble dealing with today. From this research, we picked a key need that was poorly served today, and that we thought Monzo could help with - visibility of current mortgage balance - and started working on possible solutions.
After research, it’s time to start outlining our hypotheses - the things we believe to be true, but haven’t yet proved. We don’t know for sure that the solutions we’ve come up with will solve our key customer problem. We also might not know whether our ideas are technically feasible, or whether our customers can work out how to use the app journey we’ve designed, or whether we can build this product in a way that is commercially viable for Monzo.
We might be able to test some of these hypotheses without building and launching a whole product. For example, we’d prefer to take a few Monzo customers through some prototypes to test whether they can find their way around them, than to wait weeks or months until we’ve released it to find this out.
Out of the remaining hypotheses, some of them will be more critical to the success of the product than others, and we’ll have more certainty over some than others. At the end of this process, we should have a clear idea of which two or three hypotheses are most important for us to learn about at launch. Then we can work out how to roll out the product to best test these.
To experiment or not to experiment?
We’ve blogged about experiments before. They’re one of several tools we can use to test our hypotheses. Here are some of the other options in our toolkit:
Staff testing
We can, and often do, release our products to Monzonauts early in their development lifecycle. They’re some of the most vocal critics of our products!
This is a great way to test that everything works technically, and find any lingering usability problems. But it’s not necessarily representative of what non-staff customers will think, so we need to consider the feedback in this context.
Monzo Labs
Basically a Monzo beta programme, Monzo Labs allows us to get real customer feedback from some of our most engaged users - especially our wonderful Community members. It’s great for understanding edge cases we might not have thought about.
But since anyone can opt in, features released to Labs need to be ready for potential broader public scrutiny. It’s a fine line between unpolished and incomplete. Plus, the needs of our power users might differ a little from the more than 7 million people across the UK who use Monzo - so we need to bear that in mind.
Experiments
Also known as A/B testing, we use experiments to randomly assign some customers to a new feature, while holding out a ‘control’ group that doesn’t yet receive it. This is a great way to understand discoverability (can people find it, and understand what it is?), actual demand and usage (do people actually start using it? Do they keep using it?), and business viability (is the cost sustainable?).
But experiments can take a while to give results. During that time we can’t promote the product or feature more widely without disappointing people and biasing the experiment.
And, of course, we can’t predict the outcome of the experiment! If it disproves some of our hypotheses, we might need to fall back to the ‘control’ and return to the drawing board.
Staged rollouts
Almost all products and features are released gradually rather than in one big bang. For example, we might make it available to 10% of customers on day 1, 20% on day 2, etc.
This is a great way to test that our findings scale and uncover edge cases, with a clear plan to roll back if necessary. But it makes it tricky to announce the launch, since we can’t be sure when it’ll be available to everyone.
Silent release
Even once all customers have access to a feature, we might choose to wait a week or two before promoting it more widely. We can use this time to understand whether people can find it organically, which is particularly important as we build out the universe of things you can do with Monzo.
Marketing
Only when we’re confident that this new product or feature is addressing the needs we set out at the start can we start to shout about it more widely. Even then, we might test different messaging and channels to understand what resonates best with Monzo customers.
Choosing the right approach to rolling out a product depends on the key hypotheses we’re trying to test when launching it, as well as how bad it would be to get it wrong. We’ll often use several of these approaches to help answer different questions.
Putting the theory into practice
Returning to the example from earlier, our most important hypotheses when we were building the ability to see your mortgage in Monzo were:
Technical feasibility: will the data we use to display your mortgage be reliable for all customers?
Customer needs: does this solution address the need in a useful way - will people use it? And what other needs should we focus on next?
To best answer these, we chose to release a rough version of the product to staff for a few weeks. We used a simple survey to understand whether people were seeing accurate data about their mortgage. The good news was that most of them were, and for those who weren’t, we managed to work out how we could improve it.
We then launched to Monzo Labs in December, to gather more evidence that data was accurate and to get feedback on where we could take the product next. Our amazing Community delivered in spades!
This approach helped to validate our hypotheses. We identified some data improvements we wanted to make before we were proud to make the product available more widely. We were then confident to launch to all customers in April, and we have an exciting roadmap of features we’re working on - straight out of all the insightful feedback we received - to make sure we keep solving for customer needs.
Another example that we recently launched is the ability for Monzo Flex customers to change their payment date. We heard the need for this loud and clear from your feedback. To make sure it worked properly, we first launched with staff. Then, to make sure it didn’t lead to bad outcomes (like customers being unclear how it worked, or struggling to make their payments as a result), we ran an experiment. This also allowed us to spot a few edge cases we hadn’t thought of and fix them before they led to issues.
Once we concluded the experiment and saw that it was a success, we rolled it out to all customers - and you can find the option in your app today.
How it can go wrong
We don’t always get it right! Two of the ways we get it wrong when we launch new products are:
Launching too quickly: not testing our hypotheses properly, or not testing the right hypotheses
Launching too slowly: using too many of the options in our toolkit, when we don’t really need to
An example of the first one is testing our new app layout with customers. We’d been using a lot of the tools in our locker: staff testing, Labs, and experiments for new Monzo users. But we wanted to test our hypothesis that long-time Monzo users would be able to find their way around the new interface.
The plan was to greet some customers with a screen introducing them to the new layout next time they opened their Monzo app. But although we’d tested the home screen carefully, we hadn’t tested this transition screen in the real world. We soon found out the hard way that it was interrupting people in the middle of completing important tasks, like moving money from Pots to their main balance, or approving a card payment. Not only did this stop them in their tracks, they then found themselves in a redesigned interface, unsure how to complete the task they set out to do.
We did use an experiment here, but we would have moved a little more cautiously if we’d thought more deeply about the impact this change could have.
Conclusion
We love putting new products and features in the hands of as many customers as possible, as soon as possible. But to make sure we build useful, usable, feasible and viable things, we take care to measure their impact.
So next time you spot a new app feature that your friends don’t have yet, you’ll know that you were lucky enough to be assigned to the experimental group. And if the roles are reversed… well, hopefully you can take comfort that it’s all in the name of making money work for everyone. (Including you, eventually!)