Everything is About testing

And the most successful companies understood it. Airbnb, Netflix, and Amazon are constantly A/B testing new solutions And they built their whole culture around A/B testing to ensure innovation.

Built Fast. Fail fast

Indeed A/B Tests are effective to confirm a feature's success. But it is worth nothing if run the wrong way...

Here are 15 tips about A/B tests I learned at ManoMano:

1. 1️⃣ AB test one element at the time

For example, If you test the wording of a button all the other elements of the page should be the same. Either you won't be able to attribute the success of one version to one specific element.

2. 🧠 Use common sense

Not everything should be tested, test only feature where you want to confirm assumptions and truly believe they could have an impact on your performance.

3. 📅 Estimate correctly your AB test duration

Use the daily average of users using your feature/entering your Funnel and divide it by the result of the Evan miller tool to get a precise estimation.

4. ⌛️ Wait for statistical significance

Don't use the results of the first test day to decide on the success of any feature.

5. 🤝 Consider the right users

Make sure that you count only users entering your Funnel not the whole traffic on your service.

6. 🧪 Keep a 50/50 split between your test pools

Any other split will increase your test duration. Don't change this unless you already have a clear champion in mind.

7. 💯 Define your success metric BEFORE

Either you could take any positive metric to confirm the success of your test at the end of it.

8. 2️⃣ Track 1 or 2 security metrics

This will confirm your tested feature's success if combined with positive results on your key metrics. Not replace your key metric!

9. ⛔ Avoid A/B/C/... tests

Using too many pools will increase the test duration and affect its accuracy, being more exposed to randomness. Doing proper user research should help you to set aside other solutions you think about

10. ✂️ Avoid segmenting your population

You will be tempted to segment your population during the test to reach statistical significance. Doing this is statically Wrong.

11. 📄 Document your tests and results

Documentation will help you build upon your past lessons and educate your successors.

12. 🔎 Focus experimentation towards the top of the funnel, or where important interactions occur

🙌 Thanks to all my mentors on this subject! Nicolas Huygue, Pascal Davis, veronika Lambert and Thomas Lanzoni