How does the AB testing result work ?

Question

Hi,

I've set up a test and would like to know a bit more about the results displayed. This appears as a %. Is it actually a raw % of ALL occurences ? Or is it leveled by the variant ponderations ?

Example :

Variant 1 (50%) - 35% result

Variant 2 (40%) - 40% result

Variant 3 (10%) - 25% result

Does this mean than for 100 goalAchieved triggered, 35 were from V1, 40 from V2 and 25 from V3 ? Or do you already include the 50/40/10% ratio in those figures ?

Thank you,

Answer 1

Hi,

Edited:
The ABTest has two parts, one is traffic distribution and another is conversion. When you execute any test from client side, the server sends the variant applicable for this user as per the distribution you have created. Once client gets the variant name then you call goalAcheived when the user is really converted as per the app parameters.

The result is based on the goal achieved call. Suppose out of 100 users, 50 users called goal achieved for V1, 40 called for V2 and 10 called for V3 then the percentage of goal achieved will be 50% for V1, 40% for V2 and 10% for V3.

Let me know if you have any queries.

Thanks.

Comments

Thank you Rajeev, this is precisely the information I was looking for !

However, I don't understand how this total is always going to be 100%.
I mean, getting back to your example :
What if 500 (100% of 500) users came through V1, 160(40% of 400) through V2 and 25 (25% of 100) through V3 ?
The total % would be 165%, but the results never display more than 100%.

How are the results displayed then ?

---

Hi,
I have edited my answer. Please check and let me know if you have any query.
Thanks.

---

Hey,

Sounds good, thanks. So we do have to manually infer which solution is the best depending on the traffic distribution. A 10% variant bringing 30% of the goals might be better than a 30% variant bringing 30% as well !

Thanks for your time,