> The winners of this split test were surprisingly WITH phone number fields!

That's not what the table with results says 🤔

Why not use the version A? 10,62% of conversion, better than final results.

Why not test the version A with verision C. ?

Hey Nev, I'm not in agreement with your final assessment that phone numbers did better from your test results.

Although your winning "Nev Head + Email" variation reached a statistically significant result (the test winner wasn't random), the 3 losing variations all had phone numbers in them...the phone number variations all did worse! And we can't really point to what made the winning variation do better because there were so many variables tested across the 4 variations (different designs, copy, form fields, etc).

In your second test, there was only one test variable: the hero image designs. In the second test, the signup fields were all the same (name-email-phone # vs. name-email-phone #). This only proves people liked your animated guy over the Nev Head.

That result doesn't rule out the chance that "Animated Guy + Email" vs "Nev Head + Email" would offer better conversion rates. Because of all this, I don't think it's correct to say:

"The variations with [FIRST NAME] + + [PHONE NUMBER] did better than only."

You said you did other tests to confirm that claim...why not show them here as well?

I think a better strategy would be to state the hypothesis of your A/B test up front and then design test variations systematically around trying to answer it. That way, you won't end up with ambiguous conclusions for your winners like this.

I love your Always Be Testing mentality though...keep it up!

Hi Neville! Yeah, your results aren't adding up.

The highest conversion page is not the one you're saying was the winner.

And as a follow-up test, if you are making a sales offer to each group, which group produced the most SALES.

This is where split testing can be deceiving. You might get more OPT-INS on the front end from one landing page, but more SALES from opt-ins on one of the lower-conversion pages.

Looks like Version A was supposed to be the winner with 10+%. Perhaps A was the control, and you are insisting on choosing something from B C D...

But here's come another question: why did version A have more visits than B C D? Split tests are supposed to have equal visits, ain't it?

This test has me salivating for a part 2. Which group of people converted the best on $$ ?

Can't wait!

" why did version A have more visits than B C D?"

Some people are not willing to fill out a form that requires their phone number. It's perceived as a greater invasion of privacy, and it may cost the user if you contact them by text depending on their data plan.

"The winners of this split test were A and C."

The email improperly summarized the web post.

Hey Jackson!

1st of all, I screwed up in writing this post. I left out a crucial paragraph (that's now been added):

We wanted more people to signup WITH their phone numbers, and we were willing to sacrifice that 1% or so of conversions for them.

For this reason we selected Variation C and Variation D to pit against each other for the final homepage.

Sorry for that, the info has now been updated :)

Hey Matt, that was me screwing up and leaving out a crucial piece of information (I've since added to the post):

Hey Michael, you're correct I goofed on the first version of this post.

And exactly to your point, GETTING PHONE NUMBERS was a crucial piece of what we wanted, so we sacrificed a slightly higher conversion rate (as demonstrated by Variation A) for grabbing phone numbers instead.

This is why we ended up pitting Variation C and Variation D against each other for the final homepage....focusing more on end sales than just raw conversion numbers!

Ugghh.....this is what I get for completing a post late at night 🤣

I goofed in the original post, and left out a crucial piece of information: We WANTED to only display a homepage that also collects phone numbers, and we were willing to sacrifice that 1% of extra emails to also get phone numbers.

For this reason I picked only the highest rating PHONE NUMBER pages, and excluded the actual raw conversion winner with only the email signup.

Thanks for the feedback Matt! The post is now updated and corrected :)

Hey Lili, in the initial post I goofed and left out the biggest reason for this:

We were willing to sacrifice an extra 1% in raw conversions in order to display a page that also collects phone numbers.

Because of this, the actual highest converting page (email collect only) did not make it through to the 2nd round of tests.

Hey Philip, the phone number field is optional on all these pages, not required.

So if someone doesn't want to enter a phone number it's fine (and many people don't)!

That was me screwing up and leaving out a crucial piece of information (I've since added to the post):

Hey Mike, that was a mistake on my end, apologies. Yes the email-only did more raw conversions, but as I explained in the updated post, we only wanted versions of the homepage that included phone number collection fields.

For this reason we eliminated the highest converting page from the 2nd round of tests!

Hey Mike, according to these tests that's not totally true. A BUNCH of people signup with their real phone number.

Also on every single test, the only required field is email. First Name and Phone Number fields can be skipped if they want.

However in the end keeping a more complete set of contact info helps the business :)