Jump to content

A 4-Way Homepage Split Test (w/ Numbers, Screenshots, and Results)


Neville

Split Test Variants A B C D Homepage

Split tests. They're often not very accurate, but they are fun to perform! Also if you do them correctly, they can be very profitable.

If you're careful to perform your split test with small changes, with similar calls to action, and over 1,000's of views, then they can be very telling. Just for fun I'm posting the results of a small test we did on the Copywriting Course Home Page.

We split tested 4 different variations of the page:

  • Version A: Neville Head + Email Signup
  • Version B: Neville Wall + Email Signup + Phone Signup
  • Version C: Animated Figure + Email Signup + Phone Signup
  • Version D: Neville Head + Email Signup + Phone Signup

Split test a b c d variation

 

 

 

 

We wanted to get a rough idea of which two we should select, so this test was ran across roughly 6,500 unique visits.

Results of this 4-way split test:

[table id=6 /]

The winners of this split test were A and C.

split-a-c.png

The email-only form with my face looking at it did the best, HOWEVER, it was missing a crucial thing we wanted from this test: PHONE NUMBERS!

This was not what we wanted, as getting a complete set of contact information was more important to us than slightly more emails (which a percentage of are often not very responsive).

So we selected the top performing pages with phone numbers, then had to further test these pages to make sure we got the right one to display to all homepage visitors.

 

So we did another Split Test with the top phone number collecting pages (C vs D):

c-d-split-test.pngWill the Animated Dude or the Neville Head do better??

Since this was our home page, and most of our signups come from here, we wanted to further split test the page. So we ran Version C and Version D against each other for roughly 30,000 unique visits to the homepage, and here were the results:

[table id=7 /]

The Animated Dude was the winner with 10.08% !!

Stickman-homepage-400w

 

Neville Signature


User Feedback

Recommended Comments



> The winners of this split test were surprisingly WITH phone number fields!

That's not what the table with results says 🤔

Link to comment
Guest Philip Lulu

Posted

Thanks for another masterpiece! Any idea why variations with phone number did better? Given the fact that there's no disclaimer on how the phone number will be used in the future. Obvious questions arise in my mind when I saw this field - who will call me? when will they call me? will they spam me with text messages?
Link to comment
Guest lili collins

Posted

Why not use the version A? 10,62% of conversion, better than final results.

Why not test the version A with verision C. ?

Link to comment
I'm confused too. I thought A was the winner.Also, when split testing I thought you should have just one variable. In the examples above both the image and text is different in some cases.
Link to comment
Me again. Re my comment above about split testing. I didn't look thoroughly. I was referring to the text in the image section. I realised as soon as I posted that the overall text was mainly the same in all cases. It just didn't appear in a similar place on variation B. Plus variation B had additional words - Can we give you a call? So only slightly different. Sorry about that.
Link to comment

Hey Nev, I'm not in agreement with your final assessment that phone numbers did better from your test results.

Although your winning "Nev Head + Email" variation reached a statistically significant result (the test winner wasn't random), the 3 losing variations all had phone numbers in them...the phone number variations all did worse! And we can't really point to what made the winning variation do better because there were so many variables tested across the 4 variations (different designs, copy, form fields, etc).

In your second test, there was only one test variable: the hero image designs. In the second test, the signup fields were all the same (name-email-phone # vs. name-email-phone #). This only proves people liked your animated guy over the Nev Head.

That result doesn't rule out the chance that "Animated Guy + Email" vs "Nev Head + Email" would offer better conversion rates. Because of all this, I don't think it's correct to say:

"The variations with [FIRST NAME] + + [PHONE NUMBER] did better than only."

You said you did other tests to confirm that claim...why not show them here as well?

I think a better strategy would be to state the hypothesis of your A/B test up front and then design test variations systematically around trying to answer it. That way, you won't end up with ambiguous conclusions for your winners like this.

I love your Always Be Testing mentality though...keep it up!

Link to comment
Guest Michael

Posted

Hi Neville! Yeah, your results aren't adding up.

The highest conversion page is not the one you're saying was the winner.

And as a follow-up test, if you are making a sales offer to each group, which group produced the most SALES.

This is where split testing can be deceiving. You might get more OPT-INS on the front end from one landing page, but more SALES from opt-ins on one of the lower-conversion pages.

Link to comment
Looks like the very first one in the table has the highest conversion rate. Why wasn't it the winner?
Link to comment
Guest Jackson Y

Posted

Looks like Version A was supposed to be the winner with 10+%. Perhaps A was the control, and you are insisting on choosing something from B C D...

But here's come another question: why did version A have more visits than B C D? Split tests are supposed to have equal visits, ain't it?

Link to comment
Guest Quinton Hamp

Posted

This test has me salivating for a part 2. Which group of people converted the best on $$ ?

Can't wait!

Link to comment

" why did version A have more visits than B C D?"

Some people are not willing to fill out a form that requires their phone number. It's perceived as a greater invasion of privacy, and it may cost the user if you contact them by text depending on their data plan.

Link to comment

"The winners of this split test were A and C."

The email improperly summarized the web post.

Link to comment
Guest Neville Medhora

Posted

Hey Jackson!

1st of all, I screwed up in writing this post. I left out a crucial paragraph (that's now been added):

We wanted more people to signup WITH their phone numbers, and we were willing to sacrifice that 1% or so of conversions for them.

For this reason we selected Variation C and Variation D to pit against each other for the final homepage.

Sorry for that, the info has now been updated :)

Link to comment
Guest Neville Medhora

Posted

Hey Matt, that was me screwing up and leaving out a crucial piece of information (I've since added to the post):

The email-only form with my face looking at it did the best, HOWEVER, it was missing a crucial thing we wanted from this test: PHONE NUMBERS!This was not what we wanted, as getting a complete set of contact information was more important to us than slightly more emails (which a percentage of are often not very responsive).So we selected the top performing pages with phone numbers, then had to further test these pages to make sure we got the right one to display to all homepage visitors.

 

Link to comment
Guest Neville Medhora

Posted

Hey Michael, you're correct I goofed on the first version of this post.

And exactly to your point, GETTING PHONE NUMBERS was a crucial piece of what we wanted, so we sacrificed a slightly higher conversion rate (as demonstrated by Variation A) for grabbing phone numbers instead.

This is why we ended up pitting Variation C and Variation D against each other for the final homepage....focusing more on end sales than just raw conversion numbers!

 

Link to comment
Guest Neville Medhora

Posted

Ugghh.....this is what I get for completing a post late at night 🤣

I goofed in the original post, and left out a crucial piece of information: We WANTED to only display a homepage that also collects phone numbers, and we were willing to sacrifice that 1% of extra emails to also get phone numbers.

For this reason I picked only the highest rating PHONE NUMBER pages, and excluded the actual raw conversion winner with only the email signup.

Thanks for the feedback Matt! The post is now updated and corrected :)

Link to comment
Guest Neville Medhora

Posted

Hey Lili, in the initial post I goofed and left out the biggest reason for this:

We were willing to sacrifice an extra 1% in raw conversions in order to display a page that also collects phone numbers.

Because of this, the actual highest converting page (email collect only) did not make it through to the 2nd round of tests.

Link to comment
Guest Neville Medhora

Posted

Hey Rodrigo, that was my bad. As you can see elsewhere in the comments I'm explaining that in the first iteration of this post I forgot to mention we specifically only wanted phone number collecting pages to win the final test :)
Link to comment
Guest Neville Medhora

Posted

Hey Philip, the phone number field is optional on all these pages, not required.

So if someone doesn't want to enter a phone number it's fine (and many people don't)!

Link to comment
Guest Neville Medhora

Posted

That was me screwing up and leaving out a crucial piece of information (I've since added to the post):

The email-only form with my face looking at it did the best, HOWEVER, it was missing a crucial thing we wanted from this test: PHONE NUMBERS!This was not what we wanted, as getting a complete set of contact information was more important to us than slightly more emails (which a percentage of are often not very responsive).So we selected the top performing pages with phone numbers, then had to further test these pages to make sure we got the right one to display to all homepage visitors.

 

Link to comment
Guest Neville Medhora

Posted

Hey Mike, that was a mistake on my end, apologies. Yes the email-only did more raw conversions, but as I explained in the updated post, we only wanted versions of the homepage that included phone number collection fields.

For this reason we eliminated the highest converting page from the 2nd round of tests!

Link to comment
Guest Neville Medhora

Posted

Hey Mike, according to these tests that's not totally true. A BUNCH of people signup with their real phone number.

Also on every single test, the only required field is email. First Name and Phone Number fields can be skipped if they want.

However in the end keeping a more complete set of contact info helps the business :)

Link to comment


×
×
  • Create New...
Guest