Do earlier homepage form wins still hold on a simpler form?

After we removed a step from the homepage form, we did not know if earlier winning changes still helped. We retested them together on mobile and desktop. Over 20 days, conversion rose 7.8%, short of our confidence bar.

Not significant Responsive 20 days Not called

SURFACE

Homepage

DEVICE

Responsive

VERTICAL

Ecommerce (B2C)

PRIMARY METRIC

Conversion rate

VARIATIONS

1

The hypothesis

By bringing earlier winning changes back to the simplified homepage form, we will confirm they still help, resulting in more completed orders.

We had just removed the vehicle identification number step from the homepage form. Several earlier wins had been built for the old form, so we wanted to check they still worked with the new, shorter one before building on top of it.

What we changed

A new form headline, with the service choice moved or made a dropdown

Var A gave the form a new headline that framed it as a quick registration. On mobile, the list of services became a dropdown. On desktop, the services moved below the form fields.

VAR A: BACK TEST

New form headline
Services shown as a dropdown on mobile
Services moved below the fields on desktop

The result

Conversion rose 7.8% over 20 days, short of our confidence bar

CONVERSION RATE

+7.8%

FORM SUBMITS

-1.2%

Over 20 days, the retest converted 7.8% better than the original, and revenue per visitor rose by the same amount. Form submits were nearly flat, down 1.2%. The overall gap did not reach our confidence bar. By device, mobile conversion fell while desktop conversion rose.

What we would do next

Keep the desktop change and retest mobile without the dropdown

The 2 devices moved in opposite directions, which flattened the overall result. The mobile drop pointed at the new service dropdown, so the next round kept the original mobile service selector and repeated the rest.