How to choose a CRO agency

To choose a CRO agency, judge it by what it shows you, not what it promises. A good agency shows every test it runs, losses included. It splits results by device. It can tell you, before a test starts, how it will decide the test is done. And it can explain what its monthly fee buys.

Most CRO agencies charge $5,000 to $25,000 a month. At that price, the sales deck is not enough. Here are 5 things to check before you sign, then the published tests that show why each one matters.

1. Ask to see the losses. Most tests do not win. An agency that only shows wins is showing you a highlight reel.
2. Ask how they test copy and design. A good program changes 1 thing at a time, so you know what earned the result.
3. Ask for results by device. A change can win on mobile and do nothing on desktop.
4. Ask how they decide a test is done. The rule should be set before launch, not after the numbers come in.
5. Ask what the fee includes. Research, design, build, QA, analysis and implementation are not always in the price.

Only 60 of our 237 published tests won

Our test archive holds 237 published tests. 60 of them won. The rest lost, or never showed a clear result. That is normal for real testing, and it is why the losses matter as much as the wins.

Most agencies publish a few case studies with big numbers. Those are real results, but they are picked. They do not tell you how often the agency’s ideas fail, or what it does when they do.

Here is 1 of our losses. On a homepage form, we added small help icons beside each field. Each icon opened a tooltip that explained what to enter. The idea was simple: fewer mistakes, more finished forms.

Desktop conversion fell 8.1% in 8 days, so we paused the test early. The icons added clutter without helping people finish. That loss is useful. It told us visitors on that form did not need more help text, so the next test went after something else.

When you talk to an agency, ask to see a full month of tests, not a case study. Ask what share of their tests win. If the answer is “almost all of them,” they are either not testing much, or not showing you everything.

A copy change alone lifted conversion 23.5%

A redesign changes the layout, the images and the words all at once. If it wins, nobody knows which part did the work. If it loses, nobody knows which part to fix. A good CRO agency tests copy and design as separate changes, so each result teaches you something.

On a lead form for people with tax debt, the headline and button used the industry’s formal term for unpaid tax. We changed only the words. The new headline and button said “back taxes,” the phrase people actually search for.

Conversion rose 23.5% over 9 days, and revenue per visitor rose 21.2%. Nothing else on the page changed, so we know the words earned it. That is the kind of result you can reuse: the same lesson can go into ads, emails and other pages.

Ask any agency you are talking to for an example like this. How did they isolate what won? If every test they show is a full page redesign, they may be guessing at why it worked.

A dropdown lifted mobile conversion 10.4% and did nothing on desktop

Mobile and desktop visitors use the same page in very different ways. A phone screen has room for a few lines at a time. A desktop screen shows almost everything at once. So 1 change can help on 1 device and do nothing, or harm, on the other.

On a homepage form, visitors first picked a service type from a list of options. On a phone, that list pushed the rest of the form down the screen. We moved the choice into a dropdown and reordered the fields.

Mobile conversion rose 10.4% over 26 days. On desktop, the same idea showed no lift over the control. Desktop has room to show every option, so the dropdown solved a problem desktop visitors did not have.

If an agency reports 1 blended number for every test, a win on 1 device can hide a loss on the other. Ask for results by device. Then ask whether they implement winners only where they won.

A losing test should be paused in days, not left running for weeks

Every test needs a stopping rule, and it should be written down before launch. Without 1, it is easy to stop a test the day it looks good, or let a loser run because nobody wants to call it.

Ask the agency these 4 questions about how a test ends:

• What is the main metric? It should be named before launch. For a store, revenue per visitor usually tells you more than clicks.
• How long will it run? Full weeks are safer, because weekday and weekend visitors behave differently.
• When do you pause early? The tooltip test above was paused after 8 days because it was clearly losing.
• What happens with no clear winner? Some tests end flat. A good agency explains whether to keep the change, drop it, or try a sharper version.

Traffic sets the limits on all of this. Below roughly 30,000 sessions a month, most tests take months to read. An honest agency will tell you if your traffic is too low to test well, and suggest spending on traffic first. Our guide on how to conduct a CRO audit covers what to fix when testing is not the right next step.

A good CRO program runs in 5 steps, from research to handoff

Ask any agency to walk you through its process. It should start with research, not a list of ideas. Our program follows 5 steps:

1. Find the opportunities. Review analytics, watch real users and find where visitors get stuck.
2. Prioritize what is worth testing. Rank ideas by how much revenue they could move and how fast your traffic can prove them.
3. Design and build the test. Design, build and QA every variation before it goes live.
4. Measure what actually changed. Judge the result on the main metric, by device, and check that nothing else got worse.
5. Hand off the winner. Implement it, or give your developers the tested code and findings.

Research is the step most often skipped. Analytics show where visitors leave. Watching real people use the site shows why. Our programs include usability testing with at least 10 to 12 participants each term. Ask any agency how many real users it will watch before it writes the first test.

Step 4 matters more than it looks. A test can win on its main metric and still hurt something else. For a store, ask what the agency checks besides conversion: average order value, add to cart and return rate are a good start.

3 red flags should end an agency conversation

Some answers tell you enough on the first call. CXL’s hiring guide flags the same 3 warning signs we watch for:

• Big promises before any research. An agency that predicts a large lift before it has seen your data is selling, not testing.
• No questions about your history. A good agency asks for your past tests, analytics and customer data. If it does not ask, it plans to start from scratch and repeat what you already know.
• References that are “confidential.” Most agencies sign NDAs. A good one can still put you on a call with a past client.

A CRO agency costs $5,000 to $25,000 a month

Most CRO agency retainers fall between $5,000 and $25,000 a month. A 3-person in-house team costs $35,000 or more a month. Our own programs start at $6,495 a month.

I think price is the last thing to compare, not the first. These options buy different things, so the monthly number on its own tells you very little. A retainer pays for a team, not for tests.

4 things change what a program costs:

1. Test volume. More tests means more research, build time and analysis. Your traffic also caps how many tests can run well.
2. How much of the site is covered. Working on 1 part of the site costs less than running product pages, cart and checkout at the same time.
3. Who implements the winners. The agency can hand tested code to your developers, or implement winners itself. Doing it itself costs more and saves your team time.
4. Contract length. Longer terms usually carry a lower monthly rate, because each test builds on the research from the last one.

Before you compare quotes, ask each agency what its fee includes. Design, build, QA, analysis and testing tools are sometimes extra. Ask about setup fees, too. Ours has none; the first month covers the audits and tool setup.

Also ask who keeps the record. I believe a year of tests is worth more than any single win. Where that record lives when the contract ends is the difference between a program and a series of projects.

Any agency can show you its best win. Ask to see the losses.

A partner gets your first test live in under 2 weeks

A typical CRO agency puts an account manager between you and the people doing the work. It sends a monthly report and holds scheduled calls. First tests often take 4 to 8 weeks to launch. Documentation is usually a few selected case studies.

repeatGROWTH works as a partner, not a typical agency. I have worked all 3 ways: in-house, with an agency, and as an embedded team. As a partner, the person you talk to is the person building the test. I work on every test, with research, design, build, QA and analysis from the same team. We work in your Slack and answer the same day.

The partner model puts results first. Your first test goes live in under 2 weeks, so you see real results early in the first term. Terms run 3, 6 or 12 months, and you decide whether to continue based on what the tests show. Every test is documented and handed to you, wins and losses, so the learning stays with your company if you stop.

A partner is not always the right call. If you need 10 or more tests a month, an agency with a bigger team fits better. If you want testing to become part of your culture, hire in-house. We compare all 3 side by side on our partner, agency or in-house comparison, and our conversion rate optimization page shows what the program includes. If you are weighing a single consultant instead, see whether to hire a CRO consultant.

Frequently asked questions

What does a CRO agency do?

A CRO agency finds where visitors drop off on your site, designs changes to fix it, and runs A/B tests to prove which changes work. A good one also measures the result on revenue, implements or hands off the winners, and documents every test, including the losses.

How much does a CRO agency cost?

Most CRO agencies charge $5,000 to $25,000 a month. A 3-person in-house team costs $35,000 or more a month. repeatGROWTH programs start at $6,495 a month. Price depends on test volume, how much of the site is covered, who implements the winners and contract length.

How do you choose a CRO agency for an ecommerce store?

Ask to see every test from a recent month, losses included. Ask for results by device, how they decide a test is done, and whether they judge tests on revenue per visitor. Check what the fee includes and who keeps the test records when the contract ends.

What is the typical return from hiring a CRO agency?

There is no honest single number. Most tests do not win: 60 of our 237 published tests won. The return comes from the winners added up over time, and from what the losses teach. Sites below roughly 30,000 sessions a month often cannot read tests fast enough to see a return.

How long does a CRO agency take to show results?

Many agencies take 4 to 8 weeks to launch a first test. repeatGROWTH launches the first test in under 2 weeks. Each test then runs for full weeks before it can be called.