Shopify Rollouts vs A/B testing apps: what native testing gives you, and what it leaves on the table

What Rollouts is

Rollouts lives in the Shopify admin under Markets. It splits your traffic between a control and a treatment — whole themes, and checkout configurations — decides the assignment before the page renders, and maintains visitor consistency so a returning shopper keeps seeing the same side.

Availability follows your plan: rollouts are available from Basic upward, and experiments from Grow upward. Headless and custom storefront checkouts are not supported.

What it does that no app can do

This is the part worth stating without hedging, because it is a real structural advantage and pretending otherwise would be dishonest: Rollouts assigns the visitor server-side, and a third-party app cannot.

The reason is specific. A theme app extension renders through Liquid, and Liquid's `request` object exposes exactly `host`, `origin`, `path`, `page_type`, `locale`, `design_mode` and `visual_preview_mode`. No cookies. No headers. No query parameters. No visitor identity of any kind. An app rendering inside your theme has nothing to bucket a visitor with, so its assignment has to happen in the browser. Shopify's own infrastructure sits in front of that constraint and is not subject to it.

So if what you want is a server-decided split across two whole themes, with no client-side logic involved at all, Shopify's own feature is the best available implementation of that and it costs nothing. Say so.

There is no public API

One operational fact decides a lot about how you can use Rollouts: there is no public Admin API for it. There is no `rollout` mutation in the GraphQL Admin schema, and the developer community thread asking about programmatic access has no staff answer.

That makes Rollouts an admin-operated tool. A person opens the admin, sets up the split, and reads the outcome there. Nothing schedules it, nothing creates it from a backlog, nothing pulls the result into a report you already have. If your workflow needs any of that, this is the fact that decides it — not a feature comparison.

The unit is a theme, and that has a price

Rollouts tests whole themes and checkout configurations. That is the right unit for a redesign and an expensive one for a sentence.

If you want to know whether moving your returns policy next to the add-to-cart button is worth doing, the theme-level unit means duplicating your live theme, making that one change in the copy, and keeping the copy in sync with the original for as long as the test runs. Every merchandising change, every price update, every seasonal banner your team ships during the test has to land in both. Miss one and you are no longer testing the returns policy; you are testing the returns policy plus whatever drifted.

This is not a criticism of the design — a server-side split has to serve something, and a theme is the thing Shopify can serve. It is a reason the granularity of the change you want to test should decide the tool, not the other way round.

Two more boundaries worth knowing before you plan around it. Availability follows plan: rollouts from Basic, experiments from Grow. And headless or custom storefront checkouts are not supported, which rules it out for a meaningful share of the stores most interested in testing.

What the documentation does not describe

Three questions a merchant asks on day one are not answered anywhere in Shopify’s Rollouts help pages. It is worth being exact about what that means: the documentation is silent, which is not the same as the capability being absent. If any of these decide your choice, check the admin on your own store rather than trusting either us or a help page.

• What an experiment actually reports. The pages describe how to create a rollout and how launch reach works. They do not describe which metrics the experiment surfaces, or how they are computed.
• Conversion goals. There is no documented way to declare what counts as success — that the thing you care about is add-to-cart, or newsletter signup, rather than a purchase.
• Audience targeting past reach and market. You can set the share of eligible visitors, and which markets are eligible if you run more than one. Segmentation by referral source, UTM, device, or new versus returning is not described.
• How long to run it, and when to stop. No documented duration guidance, no significance threshold, no sample-size calculation.

Which number decides it

Assume the split works perfectly and the numbers are clean. You still have to answer a question no split mechanism answers: which number are you reading?

Conversion rate is the default because it is the easiest to compute, and it is the one most likely to mislead on a storefront. A variant that pushes more visitors through checkout while lowering what they spend can raise conversion rate and lower revenue at the same time. A discount banner does this reliably. So does anything that pulls buyers toward a cheaper product.

Revenue per visitor collapses both halves into the number your profit and loss recognises: it moves only if the change earned more money from the same traffic. It is noisier than conversion rate and needs more data to resolve, which is precisely why teams quietly avoid it — not because it is the wrong measure, but because it takes longer to say something.

Shopify’s documentation does not describe which metrics an experiment reports, so we are not going to tell you what it shows. What we will say is that this is the question to take into the admin with you, because a testing program that optimises conversion rate without watching order value can spend a year winning tests and losing margin.

The three jobs it does not do

A traffic split is one of four things an experimentation program needs. Rollouts does the one; the other three are still yours.

An honest decision table

Where we sit

Looplift does not run a split inside Shopify’s infrastructure and does not integrate with Rollouts — there is no API to integrate with, and we will not imply otherwise. What we work on is the three jobs above: finding the opportunity, generating a variant that does not require standing up a second theme, and reporting the result with statistics that survive you checking the dashboard every morning.

If Shopify has commoditised the pipe, that is fine. The pipe was never the interesting part. The two hard parts — knowing which of forty possible changes deserves two weeks of your traffic, and knowing whether what came back is a result or a coincidence — are exactly as hard as they were before Winter ’26, and Rollouts does not claim otherwise.

Sources

• Shopify Help Center — Rollouts (plans, what can be rolled out)
• Shopify Help Center — Create a rollout (Grow plan requirement, launch reach)
• Shopify community — “Shopify Rollouts: access for developers”
• Shopify — Liquid `request` object