Shopify Rollouts explained: what the free app can't do

Shopify Rollouts is the free, native A/B testing app for theme-level tests. What it costs, what it can test, how to read results and where it stops.

TL;DR

• Shopify Rollouts is Shopify's own A/B testing app. It is free, installs from the App Store, and tests one full theme version against another.
• It cannot test a single element, headline or button — every test is theme A versus theme B, split 50/50.
• Results show sessions, conversion rate and revenue per session for each theme. You decide when the numbers are settled — Rollouts does not call the winner for you.
• At 5,000 monthly sessions and a 2% conversion rate, a 20% relative lift needs about 36 weeks to reach a readable result. At 50,000 sessions it needs about 4.
• If you want one theme test, use Rollouts. If you want a running program of tests without designing them yourself, that is a different category.

What is Shopify Rollouts?

Shopify Rollouts is a free A/B testing app built by Shopify. You install it from the Shopify App Store, pick two theme versions — usually your live theme and a duplicate you have edited — and Rollouts splits your storefront traffic between them. Visitors in the test see either theme A or theme B for the whole session, and the app reports sessions, orders, conversion rate and revenue per session for each side.

That is the entire product, and for a lot of merchants it is enough. If you run a store on Dawn and want to know whether the redesigned homepage your freelancer built converts better than the current one, Rollouts is the right pick. It costs nothing, it does not touch your theme code, it does not slow the page down with a third-party script, and it is supported by Shopify rather than a vendor that might disappear.

The catch is in the word 'theme'. Rollouts tests whole themes against each other. It cannot test one headline, one button or one product-page layout on its own. It does not let you choose a traffic split other than 50/50 at the time of checking, it does not target by URL or device, and it does not tell you when a result is statistically settled. The rest of this post walks through each of those limits with numbers, so you can decide whether Rollouts is enough for your store or whether you have already grown past it.

What Shopify Rollouts costs

Rollouts is free. There is no paid tier, no per-session fee and no traffic cap that pushes you into a subscription. The only requirement is an active store on a standard Shopify plan — Basic, Shopify, Advanced or Plus — with the Online Store sales channel enabled. Searches for 'shopify rollouts pricing' turn up nothing because there is nothing to price.

The cost that does exist is time. Because every test is theme versus theme, you need a second theme version to test. That means duplicating your live theme, making the change in the theme editor, and keeping the two versions in sync while the test runs. If you ship a product launch or a banner change during a 6-week test, you make it twice. Merchants on heavily customised themes like Symmetry or Brooklyn find this harder than merchants on stock Dawn or Sense, because the duplicate carries every custom section with it.

For comparison, Intelligems starts at around $99 a month, VWO's Shopify plans start higher, and Convert prices by tested visitors. Rollouts at $0 is the cheapest experimentation platform a Shopify merchant can run — as long as a whole-theme test is the shape of experiment you want.

What Shopify Rollouts can and can't test

We checked the Rollouts listing on the Shopify App Store and the setup screen inside a Dawn test store on the publish date of this post. Here is what we found.

Capability	Shopify Rollouts	Notes
Theme-level tests (theme A vs theme B)	Yes	The core feature. Both versions must be in your theme library
Element-level tests (one headline, button, section)	No	Only via a full duplicate theme carrying that one change
Traffic split control	No — fixed 50/50	You cannot send 10% to the challenger to limit risk
URL or page targeting	No	Every page on the storefront is in the test
Device or audience targeting	No
Checkout tests	No	Checkout is rendered outside the Online Store theme
Price or discount tests	No	Prices live in the catalogue, not the theme
Significance readout	No	You see sessions, conversion rate and revenue per session per theme
Who analyses the result	You	Rollouts reports; it does not declare a winner
Concurrent tests	One at a time
Price	Free	Standard Shopify plan required

Two rows matter more than the rest. 'Element-level tests: no' means that if you want to know whether a new product-page headline beats the old one, you must duplicate the whole theme just to change one line — and any other difference between the two themes contaminates the result. 'Significance readout: no' means the app hands you two conversion rates and leaves the judgement to you.

How you read Shopify Rollouts results

The results screen shows each theme side by side: sessions, orders, conversion rate, average order value and revenue per session. Rollouts does not show a confidence level, a probability that one theme beats the other, or a recommended stop date.

That leaves you with a decision most merchants have never been taught to make. If theme A converts at 2.1% on 3,000 sessions and theme B converts at 2.4% on 3,000 sessions, has B won? The honest answer is no — with those numbers the difference is well inside the noise, and rerunning the same test could easily flip the ranking. Stopping there and publishing theme B is the most common mistake we see merchants make with Rollouts, and it is not the app's fault. The app simply does not stop you.

If you use Rollouts, decide the run length before you start, using the table in the next section, and do not look at the numbers until the run is over. Peeking daily and stopping the moment the challenger pulls ahead roughly doubles your false-positive rate.

How long a Rollouts test takes at your traffic

We ran the standard sample-size calculation for a store with a 2% baseline conversion rate, a 50/50 split, 95% confidence and 80% power — the same assumptions a CRO consultant would use. The table shows how many weeks the test needs to run before the result is readable, at three traffic levels and three sizes of lift.

Monthly sessions	10% relative lift (2.0% → 2.2%)	20% relative lift (2.0% → 2.4%)	30% relative lift (2.0% → 2.6%)
5,000	~140 weeks	~36 weeks	~17 weeks
20,000	~35 weeks	~9 weeks	~4–5 weeks
50,000	~14 weeks	~4 weeks	~2 weeks

Three things fall out of this table.

First, at 5,000 monthly sessions — roughly a store doing $100k to $150k a year at typical order values — Rollouts is a tool for testing very large changes. A full redesign that lifts conversion by 30% is readable in about 4 months. A better headline that lifts it by 10% would need almost 3 years.

Second, the fixed 50/50 split works in your favour. It is the most statistically efficient allocation. Tools that let you send 10% to the challenger take far longer to converge; Rollouts does not give you that rope to hang yourself with.

Third, because every page in the storefront is in the test, seasonality hits both themes equally. That is a genuine advantage over URL-targeted tests, where a promo on one page can skew the split.

Where Rollouts stops — and when to move on

Rollouts stops being the right tool at the point where you want a program rather than a test. Signs you have reached that point:

• You have a backlog of five or more ideas — button copy, trust badges, shipping messaging, product-page layout — and each one would need its own duplicate theme.
• You want to test a change on product pages only, without putting the homepage and collections in the test too.
• You want someone or something to tell you when a result is real, instead of reading two conversion rates and guessing.
• You finished a Rollouts test and realised the two themes had drifted apart in ways you did not intend.
• You are searching for a 'shopify experimentation platform' and hoping the answer is not 'hire a consultant'.

At that point you have two directions.

The first is a classical experimentation platform. Intelligems, VWO, Convert and Visually.io all do element-level tests, URL targeting, traffic-split control and a proper significance readout. You design the test, they run it and report it. We lay out those trade-offs in Shopify Rollouts alternatives, and go head to head with the largest of them in ShopShift vs VWO.

The second is autonomous conversion optimization — a category where the AI picks what to test, writes the variant, runs the split and declares the winner from a single script tag, with no theme duplication and no hypothesis form. We wrote the full definition of autonomous conversion optimization if the term is new to you, and ShopShift vs Shopify Rollouts walks through exactly what changes when you move from one theme test to a loop that runs on its own.

Neither direction makes Rollouts a bad product. It is a free, native answer to one specific question — does this theme beat that one — and it answers it honestly. It just does not pretend to answer the next question.

Frequently asked questions

Is Shopify Rollouts really free?

Yes. Rollouts is built and published by Shopify and has no paid tier, no usage-based fee and no traffic ceiling. You need a store on a standard Shopify plan with the Online Store channel active — that is the whole requirement. The real cost is the effort of maintaining a duplicate theme for the length of the test, and the risk of publishing a challenger that only looked like a winner because you stopped early. Neither shows up on an invoice, but both are real.

Can Shopify Rollouts test a single button or headline?

Not directly. Rollouts only tests one theme version against another. The workaround is to duplicate your live theme, change the one headline in the duplicate and run the test with everything else identical. That works if you are disciplined about not touching either theme during the run. It breaks the moment you fix a typo in one theme and forget the other, because the test is now measuring two changes and telling you about neither. Element-level testing is the first feature merchants outgrow Rollouts for.

Does Shopify Rollouts tell me when a test is statistically significant?

No. At the time of checking, the results screen shows sessions, orders, conversion rate, average order value and revenue per session for each theme, and nothing else. There is no confidence level, no probability-to-beat-control and no suggested stop date. You have to bring the significance judgement yourself — either by fixing the run length in advance from a sample-size table like the one above, or by running the raw numbers through an external calculator. Most merchants do neither, which is why Rollouts results get over-read.

Does Shopify Rollouts work on the checkout?

No. Checkout on Shopify is rendered outside the Online Store theme, so a theme-level test never reaches it. Rollouts covers the homepage, collection pages, product pages, cart page and any other page your theme controls. If your conversion problem is at checkout — shipping-cost shock, forced account creation, a missing payment method — no theme test on any tool will show it. Checkout Extensibility on Shopify Plus is the route for testing that step, and it is a different project.

Does Shopify Rollouts slow my store down?

No, and this is one of its strongest points. Because Rollouts is native, the theme split happens on Shopify's servers before the page is sent. There is no third-party JavaScript to load, no flicker while a script swaps content and no dependency on a vendor's CDN. Every script-tag tool — including ShopShift — has to work around those problems with anti-flicker techniques. Rollouts never has them in the first place. If page speed is your top concern and a whole-theme test is what you need, Rollouts wins on this point outright.

Related reading

• What is autonomous conversion optimization? — the category definition, if you have outgrown one-theme tests.
• ShopShift vs Shopify Rollouts — a side-by-side of what changes when the AI runs the loop.
• Pricing — $99 a month with a 30-day free trial, against Rollouts' $0.