A/B Testing

Which checkout experience helps more customers finish a purchase? Rather than committing everyone to a redesign, use a flag to serve a control and a treatment to stable groups. Measure the result before deciding what to ship.

What Flagpool does — and what you bring

Flagpool serves the configured experience through string or JSON flags and targeting rules. Your application or experiment service assigns the groups; your analytics measures the outcomes. A percentage rollout controls exposure, but does not automatically split users evenly across all flag variations.

This makes flags useful when you want to change experiences without redeploying each experiment configuration, while keeping the measurement tools you already use.

Set up a checkout experiment

1. Define a decision, not just a variant

Write down a hypothesis: “A shorter checkout will increase completed purchases without increasing payment errors.”

Before starting, agree on:

Use the same eligibility criteria for both groups. Avoid changing the hypothesis after looking at the results.

2. Assign stable control and treatment groups

Use your application's existing assignment service or persist a randomly assigned group per user and experiment. Pass that assignment in the SDK context:

client.updateContext({
  userId: user.id,
  checkoutExperimentGroup: assignment.group, // 'control' or 'treatment'
})

assignment comes from your application's experiment system, not from Flagpool. Keep it stable across sessions, and do not assign it afresh on every render. Anonymous-user experiments also need a stable identifier and a policy for what happens when a user signs in.

3. Create the flag and targeting rule

Create a string flag called checkout-layout:

The treatment rule selects the new checkout. Users without that assignment receive the control. The split is determined by your assignment system, not by the rollout slider.

Why 100% here? The experiment audience is controlled by the targeting rule. Reducing the rollout is not an experiment exclusion mechanism: unmatched users outside the rollout receive the off variation, which is typically variation 1. In this example that would be the treatment.

To restrict the experiment to a segment, assign groups only to eligible users in your application. Do not assume that independent targeting rules behave like an AND expression; the first matching rule returns a variation.

4. Render a safe experience

After initializing the JavaScript SDK:

const layout = client.getValue('checkout-layout')

if (layout === 'single-page') {
  renderSinglePageCheckout()
} else {
  renderCurrentCheckout()
}

The current checkout is the fallback for an unavailable or unexpected flag value. In React, use useFlagValue to choose the same two paths, with the current checkout as your loading fallback.

5. Measure exposure and outcomes

When a user actually sees the checkout, record an exposure event in your analytics, including the experiment ID, stable user ID, assigned group, and rendered variation. Record completed purchases and errors using the same identity.

Do not count every flag evaluation as a distinct experiment exposure. Repeated renders, retries, and background checks can inflate those counts.

Compare the primary metric between groups, examine guardrails, and use your analytics or experimentation tools to assess uncertainty. Flagpool evaluation counts alone cannot establish conversion lift or statistical significance.

Decide what to ship

A treatment that raises conversion but also causes payment failures is not automatically a winner. Check the agreed guardrails and sample size before making the release decision; an inconclusive result is a valid outcome.

To end this example experiment:

1. Set the default variation to the chosen experience and remove experiment targeting rules.
2. Keep the rollout at 100% and confirm both previous groups now receive the same experience.
3. Deploy code that uses the chosen experience without the temporary flag.
4. Archive the flag once no supported application version relies on it.

To stop treatment early, change or remove its rule and set the default to control. Do not simply lower the rollout: targeting rules take precedence.

Ready to run the experiment?

• Hypothesis, metrics, guardrails, and decision criteria are written down.
• Group assignment is stable and eligibility is consistent.
• Control, treatment, and users without an assignment receive the correct experience.
• Exposure and conversion events share the same identity in your analytics.
• The control works while configuration loads or is unavailable.
• The team has a tested stop procedure and a cleanup owner.

Build on this setup

• Targeting — understand rule priority and matching.
• Contexts — supply the attributes that choose an experience.
• Progressive rollouts — release a selected experience in controlled stages.