Experiments

An experiment splits traffic into named arms, such as control and variant. On its own it changes nothing. A rule tests the arm, as “they are in” or “they are not in” experiment · arm, and returns its own outcome. What you compare afterwards is how the users who got each outcome behaved.

EdgeKit assigns arms itself, each time a decide request reads one. It stores no assignment, so the arm is computed again on each request, with the same answer as long as the experiment does not change.

How a unit gets its arm

Each experiment has a unit: what gets an arm.

Unit	The id that is hashed	What it means
Per device	The device id	Clearing cookies or opening another browser can change the arm. A request with no device id gets no arm.
Per user	The signed-in user id	The arm follows the user across devices once they sign in. A request with nobody signed in gets no arm.

The device id is the device cookie EdgeKit Analytics sets on your domain, the same one device.token reads. A device’s first request, before that cookie exists, gets no arm.

EdgeKit hashes the experiment’s salt, its name and the unit’s id into one of 1,000 buckets, numbered 0 to 999. Each arm takes a number of buckets, its weight, and the arms take consecutive ranges in the order they are listed. With control at 500 and variant at 500, buckets 0 to 499 are control and 500 to 999 are variant.

The same unit lands in the same bucket on every request. It changes arm only when you change the experiment:

• Changing the salt re-rolls every unit. Results from before and after the change are two experiments.
• Renaming the experiment re-rolls every unit too, because the name is part of the hash. The rules that name it keep the old name.
• Changing a weight or the order of the arms moves the bucket ranges, so every unit whose bucket now falls in another arm’s range changes arm.

Because the name is part of the hash, two experiments split units independently of each other, even with the same salt.

Targeting

Targeting decides who is in the experiment at all. It is a list of tests, written like a segment’s rule, and all of them have to hold. A unit outside the targeting is off-target: it gets no arm. Off-target is not the control arm.

If a targeting test cannot be read, the unit is off-target. Narrowing targeting after the experiment starts changes who is measured.

A unit with no arm

A unit that is off-target, or that has no id for the experiment’s unit, is in no arm:

• “they are in experiment · arm” does not hold, for any arm;
• “they are not in experiment · arm” holds, for every arm.

So a rule written as “not in variant” also catches off-target units and units with no id. To target only the control group, test “in control”.

Exposures

Reading an arm assigns the unit. The unit is exposed to the experiment only when the rule that read the arm is the one that wins, because only that rule’s outcome reached your site. An arm read by a rule that did not win is not an exposure. A feature with the operational purpose assigns arms and never exposes anyone.

Each response from the decide API lists the request’s exposures in exposures: one entry per experiment, with experiment and arm. A unit with no arm is reported with a reserved arm name, such as __off_target__ for a unit outside the targeting or __no_unit__ for a request with no device id or nobody signed in.

EdgeKit records nothing about exposures for you and computes no results. To measure an experiment, send each exposure as an experiment_exposure event, schema iglu:cloud.edgekit/experiment_exposure/jsonschema/1-0-0, with your tracker (@edgekitcloud/analytics-browser), and analyse those events beside the rest of your events. The browser client’s and the server client’s decide results do not include exposures. Read them from the decide API response.

The Experiments screen

Revenue › Decision Engine › Experiments lists every experiment in the draft:

• Experiment: the name.
• Arms: how many arms the traffic splits into.
• Unit: per device or per user.
• State: Live when the deployed document holds it exactly as shown, Draft when it changed since the last deploy, Never published when the deployed document does not hold it. not in the next deploy marks an experiment left out.
• Read by: the features whose rules test an arm, and the segments that do. nothing reads the arm means the experiment splits traffic and no decision differs.
• Targeting: who is eligible, or everyone.

Narrow the list by State, Unit and Read, search by name or arm, and sort by name, arms or read by.

Create an experiment

1. In Revenue › Decision Engine › Experiments, select New experiment.
2. Under Identity, enter a Name. Rules name the experiment by it. See names.
3. Leave Salt following the name, or type your own. The hash reads the salt, so pick it once: changing it later re-rolls every unit.
4. Choose the Unit: per device or per user.
5. Leave Next deploy at Left out until the experiment is ready, or set it to Included. A rule that names it cannot be deployed while it is left out.
6. Under The split, name each arm and give it buckets. The weights must total exactly 1,000. Even the split divides the 1,000 buckets evenly, and any remainder goes to the first arm, so three arms read 334, 333 and 333.
7. Under Targeting, add tests if only some users should be in the experiment. With none, everyone is in it.
8. Select Create experiment. It stays disabled while a check blocks it, such as weights that do not total 1,000, an arm with no name, two arms with one name, or an arm named with one of the reserved names for no arm.
9. In a feature’s canvas, add a rule that tests an arm and returns that arm’s outcome. See EdgeKit Paywall.
10. Stage and deploy. See Stage and deploy a change.

The split starts with two arms, control and variant, at 500 each. An arm with no buckets is never assigned, so a rule naming it matches nobody. The screen also flags an experiment with fewer than two arms, with no arm called control, or that nothing reads.

Change, stop or delete an experiment

Open the experiment from Revenue › Decision Engine › Experiments. Read by lists the rules and segments that test its arms.

Every change is a draft edit and reaches your sites at the next deploy. There is no switch that stops an experiment at once. To stop one, change the rules that test its arms, then leave the experiment out or delete it, and deploy. While a rule still names the experiment, staging fails as a missing reference.

To delete it, under Delete this experiment, select Delete experiment, then Delete for good.