A/B Testing

An A/B test tries different versions of a campaign on a small part of your audience first, then sends the version that did best to everyone else. Use it to find out which subject line gets more people to open, or which content gets more clicks.

How it works

1. You write up to four variants, A to D. Variant A is the campaign itself; B, C, and D can each have their own subject, preview text, and content.
2. When the campaign starts sending, Postak picks a random test group (for example 20 % of the recipients) and shares it evenly among the variants.
3. The test group gets its emails. Everyone else, the remainder, waits.
4. After a wait you choose, Postak picks the winner: the variant with the best open rate or click rate. Or you pick it yourself.
5. The remainder gets the winning variant.

Everyone gets exactly one email. The campaign shows Sending until the remainder has been sent, then Sent.

Set up a test

In the campaign editor, write the campaign as usual: its subject, preview text, and content are variant A. Then, under A/B test, tick Test variants before sending to everyone. Variant B appears.

For each variant:

• Subject: required. This is often the only thing you change.
• Preview text: leave it empty to use variant A's.
• Use different content: tick it to give the variant its own content. The box starts with a copy of variant A's content, so you can change just a sentence or a button. Untick it to use A's content.
• Remove variant: removes the variant; the later ones move up a letter. A test always keeps at least variant B. To stop testing, untick Test variants before sending to everyone.

Select Add variant for C and D. You can test up to 4 variants.

Then choose:

• Test group: the share of recipients who get the test, from 1 to 50 %. The help text shows the split, for example "About 10 % per variant. The other 80 % get the winner."
• Pick the winner by:
  • Open rate: the variant opened by the largest share of the people it was sent to. Best for comparing subject lines and preview text.
  • Click rate: the variant whose links were clicked by the largest share. Best for comparing content. It needs links in the content.
  • I'll pick it myself: Postak waits until you choose the winner in the report.
• Wait before picking (open and click rate only): how long to wait, in minutes, hours, or days. The wait starts when the last test email has gone out. It's at least 15 minutes (your administrator may set a different minimum) and at most 7 days. Longer waits give more people time to open the email, so the result is more reliable.

Open rate and click rate need Track opens and clicks. With tracking off, or for a plain-text email without HTML, only I'll pick it myself is available. If you turn tracking off while an automatic choice is selected, the editor switches to I'll pick it myself and tells you.

Select Save draft. The test settings are saved with the campaign.

Preview and test each variant

• With the test on, Preview variant above the preview chooses which variant you see.
• In Send test, choose the Variant to send. The test email has that variant's subject (after [Test]) and content.

Send or schedule

Send now and Schedule work as for any campaign. The confirmation adds how the test works, for example "First 20 % get the test variants. The rest get the winner."

Postak checks the test before anything is sent: it refuses to send an automatic test when tracking is off, or an MJML variant that can't be compiled.

While the test runs

The campaign's report has an A/B test section that shows where the test is:

• Sending to the test group: the test emails are going out.
• Test finished. The winner is picked by open rate at a time: Postak picks the winner then.
• Waiting for you to pick the winner: for tests you decide yourself.
• Variant B won: the winner has been chosen, and the remainder gets it.

The table shows each variant's recipients, sent, opened, and clicked emails with the open and click rates, and a Remainder line below it.

Pick the winner yourself

While there is no winner yet, each variant has a Pick … as winner button. Select it and confirm ("Send variant B to the remaining 400 recipients?"). The remainder starts sending straight away.

You can do this for any test, including one that would pick the winner automatically, and even before the test group has finished sending. Once a winner has been chosen, by you or by Postak, it can't be changed.

How Postak picks the winner

• The rate is counted against the emails each variant has sent so far.
• Open rate: the highest open rate wins. If two variants are level, the higher click rate decides, then the earlier letter.
• Click rate: the highest click rate wins. If two variants are level, the higher open rate decides, then the earlier letter.
• If the content has no links, every click rate is 0 and variant A wins a click-rate test.

Good to know

• The test group is chosen at random each time a campaign is sent. With a small list, Postak still gives each variant at least one person; if that covers the whole audience, there is no remainder and the campaign becomes Sent when the test emails are done.
• If a test email hits a temporary problem and is retried, the wait starts later, after the retry.
• People who unsubscribe during the wait don't get the remaining email.
• The remainder gets the winner's subject, preview text, and content. {{ campaign.subject }} is the subject of the variant each person gets.
• The remainder uses the template as it is when it's sent. If you change the template during the wait, the remainder gets the changed version. See Edit or delete a template.
• The web view shows variant A until there is a winner, then the winner.
• Duplicate copies the test settings and variants, but not the results.
• Once a campaign has started sending, its test settings can't change.
• An open rate is only an estimate (see A note on accuracy). Small differences on a small test group may be chance; test on a few hundred people per variant if you can.
• Scripts can set up tests and pick winners through the API.