An A/B test that actually improves results
Most 'tests' change everything at once (a new subject + a new CTA + a new offer) and conclude 'something works better' — but you don't know what. Or a test on 20 emails (statistically meaningless). Or testing opens, which don't translate into replies.
An A/B test in 5 steps:
- CHOOSE ONE VARIABLE: subject / opener / CTA / offer (never 2 at once),
- PREPARE 2-3 VARIANTS (they differ only in that variable),
- SEND to similar groups (the same list, random split — otherwise you compare apples to oranges),
- WAIT for replies (sending to 30-50 unique per variant gives a first signal; the full picture after 100+),
- CHOOSE the winner by REPLY RATE (not opens) and deploy. What to test in order: opener (the biggest impact on replies), subject, CTA, offer (last — because the offer changes the campaign's value). Rules: one test at a time, a sufficient sample (min 50 unique per variant), decisions based on replies not opens, adopting the WINNER is the deployment, not a test repeat. AI: generates variable variants (3 subjects, 3 openers), makes sure they differ only in that variable, proposes which variable to test next.
You test the OPENER (only the opener, the rest identical): A: 'You're hiring 3 salespeople — usually then leads get lost in spreadsheets' B: 'Saw your new branch in Gdansk' C: 'I'm writing because I see you're growing fast'. After 50 sent per variant: A 8% reply, B 6%, C 3%. You deploy A. Next test: subject with opener A (A1: 'An idea to recover leads' vs A2: '[Company] + new branch'). Every test answers one question.
You are an A/B testing expert. Context: My campaign: [PRODUCT + CURRENT EMAIL]. I want to improve the reply rate. Previous tests: [IF ANY]. Task: Design the next A/B test: 1. Which variable to test now (opener/subject/CTA/offer) + why 2. 3 variants (they differ ONLY in that variable) 3. Sample size (min unique per variant) + duration 4. The decision metric (reply rate) + the deployment threshold 5. The next test after this one (what I'll check) Format: 5 sections.
Variations & alternative scenarios
Low-volume version: 2 variants, a longer test period, a clear threshold.
Scale version: sequential tests, variants per segment.
Practical exercise
Pick 1 variable to test in your campaign. Write 2-3 variants. Set the sample (min 50 unique per variant) and the metric (reply rate). Write down what you'll deploy if the A/B variant wins.