Quick Summary
- Ran 47 A/B tests over 8 months
- These 9 changes increased replies by 340%
- Complete testing framework included
- Statistical significance explained
Small changes = massive reply rate improvements.
I ran 47 A/B tests on cold emails. These 9 changes increased reply rates by 340%.
The 9 Winning Changes
Test 1: Subject Line Length
Version A: "Quick question" (2 words)
Version B: "Quick question about [Company] sales process" (7 words)
Winner: Version A (+47% reply rate)
Learning: Shorter = better. Under 4 words wins.
Test 2: Personalization Depth
Version A: No personalization
Version B: "Saw you just raised Series B"
Winner: Version B (+189% reply rate)
Learning: One specific personal detail = huge impact
Test 3: Email Length
Version A: 150 words
Version B: 65 words
Winner: Version B (+72% reply rate)
Learning: Under 75 words performs best
Test 4: CTA Type
Version A: "Book a call here: [Calendly]"
Version B: "Worth a quick chat?"
Winner: Version B (+94% reply rate)
Learning: Ask permission before sharing calendar
Test 5: Questions vs Statements
Version A: "We help SaaS companies book meetings."
Version B: "Are you using cold email to book meetings?"
Winner: Version B (+83% reply rate)
Learning: Questions engage better than statements
Test 6: Social Proof Placement
Version A: Social proof in paragraph 1
Version B: Social proof in paragraph 2
Winner: Version B (+34% reply rate)
Learning: Lead with question, social proof second
Test 7: P.S. vs No P.S.
Version A: No P.S.
Version B: "P.S. Here's our ROI calculator: [link]"
Winner: Version B (+41% reply rate)
Learning: P.S. with value gets read
Test 8: Signature Style
Version A: Full company signature with logo
Version B: Just name
Winner: Version B (+28% reply rate)
Learning: Simple signature = more human
Test 9: Send Time
Version A: 9 AM send time
Version B: 6:30 AM send time
Winner: Version B (+52% reply rate)
Learning: Early morning = top of inbox
Combined Impact
Before optimization: 3.8% reply rate
After all 9 changes: 16.7% reply rate
Improvement: 340%
How to Run Your Own Tests
Step 1: Choose One Variable
Test ONE thing at a time:
- Subject line
- Email length
- CTA type
- Personalization
- Send time
Step 2: Split Traffic 50/50
Send Version A to 500 people, Version B to 500 people.
Step 3: Wait for Statistical Significance
Need at least 100 responses to trust results.
Step 4: Implement Winner
Use winning version, then test next variable.
Tools for A/B Testing
- Instantly: Built-in A/B testing ($97/month)
- Smartlead: Advanced split testing ($94/month)
- Lemlist: Campaign variants ($59/month)
For complete cold email optimization, see our 16% reply rate framework.
— Muhammad
AI Agenix