Skip to content
MCPSTAMP

AnswerLive data

Can Claude calculate A/B test sample size?

It knows the formula, but the exact number is easy to get wrong. Sample size depends on z-scores, pooled or unpooled variance and one or two sided tests, and online calculators don't agree on those choices, so a figure from memory can be off by thousands of users. Connect the free A/B Test Sample Size Calculator server and Claude gets the exact count per arm, says which method it used, and turns traffic into days to run the test.

Connect free, no token A/B test sample size calculator MCP server

What Claude does alone, and what needs the server

Claude on its own

  • Explains power, significance and minimum detectable effect.
  • Gives a rough sample size for a common case.

With the A/B Test Sample Size Calculator server

  • The exact sample size per arm for any baseline, lift, alpha, power and number of arms.
  • Test duration from daily traffic, rounded to full weeks.
  • Significance tests, Fisher exact test and the cost of peeking.

A real call to the live server

We sent this to the hosted server on 2026-11-03. The answer below is the server's own text, not a mock-up. Live values will differ when you ask.

You ask

My checkout converts at 4%. How many users do I need to detect a 10% relative lift?

Tool call

calculate_sample_size {"baseline_rate":0.04,"mde":0.1}

Server answer

To detect a change from 4.00% to 4.40% (10.0% relative MDE) with 80% power at alpha 0.05 two-sided, you need 39,475 users in control and in each variant: 78,950 in total across 2 arms. Evan Miller's calculator gives 38,210 per arm for the same inputs (it uses the baseline variance in both terms and always two-sided alpha).

You ask

We get 3,000 visitors a day into the test. How long will it take?

Tool call

estimate_test_duration {"daily_visitors":3000,"baseline_rate":0.04,"mde":0.1}

Server answer

39,475 users per arm x 2 arms = 78,950 users. At 3,000 test visitors a day that takes 27 days; run 4 full weeks (28 days) so every weekday is covered equally.

Connect it in 1 minute

One URL, no token, no account. In Claude.ai or Claude Desktop open Settings, then Connectors, then Add custom connector, and paste the URL.

Connector URL

https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp

Claude Code

claude mcp add --transport http ab-test-sample-size-calculator https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp

Cursor (~/.cursor/mcp.json)

{
  "mcpServers": {
    "ab-test-sample-size-calculator": {
      "url": "https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp"
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "ab-test-sample-size-calculator": {
      "type": "http",
      "url": "https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp"
    }
  }
}

All 5 tools, sources and more prompts: A/B test sample size calculator MCP server · Abwex

Questions

Can Claude calculate A/B test sample size?
It knows the formula, but the exact number is easy to get wrong. Sample size depends on z-scores, pooled or unpooled variance and one or two sided tests, and online calculators don't agree on those choices, so a figure from memory can be off by thousands of users. Connect the free A/B Test Sample Size Calculator server and Claude gets the exact count per arm, says which method it used, and turns traffic into days to run the test.
Why does Evan Miller's calculator give a different number?
It uses the baseline variance in both terms. The server uses pooled variance under the null and unpooled under the alternative, and quotes both numbers so you can compare.
Can I stop the test early if it looks significant?
Checking results again and again raises the false positive rate. The check_peeking_risk tool shows by how much.
Is the server free?
Yes. The A/B Test Sample Size Calculator server is free at its hosted URL with no token, key or account. The source is open on GitHub.

More questions people ask Claude

All answers · All free servers