Experiments

Experiments let you A/B test different agent configurations to measure which version performs better. Change a prompt, split traffic, and let data decide.

Note: experiments are created and managed from the dashboard today. The crewkit experiments commands shown on this page are the planned CLI surface and have not shipped yet.

Why experiment

Agent prompts are hard to get right. A change that seems better might:

• Increase token usage without improving quality
• Work well for one task type but worse for others
• Help senior developers but confuse junior ones

Experiments give you statistical confidence before rolling out changes.

Experiment lifecycle

1. Create

Define what you're testing:

crewkit experiments create rails-expert

Or through the dashboard: Experiments > New Experiment.

Specify:

• Agent — Which agent to test
• Control — The current version
• Variant — Your proposed changes
• Traffic split — Percentage of sessions using the variant (default: 50%)

2. Run

Sessions are automatically assigned to control or variant based on the traffic split. crewkit tracks metrics for both groups:

• Session count
• Success rate
• Average cost
• Average turns
• Token usage

3. Measure

View results in the dashboard or CLI: