---
source: CrewKit documentation
url: https://crewkit.io/docs/experiments.md
title: "Experiments"
description: "A/B test agent configurations to find what works best."
---


Experiments let you A/B test different agent configurations to measure which version performs better. Change a prompt, split traffic, and let data decide.

> **Note:** experiments are created and managed from the dashboard today. The `crewkit experiments` commands shown on this page are the planned CLI surface and have not shipped yet.

---

## Why experiment

Agent prompts are hard to get right. A change that seems better might:

- Increase token usage without improving quality
- Work well for one task type but worse for others
- Help senior developers but confuse junior ones

Experiments give you statistical confidence before rolling out changes.

---

## Experiment lifecycle

### 1. Create

Define what you're testing:

{/* crewkit-docs-check: ignore — planned CLI surface, not shipped yet */}
```bash
crewkit experiments create rails-expert
```

Or through the dashboard: **Experiments** > **New Experiment**.

Specify:
- **Agent** — Which agent to test
- **Control** — The current version
- **Variant** — Your proposed changes
- **Traffic split** — Percentage of sessions using the variant (default: 50%)

### 2. Run

Sessions are automatically assigned to control or variant based on the traffic split. crewkit tracks metrics for both groups:

- Session count
- Success rate
- Average cost
- Average turns
- Token usage

### 3. Measure

View results in the dashboard or CLI:

{/* crewkit-docs-check: ignore — planned CLI surface, not shipped yet */}
```bash
crewkit experiments metrics swift-amber-falcon
```

crewkit calculates statistical significance (p-value) so you know when you have enough data to make a decision.

### 4. Decide

When results are significant:

{/* crewkit-docs-check: ignore — planned CLI surface, not shipped yet */}
```bash
crewkit experiments deploy swift-amber-falcon
```

This promotes the winning variant to the project or organization level. The experiment is archived.

---

## Metrics tracked

| Metric | Description |
|--------|-------------|
| `session_count` | Number of sessions per variant |
| `success_rate` | Percentage of successful sessions |
| `avg_cost` | Mean cost per session |
| `avg_turns` | Mean conversation turns |
| `p95_cost` | 95th percentile cost |
| `p_value` | Statistical significance |

---

## Managing experiments

{/* crewkit-docs-check: ignore — planned CLI surface, not shipped yet */}
```bash
crewkit experiments list                        # List all experiments
crewkit experiments show swift-amber-falcon     # View details
crewkit experiments metrics swift-amber-falcon  # View metrics
crewkit experiments deploy swift-amber-falcon   # Deploy winner
```

---

## Next steps

- [Inheritance](/docs/inheritance)
- [Analytics](/docs/analytics)