Experiments API
A/B test agent configurations.
Experiments let you compare two versions of an agent configuration and measure which performs better.
All endpoints are organization-scoped: /:org_id/experiments.
List experiments
GET /:org_id/experimentsCreate experiment
POST /:org_id/experiments
Content-Type: application/json
{
"name": "Rails expert v2 test",
"resource_id": "resource_abc123",
"control_version_hash": "hash_control...",
"variant_version_hash": "hash_variant...",
"traffic_split": 50
}Show experiment
GET /:org_id/experiments/:idUpdate experiment
PATCH /:org_id/experiments/:id
Content-Type: application/json
{
"traffic_split": 70
}Experiment lifecycle
Start
POST /:org_id/experiments/:id/startBegins routing traffic to control and variant groups.
Stop
POST /:org_id/experiments/:id/stopStops the experiment. No new sessions are assigned to variants.
Deploy
POST /:org_id/experiments/:id/deploy
Content-Type: application/json
{
"variant": "variant"
}Promotes the winning variant to production. The experiment is archived.
Archive
POST /:org_id/experiments/:id/archiveArchive a completed or stopped experiment.
Metrics
GET /:org_id/experiments/:id/metricsReturns metrics for both control and variant groups:
{
"control": {
"total_sessions": 100,
"success_rate": 85.5,
"avg_cost": 0.0234,
"avg_turns": 12.5,
"p95_cost": 0.0567
},
"variant": {
"total_sessions": 98,
"success_rate": 89.2,
"avg_cost": 0.0198,
"avg_turns": 10.8,
"p95_cost": 0.0489
},
"significance": {
"p_value": 0.05,
"significant": false,
"confidence": 0.95
}
}Delete experiment
DELETE /:org_id/experiments/:id