Hey everyone! May was huge at 2027.dev — we're making the platform programmable, so you can drive your Agent Experience (AX) evals straight from your API and your CI, no dashboard required.
A public REST API
Everything you can do from the dashboard you can now do over HTTP. Create an API key in Settings, then list your prompts, trigger runs, poll their status, and pull back full evaluation data, including all friction points and agent traces, all as JSON. One key authenticates both the REST API and our MCP server. Keys are scoped to your org, shown once, and revocable any time.
Prompts with variables
One scenario rarely covers every case. You can now write prompts with {{variables}} — "Deploy a project to {{region}}", "Set up auth for a {{framework}} app" — and fill in the values when you run them. A single prompt now stretches across the matrix of environments, frameworks, and user states your agents actually hit in the wild. You can also override login hints per prompt, so evals that need an authenticated session start from the right place.
A self-hostable GitHub Action
Last month we shipped the GitHub App that comments AX results on your docs PRs. This month we added an alternative for teams that would rather own the pipeline: a self-hostable GitHub Action (team2027/evals-action) you drop into any workflow. It authenticates with an API key, polls the run, and posts the grade — now with a direct link to the agent's full trace — as a sticky comment and commit status from your own runner. The Setup page walks you through it.
Sharper benchmarking
Comparisons should reflect typical behavior, not a single lucky (or unlucky) run. Leaderboard and benchmark rankings now use a weekly average, with a hover breakdown of the runs behind each cell and a ±1σ band so you can see how consistent a result really is. We also retired the raw tool-call count in favor of an interruptions metric — a far better signal for where an agent got stuck — and added an AI-adapted error count to traces, showing how often an agent hit an error and recovered on its own.
Coming soon
- One-click fix prompts. For each friction point we catch, a ready-made prompt you can paste straight into your agent to fix it — landing now.
- Multi-model comparison. See how every model does on the same prompt side by side — time, tokens, errors, interruptions — so you know which agents your tool already serves well and which it doesn't.
Thoughts?
We're always here if you have any thoughts on what we can do better. Reach out at mika@2027.dev or DM me on X anytime — @heymikasagi.