Ori Eval

Ori Eval answers “which model is best for what I’m building?” with evidence instead of vibes. Ask the question in plain words and the agent finds test material in your repo, writes the eval, runs your candidates, and comes back with a ranked answer. Under the hood an eval is a *.eval.ts file — you can write one by hand whenever you want the control, but you never have to start there.

Why run evals through Ori

The hardest part of evals is keeping the test bench stable. Ori is a harness, not just a skill: it pins the model and settings at a level prompts can’t, so a score change means your agent changed, not the environment. And because Ori routes through OpenRouter, a bakeoff can cover every major lab’s models — not just the one vendor your usual harness supports.

Before you start

Install the CLI:

curl -fsSL https://openrouter.ai/labs/ori/install.sh | sh

ori

An eval calls a real model, so a credential is required — the browser login covers it, or export OPENROUTER_API_KEY directly. Running evals also needs Bun; if it is missing, ori eval offers to install it on an interactive terminal.

Your project does not need to be TypeScript, and TypeScript never needs a separate install: the eval file tests the agent, not your codebase.

Start from a question

From your project directory, start a session and ask:

cd my-project
ori

What's the best model for my support agent?

You do not need to know eval methodology or model names. The agent:

Scans your repo for test material — existing prompts, tool definitions, data files (.jsonl, .csv, chat logs), and gold answers. Tests in any language count: pytest files and Go tests get mined for cases, even though the eval itself is always TypeScript.
Asks what matters — a question or two like “accuracy first, or speed and cost?” — only when the answer changes the eval.
Writes and runs the eval — scaffolds the *.eval.ts, seeds candidate models from OpenRouter’s live catalog under your price cap, and runs them.
Recommends one model, with the scores, latency, and cost that back it up.

Every artifact stays in your repo as plain code: re-run it when a new model drops, tighten the criteria, or put it in CI.

Under the hood: the eval file

When you want control — or want to understand what the agent built — here is the whole surface. An eval reads like a normal bun test:

recommends.eval.ts

import { test } from 'bun:test';
import { setupAgent } from 'ori/eval';

const agent = setupAgent();

test('recommends restaurants using the search tool', async () => {
  const run = await agent.run('Where should I eat dinner in Lisbon?');

  run.tool('search').toBeCalled();
  run.tool('delete_file').toNotBeCalled();
  run.toComplete();
});

setupAgent() with no arguments is your workspace’s own resolved harness and model — the same agent you already run. Then:

ori eval

Ori discovers every *.eval.ts in the directory, boots an ephemeral runtime, and hands the files to bun test. The exit code mirrors bun test, so a failing eval fails CI.

Compare models

Seed candidates from OpenRouter’s live catalog instead of hand-listing slugs, cap the price, and run the same eval against each:

model.eval.ts

import { test } from 'bun:test';
import { candidateModels, setupAgent } from 'ori/eval';

const candidates = await candidateModels({
  limit: 5,
  maxPromptPrice: 0.000005,
});

for (const model of candidates) {
  test(`handles a refund request on ${model}`, async () => {
    const run = await setupAgent({ model }).run(
      'A customer wants a refund for order #1234. What do you do?',
    );
    run.tool('lookup_order').toBeCalled();
    run.toComplete();
  });
}

Grade open-ended answers

When there is no single right string, grade with an LLM judge. The judge defaults to a dedicated agent on a strong grading model, so scoring stays independent of the model under test:

quality.eval.ts

import { test } from 'bun:test';
import { setupAgent, setupJudge } from 'ori/eval';

const agent = setupAgent();
const judge = setupJudge({ minScore: 0.8 });

test('gives an accurate, grounded answer', async () => {
  const run = await agent.run('What is our refund policy for digital goods?');
  await judge.autoEvals({
    criteria: 'Cites the 14-day window and does not invent exceptions.',
    run,
  });
});

Eval against your own data

Real usage data beats synthetic prompts. Loop chat logs or Q&A pairs with test.each:

import supportPairs from './support-pairs.json';

test.each(supportPairs)(
  'answers: $question',
  async ({ question, mustMention }) => {
    const run = await agent.run(question);
    run.toMention(mustMention);
    run.toComplete();
  },
);

Run it in CI

Evals spend model calls, so put them in a credentialed, opt-in job — not your default unit-test run — with the key as a secret. A failing eval fails the build, so a worse agent stops the ship instead of your users. ori eval --list --allow-no-key needs no key and cheaply asserts evals exist.

Overview

Models & Routing

Features

Ori

Privacy

Best Practices

Community

Why run evals through Ori

Before you start

Start from a question

Under the hood: the eval file

Compare models

Grade open-ended answers

Eval against your own data

Run it in CI

​Why run evals through Ori

​Before you start

​Start from a question

​Under the hood: the eval file

​Compare models

​Grade open-ended answers

​Eval against your own data

​Run it in CI

Why run evals through Ori

Before you start

Start from a question

Under the hood: the eval file

Compare models

Grade open-ended answers

Eval against your own data

Run it in CI