HazelJS Benchmark Package

npm downloads

@hazeljs/benchmark provides Agent OS benchmark helpers — run comparable case sets, summarize score/duration/pass-rate, and compare a candidate run against a baseline to catch regressions in CI.

Quick Reference

  • Purpose: Benchmark agent (or pipeline) quality across commits/labels and fail CI when score regressions appear.
  • When to use: After you have golden cases or scored runs and want commit-to-commit comparison — complementary to @hazeljs/testing (describeAgent) and @hazeljs/eval.
  • Key concepts: runBenchmark → summarizeBenchmarkRun → compareBenchmarkRuns (regressions / improvements).
  • Dependencies: None required at runtime beyond Node. Works with any scoring function you provide.
  • Common patterns: Store a baseline JSON from main, run candidates on PRs, gate on comparison.regressions.length === 0.
  • Common mistakes: Comparing runs with different case IDs (cases must share ids); treating raw LLM variance as a hard gate without a threshold (use regressionThreshold).

Installation

npm install @hazeljs/benchmark

Quick start

import {
  runBenchmark,
  summarizeBenchmarkRun,
  compareBenchmarkRuns,
} from '@hazeljs/benchmark';

const baseline = summarizeBenchmarkRun('baseline', [
  { id: 'refund', score: 0.9, durationMs: 300, passed: true },
  { id: 'shipping', score: 0.8, durationMs: 220, passed: true },
]);

const candidate = await runBenchmark({
  label: 'candidate',
  commit: process.env.GIT_SHA,
  cases: [
    { id: 'refund', input: 'I want a refund' },
    { id: 'shipping', input: 'Where is my package?' },
  ],
  run: async (input, id) => {
    const result = await myAgentRunner(input);
    return {
      score: result.score,
      durationMs: result.durationMs,
      passed: result.score >= 0.7,
      costUsd: result.costUsd,
      error: result.error,
    };
  },
});

const comparison = compareBenchmarkRuns(baseline, candidate, 0.05);
if (comparison.regressions.length > 0) {
  console.log('Regressions found:', comparison.regressions);
  process.exitCode = 1;
}

API overview

ExportRole
runBenchmarkExecute cases with your scorer and return a BenchmarkRun
summarizeBenchmarkRunAggregate cases into average score, duration, and pass rate
compareBenchmarkRunsDiff baseline vs candidate; list regressions and improvements

runBenchmark

await runBenchmark({
  label: string;
  commit?: string;
  cases: Array<{ id: string; input: string }>;
  run: (input: string, id: string) => Promise<{
    score: number;
    durationMs: number;
    costUsd?: number;
    passed?: boolean; // defaults to score >= 0.7
    error?: string;
  }>;
});

Thrown errors from run become failed cases (score: 0, passed: false) instead of aborting the suite.

compareBenchmarkRuns

compareBenchmarkRuns(baseline, candidate, regressionThreshold = 0.05);

Returns scoreDelta, durationDeltaMs, passRateDelta, plus regressions / improvements for shared case IDs.

CLI

The Hazel CLI exposes hazel benchmark for Agent OS workflows. See @hazeljs/cli and the Agent OS guide.