HazelJS Testing Package

npm downloads

@hazeljs/testing is the Agent OS testing DSL for HazelJS — describeAgent on top of @hazeljs/eval. Use it to regression-test agents in Jest/Vitest with latency, cost, tool, and output assertions.

Quick Reference

  • Purpose: Write agent regression suites with describeAgent / runAgentSuite, then assert tools, latency, cost, and output shape.
  • When to use: After you have a working Agent and want CI checks that feel like unit tests — not ad-hoc prompt scripts.
  • Key concepts: describeAgent → test cases → runAgentSuite / bindAgentSuite → assertAgentResult / expectTools.
  • Dependencies: Peer @hazeljs/eval. Wire your own runtime.execute (or mock) into the suite runner.
  • Common patterns: One suite per agent → map run(input) to AgentRuntime.execute → fail CI when failed > 0.
  • Common mistakes: Expecting describeAgent to replace Jest globals (it does not — call runAgentSuite inside it, or use bindAgentSuite); omitting tool/cost thresholds so flaky LLM runs always pass.

Installation

npm install @hazeljs/testing @hazeljs/eval

Usage

import {
  describeAgent,
  runAgentSuite,
  assertAgentResult,
  expectTools,
} from '@hazeljs/testing';

const suite = describeAgent('Support Agent', ({ test }) => {
  test('Refund', async ({ run }) => {
    const result = await run('I want a refund for order 123');
    expectTools(result, ['lookup-order']);
    assertAgentResult(result, {
      maxLatencyMs: 5000,
      maxCostUsd: 0.05,
      outputIncludes: 'refund',
    });
  });

  test('Shipping Delay', async ({ run }) => {
    const result = await run('Where is my package?');
    assertAgentResult(result, { maxLatencyMs: 5000 });
  });
});

it('support agent regression', async () => {
  const { failed, errors } = await runAgentSuite(suite, {
    agentName: 'support-agent',
    run: async (input) => {
      const out = await runtime.execute('support-agent', input);
      return {
        output: out.response ?? '',
        durationMs: out.duration,
        toolCalls: out.steps
          .filter((s) => s.action?.toolName)
          .map((s) => s.action!.toolName!),
      };
    },
  });
  expect(failed).toBe(0);
  expect(errors).toEqual([]);
});

CI helpers

import { runAgentGolden, reportAgentCi } from '@hazeljs/testing';

const result = await runAgentGolden(dataset, ctx, {
  maxLatencyMs: 8000,
  maxCostUsd: 0.1,
  minAverageScore: 0.8,
});
reportAgentCi(result, { exitOnFail: true });

Jest / Vitest

Use bindAgentSuite(suite, () => ctx) to expand each case into a native it(), or call runAgentSuite inside a single test as shown above.

API overview

ExportRole
describeAgentDefine a named suite of agent cases
runAgentSuiteExecute a suite against your run adapter
bindAgentSuiteExpand cases into Jest/Vitest it() blocks
assertAgentResultAssert latency, cost, and output substrings
expectToolsAssert expected tool names were called
runAgentGolden / reportAgentCiGolden-dataset CI helpers (via @hazeljs/eval)