The HazelJS Production Ops Stack: Config Server, Self-Healing, and Predictive Scaling

Git-backed configuration, automatic recovery, and proactive Kubernetes HPA — a deep dive into @hazeljs/config-server, @hazeljs/self-healing, and @hazeljs/predictive-scaling, and how they compose into one operations stack.

Circuit breakers stop a failing call. They do not roll back a bad config, restart a stuck pod, or have extra replicas waiting before Black Friday traffic arrives. That gap — between “the request failed” and “the system recovered” — is where on-call lives.

HazelJS already shipped the first half of that story with @hazeljs/resilience, @hazeljs/gateway, and @hazeljs/discovery. This post is about the second half: shared Git-backed configuration, automatic recovery, and proactive Kubernetes HPA.

We are shipping three packages that compose into one operations stack:

Package	Job	Timing
`@hazeljs/config-server`	One Git repo of YAML/JSON for every service, with `{cipher}` secrets and HTTP refresh	Before and during a change
`@hazeljs/self-healing`	Diagnose repeating failures and recover: rollback, drain, pod restart, reactive HPA boost	After something breaks
`@hazeljs/predictive-scaling`	Forecast load ~30 minutes ahead and raise HPA `minReplicas` before the spike	Before traffic arrives

They share a Kubernetes scaling client, optional @hazeljs/ai diagnostics, and @hazeljs/ops-agent for Jira. You can adopt them independently. Together they are how a TypeScript microservice fleet stops treating production as a page-and-hope loop.

The problem these packages exist to solve

A typical Node fleet in 2026 still looks like this:

Each service owns a .env file. Staging and prod drift. A hotfix is a Slack paste of DATABASE_URL.
When a dependency dies, @CircuitBreaker opens. The pod keeps serving 503s until a human rolls the deployment.
HPA reacts to CPU after latency is already bad. Weekend traffic and product launches are tribal knowledge in a spreadsheet.

Spring Cloud Config, Kubernetes operators, and Prometheus adapters solved pieces of this in the JVM and Go worlds. The TypeScript backend stack mostly left you to wire Git, AES, HPA patches, and Slack yourself.

These three packages are that wiring — decorator-first, DI-native, and honest about what they do not do (no Java .jks keystores, no magic cluster-wide operator, no guarantee that a forecast is free).

How the three fit together

Think of a payments API on Kubernetes:

Config server is the source of truth for database.url, stripe.secret, and features.newAlgorithm. A Git push + POST /refresh updates every client that called ConfigClient.load() without a rebuild.
Self-healing watches method errors and latency. A bad config snapshot can roll back. A stuck process can drain in-flight work and PATCH the deployment. A latency cliff can temporarily raise HPA minReplicas.
Predictive scaling reads Prometheus http_requests_total, forecasts the next 30 minutes, and raises the same HPA target before the cliff — then scales down slowly when confidence drops.

Reactive boost and proactive forecast can fight each other if you point them at different HPAs. Point both at one client (adaptSelfHealingScalingClient / createOperationsStack) so one process owns minReplicas.

Git config repo ──► ConfigServer (HTTP :8888)
                         │
                         ▼
                   ConfigClient / @ConfigValue
                         │
              ┌──────────┴──────────┐
              │                     │
     Self-healing              Predictive scaler
     (diagnose + recover)      (forecast + HPA)
              │                     │
              └──────────┬──────────┘
                         ▼
              Kubernetes HPA minReplicas
              Slack / PagerDuty / Jira

Part 1 — `@hazeljs/config-server`: Spring Cloud Config for TypeScript

@hazeljs/config is the right tool for one process: load .env, validate a schema, inject ConfigService. It is the wrong tool the moment five services need the same payments.timeout and you want a PR, a history, and a refresh without five deploys.

@hazeljs/config-server is the shared Git pattern: clone a repo, overlay files by application name and profile, decrypt secrets, serve JSON over HTTP.

Overlay order (later wins)

application.yml — shared defaults
application-{profile}.yml — shared env overlay (prod, staging, …)
{application}.yml — service defaults
{application}-{profile}.yml — service + env
Extra files under searchPaths such as configs/{application}/{profile}

Formats: YAML, JSON, .properties, .env. Nested keys resolve with dotted getters: database.url.

A repo can look like:

application.yml
application-prod.yml
user-service.yml
user-service-prod.yml
configs/user-service/prod/extra.yml

profile may be a comma list (prod,cloud). The label is a Git branch or tag (main, release/2026-09). Pinning a label is how you roll config the same way you pin a container digest.

Run the server

npm install @hazeljs/config-server

The server process needs a git binary. HTTP clients do not.

import { ConfigServer } from '@hazeljs/config-server';

const server = new ConfigServer({
  git: {
    uri: 'https://github.com/org/config-repo',
    searchPaths: ['configs/{application}/{profile}'],
    defaultLabel: 'main',
  },
  encryption: {
    enabled: true,
    key: process.env.CONFIG_SERVER_ENCRYPT_KEY,
  },
  port: 8888,
  refreshInterval: 60_000,
});

await server.start();

HTTPS Git: git.username + git.password (a PAT). Local path or file:// works. Tests and air-gapped images can use nativePath instead of Git — production should not.

Module form for HazelJS apps:

import { EnableConfigServer, ConfigServerModule } from '@hazeljs/config-server';

@EnableConfigServer({
  git: { uri: process.env.CONFIG_GIT_URI!, defaultLabel: 'main' },
  encryption: { enabled: true, key: process.env.CONFIG_SERVER_ENCRYPT_KEY },
  profiles: ['dev', 'staging', 'prod'],
})
export class AppConfigServer {}

ConfigServerModule.forRoot({
  git: { uri: process.env.CONFIG_GIT_URI! },
  port: 8888,
});

CLI: hazel add config-server.

HTTP API

Method	Path	Purpose
GET	`/{application}/{profile}`	Merged environment
GET	`/{application}/{profile}/{label}`	Same, pinned to a Git ref
POST	`/refresh`	`git fetch` + checkout
POST	`/encrypt`	plaintext → `{cipher}v1:...`
POST	`/decrypt`	cipher → plaintext
GET	`/health`	liveness + current SHA
GET	`/audit`	clone / resolve / encrypt events

Response shape:

{
  "name": "user-service",
  "profiles": ["prod"],
  "label": "main",
  "version": "abc123...",
  "propertySources": [{ "name": "application.yml", "source": {} }],
  "config": { "database": { "url": "postgres://..." } }
}

version is the Git SHA. Treat it as the config equivalent of a container image id in incident notes.

Client, refresh, and `@ConfigValue`

import { ConfigClient, ConfigValue } from '@hazeljs/config-server';

const client = new ConfigClient({
  uri: 'http://config-server:8888',
  application: 'user-service',
  profiles: ['prod'],
  label: 'main',
  refreshInterval: 30_000,
});
await client.load();

client.get('database.url');
await client.refresh();

class AppConfig {
  @ConfigValue('database.url', { refresh: true })
  dbUrl!: string;

  @ConfigValue('features.newAlgorithm', { default: false, type: 'boolean' })
  useNewAlgorithm!: boolean;

  @ConfigValue('api.timeout', { type: 'number' })
  apiTimeout!: number;
}

refresh: true means a Git push + server POST /refresh (or the server’s refreshInterval) can update the field without restarting the Node process. That is the whole point of a config server.

In-process (no HTTP), pass server instead of uri — useful in tests.

Encryption: AES-256-GCM, not a Java keystore

Spring Cloud Config often used a .jks keystore. That is JVM-specific. This package uses AES-256-GCM and a {cipher}v1:... prefix.

Put the passphrase in CONFIG_SERVER_ENCRYPT_KEY or encryption.keyFile. Encrypt once:

const cipher = server.encrypt('my-db-password');
// database:
//   password: '{cipher}v1:...'

Values decrypt when the environment is served. Ciphertext in Git is fine. The key next to the ciphertext is not. Rotate the key the same way you rotate any other secret; old {cipher} values will not decrypt.

Config server pitfalls

Using @hazeljs/config alone when five services should share one repo.
Committing CONFIG_SERVER_ENCRYPT_KEY in the same repo as {cipher} values.
Expecting a Java keystore to work.
Refreshing secrets that the process already opened as long-lived connections (a new database.url does not reconnect Prisma for you — plan a drain or restart).
Treating nativePath as production. It is for tests and air-gapped images.

Use @hazeljs/config in the process. Use @hazeljs/config-server when the source of truth is Git across services.

Docs: Config Server · Config · npm @hazeljs/config-server

Part 2 — `@hazeljs/self-healing`: diagnose, then recover

@hazeljs/resilience answers: should this call proceed? Circuit breaker, retry, timeout, bulkhead, rate limit.

@hazeljs/self-healing answers: this already failed — what should the process or cluster do next?

That distinction matters. A circuit breaker that stays open forever is a quiet outage. Self-healing is the next step: restart the module, roll back config, clean memory, drain and bounce the pod, or temporarily raise HPA.

What shipped, by phase

Phase	What you get
1	Rule-based diagnosis + recovery primitives (`@SelfHealing`, `@SelfHeal`, `@MemoryGuard`, `createHealingCoordinator`)
2	LLM diagnosis via `@hazeljs/ai`, Kubernetes `pod-restart`, Slack / PagerDuty notifiers
3	Graceful drain, `hpa-boost` with auto-restore, `recordLatency` performance scaling, Jira via `@hazeljs/ops-agent`

Install:

npm install @hazeljs/self-healing @hazeljs/core
# optional
npm install @hazeljs/resilience @hazeljs/ai @hazeljs/ops-agent

Decorators

import {
  SelfHealing,
  SelfHeal,
  MemoryGuard,
  createSlackHealingNotifier,
} from '@hazeljs/self-healing';

@SelfHealing({
  enabled: true,
  strategies: ['auto-restart', 'config-rollback', 'pod-restart'],
  aiDiagnostics: true,
  notifyOn: ['critical-healing', 'auto-rollback', 'healing-failed', 'pod-restart'],
  notifications: createSlackHealingNotifier({ channel: '#incidents' }),
  kubernetes: {
    deployment: process.env.K8S_DEPLOYMENT!,
    namespace: process.env.K8S_NAMESPACE ?? 'default',
  },
})
export class AppModule {}

@MemoryGuard({ threshold: '500MB', action: 'memory-cleanup' })
export class PaymentService {
  @SelfHeal({ onError: 'diagnose-and-fix', maxAttempts: 3, fallback: 'chargeSafe' })
  async charge(): Promise<string> {
    return 'ok';
  }

  async chargeSafe(): Promise<string> {
    return 'safe-mode';
  }
}

@SelfHeal wraps the hot path. onError can be diagnose-and-fix, retry-only, or safe-mode-only. fallback is a method name on the same class — that is safe-mode.

CLI: hazel add self-healing.

Strategies

Strategy	When	Action
`auto-restart`	Dependency / timeout errors	Re-run `onModuleDestroy` + `onModuleInit`
`config-rollback`	Config errors	Restore last config snapshot
`memory-cleanup`	Memory pressure	`clearCache()` + `global.gc()` if exposed
`safe-mode`	Unrecoverable errors	Named fallback method
`pod-restart`	Cluster-level failures	Drain in-flight work, then PATCH deployment `restartedAt`
`hpa-boost`	Performance / load spikes	Temporarily raise HPA `minReplicas`, restore after cooldown

config-rollback is why config-server and self-healing belong in the same post. If the last good snapshot lives in Git, rollback is a known SHA — not a guess from a 3am .env.

Drain, HPA boost, Jira

Do not bounce a pod while checkout is mid-charge. Phase 3 drain waits up to drain.timeoutMs, then restarts.

import {
  createHealingCoordinator,
  createJiraHealingNotifier,
  createHealingNotifierChain,
  FetchKubernetesScalingClient,
} from '@hazeljs/self-healing';
import { createJiraTool } from '@hazeljs/ops-agent';

const healing = createHealingCoordinator({
  drain: { timeoutMs: 30_000 },
  performance: {
    enabled: true,
    autoScaleOnDegradation: true,
    thresholds: { criticalLatencyMs: 2000, sampleSize: 10 },
  },
  strategies: ['hpa-boost', 'pod-restart', 'config-rollback'],
  notifications: createHealingNotifierChain([
    createJiraHealingNotifier({ jira: createJiraTool(), project: 'OPS' }),
  ]),
  kubernetes: {
    deployment: 'payments-api',
    namespace: 'prod',
    drainBeforeRestart: true,
    hpa: {
      name: 'payments-hpa',
      client: new FetchKubernetesScalingClient(),
      boostMinReplicas: 4,
      restoreAfterMs: 300_000,
    },
  },
});

await healing.recordLatency('PaymentService.charge', durationMs);

recordLatency is the reactive half of scaling: if p95 stays above criticalLatencyMs for sampleSize samples, hpa-boost fires. Predictive scaling (Part 3) tries to make that boost unnecessary.

AI diagnostics

Rules catch timeouts and OOM. Ambiguous errors benefit from an LLM:

createHealingCoordinator({ aiDiagnostics: true });

createHealingCoordinator({
  aiDiagnostics: createHazelAIDiagnosticsProvider(aiService, { model: 'gpt-4o-mini' }),
});

createHealingCoordinator({
  aiDiagnostics: createAIDiagnosticsProvider({
    complete: async (messages) => myLlm.chat(messages),
  }),
});

Do not turn this on without a model budget and a human channel. Diagnostics should recommend a strategy; they should not encrypt a new production key.

Notifications

const notifications = createHealingNotifierChain([
  createSlackHealingNotifier({ channel: '#incidents' }), // SLACK_BOT_TOKEN
  createPagerDutyHealingNotifier({}), // PAGERDUTY_ROUTING_KEY
]);

Wire Jira when healing fails, not on every retry. Noise trains people to ignore the channel.

Self-healing pitfalls

Enabling pod-restart without a Kubernetes client (in-cluster FetchKubernetesRestartClient or an in-memory client in tests).
Skipping drain so in-flight requests die on SIGTERM.
hpa-boost without restoreAfterMs — you will pay for replicas until the next deploy.
AI diagnostics with no rate limit and no audit of prompts (they may contain request payloads).
Using self-healing instead of circuit breakers. Keep @hazeljs/resilience on the outbound call; heal the process when the pattern repeats.

Docs: Self-Healing · Resilience · Ops Agent

Part 3 — `@hazeljs/predictive-scaling`: replicas before the spike

HPA that only watches CPU is late. The request rate is already climbing; pods are still cold.

@hazeljs/predictive-scaling forecasts traffic (exponential smoothing + hour-of-week seasonality + event boosts) and raises HPA minReplicas ahead of the horizon (default 30m). Optional Prometheus poll. Optional LLM forecast hook. Optional bridge to the same K8s client self-healing uses.

Self-healing hpa-boost is reactive (latency already bad). This package is proactive. Use both; do not confuse them.

npm install @hazeljs/predictive-scaling @hazeljs/core
# optional
npm install @hazeljs/self-healing @hazeljs/ai

Decorators

import {
  PredictiveScaling,
  ScalePredict,
  ScaleOnEvent,
  adaptSelfHealingScalingClient,
} from '@hazeljs/predictive-scaling';
import { InMemoryKubernetesScalingClient } from '@hazeljs/self-healing';

@PredictiveScaling({
  model: 'time-series-forecast',
  metrics: ['requests', 'latency'],
  horizon: '30m',
  confidence: 0.85,
  costOptimization: true,
  hpa: {
    name: 'video-hpa',
    namespace: 'prod',
    client: adaptSelfHealingScalingClient(new InMemoryKubernetesScalingClient()),
    maxReplicas: 100,
  },
})
@ScaleOnEvent({
  events: ['product-launch', 'black-friday'],
  maxScale: 100,
  scaleFactor: 2,
})
export class AppModule {}

export class VideoStreamingService {
  @ScalePredict({
    triggers: ['weekend-pattern', 'viral-content'],
    scaleUp: { before: '15m', factor: 2 },
  })
  async streamVideo() {
    // Demand signals recorded automatically
  }
}

confidence: 0.85 is a cost gate. Below that, the scaler should not blow the replica cap. costOptimization: true prefers gradual scale-down so you do not thrash the cluster every time a forecast dips.

CLI: hazel add predictive-scaling.

Programmatic API

import { createPredictiveScaler, InMemoryScalingClient } from '@hazeljs/predictive-scaling';

const scaler = createPredictiveScaler({
  horizon: '30m',
  metrics: ['requests'],
  hpa: { name: 'api-hpa', namespace: 'prod', client: new InMemoryScalingClient() },
});

scaler.recordMetric('requests', 420);
scaler.start();

await scaler.triggerEvent('black-friday');

triggerEvent is for known launches. Spreadsheets and Slack “we’re going live at 10:00” become scaler.triggerEvent('product-launch') from a controller or cron.

Prometheus ingestion

In-process recordMetric is enough for demos. Production should scrape what you already scrape:

import { createPredictiveScaler, attachPrometheusCollector } from '@hazeljs/predictive-scaling';

const scaler = createPredictiveScaler({
  metrics: ['requests', 'latency', 'cpu'],
  hpa: { name: 'api-hpa', namespace: 'prod', client },
});

const prometheus = attachPrometheusCollector(scaler, {
  baseUrl: process.env.PROMETHEUS_URL ?? 'http://localhost:9090',
  pollIntervalMs: 60_000,
  queries: {
    requests: 'sum(rate(http_requests_total{service="api"}[5m]))',
    latency:
      'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le))',
    cpu: 'avg(rate(container_cpu_usage_seconds_total{pod=~"api-.*"}[5m]))',
  },
});

scaler.start();
prometheus.start();

If Prometheus is wrong, the forecast is wrong. Treat query labels (service="api") as part of the contract.

Predictive scaling pitfalls

Scaling without a confidence gate — cost blow-ups on noisy metrics.
Instant scale-down (costOptimization off) — replica churn and cold starts.
Using this package instead of circuit breakers for a failing dependency. Forecasts do not fix 500s.
Two controllers patching the same HPA (a custom operator and this scaler). Pick one owner.
Event boosts left on after the launch. triggerEvent is not a forever multiplier.

Docs: Predictive Scaling

Part 4 — One operations stack

The intended production shape is one HPA client, healing + forecasting + optional Prometheus:

import { createOperationsStack } from '@hazeljs/predictive-scaling';
import { InMemoryKubernetesScalingClient } from '@hazeljs/self-healing';

const client = new InMemoryKubernetesScalingClient();

const ops = createOperationsStack({
  healing: {
    strategies: ['hpa-boost', 'pod-restart', 'config-rollback'],
    kubernetes: {
      deployment: 'payments-api',
      hpa: { name: 'payments-hpa', client },
    },
  },
  scaling: {
    horizon: '30m',
    metrics: ['requests', 'latency'],
    hpa: { name: 'payments-hpa', client },
  },
  prometheus: {
    baseUrl: 'http://prometheus.monitoring:9090',
    queries: { requests: 'sum(rate(http_requests_total[5m]))' },
  },
});

ops.start();

A fuller wiring (Slack, drain, AI diagnostics, capacity per replica) lives in the monorepo as packages/predictive-scaling/examples/production-ops-stack.ts.

Who does what on that HPA:

Signal	Owner	Action
Forecasted request rate 30m out	Predictive scaler	Raise `minReplicas` if confidence ≥ threshold
p95 latency already over SLO	Self-healing `hpa-boost`	Raise `minReplicas` for `restoreAfterMs`
Repeated timeout / bad config	Self-healing strategies	Rollback, drain, pod restart
Git config change	Config server + client	`refresh()` without a rebuild

If both scaler and healer want more replicas, the higher floor should win. If they disagree, you have two owners — fix the wiring, not the YAML.

A concrete incident, end to end

Friday 18:40. Marketing starts a flash sale. Prometheus request rate is already 3× weekday. Forecast confidence is 0.91. The scaler raises payments-hpa minReplicas from 3 to 8 twelve minutes before checkout p95 would have crossed 2s.

18:55. A config PR sets stripe.timeout to 10 (milliseconds, not seconds). Config server serves the overlay. Clients with @ConfigValue({ refresh: true }) pick it up. Charges start failing.

Self-healing categorizes a config error, rolls back the last snapshot, notifies #incidents, and opens an OPS Jira ticket if rollback is not enough. Drain + pod-restart is reserved for process-level failure, not a one-line YAML typo.

19:10. Sale ends. Confidence drops. Cost optimization scales down slowly. hpa-boost restore timer is a no-op because predictive already owns the floor.

That is the stack: Git as config truth, healing as the safety net, forecasts as the early replica.

When not to use these packages

Single-process app, one .env. Stay on @hazeljs/config.
No Kubernetes. Predictive HPA and pod-restart need a cluster (or in-memory clients in tests). Self-heal methods and config rollback still work in-process.
You already have a mature GitOps + KEDA + custom operator story. These packages are TypeScript-native glue, not a replacement for a platform team’s operator. Adopt the pieces you lack (often: Git config HTTP + method-level heal).
You need Java {cipher} with a JKS keystore. This stack will not decrypt those blobs. Re-encrypt with AES-256-GCM or keep Spring Cloud Config for that repo.

Production checklist

Config server

Config Git repo is private; PAT via env, not committed
CONFIG_SERVER_ENCRYPT_KEY lives in the cluster secret store, not next to YAML
/health and current SHA are in your dashboards
Clients call load() at boot; refresh is either interval or explicit after deploy bots

Self-healing

Resilience decorators still wrap outbound HTTP
pod-restart has drain timeout ≥ your longest in-flight request
Slack/PagerDuty/Jira fire on healing-failed / critical-healing, not every retry
In-memory K8s clients in unit tests; fetch clients only in-cluster

Predictive scaling

Prometheus queries match the service labels you actually emit
confidence and maxReplicas are set before the first start()
One HPA name shared with self-healing
Launch events have an owner who turns them off

Get started

npm install @hazeljs/config-server @hazeljs/self-healing @hazeljs/predictive-scaling @hazeljs/core

# or via CLI
hazel add config-server
hazel add self-healing
hazel add predictive-scaling

Installation “what to install” table: Installation. Recipe index: Recipes. Glossary: Config Server, Self-Healing, Predictive Scaling.

Earlier microservices post (gateway + resilience + discovery): Building Production Microservices. This post is the ops layer on top of that.

Read the docs

If you already run HazelJS on Kubernetes, start with config-server for shared YAML, add @SelfHeal on one hot method, then point createOperationsStack at a single HPA. Share what breaks in GitHub issues or Discord — that feedback is how Phase 3 stays honest.

The HazelJS Production Ops Stack: Config Server, Self-Healing, and Predictive Scaling

The problem these packages exist to solve

How the three fit together

Part 1 — @hazeljs/config-server: Spring Cloud Config for TypeScript

Overlay order (later wins)

Run the server

HTTP API

Client, refresh, and @ConfigValue

Encryption: AES-256-GCM, not a Java keystore

Config server pitfalls

Part 2 — @hazeljs/self-healing: diagnose, then recover

What shipped, by phase

Decorators

Strategies

Drain, HPA boost, Jira

AI diagnostics

Notifications

Self-healing pitfalls

Part 3 — @hazeljs/predictive-scaling: replicas before the spike

Decorators

Programmatic API

Prometheus ingestion

Predictive scaling pitfalls

Part 4 — One operations stack

A concrete incident, end to end

When not to use these packages

Production checklist

Get started

Read the docs

Part 1 — `@hazeljs/config-server`: Spring Cloud Config for TypeScript

Client, refresh, and `@ConfigValue`

Part 2 — `@hazeljs/self-healing`: diagnose, then recover

Part 3 — `@hazeljs/predictive-scaling`: replicas before the spike