Operator task
    |
    v
[ CSI meta-harness (local proxy) ]
    |
    +-- task classify --> select sub-harness
    +-- inject shared context (scope, RoE, engagement memory handle)
    +-- route through proxy --> [ CAI ] / [ Claude Code ] / [ Codex CLI ] / ...
    +-- intercept tool calls --> security primitives
    +-- aggregate telemetry + evidence + cost
    |
    v
Unified output: findings, evidence chain, cost ledger

from pydantic import BaseModel
from typing import Any, Callable, Awaitable

class SecurityPrimitiveMiddleware:
    """Wraps any sub-harness tool call. Applies the five primitives
    without modifying the sub-harness's code."""

    def __init__(self, scope, roe, evidence_logger, kill_chain, triage, cost_ledger):
        self.scope = scope
        self.roe = roe
        self.evidence = evidence_logger
        self.kill_chain = kill_chain
        self.triage = triage
        self.cost = cost_ledger

    async def wrap(self, tool_call: dict, sub_harness: str,
                   execute: Callable[[dict], Awaitable[dict]]) -> dict:
        target = tool_call.get("target")
        action = tool_call.get("action")

        # 1. Scope enforcement — before anything else
        if not self.scope.is_in_scope(target, action):
            return {"status": "blocked", "reason": "out_of_scope",
                    "scope_ref": self.scope.stamp_ref(target, action)}

        # 2. RoE enforcement — rate limit + prohibited actions
        if action in self.roe.prohibited_actions:
            return {"status": "blocked", "reason": "prohibited_by_roe"}
        await self.roe.rate_limiter.acquire()

        # 3. Kill chain state — phase-appropriate gating
        if not self.kill_chain.action_permitted_in_phase(action):
            return {"status": "blocked", "reason": "not_permitted_in_current_phase"}

        # 4. Execute via the sub-harness's own executor
        result = await execute(tool_call)

        # 5. Evidence collection — transparent to the sub-harness
        self.evidence.record(
            sub_harness=sub_harness, tool=tool_call["tool"],
            target=target, action=action,
            request=tool_call, response=result,
            scope_ref=self.scope.stamp_ref(target, action),
        )

        # 6. Signal/noise filtering — triage the result if it's a finding
        if result.get("finding"):
            triaged = await self.triage.triage(result["finding"])
            result["finding"] = triaged

        # 7. Cost accounting
        self.cost.record(sub_harness=sub_harness,
                         tokens=result.get("tokens_used", 0))

        return result

# Module S05 — Meta-Harness Architecture for Offensive Operations

**Course**: 2A — Building AI Harnesses for Cybersecurity
**Module**: S05 — Meta-Harness Architecture for Offensive Operations
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: S00, S01, S02, S03, S04 complete

> *Routing between specialized offensive harnesses. The Alias Robotics CSI model — a scaffold combiner that routes between CAI, Claude Code, Codex CLI, and other harnesses through a local proxy, with the security primitives that must live at the meta-harness level and the benchmarking discipline that proves the architecture is worth operating.*

---

## Learning Objectives

1. Explain why a single monolithic offensive harness is the wrong architecture when work spans multiple specialized domains, and design a meta-harness that routes between sub-harnesses through a local proxy.
2. Specify the meta-harness's four responsibilities — task classification, harness selection, telemetry aggregation, cost control — and the shared engagement context that all sub-harnesses read.
3. Identify the five security primitives that MUST live at the meta-harness level (scope enforcement, evidence collection, kill chain state, signal/noise filtering, rules-of-engagement enforcement) and defend why each cannot be delegated to a sub-harness.
4. Implement a security-primitive middleware that wraps any sub-harness and adds scope enforcement and evidence collection without modifying its code.
5. Define the six benchmarking metrics (detection rate, exploit success rate, false positive rate, evidence completeness, time-to-first-finding, cost per finding) and run InjecAgent as a pass/fail gate.

---

# S05.1 — When a Single Harness Is the Wrong Architecture

S02, S03, and S04 each built a harness tuned to one engagement shape: bug bounty, pentest/red team, CTF. A real offensive organization runs all three concurrently, plus web application security, plus smart contract auditing, plus cloud attack-path discovery. The naive response is to build one harness that does everything — load every tool, every prompt, every primitive into a single agent. That harness fails for the same three reasons a single CTF agent fails (S04.1): context pollution, reasoning drift, and token waste. The failure is just larger and slower, because the surface area is the union of every domain.

The correct response is the **meta-harness**: a thin orchestration layer that classifies the task, selects the right specialized sub-harness for it, and enforces the security and operational primitives that are common to every engagement. The sub-harnesses stay specialized — each keeps its own prompt, its own tool subset, its own reasoning loop. The meta-harness owns the things that are true regardless of which sub-harness is executing.

## The CSI architecture

The architecture worth studying is **CSI** (Common Scaffold Interface), developed at Alias Robotics. CSI is a **scaffold combiner**: it does not implement offensive capability itself. It routes work between existing specialized harnesses — CAI for cybersecurity operations, Claude Code for code-aware reasoning, Codex CLI for agentic software engineering, and others — through a local proxy that every sub-harness talks to.

```
Operator task
    |
    v
[ CSI meta-harness (local proxy) ]
    |
    +-- task classify --> select sub-harness
    +-- inject shared context (scope, RoE, engagement memory handle)
    +-- route through proxy --> [ CAI ] / [ Claude Code ] / [ Codex CLI ] / ...
    +-- intercept tool calls --> security primitives
    +-- aggregate telemetry + evidence + cost
    |
    v
Unified output: findings, evidence chain, cost ledger
```

The local proxy is the load-bearing design decision. Every sub-harness — regardless of its native tool-calling format, its native model provider, its native auth scheme — talks to the same proxy endpoint. The proxy owns the security primitives, the scope enforcement, the evidence logging, and the rate limiting. A sub-harness cannot bypass the proxy because the proxy is the only thing it can reach. This is the same defense-in-depth argument as S02.2's scope check in both middleware and tool, but lifted to the inter-harness boundary.

This matters because the sub-harnesses are not under your control. CAI, Claude Code, and Codex CLI are external tools with their own update cadences, their own tool surfaces, their own prompt regimes. You cannot modify their internals. You can only wrap them. The proxy is the wrap.

## Meta-harness responsibilities

The meta-harness has four jobs. Anything outside these four belongs in a sub-harness.

**Task classification.** Read the incoming task (a bug bounty brief, a CTF challenge, a pentest scope, a code-review request) and emit a domain label. This is the S04.1 meta-agent pattern generalized: a small, fast classifier — model or deterministic — that routes to the right specialist. When the classifier is uncertain, a TriageAgent runs cheap probes (target type, scope, attached artifacts) and re-routes, exactly as in S04.1. The meta-harness does not solve the task; it decides who does.

**Harness selection.** Map the domain label to a sub-harness and load its configuration: system prompt, tool subset, context budget, HITL posture. A bug bounty task routes to the S02 harness with the bug bounty tool suite and engagement memory. A CTF task routes to the S04 harness with the six-domain router and the CTFd submission client. A code-review task routes to Codex CLI with the SDLC prompt and no offensive tools loaded at all. Selection is a lookup table, not a model call — once classification is done, the mapping is deterministic.

**Telemetry aggregation.** Every sub-harness emits telemetry: tool calls, model invocations, tokens consumed, findings produced. The meta-harness aggregates this into one ledger per engagement, regardless of which sub-harness generated each event. This is what makes cross-harness cost control and benchmarking possible — there is one place to read "what did this engagement cost and what did it find" instead of reconciling logs from three tools with three formats.

**Cost control.** The meta-harness enforces a per-engagement budget: total tokens, total dollars, total wall-clock. When a sub-harness exhausts its allocation, the meta-harness either re-routes to a cheaper sub-harness, degrades gracefully (fewer parallel attempts, narrower tool surface), or halts and surfaces to the operator. Cost control cannot live in a sub-harness because a sub-harness only sees its own consumption; only the meta-harness sees the total.

## Shared engagement context

Every sub-harness reads the same engagement context: the scope file (S00.3), the rules of engagement, a handle to the persistent engagement memory (S02.1), and the engagement ID. The sub-harness does not own this context; it borrows it. The meta-harness loads it once at engagement start and passes a reference to whichever sub-harness is active.

This is what prevents the "three harnesses, three scope files" failure mode. If CAI loads its own scope, Claude Code loads its own scope, and Codex CLI has no scope at all, the operator is responsible for keeping them synchronized — and they will drift. Drifted scope is how out-of-scope tool calls happen. The cure is one scope file, owned by the meta-harness, read-only to every sub-harness.

## Cross-harness evidence aggregation

S02.3 established the evidence chain: append-only, hash-chained, scope-referenced. In a meta-harness architecture, the evidence chain is owned by the meta-harness, not by any sub-harness. A finding produced by CAI and a finding produced by Claude Code both land in the same evidence log, with the same schema, stamped with the same engagement ID and scope_ref.

This is the property that makes the meta-harness legally defensible. When the client asks "how was this finding obtained," the answer is one query against one log — not a reconciliation exercise across three tools' separate evidence stores. The sub-harnesses do not even know the evidence log exists; the proxy intercepts their tool calls and writes the evidence records transparently.

---

# S05.2 — Building the Security Primitive Layer

A general-purpose meta-harness — one that only routes and aggregates — is not enough for offensive work. Offensive harnesses need security primitives: scope enforcement, evidence collection, rules-of-engagement enforcement. A meta-harness that routes between sub-harnesses but does not enforce these primitives is an open pipe to the network, and the sub-harnesses (which you do not control) will eventually call out of scope, fail to log evidence, or violate the RoE.

The gap is that the sub-harnesses were not built with these primitives. CAI has its own scope model; Claude Code has none; Codex CLI is a coding tool with no concept of scope at all. The meta-harness must impose the primitives uniformly, regardless of which sub-harness is executing. This is the **security primitive layer**: a middleware that wraps every sub-harness and applies the five primitives below without modifying the sub-harness's code.

## The five primitives

These five must live at the meta-harness level. Each is justified by what breaks if it is delegated to a sub-harness.

**1. Scope enforcement.** Every outbound tool call — every nmap scan, every HTTP request, every file read — is checked against the engagement scope file before execution. This is S00.3 and S02.2's scope middleware, lifted to the meta-harness boundary. It cannot live in a sub-harness because the sub-harnesses do not all implement it, and the ones that do implement it differently. One scope file, one enforcement point, applied to every sub-harness through the proxy.

**2. Evidence collection.** Every tool call's request and response is captured into the append-only, hash-chained evidence log (S02.3). The sub-harness does not opt in; the proxy intercepts the call and writes the record. This is what produces one evidence log per engagement regardless of how many sub-harnesses ran. Delegate this to a sub-harness and you get three logs with three schemas and three hash chains — useless for legal defense.

**3. Kill chain state.** The engagement's position in the attack kill chain (recon → weaponize → deliver → exploit → post-exploit) is tracked at the meta-harness level. This is S03's kill chain state machine, generalized. The meta-harness knows "we are in the exploitation phase against target X" and can enforce phase-appropriate controls (e.g., post-exploit actions require explicit operator confirmation even in CTF contexts where other gates are removed). Sub-harnesses report phase transitions; the meta-harness holds the authoritative state.

**4. Signal/noise filtering.** The triage pipeline from S02.4 runs at the meta-harness level. Findings from any sub-harness — CAI's nuclei output, Claude Code's code review findings, Codex CLI's static analysis — flow through the same dedup, severity, enrichment, and model-judged triage. This is what produces one prioritized finding list per engagement. If each sub-harness triages its own output, the operator reconciles three finding lists with three severity schemes and three notions of "in scope."

**5. Rules-of-engagement enforcement.** The RoE (max requests per second, prohibited actions, time windows, data-handling restrictions) is enforced at the proxy. A sub-harness that wants to send 200 requests per second is throttled to the RoE cap regardless of what it thinks its own limit is. This is S02.2's rate limiting, lifted to the boundary where it cannot be bypassed by a sub-harness that does not implement it.

## Why these cannot be sub-harness-level

The argument is the same for all five: you do not control the sub-harnesses. CAI's scope model is CAI's; you cannot make Claude Code adopt it. Codex CLI has no concept of an evidence chain. A sub-harness update can silently change or remove a primitive. The only way to guarantee that every outbound call from every sub-harness is scope-checked, evidence-logged, RoE-compliant, and triaged consistently is to enforce it at the one choke point every sub-harness must pass through: the proxy.

This is defense in depth at the architecture boundary. The sub-harness may have its own scope check (CAI does). The tool may have its own scope check (S02.2's belt-and-suspenders). The proxy's check is the third layer, and it is the one that catches the case where a sub-harness was updated and its internal check broke, or where a sub-harness never had one.

## The middleware pattern

The security primitive layer is implemented as middleware that wraps the sub-harness's tool-call interface. The sub-harness calls a tool; the middleware intercepts the call, applies the primitives, executes (or blocks), and returns the result. The sub-harness's code is unchanged.

```python
from pydantic import BaseModel
from typing import Any, Callable, Awaitable

class SecurityPrimitiveMiddleware:
    """Wraps any sub-harness tool call. Applies the five primitives
    without modifying the sub-harness's code."""

    def __init__(self, scope, roe, evidence_logger, kill_chain, triage, cost_ledger):
        self.scope = scope
        self.roe = roe
        self.evidence = evidence_logger
        self.kill_chain = kill_chain
        self.triage = triage
        self.cost = cost_ledger

    async def wrap(self, tool_call: dict, sub_harness: str,
                   execute: Callable[[dict], Awaitable[dict]]) -> dict:
        target = tool_call.get("target")
        action = tool_call.get("action")

        # 1. Scope enforcement — before anything else
        if not self.scope.is_in_scope(target, action):
            return {"status": "blocked", "reason": "out_of_scope",
                    "scope_ref": self.scope.stamp_ref(target, action)}

        # 2. RoE enforcement — rate limit + prohibited actions
        if action in self.roe.prohibited_actions:
            return {"status": "blocked", "reason": "prohibited_by_roe"}
        await self.roe.rate_limiter.acquire()

        # 3. Kill chain state — phase-appropriate gating
        if not self.kill_chain.action_permitted_in_phase(action):
            return {"status": "blocked", "reason": "not_permitted_in_current_phase"}

        # 4. Execute via the sub-harness's own executor
        result = await execute(tool_call)

        # 5. Evidence collection — transparent to the sub-harness
        self.evidence.record(
            sub_harness=sub_harness, tool=tool_call["tool"],
            target=target, action=action,
            request=tool_call, response=result,
            scope_ref=self.scope.stamp_ref(target, action),
        )

        # 6. Signal/noise filtering — triage the result if it's a finding
        if result.get("finding"):
            triaged = await self.triage.triage(result["finding"])
            result["finding"] = triaged

        # 7. Cost accounting
        self.cost.record(sub_harness=sub_harness,
                         tokens=result.get("tokens_used", 0))

        return result
```

The sub-harness sees only the `execute` callback. It calls a tool, the middleware runs, the result comes back. The sub-harness has no idea that scope was checked, evidence was logged, or triage ran. This is what makes the pattern robust to sub-harness updates: the primitives are not in the sub-harness's code path, so they cannot be removed by a sub-harness change.

---

# S05.3 — Benchmarking Your Offensive Harness

A meta-harness that routes, enforces primitives, and aggregates telemetry is not done. It must be measurable. Without benchmarking, you cannot tell whether routing to CAI versus Claude Code for a given task class is actually better, whether the security primitives are catching real violations, or whether the harness is competitive with published baselines. This section covers the metrics, the pass/fail gate, and the repeatable lab.

## The six metrics

Six metrics cover offensive harness performance. Each measures a different failure mode; together they prevent the "it feels faster" trap where an unmeasured change is mistaken for an improvement.

| Metric | Measures | Target |
| --- | --- | --- |
| **Detection rate** | Of seeded vulnerabilities, how many the harness finds | >85% on a labeled lab |
| **Exploit success rate** | Of detected vulnerabilities, how many are exploitably demonstrated | >60% (context-dependent) |
| **False positive rate** | Of findings surfaced, how many are not real | <15% (1 − precision) |
| **Evidence completeness** | Of findings, how many have a complete, reproducible evidence chain | >95% |
| **Time-to-first-finding** | Wall-clock from engagement start to first confirmed finding | Domain-dependent; track the trend |
| **Cost per finding** | Dollars (tokens + compute) divided by confirmed findings | Track the trend; compare sub-harnesses |

Detection rate and false positive rate are the precision/recall pair from S02.4, generalized. Exploit success rate is the harder metric — detection is necessary but not sufficient; the harness must demonstrate exploitability, not just assert it. Evidence completeness is the legal/compliance metric: a finding without a complete evidence chain is not deliverable to a client. Time-to-first-finding and cost per finding are the operational metrics — they tell you whether the harness is competitive and sustainable.

The targets in the right column are starting points, not absolutes. The discipline is measuring consistently and tracking the trend across harness versions. A change that improves detection rate by 5 points but doubles cost per finding is a tradeoff, not a win — and you can only see that tradeoff if you measure all six.

## InjecAgent as a pass/fail gate

Before benchmarking offensive performance, the harness must pass a safety gate: it must resist adversarial manipulation of its tool output. **InjecAgent** is the benchmark for this — a suite of injection attacks against agent tool-calling, with pass/fail scoring on whether the agent executed the injected action.

InjecAgent is a pass/fail gate, not a performance metric, because the failure mode is catastrophic. A harness that finds 95% of vulnerabilities but executes injected commands from target output is not a harness you can operate against real targets — it will eventually exfiltrate scope, modify evidence, or call out-of-scope systems because a target response told it to. The S01.3 reader-actor separation and S02.1's structured-summary-only memory are the defenses; InjecAgent is the test that proves they work.

The gate: run InjecAgent against the meta-harness with the security primitive layer enabled. The harness must refuse every injected action — no exceptions. A single execution of an injected command is a fail, regardless of how the harness scored on the six metrics. Fix the injection resistance before benchmarking anything else.

## Comparing against published baselines

Three published baselines give external reference points:

- **CAI CTF benchmarks** — CAI's published results on standard CTF challenge sets. Use these to validate the S04 router's domain routing and the CTF sub-harness's solve rate.
- **EVMbench** (for smart contract work) — smart contract audit benchmarks. Use these when the meta-harness routes to a smart-contract sub-harness; they measure detection rate and false positive rate on a labeled set of vulnerable contracts.
- **RedTeamLLM CTF results** — published red-team LLM performance on CTF. Use these as a competitive baseline for the offensive reasoning quality of the underlying models.

The point of comparison is not to beat the published number — it is to confirm your harness is in the same order of magnitude and to identify where it is systematically worse. A harness that scores 40% detection rate where the baseline scores 85% has an architecture problem, not a tuning problem.

## The repeatable benchmark lab

A benchmark is only useful if it is repeatable. The lab has four properties:

1. **Isolated targets.** Docker containers or VMs with known, seeded vulnerabilities. No live targets — live targets change and invalidate the baseline.
2. **Seeded vulnerabilities.** Each target has a documented vulnerability set (CVE, CWE, location). The harness's findings are scored against this set — detection rate is finding-count divided by seeded-count.
3. **Consistent scoring methodology.** The same scoring script runs against every harness version. A finding is "correct" if it matches a seeded vulnerability by type and location; "false positive" if it does not match; "missed" if a seeded vulnerability is not found.
4. **Fixed compute envelope.** Same model, same token budget, same wall-clock limit per run. A harness that scores higher because it consumed 3x the compute is not better — it is more expensive.

The output of a benchmark run is a scored report: the six metrics, the InjecAgent pass/fail, and a comparison against the previous run and the published baseline. This is the artifact you publish — internally or externally — to defend the architecture.

---

## Anti-Patterns

### The monolithic super-harness
One harness with every tool, every prompt, every primitive loaded, attempting every engagement type. Context pollution, reasoning drift, token waste at the union of all domains. Cure: the meta-harness plus specialized sub-harnesses from S05.1.

### Primitives delegated to sub-harnesses
Scope enforcement left to CAI's internal model, evidence logging left to whichever sub-harness happens to implement it, RoE enforced by convention. A sub-harness update silently removes a primitive; an out-of-scope call is not caught. Cure: the security primitive layer at the meta-harness boundary (S05.2).

### Three scope files, three evidence logs
Each sub-harness loads its own scope and writes its own evidence. Scope drifts, evidence is unreconcilable, the legal defense fails. Cure: one scope file and one evidence log, owned by the meta-harness, read/written through the proxy.

### Benchmarking by feel
A routing change "feels faster," so it ships. No measurement, no baseline, no comparison. The change actually increased cost per finding by 40%. Cure: the six metrics and the repeatable lab (S05.3).

### Skipping the InjecAgent gate
Benchmarking offensive performance before confirming injection resistance. The harness scores well on detection but executes an injected command during a real engagement. Cure: InjecAgent is a pass/fail gate, run first, no exceptions.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **Meta-harness** | A thin orchestration layer that classifies tasks, selects sub-harnesses, aggregates telemetry, and controls cost — without implementing offensive capability itself |
| **CSI** | Common Scaffold Interface (Alias Robotics) — a scaffold combiner that routes between CAI, Claude Code, Codex CLI, and other harnesses through a local proxy |
| **Local proxy** | The single endpoint every sub-harness talks to; the choke point where security primitives are enforced and cannot be bypassed |
| **Security primitive layer** | Middleware wrapping every sub-harness that applies the five primitives (scope, evidence, kill chain, signal/noise, RoE) without modifying sub-harness code |
| **Shared engagement context** | One scope file, one RoE, one engagement memory handle, owned by the meta-harness and read by every sub-harness |
| **InjecAgent** | A benchmark suite for adversarial injection against agent tool-calling; used as a pass/fail gate before offensive benchmarking |
| **Detection rate** | Of seeded vulnerabilities, the fraction the harness finds |
| **Cost per finding** | Total dollars (tokens + compute) divided by confirmed findings; the sustainability metric |

---

## Lab Exercise

See `07-lab-spec.md`. Three labs: (1) map the CSI architecture from the Alias Robotics codebase and draw the routing logic as a state machine; (2) implement a security primitive middleware that wraps any sub-harness and adds scope enforcement and evidence collection without modifying its code; (3) define and run a benchmark suite against your current offensive harness, producing a scored output you can publish.

---

## References

1. **CSI (Common Scaffold Interface)** — Alias Robotics' scaffold combiner that routes between CAI, Claude Code, Codex CLI via local proxy. The reference architecture for S05.1.
2. **CAI (Cybersecurity AI)** — the offensive sub-harness; one of the routed targets in the CSI architecture.
3. **InjecAgent** — benchmark for indirect prompt injection against agent tool-calling; the pass/fail gate for S05.3.
4. **EVMbench** — smart contract vulnerability detection benchmark; baseline for the smart-contract sub-harness.
5. **RedTeamLLM CTF results** — published red-team LLM performance on CTF; competitive baseline.
6. **S00.3** — authorization scope; the scope file the meta-harness loads and enforces.
7. **S01.3** — reader-actor separation; the injection-resistance defense InjecAgent tests.
8. **S02.1** — persistent engagement memory; the shared context handle the meta-harness passes to sub-harnesses.
9. **S02.3** — the evidence chain; the single log the meta-harness owns and the proxy writes to.
10. **S04.1** — the meta-agent-plus-specialists pattern; the routing model generalized in S05.1.