Agent Security Corpus

A public, tool-agnostic benchmark of agent security scenarios, and Aspex’s detection contract.

testdata/corpus/ in the repository is two things at once: the tests that stop Aspex from regressing, and a benchmark other tools can run against.

Layer	What it states
Server fixtures (malicious/, benign/)	one MCP server’s tools and config; malicious fixtures name the rules that must fire, benign ones the highest severity allowed
Environment scenarios (scenarios/*.yaml)	a whole setup (servers, hooks); truth: in generic vocabulary, expect: in Aspex’s, must_not_report: for false positives

aspex scan corpus test

  PASS  benign-sequential-thinking                   benign               3 checks
  PASS  hook-persistence-curl-pipe-sh                hook-persistence     2 checks
  PASS  memory-poisoning-via-web-search              memory-poisoning     5 checks
  PASS  project-scoped-filesystem-and-github         false-positive       8 checks
  PASS  shell-plus-fetch-remote-control              destructive-actions  6 checks
  PASS  ssh-key-cross-server-exfiltration            cross-mcp            11 checks

  ─ 6/6 scenarios passed · 35 true positives · 0 false negatives · 0 false positives

Scenarios check expected capabilities per server, attack path IDs, blast radius, and explain verdicts, and fail on anything listed under must_not_report. Writing the first six found one wrong expectation (a shell-only server is remote control, not a separate file-read exfiltration path); the corpus now pins that.

The format, both vocabularies, and how to contribute a scenario or an anonymized real-world configuration are in testdata/corpus/README.md. A scenario where Aspex disagrees with truth: is a welcome pull request: it is a bug report with a test attached.