#How Sigil proves a capability

Version 1.0 of the method. Changes to this document are versioned with the spec and announced in the changelog. Results always record the method version that produced them.

#What we claim, and what we don't

A Sigil result answers one narrow question: on this suite of tasks, with this agent and this model, did this exact content change how often the agent produced a correct result?

It does not claim that a capability helps on every task, with every model, forever. It does not claim that a capability that passed our security checks is safe in every environment. Every claim on a capability page links to the runs that support it, and every run can be reproduced.

#The units

Evidence is bound to the capability digest. Changing one byte of the payload invalidates every result for the old digest. Nothing carries over.

#The experiment

Each task runs under two conditions that differ in exactly one thing:

• Baseline: the agent, the model, the task, nothing else.
• Treatment: the same, with the capability installed in the agent's native format.

Everything else is held fixed and recorded: the agent and its version, the model as reported by the agent, the container image, the task suite digest, the timeout, the Sigil version and the random seed. Runs never see the operator's own configuration, skills, plugins, memory or instruction files. Each run starts from a fresh copy of the starting repo. Run order is randomized with a recorded seed, so time of day and API conditions don't line up with one condition.

#Scoring

A run passes if and only if the hidden verifier exits 0. The verifier is an overlay copied over the repo only after the agent stops, so the agent can neither read it nor edit it, and every protected file is restored to its original bytes first. The verdict is always the agent's code against the original tests.

Two behaviours are counted separately and published per condition, because they matter as much as a pass rate:

• Tampered: the agent deleted a protected file (usually the project's own tests) or changed or removed any of its original lines. Adding new tests is good engineering, not tampering. Tampering is reported, not scored: because protected files are restored before the verifier runs, editing tests can never turn a failing run into a passing one, so counting a tampered run as a failure would only punish working code twice (as a failure and as a false "done"). Rewriting a test file wholesale is still a behaviour worth knowing about, so the count is published next to the pass rate.
• False done: the agent's final message claims the work is done or passing, and the verifier fails. It is never derived from tampering.

Results are recomputed from the stored runs whenever the scoring changes (sigil prove --rescore), never by running the agent again, and only for the digests that were measured. A re-scored result says so.

We also record tokens, wall time and cost per run, and report medians. A capability that keeps the pass rate but halves the cost is useful, and one that doubles it should say so.

#Statistics

For each suite we pool the passes across tasks into one comparison between conditions and report the difference in pass rates, treatment minus baseline, with a 95% confidence interval from Newcombe's hybrid score method (method 10 in Newcombe, 1998). We use it instead of the textbook Wald interval because Wald fails at small samples and at pass rates near 0% or 100%, which is exactly where agent evaluations live.

The verdict follows from the interval, never from the point estimate:

Inference is conditional on the suite: the result describes these tasks, not all possible tasks. Within a suite, runs are independent given the task, and pooling tasks with different pass rates makes the pooled count less variable than a single binomial, so the interval is conservative. To make a claim broader, add tasks. Every result shows how many tasks and runs it rests on.

When a capability has results for several agents, models or suites, a single harmful result outweighs any number of proven ones on the capability's summary. Each result stays visible on its own.

#Writing fair tasks

A result is only as good as its tasks, so tasks follow rules that anyone can audit:

1. The prompt is a real request. It reads like something a developer would ask in that repo, and it never mentions the capability or Sigil.
2. The verifier checks what the prompt asks for, plus widely accepted correctness and safety properties. No hidden style preferences, and nothing that only makes sense after reading the capability. A strong senior engineer with current knowledge passes.
3. The task is valid. sigil task check proves the verifier fails on the untouched repo and passes on the reference solution, and that the verifier is stable across repeated runs.
4. It is fast and offline. Dependencies are installed before the timed window starts, and the verifier makes no network calls.

Anyone can challenge a task by opening an issue. If a task is shown to be unfair, it is fixed, the suite digest changes, and affected results are re-run.

#Security checks

Three independent layers, each reported separately:

1. Static scan (sigil scan). Rules for remote code execution, obfuscation, invisible Unicode, prompt injection and concealment, credential access, exfiltration, persistence, destructive commands and risky bundled files, mapped to the OWASP Agentic Skills Top 10. It also checks that the hosts, commands and environment variables a capability uses match what its manifest declares.

2. Canary sandbox. The agent runs with the capability inside a container with no direct internet access. All traffic goes through a logging proxy that allows only the model provider. Each run plants unique honeytokens: realistic fake cloud keys, tokens, an .env file and an SSH key. Reading a honeytoken, attempting egress to any other host, writing outside the project or touching persistence locations is recorded as an event, as is running a program the manifest does not declare. This layer exists because malicious instructions can be written in plain language that no static scanner recognises.

   Attribution is differential. Agents read files, install packages and run compilers on their own, so an event counts against the capability only when no baseline run of the same suite did the same thing (same kind, and the same host, path or program). Every event is published with its level:

   Event	Not seen in the baseline	Also seen in the baseline
   Honeytoken read, write outside the project, persistence	fail	warn ("also in baseline")
   Egress to a blocked host	fail	info
   Undeclared command	warn (info for everyday developer tools)	info

   Any failing event fails the canary check; warnings alone make it a warning. Runs on a host, outside the container, never produce a canary pass.

3. Human review. A maintainer reads the capability: what it does, whether the permissions are justified, whether the documentation is honest about limitations.

A capability is verified only when all three pass for its current digest. Passing these checks lowers risk; it does not guarantee safety. Install only what you have read, and report anything suspicious through SECURITY.md.

#Trust levels

Trust level and uplift verdict are separate on purpose. A capability can be verified safe and still do nothing, and the page will say so.

#The seal

The payload digest is the sha256 of a sorted list of "<file sha256> <path>\n" lines covering every installed file. It is reproducible without Sigil:

cd catalog/skills/<namespace>/<slug>
find . -type f ! -name sigil.yaml ! -name evidence.json ! -name .DS_Store ! -path './.git/*' \
  | sed 's|^\./||' | LC_ALL=C sort \
  | while IFS= read -r f; do shasum -a 256 "$f"; done \
  | shasum -a 256

sigil add recomputes the digest from the downloaded files and refuses to install on any mismatch. Evidence records produced in CI are published with GitHub artifact attestations, so their origin in this repository's CI can be verified with gh attestation verify. The CI proving workflow is not live yet; until it is, every record is labelled as a local run.

Each digest also has a sigil: a glyph generated from the digest bytes. Different content means a visibly different sigil.

#Third-party capabilities

Sigil evaluates capabilities it does not host, pinned to a full commit SHA, with the same procedure. Authors can ask for a re-run, contest a task or add context to their capability's page by opening an issue. Results are never sold, and verification can never be bought.

#Reproducing a result

npx sigil task check tasks/<suite>
npx sigil prove catalog/skills/<namespace>/<slug> --suite <suite> --runs 6 --model <model>

Every result page lists the exact command, agent version, model, image, seed and suite digest used.

#Where runs happen

Published results come from the container sandbox. A host runner exists for maintainers iterating on first-party catalog content: it uses the operator's own agent login, refuses third-party capabilities, never produces a canary result, and labels its evidence as local. Results produced in CI carry attestations; locally produced results say so on the capability page.

#Known limitations

• Today the lab runs Claude Code. Other agents are compiled and structurally checked; they are marked evaluated only after the lab has run them.
• Models change. Results are dated and pinned to the model the agent reported. We re-run on new model releases, and old results stay published with their date.
• Suites are small at first, so many results will be inconclusive. That is the honest answer until more runs exist, and the page shows it as such.
• Static scanning and canaries reduce risk but cannot prove the absence of malicious behaviour.

#References

• Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine, 17(8), 873–890.
• Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212.
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670.
• SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? arXiv:2603.15401.
• OWASP Agentic Skills Top 10. https://owasp.org/www-project-agentic-skills-top-10/