SkillTrial docs

Overview

SkillTrial answers one question before you release a new version of an agent skill: on which Solana tasks does it help, and where does it break? It runs a baseline (no skill), version A and version B on the same fixed tasks, scores every answer with an external checker, and gives you a report with each trial in it.

It evaluates the text instructions and Markdown references of a skill. It does not execute scripts, touch mainnet RPC or wallets, browse, or download dependencies. It measures bounded interpret and plan tasks — not every coding or trading action a skill might be used for.

Quickstart

Open New run and drop both versions as a ZIP or a folder. Validation happens in your browser.
Pick a model profile, repetitions (3 by default) and a budget cap in SOL.
Approve the manifest hash. Your wallet signs one transaction that reserves the upper cost of the run.
Watch the trials land. When it finishes, open the report, export it, or anchor it on Solana.

Skill bundles

A bundle is a folder with a SKILL.md at its root plus optional references. name and version are read from the SKILL.md front matter; the SHA-256 digest of the content identifies the bundle in the manifest.

Rule	Limit / behaviour
ZIP size	≤ 5 MiB, ≤ 100 files, ≤ 1 MiB of unpacked text
Rejected outright	Path traversal, absolute paths, symlinks, duplicate or case-conflicting paths, nested archives, decompression bombs
References	Only files inside the bundle. External links are never fetched.
Scripts and binaries	Listed as non-executable. A task that needs them gets `unsupported_case`, never full coverage.
Text aimed at the grader	Flagged on import and treated as test data. It cannot reach the checker or change a score.

Fixture pack

solana-core@1.0.3 has 6 tasks in three families. Every fixture has an immutable ID, network tag, raw bytes and hash, the slot it came from where there is one, the expected structured output, the checker version and the reason codes it can return. Each task ships with known-wrong answers that the checker must reject.

Family	Input	External check
Mint / account interpretation	Raw account bytes plus network and owning program	Account type, required fields and authority / null match a verified decoder fixture
Transaction interpretation	A saved finalized transaction response of a known version	Decoded instructions, account references and outcome match the golden record
Unsigned transfer plan	Explicit mint, decimals, owner, amount and constraints	JSON plan has the right atomic units, addresses and asset; no signatures, no sending, no extra instructions

Transaction versions: legacy and 0. Any other version is excluded explicitly and never turned into a pass.

Paired protocol

The manifest pins the A/B bundle hashes, the baseline, the pack and checker hashes, the system prompt template, the model revision, sampling and output limits, repetitions, order, budget, environment and app version. Inputs are content-addressed: a run never re-reads a changed file from the same path.

Default size: 6 tasks × 3 repetitions × 3 arms = 54 trials (36 with one skill).
Every trial is a fresh context with the same task, fixtures and tools. Skill arms add only their own bundle.
No answers, memory or corrections carry over between arms. Arm order is shuffled per task from the stored seed.
Success = pass / executed_model_trials, always shown with planned, executed, infra errors, interrupted and not-run.
A paired difference uses only (task, repetition) tuples where every compared arm has a model result. The number of complete pairs and every excluded tuple are published.
Different model fingerprint, pack or checker → incomparable. A partial run never names a winner. Three repetitions give counts and the observed difference, not statistical significance.

Outcome codes

Code	Meaning	In the success rate
pass	Checker matched every required field against the golden record	Success
fail	Valid JSON, wrong answer. Comes with a reason code from the checker	Executed, not a success
invalid_output	The model answered, but not in the required structure	Executed, not a success
unsupported_case	The task needs something the skill or runner does not support, e.g. executing a script	Shown, never a pass
infrastructure_error	The request never reached the model: rate limit, network, provider 5xx before dispatch	Not the model's fault
unknown	Dispatched, then no answer: timeout or restart. Kept as is, never retried silently	Excluded until you decide
interrupted	A crash during dispatch. Usage is kept; the attempt is closed	Excluded from pairs
not_run	Never dispatched: cancelled, or the next call could break the budget cap	Excluded from pairs

Budget and recovery

Before the run you see the maximum number of requests and tokens and the upper cost bound. Execution is sequential with one request in flight.

The upper cost of each request (its input and output limits) is reserved before dispatch. A request that could exceed what is left of the cap is not sent: not_run_budget.
Every request has a unique attempt ID and a durable planned → dispatched → result / unknown record. Nothing is retried automatically after a timeout or restart.
An unknown attempt holds its reserve until you decide. A new attempt carries a visible reason; the old one stays in the ledger and is never presented as the same observation.
Cancel stops new dispatches. A request already sent may still cost money, and its usage is never zeroed.
Profiles without a verifiable price get no monetary cap: hard request and token caps plus manual confirmation, CLI only.

CLI

The CLI runs everything locally with your own model key from the OS keychain or an environment variable — never stored in the manifest. State lives in a SQLite journal plus content-addressed files; every answer is flushed to disk before the checker sees it.

$ skilltrial validate ./v1.4.2 ./v1.5.0
$ skilltrial plan --pack solana-core@1.0.3 --model claude-sonnet-5-5@2026-09-30 \
    --reps 3 --cap-usd 4 --out plan.json
$ skilltrial run --manifest <hash>       # refuses if any input changed
$ skilltrial status
$ skilltrial cancel                      # stops new dispatches, keeps in-flight
$ skilltrial recheck <run>               # re-scores stored answers, no model calls
$ skilltrial report <run> --html --json

Raw payloads are kept locally for 30 days or until you delete them. Exports warn that they may contain sensitive content and are never uploaded automatically.

Reports and anchoring

A report has six views: summary and coverage, pair comparison, tasks and trials, evidence and reasons, cost and unknowns, and the manifest. Model output is shown as escaped plain text; links and scripts in answers are never active. Hidden expected assertions never enter the model prompt.

Anchoring writes the manifest hash and the report hash to Solana with a Memo transaction signed by your wallet. Anyone can export the JSON, recompute both hashes and compare them with the memo.

Limits

One run: up to 100 requests and 60 minutes by default. One request: 60 s timeout.
Checker: up to 5 s per case. Fixture tests run offline.
Model latency and cost are reported separately and never attributed to the runner.

Fees

Hosted runs pay model usage at cost, plus a runner fee of 0.0002 SOL (~$0.03) per executed trial. Trials that were never dispatched pay nothing. The whole runner fee buys $SKTRIAL and burns it every six hours — see the buyback log. The CLI is free.