Does your skill actually help?
Run the new version, the old version and no skill at all on the same Solana tasks. A checker that ships with the runner — not the model — scores every answer. You get a report with every single trial in it
Every skill update ships on faith
A SKILL.md is a promise
The description says the skill helps with Solana. Nothing in the file shows how often it gets a mint, a transaction or a transfer right
Verified is not helpful
A verified publisher and a clean security scan tell you the bundle is safe to load. They say nothing about whether answers got better
One good answer is not a measurement
The model grading itself, or one lucky reply, will not catch the version that quietly broke atomic units
Four steps. No vibes
The same flow in the browser and in the CLI. The checker is the same binary in both
Import
Drop two versions as a folder or ZIP. You see the digest, the files, what the run covers — and which scripts will not be executed
Plan
Pick the fixture pack, the model profile, repetitions and a budget cap. The manifest is hashed before anything is sent
Run
Fresh context per trial, same tools, one request in flight. Arms are shuffled per task, so baseline is never always first
Report
Success rate with its full denominator, paired differences, reason codes and the raw answer behind every cell
Real chain data. Golden answers
Every fixture is immutable and content-addressed: raw bytes, network, slot, the expected structured output and the checker version that scores it. Golden records are verified by an independent decoder and a second reviewer
Raw account bytes plus network and owning program
Account type, required fields and authority / null match a verified decoder fixture
Same task, same tools, three arms
A difference only counts on a (task, repetition) where every compared arm got a real model answer. Anything else is listed, never quietly dropped
One task, three repetitions
order from seed 11002Outcome codes
docs →Built to be hard to fool
The checker is not the model
Scores come from a trusted checker that ships with the runner. It never runs model-generated code, never follows a URL from an answer and never lets the skill set its own grade
No hidden retries
Every request has an attempt ID and a durable planned → dispatched → result record. A timeout stays unknown until you decide
Budget before dispatch
The upper cost of the next call is reserved first. If it could break the cap, it is not sent — the trial is not_run_budget
The skill is test data
Instructions inside a bundle cannot widen access. No shell, no arbitrary HTTP, no file walking, no signing. Read-only fixture tools only
No winner when it is unfair
If the model fingerprint, pack or checker differ, the comparison is marked incomparable. Partial runs never name a winner
Reports anchored on Solana
Publish a report and its manifest hash is written on chain with a Memo transaction from your wallet. Anyone can recompute the hash from the export and check it
Recheck without the model
Saved answers re-scored by the same checker give the same results. Zero model calls
Keys stay on your machine
Prefer to run it yourself? The CLI uses your own model key from the OS keychain, writes a journal and content-addressed files next to it, and exports the same JSON and HTML report
Every hosted trial burns supply
Hosted runs pay model usage at cost plus a flat runner fee of 0.0002 SOL (~$0.03) per executed trial. All of the runner fee buys $SKTRIAL on the market and burns it, every six hours
Questions, answered
Ship the version that wins
Your first comparison takes a few minutes and costs less than a coffee. The report is yours to keep, export or anchor