Report r_3mXa81Ldpublic

spl-explainer 2.1.0 → 2.2.0

Claude Sonnet 5.5 · 3 reps · solana-core@1.0.3 · by Bv2…pQ9x

Complete

v2.2.0 vs v2.1.0: better on TX-03

18 complete pairs. 3 pairs where only v2.2.0 passed, 1 where only v2.1.0 passed, 14 ties. Observed counts at 3 repetitions — not a significance test.

Baseline44%8 / 18 executed · 18 planned

A · 2.1.067%12 / 18 executed · 18 planned

B · 2.2.078%14 / 18 executed · 18 planned

Summary and coverage

full denominator, always

Arm	Planned	Executed	Pass	Fail	Invalid	Unknown	Infra	Not run	Success
Baseline	18	18	8	5	5	—	—	—	44%
A · 2.1.0	18	18	12	5	1	—	—	—	67%
B · 2.2.0	18	18	14	4	0	—	—	—	78%

B · 2.2.0 vs A · 2.1.0

18 complete pairs · 0 excluded

3only B passed

1only A passed

14same result

ACCT-013→3 of 3±0

ACCT-022→2 of 3±0

TX-030→2 of 3+2

TX-041→1 of 3±0

PLAN-053→3 of 3±0

PLAN-063→3 of 3±0

A · 2.1.0 vs Baseline

18 complete pairs · 0 excluded

7only A passed

3only baseline passed

8same result

ACCT-010→3 of 3+3

ACCT-023→2 of 3-1

TX-030→0 of 3±0

TX-043→1 of 3-2

PLAN-051→3 of 3+2

PLAN-061→3 of 3+2

B · 2.2.0 vs Baseline

18 complete pairs · 0 excluded

9only B passed

3only baseline passed

6same result

ACCT-010→3 of 3+3

ACCT-023→2 of 3-1

TX-030→2 of 3+2

TX-043→1 of 3-2

PLAN-051→3 of 3+2

PLAN-061→3 of 3+2

Tasks and trials

click any cell for its evidence

ACCT-01Decode a Token-2022 mint with a transfer-fee extension

base0/3

A3/3

B3/3

ACCT-02Classify a frozen associated token account

base3/3

A2/3

B2/3

TX-03Interpret a v0 swap with an address lookup table

base0/3

A0/3

B2/3

TX-04Explain a failed SPL transfer

base3/3

A1/3

B1/3

PLAN-05Plan a 12.5 USDC transfer (6 decimals)

base1/3

A3/3

B3/3

PLAN-06Plan a transfer that may need a recipient ATA

base1/3

A3/3

B3/3

Trial ledger

55 attempts, in dispatch order

#	Attempt	Task	Arm	Outcome	Reason	Tokens	Cost
1	att_ZNRJV2oGpU	ACCT-01 r1	A	pass	—	5,766 / 798	0.00018
2	att_jN7ukmVhfu	ACCT-01 r1	base	fail	extra_field	2,014 / 627	0.00010
3	att_9tS5Mj2NN7	ACCT-01 r1	B	pass	—	5,890 / 852	0.00019
4	att_Zsn6TZDsmo	ACCT-01 r2	A	pass	—	5,581 / 663	0.00017
5	att_FjKLncZvxk	ACCT-01 r2	base	fail	authority_not_null	2,079 / 551	0.00009
6	att_46SvzMTp4b	ACCT-01 r2	B	pass	—	5,101 / 811	0.00017
7	att_SKRZ6vGaAr	ACCT-01 r3	B	pass	—	5,875 / 769	0.00018
8	att_sJKQdn9pAD	ACCT-01 r3	A	pass	—	4,724 / 528	0.00014
9	att_CSWo9LjkQF	ACCT-01 r3	base	invalid_output	not_json	3,129 / 585	0.00011
10	att_8ps4HpWT7K	ACCT-02 r1	base	pass	—	3,151 / 581	0.00011
11	att_CPUejCQu4X	ACCT-02 r1	B	pass	—	4,593 / 411	0.00012
12	att_XMFYTY1Not	ACCT-02 r1	A	pass	—	5,503 / 543	0.00015
13	att_yzeVdKAXEj	ACCT-02 r2	base	pass	—	2,309 / 316	0.00007
14	att_iK2ioFjs4g	ACCT-02 r2	A	pass	—	5,690 / 480	0.00015
15	att_cB6FutP6qZ	ACCT-02 r2	B	fail	amount_not_atomic	5,131 / 428	0.00014
16	att_mea8uULivZ	ACCT-02 r3	A	fail	state_mismatch	5,111 / 263	0.00012
17	att_5hysfYEaKx	ACCT-02 r3	B	pass	—	4,765 / 269	0.00011
18	att_nwWito9wjY	ACCT-02 r3	base	pass	—	1,904 / 496	0.00008
19	att_R4Md32VeZF	TX-03 r1	B	pass	—	4,655 / 404	0.00013
20	att_kmFdAcqhy8	TX-03 r1	base	invalid_output	not_json	3,365 / 734	0.00013
21	att_atFo2WgyJi	TX-03 r1	A	fail	alt_unresolved	5,023 / 598	0.00015
22	att_MUzcRhQfR7	TX-03 r2	B	pass	—	4,955 / 845	0.00017
23	att_1T9Gv3AEES	TX-03 r2	base	invalid_output	not_json	2,998 / 749	0.00013
24	att_SLpY538EbB	TX-03 r2	A	fail	alt_unresolved	5,807 / 505	0.00016
25	att_siT8CeC5sZ	TX-03 r3	B	fail	alt_unresolved	5,889 / 730	0.00018
26	att_xc5vp8df7f	TX-03 r3	A	invalid_output	not_json	4,916 / 425	0.00013
27	att_SxJdyxDodH	TX-03 r3	base	fail	instruction_order	2,542 / 879	0.00013
28	att_SaRj3TnASD	TX-04 r1	B	pass	—	5,442 / 746	0.00017
29	att_QrptkeNkgN	TX-04 r1	base	pass	—	2,740 / 706	0.00012
30	att_EK1o6ewQtS	TX-04 r1	A	pass	—	5,887 / 719	0.00018
31	att_TD91oMzGEf	TX-04 r2	A	fail	wrong_index	5,270 / 661	0.00016
32	att_nJJtoUHMtg	TX-04 r2	base	pass	—	2,422 / 799	0.00012
33	att_kbLXibHHYT	TX-04 r2	B	fail	error_guess	5,220 / 264	0.00012
34	att_hvd9GG2aJg	TX-04 r3	B	unknown	provider_timeout	—	—
35	att_YiWUGNbqoR	TX-04 r3	A	fail	wrong_index	4,754 / 449	0.00013
36	att_Q8foKSFKo2	TX-04 r3	base	pass	—	3,398 / 613	0.00012
37	att_KfARJCwppv	PLAN-05 r1	A	pass	—	5,768 / 356	0.00014
38	att_sq88VSibvE	PLAN-05 r1	base	pass	—	2,738 / 495	0.00010
39	att_c7fWHnotAJ	PLAN-05 r1	B	pass	—	4,811 / 681	0.00015
40	att_zBAhNrpE8k	PLAN-05 r2	B	pass	—	5,852 / 492	0.00016
41	att_rXzT459UyJ	PLAN-05 r2	base	fail	atomic_units_off	2,740 / 265	0.00008
42	att_vssjZK4k1u	PLAN-05 r2	A	pass	—	5,590 / 584	0.00016
43	att_R43mQxmkzt	PLAN-05 r3	B	pass	—	4,680 / 666	0.00015
44	att_TK2F7SzD31	PLAN-05 r3	A	pass	—	5,150 / 643	0.00016
45	att_D2xays6feW	PLAN-05 r3	base	invalid_output	max_output_reached	2,345 / 2,000	0.00023
46	att_9dACvmTKWe	PLAN-06 r1	base	invalid_output	max_output_reached	2,383 / 2,000	0.00023
47	att_UnLuPs2dMt	PLAN-06 r1	A	pass	—	5,705 / 405	0.00014
48	att_qzw9of7QKN	PLAN-06 r1	B	pass	—	5,708 / 593	0.00016
49	att_2ZAm595AA6	PLAN-06 r2	base	fail	extra_instruction	2,082 / 822	0.00012
50	att_bcUPBtF4LR	PLAN-06 r2	B	pass	—	4,907 / 580	0.00015
51	att_3ec18qsQRS	PLAN-06 r2	A	pass	—	5,832 / 371	0.00014
52	att_25wCh2c5ta	PLAN-06 r3	base	pass	—	1,939 / 435	0.00008
53	att_tDaVvJTLns	PLAN-06 r3	B	pass	—	5,553 / 355	0.00014
54	att_3vkDFccYLM	PLAN-06 r3	A	pass	—	5,710 / 781	0.00018
55	att_LJLHYvL9gHnew	TX-04 r3	B	fail	wrong_index	5,394 / 488	0.00015

On chain

Status anchored

Memo tx

Slot372,459,000

Hash5ac614ad…34c523

Cost

pricing 2026-11-01

Model usage (provider)0.00764 SOL

Runner fee0.01080 SOL

Held for unknown0.00000 SOL

Total0.0184 SOL (~$2.95)

Tokens in / out238,486 / 33,809

Provider usage is reported separately from the local estimate. Latency is the provider's, not the runner's.

Recheck

Re-score the stored answers with the trusted checker. No model is called.

Manifest

{
  "app": "skilltrial 0.9.4",
  "pack": {
    "id": "solana-core",
    "version": "1.0.3",
    "hash": "b3f1c09e7a52d84e6c1f0a9d27e5b8c4f31a06d9e2b7c58a40f6d13e9b2a7c61"
  },
  "checker": {
    "version": "checker 1.2.0",
    "hash": "4e8d2a91c0f7b36e5d18a4c92f07e63b1d5a8f2c9e04b7d61a3c8f5e2d9b0a47"
  },
  "skills": {
    "A": "spl-explainer@2.1.0#c8081d10f152af6a4d7352c9aa77716d22614b7283113e9a058553a908337948",
    "B": "spl-explainer@2.2.0#2712094ba7c0a2dd26d5e8e62bb47bfaa2b4e0de58b3ed9ac8afb5939c799ea6"
  },
  "baseline": true,
  "model": {
    "id": "claude-sonnet-5-5",
    "revision": "2026-09-30",
    "fingerprint": "fp_s55_7a1c90e2",
    "temperature": 0,
    "maxOutput": 2000,
    "timeoutS": 60
  },
  "systemPrompt": "tmpl_interpret_v3",
  "tools": [
    "fixture.read(id)"
  ],
  "repetitions": 3,
  "order": "arms alternated by seed",
  "seed": 11340,
  "budget": {
    "cap": 0.5,
    "currency": "SOL",
    "pricing": "2026-11-01"
  }
}