Report r_Z8cJr4Fspublic

anchor-idl-guide 3.0.1 → 3.1.0

Claude Sonnet 5.5 · 5 reps · solana-core@1.0.3 · by Hn8…c3Wd

Complete

v3.1.0 vs v3.0.1: worse on TX-04

30 complete pairs. 3 pairs where only v3.1.0 passed, 5 where only v3.0.1 passed, 22 ties. Observed counts at 5 repetitions — not a significance test.

Baseline30%9 / 30 executed · 30 planned

A · 3.0.173%22 / 30 executed · 30 planned

B · 3.1.067%20 / 30 executed · 30 planned

Summary and coverage

full denominator, always

Arm	Planned	Executed	Pass	Fail	Invalid	Unknown	Infra	Not run	Success
Baseline	30	30	9	17	4	—	—	—	30%
A · 3.0.1	30	30	22	5	3	—	—	—	73%
B · 3.1.0	30	30	20	9	1	—	—	—	67%

B · 3.1.0 vs A · 3.0.1

30 complete pairs · 0 excluded

3only B passed

5only A passed

22same result

ACCT-014→4 of 5±0

ACCT-025→5 of 5±0

TX-032→2 of 5±0

TX-044→2 of 5-2

PLAN-055→5 of 5±0

PLAN-062→2 of 5±0

A · 3.0.1 vs Baseline

30 complete pairs · 0 excluded

14only A passed

1only baseline passed

15same result

ACCT-010→4 of 5+4

ACCT-023→5 of 5+2

TX-030→2 of 5+2

TX-041→4 of 5+3

PLAN-053→5 of 5+2

PLAN-062→2 of 5±0

B · 3.1.0 vs Baseline

30 complete pairs · 0 excluded

12only B passed

1only baseline passed

17same result

ACCT-010→4 of 5+4

ACCT-023→5 of 5+2

TX-030→2 of 5+2

TX-041→2 of 5+1

PLAN-053→5 of 5+2

PLAN-062→2 of 5±0

Tasks and trials

click any cell for its evidence

ACCT-01Decode a Token-2022 mint with a transfer-fee extension

base0/5

A4/5

B4/5

ACCT-02Classify a frozen associated token account

base3/5

A5/5

B5/5

TX-03Interpret a v0 swap with an address lookup table

base0/5

A2/5

B2/5

TX-04Explain a failed SPL transfer

base1/5

A4/5

B2/5

PLAN-05Plan a 12.5 USDC transfer (6 decimals)

base3/5

A5/5

B5/5

PLAN-06Plan a transfer that may need a recipient ATA

base2/5

A2/5

B2/5

Trial ledger

91 attempts, in dispatch order

#	Attempt	Task	Arm	Outcome	Reason	Tokens	Cost
1	att_kJcj6u5au1	ACCT-01 r1	base	fail	extra_field	2,898 / 712	0.00012
2	att_2hK1kmi9UJ	ACCT-01 r1	A	pass	—	6,298 / 633	0.00018
3	att_2ogGvTnZG4	ACCT-01 r1	B	fail	authority_not_null	5,535 / 647	0.00016
4	att_PVCFVN2L7j	ACCT-01 r2	A	invalid_output	max_output_reached	5,566 / 2,000	0.00029
5	att_GcLK1iJVg6	ACCT-01 r2	B	pass	—	5,941 / 826	0.00019
6	att_yHqtsNxPWf	ACCT-01 r2	base	fail	extra_field	2,948 / 798	0.00013
7	att_XHkWqFeE4e	ACCT-01 r3	A	pass	—	5,595 / 727	0.00017
8	att_Xa1v3A4BHa	ACCT-01 r3	base	fail	fee_bps_mismatch	2,390 / 393	0.00008
9	att_J73gZWSMKw	ACCT-01 r3	B	pass	—	6,224 / 375	0.00015
10	att_UmieKhLhpr	ACCT-01 r4	A	pass	—	5,381 / 260	0.00013
11	att_cwaacVaGbo	ACCT-01 r4	base	fail	fee_bps_mismatch	3,057 / 584	0.00011
12	att_LpzMUtF3RP	ACCT-01 r4	B	pass	—	6,200 / 811	0.00019
13	att_GKaXWwbsQz	ACCT-01 r5	A	pass	—	5,933 / 816	0.00019
14	att_tqfHJkCvAy	ACCT-01 r5	B	pass	—	6,122 / 795	0.00019
15	att_bUAh5GP1xR	ACCT-01 r5	base	fail	fee_bps_mismatch	2,228 / 886	0.00012
16	att_n5ABwnPWzk	ACCT-02 r1	base	pass	—	2,140 / 514	0.00009
17	att_6TQ9Yr8AXT	ACCT-02 r1	B	pass	—	5,978 / 291	0.00014
18	att_JCbmgBdSJB	ACCT-02 r1	A	pass	—	5,235 / 450	0.00014
19	att_X27YvJYo4e	ACCT-02 r2	B	pass	—	5,591 / 734	0.00017
20	att_esK4SsUxpV	ACCT-02 r2	A	pass	—	6,333 / 774	0.00019
21	att_9Fne85DqAz	ACCT-02 r2	base	pass	—	2,603 / 830	0.00013
22	att_8sZ2jVoyAV	ACCT-02 r3	A	pass	—	5,274 / 502	0.00015
23	att_ui7k6fLzt9	ACCT-02 r3	base	fail	state_mismatch	2,328 / 299	0.00007
24	att_uAbtF55ANi	ACCT-02 r3	B	pass	—	5,548 / 653	0.00017
25	att_Z24WkR8fhh	ACCT-02 r4	base	fail	state_mismatch	2,023 / 738	0.00011
26	att_mw3Pqf3dFy	ACCT-02 r4	B	pass	—	5,694 / 771	0.00018
27	att_wi46Nj3dbw	ACCT-02 r4	A	pass	—	5,868 / 417	0.00015
28	att_1Kzd97Fxwc	ACCT-02 r5	base	pass	—	3,276 / 771	0.00013
29	att_hpHMFnN5yU	ACCT-02 r5	A	pass	—	5,339 / 563	0.00015
30	att_ibyQexkRyC	ACCT-02 r5	B	pass	—	5,930 / 515	0.00016
31	att_3Q75bYyqGW	TX-03 r1	A	fail	instruction_order	5,643 / 667	0.00017
32	att_nCpuKgb7E6	TX-03 r1	base	fail	instruction_order	1,961 / 647	0.00010
33	att_isbtJQsRFJ	TX-03 r1	B	fail	instruction_order	6,449 / 591	0.00018
34	att_J1cVReT1qK	TX-03 r2	B	pass	—	6,502 / 616	0.00018
35	att_Q4Wg5CLyCv	TX-03 r2	base	fail	alt_unresolved	3,288 / 777	0.00013
36	att_t7a2p81F2B	TX-03 r2	A	invalid_output	not_json	5,584 / 782	0.00018
37	att_RDKesvB7yo	TX-03 r3	B	fail	alt_unresolved	6,871 / 767	0.00020
38	att_ifB9uXeBZ3	TX-03 r3	base	fail	instruction_order	1,913 / 804	0.00011
39	att_FDdHYbZc4k	TX-03 r3	A	fail	alt_unresolved	6,318 / 779	0.00019
40	att_FQwXKkZK2j	TX-03 r4	base	invalid_output	not_json	1,928 / 460	0.00008
41	att_1xP8mhJR26	TX-03 r4	B	fail	instruction_order	6,239 / 542	0.00017
42	att_gyjCV8AfF4	TX-03 r4	A	pass	—	5,371 / 798	0.00018
43	att_WvM2zRHwZj	TX-03 r5	B	pass	—	5,494 / 396	0.00014
44	att_kah2CyY5mS	TX-03 r5	A	pass	—	5,206 / 503	0.00014
45	att_1uqo9ubG2G	TX-03 r5	base	invalid_output	not_json	3,396 / 832	0.00014
46	att_Ugtqvj3iQB	TX-04 r1	B	pass	—	5,961 / 403	0.00015
47	att_5349nwzvbR	TX-04 r1	A	pass	—	5,515 / 390	0.00014
48	att_i6Mdwrvv8A	TX-04 r1	base	pass	—	2,179 / 840	0.00012
49	att_2H7BkNh5fz	TX-04 r2	B	fail	wrong_index	5,545 / 625	0.00016
50	att_nANz3Bfm6n	TX-04 r2	base	fail	wrong_index	3,025 / 330	0.00009
51	att_2Mf7M3qB4b	TX-04 r2	A	fail	error_guess	6,427 / 421	0.00016
52	att_18dPwX4p7D	TX-04 r3	base	fail	error_guess	2,559 / 748	0.00012
53	att_XF2GTpVGJ4	TX-04 r3	A	pass	—	5,921 / 847	0.00019
54	att_RrxKKoGbet	TX-04 r3	B	pass	—	6,927 / 600	0.00019
55	att_g2kaZEVXKL	TX-04 r4	base	fail	wrong_index	2,112 / 854	0.00012
56	att_iha7569weN	TX-04 r4	B	invalid_output	not_json	6,372 / 371	0.00015
57	att_XhmuXL5BBM	TX-04 r4	A	pass	—	5,342 / 761	0.00017
58	att_U89KeMPojQ	TX-04 r5	base	fail	wrong_index	2,108 / 700	0.00011
59	att_otXUu3PDow	TX-04 r5	B	fail	wrong_index	6,930 / 731	0.00020
60	att_2thboB2Krj	TX-04 r5	A	pass	—	6,517 / 724	0.00019
61	att_JV6XjCAQPQ	PLAN-05 r1	A	pass	—	5,633 / 286	0.00013
62	att_FLN4hpF4pt	PLAN-05 r1	B	pass	—	6,598 / 869	0.00021
63	att_cKnBvoh9tQ	PLAN-05 r1	base	unknown	provider_timeout	—	—
64	att_idXk2xcvew	PLAN-05 r2	A	pass	—	6,570 / 517	0.00017
65	att_r1bxSnGSMY	PLAN-05 r2	B	pass	—	6,827 / 857	0.00021
66	att_QiaQsommqF	PLAN-05 r2	base	pass	—	2,928 / 725	0.00012
67	att_dG2jY615VN	PLAN-05 r3	B	pass	—	5,745 / 261	0.00013
68	att_kHAcsfiHcE	PLAN-05 r3	base	pass	—	2,645 / 689	0.00011
69	att_dkX1bJ4nDq	PLAN-05 r3	A	pass	—	5,760 / 779	0.00018
70	att_XNQBTjXP7a	PLAN-05 r4	base	pass	—	2,623 / 794	0.00012
71	att_kyEMaNKZB9	PLAN-05 r4	B	pass	—	5,964 / 892	0.00020
72	att_LVRzJ2QgPX	PLAN-05 r4	A	pass	—	6,481 / 533	0.00017
73	att_NbTij9XLfA	PLAN-05 r5	A	pass	—	5,638 / 740	0.00018
74	att_nQoCSFzL8k	PLAN-05 r5	base	fail	atomic_units_not_integer	2,483 / 744	0.00012
75	att_BafCFE4znn	PLAN-05 r5	B	pass	—	6,671 / 437	0.00017
76	att_nDKUNchah7	PLAN-06 r1	base	invalid_output	max_output_reached	2,089 / 2,000	0.00023
77	att_UzCHTAxuYx	PLAN-06 r1	B	pass	—	6,489 / 463	0.00017
78	att_BYS53jXHcC	PLAN-06 r1	A	fail	non_idempotent_create	5,237 / 767	0.00017
79	att_fXzSBW42mV	PLAN-06 r2	A	pass	—	5,318 / 525	0.00015
80	att_cJhuCB1Qfv	PLAN-06 r2	base	pass	—	2,129 / 296	0.00007
81	att_HU9fCdtCRr	PLAN-06 r2	B	pass	—	6,511 / 645	0.00018
82	att_SHxtmsbPzn	PLAN-06 r3	base	pass	—	2,421 / 895	0.00013
83	att_w6uckZjgBY	PLAN-06 r3	A	invalid_output	not_json	6,192 / 656	0.00018
84	att_mEx2VfUqSn	PLAN-06 r3	B	fail	non_idempotent_create	6,634 / 288	0.00015
85	att_fce7wAVJEJ	PLAN-06 r4	B	fail	extra_instruction	6,533 / 839	0.00020
86	att_PwCXGuLv6W	PLAN-06 r4	A	fail	non_idempotent_create	5,876 / 623	0.00017
87	att_mhtYuaiVn5	PLAN-06 r4	base	fail	non_idempotent_create	2,827 / 487	0.00010
88	att_AmGkjMcm4t	PLAN-06 r5	A	pass	—	6,306 / 850	0.00020
89	att_ypRgS1VUWG	PLAN-06 r5	base	fail	extra_instruction	2,658 / 426	0.00009
90	att_YQLt2aCqAD	PLAN-06 r5	B	fail	non_idempotent_create	5,694 / 859	0.00019
91	att_oSZts5Qsgunew	PLAN-05 r1	base	invalid_output	not_json	2,149 / 422	0.00008

On chain

Status anchored

Memo tx

Slot372,387,000

Hash8decff2e…8118dd

Cost

pricing 2026-11-01

Model usage (provider)0.01373 SOL

Runner fee0.01800 SOL

Held for unknown0.00000 SOL

Total0.0317 SOL (~$5.08)

Tokens in / out434,708 / 59,555

Provider usage is reported separately from the local estimate. Latency is the provider's, not the runner's.

Recheck

Re-score the stored answers with the trusted checker. No model is called.

Manifest

{
  "app": "skilltrial 0.9.4",
  "pack": {
    "id": "solana-core",
    "version": "1.0.3",
    "hash": "b3f1c09e7a52d84e6c1f0a9d27e5b8c4f31a06d9e2b7c58a40f6d13e9b2a7c61"
  },
  "checker": {
    "version": "checker 1.2.0",
    "hash": "4e8d2a91c0f7b36e5d18a4c92f07e63b1d5a8f2c9e04b7d61a3c8f5e2d9b0a47"
  },
  "skills": {
    "A": "anchor-idl-guide@3.0.1#98fb0ce8dd259d6975c0402a293a51ba8cf07b8d7f1e772e33c0d5156d59ecca",
    "B": "anchor-idl-guide@3.1.0#e6f4ab238d6dce245ef8cff0f1153de5f7a0cd15f98caf5d5990dc9c0d82016f"
  },
  "baseline": true,
  "model": {
    "id": "claude-sonnet-5-5",
    "revision": "2026-09-30",
    "fingerprint": "fp_s55_7a1c90e2",
    "temperature": 0,
    "maxOutput": 2000,
    "timeoutS": 60
  },
  "systemPrompt": "tmpl_interpret_v3",
  "tools": [
    "fixture.read(id)"
  ],
  "repetitions": 5,
  "order": "arms alternated by seed",
  "seed": 11054,
  "budget": {
    "cap": 0.5,
    "currency": "SOL",
    "pricing": "2026-11-01"
  }
}