Report r_9vNb6KpEpublic

token22-notes 1.2.0 → 1.3.0-rc1

Claude Sonnet 5.5 · 3 reps · solana-core@1.0.3 · by Ck3…Ve7r

Incomparable

Incomparable — no automatic winner

The provider fingerprint changed at trial #32. Trials before and after it are different observations; open both halves side by side instead.

Baseline39%7 / 18 executed · 18 planned

A · 1.2.072%13 / 18 executed · 18 planned

B · 1.3.0-rc156%10 / 18 executed · 18 planned

Summary and coverage

full denominator, always

Arm	Planned	Executed	Pass	Fail	Invalid	Unknown	Infra	Not run	Success
Baseline	18	18	7	9	2	—	—	—	39%
A · 1.2.0	18	18	13	4	1	—	—	—	72%
B · 1.3.0-rc1	18	18	10	6	2	—	—	—	56%

B · 1.3.0-rc1 vs A · 1.2.0

18 complete pairs · 0 excluded

4only B passed

7only A passed

7same result

ACCT-011→1 of 3±0

ACCT-023→2 of 3-1

TX-032→1 of 3-1

TX-042→3 of 3+1

PLAN-053→1 of 3-2

PLAN-062→2 of 3±0

A · 1.2.0 vs Baseline

18 complete pairs · 0 excluded

7only A passed

1only baseline passed

10same result

ACCT-011→1 of 3±0

ACCT-023→3 of 3±0

TX-030→2 of 3+2

TX-042→2 of 3±0

PLAN-050→3 of 3+3

PLAN-061→2 of 3+1

B · 1.3.0-rc1 vs Baseline

18 complete pairs · 0 excluded

5only B passed

2only baseline passed

11same result

ACCT-011→1 of 3±0

ACCT-023→2 of 3-1

TX-030→1 of 3+1

TX-042→3 of 3+1

PLAN-050→1 of 3+1

PLAN-061→2 of 3+1

Tasks and trials

click any cell for its evidence

ACCT-01Decode a Token-2022 mint with a transfer-fee extension

base1/3

A1/3

B1/3

ACCT-02Classify a frozen associated token account

base3/3

A3/3

B2/3

TX-03Interpret a v0 swap with an address lookup table

base0/3

A2/3

B1/3

TX-04Explain a failed SPL transfer

base2/3

A2/3

B3/3

PLAN-05Plan a 12.5 USDC transfer (6 decimals)

base0/3

A3/3

B1/3

PLAN-06Plan a transfer that may need a recipient ATA

base1/3

A2/3

B2/3

Trial ledger

55 attempts, in dispatch order

#	Attempt	Task	Arm	Outcome	Reason	Tokens	Cost
1	att_axeCGec3j2	ACCT-01 r1	B	fail	authority_not_null	7,293 / 608	0.00019
2	att_1k33nj92yZ	ACCT-01 r1	A	pass	—	7,580 / 340	0.00017
3	att_YQrQwXWs4Q	ACCT-01 r1	base	pass	—	2,862 / 878	0.00014
4	att_J3REcvEDVn	ACCT-01 r2	B	fail	extra_field	6,923 / 876	0.00021
5	att_VgBXbHqjty	ACCT-01 r2	base	fail	fee_bps_mismatch	2,772 / 327	0.00008
6	att_Sbfk8Ceh81	ACCT-01 r2	A	fail	authority_not_null	6,409 / 308	0.00015
7	att_LJEVsPJKPo	ACCT-01 r3	B	pass	—	7,313 / 414	0.00018
8	att_C62gLaRPA9	ACCT-01 r3	A	fail	authority_not_null	7,235 / 742	0.00021
9	att_vWoq6dJn9g	ACCT-01 r3	base	fail	authority_not_null	2,026 / 888	0.00012
10	att_d5D8hsupH8	ACCT-02 r1	A	pass	—	7,363 / 856	0.00022
11	att_zUqKYKe4Qw	ACCT-02 r1	B	pass	—	7,562 / 406	0.00018
12	att_EMVqMoBjW9	ACCT-02 r1	base	pass	—	3,332 / 889	0.00015
13	att_UY9g4eaq3y	ACCT-02 r2	base	pass	—	2,747 / 822	0.00013
14	att_RH9VdJts74	ACCT-02 r2	B	fail	amount_not_atomic	7,389 / 530	0.00019
15	att_fzHusNrjH5	ACCT-02 r2	A	pass	—	7,202 / 641	0.00020
16	att_nF3Hv14YNG	ACCT-02 r3	A	pass	—	7,018 / 262	0.00016
17	att_bWrMiGXi74	ACCT-02 r3	B	pass	—	6,386 / 326	0.00015
18	att_1oQeVgiUKh	ACCT-02 r3	base	pass	—	2,496 / 357	0.00008
19	att_vf6vYWFQry	TX-03 r1	A	pass	—	6,913 / 350	0.00016
20	att_joYtLZAsCZ	TX-03 r1	base	fail	alt_unresolved	3,032 / 711	0.00012
21	att_8GcbWnUbXT	TX-03 r1	B	fail	alt_unresolved	6,816 / 395	0.00016
22	att_CMDJQtRCNz	TX-03 r2	A	pass	—	6,570 / 301	0.00015
23	att_Qf6kwyDYVY	TX-03 r2	base	invalid_output	max_output_reached	3,140 / 2,000	0.00025
24	att_5eEsj1qZXC	TX-03 r2	B	unknown	provider_timeout	—	—
25	att_V7bKxqJLnZ	TX-03 r3	A	invalid_output	not_json	6,723 / 298	0.00015
26	att_7C9RKziL1M	TX-03 r3	B	pass	—	6,566 / 885	0.00021
27	att_7rG3fGyQRk	TX-03 r3	base	fail	alt_unresolved	2,246 / 739	0.00011
28	att_tn6gMt95bU	TX-04 r1	A	pass	—	6,811 / 288	0.00015
29	att_wj7XEsjx7z	TX-04 r1	base	invalid_output	max_output_reached	2,751 / 2,000	0.00024
30	att_sft7yRH5MW	TX-04 r1	B	pass	—	6,388 / 694	0.00018
31	att_32PYwyqfBH	TX-04 r2	base	pass	—	2,416 / 388	0.00008
32	att_8ZENro4tum	TX-04 r2	A	fail	error_guess	7,600 / 866	0.00022
33	att_2nkSL9ELFL	TX-04 r2	B	pass	—	6,145 / 887	0.00020
34	att_xnLGt2xWVW	TX-04 r3	A	pass	—	7,547 / 729	0.00021
35	att_h1zYAFXeEs	TX-04 r3	B	pass	—	6,601 / 273	0.00015
36	att_uxxEzcNtMv	TX-04 r3	base	pass	—	2,819 / 519	0.00010
37	att_mbTyJnDwVM	PLAN-05 r1	B	invalid_output	not_json	6,870 / 675	0.00019
38	att_NVjSUpse5G	PLAN-05 r1	base	fail	signature_present	2,114 / 348	0.00007
39	att_VU6sGEYxaL	PLAN-05 r1	A	pass	—	6,962 / 717	0.00020
40	att_gYkPEnCAy3	PLAN-05 r2	base	fail	atomic_units_off	2,055 / 319	0.00007
41	att_2SZWZtPion	PLAN-05 r2	B	pass	—	7,054 / 733	0.00020
42	att_j8rojan5Zj	PLAN-05 r2	A	pass	—	7,261 / 795	0.00021
43	att_b915MS85SY	PLAN-05 r3	B	fail	signature_present	7,392 / 409	0.00018
44	att_DGfRP5JNiE	PLAN-05 r3	A	pass	—	6,528 / 435	0.00016
45	att_jozeWqHaio	PLAN-05 r3	base	fail	atomic_units_not_integer	2,930 / 523	0.00010
46	att_GiEkn3H41L	PLAN-06 r1	base	fail	extra_instruction	2,830 / 824	0.00013
47	att_cFWDkdSeyC	PLAN-06 r1	B	pass	—	6,651 / 344	0.00016
48	att_aeBCCnFmh1	PLAN-06 r1	A	fail	extra_instruction	7,542 / 498	0.00019
49	att_Y6SjXyfoNe	PLAN-06 r2	A	pass	—	7,115 / 846	0.00021
50	att_SW9HqjniND	PLAN-06 r2	B	fail	non_idempotent_create	6,506 / 682	0.00019
51	att_tewZKBrKgR	PLAN-06 r2	base	fail	extra_instruction	1,967 / 854	0.00012
52	att_bb5rgxAqSm	PLAN-06 r3	B	pass	—	6,377 / 587	0.00017
53	att_GMyEpNU83a	PLAN-06 r3	A	pass	—	6,744 / 890	0.00021
54	att_vpHwTzHK5A	PLAN-06 r3	base	pass	—	2,468 / 623	0.00010
55	att_JmnX6qYvrxnew	TX-03 r2	B	invalid_output	max_output_reached	6,717 / 2,000	0.00031

On chain

Status anchored

Memo tx

Slot372,344,250

Hash03ee7751…e2d8d0

Cost

pricing 2026-11-01

Model usage (provider)0.00894 SOL

Runner fee0.01080 SOL

Held for unknown0.00000 SOL

Total0.0197 SOL (~$3.16)

Tokens in / out297,075 / 35,895

Provider usage is reported separately from the local estimate. Latency is the provider's, not the runner's.

Recheck

Re-score the stored answers with the trusted checker. No model is called.

Manifest

{
  "app": "skilltrial 0.9.4",
  "pack": {
    "id": "solana-core",
    "version": "1.0.3",
    "hash": "b3f1c09e7a52d84e6c1f0a9d27e5b8c4f31a06d9e2b7c58a40f6d13e9b2a7c61"
  },
  "checker": {
    "version": "checker 1.2.0",
    "hash": "4e8d2a91c0f7b36e5d18a4c92f07e63b1d5a8f2c9e04b7d61a3c8f5e2d9b0a47"
  },
  "skills": {
    "A": "token22-notes@1.2.0#bb12bc2fafdeb67747a434963fbd6d967debe716ce4f8c079fb5cac719f1a3dc",
    "B": "token22-notes@1.3.0-rc1#dcd3fd25a2411b93662bde29339220b44961d894657e162a76e59843c5219bca"
  },
  "baseline": true,
  "model": {
    "id": "claude-sonnet-5-5",
    "revision": "2026-09-30",
    "fingerprint": "fp_s55_7a1c90e2",
    "temperature": 0,
    "maxOutput": 2000,
    "timeoutS": 60
  },
  "systemPrompt": "tmpl_interpret_v3",
  "tools": [
    "fixture.read(id)"
  ],
  "repetitions": 3,
  "order": "arms alternated by seed",
  "seed": 13231,
  "budget": {
    "cap": 0.5,
    "currency": "SOL",
    "pricing": "2026-11-01"
  }
}