Report r_7Hk2QpV9public

solana-token-helper 1.4.2 → 1.5.0

Claude Sonnet 5.5 · 3 reps · solana-core@1.0.3 · by 7Gq…Tn4e

Complete

v1.5.0 vs v1.4.2: better on TX-03, worse on PLAN-05

18 complete pairs. 2 pairs where only v1.5.0 passed, 1 where only v1.4.2 passed, 15 ties. Observed counts at 3 repetitions — not a significance test.

Baseline50%9 / 18 executed · 18 planned

A · 1.4.283%15 / 18 executed · 18 planned

B · 1.5.089%16 / 18 executed · 18 planned

Summary and coverage

full denominator, always

Arm	Planned	Executed	Pass	Fail	Invalid	Unknown	Infra	Not run	Success
Baseline	18	18	9	8	1	—	1	—	50%
A · 1.4.2	18	18	15	3	0	—	—	—	83%
B · 1.5.0	18	18	16	2	0	—	—	—	89%

B · 1.5.0 vs A · 1.4.2

18 complete pairs · 0 excluded

2only B passed

1only A passed

15same result

ACCT-012→2 of 3±0

ACCT-023→3 of 3±0

TX-031→3 of 3+2

TX-043→3 of 3±0

PLAN-053→2 of 3-1

PLAN-063→3 of 3±0

A · 1.4.2 vs Baseline

18 complete pairs · 0 excluded

8only A passed

2only baseline passed

8same result

ACCT-011→2 of 3+1

ACCT-023→3 of 3±0

TX-031→1 of 3±0

TX-041→3 of 3+2

PLAN-053→3 of 3±0

PLAN-060→3 of 3+3

B · 1.5.0 vs Baseline

18 complete pairs · 0 excluded

9only B passed

2only baseline passed

7same result

ACCT-011→2 of 3+1

ACCT-023→3 of 3±0

TX-031→3 of 3+2

TX-041→3 of 3+2

PLAN-053→2 of 3-1

PLAN-060→3 of 3+3

Tasks and trials

click any cell for its evidence

ACCT-01Decode a Token-2022 mint with a transfer-fee extension

base1/3

A2/3

B2/3

ACCT-02Classify a frozen associated token account

base3/3

A3/3

B3/3

TX-03Interpret a v0 swap with an address lookup table

base1/3

A1/3

B3/3

TX-04Explain a failed SPL transfer

base1/3

A3/3

B3/3

PLAN-05Plan a 12.5 USDC transfer (6 decimals)

base3/3

A3/3

B2/3

PLAN-06Plan a transfer that may need a recipient ATA

base0/3

A3/3

B3/3

Trial ledger

56 attempts, in dispatch order

#	Attempt	Task	Arm	Outcome	Reason	Tokens	Cost
1	att_tgXUGZ5GiP	ACCT-01 r1	base	pass	—	2,156 / 526	0.00009
2	att_WjBPLzSSpY	ACCT-01 r1	B	fail	fee_bps_mismatch	6,018 / 372	0.00015
3	att_emheKrHXpe	ACCT-01 r1	A	fail	authority_not_null	5,399 / 422	0.00014
4	att_nNSMsWPN57	ACCT-01 r2	A	pass	—	5,427 / 427	0.00014
5	att_b4AxjgE3Lh	ACCT-01 r2	B	pass	—	5,939 / 310	0.00014
6	att_gDfrKy7oxb	ACCT-01 r2	base	fail	extra_field	2,065 / 322	0.00007
7	att_SwxVWpPEfS	ACCT-01 r3	base	fail	extra_field	2,679 / 426	0.00009
8	att_SdkZRLgEB5	ACCT-01 r3	A	pass	—	5,694 / 666	0.00017
9	att_ouNZ9ppNRD	ACCT-01 r3	B	pass	—	6,601 / 888	0.00021
10	att_FpXzqa3DuC	ACCT-02 r1	A	pass	—	6,180 / 689	0.00018
11	att_nVP4PAK7U2	ACCT-02 r1	base	infrastructure_error	provider_429	—	—
12	att_ro2XMWDWKqnew	ACCT-02 r1	base	pass	—	2,471 / 627	0.00011
13	att_KwbedGaSu1	ACCT-02 r1	B	pass	—	6,797 / 327	0.00016
14	att_cALWQAH12K	ACCT-02 r2	base	pass	—	2,716 / 597	0.00011
15	att_Nfx9wQpKLQ	ACCT-02 r2	B	pass	—	6,094 / 469	0.00016
16	att_LpVnrV36n7	ACCT-02 r2	A	pass	—	6,244 / 591	0.00017
17	att_nnWLchrxjg	ACCT-02 r3	A	pass	—	6,737 / 754	0.00020
18	att_ZjzExRLkDc	ACCT-02 r3	B	pass	—	6,942 / 548	0.00018
19	att_bdvjvbU5fh	ACCT-02 r3	base	pass	—	2,779 / 683	0.00012
20	att_5PwmgK94nz	TX-03 r1	B	pass	—	6,690 / 794	0.00020
21	att_qws9aP235t	TX-03 r1	A	fail	alt_unresolved	5,450 / 396	0.00014
22	att_mHWf4xN2Xx	TX-03 r1	base	pass	—	2,615 / 651	0.00011
23	att_3tEWoZ2Taw	TX-03 r2	A	pass	—	5,512 / 595	0.00016
24	att_bkxQCxWiN5	TX-03 r2	base	fail	alt_unresolved	2,737 / 886	0.00013
25	att_42mDV43hjK	TX-03 r2	B	pass	—	5,704 / 796	0.00018
26	att_PrLxuUs6TS	TX-03 r3	base	fail	alt_unresolved	2,379 / 588	0.00010
27	att_GLMW7X4yQb	TX-03 r3	A	fail	instruction_order	6,517 / 267	0.00015
28	att_JTqkfDrQRp	TX-03 r3	B	pass	—	6,609 / 493	0.00017
29	att_WoTKg8W73T	TX-04 r1	B	pass	—	6,377 / 447	0.00016
30	att_xWh6PffsAi	TX-04 r1	A	pass	—	6,241 / 533	0.00017
31	att_FsdgbZ9RCh	TX-04 r1	base	fail	error_guess	1,989 / 764	0.00011
32	att_MKJG7SKqLd	TX-04 r2	base	pass	—	3,114 / 869	0.00014
33	att_PFVo7PRLfq	TX-04 r2	A	pass	—	5,451 / 601	0.00016
34	att_a4NqLLuQNC	TX-04 r2	B	pass	—	6,279 / 460	0.00016
35	att_DyPm8zFXQF	TX-04 r3	B	pass	—	6,253 / 358	0.00015
36	att_tKkLTX5z15	TX-04 r3	base	fail	error_guess	3,345 / 420	0.00010
37	att_8hdMyhEYDG	TX-04 r3	A	pass	—	5,362 / 464	0.00014
38	att_KrAeukoCXe	PLAN-05 r1	A	pass	—	5,803 / 892	0.00019
39	att_4dd5VD7x1s	PLAN-05 r1	base	unknown	provider_timeout	—	—
40	att_3G28iYajrx	PLAN-05 r1	B	pass	—	6,399 / 678	0.00018
41	att_1Vsszr7Gsb	PLAN-05 r2	base	pass	—	2,187 / 691	0.00011
42	att_McQdxxhrHU	PLAN-05 r2	A	pass	—	6,655 / 411	0.00016
43	att_qqpUozcAcS	PLAN-05 r2	B	pass	—	6,450 / 446	0.00016
44	att_8wzjPGBzdK	PLAN-05 r3	B	fail	atomic_units_off	6,748 / 819	0.00020
45	att_Px36qZ4hFb	PLAN-05 r3	base	pass	—	1,922 / 806	0.00011
46	att_L4yts7bMXn	PLAN-05 r3	A	pass	—	6,227 / 834	0.00019
47	att_QcJAPcdZ1b	PLAN-06 r1	A	pass	—	5,510 / 530	0.00015
48	att_5DZwK4ePZP	PLAN-06 r1	base	fail	non_idempotent_create	2,115 / 703	0.00011
49	att_xLzcEU3imm	PLAN-06 r1	B	pass	—	5,511 / 723	0.00017
50	att_c1mqALGFTi	PLAN-06 r2	base	invalid_output	not_json	2,019 / 427	0.00008
51	att_HKGikpB6zk	PLAN-06 r2	B	pass	—	5,617 / 850	0.00019
52	att_oJqSzEbfLF	PLAN-06 r2	A	pass	—	5,286 / 834	0.00018
53	att_MHNG3W3Ac1	PLAN-06 r3	B	pass	—	5,596 / 494	0.00015
54	att_jQuU8MvrP6	PLAN-06 r3	base	fail	non_idempotent_create	2,289 / 418	0.00008
55	att_cEaoGWrHN9	PLAN-06 r3	A	pass	—	5,505 / 564	0.00016
56	att_dXApN3ekfsnew	PLAN-05 r1	base	pass	—	2,596 / 640	0.00011

On chain

Status anchored

Memo tx

Slot372,474,300

Hash4e2737e9…fd56fd

Cost

pricing 2026-11-01

Model usage (provider)0.00789 SOL

Runner fee0.01080 SOL

Held for unknown0.00000 SOL

Total0.0187 SOL (~$2.99)

Tokens in / out261,997 / 31,786

Provider usage is reported separately from the local estimate. Latency is the provider's, not the runner's.

Recheck

Re-score the stored answers with the trusted checker. No model is called.

Manifest

{
  "app": "skilltrial 0.9.4",
  "pack": {
    "id": "solana-core",
    "version": "1.0.3",
    "hash": "b3f1c09e7a52d84e6c1f0a9d27e5b8c4f31a06d9e2b7c58a40f6d13e9b2a7c61"
  },
  "checker": {
    "version": "checker 1.2.0",
    "hash": "4e8d2a91c0f7b36e5d18a4c92f07e63b1d5a8f2c9e04b7d61a3c8f5e2d9b0a47"
  },
  "skills": {
    "A": "solana-token-helper@1.4.2#a3c1c25b014415314c7ca26e01748931d413b46d9df11e02801f4f32c5d671a4",
    "B": "solana-token-helper@1.5.0#a0724a3fa93a966bafbd20dd4bf624a7687ab36f81177d795979ab687736da94"
  },
  "baseline": true,
  "model": {
    "id": "claude-sonnet-5-5",
    "revision": "2026-09-30",
    "fingerprint": "fp_s55_7a1c90e2",
    "temperature": 0,
    "maxOutput": 2000,
    "timeoutS": 60
  },
  "systemPrompt": "tmpl_interpret_v3",
  "tools": [
    "fixture.read(id)"
  ],
  "repetitions": 3,
  "order": "arms alternated by seed",
  "seed": 11002,
  "budget": {
    "cap": 0.5,
    "currency": "SOL",
    "pricing": "2026-11-01"
  }
}