Report r_Lw4Ds0Mhpublic

wallet-plan-kit 0.4.0 → 0.5.0

Claude Haiku 5.5 · 3 reps · solana-core@1.0.3 · by 9Pd…mT2k

Complete

v0.5.0 vs v0.4.0: better on TX-03, TX-04

18 complete pairs. 8 pairs where only v0.5.0 passed, 4 where only v0.4.0 passed, 6 ties. Observed counts at 3 repetitions — not a significance test.

Baseline22%4 / 18 executed · 18 planned

A · 0.4.050%9 / 18 executed · 18 planned

B · 0.5.072%13 / 18 executed · 18 planned

Summary and coverage

full denominator, always

Arm	Planned	Executed	Pass	Fail	Invalid	Unknown	Infra	Not run	Success
Baseline	18	18	4	10	4	—	—	—	22%
A · 0.4.0	18	18	9	6	3	—	—	—	50%
B · 0.5.0	18	18	13	3	2	—	1	—	72%

B · 0.5.0 vs A · 0.4.0

18 complete pairs · 0 excluded

8only B passed

4only A passed

6same result

ACCT-011→1 of 3±0

ACCT-022→2 of 3±0

TX-030→3 of 3+3

TX-042→3 of 3+1

PLAN-052→2 of 3±0

PLAN-062→2 of 3±0

A · 0.4.0 vs Baseline

18 complete pairs · 0 excluded

6only A passed

1only baseline passed

11same result

ACCT-010→1 of 3+1

ACCT-021→2 of 3+1

TX-030→0 of 3±0

TX-042→2 of 3±0

PLAN-051→2 of 3+1

PLAN-060→2 of 3+2

B · 0.5.0 vs Baseline

18 complete pairs · 0 excluded

10only B passed

1only baseline passed

7same result

ACCT-010→1 of 3+1

ACCT-021→2 of 3+1

TX-030→3 of 3+3

TX-042→3 of 3+1

PLAN-051→2 of 3+1

PLAN-060→2 of 3+2

Tasks and trials

click any cell for its evidence

ACCT-01Decode a Token-2022 mint with a transfer-fee extension

base0/3

A1/3

B1/3

ACCT-02Classify a frozen associated token account

base1/3

A2/3

B2/3

TX-03Interpret a v0 swap with an address lookup table

base0/3

A0/3

B3/3

TX-04Explain a failed SPL transfer

base2/3

A2/3

B3/3

PLAN-05Plan a 12.5 USDC transfer (6 decimals)

base1/3

A2/3

B2/3

PLAN-06Plan a transfer that may need a recipient ATA

base0/3

A2/3

B2/3

Trial ledger

56 attempts, in dispatch order

#	Attempt	Task	Arm	Outcome	Reason	Tokens	Cost
1	att_G5Tb7ERE6S	ACCT-01 r1	B	invalid_output	not_json	6,132 / 688	0.00006
2	att_xQ3eiHx9Lh	ACCT-01 r1	A	pass	—	6,360 / 438	0.00005
3	att_LgCkbvzKsa	ACCT-01 r1	base	fail	extra_field	2,991 / 611	0.00004
4	att_2DtXJgVqvQ	ACCT-01 r2	base	fail	authority_not_null	2,206 / 881	0.00004
5	att_Exnz9MK288	ACCT-01 r2	A	invalid_output	not_json	6,539 / 664	0.00006
6	att_kwfuy4cjLc	ACCT-01 r2	B	invalid_output	not_json	6,654 / 477	0.00006
7	att_oNbNiJUuGN	ACCT-01 r3	A	fail	fee_bps_mismatch	7,251 / 813	0.00007
8	att_4MAPCDs2XQ	ACCT-01 r3	base	fail	authority_not_null	2,637 / 306	0.00003
9	att_JFyknEdTC3	ACCT-01 r3	B	pass	—	6,735 / 857	0.00007
10	att_DkbN8ct8fm	ACCT-02 r1	base	fail	state_mismatch	3,260 / 790	0.00005
11	att_Vmy1R7mWe2	ACCT-02 r1	B	infrastructure_error	provider_429	—	—
12	att_sg62LeuY66new	ACCT-02 r1	B	pass	—	6,300 / 883	0.00007
13	att_iq8WapvxDb	ACCT-02 r1	A	pass	—	6,466 / 598	0.00006
14	att_DdPGfeJruZ	ACCT-02 r2	A	fail	amount_not_atomic	6,001 / 868	0.00006
15	att_B44HDPG3nM	ACCT-02 r2	B	pass	—	7,091 / 310	0.00005
16	att_8Cn9yMCR3F	ACCT-02 r2	base	invalid_output	max_output_reached	2,842 / 2,000	0.00008
17	att_SwbSNXXwLQ	ACCT-02 r3	A	pass	—	6,281 / 858	0.00007
18	att_13C7Pz84eS	ACCT-02 r3	B	fail	state_mismatch	6,291 / 746	0.00006
19	att_1W8WogTthy	ACCT-02 r3	base	pass	—	2,484 / 424	0.00003
20	att_8NYhPfZtyK	TX-03 r1	A	invalid_output	max_output_reached	6,508 / 2,000	0.00010
21	att_PLVoeDUtQd	TX-03 r1	B	pass	—	6,196 / 327	0.00005
22	att_iL8BXTjXyt	TX-03 r1	base	fail	instruction_order	2,506 / 817	0.00004
23	att_eC6Ty1CwEH	TX-03 r2	B	pass	—	6,232 / 885	0.00007
24	att_6gsWfxWd17	TX-03 r2	A	fail	instruction_order	6,725 / 346	0.00005
25	att_6wvQjfDXAA	TX-03 r2	base	unknown	provider_timeout	—	—
26	att_CFwTMMuTLw	TX-03 r3	B	pass	—	6,250 / 262	0.00005
27	att_GiWNrX2xFR	TX-03 r3	A	fail	alt_unresolved	5,936 / 375	0.00005
28	att_bTPRtR4sXx	TX-03 r3	base	invalid_output	not_json	3,274 / 405	0.00003
29	att_a2uUgo67Q2	TX-04 r1	base	pass	—	2,333 / 821	0.00004
30	att_HjNdfsUdmp	TX-04 r1	A	pass	—	6,428 / 633	0.00006
31	att_t94BLZJ6Wf	TX-04 r1	B	pass	—	7,227 / 533	0.00006
32	att_EogtJoPrp5	TX-04 r2	A	pass	—	5,970 / 382	0.00005
33	att_n7jVccpGH9	TX-04 r2	base	fail	wrong_index	2,155 / 584	0.00003
34	att_e1qZ9U4xJY	TX-04 r2	B	pass	—	7,202 / 888	0.00007
35	att_W57iGC2dpJ	TX-04 r3	base	pass	—	3,216 / 855	0.00005
36	att_1ayr9izGeu	TX-04 r3	A	fail	wrong_index	7,243 / 653	0.00007
37	att_aMyM4ahq4B	TX-04 r3	B	pass	—	6,707 / 891	0.00007
38	att_ZxF5qdhGUT	PLAN-05 r1	base	invalid_output	not_json	2,988 / 545	0.00004
39	att_jAe3wiNsTr	PLAN-05 r1	A	fail	atomic_units_not_integer	6,485 / 577	0.00006
40	att_qchMxzgzS5	PLAN-05 r1	B	pass	—	6,288 / 493	0.00005
41	att_Z3TpgXBfuU	PLAN-05 r2	A	pass	—	6,419 / 499	0.00006
42	att_wZkMFrhRYR	PLAN-05 r2	B	pass	—	5,949 / 835	0.00006
43	att_XqHYJJCahA	PLAN-05 r2	base	pass	—	2,890 / 677	0.00004
44	att_NCyB1aJdjU	PLAN-05 r3	base	invalid_output	not_json	2,056 / 498	0.00003
45	att_fEwa4QBt2W	PLAN-05 r3	A	pass	—	5,892 / 580	0.00005
46	att_CUa1UoQGAV	PLAN-05 r3	B	fail	atomic_units_not_integer	6,899 / 321	0.00005
47	att_cJGHZC24sA	PLAN-06 r1	B	fail	non_idempotent_create	6,113 / 361	0.00005
48	att_wxfjCYV9iq	PLAN-06 r1	A	pass	—	7,065 / 523	0.00006
49	att_pVUjEhxCrs	PLAN-06 r1	base	fail	non_idempotent_create	2,328 / 790	0.00004
50	att_zo9D113iBn	PLAN-06 r2	base	fail	extra_instruction	3,077 / 803	0.00004
51	att_WVt2gZNq3V	PLAN-06 r2	B	pass	—	6,443 / 437	0.00005
52	att_oX5NPQjL1z	PLAN-06 r2	A	pass	—	6,723 / 304	0.00005
53	att_2ucx8ifnCg	PLAN-06 r3	base	fail	non_idempotent_create	2,067 / 732	0.00004
54	att_UjvvtaHhjx	PLAN-06 r3	B	pass	—	7,145 / 267	0.00005
55	att_gMXKJHxqzQ	PLAN-06 r3	A	invalid_output	not_json	6,109 / 888	0.00007
56	att_6tZ3KyGr4Anew	TX-03 r2	base	fail	alt_unresolved	3,178 / 425	0.00003

On chain

Status anchored

Memo tx

Slot372,283,500

Hash53574935…e25d4f

Cost

pricing 2026-11-01

Model usage (provider)0.00287 SOL

Runner fee0.01080 SOL

Held for unknown0.00000 SOL

Total0.0137 SOL (~$2.19)

Tokens in / out282,743 / 35,424

Provider usage is reported separately from the local estimate. Latency is the provider's, not the runner's.

Recheck

Re-score the stored answers with the trusted checker. No model is called.

Manifest

{
  "app": "skilltrial 0.9.4",
  "pack": {
    "id": "solana-core",
    "version": "1.0.3",
    "hash": "b3f1c09e7a52d84e6c1f0a9d27e5b8c4f31a06d9e2b7c58a40f6d13e9b2a7c61"
  },
  "checker": {
    "version": "checker 1.2.0",
    "hash": "4e8d2a91c0f7b36e5d18a4c92f07e63b1d5a8f2c9e04b7d61a3c8f5e2d9b0a47"
  },
  "skills": {
    "A": "wallet-plan-kit@0.4.0#2cdbe66837f5016de8e5d4def2088cc03fa5ec4e764eca0a1f75d728d0d06c1a",
    "B": "wallet-plan-kit@0.5.0#d541d481da16fe51b57353002a15e9a861f4c55a9052e7ce547302e1f9004d84"
  },
  "baseline": true,
  "model": {
    "id": "claude-haiku-5-5",
    "revision": "2026-10-14",
    "fingerprint": "fp_h55_3e7d41b8",
    "temperature": 0,
    "maxOutput": 2000,
    "timeoutS": 60
  },
  "systemPrompt": "tmpl_interpret_v3",
  "tools": [
    "fixture.read(id)"
  ],
  "repetitions": 3,
  "order": "arms alternated by seed",
  "seed": 10287,
  "budget": {
    "cap": 0.5,
    "currency": "SOL",
    "pricing": "2026-11-01"
  }
}