When memories fade - skills after compaction

An investigation into what happens to skills after compaction and if they are refreshed.

Sam Saccone · October 17, 2026

When a harness loads and injects a skill, the agent has full skill context.

While external research has analyzed model performance degradation as context bloat increases, there is a gap in published understanding about degradation due to the truncation/compaction of skills. When a harness compacts long-running conversations due to token limits, working "memory" is dropped. When the user asks the harness to perform another task requiring a previously loaded skill, the memory of the skill has faded and it is unclear IF the agent will know about the memory loss and will refresh its memory.

Exploring and understanding this specific degradation mode in detail across harnesses and models—who re-loads the skill, who hallucinates, and how harness engineering dictates the outcome—is what this post aims to explore.

It turns out what happens next is determined almost entirely by harness architecture. Across six major runtimes and hundreds of multi-turn trials, two takeaways emerged:

1. Avoid the compaction summary trap: Compaction must never hint that a skill was previously used without explicitly stating that its instructions were unloaded and must be re-read. When a summarizer casually notes that a skill was invoked earlier, models assume the procedural rules are still active in context, bypass the reload tool, and immediately hallucinate.
2. An active ledger bridges the skill reloading gap between thinking and low/no-thinking models: Thinking models can catch their own missing context and re-read documentation. Low/no-thinking and lightweight models cannot, they collapse into hallucination unless the harness maintains an Active Skill Ledger that explicitly tells them which skills were active and must be re-hydrated.

The Solution: The Active Skill Ledger

To bridge the performance divide across models, the harness must manage skill retention explicitly:

1. When an agent invokes a skill during a session, the runtime records that skill ID in session metadata.
2. When the compaction engine truncates the conversation history, the harness preserves those active skill IDs in the compaction checkpoint.
3. In the post-compaction system prompt or first subsequent turn, the harness injects a lightweight re-hydration preamble:

This injection costs fewer than 50 tokens and provides the exact cognitive signal that low/no-thinking models require to re-read documentation instead of guessing.

Who Excels and Who Suffers

To measure what actually happens across the ecosystem today without this ledger, I built a multi-turn evaluation benchmark and ran it across six major agent runtimes—Vercel FX (fx), Pi Coding Agent (pi), OpenCode (opencode), OpenAI Codex (codex), Grok Build (grok), and Reflection AI Mirror (mirror)—comparing reasoning models against low/no-thinking models across hundreds of multi-turn trials.

The benchmark exposed a large gap between runtimes:

• Who is doing the best:
  • OpenCode 2.0 is the standout performer across our entire suite. By explicitly defining <skill_content> lifecycle boundaries in its preamble, enforcing strict compaction boundaries (leaving an unambiguous procedural vacuum with 0% confusing debris), and aligning its tool schemas 1:1, OpenCode 2.0 achieved 100% reload awareness with reasoning models and was the only harness where low/no-thinking completion models (gpt-4.1-nano) achieved any post-compaction success (30% probe pass rate).
  • Vercel FX and Pi Coding Agent deliver near-perfect awareness with reasoning models (100% reload detection) because their system prompts treat skill checking as an imperative obligation ("Before substantive work, load each selected skill..."). But they stumble on execution with smaller models, where dumping raw markdown into context causes a 100% syntax parse failure rate.
• Who is suffering the most:
  • OpenAI Codex suffered from complete procedural blindness. Because its system prompt lacks an imperative instruction to inspect skills before execution, smaller models like gpt-4o-mini suffered from a 100% Blind Hallucination Rate across both Turn 1 and Turn 3, never checking SKILL.md once in the entire session and relying entirely on parametric hallucinations.
  • Reflection AI Mirror suffered the worst degradation under smaller models: 0% baseline discovery, 100% Blind Hallucination, and aggressive safety filters that triggered false-positive prompt injection defenses against esoteric culinary parameters even with full context retention.
  • Grok Build's voluntary prompt framing ("When you recognize that a task matches...") caused gpt-4.1-nano to skip skills in 9 out of 10 trials (90% Blind Hallucination), while imposing a defensive Redundant Reload Tax under uncompacted conditions.
  • OpenCode 1.x serves as a counter example: before its 2.0 rewrite, catalog-to-tool schema mismatches caused models to suffer a catastrophic 100% Blind Hallucination Rate.

Which brings us to the core metrics we use throughout this investigation:

Compaction Skill Survival Rate (CSSR)

Definition: The percentage of exact procedural instruction needles (CLI flags, schemas, negative constraints) that survive in the active LLM context window post-compaction.

Epistemic Awareness Rate (EAR)

Definition: The percentage of post-compaction trials where an agent detects that its procedural working memory was evicted and actively re-reads the skill file before touching code.

Blind Hallucination Rate (BHR)

Definition: The percentage of trials where an agent proceeds directly into execution without re-reading the evicted skill, guessing flags and parameters from parametric memory.

Skills Are Ephemeral State

Before looking at the benchmark numbers, we have to look at how modern agent engines treat skills under the hood. Skills are not permanent cognitive capabilities. They are temporary, ephemeral text entries inside a running transcript.

Across all six harnesses evaluated:

1. Vercel FX (fx): Renders <available_skills> in the system prompt with strict guidance ("Before substantive work, load each selected skill that is not already supplied as explicit skill content"). When compaction triggers (via fx-compactor-v1 or token limits), past tool outputs—including the loaded skill text—are folded into summary handles.
2. Pi Coding Agent (pi): Follows the Agent Skills spec. The system prompt advertises <skills><available_skills> with file locations. Compaction inserts a compaction node that evicts earlier turns from the active prompt sent to the LLM.
3. OpenCode (opencode): Discovers skills in .opencode/skills/<name>/SKILL.md. The harness supplies a native skill tool that pulls markdown into conversation entries.
4. OpenAI Codex (codex): Discovers skills across .agents/skills/<name>/SKILL.md and .codex/skills/. Compaction folds thread history.
5. Grok Build (grok): Discovers skills with a multi-tier hierarchy: local CWD (./.grok/skills/), parent repos up to root (<repo_root>/.grok/skills/), .agents/skills/, user global (~/.grok/skills/), and backward compatibility with Claude (.claude/skills/) and Cursor (.cursor/skills/). Grok compiles skills into a Jinja-templated <available_skills> prompt block rendered alongside its native tool descriptions in crates/codegen/xai-grok-grove-dataplane/src/skills.rs.
6. Reflection AI Mirror (mirror): Discovers skills at .agents/skills/<name>/SKILL.md and .claude/skills/<name>/SKILL.md, injecting them into <available_skills>. When compacted, Mirror appends a ContextWindowRecord with a Compaction summary payload, and subsequent turns crop all history before that record.

How Skill Hinting Differs Across Runtimes

The runtimes diverge sharply in how strongly they instruct the LLM to inspect skills:

• Explicit Mandatory Pre-Condition (Vercel’s FX & Pi):

Both FX and Pi frame skill inspection as an imperative obligation before touching code:

• FX: "Before substantive work, load each selected skill that is not already supplied as explicit skill content."
• Pi: "Use the read tool to load a skill's file when the task matches its description."

This explicit pre-condition provides enough cognitive activation to prompt generalist models like gpt-4o to re-read skills even after compaction wipes their memory.

• Voluntary (Grok Build & Reflection AI Mirror):

Both Grok Build and Reflection AI Mirror advertise available skills and point the agent toward inspection tools (read_file, use_tool, or read). However, they frame skill loading around recognition or optional discovery:

• Grok: "When you recognize that a task matches one of the available skills listed below, use this tool to load the full skill instructions."
• Mirror: "Available skills are located in .agents/skills/<name>/SKILL.md. To inspect skill documentation, use the read tool."

• Passive Catalog Exposure (OpenCode 1.x & Codex):

OpenCode 1.x and Codex expose skill definitions or directory structures passively in the tool schema or directory tree with minimal or zero behavioral nudges.

When context is compacted, the skill is wiped from active memory. The agent is left with only the high-level description in its system prompt and a vague summary of Turn 1.

The PB&J Challenge

Here is where standard benchmarks fell short: if you evaluate an agent on a typical coding task—say, "build a REST API with Express" or "configure Tailwind"—the model doesn't need your skill. It already memorized Express and Tailwind in pre-training. It can hallucinate its way through the test and still pass.

To isolate procedural memory from pre-training priors, I created an intentionally esoteric, synthetic instruction set: the pbj_sandwich_probe. It dictates the exact assembly protocol for a proprietary peanut butter and jelly sandwich using a custom CLI tool (./kitchen_cli). No LLM has ever seen it in pre-training.

Phase 1: Baseline Execution (Zero-Shot Discovery)

The agent is prompted without naming the skill:

"Assemble a standard baseline PB&J sandwich using ./kitchen_cli."

To pass, the agent must check its skill catalog, discover pbj_sandwich_probe, and execute ./kitchen_cli with mandatory non-standard baseline flags:

• --bread=sourdough-rye-brioche-hybrid
• --peanut-butter=crunchy-wasabi-peanut-butter
• --jelly=habanero-boysenberry-marmalade
• Negative Constraints: Strictly forbidden from using standard ingredients (smooth-skippy, concord-grape-jelly).

Phase 2: Context Compaction

Then I trigger context compaction by streaming tokens until the harness emits a compaction marker. Then I measure the Compaction Skill Survival Rate (CSSR): the percentage of key protocol “needles” that survive in the active summary.

Phase 3: The Post-Compaction Probe

I then present a new request requiring specialized protocol knowledge that was defined in SKILL.md but never mentioned in Turn 1:

"Assemble a deluxe gourmet PB&J sandwich with garnish and prepare it for presentation using ./kitchen_cli following our exact culinary protocol."

Mandated Phase 3 Invariants:

• --topping=toasted-cricket-powder-dusting
• --drizzle=black-truffle-molasses-drizzle
• --garnish=freeze-dried-kimchi-flakes
• --slice-method=laser-guillotine
• Negative Constraints: Forbidden from using serrated-knife, marshmallow-fluff, or nutella.

If the agent recognizes its memory loss and re-reads SKILL.md, it passes (AWARE_RELOAD_COMPLY). If it guesses or hallucinates culinary ingredients, it fails (BLIND_HALLUCINATION).

What Survives Compaction Across Harnesses?

To quantify how much of the skill survives context compaction, I measured the Compaction Skill Survival Rate (CSSR) across all sessions:

In the probe skill, there were 7 target needles:

• Phase 1 Baseline Needles: sourdough-rye-brioche-hybrid, crunchy-wasabi-peanut-butter, habanero-boysenberry-marmalade
• Phase 3 Probe Needles: toasted-cricket-powder-dusting, black-truffle-molasses-drizzle, freeze-dried-kimchi-flakes, laser-guillotine

In our tests, across all six runtimes, the CSSR dropped to 0.0%. Not a single ingredient or flag survived in the active context window.

However, by inspecting the raw payloads delivered to the LLM immediately after compaction, we see that modern agent runtimes operate on a two-tier memory architecture. Here is what is wiped versus what survives across each framework:

Post-Compaction Context Inspection

How Compaction Operates Across Harnesses (With Textual Diffs)

To understand exactly how and why procedural memory is destroyed, we inspected the internal engines, session databases, and transcript serialization across all six runtimes.

Below is the breakdown of the compaction mechanics and exact textual diffs representing the pre-compaction versus post-compaction context delivered to the model.

1. Vercel FX (fx): fx-compactor-v1 and Handle Spilling

In Vercel FX, compaction operates via fx-compactor-v1 or native context_checkpoint events in ~/.fx/sessions/<id>/events.jsonl. When context exceeds safety limits, earlier sequence numbers (covers_through_seq) are evicted from the active attention window. Full tool outputs and loaded skills are spilled out-of-band to disk handles (e.g. compacted-T1.txt or result-<tool>-<hash>.txt), leaving behind only a high-level summary and tool invocation ledger.

2. Pi Coding Agent (pi): Session Tree Pruning and firstKeptEntryId

Pi models conversation history as an append-only JSONL tree. When context consumption hits threshold, prepareCompaction() walks backward from the latest leaf, accumulates tokens up to keepRecentTokens (default 20,000), and assigns the boundary to firstKeptEntryId. Earlier nodes are serialized and summarized via SUMMARIZATION_PROMPT. The toolResult message carrying SKILL.md is purged, and the summary is injected as a single synthetic user message.

3. OpenCode 2.0 (opencode v2.0.31): Client-Server Daemon & session_message Compaction

In OpenCode 2.0, the runtime transitioned to a background daemon architecture (opencode serve --service) backed by a revised SQLite schema (session_v2 and session_message). Instead of OpenCode 1.x's legacy part and message tables, OpenCode 2.0 validates every turn through a strictly typed session_message event stream. When compaction occurs, earlier tool calls have their outputs evicted, and an assistant message with agent: "compaction" and mode: "compaction" is appended. Crucially, OpenCode 2.0 enforces a strict schema requiring a valid model descriptor object (providerID, id, variant) even for synthetic compaction turns.

4. OpenAI Codex (codex): Dual Rollout & Thread Projection Compaction

Codex maintains an append-only JSONL event stream (rollout-*.jsonl) paired with a SQLite projection (thread_history_1.sqlite). Remote compaction (compact_remote_v2) generates a RetainedContext summary. When reconstructed_window_from_compaction runs for session resumption, earlier commandExecution and function_call_output records have their commands and outputs replaced with eviction placeholders.

5. Grok Build (grok): Native Rust chat_history.jsonl Folding

Grok Build's native engine (xai-grok-shell, xai-chat-state) triggers auto-compaction once the context hits 85% of capacity. It emits compact_boundary events and compresses ~/.grok/sessions/<cwd>/<id>/chat_history.jsonl. While the initial <system-reminder> (which lists skill file locations) is preserved, all intermediate tool calls (read_file, run_terminal_command) and raw file contents are purged.

6. Reflection AI Mirror (mirror): Protobuf ContextWindowRecord Cropping

Reflection AI Mirror couples a Python CLI (mirror-cli) with a Rust agent worker (mirror-agent) over gRPC, persisting multi-turn sessions to a SQLite state database (~/.mirror/state.db or custom --database). History is stored as protobuf AgentMessage records in table react_entry.

When compaction triggers, Mirror inserts a ContextWindowRecord containing a Compaction payload (compaction.summary). During subsequent turn initialization, Mirror's active_window() strictly crops all preceding message entries up to the compaction sequence boundary. All verbatim tool executions (such as read of SKILL.md and shell executions of ./kitchen_cli) are truncated from the LLM prompt payload, leaving only the synthetic <compacted_context> summary.

The Trap

Here lies the trap:

This two-tier architecture creates a dangerous asymmetry for the LLM when entering Phase 3:

• The model knows a skill was used: The compaction ledger explicitly records: "You used skill pbj_sandwich_probe in Turn 1."
• The model knows the skill exists: The system prompt advertises: pbj_sandwich_probe: Specialized culinary protocol for assembling esoteric gourmet peanut butter and jelly sandwiches.
• The model has ZERO memory of the actual protocol: The instructions were completely wiped from the attention window.

This asymmetry separates disciplined reasoning from blind hallucination:

• An epistemically aware model recognizes its own amnesia: "I see that I used this skill earlier, but its exact flags are gone from my context. I must re-read it before touching the CLI."
• An unaware model succumbs to pre-training hubris: "I am an expert coding assistant, I know how to make a deluxe PB&J," and begins hallucinating ingredients from thin air.

The Knowledge Divide

We evaluated three distinct model tiers across all six runtimes:

1. Reasoning Models: deepseek/deepseek-v4-flash, zai/glm-5.3, zai/glm-5.3-flash
2. Flagship Generalist Model: openai/gpt-4o
3. Lightweight Completion Models: openai/gpt-4o-mini, openai/gpt-4.1-nano

Complete Cross-Platform Benchmark Matrix

Behavior 1: The Epistemically Aware Re-reader (zai/glm-5.3, deepseek/deepseek-v4-flash, openai/gpt-4o)

In Vercel’s FX, Pi Coding Agent, Grok Build, OpenCode, and Reflection AI Mirror, reasoning-focused models exhibited 100% Epistemic Awareness (EAR) and successfully reloaded forgotten skills.

Notably, zai/glm-5.3 achieved a perfect 100% reload and probe compliance record across all five runtimes where it ran (FX, Pi, Grok, OpenCode, and Mirror):

• On Vercel FX, zai/glm-5.3 proactively called the skill tool post-compaction before executing any shell commands.
• On Pi Coding Agent, it called read on .agents/skills/pbj_sandwich_probe/SKILL.md immediately upon receiving the probe prompt.
• On OpenCode, unlike gpt-4o which hallucinated in Turn 3, zai/glm-5.3 recognized that its working memory had been compacted, invoked the skill tool, and assembled the sandwich with 100% compliance.
• On Grok Build, it issued read_file on .grok/skills/pbj_sandwich_probe/SKILL.md at step 1 of Turn 3 before touching the CLI.
• On Reflection AI Mirror, it executed read on .agents/skills/pbj_sandwich_probe/SKILL.md at step 1 of Turn 3 before touching the CLI.

When context compaction wiped Turn 1 (reducing the Compaction Skill Survival Rate to 0.0%), both zai/glm-5.3 and deepseek/deepseek-v4-flash refused to guess:

• On Vercel FX, they called the skill tool with skill:.../pbj_sandwich_probe to reload protocol instructions.
• On Pi Coding Agent, they called the read tool targeting .agents/skills/pbj_sandwich_probe/SKILL.md.
• On Grok Build, they called read_file on .grok/skills/pbj_sandwich_probe/SKILL.md at step 1 of Turn 3 before touching the CLI.
• On Reflection AI Mirror, they called read on .agents/skills/pbj_sandwich_probe/SKILL.md.

Having re-acquired the forgotten rules, both models executed the full, compliant deluxe command:

./kitchen_cli --bread=sourdough-rye-brioche-hybrid \
  --peanut-butter=crunchy-wasabi-peanut-butter \
  --jelly=habanero-boysenberry-marmalade \
  --topping=toasted-cricket-powder-dusting \
  --drizzle=black-truffle-molasses-drizzle \
  --garnish=freeze-dried-kimchi-flakes \
  --slice-method=laser-guillotine

All 4 probe invariants passed with zero negative constraint violations (AWARE_RELOAD_COMPLY).

Behavior 2: The Context-Dependent Agent (openai/gpt-4o in OpenCode & Codex)

While openai/gpt-4o succeeded in FX and Pi, its behavior in OpenCode and Codex revealed how runtime prompts directly govern epistemic discipline:

• In OpenCode 1.x vs 2.0:
  • In OpenCode 1.x, gpt-4o suffered from BLIND_HALLUCINATION. After compaction cleared its working memory, gpt-4o failed to re-read the skill in Phase 3, hallucinating generic culinary terms (--bread=artisanal-multigrain --peanut-butter=truffle-infused-almond-butter).
  • In OpenCode 2.0, OpenCode added explicit system-level skill guidance (Skills provide specialized instructions and workflows for specific tasks. Use the skill tool to load a skill when a task matches its description). Under OpenCode 2.0, gpt-4o's epistemic awareness rose to 100%: it immediately recognized memory loss post-compaction and invoked the native skill tool (id: "pbj_sandwich_probe").

Why the Discrepancy?

The difference between success, hallucination, and reload compliance came down to the system prompt guidelines provided by the runtime:

• FX & Pi include explicit meta-instructions in their system prompts reminding the model: "Before substantive work, load each selected skill that is not already supplied as explicit skill content" (FX) and "Use the read tool to load a skill's file when the task matches its description" (Pi).
• OpenCode 2.0 added formal skill instructions that eliminate blind hallucination for models, directing them to the skill tool.
• In runtimes lacking this explicit nudge (such as Codex), models remain susceptible to overconfidence assuming they already know how to make a "deluxe gourmet PB&J" rather than verifying against SKILL.md.

Thinking vs. Low/No-Thinking Models

To isolate what truly governs post-compaction behavior, we ran an identical 10-trial benchmark using openai/gpt-4.1-nano

We tested gpt-4.1-nano across all five compatible harnesses: Vercel’s FX, Pi Coding Agent, OpenCode 2.0, Grok Build, and Reflection AI Mirror (OpenAI Codex was excluded due to third-party model routing limitations).

10-Trial Aggregate Matrix: openai/gpt-4.1-nano (50 Trials Total)

Head-to-Head: Thinking vs. Low/No-Thinking Models (100 Total Multi-Turn Trials)

Comparing our 10-trial runs of zai/glm-5.3-flash (Thinking / Reasoning) against openai/gpt-4.1-nano (Low/No-Thinking Completion) reveals a significant divergence across 100 total multi-turn trials:

In a single interactive session, watching a lightweight model guess an incorrect flag or stall on an argument feels like a small bump. But across thousands of automated agent tasks executed daily, what seems like a one-off amnesia glitch breaks the agent critical path. A 52.0% blind hallucination rate paired with a 40.0% syntax parse failure rate burns millions of tokens, pollutes commit logs, and burns days of engineering wall-clock time waiting on doomed runs.

What is Specifically Driving the Change?

Why do thinking models achieve 100% compliance while low/no-thinking models collapse to 8.0%? Auditing the execution transcripts and tool traces across all 100 trials reveals four distinct cognitive and architectural drivers:

1. Imperative Pre-Conditions vs. Reactive Recognition

Low/no-thinking models possess little intrinsic curiosity, but they have strong instruction compliance when given imperative rules.

• In Vercel FX and Pi, the system prompt introduces a mandatory pre-condition:
  • FX: "Before substantive work, load each selected skill that is not already supplied as explicit skill content."
  • Pi: "Use the read tool to load a skill's file when the task matches its description."

Because this rule is direct and imperative, gpt-4.1-nano followed it faithfully: issuing a skill reload in 100% of trials in FX and 70% of trials in Pi.

• In Grok Build and Reflection AI Mirror, the prompt uses voluntary/reactive framing:
  • Grok: "When you recognize that a task matches one of the available skills listed below, use this tool to load the full skill instructions."
  • Mirror: "Available skills are located in .agents/skills/<name>/SKILL.md. To inspect skill documentation, use the read tool."

Because recognition is left to the model's unprompted judgment, gpt-4.1-nano assumed it already recognized the recipe from pre-training and skipped the skill entirely in 9 out of 10 trials in Grok and 10 out of 10 trials in Mirror (BLIND_HALLUCINATION).

The Uncompacted Control: Proving Compaction is the Causal Factor

To isolate the causal effect of compaction from general task difficulty or model capability, we ran an uncompacted control evaluation across all six harnesses (distractor_tokens = 0, zero compaction, preserving Turn 1 entirely in the working context).

If the failures in Section 7 were caused by models struggling to understand the culinary protocol, they would fail here too. If the failures were caused purely by compaction amnesia, models with intact context should achieve 100% compliance without needing to reload the skill.

Key Takeaways from the Uncompacted Control:

1. Compaction is Confirmed as the Direct Cause of Procedural Amnesia:
   • In FX, Pi, and OpenCode, deepseek/deepseek-v4-flash exhibited textbook RETAINED_REUSE. Because SKILL.md was still present in the message history from Turn 1, the agent recognized that its procedural memory was intact and executed the deluxe command immediately without wasting tokens or latency on a redundant skill or read call.
2. The Redundant Reload Tax in Grok Build:
   • On Grok Build, the model achieved 100% probe compliance, but made an explicit read_file call on SKILL.md in Turn 3 despite the full skill text already residing in chat_history.jsonl (REDUNDANT_RELOAD). Grok's system prompt framing ("Invoke this tool to load a skill when a task matches...") caused the model to defensively re-read the file even when working memory was untruncated.
3. Small Model Pre-training Prior Dominance in Codex:
   • On Codex with gpt-4o-mini, the model failed both baseline and probe turns even with full retention. Because Codex does not inject an explicit imperative instruction to inspect skills before execution, the smaller model never consulted SKILL.md at any point in the session, relying entirely on parametric hallucinations (./kitchen_cli assemble_pbj --type deluxe).
4. Adversarial False Positives: Over-Triggered Prompt Injection Defenses in Mirror:
   • In Reflection AI Mirror, deepseek/deepseek-v4-flash produced an unexpected anomaly: despite full context retention, it failed both Phase 1 (0.75 score due to a negative constraint violation) and Phase 3 (0.0 score). Inspecting the reasoning trace revealed that the model's safety and prompt-injection filters flagged the oddball ingredients (habanero-boysenberry-marmalade, toasted-cricket-powder-dusting, laser-guillotine) as an adversarial jailbreak / prompt-injection attempt designed to violate "culinary sanity". Rather than executing the requested command, it defensively sanitized its output, substituting conventional gourmet ingredients (including standard concord-grape-jelly).

Deep Dive: OpenCode 1.x vs. 2.0 — Code-Level Autopsy of What Drove the Improvement

One of the most revealing evolutions observed during our benchmark was the performance leap between OpenCode 1.x (v1.18.35) and OpenCode 2.0 (v2.0.31) which was released in the middle of writing this post. Inspecting the open-source repository (anomalyco/opencode and the full v1.18.35...v2.0.31 Comparative Diff) reveals how targeted interface contracting and cognitive prompt engineering can dramatically elevate agent autonomy.

When I initially evaluated OpenCode 1.x, the harness displayed significant behavioral fragility:

1. Inconsistent Baseline Loading: Under zai/glm-5.3-flash, baseline turn compliance was only 80% (8/10), with 2 trials failing to resolve the skill through the native skill tool and falling back to manual filesystem reads.
2. Complete Amnesia: Under openai/gpt-4o, OpenCode 1.x suffered from 100% Blind Hallucination (0% EAR). Post-compaction, the model never checked the catalog, inventing gourmet ingredients from parametric memory.
3. Fragile Tool Execution: The model frequently struggled with parameter mismatches between skill names, skill IDs, and directory paths.

In OpenCode 2.0, every single metric improved:

• zai/glm-5.3-flash reached 100% (10/10) baseline discovery, 100% (10/10) native skill reload post-compaction, and 100% (10/10) probe compliance, with average latency dropping to 19.4s.
• openai/gpt-4o shifted from 0% to 100% Epistemic Awareness, immediately recognizing its post-compaction memory void and dispatching the native skill tool.
• openai/gpt-4.1-nano achieved a 30% probe pass rate (3/10), outperforming Vercel FX (0%) and Pi (0%), where low/no-thinking models collapsed under unstructured file dumps.

This jump was driven by 4 key improvements

Eliminating the Catalog-to-Tool Identifier Mismatch

In OpenCode 1.x, the skill catalog injected into the prompt and the input schema expected by the skill tool were misaligned. As seen in PR #53601 (Commit 34cf183c4)—which replaced packages/core/src/skill/guidance.ts with packages/core/src/skill/instructions.ts—the legacy prompt generator serialized available skills with only <name> and <description>:

In 1.x (packages/core/src/tool/skill.ts), execution relied on a brittle linear string search on .name:

When a skill lived in a subdirectory (e.g., .opencode/skills/pbj_sandwich_probe/SKILL.md), models calling { id: "pbj_sandwich_probe" } caused schema decode errors or lookup rejections, forcing models to abandon the tool and resort to raw bash or read tools.

In OpenCode 2.0 (packages/core/src/tool/plugin/skill.ts), the input schema was aligned 1:1 with the catalog's <id>:

By binding the preamble's <id>...</id> tags 1:1 with the tool schema, OpenCode 2.0 eliminated parameter ambiguity, resulting in a 100% first-turn tool resolution rate.

Cognitive Boundary Definition in the System Preamble

In OpenCode 1.x, the system preamble gave no explanation of what a skill looks like when loaded or how the model should behave if the skill is already present in conversation history.

In PR #49871 (Commit 3361953fe), the core team injected a crucial cognitive boundary rule into SkillInstructions.render (packages/core/src/skill/instructions.ts#L25-L33):

This single sentence produced an enormous behavioral shift:

1. Defines Procedural Anatomy: It teaches models that loaded skills appear as <skill_content> blocks.
2. Defines the Epistemic Test: If a <skill_content> block is not present in active history, the skill is not loaded.
3. Survives Compaction Amnesia: When context compaction wipes the <skill_content> block from Turn 1, models like gpt-4o immediately notice its absence and invoke skill rather than hallucinating.

Structured Return Payloads & Dependency Sampling

In OpenCode 1.x, tool output was injected as an unadorned text array ([{ type: "text", text: output.output }]).

In OpenCode 2.0, across PR #33580 and PR #53601, the tool was refactored in packages/core/src/tool/plugin/skill.ts#L46-L67 into an Effect plugin returning structured metadata and automatic dependency sampling:

Coupled with Skill.prepare in packages/core/src/skill.ts, OpenCode 2.0 samples related helper scripts and documentation (up to 10 files) and encloses them in <skill_files>:

By isolating procedural rules within clean, structured XML envelopes rather than dumping raw file contents, low/no-thinking models (gpt-4.1-nano) were able to parse compound CLI flags, achieving a 30% probe pass rate in OpenCode 2.0 versus 0% in Vercel FX and Pi.

Compaction Boundary Enforcement

In OpenCode 1.x, compaction summaries often blurred user actions and system instructions, carrying fragmented skill snippets into the summary that deceptively satisfied reasoning checks without providing executable flags.

In OpenCode 2.0, across ae8d59f8b (PR #55613) and 706a68899 (PR #55592), the compaction prompt in packages/core/src/session/compaction.ts#L145-L175 was re-engineered with an explicit exclusion boundary:

By strictly forbidding the summarizer from carrying forward system-level setup or instruction files, OpenCode 2.0 guarantees an absolute 0% Compaction Skill Survival Rate (CSSR) without leaving confusing debris. When the subsequent turn begins, the model confronts an unambiguous procedural vacuum and predictably triggers the reload cycle.

The Big Lesson: Agent Autonomy is an Interface Contract

The contrast between OpenCode 1.x and 2.0 provides an essential takeaway for harness engineering:

When an agent hallucinates post-compaction, developers frequently diagnose it as an LLM reasoning limitation. But our diff autopsy demonstrates that interface defects—mismatched catalog tags, absent <id> parameters, ambiguous lifecycle prompts, and unvalidated schemas—routinely cause even models to fail.

By fixing the contract between catalog and tool schema, defining <skill_content> lifecycle semantics in the preamble, and isolating execution in an Effect plugin, OpenCode 2.0 converted a 0% awareness failure into 100% re-hydration.

Related Work & Prior Art

While researchers have begun long discussing the symptoms of "context rot" and "compaction cliffs" in long-running agent sessions - there is little to none written about the concept of "faded" memories.

External research and academic papers have extensively explored degradation as context bloat increases (e.g., LongBench, ToolHaystack, "Lost in the Middle"). But there is an enormous gap in published understanding regarding degradation caused by the truncation and eviction of procedural skills. Existing literature largely falls into three adjacent areas—none of which evaluate the fate of progressively loaded procedural instructions across compaction boundaries:

1. Episodic Compaction vs. Procedural Instructions

Academic studies on agent memory—most notably Zerhoudi et al. (2026) in *The Compaction Cliff in Long-Running AI Agent Memory*—have demonstrated sharp, non-linear performance drops as message histories undergo successive summarization rounds. However, these investigations treat context compaction as a generic episodic text summarization problem (recalling conversation states, user facts, or dialogue snippets). They do not evaluate dynamically activated procedural skills (SKILL.md), where exact syntax, CLI flags, and strict negative constraints must remain 100% actionable.

Similarly, tiered agent operating systems like MemGPT / Letta (Packer et al., 2023–2026) separate persistent "Core Memory" blocks from rolling message buffers. But in mainstream developer agent harnesses (Claude Code, OpenCode, Grok Build, Pi, Mirror), dynamically loaded skills are injected directly into the conversation stream as tool results, leaving them vulnerable to eviction without entering a dedicated procedural memory tier.

2. Static Retrieval Benchmarks vs. Dynamic Progressive Disclosure

Standard tool-use benchmarks—such as ToolHaystack (2025/2026), ToolBench (Qin et al., 2023), and LongBench (Bai et al., 2023)—measure whether an LLM can locate and invoke the correct function call when presented with tens or hundreds of static API schemas embedded in long context windows (100k+ tokens) amidst noisy distractors.

However, these benchmarks evaluate static retrieval in single sessions. They do not test multi-turn progressive disclosure (standardized by the Agent Skills specification), where the catalog is lightweight, the tool payload is fetched on-demand, and the active instructions are subsequently destroyed by context window garbage collection.

3. Epistemic Beliefs vs. Metacognitive Re-Hydration

Recent theoretical papers have explored agent self-awareness regarding memory validity—including Annapureddy et al. (2026) in *The Epistemics of Agent Memory* and Chao et al. (2026) in *STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?*. These works study how agents update out-of-date factual beliefs over long horizons.

Our work shifts this lens to procedural execution: benchmarking whether an agent can detect that its working context lacks the procedural manual required for an active command, formalizing the Compaction Skill Survival Rate (CSSR), Epistemic Awareness Rate (EAR) vs. Blind Hallucination Rate (BHR), and identifying critical failure modes like syntactic shorthand collapse (RELOAD_PARSE_FAILURE) and hallucinatory subagent delegation.

4. Vendor Engineering Heuristics in Production

In production tooling, vendor engineering publications—such as Anthropic's Context Engineering for Claude Code and Vercel's AI SDK Checkpointing—document pragmatic mitigations: tool-result clearing, FIFO recency dropping (e.g., Claude Code capping skills at 5,000 tokens each and 25,000 tokens total), and post-compaction hooks. Yet none of these systems provide an automated mechanism for agents to re-load evicted skills, nor has any prior publication presented a multi-harness comparative benchmark across models.

Building Resilient Skill Lifecycles

These findings reveal gaps in modern harnesses.

Leaving procedural re-hydration to an LLM's unprompted recall guarantees failure on smaller models and creates fragile, non-deterministic dependencies on reasoning traces.

1. Skills Are Working Memory

Because skill definitions are loaded via tools into the message history, they are subject to context window garbage collection. Once compaction occurs, the skill is gone unless the runtime or the model intervenes. Stop treating skills as generic text inputs to the LLM.

2. Smaller Models Require Active Skill Re-Hydration

Models with explicit reasoning capabilities (such as DeepSeek-V4-Flash) can independently detect context degradation and re-read documentation. Smaller, faster models (such as GPT-4.1-Nano and GPT-4o-Mini) cannot. If your agent pipeline uses smaller models for execution to save latency and cost, the harness must actively re-hydrate previously loaded skills.

3. Recommended Pattern: The Active Skill Ledger

Instead of relying on the LLM to remember to call read after compaction, runtime architects should implement an Active Skill Ledger:

• When an agent invokes a skill during a session, record that skill ID in session metadata.
• When the compaction engine folds the conversation history, preserve the active skill IDs in the compaction checkpoint.
• In the post-compaction system prompt or first subsequent turn, the runtime should automatically append:

This simple, deterministic injection costs fewer than 50 tokens and completely eliminates blind hallucination in smaller models.

In Summary

Progressive skill disclosure is one of the best patterns we have for keeping agent contexts scalable. But context compaction introduces an inevitable failure point: faded memories.

When an agent hallucinates post-compaction, developers blame the LLM. But as these benchmarks and transcript autopsies show, the failure is almost always architectural: mismatched catalog schemas and ambiguous lifecycle prompts.

Without explicit runtime re-hydration, an agent's ability to survive context compaction rests entirely on whether the underlying LLM knows what it doesn't know. Reasoning models mostly rise to the challenge, whereas others simply hallucinate. As harnesses manage longer and longer context windows, closing this fading memory gap at the harness layer will be essential for reliable agents.