Skills vs. Always-Loaded Instructions: When Each One Actually Works

If your CLAUDE.md or AGENTS.md keeps growing, moving parts of it into Skills looks like an easy way to save context. There is a catch. A skill's full instructions reach the model only when the skill is invoked, and unless you invoke it yourself, that depends on the model judging it relevant. You trade predictable loading for a smaller prompt. Three published evaluations show when that trade pays off and when it quietly fails, and the vendor docs show how to tilt the odds.

This guide is for people who already run Claude Code, Codex or ChatGPT with a project instruction file and want a rule for what goes where. I have not run my own trigger-rate tests; everything below comes from the vendors' documentation and the three evaluations, each with its limits stated.

How a skill actually gets loaded

Anthropic introduced Agent Skills in October 2025 as folders of instructions, scripts and resources that an agent can find and load when a task calls for them. The open specification describes loading in three tiers:

1. Metadata (about 100 tokens per skill). Each skill's name and description load at startup. Hosts cap this listing, which matters once you have many skills (see the listing budget below).
2. The SKILL.md body. Loaded only when the skill is activated. The spec recommends keeping it under 5,000 tokens and under 500 lines, with detailed reference material moved to separate files.
3. Bundled resources. Reference files and scripts, read or executed on demand.

The important part is step two. Claude Code's skills documentation says of the description field: "Claude uses this to decide when to apply the skill." OpenAI's skills documentation describes the same pattern for ChatGPT and Codex: they begin with each skill's name and description and read the full SKILL.md once they decide to use it. So on both platforms, unless you call a skill by hand, its instructions reach the model only if a short description wins a relevance judgment.

In this guide, "always-loaded" means instruction files the host puts into context without that judgment. The scope is narrower than the name suggests. Claude Code's memory documentation says CLAUDE.md files in the directory hierarchy above where you start load at launch, while those in subdirectories load on demand. Whatever does load skips the relevance check, and it takes context for as long as it stays in the session. That is the trade.

Loading is not obedience, though. The same docs describe CLAUDE.md as "context, not enforced configuration," say there is no guarantee of strict compliance, and point to a PreToolUse hook when an action must be blocked regardless of what Claude decides. Neither an instruction file nor a skill is a permission or sandbox control.

Skill content also has limits once loaded. In Claude Code, an invoked skill's content enters the conversation as one message and stays there, and the file is not re-read on later turns, so guidance meant to apply throughout a task should be written "as standing instructions rather than one-time steps." When auto-compaction summarizes a long session, Claude Code re-attaches only the first 5,000 tokens of each skill's most recent invocation, within a 25,000-token budget shared by all skills, so older skills can drop out entirely. These are host-specific context rules, not a promise that skill content stays forever.

What the three evaluations found

Evaluation	What it measured	Headline result	Main limit
Vercel (January 2026)	A docs skill versus a docs index in AGENTS.md, on Next.js 16 APIs absent from model training data	Skill never invoked in 56% of cases; AGENTS.md index reached a 100% pass rate vs 53% for the skill by default	One vendor, one framework; Vercel's post does not state case count or model
SkillsBench (arXiv v4, June 2026)	Curated skills across 87 tasks, 8 domains, 18 model and harness configurations	Average pass rate rose from 33.9% to 50.5%	Skills curated for the benchmark; gains ranged from +4.1 to +25.7 points by configuration
SWE-Skills-Bench (arXiv v1, March 2026)	49 public software engineering skills on roughly 565 real-repository tasks	39 of 49 skills gave zero improvement; average gain +1.2%	Software engineering only; public skills, not ones written for your project

Vercel: the skill often never loaded

Vercel's eval tested agents on Next.js 16 APIs that are not in model training data, which is exactly the case where outside guidance should help most. In 56% of its eval cases, the skill was never invoked. The pass rates it reported were 53% with no added docs, 53% with the skill available by default, 79% with the skill plus explicit instructions, and 100% with a compressed docs index placed directly in AGENTS.md. Vercel squeezed that index to 8KB, which it describes as an 80% reduction.

Vercel's explanation is that the always-loaded index leaves "no decision point": it does not depend on the model choosing to fetch it. The underlying problem is easy to see. A model that does not know an API changed has no reason to go looking for a skill about it.

Vercel added two caveats worth keeping. It expects the gap may narrow as models improve at tool use, and it still sees a place for skills in narrow, action-specific workflows that users trigger explicitly, such as upgrading a Next.js version or migrating to the App Router. This is one company's eval on its own framework, so I read it as strong evidence about one failure mode, not a general law.

SkillsBench: curated, focused skills help on average

I used version 4 of the SkillsBench paper (June 14, 2026). It reports that curated skills raised the average pass rate from 33.9% to 50.5% across 87 tasks and 18 model and harness configurations, a gain of 16.6 points, with per-configuration gains between +4.1 and +25.7 points. Two further findings are directly useful: focused skills with no more than three modules beat larger or exhaustive bundles, and smaller models with skills can match larger models without them.

The first version of the paper (February 13, 2026, seven configurations) carried details the v4 abstract does not restate, so I treat them as that version's findings. It reported that effects ranged from +4.5 points for software engineering to +51.9 points for healthcare, that 16 of 84 tasks got worse, and that "Self-generated Skills provide no benefit on average."

SWE-Skills-Bench: most public coding skills did nothing

SWE-Skills-Bench (v1, March 16, 2026) took 49 publicly available software engineering skills and ran them against real repositories pinned at fixed commits. Thirty-nine of the 49 produced no pass-rate improvement, and the average gain was 1.2%. Token overhead ranged from small savings to a 451% increase with pass rates unchanged. Seven specialized skills produced meaningful gains of up to 30%, while three made things worse, by up to 10%, because guidance written for a different version conflicted with the project. The authors conclude that skills are "a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility."

Why these results do not contradict each other

It is tempting to read SkillsBench as "skills work" and the other two as "skills don't." They measured different things. Vercel measured whether the model chose to load a skill at all. SkillsBench measured curated skills paired with its tasks, and even there, its first version found the smallest gain in software engineering. SWE-Skills-Bench measured generic public skills dropped into real codebases. Together they point to two variables: whether the skill loads when it should, and whether its content fits the domain and the project. A skill fails if either one goes wrong.

The decision rule: what goes where

Claude Code's documentation gives the cleanest starting rule: make a skill when you keep pasting the same instructions, checklist or multi-step procedure into chat, or when a section of CLAUDE.md has become "a procedure rather than a fact." Because a skill's body loads only when used, long reference material costs little until you need it. Combining that with the evaluations, this is how I would split things:

Put it in always-loaded instructions	Put it in a skill
Facts and conventions that should shape every turn: build and test commands, code style, project rules	Multi-step procedures you run occasionally: release notes, migrations, a review checklist
Knowledge the model lacks and does not know it lacks, such as APIs newer than its training data (the Vercel case), ideally as a compact index pointing to fuller docs	Long reference material that only matters for one kind of task
Guidance you cannot afford to have skipped because the model misjudged relevance	Work that a bundled script can make deterministic; Anthropic notes Claude "can run deterministic code directly" instead of generating the result
Short, stable rules that fit your context budget	Actions with side effects that you want to trigger yourself, such as a deploy (in Claude Code, make them manual-only)

Neither column is the place for actions that must never happen. Text in either one is guidance the model can still misread, so hard limits belong in the host's permission settings or, in Claude Code, a hook. My agent permissions playbook covers that side.

The cost of the left column is real: everything there takes context for the whole session. Vercel kept the cost down by compressing its docs to an 8KB index rather than pasting the full documentation, and I recommend the same approach: keep always-loaded content to what the agent needs to know exists, and let it read details from files.

How to make a skill trigger when it should

Write the description for matching, not for humans

The open spec requires a description of at most 1,024 characters that "describes what the skill does and when to use it." OpenAI's guidance is more pointed: "Front-load the key use case and trigger words so a host can still match the skill if descriptions are shortened." Use the words people will actually type in a request, early in the description.

Watch the listing budget

Descriptions are not guaranteed to reach the model intact, and each host sets its own limit. Claude Code truncates the combined description and when_to_use text at 1,536 characters, and when you have many skills it drops some descriptions to fit a listing budget of 1% of the model's context window, which removes the keywords Claude needs to match your request. Codex caps its initial skills list at 2% of the context window, or 8,000 characters when the window is unknown. It shortens descriptions first and, for large skill sets, may leave some skills out of the initial list and show a warning; a skill it does select is still read in full. On either host, installing dozens of skills can make each one less likely to trigger. Prune the ones you do not use.

Ask for the skill when it matters

Vercel's 79% condition, the skill with explicit instructions, came from adding a line to AGENTS.md telling the agent to use the skill. Vercel's line asks the agent to explore the project structure before writing code, and only then to invoke its Next.js docs skill. That raised the pass rate from 53% with the skill simply available, though it stayed short of the always-loaded index. Vercel also found that the outcome shifted with subtle changes in wording, even with the same skill and the same docs. A blunt instruction that the agent must invoke the skill did worse than telling it to explore the project first and then invoke the skill.

If a task depends on a skill, I recommend a short line in your always-loaded file that points to the skill for that kind of task, written as a step in the workflow like Vercel's better-performing example, and checking in your own sessions that it actually fires. Naming the skill in your request yourself is another option, but Vercel's eval did not measure it.

Make side-effect skills manual-only in Claude Code

This is a Claude Code frontmatter field, so check your own host's docs before assuming the same control elsewhere. In Claude Code, setting disable-model-invocation: true means Claude cannot invoke the skill on its own; you run it with /name. The docs state that its description is not in context until you invoke it, which also frees listing budget.

Keep each skill focused

SkillsBench found that focused skills with at most three modules beat exhaustive bundles. Combined with the spec's 500-line guidance, the pattern is one job per skill, with reference material split into separate files.

Here is an illustrative pair of SKILL.md headers that apply these points. I wrote them for this article; they are not tested output.

---
name: release-notes
description: Draft release notes or a changelog from merged pull requests since the last tag. Use when asked for release notes, a changelog, or what shipped in a version.
---

---
name: deploy-staging
description: Deploy the current branch to staging.
disable-model-invocation: true
---

The first front-loads its trigger words. The second, in Claude Code, runs only when you type /deploy-staging.

Before you install someone else's skill

The evidence above gives two reasons for caution, and security gives a third.

• Most generic skills may add nothing. In SWE-Skills-Bench, 39 of 49 public skills gave zero improvement, and some cost far more tokens.
• Stale guidance can hurt. The three skills that degraded performance did so because version-mismatched guidance conflicted with the project. Check that a skill matches your framework versions.
• Skills can carry code. Anthropic recommends installing skills only from sources you trust, and auditing any skill from a less-trusted source by reading every bundled file, with particular attention to scripts and code dependencies. In Claude Code, also review the allowed-tools field of skills committed to a repository before you run Claude Code there, because a skill can grant itself broad tool access.

Asking a model to write your skills is not a reliable shortcut either. The first version of SkillsBench reported that self-generated skills gave no benefit on average.

Where to start

My recommendation is to keep facts and must-follow rules in CLAUDE.md or AGENTS.md, and to move procedures into skills only when you can accept that they will sometimes not load on their own. The smallest useful step: pick one section of your instruction file that is a procedure rather than a fact, move it into a skill with a specific, front-loaded description, and keep a copy of the original section. Then try the skill on a few realistic requests phrased the way you normally would, and check whether the procedure's specific steps show up in the agent's work. If they do not appear reliably, point to the skill from your always-loaded file, name it in your request, or restore the essential part of the original section.

For the broader Claude Code feature set that skills sit alongside, see my Claude Code 2.1 review.

Sources

• Anthropic engineering post introducing Agent Skills (October 16, 2025)
• Claude Code documentation on skills and on CLAUDE.md memory (living docs, read October 26, 2026)
• Agent Skills open specification
• OpenAI documentation on building skills for ChatGPT and Codex (living doc, read October 26, 2026)
• Vercel's AGENTS.md versus skills eval (January 27, 2026)
• SkillsBench, version 4 (June 14, 2026) and version 1 (February 13, 2026)
• SWE-Skills-Bench, version 1 (March 16, 2026)