tim/SlipItIn
1
0
Fork 0
Code Pull Requests 1 Actions 1 Packages Releases Wiki Activity

Neue Skills, Referenzen & OpenWiki-Doku integriert

Browse Source 
Umfangreiche Erweiterung der Skill-Bibliothek: Neue Skills für Humanisierung (Englisch/PT-BR), Design-Validierung, AI-SEO und Coolify-Deployment inkl. Regelwerke, Presets, Pattern-Referenzen, Testfälle und Automatisierungsskripte. Zusätzliche Skills für Revenue-Centric Design, Pier Cloud, OKF, Lebenslauf- und LinkedIn-Optimierung sowie zahlreiche Referenzdateien, Checklisten und YAML/JSON/Markdown-Templates. Einführung einer vollständigen OpenWiki-Dokumentation mit Architektur-, Domain- und Workflow-Beschreibungen, zentralem Index und automatisierten Updates. Modularer Aufbau, restriktive Lizenzen und umfassende Qualitäts- und Evaluationsmechanismen für alle neuen Inhalte.
This commit is contained in:
Tim Krampitz
2026-07-26 14:00:58 +02:00
parent 070727d5cd
commit 01046b01e4
202 changed files with 31290 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

299
.github/skills/skill-evaluation/SKILL.md vendored Normal file
View File

@@ -0,0 +1,299 @@
---
name: skill-evaluation
description: >
Evaluate any agent skill against a merged framework — Anthropic's Claude Code
best practices plus Matt Pocock's writing-great-skills methodology — across
4 axes (Trigger, Structure, Steering, Pruning). Produces an evidence-cited
scorecard (0–100), a weighted overall score, and diagnosed failure modes
with prioritized fixes. Use when the user asks to evaluate, rate, or audit
a skill ("evaluate this skill", "skill scorecard", "review SKILL.md"), or
to compare two skills.
metadata:
author: ft.ia.br
version: "2.1.0"
date: 2026-07-03
repository: https://github.com/fabricioctelles/skills
license: Apache-2.0
category: code-quality-and-review
---
# Skill Evaluation
If you need the vocabulary and tests behind Axes 1, 3, and 4 (leading words,
completion criteria, context pointers, the deletion test, failure-mode
definitions), read `references/mechanics.md` before scoring those axes.
## Source
- [Lessons from building Claude Code: How we use skills](https://claude.com/blog/lessons-from-building-claude-code-how-we-use-skills) — Anthropic, Jun 2026
- "The Missing Manual: How to Write Great Skills" — Matt Pocock, AI Engineer World's Fair 2026 ([video](https://www.youtube.com/watch?v=UNzCG3lw6O0)), and his `writing-great-skills` skill
## Parameters
| Parameter | Description | Default |
|-----------|-------------|---------|
| `target` | Path to skill directory or SKILL.md to evaluate | Ask user |
| `output` | Path to write the scorecard | `<target>/EVALUATION.md` |
| `compare` | Optional second skill to compare side-by-side | None |
Also runs unattended: in CI, point `target` at skills changed in a PR and
gate with `scripts/score.py --fail-below 60 ...` — non-zero exit below the
threshold fails the check.
## Criteria
18 criteria: 14 core, scored on every skill, plus 4 conditional criteria
scored only when the skill's category makes them apply — otherwise mark
**N/A** and exclude the criterion from both the numerator and denominator of
the weighted average. Every score is 0–100 with evidence citing file,
section, or line.
### Axis 1 — Trigger (invocation)
| # | Criterion | Weight | Key question |
|---|-----------|--------|---------------|
| 1 | Invocation design | 2x | Is model-invoked vs. user-invoked deliberate and fitting? Model-invoked pays **context load** (the description loads every turn); user-invoked pays **cognitive load** (the human is the index). A skill that only ever fires by hand should be user-invoked. |
| 2 | Description quality | 2x | Model-invoked: leading word up front, one trigger per branch (synonyms renaming the same branch are duplication), no identity that's redundant with the body. User-invoked (`disable-model-invocation: true`): a human-facing one-liner, no trigger list. Score against the mode the skill actually uses — never penalize a user-invoked skill for lacking trigger phrases. |
### Axis 2 — Structure
| # | Criterion | Weight | Key question |
|---|-----------|--------|---------------|
| 3 | Steps vs. reference clarity | 1x | Does the skill distinguish ordered steps from on-demand reference? All-reference and all-steps skills are both valid — score clarity, not the mix. Is related material co-located (definition, rules, caveats under one heading)? |
| 4 | Branch-aware disclosure & pointers | 2x | Is material every branch needs inline, and material only some branches need behind a context pointer? Does each pointer's wording say when to follow it ("if you need X, read Y")? A weakly worded pointer to must-have material is a variance bug. |
| 5 | Conciseness (no sprawl) | 2x | Is SKILL.md lean — under 500 lines as a ceiling, smaller is better — with every line earning its context cost? |
| 6 | Coherent scope | 1x | Does the skill do one thing and compose with others, rather than covering too much? |
### Axis 3 — Steering
| # | Criterion | Weight | Key question |
|---|-----------|--------|---------------|
| 7 | Leading words | 2x | Does the skill use compact, high-prior terms ("vertical slice", "tight", "red") to anchor behavior, repeated consistently? Could any verbose passage collapse into one? |
| 8 | Completion criteria & legwork | 2x | Skills with steps: does each step end on a checkable, exhaustive completion criterion? A vague one invites premature completion. Skills that are pure reference: is there an exhaustiveness bar over the reference itself ("every rule applied")? If neither applies, mark N/A. |
| 9 | Gotchas section | 2x | Is there explicit capture of failure points, edge cases, footguns? |
| 10 | Grounded in expertise | 2x | Does content come from observed failures and real project facts, or generic "best practices"? |
| 11 | Avoids railroading | 1x | Does the skill leave room to adapt — procedures over declarations, defaults over menus — without over-prescribing? |
### Axis 4 — Pruning
| # | Criterion | Weight | Key question |
|---|-----------|--------|---------------|
| 12 | No-ops (deletion test) | 2x | Running the deletion test sentence by sentence: if removing a sentence leaves behavior unchanged, it's a no-op — including restatements of what the model already does by default. Cite line numbers for candidates. |
| 13 | Single source of truth | 1x | Does each meaning live in exactly one place? Duplication between SKILL.md and references/ counts too. |
| 14 | Relevance & sediment | 1x | Are there stale lines, accumulated layers, or material that no longer influences what the skill does? |
### Conditional criteria
Score only when the skill's category (from `references/categories.md`) makes
the criterion apply; otherwise mark N/A and drop it from the weighted
average entirely.
| # | Criterion | Weight | Applies to category |
|---|-----------|--------|----------------------|
| 15 | Setup flow | 1x | library-and-api-reference, data-fetching-and-analysis, ci-cd-and-deployment, infrastructure-operations |
| 16 | Memory mechanism | 1x | business-process-automation, data-fetching-and-analysis, runbooks |
| 17 | Scripts & libraries | 1x | product-verification, code-scaffolding-and-templates, code-quality-and-review, data-fetching-and-analysis, infrastructure-operations |
| 18 | On-demand hooks | 1x | code-quality-and-review, ci-cd-and-deployment |
Override this table with judgment, in either direction: score a criterion
for a skill outside these categories when it would clearly benefit (e.g., a
non-`product-verification` skill that obviously needs a helper script), and
mark it N/A even within an applicable category when the pattern doesn't fit
the skill's shape (e.g., a pure-reference vocabulary skill filed under
`code-quality-and-review` has nothing for a hook to enforce). Explain the
override in the scorecard either way.
### Overall score
```
overall = sum(score × weight) / sum(weight)
```
N/A criteria are excluded from both sums — never scored as 0, never counted
as weight.
## Scoring Guide
| Score | Meaning |
|-------|---------|
| 0 | Not present at all |
| 1–25 | Minimal/token effort, barely addresses the criterion |
| 26–50 | Partially addressed but with significant gaps |
| 51–75 | Solid implementation with room for improvement |
| 76–90 | Strong implementation, minor gaps only |
| 91–100 | Exemplary — would use as a reference for others |
## Grade Scale
| Grade | Range | Meaning |
|-------|-------|---------|
| A | 80–100 | Production-quality, reference skill |
| B | 60–79 | Good skill, minor improvements needed |
| C | 40–59 | Functional but significant gaps |
| D | 20–39 | Needs substantial rework |
| F | 0–19 | Skeleton only, not production-ready |
## Workflow
1. **Read the target skill** — SKILL.md, its frontmatter (check for
`disable-model-invocation`), and every file in the skill directory.
2. **Read `references/mechanics.md`** — the vocabulary and tests Axes 1, 3,
and 4 depend on, including what makes a context pointer's wording
effective.
3. **Classify** — use `references/categories.md` and its decision tree to
assign a category. The category determines which conditional criteria
apply.
4. **Score all applicable criteria** — **cite-or-cut**: a criterion is only
scored once its justification cites specific evidence (file, section, or
line); no citation, no score. Mark N/A wherever the conditional table, or
your own judgment, says a criterion doesn't apply. Done when every
applicable criterion carries a score and a citation, and every N/A a
reason.
5. **Trigger eval** — empirical test of whether the skill's description
actually causes invocation. See the **Trigger Eval** section below for
the full mechanic. Skip this step for user-invoked skills
(`disable-model-invocation: true`) — they have no description to test.
6. **Diagnose failure modes** — done when every mode in the table below has
been checked against the skill and either cited (file:line) or dismissed.
7. **Assess bonus patterns** — the 4 carried over from v1, plus a fifth:
| Bonus | Applies when | What to look for |
|-------|-------------|-----------------|
| Validation loops | Skill produces output or modifies state | Instructs the agent to self-check before finalizing |
| Output templates | Skill generates structured output | Includes a concrete template/example of expected format |
| Procedures over declarations | Skill teaches a method | Teaches *how to approach* problems, not *what to produce* for one case |
| Defaults over menus | Skill offers tool/approach choices | Picks a clear default, mentions alternatives briefly |
| Trace-checkable steering | Skill uses leading words | The leading words are distinctive enough that a user could grep the agent's reasoning traces to confirm the skill actually fired |
Report each as Present / Absent / N/A.
8. **Compute the weighted score** — run `scripts/score.py` with one
`criterion:score:weight` triple per criterion (score `NA` to exclude); it
prints both sums, the overall, and the grade. Don't do this arithmetic by
hand.
9. **Write the scorecard** to the output path — read
`references/output-template.md` first (it also holds the comparison-mode
template used when `compare` is set) and emit exactly that structure.
## Trigger Eval
Empirical test of whether the skill's description causes a model to invoke
it when it should — and ignore it when it shouldn't. This is not a pass/fail
gate; it produces observational data that feeds the scorecard and informs
the failure-mode diagnosis.
### When to run
- Model-invoked skills only. User-invoked skills (`disable-model-invocation:
true`) have no description to test — skip and mark the section N/A.
### Prompt generation
Generate **10 prompts** from the skill's description, scope, and gotchas:
- **5 should-trigger** — realistic user requests that fall squarely within
the skill's stated scope. Vary phrasing: some use the skill's vocabulary,
others describe the same need in naive/indirect language.
- **5 should-not-trigger** — requests that are adjacent but clearly outside
scope (e.g., a sibling skill's territory, a task the description
explicitly excludes, or a generic request a model handles without any
skill).
Each prompt should read like something a real user would type — no
meta-language about skills, no hints.
### Sub-agent execution
Run each prompt in an independent sub-agent session with the target skill
available. The sub-agent receives a single additional instruction appended to
its system context:
```
At the end of your response, output exactly one line in this format:
SKILLS_USED: <comma-separated list of skill names you loaded during this task, or "none">
```
This instruction is generic — it does not name the skill under test or hint
at what should be triggered. The sub-agent operates normally; it either loads
the skill or doesn't based on the prompt alone.
### Detection
Parse the `SKILLS_USED:` line from each sub-agent's response. Record per
prompt:
| Field | Value |
|-------|-------|
| Prompt | The test prompt text |
| Expected | should-trigger / should-not-trigger |
| Triggered | yes / no (was the target skill name in the list?) |
| Other skills | Any other skills that fired |
### What to report
Report raw counts — no pass/fail judgment:
- **Should-trigger hit rate** — X/5 triggered
- **Should-not-trigger leak rate** — X/5 triggered (lower is better)
- **Other skills observed** — which siblings fired on the same prompts
These numbers feed criterion #1 (invocation design) and #2 (description
quality) with empirical evidence, and may reveal failure modes like
over-triggering or description weakness.
### Practical notes
- If the evaluation environment cannot spawn sub-agents (e.g., CI without
agent access), skip the trigger eval and note "trigger eval: skipped
(no agent access)" in the scorecard.
- A single trial per prompt is acceptable given the observational (non-gating)
nature. Run multiple trials only if results are ambiguous.
- Keep prompts in the scorecard output so the skill author can reuse them as
a regression set.
## Failure-mode diagnosis
Name the failure mode, cite evidence, prescribe the defense. Each mode's
defense is defined once in `references/mechanics.md` §5 — prescribe from
there. This replaces a generic "top improvements" list.
| Mode | Evidence to look for |
|------|----------------------|
| Premature completion | Vague completion criteria with future steps still visible |
| Weak steering | Instruction present but the agent doesn't reliably follow it |
| Duplication | Same meaning in 2+ places, including SKILL.md vs. references/ |
| Sediment | Stale layers, outdated references, dead instructions |
| Sprawl | Long even with no duplication or sediment |
| No-ops | Lines that don't change behavior versus the model's default |
| Buried steps | Inline reference so heavy it soaks the steps |
After the table, write a **Prioritized Actions** section: 3–5 highest-impact
actions derived directly from the detected failure modes, each citing its
evidence.
Note: **context overload** — too many model-invoked skills competing for
attention in one environment — is a portfolio-level problem, out of scope
for evaluating a single skill. Record the description's context-load cost
when it's notable; don't score the portfolio.
## Gotchas
- Tiny skills (under ~50 lines) flood the scorecard with N/A — score what's
there; a small, sharp skill can reach grade A on few criteria.
- Self-evaluation bias: when the skill under review is one you (or this
session) wrote, apply the deletion test with extra skepticism — you will
want your own lines to matter.
- Fresh rewrites still carry duplication: sediment needs time to settle, but
duplication can ship on day one. Run the pruning axis even on brand-new
skills.
## Quality Checklist
Final gate before delivering — each item names the step whose completion it
re-checks, nothing new:
- [ ] cite-or-cut held everywhere (step 4)
- [ ] every N/A justified (step 4)
- [ ] trigger eval run or skipped with reason (step 5)
- [ ] every failure mode cited or dismissed (step 6)
- [ ] 5 bonus patterns assessed (step 7)
- [ ] score computed by `scripts/score.py`, not by hand (step 8)

112
.github/skills/skill-evaluation/references/categories.md vendored Normal file
View File

@@ -0,0 +1,112 @@
# Skill Categories Reference
Sources:
- [Lessons from building Claude Code: How we use skills](https://claude.com/blog/lessons-from-building-claude-code-how-we-use-skills) — Anthropic, Jun 2026
- [Best practices for skill creators](https://agentskills.io/skill-creation/best-practices) — Agent Skills spec
- [Extend Claude with skills](https://code.claude.com/docs/en/skills) — Claude Code docs
---
## 1. `library-and-api-reference`
Skills that explain how to correctly use a library, CLI, or SDK. Can be internal or public libraries that the model struggles with. Often include reference code snippets and gotchas lists.
**Signals:** Has API endpoint docs, CLI command reference, code examples, "how to call X" patterns.
**Examples:** billing-lib, internal-platform-cli, sandbox-proxy
---
## 2. `product-verification`
Skills that describe how to test or verify code is working. Often paired with Playwright, tmux, or other external tools. These have the most measurable impact on output quality — worth investing an engineer-week.
**Signals:** Has test scripts, assertion patterns, Playwright/Cypress flows, "verify that X" instructions.
**Examples:** signup-flow-driver, checkout-verifier, tmux-cli-driver
---
## 3. `data-fetching-and-analysis`
Skills that connect to data and monitoring stacks. Include libraries to fetch data with credentials, dashboard IDs, common query patterns.
**Signals:** Has database queries, dashboard references, metric/event schemas, "how to find X in our data" patterns.
**Examples:** funnel-query, cohort-compare, grafana, datadog
---
## 4. `business-process-automation`
Skills that automate repetitive workflows into one command. Usually simple instructions but may depend on other skills or MCPs. Saving results in log files helps consistency.
**Signals:** Has "do this weekly/daily" patterns, aggregates from multiple sources, posts to Slack/channels, formats structured output.
**Examples:** standup-post, create-ticket, weekly-recap
---
## 5. `code-scaffolding-and-templates`
Skills that generate framework boilerplates for a specific function. May combine with composable scripts. Especially useful when scaffolding has natural-language requirements beyond pure code.
**Signals:** Has templates, "new X" generators, boilerplate structures, asset files to copy.
**Examples:** new-workflow, new-migration, create-app
---
## 6. `code-quality-and-review`
Skills that enforce code quality and help review code. Can include deterministic scripts for robustness. May run as hooks or in GitHub Actions.
**Signals:** Has style rules, review checklists, linting patterns, "reject if X" logic, adversarial review patterns.
**Examples:** adversarial-review, code-style, testing-practices
---
## 7. `ci-cd-and-deployment`
Skills that help fetch, push, and deploy code. May reference other skills to collect data.
**Signals:** Has deploy commands, build pipelines, PR management, rollout/rollback logic, environment configs.
**Examples:** babysit-pr, deploy-service, cherry-pick-prod
---
## 8. `runbooks`
Skills that take a symptom (alert, error, Slack thread) and walk through multi-tool investigation producing a structured report.
**Signals:** Has symptom→tool→diagnosis flows, "if you see X check Y" decision trees, report templates.
**Examples:** service-debugging, oncall-runner, log-correlator
---
## 9. `infrastructure-operations`
Skills that perform routine maintenance and ops, some involving destructive actions with guardrails. Make it easier to follow best practices in critical operations.
**Signals:** Has cleanup/orphan detection, cost investigation, dependency approval, confirmation gates for destructive actions.
**Examples:** resource-orphans, dependency-management, cost-investigation
---
## Classification Decision Tree
1. Does it primarily teach how to **call an API/CLI/SDK**? → `library-and-api-reference`
2. Does it **verify** that something works (test, assert, validate)? → `product-verification`
3. Does it **query data** from monitoring/analytics/databases? → `data-fetching-and-analysis`
4. Does it **automate a repeating team process** (standup, report, ticket)? → `business-process-automation`
5. Does it **generate new code/files** from templates? → `code-scaffolding-and-templates`
6. Does it **review/lint/enforce quality** on existing code? → `code-quality-and-review`
7. Does it **build/deploy/ship** code to environments? → `ci-cd-and-deployment`
8. Does it **diagnose problems** from symptoms to structured findings? → `runbooks`
9. Does it perform **infrastructure maintenance/cleanup** with guardrails? → `infrastructure-operations`
If a skill spans multiple categories, pick the one that describes its **primary action** — what the user gets when they invoke it.

116
.github/skills/skill-evaluation/references/mechanics.md vendored Normal file
View File

@@ -0,0 +1,116 @@
# Mechanics: Predictability, Invocation, Hierarchy, Steering, Failure Modes
Reference for scoring Axes 1, 3, and 4 of the skill-evaluation rubric — a
deliberately self-contained condensation of Matt Pocock's `writing-great-skills`
GLOSSARY, kept in-skill so the evaluator runs anywhere without that skill
installed (sync manually if the upstream GLOSSARY changes). Not a tutorial:
look a bolded term up here rather than re-deriving it.
## 1. Root virtue: Predictability
A skill exists to wrangle determinism out of a stochastic system.
**Predictability** is the agent taking the same *process* every run, not
producing the same output — a brainstorming skill should predictably diverge;
its tokens vary, its behavior doesn't. Every criterion in the rubric is a lever
on this one virtue: conciseness, steering, and pruning are symptoms of
predictability, not separate virtues competing with it.
## 2. Invocation trade-off
Two invocation modes, each paying a different cost:
- **Model-invoked** (default; no `disable-model-invocation`): keeps a
description the agent reads every turn. Pays permanent **context load** —
tokens and attention spent on every turn — in exchange for autonomous
firing and reachability by other skills.
- **User-invoked** (`disable-model-invocation: true`): the description is
stripped from the agent's reach; only a human typing the skill's name can
fire it, and no other skill can reach it either. Zero context load, but
spends **cognitive load** — the human becomes the index of which skills
exist and when to reach for each.
Pick model-invocation only when the agent must reach the skill on its own, or
another skill must reach it. A skill that only ever fires by hand should be
user-invoked and carry no trigger scaffolding it doesn't need. When
user-invoked skills multiply past what a human can remember, a **router
skill** — one user-invoked skill naming the others and when to reach for each
— cures the accumulated cognitive load. That fix operates at the portfolio
level, not the single-skill level this evaluation scores.
## 3. Content types & hierarchy
A skill mixes two content types freely: **steps** (ordered actions, each
ending on a **completion criterion**) and **reference** (definitions, rules,
facts consulted on demand). All-steps, all-reference, and mixed skills are
equally valid — neither shape is a smell.
The **information hierarchy** ranks material by how immediately the agent
needs it: in-skill step, then in-skill reference, then reference disclosed
behind a **context pointer** in a linked file. Material every **branch** (a
distinct way the skill is invoked) needs belongs inline; material only some
branches need belongs behind a pointer — branching is the disclosure test. A
pointer's *wording*, not its target, decides whether the agent reaches it and
how reliably; a must-have target behind weak wording is a variance bug, and
the fix is sharper wording, tried before pulling the material back inline.
**Co-location** governs what sits beside a piece of content once placed: a
concept's definition, rules, and caveats belong under one heading, not
scattered, so reading one part brings its neighbors with it.
## 4. Steering
**Leading words** are compact, pretrained concepts (*tight*, *red*, *lesson*)
the agent thinks with while executing. Repeated consistently, they recruit
priors the model already holds and anchor a region of behavior in the fewest
tokens — cheaper and stickier than spelling the same quality out in prose. A
leading word works twice: in the body it anchors execution (the same behavior
fires every time the word appears); in the description it anchors invocation.
It is also **trace-checkable** — distinctive enough that its appearance in the
agent's reasoning traces confirms the skill actually shaped behavior.
A completion criterion must be *checkable* (can the agent tell done from
not-done?) and, where it matters, *exhaustive* ("every X accounted for", not
"produce a list"). A vague criterion invites **premature completion** —
attention slipping to being done rather than to the work. The exhaustiveness
demand also binds flat reference with no steps: "every rule applied" drives
thorough **legwork** over a checklist the same way a sharp step criterion
drives it over an action.
Skills should **avoid railroading**: procedures the agent adapts, not
declarations of exact output; defaults with brief alternatives, not
exhaustive menus.
## 5. Failure modes
- **Premature completion** — ending a step before it's genuinely done.
Defense, in order: sharpen the completion criterion first (cheap, local);
only if it's irreducibly vague *and* the rush is actually observed, split
the sequence so later steps are hidden.
- **Duplication** — the same meaning in more than one place. Costs
maintenance and tokens, and inflates that meaning's rank past its real
weight. Fix: collapse to a **single source of truth**, often via a leading
word.
- **Sediment** — stale layers that accumulate because adding feels safe and
removing feels risky. The default fate of any skill without a pruning
discipline.
- **Sprawl** — a skill simply too long, independent of whether lines are
stale or duplicated. Cure: disclose reference behind pointers, split by
branch or sequence so each path carries only what it needs.
- **No-op** — a line that changes nothing because the model already does it
by default. The test: does it change behavior versus the default? Apply
the **deletion test** sentence by sentence, not paragraph by paragraph — if
removing the sentence leaves behavior unchanged, delete the whole sentence,
don't trim words from it. A weak leading word (*be thorough* when the agent
is already thorough-ish) is a no-op; the fix is a stronger word
(*relentless*), not a different technique.
- **Weak steering** — an instruction is present but the agent doesn't
reliably follow it. Usually a leading word too weak to beat the default, or
no leading word at all where a verbose passage is trying to do its job.
- **Buried steps** — in-file reference so heavy it soaks the steps beneath
it, turning attention to them into a coin-flip. Defense: progressive
disclosure — push the reference behind a pointer.
**Relevance vs. no-op**: relevance asks whether a line still bears on the
task; no-op asks whether it changes behavior. A line can be relevant (right
topic) and still be a no-op (the model would do it anyway) — run both checks,
they don't imply each other.

158
.github/skills/skill-evaluation/references/output-template.md vendored Normal file
View File

@@ -0,0 +1,158 @@
# Output Template
Emit the scorecard exactly in this structure (step 9 of the workflow).
```markdown
# Skill Evaluation — {skill name}
> Evaluated: {date}
> Source: {path}
> Evaluator: skill-evaluation v2.1.0
> Framework: [Anthropic Skill Best Practices](https://claude.com/blog/lessons-from-building-claude-code-how-we-use-skills) + Matt Pocock's [writing-great-skills](https://www.youtube.com/watch?v=UNzCG3lw6O0)
## Summary
| Metric | Value |
|--------|-------|
| Overall Score | {weighted}/100 |
| Grade | {A/B/C/D/F} |
| Category | {category} |
| Invocation | {model-invoked / user-invoked} |
| Files | {count} |
| Criteria scored / N/A | {n} scored, {m} N/A |
## Scorecard
### Axis 1 — Trigger
| # | Criterion | Weight | Score | Notes |
|---|-----------|--------|-------|-------|
| 1 | Invocation design | 2x | {n}/100 | {evidence} |
| 2 | Description quality | 2x | {n}/100 | {evidence} |
### Axis 2 — Structure
| # | Criterion | Weight | Score | Notes |
|---|-----------|--------|-------|-------|
| 3 | Steps vs. reference clarity | 1x | {n}/100 | {evidence} |
| 4 | Branch-aware disclosure & pointers | 2x | {n}/100 | {evidence} |
| 5 | Conciseness | 2x | {n}/100 | {evidence} |
| 6 | Coherent scope | 1x | {n}/100 | {evidence} |
### Axis 3 — Steering
| # | Criterion | Weight | Score | Notes |
|---|-----------|--------|-------|-------|
| 7 | Leading words | 2x | {n}/100 | {evidence} |
| 8 | Completion criteria & legwork | 2x | {n/100 or N/A} | {evidence} |
| 9 | Gotchas section | 2x | {n}/100 | {evidence} |
| 10 | Grounded in expertise | 2x | {n}/100 | {evidence} |
| 11 | Avoids railroading | 1x | {n}/100 | {evidence} |
### Axis 4 — Pruning
| # | Criterion | Weight | Score | Notes |
|---|-----------|--------|-------|-------|
| 12 | No-ops (deletion test) | 2x | {n}/100 | {evidence with line citations} |
| 13 | Single source of truth | 1x | {n}/100 | {evidence} |
| 14 | Relevance & sediment | 1x | {n}/100 | {evidence} |
### Conditional criteria
| # | Criterion | Weight | Score | Notes |
|---|-----------|--------|-------|-------|
| 15 | Setup flow | 1x | {n/100 or N/A} | {evidence or reason for N/A} |
| 16 | Memory mechanism | 1x | {n/100 or N/A} | {evidence or reason for N/A} |
| 17 | Scripts & libraries | 1x | {n/100 or N/A} | {evidence or reason for N/A} |
| 18 | On-demand hooks | 1x | {n/100 or N/A} | {evidence or reason for N/A} |
## Trigger Eval
{For user-invoked skills, write: "N/A — user-invoked skill, no description to test."}
### Prompts tested
| # | Prompt | Expected | Triggered | Other skills |
|---|--------|----------|-----------|--------------|
| 1 | {prompt text} | should-trigger | yes/no | {list or none} |
| 2 | {prompt text} | should-trigger | yes/no | {list or none} |
| 3 | {prompt text} | should-trigger | yes/no | {list or none} |
| 4 | {prompt text} | should-trigger | yes/no | {list or none} |
| 5 | {prompt text} | should-trigger | yes/no | {list or none} |
| 6 | {prompt text} | should-not-trigger | yes/no | {list or none} |
| 7 | {prompt text} | should-not-trigger | yes/no | {list or none} |
| 8 | {prompt text} | should-not-trigger | yes/no | {list or none} |
| 9 | {prompt text} | should-not-trigger | yes/no | {list or none} |
| 10 | {prompt text} | should-not-trigger | yes/no | {list or none} |
### Results
| Metric | Value |
|--------|-------|
| Should-trigger hit rate | {X}/5 |
| Should-not-trigger leak rate | {X}/5 |
| Other skills observed | {list or none} |
### Observations
{Free-form notes: patterns in what triggered or didn't, description wording
gaps revealed, sibling skills that competed, etc.}
## Failure Modes Detected
| Mode | Evidence | Root cause | Defense |
|------|----------|------------|---------|
| {mode, or a single row "None detected"} | {file:line} | {cause} | {defense} |
## Prioritized Actions
### 1. {action}
**Evidence:** {file:line or section}
**Fix:** {specific recommendation}
### 2. {action}
**Evidence:** {file:line or section}
**Fix:** {specific recommendation}
(3–5 total, each tied to a detected failure mode)
## Bonus Patterns
| Pattern | Status | Notes |
|---------|--------|-------|
| Validation loops | {Present/Absent/N/A} | {detail} |
| Output templates | {Present/Absent/N/A} | {detail} |
| Procedures over declarations | {Present/Absent/N/A} | {detail} |
| Defaults over menus | {Present/Absent/N/A} | {detail} |
| Trace-checkable steering | {Present/Absent/N/A} | {detail} |
## Grade Scale
{copy the Grade Scale table from SKILL.md}
---
*Generated by [skill-evaluation](https://github.com/fabricioctelles/skills) v2.1.0, merging the [Anthropic skill quality framework](https://claude.com/blog/lessons-from-building-claude-code-how-we-use-skills) with Matt Pocock's [writing-great-skills](https://www.youtube.com/watch?v=UNzCG3lw6O0) methodology.*
```
## Comparison mode
When `compare` is set, add a side-by-side table across all 18 criteria.
Leave a cell N/A rather than scoring it 0, and exclude N/A rows from the
Overall row's weighted math for that skill.
```markdown
## Comparison: {skill A} vs {skill B}
| # | Criterion | {A} | {B} | Delta |
|---|-----------|-----|-----|-------|
| 1 | Invocation design | 60 | 85 | +25 |
| 2 | Description quality | 25 | 70 | +45 |
| ... | ... | ... | ... | ... |
| 15 | Setup flow | N/A | 80 | — |
| **Overall** | | **43** | **72** | **+29** |
```

66
.github/skills/skill-evaluation/scripts/score.py vendored Normal file
View File

@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Weighted overall score for a skill-evaluation scorecard.
Usage:
score.py [--fail-below N] 1:80:2 2:65:2 3:85:1 ... 15:NA:1 16:NA:1
One arg per criterion, formatted criterion:score:weight.
Score NA (or N/A) excludes the criterion from both sums.
Prints sum(score x weight), sum(weight), overall, and grade.
--fail-below N exits non-zero when overall < N (CI gate).
"""
import sys
def grade(score: float) -> str:
if score >= 80:
return "A"
if score >= 60:
return "B"
if score >= 40:
return "C"
if score >= 20:
return "D"
return "F"
def main() -> None:
args = sys.argv[1:]
fail_below = None
if "--fail-below" in args:
i = args.index("--fail-below")
try:
fail_below = float(args[i + 1])
except (IndexError, ValueError):
sys.exit("--fail-below requires a numeric threshold")
del args[i : i + 2]
if not args:
sys.exit(__doc__)
num = den = 0.0
na = []
for arg in args:
try:
crit, score, weight = arg.split(":")
except ValueError:
sys.exit(f"bad arg {arg!r}: expected criterion:score:weight")
if score.strip().upper() in ("NA", "N/A"):
na.append(crit)
continue
s, w = float(score), float(weight)
if not 0 <= s <= 100:
sys.exit(f"criterion {crit}: score {s} outside 0-100")
num += s * w
den += w
if den == 0:
sys.exit("no applicable criteria")
overall = num / den
print(f"applicable criteria: {len(args) - len(na)} | N/A: {', '.join(na) or 'none'}")
print(f"sum(score x weight) = {num:g}")
print(f"sum(weight) = {den:g}")
print(f"overall = {overall:.2f} -> grade {grade(overall)}")
if fail_below is not None and overall < fail_below:
sys.exit(f"FAIL: overall {overall:.2f} below threshold {fail_below:g}")
if __name__ == "__main__":
main()