SKILLRET Dataset — Exploratory Data Analysis

1. What Is SKILLRET?

SkillRet is a retrieval benchmark for matching natural-language user requests to AI agent skills. It contains a curated library of 17,810 public agent skills harvested from GitHub, along with synthetic training queries and human-quality evaluation queries.

The dataset consists of three core components:

• Skills — structured metadata + full SKILL.md content for each agent skill
• Queries — natural-language user requests that should match one or more skills
• Qrels — binary relevance judgments linking queries to their correct skills

Additionally, the dataset includes a taxonomy with 6 major categories and 18 sub-categories that classify every skill.

Data Structure

Each JSONL file stores one record per line. Below are example records (long text values truncated with …).

Skill Record (skills.jsonl)

18 fields per skill: identifiers, metadata, taxonomy labels, and the SKILL.md content.

{
  "id": "a1299c1e-12ab-44af-a931-d7fa0254de10",
  "name": "brainstorming",
  "namespace": "@obra/superpowers/brainstorming",
  "description": "You MUST use this before any creative work - creating features, building components, adding functionality, or modifying ...",
  "author": "obra",
  "stars": 16795,
  "installs": 2847,
  "license": "mit",
  "repo": "obra/superpowers",
  "source_url": "https://github.com/obra/superpowers/tree/main/skills/brainstorming",
  "raw_url": "https://raw.githubusercontent.com/obra/superpowers/main/skills/brainstorming/SKILL.md",
  "major": "Software Engineering",
  "sub": "Development",
  "primary_action": "design",
  "primary_object": "agent_skill",
  "domain": "ai_agents",
  "body": "---\nname: brainstorming\ndescription: \"You MUST use this before any creative work - creating features, building component...",
  "skill_md": "---\nname: brainstorming\ndescription: \"You MUST use this before any creative work - creating features, building component..."
}

Query Record (queries/*.jsonl)

7 fields: the natural-language request, its matching skill IDs, the number of relevant skills (k), and the generator model.

{
  "id": "q-00001",
  "original_id": "q-00001",
  "query": "Design a system where an LLM-based assistant can autonomously manage a small e-commerce store's customer support: it sho...",
  "skill_ids": [
    "57ca9c6b-c2c3-446a-b695-09f824efc80d"
  ],
  "skill_names": [
    "agents"
  ],
  "k": 1,
  "generator_model": "Claude Opus 4.6"
}

Qrel Record (qrels/*.jsonl)

3 fields: a binary relevance judgment linking a query to a skill.

{
  "query_id": "q-00001",
  "skill_id": "57ca9c6b-c2c3-446a-b695-09f824efc80d",
  "relevance": 1
}

2. Skill Directory Structure

Per the Agent Skills specification, a skill is a directory containing a required SKILL.md file plus optional companion folders:

• scripts/ — executable code the agent can run
• references/ — additional documentation loaded on demand
• assets/ — templates, data files, static resources

The dataset only captured the SKILL.md content for each skill (body == skill_md for all 17,810 records). To determine how many skills are bare SKILL.md (nothing else in the directory) versus skills that have companion sub-folders, we analyzed the SKILL.md text for references to scripts/, references/, assets/, and other local file paths.

Important caveat: While these skills reference sub-folders and companion files in their SKILL.md instructions (e.g., “Run scripts/extract.py”), the actual content of those files is not included in the dataset. The dataset only provides the SKILL.md text. This means the retrieval benchmark operates exclusively on SKILL.md content, missing the richer context that ~36% of skills were designed to include.

3. Taxonomy & Categories

The SKILLRET taxonomy organizes skills into 6 major categories and 18 sub-categories:

Software Engineering dominates with 11,085 skills (62.2%), followed by AI Agents (2,074) and Data & ML (1,311).

4. Skill Metadata Statistics

5. Queries Component

Queries are synthetic natural-language user requests designed to match one or more skills. They were generated by two different LLMs:

• Test split (4,392 queries) — generated by Claude Opus 4.6. These are the evaluation queries, typically longer and more complex.
• Train split (63,259 queries) — generated by Qwen3.5-122B-A10B. These are the synthetic training queries.

Each query record includes:

• id / original_id — unique query identifier
• query — the full natural-language request text
• skill_ids — list of relevant skill UUIDs
• skill_names — human-readable names of relevant skills
• k — number of relevant skills for this query
• generator_model — which LLM generated the query

6. Qrels (Relevance Judgments)

Qrels (query relevance judgments) are the ground-truth labels that link queries to their correct skills. Each qrel is a triple: (query_id, skill_id, relevance) where relevance is binary (1 = relevant).

7. Train / Test Split Design

The dataset uses a disjoint skill split: train and test skills do NOT overlap. This ensures that evaluation measures true generalization to unseen skills, not memorization.

• Train skills: 10,123 skills with 63,259 queries and 127,190 qrels
• Test skills: 6,006 skills with 4,392 queries and 7,187 qrels
• Out-of-split skills: 1,681 skills in the full corpus that appear in neither train nor test splits (serve as distractors)

8. Key Findings Summary