autoresearch-mcp

An MCP server that brings Andrej Karpathy's autoresearch pattern to every AI coding session, with a composable technique catalog, experiment scaffolding, and SQLite + FTS5-backed tracking.

What is Autoresearch?

Autoresearch is a simple but powerful pattern popularized by Andrej Karpathy's autoresearch project, one of the most-starred AI research repositories on GitHub: give an AI agent a real experiment setup, let it modify code, prompts, or configs, run a fixed-time experiment, check whether the target metric improved, keep or discard the change, and repeat.

In Karpathy's framing, that can mean roughly 12 experiments per hour and around 100 overnight. The important idea is broader than any one implementation: if you have a measurable metric, you can ratchet toward better results.

autoresearch-mcp packages that pattern as an MCP server so any compatible AI client can discover techniques, scaffold experiments, track iterations, and accumulate meta-learning across projects.

This project is inspired by Karpathy's work, but it is not affiliated with his project and no code was copied.

Data Safety

Use synthetic, non-sensitive data only; examples and tests use synthetic data only. Do not supply PHI, patient identifiers, PHI-bearing prompts or model responses, clinical records, secrets, or production datasets to this MCP server, its evaluators or review tooling, logs, fixtures, or CI.

Release Status

Version 0.4.0 is not yet published or tagged. Package safety and protected Forgejo CI are in place; the remaining release work is the authenticated npm publish, installed-package smoke, and Forgejo-first tag push described in CONTRIBUTING.md. Until those steps succeed, use a reviewed source checkout rather than assuming autoresearch-mcp is available from npm.

Quick Start

Runtime requirements:

• MCP server (autoresearch-mcp): requires Bun >= 1.3.10, because the server uses bun:sqlite
• Standalone skill installer (autoresearch-install-skill): supports Node.js 22 and 24, no Bun needed
• Any MCP-compatible client such as Claude Code or OpenCode

After 0.4.0 is published, install globally (puts both commands on your PATH):

npm install -g autoresearch-mcp

Or, after publication, run without a global install:

# Start the MCP server (requires Bun)
bunx autoresearch-mcp

# Install the bundled skill (works with Node alone)
npx -p autoresearch-mcp autoresearch-install-skill

Note: npm install -g succeeds on a machine without Bun, but the autoresearch-mcp server command will not run until Bun >= 1.3.10 is installed. The standalone skill installer runs on supported Node.js 22 or 24 alone.

Install as Skill (Recommended)

autoresearch-mcp ships with a skill file that teaches your AI agent the autoresearch methodology: when to use which technique, how to compose recipes, and how to run ratchet loops. The skill is lightweight (~100-400 tokens in context) while the MCP server provides the heavy machinery (catalog search, experiment tracking, scaffolding).

Skill + MCP = Brain + Hands

OpenCode

# Install the bundled skill into ~/.opencode/skills/autoresearch
npx -p autoresearch-mcp autoresearch-install-skill --target opencode

# If autoresearch-mcp is already installed globally (requires Bun;
# on Node-only machines use autoresearch-install-skill instead):
autoresearch-mcp install-skill --target opencode

Skills are auto-discovered from ~/.opencode/skills/. The skill lazy-loads when your agent encounters optimization problems.

Claude Code

npx -p autoresearch-mcp autoresearch-install-skill --target claude

pi.dev

pi --skill $(npm root -g)/autoresearch-mcp/skills/autoresearch/SKILL.md

The installer copies skill files by default so npx temporary package caches do not leave broken symlinks. Use --dry-run to preview changes or --overwrite to replace an existing skill directory.

Install as MCP Server (Machinery)

The MCP server provides tools and state. Install alongside the skill for full capability.

Claude Code

Add this to your MCP settings:

{
  "mcpServers": {
    "autoresearch": {
      "command": "bunx",
      "args": ["autoresearch-mcp"]
    }
  }
}

OpenCode

Add this to ~/.config/opencode/opencode.json:

{
  "mcp": {
    "autoresearch": {
      "type": "local",
      "command": ["bunx", "autoresearch-mcp"]
    }
  }
}

Once connected, ask your agent things like:

• "What autoresearch technique should I use for prompt optimization?"
• "Scaffold a code-performance experiment for this project."
• "Log this iteration result and track the total cost."

How It Works

At the core is a ratchet loop:

Edit artifact -> Run evaluator -> Score improved? -> Yes: Keep -> Repeat
                                                  -> No: Revert -> Repeat

The server gives your agent the pieces needed to run that loop in a structured way:

1. Pick a technique or recipe.
2. Scaffold an experiment with a program and evaluator harness.
3. Run iterations against a measurable metric.
4. Keep improvements, discard regressions.
5. Track costs, timing, and outcomes.
6. Reuse what works across future projects.

Technique Catalog

autoresearch-mcp ships with a 30-item catalog organized into four composable layers.

1. Search Strategies

These define how candidate changes are proposed.

• hill-climbing
• evolutionary
• bayesian-optimization
• beam-search
• multi-armed-bandit
• simulated-annealing
• ablation-elimination
• self-refine

2. Evaluators

These define how candidates are scored.

• benchmark-harness
• binary-evaluator
• rubric-scorer
• llm-as-judge
• pairwise-comparison
• cost-latency-evaluator
• human-approval-gate
• regression-detector

3. Execution Patterns

These define how the loop is run and controlled.

• single-ratchet
• two-loop
• bounded-episode
• branch-and-merge
• champion-challenger
• checkpoint-and-resume

4. Recipes

Recipes compose a strategy, evaluator, and execution pattern into a ready-to-use starting point.

• prompt-optimization
• code-performance
• config-tuning
• content-revision
• test-amplification
• ml-training
• literature-synthesis
• general-ratchet

MCP Tools

The server exposes 12 MCP tools.

Tool	Description
search_techniques	Search the catalog by query, or list all techniques when the query is empty.
get_technique	Return full details for a technique by ID.
suggest_technique	Describe a problem and get a recommended approach.
register_experiment	Create a tracked experiment record.
update_experiment	Update experiment status with automatic timestamps.
log_result	Log an iteration result with score, time, token, and dollar tracking.
get_experiment	Retrieve experiment details and optional iteration history.
list_experiments	List experiments filtered by status or project.
scaffold_experiment	Generate program.md, eval.sh, and results.tsv from a recipe.
get_template	Return a recipe template file.
get_server_info	Return server version, catalog stats, and the active database path.
log_technique_outcome	Record what worked for cross-project meta-learning.

Usage Examples

Conversational workflow

You: "I want to optimize my chatbot's system prompt. I have 50 test questions."

Agent calls: suggest_technique(problem: "optimize chatbot prompt with eval set")
-> Recommends: prompt-optimization recipe
   (hill-climbing + llm-as-judge + single-ratchet)

Agent calls: scaffold_experiment(recipe_id: "prompt-optimization", ...)
-> Creates: autoresearch/program.md, eval.sh, results.tsv

You: "Run the ratchet loop"
Agent: reads program.md, edits prompt, runs eval.sh, logs results...

After 10 iterations: Score improved from 62 to 94 (+52%)

Scripted MCP flow

If you prefer explicit tool orchestration, the lifecycle looks like this:

1. suggest_technique(problem="reduce API latency without hurting quality")
2. scaffold_experiment(recipe_id="code-performance", project_path="/repo", metric_name="requests/sec")
3. update_experiment(experiment_id="...", status="running")
4. log_result(iteration=0, score=1100, is_baseline=true, improved=false, change_description="baseline before candidate changes")
5. log_result(iteration=1, score=1180, change_description="inlined hot path")
6. log_result(iteration=2, score=1165, change_description="added extra serialization")
7. get_experiment(experiment_id="...", include_results=true)
8. log_technique_outcome(technique_id="code-performance", domain="backend", outcome="success")

Log exactly one baseline before candidate iterations. Baselines are never improvements, so the explicit improved=false assertion matches the server-derived result. For candidates, omit improved as above and let the server derive it; if supplied, the assertion must match the server-derived result.

After scaffolding, your agent gets a working starting point:

• autoresearch/program.md for the loop instructions
• autoresearch/eval.sh for the evaluation harness
• autoresearch/results.tsv for iteration history

Example Domains

Anything with a measurable target can use the pattern.

Domain	Target	Evaluator Example
Prompt engineering	System prompts	Eval set accuracy
Code performance	Source code	Benchmark score
Config tuning	Config files	Performance metric
Content quality	Articles and docs	Quality rubric score
Test coverage	Test suites	Coverage percentage
ML training	Training code	Validation loss

Recipes

Recipes are the fastest way to get started because they encode a practical composition of the three lower layers:

recipe = search strategy + evaluator + execution pattern

For example:

• prompt-optimization combines a search strategy suited to prompt mutation, an evaluator that can score prompt outputs, and a ratchet pattern that preserves improvements.
• code-performance pairs code changes with benchmark-driven evaluation.
• general-ratchet gives you a flexible default when your domain is unusual but still measurable.

You can use recipes as-is, inspect their parts with get_technique, or search the catalog to build your own combination.

Configuration

Claude Code

{
  "mcpServers": {
    "autoresearch": {
      "command": "bunx",
      "args": ["autoresearch-mcp"]
    }
  }
}

OpenCode

{
  "mcp": {
    "autoresearch": {
      "type": "local",
      "command": ["bunx", "autoresearch-mcp"]
    }
  }
}

Other MCP clients

Any client that supports launching a local stdio MCP server can use autoresearch-mcp with the same pattern:

• command: bunx
• args: autoresearch-mcp

If your client expects a single executable command, point it at the same Bun-based invocation.

Roadmap

• Phase 0.5: Catalog discovery + FTS5 search
• Phase 1: Experiment tracking + scaffolding
• Phase 2: Skill + tests + public release (current)
• Phase 3: Autonomous runner with agent-driven execution and approval-aware loops
• Phase 4: Docker sandbox for safer code execution and isolated experiments
• Phase 5: Nightcrawler-style bounded episodes for longer autonomous optimization runs

The direction is simple: start with trustworthy building blocks, then expand toward increasingly autonomous experiment execution.

Inspired By

This project is prominently inspired by Andrej Karpathy's autoresearch work:

• GitHub: karpathy/autoresearch
• Posts and discussion: @karpathy on X

autoresearch-mcp adapts the underlying pattern for MCP-native workflows so coding agents can use it across prompts, code, configs, content, tests, and research tasks.

It is inspired by Karpathy's idea, not affiliated with his project, and no code was copied.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md.

License

Apache-2.0