All posts
AI AgentsOctober 28, 20265 min read

Skill Dreaming: How PR Streams Teach Coding Agents to Learn - and Keep Learning

A new framework called Skill Dreaming lets LLM coding agents extract reusable knowledge from a repository's pull request history and continuously evolve that knowledge as the codebase changes - pushing the SWE-bench Pro resolved rate from 79% to 88%.

Skill Dreaming: How PR Streams Teach Coding Agents to Learn - and Keep Learning

Coding agents have made remarkable strides on software engineering benchmarks, but they still share a fundamental limitation: their knowledge of a specific codebase is static. A model may know Python deeply, but it doesn't know your repository - its quirks, its conventions, or the architectural decisions layered in over years of merged pull requests. A new paper published on arXiv proposes a clean solution: let the PR stream itself be the teacher.

The framework is called Skill Dreaming 1, and it arrives with a headline result that should get serious attention from the engineering community: it lifts the official resolved rate on SWE-bench Pro from 79% to 88% for the base agent it augments.

The Core Problem: Skills Go Stale

Modern LLM agents increasingly rely on skill libraries - reusable, structured instructions that guide how a task should be approached. These are powerful tools for reducing repetition and improving consistency. But there's a catch.

As the paper describes it, repository-specific skills must evolve with changing code structures, implementation conventions, and maintenance requirements 1. A skill that correctly described how to add a feature six months ago may be outdated after a large refactor. Static skill libraries don't track that drift - they just quietly become wrong.

Pull requests offer a natural remedy. PRs provide a natural source of such knowledge by recording how repositories evolve through implementation and review 1. Every merged PR is a validated, human-reviewed record of what good work looks like in a specific codebase at a specific moment in time.

The Three-Layer Architecture

Skill Dreaming formalizes the relationship between PRs, skills, and the codebase into a self-reinforcing cycle. PRs provide knowledge for updating skills, while these skills improve future PRs, maintain repositories, and support further skill evolution 1.

The framework operates in three interconnected layers:

  1. PR Streams - The raw feed of development history. Each merged PR is treated as a training signal, not just a change record.
  2. Skill Evolution - Skills are dynamically generated, reused, revised, and replaced as new PRs arrive 1. Reused skills persist; new features generate new skills; obsolete rules get superseded 2.
  3. Repository Development - The evolved skill library actively guides the coding agent when it tackles future PRs, maintaining consistency with the project's architecture and test suite.

By integrating PR analysis, skill management, code execution, and continuous integration, Skill Dreaming provides an end-to-end workflow for applying learned skills to repository changes and incorporating subsequent PR outcomes into future skill updates 1.

Crucially, the skills that emerge aren't just high-level hints. These skills guide coding agents not only in implementing requested features but also in maintaining associated tests, documentation, compatibility, and repository conventions 1. This is the kind of holistic awareness that separates a developer who deeply knows a codebase from one who's just reading the issue tracker.

Why SWE-bench Pro Is the Right Test

SWE-bench Verified - the original benchmark - has been largely saturated at the frontier, and the community has increasingly turned to SWE-bench Pro, Scale AI's contamination-resistant successor. SWE-bench Pro comprises 1,865 tasks across 41 repositories in Python, Go, TypeScript, and JavaScript - including private proprietary codebases that models cannot have seen during training 3.

That makes SWE-bench Pro the genuinely hard, real-world test - and it's on that benchmark that Skill Dreaming delivers its most meaningful result. Going from 79% to 88% resolved on a contamination-resistant eval isn't a leaderboard trick; it reflects the framework helping an agent navigate real repository context it could never have memorized.

What "Skill Dreaming" Actually Means

The name itself is evocative. Classical AI "dreaming" - replaying experience during off-policy phases to consolidate learning - has roots in neuroscience-inspired reinforcement learning. Skill Dreaming applies that intuition to the software engineering domain: the agent isn't just executing tasks; it is continuously consolidating lessons from past repository work into a growing, structured memory that gets applied to future work.

Given a repository and its PR history, the framework extracts reusable knowledge from validated PRs into a dynamic skill library 1. Each new round of PRs then both tests those skills and teaches the system where they need updating. This closed loop - test, learn, revise - is what gives the framework its staying power in a live codebase.

Connections to a Broader Trend

Skill Dreaming fits into a fast-moving research space around persistent, evolving agent memory for code. The AGENTS.md format has established that structured skill files can live natively in a repository alongside source code, versioned and reviewed like any other artifact. Skill Dreaming takes the complementary view: those skill files shouldn't be manually authored - they should be distilled automatically from the lived history of the repo.

Meanwhile, on the capability side, the SWE-bench Pro leaderboard as of October 2026 shows frontier models clustered in the 44 - 62% range on the standardized public set 4. A framework that lifts a base agent nine percentage points on that benchmark - on tasks the model has never seen - represents a meaningful practical advance, independent of whatever model sits underneath.

Code Is Available Now

The implementation is already public. The GitHub repository 2 includes a full project structure: a skill_dream core module covering skill storage, PR parsing, replay, retrieval, patch application, and model clients; benchmark drivers for both SWE-bench Verified and SWE-bench Pro; offline test suites; and configuration files. The README includes installation and quickstart instructions for running the skill-dreaming pipeline on a local repository. Model weights are not bundled - the framework is model-agnostic and calls external LLM APIs - but everything needed to reproduce the benchmark pipeline is present.

Why It Matters

The engineering implication is concrete: if your team runs a coding agent on a large, long-lived repository, static system prompts and one-time skill files are going to drift out of sync with the codebase. Skill Dreaming suggests a better architecture - one where the agent's knowledge of your repository grows with it, drawing directly from the institutional memory already encoded in your PR history.

With both the paper and the full codebase now publicly available, teams investing in agentic coding infrastructure can start experimenting today. The core idea is immediately practical: treat your PR stream not just as a changelog, but as a continuous curriculum for your agents.

This article was researched and drafted by an AI writer agent (claude-sonnet-4-6) and reviewed by an editor agent before publishing.

Ask about Julian Walder

Grounded in his real work

Hi! I'm Julian Walder's assistant. Ask me anything about his work, projects, or background in AI.