ADLC · Practice Area 16

Agent Learning & Adaptation

How an agent gets better over time: from feedback loops that need no training at all, through fine-tuning, preference optimization and reinforcement learning, to systems that can improve themselves.

Feedback loopsFine-tuningPreference optimizationReinforcement learningOnline learningAgent evolution
Agent Learning and Adaptation overview slide
Overview

Two loops: the system and the weights

An agent can improve without a single gradient update. The first decision is whether the problem belongs in the fast, reversible system loop or in the slower, more expensive weights loop.

Hours · cheap · reversible

The system loop

  • Improve prompts, context, retrieval and tools.
  • Use memory and few-shot examples from real runs.
  • Close feedback loops without changing model weights.
  • This is where the source deck says most real improvement comes from.
Weeks · costly · hard to undo

The weights loop

  • Fine-tuning, preference optimization and reinforcement learning.
  • Requires curated data, compute, evaluation and rollback planning.
  • Can buy behavior that prompting cannot reach — along with new failure modes.
  • Best justified after the system loop has genuinely plateaued.
Improve the system before you touch the weights. Training is a commitment; a prompt change is an experiment.
Lever 1

Feedback loops

Every downstream learning method is only as good as the signal feeding it. Feedback is the raw material — and most of it is already flowing through production systems.

Cheap but sparse

Explicit

Thumbs, ratings, corrections and human approvals.

Abundant but noisy

Implicit

Edits, retries, abandonment, escalation and task completion.

The gold standard

Verifiable

Tests pass, a compiler runs, math checks out, or a transaction settles.

Turn signal into data

  • Attach feedback to the trace so you know what produced it.
  • Categorize failures; the taxonomy becomes the improvement backlog.
  • Turn every correction into an eval case before training on it.
  • Capture the reason behind rejection, not only the rejection itself.

Where loops go wrong

  • Survivorship bias: only the users who complain are heard.
  • Training on the agent's own outputs narrows the distribution.
  • Teams optimize the metric they can measure instead of the outcome that matters.
  • Without a verifier there is no reliable ground truth, and errors compound.
Lever 2

Fine-tuning

Supervised fine-tuning teaches format, style and domain conventions by showing the model curated examples. In the source, it is described as the cold start — not the finish line.

What it fixes

  • Output formats that prompting fails to enforce reliably.
  • Domain vocabulary, tone and house conventions.
  • Tool-call syntax on curated trajectories.
  • Long repeated instructions that can be distilled into the weights.
  • Distilling a large model's behavior into a smaller one.

What it doesn't

  • Add fresh knowledge — retrieval is the mechanism for that.
  • Automatically raise the reasoning ceiling.
  • Fix a bad task definition or broken tool.
  • Survive every base-model upgrade without rework.
  • Guarantee better truthfulness when the data is thin.
Parameter-efficient by default. The deck recommends LoRA and other adapter-style approaches before full fine-tuning: cheaper, faster, swappable by task, and still guarded by an eval suite.
Lever 3

Preference optimization

When there is no single correct answer, teach the model which answer is better. The source contrasts classic RLHF with direct preference optimization (DPO).

Classic RLHF

Three stages

SFTCold start
Reward modelLearn preferences
RL (PPO)Optimize against reward
DPO

Direct optimization

  • Train on chosen vs. rejected responses to the same prompt.
  • Raise likelihood of the chosen response and lower the rejected one.
  • Stay anchored near a reference model to reduce drift.
  • No separate reward model and no RL loop, making the process simpler.
Preference optimization shapes alignment and taste. The deck explicitly treats it as a different problem from adding capability.
Lever 4

Reinforcement learning

The deck presents RL as the mechanism for pushing past supervised-training plateaus, especially when the environment can verify success directly.

RLVR

Verifiable rewards

Skip subjective human rankings when the environment can score the answer. Run the tests, check the math, compile the code, or verify the transaction, then reward success.

GRPO

Group-relative optimization

  • Sample a group of responses to one prompt and score them relative to each other.
  • Avoid a separate value critic.
  • Use less memory than PPO-style four-model setups, according to the source deck.
  • Pair with verifiable rewards where objective checking is available.

Failure modes

  • Reward hacking: the model games the metric instead of solving the task.
  • Entropy, diversity and advantage collapse.
  • Drift away from the intended behavior if constraints are weak.

Agent-specific challenge

Long-horizon agents may take many tool calls and thousands of tokens before anyone can score the run. The source therefore highlights frameworks that decouple rollout from gradient updates.

Synthesis

The 2026 post-training stack

The three weight-loop levers are presented as a pipeline. Each stage addresses a limitation left by the one before it.

Pretrained base modelRaw capability
SFTFormat, tool syntax, cold start
Preference optimizationAlignment and taste · DPO / SimPO
RL with verifiable rewardsReasoning and generalization · GRPO / DAPO

Why order matters

  • SFT gets the model started but can plateau.
  • Preference optimization shapes alignment more than raw capability.
  • RL pushes beyond the ceiling, but needs a stable starting point.
  • Skipping a stage can amplify the deficiencies of the previous one.

What changed from classic RLHF

  • Critic-free and group-relative methods reduce complexity.
  • Verifiers replace reward models when ground truth is available.
  • Post-training is increasingly modular rather than one monolithic recipe.
Lever 5

Online learning

The source argues that continuously updating model weights from live production traffic is usually too risky. In practice, teams update the system around the model.

Why live weight updates fail

  • Catastrophic forgetting can erase old competence.
  • The model trains on its own outputs and narrows the distribution.
  • Poisoned or adversarial input can become permanent behavior.
  • Updates become hard to reproduce and difficult to roll back.
  • Safety and alignment properties can silently regress.

What “online” means in practice

  • Update memory while keeping weights fixed.
  • Refresh retrieval indexes with new knowledge.
  • Change prompts, routing and configuration behind flags.
  • Retrain periodically, offline, on accumulated curated data.
  • Gate every update with evals, canaries and a reversible path.
An agent that “learns from every conversation” is usually writing to memory, not weights. The source frames that as a feature: memory is inspectable, correctable and user-scoped.
Lever 6

Agent evolution

The frontier is a closed loop in which an agent identifies weaknesses, generates tasks that target them, attempts the tasks, verifies the result, and trains on what worked.

Identify a weakness
Generate targeted tasks
Attempt and verify
Train on what worked

What is already real

  • Self-play and self-generated curricula tuned to current ability.
  • Self-verification and reflection emerging from RL.
  • Evolution of prompts, tools and workflows — not only weights.
  • Search over agent designs scored by an eval suite.

Verification is the bottleneck

  • Self-improvement works best where outcomes can be checked.
  • Without a verifier the loop amplifies the agent's own errors.
  • A confident wrong answer can become tomorrow's training data.
  • Keep a human at the boundary and keep the evals independent.
Putting it together

Which learning mechanism, when?

The deck closes with a practical routing guide: choose the cheapest, most reversible mechanism that actually matches the problem.

If: the agent lacks facts, not skill
Retrieval — never training
If: it ignores formatting rules
Fine-tune (LoRA) on curated examples
If: outputs are correct but badly judged
Preference optimization (DPO)
If: answers are checkable and performance plateaus
RL with verifiable rewards (GRPO)
If: it should remember this user
Memory — not a gradient update
If: you want it to improve itself
Only behind a trustworthy verifier
If: you have not tried prompting properly
Fix the system loop first
Learning is cheap where you can verify and expensive where you cannot. Build the verifier first; everything else follows from it.
Original deck

All 10 slides

Each image uses responsive variants from the Slide600, Slide700, Slide900 and Slide1000 folders. Select any slide to enlarge it.

Slide 1 · Agent Learning & Adaptation
Slide 2 · Two Loops: The System and the Weights
Slide 3 · Feedback Loops
Slide 4 · Fine-Tuning
Slide 5 · Preference Optimization
Slide 6 · Reinforcement Learning
Slide 7 · The 2026 Post-Training Stack
Slide 8 · Online Learning
Slide 9 · Agent Evolution
Slide 10 · Which Learning Mechanism, When