elvis@omarsar0 · 5月20日74http://x.com/i/article/2056851733582880768
# Automating LLM Fine-Tuning with Fireworks Agent
## From Context Window to Weights
Andrej Karpathy (@karpathy) recently described the personal LLM Wiki as a kind of pre-AGI memory aid, a curated repo of notes about papers, tools, and ideas you read into context when you want a model to reason over them.
In his viral post, Karpathy flagged the obvious next move: "As the repo grows, the natural desire is to also think about synthetic data generation + finetuning to have your LLM 'know' the data in its weights instead of just context windows."
Building LLM Knowledge Bases or LLM Wikis is already possible with agents like Claude Code or Codex, but this approach can quickly get inefficient and expensive as you try to scale them. Fine-tuning LLMs to maintain your knowledge bases is often a more efficient path forward.
This post takes that next step by putting the wiki's output style into the weights. In under ten minutes of GPU time and a couple of cents of compute, a small open-weight model writes summaries of new papers in the exact format the wiki uses, with no system-prompt gymnastics, no few-shot exemplars, and no router logic. Once deployed, the summary comes back in a single fast call, fast enough to use inline inside a larger agent loop rather than as a batch job. The harder version (parametric knowledge injection of the wiki's contents) is the natural follow-up to Karpathy's framing, and I treat it as future work at the end.
The interesting part is not the model itself, but that one @FireworksAI_HQ Agent session did the entire pipeline (dataset inspection, hyperparameter sweep, full training, deployment, and a working inference endpoint). Fireworks Agent is the autonomous orchestration layer for fine-tuning runs, where you give it a natural-language goal, and it plans, executes, and surfaces decision gates back to you. The whole flow can be driven from a coding agent you already use (Claude Code, Codex, or similar), which is how I ran it.
The bigger picture this points to is self-improving LLMs and agents. Once training is a callable step inside an agent loop, the same coding agent that drives your workflow can also kick off fine-tuning runs to bake recurring patterns (a wiki's voice, a coding style, a triage policy) into the model itself, closing the loop between using a model and improving it.
The rest of this post is the full walkthrough.
All resources from this run are available in a companion repo, including the training and validation splits (train.jsonl, val.jsonl, wiki-sft-2026.jsonl), the data-build scripts (parse_2026.py, fetch_abstracts.py, build_jsonl.py), the pilot-agent.md slash command, the smoke-test script (test_new_deployment.py), and the baseline-vs-fine-tuned comparison code (before_after.py). Grab it at github.com/dair-ai/wiki-sft, clone it, point it at your own corpus, and reproduce the run end to end.
## Why Output Style Is the Right First SFT Target
For a personal wiki, the high-leverage thing is consistency. Readers recognize a summary by its shape, which is a one-paragraph lede that names the authors' affiliation and the core contribution, followed by three to five bulleted takeaways with bolded short labels. A capable base model can be coaxed into this format with a careful system prompt, but the failure modes are familiar. It reverts to title-case headers, drops the affiliation line, varies bullet count, and sneaks in marketing language.
Supervised fine-tuning (SFT) fixes this at the parameter level. Once the format is in the weights, every generation conforms by default, and the system prompt collapses to a single sentence (or drops out entirely). The cost stays small when the dataset stays small, and a clean stylistic dataset of 50 to 100 examples is usually enough to get started.
## Handing the Work to an Agent
Most fine-tuning tutorials walk you through ten distinct steps. You format your data, upload it, choose a base model, decide on LoRA rank and learning rate, launch a job, parse logs, pick a winner, retrain on full data, deploy, and smoke test. Each step is its own surface to mess up, and you end up playing the role of a tuning agent yourself.
Fireworks Agent inverts this. The interface is firectl session create -n "<your instruction>", where firectl is the Fireworks CLI. After that, you watch events stream and respond to gates when the agent surfaces a decision, such as the proposed plan or the hyperparameter (HP) sweep results.
Fireworks also ships a Claude Code slash command (or you can format it as an agent skill), pilot-agent.md (previously known as Pilot Agent), that wraps the firectl commands and handles event streaming, gate detection, and resume-from-last-timestamp logic.
## Full Walkthrough
Step 0: Setup
Install the Fireworks CLI and confirm your account.
In the Fireworks dashboard, create a service account that has the permissions Training Agent needs (the role that lets it launch training jobs and deployments on your behalf), then generate an API key tied to that service account. Also, create a separate user-level API key for inference and deployment inspection. Drop both into a .env file next to the project.
Step 1: Build the Dataset
The training data I use consists of chat-format records derived from the DAIR.AI Top AI Papers of the Week wiki, drawn from the top 5 papers per week in 2026 and paired with their arXiv abstracts. Three small Python scripts handle the pipeline, namely parse_2026.py (wiki to structured entries), fetch_abstracts.py (arXiv abstract lookup), and build_jsonl.py (chat-format assembly). The chat schema is the standard Fireworks shape:
The final outputs are train.jsonl and val.jsonl (plus the combined wiki-sft-2026.jsonl for reference), with about 90 percent of records reserved for training and 10 percent for validation.
Step 2: Upload the Dataset to Fireworks
Confirm the dataset is `READY`:
The dataset path you will pass to the Fireworks Agent looks like accounts/<your-account>/datasets/wiki-sft-2026.
Step 3: Kick Off the Fireworks Agent
This is the entire user-facing config for the run, just one instruction.
The session returns an ID like 1777224532-7ddb. Stream the events:
The --wait flag is important; without it, the command dumps existing events and exits. The Claude Code slash command handles this for you.
Step 4: Approve the Plan and Promote the Winner
The agent surfaces two gates. The first is a plan with a cost estimate and three HP configs to sweep in parallel, with validation loss as the evaluator, which you approve to resume streaming.
The HP sweep then runs three SFT jobs in parallel and returns a ranked table, after which the agent surfaces a second gate with the winning config. In my run, the top three configs landed very close to each other on eval loss, which tells you the task is not particularly HP-sensitive at this dataset size, so approving full training is the obvious next step.
Full training takes about eight minutes of GPU time and costs a few cents.
Step 5: Verify the Deployment
Deployment is where ad-hoc fine-tuning workflows usually go sideways, picking the wrong accelerator, missing a compatible shape, or stalling on capacity. The agent handles the recovery itself, so the session lands at status succeeded with a READY scale-to-zero deployment. Confirm the deployment with the following command:
Step 6: Call the Model
Inference uses the standard Fireworks chat completions endpoint, with a deployment-pinned model ID so requests route to your custom deployment:
Once warm, calls return fast enough to use as an inline step inside an agent rather than a batch job.
## Why This Workflow Pays Off
I tested the fine-tuned model on a few papers that sit outside the training set, sending the same system prompt and abstract to both the baseline qwen3-8b and the fine-tuned model. The fine-tuned model produces affiliation-led ledes that name the researchers' lab, followed by three to five bullets with bolded short-label prefixes (Method:, Performance Gains:, Scalability:), and an analytical, non-promotional tone. For instance, on Chain-of-Thought, it opened with "Researchers at Stanford University demonstrate that chain-of-thought prompting significantly enhances large language models' reasoning capabilities..." That is the wiki's voice, baked into the weights and produced in a single fast call.
The practical payoff is that you no longer need a large, inefficient LLM or agent to write the summaries for your LLM Wiki. A smaller fine-tuned model can do it effectively, efficiently, and cheaply. Getting the style and tone right matters for this use case, and no amount of tuning a skill or system prompt can replace what a properly fine-tuned LLM gives you.
Two more things make this useful beyond a one-off experiment. First, training becomes a tool, not a project, with one CLI command, cents of compute, and a real callable endpoint at the end, while the agent handles the boring failure modes. Second, you own the resulting model. The weights live in your account, deployed on infrastructure you control, and the idle cost is zero. At this price and friction, reaching for SFT becomes a reasonable answer to a much wider set of style and format problems.
## What's Next, Knowledge in the Weights
I intentionally stopped at style transfer because it is the cleanest first SFT target on a small dataset. The harder version Karpathy described (your wiki's contents in the weights) is the natural follow-up, with synthetic data generation, more training records, and knowledge-recall evaluators in the loop.
The pattern generalizes beyond a personal papers wiki. Any structured knowledge surface (an internal docs wiki, a product manual, a research vault) is a candidate for the same two-step recipe, where you SFT on style first and layer knowledge injection on top. A model that has internalized both the voice and the substance of a corpus is what makes a personalized agent on top of it genuinely useful.
Fireworks Agent is currently in private preview and will be generally available soon. If you are thinking about applying this workflow to your own corpus and want to request access or talk it through with the Fireworks team, reach out at fireworks.ai/contact-training.
译本文探讨了通过微调，将个人知识库（如LLM Wiki）的内容从依赖上下文窗口，转变为固化到模型自身权重中的方法。关键在于利用如Fireworks Agent这样的自主AI代理，仅需提供自然语言目标，它就能自动完成从数据准备、训练到部署的完整微调流程。这标志着模型自我改进的闭环成为可能：当训练成为AI工作流中一个可调用的步骤时，模型能主动将反复使用的模式（如特定写作风格或决策逻辑）学习并内化到权重中，从而实现使用与优化的持续迭代。
AYi@AYi_AInotes