Chapter 14: Fine-tuning & Adaptation Techniques¶

Teach pre-trained models new behaviors with your data—when to fine-tune, instruction datasets, supervised fine-tuning loops, parameter-efficient methods (LoRA, QLoRA, adapters, IA3), preference data (DPO), and rigorous evaluation.

Metadata¶

Learning Objectives¶

• Decide when to fine-tune vs prompt vs RAG, with cost / latency / quality trade-offs
• Prepare instruction datasets (formatting, splits, token budgets, response masking)
• Run a supervised fine-tuning (SFT) loop with masked loss and early stopping
• Implement LoRA from scratch and apply PEFT methods (QLoRA, adapters, prefix tuning, IA3)
• Use preference data: RLHF and DPO concepts with a NumPy DPO loss
• Evaluate adapted models rigorously and plan deployment via a model registry

What's Included¶

Notebooks¶

Scripts¶

• config.py — Chapter config, dataset paths, optional-framework flags
• dataset_utils.py — Instruction formatting, splits, tokenization budgets, response masking
• training_utils.py — Tiny SFT loop helpers, loss masking, schedules, early stopping
• peft_utils.py — NumPy LoRA adapter (rank, alpha, scaling), merge / serve helpers

Exercises¶

• Problem Set 1 (notebook) — Format an instruction dataset, token budgets, loss masking, choose hyperparameters, FT vs RAG, tiny SFT loop
• Problem Set 2 (notebook) — Implement LoRA forward, parameter-efficiency ratios, merge adapters, DPO loss in NumPy, win-rate eval, registry entry
• Solutions — In exercises/solutions/ (notebooks and solutions.py for CI)

Diagrams (Mermaid)¶

• fine_tuning_spectrum.mermaid, lora_architecture.mermaid, training_pipeline.mermaid

Read Online¶

• 14.1 Introduction — When to fine-tune, dataset prep, SFT loop, evaluation basics
• 14.2 Intermediate — LoRA math + NumPy implementation, QLoRA, adapters, IA3, merging
• 14.3 Advanced — Instruction tuning, DPO, evaluation, forgetting, registry, capstone

Or try the code in the Playground.

How to Use This Chapter¶

1. Clone and install dependencies

git clone https://github.com/luigipascal/berta-chapters.git
cd berta-chapters
pip install -r requirements.txt

2. Navigate to the chapter

cd chapters/chapter-14-fine-tuning-and-adaptation
pip install -r requirements.txt

3. (Optional) Install the heavy framework extras (GPU helpful)

pip install torch transformers peft accelerate datasets trl bitsandbytes

4. Launch Jupyter

jupyter notebook notebooks/01_fine_tuning_basics.ipynb

Created by Luigi Pascal Rondanini | Generated by Berta AI