Ch 11: Large Language Models & Transformers - Intermediate¶
Track: Practitioner | Try code in Playground | Back to chapter overview
Read online or run locally
To run the code interactively, clone the repo and open chapters/chapter-11-large-language-models-and-transformers/notebooks/02_pretrained_llms.ipynb in Jupyter.
Chapter 11: LLMs & Transformers — Notebook 02 (Working with Pretrained LLMs)¶
This notebook moves from theory to practice: load pretrained models (BERT, DistilBERT, GPT-style) with Hugging Face transformers, tokenize, extract embeddings, and build a frozen-embedding classifier.
What you'll learn¶
| Topic | Section |
|---|---|
Loading pretrained models with transformers (and graceful fallback) | §1 |
AutoTokenizer and tokenization details | §2 |
| Extracting and visualizing token / sentence embeddings | §3 |
| Mean pooling and similarity search | §4 |
| Frozen-embedding classifier with scikit-learn | §5 |
| Choosing among BERT / RoBERTa / DistilBERT / GPT | §6 |
Time estimate: 3 hours
Key concepts¶
- Pretrained LLM — A model already trained on huge corpora; reuse its representations instead of training from scratch.
- Tokenizer — Maps text to subword IDs (BPE / WordPiece / SentencePiece) the model expects.
- Embeddings — Hidden states (per token or pooled) make excellent fixed features for downstream tasks.
- Frozen-embedding classifier — Encode text once with an LLM, then train a small sklearn classifier on top — fast and strong.
- Model selection — Pick by task (classification vs generation), latency budget, and license.
Run the full notebook for code and outputs.
Generated by Berta AI