Fine-Tuning
Fine-tuning is the process of further training a pre-trained language model on a smaller, task-specific dataset to improve its performance on a particular type of task or domain.
Definition¶
Fine-tuning is a machine learning technique where a model that has already been trained on a large general corpus (the base model) is further trained on a smaller, curated dataset focused on a specific task, domain, or style. The additional training adjusts the model's weights to improve performance on the target task without starting from scratch.
Pre-training vs. Fine-tuning¶
Pre-training is the initial, large-scale training phase where the model learns general language understanding from massive text corpora. This is computationally expensive and done once.
Fine-tuning happens after pre-training and refines the model for specific applications. It's faster and cheaper because the model already understands language — fine-tuning just adjusts its behavior for a narrower task.
Examples of Fine-Tuning¶
- Training a base LLM on medical literature to improve performance on clinical documentation tasks
- Training on customer service conversations to create a domain-specific support chatbot
- Training on legal documents to improve contract analysis accuracy
- Training on examples of humanized text (AI input → human output pairs) to improve an AI humanizer's output quality
Fine-Tuning vs. Prompt Engineering¶
Both techniques improve model output quality, but they work differently:
- Prompt engineering shapes the model's behavior through carefully crafted instructions at inference time — no training required
- Fine-tuning shapes the model's behavior by updating its weights — requires a training dataset and compute
Fine-tuning tends to produce more consistent results for specific tasks but requires more resources. Prompt engineering is more flexible and faster to iterate.
Instruction Fine-Tuning and RLHF¶
Modern chat models like ChatGPT are not just pre-trained but also fine-tuned through instruction fine-tuning (training on example instruction-following) and RLHF (Reinforcement Learning from Human Feedback), where human raters score model outputs and those scores are used to further train the model to produce preferred outputs.