Continued pretraining is how many specialized AI models actually get built.
In this video, I explain where continued pretraining fits in the AI training pipeline, why it is different from fine-tuning, and how models like Code Llama, Composer 2, Meditron, finance models, and legal models become good at specific domains without being trained from scratch.
Timestamps:
0:00 Continued pretraining, explained
0:31 Stage 1: pretraining from scratch
1:00 Stage 2: continued pretraining
1:44 Stage 3: fine-tuning
1:59 What continued pretraining teaches vs fine-tuning
2:50 Code Llama example
3:22 Composer 2 example
3:35 Catastrophic forgetting
4:11 Why specialized AI models are built this way
If this was helpful, like and subscribe for more AI engineering breakdowns.
Check out my book: Get Insanely Good at AI 👉 https://getaibook.com/book
#ai #coding #programming
