Creating a Lexer in C - Part 2

In Part 2 of this lexer series, we continue building a lexer in C by implementing two core functions: nextToken() createToken() These functions form the heart of the lexer, allowing us to scan input, recognise patterns, and return structured tokens that a parser can later consume. In this video, you’ll learn: How a nextToken() function drives lexical analysis How to design a clean createToken() function in C How tokens are represented and returned safely How to extend your lexer without breaking existing logic This episode builds directly on Part 1, where we handled input and lexer structure, and moves us closer to a real compiler-style workflow. This series is for you if you want to: Understand compilers from the ground up Write real systems-level C code Learn how lexers work beyond theory 📌 Previous video: ▶️ https://www.youtube.com/watch?v=kXZhhxYYtU0 GitHub Inspiration: https://github.com/ThePrimeagen/ts-rust-zig-deez/blob/master/c/src/lexer/lexer.c #lexer #tokenization #programming #cprogramming Chapters: Introduction - 0:00 Initial Setup - 1:53 nextToken - 3:12 createToken - 11:50 Tidying Up - 22:30 Conclusion - 25:25