Crash Course IR - Fundamentals

In this lecture we explore two fundamental building blocks of information retrieval (IR): indexing and ranked retrieval with TF-IDF and BM25 scoring models. Slides & transcripts are available at: https://github.com/sebastian-hofstaetter/teaching 📖 Check out Youtube's CC - we added our high quality (human corrected) transcripts here as well. Slide Timestamps: 0:00:00 1 - Welcome 0:00:19 2 - Today 0:00:58 3 - Information Retrieval 0:01:19 4 - Information Retrieval (Finding the needle in the haystack) 0:01:47 5 - Notes on terminology 0:03:04 6 - Relevance (based on text content) 0:04:10 7 - Inverted Index 0:04:18 8 - Inverted Index 0:05:43 9 - Inverted Index 0:07:00 10 - Creating the Inverted Index 0:08:41 11 - Tokenization 0:09:57 12 - Stemming 0:10:55 13 - Search 0:11:21 14 - Querying the Inverted Index 0:13:13 15 - Types of queries (including, but not limited to) 0:14:19 16 - Inverted Index: Dictionary 0:15:31 17 - Dictionary data structures 0:17:39 18 - Hash table 0:18:39 19 - Spell-checking 0:19:58 20 - Spell-checking by Peter Norvig 0:21:24 21 - Relevance Models 0:21:42 22 - Scoring model 0:22:44 23 - Search algorithm 0:23:51 24 - Relevance 0:24:38 25 - Relevance 0:25:22 26 - Relevance limitations 0:26:25 27 - TF-IDF 0:26:41 28 - Term Frequency – conceptional data view 0:27:32 29 - Term Frequency – actual data storage 0:28:24 30 - TF - Term Frequency 0:29:07 31 - Term Frequency & Logarithm 0:30:05 32 - Document Frequency 0:31:05 33 - IDF – Inverse Document Frequency 0:31:56 34 - TF-IDF 0:33:23 35 - TF-IDF – Usage 0:34:19 36 - BM25 0:34:49 37 - BM25 0:35:32 38 - BM25 (as defined by Robertson et al. 2009) 0:37:59 39 - BM25 vs. TF-IDF 0:38:35 40 - BM25 vs. TF-IDF - Saturation 0:39:09 41 - BM25 vs. TF-IDF - Example 0:40:41 42 - Hyperparameters 0:41:55 43 - BM25F 0:42:55 44 - BM25F (as defined by Robertson et al. 2009) 0:43:29 45 - BM25F 0:44:31 46 - 1998: Google 0:45:34 47 - Summary: Crash Course – Fundamentals 0:46:23 48 - Thank You