Large Language Models and Knowledge Graphs: Merging Flexibility and Structure

We discuss how to infuse Large Language Models (LLMs) with Knowledge Graphs (KGs)! This is a very exciting approach, as we can combine the flexibility and generalisability of LLMs with the structure and reliability of KGs, and is a first step towards neurosymbolic architectures! I will also be going through a LangChain implementation of LLMs with knowledge graphs as inputs, demonstrate some of the limitations currently faced, and show how we can better prompt engineer KG usage with LLMs using my very own StrictJSON Framework. ~~~~~~~~~~~~~ Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/LLM%20with%20Knowledge%20Graphs.pdf Jupyter Notebook: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Tutorial/LLM%20with%20Knowledge%20Graphs.ipynb Jupyter Notebook (updated for StrictJSON v4.0.0): https://github.com/tanchongmin/strictjson/blob/main/Experiments/LLM%20with%20Knowledge%20Graphs.ipynb StrictJSON Framework: https://github.com/tanchongmin/strictjson LangChain Documentation: https://python.langchain.com/docs/get_started/introduction.html Tutorial on how to use LangChain and StrictJSON Framework for Knowledge Graphs and LLMs: https://www.youtube.com/watch?v=h2RitC7DUvo Paper on Unifying LLMs and Knowledge Graphs: https://arxiv.org/abs/2306.08302 LLMs as Graph Neural Networks / Embeddings ERNIE: https://arxiv.org/abs/1905.07129 TransE embeddings used by ERNIE: https://proceedings.neurips.cc/paper/2013/file/1cecc7a77928ca8133fa24680a88d2f9-Paper.pdf QA-GNN: https://arxiv.org/abs/2104.06378 FactKB: https://arxiv.org/abs/2305.08281 ~~~~~~~~~~~~~ 0:00 Introduction 1:55 Pros and Cons of LLMs and Knowledge Graphs (KGs) 4:55 Retrieval Augmented Generation (RAG) 8:10 Problems with LLMs and RAG 17:40 Basics of KG 26:09 Hierarchy in KG 31:13 KGs can be structurally parsed 33:17 KG can represent environmental transitions 33:58 KG as tool/memory for LLM 39:16 3 approaches to integrate KG and LLMs 40:21 Approach 1: KG-augmented LLMs 59:05 Approach 2: LLM-augmented KG 1:05:37 Approach 3: LLMs and KG two-way interaction 1:10:16 LangChain Graph QA Example 1:16:35 Strict JSON Framework Graph QA Example 1:23:00 Discussion ~~~~~~~~~~~~~ AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator. Discord: https://discord.gg/bzp87AHJy5 LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/ Online AI blog: https://delvingintotech.wordpress.com/ Twitter: https://twitter.com/johntanchongmin Try out my games here: https://simmer.io/@chongmin