Build Motion Control Interfaces with Multimodal LLMs

Talk: Charlie Gerard - "Motion Control With Multimodal LLMs", JSNation 2025 What if you could use multimodal LLMs to interact with websites or IoT devices using motion control? As advancements in multimodal AI offer new opportunities to push the boundaries of what can be done with this technology, I started wondering how it could be leveraged from the perspective of human-computer interaction. In this talk, I will take you through my research experimenting with building motion-controlled prototypes using LLMs in JavaScript. Watch the full talk with Q&A on GitNation website: https://gitnation.com/contents/motion-control-with-multimodal-llms Follow the link to watch the full version of all the conference talks, QnA’s with speakers and hands-on workshop recordings → https://gitnation.com/events/jsnation-2025 Timestamps: 00:00 Introduction to Motion Control with Multimodal AI 01:24 Exploring TensorFlow.js Models for Motion Control 02:47 Enhancing User Interactions with Gesture Control 04:45 Redefining Coding Interfaces with Minimal Inputs 06:58 Leveraging LLMs for Gesture-Based Interaction 08:55 Investigating LLMs for Motion Control Experiences 10:18 Exploring Gemiini for Gesture Recognition 12:34 Webcam-Based Gesture Recognition with Gemini 14:28 Controlling Light States with Gemini Functions 15:46 Interacting with Gemini and WiFi Light Bulbs 17:44 Enabling Multi-Light Control with ID and Position 19:05 Custom Light Names and Gestural Control Logic 20:13 Live API Functions and Directional Light Control 22:57 Integrating TensorFlow.js for Color Control 24:08 Color Changes with Hand Gestures 26:07 Custom Gesture Databases and Autonomous Interfaces 28:28 Personalized Interfaces and Motion Control 🔥 Want to hear talks like this live? Join us at JSNation US 2025 in New York: https://jsnation.us/?utm_source=youtube&utm_medium=charliegerard JSNation website: https://jsnation.com JSNation on X: https://x.com/thejsnation JSNation on LinkedIn: https://www.linkedin.com/showcase/jsnation-the-main-js-conference/ JSNation on Bluesky: https://bsky.app/profile/jsnation.gitnation.org JSNation on Instagram: https://www.instagram.com/thejsnation/ GitNation on TikTok: https://www.tiktok.com/@gitnationorg #JSNation #MultimodalLLMs #GitNation #JSNation2025 #MotionControl #TechTalks