INTRODUCTION: Learn real-time video processing in C++

Subscribe to this channel to take part in the weekly live-streams! This video serves as an introduction to the programming project I'm doing exclusively on stream. MUSIC FEATURED: Jacob Collier - Don't You Worry 'Bout A Thing Yussef Dayes x Alfa Mist - Love is The Message (at Abbey Road) VIDEO TRANSCRIPT: Hello my name is Bartholomew and I’m a software developer and musician. And I want to build a visualisation tool that musicians can use to enhance their live performances. To explain the kind of visuals I want to create, let me show you an old music video by Jacob Collier. This is “Don’t You Worry Bout A Thing” and in this song he’s playing every part himself. In the video he shows us in these grids the different parts that together comprise the song. It’s a joy to watch. You hear a sound and you go and look for it in the grid, you find it, and you see exactly what he’s playing. The music video serves as a really great visualisation of the music, and what I want to do is create these kinds of grids live. So what do I mean by that? Well, let me demonstrate with an example solo performance. This is in my room at 1 in the morning, but imagine that it’s actually in a venue with an audience. I’m using live looping and samples to layer multiple tracks on top of each other. There is a drum loop and a guitar loop that are pre-recorded, and then there are backing vocals that were recorded live earlier in the set, that get played when I reach the chorus. I want to create a grid live placed behind me, that will have a video for every sound you hear in the performance. So in this case it will have videos for those pre-recorded samples as well as videos of those live-looped vocals. How can this be achieved? Well, in the case of pre-recorded samples, for every sample that I have in my Ableton Live set, I will give my visualisation tool a video to go along with the sample. Whenever I trigger a sample within Live, my tool will play the associated video in the grid behind me. So every sample that the audience can hear, the audience can see. Now, in the case of live loops, I’ll have a webcam or GoPro set up on stage facing me. Whenever I start recording a loop, the camera will start recording me. Whenever I play back the loop, the camera’s video will be played back in the grid. Does that make sense? I think a tool like this could be really fun, and could greatly enhance solo performances where the performer is relying a lot on a sampler or DAW to create their sound. So this is the visual tool I want to develop, and starting this Saturday I will be live-streaming the entire development process on YouTube. All the real-time video processing code will be done in C++ using FFmpeg and OpenGL. The layer that communicates with Ableton Live might be done in a scripting language, but I’m not sure yet. I think this would be a valuable programming stream to watch for any programmers interested in video graphics. Very few programmers actually ever dive into the systems underpinning real-time video processing. Usually, you can get a video player component to take care of all the heavy lifting, and you just have to tell it when to play and to stop. But here I’ve got a very specific challenge where I have to stream multiple video sources as well as record multiple video sources in an efficient manner. Not to mention that all these videos have to be correctly synchronised to the live performance. So if you want to learn more about how video processing works, then you can learn with me on these live streams. So, Saturday the 9th of November at 6pm GMT I will start the first stream where I will cover the general architecture of the project and then set up FFmpeg and start to decode video files. I will link the Youtube channel below, so if you’re interested, subscribe and I’ll see you on Saturday!