How Do Videos Work?

Modern video files rely on some of the most advanced algorithms ever created to achieve high compression. So, writing a video file is far from simple, and explaining the process is even harder. I did my best to keep this video as short as possible while still making it easy to follow. The H264/AAC MP4 file, also known as AVC/AAC, is currently used by the majority of all videos on the internet. However, almost nobody knows exactly how these files work. In this video, we will uncover the exact processes used by real encoders to encode an uncompressed video into an MP4 file. ******** PLEASE CHECK THESE OUT!!! ******** H264 (AVC) Specs: https://www.itu.int/rec/T-REC-H.264-202108-S/en AAC Specs: https://ossrs.io/lts/zh-cn/assets/files/ISO_IEC_13818-7-AAC-2004-67b015c6ddfc9a4af83665738477124a.pdf MP4 container format documentation: https://developer.apple.com/documentation/quicktime-file-format Online MP4 parser: https://onlinemp4parser.com/ *Open source encoders:* H264: https://github.com/cisco/openh264/tree/master/codec/encoder/core AAC: https://github.com/knik0/faac/tree/master/libfaac *Side notes:* At 24:45, because the chroma coded block pattern is already defined in the mb_type, it doesn't need to be encoded. The luma coded block pattern is only encoded if the 3rd suffix bit is 0. This only applies for intra_16x16, not in P slices. Also, only one intra prediction mode is stored for both chroma blocks, stored in intra_chroma_pred_mode. *Timestamps:* 0:00 Intro 1:23 What is H264 and AAC? 2:23 Uncompressed video and audio format *VIDEO COMPRESSION (I slice)* 3:13 Slices, Macroblocks, IDR frame 3:29 Color Space Conversion: RGB → YCrCb 4:01 4:2:0 Chroma Subsampling 5:13 Intra Prediction 7:17 Residuals 8:29 Frequency Transform (Luma) 11:44 Frequency Transform (Chroma) 12:26 Quantization 14:16 Zigzag scan 15:11 Video compression summary *BIT ENCODING PROCESSES* 15:27 CABAC (Context Adaptive Binary Arithmetic Coding) 16:18 CABAC Initialization 18:41 CABAC Encoding process (Important!!!!) 21:19 CABAC Termination symbol (end_of_slice_flag = 1, ctxidx=276) 22:32 CABAC bypass mode 23:09 Unary, Signed exp-golomb, Unsigned exp-golomb *I-SLICE ENCODING* 24:16 Macroblock type 25:53 (Only for Intra_4x4) Table of prediction modes 26:54 Intra chroma prediction mode 27:10 CBP Luma (See side notes in description) 27:54 CBP Chroma 28:40 QP Delta 28:56 Luma residuals 31:45 Chroma DC (Blue then red) 32:26 Chroma AC (Blue then red) *Slice headers/info* 33:10 Slice types (I slice, P slice, B slice, IDR frame) 34:29 SPS and PPS (Sequence/Picture parameter sets) 35:56 NAL Unit header and slice header 39:10 RSBP Trailing bits (Put after CABAC termination symbol) 39:23 NAL Unit 4 byte size prefix *VIDEO COMPRESSION (P slice)* 39:34 P slice header walkthrough 39:56 Luma inter prediction 41:20 Hexagon search (Inter prediction) 42:10 Half-pel (Half-pixel) refinement 42:56 6-tap interpolation (Half-pixel search) 44:12 Quarter-pixel refinement 45:04 Inter prediction heuristics 46:04 Chroma inter prediction 46:28 Inter prediction residuals *P-SLICE ENCODING* 46:46 Macroblock skip flag (Has nothing changed?) 47:12 Macroblock type (P slices) 47:41 Sub macroblock types 47:52 Reference frame index 48:19 Motion vectors (All Xs then all Ys) 50:11 CBP luma, CBP chroma, QP_delta, residuals... 50:31 Slice type assignments *PART 2: AUDIO* 50:50 Uncompressed audio *Audio compression* 51:22 Split into audio frames 52:09 Frequency transform (Modified discrete cosine transform) 53:38 (Informative) How to find the pitch of a frequency coefficient 53:53 Audio quantization 54:10 Split frames into 49 bands 54:58 Determine quantization weights, then quantize *Audio encoding* 56:41 ↓Read the AAC specs in description for more info!↓ 57:04 Audio frame info 58:16 Scale factor data 59:01 ↓Read the AAC specs in description for more info!↓ 59:11 Spectral data (Frequency coefficients) 1:01:18 Section data (Band codebooks) *MP4 Container* 1:02:03 The MP4 file 1:02:58 ftyp 1:03:23 mdat 1:03:57 moov 1:05:49 stbl 1:06:03 avcC (Put SPS and PPS here) 1:08:01 Audio track 1:09:16 ↓Check the description for more info!↓ 1:09:45 Recap 1:10:13 Decoding an MP4 file *H264 glitches I found* Missing the initial IDR frame: https://www.youtube.com/watch?v=QlMVNMo7DM0 #multimedia #science #computers #computerscience #computerengineering #algorithm #file #fileformat #mp4 #mp3 #h264 #compression #jpeg #imagecompression #videocompression #aac #audio #film #movie #motionpicture #effectking #localstarbinit #comedyeffect #vikramsarkar #tiktokeffects #moneylove #catloverlife #ciddubbing #flyxoedits #effects #processing #foryou #recommended #technical #bit #bytes #coding #encoding #encoder #codec #decoder #itu #software #megabytes #fft #dct #tech #math #residuals #luma #chroma #rgb #pixels #picture #png #jpg #mov #avi