DETR for panoptic segmentation - algorithm and code reading

The video is a part of a larger playlist about image segmentation: https://studio.youtube.com/playlist/PL1HdfW5-F8AR9p2zKvFO-bdQsGp6GVHQi/videos This video talks about DETR - the first transformer-based object detector, this time focusing on the panoptic segmentation part of the model. It talks about panoptic segmentation task itself, PQ metric, panoptic segmentation labels for CoCo dataset, and explains how DETR is used for segmentation. Second part of the video demonstrates an example of a jupyter notebook with DETR inference and loss calculation, and dives into the source code of the model itself. Important links: - DETR paper https://arxiv.org/pdf/2005.12872 - DETR source code https://github.com/facebookresearch/detr - Panoptic Segmentation paper shown in the video https://arxiv.org/pdf/1801.00868 - Paper about CoCo panoptic annotations https://arxiv.org/pdf/1612.03716 - Segment Anything paper shown in the video: https://arxiv.org/pdf/2304.02643 - Jupyter notebook shown in the video https://github.com/adensur/blog/blob/main/computer_vision_zero_to_hero/35_detr_segmentation/sandbox.ipynb - Installation instructions: https://github.com/adensur/blog/blob/main/computer_vision_zero_to_hero/35_detr_segmentation/Install.md - My video about DETR for object detection: https://youtu.be/A2f4w54fSsM - My video about reading DETR source code: https://youtu.be/UfutqqyFjEM 00:00 - Intro 01:30 - Panoptic Segmentation Task 06:13 - PQ Metric 10:46 - Panoptic for CoCo 17:18 - DETR Segmentation Head 30:12 - Reading The Code 38:13 - Query to Pixel Attention Maps 43:59 - Convolutional Segmentation Head 50:50 - Loss