Yale Song

Senior Staff Research Scientist and Manager, Google
<first_name><last_name> AT gmail DOT com

Scholar | Github

Latest News

Professional Service

    • Organizing Committee: CVPR 2025/2024/2023 (Socials Chair), ICLR 2023/2021 (Paper Award Committee), NeurIPS 2021/2020 (Expo Chair), ICMI 2021 (Sponsorship Chair), ICMI 2019 (Program Chair)
    • Area Chair: NeurIPS 2025/2024/2023/2022/2021/2020, ICLR 2026/2024/2023/2021, ICML 2024/2023, CVPR 2026*/2025*/2024/2020, ECCV 2024*/2022, ICCV 2025/2023/2021, WACV 2023/2020/2018, FG 2018, ICMI 2018/2017/2016, ACL 2019 (*: Lead AC)
    • Journal Editor: Transactions on Machine Learning Research (2022-Current)
    • Outstanding Area Chair: ICLR 2023
    • Outstanding reviewer: ICML 2021/2020, NeurIPS 2019, CVPR 2017

Publications (see also at Google Scholar, DBLP)

    2025

  1. PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
    Yale Song†, Jang Hyun Cho†, Andrea Madotto†, Effrosyni Mavroudi†, Triantafyllos Afouras†, Tushar Nagarajan†, Muhammad Maaz†, Tengyu Ma†, Shuming Hu†, Hanoona Rasheed, Peize Sun, Po-Yao Huang, Daniel Bolya, Suyog Jain, Miguel Martin, Huiyu Wang, Nikhila Ravi, Shashank Jain, Tammy Stark, Shane Moon, Babak Damavandi, Vivian Lee, Andrew Westbury, Salman Khan, Piotr Dollar, Lorenzo Torresani, Kristen Grauman, Christoph Feichtenhofer (†: joint first author)
    NeurIPS 2025 (Spotlight) [Preprint] [Model] [Code] [Data]

  2. Enrich and Detect: Video Temporal Grounding with Multimodal LLMs
    Shraman Pramanick, Effrosyni Mavroudi, Yale Song, Rama Chellappa, Lorenzo Torresani, Triantafyllos Afouras
    ICCV 2025 (Highlight)

  3. Streaming VideoLLMs for Real-Time Procedural Video Understanding
    Dibyadip Chatterjee, Edoardo Remelli, Yale Song, Bugra Tekin, Abhay Mittal, Bharat Lal Bhatnagar, Necati Cihan Camgoz, Shreyas Hampali, Eric Sauser, Shugao Ma, Angela Yao, Fadime Sener
    ICCV 2025

  4. VITED: Video Temporal Evidence Distillation
    Yujie Lu, Yale Song, Lorenzo Torresani, William Yang Wang, Tushar Nagarajan
    CVPR 2025 [Preprint]

    5. 2024

  5. Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, David Crandall, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, Michael Wray
    CVPR 2024 (Oral) [Preprint] [Blog] [Data]

    6. 2023

  6. Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities
    Yale Song, Gene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, Lorenzo Torresani
    NeurIPS 2023 Datasets & Benchmarks (Spotlight) [Paper] [Data]

  7. EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, Pengchuan Zhang
    ICCV 2023 (Highlight) [Preprint]

  8. Egocentric Video Task Translation
    Zihui Xue, Yale Song, Kristen Grauman, Lorenzo Torresani
    CVPR 2023 (Highlight) [Preprint]

  9. Scaling Novel Object Detection with Weakly Supervised Detection Transformers
    Tyler LaBonte, Yale Song, Xin Wang, Vibhav Vineet, Neel Joshi
    WACV 2023 [Preprint][Poster]

    10. 2022

  10. Neural-Sim: Learning to Generate Training Data with NeRF
    Yunhao Ge, Harkirat Behl, Jiashu Xu, Suriya Gunasekar, Neel Joshi, Yale Song, Xin Wang, Laurent Itti, Vibhav Vineet
    ECCV 2022

  11. COMPASS: Contrastive Multimodal Pretraining for Autonomous Systems
    Shuang Ma, Sai Vemprala, Wenshan Wang, Jayesh K. Gupta, Yale Song, Daniel McDuff, Ashish Kapoor
    IROS 2022 [Preprint] [Blog] [Code]

  12. Visual Attention Emerges from Recurrent Sparse Reconstruction
    Baifeng Shi, Yale Song, Neel Joshi, Trevor Darrell, Xin Wang
    ICML 2022 [Preprint] [Slides] [Code]

  13. Robust Contrastive Learning against Noisy Views
    Ching-Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, Yale Song
    CVPR 2022 [Preprint] [Code]

  14. CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
    Daniel McDuff, Yale Song, Jiyoung Lee, Vibhav Vineet, Sai Vemprala, Nicholas Alexander Gyde, Hadi Salman, Shuang Ma, Kwanghoon Sohn, Ashish Kapoor
    CLeaR 2022 [Preprint] [Project and code] [Blog]

  15. DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
    Tsu-Jui Fu, William Yang Wang, Daniel McDuff, Yale Song
    AAAI 2022 [Preprint] [Project and dataset]

  16. Anomaly Detection in Time Series with Robust Variational Quasi-Recurrent Autoencoders
    Tung Kieu, Razvan Cirstea, Yan Zhao, Bin Yang, Chenjuan Guo, Yale Song, Christian Jensen
    ICDE 2022

    17. 2021

  17. Contrastive Learning of Global and Local Video Representations
    Shuang Ma, Zhaoyang Zeng, Daniel McDuff, Yale Song
    NeurIPS 2021 [Paper] [Code]

  18. ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
    Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, Yale Song
    ICCV 2021 [Preprint] [ACAV100M Dataset] [MSR Blog]

  19. Parameter Efficient Multimodal Transformers for Video Representation Learning
    Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, Yale Song
    ICLR 2021 [Preprint] [Code] [MSR Blog]

  20. Self-Supervised Learning of Compressed Video Representations
    Youngjae Yu, Sangho Lee, Gunhee Kim, Yale Song
    ICLR 2021 [Preprint]

  21. Active Contrastive Learning of Audio-Visual Video Representations
    Shuang Ma, Zhaoyang Zeng, Daniel McDuff, Yale Song
    ICLR 2021 [Preprint] [Code]

    22. 2020

  22. Attention-Based Deep Metric Learning for Near-Duplicate Video Retrieval
    Kuan-Hsun Wang, Chia Chun Cheng, Yi-Ling Chen, Yale Song, Shang-Hong Lai
    ICPR 2020 (Oral)

  23. Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency
    Matt Whitehill, Shuang Ma, Daniel McDuff, Yale Song
    INTERSPEECH 2020 [Preprint]

  24. Phans, Stans and Cishets: Self-Presentation Effects on Content Propagation in Tumblr
    Michael M. Yoder, Qinlan Shen, Alex Coda, Yunseok Jang, Yale Song, Kapil Thadani, Carolyn P. Rose
    WebSci 2020

  25. Image to Video Domain Adaptation Using Web Supervision
    Andrew Kae and Yale Song
    WACV 2020 [Preprint]

    26. 2019

  26. Characterizing Bias in Classifiers using Generative Models
    Daniel McDuff, Shuang Ma, Yale Song, Ashish Kapoor
    NeurIPS 2019 [Preprint]

  27. Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
    Shuang Ma, Daniel McDuff, Yale Song
    ICCV 2019 [Paper] [Project]

  28. Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
    Yale Song and Mohammad Soleymani
    CVPR 2019 [Paper] [Project] [Code and Dataset]

  29. Neural TTS Stylization with Adversarial and Collaborative Games
    Shuang Ma, Daniel McDuff, Yale Song
    ICLR 2019 [Paper] [Code] [Press]

  30. Visual Question Answering with Spatio-Temporal Reasoning
    Yunseok Jang, Yale Song, Chris Dongjoo Kim, YoungJae Yu, Youngjin Kim, Gunhee Kim
    IJCV 2019 [Paper] [Code and dataset]

    31. 2018

  31. Video Prediction with Appearance and Motion Conditions
    Yunseok Jang, Gunhee Kim, Yale Song
    ICML 2018 [PDF] [Project] [Code]

  32. Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
    Yipin Zhou, Yale Song, Tamara L. Berg
    WACV 2018 [arxiv] [project]

    33. 2017

  33. Learning from Noisy Labels with Distillation
    Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, Jia Li
    ICCV 2017 [arxiv] [slides] [YFCC100M Entity Dataset]

  34. ElasticPlay: Interactive Video Summarization with Dynamic Time Budget
    Haojian JIn, Yale Song, Koji Yatani
    ACM Multimedia 2017 (Oral) [arxiv] [demo] [video]

  35. TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
    Yunseok Jang, Yale Song, YoungJae Yu, Youngjin Kim, Gunhee Kim
    CVPR 2017 (Spotlight) [arxiv] [Code and dataset]

  36. Improving Pairwise Ranking for Multi-label Image Classification
    Yuncheng Li, Yale Song, Jiebo Luo
    CVPR 2017 [arxiv]

    37. 2016

  37. Real-Time Video Highlights for Yahoo Esports
    Yale Song
    NIPS Workshop, LSCVS 2016, [arxiv]
    In production at Yahoo eSports (Match Highlights)

  38. To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videos
    Yale Song, Miriam Redi, Jordi Vallmitjana, Alejandro Jaimes
    CIKM 2016, [arxiv] [Slides] [Code] [Dataset]
    In production at Tumblr and Flickr (Thumbnails from user-generated videos)

  39. TGIF: A New Dataset and Benchmark on Animated GIF Description
    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, Jiebo Luo
    CVPR 2016 (Spotlight), [arxiv] [Dataset] [Project]

  40. Video2GIF: Automatic Generation of Animated GIFs from Video
    Michael Gygli, Yale Song, Liangliang Cao
    CVPR 2016, [arxiv] [Demo] [Code] [Dataset]
    Press coverage: Yahoo, Motherboard, Le Monde (French)

  41. Fast, Cheap, and Good: Why Animated GIFs Engage Us
    Saeideh Bakhshi, David A. Shamma, Lyndon Kennedy, Yale Song, Paloma de Juan, Joseph 'Jofish' Kaye
    CHI 2016, [PDF] [Dataset] [Video]

  42. Balancing Appearance and Context in Sketch Interpretation
    Yale Song, Randall Davis, Kaichen Ma, Dana L. Penney
    IJCAI 2016, [arxiv]

  43. Mouse Activity as an Indicator of Interestingness in Video
    Gloria Zen, Paloma de Juan, Yale Song, Alejandro Jaimes
    ICMR 2016 (Long paper), [PDF] [Dataset]

    44. 2015

  44. TVSum: Summarizing Web Videos using Titles
    Yale Song, Jordi Vallmitjana, Amanda Stent, Alejandro Jaimes
    CVPR 2015, [PDF] [Poster] [TVSum50 Dataset]

  45. Video Co-summarization: Video Summarization by Visual Co-occurrence
    Wen-Sheng Chu, Yale Song, Alejandro Jaimes
    CVPR 2015, [PDF] [Poster] [Project]

  46. Continuous Body and Hand Gesture Recognition for Natural Human-Computer Interaction
    Yale Song, Randall Davis
    IJCAI 2015 Journal Track, [PDF]

  47. Exploiting Sparsity and Co-occurrence Structure for Action Unit Recognition
    Yale Song*, Daniel McDuff*, Deepak Vasisht, Ashish Kapoor (* equal contribution)
    FG 2015, [PDF] [Project] [Code]

    48. 2014

  48. #FluxFlow: Visual Analysis of Anomalous Information Spreading on Social Media
    Jian Zhao, Nan Cao, Zhen Wen, Yale Song, Yu-Ru Lin, Christopher Collins
    IEEE Trans. Visual. Comput. Graphics (VAST 2014), [PDF] [Video]
    Honorable Mention Award (3 out of 146 submissions)

    49. 2013

  49. Action Recognition by Hierarchical Sequence Summarization
    Yale Song, Louis-Philippe Morency, Randall Davis
    CVPR 2013, [PDF] [Code]

  50. One-Class Conditional Random Fields for Sequential Anomaly Detection
    Yale Song, Zhen Wen, Ching-Yung Lin, Randall Davis
    IJCAI 2013, [PDF]

  51. Distribution-Sensitive Learning for Imbalanced Datasets
    Yale Song, Louis-Philippe Morency, Randall Davis
    FG 2013, [PDF]

  52. Learning a Sparse Codebook of Facial and Body Microexpressions for Emotion Recognition
    Yale Song, Louis-Philippe Morency, Randall Davis
    ICMI 2013, [PDF] [Slides]

    53. 2012

  53. Multi-View Latent Variable Discriminative Models for Action Recognition
    Yale Song, Louis-Philippe Morency, Randall Davis
    CVPR 2012, [PDF] [Project] [Code]

  54. Multimodal Human Behavior Analysis: Learning Correlation and Interaction Across Modalities
    Yale Song, Louis-Philippe Morency, Randall Davis
    ICMI 2012, [PDF] [Slides]

  55. Continuous Body and Hand Gesture Recognition for Natural Human-Computer Interaction
    Yale Song, David Demirdjian, Randall Davis
    ACM Trans. Interact. Intell. Syst. 2(1), 2012, [PDF]
    Press coverage: MIT News, Economist, The Verge, CNET, Gizmodo, DailyBRINK

    56. 2011

  56. Tracking Body and Hands For Gesture Recognition: NATOPS Aircraft Handling Signals Database
    Yale Song, David Demirdjian, Randall Davis
    FG 2011, [PDF] [Dataset]

  57. Multi-Signal Gesture Recognition Using Temporal Smoothing Hidden Conditional Random Fields
    Yale Song, David Demirdjian, Randall Davis
    FG 2011, [PDF]

Theses

  1. Structured Video Content Analysis: Learning Spatio-Temporal and Multimodal Structures
    Yale Song
    PhD Thesis, Massachusetts Institute of Technology, 2014 [DSpace@MIT]

  2. Multi-Signal Gesture Recognition using Body and Hand Poses
    Yale Song
    SM Thesis, Massachusetts Institute of Technology, 2010 [DSpace@MIT]

Talks (selected)

  1. Life in Industrial Research Labs
    NeurIPS 2021 Workshop on New in ML, Dec 2021 [Video]

  2. Towards Self-Supervised Holistic Video Representations
    ICCV Tutorial on Holistic Video Understanding, Oct 2021 [Video] [Slides]

  3. An Introduction to Learning from Unlabeled Video
    ICCV Tutorial on Efficient Video Understanding, Oct 2021 [Video] [Slides]

  4. Learning from Unlabeled Video
    UBC Topics in Artificial Intelligence (guest lecture). Apr 2021
    5. SNU Data Science Seminar. Dec 2020 [Video] [Slides]

Interns / Students

Etc.

PageRank