Jaemin Cho

Jaemin Cho

PhD student @ UNC Chapel Hill

I am a PhD student at UNC Chapel Hill (MURGe-Lab, UNC NLP group), advised by Prof. Mohit Bansal.

While at UNC, I spent my summer time at Google Research (in 2023), Microsoft Research (in 2022) and Adobe Research (in 2021). Prior to UNC, I did machine learning research at AI2, Naver, and SNU.

I am interested in building reliable machine learning systems that understand various types of real-world data, such as language, vision, music and dance.

News

Interests
  • Machine Learning
  • Multimodal Learning
  • Natural Language Processing
  • Generative Model
Education

  • M.S. in Computer Science, 2022

    University of North Carolina at Chapel Hill

  • B.S. in Industrial Engineering, 2018

    Seoul National University

Publications

Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-Image Generation
Reliable and Fine-grained QG/A framework for T2I Evaluation based on Davisdonian Semantics
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, Su Wang
Preprint Cite Code Project
Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-Image Generation
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
Using LLM (GPT-4) to generate a ‘diagram plan’ for fine-grained layouts (object/text labels/arrows, etc.) and render in either raster images (via diffusion) and vector graphics (via PowerPoint / Inkscape or any tools).
Abhay Zala, Han Lin, Jaemin Cho, Mohit Bansal
Preprint Cite Code Project
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
Using LLM (GPT-4) to generate a ‘video plan’ for consistent multi-scene video generation
Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal
Preprint Cite Code Project
VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
Visual Programming for Text-to-Image Generation and Evaluation
Interpretable/explainable visual programming frameworks for T2I generation (VPGen) and evaluation (VPEval) - NeurIPS 2023
Jaemin Cho, Abhay Zala, Mohit Bansal
Preprint Cite Code Project
Visual Programming for Text-to-Image Generation and Evaluation
Self-Chained Image-Language Model for Video Localization and Question Answering
To handle video QA, we self-chain BLIP-2 for 2-stage inference (localize+QA) & refining localization via QA feedback - NeurIPS 2023
Shoubin Yu, Jaemin Cho, Prateek Yadav, Mohit Bansal
Preprint Cite Code
Self-Chained Image-Language Model for Video Localization and Question Answering
Paxion: Patching Action Knowledge in Video-Language Foundation Models
Analyzing and patching action knowledge in video-language models - NeurIPS 2023 (Spotlight)
Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, Heng Ji
Preprint Cite Code
Paxion: Patching Action Knowledge in Video-Language Foundation Models
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
Evaluation of Text-to-Image Generation Models in Reasoning Skills and Social Biases - ICCV 2023
Jaemin Cho, Abhay Zala, Mohit Bansal
Preprint Cite Code
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
a New Diagnostic Benchmark (LayoutBench) and a new Baseline model (IterInpaint) for Layout-Guided Image Generation
Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, Mohit Bansal
Preprint Cite Code Dataset Project
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Hierarchical Video-Moment Retrieval and Step-Captioning
HiREST is a holistic, hierarchical benchmark of multimodal retrieval and step-by-step summarization for a video corpus - CVPR 2023
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oğuz, Yasahar Mehdad, Mohit Bansal
Preprint Cite Code Project
Hierarchical Video-Moment Retrieval and Step-Captioning
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
Efficient VL modeling with Perceiver-based iterative cross-attentions - WACV 2023
Zineng Tang, Jaemin Cho, Jie Lei, Mohit Bansal
Preprint Cite Code
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
TVLT: Textless Vision-Language Transformer
Vision-and-Language modeling without text, by using a transformer which takes only raw visual and audio inputs - NeurIPS 2022 (Oral)
Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal
Preprint Cite Code
TVLT: Textless Vision-Language Transformer
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
LST brings Memory efficiency into Parameter-efficient transfer learning - NeurIPS 2022
Yi-Lin Sung, Jaemin Cho, Mohit Bansal
Preprint Cite Code
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Fine-grained Image Captioning with CLIP Reward
CLIP as reward function for fine-grained image captioning - Findings of NAACL 2022
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal
Preprint Cite Code
Fine-grained Image Captioning with CLIP Reward
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Adapter-based Parameter-Efficient Training for V&L tasks - CVPR 2022
Yi-Lin Sung, Jaemin Cho, Mohit Bansal
Preprint Cite Code
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding
A question answering benchmark on real-world news articles for multi-media and multi-hop reasoning - AAAI 2022
Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schiwing, Heng Ji
Preprint Cite
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
Video-based grounding can improve diverse NLU tasks - NeurIPS 2021
Zineng Tang, Jaemin Cho, Hao Tan, Mohit Bansal
Preprint Cite Code
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
Unifying Vision-and-Language Tasks via Text Generation
Tackle different V&L tasks via text generation with a single unified architecture - ICML 2021
Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal
Preprint Cite Code
Unifying Vision-and-Language Tasks via Text Generation
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
Generate image from text by predicting masked patches with multi-modal transformers - EMNLP 2020
Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi
Preprint Cite Code Project
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
Mixture Content Selection for Diverse Sequence Generation
Separate Diversification from Generation to improve both diversity and accuracy in sequence generation - EMNLP 2019
Jaemin Cho, Minjoon Seo, Hannaneh Hajishirzi
Preprint Cite Code
Mixture Content Selection for Diverse Sequence Generation
A Hierarchical Latent Structure for Variational Conversation Modeling
Propose a hierarchical VAE model and utterance drop regularization to mitigate posterior collapse problem - NAACL 2018 (Oral)
Yookoon Park, Jaemin Cho, Gunhee Kim
Preprint Cite Code Video
A Hierarchical Latent Structure for Variational Conversation Modeling