Jaemin Cho

Jaemin Cho

PhD student @ UNC Chapel Hill

🚨 I’m now on the 2024-2025 job market and open to both academic and industry positions. Feel free to reach out if you’d like to discuss opportunities

I am a PhD student at UNC Chapel Hill (MURGe-Lab, UNC NLP group), advised by Prof. Mohit Bansal. I am supported by Bloomberg Data Science Ph.D. Fellowship.

While at UNC, I have spent my summer time at Bloomberg (in 2024), Google Research (in 2023), Microsoft Research (in 2022) and Adobe Research (in 2021). Prior to UNC, I did machine learning research at AI2, Naver, and SNU.

I am interested in building reliable machine learning systems that understand various types of real-world data. This includes but not limited to:

News

Education

  • Ph.D. in Computer Science, 2025 (expected)

    University of North Carolina at Chapel Hill

  • M.S. in Computer Science, 2022

    University of North Carolina at Chapel Hill

  • B.S. in Industrial Engineering, 2018

    Seoul National University

All Publications

Quickly search over publications. Also see my Google Scholar for a more complete list.
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
Multi-modal RAG framework for multi-page multi-document understanding
Jaemin Cho, Debanjan Mahata, Ozan Ä°rsoy, Yujie He, Mohit Bansal
Preprint Cite Project
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
SELMA improves T2I models by fine-tuning on automatically generated multi-skill image-text datasets, with skill-specific LoRA expert learning & merging. - NeurIPS 2024
Jialu Li, Jaemin Cho, Yi-Lin Sung, Jaehong Yoon, Mohit Bansal
Preprint Cite Code Project
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
A new testbed of teacher environments for data generation agents for diverse tasks.
Zaid Khan, Elias Stengel-Eskin, Jaemin Cho, Mohit Bansal
Preprint Cite Code Project
DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
Using LLM (GPT-4) to generate a ‘diagram plan’ for fine-grained layouts (object/text labels/arrows, etc.) and render in either raster images (via diffusion) and vector graphics (via PowerPoint / Inkscape or any tools). - COLM 2024
Abhay Zala, Han Lin, Jaemin Cho, Mohit Bansal
Preprint Cite Code Project
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
EnvGen is a novel framework that uses LLMs to adaptively create training environments to help smaller embodied RL agents learn useful skills that they are weak at. - COLM 2024
Abhay Zala, Jaemin Cho, Han Lin, Jaehong Yoon, Mohit Bansal
Preprint Cite Code Project
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
Using LLM (GPT-4) to generate a ‘video plan’ for consistent multi-scene video generation - COLM 2024
Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal
Preprint Cite Code Project
VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
An Assessment of Reported Biases and Harms of Large Language Models
Analysis of how biases and harms are reported and understood in recent LLM papers - ICA 2024 (Top Paper Award)
Heesoo Jang, Jaemin Cho
Cite
DOCCI: Descriptions of Connected and Contrasting Images
High-quality, long, human-annotated descriptions of 15K images - ECCV 2024
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, Jason Baldridge
Preprint Cite Dataset Project
DOCCI: Descriptions of Connected and Contrasting Images
Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
A plug-and-play framework that reuses any existing ControlNet for any video/image diffusion model
Han Lin, Jaemin Cho, Abhay Zala, Mohit Bansal
Preprint Cite Code Project
Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
a New Diagnostic Benchmark (LayoutBench) and a new Baseline model (IterInpaint) for Layout-Guided Image Generation - CVPR Workshop 2024 (Oral)
Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, Mohit Bansal
Preprint Cite Code Dataset Project
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts
Combining the best of both of specialist (e.g., SimpleClick) and generalist (e.g., SAM) designs to achieve all of low latency, high quality, and diverse prompts - CVPR 2024
Qin Liu, Jaemin Cho, Mohit Bansal, Marc Niethammer
Preprint Cite Code Project
Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
CRG is a training-free method that guides VLMs to help understand the visual prompts, by contrasting the outputs with & without visual prompts. - ECCV 2024
David Wan, Jaemin Cho, Elias Stengel-Eskin, Mohit Bansal
Preprint Cite Code Project
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation
Reliable QG/A framework for T2I Evaluation based on Davidsonian Semantics - ICLR 2024
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, Su Wang
Preprint Cite Code Project
Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation
Visual Programming for Text-to-Image Generation and Evaluation
Interpretable/explainable visual programming frameworks for T2I generation (VPGen) and evaluation (VPEval) - NeurIPS 2023
Jaemin Cho, Abhay Zala, Mohit Bansal
Preprint Cite Code Project
Visual Programming for Text-to-Image Generation and Evaluation
Self-Chained Image-Language Model for Video Localization and Question Answering
To handle video QA, we self-chain BLIP-2 for 2-stage inference (localize+QA) & refining localization via QA feedback - NeurIPS 2023
Shoubin Yu, Jaemin Cho, Prateek Yadav, Mohit Bansal
Preprint Cite Code
Self-Chained Image-Language Model for Video Localization and Question Answering
Paxion: Patching Action Knowledge in Video-Language Foundation Models
Analyzing and patching action knowledge in video-language models - NeurIPS 2023 (Spotlight)
Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, Heng Ji
Preprint Cite Code
Paxion: Patching Action Knowledge in Video-Language Foundation Models
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
Evaluation of Text-to-Image Generation Models in Reasoning Skills and Social Biases - ICCV 2023
Jaemin Cho, Abhay Zala, Mohit Bansal
Preprint Cite Code
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
Hierarchical Video-Moment Retrieval and Step-Captioning
HiREST is a holistic, hierarchical benchmark of multimodal retrieval and step-by-step summarization for a video corpus - CVPR 2023
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas OÄŸuz, Yasahar Mehdad, Mohit Bansal
Preprint Cite Code Project
Hierarchical Video-Moment Retrieval and Step-Captioning
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
Efficient VL modeling with Perceiver-based iterative cross-attentions - WACV 2023
Zineng Tang, Jaemin Cho, Jie Lei, Mohit Bansal
Preprint Cite Code
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
TVLT: Textless Vision-Language Transformer
Vision-and-Language modeling without text, by using a transformer which takes only raw visual and audio inputs - NeurIPS 2022 (Oral)
Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal
Preprint Cite Code
TVLT: Textless Vision-Language Transformer
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
LST brings Memory efficiency into Parameter-efficient transfer learning - NeurIPS 2022
Yi-Lin Sung, Jaemin Cho, Mohit Bansal
Preprint Cite Code
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Fine-grained Image Captioning with CLIP Reward
CLIP as reward function for fine-grained image captioning - Findings of NAACL 2022
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal
Preprint Cite Code
Fine-grained Image Captioning with CLIP Reward
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Adapter-based Parameter-Efficient Training for V&L tasks - CVPR 2022
Yi-Lin Sung, Jaemin Cho, Mohit Bansal
Preprint Cite Code
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding
A question answering benchmark on real-world news articles for multi-media and multi-hop reasoning - AAAI 2022
Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schiwing, Heng Ji
Preprint Cite
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
Video-based grounding can improve diverse NLU tasks - NeurIPS 2021
Zineng Tang, Jaemin Cho, Hao Tan, Mohit Bansal
Preprint Cite Code
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
Unifying Vision-and-Language Tasks via Text Generation
Tackle different V&L tasks via text generation with a single unified architecture - ICML 2021
Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal
Preprint Cite Code
Unifying Vision-and-Language Tasks via Text Generation
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
Text-to-Image Generation via predicting vector-quantized image patches with multimodal LMs - EMNLP 2020
Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi
Preprint Cite Code Project
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
Mixture Content Selection for Diverse Sequence Generation
Separate Diversification from Generation to improve both diversity and accuracy in sequence generation - EMNLP 2019
Jaemin Cho, Minjoon Seo, Hannaneh Hajishirzi
Preprint Cite Code
Mixture Content Selection for Diverse Sequence Generation
A Hierarchical Latent Structure for Variational Conversation Modeling
Propose a hierarchical VAE model and utterance drop regularization to mitigate posterior collapse problem - NAACL 2018 (Oral)
Yookoon Park, Jaemin Cho, Gunhee Kim
Preprint Cite Code Video
A Hierarchical Latent Structure for Variational Conversation Modeling