Jaemin Cho

UNC NLP Group

I’m a PhD student at MURGe-Lab, UNC NLP group, advised by Prof. Mohit Bansal. Prior to UNC, I did machine learning research at AI2, Naver, and SNU.

I’m interested in building reliable machine learning systems that understand various types of real-world data, such as language, vision, music and dance.

News

May 2023 - New preprints:
Mar 2023 - 1 paper has been accepted to CVPR 2023:
- HiREST: Hierarchical Video-Moment Retrieval and Step-Captioning
Sep 2022 - 2 papers have been accepted to NeurIPS 2022:
- LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
- TVLT: Textless Vision-Language Transformer (Oral)
Aug 2022: 1 paper has been accepted to WACV 2023:
- Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
Apr 2022 - 1 paper has been accepted to Findings of NAACL 2022:
- Fine-grained Image Captioning with CLIP Reward <!– * Mar 2022 - 1 paper has been accepted to CVPR 2022:
- VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Feb 2022 - New preprint: DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers. –>

Interests

Machine Learning
Natural Language Processing
Multimodal Learning
Generative Model

Education

M.S. in Computer Science, 2022
University of North Carolina at Chapel Hill
B.S. in Industrial Engineering, 2018
Seoul National University

Publications

Visual Programming for Text-to-Image Generation and Evaluation

Interpretable/explainable visual programming frameworks for T2I generation (VPGen) and evaluation (VPEval)

Jaemin Cho, Abhay Zala, Mohit Bansal

Preprint Code Project

Paxion: Patching Action Knowledge in Video-Language Foundation Models

Dignosic and patching action knowledge in video-language models

Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, Heng Ji

Preprint Code

Self-Chained Image-Language Model for Video Localization and Question Answering

To handle video QA, we self-chain BLIP-2 for 2-stage inference (localize+QA) & refining localization via QA feedback

Shoubin Yu, Jaemin Cho, Prateek Yadav, Mohit Bansal

Preprint Code

Hierarchical Video-Moment Retrieval and Step-Captioning

HiREST is a holistic, hierarchical benchmark of multimodal retrieval and step-by-step summarization for a video corpus - CVPR 2023

Abhay Zala*, Jaemin Cho*, Satwik Kottur, Xilun Chen, Barlas Oğuz, Yasahar Mehdad, Mohit Bansal

Preprint Code Project

TVLT: Textless Vision-Language Transformer

Vision-and-Language modeling without text, by using a transformer which takes only raw visual and audio inputs - NeurIPS 2022 (Oral)

Zineng Tang*, Jaemin Cho*, Yixin Nie*, Mohit Bansal

Preprint Code

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

LST brings Memory efficiency into Parameter-efficient transfer learning - NeurIPS 2022

Yi-Lin Sung, Jaemin Cho, Mohit Bansal

Preprint Code

Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention

Efficient VL modeling with Perceiver-based iterative cross-attentions - WACV 2023

Zineng Tang*, Jaemin Cho*, Jie Lei, Mohit Bansal

Preprint Code

Fine-grained Image Captioning with CLIP Reward

CLIP as reward function for fine-grained image captioning - Findings of NAACL 2022

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal

Preprint Code

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Adapter-based Parameter-Efficient Training for V&L tasks - CVPR 2022

Yi-Lin Sung, Jaemin Cho, Mohit Bansal

Preprint Code

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

Probing the Reasoning Skills and Social Biases of Text-to-Image Models

Jaemin Cho, Abhay Zala, Mohit Bansal

Preprint Code

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

A question answering benchmark on real-world news articles for multi-media and multi-hop reasoning - AAAI 2022

Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schiwing, Heng Ji

Preprint

VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer

Video-based grounding can improve diverse NLU tasks - NeurIPS 2021

Zineng Tang, Jaemin Cho, Hao Tan, Mohit Bansal

Preprint Code

Unifying Vision-and-Language Tasks via Text Generation

Tackle different V&L tasks via text generation with a single unified architecture - ICML 2021

Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal

Preprint Code

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Generate image from text by predicting masked patches with multi-modal transformers - EMNLP 2020

Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi

Preprint Code Project

Mixture Content Selection for Diverse Sequence Generation

Separate Diversification from Generation to improve both diversity and accuracy in sequence generation - EMNLP 2019

Jaemin Cho, Minjoon Seo, Hannaneh Hajishirzi

Preprint Code

A Hierarchical Latent Structure for Variational Conversation Modeling

Propose a hierarchical VAE model and utterance drop regularization to mitigate posterior collapse problem - NAACL 2018 (Oral)

Yookoon Park, Jaemin Cho, Gunhee Kim

Preprint Code Video

Experience

Student Researcher

Google Research

May 2023 – Aug 2023 Austin, TX

Working on

Vision and Language

Research Intern

Microsoft Research

May 2022 – Aug 2022 Redmond, WA (Remote)

Publications

Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, Mohit Bansal, Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation.

Worked on

Vision and Language

Advisors

Research Intern

Adobe Research

May 2021 – Aug 2021 San Jose, CA (Remote)

Publications

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal, Fine-grained Image Captioning with CLIP Reward, Findings of NAACL 2022.

Worked on