1

SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
SELMA improves T2I models by fine-tuning on automatically generated multi-skill image-text datasets, with skill-specific LoRA expert learning & merging. - NeurIPS 2024
Jialu Li, Jaemin Cho, Yi-Lin Sung, Jaehong Yoon, Mohit Bansal
Preprint Cite Code Project
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
Using LLM (GPT-4) to generate a ‘diagram plan’ for fine-grained layouts (object/text labels/arrows, etc.) and render in either raster images (via diffusion) and vector graphics (via PowerPoint / Inkscape or any tools). - COLM 2024
Abhay Zala, Han Lin, Jaemin Cho, Mohit Bansal
Preprint Cite Code Project
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
EnvGen is a novel framework that uses LLMs to adaptively create training environments to help smaller embodied RL agents learn useful skills that they are weak at. - COLM 2024
Abhay Zala, Jaemin Cho, Han Lin, Jaehong Yoon, Mohit Bansal
Preprint Cite Code Project
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
Using LLM (GPT-4) to generate a ‘video plan’ for consistent multi-scene video generation - COLM 2024
Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal
Preprint Cite Code Project
VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
An Assessment of Reported Biases and Harms of Large Language Models
Analysis of how biases and harms are reported and understood in recent LLM papers - ICA 2024 (Top Paper Award)
Heesoo Jang, Jaemin Cho
Cite
DOCCI: Descriptions of Connected and Contrasting Images
High-quality, long, human-annotated descriptions of 15K images - ECCV 2024
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, Jason Baldridge
Preprint Cite Dataset Project
DOCCI: Descriptions of Connected and Contrasting Images
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
a New Diagnostic Benchmark (LayoutBench) and a new Baseline model (IterInpaint) for Layout-Guided Image Generation - CVPR Workshop 2024 (Oral)
Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, Mohit Bansal
Preprint Cite Code Dataset Project
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts
Combining the best of both of specialist (e.g., SimpleClick) and generalist (e.g., SAM) designs to achieve all of low latency, high quality, and diverse prompts - CVPR 2024
Qin Liu, Jaemin Cho, Mohit Bansal, Marc Niethammer
Preprint Cite Code Project
Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
CRG is a training-free method that guides VLMs to help understand the visual prompts, by contrasting the outputs with & without visual prompts. - ECCV 2024
David Wan, Jaemin Cho, Elias Stengel-Eskin, Mohit Bansal
Preprint Cite Code Project
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation
Reliable QG/A framework for T2I Evaluation based on Davidsonian Semantics - ICLR 2024
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, Su Wang
Preprint Cite Code Project
Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation