Zirui "Colin" Wang

I am a second-year master's student in Computer Science at Princeton University. My research is currently advised by Prof. Danqi Chen, and I serve as a Teaching Assistant in the department. I am generally interested in enhancing large language models with multimodal capabilities.

Previously, I obtained B.S. in Data Science at the Halicioglu Data Science Institute (HDSI) and B.A. in Cognitive Science at the CogSci Department at the University of California, San Diego (UCSD). I was advised by Prof. Zhuowen Tu and Prof. Zhiting Hu for generative models in computer vision during my undergraduate years.

Email  /  Resume  /  CV  /  GitHub  /  Twitter  /  Google Scholar  /  LinkedIn

Updates

• 11/04/2024: Awarded as a Top Reviewer in NeurIPS 2024"
• 10/02/2024: Invited Talk at University of Michigan, Ann Arbor on "Evaluations of Multimodal Large Language Models"
• 09/26/2024: CharXiv accepted into NeurIPS 2024 (D&B Track) & ECCV FoMo-Eval Workshop
• 09/20/2024: Nominated for Siebel Scholar 2025
• 08/05/2024: Invited Talk at Google on "CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs"

Schedule

My calendar. Note that all the schedulings are tentative and subjective to change

Research

CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

CharXiv reveals significant shortcomings in MLLMs’ chart understanding, showing a large performance gap between models and humans.

Improving Language Understanding from Screenshots

We close the performance gap between screenshot Language Models and text-only Language Models on language understanding tasks with our PTP objective.

Language Models as Science Tutors

We propose TutorChat and TutorEval, a dataset of long synthetic dialogues about textbooks and a question-ansering benchmark consisting questions about long chapters from STEM textbooks written by human experts.

TokenCompose: Grounding Diffusion with Token-level Supervision

We introduce token-wise consistency terms between the image content and object segmentation maps in training text-to-image models for enhanced multi-category instance composition and photorealism.

OmniControlNet: Dual-stage Integration for Conditional Image Generation

We provide a two-way integration for the widely-adopted ControlNet method by integrating four external condition generation algorithms into a single dense image labeling method, and by integrating its individually trained image generation processes into a single model.

Language Models Meet World Models: Embodied Experiences Enhance Language Models

We establish a framework that effectively and efficiently finetunes a language model with embodied experience while retaining its language modeling abilities.

On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning

We investigate whether internal models learned by modern model-based RL algorithms can be leveraged to solve new, distinctly different tasks faster.

Services

• ICML: Reviewer (2024)
• ICLR: Reviewer (2023, 2024)
• NeurIPS: Reviewer (2024)

Teaching

Full lists and details about classes I have served as a teaching assistant in the past. Instructor names are listed based on the time I worked with them. Staff names are listed in alphebatical order based on their first names. Instructor names and staff names are separated by a semicolon. Instructor evaluation is attached if available.

NLP

An introductory class about Natural Language Processing. Topics covered include language modeling, representation learning, text classification, sequence tagging, machine translation, Transformers, and others.

Human-Computer Interaction

Project-based class where students are introduced with basics of HCI and either conduct a study-based (e.g., survey + interviews) project or a system-based (e.g., implementations) project. I lead HCI + AI domain.

Deep Learning

This course covers the fundamentals of neural networks. We introduce linear regression, logistic regression, perceptrons, multilayer networks and back-propagation, convolutional neural networks, recurrent networks, and deep networks trained by reinforcement learning.

Introduction to Machine Learning

The topics include some topics in supervised learning, such as k-nearest neighbor classifiers, decision trees, boosting, and perceptrons; and topics in unsupervised learning, such as k-means and hierarchical clustering. In addition to the actual algorithms, the course focuses on the principles behind the algorithms.

Practical Data Science in R

Learn coding for data analysis using the R programming language. Course focus will be on practical and applied skills in asking data-informed questions, data wrangling, data visualization, building statistical learning models, and communication.

The Practice and Application of Data Science

Students master the data science life-cycle and learn many of the fundamental principles and techniques of data science spanning algorithms, statistics, machine learning, visualization, and data systems.

Data Structures and Algorithms for Data Science (x4)

Programming techniques including encapsulation, abstract data types, interfaces, algorithms and complexity, and data structures such as stacks, queues, priority queues, heaps, linked lists, binary trees, binary search trees, and hash tables with Java.

Programming and Basic Data Structures for Data Science (x2)

Programming techniques including recursion, higher-order functions, function composition, object-oriented programming, interpreters, classes, and simple data structures such as arrays, lists, and linked lists.

Principles of Data Science

This introductory course develops computational thinking and tools necessary to answer questions that arise from large-scale datasets, emphasizing an end-to-end approach to data science, introducing programming techniques in Jupyter Notebook that cover data processing, modeling, and analysis.

Other Projects

These include courseworks, projects and other research-related work not for publishing. contents to be updated (10/11/2022). To remind myself what to put: DSC180 Capstone, DSC190 DataMining, COGS 108, MATH 189, DSC 102, DSC 106, COGS 189, DataHacks Adv, DataHacks Bus, Tencent, SS (CMU)

On the Domain Robustness with Prompt & Prefix Tuning

We analyze the robustness of Language Models using prompt tuning and prefix tuning toward a domain shift (i.e. learning a task from data in a specific domain and evaluating that model on the same task, but data is out-of-domain).

EEG Transformer

We implement a naive EEG transformer that explores the possibility of using a ViT based transformer for inferring 3-class motor imagery based on multichannel time-series EEG data recorded at 1000 Hz for 8 seconds (in which 4 seconds are used). We propose future directions.

Design and source code from Jon Barron's website. Forked from the Jekyll varient by Leonid Keselman.