Xingjian Diao

About Me



I am currently a Ph.D. student at Dartmouth College. My research interests focus on multimodal learning. I have published papers and developed codes for temporal modeling, efficient training, and audio-video-language integration, advancing state-of-the-art multimodal large language models (MLLMs) in audio-visual question answering, video captioning, and text-to-talking face generation.


Prior to Dartmouth, I earned a Master's degree in Computer Science from Northwestern University (2021), advised by Prof. Nabil Alshurafa (Thank you, Nabil!), and a B.S. degree in Computer Science from the University of Pittsburgh (2020).



Interests

  • Multimodal Large Language Models
  • Video Understanding
  • Natural Language and Speech Processing

Education

  • Ph.D. in Computer Science, -Present
    Dartmouth College
  • M.S. in Computer Science, 2021
    Northwestern University
  • B.S. in Computer Science, 2020
    University of Pittsburgh
Publications

* indicates equal contribution

Temporal Working Memory: Query-Guided Temporal Segment Refinement for Enhanced Multimodal Understanding
Xingjian Diao*, Chunhui Zhang*, Weiyi Wu, Zhongyu Ouyang, Peijun Qing, Ming Cheng, Soroush Vosoughi, Jiang Gui
Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2025) Findings
FT2TF: First-Person Statement Text-To-Talking Face Generation
Xingjian Diao, Ming Cheng, Wayner Barrios, SouYoung Jin
Winter Conference on Applications of Computer Vision (WACV) , 2025
Learning Musical Representations for Music Performance Question Answering
Xingjian Diao, Chunhui Zhang, Tingxuan Wu, Ming Cheng, Zhongyu Ouyang, Weiyi Wu, Jiang Gui
Conference on Empirical Methods in Natural Language Processing (EMNLP 2024) Findings
AlphaExpert: Assigning LoRA Experts Based on Layer Training Quality
Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, Soroush Vosoughi
Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

Analytical free vibration solutions of rectangular thin plates subjected to three edges rotationally-restrained and one edge free
Jinghui Zhang, Pin Gao, Xingjian Diao, Salamat Ullah, Jiapeng Li, Yuwei Zhang, Wenyue Qi
International Journal of Structural Stability and Dynamics (IJSSD), 2024
SAIC: Integration of Speech Anonymization and Identity Classification
Ming Cheng*, Xingjian Diao*, Shitong Cheng, Wenjun Liu
AI for Health Equity and Fairness: Leveraging AI to Address Social Determinants of Health, Cham: Springer Nature Switzerland, 2024
GluMarker: A Novel Predictive Modeling of Glycemic Control Through Digital Biomarkers
Ziyi Zhou*, Ming Cheng*, Xingjian Diao*, Yanjun Cui, Xiangling Li
Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2024
🎖 IEEE EMBC NextGen Scholar Award
Efflex: Efficient and Flexible Pipeline for Spatio-Temporal Trajectory Graph Modeling and Representation Learning
Ming Cheng*, Ziyi Zhou*, Bowen Zhang*, Ziyu Wang, Jiaqi Gan, Ziang Ren, Weiqi Feng, Yi Lyu, Hefan Zhang, Xingjian Diao
Conference on Computer Vision and Pattern Recognition (CVPR) SG2RL2024, 2024
AV-MaskEnhancer: Enhancing Video Representations through Audio-Visual Masked Autoencoder
Xingjian Diao*, Ming Cheng*, Shitong Cheng
International Conference on Tools with Artificial Intelligence (ICTAI), 2023
An End-to-End Energy-Efficient Approach for Intake Detection With Low Inference Time Using Wrist-Worn Sensor
Boyang Wei, Shibo Zhang, Xingjian Diao, Qiuyang Xu, Yang Gao, Nabil Alshurafa
IEEE Journal of Biomedical and Health Informatics (JBHI), 2023
Building a Cloud-based Energy Storage System through Digital Transformation of Distributed Backup Batteries in Mobile Base Stations
Song Ci, Yanglin Zhou, Yuan Xu, Xingjian Diao, Junwei Wang
China Communications, 2020
Projects
Selected Projects
Intake Detection Tool with Multiple Classifiers

An Android application for wrist-worn devices to detect feeding patterns with low energy consumption and fast inference times. It applied template-based multi-centroid classifier which could provide an end-to-end battery-efficient approach for feeding detection.

Interactive Active Learning Annotation Tool

An interactive annotation software that utilizes active learning to reduce data labeling time and cost. The front-end was created with PyQt5 and pyqtgraph, offering features such as time synchronization and video frame-by-frame rewinding. The back-end, utilizing cv2, sklearn and xgboost, performed data processing, K-means clustering, and clustered entropy active learning.

iPADshiny

iPADshiny (integrated Protein Array Data management,analysis and visualization tools) is a desktop application that simplifies protein analysis for biologists. It integrates multiple algorithms, including the auto-antibody Profiling Analysis, and utilizes state-of-the-art computational methods for efficient and effective analysis.

Online Drawing Management System

An Online Drawing Management System with B/S structure and Windows OS, including features such as notice announcement, navigation menu, user and role management, flexible authorization, and online management and preview of large drawing documents. It automatically loads existing document storage structures, eliminating the need for manual entry of basic information. (Copyright: 2018SR071476)

Remote Voting System

A remote voting system that uses SMS texts to count unique votes while recording phone numbers to prevent repetitive voting, offering an accessible and transparent solution for remote voting scenarios.

Introvert

An inclusive online chat environment for introverted students, utilizing JavaScript, Python, and Google Cloud platform to implement anonymous chatting and user-friendly direct messaging features, aimed at promoting engagement and improving the chat experience for introverted individuals.

Teaching

TA indicates Teaching Assistant

Services
Contact