Publications

2024


Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models

Preprint, 2024

Towards Bidirectional Human-AI Alignment: A Systematic Review for Clarifications, Framework, and Future Directions

Preprint, 2024

2023


Partition-Based Active Learning for Graph Neural Networks

Transactions on Machine Learning Research (TMLR), 2023

2022


Spoken Language Interaction with Robots: Recommendations for Future Research

Computer Speech & Language, 2022

2021


CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models

iScience Cell Press Journal, 2021

Zero-Shot Compositional Concept Learning

ACL workshop on MetaNLP (MetaNLP @ ACL), 2021

Are We There Yet? Learning to Localize in Embodied Instruction Following

AAAI Workshop on Hybrid Artificial Intelligence (HAI @ AAAI), 2021

2020


Experience Grounds Language

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020

2019


X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust

Arxiv, 2019

Teaching Robots New Tasks through Natural Interaction

Teaching Robots New Tasks through Natural Interaction, 2019

Report of 2017 NSF Workshop on Multimedia Challenges, Opportunities and Research Roadmaps

NSF Workshop on Multimedia Challenges, Opportunities and Research Roadmaps, 2019

Natural Language Interaction with Explainable AI Models

Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019

Explainable AI as Collaborative Task Solving

Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019

2018


What Action Causes This? Towards Naive Physical Action-Effect Prediction

Annual Meeting of the Association for Computational Linguistics (ACL), 2018

Language to Action: Towards Interactive Task Learning with Physical Agents (Keynote Presentation)

Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018

Language to Action: Towards Interactive Task Learning with Physical Agents

International Joint Conferences on Artificial Intelligence (IJCAI), 2018

Commonsense Justification for Action Explanation

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018

2017


Interactive Learning of Grounded Verb Semantics towards Human-Robot Communication

Annual Meeting of the Association for Computational Linguistics (ACL), 2017

Detecting Clinically Related Content in Online Patient Posts

Journal of Biomedical Informatics (JBI), 2017

2016


What's Hot in Human Language Technology: Highlights from NAACL HLT 2015

AAAI Conference on Artificial Intelligence (AAAI), 2016

Task Learning through Visual Demonstration and Situated Dialogue

AAAI Workshop on Symbiotic Cognitive Systems, 2016

Program Robots Manufacturing Tasks by Natural Language Instructions

IEEE International Conference on Automation Science and Engineering (CASE), 2016

Physical Causality of Action Verbs in Grounded Language Understanding

Annual Meeting of the Association for Computational Linguistics (ACL), 2016

Jointly Learning Grounded Task Structures from Language Instruction and Visual Demonstration

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016

Incremental Acquisition of Verb Hypothesis Space towards Physical World Interaction

Annual Meeting of the Association for Computational Linguistics (ACL), 2016

Grounded Semantic Role Labeling

North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), 2016

2015


Question Types in Online Health Communities

American Medical Informatics Association Annual Symposium (AMIA), 2015

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

AAAI Conference on Artificial Intelligence (AAAI), 2015

Exception Handling for Natural Language Control of Robots

ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2015

Embodied Collaborative Referring Expression Generation in Situated Human-Robot Interaction

ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2015

2014


Teaching Robots New Actions through Natural Language Instructions

IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2014

Proceedings of the 19th International Conference on Intelligent User Interfaces

International Conference on Intelligent User Interfaces (IUI), 2014

Probabilistic Labeling for Efficient Referential Grounding based on Collaborative Discourse

Annual Meeting of the Association for Computational Linguistics (ACL), 2014

Perceptive feedback for natural language control of robotic operations

International Conference on Robotics and Automation (ICRA), 2014

Context-based Word Acquisition for Situated Dialogue in a Virtual World

Journal of Artificial Intelligence Research (JAIR), 2014

Collaborative Models for Referring Expression Generation in Situated Dialogue

AAAI Conference on Artificial Intelligence (AAAI), 2014

Collaborative effort towards common ground in situated human-robot dialogue

ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2014

Back to the Blocks World: Learning New Actions through Situated Human-Robot Dialogue

Special Interest Group on Discourse and Dialogue (SIGDIAL), 2014

2013


Towards Situated Dialogue: Revisiting Referring Expression Generation

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013

Shared Gaze in Situated Referential Grounding: An Empirical Study

Eye Gaze in Intelligent User Interfaces, 2013

Modeling Collaborative Referring for Situated Referential Grounding

Special Interest Group on Discourse and Dialogue (SIGDIAL), 2013

Introduction to the special section on eye gaze and conversation

ACM Transactions on Interactive Intelligent Systems (TiiS), 2013

2012


Towards online adaptation and personalization of key-target resizing for mobile devices

International Conference on Intelligent User Interfaces (IUI), 2012

Towards Mediating Shared Perceptual Basis in Situated Dialogue

Special Interest Group on Discourse and Dialogue (SIGDIAL), Best Paper Nominee, 2012

Introduction to the Special Issue on Eye Gaze in Intelligent Human-machine Interaction

ACM Transactions on Interactive Intelligent Systems (TiiS), 2012

Integrating word acquisition and referential grounding towards physical world interaction

*International Conference on Multimodal Interaction (ICMI), 2012

Autonomous Self-Assessment of Autocorrections: Exploring Text Message Dialogues

North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), 2012

2011


Cognitive Principles in Robust Multimodal Interpretation

Journal of Artificial Intelligence Research (JAIR), 2011

Beyond Normalization: Pragmatics of Word Form in Text Messages

International Joint Conference on Natural Language Processing (IJCNLP), 2011

A Joint Model of Implicit Arguments for Nominal Predicates

ACL Workshop on Relational Models of Semantics, 2011

2010


Workshop: Eye Gaze in Intelligent Human Machine Interaction

International Conference on Intelligent User Interfaces (IUI), 2010

Towards Conversation Entailment: An Empirical Investigation

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2010

Hand Gestures in Disambiguating Types of You Expressions in Multiparty Meetings

Special Interest Group on Discourse and Dialogue (SIGDIAL), 2010

Fusing Eye Gaze with Speech Recognition Hypotheses to Resolve Exophoric References in Situated Dialogue

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2010

Context-based Word Acquisition for Situated Dialogue in a Virtual World

Journal of Artificial Intelligence Research (JAIR), 2010

Beyond NomBank: A Study of Implicit Arguments for Nominal Predicates

Annual Meeting of the Association for Computational Linguistics (ACL), Best Long Paper Award, 2010

Ambiguities in Spatial Language Understanding in Situated Human Robot Dialogue

2010 AAAI Fall Symposium Series, 2010

2009


What do We Know about Conversation Participants: Experiments on Conversation Entailment

Special Interest Group on Discourse and Dialogue (SIGDIAL), 2009

The Role of Interactivity in Human-Machine Conversation for Automatic Word Acquisition

Special Interest Group on Discourse and Dialogue (SIGDIAL), 2009

The Role of Implicit Argumentation in Nominal SRL

North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), 2009

Communicative gestures in coreference identification in multiparty meetings

International Conference on Multimodal Interfaces (ICMI), 2009

Between linguistic attention and gaze fixations inmultimodal conversational interfaces

International Conference on Multimodal Interfaces (ICMI), 2009

2008


What's in a gaze?: the role of eye-gaze in reference resolution in multimodal conversational interfaces

International Conference on Intelligent User Interfaces (IUI), 2008

Incorporating Temporal and Semantic Information with Eye Gaze for Automatic Word Acquisition in Multimodal Conversational Systems

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2008

Beyond attention: the role of deictic gesture in intention recognition in multimodal conversational interfaces

International Conference on Intelligent User Interfaces (IUI), 2008

2007


Michigan State University at the 2007 TREC ciQA Task

Text REtrieval Conference (TREC), 2007

Eye Gaze for Attention Prediction in Multimodal Human-Machine Conversation

2007 AAAI Spring Symposium: Interaction Challenges for Intelligent Assistants, 2007

Automated Vocabulary Acquisition and Interpretation in Multimodal Conversational Systems

Annual Meeting of the Association for Computational Linguistics (ACL), 2007

An Exploration of Eye Gaze in Spoken Language Processing for Multimodal Conversational Interfaces

North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), 2007

An Empirical Investigation of User Term Feedback in Text-based Targeted Image Search

ACM Transactions on Interactive Intelligent Systems (TiiS), 2007

2006


Towards intelligent QA interfaces: discourse processing for context questions

International Conference on Intelligent User Interfaces (IUI), 2006

Towards Conversational QA: Automatic Identification of Problematic Situations and User Intent

Annual Meeting of the Association for Computational Linguistics (ACL), 2006

Salience modeling based on non-verbal modalities for spoken language understanding

International Conference on Multimodal Interfaces (ICMI), 2006

Cognitive Principles in Robust Multimodal Interpretation

Journal of Artificial Intelligence Research (JAIR), 2006

Automated performance assessment in interactive QA

International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2006

A Statistical Framework for Query Translation Disambiguation

ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 2006

2005


User term feedback in interactive text-based image retrieval

International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2005

Study of cross lingual information retrieval using on-line translation systems

International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2005

Linguistic theories in efficient multimodal reference resolution: an empirical investigation

International Conference on Intelligent User Interfaces (IUI), 2005

Learn to weight terms in information retrieval using category information

International Conference on Machine Learning (ICML), 2005

A Salience Driven Approach to Robust Input Interpretation in Multimodal Conversational Systems

Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT-EMNLP), 2005

A maximum coherence model for dictionary-based cross-language information retrieval

International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2005

2004


Regularizing translation models for better automatic image annotation

Proceedings of the thirteenth ACM International Conference on Information and Knowledge Management (CIKM), 2004

Performance Evaluation and Error Analysis for Multimodal Reference Resolution in a Conversation System

North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), 2004

Optimization in Multimodal Interpretation

Annual Meeting of the Association for Computational Linguistics (ACL), 2004

MSU at ImageCLEF: Cross Language and Interactive Image Retrieval

The Cross-Language Evaluation Forum (CLEF) Workshop on Multilingual Information Access for Text, Speech and Images, 2004

An automatic weighting scheme for collaborative filtering

International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2004

A probabilistic approach to reference resolution in multimodal user interfaces

International Conference on Intelligent User Interfaces (IUI), 2004