Coherence Measurement and Verifiable Natural Language Understanding

Overview

Datasets

Please contact Shane Storks with any questions.

• Verifiable-Coherent-NLU

Getting Started

Please see the GitHub repo linked above for the latest information on how to reproduce results.

Contributed Results

Some of our students in EECS 595 (Natural Language Processing) worked on TRIP in fall 2021. With permission, we have shared their reported results here:

• Saurabh Budholiya and Isaac Fung. Common Sense Language Understanding - Improving Baseline for TRIP.
• Muhammad Khalifa and James Tavernor. Pretrained Commonsense Language Models and Memory Networks for TRIP.
• Le Qin. Evaluating XLNet on Trip Dataset.
• Ikhee Shin. Tiered Reasoning for Intuitive Physics: a Dialogue State Tracking approach.
• Wenfei Tang and Juejue Wang. Improving the Baseline Performance of the TRIP Model.
• Yicheng Tao and Leheng Lu. An Improved Baseline for TRIP with Attention and New Backbones.
• Ben VanDerPloeg. Dependency Graph Baseline for TRIP.
• Ziyan Wang and Jim Yang. An Approach to Boost the Baseline Performance on TRIP.
• Zhihao Xu and Zheyu Zhang. Improving Verifiability of TRIP with Data Augmentation and Graph Neural Networks.

If you refer to any of these unpublished results in your own work, please cite them appropriately.

Cite Us

If you use our code or models in your work, please cite one of our following papers from Findings of EMNLP 2021:

  @misc{storks2021tiered,
        title={Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding}, 
        author={Shane Storks and Qiaozi Gao and Yichi Zhang and Joyce Chai},
        year={2021},
        booktitle={Findings of the Association for Computational Linguistics: EMNLP 2021},
        location={Punta Cana, Dominican Republic},
        publisher={Association for Computational Linguistics},
  }

  @misc{storks2021tip,
        title={Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers}, 
        author={Shane Storks and Joyce Chai},
        year={2021},
        booktitle={Findings of the Association for Computational Linguistics: EMNLP 2021},
        location={Punta Cana, Dominican Republic},
        publisher={Association for Computational Linguistics},
  }

Additionally, please consider citing Conversational Entailment and ART, which are used in experiments from the latter paper (and included in this repo):

  @inproceedings{zhang-chai-2010-towards,
      title = "Towards Conversation Entailment: An Empirical Investigation",
      author = "Zhang, Chen  and
        Chai, Joyce",
      booktitle = "Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing",
      month = oct,
      year = "2010",
      address = "Cambridge, MA",
      publisher = "Association for Computational Linguistics",
      url = "https://aclanthology.org/D10-1074",
      pages = "756--766",
  }

  @inproceedings{
      bhagavatula2020abductive,
      title={Abductive Commonsense Reasoning},
      author={Chandra Bhagavatula and Ronan Le Bras and Chaitanya Malaviya and Keisuke Sakaguchi and Ari Holtzman and Hannah Rashkin and Doug Downey and Wen-tau Yih and Yejin Choi},
      booktitle={International Conference on Learning Representations},
      year={2020},
      url={https://openreview.net/forum?id=Byg1v1HKDB}
  }