publications

publications by categories in reversed chronological order.

Visit Google Scholar for a full list.

2024

Softmax probabilities (mostly) predict large language model correctness on multiple-choice Q&A

Benjamin Plaut, Khanh Nguyen, and Tu Trinh

ArXiv, 2024

💡 LLM probabilities are predictive of the correctness of their answers on Q&A tasks

@article{plaut2024softmax,
  author = {Plaut, Benjamin and Nguyen, Khanh and Trinh, Tu},
  title = {Softmax probabilities (mostly) predict large language model correctness on multiple-choice Q&A},
  journal = {ArXiv},
  month = {},
  year = {2024},
  oneline = {LLM probabilities are predictive of the correctness of their answers on Q&A tasks}
}

Language-guided world models: A model-based approach to AI control

Alex Zhang, Albert Lin, Jens Tuyls, Khanh Nguyen, and Karthik Narasimhan

ArXiv, 2024

💡 Models that can read texts to simulate environments

@article{zhang2024lwm,
  author = {Zhang, Alex and Lin, Albert and Tuyls, Jens and Nguyen, Khanh and Narasimhan, Karthik},
  title = {Language-guided world models: A model-based approach to AI control},
  journal = {ArXiv},
  month = {},
  year = {2024},
  oneline = {Models that can read texts to simulate environments}
}

2023

Progressively efficient learning

Ruijie Zheng, Khanh Nguyen, Furong Huang, Hal Daumé III, and Karthik Narasimhan

Workshop on Intrinsically Motivated Open-ended Learning (NeurIPS), 2023

💡 Learn increasingly abstract language to reduce communication effort

@inproceedings{zheng2023progressive,
  author = {Zheng, Ruijie and Nguyen, Khanh and Huang, Furong and Daum{\'e} III, Hal and Narasimhan, Karthik},
  title = {Progressively efficient learning},
  booktitle = {Workshop on Intrinsically Motivated Open-ended Learning
   (NeurIPS)},
  month = {},
  year = {2023},
}

Hallucination detection for instruction generation models

Lingjun Zhao, Khanh Nguyen, and Hal Daumé III

EMNLP Findings, 2023

💡 Detect hallucinations in model-generated navigation instructions

@inproceedings{zhao2023hallucination,
  author = {Zhao, Lingjun and Nguyen, Khanh and Daum{\'e} III, Hal},
  title = {Hallucination detection for instruction generation models},
  booktitle = {EMNLP Findings},
  month = {},
  year = {2023},
  oneline = {Detect hallucinations in model-generated navigation instructions}
}

Language models are bounded pragmatic speakers: Understanding RLHF from a Bayesian cognitive modeling perspective

Khanh Nguyen

Workshop on Theory of Mind (ICML), 2023

💡 LLMs as limited communicative agents and how to augment them

@inproceedings{nguyen2023bounded,
  author = {Nguyen, Khanh},
  title = {Language models are bounded pragmatic speakers: Understanding RLHF from a Bayesian cognitive modeling perspective},
  booktitle = {Workshop on Theory of Mind (ICML)},
  month = {},
  year = {2023},
  oneline = {LLMs as limited communicative agents and how to augment them}
}

Define, evaluate, and improve task-oriented cognitive capabilities for instruction generation models

Lingjun Zhao, Khanh Nguyen, and Hal Daumé III

ACL Findings, 2023

💡 Improve instruction generation by predicting human interpretation

@inproceedings{zhao2023cognitive,
  author = {Zhao, Lingjun and Nguyen, Khanh and Daum{\'e} III, Hal},
  title = {Define, evaluate, and improve task-oriented cognitive capabilities for instruction generation models},
  booktitle = {ACL Findings},
  month = {},
  year = {2023},
  oneline = {Improve instruction generation by predicting human interpretation}
}

2022

A framework for learning to request rich and contextually useful information from humans

Khanh Nguyen, Yonatan Bisk, and Hal Daumé III

ICML, Jul 2022

💡 Agents that can decide when and what question to ask, and incoporate answer to make progress

@inproceedings{nguyen2022hari,
  author = {Nguyen, Khanh and Bisk, Yonatan and Daum{\'e} III, Hal},
  title = {A framework for learning to request rich and contextually useful information from humans},
  booktitle = {ICML},
  month = jul,
  year = {2022},
  oneline = {Agents that can decide when and what question to ask, and incoporate answer to make progress}
}

2021

Interactive learning from activity description

Khanh Nguyen, Dipendra Misra, Robert Schapire, Miro Dudı́k, and Patrick Shafto

ICML, Jul 2021

💡 Framework for learning from language feedback with theoretical guarantees

@inproceedings{nguyen2021iliad,
  title = {Interactive learning from activity description},
  author = {Nguyen, Khanh and Misra, Dipendra and Schapire, Robert and Dud{\'\i}k, Miro and Shafto, Patrick},
  booktitle = {ICML},
  year = {2021},
  oneline = {Framework for learning from language feedback with theoretical guarantees}
}

2019

Help, Anna! Visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning

Khanh Nguyen, and Hal Daumé III

EMNLP, 2019

💡 Agents that can ask for help and interpret language instructions

@inproceedings{nguyen2019hanna,
  author = {Nguyen, Khanh and Daum{\'e} III, Hal},
  title = {Help, Anna! Visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning},
  booktitle = {EMNLP},
  month = {},
  year = {2019},
  oneline = {Agents that can ask for help and interpret language instructions}
}

Vision-based navigation with language-based assistance via imitation learning with indirect intervention

Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan

CVPR, 2019

💡 Agent that can ask for help

@inproceedings{nguyen2019vnla,
  author = {Nguyen, Khanh and Dey, Debadeepta and Brockett, Chris and Dolan, Bill},
  title = {Vision-based navigation with language-based assistance via imitation learning with indirect intervention},
  booktitle = {CVPR},
  month = {},
  year = {2019},
  oneline = {Agent that can ask for help}
}

Global Voices: Crossing borders in automatic news summarization

Khanh Nguyen, and Hal Daumé III

New Frontiers in Summarization Workshop at EMNLP, Nov 2019

💡 Evaluation dataset for cross-lingual summarization

@inproceedings{nguyen2019gv,
  title = {Global Voices: Crossing borders in automatic news summarization},
  author = {Nguyen, Khanh and Daum{\'e} III, Hal},
  booktitle = {New Frontiers in Summarization Workshop at EMNLP},
  month = nov,
  year = {2019},
  oneline = {Evaluation dataset for cross-lingual summarization}
}

2017

Reinforcement learning for bandit neural machine translation with simulated human feedback

Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber

EMNLP, Sep 2017

💡 Improve machine translation with reinforcement learning from noisy ratings

Abs Paper Bib

Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training focuses on expensive human-generated reference translations. We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder-decoder architecture (Luong et al., 2015). This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine translation metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors.
@inproceedings{nguyen2017banditnmt, title = {Reinforcement learning for bandit neural machine translation with simulated human feedback}, author = {Nguyen, Khanh and Daum{\'e} III, Hal and Boyd-Graber, Jordan}, booktitle = {EMNLP}, month = sep, year = {2017}, address = {Copenhagen, Denmark}, publisher = {Association for Computational Linguistics}, url = {https://www.aclweb.org/anthology/D17-1153}, doi = {10.18653/v1/D17-1153}, pages = {1464--1474}, oneline = {Improve machine translation with reinforcement learning from noisy ratings} }

The UMD neural machine translation systems at WMT17 bandit learning task

Amr Sharaf, Shi Feng, Khanh Nguyen, Kiante Brantley, and Hal Daumé III

WMT, Sep 2017

@inproceedings{sharaf17wmt,
  title = {The UMD neural machine translation systems at WMT17 bandit learning task},
  author = {Sharaf, Amr and Feng, Shi and Nguyen, Khanh and Brantley, Kiante and Daum{\'e} III, Hal},
  booktitle = {WMT},
  month = sep,
  year = {2017},
  address = {Copenhagen, Denmark},
  publisher = {Association for Computational Linguistics},
  url = {https://www.aclweb.org/anthology/W17-4778},
  doi = {10.18653/v1/W17-4778},
  pages = {667--673},
}

2015

Posterior calibration and exploratory analysis for natural language processing models

Khanh Nguyen, and Brendan O’Connor

EMNLP, Sep 2015

💡 Calibrated probabilities for detecting errors of NLP models

@inproceedings{nguyen15calibration,
  title = {Posterior calibration and exploratory analysis for natural language processing models},
  author = {Nguyen, Khanh and O{'}Connor, Brendan},
  booktitle = {EMNLP},
  year = {2015},
  address = {Lisbon, Portugal},
  publisher = {Association for Computational Linguistics},
  url = {https://www.aclweb.org/anthology/D15-1182},
  doi = {10.18653/v1/D15-1182},
  pages = {1587--1598},
  oneline = {Calibrated probabilities for detecting errors of NLP models}
}