I primarily work on grounding language in situated environments to explore highly complex interactive grounded language learning without the complications that arise when modeling physical motor control and vision—situations that voice assistants such as Siri or Alexa might find themselves in when improvising responses. My work focuses imbuing agents with the priors required to not only operate, both act and speak, in interactive worlds, but to create them.

Projects:   Learning from Feedback and Aligning to Human Preferences   Grounding Language in Textual Worlds   World Modeling   Interactive Situated Dialogue   Automated Story Generation   Procedural Content Generation  

Learning from Feedback and Aligning to Human Preferences

The ultimate aim of language technology is to interact with humans. However, most such systems are trained without direct signals of human preference, with supervised target strings serving as (a sometimes crude) proxy. This work focuses on using reinforcement learning to interact and align to human preferences.

Representative Publications:

  • To facilitate research in this area, we release a library RL4LMs, a benchmark GRUE for NLP tasks paired with human preference rewards, and a novel RL algorithm NLPO for language in large action spaces (https://rl4lms.apps.allenai.org/).
    Rajkumar Ramamurthy*, Prithviraj Ammanabrolu*, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi
    Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
    International Conference on Learning Representations (ICLR) (2023).
    arXiv Conference bibtex
  • We teach LMs to unlearn undesired behaviors such as toxicity via a novel off-policy RL algorithm.
    Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi
    Quark: Controllable Text Generation with Reinforced Unlearning
    Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS) (2022).
    arXiv Conference bibtex
  • We focus on creating agents that act in alignment with socially beneficial norms and values in interactive textual worlds by introducing the GALAD agent that uses the social commonsense knowledge present in specially trained language models to contextually restrict its action space to only those actions that are aligned with socially beneficial values.
    Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hannaneh Hajizhirzi, and Yejin Choi
    Aligning to Social Norms and Values in Interactive Narratives
    North American Chapter of the Association for Computational Linguistics (NAACL) (2022).
    arXiv Conference bibtex
  • We release a benchmark and models for teaching agents when and how to take the initiative to ask humans for clarifying information when engaged in a conversation.
    Zeqiu Wu, Ryu Parish, Hao Cheng, Sewon Min, Prithviraj Ammanabrolu, Mari Ostendorf, and Hannaneh Hajishirzi
    INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions
    Transactions of the Association for Computational Linguistics (TACL) (2022).
    arXiv Journal bibtex

Grounding Language in Textual Worlds

I study Interactive Narrative games : partially observable simulations in which an agent interacts with the world through natural language—“perceiving”, “acting upon”, “and talking to” the world using textual descriptions, commands, and dialogue. These games are usually structured as puzzles or quests with long-term dependencies in which a player must complete a sequence of actions to succeed. Operating in these worlds requires an agent to reason about partially observable worlds in a combinatorially-sized state-action spaces. These works focus on the use of knowledge graphs to aid in these challenges.

Representative Publications:

  • Jericho (https://github.com/microsoft/jericho), the original human-made text game benchmark.
    Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Cot'e, and Xingdi Yuan
    Interactive fiction games: A colossal adventure
    Proceedings of the AAAI Conference on Artificial Intelligence 34 (2020).
    arXiv Conference bibtex
  • Introduces the KG-DQN, using KG state representations for more efficient off-policy RL learning.
    Prithviraj Ammanabrolu and Mark Riedl
    Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning
    North American Chapter of the Association for Computational Linguistics (NAACL-HLT) 2019 (2019).
    Link Conference bibtex
  • Explores how KG-DQN trained agents can transfer commonsense and domain-specific knowledge between domains and online guides through their graphs.
    Prithviraj Ammanabrolu and Mark Riedl
    Transfer in Deep Reinforcement Learning Using Knowledge Graphs
    Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13) at EMNLP (2019).
    Link Workshop bibtex
  • Introduces the KG-A2C, adapting KG-based agents to larger action spaces and on-policy training.
    Prithviraj Ammanabrolu and Matthew Hausknecht
    Graph Constrained Reinforcement Learning for Natural Language Action Spaces
    International Conference on Learning Representations (2020).
    OpenReview Conference bibtex
  • The Q*BERT agent is intrinsically motivated to ask questions about the world and learn more about it to more systematically explore it, all the while leveraging the commonsense reasoning abilities of a large language model.
    Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl
    How to avoid being eaten by a grue: Structured exploration strategies for textual worlds
    arXiv preprint arXiv:2006.07409 (2020).
    arXiv bibtex

World Modeling

Operating in most interactive narratives requires navigation and interaction with hundreds of locations, characters, and objects. These works focus on knowledge representation, i.e. building better world models of the transitions in such worlds.

Representative Publications:

  • The JerichoWorld (https://github.com/JerichoWorld/JerichoWorld) benchmark consisting of text state to world knowledge graphs from over 30 games.
    Prithviraj Ammanabrolu and Mark Riedl
    Modeling Worlds in Text
    Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (Round 1) (2021).
    OpenReview Conference bibtex
  • The Worldformer, a state-of-the-art textual world model that predicts world state and plausible agent actions simultaneously.
    Prithviraj Ammanabrolu and Mark Riedl
    Learning Knowledge Graph-based World Models of Textual Environments
    Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS) (2021).
    arXiv Conference bibtex

Interactive Situated Dialogue

These works focus on developing agents that can produce contextually relevant dialogue utterances while remaining true to their personas and motivations, situations that voice assistants such as Siri or Alexa might find themselves in when improvising responses.

Representative Publications:

  • We train Dungeons and Dragons player agents using transcripts of human-played games, finding that simultaneously predicting relationships between characters in a multi-user setting improves collaborative storytelling/dialogue abilities of agents.
    Wai Man Si, Prithviraj Ammanabrolu, and Mark O Riedl
    Telling Stories through Multi-User Dialogue by Modeling Character Relations
    SIGDIAL 2021 (2021).
    arXiv Conference bibtex
  • Builds on LIGHT (https://parl.ai/projects/light/), a large-scale crowd-sourced fantasy text game, by releasing a dataset of motivations or quests for characters and further an entire RL framework to train them. Agents we train interactively are not only more consistent with respect to the actions they do but more natural in terms of dialogues uttered.
    Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur Szlam, Tim Rockt"aschel, and Jason Weston
    How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds
    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2021).
    Link Conference bibtex

Automated Story Generation

Storytelling is one of the most natural forms of human communication and imagine how much more we can do if our computers can understand language.

Representative Publications:

  • Foundational automated storytelling paper that focused on the knowledge representations to be used for neural net-based stories.
    Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O Riedl
    Event Representations for Automated Story Generation with Deep Neural Nets
    Thirty-Second AAAI Conference on Artificial Intelligence (2018).
    Link Conference bibtex
  • A system that was able to realize and flesh out a plot into a full story.
    Prithviraj Ammanabrolu, Ethan Tien, Wesley Cheung, Zhaochen Luo, William Ma, Lara J. Martin, and Mark O. Riedl
    Story Realization: Expanding Plot Events into Sentences
    Proceedings of the AAAI Conference on Artificial Intelligence 34 (2020).
    Link Conference bibtex
  • Generating commonsensey consistent plots for stories with large scale commonsense transformers and plot graphs.
    Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O Riedl
    Automated storytelling via causal, commonsense plot ordering
    Thirty-Second AAAI Conference on Artificial Intelligence (2020).
    arXiv Conference bibtex

Procedural Content Generation

Agents trained to solve these games are limited by the scenarios described in them. Although the range of scenarios is vast, this brings about the question of what the agent is actually capable of understanding even if it has learned to solve all the puzzles in a particular game. These works are based on the idea that a potential way of testing an AI system’s understanding of a domain is to use the knowledge it has gained in a novel way and to create more instances of that domain.

Representative Publications:

  • Focuses on quest generation as a sequence of actions to be completed and then anchoring them in a given world using knowledge graphs.
    Prithviraj Ammanabrolu, William Broniec, Alex Mueller, Jeremy Paul, and Mark O. Riedl
    Toward Automated Quest Generation in Text-Adventure Games
    International Conference on Computational Creativity (ICCC) (2020).
    arXiv Conference bibtex
  • Focuses on world generation by extracting a knowledge graph of a world from a given story. Basically turning a linear reading experience into something you can interact with.
    Prithviraj Ammanabrolu, Wesley Cheung, Dan Tu, William Broniec, and Mark O Riedl
    Bringing stories alive: Generating interactive fiction worlds
    Proceedings of the Sixteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE-20) (2020).
    Link Conference bibtex