1

Language model acceptability judgements are not always robust to context
Targeted syntactic evaluations of language models ask whether models show stable preferences for syntactically acceptable content over …
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, Adina Williams
Cite Arxiv ACL Anthology
The Curious Case of Absolute Position Embeddings
Transformer language models encode the notion of word order using positional information. Most commonly, this positional information is …
Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, Adina Williams
Cite DOI Arxiv
How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts
Neural Machine Translation systems built on top of Transformer-based architectures are routinely improving the state-of-the-art in …
Shanya Sharma, Manan Dey, Koustuv Sinha
Cite DOI Arxiv
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to …
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela
Cite Arxiv ACL Anthology Code
UnNatural Language Inference
Natural Language Understanding has witnessed a watershed moment with the introduction of large pre-trained Transformer networks. These …
Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, Adina Williams
Cite Arxiv ACL Anthology Code
Sometimes We Want Ungrammatical Translations
Rapid progress in Neural Machine Translation (NMT) systems over the last few years has been driven primarily towards improving …
Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau, Adina Williams
Cite ACL Anthology
Ideas for Improving the Field of Machine Learning: Summarizing Discussion from the NeurIPS 2019 Retrospectives Workshop
This report documents ideas for improving the field of machine learning, which arose from discussions at the ML Retrospectives workshop …
Shagun Sodhani, Mayoore S Jaiswal, Lauren Baker, Koustuv Sinha, Carl Shneider, Peter Henderson, Joel Lehman, Ryan Lowe
Cite Arxiv
Learning an Unreferenced Metric for Online Dialogue Evaluation
Evaluating the quality of a dialogue interaction between two agents is a difficult task, especially in open-domain chit-chat style …
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, Joelle Pineau
Cite Arxiv ACL Anthology Code
Measuring Systematic Generalization in Neural Proof Generation with Transformers
We are interested in understanding how well Transformer language models (TLMs) can perform reasoning tasks when trained on knowledge …
Nicolas Gontier, Koustuv Sinha, Siva Reddy, Christopher Pal
Cite Arxiv NeurIPS Code
Probing Linguistic Systematicity
Recently, there has been much interest in the question of whether deep natural language understanding models exhibit systematicity; …
Emily Goodwin, Koustuv Sinha, Timothy J. O'Donnell
Cite Arxiv ACL Anthology Code