Devendra Singh Chaplot

I am a Research Scientist at Facebook AI Research. I am interested in Embodied AI and Computer Vision. Earlier I was a Ph.D. student at the Machine Learning Department, School of Computer Science, Carnegie Mellon University. During my Ph.D., I worked with Prof. Ruslan Salakhutdinov on Building Intelligent Autonomous Navigation Agents. I also worked closely with Abhinav Gupta and Saurabh Gupta. Prior to joining CMU, I graduated from IIT Bombay, India with a B.Tech in Computer Science & Engineering and a minor in Applied Statistics in August 2014, where I worked with Prof. Pushpak Bhattacharyya. I also worked for about a year at Samsung Electronics HQ in South Korea.

CV | Google Scholar | Github | Bio
Email: chaplot[at]cs.cmu.edu

Ph.D. Thesis

Building Intelligent Autonomous Navigation Agents
PDF | Talk | Slides

Updates

Jan '22
Paper on Multi-skill Mobile Manipulation (M3) accepted to ICLR 2023 as spotlight!
[PDF] [Webpage]
Dec '22
Our Multi-skill Mobile Manipulation (M3) model wins the Habitat Rearrangement Challenge at NeurIPS 2022!
[Challenge] [Code]
Dec '22
New paper on Navigating to Objects in the Real World.
[PDF] [Webpage]
Oct '22
Released the HM3D-Semantics Dataset, the largest public dataset of real-world 3D spaces with dense semantic annotations!
[PDF] [Webpage]
Oct '22
New paper on Retrospectives on the embodied ai workshop .
[PDF]
Oct '22
New paper on Instance-Specific Image Goal Navigation task.
[PDF]
Jun '22
Co-organizing the Embodied AI Workshop and Habitat ObjectNav Challenge at CVPR 2022
[Webpage]
Feb '22
PONI accepted to CVPR 2022 as oral!
[PDF] [Webpage]
Jan '22
FILM accepted to ICLR 2022
[PDF] [Webpage]
Sep '21
3 papers accepted at NeurIPS 2021: SEAL, NRNS, Habitat 2.0!
[Publications]
Aug '21
For students applying for graduate fellowships, I uploaded my research statement from the 2020 Facebook Fellowship application.
[PDF]
Jul '21
New ICML 2021 paper on Differentiable Spatial Planning using Transformers.
[PDF] [Webpage]
Apr '21
Excited to join Facebook AI Research as a Research Scientist!
Apr '21
Released code and pre-trained models for NeurIPS-20 paper on Object Goal Navigation.
[Code]
Mar '21
I successfully defended my Ph.D. Thesis on "Building Intelligent Autonomous Navigation Agents".
[Recording] [Slides]
Sep '20
Paper on Object Goal Navigation accepted at NeurIPS-20.
[PDF] [Webpage]
Jul '20
Our work on Object Goal Navigation was featured in TechCrunch, Gizmodo, The Hindu & CMU News.
[PDF] [Webpage]
Jun '20
We won the CVPR 2020 Habitat ObjectNav Challenge! Thanks Google for $10,000 GCP Credits.
[PDF] [Talk] [Slides]
Summer '20
Excited to intern with Jitendra Malik and Deepak Pathak at Facebook AI Research!
Apr '20
Our work on Active Neural SLAM was featured in VentureBeat, Synced and Analytics India Mag.
Apr '20
Released code and pre-trained models for ICLR-20 Active Neural SLAM paper.
[Code]
Feb '20
Paper on Neural Topological SLAM for Visual Navigation accepted at CVPR-20.
[PDF] [Webpage]
Jan '20
Received the Facebook Graduate Fellowship!
[Link] [CMU News]
Dec '19
Received the Nvidia Graduate Fellowship! (Declined)
Dec '19
New paper on Learning to Explore using Active Neural SLAM accepted at ICLR-20.
[PDF] [Webpage]
Oct '19
We received the Facebook Research Award for PyRobot: Democratizing Robotics!
Jun '19
We won the CVPR-19 Habitat Navigation Challenge! Thanks Google for $20,000 GCP Credits.
[Challenge] [PDF]
Summer '19
Interned with Abhinav Gupta and Saurabh Gupta at Facebook AI Research, Pittsburgh.
[PDF]
Feb '19
New paper on Embodied Multimodal Multitask Learning posted on arxiv.
[PDF] [Videos]
Oct '18
Excited to be a Workflow chair for ICML 2019.
Jun '18
Gave a talk at AIED-18 for our paper on Learning Cognitive Models using Neural Networks.
[PDF]
Summer '18
Interned with Dhruv Batra and Devi Parikh at Facebook AI Research, Menlo Park.
[PDF]
Jun '18
New AIED-18 paper on Learning Cognitive Models using Neural Networks posted on arxiv.
[PDF]
Jun '18
Paper on end-to-end global pose estimation using Neural Graph Optimization received Best Paper Award at the CVPR-18 Deep Learning for Visual SLAM workshop!
[PDF]
Jun '18
New ICML-18 paper on Gated Path Planning Networks posted on arxiv.
[PDF] [Code]
Apr '18
Released code, environment and pre-trained models for ICLR-18 Localization paper.
[Code]
Mar '18
Gave a talk on DeepRL in 3D environments at the Nvidia GTC 2018.
[Video] [Slides]
Feb '18
New paper on our work at Apple AI Research on end-to-end global pose estimation using Neural Graph Optimization posted on arxiv.
[PDF]
Feb '18
Gave two talks at AAAI-18, on WSD and Language Grounding.
[Slides-WSD] [Slides-LG]
Jan '18
Paper on Active Neural Localization accepted at ICLR-18.
[PDF] [Slides]
Jan '18
Released code, environment and pre-trained model for AAAI-18 Language Grounding paper.
[Code]
Jan '18
AAAI-18 WSD paper available on arXiv.
[PDF]
Dec '17
Released code and pre-trained models for our Doom AI agent, Arnold. Play against our Doom AI agent which won the Visual Doom AI Competition 2017 Full Deathmatch.
[Code]
Dec '17
Gave a talk on Active Neural Localization at the NIPS-17 DeepRL Symposium.
[Video] [Slides]
Nov '17
Two papers accepted for oral presentation at AAAI-18 on WSD and Language Grounding.
[PDF-WSD] [PDF-LG]
Sep '17
Our Doom AI agent, Arnold won the Visual Doom AI Competition 2017 Full Deathmatch.
[Competition Results]
Jun '17
MIT TechReview and Inverse write about our recent paper on language grounding!
[MIT TechReview] [Inverse]
Jun '17
New paper on Language Grounding posted on arXiv.
[PDF]

Select Research Projects

Navigating to Objects in the Real World

Navigating to Objects in the Real World
Theo Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, Devendra Singh Chaplot. (2022)
arXiv preprint arXiv:2212.00922
PDF | Webpage | Talk | Slides

SEAL: Self-supervised Embodied Active Learning

SEAL: Self-supervised Embodied Active Learning
Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, Ruslan Salakhutdinov. (2021)
Neural Information Processing Systems (NeurIPS-21)
PDF | Webpage | Talk | Slides

Goal-Oriented Semantic Exploration

Object Goal Navigation using Goal-oriented Semantic Exploration
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, Ruslan Salakhutdinov. (2020)
Neural Information Processing Systems (NeurIPS-20)
PDF | Webpage | Code | Pre-trained models | Talk | Slides
Media: TechCrunch, Gizmodo, The Hindu, CMU News
Winner CVPR 2020 Habitat ObjectNav Challenge.

Semantic Curiosity

Semantic Curiosity for Active Visual Learning
Devendra Singh Chaplot*, Helen Jiang*, Saurabh Gupta, Abhinav Gupta. (2020)
European Conference on Computer Vision (ECCV-20) (spotlight)
PDF | Webpage | Talk | Slides

Neural Topological SLAM

Neural Topological SLAM for Visual Navigation
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, Saurabh Gupta. (2020)
Computer Vision and Pattern Recognition (CVPR-20), Seattle, USA
PDF | Webpage | Talk | Slides

Active Neural SLAM

Learning to Explore using Active Neural SLAM
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta,
Ruslan Salakhutdinov. (2020)
8th International Conference on Learning Representations (ICLR-20), Addis Adaba, Ethiopia
PDF | Webpage | Code | Pre-trained models | Talk | Slides | Blog
Media: VentureBeat, Synced, Analytics India Mag
Winner CVPR 2019 AI Habitat Navigation Challenge: 1st place (RGBD), joint 1st place (RGB)

Embodied Multimodal Multitask Learning

Embodied Multimodal Multitask Learning
Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh,
Dhruv Batra. (2020)
29th International Joint Conference on Artificial Intelligence (IJCAI-20), Yokohama, Japan
PDF | Webpage

Learning to localize

Active Neural Localization
Devendra Singh Chaplot, Emilio Parisotto, Ruslan Salakhutdinov. (2018)
6th International Conference on Learning Representations (ICLR-18), Vancouver, Canada
PDF | Code | Pre-trained models | Webpage | Talk | Slides

Learning to follow language instructions

Gated-Attention Architectures for Task-Oriented Language Grounding
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, Ruslan Salakhutdinov. (2018)
32nd AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, USA.
PDF | Code | Environment | Pre-trained models | Webpage
Media: MIT TechReview, Inverse

Learning to play deathmatches in Doom

Playing FPS Games with Deep Reinforcement Learning
Guillaume Lample*, Devendra Singh Chaplot*. (2017)
31st AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, USA.

Arnold: An Autonomous Agent to play FPS Games (Best Demo Award)
Devendra Singh Chaplot*, Guillaume Lample*. (2017)
31st AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, USA. (demo)

PDF | Code | Pre-trained models | Demo videos (300K+ views)
Media: TechCrunch, Popular Science, Engadget, Daily Mail, Salon, Kotaku, ScienceAlert, Pittsburgh Post-Gazette, Inverse, CMU News
First place Visual Doom AI Competition 2017 Full Deathmatch
Second place Visual Doom AI Competition 2016 Full Deathmatch

Publications

Google Scholar

Navigating to Objects in the Real World

Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, Devendra Singh Chaplot (2022)
arXiv preprint arXiv:2212.00922
PDF | Webpage | Talk | Slides

Multi-skill Mobile Manipulation for Object Rearrangement (spotlight)

Jiayuan Gu, Devendra Singh Chaplot, Hao Su, Jitendra Malik (2023)
International Conference on Learning Representations (ICLR-23), Kigali, Rwanda
PDF | Webpage | Talk | Code

Habitat-Matterport 3D Semantics Dataset

Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, Alexander William Clegg, Devendra Singh Chaplot Devendra Singh Chaplot (2022)
arXiv preprint arXiv:2210.05633
PDF | Webpage | Data

Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances

Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, Devendra Singh Chaplot (2022)
arXiv preprint arXiv:2211.15876
PDF | Code

Retrospectives on the embodied ai workshops

Matt Deitke et al. (2022)
arXiv preprint arXiv:2210.06849
PDF

PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning (oral)

Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, Kristen Grauman. (2022)
Computer Vision and Pattern Recognition (CVPR-22), New Orleans, USA
PDF | Webpage | Talk | Code

FILM: Following Instructions in Language with Modular Methods

So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, Ruslan Salakhutdinov. (2022)
International Conference on Learning Representations (ICLR-22)
PDF | Webpage | Code

SEAL: Self-supervised Embodied Active Learning

Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, Ruslan Salakhutdinov. (2021)
Neural Information Processing Systems (NeurIPS-21)
PDF | Webpage | Talk | Slides

No RL, No Simulation: Learning to Navigate without Navigating

Meera Hahn, Devendra Singh Chaplot, Shubham Tulsiani, Mustafa Mukadam, James M. Rehg, Abhinav Gupta
Neural Information Processing Systems (NeurIPS-21)
PDF | Webpage | Talk | Code

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra
Neural Information Processing Systems (NeurIPS-21)
PDF | Webpage | Blog | Code

Differentiable Spatial Planning using Transformers

Devendra Singh Chaplot, Deepak Pathak, Jitendra Malik. (2020)
International Conference on Machine Learning (ICML-21)
PDF | Webpage | Talk | Slides

Object Goal Navigation using Goal-oriented Semantic Exploration

Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, Ruslan Salakhutdinov. (2020)
Neural Information Processing Systems (NeurIPS-20)
Also presented at the CVPR-20 Habitat Embodied Agents Workshop
PDF | Webpage | Code | Pre-trained models | Talk | Slides
Media: TechCrunch, Gizmodo, The Hindu, CMU News

Semantic Curiosity for Active Visual Learning (spotlight)

Devendra Singh Chaplot*, Helen Jiang*, Saurabh Gupta, Abhinav Gupta. (2020)
European Conference on Computer Vision (ECCV-20)
PDF | Webpage | Talk | Slides

Neural Topological SLAM for Visual Navigation

Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, Saurabh Gupta. (2020)
Computer Vision and Pattern Recognition (CVPR-20), Seattle, USA
PDF | Webpage | Talk | Slides

Learning to Explore using Active Neural SLAM

Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, Ruslan Salakhutdinov. (2020)
8th International Conference on Learning Representations (ICLR-20), Addis Adaba, Ethiopia
Also presented at the CVPR-19 Habitat Embodied Agents Workshop, Long Beach, USA
PDF | Webpage, Code | Pre-trained models | Talk | Slides | Blog
Media: VentureBeat, Synced, Analytics India Mag

Embodied Multimodal Multitask Learning

Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, Dhruv Batra. (2020)
8th International Conference on Learning Representations (IJCAI-20), Yokohama, Japan
PDF | Demo videos

Gated Path Planning Networks

Lisa Lee, Emilio Parisotto, Devendra Singh Chaplot, Eric Xing, Ruslan Salakhutdinov. (2018)
35th International Conference on Machine Learning (ICML-18), Stockholm, Sweden
PDF | Code

Learning Cognitive Models using Neural Networks (oral)

Devendra Singh Chaplot, Christopher MacLellan, Ruslan Salakhutdinov, Kenneth Koedinger. (2018)
19th International Conference on Artificial Intelligence in Education (AIED-18), London, UK
PDF

Global Pose Estimation with an Attention-based Recurrent Network (oral, Best Paper Award)

Emilio Parisotto*, Devendra Singh Chaplot*, Jian Zhang, Ruslan Salakhutdinov. (2018)
Conference on Computer Vision and Pattern Recognition (CVPR-18), Deep Learning for Visual SLAM workshop, Salt Lake City, USA.
PDF

Active Neural Localization

Devendra Singh Chaplot, Emilio Parisotto, Ruslan Salakhutdinov. (2018)
6th International Conference on Learning Representations (ICLR-18), Vancouver, Canada
Also presented as contributed talk at the NIPS-17 Deep RL Symposium, Long Beach, USA
PDF | Code | Environment | Pre-trained models | Demo videos | Talk | Slides

Gated-Attention Architectures for Task-Oriented Language Grounding (oral)

Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, Ruslan Salakhutdinov. (2018)
32nd AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, USA.
PDF | Code | Environment | Pre-trained Models | Demo videos
Media: MIT TechReview, Inverse

Knowledge-based Word Sense Disambiguation using Topic Models (oral)

Devendra Singh Chaplot, Ruslan Salakhutdinov. (2018)
32nd AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, USA.
PDF

Playing FPS Games with Deep Reinforcement Learning

Guillaume Lample*, Devendra Singh Chaplot*. (2017)
31st AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, USA.
PDF | Code | Environment | Pre-trained models | Demo videos
Media: TechCrunch, Popular Science, Daily Mail, Salon, Kotaku, ScienceAlert, Pittsburgh Post-Gazette

Arnold: An Autonomous Agent to play FPS Games (Best Demo Award)

Devendra Singh Chaplot*, Guillaume Lample*. (2017)
31st AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, USA. (demo)
PDF | Code | Environment | Pre-trained models | Demo videos
Media: Engadget, CMU News, Inverse

Transfer Deep Reinforcement Learning in 3D Environments: An Empirical Study

Devendra Singh Chaplot, Guillaume Lample, Kanthashree Mysore Sathyendra, Ruslan Salakhutdinov (2016).
30th Annual Conference on Neural Information Processing Systems (NIPS-16), Deep RL Workshop, Barcelona, Spain.
PDF

Unsupervised Word Sense Disambiguation using Markov Random Field and Dependency Parser

Devendra Singh Chaplot, Pushpak Bhattacharyya, Ashwin Paranjape. (2015)
29th AAAI Conference on Artificial Intelligence (AAAI-15), Austin, USA.
PDF, Demo

Data-driven Automated Induction of Prerequisite Structure Graphs

Devendra Singh Chaplot, Yiming Yang, Jaime Carbonell, Kenneth Koedinger. (2016)
9th International Conference on Educational Data Mining (EDM-16), Raleigh, USA.
PDF

Personalized Adaptive Learning using Neural Networks

Devendra Singh Chaplot, Eunhee Rhim, Jihie Kim. (2016)
3rd ACM Conference on Learning at Scale (L@S-16), Edinburgh, UK.
PDF

Predicting Student Attrition in MOOCs using Sentiment Analysis and Neural Networks

Devendra Singh Chaplot, Eunhee Rhim, Jihie Kim. (2015)
17th International Conference on Artificial Intelligence in Education (AIED-15), ISLG Workshop, Madrid, Spain.
PDF

SAP: Student Attrition Predictor

Devendra Singh Chaplot, Eunhee Rhim, Jihie Kim. (2015)
8th International Conference on Educational Data Mining (EDM-15), Madrid, Spain. (demo)
PDF

IndoWordnet Visualizer: A Graphical User Interface for Browsing and Exploring Wordnets of Indian Languages

Devendra Singh Chaplot, Sudha Bhingardive, Pushpak Bhattacharyya. (2014)
7th Global WordNet Conference (GWC-14), Tartu, Estonia.
PDF

Comparing Model Comparison Methods

Holger Schultheis, Ankit Singhaniya, Devendra Singh Chaplot. (2013)
35th annual conference of the Cognitive Science Society (CogSci-13), Berlin, Germany.
PDF

(* Equal Contribution)

Talks


Building Intelligent Autonomous Navigation Agents

Ph.D. Thesis Defense
Video, Slides

Semantic Curiosity for Active Visual Learning

ECCV 2020 (spotlight)
Video, Slides

Object Goal Navigation using Goal-Oriented Semantic Exploration

CVPR 2020 Embodied AI Workshop
Winning entry for Habitat ObjectNav Challenge
Video, Slides

Neural Topological SLAM for Visual Navigation

CVPR 2020 Main Conference
CVPR 2020 Workshop on 3D Scene Understanding for Vision, Graphics, and Robotics
Video, Slides

Learning to Explore using Active Neural SLAM

ICLR 2020
CVPR 2019 Habitat Embodied Agents Workshop, Winning entry
Video, Slides

Tutorial on Deep Reinforcement Learning

2019 Summer Workshop on Machine Learning, Tepper School of Business, CMU, Pittsburgh
Workshop, Google Colab Notebook

Playing FPS Games with Deep Reinforcement Learning

Nvidia GTC 2018
Video, Slides

Doom and Unreal Game Engines

Embodied Agents and Environments Workshop 2018, Facebook AI Research, Menlo Park
Slides

Gated-Attention Architectures for Task-Oriented Language Grounding

AAAI 2018, oral
Slides

Knowledge-based Word Sense Disambiguation using Topic Models

AAAI 2018, oral
Slides

Active Neural Localization

NIPS 2017, Deep Reinforcement Learning Symposium
Video, Slides

Website Template