Semantic Curiosity for Active Visual Learning

Devendra Singh Chaplot*
Helen Jiang*
Saurabh Gupta
Abhinav Gupta
Published at ECCV, 2020 (spotlight)

[Talk (short)]
[Talk (long)]

In this paper, we study the task of embodied interactive learning for object detection. Given a set of environments (and some labeling budget), our goal is to learn an object detector by having an agent select what data to obtain labels for. How should an exploration policy decide which trajectory should be labeled? One possibility is to use a trained object detector's failure cases as an external reward. However, training RL policies requires millions of labelled samples, hence making any supervision infeasible. Instead, we explore a self-supervised approach for training our exploration policy by introducing a notion of semantic curiosity. Our semantic curiosity policy is based on a simple observation -- the detection outputs should be consistent. Therefore, our semantic curiosity rewards trajectories with inconsistent labeling behavior and encourages the exploration policy to explore such areas. The exploration policy trained via semantic curiosity generalizes to novel scenes and helps train an object detector that outperforms baselines trained with other possible alternatives such as random exploration, prediction-error curiosity and coverage-maximizing exploration.

Embodied Active Visual Learning

We use semantic curiosity to learn an exploration policy on set of the training environments. The exploration policy is learned by projecting segmentation masks on the top-down view to create semantic maps. The entropy of semantic map defines the inconsistency of the object detection module. The learned exploration policy is then used to generate training data for object detection/segmentation module. The labeled data is then used to finetune and evaluate the object detection/segmentation.

Short Presentation

Long Presentation

Demo Video

Paper and Bibtex


Chaplot, D.S., Jiang H., Gupta, S. and Gupta, A. 2020.
Semantic Curiosity for Active Visual Learning. In ECCV.

  title={Semantic Curiosity for Active Visual Learning},
  author={Chaplot, Devendra Singh and Jiang, Helen and Gupta, Saurabh and 
          Gupta, Abhinav},

Related Projects

Active Neural SLAM

Neural Topological SLAM

Semantic Exploration


This work was supported by IARPA DIVA D17PC00340, ONR MURI, ONR Grant N000141812861, ONR Young Investigator, DARPA MCS and NSF Graduate Research Fellowship. We thank Shubham Tulsiani for help in parts of the implementation and visualizations. We would also like to thank NVIDIA for GPU support. Website template from here and here.