Interactive Imitation Learning

This collection of projects brings together multiple works on interactive imitation learning, where robot policies are learned from online human feedback. This approach not only enhances ease of use but also improves data efficiency, making robots more adaptable to both dynamic and unstructured environments.

1. Interactive Imitation Learning for Robotics: A Survey

In this work, we attempt to facilitate research in interactive imitation learning and lower entry barriers for new practitioners by providing a survey of the field that unifies and structures it. In addition, we aim to raise awareness of its potential, what has been accomplished and what are still open research questions.

Paper:

@Article{Celemin2022FTR,
  author  = {Celemin, Carlos AND P\'{e}rez-Dattari, Rodrigo AND Chisari, Eugenio AND Franzese, Giovanni AND de Souza Rosa, Leandro AND Prakash, Ravi AND Ajanovi\'{c}, Zlatan AND Ferraz, Marta AND Valada, Abhinav AND Kober, Jens},
  journal = {Foundations and Trends\textregistered{} in Robotics},
  title   = {Interactive Imitation Learning in Robotics: A Survey},
  year    = {2022},
  pages   = {1--197},
  volume  = {10},
  number  = {1--2},
  doi     = {10.1561/2300000072},
}
      

Links:

Project members:
C. Celemin*, R. Pérez-Dattari*, E. Chisari*, G. Franzese*, L. de Souza Rosa, R. Prakash, Z. Ajanović, M. Ferraz, A. Valada and J. Kober.
*Authors contributed equally
2022

2. Integrating Human Corrective Feedback and State Representation Learning

This project presents a collection of papers introducing the Deep COACH framework. Deep COACH was the first method to integrate relative corrective feedback into deep neural network frameworks. Prior to this, such approaches were limited to simpler function approximators, like linear combinations of basis functions. This integration was achieved by reformulating the feedback as a training label and designing an architecture that simultaneously minimizes auxiliary loss functions—such as observation reconstruction or next observation prediction—while improving the robot’s behavior to solve the task at hand.

Paper 1:

@inproceedings{perez2020interactive,
  title={Interactive learning with corrective feedback for policies based on deep neural networks},
  author={P{\'e}rez-Dattari, Rodrigo and Celemin, Carlos and Ruiz-del-Solar, Javier and Kober, Jens},
  booktitle={Proceedings of the 2018 International Symposium on Experimental Robotics},
  pages={353--363},
  year={2020},
  organization={Springer}
}
      

Links:

Paper 2:

@inproceedings{perez2019continuous,
  title={Continuous control for high-dimensional state spaces: An interactive learning approach},
  author={P{\'e}rez-Dattari, Rodrigo and Celemin, Carlos and Ruiz-del-Solar, Javier and Kober, Jens},
  booktitle={2019 International Conference on Robotics and Automation (ICRA)},
  pages={7611--7617},
  year={2019},
  organization={IEEE}
}
      

Links:

Paper 3:

@article{perez2020interactive,
  title={Interactive learning of temporal features for control: Shaping policies and state representations from human feedback},
  author={P{\'e}rez-Dattari, Rodrigo and Celemin, Carlos and Franzese, Giovanni and Ruiz-del-Solar, Javier and Kober, Jens},
  journal={IEEE Robotics \& Automation Magazine},
  volume={27},
  number={2},
  pages={46--54},
  year={2020},
  publisher={IEEE}
}
      

Links:

Project members:
R. Pérez-Dattari, C. Celemin, G. Franzese, J. Ruiz del Solar and J. Kober.
2018 – 2020

3. Contrastive Policy Shaping via Human Corrective Feedback

This project introduces CLIC, a novel approach to shaping a robot’s policy using corrective feedback. The formulation generates contrastive examples from the feedback, providing a general method that enables learning from both relative and absolute corrective feedback. Additionally, it supports learning with arbitrarily large replay buffers, addressing a key limitation of previous methods (e.g., Deep COACH) that learn from this type of feedback.

x1

x1

Project members:
Z. Li, R. Pérez-Dattari, R. Babuska, C. Della Santina, and J. Kober
ongoing

4. Visually-Guided Motion Planning from Human Feedback

This project introduced a Model Predictive Control (MPC) framework that incorporates high-level learned behaviors into its cost function using human feedback. This approach allows the integration of complex driving behaviors into an MPC, which would otherwise be difficult to model. As a result, we achieve an autonomous vehicle capable of driving safely and in compliance with other agents in an unstructured and dynamic environment.

Paper:

@article{perez2022visually,
  title={Visually-guided motion planning for autonomous driving from interactive demonstrations},
  author={P{\'e}rez-Dattari, Rodrigo and Brito, Bruno and de Groot, Oscar and Kober, Jens and Alonso-Mora, Javier},
  journal={Engineering Applications of Artificial Intelligence},
  volume={116},
  pages={105277},
  year={2022},
  publisher={Elsevier}
}
      

Links:

Project members:
R. Pérez-Dattari*, B. Brito*, O. de Groot, J. Kober, and J. Alonso-Mora.
*Authors contributed equally
2022