Nonlinear Model Predictive Control for Multi-rotor Aerial Vehicles under Visual-perceptive Constraints

1. Description

The objective of this work is to develop a controller that allows an aerial robot to safely and ergonomically hand over an object to a human operator.

The robot is equipped with an onboard sensor, for instance a monocular camera, through which it can perceive the human. The human can move freely in the environment. The goal of the robot is to hand over the tool while taking into consideration the safety and ergonomics of the human operator.

Precisely, a Nonlinear Model Predictive Control (NMPC) algorithm is formulated which accounts for: . Maintaining within the sensor field of view (FoV) the human, in order to guarantee a proper human state estimation. . Avoiding collision between the robot and human, which is of paramount importance in Human-Robot Interaction (HRI). . Maximizing the ergonomics of the human operator during the object transfer phase.

2. Simulations

Simulations are performed withing the Gazebo simulator.

In the following, a screenshot of a simulation is provided, where the robot is in front of the human operator in the process of closing the distance and handing over the object.

Snapshot of the simulation environment …​ .
Figure 1. Snapshot of the simulation environment in Gazebo.

On the top right, it is shown what the human sees from his perspective. On the bottom right, instead, the onboard camera video stream is displayed.

As shown there, the human is detected by means AruCo tags and the correct detection is marked with a red dot over the tag. This simple but yet effective method is used as possible implementation of the human detection pipeline. One could imagine that the human detection could take place by means of more sophisticated means and algorithms, for instance based on learning-based methods.

Another snapshot taken during the simulations is provided below. In this other figure, the robot is navigating around the human in order to reach a location in front of his body.

Simulation snapshot of the robot navigating around human operator.
Figure 2. Snapshot of the simulation when the robot is reaching a location in front of the human operator.

3. Technical details

3.1. Control architecture

In the following, a block diagram of the developed control framework is provided.

Block diagram of the developed control framework.
Figure 3. Block diagram of the developed control framework.

The Software collects all the modules related to the control framework, namely: * a reference generator in charge of providing the desired motion task that the robot has to perform (reach a position in front of the human); * the NMPC controller which computes the low-level commands for the robot actuators and takes into consideration the motion, visibility, ergonomics and safety tasks; * the robot and human state estimators which provide an estimation of the state for both agents.

The Hardware group contains the interface to the flight controller mounted on the robot, which receives the motor commands and sends them to the actuators. Additionally, it groups also the device to detect the human from the onboard sensor (Human Detector), and the interfaces to the other sensing devices providing the measures used in the robot state estimation (onboard IMU and external motion capture system).

3.2. Software implementation

We rely on the Telekyb3 architecture to simulate and interface to the low-level hardware.

All the software modules are realized exploiting GenOM3 and running within the middleware pocolibs. The output code is written in C/C++.

The NMPC controller is based on the MATMPC package and the CasADi library.


Nonlinear Model Predictive Control for Human-Robot Handover with Application to the Aerial Case


Gianluca Corsini[1], Martin Jacquet[1], Hemjyoti Das[2], Amr Afifi[2], Daniel Sidobre[1], Antonio Franchi[1][2]

Cite as:

G. Corsini, M. Jacquet, H. Das, A. Afifi, D. Sidobre, A. Franchi, "Nonlinear Model Predictive Control for Human-Robot Handover with Application to the Aerial Case", in The 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022), Oct 2022, Kyoto, Japan. doi:10.1109/IROS47612.2022.9981045.


Aerial Robotics; Human-Aerial Robot Interaction; Human-Robot Handover; Nonlinear Model Predictive Control (NMPC)

4.2. Released software

Open-source code for simulations/experiments:

NMPC for Human Aerial Handover

1. LAAS-RIS - Équipe Robotique et InteractionS.
2. EEMCS - Faculty of Electrical Engineering, Mathematics and Computer Science.