Design and development of generic framework for VR-headset-enabled drone control

Currently a large variety of relatively low-cost drones, all with differing sensor abilities and control methods, are coming to market. These drones can be remote controlled by an operator, and their sensor input can be retrieved while doing so. Most drones are controlled by a smartphone or a remote controller. Sensor in- put, like camera footage and altitude information, is visible in the vendor’s mobile application. The emergence of virtual reality headsets open new possibilities for controlling these drones and visualizing their sensor information. Using a VR head- set for controlling the drone can give a real “I’m there’ experience as they allow to control the direction of the drone’s camera simply by moving the head. This is meant to mimic the natural motion of the head, so looking up and down will move the camera in those directions.

 

Due to the variety in drones and VR headsets, regarding the control methods and sensor abilities, a software framework that can provide the necessary abstractions is required. The framework will need to provide abstractions for dealing with different types / brands of drones on the one hand, and different Virtual Reality goggles on the other.

Connecting a number of drones to the framework would allow an operator to have a nice overview of all the different drone camera images and would allow an operator to switch between the active drones on the fly, and take over the control with his head.

 

 

As currently all drones and VR headsets on the market have different control methods, research on this topic is paramount. The main objective is the design and creation of a framework prototype focusing on interoperability, responsiveness and modifiability. The framework will need to provide abstractions for dealing with different types / brands of drones on the one hand, and different Virtual Reality goggles on the other. The second focus point will be on techniques to minimize the experienced lag between head movements and the visual representation in the headset, when controlling the different drones. The final focus point is on easy modification of existing plugins to deal with SDK updates.

Practical applications :

For example, if a public place or an event needs to be monitored, this is usually done by placing a few cameras spread all across the field. Then, the camera images are monitored by a security member. However, when having a few drones in the air and all the drone camera images in a VR headset, it would allow the operator to keep a nice overview of the whole field, but also to take a closer look when an incident happens.

Master’s dissertation submitted in order to obtain the academic degree of Master of Science in Computer Science Engineering

by
 Stefaan VERMASSEN

Supervisors: Prof. dr. ir. Filip De Turck, Bruno Volckaert Counsellors: Ir. Thomas Dupont, Ir. Wannes Kerckhove

Faculty of Engineering and Architecture Ghent University