Autonomous UAV Navigation in Unknown Terrain/Environment using Reinforcement Learning

Abstract

Over the last few years, UAV applications have grown immensely from delivery services to military use. Major goal of UAV applications is to be able to operate and implement various tasks without any human aid. To best of our knowledge, in the existing works for autonomous navigation for UAV's, ideal environments (e.g., 2D) are considered instead of realistic or special hardware are used (e.g., nine range sensors) to navigate through an ideal environment. Therefore, in this thesis, we aim to overcome the limitations of the existing works by proposing a model for navigating a drone in an unknown environment without any human help or aid. The goal of this research is to navigate from location A to location B in unknown terrain without having any prior knowledge about the terrain using default drone sensors only. We present a model which is compatible with almost every off-the-shelf drone available in the market. Our methodology utilizes only standard drone sensors which are attached to almost every drone. These include a camera, GPS, IMU, magnetometer, and barometer. Our methodology uses 3D, POMDP, and continuous environment. We have experimented with three different types of simulation environments in this work; Blocks, Landscape, Neighborhood. In our approach, we use a DNN for predicting UAV's next movement. First few layers are convolutional layers with propose of generating deep vector representation of camera image. This deep vector representation is combined with other sensor data in fully connected layers. The network outputs the next movement for our UAV. Our neural network architecture is sort of CNN but the key difference is instead of classification, it generates probability distribution for the next possible movement of UAV. We achieved completely autonomous (unaided) flight and navigation using Deep Q-learning, which is a subfield of RL. We implemented two different versions of the algorithm, i.e. policy-based DQN and value-based DQN. We were able to achieve 97.24% success rate for policy-based DQN and 96.74% success rate for value base DQN. We were able to demonstrate that our proposed approach was able to navigate the unknown environment successfully when it was trained for over 1000 iterations. The results showed that the rewards for the first 500 iterations were low. This was because the DQN was exploring different strategies and finding ones which work. Post 500 iterations the reward started to go up, and the performance started to improve. After 1000 iterations the DQN was successfully able to navigate drone in an unknown environment with ease. Our main contributions are: 1) We are using realistic environment model including factors like rain and wind. 2) We are only using onboard computing resources to run our model instead of some external server. 3) We were able to achieve improved results in terms of success (97.24%), failure (1.09%), and stray rate (1.66%). Another factor that distinguishes this work from other works is its potential for mass adaptability. In this work we are only using standard sensors without any special hardware requirements. This make our work widely adaptable for any off-the-shelf drone in the market.