Real time application of neural networks and hardware-accelerated image processing pipeline for precise autonomous agricultural systems

Abstract

The agricultural industry is undergoing a significant transformation as it increasingly adopts automation and precision technology to optimize crop management practices. In this context, this research focuses on developing an autonomous pesticide spraying rover that leverages advanced technologies to revolutionize precision agriculture. The primary objective of this project is to utilize a neural network for real-time aphid detection in sorghum crops, enabling the targeted application of pesticides only to infested plants. To achieve this goal, cutting-edge technologies and software frameworks, including ROS 2 (Robot Operating System), have been integrated to create a sophisticated software system for the rover. One of the major hurdles in deploying neural networks for real-time feedback in the field is the limitations of wireless communication. Transmitting large amounts of high-resolution images can be unreliable and slow, making it impractical to deploy the neural network on a remote server. To ensure real-time processing and feedback for the sprayer, the neural network is deployed directly on the rover’s computing system. For this purpose, Nvidia’s Jetson AGX Orin platform has been chosen, which combines an ARM processor with a powerful GPU. This setup ensures exceptional model inference performance compared to CPU-based solutions, enabling real-time processing and feedback for the sprayer. In addition to deploying the neural network on the rover, this research also focuses on efficiently handling image frames captured by the cameras. The GPU in the Jetson platform is utilized to accelerate the image pipeline and leverage the hardware encoder to compress image frames to H.264 format, resulting in streamlined data recording. This comprehensive approach enhances the rover’s capabilities while ensuring energy efficiency, a critical factor for field operations. The entire system is seamlessly integrated into ROS 2, and benefits from Nvidia’s Isaac ROS packages, which provide GPU-accelerated ROS nodes. The use of hardware acceleration is a pivotal component of this research, as it enables substantial computational power while maintaining efficiency. Furthermore, a method has been implemented to estimate the depth from the camera to the aphid from a 2D aphid detection. This depth estimation is subsequently used to determine the 3D coordinates of the aphid in the world, further enhancing the precision of the pesticide application. By offloading intensive processes from the CPU to the GPU and other accelerators, it is ensured that the autonomous pesticide spraying rover can operate effectively and deliver precise results, making it a valuable asset in the quest for optimized and sustainable agriculture. This thesis will provide an in-depth look at this innovative approach and showcase the practical implications of real-time neural network deployment in precision agriculture.