Machine vision in hay bale production

Abstract

The goal of this project is to develop a system capable of real-time detection, pass/fail classification, and location tracking of large square hay bales under field conditions. A review of past and current methods of object detection was carried out. This led to the selection of the YOLO family of detectors, due to its relatively high precision, recall and mAP capabilities while still being able to maintain 10 to 15 fps on a Jetson Nano during detection. A dataset with over 6000 images of both good and bad hay bales was collected. Unfortunately, the dataset developed a class imbalance, with more good bale images than bad bales. This caused the bad bale class to over train and the good bale class to under train, severely impacting precision, and recall. To correct this imbalance, three different methods of image generation were tested, PFGM++, DCGAN, and Least Squares GAN. The methods were tested against two baselines from the original imbalanced data, one from just the data without any pre-training and the other with pre-training on Microsoft’s MSCOCO multiclass image dataset, commonly used a baseline by the computer vision community. The results from this showed a clear improvement in model quality for the PFGM++, a slight improvement was seen with the Least Squares GAN, but the results from the DCGAN were mixed. The results also showed a clear improvement when using pre-training on the MSCOCO dataset over the from scratch training. For the prototype, the system would need to be small but still powerful enough to run real-time detection. The Nvidia Jetson Nano was selected to be the main computing platform for the system. Using a YOL0V8n model, the smallest parameter profile, the Jetson Nano can maintain 15fps even with multiple detected objects in camera view. The Luxonis Oak-D pro POE camera was chosen to carry out the detection and provide depth estimation of the bales. With IR vision, IR laser active stereo vision, and an IP-67 rating, it is a good choice for use on agricultural equipment that might be exposed to the elements and can also support night-time baling operations. Results from testing showed that the system can detect good and bad bales. However, changes in bale material such as corn stover vs alfalfa and the reduction of the model from the v8x to the v8n, impacted the performance of the system. The system was able to achieve a precision of 0.655 and recall of 0.799 during testing on corn stover bales. Using the v8x model would improve this moving forward.