Exploring the Impact of Explainable Heterogeneous Fusion Performance for Target Tracking

Abstract

The prolific use of deep learning models in the information age has reached a point where it is almost ubiquitous with sensor fusion. The combination of available data and the ever-increasing processing power of available hardware has made many believe that data science is just a matter of throwing obscene amounts data into a deep learning algorithm. Rather than considering factors like the quality or available quantity of the data or trying to break down the complexity of the problem the model is designed to solve, the naive use of brute force training is the chosen approach. This misconception, coupled with the inherent blackbox nature of deep learning algorithms, makes that the importance of explainability and transparency is integral to the success of machine learning, particularly with respect to using data we inherently lack an understanding of. This is especially the case for Passive Radiofrequency (P-RF) data, which despite radar having existed since the 19th century, there is a lack of literature regarding the usage of available I/Q data for detection purposes. While P-RF data has many potential benefits for detection, with our research group’s previous research and tentative patent requiring considerably less physical hardware to implement (using commercially available software defined radio dipole antennas), the ability to utilize the modality is primarily dependent on the application of AI. In this thesis, we showcase research to the detection of vehicle targets via multimodal fusion in the ESCAPE dataset. The research presented includes a comparison of over thirty multimodal fusion models for the shared objective of detection and differentiation of vehicle targets and examine the sensor data’s impact on the model’s performance. The integration of acoustic, electro optical, passive radiofrequency, and seismic data is implemented over three scenarios of data to determine the impact of different modalities using explainable AI methods. By comparing the local and global impact of each modality, as well as the F1 Score of the trained model, we can draw conclusions regarding how the modality was utilized with the benefit of the context of the training data and scenario. Computational costs of each model are considered in the context of FLoating Point Operations (FLOPs), and used to make determinations as to which modalities provided the most value to the fusion process.