Deep learning for multimodal perception: Improving lidar and radar integration in self-driving cars

Autonomous driving relies on a range of advanced technologies, including computer vision, radar, lidar, and deep learning. Each of these components brings distinct strengths and limitations, but when used together, they create powerful synergies. While many self-driving systems employ radar or lidar independently, this separation often leaves critical gaps in perception. Combining radar and lidar with deep learning can significantly enhance a vehicle's environmental awareness by leveraging the complementary properties of each technology, thereby reducing individual weaknesses and improving overall safety. This paper explores the historical evolution and current challenges associated with deep learning, radar, and lidar in autonomous systems. It also examines methods for integrating radar and lidar data through a deep learning approach called multimodal learning, focusing on two key frameworks: M2-Fusion and ST-MVDNet. Although prior research on radar-lidar fusion remains limited due to the ongoing advancement of these technologies, various integration strategies have been introduced.

The objective of this paper is to present a systematic review of lidar, radar, and deep learning, offering a cohesive summary of their technological development. We also provide a critical evaluation of the M2-Fusion and ST-MVDNet models, highlighting their potential as well as the limitations that may affect their real-world implementation in modern autonomous vehicles.