A Complete Deep Learning-Based Driver Monitoring System with Low Cost

Abstract

This research provides a solution to build a deep learning-based driving monitoring system (DDMS) to verify the driver’s ID, estimate the driver’s status and the vehicle’s occupancy status. In our DDMS, we built and trained three deep learning neural networks: object detection neural network (ODNN), facial recognition neural network (FRNN) and facial landmark detection neural network (FLDNN). We built software to drive a camera to capture images in the vehicle and integrate these three neural networks to build our DDMS. The pictures of the driver seat, passenger seat and back seat are sent to ODNN. ODNN detects common objects in vehicles such as humans, faces, dogs, and so on; the driver’s facial images are sent to FRNN. Our DDMS uses the facial recognition result to verify if the person is authorized to drive; the driver’s face images are also sent to FLDNN. FLDNN detects facial keypoints including eyebrows, eyes, the nose, lips, and the chin continuously. Our DDMS estimates driver’s state such as activeness, drowsiness, distractedness and so on. The movement of lips detected by FLDNN and other detected objects by ODNN near the driver’s face such as cell phone are used to estimate if the driver is speaking on the phone. Public datasets that meet our requirements to train our ODNN do not exist to the best of our knowledge. We built an Auto-Labeling Model (ALM) to expand an existing dataset by assigning labels to the data automatically. Quantizing deep learning models from floating point to 8-bit or lower-bit scaled integer data types is required to reduce the cost of using DDMS. Quantization is applied to convert the parameters in ODNN, FRNN and FLDNN from floating point numbers to integer numbers. The contribution in our study includes building and training an ALM to create a new dataset for ODNN; building and training ODNN, FRNN and FLDNN; developing software to drive the camera and integrating ODNN, FRNN and FLDNN together to provide vehicle’s occupancy status, face ID and driver status estimation results; quantizing ODNN, FRNN and FLDNN.