Ambient Awareness on Sidewalk for Blind and Visually Impaired

Ambient awareness is critical for safe navigation, especially for the people who are blind or visually impaired. In particular, awareness of obstacles such as debris, potholes, construction site, and traffic movement pattern on the sidewalk is very helpful to improve mobility and independence. To address this problem, we developed an interactive and portable system. The system has option to use human intelligence through crowd-sourcing and caregiver(s) to assist the users in near real-time.

Ambient Awareness Application

The ambient awareness on the sidewalk model is developed using a Deep Neural Network (DNN). The problem was set as image classification problem and found solution using CNN. We report three studies that were performed in developing the system. Firstly, an interview with 50 individuals (who are blind or visually impaired) was conducted to select a set of obstacles to model the ambient awareness on the sidewalk. Secondly, we built a Sidewalk Obstacle Image Dataset (SOID) comprising of 50K obstacle images to train the DNN model. Finally, we built fully integrated system prototype on low form-factor devices such as Raspberry Pi3 (RPi3) and Android Smartphone. Audio and haptic feedback schemes implemented to accommodate user preferences and personalization. We perform quantitative evaluation of the prototype system on data communication, system latency, the accuracy of the DNN model.

To make the system more robust we are considering couple of features related to level of threat/danger the obstacle poses to the visually impaired. Some of the features are,

  • Localization of the obstacle
  • Motion of the obstacle
  • Speed of the moving obstacle
  • Distance of the obstacle

We are working towards a more robust solution by considering sidewalk obstacle annotation as well as is localized and segmented or annotated in real-time.