Internet of Things Care Device for Visually Impaired and Old People

Introduction

Eyesight gives the capability of seeing and is a gateway for what a human perceives or understands. According to a report of the World Health Organization (Media Center, 2014), Blind persons cannot experience the world as a normal person does and face challenges in performing day to day activities. These challenges include the detection of obstacles and differentiating between them, calling someone in case of emergency, gaining knowledge about the surrounding environment, and navigating. According to estimation, more than 82% of blind people are aged above fifty year.

People having visual impairment are dependent on assisting devices mostly while travelling. Traditional devices include long white canes, shown in (A. G. t. B. P. Accessible Pedestrian Signals., n.d.), which are used for the purpose of obstacle detection and navigation. Various researches have been done for the development of assisting devices. These devices include blind sticks (Willing, n.d.), assistive devices (Prasanthi & Tejaswitha, 2015) which uses ultrasonic or sonar-based sensors that detect the obstacle and alerts the blind persons through notifications with the help of a speaker (Senthilingam, 2011).

It is desolate to study that more than 90% of the blind persons around the globe are living in low-income settings, so a device that is cost-effective, easy to use, and must incorporate fast processing for switching information between the user and the sensors.

The proposed technology solution is wearable as a jacket and has diverse features that provide assistance to the blind person. The proposed system detects the obstacle surrounded by the person and calculates the distance of these obstacles from him. The approximate height of the obstacle can also be determined. The proposed design enables the person to book a cab on a single touch of a switch and provide the choice to send his location to a relative or any emergency service. The commands are converted into voice signals and given to the user. The initial part of the paper introduces device and also discuss the motivation for taking up this work. In the second section, some of the existing works. The third section proposes the product and outlines an overview of the design and implementation of work. Section 4 concludes with experiments and results, and performance. Finally the paper ends with conclusion and future scope of the work.

Literature Review

Several numbers of technology solutions aiding blind are existing. Traditional systems use long white canes for assistance. The integration of various sensors on these white corners was a big advancement in these traditional systems. A couple of devices for navigation have been designed for assisting blind people and help them gain more senses to the surroundings. Eye stick (T. Mishra, 2015) and Ultra stick (Agarwal et al., 2015) uses sensors on traditional white canes for helping blind people to perform various daily life activities. These devices use sensors such as sonar and ultrasonic sensors to detect the presence of an obstacle and provide feedback to the user. There are assisting devices that detect obstacles around the blind person and also detect the location where the person holding device is present. They provide voice-based assistance and use Global Positioning System (GPS) and Global System for Mobile (GSM) for getting the location of the person handling the device (T. M. magazine, n.d.). Another system used GPS, RFID Module, and ultrasonic sensors for taking input and interfaced with ATmega16 as a microcontroller. GSM was used for taking the output and an IC was used for transmitting voice to the speaker and data transmitted was shown on Liquid Crystal Display (LCD) (John Dew,  2010).

Various devices provide acoustic signals to blind people for providing them navigation assistance. The location of stimulation in the body is critical in devices where the sense of touch is utilized to give feedback. The VEST (Assistive Technology Blog, 2014) is a device that has vibration units and is wearable under the shirt. The surrounding sounds collected by an advanced mobile phone which is then converted into signals that are used to trigger the vibration units.  These vibrations are then transferred to the cerebrum where they translate into data. The sensors for capturing surroundings such as ultrasonic sensors and cameras must be attached in a specific position, as it is required to cover the complete field view of the blind person (G. Open Source, n.d.). 

The device needs to be portable and should be meeting both user’s needs and the engineer’s points of view. These portable devices should be compact, light, and easy to carry. This increases the difficulties in establishing connections between various components involved. The portability of devices also puts limitations on the power supply and management system. Wireless technology such as Bluetooth (espeak.sourceforge.net, n.d.) is used to interface various components in various places. A ZigBee based wireless system is used to communicate between the user’s ZigBee module to the server’s ZigBee module (Khlaikhayai et al., 2010). A TTS module was integrated which was further connected to a headphone as an output device.

Proposed Design

The proposed system is an Electronic Travelling Aid (ETA), a wearable assistive device that comprises a control box tied up on the arm of the jacket (see Fig. 1).

Figure 1. Control Box

The numerous sections on the control box are as follows:

Strap: It is used to fasten the control box on the arm of a person wearing a jacket.

HDMI (High-Definition Multimedia Interface) Port: It provides an audio/video interface for transmitting data.USB (Universal Serial Bus) Port: It is a standard connection interface that allows transferring digital data. 2.5 A, 5 V USB ports for charging personal devices are additionally provided. Switch (1-4): These switches will serve the purpose of selection among the various assisting features. Two switches for switching OS (Operating System) and main supply are provided additionally. Earphone Jack: A 3.5 mm audio jack enables headphones or speakers.

Power Jack: It is used for charging the battery of the system. Sensor Input: All ultrasonic sensors are attached to the control box using FRC (Flat Ribbon Cable) connector. The jacket holds all the essential hardware and the user only needs to wear it (see Fig. 2).

Figure 2. Different orientation of the Jacket holding essential Hardware.

The device enables the person to trigger numerous operations using the four switches on the control box for the assistance of blind paired persons. The first switch triggers the system to monitor the various obstacles around, and how far each obstacle is located. The second switch initiates the system to determine the height of an obstacle encountered in front of the blind person wearing the jacket. Taxi services are an economical and time-saving way to commute from one area to another. It is difficult for the blind person to book a cab because it requires a series of steps. The third switch will transfer the location of the person to a taxi service providing company via an email. The person can tell the destination to the driver once the taxi reaches on the location of the person holding the device.    This system will help the blind person to book a cab on a single click of a switch. One of the common issues faced by the blind person is to contact relatives or services like police and ambulance in case of emergency. The fourth switchon the control box will fetch the location and send it to the concerned person via email, thus helping to reduce the rescuing time for a helper. Furthermore, this device is a lightweight, economical, convenient, and wearable jacket and gives notification to person by an earpiece. This design helps to solve the issues faced by blind persons and make them self confident.

Components Required

Hardware Specification

Rasp Pi, Earphones with 3.5mm jack are used which are mostly compatible with the majority of devices. A 5 V, 2.5 Amp Lithium Ion battery support is provided to supply power to the system, having a total capacity of 20,000 mAh. Ultrasonic Sensors(HC-SR04) of  range of HC-SR04 varies between 2cm - 400 cm, with an accuracy of 3mm (Sparkfun, n.d.). GPIO pins are present on the Rasp Pi which manages the input and output operations. There are a total of 40 pins which are divided into two rows, each containing 20 of them (Yi, 2013). In GPS Module (NEO-6M) the output of the GPS receiver module is in standard string format (Electronic Wings, n.d.).It uses UART communication to communicate with the Rasp Pi controller. The output is provided serially on the Tx pin with a default baud rate of 9600 (Electronic Wings, n.d.). The NodeMCU ESP8266 development board, The MPU-6050 device is a complete 6-axis Motion Tracking Device. It has 3-axis gyroscope and a 3-axis accelerometer and also embedded with a Digital Motion Processor and an on-chip temperature sensor. Arduino Uno is a ATmega328P which used for work.

Figure 3. Potential divider circuit for HC-SR04.

Software Specification

Operating System used is Rasp Pi operates on Raspbian, which is a free operating system based on Debian, optimized for the Rasp-Pihardware (Rasp-PipPi, n.d.). Python IDE and Arduino IDE are used as free software in the work. In (Sareeka at el., 2018), (Kumar KN at el., 2019), (Khan at el., 2018) and (Jain B at el., 2018) suggested some models for blind and old people. Also (Kumar at el., 2019) uses IoT and cloud for designing intelligent home automation system. So IoT Technology is best suitable for designing IoT based care device for blind and old people.

Approach

This segment elaborates on the functioning of the system and the synchronization of the components with each other to accomplish the final goal. The main aim of the system is to make the user of the system more aware of his surroundings and make the task much safer and simplified. The basic workflow can be understood by the flow chart (see Fig. 4).

Figure 4. Flow Chart of the Control Box

The main python script will run on the Rasp-Pi Pi at start-up. The script will use the eSpeak package to make the system speak a welcome message to the user. The system will run in a continuous loop to check the transition of a specific switch and execute a different script to provide specific assistance to a blind person. When the user presses two switches at the same time, then the system will detect it as a wrong selection of switches and be aware of the person about the same. The numerous assistance provided are as follows - .

Detection of Obstacle’s Range

When the first switch on the control box is pressed, the ultrasonic sensors attached to numerous positions of the jacket will send a sequence of ultrasonic pulses. If any obstacle is detected then the pulse will be reflected back to the receiver in Fig. 5.  The received ultrasonic wave generates a control signal that triggers the echo pin of the microcontroller. The range is calculated through the time interval between transferred trigger signals and received echo signals (Sparkfun, n.d.). The microcontroller will process the information and convert it into text. The text is then converted into an audio signal. The user can hear the audio using a headphone or speaker attached to the control box. The audio signal will assist the person regarding the distance of the obstacle from the person which he is surrounded by.

Figure 5. Transmission and Reception of Ultrasonic Pulses.

Detection of Obstacle’s Height

Two ultrasonic sensors are attached to the front section of the jacket at different heights in figure 2. When the second switch on the control box is pressed, both sensors transmit the trigger signals. The obstacle will be considered as a long obstacle when both sensors receive the echo signal in figure 6 while the obstacle is considered a short obstacle when only the bottom sensor receives the echo signal in figure 7. The microcontroller will process the information and convert it into text. The text is then converted into an audio signal. By getting the information regarding the height of the obstacle, it would be easy for the blind person to evaluate the path planning.

Figure 6. Detection of long obstacles.

Figure 7. Detection of short obstacles.

On-demand Cab Booking Facility

The system will initiate the GPS module when the user triggers the third switch on the control box. The GPS will give the output in standard NMEA string format. The microcontroller will process the string and extract latitude and longitude from it. The latitude and longitude will be sent to a taxi service providing company via mail. The person can tell the destination to the driver once the taxi reaches on the location of the person holding the device. This facility requires a tie-up with a current cab service provider.

Emergency Contacting

If the blind person feels that he has lost his path or wanted the help of his relatives and known person. When the person presses the fourth switch, the latitude and longitude fetched by the GPS will be converted into a Google Maps link and can be sent to his relatives. The link can be used to see where the person holding the device is using Google maps. The same solution can be applied when a person feels unsafe. His location will be sent to the local authorities like police, ambulance according to his need. With the help of voice assistance, the user can hear audio signals via earphones attached stating the distance from the obstacle, its height, thus helping the user to pass the obstacle without any injuries. Python provides access to various APIs that can be used to convert text into speech. One of such APIs is the eSpeak API.

Fall Detection

The fall detection system is designed where the logic is based on the concept that when a person falls, the sudden change in the acceleration happens.

An algorithm must detect a fall in and a threshold is decided for detecting this sudden change . 

In a fall situation, there's an outsized change in acceleration within a blink of an eye, after the fall of the person, the person lay still for a few time, showing no change in orientation.

The data will be collected from the accelerometer then the acceleration magnitude is calculated. The three values ax, ay, and az are the acceleration of X-, Y-, and Z-axis respectively as per data collected.

The next task is to check whether this value crosses the lower and upper threshold. Also, it needs to verify that the threshold should be crossed in a particular time interval and there should be change in the orientation in that time interval. This should be checked in a continuous loop.

Results

In this section, we affirm the working of the technologies used in this process by rigorous testing and probing the project to extreme limits. All the data from various sensors and input sources were grouped and checked for the deviation from expected results. Fig. 8 represents the jacket and the ultrasonic sensors attached to numerous positions.

Fig. 8 Jacket and Ultrasonic Sensors

In Fig. 9 represents the upper part of the actual control box which is having four switches to choose various assistances and a strap is attached. HDMI port and earphone jack are also given on the control box in figure 9.

Figure 9. Control Box.

Table 1.Test results during detection of obstacle’s range.

Table 2 shows the output of the device when the second switch on the control box is triggered. When an obstacle less than the range where the upper sensor is attached is detected, that obstacle is considered as a short obstacle. Both sensors on the front side of the jacket will not receive any echo signal or only the bottom sensor will receive the echo signal in such a case. When both sensors receive the echo signal, that obstacle is considered as a long obstacle.

Table 2. Test results during detection of obstacle’s height.

In Fig. 10 shows the mail generated by the system sent to any Taxi service, containing the latitude and longitude of the person holding the device. The mail title will be the user id of the blind person and the body will contain the coordinates. The mail will be generated and sent by triggering the third switch of the device.

Figure 10. Mail Received by Taxi Service.Figure 11. Mail Received by Emergency Contact.

Fig. 11 shows the mail generated by the system sent to the relative of the person whose details are saved in the database. The mail body consists of a Google map link that can be used to see where the person holding the device is using Google maps. Once the link is clicked the location will get pointed on the Google maps. By clicking directions a path between the person and his relative will be created, which will help his relative to reach the blind person seeking help. The same solution can be applied to send the location to the local authorities like police; ambulance according to the user's need. The latitude and longitude generated by the Neo-6m GPS receiver was captured and compared with actual values of the latitude and longitude in Table 3. It was observed that the values generated by Neo-6m have 99% accuracy. All the numerous outputs are converted into text, and later on into audio using eSpeak API of python. The module was tested for English and Hindi language.

Table 3.Test results of Neo-6m.

The circuit for the fall detection system was successfully built shown in figure 13. The ESP8266 module was enabled in STA mode and connected to Wi-Fi for internet access. Once the fall is detected the Arduino triggers the NodeMCU which triggers the IFTTT API and SMS is sent to the emergency contact.

.

Figure 13. Interface of fall detection system.

Contribution and Conclusion

This research was associated with designing an assistance system for blind people who could help in real-time obstacle detection as well as real-time location access. All the features have been tested and implemented on practical grounds. The designed system was easy and convenient to use for the person. The features provided can give the person more senses to the surrounding environment. The blind person will be able to use it with a simple one-week training program. The performance of the system meets both the user’s needs and the engineer’s viewpoint and its specification for both hardware and software. It is more flexible and comfortable for the user to wear the device rather than carrying it.The manufacturing of the proposed project was cost-efficient as well as economical. The commercial version can be made even further more affordable and can be implemented at various levels. Despite the promising results of the experiment, there are several limitations of the system which are needed to be considered and addressed. The system is needed to be fully stabilized and optimized. With new technologies in the near future, further studies can be conducted to develop a more cost-effective and reliable version of the device. Since it is not acceptable to subject blind people to simulated falls, the evaluation of the detectors is severely limited.

Future Scope

The weight of the unit is around 5 Kg. The unit needs to be as light as possible. Enhancements can be made by replacing the single-board computer with an embedded system structure. Upon further research and testing, the prototype model can be made more accurate and efficient and can be manufactured on an industrial level. Features such as obstacle categorization and faster feedback can be implemented. The size of the setup could be made more compact and upon using industrial-grade components the results could tend to near accuracy. The distance received from sensors attached on various positions of the jacket will be saved in the CSV file regularly. These values will be compared to the pre-saved values and estimation would be made regarding the surrounding situation of the person. The system will track the situation of a person feeling trapped and not able to make any navigation decision. The system will trigger a message to the relative of the person saved in the database. The latitude and longitude values captured from the GPS receiver will be saved in JSON format and can be sent to the server for regular monitoring and tracking of the person. Fall detection system requires more testing in a real time environment and can be integrated with the control box.

Conflict of interest

The authors declare no potential conflict of interest regarding the publication of this work. In addition, the ethical issues including plagiarism, informed consent, misconduct, data fabrication and, or falsification, double publication and, or submission, and redundancy have been completely witnessed by the authors.

Funding

The author(s) received financial support for this research under KCIIS from A.K.T.U. Lucknow (UP) and Council of Science and Technology, Department of Science and Technology, Govt. of U.P.