Document Type : Research Paper
Authors
1 Department of Information Technology, Amity University Uttar Pradesh – Noida, India
2 Department of Information Technology, Amity University Uttar Pradesh – Noida, India.
Abstract
Keywords
Human Identification has proven to be a crucial part of numerous fields including authorization and forensics. With the advancements in technology, many open-source software and systems exist which can identify individuals with decent accuracy, but most of these systems rely on first-generation biometric modalities like facial recognition which are incompatible with the common CCTV cameras (recording in low-resolution) being used in almost every surveillance system. For the last few years, human identification using gait (i.e., an individual’s unique walking pattern/style) has been attracting all the researchers working in the fields such as surveillance and security, as gait-biometric is unobtrusive and can easily be extracted from low-resolution camera feeds. Existing systems working on gait analysis are yielding satisfactory results with some drawbacks, such as abnormalities in extraction (noisy silhouette) and exorbitant processing times.
The objective of this paper is to propose an automatic identification system that implements an artificially intelligent model which tracks and analyses an individual’s physical features while in motion from low-resolution camera feeds, keeping the speed-accuracy trade-off in mind. To achieve this, the low-resolution camera feeds are first pre-processed where the frame size is standardized and the clips are converted to grey scale, then using a series of image processing algorithms, the gait profile is extracted from the clips and saved in the form of numeric data. This data is then used to identify individuals using a trained AI (artificial intelligence) model.
The curiosity about gait as a biometric is strongly inspired by the want for an automated identification system for visible surveillance and tracking individuals. Currently, the deployment of gait as a biometric for identifying individuals in surveillance systems has attracted many researchers from the computer vision domain. The improvement in this research is being propelled by using improved extraction of features and optimized models. The suitability of gait recognition for surveillance systems emerges from the fact that gait can be perceived from a distance and that it is unobtrusive, i.e., non-invasive. Even though gait-biometric is still a brand-new concept, it overcomes almost all the issues that other biometrics suffer from including facial recognition, fingerprints, and iris recognition.
The pattern in which an individual’s limbs move while in locomotion is known as Gait. It can be majorly categorized into two types: natural and trained, where natural gait is the style in which people instinctively walk and trained is learned through training. The gait of an individual can depend on many features including personality, sociocultural factors, age, and mood, and is immensely influenced by the nervous, musculoskeletal, and cardiorespiratory systems.
As (Khan, Jain, and Sharma, 2022) have shown, numerous features collectively form an individual’s gait profile, including step, step-time, step width, stride, cadence, speed, etc.
Gait consists of numerous parameters:
Although gait analysis is the most recent biometric system, it is not a new concept in the field of study. Gait analysis has been explored and analyzed in numerous areas in the previous two decades. The Federal Bureau of Investigation (FBI) implemented gait analysis to narrow down their search on a list of suspects in the pipe-bomb investigation, on January 5th, 2021. In addition to biometrics, gait studies were pursued medical purposes, such as tracking relocations and as a screening tool.
The authors (Niyogi and Adelson,1994) presented the first gait identification system in 1994, which was built on a limited gait database. The Defense Advanced Research Projects Agency (DARPA) then created the famous HumanID initiative, which created the very first publicly available gait recognition database. Since then, several studies on gait recognition have been conducted. Most of the early gait recognition systems rely on video. Model-free and model-based techniques are the two types of video-based methodologies.
Gait recognition has several characteristics, such as unobtrusiveness, that alternative biometrics don’t. These characteristics are briefly explained below:
Gait characteristics are difficult to imitate. This is because human silhouettes and activities are commonly used in gait recognition. This effectively means that it is almost impossible for an induvial to hide or change their own gait. This property is critical in criminal investigations.
Literature Review
Table 1. Background work review
|
S. no. |
Title |
Dataset Used |
Methodology |
Limitations |
Accuracy obtained/Conclusion |
|
01. |
Biometric authorization system using gait biometry. |
NLPR |
Different gait components are extracted from the spatial, temporal and wavelet plot and Support Vector Machine (SVM) model is implemented. |
The model is only limited to videos in which the person is present in the side views. |
Maximum accuracy achieved was 98%. |
|
02. |
Gait recognition using gait gaussian image. |
Casia-B and Soton. |
A new approach has been proposed in which a new spatial-temporal based method known as Gait Gaussian Image (GGI) has been used for human recognition. |
It is only effective for normal walking conditions. |
The algorithm was used on both datasets, from which Soton database showed higher accuracy. |
|
03. |
Gait recognition using multiple projections. |
CMU, NLPR, USF. |
By using Principal Component Analysis (PCA), multiple projections of silhouette are analyzed from different databases. |
|
Silhouette projections can be used as a basic feature for classification. |
|
04. |
Gait based recognition of humans using continuous HMM’s. |
Little and Boyd’s, UMD, CMU. |
The width of the outer contour of the silhouette is used as the feature which is then continuously trained using HMM. |
It is not robust to drastic changes made due to illumination, change in clothes and angle view more than 10 degrees in the silhouette. |
It gave a satisfying result, no accuracy mentioned. |
|
05. |
Gait recognition using radon transform and linear discriminant analysis. |
USF, gait challenge database. |
A new method was used, called as Radon Transform of binary silhouettes in which each gait sequence is transformed for the computation of template. |
|
Using this method, a considerable improvement was seen as compared to those State-of-the-Art methods. |
|
06. |
A stochastic model of human gait dynamics. |
Gait maturation database. |
A stochastic model was proposed which can describe alterations in gait dynamics from childhood to adulthood. |
The model is unlikely to be accurate in mirroring anatomical structure. |
The model can extract different aspects of gait dynamics as they change with maturation. |
|
07. |
Gait biometrics |
USF |
Different methods like canonical space transform, Hidden Markov, etc. were used for the extraction of features and for the analysis of the gait for classification. |
|
Different approaches can be used for the classification. |
|
08. |
A lightweight attention-based CNN model for efficient gait recognition with wearable IMU sensors. |
cuhuGait and OU-ISR. |
A CNN model was proposed which has lightweight structure along with attention mechanism for gait recognition using wearable sensors. |
|
There was an improvement in the performance with smallest accuracy of 7.37% and the largest of 92.8% on F1 score. |
|
09. |
An appearance invariant gait recognition technique using dynamic gait features. |
SACV gait database, CASIA-B, OUISIK-B, TUM-IITKGB database. |
The paper proposes the use of cross-correlation strength and support vector machine for gait verification. |
The methods used weren’t efficient practically. |
98.5% accuracy was achieved using support vector machine. |
|
10. |
Human classification using gait features. |
Microsoft Kinect 3D data |
Gait characterization can be done by using Microsoft Kinect for the extraction of skeleton model which would further be trained on SVM. |
|
Accuracy of 96.25% was achieved. |
Methodology
To modelize the proposed approach, we used many techniques including data pre-processing, feature extraction, vectorization, etc. These methodologies are briefly explained below:
Several datasets exist such as the National Laboratory of Pattern Recognition dataset (NLPR), Carnegie Mellon University dataset (CMU), and Karadeniz Technical University dataset (KTU). But there is a common drawback to these datasets, that is, they are made with an emphasis on the physical features of the upper half of the human body. Additionally, many of such pre-existing datasets were limited in terms of noisy data and resolution. Thus, a custom dataset was required. A static camera position, to mimic the viewpoint of a CCTV camera (or any other surveillance camera) and plain background was used to set up our own data collection structure. 15 individuals walking laterally concerning the camera were recorded, where everyone was recorded 4 times with different view angles and camera positions. Out of the 60 video clips, 15 clips were used to generate the gait profile of each individual (i.e., the first clip for each individual generates their gait profile) while the remaining 3 clips for each individual are cross fed to generate the dataset. This cross-feeding approach works well when the problem is treated as a binary classification problem, and in turn, produces a larger training dataset
For example, out of the 15 individuals, if for individual X1, we recorded 4 walking captures V1, V2, V3, and V4, then V1 was used to generate the gait profile of X1, and V2, V3, and V4 were used to match the gait profile against the individual X1 as a positive match, and against some other individual X2 as a negative match. This way, a dataset of 15 individuals with four walking captures each can be used to make a dataset large enough to run training and tests.
Getting the data ready for feature extraction is the first step toward gait analysis. The clips can be of different aspect ratios and frame sizes, this is generalized by setting the frame size to a standard value. Next, the clips are read frame by frame and each frame is simultaneously converted to greyscale so that the digital image processing algorithms can extract the features efficiently.
Figure 1. Screenshot of original video clip
Figure 2. Frame converted to greyscale
The resized greyscale frames are then used to extract the numeric gait data by running a sequence of algorithms on the frames. This whole extraction process can be broken down into steps:
Background Subtraction and Thresholding
This step is the foundation of this sequence of algorithms as every other step relies on the output of background subtraction for an efficient result. While several inbuilt functions are available, such as Open CV functions (BackgroundSubtractor, BackgroundSubtractorMOG2, and BackgroundSubtractorKNN), the proposed model implements median frame subtraction for a clean result. At random, 50 frames are selected from the clip which is then used to calculate the median frame. This median frame is used to obtain the absolute difference concerning each frame in the clip. The absolute difference is then blurred using median blurring. Further, adaptive thresholding is implemented to obtain a binary image with just the foreground being highlighted and all the rest of the information being dropped.
Figure 3. Median Frame used for Background Subtraction
Figure 4. Absolute difference between median frame and individual frame
Figure 5. Frame after median blurring
Figure 6. Binary foreground image obtained after thresholding
To drop computational costs, and reduce time and space complexity, it is always preferred to process only the region of interest (ROI) rather than processing the whole frame. To obtain the ROI, a bounding box is placed using the contour of the human body, where firstly the whole frame is used to obtain a list of contours and then the largest contour (by area) is extracted and assumed to be that of the human body. Then a bounding box is placed on the contour, using the minimum and maximum values on the x and y axes of the largest contour.
This ROI boundary can also be used to filter the data extraction process to avoid extracting incomplete cycles. This is done by keeping a positive flag variable that gets a negative value as soon as the boundary of ROI touches the boundary of the frame. This flag variable is referenced when extracting the gait cycles and only the values where the flag variable was positive are computed.
Figure 7. Region of Interest.
Once a clean region-of-interest, containing the human body silhouette, is obtained, the outline of the silhouette is extracted for further computation. But, before the outline is used for extracting the projections, the direction of motion of the human in the frame needs to be standardized, this is important as when the person is going from left to right, the projections tend to be more right-leaning (because of the natural leaning of a human in the direction of walk) and likewise the projections are left-leaning when the motion is in right to left direction. To filter this, the ROI is flipped when the motion is detected to be in the right-to-left direction, converting all the data for movement in the left-to-right direction.
Figure 8. Extracting silhouette outline from left-to-right movement
Figure 9. Extracting silhouette outline from right-to-left movement.
The unidirectional outlines extracted in the previous step are then processed using a simple algorithm that returns a list of coordinates of the top and bottom projections of the silhouette. The top projection is extracted by calculating the distance between the top outline of the silhouette and the top border of the region of interest, i.e., the number of rows between the top border of ROI and the first pixel of the silhouette outline from above for each column. Similarly, the bottom projection is extracted by calculating the distance between the bottom outline of the silhouette and the bottom border of the ROI. Left and right projections don’t hold significant data when the subjects are walking laterally concerning the camera, hence are not included in this study. In Figure 10 below, the top projection is represented by red, and the bottom projection is represented by green.
Figure 10. Top Projections (red) and Bottom Projections (green).
The correlation of consecutive frames is calculated to measure the randomness in motion. This data is the core data for a gait profile. To measure the randomness component, Pandas function corr() is used, which takes the current and the consecutive frame as inputs and calculates the correlation of the two images. Similarly, it takes all the other frames of the clip and calculates the correlation for each consecutive pair of frames. This process is repeated two times, one for each top and bottom projections. The results obtained in this stage are two vectors representing the randomness component of the top and bottom projections of the human body. These arrays are then normalized to fit the standard range and used to extract the periodic gait cycle.
Figure 11. Normalized Correlation of consecutive frames
The human gait cycle is periodic. This property is very useful in terms of data collection. Generally, researchers split the complete gait cycle into two parts/half-cycles, this is practical as the first and the second half-cycles in a normal gait cycle are indistinguishable to a machine using only one camera, but this property can be further exploited to our benefit. The traditional method of cycle extraction describes the use of autocorrelation, but since we have one more periodic aspect in the frame (variations in the width of the ROI), the gait cycle can be divided into 4 parts/quarter-cycles (two quarter-cycles starting at the point where the ROI has a maximum width and ending with a minimum width and other two quarter-cycles starting with a minimum width and ending with maximum width). The complete gait cycle and the quarter cycles are depicted in Figure 12 below.
Figure 12. Gait Cycle.
Using the width of the ROI box, we can even filter out the incomplete gait cycles by marking the points where the subject being monitored leaves the frame. This is done by pointing out the abnormalities in the graph plot of the width of ROI (as when the subject leaves the frame, the width of the ROI jumps from the last value to the next value when he enters back in the frame, the smooth curve is broken, and a local maximum/minimum is achieved). This can be filtered easily. Figure 13 below shows the plots of autocorrelation and width of ROI, showing that the width plot captures the abnormalities of incomplete cycles much more efficiently than autocorrelation.
Figure 13. Width of ROI and Autocorrelation Plot.
The width plot is used to find all the maxima and minima, which are then processed to filter the incomplete cycles, Figure 15 below shows the quarter-cycles with the multi-colored lines, and the incomplete cycles are dropped where the line breaks.
Figure 14. Width plot highlighting the maxima and minima
Figure 15. Complete gait cycles (quarter-cycles) represented by the multi-colored lines
The projection vectors for the complete quarter-cycles are extracted and averaged, thus 4 curves are obtained, one for each projection, and this comprises of the gait profile that will be saved as a 4-feature value in the database.
Figure 16. Final Gait Profile of an individual (Correlation Projections)
The gait profile extracted is saved in the database after converting it to numeric data. This data is read from the database where unique identification numbers are used to index the profiles. When a new video is analyzed, the ‘UID’ is required as an input, this number points to a gait profile saved in the database, against which the gait extracted from the test video is analyzed. Both the gait profiles are displayed for graphical comparison and finally a textual output with the prediction, “Positive”/”Negative” match label, and other metrics and information is displayed.
Since identification problems have multiple subjects to identify, hence these problems are usually categorized as multi-class problems. This approach is reasonable but not computationally very efficient, as it will automatically result in the dynamic number of classes (one new class for each person who is added to the database). This is avoided by using a binary classification approach, where the model is trained to recognize the difference between a positive match and a negative match. Hence, there are only two classes, 1 for positive matches and 0 for negative matches.
Results
The model can successfully identify individuals using gait profiles, which are extracted from low-resolution video clips, and it can do so with remarkable speeds (in a range of 20 to 35 Hz depending on the hardware specifications of the machine).
Let’s consider two test cases, first test case A where the person in the test video is the same person whose ‘UID’ is entered for the database profile, and another test case B where the person is the test video is not the same person and is being analyzed against someone else’s gait profile. The outputs for both test cases are given below in Figure 17-20.
Figure 17. Graphical Output for Test Case A
Figure 18. Metrics and Final Predictions for Test Case A
Figure 19. Graphical Output for Test Case B
Figure 20. Metrics and Final Predictions for Test Case B
In the test cases, textual outputs in the figures, Figure 17, and Figure 19 shows that the average processing speed was 34 FPS when the Nvidia CUDA toolkit was used and 22 FPS when standard toolkit and processing systems were used.
Further, this proposed artificial neural network was compared with other machine learning models (including logistic regressor, decision tree classifier and random forest classifier) and their Receiver Operating Characteristic (ROC) curves are plot and the area under the curves are calculated. Figure 21 below shows the ROC plots, and it is visible that although the proposed model outperforms the other models, machine learning models are also performing decently as the input features were extracted efficiently and is a numeric fashion.
Figure 21. ROC Plot
Conclusion
The objective of the paper, i.e., “Human Identification using Gait Analysis and AI” is achieved, and an automatic identification system based on gait analysis is proposed. The applications of this research in the domain of security, surveillance, and authorization are studied and a conclusion is drawn that gait as biometric is not only feasible, but it is also the next practical upgrade in the surveillance and security domain.
The existing models capable of analyzing gait achieve decent accuracies but at the cost of high processing times. The proposed model maintains the speed/accuracy trade-off, where it processes the frames at a speed of 20-35 Hz and achieves more than 85% accuracy in average cases, with as high as 99% accuracy in ideal cases.
The major breakthroughs in the proposed model are achieved based on two optimizations, firstly implementing algorithm optimizations, and using multi-threading, and secondly dividing the gait cycle into 4 quarter-cycles rather than 2 half-cycles. This facilitates efficient gait extraction and most of the data is utilized, without data loss.
Conflict of interest
The authors declare no potential conflict of interest regarding the publication of this work. In addition, the ethical issues including plagiarism, informed consent, misconduct, data fabrication and, or falsification, double publication and, or submission, and redundancy have been completely witnessed by the authors.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
)