research paper on gesture technology

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 12 December 2022

Exploiting domain transformation and deep learning for hand gesture recognition using a low-cost dataglove

• Md. Ahasan Atick Faisal 1   na1 ,
• Farhan Fuad Abir 1   na1 ,
• Mosabber Uddin Ahmed 1 &
• Md Atiqur Rahman Ahad 2  

Scientific Reports volume  12 , Article number:  21446 ( 2022 ) Cite this article

7009 Accesses

16 Citations

10 Altmetric

Metrics details

• Biomedical engineering
• Electrical and electronic engineering
• Information technology

Hand gesture recognition is one of the most widely explored areas under the human–computer interaction domain. Although various modalities of hand gesture recognition have been explored in the last three decades, in recent years, due to the availability of hardware and deep learning algorithms, hand gesture recognition research has attained renewed momentum. In this paper, we evaluate the effectiveness of a low-cost dataglove for classifying hand gestures in the light of deep learning. We have developed a cost-effective dataglove using five flex sensors, an inertial measurement unit, and a powerful microcontroller for onboard processing and wireless connectivity. We have collected data from 25 subjects for 24 static and 16 dynamic American sign language gestures for validating our system. Moreover, we proposed a novel Spatial Projection Image-based technique for dynamic hand gesture recognition. We also explored a parallel-path neural network architecture for handling multimodal data more effectively. Our method produced an F1-score of 82.19% for static gestures and 97.35% for dynamic gestures from a leave-one-out-cross-validation approach. Overall, this study demonstrates the promising performance of a generalized hand gesture recognition technique in hand gesture recognition. The dataset used in this work has been made publicly available.

Similar content being viewed by others

A high-density 1,024-channel probe for brain-wide recordings in non-human primates

SuperAnimal pretrained pose estimation models for behavioral analysis

Transfer learned deep feature based crack detection using support vector machine: a comparative study

Introduction.

From the dawn of human civilization, communication between humans has been the single most important trait for our survival. At the same time, it created the social attributes among us which had been modified over the centuries and transformed us into civilized beings. However, the first mode of communication was not a structured vocal language but involved gestures, often using hands. Later, with the progress of civilization, people adopted structured languages and used hand gesture-based communication in special cases. Recent researchers have found around 6700 spoken languages 1 and several hundred sign languages 2 , although a good number of them are not currently in use. Despite having structured vocal languages in every country, sign languages are still used primarily for communication with the deaf and hard-of-hearing community. However, since non-signers are not generally familiar with these sign languages, deaf and hard-of-hearing people face communication barriers. On the other hand, with the technological development in sensor technologies, embedded systems, camera technologies, and efficient learning systems, hand gesture recognition research has found efficient and pragmatic solutions to the communication problem.

Hand gestures and their use vary greatly depending on the field of application. Apart from sign-language communication, several other tasks, namely military coordination, interaction with digital devices, and virtual gaming consoles involve hand gestures. Based on hand motion, hand gestures can be divided into two types—static hand gestures and dynamic ones. Moreover, based on the application, one or both hands can be involved to complete a sign. Over the year, researchers have been trying to develop several technologies to use human hand gestures to communicate with the cyber world. Hence, the task of hand gesture recognition has been one of the most widely explored areas in the research domain of Human–Computer Interaction (HCI). The detection systems can be classified into two categories—contactless detection systems where the detecting device is kept at a distance from the hand and are not in any sort of contact, and wearable detection systems that are often implemented with several sensors at close contact with the hand 3 .

The researchers have explored contactless hand gesture recognition systems using several modalities, namely Radio Frequency (RF), ultrasound, and Computer Vision (CV). Google’s Project Soli is a 60 GHz millimeter-wave radar on-chip that can detect fine-grained hand gestures along with micro-finger movement 4 . Wang et al. used this chip and employed a deep learning algorithm to detect 11 dynamic hand gestures from 10 subjects 5 . However, this chip is not usable for detecting gestures at a meter-long distance. In this regard, WiFi has been used as a ubiquitous modality for detecting hand gestures from a greater distance than the Soli chip 6 , 7 , 8 ; however, it fails in the precision of detection 3 . On the other hand, several studies have discovered the potential of ultrasound for detecting hand gestures with a clear line of sight 9 , 10 , 11 . Although in recent years, these RF-based and sound-based modalities have been improved in performance and reliability in particular applications, they are still not dependable in a regular use case where the environmental parameters vary frequently 3 .

Owing to the tremendous development in Artificial Intelligence (AI) and camera technology, computer vision-based gesture detection systems are the most widely explored field of research in recent years. Although fundamentally employing computer vision modality, hand tracking and gesture recognition can be achieved in a variety of techniques namely, skin color detection 12 , 13 , 14 , appearance detection 15 , 16 , 17 , 18 , motion-based detection 19 , 20 , skeleton-based detection 21 , 22 , and depth detection 23 , 24 , 25 , 26 . Apart from the conventional RGB camera, IR-based leap motion controller 27 , 28 and Microsoft Kinect depth camera 26 , 29 , 30 , 31 are two of the most widely used hardware for depth and skeletal information detection. However, the primary shortcomings of these methods are the environmental dynamics, namely lighting conditions, line of sight, and detector proximity. Although depth-based and skeleton-based approaches have become more robust over the year, such as the MediaPipe by Google 32 , they still have not overcome those shortcomings completely. Moreover, due to the cost of the high-quality depth sensor, the usability of such systems is rather still limited.

On the other hand, sensor-based wearable datagloves are one of the most widely used contact-based hand gesture recognition systems that have overcome most of the shortcomings of contactless detection methods. VPL Inc. first introduced a commercial sensor-based dataglove back in 1987 33 . The researcher of this dataglove invented optical flex sensors which enabled them to track finger flexion. Despite inventing the technology at such an early age of HCI, these datagloves were not widely adoptable due to the high cost and lack of feasibility in regular use. In the last decade, owing to the development of low-cost high-performance sensors, processing devices, connectivity, and algorithms, researchers have explored new avenues of sensor-based hand gesture recognition.

Over the year, a wide range of both commercially available and custom-made sensors are used on datagloves for accurately capturing hand gesture dynamics. Several studies have explored surface electromyography (sEMG) sensors to capture the electrical activity inside the hand muscles during gesture performance 34 , 35 , 36 , 37 , 38 , 39 , 40 . Moreover, Wang et al. 41 , Abreu et al. 42 , and Su et al. 43 used variants of the Myo band which are commercial sEMG armbands and are specifically designed to track hand gestures. Although sEMG shows reliable performance in a wide range of gestures including sign languages, the detection algorithms were subjected to the sensor placement and the signers. Moreover, several studies have used resistive flex sensors and their variants for actively tracking finger flexions 44 , 45 , 46 and an Inertial Measurement Unit (IMU) to detect the hand movements 40 , 44 , 45 , 47 , 48 . Wen et al. developed a smart glove with 15 triboelectric nanogenerator (TENG)-based sensors for tracking 50 American Sign Language (ASL) words and 20 sentences 49 . Furthermore, using a fusion of multiple sensors has shown greater performance in several studies than a single sensor method 40 , 41 , 43 , 45 , 46 , 50 .

In our previous study, we presented a dataglove with five flex sensors and one IMU and evaluated its performance in a limited number of gestures and subjects 44 . In this work, we adopted the same sensor fusion configuration and developed and combined it with state-of-the-art deep learning techniques. We proposed a Spatial Projection Image based deep learning technique for dynamic hand gesture recognition and parallel-path neural network architecture for multimodal sensor data analysis. The system successfully recognizes 40 words from the ASL dictionary, including 24 static and 16 dynamic signs collected from 25 subjects. In a nutshell, the key contributions of this work are as follows:

We constructed a low-cost wireless capable dataglove combining flex sensors and IMU and explored state-of-the-art deep learning techniques on it.

We provided a large dataset of 40 ASL letters and words collected from 25 subjects using the proposed dataglove.

We introduced Spatial Projection to convert the 1D time-series signals into 2D images for dynamic gesture recognition which outperformed 1D CNN and classical machine learning-based approaches.

The proposed parallel-path neural network architecture showed superior feature extraction capability from multimodal data over conventional architectures.

Methods and materials

Hardware configuration.

The primary hardware is a dataglove consisting of three units, namely sensing, processing, and onboard power regulation unit. The sensing unit is comprised of five 2.2" flex sensors (SEN-10264) and an IMU (MPU-6050) which has a triaxial accelerometer and a triaxial gyroscope. The overall hardware configuration is illustrated in Fig.  1 .

The dataglove architecture: On the left, we have the glove with all the mounted sensors and electronics. A flex sensor is shown in the top right corner. The components of the main controller board are shown in the bottom right corner. It consists of an ESP32 microcontroller, an MPU-6050 IMU, and some complementary electronics.

Sensing unit

The flex sensors are, in fact, variable resistors with flat resistance of \(25\;{\text{K}}\Omega \;\left( { \pm \;30\% } \right)\) , which are placed above the five fingers of the dataglove using fabric pockets to sense the fingers’ flex. A voltage divider was created with each flex sensor and a \(0.25 \;{\text{W}}\;100\) KΩ ( \(\pm \;5\% )\) resistor was used to convert the resistance difference during the finger flexion to the voltage difference across the sensor using the processing unit 51 .

The accelerometer and gyroscope of the IMU are configured to track the linear acceleration within \(\pm \;19.6 \;{\text{ms}}^{ - 2}\) and angular velocity within \(\pm \;4.36\;{\text{rad}}\;{\text{s}}^{ - 1}\) , respectively, which is well within the range of any human hand motion. Moreover, the IMU contains a Digital Motion Processor (DMP) which can derive the quaternions in-chip from the accelerometers and gyroscope data and thus, provides the hand orientation data along with the motion information 52 .

Processing unit

The processing unit is a WiFi-enabled development module called DOIT ESP32 Devkit V1 that has a Tensilica Xtensa LX microprocessor with a maximum clock frequency of \(240\;{\text{MHz}}\) . The 12–bit analog to digital converter (ADC) with 200-kilo samples per second maximum sampling rate is capable of sampling the flex sensors’ analog data with sufficient resolution. Moreover, the module is capable of communicating with external computers via USB which enables wired data communication 53 .

Onboard power regulation

The ESP32 module and the IMU have an operating voltage of \(3.3\;{\text{V}}\) 52 , 53 . On the other hand, the flex sensors do not have a strict operating voltage 51 . Hence, we used an LM1117 low-dropout (LDO) 3.3 V linear voltage regulator to regulate the supply voltage from the \(3.7\;{\text{V}}\) single cell LiPo battery. Moreover, we used \(10\) μF and \(100\) μF filtering capacitors to filter out the supply noise.

We explored 40 signs from the standard ASL dictionary that including 26 letters and 14 words. Among these signs, 24 require only a certain finger flexion and no hand motion; hence, are addressed as static signs or gestures. Conversely, the remaining 16 signs need hand motion alongside finger flexion to portray meaningful expression according to the ASL dictionary. Moreover, we collected the signs from 25 subjects (19 Male and 6 Female) in separate data recording sessions with a consistent protocol. Overall, three channels for acceleration in both body and earth axis, three for angular velocity, four for quaternion, and five for flex sensors were recorded in the dataset.

The data was recorded by the dataglove processing unit which was connected to a laptop for data storage via USB. The sampling frequency is set to 100 Hz and each gesture was repeated 10 times to record the performance variabilities of each subject. However, during a few sessions denoted in the dataset supplementary information, the laptop charger was connected which resulted in AC-induced noise all over those specific recorded data.

Data recording protocol

Before starting the recording process, each subject signed an approval form for the usage of their data in this research and was briefed about the data recording steps. As the subjects were not familiar with the signs before the study, they were taught each sign before the data recording via online video materials 54 . The data was recorded by the dataglove and stored on the laptop at the same time. Hence, a Python script was used on the laptop to make the handshake between the two devices and to store the data in separate folders as per the signs and the subjects.

At the beginning of each data recording session, the subjects were prompted to declare their subject id and the gesture name. Afterward, a five-second countdown is prompted on the laptop screen for preparation. Each instance of the gesture data is recorded for a 1.5 s window and the subjects can easily perform their gesture once within that window. In a single gesture recording session, this process is repeated 10 times. The gesture recording flow for each session is shown in Fig.  2 . All methods were carried out following the relevant guidelines, and the correctness of gestures was evaluated by visual inspection. All experimental protocols were approved by the University of Dhaka, Dhaka, Bangladesh. Note that informed consent was obtained from all subjects.

The flowchart showing the data collection protocol. The diagram shows all the different steps of the data collection process. This protocol was followed during the data collection for all the subjects.

Data preprocessing

Gravity compensation.

The triaxial accelerometer of the IMU body records acceleration, which is subjected to gravity. Hence, the gravity component has to be adjusted from the recorded raw acceleration to interpret the actual motion characteristics of the dataglove. The gravity vector can be derived from the orientation of the dataglove. Quaternions express the 3d orientation of an object which is a robust alternative to the Euler angles which are often affected by gimbal-lock 55 . The digital motion processor (DMP) of the MPU-6050 processes the raw acceleration and angular velocity internally and produces quaternion. The quaternions can be expressed by Eq. ( 1 ).

where \({\varvec{Q}}\) stands for a quaternion that contains a scaler, \({q}_{w}\) and a vector, \(\mathbf{q}\left({q}_{x},{q}_{y},{q}_{z}\right)\) . The overall gravity compensation process is described in Eqs. ( 2 ) and ( 3 ) 56 .

where \({\varvec{g}}\left({g}_{x},{g}_{y},{g}_{z}\right)\) , \({\varvec{Q}}\left({q}_{w}, {q}_{x},{q}_{y},{q}_{z}\right),\) \({\varvec{l}}{\varvec{a}}\left({la}_{x},{la}_{y},{la}_{z}\right)\) , and \({\varvec{a}}\left({a}_{x},{a}_{y},{a}_{z}\right)\) denotes the gravity vector, quaternion, linear acceleration vector, and raw acceleration vector, respectively. The resultant linear acceleration ( \({\varvec{l}}{\varvec{a}}\) ) represents the body axis acceleration which is compensated for the gravity offset. This step was done in the processing unit of the dataglove.

Axis rotation

The recorded raw acceleration and the gravity-compensated linear acceleration both were in the body axis of the dataglove and the body axis is dependent on the initial orientation of the dataglove when it powers up. However, this nature of axis dependency on the initial orientation is problematic for real-world applications. Hence, we converted the triaxial acceleration vector from the body axis to the North-East-Down (NED) coordinate system which follows the directions based on the earth itself 57 . At first, a rotation matrix was calculated using the quaternions. Afterward, the NED linear acceleration is derived using matrix multiplication between the rotation matrix and the body axis linear acceleration. Equations ( 4 ) and ( 5 ) show this axis transformation process using quaternions 58 .

where \(\mathbf{R}\) , \({\varvec{Q}}\left({q}_{w}, {q}_{x},{q}_{y},{q}_{z}\right)\) , \({\varvec{L}}{\varvec{A}}\left({LA}_{x},{LA}_{y},{LA}_{z}\right)\) , and \({\varvec{l}}{\varvec{a}}\left({la}_{x},{la}_{y},{la}_{z}\right)\) stands for the rotation matrix, quaternion, NED linear acceleration, and the body axis linear acceleration, respectively. Similar to the previous step, this axis transformation is also done in the processing unit of the dataglove. Figure  3 illustrates the axial diagram of the dataglove and the axis rotation.

The IMU orientation diagram: On left, we have the X, Y, and Z coordinates of the MPU-6050. Along these 3 axes, the accelerometer and gyroscope values are recorded. The figure on the right shows the body axis to earth axis conversion diagram.

Rolling filters

After closer inspection, we found a few random spikes in the IMU data. Hence, firstly, we removed using a rolling median filter of 10 data points to get rid of such spikes. After the spike removal, secondly, we used an extra step of applying moving average filters for the only specific sessions where the recordings were subjected to AC-induced noise which resulted in comparable waveforms for all data recordings. The implementation of the moving average filter is shown in Eq. ( 6 ) 59 :

where \(x\left[n\right]\) is the input signal, \(N\) stands for the number of data points, and \(y\left[n\right]\) denotes the output signal. However, after applying the rolling average there were a few null values at the end of each signal frame which were replaced by the nearest values in that signal. According to the data recording protocol, the gestures were performed in the middle of each 1.5-s window. Hence, replacing the few terminal data points with the nearest available valid data point does not change the signal morphology. Lastly, we used another level of rolling average filter of 10 data points, this time for the whole dataset, to further smooth the signal and also replaced the terminal null values with the nearest valid data point in each frame.

Normalization

The processed acceleration and flex sensor data are not in the same range. Hence, before employing the AI-based classification technique, data normalization is widely practiced for better convergence of the loss function 60 . We used min–max scaling as the normalization technique with a range of \(\left[ {0,1} \right]\) . It is shown in Eq. ( 7 ) 61 :

where \(x\) is the input and \({x}_{normalized}\) is the normalized output. \({x}_{\mathrm{max}}\) and \({x}_{\mathrm{min}}\) respectively denote the maximum and minimum values of the input.

Spatial projection images generation

There are several challenges associated with dynamic sign language recognition. In our case, the temporal dependency and the size of the hand were the most challenging issues. A signer can perform a sign at many different speeds. Moreover, the speed does not match up from signer to signer. To successfully recognize signs from all the subjects, first, this temporal dependency needs to be removed from the signals. The second challenge was the hand size of the signer which introduced variability in the gestures performed by different signers. In the proposed method, we tried to eliminate these two issues by utilizing the Spatial Projection Images of the dynamic gestures. However, the static gestures do not generate a meaningful pattern in the projections due to their stationary nature. Hence, this step is omitted for static signs.

When interpreting a sign, the speed of performing the sign and the signer's hand size does not matter. The spatial pattern created by the motion of the signer’s hand defines the sign. As long as the pattern is correct, the sign will be considered valid regardless of its temporal and spatial states. To capture this pattern of sign language gestures we utilized the accelerometer sensor data from our device. Using Eqs. ( 8 – 9 ), we converted the 3D acceleration into 3D displacement vectors. These vectors represent the path followed by the hand in 3D space during the performance of the gesture.

These 3D displacement vectors were then projected onto the XY, YZ, and ZX 2D planes. If the vectors are projected onto these planes for the entire timeframe of the sign, the projections form a 2D path that captures the pattern of the sign in the 3 planes as shown in Fig.  4 . No matter at which speed the gesture was performed, these 2D projections of the gesture always provide similar patterns. Hence the temporal dependency is eliminated in this process.

Spatial projection generation process. We start with the 3-axis acceleration and then convert them into 3-axis displacement vectors. These vectors are projected onto the 2D spatial planes to generate the projection images.

After capturing the pattern of a particular gesture, we normalize the projections using the maximum and minimum values along axes. In this way, the projection from different signers results in a pattern that is similar regardless of their hand size.

The projections were generated using the Python Matplotlib 62 library where the components of the displacement were calculated along the 3 axes and they were plotted 2 at a time for the three-axis planes (XY, YZ, and ZX). We used the line plot for this with the “linewidth” parameter set to 7 and the color of the line set to black. This resulted in 3 grayscale images for the 3 projection planes for each gesture. The images were then resized to 224 × 224 pixels dimensions and we used these images for the input of our proposed model.

The proposed architecture

In this section, we present the network architecture of our proposed framework (Fig.  5 ). We have used two variations of the architecture for static and dynamic signs.

Proposed network architectures: ( a ) Overall diagram of the proposed architecture. For static gestures, the sensor channels are processed by parallel 1D ConvNet blocks. For dynamic gestures, the accelerations are first converted into spatial projection images and features are extracted from them using the pre-trained MobileNetV2 network, ( b ) the architecture of the 1D ConvNet Blocks, and ( c ) the architecture of MobileNetV2.

Architecture for static gestures

As mentioned in the Data Preprocessing subsection, Spatial Projection Images are not used for static gestures. The normalized time series channels are passed to separate 1D ConvNet blocks to produce embeddings. These embeddings are afterward concatenated in a fully connected layer which in turn, makes the prediction. Figure  5 a shows the stacked 1D ConvNet block architecture for static gesture detection.

Architecture for dynamic gestures

We have utilized two different types of signals for the input to our model. First, we have the 3 spatial projection images generated from the acceleration data. Then we also have the 1D time-series signals from the flex sensors. So, in total, we have 8 channels of input data with 3 image channels and 5 time-series signal channels. Each of these channels was processed using separate ConvNet blocks to produce the embeddings from that particular channel. For the static gestures, the 8 time-series signals were processed using the parallel path ConvNet architecture shown in Fig.  5 b. On the other hand, the projection images were processed by a 2D ConvNet architecture (MobileNetV2 63 ) as shown in Fig.  5 c. The architectural details of these two ConvNet blocks are discussed below.

1D ConvNet block

The 1D ConvNet blocks are composed of 4 convolution layers. Each pair of convolution layers is followed by a BatchNormalization layer and a MaxPooling layer. The kernel size used in the convolution layers was set to 3, the stride was set to 1 and the padding was set to 1. The MaxPooling kernel size was set to 2 and the ReLU activation function was used. After the 4 convolution layers, the fully-connected layer with 50 neurons was used to extract the embeddings.

2D ConvNet block

The 2D ConvNet blocks are constructed using the MobileNetV2 64 architecture. MobileNet is an efficient architecture for mobile and embedded vision applications. It utilizes depthwise separable convolutions 65 to significantly reduce the computational burden compared to regular convolution. In depthwise separable convolution, each of the channels is processed with the convolution filters separately and the resultants are combined using a 1 × 1 pointwise convolution. This is known as factorization and it drastically reduces the computation and model size.

The MobileNetV2 63 is the result of the improvements done to the regular MobileNet architecture. It uses an inverted residual structure 66 where the skip connections are between the thin bottleneck layers which improves the performance compared to the classical structure. The MobileNetV2 architecture starts with a regular convolution layer with 32 filters followed by 19 residual bottleneck layers. The kernel size was set to 3 × 3 and ReLU6 64 was used as the activation function.

We used the Tensorflow 67 Python library to implement the proposed network. For the loss function, we used the Sparse Categorical Cross-Entropy loss. The loss was minimized using the Adam 68 optimizer with a learning rate of 0.0001. The network was trained for a maximum of 300 epochs with an early stopping criterion set on the validation loss with a tolerance of 30 epochs.

Ethical approval

We took written consent from all the subjects participating in the data collection process. It was mentioned in the consent form that the data will only be used for research purposes. Moreover, the dataset does not contain any personal information of the subjects but their sex and age information.

Evaluation criteria

Evaluation metrics.

To evaluate our architecture for the static and dynamic gestures, we adopted four evaluation criteria, namely macro-averaged precision, macro-averaged recall, macro-averaged F1, and accuracy which are described in Eqs. ( 10 – 16 ).

where \(TP\) , \(FP\) , and \(FN\) denote true positive, false positive, and false negative, respectively. Moreover, the \(i\) indicates the particular gesture or subject and \(N\) stands for the total number of that gesture or subject. For evaluating per-gesture performance we have used the per-class precision, recall, and F1-score, and for overall reporting, we adopted the macro-average method.

Validation method

There are several validation techniques used for evaluating a machine-learning (ML) model. Among these techniques, we have used the leave-one-out-cross-validation (LOOCV) method to determine the performance of the architecture. LOOCV is regarded as one of the most challenging validation techniques because for each training and evaluation session, the model is exposed to a single unseen subject’s data. Hence, if that particular subject’s data contains significant variation from other subjects in the training set, the resultant matrices are heavily penalized. Increasing the number of subjects in the training set also increases the chance of having more representative data in the test set.

However, our rationales behind using the LOOCV technique are to challenge the generalization of our trained model and test the model’s capability on unseen subject data. Here, we have separated one subject from the dataset as the test set and used the rest of the subject data as the training set. Thus, we repeated the process for all 25 subjects and evaluated the overall results at last.

Experiments

Baseline methods.

Since we have used a custom-made dataglove for this study and our dataset has not been benchmarked before, two classical ML and one deep learning model are employed to generate the overall result. These two classical ML algorithms provided the top performance for our previous study with the same dataglove. Moreover, 1D CNN is one of the most widely used deep learning algorithms with time-series data. Wen et al. 49 used this architecture as the AI algorithm for their study. Hence, we chose these methods for the baseline determination. Table 1 shows the results of these baseline methods for both static and dynamic gestures.

Performance evaluation of the proposed method

We have evaluated the proposed architecture for static and dynamic gestures separately. The confusion matrices illustrated in Fig.  6 projects the performance evaluation for each class. Moreover, Table 2 presents the evaluation metrics for each gesture per gesture category, and Table 3 shows the overall metrics for static and dynamic gestures.

Confusion matrices: ( a ) confusion matrix for the static signs; ( b ) confusion matrix for the dynamic signs.

Static gestsures

In the proposed architecture, we used individual 1D ConvNet blocks for each channel of the flex and IMU to produce embeddings. The flex sensors capture the finger movements whereas the orientation can be interpreted from the acceleration. The confusion matrix in Fig.  6 a shows the majority of the detection at the diagonal with a few misclassifications. Among the 24 static gestures, 14 were classified with F1-scores over 0.8, two (k, x) had F1-scores between 0.7 and 0.8, and the F1-scores dropped below 0.7 for seven static gestures (c, e, o, s, t, u, v).

According to Fig.  7 c and o are very similar to each other in gesture shape and hand orientation 69 . The only difference is the position of the thumb with respect to the other four fingers, which touch each other during o but remain separate during c. The use of a contact sensor on the tip of the thumb might improve this classification.

The standard ASL signs for letters and numbers 69 .

Moreover, u and v have similar finger flexion and orientation. The only subtle difference between these two gestures is that the index touches the middle finger during u but does not do so during v . A contact sensor between these two fingers might improve the detection ability of the model.

Based on Fig.  7 , we found similarities between e and s as well. While the thumb is kept below the other fingertips during e , it remains on top of the fingers like a fist during s . Although the flexion of the four fingers is a bit different, the subtle differences in the flex sensor data are not learned by the model.

Lastly, the performance of t is one of the most complex ones using a dataglove where the gesture is performed with the thumb kept in between the index and the middle fingers. The finger flexion is similar in x as well. Moreover, for some subjects, the index finger was not bent enough which resulted in a similar flexion as d . Therefore, the model sometimes misclassified t with x and d .

Among the 0.7–0.8 F1-score range, the model falsely predicted x as t and k as p in a few cases. This is also due to the similarities between the gestures.

Dynamic gestures

Compared to the static gestures, our model performed significantly well for the dynamic gestures with an F1-score ranging from perfect 1 for please , to 0.9295 for hello . Although the gesture hello is significantly different from sorry or yes , according to the confusion matrix there were some misclassifications between these classes (Fig.  8 demonstrates the differences among these 3 classes). However, since we used the LOOCV technique to generate these results, the subject-induced bias in one gesture might affect the validation for a different gesture performed by another subject.

Differences among ( a ) ‘hello’, ( b ) ‘sorry’, and ( c ) ‘yes’ gestures.

Comparison with previous works

Based on our literature review, we showed different sensor-based gesture recognition works from 2016 in Table 4 for ease of comparison.

According to the comparison, several studies show better accuracy compared to this work. However, the number of volunteers, number of gestures, and validation method are not the same in all these studies. Moreover, due to the mode of our experiments and system, we are unable to compare our method with other systems. For example, among these works, Wen et al. 49 , Lee et al. 45 , and Abhishek et al. 72 did not provide enough information in their manuscripts regarding the number of volunteers in their dataset. Although other works have mentioned the number of users, most of them, for example, Su et al. 43 , did not consider user-independent performance. In practice, AI-based models show discrepancies in their performances on new subjects, making the user-specific metric unreliable.

However, Wen et al., 2021 49 , Lee et al. 45 , and Saquib et al. 70 customized their dataglove with sensor placements at some specific points to detect the touch at the fingertips. Such sensor placements have improved the detection capability of some specific ASL alphabets. In this work, we proposed a generalized hand gesture recognition system and used ASL signs only for validation. On the other hand, such ASL-specific systems in the abovementioned studies might not show similar performance in other application domains.

Moreover, the number of gestures, number of subjects, and gesture type are three significant parameters for the performance comparison. For example, in our previous work 44 , we used K-nearest neighbors (KNN) with the same dataglove which resulted in an accuracy of 99.53% for static and 98.64% for dynamic gestures. However, that study included only 14 static and 3 dynamic gestures collected from in total of 35 volunteers. However, the gestures chosen for the study were very distinct from each other compared to the ones we used in this study.

The comparison among several systems cannot be done based on only the accuracies of the systems. Based on the gesture type, number of gestures, number of volunteers, application, and validation method, this study presented a more robust and economic hand gesture recognition solution compared to the other works in recent years.

Limitations

Domain-specific improvement.

Each application of hand gesture recognition is different. Hence, some domain-dependent limitations are encountered in the model’s performance for a few classes which might vary for different sign language dictionaries. In this particular application, contact sensors are required at the tip of the thumb and between the index finger and the middle finger for performance improvement.

Limitation in everyday use

Although made using low-cost commercially available sensors and modules, the dataglove is not feasible for everyday outdoor use which limits the use of such systems in particular domains.

Applications

Video conference.

Due to the COVID-19 pandemic, the use of video conferences has increased in a steep curve. However, for the deaf and hard-of-hearing community, access to these video conferences is a challenge, since some platforms might not have a real-time computer vision-based sign interpreter. In this case, an accessibility software using our dataglove and proposed AI-based gesture detection system might open new avenues for the deaf and hard-of-hearing community.

Robot control

One of the primary applications of hand gesture recognition is controlling a remote cyber body, namely a robot using hand commands. Due to the promising performance of our dataglove and the detection algorithm, it can be a promising low-cost solution for a wide range of robot control applications.

Virtual reality

Nowadays, virtual reality (VR) devices are within our reach and with the announcement of Meta Verse, new avenues of VR technology have been opened. In this regard, the fundamental necessity of communicating with the cyber world is still done using wearable dataglove-based hand gestures. Our proposed dataglove can be used in conjunction with the VR headset as well.

In this paper, we developed a dataglove to detect static and dynamic hand gestures and presented a novel deep learning-based to make predictions. To validate the system, we constructed a dataset of 40 ASL signs, including 24 static signs and 16 dynamic ones, from 25 subjects. For static gestures, after data filtering, we compensated the gravity from the acceleration and converted it from the body axis to the earth axis. In the case of dynamic gestures, we generated Spatial Projection Images from 1D time series acceleration data. We also introduced a parallel path neural network architecture to extract features from different sensor channels more efficiently. Our method produced better results than classical ML and CNN-based methods for both static and dynamic gestures. The achieved results are extremely promising for various applications.

In future work, we will employ our method on several applications and create a larger dataset to explore further. Moreover, by employing a multimodal technique, we can include videos with the sensor data to accumulate additional features.

Data availability

The datasets analyzed during the current study are available in Figshare 73 ( https://figshare.com/articles/dataset/ASL-Sensor-Dataglove-Dataset_zip/20031017 ).

Comrie, B. Languages of the world. In The Handbook of Linguistics (eds Aronoff, M. & Rees-Miller, J.) 21–38 (Wiley, 2017).

Chapter   Google Scholar  

Zeshan, U. & Palfreyman, N. Typology of sign languages. Camb. Handb. Linguist. Typology 1–33 (2017).

Abir, F. F., Faisal, M. A. A., Shahid, O. & Ahmed, M. U. Contactless human activity analysis: An overview of different modalities. In Contactless Human Activity Analysis (eds Ahad, M. A. R. et al. ) 83–112 (Springer International Publishing, 2021).

Lien, J. et al. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Trans. Graph. TOG 35 , 1–19 (2016).

Article   Google Scholar  

Wang, S., Song, J., Lien, J., Poupyrev, I. & Hilliges, O. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum 851–860 (2016).

Pu, Q., Gupta, S., Gollakota, S. & Patel, S. Whole-home gesture recognition using wireless signals 27–38 (2013).

He, W., Wu, K., Zou, Y. & Ming, Z. WiG: WiFi-based gesture recognition system 1–7 (IEEE, 2015).

Ma, Y., Zhou, G., Wang, S., Zhao, H. & Jung, W. Signfi: Sign language recognition using wifi. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2 , 1–21 (2018).

Article   CAS   Google Scholar  

Wang, W., Liu, A. X. & Sun, K. Device-free gesture tracking using acoustic signals 82–94 (2016).

Nandakumar, R., Iyer, V., Tan, D. & Gollakota, S. Fingerio: Using active sonar for fine-grained finger tracking 1515–1525 (2016).

Gupta, S., Morris, D., Patel, S. & Tan, D. Soundwave: Using the doppler effect to sense gestures 1911–1914 (2012).

Pansare, J. R., Gawande, S. H. & Ingle, M. Real-time static hand gesture recognition for American Sign Language (ASL) in complex background (2012).

Choudhury, A., Talukdar, A. K. & Sarma, K. K. A novel hand segmentation method for multiple-hand gesture recognition system under complex background 136–140 (IEEE, 2014).

Stergiopoulou, E., Sgouropoulos, K., Nikolaou, N., Papamarkos, N. & Mitianoudis, N. Real time hand detection in a complex background. Eng. Appl. Artif. Intell. 35 , 54–70 (2014).

Chen, Q., Georganas, N. D. & Petriu, E. M. Real-time vision-based hand gesture recognition using haar-like features 1–6 (IEEE, 2007).

Kulkarni, V. S. & Lokhande, S. Appearance based recognition of american sign language using gesture segmentation. Int. J. Comput. Sci. Eng. 2 , 560–565 (2010).

Google Scholar  

Zhou, Y., Jiang, G. & Lin, Y. A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recognit. 49 , 102–114 (2016).

Article   ADS   Google Scholar  

Wadhawan, A. & Kumar, P. Deep learning-based sign language recognition system for static signs. Neural Comput. Appl. 32 , 7957–7968 (2020).

Pun, C.-M., Zhu, H.-M. & Feng, W. Real-time hand gesture recognition using motion tracking. Int. J. Comput. Intell. Syst. 4 , 277–286 (2011).

Molina, J., Pajuelo, J. A. & Martínez, J. M. Real-time motion-based hand gestures recognition from time-of-flight video. J. Signal Process. Syst. 86 , 17–25 (2017).

Devineau, G., Moutarde, F., Xi, W. & Yang, J. Deep learning for hand gesture recognition on skeletal data 106–113 (IEEE, 2018).

Chen, Y., Luo, B., Chen, Y.-L., Liang, G. & Wu, X. A real-time dynamic hand gesture recognition system using kinect sensor 2026–2030 (IEEE, 2015).

Ren, Z., Meng, J. & Yuan, J. Depth camera based hand gesture recognition and its applications in human-computer-interaction 1–5 (IEEE, 2011).

Ma, X. & Peng, J. Kinect sensor-based long-distance hand gesture recognition and fingertip detection with depth information. J. Sens. https://doi.org/10.1155/2018/5809769 (2018).

Song, L., Hu, R. M., Zhang, H., Xiao, Y. L. & Gong, L. Y. Real-Time 3d Hand Gesture Detection from Depth Images Vol. 756, 4138–4142 (Trans Tech Publ, 2013).

Aly, W., Aly, S. & Almotairi, S. User-independent American sign language alphabet recognition based on depth image and PCANet features. IEEE Access 7 , 123138–123150 (2019).

Potter, L. E., Araullo, J. & Carter, L. The leap motion controller: A view on sign language 175–178 (2013).

Mittal, A., Kumar, P., Roy, P. P., Balasubramanian, R. & Chaudhuri, B. B. A modified LSTM model for continuous sign language recognition using leap motion. IEEE Sens. J. 19 , 7056–7063 (2019).

Zhang, Z. Microsoft kinect sensor and its effect. IEEE Multimed. 19 , 4–10 (2012).

Xiao, Q., Zhao, Y. & Huan, W. Multi-sensor data fusion for sign language recognition based on dynamic Bayesian network and convolutional neural network. Multimed. Tools Appl. 78 , 15335–15352 (2019).

Kumar, P., Saini, R., Roy, P. P. & Dogra, D. P. A position and rotation invariant framework for sign language recognition (SLR) using Kinect. Multimed. Tools Appl. 77 , 8823–8846 (2018).

Lugaresi, C. et al. Mediapipe: A framework for perceiving and processing reality (2019).

Burdea, G. C. & Coiffet, P. Virtual Reality Technology (John Wiley & Sons, 2003).

Book   Google Scholar  

Ding, Z. et al. sEMG-based gesture recognition with convolution neural networks. Sustainability 10 , 1865 (2018).

Hu, Y. et al. A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition. PLoS ONE 13 , e0206049 (2018).

Ovur, S. E. et al. A novel autonomous learning framework to enhance sEMG-based hand gesture recognition using depth information. Biomed. Signal Process. Control 66 , 102444 (2021).

Pomboza-Junez, G. & Terriza, J. H. Hand gesture recognition based on sEMG signals using Support Vector Machines 174–178 (IEEE, 2016).

Tsinganos, P., Cornelis, B., Cornelis, J., Jansen, B. & Skodras, A. Improved gesture recognition based on sEMG signals and TCN 1169–1173 (IEEE, 2019).

Savur, C. & Sahin, F. American sign language recognition system by using surface EMG signal 002872–002877 (IEEE, 2016).

Wu, J., Sun, L. & Jafari, R. A wearable system for recognizing American sign language in real-time using IMU and surface EMG sensors. IEEE J. Biomed. Health Inform. 20 , 1281–1290 (2016).

Wang, Z. et al. Hear sign language: A real-time end-to-end sign language recognition system. IEEE Trans. Mob. Comput. https://doi.org/10.1109/TMC.2020.3038303 (2020).

Abreu, J. G., Teixeira, J. M., Figueiredo, L. S. & Teichrieb, V. Evaluating sign language recognition using the Myo Armband. In 2016 XVIII Symposium on Virtual and Augmented Reality (SVR) 64–70. https://doi.org/10.1109/SVR.2016.21 (2016).

Su, R., Chen, X., Cao, S. & Zhang, X. Random forest-based recognition of isolated sign language subwords using data from accelerometers and surface electromyographic sensors. Sensors 16 , 100 (2016).

Faisal, M. A. A., Abir, F. F. & Ahmed, M. U. Sensor dataglove for real-time static and dynamic hand gesture recognition. In 2021 Joint 10th International Conference on Informatics, Electronics Vision (ICIEV) and 2021 5th International Conference on Imaging, Vision Pattern Recognition (icIVPR) 1–7. https://doi.org/10.1109/ICIEVicIVPR52578.2021.9564226 (2021).

Lee, B. G. & Lee, S. M. Smart wearable hand device for sign language interpretation system with sensors fusion. IEEE Sens. J. 18 , 1224–1232 (2018).

Jani, A. B., Kotak, N. A. & Roy, A. K. Sensor based hand gesture recognition system for English alphabets used in sign language of deaf-mute people. In 2018 IEEE SENSORS 1–4. https://doi.org/10.1109/ICSENS.2018.8589574 (2018).

Chong, T.-W. & Kim, B.-J. American sign language recognition system using wearable sensors with deep learning approach. J. Korea Inst. Electron. Commun. Sci. 15 , 291–298 (2020).

Gałka, J., Mąsior, M., Zaborski, M. & Barczewska, K. Inertial motion sensing glove for sign language gesture acquisition and recognition. IEEE Sens. J. 16 , 6310–6316 (2016).

Wen, F., Zhang, Z., He, T. & Lee, C. AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nat. Commun. 12 , 1–13 (2021).

Yu, Y., Chen, X., Cao, S., Zhang, X. & Chen, X. Exploration of Chinese sign language recognition using wearable sensors based on deep belief net. IEEE J. Biomed. Health Inform. 24 , 1310–1320 (2020).

SparkFun. Flex Sensor 2.2—SEN-10264—SparkFun Electronics. SparkFun https://www.sparkfun.com/products/10264 .

TDK. MPU-6050—TDK, InvenSense Corporation. https://invensense.tdk.com/products/motion-tracking/6-axis/mpu-6050/ .

Espressif. ESP32 Wi-Fi & bluetooth MCU—Espressif systems. ESPRESSIF-ESP32 https://www.espressif.com/en/products/socs/esp32 .

Lapiak, J. American sign language dictionary—HandSpeak. https://www.handspeak.com/ .

Canuto, E., Novara, C., Massotti, L., Carlucci, D. & Montenegro, C. P. Chapter 2—Attitude representation. In Spacecraft Dynamics and Control (eds Canuto, E. et al. ) 17–83 (Butterworth-Heinemann, 2018).

Kim, A. & Golnaraghi, M. A quaternion-based orientation estimation algorithm using an inertial measurement unit 268–272 (IEEE, 2004).

Cai, G., Chen, B. M. & Lee, T. H. Coordinate systems and transformations. In Unmanned Rotorcraft Systems 23–34 (Springer, 2011).

Chapter   MATH   Google Scholar  

Ahmed, M., Antar, A. D., Hossain, T., Inoue, S. & Ahad, M. A. R. Poiden: Position and orientation independent deep ensemble network for the classification of locomotion and transportation modes 674–679 (2019).

Smith, S. W. Chapter 15—Moving average filters. In Digital Signal Processing (ed. Smith, S. W.) 277–284 (Newnes, USA, 2003).

Bhanja, S. & Das, A. Impact of data normalization on deep neural network for time series forecasting. https://arxiv.org/abs/812.05519 Cs Stat (2019).

Patro, S. G. K. & Sahu, K. K. Normalization: A preprocessing stage. https://arxiv.org/abs/1503.06462 Cs (2015).

Hunter, J. D. Matplotlib: A 2D graphics environment. Comput. Sci. Eng. 9 , 90–95 (2007).

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. & Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks 4510–4520 (2018).

Howard, A. G. et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv Prepr. https://arxiv.org/abs/1704.04861 (2017).

Sifre, L. & Mallat, S. Rigid-motion scattering for texture classification. https://arxiv.org/abs/1403.1687 (2014).

He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition 770–778 (2016).

Abadi, M. et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. ArXiv Prepr. https://arxiv.org/abs/1603.04467 (2016).

Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. ArXiv Prepr. https://arxiv.org/abs/1412.6980 (2014).

Chong, T. W. & Lee, B. G. American sign language recognition using leap motion controller with machine learning approach. Sensors 18 , 3554 (2018).

Saquib, N. & Rahman, A. Application of machine learning techniques for real-time sign language detection using wearable sensors. In Proceedings of the 11th ACM Multimedia Systems Conference 178–189. (Association for Computing Machinery, 2020). https://doi.org/10.1145/3339825.3391869 .

Zhang, Y. et al. Static and dynamic human arm/hand gesture capturing and recognition via multiinformation fusion of flexible strain sensors. IEEE Sens. J. 20 , 6450–6459 (2020).

Abhishek, K. S., Qubeley, L. C. F. & Ho, D. Glove-based hand gesture recognition sign language translator using capacitive touch sensor. In 2016 IEEE International Conference on Electron Devices and Solid-State Circuits (EDSSC) 334–337. https://doi.org/10.1109/EDSSC.2016.7785276 (2016).

ASL-Sensor-Dataglove-Dataset.zip. 10.6084/m9.figshare.20031017.v1 (2022).

Download references

Acknowledgements

This work was supported by the Centennial Research Grant, University of Dhaka, Bangladesh, and APC was sponsored by University of East London, UK.

Author information

These authors contributed equally: Md. Ahasan Atick Faisal and Farhan Fuad Abir.

Authors and Affiliations

Department of Electrical and Electronic Engineering, University of Dhaka, Dhaka, 1000, Bangladesh

Md. Ahasan Atick Faisal, Farhan Fuad Abir & Mosabber Uddin Ahmed

Department of Computer Science and Digital Technologies, University of East London, London, UK

Md Atiqur Rahman Ahad

You can also search for this author in PubMed   Google Scholar

Contributions

M.A.A.F. and F.F.A. did the experiments and wrote the main manuscript text and prepared figures. Both of them contributed equally. All authors formulated the methods and design, and reviewed the manuscript. M.U.A. and M.A.R.A. are the corresponding authors. The paper has 2 corresponding authors.

Corresponding authors

Correspondence to Mosabber Uddin Ahmed or Md Atiqur Rahman Ahad .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Faisal, M.A.A., Abir, F.F., Ahmed, M.U. et al. Exploiting domain transformation and deep learning for hand gesture recognition using a low-cost dataglove. Sci Rep 12 , 21446 (2022). https://doi.org/10.1038/s41598-022-25108-2

Download citation

Received : 04 June 2022

Accepted : 24 November 2022

Published : 12 December 2022

DOI : https://doi.org/10.1038/s41598-022-25108-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• PubMed/Medline
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Hand gesture recognition based on computer vision: a review of techniques.

1. Introduction

2. hand gesture methods, 2.1. hand gestures based on instrumented glove approach, 2.2. hand gestures based on computer vision approach, 2.2.1. color-based recognition:, color-based recognition using glove marker, color-based recognition of skin color.

• red, green, blue (R–G–B and RGB-normalized);
• hue and saturation (H–S–V, H–S–I and H–S–L);
• luminance (YIQ, Y–Cb–Cr and YUV).

2.2.2. Appearance-Based Recognition

2.2.3. motion-based recognition, 2.2.4. skeleton-based recognition, 2.2.5. depth-based recognition, 2.2.6. 3d model-based recognition, 2.2.7. deep-learning based recognition, 3. application areas of hand gesture recognition systems, 3.1. clinical and health, 3.2. sign language recognition, 3.3. robot control, 3.4. virtual environment, 3.5. home automation, 3.6. personal computer and tablet, 3.7. gestures for gaming, 4. research gaps and challenges, 5. conclusions, author contributions, acknowledgments, conflicts of interest.

• Zhigang, F. Computer gesture input and its application in human computer interaction. Mini Micro Syst. 1999 , 6 , 418–421. [ Google Scholar ]
• Mitra, S.; Acharya, T. Gesture recognition: A survey. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 2007 , 37 , 311–324. [ Google Scholar ] [ CrossRef ]
• Ahuja, M.K.; Singh, A. Static vision based Hand Gesture recognition using principal component analysis. In Proceedings of the 2015 IEEE 3rd International Conference on MOOCs, Innovation and Technology in Education (MITE), Amritsar, India, 1–2 October 2015; pp. 402–406. [ Google Scholar ]
• Kramer, R.K.; Majidi, C.; Sahai, R.; Wood, R.J. Soft curvature sensors for joint angle proprioception. In Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA, 25–30 September 2011; pp. 1919–1926. [ Google Scholar ]
• Jesperson, E.; Neuman, M.R. A thin film strain gauge angular displacement sensor for measuring finger joint angles. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, New Orleans, LA, USA, 4–7 November 1988; pp. 807–vol. [ Google Scholar ]
• Fujiwara, E.; dos Santos, M.F.M.; Suzuki, C.K. Flexible optical fiber bending transducer for application in glove-based sensors. IEEE Sens. J. 2014 , 14 , 3631–3636. [ Google Scholar ] [ CrossRef ]
• Shrote, S.B.; Deshpande, M.; Deshmukh, P.; Mathapati, S. Assistive Translator for Deaf & Dumb People. Int. J. Electron. Commun. Comput. Eng. 2014 , 5 , 86–89. [ Google Scholar ]
• Gupta, H.P.; Chudgar, H.S.; Mukherjee, S.; Dutta, T.; Sharma, K. A continuous hand gestures recognition technique for human-machine interaction using accelerometer and gyroscope sensors. IEEE Sens. J. 2016 , 16 , 6425–6432. [ Google Scholar ] [ CrossRef ]
• Lamberti, L.; Camastra, F. Real-time hand gesture recognition using a color glove. In Proceedings of the International Conference on Image Analysis and Processing, Ravenna, Italy, 14–16 September 2011; pp. 365–373. [ Google Scholar ]
• Wachs, J.P.; Kölsch, M.; Stern, H.; Edan, Y. Vision-based hand-gesture applications. Commun. ACM 2011 , 54 , 60–71. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Pansare, J.R.; Gawande, S.H.; Ingle, M. Real-time static hand gesture recognition for American Sign Language (ASL) in complex background. JSIP 2012 , 3 , 22132. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Van den Bergh, M.; Carton, D.; De Nijs, R.; Mitsou, N.; Landsiedel, C.; Kuehnlenz, K.; Wollherr, D.; Van Gool, L.; Buss, M. Real-time 3D hand gesture interaction with a robot for understanding directions from humans. In Proceedings of the 2011 Ro-Man, Atlanta, GA, USA, 31 July–3 August 2011; pp. 357–362. [ Google Scholar ]
• Wang, R.Y.; Popović, J. Real-time hand-tracking with a color glove. ACM Trans. Graph. 2009 , 28 , 1–8. [ Google Scholar ]
• Desai, S.; Desai, A. Human Computer Interaction through hand gestures for home automation using Microsoft Kinect. In Proceedings of the International Conference on Communication and Networks, Xi’an, China, 10–12 October 2017; pp. 19–29. [ Google Scholar ]
• Rajesh, R.J.; Nagarjunan, D.; Arunachalam, R.M.; Aarthi, R. Distance Transform Based Hand Gestures Recognition for PowerPoint Presentation Navigation. Adv. Comput. 2012 , 3 , 41. [ Google Scholar ]
• Kaur, H.; Rani, J. A review: Study of various techniques of Hand gesture recognition. In Proceedings of the 2016 IEEE 1st International Conference on Power Electronics, Intelligent Control and Energy Systems (ICPEICES), Delhi, India, 4–6 July 2016; pp. 1–5. [ Google Scholar ]
• Murthy, G.R.S.; Jadon, R.S. A review of vision based hand gestures recognition. Int. J. Inf. Technol. Knowl. Manag. 2009 , 2 , 405–410. [ Google Scholar ]
• Khan, R.Z.; Ibraheem, N.A. Hand gesture recognition: A literature review. Int. J. Artif. Intell. Appl. 2012 , 3 , 161. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Suriya, R.; Vijayachamundeeswari, V. A survey on hand gesture recognition for simple mouse control. In Proceedings of the International Conference on Information Communication and Embedded Systems (ICICES2014), Chennai, India, 27–28 February 2014; pp. 1–5. [ Google Scholar ]
• Sonkusare, J.S.; Chopade, N.B.; Sor, R.; Tade, S.L. A review on hand gesture recognition system. In Proceedings of the 2015 International Conference on Computing Communication Control and Automation, Pune, India, 26–27 February 2015; pp. 790–794. [ Google Scholar ]
• Garg, P.; Aggarwal, N.; Sofat, S. Vision based hand gesture recognition. World Acad. Sci. Eng. Technol. 2009 , 49 , 972–977. [ Google Scholar ]
• Dipietro, L.; Sabatini, A.M.; Member, S.; Dario, P. A survey of glove-based systems and their applications. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 2008 , 38 , 461–482. [ Google Scholar ] [ CrossRef ]
• LaViola, J. A survey of hand posture and gesture recognition techniques and technology. Brown Univ. Provid. RI 1999 , 29 . Technical Report no. CS-99-11. [ Google Scholar ]
• Ibraheem, N.A.; Khan, R.Z. Survey on various gesture recognition technologies and techniques. Int. J. Comput. Appl. 2012 , 50 , 38–44. [ Google Scholar ]
• Hasan, M.M.; Mishra, P.K. Hand gesture modeling and recognition using geometric features: A review. Can. J. Image Process. Comput. Vis. 2012 , 3 , 12–26. [ Google Scholar ]
• Shaik, K.B.; Ganesan, P.; Kalist, V.; Sathish, B.S.; Jenitha, J.M.M. Comparative study of skin color detection and segmentation in HSV and YCbCr color space. Procedia Comput. Sci. 2015 , 57 , 41–48. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Ganesan, P.; Rajini, V. YIQ color space based satellite image segmentation using modified FCM clustering and histogram equalization. In Proceedings of the 2014 International Conference on Advances in Electrical Engineering (ICAEE), Vellore, India, 9–11 January 2014; pp. 1–5. [ Google Scholar ]
• Brand, J.; Mason, J.S. A comparative assessment of three approaches to pixel-level human skin-detection. In Proceedings of the 15th International Conference on Pattern Recognition. ICPR-2000, Barcelona, Spain, 3–7 September 2000; Volume 1, pp. 1056–1059. [ Google Scholar ]
• Jones, M.J.; Rehg, J.M. Statistical color models with application to skin detection. Int. J. Comput. Vis. 2002 , 46 , 81–96. [ Google Scholar ] [ CrossRef ]
• Brown, D.A.; Craw, I.; Lewthwaite, J. A som based approach to skin detection with application in real time systems. BMVC 2001 , 1 , 491–500. [ Google Scholar ]
• Zarit, B.D.; Super, B.J.; Quek, F.K.H. Comparison of five color models in skin pixel classification. In Proceedings of the International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, In Conjunction with ICCV’99 (Cat. No. PR00378). Corfu, Greece, 26–27 September 1999; pp. 58–63. [ Google Scholar ]
• Albiol, A.; Torres, L.; Delp, E.J. Optimum color spaces for skin detection. In Proceedings of the 2001 International Conference on Image Processing (Cat. No. 01CH37205), Thessaloniki, Greece, 7–10 October 2001; Volume 1, pp. 122–124. [ Google Scholar ]
• Sigal, L.; Sclaroff, S.; Athitsos, V. Estimation and prediction of evolving color distributions for skin segmentation under varying illumination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2000 (Cat. No. PR00662). Hilton Head Island, SC, USA, 15 June 2000; Volume 2, pp. 152–159. [ Google Scholar ]
• Chai, D.; Bouzerdoum, A. A Bayesian approach to skin color classification in YCbCr color space. In Proceedings of the 2000 TENCON Proceedings. Intelligent Systems and Technologies for the New Millennium (Cat. No. 00CH37119), Kuala Lumpur, Malaysia, 24–27 September 2000; Volume 2, pp. 421–424. [ Google Scholar ]
• Menser, B.; Wien, M. Segmentation and tracking of facial regions in color image sequences. In Proceedings of the Visual Communications and Image Processing 2000, Perth, Australia, 20–23 June 2000; Volume 4067, pp. 731–740. [ Google Scholar ]
• Kakumanu, P.; Makrogiannis, S.; Bourbakis, N. A survey of skin-color modeling and detection methods. Pattern Recognit. 2007 , 40 , 1106–1122. [ Google Scholar ] [ CrossRef ]
• Perimal, M.; Basah, S.N.; Safar, M.J.A.; Yazid, H. Hand-Gesture Recognition-Algorithm based on Finger Counting. J. Telecommun. Electron. Comput. Eng. 2018 , 10 , 19–24. [ Google Scholar ]
• Sulyman, A.B.D.A.; Sharef, Z.T.; Faraj, K.H.A.; Aljawaryy, Z.A.; Malallah, F.L. Real-time numerical 0-5 counting based on hand-finger gestures recognition. J. Theor. Appl. Inf. Technol. 2017 , 95 , 3105–3115. [ Google Scholar ]
• Choudhury, A.; Talukdar, A.K.; Sarma, K.K. A novel hand segmentation method for multiple-hand gesture recognition system under complex background. In Proceedings of the 2014 International Conference on Signal Processing and Integrated Networks (SPIN), Noida, India, 20–21 February 2014; pp. 136–140. [ Google Scholar ]
• Stergiopoulou, E.; Sgouropoulos, K.; Nikolaou, N.; Papamarkos, N.; Mitianoudis, N. Real time hand detection in a complex background. Eng. Appl. Artif. Intell. 2014 , 35 , 54–70. [ Google Scholar ] [ CrossRef ]
• Khandade, S.L.; Khot, S.T. MATLAB based gesture recognition. In Proceedings of the 2016 International Conference on Inventive Computation Technologies (ICICT), Coimbatore, India, 26–27 August 2016; Volume 1, pp. 1–4. [ Google Scholar ]
• Karabasi, M.; Bhatti, Z.; Shah, A. A model for real-time recognition and textual representation of malaysian sign language through image processing. In Proceedings of the 2013 International Conference on Advanced Computer Science Applications and Technologies, Kuching, Malaysia, 23–24 December 2013; pp. 195–200. [ Google Scholar ]
• Zeng, J.; Sun, Y.; Wang, F. A natural hand gesture system for intelligent human-computer interaction and medical assistance. In Proceedings of the 2012 Third Global Congress on Intelligent Systems, Wuhan, China, 6–8 November 2012; pp. 382–385. [ Google Scholar ]
• Hsieh, C.-C.; Liou, D.-H.; Lee, D. A real time hand gesture recognition system using motion history image. In Proceedings of the 2010 2nd international conference on signal processing systems, Dalian, China, 5–7 July 2010; Volume 2, pp. V2–394. [ Google Scholar ]
• Van den Bergh, M.; Koller-Meier, E.; Bosché, F.; Van Gool, L. Haarlet-based hand gesture recognition for 3D interaction. In Proceedings of the 2009 Workshop on Applications of Computer Vision (WACV), Snowbird, UT, USA, 7–8 December 2009; pp. 1–8. [ Google Scholar ]
• Van den Bergh, M.; Van Gool, L. Combining RGB and ToF cameras for real-time 3D hand gesture interaction. In Proceedings of the 2011 IEEE workshop on applications of computer vision (WACV), Kona, HI, USA, 5–7 January 2011; pp. 66–72. [ Google Scholar ]
• Chen, L.; Wang, F.; Deng, H.; Ji, K. A survey on hand gesture recognition. In Proceedings of the 2013 International conference on computer sciences and applications, Wuhan, China, 14–15 December 2013; pp. 313–316. [ Google Scholar ]
• Shimada, A.; Yamashita, T.; Taniguchi, R. Hand gesture based TV control system—Towards both user-& machine-friendly gesture applications. In Proceedings of the 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision, Incheon, Korea, 30 January–1 February 2013; pp. 121–126. [ Google Scholar ]
• Chen, Q.; Georganas, N.D.; Petriu, E.M. Real-time vision-based hand gesture recognition using haar-like features. In Proceedings of the 2007 IEEE instrumentation & measurement technology conference IMTC 2007, Warsaw, Poland, 1–3 May 2007; pp. 1–6. [ Google Scholar ]
• Kulkarni, V.S.; Lokhande, S.D. Appearance based recognition of american sign language using gesture segmentation. Int. J. Comput. Sci. Eng. 2010 , 2 , 560–565. [ Google Scholar ]
• Fang, Y.; Wang, K.; Cheng, J.; Lu, H. A real-time hand gesture recognition method. In Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, Beijing, China, 2–5 July 2007; pp. 995–998. [ Google Scholar ]
• Licsár, A.; Szirányi, T. User-adaptive hand gesture recognition system with interactive training. Image Vis. Comput. 2005 , 23 , 1102–1114. [ Google Scholar ] [ CrossRef ]
• Zhou, Y.; Jiang, G.; Lin, Y. A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recognit. 2016 , 49 , 102–114. [ Google Scholar ] [ CrossRef ]
• Pun, C.-M.; Zhu, H.-M.; Feng, W. Real-time hand gesture recognition using motion tracking. Int. J. Comput. Intell. Syst. 2011 , 4 , 277–286. [ Google Scholar ] [ CrossRef ]
• Bayazit, M.; Couture-Beil, A.; Mori, G. Real-time Motion-based Gesture Recognition Using the GPU. In Proceedings of the MVA, Yokohama, Japan, 20–22 May 2009; pp. 9–12. [ Google Scholar ]
• Molina, J.; Pajuelo, J.A.; Martínez, J.M. Real-time motion-based hand gestures recognition from time-of-flight video. J. Signal Process. Syst. 2017 , 86 , 17–25. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Prakash, J.; Gautam, U.K. Hand Gesture Recognition. Int. J. Recent Technol. Eng. 2019 , 7 , 54–59. [ Google Scholar ]
• Xi, C.; Chen, J.; Zhao, C.; Pei, Q.; Liu, L. Real-time Hand Tracking Using Kinect. In Proceedings of the 2nd International Conference on Digital Signal Processing, Tokyo, Japan, 25–27 February 2018; pp. 37–42. [ Google Scholar ]
• Devineau, G.; Moutarde, F.; Xi, W.; Yang, J. Deep learning for hand gesture recognition on skeletal data. In Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China, 15–19 May 2018; pp. 106–113. [ Google Scholar ]
• Jiang, F.; Wu, S.; Yang, G.; Zhao, D.; Kung, S.-Y. independent hand gesture recognition with Kinect. Signal Image Video Process. 2014 , 8 , 163–172. [ Google Scholar ] [ CrossRef ]
• Konstantinidis, D.; Dimitropoulos, K.; Daras, P. Sign language recognition based on hand and body skeletal data. In Proceedings of the 2018-3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), Helsinki, Finland, 3–5 June 2018; pp. 1–4. [ Google Scholar ]
• De Smedt, Q.; Wannous, H.; Vandeborre, J.-P.; Guerry, J.; Saux, B.L.; Filliat, D. 3D hand gesture recognition using a depth and skeletal dataset: SHREC’17 track. In Proceedings of the Workshop on 3D Object Retrieval, Lyon, France, 23–24 April 2017; pp. 33–38. [ Google Scholar ]
• Chen, Y.; Luo, B.; Chen, Y.-L.; Liang, G.; Wu, X. A real-time dynamic hand gesture recognition system using kinect sensor. In Proceedings of the 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO), Zhuhai, China, 6–9 December 2015; pp. 2026–2030. [ Google Scholar ]
• Karbasi, M.; Muhammad, Z.; Waqas, A.; Bhatti, Z.; Shah, A.; Koondhar, M.Y.; Brohi, I.A. A Hybrid Method Using Kinect Depth and Color Data Stream for Hand Blobs Segmentation ; Science International: Lahore, Pakistan, 2017; Volume 29, pp. 515–519. [ Google Scholar ]
• Ren, Z.; Meng, J.; Yuan, J. Depth camera based hand gesture recognition and its applications in human-computer-interaction. In Proceedings of the 2011 8th International Conference on Information, Communications & Signal Processing, Singapore, 13–16 December 2011; pp. 1–5. [ Google Scholar ]
• Dinh, D.-L.; Kim, J.T.; Kim, T.-S. Hand gesture recognition and interface via a depth imaging sensor for smart home appliances. Energy Procedia 2014 , 62 , 576–582. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Raheja, J.L.; Minhas, M.; Prashanth, D.; Shah, T.; Chaudhary, A. Robust gesture recognition using Kinect: A comparison between DTW and HMM. Optik 2015 , 126 , 1098–1104. [ Google Scholar ] [ CrossRef ]
• Ma, X.; Peng, J. Kinect sensor-based long-distance hand gesture recognition and fingertip detection with depth information. J. Sens. 2018 , 2018 , 5809769. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Kim, M.-S.; Lee, C.H. Hand Gesture Recognition for Kinect v2 Sensor in the Near Distance Where Depth Data Are Not Provided. Int. J. Softw. Eng. Its Appl. 2016 , 10 , 407–418. [ Google Scholar ] [ CrossRef ]
• Li, Y. Hand gesture recognition using Kinect. In Proceedings of the 2012 IEEE International Conference on Computer Science and Automation Engineering, Beijing, China, 22–24 June 2012; pp. 196–199. [ Google Scholar ]
• Song, L.; Hu, R.M.; Zhang, H.; Xiao, Y.L.; Gong, L.Y. Real-time 3d hand gesture detection from depth images. Adv. Mater. Res. 2013 , 756 , 4138–4142. [ Google Scholar ] [ CrossRef ]
• Pal, D.H.; Kakade, S.M. Dynamic hand gesture recognition using kinect sensor. In Proceedings of the 2016 International Conference on Global Trends in Signal Processing, Information Computing and Communication (ICGTSPICC), Jalgaon, India, 22–24 December 2016; pp. 448–453. [ Google Scholar ]
• Karbasi, M.; Bhatti, Z.; Nooralishahi, P.; Shah, A.; Mazloomnezhad, S.M.R. Real-time hands detection in depth image by using distance with Kinect camera. Int. J. Internet Things 2015 , 4 , 1–6. [ Google Scholar ]
• Bakar, M.Z.A.; Samad, R.; Pebrianti, D.; Aan, N.L.Y. Real-time rotation invariant hand tracking using 3D data. In Proceedings of the 2014 IEEE International Conference on Control System, Computing and Engineering (ICCSCE 2014), Batu Ferringhi, Malaysia, 28–30 November 2014; pp. 490–495. [ Google Scholar ]
• Desai, S. Segmentation and Recognition of Fingers Using Microsoft Kinect. In Proceedings of the International Conference on Communication and Networks, Paris, France, 21–25 May 2017; pp. 45–53. [ Google Scholar ]
• Lee, U.; Tanaka, J. Finger identification and hand gesture recognition techniques for natural user interface. In Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction, Bangalore, India, 24–27 September 2013; pp. 274–279. [ Google Scholar ]
• Bakar, M.Z.A.; Samad, R.; Pebrianti, D.; Mustafa, M.; Abdullah, N.R.H. Finger application using K-Curvature method and Kinect sensor in real-time. In Proceedings of the 2015 International Symposium on Technology Management and Emerging Technologies (ISTMET), Langkawai Island, Malaysia, 25–27 August 2015; pp. 218–222. [ Google Scholar ]
• Tang, M. Recognizing Hand Gestures with Microsoft’s Kinect ; Department of Electrical Engineering of Stanford University: Palo Alto, CA, USA, 2011. [ Google Scholar ]
• Bamwenda, J.; Özerdem, M.S. Recognition of Static Hand Gesture with Using ANN and SVM. Dicle Univ. J. Eng. 2019 , 10 , 561–568. [ Google Scholar ]
• Tekin, B.; Bogo, F.; Pollefeys, M. H+ O: Unified egocentric recognition of 3D hand-object poses and interactions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 4511–4520. [ Google Scholar ]
• Wan, C.; Probst, T.; Van Gool, L.; Yao, A. Self-supervised 3d hand pose estimation through training by fitting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 10853–10862. [ Google Scholar ]
• Ge, L.; Ren, Z.; Li, Y.; Xue, Z.; Wang, Y.; Cai, J.; Yuan, J. 3d hand shape and pose estimation from a single rgb image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 10833–10842. [ Google Scholar ]
• Taylor, J.; Bordeaux, L.; Cashman, T.; Corish, B.; Keskin, C.; Sharp, T.; Soto, E.; Sweeney, D.; Valentin, J.; Luff, B. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM Trans. Graph. 2016 , 35 , 1–12. [ Google Scholar ] [ CrossRef ]
• Malik, J.; Elhayek, A.; Stricker, D. Structure-aware 3D hand pose regression from a single depth image. In Proceedings of the International Conference on Virtual Reality and Augmented Reality, London, UK, 22–23 October 2018; pp. 3–17. [ Google Scholar ]
• Tsoli, A.; Argyros, A.A. Joint 3D tracking of a deformable object in interaction with a hand. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 484–500. [ Google Scholar ]
• Chen, Y.; Tu, Z.; Ge, L.; Zhang, D.; Chen, R.; Yuan, J. So-handnet: Self-organizing network for 3d hand pose estimation with semi-supervised learning. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 6961–6970. [ Google Scholar ]
• Ge, L.; Ren, Z.; Yuan, J. Point-to-point regression pointnet for 3d hand pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 475–491. [ Google Scholar ]
• Wu, X.; Finnegan, D.; O’Neill, E.; Yang, Y.-L. Handmap: Robust hand pose estimation via intermediate dense guidance map supervision. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 237–253. [ Google Scholar ]
• Cai, Y.; Ge, L.; Cai, J.; Yuan, J. Weakly-supervised 3d hand pose estimation from monocular rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 666–682. [ Google Scholar ]
• Alnaim, N.; Abbod, M.; Albar, A. Hand Gesture Recognition Using Convolutional Neural Network for People Who Have Experienced A Stroke. In Proceedings of the 2019 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Ankara, Turkey, 11–13 October 2019; pp. 1–6. [ Google Scholar ]
• Chung, H.; Chung, Y.; Tsai, W. An efficient hand gesture recognition system based on deep CNN. In Proceedings of the 2019 IEEE International Conference on Industrial Technology (ICIT), Melbourne, Australia, 13–15 February 2019; pp. 853–858. [ Google Scholar ]
• Bao, P.; Maqueda, A.I.; del-Blanco, C.R.; García, N. Tiny hand gesture recognition without localization via a deep convolutional network. IEEE Trans. Consum. Electron. 2017 , 63 , 251–257. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Li, G.; Tang, H.; Sun, Y.; Kong, J.; Jiang, G.; Jiang, D.; Tao, B.; Xu, S.; Liu, H. Hand gesture recognition based on convolution neural network. Cluster Comput. 2019 , 22 , 2719–2729. [ Google Scholar ] [ CrossRef ]
• Lin, H.-I.; Hsu, M.-H.; Chen, W.-K. Human hand gesture recognition using a convolution neural network. In Proceedings of the 2014 IEEE International Conference on Automation Science and Engineering (CASE), Taipei, Taiwan, 18–22 August 2014; pp. 1038–1043. [ Google Scholar ]
• John, V.; Boyali, A.; Mita, S.; Imanishi, M.; Sanma, N. Deep learning-based fast hand gesture recognition using representative frames. In Proceedings of the 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Gold Coast, Australia, 30 November–2 December 2016; pp. 1–8. [ Google Scholar ]
• Wu, X.Y. A hand gesture recognition algorithm based on DC-CNN. Multimed. Tools Appl. 2019 , 1–13. [ Google Scholar ] [ CrossRef ]
• Nguyen, X.S.; Brun, L.; Lézoray, O.; Bougleux, S. A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 12036–12045. [ Google Scholar ]
• Lee, D.-H.; Hong, K.-S. Game interface using hand gesture recognition. In Proceedings of the 5th International Conference on Computer Sciences and Convergence Information Technology, Seoul, Korea, 30 November–2 December 2010; pp. 1092–1097. [ Google Scholar ]
• Gallo, L.; Placitelli, A.P.; Ciampi, M. Controller-free exploration of medical image data: Experiencing the Kinect. In Proceedings of the 2011 24th international symposium on computer-based medical systems (CBMS), Bristol, UK, 27–30 June 2011; pp. 1–6. [ Google Scholar ]
• Zhao, X.; Naguib, A.M.; Lee, S. Kinect based calling gesture recognition for taking order service of elderly care robot. In Proceedings of the The 23rd IEEE international symposium on robot and human interactive communication, Edinburgh, UK, 25–29 August 2014; pp. 525–530. [ Google Scholar ]

Click here to enlarge figure

Share and Cite

Oudah, M.; Al-Naji, A.; Chahl, J. Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. J. Imaging 2020 , 6 , 73. https://doi.org/10.3390/jimaging6080073

Oudah M, Al-Naji A, Chahl J. Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. Journal of Imaging . 2020; 6(8):73. https://doi.org/10.3390/jimaging6080073

Oudah, Munir, Ali Al-Naji, and Javaan Chahl. 2020. "Hand Gesture Recognition Based on Computer Vision: A Review of Techniques" Journal of Imaging 6, no. 8: 73. https://doi.org/10.3390/jimaging6080073

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Advanced Search

Hand Gesture Recognition Methods and Applications: A Literature Survey

New Citation Alert added!

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, editorial notes.

• Dayananda Kumar N Suresh K Dinesh R (2022) CNN based Static Hand Gesture Recognition using RGB-D Data 2022 2nd International Conference on Artificial Intelligence and Signal Processing (AISP) 10.1109/AISP53593.2022.9760658 (1-6) Online publication date: 12-Feb-2022 https://doi.org/10.1109/AISP53593.2022.9760658
• Xia C Saito A Sugiura Y (2022) Using the virtual data-driven measurement to support the prototyping of hand gesture recognition interface with distance sensor Sensors and Actuators A: Physical 10.1016/j.sna.2022.113463 338 (113463) Online publication date: May-2022 https://doi.org/10.1016/j.sna.2022.113463
• Zhou X Guo Y Jia L Jin Y Li H Xue C (2022) A study of button size for virtual hand interaction in virtual environments based on clicking performance Multimedia Tools and Applications 10.1007/s11042-022-14038-w Online publication date: 15-Oct-2022 https://doi.org/10.1007/s11042-022-14038-w

Index Terms

Applied computing

Computing methodologies

Artificial intelligence

Natural language processing

Machine learning

Human-centered computing

Human computer interaction (HCI)

Social and professional topics

Professional topics

Computing profession

Recommendations

Robust hand gesture recognition with kinect sensor.

Hand gesture based Human-Computer-Interaction (HCI) is one of the most natural and intuitive ways to communicate between people and machines, since it closely mimics how human interact with each other. In this demo, we present a hand gesture recognition ...

A Survey on Hand Gesture Recognition

Hand gesture recognition has become one of the key techniques of human-computer interaction (HCI). Many researchers are devoted in this field. In this paper, firstly the history of hand gesture recognition is discussed and the technical difficulties are ...

Finger identification and hand gesture recognition techniques for natural user interface

The natural user interface using hand gesture have been popular field in Human-Computer-Interaction(HCI). Many research papers have been proposed in this field. They proposed vision-based, glove-based and depth-based approach for hand gesture ...

Information

Published in.

Rector of International Information Technology University, IITU, Kazakhstan

PhD, Professor, IITU, Kazakhstan

Associate professor, IITU, Kazakhstan

JUST, Jordan

IITU, Kazakhstan

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates, author tags.

• computer vision
• deep learning
• feature extraction
• hand gesture recognition
• sign language
• Research-article
• Refereed limited

Acceptance Rates

Contributors, other metrics, bibliometrics, article metrics.

• 3 Total Citations View Citations
• 823 Total Downloads
• Downloads (Last 12 months) 216
• Downloads (Last 6 weeks) 25

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

HTML Format

View this article in HTML Format.

Share this Publication link

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Advertisement

Hand Gesture Control for Human–Computer Interaction with Deep Learning

• Original Article
• Published: 21 January 2022
• Volume 17 , pages 1961–1970, ( 2022 )

Cite this article

• S. N. David Chua   ORCID: orcid.org/0000-0003-4149-8696 1 ,
• K. Y. Richard Chin 1 ,
• S. F. Lim 1 &
• Pushpdant Jain 2  

877 Accesses

13 Citations

Explore all metrics

The use of gesture control has numerous advantages compared to the use of physical hardware. However, it has yet to gain popularity as most gesture control systems require extra sensors or depth cameras to detect or capture the movement of gestures before a meaningful signal can be triggered for corresponding course of action. This research proposes a method for a hand gesture control system with the use of an object detection algorithm, YOLOv3, combined with handcrafted rules to achieve dynamic gesture control on the computer. This project utilizes a single RGB camera for hand gesture recognition and localization. The dataset of all gestures used for training and its corresponding commands, are custom designed by the authors due to the lack of standard gestures specifically for human–computer interaction. Algorithms to integrate gesture commands with virtual mouse and keyboard input through the Pynput library in Python, were developed to handle commands such as mouse control, media control, and others. The mAP result of the YOLOv3 model obtained 96.68% accuracy based on testing result. The use of rule-based algorithms for gesture interpretation was successfully implemented to transform static gesture recognition into dynamic gesture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or Ebook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

A review of convolutional neural networks in computer vision

A systematic review of object detection from images using deep learning

Convolutional neural network: a review of models, methodologies and applications to object detection

Al-Shamayleh AS, Ahmad R, Abushariah MAM, Alam KA, Jomhari N (2018) A systematic literature review on vision based gesture recognition techniques. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-5971-z

Article   Google Scholar  

Anwar S, Sinha SK, Vivek S, Ashank V (2019). Hand gesture recognition: a survey. Lecture notes in electrical engineering. https://doi.org/10.1007/978-981-13-0776-8_33

Ayooshkathuria (2018) pytorch-yolo-v3. Github

Bai Y, Zhang L, Wang T, Zhou X (2019) A skeleton object detection-based dynamic gesture recognition method. In: Proceedings of the 2019 IEEE 16th international conference on networking, sensing and control, ICNSC 2019. https://doi.org/10.1109/ICNSC.2019.8743166

Beyer G, Meier M (2011) Music interfaces for novice users: composing music on a public display with hand gestures. In: Proceedings of the international conference on new interfaces for musical expression

Bochkovskiy A, Wang CY, Liao M (2020) YOLOv4: optimal speed and accuracy of object detection. https://arxiv.org/pdf/2004.10934v1.pdf

Bush IJ, Abiyev R, Arslan M (2019) Impact of machine learning techniques on hand gesture recognition. J Intell Fuzzy Syst. https://doi.org/10.3233/JIFS-190353

Camgoz NC, Hadfield S, Bowden R (2017) Particle filter based probabilistic forced alignment for continuous gesture recognition. In: Proceedings—2017 IEEE international conference on computer vision workshops, ICCVW 2017. https://doi.org/10.1109/ICCVW.2017.364

Chandrasekaran G, Periyasamy S, Panjappagounder Rajamanickam K (2020) Minimization of test time in system on chip using artificial intelligence-based test scheduling techniques. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04039-6

Chai X, Liu Z, Yin F, Liu Z, Chen X (2016) Two streams recurrent neural networks for large-scale continuous gesture recognition. In: Proceedings—international conference on pattern recognition. https://doi.org/10.1109/ICPR.2016.7899603

Chen D, Li G, Sun Y, Kong J, Jiang G, Tang H, Ju Z, Yu H, Liu H (2017) An interactive image segmentation method in hand gesture recognition. Sensors (Switzerland). https://doi.org/10.3390/s17020253

Chua SND, Lim SF, Lai SN et al (2019) Development of a child detection system with artificial intelligence using object detection method. J Electr Eng Technol 14:2523–2529. https://doi.org/10.1007/s42835-019-00255-1

Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis. https://doi.org/10.1007/s11263-009-0275-4

Flores CJL, Cutipa AEG, Enciso RL (2017) Application of convolutional neural networks for static hand gestures recognition under different invariant features. In: Proceedings of the 2017 IEEE 24th international congress on electronics, electrical engineering and computing, INTERCON 2017. https://doi.org/10.1109/INTERCON.2017.8079727

Geirhos R, Schütt HH, Medina Temme CR, Bethge M, Rauber J, Wichmann FA (2018) Generalisation in humans and deep neural networks. In: Advances in neural information processing systems

Huang H, Chong Y, Nie C, Pan S (2019) Hand gesture recognition with skin detection and deep learning method. J Phys Conf Ser. https://doi.org/10.1088/1742-6596/1213/2/022001

Islam MZ, Hossain MS, Ul Islam R, Andersson K (2019) Static hand gesture recognition using convolutional neural network with data augmentation. In: 2019 Joint 8th international conference on informatics, electronics and vision, ICIEV 2019 and 3rd international conference on imaging, vision and pattern recognition, IcIVPR 2019 with international conference on activity and behavior computing, ABC 2019. https://doi.org/10.1109/ICIEV.2019.8858563

Ji S, Xu W, Yang M, Yu K (2013) 3D Convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2012.59

Kim H, Albuquerque G, Havemann S, Fellner DW (2005) Tangible 3D: hand gesture interaction for immersive 3D modeling. In: 9th international workshop on immersive projection technology—11th Eurographics symposium on virtual environments, IPT/EGVE 2005

Kim S, Ji Y, Lee KB (2018) An effective sign language learning with object detection based ROI segmentation. In: Proceedings—2nd IEEE international conference on robotic computing, IRC 2018. https://doi.org/10.1109/IRC.2018.00069

Köpüklü O, Gunduz A, Kose N, Rigoll G (2019) Real-time hand gesture detection and classification using convolutional neural networks. In: Proceedings—14th IEEE international conference on automatic face and gesture recognition, FG 2019. https://doi.org/10.1109/FG.2019.8756576

Maqueda AI, Del-Blanco CR, Jaureguizar F, García N (2015) Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns. Comput Vis Image Underst. https://doi.org/10.1016/j.cviu.2015.07.009

Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural networks. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2016.456

Ni Z, Chen J, Sang N, Gao C, Liu L (2018) Light YOLO for high-speed gesture recognition. In: Proceedings—international conference on image processing, ICIP. https://doi.org/10.1109/ICIP.2018.8451766

Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl. https://doi.org/10.1007/s00521-016-2294-8

Rahmat RF, Chairunnisa T, Gunawan D, Pasha MF, Budiarto R (2019) Hand gestures recognition with improved skin color segmentation in human-computer interaction applications. J Theor Appl Inf Technol 97(3):727–739

Google Scholar  

Redmon J, Farhadi A (2018) YOLO v.3. Tech Report

Tzutalin (2015) LabelImg. LabelImg. https://github.com/tzutalin/labelImg

Walker A (2013) Voice commands or gesture recognition: how will we control the computers of the future? https://www.independent.co.uk/life-style/gadgets-and-tech/voice-commands-or-gesture-recognition-how-will-we-control-the-computers-of-the-future-8899614.html

Wan J, Li SZ, Zhao Y, Zhou S, Guyon I, Escalera S (2016) ChaLearn looking at people RGB-D isolated and continuous datasets for gesture recognition. In: IEEE computer society conference on computer vision and pattern recognition workshops. https://doi.org/10.1109/CVPRW.2016.100

Yang X, Tian Y (2014) Super normal vector for activity recognition using depth sequences. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2014.108

Zhang Y, Cao C, Cheng J, Lu H (2018) EgoGesture: a new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans Multimed. https://doi.org/10.1109/TMM.2018.2808769

Download references

Acknowledgements

This research was funded by Universiti Malaysia Sarawak under the UNIMAS publication support fee fund.

Author information

Authors and affiliations.

Faculty of Engineering, Universiti Malaysia Sarawak, 94300, Kota Samarahan, Malaysia

S. N. David Chua, K. Y. Richard Chin & S. F. Lim

School of Mechanical Engineering, VIT Bhopal University, Bhopal Indore Bypass, Sehore, 466114, India

Pushpdant Jain

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to S. N. David Chua .

Ethics declarations

Conflict of interest.

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Chua, S.N.D., Chin, K.Y.R., Lim, S.F. et al. Hand Gesture Control for Human–Computer Interaction with Deep Learning. J. Electr. Eng. Technol. 17 , 1961–1970 (2022). https://doi.org/10.1007/s42835-021-00972-6

Download citation

Received : 14 September 2021

Revised : 21 November 2021

Accepted : 23 November 2021

Published : 21 January 2022

Issue Date : May 2022

DOI : https://doi.org/10.1007/s42835-021-00972-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Hand gesture
• Human computer interaction
• Deep learning
• Object detection
• Find a journal
• Publish with us
• Track your research

School of Electrical and Computer Engineering

College of engineering, laboratory for emerging devices and circuits team wins best paper award for ai computing memory research.

The team, led by ECE professor Shimeng Yu, analyzed different combinations of settings for emerging non-volatile memory (eNVM) technologies in hopes of improving AI hardware efficiency and power.

Georgia Tech School of Electrical and Computer Engineering (ECE) professor Shimeng Yu and his team at the Laboratory for Emerging Devices and Circuits  won the Association for Computer Memory (ACM) Transactions on Design Automation of Electronic Systems (TODAES) 2024 Best Paper Award.

The prestigious award recognizes the best paper published in the TODAES, the ACM's flagship publications in the area of electronic design automation (EDA).

Yu accepted the award at the 61st Design Automation Conference in San Francisco, Calif. in June.

This is the second consecutive year Yu’s team won an award for research printed in a flagship publication in the area of EDA, and the third year in a row research from ECE has received such an honor.

The paper titled, “ Hardware-aware quantization/mapping strategies for compute-in-memory accelerators, ” analyzed different combinations of settings for emerging non-volatile memory (eNVM) technologies.

This new technology is important for mixed-signal Compute-in-Memory (CIM) accelerators, which are very energy efficient, thus making them crucial for artificial intelligence hardware design, which are notoriously resource intensive platforms.

Ultimately, the research found the right configuration and settings can significantly improve output and efficiency. Yu’s team was able to achieve an increase in processing speed by up to 60 percent, while doubling the energy efficiency and reducing overall hardware size by up to 25 percent.

The findings provide design guidelines to engineers who continue to research eNVM and CIM technology.

Yu co-authored the paper with ECE Ph.D. graduates Shanshi Huang and Hongwu Jiang, who are now both assistant professors in Hong Kong University of Science and Technology.

Yu’s lab won the IEEE’s Donald O. Pederson Best Paper Award in 2023 for their research on an end-to-end benchmark framework to evaluate state-of-the-art CIM accelerators. The award honors the best paper in the IEEE’s Transactions on Computer-Aided Design of Integrated Circuits and Systems, the flagship journal of the IEEE Council on Electronic Design Automation.

In 2022, ECE Professor Sung Kyu Lim and his research team won the Donald O. Pederson Best Paper Award for their paper on a physical design tool named Compact-2D that automatically builds high-density and commercial-quality monolithic three-dimensional integrated circuits.

Yu also recently received a 2023 Intel Outstanding Research Award for his work on a chip that will help quantify uncertainty that is beyond the capabilities of existing binary computing systems, and improve computing robustness.

Zachary Winiecki

[email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Bioengineering (Basel)
• PMC10215961

Empowering Hand Rehabilitation with AI-Powered Gesture Recognition: A Study of an sEMG-Based System

1 School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China

2 Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China

Mostafa Orban

3 Mechanical Department, Faculty of Engineering at Shoubra, Benha University, Cairo 11672, Egypt

4 School of Mechanical and Electrical Engineering, Changchun University of Science and Technology, Changchun 130001, China

Maged S. Al-Quraishi

5 Faculty of Engineering, Thamar University, Dhamar 87246, Yemen

Hongbo Yang

Mahmoud elsamanty.

6 Mechatronics and Robotics Department, School of Innovative Design Engineering, Egypt-Japan University of Science and Technology, Alexandria 21934, Egypt

Associated Data

The data presented in this study are available on request from the corresponding author.

Stroke is one of the most prevalent health issues that people face today, causing long-term complications such as paresis, hemiparesis, and aphasia. These conditions significantly impact a patient’s physical abilities and cause financial and social hardships. In order to address these challenges, this paper presents a groundbreaking solution—a wearable rehabilitation glove. This motorized glove is designed to provide comfortable and effective rehabilitation for patients with paresis. Its unique soft materials and compact size make it easy to use in clinical settings and at home. The glove can train each finger individually and all fingers together, using assistive force generated by advanced linear integrated actuators controlled by sEMG signals. The glove is also durable and long-lasting, with 4–5 h of battery life. The wearable motorized glove is worn on the affected hand to provide assistive force during rehabilitation training. The key to this glove’s effectiveness is its ability to perform the classified hand gestures acquired from the non-affected hand by integrating four sEMG sensors and a deep learning algorithm (the 1D-CNN algorithm and the InceptionTime algorithm). The InceptionTime algorithm classified ten hand gestures’ sEMG signals with an accuracy of 91.60% and 90.09% in the training and verification sets, respectively. The overall accuracy was 90.89%. It showed potential as a tool for developing effective hand gesture recognition systems. The classified hand gestures can be used as a control command for the motorized wearable glove placed on the affected hand, allowing it to mimic the movements of the non-affected hand. This innovative technology performs rehabilitation exercises based on the theory of mirror therapy and task-oriented therapy. Overall, this wearable rehabilitation glove represents a significant step forward in stroke rehabilitation, offering a practical and effective solution to help patients recover from stroke’s physical, financial, and social impact.

1. Introduction

Stroke has become a major contributor to human death globally, with over 10 million people suffering from it annually. This debilitating disease affects the lives of patients and their families, significantly reducing the quality of life of all involved [ 1 ]. Despite progress in stroke treatment and increased survival rates in recent years, there is still a critical need for advanced rehabilitation methods to speed up recovery and improve motor function in post-stroke patients. This is where the concept of assistive devices in rehabilitation comes into play. Stroke can cause severe damage to the central nervous system, resulting in various physical impairments and limitations. One of the most common injuries in stroke patients is the loss of wrist motion, a crucial joint connecting the hand and the arm. As such, rehabilitation training for patients with stroke and wrist injuries is essential for improving their overall health and well-being [ 2 , 3 ]. With the help of specialized rehabilitation techniques and assistive devices, stroke patients can regain some of their motor abilities and improve their quality of life.

The traditional approach to rehabilitation, which relies on one-on-one training involving a patient and a therapist, has proven ineffective and cost-prohibitive in today’s world. With the outbreak of the COVID-19 epidemic, face-to-face therapy sessions in hospitals had to be suspended, making the traditional approach to rehabilitation untenable [ 4 ]. To address this issue, rehabilitation robots have emerged as a solution by offering home-based rehabilitation programs. These programs allow patients to easily access rehabilitation training from the comfort of their own homes, reducing the need for face-to-face sessions during the pandemic [ 5 ]. Using assistive robots in rehabilitation has numerous advantages, including improved safety, efficiency, flexibility, and practicality. Many countries have recognized the need to use technology to mitigate the impact of disabilities and improve the quality of life of people with disabilities [ 6 ]. As a result, more people with disabilities rely on assistive devices to carry out daily activities and participate more actively and productively in community life [ 7 ]. However, assistive devices and technologies are still limited in many low- and middle-income countries. Only 5–15% of people who require these devices have access to them due to low production, limited quality, a lack of trained personnel, and prohibitive costs [ 8 ]. The need for affordable and accessible assistive technologies remains a critical challenge in many parts of the world, and more needs to be done to address this disparity.

The availability of assistive devices is critical for many countries’ economic and social development. It enables people with disabilities to participate in community activities, work, and earn an income, reducing the poverty levels [ 9 ]. As a result, rehabilitation and the development of assistive devices for stroke patients have become urgent and critical issues for many countries. To design effective rehabilitation assistive devices, it is crucial to understand their ideal properties for hand function and simulate simple and complex hand movements. The device should be comfortable, affordable, portable, and easy to use in clinical rehabilitation settings and at home [ 10 ]. A timely and efficient rehabilitation is critical, as prolonged gaps between therapy sessions can negatively impact rehabilitation. Assistive devices have been shown to speed up the rehabilitation process, and incorporating complementary therapies, such as mirror therapy and task-oriented therapy, can further enhance the effectiveness of rehabilitation [ 11 , 12 ]. Overall, the development and availability of assistive devices for rehabilitation are essential for promoting economic and social development, reducing the poverty levels, and improving the quality of life of people with disabilities.

In 2018, Randazzo et al. proposed a novel hand exoskeleton design to assist and rehabilitate individuals with impaired movement function and improve their quality of life. The device is wearable, portable, and does not impede hand mobility. It utilizes an EEG brain–machine interface to control hand movements through pattern detection. Finger actuation is achieved using Bowden cables, a linear actuator, with each finger having an active DOF for flexion and extension. The user can passively control abduction and adduction. The device is cost-effective, utilizing off-the-shelf parts as well as 3D printing and laser cutting. The actuator is housed in a closed box mounted on the chest, connected to the fingers via Bowden cables. The device is powered by a lithium–polymer (LiPo) battery, capable of continuous operation for up to 3 h [ 13 ].

In 2020, Butzer T. et al. designed the RELab tenoexo, a fully wearable, soft hand exoskeleton for daily activities, It consists of a hand module attached to the hand and a backpack containing electronic components, motors, and batteries. The backpack and hand module are connected by a force transmission system based on Bowden cables and can be attached by a clip mechanism. In the hand module, three DOF achieve dynamic, precision, and lateral grasping: a combination of 2–5-digit actuators for flexion/extension, an individual actuator for the thumb flexion/extension, and a manual lateral and oppositional pad for the thumb. Two DC motors are fed forward and controlled to drive the flexion/extension of the fingers and thumb through gear-rack mechanisms. The Myo armband (a wireless sEMG sensor for monitoring) is used [ 14 ].

In 2022, Marek Sierotowicz and colleagues from the Robotics and Mechatronics Department at the German Aerospace Center (DLR) designed an EMG-driven machine learning-controlled soft glove for assisting grasp and rehabilitation. The glove can assist the flexion and extension of the index finger and thumb through a tendon-driven system. The two actions, thumb and finger flexion, are independently assisted by electric motors that pull and release their respective tendon lines. The glove has a control system that recognizes attention through sEMG [ 15 ].

Despite these efforts, we think that the shape of the device and the fact that it has to be carried on the chest or a backpack still have some negative implications for the user, especially for stroke patients who usually suffer from a lack of self-confidence within a community, which can affect the progress of the training beside the short time of use of 3 h. In addition, the size and the weight of the device have a negative effect on the comfort of the user.

This article presents a hand rehabilitation system with a soft glove that supports mirror, task-oriented therapies, and neural plasticity. The actuated glove, integrated with a linear actuator, provides the driving force for the affected hand during rehabilitation training. The sEMG sensor enables cooperation between both hands and was used to control the actuated glove to let the affected and the non-affected hands move simultaneously. An integrated sensor measures the bending angle and the progress of the rehabilitation programmer. A machine learning algorithm was developed to classify the sEMG gestures as a control command for the actuated glove. This gives the affected hand the suitable driving force to perform the corresponding motions. Our glove is superior to previous devices because of its lightweight, the soft material used for its fabrication, its long-lifetime batteries that can support 4–5 h of use, and its low cost, which is a good advantage for low-income countries, as discussed before. The device is built so that all of its parts are hidden under the shoulder and under the clothes, and the only visible part is the glove placed on the wrist, which positively affects the user’s self-confidence.

Moreover, our glove can train each finger alone to perform flexion and extension with the integrated sensor to measure the angle of bending that reflects the progress of the rehabilitation program. Finally, our glove is based on two verified therapy theories, mirror therapy and task-oriented therapy, using sEMG. The telecooperation between the affected and the unaffected hand has also a good effect on the progress of the therapy training.

2. Materials and Methods

2.1. related work, 2.1.1. hand design and actuation.

The research and development of hand rehabilitation equipment are still in their infancy. Most exoskeleton-hand robots focus on improving finger joints and increasing muscle strength, ignoring the vital role of active brain participation and the sensory function input into the motor function. Therefore, the effect of fine motor rehabilitation on hand function is not considered. At present, the hand-function rehabilitation robots design adopts an exoskeleton structure, which mainly provides a driving force through motors and pneumatics to complete flexion and extension of the fingers and joints. Four main driving methods have been used for hand-function rehabilitation robots. They mainly include a motor drive, a pneumatic artificial muscle drive, a memory alloy drive, and a lasso drive. Because the lasso drive has good flexibility and linearity, we used a lasso drive to design hand-functional rehabilitation robots.

2.1.2. Recognition Methods of the sEMG Signals

The electrical voltages in sEMG signals range from −5 to +5 (mV) and are influenced by both the movements performed and the muscle extraction and contraction level. The continuous availability and variability of the signal can be measured using a suitable detection element [ 16 , 17 , 18 , 19 , 20 , 21 ]. These signals have significant potential in rehabilitation, where they can be combined with an appropriate recognition system. However, given that the sEMG signals traverse multiple tissues and muscles before being acquired from human muscles [ 22 ], they are susceptible to interference from crosstalk, obstructions, and noise. Different machine learning and A.I. techniques have been used to classify and recognize the EMG signals. The classifier can be a Supported Vector Machine (SVM) [ 23 ], the k-nearest neighbor (kNN) algorithm [ 24 ], linear discriminant analysis (LDA) [ 25 ], and a neural network (N.N.) [ 26 ], with different classification accuracy (C.A.) and complexity for different methods.

Using a neural network with a backpropagation or fuzzy recognition system [ 27 , 28 ] may not be optimal due to its slow learning in many cases. In the EMG-decoding window length study, a mean testing performance of 94.21% ± 4.84% after voting was demonstrated through visual inspection [ 29 ]. In another study, supervised feature extraction techniques achieved initial accuracy rates of 91.54% and 80.40%, with additional accuracy rates of 91.71% and 91.44% [ 30 ].

A highly accurate decoding and classification method for sEMG signals was developed in this research, which can be used as mirror therapy to mirror the motion from the non-affected hand in the affected hand. The wearable glove developed in this study can perform 16 desired motions, including 6 task-oriented therapy-based motions, and is cost-effective and comfortable. This glove can be used for home-based rehabilitation and is not limited to clinical environments. The individual finger movement and the cooperation of multiple fingers can be trained using this glove. The system design and fabrication process, as well as the methodology used in the research, are described in this article, which begins with sEMG implementation and experiment details in the mirror therapy scenario and ends with data preprocessing and classification.

2.2. Design and Manufacture of the Hand Robot

This study aimed to introduce a wearable rehabilitation glove based on flexible lasso transmission driven by a flexible rope and a linear actuator. The glove was designed to assist patients with hand impairments in the rehabilitation process and enable them to perform routine activities in daily life, which can boost their confidence and promote their independence. As shown in Figure 1 , the glove can simulate the functions of the human hand and facilitate the recovery of patients with various hand injuries or conditions.

Flexible Hand Rehabilitation Robot with a Linear Actuator and an sEMG Sensor.

The flexible-rope driving method used in the glove design provides several advantages, including a simple structure, a lightweight design, substantial flexibility, comfortable wearing, and convenient use. The flexible rope acts as a transmission mechanism that can transmit power from the linear actuator to the glove without a significant loss of power. This design enables the glove to provide the necessary driving force to assist the patient’s hand movements during the rehabilitation process, while also ensuring the patient’s comfort and safety.

2.2.1. Host Machine for the Hand Rehabilitation Robot

The lightweight design of the glove allows patients to wear it for extended periods without feeling uncomfortable or fatigued, making it suitable for long-term use. Additionally, the flexible lasso transmission provides a substantial range of motion, allowing patients to perform various hand movements without feeling restricted or inhibited. This range of motion is essential in promoting hand function and mobility recovery, which can be challenging for patients with hand injuries or conditions.

The wearable glove presented in this study is primarily composed of a main control module, a separation module, a glove connection module, and wearing gloves, as illustrated in Figure 2 . The portable, wearable robot main control module comprises five linear actuators, a battery, hardware, and a screen, all housed within a compact structure. The separator module effectively combines the central control module and the rehabilitation gloves, enhancing the system’s portability and ease of use. The host part dimensions are 220 mm × 94.15 mm × 34 mm, and the host weighs 476.68 g.

Main module of the hand rehabilitation robot.

The system components are shown in Figure 3 . The compact and lightweight design of the wearable glove makes it easy to use for patients, and the system’s portability allows it to be used in various settings, including clinical and home-based settings. The main control module, which includes the linear actuators, battery, hardware, and screen, was specifically designed to be portable and wearable, allowing for a more flexible patient rehabilitation experience. The separation module is an essential system component, effectively combining the central control and the rehabilitation gloves. This approach makes managing and using the system in various settings easier and enhanced its overall portability and ease of use.

Flexible Glove System Components for sEMG-based Hand Rehabilitation.

The dimensions and weight of the host part were carefully selected to ensure that the wearable glove is compact and lightweight. These features ensure that patients could use the glove comfortably and without undue strain, allowing for the glove to be used over an extended period.

2.2.2. Glove and Finger Structure

The structure of the wearable glove used for hand rehabilitation is shown in Figure 4 . The outer layer of the glove is composed of a spring tube that remains stationary, while the inner layer is a cored wire with a steel wire lining, which drives the movement of the fingers through the movement of the cored wire. The design incorporates two groups of ropes for each finger, one to drive the finger to bend on the inner side, and the other to drive the finger to straighten on the dorsal side. By pulling the ropes, the glove mimics the natural movement of the fingers, allowing the wearer to perform a range of exercises to improve the motor function. We used a pneumatic tube to reduce friction between the cored wire and the outer spring sleeve. A wire sleeve-fixing structure was used inside the palm. Considering the interference with finger movement, we designed the casing-fixing device and flexible transmission connection as shown in Figure 4 .

Flexible transmission and glove with sensors.

To monitor the wearer’s progress, pressure sensors were placed at the fingertips to detect the pressure exerted by the fingers during exercises. This information can be used to adjust the glove’s level of assistance, ensuring that the rehabilitation program is tailored to the wearer’s needs.

2.2.3. Driving System with Linear Actuator and Flexible Lasso

The wearable glove utilizes linear actuators as a power source to address the need for a portable and compact wearable glove for home-based rehabilitation and daily life assistance. Given that the flexible glove requires bending and straightening each finger, an electric cylinder is not feasible since it can only provide one function. To address this, a specialized linear motor was designed with meshing gears behind the motor reducer, resulting in two outputs in opposite directions, as illustrated in Figure 5 . Specifically, the two output sliders move in opposite directions, with one bending the finger by pulling on the inner side of the finger, and the other used to straighten the finger by pulling on the back of the finger. This design ensures the appropriate stroke of each finger for effective rehabilitation and assistive purposes.

Structure of the linear actuator with bidirectional output.

The control machine and separation structure are connected flexibly, as depicted in Figure 6 . The equipment host contains the motion unit, which is the electric cylinder designed in this study, and each electric cylinder is linked to two flexible transmission lines. The transmission line comprises a spring tube, a plastic tube, and spring steel wire. The outer layer is covered with cloth to transmit power and ensure bending while simplifying the assembly process. The internal execution device consists of a moving wire that pulls a finger back and forth under the action of a linear actuator.

Control machine and separation structure.

Consider the decoupling design between the central control module and the glove module of wearable rehabilitation robots. In this paper, a separator was designed between the central control module and the glove module to decouple the movement. The splitter makes the signal transmission part of the sensor separable, through spring pins and contacts on the circuit board. The structure and hardware composition of the separator are shown in Figure 7 . The motion-coupling structure is completed by a T-type device and a groove-type device. Kevlar wires connect the T-type components and the glove end, and the groove-type components and the central control module are connected by wire. The steel ribbon moves the grooved components, thereby driving the T-shaped components.

Structure and composition of the separator.

2.2.4. Control Protocol of the Hand Robot

The wearable glove controller utilizes the STM32 microcontroller as its central processing unit, with a 12 V lithium battery as the power source for driving the actuators. A serial touch screen enables communication via a serial interface with the central control hardware. Additionally, switch, mode selection, and intensity adjustment buttons were incorporated for selecting the rehabilitation mode during home-based rehabilitation training. A pressure sensor and a resistive linear displacement sensor were integrated into the design to monitor the fingers’ condition and movements within the rehabilitation device. The device is equipped with a lithium battery with a capacity of 11.84 Wh, a standard voltage of 7.4 V, and a nominal capacity of 3200 mAh. During grip mode testing, the power consumption was measured to be 3.7 W, and the device’s continuous exercise time was calculated to be 3.2 h. However, due to the device’s low power consumption during exercise gaps, actual laboratory test rehabilitation training can run for up to 4 h. Figure 8 illustrates the various hardware components of the controller.

Controller’s hardware components.

The WiFi module of USR-C322 was built into the device to communicate with other control modules through UDP. Network communication ensures that the upper computer can control and display the real-time state of the glove, as shown in Figure 9 .

Control instructions of the hand robot.

Equipment control and equipment status monitoring can be realized through the instructions in the figure above.

The equipment described in this paper mainly uses finger motion control instructions, and the equipment developed and used in this paper does not include vibration and electrical stimulation accessories.

The equipment sends the return instruction of equipment status (data type 0x01) as follows:

It includes the protocol frame header 0x00AA00CC, data type 0x01, equipment number 0x0001, three bytes used by a five-way vibration motor 0x111110, five-way displacement status 0xEEEEE0, whether electrical stimulation 0x01, frame tail 0x0D0A0D0A.

When we need to control the equipment, we send a control instruction with data type 0xA2 to the corresponding equipment.

At the same time, we use the following instructions as the heartbeat packet mechanism of the device.

The upper computer sends heartbeat packets (data type A0) to the device regularly:

After receiving the heartbeat packet, the device replies to the data with the device number, usage mode, and power information:

By adding a WiFi module in the development and compiling the program to realize a UDP device control protocol, the device can be controlled by other devices.

2.3. Research on Gesture Recognition Algorithms

2.3.1. gestures and device settings.

Gesture recognition was designed to simulate and fit the complexity of real hand-finger movement [ 31 , 32 , 33 , 34 , 35 , 36 , 37 ]. Therefore, human hand gesture can be classified into four groups as follows: the first group includes F.H., fisted hand, H.O., hand open, and F.F., finger flexion, and represents different movements of all fingers together. The second group includes the IMRlF, i.e., index, middle, ring, and little fingers, flexion that mimics the good sign, the MRLF—middle, ring, little fingers—flexion, mimicking using a touch screen of an elevator, the TRL—thumb, ring, and little fingers—flexion, mimicking the peace sign, the TMRI—thumb, middle, ring, and little fingers—flexion, mimicking pointing at something, the TIM—thumb, index, and middle fingers—flexion, mimicking grasping some paper. The second group gestures are complex gestures of the hand that exhibit hand flexibility for performing different and complex hand daily movement. The third group includes T.F., thumb, flexed, TIF, thumb and index fingers, flexed to indicate the thumb movement and its cooperation with other fingers. The fourth group includes finger flexion (T.F., IF, M.F., and R.F.), focusing on single finger movements; training can be home-based using task-oriented rehabilitation. These mimicked gestures are shown in Figure 10 .

Different gesture designs fit the complexity of the real hand–finger movements.

A total of eight healthy subjects, six males and two females, all right-handed except one male and between the ages of 21 and 30 years, participated in the data acquisition. The data were collected using (yw-wireless. Beijing Changfeng Technology Co., Ltd., Beijing, China). Four surface electrodes were placed on the forearm muscles, which are four muscles in total (brachioradialis, extensor carpi radialis longus, flexor carpiulnaris, and extensor digitorum), as shown in Figure 11 .

EMG-electrodes placement on the forearm muscles.

sEMG signals were collected at a sampling rate of 1000 Hz, and each gesture was performed 12 times, each time consisting of four runs, each one lasting for 4000 ms; and there was a rest period of 3000 ms to avoid causing muscle fatigue, as shown in Figure 12 . The data were collected while maintaining each gesture for 4000 ms, to ensure the presence and the quality of the collected sEMG signals.

The Setup and the scheduling for the EMG experiment.

This work divided pattern recognition into three main steps: data preprocessing, feature extraction, and pattern recognition. Pattern recognition was divided into two stages, i.e., classifier training and classifier prediction, using new signals from the non-training data.

2.3.2. Data Preprocessing and Feature Extraction

Initially, we selected pertinent data located in the intervals of [(6001:10,000), (18,001:22,000), (30,001:34,000), (42,001:46,000)], which contained EMG signals relevant to the targeted motion. Next, we proceeded with the raw data pre-processing, where the raw signals were trimmed by a butter band-pass filter, ranging from 5 to 480 Hz in areas where an sEMG was present, and further filtered by a notch filter between 48–52 Hz.

To obtain a large set of training data that could be used for pattern recognition training, we used the segmentation overlapped method for extracting useful time-domain features such as mean square value (MAV), root-mean-square (RMS), variance absolute value (VAV), integrated electromyography (Iemg), simple square integral (SSI), waveform length (WL). The extracted featured data contained useful information such as signal strength and amplitude. Later, the wrapper backward method was used to obtain new features. The wrapper method was also used as a feature selection method to select the optimal feature subset by determining the importance of each feature subset independently using a real training test to identify the feature that positively affected the recognition system accuracy. Therefore, we could select the optimal feature subset from the original features. In the feature extraction process, the window length was 200 ms, and the overlap of the adjacent window was 50 ms. The data collected from a single muscle were divided into 317 windows. The 200 ms data window set was a good choice containing enough information to define the contained gesture fully. The 50 ms overlap window utilized the data set to produce a refined, dense classification technique to meet the real-time classification needs.

3.1. Accuracy of Gesture Recognition

This article used the PyQt5 framework to build software for EMG acquisition and raw data storage, as shown in Figure 13 . In real time, the software effectively displayed the four lead EMG signals alongside the real-time predicted gestures, providing accurate results to the users.

Software interface of four lead EMG.

We adopted the 1D-CNN algorithm and the InceptionTime algorithm. The above-collected data were intelligently identified. The obtained iteration accuracy and confusion matrix of ten gestures are shown in Figure 14 .

Confusion matrix and iteration accuracy of different algorithms, ( a ) using the 1D-CNN algorithm, ( b ) using the InceptionTime algorithm.

Figure 14 shows the accuracy achieved by using the two algorithms and the accuracy for specific gestures. In Figure 14 a, we adopted the 1D-CNN algorithm, achieving an accuracy of 89.52% in the training set and of 76.84% in the verification set, with an overall accuracy of 80.98%. In Figure 14 b, we adopted the InceptionTime algorithm, achieving an accuracy of 91.60% in the training set and of 90.09% in the verification set, with an overall accuracy of 90.89%.

3.2. Control of the Hand Rehabilitation Robot Based on sEMG

The sEMG drive of the hand rehabilitation robot using the framework is shown in Figure 15 .

Framework of communication and control.

Based on the above framework, we developed an online recognition software for grasping and extending a hand using the above algorithm, as shown in Figure 16 . We implemented the control of the rehabilitation robot based on this interface.

Online recognition software for grasping and stretching.

The corresponding flowchart is shown in Figure 17 . The algorithm was developed with the help of the eight people, and data collection was completed after setting gesture sequences and collection methods. Subsequently, data filtering and other preprocessing operations were carried out, and relevant features in the time and frequency domains were extracted, ultimately completing the development of the algorithm. For the online control of robots, the first step is to select the mirror mode through a screen, and the computer and device communicate through the TCP protocol. After obtaining electromyographic data through the computer software, the data were processed and algorithmically recognized, ultimately outputting the motion instructions for the robot.

Flowchart of the hand robot control with sEMG.

The complete hand rehabilitation robot we constructed as described in this study is shown in Figure 18 .

Hand rehabilitation robot that was developed in this study.

The control of the rehabilitation robot was realized by sEMG driving, as shown in Figure 19 . This Figure also shows different snapshots of the affected hand assisted by the wearable glove.

Use of the sEMG device and software for mirror therapy.

4. Discussion

A highly accurate decoding and classification method for sEMG signals was developed in this research, resulting in a classification accuracy of 90.89%. The contribution of this work surpasses that of previous hand rehabilitation systems. The wearable glove that was created can accurately perform ten desired motions, including six task-oriented therapy-based motions. Additionally, the glove is inexpensive and exceptionally comfortable. It can be used as a home-based rehabilitation system and is not limited to clinical environments. The glove can train individual finger movements and multiple finger cooperation, with a latency time of less than 300 ms, thus satisfying online training requirements. The controller, glove, and glove structure were designed to align with human–machine integration and ergonomics, as hand-function robots are used for rehabilitation and muscle strength enhancement. The host structure has no edges or sharp corners, with round edges made of metallic silver. The wearable device mainly operates through a touch screen, offering good man–machine interaction characteristics.

The core actuator of the robot was designed independently and can output a two-way rope-pulling force. Generally, the rope can only transmit the pulling force, and two actuators are needed to realize a bidirectional drive. In this paper, the actuator has two reverse outputs; so, a single actuator can directly drive a finger forward and backward. The transmission adopts an optimized lasso transmission design, which is closer to the movement characteristics of human muscles. Compared with other flexible drives, the lasso drive has the characteristics of fine-grained operation, reliable transmission, and high safety. In this paper, the lasso drive presents a single-tube bidirectional lasso that can realize a finger’s flexion and extension.

The glove part considers the palm movement, a challenge for most wearable hand devices, and the design is lighter and more convenient to wear. A flexible thin-film pressure sensor was placed at the fingertips of the glove to collect pressure intended for muscle strength enhancement. The driving rope was sewn into the glove, obtaining a solution close to the real driving force of the muscle. The whole machine has a WIFI communication interface that grants intelligent control through the protocol.

In the offline training, the InceptionTime algorithm had the highest accuracy among all the algorithms. It showed less prediction time and required less training time than the 1D-CNN algorithm, and the average difference in accuracy between them was ±9.2%. We decided to use the InceptionTime algorithm for its low prediction time, which is important for our application. The highest recorded accuracy was 90.89%, which satisfies our wearable glove application.

5. Conclusions

In conclusion, the presented wearable hand rehabilitation system has the potential to significantly impact the motor recovery of paresis and hemiparesis patients. Actuated gloves with sEMG sensing are a flexible, wearable technology that is safe, comfortable, and portable. Additionally, the system can be affordable for low- and middle-income countries, which have limited access to rigid exoskeleton devices due to their high cost. Compared to dedicated-data gloves with targeted sensors, the wearable glove utilizing biomedical signals offers improved signal quality and higher accuracy in detecting the desired motions during rehabilitation training. This connection between the desired motion activated by the brain and sent to the muscle and the actual movement executed by the actuated glove shown on a screen can speed up the rehabilitation process based on the mirror therapy technique. However, it is worth noting that achieving a fine-grained classification of the training gestures still requires the precise placement of the EMG electrode. This study presents a promising wearable rehabilitation system for hand motor recovery that can be used in various settings, including clinical and home-based environments.

Funding Statement

This work was supported by the Natural Fund of Shandong Province (Grant no. ZR2022QH214), and the Key R&D Plan of Jiangsu Province (Grant no. BE2021012).

Author Contributions

Conceptualization, H.Y. and K.G.; methodology, M.O. and K.G.; software, K.G.; validation, M.O., M.S.A.-Q., and J.L.; formal analysis, M.O.; resources, H.Y.; data curation, M.O.; writing—original draft preparation, K.G., M.E., and M.O.; writing—review and editing, K.G., M.E. and M.O.; visualization, M.O.; supervision, H.Y.; project administration, H.Y. and K.G.; funding acquisition, K.G. and H.Y.; K.G. and M.O. contributed equally to this work as first authors. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Conflicts of interest.

The authors declare no conflict of interest.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Skip to Content

Current Students

Interested in more? Search Courses

• Search Input Submit Search

9 of DU’s Coolest Undergrad Research Projects

Emma atkinson, supported by the undergraduate research center, dozens of students got the chance to present their work at this spring’s showcase..

From science to the arts, students from departments across the University of Denver gathered this spring to present their research projects at the annual Undergraduate Research Showcase.

The Undergraduate Research Center awarded many of these students grants to conduct their research, which culminated in a diverse display of creative and engaging projects. The research represents the a dvancing intellectual growth dimension of the 4D Experience. 

“DU’s undergraduate research activities are truly premier, providing students with rigorous opportunities to work with thought leaders from across campus—flexing their muscles as thought explorers, translating and learning critical skills, and building the confidence to tackle the problems of tomorrow,” says Corinne Lengsfeld, senior vice provost for research and graduate education.

Here are descriptions of nine of this year’s coolest research projects, along with what students told us about their experience.

Saint Brigit and Her Habits: Exploring Queerness in Early Medieval Ireland

Jacqueline Stephenson

“By examining how scholars have used a lens of queerness and mediation to analyze key hagiographies—first of saints, more broadly, and then of female saints—and using these approaches to examine the inherently queer actions, positions and roles found in Saint Brigit’s First Life, we can gain a clearer view of societal views towards queerness during the early medieval period and in early medieval Ireland, in particular.

In doing so, this thesis will help chip away at the monolithic view of the period as well as the queer erasure within—demonstrating that queerness has always been a fundamental part of human society.”

What was the best part about doing this research?

The best part about doing this research was actually contributing to the scholarship and demonstrating that queerness has always been here, even in the places we might least expect. Additionally, seeing concepts we often don't associate with religion and this period were fascinating.

An example is one of her miracles being essentially an abortion, where a young unmarried woman didn't want to be pregnant and came to Saint Brigit, who made the fetus disappear. This is literally a celebrated miracle and not condemned—something shocking considering present-day religious attitudes toward abortion.

Questioning the Effectiveness of the Olympic Truce

Vincent Pandey

“A modern model for peace is the Olympic Truce, a United Nations General Assembly resolution that calls for the pausing and prevention of new conflicts from one week before the Olympic Games through one week after the Paralympic Games. Some argue that the symbolic nature of the truce allows it to create moments of peace in conflicts, while others argue that it is nothing more than a gesture of goodwill that has not actually been used for peace.

My research question is: Has the Olympic Truce ever been successfully implemented to prevent the onset of new conflict or in creating a ceasefire during an ongoing conflict? I use conflict data and case studies to determine the prevalence of conflict during Olympic Truce periods and four case studies to analyze attempts to use the Olympic Truce and some of the challenges faced.”

Hawai'i Uncovered

Lauren Tapper

“Through photography, Hawai'i Uncovered aims to explore the conflicting aspects of identity in Hawai'i, reveal the true characteristics of the state and separate its perception by tourists from the reality that locals know all too well. Photos were taken at popular tourist and lesser known ‘locals-only’ destinations on multiple of the Hawaiian Islands. The photos focused mostly on the way that people interact and exist within these places and amongst each other through a candid and observational lens.

The idea of community and belonging are both the saving grace and downfall of these islands, allowing many to be proud and excited about who they are while also leaving some cast out and forgotten. Both the romanticized and ignored aspects of these islands are what make the state unlike anywhere else in the world and are essential in defining Hawai'i in an honest fashion.”

How did the idea for this research come about?

I got the idea for this research as someone who was born and raised in Hawaii and came to the mainland to notice that there was a huge discrepancy between what people think they know about Hawaii versus what is true, or at least what I know to be true.

Flower Study

Haven Hinds

“This project is a study of extinct flowers, their histories, environments, biology, colors and assigned personalities—manifested in 3D models and a digital book. The goal was to select six flowers with interesting histories and/or cultural connections. Since the flowers can, for the most part, not be seen or kept, they were created in Blender as accurately as possible. To give people a means of connection with the flowers, each was assigned a personality based on different factors. These factors could be the colors they possess, where geographically they thrived, parts of their history or biological factors.

To allow these personalities to further flourish, color spaces were created in Affinity Designer to be applied not only as the background of the models but also as the background of half of the book’s pages. Each person who interacts with the book is encouraged to pick out flowers they connect to and create their own garden of these now-gone flowers.”

The best part about doing this research was finding out some of the love stories connected to these flowers. I am a hopeless romantic, so I enjoyed even the tragic ending to the ancient stories connected to the flowers. Their stories allowed me the opportunity to give the flowers personality traits, allowing others to further connect.

A Midsummer Night’s Dream Specialty Props

“ʽA Midsummer Night’s Dream’ is one of Shakespeare’s most renowned plays, steeped in a world of dreams and magic, and will be the second show in the Department of Theatre’s winter quarter season. This show has the goal of bringing in college audiences that have not had the full experience of witnessing live theatre and who may have previous, negative notions about Shakespearean plays. The goal was to transform our audience for the two hours they witnessed this play into people that can enter a world of magic, try something new, appreciate it, challenge their own thinking and revitalize an art that took a heavy blow in recent years.

In order to achieve this fantastical reality onstage, I worked closely with the costume designer (Janice Lacek) and director (Sabin Epstein) to research, design, prototype and fabricate specialty props and costumes to create the multiple distinct worlds present in ‘A Midsummer Night’s Dream.’ By the end of the project, I designed, prototyped and fabricated five large-scale specialty props sporting at least 50 separate pieces, each using advanced painting and texturing methods, LEDs lights and wiring techniques, foam molding, paper mâché, and wig making techniques.”

Immersive Stargazing: Leveraging VR for Astronomy Education

“This project leverages virtual reality (VR) technology to create an immersive stargazing experience that makes the wonders of the cosmos accessible to all. By integrating VR with educational strategies, we aim to revolutionize the teaching and experience of astronomy. Our primary objective is to enhance astronomy education by developing a VR platform that transcends conventional teaching methods, making celestial phenomena accessible and engaging for users from diverse educational and geographical backgrounds.

This initiative will democratize access to astronomy, promote STEM education and introduce users to advanced technological learning tools. The approach involves constructing detailed celestial models and integrating them into a user-friendly VR interface.”

Flamethrower Vol. 1-3: Innovation in Multidisciplinary Electronic and Acoustic Music

Trevor Briggs

“Flamethrower Vol. 1-3 is a series of three EPs (short albums) that combine jazz, classical and commercial electronic music to produce an artistic work in which the perception, function and musical context of electronic instruments is challenged. I approached these compositions treating electronic components as instruments, writing out scores as I would for brass and wind instruments. I also utilized cutting-edge electronic hardware to generate degenerated, fragmented and evolving sounds.

This type of in-depth sound design allowed me to write, perform and record novel parts for unique instruments. The remainder of the sounds on these recordings incorporate sampling of field recordings to generate new instruments or soundscapes and unconventional recordings of woodwinds (saxophone, clarinet, flute, bass clarinet and alto flute) from Professor Remy Le Boeuf, my faculty partner.”

The best parts of this project were tiny moments where I was really able to see how the final product was coming together. These times came through all parts of the writing and production process. I recorded Professor Remy Le Boeuf on woodwinds a few times for the album, and those sessions were full of those moments. There's a lot of momentary joy in making music with other people. As a recording engineer and producer, I have been feeling very lucky and grateful that I'm the one that gets to capture and shape these moments for others to listen to.

Technology and Homelessness: How Accessibility and Blockchain Technology Could Impact the Unhoused

“This paper discusses how the unhoused population suffers at the hand of technological inequality despite being relatively offline. It presents theories on how this would change if we reapproached how technology is used to assist the unhoused. It suggests implementing blockchain as a resource as well as modifying the websites built for the unhoused.

Employees at shelters are interviewed for this paper about their experiences with using digital resources to rehouse and restabilize the vulnerable. They are asked how the sites can be improved for more optimized use. The sites are also tested against current user experience (UX) standards for accessibility.”

My idea for this research came from two different sources. The first was the approximately 100 hours of volunteer work that I did with a local organization that works with the unhoused. I worked with caseworkers and other employees to get a better understanding of how the organization was run and what major needs were held by the unhoused population. It opened my eyes to a lot of problems that I hadn’t thought about before, especially how difficult it is to escape homelessness and why it is so difficult. 

The second source came later, when I was already working on my thesis paper. I was studying abroad in Greece and took a really interesting UX design class. That class made me start thinking about what UX looked like throughout my life, so when I was looking at government sites to understand where the unhoused would need to be using personal identification, I started noticing all these UX problems that would be easy to fix, which was baffling and frustrating to me, so I decided to add a second part to my paper.

Data Quality Matters: Suicide Intention Detection on Social Media Posts Using RoBERTa-CNN

“Suicide remains a global health concern for the field of health, which urgently needs innovative approaches for early detection and intervention. This paper focuses on identifying suicidal intentions in SuicideWatch Reddit posts and presents a novel approach to detect suicide using the cutting-edge RoBERTa-CNN model, a variant of RoBERTa (Robustly Optimized BERT Approach). RoBERTa is a language model that captures textual information and forms semantic relationships within texts.

By adding the Convolution Neural Network (CNN) head, RoBERTa enhances its ability to capture important patterns from heavy datasets. To evaluate RoBERTa-CNN, we experimented on the Suicide and Depression Detection dataset and obtained solid results. For example, RoBERTa-CNN achieves 98% mean accuracy with the standard deviation (STD) of 0.0009. It also reaches over 97.5% mean area under the curve (AUC) value with an STD of 0.0013. Then, RoBERTa-CNN outperforms competitive methods, demonstrating the robustness and ability to capture nuanced linguistic patterns for suicidal intentions. Hence, RoBERTa-CNN can detect suicide intention on text data very well.”

Related Articles

Girls in STEM: What 3 Professors Are Doing to Empower the Next Generation

DU Cyber Security Researcher Unveils Risks of Virtual Assistants

Ritchie School Students Build Assistive Technology

Suggestions or feedback?

MIT News | Massachusetts Institute of Technology

• Machine learning
• Social justice
• Black holes
• Classes and programs

Departments

• Aeronautics and Astronautics
• Brain and Cognitive Sciences
• Architecture
• Political Science
• Mechanical Engineering

Centers, Labs, & Programs

• Abdul Latif Jameel Poverty Action Lab (J-PAL)
• Picower Institute for Learning and Memory
• Lincoln Laboratory
• School of Architecture + Planning
• School of Engineering
• School of Humanities, Arts, and Social Sciences
• Sloan School of Management
• School of Science
• MIT Schwarzman College of Computing

A prosthesis driven by the nervous system helps people with amputation walk naturally

Press contact :.

Previous image Next image

State-of-the-art prosthetic limbs can help people with amputations achieve a natural walking gait, but they don’t give the user full neural control over the limb. Instead, they rely on robotic sensors and controllers that move the limb using predefined gait algorithms.

Using a new type of surgical intervention and neuroprosthetic interface, MIT researchers, in collaboration with colleagues from Brigham and Women’s Hospital, have shown that a natural walking gait is achievable using a prosthetic leg fully driven by the body’s own nervous system. The surgical amputation procedure reconnects muscles in the residual limb, which allows patients to receive “proprioceptive” feedback about where their prosthetic limb is in space.

In a study of seven patients who had this surgery, the MIT team found that they were able to walk faster, avoid obstacles, and climb stairs much more naturally than people with a traditional amputation.

“This is the first prosthetic study in history that shows a leg prosthesis under full neural modulation, where a biomimetic gait emerges. No one has been able to show this level of brain control that produces a natural gait, where the human’s nervous system is controlling the movement, not a robotic control algorithm,” says Hugh Herr, a professor of media arts and sciences, co-director of the K. Lisa Yang Center for Bionics at MIT, an associate member of MIT’s McGovern Institute for Brain Research, and the senior author of the new study.

Patients also experienced less pain and less muscle atrophy following this surgery, which is known as the agonist-antagonist myoneural interface (AMI). So far, about 60 patients around the world have received this type of surgery, which can also be done for people with arm amputations.

Hyungeun Song, a postdoc in MIT’s Media Lab, is the lead author of the paper , which appears today in Nature Medicine .

Sensory feedback

Most limb movement is controlled by pairs of muscles that take turns stretching and contracting. During a traditional below-the-knee amputation, the interactions of these paired muscles are disrupted. This makes it very difficult for the nervous system to sense the position of a muscle and how fast it’s contracting — sensory information that is critical for the brain to decide how to move the limb.

People with this kind of amputation may have trouble controlling their prosthetic limb because they can’t accurately sense where the limb is in space. Instead, they rely on robotic controllers built into the prosthetic limb. These limbs also include sensors that can detect and adjust to slopes and obstacles.

To try to help people achieve a natural gait under full nervous system control, Herr and his colleagues began developing the AMI surgery several years ago. Instead of severing natural agonist-antagonist muscle interactions, they connect the two ends of the muscles so that they still dynamically communicate with each other within the residual limb. This surgery can be done during a primary amputation, or the muscles can be reconnected after the initial amputation as part of a revision procedure.

“With the AMI amputation procedure, to the greatest extent possible, we attempt to connect native agonists to native antagonists in a physiological way so that after amputation, a person can move their full phantom limb with physiologic levels of proprioception and range of movement,” Herr says.

In a 2021  study , Herr’s lab found that patients who had this surgery were able to more precisely control the muscles of their amputated limb, and that those muscles produced electrical signals similar to those from their intact limb.

After those encouraging results, the researchers set out to explore whether those electrical signals could generate commands for a prosthetic limb and at the same time give the user feedback about the limb’s position in space. The person wearing the prosthetic limb could then use that proprioceptive feedback to volitionally adjust their gait as needed.

In the new Nature Medicine study, the MIT team found this sensory feedback did indeed translate into a smooth, near-natural ability to walk and navigate obstacles.

“Because of the AMI neuroprosthetic interface, we were able to boost that neural signaling, preserving as much as we could. This was able to restore a person's neural capability to continuously and directly control the full gait, across different walking speeds, stairs, slopes, even going over obstacles,” Song says.

A natural gait

For this study, the researchers compared seven people who had the AMI surgery with seven who had traditional below-the-knee amputations. All of the subjects used the same type of bionic limb: a prosthesis with a powered ankle as well as electrodes that can sense electromyography (EMG) signals from the tibialis anterior the gastrocnemius muscles. These signals are fed into a robotic controller that helps the prosthesis calculate how much to bend the ankle, how much torque to apply, or how much power to deliver.

The researchers tested the subjects in several different situations: level-ground walking across a 10-meter pathway, walking up a slope, walking down a ramp, walking up and down stairs, and walking on a level surface while avoiding obstacles.

In all of these tasks, the people with the AMI neuroprosthetic interface were able to walk faster — at about the same rate as people without amputations — and navigate around obstacles more easily. They also showed more natural movements, such as pointing the toes of the prosthesis upward while going up stairs or stepping over an obstacle, and they were better able to coordinate the movements of their prosthetic limb and their intact limb. They were also able to push off the ground with the same amount of force as someone without an amputation.

“With the AMI cohort, we saw natural biomimetic behaviors emerge,” Herr says. “The cohort that didn’t have the AMI, they were able to walk, but the prosthetic movements weren’t natural, and their movements were generally slower.”

These natural behaviors emerged even though the amount of sensory feedback provided by the AMI was less than 20 percent of what would normally be received in people without an amputation.

“One of the main findings here is that a small increase in neural feedback from your amputated limb can restore significant bionic neural controllability, to a point where you allow people to directly neurally control the speed of walking, adapt to different terrain, and avoid obstacles,” Song says.

“This work represents yet another step in us demonstrating what is possible in terms of restoring function in patients who suffer from severe limb injury. It is through collaborative efforts such as this that we are able to make transformational progress in patient care,” says Matthew Carty, a surgeon at Brigham and Women’s Hospital and associate professor at Harvard Medical School, who is also an author of the paper.

Enabling neural control by the person using the limb is a step toward Herr’s lab’s goal of “rebuilding human bodies,” rather than having people rely on ever more sophisticated robotic controllers and sensors — tools that are powerful but do not feel like part of the user’s body.

“The problem with that long-term approach is that the user would never feel embodied with their prosthesis. They would never view the prosthesis as part of their body, part of self,” Herr says. “The approach we’re taking is trying to comprehensively connect the brain of the human to the electromechanics.”

The research was funded by the MIT K. Lisa Yang Center for Bionics and the Eunice Kennedy Shriver National Institute of Child Health and Human Development.

Previous item Next item

Share this news article on:

Press mentions, the guardian.

MIT scientists have conducted a trial of a brain controlled bionic limb that improves gait, stability and speed over a traditional prosthetic, reports Hannah Devlin for The Guardian . Prof. Hugh Herr says with natural leg connections preserved, patients are more likely to feel the prosthetic as a natural part of their body. “When the person can directly control and feel the movement of the prosthesis it becomes truly part of the person’s anatomy,” Herr explains. 

The Economist

Using a new surgical technique, MIT researchers have developed a bionic leg that can be controlled by the body’s own nervous system, reports The Economist . The surgical technique “involved stitching together the ends of two sets of leg muscles in the remaining part of the participants’ legs,” explains The Economist . “Each of these new connections forms a so-called agonist-antagonist myoneural interface, or AMI. This in effect replicates the mechanisms necessary for movement as well as the perception of the limb’s position in space. Traditional amputations, in contrast, create no such pairings.”  

The Boston Globe

Researchers at MIT and Brigham and Women’s Hospital have created a new surgical technique and neuroprosthetic interface for amputees that allows a natural walking gait driven by the body’s own nervous system, reports Adam Piore for The Boston Globe . “We found a marked improvement in each patient’s ability to walk at normal levels of speed, to maneuver obstacles, as well as to walk up and down steps and slopes," explains Prof. Hugh Herr. “I feel like I have my leg — like my leg hasn’t been amputated,” shares Amy Pietrafitta, a participant in the clinical trial testing the new approach.

Researchers at MIT have developed a novel surgical technique that could “dramatically improve walking for people with below-the-knee amputations and help them better control their prosthetics,” reports Timmy Broderick for STAT . “With our patients, even though their limb is made of titanium and silicone, all these various electromechanical components, the limb feels natural, and it moves naturally, without even conscious thought," explains Prof. Hugh Herr. 

Financial Times

A new surgical approach developed by MIT researchers enables a bionic leg driven by the body’s nervous system to restore a natural walking gait more effectively than other prosthetic limbs, reports Clive Cookson for the Financial Times . “The approach we’re taking is trying to comprehensively connect the brain of the human to the electro-mechanics,” explains Prof. Hugh Herr.  

The Washington Post

A new surgical procedure and neuroprosthetic interface developed by MIT researchers allows people with amputations to control their prosthetic limbs with their brains, “a significant scientific advance that allows for a smoother gait and enhanced ability to navigate obstacles,” reports Lizette Ortega for The Washington Post . “We’re starting to get a glimpse of this glorious future wherein a person can lose a major part of their body, and there’s technology available to reconstruct that aspect of their body to full functionality,” explains Prof. Hugh Herr. 

Related Links

• Biomechatronics Group
• McGovern Institute
• K. Lisa Yang Center for Bionics

Related Topics

• Assistive technology
• Prosthetics
• Neuroscience
• School of Architecture and Planning

Related Articles

Magnetic sensors track muscle length

New surgery may enable better control of prosthetic limbs

Making prosthetic limbs feel more natural

More mit news.

Studying astrophysically relevant plasma physics

Read full story →

What is language for?

Signal processing: How did we get to where we’re going?

How to increase the rate of plastics recycling

Summer 2024 reading from MIT

The rules of the game

• More news on MIT News homepage →

Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA

• Map (opens in new window)
• Events (opens in new window)
• People (opens in new window)
• Careers (opens in new window)
• Accessibility
• Social Media Hub
• MIT on Facebook
• MIT on YouTube
• MIT on Instagram

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.