disease detection using machine learning research paper

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Plants (Basel)

Plant Disease Detection and Classification by Deep Learning

Muhammad hammad saleem.

1 Department of Mechanical and Electrical Engineering, School of Food and Advanced Technology, Massey University, Auckland 0632, New Zealand; [email protected]

Johan Potgieter

2 Massey Agritech Partnership Research Centre, School of Food and Advanced Technology, Massey University, Palmerston North 4442, New Zealand; [email protected]

Khalid Mahmood Arif

Plant diseases affect the growth of their respective species, therefore their early identification is very important. Many Machine Learning (ML) models have been employed for the detection and classification of plant diseases but, after the advancements in a subset of ML, that is, Deep Learning (DL), this area of research appears to have great potential in terms of increased accuracy. Many developed/modified DL architectures are implemented along with several visualization techniques to detect and classify the symptoms of plant diseases. Moreover, several performance metrics are used for the evaluation of these architectures/techniques. This review provides a comprehensive explanation of DL models used to visualize various plant diseases. In addition, some research gaps are identified from which to obtain greater transparency for detecting diseases in plants, even before their symptoms appear clearly.

1. Introduction

The Deep Learning (DL) approach is a subcategory of Machine Learning (ML), introduced in 1943 [ 1 ] when threshold logic was introduced to build a computer model closely resembling the biological pathways of humans. This field of research is still evolving; its evolution can be divided into two time periods-from 1943–2006 and from 2012–until now. During the first phase, several developments like backpropagation [ 2 , 3 ], chain rule [ 4 ], Neocognitron [ 5 ], hand written text recognition (LeNET architecture) [ 6 ], and resolving the training problem [ 7 , 8 ] were observed (as shown in Figure 1 ). However, in the second phase, state-of-the-art algorithms/architectures were developed for many applications including self-driving cars [ 9 , 10 , 11 ], healthcare sector [ 12 , 13 , 14 ], text recognition [ 6 , 15 , 16 , 17 ], earthquake predictions [ 18 , 19 , 20 ], marketing [ 21 ], finance [ 22 , 23 ], and image recognition [ 24 , 25 , 26 , 27 , 28 , 29 ]. Among those architectures, AlexNet [ 30 ] is considered to be a breakthrough in the field of DL as it won the ImageNet challenge for object recognition known as ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in the year 2012. Soon after, several architectures were introduced to overcome the loopholes observed previously. For the evaluation of these algorithms/architectures, various performance metrics were used. Among these metrics, top-1%/top-5% error [ 24 , 26 , 30 , 31 ], precision and recall [ 25 , 32 , 33 , 34 ], F1 score [ 32 , 35 ], training/validation accuracy and loss [ 34 , 36 ], classification accuracy (CA) [ 37 , 38 , 39 , 40 , 41 ] are the most popular. For the implementation of DL models, several steps are required, from the collection of datasets to visualization mappings are explained in Figure 2 .

Summary of the evolution of deep learning from 1943–2006.

Flow diagram of DL implementation: First, the dataset is collected [ 25 ] then split into two parts, normally into 80% of training and 20% of validation set. After that, DL models are trained from scratch or by using transfer learning technique, and their training/validation plots are obtained to indicate the significance of the models. Then, performance metrics are used for the classification of images (type of particular plant disease), and finally, visualization techniques/mappings [ 55 ] are used to detect/localize/classify the images.

When DL architectures started to evolve with the passage of time, researchers applied them to image recognition and classification. These architectures have also been implemented for different agricultural applications. For example, in [ 42 ], classification of leaves was performed by using author-modified CNN and Random Forest (RF) classifier among 32 species in which the performance was evaluated through CA at 97.3%. On the other hand, it was not as efficient at detecting occluded objects [ 43 ]. Leaf and fruit counting were also performed by deep CNN in [ 44 , 45 ] and [ 46 ] respectively. For classification of crop type, [ 47 ] used author-modified CNN, [ 36 ] applied VGG 16, [ 34 ] implemented three unit LSTM, and [ 33 ] used CNN and RGB histogram technique. [ 47 ] used CA, [ 36 ] used CA and Intersection over Union (IoU), [ 34 ] used CA and F1, and [ 33 ] used F1-score as a performance metric. Among them, [ 33 , 47 ] did not provide training/validation accuracy and loss. Moreover, recognition of different plants has been done by the DL approach in [ 48 , 49 , 50 ]. [ 48 , 50 ] employed user-modified CNN while [ 49 ] used AlexNet architecture. All were evaluated on the basis of CA. [ 49 ] outperformed the other two in terms of CA. Similarly, crop/weed discrimination was performed in [ 51 , 52 ], in which the author proposed CNN be used, and two datasets were utilized for the evaluation of the model. [ 51 ] evaluated precision and recall; however, [ 52 ] obtained CA for the validation of the proposed models respectively. The identification of plants by the DL approach was studied and achieved a success rate of 91.78% [ 53 ]. On top of that, DL approaches are also used for critical tasks like plant disease detection and classification, which is the main focus of this review. There are some research papers previously presented to summarize the research based on agriculture (including plant disease recognition) by DL [ 43 , 54 ], but they lacked some of the recent developments in terms of visualization techniques implemented along with the DL and modified/cascaded version of famous DL models, which were used for plant disease identification. Moreover, this review also provides the research gaps in order to get a clearer/more transparent vision of symptoms observed due to diseases in the plants.

The remaining part of the paper is comprised of Section 2 , describing the famous and new/modified DL architectures along with visualization mapping/techniques used for plant disease detection; Section 3 , elaborating upon the Hyperspectral Imaging with DL models; and finally, Section 4 , concluding the review and providing future recommendations for achieving more advancements in the visualization, detection, and classification of plants’ diseases.

2. Plant Disease Detection by Well-Known DL Architectures

Many state-of-the-art DL models/architectures evolved after the introduction of AlexNet [ 30 ] (as shown in Figure 3 and Table 1 ) for image detection, segmentation, and classification. This section presents the researches done by using famous DL architectures for the identification and classification of plants’ diseases. Moreover, there are some related works in which new visualization techniques and modified/improved versions of DL architectures were introduced to achieve better results. Among all of them, the PlantVillage dataset has been used widely as it contains 54,306 images of 14 different crops having 26 plant diseases [ 25 ]. Moreover, they used several performance metrics to evaluate the selected DL models, which are described as below.

Summary of the evolution of various deep learning models from 2012 until now.

Comparison of state-of-the-art deep learning models.

2.1. Implementation of DL Models

2.1.1. without visualization technique.

In [ 56 ], CNN was used for the classification of diseases in maize plants and histogram techniques to show the significance of the model. In [ 57 ], basic CNN architectures like AlexNet, GoogLeNet and ResNet were implemented for identifying the tomato leaf diseases. Training/validation accuracy were plotted to show the performance of the model; ResNet was considered as the best among all the CNN architectures. In order to detect the diseases in banana leaf, LeNet architecture was implemented and CA, F1-score were used for the evaluation of the model in Color and Gray Scale modes [ 32 ]. Five CNN architectures were used in [ 58 ], namely, AlexNet, AlexNetOWTbn, GoogLeNet, Overfeat, and VGG architectures in which VGG outclassed all the other models. In [ 35 ], eight different plant diseases were recognized by three classifiers, Support Vector Machines (SVM), Extreme Learning Machine (ELM), and K-Nearest Neighbor (KNN)), used with the state-of-the-art DL models like GoogLeNet, ResNet-50, ResNet-101, Inception-v3, InceptionResNetv2, and SqueezeNet. A comparison was made between those models, and ResNet-50 with SVM classifier got the best results in terms of performance metrics like sensitivity, specificity, and F1-score. According to [ 59 ], a new DL model—Inception-v3—was used for the detection of cassava disease. In [ 60 ], plant diseases in cucumber were classified by the two basic versions of CNN and got the highest accuracy, equal to 0.823. The traditional plant disease recognition and classification method was replaced by Super-Resolution Convolutional Neural Network (SRCNN) in [ 61 ]. For the classification of tomato plant disease, AlexNet and SqueezeNet v1.1 models were used in which AlexNet was found to be the better DL model in terms of accuracy [ 62 ]. A comparative analysis was presented in [ 63 ] to select the best DL architecture for detection of plant diseases. Moreover in [ 64 ], six tomato plant diseases were classified by using AlexNet and VGG-16 DL architectures, and a detailed comparison was provided with the help of classification accuracy. In the above approaches, no visualization technique was applied to spot the symptoms of diseases in the plants.

2.1.2. With Visualization Techniques

The following approaches employed DL models/architectures and also visualization techniques which were introduced for a clearer understanding of plants’ diseases. For example, [ 55 ] introduced the saliency map for visualizing the symptoms of plant disease; [ 27 ] identified 13 different types of plant disease with the help of CaffeNet CNN architecture, and achieved CA equal to 96.30%, which was better than the previous approach like SVM. Moreover, several filters were used to indicate the disease spots. Similarly, [ 25 ] used AlexNet and GoogLeNet CNN architectures by using the publicly available PlantVillage dataset. The performance was evaluated by means of precision (P), recall (R), F1 score, and overall accuracy. The uniqueness of this paper was the implication of three scenarios (color, grayscale, and segmented) for evaluating the performance metrics and comparison of the two famous CNN architectures. It was concluded that GoogLeNet outperformed AlexNet. Moreover, visualization activation in the first layers clearly showed the spots of diseases. In [ 65 ], a modified LeNet model was used to detect olive plant diseases. The segmentation and edges maps were used to spot the diseases in the plants. Detection of four cucumber diseases was done in [ 66 ] and accuracy was compared with Random Forest, Support Vector Machines, and AlexNet models. Moreover, the image segmentation method was used to view the symptoms of diseases in the plants. A new DL model was introduced in [ 67 ] named teacher/student network and proposed a novel visualization method to identify the spots of plant diseases. DL models with some detectors were implemented in [ 68 ], in which the diseases in plants were marked along with their prediction percentage. Three detectors, named Faster-RCNN, RFCN and SSD, were used with the famous architectures like AlexNet, GoogLeNet, VGG, ZFNet, ResNet-50, ResNet-101 and ResNetXt-101 for a comparative study which outlined the best among all the selected architectures. It was concluded that ResNet-50 with the detector R-FCN gave the best results. Furthermore, a kind of bounding box was drawn to identify the particular type of disease in the plants. In [ 69 ], a banana leaf disease and pest detection was performed by using three CNN models (ResNet-50, Inception-V2 and MobileNet-V1) with Faster-RCNN and SSD detectors. According to [ 70 ], different combinations of CNN were used and presented heat maps as input to the diseased plants’ images and provided the probability related to the occurrence of a particular type of disease. Moreover, ROC curve evaluates the performance of the model. Furthermore, feature maps for rice disease were also included in the paper. LeNet model was used in [ 71 ] to detect and classify diseases in the soybean plant. In [ 72 ], a comparison between AlexNet and GoogLeNet architectures for tomato plant diseases was done, in which GoogLeNet performed better than the AlexNet; also, it proposed occlusion techniques to recognize the regions of diseases. The VGG-FCN and VGG-CNN models were implemented in [ 73 ], for the detection of wheat plant diseases and visualization of features in each block. In [ 74 ], VGG-CNN model was used for the detection of Fusarium wilt in radish and K-means clustering method was used to show the marks of diseases. A semantic segmentation approach by CNN was proposed in [ 75 ] to detect the disease in cucumber. In [ 76 ], an approach based on the individual symptoms/spots of diseases in the plants was introduced by using a DL model for detecting plant diseases. A Deep CNN framework was developed for identification, classification, and quantification of eight soybean stresses in [ 77 ]. In [ 78 ], rice plant diseases were identified by CNN, and feature maps were obtained to identify the patches of diseases. A deep residual neural network was extended in [ 79 ] for the development of a mobile application in which a clear identification of diseases in plants was done by the hot spot. An algorithm based on the hot spot technique was also used in [ 80 ], in which those spots were extracted by modification in the segmented image to attain color constancy. Furthermore, each obtained hot-spot was described by two descriptors, one was used to evaluate the color information of the disease and other was used to identify the texture of the hot-spots. The cucumber plant diseases were identified in [ 81 ] by using the dilation convolutional neural network. A state-of-the-art visualization technique was proposed in [ 82 ] by correlation coefficient and DL models like AlexNet and VGG-16 architectures. In [ 83 ], color space and various vegetation indices combined with CNN model (LeNet) to detect the diseases in grapes. To summarize, Table 2 outlines some of the visualization mapping/techniques.

Visualization mapping/techniques used in several approaches.

For the practical experimentation of detection of plants’ diseases, an actual/real background/environment should be considered in order to evaluate the performance of the DL model more accurately. In most of the above approaches, the selected datasets considered plain backgrounds which are not realistic scenarios for identification and classification of the diseases [ 25 , 27 , 32 , 56 , 57 , 58 , 60 , 61 , 65 , 72 , 77 , 78 ], except for a few of them that have considered the original backgrounds [ 35 , 59 , 68 , 70 , 73 , 74 ]. The output of the visualization techniques used in several researches are shown in Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 10 and Figure 11 .

Feature maps after the application of convolution to an image: ( a ) real image, ( b ) first convolutional layer filter, ( c ) rectified output from first layer, ( d ) second convolutional layer filter, ( e ) output from second layer, ( f ) output of third layer, ( g ) output of fourth layer, ( h ) output of fifth layer [ 27 ].

Tomato plant disease detection by heat map: on left hand side ( a ) tomato early blight, ( b ) tomato septoria leaf spot, ( c ) tomato late blight and ( d ) tomato leaf mold) and saliency map; on right hand side ( a ) tomato healthy, ( b ) tomato late blight, ( c ) tomato early blight, ( d ) tomato septoria leaf spot, ( e ) tomato early blight, ( f ) tomato leaf mold) [ 55 ].

Detection of maize disease (indicated by red circles) by heat map [ 70 ].

Bounding box indicates the type of diseases along with the probability of their occurrence [ 68 ]. A bounding box technique was used in Figure 7 in which ( a ) represents the one type of disease along with its rate of occurrence, ( b ) indicates three types of plant disease (miner, temperature, and gray mold) in a single image, ( c , d ) shows one class of disease but contains different patterns on the front and back side of the image, ( e , f ) displays different patterns of gray mold in the starting and end stages [ 68 ].

( a ) Teacher/student architecture approach; ( b ) segmentation using a binary threshold algorithm [ 67 ].

Comparison of Teacher/student approach visualization map with the previous approaches [ 67 ].

Activation visualization for detection of apple plant disease to show the significance of a VGG-Inception model (the plant disease is indicated by the red circle) [ 85 ].

Segmentation and edge map for olive leaf disease detection [ 65 ].

In Figure 4 , feature maps from the first to the fifth hidden layer are shown as the neuron in a feature map having identical features at different positions of an image. Starting from the first layer (a), the features in feature maps represent separate pixels to normal lines, whereas the fifth layer shows some particular parts of the image (h).

Two types of visualization maps are shown in Figure 5 , namely, heat map and saliency map techniques. The heat maps identify the diseases shown as red boxes in the input image, but it should be noted that one disease marked in (d) has not been detected. This problem was resolved in the saliency map technique after the application of the guided back-propagation [ 55 ]; all the spots of plant disease were successfully identified thanks to a method which is superior to the heat map.

Figure 6 represents the heat map to detect the disease in maize plants. First, the image was represented in the form of the probability of each portion containing disease. Then, the probabilities were placed into the form of a matrix in order to denote the outcome of all the areas of the input image.

A new visualization technique was proposed in [ 67 ] as shown in Figure 8 and Figure 9 . In Figure 8 a, the input image was regenerated for student/teacher architecture [ 67 ], and a single channel heat map was produced after the application of simple aggregation on the channels of the regenerated image ( Figure 8 b). Then, a simple binary threshold algorithm was applied to obtain sharp symptoms of diseases in the plant. Then, [ 67 ] indicated the significance of the proposed technique by comparing it with the other visualization techniques as shown in Figure 9 . On the left hand side, LRP-Z, LRP-Epsilon, and gradient did not identify plant diseases clearly. However, the Deep Taylor approach produced better results but indicated some portion of the leaf disease. On the right hand side, an imperfect localization of the plant disease was shown in grad-cam techniques which was resolved in the proposed technique by the use of a decoder [ 67 ].

In order to find the significance of CNN architectures to differentiate between various diseases of plants, the feature maps were obtained as shown in Figure 10 . The result proves a good performance of the proposed CNN model as it clearly identifies the disease in plants [ 85 ].

In Figure 11 the segmentation and edged maps were obtained to identify the diseases in plants. It is noted that the yellow colored area is marked as white surface in the segmentation map to show the affected part of the leaf.

2.2. New/Modified DL Architectures for Plant-Disease Detection

According to some of the research papers, new/modified DL architectures have been introduced to obtain better/transparent detection of plant disease, such as [ 86 ] presented improved GoogLeNet and Cifar-10 models and their performance compared with AlexNet and VGG. It was found that improved versions of these state-of-the-art models produced a remarkable accuracy of 98.9%. In [ 87 ], a new DL model was introduced to obtain more accurate detection of plant diseases as compared to SVM, AlexNet, GoogLeNet, ResNet-20, and VGG-16 models. This model achieved 97.62% accuracy for classifying apple plant diseases. Moreover, the dataset extended in 13 different ways (rotation of 90°, 180°, 270° and mirror symmetry (horizontal symmetry), change in contrast, sharpness and brightness). Moreover, the whole dataset was transformed into Gaussian noise and PCA jittering as well. Furthermore, the selection of dataset was explained by the help of plots to prove the significance of extending the dataset. A new CNN model named LeafNet was introduced in [ 88 ] to classify the tea leaf diseases and achieved higher accuracy than Support Vector Machine (SVM) and Multi-Layer Perceptron (MLP). In [ 89 ], two DL models named modified MobileNet and reduced MobileNet were introduced, and their accuracy was near to the VGG model; the reduced MobileNet actually got 98.34% classification accuracy and had a fewer number of parameters as compared to VGG which saves time in training the model. A state-of-the-art DL model was proposed in [ 90 ] named PlantdiseaseNet which was remarkably suitable for the complex environment of an agricultural field. In [ 85 ], five types of apple plant diseases were classified and detected by the state-of-the-art CNN model named VGG-inception architecture. It outclassed the performance of many DL architectures like AlexNet, GoogLeNet, several versions of ResNet, and VGG. It also presented inter object/class detection and activation visualization; it was also mentioned for its clear vision of diseases in the plants.

A bar chart presented in Figure 12 indicates, from the most to the least frequently used, DL models for plant disease detection and classification. It can be clearly seen that the AlexNet model has been used in most of the researches. GoogLeNet, VGG-16, and ResNet-50 are the next most commonly used DL models. Similarly, there are some improved/cascaded versions (Improved Cifar-10, VGG-Inception, Cascaded AlexNet with GoogLeNet, reduced/modified MobileNet, modified LeNet, and modified GoogLeNet), which have been used for plant disease identification.

Deep learning models used in the particular number of research papers.

Summing up Section 2 , all the DL approaches along with the selected plant species and performance metrics are shown in Table 3 .

Comparison of several DL approaches in terms of various performance metrics.

3. Hyper-Spectral Imaging with DL Models

For early detection of plant diseases, several imaging techniques like multispectral imaging [ 91 ], thermal imaging, fluorescence and hyperspectral imaging are used [ 92 ]. Among them, hyperspectral imaging (HSI) is the focus of recent research. For example, [ 93 ] used hyperspectral imaging (HSI) to detect tomato plant diseases by identifying the region of interest, and a feature ranking-KNN (FR-KNN) model produced a satisfactory result for the detection of diseased and healthy plants. In the recent approach, HSI was used for the detection of an apple disease. Moreover, the redundancy issue was resolved by an unsupervised feature selection procedure known as Orthogonal Subspace Projection [ 94 ]. In [ 95 ], leaf diseases on peanuts were detected by HSI by identifying sensitive bands and hyperspectral vegetation index. The tomato disease detection was done by SVM classifiers based on HSI, and their performance was evaluated by F1-score, accuracy, specificity, and sensitivity [ 96 ].

Recently, HSI has been used with machine learning (ML) for the detection of plant diseases. For example, [ 97 ] described ML techniques for hyperspectral imaging for many agricultural applications. Moreover, ML with HSI have been used for three ML models, implemented by using hyperspectral measurement technique for the detection of leaf rust disease [ 98 ]. For wheat disease detection, [ 99 ] used Random Forest (RF) classifier with multispectral imaging technique and achieved accuracy of 89.3%. Plants’ diseases were also detected by SVM based on hyperspectral data and achieved accuracy of more than 86% [ 100 ]. There are some other ML approaches based on HSI [ 101 ], but this review is focused on DL approaches based on HSI, presented below.

The DL has been used to classify the hyperspectral images for many applications. For medical purposes, this technology is very useful as it is used for the classification of head/neck cancer in [ 102 ]. In [ 103 ], a DL approach based on HSI was proposed through contextual information as it provides spectral and spatial features. A new 3D-CNN architecture allowed for a fast, accurate, and efficient approach to classify the hyperspectral images in [ 104 ]. This architecture not only used the spectral information (as used in previous CNN techniques [ 105 ]) but also ensured that the spatial information was also taken into account. In [ 106 ], the feature extraction procedure was used with CNN for hyperspectral image classification and used dropout and L2 regularization methods in order to prevent overfitting. Just like CNN models used for hyperspectral imaging classification, RNN models are also used with HSI as described in [ 107 , 108 ]. In the domain of plant disease detection, some researches utilized Hyperspectral Imaging (HSI) along with DL models to observe clearer vision for symptoms of plant diseases. A hybrid method to classify the hyperspectral images was proposed in [ 109 ] consisting of DCNN, LR, and PCA and got better results compared to the previous methods for classification tasks. In [ 110 ], a detailed review of DL with HSI technique was provided. In order to avoid the overfitting and improve accuracy, a detailed comparison provided between several DL models like 1D/2D-CNN (2D-CNN better result), LSTM/GRU (both faced overfitting), 2D-CNN-LSTM/GRU (still overfitting) was observed. Therefore, a new hybrid approach from Convolutional and Bidirectional Gated Recurrent Network named 2D-CNN-BidLSTM/GRU was proposed for the hyperspectral images, which resolved the problem of overfitting and achieved 0.75 F1-score and 0.73 accuracy for wheat diseases detection [ 111 ]. According to [ 112 ], a hyperspectral proximal-sensing procedure based on the newest DL technique named Generative Adversarial Nets (GAN) was proposed in order to detect tomato plant disease before its clear symptoms appeared (as shown in Figure 13 ). In [ 84 ], a 3D-CNN approach was proposed for hyperspectral images to identify the Charcoal rot disease in soybeans and the CNN model was evaluated by accuracy (95.76%) and F1-score (0.87). The saliency map visualization was used, and the most delicate wavelength resulted as 733 nm, which approximately lies in the region of the wavelength of NIR. For the detection of potato virus, [ 113 ] described it by DL on the hyperspectral images and achieved acceptable values of precision (0.78) and recall (0.88). In [ 114 ], a DL model named multiple Inception-Resnet model was developed by using both spatial and spectral data on hyperspectral UAV images to detect the yellow rust in wheat (as shown in Figure 14 ). This model achieved an 85% accuracy, which is quite a lot higher than the RF-classifier (77%).

Sample images of OR-AC-GAN (a hyperspectral imaging model) [ 112 ].

Hyperspectral images by UAV: ( a ) RGB color plots, ( b ) Random-Forest classifier, and ( c ) proposed multiple Inception-ResNet model [ 114 ].

From this section, we can conclude that, although there are some DL models/architectures developed for hyperspectral image classification in the application of plant disease detection, this is still a fertile area of research and should lead to improvements for better detection of plants’ diseases [ 115 ] in different situations, like various conditions of illumination, considering real background, etc.

In Figure 13 , the resultant images are taken from the proposed method described in [ 112 ]. The green-colored portion indicates the healthy part of the plant; the red portion denotes the infected portion. Note that ( a ) and ( b ) are the healthy plant images as there is no red color indication, whereas ( c ) has infected disease which can be seen in its corresponding figure ( d ).

A comparison of proposed DCNN with RF classifier and RGB colored hyperspectral images are shown in Figure 14 . The red color label indicates the portion infected by rust. It should be observed that the rust plots were identified in an almost similar manner (see (b) and (c) of first row), but in the healthy plot, there was a large portion covered by the red label in (b) as compared to (c), which shows a wrong classification by RF model [ 114 ].

4. Conclusions and Future Directions

This review explained DL approaches for the detection of plant diseases. Moreover, many visualization techniques/mappings were summarized to recognize the symptoms of diseases. Although much significant progress was observed during the last three to four years, there are still some research gaps which are described below:

• In most of the researches (as described in the previous sections), the PlantVillage dataset was used to evaluate the accuracy and performance of the respective DL models/architectures. Although this dataset has a lot of images of several plant species with their diseases, it has a simple/plain background. However, for a practical scenario, the real environment should be considered.
• Hyperspectral/multispectral imaging is an emerging technology and has been used in many areas of research (as described in Section 3 ). Therefore, it should be used with the efficient DL architectures to detect the plants’ diseases even before their symptoms are clearly apparent.
• A more efficient way of visualizing the spots of disease in plants should be introduced as it will save costs by avoiding the unnecessary application of fungicide/pesticide/herbicide.
• The severity of plant diseases changes with the passage of time, therefore, DL models should be improved/modified to enable them to detect and classify diseases during their complete cycle of occurrence.
• DL model/architecture should be efficient for many illumination conditions, so the datasets should not only indicate the real environment but also contain images taken in different field scenarios.
• A comprehensive study is required to understand the factors affecting the detection of plant diseases, like the classes and size of datasets, learning rate, illumination, and the like.

Abbreviations

The abbreviations used in this manuscript are given as under:

Author Contributions

Conceptualization, M.H.S. and K.M.A.; methodology, M.H.S. and K.M.A.; writing—original draft preparation, M.H.S. and K.M.A.; writing—review and editing, M.H.S., J.P., and K.M.A; visualization, M.H.S., J.P., and K.M.A; supervision, J.P., and K.M.A.; project administration, J.P., and K.M.A.

This research was funded by the Ministry of Business, Innovation and Employment (MBIE), New Zealand, Science for Technological Innovation (SfTI) National Science Challenge.

Conflicts of Interest

The authors declare no conflict of interest.

• Open access
• Published: 24 February 2021

Plant diseases and pests detection based on deep learning: a review

• Jun Liu   ORCID: orcid.org/0000-0001-8769-5981 1 &
• Xuewei Wang 1  

Plant Methods volume  17 , Article number:  22 ( 2021 ) Cite this article

135k Accesses

411 Citations

17 Altmetric

Metrics details

Plant diseases and pests are important factors determining the yield and quality of plants. Plant diseases and pests identification can be carried out by means of digital image processing. In recent years, deep learning has made breakthroughs in the field of digital image processing, far superior to traditional methods. How to use deep learning technology to study plant diseases and pests identification has become a research issue of great concern to researchers. This review provides a definition of plant diseases and pests detection problem, puts forward a comparison with traditional plant diseases and pests detection methods. According to the difference of network structure, this study outlines the research on plant diseases and pests detection based on deep learning in recent years from three aspects of classification network, detection network and segmentation network, and the advantages and disadvantages of each method are summarized. Common datasets are introduced, and the performance of existing studies is compared. On this basis, this study discusses possible challenges in practical applications of plant diseases and pests detection based on deep learning. In addition, possible solutions and research ideas are proposed for the challenges, and several suggestions are given. Finally, this study gives the analysis and prospect of the future trend of plant diseases and pests detection based on deep learning.

Plant diseases and pests detection is a very important research content in the field of machine vision. It is a technology that uses machine vision equipment to acquire images to judge whether there are diseases and pests in the collected plant images [ 1 ]. At present, machine vision-based plant diseases and pests detection equipment has been initially applied in agriculture and has replaced the traditional naked eye identification to some extent.

For traditional machine vision-based plant diseases and pests detection method, conventional image processing algorithms or manual design of features plus classifiers are often used [ 2 ]. This kind of method usually makes use of the different properties of plant diseases and pests to design the imaging scheme and chooses appropriate light source and shooting angle, which is helpful to obtain images with uniform illumination. Although carefully constructed imaging schemes can greatly reduce the difficulty of classical algorithm design, but also increase the application cost. At the same time, under natural environment, it is often unrealistic to expect the classical algorithms designed to completely eliminate the impact of scene changes on the recognition results [ 3 ]. In real complex natural environment, plant diseases and pests detection is faced with many challenges, such as small difference between the lesion area and the background, low contrast, large variations in the scale of the lesion area and various types, and a lot of noise in the lesion image. Also, there are a lot of disturbances when collecting plant diseases and pests images under natural light conditions. At this time, the traditional classical methods often appear helpless, and it is difficult to achieve better detection results.

In recent years, with the successful application of deep learning model represented by convolutional neural network (CNN) in many fields of computer vision (CV, computer-vision), for example, traffic detection [ 4 ], medical Image Recognition [ 5 ], Scenario text detection [ 6 ], expression recognition [ 7 ], face Recognition [ 8 ], etc. Several plant diseases and pests detection methods based on deep learning are applied in real agricultural practice, and some domestic and foreign companies have developed a variety of deep learning-based plant diseases and pests detection Wechat applet and photo recognition APP software. Therefore, plant diseases and pests detection method based on deep learning not only has important academic research value, but also has a very broad market application prospect.

In view of the lack of comprehensive and detailed discussion on plant diseases and pests detection methods based on deep learning, this study summarizes and combs the relevant literatures from 2014 to 2020, aiming to help researchers quickly and systematically understand the relevant methods and technologies in this field. The content of this study is arranged as follows: “ Definition of plant diseases and pests detection problem ” section gives the definition of plant diseases and pests detection problem; “ Image recognition technology based on deep learning ” section focuses on the detailed introduction of image recognition technology based on deep learning; “ Plant diseases and pests detection methods based on deep learning ” section analyses the three kinds of plant diseases and pests detection methods based on deep learning according to network structure, including classification, detection and segmentation network; “ Dataset and performance comparison ” section introduces some datasets of plant diseases and pests detection and compares the performance of the existing studies; “ Challenges ” section puts forward the challenges of plant diseases and pests detection based on deep learning; “ Conclusions and future directions ” section prospects the possible research focus and development direction in the future.

Definition of plant diseases and pests detection problem

Definition of plant diseases and pests

Plant diseases and pests is one kind of natural disasters that affect the normal growth of plants and even cause plant death during the whole growth process of plants from seed development to seedling and to seedling growth. In machine vision tasks, plant diseases and pests tend to be the concepts of human experience rather than a purely mathematical definition.

Definition of plant diseases and pests detection

Compared with the definite classification, detection and segmentation tasks in computer vision [ 9 ], the requirements of plant diseases and pests detection is very general. In fact, its requirements can be divided into three different levels: what, where and how [ 10 ]. In the first stage, “what” corresponds to the classification task in computer vision. As shown in Fig.  1 , the label of the category to which it belongs is given. The task in this stage can be called classification and only gives the category information of the image. In the second stage, “where” corresponds to the location task in computer vision, and the positioning of this stage is the rigorous sense of detection. This stage not only acquires what types of diseases and pests exist in the image, but also gives their specific locations. As shown in Fig.  1 , the plaque area of gray mold is marked with a rectangular box. In the third stage, “how” corresponds to the segmentation task in computer vision. As shown in Fig.  1 , the lesions of gray mold are separated from the background pixel by pixel, and a series of information such as the length, area, location of the lesions of gray mold can be further obtained, which can assist the higher-level severity level evaluation of plant diseases and pests. Classification describes the image globally through feature expression, and then determines whether there is a certain kind of object in the image by means of classification operation; while object detection focuses on local description, that is, answering what object exists in what position in an image, so in addition to feature expression, object structure is the most obvious feature that object detection differs from object classification. That is, feature expression is the main research line of object classification, while structure learning is the research focus of object detection. Although the function requirements and objectives of the three stages of plant diseases and pests detection are different, yet in fact, the three stages are mutually inclusive and can be converted. For example, the “where” in the second stage contains the process of “what” in the first stage, and the “how” in the third stage can finish the task of “where” in the second stage. Also, the “what” in the first stage can achieve the goal of the second and the third stages through some methods. Therefore, the problem in this study is collectively referred to as plant diseases and pests detection as conventions in the following text, and the terminology differentiates only when different network structures and functions are adopted.

Comparison with traditional plant diseases and pests detection methods

To better illustrate the characteristics of plant diseases and pests detection methods based on deep learning, according to existing references [ 11 , 12 , 13 , 14 , 15 ], a comparison with traditional plant diseases and pests detection methods is given from four aspects including essence, method, required conditions and applicable scenarios. Detailed comparison results are shown in Table 1 .

Image recognition technology based on deep learning

Compared with other image recognition methods, the image recognition technology based on deep learning does not need to extract specific features, and only through iterative learning can find appropriate features, which can acquire global and contextual features of images, and has strong robustness and higher recognition accuracy.

Deep learning theory

The concept of Deep Learning (DL) originated from a paper published in Science by Hinton et al. [ 16 ] in 2006. The basic idea of deep learning is: using neural network for data analysis and feature learning, data features are extracted by multiple hidden layers, each hidden layer can be regarded as a perceptron, the perceptron is used to extract low-level features, and then combine low-level features to obtain abstract high-level features, which can significantly alleviate the problem of local minimum. Deep learning overcomes the disadvantage that traditional algorithms rely on artificially designed features and has attracted more and more researchers’ attention. It has now been successfully applied in computer vision, pattern recognition, speech recognition, natural language processing and recommendation systems [ 17 ].

Traditional image classification and recognition methods of manual design features can only extract the underlying features, and it is difficult to extract the deep and complex image feature information [ 18 ]. And deep learning method can solve this bottleneck. It can directly conduct unsupervised learning from the original image to obtain multi-level image feature information such as low-level features, intermediate features and high-level semantic features. Traditional plant diseases and pests detection algorithms mainly adopt the image recognition method of manual designed features, which is difficult and depends on experience and luck, and cannot automatically learn and extract features from the original image. On the contrary, deep learning can automatically learn features from large data without manual manipulation. The model is composed of multiple layers, which has good autonomous learning ability and feature expression ability, and can automatically extract image features for image classification and recognition. Therefore, deep learning can play a great role in the field of plant diseases and pests image recognition. At present, deep learning methods have developed many well-known deep neural network models, including deep belief network (DBN), deep Boltzmann machine (DBM), stack de-noising autoencoder (SDAE) and deep convolutional neural network (CNN) [ 19 ]. In the area of image recognition, the use of these deep neural network models to realize automate feature extraction from high-dimensional feature space offers significant advantages over traditional manual design feature extraction methods. In addition, as the number of training samples grows and the computational power increases, the characterization power of deep neural networks is being further improved. Nowadays, the boom of deep learning is sweeping both industry and academia, and the performance of deep neural network models are all significantly ahead of traditional models. In recent years, the most popular deep learning framework is deep convolutional neural network.

• Convolutional neural network

Convolutional Neural Networks, abbreviated as CNN, has a complex network structure and can perform convolution operations. As shown in Fig.  2 , the convolutional neural network model is composed of input layer, convolution layer, pooling layer, full connection layer and output layer. In one model, the convolution layer and the pooling layer alternate several times, and when the neurons of the convolution layer are connected to the neurons of the pooling layer, no full connection is required. CNN is a popular model in the field of deep learning. The reason lies in the huge model capacity and complex information brought about by the basic structural characteristics of CNN, which enables CNN to play an advantage in image recognition. At the same time, the successes of CNN in computer vision tasks have boosted the growing popularity of deep learning.

The basic structure of CNN

In the convolution layer, a convolution core is defined first. The convolution core can be considered as a local receptive field, and the local receptive field is the greatest advantage of the convolution neural network. When processing data information, the convolution core slides on the feature map to extract part of the feature information. After the feature extraction of the convolution layer, the neurons are input into the pooling layer to extract the feature again. At present, the commonly used methods of pooling include calculating the mean, maximum and random values of all values in the local receptive field [ 20 , 21 ]. After the data entering several convolution layers and pooling layers, they enter the full-connection layer, and the neurons in the full-connection layer are fully connected with the neurons in the upper layer. Finally, the data in the full-connection layer can be classified by the softmax method, and then the values are transmitted to the output layer for output results.

Open source tools for deep learning

The commonly used third-party open source tools for deep learning are Tensorflow [ 22 ], Torch/PyTorch [ 23 ], Caffe [ 24 ], Theano [ 25 ]. The different characteristics of each open source tool are shown in Table 2 .

The four commonly used deep learning third-party open source tools all support cross-platform operation, and the platforms that can be run include Linux, Windows, iOS, Android, etc. Torch/PyTorch and Tensorflow have good scalability and support a large number of third-party libraries and deep network structures, and have the fastest training speed when training large CNN networks on GPU.

Plant diseases and pests detection methods based on deep learning

This section gives a summary overview of plant diseases and pests detection methods based on deep learning. Since the goal achieved is completely consistent with the computer vision task, plant diseases and pests detection methods based on deep learning can be seen as an application of relevant classical networks in the field of agriculture. As shown in Fig.  3 , the network can be further subdivided into classification network, detection network and segmentation network according to the different network structures. As can be seen from Fig.  3 , this paper is subdivided into several different sub-methods according to the processing characteristics of each type of methods.

Framework of plant diseases and pests detection methods based on deep learning

Classification network

In real natural environment, the great differences in shape, size, texture, color, background, layout and imaging illumination of plant diseases and pests make the recognition a difficult task. Due to the strong feature extraction capability of CNN, the adoption of CNN-based classification network has become the most commonly used pattern in plant diseases and pests classification. Generally, the feature extraction part of CNN classification network consists of cascaded convolution layer + pooling layer, followed by full connection layer (or average pooling layer) + softmax structure for classification. Existing plant diseases and pests classification network mostly use the muture network structures in computer vision, including AlexNet [ 26 ], GoogleLeNet [ 27 ], VGGNet [ 28 ], ResNet [ 29 ], Inception V4 [ 30 ], DenseNets [ 31 ], MobileNet [ 32 ] and SqueezeNet [ 33 ]. There are also some studies which have designed network structures based on practical problems [ 34 , 35 , 36 , 37 ]. By inputting a test image into the classification network, the network analyses the input image and returns a label that classifies the image. According to the difference of tasks achieved by the classification network method, it can be subdivided into three subcategories: using the network as a feature extractor, using the network for classification directly and using the network for lesions location.

Using network as feature extractor

In the early studies on plant diseases and pests classification methods based on deep learning, many researchers took advantage of the powerful feature extraction capability of CNN, and the methods were combined with traditional classifiers [ 38 ]. First, the images are input into a pretrained CNN network to obtain image characterization features, and the acquired features are then input into a conventional machine learning classifier (e.g., SVM) for classification. Yalcin et al. [ 39 ] proposed a convolutional neural network architecture to extract the features of images while performing experiments using SVM classifiers with different kernels and feature descriptors such as LBP and GIST, the experimental results confirmed the effectiveness of the approach. Fuentes et al. [ 40 ] put forward the idea of CNN based meta architecture with different feature extractors, and the input images included healthy and infected plants, which were identified as their respective classes after going through the meta architecture. Hasan et al. [ 41 ] identified and classified nine different types of rice diseases by using the features extracted from DCNN model and input into SVM, and the accuracy achieved 97.5%.

Using network for classification directly

Directly using classification network to classify lesions is the earliest common means of CNN applied in plant diseases and pests detection. According to the characteristics of existing research work, it can be further subdivided into original image classification, classification after locating Region of Interest (ROI) and multi-category classification.

Original image classification. That is, directly put the collected complete plant diseases and pests image into the network for learning and training. Thenmozhi et al. [ 42 ] proposed an effective deep CNN model, and transfer learning is used to fine-tune the pre-training model. Insect species were classified on three public insect datasets with accuracy of 96.75%, 97.47% and 95.97%, respectively. Fang et al. [ 43 ] used ResNet50 in plant diseases and pests detection. The focus loss function was used instead of the standard cross-entropy loss function, and the Adam optimization method was used to identify the leaf disease grade, and the accuracy achieved 95.61%.

Classification after locating ROI. For the whole image acquired, we should focus on whether there is a lesion in a fixed area, so we often obtain the region of interest (ROI) in advance, and then input the ROI into the network to judge the category of diseases and pests. Nagasubramanian et al. [ 44 ] used a new three-dimensional deep convolution neural network (DCNN) and salience map visualization method to identify healthy and infected samples of soybean stem rot, and the classification accuracy achieved 95.73%.

Multi-category classification. When the number of plant diseases and pests class to be classified exceed 2 class, the conventional plant diseases and pests classification network is the same as the original image classification method, that is, the output nodes of the network are the number of plant diseases and pests class + 1 (including normal class). However, multi-category classification methods often use a basic network to classify lesions and normal samples, and then share feature extraction parts on the same network to modify or increase the classification branches of lesion categories. This approach is equivalent to preparing a pre-training weight parameter for subsequent multi-objective plant diseases and pests classification network, which is obtained by binary training between normal samples and plant diseases and pests samples. Picon et al. [ 45 ] proposed a CNN architecture to identify 17 diseases in 5 crops, which seamlessly integrates context metadata, allowing training of a single multi-crop model. The model can achieve the following goals: (a) obtains richer and more robust shared visual features than the corresponding single crop; (b) is not affected by different diseases in which different crops have similar symptoms; (c) seamlessly integrates context to perform crop conditional disease classification. Experiments show that the proposed model alleviates the problem of data imbalance, and the average balanced accuracy is 0.98, which is superior to other methods and eliminates 71% of classifier errors.

Using network for lesions location

Generally, the classification network can only complete the classification of image label level. In fact, it can also achieve the location of lesions and the pixel-by-pixel classification by combining different techniques and methods. According to the different means used, it can be further divided into three forms: sliding window, heatmap and multi-task learning network.

Sliding window. This is the simplest and intuitive method to achieve the location of lesion coarsely. The image in the sliding window is input into the classification network for plant diseases and pests detection by redundant sliding on the original image through a smaller size window. Finally, all sliding windows are connected to obtain the results of the location of lesion. Chen et al. [ 46 ] used CNN classification network based on sliding window to build a framework for characteristics automatic learning, feature fusion, recognition and location regression calculation of plant diseases and pests species, and the recognition rate of 38 common symptoms in the field was 50–90%.

Heatmap. This is an image that reflects the importance of each region in the image, the darker the color represents the more important. In the field of plant diseases and pests detection, the darker the color in the heatmap represents the greater the probability that it is the lesion. In 2017, Dechant et al. [ 47 ] trained CNN to make heatmap to show the probability of infection in each region in maize disease images, and these heatmaps were used to classify the complete images, dividing each image into containing or not containing infected leaves. At runtime, it takes about 2 min to generate a heatmap for an image (1.6 GB of memory) and less than one second to classify a set of three heatmaps (800 MB of memory). Experiments show that the accuracy is 96.7% on the test dataset. In 2019, Wiesner-Hanks et al. [ 48 ] used heatmap method to obtain accurate contour areas of maize diseases, the model can accurately depict lesions as low as millimeter scale from the images collected by UAVs, with an accuracy rate of 99.79%, which is the best scale of aerial plant disease detection achieved so far.

Multi-task learning network. If the pure classified network does not add any other skills, it could only realize the image level classification. Therefore, to accurately locate the location of plant diseases and pests, the designed network should often add an extra branch, and the two branches would share the results of the feature extracting. In this way, the network generally had the classification and segmentation output of the plant diseases and pests, forming a multi-task learning network. It takes into account the characteristics of both network. For segmentation network branches, each pixel in the image can be used as a training sample to train the network. Therefore, the multi-task learning network not only uses the segmentation branches to output the specific segmentation results of the lesions, but also greatly reduces the requirements of the classification network for samples. Ren et al. [ 49 ] constructed a Deconvolution-Guided VGNet (DGVGNet) model to identify plant leaf diseases which were easily disturbed by shadows, occlusions and light intensity. The deconvolution was used to guide the CNN classifier to focus on the real lesion sites. The test results show that the accuracy of disease class identification is 99.19%, the pixel accuracy of lesion segmentation is 94.66%, and the model has good robustness in occlusion, low light and other environments.

To sum up, the method based on classification network is widely used in practice, and many scholars have carried out application research on the classification of plant diseases and pests [ 50 , 51 , 52 , 53 ]. At the same time, different sub-methods have their own advantages and disadvantages, as shown in Table 3 .

Detection network

Object positioning is one of the most basic tasks in the field of computer vision. It is also the closest task to plant diseases and pests detections in the traditional sense. Its purpose is to obtain accurate location and category information of the object. At present, object detection methods based on deep learning emerge endlessly. Generally speaking, plant diseases and pests detection network based on deep learning can be divided into: two stage network represented by Faster R-CNN [ 54 ]; one stage network represented by SSD [ 55 ] and YOLO [ 56 , 57 , 58 ]. The main difference between the two networks is that the two-stage network needs to first generate a candidate box (proposal) that may contain the lesions, and then further execute the object detection process. In contrast, the one-stage network directly uses the features extracted in the network to predict the location and class of the lesions.

Plant diseases and pests detection based on two stages network

The basic process of two-stage detection network (Faster R-CNN) is to obtain the feature map of the input image through the backbone network first, then calculate the anchor box confidence using RPN and get the proposal. Then, input the feature map of the proposal area after ROIpooling to the network, fine-tune the initial detection results, and finally get the location and classification results of the lesions. Therefore, according to the characteristics of plant diseases and pests detection, common methods often improve on the backbone structure or its feature map, anchor ratio, ROIpooling and loss function. In 2017, Fuentes et al. [ 59 ] first used Faster R-CNN to locate tomato diseases and pests directly, combined with deep feature extractors such as VGG-Net and ResNet, the mAP value reached 85.98% in a dataset containing 5000 tomato diseases and pests of 9 categories. In 2019, Ozguven et al. [ 60 ] proposed a Faster R-CNN structure for automatic detection of beet leaf spot disease by changing the parameters of CNN model. 155 images were trained and tested. The results show that the overall correct classification rate of this method is 95.48%. Zhou et al. [ 61 ] presented a fast rice disease detection method based on the fusion of FCM-KM and Faster R-CNN. The application results of 3010 images showed that: the detection accuracy and time of rice blast, bacterial blight, and sheath blight are 96.71%/0.65 s, 97.53%/0.82 s and 98.26%/0.53 s respectively. Xie et al. [ 62 ] proposed a Faster DR-IACNN model based on the self-built grape leaf disease dataset (GLDD) and Faster R-CNN detection algorithm, the Inception-v1 module, Inception-ResNet-v2 module and SE are introduced. The proposed model achieved higher feature extraction ability, the mAP accuracy was 81.1% and the detection speed was 15.01FPS. The two-stage detection network has been devoted to improving the detection speed to improve the real-time and practicability of the detection system, but compared with the single-stage detection network, it is still not concise enough, and the inference speed is still not fast enough.

Plant diseases and pests detection based on one stage network

The one-stage object detection algorithm has eliminated the region proposal stage, but directly adds the detection head to the backbone network for classification and regression, thus greatly improving the inference speed of the detection network. The single-stage detection network is divided into two types, SSD and YOLO, both of which use the whole image as the input of the network, and directly return the position of the bounding box and the category to which it belongs at the output layer.

Compared with the traditional convolutional neural network, the SSD selects VGG16 as the trunk of the network, and adds a feature pyramid network to obtain features from different layers and make predictions. Singh et al. [ 63 ] built the PlantDoc dataset for plant disease detection. Considering that the application should predict in mobile CPU in real time, an application based on MobileNets and SSD was established to simplify the detection of model parameters. Sun et al. [ 64 ] presented an instance detection method of multi-scale feature fusion based on convolutional neural network, which is improved on the basis of SSD to detect maize leaf blight under complex background. The proposed method combined data preprocessing, feature fusion, feature sharing, disease detection and other steps. The mAP of the new model is higher (from 71.80 to 91.83%) than that of the original SSD model. The FPS of the new model has also improved (from 24 to 28.4), reaching the standard of real-time detection.

YOLO considers the detection task as a regression problem, and uses global information to directly predict the bounding box and category of the object to achieve end-to-end detection of a single CNN network. YOLO can achieve global optimization and greatly improve the detection speed while satisfying higher accuracy. Prakruti et al. [ 65 ] presented a method to detect pests and diseases on images captured under uncontrolled conditions in tea gardens. YOLOv3 was used to detect pests and diseases. While ensuring real-time availability of the system, about 86% mAP was achieved with 50% IOU. Zhang et al. [ 66 ] combined the pooling of spatial pyramids with the improved YOLOv3, deconvolution is implemented by using the combination of up-sampling and convolution operation, which enables the algorithm to effectively detect small size crop pest samples in the image and reduces the problem of relatively low recognition accuracy due to the diversity of crop pest attitudes and scales. The average recognition accuracy can reach 88.07% by testing 20 class of pests collected in real scene.

In addition, there are many studies on using detection network to identify diseases and pests [ 47 , 67 , 68 , 69 , 70 , 71 , 72 , 73 ]. With the development of object detection network in computer vision, it is believed that more and more new detection models will be applied in plant diseases and pests detection in the future. In summary, in the field of plant diseases and pests detection which emphasizes detection accuracy at this stage, more models based on two-stage are used, and in the field of plant diseases and pests detection which pursue detection speed more models based on one-stage are used.

Can detection network replace classification network? The task of detection network is to solve the location problem of plant diseases and pests. The task of classification network is to judge the class of plant diseases and pests. Visually, the hidden information of detection network includes the category information, that is, the category information of plant diseases and pests that need to be located needs to be known beforehand, and the corresponding annotation information should be given in advance to judge the location of plant diseases and pests. From this point of view, the detection network seems to include the steps of the classification network, that is, the detection network can answer “what kind of plant diseases and pests are in what place”. But there is a misconception, in which “what kind of plant diseases and pests” is given a priori, that is, what is labelled during training is not necessarily the real result. In the case of strong model differentiation, that is, when the detection network can give accurate results, the detection network can answer “what kind of plant diseases and pests are in what place” to a certain extent. However, in the real world, in many cases, it cannot uniquely reflect the uniqueness of plant diseases and pests categories, only can answer “what kind of plant diseases and pests may be in what place”, then the involvement of the classification network is necessary. Thus, the detection network cannot replace the classification network.

Segmentation network

Segmentation network converts the plant diseases and pests detection task to semantic and even instance segmentation of lesions and normal areas. It not only finely divides the lesion area, but also obtains the location, category and corresponding geometric properties (including length, width, area, outline, center, etc.). It can be roughly divided into: Fully Convolutional Networks (FCN) [ 74 ] and Mask R-CNN [ 75 ].

Full convolution neural network (FCN) is the basis of image semantics segmentation. At present, almost all semantics segmentation models are based on FCN. FCN first extracts and codes the features of the input image using convolution, then gradually restores the feature image to the size of the input image by deconvolution or up sampling. Based on the differences in FCN network structure, the plant diseases and pests segmentation methods can be divided into conventional FCN, U-net [ 76 ] and SegNet [ 77 ].

Conventional FCN. Wang et al. [ 78 ] presented a new method of maize leaf disease segmentation based on full convolution neural network to solve the problem that traditional computer vision is susceptible to different illumination and complex background, and the segmentation accuracy reached 96.26. Wang et al. [ 79 ] proposed a plant diseases and pests segmentation method based on improved FCN. In this method, a convolution layer was used to extract multi-layer feature information from the input maize leaf lesion image, and the size and resolution of the input image were restored by deconvolution operation. Compared with the original FCN method, not only the integrity of the lesion was guaranteed, but also the segmentation of small lesion area was highlighted, and the accuracy rate reached 95.87%.

U-net. U-net is not only a classical FCN structure, but also a typical encoder-decoder structure. It is characterized by introducing a layer-hopping connection, fusing the feature map in the coding stage with that in the decoding stage, which is beneficial to the recovery of segmentation details. Lin et al. [ 80 ] used U-net based convolutional neural network to segment 50 cucumber powdery mildew leaves collected in natural environment. Compared with the original U-net, a batch normalization layer was added behind each convolution layer, making the neural network insensitive to weight initialization. The experiment shows that the convolutional neural network based on U-net can accurately segment powdery mildew on cucumber leaves at the pixel level with an average pixel accuracy of 96.08%, which is superior to the existing K-means, Random-forest and GBDT methods. The U-net method can segment the lesion area in a complex background, and still has good segmentation accuracy and segmentation speed with fewer samples.

SegNet. It is also a classical encoder–decoder structure. Its feature is that the up-sampling operation in the decoder takes advantage of the index of the largest pooling operation in the encoder. Kerkech et al. [ 81 ] presented an image segmentation method for unmanned aerial vehicles. Visible and infrared images (480 samples from each range) were segmented using SegNet to identify four categories: shadows, ground, healthy and symptomatic grape vines. The detection rates of the proposed method on grape vines and leaves were 92% and 87%, respectively.

Mask R-CNN is one of the most commonly used image instance segmentation methods at present. It can be considered as a multitask learning method based on detection and segmentation network. When multiple lesions of the same type have adhesion or overlap, instance segmentation can separate individual lesions and further count the number of lesions. However, semantic segmentation often treats multiple lesions of the same type as a whole. Stewart et al. [ 82 ] trained a Mask R-CNN model to segment maize northern leaf blight (NLB) lesions in an unmanned aerial vehicle image. The trained model can accurately detect and segment a single lesion. At the IOU threshold of 0.50, the IOU between the baseline true value and the predicted lesion was 0.73, and the average accuracy was 0.96. Also, some studies combine the Mask R-CNN framework with object detection networks for plant diseases and pests detection. Wang et al. [ 83 ] used two different models, Faster R-CNN and ask R-CNN, in which Faster R-CNN was used to identify the class of tomato diseases and Mask R-CNN was used to detect and segment the location and shape of the infected area. The results showed that the proposed model can quickly and accurately identify 11 class of tomato diseases, and divide the location and shape of infected areas. Mask R-CNN reached a high detection rate of 99.64% for all class of tomato diseases.

Compared with the classification and detection network methods, the segmentation method has advantages in obtaining the lesion information. However, like the detection network, it requires a lot of annotation data, and its annotation information is pixel by pixel, which often takes a lot of effort and cost.

Dataset and performance comparison

This section first gives a brief introduction to the plant diseases and pests related datasets and the evaluation index of deep learning model, then compares and analyses the related models of plant diseases and pests detection based on deep learning in recent years.

Datasets for plant diseases and pests detection

Plant diseases and pests detection datasets are the basis for research work. Compared with ImageNet, PASCAL-VOC2007/2012 and COCO in computer vision tasks, there is not a large and unified dataset for plant diseases and pests detection. The plant diseases and pests dataset can be acquired by self-collection, network collection and use of public datasets. Among them, self-collection of image dataset is often obtained by unmanned aerial remote sensing, ground camera photography, Internet of Things monitoring video or video recording, aerial photography of unmanned aerial vehicle with camera, hyperspectral imager, near-infrared spectrometer, and so on. Public datasets typically come from PlantVillage, an existing well-known public standard library. Relatively, self-collected datasets of plant diseases and pests in real natural environment are more practical. Although more and more researchers have opened up the images collected in the field, it is difficult to compare them uniformly based on different class of diseases under different detection objects and scenarios. This section provides links to a variety of plant diseases and pests detection datasets in conjunction with existing studies. As shown in Table 4 .

Evaluation indices

Evaluation indices can vary depending on the focus of the study. Common evaluation indices include \(Precision\) , \(Recall\) , mean Average Precision (mAP) and the harmonic Mean F1 score based on \(Precision\) and \(Recall\) .

\(Precision\) and \(Recall\) are defined as:

In Formula ( 1 ) and Formula ( 2 ), TP (True Positive) is true-positive, predicted to be 1 and actually 1, indicating the number of lesions correctly identified by the algorithm. FP (False Positive) is false-positive, predicted to be 1 and actually 0, indicating the number of lesions incorrectly identified by the algorithm. FN (False Negative) is false-negative, predicted to be 0 and actually 1, indicating the number of unrecognized lesions.

Detection accuracy is usually assessed using mAP. The average accuracy of each category in the dataset needs to be calculated first:

In the above-mentioned formula, \(N\left( {class} \right)\) represents the number of all categories, \(Precision\left( j \right)\) and \(Recall\left( j \right)\) represents the precision and recall of class j respectively.

Average accuracy for each category is defined as mAP:

The greater the value of \(mAP\) , the higher the recognition accuracy of the algorithm; conversely, the lower the accuracy of the algorithm.

F1 score is also introduced to measure the accuracy of the model. F1 score takes into account both the accuracy and recall of the model. The formula is

Frames per second (FPS) is used to evaluate the recognition speed. The more frames per second, the faster the algorithm recognition speed; conversely, the slower the algorithm recognition speed.

Performance comparison of existing algorithms

At present, the research on plant diseases and pests based on deep learning involves a wide range of crops, including all kinds of vegetables, fruits and food crops. The tasks completed include not only the basic tasks of classification, detection and segmentation, but also more complex tasks such as the judgment of infection degree.

At present, most of the current deep learning-based methods for plant diseases and pests detection are applied on specific datasets, many datasets are not publicly available, there is still no single publicly available and comprehensive dataset that will allow all algorithms to be uniformly compared. With the continuous development of deep learning, the application performance of some typical algorithms on different datasets has been gradually improved, and the mAP, F1 score and FPS of the algorithms have all been increased.

The breakthroughs achieved in the existing studies are amazing, but due to the fact that there is still a certain gap between the complexity of the infectious diseases and pests images in the existing studies and the real-time field diseases and pests detection based on mobile devices. Subsequent studies will need to find breakthroughs in larger, more complex, and more realistic datasets.

Small dataset size problem

At present, deep learning methods are widely used in various computer vision tasks, plant diseases and pests detection is generally regarded as specific application in the field of agriculture. There are too few agricultural plant diseases and pests samples available. Compared with open standard libraries, self-collected data sets are small in size and laborious in labeling data. Compared with more than 14 million sample data in ImageNet datasets, the most critical problem facing plant diseases and pests detection is the problem of small samples. In practice, some plant diseases have low incidence and high cost of disease image acquisition, resulting in only a few or dozen training data collected, which limits the application of deep learning methods in the field of plant diseases and pests identification. In fact, for the problem of small samples, there are currently three different solutions.

Data amplification, synthesis and generation

Data amplification is a key component of training deep learning models. An optimized data amplification strategy can effectively improve the plant diseases and pests detection effect. The most common method of plant diseases and pests image expansion is to acquire more samples using image processing operations such as mirroring, rotating, shifting, warping, filtering, contrast adjustment, and so on for the original plant diseases and pests samples. In addition, Generative Adversarial Networks (GANs) [ 93 ] and Variational automatic encoder (VAE) [ 94 ] can generate more diverse samples to enrich limited datasets.

Transfer learning and fine-tuning classical network model

Transfer learning (TL) transfers knowledge learned from generic large datasets to specialized areas with relatively small amounts of data. When transfer learning develops a model for newly collected unlabeled samples, it can start with a training model by a similar known dataset. After fine-tuning parameters or modifying components, it can be applied to localized plant disease and pest detection, which can reduce the cost of model training and enable the convolution neural network to adapt to small sample data. Oppenheim et al. [ 95 ] collected infected potato images of different sizes, hues and shapes under natural light and classified by fine-tuning the VGG network. The results showed that, the transfer learning and training of new networks were effective. Too et al. [ 96 ] evaluated various classical networks by fine-tuning and contrast. The experimental results showed that the accuracy of Dense-Nets improved with the number of iterations. Chen et al. [ 97 ] used transfer learning and fine-tuning to identify rice disease images under complex background conditions and achieved an average accuracy of 92.00%, which proves that the performance of transfer learning is better than training from scratch.

Reasonable network structure design

By designing a reasonable network structure, the sample requirements can be greatly reduced. Zhang et al. [ 98 ] constructed a three-channel convolution neural network model for plant leaf disease recognition by combining three color components. Each channel TCCNN component is composed of three color RGB leaf disease images. Liu et al. [ 99 ] presented an improved CNN method for identifying grape leaf diseases. The model used a depth-separable convolution instead of a standard convolution to alleviate overfitting and reduce the number of parameters. For the different size of grape leaf lesions, the initial structure was applied to the model to improve the ability of multi-scale feature extraction. Compared with the standard ResNet and GoogLeNet structures, this model has faster convergence speed and higher accuracy during training. The recognition accuracy of this algorithm was 97.22%.

Fine-grained identification of small-size lesions in early identification

Small-size lesions in early identification.

Accurate early detection of plant diseases is essential to maximize the yield [ 36 ]. In the actual early identification of plant diseases and pests, due to the small size of the lesion object itself, multiple down sampling processes in the deep feature extraction network tend to cause small-scale objects to be ignored. Moreover, due to the background noise problem on the collected images, large-scale complex background may lead to more false detection, especially on low-resolution images. In view of the shortage of existing algorithms, the improvement direction of small object detection algorithm is analyzed, and several strategies such as attention mechanism are proposed to improve the performance of small target detection.

The use of attention mechanism makes resources allocated more rationally. The essence of attention mechanism is to quickly find region of interest and ignore unimportant information. By learning the characteristics of plant diseases and pests images, features can be separated using weighted sum method with weighted coefficient, and the background noise in the image can be suppressed. Specifically, the attention mechanism module can get a salient image, and seclude the object from the background, and the Softmax function can be used to manipulate the feature image, and combine it with the original feature image to obtain new fusion features for noise reduction purposes. In future studies on early recognition of plant diseases and pests, attention mechanisms can be used to effectively select information and allocate more resources to region of interest to achieve more accurate detection. Karthik et al. [ 100 ] applied attention mechanism on the residual network and experiments were carried out using the plantVillage dataset, which achieved 98% overall accuracy.

Fine-grained identification

First, there is a large difference within the class, that is, the visual characteristics of plant diseases and pests belonging to the same class are quite different. The reason is that the aforementioned external factors such as uneven illumination, dense occlusion, blurred equipment dithering and other interferences, resulting in different image samples belonging to the same kind of diseases and pests differ greatly. Plant diseases and pests detection in complex scenarios is a very challenging task of fine-grained recognition [ 101 ]. The existence of growth variations of diseases and pests results in distinct differences in the characterization of the same diseases and pests at different stages, forming the “intra-class difference” fine-grained characteristics.

Secondly, there is fuzziness between classes, that is, objects of different classes have some similarity. There are many detailed classifications of biological subspecies and subclasses of different kinds of diseases and pests, and there are some similarities of biological morphology and life habits among the subclasses, which lead to the problem of fine-grained identification of “inter-class similarity”. Barbedo believed that similar symptoms could be produced, which even phytopathologists could not correctly distinguish [ 102 ].

Thirdly, background disturbance makes it impossible for plant diseases and pests to appear in a very clean background in the real world. Background can be very complex and interfere with objects of interest, which makes plant diseases and pests detection more difficult. Some literature often ignores this issue because images are captured under controlled conditions [ 103 ].

Relying on the existing deep learning methods can not effectively identify the fine-grained characteristics of diseases and pests that exist naturally in the application of the above actual agricultural scenarios, resulting in technical difficulties such as low identification accuracy and generalization robustness, which has long restricted the performance improvement of decision-making management of diseases and pests by the Intelligent Agricultural Internet of Things [ 104 ]. The existing research is only suitable for fine-grained identification of fewer class of diseases and pests, can not solve the problem of large-scale, large-category, accurate and efficient identification of diseases and pests, and is difficult to deploy directly to the mobile terminals of smart agriculture.

Detection performance under the influence of illumination and occlusion

Lighting problems.

Previous studies have collected images of plant diseases and pests mostly in indoor light boxes [ 105 ]. Although this method can effectively eliminate the influence of external light to simplify image processing, it is quite different from the images collected under real natural light. Because natural light changes very dynamically, and the range in which the camera can accept dynamic light sources is limited, it is easy to cause image color distortion when above or below this limit. In addition, due to the difference of view angle and distance during image collection, the apparent characteristics of plant diseases and pests change greatly, which brings great difficulties to the visual recognition algorithm.

Occlusion problem

At present, most researchers intentionally avoid the recognition of plant diseases and pests in complex environments. They only focus on a single background. They use the method of directly intercepting the area of interest to the collected images, but seldom consider the occlusion problem. As a result, the recognition accuracy under occlusion is low and the practicability is greatly reduced. Occlusion problems are common in real natural environments, including blade occlusion caused by changes in blade posture, branch occlusion, light occlusion caused by external lighting, and mixed occlusion caused by different types of occlusion. The difficulties of plant diseases and pests identification under occlusion are the lack of features and noise overlap caused by occlusion. Different occlusion conditions have different degrees of impact on the recognition algorithm, resulting in false detection or even missed detection. In recent years, with the maturity of deep learning algorithms under restricted conditions, some researchers have gradually challenged the identification of plant diseases and pests under occluded conditions [ 106 , 107 ], and significant progress has been made, which lays a good foundation for the application of plant diseases and pests identification in real-world scenarios. However, occlusion is random and complex. The training of the basic framework is difficult and the dependence on the performance of hardware devices still exists, we should strengthen the innovation and optimization of the basic framework, including the design of lightweight network architecture. The exploration of GAN and other aspects should be enhanced, while ensuring the accuracy of detection, the difficulty of model training should be reduced. GAN has prominent advantages in dealing with posture changes and chaotic background, but its design is not yet mature, and it is easy to crash in learning and cause model uncontrollable problems during training. We should strengthen the exploration of network performance to make it easier to quantify the quality of the model.

Detection speed problem

Compared with traditional methods, deep learning algorithms have better results, but their computational complexity is also higher. If the detection accuracy is guaranteed, the model needs to fully learn the characteristics of the image and increase the computational load, which will inevitably lead to slow detection speed and can not meet the needs of real-time. In order to ensure the detection speed, it is usually necessary to reduce the amount of calculation. However, this will cause insufficient training and result in false or missed detection. Therefore, it is important to design an efficient algorithm with both detection accuracy and detection speed.

Plant diseases and pests detection methods based on deep learning include three main links in agricultural applications: data labeling, model training and model inference. In real-time agricultural applications, more attention is paid to model inference. Currently, most plant diseases and pests detection methods focus on the accuracy of recognition. Little attention is paid to the efficiency of model inference. In reference [ 108 ], to improve the efficiency of the model calculation process to meet the actual agricultural needs, a deep separable convolution structure model for plant leaf disease detection was introduced. Several models were trained and tested. The classification accuracy of Reduced MobileNet was 98.34%, the parameters were 29 times less than VGG, and 6 times less than MobileNet. This shows an effective compromise between delay and accuracy, which is suitable for real-time crop diseases diagnosis on resource-constrained mobile devices.

Conclusions and future directions

Compared with traditional image processing methods, which deal with plant diseases and pests detection tasks in several steps and links, plant diseases and pests detection methods based on deep learning unify them into end-to-end feature extraction, which has a broad development prospects and great potential. Although plant diseases and pests detection technology is developing rapidly, it has been moving from academic research to agricultural application, there is still a certain distance from the mature application in the real natural environment, and there are still some problems to be solved.

Plant diseases and pests detection dataset

Deep learning technology has made some achievements in the identification of plant diseases and pests. Various image recognition algorithms have also been further developed and extended, which provides a theoretical basis for the identification of specific diseases and pests. However, the collection of image samples in previous studies mostly come from the characterization of disease spots, insect appearance characteristics or the characterization of insect pests and leaves. Most of the research results are limited to the laboratory environment and are applicable only to the plant diseases and pests images obtained at the time. The main reason for this is that the growth of plants is cyclical, continuous, seasonal and regional. Similarly, the characteristics of the same disease or pest at different growing stages of crops are different. Images of different plant species vary from region to region. As a result, most of the existing research results are not universal. Even with a high recognition rate in a single trial, the validity of the data obtained at other times cannot be guaranteed.

Most of the existing studies are based on the images generated in the visible range, but the electromagnetic wave outside the visible range also contains a lot of information, so the comprehensive information such as visible light, near infrared, multi-spectral should be fused to achieve the acquisition of plant diseases and pests dataset. Future research should focus on multi-information fusion method to obtain and identify plant diseases and pests information.

In addition, image databases of different kinds of plant diseases and pests in real natural environments are still in the blank stage. Future research should make full use of the data information acquisition platform such as portable field spore auto-capture instrument, unmanned aerial vehicle aerial photography system, agricultural internet of things monitoring equipment, which performs large-area and coverage identification of farmland and makes up for the lack of randomness of image samples in previous studies. Also, it can ensures the comprehensiveness and accuracy of dataset, and improves the generality of the algorithm.

Early recognition of plant diseases and pests

In the application of plant diseases and pests identification, the manifestation symptoms are not obvious, so early diagnosis is very difficult whether it is by visual observation or computer interpretation. However, the research significance and demand of early diagnosis are greater, which is more conducive to the prevention and control of plant diseases and pests and prevent their spread and development. The best image quality can be obtained when the sunlight is sufficient, and taking pictures in cloudy weather will increase the complexity of image preprocessing and reduce the recognition effect. In addition, in the early stage of plant diseases and pests occurrence, even high-resolution images are difficult to analyze. It is necessary to combine meteorological and plant protection data such as temperature and humidity to realize the recognition and prediction of diseases and pests. By consulting the existing research literatures, there are few reports on the early diagnosis of plant diseases and pests.

Network training and learning

When plant diseases and pests are visually identified manually, it is difficult to collect samples of all plant diseases and pests types, and many times only healthy data (positive samples) are available. However, most of the current plant diseases and pests detection methods based on deep learning are supervised learning based on a large number of diseases and pests samples, so manual collection of labelled datasets requires a lot of manpower, so unsupervised learning needs to be explored. Deep learning is a black box, which requires a large number of labelled training samples for end-to-end learning and has poor interpretability. Therefore, how to use the prior knowledge of brain-inspired computing and human-like visual cognitive model to guide the training and learning of the network is also a direction worthy of studying. At the same time, deep models need a large amount of memory and are extremely time-consuming during testing, which makes them unsuitable for deployment on mobile platforms with limited resources. It is important to study how to reduce complexity and obtain fast-executing models without losing accuracy. Finally, the selection of appropriate hyper-parameters has always been a major obstacle to the application of deep learning model to new tasks, such as learning rate, filter size, step size and number, these hyper-parameters have a strong internal dependence, any small adjustment may have a greater impact on the final training results.

Interdisciplinary research

Only by more closely integrating empirical data with theories such as agronomic plant protection, can we establish a field diagnosis model that is more in line with the rules of crop growth, and will further improve the effectiveness and accuracy of plant diseases and pests identification. In the future, it is necessary to go from image analysis at the surface level to identification of the occurrence mechanism of diseases and pests, and transition from simple experimental environment to practical application research that comprehensively considers crop growth law, environmental factors, etc.

In summary, with the development of artificial intelligence technology, the research focus of plant diseases and pests detection based on machine vision has shifted from classical image processing and machine learning methods to deep learning methods, which solved the difficult problems that could not be solved by traditional methods. There is still a long distance from the popularization of practical production and application, but this technology has great development potential and application value. To fully explore the potential of this technology, the joint efforts of experts from relevant disciplines are needed to effectively integrate the experience knowledge of agriculture and plant protection with deep learning algorithms and models, so as to make plant diseases and pests detection based on deep learning mature. Also, the research results should be integrated into agricultural machinery equipment to truly land the corresponding theoretical results.

Availability of data and materials

For relevant data and codes, please contact the corresponding author of this manuscript.

Lee SH, Chan CS, Mayo SJ, Remagnino P. How deep learning extracts and learns leaf features for plant classification. Pattern Recogn. 2017;71:1–13.

Article   Google Scholar  

Tsaftaris SA, Minervini M, Scharr H. Machine learning for plant phenotyping needs image processing. Trends Plant Sci. 2016;21(12):989–91.

Article   CAS   PubMed   Google Scholar  

Fuentes A, Yoon S, Park DS. Deep learning-based techniques for plant diseases recognition in real-field scenarios. In: Advanced concepts for intelligent vision systems. Cham: Springer; 2020.

Google Scholar  

Yang D, Li S, Peng Z, Wang P, Wang J, Yang H. MF-CNN: traffic flow prediction using convolutional neural network and multi-features fusion. IEICE Trans Inf Syst. 2019;102(8):1526–36.

Sundararajan SK, Sankaragomathi B, Priya DS. Deep belief cnn feature representation based content based image retrieval for medical images. J Med Syst. 2019;43(6):1–9.

Melnyk P, You Z, Li K. A high-performance CNN method for offline handwritten chinese character recognition and visualization. Soft Comput. 2019;24:7977–87.

Li J, Mi Y, Li G, Ju Z. CNN-based facial expression recognition from annotated rgb-d images for human–robot interaction. Int J Humanoid Robot. 2019;16(04):504–5.

Kumar S, Singh SK. Occluded thermal face recognition using bag of CNN(BoCNN). IEEE Signal Process Lett. 2020;27:975–9.

Wang X. Deep learning in object recognition, detection, and segmentation. Found Trends Signal Process. 2016;8(4):217–382.

Article   CAS   Google Scholar  

Boulent J, Foucher S, Théau J, St-Charles PL. Convolutional neural networks for the automatic identification of plant diseases. Front Plant Sci. 2019;10:941.

Article   PubMed   PubMed Central   Google Scholar  

Kumar S, Kaur R. Plant disease detection using image processing—a review. Int J Comput Appl. 2015;124(2):6–9.

Martineau M, Conte D, Raveaux R, Arnault I, Munier D, Venturini G. A survey on image-based insect classification. Pattern Recogn. 2016;65:273–84.

Jayme GAB, Luciano VK, Bernardo HV, Rodrigo VC, Katia LN, Claudia VG, et al. Annotated plant pathology databases for image-based detection and recognition of diseases. IEEE Latin Am Trans. 2018;16(6):1749–57.

Kaur S, Pandey S, Goel S. Plants disease identification and classification through leaf images: a survey. Arch Comput Methods Eng. 2018;26(4):1–24.

CAS   Google Scholar  

Shekhawat RS, Sinha A. Review of image processing approaches for detecting plant diseases. IET Image Process. 2020;14(8):1427–39.

Hinton GE, Salakhutdinov R. Reducing the dimensionality of data with neural networks. Science. 2006;313(5786):504–7.

Liu W, Wang Z, Liu X, et al. A survey of deep neural network architectures and their applications. Neurocomputing. 2017;234:11–26.

Fergus R. Deep learning methods for vision. CVPR 2012 Tutorial; 2012.

Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell. 2013;35(8):1798–828.

Article   PubMed   Google Scholar  

Boureau YL, Le Roux N, Bach F, Ponce J, Lecun Y. [IEEE 2011 IEEE international conference on computer vision (ICCV)—Barcelona, Spain (2011.11.6–2011.11.13)] 2011 international conference on computer vision—ask the locals: multi-way local pooling for image recognition; 2011. p. 2651–8.

Zeiler MD, Fergus R. Stochastic pooling for regularization of deep convolutional neural networks. Eprint Arxiv. arXiv:1301.3557 . 2013.

TensorFlow. https://www.tensorflow.org/ .

Torch/PyTorch. https://pytorch.org/ .

Caffe. http://caffe.berkeleyvision.org/ .

Theano. http://deeplearning.net/software/theano/ .

Krizhenvshky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional networks. In: Proceedings of the conference neural information processing systems (NIPS), Lake Tahoe, NV, USA, 3–8 December; 2012. p. 1097–105.

Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: Proceedings of the 2015 IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 7–12 June; 2015. p. 1–9.

Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv. arXiv:1409.1556 . 2014.

Xie S, Girshick R, Dollár P, Tu Z, He K. Aggregated residual transformations for deep neural networks. arXiv. arXiv:1611.05431 . 2017.

Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the AAAI conference on artificial intelligence. 2016.

Huang G, Lrj Z, Maaten LVD, et al. Densely connected convolutional networks. In: IEEE conference on computer vision and pattern recognition. 2017. p. 2261–9.

Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv. arXiv:1704.04861 . 2017.

Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K. SqueezeNet: AlexNet-level accuracy with 50 × fewer parameters and < 0.5 MB model size. arXiv. arXiv:1602.07360 . 2016.

Priyadharshini RA, Arivazhagan S, Arun M, Mirnalini A. Maize leaf disease classification using deep convolutional neural networks. Neural Comput Appl. 2019;31(12):8887–95.

Wen J, Shi Y, Zhou X, Xue Y. Crop disease classification on inadequate low-resolution target images. Sensors. 2020;20(16):4601.

Article   PubMed Central   Google Scholar  

Thangaraj R, Anandamurugan S, Kaliappan VK. Automated tomato leaf disease classification using transfer learning-based deep convolution neural network. J Plant Dis Prot. 2020. https://doi.org/10.1007/s41348-020-00403-0 .

Atila M, Uar M, Akyol K, Uar E. Plant leaf disease classification using efficientnet deep learning model. Ecol Inform. 2021;61:101182.

Sabrol H, Kumar S. Recent studies of image and soft computing techniques for plant disease recognition and classification. Int J Comput Appl. 2015;126(1):44–55.

Yalcin H, Razavi S. Plant classification using convolutional neural networks. In: 2016 5th international conference on agro-geoinformatics (agro-geoinformatics). New York: IEEE; 2016.

Fuentes A, Lee J, Lee Y, Yoon S, Park DS. Anomaly detection of plant diseases and insects using convolutional neural networks. In: ELSEVIER conference ISEM 2017—The International Society for Ecological Modelling Global Conference, 2017. 2017.

Hasan MJ, Mahbub S, Alom MS, Nasim MA. Rice disease identification and classification by integrating support vector machine with deep convolutional neural network. In: 2019 1st international conference on advances in science, engineering and robotics technology (ICASERT). 2019.

Thenmozhi K, Reddy US. Crop pest classification based on deep convolutional neural network and transfer learning. Comput Electron Agric. 2019;164:104906.

Fang T, Chen P, Zhang J, Wang B. Crop leaf disease grade identification based on an improved convolutional neural network. J Electron Imaging. 2020;29(1):1.

Nagasubramanian K, Jones S, Singh AK, Sarkar S, Singh A, Ganapathysubramanian B. Plant disease identification using explainable 3D deep learning on hyperspectral images. Plant Methods. 2019;15(1):1–10.

Picon A, Seitz M, Alvarez-Gila A, Mohnke P, Echazarra J. Crop conditional convolutional neural networks for massive multi-crop plant disease classification over cell phone acquired images taken on real field conditions. Comput Electron Agric. 2019;167:105093.

Tianjiao C, Wei D, Juan Z, Chengjun X, Rujing W, Wancai L, et al. Intelligent identification system of disease and insect pests based on deep learning. China Plant Prot Guide. 2019;039(004):26–34.

Dechant C, Wiesner-Hanks T, Chen S, Stewart EL, Yosinski J, Gore MA, et al. Automated identification of northern leaf blight-infected maize plants from field imagery using deep learning. Phytopathology. 2017;107:1426–32.

Wiesner-Hanks T, Wu H, Stewart E, Dechant C, Nelson RJ. Millimeter-level plant disease detection from aerial photographs via deep learning and crowdsourced data. Front Plant Sci. 2019;10:1550.

Shougang R, Fuwei J, Xingjian G, Peishen Y, Wei X, Huanliang X. Deconvolution-guided tomato leaf disease identification and lesion segmentation model. J Agric Eng. 2020;36(12):186–95.

Fujita E, Kawasaki Y, Uga H, Kagiwada S, Iyatomi H. Basic investigation on a robust and practical plant diagnostic system. In: IEEE international conference on machine learning & applications. New York: IEEE; 2016.

Mohanty SP, Hughes DP, Salathé M. Using deep learning for image-based plant disease detection. Front Plant Sci. 2016;7:1419. https://doi.org/10.3389/fpls.2016.01419 .

Brahimi M, Arsenovic M, Laraba S, Sladojevic S, Boukhalfa K, Moussaoui A. Deep learning for plant diseases: detection and saliency map visualisation. In: Zhou J, Chen F, editors. Human and machine learning. Cham: Springer International Publishing; 2018. p. 93–117.

Chapter   Google Scholar  

Barbedo JG. Plant disease identification from individual lesions and spots using deep learning. Biosyst Eng. 2019;180:96–107.

Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell. 2017;39(6):1137–49.

Liu W, Anguelov D, Erhan D, Szegedy C, Berg AC. SSD: Single shot MultiBox detector. In: European conference on computer vision. Cham: Springer International Publishing; 2016.

Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

Redmon J, Farhadi A. Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. p. 6517–25.

Redmon J, Farhadi A. Yolov3: an incremental improvement. arXiv preprint. arXiv:1804.02767 . 2018.

Fuentes A, Yoon S, Kim SC, Park DS. A robust deep-learning-based detector for real-time tomato plant diseases and pests detection. Sensors. 2017;17(9):2022.

Ozguven MM, Adem K. Automatic detection and classification of leaf spot disease in sugar beet using deep learning algorithms. Phys A Statal Mech Appl. 2019;535(2019):122537.

Zhou G, Zhang W, Chen A, He M, Ma X. Rapid detection of rice disease based on FCM-KM and faster R-CNN fusion. IEEE Access. 2019;7:143190–206. https://doi.org/10.1109/ACCESS.2019.2943454 .

Xie X, Ma Y, Liu B, He J, Wang H. A deep-learning-based real-time detector for grape leaf diseases using improved convolutional neural networks. Front Plant Sci. 2020;11:751.

Singh D, Jain N, Jain P, Kayal P, Kumawat S, Batra N. Plantdoc: a dataset for visual plant disease detection. In: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD. 2019.

Sun J, Yang Y, He X, Wu X. Northern maize leaf blight detection under complex field environment based on deep learning. IEEE Access. 2020;8:33679–88. https://doi.org/10.1109/ACCESS.2020.2973658 .

Bhatt PV, Sarangi S, Pappula S. Detection of diseases and pests on images captured in uncontrolled conditions from tea plantations. In: Proc. SPIE 11008, autonomous air and ground sensing systems for agricultural optimization and phenotyping IV; 2019. p. 1100808. https://doi.org/10.1117/12.2518868 .

Zhang B, Zhang M, Chen Y. Crop pest identification based on spatial pyramid pooling and deep convolution neural network. Trans Chin Soc Agric Eng. 2019;35(19):209–15.

Ramcharan A, McCloskey P, Baranowski K, Mbilinyi N, Mrisho L, Ndalahwa M, Legg J, Hughes D. A mobile-based deep learning model for cassava disease diagnosis. Front Plant Sci. 2019;10:272. https://doi.org/10.3389/fpls.2019.00272 .

Selvaraj G, Vergara A, Ruiz H, Safari N, Elayabalan S, Ocimati W, Blomme G. AI-powered banana diseases and pest detection. Plant Methods. 2019. https://doi.org/10.1186/s13007-019-0475-z .

Tian Y, Yang G, Wang Z, Li E, Liang Z. Detection of apple lesions in orchards based on deep learning methods of CycleGAN and YOLOV3-dense. J Sens. 2019. https://doi.org/10.1155/2019/7630926 .

Zheng Y, Kong J, Jin X, Wang X, Zuo M. CropDeep: the crop vision dataset for deep-learning-based classification and detection in precision agriculture. Sensors. 2019;19:1058. https://doi.org/10.3390/s19051058 .

Arsenovic M, Karanovic M, Sladojevic S, Anderla A, Stefanović D. Solving current limitations of deep learning based approaches for plant disease detection. Symmetry. 2019;11:21. https://doi.org/10.3390/sym11070939 .

Fuentes AF, Yoon S, Lee J, Park DS. High-performance deep neural network-based tomato plant diseases and pests diagnosis system with refinement filter bank. Front Plant Sci. 2018;9:1162. https://doi.org/10.3389/fpls.2018.01162 .

Jiang P, Chen Y, Liu B, He D, Liang C. Real-time detection of apple leaf diseases using deep learning approach based on improved convolutional neural networks. IEEE Access. 2019. https://doi.org/10.1109/ACCESS.2019.2914929 .

Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell. 2015;39(4):640–51.

He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: 2017 IEEE international conference on computer vision (ICCV). New York: IEEE; 2017.

Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Berlin: Springer; 2015. p. 234–41. https://doi.org/10.1007/978-3-319-24574-4_28 .

Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell. 2019;39(12):2481–95.

Wang Z, Zhang S. Segmentation of corn leaf disease based on fully convolution neural network. Acad J Comput Inf Sci. 2018;1:9–18.

Wang X, Wang Z, Zhang S. Segmenting crop disease leaf image by modified fully-convolutional networks. In: Huang DS, Bevilacqua V, Premaratne P, editors. Intelligent computing theories and application. ICIC 2019, vol. 11643. Lecture Notes in Computer Science. Cham: Springer; 2019. https://doi.org/10.1007/978-3-030-26763-6_62 .

Lin K, Gong L, Huang Y, Liu C, Pan J. Deep learning-based segmentation and quantification of cucumber powdery mildew using convolutional neural network. Front Plant Sci. 2019;10:155.

Kerkech M, Hafiane A, Canals R. Vine disease detection in UAV multispectral images using optimized image registration and deep learning segmentation approach. Comput Electron Agric. 2020;174:105446.

Stewart EL, Wiesner-Hanks T, Kaczmar N, Dechant C, Gore MA. Quantitative phenotyping of northern leaf blight in UAV images using deep learning. Remote Sens. 2019;11(19):2209.

Wang Q, Qi F, Sun M, Qu J, Xue J. Identification of tomato disease types and detection of infected areas based on deep convolutional neural networks and object detection techniques. Comput Intell Neurosci. 2019. https://doi.org/10.1155/2019/9142753 .

Hughes DP, Salathe M. An open access repository of images on plant health to enable the development of mobile disease diagnostics through machine learning and crowdsourcing. Comput Sci. 2015.

Shah JP, Prajapati HB, Dabhi VK. A survey on detection and classification of rice plant diseases. In: IEEE international conference on current trends in advanced computing. New York: IEEE; 2016.

Prajapati HB, Shah JP, Dabhi VK. Detection and classification of rice plant diseases. Intell Decis Technol. 2017;11(3):1–17.

Barbedo JGA, Koenigkan LV, Halfeld-Vieira BA, Costa RV, Nechet KL, Godoy CV, Junior ML, Patricio FR, Talamini V, Chitarra LG, Oliveira SAS. Annotated plant pathology databases for image-based detection and recognition of diseases. IEEE Latin Am Trans. 2018;16(6):1749–57.

Brahimi M, Arsenovic M, Laraba S, Sladojevic S, Boukhalfa K, Moussaoui A. Deep learning for plant diseases: detection and saliency map visualisation. In: Zhou J, Chen F, editors. Human and machine learning. Human–computer interaction series. Cham: Springer; 2018. https://doi.org/10.1007/978-3-319-90403-0_6 .

Tyr WH, Stewart EL, Nicholas K, Chad DC, Harvey W, Nelson RJ, et al. Image set for deep learning: field images of maize annotated with disease symptoms. BMC Res Notes. 2018;11(1):440.

Thapa R, Snavely N, Belongie S, Khan A. The plant pathology 2020 challenge dataset to classify foliar disease of apples. arXiv preprint. arXiv:2004.11958 . 2020.

Wu X, Zhan C, Lai YK, Cheng MM, Yang J. IP102: a large-scale benchmark dataset for insect pest recognition. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). New York: IEEE; 2019.

Huang M-L, Chuang TC. A database of eight common tomato pest images. Mendeley Data. 2020. https://doi.org/10.17632/s62zm6djd2.1 .

Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. In: Proceedings of the 2014 conference on advances in neural information processing systems 27. Montreal: Curran Associates, Inc.; 2014. p. 2672–80.

Pu Y, Gan Z, Henao R, et al. Variational autoencoder for deep learning of images, labels and captions [EB/OL]. 2016–09–28. arxiv:1609.08976 .

Oppenheim D, Shani G, Erlich O, Tsror L. Using deep learning for image-based potato tuber disease detection. Phytopathology. 2018;109(6):1083–7.

Too EC, Yujian L, Njuki S, Yingchun L. A comparative study of fine-tuning deep learning models for plant disease identification. Comput Electron Agric. 2018;161:272–9.

Chen J, Chen J, Zhang D, Sun Y, Nanehkaran YA. Using deep transfer learning for image-based plant disease identification. Comput Electron Agric. 2020;173:105393.

Zhang S, Huang W, Zhang C. Three-channel convolutional neural networks for vegetable leaf disease recognition. Cogn Syst Res. 2018;53:31–41. https://doi.org/10.1016/j.cogsys.2018.04.006 .

Liu B, Ding Z, Tian L, He D, Li S, Wang H. Grape leaf disease identification using improved deep convolutional neural networks. Front Plant Sci. 2020;11:1082. https://doi.org/10.3389/fpls.2020.01082 .

Karthik R, Hariharan M, Anand S, et al. Attention embedded residual CNN for disease detection in tomato leaves. Appl Soft Comput J. 2020;86:105933.

Guan W, Yu S, Jianxin W. Automatic image-based plant disease severity estimation using deep learning. Comput Intell Neurosci. 2017;2017:2917536.

Barbedo JGA. Factors influencing the use of deep learning for plant disease recognition. Biosyst Eng. 2018;172:84–91.

Barbedo JGA. Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification. Comput Electron Agric. 2018;153:46–53.

Nawaz MA, Khan T, Mudassar R, Kausar M, Ahmad J. Plant disease detection using internet of thing (IOT). Int J Adv Comput Sci Appl. 2020. https://doi.org/10.14569/IJACSA.2020.0110162 .

Martinelli F, Scalenghe R, Davino S, Panno S, Scuderi G, Ruisi P, et al. Advanced methods of plant disease detection. A review. Agron Sustain Dev. 2015;35(1):1–25.

Liu J, Wang X. Early recognition of tomato gray leaf spot disease based on MobileNetv2-YOLOv3 model. Plant Methods. 2020;16:83.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Liu J, Wang X. Tomato diseases and pests detection based on improved Yolo V3 convolutional neural network. Front Plant Sci. 2020;11:898.

Kamal KC, Yin Z, Wu M, Wu Z. Depthwise separable convolution architectures for plant disease classification. Comput Electron Agric. 2019;165:104948.

Download references

Acknowledgements

Appreciations are given to the editors and reviewer of the Journal Plant Method.

This study was supported by the Facility Horticulture Laboratory of Universities in Shandong with Project Numbers 2019YY003, 2018YY016, 2018YY043 and 2018YY044; school level High-level Talents Project 2018RC002; Youth Fund Project of Philosophy and Social Sciences of Weifang College of Science and Technology with project numbers 2018WKRQZ008 and 2018WKRQZ008-3; Key research and development plan of Shandong Province with Project Number 2019RKA07012, 2019GNC106034 and 2020RKA07036; Research and Development Plan of Applied Technology in Shouguang with Project Number 2018JH12; 2018 innovation fund of Science and Technology Development centre of the China Ministry of Education with Project Number 2018A02013; 2019 basic capacity construction project of private colleges and universities in Shandong Province; and Weifang Science and Technology Development Programme with project numbers 2019GX081 and 2019GX082, Special project of Ideological and political education of Weifang University of science and technology (W19SZ70Z01).

Author information

Authors and affiliations.

Shandong Provincial University Laboratory for Protected Horticulture, Blockchain Laboratory of Agricultural Vegetables, Weifang University of Science and Technology, Weifang, 262700, Shandong, China

Jun Liu & Xuewei Wang

You can also search for this author in PubMed   Google Scholar

Contributions

JL designed the research. JL and XW conducted the experiments and data analysis and wrote the manuscript. XW revised the manuscript. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Xuewei Wang .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Liu, J., Wang, X. Plant diseases and pests detection based on deep learning: a review. Plant Methods 17 , 22 (2021). https://doi.org/10.1186/s13007-021-00722-9

Download citation

Received : 20 September 2020

Accepted : 13 February 2021

Published : 24 February 2021

DOI : https://doi.org/10.1186/s13007-021-00722-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Deep learning
• Plant diseases and pests
• Classification
• Object detection
• Segmentation

Plant Methods

ISSN: 1746-4811

• Submission enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 07 April 2021

Development of machine learning model for diagnostic disease prediction based on laboratory tests

• Dong Jin Park 1 ,
• Min Woo Park 2 ,
• Homin Lee 3 ,
• Young-Jin Kim 4 ,
• Yeongsic Kim 5 &
• Young Hoon Park 6  

Scientific Reports volume  11 , Article number:  7567 ( 2021 ) Cite this article

39k Accesses

78 Citations

7 Altmetric

Metrics details

• Experimental models of disease
• Information theory and computation
• Machine learning

The use of deep learning and machine learning (ML) in medical science is increasing, particularly in the visual, audio, and language data fields. We aimed to build a new optimized ensemble model by blending a DNN (deep neural network) model with two ML models for disease prediction using laboratory test results. 86 attributes (laboratory tests) were selected from datasets based on value counts, clinical importance-related features, and missing values. We collected sample datasets on 5145 cases, including 326,686 laboratory test results. We investigated a total of 39 specific diseases based on the International Classification of Diseases, 10th revision (ICD-10) codes. These datasets were used to construct light gradient boosting machine (LightGBM) and extreme gradient boosting (XGBoost) ML models and a DNN model using TensorFlow. The optimized ensemble model achieved an F1-score of 81% and prediction accuracy of 92% for the five most common diseases. The deep learning and ML models showed differences in predictive power and disease classification patterns. We used a confusion matrix and analyzed feature importance using the SHAP value method. Our new ML model achieved high efficiency of disease prediction through classification of diseases. This study will be useful in the prediction and diagnosis of diseases.

Similar content being viewed by others

An active learning machine technique based prediction of cardiovascular heart disease from UCI-repository database

Machine learning for characterizing risk of type 2 diabetes mellitus in a rural Chinese population: the Henan Rural Cohort Study

Machine learning analysis of lab tests to predict bariatric readmissions

Introduction.

Machine learning (ML) has had tremendous impacts on numerous areas of modern society. For example, it is used for filtering spam messages from text documents, such as e-mail, analyzing various images to distinguish differences, and extraction of important data from large datasets through data mining. ML makes it possible to uncover patterns, construct models, and make predictions by learning from training data 1 , 2 . ML algorithms are used in a broad range of domains, including biology and genomics 3 , 4 . Deep learning (DL) is a subset of ML that differs from other ML processes in many ways. Most ML models perform well due to their custom-designed representation and input features. Using the input data generated through that process, ML learns algorithms, optimizes the weights of each feature, and optimizes the final prediction. DL attempts to learn multiple levels of representation using a hierarchy of multiple layers 5 . In recent years, DL has overtaken ML in many areas, including speech, vision, and natural language processing 1 , 6 . DL and ML are also increasingly used in the medical field, mainly in the areas of ophthalmology and speech 3 , 7 . The deep neural network (DNN) is a type of DL that uses multiple hidden layers 8 and is renowned for analysis of high-dimensional data. As bioinformatics data are usually high dimensional, DNN may be a suitable model for bioinformatics research 2 , 5 , 9 . In addition to image and text data from medical charts generated in hospitals, various types of laboratory data must be analyzed, which are mostly composed of numbers. However, very few studies have applied DNN to structured numerical medical data.

In practice, the symptoms described by patients, physical examinations performed by physicians, laboratory test results, and imaging studies such as X-ray and computed tomography (CT) are generally needed to evaluate a patient’s status and diagnose a specific disease. However, little research has been conducted into the predictive power and accuracy that can be achieved using laboratory data alone for the diagnosis of specific diseases. Therefore, the purpose of this study was to develop predictive models that can be used by physicians to make decisions in the hospital setting based on DL and ML using laboratory data alone, and then to validate our model through comparison of its predictions with the diagnoses of physicians. In addition, we generated an ensemble of DL and ML models to improve performance. The Shapley additive explanation (SHAP) method 10 , 11 , which was recently developed, was used to determine the features that are important to each disease and to identify predictive relationships between diseases and features. In this manner, it is possible to obtain utility from laboratory tests that would not be conclusive otherwise.

Performance evaluation of neural network

We developed a new neural network (DL model) and trained it using 88 different parameters (86 laboratory test features, sex and age). We validated our DL algorithm using stratified five-fold cross-validation. We applied the TOP5 criterion (five most likely diseases) for evaluation of the model, as the purpose of our study was to create an artificial intelligence (AI) model to assist physicians with diagnosing diseases. In addition, we assessed the performance of each model using F1-scores, because there was an imbalance problem in the number of each 39 diseases. The F1-score and accuracy of our DL model were 80% and 91%, respectively. Precision and recall were 77% and 87%, respectively, for the five most common disease categories in Supplementary Table S2 .

Interestingly, the DL model showed a different result in prediction of disease categories compared to the two tree-based boosting ML models (LightGBM, XGBoost). The top 10 diseases in terms of F1-score for the DL model were tuberculosis pleurisy, acute hepatitis B, malaria, acute lymphoblastic leukemia, acute leukemia of unspecified cell type, aplastic anemia, scrub typhus, acute myocardial infarction, acute hepatitis A, and acute pyelonephritis, in that order in Table 1 . In addition, differences in performance were observed between the DNN and ML models for prediction of 38 different diseases. That is, DNN showed higher prediction performance for specific disease categories (sepsis, scrub typhus, acute hepatitis A, other specified acute viral hepatitis, acute respiratory distress syndrome, liver abscess, and urinary tract infection (UTI)).

ML results (XGBoost, LightGBM)

We developed two ML algorithms using LightGBM and XGBoost. These algorithms were trained using 88 different parameters in the same dataset used with the DL model. We proceeded with the learning process, leaving missing values in place due to the possibility of bias when missing values are replaced with other values (mean, median, etc.). We validated our two ML algorithms using stratified five-fold cross-validation. For the LightGBM prediction algorithm, the F1 score, accuracy, precision, and recall were 76%, 91%, 73%, and 85% for the five most likely diseases, respectively in Supplementary Table S3 . For the XGBoost prediction model, the F1 score, accuracy, precision, and recall were 78%, 93%, 76%, and 86% for the five most likely diseases, respectively in Supplementary Table S4 . Using the F1-score criterion, LightGBM sequentially showed malaria, toxic liver disease with hepatitis, acute myocardial infarction, unstable angina, acute pancreatitis, liver cirrhosis, acute hepatitis A, diabetic ketoacidosis, end-stage renal disease (ESRD), and tuberculosis as having the 10 highest F1-scores in Supplementary Table S5 . The XGBoost model sequentially showed malaria, toxic liver disease with hepatitis, acute myocardial infarction, unstable angina, liver cirrhosis, infectious colitis, acute hepatitis, ESRD, acute hepatitis A, and diabetic ketoacidosis to have the highest 10 F1-scores in Supplementary Table S6 . Thus, a slight difference was observed between the two tree-based ML models in ranking and disease classification. While no significant differences in the predictive power for disease classification were observed between the two boosting models of the tree series, a significant difference was found in predictive power between the two ML models and DL.

Ensemble model results (DNN, ML)

We developed a new ensemble model by combining our DL model with our two ML models to improve AI performance. We used the validation loss for model optimization. Finally, we created the optimized ensemble model. It achieved an F1-score of 81%, 92% prediction accuracy, 78% precision, and 88% recall in Supplementary Table S7 . Compared to our DL and two ML models, the new ensemble model achieved improved performance in all aspects, including F1-score, accuracy, precision, and recall. It achieved the best F1-score, 81%. To improve accuracy further, we constructed an additional ensemble model based on accuracy, which raised the accuracy to 93% in Supplementary Table S8 . The ensemble model showed differences in disease prediction compared to the ML and DL. Using the F1-score criterion, the top 10 diseases were acute hepatitis B, malaria, aplastic anemia, meningitis, acute myocardial infarction, acute pyelonephritis, infectious colitis, alcoholic hepatitis, acute pancreatitis, and ESRD in Table 2 . Similarly, the ensemble model optimized for accuracy differed from other models in the prediction of disease categories in Table 3 .

Confusion matrix of the ensemble model (optimal accuracy model)

We used confusion matrices for the ensemble model optimized by accuracy. The horizontal axis in Fig.  1 is predicted diseases and the vertical axis is true diseases. Specific label disease classifications were described in Supplementary Table S9 . The predictive power (accuracy) of TOP1 (representing the most likely disease) and TOP5 (the five most likely diseases) in the ensemble model was 65% and 93%, respectively in Fig.  1 A,B. Specifically, the TOP1 accuracy of the ensemble model was greater than 60% for a large number of diseases. The diseases with accuracies greater than 70% were A7 (scrub typhus), B1A (acute hepatitis A), C9B (acute myeloid leukemia), D6 (aplastic anemia, unspecified), E1 (DKA), I2B (acute myocardial infarction, unspecified), J1 (pneumonia, unspecified), LC (liver cirrhosis), K7D (liver abscess), K8B (cholangitis), K8C (acute pancreatitis), N1A (acute pyelonephritis), and N1C (ESRD). In particular, acute hepatitis and chronic hepatitis tended to be assigned to B1A (acute hepatitis A) by the TOP1 accuracy ensemble model. In addition, several other diseases tended to be predicted as J1 (pneumonia, unspecified). Using the TOP5 accuracy ensemble model, overall disease prediction was very high, with accuracy of 93%. For most diseases, we observed high accuracy, with the model achieving improved prediction accuracy for identifying acute hepatitis and chronic hepatitis as B1A (acute hepatitis A), B1B (acute hepatitis B), B1C (other specified acute viral hepatitis), or B1D (chronic hepatitis B, active). In addition, the TOP5 accuracy ensemble model exhibited improved accuracy for classifying pneumonia, unspecified (J1) compared to the TOP1 accuracy ensemble model. However, non-Hodgkin lymphoma, unspecified (C8) and acute leukemia of unspecified cell type (C9C) were among 39 disease categories that were not predicted at all.

Confusion matrix of the ensemble model (optimal accuracy model): ( A ) The predictive power (accuracy) of TOP1 (representing the most likely disease) result, ( B ) TOP5 (the five most likely diseases) result.

Differences in the predictive power for disease classification by DL and ML

Compared to two ML models (LightGBM, XGBoost), the DNN model showed higher precision, recall, and F1-score values for sepsis (A4), scrub typhus (A7), acute hepatitis A (B1A), other specified acute viral hepatitis (B1C), acute respiratory distress disease (J8), liver abscess (K7D), and UTI (N3). Meanwhile, the ML models showed higher scores for infectious colitis (A0B), pulmonary tuberculosis (A1A), unstable angina (I2A), congestive heart failure (I5), pneumonia, unspecified (J1), acute respiratory failure (J9), liver cirrhosis (LC), cholangitis (K8B), acute pancreatitis (K8C), acute renal failure (N1B), and ESRD (N1C).

Feature importance in ML (LightGBM, XGBoost)

The two ML models are similar tree series boosting models, but had differences in feature importance. In LightGBM, the important features were CP (C-reactive protein, CRP), LD (lactate dehydrogenase), GP (γ-glutamyl transpeptidase, γ-GTP), AL (alanine transaminase, ALT), PL (platelet). In the XGBoost model, the important features were CM (CK-MB, creatine kinase-myocardial band isoenzyme), WM (white blood cell in urine by microscopic examination), ALN (ALT/ALT_normal), S (sex), PB (pro-brain natriuretic peptide, Pro BNP), in order of importance. In addition, the Pusan National University team performed feature extraction differently from XGBoost, using a different number of attributes, and the results are shown in Supplement table S10. These results show that even within the same XGBoost ML model, the ranking of feature importance differed according to differences in the engineering process.

Evaluation of the average impact on model performance of using the mean SHAP value (correlations between feature importance and individual diseases)

Each disease has a unique ICD-10 code. While the feature importance method (analysis or coding) simply indicates the importance of a given feature (parameter), the mean SHAP method allows the effect of the features on classification (specific disease) to be calculated. We analyzed the mean SHAP values of the two ML models to determine how many features (laboratory tests) correlated with diseases (Figs.  2 and 3 ). For LightGBM, the TOP10 features with the highest predictive power among the 88 features tested were in the order of CP, LD, GP (γ-GTP), AL, PL, AG (age), TB (total bilirubin), Amyl (amylase), TrT (troponin-T), and W (white blood cell count). In particular, LD and GP were associated with malaria (B5) and acute hepatitis B (B1B), respectively (Supplementary Table S1 , S9 and Fig.  2 ). For XGBoost, CP, PL, AL, LD, GP, W, AG, Amyl, TB, and CR were the ten most important features. Investigating the associations between diseases and features, we confirmed that acute myocardial infarction (I2B) contributed most to the increased CM level. In addition, increased CR was related to ESRD (N1C). Dilated cardiomyopathy (I4) and congestive heart failure (I5) were strongly associated with PB. AST (aspartate aminotransferase) was associated with toxic liver disease (K7B), while an increase in AL was associated with acute hepatitis A (B1A) (Supplementary Table S1 , S9 and Fig.  3 ). For the DL model, the TOP10 features were CP, AG, CR, PB (Pro BNP), PL, TB, AB-O-Con (ABGA-O2 Content), GP, T/A (total protein/albumin), AB (albumin) using mean SHAP value.

The mean SHAP method result between parameters and disease classifications in LightGBM.

The mean SHAP method result between parameters and disease classifications in XGBoost.

Comparison of AI models (DNN, ML models, and ensemble model) with physicians

Using 88 laboratory parameters (86 laboratory tests, sex and age) for 39 diseases, we compared the results obtained from both our AI models (ML, DL and ensemble models) with physicians’ performance. The same input data were analyzed by five internal medicine specialists, who were assigned a total of 390 analyses with 78 questions each.

Five different physicians showed mean accuracy of 20% for the TOP1 (most likely disease) disease selection (accuracy range: 15–27%, median value: 19%) while the accuracy of AI models was higher (accuracy of ML XGBoost: 65%, LightGBM: 63%, DL: 61%, ensemble: 65%). Five physicians showed diagnostic mean accuracy of 47% for TOP5 disease selection (accuracy range: 33–54%, median value: 50%). However, the accuracies of XGBoost, LightGBM, DL, and the ensemble were 93%, 91%, 91%, and 93%, respectively, for the TOP5 diseases. Overall, the AI model had better diagnostic accuracy than the physicians. We found that the features that contributed to the physicians answering correctly were CP, Pro-BNP, CM, ALT, and AST, in order of importance.

ROC curve of the optimized ensemble model

For the optimized ensemble model, the modified ICD-10 codes (ICD 10 CODE(M)) for each disease were described in Supplementary Table S9 . We classified each disease into class I (group A: A0A, A0B, A0C, A1A, A1B, A4, A7, group B: B1A, B1B, B1C, B1D, B5), class II (C8, C9, C9A, C9B, C9C, D6, E1, G0, I2A, I2B, I4, I5) and Class III (J1, J8, J9, K7A, K7B, K7C, K7D, K8A, K8B, K8C, N0, N1A, N1B, NAC, N3) based on ICD 10 CODE(M). Using python's scikit-learn library, ROC (receiver operating characteristic) curves were drawn for each disease to measure AUC (area under curve). They were shown in Supplementary Figure  1A , B and C . In Class I, the AUC is higher in group B (AUC range: 0.97–1.0) than in group A (AUC range: 0.87–0.98). Among group A, the lowest AUC disease was sepsis (A4) (AUC: 0.87), and the highest disease was tuberculosis pleurisy (A1B). Among group B, the lowest AUC disease was other specified acute viral hepatitis (B1C) (AUC: 0.97), and the highest AUC disease was malaria (B5) (AUC: 1.0). In class II, the overall AUC results ranged from 0.94 to 1.00. In class III, the results of AUC ranged from 0.89 to 0.99. Among them, ARDS (J8), acute cholecystitis (K8A), and UTI (N3) diseases showed relatively low AUC results. The rest of the class III showed AUC results ranging from 0.93 to 0.99.

In this study, our research has a novel point that we could predict 39 diseases accurately that are relatively commonly observed in patients visiting the emergency room through our model. We applied DL and ML models to laboratory data (features). For DL, we used a neural network with two hidden layers, with Relu as the activation function for the input and hidden layers and Softmax as the activation function for the output layer. We tried to improve performance through a hyperparameter optimization process. In general, deepening the neural network layer caused the serious problem of gradient vanishing. In this study, the use of three or more hidden layers caused gradient vanishing and overfitting problems, such as validation loss, to increase. Our neural net model demonstrated very good performance because all data (features), except for sex, were numerical and we optimized performance through tuning of hyperparameters. In this study, the DL (neural network) model also performed well in a categorical classification problem (disease classification) using structured medical data. Among ML models, we selected the LightGBM and XGBoost ML models because they are the state of the art (SOTA) boosting models that show the best performance for a general classification problem. In general, tree-based ML models are known to show good results for classification 12 , 13 , 14 . In contrast, the support vector machine (SVM) and random forest (RF) models showed poor performance (accuracy of SVM: 57%, RF: 61%). For each ML technique, LightGBM and XGBoost showed better performance when the maximum depth was shallow, i.e., when the patient's laboratory data were relatively uncomplicated. We made various efforts to determine the optimal conditions for DL, and the optimal result was obtained using two hidden layers. This result indicates that the data were relatively simple, and we used the dropout method. In the AI of our DL (neural network) model, we used a batch size of 128 and 10 epochs to cover 5145 cases that included 326,686 laboratory test results, meaning that 5145 cases were divided into 128 units and 10 analyses were performed. To calculate the optimal validation loss, we adopted an early stopping method, which caused calculation to stop if the validation loss value did not improve. In this study, analysis was conducted using a patience value of 10 as an option. We analyzed the test set with the optimal weight and bias values obtained through the validation process for a training dataset. To avoid data loss during validation, we used five-fold validation. After the optimization process above, the optimal weight (w) and bias were calculated and used with the test set. An important characteristic of this dataset was that most of its data were numerical. For numerical data, ML generally shows good performance in computational prediction 4 , 15 . In this study, we aimed to compare DL (neural net) with ML methods. We found that the easiest way to increase performance of ML is proper model selection considering the dataset. Although the completeness of our dataset varied among the 88 total features (parameters), our dataset had approximately 26% missing values. Some features had more than 50% missing values.

For classification problems, the random forest method, a traditional bagging technique, shows good results 16 , 17 , 18 . However, in this study, the results from that method were worse than those of the two ML (LightGBM and XGBoost), which might have been due to the high frequency of missing values. A random forest can be analyzed only if there are no missing values. Thus, missing values must be replaced with the mean, median, or some other value prior to analysis. However, LightGBM and XGBoost automatically fill in the optimal values to replace missing values to calculate the results in an efficient manner 19 . We replaced missing values in LightGBM and XGBoost with the mean and median values and then analyzed performance, which was poor. The decision of whether to replace the missing values for analysis of a dataset should be made considering the characteristics of the data. In our study, we found that replacing missing values with mean or median values biased the results in ML. With regard to DL, because data could only be analyzed if it lacked missing values, we replaced the missing data with median values in DL.

In the present study, physicians showed very low diagnostic accuracy compared to our AI models. Practically, physicians’ diagnosis of a specific disease is made based on the symptom of the patient, physical examination, variable hospital data including laboratory test, image data (computed tomography, x-ray, MRI) and so on. Addition of a lot of unstructured data such as patients’ symptoms and signs, image data (x-ray or computed tomography findings), and biosignals (heart rate, body temperature, etc.) could result in a better and accurate prediction model. However, in real-world practice, it is difficult to use unstructured data because most of these data are in the form of free text, which is not standardization between the hospitals and there is a lot of missing data. Also, it takes a lot of manpower and computing power to process and handle these unstructured data. However, because the performance of the AI model was better than that of physicians using the same limited laboratory dataset, we have the opportunity to re-examine underutilized laboratory data (features, parameters). In particular, both TOP1 and TOP5 showed huge differences in accuracy between the physicians and ML models, showing that the ML model developed in this study could be very helpful for diagnosing diseases. This study further explored correlations between each feature and specific diseases using the newly developed SHAP value library. Interestingly, we found some generally unknown correlations between features (laboratory tests) and classifications (disease), such as increased LDH in malaria and high ALT in viral hepatitis, which are consistent with the results of previous studies 20 , 21 , 22 . In this study, we not only developed meaningful AI models and compared their performance but also revealed associations between diseases and features from the input data.

Regarding the overfitting problem, we separated about 1029 cases (independent data) from the total cases using python library “train test split”. Because our data set is imbalance data, we randomly and evenly selected each specific disease from the data set using stratify option. Also, for a robust model, k-fold cross validation was used. We compared the classic supervised classification models (SVM and RF) and our models. To overcome the overfitting issue, we used two hidden layers in the DNN model, but if more than three were used, the performance was rather degraded. This is thought to be because hospital laboratory data are numerical data and are not somewhat complicated. The reason why we used the DNN model is that deep learning models learn patterns which the tree model cannot solve, so deep learning actually well predicts different diseases (Table 1 , Supplementary Table S5 , Supplementary Table S6 ).

In summary, in the present study, we demonstrated that DL and ML could achieve favorable outcomes for a disease classification problem using hospital data. The application of our AI model to large datasets shed new light on the values of various laboratory tests that were not previously recognized. This study contributes to the development of medical informatics by helping AI to learn new patterns of information beyond determining relationships between laboratory data and diseases.

The principal procedures of the disease prediction model based on laboratory tests (DPMLT) are described in the following sections. The overall workflow of DPMLT is schematically demonstrated in Fig.  4 .

Overall framework of DPMLT. The development of DPMLT methodology involved for three major steps. (1) Data collection and preprocessing (2) Model selection, training and ensemble modeling (3) Performance evaluation.

Data collection and preprocessing

We analyzed datasets provided by the Department of Internal Medicine from patients visiting the emergency room and those admitted to Catholic University of Korea St. Vincent’s Hospital in Suwon, Korea, between 2010 and 2019. All patients were at least 19 years old. We collected anonymized laboratory test datasets, including blood and urine test results, along with each patient’s final diagnosis on discharge. This study was approved by the institutional review board of St. Vincent’s Hospital, the Catholic University of Korea, which waived the need of informed consent for this study. All methods and datasets were carried out in accordance with relevant guideline and regulations by the institutional review board. We included data from each patient’s first admission to eliminate any bias caused by previous medications and treatments. We curated the datasets and selected 86 attributes (different laboratory tests) based on value counts, clinical importance-related features, and missing values. We confirmed a total of 88 attributes, including sex and age. Finally, we collected sample datasets of 5145 cases, including 326,686 (73.83%, 326,686/442,470) laboratory test results. For DL, missing values were replaced with the median value for each disease. Finally, 88 selected parameters were used for analysis of the results using DL and ML. In addition, we selected 390 cases covering a total of 39 different classifications (diagnosis), which were evenly divided among five physicians for analysis.

Feature extraction

Feature extraction plays a major role in the creation of ML models. We performed minimum feature extractions as shown in Supplement 1 (AST/AST normal: aspartate aminotransferase value/50, ALT/ALT normal: alanine aminotransferase value/50, T/A: total protein/albumin).

In this study, we compared the results of DL and ML using structured data and developed a new ensemble model combining DL and ML.

Model selection and training

Dl selection.

Recently, numerous attempts have been made to utilize DL for bioinformatics. DL has been shown to approximate the function of a complex structure 23 . Another study 5 reported that DL technologies, including CNN, SAE (stacked auto-encoder), DBN (deep belief network), DNN, and RNN, could be applied to various subjects such as medical images, DNA/RNA, and protein structures. The research in this study was conducted using a DNN for structured data.

MLP (multi-layer perceptron)

MLP is a class of DNN 24 , 25 . An MLP consists of at least three layers: an input layer, a hidden layer, and an output layer. Aside from the input layer, each layer uses a non-linear activation function. This method can analyze data that are not linearly separable. All features used in this study are numeric data except for the ‘sex’ feature. MLP recognizes only numerical data, so we transformed the categorical feature of ‘sex’ into a number using LabelEncoder of the scikit-learn library. MLP does not allow for null values, so we replaced null values with the median value of each feature.

Feature normalization and parallelization

Each feature had a different range. However, if the range differs among features, the model may misinterpret the feature range as a real difference, causing it to assign incorrect weights (W) to some features. Therefore, we applied a standard scale to normalize the mean and standard deviation of each feature to (0, 1) by subtracting the mean value of the feature and dividing by its standard deviation value.

Hidden layer composition

In our study, the hidden layer was comprised of two layers. We employed the Relu (rectified linear unit) activation function for each layer. We applied the dropout technique to each hidden layer, which is a simple method to prevent overfitting in neural networks 26 , 27 , 28 . Dropout is a method of learning based on random deletion of nodes. It selects and deletes hidden layer nodes during training. The deleted nodes are not forwarded to the next step. During training, whenever data was spilled, nodes were randomly selected for deletion, and all nodes remaining at the time of the test were used. During the test, each node's output was multiplied by the percentage deleted during training.

The activation function of the output layer used the Softmax function to assign a value between 0 and 1 to each class. To train the MLP, we calculated the cross-entropy loss for the difference between the target value and the predicted value. We updated the MLP using the Adam optimizer with a calculated parameter gradient. The Adam optimizer is a method of stochastic optimization 29 that was introduced as a noise optimizer and is suitable for various objective functions. The optimal model was selected based on low validation set loss. Training was stopped if validation set loss no longer improved after 10 epochs. In our experiments, we implemented this optimization using Keras for the MLP model and scikit-learn for data preprocessing and splitting of datasets.

ML: boosting model selection

Ensemble modeling is a method of creating strong learners by combining weak learners, and has been widely used for ML recently. Boosting and bagging 30 , 31 , 32 are the most common ensemble methods. Bagging creates a generalized model through bootstrapping (random sampling) of datasets followed by aggregation into different datasets. Both boosting and bagging are similar learning models based on bootstrapping. However, boosting allows data that were not identified in previous steps to be weighted and classified. Between these methods, we decided to use boosting through the XGBoost and LightGBM algorithms, which are the most popular boosting algorithms. To determine whether performance of boosting was better than that of bagging, and also whether it was vulnerable to overfitting, various hyperparameters of the two algorithms were solved and optimized.

The numbers of disease prediction papers using XGBoost with medical data have increased recently 33 , 34 , 35 , 36 . XGBoost is an algorithm that overcomes the shortcomings of GBM (gradient boosting machine). The disadvantages of GBM include long learning times and overfitting problems. The most common ways to solve these problems are through parallelization and regularization. Our dataset contained null values, which MLP replaced with the corresponding median values, but XGBoost has a procedure to process null values, so utilized that procedure. The max_depth argument in XGBoost is one factor determining the depth of the decision tree. Setting max_depth to a large number increases complexity and can lead to overfitting. This study found that max_depth was optimally set to 2.

Many previous studies have used LightGBM for analysis of medical data 37 , 38 , 39 , 40 . LightGBM is a GBM-based model that follows XGBoost. LightGBM is also the most recent algorithm to win an award from Kaggle. The difference between LightGBM and XGBoost is the method by which the tree grows. XGBoost creates a deeper level within the leaf (level-wise/depth-wise), and LightGBM generates a leaf at the same level (leaf-wise). XGBoost uses a level-centered tree-splitting method to keep the tree balanced when it is deepened. LightGBM uses a leaf-centered tree-splitting method to split leaf nodes with the maximum loss value, creating an asymmetric tree. To avoid overfitting in LightGBM, an experiment was conducted by adjusting num_leaves and min_child_samples. Through that experiment, the optimal value of num_leaves was determined to be 2 and that of min_child_samples was 30.

K-fold cross-validation

In our study, we divided a total of 5145 datasets at a ratio of 8:2 to create the training set and test set. We set the validation data ratio to 0.2 for the training set, which was evaluated using validation loss for model optimization based on the training data.

The number of cases was 5145, which is a relatively small dataset. If the size of the dataset is small, high variance can cause performance problems for the evaluation of the validation dataset. However, if the number of validation data is increased, the number of training data decreases, leading to a problem of high bias. To resolve these trade-offs, the cross-validation method proposed in Sensitivity Analysis of K-Fold Cross-Validation 41 , 42 , 43 , 44 , 45 in Prediction Error Estimation was used. We used k-fold cross validation to prevent data loss of the training set.

SHAP (Shapley Adaptive Explanations)

There are many ways to calculate feature importance. Among them, SHAP value can be seen through ‘Consistent Individualized Feature Attribution for Tree Ensembles’ 11 that have good consistency and accuracy in calculating feature importance. It is a method for the Tree Ensemble model in the experiment of the paper 11 . In our experiment, MLP did not correspond to a Tree ensemble model, so although we didn’t use the same method, we can calculate SHAP value using DeepLIFT 10 .

SHAP is an acronym for Shapley Adaptive Explanations. Relating to the Shapley value, as the name suggests. This Shapley value is a concept in game theory that indicates how much each contributor contributes to a particular outcome. SHAP values provide a strict theoretical improvement by eliminating significant consistency problems.

DeepLIFT 10 , 46 is a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. DeepLIFT compares the activation of each neuron to its ‘reference activation’ and assigns contribution scores according to the difference.

Performance measures

We used the F1 score, accuracy and ROC curve for performance measures.

Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25 , 24–29 (2019).

Article   CAS   PubMed   Google Scholar  

Wainberg, M., Merico, D., Delong, A. & Frey, B. J. Deep learning in biomedicine. Nat. Biotechnol. 36 , 829–838 (2018).

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521 , 436–444 (2015).

Article   ADS   CAS   PubMed   Google Scholar  

Zhang, Y. et al. Computational analysis and prediction of lysine malonylation sites by exploiting informative features in an integrative machine-learning framework. Brief. Bioinform. 20 , 2185–2199 (2019).

Min, S., Lee, B. & Yoon, S. Deep learning in bioinformatics. Brief. Bioinform. 18 , 851–869 (2017).

PubMed   Google Scholar  

Jurtz, V. I. et al. An introduction to deep learning on biological sequence data: Examples and solutions. Bioinformatics (Oxford, England) 33 , 3685–3690 (2017).

Article   CAS   Google Scholar  

Shin, H. C. et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35 , 1285–1298 (2016).

Article   PubMed   Google Scholar  

Bizopoulos, P. & Koutsouris, D. Deep learning in cardiology. IEEE Rev. Biomed. Eng. 12 , 168–193 (2019).

Jackups, R. Jr. Deep learning makes its way to the clinical laboratory. Clin. Chem. 63 , 1790–1791 (2017).

Shrikumar, A., Greenside, P. & Kundaje, A. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning , Vol. 70, 3145–3153 (JMLR.org, 2017).

Lundberg, S. M., Erion, G. G. & Lee, S.-I. Consistent Individualized Feature Attribution for Tree Ensembles Vol. 2019 (Springer, 2019).

Google Scholar  

Freeman, C., Kuli, D. & Basir, O. Feature-selected tree-based classification. IEEE Trans. Cybern. 43 , 1990–2004 (2013).

Parikh, S. A. et al. Decision tree based classification of abdominal aortic aneurysms using geometry quantification measures. Ann. Biomed. Eng. 46 , 2135–2147 (2018).

Article   PubMed   PubMed Central   Google Scholar  

Rhodes, N. J. et al. Tree-based models for predicting mortality in Gram-negative bacteremia: Avoid putting the CART before the horse. Antimicrob. Agents Chemother. 60 , 838–844 (2016).

Article   CAS   PubMed   PubMed Central   Google Scholar  

DeGregory, K. W. et al. A review of machine learning in obesity. Obes. Rev. 19 , 668–685 (2018).

Díaz-Uriarte, R. & Alvarez de Andrés, S. Gene selection and classification of microarray data using random forest. BMC Bioinform. 7 , 3 (2006).

Roguet, A., Eren, A. M., Newton, R. J. & McLellan, S. L. Fecal source identification using random forest. Microbiome 6 , 185 (2018).

Sarica, A., Cerasa, A. & Quattrone, A. Random forest algorithm for the classification of neuroimaging data in Alzheimer’s disease: A systematic review. Front. Aging Neurosci. 9 , 329 (2017).

Romagnoni, A., Jégou, S., Van Steen, K., Wainrib, G. & Hugot, J. P. Comparative performances of machine learning methods for classifying Crohn Disease patients using genome-wide genotyping data. Sci. Rep. 9 , 10351 (2019).

Article   ADS   PubMed   PubMed Central   CAS   Google Scholar  

Chang, M. L. & Liaw, Y. F. Hepatitis B flares in chronic hepatitis B: Pathogenesis, natural course, and management. J. Hepatol. 61 , 1407–1417 (2014).

Harvala, H., Wong, V., Simmonds, P., Johannessen, I. & Ramalingam, S. Acute viral hepatitis—Should the current screening strategy be modified?. J. Clin. Virol. 59 , 184–187 (2014).

Simpalipan, P., Pattaradilokrat, S. & Harnyuttanakorn, P. Global sequence diversity of the lactate dehydrogenase gene in Plasmodium falciparum . Malar. J. 17 , 16 (2018).

Article   PubMed   PubMed Central   CAS   Google Scholar  

Castro, W., Oblitas, J., Santa-Cruz, R. & Avila-George, H. Multilayer perceptron architecture optimization using parallel computing techniques. PLoS ONE 12 , e0189369 (2017).

Kwon, K., Kim, D. & Park, H. A parallel MR imaging method using multilayer perceptron. Med. Phys. 44 , 6209–6224 (2017).

Tang, J., Deng, C. & Huang, G. B. Extreme learning machine for multilayer perceptron. IEEE Trans. Neural Netw. Learn. Syst. 27 , 809–821 (2016).

Article   MathSciNet   PubMed   Google Scholar  

Abdelhafiz, D., Yang, C., Ammar, R. & Nabavi, S. Deep convolutional neural networks for mammography: Advances, challenges and applications. BMC Bioinform. 20 , 281 (2019).

Article   Google Scholar  

Khan, S. H., Hayat, M. & Porikli, F. Regularization of deep neural networks with spectral dropout. Neural Netw. 110 , 82–90 (2019).

Poernomo, A. & Kang, D. K. Biased dropout and crossmap dropout: Learning towards effective dropout regularization in convolutional neural network. Neural Netw. 104 , 60–67 (2018).

Hamm, C. A. et al. Deep learning for liver tumor diagnosis part I: Development of a convolutional neural network classifier for multi-phasic MRI. Eur. Radiol. 29 , 3338–3347 (2019).

Cannas, M. & Arpino, B. A comparison of machine learning algorithms and covariate balance measures for propensity score matching and weighting. Biom. J. 61 , 1049–1072 (2019).

MathSciNet   PubMed   MATH   Google Scholar  

Curtis, C., Liu, C., Bollerman, T. J. & Pianykh, O. S. Machine learning for predicting patient wait times and appointment delays. J. Am. Coll. Radiol. 15 , 1310–1316 (2018).

Yaman, E. & Subasi, A. Comparison of bagging and boosting ensemble machine learning methods for automated EMG signal classification. Biomed. Res. Int. 2019 , 9152506 (2019).

Li, W., Yin, Y., Quan, X. & Zhang, H. Gene Expression value prediction based on XGBoost algorithm. Front. Genet. 10 , 1077 (2019).

Ogunleye, A. A. & Qing-Guo, W. XGBoost model for chronic kidney disease diagnosis. IEEE/ACM Trans. Comput. Biol. Bioinform. (2019).

Wang, J. & Gribskov, M. IRESpy: An XGBoost model for prediction of internal ribosome entry sites. BMC Bioinform. 20 , 409 (2019).

Zhong, J. et al. XGBFEMF: An XGBoost-based framework for essential protein prediction. IEEE Trans. Nanobioscience 17 , 243–250 (2018).

Deng, L. et al. PDRLGB: Precise DNA-binding residue prediction using a light gradient boosting machine. BMC Bioinform. 19 , 522 (2018).

Su, R., Wu, H., Liu, X. & Wei, L. Predicting drug-induced hepatotoxicity based on biological feature maps and diverse classification strategies. Brief. Bioinform. 21 , 428–437 (2020).

Zhan, Z. H., You, Z. H., Li, L. P., Zhou, Y. & Yi, H. C. Accurate prediction of ncRNA–protein interactions from the integration of sequence and evolutionary information. Front. Genet. 9 , 458 (2018).

Zhang, J., Mucs, D., Norinder, U. & Svensson, F. LightGBM: An effective and scalable algorithm for prediction of chemical toxicity-application to the Tox21 and mutagenicity data sets. J. Chem. Inf. Model. 59 , 4150–4158 (2019).

Hellton, K. H. & Hjort, N. L. Fridge: Focused fine-tuning of ridge regression for personalized predictions. Stat. Med. 37 , 1290–1303 (2018).

Panda, B., Majhi, B. & Thakur, A. An integrated-OFFT model for the prediction of protein secondary structure class. Curr. Comput. Aid. Drug Des. 15 , 45–54 (2019).

Poldrack, R. A., Huckins, G. & Varoquaux, G. Establishment of best practices for evidence for prediction: A review. JAMA Psychiat. https://doi.org/10.1001/jamapsychiatry.2019.3671 (2019).

Watson, G. L., Telesca, D., Reid, C. E., Pfister, G. G. & Jerrett, M. Machine learning models accurately predict ozone exposure during wildfire events. Environ. Pollut. 254 , 112792 (2019).

Yalamanchi, K. K. et al. Machine learning to predict standard enthalpy of formation of hydrocarbons. J. Phys. Chem. A 123 , 8305–8313 (2019).

Lundberg, S. M. & Lee, S.-I. A Unified Approach to Interpreting Model Predictions , Vol. 2019 (2017).

Download references

Acknowledgements

This work was supported by the Korean Society of Medical Informatics.

This work was financially supported by research project grants for the Korean Society of Medical Informatics in 2019.

Author information

Authors and affiliations.

Department of Laboratory Medicine, College of Medicine, Ewha Womans University of Korea, Seoul, South Korea

Dong Jin Park

Department of Laboratory Medicine, St. Vincent’s Hospital, The Catholic University of Korea, Seoul, South Korea

Min Woo Park

Department of Research, Future Lab, Seoul, South Korea

Finance, Fishery, Manufacture Industrial Mathematics Center on Big Data, Pusan National University, Pusan, South Korea

Young-Jin Kim

Department of Laboratory Medicine, College of Medicine, The Catholic University of Korea, Seoul, South Korea

Yeongsic Kim

Division of Hematology, Department of Internal Medicine, College of Medicine, The Catholic University of Korea, Seoul, South Korea

Young Hoon Park

You can also search for this author in PubMed   Google Scholar

Contributions

D.J.P., M.W.P., Y.J.K., and Y.H.P. designed the experiments, analyzed the data. D.J.P. and Y.H.P wrote the manuscript. D.J.P., Y.H.P., M.W.P., and Y.K. conducted data curation. D.J.P., Y.H.P., and H.L. considered the deep learning algorithm and analyzed numerical results.

Corresponding author

Correspondence to Young Hoon Park .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary figure 1a., supplementary figure 1b., supplementary figure 1c., supplementary table 1., supplementary table 2., supplementary table 3., supplementary table 4., supplementary table 5., supplementary table 6., supplementary table 7., supplementary table 8., supplementary table 9., supplementary table 10., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Park, D.J., Park, M.W., Lee, H. et al. Development of machine learning model for diagnostic disease prediction based on laboratory tests. Sci Rep 11 , 7567 (2021). https://doi.org/10.1038/s41598-021-87171-5

Download citation

Received : 31 August 2020

Accepted : 19 March 2021

Published : 07 April 2021

DOI : https://doi.org/10.1038/s41598-021-87171-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.