Artificial Intelligence-Assisted Polyp Detection for Colonoscopy: Initial Experience

The adenoma detection rate is an established quality indicator for colonoscopy. For instance, a 1% increase in the adenoma detection rate was associated with a 3% decrease in interval colorectal cancer incidence.1Corley D.A. Jensen C.D. Marks A.R. et al.Adenoma detection rate and risk of colorectal cancer and death.N Engl J Med. 2014; 370: 1298-1306Crossref PubMed Scopus (1166) Google Scholar However, a previous meta-analysis showed that approximately 26% of neoplastic diminutive polyps were missed in single colonoscopy.2van Rijn J.C. Reitsma J.B. Stoker J. et al.Polyp miss rate determined by tandem colonoscopy: a systematic review.Am J Gastroenterol. 2006; 101: 343-350Crossref PubMed Scopus (1031) Google Scholar Two factors are considered to affect this rate; one is blind spots and the other is human error. The first factor could be solved using a wide-angle scope or distal attachments, but human error is not easily overcome. As a solution to address human error, artificial intelligence has been attracting attention.3Byrne M.F. Shahidi N. Rex D.K. Will Computer-Aided Detection and Diagnosis Revolutionize Colonoscopy?.Gastroenterology. 2017; 153: 1460-1464.e1Abstract Full Text Full Text PDF PubMed Scopus (45) Google Scholar, 4Misawa M. Kudo S.E. Mori Y. et al.Characterization of colorectal lesions using a computer-aided diagnostic system for narrow-band imaging endocytoscopy.Gastroenterology. 2016; 150: 1531-1532.e3Abstract Full Text Full Text PDF PubMed Scopus (113) Google Scholar, 5Chen P.J. Lin M.C. Lai M.J. et al.Accurate classification of diminutive colorectal polyps using computer-aided analysis.Gastroenterology. 2018; 154: 568-575Abstract Full Text Full Text PDF PubMed Scopus (227) Google Scholar Karkanis et al6Karkanis S.A. Iakovidis D.K. Maroulis D.E. et al.Computer-aided tumor detection in endoscopic video using color wavelet features.IEEE Trans Inf Technol Biomed. 2003; 7: 141-152Crossref PubMed Scopus (371) Google Scholar first reported using computer-aided detection (CADe) systems for colorectal polyps and achieved a >90% detection rate.6Karkanis S.A. Iakovidis D.K. Maroulis D.E. et al.Computer-aided tumor detection in endoscopic video using color wavelet features.IEEE Trans Inf Technol Biomed. 2003; 7: 141-152Crossref PubMed Scopus (371) Google Scholar However, the study results could not be applied clinically because the system was based on static images. Recently, Fernández-Esparrach et al7Fernández-Esparrach G. Bernal J. López-Cerón M. et al.Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps.Endoscopy. 2016; 48: 837-842Crossref PubMed Scopus (83) Google Scholar reported using a CADe system based on routine colonoscopy videos. Although their system could localize the polyps, sensitivity was only approximately 70% because of the limited number of study samples. To tackle these issues, we used an algorithm designed to analyze videos, and we secured a large number of routine colonoscopy videos. Subsequently, we conducted a pilot study to evaluate the performance of the developed CADe system. In this study, we developed an original artificial intelligence-assisted CADe system. Figure 1 shows an output sample from the system; the full algorithm appears in the Supplementary Document. To develop the CADe, we retrospectively collected colonoscopy videos from study participants who underwent colonoscopy from April 2015 to October 2015 in our institution. Recording for each colonoscopy video ran from cecal intubation to withdrawal of the scope across the anus. The inclusion criterion was patients with colorectal polyp(s) and the exclusion criteria were (1) advanced colorectal cancer, (2) inflammatory bowel disease, and (3) non-epithelial lesions. We collected 73 colonoscopy videos (total duration, 997 minutes; 1.8 million frames) from 73 patients, which included 155 colorectal polyps. Two expert endoscopists retrospectively annotated the presence of polyps in each frame of each video, and this annotation was treated as the gold standard for the presence of polyps. To use these videos for machine leaning and its evaluation, we divided these full-length videos into short videos based on the method described in the Supplementary Document, which produced 155 polyp-positive short videos and 391 polyp-negative videos. These 546 short videos were randomly divided into 2 groups: learning samples (105 positive and 306 negative) and test samples (50 positive and 85 negative). Learning samples were used for machine learning, and test samples were used to evaluate our system’s performance. Thus, test samples were a completely separate dataset that was never used during machine learning. The CADe system presented the probability of the presence of polyps as a percentage (0%–100%) in each frame. This probability value simulated the confidence level of human endoscopists on a given image frame and was not related to adenoma detection rate. To decide the cutoff value for this probability and to develop the CADe with a sensitivity of ≥90%, we performed receiver-operating characteristic analysis. To evaluate the CADe performance, we then calculated the system’s sensitivity, specificity, and accuracy on an image-frame basis. When the probability exceeded the cutoff value, the system considered this a positive detection. Sensitivity was calculated by dividing the number of frames correctly detected by the system by the total number of polyp frames in the test sample. Specificity was calculated by dividing the number of negative frames correctly diagnosed by the system by the total number of negative frames in the test samples. We also performed a polyp-based analysis where we defined polyp detection by the CADe as the system output over the cutoff value for >75% of the duration of each short video. We also calculated the sensitivity and false positive ratio for the polyp-based analysis. Ethics statements are included in the Supplementary Document. Video 1 is a sample with the CADe providing the probability in the upper left of the endoscopy image. When the probability exceeded the cutoff, the CADe warned of the possibility of the presence of polyps by changing the color in the 4 corners of the endoscopic image to red. When a lesion appeared on the screen, a warning was issued, and the warning stopped as the lesion disappeared from the screen. The clinicopathologic features of the study participants and the polyps are shown in Supplementary Table 1. In this study, flat lesions, which are considered difficult for CADe to find, were included at a rate of 64.5% (100 of 155). Figure 2 shows the CADe receiver-operating characteristic curve. Based on the receiver-operating characteristic analysis, we set the cutoff value for the probability of detecting a polyp at 15%. The sensitivity, specificity, and accuracy for the frame-based analysis, were 90.0%, 63.3%, and 76.5%, respectively. Regarding the polyp-based analysis, the CADe detected 94% of the test polyps (47 of 50), and the false-positive detection was 60% (51 of 85). This sensitivity and false-positive rate were similar to results of a recent deep learning-based CADe system for gastric cancer.8Hirasawa T. Aoyama K. Tanimoto T. et al.Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.Gastric Cancer. 2018 Jan 15; ([Epub ahead of print])Crossref Scopus (390) Google Scholar Supplementary Table 2 shows the result of each test lesions. Our proposed CADe system showed that artificial intelligence has the potential to provide automated detection of colorectal polyps. Further machine learning and prospective evaluation are mandatory; however, such CADe systems are expected to fill the gap between endoscopists with different levels of experience. The authors express their gratitude to Mr Takashi Wakisaka and Mr Hideo Kahara (Cybernet Systems). In this study, we developed an original algorithm based on a convolutional 3-dimensional neural network, which is a type of deep learning. Deep learning is a recent machine learning method using a deep neural network that automatically extracts specific features from data without human power if very high numbers of learning samples are available. Convolutional 3-dimensional network is designed for spatiotemporal data; therefore, it is more suitable for video datasets compared with previous deep learning methods.1Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, 2015.Google Scholar First, because this study included 155 polyps, we extracted all polyp-positive frame sequences with white-light imaging among the 73 colonoscopy videos. We then divided the polyp-positive frames into 155 short videos (total duration of polyp positive frames: 35 minutes; 63,135 frames) based on the individual polyps. We excluded frames with narrow-band imaging and chromoendoscopy because these methods were usually used after detecting the polyps. Next, we randomly extracted polyp-negative frame sequences of random lengths. By learning all negative frames, the specificity of the system increases, but the sensitivity decreases; therefore, because CADe systems require high sensitivity, we decided not to use all of the negative frames. Regarding this random extraction, we set the number of negative frames so that the ratio of the number of positive frames to negative frames was approximately 1:2. Thus, the total number of extracted negative frames was 133,496 frames (80 minutes), which also created 391 negative short videos. We chose a 1:2 ratio because it provided high sensitivity with sufficient specificity for this initial experiment. The 155 positive short videos and 391 negative short videos were used for the final analysis. After discussion with our institutional ethics committee, it was decided that informed consent would be obtained on an “opt-out” basis in the present trial because the intervention was considered to be minimally serious (No. 1411-02), and no patients refused to participate in the study. All procedures performed in this study were in accordance with the Declaration of Helsinki. Information about the current trial and the method of informing staff on presenting the option for refusal to participate appears on Showa University's home page (www.showa-u.ac.jp/pick_up/chiken/frdi8b000000mopb-att/17H056.pdf; posted until March 2020).Supplementary Table 1Clinicopathologic Features of the Study Participants and the PolypsLearning SamplesTest SamplesPPatients, n5914-Male/Female34/2510/4.38aFisher’s exact test was used.Mean age ± SD (y)61.3 ± 11.160.8 ± 14.8.88bStudent t test was used.Lesions, n10550-Morphology (protruded/flat and depressed)38/6716/34.72aFisher’s exact test was used.Mean size ± SD (mm)4.8 ± 3.04.9 ± 5.0.45cMann-Whitney U testPathologic diagnosis.50aFisher’s exact test was used. High-grade adenoma21 Low-grade adenoma7139 Hyperplastic polyp288 Sessile serrated adenoma/polyp20 Other non-neoplasmsdOther non-neoplasms included inflammatory polyps and juvenile polyps.32Location.45aFisher’s exact test was used. Right/left/rectum47/45/1326/16/8SD, standard deviation.a Fisher’s exact test was used.b Student t test was used.c Mann-Whitney U testd Other non-neoplasms included inflammatory polyps and juvenile polyps. Open table in a new tab Supplementary Table 2Results of the Test LesionsCase No.MorphologySize (mm)LocationPathologic diagnosisDetected ratio (%)aPercentage of the length of each short video which the system output over the cutoff value.1Ip6SJuvenile polyp1002IIa2ALow-grade adenoma1003IIa3ALow-grade adenoma1004IIa3SLow-grade adenoma975IIa3THyperplastic polyp896IIa2DLow-grade adenoma1007Ip10RLow-grade adenoma918IIa4SLow-grade adenoma979Isp8ALow-grade adenoma9410Isp6ALow-grade adenoma10011Is5RLow-grade adenoma10012Is10RSLow-grade adenoma10013IIa2RLow-grade adenoma10014IIa4THyperplastic polyp8815IIa3ALow-grade adenoma10016IIa2ALow-grade adenoma10017IIa2ALow-grade adenoma9718Is4ALow-grade adenoma8819IIa4ALow-grade adenoma10020IIa3SLow-grade adenoma10021IIa3TLow-grade adenoma10022IIa4SHyperplastic polyp10023Is6SLow-grade adenoma10024IIa3ALow-grade adenoma10025IIa2ALow-grade adenoma10026IIa2ALow-grade adenoma9827IIa3ALow-grade adenoma10028IIa3SLow-grade adenoma10029IIa4ALow-grade adenoma10030IIa2SHyperplastic polyp9531IIa3TLow-grade adenoma9532Is12RSLow-grade adenoma10033IIa35RLow-grade adenoma9834IIa2SHyperplastic polyp10035Is12RHigh-grade adenoma10036Is4ALow-grade adenoma9537Is3RInflammatory polyp10038Isp5SLow-grade adenoma9839IIa3ALow-grade adenoma8240IIa2SHyperplastic polyp6041IIa3SHyperplastic polyp10042IIa4CHyperplastic polyp10043Isp8ALow-grade adenoma9744Is5SLow-grade adenoma10045IIa8SLow-grade adenoma9546IIa7CLow-grade adenoma5047IIa3ALow-grade adenoma10048IIa3TLow-grade adenoma6749IIa5DLow-grade adenoma10050Is2TLow-grade adenoma80A, ascending colon; C, cecum; D, descending colon; Ip, pedunculated; IIa, flat elevated; Isp, subpedunculated; Is, sessile; R, rectum; RS, rectosigmoid colon; S, sigmoid colon; T, transverse colon.a Percentage of the length of each short video which the system output over the cutoff value. Open table in a new tab SD, standard deviation. A, ascending colon; C, cecum; D, descending colon; Ip, pedunculated; IIa, flat elevated; Isp, subpedunculated; Is, sessile; R, rectum; RS, rectosigmoid colon; S, sigmoid colon; T, transverse colon. eyJraWQiOiI4ZjUxYWNhY2IzYjhiNjNlNzFlYmIzYWFmYTU5NmZmYyIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJhODhjNWE4NDIyMjdiMGFmMDE4ZGExNmI4YjBjNDc0MiIsImtpZCI6IjhmNTFhY2FjYjNiOGI2M2U3MWViYjNhYWZhNTk2ZmZjIiwiZXhwIjoxNjc4MTkyMzkwfQ.EaXgtHBU1HK5yYllhG_wbwK1zClIbgO5-MYxehkFFvvbAOX7xGXMl4YYjHTa66Yr3yhLxsHRmn7a3RVROFw_IhlK9kDq6CpmW11SHs1S5aEHp58Cm21R7F6YiT1lKiJcvbAUANlnqUYr8oz4HH9mK3ggOV3-wV1wVrdYwNrwL-I348usKbha0GCp1EWS3gNfkOJKZDRRYBMcS4G_EYSK7qNBu1DnF8l-Bs7ICh7eJIg9sm9gX2NeXW0qrAbjjd6G1bVcRFSWou09wd8bdFga3eKiaPVO85HnR6EI8J2df8c6bH5PwjBY8PeRpkPKp97NHZY0lqlfJOkXNht1BmJ8Fw Download .mp4 (81 MB) Help with .mp4 files Video 1