Hanyang Med Rev.  2017 Nov;37(2):61-70. 10.7599/hmr.2017.37.2.61.

Deep Learning for Medical Image Analysis: Applications to Computed Tomography and Magnetic Resonance Imaging

• Jung KH
• Park H
• Hwang W

• KMID: 2424830
• DOI: http://doi.org/10.7599/hmr.2017.37.2.61

Abstract

Recent advances in deep learning have brought many breakthroughs in medical image analysis by providing more robust and consistent tools for the detection, classification and quantification of patterns in medical images. Specifically, analysis of advanced modalities such as computed tomography (CT) and magnetic resonance imaging (MRI) has benefited most from the data-driven nature of deep learning. This is because the need of knowledge and experience-oriented feature engineering process can be circumvented by automatically deriving representative features from the complex high dimensional medical images with respect to the target tasks. In this paper, we will review recent applications of deep learning in the analysis of CT and MR images in a range of tasks and target organs. While most applications are focused on the enhancement of the productivity and accuracy of current diagnostic analysis, we will also introduce some promising applications which will significantly change the current workflow of medical imaging. We will conclude by discussing opportunities and challenges of applying deep learning to advanced imaging and suggest future directions in this domain.

Keyword

Deep Learning; Medical Imaging; Computed Tomography; Magnetic Resonance Imaging

MeSH Terms

Classification
Diagnostic Imaging
Efficiency
Learning*
Magnetic Resonance Imaging*

Figure

• 

  Fig. 1A Visualization of ‘evidence hotspot’ in spinal MRI [10]. Adapted with permission

• 

  Fig. 1B HIV patient and normal control HIV patient and healthy control

• 

  Fig. 2 Multi-stream, multi-scale CNN architecture for pulmonary nodule classification [16]. Adapted with permission.

• 

  Fig. 3 Segmentation of aggressive prostate cancer lesion using GAN [29]. Adapted with permission.

• 

  Fig. 4 Conversion of MRI to CT using GAN [38]. Adapted with permission

• 

  Fig. 5 Example of a model adopting deep learning for survival analysis [40]. Adapted with permission.

Reference

1. Litjens G, Kooi T, Bejnordi BE, et al. A Survey on Deep Learning in Medical Image Analysis. arXiv [Internet]. 2017. 1702.05747:1-34. Available from: http://arxiv.org/abs/1702.05747/.

2. Lot SB, Lint JJ, Freedmant MT, Munt SK. Computer-Assisted Diagnosis of Lung Nodule Detection Using Artificial Convolution Neural Network. Med Imaging. 1993; 1898:859–869.

3. Chan HP, Lo SC, Sahiner B, Lam KL, Helvie MA. Computer-aided detection of mammographic microcalcifications: pattern recognition with an artificial neural network. Med Phys. 1995; 22:1555–1567.

4. Imagenet Challenge. Available from: http://image-net.org/challenges/LSVRC/2012/results.html.

5. Ciresan DC, Giusti A, Gambardella LM, Schmidhuber J. Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks. Med Image Comput Comput Assist Interv. 2013; 16:411–418.

6. Gulshan V, Peng L, Coram M, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016; 316:2402–2410.

7. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542:115–118.

8. Hosseini-Asl E, Keynton R, El-Baz A. Alzheimer's disease diagnostics by adaptation of 3D convolutional network. ICIP. 2016; 126–130.

9. Gao M, Xu Z, Lu L, Harrison AP, Summers RM, Mollura DJ. Holistic Interstitial Lung Disease Detection using Deep Convolutional Neural Networks: Multi-label Learning and Unordered Pooling. arXiv [Internet]. 2017. 9352:1-9. Available from: http://arxiv.org/abs/1701.05616/.

10. Jamaludin A, Kadir T, Zisserman A. SpineNet: Automatically pinpointing classification evidence in spinal MRIs. Lect Notes Comput Sci. 2016; 9901:166–175.

11. Zintgraf L, Cohen T, Adel T, Welling M. Visualizing Deep Neural Network Decisions: Prediction Difference Analysis. arXiv [Internet]. 2016. 1511.06488:1-12. Available from: http://arxiv.org/abs/1511.06488/.

12. Zeiler MD, Fergus R. Visualizing and Understanding Convolutional Networks. Comput Vis ECCV. 2014; 8689:818–833.

13. Cheng JZ, Ni D, Chou YH, et al. Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans. Sci Rep. 2016; 6:24454.

14. Shin HC, Roth HR, Gao M, et al. Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning. IEEE Trans Med Imaging. 2016; 35:1285–1298.

15. Dou Q, Chen H, Yu L, Qin J, Heng PA. Multi-level Contextual 3D CNNs for False Positive Reduction in Pulmonary Nodule Detection. IEEE Trans Biomed Eng. 2017; 64:1558–1567.

16. Ciompi F, Chung K, van Riel SJ, et al. Towards automatic pulmonary nodule management in lung cancer screening with deep learning. Sci Rep. 2017; 7:46479.

17. Ghafoorian M, Karssemeijer N, Heskes T, et al. Deep multi-scale location-aware 3D convolutional neural networks for automated detection of lacunes of presumed vascular origin. NeuroImage Clin. 2017; 14:391–399.

18. Guo Y, Wu G, Commander LA, et al. Segmenting hippocampus from infant brains by sparse patch matching with deep-learned features. Med Image Comput Comput Assist Interv. 2014; 17:308–315.

19. Lo P, Van Ginneken B, Reinhardt JM, et al. Extraction of airways from CT (EXACT’09). IEEE Trans Med Imaging. 2012; 31:2093–2107.

20. Charbonnier JP, Rikxoort EMV, Setio AAA, Schaefer-Prokop CM, Ginneken BV, Ciompi F. Improving airway segmentation in computed tomography using leak detection with convolutional networks. Med Image Anal. 2017; 36:52–60.

21. Hu P, Wu F, Peng J, Liang P, Kong D. Automatic 3D liver segmentation based on deep learning and globally optimized surface evolution. Phys Med Biol. 2016; 61(24):8676–8698.

22. Thong W, Kadoury S, Piché N, Pal CJ. Convolutional networks for kidney segmentation in contrast-enhanced CT scans. Comput Methods Biomech Biomed Eng Imaging Vis. 2016; 1163:1–6.

23. Roth HR, Lu L, Farag A, Sohn A, Summers RM. Spatial aggregation of holistically-nested networks for automated pancreas segmentation. Lect Notes Comput Sci. 2016; 9901:451–459.

24. Yu L, Yang X, Chen H, Qin J, Heng P-A. Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images. In : Thirty-First AAAI Conference on Artificial Intelligence; 2017. p. 66–72.

25. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI. 2015; 234–241.

26. Poudel RPK, Lamata P, Montana G. Recurrent fully convolutional neural networks for multi-slice MRI cardiac segmentation. Lect Notes Comput Sci. 2017; 10129:83–94.

27. Brosch T, Tang LYW, Yoo Y, Li DKB, Traboulsee A, Tam R. Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation. IEEE Trans Med Imaging. 2016; 35(5):1229–1239.

28. Ghafoorian M, Karssemeijer N, Heskes T, et al. Location Sensitive Deep Convolutional Neural Networks for Segmentation of White Matter Hyperintensities. Sci Rep. 2017; 7:5110.

29. Kohl S, Bonekamp D, Schlemmer H-P, Yaqubi K, Hohenfellner M, Hadaschik B, et al. Adversarial Networks for the Detection of Aggressive Prostate Cancer. arXiv [Internet]. 2017. 1702.08014:1-12. Available from: http://arxiv.org/abs/1702.08014/.

30. Cheng X, Zhang L, Zheng Y. Deep similarity learning for multimodal medical images. Comput Methods Biomech Biomed Eng Imaging Vis. 2016; 1163:1–5.

31. Simonovsky M, Gutiérrez-Becker B, Mateus D, Navab N, Komodakis N. A Deep Metric for Multimodal Registration. arXiv [Internet]. 2016. 1609.05396:1-10. Available from: http://arxiv.org/abs/1609.05396/.

32. Miao S, Wang ZJ, Liao R. A CNN Regression Approach for Real-Time 2D/3D Registration. IEEE Trans Med Imaging. 2016; 35(5):1352–1363.

33. Yang X, Kwitt R, Styner M, Niethammer M. Fast Predictive Image Registration. arXiv [Internet]. 2017. 1703.10902:1-10. Available from: http://arxiv.org/abs/1703.10902/.

34. Chen H, Zhang Y, Kalra MK, et al. Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN). IEEE Trans Med Imaging. 2017; 36:2524–2535.

35. Bahrami K, Shi F, Rekik I, Shen D. Convolutional Neural Network for Reconstruction of 7T-like Images from 3T MRI Using Appearance and Anatomical Features. In : MICCAI 2016 DL workshop; 2016. p. 39–47.

36. Oktay O, Bai W, Lee M, Guerrero R, Kamnitsas K, Caballero J, et al. Multiinput cardiac image super-resolution using convolutional neural networks. Lect Notes Comput Sci. 2016; 9902:246–254.

37. Li R, Zhang W, Suk H-I, et al. Deep learning based imaging data completion for improved brain disease diagnosis. Med Image Comput Comput Assist Interv. 2014; 17:305–312.

38. Nie D, Trullo R, Petitjean C, Ruan S, Shen D. Medical Image Synthesis with Context-Aware Generative Adversarial Networks. arXiv [Internet]. 2016. 1612.05362:1-11. Available from: http://arxiv.org/abs/1612.05362/.

39. Nie D, Zhang H, Adeli E, Liu L, Shen D. 3D deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients. Med Image Comput Comput Assist Interv. Med Image Comput Comput Assist Interv. 2016; 9901:212–220.

40. Van der Burgh HK, Schmidt R, Westeneng HJ, de Reus MA, de Reus MA, van den Berg LH. Deep learning predictions of survival based on MRI in amyotrophic lateral sclerosis. NeuroImage Clin. 2017; 13:361–369.

41. Yoo Y, Tang LW, Brosch T, Li DKB, Metz L, Traboulsee A, Tam R. Deep learning of brain lesion patterns for predicting future disease activity in patients with early symptoms of multiple sclerosis. Deep Learn Data Label Med Appl (2016). Lect Notes Comput Sci. 2016; 10008:86–94.

42. Oakden-Rayner L, Carneiro G, Bessen T, Nascimento JC, Bradley AP, Palmer LJ. Precision Radiology: Predicting longevity using feature engineering and deep learning methods in a radiomics framework. Sci Rep. 2017; 7:1648.

43. Leibig C, Allken V, Berens P, Wahl S. Leveraging uncertainty information from deep neural networks for disease detection. bioRxiv. 2016; 084210:1–21. Available from: http://www.biorxiv.org/content/early/2016/10/28/084210/.

44. Burak Akgül C, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B. Content-based image retrieval in radiology: Current status and future directions. J Digit Imaging. 2011; 24:208–222.

45. Shokri R, Shmatikov V. Privacy-preserving deep learning. In : 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton; 2015. p. 909–910.

46. Papernot N, Abadi MM, Erlingsson Ú, Goodfellow I, Talwar K. Semisupervised knowledge transfer for deep learning from private training data. arXiv [Internet]. 2016. 1610.05755:1-16. Available from: http://arxiv.org/abs/1610.05755/.