Healthc Inform Res.  2012 Mar;18(1):3-9. 10.4258/hir.2012.18.1.3.

Medical Image Retrieval: Past and Present

• Hwang KH
• Lee H
• Choi D

• KMID: 2166612
• DOI: http://doi.org/10.4258/hir.2012.18.1.3

Abstract

With the widespread dissemination of picture archiving and communication systems (PACSs) in hospitals, the amount of imaging data is rapidly increasing. Effective image retrieval systems are required to manage these complex and large image databases. The authors reviewed the past development and the present state of medical image retrieval systems including text-based and content-based systems. In order to provide a more effective image retrieval service, the intelligent content-based retrieval systems combined with semantic systems are required.

Keyword

Medical Image; Content-Based Image Retrieval; Text-Based Image Retrieval; SNOMED-CT; RadLex; Ontology

MeSH Terms

Radiology Information Systems
Semantics

Figure

• 

  Figure 1 Classification of medical image retrieval methods.

• 

  Figure 2 Text-based query commonly retrieves irrelevant images on the internet web browser. For user query "bike (bicycle)," irrelevantly retrieved images (not relevant to bike) are shown because image annotations contain a word "bike" such as bike tour, school bike rack, water bike, abbreviation, etc. (upper two layers). Miscellaneous medical images retrieved by query of "Reconstruction and computed tomography (CT)" are displayed in the lower 3rd layer of the figure (CT for image reconstruction, 3-dimensional reconstruction CT image, CT reconstruction algorithm, bone reconstruction with CT, breast reconstruction using CT).

• 

  Figure 3 Image retrieval for medical applications (IRMA) content-based image retrieval results (http://irma-project.org/).

• 

  Figure 4 Flexible image retrieval engine (FIRE) content-based image retrieval results (http://thomas.deselaers.de/fire/).

• 

  Figure 5 CliniClue Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) web browser (http://www.cliniclue.com/).

• 

  Figure 6 Foundational model explorer browser (http://fme.biostr.washington.edu/FME/index.html/).

• 

  Figure 7 Radiology Lexicon (RadLex) term browser (http://www.radlex.org/).

• 

  Figure 8 An example of combined text and content-based medical image retrieval system - iPad, a plug-in to OsiriX, the application tool of Annotation and Image Markup Project [31].

Reference

1. Liu Y, Zhang D, Lu G, Ma WY. A survey of content-based image retrieval with high-level semantics. Pattern Recogn. 2007. 40:262–282.

2. Brahmi D, Ziou D. Improving CBIR systems by integrating semantic features. Proceedings of the 1st Canadian Conference on Computer and Robot Vision. 2004. 233–240.

3. Faloutsos C, Barber R, Flickner M, Hafner J, Niblack W, Petkovic D, Equitz W. Efficient and effective querying by image content. J Intell Inf Syst. 1994. 3:231–262.

4. Pentland A, Picard RW, Sclaroff S. Photobook: content-based manipulation of image databases. Int J Comput Vis. 1996. 18:233–254.

5. Chang SF, Smith JR, Beigi M, Benitez A. Visual information retrieval from large distributed online repositories. Commun ACM. 1997. 40:63–71.

6. Smith JR, Chang SF. VisualSEEk: a fully automated content-based image query system. Proceedings of the fourth ACM international conference on Multimedia. 1996. 87–98.

7. Ma WY, Manjunath BS. NeTra: a toolbox for navigating large image databases. Proceedings of International Conference on Image Processing. 1997. 568–571.

8. Eakins JP. Towards intelligent image retrieval. Pattern Recogn. 2002. 35:3–14.

9. Brodley C, Kak A, Shyu C, Dy J, Broderick L, Aisen AM. Content-based retrieval from medical image databases: a synergy of human interaction, machine learning and computer vision. Proceedings of the 16th National Conference on Artificial Intelligence and the 11th Innovative Applications of Artificial Intelligence Conference Innovative Applications of Artificial Intelligence. 1999. 760–767.

10. Kelly PM, Cannon M. Query by image example: the comparison algorithm for navigating digital image databases (CANDID) approach. Proceedings of the Storage and Retrieval for Image and Video Databases. 1995. 238–248.

11. Orphanoudakis SC, Chronaki C, Kostomanolakis S. I2C: a system for the indexing, storage, and retrieval of medical images by content. Med Inform (Lond). 1994. 19:109–122.

12. Shyu CR, Brodley CE, Kak AC, Kosaka A, Aisen AM, Broderick LS. ASSERT: a physician-in-the-loop content-based retrieval system for HRCT image databases. Comput Vis Image Understand. 1999. 75:111–132.

13. Keysers D, Dahmen J, Ney H, Wein BB, Lehmann TM. Statistical framework for model-based image retrieval in medical applications. J Electron Imaging. 2003. 12:59–68.

14. Deselaers T. Features for Image Retrieva [dissertation]. 2003. Aachen, Germany: Rheinisch-Westfalische Technische Hochschule Aachen.

15. Muller H, Michoux N, Bandon D, Geissbuhler A. A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. Int J Med Inform. 2004. 73:1–23.

16. Deselaers T. Fire [Internet]. c2009. cited at 2012 Mar 26. Tomas Deselaers;Available from: http://thomas.deselaers.de/fire.

17. El-Kwae EA, Xu H, Kabuka MR. Content-based retrieval in picture archiving and communication systems. J Digit Imaging. 2000. 13:70–81.

18. Qi H, Snyder WE. Content-based image retrieval in picture archiving and communications systems. J Digit Imaging. 1999. 12:81–83.

19. Stearns MQ, Price C, Spackman KA, Wang AY. SNOMED clinical terms: overview of the development process and project status. Proc AMIA Symp. 2001. 662–666.

20. Rosse C, Mejino JL Jr. A reference ontology for biomedical informatics: the foundational model of anatomy. J Biomed Inform. 2003. 36:478–500.

21. Langlotz CP. RadLex: a new method for indexing online educational materials. Radiographics. 2006. 26:1595–1597.

22. Rubin DL. Creating and curating a terminology for radiology: ontology modeling and analysis. J Digit Imaging. 2008. 21:355–362.

23. Rubin DL, Flanders A, Kim W, Siddiqui KM, Kahn CE Jr. Ontology-assisted analysis of Web queries to determine the knowledge radiologists seek. J Digit Imaging. 2011. 24:160–164.

24. Hazen R, Van Esbroeck AP, Mongkolwat P, Channin DS. Automatic extraction of concepts to extend RadLex. J Digit Imaging. 2011. 24:165–169.

25. Luo B, Wang X, Tang X. World Wide Web based image search engine using text and image content features. Proc SPIE. 2003. 5018:123–130.

26. Barrios JM, Diaz-Espinoza D, Bustos B. Text-based and content-based image retrieval on Flickr: DEMO. Proceedings of the 2nd International Workshop on Similarity Search and Applications. 2009. 156–157.

27. Deselaers T, Weyand T, Keysers D, Macherey W, Ney H. FIRE in ImageCLEF 2005: combining content-based image retrieval with textual information retrieval. Proceedings of the 6th International Conference on Cross-Language Evalution Forum: accessing Multilingual Information Repositories. 2005. 652–661.

28. Dinakaran B, Annapurna J, Kumar CA. Interactive image retrieval using text and image content. Cybern Inf Tech. 2010. 10:20–30.

29. Zhou XS, Zillner S, Moeller M, Sintek M, Zhan Y, Krishnan A, Gupta A. Semantics and CBIR: a medical imaging perspective. Proceedings of the 2008 International Conference on Content-based Image and Video Retrieval. 2008. 571–580.

30. Rubin DL, Mongkolwat P, Kleper V, Supekar K, Channin DS. Medical imaging on the semantic web: annotation and image markup. Proceedings of the Semantic Scientific Knowledge Integration. 2008.

31. Rubin DL, Rodriguez C, Shah P, Beaulieu C. iPad: semantic annotation and markup of radiological images. AMIA Annu Symp Proc. 2008. 626–630.