Healthc Inform Res.  2018 Oct;24(4):394-401. 10.4258/hir.2018.24.4.394.

Augmented Reality to Localize Individual Organ in Surgical Procedure

• Lee D
• Yi JW
• Hong J
• Chai YJ
• Kim HC
• Kong HJ

• KMID: 2434556
• DOI: http://doi.org/10.4258/hir.2018.24.4.394

Abstract

OBJECTIVES
Augmented reality (AR) technology has become rapidly available and is suitable for various medical applications since it can provide effective visualization of intricate anatomical structures inside the human body. This paper describes the procedure to develop an AR app with Unity3D and Vuforia software development kit and publish it to a smartphone for the localization of critical tissues or organs that cannot be seen easily by the naked eye during surgery.
METHODS
In this study, Vuforia version 6.5 integrated with the Unity Editor was installed on a desktop computer and configured to develop the Android AR app for the visualization of internal organs. Three-dimensional segmented human organs were extracted from a computerized tomography file using Seg3D software, and overlaid on a target body surface through the developed app with an artificial marker.
RESULTS
To aid beginners in using the AR technology for medical applications, a 3D model of the thyroid and surrounding structures was created from a thyroid cancer patient's DICOM file, and was visualized on the neck of a medical training mannequin through the developed AR app. The individual organs, including the thyroid, trachea, carotid artery, jugular vein, and esophagus were localized by the surgeon's Android smartphone.
CONCLUSIONS
Vuforia software can help even researchers, students, or surgeons who do not possess computer vision expertise to easily develop an AR app in a user-friendly manner and use it to visualize and localize critical internal organs without incision. It could allow AR technology to be extensively utilized for various medical applications.

Keyword

Virtual Reality; Medical Education; Three-Dimensional Imaging; Thyroidectomy; Augmented Reality

MeSH Terms

Carotid Arteries
Education, Medical
Esophagus
Human Body
Humans
Imaging, Three-Dimensional
Jugular Veins
Manikins
Methyltestosterone
Neck
Smartphone
Surgeons
Thyroid Gland
Thyroid Neoplasms
Thyroidectomy
Trachea
Methyltestosterone

Figure

• 

  Figure 1 Installation procedures of Unity and Vuforia.

• 

  Figure 2 Usage of License Manager to get development key, which is used in Unity environment for Vuforia configuration.

• 

  Figure 3 Usage of Target Manager to add target database. There are four target categories that can be selected: Single Image, Cuboid, Cylinder, and 3D Object.

• 

  Figure 4 Interface of Unity, consisting of Inspector, Scene, Project, and Hierarchy.

• 

  Figure 5 Basic settings.

• 

  Figure 6 Registering the license key.

• 

  Figure 7 Loading and activating the target datasets.

• 

  Figure 8 Build settings for PC and Android version.

• 

  Figure 9 Example of application of augmented reality technology: thyroid and internal organs on a robotic thyroidectomy training model through a smartphone display.

Reference

1. Chang H, Choi M. Big data and healthcare: building an augmented world. Healthc Inform Res. 2016; 22(3):153–155.

2. Bernhardt S, Nicolau SA, Soler L, Doignon C. The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal. 2017; 37:66–90.

3. Khor WS, Baker B, Amin K, Chan A, Patel K, Wong J. Augmented and virtual reality in surgery-the digital surgical environment: applications, limitations and legal pitfalls. Ann Transl Med. 2016; 4(23):454.

4. Kim Y, Kim H, Kim YO. Virtual reality and augmented reality in plastic surgery: a review. Arch Plast Surg. 2017; 44(3):179–187.

5. Tang R, Ma L, Xiang C, Wang X, Li A, Liao H, et al. Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: a case report. Medicine (Baltimore). 2017; 96(37):e8083.

6. Soler L, Nicolau S, Pessaux P, Mutter D, Marescaux J. Real-time 3D image reconstruction guidance in liver resection surgery. Hepatobiliary Surg Nutr. 2014; 3(2):73–81.

7. Barsom EZ, Graafland M, Schijven MP. Systematic review on the effectiveness of augmented reality applications in medical training. Surg Endosc. 2016; 30(10):4174–4183.

8. Hassan K, Dort JC, Sutherland GR, Chan S. Evaluation of software tools for segmentation of temporal bone anatomy. Stud Health Technol Inform. 2016; 220:130–133.