J Korean Neurosurg Soc.  2025 Mar;68(2):137-149. 10.3340/jkns.2024.0160.

Envisioning the Future of the Neurosurgical Operating Room with the Concept of the Medical Metaverse

• Nam SM
• Byun YH
• Dho YS
• Park CK

• KMID: 2565272
• DOI: http://doi.org/10.3340/jkns.2024.0160

Abstract

The medical metaverse can be defined as a virtual spatiotemporal framework wherein higher-dimensional medical information is generated, exchanged, and utilized through communication among medical personnel or patients. This occurs through the integration of cutting-edge technologies such as augmented reality (AR), virtual reality (VR), artificial intelligence (AI), big data, cloud computing, and others. We can envision a future neurosurgical operating room that utilizes such medical metaverse concept such as shared extended reality (AR/VR) of surgical field, AI-powered intraoperative neurophysiological monitoring, and real-time intraoperative tissue diagnosis. The future neurosurgical operation room will evolve into a true medical metaverse where participants of surgery can communicate in overlapping virtual layers of surgery, monitoring, and diagnosis.

Keyword

Artificial intelligence; Augmented reality; Metaverse; Neurosurgery; Telemedicine

Figure

• 

  Fig. 1. Conceptual diagram of future neurosurgical operation room. Multi-layer of medical metaverse where various surgical participants communicate throughout the entire process of surgery, monitoring, and diagnosis. AI : artificial intelligence, AR : augmented reality.

• 

  Fig. 2. Timeline of head-mounted display development and clinical applications as augmented reality navigation systems in neurosurgery. VP shunt, ventriculoperitoneal shunt.

Reference

References

1. Abramov I, Park MT, Belykh E, Dru AB, Xu Y, Gooldy TC, et al. Intraoperative confocal laser endomicroscopy: prospective in vivo feasibility study of a clinical-grade system for brain tumors. J Neurosurg. 138:587–597. 2023.

2. Abramov I, Park MT, Gooldy TC, Xu Y, Lawton MT, Little AS, et al. Real-time intraoperative surgical telepathology using confocal laser endomicroscopy. Neurosurg Focus. 52:E9. 2022.

3. Ahuja AS, Polascik BW, Doddapaneni D, Byrnes ES, Sridhar J. The digital metaverse: applications in artificial intelligence, medical education, and integrative health. Integr Med Res. 12:100917. 2023.

4. Amann J, Blasimme A, Vayena E, Frey D, Madai VI; Precise4Q consortium. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Med Inform Decis Mak. 20:310. 2020.

5. Amos WB, White JG. How the confocal laser scanning microscope entered biological research. Biol Cell. 95:335–342. 2003.

6. Andrews C, Southworth MK, Silva JNA, Silva JR. Extended reality in medical practice. Curr Treat Options Cardiovasc Med. 21:18. 2019.

7. Aschke M, Wirtz CR, Raczkowsky J, Worn H, Kunze S. Augmented reality in operating microscopes for neurosurgical interventions. In : First International IEEE EMBS Conference on Neural Engineering, 2003.; IEEE;2003. p. 652–655.

8. Balzer JR, Caviness J, Krieger D. The evolution of real-time remote intraoperative neurophysiological monitoring. Computer. 56:28–38. 2023.

9. Baum ZMC, Lasso A, Ryan S, Ungi T, Rae E, Zevin B, et al. Augmented reality training platform for neurosurgical burr hole localization. J Med Robot Res. 04:1942001. 2019.

10. Belykh E, Zhao X, Ngo B, Farhadi DS, Kindelin A, Ahmad S, et al. Visualization of brain microvasculature and blood flow in vivo: feasibility study using confocal laser endomicroscopy. Microcirculation. 28:e12678. 2021.

11. Besharati Tabrizi L, Mahvash M. Augmented reality-guided neurosurgery: accuracy and intraoperative application of an image projection technique. J Neurosurg. 123:206–211. 2015.

12. Byun YH, Won JK, Hong DH, Kang H, Kim JH, Yu MO, et al. A prospective multicenter assessor blinded pilot study using confocal laser endomicroscopy for intraoperative brain tumor diagnosis. Sci Rep. 14:6784. 2024.

13. Cabrilo I, Schaller K, Bijlenga P. Augmented reality-assisted bypass surgery: embracing minimal invasiveness. World Neurosurg. 83:596–602. 2015.

14. Centers for Medicare & Medicaid Services. Physician Supervision of Diagnostic Tests. Available at : https://www.hhs.gov/guidance/document/physician-supervision-diagnostic-tests.

15. Charalampaki P, Nakamura M, Athanasopoulos D, Heimann A. Confocal-assisted multispectral fluorescent microscopy for brain tumor surgery. Front Oncol. 9:583. 2019.

16. Cheng JX, Jia YK, Zheng G, Xie XS. Laser-scanning coherent anti-Stokes Raman scattering microscopy and applications to cell biology. Biophys J. 83:502–509. 2002.

17. Colombo E, Regli L, Esposito G, Germans MR, Fierstra J, Serra C, et al. Mixed reality for cranial neurosurgical planning: a single-center applicability study with the first 107 subsequent holograms. Oper Neurosurg (Hagerstown). 26:551–558. 2023.

18. Cui H, Xie X, Xu S, Hu Y. A Dynamic Prediction Model for Intraoperative Somatosensory Evoked Potential Monitoring. In : 2015 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA); Shenzhen, China. IEEE;2015. p. 31–35.

19. Cui Y, Zhou Y, Zhang H, Yuan Y, Wang J, Zhang Z. Application of glasses-free augmented reality localization in neurosurgery. World Neurosurg. 180:e296–e301. 2023.

20. DePaoli D, Lemoine É, Ember K, Parent M, Prud’homme M, Cantin L, et al. Rise of Raman spectroscopy in neurosurgery: a review. J Biomed Opt. 25:050901. 2020.

21. Desroches J, Lemoine É, Pinto M, Marple E, Urmey K, Diaz R, et al. Development and first in-human use of a Raman spectroscopy guidance system integrated with a brain biopsy needle. J Biophotonics. 12:e201800396. 2019.

22. Dho YS, Lee BC, Moon HC, Kim KM, Kang H, Lee EJ, et al. Validation of real-time inside-out tracking and depth realization technologies for augmented reality-based neuronavigation. Int J Comput Assist Radiol Surg. 19:15–25. 2024.

23. Dho YS, Park SJ, Choi H, Kim Y, Moon HC, Kim KM, et al. Development of an inside-out augmented reality technique for neurosurgical navigation. Neurosurg Focus. 51:E21. 2021.

24. Diaz R, Yoon J, Chen R, Quinones-Hinojosa A, Wharen R, Komotar R. Real-time video-streaming to surgical loupe mounted head-up display for navigated meningioma resection. Turk Neurosurg. 28:682–688. 2018.

25. Egger J, Gsaxner C, Luijten G, Chen J, Chen X, Bian J, et al. Is the apple vision pro the ultimate display? A first perspective and survey on entering the wonderland of precision medicine. JMIR Serious Games. 12:e52785. 2024.

26. Egger MD, Petrăn M. New reflected-light microscope for viewing unstained brain and ganglion cells. Science. 157:305–307. 1967.

27. Eschbacher J, Martirosyan NL, Nakaji P, Sanai N, Preul MC, Smith KA, et al. In vivo intraoperative confocal microscopy for real-time histopathological imaging of brain tumors. J Neurosurg. 116:854–860. 2012.

28. Fan B, Li HX, Hu Y. An intelligent decision system for intraoperative somatosensory evoked potential monitoring. IEEE Trans Neural Syst Rehabil Eng. 24:300–307. 2016.

29. Fick T, van Doormaal JAM, Hoving EW, Willems PWA, van Doormaal TPC. Current accuracy of augmented reality neuronavigation systems: systematic review and meta-analysis. World Neurosurg. 146:179–188. 2021.

30. Freudiger CW, Min W, Saar BG, Lu S, Holtom GR, He C, et al. Label-free biomedical imaging with high sensitivity by stimulated raman scattering microscopy. Science. 322:1857–1861. 2008.

31. Gibby J, Cvetko S, Javan R, Parr R, Gibby W. Use of augmented reality for image-guided spine procedures. Eur Spine J. 29:1823–1832. 2020.

32. Heinrich F, Schwenderling L, Becker M, Skalej M, Hansen C. HoloInjection: augmented reality support for CT-guided spinal needle injections. Healthc Technol Lett. 6:165–171. 2019.

33. Höhne J, Schebesch KM, Zoubaa S, Proescholdt M, Riemenschneider MJ, Schmidt NO. Intraoperative imaging of brain tumors with fluorescein: confocal laser endomicroscopy in neurosurgery. Clinical and user experience. Neurosurg Focus. 50:E19. 2021.

34. Hollon T, Jiang C, Chowdury A, Nasir-Moin M, Kondepudi A, Aabedi A, et al. Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging. Nat Med. 29:828–832. 2023.

35. Hollon T, Kondepudi A, Pekmezci M, Hou X, Scotford K, Jiang C, et al. Visual foundation models for fast, label-free detection of diffuse glioma infiltration. Available at : https://doi.org/10.21203/rs.3.rs-4033133/v1.

36. Hollon TC, Pandian B, Adapa AR, Urias E, Save AV, Khalsa SSS, et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks. Nat Med. 26:52–58. 2020.

37. Incekara F, Smits M, Dirven C, Vincent A. Clinical feasibility of a wearable mixed-reality device in neurosurgery. World Neurosurg. 118:e422–e427. 2018.

38. Iseki H, Masutani Y, Iwahara M, Tanikawa T, Muragaki Y, Taira T, et al. Volumegraph (overlaid three-dimensional image-guided navigation). Clinical application of augmented reality in neurosurgery. Stereotact Funct Neurosurg. 68(1-4 Pt 1):18–24. 1997.

39. Ivan ME, Eichberg DG, Di L, Shah AH, Luther EM, Lu VM, et al. Augmented reality head-mounted display-based incision planning in cranial neurosurgery: a prospective pilot study. Neurosurg Focus. 51:E3. 2021.

40. Jain S, Gao Y, Yeo TT, Ngiam KY. Use of mixed reality in neuro-oncology: a single centre experience. Life (Basel). 13:398. 2023.

41. Jamaludin MR, Lai KW, Chuah JH, Zaki MA, Hasikin K, Abd Razak NA, et al. Machine learning application of transcranial motor-evoked potential to predict positive functional outcomes of patients. Comput Intell Neurosci. 2022:2801663. 2022.

42. Jermyn M, Mok K, Mercier J, Desroches J, Pichette J, Saint-Arnaud K, et al. Intraoperative brain cancer detection with Raman spectroscopy in humans. Sci Transl Med. 7:274ra219. 2015.

43. Jiang W, Zhan Q, Wang J, Wei M, Li S, Mei R, et al. Quantitative identification of ventral/dorsal nerves through intraoperative neurophysiological monitoring by supervised machine learning. Front Pediatr. 11:1118924. 2023.

44. Koljenović S, Choo-Smith LP, Bakker Schut TC, Kros JM, van den Berge HJ, Puppels GJ. Discriminating vital tumor from necrotic tissue in human glioblastoma tissue samples by Raman spectroscopy. Lab Invest. 82:1265–1277. 2002.

45. Kuo TT, Kim HE, Ohno-Machado L. Blockchain distributed ledger technologies for biomedical and health care applications. J Am Med Inform Assoc. 24:1211–1220. 2017.

46. Lai M, Skyrman S, Shan C, Babic D, Homan R, Edström E, et al. Fusion of augmented reality imaging with the endoscopic view for endonasal skull base surgery; a novel application for surgical navigation based on intraoperative cone beam computed tomography and optical tracking. PLoS One. 15:e0227312. 2020.

47. Mahvash M, Besharati Tabrizi L. A novel augmented reality system of image projection for image-guided neurosurgery. Acta Neurochir (Wien). 155:943–947. 2013.

48. Martirosyan NL, Cavalcanti DD, Eschbacher JM, Delaney PM, Scheck AC, Abdelwahab MG, et al. Use of in vivo near-infrared laser confocal endomicroscopy with indocyanine green to detect the boundary of infiltrative tumor. J Neurosurg. 115:1131–1138. 2011.

49. Minsky M. Memoir on inventing the confocal scanning microscope. Scanning. 10:128–138. 1988.

50. Mizuno A, Kitajima H, Kawauchi K, Muraishi S, Ozaki Y. Near-infrared Fourier transform Raman spectroscopic study of human brain tissues and tumours. J Raman Spectrosc. 25:25–29. 1994.

51. Mooney MA, Zehri AH, Georges JF, Nakaji P. Laser scanning confocal endomicroscopy in the neurosurgical operating room: a review and discussion of future applications. Neurosurg Focus. 36:E9. 2014.

52. Nebeker C, Torous J, Bartlett Ellis RJ. Building the case for actionable ethics in digital health research supported by artificial intelligence. BMC Med. 17:137. 2019.

53. Paleologos TS, Wadley JP, Kitchen ND, Thomas DG. Clinical utility and cost-effectiveness of interactive image-guided craniotomy: clinical comparison between conventional and image-guided meningioma surgery. Neurosurgery. 47:40–47. discussion 47-48. 2000.

54. Pavlov V, Meyronet D, Meyer-Bisch V, Armoiry X, Pikul B, Dumot C, et al. Intraoperative probe-based confocal laser endomicroscopy in surgery and stereotactic biopsy of low-grade and high-grade gliomas: a feasibility study in humans. Neurosurgery. 79:604–612. 2016.

55. Pescador AM, Lavrador JP, Lejarde A, Bleil C, Vergani F, Baamonde AD, et al. Bayesian networks for risk assessment and postoperative deficit prediction in intraoperative neurophysiology for brain surgery. J Clin Monit Comput. 38:1043–1055. 2024.

56. Porras JL, Khalid S, Root BK, Khan IS, Singer RJ. Point-of-view recording devices for intraoperative neurosurgical video capture. Front Surg. 3:57. 2016.

57. Potma EO, de Boeij WP, van Haastert PJ, Wiersma DA. Real-time visualization of intracellular hydrodynamics in single living cells. Proc Natl Acad Sci U S A. 98:1577–1582. 2001.

58. Qiao N, Song M, Ye Z, He W, Ma Z, Wang Y, et al. Deep learning for automatically visual evoked potential classification during surgical decompression of sellar region tumors. Transl Vis Sci Technol. 8:21. 2019.

59. Raman CV, Krishnan KS. A new type of secondary radiation. Nature. 121:501–502. 1928.

60. Roberts DW, Strohbehn JW, Hatch JF, Murray W, Kettenberger H. A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. J Neurosurg. 65:545–549. 1986.

61. Sanai N, Eschbacher J, Hattendorf G, Coons SW, Preul MC, Smith KA, et al. Intraoperative confocal microscopy for brain tumors: a feasibility analysis in humans. Neurosurgery. 68(2 Suppl Operative):282–290. discussion 290. 2011.

62. Sanai N, Snyder LA, Honea NJ, Coons SW, Eschbacher JM, Smith KA, et al. Intraoperative confocal microscopy in the visualization of 5-aminolevulinic acid fluorescence in low-grade gliomas. J Neurosurg. 115:740–748. 2011.

63. Sankar T, Delaney PM, Ryan RW, Eschbacher J, Abdelwahab M, Nakaji P, et al. Miniaturized handheld confocal microscopy for neurosurgery: results in an experimental glioblastoma model. Neurosurgery. 66:410–417. discussion 417-418. 2010.

64. Shenai MB, Tubbs RS, Guthrie BL, Cohen-Gadol AA. Virtual interactive presence for real-time, long-distance surgical collaboration during complex microsurgical procedures. J Neurosurg. 121:277–284. 2014.

65. Shu XJ, Wang Y, Xin H, Zhang ZZ, Xue Z, Wang FY, et al. Real-time augmented reality application in presurgical planning and lesion scalp localization by a smartphone. Acta Neurochir (Wien). 164:1069–1078. 2022.

66. Sievert M, Stelzle F, Aubreville M, Mueller SK, Eckstein M, Oetter N, et al. Intraoperative free margins assessment of oropharyngeal squamous cell carcinoma with confocal laser endomicroscopy: a pilot study. Eur Arch Otorhinolaryngol. 278:4433–4439. 2021.

67. Skyrman S, Lai M, Edström E, Burström G, Förander P, Homan R, et al. Augmented reality navigation for cranial biopsy and external ventricular drain insertion. Neurosurg Focus. 51:E7. 2021.

68. Tashibu K. Analysis of water content in rat brain using Raman spectroscopy. No to shinkei. 42:999–1004. 1990.

69. Van Gestel F, Frantz T, Buyck F, Geens W, Neuville Q, Bruneau M, et al. Neuro-oncological augmented reality planning for intracranial tumor resection. Front Neurol. 14:1104571. 2023.

70. Wang JY, Qu V, Hui C, Sandhu N, Mendoza MG, Panjwani N, et al. Stratified assessment of an FDA-cleared deep learning algorithm for automated detection and contouring of metastatic brain tumors in stereotactic radiosurgery. Radiat Oncol. 18:61. 2023.

71. Wang Y, Su Z, Zhang N, Xing R, Liu D, Luan TH, et al. A survey on metaverse: fundamentals, security, and privacy. IEEE Commun Surv Tutor. 25:319–352. 2023.

72. Wilson JP Jr, Kumbhare D, Ronkon C, Guthikonda B, Hoang S. Application of machine learning strategies to model the effects of sevoflurane on somatosensory-evoked potentials during spine surgery. Diagnostics (Basel). 13:3389. 2023.

73. Wilson JP Jr, Kumbhare D, Kandregula S, Oderhowho A, Guthikonda B, Hoang S. Proposed applications of machine learning to intraoperative neuromonitoring during spine surgeries. Neurosci Inform. 3:100143. 2023.

74. Xu Y, Abramov I, Belykh E, Mignucci-Jiménez G, Park MT, Eschbacher JM, et al. Characterization of ex vivo and in vivo intraoperative neurosurgical confocal laser endomicroscopy imaging. Front Oncol. 12:979748. 2022.

75. Xu Y, Mathis AM, Pollo B, Schlegel J, Maragkou T, Seidel K, et al. Intraoperative in vivo confocal laser endomicroscopy imaging at glioma margins: can we detect tumor infiltration? J Neurosurg. 40:357–366. 2023.

76. Yoon JW, Chen RE, Han PK, Si P, Freeman WD, Pirris SM. Technical feasibility and safety of an intraoperative head-up display device during spine instrumentation. Int J Med Robot. 13:e1770. 2017.

77. Yoon JW, Chen RE, ReFaey K, Diaz RJ, Reimer R, Komotar RJ, et al. Technical feasibility and safety of image-guided parieto-occipital ventricular catheter placement with the assistance of a wearable head-up display. Int J Med Robot. 13:e1836. 2017.

78. Zehri AH, Ramey W, Georges JF, Mooney MA, Martirosyan NL, Preul MC, et al. Neurosurgical confocal endomicroscopy: a review of contrast agents, confocal systems, and future imaging modalities. Surg Neurol Int. 5:60. 2014.

79. Zha X, Wehbe L, Sclabassi RJ, Mace Z, Liang YV, Yu A, et al. A deep learning model for automated classification of intraoperative continuous emg. IEEE Trans Med Robot Bionics. 3:44–52. 2021.

80. Zhang J, Yang Z, Jiang S, Zhou Z. A spatial registration method based on 2D-3D registration for an augmented reality spinal surgery navigation system. Int J Med Robot. 20:e2612. 2024.

81. Zhang ZY, Duan WC, Chen RK, Zhang FJ, Yu B, Zhan YB, et al. Preliminary application of mxed reality in neurosurgery: development and evaluation of a new intraoperative procedure. J Clin Neurosci. 67:234–238. 2019.

82. Ziebart A, Stadniczuk D, Roos V, Ratliff M, von Deimling A, Hänggi D, et al. Deep neural network for differentiation of brain tumor tissue displayed by confocal laser endomicroscopy. Front Oncol. 11:668273. 2021.

83. Zumbusch A, Holtom GR, Xie XS. Three-dimensional vibrational imaging by coherent anti-stokes Raman scattering. Phys Rev Lett 82. 82:4142–4145. 1999.