A Marker-Less Registration Approach for Mixed Reality–Aided Maxillofacial Surgery: a Pilot Evaluation

Antonio Pepe, Gianpaolo F Trotta, Peter Mohr-Ziak, Christina Gsaxner, Jürgen Wallner, Vitoantonio Bevilacqua, Jan Egger

Research output: Contribution to journalArticleResearchpeer-review

Abstract

As of common routine in tumor resections, surgeons rely on local examinations of the removed tissues and on the swiftly made microscopy findings of the pathologist, which are based on intraoperatively taken tissue probes. This approach may imply an extended duration of the operation, increased effort for the medical staff, and longer occupancy of the operating room (OR). Mixed reality technologies, and particularly augmented reality, have already been applied in surgical scenarios with positive initial outcomes. Nonetheless, these methods have used manual or marker-based registration. In this work, we design an application for a marker-less registration of PET-CT information for a patient. The algorithm combines facial landmarks extracted from an RGB video stream, and the so-called Spatial-Mapping API provided by the HMD Microsoft HoloLens. The accuracy of the system is compared with a marker-based approach, and the opinions of field specialists have been collected during a demonstration. A survey based on the standard ISO-9241/110 has been designed for this purpose. The measurements show an average positioning error along the three axes of (x, y, z) = (3.3 ± 2.3, − 4.5 ± 2.9, − 9.3 ± 6.1) mm. Compared with the marker-based approach, this shows an increment of the positioning error of approx. 3 mm along two dimensions (x, y), which might be due to the absence of explicit markers. The application has been positively evaluated by the specialists; they have shown interest in continued further work and contributed to the development process with constructive criticism.
Original languageEnglish
Pages (from-to)1008-1018
Number of pages11
JournalJournal of digital imaging
Volume32
Issue number6
DOIs
Publication statusPublished - 4 Sep 2019

Fingerprint

Oral Surgery
Surgery
Tissue
Helmet mounted displays
Operating rooms
Augmented reality
Medical Staff
Operating Rooms
Application programming interfaces (API)
Tumors
Microscopy
Microscopic examination
Demonstrations
Technology
Neoplasms
Surgeons
Surveys and Questionnaires
Pathologists
Positron Emission Tomography Computed Tomography

Cite this

A Marker-Less Registration Approach for Mixed Reality–Aided Maxillofacial Surgery: a Pilot Evaluation. / Pepe, Antonio; Trotta, Gianpaolo F; Mohr-Ziak, Peter; Gsaxner, Christina; Wallner, Jürgen; Bevilacqua, Vitoantonio; Egger, Jan.

In: Journal of digital imaging, Vol. 32, No. 6, 04.09.2019, p. 1008-1018.

Research output: Contribution to journalArticleResearchpeer-review

Pepe, Antonio ; Trotta, Gianpaolo F ; Mohr-Ziak, Peter ; Gsaxner, Christina ; Wallner, Jürgen ; Bevilacqua, Vitoantonio ; Egger, Jan. / A Marker-Less Registration Approach for Mixed Reality–Aided Maxillofacial Surgery: a Pilot Evaluation. In: Journal of digital imaging. 2019 ; Vol. 32, No. 6. pp. 1008-1018.
@article{0033c5b9e45448a9a467f3da83127913,
title = "A Marker-Less Registration Approach for Mixed Reality–Aided Maxillofacial Surgery: a Pilot Evaluation",
abstract = "As of common routine in tumor resections, surgeons rely on local examinations of the removed tissues and on the swiftly made microscopy findings of the pathologist, which are based on intraoperatively taken tissue probes. This approach may imply an extended duration of the operation, increased effort for the medical staff, and longer occupancy of the operating room (OR). Mixed reality technologies, and particularly augmented reality, have already been applied in surgical scenarios with positive initial outcomes. Nonetheless, these methods have used manual or marker-based registration. In this work, we design an application for a marker-less registration of PET-CT information for a patient. The algorithm combines facial landmarks extracted from an RGB video stream, and the so-called Spatial-Mapping API provided by the HMD Microsoft HoloLens. The accuracy of the system is compared with a marker-based approach, and the opinions of field specialists have been collected during a demonstration. A survey based on the standard ISO-9241/110 has been designed for this purpose. The measurements show an average positioning error along the three axes of (x, y, z) = (3.3 ± 2.3, − 4.5 ± 2.9, − 9.3 ± 6.1) mm. Compared with the marker-based approach, this shows an increment of the positioning error of approx. 3 mm along two dimensions (x, y), which might be due to the absence of explicit markers. The application has been positively evaluated by the specialists; they have shown interest in continued further work and contributed to the development process with constructive criticism.",
author = "Antonio Pepe and Trotta, {Gianpaolo F} and Peter Mohr-Ziak and Christina Gsaxner and J{\"u}rgen Wallner and Vitoantonio Bevilacqua and Jan Egger",
year = "2019",
month = "9",
day = "4",
doi = "10.1007/s10278-019-00272-6",
language = "English",
volume = "32",
pages = "1008--1018",
journal = "Journal of digital imaging",
issn = "0897-1889",
publisher = "Springer New York",
number = "6",

}

TY - JOUR

T1 - A Marker-Less Registration Approach for Mixed Reality–Aided Maxillofacial Surgery: a Pilot Evaluation

AU - Pepe, Antonio

AU - Trotta, Gianpaolo F

AU - Mohr-Ziak, Peter

AU - Gsaxner, Christina

AU - Wallner, Jürgen

AU - Bevilacqua, Vitoantonio

AU - Egger, Jan

PY - 2019/9/4

Y1 - 2019/9/4

N2 - As of common routine in tumor resections, surgeons rely on local examinations of the removed tissues and on the swiftly made microscopy findings of the pathologist, which are based on intraoperatively taken tissue probes. This approach may imply an extended duration of the operation, increased effort for the medical staff, and longer occupancy of the operating room (OR). Mixed reality technologies, and particularly augmented reality, have already been applied in surgical scenarios with positive initial outcomes. Nonetheless, these methods have used manual or marker-based registration. In this work, we design an application for a marker-less registration of PET-CT information for a patient. The algorithm combines facial landmarks extracted from an RGB video stream, and the so-called Spatial-Mapping API provided by the HMD Microsoft HoloLens. The accuracy of the system is compared with a marker-based approach, and the opinions of field specialists have been collected during a demonstration. A survey based on the standard ISO-9241/110 has been designed for this purpose. The measurements show an average positioning error along the three axes of (x, y, z) = (3.3 ± 2.3, − 4.5 ± 2.9, − 9.3 ± 6.1) mm. Compared with the marker-based approach, this shows an increment of the positioning error of approx. 3 mm along two dimensions (x, y), which might be due to the absence of explicit markers. The application has been positively evaluated by the specialists; they have shown interest in continued further work and contributed to the development process with constructive criticism.

AB - As of common routine in tumor resections, surgeons rely on local examinations of the removed tissues and on the swiftly made microscopy findings of the pathologist, which are based on intraoperatively taken tissue probes. This approach may imply an extended duration of the operation, increased effort for the medical staff, and longer occupancy of the operating room (OR). Mixed reality technologies, and particularly augmented reality, have already been applied in surgical scenarios with positive initial outcomes. Nonetheless, these methods have used manual or marker-based registration. In this work, we design an application for a marker-less registration of PET-CT information for a patient. The algorithm combines facial landmarks extracted from an RGB video stream, and the so-called Spatial-Mapping API provided by the HMD Microsoft HoloLens. The accuracy of the system is compared with a marker-based approach, and the opinions of field specialists have been collected during a demonstration. A survey based on the standard ISO-9241/110 has been designed for this purpose. The measurements show an average positioning error along the three axes of (x, y, z) = (3.3 ± 2.3, − 4.5 ± 2.9, − 9.3 ± 6.1) mm. Compared with the marker-based approach, this shows an increment of the positioning error of approx. 3 mm along two dimensions (x, y), which might be due to the absence of explicit markers. The application has been positively evaluated by the specialists; they have shown interest in continued further work and contributed to the development process with constructive criticism.

U2 - 10.1007/s10278-019-00272-6

DO - 10.1007/s10278-019-00272-6

M3 - Article

VL - 32

SP - 1008

EP - 1018

JO - Journal of digital imaging

JF - Journal of digital imaging

SN - 0897-1889

IS - 6

ER -