A new project investigates how artificial intelligence (AI) and augmented reality (AR) can assist surgical teams in the operating theatre.
The aim is to enhance hospital workflows to improve efficiency and safety.
The KARVIMIO research project, led by the Institute for Work and Technology, also raises important ethical questions and underscores potential risks.
Funded by the German Federal Ministry of Education and Research (BMBF), the project examines how AI and AR can be utilised in operating theatres to support surgical assistants and enhance patient safety.
The project focuses on developing so-called ‘in-situ instructions’ – visual, step-by-step guides displayed through AR glasses or head-mounted devices such as Microsoft’s HoloLens 2.
These instructions appear directly within the user’s field of view. The AI recognises the surgical instrument in use and overlays the corresponding setup or usage guide onto the real-world object, eliminating the need for printed manuals that are impractical in sterile environments. A depth camera further enhances the system by detecting objects even outside the user’s immediate line of sight.
Led by Elena Fitzner and Dr Peter Enste, the project team follows a strictly user-centred development approach.
Throughout all phases, surgical assistants, equipment manufacturers, and central sterilisation staff have been actively involved through workshops and hands-on testing.
Participants emphasised the importance of intuitive controls, clear colour displays and easy-to-read visuals. However, some expressed concerns that AR systems might feel intrusive, particularly during high-pressure situations or routine tasks.
Staff reliance on technology may reduce their routine knowledge, while technical issues like misrecognition or failures can disrupt workflows.
Ethical and legal issues also arise, including liability for erroneous instructions and ensuring that technology upholds human responsibility.
The project is also examining ethical, legal, and social aspects through the ELSI framework, emphasising trust, data protection, responsibility and skills development.
The goal is to develop systems that augment rather than replace human abilities. AI and AR hold broad potential in the operating room, not only for surgical assistance but also for training, documentation, and workflow improvements in sterile services.
