‘Pokémon GO’ technology provides an interactive 3D view of organs

A new technology that is ‘a bit like Pokémon GO, but with much greater precision’ and generates an interactive 3D image of an organ during surgery can reveal tumours hidden beneath the surface, helping surgeons target them for removal.
The surgeon places a small QR-code sticker on the organ. So far, the technology has been tested only on the liver. While the camera tracks the marker, underlying tumours and other structures remain visible.
Niclas Kvarnström is a transplant surgeon and head of the Transplant Centre at Sahlgrenska University Hospital, and a researcher at Sahlgrenska Academy at the University of Gothenburg.
Niclas said: ‘It is a bit like having GPS navigation inside the organ. Instead of simply having a map before surgery, we receive continuous guidance based on the current camera angle. We could previously create a 3D image, but it wasn’t reliable because the organ changes during surgery. If we wanted an updated ultrasound image, we had to interrupt the procedure.’
The new technology links a 3D image of the patient’s anatomy to the surgical field and displays it in relation to the surgeon’s camera view.
Using augmented reality, tumours, blood vessels, and other anatomical structures hidden beneath the organ’s surface can be made visible to the surgeon directly in the surgical field. The information updates continuously as the camera moves.
Niclas added: ‘This means that computer-generated content is overlaid on the physical world. It is a bit like Pokémon GO, but with much greater precision.’
The technology was developed through a collaboration between Sahlgrenska University Hospital, Chalmers University of Technology, and the University of Gothenburg. Chalmers Ventures helped establish a company to further develop the technology.
The collaboration dates back 10 years, to when the Imaging and Intervention Centre at Sahlgrenska University Hospital had recently opened.
Surgeons Niclas Kvarnström and Mårten Falkenberg contacted Torbjörn Lundh and Klas Modin, professors in the Department of Mathematical Sciences at Chalmers and the University of Gothenburg.
The initial focus was on using diagnostic imaging in hybrid operating rooms and developing minimally invasive methods, including in liver and tumour surgery.
The road to the first operation has been long, involving several years of technical development, preclinical studies, and testing of the system in different scenarios. The first operation on a human patient was performed in June of this year.
Although the first clinical application has been in liver surgery, Niclas Kvarnström sees potential in other areas too.
‘This technology can be used in many routine procedures, for tumours in other organs, or to track specific anatomical structures. We are starting with liver surgery, but in time the technology could have much broader applications,’ he said.