Deep Neural Networks (DNNs) have demonstrated substantial prowess in improving surgical precision by accurately identifying robotic instruments and tissues through semantic segmentation. However, DNNs grapple with catastrophic forgetting, signifying a rapid performance decline on previously learned tasks when new ones are introduced. This poses significant problems, especially in cases where old data is not accessible due to privacy concerns or when new data is introduced.
To address this challenge, an innovative privacy-preserving synthetic continual semantic segmentation framework has been proposed for robotic surgery. This approach combines existing instrument images with synthesized and real backgrounds. It uses overlapping class-aware temperature normalization (CAT) and multi-scale shifted-feature distillation (SD) to increase the utility of model learning.
The proposed methodology brings several innovative solutions to tackle the problems of continual learning in semantic segmentation in robotic surgery. One such innovation is the use of StyleGAN-XL to create synthetic data. By generating realistic background tissue images synthetically, it ensures no compromise on patient privacy. Techniques like blending and harmonization are used to improve the synthetic images’ realism and cover environmental variations, thus adding to the model robustness. Additionally, introducing CAT allows for control over learning utility for different classes, preventing imbalances without causing catastrophic forgetting.
In addition, the method uses multi-scale shifted-feature distillation to maintain spatial relationships amongst semantic objects, which conventional feature distillation methods do not provide. This methodology blends distillation losses, including both logits and feature distillation, to strike the right balance between model rigidity and flexibility. All these innovations together make this methodology a comprehensive solution tailored specifically to the unique demands of semantic segmentation in robotic surgery.
The efficacy of the method was verified using EndoVis 2017 and 2018 datasets. It demonstrated effective mitigation of catastrophic forgetting and balanced performance across old and new instruments. Robustness tests also showed superior performance under various uncertainties compared to the baseline methods. An ablation study was conducted to understand the hyperparameters impact, revealing optimal settings that significantly improved learning outcomes.
In conclusion, this novel privacy-preserving synthetic continual semantic segmentation approach for robotic instrument segmentation offers a promising solution to the problem of catastrophic forgetting in medical data. Extensive experiments have validated its superiority over existing state-of-the-art techniques. Future work will focus on exploring incremental domain adaptation techniques to further enhance model adaptability.
