Self-adaptive fine-tuning of deep learning super-resolution microscopy for artifact suppression in live-cell imaging

Public summary

* AI can enhance the spatial-temporal resolution of diverse microscopy modalities.

* We present SAFT, a self-adaptive fine-tuning method for deep learning super-resolution microscopy.

* SAFT dynamically optimizes model parameters by integrating artifact quantification into the loss function.

* SAFT allows microscopes to intelligently adapt to changing imaging conditions and suppress artifacts.

Abstract

In deep learning super-resolution microscopy, concerns exist about the generation of artifacts, and methods for artifact suppression are lacking. We developed a self-adaptive fine-tuning method that dynamically adjusts the parameters of the models to minimize the loss function, which includes direct quantification of artifacts from live-cell imaging. Integrating self-adaptive fine-tuning with super-resolution models enables significant artifact reduction in the visualization of nanoscale organelle interactions at high spatial-temporal resolution.

Introduction

Deep learning models can be used with different microscopy modalities to increase their spatial and temporal resolution. However, artifacts that may be generated by deep learning models continue to raise significant concerns. While several methods have been developed to quantify artifacts, effective methods for suppressing artifacts are still lacking. Researchers have strived to enhance model performance in deep learning super-resolution (SR) microscopy by employing advanced architectures, refining loss functions, and using pretrained foundation models. However, these strategies do not explicitly address the need to minimize artifacts. In practical applications, upon the deployment of a pretrained model for a local microscopy system, diverse user practices may result in large variations in imaging conditions. Such variations are also common in live-cell imaging. These conditions further complicate the issue of artifact minimization. In practice, training data that cover all usage scenarios and imaging conditions is time-consuming and impractical to obtain. However, when provided with inadequate training data, deep learning models may generate fuzzy image details and artifacts.15 Self-supervised learning frameworks have been employed in the computer vision community to reduce artifacts and improve the generalization of SR models.16 However, the image degradation model in fluorescence microscopy is fundamentally different from that in natural image processing, as the latter often relies purely on data-driven priors and lacks the incorporation of physics-based knowledge of the optical system. To solve this problem, we propose a self-adaptive fine-tuning (SAFT) strategy (Figure 1). The core idea of SAFT is to use an evaluation metric to assess the quality of SR outputs during the inference stage and fine-tune the models to minimize the loss function on the basis of this evaluation. In daily-use scenarios, after a pretrained SR network (SRN) is loaded onto a local microscope, the SAFT technique enables adjustments to the foundational model parameters to adapt to different imaging configurations as users load various fluorescence samples.