Microsnoop: A generalist tool for microscopy image representation

Dejin Xun; Rui Wang; Xingcai Zhang; Yi Wang

doi:10.1016/j.xinn.2023.100541

The Innovation > 2024 Vol. 5 > No. 1

Next Article Previous Article

REPORT Open Access

Cite PDF

Microsnoop: A generalist tool for microscopy image representation

1.
Pharmaceutical Informatics Institute, College of Pharmaceutical Sciences, Zhejiang University, Hangzhou 310058, China
2.
State Key Lab of Computer-Aided Design & Computer Graphics, Zhejiang University, Hangzhou 310058, China
3.
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA
4.
Innovation Institute for Artiﬁcial Intelligence in Medicine of Zhejiang University, Hangzhou 310018, China
5.
National Key Laboratory of Chinese Medicine Modernization, Innovation Center of Yangtze River Delta, Zhejiang University, Jiaxing 314100, China

More Information

Corresponding authors: ruiwang@zju.edu.cn (R.W.); xingcai@seas.harvard.edu (X.Z.); zjuwangyi@zju.edu.cn (Y.W.)
Received Date: May 31, 2023

Accepted Date: November 17, 2023

Published Online: January 02, 2024

The Innovation 5(1), https://doi.org/10.1016/j.xinn.2023.100541 | Cite this article

GRAPHICAL ABSTRACT

GRAPHICAL ABSTRACT

DownLoad: Full size image

PUBLIC SUMMARY

PUBLIC SUMMARY
1. ■ Microsnoop is a deep learning tool for profiling heterogeneous microscopy images.
2. ■ Microsnoop provides generalist pipelines for processing various types of images.
3. ■ Microsnoop achieves cutting-edge microscopy image representation ability with great potential for expansion.
4. ■ Microsnoop is highly scalable for studies from small scale to high throughput.

ABSTRACT

ABSTRACT

Accurate profiling of microscopy images from small scale to high throughput is an essential procedure in basic and applied biological research. Here, we present Microsnoop, a novel deep learning–based representation tool trained on large-scale microscopy images using masked self-supervised learning. Microsnoop can process various complex and heterogeneous images, and we classified images into three categories: single-cell, full-field, and batch-experiment images. Our benchmark study on 10 high-quality evaluation datasets, containing over 2,230,000 images, demonstrated Microsnoop’s robust and state-of-the-art microscopy image representation ability, surpassing existing generalist and even several custom algorithms. Microsnoop can be integrated with other pipelines to perform tasks such as superresolution histopathology image and multimodal analysis. Furthermore, Microsnoop can be adapted to various hardware and can be easily deployed on local or cloud computing platforms. We will regularly retrain and reevaluate the model using community-contributed data to consistently improve Microsnoop.

FullText(HTML)

REFERENCES

[1]	Caicedo, J.C., Singh, S., and Carpenter, A.E. (2016). Applications in image-based profiling of perturbations. Curr. Opin. Biotechnol. 39, 134-142. View in Article CrossRef Google Scholar
[2]	Pratapa, A., Doron, M., and Caicedo, J.C. (2021). Image-based cell phenotyping with deep learning. Curr. Opin. Chem. Biol. 65, 9-17. View in Article CrossRef Google Scholar
[3]	Lu, A.X., Chong, Y.T., Hsu, I.S., et al. (2018). Integrating images from multiple microscopy screens reveals diverse patterns of change in the subcellular localization of proteins. Elife 7, e31872. View in Article CrossRef Google Scholar
[4]	Eulenberg, P., Köhler, N., Blasi, T., et al. (2017). Reconstructing cell cycle and disease progression using deep learning. Nat. Commun. 8, 463. View in Article CrossRef Google Scholar
[5]	Pawlowski, N., Caicedo, J.C., Singh, S., et al. (2016). Automating morphological profiling with generic deep convolutional networks. Preprint at. bioRxiv. http://biorxiv.org/lookup/doi/10.1101/085118. View in Article Google Scholar
[6]	Cuccarese, M.F., Earnshaw, B.A., Heiser, K., et al. (2020). Functional immune mapping with deep-learning enabled phenomics applied to immunomodulatory and COVID-19 drug discovery. Preprint at. bioRxiv. http://biorxiv.org/lookup/doi/10.1101/2020.08.02.233064. View in Article Google Scholar
[7]	Schürch, C.M., Bhate, S.S., Barlow, G.L., et al. (2020). Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front. Cell *182*, 1341-1359.e19. View in Article CrossRef Google Scholar
[8]	Eng, C.-H.L., Lawson, M., Zhu, Q., et al. (2019). Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH+. Nature *568*, 235-239. View in Article CrossRef Google Scholar
[9]	Carpenter, A.E., Jones, T.R., Lamprecht, M.R., et al. (2006). CellProfiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7, R100. View in Article CrossRef Google Scholar
[10]	Pau, G., Fuchs, F., Sklyar, O., et al. (2010). EBImage–an R package for image processing with applications to cellular phenotypes. Bioinformatics 26, 979-981. View in Article CrossRef Google Scholar
[11]	Singh, S., Bray, M.-A., Jones, T.R., et al. (2014). Pipeline for illumination correction of images for high-throughput microscopy. J. Microsc. *256*, 231-236. View in Article CrossRef Google Scholar
[12]	Caicedo, J.C., McQuin, C., Goodman, A., et al. (2018). Weakly supervised learning of singlecell feature embeddings. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9309–9318. View in Article Google Scholar
[13]	Lu, A.X., Kraus, O.Z., Cooper, S., et al. (2019). Learning unsupervised feature representations for single cell microscopy images with paired cell inpainting. PLoS Comput. Biol. 15, e1007348. View in Article CrossRef Google Scholar
[14]	Adnan, M., Kalra, S., and Tizhoosh, H.R. (2020). Representation learning of histopathology images using graph neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 988–989. View in Article Google Scholar
[15]	Perakis, A., Gorji, A., Jain, S., et al. (2021). Contrastive learning of single-cell phenotypic representations for treatment classification. In Machine Learning in Medical Imaging, 12966, C. Lian, X. Cao, I. Rekik, X. Xu, and P. Yan, eds., pp. 565–575. View in Article Google Scholar
[16]	Russakovsky, O., Deng, J., Su, H., et al. (2015). ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. *115*, 211-252. View in Article CrossRef Google Scholar
[17]	Bao, F., Deng, Y., Wan, S., et al. (2022). Integrative spatial analysis of cell morphologies and transcriptional states with MUSE. Nat. Biotechnol. 40, 1200-1209. View in Article CrossRef Google Scholar
[18]	Hua, S.B.Z., Lu, A.X., and Moses, A.M. (2021). CytoImageNet: a large-scale pretraining dataset for bioimage transfer learning. In Proc. Advances in Neural Information Processing Systems. View in Article Google Scholar
[19]	Kobayashi, H., Cheveralls, K.C., Leonetti, M.D., et al. (2022). Self-supervised deep learning encodes high-resolution features of protein subcellular localization. Nat. Methods 19, 995-1003. View in Article CrossRef Google Scholar
[20]	Wong, D.R., Conrad, J., Johnson, N., et al. (2022). Trans-channel fluorescence learning improves high-content screening for Alzheimer’s disease therapeutics. Nat. Mach. Intell. 4, 583-595. View in Article CrossRef Google Scholar
[21]	He, K., Chen, X., Xie, S., et al. (2022). Masked autoencoders are scalable vision learners. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 16000–16009. View in Article Google Scholar
[22]	Liu, X., Zhou, J., Kong, T., et al. (2022). Exploring target representations for masked autoencoders. Preprint at. arXiv. https://arxiv.org/abs/2209.03917. View in Article Google Scholar
[23]	Li, Z., Chen, Z., Yang, F., et al. (2021). MST: masked self-supervised transformer for visual representation. In SAVE Proc. Advances in Neural Information Processing Systems, p. 35. View in Article Google Scholar
[24]	Wei, C., Fan, H., Xie, S., et al. (2022). Masked feature prediction for self-supervised visual pretraining. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 14668–14678. View in Article Google Scholar
[25]	Pandey, V., Brune, C., and Strisciuglio, N. (2022). Self-supervised learning through colorization for microscopy images. In Image Analysis and Processing - ICIAP 2022, S. Sclaroff, C. Distante, and M. Leo, et al., eds., pp. 621–632. View in Article Google Scholar
[26]	Mascolini, A., Cardamone, D., Ponzio, F., et al. (2022). Exploiting generative self-supervised learning for the assessment of biological images with lack of annotations. BMC Bioinf. 23, 295. View in Article CrossRef Google Scholar
[27]	Stringer, C., Wang, T., Michaelos, M., et al. (2021). Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100-106. View in Article CrossRef Google Scholar
[28]	Edlund, C., Jackson, T.R., Khalid, N., et al. (2021). LIVECell-a large-scale dataset for label-free live cell segmentation. Nat. Methods 18, 1038-1045. View in Article CrossRef Google Scholar
[29]	Greenwald, N.F., Miller, G., Moen, E., et al. (2022). Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat. Biotechnol. 40, 555-565. View in Article CrossRef Google Scholar
[30]	Bray, M.-A., Singh, S., Han, H., et al. (2016). Cell Painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes. Nat. Protoc. 11, 1757-1774. View in Article CrossRef Google Scholar
[31]	Leek, J.T., Scharpf, R.B., Bravo, H.C., et al. (2010). Tackling the widespread and critical impact of batch effects in high-throughput data. Nat. Rev. Genet. 11, 733-739. View in Article CrossRef Google Scholar
[32]	Lin, A., and Lu, A.X. (2022). Incorporating knowledge of plates in batch normalization improves generalization of deep learning for microscopy images. In Proc. International Conference on Machine Learning, pp. 74–93. View in Article Google Scholar
[33]	Kumar, N., Verma, R., Anand, D., et al. (2020). A multi-organ nucleus segmentation challenge. IEEE Trans. Med. Imag. 39, 1380-1391. View in Article CrossRef Google Scholar
[34]	Verma, R., Kumar, N., Patil, A., et al. (2021). MoNuSAC2020: a multi-organ nuclei segmentation and classification challenge. IEEE Trans. Med. Imag. 40, 3413-3423. View in Article CrossRef Google Scholar
[35]	Amgad, M., Atteya, L.A., Hussein, H., et al. (2022). NuCLS: a scalable crowdsourcing, deep learning approach and dataset for nucleus classification, localization and segmentation. GigaScience 11, giac037. View in Article CrossRef Google Scholar
[36]	Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: convolutional networks for biomedical image segmentation. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. View in Article Google Scholar
[37]	Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. In Proc. Advances in Neural Information Processing Systems, 30. View in Article Google Scholar
[38]	Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations. View in Article Google Scholar
[39]	Ando, D.M., McLean, C.Y., and Berndl, M. (2017). Improving phenotypic measurements in high-content imaging screens. Preprint at. bioRxiv. http://biorxiv.org/lookup/doi/10.1101/161422. View in Article Google Scholar
[40]	Bray, M.-A., Weck, A.D., Durand, E., et al. (2022). High-content cellular screen image analysis benchmark study. Preprint at. bioRxiv. https://www.biorxiv.org/content/10.1101/2022.05.15.491989v1.abstract. View in Article Google Scholar
[41]	Lu, A., Lu, A., Schormann, W., et al. (2019). The Cells Out of Sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers. In Proc. Advances in Neural Information Processing Systems, p. 32. View in Article Google Scholar
[42]	Maaten, L., and Hinton, G. (2008). Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579-2605. View in Article Google Scholar
[43]	Caie, P.D., Walls, R.E., Ingleston-Orme, A., et al. (2010). High-content phenotypic profiling of drug response signatures across distinct cancer cells. Mol. Cancer Therapeut. 9, 1913-1926. View in Article CrossRef Google Scholar
[44]	Graham, S., Vu, Q.D., Raza, S.E.A., et al. (2019). Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. Image Anal. 58, 101563. View in Article CrossRef Google Scholar
[45]	Schraivogel, D., Kuhn, T.M., Rauscher, B., et al. (2022). High-speed fluorescence image-enabled cell sorting. Science *375*, 315-320. View in Article CrossRef Google Scholar
[46]	Tan, M., and Le, Q.V. (2019). EfficientNet: rethinking model scaling for convolutional neural networks. In Proc. International Conference on Machine Learning, pp. 6105–6114. View in Article Google Scholar
[47]	Szegedy, C., Vanhoucke, V., Ioffe, S., et al. (2016). Rethinking the Inception architecture for computer vision. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826. View in Article Google Scholar
[48]	Xun, D., Chen, D., Zhou, Y., et al. (2022). Scellseg: a style-aware deep learning tool for adaptive cell instance segmentation by contrastive fine-tuning. iScience 25, 105506. View in Article CrossRef Google Scholar
[49]	Pachitariu, M., and Stringer, C. (2022). Cellpose 2.0: how to train your own model. Nat. Methods 19, 1634-1641. View in Article CrossRef Google Scholar
[50]	Lazard, T., Bataillon, G., Naylor, P., et al. (2022). Deep learning identifies morphological patterns of homologous recombination deficiency in luminal breast cancers from whole slide images. Cell Rep. Med. 3, 100872. View in Article CrossRef Google Scholar
[51]	Moshkov, N., Bornholdt, M., Benoit, S., et al. (2022). Learning representations for imagebased profiling of perturbations. Preprint at. bioRxiv. http://biorxiv.org/lookup/doi/10.1101/2022.08.12.503783. View in Article Google Scholar
[52]	Taylor, J., Earnshaw, B., Mabey, B., et al. (2019). RxRx1: an image set for cellular morphological variation across many experimental batches. In 7th International Conference on Learning Representations. View in Article Google Scholar
[53]	Caicedo, J.C., Cooper, S., Heigwer, F., et al. (2017). Data-analysis strategies for image-based cell profiling. Nat. Methods 14, 849-863. View in Article CrossRef Google Scholar
[54]	Chandrasekaran, S.N., Ceulemans, H., Boyd, J.D., et al. (2021). Image-based profiling for drug discovery: due for a machine-learning upgrade? Nat. Rev. Drug Discov. 20, 145-159. View in Article CrossRef Google Scholar
[55]	Devlin, J., Chang, M.-W., Lee, K., et al. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. Preprint at. arXiv. https://arxiv.org/abs/1810.04805. View in Article Google Scholar
[56]	Brown, T.B., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems, p. 33. View in Article Google Scholar
[57]	Min, B., Ross, H., Sulem, E., et al. (2021). Recent advances in natural language processing via large pre-trained language models: a survey. Preprint at. arXiv. http://arxiv.org/abs/2111.01243. View in Article Google Scholar
[58]	Davari, M., Asadi, N., Mudur, S., et al. (2022). Probing representation forgetting in supervised and unsupervised continual learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 16691–16700. View in Article Google Scholar
[59]	Mundt, M., Hong, Y., Pliushch, I., et al. (2023). A wholistic view of continual learning with deep neural networks: forgotten lessons and the bridge to active and open world learning. Neural Network. *160*, 306-336. View in Article CrossRef Google Scholar
[60]	Abdalla, M., Abdalla, M., Hirst, G., et al. (2020). Exploring the privacy-preserving properties of word embeddings: algorithmic validation study. J. Med. Internet Res. 22, e18055. View in Article CrossRef Google Scholar
[61]	Wang, B., Li, Y., Zhou, M., et al. (2023). Smartphone-based platforms implementing microfluidic detection with image-based artificial intelligence. Nat. Commun. 14, 1341. View in Article Google Scholar
[62]	Sanchez-Fernandez, A., Rumetshofer, E., and Hochreiter, S. (2022). Contrastive learning of image- and structure- based representations in drug discovery. In International Conference on Learning Representations. View in Article Google Scholar
[63]	Tian, G., Harrison, P.J., Sreenivasan, A.P., et al. (2022). Combining molecular and cell painting image data for mechanism of action prediction. Preprint at. bioRxiv. http://biorxiv.org/lookup/doi/10.1101/2022.10.04.510834. View in Article Google Scholar
[64]	Haghighi, M., Caicedo, J.C., Cimini, B.A., et al. (2022). High-dimensional gene expression and morphology profiles of cells across 28,000 genetic and chemical perturbations. Nat. Methods 19, 1550-1557. View in Article CrossRef Google Scholar
[65]	Liu, L., Bi, M., Wang, Y., et al. (2021). Artificial intelligence-powered microfluidics for nanomedicine and materials synthesis. Nanoscale 13, 19352-19366. View in Article CrossRef Google Scholar
[66]	Wang, X., Xie, P., Chen, B., et al. (2022). Chip-based high-dimensional optical neural network. Nano-Micro Lett. 14, 221. View in Article CrossRef Google Scholar

Cited by

ABOUT THIS ARTICLE

Cite this article:

Dejin Xun, Rui Wang, Xingcai Zhang, Yi Wang. Microsnoop: A generalist tool for microscopy image representation[J]. The Innovation, 2024, 5(1). https://doi.org/10.1016/j.xinn.2023.100541

Dejin Xun, Rui Wang, Xingcai Zhang, Yi Wang. Microsnoop: A generalist tool for microscopy image representation[J]. The Innovation, 2024, 5(1). https://doi.org/10.1016/j.xinn.2023.100541

Welcome!

To request copyright permission to republish or share portions of our works, please visit Copyright Clearance Center's (CCC) Marketplace website at marketplace.copyright.com.

PDF

Figures(6)

Article Metrics

Article views(4116) PDF downloads(1551)

Microsnoop: A generalist tool for microscopy image representation

GRAPHICAL ABSTRACT

PUBLIC SUMMARY

ABSTRACT

REFERENCES

ABOUT THIS ARTICLE

Cite this article:

Welcome!

Share

Article Metrics

Relative Articles

Cited by

Catalog

{{newsColumn.name}}

Official Account

{{newsColumn.name}}

Collections

Stay connected

{{newsColumn.name}}

Microsnoop: A generalist tool for microscopy image representation

GRAPHICAL ABSTRACT

PUBLIC SUMMARY

ABSTRACT

REFERENCES

ABOUT THIS ARTICLE

Cite this article:

Welcome!

Share

Article Metrics

Relative Articles

Cited by

Catalog

Export File

Citation

Format

Content

{{newsColumn.name}}

Official Account

{{newsColumn.name}}

Collections

Stay connected

{{newsColumn.name}}