Extended depth-of-field resolution enhancement microscopy imaging for neutralizing the impact of mineral inhomogeneous surface

Heng Sun; Xinran Xu; Qianxiong Shi; Junzhang Chen; Darui Jin; Yan Li; Dong Ye; Yong Lai; Xiangzhi Bai

doi:10.59717/j.xinn-geo.2024.100083

The Innovation Geoscience > 2024 Vol. 2 > No. 3 > 100083

Next Article Previous Article

ARTICLE Open Access

Cite PDF

Extended depth-of-field resolution enhancement microscopy imaging for neutralizing the impact of mineral inhomogeneous surface

1.
Image Processing Center, Beihang University, Beijing 102206, China
2.
The Key Laboratory of Orogenic Belts and Crustal Evolution, Peking University, Beijing 100871, China
3.
School of Art and Design, Shenzhen Polytechnic University, Shenzhen 518055, China
4.
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China
5.
Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100083, China
6.
These authors contributed equally

Show all affliationsShow less

More Information

Corresponding authors: yonglai@pku.edu.cn (Y. L.); jackybxz@buaa.edu.cn (X. B.)
Received Date: September 13, 2023

Accepted Date: June 21, 2024

Published Online: July 01, 2024

The Innovation Geoscience 2(3), https://doi.org/10.59717/j.xinn-geo.2024.100083 | Cite this article

GRAPHICAL ABSTRACT

GRAPHICAL ABSTRACT

DownLoad: Full size image

PUBLIC SUMMARY

PUBLIC SUMMARY
1. Mineral uneven surface aggravates conflict between depth-of-field (DOF) and resolution in microscopy imaging.
  
  DOF from low-resolution image is revealed as oracles for neutralizing impact of mineral inhomogeneous surface.
  
  Optically induced generative adversarial network (OIGAN) is proposed by integrating DOF information from minerals.
  
  OIGAN significantly improves the performance in microthermometry of fluid inclusions and mineral classification.

ABSTRACT

ABSTRACT

One of the most fundamental experimental methods in geoscience is to observe minerals under high magnification objectives. However, uneven microsurfaces in thin sections occur due to the irregular constituent distribution and varying hardness of minerals in natural rocks. Consequently, the conflict between large depth-of-field (DOF) and high-resolution in microscopy imaging leads to random out-of-focus issues when observing thin sections with high resolution microscopy. Although existing super-resolution algorithms promise to improve visual performance, reconstructing images with both large DOF and high-resolution simultaneously remains challenging. We address this problem by guiding the networks with optical information. Utilizing DOF information from low-resolution data, we propose an optically induced generative adversarial network (OIGAN) to neutralize the impact through computational imaging. In OIGAN, optical DOF information from low-resolution data facilitates to achieve spatial-adaptive extended-DOF resolution enhancement imaging, without incorporating extended DOF high-resolution data for supervision. The approach, trained and evaluated on the dataset with 233,156 images (115,346 pairs of low- and high-resolution data), outperforms four comparison methods on various minerals and optical conditions, leading to at least 1.54dB increase on peak signal-to-noise ratio (PSNR). Specifically, OIGAN significantly improves the accuracy of fluid inclusion ice-melting temperature measurement, reducing mean error by 65%, and enhances mineral classification accuracy with 1.5%~15% increase. OIGAN offers an insight of integrating physical knowledge into neural networks, facilitating self-identification of minerals, automatic microthermometry of fluid inclusions and other geoscience tasks via microscopy.

FullText(HTML)

REFERENCES

[1]	Mitchell, R.N. (2021). Chang’E-5 reveals the Moon's secrets to a longer life. The Innovation 2: 100177. DOI: 10.1016/j.xinn.2021.100177. View in Article CrossRef Google Scholar
[2]	Chen, Y., Hu, S., Li, J.-H., et al. (2023). Chang’e-5 lunar samples shed new light on the Moon. The Innovation Geoscience 1: 100014. DOI: 10.59717/j.xinn-geo.2023.100014. View in Article CrossRef Google Scholar
[3]	Hell, S.W. and Wichmann, J. (1994). Breaking the diffraction resolution limit by stimulated-emission—stimulated-emission-depletion fluorescence microscopy. Opt. Lett. 19: 780−782. DOI: 10.1364/ol.19.000780. View in Article CrossRef Google Scholar
[4]	Gustafsson, M.G.L. (2000). Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc-Oxford 198: 82−87. DOI: 10.1046/j.1365-2818.2000.00710.x. View in Article CrossRef Google Scholar
[5]	Gustafsson, M.G.L. (2005). Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution. Proc. Natl. Acad. Sci. USA 102: 13081−13086. DOI: 10.1073/pnas.0406877102. View in Article CrossRef Google Scholar
[6]	Rust, M.J., Bates, M., and Zhuang, X. (2006). Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3: 793−796. DOI: 10.1038/nmeth929. View in Article CrossRef Google Scholar
[7]	Wang, Z., Zhao, T., Cai, Y., et al. (2023). Rapid, artifact-reduced, image reconstruction for super-resolution structured illumination microscopy. The Innovation 4: 100425. DOI: 10.1016/j.xinn.2023.100425. View in Article CrossRef Google Scholar
[8]	Schultz, R.R. and Stevenson, R.L. (1994). A Bayesian approach to image expansion for improved definition. IEEE T. Image Process 3: 233−242. DOI: 10.1109/83.287017. View in Article CrossRef Google Scholar
[9]	Schultz, R.R. and Stevenson, R.L. (1996). Extraction of high-resolution frames from video sequences. IEEE T. Image Process 5: 996−1011. DOI: 10.1109/83.503915. View in Article CrossRef Google Scholar
[10]	Chang, H., Yeung, D.-Y., and Xiong, Y. (2004). Super-resolution through neighbor embedding. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. DOI: 10.1109/CVPR.2004.1315043. View in Article Google Scholar
[11]	Kim, K.I., Franz, M.O., and Scholkopf, B. (2005). Iterative kernel principal component analysis for image modeling. IEEE T. Pattern Anal. 27: 1351−1366. DOI: 10.1109/TPAMI.2005.181. View in Article CrossRef Google Scholar
[12]	Yang, J., Wright, J., Huang, T.S., et al. (2010). Image super-resolution via sparse representation. IEEE T. Image Process 19: 2861−2873. DOI: 10.1109/TIP.2010.2050625. View in Article CrossRef Google Scholar
[13]	Dowski, E.R. and Cathey, W.T. (1995). Extended depth of field through wave-front coding. Appl. Opt. 34: 1859−1866. DOI: 10.1364/AO.34.001859. View in Article CrossRef Google Scholar
[14]	Botcherby, E.J., Booth, M.J., Juškaitis, R., et al. (2008). Real-time extended depth of field microscopy. Opt. Express 16: 21843−21848. DOI: 10.1364/OE.16.021843. View in Article CrossRef Google Scholar
[15]	Jin, L., Tang, Y., Wu, Y., et al. (2020). Deep learning extended depth-of-field microscope for fast and slide-free histology. Proc. Natl. Acad. Sci. USA 117: 33051−33060. DOI: 10.1073/pnas.201357111. View in Article CrossRef Google Scholar
[16]	Wu, Y., Rivenson, Y., Wang, H., et al. (2019). Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat. Methods 16: 1323−1331. DOI: 10.1038/s41592-019-0622-5. View in Article CrossRef Google Scholar
[17]	Luo, Y., Huang, L., Rivenson, Y., et al. (2021). Single-shot autofocusing of microscopy images using deep learning. ACS Photonics 8: 625−638. DOI: 10.1021/acsphotonics.0c01774. View in Article CrossRef Google Scholar
[18]	Dong, C., Loy, C.C., He, K., et al. (2015). Image super-resolution using deep convolutional networks. IEEE T. Pattern Anal. 38: 295−307. DOI: 10.1109/DSC.2016.104. View in Article CrossRef Google Scholar
[19]	Zhang, Y., Li, K., Li, K., et al. (2018). Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV). DOI: 10.1007/978-3-030-01234-2_18. View in Article Google Scholar
[20]	Liang, J., Cao, J., Sun, G., et al. (2021). Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. DOI: 10.1109/ICCVW54120.2021.00210. View in Article Google Scholar
[21]	Ma, C., Rao, Y., Cheng, Y., et al. (2020). Structure-preserving super resolution with gradient guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. DOI: 10.1109/CVPR42600.2020.00779. View in Article Google Scholar
[22]	Qiao, C., Li, D., Guo, Y., et al. (2021). Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18: 194−202. DOI: 10.1038/s41592-020-01048-5. View in Article CrossRef Google Scholar
[23]	Wang, H., Rivenson, Y., Jin, Y., et al. (2019). Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16: 103−110. DOI: 10.1038/s41592-018-0239-0. View in Article CrossRef Google Scholar
[24]	Qiao, C., Li, D., Liu, Y., et al. (2023). Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat. Biotechnol. 41: 367−377. DOI: 10.1038/s41587-022-01471-3. View in Article CrossRef Google Scholar
[25]	Wu, Y., Han, X., Su, Y., et al. (2021). Multiview confocal super-resolution microscopy. Nature 600: 279−284. DOI: 10.1038/s41586-021-04110-0. View in Article CrossRef Google Scholar
[26]	Zhang, B., Ouyang, C., Cui, P., et al. (2024). Deep learning for cross-region streamflow and flood forecasting at a global scale. The Innovation 5: 100617. DOI: 10.1016/j.xinn.2024.100617. View in Article CrossRef Google Scholar
[27]	Yuan, X. and Han, S. (2021). Single-pixel neutron imaging with artificial intelligence: Breaking the barrier in multi-parameter imaging, sensitivity, and spatial resolution. The Innovation 2: 100100. DOI: 10.1016/j.xinn.2021.100100. View in Article CrossRef Google Scholar
[28]	Ye, Z., Wang, N., Zhou, J., et al. (2024). Organic crystal structure prediction via coupled generative adversarial networks and graph convolutional networks. The Innovation 5: 100562. DOI: 10.1016/j.xinn.2023.100562. View in Article CrossRef Google Scholar
[29]	Li, S. and Xing, J. (2024). DeepSAT4D: Deep learning empowers four-dimensional atmospheric chemical concentration and emission retrieval from satellite. The Innovation Geoscience 2: 100061−100011. DOI: 10.59717/j.xinn-geo.2024.100061. View in Article CrossRef Google Scholar
[30]	Wang, Z., Zhang, J., Hua, P., et al. (2023). Filling in missing pieces in the co-development of artificial intelligence and environmental science. The Innovation Geoscience 1: 100007−100015. DOI: 10.59717/j.xinn-geo.2023.100007. View in Article CrossRef Google Scholar
[31]	Ma, H., Jiao, Y., Guo, W., et al. (2024). Machine learning predicts atomistic structures of multielement solid surfaces for heterogeneous catalysts in variable environments. The Innovation 5: 100571. DOI: 10.1016/j.xinn.2024.100571. View in Article CrossRef Google Scholar
[32]	Xun, D., Wang, R., Zhang, X., et al. (2024). Microsnoop: A generalist tool for microscopy image representation. The Innovation 5: 100541. DOI: 10.1016/j.xinn.2023.100541. View in Article CrossRef Google Scholar
[33]	Zhu, R., Zhang, F., Yan, J., et al. (2023). A sustainable solar city: From utopia to reality facilitated by GIScience. The Innovation Geoscience 1: 100006−100013. DOI: 10.59717/j.xinn-geo.2023.100006. View in Article CrossRef Google Scholar
[34]	Luo, L., Zong, X., Shao, J., et al. (2023). LiDARchaeology: A revolutionary approach to archaeological prospection in forested areas. The Innovation Geoscience 1: 100017. DOI: 10.59717/j.xinn-geo.2023.100017. View in Article CrossRef Google Scholar
[35]	Lü, G., Yue, S., Yu, Z., et al. (2023). Ubiquitous geographic information for building digital twins of geographic environments. The Innovation Geoscience 1: 100023−100022. DOI: 10.59717/j.xinn-geo.2023.100023. View in Article CrossRef Google Scholar
[36]	Gomariz, A., Portenier, T., Helbling, P.M., et al. (2021). Modality attention and sampling enables deep learning with heterogeneous marker combinations in fluorescence microscopy. Nat. Mach. Intell. 3: 799−811. DOI: 10.1038/s42256-021-00379-y. View in Article CrossRef Google Scholar
[37]	Ziatdinov, M., Ghosh, A., Wong, C.Y., et al. (2022). AtomAI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy. Nat. Mach. Intell. 4: 1101−1112. DOI: 10.1038/s42256-022-00555-8. View in Article CrossRef Google Scholar
[38]	Kench, S. and Cooper, S.J. (2021). Generating three-dimensional structures from a two-dimensional slice with generative adversarial network-based dimensionality expansion. Nat. Mach. Intell. 3: 299−305. DOI: 10.1038/s42256-021-00322-1. View in Article CrossRef Google Scholar
[39]	Ledig, C., Theis, L., Huszár, F., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR.2017.19. View in Article Google Scholar
[40]	Jiang, K., Wang, Z., Yi, P., et al. (2019). Edge-enhanced GAN for remote sensing image superresolution. IEEE Transactions on Geoscience and Remote Sensing 57: 5799−5812. DOI: 10.1109/TGRS.2019.2902431. View in Article CrossRef Google Scholar
[41]	Zhang, W., Liu, Y., Dong, C., et al. (2019). RankSRGAN: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. DOI: 10.1109/ICCV.2019.00319. View in Article Google Scholar
[42]	Jiang, K., Wang, Z., Yi, P., et al. (2020). Hierarchical dense recursive network for image super-resolution. Pattern Recog. 107: 107475. DOI: 10.1016/j.patcog.2020.107475. View in Article CrossRef Google Scholar
[43]	Xiao, Y., Su, X., Yuan, Q., et al. (2022). Satellite video super-resolution via multiscale deformable convolution alignment and temporal grouping projection. IEEE T. Geoscience and Remote Sensing 60: 1−19. DOI: 10.1109/TGRS.2021.3107352. View in Article CrossRef Google Scholar
[44]	Hu, M., Jiang, K., Wang, Z., et al. (2023). CycMuNet+: Cycle-projected mutual learning for spatial-temporal video super-resolution. IEEE T. Pattern Analysis and Machine Intelligence 45: 13376−13392. DOI: 10.1109/TPAMI.2023.3293522. View in Article CrossRef Google Scholar
[45]	Xiao, Y., Yuan, Q., Jiang, K., et al. (2023). From degrade to upgrade: Learning a self-supervised degradation guided adaptive network for blind remote sensing image super-resolution. Inform. Fusion 96: 297−311. DOI: 10.1016/j.inffus.2023.03.021. View in Article CrossRef Google Scholar
[46]	Qiu, X., Li, M., Zhang, L., et al. (2019). Guided filter-based multi-focus image fusion through focus region detection. Signal Processing: Image Communication 72: 35−46. DOI: 10.1016/j.image.2018.12.004. View in Article CrossRef Google Scholar
[47]	Dai, Y., Gieseke, F., Oehmcke, S., et al. (2021). Attentional feature fusion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. DOI: 10.1109/WACV48630.2021.00360. View in Article Google Scholar
[48]	Li, W., Zhou, K., Qi, L., et al. (2022). Best-buddy gans for highly detailed image super-resolution. In Proceedings of the AAAI Conference on Artificial Intelligence. DOI: 10.1609/aaai.v36i2.20030. View in Article Google Scholar
[49]	Huynh-Thu, Q. and Ghanbari, M. (2008). Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 44: 800−801. DOI: 10.1049/el:20080522. View in Article CrossRef Google Scholar
[50]	Zhang, R., Isola, P., Efros, A.A., et al. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. DOI: 10.1109/CVPR.2018.00068. View in Article Google Scholar
[51]	Heusel, M., Ramsauer, H., Unterthiner, T., et al. (2017). GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems. DOI: 10.5555/3295222.3295408. View in Article Google Scholar
[52]	Blau, Y., Mechrez, R., Timofte, R., et al. (2018). The 2018 PIRM Challenge on Perceptual Image Super-Resolution. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. DOI: 10.1007/978-3-030-11021-5_21. View in Article Google Scholar
[53]	Wang, X., Yu, K., Wu, S., et al. (2018). ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. DOI: 10.1007/978-3-030-11021-5_5. View in Article Google Scholar
[54]	Prajapati, K., Chudasama, V., Patel, H., et al. (2020). Unsupervised single image super-resolution network (USISResNet) for real-world data using generative adversarial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. DOI: 10.1109/CVPRW50498.2020.00240. View in Article Google Scholar
[55]	Prajapati, K., Chudasama, V., Patel, H., et al. (2021). Direct unsupervised super-resolution using generative adversarial network (DUS-GAN) for real-world data. IEEE T. Image Process. 30: 8251−8264. DOI: 10.1109/TIP.2021.3113783. View in Article CrossRef Google Scholar
[56]	Marr, D. and Hildreth, E. (1980). Theory of Edge Detection. P. Roy. Society B-Biol. Sci. 207: 187−217. DOI: 10.1098/rspb.1980.0020. View in Article CrossRef Google Scholar
[57]	Paris, S., Hasinoff, S.W., and Kautz, J. (2011). Local Laplacian filters: Edge-aware image processing with a Laplacian pyramid. In SIGGRAPH '11: Special Interest Group on Computer Graphics and Interactive Techniques Conference. DOI: 10.1145/1964921.1964963. View in Article Google Scholar
[58]	Kong, H., Akakin, H.C., and Sarma, S.E. (2013). A generalized laplacian of gaussian filter for blob detection and its applications. IEEE T. Cybernetics 43: 1719−1733. DOI: 10.1109/TSMCB.2012.2228639. View in Article CrossRef Google Scholar
[59]	Jin, D., Chen, Y., Lu, Y., et al. (2021). Neutralizing the impact of atmospheric turbulence on complex scene imaging via deep learning. Nat. Mach. Intell. 3: 876−884. DOI: 10.1038/s42256-021-00392-1. View in Article CrossRef Google Scholar
[60]	Bradley, R.A. and Terry, M.E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39: 324−345. DOI: 10.2307/2334029. View in Article CrossRef Google Scholar
[61]	Maritz, M.G. and Schoeman, J. (2022). Programmable aperture using a digital micromirror device for in-line holographic microscopy. IEEE J. Quantum Elect. 58: 1−8. DOI: 10.1109/JQE.2022.3190501. View in Article CrossRef Google Scholar
[62]	Yang, J., Zhang, Z., and Cheng, Q. (2022). Resolution enhancement in micro-XRF using image restoration techniques. J. Anal. Atom. Spectrom. 37: 750−758. DOI: 10.1039/D1JA00425E. View in Article CrossRef Google Scholar
[63]	Shi, Q., Lai, Y., Guo, H., et al. (2021). Fluid inclusion and C-O isotopic constrains on the origin and evolution of ore-forming fluids of the Badaguan Cu-Mo deposit, Inner Mongolia. Ore Geol. Rev. 136: 104267. DOI: 10.1016/j.oregeorev.2021.104267. View in Article CrossRef Google Scholar

Cited by

ABOUT THIS ARTICLE

Cite this article:

Sun H., Xu X., Shi Q., et al., (2024). Extended depth-of-field resolution enhancement microscopy imaging for neutralizing the impact of mineral inhomogeneous surface. The Innovation Geoscience 2(3): 100083. https://doi.org/10.59717/j.xinn-geo.2024.100083

Sun H., Xu X., Shi Q., et al., (2024). Extended depth-of-field resolution enhancement microscopy imaging for neutralizing the impact of mineral inhomogeneous surface. The Innovation Geoscience 2(3): 100083. https://doi.org/10.59717/j.xinn-geo.2024.100083

Welcome!

To request copyright permission to republish or share portions of our works, please visit Copyright Clearance Center's (CCC) Marketplace website at marketplace.copyright.com.

Standard PDF

Extended PDF

Figures(7) Tables(1)

Article Metrics

Article views(5219) PDF downloads(217)

Extended depth-of-field resolution enhancement microscopy imaging for neutralizing the impact of mineral inhomogeneous surface

GRAPHICAL ABSTRACT

PUBLIC SUMMARY

ABSTRACT

REFERENCES

ABOUT THIS ARTICLE

Cite this article:

Welcome!

Share

Article Metrics

Relative Articles

Cited by

Catalog

{{newsColumn.name}}

Official Account

{{newsColumn.name}}

Stay connected

{{newsColumn.name}}

Extended depth-of-field resolution enhancement microscopy imaging for neutralizing the impact of mineral inhomogeneous surface

GRAPHICAL ABSTRACT

PUBLIC SUMMARY

ABSTRACT

REFERENCES

ABOUT THIS ARTICLE

Cite this article:

Welcome!

Share

Article Metrics

Relative Articles

Cited by

Catalog

Export File

Citation

Format

Content

{{newsColumn.name}}

Official Account

{{newsColumn.name}}

Stay connected

{{newsColumn.name}}