Research Progress of imaging through scattering media based on the deep learning

doi:10.3969/j.issn.1007-5461.2022.06.005

Abstract

Abstract: Scattering media will change the transmission direction and transmission path of photons, resulting in a decrease in image quality and even forming speckle. Theoretically, information of targets can be recovered by the transmission matrix of the scattering medium, but the process to solve the transmission matrix is too complicated. Then, the rapid development of deep learning provides a new method to solve the problem of imaging through scattering media. As a typical method for solving inverse problems, it can recover the target information accurately, improve the imaging quality, solve many difficult problems, and achieve important research results through scattering imaging. Existing learning methods could be divided into two categories: supervised learning and unsupervised learning. Herein, we summarize the progress of imaging through scattering media based on the two methods, and analyze the advantages and challenges of this technology. Besides, based on the aspects of network structure, imaging quality, generalization and so on, a various of scattering imaging technologies are analyzed and compared. Finally, the development trends are prospected.

Key words: information optics, scattering imaging, target reconstruction, deep learning, scattering media

CLC Number:

TN29

LIN Bing , FAN Xueqiang , LI Dekui , PENG Zhiyong , GUO Zhongyi ∗. Research Progress of imaging through scattering media based on the deep learning[J]. Chinese Journal of Quantum Electronics, 2022, 39(6): 880-898.

References

[1]Xu Q, Guo Z Y, Tao Q Q, Jiao W Y, Qu S L, Gao J.Multi-spectral characteristics of polarization retrieve in various atmospheric conditions[J].Optics Communications, 2015, 339(15):167-170 [2]Hu T W, Shen F, Wang K P, Guo K, Liu X, Wang F, Peng Z Y, Cui Y M, Sun R, Ding Z Z, Gao J, Guo Z Y.Broad-band transmission characteristics of Polarizations in foggy environments[J].Atomosphere, 2019, 10(6):342- [3]Han J, Yang K, Xia M.Resolution enhancement in active underwater polarization imaging with modulation transfer function analysis[J].Applied optics, 2015, 54(11):3294-3302 [4]Huang B J, Liu T G, Hu H F, Han J H, Yu M X.Underwater image recovery considering polarization effects of objects[J].Optics Express, 2016, 24(9):9826-9838 [5]Horstmeyer R, Ruan H W, Yang C H.Guidestar-assisted wavefront-shaping methods for focusing light into biological tissue[J].Nature Photonics, 2015, 9(9):563-571 [6]Li D K, Xu C X, Zhang M, Wang X Y, Guo K, Sun Y X, Gao J, Guo ZY.Measuring glucose concentration in a solution based on the indices of polarimetric purity[J].Biomed. Opt. Express, 2021, 12(4):2447-2459 [7]Shen F, Zhang M, Guo K, Zhou H P, Peng Z Y, Cui Y M, Wang F, Gao J, Guo Z Y.The Depolarization Performances of Scattering Systems Based on Indices of Polarimetric Purity[J].Optics Express, 2019, 20(27):28337-28349 [8]Ntziachristos V.Going deeper than microscopy the optical imaging frontier in biology[J].Nature Methods, 2010, 7(8):603-614 [9]Bertolotti J, van Putten E G, Blum C, et al.Noninvasive imaging through opaque scattering layers[J].Nature, 2012, 491(7423):232-234 [10]Osnabrugge G, Horstmeyer R, Papadopoulos I N.Generalized optical memory effect[J].Optica, 2017, 4(8):886-892 [11]Haskel M, Stern A.Modeling optical memory effects with phase screens[J].Optics express, 2018, 26(22):29231-29243 [12]Yoon J, Lee K R, Park J.Measuring optical transmission matrices by wavefront shaping[J].Optics Express, 2015, 23(8):10158-10167 [13]Drémeau A, Liutkus A, Martina D.Reference-less measurement of the transmission matrix of a highly scattering material using a DMD and phase retrieval techniques[J].Optics express, 2015, 23(9):11898-11911 [14]Paudel H P, Stockbridge C, Mertz J.Focusing polychromatic light through strongly scattering media[J].Optics express, 2013, 21(14):17299-17308 [15]Osnabrugge G, Amitonova L V, Vellekoop I M.Blind focusing through strongly scattering media using wavefront shaping with nonlinear feedback[J].Optics express, 2019, 27(8):11673-11688 [16]Li D K, Xu C X, Yan L S, Guo Z Y.High-performance scanning-mode polarization based computational ghost imaging (SPCGI)[J].Opt. Express, 2022, 30(11):17909-17921 [17]Ferri F, Magatti D, Lugiato L A.Differential ghost imaging[J].Physical review letters, 2010, 104(25):253603- [18]Chen H, Gao Y, Liu X, et al.Imaging through scattering media via support vector regression[J].Optics Communications, 2019, 433(15):126-131 [19]Borhani N, Kakkava E, Moser C.Learning to see through multimode fibers[J].Optica, 2018, 5(8):960-966 [20]Schulein R, Do C M, Javidi B.Distortion-tolerant 3D recognition of underwater objects using neural networks[J].JOSA A, 2010, 27(3):461-468 [21]Frauel Y, Javidi B.Neural network for three-dimensional object recognition based on digital holography[J].Optics letters, 2001, 26(19):1478-1480 [22]Ando T, Horisaki R, Tanida J.Speckle-learning-based object recognition through scattering media[J].Optics Express, 2015, 23(26):33902-33910 [23]Horisaki R, Takagi R, Tanida J.Learning-based imaging through scattering media[J].Optics express, 2016, 24(13):13738-13743 [24]Ronneberger O, Fischer P, Brox T.U-net: Convolutional networks for biomedical image segmentation [C]//International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015: 234-241. [25]Li S, Deng M, Lee J, Sinha A, Barbastathis George.Imaging through glass diffusers using densely connected convolutional networks[J].Optica, 2018, 5(7):803-813 [26]Li Y, Xue Y, Tian L.Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media[J].Optica, 2018, 5(10):1181-1190 [27]Zhu S, Guo E L, Gu J, Bai L F, and Han J, " Imaging through unknown scattering media based on physics-informed learning, " Photon.Res. 9, B210-B219 (2021) [28]Zhu S, Guo E L, Gu J, Cui Q Y, Zhou C Y, Bai L F, and Han J, " Efficient color imaging through unknown opaque scattering layers via physics-aware learning, " Opt.Express 29, 40024-40037 (2021) [29]Lyu M, Wang H, Li G, Zheng S, Situ G H.Learning-based lensless imaging through optically thick scattering media[J].Advanced Photonics, 2019, 1(3):036002- [30]Yang M, Liu Z H, Cheng Z D, Xu J S, Li C F, Guo G C.Deep hybrid scattering image learning[J].Journal of Physics D: Applied Physics, 2019, 52(11):115105- [31]Zheng S S, Wang H, Dong S, Wang F, and Situ G H, " Incoherent imaging through highly nonstatic and optically thick turbid media based on neural network, " Photon.Res. 9, B220-B228 (2021) [32]Sun L, Shi J, Wu X，Sun Y, Zeng G.Photon-limited imaging through scattering medium based on deep learning[J].Optics express, 2019, 27(23):33120-33134 [33]Guo E L, Zhu S, Sun Y, Bai L F, Zuo C, Han J.Learning-based method to reconstruct complex targets through scattering medium beyond the memory effect[J].Optics express, 2020, 28(2):2433-2446 [34]Zhu S et al.“Locating and Imaging through Scattering Medium in a Large Depth.” Sensors (Basel, Switzerland) vol. 21, 1 90. 25 Dec. 2020, doi:10.3390/s21010090 [35]Sun R, Sun X, Chen F, Song Q, Pan H.Polarimetric imaging detection using a convolutional neural network with three-dimensional and two-dimensional convolutional layers[J].Applied optics, 2020, 59(1):151-155 [36]Wang Y, Liu Q, Zu H, Xie R, Wang F. An end-to-end CNN framework for polarimetric vision tasks based on polarization-parameter-constructing network [J]. arXiv prep.[J].rXiv:2004.08740, 2020., rint, :- [37]Hu H F, Zhang Y B, Li X B, Lin Y, Cheng Z Z, Liu T G, Polarimetric underwater image recovery via deep learning, Optics and Lasers in Engineering, Volume 133, 2020, 106152, ISSN 0143-8166, https://doi.org/10.1016/j.optlaseng.2020.106152. [38]Zhang J, Shao J, Luo H, Zhang X, Hui B, Chang Z, Liang R.Learning a convolutional demosaicing network for microgrid polarimeter imagery[J].Optics letters, 2018, 43(18):4534-4537 [39]Zhang J, Shao J, Chen J, Yang D, Liang B, Liang R.PFNet: an unsupervised deep network for polarization image fusion[J].Optics letters, 2020, 45(6):1507-1510 [40]Jiang Huilin, Fu Qiang, Duan Jin, et al.Infrared polarization imaging detection technology and application research[J].Infrared Technology, 2014, 36(5):345-349 [41]姜会林, 付强, 段锦, 等.红外偏振成像探测技术及应用研究[J].红外技术, 2014, 36(5):345-349 [42]Li X, Li H, Lin Y, Guo J, Yang J, Yue H, Li K, Li C, Cheng Z, Hu H, and Liu T, " Learning-based denoising for polarimetric images, " Opt.Express 28, 16309-16321 (2020) [43]Schechner Y.Y, Narasimhan S. G, Nayar S. K. Instant dehazing of images using polarization [C]. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. IEEE, 2001, 1: I-I. [44]Zhang R, Gui X Y, Cheng H Y, and Chu J K, " Underwater image recovery utilizing polarimetric imaging based on neural networks, " Appl.Opt. 60, 8419-8425 (2021) [45]Li D K, Lin B, Wang X Y and Guo Z Y, " High-Performance Polarization Remote Sensing With the Modified U-Net Based Deep-Learning Network, " in IEEE Transactions on Geoscience and Remote Sensing, vol.60, pp. 1-10, 2022, Art no. 5621110, doi: 10.1109/TGRS.2022.3164917. [46]Wang X Y, Hu T W, Li D K, Guo K, Gao J, Guo Z Y.Performances of Polarization-Retrieve Imaging in Stratified Dispersion Media. Remote Sensing. 2020; 12(18):2895. https://doi.org/10.3390/rs12182895 [47]LeCun Y, Bengio Y, and Hinton G, “Deep learning, ” Nature 521(7553), 436–444 (2015). [48]Kang I, Pang S, Zhang Q, Fang N, and Barbastathis G, " Recurrent neural network reveals transparent objects through scattering media, " Opt.Express 29, 5316-5326 (2021) [49]Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, .Courville A, and. Bengio Y, “Generative adversarial nets, ” in Proceedings of the 27th International Conference on Neural Information Processing Systems, vol. 2 (MIT Press, 2014), pp. 2672–2680 [50]Mirza M, Osindero S.Conditional Generative Adversarial Nets[J]. Computer Science, 2014:2672-2680. [51]Ding X Y, Wang Y F, Fu X P.(2022). Multi-polarization fusion generative adversarial networks for clear underwater imaging[J]. Optics and Lasers in Engineering, 2022, 152: 06971. [52]Zhu Y M, Chok H Y, Edmund Y L, Digital holography with deep learning and generative adversarial networks for automatic microplastics classification, Proc.SPIE 11551, 115510A (2020) [53]Sun Y W, Shi J H, Sun L, et al.Image reconstruction through dynamic scattering media based on deep learning[J].Optics Express, 2019, 27(11):16032-16046 [54]Lai X T, Li Q Y, Chen Z Y, et al.Reconstructing images of two adjacent objects passing through scattering medium via deep learning[J].Optics Express, 2021, 29(26):43280-43291 [55]Si L, Huang T Y, Wang X J, et al.Deep learning Mueller matrix feature retrieval from a snapshot Stokes image[J].Optics Express, 2022, 30(6):8676-8689 [56]Zhang Y, Mke A N, Patrik V, et al.PhaseGAN: A deep-learning phase-retrieval approach for unpaired dataset[J].Optics Express, 2021, 29(13):19593-19604 [57]Zhu J Y, Park T, Isola P, et al.Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp:2223–2232 [58]Kaoru Y, Ryochi H, Jun T.Imaging through scattering media based on semi-supervised learning[J].Applied Optics, 2020, 59(31):9850-9854 [59]Reinhard H, Paul H.Deep decoder: Concise image representations from untrained non-convolutional networks. in Proceedings of Seventh International Conference on Learning Representations, (2019). [60]Yang Y, Sun J, Li H, et al.ADMM-CSNet: A deep learning approach for image compressive sensing[J].in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(3):521-538 [61]Yang S L, Qu B W, Liu G S, et al.Unsupervised learning polarimetric underwater image recovery under nonuniform optical fields[J].Applied Optics, 2021, 60(26):8198-8205 [62]Shi Y J, Guo E L, Bai L F, et al.Prior-free imaging unknown target through unknown scattering medium[J].Optics Express, 2022, 30(10):17635-17651 [63]Dmitry U, Andrea V, Victor L.Deep Image Prior. International Journal of Computer Vision, 2020, 128: 1867–1888. [64]Wang F., Bian Y M, Wang H C, et al. Phase imaging with an untrained neural network. Light: Science & Applications, 2022, 9:77. [65]Zhang X Y, Wang F, Situ G H.BlindNet: an untrained learning approach toward computational imaging with model uncertainty[J].Journal of Physics D: Applied Physics, 2022, 55(3):034001- [66]Zhu Y M, Zeng T J, Liu K W, et al.Full scene underwater imaging with polarization and an untrained network[J].Optics Express, 2021, 29(25):41865-41881 [67]Shi Y J, Guo E L, Bai L F, et al.Polarization-Based Haze Removal Using Self-Supervised Network. Frontiers in Physics, 2022, 9: 789232. [68]Sinno J P, Yang Q.A survey on transfer learning[J].in IEEE Transactions on Knowledge and Data Engineering, 2021, 22(10):1345-1359 [69]Ashish V, Noam S, Niki P, et al.Attention is all you need. in the 31st International Conference on Neural Information Processing Systems, 2017, pp:6000–6010