Research progress of image registration methods based on deep learning

doi:10.3969/j.issn.1007-5461.2022.06.006

Abstract

Abstract: In recent years, the rapid development of image acquisition equipment has greatly enriched the types and quantities of images. As the key of image analysis and processing, image registration technology has become increasingly important in the fields of image fusion, pattern recognition and computer vision, and how to register images with high accuracy and in real time has become the focus of research. At the same time, deep learning techniques shine, and convolutional neural networks show unique advantages in image representation and feature extraction. The aim is to provide a systematic review of research on image registration using deep learning techniques. By discussing typical deep learning-based image registration methods from deep iterative registration, fully supervised image registration, weak/dually supervised image registration, and unsupervised image registration, we highlight common challenges faced by related researchers and explore possible future research directions to address these challenges.

Key words: image processing, image registration, deep learning, convolutional neural network

CLC Number:

TP391.41

CHEN Jianming , , ZENG Xiangjin , , ZHONG Liyun , , DI Jianglei , ∗ , QIN Yuwen , ∗. Research progress of image registration methods based on deep learning[J]. Chinese Journal of Quantum Electronics, 2022, 39(6): 899-926.

References

[1]Zitová B, Flusser J.Image registration methods: a survey[J].Image and Vision Computing, 2003, 21(11):977-1000 [2]Mao Xiaoying, Yi Mengxiao, et al.A survey of deep learning methods for medical image registration[J].Journal of Chinese Computer Systems, 2021, 42(08):1706-1714 [3]莫晓盈, 杨锋, 尹梦晓, 等.医学图像配准的深度学习方法综述[J].小型微型计算机系统, 2021, 42(08):1706-1714 [4]Lester H, Arridge S R.A survey of hierarchical non-linear medical image registration[J].Pattern Recognition, 1999, 32(1):129-149 [5]Goshtasby A.Registration of images with geometric distortions[J].IEEE Transactions on Geoscience and Remote Sensing, 1988, 26(1):60-64 [6]Friston K J, Ashburner J, Frith C D, et al.Spatial registration and normalization of images[J].Human brain mapping, 1995, 3(3):165-189 [7]Wells W M, Viola P, Atsumi H, et al.Multi-modal volume registration by maximization of mutual information[J].Medical Image Analysis, 1996, 1(1):35-51 [8]Studholme C, Drapaca C, Iordanova B, et al.Deformation-based mapping of volume change from serial brain MRI in the presence of local tissue contrast change[J].IEEE Transactions on Medical Imaging, 2006, 25(5):626-639 [9]Balntas V, Johns E, Tang L, et al. PN-Net: conjoined triple deep network for learning local image descriptors[J]. arXiv e-prints, 2.[J].rXiv:1601.05030., 016:, :- [10] Kuglin C D, Hines D C.The phase correlation image registration method[C], Conference Cybernetics Society, 1975: 163–165. [11]Reddy B S, Chatterji B N.An FFT-based technique for translation,rotation,and scale-invariant image registration[J].IEEE transactions on image processing, 1996, 5(8):1266-1271 [12] Rocco I, Arandjelovic R, Sivic J.Convolutional neural network architecture for geometric matching[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 39-48. [13] Chauhan R, Ghanshala K K, Joshi R C.Convolutional Neural Network (CNN) for image detection and recognition[C]. 2018 First International Conference on Secure Cyber Computing and Communication (ICSCCC), 2018: 278-282. [14]Long J, Shelhamer E, Darrell T.Fully convolutional networks for semantic segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39(4):640-651 [15] Liu C, Shang Z, Qin A.A multiscale image denoising algorithm based on dilated residual convolution network[C]. Image and Graphics Technologies and Applications, 2019: 193-203. [16]Ren S, He K, Girshick R, et al.Faster R-CNN: towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149 [17]LeCun Y, Boser B, Denker J S, et al.Backpropagation applied to handwritten zip code recognition[J].Neural computation, 1989, 1(4):541-551 [18]LeCun Y, Bottou L, Bengio Y, et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE, 1998, 86(11):2278-2324 [19] Gu J, Wang Z, Kuen J, et al.Recent advances in convolutional neural networks[J].[J]. Pattern Recognition, 2018, 77(*):354-377 [20] Krizhevsky A, Sutskever I, Hinton G E.ImageNet classification with deep convolutional neural networks[J]. [J].Advances in neural information processing systems, , 2017, 60(*):84-90 [21]Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv e-prints, 2.[J].rXiv:1409.1556., 014:, :- [22] Szegedy C, Wei L, Yangqing J, et al.Going deeper with convolutions[C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 1-9. [23] He K, Zhang X, Ren S, et al.Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 770-778. [24] Ronneberger O, Fischer P, Brox T.U-Net: convolutional networks for biomedical image segmentation[C]. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, 2015: 234-241. [25] Goodfellow I J, Pouget-Abadie J, Mirza M, et al.Generative adversarial networks[J]. [J].Advances in Neural Information Processing Systems, 2014, 3:(*):2672-2680 [26] Vaswani A, Shazeer N, Parmar N, et al.Attention is all you need[C]. Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000–6010. [27] Wu G, Kim M, Wang Q, et al.Unsupervised deep feature learning for deformable registration of MR brain images[C]. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013, 2013: 649-656. [28]Cheng X, Zhang L, Zheng Y.Deep similarity learning for multimodal medical images[J].Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 2018, 6(3):248-252 [29] Sedghi A, O’donnell L J, Kapur T, et al.Image registration: maximum likelihood, minimum entropy and deep learning[J]. [J].Medical Image Analysis, , 2021, 69(*):101939-101939 [30] Simonovsky M, Gutiérrez-Becker B, Mateus D, et al.A deep metric for multimodal registration[C]. Medical Image Computing and Computer-Assisted Intervention-MICCAI 2016, 2016: 10-18. [31]Czolbe S, Krause O, Feragen A. DeepSim: semantic similarity metrics for learned image registration[J]. arXiv e-prints, 2.[J].rXiv:2011.05735., 020:, :- [32] Sloan J M, Goatman K A, Siebert J P.Learning rigid image registration-utilizing convolutional neural networks for Medical Image Registration[C]. BIOIMAGING, 2018. [33]Miao S, Wang Z J, Liao R.A CNN regression approach for real-time 2D3D registration[J].IEEE Transactions on Medical Imaging, 2016, 35(5):1352-1363 [34] Miao S, Wang Z J, Zheng Y, et al.Real-time 2D/3D registration via CNN regression[C]. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), 2016: 1430-1434. [35]Salehi S S M, Khan S, Erdogmus D, et al.Real-time deep pose estimation with geodesic loss for image-to-template rigid registration[J].IEEE Transactions on Medical Imaging, 2019, 38(2):470-481 [36]Zou M, Hu J, Zhang H, et al.Rigid medical image registration using learning-based interest points and features[J].Computers, Materials & Continua, 2019, 60(2):511-525 [37]Chee E, Wu Z. AIRNet: Self-supervised affine registration for 3D medical images using neural networks[J]. arXiv e-prints, 2.[J].rXiv:1810.02583., 018:, :- [38]Jiannan Z, Shun M, Wang Z J, et al.Pairwise domain adaptation module for CNN-based 2-D3-D registration[J].Journal of Medical Imaging, 2018, 5(2):1-10 [39] Guo H, Kruger M, Xu S, et al.Deep adaptive registration of multi-modal prostate images[J]. [J].Computerized Medical Imaging and Graphics, 2020, 84: (*):101769-101769 [40] Yang X, Kwitt R, Niethammer M.Fast predictive image registration[C]. Deep Learning and Data Labeling for Medical Applications, 2016: 48-57. [41] Cao X, Yang J, Zhang J, et al.Deformable image registration based on similarity-steered CNN regression[C]. Medical Image Computing and Computer Assisted Intervention?MICCAI 2017, 2017: 300-308. [42]Cao X, Yang J, Zhang J, et al.Deformable image registration using a cue-aware deep regression network[J].IEEE Transactions on Biomedical Engineering, 2018, 65(9):1900-1911 [43]Teng X, Y Chen, Y Zhang, et al.Respiratory deformation registration in 4D-CTcone beam CT using deep learning[J].Quantitative imaging in medicine and surgery, 2020, 11(2):737-748 [44]Koen A J E, Josien P W P.Error estimation of deformable image registration of pulmonary CT scans using convolutional neural networks[J].Journal of Medical Imaging, 2018, 5(2):1-11 [45] Sokooti H, De Vos B, Berendsen F, et al.Nonrigid image registration using multi-scale 3D convolutional neural networks[C]. Medical Image Computing and Computer Assisted Intervention ? MICCAI 2017, 2017: 232-239. [46]Sokooti H, De Vos B, Berendsen F, et al. 3D convolutional neural networks image registration based on efficient supervised learning from artificial deformations[J]. arXiv e-prints, 2.[J].rXiv:1908.10235., 019:, :- [47] Fu Y, Lei Y, Wang T, et al.Biomechanically constrained non-rigid MR-TRUS prostate registration using deep learning based 3D point cloud matching[J]. [J].Medical Image Analysis, 2021, 67(*):101845-101845 [48] Hu Y, Modat M, Gibson E, et al.Label-driven weakly-supervised learning for multimodal deformable image registration[C]. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018: 1070-1074. [49]Blendowski M, Hansen L, Heinrich M P.Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration[J].Medical Image Analysis, 2021, 67(8):101822-101822 [50] Hering A, Kuckertz S, Heldmann S, et al.Enhancing label-driven deep deformable image registration with local distance metrics for state-of-the-Art cardiac motion tracking[C]. Bildverarbeitung für die Medizin 2019, 2019: 309-314. [51] Shao W, Bhattacharya I, Soerensen S J C, et al.Weakly supervised registration of prostate MRI and histopathology images[C]. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, 2021: 98-107. [52]Zhu Q, Sun Y, Wu Y, et al.Whole-brain functional MRI registration based on a semi-supervised deep learning model[J].Medical Physics, 2021, 48(6):2847-2858 [53]Zhu Z, Cao Y, Qin C, et al.Joint affine and deformable three-dimensional networks for brain MRI registration[J].Medical Physics, 2021, 48(3):1182-1196 [54] Wang Y, Zhang J, Cavichini M, et al.Robust content-adaptive global registration for multimodal retinal images using weakly supervised deep-learning framework[J].[J]. IEEE Transactions on Image Processing, 2021, 30(*):3167-3178 [55] Yunzhen P, Xinjian C, Dehui X, et al.Keypoint matching networks for longitudinal fundus image affine registration[C]. Proc.SPIE, 2021. [56]Wang X, Mao L, Huang X, et al.Multimodal MR image registration using weakly supervised constrained affine network[J].Journal of Modern Optics, 2021, 68(13):679-688 [57]Lei Y, Fu Y, Wang T, et al.D-CT deformable image registration using multiscale unsupervised deep learning[J].Physics in Medicine & Biology, 2020, 65(8):085003-085003 [58] Hu Y, Gibson E, Ghavami N, et al.Adversarial deformation regularization for training image registration neural networks[C]. Medical Image Computing and Computer Assisted Intervention–MICCAI 2018, 2018: 774-782. [59] Chen X, Xia Y, Ravikumar N, et al.A deep discontinuity-preserving image registration network[C]. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, 2021: 46-55. [60]Ma Lufan, Luo Feng, Yan Jiangpeng, et al.Advances in deep medical image registration: towards unsupervised learning[J].Journal of Image and Graphics, 2021, 26(09):2037-2057 [61]马露凡, 罗凤, 严江鹏, 等.深度医学图像配准研究进展: 迈向无监督学习[J].中国图象图形学报, 2021, 26(09):2037-2057 [62] Fan J, Cao X, Yap P-T, et al.BIRNet: brain image registration using dual-supervised fully convolutional networks[J]. [J].Medical Image Analysis, 2019, 54(*):193-206 [63] Cao X, Yang J, Wang L, et al.Deep learning based inter-modality image registration supervised by intra-modality similarity[C]. Machine Learning in Medical Imaging, 2018: 55-63. [64] Yan P, Xu S, Rastinehad A R, et al.Adversarial image registration with application for MR and TRUS image fusion[C]. Machine Learning in Medical Imaging, 2018: 197-204. [65] Arjovsky M, Chintala S, Bottou L.Wasserstein generative adversarial networks[C]. Proceedings of the 34th International Conference on Machine Learning, 2017: 214--223. [66]Qiu L, Ren H.RSegNet: A joint learning framework for deformable registration and segmentation[J].IEEE Transactions on Automation Science and Engineering, 2022, 19(3):2499-2513 [67] Jaderberg M, Simonyan K, Zisserman A, et al.Spatial transformer networks[C]. Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2, 2015: 2017–2025. [68] De Vos B D, Berendsen F F, Viergever M A, et al.End-to-End unsupervised deformable image registration with a convolutional neural network[C]. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2017: 204-212. [69] De Vos B D, Berendsen F F, Viergever M A, et al.A deep learning framework for unsupervised affine and deformable image registration[J]. [J].Medical Image Analysis, 2019, 52(*):128-143 [70] Balakrishnan G, Zhao A, Sabuncu M R, et al.An unsupervised learning model for deformable medical image registration[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 9252-9260. [71]Avants B B, Epstein C L, Grossman M, et al.Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain[J].Medical Image Analysis, 2008, 12(1):26-41 [72]Balakrishnan G, Zhao A, Sabuncu M R, et al.VoxelMorph: a learning framework for deformable medical image registration[J].IEEE Transactions on Medical Imaging, 2019, 38(8):1788-1800 [73] Zhao S, Dong Y, Chang E, et al.Recursive cascaded networks for unsupervised medical image registration[C]. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019: 10599-10609. [74] Zhang L, Zhou L, Li R, et al.Cascaded feature warping network for unsupervised medical image registration[C]. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 2021: 913-916. [75] Dalca A V, Balakrishnan G, Guttag J, et al.Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces[J]. [J].Medical Image Analysis, 2019, 57(*):226-236. [76] Dalca A V, Balakrishnan G, Guttag J, et al.Unsupervised learning for fast probabilistic diffeomorphic registration[C]. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, 2018: 729-738. [77] Mok T C W, Chung A C S.Fast symmetric diffeomorphic image registration with convolutional neural networks[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 4643-4652. [78]Chen Q, Li Z, Lui L M. A learning framework for diffeomorphic image registration based on quasi-conformal geometry[J]. arXiv e-prints, 2.[J].rXiv:2110.10580., 021:, :- [79] Kim B, Kim D H, Park S H, et al.CycleMorph: cycle consistent unsupervised deformable image registration[J]. [J].Medical Image Analysis, , 2021, 71(*):102036-102036 [80]Xu Z, Luo J, Yan J, et al.F3RNet: full-resolution residual registration network for deformable image registration[J].International Journal of Computer Assisted Radiology and Surgery, 2021, 16(6):923-932 [81] Han L, Dou H, Huang Y, et al.Deformable registration of brain MR images via a hybrid loss[C]. Biomedical Image Registration, Domain Generalisation and Out-of-Distribution Analysis, 2022: 141-146. [82]Kori A, Krishnamurthi G. Zero Shot Learning for multi-modal real time image registration[J]. arXiv e-prints, 2.[J].rXiv:1908.06213., 019:, :- [83] Xu Z, Yan J, Luo J, et al.Unsupervised multimodal image registration with adaptative gradient guidance[C]. ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021: 1225-1229. [84] Sideri-Lampretsa V, Kaissis G, Rueckert D.Multi-modal unsupervised brain image registration using edge maps[C]. 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 2022: 1-5. [85] Pluim J P W, Maintz J B A, Viergever M A.Image registration by maximization of combined mutual information and gradient information[C]. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2000, 2000: 452-461. [86] Tian Y, Hu Y, Ma Y, et al.Multi-scale U-net with edge guidance for multimodal retinal image deformable registration[C]. 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2020: 1360-1363. [87]Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: transformers for image recognition at scale[J]. arXiv e-prints, 2.[J].rXiv:2010.11929., 020:, :- [88]Chen J, He Y, Frey E C, et al. ViT-V-Net: vision transformer for unsupervised volumetric medical image registration[J]. arXiv e-prints, 2.[J].rXiv:2104.06468., 021:, :- [89] Liu Z, Lin Y, Cao Y, et al.Swin transformer: hierarchical vision transformer using shifted windows[C]. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021: 9992-10002. [90]Chen J, Frey E C, He Y, et al. TransMorph: transformer for unsupervised medical image registration[J]. arXiv e-prints, 2.[J].rXiv:2111.10480., 021:, :- [91]Wang Y, Qian W, Zhang X. A transformer-based network for deformable medical image registration[J]. arXiv e-prints, 2.[J].rXiv:2202.12104., 022:, :- [92]Wang Z, Delingette H. Attention for Image Registration (AiR): an unsupervised transformer approach[J]. arXiv e-prints, 2.[J].rXiv:2105.02282., 021:, :- [93] Singh N K, Raza K: .Medical image generation using generative adversarial networks: a review[J][J].Medical Image Analysis, 2021:, *(*):76-99 [94]Zhang X, Jian W, Chen Y, et al. Deform-GAN:an unsupervised learning model for deformable registration[J]. arXiv e-prints, 2.[J].rXiv:2002.11430., 020:, :- [95] Arar M, Ginger Y, Danon D, et al.Unsupervised multi-modal image registration via geometry preserving image-to-image translation[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 13407-13416. [96]Qin C, Shi B, Liao R, et al. Unsupervised deformable registration for multi-modal images via disentangled representations[J]. arXiv e-prints, 2.[J].rXiv:1903.09331., 019:, :- [97] Xu Z, Luo J, Yan J, et al.Adversarial uni- and multi-modal stream networks for multimodal image registration[C]. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 2020: 222-232. [98]Hoopes A, Hoffmann M, Fischl B, et al. HyperMorph: amortized hyperparameter learning for image registration[J]. arXiv e-prints, 2.[J].rXiv:2101.01035., 021:, :- [99]Mok T C W, Chung A C S. Conditional deformable image registration with convolutional neural network[J]. arXiv e-prints, 2.[J].rXiv:2106.12673., 021:, :- [100]Jia D, Gao S, Chen Q, et al. A low-rank representation for unsupervised registration of medical images[J]. arXiv e-prints, 2.[J].rXiv:2105.09548., 021:, :-