MHF-Net: An Interpretable Deep Network for Multispectral and Hyperspectral Image Fusion.

Researchers

Journal

Modalities

Models

Abstract

Multispectral and hyperspectral image fusion (MS/HS fusion) has become one of the most commonly addressed problems for hyperspectral image processing. In this paper, we specifically design a network architecture for the MS/HS fusion task, called MHF-net, which not only contains clear interpretability, but also well embeds the intrinsic generalization mechanism of low-resolution images. In particular, we first construct an MS/HS fusion model which merges the generalization models of low-resolution images and the low-rankness prior knowledge of hyperspectral image into a concise formulation, and then we build the proposed network by unfolding the proximal gradient algorithm for solving the proposed model. As a result of the careful design of the model and algorithm, all of the fundamental modules in MHF-net have clear physical meanings and are thus easily interpretable. Based on the architecture of MHF-net, we further design two deep learning regimes: consistent MHF-net and blind MHF-net, which are suitable in the presences that spectral and spatial responses of train and test data are consistent and inconsistent, respectively. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.