Hyperspectral imaging from a raw mosaic image with end-to-end learning.

Researchers

Journal

Modalities

Models

Abstract

Hyperspectral imaging provides rich spatial-spectral-temporal information with wide applications. However, most of the existing hyperspectral imaging systems require light splitting/filtering devices for spectral modulation, making the system complex and expensive, and sacrifice spatial or temporal resolution. In this paper, we report an end-to-end deep learning method to reconstruct hyperspectral images directly from a raw mosaic image. It saves the separate demosaicing process required by other methods, which reconstructs the full-resolution RGB data from the raw mosaic image. This reduces computational complexity and accumulative error. Three different networks were designed based on the state-of-the-art models in literature, including the residual network, the multiscale network and the parallel-multiscale network. They were trained and tested on public hyperspectral image datasets. Benefiting from the parallel propagation and information fusion of different-resolution feature maps, the parallel-multiscale network performs best among the three networks, with the average peak signal-to-noise ratio achieving 46.83dB. The reported method can be directly integrated to boost an RGB camera for hyperspectral imaging.