Spectral rendering approaches are nowadays the most accurate ones for predictively simulating the light transport equation with advanced wavelength-dependent phenomena such as light dispersion or metamerism. However, these approaches require a much larger number of samples and longer computational times than standard trichromatic ones\textemdash e.g. expressed in RGB color spaces\textemdash to properly sample the visible spectral/spectrum domain and achieve noise-free images. To address these issues, denoising solutions can be applied to a spectral renderer in order to remove potential color noise, while ensuring that the reconstruction can properly conserve wavelength-dependant phenomena. Inspired from a feature fusion based deep dual-encoder network, we propose a deep-learning approach applied to multi-spectral images. Instead of denoising already-integrated RGB data, and to preserve crucial spectral information, we discretize the output of a spectral renderer into multiple spectral bins. As a way for our architecture to remain agnostic in regard to the number of bins, we denoise each bin independently and use a PCA-like batch normalization transformation, also known as whitening tranformation, to compress all spectral bins into a single parametric image. This parametric feature used as an auxiliary input allows us to conserve critical context information when denoising each bin. In addition to other auxiliary features such as normal and depth, we provide the denoiser with a spectral albedo discretized in the same manner as the noisy input. Our experiments demonstrate that our method outperforms state-of-the-art denoising techniques for spectral rendering while significantly reducing the number of samples required for noise-free images, and provides a better reconstruction than trichromatic regular approaches, preventing the appearance of most artefacts.