360° Image Saliency Prediction by Embedding Self-Supervised Proxy Task - Pôle Télécoms et Réseaux Access content directly
Journal Articles IEEE Transactions on Broadcasting Year : 2023

360° Image Saliency Prediction by Embedding Self-Supervised Proxy Task


The development of Metaverse industry produces many 360° images and videos. Transmitting these images or videos efficiently is the key to success of Metaverse. Since the subject’s field of view is limited in Metaverse, from the perception perspective, bit rates can be saved by focusing video encoding on salient regions. On different ways of handling 360° image projections, the existing works either consider combining local and global projections or just use only global projection for saliency prediction, which results in slow detection speed or low accuracy. In this work, we address this problem by Embedding a self-supervised Proxy task in the Saliency prediction Network, dubbed as EPSNet. The main architecture follows an autoencoder with an encoder for feature extraction and a decoder for saliency prediction. The proxy task is combined with the encoder to enforce it to learn local and global information. It is designed to find the location of a certain local projection in the global projection via self-supervised learning. A cross-attention fusion mechanism is used to fuse the global and local features for location prediction. Then, the decoder is trained based on the sole global projection. In this way, the time-consuming local-global feature fusion is placed in the training stage only. Experiments on public dataset show that our method has achieved satisfactory results in terms of inference speed and accuracy. The dataset and code are available at https://github.com/zzz0326/EPSNet.
Fichier principal
Vignette du fichier
2023_TBC_Zou_et_al.pdf (12.74 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04028523 , version 1 (16-03-2023)



Zizhuang Zou, Mao Ye, Shuai Li, Xue Li, Frédéric Dufaux. 360° Image Saliency Prediction by Embedding Self-Supervised Proxy Task. IEEE Transactions on Broadcasting, 2023, pp.1-11. ⟨10.1109/TBC.2023.3254143⟩. ⟨hal-04028523⟩
29 View
3 Download



Gmail Facebook Twitter LinkedIn More