Image segmentation is an essential initial stage in several computer vision applications. However, unsupervised image segmentation is still a challenging task in some cases such as when objects with a similar visual appearance overlap. Unlike 2D images, 4D Light Fields (LFs) convey both spatial and angular scene information facilitating depth/disparity estimation, which can be further used to guide the segmentation. Existing 4D LF segmentation methods that target object level (i.e., mid-level and high-level) segmentation are typically semi-supervised or supervised with ground truth labels and mostly support only densely sampled 4D LFs. This paper proposes a novel unsupervised mid-level 4D LF Segmentation method using Graph Neural Networks (LFSGNN), which segments all LF views consistently. To achieve that, the 4D LF is represented as a hypergraph, whose hypernodes are obtained based on hyperpixel over-segmentation. Then, a graph neural network is used to extract deep features from the LF and assign segmentation labels to all hypernodes. Afterwards, the network parameters are updated iteratively to achieve better object separation using backpropagation. The proposed segmentation method supports both densely and sparsely sampled 4D LFs. Experimental results on synthetic and real 4D LF datasets show that the proposed method outperforms benchmark methods both in terms of segmentation spatial accuracy and angular consistency.

This work was supported in part by FCT/MECI through National Funds and EU Funds under Grant UID/50008 and in part by the Instituto de Telecomunicações under Project PTDC/EEI-COM/7096/2020.

Light field,Unsupervised segmentation,Deep learning,Angular consistency,Graph neural network

• Computer and Information Sciences - Natural Sciences
• Electrical Engineering, Electronic Engineering, Information Engineering - Engineering and Technology

This publication is an output of the following project(s):

• Light Field Processing for Immersive Media Streaming Applications