Abstract

Typical methods for semantic image segmentation rely on large training sets comprising per-pixel semantic segmentations. In medical-imaging applications, obtaining a large number of expert segmentations can be difficult because of the underlying demands on the experts’ time and the budget. However, in many such applications, it is much easier to obtain image-level information indicating the class labels of the objects of interest present in the image. We propose a novel deep-neural-network (DNN) framework for the semantic segmentation of images relying on weakly-and-semi-supervised learning from a training set comprising (i) very few images having per-pixel semantic segmentations and (ii) all images having class labels for the objects of interest present within. To enable weakly-and-semi-supervised learning, our framework proposes to couple the tasks of semantic segmentation and image classification, by incorporating a semantic-segmenter DNN followed by a translator DNN with end-to-end learning. We propose variational learning relying on Monte-Carlo expectation maximization, infering a posterior distribution on the hidden variable that models the segmenter-DNN’s latent space. We propose a Metropolis-Hastings sampler for the posterior distribution, along with sample reparametrizations to enable end-to-end backpropagation. Results on three publicly available real-world microscopy datasets show the benefits of our framework over existing methods, along with empirical insights into the workings of various approaches.