As shown in Fig. 1, the dual-branch network, which extends from the VNet architecture (Milletari et al., 2016), comprises a shared encoder and two decoders inspired by Luo et al. (2022a). Let’s denote the inputs of the encoder as $x$ and the output features as $f$, which include features from the bottleneck layer and skip connections at different resolutions. The two decoders share the same structure, but have different inputs and parameters. The main decoder directly utilizes the features $f$ from the encoder as input, while the auxiliary decoder takes perturbed $f$ through dropout as input. We define the mappings of features from $f$ to the outputs of the main and auxiliary decoders as $g_{m}(\cdot)$ and $g_{a}(\cdot)$, respectively. In detail, one convolution block contains two convolution layers and a residual connection with Instance Normalization (IN) and Leaky ReLU activation function. Both the encoder and decoder are symmetrical structures with five different resolutions.

An appropriate pretext task can empower neural networks to learn low-level and high-level features that are conducive to downstream tasks (Jing and Tian, 2020). Zhou et al. (2021) introduced a self-supervised method called Model Genesis, which performs an image reconstruction process, and has shown promising results for downstream supervised segmentation tasks. Therefore, we use Model Genesis to pre-train the dual-branch network. Unlike the original Model Genesis that only trains one decoder, we extend it by training two decoders for the reconstruction process during pre-training.

Model Genesis (Zhou et al., 2021) employed three types of transformations on the original images, as detailed in Fig. 2: 1) Non-linear transformation integrates the Bézier Curve (Mortenson, 1999) to assign a unique determined value to each pixel, encouraging the model to focus on the information of image appearance and intensity distribution. 2) Local pixel shuffling samples a window smaller than the model’s receptive field in the patch and rearranges the internal pixels to encourage the model to learn the local texture and boundary. 3) Out-painting or In-painting: Out-painting sets the outer pixels of a shape to random values, while the inner pixels retain their original intensities. In-painting follows the opposite way. The network learns visual features of images by reconstructing the original images from the corrupted version. In the self-supervised pre-training as shown in Fig. 1(a), the outputs from the main and auxiliary decoders compute the Mean Squared Error (MSE) $\mathcal{L}_{MSE}$ separately with the original image $y$. The reconstruction loss is defined as:

where $p^{m}=g_{m}(f)$ and $p^{a}=g_{a}(f)$ are the predictions of the main and auxiliary decoder, respectively. And $y$ denotes the reconstruction target, i.e., the original input image.

Partial instance annotations can be considered as a noisy label learning problem, where some foreground regions are incorrectly labeled as the background. However, compared to conventional noisy label learning scenarios, the noise in the background of partial instance annotations can be excessive, with a large amount of false negatives. Therefore, we integrate weakly supervised learning and noisy label learning methods to effectively learn from partial instance annotations and generate reliable supervision signals. This combination enables us to mitigate the impact of excessive noise in the background and produce more effective supervision signals for the learning process.

Firstly, in our quest for more reliable supervision signals, we employ a noisy label learning technique to learn from partial instance annotation. Given a large amount of false negatives in the labels, we utilize the Symmetric Cross-Entropy (SCE) loss to balance the confidence between the partial instance annotation and the model’s prediction, as proposed by Wang et al. (2019):

where $\mathcal{L}_{CE}$ is the widely used Cross-Entropy loss. $p$ and $y$ are the predicted probability map in a certain branch and partial instance annotation. The relationship between $\gamma$ and 1 indicates which one is more trustworthy between the prediction and the label. In this work we set $\gamma$ to 0.8 due to that the partial instance annotation is less credible than the model’s prediction.

Secondly, we introduce the Partial Cross-Entropy (PCE) loss (Lee and Jeong, 2020) to ensure that the foreground in the partial instance label can be reliably learned. Unlike the PCE loss used in scribble-level annotations (Luo et al., 2022a), which supervises all the labeled voxels (including all target categories as well as the background) and does not calculate on unlabeled voxels, for partial instance annotations, we compute the cross-entropy only for the foreground voxels:

where $\Omega$ is the foreground voxels in partial instance annotation. $p_{i}$ and $y_{i}$ denote the predicted probability and partial instance annotation of voxel $i$.

Finally, despite that $\mathcal{L}_{PCE}$ helps to improve recall of lymph nodes, it increases the risk of false positives. To deal with this problem, we additionally introduce a Tversky loss (Salehi et al., 2017) for supervision. Unlike Dice loss, which treats False Positives (FPs) and False Negatives (FNs) samples equally, the Tversky loss can balance the importance of both with different weights and mitigate class imbalance simultaneously:

where $N$ is the number of voxels, $TP=\sum_{i=1}^{N}p_{i}y_{i}$, $FP=\sum_{i=1}^{N}p_{i}(1-y_{i})$ and $FN=\sum_{i=1}^{N}(1-p_{i})y_{i}$. By adjusting the hyper-parameter $\alpha$, we can control the importance between FPs and FNs. To predict more foreground voxels (false positive samples relative to partial instance annotation), $\alpha$ is set to 0.4 based on experiments.

For partial instance annotation, the supervised loss for each decoder is a combination of $\mathcal{L}_{PCE}$, $\mathcal{L}_{SCE}$ and $\mathcal{L}_{Tversky}$:

Due to the presence of incorrectly labeled background voxels in partial instance annotations, it is unreliable to directly extract supervisory signals from them. Inspired by pseudo label learning for scribble annotations (Luo et al., 2022a), we first dynamically mix the predictions from the two decoders:

where $\theta$ is randomly generated from a uniform distribution between 0 and 1 at each iteration, enhancing the diversity of the pseudo label and compelling the model to continually update its predictions (Huo et al., 2021).

Then, we apply a sharpening function to adjust the entropy of the label distribution. The predicted probability of class $k$ can be define as:

where $\tilde{y}_{mix}^{k}$ is the mixed output of class $k$ and $C$ is the set of all categories. $\tau$ is a temperature that is normally set to 1 (Hinton et al., 2015). When $\tau>1$, the labels become smoother, leading to increased entropy within the labels. Consequently, the information carried by negative labels is relatively amplified, directing the model training to pay more attention to negative labels. Conversely, when $\tau<1$, the labels become sharper. Properly sharpening the labels can enhance their robustness to noise while also maintaining the differences between classes. We set $\tau$ to 0.3 in our implementation.

Finally, we integrate the mixed pseudo label with the partial instance annotation to obtain the final pseudo label, ensuring that the pseudo label complements the partial annotation. The final pseudo label is denoted as $\hat{y}$, and its $i^{th}$ element is defined as $\hat{y}_{i}=y_{i}+(1.0-y_{i})\tilde{y}_{i}$, i.e., the zero region in $y$ is replaced by the corresponding values from $\tilde{y}$.

The perturbation introduced in the auxiliary decoder may lead to uncontrollable effects. Ideally, predictions for background voxels near the classification boundary should shift towards the foreground space. However, foreground voxels in partial instance annotations may be predicted as background, leading to misleading effects in the model’s training. To mitigate such adverse effects, we only learn from pixels with minor discrepancies based on the consistency of the two outputs, ensuring a smooth and gradual learning process. We utilize Kullback-Leibler (KL) divergence to estimate the consistency of the two outputs and use it to generate voxel-wise weights for the Cross-Entropy loss. Following the approach outlined in Zheng and Yang (2021), the weight for voxel $i$ is defined as:

where $KL(p^{m}_{i},p^{a}_{i})=p^{a}_{i}log(p^{a}_{i}/p^{m}_{i})$ is the KL divergence loss calculated from the $i^{th}$ voxel of the two probability maps $p^{m}$ and $p^{a}$. When the predictions of a certain voxel from the main decoder and the auxiliary decoder are highly dissimilar, Eq. (8) will lead to a lower value of $\mathcal{W}_{i}$. Based on this observation, the learning loss of the main branch for pseudo label can be formulated as:

where $W$ is the sum of $\mathcal{W}_{i}$ for all voxels. The introduction of $KL$ can avoid excessive discrepancies between the predictions of the two decoders.

The proposed DBDMP framework learns from both partial instance annotation and pseudo label by minimizing the following combined objective function:

where $y$ and $\hat{y}$ are partial instance annotation and the generated pseudo label, respectively. $\lambda_{p}$ is the trade-off weight that schedules with an epoch-dependent sigmoid-like ramp-up function in the first 100 epochs as the pseudo labels in the early training stage can be in poor equality:

where $\lambda$ is a hyper-parameter that represents the final value of $\lambda_{p}$. $t_{max}$ is set to 99 which means the maximal epoch for ramp-up and $t$ is the current epoch.

Our method was implemented in nnUNet (Isensee et al., 2021), which is a Pytorch-based (Paszke et al., 2019) toolkit for image computing with deep learning. The implementation was carried out on a single NVIDIA 2080Ti GPU with 11GB VRAM. We utilized a VNet-like (Milletari et al., 2016) network as the backbone for all experiments, and extended it to two decoders, as detailed in Section 3.1.

For preprocessing, we first cropped the volumes to the lung region based on intensity. Subsequently, we resampled each volume into the resolution of 3.0 $mm\times$0.8 $mm\times$0.8 $mm$. Finally, we normalized each volume to have zero mean and unit variance. Our networks were trained using a patch-based approach with a patch size of $224\times 128\times 64$ and a batch size of 2. We employed the polynomial learning rate strategy to decay the learning rate in each epoch.

For self-supervised pre-training in Fig. 1(a), we used Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.99, an initial learning rate of 0.01, and weight decay of $3\times 10^{-5}$ to minimize the reconstruction loss in Eq. (1). The training process lasted for 1000 epochs, with 250 iterations in each epoch. During weakly supervised training, the segmentation model was initialized with the weights obtained from self-supervised pre-training. We minimized the loss functions in Eq. (10) using SGD optimizer with a momentum of 0.9, while keeping the other parameters the same as those in the self-supervised pre-training stage. The training epoch was 300 with 250 iterations in each epoch.

During the inference stage, we loaded the weights from the final epoch and only utilized predictions from the main decoder as the final outputs. All inference processes were conducted using a sliding window strategy. We also applied a specific post-processing method, which involved removing lymph node regions at the boundaries of an image and eliminating a portion of the lymph nodes based on voxel intensity and the actual volume. For quantitative evaluation, we calculated the Dice Similarity Coefficient (DSC) and the Average Symmetric Surface Distance (ASSD) between a segmentation result and the ground truth. In light of the potential for samples with failed predictions, the ASSD for these samples is missing, so we fill them with the maximum value of the successfully calculated ASSD and then average them. This may result in larger mean ASSD values.