Based on the formulation in Section 2.1, we first derive the approximation of the classification loss for binary cell detection. $\mathcal{L}_{\mathrm{cls}}$ is an approximation (empirical mean) of the expectation $\mathbb{E}_{(x,z)}[H(c,z)]$, which measures the classification inaccuracy of $c$ with respect to the ground truth label $z$, and we reformulate the computation of $\mathbb{E}_{(x,z)}[H(c,z)]$ as

Here, we use $p(\cdot)$ to represent a probability density function, and we denote the positive class prior $\mathrm{Pr}(z=1)$ by $\pi$, which is assumed to be known.

In incomplete annotations, positive training samples are available, whereas negative training samples cannot be determined. Therefore, the second term in Eq. (2) can be directly approximated with the incompletely annotated training samples but not the first term. However, the first term can be approximated with both positive and unlabeled training samples via PU learning (Kiryo et al., 2017). Specifically, as $p(x)=\mathrm{Pr}(z=0)p(x|z=0)+\mathrm{Pr}(z=1)p(x|z=1)$, we have

and the first term $(1-\pi)\mathbb{E}_{x|z=0}[H(c,0)]$ in Eq. (2) can be rewritten as

Then, based on Eqs. (2) and (4), $\mathbb{E}_{(x,z)}[H(c,z)]$ can be rewritten as

With this derivation, the second and third terms ($\pi\mathbb{E}_{x|z=1}[H(c,0)]$ and $\pi\mathbb{E}_{x|z=1}[H(c,1)]$, respectively) on the right hand side of Eq. (5) can be approximated with positive training samples, and we still need to approximate the first term $\mathbb{E}_{x}[H(c,0)]$.

The original PU learning framework developed for classification problems assumes that the distribution of the unlabeled data $x_{\mathrm{u}}$ is identical to the distribution of $x$, and thus $\mathbb{E}_{x}[H(c,0)]$ can be approximated by $\mathbb{E}_{x_{\mathrm{u}}}[H(c,0)]$. For convenience, this approximation developed for classification instead of object detection is referred to as the naive approximation hereafter. The naive approximation has been directly applied to object detection problems (Yang et al., 2020). However, since in detection problems the unlabeled samples and positively labeled samples are drawn from the same images, the assumption that the distribution of $x_{\mathrm{u}}$ is identical to the distribution of $x$ in the naive approximation can be problematic, where some positive samples are excluded from the distribution of $x_{\mathrm{u}}$, leading to a biased approximation of $\mathbb{E}_{x}[H(c,0)]$. To better approximate $\mathbb{E}_{x}[H(c,0)]$ for cell detection, we combine the positively labeled and unlabeled samples in the same images, and the combined samples can represent samples drawn from the distribution of $x$. Then, $\mathbb{E}_{x}[H(c,0)]$ can be approximated as

Here, $N_{\mathrm{p}}$ becomes the number of samples associated with the annotated cells in the incomplete annotations, $N_{\mathrm{u}}$ represents the number of unlabeled samples that are not associated with any annotated cells in the incomplete annotations, $k$ is the index of the unlabeled training samples, and $c^{k}_{\mathrm{u}}$ is the predicted classification probability of the $k$-th unlabeled sample $x^{k}_{\mathrm{u}}$. Now, all three terms on the right hand side of Eq. (5) can be approximated with the incompletely annotated training samples.

Note that as shown in Kiryo et al. (2017), when $(1-\pi)\mathbb{E}_{x|z=0}[H(c,0)]=\mathbb{E}_{x}[H(c,0)]-\pi\mathbb{E}_{x|z=1}[H(c,0)]$ is approximated by an expressive CNN, negative values can be produced due to overfitting. This can adversely affect the computation of $\mathbb{E}_{(x,z)}[H(c,z)]$ with Eq. (5). Thus, like Kiryo et al. (2017) we use the following nonnegative approximation of $(1-\pi)\mathbb{E}_{x|z=0}[H(c,0)]$:

To summarize, the derivation steps described above give us the revised classification loss $\mathcal{L}_{\mathrm{cls}}^{\mathrm{pu}}$ that approximates $\mathbb{E}_{(x,z)}[H(c,z)]$ with the PU learning framework, and it is computed as

With $\mathcal{L}_{\mathrm{cls}}^{\mathrm{pu}}$, when only incomplete annotations are available for binary cell detection, the overall loss function to minimize for network training becomes

Based on the derivation in Section 2.2.1 for binary cell detection, we further derive the approximation of the classification loss for multi-class cell detection, where the positive samples are annotated incompletely for each positive class. Mathematically, suppose that there are $M$ classes in total, which comprise $M-1$ positive classes (cell types of interest) and one background negative class; then the ground truth label $z$ of a bounding box $x$ becomes $z\in\{0,\ldots,M-1\}$, where $z=0$ still represents the negative class and $z\in\{1,\ldots,M-1\}$ represents the positive classes. The expectation $\mathbb{E}_{(x,z)}[H(c,z)]$ in Eq. (2) that is associated with the classification loss now becomes

where $m\in\{1,\ldots,M-1\}$ is the positive class index and $\pi_{m}=\mathrm{Pr}(z=m)$ is the class prior (assumed to be known) for the $m$-th positive class. Note that here for multi-class detection, $c$ is a vector that comprises the predicted probabilities of all classes, and $H(\cdot,\cdot)$ computes the categorical cross entropy. Due to the incomplete annotations, $(1-\sum_{m=1}^{M-1}\pi_{m})\mathbb{E}_{x|z=0}[H(c,0)]$ in Eq. (10) cannot be directly approximated.

Similar to Eqs. (3) and (4), when there are $M-1$ positive classes, because $\mathrm{Pr}(z=0)p(x|z=0)=p(x)-\sum_{m=1}^{M-1}\mathrm{Pr}(z=m)p(x|z=m)$, we have

and Eq. (10) becomes

Like in the binary case, the first term $\mathbb{E}_{x}[H(c,0)]$ in Eq. (12) still needs to be determined, whereas the other terms can be computed with the labeled instances of each positive class.

As discussed in Section 2.2.1, the distribution of $x$ can be approximated with the combination of all positive and unlabeled samples. Thus, we approximate $\mathbb{E}_{x}[H(c,0)]$ as

where $N_{\mathrm{p}}^{m}$ represents the number of annotated samples for the $m$-th positive class and $c_{\mathrm{p}}^{m,i}$ represents the prediction probability for the $i$-th positive sample $x_{\mathrm{p}}^{m,i}$ that belongs to class $m$. With Eq. (13), we can approximate $\mathbb{E}_{(x,z)}[H(c,z)]$ using the incomplete annotations based on Eq. (12). Note that again a nonnegative approximation of $(1-\sum_{m=1}^{M-1}\pi_{m})\mathbb{E}_{x|z=0}[H(c,0)]=\mathbb{E}_{x}[H(c,0)]-\sum_{m=1}^{M-1}\pi_{m}\mathbb{E}_{x|z=m}[H(c,0)]$ is used to avoid overfitting, which leads to

Now, we have the classification loss $\mathcal{L}_{\mathrm{cls}}^{\mathrm{mpu}}$ for multi-class cell detection:

$\mathcal{L}_{\mathrm{cls}}^{\mathrm{mpu}}$ is used together with the localization loss (extended to multi-class detection by considering instances of all positive classes) for network training when multiple types of cells are to be detected given incomplete annotations.

The MITOS-ATYPIA-14 dataset (Roux et al., 2013) aims to detect mitosis in breast cancer cells. It comprises 393 images belonging to 11 slides at $\times 40$ magnification. The slides were stained with standard Hematoxylin & Eosin (H&E) dyes, and they were imaged with an Aperio Scanscope XT scanner. The image size is about 1539$\times$1376, and the image resolution is 0.2455 $\mu$m/pixel. Each mitosis in this dataset was annotated with a key point by experienced pathologists, and 749 cells have been annotated. Following Li et al. (2020), for each annotated cell we generated a 32$\times$32 bounding box centered around the key point. The 11 slides were split into training, validation, and test sets, and the images belonging to these slides were split accordingly for our experiment. The ratio of the number of images in the training, validation, and test sets was about 4:1:1. We performed 5-fold cross-validation for evaluation. In each fold, the validation set was fixed, and we regrouped the training and test sets.

Due to the large image size of the MITOS-ATYPIA-14 dataset, we cropped the original images into $500\times 500$ patches, where an overlap of 100 pixels between adjacent patches in the horizontal and vertical directions was used. To simulate incomplete annotations, like Li et al. (2020) we randomly deleted the annotations in the training and validation sets until only one annotated cell per patch was kept. Since the total number of annotated cells in the complete annotations is not large on each image patch, about 73% of the annotated cells were kept in the training and validation sets after deletion. Note that the deletion was performed before network training for this experiment and all other experiments as well. Since the detection is binary for the MITOS-ATYPIA-14 dataset, network training was performed according to Section 2.2.1. The annotations in the test set were intact, and they were only used for the evaluation purpose.

For test images, we first detected the cells on each $500\times 500$ patch, and the prediction boxes on the image patches were merged to produce the final prediction, where the coordinates of these boxes were mapped back into the image and duplicate bounding boxes were removed with non-maximum suppression (NMS) (Neubeck and Van Gool, 2006).

The results on the MITOS-ATYPIA-14 dataset will be presented in Section 3.2. First, the detection performance of the proposed method is given in Section 3.2.1. In addition, to confirm the benefit of the approximation proposed in Eq. (6) for cell detection, we have used the MITOS-ATYPIA-14 dataset to compare the proposed approximation with the naive approximation in PU learning originally developed for classification problems (described in Section 2.2.1). The comparison of the approximation strategies will be reported in Section 3.2.2.

Moreover, we used the MITOS-ATYPIA-14 dataset to investigate the impact of detection backbones. Specifically, besides the VGG16 backbone (Simonyan and Zisserman, 2015), we considered the ResNet50 and ResNet101 backbones (He et al., 2016), which are also commonly used for objection detection with Faster R-CNN. These backbones were integrated with the proposed method to detect cells of interest in histopathology images. The results achieved with these backbones will be reported in Section 3.2.3.

Finally, in addition to the random deletion strategy described above for generating incomplete annotations, as information about the agreement on the annotations between pathologists was available in the MITOS-ATYPIA-14 dataset, we considered another scenario where pathologists choose to annotate the more confident cells. These cells are likely to be those that are easy to annotate. Specifically, the annotated cell with the highest agreement was kept on each image patch in the training or validation set. The other experimental settings were not changed. The results achieved with this deletion strategy will be presented in Section 3.2.4.

To show that the proposed method is applicable to different datasets, we evaluated the detection performance of the proposed method on the publicly available CRCHistoPhenotypes dataset (Sirinukunwattana et al., 2016). The CRCHistoPhenotypes dataset targets the detection of cell nuclei in colorectal adenocarcinomas. It comprises 100 H&E stained images. All images have the same size of 500$\times$500 pixels, and they were cropped from non-overlapping areas of whole-slide images at a resolution of 0.55 $\mu$m/pixel. The whole-slide images were obtained with an Omnyx VL120 scanner. A total number of 29756 nuclei were marked by experts at the center of each nucleus for detection purposes. We followed Sirinukunwattana et al. (2016) and generated a 12$\times$12 bounding box centered around each annotated nucleus. We also performed 5-fold cross-validation for this dataset, where the images were split into training, validation, and test sets with a ratio of about 4:1:1. Like in Section 3.1.1 the validation set was fixed, and the training and test sets were regrouped in each fold.

We cropped the images into $250\times 250$ patches with an overlap of 50 pixels horizontally and vertically between adjacent patches. To simulate the scenario of incomplete annotations, we considered three different cases, where the annotations were deleted at random until there was only one annotation, two annotations, or five annotations on each image patch in the training or validation set, and about 2%, 4%, and 9% of the annotated cells were kept in the training and validation sets, respectively. Network training was performed with the incomplete annotations according to Section 2.2.1, as only one type of cell is of interest for this dataset. The annotations in the test set were complete and used for evaluation only.

For test images, we generated prediction boxes for each $250\times 250$ patch. These boxes were then mapped back into the image with NMS to produce the final prediction on each test image. The detection performance of the proposed method on the CRCHistoPhenotypes dataset will be presented in Section 3.3.

The TUPAC dataset (Veta et al., 2019) with the alternative labels given by Bertram et al. (2020) was also included for evaluation. We selected the first auxiliary dataset of the TUPAC dataset, which aims to detect mitosis in breast cancer. The dataset consists of H&E images acquired at three centers, and we used the 23 cases from the first center. The 23 cases were split into training, validation, and test sets with a ratio of about 4:1:1. Each case is associated with an image, the size of which is about $20000\times 20000$. The images were acquired on an Aperio ScanScope XT scanner at $\times 40$ magnification with a resolution of 0.25 $\mu$m/pixel.

Due to the large image size of this dataset, we cropped the images into $500\times 500$ patches (without overlap), and patches without cells of interest were discarded. The dataset provides both complete and incomplete annotations for the images. However, the difference in the number of annotated cells between the complete and incomplete annotations is small (1359 vs 1273 for the 23 cases). Therefore, based on the original incomplete annotations, we further randomly deleted the annotations in the training and validation sets until there was only one annotated cell per patch. This led to new incomplete annotations that comprised about 63% of all annotated cells in the training and validation sets. Like for the MITOS-ATYPIA-14 dataset, for each annotated mitosis we generated a $32\times 32$ bounding box centered around it, and network training was performed with the new incomplete annotations according to Section 2.2.1. The annotations in the test set were complete and only used for evaluation.

Since the size of the original image is large, evaluation was performed directly on the image patches. The detection performance of the proposed method on the TUPAC dataset will be presented in Section 3.4.

To demonstrate the applicability of the proposed method to multi-class cell detection, we performed experiments on the NuCLS dataset (Amgad et al., 2021). The dataset provides labeled nuclei of seven classes of cells in breast cancer images from The Cancer Genome Atlas (TCGA) (Tomczak et al., 2015). Note that the NuCLS dataset provides annotations (bounding boxes) of different quality, and we used the subset of the images associated with the high-quality cell annotations for evaluation, where initial annotations have been manually corrected by study coordinators based on the feedback from a senior pathologist. This subset comprises 1744 images, and the image size is about 400 $\times$ 400. The images are from the scanned diagnostic H&E slides (mostly at $\times$20–40 magnification) generated by the TCGA Research Network and were accessed with the Digital Slide Archive repository. Since not all seven cell types have a large number of annotated instances, for our experiment, we selected three types of cells for which a large number of annotations were made on these images, and they are the tumor class (with 21067 annotated nuclei), the lymphocyte class (with 13630 annotated nuclei), and the stromal class (with 9132 annotated nuclei).

The images were split into training, validation, and test sets with a ratio of about 4:1:1, and they were directly used for training and testing without cropping. To simulate the scenario of incomplete annotations, for each cell type, if there were more than ten annotated instances in an image in the training or validation set, we randomly deleted the annotations until only ten annotations remained. After deletion 50%, 60%, and 71% of the annotated tumor, lymphocyte, and stromal cells were kept in the training and validation sets, respectively. Since multiple types of cells were of interest here, network training was performed with the incomplete annotations according to Section 2.2.2. The annotations in the test set were complete, and they were only used for evaluation. The detection performance of the proposed method on the NuCLS dataset will be presented in Section 3.5.

In the experiment, the proposed method was compared with two competing methods, which, for fair comparison, used the same backbone Faster R-CNN detection network. The first one is the baseline Faster R-CNN model (also pretrained on the ImageNet dataset as described in Section 2.3), which neglected that the annotations were incomplete and simply considered the unlabeled regions truly negative. Note that here the baseline Faster R-CNN was trained with the standard cross entropy loss. Although it is also possible to use other losses that address imbalanced samples, such as the weighted cross entropy loss or focal loss (Lin et al., 2017), we have empirically observed that they led to worse performance, and thus they were not considered.⁴⁴4The performance achieved with these alternative losses is discussed in Appendix A. The second one is the BDE method (Li et al., 2020, 2021) that addresses the problem of incomplete annotations for cell detection with a calibrated loss function, and it was integrated with the Faster R-CNN architecture with network weights initialized on ImageNet.

In addition to these competing methods, we have also computed the upper bound performance that was achieved with the complete annotations for training. Specifically, the original complete annotations without deletion were used in the training and validation sets, and Faster R-CNN was trained with these training and validation sets with the standard training procedure, as no PU learning was needed for complete annotations.

We first present the detection results of the proposed method on the MITOS-ATYPIA-14 dataset when incomplete annotations were obtained with random deletion. As described in Section 2.3, the class prior $\pi$ was determined based on the validation set. The candidate values of $\pi$ ranged from 0.025 to 0.050 with an increment of 0.005. Note that $\pi$ was selected for each fold independently, and the selected value was consistent (0.035 to 0.045) across the folds.⁵⁵5A detailed analysis of the sensitivity to the class prior $\pi$ is given in Appendix B.

Examples of the detection results of the proposed and competing methods are shown in Fig. 1, where the bounding boxes predicted by each method on representative test patches are displayed. For reference, the gold standard full annotations on the test patches are also shown, and the numbers of true positive and false positive detection results on a patch are indicated for each method. In these cases, our method compares favorably with the competing methods by either producing more true positive boxes without increasing the number of false positive boxes or reducing the number of false positive boxes with preserved true positive boxes.

For quantitative evaluation, we computed the average recall, average precision, and average F1-score of the detection results on the test set for each method and each fold, as well as the upper bound performance achieved with the complete annotation for training, and they are shown in Tables 1 and 2.⁶⁶6To investigate the impact of random effects, we further repeated the experiment with multiple independent runs for the proposed and competing methods, and these results are reported in Appendix C. Compared with the competing methods, the proposed method has higher recall, precision, and F1-score, which indicate the better detection accuracy of our method, and this improvement is consistent across the folds. In addition, the F1-score of our method is closer to the upper bound than the competing methods. We also computed the means and standard deviations of the average recall, average precision, and average F1-score of the five folds, and compared the proposed method with the competing methods using paired Student’s $t$-tests. These results are shown in Table 3. Consistent with Tables 1 and 2, the proposed method has higher recall, precision, and F1-score, and the improvement of our method is statistically significant.

We then performed experiments to confirm the benefit of the approximation developed in Eq. (6) for detection problems. As described in Section 3.1.1, cell detection with incompletely annotated training data was performed with the naive approximation used in PU learning for classification problems on the MITOS-ATYPIA-14 dataset.

The average recall, average precision, and average F1-score of each fold achieved with the naive approximation are listed in Table 4, as well as their mean values and standard deviations. By comparing Table 4 with Table 2, we can see that for each fold the F1-score of the naive approximation is worse than the result of the proposed method, and it is even worse than the BDE result for the first fold. These results indicate the benefit of the proposed approximation.

Next, we investigated the applicability of the proposed method to different detection backbones with the MITOS-ATYPIA-14 dataset as described in Section 3.1.1, where ResNet50 and ResNet101 (He et al., 2016) were considered for Faster R-CNN. The competing methods were also integrated with these backbones, and they were compared with the proposed method.

The results are summarized in Table 5 (together with the upper bound computed with the different backbones), where the means and standard deviations of the average recall, average precision, and average F1-score of the five folds are listed. The proposed method is also compared with the competing methods using paired Student’s $t$-tests in Table 5. With these different backbones, the proposed method still has higher recall, precision, and F1-score than the competing methods, and the improvement is statistically significant.

We then performed experiments with the other annotation strategy, where incomplete annotations were generated based on the agreement on the annotations between pathologists as described in Section 3.1.1. The quantitative results are summarized in Table 6, where the means and standard deviations of the average recall, average precision, and average F1-score of the five folds are listed. The proposed method is also compared with the competing methods using paired Student’s $t$-tests. The proposed method still has higher recall, precision, and F1-score than the two competing approaches, and the improvement is significant.