The proposed image quality assessment framework is comprised of two parametric functions: 1) the task predictor, $f(\cdot;w):\mathcal{X}\rightarrow\mathcal{Y}$, which performs the target task by producing a prediction $y\in\mathcal{Y}$ for a given image sample $x\in\mathcal{X}$; 2) the controller, $h(\cdot;\theta):\mathcal{X}\rightarrow[0,1]$, which scores image samples $x$ based on their task-specific quality ²²2Task-specific quality means quality scores that represent the impact of an image on a target task; the target task may include any machine learning task including classification, segmentation, or self-reconstruction where these task may not necessarily be clinical tasks.. The image and label domains for the target task are denoted by $\mathcal{X}$ and $\mathcal{Y}$, respectively. Thus in this formulation, we define the image and joint image-label distributions as $\mathcal{P}_{X}$ and $\mathcal{P}_{XY}$, respectively. These distributions have probability density functions $p(x)$ and $p(x,y)$, respectively.

The loss function to be minimised for optimising the task predictor is $L_{f}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}_{\geq 0}$. This loss function $L_{f}$ measures how well the target task is performed by the task predictor $f(\cdot;w)$. This loss weighted by controller-predicted IQA scores can be minimised such that poor target task performance for image samples with low controller-predicted scores are weighted less:

On the other hand the metric function $L_{h}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}_{\geq 0}$ measures task performance on a validation set. Intuitively, performing the task on images with lower task-specific quality tends to be difficult. To encourage the controller to predict lower quality scores for samples with higher metric values (lower task performance), the following weighted metric function can be minimised:

Here, $c$ is a constant that ensures non-zero quality scores.

The controller thus learns to predict task-specific quality scores for image samples while the task predictor learns to perform the target task. The IQA framework can thus be assembled as the following bi-level minimisation problem (Sinha et al., 2018):

A re-formulation to permit sample selection based on task-specific quality scores, where the the data $x$ and $(x,y)$ are sampled from the controller-selected or -sampled distributions $\mathcal{P}_{X}^{h}$ and $\mathcal{P}_{XY}^{h}$, with probability density functions $p^{h}(x)\propto p(x)h(x;\theta)$ and $p^{h}(x,y)\propto p(x,y)h(x;\theta)$ respectively, can be defined as follows:

This bi-level minimisation problem can be formulated in a RL setting. We propose to formulate this problem in a RL-based meta learning setting where the controller interacting with the task predictor via sample-selection or -weighting can be considered a Markov decision process (MDP). The MDP in this problem setting can be described by a 5-tuple $(\mathcal{S},\mathcal{A},p,r,\pi)$. The state transition distribution $p:\mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ denotes the probability of the next state $s_{t+1}$ given the current state $s_{t}$ and action $a_{t}$; here $s_{t+1},~{}s_{t}\in\mathcal{S}$ and $a_{t}\in\mathcal{A}$, where $\mathcal{S}$ is the state space and $\mathcal{A}$ is the action space. The reward function is $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ and $R_{t}=r(s_{t},a_{t})$ denotes the reward given $s_{t}$ and $a_{t}$. The policy is $\pi(a_{t}\mid s_{t}):\mathcal{S}\times\mathcal{A}\in[0,1]$, which represents probability of performing action $a_{t}$ given $s_{t}$. The rewards accumulated starting from time-step $t$ can be denoted by: $Q^{\pi}(s_{t},a_{t})=\sum_{k=0}^{T}\gamma^{k}R_{t+k}$, where the discount factor for future rewards is $\gamma\in[0,1]$. A trajectory or sequence can be observed as the MDP interaction takes place, the sequence takes the form $(s_{1},a_{1},R_{1},s_{2},a_{2},R_{2},\ldots,s_{T},a_{T},R_{T})$. If the interactions take place according to a parameterised policy $\pi_{\theta}$, then the objective in RL is to learn optimal policy parameters $\theta^{*}=\text{argmax}_{\theta}\mathbb{E}_{\pi_{\theta}}[Q^{\pi}(s_{t},a_{t})]$.

Formulating the IQA framework, outlined in Sect. 2.1, as a RL-based meta-learning problem, the finite dataset together with the task predictor can be considered to be contained inside an environment with which the controller interacts. Then at time-step $t$, from the training dataset $\mathcal{D}_{\text{train}}=\{(x_{t}^{i},y_{t}^{i})\}_{i=1}^{N}$, a batch of samples $\mathcal{B}_{t}=\{(x_{t}^{i},y_{t}^{i})\}_{i=1}^{B}$ together with the task predictor $f(\cdot;w_{t})$ may be considered an observed state $s_{t}=(f(\cdot;w_{t}),\mathcal{B}_{t})$. The controller $h(\cdot;\theta)$ outputs sampling probabilities which dictate selection decisions for samples based on $a_{t}^{i}\sim\text{Bernoulli}(h(x_{t}^{i};\theta))$ where action $a_{t}=\{a_{t}^{i}\}_{i=1}^{B}\in\{0,1\}^{B}$ contains the selection decision for samples in the batch. Since the actions are binary selection decisions, at each time-step $t$, the possible actions are $2^{B}$. If $a_{t}^{i}=1$, sample $(x_{t}^{i},y_{t}^{i})$ is selected for training the task predictor. The policy $\pi_{\theta}(a_{t}\mid s_{t})$ may be defined as $\log\pi_{\theta}(a_{t}\mid s_{t})=\sum_{i=1}^{D}h(x_{t}^{i};\theta)a_{t}^{i}+(1-h(x_{t}^{i};\theta)(1-a_{t}^{i}))$.

In this RL-based meta-learning framework, the reward is computed based on the task predictor’s performance $\{l_{t}^{j}\}_{j=1}^{M}=\{L_{h}(f(x^{j};w_{t}),y^{j})\}_{j=1}^{M}$ on a validation set $\mathcal{D}_{\text{val}}=\{(x^{j},y^{j})\}_{j=1}^{M}$. The controller outputs for the validation set $\{h^{j}\}_{j=1}^{M}=\{h(x^{j};\theta)\}_{j=1}^{M}$ may be utilised to formulate the final reward. While several strategies exist for reward computation, we investigate three strategies to compute an un-clipped reward $\tilde{R}_{t}$ Three strategies to compute $\tilde{R}_{t}$ are as follows:

1. 1.

   $\tilde{R}_{\text{avg},t}=-\frac{1}{M}\sum_{j=1}^{M}l_{t}^{j}$, the average performance,

2. 2.

   $\tilde{R}_{\text{w},t}=-\frac{1}{M}\sum_{j=1}^{M}l_{t}^{j}h^{j}$, the weighted sum,

3. 3.

   $\tilde{R}_{\text{sel},t}=-\frac{1}{M^{\prime}}\sum_{j^{\prime}=1}^{M^{\prime}}l_{t}^{j^{\prime}}$, the average of the selected $M^{\prime}$ samples;

For the selective reward formulation $\tilde{R}_{\text{sel},t}$ $\{{j^{\prime}}\}_{j^{\prime}=1}^{M^{\prime}}\subseteq\{j\}_{j=1}^{M}$ with $h^{j^{\prime}}\leq h^{k^{\prime}}$ for $\forall{k^{\prime}\in\{j^{\prime}\}^{c}}$, $\forall{j^{\prime}\in\{j^{\prime}\}}$, i.e. the unclipped reward $\tilde{R}_{\text{sel},t}$ is the average of $\{l_{j^{\prime}}\}$ from the subset of $M^{\prime}=\lfloor(1-s^{rej})M\rfloor$ samples, by removing the first $s^{rej}\times 100\%$ samples, after sorting $h_{j}$ in decreasing order. The first reward formulation $\tilde{R}_{\text{avg},t}$ requires pre-selection of data with high task-specific quality whereas the other two reward formulations do not require any pre-selection of data and the validation set may be of mixed quality. In the selective and weighted reward formulations data is selected or weighted based on controller-predicted task-specific quality. The use of a validation set ensures that the system encourages generalisability therefore extreme cases, where the controller values one sample very highly and the remaining samples as very low, are discouraged. The validation set is formed of data which is weighted or selected based on task-specific quality, either using on human labels or controller predictions. This ensures that selection based on task-specific quality, in the train set, is encouraged as opposed to selecting all samples to improve generalisability. It should be noted that other strategies may be used to compute $\tilde{R}_{t}$, however, in this work we only evaluate the three outlined above.

To form the final reward $R_{t}$, we clip the performance measure $\tilde{R}_{t}$ using a clipping quantity $\bar{R}_{t}$. The final reward thus takes the form:

Similar to $\tilde{R}_{t}$, several different formulations may be used for $\bar{R}_{t}$. In this work we use a moving average $\bar{R}_{t}={\alpha}_{R}\bar{R}_{t-1}+(1-{\alpha}_{R})\tilde{R}_{t}$, where ${\alpha}_{R}$ is a hyper-parameter set to 0.9. It should be noted that, although optional, moving average-based clipping serves to promote continual improvement. A random selection baseline model or a non-selective baseline model may also be used as $\bar{R}_{t}$, however, these are not investigated in this work.

The RL-based meta-learning framework to learn IQA is summarised in Fig. 2 and is outlined in Algo. 1.

For the prostate presence classification task, which is a binary classification of whether a 2D US slice contains the prostate gland or not, the Alex-Net (Krizhevsky et al., 2012) was used as the task predictor with a cross-entropy loss function and reward based on classification accuracy (Acc.), i.e. classification correction rate. The controller was trained using the deep deterministic policy gradient (DDPG) RL algorithm (Lillicrap et al., 2019) where both the actor and critic used a 3-layer convolutional encoder for the image before passing to 3 fully connected layers.

For the prostate gland segmentation task, which is a segmentation of the gland on a 2D slice of US, a 2D U-Net (Ronneberger et al., 2015) was used as the task predictor with a pixel-wise cross-entropy loss and reward based on mean binary Dice score (Dice). The controller training and architecture details are the same as the prostate presence classification task.

Further implementation details can be found in the GitHub repository: https://github.com/s-sd/task-amenability/tree/v1.

The experiments for the two tasks of prostate presence classification and gland segmentation have been previously described in Saeed et al. (2021) and the results are summarised in Sect. 4. For these experiments, the task-specific IQA was investigated, i.e. the agent was trained with $\phi=1$. Evaluation of the proposed task amenability agent for both tasks, including the three proposed reward strategies, and comparisons to human labels of IQA were presented in our previous work Saeed et al. (2021).

Acc. and Dice were used as measures of performance for the classification and segmentation tasks, respectively. These serve as direct measures of performance for the task in question and as indirect measures of performance for the controller with respect to the learnt IQA. Standard deviation (St.D.) is reported as a measure of inter-patient variance and T-test results are reported at a significance level of $\alpha=0.05$, where comparisons are made. Details on controller selection are as described in Sect. 3.2.2.

The data was acquired as part of the SmartTarget:Biopsy and SmartTarget:Therapy clinical trials (NCT02290561, NCT02341677). Images were acquired form 259 patients who underwent ultrasound-guided prostate biopsy. In total 50-120 2D frames of TRUS were acquired for each patient using the side-firing transducer of a bi-plane transperineal ultrasound probe (C41L47RP, HI-VISION Preirus, Hitachi Medical Systems Europe). These 2D frames were acquired during manual positioning of a digital transperineal stepper (D&K Technologies GmbH, Barum, Germany) for navigation or the rotation of the stepper with recorded relative angles for scanning the entire gland. The TRUS images were sampled at approximately every 4 degrees resulting in a total of 6712 images. These 2D images were then segmented by three trained biomedical engineering researchers. For the first task of prostate presence classification, a binary scalar indicating prostate presence was derived from the segmentation for all three observers and then a majority vote was conducted in order to form the final labels for used for this task. For the prostate gland segmentation, binary segmentation masks of the prostate gland were used and a majority vote at the pixel level was used to form the final labels used for this task. The images were split into three sets, train, validation and holdout, with 4689, 1023, and 1000 images from 178, 43, and 38 subjects, respectively. Samples from the TRUS data are shown in Fig. 3.

A 3D U-Net (Özgün Çiçek et al., 2016) was used as the task predictor with a loss formed of equally weighted pixel-wise cross-entropy and Dice (i.e. $1-\text{Dice}$) losses and the reward was based on Dice. The reward formulation used for the computation of $\tilde{R}$ was the weighted formulation. A depth of 4 was used with the U-Net where 4 down-sampling and 4 up-sampling layers were used. Each of the down-sampling modules consisted of two convolutional layers with batch normalisation and ReLU activation, and a max-pooling operation. Analogously, de-convolutional layers are used instead in the up-sampling part. Skip connections were used to connect layers in encoding part to the corresponding layers in decoding part.

The input to the network was in the form of a 3-channel 3D image with the channels corresponding to T2-weighted, diffusion-weighted with high b-value and apparent diffusion coefficient MR images. The output is a one-channel probability map of the lesion for computing the loss, which can then be converted to a binary map using thresholding when computing testing results. Other network training details are described in Sect. 3.2.2.

For experiments with the prostate mpMR images, the deep deterministic policy gradient (DDPG) algorithm (Lillicrap et al., 2019) was used as the RL algorithm for training with the only difference compared to experiments using the TRUS data being that the 3-layer convolutional encoders for the actor and critic networks used 3D convolutions rather than 2D. This agent may be considered as having $\phi=1$.

More implementation details can be found in the GitHub repository: https://github.com/s-sd/task-amenability/tree/v1.

Dice was used a direct performance measure to evaluate the segmentation task and as an indirect measure of performance for the controller with respect to learnt IQA. Standard deviation (St.D.) is reported as a measure of inter-patient variance. Wherever comparisons are made, T-test results with a significance level of 0.05 are reported. Where controller selection is applicable on the holdout set, samples are ordered according to their controller predicted values and the lowest $k\times 100\%$ samples are removed, where $k$ is referred to as the holdout set rejection ratio. This rejection ratio is specified where appropriate. The results are presented in Sect. 4.

With the prostate mpMR images, we compare the proposed task-agnostic IQA strategy, presented in Sect. 2.2, with task-specific IQA learnt using the framework outlined in Sect. 2.1.3. To train the task-agnostic IQA agent, the actor and critic networks remain the same as in Sect. 3.2.2. The task predictor, is a fully convolutional auto-encoder with 4 down-sampling and 4 up-sampling layers, with convolutions being in 3D. The loss was based on mean squared error (ie. $1/n\sum_{i=1}^{n}(y_{t}^{i}-\hat{y}_{t}^{i})^{2}$) and the reward was based on mean absolute error (ie. $1/n\sum_{i=1}^{n}|y_{t}^{i}-\hat{y}_{t}^{i}|$) which serves as a measure of image reconstruction performance. This agent may be considered as having $\phi=0$.

We also evaluate the effect of different values of the weighting between the task-agnostic and task-specific IQA, $\phi$, when using the reward shaping strategy presented in Sect. 2.3. Moreover, samples are presented to further qualitatively evaluate the learnt IQA. These results are presented in Sect. 4.

There were 878 sets of mpMR image data acquired from 850 prostate cancer patients. Patients data were acquired as part several clinical trials carried out at University College London Hospitals, with a mixture of biopsy and therapy patient cohorts, including SmartTarget (Hamid et al., 2019), PICTURE (Simmons et al., 2018), ProRAFT (Orczyk et al., 2021), Index (Dickinson et al., 2013), PROMIS (Bosaily et al., 2015) and PROGENY (Linch et al., 2017). All trial patients gave written consents and the ethics was approved as part of the respective trial protocols. Radiologist contours were done manually and were included for all lesions with a likert score equal or greater than three as the radiological ground-truth in this study. All the data volumes were resampled to $0.5\times{0.5}\times{0.5}mm^{3}$ in 3D space. A region of interest (ROI) with a volume size of $192\times{192}\times{96}$ voxels centring at the prostate was cropped according to the prostate segmentation mask. All the volumes were then normalised to an intensity range of [0,1]. The 3D image sequences used in our study include those with T2-weighted, diffusion-weighted with b values of 1000 or 2000, and apparent diffusion coefficient MR, all available. These were then split into three sets, the train, validation and holdout sets with 445, 64 and 128 images in each set, respectively, where each image corresponds to a separate patient.

The results for the task-agnostic IQA agent, for the lesion segmentation task using mpMR images, are summarised in Table 1 and Fig. 6. The task-agnostic IQA agent refers to the agent trained with $\phi=0$. Controller selection of holdout samples for this agent shows a small yet statistically significant improvement in performance compared to the non-selective baseline (p$<$0.01). A contingency table comparing task-agnostic IQA ($\phi=0.00$) with task-specific IQA ($\phi=1.00$) shows the level of disagreement between the two, with a Cohen’s kappa value less than 0.001. Samples for controller predictions are presented in Fig. 7.

The results for the shaped reward formulations, for the lesion segmentation task using mpMR images, are summarised in Table 2 and Fig. 6b. Higher performance was observed for the controller-selected holdout set compared with the non-selective baseline, for all tested values of $\phi$, where the differences were statistically significant (p$<$0.01 for all). Moreover, the shaped reward formulation with $\phi\geq 0.85$ showed improved performance compared to the shaped reward with $\phi=0.00$ (task-agnostic IQA agent), with statistical significance (p$<$0.01 for all). Interestingly, comparing the the formulation with $\phi=0.00$ to the formulation with $\phi=0.80$, statistical significance was not found (p-value$=$0.051).

To further qualitatively assess the proposed IQA agent in identifying the general task-agnostic quality issues, 20 example cases were shown blindly to an experienced radiologist. Among the 8 cases reported as high task-agnostic quality by the IQA agent, the radiologist agreed with 7, with one having “minor artefact”. Among the other 12 cases deemed low task-agnostic quality with varying task-specific quality by the IQA agent, the radiologist agreed with 3 having “significant quality issues that may affect diagnosis” and 6 having “minor quality issues that are unlikely to affect the diagnostic task”, and disagreed with the other 3 having “little quality issues”.