The problem of matching shapes has been widely addressed in the literature in the past decades (van Kaick et al., 2011). In the specific case of partial matching, one can find two main approaches to such problem: either by finding correspondences, sparse or dense, between structures from descriptors that are invariant to different transformations (Aiger et al., 2008; Rodolà et al., 2017; Halimi et al., 2019), or by looking for a deformation aligning the shapes with respect to a given metric (Aiger et al., 2008; Mori et al., 2013; Zhao et al., 2015; Feragen et al., 2015; Halimi et al., 2020).

The early works on partial shape correspondence as reviewed in van Kaick et al. (2011) rely on correspondences between points computed from geometric descriptors extracted from an isotropic local region around the selected points. The method is refined in van Kaick et al. (2013) by selecting pairs of points to better fit the local geometry using bilateral map. The features extracted can also be invariant to different transformation, as in Rodolà et al. (2013) where the descriptors extracted are scale invariant. Such sparse correspondences are naturally adapted to partial matching, yet they cannot take the whole shapes into account in the matching process.

Using a different approach, functional maps were introduced in Ovsjanikov et al. (2012) allowing dense correspondences between shapes by transferring the problem to linear functions between spaces of functions defined over the shapes. In Rodolà et al. (2017) the non-rigid partial shape correspondence is based on the Laplace-Beltrami eigenfunctions used as prior in the spectral representation of the shapes. Recently in Halimi et al. (2019), such functional map models were adapted in a deep unsupervised framework by finding correspondences minimizing the distortion between shapes. Such methods are yet limited to surface correspondences.

The second kind of methods relies on the deformations that can be generated to align the shapes with each other. It usually involves minimizing a function called data attachment that quantifies the alignment error between the shapes. A deformation cost is sometimes added to regularize these deformations. The sparse correspondences being naturally suited to partial matching, they are notably used in Iterative Closest Point (ICP) methods and their derivatives to guide a registration of one shape onto the other. In Aiger et al. (2008) a regularized version of the ICP selects the sets of four co-planar points in the points cloud. In Mori et al. (2013) the ICP is adapted to the specific case of vascular trees and compute curves’ correspondences through an Iterative Closest Curve method. Working on trees of 3D curves as well, Feragen et al. (2015) hierarchically selects the overall curves correspondences minimizing the tree space geodesic distance between the trees. This latter method, although specific to tree-structures, allows topological changes in the deformation.

On the other hand some authors compute the deformation guided by a dense data attachment term. In Bronstein et al. (2006) isometry-invariant minimum distortion deformations are applied to 2D Riemannian manifolds thanks to a multiscale framework using partial matching derived from Gromov’s theory. This work is extended in Bronstein et al. (2009) by finding the optimal trade-off between partial shape matching and similarity between these shapes embedded in the same metric space. Recently in Halimi et al. (2020) a partial correspondence is performed through a non-rigid alignment of one shape and its partial scan seen as points clouds embedded in the same representation space. The non-rigid alignment is done with a Siamese architecture network. This approach seems promising and is part of a completion framework, however it requires a huge amount of data to train the network.

Interestingly, the regularization cost can be seen as a distance between shapes itself (Feragen et al., 2015) by quantifying the deformation amount necessary to register one shape onto the other. This provides a complementary tool to the metrics used to quantify the shapes dissimilarities.

A well established and versatile framework to compute meaningful deformations is the Large Deformation Diffeomorphic Metric Mapping (LDDMM) (Trouvé, 1995; Christensen et al., 1996; Beg et al., 2005). It allows to see the difference between shapes through the optimal deformation to register a source shape $S$ onto a target shape $T$. However, the data attachment metrics proposed so far aim to compare the source and target shapes in their entirety (Charon et al., 2020), or look for explicit correspondences between subparts of these shapes (Feydy et al., 2017). In Kaltenmark and Trouvé (2018) the proposed growth model introduces a first notion of partial matching incorporated to the LDDMM framework. More recently, the bijectivity constraint is circumvented by the introduction of weighted shapes: in Hsieh and Charon (2021); Sukurdeep et al. (2021), authors optimize a mask defined on the shapes to exclude some subparts of these shapes (source or target). In Antonsanti et al. (2021) a partial matching data fidelity term was introduced in the space of varifolds. We use it in this paper and extend its formulation to better control the non-rigid deformations. We then apply the overall method to guide multi-modal volumes registration.

In the first part we will shortly review the main elements of the representation of shapes in the space of varifolds (Charon and Trouvé, 2013) and the expression of the partial matching we use in this space, introduced in Antonsanti et al. (2021). In particular, we recall how it can be used with LDDMMs to compute diffeomorphic registration of a source shape to a subset of a target one. We will also discuss the use of the partial matching with a rigid registration. We then provide more details on the regularization term used for the application to the registration of truncated liver surfaces to limit the shrinkage effect that can occur when using partial matching with LDDMM.

The second section of this paper is dedicated to its clinical application to multi-modality volume registration based on the partial matching of liver surfaces. In this application we register liver surfaces obtained from segmentations of pre-interventional CT volumes intended for diagnostic purposes, and per-interventional CBCT volumes in which the livers are wider than the field of view, leading to truncation of the surfaces in the latter modality.

The varifold method for shape matching was first introduced in Charon and Trouvé (2013). In this section we use the notations and concepts of the more general oriented varifold framework, developed in Kaltenmark et al. (2017). This framework associates with any shape $S$ a representer function, defined on $\mathbb{R}^{d}\times\mathbb{S}^{d-1}$, as follows:

where $\tau_{x}$ denotes the unit tangent vector –for curves– or unit normal vector –for hypersurfaces– at point $x$ on the shape $S$, and $k_{e},k_{t}$ are predefined fixed kernels functions over $\mathbb{R}^{d}$ and $\mathbb{S}^{d-1}$. In practice, we use $k_{e}:{(\mathbb{R}^{d})}^{2}\rightarrow\mathbb{R}$ a gaussian kernel $k_{e}(x,y)=e^{-\|x-y\|^{2}/\sigma_{W}^{2}}$, where $\sigma_{W}$ is a scale parameter, and for $k_{t}:{(\mathbb{S}^{d-1})}^{2}\rightarrow\mathbb{R}$ the kernel $k_{t}(u,v)=e^{\langle\,u,v\rangle_{\mathbb{R}^{d}}}$. Mathematically, the product kernel $k_{e}\otimes k_{t}$ defines a Reproducing Kernel Hilbert Space (RKHS) structure $W$, and $\omega_{S}$ corresponds to the Riesz representer of the oriented varifold $\mu_{S}\in W^{\prime}$, associated with $S$, and $W^{\prime}$ the space of continuous linear forms on $W$.

The shape matching dissimilarity between $S$ and $T$ is then the squared $W$-norm between their representers, which is also the squared $W^{\prime}$ (dual) norm between their varifolds:

where the scalar product has an explicit form using the kernels:

The proposed partial dissimilarity term is designed to look locally at the inclusion of the deformed source shape into the target one, and to compensate (with the $max(.,.)$ function) for the imbalance of weights between the simple shape and the more complex one with a normalization. The following term tends to 0 when the deformed shape is included in the target one (for more details and discussion, see Antonsanti et al. (2021)).

In the LDDMM framework Beg et al. (2005), the partial matching problem consists in minimizing

where $\phi_{1}^{v}$ is the flow of the time-dependent square integrable velocity fields $t\in[0,1]\mapsto v_{t}$, and $\|\cdot\|_{V}$ is the regularization Hilbert norm over vector fields.

The LDDMM registration procedure is numerically solved via a geodesic shooting algorithm introduced by Miller et al. (2006), optimizing on a set of initial momentum vectors located at the discretization points of the source shape. Implementations of the dissimilarity terms are available on our github repository ¹¹1https://github.com/plantonsanti/PartialMatchingVarifolds.

We can define a second minimization problem for a rigid registration, by minimizing the function:

with $r$ a rigid deformation composed of a translation and a rotation.

For any rigid deformation $\omega_{r(S)}(r(\vec{\mathbf{x}}))=\omega_{S}(\vec{\mathbf{x}})$, $\forall x\in S$, and we can thus write in the discrete setting:

We have that $J_{rig}$ is a composition of continuous functions and that : $k(r(\vec{\mathbf{x}}_{i}),\vec{\mathbf{y}}_{l})\longrightarrow 0$ when the translation goes to infinity from the construction of the kernel $k$. We can deduce that for all rigid deformation we have ${\min}_{\epsilon}\left(1,\dfrac{\omega_{S}(\vec{\mathbf{x}}_{i})}{\omega_{T}(\vec{\mathbf{y}}_{l})}\right)k(r(\vec{\mathbf{x}}_{i}),\vec{\mathbf{y}}_{l})>0$ and $J_{rig}(r)<\sum\limits_{i=1}^{N_{S}}g\left(\omega_{S}(\vec{\mathbf{x}}_{i})\right)$.

$J_{rig}$ is continuous and bounded over the space of finite dimension of rotations and translations, the minimization problem has a solution.

It is known from experiments that the deformations can lead to abnormal shrinkage or stretching of the deformed shapes. This phenomenon comes from two things combined: first, we introduce an attachment term to the data favoring the inclusion of a deformed object in a target, thus an asymmetric term. In the case of non-rigid deformations, there is a multitude of local minima, and this must be controlled with a regularization of the deformations. Second, in the regularization of the LDDMM model (2.3), the deformations tend to shrink the objects along the geodesics, so it is possible that the diffeomorphisms create a shrinkage of the non-realistic source shape. In order to limit this we can add a regularization term in the function $J$ to minimize.

Transcatheter directed liver therapies are part of the therapeutic arsenal of primary and secondary liver malignancies. The objective of these procedures is to locally treat the tumor and be as selective as possible (meaning placing the microcatheter used to inject the treatment as close as possible to the tumor) to preserve surrounding healthy tissues all while ensuring the destruction of the malignant cells.

These minimally invasive procedures are performed by navigating through the patient’s arteries under real time 2D angiography, acquired through an imaging device called C-arm. Additionally, the latter can perform a 200 degrees rotation to allow 3D reconstruction of the patient’s anatomy, called Cone Beam Computed Tomography (CBCT) (Tacher et al., 2015), to obtain a “live” 3D imaging of the patient at point of care. Performing CBCT during such procedure improves tumor detection and navigation guidance (Fig. 1a).

Classically, preprocedural diagnostic CT scan or MRI are reviewed by the interventional radiologist to plan the procedure accordingly. The preprocedural acquisitions provide information on the entire liver anatomy, tumor burden and tumor feeding arteries that are decisive for procedural planning, such as number of tumors to be treated in one session and the dose of therapeutic agent to inject. Contrary to CT or MRI, CBCT is performed during the procedure, and the operator can compare procedural CBCT to preprocedural CT (Fig. 1b) to ensure adequate treatment delivery.

While CBCT offers a superior spatial resolution compared to conventional CT scan, with intra-arterial injection of contrast agent providing a detailed visualization of the arteries, low contrast visibility is better in CT (Fig. 1a). CBCT can also be subject to several artifacts such as beam hardening and motion artifacts that might decrease the CBCT performance to visualize the tumor, which is key to selective and successful treatment. A major difficulty in the fusion of a CT volume with a CBCT one comes from the fact that, unlike CT, liver is only partially visible in CBCT (due to the limited size of the field of view in the latter modality). In addition the acquisitions are taken at different times, potentially several weeks apart, and with different patient stances introducing deformations of the liver. For all these reasons, the two types of volumes are very different one from another as illustrated in Fig. 1.

We propose a registration method based on liver surfaces (one of the feature that is visible in both the CT and CBCT whatever the clinical acquisition protocol) providing a deformation of the entire volume. To that extent we apply our partial matching dissimilarity term allowing to tackle the issue of partial correspondence between the truncated surface extracted from the CBCT (Live) and the one extracted from the CT (Liver). The registrations are evaluated on landmarks inside the liver, which were annotated by a physician.

The database is composed of CBCT/CT pairs where CBCT have been acquired during hepatic arteriography and CT scans obtained at early or late arterial phase. We do not provide the acquisition parameters here, yet the spatial resolution (in $mm$) of the volumes are $(0.45,0.45,0.45)$ for the CBCT and in average $(0.75,0.75,1.25)$ for the CT. Both were selected to show good visualization of the vessels and tumors. In total, 19 pairs of CT/CBCT liver volumes were evaluated, as the one illustrated in Fig. 1.

As a first registration step, the truncated liver surfaces from CBCT were registered onto complete liver surfaces from CT scans with a LDDMM deformation model using the discrete framework described in Sec. 2.2. The LDDMM deformations of the truncated livers surfaces with partial dissimilarity function may lead to small shrinkage of the borders. To compensate this phenomenon in the application we added the a priori regularization of Eq. 4 to the partial data attachment term that prevents from strong local deformations. For illustration purpose, one subject was registered twice with this model: once with the distance in the space of Varifolds, once with the normalized partial dissimilarity term (Def. 1) with the local regularization (Eq. 4). This particular experiment is illustrated in Fig. 4. Both results are generated with the same deformations and regularization parameters. As expected the Varifold distance leads to unrealistic deformations that tend to fill the holes in the source shape to cover the entire target. From the anatomical and medical point of view this is misleading and can not be used in a clinical application of multi-modality volumes registration. On the contrary the partial matching produces a more realistic deformation of the source onto a subset of the complete surface. We will only use and discuss this model in the following.

Both rigid deformation and LDDMM can be extended to the whole volume.

To do so, the voxel grid of the source volume is deformed and interpolated on the target volume. In the case of LDDMM deformations, the voxel grid is deformed with the diffeomorphism of $\mathbb{R}^{3}$ as described in 2.3. The values of the interpolated grid are then reported in the initial volume, providing a registration of the CT volume onto the CBCT one as illustrated in Fig. 5.

The registrations of the volumes can be visualized to qualitatively assess the registration in the livers. To provide a 2D visualization of the results, we use in Fig. 5 and Fig. 6 a tiled representation that alternatively shows two volumes. Such visualization allows to see the continuities between the volumes. Each tile contains a 2D view of the CT volume (light tiles) or the CBCT one (dark tiles). As the dynamics of the images are very different, and the tissues that emerge differ from one modality to another, we are interested in the continuity of the emerging structures such as vessels, liver parenchyma or tumors.

We recall that the key to clinical success is to register precisely the local area around the tumor despite the fact that this tumor is not segmented in CBCT (live) clinical routine (and thus not usable in the registration procedure). Therefore the liver registration is evaluated through the euclidean distance between the deformed points of interest of the CBCT and those of the CT. It is done similarly with the tumors landmarks. Since the POIs are more centered than the tumor landmarks (see Fig.3), this second evaluation is significant as we will further explain in the discussion. The detailed results per case are provided in Table 1 and Table 2. In addition, we evaluate the registration at the surface level, which provides an indication of the overall registration quality, and gives the physician an additional benchmark for the comparison of CT and CBCT volumes. We first compute the barycenters registrations between the surfaces and apply the resulting translations to the volumes.We obtain an average distance of $18.3mm$ between the POIs, and of $21.5mm$ between the tumors landmarks. We show these values as red lines in the corresponding figures. They illustrate what the physicians can quickly obtain during the procedures, registering the volumes in translation by clicking one corresponding point in both modalities. As a reference method for the rigid registration, we also computed the ICP registration directly based on the distance between the points of interest. This registration setting is the only one to exploit the annotated landmarks as input and it will only be used for quantitative comparison.

Therefore the liver registration is evaluated through the euclidean distance between the deformed points of interest of the CBCT and those of the CT. It is done similarly with the tumors landmarks. Since the POIs are more centered than the tumor landmarks (see Fig.3), this second evaluation is significant as we will further explain in the discussion. The detailed results per case are provided in Table 1 and Table 2. In addition, we evaluate the registration at the surface level, which provides an indication of the overall registration quality, and gives the physician an additional benchmark for the comparison of CT and CBCT volumes. We first compute the barycenters registrations between the surfaces and apply the resulting translations to the volumes.We obtain an average distance of $18.3mm$ between the POIs, and of $21.5mm$ between the tumors landmarks. We show these values as red lines in the corresponding figures. They illustrate what the physicians can quickly obtain during the procedures, registering the volumes in translation by clicking one corresponding point in both modalities.

We applied the proposed partial matching dissimilarity term to the registration of pre-op CT volumes on CBCT volumes acquired during the interventions based on segmented liver surfaces. These surfaces, truncated in CBCT, present non-rigid deformations between them due to differences in time point and patient stances in each modality. Partial matching can be used in a rigid registration process, providing results equivalent to a standard method like ICP. It is important to note that the registered surfaces in this application are relatively smooth, which favors the ICP for the rigid registration part. However, this approach in the non-rigid case can lead to projecting several points on one point of the target, and does not take into account the local orientations of the objects or their resolution. On less regular anatomies, one could expect less good performances. Moreover, this approach tends to minimize the average distance from the deformed source to the target. In some cases, as for patient 2 or patient 9, the truncation allows a freedom of deformation which makes the method unsuitable and leads to errors, whereas the reference method generating rigid deformations allowed to obtain good results on both POIs and tumor landmarks. In some cases (Patients 1, 2, 9 for instance), the reference rigid method is providing better results than the rigid ICP based on the livers surfaces, which ensures that despite an acceptable global rigid deformation, the rigid ICP fails at retrieving the correct one using the surfaces to drive the registrations.

The proposed Partial Matching can also be used in LDDMM, providing the tools of computational anatomy and allowing non-rigid, yet regular, deformations. The diffeomorphic deformations that this generates lead to accurate registration of the surfaces of the livers (about $2mm$ on average), which gives the physician a first tool to easily compare CT and CBCT volumes during the procedure. However, the areas of interest for the procedures are also within the liver, and care must be taken to ensure that the non-rigid deformations generated from the surfaces generalize well within the liver volume.

The distance between the Points of Interest, close to the bifurcation of the proper hepatic artery, provide a first evaluation metric to the registration methods. However, since they do not cover the volume of the livers correctly, it does not allow to discriminate between the deformation models. Regarding their location, a rigid registration is sufficient to align them when it fails to extend to the tumors landmarks as illustrated in Fig. 5. In particular, the extremities of the livers seem to be deformed by non-rigid deformations which can be explained by the better performance of the translation+LDDMM regarding the distance on the tumor landmarks. In this case, the rigid deformations with rotation lead to a local minimum that the LDDMM are unable to compensate. On the contrary, translation+LDDMM provide a consistent registration of both POIs and tumors. This last finding indicates that the correct deformations to generate to register a liver from a CBCT acquisition to the liver from the CT acquisition are translations and local non-rigid deformations without large (or even no) rotations.

Although the methods based on surface registrations are sensitive to surface extraction, we have seen through one case illustrated in Appendix D that the proposed registrations remain suitable, despite the outlier (Patient 14) with respect to the metric on the POIs. In particular natural the regularization of the LDDMM associated to the one proposed in this paper provide a smooth and consistent non rigid registration of the surface and its ambient space, ensuring realistic registrations of the volumes.

Regarding the computation time, we are studying a non-rigid multi-modality volume registration solution with partial matching. The current computation time does not allow to use this registration as is in a real time procedure, however there are many levers to accelerate the diffeomorphic deformations, main source of computation cost. These solutions, such as limiting the number of control points (and therefore variables), could allow the use of this solution in an application used during the procedures.

The purpose of this study is to provide a visualization tool for the transcatheter directed liver therapies in minimally invasive procedures. The results obtained with the translation+LDDMM method thus provide robust metrics around $5mm$ on average, which is useful for physicians in the perspective of navigating their tools in the patient’s anatomy to locate structures that are hardly visible in the CBCT used during the procedure. Such registrations could facilitate the physician’s intervention by providing, for example through image fusion, an improved visualization of the pre-procedure CT volume tumor placed in the CBCT volume. This would allow the physician to avoid redoing a traditional CBCT to see the tumor correctly, thus limiting the X-ray dose sent to the patient and the time of the procedure.