Multimodal Surface Matching (MSM): a flexible tool for aligning generic data on the cortical surface

Let’s say you have run a functional imaging experiment on a group of different subjects: you have asked all the subjects to perform a task in the scanner, and you want to know whether the response of all subjects to that task is the same. For example, you have some healthy subjects, and some with Dementia. Do all subjects perform as well on a memory task? How can you perform this comparison?

Cortical registration is vital for this type of comparison as it allows studies to compare and contrast brain activity across different subjects by bringing all imaging data into a common template space. When done correctly, image registration ensures that each data point in the template space represents the same functional activation in all subjects.

Many different types of registration exist. In particular there are two sub-streams volumetric registration (that works on 3D volumes) and cortical surface approaches that first fit a tessellated mesh to each brain surface. In my opinion the latter approach has distinct advantages. But, why register cortical surfaces rather than volumes?


The cortical surface is highly convoluted. The folds vary considerably across individuals. In general it is not possible to achieve a complete matching of folds between brains, and especially if you are concerned with retaining smooth deformation fields you can expect significant differences.

In the volume a relatively small mismatch of folding patterns in three-dimensions can have considerable impact in terms of the alignment of sub-structures within the cortex. For example, a slight translation of a folding pattern of just a few millimetres could match the left bank of a sulcus, in one subject, with the right bank in another. This is problematic for any subsequent functional analysis. as each sulcus can be subdivided into several functionally specialised regions. This means that you could end up comparing different functional regions as the same thing.

Then the question is: how do I register cortical surfaces? There are a large number of approaches, so I’m not going to go into all of them. But in general  they can be roughly subdivided into methods that warp the surfaces directly eg [1]; and methods that first project surfaces to a sphere and perform alignment of the spherical projections [2][3].

Spherical methods are are by far the most popular and include the FreeSurfer method, whilst despite being almost 20 years old, is probably the most widely used cortical surface tool in the neuroimage community. They have the advantage of greatly simplifying the registration problem by representing the problem in two, rather than three, dimensions. In addition, projection to a sphere removes dependence on the geometry for alignment, which for reasons I will describe below, can be a disadvantage.

MSM is also spherical projection approach, but from there it diverges from the methods used before for several reasons:

  • It uses discrete optimisation
  • It is not constrained to use any particular data type to drive alignment.

In fact these two reasons are related; it is because we use discrete optimisation that the method is highly unconstrained as to what data we can use to drive alignment. This is because the similarity force that drives alignment is completely uncoupled from the optimisation approach, unlike Spherical Demons for example which uses Gauss Newton optimisation and therefore requires a Sum of Square Difference (SSD) similarity term, which assumed datasets only differ through Gaussian noise. Also, in [4] et al propose an elegant solution to the alignment of functional connectivity, but as the optimisation is specifically solved for covariance matrices, it restricts generalisability.

Flexibility with regards to data is a huge advantage. Not least because there is a huge variability in the types of data that can be projected onto the cortical surface. Each of these data sets has strengths and weakness and together they often supply complementary information that can help aid analysis.


Further there are known limits to cortical folding or geometric based alignment that hinder approaches that depend on them [1][2]. For one, cortical folding is highly variable across subjects, which means that in some subjects major neuroanatomical structures such as the cingulate can be represented by two folds, whereas in other subjects there is only one [5]. Further, there have been noted topological inconsistencies between folding patterns and cytoarchitectural subdivisions of the brain [6]. This means that the same functionally specialised sub-region of the cortex can appear in different relative position in a fold for different individuals. This means that driving cortical alignment using geometry alone does not guarantee optimal alignment of brain functional activations [7].

For these reasons we developed MSM or multi-modal (or multi-data-type) surface matching, and achieved this using a discrete displacement approach [10].


The idea is first that you constrain the deformation of your spherical mesh, though the displacement of a relatively few, widely-spaced control points. Registration is refined by iteratively upsampling the control point grid. This approach is also used in Spherical Demons [3] and Free-Form deformation volumetric approaches [9], including the method that inspired MSM [10]. The advantage is that they further constrain the smoothness of the deformation and allow structures in the data to be aligned in a coarse-to-fine fashion

Then rather than allowing control points on the surface to deform to any new  feasible position, instead you restrict the number of endpoints to a finite number spaced evenly around on the surface. You then solve your registration as a multi-labelling problem, where each label represents a possible endpoint on the surface for each and your algorithm tries to optimise label configurations across the surface until the features on the moving surface best match that on the target surface.

One caveat is that it is not as simple as just looking for the best matching of the data. It is also important that the deformation is smooth, which means that the displacement of the points does not cause folding of the spherical mesh, and doesn’t shrink or expand the triangular faces of the mesh outside of a range that is considered normal variation for the human brain. Therefore constraints are put on the registration in order to discourage neighbouring control points from deforming in very different directions. As each discrete deformation can be seen as rotations of the control points about the centre of the sphere, this can be achieved by penalising differences in the angles of rotation. Therefore, the optimal displacement of each point is a balance between getting the best match of all data locally centred on that point, and making sure that the deformation of the point follows that of its neighbours.

This approach for estimating smooth deformations is a simple one and it does have its limitations (keep your eyes open for the upcoming release of a new MSM with advanced regularisation of white-matter surface deformations). Nevertheless, we has observed significant advantages for the alignment of functional data.

MSM has so far been used for the alignment of cortical folding, resting state fMRI, retinotopy and cortical myelin. fMRI task maps, aligned using resting state, show significant improvements in the size and peak values of group z statistics due to the improved alignment of functional activations.


Further MSM allows joint alignment of different feature sets: for example myelin, and folding.


A major challenge remains however: how to combine feature sets that fundamentally don’t agree. In the above example we combine myelin and folding using subjective cost functions that simply downweight myelin where it isn’t trusted: something of a hack to say the least!

In future, weightings will need to be learnt from the data, based on groupwise consistencies or prior information. Given that the theoretical origins of cortical folding may be based on the formation of nerve fibres, efforts should be made to reconcile the differences between folding and function. One option may be to move away from the assumptions of a single population consistent model of folding and function and start to look for families or subpopulations of brain topologies, within which function and structure might agree.

****Note this isn’t intended to be a technical guide or a comprehensive user manual; for those please look at the original paper or the tool page on the  FSL software library website where MSM will soon become available  ( The purpose of this post is to motivate at more length about why it is useful (and sometimes vital) to register brain surfaces rather than volumes; to justify some of the implementational choices; and to highlight MSMs great flexibility with regard to the choice of data you chose to drive your alignments.***


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s