Super Resolution Imaging – For Better Medical Image Analysis
If you are fan of science fiction films and you are 40+ like me, you will probably remember the classical scene in the movie Blade Runner where Harrison Ford progressively enhances a photo to finally find the face of a young woman reflected in a mirror completely invisible when looking at the whole scene. This is exactly the aim of super resolution (SR) techniques: to extract additional information from images by going beyond the original image resolution. The potential applications of these techniques in radiology are countless and, as we are approaching 2019 (the year of the sci fi story) SR has become a reality, thanks to the hard work of clever scientists working on this topic.
Initial work on SR can be traced back to the beginning of the ‘80s (probably at the same time that Ridley Scott was shooting Blade Runner!) with applications to image enhancement and restoration1. The underlying idea behind SR is pretty simple: the combination of several sources provides more information than each of the individual sources. This sounds quite simple, right? However, as with most engineering problems, the main idea is not the biggest concern, what is challenging is the implementation. In this particular case, how does one obtain and combine different low resolution (LR) sources to obtain the SR image. Even though this is a fascinating engineering problem, the algorithmic details of such methods are out of scope for this short article, which focuses on providing an overview of applications to medical imaging.
Figure 1. The reconstruction of volumetric fetal brain MRI for C12 (GA 36w2d); (a)–(c): the acquired SSFSE scans in (a) axial, (b) coronal, and (c) sagittal orthogonal planes, (d) the initial reconstructed volume by averaging (AVE), and (e) the volume reconstructed by using super resolution techniques. (Credit Gholipour et al 2)
So let’s dive into the radiology applications of SR and consider the particular case of T2-weighted MRI. The acquisition of 3D imagesa by using this specific sequence is very difficult to perform in reasonable imaging times. The problem comes from the need for long signal recovery between excitations to enable the operation of the spin-echo mechanism that provides T2 contrast. Since all the spins are excited by every pulse, the recovery time cannot be utilized and the sequence takes a long time. This is the reason why technologists usually acquire standard T2 axial, coronal and sagittal images for brain examinations. As you have probably already guessed, these are independent LR acquisitions of the same organ that can be combined to create a single image with a pixel resolution higher than independent acquisitions, that is, the SR image.
Figure 1 illustrates this concept through an application to fetal brain analysis. The advantage of using the SR is that the radiologist counts on high resolution images in all directions at the same time and he/she can navigate over the image to obtain extremely detailed anatomical information at each physical point. Figure 2 shows an example of the application of SR images to provide an anatomical context for the fiber bundles estimated by using tractography techniques.
Figure 2. Example of fiber bundles in a fetal brain displayed in the anatomical context provided by the SR image. (Credit CNRS Photothèque / François Rousseau / Estanislao Oubel3)
Even though MRI is the primary target modality of SR techniques, they have also been applied successfully to other imaging modalities like PET4 and mammography5. Several review articles about novel methods and clinical applications are already available, like the one by Greenspan et al 6. Figure 3 shows an example of Digital Breast Tomosynthesis, which can be considered as a SR technique where the LR inputs of the algorithm are projection images.
Figure 3. Conventional film screen mediolateral oblique mammography view (A) of a patient with invasive ductal cancer. The lesion, although vaguely apparent on the conventional mammogram (arrow), is much better visualized on the 1 mm thick tomosynthesis image (B) (arrow). (Credit M. A. Helvie7)
A drawback of SR techniques is that pixel values are estimated by using computer vision methods from a specific set of pixels in the original images. These “artificially-generated” pixels are the result of an optimization process aiming to minimize the error with respect to different sources, and therefore cannot be considered as reliable as pixels in the original images which are directly generated by the reconstruction methods of image acquisition systems. This raises some questions about the use of such images for diagnosis purposes. There seems to be a consensus in the scientific community that these images could be used as complementary information and whenever the original images are available to be explored independently to confirm/reject potential findings. This situation is similar to the interpretation of results when applying tractography methods: the fibers generated depend on multiple parameters and a non-negligible variability is associated to them. As a consequence, end users must be careful when interpreting the information.
Even though current results are encouraging, real life scenarios present specific issues that need to be addressed, like patient motion and acquisition time/processing constraints. The main competitor of SR techniques is perhaps hardware, which may provide additional means for resolution augmentation. Future advances in this technology are focused on two different areas:
- Finding applications that could take advantage of the SR technology from a clinical point of view. This could be promoted by making it available on standard platforms provided by current manufacturers, which would allow the identification of novel clinical applications.
- From an algorithmic point of view, by extending research to additional medical imaging modalities, which will improve current methods and help find solutions to the current limitations.
These advances in medical imaging are groundbreaking and open the doors to infinite diagnostic possibilities. Time will tell us which approach wins for providing SR capabilities in radiology platforms (hardware, image processing techniques, a combination of both?) Regardless of the specific technological solution, we could start imagining the radiologist of the future seated in front of the screen, just like Harrison Ford, navigating throughout the image, zooming in suspicious regions, zooming out after ruling out lesions, changing point of views, to exploit all the available information and guarantee the best possible diagnosis with a single final objective: to improve patient care.
Post Authored by:
Estanislao OUBEL, MSc, PhD, Lead Scientist, Imaging Processing and Science – Median Technologies
References:
[1] Huang, T. S.., Tsay, R. Y., “Multiple frame image restoration and registration,” Adv. Comput. Vis. Image Process. 1, 317–339, JAI, Greenwich (1984).
[2] Gholipour, A., Estroff, J. A.., Warfield, S. K., “Robust super-resolution volume reconstruction from slice acquisitions: application to fetal brain MRI.,” IEEE Trans. Med. Imaging 29(10), 1739–1758, NIH Public Access (2010).
[3] CNRS Phototheque/ Francois Rousseau / Estanislao Oubel, https://lejournal.cnrs.fr/diaporamas/du-neurone-a-la-pensee
[4] Kennedy, J. A., Israel, O., Frenkel, A., Bar-Shalom, R.., Haim Azhari, H., “Super-resolution in PET imaging,” IEEE Trans. Med. Imaging 25(2), 137–147 (2006).
[5] Zheng, J., Fuentes, O.., Leung, M.-Y., “Super-resolution of mammograms,” 2010 IEEE Symp. Comput. Intell. Bioinforma. Comput. Biol., 1–7, IEEE (2010).
[6] Greenspan, H.., Hayit., “Super-Resolution in Medical Imaging,” Comput. J. 52(1), 43–63, Oxford University Press (2008).
[7] Helvie, M. A., “Digital mammography imaging: breast tomosynthesis and advanced applications.,” Radiol. Clin. North Am. 48(5), 917–929, NIH Public Access (2010).
[a] In the context of this paper, 3D image means an image with isotropic voxels, i.e. voxels with the same size in each directions as opposed to most of acquisitions with voxels longer in the z direction (anisotropic voxels).