Computational imaging

Computational imaging is a class of imaging methods in which images or quantitative maps are reconstructed from measurements using algorithms. In a conventional camera or microscope, the hardware usually forms a directly recognizable image on a detector. In computational imaging, the detector may instead record indirect data, such as projections, coded measurements, diffraction patterns, phase-shifted images, time-of-flight signals, or images captured under different illumination conditions. A computational model then estimates the object or property being measured, such as intensity, phase, depth, chemical composition, electron density, strain, refractive index, or motion.¹

Computational imaging is used in computational microscopy, medical imaging, computed tomography, magnetic resonance imaging, ultrasound, synthetic aperture radar, seismic imaging, computational photography, coded-aperture imaging, and hyperspectral imaging. It differs from ordinary image processing because the measurement system and the reconstruction algorithm are usually designed together, rather than applying software only after a conventional image has already been formed.

A major area of computational imaging is computational microscopy, including coherent diffractive imaging (CDI), ptychography, and Fourier ptychography. These methods use coherent light, X-rays, or electrons together with phase-retrieval algorithms to reconstruct images when a lens is absent, imperfect, or insufficient to capture the desired information. They can recover both amplitude and phase, enabling quantitative imaging of transparent specimens, nanomaterials, electronic devices, and biological samples.²³

Computational imaging grew from several older traditions, including tomographic reconstruction, radar signal processing, inverse problems in geophysics, coded apertures for high-energy radiation, and digital photography. In computed tomography, many X-ray projections are mathematically combined to reconstruct a cross-sectional or three-dimensional image. Related inverse-problem ideas are used in magnetic resonance imaging, ultrasound, synthetic aperture radar, and seismic imaging.

The development of digital detectors, fast computers, and numerical optimization made it practical to design imaging hardware and reconstruction algorithms as a combined system. In medical imaging, iterative and model-based reconstruction methods have been used to improve image quality and reduce radiation dose in CT.⁴ In MRI, compressed sensing has been used to accelerate image acquisition by exploiting sparsity in the reconstructed image.⁵

In optics and microscopy, phase retrieval became an important foundation. In 1972, Gerchberg and Saxton introduced a practical iterative algorithm to recover phase information from image-plane and diffraction-plane data.⁶ Fienup later developed widely used error-reduction and hybrid input-output algorithms for phase retrieval.⁷ These algorithms helped make it possible to reconstruct information that detectors do not directly measure, especially optical or electron phase. Mathematically, phase retrieval from diffraction intensity becomes possible in principle when the number of independently measured intensity points exceeds the number of unknown object variables by a sufficient amount.⁸

Coherent diffractive imaging was experimentally demonstrated by Miao and collaborators in 1999 by extending crystallographic ideas to non-crystalline specimens. In CDI, an objective lens is replaced by coherent diffraction measurements and computational phase retrieval.⁹² Ptychography, originally proposed in the context of the crystallographic phase problem, developed into a general computational imaging method in the 2000s by combining overlapping coherent diffraction patterns with iterative reconstruction algorithms.¹⁰¹¹²³ Fourier ptychography was introduced in optical microscopy in 2013 to synthesize a larger numerical aperture from multiple low-resolution images recorded under different illumination angles.¹²¹³

Computational photography also developed rapidly with digital cameras and mobile devices. Techniques such as high-dynamic-range imaging, panoramic imaging, light-field imaging, and multi-frame image fusion use algorithms to combine measurements that would not be available in a single conventional exposure.¹⁴¹⁵

Computational microscopy uses engineered illumination, scattering, detection, and reconstruction algorithms to recover information that is not directly recorded by the detector. In many forms of computational microscopy, the measured data are diffraction patterns or image stacks recorded under different illumination conditions. The reconstructed output may be an intensity image, an optical phase map, an electron-density map, a strain map, a refractive-index map, or a three-dimensional structure.²

A central problem in coherent computational microscopy is the phase problem. Detectors usually measure intensity, but not the phase of the scattered wave. Because phase contains structural information, computational algorithms must recover it from intensity-only data. Common approaches use iterative algorithms that move between real space and reciprocal space while enforcing physical constraints, such as finite sample support, positivity, sparsity, overlap between neighboring scan positions, or consistency with a wave-propagation model.⁶⁷¹⁶

CDI reconstructs an object from an oversampled diffraction pattern recorded with coherent X-rays, electrons, or light. In a typical CDI experiment, a coherent beam illuminates an isolated or bounded object, the scattered wave forms a diffraction pattern on a detector, and phase-retrieval algorithms reconstruct the object in real space.⁹

CDI can avoid aberrations associated with objective lenses because the image is reconstructed computationally rather than formed directly by a lens. However, successful reconstruction depends on source coherence, detector dynamic range, signal-to-noise ratio, radiation dose, sample stability, and the accuracy of the physical model. Radiation damage is an important limitation in biological and other beam-sensitive samples.²

Several variants of CDI have been developed, including conventional CDI, Bragg CDI, Fresnel CDI, reflection or grazing-incidence CDI, coherent modulation imaging, and holographic CDI. Bragg CDI is especially useful for mapping strain and lattice distortions in nanocrystals. X-ray free-electron lasers can be used for single-shot diffractive imaging of rapidly changing or radiation-sensitive samples.¹⁷²

Ptychography extends coherent diffraction imaging by recording many diffraction patterns from overlapping regions of a sample. A localized coherent probe is scanned across the specimen, and a diffraction pattern is recorded at each position. The overlap between neighboring probe positions provides redundancy, allowing algorithms to reconstruct both the object and, in modern implementations, the probe itself.¹⁰¹¹

Ptychography is useful for extended objects that are difficult to image by single-pattern CDI. It is used with X-rays, electrons, visible light, and extreme-ultraviolet radiation. X-ray ptychography is widely used for quantitative phase imaging and three-dimensional tomography of materials, biological specimens, and microelectronic devices. Electron ptychography, often implemented in four-dimensional scanning transmission electron microscopy, can achieve sub-ångström resolution and is sensitive to both light and heavy atoms.³¹⁷²

Fourier ptychography is an optical variant in which a conventional microscope records a series of low-resolution images under different illumination angles, often produced by an LED array. The measurements sample different regions of the specimen's Fourier spectrum. Algorithms then combine the data to synthesize a larger numerical aperture and reconstruct a high-resolution image with quantitative phase over a wide field of view.¹²¹³

Tomographic imaging reconstructs internal structure from multiple measurements taken from different directions, positions, frequencies, or encoding conditions. X-ray computed tomography reconstructs cross-sectional or three-dimensional images from many X-ray projections. Magnetic resonance imaging reconstructs images from measurements in k-space, using magnetic-field gradients and radio-frequency signals. Medical ultrasound forms images from echoes of acoustic waves and may use beamforming, model-based reconstruction, or quantitative inversion.

Computational methods in medical imaging can improve image quality, reduce noise, reduce acquisition time, or lower radiation dose. These benefits depend on the accuracy of the measurement model and on clinical validation, because reconstruction algorithms can introduce artifacts when assumptions are violated.⁴⁵

Synthetic-aperture radar uses motion of the radar platform to synthesize a large effective aperture. The received radar echoes are processed computationally to form high-resolution images of terrain, buildings, ships, ice, or other targets. Related ideas are used in sonar and other wave-based remote-sensing systems.

Seismic imaging and seismic inversion use measured seismic waves to estimate underground structure. These methods are important in geophysics, earthquake studies, and resource exploration. Like many other forms of computational imaging, seismic imaging depends on a physical model of wave propagation and on algorithms that solve a large inverse problem.

In coded-aperture imaging, a mask with a known pattern modulates the incoming radiation before it reaches the detector. The recorded data look unlike the object itself, but the object can be estimated by computational decoding. Coded apertures are useful when conventional lenses are impractical, such as for X-rays, gamma rays, or some infrared and terahertz systems.

A pinhole camera is the simplest lensless imager, but it has a trade-off between light collection and resolution. A smaller pinhole improves sharpness but collects fewer photons. A larger opening collects more photons but blurs the image. Coded apertures address this problem by using many openings arranged in a known pattern, increasing throughput while retaining information that can be decoded computationally.¹⁸

Coded-aperture methods have been used in astronomical imaging, nuclear imaging, spectroscopy, and snapshot spectral imaging. They are often combined with deconvolution, correlation, compressed sensing, or statistical reconstruction.

Conventional spectral imaging often scans a scene point by point, line by line, or wavelength by wavelength to build a spectral data cube. Compressive spectral imaging instead records multiplexed measurements that mix spatial and spectral information. Reconstruction algorithms then estimate the full data cube from fewer measurements than would be required by direct scanning.¹⁹²⁰

The coded aperture snapshot spectral imager (CASSI) is a well-known example. It uses a coded aperture and a dispersive element to map spatial and spectral information onto a detector in a compressed form. Numerical optimization is then used to reconstruct the spectral data cube.²¹²²

Compressive spectral imaging is closely related to compressed sensing, which exploits sparsity or other structure in the object being measured. It can reduce acquisition time and enable snapshot measurements, but reconstruction quality depends on calibration, signal-to-noise ratio, scene complexity, and the assumptions used in the reconstruction model.²³²⁴

Computational photography uses algorithms to extend or change the capabilities of cameras. Examples include high-dynamic-range imaging, panoramic stitching, light-field photography, depth estimation, deblurring, denoising, and multi-frame fusion. These methods often use multiple images, coded optics, structured illumination, or sensor data from different viewpoints.¹⁴¹⁵

In consumer cameras and mobile phones, computational photography can improve low-light performance, extend depth of field, estimate depth maps, and combine multiple exposures into a single image. In scientific imaging, related methods are used for quantitative measurement rather than only visual improvement.

Computational imaging algorithms are usually formulated as inverse problems. The measurements are related to the unknown object by a mathematical model of the imaging system. The reconstruction algorithm then estimates the object that best explains the measured data under that model.

A typical workflow includes the following steps:

1. Define a forward model that predicts the measurements from a candidate object. For example, in X-ray CT the model describes how X-rays are attenuated along paths through the object. In coherent diffraction imaging, the model describes how a coherent wave scatters and forms a diffraction pattern.
2. Choose an objective function or estimator. This may be a least-squares error, a likelihood function based on noise statistics, or a regularized objective that includes prior information about the object.
3. Apply an algorithm to estimate the object. Common methods include direct inversion, filtered back projection, Fourier reconstruction, iterative reconstruction, convex optimization, gradient-based optimization, alternating projections, and machine-learning-based reconstruction.
4. Validate the reconstruction using simulations, calibration objects, independent measurements, uncertainty estimates, or reproducibility tests.

Many computational imaging methods use constraints or priors. Examples include non-negativity, finite support, sparsity, smoothness, physical wave-propagation constraints, and overlap consistency in ptychography. These constraints can improve reconstruction from incomplete or noisy data, but they can also bias the result if they do not match the real object or measurement conditions.¹⁶²³²⁴

Computational imaging can recover information that is difficult or impossible to obtain with conventional direct imaging, but it is not independent of experimental assumptions. The reconstructed image is an inferred result. Errors in calibration, alignment, noise modeling, detector response, illumination, motion, or the assumed physical model can produce artifacts.

Validation is therefore central to reliable computational imaging. Common approaches include imaging known test objects, comparing with independent methods, using simulated data with known ground truth, checking reconstruction stability under changes in parameters, estimating uncertainty, and making data and code available for independent analysis. These issues are especially important in high-stakes settings such as medical imaging and in scientific claims that depend on small quantitative differences.

Examples of software packages used in computational imaging, image reconstruction, microscopy, or biomedical image analysis include:

• ImageJ
• 3D Slicer
• ITK-SNAP
• CellProfiler
• FreeSurfer