Designing Imaging Systems by Measuring Information Content
Traditional imaging system design often relies on metrics like resolution or signal-to-noise ratio that assess quality in isolation, or on training neural networks that conflate hardware and algorithm performance. But for systems like self-driving car sensors or MRI scanners, what matters is how much useful information the raw measurements carry, even if humans can't interpret them directly. A new framework, presented at NeurIPS 2025, estimates mutual information directly from noisy measurements to evaluate and optimize imaging systems based on information content alone. Below, we answer key questions about this approach.
Why can't we rely on traditional image quality metrics for modern imaging systems?
Traditional metrics such as resolution, signal-to-noise ratio (SNR), and modulation transfer function each assess only one aspect of image quality. In a modern imaging pipeline—whether from a smartphone sensor processed through algorithms, an MRI collecting frequency-space data, or a LiDAR feeding a neural network—the measurements may not even be viewed by humans. What matters is the information they contain for downstream tasks like object detection or classification. Separating quality into independent factors makes it impossible to compare systems that trade off, say, resolution for noise reduction. Furthermore, the common alternative of training a neural network to reconstruct or classify images conflates the performance of the hardware with that of the algorithm, making it hard to know whether sensor improvements are genuine. A direct information metric avoids these entanglements and provides a single, task-independent measure of how well measurements distinguish different objects.
What is mutual information and why is it useful for imaging?
Mutual information (MI) is a quantity from information theory that measures how much a measurement reduces uncertainty about the object that produced it. In imaging, two systems with the same MI are equally good at distinguishing objects, even if their raw measurements look completely different. This is powerful because MI naturally accounts for the combined effect of resolution, noise, sampling, spectral sensitivity, and any other factor that influences measurement quality. A blurry, noisy image that preserves critical features for discrimination may have higher MI than a sharp, clean image that loses those features. By using MI as a single objective, designers can directly compare disparate optical and sensor configurations on a common scale, and optimize them without needing to pre-define what “good” looks like in pixel terms.
How does the information-driven design approach differ from end-to-end neural network methods?
End-to-end learning trains a neural network jointly with a differentiable model of the imaging system, so that the network learns to reconstruct or classify images from sensor data. The drawback is that the network is specific to the task (e.g., segmentation or classification) and must be re-designed if the task changes. The network also uses the entire measurement, consuming more memory and compute. The information-based method, in contrast, evaluates the imaging system purely by estimating the mutual information between objects and measurements, independent of any downstream algorithm. In the NeurIPS 2025 paper, optimizing an imaging system using this information metric produced designs that matched the performance of state-of-the-art end-to-end approaches while requiring less memory, less compute, and no task-specific decoder design. This makes the approach more general and efficient for co-optimizing optics and sensors before deciding on the processing algorithm.
What were the previous challenges in applying information theory to imaging?
Earlier attempts to use information theory in imaging faced two main problems. The first was treating the imaging system as an unconstrained communication channel, ignoring the physical limitations of lenses and sensors. This gave wildly inaccurate estimates because real optics impose diffraction, aberrations, and finite apertures that reduce channel capacity. The second problem was requiring explicit models of the objects being imaged, such as a known probability distribution over scenes. Such models are difficult to obtain for general natural images or medical data, limiting the generality of the approach. The new method avoids both issues by estimating mutual information directly from the noisy measurements themselves, using only a noise model and the observed data. No explicit object model is needed, and the physical constraints of the system are naturally included because they affect the measurement distribution.
How does your method estimate mutual information from measurements directly?
The key innovation is an information estimator that operates solely on the noisy measurements and a known noise model, without needing ground truth objects. The imaging system is modeled as an encoder (optical system) that maps objects to noiseless images, which are then corrupted by noise into measurements. The estimator uses these noisy measurements to compute a lower bound on mutual information. By relying on the noise model and the empirical distribution of the measurements, it can be applied to any physical system without requiring explicit object models. The estimator is also differentiable, allowing it to be used as a loss function for optimizing optical parameters (e.g., lens shape, sensor size). In practice, this enables designers to directly optimize the imaging hardware to maximize information content, producing designs that are provably better for any downstream task that relies on distinguishing objects.
What are the practical benefits of this information-based optimization for designing cameras and sensors?
Adopting information-driven design provides several concrete advantages. First, it gives a single, principled metric that unifies resolution, noise, sensitivity, and other factors into one number, making it easy to compare radically different hardware configurations. Second, optimization using this metric is more memory- and compute-efficient than end-to-end learning because it does not require training a separate network for each candidate design—only the estimator is needed. Third, the resulting hardware is task-agnostic: the same optimized sensor works equally well for classification, detection, or reconstruction, without re-optimizing the optics. Fourth, the method directly puts the focus on what matters for AI-driven systems: information content, rather than human interpretability. As shown in the NeurIPS 2025 paper, this approach matches the performance of end-to-end learned designs across four imaging domains while requiring significantly fewer computational resources.