PRISM: High-Resolution & Precise Counterfactual Medical Image Generation

Abstract

Developing reliable and generalizable deep learning systems for medical imaging faces significant obstacles due to spurious correlations, data imbalances, and limited text annotations in datasets. Addressing these challenges requires architectures robust to the unique complexities posed by medical imaging data. The rapid advancements in vision-language foundation models within the natural image domain prompt the question of how they can be adapted for medical imaging tasks. In this work, we present PRISM, a framework that leverages foundation models to generate high-resolution, language-guided medical image counterfactuals using Stable Diffusion. Our approach demonstrates unprecedented precision in selectively modifying spurious correlations (the medical devices) and disease features, enabling the removal and addition of specific attributes while preserving other image characteristics. Through extensive evaluation, we show how PRISM advances counterfactual generation and enables the development of more robust downstream classifiers for clinically deployable solutions. To facilitate broader adoption and research, we make our code publicly available at https://github.com/Amarkr1/PRISM..

Synthesized Images

PRISM can generate high-fidelity counterfactual medical images with precise control over specific attributes. Below we showcase a variety of examples demonstrating our model's capability to sythesize text given a text prompt (hover over the images to see the text prompt).

Cardiomegaly

Chest x-ray of a patient with severe cardiomegaly and lots of support devices

Healthy

Chest x-ray of a patient with no findings

Healthy

Chest x-ray of a patient with no findings and support devices

Cardiomegaly

Chest x-ray of a patient with cardiomegaly and no devices

Cardiomegaly & Pleural Effusion

Chest x-ray of a patient with cardiomegaly and pleural effusion

Pleural Effusion

Chest x-ray of a patient with pleural effusion

Pleural Effusion

Chest x-ray of a patient with pleural effusion and support devices

Method

            Key Contributions
            A novel pipeline for generating high-fidelity counterfactual medical images with unprecedented resolution and precision
A specialized language-guided stable diffusion model fine-tuned on diverse medical imaging datasets
A comprehensive evaluation framework for assessing the clinical accuracy and utility of generated counterfactual images

        

Figure 2: Overview of the PRISM pipeline. Given an input medical image and a text description of the desired edit, our model generates a counterfactual image that implements the requested changes while preserving patient-specific anatomical details.

Results

Devices --> No Devices

PRISM results visualization - device removal

Pleural Effusion --> Healthy

Cardiomegaly --> Healthy

Figure 3: Demonstration of PRISM's ability to perform precise counterfactual edits on chest X-rays. The GIFs show the progression from original images to counterfactual variants with specific pathologies added or removed while preserving anatomical integrity. Use the pause/play buttons to control the animations.

Our extensive evaluation demonstrates PRISM's ability to generate high-quality counterfactual medical images with unprecedented precision. The model successfully preserves patient-specific anatomical details while selectively modifying targeted disease features or removing spurious correlations as directed by language prompts.

Citation

@article{kumar2025prism, title={PRISM: High-Resolution \& Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion}, author={Kumar, Amar and Kriz, Anita and Havaei, Mohammad and Arbel, Tal}, year={2025} }

Acknowledgements

The authors are grateful for funding provided by the Natural Sciences and Engineering Research Council of Canada, the Canadian Institute for Advanced Research (CIFAR) Artificial Intelligence Chairs program, Mila - Quebec AI Institute, Google Research, Calcul Quebec, Fonds de recherche du Québec (FRQNT), and the Digital Research Alliance of Canada.