Abstract
Spatial transcriptomics (ST) has demonstrated enormous potential for generating intricate molecular maps of cells within tissues. Here we present iStar, a method based on hierarchical image feature extraction that integrates ST data and high-resolution histology images to predict spatial gene expression with super-resolution. Our method enhances gene expression resolution to near-single-cell levels in ST and enables gene expression prediction in tissue sections where only histology images are available.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
We analyzed the following publicly available ST datasets: (1) 10x Xenium human breast cancer data (https://www.10xgenomics.com/products/xenium-in-situ/preview-dataset-human-breast); (2) 10x Xenium mouse brain data (https://www.10xgenomics.com/resources/datasets/fresh-frozen-mouse-brain-replicates-1-standard); (3) human HER2-positive breast cancer ST data reported in Anderson et al. (https://github.com/almaan/her2st); (4) 10x Visium human breast cancer data (https://www.10xgenomics.com/resources/datasets/human-breast-cancer-visium-fresh-frozen-whole-transcriptome-1-standard); (5) 10x Visium human colorectal cancer data (https://www.10xgenomics.com/resources/datasets/human-colorectal-cancer-whole-transcriptome-analysis-1-standard-1-2-0); (6) 10x Visium human prostate cancer data (https://www.10xgenomics.com/resources/datasets/human-prostate-cancer-adenocarcinoma-with-invasive-carcinoma-ffpe-1-standard-1-3-0); (7) human prostate cancer data reported in Erickson et al. (https://doi.org/10.17632/svw96g68dv.1); (8) human clear cell renal cell carcinoma primary tumors reported in Meylan et al. (GSE175540); (9) 10x Visium mouse kidney data (https://www.10xgenomics.com/resources/datasets/adult-mouse-kidney-ffpe-1-standard-1-3-0); (10) 10x Visium mouse brain coronal cut data (https://www.10xgenomics.com/resources/datasets/mouse-brain-coronal-section-2-ffpe-2-standard); (11) 10x Visium mouse brain sagittal cut posterior data (https://www.10xgenomics.com/resources/datasets/mouse-brain-serial-section-2-sagittal-posterior-1-standard); (12) 10x Visium mouse brain olfactory bulb data (https://www.10xgenomics.com/resources/datasets/adult-mouse-olfactory-bulb-1-standard-1). Details of the datasets analyzed in this paper are described in Supplementary Table 3. Gene expression visualizations for other spatial resolutions in the 10x Xenium breast cancer and mouse brain data are available at https://zenodo.org/doi/10.5281/zenodo.10071636. Source data are provided with this paper.
Code availability
The iStar algorithm was implemented in Python and is available on GitHub at https://github.com/daviddaiweizhang/istar.
References
Burgess, D. J. Spatial transcriptomics coming of age. Nat. Rev. Genet. 20, 317 (2019).
Asp, M., Bergenstrahle, J. & Lundeberg, J. Spatially resolved transcriptomes-next generation tools for tissue exploration. Bioessays 42, e1900221 (2020).
Crosetto, N., Bienko, M. & van Oudenaarden, A. Spatially resolved transcriptomics and beyond. Nat. Rev. Genet. 16, 57–66 (2015).
Moor, A. E. & Itzkovitz, S. Spatial transcriptomics: paving the way for tissue-level systems biology. Curr. Opin. Biotechnol. 46, 126–133 (2017).
Hu, J. et al. SpaGCN: integrating gene expression, spatial location and histology to identify spatial domains and spatially variable genes by graph convolutional network. Nat. Methods 18, 1342–1351 (2021).
Sun, S., Zhu, J. & Zhou, X. Statistical analysis of spatial expression patterns for spatially resolved transcriptomic studies. Nat. Methods 17, 193–200 (2020).
Svensson, V., Teichmann, S. A. & Stegle, O. SpatialDE: identification of spatially variable genes. Nat. Methods 15, 343–346 (2018).
Dries, R. et al. Giotto: a toolbox for integrative analysis and visualization of spatial expression data. Genome Biol. 22, 78 (2021).
Pham, D. et al. stLearn: integrating spatial location, tissue morphology and gene expression to find cell types, cell–cell interactions and spatial trajectories within undissociated tissues. Preprint at bioRxiv https://doi.org/10.1101/2020.05.31.125658 (2020).
Asp, M. et al. A spatiotemporal organ-wide gene expression and cell atlas of the developing human heart. Cell 179, 1647–1660 e1619 (2019).
Wang, X. et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science 361, eaat5691 (2018).
Lubeck, E., Coskun, A. F., Zhiyentayev, T., Ahmad, M. & Cai, L. Single-cell in situ RNA profiling by sequential hybridization. Nat. Methods 11, 360–361 (2014).
Shah, S., Lubeck, E., Zhou, W. & Cai, L. In situ transcription profiling of single cells reveals spatial organization of cells in the mouse hippocampus. Neuron 92, 342–357 (2016).
Eng, C. L. et al. Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH. Nature 568, 235–239 (2019).
Moffitt, J. R. et al. Molecular, spatial, and functional single-cell profiling of the hypothalamic preoptic region. Science 362, eaau5324 (2018).
Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S. & Zhuang, X. RNA imaging. Spatially resolved, highly multiplexed RNA profiling in single cells. Science 348, aaa6090 (2015).
Stickels, R. R. et al. Highly sensitive spatial transcriptomics at near-cellular resolution with Slide-seqV2. Nat. Biotechnol. 39, 313–319 (2020).
Chen, A. et al. Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays. Cell 185, 1777–1792 e1721 (2022).
Badea, L. & Stanescu, E. Identifying transcriptomic correlates of histology using deep learning. PLoS ONE 15, e0242858 (2020).
Ash, J. T., Darnell, G., Munro, D. & Engelhardt, B. E. Joint analysis of expression levels and histological images identifies genes associated with tissue morphology. Nat. Commun. 12, 1609 (2021).
Schmauch, B. et al. A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nat. Commun. 11, 3877 (2020).
Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 16144–16155 (IEEE, 2022).
Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In Proc. IEEE/CVF International Conference on Computer Vision, 10012–10022 (2021).
Han, K. et al. Transformer in transformer. Adv. Neural Inf. Process. Syst. 34, 15908–15919 (2021).
Caron, M. et al. Emerging properties in self-supervised vision transformers. In Proc. IEEE/CVF International Conference On Computer Vision 9650–9660 (2021).
Bao, H., Dong, L., Piao, S. & Wei, F. Beit: Bert pre-training of image transformers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. (ICLR, 2022).
Janesick, A. et al. High resolution mapping of the breast cancer tumor microenvironment using integrated single cell, spatial and in situ analysis of FFPE tissue. Preprint at bioRxiv https://doi.org/10.1101/2022.10.06.510405 (2022).
Bergenstråhle, L. et al. Super-resolved spatial transcriptomics by deep data fusion. Nat. Biotechnol. 40, 476–479 (2022).
Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).
Hamerly, G. & Elkan, C. Learning the k in k-means. Adv. Neural Inf. Process. Syst. 16, 281–288 (2003).
Andersson, A. et al. Spatial deconvolution of HER2-positive breast cancer delineates tumor-associated cell type interactions. Nat. Commun. 12, 6012 (2021).
Stahl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353, 78–82 (2016).
Wu, S. Z. et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet. 53, 1334–1347 (2021).
Sautes-Fridman, C., Petitprez, F., Calderaro, J. & Fridman, W. H. Tertiary lymphoid structures in the era of cancer immunotherapy. Nat. Rev. Cancer 19, 307–325 (2019).
Fridman, W. H. et al. B cells and tertiary lymphoid structures as determinants of tumour immune contexture and clinical outcome. Nat. Rev. Clin. Oncol. 19, 441–457 (2022).
Helmink, B. A. et al. B cells and tertiary lymphoid structures promote immunotherapy response. Nature 577, 549–555 (2020).
Petitprez, F. et al. B cells are associated with survival and immunotherapy response in sarcoma. Nature 577, 556–560 (2020).
Cabrita, R. et al. Tertiary lymphoid structures improve immunotherapy and survival in melanoma. Nature 577, 561–565 (2020).
Lin, J. R. et al. Multiplexed 3D atlas of state transitions and immune interaction in colorectal cancer. Cell 186, 363–381 e319 (2023).
benchmark. University of Pennsylvania https://upenn.box.com/v/istar-results-benchmark
Steiner, A. et al. How to train your ViT? Data, augmentation, and regularization in vision transformers. Trans. Mach. Learn Res. https://openreview.net/pdf?id=4nPswr1KcP (2022).
Hölscher, D. L. et al. Next-generation morphometry for pathomics-data mining in histopathology. Nat. Commun. 14, 470 (2023).
Rappez, L. et al. SpaceM reveals metabolic states of single cells. Nat. Methods 18, 799–805 (2021).
Xu, B., Wang, N., Chen, T. & Li, M. Empirical evaluation of rectified activations in convolutional network. Preprint at arXiv https://doi.org/10.48550/arXiv.1505.00853 (2015).
Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. (ICLR, 2016).
Ringnér, M. What is principal component analysis? Nat Biotechnol. 26, 303–304 (2008).
McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at arXiv https://doi.org/10.48550/arXiv.1802.03426 (2018).
Schurch, C. M. et al. Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front. Cell 182, 1341–1359 e1319 (2020).
Aran, D. et al. Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage. Nat. Immunol. 20, 163–172 (2019).
Hu, J. et al. Iterative transfer learning with neural network for clustering and cell type classification in single-cell RNA-seq analysis. Nat. Mach. Intell. 2, 607–618 (2020).
Lu, Y. The level weighted structural similarity loss: a step away from MSE. In Proc. AAAI Conference on Artificial Intelligence 33, 9989–9990 (2019).
Lai, W.-S., Huang, J.-B., Ahuja, N. & Yang, M.-H. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 624–632 (2017).
Dahl, R., Norouzi, M. & Shlens, J. Pixel recursive super resolution. Proc. IEEE International Conference on Computer Vision 5439–5448 (2017).
Masutani, E. M., Bahrami, N. & Hsiao, A. Deep learning single-frame and multiframe super-resolution for cardiac MRI. Radiology 295, 552–561 (2020).
Anwar, S., Khan, S. & Barnes, N. A deep journey into super-resolution: a survey. ACM Comput. Surv. 53, 1–34 (2020).
Wang, Z., Chen, J. & Hoi, S. C. Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3365–3387 (2020).
Ma, Y. & Zhou, X. Spatially informed cell-type deconvolution for spatial transcriptomics. Nat. Biotechnol. 40, 1349–1359 (2022).
Acknowledgements
M.L. was supported by the following NIH grants: R01GM125301, R01EY030192, R01HL150359, R01HG013185 and P01AG066597. E.B.L. was supported by NIH grant P01AG066597. L.W. was supported in part by NIH grant R01CA266280, the Cancer Prevention and Research Institute of Texas (CPRIT) award RP200385, the University Cancer Foundation via the Institutional Research Grant Program at the University of Texas MD Anderson Cancer Center, the Andrew Sabin Family Foundation, and the Break Through Cancer Foundation. We thank M. Meylan and W. H. Fridman for sharing the kidney cancer histology image data. We also thank Erickson, Lamb and Lundberg for sharing the prostate cancer histology image and clone annotation data.
Author information
Authors and Affiliations
Contributions
This study was conceived of and led by M.L. D.Z. designed the model and algorithm, implemented the iStar software and led data analyses with input from M.L., E.E.F., L.W., K.S., G.X.X., M.D.F. and E.B.L. E.E.F. examined histology images. A.S., H.Y., M.Y.Y.L., K.S.C. and J.H. helped with data analyses. L.W. provided marker genes for TLS detection and interpreted results related to cancer. D.Z. and M.L. wrote the paper with feedback from the other co-authors.
Corresponding authors
Ethics declarations
Competing interests
M.L. receives research funding from Biogen Inc. unrelated to the current manuscript. The other authors declare no competing financial interests.
Peer review
Peer review information
Nature Biotechnology thanks the anonymous reviewers for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Correlation between super-resolution gene expression and histology.
Comparison of iStar’s super-resolution gene expression patterns with the paired histology image in the Xenium-derived pseudo-Visium data. Spot boundaries are highlighted.
Extended Data Fig. 2 Resolution enhancement at various scales.
Visualization of predicted super-resolution gene expressions by iStar at various scales of resolution enhancement for three breast cancer-related genes (ESR1, ERBB2, and PGR) in the Xenium-derived pseudo-Visium data.
Extended Data Fig. 3 Numerical evaluation of prediction accuracy.
Prediction accuracy of iStar and XFuse as measured by root mean squared error (RMSE) and structural similarity index measure (SSIM) for all the 313 genes in the Xenium-derived pseudo-Visium data obtained from a breast cancer patient. In each scatter plot, a dot represents a gene. In this analysis, Section 1 was first treated as the training sample in which in-sample prediction was perfromed for Section 1 and then out-of-sample prediction was perfromed for Section 2. We then repeated this analysis by treating Section 2 as the ‘in-sample’ and Section 1 as the ‘out-of-sample’. The evaluation metrics in’Average’ is the average of those in ‘Sections 1’ and ‘Section 2’.
Extended Data Fig. 4 Visualization of single-cell level gene expression prediction.
Visualization of single-cell level gene expression predicted by iStar for the pseudo-Visium breast cancer data derived from Xenium data. Shown on the left is the ground truth single-cell level gene expression directly measured by Xenium, and shown on the right is the single-cell level gene expression predicted by iStar. For each gene, the root mean squared error (RMSE) and Pearson’s correlation coefficient (PCC) between the prediction and the ground truth across the whole tissue and within the shown region are displayed.
Extended Data Fig. 5 Prediction accuracy evaluation of single-cell level gene expression prediction.
Prediction accuracy of iStar and XFuse for single-cell level gene expression prediction as measured by root mean squared error (RMSE) for all the 313 genes in the Xenium-derived pseudo-Visium data obtained from a breast cancer patient. In each scatter plot, a dot represents a gene. In this analysis, Section 1 was first treated as the training sample in which in-sample prediction was performed for Section 1 and then out-of-sample prediction was performed for Section 2. We then repeated this analysis by treating Section 2 as the ‘in-sample’ and Section 1 as the ‘out-of-sample’. The evaluation metrics in’Average’ is the average of those in ‘Sections 1’ and ‘Section 2’. We stratified cells by the quantiles of their cell size.
Extended Data Fig. 6 Super-resolution vs spot-level signature scoring of tertiary lymphoid structures (TLSs).
iStar detected tertiary lymphoid structures (TLSs) in the Anderson et al. (2021) HER2+ breast cancer dataset. Displayed are the predicted TLS scores by iStar and the original publication, along with the pathologist’s manual annotation reported in the original publication. Super-resolution was performed with 128x resolution enhancement.
Extended Data Fig. 7 Super-resolution gene expression prediction in the Xenium-derived pseudo-Visium mouse brain data.
Visualization of the spot-level training data, ground truth gene expression, and predicted super-resolution gene expressions by iStar and XFuse for 24 highly variable genes, whose variances are in the 80%-100% quantiles among all the 248 genes in the Xenium-derived pseudo-Visium data obtained from mouse brain. The variance quantiles of the genes in the order of top-left, top-right, bottom-left, bottom-right are equally spaced from 100% to 80% (in descending order). Super-resolution gene expressions are visualized at the scale of 8x resolution enhancement.
Extended Data Fig. 8 Prediction accuracy evaluation and gene-based segmentation comparison in the Xenium-derived pseudo-Visium mouse brain data.
a. Prediction accuracy of by iStar and XFuse as measured by root mean squared error (RMSE) and structural similarity index measure (SSIM) for all the 248 genes in the Xenium-derived pseudo-Visium data obtained from mouse brain. In each scatter plot, a dot represents one gene. b. Segmentation of the Xenium-derived pseudo-Visium data obtained from mouse brain by iStar and XFuse using all 248 genes available in this dataset. Super-resolution was performed with 128x resolution enhancement.
Extended Data Fig. 9 Gene-based segmentation of mouse brain datasets.
Analyses of a. mouse brain (coronal cut), b. mouse brain posterior (sagittal cut), and c. mouse brain ofactory bulb Visium datasets generated by 10x Genomics. For each dataset, iStar was applied to enhance the resolution of the top 1000 most highly variable genes. Super-resolution was performed with 128x resolution enhancement. Segmentations by iStar identified fine-grained tissue structures in the mouse brain and agreed with the Allen Brain Atlas annotations.
Extended Data Fig. 10 Gene-based tissue segmentation and tertiary lymphoid structure (TLS) signature scoring of cancer datasets and a mouse kidney dataset.
Comparison of gene-based segmentation by iStar with manual tissue annotation in a. mouse kidney and b. prostate cancer Visium datasets generated by 10x Genomics. c. Gene-based segmentation by iStar in a colorectal cancer Visium dataset by 10x Genomics. d. Comparison of tertiary lymphoid structure (TLS) signature score by iStar with manual TLS annotation in a kidney cancer Visium dataset generated by Meylan et al. (2022).
Supplementary information
Supplementary Information
Supplementary Tables 1–3 and Figs. 1–24.
Source data
Source Data Extended Data Fig. 3
Statistical source data.
Source Data Extended Data Fig. 5
Statistical source data.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, D., Schroeder, A., Yan, H. et al. Inferring super-resolution tissue architecture by integrating spatial transcriptomics with histology. Nat Biotechnol 42, 1372–1377 (2024). https://doi.org/10.1038/s41587-023-02019-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41587-023-02019-9
This article is cited by
-
Spotiphy enables single-cell spatial whole transcriptomics across an entire section
Nature Methods (2025)
-
Benchmarking the translational potential of spatial gene expression prediction from histology
Nature Communications (2025)
-
Profiling cell identity and tissue architecture with single-cell and spatial transcriptomics
Nature Reviews Molecular Cell Biology (2025)
-
Systematic inference of super-resolution cell spatial profiles from histology images
Nature Communications (2025)
-
ImSpiRE: image feature-aided spatial resolution enhancement method
Science China Life Sciences (2025)