+
Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI

Nature Medicine volume 27, pages 1663–1665 (2021)Cite this article

Subjects

Access through your institution
Buy or subscribe

To the Editor — Over the next decade, systems that are centered on artificial intelligence (AI), particularly machine learning, are predicted to become key components of several workflows within the health sector. Medical diagnosis is seen as one of the first areas that may be revolutionized by AI innovations. Indeed, more than 90% of health-related AI systems that have reached regulatory approval by the US Food and Drug Administration belong to the field of diagnostics1.

In the current paradigm, most diagnostic investigations require interpretation from a clinician to identify the presence of a target condition — a crucial step in determining subsequent treatment strategies. Despite being an essential step in the provision of patient care, many health systems find it increasingly difficult to meet the demand for the interpretation of diagnostic tests. To address this issue, diagnostic AI systems have been characterized as medical devices that may alleviate the burden placed on diagnosticians: by serving as case triage tools, enhancing diagnostic accuracy and stepping in as a second reader when necessary. As AI-centered diagnostic test accuracy (AI DTA) studies emerge, there has been a concurrent rise in systematic reviews that amalgamate the findings of comparable studies.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Benjamens, S., Dhunnoo, P. & Meskó, B. npj Digit. Med. 3, 118 (2020).

    Article  Google Scholar 

  2. Jayakumar, S. et al. npj Digital Med. (in the press).

  3. Whiting, P. F. Ann. Intern. Med. 155, 529 (2011).

    Article  Google Scholar 

  4. Page, M. J. et al. BMJ. 372, n71 (2021).

    Article  Google Scholar 

  5. Yang, B. et al. Open Science Framework https://doi.org/10.17605/OSF.IO/HQ8MF (2018).

  6. Sounderajah, V. et al. Nat. Med. 26, 807–808 (2020).

    Article  CAS  Google Scholar 

  7. Collins, G. & Moons, K. Lancet 393, 1577–1579 (2019).

    Article  Google Scholar 

  8. Liu, X. & Rivera, S. C. Nat. Med. 26, 1364–1374 (2020).

    Article  CAS  Google Scholar 

  9. Harris, M. et al. PLoS One 14, e0226134 (2019).

    Google Scholar 

  10. Roberts, M. et al. Nat. Mach. Intell. 3, 199–217 (2021).

    Article  Google Scholar 

  11. The EQUATOR Network.Enhancing the QUAlity and Transparency Of Health Research; https://www.equator-network.org/ (accessed 27 September 2021).

Download references

Acknowledgements

Infrastructure support for this research was provided by the National Institute for Health Research (NIHR) Imperial Biomedical Research Centre (BRC). G.S.C. is supported by the NIHR Biomedical Research Centre and Cancer Research UK (programme grant C49297/A27294). D.T. is funded by National Pathology Imaging Co-operative, NPIC (project no. 104687), supported by a £50 million investment from the Data to Early Diagnosis and Precision Medicine strand of the government’s Industrial Strategy Challenge Fund, and managed and delivered by UK Research and Innovation (UKRI). F.G. is supported by the NIHR Applied Research Collaboration Northwest London. The views and opinions expressed herein are those of the authors and do not necessarily reflect the views of their employers or funders.

Author information

Author notes

  1. These authors jointly supervised this work: Patrick M. Bossuyt, Ara Darzi.

Authors and Affiliations

  1. Institute of Global Health Innovation, Imperial College London, London, UK

    Viknesh Sounderajah, Hutan Ashrafian, Dominic King, Leanne Harling & Ara Darzi

  2. Department of Surgery and Cancer, Imperial College London, London, UK

    Viknesh Sounderajah, Hutan Ashrafian, Leanne Harling, Pasha Normahani, Sheraz R. Markar & Ara Darzi

  3. Center for Health Policy and Center for Primary Care and Outcomes Research, Stanford University, Stanford, CA, USA

    Sherri Rose

  4. Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA

    Nigam H. Shah

  5. Institute for Medical Engineering & Science, Massachusetts Institute of Technology, Cambridge, MA, USA

    Marzyeh Ghassemi

  6. Journal of the American Medical Association (JAMA), Chicago, IL, USA

    Robert Golub

  7. University of Pennsylvania, Philadelphia, Pennsylvania, PA, USA

    Charles E. Kahn Jr

  8. Salesforce Research, San Francisco, CA, USA

    Andre Esteva

  9. Google Health, Palo Alto, CA, USA

    Alan Karthikesalingam, Dale Webster & Shravya Shetty

  10. Wellcome Trust, London, UK

    Bilal Mateen

  11. Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore

    Dan Milea & Daniel Ting

  12. Leeds Teaching Hospitals NHS Trust, Leeds, UK

    Darren Treanor

  13. University of Leeds, Leeds, UK

    Darren Treanor

  14. Department of Clinical Pathology, and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden

    Darren Treanor

  15. Center for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden

    Darren Treanor

  16. NHSX, London, UK

    Dominic Cushnan

  17. Optum, London, UK

    Dominic King

  18. Medicines and Healthcare Products Regulatory Agency, London, UK

    Duncan McPherson & Johan Ordish

  19. Faculty of Engineering, Department of Computing, Imperial College London, London, UK

    Ben Glocker

  20. National Institute for Health and Care Excellence, London, UK

    Felix Greaves

  21. Department of Pediatrics, Centre of Research in Epidemiology and Statistics, Inserm UMR 1153, Necker- Enfants Malades Hospital, Assistance Publique-Hôpitaux de Paris, Université de Paris, Paris, France

    Jérémie F. Cohen

  22. Institute of Applied Health Research, University of Birmingham, Birmingham, UK

    Jon Deeks

  23. Department of Epidemiology and Data Science, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, The Netherlands

    Mariska Leeflang & Patrick M. Bossuyt

  24. Food and Drug Administration, Silver Spring, MD, USA

    Matthew Diamond

  25. Departments of Radiology and Epidemiology, University of Ottawa, The Ottawa Hospital Research Institute, Ottawa, Ontario, Canada

    Matthew D. F. McInnes

  26. Department of Bioethics, The Hospital for Sick Kids, Toronto, Ontario, Canada

    Melissa McCradden

  27. Department of Ophthalmology and Visual Sciences, University of Iowa, Iowa City, IA, USA

    Michael D. Abràmoff

  28. Annals of Internal Medicine, American College of Physicians, Philadelphia, PA, USA

    Stephanie Chang

  29. Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK

    Xiaoxuan Liu & Alastair Denniston

  30. University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK

    Xiaoxuan Liu & Alastair Denniston

  31. Health Data Research UK, London, UK

    Xiaoxuan Liu & Alastair Denniston

  32. Centre for Medical Imaging, University College London, London, UK

    Susan Mallett

  33. Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK

    Gary S. Collins

  34. NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK

    Gary S. Collins

  35. Ottawa Hospital Research Institute, Ottawa, Ontario, Canada

    David Moher

  36. Bristol Medical School, University of Bristol, Bristol, UK

    Penny Whiting

Authors
  1. Viknesh Sounderajah
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Hutan Ashrafian
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Sherri Rose
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Nigam H. Shah
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Marzyeh Ghassemi
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Robert Golub
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Charles E. Kahn Jr
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Andre Esteva
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Alan Karthikesalingam
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Bilal Mateen
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Dale Webster
    View author publications

    You can also search for this author inPubMed Google Scholar

  12. Dan Milea
    View author publications

    You can also search for this author inPubMed Google Scholar

  13. Daniel Ting
    View author publications

    You can also search for this author inPubMed Google Scholar

  14. Darren Treanor
    View author publications

    You can also search for this author inPubMed Google Scholar

  15. Dominic Cushnan
    View author publications

    You can also search for this author inPubMed Google Scholar

  16. Dominic King
    View author publications

    You can also search for this author inPubMed Google Scholar

  17. Duncan McPherson
    View author publications

    You can also search for this author inPubMed Google Scholar

  18. Ben Glocker
    View author publications

    You can also search for this author inPubMed Google Scholar

  19. Felix Greaves
    View author publications

    You can also search for this author inPubMed Google Scholar

  20. Leanne Harling
    View author publications

    You can also search for this author inPubMed Google Scholar

  21. Johan Ordish
    View author publications

    You can also search for this author inPubMed Google Scholar

  22. Jérémie F. Cohen
    View author publications

    You can also search for this author inPubMed Google Scholar

  23. Jon Deeks
    View author publications

    You can also search for this author inPubMed Google Scholar

  24. Mariska Leeflang
    View author publications

    You can also search for this author inPubMed Google Scholar

  25. Matthew Diamond
    View author publications

    You can also search for this author inPubMed Google Scholar

  26. Matthew D. F. McInnes
    View author publications

    You can also search for this author inPubMed Google Scholar

  27. Melissa McCradden
    View author publications

    You can also search for this author inPubMed Google Scholar

  28. Michael D. Abràmoff
    View author publications

    You can also search for this author inPubMed Google Scholar

  29. Pasha Normahani
    View author publications

    You can also search for this author inPubMed Google Scholar

  30. Sheraz R. Markar
    View author publications

    You can also search for this author inPubMed Google Scholar

  31. Stephanie Chang
    View author publications

    You can also search for this author inPubMed Google Scholar

  32. Xiaoxuan Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  33. Susan Mallett
    View author publications

    You can also search for this author inPubMed Google Scholar

  34. Shravya Shetty
    View author publications

    You can also search for this author inPubMed Google Scholar

  35. Alastair Denniston
    View author publications

    You can also search for this author inPubMed Google Scholar

  36. Gary S. Collins
    View author publications

    You can also search for this author inPubMed Google Scholar

  37. David Moher
    View author publications

    You can also search for this author inPubMed Google Scholar

  38. Penny Whiting
    View author publications

    You can also search for this author inPubMed Google Scholar

  39. Patrick M. Bossuyt
    View author publications

    You can also search for this author inPubMed Google Scholar

  40. Ara Darzi
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

V.S., S.R., N.H.S., M.G., R.G., C.E.K., X.L., G.S.C., D.W., A.E., H.A., D. Milea, D. McPherson, J.O., D. Treanor, J.F.C., M.L., M.M., M.D.F.M., M.D.A., S.M., P.W. and P.M.B. prepared the first draft of the manuscript. Critical edits, further direction and feedback have been attained from all co-authors (including A.K., B.M., D. Ting, D.C., D.K., F.G., L.H., J.D., M.D., P.N., S.M., S.C., S.S., A.D., D.M. and A.D.). The study described in the manuscript has been conceptualized, discussed and agreed upon between all co-authors.

Corresponding authors

Correspondence to Viknesh Sounderajah, Patrick M. Bossuyt or Ara Darzi.

Ethics declarations

Competing interests

A.K., S.S. and D.W. are employees at Google. A.D. and H.A. are employees at Flagship Pioneering UK Ltd. A.E. is an employee at Salesforce. DK is an employee at Optum. None of the other authors have any competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sounderajah, V., Ashrafian, H., Rose, S. et al. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med 27, 1663–1665 (2021). https://doi.org/10.1038/s41591-021-01517-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-021-01517-0

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research
点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载