Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence

White paper Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence siemens-healthineers.com SIEMENS Healthineers Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence While a number of useful stand-alone artificial intelligence (AI) algorithms already are in clinical use for the interpretation of medical images, the broader implementation of the technology in everyday radiology requires versatile AI platforms that support and augment image reading and reporting for multiple organs and can be easily integrated into existing workflows and IT architectures. Such intelligent assistance systems with increasing degrees of autonomy promise to make radiology as a whole more efficient and precise, thereby absorbing the growing workload, and reducing diagnostic errors. An exemplary application is chest imaging, which often involves the evaluation of several organs and anatomical structures. Integrated AI platforms can be used here to automate a wide variety of biomarker-based measurements, to highlight and characterize suspicious lesions, and to prepare structured reports. This should also significantly reduce the number of missed pathological findings. Likewise, the interpretation of whole-body scans may increasingly benefit from multi-organ, multimodal AI assistance systems in the future. Contents Introduction 3 Toward a comprehensive use of AI in radiology 3 AI is already a partial reality in medical image interpretation today 3 Implementing AI on a broader basis: the need for integrated routine solutions 4 Tangible benefits of AI assistance: chest imaging – and beyond 6 Leveraging a learning system 7 Additional Resources 8 2 White paper · 10.2018 Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence Introduction a recent analysis by healthcare technology consulting firm Signify Research (Harris 2018). Toward a comprehensive use Strategies are now emerging to use such integrated AI platforms to, for example, assess of AI in radiology multiple anatomical structures on a chest CT more quickly and precisely, or to evaluate whole-body scans in the future. This would The interpretation of medical images with make AI a cross-sectional technology in the aid of artificial intelligence (AI) is becom- radiology – and would make AI support a ing a clinical reality. While the technology self-evident aspect of image interpretation. ranks undoubtedly among the most visionary topics at medical congresses and in media reporting, a number of specific AI algorithms have already proven their practicality and AI is already a partial benefits. reality in medical image Accordingly, the question is no longer whether interpretation today AI can in principle be advantageous for image analysis. Most experts already take this for granted. The challenge today is rather to Case studies from various fields of application promote the work of radiologists in an everyday show that AI can bring tangible advantages. context with intelligent software and thus to AI-powered image interpretation is already achieve a significant increase in added value partly reality. for the entire discipline. Indeed, AI in medical imaging promises to help improve the A well-known example is bone age assessment physician’s experience and prevent burnout, in children based on X-rays of the hand. For thereby safeguarding the important fourth example, Danish researchers have developed component of the “quadruple aim” in health- image-analysis software that has been approved care (Bodenheimer & Sinsky 2014). Some in Europe for some time and is now routinely specialists speak of radiology being “augmented” used for this purpose in around 100 clinics with AI (Liew 2018). (Thodberg et al. 2009, Thodberg 2017). According to a recent study with a deep-learning system, automated analysis can indeed provide age results as accurately as a time-consuming “The mainstream market will require image reading by a radiologist (Hyunkwang end-to-end AI-powered solutions, et al. 2017). Another algorithm has been proven to detect wrist fractures, thereby bringing the with proven productivity gains.” expertise of an orthopedic surgeon to the emergency department. (Deep neural network Source: Harris 2018 improves fracture detection by clinicians Robert Lindsey, Aaron Daluiski, Sumit Chopra, Alexander Lachapelle, Michael Mozer, This requires software platforms that support Serge Sicular, Douglas Hanel, Michael Gardner, imaging in an easy way for a wide variety of AnuragGupta, Robert Hotchkiss, Hollis Potter, organs and modalities. While existing AI Proceedings of the National Academy of applications are usually specialized stand-alone Sciences Oct 2018, 201806905; DOI:10.1073/ solutions for individual tasks, the broader pnas.1806905115). implementation of the technology will be based on versatile assistance systems that can No less remarkable is AI-supported tuberculosis be seamlessly integrated into existing work- screening, especially in regions with limited flows and IT architectures. “When considering medical resources and few radiologists. AI in medical imaging, a deep-learning In a number of developing countries today, algorithm on its own is not the total solution, machine analyses of chest X-rays are merely a component. The mainstream market efficiently used to identify people who have an will require end-to-end AI-powered solutions, increased likelihood of disease and need to with proven productivity gains,” emphasizes undergo further testing (Philipsen et al. 2015). 10.2018 · White paper 3 Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence A related approach to TB detection is that “Ultimately, the driver of clinical radiologists only examine chest radiographs that cannot be clearly classified by artificial adoption may reside in the neural networks (Lakhani & Sundaram 2017). implementation and availability of AI applications integrated into the Likewise, AI applications for other imaging PACS system at the reading station.” modalities have proven their value under everyday conditions. As a three-month clinical Source: Tang et al. 2018 implementation phase in a U.S. healthcare network has shown, head CTs can be evaluated in seconds using an intelligent algorithm to detect unknown intracranial bleeding and A crucial prerequisite for advancing the prioritize the scans for rapid interpretation by a implementation of AI and fully exploiting its radiologist, which in some cases could even benefits is therefore the availability of easy-to- save lives (Arbabshirani et al. 2018). Many use, comprehensive solutions for clinical routine. other AI applications – for example, for the This applies in particular to the large number of analysis of lung or liver cancers – are at a radiologists who generally welcome AI but do practical development stage, and a growing not see themselves as tech pioneers. “The number of algorithms are now approved for early-majority customers expect total solutions clinical use by authorities such as the U.S. Food for a given business or clinical problem, rather and Drug Administration. than discrete products and technologies. These solutions must seamlessly integrate into their existing infrastructure,” underscores a current market assessment (Harris 2018). Implementing AI on a broader basis: In particular, compatibility with existing picture archiving and communication systems (PACS) is key to the successful use of AI in healthcare the need for integrated organizations. “Ultimately, the driver of clinical routine solutions adoption may reside in the implementation and availability of AI applications integrated into the PACS system at the reading station,” While these examples underscore the enormous confirms a technology white paper from the potential of artificial intelligence for radiology, Canadian Association of Radiologists (Tang et its broader implementation in routine imag- al. 2018). In other words: AI should not ing is still pending. One of the challenges lies reinvent workflows, but instead improve and in the nature of the algorithms themselves, accelerate what radiologists do every day in as which usually only perform single tasks, such many different ways as possible. as the segmentation of a particular organ. “The traditional view of machine learning and In the meantime, various strategies for such a neural networks is that a given system can comprehensive implementation of AI are only solve one well-defined problem,” Bradley emerging. On the one hand, many software Erickson and his colleagues from Mayo Clinic providers enter into cooperations in order to write in an article on the state of deep learning coordinate their AI applications or to bundle them (Erickson et al. 2018). However, they continue, into market places. These “AI app stores” give “it is rare for an examination to have only hospitals access to curated algorithm libraries one question.” For example, the interpretation with applications for a wide variety of radio- of a thoracic CT can involve multiple questions logical issues (Harris 2017, Signify Research 2018). about several organs. This presents radiologists with the dilemma of either restricting the On the other hand, it is particularly suitable for use of AI to specific cases, or integrating vari- larger companies to design integrated AI assistance ous algorithms from different developers into systems from the outset that cover entire their IT systems, which in turn can compromise areas of imaging and provide multifunctional practicability and raise compatibility problems. support – a strategy that Siemens Healthineers also pursues on the basis of its extensive AI research and expertise (Ghesu et al. 2017, Liu et al. 2017, Yang D et al. 2017) (Fig. 1). 4 White paper · 10.2018 Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence Conceptual 2 Intelligent Dispatching framework for an integrated AI platform assisting multimodal multi-organ imaging 1 Multi-Modal Input The algorithms to be executed are automati- cally selected from a global library depend- ing on the data content. All data produced by any modality for any 3 Automated Results Generation examination is automatically sent to an integrated AI assistant. 000 4 Multi-Channel Output AI-powered image analyses (e.g. measure- ments) are performed and structured reports T are generated. 5 Continuous Improvement Results are dispatched to the appropriate target systems (e.g. PACS, RIS, EMR). web-based infrastructure and plug-ins in partner systems enable user feedback and Figure 1: The system, which comprises a collection of continuous improvements of algorithm continuously learning algorithms, integrates itself into functionality. the existing image-processing IT environment and can be cloud-based or installed locally. (PACS: Picture archiving and communication system; RIS: Radiology information system; EMR: Electronic medical records) 10.2018 · White paper 5 Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence Tangible benefits of Numerous current software developments aim, AI assistance: for instance, to facilitate detection and classification of lung nodules, thereby potentially chest imaging – and beyond improving cancer screening and diagnosis and minimizing false positive findings (Yang Y et al. 2018). The mere ability to automatically An exemplary case for AI-supported multi- locate suspicious lesions using AI-powered functional image analysis is chest imaging, lung segmentation (Humphries et al. 2018) and to measure their size in 2D and 3D could which is one of the most important radiological save an enormous amount of time. fields of work. In the U.S., for example, approximately 900 chest X-ray examinations and 90 chest CTs are performed per 1,000 Medicare Equally promising are algorithms for determining beneficiaries every year (Kamel et al. 2017). airway obstructions in COPD and the extent Lung cancer screening with low-dose CT of emphysema (Das et al. 2018) or for in particular could further increase the need quantifying the severity of pulmonary fibrosis for fast and reliable image interpretation in on CT scans (Humphries et al. 2017). Last but many countries worldwide in the coming years. not least, AI-supported 3D visualizations such as cinematic renderings can simplify the In general, experts assume that completely reading process and make it more intuitive independent diagnostic algorithms will find (Dappa et al. 2016). their way into radiological routines only in the medium to long term and that AI will, in the near “Radiologists tend to overlook the heart future, rather serve to accelerate workflows and facilitate image interpretation (Loria 2018). while interpreting a routine chest CT.” Indeed, many supporting functions can already Source: Kanza et al. 2016 be integrated into AI systems today, for example, to perform biomarker-based measurements for various organs on a thoracic CT automatically, to highlight anatomical and pathological Siemens Healthineers is currently developing structures, or to prepare reproducible structured software platforms that integrate many of reports. In this way, readily actionable informa- these algorithms for assisted multi-organ tion is provided (Fig. 2). image interpretation. An envisioned advantage of Exemplary benefits Guideline- and biomarker-based automatic measure- ments of multiple structures, such as aorta, lung nod- ules, heart, coronary tree, and vertebrae through comprehensive AI assistance in chest CT reading and reporting Automated consideration of prior studies, as well as progression quantification e.g. for lung nodules and aortic aneurysms Accelerating reading and Automated transfer of results into structured reports and reporting workflows export to appropriate IT systems, e.g. PACS, RIS, and EMR Highlighting of lung lobes and detected nodules, airways, heart structures, or vertebrae through cinematic rendering Making patient care more effective and efficient1 Characterization and classification of bone mineral density, vertebral fractures, emphysema, coronary plaque, cardiomegaly, and aortic aneurysms Figure 2: Exemplary benefits through comprehensive AI assistance in chest CT reading and reporting Integration of reference values and risk scores, such as Lung RADS, calcium score, or bone mineral density 1medcitynews.com/2018/04/how-radiologists-will-use-ai/ score, as a guidance for reading and reporting 6 White paper · 10.2018 Assisted Multi-Organ Image Interpretation in the Age of Artificial Intelligence such an organ-spanning AI system is that, for in the nature of intelligent algorithms them- example, cardiopulmonary diseases are easier selves that they learn by processing large to assess and incidental findings are less often amounts of data and adjust and optimize their missed. Due to their high spatial and temporal internal parameters. This optimization process resolution, modern scanners enable compre- is key, in particular when AI applications are hensive cardiothoracic evaluations even on used with different scanners or imaging non-ECG-synchronized, non-contrast-enhanced protocols, or in different patient populations. thoracic CTs (Marano et al. 2015). However, “radiologists tend to overlook the heart while It is therefore obvious that AI systems need interpreting a routine chest CT,” as Canadian regular, well-planned updates to take full radiologist René Kanza and his colleagues advantage of the technology. The FDA in the comment (Kanza et al. 2016). Indeed, up to U.S., for example, is currently developing a two-thirds of incidental cardiac findings regulatory framework to take into account this detectable on a noncardiac CT, such as coronary dynamic character of artificial intelligence, calcifications or aortic dilatation, remain and at the same time to be able to implement unmentioned in the radiological report (Secchi incremental technological developments et al. 2017, Balakrishnan et al. 2017). This safely and on the basis of their clinical benefit could probably be largely avoided by automat- (Petro & Lyapustina 2018, Miliard 2018). ed image analysis and AI-supported reporting. For AI-supported image interpretation, this The same is true for thoracic bone tumors or means that solutions that are already tangible metastases, which are by no means rare or and feasible today are poised for expansion unexpected findings on chest CTs but still and improvement in the future. Cloud-based tend to slip through the diagnostic net, with infrastructures and user feedback will allow potentially serious clinical consequences algorithms to be adapted at a fast pace, and (Jokerst et al. 2016). new applications to be integrated into AI systems. Assisted multi-organ image interpre- For metastasis detection in particular, it is tation thus constitutes a key milestone on the desirable to have assisting AI platforms path to comprehensive, AI-powered whole- available in the future to evaluate not only body imaging. images of individual body areas such as the chest, but also whole-body scans. For exam- ple, in advanced stages of breast or prostate cancer, metastases typically occur in the skeleton, or in the lungs, liver, or brain. The reliable detection and quantification of such tumor lesions is of enormous importance for i Information therapy and prognosis, but it is also labor- Want more insights for intensive and prone to errors. Here, further healthcare leadership? developed, integrated AI systems may siemens.com/executive-alliance significantly improve whole-body evaluation in the coming years. Leveraging a learning system Artificial intelligence is a learning technology. On the one hand, AI as a whole has great development potential. New architectures of artificial neural networks have made possible remarkable progress in image analysis in recent years and will most probably continue to do so in the future. On the other hand, it is 10.2018 · White paper 7 Additional Resources Analysis of Extent of Fibrosis at Baseline 22. Miliard M (April 30, 2018) “FDA chief and 15-Month Follow-up. Radiology sees big things for AI in healthcare”. 285:270-278 http://www.healthcareitnews.com/ 01. Arbabshirani MR, Fornwalt BK, 11. Humphries SM, Lynch DA, Charbonnier J news/fda-chief-sees-big-things-ai- Mongelluzzo GJ et al. (2018) Advanced machine learning in action: identification et al. (2018) Initial validation of an healthcare (accessed June 12, 2018) of intracranial hemorrhage on computed artificial intelligence radiology assistant 23. Petro LG, Lyapustina S (April 24, 2018) tomography scans of the head with for chest CT analysis. Abstract submitted “US FDA Approaches to Artificial clinical workflow integration. npj Digital to the 2018 RSNA Annual Meeting Intelligence”. https://www.lexology. Medicine 1, article no. 9 (unpublished). com/library/detail.aspx?g=4d832c9d- 12. Hyunkwang L, Shahein T, Jenny L et al. 87be-4307-a1f8-db20124399c8 02. Balakrishnan R, Nguyen B, Raad R et al. (2017) Fully Automated Deep Learning (accessed June 12, 2018) (2017) Coronary artery calcification is common on nongated chest computed System for Bone Age Assessment. J Digit 24. Philipsen RH, Sánchez CI, Maduskar P tomography imaging. Clin Cardiol Imaging 2017 30: 427-441 et al. (2015) Automated chest-radiography 40:498-502 13. Jokerst C, McFarland W, Swanson J, as a triage for Xpert testing in resource- constrained settings: a prospective 03. Bodenheimer T, Sinsky C (2014) From Mohammed TL (2016) Thoracic Bone Triple to Quadruple Aim: Care of the Tumors Every Radiologist Should Know. study of diagnostic accuracy and costs. Sci Rep 5:12215 Patient Requires Care of the Provider. Curr Probl Diagn Radiol 45:71-9 Ann Fam Med 12:573-576 14. Kamel SI, Levin DC, Parker L, Rao VM 25. Secchi F, Di Leo G, Zanardo M et al. (2017) Detection of incidental cardiac 04. Dappa E, Higashigaito K, Fornaro J et (2017) Utilization Trends in Noncardiac findings in noncardiac chest computed al. (2016) Cinematic rendering – an Thoracic Imaging, 2002-2014. J Am alternative to volume rendering for Coll Radiol 14:337-342 tomography. Medicine (Baltimore) 96:e7531 3D computed tomography imaging. 15. Kang G, Liu K, Hou B, Zhang N (2017) Insights Imaging 7:849-56 3D multi-view convolutional neural 26. Signify Research (March 12, 2018) 05. Das N, Topalovic M, Janssens W (2018) networks for lung nodule classification. Signify Research Show Report from ECR 2018. https://www.signifyresearch. Artificial intelligence in diagnosis of PLoS One 12:e0188290 net/healthcare-it/ecr-2018-post-show- obstructive lung disease: current status 16. Kanza RE, Allard C, Berube M (2016) report/ (accessed June 12, 2018) and future potential. Curr Opin Pulm Cardiac findings on non-gated chest Med 24:117-123 computed tomography: A clinical and 27. Tang A, Tam R, Cadrin-Chênevert A et al. 06. Erickson BJ, Korfiatis P, Kline TL et al. pictorial review. Eur J Radiol 85:435-51 (2018) Canadian Association of Radiologists White Paper on Artificial (2018) Deep Learning in Radiology: 17. Lakhani P, Sundaram B (2017) Deep Intelligence in Radiology. Can Assoc Does One Size Fit All? J Am Coll Radiol Learning at Chest Radiography: Radiol J 69:120-135 15:521-526 Automated Classification of Pulmonary 28. Thodberg HH, Kreiborg S, Juul A, 07. Ghesu FC, Georgescu B, Zheng Y et al. Tuberculosis by Using Convolutional Pedersen KD (2009) The BoneXpert (2017) Multi-Scale Deep Reinforcement Neural Networks. Radiology 284:574-582 method for automated determination of Learning for Real-Time 3D-Landmark 18. Liew C (2018) The future of radiology skeletal maturity. IEEE Trans Med Detection in CT Scans. IEEE Trans Pattern augmented with Artificial Intelligence: Imaging 28:52-66 Anal Mach Intell. doi: 10.1109/ A strategy for success. Eur J Radiol TPAMI.2017.2782687 [Epub ahead of 29. Thodberg HH (2017) “October 2017: 102:152-156 print] BoneXpert licensed to 100 hospitals” 19. Liu S, Xu D, Zhou SK et al. (2017) 3D (press release). https://www.bonexpert.com/ 08. Harris S (December 16, 2017) “AI at Anisotropic Hybrid Network: Transferring news/latest-news/74-october-2017- RSNA – What a Difference a Year Makes”. Convolutional Features from 2D Images bonexpert-licensed-to-100-hospitals https://www.signifyresearch.net/med- to 3D Anisotropic Volumes. https://arxiv. (accessed June 12, 2018) ical-imaging/ai-rsna-difference-year- org/abs/1711.08580 (accessed makes/ (accessed June 12, 2018) June 12, 2018) 30. Yang D, Xu D, Zhou SK et al. (2017) Automatic Liver Segmentation using an 09. Harris S (February 8, 2018) “Will AI in 20. Loria K (2018) “Putting the AI in Radiology”. Adversarial Image-to-Image Network. Medical Imaging ‘Cross the Chasm’?” Radiology Today Vol. 19 No. 1 P. 10. https://arxiv.org/abs/1707.08037 https://www.signifyresearch.net/medical- http://www.radiologytoday.net/archive/ (accessed June 12, 2018) imaging/will-ai-in-medical-imaging- rt0118p10.shtml (accessed June 12, 2018) cross-the-chasm/ (accessed June 12, 2018) 31. Yang Y, Feng X, Chi W et al. (2018) 21. Marano R, Pirro F, Silvestri V et al. (2015) Deep learning aided decision support 10. Humphries SM, Yagihashi K, Huck- Comprehensive CT cardiothoracic for pulmonary nodules diagnosing: leberry J et al. (2017) Idiopathic imaging: a new challenge for chest a review. J Thorac Dis 10(Suppl 7): Pulmonary Fibrosis: Data-driven Textural imaging. Chest 147:538-551 S867-S875 Siemens Healthineers Headquarters Siemens Healthcare GmbH Henkestr. 127 91052 Erlangen, Germany Phone: +49 9131 84-0 siemens-healthineers.com Published by Siemens Healthcare GmbH · Order No. xxxxxxxxxxx · Printed in Country · 0000.x · ©Siemens Healthcare GmbH, 2018