WO2023063845A1 - System and method for using automated machine learning (automl) to train computer vision models for analyzing biomedical images - Google Patents

Abstract

The invention relates to a system and method for using automated machine learning (AutoML) to train computer vision models for analyzing biomedical images. The system comprises a database of biomedical images, and a server which: uploads biomedical image data for testing, training and validating computer vision models; transforms biomedical image data into the required format for searching, training and evaluation; searches computer vision models with the aid of a training set and a test set; searches and optimizes the architecture parameters of the models found; trains computer vision models with the aid of a training set generated on the basis of biomedical image data; selects the best training model; evaluates the selected best computer vision model with the aid of a validation set generated on the basis of biomedical image data.

Description

SYSTEM AND METHOD FOR AUTOMATED MACHINE LEARNING (AutoML) OF COMPUTER VISION MODELS FOR ANALYSIS OF BIOMEDICAL IMAGES

FIELD OF TECHNOLOGY

The invention relates to the field of information and communication technologies for processing medical data, in particular, to a system and method for automatic machine learning (AutoML) of computer vision models for analyzing biomedical images.

The presented solution can be used in medical decision support systems (DMSS), by doctors, for example, CT diagnostic doctors, MRI doctors, radiologists, radiologists, mammologists, oncologists and other specialists who analyze biomedical images obtained using various diagnostic methods. (e.g. CT scans, MPT scans, ultrasound scans, x-rays, mammography, etc.).

BACKGROUND OF THE INVENTION

Patent US10282835B2, publication date 05/07/2019, describes a method and system for automatic analysis of clinical images using models developed using machine learning. The system includes a server with an electronic processor and an interface for communication with the data source. The electronic processor is configured to receive training information from a data source via an interface. The training information includes a plurality of images and graphic reports associated with each of the plurality of images. Each graphical report includes a graphic marker denoting a portion of one of the plurality of images and diagnostic information associated with a portion of one of the plurality of images. The electronic processor is also configured to perform machine learning to develop a model using the training information. The electronic processor is also configured to receive images for analysis and automatic image processing using a model to generate a diagnosis for the image.

International application WO2021035412A1, publication date 03/04/2021 describes an automatic machine learning (AutoML) method. The method includes: receiving the user's target and the first data set by the AutoML system; determining, according to the target, that the original artificial intelligence (AI) model is used to implement the user's target; training the AutoML system, according to the obtained first data set, the initial AI model to obtain the trained AI model; further analyzing, according to the first data set, the training of the initial AI model to obtain an analysis result, the analysis result including the effect of at least one type of data in the first data set on the training of the initial AI model. An AutoML system is also described, providing, depending on the analysis result and the user, an optimization mode for the trained AI model, while the optimization mode can load a second data set to optimize the trained AL model. With this solution, according to the analysis of the training of the original AI model, the optimization mode provided by the AutoML system to the user can effectively optimize the degree of prediction accuracy of the AL model

However, in these solutions there is no automatic search for several computer vision models for biomedical image analysis, training of the found several models and selection of the best of the trained models for its subsequent evaluation, and there is also no automatic collection of biomedical images.

The technical problem to be solved by the claimed invention is the development of methods and systems for automatic training of computer vision models for tasks related to biomedical images, the development of automated methods for evaluating and validating trained models, the development of a data and markup management system to provide the AutoML process, increasing accuracy of AutoML machine learning models for biomedical image analysis.

SUMMARY OF THE INVENTION

The technical result of the claimed invention is to expand the arsenal of technical tools for automating the creation of models machine learning for biomedical image analysis (e.g. CT scans, MPT scans, ultrasound scans, x-rays, mammography, angiography, and others), improving the accuracy of biomedical image analysis by choosing the best model, reducing biomedical image analysis time by automating search, training and evaluation of computer vision models, increasing the speed of processing a large number of biomedical images simultaneously with increasing accuracy, increasing the ability to adapt computer vision models to new cases, devices, research modes, etc. - for example, to the emergence of a large number of CT studies with signs of viral pneumonia, increasing the scalability of the processes of building computer vision models in the tasks of analyzing biomedical images, reducing the participation of researchers in building computer vision models in the tasks of analyzing biomedical images and, thereby, saving on the most scarce resource - human expertise, improving the quality of the resulting computer vision models in the analysis of biomedical images by automating the study of the space of configurations of computer vision models and training parameters.

This technical result is achieved due to the fact that

A computer-implemented automatic machine learning (AutoML) system for computer vision models for biomedical image analysis comprises: a database, the database storing biomedical image data; moreover, the data on the basis of which the biomedical image data is obtained is collected automatically; server containing:

- a loader, wherein the loader automatically loads the biomedical image data required for testing, training and validating computer vision models from the database;

- a transformation unit, wherein the transformation unit automatically transforms the biomedical image data received from the download unit into a format accepted by the search, learning and evaluation units; - a search unit, and with the help of the search unit, computer vision models are automatically searched using training and test samples generated on the basis of biomedical image data received from the transformation unit, and the architecture parameters of the found models are automatically searched and optimized;

- a training unit, wherein the training unit automatically trains the computer vision models found by the search unit using a training sample generated on the basis of biomedical image data received from the transformation unit and using architecture parameters received from the search unit; wherein the best of said trained models is automatically selected and the selected model is passed to the estimator;

- an estimator, wherein the estimator automatically evaluates the best selected computer vision model trained by the training unit using a validation set formed on the basis of biomedical image data received from the transformation unit.

In the system, data collection can be automatically carried out using a clinic agent, on the basis of which biomedical image data is obtained.

In the system, using the search block, a model can be searched until the specified metric values are reached or until the search budget is exhausted.

In the system, using the search learning unit, additional training of the computer vision model found by the search unit can be carried out using a training sample, which is supplemented with data from additional biomedical images received from the transformation unit, if the specified model has not passed validation.

In the system, the training and evaluation units may be configured to initiate a repeated search and training process for computer vision models for biomedical image analysis.

In a computer-implemented method for automatic machine learning (AutoML) of computer vision models for biomedical image analysis:

- automatically load biomedical image data required for testing, training and validation of computer models vision; moreover, the data on the basis of which the biomedical image data is obtained is collected automatically;

- automatically perform the transformation of the downloaded biomedical image data into a format accepted for automatic search, training and evaluation;

- automatically search for computer vision models using training and test sets formed on the basis of transformed biomedical image data, and automatically search and optimize the architecture parameters of the found models;

- automatically perform training of the found computer vision models using a training set formed on the basis of the transformed data of biomedical images, and using the found and optimized parameters of the architectures of the found models; and automatically choose the best of these trained models;

- automatically evaluate the best selected trained computer vision model using a validation set formed on the basis of the transformed biomedical image data.

In the method, loading of biomedical image data can be automatically performed using a loading block, automatic collection of data from which biomedical image data is obtained can be performed using a clinic agent, transformation of the loaded biomedical image data can be automatically performed using a transformation block, the computer vision models can be automatically searched with the search block, the found computer vision models can be automatically trained with the training block, the best selected trained computer vision model can be automatically evaluated with the evaluation block.

The method can be used to search for a model until the specified values of the metrics are reached or until the search budget is exhausted.

In the method, additional training of the computer vision model found by the search unit can be carried out using a training set, in which additional biomedical image data received from the transformation unit is added, if the specified model has not passed the validation. The method may further initiate a repeated process of searching and training computer vision models for biomedical image analysis.

DESCRIPTION OF THE DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention.

The claimed invention is illustrated by figures 1-6, which depict:

Fig. 1 illustrates an example of a general architecture of which an automatic machine learning (AutoML) system of computer vision models for biomedical image analysis is a part.

Fig. 2 - illustrates the general scheme for constructing AutoML computer vision models for biomedical image analysis using hybrid intelligence.

Fig. 3 - illustrates the general scheme of the learning agent device.

Fig. 4 illustrates the general scheme for updating AutoML computer vision models for biomedical image analysis.

Fig. 5 - illustrates the general scheme of the device of the clinic agent.

Fig. 6 illustrates a general diagram of a computing device for implementing the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention can be used, both with and without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to unnecessarily obscure the features of the present invention.

Furthermore, it will be clear from the foregoing that the invention is not limited to the present implementation. Numerous possible modifications, changes, variations and substitutions that retain the essence and form of the present invention will be obvious to those skilled in the art.

The present invention discloses an automatic machine learning (AutoML) system for computer vision models for analyzing biomedical images. The system is designed to automate the stages of development and training of computer vision models in the tasks of biomedical image analysis. Biomedical images are medical images obtained by various methods, for example, methods of radiation diagnostics (X-ray, magnetic resonance, radionuclide, ultrasound, etc.) - computed tomography (CT) images, magnetic resonance imaging (MRI) images, ultrasound images ( ultrasonography), positron emission tomography (PET) images, x-rays, mammography, angiography images, elastography images, etc.), through an endoscope (endoscopic images), using photographic methods (for example, medical photographs of skin conditions and other superficial conditions, such as the palate, birthmarks, moles, etc.), etc.

The task of building machine learning models consists of the following steps:

1. Data collection:

- access to data sources;

- technical integration;

- data validation;

- data download;

- data storage;

- data search.

2. Data preparation: normalization, cleaning, search for outliers.

3. Separation of the prepared data into test, validation and training sets.

4. Choice of model architecture.

5. Choice of hyperparameters.

6. Training of the selected model.

7. Model evaluation.

The Clinic Agent provides automation of data collection from clinics based on a system of rules and filters. The clinic agent is also responsible for technical integration and data download, validation and storage. At the core The work of a clinic agent is based on sets of rules, filters and lists of DICOM tags. Based on this data, it is possible to automate the processes of access, technical integration, validation, download, storage and retrieval of biomedical image data. Data collection is carried out from internal sources (for example, mini and postgre databases storing biomedical images) by automatic copying to the point of work - to the server where the training model will be launched.

The training agent is responsible for dividing the prepared data into test, validation and training sets, choosing the model architecture, choosing hyperparameters, training the selected model, and evaluating the model, which collects all actions into a chain of tasks that are performed on computing resources in sequential mode.

Automatic sampling relies on industry-leading AutoML approaches based on parsing the markup in the data to partition the samples in a stratified manner.

The choice of model architecture is based on the methods of Neural Architecture Serach (NAS) - a branch of machine learning that solves the problem of finding the best model in the context of a training set. Within the framework of this invention, a method is used based on the adaptation of NAS methods to the specifics of medical data - small sample sizes, the task of segmenting biomedical images as a key task of analysis, the use of existing solutions as a starting point for searching for computer vision models for analyzing biomedical images.

The found architecture is also trained in automatic mode, which eliminates the need for manual launches and selection of training parameters, which reduces human participation in this cycle.

Model evaluation is performed on the basis of a prepared protocol, which allows you to evaluate all the necessary model metrics in automatic mode.

On FIG. 1 shows an example of a general architecture, of which an automatic machine learning (AutoML) system for computer vision models for biomedical image analysis is a part.

Botkin Main Platform - the main platform, the central cloud of the Botkin.AI ecosystem. Carries out the relationship between all agents and subsystems, including managing data flows used for training and labeling models. Here are the following groups of services: 1. Data Management - system data management services: studies, medical images, support for the DICOM standard, management of datasets and groups of studies (DataSet, DataFolder).

2. User Management - user management services that perform the following functions:

- accounting and registration of users;

- maintenance of user rights;

- audit;

- OAuth 2 authorization services.

3. Agent Manager - infrastructure management services that perform the following functions:

- accounting and register of agents, agent descriptors;

- interaction with the API of cloud providers to raise virtual machines, deploy local Kubernetes clusters;

- interaction with Kubernetes cluster controllers to deploy and update agent services.

4. AutoML Management - model learning management services that perform the following functions:

- accounting for model learning metrics (Leader Board);

- storage of model artifacts (Model Registry).

5. Process Schedule Management - process planning service. This service performs the following functions:

- purpose of the process (including sub-process), choice of agent;

- control of resource utilization.

6. Platform Controller - a service for coordinating system processes.

Botkin Secondary Platform - secondary secondary platform Botkin.AL It differs from the main platform in that there are no AutoML management services, and process scheduling tasks are delegated to the main platform.

Inference Agent is an inference agent whose task is to process medical images using already trained models.

Learning Agent - a learning agent whose task is to find and train new machine learning models. The learning agent contains several subcomponents: a module for interacting with the system, a module for training computer vision models, a module for automatically deploying a model in industrial outline. This module is deployed on servers with sufficient computing resources. Multiple copies may be deployed.

Clinic Agent is a clinic agent that is deployed on the side of the clinic and provides a means of interaction with the clinic's information systems.

Satellite - agent management service.

ML Service - a service that performs the processing of studies by a computer vision model.

Report Service - a service that generates reports in the DICOM standard based on the results of processing a series of studies by a machine learning model.

Learning Service is a service that trains machine learning models, including machine learning algorithms.

Clinic Side - the internal network of the clinic.

Cloud Provider - provider of cloud servers.

HIS - clinic information system.

PACS - (English Picture Archiving and Communication System) - systems for the transmission and archiving of DICOM images.

Scanners - devices that perform research (CT machine, mat machine, etc.).

User - system user.

Botkin Resource Layer - resource management layer.

The 3rd Party DICOM Viewer is a doctor-supplied viewer, such as a web viewer or a standalone viewer, that contains all the necessary tools for biomedical image analysis, labeling biomedical images according to required protocols, and interacts with the system in terms of data addressing and tasks.

On FIG. Figure 2 shows a general scheme for building computer vision models for analyzing biomedical images based on two key technologies - AutoML technology, which automates the routine work of computer vision specialists, and hybrid intelligence - a group of methods that allow taking into account feedback from a person (for example, a radiologist) and using it to update AutoML models. The stages where AutoML and hybrid intelligence are used are highlighted in color. For example, the physician(s) mark up a pool of biomedical image data. In automatic mode, the data is uploaded to the server for training. The AutoML algorithm is launched, which consists of the following steps: data preparation, search for suitable model architectures, training of selected architectures, selection of the best model, testing on a delayed sample. If the quality of the model exceeds the specified threshold, the model is updated in the industrial loop, otherwise this step is skipped. The data is processed by the current version of the model and provided to the doctor for validation. If the validation result is unsatisfactory (FAIL), the data is returned to the markup and the process is repeated.

On FIG. Figure 3 shows the general layout of the learning agent device.

Learning Agents is a group of services managed by the Satellite service designed to train artificial intelligence models for biomedical image analysis tasks.

On FIG. 3 shows the following learning agent services:

1. Satellite - managing agent service.

2. Learning Service - a service that performs learning. The service consists of the following components:

- data loader - a block that loads the data necessary for training and testing models from the storage directly to the server where the agent is deployed;

- data preparation module (Data Preprocessor) - a block that performs the transformation of data received from the data loading block into a format accepted by the blocks for searching for models and their training;

- Model Search block - a block that implements a set of AutoML methods for searching and optimizing metaparameters. Starts and controls the model search process;

- Model Train block - a block that trains the model according to the found architecture parameters. If necessary, may initiate a second learning search process; (In case of incorrect completion of training or problems of an infrastructural nature (temporary communication problems, equipment reboot, etc.).

- Model Test block - a block that performs testing and evaluation of model metrics on a delayed sample. If necessary, it can initiate a repeated learning search process, for example, if the specified metric values are not reached on the test sample. On FIG. 4 shows a general scheme for updating AutoML computer vision models for biomedical image analysis.

On FIG. 5 shows a general diagram of the device of the clinic agent.

Clinic agents are a group of services managed by the Satellite service, deployed on the side of the clinic, designed to be integrated with the clinic's information systems, devices, radiologists' tools, etc. The clinic agent periodically, for example, once a day at midnight, selects all studies that have entered the clinic's PACS in the last 24 hours. Next, the clinic agent sends the collected biomedical image data to the main or auxiliary platform for processing, and returns the results of the biomedical image analysis to the responsible doctor.

Below is an example of automatic machine learning (AutoML) computer vision models for mammography image analysis.

The learning agent loads from the storage locations specified in the configuration file the mammography data as images and the generated annotations for the specified images. Annotation is created by physicians and usually consists of a class of study (normal or pathological, such as breast cancer) and a set of regions of interest associated with the mammographic image. The configuration file specifies the necessary parameters for the operation of the learning agent, for example, the search budget (how many hours of computing resources can be spent on searching), the type of problem being solved (classification, segmentation), service information (for example, addresses of alloying servers), the share of training and test examples in the sample, image parameters in the study (their number) and the number of channels (classes) into which the samples are divided, etc. Next, the learning agent processes the received data (for example, for raw data from the DICOM viewports embedded in the file, determines the projections of the image) and saves the data in the accepted format on the server (for example, in the form of binary files containing 4 images (images of each breast in two projections, and images of regions of interest.) Next, the learning agent launches methods for preparing data partitioning into training and test data. For example, a stratified partition by the presence of a norm and a pathology into two samples according to specified proportions. One patient can enter only one sample - training or a test one, even if it has more than one study.The learning agent then runs the learning methods that represent is a variation of a method called Neural Architecture Search (NAS) based on a gradient architecture search. For this, a basic architecture consisting of large blocks (for example, Linet) is used. Each block is searched by optimizing links between nodes. Thus, the search process is a search for such a set of weights that achieves a minimum of training error. The final architecture itself is obtained by binarizing (removing) links that have too low a weight. In the learning process, unified models are used that differ only in parameters. Alloying takes place in the ML Flow service. At each epoch of DS (Data Science) training, a specialist has access to logs to evaluate the performance of the model. The search for suitable models occurs until the specified values of the metrics are reached. The search is carried out by running the learning method with different metaparameters (training step size, regularization parameters, data augmentation parameters, etc.). The criterion for choosing models for the analysis of mammograms is, for example, maximizing the value of the AUC metric (area under the ROC-curve) to determine the norm / pathology for the study on the entire test sample. The traditional threshold value AUC = 0.85. When it is reached, the training is considered successfully completed, or until the search budget is exhausted. The budget is the number of machine hours allocated for the search. If the specified quality is not achieved within the allotted time, the process ends. In this case, the DS specialist receives a notification indicating the reason for the stop "search budget exhausted". If the quality of the model is achieved at any epoch, the learning agent deploys the model to the validation loop. To validate the model, a validation dataset is formed. The validation set is created from a separate data source that is not represented in the test or training dataset, otherwise the process is similar to the process of creating training and test sets. Next, a workflow is launched that sends mammographic data from the validation dataset to the trained model, which performs processing, and as a result, annotated mammographic images generated by the model are obtained. Mammography images processed by the trained model are assigned to a doctor who checks the quality of the model on the data provided. If the model fails validation, the decision is usually made to add training data and repeat the training process. On FIG. 6 shows a general diagram of a computing device (600) that provides the data processing necessary to implement the claimed solution.

In general, the device (600) contains components such as: one or more processors (601), at least one memory (602), storage media (603), input/output interfaces (604), I/O ( 605), networking tools (606).

The processor (601) of the device performs the basic computing operations necessary for the operation of the device (600) or the functionality of one or more of its components. The processor (601) executes the necessary machine-readable instructions contained in the main memory (602).

The memory (602) is typically in the form of RAM and contains the necessary software logic to provide the desired functionality.

The data storage means (603) can be in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The means (603) allows long-term storage of various types of information.

Interfaces (604) are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc.

The choice of interfaces (604) depends on the specific implementation of the device (N00), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

The data I/O means (605) in any embodiment of the system must be a keyboard. The keyboard hardware can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computer device. In this case, the connection can be either wired, in which the keyboard connection cable is connected to the PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB- ports. In addition to the keyboard, the following I/O devices can also be used: joystick, display (touchscreen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Means of networking (606) are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. With the help of tools (605) the organization of data exchange over a wired or wireless data transmission channel, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM, 3G, 4G, 5G, is provided.

The components of the device (600) are coupled via a common data bus (607).

The present application materials provide a preferred disclosure of the implementation of the claimed technical solution, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested legal protection and are obvious to specialists in the relevant field of technology.

It should be clear to a person skilled in the art that various variations of the proposed method and system do not change the essence of the invention, but only determine its specific embodiments and applications.

Claims

CLAIM 1. Computer-implemented automatic machine learning (AutoML) computer vision models for biomedical image analysis, comprising: a database, the database storing biomedical image data; moreover, the data on the basis of which the biomedical image data is obtained is collected automatically; server containing: - a loader, wherein the loader automatically loads the biomedical image data required for testing, training and validating computer vision models from the database; - a transformation unit, wherein the transformation unit automatically transforms the biomedical image data received from the download unit into a format accepted by the search, learning and evaluation units; - a search unit, and with the help of the search unit, computer vision models are automatically searched using training and test samples generated on the basis of biomedical image data received from the transformation unit, and the architecture parameters of the found models are automatically searched and optimized; - a training unit, wherein the training unit automatically trains the computer vision models found by the search unit using a training sample generated on the basis of biomedical image data received from the transformation unit and using architecture parameters received from the search unit; wherein the best of said trained models is automatically selected and the selected model is passed to the estimator; - an estimator, wherein the estimator automatically evaluates the best selected computer vision model trained by the training unit using a validation set formed on the basis of biomedical image data received from the transformation unit. 2. The system according to claim 1, characterized in that the clinic agent automatically collects data, on the basis of which biomedical image data is obtained. 3. The system according to claim 1, characterized in that the search block is used to search for the model until the specified metric values are reached or until the search budget is exhausted. 4. The system according to claim 1, characterized in that with the help of the training unit, the computer vision model found by the search unit is retrained using the training set, in which additional biomedical image data obtained from the transformation unit is added, if the specified the model has not been validated. 5. The system according to claim 1, characterized in that the training and evaluation units are configured to initiate a repeated process of searching and training computer vision models for analyzing biomedical images. 6. A computer-implemented method for automatic machine learning (AutoML) of computer vision models for analyzing biomedical images, in which: - automatically load biomedical image data required for testing, training and validation of computer vision models; moreover, the data on the basis of which the biomedical image data is obtained is collected automatically; - automatically perform the transformation of the downloaded biomedical image data into a format accepted for automatic search, training and evaluation; - automatically search for computer vision models using training and test sets formed on the basis of transformed biomedical image data, and automatically search and optimize the architecture parameters of the found models; - automatically perform training of the found computer vision models using a training set formed on the basis of the transformed data of biomedical images, and using the found and optimized parameters of the architectures of the found models; and automatically choose the best of these trained models; - automatically evaluate the best selected trained computer vision model using a validation set formed on the basis of the transformed biomedical image data. 7. The method according to claim 6, characterized in that the loading block automatically performs the loading of biomedical image data, with using the agent of the clinic, they automatically collect data, on the basis of which biomedical image data is obtained, using the transformation unit, they automatically perform the transformation of the loaded biomedical image data, using the search unit, they automatically search for computer vision models, using the training unit, the found computer vision models are automatically trained , the estimator automatically evaluates the best selected trained computer vision model. 8. The method according to claim 6, characterized in that the model is searched until the specified metric values are reached or until the search budget is exhausted. 9. The method according to claim 6, characterized in that the computer vision model found by the search unit is retrained using a training set, in which additional biomedical image data received from the transformation unit is added, if the specified model has not passed validation . 10. The method according to claim 6, characterized in that it additionally initiates a repeated process of searching and training computer vision models for analyzing biomedical images. 18