CN117542469A - Diagnostic report generation method, system and medium for multi-modal medical data - Google Patents

Abstract

The disclosure provides a diagnostic report generation method, a system and a medium for multi-modal medical data, wherein the diagnostic report generation method for the multi-modal medical data comprises the following steps: acquiring a multi-modal training data set, wherein the multi-modal training data set comprises a medical image and a text report; splitting the text report according to preset keywords, and determining a negative description text and a positive description text; inputting the medical image, the negative description text and the positive description text into a pre-trained diagnostic report generation model, and outputting a loss function; optimizing the pre-trained diagnostic report generation model according to the loss function, and determining a multi-mode medical diagnostic report generation model; inputting the medical image to be detected into a multi-mode medical diagnosis report generating model to generate a diagnosis report corresponding to the medical image to be detected. The method solves the problem of unbalanced training samples in the model training stage, and improves the detection capability of the model on abnormal description in the medical image.

Description

Diagnostic report generation method, system and medium for multi-modal medical data

Technical field

The present disclosure relates to the technical fields of computer vision and natural language processing, and in particular, to a diagnostic report generation method, system and medium for multi-modal medical data.

Background technique

With the improvement of medical standards and the development of medical imaging technology, medical imaging data has become an important part of patients’ electronic files. Writing a qualified medical imaging report requires doctors to have professional medical imaging knowledge, clinical medicine knowledge and clinical experience, and Have certain ability to evaluate changes in lesions over time, and have the ability to make comprehensive judgments on patient medical history and other examination results.

However, nowadays, medical imaging data is growing exponentially, and the excess data puts tremendous pressure on clinicians. The medical field has accumulated a large amount of "Medical Image-Text Report" data. These graphic and text data have strong intrinsic correlation. If based on these massive hospital data, a computer-aided diagnosis system can be used to optimize the doctor's workflow, and computer-aided diagnosis can be used to optimize the doctor's workflow. Medical images are analyzed and processed, information is extracted from the examination report, and the image report is automatically generated. The doctor only reviews and modifies the final report, which can greatly relieve the doctor's work pressure and greatly reduce misdiagnosis and missed diagnosis. function, this application has important practical application value.

When doctors write imaging reports, they will also refer to features such as medical images and examination reports. Text reports include laboratory test reports, clinical records, etc. This information is difficult to comprehensively process through traditional technical methods. Deep learning methods need to be used to extract and align features of image data and text data, and use the aligned features to generate images. Report text. At the same time, there are a large number of non-abnormal sentence descriptions in the image report text description, which are defined as negative descriptions, and descriptions of abnormal features such as lesions in the images, which are defined as positive descriptions. The large number of negative descriptions lead to inconsistency between positive and negative samples. The balancing problem affects the detection rate of anomaly descriptions in the final report generation task.

Contents of the invention

In view of the deficiencies in the prior art, the purpose of this disclosure is to provide a diagnostic report generation method, system and medium for multi-modal medical data.

In order to achieve the above object, according to the first aspect of the present disclosure, a diagnostic report generation method for multi-modal medical data is provided, including:

Obtaining a multi-modal training data set, the multi-modal training data set includes medical images and text reports;

Split the text report according to preset keywords to determine negative description text and positive description text;

The medical image, the negative description text, and the positive description text are input into a pre-trained diagnostic report generation model, a loss function is output, and the pre-trained diagnostic report generation model is model trained, and the loss function includes Negative loss function and positive loss function;

Optimize the pre-trained diagnostic report generation model according to the loss function to determine a multi-modal medical diagnosis report generation model;

The medical image to be tested is input into the multi-modal medical diagnosis report generation model, and a diagnosis report corresponding to the medical image to be tested is generated.

Optionally, the pre-trained diagnostic report generation model includes a text editor, an image editor, and a multi-modal text generator.

Optionally, the medical image, the negative description text, and the positive description text are input into a pre-trained diagnostic report generation model, a loss function is output, and model training is performed on the pre-trained diagnostic report generation model. ,include:

Input the medical image into the image editor for image feature extraction processing, and output the image features of the medical image;

Input the negative description text and the positive description text into the text editor, and output the text features of the text report, where the text features of the text report include negative description text features and positive description text features;

The image features and the text features are input into the multi-modal text generator, and the negative loss function and the positive loss function are output.

Optionally, inputting the image features and the text features into the multi-modal text generator, and outputting the negative loss function and the positive loss function include:

Perform feature alignment processing on the image features and text features in the same feature space, and determine the image features and text features that have undergone the alignment processing;

Generate a diagnosis report corresponding to the medical image according to the image features and the text features that have undergone the alignment process;

The negative loss function and the positive loss function are output according to the diagnosis report corresponding to the medical image and the text report.

Optionally, optimizing the pre-trained diagnostic report generation model according to the loss function to determine a multi-modal medical diagnosis report generation model includes:

Set the proportion of the negative description text and the positive description text in the text report according to the negative loss function and the positive loss function, optimize the parameters of the pre-trained diagnostic report generation model, and determine the multi-modality Medical diagnosis report generation model.

Optionally, the pre-trained diagnostic report generation model uses text reports containing all negative description texts for pre-training processing.

Optionally, inputting the medical image to be tested into the multi-modal medical diagnosis report generation model and generating a diagnostic report corresponding to the medical image to be tested includes:

Input the medical image to be tested into the image editor, and output the image features of the medical image to be tested;

The image features of the medical image to be tested are input into the multi-modal text generator to generate a diagnostic report corresponding to the medical image to be tested.

According to a second aspect of the present disclosure, a diagnostic report generation system for multi-modal medical data is provided, including:

According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the multi-modal medical data-oriented diagnosis provided by the first aspect of the present disclosure is implemented. Report generation method.

According to a fourth aspect of the present disclosure, an electronic device is provided, including:

A memory on which a computer program is stored;

A processor, configured to execute the computer program in the memory to implement the steps of the diagnostic report generation method for multi-modal medical data provided in the first aspect of the present disclosure.

Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

Through the above technical solution, during the model training process of the pre-trained diagnostic report generation model, the negative loss function and the positive loss function are output, and the negative description text and the positive description text in the text report can be set according to the negative loss function and the positive loss function. Ratio, guide the positive description text to train the pre-trained diagnostic report generation model to solve the problem of imbalanced training samples, adjust the loss weight of negative description text and positive description text, and improve the detection accuracy of abnormal descriptions with medical impact, effectively It prevents the missed detection of abnormal lesions, and uses the trained multi-modal medical diagnosis report generation model to automatically generate diagnostic reports to relieve doctors' work pressure and improve work efficiency.

Description of drawings

Other features, objects and advantages of the present disclosure will become more apparent upon reading the detailed description of the non-limiting embodiments with reference to the following drawings:

FIG. 1 is a flow chart of a method for generating a diagnostic report for multi-modal medical data according to an exemplary embodiment.

Figure 2 is a schematic structural diagram of a pre-trained diagnostic report generation model according to an exemplary embodiment.

FIG. 3 is a flowchart of a method for training a pre-trained diagnostic report generation model according to an exemplary embodiment.

FIG. 4 is a block diagram of a diagnostic report generation system for multi-modal medical data according to an exemplary embodiment.

FIG. 5 is a block diagram of an electronic device according to an example embodiment.

Detailed ways

The present disclosure will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art further understand the present disclosure, but do not limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure. These all belong to the protection scope of this disclosure.

FIG. 1 is a flow chart of a method for generating a diagnostic report for multi-modal medical data according to an exemplary embodiment. As shown in Figure 1, a diagnostic report generation method for multi-modal medical data includes S11 to S15.

S11 obtains the multi-modal training data set.

Among them, the multi-modal training data set includes medical images and text reports, and the multi-modal training data set can be in the form of a combined pair consisting of multiple medical images and their corresponding text reports. The multimodal training dataset is used for model training of the pretrained diagnostic report generation model.

S12 splits the text report according to the preset keywords and determines the negative description text and the positive description text.

Among them, the negative description text represents the text description in the text report involving the abnormal characteristics of lesions that do not appear in the medical images, and the positive description text represents the text description in the text report that involves the abnormal characteristics of lesions in the medical images.

S13 inputs the medical image, negative description text, and positive description text into the pre-trained diagnostic report generation model, outputs the loss function, and performs model training on the pre-trained diagnostic report generation model.

Among them, the loss function includes a negative loss function and a positive loss function. The negative loss function represents the loss function corresponding to the negative description text, and the positive loss function represents the loss function corresponding to the positive description text.

In some possible embodiments, first, a text report with a full amount of negative description text and its corresponding medical image can be used to pre-train a preset diagnostic report generation model to obtain a pre-trained diagnostic report generation model; secondly, The pre-trained diagnostic report generation model is gradually trained using text reports containing negative description text and positive description text to improve the diagnostic report generation model's ability to recognize positive description text.

Figure 2 is a schematic structural diagram of a pre-trained diagnostic report generation model according to an exemplary embodiment.

As shown in Figure 2, the diagnostic report generation model includes a text editor, an image editor, and a multi-modal text generator. The text editor is used to extract text features from text reports, the image editor is used to extract image features from medical images, and the multimodal text generator is used to align text features and image features in the same feature space, and generate text based on the aligned text. Features and image features to generate diagnostic reports.

As an example, pre-training a preset diagnostic report generation model includes:

Input the text report of the full amount of negative description text and its corresponding medical image into the preset diagnosis report generation model, and output the negative loss function.

Specifically, the entire negative description text is input into the text editor, and the text features of the full negative description text are output; the medical image image is input into the image editor, and the image features of the medical image are output; and the text features of the full negative description text and the medical image are output. The image features of the image are input into the multi-modal text generator and the negative loss function is output.

As another example, a pre-trained diagnostic report generation model is subjected to a model training process using a text report containing negative description text and positive description text. Guide the positive description text to train the pre-trained diagnostic report generation model to prevent the imbalance of training samples used for model training from causing the multi-modal medical diagnosis report generation model that has completed training to be difficult to detect abnormal descriptions.

S14 optimizes the pre-trained diagnostic report generation model according to the loss function to determine the multi-modal medical diagnosis report generation model.

In a possible embodiment, the proportion of negative description text and positive description text in the text report is set according to the negative loss function and the positive loss function, the pre-trained diagnostic report generation model is optimized, and the multi-modal medical diagnosis report generation model is determined.

In the model training stage, you can adjust the proportion of negative description text and positive description text independently. By adjusting the proportion of positive description text in the text report, you can then adjust the weight of the negative loss function and the positive loss function, and improve the trained results to a certain extent. The multi-modal medical diagnosis report generation model has the ability to identify the abnormal characteristics of lesions appearing in medical images, and accurately generates diagnostic reports based on medical images, effectively preventing the missed detection of abnormal lesions.

S15 inputs the medical image to be tested into the multi-modal medical diagnosis report generation model, and generates a diagnostic report corresponding to the medical image to be tested.

As an example, in the testing phase, the medical image to be tested is input into the multi-modal medical diagnosis report generation model that has been trained. First, the medical image to be tested is input into the image editor, and the image of the medical image to be tested is output. features; secondly, input the image features of the medical image to be tested into the multi-modal text generator to generate a diagnostic report corresponding to the medical image to be tested. Finally, the multi-modal medical diagnosis report generation model outputs the text corresponding to the medical impact to be tested. Diagnose report.

Through the above technical solution, during the model training process of the pre-trained diagnostic report generation model, the negative loss function and the positive loss function are output, and the negative description text and the positive description text in the text report can be set according to the negative loss function and the positive loss function. Ratio, guide the positive description text to train the pre-trained diagnostic report generation model to solve the problem of imbalanced training samples, adjust the loss weight of negative description text and positive description text, and improve the detection accuracy of abnormal descriptions with medical impact, effectively It prevents the missed detection of abnormal lesions, and uses the trained multi-modal medical diagnosis report generation model to automatically generate diagnostic reports to relieve doctors' work pressure and improve work efficiency.

FIG. 3 is a flowchart of a method for training a pre-trained diagnostic report generation model according to an exemplary embodiment.

As shown in Figure 3, in some possible embodiments, medical images, negative description text, and positive description text are input into a pre-trained diagnostic report generation model, a loss function is output, and model training is performed on the pre-trained diagnostic report generation model. , including S21 to S23.

S21. Input the medical image into the image editor for image feature extraction processing, and output the image features of the medical image.

S22: Input the negative description text and the positive description text into a text editor, and output the text characteristics of the text report.

Among them, the text features of the text report include negative description text features and positive description text features.

S23, input the image features and text features into the multi-modal text generator, and output the negative loss function and the positive loss function.

In a possible embodiment, in the multi-modal text generator, the image features and text features are subjected to feature alignment processing in the same feature space, and the aligned image features and text features are determined; according to the aligned features Image features and text features are used to generate diagnostic reports corresponding to medical images; negative loss functions and positive loss functions are output based on reports and text reports corresponding to medical images.

Through the above technical solution, by adjusting the loss weight of the negative description text and the positive description text, the positive description text is guided to perform model training and optimization processing on the pre-trained diagnostic report generation model, and the text editor, image editor, and image editor of the diagnostic report generation model are optimized. and the parameters of the multi-modal text generator, complete the model training of the pre-trained diagnostic report generation model, and obtain the multi-modal medical diagnosis report generation model.

FIG. 4 is a block diagram of a diagnostic report generation system for multi-modal medical data according to an exemplary embodiment.

Based on the same concept, the present disclosure also provides a diagnostic report generation system for multi-modal medical data. Referring to Figure 4 , the diagnostic report generation system 100 for multi-modal medical data includes: an acquisition module 110 and a text processing module 120 , model training module 130, model optimization module 140, and diagnostic report generation module 150.

The acquisition module 110 is used to acquire a multi-modal training data set, which includes medical images and text reports;

The text processing module 120 is used to split the text report according to preset keywords and determine negative descriptive text and positive descriptive text;

The model training module 130 is used to input the medical image, the negative description text, and the positive description text into a pre-trained diagnostic report generation model, output a loss function, and model the pre-trained diagnostic report generation model. Training, the loss function includes a negative loss function and a positive loss function;

The model optimization module 140 is configured to optimize the pre-trained diagnostic report generation model according to the loss function and determine a multi-modal medical diagnosis report generation model;

The diagnostic report generation module 150 is configured to input the medical image to be tested into the multi-modal medical diagnosis report generation model, and generate a diagnostic report corresponding to the medical image to be tested.

Through the above technical solution, during the model training process of the pre-trained diagnostic report generation model, the negative loss function and the positive loss function are output, and the negative description text and the positive description text in the text report can be set according to the negative loss function and the positive loss function. Ratio, guide the positive description text to train the pre-trained diagnostic report generation model to solve the problem of imbalanced training samples, adjust the loss weight of negative description text and positive description text, and improve the detection accuracy of abnormal descriptions with medical impact, effectively It prevents the missed detection of abnormal lesions, and uses the trained multi-modal medical diagnosis report generation model to automatically generate diagnostic reports to relieve doctors' work pressure and improve work efficiency.

Optionally, the pre-trained diagnostic report generation model includes a text editor, an image editor, and a multi-modal text generator.

Optionally, the model training module 130 includes:

An image feature extraction submodule, used to input the medical image into the image editor for image feature extraction processing, and output the image features of the medical image;

Text feature extraction submodule, used to input the negative description text and the positive description text into the text editor, and output the text features of the text report. The text features of the text report include negative description text features and positive description text. Describe text features;

The multi-modal processing submodule is used to input the image features and the text features into the multi-modal text generator, and output the negative loss function and the positive loss function.

Optionally, the multimodal processing submodule includes:

Alignment processing submodule, used to perform feature alignment processing on the image features and the text features in the same feature space, and determine the image features and the text features that have undergone the alignment processing;

A diagnostic report generation submodule, configured to generate a diagnostic report corresponding to the medical image according to the image features and the text features that have undergone the alignment process;

A loss submodule, configured to output the negative loss function and the positive loss function according to the diagnosis report corresponding to the medical image and the text report.

Optionally, the model optimization module 140 includes:

An optimization submodule configured to set the ratio of the negative description text and the positive description text in the text report according to the negative loss function and the positive loss function, and optimize the pre-trained diagnostic report generation model parameters, The multimodal medical diagnosis report generation model is determined.

Optionally, the pre-trained diagnostic report generation model uses text reports containing all negative description texts for pre-training processing.

Optionally, the diagnostic report generation module 150 includes:

The image feature extraction submodule to be tested is used to input the medical image to be tested into the image editor and output the image features of the medical image to be tested;

The diagnostic report generation submodule to be tested is used to input the image features of the medical image to be tested into the multi-modal text generator and generate a diagnostic report corresponding to the medical image to be tested.

Regarding the embodiments of the above system, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.

FIG. 5 is a block diagram of an electronic device according to an example embodiment.

As shown in FIG. 5 , in some possible embodiments, the present disclosure can also provide an electronic device, such as a medical diagnosis report terminal. The electronic device 500 can include: a processor 501 and a memory 502 . The electronic device 500 may also include one or more of a multimedia component 503 , an input/output interface 504 , and a communication component 505 .

The processor 501 is used to control the overall operation of the electronic device 500 to complete all or part of the steps in the first aspect of the diagnostic report generation method for multi-modal medical data. The memory 502 is used to store various types of data to support operations on the electronic device 500. These data may include, for example, instructions for any application program or method operating on the electronic device 500, as well as application-related data. For example, contact data, messages sent and received, pictures, audios, videos, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (Static Random Access Memory, SRAM for short), electrically erasable programmable read-only memory ( Electrically Erasable Programmable Read-Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM for short), Programmable Read-Only Memory (PROM for short), Read-Only Memory ( Read-Only Memory (ROM for short), magnetic memory, flash memory, magnetic disk or optical disk. Multimedia components 503 may include screen and audio components. The screen may be a touch screen, for example, and the audio component is used to output and/or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in memory 502 or transmitted through communication component 505 . The audio component also includes at least one speaker for outputting audio signals. The input/output interface 504 provides an interface between the processor 501 and other interface modules. The other interface modules may be keyboards, mice, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or one or more of them The combination is not limited here. Therefore, the corresponding communication component 305 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

In another exemplary embodiment, a non-transitory computer-readable storage medium including program instructions is also provided. When the program instructions are executed by a processor, the multi-modal medical data-oriented diagnosis of the first aspect is implemented. Steps in the report generation method. For example, the computer-readable storage medium may be the above-mentioned memory including program instructions, and the program instructions may be executed by a processor of the electronic device to complete the diagnostic report generation method for multi-modal medical data.

In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a function for performing the above when executed by the programmable device. The code part of the diagnostic report generation method for multi-modal medical data.

Specific embodiments of the present disclosure have been described above. It should be understood that the present disclosure is not limited to the specific embodiments described above, and those skilled in the art can make various variations or modifications within the scope of the claims, which does not affect the essential content of the present disclosure. The above preferred features can be used in any combination as long as they do not conflict with each other.

Claims (10)

1. A diagnostic report generation method for multi-modal medical data, characterized by: Obtaining a multi-modal training data set, the multi-modal training data set includes medical images and text reports; Split the text report according to preset keywords to determine negative description text and positive description text; The medical image, the negative description text, and the positive description text are input into a pre-trained diagnostic report generation model, a loss function is output, and the pre-trained diagnostic report generation model is model trained, and the loss function includes Negative loss function and positive loss function; Optimize the pre-trained diagnostic report generation model according to the loss function to determine a multi-modal medical diagnosis report generation model; The medical image to be tested is input into the multi-modal medical diagnosis report generation model, and a diagnosis report corresponding to the medical image to be tested is generated. 2. The method according to claim 1, characterized in that the pre-trained diagnostic report generation model includes a text editor, an image editor, and a multi-modal text generator. 3. The method according to claim 2, characterized in that the medical image, the negative description text, and the positive description text are input into a pre-trained diagnostic report generation model, and a loss function is output to calculate the The above pre-trained diagnostic report generation model is used for model training, including: Input the medical image into the image editor for image feature extraction processing, and output the image features of the medical image; Input the negative description text and the positive description text into the text editor, and output the text features of the text report, where the text features of the text report include negative description text features and positive description text features; The image features and the text features are input into the multi-modal text generator, and the negative loss function and the positive loss function are output. 4. The method according to claim 3, characterized in that the image features and the text features are input into the multi-modal text generator, and the negative loss function and the positive loss function are output ,include: Perform feature alignment processing on the image features and text features in the same feature space, and determine the image features and text features that have undergone the alignment processing; Generate a diagnosis report corresponding to the medical image according to the image features and the text features that have undergone the alignment process; The negative loss function and the positive loss function are output according to the diagnosis report corresponding to the medical image and the text report. 5. The method according to claim 1, characterized in that optimizing the pre-trained diagnostic report generation model according to the loss function and determining a multi-modal medical diagnosis report generation model includes: Set the proportion of the negative description text and the positive description text in the text report according to the negative loss function and the positive loss function, optimize the parameters of the pre-trained diagnostic report generation model, and determine the multi-modality Medical diagnosis report generation model. 6. The method according to claim 5, characterized in that the pre-trained diagnostic report generation model uses a text report with a full amount of negative description text for pre-training processing. 7. The method according to claim 2, characterized in that said inputting the medical image to be tested into the multi-modal medical diagnosis report generation model and generating a diagnosis report corresponding to the medical image to be tested includes: : Input the medical image to be tested into the image editor, and output the image features of the medical image to be tested; The image features of the medical image to be tested are input into the multi-modal text generator to generate a diagnosis report corresponding to the medical image to be tested. 8. A diagnostic report generation system for multi-modal medical data, characterized by including: An acquisition module, used to acquire a multi-modal training data set, where the multi-modal training data set includes medical images and text reports; A text processing module, configured to split the text report according to preset keywords and determine negative description text and positive description text; A model training module, configured to input the medical image, the negative description text, and the positive description text into a pre-trained diagnostic report generation model, output a loss function, and perform model training on the pre-trained diagnostic report generation model. , the loss function includes a negative loss function and a positive loss function; A model optimization module, configured to optimize the pre-trained diagnostic report generation model according to the loss function and determine a multi-modal medical diagnosis report generation model; A diagnostic report generation module is used to input the medical image to be tested into the multi-modal medical diagnosis report generation model and generate a diagnostic report corresponding to the medical image to be tested. 9. A non-transitory computer-readable storage medium with a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method according to any one of claims 1-7 are implemented. 10. An electronic device, characterized in that it includes: A memory on which a computer program is stored; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1-7.