CN109697285B - Hierarchical BilSt Chinese electronic medical record disease coding and labeling method for enhancing semantic representation - Google Patents

Abstract

The invention discloses a hierarchical BiLSTM Chinese electronic medical record disease coding and labeling method for enhancing semantic representation. After preprocessing the input electronic medical record text, considering that in the composition of Chinese words, a single Chinese character contains specific semantics, the BiLSTM extracting the attention mechanism is used to extract Character-level feature vector representation to obtain the semantic and word formation features of a single Chinese character; concatenate the character-level word vector representation with the word-level vector representation obtained by word2vec training to obtain a character-level feature-enhanced word vector representation; represented by feature word vectors As input, BiLSTM is used again to learn the contextual features in the entire electronic medical record, and the attention mechanism is used to calculate the contribution of each feature word to obtain a text vector representation weighted by contextual features, which improves the prediction effect. The method of the invention is suitable for the task of disease label classification based on Chinese electronic medical record text, and effectively improves the classification effect.

Description

A Hierarchical BiLSTM Chinese Electronic Medical Record Disease Coding and Annotation Method with Enhanced Semantic Representation

technical field

The invention relates to the field of medical informatics, in particular to a hierarchical BiLSTM Chinese electronic medical record disease coding and labeling method for enhancing semantic representation.

Background technique

Electronic health records (Electronic Health Records, EHRs, electronic medical records for short) have become one of the important data resources for medical clinical research. It stores all kinds of information in the process of patients' medical treatment as digital data, which is convenient for us to use computers to analyze and process clinical data. For an electronic medical record, there needs to be a unified labeling specification describing the patient's disease status, which is conducive to rational classification of patient information to help clinical decision-making. The International Classification of Diseases (ICD), published by the World Health Organization and continuously updated, is an internationally accepted disease coding scheme, which is often used as a label in clinical records to identify symptoms, signs, diseases, abnormal findings or actions. Wait. At present, the 10th edition of the newly revised ICD code has been widely used in the hospital information system of our country.

Annotating ICD codes for electronic medical records is an important and basic work of using electronic medical records. The absence of diagnostic names and ICD codes in electronic medical records is not conducive to our analysis of clinical data. Usually, the labeling work of the ICD code is manually judged by the medical staff in the medical record room of each hospital according to the clinical diagnosis description given by the doctor. Manual coding not only requires coders to master certain medical knowledge, coding rules and medical terminology, but also takes time and effort. Therefore, the use of computer for automatic coding can provide effective assistance for coding and labeling work, and improve the labeling efficiency of ICD coding.

At present, most of the automatic labeling of disease codes is based on clinical text data, such as radiology reports, death certificates, discharge summaries, etc. However, most of the research work focuses on English corpus, and there is less disease coding prediction work on Chinese clinical texts, and the main method is the semantic comparison of strings based on diagnosis names. The comparison of semantic similarity has high requirements on the quality of the description of diagnosis names, and automatic coding and annotation cannot be performed in the case of missing diagnosis names. At present, there is no related research work using neural network models for disease coding and labeling tasks in Chinese electronic medical records.

The processing of Chinese electronic medical record text has two characteristics: one is that the electronic medical record text is long, and it is difficult to obtain the context information of the long text; , body parts, etc. are all described by a Chinese character, so the semantic representation containing character features can better express the semantics of words.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a hierarchical BiLSTM Chinese electronic medical record disease coding and labeling method with enhanced semantic representation, which can complete automatic labeling in an end-to-end manner and improve the prediction effect.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

A hierarchical BiLSTM Chinese electronic medical record disease coding and annotation method with enhanced semantic representation includes the following steps:

1) Using the Chinese word segmentation tool, introduce a user-defined medical clinical term dictionary for word segmentation, remove stop words, and filter out feature words according to word frequency;

2) The character-level and word-level vectorized representations of the feature words are respectively performed, and the character-level vector and the word-level vector are concatenated to construct the character-enhanced feature vector representation of the word;

3) Use the spliced feature words to obtain the contextual features of the entire text, and use the attention mechanism to calculate the contribution of each feature word to obtain the contextual feature weighted vector representation of the entire text.

其中S_fw表示特征词集合，

表示词w_i的频率，N_d表示电子病历样本总数。In step 1), the feature words are selected according to the following rules:

where S _fw represents the feature word set,

represents the frequency of word _wi , and N _d represents the total number of electronic medical record samples.

In step 2), the character-level feature vector representation of the feature word is trained using the bidirectional LSTM fused with the attention mechanism, and the word-level vector representation of the feature word is obtained by using the word vector representation method word2vec based on word distributed representation.

其中

表示前向LSTM在第t个单元或t时刻的隐层输出，

则为后向LSTM在第t个单元的隐层输出。The output mode of bidirectional long short-term memory network training is:

in

represents the hidden layer output of the forward LSTM at the t-th unit or time t,

Then it is the output of the hidden layer of the backward LSTM in the t-th unit.

The attention mechanism is calculated as:

u _ij =tanh(W _c h _ij +b _c );

为第i个词的上下文加权特征向量表示。h _ij is the hidden layer output of the j-th character of the i-th word after BiLSTM training, W _c is the weight matrix, b _c is the bias vector, u _c is the randomly initialized character-level context feature vector, α _ij is The weight of the j-th character for the i-th word calculated by the softmax function,

is the context-weighted feature vector representation for the ith word.

In step 3), the method for calculating the context feature weighted vector of the entire text includes: inputting the text represented by the spliced feature word vectors into the second-layer bidirectional long-term and short-term memory network, learning to obtain the context feature of the entire text, and adopting an attention mechanism, Calculate the weight of each feature word to obtain the text feature vector weighted by context information.

The attention mechanism is calculated as:

u _i =tanh(Wh _i +b _w );

v=∑ _i α _i h _i ;

h _i is the output of the hidden layer obtained after the BiLSTM training of the character-enhanced feature vector of the i-th word in the text sequence, W is the weight matrix, and b _w is the bias vector. When applying the attention mechanism, a corresponding one is introduced and randomly initialized. The word-level document context feature vector _uw is used to complete the weight calculation, α _i is the weight corresponding to each word, v is the context weighted feature vector representation of the entire text, and the vector is input into the fully connected layer, which is calculated by the sigmoid function. The probability of occurrence of each disease code.

Compared with the prior art, the present invention has the following beneficial effects: aiming at the characteristics of Chinese itself, the present invention integrates the semantic features of a single Chinese character into the feature vector representation of words, and combines the attention mechanism to truly contribute to the features in the input sequence. The words are weighted to improve the prediction effect of disease coding; this method is suitable for Chinese clinical text data, using neural network model to automatically extract text features, and complete automatic labeling in an end-to-end manner.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Figure 2. Hierarchical BiLSTM feature learning model fused with attention mechanism;

Fig. 3 Calculation of attention mechanism; (a) change h _ij into u _ij ; (b) calculate the weight of each u _ij using the context feature vector; (c) the weighted summation of h _ij obtains the feature vector representation of applying the attention mechanism ;

FIG. 4 is a graph showing the experimental results of the implementation of the present invention.

Detailed ways

1. Preprocessing of clinical text data

其中S_fw表示特征词集合，

表示词w_i的频率，N_d表示电子病历总数。Use the Chinese word segmentation tool "Jieba" and a user-defined medical thesaurus to segment the input text of the discharge summary, remove stop words, count the word frequencies of valid words, and select feature words based on the word frequency in descending order, and press Choose from the following rules:

where S _fw represents the feature word set,

represents the frequency of word _wi , and N _d represents the total number of electronic medical records.

Second, the word vector representation of feature words

1) Character-based word vector representation

First, initialize a vector representation for each character, and then input the BiLSTM fused with the attention mechanism, and train to obtain the character-level word vector representation of each feature word. The state value c _t and output value h _t of each neural unit in BiLSTM are calculated specifically The process is (t=1,2,...,n, t represents the t-th neural unit in the network or the neural unit at time t):

i _t =sigmoid(W _i [x _t ; h _t-1 ]+ _bi ) (1)

f _t =sigmoid(W _f [x _t ; h _t-1 ]+b _f ) (2)

g _t =tanh(W _g [x _t ; h _t-1 ]+b _g ) (3)

o _t =sigmoid(W _o [x _t ; h _t-1 ]+b _o ) (4)

c _t =f _t *c _t-1 +i _t *g _t (5)

h _t =o _t *tanh(c _t ) (6)

tanh函数的计算为

BiLSTM的输出方式为

Each neural unit contains an input gate i, an output gate o, a forget gate f, a storage unit g, a state-saving unit c and a hidden state h, all of which are vectors, W _i , W _f , W _g , W _o is the weight matrix, b _i , b _f , b _g , b _o are the bias vectors, ";" indicates the connection operation, "*" indicates the element-wise multiplication, and the calculation of the sigmoid function is

The tanh function is calculated as

The output method of BiLSTM is

2) Application of attention mechanism

The attention mechanism calculation method is:

u _ij =tanh(W _c h _ij +b _c ) (7)

即为第i个词的上下文加权特征向量表示。h _ij is the hidden layer output of the j-th character of the i-th word after BiLSTM training, W _c is the weight matrix, b _c is the bias vector, u _c is the randomly initialized character-level context feature vector, α _ij is is the weight of the j-th character to the i-th word calculated by the softmax function,

It is the context-weighted feature vector representation of the ith word.

3) Splicing the character-level word vector obtained by training with the word vector generated by word2vec to obtain a word-level feature vector enhanced by character-level contextual features.

3. Context Feature Extraction

The character-enhanced feature vector sequence is input into the BiLSTM of the second-layer fusion attention mechanism, and the text context information features are extracted. The calculation of the BiLSTM neural unit and the calculation of the context feature weighting are the same as those of the character-level word vector representation. The specific calculation formula is as follows :

u _i =tanh(Wh _i +b _w ) (10)

v=∑ _i α _i h _i (12)

h _i is the output of the hidden layer obtained after the BiLSTM training of the character-enhanced feature vector of the i-th word in the text sequence, W is the weight matrix, and b _w is the bias vector. When applying the attention mechanism, a corresponding one is introduced and randomly initialized. The word-level document context feature vector _uw is used to complete the weight calculation, α _i is the weight corresponding to each word, v is the context weighted feature vector representation of the entire text, and the vector is input into the fully connected layer, which is calculated by the sigmoid function. The probability of occurrence of each disease code.

4. Experimental verification

1) Experimental process

In order to verify the effectiveness of this method, we conducted experiments on real Chinese electronic medical record clinical data. The dataset contains 7732 discharge records, involving a total of 1177 ICD-10 disease code labels. The ICD-10 code is a dotted six-digit code consisting of letters and numbers, starting with a letter, and the first three codes are the first-level codes. Specify the disease category. The average length of discharge summaries was 610 words, with an average of 3.6 disease codes per discharge summary.

The experiments were done on a server containing 256GB of memory and an NVIDIA GeForce Titan XPascal CUDA GPU processor. We split the dataset into training and test sets in a 9:1 ratio, and performed validation by randomly shuffling the data ten times. The evaluation indicators selected the precision (P), recall rate (R) of the micro-average and the F1 value of the combination of the two, as well as the Hamming loss value to evaluate the false positive situation from the perspective of the sample. The higher the F1 value and the lower the Hamming loss value, the better the model performance.

2) Experimental results

Because the related research work has pointed out that the deep learning method is superior to the traditional machine learning method, we mainly conducted comparative experiments with other common neural network models. The results are shown in Table 1. MA-BiLSTM represents our model, D2V+CNN As a method in related research work, this method has achieved the best results on the public English dataset MIMIC III. The experimental results show that MA-BiLSTM is superior to other neural network models in various evaluation indicators, indicating that BiLSTM combined with attention mechanism can effectively capture the contextual information features of long texts and improve the prediction effect.

Table 1 Comparative experimental results

Model Micro_P (CI: 95%) Micro_R (CI: 95%) Micro_F1 (CI: 95%) hLoss (CI: 95%) CBOW 0.614(±6.43e-03) 0.522(±5.30e-03) 0.564(±4.52e-03) 0.00248(±3.14e-05) CNN 0.647(±6.67e-03) 0.509(±6.51e-03) 0.569(±4.71e-03) 0.00237(±3.52e-05) D2V+CNN 0.661(±9.57e-03) 0.514(±8.74e-03) 0.579(±7.14e-03) 0.00231(±3.70e-05) MA-BiLSTM 0.704(±1.13e-02) 0.586(±5.84e-03) 0.639(±4.45e-03) 0.00204(±3.47e-05)

In order to analyze the role of each module of the model, we designed an ablation experiment for analysis, and the results are shown in Table 2. From the experimental results, only word vectors or character vectors represent the features of words in the text, and the prediction results have declined. Therefore, the word vector representation enhanced by character vectors does bring better text feature representation. The attention mechanism plays an important role in the model. If the attention mechanism is removed, the performance of the model decreases significantly.

Predicted on both ICD-10 full coding and primary coding, 7732 samples, corresponding to 488 primary coding. The experimental results are shown in Figure 4. The accuracy of the prediction results on the first-level coding has reached 80.5%, which can better assist the medical staff in the medical record room in the work of disease coding and labeling.

Table 2 Model ablation experimental results

Claims (4)

1. A hierarchical BilSt Chinese electronic medical record disease coding and labeling method for enhancing semantic representation is characterized by comprising the following steps:

1) utilizing a Chinese word segmentation tool, introducing a user-defined medical clinical word dictionary to segment words of the discharge summary text, removing stop words, and screening out characteristic words according to word frequency;

2) respectively carrying out character level and word level vectorization representation on the feature words, splicing the character level vectors and the word level vectors, and constructing character enhancement feature vector representation of the words; using character-level feature vector representation of a BilSTM training feature word fused with an attention mechanism, and using a word vector representation method word2vec based on word distributed representation to obtain a word-level vector representation form of the feature word;

3) obtaining a word vector representation sequence of the whole text by using the spliced feature words, calculating the contribution degree of each feature word by using an attention mechanism, obtaining context feature weighted vector representation of the whole text, namely inputting the text represented by the spliced feature word vector into a second-layer bidirectional long-short term memory network, learning to obtain context features of the whole text, and calculating the weight of each feature word by using the attention mechanism, so as to obtain a text feature vector weighted by context information;

the calculation mode of the attention mechanism is as follows:

v＝∑_iα_ih_i；

h_iis the output of a hidden layer obtained after the character reinforcing characteristic vector of the ith word of the text sequence is trained by BilSTM, W is a weight matrix, b_wFor a bias vector, when an attention mechanism is applied, a document context feature vector u at a word level is correspondingly introduced and randomly initialized_wTo complete the calculation of the weight value, α_iAnd v is represented by the context weighted feature vector of the whole text for the weight corresponding to each word, the vector is input into a full connection layer, and the occurrence probability of each disease code is calculated by a sigmoid function.

2. The method for disease-coding and labeling of BiLSTM Chinese electronic medical record with enhanced semantic representation according to claim 1, wherein in step 1), the feature words are selected according to the following rules:

wherein S_fwA set of characteristic words is represented,

the expression w_iFrequency of (N), N_dAnd the total number of samples of the electronic medical record is shown.

3. The hierarchical BilSTM Chinese electronic medical record disease coding labeling method based on enhanced semantic representation according to claim 1, wherein the output mode of the BilSTM is as follows:

wherein

Representing the hidden layer output of the forward LSTM at the t-th element or time t,

then it is output at the hidden layer of the t-th cell for backward LSTM.

4. The method for disease coding and labeling of BiLSTM Chinese electronic medical record with enhanced semantic representation according to claim 1, wherein in step 2), the calculation mode of the attention mechanism is as follows:

u_ij＝tanh(W_ch_ij+b_c)；

h_ijis the output of the j character of the i word in the hidden layer after the BilSTM training, W_cAs a weight matrix, b_cAs an offset vector, u_cFor the random initialization of the character-level context feature vector, α_ijFor the weight of the jth character to the ith word calculated by the softmax function,

the feature vector representation is weighted for the context of the ith word.

Images