`，被忽略的情况与attn_mask一致。
-
-```python
-# 将key_padding_mask值改为布尔值
-if key_padding_mask is not None and key_padding_mask.dtype == torch.uint8:
- warnings.warn("Byte tensor for key_padding_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.")
- key_padding_mask = key_padding_mask.to(torch.bool)
-```
-
-先介绍两个小函数，`logical_or`，输入两个tensor，并对这两个tensor里的值做`逻辑或`运算，只有当两个值均为0的时候才为`False`，其他时候均为`True`，另一个是`masked_fill`，输入是一个mask，和用以填充的值。mask由1，0组成，0的位置值维持不变，1的位置用新值填充。
-```python
-a = torch.tensor([0,1,10,0],dtype=torch.int8)
-b = torch.tensor([4,0,1,0],dtype=torch.int8)
-print(torch.logical_or(a,b))
-# tensor([ True, True, True, False])
-```
-
-```python
-r = torch.tensor([[0,0,0,0],[0,0,0,0]])
-mask = torch.tensor([[1,1,1,1],[0,0,0,0]])
-print(r.masked_fill(mask,1))
-# tensor([[1, 1, 1, 1],
-# [0, 0, 0, 0]])
-```
-其实attn_mask和key_padding_mask有些时候对象是一致的，所以有时候可以合起来看。`-inf`做softmax之后值为0，即被忽略。
-```python
-if key_padding_mask is not None:
- assert key_padding_mask.shape == (bsz, src_len), \
- f"expecting key_padding_mask shape of {(bsz, src_len)}, but got {key_padding_mask.shape}"
- key_padding_mask = key_padding_mask.view(bsz, 1, 1, src_len). \
- expand(-1, num_heads, -1, -1).reshape(bsz * num_heads, 1, src_len)
- # 若attn_mask为空，直接用key_padding_mask
- if attn_mask is None:
- attn_mask = key_padding_mask
- elif attn_mask.dtype == torch.bool:
- attn_mask = attn_mask.logical_or(key_padding_mask)
- else:
- attn_mask = attn_mask.masked_fill(key_padding_mask, float("-inf"))
-
-# 若attn_mask值是布尔值，则将mask转换为float
-if attn_mask is not None and attn_mask.dtype == torch.bool:
- new_attn_mask = torch.zeros_like(attn_mask, dtype=torch.float)
- new_attn_mask.masked_fill_(attn_mask, float("-inf"))
- attn_mask = new_attn_mask
-
-```
-
-***
-##### 点积注意力
-
-
-```python
-from typing import Optional, Tuple, Any
-def _scaled_dot_product_attention(
- q: Tensor,
- k: Tensor,
- v: Tensor,
- attn_mask: Optional[Tensor] = None,
- dropout_p: float = 0.0,
-) -> Tuple[Tensor, Tensor]:
- r'''
- 在query, key, value上计算点积注意力，若有注意力遮盖则使用，并且应用一个概率为dropout_p的dropout
-
- 参数：
- - q: shape:`(B, Nt, E)` B代表batch size， Nt是目标语言序列长度，E是嵌入后的特征维度
- - key: shape:`(B, Ns, E)` Ns是源语言序列长度
- - value: shape:`(B, Ns, E)`与key形状一样
- - attn_mask: 要么是3D的tensor，形状为:`(B, Nt, Ns)`或者2D的tensor，形状如:`(Nt, Ns)`
-
- - Output: attention values: shape:`(B, Nt, E)`，与q的形状一致;attention weights: shape:`(B, Nt, Ns)`
-
- 例子：
- >>> q = torch.randn((2,3,6))
- >>> k = torch.randn((2,4,6))
- >>> v = torch.randn((2,4,6))
- >>> out = scaled_dot_product_attention(q, k, v)
- >>> out[0].shape, out[1].shape
- >>> torch.Size([2, 3, 6]) torch.Size([2, 3, 4])
- '''
- B, Nt, E = q.shape
- q = q / math.sqrt(E)
- # (B, Nt, E) x (B, E, Ns) -> (B, Nt, Ns)
- attn = torch.bmm(q, k.transpose(-2,-1))
- if attn_mask is not None:
- attn += attn_mask
- # attn意味着目标序列的每个词对源语言序列做注意力
- attn = F.softmax(attn, dim=-1)
- if dropout_p:
- attn = F.dropout(attn, p=dropout_p)
- # (B, Nt, Ns) x (B, Ns, E) -> (B, Nt, E)
- output = torch.bmm(attn, v)
- return output, attn
-
-```
-### **完整的多头注意力机制-MultiheadAttention
**
-[点击快速回到：拆开看多头注意力机制](#拆开看多头注意力机制)
-
-```python
-class MultiheadAttention(nn.Module):
- r'''
- 参数：
- embed_dim: 词嵌入的维度
- num_heads: 平行头的数量
- batch_first: 若`True`，则为(batch, seq, feture)，若为`False`，则为(seq, batch, feature)
-
- 例子：
- >>> multihead_attn = MultiheadAttention(embed_dim, num_heads)
- >>> attn_output, attn_output_weights = multihead_attn(query, key, value)
- '''
- def __init__(self, embed_dim, num_heads, dropout=0., bias=True,
- kdim=None, vdim=None, batch_first=False) -> None:
- # factory_kwargs = {'device': device, 'dtype': dtype}
- super(MultiheadAttention, self).__init__()
- self.embed_dim = embed_dim
- self.kdim = kdim if kdim is not None else embed_dim
- self.vdim = vdim if vdim is not None else embed_dim
- self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim
-
- self.num_heads = num_heads
- self.dropout = dropout
- self.batch_first = batch_first
- self.head_dim = embed_dim // num_heads
- assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"
-
- if self._qkv_same_embed_dim is False:
- self.q_proj_weight = Parameter(torch.empty((embed_dim, embed_dim)))
- self.k_proj_weight = Parameter(torch.empty((embed_dim, self.kdim)))
- self.v_proj_weight = Parameter(torch.empty((embed_dim, self.vdim)))
- self.register_parameter('in_proj_weight', None)
- else:
- self.in_proj_weight = Parameter(torch.empty((3 * embed_dim, embed_dim)))
- self.register_parameter('q_proj_weight', None)
- self.register_parameter('k_proj_weight', None)
- self.register_parameter('v_proj_weight', None)
-
- if bias:
- self.in_proj_bias = Parameter(torch.empty(3 * embed_dim))
- else:
- self.register_parameter('in_proj_bias', None)
- self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
-
- self._reset_parameters()
-
- def _reset_parameters(self):
- if self._qkv_same_embed_dim:
- xavier_uniform_(self.in_proj_weight)
- else:
- xavier_uniform_(self.q_proj_weight)
- xavier_uniform_(self.k_proj_weight)
- xavier_uniform_(self.v_proj_weight)
-
- if self.in_proj_bias is not None:
- constant_(self.in_proj_bias, 0.)
- constant_(self.out_proj.bias, 0.)
-
-
-
- def forward(self, query: Tensor, key: Tensor, value: Tensor, key_padding_mask: Optional[Tensor] = None,
- need_weights: bool = True, attn_mask: Optional[Tensor] = None) -> Tuple[Tensor, Optional[Tensor]]:
- if self.batch_first:
- query, key, value = [x.transpose(1, 0) for x in (query, key, value)]
-
- if not self._qkv_same_embed_dim:
- attn_output, attn_output_weights = multi_head_attention_forward(
- query, key, value, self.num_heads,
- self.in_proj_weight, self.in_proj_bias,
- self.dropout, self.out_proj.weight, self.out_proj.bias,
- training=self.training,
- key_padding_mask=key_padding_mask, need_weights=need_weights,
- attn_mask=attn_mask, use_separate_proj_weight=True,
- q_proj_weight=self.q_proj_weight, k_proj_weight=self.k_proj_weight,
- v_proj_weight=self.v_proj_weight)
- else:
- attn_output, attn_output_weights = multi_head_attention_forward(
- query, key, value, self.num_heads,
- self.in_proj_weight, self.in_proj_bias,
- self.dropout, self.out_proj.weight, self.out_proj.bias,
- training=self.training,
- key_padding_mask=key_padding_mask, need_weights=need_weights,
- attn_mask=attn_mask)
- if self.batch_first:
- return attn_output.transpose(1, 0), attn_output_weights
- else:
- return attn_output, attn_output_weights
-```
-
-
-接下来可以实践一下，并且把位置编码加起来，可以发现加入位置编码和进行多头注意力的前后形状都是不会变的
-
-
-```python
-# 因为batch_first为False,所以src的shape：`(seq, batch, embed_dim)`
-src = torch.randn((2,4,100))
-src = positional_encoding(src,100,0.1)
-print(src.shape)
-multihead_attn = MultiheadAttention(100, 4, 0.1)
-attn_output, attn_output_weights = multihead_attn(src,src,src)
-print(attn_output.shape, attn_output_weights.shape)
-
-# torch.Size([2, 4, 100])
-# torch.Size([2, 4, 100]) torch.Size([4, 2, 2])
-```
-
- torch.Size([2, 4, 100])
- torch.Size([2, 4, 100]) torch.Size([4, 2, 2])
-
-
-***
-## **TransformerEncoderLayer
**
-- Encoder Layer
-
-
-
-
-```python
-class TransformerEncoderLayer(nn.Module):
- r'''
- 参数：
- d_model: 词嵌入的维度（必备）
- nhead: 多头注意力中平行头的数目（必备）
- dim_feedforward: 全连接层的神经元的数目，又称经过此层输入的维度（Default = 2048）
- dropout: dropout的概率（Default = 0.1）
- activation: 两个线性层中间的激活函数，默认relu或gelu
- lay_norm_eps: layer normalization中的微小量，防止分母为0（Default = 1e-5）
- batch_first: 若`True`，则为(batch, seq, feture)，若为`False`，则为(seq, batch, feature)（Default：False）
-
- 例子：
- >>> encoder_layer = TransformerEncoderLayer(d_model=512, nhead=8)
- >>> src = torch.randn((32, 10, 512))
- >>> out = encoder_layer(src)
- '''
-
- def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=F.relu,
- layer_norm_eps=1e-5, batch_first=False) -> None:
- super(TransformerEncoderLayer, self).__init__()
- self.self_attn = MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=batch_first)
- self.linear1 = nn.Linear(d_model, dim_feedforward)
- self.dropout = nn.Dropout(dropout)
- self.linear2 = nn.Linear(dim_feedforward, d_model)
-
- self.norm1 = nn.LayerNorm(d_model, eps=layer_norm_eps)
- self.norm2 = nn.LayerNorm(d_model, eps=layer_norm_eps)
- self.dropout1 = nn.Dropout(dropout)
- self.dropout2 = nn.Dropout(dropout)
- self.activation = activation
-
-
- def forward(self, src: Tensor, src_mask: Optional[Tensor] = None, src_key_padding_mask: Optional[Tensor] = None) -> Tensor:
- src = positional_encoding(src, src.shape[-1])
- src2 = self.self_attn(src, src, src, attn_mask=src_mask,
- key_padding_mask=src_key_padding_mask)[0]
- src = src + self.dropout1(src2)
- src = self.norm1(src)
- src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))
- src = src + self.dropout(src2)
- src = self.norm2(src)
- return src
-
-```
-
-
-```python
-# 用小例子看一下
-encoder_layer = TransformerEncoderLayer(d_model=512, nhead=8)
-src = torch.randn((32, 10, 512))
-out = encoder_layer(src)
-print(out.shape)
-# torch.Size([32, 10, 512])
-```
-
-
-## Transformer layer组成Encoder
-
-```python
-class TransformerEncoder(nn.Module):
- r'''
- 参数：
- encoder_layer（必备）
- num_layers： encoder_layer的层数（必备）
- norm: 归一化的选择（可选）
-
- 例子：
- >>> encoder_layer = TransformerEncoderLayer(d_model=512, nhead=8)
- >>> transformer_encoder = TransformerEncoder(encoder_layer, num_layers=6)
- >>> src = torch.randn((10, 32, 512))
- >>> out = transformer_encoder(src)
- '''
-
- def __init__(self, encoder_layer, num_layers, norm=None):
- super(TransformerEncoder, self).__init__()
- self.layer = encoder_layer
- self.num_layers = num_layers
- self.norm = norm
-
- def forward(self, src: Tensor, mask: Optional[Tensor] = None, src_key_padding_mask: Optional[Tensor] = None) -> Tensor:
- output = positional_encoding(src, src.shape[-1])
- for _ in range(self.num_layers):
- output = self.layer(output, src_mask=mask, src_key_padding_mask=src_key_padding_mask)
-
- if self.norm is not None:
- output = self.norm(output)
-
- return output
-```
-
-```python
-# 例子
-encoder_layer = TransformerEncoderLayer(d_model=512, nhead=8)
-transformer_encoder = TransformerEncoder(encoder_layer, num_layers=6)
-src = torch.randn((10, 32, 512))
-out = transformer_encoder(src)
-print(out.shape)
-# torch.Size([10, 32, 512])
-```
-
-***
-## Decoder Layer:
-
-
-```python
-class TransformerDecoderLayer(nn.Module):
- r'''
- 参数：
- d_model: 词嵌入的维度（必备）
- nhead: 多头注意力中平行头的数目（必备）
- dim_feedforward: 全连接层的神经元的数目，又称经过此层输入的维度（Default = 2048）
- dropout: dropout的概率（Default = 0.1）
- activation: 两个线性层中间的激活函数，默认relu或gelu
- lay_norm_eps: layer normalization中的微小量，防止分母为0（Default = 1e-5）
- batch_first: 若`True`，则为(batch, seq, feture)，若为`False`，则为(seq, batch, feature)（Default：False）
-
- 例子：
- >>> decoder_layer = TransformerDecoderLayer(d_model=512, nhead=8)
- >>> memory = torch.randn((10, 32, 512))
- >>> tgt = torch.randn((20, 32, 512))
- >>> out = decoder_layer(tgt, memory)
- '''
- def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=F.relu,
- layer_norm_eps=1e-5, batch_first=False) -> None:
- super(TransformerDecoderLayer, self).__init__()
- self.self_attn = MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=batch_first)
- self.multihead_attn = MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=batch_first)
-
- self.linear1 = nn.Linear(d_model, dim_feedforward)
- self.dropout = nn.Dropout(dropout)
- self.linear2 = nn.Linear(dim_feedforward, d_model)
-
- self.norm1 = nn.LayerNorm(d_model, eps=layer_norm_eps)
- self.norm2 = nn.LayerNorm(d_model, eps=layer_norm_eps)
- self.norm3 = nn.LayerNorm(d_model, eps=layer_norm_eps)
- self.dropout1 = nn.Dropout(dropout)
- self.dropout2 = nn.Dropout(dropout)
- self.dropout3 = nn.Dropout(dropout)
-
- self.activation = activation
-
- def forward(self, tgt: Tensor, memory: Tensor, tgt_mask: Optional[Tensor] = None,
- memory_mask: Optional[Tensor] = None,tgt_key_padding_mask: Optional[Tensor] = None, memory_key_padding_mask: Optional[Tensor] = None) -> Tensor:
- r'''
- 参数：
- tgt: 目标语言序列（必备）
- memory: 从最后一个encoder_layer跑出的句子（必备）
- tgt_mask: 目标语言序列的mask（可选）
- memory_mask（可选）
- tgt_key_padding_mask（可选）
- memory_key_padding_mask（可选）
- '''
- tgt2 = self.self_attn(tgt, tgt, tgt, attn_mask=tgt_mask,
- key_padding_mask=tgt_key_padding_mask)[0]
- tgt = tgt + self.dropout1(tgt2)
- tgt = self.norm1(tgt)
- tgt2 = self.multihead_attn(tgt, memory, memory, attn_mask=memory_mask,
- key_padding_mask=memory_key_padding_mask)[0]
- tgt = tgt + self.dropout2(tgt2)
- tgt = self.norm2(tgt)
- tgt2 = self.linear2(self.dropout(self.activation(self.linear1(tgt))))
- tgt = tgt + self.dropout3(tgt2)
- tgt = self.norm3(tgt)
- return tgt
-```
-
-
-```python
-# 可爱的小例子
-decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
-memory = torch.randn((10, 32, 512))
-tgt = torch.randn((20, 32, 512))
-out = decoder_layer(tgt, memory)
-print(out.shape)
-# torch.Size([20, 32, 512])
-```
-
-```python
-## Decoder
-```
-
-
-```python
-class TransformerDecoder(nn.Module):
- r'''
- 参数：
- decoder_layer（必备）
- num_layers: decoder_layer的层数（必备）
- norm: 归一化选择
-
- 例子：
- >>> decoder_layer =TransformerDecoderLayer(d_model=512, nhead=8)
- >>> transformer_decoder = TransformerDecoder(decoder_layer, num_layers=6)
- >>> memory = torch.rand(10, 32, 512)
- >>> tgt = torch.rand(20, 32, 512)
- >>> out = transformer_decoder(tgt, memory)
- '''
- def __init__(self, decoder_layer, num_layers, norm=None):
- super(TransformerDecoder, self).__init__()
- self.layer = decoder_layer
- self.num_layers = num_layers
- self.norm = norm
-
- def forward(self, tgt: Tensor, memory: Tensor, tgt_mask: Optional[Tensor] = None,
- memory_mask: Optional[Tensor] = None, tgt_key_padding_mask: Optional[Tensor] = None,
- memory_key_padding_mask: Optional[Tensor] = None) -> Tensor:
- output = tgt
- for _ in range(self.num_layers):
- output = self.layer(output, memory, tgt_mask=tgt_mask,
- memory_mask=memory_mask,
- tgt_key_padding_mask=tgt_key_padding_mask,
- memory_key_padding_mask=memory_key_padding_mask)
- if self.norm is not None:
- output = self.norm(output)
-
- return output
-```
-
-```python
-# 可爱的小例子
-decoder_layer =TransformerDecoderLayer(d_model=512, nhead=8)
-transformer_decoder = TransformerDecoder(decoder_layer, num_layers=6)
-memory = torch.rand(10, 32, 512)
-tgt = torch.rand(20, 32, 512)
-out = transformer_decoder(tgt, memory)
-print(out.shape)
-# torch.Size([20, 32, 512])
-```
-
-
-总结一下，其实经过位置编码，多头注意力，Encoder Layer和Decoder Layer形状不会变的，而Encoder和Decoder分别与src和tgt形状一致
-
-## Transformer
-
-
-```python
-class Transformer(nn.Module):
- r'''
- 参数：
- d_model: 词嵌入的维度（必备）（Default=512）
- nhead: 多头注意力中平行头的数目（必备）（Default=8）
- num_encoder_layers:编码层层数（Default=8）
- num_decoder_layers:解码层层数（Default=8）
- dim_feedforward: 全连接层的神经元的数目，又称经过此层输入的维度（Default = 2048）
- dropout: dropout的概率（Default = 0.1）
- activation: 两个线性层中间的激活函数，默认relu或gelu
- custom_encoder: 自定义encoder（Default=None）
- custom_decoder: 自定义decoder（Default=None）
- lay_norm_eps: layer normalization中的微小量，防止分母为0（Default = 1e-5）
- batch_first: 若`True`，则为(batch, seq, feture)，若为`False`，则为(seq, batch, feature)（Default：False）
-
- 例子：
- >>> transformer_model = Transformer(nhead=16, num_encoder_layers=12)
- >>> src = torch.rand((10, 32, 512))
- >>> tgt = torch.rand((20, 32, 512))
- >>> out = transformer_model(src, tgt)
- '''
- def __init__(self, d_model: int = 512, nhead: int = 8, num_encoder_layers: int = 6,
- num_decoder_layers: int = 6, dim_feedforward: int = 2048, dropout: float = 0.1,
- activation = F.relu, custom_encoder: Optional[Any] = None, custom_decoder: Optional[Any] = None,
- layer_norm_eps: float = 1e-5, batch_first: bool = False) -> None:
- super(Transformer, self).__init__()
- if custom_encoder is not None:
- self.encoder = custom_encoder
- else:
- encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout,
- activation, layer_norm_eps, batch_first)
- encoder_norm = nn.LayerNorm(d_model, eps=layer_norm_eps)
- self.encoder = TransformerEncoder(encoder_layer, num_encoder_layers)
-
- if custom_decoder is not None:
- self.decoder = custom_decoder
- else:
- decoder_layer = TransformerDecoderLayer(d_model, nhead, dim_feedforward, dropout,
- activation, layer_norm_eps, batch_first)
- decoder_norm = nn.LayerNorm(d_model, eps=layer_norm_eps)
- self.decoder = TransformerDecoder(decoder_layer, num_decoder_layers, decoder_norm)
-
- self._reset_parameters()
-
- self.d_model = d_model
- self.nhead = nhead
-
- self.batch_first = batch_first
-
- def forward(self, src: Tensor, tgt: Tensor, src_mask: Optional[Tensor] = None, tgt_mask: Optional[Tensor] = None,
- memory_mask: Optional[Tensor] = None, src_key_padding_mask: Optional[Tensor] = None,
- tgt_key_padding_mask: Optional[Tensor] = None, memory_key_padding_mask: Optional[Tensor] = None) -> Tensor:
- r'''
- 参数：
- src: 源语言序列（送入Encoder）（必备）
- tgt: 目标语言序列（送入Decoder）（必备）
- src_mask: （可选)
- tgt_mask: （可选）
- memory_mask: （可选）
- src_key_padding_mask: （可选）
- tgt_key_padding_mask: （可选）
- memory_key_padding_mask: （可选）
-
- 形状：
- - src: shape:`(S, N, E)`, `(N, S, E)` if batch_first.
- - tgt: shape:`(T, N, E)`, `(N, T, E)` if batch_first.
- - src_mask: shape:`(S, S)`.
- - tgt_mask: shape:`(T, T)`.
- - memory_mask: shape:`(T, S)`.
- - src_key_padding_mask: shape:`(N, S)`.
- - tgt_key_padding_mask: shape:`(N, T)`.
- - memory_key_padding_mask: shape:`(N, S)`.
-
- [src/tgt/memory]_mask确保有些位置不被看到，如做decode的时候，只能看该位置及其以前的，而不能看后面的。
- 若为ByteTensor，非0的位置会被忽略不做注意力；若为BoolTensor，True对应的位置会被忽略；
- 若为数值，则会直接加到attn_weights
-
- [src/tgt/memory]_key_padding_mask 使得key里面的某些元素不参与attention计算，三种情况同上
-
- - output: shape:`(T, N, E)`, `(N, T, E)` if batch_first.
-
- 注意：
- src和tgt的最后一维需要等于d_model，batch的那一维需要相等
-
- 例子:
- >>> output = transformer_model(src, tgt, src_mask=src_mask, tgt_mask=tgt_mask)
- '''
- memory = self.encoder(src, mask=src_mask, src_key_padding_mask=src_key_padding_mask)
- output = self.decoder(tgt, memory, tgt_mask=tgt_mask, memory_mask=memory_mask,
- tgt_key_padding_mask=tgt_key_padding_mask,
- memory_key_padding_mask=memory_key_padding_mask)
- return output
-
- def generate_square_subsequent_mask(self, sz: int) -> Tensor:
- r'''产生关于序列的mask，被遮住的区域赋值`-inf`，未被遮住的区域赋值为`0`'''
- mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
- mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
- return mask
-
- def _reset_parameters(self):
- r'''用正态分布初始化参数'''
- for p in self.parameters():
- if p.dim() > 1:
- xavier_uniform_(p)
-```
-
-```python
-# 小例子
-transformer_model = Transformer(nhead=16, num_encoder_layers=12)
-src = torch.rand((10, 32, 512))
-tgt = torch.rand((20, 32, 512))
-out = transformer_model(src, tgt)
-print(out.shape)
-# torch.Size([20, 32, 512])
-```
-
-到此为止，PyTorch的Transformer库我们已经全部实现，相比于官方的版本，手写的这个少了较多的判定语句。
-## 致谢
-本文由台运鹏撰写，本项目成员重新组织和整理。最后，期待您的阅读反馈和star，谢谢。
diff --git a/requirements.txt b/requirements.txt
index 00806ef..21f349f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -4,3 +4,7 @@ notebook
torch==1.9.0
sklearn
scipy
+matplotlib
+torchtext
+seaborn
+spacy