论文-Recurrent neural network based language model(RNNLM)

1. 简称

论文《Recurrent neural network based language model》简称RNNLM，作者Tomas Mikolov，经典的循环/递归神经语言模型。

2. 摘要

提出了一种新的基于递归神经网络的语言模型(RNN LM)及其在语音识别中的应用。

结果表明，与现有的退避语言模型相比，通过使用几个RNN LMs的混合，可以获得大约50%的困惑减少。

语音识别实验表明，当比较针对相同数据量训练的模型时，“华尔街日报”任务的单词错误率降低约18%，而在难度更大的NIST RT05任务上，即使退避模型训练的数据量比RNN LM多得多，单词错误率也减少约5%。

我们提供了充足的经验证据，以表明连接主义语言模型优于标准的n-gram技术，除了它们的高计算(训练)复杂性。

3. 核心

RNNLM

在我们的工作中，我们使用了一种通常被称为简单的递归神经网络或Elman网络的架构。这可能是递归神经网络的最简单的可能版本，并且非常容易实现和训练。

网络具有输入层 $x$ 、隐藏层 $s$ (也称为上下文层或状态)和输出层 $y$ 。对网络的时间t的输入是 $x(t)$ ，输出表示为 $y(t)$ ，并且 $s(t)$ 是网络的状态(隐藏层)。输入向量 $x(t)$ 是通过连接表示当前单词的向量 $w$ 而形成的，并且在时间 $t−1$ 从上下文层 $s$ 中的神经元输出。然后，输入、隐藏和输出层被计算如下：

$x(t)=w(t)+s(t-1)\tag{3.1}$
$s_j(t)=f(\sum_ix_i(t)u_{ji})\tag{3.2}$
$y_k(t)=g(\sum_js_j(t)v_{kj})\tag{3.3}$

其中 $f(z)$ 是S型激活函数：
$f(z)=\frac{1}{1+e^{-z}}\tag{3.4}$

并且 $g(z)$ 是softmax函数：
$g(z_m)=\frac{e^{z_m}}{\sum_ke^{z_k}}\tag{3.5}$

对于初始化，可以将 $s(0)$ 设置为小值的向量，如0.1-当处理大量数据时，初始化并不重要。

在接下来的时间步长中， $s(t+1)$ 是 $s(t)$ 的副本。输入向量 $x(t)$ 表示使用1-of-N编码和先前上下文层编码的时间 $t$ 中的字-向量 $x$ 的大小等于词汇表V的大小(实际上可以是30000−200000)加上上下文层的大小。上下文(隐藏)层的大小通常是30−500个隐藏单元。

基于我们的实验，隐藏层的大小应该反映训练数据量-对于大量的数据，需要大的隐藏层。

把上面的形式写成向量形式：
$\begin{cases} s(t)=f(Uw(t)+Ws(t-1)) \\ y(t)=g(Vs(t)) \end{cases}\tag{3.6}$

4. 代码编写

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset
import torch.nn.functional as F

sentence = """When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gazed on now,
Will be a totter'd weed of small worth held:
Then being asked, where all thy beauty lies,
Where all the treasure of thy lusty days;
To say, within thine own deep sunken eyes,
Were an all-eating shame, and thriftless praise.
How much more praise deserv'd thy beauty's use,
If thou couldst answer 'This fair child of mine
Shall sum my count, and make my old excuse,'
Proving his beauty by succession thine!
This were to be new made when thou art old,
And see thy blood warm when thou feel'st it cold.""".split()

# 准备词表与相关字典
vocab = set(sentence)
print(vocab)
word2index = {w:i for i, w in enumerate(vocab)}
print(word2index)
index2word = {i:w for i, w in enumerate(vocab)}
print(index2word)

# 准备N-gram训练数据 each tuple is ([word_i-1], target word)
data_source = []
data_target = []
for i in range(len(sentence)-1):
    data_source.append(word2index[sentence[i]])
    data_target.append(word2index[sentence[i+1]])

print(data_source)
print(data_target)

class MyDataset(Dataset):
    def __init__(self, words, labels):
        self.words = words
        self.labels = labels

    def __getitem__(self, index):  # 返回的是tensor
        input, target = self.words[index], self.labels[index]
        return input, target

    def __len__(self):
        return len(self.words)

# 数据批量化
dataset = MyDataset(data_source, data_target)
train_loader = DataLoader(dataset, batch_size=2, shuffle=True)

# for _, x in enumerate(train_loader):
#     print(x[0].unsqueeze(-1).size())
#     print(x[1])
#     break

# 模型所需参数
EMBEDDING_DIM = 10
HIDDEN_DIM = 128

# 创建模型
class RNNLanguageModler(nn.Module):

    def __init__(self, vocab_size, embedding_dim, hidden_dim):
        super(RNNLanguageModler, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.rnn = nn.RNN(embedding_dim, hidden_dim, batch_first=True, num_layers=1)
        self.linear = nn.Linear(hidden_dim, vocab_size)

    def forward(self, inputs):
        embeds = self.embedding(inputs)
        output, hidden = self.rnn(embeds)
        out = self.linear(hidden.view(hidden.size(1), -1))
        return out

losses = []
loss_function = nn.CrossEntropyLoss()
model = RNNLanguageModler(len(vocab), EMBEDDING_DIM, HIDDEN_DIM)
optimizer = optim.SGD(model.parameters(), lr = 0.001)

for epoch in range(100):
    total_loss = 0

    for _, batch in enumerate(train_loader):
        # Step 1. Prepare the inputs to be passed to the model
        x = batch[0].view(-1, 1)

        # Step 2. Before passing in a new instance, you need to zero out the gradients from the old instance
        model.zero_grad()

        # Step 3. Run forward pass
        out = model(x)

        # Step 4. Compute your loss function.
        loss = loss_function(out, batch[1])

        # Step 5. Do the backword pass and update the gradient
        loss.backward()
        optimizer.step()

        # Get the Python number from a 1-element Tensor by calling tensor.item()
        total_loss += loss.item()
    
    losses.append(total_loss)

print(losses) # The loss decreased every iteration over the training data!

# 结果
{"excuse,'", "youth's", 'thou', 'sunken', 'gazed', 'cold.', 'mine', 'his', 'all', 'worth', 'thine', 'within', 'besiege', "deserv'd", 'forty', 'field,', 'so', 'see', 'praise', "beauty's", 'trenches', 'the', 'small', 'old,', 'to', 'an', 'by', 'be', "totter'd", 'Thy', 'child', 'more', 'own', 'When', 'where', 'much', 'beauty', 'Then', 'days;', 'lusty', 'were', 'warm', 'count,', 'new', 'proud', 'in', 'now,', 'If', 'To', 'deep', 'shame,', "feel'st", 'Will', 'winters', 'dig', 'old', 'held:', 'weed', 'treasure', 'sum', 'art', 'blood', 'Shall', 'And', 'my', 'Proving', 'This', 'thriftless', 'made', 'Were', "'This", 'brow,', 'asked,', 'and', 'eyes,', 'livery', 'couldst', 'How', 'praise.', 'Where', 'succession', 'make', 'answer', 'a', 'being', 'lies,', 'on', 'thine!', 'say,', 'thy', 'all-eating', 'use,', 'it', 'fair', 'of', 'when', 'shall'}
{"excuse,'": 0, "youth's": 1, 'thou': 2, 'sunken': 3, 'gazed': 4, 'cold.': 5, 'mine': 6, 'his': 7, 'all': 8, 'worth': 9, 'thine': 10, 'within': 11, 'besiege': 12, "deserv'd": 13, 'forty': 14, 'field,': 15, 'so': 16, 'see': 17, 'praise': 18, "beauty's": 19, 'trenches': 20, 'the': 21, 'small': 22, 'old,': 23, 'to': 24, 'an': 25, 'by': 26, 'be': 27, "totter'd": 28, 'Thy': 29, 'child': 30, 'more': 31, 'own': 32, 'When': 33, 'where': 34, 'much': 35, 'beauty': 36, 'Then': 37, 'days;': 38, 'lusty': 39, 'were': 40, 'warm': 41, 'count,': 42, 'new': 43, 'proud': 44, 'in': 45, 'now,': 46, 'If': 47, 'To': 48, 'deep': 49, 'shame,': 50, "feel'st": 51, 'Will': 52, 'winters': 53, 'dig': 54, 'old': 55, 'held:': 56, 'weed': 57, 'treasure': 58, 'sum': 59, 'art': 60, 'blood': 61, 'Shall': 62, 'And': 63, 'my': 64, 'Proving': 65, 'This': 66, 'thriftless': 67, 'made': 68, 'Were': 69, "'This": 70, 'brow,': 71, 'asked,': 72, 'and': 73, 'eyes,': 74, 'livery': 75, 'couldst': 76, 'How': 77, 'praise.': 78, 'Where': 79, 'succession': 80, 'make': 81, 'answer': 82, 'a': 83, 'being': 84, 'lies,': 85, 'on': 86, 'thine!': 87, 'say,': 88, 'thy': 89, 'all-eating': 90, 'use,': 91, 'it': 92, 'fair': 93, 'of': 94, 'when': 95, 'shall': 96}
{0: "excuse,'", 1: "youth's", 2: 'thou', 3: 'sunken', 4: 'gazed', 5: 'cold.', 6: 'mine', 7: 'his', 8: 'all', 9: 'worth', 10: 'thine', 11: 'within', 12: 'besiege', 13: "deserv'd", 14: 'forty', 15: 'field,', 16: 'so', 17: 'see', 18: 'praise', 19: "beauty's", 20: 'trenches', 21: 'the', 22: 'small', 23: 'old,', 24: 'to', 25: 'an', 26: 'by', 27: 'be', 28: "totter'd", 29: 'Thy', 30: 'child', 31: 'more', 32: 'own', 33: 'When', 34: 'where', 35: 'much', 36: 'beauty', 37: 'Then', 38: 'days;', 39: 'lusty', 40: 'were', 41: 'warm', 42: 'count,', 43: 'new', 44: 'proud', 45: 'in', 46: 'now,', 47: 'If', 48: 'To', 49: 'deep', 50: 'shame,', 51: "feel'st", 52: 'Will', 53: 'winters', 54: 'dig', 55: 'old', 56: 'held:', 57: 'weed', 58: 'treasure', 59: 'sum', 60: 'art', 61: 'blood', 62: 'Shall', 63: 'And', 64: 'my', 65: 'Proving', 66: 'This', 67: 'thriftless', 68: 'made', 69: 'Were', 70: "'This", 71: 'brow,', 72: 'asked,', 73: 'and', 74: 'eyes,', 75: 'livery', 76: 'couldst', 77: 'How', 78: 'praise.', 79: 'Where', 80: 'succession', 81: 'make', 82: 'answer', 83: 'a', 84: 'being', 85: 'lies,', 86: 'on', 87: 'thine!', 88: 'say,', 89: 'thy', 90: 'all-eating', 91: 'use,', 92: 'it', 93: 'fair', 94: 'of', 95: 'when', 96: 'shall'}
[33, 14, 53, 96, 12, 89, 71, 63, 54, 49, 20, 45, 89, 19, 15, 29, 1, 44, 75, 16, 4, 86, 46, 52, 27, 83, 28, 57, 94, 22, 9, 56, 37, 84, 72, 34, 8, 89, 36, 85, 79, 8, 21, 58, 94, 89, 39, 38, 48, 88, 11, 10, 32, 49, 3, 74, 69, 25, 90, 50, 73, 67, 78, 77, 35, 31, 18, 13, 89, 19, 91, 47, 2, 76, 82, 70, 93, 30, 94, 6, 62, 59, 64, 42, 73, 81, 64, 55, 0, 65, 7, 36, 26, 80, 87, 66, 40, 24, 27, 43, 68, 95, 2, 60, 23, 63, 17, 89, 61, 41, 95, 2, 51, 92]
[14, 53, 96, 12, 89, 71, 63, 54, 49, 20, 45, 89, 19, 15, 29, 1, 44, 75, 16, 4, 86, 46, 52, 27, 83, 28, 57, 94, 22, 9, 56, 37, 84, 72, 34, 8, 89, 36, 85, 79, 8, 21, 58, 94, 89, 39, 38, 48, 88, 11, 10, 32, 49, 3, 74, 69, 25, 90, 50, 73, 67, 78, 77, 35, 31, 18, 13, 89, 19, 91, 47, 2, 76, 82, 70, 93, 30, 94, 6, 62, 59, 64, 42, 73, 81, 64, 55, 0, 65, 7, 36, 26, 80, 87, 66, 40, 24, 27, 43, 68, 95, 2, 60, 23, 63, 17, 89, 61, 41, 95, 2, 51, 92, 5]
[261.4552655220032, 261.2737054824829, 261.0931797027588, 260.9118666648865, 260.73165369033813, 260.55105447769165, 260.3709635734558, 260.1912055015564, 260.011239528656, 259.8317127227783, 259.65210247039795, 259.47280836105347, 259.2937173843384, 259.1144289970398, 258.93572902679443, 258.75625705718994, 258.5772943496704, 258.398380279541, 258.2191872596741, 258.04021644592285, 257.8635997772217, 257.68174028396606, 257.50267362594604, 257.3237919807434, 257.1443405151367, 256.96484422683716, 256.7852563858032, 256.60576152801514, 256.4258427619934, 256.2457194328308, 256.06648778915405, 255.8852858543396, 255.70466136932373, 255.52420616149902, 255.34329175949097, 255.16089391708374, 254.9813151359558, 254.79758167266846, 254.61532545089722, 254.43314790725708, 254.25010776519775, 254.06746768951416, 253.88412618637085, 253.69995212554932, 253.51536560058594, 253.33151292800903, 253.14607954025269, 252.9601068496704, 252.77443265914917, 252.58800411224365, 252.40100288391113, 252.2142734527588, 252.02623414993286, 251.83781957626343, 251.64954042434692, 251.45993757247925, 251.27176523208618, 251.08007860183716, 250.88931608200073, 250.69780254364014, 250.50601959228516, 250.3132610321045, 250.11953020095825, 249.92643022537231, 249.7322335243225, 249.5378646850586, 249.342125415802, 249.14556121826172, 248.94805097579956, 248.7504014968872, 248.55196523666382, 248.35296511650085, 248.1533761024475, 247.95326471328735, 247.751886844635, 247.5496575832367, 247.34726858139038, 247.14575910568237, 246.9398694038391, 246.7349500656128, 246.52930426597595, 246.3258776664734, 246.11556768417358, 245.90801095962524, 245.69976925849915, 245.48987817764282, 245.27924585342407, 245.06853342056274, 244.85593819618225, 244.64315629005432, 244.42985773086548, 244.21367502212524, 243.99954438209534, 243.78382658958435, 243.56601929664612, 243.3485345840454, 243.12900829315186, 242.9099476337433, 242.68885612487793, 242.46759343147278]

参考文献

Recurrent neural network based language model

论文-Recurrent neural network based language model(RNNLM)

1. 简称

2. 摘要

3. 核心

4. 代码编写

推荐阅读更多精彩内容