大数据文摘出品
作者：钱天培、魏子敏
训练神经网络是个极为枯燥的工作。与其盯着Learning Curves发呆，或许可以调动一下其他感官，一起做点更有意思的事情。
比如说，眼睛看久了，可以让耳朵也活动活动。
一位酷爱弹吉他的数据科学家就希望，在调参时把其他器官也调动起来共同监督神经网络的训练。
他用一段程序将神经网络训练时的梯度转化成音频，这样，你就可以通过听不同的声音模式知晓训练情况。
先来一段我们制作的小样——梯度的声音！
听出来了嘛？这可是Adam optimizer的弹拨下，梯度发出的声音。
umm没错，确实很难听… 不过先别急着下结论，因为… 更难听的还在后头呢……
通常，我们需要通过测量许多不同的指标来完成训练，例如准确度、损失、梯度等。多数调参工程师会选择将这些指标整合，并在TensorBoard上绘制可视化图。
而这位名叫Christian S. Perone的数据科学家就厌倦了一直盯着各种参数的传统训练方式，经常玩音乐的他开发了一个小系统，把梯度训练变成声音，并且发布了全部120行代码。
Christian S. Perone也是位吉他手
他用一段程序将神经网络训练时的梯度转化成音频，通过听不同的声音模式就知道训练情况。
这是个讨巧的训练监督方式，毕竟，听觉是目前在神经网络训练中很少被用到的感官。而事实上，人类的听觉感官也非常敏锐，可以非常好地区分非常小的特征，例如节奏和音调，即便是很微小或者短暂的变动，人们也很容易有直观的感受。
先一起来看几个非常简单的训练例子。
以下的几段声音显示了我们使用每层的梯度范数进行的合成声音，以及使用不同设置（如不同学习率、优化器、动量）对MNIST进行卷积神经网络训练的训练步骤等。
因为微信编辑限制，每篇文章只能插入一段音频，我们将后三段声音转化成了视频，请大家点击收听。
此段表示，在第一个epoch的前200个step中使用batch size为10的训练结果。我们选取了0.01的learning rate。音高越高，层的范数（norm）就越高，不同批次之前我们插入了短暂的静音。注意渐变在时间内增加。
与上述相同，但我们把learning rate调高到了0.1。
与上述相同，但是学习率更高，梯度爆炸并最后发散了，注意高音。嗯，听到最后觉得这个网络大概是死了吧，
相同的设置，但学习率高达1.0，批量大小为256.注意渐变如何爆炸，然后有NaNs导致最终声音。
花了这么大力气，我就想知道，靠耳朵调参真的靠谱吗？
没错，如你所料，可能没什么卵用。虽然在上面的例子中，我们可以很明显得听出不同参数的差别，但这些例子都是比较极端的情况。
所以，为什么还要制作这些音频呢？
大概是因为，调参真的是太无聊了吧。
最后，还是放上Christian发布的完整开源代码，你需要安装PyAudio和PyTorch来运行代码。感兴趣的读者，不妨自己试试看。
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">import pyaudio</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">import numpy as np</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">import wave</span></span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">import torch</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">import torch.nn as nn</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">import torch.nn.functional as F</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">import torch.optim as optim</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">from torchvision import datasets, transforms</span></span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">class Net(nn.Module):</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">def __init__(self):</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">super(Net, self).__init__()</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">self.conv1 = nn.Conv2d(1, 20, 5, 1)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">self.conv2 = nn.Conv2d(20, 50, 5, 1)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">self.fc1 = nn.Linear(4*4*50, 500)</span></span> |
1
| <span class="code-snippet_outer">self.fc2 = nn.Linear(500, 10)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">self.ordered_layers = [self.conv1,</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">self.conv2,</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">self.fc1,</span></span> |
1
| <span class="code-snippet_outer">self.fc2]</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">def forward(self, x):</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">x = F.relu(self.conv1(x))</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">x = F.max_pool2d(x, 2, 2)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">x = F.relu(self.conv2(x))</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">x = F.max_pool2d(x, 2, 2)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">x = x.view(-1, 4*4*50)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">x = F.relu(self.fc1(x))</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">x = self.fc2(x)</span></span> |
1
| <span class="code-snippet_outer">return F.log_softmax(x, dim=1)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">def open_stream(fs):</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">p = pyaudio.PyAudio()</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">stream = p.open(format=pyaudio.paFloat32,</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">channels=1,</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">rate=fs,</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">output=True)</span></span> |
1
| <span class="code-snippet_outer">return p, stream</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">def generate_tone(fs, freq, duration):</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">npsin = np.sin(2 * np.pi * np.arange(fs*duration) * freq / fs)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">samples = npsin.astype(np.float32)</span></span> |
1
| <span class="code-snippet_outer">return 0.1 * samples</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">def train(model, device, train_loader, optimizer, epoch):</span></span> |
1
| <span class="code-snippet_outer">model.train()</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">fs = 44100</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">duration = 0.01</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">f = 200.0</span></span> |
1
| <span class="code-snippet_outer">p, stream = open_stream(fs)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer">frames = []</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">for batch_idx, (data, target) in enumerate(train_loader):</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">data, target = data.to(device), target.to(device)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">optimizer.zero_grad()</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">output = model(data)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">loss = F.nll_loss(output, target)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">loss.backward()</span></span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">norms = []</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">for layer in model.ordered_layers:</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">norm_grad = layer.weight.grad.norm()</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">norms.append(norm_grad)</span></span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">tone = f + ((norm_grad.numpy()) * 100.0)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">tone = tone.astype(np.float32)</span></span> |
1
| <span class="code-snippet_outer">samples = generate_tone(fs, tone, duration)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer">frames.append(samples)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">silence = np.zeros(samples.shape[0] * 2,</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">dtype=np.float32)</span></span> |
1
| <span class="code-snippet_outer">frames.append(silence)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer">optimizer.step()</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;"># Just 200 steps per epoach</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">if batch_idx == 200:</span></span> |
1
| <span class="code-snippet_outer">break</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">wf = wave.open("sgd_lr_1_0_bs256.wav", 'wb')</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">wf.setnchannels(1)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">wf.setsampwidth(p.get_sample_size(pyaudio.paFloat32))</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">wf.setframerate(fs)</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">wf.writeframes(b''.join(frames))</span></span> |
1
| <span class="code-snippet_outer">wf.close()</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">stream.stop_stream()</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">stream.close()</span></span> |
1
| <span class="code-snippet_outer">p.terminate()</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">def run_main():</span></span> |
1
| <span class="code-snippet_outer">device = torch.device(<span class="code-snippet__string">"cpu"</span>)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">train_loader = torch.utils.data.DataLoader(</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">datasets.MNIST('../data', train=True, download=True,</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">transform=transforms.Compose([</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">transforms.ToTensor(),</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">transforms.Normalize((0.1307,), (0.3081,))</span></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">])),</span></span> |
1
| <span class="code-snippet_outer">batch_size=256, shuffle=True)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">model = Net().to(device)</span></span> |
1
| <span class="code-snippet_outer">optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">for epoch in range(1, 2):</span></span> |
1
| <span class="code-snippet_outer">train(model, device, train_loader, optimizer, epoch)</span> |
1
| <span class="code-snippet_outer"><br></span> |
1
| <span class="code-snippet_outer"><span style="font-size: 15px;color: rgb(51, 51, 51);letter-spacing: 2px;">if __name__ == "__main__":</span></span> |
1
| <span class="code-snippet_outer">run_main()</span> |
1
| <span class="code-snippet_outer"><br></span> |
朋友会在“发现-看一看”看到你“在看”的内容