0. 安装流程

• 首先安装 Anaconda；
• 然后安装 cuda, cudnn；注意：cuda编译安装很慢并且不方便；

conda install cudatoolkit=10.1 
conda install cudnn=7.6.5

• 最后安装 tensorflow，使用的是豆瓣源；

pip install tensorflow-gpu==2.3.0 -i https://pypi.douban.com/simple

1. 基本操作

• hello, word打印： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/1_Introduction/helloworld.ipynb
• 张量基本操作（加\减\乘\除、求和、矩阵乘法）： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/1_Introduction/basic_operations.ipynb

2. 数据管理

2.1 加载&解析数据

• 数据格式包括：numpy数组、图片、csv格式、自定义数据： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/5_DataManagement/load_data.ipynb

数据格式.png

2.2 TFRecord数据格式

• 创建&加载TFRecord数据格式： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/5_DataManagement/tfrecords.ipynb
• 多进程创建tfrecord数据格式： 依赖python multiprocessing模块；
• spark创建tfrecord数据格式： https://github.com/linkedin/spark-tfrecord

spark tfrecord举例.png

3. 模型管理

• 保存&加载 TF checkpoint（使用low-level和high-level API接口）
  https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/4_Utils/save_restore_model.ipynb
• 低级别API，保存和加载模型；

# Save weights and optimizer variables.
# Create a dict of variables to save.
vars_to_save = {"W": W, "b": b, "optimizer": optimizer}
# TF Checkpoint, pass the dict as **kwargs.
checkpoint = tf.train.Checkpoint(**vars_to_save)
# TF CheckpointManager to manage saving parameters.
saver = tf.train.CheckpointManager(
      checkpoint, directory="./tf-example", max_to_keep=5)

# Save variables.
saver.save()

# Set checkpoint to load data.
vars_to_load = {"W": W, "b": b, "optimizer": optimizer}
checkpoint = tf.train.Checkpoint(**vars_to_load)
# Restore variables from latest checkpoint.
latest_ckpt = tf.train.latest_checkpoint("./tf-example")
checkpoint.restore(latest_ckpt)

• 高级别API，保存和加载模型；

from tensorflow.keras import Model, layers

# Create TF Model.
class NeuralNet(Model):
    # Set layers.
    def __init__(self):
        super(NeuralNet, self).__init__(name="NeuralNet")
        # First fully-connected hidden layer.
        self.fc1 = layers.Dense(64, activation=tf.nn.relu)
        # Second fully-connected hidden layer.
        self.fc2 = layers.Dense(128, activation=tf.nn.relu)
        # Third fully-connecter hidden layer.
        self.out = layers.Dense(num_classes, activation=tf.nn.softmax)

    # Set forward pass.
    def __call__(self, x, is_training=False):
        x = self.fc1(x)
        x = self.out(x)
        if not is_training:
            # tf cross entropy expect logits without softmax, so only
            # apply softmax when not training.
            x = tf.nn.softmax(x)
        return x

# Build neural network model.
neural_net = NeuralNet()

# 模型训练...

# Save TF model.
neural_net.save_weights(filepath="./tfmodel.ckpt")

# Load saved weights.
neural_net.load_weights(filepath="./tfmodel.ckpt")

4. 自定义layers, modules

• 自定义层、模型：https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/4_Utils/build_custom_layers.ipynb
• 需要导入keras模块：from tensorflow.keras import Model, layers

4.1 自定义layers

• 自定义layer类必须实现：__init__, build, call 三个方法； build方法用于定义本层使用的网络参数；call方法用于定义本层的前向传播过程；get_config是可选的；

# Create a custom layer, extending TF 'Layer' class.
# Layer compute: y = relu(W * x + b)
class CustomLayer1(layers.Layer):
    
    # Layer arguments.
    def __init__(self, num_units, **kwargs):
        # Store the number of units (neurons).
        self.num_units = num_units
        super(CustomLayer1, self).__init__(**kwargs)
        
    def build(self, input_shape):
        # Note: a custom layer can also include any other TF 'layers'.
        shape = tf.TensorShape((input_shape[1], self.num_units))
        # Create weight variables for this layer.
        self.weight = self.add_weight(name='W',
                                      shape=shape,
                                      initializer=tf.initializers.RandomNormal,
                                      trainable=True)
        self.bias = self.add_weight(name='b',
                                    shape=[self.num_units])
        # Make sure to call the `build` method at the end
        super(CustomLayer1, self).build(input_shape)

    def call(self, inputs):
        x = tf.matmul(inputs, self.weight)
        x = x + self.bias
        return tf.nn.relu(x)

    def get_config(self):
        base_config = super(CustomLayer1, self).get_config()
        base_config['num_units'] = self.num_units
        return base_config

4.2 自定义modules

• 自定义module类必须实现两个方法：__init__, __call__方法；__call__方法用于定义模型前向传播过程；

# Create TF Model.
class CustomNet(Model):
    
    def __init__(self):
        super(CustomNet, self).__init__()
        # Use custom layers created above.
        self.layer1 = CustomLayer1(64)
        self.layer2 = CustomLayer2(64)
        self.out = layers.Dense(num_classes, activation=tf.nn.softmax)

    # Set forward pass.
    def __call__(self, x, is_training=False):
        x = self.layer1(x)
        x = tf.nn.relu(x)
        x = self.layer2(x)
        if not is_training:
            # tf cross entropy expect logits without softmax, so only
            # apply softmax when not training.
            x = tf.nn.softmax(x)
        return x

# Build neural network model.
custom_net = CustomNet()

4.3 自定义损失函数、评价指标、训练过程

• 自定义损失函数，cross_entropy交叉熵损失函数；评价指标accuracy；

# Cross-Entropy loss function.
def cross_entropy(y_pred, y_true):
    y_true = tf.cast(y_true, tf.int64)
    crossentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y_true, logits=y_pred)
    return tf.reduce_mean(crossentropy)

# Accuracy metric.
def accuracy(y_pred, y_true):
    # Predicted class is the index of highest score in prediction vector (i.e. argmax).
    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))
    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Adam optimizer.
optimizer = tf.optimizers.Adam(learning_rate)

• 自定义训练过程；

# Optimization process. 
def run_optimization(x, y):
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = custom_net(x, is_training=True)
        loss = cross_entropy(pred, y)

        # Compute gradients.
        gradients = g.gradient(loss, custom_net.trainable_variables)

        # Update W and b following gradients.
        optimizer.apply_gradients(zip(gradients, custom_net.trainable_variables))

# Run training for the given number of steps.
for step, (batch_x, batch_y) in enumerate(train_data.take(training_steps), 1):
    # Run the optimization to update W and b values.
    run_optimization(batch_x, batch_y)
    
    if step % display_step == 0:
        pred = custom_net(batch_x, is_training=False)
        loss = cross_entropy(pred, batch_y)
        acc = accuracy(pred, batch_y)
        print("step: %i, loss: %f, accuracy: %f" % (step, loss, acc))

5. TensorBoard可视化

https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/4_Utils/tensorboard.ipynb

6. 模型实现例子

6.1 基础模型

• linear regression模型： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/2_BasicModels/linear_regression.ipynb
• Logistic regression模型： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/2_BasicModels/logistic_regression.ipynb
• word2vec模型： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/2_BasicModels/word2vec.ipynb
• GBDT模型： https://github.com/nlpming/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/2_BasicModels/gradient_boosted_trees.ipynb

6.2 前馈神经网络

6.3 卷积神经网络

6.4 循环神经网络

6.5 非监督式算法

参考资料

• TensorFlow Examples
  https://github.com/aymericdamien/TensorFlow-Examples
• 最新 Tensorflow 2.3 极简安装，五分钟搞定 https://zhuanlan.zhihu.com/p/183981491
• TFRecord 使用例子：https://gist.github.com/MSWon/124e0d3aa22e38b17c347afdcbf5bcc5