NVCC / CUDA / cuDNN / TensorRT是什么关系
CUDA
- CUDA是nvidia封装了对GPU的编程模型而开发的编程平台，能够方便开发者做GPU程序开发，其中集成nvcc编译器驱动
In November 2006, NVIDIA introduced CUDA®, a general purpose parallel computing platform and programming model that leverages the parallel compute engine in NVIDIA GPUs to solve many complex computational problems in a more efficient way than on a CPU.
NVCC
- NVCC是GPU程序的编译器，将GPU程序分阶段编译成可执行程序，类似于gcc
The compilation trajectory involves several splitting, compilation, preprocessing, and merging steps for each CUDA source file. It is the purpose of nvcc, the CUDA compiler driver, to hide the intricate details of CUDA compilation from developers.
cuDNN
- cuDNN是针对深度学习做的更上层的SDK封装，底层基于CUDA API。主要针对training阶段的实现
tensorRT
- TensorRT目的和cuDNN相同，方便深度学习开发并在实现上做了相应的优化。但是主要针对的是inference阶段的实现
cuDNN 和 tensorRT区别
TENSORRT的目标很简单，当你训练好了网络，需要部署的时候，能够尽量地加速线上的inference的throughput。TENSORRT跟CUDNN最大的区别在于，CUDNN里面的API都是ProModule，或者说ProLayer,也就是说，在优化卷积的时候，我只知道卷积的参数，然后再做优化；而输入给TENSORRT的信息，是包含整个网络的信息，如layer结构是什么样子，连接关系是怎样的。
所以说，TENSORRT可以做很多CUDNN做不了的优化，这里面列举了一些常用的：
举一个例子，卷积属于比较大的操作。在卷积之后，经常会连接一些小的操作，比如pooling。如果用CUDNN,实际上把计算过程会变成两个API的调用：一个做卷积,一个pooling。这样的话，卷积和pooling之间，首先卷积的结果，要把它存到memory里面去，然后再去做pooling操作，你需要把数据从memory读回来，然后再把最终结果存回去。
Network LayerFusion的一个思想，指的是我既然知道前面是卷积，后面是pooling或者其他操作的话，在有可能的情况下，在前面的这个layer做完之后，我不是把结果立马写回memory里面去，而有一些后面的小操作可以直接做的情况下，我可以先做了之后，再寄存器或者memory。把数据存在那个位置，然后把最终结果算完之后，再一次存回内存里面去，可以显著地提高性能。
MXNET问题排查
-
mxnet程序编译完成后上线运行，对于第一次运行的机器启动时间超过5分钟，但是随后运行的时间都在5~8s左右
怀疑是OS Cache问题：清空系统缓存重试，无效果
怀疑是cudnn版本问题：从cudnn5换成cudnn6后，无效果
-
怀疑是本地local cache的问题，找到CUDA的JIT编译 描述相关的文章
最终问题定位到是因为mxnet配置的CUDA编译参数-arch -code中没有对应GPU型号的配置，导致在编译阶段无法生成最终的fatbinary。而程序上线后发现运行GPU型号不匹配，重新进行JIT编译并在本地生成JIT Cache，所以随后的运行操作速度都会加快
-
解决方案：
1. 如果编译机器和运行机器GPU型号相同，则去掉CUDA编译参数-arch -code，则直接生成匹配GPU型号的fat binary
2. 直接配置对应GPU型号的配置，可以参考：GPU Feature List、 Virtual Architecture Feature List
-
下图是NVCC的两阶段编译图示
image.png<input type="file" accept=".jpg, .jpeg, .png, .gif" style="display: none;">
Roughly speaking, the code compilation flow goes like this:
CUDA C/C++ device code source --> PTX --> SASS
The virtual architecture (e.g. compute_20, whatever is specified by -arch compute...) determines what type of PTX code will be generated. The additional switches (e.g. -code sm_21) determine what type of SASS code will be generated. SASS is actually executable object code for a GPU (machine language). An executable can contain multiple versions of SASS and/or PTX, and there is a runtime loader mechanism that will pick appropriate versions based on the GPU actually being used.
As you point out, one of the handy features of GPU operation is JIT-compile. JIT-compile will be done by the GPU driver (does not require the CUDA toolkit to be installed) anytime a suitable PTX code is available but a suitable SASS code is not.
One advantage of including multiple virtual architectures (i.e. multiple versions of PTX), then, is that you have executable compatibility with a wider variety of target GPU devices (although some devices may trigger a JIT-compile to create the necessary SASS).
One advantage of including multiple "real GPU targets" (i.e. multiple SASS versions) is that you can avoid the JIT-compile step, when one of those target devices is present.
If you specify a bad set of options, it's possible to create an executable that won't run (correctly) on a particular GPU.
One possible disadvantage of specifying a lot of these options is code size bloat. Another possible disadvantage is compile time, which will generally be longer as you specify more options.
It's also possible to create excutables that contain no PTX, which may be of interest to those trying to obscure their IP.
Creating PTX suitable for JIT should be done by specifying a virtual architecture for the codeswitch.
GPU相关资料：
NVIDIA CUDA Compiler Driver NVCC
CUDA C Programming Guide
CUDA Samples
cuDNN Developer Guide
