前言
当前，在目标检测领域，基于深度学习的目标检测方法在准确度上碾压传统的方法。基于深度学习的目标检测先后出现了RCNN,FastRCNN,FasterRCNN, 端到端目标检测方法YOLO,YOLO-9000,YOLO-v3, MobileNet-SSD,以及Mask-RCNN等。MobileNet是一种轻量级的网络，本文基于MobileNet-SSD+opencv实现目标检测。
开发环境
- windows10 x64
- IntellJ IDEA
- opencv3.4.2
- Visual Studio 2017(测试C++版本MobileNet-SSD)
MobileNet-SSD简介
opencv调用MobileNet-SSD
C++版本MobileNet-SSD的运行
目前MobileNet有基于caffe框架训练好的，caffe本身就是C++实现的，因此网上的大部分opencv调用MobileNet都是C++代码。本人先采用vs+opencv3.4.1成功测试之后，再用Java代码进行移植。
Visual Stuido 2017配置opencv过程就不赘述了
MobileNet-SSD训练好的caffe模型在上面的MobileNet-SSD caffe链接下载
C++代码：
#include<iostream>
#include<opencv2/opencv.hpp>
#include<opencv2/dnn.hpp>
using namespace std;
using namespace cv;
using namespace cv::dnn;
class Object
{
public:
Object();
Object(int index, float confidence, String name, Rect rect);
~Object();
public:
int index;
String name;
float confidence;
Rect rect;
private:
};
Object::Object() {
}
Object::Object(int index,float confidence,String name,Rect rect) {
this->index = index;
this->confidence = confidence;
this->name = name;
this->rect = rect;
}
Object::~Object() {
}
//----------------------------全局常量----------------------------------
//配置好protxt文件，网络结构描述文件
//配置好caffemodel文件，训练好的网络权重
const String PROTOTX_FILE ="MobileNetSSD\\MobileNetSSD_deploy.prototxt";
const String CAFFE_MODEL_FILE = "MobileNet-SSD\\MobileNetSSD_deploy.caffemodel";
const String classNames[] = { "background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair",
"cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor" };
const float CONF_THRESH = 0.7f;
int main() {
//------------------------实例化网络----------------------------
Net mobileNetSSD = readNetFromCaffe(PROTOTX_FILE, CAFFE_MODEL_FILE);
if (mobileNetSSD.empty()) {
cerr << "加载网络失败！" << endl;
return -1;
}
TickMeter t;
//----------------------设置网络输入-----------------------
Mat srcImg = imread("D:\\小可爱\\Java学习\\day-6\\代码\\opencv调用MobileNet-SSD\\MobileNet-test.jpg");
//将二维图像转换为CNN输入的张量Tensor,作为网络的输入
mobileNetSSD.setInput(blobFromImage(srcImg, 1.0 / 127.5, Size(300, 300), Scalar(127.5, 127.5, 127.5), true, false));
t.start();
//--------------------CNN网络前向计算----------------------
Mat netOut = mobileNetSSD.forward();
t.stop();
cout << "检测时间=" << t.getTimeMilli() << "ms" << endl;
//----------------------解析计算结果-----------------------
vector<Object> detectObjects;
Mat detectionResult(netOut.size[2], netOut.size[3], CV_32F, netOut.ptr<float>());
for (int i = 0; i < detectionResult.rows; i++) {
//目标类别的索引
int objectIndex = detectionResult.at<float>(i, 1);
//检测结果置信度
float confidence = detectionResult.at<float>(i, 2);
//根据置信度阈值过滤掉置信度较小的目标
if (confidence<CONF_THRESH) {
continue;
}
//反归一化，得到图像坐标
int xLeftUp = static_cast<int>(detectionResult.at<float>(i, 3)*srcImg.cols);
int yLeftUp = static_cast<int>(detectionResult.at<float>(i, 4)*srcImg.rows);
int xRightBottom = static_cast<int>(detectionResult.at<float>(i, 5)*srcImg.cols);
int yRightBottom = static_cast<int>(detectionResult.at<float>(i, 6)*srcImg.rows);
//矩形框
Rect rect(Point{ xLeftUp,yLeftUp }, Point{ xRightBottom,yRightBottom });
//保存结果
detectObjects.push_back(Object{ objectIndex,confidence,classNames[objectIndex],rect });
}
//------------------------显示结果-----------------------------------
int count = 0;
for (auto& i:detectObjects) {
rectangle(srcImg, i.rect, Scalar(0, 255, 255), 2);
putText(srcImg, i.name, i.rect.tl(), 1, 1.8, Scalar(255, 0, 0),2);
cout << "第" << count << "个目标：" << i.name << "\t" << i.rect << "\t" << i.confidence << endl;
count++;
}
imshow("MobileNet-SSD", srcImg);
waitKey(0);
}
测试图像：
MobileNet-test.jpg
代码说明
- 相关API
-
blobFromImage()将输入的二维图像转换为一个4维的张量/高维矩阵，4维张量的顺序为NCHW(N个数,C通道数，H高度，W宽度 )
-
/** @brief Creates 4-dimensional blob from series of images. Optionally resizes and
* crops @p images from center, subtract @p mean values, scales values by @p scalefactor,
* swap Blue and Red channels.
* @param images input images (all with 1-, 3- or 4-channels).
* @param size spatial size for output image
* @param mean scalar with mean values which are subtracted from channels. Values are intended
* to be in (mean-R, mean-G, mean-B) order if @p image has BGR ordering and @p swapRB is true.
* @param scalefactor multiplier for @p images values.
* @param swapRB flag which indicates that swap first and last channels
* in 3-channel image is necessary.
* @param crop flag which indicates whether image will be cropped after resize or not
* @details if @p crop is true, input image is resized so one side after resize is equal to corresponing
* dimension in @p size and another one is equal or larger. Then, crop from the center is performed.
* If @p crop is false, direct resize without cropping and preserving aspect ratio is performed.
* @returns 4-dimansional Mat with NCHW dimensions order.
*/
CV_EXPORTS_W Mat blobFromImages(const std::vector<Mat>& images, double scalefactor=1.0,
Size size = Size(), const Scalar& mean = Scalar(), bool swapRB=true, bool crop=true);
-
net.forward()深度神经网络的前向传播计算,返回值也是一个4D的张量
/** @brief Runs forward pass to compute output of layer with name @p outputName.
* @param outputName name for layer which output is needed to get
* @return blob for first output of specified layer.
* @details By default runs forward pass for the whole network.
*/
CV_WRAP Mat forward(const String& outputName = String());
-
调试步骤，输入图像为RGB图像，包含dog,person
image.png -
调试步骤，网络的最终计算输出是一个张量，需要转换为一个2维矩形Mat,是一个7(width)*6(height)的float矩阵，第2列的整数代表目标类别的索引，第3列代表检测结果的置信度，最后4列是归一化的目标矩形框。网络只输出置信度最大的前6个目标，因此输出矩阵的形状为7x6.
image.png
运行结果
image.png
image.png
检测到了5个目标，2个dog+3个person，后面坐着的2个人以及椅子上中间的人露检了。
Java版本MobileNet-SSD的移植
笔者查看了好几遍Java封装的Mat,没有发现如何将一个高维/4D张量转换为2D矩阵的方法，比较坑。在C++ Mat中，Mat有一个成员变量size,进一步查看之后发现是一个结构体，里面封装了指针，在Java Mat中找不到对应的。卡在这里了，只能用JNI了，好麻烦。
以后有时间再继续倒腾Java版本的 MobileNet-SSD。
