./experiment/script/faster_rcnn_alt_opt.sh 和 ./experiment/script/faster_rcnn_end2end.sh 都调用了./tools/test_net.py
imdb = get_imdb(args.imdb_name)初始化imdb类，调用factory.py和pascal.py
test_net(net, imdb, max_per_image=args.max_per_image, vis=args.vis)调用./lib/fast_rcnn/test.py
test_net函数==>./lib/fast_rcnn/test.py
1、output_dir = get_output_dir(imdb, net)输出路径
2、scores, boxes = im_detect(net, im, box_proposals)通过网络forward得到scores和boxes
3、遍历除背景外的所有类别，取出每个类别score>thresh(=0.05)的所有框，NMS(阈值为0.3)
4、max_per_image默认为100，每张图像框大于100则取score最大的100个
5、cPickle.dump(all_boxes, f, cPickle.HIGHEST_PROTOCOL)将all_bixes写入文件，f = os.path.join(output_dir, 'detections.pkl')。all_boxes[j][i]:==>j代表类别(不包含背景)，i代表不同图像，all_boxes[j][i]存储二维数组，行数是第i个图像第j个类别的所有框数目，有5列，前四列是框的坐标，最后一列是分数
6、imdb.evaluate_detections(all_boxes, output_dir): ./lib/datasets/pascal_voc.py
def evaluate_detections(self, all_boxes, output_dir):
self._write_voc_results_file(all_boxes)
self._do_python_eval(output_dir)
if self.config['matlab_eval']:
self._do_matlab_eval(output_dir)
if self.config['cleanup']:
for cls in self._classes:
if cls == '__background__':
continue
filename = self._get_voc_results_file_template().format(cls)
os.remove(filename)
evaluate_detections函数
_write_voc_results_file: 根据不同类别写result file，文件中遍历不同图像，所有的boxes，一次写入一个框的img_id score x0+1 y0+1 x1+1 y1+1 六个数
_do_python_eval:根据上面的result file，对不同类别的结果进行评估，写入output文件
_do_python_eval函数:
rec, prec, ap = voc_eval(filename, annopath, imagesetfile, cls, cachedir, ovthresh=0.5, use_07_metric=use_07_metric)是主要函数。(./lib/datasets/voc_eval.py)
filename表示_write_voc_results_file得到的不同类别对应的文件
annopath表示ground-truth的xml文件
imagesetfile表示测试的图像列表
cls是类别
cachedir根据imagesetfile图片名解析annopath－xml文件得到的文件，文件格式是每个图像名对应图像中的boxes和类别等，如不存在会在voc_eval函数生成。
ovthresh overlap的阈值
返回值 recall、precision、AP(Average Precision)平均正确率
def _do_python_eval(self, output_dir = 'output'):
annopath = os.path.join(
self._devkit_path,
'VOC' + self._year,
'Annotations',
'{:s}.xml')
imagesetfile = os.path.join(
self._devkit_path,
'VOC' + self._year,
'ImageSets',
'Main',
self._image_set + '.txt')
cachedir = os.path.join(self._devkit_path, 'annotations_cache')
aps = []
# The PASCAL VOC metric changed in 2010
use_07_metric = True if int(self._year) < 2010 else False
print 'VOC07 metric? ' + ('Yes' if use_07_metric else 'No')
if not os.path.isdir(output_dir):
os.mkdir(output_dir)
for i, cls in enumerate(self._classes):
if cls == '__background__':
continue
filename = self._get_voc_results_file_template().format(cls)
rec, prec, ap = voc_eval(
filename, annopath, imagesetfile, cls, cachedir, ovthresh=0.5,
use_07_metric=use_07_metric)
aps += [ap]
print('AP for {} = {:.4f}'.format(cls, ap))
with open(os.path.join(output_dir, cls + '_pr.pkl'), 'w') as f:
cPickle.dump({'rec': rec, 'prec': prec, 'ap': ap}, f)
print('Mean AP = {:.4f}'.format(np.mean(aps)))
print('~~~~~~~~')
print('Results:')
for ap in aps:
print('{:.3f}'.format(ap))
print('{:.3f}'.format(np.mean(aps)))
...
voc_eval函数:
1、加载或生成cachedir
2、得到当前传入的类别cls，根据cachedir生成每个图像对应的class_recs[imagename] = {'bbox': bbox,'difficult': difficult, 'det': det}difficult通常均为0，det均为False
3、从filename/detpath中读取网络生成的cls类别box结果，所有的box坐标和图像id都按score高低排序，注意：这里图像id的数目不等于总的test文件数，可能有多个相同的文件名，以为取图像id的某一行只对应一个box
４、按顺序取box和img_id，与img_id对应的所有ground-truth的box计算overlaps，如果大于阈值ovthresh = 0.5 且R['det'][jmax]=false则为tp，否则为fp。R['det'][jmax]应该是当一个框和一个ground-truth计算是tp，其他的框就不能和这个ground-truth是tp
5、计算recall、precision
6、计算ap = voc_ap(rec, prec, use_07_metric)，当use_07_metric使用11点差值法计算ap，否则Average precision as defined by TREC. 计算方法是找到recall变化时（TP被recall时）的点，求这些点对应的precision的均值。
问题:
BB = BB[sorted_ind, :]
IndexError: too many indices for array
运行时出现错误，应该是网络问题，没有学到boxes
def voc_ap(rec, prec, use_07_metric=False):
""" ap = voc_ap(rec, prec, [use_07_metric])
Compute VOC AP given precision and recall.
If use_07_metric is true, uses the
VOC 07 11 point method (default:False).
"""
if use_07_metric:
# 11 point metric
ap = 0.
for t in np.arange(0., 1.1, 0.1):
if np.sum(rec >= t) == 0:
p = 0
else:
p = np.max(prec[rec >= t])
ap = ap + p / 11.
else:
# correct AP calculation
# first append sentinel values at the end
mrec = np.concatenate(([0.], rec, [1.]))
mpre = np.concatenate(([0.], prec, [0.]))
# compute the precision envelope
for i in range(mpre.size - 1, 0, -1):
mpre[i - 1] = np.maximum(mpre[i - 1], mpre[i])
# to calculate area under PR curve, look for points
# where X axis (recall) changes value
i = np.where(mrec[1:] != mrec[:-1])[0]
# and sum (\Delta recall) * prec
ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1])
return ap
