背景
公司视频应用,需要针对用户的操作数据分析,得出应用内所有的视频的权重分析,方便以后更好的进行推荐视频操作.目前已有的log格式主要有:
action vid uid
view video video_001 NaN
play video video_001 user_00x
用户分为注册用户与游客两种,操作主要有view,like,comment,upload,download几种.为了方便处理,第一阶段只考虑针对游客访问的视频进行处理.这样形成一个只有action与uid的字典形式.处理数据主要使用python的pandas库.
清洗数据
日志初期action包括view,like,comment,upload,uid包括用户id以及代表游客的NaN空值.第一步需要把这些action具体量化,为了简单量化初期:view = 1 , like = 2 , comment = 3 , upload = 3 , download = 3.日志文件以csv格式存储.导入数据后进行处理:
import pandas as pd
# pandas读取csv
data = pd.read_csv('data.csv')
# action为playvideo的设置为1,即打分为1
data.loc[data['action']=='playvideo','action']=1
# 把NaN(即游客)值重置为0
data=data.fillna(value = '0')
# 过滤出uid值为0的数据
tourist_data = data.loc[data['uid']=='0']
del tourist_data['uid']
# tourist_data为action与uic的矩阵
DataFrame如下
action vid
0 1 2c9f91345c3ed855015c5ee9cb904681
1 1 2c9f91345c3ed855015c52649f962d4f
2 -1 2c9f91345c2007f8015c2deee14c18cb
3 1 2c9f91345c3ed855015c5ee9cb904681
4 3 2c9f91345c2007f8015c2deee14c18cb
针对这个数据,需要进行清洗.找出重复的vid,并相加对应的action.得出的矩阵就是需要的vid的score排列.
首先DataFrame生成vid的list
vid_list=tourist_data['vid'].values.tolist()
问题分解为寻找list中重复的数值,并把数值对应的index记录下来.需要用到两个库enumerate与defaultdict
enumerate可以把list生成带有index的dict
defaultdict可以对list形成的dict进行统计处理
# 生成带有index的列表
In [189]: [(v,i) for i,v in enumerate(vid_list)]
Out[189]:
[('2c9f91345c3ed855015c5ee9cb904681', 0),
('2c9f91345c3ed855015c52649f962d4f', 1),
('2c9f91345c2007f8015c2deee14c18cb', 2),
('2c9f91345c3ed855015c5ee9cb904681', 3),
('2c9f91345c2007f8015c2deee14c18cb', 4),
('2c9f91345c3ed855015c52649f962d4f', 5),
('2c9f91345c2007f8015c2deee14c18cb', 6),
('2c9f91345bf13cac015bfce28ef31002', 7),
('2c9f91345c3ed855015c5ee9cb904681', 8),
('2c9f91345c3ed855015c52649f962d4f', 9)]
# 利用defaultdict生成对重复vid处理后的dict
In [190]: vid_dict = defaultdict(list)
In [191]: for key, value in [(v, i) for i, v in enumerate(vid_list)]:
...: vid_dict[key].append(value)
...:
In [192]: vid_dict
Out[192]:
defaultdict(list,
{'2c9f91345bf13cac015bfce28ef31002': [7],
'2c9f91345c2007f8015c2deee14c18cb': [2, 4, 6],
'2c9f91345c3ed855015c52649f962d4f': [1, 5, 9],
'2c9f91345c3ed855015c5ee9cb904681': [0, 3, 8]})
vid对应的list即为对应的index位置,利用index位置就可以为score_list进行处理累加
# 生成score list
score_list = tourist_data['action'].values.tolist()
In [222]: vid_dict = defaultdict(list)
In [223]: for key, value in [(v, i) for i, v in enumerate(vid_list)]:
...: vid_dict[key].append(value)
In [224]: vid_dict
Out[224]:
defaultdict(list,
{'2c9f91345bf13cac015bfce28ef31002': [7],
'2c9f91345c2007f8015c2deee14c18cb': [2, 4, 6],
'2c9f91345c3ed855015c52649f962d4f': [1, 5, 9],
'2c9f91345c3ed855015c5ee9cb904681': [0, 3, 8]})
In [227]: rank_list=[]
...: for i in vid_dict:
...: score = 0
...: for index in vid_dict[i]:
...: score += int(score_list[index])
...: rank_list.append(score)
In [231]: vid_list = []
In [232]: for i in vid_dict:
...: vid_list.append(i)
In [233]: vid_list
Out[233]:
['2c9f91345bf13cac015bfce28ef31002',
'2c9f91345c2007f8015c2deee14c18cb',
'2c9f91345c3ed855015c5ee9cb904681',
'2c9f91345c3ed855015c52649f962d4f']
In [234]: rank_list
Out[234]: [1, 4, 5, 3]
In [237]: vid_score = pd.DataFrame({'score':rank_list,'vid':vid_list})
生成score与vid的矩阵
针对生成的数据进行分析
使用altair对数据进行画图分析
# 导入csv文件
import pandas as pd
from altair import Chart, load_dataset
%matplotlib inline
vids_score = pd.read_csv('./res/vid_score.csv')
bins = [0, 10, 20, 30, 40, 50,60,70,80,90, 100,150,200,300,400,500,1000,1500]
scores = pd.cut(vids_score['score'], bins)
def get_stats(group):
return {'count': group.count()}
grouped = vids_score['score'].groupby(scores)
bin_counts = grouped.apply(get_stats).unstack()
bin_counts
bin_counts.index = ['0~10', '10~20', '20~30', '30~40', '40~50', '50~60', '60~70',
'70~80', '80~90', '90~100','100-150','150-200','200-300','300-400','400-500','500-1000','1000-1500']
bin_counts.index.name = 'score'
plt=bin_counts.plot(kind='bar', alpha=0.5, rot=1,width = 0.8,align='center',figsize=(15,4))
