利用 Python 分析网站访问日志文件

场景介绍

有一个前台的 Web 应用，框架会记录访问日志，并定期归档，存储在特定的目录，目录格式如下：
/onlinelogs/应用名/环境名/年/月/日/小时/，例如 /onlinelogs/<app_name>/Prod/2018/08/07/01。
在该目录下：

  • 访问日志文件可能有多个，文件名以 access-log 开头
  • 已压缩为 .gz 文件，并且只读，例如：
    访问日志文件
  • 访问日志文件中，有部分行是记录 HTTP 请求的，格式如下所示：
    • 从中可以看出，请求的目标资源，响应码，客户端信息
222.67.225.134 - - [04/Aug/2018:01:16:44 +0000] "GET /?ref=as_cn_ags_resource_tb&ck-tparam-anchor=123067 HTTP/1.1" 200 7798 "https://gs.amazon.cn/resources.html/ref=as_cn_ags_hnav1_re_class" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15"

222.67.225.134 - - [04/Aug/2018:01:16:54 +0000] "GET /tndetails?tnid=3be3f34dee8a4bf08baa072a478fc882 HTTP/1.1" 200 9152 "https://gs.amazon.cn/sba/?ref=as_cn_ags_resource_tb&ck-tparam-anchor=123067&tnm=Offline" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15"

222.67.225.134 - - [04/Aug/2018:01:17:37 +0000] "GET /paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435 HTTP/1.1" 200 7138 "https://gs.amazon.cn/sba/paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435" "Mozilla/5.0 (Linux; Android 8.1.0; DE106 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 AliApp(DingTalk/4.5.3) com.alibaba.android.rimet/0 Channel/10006872 language/zh-CN"

222.67.225.134 - - [04/Aug/2018:01:17:39 +0000] "GET /paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435 HTTP/1.1" 200 7138 "https://gs.amazon.cn/sba/paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435" "Mozilla/5.0 (Linux; Android 8.1.0; DE106 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 AliApp(DingTalk/4.5.3) com.alibaba.android.rimet/0 Channel/10006872 language/zh-CN"

222.67.225.134 - - [04/Aug/2018:01:17:40 +0000] "GET /paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435 HTTP/1.1" 200 7138 "https://gs.amazon.cn/sba/paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435" "Mozilla/5.0 (Linux; Android 8.1.0; DE106 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 AliApp(DingTalk/4.5.3) com.alibaba.android.rimet/0 Channel/10006872 language/zh-CN"

140.243.121.197 - - [04/Aug/2018:01:17:41 +0000] "GET /?ref=as_cn_ags_resource_tb HTTP/1.1" 302 - "https://gs.amazon.cn/resources.html/ref=as_cn_ags_hnav1_re_class" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"

分析目标

  • 记录不同时间段的访问量
  • 统计 PC 和 Mobile 端访问量
  • 统计不同页面的访问量

基本思想：

  • 由于原日志文件已压缩，并且只读，所以需要创建一个临时目录 /tmp/logs/unziped_logs 来解压缩日志文件
  • 利用正则表达式 ^access\-log.*gz+$ 过滤日志文件
  • 利用正则表达式 ^.*GET (.*) HTTP.*$ 过滤日志文件中的 HTTP 请求
  • 通过日志行是否有 Mobile 来判断客户端

代码如下：（部分内容屏蔽）

#!/usr/bin/python3

import os
import os.path
import re
import shutil
import gzip
from collections import defaultdict

# define the ziped and unziped log file directorys
source_logs_dir = '/onlinelogs/<app_name>/Prod'
unziped_logs_dir = '/tmp/logs/unziped_logs'

# clear the unziped log file directory if exists
if os.path.exists(unziped_logs_dir):
    shutil.rmtree(unziped_logs_dir)

# create the unziped log file directory
os.mkdir(unziped_logs_dir)

# regex used to match target log file name
log_file_name_regex = re.compile(r'^access\-log.*gz+$')

# regex used to match HTTP request
http_request_regex = re.compile(r'^.*GET.*gs.amazon.cn')

# regex used to match request page
request_page_regex = re.compile(r'^.*GET (.*) HTTP.*$')
# request_page_regex = re.compile(r'^.*GET (.*)\?.*$')

# a dictionary to store the HTTP request count of each day
day_count = defaultdict(int)

# a dictionary to store the count of each device (PC or Mobile)
device_count = defaultdict(int)
device_count['PC'] = 0
device_count['Mobile'] = 0

# a dictionary to store the count of each request page
request_page_count = defaultdict(int)

for root, dirs, files in os.walk(source_logs_dir):
    for name in files:
        # find the target log files
        if log_file_name_regex.search(name):
            # parst the day
            day = root[-13:-3]

            # copy the target log files
            shutil.copyfile(os.path.join(root, name), os.path.join(unziped_logs_dir, name))

            # unzip the log files
            unziped_log_file = gzip.open(os.path.join(unziped_logs_dir, name), 'rb')

            http_request_count = 0
            pc_count = 0
            mobile_count = 0
            for line in unziped_log_file:
                if(http_request_regex.search(line)):
                    # parse the request page
                    regex_obj = request_page_regex.search(line)
                    request_page = regex_obj.group(1)
                    # remove params of the request page
                    if('?' in request_page):
                        request_page = request_page[:request_page.find('?')]

                    http_request_count = http_request_count + 1

                    if('Mobile' in line):
                        mobile_count = mobile_count + 1
                    else:
                        pc_count = pc_count + 1

                    # update the count of each request page
                    if(request_page in request_page_count):
                                        request_page_count[request_page] = request_page_count[request_page] + 1
                                else:
                                        request_page_count[request_page] = 1

            # update the HTTP request count of each day
            if(day in day_count):
                day_count[day] = day_count[day] + http_request_count
            else:
                day_count[day] = http_request_count

            # update the count of each device (PC or Mobile)
            device_count['PC'] = device_count['PC'] + pc_count
            device_count['Mobile'] = device_count['Mobile'] + mobile_count

            # remvoe the original zip log files
            os.remove(os.path.join(unziped_logs_dir, name))

# print the HTTP request count of each day
total = 0
print 'HTTP request count of each day'
for day, count in sorted(day_count.items()):
    print day, ':', count
    total = total + count
print 'Total = ', total

print '###############################'

total = 0
print 'count of each device (PC or Mobile)'
# print the count of each device (PC or Mobile)
for device, count in sorted(device_count.items()):
        print device, ':', count
    total = total + count
print 'Total = ', total

print '###############################'

total = 0
print 'count of each request page'
# print the count of each request page
for request_page, count in sorted(request_page_count.items()):
        print request_page, ':', count
    total = total + count
print 'Total = ', total
©著作权归作者所有,转载或内容合作请联系作者
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

推荐阅读更多精彩内容

  • oracle错误集合
    ORA-00001: 违反唯一约束条件 (.) 错误说明：当在唯一索引所对应的列上键入重复值时，会触发此异常。 O...
    我想起个好名字阅读 5,498评论 0赞 9
  • 关于Mongodb的全面总结
    关于Mongodb的全面总结 MongoDB的内部构造《MongoDB The Definitive Guide》...
    中v中阅读 32,099评论 2赞 89
  • Apache httpd.conf配置详解
    常用配置指令说明 1. ServerRoot：服务器的基础目录，一般来说它将包含conf/和logs/子目录，其它...
    小僧有礼了阅读 4,600评论 0赞 5
  • Spring Cloud
    Spring Cloud为开发人员提供了快速构建分布式系统中一些常见模式的工具（例如配置管理，服务发现，断路器，智...
    卡卡罗2017阅读 135,280评论 19赞 139
  • 可你是蓝颜
    早已过了浪荡不羁的年纪，时光磨平了我年少时的棱角，我懊悔的是，你再也看不到，我的男孩，祝你一生平安喜乐。 一别两...
    安念kk阅读 287评论 1赞 3