在github看到了开源项目IPProxyTool,因而以此为基础自己维护一个抓取免费代理ip的项目。个人对项目进行了一些改动，细节这里就不提了，这里简单记录下我新添加的爬虫中的goubanjia，nyloner。因为这这个网址使用了一些反爬技巧。
爬虫过程经常会遇到JavaScript反爬虫，如果JavaScript代码可读性强，那么写出相应的python代码，效率是最高的。如果JavaScript代码可读性非常差，也可以考虑使用基于selenium的phantomjs。本文采用的是第一种。
首先查看goubanjia的网页源码
<td class="ip">
<span style='display: inline-block;'></span>
<span style='display: inline-block;'>1</span>
<p style='display: none;'>5</p>
<span>5</span>
<div style='display: inline-block;'>6</div>
<p style='display: none;'></p>
<span></span>
<span style='display: inline-block;'>.</span>
<p style='display: none;'></p>
<span></span>
<p style='display: none;'>67</p>
<span>67</span>
<span style='display: inline-block;'>.</span>
<div style='display: inline-block;'></div>
<p style='display: none;'>2</p>
<span>2</span>
<p style='display: none;'></p>
<span></span>
<div style='display: inline-block;'>19</div>
<span style='display: inline-block;'></span>
<span style='display: inline-block;'>.6</span>
<div style='display: inline-block;'>1</div>:
<span class="port CFACE">8364</span>
</td>
代理ip好办，用xpath提取即可。
ipList = row.xpath('./td[1]/div[@style]/text() | ./td[1]/span[@style]/text() | ./td[1]/span/text()')[:-1]
ip = ''.join(ipList)
item['ip'] = ip
portstr = row.xpath('./td[1]/span[contains(@class,"port")]/@class')[0] #['port EAEDCA']
端口<span class="port CFACE">8255</span>，8255却与实际端口3128不一样，玄机在哪呢？
首先找个相同的端口3128看看： <span class="port CFACE">8110</span> ，数字不同，"port CFACE"完全相同，我们把关注点放在"port XXXXX"上面，很容易知道实际端口可以通过"CFACE"解密而得。直接上解密代码：
def get_poxy(self,word):
num_list = []
for item in word:
num = 'ABCDEFGHIZ'.find(item)
num_list.append(str(num))
port = int("".join(num_list)) >> 0x3
return port
再来看看nyloner，这是个个人维护的网站，首先还是右键查看网页源码，会发现完全没有代理ip的信息，不必惊慌，我们先F12打开开发者工具，然后点下一页，第二页的ip显示出来了，同时会发现仅仅多出了一个请求https://www.nyloner.cn/proxy?page=2&num=15&token=7a65c88d591e3096135db3d9770f26b8&t=1508988424.显然第二页的ip信息，就在这个请求的回复里面，我们看下回复的内容:
{"status": "true", "list": "OUofAQ89MwA2BS4AKCY/SCAFJx89PU1YIi1RCiM9OQEiJjMAGitWFwExDU8lLA0fPzowWCUXEhUkSj4fDDInADYFLgAoGC8BIhZeHT8tSRUhByQKIy0DFiAPHV83FQtXKTEsTiZdAhI4FBYLJRQgCiMAWBYjMSMbNDgiGyghDQolLAoSEF0zXCUUChUkAwsYIyEzADUvLgABNQINNjwNUzstGhUiKiBBIi0pGCIxO1wwKCoWKhgnSyMFO1Y4XUkfJiYWHA8hLQEhDDMANDgmGSkYDUogBScSPz1NXCEHBgEnOggYDFY4XjMWAAkvCAkOIysvCz4tOAILKQkGNCoPWScmERc0KCpdKSYrDiA7J1c7KjwUIAQsQCETOVwkVkIdMCQcAAQqLxcjBi8LPzo8WSMELEYiEyleICZGFjQ4IgApMS8XDyhXHBYtGlomLQZGIz0LGCQcBQkzXz0eByU7FyMGLws/BDgUIRdVCiM6XBYgDDMWNDgAFysYAUgiFgpcPi07WyVcCQUkAwMIJB83FjYvW1wqIVpLITxaVDwXGh8mLQEFDEomXyQfHQkzFghaKjYJFyAsLwsWKRUYNT0GRCc6DxYjITddNTgqGSkmJ0slKysdPQQwXiAEMEEkSlwcJy0BABg0LgAqGy8XITsrVz4EPFkgPVELIhMlGCEmER0wLwsZB0EkSSYVAQI4FB4bJRcSFSRKPh8MMicANgUuACgYLwEiFl4dPy1JFSEHJAojLQMWIA8dXzcVC1cpMSxOJl0CEjgUFgslFCAKIBBYFyExGRs0BVsWKiYNFyAsLwsRKUAVCy0GRCc6D1sjJhEdMC8LXgQlXxUmFQECOBQwGyI6BgYjLS0bIw8rCTQ4KlgoGDdPIjsNCxQ9DgsKFwECDToPWScmERY0ODYbKCYFTCAFJxA/OhJeJRcSFSRKLhkNCyMANgUuACo2LxcgLC8LFikVGDU9BkQnOg8WIyE3XTU4KhkpJidLJSsrHT0EMF4gBDBBJEpcHCctAQAYNC4AKhsvFyE7IxA8FB4ZIjosQyITKV0gHBEdMC8LGQdBJEkmFQECOBQeGyAqJBwiOi0BCiIeGiM/DFgsMQ0AISsrVj46PBsjOixAJz0pFyEPO1w2BjpcL0FeCiUnHQsTNjgCIAckHCMtC14iDxlfNQYqWisbWkojLA0WOy0dGw1dL0IkAwMIJB8rXzQ4LhQvCxkeJlw8FRA5LAIgByQcIxMtFyAcQhY0L18XKxsvASE7ARw8BBZdIT0BSyI6LlgkVx4ZMxYACS8IJwwhOwlWPgQ0XSMEIAsjEA8cJyYWGRtfJV4vCAEeJhUJEj8qOAIjLSQcCj4AGzQ2EVgwLwwXKDYrSiA7KxI+OjBeJiogCiETJV0hDydfM19fHSw6HRcNNy8LPQc4AiI6DkIiEyleIg83FzQFWxcrNjcXICwvCxEpQBULLQZEJzoPXiMfO1o0PwwdLDEKSQ04Xwk4FBYLJRQsBSMtDxsjMTMaNAY2CSgmK08hBTdTPDooAgk9EhULAAgfDSYRWDAvDBYqJjcMITs7ED8EKBYjBCwFJAAbCCRWMBgaAj4AKhsvFyMrL1E/LRofJi0BQg8uXQMkHx0JMxYmGSgmDQ0hOy8RPwQgCyI6IEQjEzVZIDEjAB8/GAkACwoJDywNUzstGhQhBDQHIy01GCIPN140BVtcKDENCiUsChIQXTNcJRQKFSQDORUjDxUWMxUYCS9BPAkOODsLPQc4AiIEJAogAFwXIyZCFzcFLhYoJgEAIgUBVD8XHVUjLSdFJEsAGCQfHQkzFipbKCFaSiAFK1U/B00UIgQCHCI6LQENIksXHS8MWCwxDUghFSdRPz0aHyYtAUIPLl0DJB8dCTMWJhkoJg0NITsvET8EIAsiOiBEIxM1WSAxPwAfOl9T"}
什么鬼？list的内容看起来像乱码，我们回头找网页的javascript，发现有如下几个脚本:
<script src="../static/js/jquery.min.js"></script>
<script src="../static/js/bootstrap.min.js"></script>
<script src="../static/js/md5.js"></script>
<script src="../static/js/proxy.js"></script>
<script src="../static/js/base64.js"></script>
这其中md5，base64是常见的加密方法，很自然就关注到proxy.js，我们打开proxy.js看看里面的源码：
function get_proxy_ip(page, num, click_btn) {
var timestamp = Date.parse(new Date());
timestamp = timestamp / 1000;
var token = md5(String(page) + String(num) + String(timestamp));
$.get('../proxy?page=' + page + '&num=' + num + '&token=' + token + '&t=' + timestamp, function (result) {
if (result.status === 'true') {
var setHtml = "";
$("#ip-list").html(setHtml);
var encode_str = result.list;
var items = strToJson(decode_str(encode_str));
for (var index = 0; index < items.length; ++index) {
item = items[index];
setHtml += "<tr>\n<td>" + (index + 1) + "</td>\n";
setHtml += "<td>" + item.ip.toString() + "</td>\n";
setHtml += "<td>" + item.port.toString() + "</td>\n";
setHtml += "<td>" + item.time.toString() + "</td>\n</tr>\n";
}
$("#ip-list").html(setHtml);
if (click_btn === 'next') {
document.getElementById("last-page").disabled = false;
if (items.length < 15) {
document.getElementById("next-page").disabled = true;
}
} else {
document.getElementById("next-page").disabled = false;
if (page === 1) {
document.getElementById("last-page").disabled = true;
}
}
}
});
}
很开心里面的代码可读性非常高，看到这个函数function get_proxy_ip(page, num, click_btn)了吗？里面的代码简单分为1：先根据页数+每页ip数+时间戳经过md5加密获得一个token，2：然后根据页数&每页ip数&token&时间戳组成一个url，我们依样画瓢写出get_url()函数。
def geturl(self,page,limitnum=15):
timestamp = datetime.now().timestamp()
timestamp = int(timestamp)
token = str(page) + str(limitnum) + str(timestamp)
token = hashlib.md5(token.encode()).hexdigest()
url = "https://www.nyloner.cn/proxy?page={page}&num={num}&token={token}&t={t}".format(page=page,num=limitnum,token=token,t=timestamp)
return url
使用requests测试一下，返回了一长串类似乱码的字符串。这里面的信息应该如何解密呢？我们继续看刚才的get_proxy_ip(),可以看到解密的代码，先使用decode_str解密，再转成json。
var items = strToJson(decode_str(encode_str))
function decode_str(scHZjLUh1) {
scHZjLUh1 = Base64["\x64\x65\x63\x6f\x64\x65"](scHZjLUh1);
key = '\x6e\x79\x6c\x6f\x6e\x65\x72';
len = key["\x6c\x65\x6e\x67\x74\x68"];
code = '';
for (i = 0; i < scHZjLUh1["\x6c\x65\x6e\x67\x74\x68"]; i++) {
var coeFYlqUm2 = i % len;
code += window["\x53\x74\x72\x69\x6e\x67"]["\x66\x72\x6f\x6d\x43\x68\x61\x72\x43\x6f\x64\x65"](scHZjLUh1["\x63\x68\x61\x72\x43\x6f\x64\x65\x41\x74"](i) ^ key["\x63\x68\x61\x72\x43\x6f\x64\x65\x41\x74"](coeFYlqUm2))
}
return Base64["\x64\x65\x63\x6f\x64\x65"](code)
}
function strToJson(str) {
return (new Function("return " + str))();
}
通过代码，可以知道decode_str()的原理：先把 scHZjLUh1通过base64解密，然后每7个字节分别与key="nyloner"进行异或运算，得到一个新字符串，再base64解密。同样依样画瓢可以下出decode_str(),代码如下：
def decode_str(self,scHZjLUh1):
scHZjLUh1 = base64.decodestring(scHZjLUh1)
key=b'nyloner'
lenth= len(key)
schlenth=len(scHZjLUh1)
code=''
for i in range(schlenth):
coeFYlqUm2 = i % lenth
code+= chr(scHZjLUh1[i] ^ key[coeFYlqUm2])
code = base64.decodestring(code.encode())
iplist = code.decode()
return iplist
我们运行一下代码，即可成功获取ip列表。
