一、需求：
用python实现去内涵段子里面下载网页当中的图片到本地当中
二、实现：
1、获取要爬取的URL地址
2、设置headers
3、请求网页内容，把html内容转换成XML
4、解析地址内容，进行图片下载
三、开始操作：以下图为例子
1、获取要爬取的URL地址:
url="http://www.neihan8.com/gaoxiaomanhua/index_2.html"
2、设置headers:
headers={"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
3、请求网页内容，把html内容转换成XML
request = urllib2.Request(url,headers=headers)
response = urllib2.urlopen(request).read()
xml = etree.HTML（response）#这个etree是需要在前面导入包的 : from lxml import etree
4、解析地址内容，进行图片下载，我们通过上面的图片进行获取到具体的xpath图片地址.
linklist = content.xpath('/html/body/div[@class="main wrap"]//div[@class="left"]/div[@class="pic-column-list mt10"]/div/a/img/@src')
ps:这个linklist里面存放的是所有这个xpath里面的内容，所以如果需要下载的话需要依次提取
for link in linklist:
image_request = urllib2.Request(link)
response = urllib2.urlopen(image_request).read()
filename = link[10:0]
with open(fileName,"wb") as f:
f.write(response)
上面是分别解释了一下流程，都是手写的代码，第一次写文章比较粗糙大家见谅了。下面是整个代码的内容
import urllib2
from lxmlimport etree
class Spider:
pass
def __init__(self):
self.pageNum =2
self.switch =True
def loadImage(self):
url ="http://www.neihan8.com/gaoxiaomanhua/index_"+str(self.pageNum)+".html"
headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
request = urllib2.Request(url,headers=headers)
response= urllib2.urlopen(request).read()
content = etree.HTML(response)
linklist = content.xpath('/html/body/div[@class="main wrap"]//div[@class="left"]/div[@class="pic-column-list mt10"]/div/a/img/@src')
for image_linkin linklist:
print "downLoading..."
self.writeImage(image_link)
def writeImage(self,link_address):
headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
download_request =urllib2.Request(link_address)
response = urllib2.urlopen(download_request).read()
fileName = link_address[-10:]
with open(fileName,"wb")as f:
f.write(response)
print "downLoad---FINISH"
if __name__ =="__main__":
spider = Spider()
spider.loadImage()
