在实现了用session登录成功后(https://www.jianshu.com/p/be0e73b52776
),尝试在Scrapy中如何携带cookie来实现登入

难点:

需要全程cookie传递，带着cookie去下载验证码图片

参考:

https://www.jianshu.com/p/72bca2dcac03

https://www.cnblogs.com/think-a-lot/p/9597952.html

思路:

(1)重写start_url
(2)保存验证码到本地，传递cookie
(3)获取参数，post方法带参数请求实现登录

import scrapy
import requests

headers = {
            'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'
        }
login_url = 'https://so.gushiwen.org/user/login.aspx?from=http://so.gushiwen.org/user/collect.aspx'

class GushiwenSpider(scrapy.Spider):
    name = 'gushiwen'
    allowed_domains = ['gushiwen.org']
    start_urls = ['https://www.gushiwen.org/']
    # login_url='https://so.gushiwen.org/user/login.aspx?from=http://so.gushiwen.org/user/collect.aspx'


    # 重写start_url 方法，获取验证码图片，传递cookie
    def start_requests(self):
        return [scrapy.Request(url='https://so.gushiwen.org/RandCode.ashx',headers=headers,callback=self.get_code,meta={'cookiejar':1})]

    #保存验证码图片，带cookie
    def get_code(self,response):
        with open ('code4.png','wb') as f:
            f.write(response.body)
        return [scrapy.Request(url=login_url,callback=self.get_VIEWSTATE,meta={'cookiejar':response.meta['cookiejar']})]

    # 获取 VIEWSTATE 参数
    def get_VIEWSTATE(self,response):
        VIEWSTATE = response.xpath('//*[@id="__VIEWSTATE"]/@value').extract_first()
        print(VIEWSTATE)
        code=input('请输入验证码：')
        formdata = {
                '__VIEWSTATE': VIEWSTATE,
                '__VIEWSTATEGENERATOR': 'C93BE1AE',
                'from': 'http://so.gushiwen.org/user/collect.aspx',
                'email': ##,
                'pwd': ##,
                'code': code,
                'denglu': '登录'
            }

        # post 方法请求
        return scrapy.FormRequest(url=login_url,formdata=formdata,callback=self.login,meta={'cookiejar':response.meta['cookiejar']})

    # 验证登录结果
    def login(self,response):
        if response.url== 'https://so.gushiwen.org/user/collect.aspx':
                print('登陆成功')
        else:
                print('登入失败')
        print(response.text)

ps:本人是小白一枚，内容可能不太专业，有不严谨的地方请指点，谢谢呢~