Python网络数据采集7-单元测试与Selenium自动化测试

单元测试

Python中使用内置库unittest可完成单元测试。只要继承unittest.TestCase类，就可以实现下面的功能。

• 为每个单元测试的开始和结束提供setUp和tearDown函数。
• 提供不同类型的断言让测试成功或者失败
• 所有以test_打头的函数，都会当成单元测试来运行，他们彼此独立，互不影响。

下面来看一个简单的例子

import unittest


class TestSimple(unittest.TestCase):
    def setUp(self):
        print('set up')

    def test_simple(self):
        a = 2
        l = [2, 3, 43]
        self.assertIn(a, l)

    def tearDown(self):
        print('teardown')



if __name__ == '__main__':
    unittest.main(argv=['ignored', '-v'], exit=False)

test_simple (__main__.TestSimple) ... 

set up
teardown


ok

----------------------------------------------------------------------
Ran 1 test in 0.006s

OK

在Jupyter中，main()需要填入以上参数才能运行，参数列表中第一个参数会被忽略，而exit=False则不会kill掉kernel。详见stackoverflow

但是在Pycharm中运行则不会任何参数。

测试维基百科

将Python的单元测试和网络爬虫结合起来，就可以实现简单的网站前端功能测试。

import requests
from bs4 import BeautifulSoup
import unittest


class TestWiki(unittest.TestCase):
    soup = None

    def setUp(self):
        global soup
        r = requests.get('https://en.wikipedia.org/wiki/Monty_Python')
        soup = BeautifulSoup(r.text, 'lxml')

    def test_title(self):
        global soup
        title = soup.h1.string
        self.assertEqual(title, 'Monty Python')

    def test_content_exists(self):
        global soup
        content = soup.find('div', id='mw-content-text')
        self.assertIsNotNone(content)

        
if __name__ == '__main__':
    unittest.main(argv=['ignored', '-v'], exit=False)

test_simple (__main__.TestSimple) ... ok
test_content_exists (__main__.TestWiki) ... 

set up
teardown


D:\Anaconda3\lib\site-packages\bs4\builder\_lxml.py:250: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  self.parser.feed(markup)
ok
test_title (__main__.TestWiki) ... ok

----------------------------------------------------------------------
Ran 3 tests in 3.651s

OK

Selenium单元测试

如果使用selenium进行网站测试呢？（它的初衷就是用来干这个的）

from selenium import webdriver

driver = webdriver.PhantomJS(executable_path=r'C:\Program Files (x86)\phantomjs\bin\phantomjs.exe')
driver.get('https://en.wikipedia.org/wiki/Monty_Python')

print(driver.title)

assert 'Monty Python' in driver.title

Monty Python - Wikipedia

from selenium import webdriver


driver = webdriver.PhantomJS(executable_path=r'C:\Program Files (x86)\phantomjs\bin\phantomjs.exe')

driver.get('http://pythonscraping.com/pages/files/form.html')
# 找到输入框和提交按钮
first_name = driver.find_element_by_name('firstname')
last_name = driver.find_element_by_name('lastname')
submit = driver.find_element_by_id('submit')
# 输入
first_name.send_keys('admin')
last_name.send_keys('Sun')
# 提交
submit.click()
print(driver.find_element_by_tag_name('body').text)
driver.close()

Hello there, admin Sun!

或者使用动作链。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver import ActionChains

driver = webdriver.PhantomJS(executable_path=r'C:\Program Files (x86)\phantomjs\bin\phantomjs.exe')

driver.get('http://pythonscraping.com/pages/files/form.html')
# 找到输入框和提交按钮
first_name = driver.find_element_by_name('firstname')
last_name = driver.find_element_by_name('lastname')
submit = driver.find_element_by_id('submit')

actions = ActionChains(driver).send_keys_to_element(first_name, 'admin') \
    .send_keys_to_element(last_name, 'Sun') \
    .send_keys(Keys.ENTER)
# 执行动作链
actions.perform()
print(driver.find_element_by_tag_name('body').text)
driver.close()

Hello there, admin Sun!

除了简单的单击双击，发送文本到输入框，还能实现复杂的动作。比如拖放。

from selenium import webdriver
from selenium.webdriver import ActionChains


driver = webdriver.Firefox()

driver.get('http://pythonscraping.com/pages/javascript/draggableDemo.html')
print(driver.find_element_by_id('message').text)

element = driver.find_element_by_id('draggable')
target = driver.find_element_by_id('div2')
actions = ActionChains(driver).drag_and_drop(element, target)
actions.perform()
print(driver.find_element_by_id('message').text)
driver.close()

Prove you are not a bot, by dragging the square from the blue area to the red area!
You are definitely not a bot!

上面的代码用FireFox可以成功，Chrome和PhantomJs不成功，不知道为什么。先是打印了未拖拽方块时候显示的文字，当将方块拖到下面的div区域之后，页面上的文字变化了。

Selenium还有个有意思的功能--截屏。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver import ActionChains

driver = webdriver.PhantomJS(executable_path=r'C:\Program Files (x86)\phantomjs\bin\phantomjs.exe')
driver.get('https://www.pythonscraping.com/')

driver.get_screenshot_as_file('screenshot.png')

True

保存成功就会返回打印True。

Selenium与单元测试结合

还是面拖拽的例子，加入了Python单元测试与Selenium结合使用。

from selenium import webdriver
from selenium.webdriver import ActionChains
import unittest


class TestSelenium(unittest.TestCase):
    driver = None

    def setUp(self):
        global driver
        driver = webdriver.Firefox()
        driver.get('http://pythonscraping.com/pages/javascript/draggableDemo.html')

    def test_drag(self):
        global driver
        element = driver.find_element_by_id('draggable')
        target = driver.find_element_by_id('div2')
        actions = ActionChains(driver).drag_and_drop(element, target)
        actions.perform()

        self.assertEqual('You are definitely not a bot!', driver.find_element_by_id('message').text)


if __name__ == '__main__':
    unittest.main(argv=['ignored', '-v'], exit=False)

test_drag (__main__.TestSelenium) ... ok
test_simple (__main__.TestSimple) ... ok
test_content_exists (__main__.TestWiki) ... 

set up
teardown


D:\Anaconda3\lib\site-packages\bs4\builder\_lxml.py:250: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  self.parser.feed(markup)
ok
test_title (__main__.TestWiki) ... ok

----------------------------------------------------------------------
Ran 4 tests in 23.923s

OK

依然得使用FireFox, 以及main中加上参数argv=['ignored', '-v'], exit=False。看打印结果，上面的测试也执行了，好像在Jupyter中，虽然在一个Cell中执行单元测试，所有测试都会得到执行。普通Python代码就不是这样，每个Cell独立运行，还可以使用上面Cell定义过的变量和已经导入的库。

by @sunhaiyu

2017.7.19