pika URLParameters异常Name or service not known

[TOC]

情况描述

使用python pika包连接mq服务器时, 使用amqp协议用URLParameters连接方式时异常, 错误为pika.exceptions.AMQPConnectionError: [Errno -2] Name or service not known, 但使用ConnectionParameters连接方式则无错误.

amqp连接Url: amqp://app:3W!@4WUWo#5Jae^5@192.168.1.100:5672/app

连接配置

连接方式: selectConnection
连接参数: URLParameters("amqp://3W!@4WUWo#5Jae^5@192.168.1.100:5672/app")
服务器:

host = '192.168.1.100'
port = 5672
username = 'app'
password = '3W!@4WUWo#5Jae^5'
vhost = 'app'
queue = 'app.msg'

错误详情

connection OK
Traceback (most recent call last):
  File "consumer_test.py", line 40, in <module>
    consume.run()
  File "/app/python/treasure_box/mq/consumer.py", line 56, in run
    self.connect()
  File "/app/python/treasure_box/mq/client.py", line 135, in connect
    self.connection.ioloop.start()
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 209, in start
    self._poller.start()
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 496, in start
    self.process_timeouts()
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 366, in process_timeouts
    timer['callback']()
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/connection.py", line 1836, in _on_connect_timer
    error)
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/callback.py", line 60, in wrapper
    return function(*tuple(args), **kwargs)
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/callback.py", line 92, in wrapper
    return function(*args, **kwargs)
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/callback.py", line 236, in process
    callback(*args, **keywords)
  File "/root/.virtualenvs/py3/lib/python3.6/site-packages/pika/connection.py", line 1768, in _on_connection_error
    self.params.connection_attempts)
pika.exceptions.AMQPConnectionError: [Errno -2] Name or service not known

期望结果

使用amqp协议连接正常, 确定Name or service not known错误原因.

找错过程

name or service not known是一个关于域名与dns解析方面的错误, 一般是出现在域名对应的ip地址找不到或无法连接到对应错误服务器.
先确定/etc/hosts文件, hosts文件除了localhost 127.0.0.1的对应外没有其他域名解析.

# /etc/hosts
127.0.0.1       localhost
::1     localhost       ip6-localhost   ip6-loopback

查看/etc/resolv.conf, 均指向阿里云nameserver, 没有问题.

# /etc/resolv.conf
nameserver 100.100.2.136
nameserver 100.100.2.138

使用同样代码尝试连接其他MQ服务器. 两种方式均无异常, 证明代码无问题.
通过两个服务器同时使用tcpdump抓包, 确定没有从client连接到mq server服务. 根本没有建立连接.
印象中amqp url与http url解析方式相同. 查看pika模块源代码, 确认amqp url解析方式.

# pika URLParameters
class URLParameters(Parameters):
    __slots__ = ('_all_url_query_values',)
    _SETTER_PREFIX = '_set_url_'

    def __init__(self, url):
        super(URLParameters, self).__init__()


        self._all_url_query_values = None
        if url[0:4].lower() == 'amqp':
            url = 'http' + url[4:]
        parts = urlparse.urlparse(url)
        if parts.scheme == 'https':
            self.ssl = True
        elif parts.scheme == 'http':
            self.ssl = False
        elif parts.scheme:
            raise ValueError('Unexpected URL scheme %r; supported scheme '
                             'values: amqp, amqps' % (parts.scheme,))

        ...

        if parts.username is not None:
            self.credentials = pika_credentials.PlainCredentials(url_unquote(parts.username),
                                                                 url_unquote(parts.password))

        # Get the Virtual Host
        if len(parts.path) > 1:
            self.virtual_host = url_unquote(parts.path.split('/')[1])

        ...

从pika源码中观察, amqp在确认协议开头为amqp://后, 会转为http或https. 此时可以完全做为http来处理url
按http url来解析amqp://app:3W!@4WUWo#5Jae^5@192.168.1.100:5672/app ==> http://app:3W!@4WUWo#5Jae^5@192.168.1.100:5672/app

预期结果
- username: app
- password: 3W!@4WUWo#5Jae^5
- host: 192.168.1.100
- port: 5672
- vhost: app
解析结果
- username: app
- password: 3W!
- host: 4wuwo
- port: 5672
- vhost: /

明显password与host及vhost解析错误, 已找到问题, 现在就只差解决了.

问题症结

url在解析时使用

username:password@host:port/path?key1=value1&key2=value2#...

@前后分别为auth和netloc
第一个/前是netloc与path
在url中, 有些字符是做标识与特殊作用的.

解决方案

URL中绝对安全的字符:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
0123456789
_.-

其他字符如何在传输过程中会转义为%HEX格式, 此问题解决方案有两个.

帐号密码和host中尽量不包含特殊字符, 要使用以上安全字符.
如果条件允许, 使用ConnectionParameters方式连接, 该方式没有特殊字符限制.

总结

所谓, Bug再小也是虫. 没想到问题是出在url上. 此次问题虽然很低级, 但也浪费了不少的时间, 但问题排查过程给了自己一个思路及处理Name or service not known的经验, 也对URL解析有了各具象的认识.
依照本次经验举一反三, 在redis, mongo等url连接的方式中是否也存在同样的情况?