问题
实际应用中,使用 airflow 运行 BashOperator 操作时,由于业务日志包含中文导致 airflow task log 卡住,任务其实是正常运行,但是没办法 hook task status,问题原因是 python 编码的问题,work 日志中会出现编码错误,如下:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py", line 95, in _read_task_logs
self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u6ce8' in position 55: ordinal not in range(128)
解决办法
打开 base_task_runner.py 文件,
vi /usr/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py
将
self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
做如下修改
修改前:
def _read_task_logs(self, stream):
while True:
line = stream.readline().decode('utf-8')
if len(line) == 0:
break
self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
修改后
def _read_task_logs(self, stream):
while True:
line = stream.readline().decode('utf-8')
if len(line) == 0:
break
self.logger.info(u'Subtask: {}'.format(line.rstrip('\n')))