做文本处理的同学在windows下工作可能经常会遇到带BOM的utf-8编码文件,这时需要对文件头BOM处理一下,不然会带来一些不好的影响。BOM字符的表示是codecs.BOM_UTF8。
首先要明确一点,在Python2中,codecs.BOM_UTF8是str类型,如果要与unicode进行对比的话需要转换一下。
下面展示Python2采用不同的读文件方式时处理BOM的代码:
with open("test.txt", "r") as fr:
for line in fr:
if line[:3] == codecs.BOM_UTF8:
line = line[3:]
print line
import codecs
with codecs.open("test.txt", "r", "utf-8") as fr:
for line in fr:
if line[0].encode("utf-8") == codecs.BOM_UTF8:
# 或使用
# if line[0] == codecs.BOM_UTF8.decode("utf-8"):
line = line[1:]
print line
python3中比较简单,只有一种方式,代码如下:
with open("test.txt", "r") as fr:
for line in fr:
if line[0] == codecs.BOM_UTF8.decode("utf-8"):
line = line[1:]
print line
import codecs
with codecs.open("test.txt", "r", "utf-8") as fr:
for line in fr:
# 代码与上面一样
if line[0] == codecs.BOM_UTF8.decode("utf-8"):
line = line[1:]
print line
最好养成这样写代码的好习惯,永不再出错。
得大神指点,有一种最简便的方式
指定编码方式为“utf_8_sig”即可解决删除文件头BOM
import codecs
with codecs.open("test.txt", "r", "utf_8_sig") as fr:
for line in fr:
print line