问题发布功能的重点在于如何实现敏感词过滤,基本的算法是前缀树算法,前缀树也就是字典树,通过前缀树匹配可以加快敏感词匹配的速度
下面是具体的代码
- 前缀树的结构
其中包括有是否为结尾的标志符end,当在树中找到这个位置时,说明找到了敏感词,可以将其打码设为**,从后面的字符继续匹配
private class TrieNode {
private boolean end = false;
private Map<Character, TrieNode> map = new HashMap<Character, TrieNode>();
private Character c;
TrieNode() {
}
TrieNode(Character ch) {
c = ch;
}
public Character getch() {
return c;
}
public void setend(boolean a) {
this.end = a;
}
public boolean getend() {
return end;
}
public void addTrieNode(TrieNode tn) {
map.put(tn.getch(), tn);
}
public TrieNode getTrieNode(Character key) {
return map.get(key);
}
}
- 前缀树的生成(从txt文件中读取,每读到一个词语(一个词语一行),利用addword来添加)
注意:这里该类implements InitializingBean其中的一个函数——public void afterPropertiesSet() throws Exception,这个函数表示在实例化bean之后,即Bean属性注入之后,执行的操作,我们通过这个函数,在每次注入该服务时,能够初始化前缀树
@Override
public void afterPropertiesSet() throws Exception {
rootNode = new TrieNode();
try {
BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(new File("C:\\Users\\erika\\Videos\\offer\\askme\\src\\main\\resources\\SensitiveWords.txt")),
"UTF-8"));
String lineTxt = null;
while ((lineTxt = br.readLine()) != null) {
addword(lineTxt.trim());
System.out.println(lineTxt);
}
br.close();
} catch (Exception e) {
logger.error("读取文件失败" + e);
System.out.println("111");
}
}
- 向前缀树中添加词语(每个字占一个节点)
public void addword(String word) {
TrieNode tmp = rootNode;
for (int i = 0; i < word.length(); i++) {
if (isSymbol(word.charAt(i)))
continue;
if (tmp.getTrieNode(word.charAt(i)) != null) ;
else {
TrieNode tt = new TrieNode(word.charAt(i));
tmp.addTrieNode(tt);
}
tmp = tmp.getTrieNode(word.charAt(i));
if (i == word.length() - 1) {
tmp.setend(true);
}
}
}
- filter匹配字符串
public String filterword(String content) {
TrieNode root = rootNode;
TrieNode tmp = root;
StringBuilder result = new StringBuilder();
int head = 0;
int current = 0;
while (head < content.length()) {
Character c = content.charAt(head);
tmp = root;
if (isSymbol(c)) {
head++;
current++;
result.append(c);
continue;
}
boolean flag = false;
while (current < content.length() && tmp.getTrieNode(c) != null) {
if (tmp.getTrieNode(c).getend()) {
result.append("**");
flag = true;
break;
}
current++;
tmp = tmp.getTrieNode(c);
while(current<content.length() && isSymbol(content.charAt(current))) {
current++;
continue;
}
c = content.charAt(current);
}
if (flag) {
current++;
head = current;
} else {
result.append(content.charAt(head));
head++;
current = head;
}
}
return result.toString();
}
- 一个辅助函数,用来判断字符是否为空格,或者特殊字符
private boolean isSymbol(char c) {
int ic = (int) c;
Character.UnicodeBlock ub = Character.UnicodeBlock.of(c);
//判断是否为中文标点
if (ub == Character.UnicodeBlock.GENERAL_PUNCTUATION
|| ub == Character.UnicodeBlock.CJK_SYMBOLS_AND_PUNCTUATION
|| ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS
|| ub == Character.UnicodeBlock.CJK_COMPATIBILITY_FORMS
|| ub == Character.UnicodeBlock.VERTICAL_FORMS)
return true;
// 0x2E80-0x9FFF 东亚文字范围
return Character.isSpaceChar(c)||(!CharUtils.isAsciiAlphanumeric(c) && (ic < 0x2E80 || ic > 0x9FFF));
}