1. One-Hot
2. 词袋
Bag of Words（词袋表示），也称为Count Vectors，每个文档的字/词可以使用其出现次数来进行表示。
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
'This is the first document.',
'This document is the second document.',
'And this is the third one.',
'Is this the first document?',
]
vectorizer = CountVectorizer()
vectorizer.fit_transform(corpus).toarray()
Output：
array([[0, 1, 1, 1, 0, 0, 1, 0, 1],
[0, 2, 0, 1, 0, 1, 1, 0, 1],
[1, 0, 0, 1, 1, 0, 1, 1, 1],
[0, 1, 1, 1, 0, 0, 1, 0, 1]], dtype=int64)
3. N-gram
4. TF-IDF
由两部分组成：
第一部分是词语频率（Term Frequency），
第二部分是逆文档频率（Inverse Document Frequency）。
其中计算语料库中文档总数除以含有该词语的文档数量，然后再取对数就是逆文档频率。TF(t)= 该词语在当前文档出现的次数 / 当前文档中词语的总数
IDF(t)= log_e（文档总数 / 出现该词语的文档总数）
对比不同文本表示算法的精度，通过本地构建验证集计算F1得分
PlanA：Count Vectors + RidgeClassifier
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn.metrics import f1_score
train_df = pd.read_csv('/Users/summer/Desktop/xul_data/learning/DataWhale/20200719NLP/task01_preparing_20200719/input/train_set.csv', *sep*='t', *nrows*=15000)
vectorizer = CountVectorizer(*max_features*=3000)
train_test = vectorizer.fit_transform(train_df['text'])
# https://blog.csdn.net/LOLUN9/article/details/106012418/
# https://blog.csdn.net/fantacy10000/article/details/90647686
'''RidgeClassifier()通过Ridge()以下方式使用回归模型来创建分类器：
为了简单起见，让我们考虑二进制分类，目标变量等于+1或-1。
建立一个Ridge()回归模型（这是一个回归模型）来预测我们的目标变量。损失函数是RMSE + l2 penality
如果Ridge()回归的预测值（基于decision_function()函数计算）大于0，则将其预测为正类，否则为负类。
'''
# L2岭回归，压缩最优解的系数，计算效率高，模型稳定性好；L1减少项的个数
clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])
val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, *average*='macro'))
Output：
>>>> 0.65441877581244
PlanB：TF-IDF + RidgeClassifier
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn.metrics import f1_score
# train_df = pd.read_csv('../data/train_set.csv', sep='t', nrows=15000)
tfidf = TfidfVectorizer(*ngram_range*=(1,3), *max_features*=3000)
train_test = tfidf.fit_transform(train_df['text'])
clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])
val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, *average*='macro'))
Output：
>>> 0.8719372173702
本章作业
Q1：尝试改变TF-IDF的参数，并验证精度
A1：Tfidf Vectorizer
train_df_hw = pd.read_csv('/Users/summer/Desktop/xul_data/learning/DataWhale/20200719NLP/task01_preparing_20200719/input/train_set.csv', sep='\t', nrows=10000)
tfidf_hw = TfidfVectorizer(ngram_range=(1,5), max_features=3000)
train_hw_test = tfidf_hw.fit_transform(train_df_hw['text'])
clf = RidgeClassifier()
clf.fit(train_hw_test[:7000], train_df_hw['label'].values[:7000])
val_pred_hw = clf.predict(train_hw_test[7000:10000]) # [:N]表示从第一个开始取到第N个
print(f1_score(train_df_hw['label'].values[7000:10000], val_pred_hw, average='macro'))
Output：
# Test1-ngram_range=(1,3)
>>> 0.9317302315325816
# Test2-ngram_range=(1,5)
>>> 0.9326016109802603
# Test3-增加停用词stop_words='world'
>>> 0.9322304435086377
# Test4-norm='l2'
>>> 0.9322304435086377
# Test5-norm='l1'
>>> 0.5073894598279685
Q2：尝试使用其他机器学习模型，完成训练和验证
常用分类器 线性：LR、SVM 非线性：DF、RF、GBDT、XGBOOST
原理：https://www.cnblogs.com/andy-0212/p/10630608.html
对比：https://www.cnblogs.com/wkang/p/9657032.html
https://blog.csdn.net/twt520ly/article/details/79769705
https://www.jianshu.com/p/96173f2c2fb4
GBDT：https://blog.csdn.net/weixin_40924580/article/details/85043801?utm_medium=distribute.pc_relevant.none-task-blog-baidujs-2&spm=1001.2101.3001.4242
Test1：GBDT
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder as OHE
from xgboost.sklearn import XGBClassifier
train_df_hw = pd.read_csv('/Users/summer/Desktop/xul_data/learning/DataWhale/20200719NLP/task01_preparing_20200719/input/train_set.csv', sep='\t', nrows=10000)
tfidf_hw = TfidfVectorizer(ngram_range=(1,5), max_features=3000)
train_hw_test = tfidf_hw.fit_transform(train_df_hw['text'])
x_train = train_hw_test[:7000]
y_train = train_df_hw['label'].values[:7000]
x_test = train_hw_test[7000:10000]
y_test = train_df_hw['label'].values[7000:10000]
gbm1 = GradientBoostingClassifier(n_estimators=50, random_state=10, subsample=0.6, max_depth=4,
min_samples_split=400)
gbm1.fit(x_train, y_train)
gbm1_pred_hw = gbm1.predict(x_test)
print(f1_score(y_test, gbm1_pred_hw, average='macro'))
Output：
>>> 0.8165503231061779
Test2：TF-IDF+GBDT+LR，基于Test1
import numpy as np
## 特征转换
## model.apply(x_train)返回训练数据x_train在训练好的模型里每棵树中所处的叶子节点的位置（索引）
y_pred = gbm1.apply(x_train)
y_pred = y_pred.reshape(7000, -1) # 一个ID对应一个特征，训练集中有7000个ID，因此reshape(7000, -1)
## 打印上面结果的输出，可以看到shape是(7000, 50)，即训练数据量*树的棵树
print(np.array(y_pred).shape)
print(y_pred[0])
enc = OneHotEncoder()
enc.fit(y_pred)
y_pred2 = np.array(enc.transform(y_pred).toarray())
### 对测试集相同操作
y_pred_test = gbm1.apply(x_test)
y_pred_test = y_pred_test.reshape(3000, -1)
print(np.array(y_pred_test).shape) #(3000, 700)
print(y_pred_test[0])
y_pred_test2 = np.array(enc.transform(y_pred_test).toarray())
## 预测
LR = LogisticRegression(penalty='l2')
LR.fit(y_pred2, y_train)
lr_pred_hw = LR.predict(y_pred_test2)
print(f1_score(y_test, lr_pred_hw, average='macro'))
Output：
>>> 0.8515161019132009
通过LR提升3.5%
