离散特征
举例
离散特征 | 处理
- one-hot 编码，就是一维的编码，比如性别可以抽象成二维的向量，如果是男就是 (1, 0)，女就是(0, 1)；
- 如果离散的特征分布的特别广泛， 比如有 10 种分类的方法，one-hot 编码的向量就是十维，其落在哪个维度上面，其对应的维度就是 1，其他的都是 0；
连续特征
举例
- age 从 0 到 100 就是连续的特征；
- price_per_man 也是连续的特征；
- 连续的特征一般不会直接进模型；
连续特征 | 标准化 | 处理
- z-score 标准化(x-mean) / std
- 计算特征值，比如 price_per_man 的平均数（mean）和标准差（std）;
- 这样，就可以使 price_per_man 压缩到 0~1 之间；
- max-min 标准化 (x-min) / (max-min)；
- 这样也可以把 price_per_man 的值压缩在 0~1 之间；
连续特征 | 离散化 | 处理
- bucket 编码；
- 比如 age，比如 1~10 岁的定义为孩子，10~30 定义为青年，30~50 定义为中年，50 以上定义为老年；虽然 age 是离散特征，可以把它当做离散特征落在不同的 bucket 中，然后在基于 bucket 做 one-hot 的编码；
特征处理
featurevalue.csv
"用户id","年龄","性别","门店id","评分","人均价格","是否点击"
"1","22","M","315","4","193","0"
"1","16","F","431","3","193","1"
"1","62","F","489","3","72","1"
"1","12","M","398","0","216","1"
"1","76","M","307","3","131","0"
"1","54","M","490","1","205","0"
"1","38","M","308","2","227","1"
"1","56","M","400","3","82","1"
"1","65","F","426","0","136","0"
"2","48","F","328","3","64","1"
feature.csv
- 去掉 featurevalue.csv 中 userid，shopid 这些没有意义的字段，然后将其他内容做了映射；
- age 分成前 4 列；
- 性别分在 5, 6 列；
- 评分使用 max-min 标准化分在第 7 列；
- 人均价格使用 bucket 编码分布自 8 ～ 11 列；
- 是否点击落在最后一列；
"1","0","0","0","1","0","0.8","0","0","1","0","0"
"1","0","0","0","0","1","0.6","0","0","1","0","1"
"0","0","0","1","0","1","0.6","0","1","0","0","1"
"1","0","0","0","1","0","0.0","0","0","0","1","1"
"0","0","0","1","1","0","0.6","0","0","1","0","0"
"0","0","1","0","1","0","0.2","0","0","0","1","0"
"0","1","0","0","1","0","0.4","0","0","0","1","1"
"0","0","1","0","1","0","0.6","0","1","0","0","1"
"0","0","0","1","0","1","0.0","0","0","1","0","0"
"0","0","1","0","0","1","0.6","0","1","0","0","1"
LR 模型生成
LR 模型生成 | 步骤
- 用预处理过的特征值，训练生成模型；
- 生成完了评估一下模型；
LR 模型生成 | 代码
package tech.lixinlei.dianping.recommand;
import java.io.IOException;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.ml.classification.LogisticRegression;
import org.apache.spark.ml.classification.LogisticRegressionModel;
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator;
import org.apache.spark.ml.linalg.VectorUDT;
import org.apache.spark.ml.linalg.Vectors;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.Metadata;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
public class LRTrain {
public static void main(String[] args) throws IOException {
// 初始化spark运行环境
SparkSession spark = SparkSession.builder().master("local").appName("DianpingApp").getOrCreate();
// 加载特征及 label 训练文件
JavaRDD<String> csvFile = spark.read().textFile("file:///home/lixinlei/project/gitee/dianping/src/main/resources/feature.csv").toJavaRDD();
// 做转化
JavaRDD<Row> rowJavaRDD = csvFile.map(new Function<String, Row>() {
/**
*
* @param v1 feature.csv 中的一行数据；
* @return
* @throws Exception
*/
@Override
public Row call(String v1) throws Exception {
v1 = v1.replace("\"", "");
String[] strArr = v1.split(",");
return RowFactory.create(new Double(strArr[11]),
Vectors.dense(
Double.valueOf(strArr[0]),
Double.valueOf(strArr[1]),
Double.valueOf(strArr[2]),
Double.valueOf(strArr[3]),
Double.valueOf(strArr[4]),
Double.valueOf(strArr[5]),
Double.valueOf(strArr[6]),
Double.valueOf(strArr[7]),
Double.valueOf(strArr[8]),
Double.valueOf(strArr[9]),
Double.valueOf(10)));
}
});
// 定义列
StructType schema = new StructType(
new StructField[]{
new StructField("label", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("features",new VectorUDT(),false, Metadata.empty())
}
);
// data 只有两列，第一列 label，第二列是个 11 维的向量；
Dataset<Row> data = spark.createDataFrame(rowJavaRDD, schema);
// 训练集和测试集
Dataset<Row>[] dataArr = data.randomSplit(new double[]{0.8, 0.2});
Dataset<Row> trainData = dataArr[0];
Dataset<Row> testData = dataArr[1];
// 模型训练 | 逻辑回归
LogisticRegression lr = new LogisticRegression()
.setMaxIter(10) // 迭代次数
.setRegParam(0.3)
.setElasticNetParam(0.8)
.setFamily("multinomial");
LogisticRegressionModel lrModel = lr.fit(trainData);
lrModel.save("file:///home/lixinlei/project/gitee/dianping/src/main/resources/lrmode");
// 测试评估
Dataset<Row> predictions = lrModel.transform(testData);
MulticlassClassificationEvaluator evaluator = new MulticlassClassificationEvaluator();
double accuracy = evaluator.setMetricName("accuracy").evaluate(predictions);
System.out.println("auc = " + accuracy);
}
}
