Elasticsearch集成SpringBoot(二)

接上篇，补充几个要点：

分词器
@Field-->FieldType
深度分页

分词器插件安装：

https://github.com/medcl/elasticsearch-analysis-ik/releases
下载ES对应的版本，解压到plugins文件中，注意：需要在plugins文件中创建一个ik的文件夹，在解压到ik文件夹中，否则重启ES会报错。

分词器介绍：

ik_smart （最少切分）
ik_max_word （最细粒度划分）

分词器测试：

ik_max_word分词器.png

ik_smart分词器.png

FieldType:

public enum FieldType {
    Auto, //
    Text, //
    Keyword, //不支持分词
    Long, //
    Integer, //
    Short, //
    Byte, //
    Double, //
    Float, //
    Half_Float, //
    Scaled_Float, //
    Date, //
    Date_Nanos, //
    Boolean, //
    Binary, //
    Integer_Range, //
    Float_Range, //
    Long_Range, //
    Double_Range, //
    Date_Range, //
    Ip_Range, //
    Object, //
    Nested, //
    Ip, //
    TokenCount, //
    Percolator, //
    Flattened, //
    Search_As_You_Type //
}

在ES插入的VO中可以不指定类型：框架会默认匹配字段类型，也可以声明式的指定字段类型：

    /**
     * orderId
     */
    @Field(name = "order_id",type = FieldType.Long)
    private Long orderId;

    /**
     * 订单编号
     */
    @Field(name = "order_sn",type = FieldType.Text)
    private String orderSn;

    /**
     * 收货人
     */
    @Field(name = "receiver_name",type = FieldType.Keyword)
    private String receiverName;

    /**
     * 省市区
     */
    @Field(name = "receiver_area_name",type = FieldType.Auto)
    private String receiverAreaName;

深度分页：

深度分页介绍：
官方推荐使用：PIT+search_after，但是PIT只支持ES7.10.X以上，我用的是elasticsearch-7.6.2，无法使用PIT，就直接基于search_after做深度分页。
深度分页有几个必要的条件：需要一个全局唯一的值做游标(一般可以用数据库主键)，下一次查询带上一次返回的游标；只能按顺序翻页，所以form标签必须要设置成0或者-1；(PS:如果没全局唯一字段，多字段唯一也是可以的)
Kibana演示search_after的基本使用
ES里面现在有10条数据，根据orderId倒序，orderId分别是1~10：

total.png

现在以2条为一页来分页，执行：

数量为2.png

拿到返回的游标：9，1654736359331，下次请求的时候带上这个游标，就能查到orderId=8~7 ，下一页6~5......：

search_after下一页.png
深度分页基于代码实现：
$\color{#dd0000}{ 这里我有必要用红色字体说明下我遇到的问题：}$
既然ES支持search_after标签搜索，现在只需要在SpringBootDataElasticsearch里面找到对应的API调用就行了，带着这个问题找到了SearchSourceBuilder,然后找到SearchSourceBuilder.searchAfter(Object[] values)，那么问题来了：SearchSourceBuilder怎么集成到ElasticsearchRestTemplate.search(Query query, Class<T> clazz)里面去，最终得出结论：SpringBootDataElasticsearch不支持：SearchSourceBuilder,只能通过ES客户端实现，下面是客户端代码实现：
AbstractSearchQueryEngine抽象类添加一个查询

public abstract class AbstractSearchQueryEngine<T,R> {

    @Autowired
    protected ElasticsearchRestTemplate elasticsearchRestTemplate;


    /**
     * from+size 分页查询
     * @param requestPara
     * @param clazz
     * @param pageable
     * @return
     */
    public abstract SearchHits<R> findPage(T requestPara, Class<R> clazz, Pageable pageable);


    /**
     * searchAfter分页查询
     * @param requestPara
     * @param searchSourceBuilder
     * @return
     */
    public abstract org.elasticsearch.search.SearchHits  findPage(T requestPara, SearchSourceBuilder searchSourceBuilder);

    //todo Scroll分页...

}

SimpleSearchQueryEngine实现类

    @Value("${spring.elasticsearch.rest.uris}")
    private List<String> nodes;

    @Override
    public org.elasticsearch.search.SearchHits findPage(Object requestPara, SearchSourceBuilder searchSourceBuilder) {
        String[] hosts = nodes.get(0).split(":");
        RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(new HttpHost(hosts[0],Integer.parseInt(hosts[1]),HttpHost.DEFAULT_SCHEME_NAME)));
        SearchRequest request = EsQueryParse.convertSearchSourceQuery(requestPara,searchSourceBuilder);
        request.source(searchSourceBuilder);
        try {
            SearchResponse response = client.search(request, RequestOptions.DEFAULT);
            return Optional.ofNullable(response).map(SearchResponse::getHits).orElseThrow(()->new IllegalStateException("ES服务器异常"));
        } catch (IOException e) {
            log.error("SimpleSearchQueryEngine.findPage:e{}",e);
            throw new IllegalArgumentException(e.getMessage());
        }
    }

ESQuery自定义注解转换方法

    public static <T> SearchRequest convertSearchSourceQuery(T t, SearchSourceBuilder searchSourceBuilder){
        SearchRequest request = new SearchRequest();
        try {
            BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
            Class<?> clazz = t.getClass();
            Field[] fields = clazz.getDeclaredFields();
            for (Field field : fields) {
                Object value = ClassUtils.getPublicMethod(clazz, "get" + captureName(field.getName())).invoke(t);
                if (value == null) {
                    continue;
                }
                if (field.isAnnotationPresent(EsLike.class)) {
                    WildcardQueryBuilder query = getLikeQuery(field, (String) value);
                    boolQueryBuilder.must(query);
                }
                if (field.isAnnotationPresent(EsEquals.class)) {
                    MatchQueryBuilder query = getEqualsQuery(field, value);
                    boolQueryBuilder.must(query);
                }
                if (field.isAnnotationPresent(EsRange.class)) {
                    RangeQueryBuilder rangeQueryBuilder = getRangeQuery(field, value);
                    boolQueryBuilder.must(rangeQueryBuilder);
                }
                if (field.isAnnotationPresent(EsIn.class)) {
                    TermsQueryBuilder query = getInQuery(field, (List<?>) value);
                    boolQueryBuilder.must(query);
                }
            }
            if (clazz.isAnnotationPresent(Document.class)){
                Document document = clazz.getAnnotation(Document.class);
                request.indices(document.indexName());
            }
            searchSourceBuilder.query(boolQueryBuilder);
        } catch (IllegalAccessException | InvocationTargetException | NoSuchMethodException e) {
            e.printStackTrace();
        }
        return request;
    }

配置文件

spring:
  main:
    allow-bean-definition-overriding: true
  flyway:
    locations: classpath:db/oms
  elasticsearch:
    rest:
      uris: 127.0.0.1:9200
  data:
    elasticsearch: #ElasticsearchProperties
      cluster-name: elasticsearch #默认即为elasticsearch
      cluster-nodes: 127.0.0.1:9300 #配置es节点信息，逗号分隔，如果没有指定，则启动ClientNode

Service

    /**
     * search_after分页
     * @param elasticsearchOrderQueryVo
     * @return
     */
    Page findOrderVoListBySearchAfter(ElasticsearchOrderQueryVo elasticsearchOrderQueryVo);

ServiceImpl

    @Override
    public Page findOrderVoListBySearchAfter(ElasticsearchOrderQueryVo elasticsearchOrderQueryVo) {
        SearchSourceBuilder searchSourceBuilder =  new SearchSourceBuilder();
        List<String> sortValues = elasticsearchOrderQueryVo.getSortValues();
        searchSourceBuilder.from(0); //from不用传值，默认是-1
        searchSourceBuilder.size(elasticsearchOrderQueryVo.getSize());
        searchSourceBuilder.sort("orderId").sort("createTime");//SortOrder.DESC
        if (CollectionUtils.isNotEmpty(sortValues)){
            searchSourceBuilder.searchAfter(sortValues.toArray());
        }
        org.elasticsearch.search.SearchHits searchHits = simpleSearchQueryEngine.findPage(elasticsearchOrderQueryVo,searchSourceBuilder);
        Page page = new Page(0,elasticsearchOrderQueryVo.getSize(),searchHits.getTotalHits().value);
        page.setRecords(Arrays.stream(searchHits.getHits()).collect(Collectors.toList()));
        return page;
    }

controller

    @PostMapping("/search/after")
    public CommonResp findOrderVoListBySearchAfter(@RequestBody ElasticsearchOrderQueryVo elasticsearchOrderQueryVo){
        return iomsOrderItemService.findOrderVoListBySearchAfter(elasticsearchOrderQueryVo);
    }

requestVO(为了动态拿到索引名称，所以这个请求VO我也加了@Document注解，可以在代码写死索引名称)


@Document(indexName="oms",shards=5,replicas=1,indexStoreType="fs",refreshInterval="-1")
@Data
public class ElasticsearchOrderQueryVo implements Serializable {

    /**
     * 分页
     */
    private Integer page;

    /**
     * 数量
     */
    private Integer size;

    /**
     * orderId
     */
    private Long orderId;

    /**
     * 订单编号
     */
    private String orderSn;

    /**
     * 收货人
     */
    @EsEquals
    private String receiverName;

    /**
     * 省市区
     */
    private String receiverAreaName;

    /**
     * 详细地址
     */
    private String receiverAddress;

    /**
     * 手机
     */
    private String receiverPhone;


    /**
     * 实际支付金额
     */
    @EsRange(lt = true)
    private Long payAmount;


    /**
     * 创建时间
     */
    private Date createTime;


    /**
     * 游标：用全局唯一字段就行，也可以多个，我用orderID+createTime组合定义一个游标
     */
    private List<String> sortValues;
}

postman请求调用
ES中总共有10条数据，现在就模拟一页3条数据来分页，还是以字段orderId和createTime做游标；

SearchAfter请求.png

响应结果(orderId=1~3)：

{
    "code": "200",
    "message": "请求成功",
    "body": {
        "records": [
            {
                "score": "NaN",
                "id": "754ee3cb-80b1-4c81-9f5e-077171878a16",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "fields": {},
                "highlightFields": {},
                "sortValues": [
                    1,
                    1654045147873
                ],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "oms",
                "clusterAlias": null,
                "sourceAsMap": {
                    "orderType": 1,
                    "freightAmount": 8,
                    "orderId": 1,
                    "orderSn": "Order_1",
                    "integrationAmount": 1,
                    "receiverName": "小明1",
                    "billReceiverEmail": "12456789@qq.com",
                    "receiverAddress": "四方精创资讯大厦1楼",
                    "couponAmount": 100,
                    "receiverPhone": "12012121221",
                    "payAmount": 200,
                    "createTime": 1654045147873,
                    "receiverAreaName": "广东省深圳市南山区",
                    "_class": "com.formssi.mall.order.domain.vo.ElasticsearchOrderVo",
                    "id": "754ee3cb-80b1-4c81-9f5e-077171878a16",
                    "billHeader": "深圳市 南山区 软基1层",
                    "status": 4
                },
                "innerHits": null,
                "sourceAsString": "{\"_class\":\"com.formssi.mall.order.domain.vo.ElasticsearchOrderVo\",\"id\":\"754ee3cb-80b1-4c81-9f5e-077171878a16\",\"orderId\":1,\"orderSn\":\"Order_1\",\"receiverName\":\"小明1\",\"receiverAreaName\":\"广东省深圳市南山区\",\"receiverAddress\":\"四方精创资讯大厦1楼\",\"receiverPhone\":\"12012121221\",\"payAmount\":200,\"freightAmount\":8,\"integrationAmount\":1,\"couponAmount\":100,\"createTime\":1654045147873,\"status\":4,\"billHeader\":\"深圳市 南山区 软基1层\",\"billReceiverEmail\":\"12456789@qq.com\",\"orderType\":1}",
                "sourceRef": {
                    "fragment": true
                },
                "rawSortValues": [],
                "fragment": false
            },
            {
                "score": "NaN",
                "id": "da923359-3e2b-44be-989a-66daf14a6c26",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "fields": {},
                "highlightFields": {},
                "sortValues": [
                    2,
                    1654131547874
                ],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "oms",
                "clusterAlias": null,
                "sourceAsMap": {
                    "orderType": 1,
                    "freightAmount": 8,
                    "orderId": 2,
                    "orderSn": "Order_2",
                    "integrationAmount": 1,
                    "receiverName": "小明2",
                    "billReceiverEmail": "12456789@qq.com",
                    "receiverAddress": "四方精创资讯大厦2楼",
                    "couponAmount": 100,
                    "receiverPhone": "12012121222",
                    "payAmount": 200,
                    "createTime": 1654131547874,
                    "receiverAreaName": "广东省深圳市南山区",
                    "_class": "com.formssi.mall.order.domain.vo.ElasticsearchOrderVo",
                    "id": "da923359-3e2b-44be-989a-66daf14a6c26",
                    "billHeader": "深圳市 南山区 软基2层",
                    "status": 4
                },
                "innerHits": null,
                "sourceAsString": "{\"_class\":\"com.formssi.mall.order.domain.vo.ElasticsearchOrderVo\",\"id\":\"da923359-3e2b-44be-989a-66daf14a6c26\",\"orderId\":2,\"orderSn\":\"Order_2\",\"receiverName\":\"小明2\",\"receiverAreaName\":\"广东省深圳市南山区\",\"receiverAddress\":\"四方精创资讯大厦2楼\",\"receiverPhone\":\"12012121222\",\"payAmount\":200,\"freightAmount\":8,\"integrationAmount\":1,\"couponAmount\":100,\"createTime\":1654131547874,\"status\":4,\"billHeader\":\"深圳市 南山区 软基2层\",\"billReceiverEmail\":\"12456789@qq.com\",\"orderType\":1}",
                "sourceRef": {
                    "fragment": true
                },
                "rawSortValues": [],
                "fragment": false
            },
            {
                "score": "NaN",
                "id": "6eccde33-4b3d-44a6-97aa-73438fbadb47",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "fields": {},
                "highlightFields": {},
                "sortValues": [
                    3,
                    1654217947874
                ],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "oms",
                "clusterAlias": null,
                "sourceAsMap": {
                    "orderType": 1,
                    "freightAmount": 8,
                    "orderId": 3,
                    "orderSn": "Order_3",
                    "integrationAmount": 1,
                    "receiverName": "小明3",
                    "billReceiverEmail": "12456789@qq.com",
                    "receiverAddress": "四方精创资讯大厦3楼",
                    "couponAmount": 100,
                    "receiverPhone": "12012121223",
                    "payAmount": 200,
                    "createTime": 1654217947874,
                    "receiverAreaName": "广东省深圳市南山区",
                    "_class": "com.formssi.mall.order.domain.vo.ElasticsearchOrderVo",
                    "id": "6eccde33-4b3d-44a6-97aa-73438fbadb47",
                    "billHeader": "深圳市 南山区 软基3层",
                    "status": 4
                },
                "innerHits": null,
                "sourceAsString": "{\"_class\":\"com.formssi.mall.order.domain.vo.ElasticsearchOrderVo\",\"id\":\"6eccde33-4b3d-44a6-97aa-73438fbadb47\",\"orderId\":3,\"orderSn\":\"Order_3\",\"receiverName\":\"小明3\",\"receiverAreaName\":\"广东省深圳市南山区\",\"receiverAddress\":\"四方精创资讯大厦3楼\",\"receiverPhone\":\"12012121223\",\"payAmount\":200,\"freightAmount\":8,\"integrationAmount\":1,\"couponAmount\":100,\"createTime\":1654217947874,\"status\":4,\"billHeader\":\"深圳市 南山区 软基3层\",\"billReceiverEmail\":\"12456789@qq.com\",\"orderType\":1}",
                "sourceRef": {
                    "fragment": true
                },
                "rawSortValues": [],
                "fragment": false
            }
        ],
        "total": 10,
        "size": 3,
        "current": 1,
        "orders": [],
        "optimizeCountSql": true,
        "searchCount": true,
        "countId": null,
        "maxLimit": null,
        "pages": 4
    },
    "ok": true
}

带上orderId=3的sortValues做游标参数，查询下一页

SerchAfter第二页.png

第二页响应结果(orderId=4~6)：

{
    "code": "200",
    "message": "请求成功",
    "body": {
        "records": [
            {
                "score": "NaN",
                "id": "a47e3303-fe35-4164-b1b0-3f14809935dc",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "fields": {},
                "highlightFields": {},
                "sortValues": [
                    4,
                    1654304347874
                ],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "oms",
                "clusterAlias": null,
                "sourceAsMap": {
                    "orderType": 1,
                    "freightAmount": 8,
                    "orderId": 4,
                    "orderSn": "Order_4",
                    "integrationAmount": 1,
                    "receiverName": "小明4",
                    "billReceiverEmail": "12456789@qq.com",
                    "receiverAddress": "四方精创资讯大厦4楼",
                    "couponAmount": 100,
                    "receiverPhone": "12012121224",
                    "payAmount": 200,
                    "createTime": 1654304347874,
                    "receiverAreaName": "广东省深圳市南山区",
                    "_class": "com.formssi.mall.order.domain.vo.ElasticsearchOrderVo",
                    "id": "a47e3303-fe35-4164-b1b0-3f14809935dc",
                    "billHeader": "深圳市 南山区 软基4层",
                    "status": 4
                },
                "innerHits": null,
                "sourceAsString": "{\"_class\":\"com.formssi.mall.order.domain.vo.ElasticsearchOrderVo\",\"id\":\"a47e3303-fe35-4164-b1b0-3f14809935dc\",\"orderId\":4,\"orderSn\":\"Order_4\",\"receiverName\":\"小明4\",\"receiverAreaName\":\"广东省深圳市南山区\",\"receiverAddress\":\"四方精创资讯大厦4楼\",\"receiverPhone\":\"12012121224\",\"payAmount\":200,\"freightAmount\":8,\"integrationAmount\":1,\"couponAmount\":100,\"createTime\":1654304347874,\"status\":4,\"billHeader\":\"深圳市 南山区 软基4层\",\"billReceiverEmail\":\"12456789@qq.com\",\"orderType\":1}",
                "sourceRef": {
                    "fragment": true
                },
                "rawSortValues": [],
                "fragment": false
            },
            {
                "score": "NaN",
                "id": "ed1e53f5-1ca2-45df-8a50-fdda58abe21b",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "fields": {},
                "highlightFields": {},
                "sortValues": [
                    5,
                    1654390747874
                ],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "oms",
                "clusterAlias": null,
                "sourceAsMap": {
                    "orderType": 1,
                    "freightAmount": 8,
                    "orderId": 5,
                    "orderSn": "Order_5",
                    "integrationAmount": 1,
                    "receiverName": "小明5",
                    "billReceiverEmail": "12456789@qq.com",
                    "receiverAddress": "四方精创资讯大厦5楼",
                    "couponAmount": 100,
                    "receiverPhone": "12012121225",
                    "payAmount": 200,
                    "createTime": 1654390747874,
                    "receiverAreaName": "广东省深圳市南山区",
                    "_class": "com.formssi.mall.order.domain.vo.ElasticsearchOrderVo",
                    "id": "ed1e53f5-1ca2-45df-8a50-fdda58abe21b",
                    "billHeader": "深圳市 南山区 软基5层",
                    "status": 4
                },
                "innerHits": null,
                "sourceAsString": "{\"_class\":\"com.formssi.mall.order.domain.vo.ElasticsearchOrderVo\",\"id\":\"ed1e53f5-1ca2-45df-8a50-fdda58abe21b\",\"orderId\":5,\"orderSn\":\"Order_5\",\"receiverName\":\"小明5\",\"receiverAreaName\":\"广东省深圳市南山区\",\"receiverAddress\":\"四方精创资讯大厦5楼\",\"receiverPhone\":\"12012121225\",\"payAmount\":200,\"freightAmount\":8,\"integrationAmount\":1,\"couponAmount\":100,\"createTime\":1654390747874,\"status\":4,\"billHeader\":\"深圳市 南山区 软基5层\",\"billReceiverEmail\":\"12456789@qq.com\",\"orderType\":1}",
                "sourceRef": {
                    "fragment": true
                },
                "rawSortValues": [],
                "fragment": false
            },
            {
                "score": "NaN",
                "id": "f0ea3da0-2c1d-41a7-b250-9fc0d9500baa",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "fields": {},
                "highlightFields": {},
                "sortValues": [
                    6,
                    1654477159331
                ],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "oms",
                "clusterAlias": null,
                "sourceAsMap": {
                    "orderType": 1,
                    "freightAmount": 8,
                    "orderId": 6,
                    "orderSn": "Order_6",
                    "integrationAmount": 1,
                    "receiverName": "大白6号",
                    "billReceiverEmail": "366666666@qq.com",
                    "receiverAddress": "壹方中心 -楼6楼",
                    "couponAmount": 100,
                    "receiverPhone": "13856569999",
                    "payAmount": 100,
                    "createTime": 1654477159331,
                    "receiverAreaName": "广东省深圳市福田区",
                    "_class": "com.formssi.mall.order.domain.vo.ElasticsearchOrderVo",
                    "id": "f0ea3da0-2c1d-41a7-b250-9fc0d9500baa",
                    "billHeader": "深圳市 福田区 中国建设银行",
                    "status": 4
                },
                "innerHits": null,
                "sourceAsString": "{\"_class\":\"com.formssi.mall.order.domain.vo.ElasticsearchOrderVo\",\"id\":\"f0ea3da0-2c1d-41a7-b250-9fc0d9500baa\",\"orderId\":6,\"orderSn\":\"Order_6\",\"receiverName\":\"大白6号\",\"receiverAreaName\":\"广东省深圳市福田区\",\"receiverAddress\":\"壹方中心 -楼6楼\",\"receiverPhone\":\"13856569999\",\"payAmount\":100,\"freightAmount\":8,\"integrationAmount\":1,\"couponAmount\":100,\"createTime\":1654477159331,\"status\":4,\"billHeader\":\"深圳市 福田区 中国建设银行\",\"billReceiverEmail\":\"366666666@qq.com\",\"orderType\":1}",
                "sourceRef": {
                    "fragment": true
                },
                "rawSortValues": [],
                "fragment": false
            }
        ],
        "total": 10,
        "size": 3,
        "current": 1,
        "orders": [],
        "optimizeCountSql": true,
        "searchCount": true,
        "countId": null,
        "maxLimit": null,
        "pages": 4
    },
    "ok": true
}

ES最简单的用法就这些，后续在慢慢深入学习！！！