Search APIs简介
REST request URI
轻便快速的URI查询方法
REST request body
可以有许多限制条件的json格式查询方法
- query: 在请求消息体中的query允许我们用Query DSL的方式查询。
- term: 查询时判断某个document是否包含某个具体的值，不会对被查询的值进行分词查询
- match: 将被查询值进行分词，然后用评分机制(TF/IDF)进行打分
- match_phrase: 查询指定段落
- Bool: 结合其他真值查询，通常和must、should、mustnot(与或非)一起组合出复杂的查询
- range: 查询时指定某个字段在某个特定的范围 （gte：>=；gt：>；lte：<=；le：<）
"range": {
"FIELD": {# 指定具体过滤的字段
"gte": 1,# gte: >=, gt: >
"lte": 10
}
}
- from: 以一定的偏移量来查看我们检索的结果，缺省从检索的第一条数据开始显示
- size: 指定检索结果中输出的数据条数，缺省为10条
- sort: 允许我们将检索的结果以指定的字段进行排序显示
- _source: 指定检索结果输出的字段
- script_fields: 该类型允许我们通过一个脚本来计算document中不存在的值，比如我们需要计算install/click得到cti之类的
"script_fields": {
"FIELD": {# 指定脚本计算之后值得名称
"script": {# 脚本内的运算
}
}
}
- aggs: 基于搜索查询，可以嵌套聚合来组合复杂的需求
"aggs": {
"NAME": {# 指定结果的名称
"AGG_TYPE": {# 指定具体的聚合方法，
TODO: # 聚合体内制定具体的聚合字段
}
}
TODO: # 该处可以嵌套聚合
}
查询
查询所有数据
GET logstash-zlead-web-v3.1.1-2019.03.14/_search
{
"query": {
"match_all": {}
}
}
output:
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 6344,
"max_score": 1,
"hits": [
{
"_index": "logstash-zlead-api-2019.03",
"_type": "logs",
"_id": "Z0pff2kB2B_5yQNjn_8T",
"_score": 1,
"_source": {
"callsite": "ZENSURE.EHandWare.Core.Log.Info",
"level": "INFO",
"@timestamp": "2019-03-15T03:22:44.756Z",
"tags": [
"api",
"beats_input_codec_json_applied"
],
"callsite_linenumber": "62",
"offset": 23030,
"node_name": "webApi",
"host": "iZ97uwvkgixopzZ",
"date": "2019-03-15 11:22:39.2948",
"@version": "1",
"beat": {
"name": "iZ97uwvkgixopzZ",
"hostname": "iZ97uwvkgixopzZ",
"version": "6.2.2"
},
"prospector": {
"type": "log"
},
"message": "验签",
"counter": "18"
}
}
]
}
}
返回数据含义
- took：表示该操作的耗时（单位为毫秒）
- timed_out：表示是否超时
- hits：表示命中的记录
- total：返回记录数，本例是2条。
- max_score：最高的匹配程度，本例是1.0。
- hits：返回的记录组成的数组。
- _score：表示匹配的程序，默认是按照这个字段降序排列。
查询数据总量
GET logstash-zlead-web-v3.1.1-2019.03.14/_count
{
"query": {
"match_all": {}
}
}
output
{
"count": 4483,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
}
}
根据ID查询
GET logstash-zlead-web-v3.1.1-2019.03.14/logs/iQ7CeWkBhhgPSpnGOCS9
output
{
"_index": "logstash-zlead-web-v3.1.1-2019.03.14",
"_type": "logs",
"_id": "iQ7CeWkBhhgPSpnGOCS9",
"_version": 1,
"found": true,
"_source": {
"date": "2019-03-14 09:12:35.4887",
"@timestamp": "2019-03-14T01:12:43.379Z",
"prospector": {
"type": "log"
},
"beat": {
"name": "iZ1128g9ft4c8qZ",
"hostname": "iZ1128g9ft4c8qZ",
"version": "6.2.2"
},
"level": "FATAL",
"exception": """
System.InvalidOperationException: 此请求已被阻止，因为当用在 GET 请求中时，会将敏感信息透漏给第三方网站。若要允许 GET 请求，请将 JsonRequestBehavior 设置为 AllowGet。
在 System.Web.Mvc.JsonResult.ExecuteResult(ControllerContext context)
在 System.Web.Mvc.ControllerActionInvoker.InvokeActionResult(ControllerContext controllerContext, ActionResult actionResult)
在 System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass29.<BeginInvokeAction>b__1b()
在 System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)
""",
"message": "此请求已被阻止，因为当用在 GET 请求中时，会将敏感信息透漏给第三方网站。若要允许 GET 请求，请将 JsonRequestBehavior 设置为 AllowGet。, 在 System.Web.Mvc.JsonResult.ExecuteResult(ControllerContext context) 在 System.Web.Mvc.ControllerActionInvoker.InvokeActionResult(ControllerContext controllerContext, ActionResult actionResult) 在 System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass29.<BeginInvokeAction>b__1b() 在 System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)",
"callsite_linenumber": "35",
"@version": "1",
"source": """E:\www.wujinyunshang.com\logs\2019-03-14\nlog-Fatal\2019-03-14.json""",
"tags": [
"web",
"beats_input_codec_json_applied"
],
"host": "iZ1128g9ft4c8qZ",
"callsite": "ZENSURE.EHandWare.WebFramework.ExpFilter.OnException",
"offset": 56059,
"counter": "10",
"node_name": "webAdmin"
}
}
简单条件搜索
GET logstash-zlead-web-v3.1.1-2019.03.14/_search
{
"query": {
"match": {
"level": "INFO"
}
},
"size": 1
}
output
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 3352,
"max_score": 0.34418565,
"hits": [
{
"_index": "logstash-zlead-web-v3.1.1-2019.03.14",
"_type": "logs",
"_id": "pQ7HeWkBhhgPSpnGBCTj",
"_score": 0.34418565,
"_source": {
"date": "2019-03-14 09:17:54.6293",
"@timestamp": "2019-03-14T01:17:58.787Z",
"prospector": {
"type": "log"
},
"beat": {
"name": "iZ1128g9ft4c8qZ",
"hostname": "iZ1128g9ft4c8qZ",
"version": "6.2.2"
},
"level": "INFO",
"message": "检测到Token的Cookie值",
"callsite_linenumber": "68",
"@version": "1",
"source": """E:\www.wujinyunshang.com\logs\2019-03-14\nlog-Info\2019-03-14.json""",
"tags": [
"web",
"beats_input_codec_json_applied"
],
"host": "iZ1128g9ft4c8qZ",
"callsite": "ZENSURE.EHandWare.WebFramework.BaseAgentsAdminController.get_CurrentAgentsManager",
"offset": 5962,
"counter": "26",
"node_name": "webAdmin"
}
}
]
}
}
更复杂的搜索
GET logstash-zlead-web-v3.1.1-2019.03.14/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"level": "INFO"
}
},
{
"match": {
"callsite_linenumber": 68
}
},
{
"match": {
"offset": 1014364
}
}
],
"filter": {
"range": {
"offset": {
"gte": 1014096
}
}
}
}
},
"sort": [
{
"offset": {
"order": "desc"
}
}
]
}
output
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": null,
"hits": [
{
"_index": "logstash-zlead-web-v3.1.1-2019.03.14",
"_type": "logs",
"_id": "eA4Qe2kBhhgPSpnG3Tce",
"_score": null,
"_source": {
"date": "2019-03-14 15:18:15.1137",
"@timestamp": "2019-03-14T07:18:15.488Z",
"prospector": {
"type": "log"
},
"beat": {
"name": "iZ1128g9ft4c8qZ",
"hostname": "iZ1128g9ft4c8qZ",
"version": "6.2.2"
},
"level": "INFO",
"message": "检测到Token的Cookie值",
"callsite_linenumber": "68",
"@version": "1",
"source": """E:\www.wujinyunshang.com\logs\2019-03-14\nlog-Info\2019-03-14.json""",
"tags": [
"web",
"beats_input_codec_json_applied"
],
"host": "iZ1128g9ft4c8qZ",
"callsite": "ZENSURE.EHandWare.WebFramework.BaseAgentsAdminController.get_CurrentAgentsManager",
"offset": 1014364,
"counter": "2053",
"node_name": "webAdmin"
},
"sort": [
1014364
]
}
]
}
}
注：filter的优先级在query中会高于must，即先过滤，后查询
短语搜索
GET logstash-zlead-web-v3.1.1-2019.03.14/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"message": "请求 并增加"
}
}
]
}
}
}
这个搜索会返回message中包含“请求”或者“并增加”的数据，也就是关键词之间默认是or的关系。如果希望精确匹配这个短语呢？
GET logstash-zlead-web-v3.1.1-2019.03.14/_search
{
"query": {
"bool": {
"must": [
{
"match_phrase": {
"message": "请求"
}
}
]
}
}
}
就是用match_phrase查询。
