ES 性能测试

ES 性能测试

监控软件

marvel安装
kibana

测试环境

host 私网IP IP 配置 操作系统 ES角色
NM-WL-ES-01 10.37.30.2 100.66.137.152 4*8c,512G Mem,8T Disk CentOS7.2 MD
NM-WL-ES-02 10.37.30.3 100.66.137.153 4*8c,512G Mem,8T Disk CentOS7.2 MD
NM-WL-ES-03 10.37.30.4 100.66.137.154 4*8c,512G Mem,8T Disk CentOS7.2 MD
NM-WL-ES-04 10.37.30.5 100.66.137.155 4*8c,512G Mem,8T Disk CentOS7.2 DD
NM-WL-ES-05 10.37.30.6 100.66.137.156 4*8c,512G Mem,8T Disk CentOS7.2 DD
NM-WL-ES-06 10.37.30.7 100.66.137.157 4*8c,512G Mem,8T Disk CentOS7.2 DD

每台起两个ES实例, 3个Master, 9个Data

ES 入库效率

8进程, 连接的数据节点有6个
单条数据 888

{
    "T0": "2017-09-07 10:27:01.670",
    "T1": "43",
    "T10": "2",
    "T11": "460-11",
    "T12": "0x0003BD8B",
    "T13": "0x0000",
    "T14": "115.169.119.159",
    "T15": "TOPON.S11.GW0202-B-HW.NJ.JS.NODE.EPC.MNC011.MCC460.3GPPNETWORK.ORG",
    "T16": "460-11",
    "T17": "0x3203",
    "T18": "1",
    "T19": "0xD3C56A04",
    "T2": "64F011033201FFAB",
    "T20": null,
    "T21": "0xFFFF",
    "T22": "255",
    "T23": "0xFFFFFFFF",
    "T24": "0x00C1CECD",
    "T25": "2017-Sep-07 10:26:59",
    "T26": "460-11",
    "T27": "0x3B40",
    "T28": null,
    "T29": "0xFFFF",
    "T3": "0x13707FEF",
    "T30": null,
    "T31": "4294967295",
    "T32": "0xFFFFFFFF",
    "T33": "1970-Jan-01 07:59:59",
    "T34": null,
    "T35": "0xFFFF",
    "T36": "0x0000",
    "T37": "0x0000",
    "T38": "0x0000",
    "T39": "0",
    "T4": "0x95",
    "T40": "00",
    "T41": "0xFFFF",
    "T42": "0xFF",
    "T43": "0xFFFF",
    "T44": "0",
    "T45": "00",
    "T46": "0xFFFF",
    "T47": "0xFF",
    "T48": "0x00005FD8",
    "T49": "0x31",
    "T5": "460110171455497",
    "T50": null,
    "T51": "F070000018",
    "T6": "8618921771001",
    "T7": "3585880708120215",
    "T8": "0x1E",
    "T9": "10"
}

查看template

curl http://100.66.137.157:9201/_template?pretty
Paste_Image.png
time /slview/logpro/bin/EpcFile2ES_Invoke.pl -f 8 -t test_mmelog -v epc -s TaskWLJ  -c '/slview/test/liyr/mmelog/test_mmelog.yml' -lp 1

real 2m33.672s
echo "4072653/153" | bc
26618 /s

ES 索引占磁盘大小

CASE 1

"include_in_all" : false,
字段类型都为string,无分词
无正排索引

源文件大小 3.3G (json)
转为csv 1.77G

sed -i 's/\"//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i -E 's/T[0-9]+\://g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i 's/{//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i 's/}//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10

入库后 1.78G

约为 1:1

CASE 2

"include_in_all" : ture,
string 带分词
store true
doc_values true
入库时间 real 2m54.699s
2.84Gi

约为 1: 1.5

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

推荐阅读更多精彩内容

  • NAME dnsmasq - A lightweight DHCP and caching DNS server....
    ximitc阅读 2,906评论 0 0
  • 概述 监控预警平台, eagle + eye (鹰眼)的合体词, 寓意可以快速发现问题, 并及时作出响应,Eagl...
    Kungfu猫熊阅读 7,442评论 0 52
  • 打造轻量级OLAP(二):Hive + Elasticsearch - Treant - 博客园http://ww...
    葡萄喃喃呓语阅读 3,975评论 0 5
  • 今天是好沮丧的一天呢~ 越来越孤单了!走在人群中,明明是那么的热闹,可是我的心忽的就冷了下来! 阳光那么的灿烂,可...
    等待九月阅读 305评论 6 3
  • 早晨的雨让开在路上的我想起了外婆,或许是被反复提起的“母亲节”,记忆里那个最坚强的母亲。很多事经不起回忆,思绪还没...
    多多gxy阅读 284评论 0 0