流水数据写 ES 压测

压测工具 esrally

• 编写脚本，生成测试数据 JSONEachRow

esrally --track-path=~/track/10m_kd/ \
 --pipeline=benchmark-only \
 --target-hosts=10.111.3.19:9200,10.111.3.20:9200,10.111.3.21:9200 \
 --offline \
 --report-file=/tmp/report-10m.md

压测报告

报告名词解释

http://esrally.lyremelody.org/zh_CN/latest/summary_report.html

压测方法

• query-match-all-10k 一次查询 10k 条数据

10,000,000 压测一千万数据（2G）

Metric	Task	Value	Unit
Cumulative indexing time of primary shards		926.319	min
Min cumulative indexing time across primary shards		0	min
Median cumulative indexing time across primary shards		0	min
Max cumulative indexing time across primary shards		189.715	min
Cumulative indexing throttle time of primary shards		0	min
Min cumulative indexing throttle time across primary shards		0	min
Median cumulative indexing throttle time across primary shards		0	min
Max cumulative indexing throttle time across primary shards		0	min
Cumulative merge time of primary shards		444.689	min
Cumulative merge count of primary shards		449
Min cumulative merge time across primary shards		0	min
Median cumulative merge time across primary shards		0	min
Max cumulative merge time across primary shards		342.369	min
Cumulative merge throttle time of primary shards		48.5922	min
Min cumulative merge throttle time across primary shards		0	min
Median cumulative merge throttle time across primary shards		0	min
Max cumulative merge throttle time across primary shards		26.4601	min
Cumulative refresh time of primary shards		162.56	min
Cumulative refresh count of primary shards		7592
Min cumulative refresh time across primary shards		0	min
Median cumulative refresh time across primary shards		0	min
Max cumulative refresh time across primary shards		4.42518	min
Cumulative flush time of primary shards		36.3231	min
Cumulative flush count of primary shards		842
Min cumulative flush time across primary shards		0	min
Median cumulative flush time across primary shards		0	min
Max cumulative flush time across primary shards		12.215	min
Total Young Gen GC		29.81	s
Total Old Gen GC		0	s
Store size		600.827	GB
Translog size		3.4729e-05	GB
Heap used for segments		19.9	MB
Heap used for doc values		8.05996	MB
Heap used for terms		5.51184	MB
Heap used for norms		0.49292	MB
Heap used for points		0	MB
Heap used for stored fields		5.83527	MB
Segment count		6366
Min Throughput	bulk	32419.2	docs/s
Median Throughput	bulk	34948.8	docs/s
Max Throughput	bulk	35597	docs/s
50th percentile latency	bulk	544.618	ms
90th percentile latency	bulk	852.475	ms
99th percentile latency	bulk	6567.99	ms
100th percentile latency	bulk	10029.2	ms
50th percentile service time	bulk	544.618	ms
90th percentile service time	bulk	852.475	ms
99th percentile service time	bulk	6567.99	ms
100th percentile service time	bulk	10029.2	ms
error rate	bulk	0	%
Min Throughput	query-match-all	100.02	ops/s
Median Throughput	query-match-all	100.03	ops/s
Max Throughput	query-match-all	100.06	ops/s
50th percentile latency	query-match-all	3.23969	ms
90th percentile latency	query-match-all	4.42155	ms
99th percentile latency	query-match-all	18.7788	ms
99.9th percentile latency	query-match-all	50.5065	ms
100th percentile latency	query-match-all	61.5432	ms
50th percentile service time	query-match-all	3.13218	ms
90th percentile service time	query-match-all	4.19813	ms
99th percentile service time	query-match-all	14.7171	ms
99.9th percentile service time	query-match-all	43.7854	ms
100th percentile service time	query-match-all	61.4653	ms
error rate	query-match-all	0	%
Min Throughput	query-match-all-10k	4.83	ops/s
Median Throughput	query-match-all-10k	4.86	ops/s
Max Throughput	query-match-all-10k	4.86	ops/s
50th percentile latency	query-match-all-10k	880963	ms
90th percentile latency	query-match-all-10k	1.11775e+06	ms
99th percentile latency	query-match-all-10k	1.17266e+06	ms
99.9th percentile latency	query-match-all-10k	1.18024e+06	ms
100th percentile latency	query-match-all-10k	1.18133e+06	ms
50th percentile service time	query-match-all-10k	608.404	ms
90th percentile service time	query-match-all-10k	749.748	ms
99th percentile service time	query-match-all-10k	886.041	ms
99.9th percentile service time	query-match-all-10k	1013.36	ms
100th percentile service time	query-match-all-10k	1036.93	ms
error rate	query-match-all-10k	0	%

20,000,000 压测两千万数据（6G）

Metric	Task	Value	Unit
Cumulative indexing time of primary shards		942.175	min
Min cumulative indexing time across primary shards		0	min
Median cumulative indexing time across primary shards		0	min
Max cumulative indexing time across primary shards		189.715	min
Cumulative indexing throttle time of primary shards		0	min
Min cumulative indexing throttle time across primary shards		0	min
Median cumulative indexing throttle time across primary shards		0	min
Max cumulative indexing throttle time across primary shards		0	min
Cumulative merge time of primary shards		449.05	min
Cumulative merge count of primary shards		453
Min cumulative merge time across primary shards		0	min
Median cumulative merge time across primary shards		0	min
Max cumulative merge time across primary shards		342.369	min
Cumulative merge throttle time of primary shards		49.1307	min
Min cumulative merge throttle time across primary shards		0	min
Median cumulative merge throttle time across primary shards		0	min
Max cumulative merge throttle time across primary shards		26.4601	min
Cumulative refresh time of primary shards		164.416	min
Cumulative refresh count of primary shards		7604
Min cumulative refresh time across primary shards		0	min
Median cumulative refresh time across primary shards		0	min
Max cumulative refresh time across primary shards		4.42518	min
Cumulative flush time of primary shards		36.7088	min
Cumulative flush count of primary shards		848
Min cumulative flush time across primary shards		0	min
Median cumulative flush time across primary shards		0	min
Max cumulative flush time across primary shards		12.215	min
Total Young Gen GC		48.762	s
Total Old Gen GC		0	s
Store size		603.832	GB
Translog size		3.4729e-05	GB
Heap used for segments		20.0072	MB
Heap used for doc values		8.07825	MB
Heap used for terms		5.56839	MB
Heap used for norms		0.499146	MB
Heap used for points		0	MB
Heap used for stored fields		5.86143	MB
Segment count		6400
Min Throughput	bulk	34692.8	docs/s
Median Throughput	bulk	37088.2	docs/s
Max Throughput	bulk	40048.1	docs/s
50th percentile latency	bulk	604.131	ms
90th percentile latency	bulk	1000.26	ms
99th percentile latency	bulk	6933.49	ms
99.9th percentile latency	bulk	9321.29	ms
100th percentile latency	bulk	10969.2	ms
50th percentile service time	bulk	604.131	ms
90th percentile service time	bulk	1000.26	ms
99th percentile service time	bulk	6933.49	ms
99.9th percentile service time	bulk	9321.29	ms
100th percentile service time	bulk	10969.2	ms
error rate	bulk	0	%
Min Throughput	query-match-all	99.96	ops/s
Median Throughput	query-match-all	99.99	ops/s
Max Throughput	query-match-all	100	ops/s
50th percentile latency	query-match-all	18.0637	ms
90th percentile latency	query-match-all	22.6536	ms
99th percentile latency	query-match-all	25.7546	ms
99.9th percentile latency	query-match-all	52.2396	ms
100th percentile latency	query-match-all	80.3291	ms
50th percentile service time	query-match-all	12.5858	ms
90th percentile service time	query-match-all	13.1015	ms
99th percentile service time	query-match-all	14.5955	ms
99.9th percentile service time	query-match-all	36.1591	ms
100th percentile service time	query-match-all	80.0038	ms
error rate	query-match-all	0	%
Min Throughput	query-match-all-10k	4.8	ops/s
Median Throughput	query-match-all-10k	4.83	ops/s
Max Throughput	query-match-all-10k	4.84	ops/s
50th percentile latency	query-match-all-10k	885599	ms
90th percentile latency	query-match-all-10k	1.12163e+06	ms
99th percentile latency	query-match-all-10k	1.17445e+06	ms
99.9th percentile latency	query-match-all-10k	1.17983e+06	ms
100th percentile latency	query-match-all-10k	1.18078e+06	ms
50th percentile service time	query-match-all-10k	606.899	ms
90th percentile service time	query-match-all-10k	733.711	ms
99th percentile service time	query-match-all-10k	903.01	ms
99.9th percentile service time	query-match-all-10k	1068.78	ms
100th percentile service time	query-match-all-10k	1120.69	ms
error rate	query-match-all-10k	0	%

50,000,000 压测两千万数据（14.9gb）

Metric	Task	Value	Unit
Cumulative indexing time of primary shards		1004.26	min
Min cumulative indexing time across primary shards		0	min
Median cumulative indexing time across primary shards		0	min
Max cumulative indexing time across primary shards		189.715	min
Cumulative indexing throttle time of primary shards		0	min
Min cumulative indexing throttle time across primary shards		0	min
Median cumulative indexing throttle time across primary shards		0	min
Max cumulative indexing throttle time across primary shards		0	min
Cumulative merge time of primary shards		481.161	min
Cumulative merge count of primary shards		513
Min cumulative merge time across primary shards		0	min
Median cumulative merge time across primary shards		0	min
Max cumulative merge time across primary shards		342.369	min
Cumulative merge throttle time of primary shards		55.9592	min
Min cumulative merge throttle time across primary shards		0	min
Median cumulative merge throttle time across primary shards		0	min
Max cumulative merge throttle time across primary shards		26.4601	min
Cumulative refresh time of primary shards		170.947	min
Cumulative refresh count of primary shards		7850
Min cumulative refresh time across primary shards		0	min
Median cumulative refresh time across primary shards		0	min
Max cumulative refresh time across primary shards		4.42518	min
Cumulative flush time of primary shards		39.1434	min
Cumulative flush count of primary shards		869
Min cumulative flush time across primary shards		0	min
Median cumulative flush time across primary shards		0	min
Max cumulative flush time across primary shards		12.215	min
Total Young Gen GC		130.536	s
Total Old Gen GC		0	s
Store size		612.732	GB
Translog size		3.4729e-05	GB
Heap used for segments		20.0694	MB
Heap used for doc values		8.07607	MB
Heap used for terms		5.59167	MB
Heap used for norms		0.501709	MB
Heap used for points		0	MB
Heap used for stored fields		5.8999	MB
Segment count		6414
Min Throughput	bulk	37718.2	docs/s
Median Throughput	bulk	39696.5	docs/s
Max Throughput	bulk	47263.3	docs/s
50th percentile latency	bulk	1785.51	ms
90th percentile latency	bulk	3012.87	ms
99th percentile latency	bulk	8610.95	ms
99.9th percentile latency	bulk	13259.1	ms
100th percentile latency	bulk	14838.4	ms
50th percentile service time	bulk	1785.51	ms
90th percentile service time	bulk	3012.87	ms
99th percentile service time	bulk	8610.95	ms
99.9th percentile service time	bulk	13259.1	ms
100th percentile service time	bulk	14838.4	ms
error rate	bulk	0	%
Min Throughput	query-match-all	99.87	ops/s
Median Throughput	query-match-all	100.01	ops/s
Max Throughput	query-match-all	100.01	ops/s
50th percentile latency	query-match-all	3.62666	ms
90th percentile latency	query-match-all	4.67795	ms
99th percentile latency	query-match-all	90.2625	ms
99.9th percentile latency	query-match-all	267.873	ms
100th percentile latency	query-match-all	289.898	ms
50th percentile service time	query-match-all	3.37023	ms
90th percentile service time	query-match-all	4.41682	ms
99th percentile service time	query-match-all	17.8509	ms
99.9th percentile service time	query-match-all	230.215	ms
100th percentile service time	query-match-all	277.346	ms
error rate	query-match-all	0	%
Min Throughput	query-match-all-10k	4.86	ops/s
Median Throughput	query-match-all-10k	4.86	ops/s
Max Throughput	query-match-all-10k	4.87	ops/s
50th percentile latency	query-match-all-10k	880695	ms
90th percentile latency	query-match-all-10k	1.11436e+06	ms
99th percentile latency	query-match-all-10k	1.16724e+06	ms
99.9th percentile latency	query-match-all-10k	1.17362e+06	ms
100th percentile latency	query-match-all-10k	1.17412e+06	ms
50th percentile service time	query-match-all-10k	608.759	ms
90th percentile service time	query-match-all-10k	734.927	ms
99th percentile service time	query-match-all-10k	885.9	ms
99.9th percentile service time	query-match-all-10k	1006.49	ms
100th percentile service time	query-match-all-10k	1126.4	ms
error rate	query-match-all-10k	0	%

配置

track.json

{% import "rally.helpers" as rally with context %}

{
  "version": 2,
  "description": "StackOverflow Q&A stored as nested docs",
  "indices": [
    {
      "name": "sonested",
      "body": "index.json"
    }
  ],
  "corpora": [
    {
      "name": "nested",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 50000000,
          "uncompressed-bytes": 10590359571
        }
      ]
    }
  ],
  "challenges": [
    {
      "name": "2-index-and-query",
      "default": true,
      "schedule": [
        {
          "operation": {
            "operation-type": "delete-index"
          }
        },
        {
          "operation": {
            "operation-type": "create-index"
          }
        },
        {
          "operation": {
            "operation-type": "cluster-health",
            "request-params": {
              "wait_for_status": "green"
            }
          }
        },
        {
          "operation": {
            "operation-type": "bulk",
            "bulk-size": 10000
          },
          "warmup-time-period": 120,
          "clients": 8
        },
        {
          "operation": {
            "operation-type": "force-merge"
          }
        },
        {
          "operation": {
            "name": "query-match-all",
            "operation-type": "search",
            "body": {
              "query": {
                "match_all": {}
              }
            }
          },
          "clients": 8,
          "warmup-iterations": 1000,
          "iterations": 1000,
          "target-throughput": 100
        },
	{
          "operation": {
            "name": "query-match-all-10k",
            "operation-type": "search",
            "body": {
	      "size": 10000,
              "query": {
                "match_all": {}
              }
            }
          },
          "clients": 3,
          "warmup-iterations": 1000,
          "iterations": 1000,
          "target-throughput": 100
        }
      ]
    }
  ]
}

index.json

{
  "settings": {
    "index.number_of_shards": {{number_of_shards | default(5)}},
    "index.number_of_replicas": {{number_of_replicas | default(0)}},
    "index.store.type": "{{store_type | default('fs')}}",
    "index.requests.cache.enable": false
  },
  "mappings" : {
    "properties" : {
      "cFree" : {
        "type" : "long"
      },
      "cTotal" : {
        "type" : "long"
      },
      "direction" : {
        "type" : "long"
      },
      "free" : {
        "type" : "long"
      },
      "id" : {
        "type" : "text",
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "relId" : {
        "type" : "text",
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "time" : {
        "type" : "long"
      },
      "total" : {
        "type" : "long"
      },
      "type" : {
        "type" : "text",
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "user" : {
        "type" : "long"
      }
    }
  }
}

参考资料

1. Rally 官方文档