A light weight load generator for elasticsearch or generic HTTP servers.
10K+
You are advised to use the Elasticsearch-dedicated benchmark tool Loadgen to test the elasticsearch performance.
Highlights of Loadgen:
Download URL: http://release.infinilabs.com/loadgen/
Loadgen is easy to use. After the tool is downloaded and decompressed, two files are obtained: one executable program and one configuration file loadgen.yml. An example of the configuration file is as follows:
variables:
- name: ip
type: file
path: test/ip.txt
- name: user
type: file
path: test/user.txt
- name: id
type: sequence
- name: uuid
type: uuid
- name: now_local
type: now_local
- name: now_utc
type: now_utc
- name: now_unix
type: now_unix
requests:
- request:
method: GET
basic_auth:
username: elastic
password: pass
url: http://localhost:8000/medcl/_search
body: '{ "query": {"match": { "name": "$[[user]]" }}}'
In the above configuration, variables is used to define variable parameters and variables are identified by name. In a constructed request, $[[Variable name]] can be used to access the value of the variable. Supported variable types are as follows:
| Type | Description |
|---|---|
| file | External variable parameter of the file type |
| sequence | Variable of the auto incremental numeric type |
| range | Variable of the range numbers, support parameters from and to to define the range |
| uuid | Variable of the UUID character type |
| now_local | Current time and local time zone |
| now_utc | Current time and UTC time zone |
| now_unix | Current time and Unix timestamp |
Variable parameters of the file type are loaded from an external text file. One variable parameter occupies one line. When one variable of the file type is accessed, one variable value is taken randomly. An example of the variable format is as follows:
➜ loadgen git:(master) ✗ cat test/user.txt
medcl
elastic
The requests node is used to set requests to be executed by Loadgen in sequence. Loadgen supports fixed-parameter requests and requests constructed using template-based variable parameters. The following is an example of a common query request.
requests:
- request:
method: GET
basic_auth:
username: elastic
password: pass
url: http://localhost:8000/medcl/_search?q=name:$[[user]]
In the above query, Loadgen conducts queries based on the medcl index and executes one query based on the name field. The value of each request is from the random variable user.
Loadgen cyclically executes requests defined in the configuration file. By default, Loadgen runs for 5s and then automatically exits. If you want to prolong the running time or increase the concurrency, you can set the tool's startup parameters. The help commands are as follows:
➜ loadgen git:(master) ✗ ./bin/loadgen --help
Usage of ./bin/loadgen:
-c int
Number of concurrent threads (default 1)
-compress
Compress requests with gzip
-config string
the location of config file, default: loadgen.yml (default "loadgen.yml")
-d int
Duration of tests in seconds (default 5)
-debug
run in debug mode, loadgen will quit with panic error
-l int
Limit total requests (default -1)
-log string
the log level,options:trace,debug,info,warn,error (default "info")
-r int
Max requests per second (fixed QPS) (default -1)
-v version
Run Loadgen to perform the benchmark test as follows:
➜ loadgen git:(master) ✗ ./bin/loadgen -d 30 -c 100 -compress
__ ___ _ ___ ___ __ __
/ / /___\/_\ / \/ _ \ /__\/\ \ \
/ / // ///_\\ / /\ / /_\//_\ / \/ /
/ /__/ \_// _ \/ /_// /_\\//__/ /\ /
\____|___/\_/ \_/___,'\____/\__/\_\ \/
[LOADGEN] A http load generator and testing suit.
[LOADGEN] 1.0.0_SNAPSHOT, 83f2cb9, Sun Jul 4 13:52:42 2021 +0800, medcl, support single item in dict files
[07-19 16:15:00] [INF] [instance.go:24] workspace: data/loadgen/nodes/0
[07-19 16:15:00] [INF] [loader.go:312] warmup started
[07-19 16:15:00] [INF] [app.go:306] loadgen now started.
[07-19 16:15:00] [INF] [loader.go:316] [GET] http://localhost:8000/medcl/_search
[07-19 16:15:00] [INF] [loader.go:317] status: 200,<nil>,{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}
[07-19 16:15:00] [INF] [loader.go:316] [GET] http://localhost:8000/medcl/_search?q=name:medcl
[07-19 16:15:00] [INF] [loader.go:317] status: 200,<nil>,{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}
[07-19 16:15:01] [INF] [loader.go:316] [POST] http://localhost:8000/_bulk
[07-19 16:15:01] [INF] [loader.go:317] status: 200,<nil>,{"took":120,"errors":false,"items":[{"index":{"_index":"medcl-y4","_type":"doc","_id":"c3qj9123r0okahraiej0","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":5735852,"_primary_term":3,"status":201}}]}
[07-19 16:15:01] [INF] [loader.go:325] warmup finished
5253 requests in 32.756483336s, 524.61KB sent, 2.49MB received
[Loadgen Client Metrics]
Requests/sec: 175.10
Request Traffic/sec: 17.49KB
Total Transfer/sec: 102.34KB
Avg Req Time: 5.711022ms
Fastest Request: 440.448µs
Slowest Request: 3.624302658s
Number of Errors: 0
Number of Invalid: 0
Status 200: 5253
[Estimated Server Metrics]
Requests/sec: 160.37
Transfer/sec: 93.73KB
Avg Req Time: 623.576686ms
Loadgen executes all requests once to warm up before the formal benchmark test. If an error occurs, a prompt is displayed, asking you whether to continue. The warm-up request results are also output to the terminal. After execution, an execution summary is output.
The final results of Loadgen are the cumulative statistics after all requests are executed, and they may be inaccurate. You are advised to start the Kibana dashboard to check all operating indicators of Elasticsearch in real time.
It is very easy to use Loadgen to simulate bulk ingestion. Configure one index operation in the request body and then use the body_repeat_times parameter to randomly replicate several parameterized requests to complete the preparation of a batch of requests. See the following example.
- request:
method: POST
basic_auth:
username: test
password: testtest
url: http://localhost:8000/_bulk
body_repeat_times: 1000
body: "{ \"index\" : { \"_index\" : \"medcl-y4\",\"_type\":\"doc\", \"_id\" : \"$[[uuid]]\" } }\n{ \"id\" : \"$[[id]]\",\"field1\" : \"$[[user]]\",\"ip\" : \"$[[ip]]\",\"now_local\" : \"$[[now_local]]\",\"now_unix\" : \"$[[now_unix]]\" }\n"
You can use Loadgen and set the CLI parameter -r to restrict the number of requests that can be sent by the client per second, so as to evaluate the response time and load of Elasticsearch under fixed pressure. See the following example.
➜ loadgen git:(master) ✗ ./bin/loadgen -d 30 -c 100 -r 100
Note: The client throughput limit may not be accurate enough in the case of massive concurrencies.
You can set the -l parameter to control the total number of requests that can be sent by the client, so as to generate a fixed number of documents. Modify the configuration as follows:
requests:
- request:
method: POST
basic_auth:
username: test
password: testtest
url: http://localhost:8000/medcl-test/doc2/_bulk
body_repeat_times: 1
body: "{ \"index\" : { \"_index\" : \"medcl-test\", \"_id\" : \"$[[uuid]]\" } }\n{ \"id\" : \"$[[id]]\",\"field1\" : \"$[[user]]\",\"ip\" : \"$[[ip]]\" }\n"
Configured parameters use the content of only one document for each request. Then, the system executes Loadgen.
./bin/loadgen -config loadgen-gw.yml -d 600 -c 100 -l 50000
After execution, 50000 records are added for the Elasticsearch index medcl-test.
If the IDs of generated documents need to increase regularly to facilitate comparison, you can use the auto incremental IDs of the sequence type as the primary key and avoid using random numbers in the content. See the following example.
requests:
- request:
method: POST
basic_auth:
username: test
password: testtest
url: http://localhost:8000/medcl-test/doc2/_bulk
body_repeat_times: 1
body: "{ \"index\" : { \"_index\" : \"medcl-test\", \"_id\" : \"$[[id]]\" } }\n{ \"id\" : \"$[[id]]\" }\n"
In a request, we might want use the same variable value, such as the routing parameter to control the shard destination, also store the field in the JSON document.
You can use runtime_variables to set request-level variables, or runtime_body_line_variables to define request-body-level variables.
If the request body set body_repeat_times, each line will be different, as shown in the following example:
variables:
- name: id
type: sequence
- name: uuid
type: uuid
- name: now_local
type: now_local
- name: now_utc
type: now_utc
- name: now_unix
type: now_unix
- name: suffix
type: range
from: 10
to: 15
requests:
- request:
method: POST
runtime_variables:
batch_no: id
runtime_body_line_variables:
routing_no: uuid
basic_auth:
username: ingest
password: password
#url: http://localhost:8000/_search?q=$[[id]]
url: http://192.168.3.188:9206/_bulk
body_repeat_times: 10
body: "{ \"create\" : { \"_index\" : \"test-$[[suffix]]\",\"_type\":\"doc\", \"_id\" : \"$[[uuid]]\" , \"routing\" : \"$[[routing_no]]\" } }\n{ \"id\" : \"$[[uuid]]\",\"routing_no\" : \"$[[routing_no]]\",\"batch_number\" : \"$[[batch_no]]\", \"random_no\" : \"$[[suffix]]\",\"ip\" : \"$[[ip]]\",\"now_local\" : \"$[[now_local]]\",\"now_unix\" : \"$[[now_unix]]\" }\n"
We defined the batch_no variable to represent the same batch number in a batch of documents, and the routing_no variable to represent the routing value at each document level.
Content type
Image
Digest
sha256:25050740d…
Size
13.6 MB
Last updated
about 1 month ago
docker pull infinilabs/loadgen:1.31.0_NIGHTLY-20260813