Scalability & Latency

average of 1.58ms and p99 of 4ms for 150 parallel clients sending 95,228 requests per second

MockServer is build to support massive scale from a single instance:

  • Apache Benchmark tested up to 6,000 parallel clients and shows MockServer has an average of 1.58ms and p99 of 4ms for 150 parallel clients sending 95,228 requests per second
  • Locust tested up to 3,000 parallel clients and shows MockServer has a p99 of 4ms for 150 parallel clients

Both performance testing frameworks show similar results up to 2,000 parallel clients at which point Locust reports warnings and the figures are no longer consistent with Apache Benchmark.

The following frameworks & techniques are used to maximise scalability:

  • Netty an asynchronous event-driven network application framework to maximise the scalability of HTTP and TLS
  • LMAX Disruptor a high performance inter-thread messaging library to maximise the scalability of recording events (i.e. state) and logging
  • ScheduledThreadPoolExecutor a thread pool that can scheduled delayed tasks is used to execute delay response to avoid blocking threads

Performance Tests

MockServer has been performance tested using Apache Benchmark and Locust with the following scenario:

  • four basic expectations, including method, path and headers
  • basic GET request matching third expectation (i.e. three matches are attempted for each request)

During the test MockServer was run on a Java 13 JVM, with the following command:

java -Xmx500m -Dmockserver.logLevel=WARN -Dmockserver.disableLogging=true -jar ~/.m2/repository/org/mock-server/mockserver-netty/5.12.0-SNAPSHOT/mockserver-netty-5.12.0-SNAPSHOT-shaded.jar -serverPort 1080

The following graph shows how the p99 increases as the number of parallel clients increase.

p99 as parallel clients increase  

Apache Bench Performance Test Results

Apache Benchmark was executed as follows:

ab -k -n 10000000 -c <parallel clients> http://127.0.0.1:1080/simple

The test results are:

parallel clients 50% 66% 75% 80% 90% 95% 98% 99% requests/s mean
10 0 0 0 0 0 0 0 0 77,122 0.13
50 1 1 1 1 1 1 1 1 85,765 0.58
100 1 1 1 1 1 2 2 3 92,846 1.08
150 1 2 2 2 2 2 3 4 95,228 1.58
250 3 3 3 3 4 5 7 8 86,470 2.89
500 6 6 6 6 7 7 8 9 83,209 6.01
750 9 9 10 10 11 11 12 15 75,554 9.93
1000 11 12 13 13 14 16 17 21 75,423 13.26
2000 24 24 25 26 27 29 31 35 82,191 24.33
3000 37 39 40 40 43 46 51 58 78,171 38.38
4000 52 55 57 59 64 70 82 91 73,552 54.38
5000 65 67 70 71 75 79 90 102 74,065 67.51
6000 80 84 88 90 97 104 122 137 70,432 85.19
 

Locust Performance Test Results

Apache Benchmark was executed as follows:

locust --loglevel=WARNING --headless --only-summary -u <parallel clients> -r 100 -t 180 --host=http://127.0.0.1:1080

The test results are:

parallel clients 50% 66% 75% 80% 90% 95% 98% 99% 99.90% 99.99% requests/s mean
10 0 0 1 1 1 1 1 2 5 5 11 0
50 0 0 1 1 1 1 1 2 3 5 50 0
100 0 1 1 1 1 2 2 3 4 8 100 0
150 1 1 1 1 2 3 3 4 5 6 149 0
250 2 3 3 4 5 6 7 8 15 46 245 2
500 2 2 3 3 4 5 6 7 9 46 479 2
750 3 4 5 6 8 10 12 14 29 34 699 3
1000 3 4 6 6 8 10 13 16 36 52 909 3
2000 4 7 10 12 22 34 49 59 87 110 1626.14 8
3000 51 78 99 110 160 180 220 240 290 310 2629.92 54

The following locustfile.py was used

import locust.stats
from locust import task, between
locust.stats.CONSOLE_STATS_INTERVAL_SEC = 60
from locust.contrib.fasthttp import FastHttpLocust

class UserBehavior(FastHttpUser):
    wait_time = between(1, 1)

    @task
    def request(self):
        self.client.get("/simple", verify=False)
 

Clustering MockServer

MockServer supports a very high request throughput, however if a higher request per second rate is required it is possible to cluster MockServer so that all nodes share expectations.

Although expectations are clustered, currently there is no support for clustering the MockServer log therefore request verifications will only work against the node that received the request.

To create a MockServer cluster all instances need to:

Each node could be configured as follows (adjusting the port as necessary):

MOCKSERVER_WATCH_INITIALIZATION_JSON=true \
MOCKSERVER_INITIALIZATION_JSON_PATH=mockserverInitialization.json \
MOCKSERVER_PERSIST_EXPECTATIONS=true \
MOCKSERVER_PERSISTED_EXPECTATIONS_PATH=mockserverInitialization.json \
java -jar ~/Downloads/mockserver-netty-5.12.0-shaded.jar -serverPort 1080 -logLevel INFO

or

java \
-Dmockserver.watchInitializationJson=true \
-Dmockserver.initializationJsonPath=mockserverInitialization.json \
-Dmockserver.persistExpectations=true \
-Dmockserver.persistedExpectationsPath=mockserverInitialization.json \
-jar ~/Downloads/mockserver-netty-5.12.0-shaded.jar -serverPort 1080 -logLevel INFO
 

Scalability Configuration:

Number of threads for main event loop

These threads are used for fast non-blocking activities such as:

  • reading and de-serialise all requests
  • serialising and writing control plane responses
  • adding, updating or removing expectations
  • verifying requests or request sequences
  • retrieving logs

Expectation actions are handled in a separate thread pool to ensure slow object or class callbacks and response / forward delays do not impact the main event loop.

Type: int Default: 5

Java Code:

ConfigurationProperties.nioEventLoopThreadCount(int count)

System Property:

-Dmockserver.nioEventLoopThreadCount=...

Environment Variable:

MOCKSERVER_NIO_EVENT_LOOP_THREAD_COUNT=...

Property File:

mockserver.nioEventLoopThreadCount=...

Example:

-Dmockserver.nioEventLoopThreadCount="5"

Number of threads for the action handler thread pool

These threads are used for handling actions such as:

  • serialising and writing expectation or proxied responses
  • handling response delays in a non-blocking way (i.e. using a scheduler)
  • executing class callbacks
  • handling method / closure callbacks (using web sockets)

Type: int Default: maximum of 5 or available processors count

Java Code:

ConfigurationProperties.actionHandlerThreadCount(int count)

System Property:

-Dmockserver.actionHandlerThreadCount=...

Environment Variable:

MOCKSERVER_ACTION_HANDLER_THREAD_COUNT=...

Property File:

mockserver.actionHandlerThreadCount=...

Example:

-Dmockserver.actionHandlerThreadCount="5"

Number of threads for each expectation with a method / closure callback (i.e. web socket client) in the org.mockserver.client.MockServerClient

This setting only effects the Java client and how requests each method / closure callbacks it can handle, the default is 5 which should be suitable except in extreme cases.

Type: int Default: 5

Java Code:

ConfigurationProperties.webSocketClientEventLoopThreadCount(int count)

System Property:

-Dmockserver.webSocketClientEventLoopThreadCount=...

Environment Variable:

MOCKSERVER_WEB_SOCKET_CLIENT_EVENT_LOOP_THREAD_COUNT=...

Property File:

mockserver.webSocketClientEventLoopThreadCount=...

Example:

-Dmockserver.webSocketClientEventLoopThreadCount="5"

Number of threads for client event loop when calling downstream

These threads are used for fast non-blocking activities such as, reading and de-serialise all requests and responses

Type: int Default: 5

Java Code:

ConfigurationProperties.clientNioEventLoopThreadCount(int count)

System Property:

-Dmockserver.clientNioEventLoopThreadCount=...

Environment Variable:

MOCKSERVER_CLIENT_NIO_EVENT_LOOP_THREAD_COUNT=...

Property File:

mockserver.clientNioEventLoopThreadCount=...

Example:

-Dmockserver.clientNioEventLoopThreadCount="5"

The the minimum level of logs to record in the event log and to output to system out (if system out log output is not disabled). The lower the log level the more log entries will be captured, particularly at TRACE level logging.

Type: string Default: INFO

Java Code:

ConfigurationProperties.logLevel(String level)

System Property:

-Dmockserver.logLevel=...

Environment Variable:

MOCKSERVER_LOG_LEVEL=...

Property File:

mockserver.logLevel

The log level, which can be TRACE, DEBUG, INFO, WARN, ERROR, OFF, FINEST, FINE, INFO, WARNING, SEVERE

Example:

-Dmockserver.logLevel="DEBUG"

Disable logging to the system output

Type: boolean Default: false

Java Code:

ConfigurationProperties.disableSystemOut(boolean disableSystemOut)

System Property:

-Dmockserver.disableSystemOut=...

Environment Variable:

MOCKSERVER_DISABLE_SYSTEM_OUT=...

Property File:

mockserver.disableSystemOut=...

Example:

-Dmockserver.disableSystemOut="true"

Disable all logging and processing of log events

Type: boolean Default: false

Java Code:

ConfigurationProperties.disableLogging(boolean disableLogging)

System Property:

-Dmockserver.disableLogging=...

Environment Variable:

MOCKSERVER_DISABLE_LOGGING=...

Property File:

mockserver.disableLogging=...

Example:

-Dmockserver.disableLogging="true"