Create automated testing tool for Magistrala

[felixgateru]

We should have a tool to test Magistrala functionality and system performance and metrics.

[felixgateru]

Load Testing Tools Evaluation

1. Background

The first part of this issue is to find a tool for load testing especially for our messaging protocols.

An existing benchmark for MQTT messaging exists here:
https://github.com/absmach/benchmark

This tool currently uses mzbench.

This tool is outdated and is currently not being actively being maintained. It uses the deprecated version of python , python 2.7 and elang 21. The tool includes libraries that are deprecated and need to be replaced so that we can run the tool with the latest versions of Python and Erlang. I have been able to bump the Python section from v2 to v3 but had an issue with the Erlang part but can use this as reference:
mzbench/mzbench#36

In that regard I looked into alternative testing tools:

2. Alternative Load-Testing Tools

The following are tools that can offer performance testing, some offer MQTT support and can be a direct alternative to mzbench. Others are more suited to REST and API performance testing.

For open source alternatives to using mzbench I have found:

JMeter

• Apache license
• Has a GUI to create test scenarios.
• It is written in Java.
• Mature and very popular project.
• All features available for free.
• Last update was this year

Gatling

• Apache license
• Can use Java or Javascript to write test scenarios.
• Written in Scala.
• GUI for test creation is closed source.
• Last update was within the week

Locust

• MIT license
• Uses python to write test scenarios.
• Written in Python.
• Has a bundled web UI in open source version.
• The documentation points out that Python may not be the fastest.
• Last update was 3 days ago.

Vegeta

• MIT license
• Not as feature and protocol rich as the others.
• Written in Go.
• CLI to invoke test.
• Last update was in October

Goose

• Apache license -
• Looks like the Rust version of locust as pointed out by their documentation.
• Scripting of test scenarios is in rust.
• Not as popular as Locust, Gatlng and Jmeter.
• Written in Rust.
• Last update was August.

Taurus

• Apache license
• Uses Jmeter, Gatling and Locust under the hood.
• Uses a DSL to write test scenarios.
• Last update was in September.

Our choice will depend on what we want to achieve, if it is load testing MQTT then I have seen that Gatling has native MQTT support while Jmeter does so using community plugins. If the task involves including general load testing then any of the above looks viable.