Distributed Load Testing: Using Instant Servers for semi-automated slaves spawning (3/5)

As I mention in the first post of this series virtual machines provides a very dynamic and cheap way to create slave nodes for load testing.  There are different providers out there but I was interested on using Instant Servers (based on Joyent technology) because it is really simple to start with and it is fun to explore new solutions instead of using always the boring Amazon infrastructure.

The three small features I was interested on implementing with Instant Servers were:

• Simplify the creation of machines preconfigured to be used of locust slaves for my load testing
• Starting the slaves automatically on the machine startup so that the master detects them and is able to schedule jobs
• Stopping the machine automatically when it is not used for any test for some time to make sure we don't waste our money when forgetting to stop those unused machines

Creation of machines

To simplify the creation of machines I decided to build an instance with all the required packages and use it as template to clone the actual slave nodes.    To be able to find this instance later automatically I used Instant Server tags.  As far as I know unfortunately there is no UI for tagging in Instant Servers but you could use a script similar to mine [1].

This machine needs to have python, locust, the test file (locustfile.py), all the basic packages (make, gcc) and the packages required to start locust automatically (see next point).

Auto starting slaves

To make sure that the slaves are started during machine startup and that we start multiple instances in every box (because locust is single threaded and I want to use multicore machines) I used supervisord with the following (hopefully autoexplicative) configuration.   Tune the numprocs parameter depending on the machine you are using.

/etc/supervisor/conf.d/locust.conf

[program:locust]
command=locust -f /root/locustfile.py --master-host=81.45.23.221 --slave
stderr_logfile = /var/log/supervisord/locust-stderr.log
stdout_logfile = /var/log/supervisord/locust-stdout.log
process_name=%(program_name)s_%(process_num)02d
numprocs=4

Auto stopping machines

To monitor the usage of a machine I could be using the extensive Analytics API in instant servers but I decided to use the locust log file for simplicity.   I created a python script [2] monitoring log files for activity and if there is no activity for some minutes it invokes the stop Instant Servers API to shut down that instance.

Note: It was not possible to use the existing python SDK with Instant Servers and I had to fork it and made a small modification.  More details in next post.


PD. If I ever mention the word "cloud" in any of this posts, please feel free to insult me, I will deserve it.

[1] https://gist.github.com/ggarber/8381238
[2] https://gist.github.com/ggarber/8381263

Distributed load testing: Extending locus.io for connection oriented services (2/5)

I have to admit that when talking about testing scripts I'm a python fan because of the simplicity, low overhead and availability of libraries (Ruby could be a good option too).  So, the first think I did when started to build my distributed load testing environment was searching google for "python distributed load testing" and as usual google didn't dissapoint me and that was the beginning of fun with locust!

locust is python testing framework where you write your tests in python and the execution can be scheduled in one or multiple machines.    It has facilities for HTTP testing but can be extended to other services and provides a web interface to get easy access to the progress and results of the tests.

After digging for a while in the documentation (not that good) and the source code (nice code) I decided to give it a try and create my first non HTTP test.   When creating a Locust class (that class defines the configuration of the tests) you can specify a client to override the default HTTP client:

class MyLocust(Locust):
    task_set = MyTaskSet
    min_wait = 500
    max_wait = 500

    def __init__(self):
        super(MyLocust, self).__init__()

        self.client = MyProtocolLocustClient(host=self.host)

class MyProtocolLocustClient(object):
    def __init__(self, host):
        self.host = host
        self.connection = MyProtocolConnection(host)

    def ping(self):
        request_meta = {}
        request_meta["start_time"] = time.time()
        request_meta["method"] = 'MESSAGE'
        request_meta["name"] = 'PING'

        self.connection.send_message(self.connection.id(), "HI")

        try:
            response = self.connection.recv_message()

            if payload != "HI":
                raise Exception()

            request_meta["response_time"] = (time.time() - request_meta["start_time"]) * 1000

            events.request_success.fire(
                    method=request_meta["method"],
                    name=request_meta["name"],
                    response_time=request_meta["response_time"],
                    response_length=0
                )
        except Exception as e:
            events.request_failure.fire(
                    method=request_meta["method"],
                    name=request_meta["name"],
                    response_time=0,
                    exception=e,
                    response=None,
                )


That client object is available to all the locust TaskSets that you will define later with your specific tests.  For example:

class MyTaskSet(TaskSet):
    @task
    def my_task(l):
        l.client.ping()

And that's all, just put that code in a locustfile.py file and run it from the command line:
 locust -f locustfile.py -H xxxx.yyyy.com

And open a browser in http://localhost:8089 to start the test and get the results and you should get something like this:

Distributed load testing: Introduction (1/5)

One of the typical tasks required in all the projects when building server side components is testing and specifically load&performance testing.

The first recommendation is the usage of tools or libraries as much as possible.   Please, don't try to reinvent the wheel, there are plenty of libraries and tools for testing for every possible language and type of service.  If your project is simple enough you can probably make use of simple command line tools like apache ab and if it gets more complex there are some tools like jmeter or soapui or frameworks for testing that should fit your needs.  I have to admit my preference is not to use UI tools for testing and in general prefer simple scripts easy to hack, run and maintain.  One thing to remember is that most of those tools are specially suited for HTTP services and testing is usually more complicate for other types of connection oriented protocols.

The second recommendation is to use a solution supporting distributed execution of tests.    That will ensure that your are able to put as much load as needed in the servers under test.   Typically most of the solutions are based on a master node controlling slaves nodes where the master node schedule and submit jobs to the slaves and collect and present the results and stats received from them.

In this series of posts I will explain my approach for load testing of custom non-HTTP services using  python scripts and virtual machines.