Writing sequential test scripts with node

Today I was trying to create a node.js script to test a HTTP service but the test required multiple steps.   I gave it a try by using async module to "symplify" that code and that's the ugly code I came up with.

I'm not an expert in js/node, feel free to comment if I'm doing something wrong, I'm more than happy to learn.

(inflightSession and create are two helper functions that I have)

Test using node + jasmine: 

it("should accept valid sessionId", function(done) {
      async.waterfall([
          inflightSession,

          function(sessionId, callback) {
            create({ 'sessionId':sessionId }, callback);
          }
      ], function(error, response) {
          expect(response.statusCode).to.equal(200);
          done()
    });
  });

Same test using python + unittest:

def create_ok_test():
    session_id = inflight_session()
    response = create({ 'sessionId': session_id })

    assert_equals(200, response.status_code)

Same test using node ES6 generators (yield keyword):

it("should accept valid sessionId", function*() {
      var sessionId = yield inflightSession();
      var response = yield create({ 'sessionId': sessionId });
      expect(response.statusCode).to.equal(200);
  });

Honestly the code in python is way more readable than the existing node code, and still better even when comparing it with the new node generators.    Anway definitely looks like a promising way to move forward in the node community.   Some comments:

ES6 generators are available under a flag in node 0.11 and are supposed to be included in 0.12.

yield is a common keyword in other languages (i.e. python, C#) to exit from a function but keeping the state of that function so that you can resume the execution later.

function* is the syntax to define a generator function (a function using yield inside).

You need a runner supporting those generator functions (in this example jasmine needs to add support for it), basically calling the generator.next and waiting for the result (the result should be a promise or similar object) before calling generator.next again.

UPDATE: As I´m somehow forced to use node, I ended up creating a helper function and my tests are now like this

itx("should accept valid sessionId", inflightSession, function(sessionId, done) {
      create({ 'sessionId':sessionId }, function(error, response) {
          expect(response.statusCode).to.equal(200);
          done()
      });
});

Distributed Load Testing: Conclussions (5/5)

Let's recap what we have done in these series and try to get some conclusions.   The steps or achievements are these ones:

1. Find and test a distributed load testing tool in python: locust.
2. Extend locust for custom (non-HTTP) protocol testing.
3. Use Instant Servers to run the locust master and slaves.
4. Implement a simple way to autostop the machines when they are not being used based on the locust logs and instant servers stop API.
5. Create a template for the slaves to be easily cloned.   Use instant servers tags to define groups.
6. Fix the python Instant Server SDK and extend it with new authentication and clone features.
7. Extend locust interface adding a button to spawn machines in instant servers directly from the locust web interface.

Today I like even more python and the testing tools based in scripting instead of complex UIs.  This project gave me the oportunity to discover locust and Instant Servers and highly recommend people to use them for this kind of use case, it was very easy and a lot of fun using and combining those technologies.  Hopefully I can get more time for deeper integration of virtual machines in locust (with a good control UI and perhaps support for other providers).

Distributed Load Testing: py-smartdc and spawing slaves from locust (4/5)

After all the previous work I was able to spawn slaves easily from the Instant Servers interface (just clicking Clone) or with the command line tools but I wanted to go further and explore the extension capabilities of locust to add some very simple support to the locust web page to create the slaves.

How to clone and tag an Instant Servers machine with python

I have to recognize that my first instinct was to try to create a python Instant Servers SDK,  I even created the github repo and built the skeleton of the SDK, but 10 mins later somebody told me how stupid I was because there was already an official python SDK :(

Ok, that's great I though, but when I tried to use it I realize that it was not compatible with Instant Servers.   The "problem" is that the SDKs are maintained by joyent and even if Instant Servers is the same infrastructure, the API version is not exactly the same and the existing python SDK doesn't work with Telefonica infrastructure.

The solution was easy and I created my fork and pull requested a patch [1] that still hasn't been merged.   Feel free to use my fork! [2]  In addition I added another patch to support username&password authentication [3], and another one to add support for cloning machines

Once that's solved creating a machine by cloning the template instance we built in the previous post is very easy:

from smartdc import DataCenter, TELEFONICA_LOCATIONS

mad = DataCenter(location='eu-mad-1',
              known_locations=TELEFONICA_LOCATIONS,
              login='is0012', password='HNSnFAkc', api_version='6.5')

template_found = False
for machine in mad.machines():
        tags = machine.get_tags()

        if tags.get('locust') == 'slave-template':
                new_machine = machine.clone()
                new_machine.add_tags(locust='slave')
                print new_machine.name + ' ' + str(new_machine.get_tags())
                template_found = True

if not template_found:
        print 'slave-template instance not found'

How to integrate spawning machines in locust

Locust is easily extensible in the testing scripts as we saw in the previous post but also in the UI.   It is easy to add functionality to the website but modifying the template and adding more HTTP routes to process new API requests.  In my case I added a new button to spawn an Instant Servers machine and a route to process that request:

In locust/templates/index.html: 

                <div class="top_box box_stop box_running" id="box_reset">
                    <a href="/stats/reset">Reset Stats</a></br>
                    {% if is_distributed %}
                        <a href="/cloud/create">Spawn Slave</a>
                    {% endif %}
                </div>


In a new file locust/cloud.py:

from smartdc import DataCenter, TELEFONICA_LOCATIONS
from locust import web

mad = DataCenter(location='eu-mad-1',
              known_locations=TELEFONICA_LOCATIONS,
              login='', password='', api_version='6.5')

@web.app.route("/cloud/create")
def cloud_create():

    template_found = False
    for machine in mad.machines():
        tags = machine.get_tags()

        if tags.get('locust') == 'slave-template':
                new_machine = machine.clone()
                new_machine.add_tags(locust='slave')
                return new_machine

    return None

You can find my locust fork in [5], don't forget to change your login and password in the cloud.py file.

The result is this simple new button with the functionality required to spawn new machines automatically configured as locust slaves connected to the master for distributed testing.

[1] https://github.com/atl/py-smartdc/pull/9
[2] https://github.com/ggarber/py-smartdc/
[3] https://github.com/atl/py-smartdc/pull/10
[4] https://github.com/atl/py-smartdc/pull/11
[5] https://github.com/ggarber/locust

Distributed Load Testing: Using Instant Servers for semi-automated slaves spawning (3/5)

As I mention in the first post of this series virtual machines provides a very dynamic and cheap way to create slave nodes for load testing.  There are different providers out there but I was interested on using Instant Servers (based on Joyent technology) because it is really simple to start with and it is fun to explore new solutions instead of using always the boring Amazon infrastructure.

The three small features I was interested on implementing with Instant Servers were:

• Simplify the creation of machines preconfigured to be used of locust slaves for my load testing
• Starting the slaves automatically on the machine startup so that the master detects them and is able to schedule jobs
• Stopping the machine automatically when it is not used for any test for some time to make sure we don't waste our money when forgetting to stop those unused machines

Creation of machines

To simplify the creation of machines I decided to build an instance with all the required packages and use it as template to clone the actual slave nodes.    To be able to find this instance later automatically I used Instant Server tags.  As far as I know unfortunately there is no UI for tagging in Instant Servers but you could use a script similar to mine [1].

This machine needs to have python, locust, the test file (locustfile.py), all the basic packages (make, gcc) and the packages required to start locust automatically (see next point).

Auto starting slaves

To make sure that the slaves are started during machine startup and that we start multiple instances in every box (because locust is single threaded and I want to use multicore machines) I used supervisord with the following (hopefully autoexplicative) configuration.   Tune the numprocs parameter depending on the machine you are using.

/etc/supervisor/conf.d/locust.conf

[program:locust]
command=locust -f /root/locustfile.py --master-host=81.45.23.221 --slave
stderr_logfile = /var/log/supervisord/locust-stderr.log
stdout_logfile = /var/log/supervisord/locust-stdout.log
process_name=%(program_name)s_%(process_num)02d
numprocs=4

Auto stopping machines

To monitor the usage of a machine I could be using the extensive Analytics API in instant servers but I decided to use the locust log file for simplicity.   I created a python script [2] monitoring log files for activity and if there is no activity for some minutes it invokes the stop Instant Servers API to shut down that instance.

Note: It was not possible to use the existing python SDK with Instant Servers and I had to fork it and made a small modification.  More details in next post.


PD. If I ever mention the word "cloud" in any of this posts, please feel free to insult me, I will deserve it.

[1] https://gist.github.com/ggarber/8381238
[2] https://gist.github.com/ggarber/8381263