Distributed load testing: Introduction (1/5)

One of the typical tasks required in all the projects when building server side components is testing and specifically load&performance testing.

The first recommendation is the usage of tools or libraries as much as possible.   Please, don't try to reinvent the wheel, there are plenty of libraries and tools for testing for every possible language and type of service.  If your project is simple enough you can probably make use of simple command line tools like apache ab and if it gets more complex there are some tools like jmeter or soapui or frameworks for testing that should fit your needs.  I have to admit my preference is not to use UI tools for testing and in general prefer simple scripts easy to hack, run and maintain.  One thing to remember is that most of those tools are specially suited for HTTP services and testing is usually more complicate for other types of connection oriented protocols.

The second recommendation is to use a solution supporting distributed execution of tests.    That will ensure that your are able to put as much load as needed in the servers under test.   Typically most of the solutions are based on a master node controlling slaves nodes where the master node schedule and submit jobs to the slaves and collect and present the results and stats received from them.

In this series of posts I will explain my approach for load testing of custom non-HTTP services using  python scripts and virtual machines.

What is AOP (Aspect Oriented Programming)?

For me AOP is a all about avoiding repetitive code to support common functionally by replacing that functionality with simple declarations associated to a method, class or module.

Think about it for a moment, I'm sure there are a lot of common tasks that you have to repeat in your projects: logging, authentication, caching, transactions, locking...     Those are commonly called cross-cutting concerns. In the worse case you will have lot of lines of code duplicated in all your methods for that functionality,  in the best case you will have that code in a library and you will have only one 1 line of code in all your methods.    But 1 line of code is still infinitely more than 0 lines of code!!!

If we were able to define those functions in a declarative way it would be great.   We need a way to tell the compiler "Add logging to this method", "All the responses in the methods of this class can be cached", "Make sure this method is executed in a transaction"... but hopefully without having to put that code in the middle of the real code of our application.  Business logic of our service should be easy to read and maintain and not obscured with tons of lines implementing plumbing functionality.

Most of the modern languages have a way to add some information attached to classes and methods.   For example we have Annotations in Java, Attributes in C# or Decorators in Python.

It is very easy to define in Java an annotation saying @Logging that can be attached to any method or class, the problem is how to associate some behavior to those annotations.    This is completely language/framework dependent but taking Java as an example, we have different options:
* Code modifications: A pre/post compiler task will modify the source code or byte code to introduce the code associated with each annotation.
* Dynamic proxies: When using IoC the container can instantiate a proxy instead of the real object and that proxy will execute the code associated with the annotations.
* Middlewares/Filters: In some frameworks (like jersey) you have the concept of filters that are executed for each request and you can add specific filters that can check the annotations in the class serving the request.

// Pending (I'm lazy) adding an example

I have been using AOP for years without any idea of this terminology, but just for completeness.... When we talk about AOP we usually mention Joint points and Advices.  Joint points are the points where the code will be added (for example before the method or after the method) and Advices the actual code that will be added.   An aspect is the combination of Joint points and Advice.

Optimistic Concurrency with REDIS

When using a database there are some situations when you retrieve some values, make some verifications or modifications and then make the update to the database.  This is a typical situation:

values = read_from_database()
if (values.x == A)
    values.x = B
    update_to_database(values)

The problem in this cases is how do you ensure that nobody changed the status from A to C between you read the database and you update it.

One possible solution is using locks in all your code.   This is an easy solution if you have a single instance of your service, but difficult to implement in distributed deployments (although doable even with REDIS [1]).

lock()
values = read_from_database()
if (values.x == A)
   values.x = B
   update_to_database(values)
unlock();

Using locks to solve this concurrency issue is what we call Pesimistic Concurrency, because you assume that you are going to have concurrent modifications and you protect always against them.

Other possible solution is to try to make the modification asking the update to the database to fail if the data changed since the time we read it.   This can be accomplished easily with version numbers or timestamps:

values = read_from_database()
version = values.version
if (values.x == A)
   values.x = B
   values.version ++
   update_to_database_only_if(values, version=version)

Using this strategy to solve this concurrency issue is what we call Optimistic Concurrency, because you assume that you are not going to have concurrent modifications most of the times and the cases it happens you just throw an error and the application should recover from it.

In REDIS we don't have to manually implement this logic and we can make use of WATCH [2] command [2].   Watch allows to mark a specific entry in the database (a key in REDIS) so that if that value changes before we commit the changes we are going to make, we will get an error and the changes will be discarded:

WATCH('customer:27')   // Future EXEC should fail if customer:27 changes
state = HGET('customer:27', 'state')
if state == 'available':
     MULTI()  // Initiate a transaction
     HSET('customer:27', 'state', 'busy')
     EXEC() // Execute the transaction, failing if customer:27 changed since the WATCH command
else:
     UNWATCH()

Very important: You have to manually UNWATCH if you don't make the EXEC or otherwise if the redis connection is reused the watch will be still in place for the next EXEC to be run in that connection.


[1] https://github.com/ServiceStack/ServiceStack.Redis/wiki/RedisLocks
[2] http://redis.io/commands/watch

Back to the Future

"Back to the Future" is what I'm feeling now regarding software development after moving to San Francisco.   It is not (only) because of the environment or the people but because I'm back to develop low level components in C++ after some years developing high level services on fancy languages.

I think the main reason behind the good feelings apart from the change itself is the challenging it is to develop in such environment.

Working on low level components makes you worried about a lot of technical issues that I didn't remember paying to much attention to.   I've been testing multiple FIFO queues implementations to improve the performance, profiling the application to detect contention on different synchronization strategies, testing different thread allocation and tasks assignment to those tasks, changing memory allocation...  You feel that the way you implement the software has a big impact in the way it performs and in addition you are continuously being surprised by the results of some of the tests and continuously willing to explore new things that you had never been done before.

Working with lower level programming languages like C++ makes you feel that you are under full control, everything that happens is your fault or is thanks to you.   If the application takes a long time to start is because of you and not because of the JVM, if there is a lot of contention is because of you and not because of the GIL, if the application takes a lot of memory is because of you and not because garbage collector....  Of course great power brings great responsibility, but that responsibility makes everything much more funny.

The only thing I'm still hating about the "new" development environment is having to compile.   If I could get rid of that I would be more than happy.... at least for some months/years :-)

Love to Code?

Just saw a facebook advertisemet with the title "Love to Code?" and couldn't resist the temptation to click on it.   It ended up being a door to facebook recruitement process and I decided to spend some minutes to try to get more information on how their process is.

After clicking the ad you are redirected to a very interesting tool from an external company called InterviewStreet who asked me to solve a simple programming problem in less than an hour in any programming language.

The problem was a slightly modified version of the game of life problem where the pieces at the boundaries never change and I decided to solve it in python. I was slightly disappointed because I was really expecting something more difficult (Tuenti contest problems are more difficult for example).

I was able to finish it in the 1h timeframe but didn't pay too much attention to write a great code so they won't probably call me :-)    Find my code below in case somebody wants to improve it.

#Enter your code here. Read input from STDIN. Print output to STDOUT

#For more details, please check http://www.interviewstreet.com/recruit/challenges/faq/view#stdio

def solve(board, size):

  result = [list(row) for row in board]

  pos = [(-1, -1),(-1, 0),(-1, 1),(0, -1),(0, 0),(0, 1),(1, -1),(1, 0),(1, 1)]

  for y in range(1, size-1):

    for x in range(1, size-1):

      blacks = whites = 0

      for dx,dy in pos:

        if 0 <= y+dy < size and 0 <= x+dx < size:

          if board[y+dy][x+dx] == 'b':

            blacks += 1

          else:

            whites += 1

      result[y][x] = ('b' if blacks > whites else 'w')

  return [''.join(row) for row in result]

tests = raw_input()

for test in range(int(tests)):

  #read input

  size, iters = raw_input().split(' ')

  board = [''] * int(size)

  for i in range(int(size)):

    board[i] = raw_input()

  #calculate

  for i in range(int(iters)):

    board = solve(board, int(size))

  #write result

  print "Board %d" % (test+1)

  for row in board:

    print row

Comparing YUI vs jQuery for simple tasks

Most of the people agree on the fact that using jQuery is not enough to build complex web apps.   When building one of those apps with jQuery you will probably end up including additional libraries and tools for logging, widgets, graphs, MVC infrastructure, history tracking, unit testing, minification... or moving to a different framework like dojo, YUI or ExtJS.

So, we can not create complex apps just with jQuery, but on the other side... Should we create simple apps with those bigger frameworks?  Are they really imposing a performance overhead and making much more complex to implement simple tasks?

To answer that question I tried to implement three typical use cases of simple apps with jQuery and YUI to compare the code produced.  Those typical use cases are 1) Attaching event handlers, 2) Modifying the DOM and 3) Making async HTTP requests.

1) Attaching event handlers (f.e. to catch a click event)

jQuery: $(selector).click(function() { });
YUI: Y.one(selector).on('click', function() { });

Conclusion: YUI slightly more verbose (but Y.one could be aliased as $ and the syntax is the same with both libraries in case of using event delegation). Result: tie

2) Modifying DOM (f.e. to show a div)

jQuery: $(selector).show();
YUI: Y.one(selector).show();

Conclusion: Identical (at least for this example).  Result: tie

3) Making async HTTP requests

jQuery: $.post(url, function() {} );
YUI: Y.io(url, { method: 'POST', { on: { success: function() { }} } )

Conclusion: YUI slightly more verbose.  I'm also missing deferreds support in YUI.  Result: jQuery wins

* Selectors are pretty similar and didn't find relevant differences in the support included in jQuery and YUI

Regarding the size of the library, some quick information without too much investigation (minified but without considering compression):

jQuery 1.7.1: 90KB
YUI min (limited but allows deferred loading of rest of yui components): 70KB
YUI min + dom + events + animations + io (equivalent to jquery?): 150KB

Disclaimer: I'm not an expert on YUI, comments and corrections are more welcomed

C++ MACRO Nightmare

This afternoon I was implementing a typical circular buffer including some lines like this to make the pointers go from the end of the buffer to the beginning again when needed.

_bufferReadPos = (_bufferReadPos + len) % BUFFER_SIZE;

When testing the code I started to see very weird values of the read and write pointers, in fact much bigger than expected.

After at least 10 minutes trying to debug the problem I realized that it was because of the expansion of the BUFFER_SIZE macro that was:

#define BUFFER_SIZE 32*1024

% Has precedence over * and my code was been interpreted as:

_bufferReadPos = ((_bufferReadPos + len) % 32) * 1024;

Very sad and not easy to catch.  My solution has been using parenthesis in the BUFFER_SIZE definition to avoid it being split again when being part of the previous line:

#define BUFFER_SIZE (32*1024)

Do you have similar nightmare stories with macros?!