lunes, 24 de octubre de 2022

Using Node.js AsyncLocalContext to store tracing information

Usually when building Node.js services you want to be able to include in your logs some identifier (trace ID or user ID) that is shared for all the logs for the same request or event handler.   That way when you need to debug a failed request you can easily filter all the logs belonging to the same request.

This is very easy to implement in other execution environments thanks to the concept of thread-local variables where you can store some information for the duration of a task in a storage that is specific of the thread executing that task.   For example in Java you can make use of the ThreadLocal class to store information that then will be available for the rest of the execution of that thread. 

    private static ThreadLocal<RequestContext> context = new ThreadLocal<>(); context.set(new RequestContext(requestId));

This is not directly applicable to Node.js because all the tasks of a process are executed asynchronously in the same thread, but there is some similar functionality provided by the AsyncLocalContext cass.   In this case the context is not per thread but per call stack, so two flows of execution will have different stacks and different values for the variables stored.

Let's look at it with an example using the express nodejs framework.


First thing you want to do is assign a new context to each request and assign it to the current asyncLocalStorage.  This can be done with a express middleware to make it transparent for all the requests:

import { AsyncLocalStorage } from 'node:async_hooks';
const asyncLocalStorage = new AsyncLocalStorage();

app.use((_req, res, next) => {
const context = { traceId: crypto.randomUUID(), begin: Date.now() };
asyncLocalStorage.run(context, async () => {
next();
});
});

In this example we are associating a new UUID traceId and the begin time with every request.

Next you want your logger to add the information from the context when generating a log.   In this example we are using winston library for logging and a custom formatter to add the context data:

const addLoggingContext = winston.format(info => {
const context = asyncLocalStorage.getStore() as any;
return context ? { ...info, ...context, duration: Date.now() - context.begin } : info;
});

logger.add(new winston.transports.Console({
format: winston.format.combine(
addLoggingContext(),
winston.format.simple(),
),
}));

With that in place we can add some logs.    We can add some basic ones in the middleware and some in the request processing for example:

app.use((_req, res, next) => {
res.on('finish', () => {
logger.info('End request');
});
const context = { traceId: crypto.randomUUID(), begin: Date.now() };
asyncLocalStorage.run(context, async () => {
logger.info('Begin request');
next();
});
});

app.get('/', async (_req, res) => {
logger.info('Some interesting log');
res.sendStatus(200);
});

And if we make some requests now to our HTTP server we can see the expected output:

info: Begin request {"begin":1666640047930,"duration":0,"traceId":"ff3fcf6c-c7e5-4a70-a3a3-143bff9b2993"}
info: Some interesting log {"begin":1666640047930,"duration":1,"traceId":"ff3fcf6c-c7e5-4a70-a3a3-143bff9b2993"}
info: Begin request {"begin":1666640047933,"duration":0,"traceId":"4f2b4471-8d0f-481c-90bb-72bd33fe4ab2"}
info: Some interesting log {"begin":1666640047933,"duration":1,"traceId":"4f2b4471-8d0f-481c-90bb-72bd33fe4ab2"}
info: End request {"begin":1666640047930,"duration":1005,"traceId":"ff3fcf6c-c7e5-4a70-a3a3-143bff9b2993"}
info: End request {"begin":1666640047933,"duration":1004,"traceId":"4f2b4471-8d0f-481c-90bb-72bd33fe4ab2"}

So we have achieved what we wanted (to have a unique traceId shared for all the logs of each request) in a transparent way (using a express middleware) taking advantage of the node capabilities included in the node:async_hooks" package.

Note: You probably also want those tracing identifiers to be preserved when forwarding requests to other services but that's outside of the scope of this post.


lunes, 4 de junio de 2018

Video bubbles UI using Electronjs



After today's release of Houseparty's Mac app showing a new approach for video conversations UI based on bubbles I was wondering if it would be possible to build a similar user experience using Electron framework and web technologies.


It was easy to find that Electron has support for transparent and frameless windows so I decided to give it a try and figure out if it would be technically possible to build something similar to that.

To build the app I used the electron quick-start and only edited two files:

HTML File

First I modified the HTML file to add video capturing from the camera using the standard getUserMedia API and showed your local video stream in 3 <video> elements.  To make rounded bubbles you can use standard CSS attributes:


Note the style "webkit-app-region: drag" to make the window draggable from anywhere.

main.js File

You have to update the main.js file to create the main window as a frameless and transparent window:

const mainWindow = new BrowserWindow({ frame: false, transparent: true });

With those 2 tiny changes I was able to have something similar to the bubbles video experience created by Houseparty.



So I successfully built 0.01 % of the new Houseparty Mac app using Electron!  Enjoy it!




jueves, 3 de mayo de 2018

Two-level hashes in Redis using LUA and MsgPack

Redis hashes are a very powerful data structure allowing you to store key-value properties associated with a given Redis key.   For example you could store for each user all it's devices and the last time they were online:

UserId1
   DeviceId1=1525038228
   DeviceId2=1525038128

But at some point maybe you need to store something more than the last time each device was online.   Maybe a name, last time offline or a status (online/offline/away/busy...).

So basically we want to store nested hashes with a structure similar like this one:

UserId1
   DeviceId1
     LastOnline=1525038228
     LastOffline=1525028228
     Status=away
   DeviceId2=1525038228
     LastOnline=1525038128
     LastOffline=1525028128
     Status=busy

Unfortunately for us this is not a structure supported out of the box in Redis, so we need to flatten it a little bit and use something like Json values to store all that information:

UserId1
   DeviceId1={"LastOnline":1525038228, "LastOffline":1525028228, "Status": "away"}
   DeviceId2={"LastOnline": 1525038128, "LastOffline": 1525028128, "Status": "busy"}

With this approach the problem is solved, but when we need to update one of those values (f.e. LastOnline of DeviceId2) we need to do a HGET plus a HSET to redis.  This is problematic because it is:
  • Slower as it requires 2 round trip times to complete the operation
  • More complex as you need to use the WATCH command to run both commands simulating a transaction to avoid race conditions
  • Less efficient because you need to receive and send the whole Json value over the network
Fortunately there are two features of Redis that combined can give us something very similar to what we need.

The first feature is the ability to execute Lua scripts as part of a Redis command and the second feature are the standard Lua modules included in latest Redis versions allowing to serialize data as Json or MsgPack formats.

In this python example you can see the Lua scripts to write and read any property in these nested hashes:


The first Lua script updates a nested field.  To do that it gets the field value with HGET, deserialize it with 'cmsgpack.unpack', then update the field, serialize it again with 'cmsgpack.pack' and stores it back with HSET.  
The second Lua script returns all the nested fields.  To do that it gets the value with HGET, deserialize it with 'cmsgpack.unpack' and converts it to a list so that it can be sent in a redis response.

Disclaimer: I haven't used Lua much in the last 10y so that can probably be simpler/cleaner.

Size

MessagePack serialization is more compact than Json.  If you check the value stored in Redis after running the previous script you get this:

127.0.0.1:6379> hgetall user_id
1) "device1"
2) "\x81\xablast_online\xceZ\xeb\x0f\x13"

We should make it even smaller with shorter key names (f.e. "on" instead of "LastOnline").

Performance

I didn't have time to do a detailed performance test but just to check if something was terribly wrong I tried setting and getting one of those nested values 10.000 times in a loop against a local server and checked the time it took:
  Option 1: Lua/MessagePack:  2.15 secs
  Option 2: Use raw Redis commands storing nested hash as Json and using a transaction (GET + SET) to update a subfield: 2.42 secs


The same idea can be implemented with custom Redis modules instead of Lua scripts.   For example that is what the ReJSON module does. That is probably a little bit faster than the Lua script approach but there are many cases where you cannot install custom Redis modules (f.e. when using managed Redis instances in  AWS) 

miércoles, 22 de noviembre de 2017

Playing with Redis Geo structures

One of the things I've never used in Redis were the commands provided to store and access geolocated data.   From version 3.2 Redis includes 6 new very simple commands that can be used to store tags associated with coordinates and calculate distances between those tags or find the tags around some specific coordinates.

Let's try to build a simple prototype to see how it works.   Imagine that you have a service where users can rate anything (people, places, restaurants...)* and at some point you want to show in the UI of your app what things other people around you are rating right now. 

Almost every database right now has support to store this type of geographically located information but for use cases like this one where you want very fast access to ephemeral information an in-memory database can be a very good choice.

Storing data

The GEOADD command in Redis is the one you have to insert a new tag asociated with a specific position (latitude, longitude).

The structure supported in Redis for geolocated data has an insertion and query time that is O(log(N)) complexity where N is the number of items in the set, so probably you don't want to have all the data in the same set but partition it by country or some other grouping that makes sense for your use case.   In our example we could try partitioning it per city.

So everytime somebody rates something identified by a tag we will do this insert in redis:
GEOADD city latitude longitude tag
For example:
GEOADD sanfrancisco -122 37 goldengate
Redis stores this geographical information internally in a Sorted Set, so we can use any of the sorted set commands to manipulate or retrieve the list of items stored:
ZRANGE sanfrancisco 0 -1
1) "goldengate"

Retrieving data

There are two commands that you can use to make geographical queries on the stored data depending on your use case:


For our use case, everytime somebody opens the app we will retrieve all the tags around his position sorted by distance.

GEORADIUS sanfrancisco -122.2 37.1 5 km
1) "goldengate"

How does it work internally

As we mentioned before everything is stored inside Redis in the existing Sorted Set structures.

The way those zsets are leveraged are by using a score based on the latitude and longitude.  Basically by generating the zset scores interleaving the bits of the latitude and longitude of each entry you can later make queries to retrieve all the tags in a specific geographical square as a range of those scores.

That way with 9 ranges you can get all the areas around a specific point.  And those ranges can be of any size to be able to make queries using different radius just by trimming bits at the end of the score.

This technique is called geohashing and makes this geo commands very easy to implement on top of sorted sets.

Hope this is useful for other people implementing similar services, the truth is I never stop being amazed by Redis...

* Disclaimer: I built that service with some friends, you can see it in http://www.pleason.com/

lunes, 8 de mayo de 2017

How to (not) reuse code between Android and iOS

Most of the mobile applications we build these days have to work in two different platforms (Android and iOS).  Each of these platforms has its own frameworks, tools and programming languages so usually you end up building two completely separated applications and many times even built by separate teams.

[Note: If you are using some cross-platform development environment like react-native or Xamarin or building a Web/Hybrid app you are "lucky" and this post doesn't apply to you :)]

Unless you are working in a very simple app at some point you will realize that there are some parts of the application that you are implementing twice because you need it in both platforms (for example some business logic or the code to make requests to the HTTP server APIs).

Based on the capabilities of Android and iOS you have basically two options:
Option 1: Implement everything twice using the official language and libraries of each platform (f.e. implement the access to HTTP APIs using Swift and URLSession in the iOS app and using Java and Volley in the Android app)
Option 2: Implement the reusable code in C++ and compile it in the iOS app (creating a Objective C++ wrapper) and use it in the Android app (creating a JNI wrapper).

These are some possible advantages of Option 1:
  • Code is usually easier to read and maintain when written in modern languages (for example Swift vs C++).
  • Native integration: When using an Android library to make HTTP requests it will be probably integrated with the system proxy configuration and validates the SSL certificates with the system CAs by default.
  • No plumbing/boring code to write to provide access to the C++ library from the application (for example with JNI).  This can be partially mitigated using frameworks like SWIG to autogenerate the wrappers but it is still boring and usually problematic.
  • Simpler to debug because there is a single layer instead of having to make calls accross layers with different technologies(for example with JNI).
  • Build process faster and simpler because of less libraries/tools (for example no ndk required)
These are some possible advantages of Option 2:
  • No duplicated code to develop and maintain.
  • Avoid inconsistencies in naming, algorithms, protocols implementation because it is implemented in a single place.
  • Performance can be better.  Almost this is not an issue in most of the cases.
As we can see there are important pros and cons of both options so let's try another approach....  Let's check what are other popular mobile libraries doing?

I put some of those libraries in a diagram across two axis: Y for size/complexity of the library and X for number of platforms to support.  Other relevant variable could be how relevant is the performance optimisation but I don't want to make a 3D diagram :)  

In blue libraries using Option 1 and In green libraries using Option 2
[Apology: I picked some popular libraries I have used in the past and the lines of code and number of platforms is just an estimation, I didn't really count them]

As we can see most of the popular libraries are using Option 1 reimplementing the library twice, once for Android and once for iOS.  On the other side some big libraries related to real time communications or databases are using Option 2 implementing the core in C++ and exposing it with wrappers to Java and Objective-C applications.

Conclusion

What is the right solution probably depends on the type of project and the team building it but in my opinion in many (or most) of the cases it is less effort to develop and maintain 2 simple implementations than writing and maintaining a single more complex implementation plus the wrappers to different platforms.   In addition you can (should) mitigate the issues of Option 1 making use of tools to autogenerate code when possible, for example using protocol buffers/grpc for the client-server communication or swagger to generate clients for REST APIs.

I'm very interested on knowing your opinion on this topic, What do you think?   What are you doing right now in your projects?

miércoles, 19 de abril de 2017

Multiplatform Travis Projects (Android, iOS, Linux in the same build)

Using travis to build and test your code is usually a piece of cake and highly recommended but last week I tried to use travis for a non so conventional project and it ended up being more challenging than expected.

The project was a C library with Java and Swift wrappers and my goal was to generate Android, iOS and Linux versions of that library using Travis.   The main problem with my plan was that you have to define the "language" of project in your travis.yaml file and in my case... should it be android, objective-c or cpp project?

It would be great if travis would support multilanguage projects [1] or multiple yaml files per project [2] but apparently none of that is going to happen in the short term.

Linux
I decided to build the Linux part using docker to make sure I can use the same environment locally, in travis and in production.

iOS
Given the fact that the only way to build an iOS project is using OSX images and that there is no docker support in travis for OSX I had to use the multiple operating systems capabilities in travis [3].

Android
This ended up being the most challenging part.  Android projects require a lot of packages (tools, sdks, ndks, gradle...) so I decided to use docker also for this to make sure I had the same environment locally and in travis.    There were some docker images for this and I took many ideas form them, but I decided to generate my own [4].

To not have a too crazy travis.yaml file I put all the steps to install prerequirements and to launch the build process in shell scripts (2 scripts per platform).  That simplifies the travis configuration and also let me reuse the steps if I want to build locally or in jenkins eventually.   My project folder looks like this:

    /scripts/ios
       before_install.sh
       build.sh
    /scripts/android
       before_install.sh
       build.sh
    /scripts/linux
       before_install.sh
       build.sh

The most interesting scripts (if any) are the android and ios ones.

    #!/bin/bash
    echo "no additional requirements needed"

    #!/bin/bash
    xcodebuild build -workspace ./project.xcworkspace -scheme 'MyLibrary' -destination 'platform=iOS Simulator,name=iPhone 6,OS=10.3'

    #!/bin/bash
    docker pull ggarber/android-dev

    #!/bin/bash
    docker run --rm -it --volume=$(pwd):/opt/workspace --workdir=/opt/workspace/samples/android ggarber/android-dev gradle build


With that structure and those scripts the resulting travis.yaml file is very simple:

language: cpp

sudo: required
dist: xenial

os:
  - linux
  - osx

osx_image: xcode8.3

services:
  - docker

before_install:
  - if [[ "$TRAVIS_OS_NAME" != "osx" ]]; then ./scripts/linux/before_install.sh  ; fi
  - if [[ "$TRAVIS_OS_NAME" != "osx" ]]; then ./scripts/android/before_install.sh ; fi
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ./scripts/ios/before_install.sh     ; fi

script:
  - if [[ "$TRAVIS_OS_NAME" != "osx" ]]; then ./scripts/linux/script.sh  ; fi
  - if [[ "$TRAVIS_OS_NAME" != "osx" ]]; then ./scripts/android/script.sh ; fi
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ./scripts/ios/script.sh     ; fi

This is working fine although the build process is a little bit slow so these are some ideas to explore to try to improve it in the future:
  • Linux and Android builds could run in parallel.
  • Android docker images are very big (not only mine but all the ones I found).   According to docker hub it is 2GB compressed image.  Probably there are ways to strip this down.
  • I'm not caching the android packages being downloaded during the build process inside the docker container.

[1] https://github.com/travis-ci/travis-ci/issues/4090
[2] https://github.com/travis-ci/travis-ci/issues/3540
[3] https://docs.travis-ci.com/user/multi-os/
[4] https://github.com/ggarber/docker-android-dev

lunes, 6 de febrero de 2017

Using Kafka as the backbone for your microservices architecture

Disclaimer: I only use the word microservices here to get your attention.  Otherwise I would say your platform, your infrastructure or your services.

In many cases when your application and/or your team start growing the only way to maintain a fast development and deployment pace is to split the application and teams in different smaller units.   In case of teams/people that creates some interesting and not necessarily easier to solve challenges but this post is focused on the problems and complexity created in the software/architecture part.

When you split your solution in many components there are at least two problems to solve:
  • How to pass the information from one component to another (f.e. how do you notify all the sub-components when a user signs up so that you send him notifications, start billing him, generate recommendations...)
  • How to maintain the consistency of all the partially overlapped data stored in the different components (f.e. how do you remove all the user data from all the sub-components when the user decides to drop out from your service)

Inter component communication

At a very high level there are two communication models that are needed in most of the architectures:
  •  Synchronous request/response communications.  This has his own challenges and I recommend to use gRPC and some best practices around load balancing, service discovery, circuit breakers.... (find here my slides for TEFCON 2016) but it is usually a well understood model.
  • Asynchronous event based communications where a component generates an event and one or many components receive it and implement some logic in response to that event.
The elegant way to solve this second requirement is having in the middle a bus or a queue (depending on the reliability guarantees required for the use case) where producers send events and consumers can read those events from it.    There are many solutions to implement this pattern but when you have to handle heterogeneous consumers (that consume events at different rates or with different guarantees) or you have a massive amount of events or consumers the solution is not so obvious.

Data consistency

The biggest problem to solve in pure microservices architectures is probably how to ensure data consistency.   Once you split your application in different modules with data that is not completely independent (at the very least they all have the information about the same users) you have to figure out how to maintain that information in sync.

Obviously you have to try to maintain these dependencies and duplicated data as small as possible but usually at least you have to solve the problem of having the same users created in all of them.

To solve it you need a way to sync the data changes between different components that could be duplicated and need to be updated in other components.  So basically you need a way to replicate data that ensures the eventual consistency of it.

The Unified Log solution

If you look at those two problems they can be reduced to a single one: To have a real-time and reliable unified log that you can use to distribute events among different components with different needs and capabilities.   That's exactly the problem that LinkedIn had and what they built Kafka to solve.   The post "The Log: What every software engineer should know about real-time data's unifying abstraction" it is a very very recommended reading.

Kafka decouples the producers from the consumers including the ability to have slow consumers without affecting rest of the consumers.  Kafka does that and at the same time supports very high rates of events (it is common to have hundreds of thousands per second) with very low latencies (<20 msecs easily).  All these features while still being a very simple solution and providing some advanced features like organizing events in topics, preserving ordering of the events or handling consumer groups.

Those Kafka characteristics make it suitable to support most the inter-component communication use cases including events distribution, logs processing and data replication/synchronization.  All with a single simple solution by modeling all these communications as an infinite list of ordered events accessible for multiple consumers using a centralized unified log.

This post was about Kafka but all/most-of-it is equally applicable to the Amazon clone Kinesis. 

You can follow me in Twitter if you are interested in Software and Real Time Communications.

domingo, 15 de enero de 2017

Starting to love gRPC for interprocess communication (1/2)

In the context of a discussion around programming languages and static typing a colleague said that when you get older you stop caring about fancy technologies and you realize that is way better to just use safe and well probed solutions.  

I'm kind of tired of having been using loosely defined JSON-HTTP interfaces for many years and when I discovered gRPC last year it looked exactly what I was looking for.  I would love to start using it in production as soon as possible so I decided to play with it for a while first and explain how it went.

I will split my comments about gRPC in two posts. This first one about what is gRPC and what advantages provide and the next one on how to use it in our applications.

gRPC embraces the RPC paradigm where the APIs are defined as actions receiving some arguments and replying with a response.  Initially it feels like going 10y back when we started to use SOAP and similar technologies but we have to admit that is much simpler to map those primitives to our client and server code (for example no url path mapping) and it is more strict and explicit on what can and cannot be done for each operation and that usually makes the system more robust.

In gRPC you define your interfaces (methods, arguments and results) in an IDL using the protocol buffers format.   This definition is used to generate the server and client code automatically.   The serialization of the calls is done using the binary protobuf format too.  This makes the communication efficient and the protocol extensible being able to use all the features available in protobuf (for example composition or enum types).

Two of the advantages of this approach are automatic code generation and schema validation.  That can also be done in the "traditional" REST interfaces, but it is more tedious, less efficient and in my experience much easier to make mistakes when you add new features or refactor the code.

The communication in gRPC is based on HTTP2 transport.  This provides all the advantages of the new HTTP version (multiplexing, streaming, compression) while at the same time allows you to keep using existing HTTP infrastructure (nginx or other load balancers for example).

Another special feature of gRPC is the streaming support that is very convenient for some APIs these days.    With gRPC you are able to send a (potentially infinite) sequence of arguments to the server and receive a sequence of results from it.   That is very useful to implement applications more responsive where data can be processed and displayed even if part of it is still not ready.    It is also very useful for APIs based on notifications like in case of a chat application for example.

When compared with other IPC frameworks like Finagle (disclaimer, i'm a fan of it) gRPC is still missing important features client side load balancing (although it is wip) and some other goodies like circuit breakers, retries or service discovery.   In the mean time people is implementing those features on top of the framework.

The other missing piece is browsers support.  Even if there is support for many languages including Javascript, the browsers limitations make it not possible to implement a gRPC compatible web client nowadays.   The community is working on an extension of the protocol to support browsers and in the mean time the only solution seem to be the grpc-gateway proxy that generates a JSON-HTTP to gRPC proxy based on the IDL of the service with some extra annotations.

domingo, 6 de noviembre de 2016

Adding metrics/monitoring to the Mac menu bar

In the past I used to have an extra screen close to my desk where I was able to show different dashboards with metrics to monitor the health of our services.    Depending on the service we can use things like graphite, cloudwatch or google analytics.

These days I'm finding some challenges to keep using that approach so I decided to explore the option of showing those metrics in the menu bar of my Mac.

First thing I needed was an app allowing me to put custom stuff in the menu bar.  I explored a couple of options and I ended up using BitBar:
https://github.com/matryer/bitbar

BitBar is a free app that is able to execute almost any script (bash, python, ruby...) and put the output of it in the menu bar with many customizable options for icons, images and format.

Right now I wanted to monitor a couple of services using graphite, another from cloudwatch, show some statistics from google analytics and ideally monitor a heroku app.

- Graphite: It was trivial to write a 1 line bash script curl-ing the graphite endpoint with &format=json and parsing the output with jq.
https://gist.github.com/ggarber/9490390fdcb5db0251cdb6d3ca6faef9

- CloudWatch: I used a python script and boto3 to be able to get CloudWatch statistics for AWS Firehose but after a couple of try-error iterations the final script was very simple too.
https://gist.github.com/ggarber/8317179246ca11bfe867c93f9c6f0e2d

- Google Analytics: This was the most challenging part specially because the authentication part.   I ended up using this sample from Google tuned to print the exact information I wanted: https://developers.google.com/analytics/devguides/reporting/core/v4/quickstart/service-py

- Heroku: I was not able to figure out how to access programmatically the requests/sec metrics that are shown in the web dashboard :(

The end result is this one, you can even use emojis (:mushroom) to make the information more colorful:




miércoles, 28 de septiembre de 2016

How much plumbing is required to build and deploy a server exposing the simplest HTTP API

I got into an interesting discussion today about the future of development & deployment and one of the premises was that today there is too much plumbing involved on building and deploying everything.

I argued that it was not that much plumbing with modern frameworks or with project templates (like yeoman ones) and that deployment had been heavily simplified in environments like Heroku.

So, in this quick & dirty post I will try to prove my point building a simple HTTP server exposing a Hello-World HTTP API.

Creating the app:

➜  echo "import os
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

app.run(host='0.0.0.0', port=int(os.environ.get('PORT', 5000)))> app.py
➜  echo "flask" > requirements.txt
➜  echo "web: python app.py" > Procfile


Initializing the source control (git) and comiting the changes:

➜  git init
Initialized empty Git repository in /Users/ggb/projects/rgb/.git/
➜  git add *
➜  git commit -m "First version"

Deploying to production:

➜  heroku create rgb-ggb
Creating ⬢ rgb-ggb... done
https://rgb-ggb.herokuapp.com/ | https://git.heroku.com/rgb-ggb.git
➜  git push heroku master *

Try It!: https://rgb-ggb.herokuapp.com/

Summary:


Code: 7 LoC (4 is the plumbing of starting the server and it has to be done only once)
Deployment: 1 file (Procfile) to tell heroku how to start your app (this is not needed for node.js apps and could be autogenerated with a yeoman template) + 1 git push command in the console (and a "heroku create" command the first time)


sábado, 17 de octubre de 2015

You need a corporate framework

If you are working in a big enough software development team you probably agree that consistency in the code and development practices is very important.  Consistency is what makes you save time when joining a new project or reviewing somebody else code, or what saves ops team time when they deploy a new module and have to figure out how to monitor it, or what saves analytics team time when they have to understand and use the logs & metrics generated by a new component.

To be able to get certain degree of consistency (and quality at the same time) it is very common these days to have coding guidelines, technical plans, training plans, code reviews....  All those practices are very important and help a lot to achieve certain degree of consistency, but in my opinion they don't solve some of the most important problems and in addition they depend a lot on human responsibility (bad, very bad, you shouldn't trust any human).

So let's try to figure out what are some of the problems we have today.  Is any of these problems familiar to you?
  • You start a new project and you don't know what folders to create (should I create doc and test folders), how to name things (is it test or tests, src or lib?), should I use jasmine or mocha for testing, should I put the design of this component in a wiki page, a gdocs or a .txt in a folder, where do I put configuration, should I mention the third party licenses somewhere ...
  • Each component logs different things, with different names and in different format.  Do all your components log every request and response? Do they use WARN and ERROR consistently?  Do you always use the same format for logging?  I've seen teams using as many logging libraries as components they have.  The cost of not having good consistent logging can easily make a company waste hundreds of thousands of dollars very quickly.
  • Half of the components don't have a health or monitoring endpoint, or if they have it the amount of information shown or the format is totally inconsistent.   One service expose the average response time, the other the P99, the other only counters...  It makes hard (if not impossible) to monitor components so at the end nobody pays attention to them until a customer complains.
  • My retries strategy sucks.   Do you always retry when you make requests to third party components (very common with the popularization of "microservices" architectures)?  All your components do the same amount of retries?  The timeout before retrying is always the same?  Do you retry against a different server instance?
  • The configuration of each component is different.    One use XML, the other JSON, the other env variables?   In some components it can be changed on the fly while in others it can't?  In some components the config is in git, in others in chef recipes, in others in external configuration servers?
  • Do you have any service registration and service discovery solution?  Or some services are registered in a database, others in a config file, others in the load balancer configuration file?

Use the force Luke!

What you need is a corporate framework and a corporate project template.

You don't even need to create your own framework.  The best example of this kind of framework I know would be Finagle from Twitter and other teams like Tumblr, Pinterest or Foursquare are reusing it.

Finagle enforces a design to build Scala services (Futures based), it provides a TwitterServer class that automatically exposes a stats endpoint and read configuration properties from command line arguments, includes support for distributed logging,  provides lot of clients (MySQL, HTTP, Redis...) exposing a consistent API and automatically generating logs and statistics, integrates with zookeeper for seamless registration and discovery of services.    If you don't know it I highly recommend you to take a look.

I tried to implement my own framework some months ago (https://github.com/ggarber/snap).  It is very rudimentary (the maturity level of a hackathon project) but I'm using it in production to test if it is really helpful and even at the level of immaturity it has I found it very helpful (I don't need to care much about consistency anymore and it also saved me time).

The other piece I think it is mandatory is to have project template.   It avoids you having to make decisions and should have a reasonable amount of tools integrated to automatically run tests, review styles, initiate a pull request... and maybe even deploy.

This project template can be an Eclipse plugin, a yeoman generator or something else, but if you don't have one I don't understand why :)  As an example for node.js projects I like this one created by a friend: https://github.com/luma/generator-tok

Hopefully I convinced you of how important is to have a corporate framework and project template that you use for all your components.    Feedback is more than welcomed.     And contributors for the snap framework (https://github.com/ggarber/snap) even more! :)

sábado, 4 de julio de 2015

HTTP/2 explained in 5 minutes

After reading and playing for some days with HTTP/2 this is a summary of my understanding at a very high level.

HTTP/2 is all about reducing the latency accessing web applications.  It maintains the semantics (GET, POST... methods, headers, content)  and url schemes of existing HTTP and it is based on the improvements proposed by Google as part of his SPDY protocol that is finally replaced by HTTP/2.

The three most important changes introduced in HTTP/2 in my opinion are:
1) Reduced overhead of HTTP Headers by using binary fields and header compression (HPACK).
2) Ability to use a single TCP connection for multiple HTTP Requests without any type of first in line blocking (Responses can be sent in different order than requests).
3) Support for pushing contents from server to client without previous request (for example the server could send some images that the browser will need when it receives the request for the HTML file referencing those images).

The most controversial "feature" of HTTP/2 was making TLS mandatory.  At the end the requirement was relaxed but some browsers (firefox) plan to make it mandatory anyways.

Most of the relevant browsers (at least chrome and firefox and some versions of IE) already include support for HTTP 2 as well as the most popular opensource servers (nginx and Apache).  So you should be able to take advantage of the new version of the protocol right now.

The HTTP/2 support is negotiated using the same HTTP Upgrade mechanism used for websockets and should be transparent for users and elements in the middle (proxies),

Application developers should benefit for free automatically without any change in their apps but they can get even more benefits with some extra changes:
* Some tricks that are in use today like spriting or inlining resources are not needed any more.  So they can simplify the build/deploy pipeline.
* Server push could be automatic in some cases but in general will require developers to declare the resources to be pushed in each request.  This feature requires support in the web framework being used.

I made a hello world test pushing the javascript referenced in an HTML page automatically and it improved the latency as expected.  I plan to repeat the test with a real page with tens/hundreds of referenced js/css/img files and publish the results.


lunes, 8 de diciembre de 2014

Static type checking for Javascript (TypeScript vs Flow)

I've never been a big fan of Javascript for large applications (nothing beyond proxies and simple services) and that is partially because in my experience the lack of static typing ends up making very easy to make mistakes and very difficult to refactor code

Because of that I was very excited when I discovered TypeScript some months ago (Disclaimer: I'm not a JS expert) and I was very curious about the differences between TypeScript and Flow when some colleage pointed me to it today.   So I tried to play find the seven differences, but I'm lazy and I stopped after finding one.

Apart from cosmetic differences and tools availability both TypeScript and Flow support type definition based on annotations, type inference and class/modules support based on EcmaScript 6 syntax.    The relevant difference I found after reading/playing with them (for half an hour) is that because of the way they implement type inference Flow can detect type changes of the variables after the initial declaration making it more appropriate for legacy Javascript code where adding annotations can be not possible.

This is some code I used to play with it with some inline comments:

var s = "hello";
s.length  // Both TS and Flow know that s is a string and they check they have a length method


var s: string = null;
s = "hello";
s.length  // Both TS and Flow know that s is a string and they check they have a length method

var s = null;
s = "hello";

s.length // TS doesn't know that this is a string but Flow knows and can check it has a length method

domingo, 26 de octubre de 2014

Service discovery and getting started with etcd

After playing with some Twitter opensource components recently (mostly finagle) I became very interested on the concept of service discovery as a way to implement load balancing and failure recovery in the interconnection between internal services of your infrastructure.   This is specially critical if you are have a microservices architecture.

Basically the idea of Service Discovery solutions is having a shared repository with an updated list of existing instances of type A and having mechanisms to retrieve, update and subscribe to that list allowing other components to distribute the requests to service A in an automated and reliable way.


The traditional solution is Zookeeper (based on Google Plaxos algorithm with code opensourced by Yahoo and maintained as part of the Hadoop project) but apparently other alternatives have appeared and are very promising in the near future.  This post summarized very well the alternatives available.

One of the most interesting solutions is etcd (simpler than Zookeeper, implemented in Go and supported by the CoreOS project).  In this post I explain how to do some basic testing with it.

etcd is a simple key/value store with support for expiration and watching keys that makes it ideal for service discovery.   You can think of it like a redis server but distributed (with consistency and partition tolerance) and with a simple HTTP interface supporting GET, SET, DEL and LIST.

Installation

First step is to install etcd and the command line tool etcdctl.
You can easily download it and install it from here or if you are using a Mac you can just "brew install etcd etcdctl"

Registering a service instance

When a new service instance in your infrastructure starts it should register himself in etcd by sending a SET request with all the information that you want to store for that instance.

In this example we store the hostname and port of the service instance and we use a url schema like /services/SERVICE/DATACENTER/INSTANCE_ID.   In addition we set a ttl of 10 seconds to make sure the information expires if it is not refreshed properly because this instance is not available.

var path = require('path'),
    uuid = require('node-uuid'),
    Etcd = require('node-etcd');

var etcd = new Etcd(),

    p = path.join('/', 'services', 'service_a', 'datacenter_x', uuid.v4());


function register() {
  etcd.set(p,
    JSON.stringify({
      hostname: '127.0.0.1',
      port: '3000'
    }), {
        ttl: 60
    });


  console.log('Registered with etcd as ' + p);

}
setInterval(register, 10000);
register();

Discovering service instances

When a service in your infrastructure requires using other service it has to send a GET request to retrieve all the available instances and subscribe (WATCH) to receive notifications of nodes down or new nodes up.

var path = require('path'),
    uuid = require('node-uuid'),
    Etcd = require('node-etcd');

var etcd = new Etcd();
var p = path.join('/', 'services', 'service_a', 'datacenter_x');

var instances = {};
function processData(data) {
  if (data.action == 'set') {
    instances[data.node.key] = data.node.value;
  } else if (data.action == 'expire') {
    delete instances[data.node.key];
  }
  console.log(instances);
}


var watcher = etcd.watcher(p, null, {recursive: true});
watcher.on("change", processData);

etcd.get(p, {recursive: true}, function(res, data) {
  data.node.nodes.forEach(function(node) {
    instances[node.key] = node.value;
  });
  console.log(instances);
});


Conclusions

Service discovery solutions are becoming a central place of lot of server infrastructures because of the increasing complexity in those infrastructures specially because of the raise of microservices like architectures.  etcd is a ver simple approach that you can understand, deploy and start using in a less than an hour and looks more actively maintained and future proof than zookeeper. 

I tend to think that if Redis is able to have a good clustering solution soon it could replace specialized service discovery/configuration solutions in some cases (but I'm far from an expert in this domain).

The other thing that I found missing are good frameworks making use of these technologies integrated with connection pool management, load balancing strategies, failure detection, retries...    Kind of what finagle does for twitter, maybe that can be my next project :)

jueves, 20 de marzo de 2014

Actor Model

The more I write concurrent applications the more I hate it.    Typically you end up having a code full of locks, queues, threads and threadpools where it is from difficult to impossible to know if it is correct or it only apparently works.

Because of that I decided to do a little research on the Actor Pattern that apparently is powering frameworks like Erlang making it a very good solution for highly concurrent communication platfomrs (like Facebook Chat or WhatsApp).

These are the slides I prepared, there is no much explanation on them, so feel free to ask me any question and try it!   The Actor Model is fun and will simplify your life no matter if you use a framework for it or just keep in mind the concept in your future designs.




miércoles, 12 de febrero de 2014

Scientific way of estimating the cost of a feature in your project



I'm a fan of estimations as long as they are not used to try to figure out when a feature will be done.    I like estimations and I think they are critical when they are used to decide which features should be done and which ones shouldn't.

So, if they are so important, what is the best way to make estimations.   I'm going to share my secret formula based on the things that I have read and my personal experience in my professional career (where I have to admit that my estimations are now completely different than 15 years ago).

There are two key concepts that we need to understand before digging into the actual formula:
  • One feature working doesn't mean the feature is complete or ready.   Instrumentation, thread safety, unit tests, error handling, documentation, automation, unexpected problems, bug fixing...  most of the times takes much more time that the implementation of the basic functionality.
  • Once you write something you usually have to maintain and not break it forever.   Making sure that new features, refactors or any minor change doesn't break any existing code is a really big deal in any project with enough complexity.

Based on those key concepts we can split the cost of a feature in 3 buckets:
  • Cost to have something working (the usual engineers initial estimation): X
  • Cost to have something ready to be shipped: Y
  • Cost to keep it working for the life of the product: Z
For a total cost for adding a feature to a product of X + Y + Z 

And now is when the scientific part is applied.   Based on my experience and thousands (well, maybe 3 or 4) articles I have read I think the Pareto Principle has a perfect application in this case.

In any project the cost of implementing the basic functionality (X) is 20% vs the 80% of implementing the rest of functionality needed to ship the product (Y).   So Y = 4 * X

The Circular Estimation Conjecture: You should always multiply your estimates by pi.
I've seen a similar estimation of X + Y = PI * X that is a bit optimistic in my opinion.  I recommend you to read the visual demonstration of what is called the circular estimation conjeture







For the second part (the maintainability cost Z) we can apply the same Pareto Principle to get Z = 4 (X + Y)

With all those numbers in place the conclusion is easy.   The total cost of having a feature in a product is X + 4 * X  + 4 * (X + 4 * X) = 25 * X

Take your initial guess (or ask any engineer) to get X, then the cost of the feature that you need to use to decide if it is worth to waste your time implementing it or not is exactly 25 * X

As corollary and final demonstration of the theorem, I though this post was going to take me 5 mins to write it and it took me 25 mins and I suspect I will have to spend more than one hour discussing about it with other people.






martes, 21 de enero de 2014

Writing sequential test scripts with node

Today I was trying to create a node.js script to test a HTTP service but the test required multiple steps.   I gave it a try by using async module to "symplify" that code and that's the ugly code I came up with.

I'm not an expert in js/node, feel free to comment if I'm doing something wrong, I'm more than happy to learn.

(inflightSession and create are two helper functions that I have)

Test using node + jasmine: 

it("should accept valid sessionId", function(done) {
      async.waterfall([
          inflightSession,

          function(sessionId, callback) {
            create({ 'sessionId':sessionId }, callback);
          }
      ], function(error, response) {
          expect(response.statusCode).to.equal(200);
          done()
    });
  });


Same test using python + unittest:

def create_ok_test():
    session_id = inflight_session()
    response = create({ 'sessionId': session_id })

    assert_equals(200, response.status_code)


Same test using node ES6 generators (yield keyword):

it("should accept valid sessionId", function*() {
      var sessionId = yield inflightSession();

      var response = yield create({ 'sessionId': sessionId });
      expect(response.statusCode).to.equal(200);
  });


Honestly the code in python is way more readable than the existing node code, and still better even when comparing it with the new node generators.    Anway definitely looks like a promising way to move forward in the node community.   Some comments:

ES6 generators are available under a flag in node 0.11 and are supposed to be included in 0.12.

yield is a common keyword in other languages (i.e. python, C#) to exit from a function but keeping the state of that function so that you can resume the execution later.

function* is the syntax to define a generator function (a function using yield inside).

You need a runner supporting those generator functions (in this example jasmine needs to add support for it), basically calling the generator.next and waiting for the result (the result should be a promise or similar object) before calling generator.next again.

UPDATE: As I´m somehow forced to use node, I ended up creating a helper function and my tests are now like this

itx("should accept valid sessionId", inflightSession, function(sessionId, done) {
      create({ 'sessionId':sessionId }, function(error, response) {
          expect(response.statusCode).to.equal(200);
          done()
      });

});


viernes, 17 de enero de 2014

Distributed Load Testing: Conclussions (5/5)

Let's recap what we have done in these series and try to get some conclusions.   The steps or achievements are these ones:
  1. Find and test a distributed load testing tool in python: locust.
  2. Extend locust for custom (non-HTTP) protocol testing.
  3. Use Instant Servers to run the locust master and slaves.
  4. Implement a simple way to autostop the machines when they are not being used based on the locust logs and instant servers stop API.
  5. Create a template for the slaves to be easily cloned.   Use instant servers tags to define groups.
  6. Fix the python Instant Server SDK and extend it with new authentication and clone features.
  7. Extend locust interface adding a button to spawn machines in instant servers directly from the locust web interface.
Today I like even more python and the testing tools based in scripting instead of complex UIs.  This project gave me the oportunity to discover locust and Instant Servers and highly recommend people to use them for this kind of use case, it was very easy and a lot of fun using and combining those technologies.  Hopefully I can get more time for deeper integration of virtual machines in locust (with a good control UI and perhaps support for other providers).


miércoles, 15 de enero de 2014

Distributed Load Testing: py-smartdc and spawing slaves from locust (4/5)

After all the previous work I was able to spawn slaves easily from the Instant Servers interface (just clicking Clone) or with the command line tools but I wanted to go further and explore the extension capabilities of locust to add some very simple support to the locust web page to create the slaves.

How to clone and tag an Instant Servers machine with python

I have to recognize that my first instinct was to try to create a python Instant Servers SDK,  I even created the github repo and built the skeleton of the SDK, but 10 mins later somebody told me how stupid I was because there was already an official python SDK :(

Ok, that's great I though, but when I tried to use it I realize that it was not compatible with Instant Servers.   The "problem" is that the SDKs are maintained by joyent and even if Instant Servers is the same infrastructure, the API version is not exactly the same and the existing python SDK doesn't work with Telefonica infrastructure.

The solution was easy and I created my fork and pull requested a patch [1] that still hasn't been merged.   Feel free to use my fork! [2]  In addition I added another patch to support username&password authentication [3], and another one to add support for cloning machines

Once that's solved creating a machine by cloning the template instance we built in the previous post is very easy:

from smartdc import DataCenter, TELEFONICA_LOCATIONS

mad = DataCenter(location='eu-mad-1',
              known_locations=TELEFONICA_LOCATIONS,
              login='is0012', password='HNSnFAkc', api_version='6.5')

template_found = False
for machine in mad.machines():
        tags = machine.get_tags()

        if tags.get('locust') == 'slave-template':
                new_machine = machine.clone()
                new_machine.add_tags(locust='slave')
                print new_machine.name + ' ' + str(new_machine.get_tags())
                template_found = True

if not template_found:
        print 'slave-template instance not found'


How to integrate spawning machines in locust

Locust is easily extensible in the testing scripts as we saw in the previous post but also in the UI.   It is easy to add functionality to the website but modifying the template and adding more HTTP routes to process new API requests.  In my case I added a new button to spawn an Instant Servers machine and a route to process that request:

In locust/templates/index.html: 

                <div class="top_box box_stop box_running" id="box_reset">
                    <a href="/stats/reset">Reset Stats</a></br>
                    {% if is_distributed %}
                        <a href="/cloud/create">Spawn Slave</a>
                    {% endif %}

                </div>



In a new file locust/cloud.py:

from smartdc import DataCenter, TELEFONICA_LOCATIONS
from locust import web

mad = DataCenter(location='eu-mad-1',
              known_locations=TELEFONICA_LOCATIONS,
              login='', password='', api_version='6.5')

@web.app.route("/cloud/create")
def cloud_create():

    template_found = False
    for machine in mad.machines():
        tags = machine.get_tags()

        if tags.get('locust') == 'slave-template':
                new_machine = machine.clone()
                new_machine.add_tags(locust='slave')
                return new_machine

    return None


You can find my locust fork in [5], don't forget to change your login and password in the cloud.py file.

The result is this simple new button with the functionality required to spawn new machines automatically configured as locust slaves connected to the master for distributed testing.



[1] https://github.com/atl/py-smartdc/pull/9
[2] https://github.com/ggarber/py-smartdc/
[3] https://github.com/atl/py-smartdc/pull/10
[4] https://github.com/atl/py-smartdc/pull/11
[5] https://github.com/ggarber/locust

viernes, 3 de enero de 2014

Distributed Load Testing: Using Instant Servers for semi-automated slaves spawning (3/5)

As I mention in the first post of this series virtual machines provides a very dynamic and cheap way to create slave nodes for load testing.  There are different providers out there but I was interested on using Instant Servers (based on Joyent technology) because it is really simple to start with and it is fun to explore new solutions instead of using always the boring Amazon infrastructure.

The three small features I was interested on implementing with Instant Servers were:
  • Simplify the creation of machines preconfigured to be used of locust slaves for my load testing
  • Starting the slaves automatically on the machine startup so that the master detects them and is able to schedule jobs
  • Stopping the machine automatically when it is not used for any test for some time to make sure we don't waste our money when forgetting to stop those unused machines

Creation of machines

To simplify the creation of machines I decided to build an instance with all the required packages and use it as template to clone the actual slave nodes.    To be able to find this instance later automatically I used Instant Server tags.  As far as I know unfortunately there is no UI for tagging in Instant Servers but you could use a script similar to mine [1].

This machine needs to have python, locust, the test file (locustfile.py), all the basic packages (make, gcc) and the packages required to start locust automatically (see next point).

Auto starting slaves

To make sure that the slaves are started during machine startup and that we start multiple instances in every box (because locust is single threaded and I want to use multicore machines) I used supervisord with the following (hopefully autoexplicative) configuration.   Tune the numprocs parameter depending on the machine you are using.

/etc/supervisor/conf.d/locust.conf

[program:locust]
command=locust -f /root/locustfile.py --master-host=81.45.23.221 --slave
stderr_logfile = /var/log/supervisord/locust-stderr.log
stdout_logfile = /var/log/supervisord/locust-stdout.log
process_name=%(program_name)s_%(process_num)02d
numprocs=4

Auto stopping machines


To monitor the usage of a machine I could be using the extensive Analytics API in instant servers but I decided to use the locust log file for simplicity.   I created a python script [2] monitoring log files for activity and if there is no activity for some minutes it invokes the stop Instant Servers API to shut down that instance.

Note: It was not possible to use the existing python SDK with Instant Servers and I had to fork it and made a small modification.  More details in next post.


PD. If I ever mention the word "cloud" in any of this posts, please feel free to insult me, I will deserve it.

[1] https://gist.github.com/ggarber/8381238
[2] https://gist.github.com/ggarber/8381263