Howto speed up your monitoring system with Varnish

    03 April 2015

    If you use a monitoring system of any kind you are looking at lots of graphs. It also happens that as size of your team grows you are looking at more and more graphs and often times member of your team are looking at same graphs. In addition as you grow graphs become more complex and you may have fairly complicated aggregated graphs with 100s of data sources which can become quite a bit of a burden on your metrics system. This resulted in complaints about slowness of our monitoring. To speed it up we figured we should our best bang for the buck would be to cache page fragments. Since we run a CDN based on Varnish it was logical what we were gonna use :-).


    • Most metric systems poll on a fixed time interval e.g. 10-15 seconds. If you make a graph you can safely cache it for 10 seconds or longer since graph is not going to change
    • There are a number of static dashboard pages we can cache for longer since they don’t change. Only dependent images change
    • Even if we don’t cache or cache for really short e.g. 1-2 seconds Varnish supports Collapsed Forward which will result in collapsing multiple request for same resource into one ie. if 5 clients request same resource /img/graph1.png at the same time varnish will send only one request to the backend then respond to all 5 clients with the same resource. This is ia huge win.

    You can find an example Varnish configuration in this repo. This is Ganglia specific however you can adapt to suit your needs

    Ganglia contrib repository

    Key file you need is default.vcl which you need to put in /etc/varnish/default.vcl


    Your caching rules should be put in vcl_fetch function. For example

    if (req.url ~ "^/(ganglia2/)?$" ) {
         set beresp.ttl = 1200s;
         unset beresp.http.Cache-Control;
         unset beresp.http.Expires;
         unset beresp.http.Pragma;
         unset beresp.http.Set-Cookie;

    This is a regex match that will match /ganglia2/ or / and cache it for 20 minutes (1200 seconds). Resulting object will also be stripped of any Cache-Control, Expires, Pragma or Set-Cookie headers since we don’t want to send those to browsers.

       if (req.url ~ "/(ganglia2/)?graph.php") {
         set beresp.ttl = 15s;
         set beresp.http.Cache-Control = "public, max-age=10";
         unset beresp.http.Pragma;
         unset beresp.http.Expires;
         unset beresp.http.Set-Cookie;

    Similar to the rule above we set cache time to 15 seconds, we unset all the headers except for Cache-Control which we set to 10 seconds. What this will mean is that varnish will cache the object for 15 seconds however we’ll instruct the browser to cache it for 10 seconds.

    You could also get creative an do things based on content type of the resulting object

      if ( beresp.http.Content-Type ~ "image/png" ) {
         set beresp.ttl = 15s;

    Have fun.

    Adventures with Arduino part 2

    04 February 2015

    In my last post Adventures with Arduino part 1 I discussed some of the options of wiring up and getting metrics with Arduino. Here is the work in progress

    Arduino Wiring image

    It includes a DHT11 humidity temperature sensor, water sensor and a reid switch. The way things are set up is that it polls the sensors periodically e.g.

    • Reid switch (to see if door is open or closed) every 2-10 seconds
    • Humidity and temperature every minute

    It then sends those values as a simple comma separated file. Data format I’m using is

    device uptime,device name,metric_name=value

    with multiple metric values possibly sent in the same packet. On the receiving side I have a Raspberry Pi that follows this work flow.

    • Uses a modified raspberryfriends daemon from nrf24pihub
    • Daemon receives and parses the payload and ships it off to Statsd - using a gauge data type
    • Statsd rolls up any metrics and sends them over to Ganglia. Ganglia is used for trending and data collection e.g. this shows temperature and humidity in one of my bedrooms. You can notice the effect of a room humidifer on humidity in the room :-)

    Arduino metrics

    • I can also see if I left my garage door open :-)

    Arduino metrics

    Arduino metrics

    • In this particular instance that alert has dual meaning since this particular Arduino is driven by one of those “lip-stick” USB battery packs and Ganglia will expire a particular metric if it hasn’t been reported for a defined amount of time (in my case 1 minute). In this particular case alert state UNKNOWN tells me that most likely battery is out I need to recharge it.

    Raspberry Pi Revision B+ NRF24L01 wiring

    31 January 2015

    Raspberry Pi Revision B+ has 40 pins unlike the original Raspberry Pi. To connect NRF24L01 module to it you need to connect it as follows. This is how NRF24L01 looks like. Pins start with 1 in bottom right.

    NRF24L01 Pinout

    NRF Pin Raspberry Pi
    1 Ground PIN 4
    2 3.3 V PIN 1
    3 PIN 22
    4 PIN 24
    5 PIN 23
    6 PIN 19
    7 PIN 21

    Adventures with Arduino part 1

    01 January 2015

    Few months ago I got myself an Arduino Uno board from Adafruit. I have had couple use cases I was going to try to use them for in order of importance

    1. Water leak detection e.g. put a sensor in crawl space or attic for early detection of water leaks
    2. Garage door open/close state - find out if the garage door has been left open
    3. Humidity and temperature monitoring around the house

    One nice thing about Arduino is that some of the small boards are really inexpensive. For example Arduino Nano board or Adafruit Trinket can run you anywhere between $3.50-$7.

    After receiving the board and playing with some of the basic examples I figured it was time to resolve how to push sensor data to a central location. Some of the options I discovered were as follows

    1. XBee modules. Approx cost per module ~ $20
    2. Bluetooth - Approx cost per module $7
    3. 315/433 Mhz remote control - Approx cost per module ~ $1
    4. Nordic Semiconductor nRF24L01+ modules - Approx cost per module ~ $1

    XBee and Bluetooth are really nice options however I considered them too overpriced for this particular use case so I went ahead and bought a pair of 433 Mhz and nRF24L01+ modules.

    315/433MHz modules

    First thing I tried was 433 MHz modules. They were easy to wire and configure since e.g. transmitter has only 3 pins and receiver 4 pins (although you only use 3) and using the rc-switch project libraries I was able to communicate between my Arduino and a Raspberry Pi. The drawback is that it’s fairly low bandwidth and payload size maxes out at 24 bit so pretty limiting.

    That said an interesting side benefit of these modules is that large number of remote controlled power outlets out there use either 315/433 Mhz bands e.g. Etekcity outlets. If you have a remote controlled device in your house you can look up what frequency it uses with FCC ID search. There is also the 303 Mhz frequency however I have not been able to find the modules for it yet.

    As a result of this tinkering I know am able to turn outlets around my house with my phone :-).

    Nordic Semiconductor nRF24L01+

    These are a lot more tricky to get going as they have total of 8 pins with one unused and it is easy to mis-wire things. It took me a lot of trying to get these going however I finally got it going and was able to pass data between the Arduino and a Raspberry Pi. Max payload on this is 32 bytes which should be enough for shipping out metric data and you can ship them at a pretty rapid rate. Libraries I ended up using were these

    RF24 library and NRF24PiHub

    Do note that these may not work with Adafruit’s Trinket.

    Next steps

    I have ordered some temperature and humidity sensors, water sensors and some Arduino Nanos and will be wiring it up :-), shiping metrics to Ganglia and alerting on it.

    Bosnia and Serbia Floods aid

    18 May 2014

    As you may have heard Bosnia, Serbia and in smaller part Croatia are facing worst floods ever in recorded history

    There are a number of ways to donate. Here are few that are being posted on Twitter from

    I have not seen an easy way on those sites for people in the US to make donation. However Croatian Red Cross allows on-line donations with proceeds being transferred to Bosnian and Serbian Red Cross. You can read about it here

    The only drawback is donation page is in Croatian :-( so here is a quick guide.

    On this page you will need to pick who you are donating to

    • Pomoć za poplavljena područja u Bosni i Hercegovini - Aid for flooded regions of Bosnia

    • Pomoć za poplavljena područja u Srbiji - Aid for flooded regions of Serbia

    • Pomoć za poplavljena područja u Hrvatskoj - Aid for flooded regions of Croatia

    Pick the amount and currency e.g. 25 USD.

    Click on Autorizacija Kreditne kartice. Next screen will include your confirmation as well as currency exchange into Croatian Kunas e.g.

    10 CAD (50.90 KN)

    1 Canadian Dollar is about 5 Kunas so don’t despair :-).

    Payment info will look like this


    • Ime is name

    • Prezime Last Name

    • Adresa - Address

    • Grad - City

    • Poštanski Broj - Postal Code

    • Država - Country

    • Telefonski broj - telephone number

    Click Doniraj and that should be it.