Looks like my network connection is really slow . Changing the testing site to Croatia where I have a server I get

Slightly different since Google redirects me to their localized Google site however it leads me to believe that it's my connection that is slow not Google.

Any number of  "remotes" can be added. Want it ? Get it @GitHub

https://github.com/vvuksan/fantomtest

That is why I'm introducing fantomTest. A simple web application that allows you to generate waterfall graphs using PhantomJS. It will also take a screenshot of a rendered page. Here is what that looks like

What's interesting in this particular case is that Google is not utilizing web performance recommendations by using a HTTP redirect from google.com to www.google.com.

Anyways to get fantomTest go to

https://github.com/vvuksan/fantomtest

Recently I came to a realization that lots of sysadmins are unaware that they could easily use trending data they already capture with systems such as Ganglia, Graphite, Collectd, Munin etc. to do alerting. Standard way of doing health checks of remote nodes in Nagios is to install the Nagios Remote Plugin Executor aka. NRPE which allows you to execute Nagios plugins on remote nodes and pipe output to the Nagios server. NRPE does the job however has three major disadvantages

1. It is another daemon that needs to run on the remote host possibly introducing security concerns
2. Depending on the load of the machine can be slow thus bogging down the Nagios server
3. Last and most important is that commonly it's used to alert on common metrics such as disk, load, CPU, swap which you should be trending anyways.

Instead what you ought to be doing is use trending data for alerting. I can think of at least 4 reasons to do so

1. You may already be collecting pertinent data ie. system load, swap, CPU utilization
2. If you are alerting on a particular metric you should likely be trending it
3. It's fast
4. Allows you to do more sophisticated checks easily ie. alert me if more than 5 hosts have a load greater than 5 etc.

Years ago I used Ganglia Web PHP code to write my own generic Nagios Ganglia plugin. This has served me well. Most recently Michael Conigliaro rewrote the script in Python making it more versatile and more powerful. You can download it from here

https://github.com/ganglia/ganglia_contrib/tree/master/nagios

In a nutshell what it does is download the whole metrics tree ie. list of all hosts with their associated metrics. Caches it for a configurable amount of time then uses NagAconda to support all the threshold reporting as defined in Nagios developer guidelines.

Another alternative if you have a very large site is Ganglios which was opensourced by guys at Linden Lab. Their problem is/was that they have thousands of hosts and downloading the whole metrics tree takes ~15 seconds so they have separated the logic that downloads the metric tree and one that does alerting. You can download Ganglios from

https://bitbucket.org/maplebed/ganglios

This can easily be adapted to work with your trending system of choice.

To describe a custom graph you would create a JSON file similar to this one

{
 "report_name" : "network_report",
 "report_type" : "standard",
 "title" : "Network Report",
 "vertical_label" : "Bytes/sec",
 "series" : [
 { "metric": "bytes_in", "color": "33cc33", "label": "In", "line_width": "2", "type": "line" },
 { "metric": "bytes_out", "color": "5555cc", "label": "Out", "line_width": "2", "type": "line" }
 ]
}

This will create a line graph with bytes_in and bytes_out metrics. Since hostname and cluster are not specified it is assumed that we want metrics for the current host we are viewing. You could however specify a particular host and metric you want to graph by adding hostname and cluster attributes to series ie.

{
 "report_name" : "our_load_report",
 "report_type" : "standard",
 "title" : "Load Report vs. Database Load",
 "vertical_label" : "Loads",
 "series" : [
 { "metric": "load_one", "color": "3333bb", "label": "Load 1", "line_width": "2", "type": "line" },
 { "hostname": "db1.domain.com", "clustername": "Databases", "metric": "load_one", "color": "44ddbb", "label": "DB1 Load 1", "line_width": "2", "type": "line" },
 ]
}

To use the reports all you have to do is put the report in the $GANGLIA_WEB_ROOT/graph.d directory. Name them something_report.json and it will be available for any host in the cluster. There is one important thing to note. By default graphing function will look for PHP definitions for graphs as those in theory provide more power and flexibility and if those are not available use JSON definition.

Types of graphs

Currently both line and stacked graphs are supported. Look in graph.d/ directory for additional examples.

Future

I am particularly excited about this feature as it allows us to define aggregate graphs easily. There is even an alpha implementation of functionality which would allow you to specify a metric and a regex host entry and you would end up with an aggregate graph .

Download location

Latest version of the UI can be downloaded either from Ganglia Monitor Web 2.0 SVN branch or you can get it on Github.

• RRD doesn't offer high resolution ie. after about an hour it's all averages and I want to knows what was the metric value last year at this hour and minute
• Data drops off/is destroyed after a year - I want to keep my data forever, disk is cheap etc.

Those are valid points however none of them are the fault of RRD. RRD is a circular buffer so in order to be able to write into it you have to precreate it (otherwise it wouldn't be a circular buffer ). Obviously more data points you store bigger the RRD file will be. To illustrate the point Ganglia Monitoring uses following defaults to create RRDs

RRAs "RRA:AVERAGE:0.5:1:244" "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:244" "RRA:AVERAGE:0.5:5760:374"

This will create multiple circular buffers within the same RRD database file. In order to make sense out of this you need to know what the polling interval is ie. how often do you write into RRDs. In Ganglia's case the default is 15 seconds so

• "RRA:AVERAGE:0.5:1:244" says write actual values (:1:) for every polling interval. Save last 244 of those so in our case we'll have 61 minutes worth of actual data points. Since it's a circular buffer data older than 61 minutes will be "dropped"
• "RRA:AVERAGE:0.5:24:244" says average 24 values (:24:), 24 * 15 seconds = 360 seconds = 6 minutes. 244 of those times 6 is a whole day
• You can do the next two
• Last one "RRA:AVERAGE:0.5:5760:374" says average whole day (5760 * 15 seconds = 1440 minutes = 1 day) worth of values and store it in 374 points ie. little more than a year

When graphing RRDtool is smart enough to use the buffer which gives you the most data points. To store all this data RRD file will use about 12kBytes. Thus if you want higher resolution you will need to change the definition e.g. you could do this

"RRA:AVERAGE:0.5:1:2137440"

which will give you one year worth of data points with no averaging with 15 second interval. Trouble is the size of this RRD file is 17 Mbytes. This may not seem as bad but one of the RRD drawbacks is that every time you add data to an RRD the whole file is written over so if you have 1000 metrics you can be potentially writing 17 GBs of data every 15 seconds. This may be a problem depending how many metrics you are keeping track of. There are alternatives which increase throughput such as storing RRDs in RAMdisk or using rrdcached. Alternatively you can opt to keep 2 weeks worth of data points with e.g.

"RRA:AVERAGE:0.5:1:81984"

which will result in size of about 650 kBytes per RRD file. Or you can do something else altogether. Flip side of RRD is that there are no indexes to maintain, no tables that need to be rotated.

Update: I was wrong about the whole RRD file needing to be updated. In retrospect it makes sense and I apologize for providing the wrong info. You can read comment from Tobi Oetiker (creator of rrdtool) in comments below for more detail. This is actually awesome news since there is very little downside in making larger RRDs.

As far as Ganglia you can modify the defaults in /etc/ganglia/gmetad.conf file. You can also use gmetad-python which allows you to write your own plugins and store metric data in both RRD format, SQL or any other storage engine of your choice.

More on RRDtool can be found here

http://oss.oetiker.ch/rrdtool/tut/rrdtutorial.en.html

• Push based metrics ie. a lightweight agent on hosts that need to be monitored
• Lots of basic metrics by default such as load, cpu utilization, memory utilization
• Trivial to add new metrics ie. execute the gmetric command with metric value and graph automatically shows up
• Decent web interface that allows you to easily drill down when troubleshooting problems

I have used other monitoring systems such as Cacti, Zenoss and Zabbix and found them lacking since they were overly complicated, hard to configure and customize. That said I have also had misgivings about certain parts of the Ganglia UI. Specifically what I missed were following features

1. Ability to search hosts and metrics - looking for specific host or metric gets cumbersome even on clusters with 20-30 hosts
2. Ability to create arbitrary groupings of host metrics on one page ie. a page with web response time for each web server and mySQL lock time would be something you'd have to write custom code for
3. Easy way to create custom graphs ie. either aggregate line graphs or stacked graphs
4. Easy way to add custom graphs to either clusters or hosts ie. I have a stacked Apache report showing number of GETs vs. POSTs. It's hard or impossible to show that graph only on webservers but not on mySQL servers.
5. Mobile (WebKit) optimized experience - minimize zooming/panning etc.

Couple months ago on #ganglia Freenode IRC channel we were discussing some of the pitfalls of the UI and the idea of rewriting Ganglia UI was born. As I have been doing quite a bit of work with jQuery in months past I decided to to give it a shot.

Goals

My initial goals were

1. Implement basic search functionality ie. one search term that will show matching hosts and metrics
2. Add a way to add "optional" graphs on per cluster/per host basis ie. have a default set of graphs and allow those to be overriden using cluster or host override config files
3. Add Views ie. ability to group host/metrics
4. Add Mobile/Webkit View
5. Store view and optional graphs config information in a format that can be easily manipulated by web UI, config management system or by hand - this is one of the key omissions in most monitoring setups where adding/removing hosts requires either manual intervention or kludgy hacks. As someone who has had to spend hours manually clicking around Zabbix interface whenever we added a new server this had major importance

Implementation

Initially there was an idea to rebuild the whole interface from scratch which we still may do but I decided that that would be too much work especially since I wasn't absolutely sure whether my intended changes would make sense for most people. Thus I decided to modify the existing UI.

So far these are the features that have been implemented

Visual aides

In cluster view next to each host now you'll see the full hostname in text on top of the graph. Same goes for metric names  in host view. Now even if you have hundreds of metrics you can click CTRL-F in your browser and find the metric quickly. Also there is a hidden anchor next to each metric which is used by the search tab.

Doesn't seem like much until you need it .

Search

Search tab allows you to type in a single term which will match hosts and metrics. It will search as you type. Hosts first, metrics on host second. Clicking on hosts opens a new window with the view of the host. Clicking on a particular metric takes you to the metric in question.

Views

Views are defined using JSON configuration files. One JSON file per view. There are two types of views, standard and regex views. For example standard view will look like this

{  "view_name":"default",
   "items":[
      {"hostname":"host1.domain.com","graph":"cpu_report"},
      {"hostname":"host2.domain.com","graph":"apache_report"}
    ],
    "view_type":"standard"
}

It will group cpu report from host1 and apache_report for host2. Regex view allows you to use regular expressions to define hosts (soon also metrics) ie. you want to group all hosts that have imap, amavis or smtp in their names. That view definition would look something like this

{  "view_name":"mailservers",
    "items":[
      {"hostname":"(imap|amavis|smtp)", "graph":"cpu_report"}
    ],
    "view_type":"regex"}

If you don't want to edit JSON config files by hand you can use the UI to create standard views ie. first create a view then as you browse hosts there is a plus sign next to each graph. Clicking on it displays a dialog which allows you to add that particular host/metric to a view e.g.

Automatic rotation

Allows you to automatically rotate a view. It is an integration of GangliaView with Views. What's especially nice is that if you have multiple monitors you can open up separate browser windows and select different views to rotate.

Mobile view

There is a functional mobile view which provides mobile view of Views, Clusters and Search ie. there is very little panning or zooming. Also we are using lots of preloading ie. first page you open contains lots of hidden sub-pages in order to save on having to do subsequent requests.

You can view some of the screenshots on Flickr.

Optional Graphs

You can specify which optional graphs you want displayed for each host or cluster. Similar to views these are configured via JSON config files e.g. this is the default list of graphs

{
	"included_reports": ["load_report","mem_report","cpu_report","network_report","packet_report"]
}

You can exclude any of the default included graphs or include ones you want e.g.

Screencast

If you would like to see some of these features in action you can look at these screencasts.

Download

Ready to try ? Wait no more and check it out from SVN at

http://ganglia.svn.sourceforge.net/svnroot/ganglia/branches/monitor-web-2.0/

Future

In the future we are looking into polishing the Graphite/Ganglia integration (perhaps about that in a next post), add integrations with e.g. Nagios (you can see a hint of it in the add metric to view screenshot above), Logstash. Also another upcoming feature will be aggregate metrics and quick views. Full TODO list can be found here

http://sourceforge.net/apps/trac/ganglia/browser/branches/monitor-web-2.0/TODO

Acknowledgements

I'd like to thank Erik Kastner for helping on the Graphite/Ganglia integration. Ben Hartshorne for test driving the UI and providing a number of good suggestions/ideas.

Couple days ago I saw a tweet from Mike Brittain at Etsy on how Graphite is becoming one of his favorite graphing tools. I know that they use Ganglia at Etsy so I asked if/how they use integration between Graphite and Ganglia. He pointed me in the direction of Erik Kastner who has done Ganglia Graphite integration. I asked him if he could post the patches and he was gracious to do so. In a nutshell he uses RRD files directly and rsyncs them every few minutes. While trying to install Graphite I realized that injecting metrics into Graphite is really simple. For example graphite-web contains a simple client example that injects system load. All it does is connects to port 2003 of the graphite installation and sends a following payload

system.loadavg_1min 0.08 1285763852
system.loadavg_5min 0.02 1285763852
system.loadavg_15min 0.01 1285763852

That's simple ie. some type of a metric name, value and what looks like current UNIX timestamp. I then remembered that Kostas Georgiou showed me a ruby script that connects to gmond, retrieves the XML for the host, parses it and adds to Facter. Unfortunately that didn't seem to have much value until now . What I did  is change Kostas' script to send metrics to Graphite instead of adding them to facter. You can find the result at Ganglia Add-Ons GitHub repository. You can run the script either from cron or as a daemon.

There are two ways to do this. I have tested only the first way. I am not sure if the graphite receiver would freak out if it gets too many metrics in a payload. Let me know if you know .

1. Run this script on every host that runs gmond. This may be somewhat tricky since I usually set up gmond to only send metrics and turn off receiving by setting deaf = yes. For this approach to work you have to turn on receiving. To make it more secure we'll just listen on loopback. In global make sure you have these settings

  mute = no
  deaf = no

udp_send_channel {
  host = 127.0.0.1
  port = 8649
  ttl = 1
}
udp_recv_channel {
 bind = 127.0.0.1
 port = 8649
}
tcp_accept_channel {
   bind = 127.0.0.1
   port = 8649
}

2. Run this on the main gmond collector daemon. Main gmond collector daemon will have metrics from all hosts. Trouble is that I haven't tested injecting thousands of metrics in a single payload. I'm sure there is a way around it and perhaps someone can post a patch .

Future Improvements

I can think of couple possible improvements

1. There is a rewrite of gmetad written in Python. It supports plugins. I don't think it would be a stretch to add a plug-in where gmetad sends data to Graphite when it updates the RRDs
2. Currently metrics are sent as <hostname>.<metric_name>. It may make sense to send them into the appropriate part of the tree ie. <type_of_metric>.<hostname>.<metric_name> e.g. database.web1.mysql_selects
3. Better integrate Ganglia Web UI and Graphite. Graphite supports flexible URL parameters so this should be doable.

And obligatory screenshots. This is the stacked graph I created in 20 seconds

Most recently we wanted to get better insight into how our site and more specifically backend is performing. We wanted a tool that could provide us with per URL/page metrics such as

• total number of requests
• aggregate compute time
• average request time
• 90th percentile time (you can find more explanation what it means at monitor your site using 90th percentile response times) - this eliminates most of the really slow response times that may really affect your averages

Initial plan was to build a basic set of reports to tell us what are the pages with excessive response times or large total (aggregate) compute times. Next and yet to be implemented portion was to be able to analyze data in real time so that we'd have another data point to use in troubleshooting in case there is a site slow down.

Basic requirements for the tool were these

• Capable of crunching 100+ million daily entries
• Real-time analysis
• Produce multiple metrics with potential to add more down the line
• Low footprint

An obvious way to do this is to store all data in a heavy duty data store like a relational/SQL database or something MapReduce capable. Trouble is we may be doing in logging in excess of 3,000 hits per second (all dynamic content as static assets are served from the CDN). Doing that many inserts per second on a SQL-type database will be tricky unless you have powerful hardware. Next obvious problem is to scan through hundreds of millions or billions of rows will be slow even if I use MapReduce unless of course you throw tons of hardware at it. We wanted a low footprint remember.

Instead we decided to go with a key/value store. Major pluses were that footprint is relatively low and it performs very fast. Downside was I would not be able to run any sophisticated queries. Since we already have an app that uses memcached to give us real-time view per IP number of accesses we ended up using it for this purpose as well.

Implementation

I have been working for a while now with ganglia-logtailer which is a Python framework to crunch log data and submit it to Ganglia. There are a number of good pieces from it we could reuse and we did. What we ended up is a two part tool. A Python based log parsing piece and a PHP based web GUI and computation part. Division of "labor" was roughly this

• Python part parses the logs and creates entries/keys where the value in each key represent all the response times observed on a particular server and URL in a particular time period ie. one hour
• PHP part takes the list once the time period has ended, calculates total time, average time and 90th percentile times and stores computed values in memcache so that retrieval later can be quicker.

Graphing is achieved using simple CSS graphs while time based series are done using OpenFlashChart. I did look at Dygraphs for Javascript/DHTML based graphing however couldn't figure how to plot hourly values. I could only do daily values.

Tool is operational and so far it has led us to the realization that our mobile web pages are overall much slower than their corresponding web pages. This is due to the way we handle mobile ads since most feature phones don't support Javascript so we have to download the ad which introduces a slight delay. We did figure out that we could use Javascript on Webkit browsers similar to what we do for regular browsers so that should help a bit. We are also chasing some of the other "leads" regarding inconsistent performance for particular pages on some of the servers.

Next steps are to adapt parsing code to work with ganglia-logtailer which would give us real-time reporting. I don't expect too many problems with that. Also graphing could use some more love. Perhaps I'll even do standard deviation calculations .

Anyways you can download source code from here

http://github.com/vvuksan/pagetime-analyzer

You know what to do .

Obligatory screenshots

Hourly overview sorted by aggregate time in seconds (you can sort by any column)

This is the average response time (over an hour) for a particular URL on separate server instances

Daily view of performance for a particular URL

http://github.com/vvuksan/cronologger

Cronologger is a BASH script that stores all the cron output into a database. I am using CouchDB since it is a great document oriented database that allows me to add attachments (blobs) to a document. I assume it would not be hard to use MongoDB, Riak and others.

Some of the benefits of this utility are

• Reduce cron spam
• Provide the ability to correlate adverse affects by overlaying cron events on e.g. Ganglia graphs
• Provide a better report of all the batch jobs that ran, diff them with past jobs if they should look the same, etc.
• Provide the ability to easily view what is currently running on the whole infrastructure ie. job_duration < 0
• Review historical output

I am still working on web GUI for most of these things. I will gladly accept patches and new contributions.

Tip: To get view a list of documents in a CouchDB database you can use the _utils view e.g. http://localhost:5984/_utils/

This is how you can add deploy timeline to your Ganglia graphs or for that matter to any tool that uses RRDs such as Cacti, Munin, Collectd etc.

Background

RRDtool supports so called VRULEs which are

VRULE:time#color[:legend][:dashes[=on_s[,off_s[,on_s,off_s]...]][:dash-offset=offset]]

Draw a vertical line at time. Its color is composed from three hexadecimal numbers specifying the rgb color components (00 is off, FF is maximum) red, green and blue followed by an optional alpha. Optionally, a legend box and string is printed in the legend section. time may be a number or a variable from a VDEF. It is an error to use vnames from DEF or CDEF here. Dashed lines can be drawn using the dashes modifier. See LINE for more details.

What we want to do is add a VRULE for each deployment. For example those three lines above have been generated using these VRULEs

Implementation

Easiest way to add these to Ganglia is to modify graph.php in Ganglia Web. You need to look for following two lines at the end of the file

$command .=  array_key_exists('extras', $rrdtool_graph) ? ' '.$rrdtool_graph['extras'].' ' : '';
$command .=  " $rrdtool_graph[series]";

Then append your own VRULEs ie.

$command .= " VRULE:" . $time . "#FF00FF:\"Deploys\"";

Obviously you have to pull in the $time info from where you keep track of your deploy times. You can also get creative by using different colors for different deploys, change legend labels, add VRULEs to only certain graphs ie. load, CPU etc. This is a quick and dirty way to do it

$deploy_times = array(1278082860,1279393200);
foreach ( $deploy_times as $key => $time ) {
  # Put deploys label only once.
  if ( $key == 0 )
     $command .= " VRULE:" . $time . "#FF00FF:\"Deploys\"";
  else
     $command .= " VRULE:" . $time . "#FF00FF";
}

Now you just have to make sure you append deploy times in the array.

Alternate implementations

Alternate implementation is to create a RRD file whenever you do deploys then overlay that graph on top of an existing graph. Trouble is you have to worry about scaling the graph. Never could get it quite right.

Credit

Thanks goes to the Circonus guys since they made me think of vertical lines instead of trying the RRD overlay. Also thanks to @toredash for pointing me in the right RRDtool direction by suggesting HRULE.