Vladimir Vuksan's blog

Recently I had to get my own mobile phone service and decided to forgo the standard post-paid cell service and go prepaid. Decision was largely cost based since I already had my own GSM phone and planned to buy a Nexus 4. I did quite a bit of research and ended up with Straight Talk service

https://www.straighttalk.com/

Straight Talk is a MVNO (Mobile virtual network operator) that leases network capacity from T-Mobile USA and AT&T.  To sign up you either order a SIM from their web site or you can pick up starter package at Walmart. I did the latter option. In the package they provided me with 2 different mini-SIM cards and a micro-SIM card. SIM cards are really T-Mobile and AT&T unbranded SIM cards. Pick the card that is supported by the phone e.g. if it's a locked phone like AT&T use the AT&T SIM, change some of the phone settings (APN) and off you go. Quality of the signal is the same as if you used T-Mobile or AT&T. I  picked unlimited everything plan for $45/month with a T-Mobile SIM. If you sign up for auto refill they cut it down to $42.50. Drawback is lack of international roaming and iffy customer service ie. hold times can be 30-40 minutes.

Another option I considered was T-Mobile's prepaid service called Go Smart which is similarly priced

https://www.gosmartmobile.com/

I decided against it since cost was similar but with Straight Talk I have the option to switch to AT&T if I ever find T-Mobile coverage inadequate. That said Go Smart does have a wider array of calling plans so it may still be a good choice.

While we are at it I can also recommend an inexpensive VoIP service called GalaxyVoice. I use their free-tier which gives you up to 60 minutes of outgoing calls a month and all I pay is for taxes and 911 compliance ~ $3/month. You just need to pay the signup cost of $25 and get your own SIP device.

An extra bonus is that their web site is fairly unsophisticated and easy to automate for certain things e.g. forwarding my home phone calls to my cell phone

My friend Igor Ješe who is an all around awesome guy recently wrote a new piece of commercial software called MockupData that allows you to create realistic sample data you can use to test your applications. Without further ado you can read the press release about MockupData here and newsletter announcement. If you have a need for such a tool I would highly recommend Igor's software.

This is not Igor's first product. Prior to MockupData he wrote a tool called Mockupscreens which let's you create mockups easily.

I have couple hundred Dell systems with LSI RAID arrays however we lacked hardware monitoring which would occasionally result in situations where there would be multiple disk failures that would not get caught. Some time ago I read this post about using MegaCli to monitor Dell's RAID controller

http://timjacobs.blogspot.com/2008/05/installing-lsi-logic-raid-monitoring.html

One thing I did not like about this approach is that it may generate too many e-mails as it will send e-mails every hour until the disk has been fixed. Instead I used Ganglia with Nagios to provide me with similar type of functionality.

Create a file called analysis.awk with following content

    /Device Id/ { counter += 1; device[counter] = $3 }
    /Firmware state/ { state_drive[counter] = $3 }
    /Inquiry/ { name_drive[counter] = $3 " " $4 " " $5 " " $6 }
    END {
    for (i=1; i<=counter; i+=1) printf ( "Device %02d (%s) status is: %s\n", device[i], name_drive[i], state_drive[i]); 
    }

Get MegaCli utilities from e.g. RPMFind. Copy following BASH script into a file and run into from a cron at frequency you need e.g. every 30 minutes, hour etc.

#!/bin/sh

GMETRIC_BIN="/usr/bin/gmetric -d 7200 " 

MEGACLI_DIR="/opt/MegaRAID"

MEGACLI_PATH="/opt/MegaRAID/MegaCli/MegaCli64"

BAD_DISKS=`$MEGACLI_PATH -PDList -aALL | awk -f ${MEGACLI_DIR}/analysis.awk | grep -Ev "*: Online" | wc -l` 

if [ $BAD_DISKS -eq 0 ]; then
    STATUS="All RAID Arrays healthy"
else
    STATUS=`$MEGACLI_PATH -PDList -aALL | awk -f ${MEGACLI_DIR}/analysis.awk | grep -Ev "*: Online"` 
fi

$GMETRIC_BIN -t uint16 -n failed_unconfigured_disks -v $BAD_DISKS -u disks
$GMETRIC_BIN -t string -n raid_array_status -T "Raid Array Status" -v "$STATUS"

This will create two different Ganglia metrics. One is number of failed or unconfigured disks and another one is just a string value that gives you details on the failure e.g. Disk 4 failed. Besides being able to alert on this metric it also gives me a valuable data point I can use to correlate node behavior ie. when disk failed load on the machine went up.

If you are using Ganglia with Nagios integration you have two different options on how you want to alert.

1. Create a separate check for every host you want monitored e.g.

define command{
        command_name    check_ganglia_metric
        command_line    /bin/sh /var/www/html/ganglia/check_ganglia_metric.sh host=$HOSTADDRESS$ metric_name=$ARG1$ operator=$ARG2$ critical_value=$ARG3$
        }

define service{
        use                   generic-service
        host_name             server1
        service_description   Failed/unconfigured RAID disk(s)
        check_command         check_ganglia_metric!failed_unconfigured_disks!more!0.1
    }

2. Create a single check that gives you a single alert if at least one machine has a bad disk (how I do it ). For this purpose I'm utilizing the check_host_regex which allows me to specify a regular expression of matching hosts. In my case I check every single host. If a host doesn't have the failed_disks metric I assume it doesn't have it and I "ignore" matches. My config is similar to this

define command{
        command_name    check_host_regex_ignore_unknowns
        command_line    /bin/sh /etc/icinga/objects/check_host_regex.sh hreg=$ARG1$ checks=$ARG2$ ignore_unknowns=1
        }

define service{
        use                             generic-service
        host_name                       server2
        service_description             Failed disk - RAID array
        check_command                   check_host_regex_ignore_unknowns!'.*'!failed_disks,more,0.5
    }

Which will give you something like this

# Services OK = 236, CRIT/UNK = 2 : CRITICAL compute-4566.domain.com failed_disks = 1 disks, CRITICAL git-0341.domain.com failed_disks = 1 disks

On Twitter Grig Gheorghiu posed a number of questions about monitoring tools and what he wants in a monitoring tool. This is my attempt at describing what my setup looks like or has looked like in the past.

1. Metrics acquisition / performance trending

I use Ganglia to collect all my metrics including string metrics. Base installation for Ganglia Gmond will give you over 100 metrics. There are a number of Python modules that are disabled by default like mySQL, Redis that you can easily enable and get more. If you need even more you can check out these to Github repositories.

• Gmond Python Modules
• Gmetric scripts

Don't worry about sending too many metrics. I have hosts that send in excess of 1100 metrics per host. Ganglia can handle it so don't be shy . Also when I say all metrics go into Ganglia I mean EVERYTHING. If I want to alert on it it will be in Ganglia so I have things like these

• NTP time offset
• What version is particular key piece of software on e.g. deploy ID 123af58
• Memory utilization/CPU utilization for key daemon processes
• Number of failed disks in a RAID array
• Application uptime
• Etc.

2. Alerting

I use Nagios or Icinga for alerting. I don't really use any Nagios plugins as all the checks are driven by data coming out of Ganglia. I have written a post in the past about why you should use your trending data for alerting which you can read for some background. About a year ago Ganglia/Nagios integration has been added to Ganglia Web which makes a number of things much easier so for example I have

• A single check that checks all hosts in the cluster for failed disk in a RAID array
• A single check that checks whether time is within certain offset on all hosts ie. to make sure NTP is actually running
• A single check that makes sure version of deployed code is the same everywhere
• A single check for all file systems on a local system with each file system having their own thresholds
• A single check for elevated rates of TCP errors - useful to get a quick idea if things are globally slow, affecting a certain set of hosts in a geographic area or an individual host

Beauty in having all of the metric data in Ganglia is that you can also get creative by writing custom checks that have your own user specified logic e.g.

• Alert me only if 20% of my data nodes are down e.g. in architectures where you can withstand few nodes failing

In addition I recommend adding as much context to alerts as possible so that you get as much as information in alert as possible.

I also heavily utilize something I wrote called the alerting controller

https://github.com/vvuksan/alerting-controller

Which allows you easily enable/disable/schedule downtime for services in Nagios e.g. disable process alive check for configuration servlet on all hosts while we are doing an upgrade etc. In addition I have a tab with Naglite opened up most of the time to check on any outstanding alerts.

https://github.com/saz/Naglite3

3. Notifications

Beyond just alerting I am always interested to see what is happening with the infrastructure so we can act proactively. For that purpose I use IRC and a modified version of NagiosBot which sends echoes following things to the channel

• Nagios alerts (same script that adds context to alerts above) - helpful for quick team coordination
• Dynect DNS zone changes - those may be "invisible" so good idea to track
• Zendesk tickets - anyone can handle support requests
• Twitter mentions of a particular keyword
• Application configuration changes e.g. new version of the code deployed or in progress
• Severe Application errors - short summary of an error and which node it occured on

Closing

This is by no means an exhaustive list and this may not be the best way to do things but it does work for me.

Few weeks ago I posed a question on Twitter whether there would be interest to organize a webops hackathon   I am close to getting confirmation for conference space at Microsoft NERD for October 11, 2012. Now comes the hard part . We need to figure out what format should hackathon and unconference take as well as what topics we should talk about. There is going to be a monitoring conference at the end of March spearheaded by Jason Dixon (@obfuscurity) so we may want to leave out monitoring topics.

In a nutshell we need to decide

1. How much of the event should be a hackathon and how much should be an (un)conference
2. If we want talks what topics should we seek talks for e.g. continuous integration, noSQL, site scalability, etc.
3. Name of the event

If you have any ideas send me an e-mail conference X vuksan.com (replace X with you know what).

I have an application that uses HTTP POST to submit a JSON encoded array. Basically there are no variables that are being submitted, just JSON. This causes $_REQUEST and $_POST arrays to get messed up where "random" parts of the JSON will end up as the key and rest as the value. Instead what you need to do is get contents of the request as input then decode it   e.g.

    $input = file_get_contents('php://input');
    $my_array = json_decode( $input, TRUE);

I have recently moved this blog to be hosted on Fastly, a CDN service with bunch of great features like dynamic content caching with instant purges. Fastly utilizes HTTP headers to determine what to cache as described in the Cache Control document. While configuring my service I noticed that my WordPress (origin server) kept returning HTTP headers like these

Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

I looked through the WordPress code and couldn't see where such value was set. After some internet searches I discovered that they are set using session-cache-limiter option in php.ini. In most distributions this defaults to nocache which ends up with above headers. You can read more on the cache-limiter options here.

http://www.php.net/manual/en/function.session-cache-limiter.php

What we need is session-cache-limiter = public results in headers like these

Expires: (sometime in the future, according session.cache_expire)
Cache-Control: public, max-age=(sometime in the future, according to session.cache_expire)
Last-Modified: (the timestamp of when the session was last saved)

e.g.

Expires: Tue, 22 May 2012 16:38:33 GMT
Cache-Control: public, max-age=10800
Last-Modified: Tue, 04 Oct 2005 00:55:59 GMT

If you want to adjust the max-age you can set cache-expire in php.ini e.g.

; http://php.net/session.cache-expire
session.cache_expire = 180

For a while I have been observing a tendency of technologists/engineers to describe technology in either black or white terms ie. X technology sucks, use Y technology. Most recent example is an article by someone going by the name of Eevee  in his PHP a fractal of bad design. It is a damning expose of PHP's failings/bad decisions/inconsistencies etc. Unfortunately as most articles of this type it involves a number of ad-hominem attacks like these

It’s so broken, but so lauded by every empowered amateur who’s yet to learn anything else, as to be maddening. It has paltry few redeeming qualities and I would prefer to forget it exists at all.

or

I assert that the following qualities are important for making a language productive and useful, and PHP violates them with wild abandon. If you can’t agree that these are crucial, well, I can’t imagine how we’ll ever agree on much.

This irritates me on many levels since it makes so many misguided assumptions e.g.

- Everyone's mind is the same therefore everyone should like or hate language X

Of course not. Your mind is different than my "defective" mind. I quite prefer writing in PHP. I have written/write code in Ruby/Python/Perl and PHP is my preferred language. That may change but at this point it's my preference. You may disagree with my choice and that's OK.

- Issues we are trying to solve are similar/identical and we have same resource constraints aka one-size fits all

Of course not. If most coders on my team are well versed with PHP and we have a tight schedule you bet we are most likely to choose PHP. Technical merits are not the only consideration to be taken. They are certainly important but they often pale in comparison to other considerations such as people and culture.

- PHP core developers are incompetent

I have on a number of occasions disagreed and been frustrated with decisions made by PHP core developers however I do assume that in most/all cases they are well intentioned and are making the best decision under available circumstances. PHP has been around for a long time so I imagine making major changes is tough and involve making significant tradeoffs. If those tradeoffs become a show stopper for me I'll use a different language.

Anyways the most bothersome part of the whole post is the technology "tribalism" which results in things like this (from Kibana README file)

Q: Why is this in PHP instead of Java, Ruby, etc?
A: Because PHP is what I know. The total PHP is less than 200 lines. If you want it in something else, it shouldn't be too hard to port it to your language of choice

That makes me pretty sad.

This is a quick way to extract a 15 minute average of a metric in Ganglia. It utilizes Ganglia's CSV export function to get the values then uses awk to actually compute the average.

First of all find a metric graph you want to calculate average from. Right click over the image and copy the image location. Then append &csv=1 to the URL and UNIX time stamp from 15 minutes ago and put that as the &cs= argument. This is a simple shell script that illustrates it

MIN15AGO=`date --date="15 minutes ago" "+%s" ; 
curl --silent "http://ganglia.domain.com/ganglia/graph.php?c=NetApp&h=host1&v=&m=netapp_cpuutil&cs=$MIN15AGO&csv=1" | \
   awk -F, '{sum+=$2} END { print "Average = ",sum/NR}'

I am a big believer in adding context to alerts. This allows the recipient of an alert to make a better decision on how to deal with an alert. It's often hard to classify alerts so providing as much context to the alert is extremely helpful. For instance if I am alerting on a value of a metric I like to attach an image of that metric for the past hour. This way if I am on my mobile phone and out and about I have the alerting metric graph right there without needing to open up another window or having to start up my laptop.

In more recent versions of Ganglia there is an option to add overlay events to hosts which show up as vertical lines on the graph. I figured that would be great context to add to alerts. Since I'm using Nagios I decided to extend a mail handler I used before to query Ganglia events database and include any events that were connected to the matching host in 24 hours. This helps in a number of  scenarios to keep team on the same page and well informed e.g.

• There was a code push/config change however host/service was not scheduled for maintenance
• Recent code push is causing issues ie. web servers are crashing

This is an example e-mail you get

As an added bonus mail handler sends all alerts to a Nagios Bot . Now all you need to make sure is to record events for any major changes. You could do a lot of these things automatically by e.g.

• Adding hooks to your startup scripts so that when you purposely restart services it is logged
• Watching logs then inserting proper events in the timeline. App stoppe
• Querying external services e.g. Dynect provides an API to query zone changes

You can download the mail handler from here

https://github.com/vvuksan/misc-stuff/blob/master/nagios/send_nagios_email.php