Building your own binary packages for Cumulus Linux (PowerPC)

February 7th, 2014

Cumulus Networks  is a new entrant in the network gear space. What separates them from other players is that they are not selling hardware but their own network focused Linux distribution called Cumulus Linux. Basically you buy a switch from one of their resellers or ODMs then pay Cumulus a yearly support license. There are a number of interesting things you can do like run your own code on the switch as well as use common Linux commands to configure the switch e.g. brctl, ports are exposed as Linux network interfaces etc.

One of the first things we ended up doing is installing Ganglia agent so that we can monitor what's going on on the switch. Cumulus switch we had was running a PowerPC based control plane so that made things a bit tricky since we couldn't use any of the amd64 built packages. One way to build PowerPC packages would be to get an old PowerPC based Mac and install Linux on it. Unfortunately that seemed like a lot of work and overkill. I realized we could just use Qemu which is an Open Source machine emulator so I could run PowerPC machine on my own laptop :-). Quickest way to get up and running is as follows.

On Ubuntu you will need to install following packages

apt-get install qemu-system-ppc openbios-ppc qemu-utils

Warning: Under at least Ubuntu 13.10 openbios-ppc doesn't seem to work well. If you get a blank yellow screen after you start the install you will need to get openbios from other places e.g. https://github.com/qemu/qemu/tree/master/pc-bios

Once you get those you will need to download Debian Squeeze for PowerPC. You will need to download

  • vmlinux
  • initrd.gz

from

http://ftp.debian.org/debian/dists/squeeze/main/installer-powerpc/current/images/powerpc64/netboot/

as well as the netboot image e.g.

http://cdimage.debian.org/cdimage/archive/6.0.8/powerpc/iso-cd/debian-6.0.8-powerpc-netinst.iso

Reason why you need initrd.gz and vmlinux is that if you try to do an install straight off the CD-ROM your install will hang here

Power PC install QEMU

Power PC install QEMU

Once you have those pieces initiate the install with

qemu-img create -f qcow2 squeeze-powerpc.img 10G
sudo qemu-system-ppc -m 256 -kernel vmlinux \
 -cdrom debian-6.0.8-powerpc-netinst.iso \
          -initrd initrd.gz -hda squeeze-powerpc.img -boot d -append "root=/dev/ram" \
          -net nic,macaddr=00:16:3e:00:00:02 -net tap

Now follow the installation process as you would if you were installing Debian or Ubuntu from scratch. When you are done with the install shut down the emulator. Now to invoke your PowerPC emulator execute

sudo qemu-system-ppc -m 256 -hda squeeze-powerpc.img  \
           -net nic,macaddr=00:16:3e:00:00:02 -net tap

Congratulations you are done. What you end up with is this

root@debian:~# cat /proc/cpuinfo 
processor    : 0
cpu        : 740/750
temperature     : 62-64 C (uncalibrated)
revision    : 3.1 (pvr 0008 0301)
bogomips    : 33.14
timebase    : 16570400
platform    : PowerMac
model        : Power Macintosh
machine        : Power Macintosh
motherboard    : AAPL,PowerMac G3 MacRISC
detected as    : 49 (PowerMac G3 (Silk))
pmac flags    : 00000000
pmac-generation    : OldWorld
Memory        : 256 MB

Running Arista vEOS under Linux KVM

January 10th, 2014

At my current job we run a lot of Arista gear. They are great little boxes. You can also run Ganglia on them :-) since they are basically Fedora Core 14 OS with some Arista proprietary sauce. You can find Arista specific Ganglia gmetric scripts here

https://github.com/ganglia/gmetric/tree/master/arista

On occasion I have wanted to test some things and Arista offers VM images you can run on your choice of virtualization. You can find more details here

https://eos.aristanetworks.com/2011/11/running-eos-in-a-vm/

I use KVM on my Ubuntu laptop and although booting the imaged worked I could not SSH into vEOS from my laptop. After a bit of testing I discovered that Arista's document misses a very important option ie.

-net tap

So full invocation is really

kvm -cdrom Aboot-veos-2.0.8.iso -boot d -hda EOS-4.12.3-veos.vmdk -usb -m 1024 \
    -net nic,macaddr=52:54:00:01:02:03,model=e1000 \
    -net nic,macaddr=52:54:00:01:02:04,model=e1000 \
    -net nic,macaddr=52:54:00:01:02:05,model=e1000 \
    -net nic,macaddr=52:54:00:01:02:06,model=e1000 \
    -net nic,macaddr=52:54:00:01:02:07,model=e1000 \
    -net tap
After that I was able to configure vlan 1 to e.g. 192.168.122.2
Log in into the console you just fired up and type
localhost#configure 
localhost(config)#interface vlan 1
localhost(config-if-Vl1)#ip address 192.168.122.2/24
localhost(config)#username admin secret 0 secret

You also want to set the password e.g. here I set it to secret and voila you can now SSH into 192.168.122.2. If you have too many SSH private keys loaded log in may not work so turn of public key authentication e.g.

ssh -o PubkeyAuthentication=no admin@192.168.122.2
Only note may be that if you just install libvirt /etc/qemu-ifup doesn't quite work since it determines which bridge to connect to based on the default route To "fix" that add
switch="virbr0"
Just above this section in /etc/qemu-ifup
# only add the interface to default-route bridge if we
# have such interface (with default route) and if that
# only add the interface to default-route bridge if we
# have such interface (with default route) and if that
# interface is actually a bridge.
# It is possible to have several default routes too
for br in $switch; do
    if [ -d /sys/class/net/$br/bridge/. ]; then
        if [ -n "$ip" ]; then
          ip link set "$1" master "$br"
        else
          brctl addif $br "$1"
        fi
        exit    # exit with status of the previous command
    fi
done

 

Bring your own device cell service / VoIP

December 25th, 2012

Recently I had to get my own mobile phone service and decided to forgo the standard post-paid cell service and go prepaid. Decision was largely cost based since I already had my own GSM phone and planned to buy a Nexus 4. I did quite a bit of research and ended up with Straight Talk service

https://www.straighttalk.com/

Straight Talk is a MVNO (Mobile virtual network operator) that leases network capacity from T-Mobile USA and AT&T.  To sign up you either order a SIM from their web site or you can pick up starter package at Walmart. I did the latter option. In the package they provided me with 2 different mini-SIM cards and a micro-SIM card. SIM cards are really T-Mobile and AT&T unbranded SIM cards. Pick the card that is supported by the phone e.g. if it's a locked phone like AT&T use the AT&T SIM, change some of the phone settings (APN) and off you go. Quality of the signal is the same as if you used T-Mobile or AT&T. I  picked unlimited everything plan for $45/month with a T-Mobile SIM. If you sign up for auto refill they cut it down to $42.50. Drawback is lack of international roaming and iffy customer service ie. hold times can be 30-40 minutes.

Another option I considered was T-Mobile's prepaid service called Go Smart which is similarly priced

https://www.gosmartmobile.com/

I decided against it since cost was similar but with Straight Talk I have the option to switch to AT&T if I ever find T-Mobile coverage inadequate. That said Go Smart does have a wider array of calling plans so it may still be a good choice.

While we are at it I can also recommend an inexpensive VoIP service called GalaxyVoice. I use their free-tier which gives you up to 60 minutes of outgoing calls a month and all I pay is for taxes and 911 compliance ~ $3/month. You just need to pay the signup cost of $25 and get your own SIP device.

An extra bonus is that their web site is fairly unsophisticated and easy to automate for certain things :-) e.g. forwarding my home phone calls to my cell phone

Mockupdata sample data creator

November 25th, 2012

My friend Igor Ješe who is an all around awesome guy recently wrote a new piece of commercial software called MockupData that allows you to create realistic sample data you can use to test your applications. Without further ado you can read the press release about MockupData here and newsletter announcement. If you have a need for such a tool I would highly recommend Igor's software.

This is not Igor's first product. Prior to MockupData he wrote a tool called Mockupscreens which let's you create mockups easily.

Monitoring health of Dell/LSI RAID arrays with Ganglia

September 28th, 2012

I have couple hundred Dell systems with LSI RAID arrays however we lacked hardware monitoring which would occasionally result in situations where there would be multiple disk failures that would not get caught. Some time ago I read this post about using MegaCli to monitor Dell's RAID controller

http://timjacobs.blogspot.com/2008/05/installing-lsi-logic-raid-monitoring.html

One thing I did not like about this approach is that it may generate too many e-mails as it will send e-mails every hour until the disk has been fixed. Instead I used Ganglia with Nagios to provide me with similar type of functionality.

Create a file called analysis.awk with following content

    /Device Id/ { counter += 1; device[counter] = $3 }
    /Firmware state/ { state_drive[counter] = $3 }
    /Inquiry/ { name_drive[counter] = $3 " " $4 " " $5 " " $6 }
    END {
    for (i=1; i<=counter; i+=1) printf ( "Device %02d (%s) status is: %s\n", device[i], name_drive[i], state_drive[i]); 
    }

Get MegaCli utilities from e.g. RPMFind. Copy following BASH script into a file and run into from a cron at frequency you need e.g. every 30 minutes, hour etc.

#!/bin/sh

GMETRIC_BIN="/usr/bin/gmetric -d 7200 " 

MEGACLI_DIR="/opt/MegaRAID"

MEGACLI_PATH="/opt/MegaRAID/MegaCli/MegaCli64"

BAD_DISKS=`$MEGACLI_PATH -PDList -aALL | awk -f ${MEGACLI_DIR}/analysis.awk | grep -Ev "*: Online" | wc -l` 

if [ $BAD_DISKS -eq 0 ]; then
    STATUS="All RAID Arrays healthy"
else
    STATUS=`$MEGACLI_PATH -PDList -aALL | awk -f ${MEGACLI_DIR}/analysis.awk | grep -Ev "*: Online"` 
fi

$GMETRIC_BIN -t uint16 -n failed_unconfigured_disks -v $BAD_DISKS -u disks
$GMETRIC_BIN -t string -n raid_array_status -T "Raid Array Status" -v "$STATUS"

 

This will create two different Ganglia metrics. One is number of failed or unconfigured disks and another one is just a string value that gives you details on the failure e.g. Disk 4 failed. Besides being able to alert on this metric it also gives me a valuable data point I can use to correlate node behavior ie. when disk failed load on the machine went up.

If you are using Ganglia with Nagios integration you have two different options on how you want to alert.

1. Create a separate check for every host you want monitored e.g.

define command{
        command_name    check_ganglia_metric
        command_line    /bin/sh /var/www/html/ganglia/check_ganglia_metric.sh host=$HOSTADDRESS$ metric_name=$ARG1$ operator=$ARG2$ critical_value=$ARG3$
        }

define service{
        use                   generic-service
        host_name             server1
        service_description   Failed/unconfigured RAID disk(s)
        check_command         check_ganglia_metric!failed_unconfigured_disks!more!0.1
    }

 

2. Create a single check that gives you a single alert if at least one machine has a bad disk (how I do it :-)). For this purpose I'm utilizing the check_host_regex which allows me to specify a regular expression of matching hosts. In my case I check every single host. If a host doesn't have the failed_disks metric I assume it doesn't have it and I "ignore" matches. My config is similar to this

define command{
        command_name    check_host_regex_ignore_unknowns
        command_line    /bin/sh /etc/icinga/objects/check_host_regex.sh hreg=$ARG1$ checks=$ARG2$ ignore_unknowns=1
        }

define service{
        use                             generic-service
        host_name                       server2
        service_description             Failed disk - RAID array
        check_command                   check_host_regex_ignore_unknowns!'.*'!failed_disks,more,0.5
    }

Which will give you something like this

# Services OK = 236, CRIT/UNK = 2 : CRITICAL compute-4566.domain.com failed_disks = 1 disks, CRITICAL git-0341.domain.com failed_disks = 1 disks

 

My monitoring setup

September 1st, 2012

On Twitter Grig Gheorghiu posed a number of questions about monitoring tools and what he wants in a monitoring tool. This is my attempt at describing what my setup looks like or has looked like in the past.

1. Metrics acquisition / performance trending

I use Ganglia to collect all my metrics including string metrics. Base installation for Ganglia Gmond will give you over 100 metrics. There are a number of Python modules that are disabled by default like mySQL, Redis that you can easily enable and get more. If you need even more you can check out these to Github repositories.

Don't worry about sending too many metrics. I have hosts that send in excess of 1100 metrics per host. Ganglia can handle it so don't be shy :-). Also when I say all metrics go into Ganglia I mean EVERYTHING. If I want to alert on it it will be in Ganglia so I have things like these

  • NTP time offset
  • What version is particular key piece of software on e.g. deploy ID 123af58
  • Memory utilization/CPU utilization for key daemon processes
  • Number of failed disks in a RAID array
  • Application uptime
  • Etc.

2. Alerting

I use Nagios or Icinga for alerting. I don't really use any Nagios plugins as all the checks are driven by data coming out of Ganglia. I have written a post in the past about why you should use your trending data for alerting which you can read for some background. About a year ago Ganglia/Nagios integration has been added to Ganglia Web which makes a number of things much easier so for example I have

  • A single check that checks all hosts in the cluster for failed disk in a RAID array
  • A single check that checks whether time is within certain offset on all hosts ie. to make sure NTP is actually running
  • A single check that makes sure version of deployed code is the same everywhere
  • A single check for all file systems on a local system with each file system having their own thresholds
  • A single check for elevated rates of TCP errors - useful to get a quick idea if things are globally slow, affecting a certain set of hosts in a geographic area or an individual host

Beauty in having all of the metric data in Ganglia is that you can also get creative by writing custom checks that have your own user specified logic e.g.

  • Alert me only if 20% of my data nodes are down e.g. in architectures where you can withstand few nodes failing

In addition I recommend adding as much context to alerts as possible so that you get as much as information in alert as possible.

I also heavily utilize something I wrote called the alerting controller

https://github.com/vvuksan/alerting-controller

Which allows you easily enable/disable/schedule downtime for services in Nagios e.g. disable process alive check for configuration servlet on all hosts while we are doing an upgrade etc. In addition I have a tab with Naglite opened up most of the time to check on any outstanding alerts.

https://github.com/saz/Naglite3

3. Notifications

Beyond just alerting I am always interested to see what is happening with the infrastructure so we can act proactively. For that purpose I use IRC and a modified version of NagiosBot which sends echoes following things to the channel

  • Nagios alerts (same script that adds context to alerts above) - helpful for quick team coordination
  • Dynect DNS zone changes - those may be "invisible" so good idea to track
  • Zendesk tickets - anyone can handle support requests
  • Twitter mentions of a particular keyword
  • Application configuration changes e.g. new version of the code deployed or in progress
  • Severe Application errors - short summary of an error and which node it occured on

Closing

This is by no means an exhaustive list and this may not be the best way to do things but it does work for me.

WebOps Hackathon/(un)conference in Boston in October

August 20th, 2012

Few weeks ago I posed a question on Twitter whether there would be interest to organize a webops hackathon   I am close to getting confirmation for conference space at Microsoft NERD for October 11, 2012. Now comes the hard part :-). We need to figure out what format should hackathon and unconference take as well as what topics we should talk about. There is going to be a monitoring conference at the end of March spearheaded by Jason Dixon (@obfuscurity) so we may want to leave out monitoring topics.

In a nutshell we need to decide

  1. How much of the event should be a hackathon and how much should be an (un)conference
  2. If we want talks what topics should we seek talks for e.g. continuous integration, noSQL, site scalability, etc.
  3. Name of the event :-)

If you have any ideas send me an e-mail conference X vuksan.com (replace X with you know what).

Parsing JSON POST in PHP

June 15th, 2012

I have an application that uses HTTP POST to submit a JSON encoded array. Basically there are no variables that are being submitted, just JSON. This causes $_REQUEST and $_POST arrays to get messed up where "random" parts of the JSON will end up as the key and rest as the value. Instead what you need to do is get contents of the request as input then decode it   e.g.

    $input = file_get_contents('php://input');
    $my_array = json_decode( $input, TRUE);

PHP HTTP caching defaults

May 22nd, 2012

I have recently moved this blog to be hosted on Fastly, a CDN service with bunch of great features like dynamic content caching with instant purges. Fastly utilizes HTTP headers to determine what to cache as described in the Cache Control document. While configuring my service I noticed that my WordPress (origin server) kept returning HTTP headers like these

Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

I looked through the WordPress code and couldn't see where such value was set. After some internet searches I discovered that they are set using session-cache-limiter option in php.ini. In most distributions this defaults to nocache which ends up with above headers. You can read more on the cache-limiter options here.

http://www.php.net/manual/en/function.session-cache-limiter.php

What we need is session-cache-limiter = public results in headers like these

Expires: (sometime in the future, according session.cache_expire)
Cache-Control: public, max-age=(sometime in the future, according to session.cache_expire)
Last-Modified: (the timestamp of when the session was last saved)

e.g.

Expires: Tue, 22 May 2012 16:38:33 GMT
Cache-Control: public, max-age=10800
Last-Modified: Tue, 04 Oct 2005 00:55:59 GMT

If you want to adjust the max-age you can set cache-expire in php.ini e.g.

; http://php.net/session.cache-expire
session.cache_expire = 180

 

My programming language beat your honor roll language

April 10th, 2012

For a while I have been observing a tendency of technologists/engineers to describe technology in either black or white terms ie. X technology sucks, use Y technology. Most recent example is an article by someone going by the name of Eevee  in his PHP a fractal of bad design. It is a damning expose of PHP's failings/bad decisions/inconsistencies etc. Unfortunately as most articles of this type it involves a number of ad-hominem attacks like these

It’s so broken, but so lauded by every empowered amateur who’s yet to learn anything else, as to be maddening. It has paltry few redeeming qualities and I would prefer to forget it exists at all.

or

I assert that the following qualities are important for making a language productive and useful, and PHP violates them with wild abandon. If you can’t agree that these are crucial, well, I can’t imagine how we’ll ever agree on much.

This irritates me on many levels since it makes so many misguided assumptions e.g.

- Everyone's mind is the same therefore everyone should like or hate language X

Of course not. Your mind is different than my "defective" mind. I quite prefer writing in PHP. I have written/write code in Ruby/Python/Perl and PHP is my preferred language. That may change but at this point it's my preference. You may disagree with my choice and that's OK.

- Issues we are trying to solve are similar/identical and we have same resource constraints aka one-size fits all

Of course not. If most coders on my team are well versed with PHP and we have a tight schedule you bet we are most likely to choose PHP. Technical merits are not the only consideration to be taken. They are certainly important but they often pale in comparison to other considerations such as people and culture.

- PHP core developers are incompetent

I have on a number of occasions disagreed and been frustrated with decisions made by PHP core developers however I do assume that in most/all cases they are well intentioned and are making the best decision under available circumstances. PHP has been around for a long time so I imagine making major changes is tough and involve making significant tradeoffs. If those tradeoffs become a show stopper for me I'll use a different language.

Anyways the most bothersome part of the whole post is the technology "tribalism" which results in things like this (from Kibana README file)

Q: Why is this in PHP instead of Java, Ruby, etc?
A: Because PHP is what I know. The total PHP is less than 200 lines. If you want it in something else, it shouldn't be too hard to port it to your language of choice

That makes me pretty sad.