Archive for August, 2010

Slides from the Boston DevOps meetup

Wednesday, August 25th, 2010

Here are slides from the August 3rd, 2010 Boston DevOps meetup where Jeff Buchbinder and I spoke about deployment and other helpful hints

Slides have been slightly modified based on the feedback we received at the meetup. If you have any questions please post them in comments and I'll attempt to answer them.

Tunnel all your traffic on “hostile” networks with OpenVPN

Friday, August 20th, 2010

I am often on wireless networks that are unsecured ie. either don't use encryption or if they are I may not trust they will not tamper with my data (you never know). To protect my traffic on such networks I decided to tunnel nearly all my traffic through an OpenVPN server while I'm on such networks. I will show you how you can do it yourself on your Linux or Mac laptops. You should be able to do similar in Windows but it may be a bit more work on the client.

OpenVPN server setup

Set up OpenVPN on a network you trust e.g. home, work, cloud etc. You can either use Community Edition of OpenVPN which is free or you may want to pay OpenVPN money for their OpenVPN appliance package. I prefer using pfSense which is customized FreeBSD distribution geared for firewalls/routers with superb Web GUI. If you are gonna use the Community Edition follow the Quickstart guide.
One last step is to make sure that VPN network ie. is NATed e.g. on a Linux OpenVPN server you could do

iptables -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE

OpenVPN client setup

Configure OpenVPN client to connect to your OpenVPN server. You can find the client HOWTO here.
Make sure you can access your home/work network. This will in general provide you with "split-tunnel" access ie. only traffic intended for your home/work network will be tunneled through VPN and everything else will go the normal "insecure" way.

Tunnel all traffic

Update: Shame on me. Someone has already posted the directions on how to do this at

Thanks to @somic for pointing this out.

Tricky part in all this is that OpenVPN uses a simple TUN/TAP interface through which tunnels all the traffic. Temptation is to simply add an entry in the OpenVPN file that sets a default route through OpenVPN. This will likely fail as you will now have competing default routes. Instead what you need to do is add a route to your VPN server that uses the wireless networks default gateway and make your VPN interface the default route. This way all the traffic goes into the VPN interface and OpenVPN takes care of tunneling it through.
For this you will need to configure an external script that fires off once VPN tunnel is up. To enable post-up script put following two lines in your ovpn file
script-security 3 system
up /usr/local/bin/

Your would look something like this. Please change the VPN_SERVER_IP variable to the IP of your OpenVPN server.

# Note the wireless network default gateway
DEFAULT_GATEWAY=`netstat -nr | grep ^ | awk '{ print $2 }'`
# Find out what's the IP on the
VPN_GATEWAY=`netstat -nr | grep tun | grep -v | awk '{ print $2 }' | sort | uniq`
VPN_GATEWAY=`ifconfig  | grep 172.16 | cut -f3 -d:  | cut -f1 -d" "`
sudo /sbin/route del default
sudo /sbin/route add -host $VPN_SERVER_IP gw $DEFAULT_GATEWAY
# Don't tunnel traffic to since it's already SSLized
sudo /sbin/route add -host gw $DEFAULT_GATEWAY
sudo /sbin/route add default gw $VPN_GATEWAY

This script was tested under Ubuntu Linux but should work the same under Mac OS X. On Windows you may need to use PowerShell or use Cygwin.

Tunneling traffic for specific IPs

If you only wish to tunnel traffic for particular set of IPs you only need to add those routes to your ovpn file e.g.

You do NOT need to go through the excercise of setting up a script.
If you are looking for other OpenVPN guides Sam Johnston has a OpenVPN guide on howto set up OpenVPN in a VPS

Skipping MySQL replication errors

Thursday, August 19th, 2010

I was talking to my buddy Jeff Buchbinder and he mentioned that he recently added following to mySQL in order to reduce mySQL replication breakages


What this does is not stop replication in case following errors are encountered


Message: Table '%s' already exists


Message: Unknown table '%s'


Message: Server shutdown in progress

Error: 1062 SQLSTATE: 23000 (ER_DUP_ENTRY)

Message: Duplicate entry '%s' for key %d

Error: 1146 SQLSTATE: 42S02 (ER_NO_SUCH_TABLE)

Message: Table '%s.%s' doesn't exist

This will avoid the very common primary key collisions and "temporary tables aren't there" problems. Writing this down for posterity. Use with caution.

Marius Ducea has a post about it as well

PHP 5.3 name spaces separator

Monday, August 16th, 2010

I am posting this to help others that may encounter a similar problem.

I have been doing some PHP development recently using Predis, a PHP Redis library. While instantiating the Redis\Client object I get

Warning: Unexpected character in input:  '\' (ASCII=92) state=1 in .....

Problem was explained in this issue

If you are still running on PHP 5.2 you should use the backported version of Predis and not the mainline library which targets only PHP >= 5.3 (the backslash is the namespace separator in PHP 5.3).

More discussion of this change can be found here.

Deployment rollback

Thursday, August 12th, 2010

This is a question that often comes up in deployment discussion. How do you rollback in case of a "bad" deploy ? Bad deploy can be any of the following

  • Site completely broken
  • Significant performance degradation
  • Key feature(s) broken

There are obviously a number of ways to deal with this issue. You could put up a notice on the site that x and y feature is broken while you work to fix it. Same with performance degradation. Let's however deal with rollback ie. you decided (determined by a number of different factors) that the stuff you just deployed is broken and you should roll back to a previous last know version. In such a case you would

  • Undo any configuration changes you may have applied (often none)
  • Deploy last known good version that worked. This is one of the reasons why I prefer using labelled binary packages. I simply instruct the deployment tool to install version 1.5.2 which was last good version and off we go.

The only caveat are database changes. In general you can't easily undo DB changes especially in the situations where you discover a deployment problem couple hours after deployment has taken place since by then users may have added new posts, changed their profiles etc. It would be a major effort to undo all DB changes, evaluate newly added data and whether it needs to be changed. That said DB changes are usually not a problem if you follow these easy steps

  1. Don't do any column drops immediately after the release. You can do those in QA but in production those can wait. In most cases they only take up space. I have heard of places that would first zero out then drop "unused" columns once a quarter or so.
  2. Related to 1. never ever use SELECT * since if you drop or add a column your code may break during roll back
  3. If there are data changes you have to do ie. update carrier set name="AT&T" where name="Cingular", have the reverse SQL statement ready as the insurance policy. Those are quite easy to implement.
  4. You don't have to worry about added tables since older version will not use them.
  5. You don't have to worry about added columns provided you don't do 2. and have not placed constraints ie. NOT NULL. In that case you may need to adjust those or drop them during rollback.

The wildcard in all this is added or removed constraints ie. new foreign keys. There is no single solution for this one. Perhaps the right policy is to discuss constraints prior to deployment and have a plan ready on what to do. Good luck.