Archive for September, 2008

Smoothe As Silk – Well Almost…

// September 30th, 2008 // No Comments » // amazon ec2, chesscube, deployment, Openfire, programming, system admin


Launching a new version of your site is always stressful. Launching a site that spans multiple servers can be a nightmare. With a sound plan and practise, however, it can be less difficult than it is usually.

ChessCube Version 3 was launched on Monday and the release went relatively smoothly considering how huge the changes were to both the interface and the underlying architecture. In the past, we were a little too trigger happy in releasing a new version and failed to plan and test the release properly. For this release we had a target of 2 hours of downtime while we upgraded and uploaded software, changed configuration files and deployed the new website files.The changes to version 3, as I have already mentioned, were extensive. The main areas that were changed were:

  1. Improved interface and user experience. We wanted this version to appear to be more simple yet have more features. Instead of packing in thousands of buttons and labels everywhere, we worked hard to ensure that it all flowed correctly and that there wasn’t information overload. This is a lot more difficult than it sounds and the best way to go about it, is to forget as much as possible of the existing interface and try to come up with new and fresh ideas.
  2. A completely rebuilt and redesigned Paper. Paper is the name we have given to the component that sits alongside the board while playing or viewing a game and tracks the moves and chat between the players. You could also call it the Notation View or the Game History Panel. Due to the way Paper is implemented we had to reorganize popup windows and make sure that it resized and was positioned correctly.
  3. Multiple game server support. Clients would have to support and manage multiple connections to game servers and also work optimally with those game servers.
  4. Improved moderation & reporting tools. These changes were implemented to improve the moderators’ abilities to monitor users on the site and ensure that everything is running smoothly.

Other factors that made this release tricky were the upgrading of the version of Openfire that we were using and also having to move software onto more powerful servers, not to mention coordinate the rollout of our Amazon EC2 infrastructure.

The success of the release process was partly due to a few factors:

Testing exhaustively

I can’t stress this enough. If you build in 10% of the allotted project duration into your schedule for testing – double it. Get an independent tester – not a programmer or the boss. The boss can cause stress for the programmers when he finds a bug.

Simulating our live environment

We used to stage all our software on a little machine in our office and access the database on that machine using fancy GUI tools. If you live in South Africa and you have an average connection to the Internet, this is a mistake. The turnaround time for problems on a machine hosted internationally can cause a lot of stress. We commissioned a brand new server to go live on for our new software and set up our entire live architecture using this server, a local database and two EC2 game servers with the plan to rollout two more on launch day.

Setting up the staging environment a week before going live

This way all issues and their solutions you encounter while setting up the infrastructure and getting it running are still fresh in your system administrators’ and developers’ minds. There is nothing worse than forgetting how you solved a problem 6 months prior to release under the pressure of your system being down. We also planned to convert our staging environment to the live environment when the release launched. This meant downtime was kept to a minimum as all we had to do was change a few configuration files and point the server away from the staging database to the live database.

Making use of configuration files and logging wherever possible in our code

This is probably a moot point, but if you have a good set of external configuration files you won’t need to keep uploading new builds of your software if you have forgotten to change a server url.

Extensive log statements help to identify problems very quickly should your system stop functioning for whatever reason. Remember: Absence of logs means you are missing a log4j.xml somewhere (I learnt this one the hard way:) )

Holding on tight

You can’t plan for every eventuality and you can’t hold off your release until you feel the software is perfect. Sometimes you need to just release and see what happens. 15 minutes after releasing we realized we had a few bugs which were serious enough to warrant emergency bug fixes and we had a new version of the client software out 10 minutes later. This can be disruptive to your users but its better to disrupt early and fix the problems than handle all the bug reports and user complaints later.

Most of these tips seem obvious but they definitely played a huge roll in negating major disasters. Just remember, the hard work only really starts after you have launched and never deploy on a Friday.

Thanks to my colleagues: Dave, Tracy, James and Gideon for creating, what I think, is the best chess playing site out there and thanks to Bryan and his testing team for helping us find and swat all the bugs. Thanks to Margaret for her amazing GUI designs and thanks to Mark and the investors for giving us the opportunity to work on such a cool project.

Helper Objects

// September 28th, 2008 // No Comments » // coding style, mac osx, objective-c, programming

I think this is a brilliant comparison between using Helper Objects and Sub-Classing:

Once upon a time (before Baywatch), there was a man with no name. Knight Industries decided that if this man were given guns and wheels and booster rockets, he would be the perfect crime-fighting tool. First, they thought, “Let’s subclass him and override everything we need to add the guns and wheels and booster rockets.” The problem was that to subclass Michael Knight, you would need to know an awful lot about his guts so that you could wire them to guns and booster rockets. So instead, they created a helper object, the Knight Industries 2000, or “KITT the super car.”

Note how this is different from the RoboCop approach. RoboCop was a man subclassed and extended. The whole RoboCop project involved dozens of surgeons who extended the man’s brain into a fighting machine. This is the approach taken with many object-oriented frameworks.

While approaching the perimeter of an arms dealer’s compound, Michael Knight would speak to KITT over his watch-radio. “KITT,” he would say, “I need to get to the other side of that wall.” KITT would then blast a big hole in the wall with a small rocket. After destroying the wall, Kitt would return control to Michael, who would stroll through the rubble.

Solving The Scaling Riddle

// September 27th, 2008 // 1 Comment » // amazon ec2, amazon s3, chesscube, Openfire, programming, XMPP

Monday is a very big day for ChessCube. We are launching version 3 of our online chess playing client. Along with a redesigned interface, we have made some significant changes under the hood.

Version 2 of ChessCube was suffering under immense load created by the increasing popularity of the site. Under this load we were experiencing messages getting lost, very long login times, client interface crashes and, the worst of all, dreaded lag spikes. Lag can be described as the amount of time a message takes to go from the Flash client sitting in the browser to the server. A lag spike is when the overall lag in the system jumps for all users that are on the system. After hunting down the cause of these lag spikes we came to the conclusion that the system was unable to cope with the sheer amount of messages being sent and received between clients and the server. Putting our server software on a larger server would, in light of the steady growth of the site, just buy us a few more months. So the decision was taken to cluster the server software.

The chess playing component of ChessCube, we call it Chat internally, uses a protocol called XMPP. Now before your eyes glaze over and you go back to checking your mail, let me explain very simply what XMPP is. If you’ve ever used Google Talk or Facebook Chat, you would have used XMPP. It can simply be explained as a set of rules used to allow for chatting between people connected to a central server. For ChessCube, we chose Openfire as our XMPP server, for the reasons that its Java-based (a language the team predominantly uses), is easily extensible, due to its comprehensive plugin framework, and is Open Source.

Openfire is great for supporting a small chat community, but as soon as you need to scale above 5000 simultaneously online users it becomes very slow. This is where clustering comes in. Clustering is a term used to refer to a group of computers concurrently working together to spread load across them, thereby improving performance and in some architectures, removing a single point of failure. The company that supports Openfire does offer a clustering plugin but its prohibitively expensive – charges are on a per-user-basis rather than a per-server-basis.

In comes our trusty homemade architecture. Since each game of chess in Chat is played in a separate room we could distribute these rooms off the main Openfire server. So now we have a main Openfire server to handle all the stuff related to presence and chat; and smaller game servers that handle games. We can now have multiple game servers all communicating with the Openfire server about the status of games being played on them and the Openfire server in turn can distribute the games evenly over the game servers.

Distributing the load across game servers is handled with Amazon S3. Each game server writes its status to S3 and the Openfire server polls S3 to see which game servers are available and how much load each server is under. The Openfire server can then send clients to whichever server it feels is under the least amount of load. We can also do cool tricks with routing clients to servers that are nearest to them geographically. E.g. If two players from Europe want to play a game we can put them on a server in our German data centre. Lag is minimized and everybody is happy.

We have also created a customized instance image for game servers on Amazon EC2. Under extraordinary load we are able to bring new game servers online and running games in a matter of minutes.

This version of Chat goes live on Monday with four game servers, running on Amazon EC2. Hope to see you there.

Working Remotely – A Programmer’s Guide

// September 18th, 2008 // 2 Comments » // amazon s3, fourhourworkweek, programming, rsync, scp, ssh, working remotely

Being confined to working in an office is not ideal. Especially if you want to be more creative and have more freedom to spread your time out during the day. I have been working away from the office on Thursdays and Fridays. By “away from the office”, I mean I have been working from home.

The reasons for working from home or remotely are pretty obvious. Firstly, you don’t have to commute to work. This saves me around 1 hour of driving in the car not too mention a whole bunch of money every month in fuel costs. The other cool thing about working from home is that you never feel rushed to finish something, you can sit back relax and get it done. Don’t get me wrong though – its not about shirking your work or being lazy. It just means you don’t feel as though you have to finish whatever you are doing so that you can hop in your car and rush home at the end of the day.

There are other good things that come from being away from the office. I have found new energy and creativity in my work because I feel more relaxed. I can also go for a run/cycle during the day if I feel I need to clear the cobwebs or if I am stuck on a problem. I also get to spend more time with my girlfriend – who works from home as well.

Today I decided to go one step further and got on a plane to Johannesburg to visit my parents for my dad’s birthday. After convincing the boss that my work would not be affected, I had to make sure I had the right tools and environment so that I was not left stranded without an Internet connection – the most dreaded of scenarios in my line of work. Here are some of tips for working remotely – especially if you have a job where you rely on the Internet to get your work done.

1. Travelling

If you are travelling somewhere, try to make sure you waste as little of the middle of the day as possible. I flew at 6:15am which meant I arrived in Johannesburg at 8:10am. I could then have a shower when I got to my mom and dad’s place and start working. If you are travelling further, try to fly in the evenings.

2. Equipment

Equipment is important. If you are not prepared you are asking for trouble. I use a Mac – but I also recommend using Linux. Why not Microsoft Windows you ask? Well, I do a lot of work on remote servers, uploading and downloading files, copying between remote servers and zipping and unzipping files. This is a real pain if you don’t have a bash terminal. You can use a combination of Putty and WinSCP on Windows. If you take the time to learn bash commands you will save time in the long run. More on this later.

The other really essential device is a 3G equipped cellphone with a medium-sized data plan. (This obviously depends on how long you are going away for and how much data you expect to use.) Don’t be fooled, if you are going somewhere that has Internet connectivity, you may find that its either too slow to be usable or just downright unreliable. HSDPA cellphones are preferable as they have faster connection speeds.

Other things I like to take with me are a good set of earphones and a mouse. The earphones are a good idea in case you have to work in a noisy environment – airports, hotel lobbies etc. I take a mouse because I tend to get neck pain if I have to use the touchpad on my Mac for prolonged periods. I like the Logitech VX Revolution mouse. It has a little compartment that you can store the cordless receiver inside and is very comfortable.



3. Software, Tools & Tips

Obviously the software everyone uses is different depending on your job. I am a programmer so I have to be able to get updates to the codebase from my teammates and be able to update them with my changes. We use Subversion for this and there is a nifty little feature in the Subclipse plugin for Flex Builder and Eclipse whereby you can relocate your repository. This means that I can use the external subversion URL when I am away from the office and the internal subversion URL when I am in the office.

I also have to be able to move files around e.g. copying new plugins of our server software to our test server. To copy files I use the scp command in terminal. So if I want to copy a file to our German server I type something like:

scp filename.txt root@germanserver.chesscube.com:

Copying large files repetitively with scp can be onerous however, not to mention expensive, so I use another command called rsync. Rsync works much the same way as scp:

rsync filename.txt root@germanserver.chesscube.com:

The cool thing about rsync is that it synchronizes the files or directories you are overwriting on the remote machine and therefore only copies the differences between the two files making the transfer time faster and the transfer size smaller.

These are obviously very simple examples. If you check out the links you can learn how to do more complicated tasks. Like copy a file to a specific directory or get more detailed information while copying.

How to use rsync.
How to use scp.

Another cool tip if you need to get a large file to your boss or a client or whoever, is to use Amazon’s S3 Service. If you have an Amazon account, this is very simple. Visit http://aws.amazon.com and sign up for their S3 service. S3 stands for Simple Storage Service and its a very easy way to upload and store files online, as well as make them accessible from the Internet. Here is a whirlwind guide on how to get going:

  1. Sign up for the Amazon S3 Service
  2. Install a Firefox plugin called S3 Organizer – if you don’t have Firefox you can get it here
  3. Open S3 Organizer from the Tools menu in Firefox and click on Manage Accounts
  4. Take the access key and secret key you received from Amazon after registration and put it in the corresponding fields in S3 Organizer
  5. On the right-hand-side of S3 Organizer click on the Create Directory button – the directory name must be unique for the whole of Amazon S3, not just for you. So use something obscure if you need to.
  6. Select the directory you have just created and click Edit ACL. You can then share the directory you have just created based on the options provided. Sharing by e-mail address is the easiest.
  7. Now you can copy files from the left hand window – which is your local computer to the directory you just created in Amazon S3.
  8. Finally, for your boss, client, colleague to access the file you just uploaded – right click the file in the right hand window and select Copy URL to Clipboard. You can then paste the URL into an e-mail and they will be able to download it from their web browser.

Bear in mind that Amazon does charge you for the amount of data you upload and download from S3, but its minimal. Around US$0.12 per gigabyte.

4. Instant Messenger Advice

Using an Instant Messenger is great if you need to interact with your colleagues. The are however, downsides. It takes a lot longer to explain a concept or ask a complicated question if you have to type it out. I try to minimize the amount of questions I ask and really try to solve the problem myself before asking a colleague.

Another downside, is that it can be distracting if you are grappling with a difficult problem and your Instant Messenger starts going berserk with incoming messages and pinging noises. A simple fix for this is not to stay logged in all the time and only log in at specific periods during the day.

All in all, working remotely seems to be working out well. It is taking some getting used to, but I think the benefits are worth it in the end. I have managed to find time to start this blog and also have a more regular exercise regimen. I also don’t feel stressed after spending time in traffic. If you are having trouble working out a strategy to get your boss to allow you work from home, I highly recommend reading the chapter on it in the book The Four Hour Work.

Downloading

// September 14th, 2008 // 1 Comment » // iphone, programming, ssh, Telkom

Downloading the iPhone SDK is the simple part in getting started with creating applications for the iPhone. Accessing the iPhone Developer Centre is the complicated part. Especially if you are from South Africa. You can log into the centre but from there you are unable to access any of the documentation. Every time I clicked on a URL I would be redirected to a log in screen which would never let me in. No errors. Nothing. After some digging around, it turns out that the way Apple authenticates access to web pages doesn’t play nicely with the SAIX proxy. Thanks very much Telkom! I finally managed to get in using a combination of clearing my browser cache completely and using an SSH Tunnel through my company’s Verizon server. Happiness is…

That’s only half the story though. I didn’t really want to go through this whole rigmarole every time I wanted to get some information about a framework or a class. So I needed to find a way to access the developer documentation offline. In comes XCode’s documentation manager to the rescue. Err. Well sort of. It is possible to subscribe to the iPhone Developer Documentation in XCode, and it will download the documentation to your local machine for you, but its slow and it doesn’t resume.

So after 3 attempts and much wastage of precious bandwidth I turned back to Google search terms and in an “eureka moment”, I found a post on a blog, on how to use NetNewsWire to download the darn documentation and subsequently install it. I just hope Apple doesn’t release a new version of the documentation tomorrow. Good things come to those who bash their heads on the keyboard at consistent times during the day.

Oh and by the way, don’t bother trying to contact Apple for help. Their auto-responder works really well but their support staff obviously haven’t been outsourced to India yet. I have been waiting a week for a reply to all my e-mails, concerning the issues I have been having accessing their website. Maybe there is a trick to it.

ChessCube Award

// September 11th, 2008 // No Comments » // Uncategorized

My company, ChessCube, has been nominated for an Adobe MAX award. The Adobe site sums up what Adobe MAX is all about:

“MAX is an experience unlike any other — an opportunity to connect with thousands of designers, developers, partners, executives, and Adobe staff for education, inspiration, and community. MAX 2008/2009 will be held in San Francisco, Milan, and Tokyo. Be sure to mark your calendar for this important global event.”

We have made it to the second round of judging for two of our applications: ChessCube Chat & ChessCube Cinema. I am really excited about this. They had over 700 entries and could be one of the 15 finalists that get to present their creations at the conference.

On a more serious note though, our database server has been giving us major problems. One of our database tables got corrupted on Monday and we only managed to figure out what the problem was today. The lesson to be learned here is: don’t always try and fix something yourself. Bashing your head against a brick wall won’t solve anything. Rather call an expert you can trust and get them to take a look, they will figure it out faster and minimize the mess you have to clean up after they have fixed the problem. I had fiddled with the problem for too long and we have 4 days of database queries that need to be recovered. Fortunately, our expert knows how to fix it and as I write this post, I am waiting for him to give me the go ahead to fire our chat server back up.

Hold thumbs for us at Adobe MAX. Maybe I will on my way to San Francisco to collect an award in the next 2 months.

Bits, Bytes & The iPhone SDK

// September 5th, 2008 // No Comments » // iphone, mac osx, objective-c, programming

So I’ve been toying with the idea of building an application for the iPhone and yesterday I finally decided to download the SDK. After agreeing to give my soul to Apple as stipulated in their Terms & Conditions and 8 hours of downloading later I finally got the full SDK.

I have been trolling through their documentation and it looks like I will have to get my hands dirty in Objective-C. Which, from what I can tell, is a slightly cleaner version of C++ specifically designed for Mac OSX. In case you didn’t know, the iPhone runs on a heavily modified version of the Mac OSX kernel. This has its benefits in that by learning to program for the iPhone I will also slowly become accustomed to writing programs for Mac.


The easiest way to learn a programming language is to remember that they are all essentially the same. Don’t get confused between a programming language and a markup language. I can’t tell you how many times I have read on programmer resumés that HTML is a programming language. HTML is a markup language – a way of describing visual data in a hierarchical textual fashion. A programming language is a set of statements that can be executed sequentially and all procedural programming languages follow a similar set of rules and contain 3 specific constructs:

  1. Variables
  2. Conditional statements
  3. And Loops

With these three constructs it is possible, with great pain I might add, to build any program.

Of course a programming language is of no use if there is no way of interpreting it. Way back in the days before Microsoft and Apple, programs were created on punch cards – pieces of cardboard with rows of holes punched in them. Every hole represented a 1 and every non-hole represented a 0. The punch cards were fed through the computer which would read the rows sequentially and then execute them. Punch cards are so old they were first used back in the 18th century to control weaving looms and were used up until the last century still.

So-called ‘modern’ computers work in much the same way. High-level programming languages such as C++ and Java are compiled by another piece of software, called a compiler and converted to what is known as Assembly Language. Assembly language is a low level language used to program hardware, such as your computer’s CPU. Assembly code is then converted to what is known as Object Code. Object Code is the final sequence of 1s and 0s – the holes and non-holes on punch cards.

Every single computer program you use is constructed in this way. So next time you get the dreaded ‘blue-screen-of-death’ or your e-mail application crashes, just remember how many 1s and 0s are moving through the system. A modern car consists of approximately 12000 parts. Microsoft Windows XP consists of approximately 1 billion lines of code and it works 99% of the time. That is absolutely incredible!

I guess what I am trying to say here is that learning how to program within the iPhone SDK and learning Objective-C is trivial compared to building an operating system like Mac OSX or programming a computer’s CPU. It’s made even easier if we remember that all procedural programming languages are constructed out of similar building blocks and that if we learn and understand the basics of a subject we can go on to learn just about anything.

Exhilirating & Exhausting

// September 4th, 2008 // No Comments » // cape town, mountain biking, night riding, table mountain

I went on a night cycle last night. We left from Valley Road in Hout Bay and crossed over Main up onto the dirt road that goes behind the houses on Bokkemanskloof. I went with Peter and Lionel. It hurt like hell! After 30 minutes my lungs were aching and I was wheezing and straining like an overweight chainsmoker on a flight of stairs, except I am neither overweight nor a chainsmoker. I think I have developed a little bit of asthma over the passed couple of years. I definitely need to ride more.

cape town at night
In any case after an hour we were at the top of Constantia Nek. Peter and Lionel were riding ahead and I was lagging behind. It’s easier after resigning yourself to letting everyone else ride away from you and you can ride at your own pace. We were heading up to the dams on top of Table Mountain. The view was superb, my lungs were opening up and I was feeling stronger.

At around 400m up I turned my lights off. The light from the Cape Flats was so bright I could see the grey, concrete road snaking up ahead of me. This is where the mountain starts to kick up. The gradient was averaging around 15% now, my heartrate 180bpm. Lionel and Peter were out of sight.

heart_rate_over_distanceelevation_over_distance

I wasn’t going to give up so easily. If I had stopped, the road would have been too steep to get going again. The climb increased to 17, 18, 19 percent. This is the steepest part of the climb. Its like having a concrete wall in front of you. My heartrate was pushing 193bpm. The highest its been in a long time and my maximum. I was feeling strong but I knew I could only keep this up for a few minutes longer.

Thankfully the road started to drop in gradient. I had gotten to the metal grating that covers a chasm running down the side of the mountain. Peter and Lionel were waiting for me just beyond that. I was feeling dizzy but invigorated. It’s moments like these you wish you could put in a box and take out when you want to explain to people why exercise can make you feel so good.

After a couple of minutes of recovery time we carried on further up to the first set of dams. It was freezing. We estimated it to be 3 degrees. Thankfully there was no wind and the sky was clear.

Time for the downhill. This is an experience every person should try out at least once in their lives. The only way to describe it, is that its the closest feeling I have ever gotten to flying on a bicycle. Everything is dark around you except for your bike lights. The icy wind freezes the sweat on your forehead. And there are a million little orange lights twinkling in front of you. Its so surreal and quiet, except for the whooshing sound of your brakes and tyres. If you go too fast you can barely see the road ahead.

We got down to the top of Constantia Nek. Checked Main Road was clear and let rip down hill toward Hout Bay. We turned off Main Rd at Longkloof and made our way back home along Valley Road.

2.5 hours total. It kicked ass.

Here is the Google Earth image of the ride:

Post 1: In The Beginning

// September 3rd, 2008 // 1 Comment » // chesscube, fourhourworkweek, programming

OK here goes. This is my first post ever on a blog, so excuse the badly phrased English, terrible punctuation and generally boring parts of my mind. I promise I will get better at this.

I have been listening to the Four Hour Work Week audio book and am now inspired to start something of my own. Something that is fairly automated in terms of generating income but also fun. I am a firm believer in doing something every day that I enjoy and this has to be one of them.

Hey, but wait a sec, perhaps I should start from the beginning. I am 27 years and 345 days old. I was born in Johannesburg, South Africa and now live in Cape Town, South Africa. I went to St. Alban’s College for high school and graduated from the University of Cape Town with an Honours Degree in Computer Science. So yes, I am a programmer/software developer/general technology nerd.

I work for a company called ChessCube. It’s a site for chess enthusiasts. We are a team of 5 developers and the job rocks! Apart from participating in building a highly successful Chess Social Network, I have been involved in some really exciting projects, namely:

  1. ChessCube Cinema
  2. ChessCube Research
  3. Superlecturing
  4. Scaling on Amazon EC2

During the course of writing for this blog I will go into some more details for these projects. The day-to-day technical challenges that we face and how we solve them.

So you’re probably wondering by now why on earth am I looking at doing other work outside of my current job. Well firstly, I feel that I need to create something for myself, something that I can be proud of and something that I ultimately own and from which I can make a little bit of auxilliary income. I also want to learn new things and express myself. So enough of the sappy stuff lets get right into it.