Sharding (Gamification Server Engineering)
Kyle Vigen, cybernetic Bunchball engineer from the future, blogs about sharding- check it out, and if you want to join our engineering team, check out our job listings.
I’ve spent the last two posts talking about some of the performance challenges we’ve faced over the years, so today I want to shift the focus a bit and talk about performance’s cousin, scalability*. Scalability is a broad topic, so for this post I’m going to focus on scaling databases, a particularly acute challenge for many web applications.
To provide some background for the uninitiated: databases have to be “scaled” because, if you have enough users, no matter how much you optimize performance, at some point you won’t be able to process everything on one db server. You’re going to be forced to split your data. But, in dividing your data onto multiple servers you end up creating a new challenge for yourself - you have to be able to figure out which data is on which server. The solution: sharding**