23rd
We underestimated it a bit…
So as part of a re-branding we also rolled out a really cool contact book integration feature. We spent a ton of time looking at how it would effect performance if all of our customers loaded their contact books into the system. After a lot of testing we felt comfortable that if all of our customers had contact books in the system that it would not effect our production systems. We were correct… however we did not think about if everyone tried to upload their contact books at the same time what would happen…
So when I sent an email out to a small sampling of customers today announcing this, they basically all at the same time went and started uploading their books. We had no idea how much people were wanting this feature.
So it crashed some servers (nothing in the production) and we now have to do a little re-engineering. It will be back out their shortly.
Another lesson learned and luckily this one was not an expensive one.
