|
In my last post, I said that this whole process had progressed to the "Are you serious?" level of setbacks. So I thought I would share yet another...
Important contextual detail: We all telecommute, so all of our servers are hosted in one place and we use VPN to access them from locations spanning the entire US.
This morning, we began the process of rolling out the Content Market, as planned. The appropriate servers were taken offline, backups initiated, etc. However, not long after starting, the developer orchestrating the rollout lost his internet connection. His entire neighborhood lost internet, with an estimated outage time of 4 hours from the ISP. Thankfully, the actual updates process had not yet begun, so someone else was able to bring the servers back online, with everything operating as it had before.
But this is what I mean by "Are you serious?". We just started the rollout and the entire neighborhood where the developer lives has a massive internet outage. <sigh>
At this point, we are still waiting for the developer to regain internet access. Then we will re-assess whether to launch this afternoon, reschedule to tomorrow, or wait until next week. We hate taking down servers in the evenings when gamers may be playing, and we also hate releasing right before the weekend, just in case there's an issue we didn't catch in our testing. So all three options suck. In any case, those are the options before us, and we're weighing how best to proceed.
Thanks, Rob
|