The Patches Must Flow! Insight into the Patch Release and Communication process.


(Citizen Star News) - 2014-09-02 - Ben Lesnick dropped into one of the usual delay driven forum meltdowns with an interesting post about the development, patch, release process:

If you're interested, there's a fair bit of process that goes into this kind of decision and resulting post. (Or, if you're wondering why you get one Comm-Link later in the day rather than up to the minute updates on patches.)

Getting a release patch is sort of like mining spice; the team will work until the last possible minute to make it happen. On a Friday, we might have a release candidate build as late as early afternoon. That build goes to QA for a round of testing. Did it fix the blockers and the critical issues we identified with the previous candidate? Did it introduce any others?

After that process (which takes several hours at it fastest) we have a go/no-go meeting. Project leads meet via Skype to go over issues and give their take on whether or not to release. If you've ever seen a space shot from mission control, it's sort of that: community is go, engineering is go, platform is go, production is no-go and so on.

So: by late afternoon (Pacific time) the day we're hoping to release, we may know we're not going to patch. If we discover an issue that we THINK we might be able to quickly fix with additional engineering time, we may do that and kick off the process again, which pushes the final decision into the evening.

That's the point where I can start writing a post for the Comm-Link. I'll often sit down with a producer (usually Travis), because part of our process is telling you what's going on under the hood. That can take an hour or so, and then any post that important has to go through Chris. (Sometimes made more difficult by our being in different places; Friday's post was a collaboration between James and Travis in LA, myself in Atlanta and Chris in Seattle!)

(If we give it a go and we know we ARE going to try and patch, there's an additional layer that can end up come off as a delay to people watching for by-the-minute updates: we promote the build to the public server and then run it past QA again. Sometimes, we find that breaks it in a way that wasn't apparent internally, so we have a no-go even later in the evening.)

All of that is to say, it's not that we don't want you to know when we're patching... it's just very difficult to say until it actually happens. We can come in in the morning, eager and positive that a patch is coming out as hoped only to be felled by some unspeakable horror twelve hours later when we propagate it to the public servers. And if we spend that time telling everybody the patch is coming that day, you'll end up even more frustrated! (And while a good portion of the community understands what's going on behind the screens, there are plenty of people who don't... so it comes off looking to many like we're creating unnecessary hype.)

I know a lot of people ask: why not talk about where a patch is days in advance, just in general to prepare everyone? The answer is that it's remarkably hard to know. There have been numerous times where we were SURE we'd patch and then couldn't... and an equal number of times where we were sure it wasn't going to happen only to have our programmers pull it off. Publicizing our 'patch thinking' as it happens would ultimately greatly confuse!

So - an end-of-Friday (or, if we're working late for one last shot, a very-very-late-on-Friday) Comm-Link to tell you what's going on seems like the best solution. But I'm certainly open to other options if you can think of a way to keep everyone updated without subjecting half a million backers to will-it-or-won't-it drama!

~ Ben Lesnick [source]

A few interesting notes:

  • Up until this point there have not been Friday releases, as they like to avoid releasing something when they can't fix it if there's a problem found. The "server guys" haven't been as available over the weekends. Has this changed one wonders? Did CIG add staff or push their contractors to be able to handle a more 24/7/365 any time release schedule?
  • At the decision point this last week, Chris was in Seattle for Pax and Ben was in Atlanta for DragonCon, which is why Ben mentions them being in those places.