I’ve had a few minutes here and there to keep tabs on the EA UK forum for the BFBC2 PS3 beta, and toss in a few answers here and there. One thread turned into a discussion on the pros and cons of patching vs not patching the beta build.
There’s a limited amount of internal stability testing we can do. That testing can run down most things, but can’t test large-scale things (like server backends and what happens when 5000 people all log in at once). So, we get three things out of the beta: backend/large-scale stability tech testing, large-scale balance data and feedback from people.
I understand that people would rather see us implement our fixes resulting from all three directly into the beta. But putting out an update of the beta would require us to use up some of our internal testing to make sure the beta update is good enough. If sending a broken build to 100 people is bad, sending a broken build to 10000 people is a lot worse.
So in that situation we’re in a place where we have to choose between a spending our quality assurance resources on a beta update OR on the final product. To me, at least, that choice is quite easy. The led to this comment from poster 1Bryce1:
Isn’t a beta essentially a broken build to begin with? Any patches just eliminating problems and addressing balance issues along the way. So unless you break it more, any update would be less broken. Not only that but the “Backend/large-scale stability tech testing, large-scale balance data and feedback from people.” you get from each patched version would be more accurate to the finished game and give you better results. Wouldn’t it? I mean some of the most basic tweaks can drastically change the game and how people play. Spending some QA resources on a beta update IS contributing to the final product.
I started answering on the forum, but I figured this could be interesting for a wider audience and moved it here. A beta build is expected to be more broken than a final release — though I wouldn’t call it broken as such. A public (closed or open alike) beta is a reasonably unbroken build, from where I sit. The catch in the above line of reasoning is in this: “unless you break it more, any update would be less broken”. Essentially this is how it goes:
Lots of people in the dev team are making changes to the game. Each such change is to fix a bug or improve something. However, each time you change something, there’s a small risk of breaking something without noticing it. So together, the small risks of any one change breaking the game becomes a fairly major risk of *some* change breaking the game in one way or another.
Which means that the more work you do on a game, the bigger the risk that more stuff is broken at any given time (while, on average, the quality increases). As a developer, you can put up with that… either go back to an earlier, working, build or ignore the error for a while until fixed. Sending a build out over PSN is major though… you guys can’t just go back to an earlier build, or not jump into the tank because that crashes the game, or ignore the fact that all names in the score board come out as “PLAYERNAMEHERE_PLACEHOLDER” or whatever… a hundred small fixes can cause one large error, which you then fix as you find it.
So the way to deal with this is to stop development, test the build thoroguhly to find all the bugs. The closer you get to shipping something, the more stuff you will leave in there because of the risk of breaking something if you touch it, which means that as you get ready to ship the game off, the only bugs you fix are the really major ones. That way, when we ship, the game has been really well tested and we’re sure that it wont break.
This procedure has to be done regardless of whether it’s a beta update release or the final game… and that sucks because of the thing with “stopping development” I mentioned. So for a beta, what you do is branch the development. This essentially means you copy the entire source for the game to a separate repository, where it sits while everyone else keeps on improving (and breaking ;)) the game. The beta branch is tested, and thoroughly bug fixed. Needed bug fixes are done to both the main game line and to the beta line, while other improvements are done only on the main game line.
A slightly simplified version of this procedure as an image:
Some things are easy to see from this image: First of all, when the beta gets released, the main (somewhat broken) game line has already progressed a fair bit beyond the state in which the beta was branched. Second, there is no obvious way to update the beta from where it is… you need to start the entire procedure over again, branch another branch out of the main line game, and stabilize that the same way, and it is this process that takes resources away from the main development line.
I know there have been some comments that other betas do update. I’m sure they have rational reasons for that, which make the cost worth paying. MAG has come up as a name, and though this is pure speculation on my part, I’m guessing that their player count makes the game hard to test internally, which would mean that doing public beta updates is a very good choice for them.