A Beta Release is Born
I’ve had a few minutes here and there to keep tabs on the EA UK forum for the BFBC2 PS3 beta, and toss in a few answers here and there. One thread turned into a discussion on the pros and cons of patching vs not patching the beta build.
There’s a limited amount of internal stability testing we can do. That testing can run down most things, but can’t test large-scale things (like server backends and what happens when 5000 people all log in at once). So, we get three things out of the beta: backend/large-scale stability tech testing, large-scale balance data and feedback from people.
I understand that people would rather see us implement our fixes resulting from all three directly into the beta. But putting out an update of the beta would require us to use up some of our internal testing to make sure the beta update is good enough. If sending a broken build to 100 people is bad, sending a broken build to 10000 people is a lot worse.
So in that situation we’re in a place where we have to choose between a spending our quality assurance resources on a beta update OR on the final product. To me, at least, that choice is quite easy. The led to this comment from poster 1Bryce1:
Isn’t a beta essentially a broken build to begin with? Any patches just eliminating problems and addressing balance issues along the way. So unless you break it more, any update would be less broken. Not only that but the “Backend/large-scale stability tech testing, large-scale balance data and feedback from people.” you get from each patched version would be more accurate to the finished game and give you better results. Wouldn’t it? I mean some of the most basic tweaks can drastically change the game and how people play. Spending some QA resources on a beta update IS contributing to the final product.
I started answering on the forum, but I figured this could be interesting for a wider audience and moved it here. A beta build is expected to be more broken than a final release — though I wouldn’t call it broken as such. A public (closed or open alike) beta is a reasonably unbroken build, from where I sit. The catch in the above line of reasoning is in this: “unless you break it more, any update would be less broken”. Essentially this is how it goes:
Lots of people in the dev team are making changes to the game. Each such change is to fix a bug or improve something. However, each time you change something, there’s a small risk of breaking something without noticing it. So together, the small risks of any one change breaking the game becomes a fairly major risk of *some* change breaking the game in one way or another.
Which means that the more work you do on a game, the bigger the risk that more stuff is broken at any given time (while, on average, the quality increases). As a developer, you can put up with that… either go back to an earlier, working, build or ignore the error for a while until fixed. Sending a build out over PSN is major though… you guys can’t just go back to an earlier build, or not jump into the tank because that crashes the game, or ignore the fact that all names in the score board come out as “PLAYERNAMEHERE_PLACEHOLDER” or whatever… a hundred small fixes can cause one large error, which you then fix as you find it.
So the way to deal with this is to stop development, test the build thoroguhly to find all the bugs. The closer you get to shipping something, the more stuff you will leave in there because of the risk of breaking something if you touch it, which means that as you get ready to ship the game off, the only bugs you fix are the really major ones. That way, when we ship, the game has been really well tested and we’re sure that it wont break.
This procedure has to be done regardless of whether it’s a beta update release or the final game… and that sucks because of the thing with “stopping development” I mentioned. So for a beta, what you do is branch the development. This essentially means you copy the entire source for the game to a separate repository, where it sits while everyone else keeps on improving (and breaking 
) the game. The beta branch is tested, and thoroughly bug fixed. Needed bug fixes are done to both the main game line and to the beta line, while other improvements are done only on the main game line.
A slightly simplified version of this procedure as an image:
Some things are easy to see from this image: First of all, when the beta gets released, the main (somewhat broken) game line has already progressed a fair bit beyond the state in which the beta was branched. Second, there is no obvious way to update the beta from where it is… you need to start the entire procedure over again, branch another branch out of the main line game, and stabilize that the same way, and it is this process that takes resources away from the main development line.
I know there have been some comments that other betas do update. I’m sure they have rational reasons for that, which make the cost worth paying. MAG has come up as a name, and though this is pure speculation on my part, I’m guessing that their player count makes the game hard to test internally, which would mean that doing public beta updates is a very good choice for them.
Tags: Battlefield, BFBC2, Development, DICE, EA, Games
Games | slicedlime | 
Thursday, December 10, 2009 21:10
8 Comments
Other Links to this Post
RSS feed for comments on this post. TrackBack URI
Subscribe
By Lepper, Thursday, December 10, 2009 @ 21:52
Sounds fine to me Slicedlime. It’s good you explained this because we all know ppl been asking for a patch. We are most likely going to see these updates in the demo. The one things I do suggest is please fixed what is broken when the game releases. It’s so frustrating to see people exploit some of the broken things in-game.
By 1Bryce1, Thursday, December 10, 2009 @ 23:29
Very nice. Can I comment here or should I keep it on the forums?
By slicedlime, Thursday, December 10, 2009 @ 23:41
Either way works for me.
By hellbent oskool, Friday, December 11, 2009 @ 5:24
Thanks for the explanation, slicedlime =) It really puts me at ease knowing Dice listens to their customers/fans and interacting with us through the forums. Really means a lot to me.
Demize99, GordonVanDyke, repi and slicedlime. Your input is greatly appreciated.
By The Piano Dentist, Sunday, December 13, 2009 @ 17:05
Thanks
so long as the bugs still there at released are patched at some point I’m happy
By keyrat, Tuesday, December 15, 2009 @ 0:03
It’ll be better for players, if most bugs/annoyances are fixed, that when they buy the final game it actually feels like a better, evolved game. If you feed them every bugfix up to release it won’t have the same effect.
I do need to say this while I’m here: I would pay up to $15 extra in DLC to have a Wake Island conquest-mode map in this game
By 1Bryce1, Tuesday, December 15, 2009 @ 22:21
I guess the question remains would it even be worth it? You mentioned 3 things, of those wouldn’t 2 of those(large-scale balance data and feedback) benefit greatly from a patched version? It would seem to me that the accuracy of the data correlates to the how close the build is to the final game. With so many bugs and missing features in this Beta how useful can the data be? Besides fixes we know just by tweaking little things like points and damage you can see big trend changes in gameplay that can adversely affect other things.
Also I wouldn’t think that you would actually need work and patch the Beta build itself, but instead branch off another build of the final game that replaces the first stage of the beta. It could even be a total overwrite of the first.
By slicedlime, Tuesday, December 15, 2009 @ 22:39
Feedback would benefit. The backend testing is by its nature under our control… we can tweak the backend to work properly without patching the beta. You noticed that with the fixes for the unlocks that kept going down at the very start of the beta.
The build would come from a new branch than the first beta, but would need the same kind of testing and stabilization as the first beta, which would be effort we could otherwise spend on testing for the final game.