A Beta Release is Born

I’ve had a few minutes here and there to keep tabs on the EA UK forum for the BFBC2 PS3 beta, and toss in a few answers here and there. One thread turned into a discussion on the pros and cons of patching vs not patching the beta build.

There’s a limited amount of internal stability testing we can do. That testing can run down most things, but can’t test large-scale things (like server backends and what happens when 5000 people all log in at once). So, we get three things out of the beta: backend/large-scale stability tech testing, large-scale balance data and feedback from people.

I understand that people would rather see us implement our fixes resulting from all three directly into the beta. But putting out an update of the beta would require us to use up some of our internal testing to make sure the beta update is good enough. If sending a broken build to 100 people is bad, sending a broken build to 10000 people is a lot worse.

So in that situation we’re in a place where we have to choose between a spending our quality assurance resources on a beta update OR on the final product. To me, at least, that choice is quite easy. The led to this comment from poster 1Bryce1:

Isn’t a beta essentially a broken build to begin with? Any patches just eliminating problems and addressing balance issues along the way. So unless you break it more, any update would be less broken. Not only that but the “Backend/large-scale stability tech testing, large-scale balance data and feedback from people.” you get from each patched version would be more accurate to the finished game and give you better results. Wouldn’t it? I mean some of the most basic tweaks can drastically change the game and how people play. Spending some QA resources on a beta update IS contributing to the final product.

I started answering on the forum, but I figured this could be interesting for a wider audience and moved it here. A beta build is expected to be more broken than a final release — though I wouldn’t call it broken as such. A public (closed or open alike) beta is a reasonably unbroken build, from where I sit. The catch in the above line of reasoning is in this: “unless you break it more, any update would be less broken”. Essentially this is how it goes:

Lots of people in the dev team are making changes to the game. Each such change is to fix a bug or improve something. However, each time you change something, there’s a small risk of breaking something without noticing it. So together, the small risks of any one change breaking the game becomes a fairly major risk of *some* change breaking the game in one way or another.

Which means that the more work you do on a game, the bigger the risk that more stuff is broken at any given time (while, on average, the quality increases). As a developer, you can put up with that… either go back to an earlier, working, build or ignore the error for a while until fixed. Sending a build out over PSN is major though… you guys can’t just go back to an earlier build, or not jump into the tank because that crashes the game, or ignore the fact that all names in the score board come out as “PLAYERNAMEHERE_PLACEHOLDER” or whatever… a hundred small fixes can cause one large error, which you then fix as you find it.

So the way to deal with this is to stop development, test the build thoroguhly to find all the bugs. The closer you get to shipping something, the more stuff you will leave in there because of the risk of breaking something if you touch it, which means that as you get ready to ship the game off, the only bugs you fix are the really major ones. That way, when we ship, the game has been really well tested and we’re sure that it wont break.

This procedure has to be done regardless of whether it’s a beta update release or the final game… and that sucks because of the thing with “stopping development” I mentioned. So for a beta, what you do is branch the development. This essentially means you copy the entire source for the game to a separate repository, where it sits while everyone else keeps on improving (and breaking ;) ) the game. The beta branch is tested, and thoroughly bug fixed. Needed bug fixes are done to both the main game line and to the beta line, while other improvements are done only on the main game line.

A slightly simplified version of this procedure as an image:

branchSome things are easy to see from this image: First of all, when the beta gets released, the main (somewhat broken) game line has already progressed a fair bit beyond the state in which the beta was branched. Second, there is no obvious way to update the beta from where it is… you need to start the entire procedure over again, branch another branch out of the main line game, and stabilize that the same way, and it is this process that takes resources away from the main development line.

I know there have been some comments that other betas do update. I’m sure they have rational reasons for that, which make the cost worth paying. MAG has come up as a name, and though this is pure speculation on my part, I’m guessing that their player count makes the game hard to test internally, which would mean that doing public beta updates is a very good choice for them.

Game On

I’m not dead. First of all, we’ve moved into a house and have spent a great deal of time sorting out things and selling my old apartment. And then, just as I thought I would have some more time for the blog again, we launched into crunch mode.

I’m hoping I’ll be able to write more again soon.

For now, I urge you to try the Playstation 3 beta of Battlefield: Bad Company 2, which opened yesterday. I have a couple of beta codes laying around (EU ones), so drop a comment here if you’ve got a PS3 and haven’t been able to get one.

Black Box Black Box Testing

One of my friends is in College, and is currently feeling the full idiocy of a system that was only beginning to be rolled out as I left. Let me explain how it works.

Essentially, the system is meant to test the students’ solutions to homework problems. This is done by providing a solid definition of what the input and output of the application are supposed to be on the standard in/out channels, and setting up a whole bunch of test cases, including a memory limit and a CPU time limit. Students submit their source code to the system, which compiles it and runs all the test cases against the application in a black box test. So far, so good.

Seeing these guys at work, compared to me and my colleagues at work, makes a few things very apparent: even with a fairly solid grasp of algorithms and datastructure, their number one problem is code. Where professional programmers swim through code like sharks in the sea, the students appear to be more or less drowning. Theoretical learning aside, the education lacks practical programming, debugging, practical programming and some more practical programming.

It would seem that these programming exercises would be the perfect opportunity to get that kind of experience, if it wasn’t for the fact that the test system is itself a black box. You put in your code, and it tells you yes or no. It’s not quite a boolean pass/fail answer, but close enough: you will get told a result from the set: Didn’t compile, Passed, Failed, Crashed, Time Limit Exceeded. When I first heard of the system, it was motivated with the fact that sometimes in professional programming, that’s all you get.

I agree. Sometimes, you get gnarly bugs that give you less information than a world pro’s poker face. I’ve spent weeks tracking bugs like that sometimes, using all kinds of tools at my disposal to try to wring more  information out of the error, until finally the knot was untied. But — for all the bugs like that I’ve been through, none of them were eventually solved by guessing what was wrong and how to fix it.

Supposedly, the tool is meant to teach the students to debug their code… which it somehow does by disallowing all normal debugging tools. You can’t run a debugger on it, you can’t print traces, you’re not allowed to log to a file or socket, you’re not even allowed to know what input caused the error. The only tools you have at your disposal are your wit in coming up with your own test cases and code reviews.

Any attempts at normal debugging would be classified as cheating. If I was faced with a bug under those circumstances, I would do whatever I could to get more information out of it. Hey, I can crash it with different signals — that’s a few bits of information I could get back from it. All those kinds of tricks of the trade that real programmers use to, you know, solve problems… would be cheating.

This leads to a skewing of results… very simple bugs turn into monster problems, since you can’t identify and fix them. What they are learning is not how to debug their programs but how to painstakingly solve the very specific problem of pleasing the system. By artificially making easy things hard, the system has effectively found a way to avoid teaching the students essential skills in programming: simple debugging tools like tracing and breaking into a debugger. Instead, they learn programming by coincidence: poke something until you (hopefully, eventually) get a green light.

That’s not a lesson to learn.

The only way to go about this, faced by the obstacle made up of this system, is to learn a different skill: testing. More on that later.

More on studies: An Exceptionally Stupid Idea, Go Tinker, What’s a Good Final Year Project?

Standing in the Way of Culture

The reasoning behind the introduction of copyright was the establishment of a law which would make sure there were incentives for creating culture. There was a fear that if there wasn’t some form of exclusivity, middlemen with a large capacity for distribution would easily be able to grab all works of art, produce and distribute them more effectively than the creators themselves and thus getting the lion’s share of the profits. This was a time when the printing press was the hot new thing, and writers feared publishers would easily steal all their hard work.

The fear was that if this kept happening, the people creating works of art would tire of creating culture and seeing others profit while getting nothing for themselves (a reasonable assumption), so copyright was introduced, giving authors an unlimited right to association with their works and a limited economical exclusivity with regards to production and distribution. This would make sure middlemen would not be a problem in the production of culture.

Sadly, in this exclusivity now appears an effect that goes in the direct opposite direction related to the original intent. To explain what I mean, let’s discuss something of high cultural value — commercials. Leading up to the fifth season of The Deadliest Catch, Discovery Channel created a trailer. A music and sound design studio called Musikvergnuegen were hired to create a soundtrack for the trailer.

The trailer aired, and something somewhat unusual happened: People heard the music and rushed to the TV in order to see what it was about. On Musikvergnuegen’s blog, under a fairly short and simple post about the trailer, people started gathering in the comment fields with fantastic stories about how they reacted to the music.

Let me tell you my story of this song. I wasnt even in my house but i heard this music come through the surround sound so i darted inside to find out it was a commercial for my favorite show. But wut really mattered was this music. The celtic sailor feel and sorrow filled sound makes this one of the best pieces of music ive ever heard.

The blog post has more than 40 comments, with people asking that the song be released somewhere so they can buy it and listen to it. Several other blogs have called attention to the music and linked to the post on Musikvergnuegen’s blog.

The studio appreciate all the attention, but answer that sadly Discovery owns the rights to the soundtrack, and that  because of that they can’t sell it. Instead, they’ve mailed Discovery, and urge others to do the same. The only problem is that Discovery doesn’t reply.

So, here’s a middleman blocking the flow of culture between the author and the consumers, stopping the people from getting the culture they desire. Recognize this problem? This saga ends on a slightly upwards note, as Musikvergnuegen takes matters into their own hands and add the music to their demo reel, which means it’s now possible to stream it from their web site.

Another example of the same kind of cultural blockage is how music giant Universal let its lawyers loose a few weeks ago to prevent the performance of a theater play at the City Theater in Stockholm. Not, as you may have thought, because they hadn’t paid for it, but because Universal wanted even more money for it, claiming the music was a central theme in the play. However, it turns out the author of the songs in question, Paul Simon, doesn’t agree with the middleman that the play should not be performed.

Abandoned Goods

A similar development has been in effect when it comes to computer games for a long time (and here it’s even more clear). The whole concept “Abandonware” exists to denote older games which are no longer on the market. There are websites that specialize in catering to people feeling nostalgic about games. As with many other kinds of culture, there are a whole lot of people who long for the good old days — the older games have a higher level of quality, they claim. Others still just want the ability to play “the classics” — just like there are classic books and movies there are classic games… the difference is that you can’t get the classic games anymore.

The fact that games can’t be bought doesn’t mean the copyright on them has expired, however. The rights to various games and game intellectual properties are bought and sold between different companies and often you’ll find the rights to games far away from the people who originally created the games after a studio closed its doors.

The fact that they’re no longer selling the games doesn’t prevent companies from having a go at web sites providing Abandonware. Sometimes, whole web sites are the targets of attempted shutdowns by lawyer, at other times, only certain games are targeted and removed. But the games themselves are not sold, so the people who wanted to play them are left empty-handed.

Sometimes, people get so desperate for their old, lost games that they even gather up the people needed to form a team and go through the enormous effort of creating a remake of their classic game of choice, only to be shut down by the rights holder.

There has been a long debate about this in gaming circles, sometimes with thunderous accusations from big games companies. People who download abandonware are called pirates, and blamed for some form of loss of income, even more absurd than the normal kind of calculations.

The Upper Hand of the Middle Man

In addition to all the problems outlined above, the cost for both creation and distribution has brought back something a situation which is very similar to the world before the introduction of copyright: middlemen dictate the conditions they like and grab large parts of the pie, since the authors are so completely dependent on the chain of distribution. In these negotiations, one side has sharp lawyers with years of experience of writing contracts, fine print and cost vs revenue calculations. On the other side are inexperienced, often young talented authors with no experience of writing or even reading contracts.

The imbalance becomes extremely obvious in the average record contract. In normal venture capital business, investors come in with capital, taking a risk with that money in exchange for a slice of profit if the gamble plays out (highly simplified). This state of affairs is a quite reasonable starting point. Looking at the music industry, their standard contract is very different from this — it is more like a loan than an investment… but a loan that turns into an investment once repaid.

Let me explain. The musician or band gets money for recording an album, creating a video and other needed things. The album is released, and all the profits go directly into the record label’s pockets until the entire loan has been repaid. Only then does the artist get a first dollar for his or her hard work. At a glance, this seems incredibly skewed, and it’s just as skewed as it seems. If things had ended there, it would have been outrageous but somewhat real, but that’s not where it ends. In addition to these expenses, there needs to be marketing for the artist. Reasonable proposition if you want to sell the album, but the marketing money also comes out of the “loan” to the artist. Only it’s the label that controls the marketing spend.

So the standard contract is a shared partnership where all the risk is held by one part. In a normal start-up business, this situation would be interpreted as the artist talent and work having no value  at all (and thus should have no part of the income). Add to that the fact that the labels have binding contracts for the artists, but are free to pick and choose how they want to act themselves, and we’ve ended up incredibly far from the original intention of copyright. Let me explain further with two examples:

The New York-based band October Project created some beautiful (but hard to classify) songs during the 90s. They released two albums that were quite successful. Then, without a warning or official explanation, the record label terminated the contract, which killed the band. Two of the band’s founding members went on to start the “November Project”, a band which collected all the money they needed for their recordings themselves, directly from the fans on the Internet. On the homepage for the band, founder Emil Adler urged the fans not to buy the October Project records — “Not a single penny goes to the band”, he wrote. Not a penny, even though one of the albums had hit top-50 in the US.

Another band, Splashdown from Massachusetts, looked to have a bright future. They signed a contract with a child label for Capitol records and released an EP, which quickly sold out. The band made a song on the soundtrack for the movie Titan AE, and expectations were high. They recorded a new album, “Blueshift”. Once done, the label decides not to release the album. After a long fight with the label and because of the fear that Capitol would own any future songs, the band chose to call it quits.

Authors giving up on creating new culture because of middleman behavior was exactly what copyright laws was supposed to prevent in the first place! Something is wrong here. Instead of acting as a guarantee for the flow of culture into society, the effect is a draining of culture.

It’s Broken, Fix It!

There are already political forces in parts of the world that are growing stronger campaigning for a reformed copyright. In Europe, the Swedish Pirate Party has won a seat in the European Parliament. The German counterpart looks to be going strong in the coming elections for the national parliament. Support for both parties is extremely high among young voters.

Taken to its edge, you really could wonder about why the rights to works of art should be something that can be bought and sold at all. There seem to be plenty of problems that would go away if an author could never sell the actual right to their works.

There seems to be some very simple things to do though. Digital distribution has brought down costs of distribution by several orders of magnitude. This somewhat breaks the hold the middlemen have over the market, but there is still something of a monopolized market. Things are starting to change, but what is needed is a much more direct approach, fans meeting artists, players meeting developers directly. This has already started happening more and more in the computer games industry, and I believe we’ll see much more of it from musicians in the future.

Another thing with the Internet is that there is never any really good reason to stop selling something. Put all vintage titles you own the rights to online for a small fee. Even a fee of $5 or even $1 per game means a horde of people would buy them, and it would make you a whole lot of more money than any amount of lawyers sent hunting abandonware sites. Oh, and while you’re at it, skip the DRM. The new market is on the consumers’ terms, and with a direct connection to fans comes the responsibility of treating them like customers, rather than potential thieves.

Downtime

Due to some technical problems with a router that sadly went undiscovered for some time, the blog was down during most of the weekend. I’m sorry about the delay in fixing this — we should be up and running as normal again now.

WordPress Themes