Black Box Black Box Testing

One of my friends is in College, and is currently feeling the full idiocy of a system that was only beginning to be rolled out as I left. Let me explain how it works.

Essentially, the system is meant to test the students’ solutions to homework problems. This is done by providing a solid definition of what the input and output of the application are supposed to be on the standard in/out channels, and setting up a whole bunch of test cases, including a memory limit and a CPU time limit. Students submit their source code to the system, which compiles it and runs all the test cases against the application in a black box test. So far, so good.

Seeing these guys at work, compared to me and my colleagues at work, makes a few things very apparent: even with a fairly solid grasp of algorithms and datastructure, their number one problem is code. Where professional programmers swim through code like sharks in the sea, the students appear to be more or less drowning. Theoretical learning aside, the education lacks practical programming, debugging, practical programming and some more practical programming.

It would seem that these programming exercises would be the perfect opportunity to get that kind of experience, if it wasn’t for the fact that the test system is itself a black box. You put in your code, and it tells you yes or no. It’s not quite a boolean pass/fail answer, but close enough: you will get told a result from the set: Didn’t compile, Passed, Failed, Crashed, Time Limit Exceeded. When I first heard of the system, it was motivated with the fact that sometimes in professional programming, that’s all you get.

I agree. Sometimes, you get gnarly bugs that give you less information than a world pro’s poker face. I’ve spent weeks tracking bugs like that sometimes, using all kinds of tools at my disposal to try to wring more  information out of the error, until finally the knot was untied. But — for all the bugs like that I’ve been through, none of them were eventually solved by guessing what was wrong and how to fix it.

Supposedly, the tool is meant to teach the students to debug their code… which it somehow does by disallowing all normal debugging tools. You can’t run a debugger on it, you can’t print traces, you’re not allowed to log to a file or socket, you’re not even allowed to know what input caused the error. The only tools you have at your disposal are your wit in coming up with your own test cases and code reviews.

Any attempts at normal debugging would be classified as cheating. If I was faced with a bug under those circumstances, I would do whatever I could to get more information out of it. Hey, I can crash it with different signals — that’s a few bits of information I could get back from it. All those kinds of tricks of the trade that real programmers use to, you know, solve problems… would be cheating.

This leads to a skewing of results… very simple bugs turn into monster problems, since you can’t identify and fix them. What they are learning is not how to debug their programs but how to painstakingly solve the very specific problem of pleasing the system. By artificially making easy things hard, the system has effectively found a way to avoid teaching the students essential skills in programming: simple debugging tools like tracing and breaking into a debugger. Instead, they learn programming by coincidence: poke something until you (hopefully, eventually) get a green light.

That’s not a lesson to learn.

The only way to go about this, faced by the obstacle made up of this system, is to learn a different skill: testing. More on that later.

More on studies: An Exceptionally Stupid Idea, Go Tinker, What’s a Good Final Year Project?

Standing in the Way of Culture

The reasoning behind the introduction of copyright was the establishment of a law which would make sure there were incentives for creating culture. There was a fear that if there wasn’t some form of exclusivity, middlemen with a large capacity for distribution would easily be able to grab all works of art, produce and distribute them more effectively than the creators themselves and thus getting the lion’s share of the profits. This was a time when the printing press was the hot new thing, and writers feared publishers would easily steal all their hard work.

The fear was that if this kept happening, the people creating works of art would tire of creating culture and seeing others profit while getting nothing for themselves (a reasonable assumption), so copyright was introduced, giving authors an unlimited right to association with their works and a limited economical exclusivity with regards to production and distribution. This would make sure middlemen would not be a problem in the production of culture.

Sadly, in this exclusivity now appears an effect that goes in the direct opposite direction related to the original intent. To explain what I mean, let’s discuss something of high cultural value — commercials. Leading up to the fifth season of The Deadliest Catch, Discovery Channel created a trailer. A music and sound design studio called Musikvergnuegen were hired to create a soundtrack for the trailer.

The trailer aired, and something somewhat unusual happened: People heard the music and rushed to the TV in order to see what it was about. On Musikvergnuegen’s blog, under a fairly short and simple post about the trailer, people started gathering in the comment fields with fantastic stories about how they reacted to the music.

Let me tell you my story of this song. I wasnt even in my house but i heard this music come through the surround sound so i darted inside to find out it was a commercial for my favorite show. But wut really mattered was this music. The celtic sailor feel and sorrow filled sound makes this one of the best pieces of music ive ever heard.

The blog post has more than 40 comments, with people asking that the song be released somewhere so they can buy it and listen to it. Several other blogs have called attention to the music and linked to the post on Musikvergnuegen’s blog.

The studio appreciate all the attention, but answer that sadly Discovery owns the rights to the soundtrack, and that  because of that they can’t sell it. Instead, they’ve mailed Discovery, and urge others to do the same. The only problem is that Discovery doesn’t reply.

So, here’s a middleman blocking the flow of culture between the author and the consumers, stopping the people from getting the culture they desire. Recognize this problem? This saga ends on a slightly upwards note, as Musikvergnuegen takes matters into their own hands and add the music to their demo reel, which means it’s now possible to stream it from their web site.

Another example of the same kind of cultural blockage is how music giant Universal let its lawyers loose a few weeks ago to prevent the performance of a theater play at the City Theater in Stockholm. Not, as you may have thought, because they hadn’t paid for it, but because Universal wanted even more money for it, claiming the music was a central theme in the play. However, it turns out the author of the songs in question, Paul Simon, doesn’t agree with the middleman that the play should not be performed.

Abandoned Goods

A similar development has been in effect when it comes to computer games for a long time (and here it’s even more clear). The whole concept “Abandonware” exists to denote older games which are no longer on the market. There are websites that specialize in catering to people feeling nostalgic about games. As with many other kinds of culture, there are a whole lot of people who long for the good old days — the older games have a higher level of quality, they claim. Others still just want the ability to play “the classics” — just like there are classic books and movies there are classic games… the difference is that you can’t get the classic games anymore.

The fact that games can’t be bought doesn’t mean the copyright on them has expired, however. The rights to various games and game intellectual properties are bought and sold between different companies and often you’ll find the rights to games far away from the people who originally created the games after a studio closed its doors.

The fact that they’re no longer selling the games doesn’t prevent companies from having a go at web sites providing Abandonware. Sometimes, whole web sites are the targets of attempted shutdowns by lawyer, at other times, only certain games are targeted and removed. But the games themselves are not sold, so the people who wanted to play them are left empty-handed.

Sometimes, people get so desperate for their old, lost games that they even gather up the people needed to form a team and go through the enormous effort of creating a remake of their classic game of choice, only to be shut down by the rights holder.

There has been a long debate about this in gaming circles, sometimes with thunderous accusations from big games companies. People who download abandonware are called pirates, and blamed for some form of loss of income, even more absurd than the normal kind of calculations.

The Upper Hand of the Middle Man

In addition to all the problems outlined above, the cost for both creation and distribution has brought back something a situation which is very similar to the world before the introduction of copyright: middlemen dictate the conditions they like and grab large parts of the pie, since the authors are so completely dependent on the chain of distribution. In these negotiations, one side has sharp lawyers with years of experience of writing contracts, fine print and cost vs revenue calculations. On the other side are inexperienced, often young talented authors with no experience of writing or even reading contracts.

The imbalance becomes extremely obvious in the average record contract. In normal venture capital business, investors come in with capital, taking a risk with that money in exchange for a slice of profit if the gamble plays out (highly simplified). This state of affairs is a quite reasonable starting point. Looking at the music industry, their standard contract is very different from this — it is more like a loan than an investment… but a loan that turns into an investment once repaid.

Let me explain. The musician or band gets money for recording an album, creating a video and other needed things. The album is released, and all the profits go directly into the record label’s pockets until the entire loan has been repaid. Only then does the artist get a first dollar for his or her hard work. At a glance, this seems incredibly skewed, and it’s just as skewed as it seems. If things had ended there, it would have been outrageous but somewhat real, but that’s not where it ends. In addition to these expenses, there needs to be marketing for the artist. Reasonable proposition if you want to sell the album, but the marketing money also comes out of the “loan” to the artist. Only it’s the label that controls the marketing spend.

So the standard contract is a shared partnership where all the risk is held by one part. In a normal start-up business, this situation would be interpreted as the artist talent and work having no value  at all (and thus should have no part of the income). Add to that the fact that the labels have binding contracts for the artists, but are free to pick and choose how they want to act themselves, and we’ve ended up incredibly far from the original intention of copyright. Let me explain further with two examples:

The New York-based band October Project created some beautiful (but hard to classify) songs during the 90s. They released two albums that were quite successful. Then, without a warning or official explanation, the record label terminated the contract, which killed the band. Two of the band’s founding members went on to start the “November Project”, a band which collected all the money they needed for their recordings themselves, directly from the fans on the Internet. On the homepage for the band, founder Emil Adler urged the fans not to buy the October Project records — “Not a single penny goes to the band”, he wrote. Not a penny, even though one of the albums had hit top-50 in the US.

Another band, Splashdown from Massachusetts, looked to have a bright future. They signed a contract with a child label for Capitol records and released an EP, which quickly sold out. The band made a song on the soundtrack for the movie Titan AE, and expectations were high. They recorded a new album, “Blueshift”. Once done, the label decides not to release the album. After a long fight with the label and because of the fear that Capitol would own any future songs, the band chose to call it quits.

Authors giving up on creating new culture because of middleman behavior was exactly what copyright laws was supposed to prevent in the first place! Something is wrong here. Instead of acting as a guarantee for the flow of culture into society, the effect is a draining of culture.

It’s Broken, Fix It!

There are already political forces in parts of the world that are growing stronger campaigning for a reformed copyright. In Europe, the Swedish Pirate Party has won a seat in the European Parliament. The German counterpart looks to be going strong in the coming elections for the national parliament. Support for both parties is extremely high among young voters.

Taken to its edge, you really could wonder about why the rights to works of art should be something that can be bought and sold at all. There seem to be plenty of problems that would go away if an author could never sell the actual right to their works.

There seems to be some very simple things to do though. Digital distribution has brought down costs of distribution by several orders of magnitude. This somewhat breaks the hold the middlemen have over the market, but there is still something of a monopolized market. Things are starting to change, but what is needed is a much more direct approach, fans meeting artists, players meeting developers directly. This has already started happening more and more in the computer games industry, and I believe we’ll see much more of it from musicians in the future.

Another thing with the Internet is that there is never any really good reason to stop selling something. Put all vintage titles you own the rights to online for a small fee. Even a fee of $5 or even $1 per game means a horde of people would buy them, and it would make you a whole lot of more money than any amount of lawyers sent hunting abandonware sites. Oh, and while you’re at it, skip the DRM. The new market is on the consumers’ terms, and with a direct connection to fans comes the responsibility of treating them like customers, rather than potential thieves.

Downtime

Due to some technical problems with a router that sadly went undiscovered for some time, the blog was down during most of the weekend. I’m sorry about the delay in fixing this — we should be up and running as normal again now.

WordPress Themes