Comments on: Black Box Black Box Testing

By: Rob

Rob — Thu, 22 Oct 2009 15:17:05 +0000

Where are all the new blog posts!?

By: slicedlime

slicedlime — Thu, 17 Sep 2009 19:34:32 +0000

You’re talking about a different system Jenn. You pass 18/20 tests, you don’t know that… all you know is “Fail”. No information on what you passed or not, no partial credit, nothing.

The course I saw my friend was doing gave them 1-2 test cases, then had 100+ cases in the system test. If you failed, you had no idea if it was the first or the last test that failed.

Your autograder system sounds like a step up from KATTIS.

By: Jenn

Jenn — Thu, 17 Sep 2009 19:29:42 +0000

This is the system that was used when I was in college, the “autograder”. It wasn’t as bad as this article makes it sound.

First of all, it’s not about “pleasing the system”; the system is pleased by correct code. You do get one or two test inputs/outputs to work with, and then there are usually around 20 black box test cases which include the tests you were given originally. It should be trivial, therefore, to pass a couple of tests, and usually if you can pass a couple you’ll pass most. Say you pass 18/20, then you know you’re missing a border condition. And even if that’s the best you can do, you get partial credit.

Also, at my college they used the autograder to check the code functionality and then a TA would actually read through your code and possibly give you more credit. They tried to give you as much room to succeed as possible without handing out their test cases. Speaking of TA’s, most of them were more than willing to at least give you hints as to what test cases you might be missing. Most would even take a look at your code. It’s not like you’re totally on your own, pitted against some inscrutable machine.

My main problem with the system is that you had a very limited number of submissions (like 5). Sometimes you’d fix something that was a red herring and waste a submission. That can be stressful.

By: slicedlime

slicedlime — Thu, 17 Sep 2009 19:27:50 +0000

If you’re saying you want to teach people to debug programs, then you should let them practice… eh… debugging programs. If you want to teach them unit testing, you really need to be teaching them that, not just setting up assignments in a way that forces them to learn it on their own.

There are good techniques for it, but they don’t teach it, and you sure can’t expect people to figure those techniques out by themselves.

The situation here is more similar to what happens when you’ve done your unit testing and move on to integration testing… but then you’re unable to use debugging tools. There is no good point for that.

By: Brian Guthrie

Brian Guthrie — Thu, 17 Sep 2009 19:19:36 +0000

“So, you’re left with your own wit coming up with new test data (and making sure you get it right), or just reviewing code.”

Isn’t that sort of the point? How does that discourage the development of world-class programming skills? Unit testing is a complex topic worthy of study, but it doesn’t belong in its own course; every programmer should be doing it. If it’s one function per assignment, part of being a world-class programmer is understanding what it takes to test the dickens out of that function.

By: slicedlime

slicedlime — Thu, 17 Sep 2009 19:00:39 +0000

Paul: I agree in part that it’s a good way to test things thoroughly… thus my comment at the end. However, part of the issue on these problems is that the available test data is very limited… usually one or two example inputs. So, you’re left with your own wit coming up with new test data (and making sure you get it right), or just reviewing code.

I agree that in a real-world situation you’ll want to test everything in pieces before shipping it, but we’re not talking about a software system here… it’s more or less one function per assignment.

There are good ways of working to systematically work with this situation too, but really that’s more something for a course in unit testing than anything else. It’s a skill many professional programmers are missing too.

But more on that in a future post.

By: Winsrp

Winsrp — Thu, 17 Sep 2009 18:20:09 +0000

well, as a programmer myself, it just feels stupid. Debugging and tracking is the most useful thing when dealing with errors/bugs, most of the people that work with me, are freshmen right out of the college, and they try to make things work without event test it, and if they find an error they get frustrated, I’m trying really hard to teach them that programing is a “by-parts” job, when you have to do and test 1 thing at the time, so everything will work like a clock at the end. And working on their debugging skills since most if not all of the available tricks are pretty much new to them.

By: Paul

Paul — Thu, 17 Sep 2009 17:35:51 +0000

hmm. I’m torn on this. I’d say 98% of the time I’m forced to use the poke system. I’m used to it and can get what I want about 50% of the time and the other time I have to really really break it and work backwards. That being said, in order to do this black box programming right, you’d have to start with writing your own debugging script and then write output to it until you get to the final product and switch it from a debug/comment system to a final product system. That’s not really that bad an idea.

I do hope they can at least use try-catch otherwise I’d have to agree with the writer. Hopeless!