A coders work is only half of the game

probably even less...

Dear Monks,

I got hit by an unexpected problem while getting a project done. This project was well specified, coded and tested.

Uhm - well - did I say tested? Kind of. It was tested by the coders who did it. And because the project is a web application covering a nontrivial business process cycle, everyone tested basically his own piece. Being responsible for some overall aspects of the project, I tried to test the overall cycle, but either failed (which then led to very frustrating "but it's working here" discussions with developers) or - when succeeded - realized, that I simply cannot cover all aspects of the process due to my restricted knowledge and the fact that I'm a single person and interaction of many people is what needs to be tested.

Because of this heavy dependence on human interaction, neither a semi-automatic testing tool, nor playing different roles myself (with several alter egos using the system) are feasible.

So it all boils down to try to find betatesters. Ha! Been there, tried that - and failed miserably. The programmers who did implement the whole thing have the great gift of "circumtesting" around the possible pitfalls, some external testers (translators - which is exactly one half of the target users) had a stab at it, but until now I got exactly 0 feedback about problems. And as I definitely assume, that there ARE problems, maybe the 0 feedback is because of some utter failure.

But then again maybe not. As I mentioned translators are only one half of the target group. The other half is simply "users" or "clients" that want a translation job to get done. Maybe the betatest translators are sitting there, waiting that something happens and nothing happens because no one wants to act as "user"?

Ok - to be more concrete - what I'm talking about exactly? There is this NLP Portal our company has been operating for quite some time now. A new functionality is the Translation Platform which should provide a B2B/B2C framework for people wanting some text to be translated the quick and cheap way. By choosing to use Machine Translation, Human Translation - or something in between AND by utilizing several technologies to ensure quality of human translation without the need of a second proofreader, this framework when presented at various talks, was perceived a "pretty cool way to do it".

Maybe it will be "pretty cool" after it's burn-in. Without it, it seems to me quite hand-warm right now. So I suppose you already figured it out: I have the problem of finding the right people to test the beast, which is nothing I was trained for nor do I have any experience doing it... If I could solve these problems with coding - ok. But that doesn't apply here. What are your strategies of burn-in/betatest multi-user environments? How do you get people (who should have no prior knowledge of your system) to participate? Are there best practices? Incentives?

Thanks for helping me out of the pit I've fallen in. I have this big baby before me, and am not even able to evaluate whether it is "pretty cool with some flaws", "not cool yet" or "born dead".

Bye
PetaMem All Perl: MT, NLP, NLU

Comment on A coders work is only half of the game

Replies are listed 'Best First'.
Re: A coders work is only half of the game by spurperl (Priest) on May 13, 2005 at 10:17 UTC
If you have a product many people are interested in, it's inevitable that a few will be willing to beta-test it, with all the implication (i.e. free, no-royalties). So, your beta testers are first and foremost your customers. I think it's obvious but worth mentioning anyway. People who don't care about your product won't care enough to beta-test it (unless it's paid QA staff). One of the cornerstones of XP is frequent releases for quick interaction with users. User interaction is a great debugging tools - users will find bugs where all your testing efforts fail. So, release it to the users. Let them test it and play with the system.	[reply]
Re: A coders work is only half of the game by tbone1 (Monsignor) on May 13, 2005 at 12:49 UTC
I used to work on some fairly complex systems, and what I found to be helpful was having the project manager write up test cases right as/after the requirements were finalized. They were still fresh in everyone's mind, and she and I would sit down for 30-45 minutes to create true positives, true negatives, false positives, and false negatives. (She usually tabbed me because my background was in science, not computing, and because I had a knack for thinking of potential pitfalls that others hadn't considered.) It wasn't perfect -- nothing is -- but the timing really helped a lot. As for incentives, she and I wanted to stay employed, and nothing concentrates the mind like a noose around the neck. -- tbone1, YAPS (Yet Another Perl Schlub) And remember, if he succeeds, so what. - Chick McGee	[reply]
Re^2: A coders work is only half of the game by JanneVee (Friar) on May 13, 2005 at 13:07 UTC
Uhm, that reminds of the movie office space. Peter to Bob: "That's my only real motivation is not to be hassled, that, and the fear of losing my job. But you know, Bob, that will only make someone work just hard enough not to get fired. " As I see it there are greater incentives than losing ones job, some sort of prize if one does really well. But a confirmation of a good done job is also sufficient sometimes.	[reply]
Re: A coders work is only half of the game by dragonchild (Archbishop) on May 13, 2005 at 12:57 UTC
Read Paul Graham and look at what he says about how they did this kind of thing at ViaWeb. In general, if you think something isn't in Perl, try it out, because it usually is. :-) "What is the sound of Perl? Is it not the sound of a wall that people have stopped banging their heads against?"	[reply]
Re^2: A coders work is only half of the game by kscaldef (Pilgrim) on May 13, 2005 at 17:41 UTC
so, Paul Graham's a reasonably prolific writer. What specific essay(s) did you have in mind? Or are you referring to his book?	[reply]
Re^2: A coders work is only half of the game by BUU (Prior) on May 13, 2005 at 23:38 UTC
Doesn't he say: "You should rewrite every thing in lisp and only hire genuises like me?"	[reply]
Re^3: A coders work is only half of the game by dragonchild (Archbishop) on May 14, 2005 at 00:49 UTC
In addition to that, he also describes how a good software developer deals with their users and betatesters. That's the relevant bit. In general, if you think something isn't in Perl, try it out, because it usually is. :-) "What is the sound of Perl? Is it not the sound of a wall that people have stopped banging their heads against?"	[reply]
Re^4: A coders work is only half of the game by kscaldef (Pilgrim) on May 14, 2005 at 06:27 UTC
Re: A coders work is only half of the game by mattr (Curate) on May 14, 2005 at 17:21 UTC
Well, sounds like the problem is you don't have any potential users on board yet. But you could alienate some potential customers if it really sucks erm is not polished yet. Some options you have might be to find other people in-house who could simulate those users, find customers who have a stake in codeveloping with you over a significant period, write up scenarios of your own and hand it to your mom, or even hire a professional. You need a useability study done and it sounds like you are really not at alpha stage yet. You need honest help without any threats to the project. It would have been easier if people interviewed for planning the spec were still involved, did you spec it yourself? Maybe sales people could get involved? Of course I don't expect you have a budget, or want to admit to management the need, but for example I'm experienced in designing services, doing useability studies, interviewing future users, am also a professional translator, and also work with a large translation company that could use the service as customer or translator supplier (though we are mainly Japanese, Chinese, Korean, English, so this is really just hypothetical for you). If you have someone or a few people who can assemble those capabilities in one room then you can start to attack the problem efficiently. For starters, anybody not on the coding team (which already knows the system too well) is a potential partner on this. So if you have a pool of translators at your company like we do, try and get some to help. Maybe even you could offer compensation. When it gets launched you are bound to find useability issues you didn't realize (beyond even the straight functionality you designed in) and you need to get some allies with the viewpoints of the users otherwise you may have a showstopper show up late in the game. My guess is the first thing you could try is find a junior member of the translation and sales departments and ask them for some time. If they can join your team for alpha and beta testing then you will be far ahead. Finally, you can offer customers that get involved a chance to help direct development and add their own feedback to make sure it is something they want. This may be of value to them enough to spend the time to help you, but you will have to be sure to document everything you get them to tell you and make sure future development and testing is handling those issues. Finally I could make a couple concrete suggestions. Make sure there is a form for writing up bugs, testing issues, whatever you call them. If you see something that can't be easily reproduced at least try to get it well documented. If it is really a communication/responsibility problem then get everybody on the team in a room with a terminal (and projector if you can), and go through it with them. They won't agree with you on useability issues probably but will jump out of their seats if you show a subtle bug. also, you should get the testers into a room under your control, provide them with instructions on how to test and write up reports, and even sit with them (or videotape them) if you must. the scary thing about no feedback is that it probably means nobody will use the system, and/or there is a major design (could be just graphic design!) problem which makes it confusing to use. if you find people who are honest and helpful they may even say "well we wouldn't ever really use this", or some such thing. maybe not to your face but.. so i'd advise you to press for feedback to you and try to get the bad news as soon as you can. it may also be an organizational problem, i can say for sure that one ad agency i audited had according to the ceo spent 3 million bucks on systems that were seldom used. some systems were badly managed in other countries, or dead due to lack of an organizational structure to run it frequently and ensure involvement. but some systems were indeed used by people, though management did not know this. When you get people to pass around a report form, or you get people in a division meeting, generally they don't respond significantly (though this can be cultural I guess). So if you can make a more casual, interesting atmosphere by say asking people directly to help you out on a little problem, it may provide more feedback than you are getting now. Anyway these are just my input on where you can go from here. I don't think it sounds like you are really ready to go full beta with customers. Probably you need to first get the developers all to sit at the same table and take responsibility for your uneasiness with the project, then form an alpha testing group, and do periodic testing until you get to beta which you do first in-house. One system I know that works very well was developed by a computer manufacturer to work with their ad agency, and both sides were involved which ensured it answered real needs well. If you can create that plus a responsive development team it might save the day.	[reply]
Re: A coders work is only half of the game by Avox (Sexton) on May 16, 2005 at 22:21 UTC
While you said that automated testing isn't feasible, I feel that you might still benefit from them. Automated testing suites like mercury http://www.mercury.com/ can do quite a bit. I don't think this is a substitute for real human testing, but it may cut down on the amount of "its working here" problems. Moreover, it will allow you to develope a set of tests that you can run during your future development. Other than that, it seems like you need to identify your customer base. Once you have that, give them some free stuff and you'll be surprised at how many will want to try it. Get your marketing dept involved. If this is a potential big thing, make them do their job :D.	[reply]