Evaluate your new technology like doctors evaluate the effect of a new medicine or a new treatment on patients. In this way you will put your users’ problems into focus instead of being guided and biased by your passion for technology.
Many times I have been through the depressing experience of spending months and even years of my life developing technology that was later thrown away, without even a single user having tried it for a single day. I then started looking into new development methods, methods that promoted co-design of technology.
Co-design and iterative methods are very useful as technology innovation methods. The result is often technology that users like and use. But as a researcher I also need to publish my research results. No research is finalized before it is disseminated. Publication in high quality fora needs methodic “final evaluation” of technology. Even more important, I myself need to believe in my evaluation. Such a belief will cure my depression and clear my conscience. I know then that I have either made the perfect technology, or learned something from the experience of making a bad technology.
Good technical research consist of a structure similar to this:
- Find a relevant problem where a solution will have high impact.
- Design and develop innovative technology that will solve the problem.
- Evaluate the technology in real world (or near real world) using scientific evaluation methods.
- Report the results in high-ranking journals.
Do you see any problem with this approach? No? I do. The moment you start designing your technology bias creeps into your head. Your design becomes your darling. You start suffering from the IKEA effect (see Dan Ariely’s excellent book). You start cherry-picking your references to support your design etc. And of course your evaluation might suffer from this bias as a lot of research results do.
OK, vanity is the worst of all sins. So what can be done? Do as doctors. Doctors found out early that bias in medical research can kill people. So they became fanatic about eliminating bias. One tool that I have come to embrace is the research protocol. An approach using the research protocol looks like this:
- Find a relevant problem where a solution will have high impact.
- Start writing a research protocol, describing a new intervention aimed at solving the problem.
- The medicine in this intervention will be product X (you can later replace X with anything, including your own darling IKEA shelf).
- Once you are happy with your research protocol, freeze it and guard it as your bible (or as whatever book you believe in).
- Then you can sit down and find out what X should be. Here you can of course use co-design methods.
- Once X is developed, do the evaluation exactly as you described in your research protocol.
The difference? In this approach you plan your evaluation before you design your technology, and not after. (If you are familiar with test-driven development, it is like writing your tests before you write your code.)
You need a rock solid plan (the protocol) in order not to slip away from the goal of solving a problem for your users and into developing cool technology. In many cases you might even find existing technology that can solve the problem. Alas!
Do you have any experience using such an approach for creating technology? Discuss it here!