How to Run Major Innovation Experiments

How to Run Major Innovation Experiments

To achieve major, strategic innovations in an organization, having brilliant ideas is not enough. You also need to run experiments to test the ideas. The experimentation should aim to answer just one question: What would be the cheapest, fastest, and least risky way to learn whether a new venture or major idea is worth the investment and implementation?
Any experimentation involves 3 crucial aspects—its Finance and objectives, its Design and result evaluation, as well as its Execution.

Innovation team

I. How to FINANCE experimentation and what are the OBJECTIVES?

Finance:
Experimenting requires resources and an investment. An organization usually has two types of budgets—a project budget and an operation budget, but usually no experimentation budget. Experiments are different from both projects and operations. The time it takes to deliver result, the velocity of decision making, and the way to assess results are all not the same. Thus we need to allocate specific funding for experimentation alone.
One way of doing that is to assign 2 to 5 percent of the project budget to a venture portfolio for experimentation, as most organizations do. Why from project budget? Because they often share similar resources such as people, equipment, and so on. But a venture portfolio is managed differently from a project portfolio. A venture portfolio can be viewed as an incubator of startups, which is managed by an agile leader assuming the role of an intrapreneur. Out of, say, 10 ventures in such a portfolio, maybe just one could be wildly successful like a star, another one or two would only reach break even, and the other seven or eight wouldn’t give birth to any innovative solution at all. When we evaluate these ventures or experiments, we can’t look at the result of each separately; rather, we need to look at the whole portfolio, knowing that the stunning success of the single star experiment is built upon the learnings from those others who didn’t work out, and that there is no shortcut to identify successful ventures before learning from failure.
Objectives:
To evaluate the performance of the venture portfolio is to know whether the objectives of an organization’s intrapreneurship or innovation are met. There are three objectives in general:
1. Have experiments that in case they succeed will give birth to actual innovative solutions that could bring a major boom to the business.
2. Learning, which is also the key to achieving the first one. Although learning is not an ROI type of KPI, it still can and has to be measured. Consider a technique like innovation accounting which are designed to measure learning and progress in experimentation.
3. Engage workers. Ideas don’t come just from executives; brilliant ones can come from everywhere in an organization and involve any employee, so does the experimentation of those ideas.

II. How to DESIGN an experiment and EVALUATE the result?

Design:
A value hypothesis is a statement formulated to express what value is expected to derive from a potential solution that is not yet implemented. The goal of experimentation is to validate or invalidate value hypotheses by testing them. Here's an example:
A public transportation company faces the recurring problem of having frequent disruption and delay of the trains. One potential solution is to install an electronic service status display at the entrance of each metro station, so that people can see whether the operation is disturbed when they arrive and react accordingly. Implementing such a solution can be very expensive and certainly involves risks since no one knows to what extent people will find it useful. That means, the potential solution needs to be tested in a way that is fast, cheap, and with minimum risk.
Experimentation is designed to have usually one or more value hypotheses tested so that we can understand whether the hypotheses will really deliver the assumed value. In our example, one testable hypothesis could be to install a non-electronic board for service status announcement in front of three metro stations for one month’s time. When there is an incident affecting these metro stations, at least 50 users will be surveyed with one question: Is this display useful to you or not? The hypothesis will be considered as validated if at least 60 percent of the respondents answer “Yes,” and invalidated otherwise.
In this example, the test of the value hypothesis involves responses from actual users based on their experience with a product for trial (the non-electronic board). It's not an analysis based on a document or abstract questions asked over the phone of random users. Likewise, all the experiments need to be designed with a real prototype and feedback from actual users. Or else, they are worthless and can’t lead to any solid conclusion. More on designing experiments to support decision-making on strategic innovations.
Evaluate:
To evaluate the result after an experiment is done, we have to understand that the way to define success of an experiment is quite different from that of a project. Whether we validate or invalidate a hypothesis in an experiment, it's a success in both ways. If the hypothesis is validated, it may turn into a promising product and generate value at scale; if it's invalidated, not only a precious lesson can be drawn but a potentially huge mistake of investing in something that will not yield the expected benefits can be avoided. The only way to fail an experiment is when we don't know whether the hypotheses are valid or not. The role of an agile leader or intrapreneur is not only to embrace, support, and promote experimentation, but more importantly, to change people’s way of defining what is failure and what is not, which is an inseparable part of an organization’s culture.

III. How to EXECUTE an experiment?
To execute experiments, we need to consider the bite-sized nature of the employed resources to enforce the high velocity of the experiments. I use the word “box” below to refer to the thoughtful allocation of resources to experimentation. There are 4 types of boxes that work together in experimentation:

The budget box: When we start an experiment, we only have a fixed amount of money to do it. There will not be one more dollar available, and if we cannot do it within this budget, we just don't do it at all. The idea can be big, but it has to be split into small experiments, each fitting into a budget box.
The time box: Speed is of the essence when we're innovating, so we need to set up a maximum time frame to finish experimentation, for example one month, two months, etc. The faster our reaction to the changes in the market, the more value we can deliver. If we are too slow, even the most ground-breaking idea will yield nothing but waste.
The team box: It refers to the high level of team dedication it takes to execute experiments—dedication of the team members is as important as their expertise to succeed in experimentation. Without a fully committed team, no effective collaboration or real accountability can be achieved, thus no result can be generated from any experiment.
The location box: Sharing the same location can help people to collaborate spontaneously. If people work in different buildings and meet occasionally here and there, they can’t produce anything together effectively. It’s much better to have a virtual or physical meeting room explicitly allocated for experiments.

Besides these boxes, another must-be-met condition for the successful execution of an experiment is that the team has to be autonomous. It should be a team of intrapreneurs who have the freedom of making decisions by themselves. The role of experts and a manager or a leader on the team is merely to “serve” the team members by supporting them and offering guidance instead of controlling or directing them.
And last but not least, well executed experiments can be done with an external expert or some academics. These can contribute to the experiments of the organization through challenging its habitual way of working or shaking up its old systems.

⇨ Besides Finance, the Design and Execution, do you think there are other crucial aspects of running innovation experiments?

Sources:
Collet, B. (2019). "Agile Leadership", (online course).
Pollack, S. (2012). "The Value Hypothesis", Forbes.