Round table: “Learning from experiments”

July’s round table for the London Conflict / Fragility group focused on a question that has come up again and again in previous sessions. If we’re really committed to “demand-driven” approaches, then we’ve got to be committed to empirical investigation of what works.

That might mean trying out the minimum-viable-version of an approach before we commit to a big, multi-year investment. It might mean more use of controlled trials, or a shift from formal advance planning to something closer to impact investing. But in every case it requires pitching an approach that could well fail, in contexts where there are real human stakes for getting it wrong.

 

Over a wide-ranging discussion we looked at a number of successful, and unsuccessful, experiments. You’ll find a rough visual minutes above (click on the graphic to make it big), but here’s a few of the more interesting points.

* Much of the time, short-term and local incentives crowd out long-term and global interests. This puts a premium on leaders being the guardians of mission & vision … a task made more difficult by generally weak boards in the development / humanitarian sectors.

* It’s difficult to sell learning as a positive outcome, if there isn’t “harder” data about the beneficiaries of an intervention. Few institutional donors include learning as a relevant result in their “results-based” budgeting and strategic frameworks.

* Physical demonstration is probably the most effective way to break established mental models. The impression spreads vertically (“scaling”), and also horizontally across organisations (“replication”).

* Horizontal learning is probably the most important thing in the long run. Organisations should give more weight to the “public good” of new knowledge / approaches in their planning and evaluation models, rather than obsessing about direct impact.

* Experimental approaches by definition require different capabilities. Often-times things are easier with a pick-up team (“skunk works”) of people who self-select to work on a new approach because they believe it. However, this disrupts established human resources, planning and budgetary processes, and creates additional sources of tension.

Background

The London Conflict / Fragility group is open-access and administered via Meetup. We aim for a relaxed, inter-disciplinary conversation with no need to represent our individual institutions, and no expert / audience dynamics. Current venue is the Glasshouse space, pictured above.

If you’re London-based, or passing through, please do check out the hosting page for upcoming events.