A buyer’s guide to learning-focused evaluation

Evaluation is much too focused on impact, and not enough on process. This is counter-intuitive, and yet is true.

Imagine if Apple’s designers collected usability feedback on iPhones – their ease of use, durability, merits over other products. Then they took the forms and handed them out to workers on the factory floors in Shenzhen that produce the components.

What would they do with the feedback? How would they distinguish between hardware and software issues? Between implications for different hardware components?

The image is ridiculous. And yet this is pretty much what we try to do in the development and fragile states sectors.

The so-called gold standard for evaluation design—the randomised controlled trial—compares the program in toto against the alternative (“counterfactual”) of doing nothing. But this leaves the process of what actually happened within a black box. That might be OK for a medical trial, but certainly not for a complex social intervention where there are dozens of moving parts and stakeholder relationships that determine success or failure.

Along the same lines, the best-known evaluation criteria, written by the OECD’s Development Assistance Committee some twenty years ago, invite us to assess programs for their overall impact, efficiency, sustainability, and so forth.

This might be OK for accountability purposes, i.e. for donors who just want the bottom line, but it requires program staff to do a hell of a lot of work to infer what they, personally, need to change. A better approach must thus include two additional perspectives.

Looking down and in

It is no exaggeration to describe the starting point for modern management as the realisation that an organisation is not a very helpful unit of analysis.

Frederick Taylor, the father of “scientific management”, described this epiphany moment as the task idea. Its power was demonstrated most plainly by Taylor’s early trials to compare different methods and tools for steel-cutting operations. Over time he had reduced defect rates for individual operations enormously—by an order of magnitude—and with this the total costs of manufacture.

Of course, Taylor’s conceit that a similar “one best way” could be mapped out everywhere soon fell by the wayside. It was premised on beliefs about the standardisation of work that didn’t hold even for semi-skilled industrial work, let alone the growing service sector.

Nonetheless, the analytic approach—the task idea—has endured. Some seventy-five years after Taylor it found its bluntest and most tireless advocate in Edward Deming, who became the face of quality management in the Anglophone world.

In memorably clumsy prose, he castigated executives for relying on “slogans, exhortations and targets”, rather than hard statistical analysis of what was actually happening in their facilities. His point was simple: A fault in the product is a result that must be traced back to its causes—first the sub-components that produced the fault, and then the work processes that produced the sub-components.

Doing this requires only a few core tools. Process mapping to identify the major steps leading up to delivery. Measures of quality at each of those steps. And for those steps where problems creep in, a “work-out” method to re-organise things with the staff concerned.

Evaluations that ignore these steps are seriously short-changing your staff. They are handing them a customer feedback form on the finished iPhone.

Looking up and out

International agencies make small interventions in very large systems. On a bad day they are helpless without a ripe political environment; on a good day they can nudge their interlocutors towards a common vision.

Either way, it is essential to pay close attention to relationships. Many of the best tools for this come from specialised lobbyists, who can change anything only by way of influencing strategies. As a consequence they have every incentive to think about such strategies very carefully.

One simple, and robust, approach is the Outcome Mapping methodology popularised by the International Development Research Centre. The core idea can be explained straightforwardly:

160711 boundary partners

In words, “boundary partners” are those who inter-mediate between the program and the outside world. They co-produce the results with their resources, political support or even passive cooperation.

In consequence, any half-way competent evaluation in the development sector must thus the enabling relationships very seriously. For each significant interlocutor, you should be looking for systematic analysis of (a) their viewpoint and behaviour, and (b) how both these things have changed on the margins of your intervention.

Without this, you’re working based on assumptions. You’re assuming that the iPhone’s software will work with the hardware, rather than proactively working towards this.

Mapping the management system

All this sounds complex, and too far down in the weeds for general managers. In fact it is not, because the end results can be visualised fairly simply.

Here’s what that looked like for one client who was focused on the thorny issue of transitioning from emergency humanitarian assistance to longer-term development assistance. (Identifying information has been removed, and some details removed for clarity.)

160711 strategy map

This “dashboard” view provided:

  • A snapshot of organisational health across the specific business functions, and specific external relationships, that drove the strategy’s overall impact.
  • A tool to manage follow-up via regular senior management meetings, because the action points under each sub-heading were already identified with individual managers.
  • An action plan for institutional strengthening. In this case, our team moved straight into “work-out” processes with the planning & budgeting and human resources teams, based on the data we’d already collected.

That’s what learning-focused evaluation should enable you to do. Don’t settle for an academic approach that gives you the destination, but not a map to get there.