New Report – Review of M4P Evaluation Methods and Approaches

by Elise Wach

In a UK DFID-supported study, we conducted a review of evaluation methods and approaches for M4P (Making Markets work for the Poor) initiatives. The findings are very relevant to other private sector initiatives, including Inclusive Business, and are therefore shared here to facilitate improved evaluation approaches for Inclusive Business.

What is the M4P approach?

An initiative can be characterized as M4P if it plays a facilitative, adaptive role (as opposed to direct delivery and fixed, linear interventions), in order to contribute to systemic, large scale and sustainable market changes that positively affect the poor. These characteristics are common in many (though not all) inclusive business approaches.

How we can evaluate in complex markets

M4P programmes, and the markets within which they operate and seek to change, can be considered to be ‘complex’: they are nonlinear, dynamic, and relatively unpredictable. They consist of multiple interconnected and interdependent pieces. One small shift could catalyse a substantive change in the entire system (or not), and it can be difficult to know with certainty the nature and magnitude of change in advance.

Evaluations in complex contexts need to be designed in a way that enables the capture of complex interactions, feedback loops, and unpredicted effects. Lucho Osorio-Cortes and Marcus Jenal discuss these concepts in a recent SEEP Network MaFI paper.

The below linear logic model is often used in the evaluation of M4P (and Inclusive Business). However, it risks both underestimating impacts (through missing many of the results which M4P programmes are inherently designed to achieve), and overestimating impacts (through inaccurate assumptions about linkages between various levels of results). In short, the use of this linear model can result in inaccurate assessments and should not be used for M4P evaluation.

 

In contrast, the below diagram illustrates how M4P approaches are conceptualized as seeking to bring about systemic changes in markets.

It is important to measure changes in the market system (point B in the diagram) in order to determine whether an M4P programme has been successful. Again, given the nature of M4P, we would be looking to see if market changes are sustainable, large scale and systemic. Our review found that the evaluation of such changes is largely overlooked in M4P evaluations.

If it’s worth doing, it’s worth doing well

There are a few basic approaches that can help in the evaluation of the complex: these are the use of (and frequent testing and revision) of a theory of change, mixed methods and triangulation, and approaches that allow for the capture of unanticipated effects (i.e. ‘emergence’). (If interested to know more about evaluating the complex, I highly recommend Michael Quinn Patton’s Developmental Evaluation).

The report provides some further insight into the evaluation methodologies that are most appropriate for varying situations. But we as evaluators and development practitioners have a tendency get very excited about methods and tools (this was discussed at length in the Politics of Evidence conference). Thus, while matching the right tools to the job is the first step (and a very important one!), using them well is the next.

A theory of change is not done well when it is designed internally without sufficient justification of assumptions and linkages. Testing the assumptions with stakeholders external to the project and engaging in an iterative process of revisiting and revising are essential to ensuring the theory of change is relevant and grounded.

Mixed methods are not done well when they involve stakeholder interviews in which we select the people who we think will tell us what we want to hear (sample selection bias), and ask them only about what we want to know about our programme (self-importance bias and little room for emergence), and analyze them by ourselves (all sorts of biases here). For more on common biases, see White and Phillips Small n paper, which draws on a paper by Robert Chambers, Pawson and Tilley’s Realistic Evaluation, and Susanne Neubert’s MAPP guide.

Our review found that the M4P evaluation community needs to be better about implementing evaluation approaches in a rigorous, reliable way.

Who evaluates, and when?

Roles, responsibilities and timings of evaluation are important considerations that can often be overlooked in our focus on the methods. In the case of M4P, having a solid understanding the programme approach and the context is essential in order to accurately capture effects (prove) and ensure that the evaluation findings can enable the programme implementers to improve their approach. The latter is particularly relevant in the case of M4P, given the inherent need to be adaptive to continually changing markets.

The trick is balancing this intimate knowledge with objectivity and an outside perspective.

While one-off external evaluations can be very objective and rigorous, they may not always be entirely relevant. On the flip side, conducting an evaluation internally helps with the relevance but presents risks of bias.

To reach a balance between objectivity and deep knowledge of the programme and its context, one approach is to have long-term collaborations between the evaluator and the programme team. A second is for M&E to be conducted internally, with external audits (e.g. the DCED Standard).

Timing is also important. When we evaluate too soon, we can miss the long-term effects of the programmes. When we evaluate too late, then it is difficult to disentangle our initiative’s effects from that of other factors. Thus, we suggest that evaluation needs to happen both during the programme (to ensure contributions are measurable and also provide information that facilitates an adaptive approach, and (b) later on (post-project), when systemic, long-term change has had time to unfold.

Lessons for Inclusive Business

In reviewing the M4P evaluations that have been conducted to date, there certainly seems to be a lot that we can improve on. At the same time, there have been significant changes to M4P evaluation approaches in the last few years, with more people considering complexity, timing, and roles.

As Inclusive Business initiatives operate within the same messy and complex markets, we could apply many of the lessons from the M4P sector to improve our evaluations as well. Specifically, the following lessons from M4P could help improve our Inclusive Business evaluations: ensuring that our evaluations are grounded in a theory of change, which is frequently revisited and tested; ensuring that our qualitative data collection is conducted in a way to minimize biases and facilitate capture of unanticipated information; considering longer-term collaborations for evaluation; and focusing in on assessing any systemic, sustainable and long-term changes that have resulted from our Inclusive Business initiatives. Applying these lessons to Inclusive Business evaluation could help to increase our ability to both prove effects and improve practices.

Elise Wach is an independent consultant applying complexity science to evaluation, learning, and policy change.