CMOs + QCA + Process Tracing: A rare opportunity to apply multiple complex qualitative evaluation techniques simultaneously

From 2015-18, Databuild, now part of Winning Moves contributed substantial method design, data collection and analysis on the evaluation of BEIS’ Electricity Demand Reduction (EDR) pilot scheme, supporting projects delivering a significant and lasting reduction in peak electricity demand evaluation. The final report was published in July 2019:

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/819733/edr-final-evaluation-report.pdf

 

The evaluation utilised multiple resource-intensive techniques - delivered by Winning Moves - to strengthen the exploration of pilot participation and additionality. These techniques, uncommon individually within evaluations let alone in combination, comprised:

  • Context-Mechanism-Outcome (CMO) combinations - designed to articulate hypotheses on the reasoning of organisations (mechanisms) in arriving at particular outcomes (e.g. participating or not) and what contexts might have influenced that reasoning. These can then be systematically tested (and so refined) against the stories the data are telling.
  • Qualitative Comparative Analysis (QCA) - a means of systematically considering the extent of association between different conditions (e.g. aspects of an intervention and the wider context) and outcomes of interest, so understanding which of the former appear ‘necessary’ and/or ‘sufficient’, either individually or in combination, for the latter to be realised.
  • Process Tracing - a case-based approach to causal inference which focuses on the use of ‘clues’. Process Tracing starts with a hypothesis which is then subject to a series of tests for evidence one might expect to observe if the hypothesis is true, and evidence one might expect to observe if it is not. Passing or failing these tests (each of which carries different weight) results in an overall assessment as to the likelihood that the hypothesis is true for the case in question.

 

Subsequent to the project, we have been invited to present at numerous events and conferences – including the 2017 European Environmental Evaluators Network Forum - as to the experience (benefits and challenges) of delivering these intriguing techniques, and would be happy to discuss these further with those interested in how they can be usefully applied to other evaluations.

Contact us

We would love to hear from you

 0121 285 3800
Winning Moves  
Baskerville House
Centenary Square  
Birmingham, B1 2ND
 
Edit