Evaluating complex programmes against multiple criteria is always a challenge.
Standard of Proof conducted an evaluation of a country-wide health programme in the Pacific Islands. The programme included multiple workstreams, with each workstream working towards delivering unique outputs and achieving specific outcomes that would all contribute to reducing mortality and morbidity across the population. Although valuing programmes with such diverse outputs and workstreams is challenging, the qualitative weight and sum (QWS) approach provided an objective way in which to do this for the health programme.
The evaluation focused on the programme’s five outputs. The team worked with the clients, including health professionals and researchers to designate a weighting for each output, with this weighting estimating the perceived value of each output towards the overall goal – to reduce mortality and morbidity. Some outputs were designated as ‘high value’, some as ‘moderate value’ and others as ‘low value’ outputs. These qualitative weights allowed the team to make objective judgements about the worth of the overall programme according to the four evaluation criteria: relevance, effectiveness, impact and sustainability.
The results were visualised as weighted (output) bubbles along a rating scale how expectations were met (x-axis) and for each evaluation criteria (y-axis). Visually, the estimated value of each output was expressed in the size of each bubble (big, medium and small); big bubbles for ‘high value’ output, small bubble for ‘low value’ output. The result was a concise picture of the overall performance of a complex health programme according to four criteria. QWS was useful to break down complexities (without losing this information) and translate the results visually into a clear overview amenable to a broad audience.
