Indonesia 2010

This was an evaluation of DfID's first cash for transitional shelter project implemented by CARE International Indonesia and P3SD, CARE's local partner. The terms of reference (ToR) for the evaluation posed several impossible-to-answer questions as many humanitarian ToRs are wont to do.

Strategic Questions

We insisted on reporting not just what was done but on examining the project strategy and original decisions in terms of:

  • who was targeted and why; where and why; and what and why?
  • project outcomes analysed in light of specific contextual factors for success
 

Technical Questions

The unanswerable technical questions were as follows and we treated them as described:

  • Livelihoods: Were beneficiaries able to access good quality, sustainable materials; good quality labour, where required and did this project have a positive impact on local markets?

Given that there were more than 2,500 households involved, we dealt with this by looking at local sheltering processes in terms of pre-event: livelihoods and markets - their characteristics and interaction with shelter and housing; housing, land and property; house building processes; housing finance and housing costs.

Without the resources for a massive survey, we worked with key informants and focus groups to understand different categories of outcome in terms of household characteristics (as shown)

Livelihood Categories and Shelter Processes

  • Technical Assistance: How effective were technical training sessions; how effective were technical guidance materials; how effective was ongoing technical guidance?

With so many households and questions about gender running through the evaluation, we looked at the relative influence of different groups and how they were targeted by training.

We found that: women were often at home next to the T-shelter site on a daily basis but were not in a position either socially or as a result of training to supervise confidently an experienced mason; experienced masons were skilled at working with the materials but were sometimes reluctant to take advice from people they perceived as “non-technical” including male and female householders who were not masons and occasionally the P3SD monitors; inexperienced masons were more likely to follow advice but faced limitations in their workmanship; the young P3SD monitors felt confident to motivate and give instructions but were not always present at key moments (e.g. mixing concrete) and were not equipped with monitoring paperwork that would formally help them monitor shelters that diverged from the designed model; once the final funds had been disbursed or a family had been refused the second installment, it was much more difficult for P3SD monitoring staff to influence quality and motivate householders to progress.

 

Training and Gender graph showing who was competent to build and who had the power to influence building decisions in terms of who received training

  • Risk and Safety: was this technical guidance applied and are beneficiaries now living in safe transitional shelters? To what extent have we reduced beneficiaries’ vulnerability to future risks, including earthquakes? What have been the wider impacts of this project for beneficiaries, particularly in terms of livelihoods?

This analysis of the design borrows from the World Health Organisation’s approach to the evaluation of risks to water quality: rather than massive, expensive, “after-the-fact” sampling of water, WHO looks at risks of contamination in a whole water system. This evaluation does not rely on sampling of shelters (just a visual inspection of a 10% sample would mean 340 shelters, at 10 per day, we would be looking at a month’s work with a team of at least 2 or 3 people). Instead, we look at risks in the system of design and implementation that contribute to disaster risk reduction in final construction.