The two basic hypotheses of the iterative evaluation method with mini-seminars are as follows:

• The time at which the evaluation is conducted will in the long run determine how the action is judged and that issue weighs heavily on the quality assessment. It is indeed not only reaching the planned goals and carrying out the scheduled activities that determine the quality of a programme but also, and undoubtedly above all, its impact (including unplanned short-, mid- and long-term effects). That is expressed in terms of method through evaluation missions at different times.

in this example:

At T1, the results will be very positive (basic needs met)

At T2, the results may already appear less positive (phenomena of dependence, project non-practicability, unplanned secondary impacts?)

At T3, depending on the case, the integrated impact may be negative or positive.

 

• The efficiency of the learning mechanism is perhaps as important as the lessons learned from the evaluation mission: it is therefore important for the latter to be discussed as a group as quickly and as closely in the field as possible, but also at headquarters. This is done by organising inter-actor mini-seminars on the spot and upon return to Europe, where the mission results are presented and discussed.

October 1998: Hurricane Mitch struck Central America. Nicaragua and the Honduras were hit hard. Together NGOs invested in supporting and implementing emergency relief then reconstruction programmes using public and private funds that abounded. Most of the funds managed by NGOs were invested in housing reconstruction.

August 1999: An initial evaluation mission, conducted in August 99, drew the first series of lessons and put forth some hypotheses that were to be checked later. An initial series of mini-seminars were then organised in Managua. They were followed up by several feedback sessions in France, and in particular workshops during which the mission results and their repercussions for the NGOs were discussed.

January/February 2001: Already suffering from a decade of civil war and Hurricane Mitch, El Salvador was devastated by a series of earthquake tremors that hit both rural and urban areas. Aid was set up quickly but in a relatively limited way in view of competition from another catastrophe, an earthquake in Gujarat, India.

June 2001: The Group URD’s initial interdisciplinary evaluation mission analysed the response and organised an initial feedback series both in the field and upon return to Europe. The report in French, Spanish and a summary in English were placed in wide circulation, including through the Group URD website www.urd.org.

September 2001: So three years later, what has become of the numerous post-Mitch housing reconstruction programmes implemented by the NGOs in Nicaragua and the Honduras in 1999? How were those same humanitarian actors going to deal with the current housing reconstruction issues in El Salvador, seriously hit by earthquakes at the beginning of this year? The Quality Project team went back to Nicaragua and the Honduras to the locations it had already visited in 1999 during the first work phase capitalising on the experience to «learn lessons 3 years post-Mitch» and communicate them to the reconstruction actors in El Salvador via a workshop organised in San Salvador at the end of September 2001. The Group URD pluridisciplinary team therefore conducted a field evaluation, with direct observations, surveys of the former beneficiaries or non-beneficiaries, as well as formal and informal discussions. The analysis is structured along three technical axes (housing-economic and food security-social services, health, education…) and four transversal axes (Development-Emergency link, risk prevention, passage from donation to cost recovery, population displacement). The exercise sought to understand what worked, why and how, and what didn’t work, why and how in order to learn lessons in terms of the Quality of actions and be able to share them in El Salvador at present. Alongside that evaluation work, three young researchers set up shop in the zone (one in Nicaragua for a series of case studies, two in El Salvador, one of which was to conduct a mini-social audit).

In Nicaragua and the Honduras three years after Hurricane Mitch struck, the reconstruction projects have obtained mixed results. The hypotheses developed in 1999 ended up being confirmed to a wide extent:

• Some re-housing sites created following Mitch have been over 30% deserted. In those cases, at worst the houses have been abandoned, at best they have been sold at low prices
• Others have fallen far behind schedule and sought to do "too well", which led them to set up houses and sites unsuited to the population’s needs
• Other projects, on the contrary, have been truly successful, as reconstruction operations have served as a basis for new long-term development projects currently underway.

The varying levels of success can be explained by the different internal and external factors, whether they were technical or not.

In El Salvador, eight months after the earthquakes, a few problems or risks have already arisen that can be explained to various degrees. Most of the hypotheses developed at the time of the June 2001 mission (first mission of the iterative mission process with mini-seminars) were confirmed during mission N°2:

• Although after Mitch the reconstruction projects developed and set up too quickly were generally failures, in El Salvador, the strategy was essentially to find temporary solutions in face of the emergency. The local dynamics show that if there are not any accompaniment programmes, that temporary status is likely to last.
• The fear of being confronted with the same land programmes as in post-Mitch Nicaragua or the Honduras led lots of humanitarian actors to seek a risk-free approach. That led to favouring programmes for people capable of showing their property deeds. In an attempt to limit problems, NGOs left the most vulnerable populations by the wayside, precisely those who did not have access to land
• The donors and the lead-times they impose are often the main causes of hasty decisions made to the detriment of project quality. Every time a donor has allowed time and accepted to fund prior socio-economic evaluations before setting up programmes, the chances of achieving quality operations has been maximised.
• Every time the "population displacement" option has been chosen over reconstruction solutions on the spot, many problems have arisen which will be very hard to manage in the future
• The "cinder block" options that have prevailed at times could have been replaced by reconstruction strategies favouring local materials and know-how
• Every time a relief operator has concluded an alliance with a local development actor, the context, its constraints and its potentialities have been taken into account much better. Once again, those alliances brought about substantial gains in quality.

The following chart puts forth an empirical analysis in several stages: it uses a sample of the field observations by listing the main problems arising and attempts to explain their origins on two different levels of causality in order to put forward principles of action and tools for improving the quality of actions.

Based on this chart, different levels of reflection on the quality of the action are brought to light. The determining factors may be external or internal to the organisation. Some will be linked to the context, others wider-reaching:

1. Determining factors external to the organisation:
1. linked to the context: political environment (in this example: access to land) socio-economic level of the targeted population—geographic environment.
2. Linked to the pressure from donors: programme lead-times, funds available, etc.

 

2. Determining factors internal to the organisation:
1. organisation’s culture
2. vision of the «catastrophe phenomenon», is it an accident or not?
3. knowledge and/or taking into account of the different factors dealing with housing (which should lead to a policy of actor recruitment and training)
4. NGOs self-impose lead-times under pressure from the media: how can efforts be aimed more at news/education and less at news/sensationalism.

It can clearly be seen here that the quality of programmes is partially linked to the tools, but also to a certain number of external determining factors and ingredients which cannot necessarily be transformed into tools: ethics, moral obligations, cultural referents, etc.

As a result, a house reconstruction programme will be done differently depending on the vision of housing: is it only a roof or is it the result of a complex process integrating socio-economic and cultural aspects as well?

This vision therefore implies:

• the donors have to accept the constraint linked to time and not impose lead-times incompatible with the action to be undertaken
• working as closely as possible to the previous living location = reflecting on actions to reduce risks rather than displacing populations
• working in partnerships with partners from different operational cultures
• reflecting on whether the technical proposals are adequate for the context and the population’s capacities; (cost, evolution, reproducibility with limited resources, etc.) but also likely to create jealousy
• having different "no time" responses: the first 48 hours are the only real periods of emergency. At that stage, only the local structures are on the scene, and only proximity aid really works: there is therefore a need to work on the preparation level. As soon as that stage is past, there is a need to work with development tools to support rehabilitation processes. From that standpoint, housing reconstruction cannot be managed as an emergency.

  Recomendations for the NGO

• More flexibility for the implementation timeframe is required if a proper adequation between the constraints and capacities is to be established

It is clear that with a "learning" outlook, the method of iterative evaluations with mini-seminars has substantial potential, including a particularly suitable "cost/efficiency" ratio. However, the test currently underway made it possible to determine substantial margins for improving the method, notably with respect to:

• the preparation of the missions themselves
• the organisation, form and content of the mini-seminars

Indeed, too much trust was placed in the link via Internet between both the national and international actors. Incomplete or partially incorrect address lists and the limited time devoted to the preparation of the second feedback session in the field led to a certain amount of frustration among some actors who had not received the first mission report or who received the invitation for the second mini-seminar late. The same mistakes mustn’t be made the second time around.

The "advisory" function of such evaluations works smoothly when enough time is available for "bilateral" discussions. However, it is to be feared that the rigidity of the programmes and donors may make altering the action difficult, despite the fact that the needs for reorientation were substantially highlighted by the evaluation.

While the mission in 1999 arrived in the field eight months after Mitch, the first mission implemented in El Salvador using this particular method of "iterative evaluations with mini-seminars" took place four months after the earthquakes. It would appear that there is a need to arrive even earlier. This will be done during the next test which should take place within the framework of the Afghan crisis (mission scheduled in December if financing can be mobilised in time).

Before suggesting practical solutions (methods, tools, formulas), it is important to understand the origins of Quality problems in Humanitarian Action within the framework of a holistic reflection placing humanitarian actors in an environment that partially determines the quality of their actions (relationship between sponsors, donors, local actors, budgetary and political constraints). Although it is not a matter of freeing humanitarian actors of their share of responsibility, it is worthwhile to return to a global and holistic approach of the problem.

Moreover, the setting is dynamic. The technique is not the only level of action (political lobby, from micro to macro). As a result, a small structure that uses all its means for negotiations making it possible to obtain a small plot of land for farmers will perhaps do a better job than a bigger NGO that builds a lot of houses, even houses that meet quality standards, but which will be left empty due to a failure to take into account the economic and property stakes.