Groupe URD

La Fontaine des Marins

26170 Plaisians – France

Tél./Fax. : 04 75 28 29 35

urd@urd.org

Project supported in 2001 by :The French Ministry of Foreign Affairs, the Fondation de France, the Swiss Ministry of Foreign Affairs.

In 2002 and 2003, acceptation to be confirmed :The French Ministry of Foreign Affairs, ECHO, the Fondation de France, Ministère Suisse des Affaires Etrangères.

TABLE OF CONTENTS

This first year has made it possible to begin answering the following questions:

• How far have the various NGO partners gone into their own quality processes?
• What evaluation practices do humanitarian NGOs have?
• Where does the Project stand, relative to other international initiatives concerning the quality of humanitarian interventions?
• Using a given case study (Central America), what lessons can be learned, and which methods should be put in place, in order to help improve the quality of interventions?

To this end, several levels of activities have been carried out throughout the year:

• research activities including theoretical work and practical studies (i.e. field missions)
• work was also done upstream and downstream of the research: animation of the network of partners, mission preparation, writing for publication, presentation of the work in seminars and conferences.

The theoretical work has essentially focused on making an inventory of existing practices and initiatives concerning the quality of humanitarian interventions. The objective of this process was to know what already exists so as to avoid repeating or overlapping with current activities. It was found, actually, that the sharing of NGOs' internal practices had not been done and that it could be an interesting exercise per se.

Groupe URD turned to Marion PORTAT, a health audit professional and a student at the Institute of Economic and Social Development Studies (IEDES), Paris-Sorbonne. She carried out a 5 month internship, in partnership with the French Red Cross, a member of the Quality Project.

The project-team supervised the preparatory and methodological work (elaboration of the questionnaire frame, selection of NGOs, personal contacts, etc.). Marion then collected the information and wrote a dissertation on this subject, which included a first level of conclusions and remarks. She went on a mission in Anjouan with Aide Médicale Internationale, another partner of the Quality Project, in order to include a field perspective in her study. Her work is currently being reviewed by the project-team in order to finalise the analysis and disseminate the results. A synthesis is presented in annexe 1.

This subject was studied by Morgane LE GUENIC, a student finishing her DESS on "Evaluation of Industrial, Agricultural, Social and Environmental Projects", at the University of Economics in Rennes. She undertook this research with Groupe URD during a 5 month internship. Using the same modus operandi as Marion, she interviewed around ten humanitarian or development organisations (not only medical) and wrote a summary document presenting a first level of conclusions. Of these, the need to combine the evaluation knowledge of emergency and development actors stands out. This point will be further developed in the future. A summary of her work is presented in annexe 2.

The issue of quality is today at the heart of the aid community's concerns, from donors to actors in the field; it can therefore be approached in different ways. By following the international debates on this issue, the Quality Project is by and by defining more clearly where it stands.

The following figure suggests a "miniature mapping" of the different existing initiatives.

Among this set of initiatives, the Quality Project positions itself at two levels:

• work based on existing internal processes: mapping, systemisation, sharing, and "externalisation"
• external work involving the proposition of methods, such as iterative evaluations with mini-seminars, publications, elaboration of training modules, etc.

The method of iterative evaluations with mini-seminars -a method which was conceived and tested through the Quality Project- is based on the two following hypotheses:

• "Iterative missions" because the time when the evaluation is done will eventually determine what judgement is made of the action, and this question has a heavy bearing on the valuation of quality. The quality of a project is, indeed, not only determined by whether the objectives were reached and whether the planned activities were undertaken or not, but also -and even essentially- by the impact it has (including unplanned effects in the short, medium or long term). In terms of methodology, this means having to carry out evaluation missions over several periods.

In this example:

At T1, the results are very positive (essential needs met)

At T2, the results can already appear less positive (aid dependency, project unsustainable, unplanned side-effects?)

At T3, according to each case, the integrated impact can be negative or positive

 

 

• "with mini-seminars" because the efficiency of the learning process is perhaps as important as the lessons learned from the evaluation mission: it is therefore important that these be discussed collectively as soon and as close to the field as possible, as well as at the head-quarter level. This can be done through the organisation of mini-seminars with the various actors in the field, and upon the mission's return in Europe, where the mission's results are presented and debated.

Following the work started in 1998 on the capitalisation of experiences after Hurricane Mitch, Groupe URD suggested Central America be taken as a case study for the implementation of the IEMS method, within the framework of the Quality Project. The earthquakes in El Salvador, at the beginning of the year, made this capitalisation and consultation process even more pertinent. A series of missions was thus undertaken throughout the year 2001 in different Central American countries that were affected by natural disasters.

Mission n°1 in El Salvador

In June 2001, 4 months after the earthquakes, a team of 3 people went to El Salvador during 3 weeks. They visited projects, and talked to victims, aid workers, etc., in order to take a "snapshot" of the situation and programme issues at stake, and to propose an analysis, in real time, to the national and international actors. This analysis, which was presented and discussed during a mini-seminar in San Salvador, is exposed in detail in the mission report (available on the web); a summary is presented in annexe 3.

Mission Mitch + 3 years in Nicaragua and Honduras

At the beginning of September, a team of 3 people went to Nicaragua and Honduras to visit post-Mitch reconstruction projects and evaluate their impact (beyond the planned effects of programmes). This mission had two objectives: 1) to continue the capitalisation of experiences by adding to it a mid-to-long term perspective; 2) to assist actors of the reconstruction in El Salvador by sharing with them the lessons learned from the post-Mitch reconstruction. (A summary of this mission is presented in annexe 4.)

Mission n°2 Salvador

At the end of September, the latter team thus went to El Salvador, after Honduras and Nicaragua. This second mission in El Salvador, 8 months after the earthquakes, had two principal objectives: 1) to discuss with the different actors the evolution of the programmes and issues related to the reconstruction; and 2) to give immediate feedback of the first conclusions of the mission Mitch + 3 years. This called for new field visits and for the organisation of a second mini-seminar in the UNDP offices in San Salvador. (The summary is included in annexe 4.)

Three junior researchers were mobilised in the field for specific tasks which completed the work done by the project-team.

Non-governmental co-operation and decentralised co-operation in Nicaragua

Javier Calderon, a student at the IEDES, did his internship under a tri-partite agreement between the University of Paris-Sorbonne, Groupe URD and the Fons Catala, a decentralised Catalan co-operation fund. Through surveys (with beneficiaries, local authorities, and co-operation workers) in 4 zones of Nicaragua (Somoto, Esteli, Posoltega, Cuidad Sandino) during 6 months, he analysed the different strategies used and their respective impacts. His university dissertation is available on demand from Groupe URD, and a summary will be prepared and disseminated in the coming weeks.

Mini social audit, "the voice of beneficiaries" in El Salvador

Karla Levy, a Venezuelan student from the University of Paris-Créteil, doing a DESS in management of humanitarian aid and NGOs, is doing her end-of-study project in El Salvador with Groupe URD; she is carrying out a "mini social audit". The objective of this work is to apprehend the perception beneficiaries have of the aid brought by the different actors, following the earthquakes (relevance, speed, usefulness, mode of delivery, etc.). The study is underway, and the results will be communicated shortly.

Partnership in crisis situations, a tool for quality?

Gabrielle Savy, a student from the University of Paris-Créteil, doing a DESS on management of humanitarian aid and NGOs, is carrying out her end-of-study project in El Salvador, in partnership with ATLAS, and under the supervision of Groupe URD. Her study topic is the partnership between ATLAS, an international emergency NGO, FUNDESYRAM, a Salvadorian development NGO, and Horizonte 3000, an Austrian donor. The objective of this study is to determine what is the added value of this partnership at the different levels of the programme (conception, choice of beneficiaries, socio-economic and cultural understanding, implementation, technical choices, etc.). But it also aims to identify the critical elements of this partnership (contractualisation, distribution of tasks and responsibilities, issue of public image, role of the donor, etc.). The study is underway and the results will be disseminated shortly.

These various research activities have been accompanied by a series of complementary actions, which are also relevant to the Quality of humanitarian interventions. Upstream activities include the co-ordination of a network of actors who are mobilised around the project and the preparation of missions. Downstream activities involve the distribution of intermediary results and the dissemination of the approach in the international network.

Though Groupe URD is responsible for executing and implementing the Project, the Quality Project is inter-organisational and brings together a dozen partner NGOs. This group of NGOs has met twice this year to debate and validate the proposed process, and to discuss the intermediary results. The reports of these meetings are available on Groupe URD's web site.

Furthermore, particular attention is given to the dissemination of the approach and intermediary results beyond the French borders, to avoid limiting the project to a purely Franco-French analysis. All the reports on missions in Central America are available in Spanish, and part of the documents regarding the Quality Project are translated in English.

Moreover, Groupe URD has contributed two articles on the subject in specialised publications:

• an article presenting the Quality Project's approach was written for the journal Humanitarian Practice Network (ODI), n°19, September 2001, "Debating accountability", by François Grünewald, Claire Pirotte, and Véronique de Geoffroy.

• A chapter (p.153-171) of the book Evaluating International Humanitarian Action -published by Zed Books (London) and edited by the ALNAP network (Active Learning Network on Accountability and Performance)- describes the approach used for the capitalisation of post-Mitch experiences and introduces the Quality Project. This book compiles diverse experiences with evaluation processes of humanitarian interventions (ICRC, UNICEF, DIFID, NGO, etc.) in various areas of the world (Somalia, Cambodia, Rwanda, Kosovo, Nicaragua, etc.) and the lessons to be learned from these processes.

The project team was invited to speak in several international events on the Quality of humanitarian action. This participation represents both the first results of the Project and the preliminary steps which are necessary for its implementation.

• Within the framework of ALNAP's activities, the Quality Project was retained as one of the initiatives to be "followed up". ALNAP indeed aims to create a space for dialogue and debate between actors -donors, United Nations agencies, Red Cross, NGOs, universities- on research projects linked to evaluation practices, institutional learning and accountability. This led to two presentations during ALNAP's biennial meetings in 2001: a presentation on the IEMS approach in April, in London, and another in October, in Washington, where the first results of the method, tested in Central America, were presented.

• Also, in October 2001, the Dutch Ministry of Foreign Affairs organised a seminar with the University of Wagenigen on "Enhancing the Quality of Humanitarian Assistance", during which the Quality Project was mentioned several times. The project-team was invited to participate.

• In November 2001, ECHO asked Groupe URD to chair a session on the "relevance of indicators for the measurement of Quality" during their internal training week. This presentation, which was addressed to the field experts representing ECHO in crisis-affected regions (around 50 participants), was very much appreciated and many participants asked to be kept informed about the future developments of the project.

• Finally, 2 people from Groupe URD will participate at the annual meeting of ECHO's partners (signatories of the CCP), where the "quality of humanitarian action" is one of the issues to be discussed.

Though the training activities will be fully developed only in the final stage of the project, once the methods and tools will have been validated, a module introducing the debate on the quality of humanitarian assistance was conceived and tested, notably with interns from BIOFORCE. This module presents the terms of the present debate on the subject, as well as the foundations and first results of the Quality Project.

A training module on the set of work done in Central America, which includes a case study on "Mitch", is available; it will be presented in November 2001 to the students of the multi-faculty diploma of the University of Geneva ("development in high constraints zones").

The issue of quality of humanitarian action has started to concern more than the circles specialised on the subject, and it must be approachable by other publics. Groupe URD has therefore participated in the events around World Food Day in October 2001, in particular by chairing a debate on "the quality of food aid".

The Quality Project is presently awaiting funding. Applications for funding have been deposited with the French Ministry of Foreign Affairs (MCNG), with ECHO (on the "studies and research" budget line), and with the Swiss Ministry of Foreign Affairs; the partner NGOs have committed themselves to contribute to the budget.

The years 2002 and 2003 will constitute the 2 years of intense research for the project, with theoretical and practical work. The outcome should be a proposition of methods and concrete tools for action (c.f. project document).

In the short term, the activities which are currently being prepared are the following:

• Finalisation of the work on Central America. This includes a mission at the beginning of December 2001 in El Salvador, the organisation of a workshop with the actors in El Salvador, the restitution of the work underway (results of the "mini social audit", of the study on partnership, etc.)

• As soon as the future funding is assured, at least partially, a first activity will be proposed for the beginning of the year 2002. A discussion and training seminar will be organised with the support of Dr. Hugues MAURY, an expert on the issue of quality in hospital environments, who has already been supporting and guiding the project-team for some months. The goal of this seminar, organised for the Quality Project partners, will be to share internal practices (as a continuation of the work already achieved this year), to define terms (quality, certification, labelling, etc.) and to think collectively on the orientations of the Quality Project.

• The Project will certainly concentrate on the case of Afghanistan, which had been suggested as a potential case study for contexts of chronic crises in the project document; it would then be retained as a case study for contexts of acute crises. The IEMS would then be implemented in collaboration with the partner NGOs.

In the medium term, the theoretical work will continue, notably the research on the history of quality processes in other domains (services, or the productive private sector).

A certain number of existing methods used by some NGOs that seem particularly relevant, such as the International Rescue Committee's "Guide for programming, conception, monitoring and evaluation" -a novel method for team writing of field programmes and mid-term projections, will be tested.

Contextual studies on different types of crises will be carried out, as planned in the project document.

Finally, tools will gradually be developed, and tested in the field, such as the "questionnaire on key elements". This questionnaire came out of the analysis of "the most common and most serious errors" observed in the field, which was transformed in a list of key questions to ask when elaborating and monitoring a programme.

The debate on quality used to be, a few years ago, exclusively focused on technical indicators. Today, it has become much larger and richer. The issues surrounding the responsibility of the various actors -including donors- in programme implementation are now taken into consideration. The question of the measurement of programmes' impact, and not only that of objective achievement, is addressed. These topics make tackling the issue of quality definitely more complex, but also, and essentially, they bring the approach closer to the reality of situations.

This evolution of the debate at the international level was recognised, in the last ALNAP meeting in Washington, to be the result of 3 years of mobilisation by the French NGOs, through the Quality Project. Today, Groupe URD is strongly committed to pursuing its research work with the Quality Project's partner NGOs, in order to define, in the near future, methods and tools to improve the quality of humanitarian interventions.

Supervisor: Dr. Claire PIROTTE, Groupe URD

Office and equipment made available by the French Red Cross

Field study in Anjouan (Comoro Islands), on an Aide Médicale Internationale mission

Living allowance paid by Groupe URD

Length of internship: from June to October 2001.

• review of the history of Quality processes in hospital environments
• creation of a questionnaire for NGOs
• interview of 8 NGOs (ACF, AMI, CRF, HI, HSF, MDM, MSF, PSF)
• writing up and summary of interviews
• field mission in Anjouan
• writing up of mission report
• report-back to the partners of the Quality Project

• mapping of the tools used by NGOs (interview reports and summary tables)
• university dissertation

A basis for quality processes exists in nearly all the NGOs met, even if these processes are not always formalised.

Among small NGOs, the perception of the quality issue varies considerably according to individual profiles (job, type of programme, etc.) and remains divided by sector (drug quality, justification of accounts…).

The bigger NGOs have a more global approach, which is based on the principle that the quality process should be transversal.

Each NGO has an internal discussion on how to improve tools and methods, but each focuses on very different actors, and with different objectives: donors, to justify one's credibility; other NGOs, to continue existing; the beneficiary population, to take on one's responsibilities… These considerations lead to the implementation of actions to improve quality, but they are not followed by internal evaluations, an indispensable phase for the improvement of quality.

The notion of quality in humanitarian organisations is based on the objectives and on the actors the NGOs target through their programmes. An NGO that focuses heavily on the principle of accountability will not follow the same approach as an organisation that emphasises action only. It seems certain, nevertheless, that neither of the two will have "good quality" programmes. The first one will not satisfy its beneficiaries, while the second, even if its actions are relevant and efficient, will not survive long without the financial support of donors.

As for the actors involved, it is important, when implementing a quality process, to carefully define the beneficiaries. For example, on the AMI mission in the Comoro Islands, the programme's objective was apparently to improve the population's access to health care; however, the activities focused on improving the running of the hospital and on training the hospital staff, without this systematically having direct consequences on the population.

Rather than answering to the "initial humanitarian disorder" observed on certain programmes through a liberal analysis that imposes the application of supposed universal norms, which are not adaptable to all contexts, it seems interesting to propose a systemic analysis corresponding to a regulated system. Thus the following elements could be taken into consideration, with more or less weight according to the context and nature of the crisis: the donor constraints, the needs of beneficiaries, the political will of NGOs and the donors' preferences.

The tools that would be used, regardless of their form, would be supports for discussion, training, capitalisation, organisation, etc. They should be specific to each NGO, according to its objectives and constraints. Each organisation would then put into place quality processes within its own structure. A common document would then be elaborated based on each one's experience. This common document could be a manual for reflection on the "tools and the quality process" implemented in each organisation (for example, on responses to emergency situations, on project implementation, on the training of expatriates) with questions (e.g. "is there a security policy?") which would be answered differently according to the context. Questions such as "how is security ensured on missions?" could complete this manual with answers which would be variable according to the NGOs in their headquarters (organisation of the purchasing department) or to the contexts in the field (modality for food distribution).

Junior researcher in charge: Morgane LE GUENIC, student at the University of Economics of Rennes I, DESS "Evaluation of industrial, agricultural, social and environmental projects".

Supervisors : François GRUNEWALD and Dr. Claire PIROTTE, Groupe URD

Offices and living allowance from Groupe URD

Length of internship : from June to October 2001.

• review of the existing work available on the Net
• creation of a questionnaire for NGOs
• interview of 8 NGOs (ACF, HI, PU, GRET, CARE, MDM, A&D, CAM)
• write-up and summary of interviews
• write-up of dissertation
• report-back to the Quality Project's partners

• university dissertation
• mapping of evaluation practices and of the problems faced

Evaluations are increasingly common in the work of humanitarian actors, but they still present gaps compared to the evaluation processes used in other sectors.

Evaluations are not systematically carried out, despite the fact that, as a learning tool, they are particularly beneficial for organisations: their objective is indeed "to evaluate in order to evolve".

Furthermore, an evaluation is not always planned in the design of the project, which makes the evaluators' task more difficult. In this case, if the person in charge of the project is not informed beforehand of the coming of the consultant(s), s/he may be reticent to assist with the evaluation process, a process already made difficult by the fact that the financial resources and the time needed for its proper implementation could not be anticipated.

Moreover, an evaluation can represent a "breaking point" in the concerned project. It is therefore important to deploy all the means needed to carry out the evaluation without disturbing or interrupting the field workers' activities.

There is a lack of consensus, among the aid community, on the definition of key concepts concerning evaluation, a term which itself carries several meanings. Evaluation, impact, etc. can all have slightly different meanings, according to the consultants who use them. This prevents comparative analyses of evaluation reports from being done, and hinders the learning process.

Finally, there is often a lack of co-ordination, collaboration and information sharing between actors, such that the conclusions of an evaluative mission can be uncertain and imprecise.

It is advisable to clarify the definitions of the terms used in evaluation processes. It would be most productive, for emergency and development workers as a whole, to speak a common language, to reinforce the co-ordination and collaboration between the various actors in the field, and to facilitate the sharing of information, knowledge and skills. This would greatly improve their access to all the information they need.

Emergency and development aid workers -whose activities the last decade has shown us to be increasingly interconnected- would benefit greatly from sharing their knowledge on evaluation processes; this would help to improve and reinforce their evaluation methodology and, thereby, the quality of their programmes. The Participatory Approach tools, and especially the participatory evaluation method, which are widely used in the development world, would certainly be very useful -with certain adaptations- for emergency aid workers.

In various interviews, two additional points concerning the Quality Project stand out:

The actors all wish to reinforce their methodology for qualitative evaluations, in order to integrate the point-of-view of their beneficiaries, who are the ones able to validate the pertinence of the action. This approach must use precise and rigorous techniques (which still need to be developed) in order to be productive, otherwise the results and their interpretations would not be reliable. It is important to convince the decision-makers that qualitative evaluation is an innovative approach worth investing in.

It is therefore essential to reflect on the definition of evaluation criteria, which would not be solely technical, and which would take into consideration the specifics of a context and population. It must be noted that the lack of data and information is a recurrent factor, and that the pertinence of an evaluation mission can be limited if the consultant does not have good qualitative knowledge of the environment in which he is operating. Furthermore, it is crucial to place an action in its context: a quantitative evaluation, when isolated, cannot cover all the aspects that deserve to be addressed when dealing with quality. A quantitative evaluation used on its own, can only lead to approximate conclusions. In order to represent reality and the complexity of the field situation accurately, it is crucial to combine these two approaches.

El Salvador has been in a state of chronic crisis for decades. This situation prevents any new, more equitable and sustainable, development strategies to be initiated for the moment. El Salvador is under the control of large international institutions, such as the IDB (Inter-American Development Bank), which offer loans without special conditions and impose a model of development based on the triad: "decentralisation, participation and privatisation". The successive governments have led a neo-liberal policy favouring the most "competitive" sectors and actors in the country. For example, each governmental development institution, such as the FISDL or the FODES, supports the towns with the highest potential. As for the poorer communities, they find themselves with the responsibility of managing their own development, without receiving any particular support. The question of access to basic services was already alarming before the earthquakes. Yet it is an indispensable prerequisite for access to citizenship. The successive privatisation of certain services have caused heavy price increases (up to +300%), thus excluding an even more important part of the population.

The civil society is mobilised on the political scene through various NGOs and foundations. The latter also carry out projects with the most vulnerable communities, though they have the feeling of not being able to achieve much, compared to the scope of the needs. Faced with the government's policy, they feel powerless.

After the tropical storm Mitch, which had partially but deeply affected El Salvador, the country was hit by two earthquakes of high magnitude, over a period of 30 days, at the beginning of the year 2001. The numerous tremors recorded throughout the first semester of the year have had two major consequences: the destruction of a third of the housing, and a disruption of the psycho-social order, which was already precarious.

This "socio-natural" disaster, is above all a rural phenomenon, since the majority of the damage occurred in rural, often very remote, areas. The earthquakes revealed the precariousness and vulnerability of a world suddenly made visible. In a context of commercial and economic liberalisation, the gap and the inequalities between the urban and rural worlds, between people having a real access to citizenship and those excluded from all basic services, were made even more apparent.

Faced with the disaster, the government's approach was to bank on continuity. The earthquakes were perceived as simple natural disasters, which were temporarily upsetting the smooth-going development process. There was therefore no question of raising doubts about the paradigms that dictated the country's policy, and even less of considering social and economic vulnerability as a major cause of the destruction engendered by the earthquakes. In official declarations, the international community was nevertheless presented with a series of good intentions: satisfaction of the population's most immediate needs, supply of basic services, support to small and medium businesses, etc. But in practice, competitiveness criteria prevail over the reduction of inequalities; aid is distributed along partisan lines; and economic development is based on "economic centres", which are essentially composed of maquilas (types of free zones, for textile in particular). This makes El Salvador a reservoir of cheap labour for multinational corporations. The civil society remains marginalized, as proved by the absence of Salvadorian forums during the meeting of the Consultative Group in Madrid in March 2001, where the debate focused on the sums of money the government could collect.

International bilateral aid was rapidly mobilised. With much publicity, more or less impressive sums were announced for El Salvador. These generally do not represent much since the sums are calculated as loans, as expert consultations, as reduction of interest rates on debt, etc. The debates in the Legislative Assembly today focus more on the debt issue (facilitated by the IBD) than on the use of the funds received for reconstruction. For Mitch, El Salvador had received only 25% of the promised aid. For the earthquakes, the sums received may correspond to less than that. The response to the consolidated appeal launched by the UNDP is telling: less than one million dollars were received compared to the 40 million that were requested. WFP faced similar problems.

Nevertheless, NGOs, international aid organisations and certain large international institutions reacted. For some, the Mitch experience was the opportunity to capitalise their experiences, and a number of humanitarian practices were modified: non-intervention in albergues, refusal to create new urban centres, direct payment of funds to certain NGOs, etc.

The population is undergoing an epidemiological transition, between a developing country profile and that of a newly industrialised country: homicides, lesions, suicides and chronic degenerative diseases are the main causes of mortality. The health system, which is currently undergoing reform and decentralisation, is deficient in terms of service quality, coverage, and operational capacity.

In this context, the direct impact of the earthquakes on the population's health was very concentrated in time (few deaths, no epidemics). But if the health structures, which were already overloaded before the earthquakes, were greatly used during the time of the crisis, the indirect impact due to the destruction of numerous infrastructures will be very important over the medium term.

Relatively few organisations conducted direct health interventions. The elements that determined a project's success were the quality of human resources, a good understanding of the context and close collaboration with health units.

In a first stage, the assistance allotted to the health sector consisted essentially of emergency medical equipment, field hospitals, drug donations, and support for the water supply. These actions generally covered the most urgent needs, in the short term, but they did not offer prospects for the reconstruction of the health system.

International aid organisations and NGOs concentrated most of their efforts on mental health programmes. This sector is particularly popular in the field of international co-operation, though it suffers from great methodological gaps and from an absence of capitalisation of aid workers' experiences. Though NGOs interventions in this field are legitimate during the acute phase of the emergency, to help families overcome their trauma, it may be more difficult to justify such projects five or six months after the crisis. This type of project cannot be "self-sustainable"; they risk creating a new need and overburdening further an already deficient health system. Furthermore, the structural causes of psycho-social disorders are often more linked to deplorable living conditions than to "self-esteem disorders", which call for heavy psychotherapy.

The following table summarises the principle observations and recommendations made by the mission:

This sector is particularly important because of the leverage effect it can have on the improvement of living conditions, and also because it can help reduce the migration of Salvadorians to the United States.

Agriculture faces the problem of the concentration of land ownership. The resistance of large landowners and the lack of political will have impeded agrarian reform programmes. Moreover, El Salvador has relatively little fertile land, which makes the production costs of staple products very high. Programmes for the diversification of production, which aim to favour profitable crops (cashew nuts, peppers, etc.), are confronted with the resistance of farmers and with the monopolies that control the trade networks.

The impact of the earthquakes will have been considerable, but indirect. It has less to do with the agricultural losses than with the agricultural activity itself, since the availability of labour for the harvest was reduced.

Food aid programmes have faced difficulties with monitoring due to a lack of human resources, to the difficulty of evaluating the damage in certain zones (because of a lack of community organisation), and to the partisan approach of the National Secretariat for the Family (state organisation responsible for the logistics of food distributions). The World Food Programme sub-contracted the food distribution to 4 NGOs that distributed the food aid to other organisations in the field.

The pertinence of certain projects, such as Food-for-Work (FFW), is also questioned. The FFW projects underway in rural areas during the sowing season (May, June) face difficulties when they are in competition with the livelihood activities of populations. FFW was essentially used in shelter construction, sanitation projects, and clear-up work.

Income generating activities seem to be decisive factors in the reconstruction process since they have an effect on access to shelter, and on the improvement of health, when they are carefully supervised. Though there is a consensus on the importance of giving technical and financial support to micro and small businesses, the means allotted by the international community for this purpose seem insufficient. Salvadorian NGOs that are involved in micro-credit are short of funds, even to help people who make a living from this type of activity.

The issue of shelter, or more particularly housing, suddenly became the main concern of agencies that had never worked on this question before. The problem of shelter had already been looming for years in this country. The civil war and the tropical storm Mitch have only worsened the existing housing shortage and "favelisation" in El Salvador.

Part of the aid community thus fell in the trap of targeting the most visible problem, and gave in to the opportunism prompted by the large sums this type of disaster can attract. The most visible aspects were addressed, often in haste, without considering the structural causes. Some NGOs did not wish to stay behind the general movement focused on shelter, and, instead of sticking to their usual field of activity, got involved in heavy housing reconstruction programmes. A large number of NGOs committed themselves to the reconstruction of all types of housing, without thinking ahead about the process through which "human settlements" are built, can function, and can be integrated in an urban framework and socio-economic dynamic.

Yet, the role of "facilitators" on the question of shelter is important. The modes of intervention in shelter are very diverse. But construction activities were favoured over types of assistance that strengthen local communities' capacity.

The quality of programmes is very variable, and depends on the level of reflection on the concept of housing, on the degree of participation, and on the partnerships put in place. We are thus witnessing a levelling down of housing quality in El Salvador. Though far from the Salvadorian culture, the basic model, of the champa type, is apparently being irreversibly imposed as the model of popular housing in El Salvador, after the last disaster.

The following table summarises the main observations and recommendations.

The transversal analysis of the different sectors shows that the following issues are still insufficiently addressed in emergency interventions: understanding of contexts (including the analysis of structural vulnerability), partnerships with local actors, inter-agency co-operation, and strategies favouring sustainability (including the participation of concerned populations).

Community participation, the leitmotiv of each project, is a "carryall" term with no real meaning. Participation is often considered by agencies as a means to reduce projects' recurrent costs, by using victims as cheap, unskilled labour. There is no real learning and organisational process around a project. Furthermore, the "community" is often viewed, with a certain idealism, as a homogeneous entity or group capable of coming together for any reason. One often forgets that a community is an extremely complex social and economic structure, composed of conflicting individuals and interests. The integration of social sciences in the identification and implementation of interventions could be a way to improve the participation and organisation of social groups.

Collaboration between international aid organisations and Salvadorian NGOs is often lacking. Yet they are truly complementary. Partnership experiences are nevertheless rare.

Four types of partnerships are generally initiated:

In the light of this first mission in El Salvador, the debate on the need to modify humanitarian practices and to use new tools in emergencies remains topical. Meanwhile, the issue of the link between emergency and development also emerged. The next mission, and the co-ordination and sharing processes that will go with it, should enable us to go further. The fact that the team will carry out an evaluation "Mitch + 3 years" in Nicaragua and Honduras before going to El Salvador for the second mission of iterative evaluations with mini-seminars should further enrich the debates.

August 1999 : A first evaluation mission carried out in August 1999 draws a first set of teachings and proposes a number of hypotheses to be verified at a later date. A first series of mini-seminars are organised in Managua. The latter are followed by several restitution exercises in France, in particular through workshops during which the results of the mission and their implications for NGOs are discussed.

January / February 2001 : Already affected by a decade of civil war and by Hurricane Mitch, El Salvador is devastated by a series of earthquakes which affect both rural and urban zones. Aid is rapidly mobilised, but in a relatively limited way, since there is competition with another natural disaster: the earthquake in Gujarât.

The results of the reconstruction projects in Nicaragua and Honduras, 3 years after Hurricane Mitch, seem mixed. The hypotheses raised in 1999 were largely confirmed:

• Certain relocation sites created after Mitch are deserted by more than 30% of the population. In these cases, houses were at worst abandoned, at best resold at low prices.
• Other operators accumulated much delay, by wanting to do "too well", such that the houses and sites created are not adapted to the needs of the populations;
• Other projects, however, are true successes; the reconstruction activities served as a basis for new, long-term development projects, which are currently underway.

The variability in the level of success can be explained by different internal and external elements, technical or not.

In El Salvador, 8 months after the earthquakes, already a few problems or risks appear, which can be explained at different levels. Most hypotheses elaborated in the June 2001 mission (first mission in the iterative evaluation with mini-seminars process) were confirmed in the second mission.

• After Mitch, reconstruction projects that were too rapidly elaborated and implemented tended to be failures. In El Salvador, the strategy was to build temporary housing as part of an emergency approach. The local dynamics show that if there are no follow-up and support programmes, the temporary can become permanent;
• The fear of being confronted with the same land ownership problems as in Nicaragua or Honduras after Mitch led many humanitarian actors to seek a risk-free option. Organisations thus favoured programmes for people able acquire to land ownership titles. Because they wanted to limit the potential problems, NGOs neglected the most vulnerable populations, precisely those who did not have access to land ownership.
• The donors and the delays they impose are often the main causes of the haste with which programmes are implemented, which often affects their quality. Each time a donor granted sufficient time, and accepted to finance preliminary socio-economic evaluations, the chances to carry out good quality projects were maximised.
• Each time the "population displacement" option was preferred over reconstruction on the site of origin, many problems arose, which will be very difficult to manage.
• the breeze block options which sometimes prevailed could have been replaced by reconstruction strategies favouring local materials and know-how.
• Each time an emergency operator made an alliance with a local development actor, the context, its constraints, and its potential were taken into consideration much more. Here too, large gains in quality were made possible.

The following table presents an empirical analysis done in several steps. It uses a sample of the field observations by listing the main themes observed and attempts to explain their cause at two different levels, in order to deduce principles for action and tools for the improvement of the quality of interventions.

From this table, different levels of analysis on the quality of humanitarian action appear. The determining factors can be external or internal to the organisation. Some are linked to the context, others have a wider scope :

1. Determining factors external to the organisation
1. related to the context: political environment (in this case: access to land), socio-economic status of the targeted population, geographical environment
2. related to donor pressure: programme deadlines, available funds, etc. ;

 

2. Determining factors internal to the organisation
1. organisational culture
2. point-of-view on the "disaster phenomenon", is it an accident or not?
3. Awareness and/or consideration of the different factors related to shelter (which should lead to a recruitment and training policy)
4. NGOs impose deadlines for themselves, under the media pressure: how can we favour information focused on education over information based on emotional responses?

We note here that the quality of programmes is partly related to the tools used, but also to a number of external factors and ingredients which cannot necessarily be translated into tools: general and professional ethics, cultural references, etc.

A house reconstruction programme will thus be different according to how one views housing: is it only a roof, or is it the result of a complex process which integrates social, cultural, and economic aspects?

This approach to housing implies:

• That donors must accept constraints related to time, and do not impose deadlines that are incompatible with the actions to be undertaken.
• Working as close as possible to the previous living area = reflecting on risk reduction activities rather than displacing populations.
• Working with partners with different operational approaches.
• Reflecting on the suitability of technical propositions to the context and local population's capacity (cost, evolution, reproducibility with few resources, etc. but also risks of creating jealousy and rivalries).
• Managing time and defining "emergencies" differently: only the first 48 hours are the true emergency period. At this stage, only the local structures are present, and only local assistance works: the work must thus focus on the preparation level. As soon as this phase is over, development tools must be used to support the rehabilitation process. Housing reconstruction cannot, then, be managed in the haste of an emergency.

It appears clearly that, in a learning perspective, the method of iterative evaluations with mini-seminars has considerable potential, including an interesting cost-effectiveness ratio. On the other hand, the current test has highlighted considerable margins for improvements, especially in the following:

• the preparations of evaluation missions
• the organisation, form, and content of the mini-seminars

Too much confidence was granted to the internet liaison between both international and national actors. The lists of addresses were sometimes incomplete, or contained errors. The lack of time for the preparation of the second feedback meeting in the field led to a certain frustration among actors, who had not received the first mission report, or who received the invitation for the second mini-seminar too late. These errors should be avoided in future missions.

The advisory part of these evaluations works well when sufficient time is granted to bilateral discussions. On the other hand, it can be feared that the lack of flexibility of programmes and donors makes it difficult to implement the changes recommended by the evaluation.

Whereas the mission in 1999 arrived in the field 8 months after Mitch, the fist mission in El Salvador using the method of "iterative evaluations with mini-seminars" took place 4 months after the earthquakes. It seems that it is necessary to arrive even sooner. This will be done for the next test, which should be on the Afghan crisis (mission planned for December if the funding is mobilised on time).

Before proposing practical solutions (methods, tools, recipes), it is necessary to better understand the origins of the problems with the Quality of Humanitarian Action, through a holistic approach which places humanitarian actors in an environment which partly determines the quality of their actions (relations with public and private donors, local actors, political and budgetary constraints). The issue is not to clear humanitarian actors of their share of responsibilities, but to have a global and holistic approach to the problem.