Pakistan: Final evaluation of Concern’s Community Based Disaster Risk Management Programme in Jacobabad, Jaffarabad and Layyah districts of Pakistan

1. Background

Concern Worldwide is an international, non-governmental, humanitarian organisation dedicated to the reduction of suffering and working towards the ultimate elimination of extreme poverty in the world’s poorest countries. Concern has been working in Pakistan since 2001, when it initiated an emergency response programme to address the Afghan refugee crisis in Balochistan. Concern later moved into emergency, early recovery, and long-term development programming in Balochistan, Khyber Pakhtunkhwa (KP), Punjab, Sindh and the Federally Administered Tribal Areas (FATA).

Concern has been implementing the USAID/OFDA-funded Building Resilience through Community-Based Disaster Risk Management (CBDRM) programme in Dadu, Muzaffargarh, Charsadda, Rajanpur, Layyah, Jacobabad and Jaffarabad districts of Pakistan. The basic objective of this programme is to improve the disaster risk management capacities and strategies of the poor and vulnerable communities and other key stakeholders through sustainable preparedness, mitigation measures and increased institutional capacity. The programme is in line with national DRR and USAID/OFDA’s priorities aimed at saving lives; protecting livelihoods, assets, and infrastructure before, during, and after disasters; and increasing communities’ resilience to recurrent hazards.

Major activities planned under the programme are formation of village and UC level disaster management committees (DMCs), UC level emergency response teams (ERTs), School Disaster management Committees and participatory risk analysis and development of village and Union Council (UC) level DRM Plans, school disaster preparedness plans, community mobilization, awareness raising and capacity building, provision of emergency contingency stocks, small scale flood proofing/mitigation measures and advocacy for the integration of DRR in development processes. The programme interventions are designed to reduce the loss of lives and livelihoods, safeguard land and infrastructure, increase resilience, and lessen economic disruptions, by including activities at the community level that mitigate impacts of natural hazards by preparing for, prevention, and planning for disasters. Through fostering linkages across sectors and across organizations, and by improving understanding and implementation of hazard management, the programme will help reduce people’s vulnerability to potential disasters. The programme ensures the integration of gender, protection, considerations for disabled and elderly, HIV and AIDS, and is sensitive to all other considerations of the affected vulnerable populations.

2. Purpose and Objectives of the Evaluation

The overall purpose of the evaluation is to assess the degree to which the ‘Programme’ met the objectives as outlined in the programme proposal, with particular emphasis on appropriateness, timeliness, efficiency, effectiveness, impact and sustainability of the interventions carried out. The evaluation is also aimed to assess qualitatively as well as quantitatively, the programme achievement, on cost benefits analysis and social return on investment.

The specific objectives of this evaluation are:

· To assess the appropriateness, efficiency, effectiveness, impact, and sustainability of the programme in Jaffarabad, Jacobabad and district Layyah.

· To capture main lessons to be learned for future similar programme.

· To suggest practical recommendations for replication of the programme in different context of the country (recommendations need to be specific, practical/feasible and achievable)

· To assess Costs and Benefits / Social Return on Investment comparing losses/ damages to wide ranges of assets (including economic, social as well as environmental etc.) in floods/ disasters before and after implementation of programme

The evaluation will have a participatory nature and will focus on the following key fields and evaluation questions related to the programme results /achievements (not on the activities):

Appropriateness of the programme design:

· To what extent the programme is suited to the particular needs, expectations and priorities of the target communities, local authority, and the donor?

· How much the programme is aligned with the National, Provincial and District Disaster Risk Reduction priorities?

· Does the programme meet its stated objectives?

Efficiency in use of resources:

· Could the same or better results have been achieved with same or lower inputs or by doing things differently?

· Are the effects being achieved at an acceptable cost compared to alternative approaches of accomplishing the same objectives?

Effectiveness of programme interventions:

· Did the activities achieve satisfactory results in relation to stated objectives/results? How did the programme perform against the log frame indicators? Were there any unintended outcomes?

· To what extent have the activities contributed to enhancing local capacities and if not why?

· What is the effectiveness of a) structural mitigation measures and b) Early Warning System?

Sustainability:

· Is the program likely to continue after the end of the external funding?

· Are there sufficient forums /structures capabilities to maintain the changes produced over time?

· Are the decisions making processes by men and women likely to be continued into the future, and how/why?

· How was the exit strategy defined, and how was this managed at the end of the funding period?

Impact:

· To what extent the programme contributed to minimize losses to various assets including livelihood, shelters, WASH, infrastructure at community level.

· How this programme contributed to communities’ resilience for disasters through development of DRM Plans, Awareness Raising and providing emergency stocks.

· How the programme contributed in preparing communities to cope and withstand in disasters?

· How this programme improved linkages of the communities with other stakeholders in particularly with district line departments.

· How the programme contributed in gender equality and empowering women to take active part in the disaster risk management activities.

· What are people’s perceptions about this programme’s design, delivery and durability of its tangible and intangible assistance/ activities?

Cost Benefit Analysis and Return on Investment:

· What is the cost benefit of carrying out CBDRM activities versus emergency/ disaster response? By analysing the costs associated with losses to assets and livelihoods brought about by disasters (in selected villages that have had disasters before/during/after implementation of programme) against the cost to implement CBDRM activities

· Community Empowerment: Reviewing the programme inputs, outputs, outcomes and impact experienced by the programme participants, what is the social, economic and environmental value created (putting a monetary value on the benefits) as a result of the interventions?

Integration of Cross Cutting Issues:

· How well did the programme integrate gender equality, HIV & AIDS, disability and environment?

· To what extent was ‘accountability to beneficiaries’ promoted and the progress made against the achievement of HAP (Humanitarian Accountability Partnership) principles/benchmarks? To what extent were complaints addressed?

· How the advocacy interventions of the programme contributed to the overall community empowerment?

· What opportunities the impending local government system offers to this programme and how best the interventions can benefit from those opportunities?

· How and to what extent did Concern’s partnership with Local NGO partners affected achievement of programme objectives? Has the project enhanced the capacity of the local NGO partners?

Replicability:

· What aspects of the programme are replicable in similar areas in Pakistan and elsewhere?

· Under what circumstances and/or in what contexts would the programme be replicable?

Lessons learned, information sharing and dissemination:

· Were there any significant changes in the programme design or the programme context? What were the reasons for these and can any useful lessons is learned from this for application elsewhere?

· For whom could these lessons have relevance?

Recommendations:

· Targeted recommendations for (a. beneficiaries, b. Concern and implementing partners, c. government and d. donors) for improvements based on findings during the evaluation process (e.g. for sustainability, future programme design).

3. Methodology:

The evaluation process should comprise of mixed model approach to collect both quantitative as well as qualitative data. The process should include:

· A desk review of programme key documents listed below

· Primary data collection from sampled project beneficiaries to assess the effectiveness, efficiency and impact of the programme activities at household level and community level.

· Focus Group Discussion (FGDs) with beneficiaries, stratified on the basis of nature of different activities, for triangulation of data and other findings. The process should be participatory to the extent possible.

· Interviews with project staff (Concern and Partner) and stakeholders including government departments (DDMA, PDMA and NDMA) and from the representatives of National DRR forum to collect information on achievements and impact and difficulties faced.

· Meetings with the senior management of Local NGO Partners to document their perception about change in the organisational capacity as a result of this project.

· Presentation of draft findings to a) field/partners’ staff and b) Islamabad based staff. These feedback sessions will help finalize the conclusions for the report

· Submission of the draft evaluation report to Concern for feedback. Incorporation of Concern’s feedback in the draft report. This should have a 3 page executive summary designed as a stand-alone document as well.

· Submission of final report following comments received from Concern on the draft report within the time frame agreed.

4. Deliverables: The evaluator(s) will produce/submit a report in hard and soft form of no more than 45 pages (see Annex R for the reporting template) along with relevant annexes (in Microsoft Word using Arial font 11). The report should include:

· Basic information (1 page)

· Executive summary (max. 3 pages)

· Introduction/background of the project (1 page)

· Evaluation methodology

· Major findings against the objectives of the evaluation (i.e. serial number 2 above)

· Summary of targeted recommendations/lessons

· Annexes: Evaluation ToRs, evaluation schedule, list of persons interviewed and sites visited, documents consulted declaration of independence from the project team, data collection tools and raw data, and the updated project logframe.

Note: Main evaluation report (without annexes) will not exceed 35 pages to keep audience straight to the main point.

5. Documents to be shared with the evaluators

· The approved project proposals

· Baseline Report, Knowledge retention survey reports

· NDMA’s vulnerability assessment survey

· UCs Level DRR Plan

· Programme reports, including financial information

· Surveys reports (knowledge retention, effectiveness of early warning system)

· Case studies

· Reports/ policies of Government departments and other stakeholders

· Concern’s HCUEP (How Concern Understands Extreme Poverty) Document

· Concern’s PME guidelines and Strategic Plan and other related policies/documents.

6. Duration: The consultant will complete the work over a period of thirty (30) working days beginning, tentatively, in the Last week of January 2016. (please note the start/end date of the assignment may change due to unavoidable circumstances in which case a revised timeframe will be drawn up with the mutual agreement of both parties)at the date of signature of the contract and ending with the acceptance of the final report.

7. Reporting Line: The consultant will report to Concern’s Programmes Director and will liaise closely with the DRR Adviser, MER Coordinator and Programme Managers.

8. Consultants Expertise

· Advanced University degree (at Master’s degree level or higher) in Disaster Risk Reduction, Climate change, Environment, Research and/or related field.

· Proven knowledge of development sector particularly in the fields of community resilience along with the knowledge of financial and economic analysis.

· At least 5-8 year experience of conducting evaluations of development projects especially CBDRM related projects

· Experience of carrying out Cost Benefit Analysis and Return on Investment (ROI) studies

· Experience in the use of participatory methodologies and developing gender sensitive evaluation methodologies

· Familiarity with DAC criteria for evaluating development assistance

· Excellent analytical, facilitation and reporting skills

· Knowledge of the Government of Pakistan’s DRM policies, frameworks and architect

· Good understanding of the local context, knowledge, culture and languages will be an advantage.

Note: This consultancy is open to both National and International applicants. Individuals, group of individuals as well as consultancy firms having the requisite skills/ experience are eligible to apply. International applicants having prior work experience in Pakistan with an existing Visa/NOC to work in Pakistan are encouraged to apply however Pakistani counterparts with ability to freely move across the country would be essential.