CRAM - Concern Worldwide

Final Evaluation Community Resilience to Acute Malnutrition (CRAM) Programme Chad 2012-2015 Irish Aid Programme Evaluation September 22nd - November 15th 2015

Evaluation team Jenny Hobbs Senior Education Adviser Strategy, Advocacy and Learning, Concern Worldwide, Dublin

Alexander Carnwath Community Resilience Policy Officer Policy and Campaigns, Concern Worldwide, London

Programme Scores against DAC criteria Criteria

Score Highly satisfactory 4, Satisfactory 3, Acceptable but with some major reservations 2, Unsatisfactory 1

Relevance

4

Efficiency

2

Effectiveness

3

Impact

3

Sustainability

2

Acknowledgements This evaluation took place between September and November 2015 through two separate visits. Alexander Carnwath travelled to N’Djamena from September 27th to October 4th to evaluate macrolevel advocacy results. Jenny Hobbs travelled to N’Djamena and Goz Beida from October 25th to November 8th to evaluate programme progress against all other objectives. The evaluation was possible through the support and invaluable input of the Concern Chad team: Arjan Ottens, [email protected] Sandrine Robert, [email protected] Isaac Gahungu, [email protected] Adolphe Mubenga, [email protected]

Special thanks to Aicha Ousmane, Moussa Kolla and Ngaboye Allaressem for providing skilled translation during interviews and focus-group discussions. Thanks also to partners within government agencies, and to the drivers and wider Concern team in Goz Beida.

For further information on the review, please contact Jenny Hobbs: [email protected] For queries specific to advocacy in this evaluation please contact Alexander Carnwath: [email protected]

i|Page

Contents Acknowledgements.................................................................................................................................. i Acronyms ............................................................................................................................................... iii Executive Summary................................................................................................................................ iv Relevance (4: highly satisfactory) .................................................................................................. iv Efficiency (2: acceptable, with major reservations)........................................................................ v Effectiveness (3: satisfactory) ......................................................................................................... v Impact (3: Satisfactory) .................................................................................................................. vi Sustainability (2: Satisfactory, but with major reservations) ........................................................ vii Introduction ............................................................................................................................................ 1 Overview of the Programme................................................................................................................... 1 Evaluation Methodology Used and Limitations ...................................................................................... 3 Findings and Discussion .......................................................................................................................... 5 Relevance ............................................................................................................................................ 5 Programme Design.......................................................................................................................... 6 Programme Targeting ..................................................................................................................... 8 Levels of Engagement ................................................................................................................... 10 Efficiency ........................................................................................................................................... 11 Budget Execution .......................................................................................................................... 11 Staffing .......................................................................................................................................... 14 Monitoring and Evaluation ........................................................................................................... 16 Effectiveness ..................................................................................................................................... 16 Assets, and return on assets ......................................................................................................... 17 Inequalities .................................................................................................................................... 19 Risk and Vulnerability.................................................................................................................... 20 Advocacy ....................................................................................................................................... 22 Impact ............................................................................................................................................... 23 Sustainability ..................................................................................................................................... 25 Conclusion ............................................................................................................................................. 26 Recommendations ................................................................................................................................ 27 Annex 1: Management Response Template ........................................................................................... 1

ii | P a g e

Acronyms AGIR

Global Alliance for Resilience

AIDS

Acquired Immune Deficiency Syndrome

BRACED

Building Resilience and Adaptation to Climate Extremes and Disaster

CMAM

Community-Based Management of Acute Malnutrition

CRAM

Community Resilience to Acute Malnutrition

CSI

Coping Strategies Index

DFID

Department for International Development (UK Aid)

ECHO

Humanitarian Aid and Civil Protection department of the European Commission

ECOSAN

Ecological Sanitation

EWS

Early Warning System

FIM

Food, Income and Markets

GAM

Global Acute Malnutrition

HIV

Human Immunodeficiency Deficiency

IDP

Internally Displaced Person

NGO

Non-Governmental Organization

SAL

Strategy, Advocacy and Learning

SAM

Severe Acute Malnutrition

SMT

Senior Management Team

Tufts FIC

Tufts University Feinstein International Centre

WASH

Water, Sanitation, and Hygiene

WFP

World Food Programme

iii | P a g e

Executive Summary From 2012-2015 Concern Chad has implemented the Community Resilience to Acute Malnutrition (CRAM) programme. The programme applies the ‘CRAM model’ which aims to improve health, nutrition and livelihood security as well as resilience to shocks for the rural population of Dar Sila. This marked a significant departure from ad-hoc emergency responses and enabled the programme to address the emergencies which regularly occur in the region through a structured and multisectoral approach in which communities become better prepared to anticipate shocks, and to cope with and recover from shocks when they do inevitably occur. Partners for this programme are primarily government institutions in the health, livelihoods and WASH sectors. The Tufts University Feinstein International Centre (Tufts FIC) is engaged in a research capacity to establish a solid evidence base for CRAM. The programme continues in an evolved form with funding from DFID under the Building Resilience and Adaptation to Climate Extremes and Disaster (BRACED) programme (2015-2017). Due to timing constraints, this evaluation took place in advance of the end line assessment. An impact evaluation will be led by Tufts, to be finalised in early 2016, which will incorporate end line data collection and findings in this evaluation. In the absence of end line data, this evaluation relies on data from the baseline assessment (November-December 2012) and midline assessment (November-December 2014), along with respondent interviews and focus group discussions facilitated by the evaluators in N’Djamena and Goz Beida over a total three week period between September 22nd and November 8th 2015. The programme was evaluated against the DAC criteria and assigned a score for each criterion (four points maximum per criteria). The overall score for CRAM was 14 out of a possible 20 points (relevance 4, efficiency 2, effectiveness 3, impact 3, and sustainability 2). Overall, the evaluation finds that the programme is strong and has made significant impacts on the lives of beneficiaries, in spite of serious operational challenges.

Relevance (4: highly satisfactory) The programme design is well aligned to the needs and priorities of programme participants and, although designed by Concern globally (rather than in Chad) was applied in a contextually relevant way in consultation with programme staff. The programme design was an ambitious departure from previous programming in Goz Beida, but is considered ‘suitably ambitious’ in that the team in Chad rose to the challenge and had the appropriate level of support from Concern’s head-office units. Programme targeting is aligned with How Concern Understands Extreme Poverty and targeting criteria is clearly understood by staff, beneficiaries and partners. There is no Complaints Response Mechanism in place, apart from a helpdesk during distributions. Two groups are identified as missing in targeting – pastoralists (now included in the BRACED programme) and children over the age of five (as agents of behaviour change).

iv | P a g e

The primary level of engagement for CRAM is micro-level, in acknowledgement of the weak institutional capacities and the limited scope of coverage by the Government of Chad. This microlevel engagement is appropriate to the objectives of building community resilience. There are strong mechanisms for system strengthening at district level, although opportunities to influence planning at this level due to insufficient linkages between CRAM research and partners. At macro level advocacy efforts were targeted at the right level and strong linkages were made to influence the national resilience agenda.

Efficiency (2: acceptable, with major reservations) In terms of budget execution, the CRAM budget increased each year as compared to the budget projected in 2012, demonstrating the ability of CRAM to leverage additional funding. The projected overall budget (2012-2015) was €3.9million, but the actual budget was €7.2 million. This was achieved while maintaining a healthy balance between development funding to build resilience, and maintaining a responsive infrastructure to secure humanitarian funding and respond to emergencies. The overall contribution of Irish Aid expenditure from 2012 to 2014 was just 23% all funding secured. Further, indirect expenditure remained below 18% of overall expenditure each year, which is remarkable given the logistical constraints of working in such a remote location and the high number of international staff. While finance systems function well, other support systems, particularly logistics, transport and human resources are problematic. These challenges are interrelated – there have been chronic challenges in filling staff positions with strong personnel, and retaining strong staff. This is not a unique problem to Chad, but is particularly difficult in Chad. The remote, unaccompanied postings are challenging to fill. When filled, staff turnover is high, reportedly due to push factors (uncompetitive salaries and difficult living conditions) and pull factors (opportunities for promotion or more attractive locations, or government jobs). Systems to address these issues have not been in place, such as strong handover processes or mechanisms to improve the standard of living conditions, although there are plans to amend this. Staffing gaps contribute to weak logistics and transport systems, particularly at management level. The programme operated without a procurement manual or transport manual, and without long term procurement planning for programme staff. There was a reliance on rented vehicles due to poor maintenance of Concern vehicles. Monitoring and evaluation systems functioned well – investing in a research partnership meant that baseline, midline (and the ongoing end line) assessments were managed by experts within Tufts freeing up the CRAM team to focus on implementation. Monitoring data was routinely collected by the team and well used to inform programming. This partnership functioned well but could have been leveraged better to facilitate shared ownership of and responses to the findings. Reports (baseline and midline) and data analysis skills (e.g. Early Warning System) could have been shared better with the team, and learning documents could have been developed throughout the programme. Similarly, adviser reports could have been better written (fewer recommendations, standardised format, language) to be used by the team.

Effectiveness (3: satisfactory) Despite the long-term staffing gaps, the achievements of the programme against the results framework are impressive, and are a testament to the hard work of those who did remain in Goz Beida and N’Djamena. Measurements of progress are taken from midline data, bearing in mind a high level of intervention in the subsequent year, which should lead to increased results by end line. v|Page

Statements on the unavailability of results are not related to poor monitoring, but are a consequence of the evaluation timing. Outcomes relating to increased assets and returns on assets showed some progress by midline. Access to potable water increased significantly (from 11% at baseline to 50% at midline), mainly due to increased boreholes and well-functioning water management committees. Initial gains in the construction of family latrines and triggering disgust for open defecation were negatively impacted by the collapse of many latrines during the rainy season. ECOSAN latrines were piloted with high uptake but no evidence on results to date. Progress against reducing General Acute Malnutrition (GAM) rate was on track in 2014, when adjusted for rainy season differences. Prevalence of Severe Acute Malnutrition (SAM) reduced in small villages, but not in large, possibly linked to improved water system management from the point of extraction to the point of use, with improved hygiene practices for water containers. Outcomes relating to reduced inequalities primarily targeted pregnant and lactating women and women with children under the age of five. Mixed results are visible – dietary diversity scores decreased from baseline to end line, attributed to the exceptionally good harvest at baseline. There are some indications of a difference-in-difference, with higher dietary diversity in intervention villages than in non-intervention villages, but this cannot be confirmed until the end line. This is aligned with programme monitoring data which shows high uptake of maraîchage community gardens, with increased diversity in the production of, and consumption of vegetables and pulses from these. Improvements in gender equality are visible in increased membership of women in community groups, but not necessarily in their equal participation in these groups. Similarly, at household level there is no evidence of improved access to resources or decision-making for women. Outcomes relating to reduced risk and vulnerability are mixed. Improved knowledge of hand washing is evidenced (from 45% to 53% on critical moments for hand washing), verified in focus group discussions, and water points for hand washing are available in family latrines, but no soap, and no robust evidence is available to assess changes in behaviour. Reductions in SAM in small villages are significant, as detailed under the impact section. No data is available to measure improvements in exclusive breastfeeding practices or knowledge on HIV. Although no outcomes were specified in the results framework on advocacy, significant achievements are found at national level. The programme identified appropriate points of engagement at national level, initially hosting a round-table on resilience and later become active in the AGIR network. Contributions included commissioning a study on Chadian policies through a resilience lens in 2014 and influencing the development of a National Resilience Policy. Advocacy was led by SMT in N’Djamena and, while implemented well, was not documented in a strategy or in reports, which led to the loss of institutional knowledge when there was an SMT changeover. However, the reputation and relationships still exist and can be picked up again.

Impact (3: Satisfactory) Arguably the most important criteria in which to achieve, the programme is deemed to have had a positive impact on community resilience in Chad. Again, evidence is drawn from results at midline. Greatest CRAM impact is visible in the reduction of severe acute malnutrition in villages smaller than 150 households (74% of all villages in the sample). Other findings show increased support for malnourished children (GAM) in treatment villages from 13% at baseline to 32% at midline (while dropping 10% in control villages). This may be linked to the increased household consumption of potable water, and to improved hygiene in water chain management. Most significantly, CRAM is vi | P a g e

found to have a significant impact on reducing food insecurity during the hunger gap (July to September). Also, households in intervention villages were significantly less likely to say they consumed seed stock for the following year.

Sustainability (2: Satisfactory, but with major reservations) Designing a sustainable programme in Chad is a challenge – institutional capacity is very weak and government services are largely absent from target villages. The CRAM model took significant steps towards sustainability by shifting from ad-hoc emergency programming to long-term resilience building. This may, in the long term, lead to community-led sustainability but it is too early at this stage to view evidence of behaviour change across outcomes. There are examples in all sectors of unsustainable interventions – the payment of health worker salaries (albeit via the government), seed and equipment distributions, Concern-led support to community groups etc. There are also strong examples of innovations to address weak government services – cost-recovery strategies in the privately run veterinary pharmacy and auxiliary system, moving from mobile clinics to the stratégie avancée health outreach system, and the negotiation of a withdrawal strategy with the Ministry of Health. These initiatives are noted as strong programming, but there remain many aspects of the programme that are unsustainable and closely linked to direct service provision.

vii | P a g e

Introduction Chad is a vast landlocked country, with deserts in the north and tropical forests in the south. Eastern Chad is situated in the central semi-arid Sahel band. Though sparsely populated, subsistence farmers compete with nomadic pastoralists for access to water and arable land. Since the signing of a peace agreement with Sudan in 2010 Chad has been relatively stable, though the security situation in the border areas remains volatile. Chad is hosting over 435,000 refugees1, mainly originating from neighbouring countries CAR, Sudan and Nigeria. There are also approximately 85,000 IDPs including about 70,000 in the Sila region and 15,000 in the Lac Region. Recent crises in the neighbouring countries CAR, Nigeria and Libya have had a significant impact on migration with the return of some 245,000 migrant workers, the interruption of commercial links and the loss of remittances from Chadians previously working in these countries. Chad is ranked 184 out of 187 countries of the Human Development Index (HDI)2. In Dar Sila, the global acute malnutrition rate is 10%, 60% of the population are categorized as ‘poor’ or ‘very poor’, and 40% are severely food insecure.3 The June 2012 USGS Climate Trend Analysis4 found that rains have decreased and temperatures increased in Eastern Chad during the past 25 years.

Overview of the Programme Concern has worked in the Dar Sila region since 2007, initially responding to the massive displacement related to conflict on both sides of the Chad-Sudan border. Humanitarian support was provided to IDP and host communities in the Department of Kimiti, in the eastern half of Dar Sila. As the security situation stabilised, Concern identified the need to shift programming away from ad-hoc humanitarian responses, towards building community resilience to shocks, while concurrently improving timely humanitarian responses to predictable shocks. The Sila region located on the frontier of the Sahara Desert, the savannah woodland, and directly on the border with Sudan, experiences highly variable rainfall, seasonal food insecurity, and at times chronic levels of malnutrition. The Community Resilience to Acute Malnutrition (CRAM) model aims to address the emergencies which regularly occur in the region through a structured and multi-sectoral approach in which communities become better prepared to anticipate shocks, and to cope with and recover from shocks when they do inevitably occur. To establish a solid evidence base for the CRAM model, a rigorous research component was included from the beginning of the programme’s development and implementation. To manage this research a partnership with the Tufts University Feinstein International Centre (Tufts FIC) was formalised in 2012, giving Tufts responsibility to lead the design, analysis and dissemination of baseline, midline and end-line data, an impact evaluation, staff training on data collection, the development of an Early Warning System, and additional programme learning where appropriate.

1

Source: UNOCHA, July 2015 Source: Human Development Report 2014 3 World Food Programme, Assessment of the Food Security of Rural Households in the Sahelian and Sudanese Bands in Chad, June 2013 4 USGS, http://pubs.usgs.gov/fs/2012/3070/ 2

1|Page

In the final year of implementation a new grant was initiated by UK AID (DFID) under the Building Resilience and Adaptation to Climate Extremes and Disaster (BRACED) initiative. This evaluation does not include any analysis of the BRACED programme itself. Instead the interactions between the two are considered - the contributions of CRAM into the development and successful funding of BRACED, the relationship/transition between CRAM and BRACED, and recommendations on how lessons learnt from CRAM could enhance implementation of BRACED. The Overall Goal of the programme is to improve health, nutrition and livelihood security as well as resilience to shocks for the rural population of Dar Sila (Eastern Chad). From a livelihoods perspective the programme aims to improve food production and agricultural practices, increase crop diversity, diversify livelihood opportunities, with the goal of increasing household wealth (as measured by livestock ownership, household assets and dwelling construction). To achieve these objectives activities include the introduction of conservation agriculture, homestead gardening (referred to as maraîchage), activities to improve soil fertility, small-scale irrigation solutions, agriculture system strengthening and improved crop storage mechanisms. Animal health supports were planned through veterinary extension services.

F IGURE 1: T HEORY OF CHANGE (SOURCE : CRAM P ROPOSAL) From a health perspective the programme aims to improve access to health and nutrition services, improve the nutrition status of vulnerable groups, especially pregnant and lactating women and children under the age of five and improve health practices at household and community level. Activities towards these objectives include training and supporting voluntary community health workers (referred to as Relais Communautaires), establishing maternal and new-born health care 2|Page

groups (referred to as Groupe de soutien mère à mère), strengthening community management of health care delivery, health system strengthening at district-level and promoting improved feeding practices at household level. From a Water, Sanitation and Hygiene (WASH) perspective, the programme aims to improve access to safe water and sanitation facilities and improve health and hygiene practices at community and household levels. To achieve these objectives activities include borehole drilling, small water conveyance systems, water services management (through committees referred to as Comités de Gestion des Points d’Eau), community-led total sanitation strategies, participatory hygiene and sanitation transformation strategies, the introduction of Ecological Sanitation (ECOSAN) blocks and environmental sanitation interventions. Complimentary objectives to build community resilience through increased equality, emergency preparedness and early responses are included. Strategies include the establishment of an Early Warning System (EWS), Disaster Risk Reduction strategies and the development of a Preparing for Effective Emergency Response plan, increased participation of women in community groups, Community Conversations on gender equality, and engagement of community leaders on concepts of equality. The inclusion of quality research, monitoring and advocacy is included through the Tufts partnership to facilitate increased dialogue on the need for resilience planning in Chad and beyond. Additional cross-cutting themes are identified as partnership development and capacity building, HIV and AIDS mainstreaming, social protection and accountability.

Evaluation Methodology Used and Limitations This evaluation was conducted between September and November 2015. The evaluation draws heavily on findings from the baseline and midline assessments led by Tufts in November and December 2012 and November and December 2014 respectively. The CRAM end-line was not completed at the time of the evaluation due to time constraints - the baseline was conducted in November and December 2012, and given the need for comparability of data in CRAM it was essential to plan the end-line for the same time of year in 2015. Preliminary end-line findings will not be available until late December with the full Tufts-led impact evaluation report to be provided in March 2016. This evaluation was required for completion by November 2015 to align with Irish Aid reporting commitments so the decision was taken to base the evaluation on all available information available to date. This evaluation will inform aspects of the Tufts-led impact evaluation to be completed in 2016. 



5

For the baseline and midline, data were collected from 69 villages5. These were oversampled in the baseline in order to account for possible attrition over time, with the total number of sampled households reaching 1,420 (Table 1). In the midline, the same households were tracked in order to control for household level characteristics in the analysis that are not necessarily captured in the data. Due to mortality, migration, and relocation approximately 12% of the baseline population were not re-surveyed in the midline. While this attrition rate is not ideal, most importantly attrition was not correlated to treatment and therefore does not introduce bias in the analysis. For the selection of control and treatment villages, the sample was stratified by priority level, where Priority 1 villages are those in the northern part of the region which are

Permission for data collection was not given by the village chief in one village during the baseline.

3|Page



geographically more prone to drought, and Priority 2 villages are in the south of the region and historically experiences more rainfall. Within each of the strata, the CRAM program was randomly assigned on the village level, with participating treatment villages receiving the CRAM package of interventions from 2013 to 2015. Qualitative data was collected in February and March 2013, to gain a deeper understanding of the regional and local context, and to understand some of the patterns observed in the qualitative data. This will be repeated for the end line, but qualitative data was not collected during the midline so is not used in this evaluation.

T ABLE 1: SAMPLING AND ATTRITION FROM B ASELINE TO MIDLINE ASSESSMENT HOUSEHOLD

CHILD ROSTER

HOUSEHOLD ROSTER

Baseline Midline attrition (#) Attrition (%) Baseline Midline attrition (#) Attrition (%) Baseline Midline attrition (#) Attrition (%)

CONTROL 719 638 81 13% 860 801 59 7% 3,826 3,604 222 6%

TREATMENT 701 609 92 13% 795 772 23 3% 3,686 3,352 334 9%

TOTAL 1,420 1,247 173 12% 1,655 1,573 82 5% 7,512 6,956 556 7%

Both evaluators travelled separately to Chad - Alexander Carnwath travelled to N’Djamena from September 27th to October 4th to evaluate macro-level advocacy results. This was achieved through key informant interviews with national-level government and partner representatives, in addition to interviews with Concern staff involved in CRAM advocacy and document reviews. Jenny Hobbs travelled to N’Djamena and Goz Beida from October 25th to November 8th to evaluate programme progress against all other objectives. There is a possibility of evaluator bias in the evaluation findings. Both evaluators are Concern employees, rather than external evaluators. Alexander Carnwath’s role as Community Resilience Policy Officer with Concern UK may result in recommendations on advocacy that will contribute to the Concern UK advocacy agenda. Similarly, Jenny Hobbs may be biased towards findings and recommendations towards education, aligned with her role as Senior Education Adviser. Efforts were made by both advisers to acknowledge and minimise this bias, but it is likely to have influence on this evaluation. 

Four intervention villages and two non-intervention villages were selected for visits. The selection was based on a collaborative discussion with all programme managers. First, a list of key facilities, committees or initiatives was developed by the evaluator as priorities to see (health centres, sites of pilot initiatives) and villages with these interventions were identified. From there, a mapping process was used to identify non-intervention villages

4|Page





close-by6, to minimise travel time between sites. From these shortlisted sites, the evaluator selected three villages based on size – one large, one medium, and one small. This was crosschecked with end-line data collection visits and changes were made in the event of overlaps. A fourth small village was then randomly selected from the map. Two further villages were selected for quick stops to see specific activities not available elsewhere. A maximum travel time to villages of two hours was decided to allow sufficient time per village to meet all respondents, although this criterion meant that the furthest, most remote villages were excluded. In each village key informant interviews were held with the village Chief and the 'Chouchi', a female leader in each community. Focus group discussions were facilitated with committees and groups of beneficiaries, generally with an average of five participants, although groups tended to grow during sessions as additional people joined. PRA activities were used to facilitate priority ranking and to encourage discussions on the effectiveness of certain activities. Certain groups were male-only or female-only in each village, before bringing men and women together for other groups. Site visits (to farms, health centres, ECOSAN toilets and water points) were conducted together with individuals or groups responsible for the initiatives (model farmers, water management committees, mid-wives) and transect walks were conducted in three villages to note anything of relevance in the village (sanitary conditions, gendered workloads, visible health status of children, used/disused hardware etc.). A full schedule of villages and groups visited is provided in annex 3 Key informant interviews were held with Concern staff (SMT, managers, deputy managers and others in key positions). In Goz Beida key informant interviews were held with government partners and NGO or donor partners. Only one external meeting was held in N’Djamena as these had been accomplished by Alexander Carnwath, with just one remaining that was not possible during his visit.

Findings and Discussion The following section firstly looks at the performance of the project against the key indicators included in the Results Framework, and then looks at the programme under each of the five DAC criteria of Relevance, Efficiency, Effectiveness, Impact and Sustainability. A scoring of whether the programme is highly satisfactory, which we have taken as meaning exceeded expectations; satisfactory, which we interpret as being what would be expected, acceptable but with some major reservations and unsatisfactory is included (see table on title page). In addition to providing an overall assessment, the team have also considered this at the three different levels of intervention (micro, meso and macro) allowing a distinction to be drawn between areas where Concern and partners would traditionally be strong (in direct implementation at micro level) and areas that have been targeted for change in more recent times.

Relevance Unsatisfactory Overall Micro

6

Acceptable (with major reservations)

Satisfactory

Highly Satisfactory X X

The distance between intervention and non-intervention villages varied from 2km to 10km. The RCT design over a relatively small area means that all intervention and non-intervention villages are close enough to have possible spill-over effects.

5|Page

Meso Macro

X X

To evaluate the relevance of CRAM, we consider the alignment of the programme design to the findings of the contextual analysis, Concern’s organisational policies, and the priorities of programme participants and partners. The appropriateness of targeting in CRAM is considered through the lens of How Concern Understands Extreme Poverty (HCUEP), and this is considered geographically, in addition to groups or individuals within villages. The programme’s alignment with national and district-level priorities is assessed, considering the appropriateness of levels of engagement to meaningfully influence government agendas. In addition to the questions provided in the terms of reference, the evaluation also considered the existence of appropriate complaints response mechanisms for beneficiaries and the appropriateness of adaptations made to the programme during implementation.

Programme Design At the time of programme design, Concern was in the process of withdrawing from Goré, in the south of the country, where a development programme was ending. Activities in Goz Beida were starting to shift from the management of IDP camps (2007-2010) in response to the humanitarian crisis on both sides of the border with Darfur, towards recovery and development activities in villages where people remained or were returning to. Livelihood supports were ongoing in villages of return, and a WASH programme had started in 2010. In December 2011 - January 2012 a contextual analysis was conducted to identify the extreme poor, the drivers and maintainers of poverty, and to identify opportunities to address extreme poverty. Concern’s understanding of extreme poverty considers the multiple dimensions of poverty – a lack of assets and low return on these assets, inequality and increased risk and vulnerability to shocks – and the identification of target beneficiaries is based on these dimensions. The contextual analysis acknowledges certain limitations in its findings – there was a minimum of field work conducted due to time constraints, and only two of the data collectors spoke Arabic, the language used within communities in Goz Beida. Documents for desk review were limited in their detail or disaggregation level, which created challenges for the reviewer. That said, the process identified Goz Beida as an extremely poor and vulnerable geographical location, underserved by government departments and marked by infrastructural weaknesses at all levels. Chronic food insecurity, conflict and displacement were identified as maintainers of poverty, while the landless, those with disabilities or illness, and women suffer from societal inequalities. Three groups of extremely poor people were identified. The first is those returning to villages of origin, including those without animals, female headed-households and children. The second group is the old and infirm living in IDP sites. The third group identified as potentially extremely poor – pastoralists – were not reached by the contextual analysis and further research was recommended. Given the changing context, with improved security and more people returning to villages or remaining in their villages, Concern’s strategy was to directly support the resettlement and recovery process. However, it was acknowledged that a development-only programme would not work due to the inevitability of recurrent shocks, especially food insecurity. From stakeholder interviews, it appears that this was a time of major change in Concern’s operations in Chad – a new country director was managing the exit strategy from Goré while the team in Goz Beida were trying to shift towards development programming in a changing environment. The CRAM model was developed by this stage by Concern at a global level, and Chad was identified as a potential site for the application 6|Page

of, and testing of, the model. Tufts were subsequently engaged as research partners and a joint support visit (Concern and Tufts) was organised to visit Goz Beida, work directly with staff on the ground, and contextualise the model for application in that context. In recognition of the high workload and staffing gaps within the Concern team in Chad, there was a high level of direct support built into programme design and management. A CRAM steering group was established to meet every two weeks to include the Regional Director, Desk Officer, Country Director, Programmes Director and SAL Head of Technical Support. The terms of reference developed with Tufts gave direct responsibility for large pieces of technical work to the university – including the management of the baseline, midline and end line from design to final write-up. A large international staff presence was built into the programme management structure, bringing in technical expertise for WASH, livelihoods and Health and Nutrition, in addition to programme coordination and base management. In January 2013 a planning meeting was held in N’Djamena with the Regional Director, SAL Head of Technical Support and Senior Management Team (SMT) in Chad together, followed by sensitisation and planning meetings in Goz Beida with key Concern programme staff. Beyond the contextual analysis, participation was missing at community level – there was no clear process of engagement with beneficiaries at the time of design. However, the design itself appears to be well aligned with participant priorities. Despite the fact that the baseline was conducted in a relatively good harvest year, 53% of households did not have enough food for five or more months of the year, GAM rates exceeded 15% and a SAM rate of 3.6% was found. Almost 50% of households were practicing open defecation and only 10% of households had access to the recommended minimum of 15 litres per person per day, with the vast majority of households dependent on unsafe water supplies in ‘wadis’ (seasonal river beds), with long distances to travel to collect water. Women were identified as particularly disadvantaged, performing the majority of labour tasks but not actively participating in decision-making – only 20% of women reported being able to make decisions themselves on their own health, and even fewer (17%) participating in decisions on their children’s health. The contextual analysis found that basic services in Sila were minimal and comparatively worse than in other parts of Chad - 91% of girls and 85% of boys aged between 10 and 14 years old who lived in the zone did not attend school, and health centres were largely inactive (only seven of 17 centres were functioning) with no outreach services to the unserved populations not living near a health centre. Recurrent drought, climate change, soil erosion and displacement were identified as drivers of frequent shocks – particularly food insecurity but also disease and conflict – leading to a cycle of recurrent emergencies. A participatory priority ranking exercise was conducted with programme beneficiaries to facilitate discussion on their priorities and their perceptions of Concern’s priorities, comparing the two as a group and critiquing the importance of each activity in their lives. The exercise was conducted with four groups – two female-only, one male-only and one mixed. In all groups there was general agreement that the programme met their priorities. The male-only group did not prioritise women’s empowerment or access to decision-making, but this was prioritised F IGURE 2: PRIORITY NEEDS RANKING BY FEMALE BENEFICIARIES

7|Page

by women-only groups. When asked if there were other priorities not met by the CRAM programme, every group identified education services as a need. At the time of the contextual analysis there was no evidence of education infrastructure in the district, but during the evaluation new school buildings were seen dotted throughout Sila. The CRAM model originally planned to start education supports in year 2, but this was one of the activities that was not pursued due to funding constraints. It seems that this would be a good addition to the next phase of resilience programming in Chad moving forward. The inclusion of a robust research design into CRAM to test the model seems highly relevant. At national level, stakeholders within the government were interested in findings – they feel that they have accepted the need for resilience programming and now need more in-depth information on what does and does not work for their context, which is the need served by the research. The potential of CRAM research to secure new funding was realised through BRACED – it is generally perceived that the CRAM model, with the research partnership, was key to Concern’s success in securing new funding from DFID. Similarly, resilience features strongly on the international agenda, particularly for the Sahel, so investing in learning what works is essential to inform future interventions. It must be noted that the shift to resilience-based programming was a major step for the Chad team – systems and programme staff were all geared towards emergency response and there was a low capacity for development programming. The contextual analysis notes, in particular, challenges of applying participatory techniques for staff. Designing CRAM for Chad was very ambitious, and this is noted by all stakeholders interviewed who participated in programme design or initial roll-out. However, there is general consensus that the timeframe was appropriate (possibly with the addition of one year to allow for staffing challenges) and that the programme achieved a lot. This evaluation finds that the programme was ambitious, but suitably ambitious. It represented a complete change in programming for the team in Chad, but had sufficient organisational supports and strong SMT leadership in-country, which enabled the necessary changes to take place and good outcomes to be achieved directly related to beneficiary needs. In conclusion, it is generally agreed by those involved that an appropriate level of support was provided to facilitate the Chad team’s development and implementation of CRAM in Goz Beida, and without this level of support it is unlikely that the CRAM model could have been designed in-country. Although the original design was developed in Dublin, based on learning from Niger and Kenya, it was adequately contextualised in Chad in collaboration with the team responsible for implementing it. The design of CRAM is well aligned to needs in Goz Beida and represents a strategic shift away from ad-hoc emergency responses to building long-term development and resilience to shocks. The design was welcomed in Chad by external and internal stakeholders as an appropriate intervention, highly relevant to the contextual challenges. It was an ambitious design, but suitably ambitious, with an appropriate timeframe.

Programme Targeting As detailed above, the geographical targeting of the Sila region is justified by high levels of malnutrition and other illnesses, frequent shocks, high displacement and a vulnerable population unserved or underserved by government services – especially health, WASH and education services. In terms of geographical targeting within Dar Sila, the research design needs to be considered – due to the nature of comparative testing of the model, randomisation was used to select individual villages. However, the programme did maintain ethical targeting decisions in the design – health service strengthening was provided through health centres, targeting both intervention and non8|Page

intervention villages. Emergency response needs were responded to in all villages, based on humanitarian needs rather than on research needs. The second phase of the CRAM model, funded under the BRACED programme, includes an expansion of CRAM to non-intervention villages from 2016. From an evaluator’s perspective this is as ethical an approach as can be expected from any randomised control trial intervention, with a good balance between beneficiary needs and the needs of the research design to generate robust evidence. Targeting within villages was designed to focus primarily on pregnant and lactating women, mothers of children under the age of five, children under the age of five and women (or girls) planning to conceive. Many women live without their husbands for most of the year as the latter migrate in search of paid labour, and these women are directly targeted by the programme. Secondary beneficiaries within communities were identified as those best placed to influence certain changes in the lives of the primary beneficiaries, such village leaders engaged with to promote equality. A good example of strategic targeting observed is through the ‘paysans pilotes’, in which the most skilled farmers in the community (generally men with access to certain resources and willing to take some risks) were trained to pilot new techniques. The next stage was to link these men with groups of vulnerable people in the community, especially women, to share the skills they had learned and support the new groups to apply these techniques. This meant that the most vulnerable groups were not expected to experiment and take risks, without others with more resources doing this first. When asked to describe the targeting criteria applied by Concern, beneficiaries were strongly able to describe Concern’s targeting criteria, naming specific vulnerable groups and explaining the rationale. This was also true of stakeholders at district-level in Goz Beida – government partners could automatically describe the poorest and most vulnerable groups identified by Concern. In the case of identifying group participants (e.g. relais communautaires , a paysan pilote, or a member of the comité de gestion de point de l’eau), the planned activity was discussed with the village chief, who then held a meeting and shared criteria with the community, from which the community selected group representatives. This process presents potential risks – the poorest and most marginalised are unlikely to be selected – although in some cases mechanisms seem to be built in to avoid selecting the same people repeatedly. For example, for the relais communautaires and the Groupe de soutien mère à mère, villages were divided into sub-groups of fifteen households, and the women in those fifteen households selected a group leader from within. No formal Complaints Response Mechanism (CRM) has been developed to date. During distributions there is a helpline system established, so community members can call Concern to question the targeting and report any unfair practices they observe (this is following a system of verification also built into beneficiary selection based on wealth ranking). The helpline is used and appreciated by community members. However, when this is inactive there is no system, and no person in any community could describe an appropriate contact system if they have future problems (besides talking to the Concern staff members visiting the village, via the chief). When asked what they would do in the hypothetical case of sexual abuse of a girl in the community by a Concern staff member, women and men said they would refer the case to the chief. If the chief decided not to pursue the matter, they would have no choice but to agree. A pilot CRM is planned for five communities moving forward, potentially using complaint boxes, and mechanisms to report cases not based on literacy skills will be explored by the team. Two groups emerge as potentially missing from targeting. Children over the age of five, particularly orphans and those living with grandparents (or their grandmother) were identified as extremely vulnerable groups not currently supported by CRAM directly. There seems to be very little 9|Page

information on the number or situation of orphans or children living with grandparents in documentation or among programme staff, so this is an area that should be investigated further to see if this is a large group and to identify ways to support them under existing activities funded by BRACED. Education services would certainly directly support children living in target communities, but in general it was identified by Concern staff and partners that children could be better engaged in behaviour change aspects of the programme – such as health and hygiene promotion or environmental protection – to achieve more sustainable change. The second group identified as missing from targeting are pastoralists. Very little information is provided on pastoralists in the contextual analysis, and the village-level focus of CRAM excludes nomads and pastoralists from both research and direct interventions, even for animal health services. CRAM’s design intended to invest in research to better understand the particular needs and vulnerabilities of pastoralists, rather than pre-empting their needs and possibly missing the point. Some activities were planned, such as the provision of water points for livestock along pastoralist routes and conflict mitigation between pastoralists and villagers. However, the CRAM programme lacked sufficient funding and these activities were not pursued. In the BRACED design a strong research component has now been included and funded to gain better insight into pastoralist needs and to plan appropriate interventions. In conclusion, the evaluation finds that targeting is clear and well-aligned to How Concern Understands Extreme Poverty. Beneficiaries and partners can accurately describe the targeting profile and rationale. No CRM currently exists, although a pilot is planned moving forward. Targeting adequately reaches the right primary beneficiaries, particularly in relation to improving malnutrition. Targeting of children, particularly orphans and those living with grandparents, should be further explored, both as primary beneficiaries of services and potentially as agents of behaviour change. Pastoralists needs have not been addressed under CRAM, although actions are planned under BRACED to address this.

Levels of Engagement The CRAM model was designed to work primarily at micro-level, building community resilience. However, in Goz Beida the programme worked directly through and with government services, where they existed. This has been well achieved within the health and livelihoods sectors, with strong relationships established with the Health Department, Livestock Department, Forestry Inspectorate and Plant Health Unit (phytosanitaire). Good practices identified are cost-sharing initiatives between Concern, government and partner agencies – this is most visible through health where Concern initially provided all direct supports and now implements a withdrawal strategy, progressively handing responsibility over to the health department. At times tensions have been observed – particularly where departments demanded recurrent costs from Concern (fuel, car maintenance) or direct supplies of vehicles, but in general this seems to be managed well by the programme maintaining a balance between support and scale up. Joint training initiatives have worked particularly well, with Concern organising the training and government departments facilitating or rolling out national strategies through these training sessions. Joint follow-up after the training has not been so successful – while this seems to work well with the health, livestock and forestry departments, representatives in phytosanitaire and ONDR reported that they would have liked to do this but were not given the opportunity. At national-level, engagement has primarily been through advocacy interventions through engagement with the AGIR network, and through a roundtable organised by Concern on resilience in December 2013. This is an appropriate level of engagement – the roundtable enabled Concern to 10 | P a g e

bring together key stakeholders at national level and develop relationships with government representatives, particularly in the agriculture sector. This led to Concern’s engagement in the AGIR network, which responds to the government’s own priorities and has regional relevance. However, these relationships could have been better leveraged if an information pathway was established for CRAM learning to feed into district and national-level discussions. For example, it would have been appropriate to use a national-level forum to share findings from the midline assessment, but this was not achieved, possibly due to staff turnover and shortages within Concern and staff turnover within key government ministries. At meso-level, advocacy engagement is not strong. This may be linked to the programme design the results framework had no meso or macro-level outcomes or indicators, and advocacy initiatives were led primarily by staff based in N’Djamena (the Country Director and Programmes Director). If there had been specific objectives for advocacy within the results framework this may have led to programme staff in Goz Beida linking more with government partners (at district and national levels) to share learning from CRAM – especially when the midline report was available. The programme design could have capitalised better on the emerging research, particularly by engaging government partners more in the research process, sharing results and passing ownership of the results to government agencies who could actively lead training, follow-up and other aspects of implementation. While government partners in Goz Beida were deeply knowledgeable on programme design and implementation, there was absolutely no awareness of the research component of the programme, no knowledge of the Tufts partnership and no knowledge on programme findings to date. This is a lost opportunity, although it has already been considered in the design of BRACED with a specific outcome on information sharing and collaborative learning. In conclusion, the programme design is appropriately targeted at micro-level to improve community resilience, with strong mechanisms for system strengthening at district level. National-level advocacy efforts were targeted at the right level and strong linkages were made to influence the national resilience agenda. At district-level, opportunities to influence planning were missed and there was insufficient linkages developed between the CRAM research and partners at meso-level. This might have been strengthened if the results framework had been designed to include meso-level outcomes.

Efficiency Unsatisfactory Overall Micro Meso Macro

Acceptable (with major reservations) X X X X

Satisfactory

Highly Satisfactory

In reviewing the programme’s efficiency, the evaluators considered budget execution, staffing levels, monitoring and evaluation systems and the systems used to leverage best results from each of these. Projected Budget v Actual Budget

The original CRAM budget was projected as €3,941,058 over the four years. The budget increased significantly, when

3,500,000 3,000,000 2,500,000 2,000,000 1,500,000 1,000,000 500,000 0

Irish Aid Contribution

2015 Actual

2015 Projected

2014 Actual

2014 Projected

2013 Actual

2013 Projected

2012 Actual

2012 Projected

11 | P a g e

Amount (€)

Budget Execution

F IGURE 3: PROJECTED BUDGET VERSUS ACTUAL BUDGET

looking back from 2015, to €7,156,0967. Much of this increase is attributable to the successful BRACED application which started in April 2015, although figure four shows a consistent increase in budget each year. This increase is positive – the original budget was described as lower than desired, but aligned to expected levels of funding. Two conclusions can be drawn from this – the programme was successful in leveraging new funding from a diverse range of donors (see figure 5 for a full list of all donors). Secondly, the programme maintained a healthy balance between securing development funding to build resilience, and maintaining a responsive infrastructure to be able to secure humanitarian funding and respond to emergencies. Maintaining this balance is challenging as staffing profiles may become more

Emergency Expenditure

Development Expenditure

Donor WFP ECHO Jersey Overseas Aid Concern US IA (HPP) OFDA GD Sub-total EU FAO WFP ECHO DFID Jersey Overseas Aid Campbell Borehole OFDA Concern US Harambee GD IA (HPP) IAPF Contribution Subtotal Total funding IAPF Contribution IAPF Contribution as % of total

2012 - 2015 (€) 45,970 317,096 3,319 3,347 1,095 471,714 577,823 1,420,364 995,569 40,129 63,350 683,321 140,530 80,316 17,710 553,553 227,579 84,712 1,706,961 398,905 1,944,793 6,937,428 8,357,792 1,944,793

development-oriented – it is notable that the team in Goz Beida are already planning an ECHO response in 2016 should the harvest be affected by poor rainfall as indicated by the EWS. Irish Aid programme funding accounts for just 23% of overall funding: yet it is considered to be the ‘anchor funding’ needed by the programme to establish and test CRAM. The evaluation finds this to be an effective use of multiannual funding to build organisational expertise and develop a sustainable and diverse donor base.

Actual expenditure for CRAM is detailed in figure 6 below, covering 2012-2014 (as 2015 expenditure is ongoing). Indirect costs are maintained below 20% each year – which is impressive Summary considering the logistical 23.3% constraints of working in a remote F IGURE 4: DONOR PROFILE OF CRAM 2012-2015 and insecure environment, and considering the high number of international staff required for quality programming. Financial management and reporting was considered to be strong by all stakeholders, with no reports of delayed payments or barriers within the system in Chad. An analysis of expenditure by sector does not provide much insight as FIM expenditure covers a wide range of activities, including

7

€7,156,096 represents budgeted costs (not expenditure) for 2012-2015. In Figure 5 €8,357,791 represents actual expenditure 2012-2015 (Jan-Aug 2015). Figure 6 represents actual expenditure for 2012-2014, excluding 2015 (2012-2014: €5,480,625)

12 | P a g e

2012

2013

2014

400,000

698,995

600,000

1,698,995

19%

44%

33%

31%

1,713,029

1,304,500

1,494,490

4,512,019

377,937

289,078

301,591

968,606

2,090,966

1,593,578

1,796,081

5,480,625

18%

18%

17%

18%

FIM expenditure

1,003,108

533,830

604,009

2,140,948

Health expenditure

1,087,858

1,059,747

1,192,072

3,339,677

Expenditure from Irish Aid Irish Aid Expenditure as % of total Direct expenditure Indirect expenditure Total expenditure Indirect as % of total expenditure

F IGURE 5: CRAM EXPENDITURE 2012-2014

Total

environmental activities and emergency food distributions, while health covers all WASH, nutrition and health centre costs.

The overall cost per beneficiary for CRAM emerges at €356 per direct beneficiary or €158 per indirect beneficiary. When the added value of the research component is considered, along with the long term nature of improved resilience which should result in long term savings in health services and emergency responses, the cost per beneficiary is within an acceptable range for this evaluation. Logistics and procurement systems need to be strengthened, to facilitate improved programme implementation. There is currently no procurement manual or transport manual for Chad, although there are guidance documents with timelines and procedures for specific procurements. This has led to delays – without an approved supplier database all departments reported excessive waiting times for procurement, and management reported frustration at having to reinforce basic regulations (such as bulk ordering of materials, planned procurement processes and the need to procure items from N’Djamena where costs are lower). It was reported by SMT that there is more support needed from Concern’s head-quarters on logistics systems, which is not available to the same extent as programme technical support. Given the contextual challenges of implementing CRAM, it may have been helpful to build in more supports to the systems considerations of the programme from the outset, with increased budget for technical support and staff training (programmes and systems staff) on planning and procurement procedures. Managers interviewed during the evaluation described weak planning systems among teams, with small scale supply requests rather than forward planning and stocking materials in advance of planned activities. This was also reported as a frustration by logistics staff and SMT. The absence of a strong and consistent systems manager (or equivalent) meant it was difficult to put these systems in place or to hold programme staff accountable to the systems that are established. An example of how these procurement systems affect programming can be seen in the FIM programme where seed distribution for maraîchage gardens (dry season vegetable gardens) was late every year, arriving in Goz Beida after the optimal planting months, with mixed reports of seed quality. Seedling procurement is challenging in Chad, particularly at scale, but the programme lacks an effective strategy to address this (such as supporting seedling nurseries in Goz Beida or elsewhere), leading to repeated delays. It is hoped that the seeds will arrive in time in 2015, albeit towards the end of planting season. This may have also been due staff shortages – the position of FIM Manager was particularly difficult to fill and lacked strong, continuous management, which likely resulted in weak planning for seed procurement. There are also examples of good use of resources to achieve results despite contextual challenges. Coverage of the WASH programme with boreholes in all intervention villages is an example. The programme contracted borehole construction out to local contractors, with monitoring systems in place. Weak capacity led to the majority of these boreholes falling behind schedule. The programme team stepped in to work directly with contractors, or take over the work, identifying resources where necessary. It seems the process was very difficult for all involved, but at the end of the 13 | P a g e

programme the coverage of boreholes is a testament to the team’s ability to step in and find solutions within a challenging context. Another example is the streamlining of the end line data collection process – the team used their knowledge of the communities and roads to re-design the research plan, cutting the time and costs for data collection significantly as compared to the baseline. The management of vehicles and transport systems was problematic in 2014, although this seems to have improved considerably in 2015. The programme relies heavily on rented vehicles and only three Concern vehicles were in working order in July this year, but these have been repaired over time and the programme is on track to have ten functioning vehicles by January 2016. Identifying genuine spare parts is an ongoing challenge in Chad, but parts were often procured in small quantities in the past, causing delays – a stock is now being built up and the programme is recruiting a second mechanic. Transport systems which require teams to plan and coordinate in advance have been established and seem to work well in maximising the number of people in a car while including security measures (two cars close-by at all times). Vehicles lack functioning radios which was raised as a security concern by drivers and management, particularly given the weak mobile phone coverage in the region.

Staffing Recruiting and retaining staff was the greatest constraint experienced for the CRAM programme. Annex 4 provides an overview of international staff positions filled or vacant from January 2012 until June 2015. During this period there were just six weeks (collectively, not consecutively) when all positions were filled correctly. There has been significant involvement of SURGE team members (roving staff positions managed by the Emergency Unit of Concern’s Dublin office), or other staff members seconded for short periods, arriving for short term placements. Seventy-seven per cent of the time the programme was running without key managers in place and even when positions were filled staff retention proved extremely difficult. The continuous recruitment process placed a great strain on the programme as new staff arrived, took some time to understand the programme, and then left quite quickly. There were similar challenges in recruiting and retaining national staff. Due to low education levels in Goz Beida, most national staff are recruited from other parts of Chad, arriving to live in Goz Beida from N’Djamena or the south of Chad, where services and infrastructure are better. Identifying, recruiting and training staff is a continuous cycle within CRAM, and trained staff leave frequently for other opportunities (promotions or opportunities for work closer to home). The cycle of staff changes in itself is not an inefficiency - it is a contextual reality when Concern works in the most marginalised and fragile locations. However, the drivers of this and the ways to address staffing gaps could be improved. To find out what the drivers are, and how these could be addressed, the evaluator asked all Concern staff interviewed (or met informally) what they believe drives high staff turnover and what can be done to improve staff retention. It should be noted that existing staff were asked why they think it is difficult to recruit and retain staff, so their responses reflected their opinion on why others had left, while they themselves stayed and are still working on CRAM. Staff salaries are cited by both international and national staff as barriers to staff recruitment. International staff living outside of the Eurozone report significant losses in their salary due to exchange rates. In 2011 Concern had a reputation in Chad for being the ‘lowest paying NGO’ for national staff, according to SMT interviews, leading to an increase of 5%, and further modest 14 | P a g e

increases in 2012. It is now believed that Concern’s national staff salaries rank towards the middle of the scale when compared to other NGOs, but SMT respondents worry that this is not sufficient – the pool of highly qualified national staff is low and Concern works in a particularly remote area, so without more competitive salaries staff turnover is likely to remain high. This is a convincing argument – if a person can join Concern for a short time and benefit from training and experience, they would be in a good position to apply for positions in more desirable parts of the country which pay above average salaries. A recent survey of national staff who had left Concern in the past year showed that 60% were hired by the government (not a bad pathway in terms of national development), 35% got a new job closer to home or with a promotion and 5% did not respond. Another common response was that staff leave due to harsh living conditions in Goz Beida. Housing is very basic, teams live and work in very close quarters, there is very little variety in food options – particularly fruit and vegetables, and social activities are minimal. Staff suggested ways in which their lives could be made a little easier for staff retention. For national staff it may be helpful to have a buddy-system to help newcomers to Goz Beida to find accommodation and settle in. For international staff, living conditions could be improved through modest changes, including pest control in the offices and housing in Goz Beida and internet facilities in staff accommodation in N’Djamena. There are ongoing efforts to make changes, such as the relocation of international staff to a new location with better facilities in Goz Beida, which has the potential to make a difference. One issue arising from poor staff retention is a lack of a standardised hand-over system. The system of filling vacancies with SURGE staff or with staff from units at headquarters does help to respond to immediate needs. However, these assignments are short-term and fixed and often end before the position is filled. Managers interviewed reported different hand-over systems, whether documented, through file-sharing, or a verbal handover by a staff member on a lower grade acting in management positions during gaps. It was clear from staff interviews that significant institutional memory loss happens annually within the CRAM programme. Another challenge is that staff members on lower grades are asked to act in higher level positions during gaps in staffing, but there is no opportunity for them to be promoted themselves into these positions. There is frustration among staff holding management positions who need to engage in tasks that should be carried out by those reporting to them due to staffing gaps or capacity issues. There have been some efforts made to address the lack of mobility within the staffing structure, notably with the creation of Deputy Programme Manager positions. The organigram for CRAM has been changed repeatedly over the four years of implementation, in attempts to align it with the personnel the programme could actually recruit. Ironically, the last version of the organigram is almost identical to the proposed organigram at the conception of CRAM, which leads to the conclusion that the organigram structure itself is not the issue and if all positions were filled it is appropriately developed. In conclusion, staffing issues have consistently affected the implementation of CRAM and have been a source of institutional knowledge loss in addition to shifting unmanageable workloads to other staff members. Staff salaries do not appear to be competitive enough to attract highly skilled staff willing to remain in Goz Beida. There are also practical considerations that could improve staff retention (new staff buddy system, improvements to office and accommodation facilities), and improve institutional knowledge retention (stronger handover systems and filing systems). Some opportunities for existing staff to progress to higher level positions are being created but there remain challenges with this. 15 | P a g e

Monitoring and Evaluation The research component of CRAM cost approximately €650,000 over the four years. This shifted certain responsibilities from the programme team in terms of monitoring and evaluation, allowing them to concentrate on formative assessment and implementation. This, overall, seems to have worked well for CRAM, without which it is doubtful that the full data collection could have been managed internally, given the frequent staff shortages and changes. Good formative assessment systems were visible within each sector, with clear integration within monitoring. For example, the FIM and nutrition teams each have one person assigned to lead the collection of EWS data, working together as a team. Both individuals seem to understand the data they are collecting and the reasons for it. The WASH team has a clear system for borehole water testing (sending samples for analysis in N’Djamena) and communicating results to beneficiaries before opening the water supply. The health team regularly use data from national-level surveys, and contribute to these, including the SMART survey. Standardised monitoring forms are available – with findings collated and used by the teams – for visits to health centres and the various committees developed by Concern (comité gestion d’eau, relais, éleveurs etc.). In general, it is the use of monitoring data and learning that is the greatest point of loss in this programme. The data is collected well, with considerable investment. However, it is not then translated into useful information for the team, and partners, to use to inform programming. The baseline report and midline reports are available in English only, and are academically worded. Monthly reporting (situation reports) are developed in Goz Beida and sent to N’Djamena, although there is a lack of clear management responses to these. The same is true of all adviser reports or visits – there has been a high involvement of technical supports from headquarters over the four year period, but none of these reports includes a management response. Technical adviser visit reports come in many different formats, often difficult to navigate and with too many recommendations, again in English. There has been no capacity-building of Concern staff to independently analyse the EWS data for use in Chad – there is a sense that they send the data off to the US (to Tufts), where it then belongs to others. There are very few learning papers to share learning from the CRAM model. The only document available is an overview of CRAM developed in 2013 in English and French, which is still used by SMT in their discussions with other stakeholders. There are also some power-point presentations of CRAM from the roundtable, available in French. The current team in Chad did not have a full set of these materials, probably due to staff changeover and institutional knowledge gaps. There are currently no opportunities for government partners to understand the programme design, the learning coming through and the kinds of questions they can ask based on the data collected. CRAM needs to leverage the investment made in research to improve programme learning and government learning – it is not too late for this and it could be addressed better in the BRACED programme.

Effectiveness Unsatisfactory Overall Micro Meso Macro

16 | P a g e

Acceptable (with major reservations)

Satisfactory X X X X

Highly Satisfactory

The most striking finding of this evaluation is that CRAM achievements are considerable – significant changes are visible in the intervention villages and these are reflected in the midline assessment findings. Given the serious contextual challenges in Chad, these findings are impressive. There were significant long-term staffing gaps throughout programme implementation, so the achievements outlined below are a testament to the hard work of those who did remain in Goz Beida and N’Djamena, and Concern’s support structures. In the absence of an end-line, progress against indicators can only be measured until December 2014. It was agreed during evaluation design that no additional data should be collected at this stage, and programme monitoring data during 2015 does not add sufficient information to make judgements against further progress (nor would it be expected to). As such, the evaluation measures change according to two measures. Firstly, was CRAM on/off-track to achieving these results in 2014? Secondly, were changes in variables from baseline to midline different by control and treatment? The second is presented below as difference-in-difference (a robust measure that allows the analysis to control for any possible disparities in the indicator at baseline that might not have been evenly distributed despite randomization) at a 95% confidence interval (p ≤ 0.05). While progress is less visible in the data below due to the absence of an end line report, findings are also based on the observations of the evaluation in villages and conservative judgements on progress made due to activities since the midline was conducted. Activity reports since the midline show high levels of intervention, so this is considered when making judgements on likely change with an additional year of intervention. Additionally, more weight is placed on the reduction in severe acute malnutrition (see below) as this reflects impact-level programme success. Number of Indicators Assets Inequality Risk and Vulnerability Total

7 5 4 16

Number where midline to end line target On-track Off-track Insufficient data 3 3 1 3 2 0 0 2 2 6 7 3

Difference-inDifference 2 1 0 3

Assets, and return on assets The CRAM programme provides a good example of an integrated programme. Interventions are planned and integrated using a multi-sectoral approach, and programme staff can competently describe the linkages between programmes and the rationale for this. To improve access to food security and improve nutritional levels of children, clear linkages are made between FIM (improved returns on agriculture, improved animal health), health (reduced negative coping strategies, interventions for children identified as malnourished, improved vaccination coverage) and WASH (increased access to potable water, reduced open defecation). In the absence of end line data, findings in this section are based on evaluator interviews, focus group discussions and observation, supported by findings at the time of the midline. These are presented under the three dimensions of extreme poverty – assets, inequality and risk and vulnerability. FIM interventions to increase income levels in the household mainly focused on reducing postharvest losses, including grain storage and timely market sales. For agricultural interventions programme works in coloration with the ONDR and plant health (phytosanitaire) offices although it is unclear how far collaboration extends beyond joint training, so much of the agriculture work 17 | P a g e

within communities seems to be Concern-led. It is not possible to make judgements on improved crop yields at this stage, but the general consensus is that 2015 will be a poor harvest year due to late rains. Beneficiaries reported having enough seeds to plant, having stored them for re-planting last year, and having improved strategies for storing grain and knowing when to sell it. In general, beneficiaries report improved food security based on CRAM supports. While this knowledge was strong, some women said they will struggle to store grain for selling later this year because they need to put food needs first. There were examples of collective grain stores, including multiple communities storing grain together to reduce the number of people needed to protect it. High levels of insect infestation are reported in all villages, but some farmers (through the paysan pilotes) have used locally-made insecticide following CRAM training and this was effective. Innovative approaches were piloted to increase soil fertility (the introduction of ECOSAN latrines) which seem to have community buy-in but have yet to prove their worth in terms of improved crop yields for farmers. At meso-level, the programme supported the inception of a new veterinary pharmacy owned by the civil society group Fédération des Eleveurs in Goz Beida, directly linked to the government department (Délégation Régionale de L’Elevage). This is an excellent example of engagement with private sector – the pharmacy runs as a viable enterprise, but only provides approved veterinary drugs which are more reliable than those available within the market. Auxiliary representatives are trained to sell these drugs at community-level, replenishing their stocks when they visit Goz Beida. The pharmacy is located next to the délégation, allowing farmers to use veterinary services there and access animal vaccinations. The pharmacy was clean, well-stocked and had several customers during the evaluation visit. Auxiliary services were working well according to the pharmacy, although these need to be followed up at community level to ensure that cost recovery systems are maintained. This example of successful private sector service provision should be documented for wider learning within Concern. Health interventions to improve the nutritional status of children under five are mainly focused on community-led support groups which are then linked to health services. Originally the CRAM model included the provision of mobile health clinics, but during implementation they developed more sustainable models of support through the stratégie avancée. This strategy establishes outreach centres (very basic structures at community level) for the team from the health centres to visit according to a regional schedule. Travel time is greatly reduced (mobile clinics were based in Goz Beida and spent most of the time traveling to villages), services are integrated directly into the national health system and there are closer links between health centres and their satellite communities. This is a relatively new strategy for the programme and makes good sense, although there are still outstanding challenges (e.g. tracking actual coverage by health centres to ensure they take place). The health centres are well stocked and supported in general, with a nurse, mid-wife and health-care aide. There are hygiene issues within these centres though – latrines without water and soap, medical waste on the ground (including syringes) – the hardware is in place (latrines, wells, furnace and bins) but increased training and monitoring is needed to improve hygiene practices. Cost-sharing initiatives and direct collaboration with the regional health delegation, with a clear exit strategy, are in place and well-designed. WASH interventions are the most visible success of CRAM – coverage of water points in all intervention villages is impressive. Wells visited were managed to a high standard, in use and located centrally. Where additional wells are planned, there was a clear rationale and collaborative process to identify the new location. Water management committees were competent in their roles – examples were given of repairs to wells that had broken and fee-waiver systems for the most 18 | P a g e

vulnerable. Reducing open defecation has mixed findings – there are decent family latrines available in all villages visited, in use, with water but no soap. However, there are also many family latrines that collapsed in the rainy season that are no longer in use. Also, in general, institutional latrines in villages are not used. It appears that CTLS worked well to trigger disgust and latrines were dug, but the sandy soil led to a collapse. The WASH team has developed a good strategy to address this, through a competition with local masonries to develop solutions using local materials. There seems to be good knowledge change on hygiene – particularly hand-washing and washing jerry cans, but there is little or no soap availability so the programme should explore better ways to use ash or alternatives. Water chain management systems, to improve hygiene practices for water storage and use (e.g. washing containers) are promoted and initial findings in the mid-term assessment attribute decreased SAM rates to households washing water containers regularly.8 Evidence of effectiveness: assets and returns on assets Midline results show that the programme was on-track to achieving fewer than half of the programme targets. Increased assets and returns on assets were most visible in terms of water use – 50% of households reported access to 15 litres of potable water, per person, per day at midline, compared to just 11% at baseline. This is attributed to the construction of boreholes in target villages, together with training a water management committee in each village. Progress in reducing open defecation is off-track, however, with a significant increase (difference-in-difference) in the percentage of households practicing open defecation from baseline to midline in target villages. This is attributed to the collapse of many hand-dug latrines during the rainy season due to the sandy soil. The nature of cyclical food shortages linked to unpredictable annual rainfall leads to challenges when comparing annual indicators on food availability – the baseline was conducted in a year of good harvest, while the 2014 midline was conducted in a year of poor rainfall. The Global Acute Malnutrition Rate (GAM) is measured as on-track, decreasing from 14.8% to 13.9%. There is no direct indicator measuring the Severe Acute Malnutrition Rate in the results framework – instead it looks at the percentage of children under five with SAM who received nutritional support in the last two weeks (i.e. treatment of SAM – an output indicator rather than outcome). This indicator is measured as off-track, from 40% at baseline to 25% at midline. This means that health services reported treating fewer children for malnutrition (which could arguably be due to poor health service coverage (negative) or fewer patients needing treatment (positive)). Importantly, data collected by Tufts on the actual SAM rate showed a significant difference-in-difference between baseline and midterm, compared to the control – the rate of severe acute malnutrition was significantly reduced in the treatment villages. This is a very positive finding at outcome level. Further details on this finding are provided under discussions on impact below, but suffice it to say the programme has made progress in addressing malnutrition through CRAM.

Inequalities Interventions to improve dietary diversity for the most vulnerable mostly focus on the maraîchage gardens as these are a source of nutritious vegetables and pulses. The promotion of gender equality and women’s participation in the community and at household level in decision-making is mainstreamed through all activities. Maraîchage seasonal gardens are planned collaboratively by FIM and Nutrition (to identify appropriate seeds for nutritious value) and WASH (to identify irrigation solutions). These 8

This is a preliminary finding that will be explored further for verification at end line.

19 | P a g e

interventions are very popular and seem to enjoy good successes, despite late seed distributions by Concern. These are usually communal ventures, often led by groups of women. Analysis by FIM and Nutrition post-harvest show increased consumption of some vegetables, while others are sold at market. Irrigation schemes have not found the right solutions to date – the foot-led pump described in the CRAM proposal was not used, and the motor-based alternative used was too high in fuel costs. Overall, while best practices are still being identified, maraîchage gardens are widely used and increasing the diversity of foods available in addition to providing a second harvest option. Initiatives to improve gender equality and women’s participation are visible in the clear targeting of women for all activities. As primary beneficiaries, women enjoy more direct supports in this programme and can confidently describe their roles within specific activities. There were no challenges meeting women and men together, although it is clear that gendered seating practices are still the norm (separate mats until called together, separate food bowls). The high number of female staff within CRAM also seems to have a positive influence in communities – women call out to Concern staff and greet them openly, while certain social barriers are broken down by the visitors (sitting together for food, for example). All committees met had female representation, although men still dominated most of the conversation and women responded only when men had already spoken. During women-only groups, respondents said that women are still responsible for all housework and most farming work, and they still have no control over decisions on their own health or that of their children. There are some good strategies to address this under the BRACED programme, but the Concern team in Goz Beida is not yet equipped with the staff and skills to deliver this – the work plan is unclear and responsibility for key tasks is unclear. Community Conversations were not used within CRAM as planned in the proposal, nor were specific activities to engage men as change agents – much more focus is needed on these issues moving forward. Evidence of effectiveness: reduced inequalities Outcomes relating to reduced inequalities showed mixed progress. As described above, no progress was measurable against GAM rates and there is no significant difference-in-difference between control and treatment groups. There was a significant decrease in the dietary diversity score for both children and adults across both treatment and control groups from baseline and end line, attributed to the baseline year having an unusually good harvest. The mid-term report does indicate a difference-in-difference between target groups, with higher diversity in treatment villages, but this is not detailed further and will remain unclear until the end line results are available. Progress has been made in increasing gender equality – the percentage of women reporting participation in decision-making within community groups increased from 47% to 67%, exceeding the programme target of 65%, while attitudinal scores on gender equality among change agents (men and community leaders) increased against all measures, again exceeding programme end line targets. At midline this change had not yet influenced women’s participation in decision-making at household level, with little or no progress against this indicator from baseline to midline.

Risk and Vulnerability Considering the risks of environmental degradation in Chad, FIM activities to increase soil fertility and protection are initiated, along with the establishment of environmental protection committees. Improved hygiene practices, particularly use of soap, are encouraged. Specific interventions to address the needs of vulnerable groups, especially pregnant and lactating women are included. Responsibility for improved knowledge on HIV is left with the programme quality team.

20 | P a g e

Health interventions to meet the needs of vulnerable women and young children are met through community groups linked to the health centres – both the relais communautaire (identifying malnourished children, referring them for treatment and following up on cases) and the Groupe de soutien mère à mère committees (promoting exclusive breastfeeding, child feeding practices and vaccination schedules). These groups are used to promote multi-sectoral messages including improved hygiene practices, animal care away from children and diversifying crops to increase dietary diversity. Data is unavailable at present to measure the effectiveness of these activities in increasing breastfeeding practices or to attribute changes in SAM or GAM directly to these activities (this information may be available following the end line). Given these limitation, findings on these interventions are largely anecdotal, based on focus group discussions at community level. The initiatives were appreciated by those directly engaged and by the wider community, and are perceived to have improved practices for maternal health and childcare. Knowledge on breastfeeding practices and hygiene practices was good among the few respondents interviewed. The intensity of support in intervention villages (one relais for fifteen households instead of two per village) seems appropriate given the scale of malnutrition prevalence. One woman interviewed during the evaluation, Hawa, had been supported by Concern’s humanitarian work in 2012 as her 18 month-old daughter, 5: PHOTO: MANIRA, LEFT, AGED 5, WITH HER SISTER Manira, was admitted for therapeutic feeding RAZAKA AGED FOUR 4, TAKEN BY J ENNY HOBBS , K ADJASKE , G OZ B EIDA . over a four month period. She benefited from training on improved nutrition practices and became a leader of the Groupe de soutien mère à mère. She believes her younger children are healthier and did not need therapeutic care as a result of the changed practices, so she uses her own experience to train others. Hawa is currently practicing exclusive breastfeeding of her son – women willing to take on this practice are identified and promoted in the community as advocates. During focus group discussions a few women were identified in each community as successfully practising exclusive breastfeeding, but it is clearly a rare practice and will take more time to bring widespread change. All committees, across all sectors, have responsibility for hygiene promotion, water chain management and promotion of use of potable water. FIM interventions to promote conservation agriculture have been rolled out through the paysans pilotes initiatives. This has yielded mixed results. Fields observed had healthy crops, comparatively equal to or healthier than neighbouring fields. Crop rotation or crop mixing has been widely adapted, in addition to crop spacing. However, there is some confusion around soil coverage and this is not practiced widely. Minimal soil disturbance practices are not followed; most farmers used donkey/horse-led tills or tractors to prepare the soil for planting. Environmental committees are established and supported by the FIM team, under disaster risk reduction activities. Linkages with meso-level initiatives are well established, with the forestry inspectorate and with the phytosanitaire unit. Knowledge on tree planting, soil erosion and protection was high among committee members and among beneficiaries not directly involved. 21 | P a g e

Evidence of new tree planting was visible and no obvious signs of tree cutting were seen, although these might not be immediately visible to visitors. HIV mainstreaming is not strong within CRAM. It is deemed to be the responsibility of one staff member, who is also responsible for gender mainstreaming and manages the CRM helpline during distributions. He is given time allocations during trainings held by other sectors to teach participants key protection messages on HIV, but could not possibly reach all training sites. Condoms are not available in Concern offices or at training venues, although reportedly they were supplied and repeatedly thrown in latrines by staff indicating cultural rejection of condoms and the need to work more with staff on HIV protection. No specific work is done to support people living with HIV and AIDS or to address stigma within communities. Evidence of effectiveness: reduced risk and vulnerability Progress against reduced risk and vulnerability is not available for two of the four indicators (uptake of conservation agriculture techniques and knowledge on HIV and AIDS transmission) because these are scheduled for measurement at baseline and end line only. Progress against the remaining two indicators is off-track. The percentage of respondents with correct knowledge and practices on hand-washing increased (from 45% to 53% for knowledge on critical times for hand-washing), but none of these increases were sufficient to reach midline targets. It is also difficult to draw strong conclusions on hand-washing as data collection methods differed from baseline to midline, so end line data is needed for comparison with the more accurate methodology used at midline (Tufts did not include this indicator in the baseline stating that it is better covered in a knowledge, attitudes and practices survey by the Concern team, a decision which was reversed for the midline). No progress can be measured against attempts to increase exclusive breastfeeding practices, which are found to be among the lowest in the world when collected at midline. Again, the methodology of data collection was improved at midline, which means the findings are incomparable with baseline (the period of recall was reduced from six months to 24 hours to improve accuracy).

Advocacy Although there were no specific outcomes relating to advocacy in the CRAM results framework, significant efforts were made in N’Djamena to build and contribute to national-level resilience platforms. A Round-table Conference on Reliance was organised by Concern in December 2013, bringing together government officials, academics, donors and NGOs to achieve a common understanding of community resilience, to develop a forum for key actors to share learning on resilience, and to increase donor interest in community resilience. The round-table does appear to have effectively achieved these objectives to some extent, as evidenced through stakeholder interviews. The common forum for resilience learning shifted into the AGIR process, to which Chad is a member and signatory. Building on relationships developed at the round-table, Concern became actively involved in the AGIR process, participating in three of the four working groups – nutrition, food security, and governance. Through these working groups Concern participated in the development of the national strategy on resilience, which was approved in 2014. Relationships were maintained with key partners (particularly the EU and CARE) and Concern contributed to an analysis of national-level policies on resilience by commissioning a report on Chadian policies through a resilience lens in 2014 by Peter Gubbels. Ironically, many of these significant achievements were identified from sources who have now left the Concern programme in Chad or from government representatives within Chad. It seems that advocacy efforts were led by SMT in Chad and required a high level of individual commitment from 22 | P a g e

people with high workloads. This resulted in very little documentation – there is no advocacy strategy, work plans, contact lists or advocacy materials to share. In 2015 there was a change in SMT (currently the country director and programme director positions are vacant or filled with SURGE staff) and engagement at national level was maintained through the UN Cluster System and the semi-operational NGO partners forum, so relationships on CRAM advocacy have fallen down. However, these could be picked up again easily – all stakeholder interviews at national level indicated high levels of respect for Concern’s leadership of the resilience agenda, so the evaluation of CRAM provides a good opportunity for meaningful re-engagement. The CRAM model in Chad has also influenced Concern’s organisational understanding of resilience. Programme findings and design have informed advocacy agendas in Ireland and in the UK, and have facilitated funding opportunities (e.g. BRACED with UK AID / DFID for both Chad and Sudan). It is disappointing that there is no real documentation on learning to date, however, other than academic papers led by Tufts. Given ongoing staffing constraints this is unsurprising, but having such documentation would facilitate better advocacy based on programme learning. Efforts were made to assign responsibility for such initiatives to a Monitoring, Evaluation and Learning Manager, but there were challenges filling the position and the existing workload was very demanding when someone was in place.

Impact Unsatisfactory Overall Micro Meso Macro

Acceptable (with major reservations)

Satisfactory

Highly Satisfactory

X X X X

Judgements on the impact of this programme are based solely on findings of the midline assessment. In the absence of an end line this is used as a proxy, although it is hoped that further progress has been made in the eleven months of implementation since the midline. The greatest programme impact is visible in the reduction of severe acute malnutrition in villages smaller than 150 households (74% of all villages in the sample). CRAM was found to be successful at reducing wasting in villages that are smaller than 150 households. While there is no significant difference in GAM, SAM or weight-for-height between treatment groups using weighted population statistics, this changes when un-weighed sample statistics are used. A significant difference in SAM rates is found, which remains highly significant when controlling for child and village level characteristics. To determine why this is significant within the sample statistics F IGURE 6: WASTING PREVALENCE BY TREATMENT CATEGORY and not population-based, results are AND VILLAGE SIZE 23 | P a g e

correlated by village size. Three quarters of villages are considered small in the programme, but the cluster design assigns a higher weight to households in large villages. When only considering small villages CRAM demonstrates a significant impact on wasting levels, which may be linked to the number of inputs given to all villages (small and large villages had the same level of input until midline), which has now been amended for the final year of implementation. It is hoped that this impact on wasting will be visible again at end line, potentially in both large and small villages. Other findings showed increased support for malnourished children (GAM) in treatment villages from 13% at baseline to 32% at midline (while dropping 10% in control villages). The proportion of children that were reported to have received care, either from a mobile clinic, health centre or hospital if ill increased for treatment villages and reduced for control villages. There was a significant reduction in households relying on the marabout (spiritual healer) for health services in the treatment villages (5% difference, p<.05). The percentage of households sourcing their water for household consumption from potable water significantly increased in treatment villages, while consumption from non-potable sources or surface water decreased, both over time and just comparing control and treatment villages.

F IGURE 7: POTABLE WATER USE OVER TIME AND ACROSS TREATMENT GROUPS The coping strategy index, used to identify behavioural responses in times of shortfalls in food consumption, looks at immediate and short-term alterations of consumption patterns to cope with shocks. While there was no significant difference overall on this indicator, households in treatment villages were significantly less likely to say they consumed seed stock for the following year. Longitudinal data was also collected on the coping strategy index, from a subsample of the survey population, to show seasonal patterns. This shows higher food insecurity between July-September (the hunger gap) followed by a dramatic drop in food insecurity in October – December (harvest period). Most telling to understanding the impact of CRAM is the timing of the difference between control and treatment villages. In May, there is a significant difference, with higher levels of food insecurity in the control village. F IGURE 8: MONTHLY FOOD INSECURITY BY TREATMENT That difference vanishes in June, corresponding to blanket food distributions carried out in all villages. The peak and most continuous significant variation between control and treatment is observed at the height of the hunger gap – 24 | P a g e

July – September. This implies that CRAM does have a significant impact on reducing food insecurity during the hunger gap.

Sustainability Unsatisfactory Overall Micro Meso Macro

Acceptable (with major reservations) X X

Satisfactory

Highly Satisfactory

X X

The programme has clear mechanisms built in to improve sustainability, and its very design is based on more sustainable ways to address recurrent shocks. That said, Chad is not an environment conducive to sustainable programming in any real sense – government services are often absent and there are, at times, no systems to strengthen. For this section, CRAM is found to have good mechanisms in place to address sustainability, but to also have service delivery components that cannot continue without ongoing inputs from Concern. The score is therefore assigned as 2, bearing in mind the contextual limitations. Considering the situation before CRAM was in place, the steps taken by CRAM to shift away from ad-hoc emergency response and to build resilience were very impressive. If the end line assessment finds considerable behaviour change then this score could conceivably be too low, but in the absence of this data during the evaluation, a conservative score of 2 is considered appropriate. At micro-level the programme provides intensive support to intervention villages through both hardware and software inputs. Villagers know most Concern staff members by name and it is clear that frequent visits take place all year around. Trainings are directly delivered by Concern, or organised by Concern with facilitators from government departments. Committees and care groups are accustomed to regular visits by Concern staff to support them in planning for their upcoming activities. Health services, in particular, are possible only through the payment of salaries for health centre staff by Concern. Mechanisms have been established to improve the sustainability of these supports – a memorandum of understanding has been developed with the health ministry, documenting a phased withdrawal of Concern supports. Staff salaries are currently paid via the health ministry, so that health centre staff report directly to the health ministry. Mobile clinics have been stopped and the Stratégie Avancée is being rolled out to all sites. Direct inputs of fuel, vehicles or supplies are no longer provided and all health centres visited were well stocked with equipment and supplies from the health ministry or from UNICEF. A cost recovery strategy is in place in health centres for medicines not provided for free (malnutrition treatment and vaccinations are provided by the health ministry at no cost). While respondents in villages and in the government lamented the loss of fully stocked mobile clinics providing free medical care, the new strategies are much more practical and aligned with system strengthening. The challenge will be ensuring that services are continued when Concern has less direct involvement, through CRM systems to track activities of the stratégie avancée, for example. At meso-level, there are some excellent examples of sustainable programming, and some examples of poor sustainability. The establishment of the veterinary pharmacy was an initiative based on costrecovery that meets a real need in a sustainable way. The engagement of government stakeholders in training, using CRAM as a mechanism to roll out national-level initiatives to underserved areas, is 25 | P a g e

very positive. Through meetings in N’Djamena, the programme supported travel costs for a representative of the Hydraulics Department to be based in Goz Beida (the first representative of WASH in Sila). However, there are no mechanisms for CRAM partners to meet regularly in Goz Beida for collaborative planning, or to learn from programme learning, which inhibits the programme’s capacity to influence a multi-sectoral resilience approach by government agencies. At macro-level, advocacy efforts to date are pitched at the right level – CRAM has responded to and fit into government agendas and structures. Although this has dropped in recent months and lacks effective documentation, there is a sound basis for continued engagement. CRAM learning will be ready for sharing in early 2016 and this has the potential to renew advocacy efforts through a welldeveloped strategy, potentially working globally, regionally and nationally. To achieve this, linkages between government representatives in Goz Beida and their counterparts in N’Djamena should be actively supported. Similarly, Concern staff in Goz Beida (sector managers) should be facilitated to spend more time liaising with national-level departments to build Concern’s profile in technical areas and create opportunities to influence change.

Conclusion Developing and testing the CRAM model in Chad was an ambitious task. The contextual challenges (security, food insecurity, displacement and staffing) could have deterred Chad’s SMT from accepting the model as their own, but instead it was embraced as an opportunity to shift from adhoc emergency responses to building community resilience and addressing the long term drivers of extreme poverty. This evaluation finds that the programme has, overall, been effective in achieving what was intended. Although the evaluation is limited by the absence of an end line, there is sufficient evidence from the midline assessment, document reviews and key informant interviews to conclude that intervention communities have increased resilience to shocks as a result of CRAM. There are successes to note, in addition to lessons to learn from. Key successes are in programme implementation – particularly in terms of the integrated manner in which activities are designed and delivered. Joint planning between nutrition and FIM teams to ensure that seed distributions and household feeding practices are aligned to nutritional objectives is a key strength. The wide coverage of WASH support, particularly potable water, and the attention to all stages of water hauling from the point of extraction to the point of consumption, is a strong achievement and initial findings in the midline indicate that this may be attributable to significant reductions in severe acute malnutrition. Another key strength is the programme’s ability to take risks and experiment with innovative approaches. This is initially visible in the inclusion of a strong research component, welcoming external researchers into the Goz Beida team. It is again visible in the piloting of new ideas – the veterinary pharmacy, irrigation systems, ECOSAN latrines, the stratégie avancée for health and the uptake of conservation agriculture. These experiments have worked in most cases, or are in the process of working, but where they are not; the team shows an awareness of the issues and has strategies in place to find solutions. Overall there seems to be a healthy appetite for risk and problem solving among programme staff. Inefficiencies in programme implementation have affected progress. Logistical constraints in procurement systems and transport systems have been a source of frustration for staff and for partners, possibly due to poor planning (by programme and systems staff), staffing gaps, missing protocols/manuals, or weak adherence to the protocols. These have led to major problems in 26 | P a g e

programming – such as the late distribution of seeds for maraîchage gardens each year – with direct effects on the lives of programme participants. Human resource constraints remain a major problem – currently visible in the lack of senior management positions filled and a reliance on SURGE staffing. Systems to manage high staff turnover – such as handover and filing systems – are not in place, leading to continuous cycles of institutional knowledge loss. Women and children under the age of five are the primary beneficiaries of the programme. While targeting and direct beneficiary engagement focuses on women, more could be done to address the power imbalances and effect behaviour change through CRAM. In particular, initiatives to engage influential male leaders to promote equality and initiatives to support increased decision-making at household level are needed. HIV mainstreaming is weak in the programme and no engagement with staff to address negative attitudes to condom use or to engage them in learning on HIV is visible. The inclusion of a research component in CRAM is justified and provides a strong evidence base for Concern, donors and governments to scale up resilience programming. Tufts has been a flexible and responsive institute to work with and positive relationships have been developed. However, this investment has not been sufficiently leveraged by Concern. Research findings have not been translated and made available to programme staff in usable forms; learning has not been documented within Concern and government partners have not been engaged in the learning process to generate ownership of the findings. It is not too late to change this – as BRACED commences, programme learning from CRAM could be shared widely (in appropriate forms) and the next stage of resilience programming could better leverage the investments in research. Overall, CRAM has been effectively implemented, despite significant staffing and contextual challenges, and has been successful in contributing to improved resilience in Chad.

Recommendations 1. Improve learning and information pathways based on CRAM research. a. Communications and documentation: To improve Concern’s learning from CRAM, and to enhance opportunities for meso and macro-level advocacy, some steps could be taken to improve communications and documentation. One is to encourage jointauthorship of academic publications based on CRAM (currently these are authored by Tufts staff and their consultants only), by enabling Concern staff – or even government partners – to co-author papers that are directly related to their work. Another is to build closer ties between the advocacy team in the UK and Chad, in the form of technical support, to co-develop an advocacy strategy and to lead on the development of learning papers, posters and leaflets (in French and English) based on CRAM. Mechanisms should be explored to assign responsibility for documentation to existing staff without overburdening them. b. Joint learning and planning events: Following the CRAM end line, a joint reflection meeting is recommended in early 2016, bringing together Tufts, Concern, government representatives and partners in Goz Beida. Regular partnership meetings are needed in Goz Beida bringing together all government agencies working within CRAM (or BRACED). c. Advocacy strategy: An advocacy strategy should be developed, with mechanisms for Concern to re-engage in the AGIR network and linking programme staff based in Goz Beida to ministry structures in N’Djamena (through regular visits). 27 | P a g e

2. Improve accountability mechanisms at community level a. Complaints Response Mechanism: A CRM should be established, taking the low levels of literacy among community members into account. Programme planning and learning should be available in accessible forms to community members so they are aware of why and how activities are planned, why the research component is included and how it relates to them. 3. Increase attention to risks and vulnerabilities – particularly addressing gender inequality, HIV and AIDS mainstreaming, and children a. Gender equality: The recommendations provided in the last Equality Adviser report are strong, but are numerous and are not understood by staff. The adviser should be engaged to support the team in planning and implementing specific interventions to challenge gender inequality, possibly using mechanisms the team have worked on in the past (there is some knowledge of Community Conversations among the team). Responsibility for this should be shared among all staff, not one staff member. b. HIV and AIDS mainstreaming: One staff member is currently responsible for HIV and AIDS mainstreaming within programmes, rather than all staff. The focus of this should shift from him directly training programme participants, to training staff and building HIV and AIDS training into their work plans. Past experiences where staff rejected the idea of condom stocking in staff toilets should be dealt with – consider piloting Community Conversations techniques among Concern staff first (as this is the mechanism with which the Deputy Programme Quality Manager is experienced in facilitating). c. Children as agents of change: While children under the age of five are direct beneficiaries in the programme, children are not sufficiently engaged as agents of change. There have been some attempts to engage them in WASH messaging, but across all spheres of behaviour change there is no clear strategy to target children. This is a missed opportunity (children can be excellent vehicles for messaging), but also presents risks to the sustainability of changes – children are engaged in farming and household chores so they have a direct impact on hygiene practices, environmental practices etc. d. Children as beneficiaries: Orphans over the age of five, particularly those living with grandparents, were mentioned as vulnerable groups not currently supported. The programme should find out more about this group and see if their needs can be met through existing activities funded under BRACED. The CRAM model included education interventions in its design, aligned to the needs identified in the area and the links to resilience objectives. Education interventions were due to start in year two but funding and capacity constraints did not allow for this, and it is not currently built into the BRACED. There is a clear need and it is an expressed priority of beneficiaries and Concern staff, so funding opportunities to support education should be explored. 4. Revisit and reflect on programme interventions that demonstrate mixed results to develop strategies to improve these The eight programme interventions below have been identified as having mixed results, with examples of them working well, or working partially, or not working. Findings are provided in this report and were shared with individual managers and in many cases they have strategies to improve these (not necessarily tested or documented yet). Prior to scale up through the BRACED programme, managers should revisit each strategy and identify and 28 | P a g e

test ways to strengthen their effectiveness. Recommendations should be documented by managers on best practice for staff in supporting these strategies. a. ECOSAN latrines: The location of latrines and of fields should be clarified. b. Latrine construction: Construction solutions for sandy soil should be identified. c. Small-scale irrigation solutions: Well depth and power sources for irrigation pumps to be improved. d. Hygiene in health centres: To compliment hardware inputs, behaviour change on waste management and hygiene practices should be improved. e. Stratégie avancée: Mechanisms to track the attendance of healthcare workers at sites should be introduced. f. Conservation Agriculture: Minimal soil disturbance and soil coverage practices to be improved. g. Crop pest control: Coverage of organic pesticide use to be increased. h. Pharmacy Auxiliaries: Follow up and mechanisms to support auxiliaries to manage restocking and cost recovery are needed. 5. Support Systems need to be strengthened – particularly procurement and logistics a. Technical support: To support the team in developing and operationalising standard procedures for procurement and transport, increased technical support from logistics advisers in Dublin is recommended. This has budget implications which should be considered in upcoming budget revisions. b. Staff training and management: Building on changes currently being initiated in Goz Beida, programme and systems staff should be trained in (and held accountable to) correct planning and procurement procedures. This is especially relevant to programme managers aligning annual and quarterly activity plans to procurement plans. Given the high staff turnover this training should be built into annual plans for regular intervals. Hopefully the recent recruitment of a Logistics Coordinator will also help these initiatives.

29 | P a g e

Annex 1: Management Response Template Recommendation 1: Improve learning and information pathways based on CRAM research.   

Communications and documentation: Joint learning and planning events: Advocacy strategy:

Management Response 1: Key action(s) (to be added to as Time frame appropriate)

Responsible Unit(s)

Recommendation 2: Improve accountability mechanisms at community level (Complaints Response Mechanism)

Management Response 2: Key action(s)

Time frame

Responsible Unit(s)

2.1 2.2 Recommendation 3: Increase attention to risks and vulnerabilities – particularly addressing gender inequality, HIV and AIDS mainstreaming, and children    

Gender equality: HIV and AIDS mainstreaming Children as agents of change Children as beneficiaries:

Management Response 3: Key action(s) 3.1 3.2 Recommendation 4: 1|Page

Time frame

Responsible Unit(s)

Revisit and reflect on programme interventions that demonstrate mixed results to develop strategies to improve these        

ECOSAN latrines: Latrine construction Small-scale irrigation solutions: Hygiene in health centres Stratégie avancée: Conservation Agriculture: Crop pest control: Pharmacy Auxiliaries:

Management Response 4: Key action(s)

Time frame

Responsible Unit(s)

3.1 3.2 Recommendation 5: Support Systems need to be strengthened – particularly procurement and logistics  

Technical support: Staff training and management

Management Response 5: Key action(s) 3.1 3.2

2|Page

Time frame

Responsible Unit(s)