reflection document - Dutch Wash Alliance

Sustainability Monitoring Framework
Don’t forget
gender
Source: Maaike Kempkes, 2013
DOCUMENTATION OF PROCESS AND
MAIN LESSONS LEARNED
IRC International Water and Sanitation Centre
The Hague, January 2014 (v1.0)
Sustainability Monitoring Framework
Documentation of Process and Main Lessons Learned
Page 2 of 14
Sustainability Monitoring Framework
Content
1.
INTRODUCTION ................................................................................................... 5
2.
OVERVIEW SMF/SI PROJECT DEVELOPMENT 2012-2014 ....................................... 6
3.
THE PROJECT DELIVERABLES ................................................................................ 8
4.
THE DWA GUIDANCE GROUP AND IRC TEAM ....................................................... 8
5.
MAJOR LESSONS LEARNED .................................................................................. 8
Documentation of Process and Main Lessons Learned
Page 3 of 14
Sustainability Monitoring Framework
1. INTRODUCTION
The Dutch WASH Alliance (DWA) is a consortium of six Dutch NGOs working together
towards a society in which everybody makes hygienic use of sustainable water and
sanitation. In this alliance of Simavi, Akvo, AMREF Flying Doctors, ICCO, RAIN and WASTE,
these organisations intensify their already ongoing cooperation and coordination in
developing countries in order to further increase the effectiveness of our programs and to
learn from each other.
One of the main aspects the DWA is concerned with is the sustainability of the projects and
services it supports. IRC International Water and Sanitation Centre based in The Hague and
a thematic partner in the DWA was asked to develop an instrument allowing the DWA to
measure the sustainability of the DWA-implemented WASH systems and services. The
collaboration between IRC and DWA and the results of Phase-1 - assessments on
monitoring (including sustainability) by the local DWA partners and the WASH sector at
large in the two selected DWA countries, Uganda and Ghana - has led to the development of
this sustainability monitoring framework (SMF). The approach on SMF allows for
embedding sustainability monitoring rather than merely measuring it in the DWA projects,
the scheme level and the local government level, and possibly at national levels. The SMF
also includes a Sustainability Index (SI) as relative measure for sustainability - or the
likelihood of sustainable WASH service or practice - rather than an absolute measure of
sustainability.
The SMF/SI project was implemented during the period June 2012 and January 2014. In
this 20 month-period, IRC was guided by the SMF Guidance Group, a sub-group within the
DWA PME Group.
In collaboration with the Guidance Group, the Country Coordinators and selected local DWA
partners in Ghana and Uganda, the IRC team developed and tested the Sustainability
Monitoring Framework and the Sustainability Index (SMF/SI), a promising instrument in
measuring the likelihood of sustainable WASH service and practices.
This brief report documents the process, the deliverables and the major lessons learned.
Documentation of Process and Main Lessons Learned
Page 5 of 14
Sustainability Monitoring Framework
2. OVERVIEW SMF/SI PROJECT DEVELOPMENT 2012-2014
The proposal included three phases as in the table below (from adjusted proposal May
2012).
Assessment
Phase
1.
2.
3.
4.
5.
For Uganda, assess actual WASH
Sustainability and/or Sector
Monitoring Framework and
practice
Assess capacity and interest of
WASH Alliance partners to work
on MF
Assess interest of national and
local government on MF
Discuss findings in a national
workshop/meeting and
conclude on the ambition level
of the sustainability monitoring
For Uganda, establish a small
Task Force including also
government and non-Alliance
NGOs
Embedding Phase
Development Phase
1. Develop a version-1 of indicators,
process and tools for monitoring
WASH sustainability
2. Discuss version-1 with (i)
Alliance members/partners, and
(ii) government and other
stakeholders
3. Adjust version-1 into version-2
4. Plan pilot data collection and
train field team including local
government
5. Collect monitoring pilot data
6. Analyse monitoring pilot data
2.
3.
4.
5.
6.
1. Discuss findings on results and
pilot process/tools at local
level
Discuss data on results and pilot
process/tools at national level;
in Uganda with Task Force
Adjust version-2 into version-3
Determine the components
(indicators & process) in
version-3 required for
Programme Sustainability
Monitoring
In Uganda: determine the
potential of applicability of
indicators, process and tools for
local and national Sector
Sustainability Monitoring
In Uganda: determine the lobby
& advocacy strategy, and scaling
up process at national level
The project development went through three official phases (as in diagram above) plus an
Inception Phase at the start and a Follow-up Phase:
1. Aug/Nov 2011: IRC developed Outline Proposal and budget versions that were
discussed at various levels within DWA (organisation level, PME group level,
Programme Group level).
December 2011, SMF/SI contract signed.
2. January 2012: PME Group discussed proposal and proposed changes that led to
separate proposals for Uganda and Ghana, later merged into one proposal with
specifics for Uganda and Ghana as testing countries. In the new proposal, the scaling
up to the six other DWA countries was removed and an outline proposal for scaling
up included; for this reason the budget was also reduced.
June 2012 the new proposal was agreed and the project could start. The period
January to May 2012 can be seen as an Inception Phase with substantial inputs from
the PME/Guidance Group.
3. June 2012 the Phase-1 (Assessment Phase) started in Uganda and Ghana using IRC
staff and local consultants. In August the reports were produced and in September a
National Roundtable was organised in Uganda. In Ghana, for reasons of absence of
a national monitoring system, the focus remained at regional level that is Tamale.
4. Phase-2: At IRC the concept of the Sustainable Monitoring Framework/
Sustainability Index was developed, discussed with Guidance Group and also
presented at Country Coordinators meeting of October 2012. Between October and
December 2012, the SMF/SI was further developed incorporating comments. In
January 2013 the SMF/SI Guide v1.0 and three sets of Questionnaires V1.0 for water,
sanitation and hygiene were ready for presentation with Guidance Group. DWA
members gave comments and inputs in questionnaires. In February 2013, the
testing in Uganda and Ghana started with information sessions and training
workshops. The actual field-testing started in Ghana in April 2013 and in Uganda in
Documentation of Process and Main Lessons Learned
Page 6 of 14
Sustainability Monitoring Framework
August 2013 (start in Uganda was delayed with some four months because of
financing issues of field testing). SMF/SI was presented at a webinar of Triple-S and
in the Monitoring Conference in Addis Ababa in April 2013. In August the version
0.1 of the report on the field-testing in Ghana was produced; the pre-final draft was
shared with DWA in December 2013. The data analysis appeared more difficult
than expected; local organisations failed to make on their own sense of the survey
findings. The Uganda workshop on sense making of the findings and results was
held in December 2013, this late timing was due to availability of local DWA
partners in Uganda and IRC staff. In December 2013, the draft versions of the Ghana
and Uganda Field-testing reports were shared. In January 2014 the final drafts were
shared. The country reports give also some rationale for applying the SMF/SI at
project and local level.
Due to the encountered complexity of data collection, processing and analysis,
solutions were sought and found. Data collection was found to be easier and more
consistent if done using FLOW on smartphones, as has been done by two partners in
Uganda already with assistance from the ad-interim Country Coordinator. For the
more difficult encountered problem of data processing and analysis, Akvo could not
provide solutions and therefore IRC contacted BeDataDriven, an organisation
specialised in data management and analysis. There is no final agreement with
them on the required process and implications as yet. A proposal for developing an
on-line data management system has been developed and submitted to DWA for
further discussion. This data processing/ management/analysis problem and the
need to involve an external organisation delayed the finalisation of the SMF/SI
Guideline. The Questionnaires are now in a version-4; and these remain dynamic
for the local conditions though also fixed in the issues of sustainability
requirements. Good comments were received from WASTE on the sanitation set of
questionnaires.
5. Phase-3: the embedding of the SMF/SI took place at various levels: via the Guidance
Group, the Programme Group+ meetings, the Country Coordinators meetings in
2012 and 2013, the National Roundtable in Uganda, the workshop on sharing of
research projects results in Ghana, the webinar with Triple-S, the Monitoring
Conference in Ethiopia and the AquaConsult mapping of sustainability tools.
However, the field-testing showed the complexity of the tool in the data collection,
processing and analysis part. Therefore, it was found too early to present the
SMF/SI tool at national level in Uganda and Ghana for country sector monitoring
sustainability. The Ghana Country Coordinator and IRC’s local consultant from UDS
will - in collaboration with IRC Ghana - look for opportunities to share the principles
and promise of the SMF/SI instrument at national level. Nevertheless, there
remained great interest in the DWA Guidance Group and the Programme Group to
seriously considering continuing the final developments of the SMF/SI using
BeDataDriven solutions. The costing of this online processing is still due from
BeDataDriven. DWA has to decide if to continue with this online data management
system.
6. In November 2013, the draft outline for scaling up the SMF/SI was discussed in the
Guidance Group. It was decided that – because of the great potential of the SMF/SI
tool – the training on the tool should be a MOOC-type online training. Exploration of
MOOC training development, testing and costing took longer than expected. In
January 2014, IRC presented a final Outline Proposal for Scaling up including a Gantt
Chart and an indicative budget.
Documentation of Process and Main Lessons Learned
Page 7 of 14
Sustainability Monitoring Framework
3. THE PROJECT DELIVERABLES
There are seven major project deliverables:
1. The Uganda Assessment Report (Phase-1)
2. The Ghana Assessment Report (Phase-1)
3. The Uganda SMF/SI Field-testing Report (Phase-2)
4. The Ghana SMF/SI Field-testing Report (Phase-2)
5. The SMF/SI Guide with three sets of sustainability requirements and questions
6. Outline proposal for scaling up SMF/SI
7. Several power point presentations
4. THE DWA GUIDANCE GROUP AND IRC TEAM
IRC was guided by the SMF Guidance Group, a sub-group within the DWA PME Group. The
compositions of the Guidance Group and the IRC team are given in annex-2. The Guidance
Group met 13 times in a two-year period, see annex-3.
The Guidance Group has been a sounding board and steering entity for IRC. They formed
also a bridge between the DWA Programme Group, the PME group and the IRC team. They
brought in the DWA expectations, the DWA context and gave feedback on drafts. Maaike
Kempkes participated in a SMF/SI training workshop in Uganda, and Verele de Vreede in a
SMF/SI training workshop in Ghana.
The Country Coordinator for Ghana and the interim Country Coordinator for Uganda gave
good support and local guidance during the training and testing.
5. MAJOR LESSONS LEARNED
There are several major observations and lessons learned.
Major observations:
 SMF/SI development came at the time that various international organisations were
developing concepts and instruments to measure sustainability, including DGIS
(Sustainability Clause), UNICEF, USAID/IRF, Triple-S
 The International Conference on Monitoring Sustainable WASH Service Delivery
provided a good platform to present SMF/SI concept, to get feedback
 DWA members in the Netherlands were keen to learn about sustainability in WASH.
 DWA local partners in Uganda and Ghana used to focus their monitoring on outputs.
 DWA local partners in Uganda and Ghana welcomed the SMF as it unpacked the
complexity of WASH sustainability
 The Country Coordinator (CC) in Ghana provided good support and guidance, the CC
in Uganda not at all. Only during an interim period, the interim CC did provide
support.
 Interestingly, the views on applicability of the SMF/SI of the Ghana and (interim)
Uganda CCs were quite different. The CC of Ghana found the SMF/ SI tool and all the
related exercises very important for the Ghana programmes. According to him, the
SMF/SI added value to the FIETS principles by creating a concrete assessment/
measuring base. A positive issue was the involvement of society, the people in the
surveys. According to the CC for Ghana, programme officers would be able to do it
with some extra training. The interim CC for Uganda did not find the application of
the SMF/SI a feasible move: the training requirements are too high, the use of the
results is not clear, except if used in the design phase of a WASH project.
 GWA partners involved in the field-testing appreciated the option the SI gave to
compare locations and to compare achievements towards FIETS principles.
Documentation of Process and Main Lessons Learned
Page 8 of 14
Sustainability Monitoring Framework







UWA partners involved in the field-testing found the tool useful to measure
sustainability where they monitored before only outputs, and it helped them to
identify actions to increase sustainability.
The initiative was not scheduled in the DWA country programmes, that is, no time
nor funding had been allocated for this activity (e.g. training, field-testing, analysis,
reporting) to local partners. They had to incorporate that in their on-going activity
schedules, which created availability problems. A serious obstacle was that local
partners had no allocation for travel and allowances, and therefore could not start
the field-testing. This was particularly a problem in Uganda. The fact that local
partners had different DWA members, as funders required that Dutch DWA
members had to agree on fund allocation for travel and allowance. This fund
allocation took quite some time.
SMF/SI results in a score on the likelihood of sustainability. Such a single score has
value on progress. The SMF allows also for listing the positive achievements
towards sustainability and the present blockages for follow-up and agreed actions.
SMF/SI allow for aggregation of scheme or village SI results to overall SI results per
project or partner, or per district or sub-district. Of course, disaggregated reporting
by scheme type adds more value to such reporting
During and after the Country Coordinators meetings in The Netherlands (2012 and
2013) several DWA countries (through their CCs) expressed keen interest to learn
more about the SMF/SI. For instance, in Kenya, DWA wanted to apply it.
The Programme Group saw a good potential for mainstreaming SMF/SI in the entire
DWA programme
As there may be a wide interest in SMF/SI, the proposal to have an on-line training
(MOOC) in the public domain is an excellent idea to use public funds for world-wide
sharing.
Major lessons learned:
 Because DWA members in the Netherlands and DWA local partners in Uganda and
Ghana were keen to learn about sustainability in WASH, and the SMF/SI helped
them to unpack sustainability and monitor beyond project outputs, the SMF/SI was
very welcomed
 A pro-active Country Coordinator as in Ghana is conducive to a good collaboration
of local partners and follow-up
 Starting new initiatives without funding gives a limited commitment for
engagement of local partners
 Field-testing the SMF/SI showed the relative complexity of the proposed data
processing and management structures
 FLOW installed on smartphones as done in Uganda, made the data collection more
efficient and may have reduced the errors
 Time required for analysis of surveys is (as usual) largely under-estimated, also by
IRC (particularly as local partners and consultants could not deliver as expected)
 Drawing conclusions from processed data (findings) appeared quite hard for both
DWA local partners as for IRC’s local consultants.
 Replacing a spreadsheet-based data analysis by a online-based data management
system is not an easy step
 Giving space to local partners to be flexible on the sustainability questions, meaning
allowing them to select and/or reformulate the questions appeared not a good idea.
Their result was a significant reduction in questions and sometimes ambiguous
formulation of the questions, leading to unreliable and less useful Sis
 In Uganda, UWA partners indicated that the benchmarks need national attention
and should be strict. They also indicated that the number of questions should be
increased rather than be limited to enable good analysis of the situation. And they
learned that hygiene needed questions on menstrual hygiene management.
Documentation of Process and Main Lessons Learned
Page 9 of 14
Sustainability Monitoring Framework

Implementation does not always follow planning. The envisaged result and
application of SMF/S at both local and national government may still be a potential
but the present outcomes are not sufficient to share the SMF/SI at those levels.
However, introduction of the concept and the promise at both levels is very
opportune as government and development partners look for good tools to measure
the likelihood of sustainability next to performance monitoring.
Documentation of Process and Main Lessons Learned
Page 10 of 14
Sustainability Monitoring Framework
Annex 1
Overview of deliverables
Main deliverables by Project Phase
Phase 1: Assessment phase
a.
b.
c.
A Position paper on present Sector
Monitoring Framework, the Task Force, its
TOR and composition, and the roles of sector
stakeholders, in particular the in-country
WASH Alliance members & partners
Report of the national workshop
Detailed work plan for the development
phase
Overview of delivered products as per 31 Jan
2014
a.
b.
c.
1. Uganda Phase-1 report (Aug-12)
2. Ghana Phase-1 report (Aug-12)
National Round Table Report/minutes (Oct-12)
(not for Ghana)
1. Summary Report (2-p) and planning Phase-2
(Dec-12)
2. Uganda Detailed plan (Feb-13)
3. Minutes Guidance Group and PG+ meeting
(Dec-12)
Phase 2: Development phase
a.
b.
c.
d.
e.
f.
Monitoring Framework Version-1
Report with findings and conclusions from
consultations on Version-1
Monitoring Framework Version-2
Training programme and materials for field
team
Paper on Framework with analysed data and
conclusions on its potential
Detailed work plan for the embedding phase
a.
b.
c.
d.
e.
f.
1. Concept v0.1 PPTX (Oct-12)
2. Guideline v1.0 and Questionnaires W-S-H
v1.0 (Jan-13)
3. SMF/SI Presentation PPTX (Jan-13)
4. SMF/SI presentation (PPTX)- for Webinar
(Feb-13)
5. SMF/SI presentation (PPTX)- for CC-meeting
(Sep-13)
1. Minutes Dec-12 and Jan-13 Guidance Group
meetings
2. Comments on Questionnaires from WASTE,
RAIN, Wetlands International (Feb-12)
1. SMF/SI Questionnaires v2.0 (Mar-13)
2. SMF/SI presentation (pptx) for Monitoring
Conference (Apr-13)
1. Notes on Planning Meeting Uganda and PPTX
(Feb-13)
2. Report Facilitation workshop Ghana-Tamale
(Feb-13)
3. Uganda Training-1workshop Notes and
evaluation (Mar-13)
4. Uganda Training-2 workshop notes (Apr-13)
5. Uganda Refresher workshop notes (Jul-13)
6. Ghana workshop-1 Notes and summary
(May-13)
7. Ghana workshop-2 Notes and summary
(May-13)
1. Ghana Pilot testing report draft (Oct-13)
2. Ghana Pilot testing final draft-1 (Dec-13)
3. Uganda Pilot testing report final draft v4.2
(Dec-12)
1. Interim Report to Guidance Group (Jun-13)
2. Interim Report to Guidance Group (Sept-13)
Phase 3: Embedding phase
a.
b.
c.
Report of the various consultations and
national Task Force meeting
Monitoring Framework Version-3, a users
guideline and a Powerpoint for general
presentations of the Monitoring Framework
WASH Alliance Country Programme
Sustainability Monitoring Framework
Documentation of Process and Main Lessons Learned
a.
b.
1. Uganda National Round Table
Report/minutes (Oct-12) (not for Ghana); in
Uganda it was decided not to form a Task Force
– see roundtable report
2. Ghana, SMF/SI dissemination workshop in
Tamale 20 Dec 2013
Guideline v1.0 and Questionnaires W-S-H v1.0
Page 11 of 14
Sustainability Monitoring Framework
d.
e.
Paper on the potential of Framework or
components for Sector Sustainability
Monitoring
Documentation of Process and lessons
learned
c.
d.
e.
(Jan-13) and SMF/SI Presentation PPTX (Sep13)
This is the same as deliverable c. – the SMF/SI
Guide deals with this application at local
government, local (DWA) project and service
delivery level.
It was decided not to share SMF/SI at national
level for Sector Sustainability Monitoring
Documentation of Process and lessons learned
(Jan-14)
Extra deliverables


An outline for rolling-out this sustainability
monitoring to the other six WASH Alliance
countries
A paper with major lessons from this project
for internal learning



Documentation of Process and Main Lessons Learned
1. Outline for scaling up SMF/SI- v1.0 (Nov-13)
2. Outline for scaling up SMF/SI- v2.0 (Dec-13)
3. Outline for scaling up SMF/SI- v2.1 including
an online training using MOOC (Jan-14)
Proposal on development of a data
Management system, linking to FLOW data
collection with BeDataDriven
Part of Documentation of Process and lessons
learned (Jan-14)
Page 12 of 14
Sustainability Monitoring Framework
Annex 2
Guidance Group and IRC team compositions
DWA Guidance Group
Name
Maaike Kempkes
Martijn Marijnis
Annemieke Beekmans
Elbrich Spijksma
Stan Maessen
Verele de Vreede
Organisation
DWA
ICCO
AidEnvironment
Simavi
WASTE
WASTE
Suzanne van Rooijen
Maarten Mulder
DWA
AMREF
Function
Chair
Member
Member
Member
Member
Member
Chair
Member
Member
From /to
Feb 12-Jul13
Feb 12-Feb14
Feb12-Feb14
Feb12-Feb14
Jul12-Nov12
Dec12-Jul13
Aug13-Feb14
Sept13-Feb14
Dec13-Feb13
IRC Team
Name
Jo Smet
Kristof Bostoen
Ruzica Jacimovic
René van Lieshout
Valérie Bey
Patrick Jangeyanga
Kwame Asubonteng
Gordana KranjacBerisavljevic
Cristina Martinez
Organisation
IRC
IRC
IRC
IRC
IRC
Independent
IRC Ghana
UDS Tamale
consultant
IRC
Documentation of Process and Main Lessons Learned
Function
Manager
Team member
Database expert
Team member Uganda
Team member Uganda
Local consultant UG
Team member Ghana
Local consultant GH
From /to
Manager
Team member
Database expert
Team member Uganda
Team member Uganda
Local consultant UG
Team member Ghana
Local consultant GH
Project Assistant
Project Assistant
Page 13 of 14
Sustainability Monitoring Framework
Annex 3
Overview of Guidance Group meetings
Overview of Guidance Group meetings in period 2012-2014:
During 2011, IRC was member of the PME group and participated in the PME Group
meetings. In January 2012, it was agreed that IRC would not remain a member of the PME
group. To steer the SMF/SI project, the Guidance Group was established chaired by the
DWA Monitoring Specialist
During 2012, 2013 and 2014 many meetings took place between the Guidance Group and
IRC team on SMF/SI, below an overview. In this period, 13 Guidance Group meetings were
held, two inputs given to the Country Coordinators meetings and two to the Programme
Group Plus meetings.
• 06 February 2012
• 07 March 2012
• 05 June 2012
• 06 July 2012
• 30 August 2012
• 02 October 2012 with the SMF/SI concept and presentation (PPTX)
• 04 October 2012, SMF/SI presentation to Country Coordinators meeting with PPTX
• 19 October 2012
• 06 December 2012
• 14 December 2012, presentation to Programme Group + meeting with PPTX
• 24 January2013
• 06 April 2013, presentation SMF/SI at International Monitoring Conference in Addis
Ababa (with PPTX)
• 12 June 2013
• 21 June 2013, presentation to Programme Group + meeting with PPTX
• 03 Oct 2013, presentation to Country Coordinators meeting with PPTX (Verele)
• 19 October 2013
• 19 November 2013
• 17 December 2013
• .. February 2014
Documentation of Process and Main Lessons Learned
Page 14 of 14

Download Report