Here - Reality Check Approach

‘in touch and up-to-date with the realities of people living in poverty’
A Short Introduction to RCA
2014 Edition
WWW.REALITY-CHECK-APPROACH.COM i
WWW.EDGROUP.COM.AU
1
2
TABLE OF CONTENTS
1.
1.1.
1.2.
1.3.
1.4.
1.5.
2.
2.1.
2.2.
2.3.
3.
OVERVIEW OF THE REALITY CHECK APPROACH ................................................................... 4
CONTEXT ..................................................................................................................................... 4
WHAT IS RCA? ............................................................................................................................... 4
SITUATING THE RCA WITH OTHER METHODS .................................................................................... 5
INTENDED OUTCOMES ..................................................................................................................... 6
LIMITATIONS ................................................................................................................................... 7
CONTEMPORARY RELEVANCE ................................................................................................... 8
AID EFFECTIVENESS ........................................................................................................................ 8
VALUE FOR MONEY......................................................................................................................... 9
OWNERSHI;WHOSE VOICES COUNT? ............................................................................................... 10
RCA STUDY DESIGN & IMPLEMENTATION .............................................................................. 11
3.1.
3.2.
3.3.
3.4.
CONVERSATIONS........................................................................................................................... 11
AREAS OF ENQUIRY ...................................................................................................................... 12
THETEAM ..................................................................................................................................... 12
RCA M AIN STUDY......................................................................................................................... 13
4..
NOTES FOR RCA STUDY COMMISSIONERS........................................................................... 17
5.
FROM RESEARCH TO ACTION.................................................................................................. 17
6.
THE RCA COMMUNITY ......................................................................................... 18
7.
FUTURE USE .............................................................................................................................. 19
Acknowledgments
Special acknowledgment is made of Helena Thorfinn, Esse Nilsson and Dee Jupp who were the
inspiration and driving force for the first RCA in Bangladesh in 2007.
Most importantly we acknowledge the many host households for welcoming RCA researchers into
their homes and to their neighbours and local service providers for embracing this approach and
teaching us so much.
Photos on the front cover depict RCA researchers in
Indonesia, Mozambique and Nepal informally
interacting with members of their host households.
3
1. OVERVIEW OF THE REALITY CHECK APPROACH
1.1.
CONTEXT
‘Would the poor express amazement that people who are experts in poverty don't even bother to spend
time with them.’ - Koy Thompson, Director, Action Aid
The context, realities and aspirations of people living in
poverty are changing at accelerating rates. Development
interventions need to keep pace and provide the right
response to people’s emerging needs and ambitions.
Simultaneously, donors are demanding increasingly
responsive and effective development programme and
service provision. The challenge has never been greater for
development professionals and development interventions to
keep up to date and in touch with increasing complexity.
Acknowledging the current demand for evidence based
research (usually referring to quantitative or quasiexperimental design) to shape policy and practice, the Reality
Check Approach (RCA) offers a powerful complement to
these approaches by providing a different dimension of
evidence. As an approach that is open to multiple and
unexpected realities, RCA can ‘flag up’ new areas for
quantitative research, offset and reduce its reductionism,
provide detailed understanding of behaviours and insights
into whether and why development interventions are taken up
or not. Policy-makers can be informed of rapidly changing
realities of which they may not be aware and can be helped
to understand how context and behaviour affects the success
of programmes.
RCA IN A
NUTSHELL
RCA is a qualitative approach to
feedback and evaluation which
involves outsiders living with people
living in poverty in their own homes
and joining in their every day lives.
The relaxed environment this
provides enables easy informal
conversations with all members of
the family , their neighbours and
others who interact with the
household. It allows the outsider to
experience and observe the realities
of the family. This provides a
meaningful basis for joint reflection
on change with the family. It helps
to shed light on the disconnects
between knowledge, attitudes and
practice which conventional
evaluation is often unable to do
Since first being piloted in Bangladesh in 2007 the RCA has
been used in Indonesia, Bangladesh, Nepal, Ghana,
Nicaragua, Pakistan and Mozambique to provide insights for
Government and NGO development programmes supported by Australian Aid, Sida, the EC and DFID.
These insights have been used at national development, programme and project levels.
This short introductory guide provides an overview of critical components and thinking surrounding
RCA for researchers or research commissioners considering including RCA in their suite of research
methods. Practitioners are referred to the more detailed Rough Guide to RCA to be published in 2014.
1.2.
WHAT IS RCA?
The RCA is an experimental qualitative approach to collecting and using information from the ground.
The core of RCA is immersions during which researchers stay several days and nights with people
living in poverty to engage with, listen to, observe and document their voices, opinions and experiences.
Like other qualitative approaches, this process focuses on asking ‘how’ and ‘why’ rather than ‘what’,
‘when’ and ‘how many’. The main idea is to have sustained, detailed conversations and intense
interactions with a small number of households in their own homes. Sharing in their lives provides
opportunities to better understand and contextualise people’s opinions, experiences and perspectives.
4
The RCA is generally intended to track changes in how
people live and experience their lives and involves
repeating the RCA with the same households at
approximately the same time each year over a period of
several years. Longitudinal studies have the advantage
of engaging regularly with households and communities
enabling the researcher to build trust and increased
openness in interactions over time which other methods
may lack.
Chatting with Secondary School students on a
beach in Indonesia- 2010
1.3.
A one-time RCA can also be used as a ‘pulse taker’ (e.g.
finding out how new programmes are being received,
updating on contextual change) or as a retrospective
study on how people experienced and assessed change
over the past.
SITUATING THE RCA WITH OTHER METHODS
RCA is a hybrid approach that draws on a variety of traditions and qualitative methods, including;
ethnographic studies, people centred research, participatory learning and action, appreciative enquiry,
listening studies, observation, story-telling and participatory visual documentation and evaluation.
Perhaps the most influential method behind the RCA is participant observation and RCA has been
described as a ‘light touch participant observation’ (though not everyone likes this description).
Participant observation involves both participating in and observing people’s everyday activities. This
process usually entails extensive and detailed research into behaviour, along with exploration of
peoples’ perceptions and actions over long periods of time. The RCA is similar in that it requires
researchers to participate in everyday life within people's own environment but differs by being
comparatively quick and placing more emphasis on informal, relaxed and insightful conversations rather
than observing behaviour and the complexities of relationships.
Although RCAs may be stand alone studies, increasingly they
are being used in mixed method impact evaluations as a
means to provide a rich contextual basis for quantitative data
collection. Sequenced carefully, it can help develop better
household survey instruments, provide insights for survey
enumerators on appropriate engagement, provide early
indication of mis-understood or mis-communicated survey
questions, assist in interpretation of data, provide insights into
outliers and their significance, question generalisation and
over-simplification.
MIXED METHODS?
‘Qualitative research without
quantitative can be insightful
but vague…….
…..quantitative without
Further, there are some issues which do not lend themselves
qualitative can be precise but
to surveys or other forms of quantitative data collection e.g.
may be wrong headed,
sexual behaviour, domestic violence, intra-household decision
misdirected and contrived’
making and these are better explored in non-threatening,
informal situations of trust which RCA is especially appropriate
Kalinowski, Lai, Fidler and Cumming
for. The longitudinal nature of RCAs and the nuanced data
they collect lends RCAs particularly well to exploring change
processes. There are also small or marginalised groups which
may be left out of statistically driven surveys but whose experience and understanding of programmes
are important. Here again small scale purposive RCAs can fill an important void.
5
1.4.
INTENDED OUTCOMES
A key intention of RCA is to understand better whether and how development policies translate into
effective programmes and change for people living in poverty and how these efforts and changes are
perceived. The RCA provides an opportunity to get closer to these realities and to link them with policy
and practice within the context, limitations and resource constraints of typical development aid
programmes. It provides valid, up to date, people-centred information and is particularly valuable as a
means to provide explanations of quantitative data by giving time for people to examine and explain
why they make choices, why they take action and what prevents them from exercising choice and
agency.
RCA studies to date have;

Provided immediate insights into new programme activities, e.g. introduction of the new primary
school examination in Bangladesh, distribution of mosquito nets in Ghana

Flagged up issues for further study e.g. increasing consumption of salt in Bangladesh,
increasing drop out of boys from school in Bangladesh and Indonesia

Identified implementation problems in ongoing programmes e.g. manipulation of national
school exam conditions in Indonesia, miss-match between equipment and staffing in health
facilities in Bangladesh, leakage of free medicines to the non-ill in Nepal and Bangladesh, real
costs of education for families in Indonesia (including assertive demands for pocket money)

Identified un-met needs e.g. agricultural extension advice for cardamom farmers in Nepal,
family planning advice for men and unmarried persons in Bangladesh, safe avenues to
complain about government services in Ghana, medicine provision for people living in poverty
in Bangladesh for non-communicable diseases e.g. high blood pressure, diabetes, stress.

Provided a wider lens for evaluation by embracing negative, unintended consequences of
interventions/change and multiple realities e.g. exposed the burden of livestock care for asset
transfer recipients in Rwanda, increase in snack food and alcohol consumption resulting from
access programmes in Nepal, negative outcomes from increased mobile phone use in
Bangladesh.

Identified emerging social issues e.g. chronic indebtedness in Indonesia, increasing alcohol
abuse in Nepal, abandonment of elderly in Bangladesh and Nepal, missing middle generations
in Mozambique, Ghana, Nepal and Indonesia, the need for re-evaluation of gender and ethnic–
based quotas in Nepal

Explored the disconnect between reported data and reality e.g. exaggerated numbers of
attended births in Bangladesh, over-reported school attendance in Indonesia and Bangladesh,
under-reported teacher absenteeism in Nepal, Bangladesh and Indonesia

Provided more nuanced understanding of statistical data e.g. increasing drop out of boys from
school in Bangladesh and Indonesia due to constellation of factors including lack of incentives,
privileging girls, punishment skewed towards boys, too few male teachers, pull factor of
unskilled jobs and the desire to earn for their own recreation and status (e.g. phones, clothes,
motorbikes)
6

Helped design culturally appropriate household survey questionnaires e.g. in Nepal and Ghana
which recognise local terms and customs, provide wider and more appropriate choices of coded
answers

Provided the basis for developing people’s Theories of Change to compare with theoretical
programme Theories of Change e.g. Nepal Koshi Hills and Nepal Rural Access Programme.

Put development interventions in context. Because RCAs take a cross-sectoral stance, some
interventions are seen as unimportant or too small to have any significance for people. e.g.
chicken immunization in Mozambique was not a priority for people who very rarely ate chicken,
stipends for primary education in Bangladesh were burdensome to access and too small to
make a difference.

Provided frontline service providers with chances to share their frustrations e.g dispensary staff
in government health facilities in Bangladesh and Nepal, over-burdened midwife services in
Malawi, teachers’ time spent fulfilling many non-education related tasks in Bangladesh,
Indonesia and Mozambique

Explored difficult or sensitive subjects e,g, ethnicity, domestic violence, political empowerment,
crime, sexuality and interacted with small or marginalised groups which may be missed
through randomised surveys.

Allowed people in positions of authority to freely discuss sensitive topics, e.g. in Indonesia
people in authority tend to say the reason for children not going to school is lack of money while
when approached more informally they opened up about other factors such as pre-marital
pregnancy and truancy.
Ultimately, RCA studies are only useful if they are taken seriously by policy makers. The revelation of
unexpected and surprising realities often has a powerful effect on policy makers. Providing un-filtered
voices and experiences of people living in poverty helps policy makers put real faces to the anonymous
numbers often provided by other forms of research. While the dominant rubric of ‘analysis-thinkchange’ pervades, there is value in complementing this with findings which promote ‘see-feel-change’
(Kotter and Cohen, 2002). Evidence suggests that when complementary immersion programmes are
offered in parallel to RCA studies for policy makers, the understanding and response to the RCA study
increases.
1.5.
LIMITATIONS
RCA studies are generally small scale. They deliberately set out to explore the range of experiences
of people living in poverty, so are not intended to be generalizable. RCA studies provide depth rather
than breadth. As such they cannot replace other research, monitoring and evaluation methods but can
effectively complement them.
As the RCA purposely seeks out multiple realities and listen to people’s own voice, it does not lend itself
to providing policy recommendations. Rather, RCA researchers prefer to present policy makers with
implications collated from the RCA study findings. By doing this, they avoid the problem of outsider
interpretation, infiltration of authorial voice and confirmation bias. However, policy makers often feel
challenged by being asked to reflect on study implications rather than a set of normative
recommendations.
7
2. CONTEMPORARY RELEVANCE
As the 2015 MDG deadline approaches, donors and recipient Governments are increasingly expecting
the development community to provide rigorous and scalable approaches to evaluation to ‘prove’
effectiveness of interventions. Much emphasis has been given to evidence-based programming with a
concomitant swing towards commissioning experimental and quasi experimental evaluation design.
While helpful in determining what works, these approaches rarely address issues of why and what else
might be just as effective (or more effective). Experimental/quasi experimental evaluation are
necessarily large scale (to meet statistical significance criteria), expensive and generally slow to yield
results. By contrast, RCAs can provide alternative, complementary, relatively quick and low cost
research-based evaluation directly with the people most affected by development interventions. The
RCA can provide windows on the aspirations of ordinary people and can provide space for comment
on experienced change.
2.1.
AID EFFECTIVENESS
Since the 2005 Paris Declaration on Aid Effectiveness, attention
has been given by donors to improving the coordination,
effectiveness as well as developing country ownership of
international aid and poverty reduction efforts. One unintended
effect has been that emphasis has moved ‘upstream’ with a
concern for efficiencies and better aid management, The so-called
‘more with less’ agenda pervading aid delivery has led to widening
gaps between people on the ground and decision makers. The
relevance and responsiveness of programmes is consequently at
threat. The RCA provides a way to ‘re-connect’ the worlds of policy
makers and programme management with the realities of people
living in poverty. Understanding how programmes actually operate
and are experienced on the ground is essential to developing
operational know how, improving programmes and maintaining
creativity and innovation.
IMMERSIONS ON THE
SIDE
If commissioners of RCA research
can themselves give time to even
one immersion, their connection to
the reality of people living in
poverty
will
be
of
the
immensely
enhanced.
Members
health
and
education Sector Wide Approach
consortia in Bangladesh were
provided with opportunities for
In Provocations for Development (2012), Robert Chambers notes
'agreement seems universal that immersions give insights and
experiences that are not otherwise accessible...’ the world can be
seen from the other way round, from the perspective of people living
in poverty'.
Immersions serve to re-connect development
professionals with people. But these are rarely undertaken .It
seems that finding the time, commitment and courage to spend time
doing an immersion is limited. Where policy makers and
development professionals are unable to undertake immersions
themselves, RCAs can provide an alternative ‘re-connection by
proxy’.
8
facilitated two night immersions
with families living in poverty in
year 3 of the RCA longitudinal
study. This led immersees to
‘check’ some of the findings for
themselves. The resonance of
their own, albeit brief, experience
led them to trust the findings and
promote the findings among their
colleagues. In effect they became
champions for the RCA.
2.2.
VALUE FOR MONEY
Alongside the push for increased aid
effectiveness there has been an increased
emphasis from many donors on ‘Value for
Money’ (VfM); reflecting tax payers/voter
concerns in donor countries on the scale and use
of development aid budgets. RCA can contribute
to VfM in two key ways; RCAs are cost efficient
means to ‘find out’ and findings can lead to
design of more effective and better targeted
programmes.
RCA CONTRIBUTION TO VFM IN NEPAL
An example of how the RCA can contribute to better
VfM in programmes comes from the Nepal Koshi
Hills RCA. The RCA was a single discrete one time
study looking back at 40 years of development
assistance. It revealed that people’s perceptions of
what was significant as well as their expressions of
aspirational change were not necessarily ‘in synch’
with the priorities of development donors.
Furthermore, the RCA revealed that the heavy
sectoral focus of most development programmes
kept them from identifying significant communitylevel changes.
The original Bangladesh RCA was relatively
costly because it was pioneering the approach.
It involved development of a careful iterative
approach, combined with the testing and
promotional exercises necessary for a new
programme. More recent RCA budgets have
been much lower, with a single RCA study1 ,
including training of researchers and piloting the
approach for the country context as well as
leadership and quality assurance provided by an
international team leader, costing in the region of
US$35,000. This figure compares favourably to
Annual Reviews, conducted for donors such as
DFID which generally cost more than twice this.
For example, the RCA demonstrated that production
of cardomom (a culinery and medicinal spice) had
become very lucrative. Villages designated as
extremely poor were in fact found to have thriving
cardamom industries and families with significant
disposable incomes. Neither the national statistical
data nor development programmes had up to date
information on this transformation. Yet its
introduction and spread, more than 10 years ago,
was driven by people in communities themselves.
Development agencies were slow to pick this up and
have only relatively recently offered the needed
support.
In terms of value of outcomes, the RCA
unequivocally provides insights into what people
living in poverty need and value. Understanding
what change people hope for means better
understanding,
leveraging,
tailoring
and
prioritising complex and high cost programmes
and investments, leading to better VfM within
these programmes.
Similarly the rapid increase in overseas migration by
people from the area was spearheaded by people
themselves by making and exploiting their own
networks. The official development assistance for
this has lagged behind.
In both cases the lag has been between 7-10 years.
Reducing this lag and making these local initiatives
work better represents a major VfM saving.
Cardamom growing in Nepal
1
A discrete and complete one time study rather than a longitudinal study
9
2.3.
OWNERSHIP: WHOSE VOICES COUNT?
There is a growing concern among development aid stakeholders that ownership, identified as a key
element for aid effectiveness by the 2005 Paris Declaration on Aid Effectiveness, is too often confused
with recipient Government ownership. Since the Accra and Busan conferences, civil society
organisations have been increasingly included in evaluations and research, but those in villages, towns
and cities who are the intended target group of the development interventions are often still forgotten.
The assumption that recipient governments and civil society institutions can fully represent the views
and needs of local communities is recognised by many to be flawed 2.
Studies indicate that people, particularly those living in poverty, rarely feel able to make their voices
heard. There are few opportunities for people to voice demands and concerns and little time/inclination,
access or capability to participate in conventional formal mechanisms and platforms where voices may
be raised. Many ordinary people eschew public forums for raising issues fearing that they may not be
heard, may be ridiculed or singled out. These public spaces are often dominated by political interests
and the prevailing power dynamics may preclude open discussion on issues. Often people simply do
not have the time to engage in this kind of policy dialogue platforms or have little optimism about the
outcomes. Even though many development interventions have been aimed at creating additional
'invited spaces' for civil society participation these may be manipulated by those inviting (often national
and local governments) and/or co-opted by powerful interest groups. The lack of transparency around
who is invited and who is not as well as the politicisation of public space furthers fuels frustration and
reluctance to use these spaces by ordinary citizens.
The RCA can provide an alternative which can amplify ordinary voices. It provides an easy trusted
space for emergence of people’s real perspectives and opinions rather than ones shaped by desire to
please, conceal or manipulate for gain which often manifest themselves in public shared spaces. It
enables detailed triangulation to gauge the veracity of what people say. For example, the Ghana RCA
revealed parents were very upset about teacher absenteeism but feared losing the few teachers they
had if they complained. Patients in Bangladesh laughed at the notices in hospitals suggesting that they
lodge complaints with the hospital administration, ‘Why would we do this and put at risk future
treatment?’.
The RCA provides opportunities for local service providers to also share perspectives while chatting
informally. Teachers in both Bangladesh and Indonesia shared many insights into the manipulation of
public exams during the RCA but would not be prepared to share this in public forums or in meetings
convened by their respective education department. These forums, they explained, were ‘not times for
discussion or listening to our views but means to give us instructions from higher authorities’ Medical
staff in Nepal and Bangladesh shared their frustration with their governments’ free medicines schemes
which put demands on them to dispense medicines to the non-ill, wasting time and resources but ,
again, had not been able to raise this in conventional forums.
To redress this situation, development partners such as DFID, Sida and DFAT have indicated in recent
years that they need to better understand how aid interventions are viewed and experienced by local
populations, to get closer to understanding what works and what does not, and why. RCAs included
routinely as a complement to information gathered through their representative organisations (elected
representatives, membership organisations etc) provides a means to cross check and expand the scope
of opinion gathering.
2
Easterly, W.R. The White Man's Burden: Why the West's efforts to aid the rest have done so much ill
and so little good. New York: Penguin Press, 2006.
10
3. RCA STUDY DESIGN & IMPLEMENTATION
A major challenge for RCAs come inherently with their goal to shed light on and understand multiple
realities. RCA purposively engages with the complexity; diversity of contexts, opinions, experiences
and suggestions. This poses a huge challenge to deliver consistent, comparable and coherent
information to a client. Experience of conducting credible and well accepted RCAs has indicated that
there is a need for careful design to deliver the right amount of flexibility and reflexivity to minimize bias
and maintain objectivity.
3.1.
CONVERSATIONS
Much of the strength of the RCA lies in its flexibility and
responsiveness to context and moments. Conversations are a
relaxed and informal way to interact with people and to use
successfully require an emphasis on the role of attitudes and
behaviour which permeate all human interactions.
These
behaviours include unequal power, biases, assumptions, pride,
status, commitment and interest which shape the ‘rules of
engagement’ and can lead to mis-understanding, distortion,
manipulation and concealment of information on both sides. This
emphasis distinguishes the RCA from other methods of
engagement where power dynamics are difficult to diminish.
Having time to build trust and familiarity has proven again and again
to help people to relax and open up. Staying in people’s own
houses and fitting in with their routine, their ‘rules’ and becoming
‘learners’ contributes greatly to reducing power gaps which
normally exist between people and ‘outsiders’. Information arising
from informal conversations is rarely directly recorded either in
written or in audio-visual form. This further enhances the quality of
the interaction and the trust and confidence in the engagement
process.
IT TAKES TIME
‘One woman in the
household was very
uncomfortable with me.
She barely greeted me
when I arrived and kept
away. But next morning
we worked together in the
field and she became
relaxed. Over the rest of
the time I spent with them,
she gradually opened up
more and more.’
Nepali RCA researcher (male),
2014
Conversations can be curtailed when people lose interest, can be picked up again at a later time , can
be shared and referred to with others at other times and can take place while people are going about
their usual routine. Time and informality confer RCA considerable comparative advantage over many
other research methods.
Conversations are sometimes augmented with shared
drawings, diagrams, photographs or dramatized explanations.
These are not the core of RCA but can sometimes help the flow
of conversation. At times conversations take on a life of their
own and the researcher may become an observer of debate
and animated discussion.
There is nevertheless a need
conversations have a purpose.
Drawings of types of teachers that
children did not like which sparked
animated conversation about changes
resulting from the teacher training
programme- Bangladesh 2009
11
to
ensure
that
most
3.2.
AREAS OF ENQUIRY
RCA’s purpose is achieved through detailed briefing sessions with the RCA team members before going
to the study locations. Usually based on input from commissioners of the research and extensive
literature reviews, the team leader facilitates a review of the thematic basis of the RCA, in particular
identifying contradictory evidence, gaps in evidence and understanding as well as ambiguous findings.
Areas of enquiry are developed sometimes through the generation of a ‘people’s perspective’ theory of
change diagrams. Care is taken not to fuel assumptions or pass on received wisdom as the researchers
operate best when they can have open conversations. Based on these areas of enquiry, loose
checklists are developed by research team members to
provide an ‘aide memoire’ for their
conversations and to help team members probe and clarify issues. Such checklists are not research
questions and are not prescriptive as this inhibits the natural flow of conversations.
Example Checklist themes from Previous RCAs
Indonesia (with focus on the Basic Education)





Value of education: how parents and children perceive and value education?
Quality of education: how parents and children perceive quality?
Access to education: what are the issues behind non-attendance and drop out?
Enabling and hindering factors for education
Parental involvement: in schools and with their children's education generally
Bangladesh (with focus on the Health and Education services)



3.3.
Rights: How do people understand their rights and try to operationalise them in relation to
services?
Knowledge: How well do people understand how public systems work, their entitlements and
who is responsible for what? How do people deal with the informal aspects of the systems?
Choice; what influences choice of service provider ( formal and informal)
THE TEAM
Experience from previous RCAs indicates that the RCA team should comprise of people with high levels
of enthusiasm, appropriate and sensitive attitudes towards people living in poverty and willingness to
suspend judgment and challenge their own biases. They must also be willing to subject themselves to
quite difficult living conditions in the field (sleeping on the floor or sharing with host household family
members, basic bathing and toilet arrangements (if any!), rats, extreme heat/cold, insects, etc).
Students and recent graduates (with anthropology, sociology, development studies, journalism
backgrounds) often make the best team members.
12
TRAINING & PILOTING
Since the skills, attitudes and attributes required differ from other forms of qualitative and action
research, it is essential that the team engages in a short orientation and capacity building process (45 days) which includes pilot testing through a minimum of two day and two night immersion experience
(mini RCA). Even with an experienced RCA team, piloting should be considered when moving into a
new location or new areas of enquiry.
The pilot is intended to have the following objectives:




3.4.
to further develop the RCA for each new context and the needs of the programme
to build capacity of the RCA team to acquire the attitudes, behaviour and skills to be part of the
main RCA Study and begin (or continue) the process of building sustainable capacity to conduct
such studies in the future.
team building
to promote the approach among stakeholders, building their interest and support for the
initiative
MAIN RCA STUDY
The RCA study comprises the following steps
Detailed briefing
, selection of
locations
immersion
detailed debriefings
immediately
after immersion
analysis
&reflexive
processing
report &
communicating
findings
STEP1:BRIEFING
This always involves joint discussion and development of the areas of enquiry (see above),careful
decisions around which locations and households to select for the study and a review of the teams
own biases and assumptions. This latter is intended to make these explicit and take action to
minimize the influence of these in the conduct of the study.
The selection of locations for the RCA is challenging but RCA teams need to resist the pressure which
might exist to 'sample'. As the RCA intentionally seeks to understand multiple realities and does not
claim to provide representative findings, conventional sampling is inappropriate. Nevertheless in most
RCAs conducted to date, criteria are developed for some kind of purposive selection of study locations.
In mixed methods approaches where RCTs have been used, the RCA has also chosen to study
13
‘treatment’ and ‘control’ areas but has taken additional criteria into consideration too. There can also be
an argument for 'no criteria' so that reality is not distorted. The aim to reveal multiple realities can be
used to determine the location selection criteria which therefore embrace diversity (livelihood, ethnicity,
remoteness/access etc).
Flow diagram to show process of selecting locations
external criteria e.g.
areas where
programme
does/does not
operate
negotiated variables
e.g distance from
urban centres,
ethinicity,'control' &
'treatment'
RCA team makes
own reconnaisance
and selection of
locations
STEP2:IMMERSION
Immersion is core to the RCA and changes the dynamic between the researcher and the host families.
It involves not just staying overnight but also taking part in daily life. This creates opportunities to
experience first-hand details of people’s lives (e.g. about the difficulties collecting water, the behaviour
of service providers, the poor light with which to study, the impact of continuous rain on their livelihoods,
the hardships tilling drought affected land etc). It also provides the opportunity for detailed observation
(e.g. of household dynamics, child-parent behaviour and relationships, daily routine, who does what,
the difference between what people say they do and actually do). But most of all, it enables a relaxed
and trusted context for conversations and enhanced understanding of how people live their lives By
staying with a family, the power distance between the researcher and the family diminishes. By
purposely staying with the most disadvantaged, other voices (older people, children, poorer,
marginalised) often 'unheard' in conventional evaluations are also privileged.
Host households are a key focus for the RCA and host the immersion. Often a household is defined as
'a family unit which cohabits around a shared courtyard and often cooks together' but there are country
and context specific variations to this. The selection of host households in most RCAs to date has been
purposive, often focusing on poorer households or households with particularly relevant socio-economic
or demographic characteristics. For example, in Indonesia study households had to include schoolage children, whereas in the Nepal Koshi Hills study, households where several generations lived
together were important for the task of reflecting on forty years of development interventions.
Often an advance team facilitates identification and
explanations of the purpose of the RCA to prospective
host households. Some experienced RCA teams
have entered communities ‘cold’ i.e. without any
advance planning and this can work well too if
handled sensitively. RCA team members are alert to
‘gatekeepers’ and balancing the tensions between
paying courtesy to power holders but staying true to
the objectives of interacting with ordinary people in
ordinary ways. They work hard to reduce
expectations, explain their purpose and the need to
engage with ‘reality’, which, of course, also requires
them not to be afforded guest status.
Researcher shares a bed with family in
Bangladesh
Our experience with RCA has shown that as repeat
visits are made each year, it becomes easier and easier for the researcher to fit in with family routine.
The opportunity to share sleeping quarters (usually in our experience sharing a bed with family
14
members or sleeping on mats on the floor) confers some degree of equity in the relationship and
demonstrates a real desire on the part of the researcher to experience normal life in the household. No
RCA has been undertaken with less than two nights spent with the family per visit and this should be
considered a minimum. The first night often retains a novelty feel and the second is more productive
in terms of opportunities for open discussion.
Host households participation must be voluntary and people unfamiliar with RCA are often sceptical of
the willingness of families to host researcher and the ethics of burdening poor people with another
'mouth to feed' and a distraction from daily survival activities RCA experience to date has shown that
host families welcome the researcher and enjoy opportunities to chat in their own homes. Comfortable
in their own space and time, household members inevitably open up and readily explain their views
and their feelings, provide examples of change, debate and discuss issues of concern in ways which
they would rarely do in public forums or formal surveys. They appreciate showing the researcher what
their life is really like and the interest shown in them.
The issue of compensation to host households has been one over which all the RCA teams have
deliberated long and hard. The principle adopted is to ensure the family does not incur any cost from
the researcher’s visit. It also recognises that any contribution must be made discretely to ensure no
loss of face and to minimise local jealousies. Past experience in other countries 3 suggests that the most
appropriate form of compensation are basic food or consumable items (e.g. rice, cooking oil, salt, soap)
given to the host household upon leaving. These may be supplemented or substituted with other small
items e.g. the coloured pencils, torches, emergency lights, mosquito nets or blankets which the team
has taken with them. Cash is neither appropriate nor expected and has the danger of commoditising
the relationship. Copies of the photos taken of the family are given to them on subsequent visits and
are regarded as important gifts.
RCAs always involve interaction with people beyond the immediate household, including neighbours,
others in the community and local level service providers. This provides further opportunities for
triangulation and quickly expands the scope of the study. It is not unusual for a small scale RCA study
to include in depth conversations with well over 1.500 people.
STEP3:DE-BRIEFING
“Ask me what issues I was working on 8 years ago and I’d struggle to remember the details,
but I remember every detail of my immersion experience 8 years ago and how I felt
throughout.” Subrata De, Christian Aid
The most effective means of collating the combined experience of the RCA team members is through
very detailed de-briefings which the RCA team leader and sub-team leaders facilitate soon after the
completion of each RCA location study. This provides an opportunity to maximise the sharing and
recording of data as information is compared and added to by different team members during such debriefings. Team members recall observations and experience which they may not have noted in a
written report. This joint activity is an important additional opportunity for triangulation of findings. The
quote which starts this section illustrates how the combination of conversations, observation and
experience leads to exceptional recall. As a Ghanaian RCA practitioner noted, ‘I lived it … so I
remember it’.
3
Confirmed during the evaluation of the RCA in Bangladesh.
15
STEP4 :ANALYSIS AND REFLEXIVE PROCESSIING
The established approach of Framework Analysis can be used to examine the large quantity of
observational and conversational data collected using the following steps:



Identification of thematic framework: this phase involves identifying key issues, themes and
categories raised by the respondents which emerge from the discussion phase.
Charting: this phase involves re-visiting the entire set of data and placing summaries of the
views and experiences shared by the respondents inside the chart of themes. (categorisation)
Interpretation: this phase attempts to draw inferences from the charted summaries. However,
caution is always exercised in RCAs not to overlay the researcher’s interpretation of the
information. As the RCA is intended to explore multiple realities, it is important not to overcategorise and to ensure 'deviant' experiences and observations are also captured.
Software programmes such as NVivo can be used to manage the huge volume of qualitative data.
Some RCA studies construct grounded theory to provide further rigour to the interpretation of findings.
This involves a systematic building of theory (in reverse) through grouping and categorising ideas,
concerns and experiences of people into conceptual theoretical frames to explain action and behaviour.
A key element of the analytical phase is facilitating reflexivity. This is a purposeful and resource
intensive attempt to reduce bias and subjectivity in the research process and analysis of findings. The
team leader leads the whole team through processes which remind members to question the meanings
and constructs they have deduced from their findings, to think about their own positionality and influence
on the generation of information. Together, the team questions interpretation and by bringing their
multiple perspectives and findings to the forum allows for collaborative minimisation of bias.
STEP5:REPORT AND COMMUNICATING FINDINGS
Communicating the findings of RCA studies which always provide a wealth of information is a challenge.
This is another point at which research or research commissioning bias needs to be recognised and
contained. RCA reports are supposed to convey the opinions, perspectives and experiences of people.
Authorial voice and outsider interpretation needs to be minimised or transparently acknowledged. There
is always differences of opinion regarding the depth and breadth of information to be included in reports
and it is probably best to provide a range of products including full detailed report, summary and policy
briefs.
Report writing, like the analysis, is usually a collective activity, even if there is a main author. This
ensures that the whole team agrees on the main findings, agrees on the way these are conveyed and
has a say in the final report format. Illustrative examples are written by team members themselves
based on their first-hand experience. As mentioned above, RCA teams try to avoid being asked for
recommendations since these require a level of interpretation inconsistent with the intention to convey
people’s reality. Sometimes, RCA participants themselves have volunteered recommendations and
these can be on-conveyed as long as the context for these is clear.
The wealth of visual material produced during the course of the RCA study (photographs, drawings,
diagrams, video) provides opportunities for multi-media approaches to communication. Photographs
taken by people living in poverty themselves have proved to be powerful tools for influencing policy.
Inclusion of stories, quotes and reported conversations add to the richness of information. The
combination of narrative and visual has been shown to widen audience appeal and get RCA findings
noticed.
16
4. NOTES FOR RCA STUDY COMMISSIONERS
It is important to make the intended use of the RCA data explicit. Is the RCA a pulse taking exercise?
Should it shed more light on previous research, or help interpret quantitative studies? Or is it a standalone study of how people experience change or is it part of a mixed methods evaluation or research?
Broad areas of enquiry can be provided in Terms of Reference for further elaboration by the team. The
RCA team will want to maintain the confidentiality of the locations and households and so, while criteria
for location selection can be negotiated between the client and the team, actual identification needs to
be carried out by the RCA team themselves and they need to be able
to operate independently of any programmes or organisations.
In responses to tenders, the client should expect that the RCA
includes immersion and makes clear that engagement with people is
informal. The client should expect that the RCA team adopts
positionality aligned with people living in poverty. Sufficient days
need to be factored into the study for detailed and respectful
interactions with people.
BIAS
‘No research is free of biases,
assumptions and the
personality of the researcher’
Qualitative and quantitative research do different jobs and need to be
Sword, 1999
judged accordingly. It needs to be remembered that research
judgment is involved in both. Qualitative researchers are often
expected to go the extra mile to prove rigour/soundness. The client
should expect RCA study designs to make provisions for assuring rigour. Rigour is a means to
demonstrate plausibility, credibility and integrity and ultimately confidence in the research. The client
should require RCA design to demonstrate



rigour in procedures (e.g. justifications for site and household selections, allocation of sufficient
resources for meticulous reflexivity, demonstration of good triangulation practice and inclusion
of contradictory information)
rigour in analysis (e.g. efforts, where appropriate, to allow participants to review and validate
findings, transparent efforts to abstain from interpretation and judgement)
rigour in documentation: (e.g. efforts to keep complete records and provide a process trail of
the research (sufficient for another to follow and come to same conclusion), transparent and
explicit narratives).
Clients need to be aware that RCA produces rich and detailed information. To avoid over simplification,
report length should not be unrealistically limited. RCA reports will include visual and apt illustration.
The client should consider requesting a number of discrete products if they have concerns about the
readability of long reports rather than compromising on detail. As discussed above, clients should not
expect generalizable findings nor recommendations, but can expect RCA report authors to compile
implications from their findings.
5. FROM RESEARCH TO ACTION
RCA findings often challenge assumptions that have been central tenets of local and national level
policy making and operational planning. Whilst for some this can be an alienating experience, for many
policy makers in Bangladesh and Nepal engagement with the findings of the RCA has been a catalyst
for significant shifts in thinking on certain policy issues. In most cases the true impact of this shift will
not be felt by ordinary people for some time to come. The following are a few examples of how RCA
study findings have been deliberated upon and used.
17
BANGLADESH
A series of workshops were conducted in February 2012 at the conclusion of the Sida supported
Bangladesh RCA longitudinal study with a range of local service providers (teachers, health providers)
who had been involved in the five year RCA study (2007-2011). They shared experiences and
endorsed the findings and felt empowered (by the research and the realisation of common concerns)
to raise some of the issues within their institutions.
The Department of Education and other education stakeholders have taken seriously the perspectives
provided by through the RCA around the primary school stipends programme and people’s preference
for school feeding programmes. Perhaps most significantly there has been recognition by
representatives of the Government of Bangladesh and SWAP partners that the design of the PEDP-III
(Bangladesh National Government Primary Education Development Programme) was directly informed
by the findings of the RCA reports; particularly in relation to the complexities surrounding school boy
dropout rates. New measures to encourage and retain boys in school are explicit in the new programme.
Sida, as a relatively small bilateral donor in Bangladesh, used its comparative advantage within the
SWAP donor consortia, derived from its interest in accountability, voice and rights, to introduce and
promote the RCA. The RCA was envisioned to improve programme effectiveness through providing
new types of information from peoples’ perspectives at the grassroots. As one former Sida Embassy
staff member remarked in the reflection interview: ‘the Reality Check made us more visible, and it gave
us something to bring to the table’
Both the European Union and Sida referenced and used the RCA findings to develop their respective
Action Plans on gender and gender-based violence in Bangladesh, whilst the World Bank reported it
provided RCA reports to all Education and Health consultants working in the country as required
reading.
NEPAL
In Nepal, the DFID supported Koshi Hills RCA findings were shared with the National Planning
Commission (NPC) in a series of workshops in 2013. The reception was extremely positive and has led
to a demonstrable demand within the NPC for qualitative research studies to complement the dominant
tradition of quantitative surveys. A new knowledge sector programme has been designed to, among
other things, meet this demand. DFID has commissioned a further longitudinal RCA to assess change
in the major rural access programme in the mid and far west. This has influenced the design of the
quantitative household survey design and has flagged up issues for special studies e.g. the changing
face of poverty.
6. THE RCA COMMUNITY (TO DATE)
Whilst the RCA was an initial endeavour between the Swedish Government and GRM International in
Bangladesh, the positive response of donors has led to the RCA being replicated by a number of other
organisations. This loose group of organisations is now regularly collaborating as The RCA Community
of Practice, which includes a number of consultancies, research institutions and individual researchers:
18
The RCA website (http://reality-check-approach.com/) has been set up as a forum for this
Community of Practice as well as for more general public access.
7. FUTURE USE
At the time of publication, RCAs have been used as





Stand-alone longitudinal qualitative studies e.g. Bangladesh, Mozambique
One-off retrospective study e.g. Koshi Hills, Nepal
Pulse-taking study e.g. Indonesia
Scoping study to inform the design and implementation of larger scale household-based, mixed
method study e.g. Mid and Far West Nepal
Integrated within mixed method third party longitudinal evaluations e.g. Ghana and Mid and Far
West Nepal
It is hoped that the use of RCA will continue to grow as new development partners appreciate the value
of the approach. The application of RCA has already gone beyond its initial utilisation as a standalone
longitudinal study (as noted above). Involving the RCA in major mixed method longitudinal studies is a
new innovation and allows for an optimal method mix. Initial longitudinal studies of this nature were
funded by DFID and have built mutual respect between traditionally separate disciplines of qualitative
and quantitative research.
The increased interest in beneficiary feedback mechanisms (BFM) to improve development
programmes provides another opportunity for RCA. RCA’s informal and non -intrusive approach lends
itself to integration in methods to understand beneficiary perspectives and allow people to be critical
and demand their voices are responded to.
Additionally, decentralisation agendas in many countries offer opportunities for RCAs. Local
governments with restricted budgets will be able to afford small-scale but in depth RCA studies which
help to inform them to make their programmes context specific and responsive. This was an important
consideration in the design of the Indonesia RCA+ project, where local level research organisations will
be trained and mentored to provide this kind of research service.
While much is made of the way RCA can amplify the voices of people living in poverty, there is more
that can be done to strengthen the voices of frontline service providers, who are also marginalised from
policy dialogue space. Staying in the homes of local teachers, midwives, police, local government
officials can potentially provide important insights into their experienced reality and the constraints and
frustrations they face.
RCAs to date have been commissioned by development partners around education, rural access,
health, agriculture, social protection and rights based programmes. However, there is scope for private
sector and corporate social responsibility programmes operating in complex environments (e.g. in the
extractives industry) to commission RCAs to connect programmes to ground realities and to better meet
people’s aspirations, mitigate environmental and other risks.
References:
Chambers, R 2012, Provocations for Development, Practical Action Publishing
Kotter, J.P.and D.S.Cohen, 2002 The Heart of Change: real life stories of how people change their organisations’
Harvard Business Review Press
19
20
The 'A Short Introduction to RCA' is a Effective Development
Group production finalised in collaboration with
21
the Indonesia Reality Check Approach + Project an
Australian aid project, funded by the Australian Department of
Foreign Affairs and Trade.

Download Report