Scott Chaplowe. November, 2017

Introduction

Much of my work in monitoring and evaluation (M&E) has not simply been practicing M&E, but developing the capacity of others to practice, manage, support and/or use M&E. I am not alone as other evaluation colleagues have echoed similar experiences.  Drawing upon our book, Monitoring and Evaluation Training. A Systematic Approach, and an Expert Lecture I gave at the American Evaluation Association (AEA) 2017 conference in Washington, D.C., this post identifies ten key considerations for evaluation capacity development (ECD) as well as M&E capacity development in organizations. Because the principles are about human behavior, they are not absolutes, nor are they exhaustive.

Before we dive into the top ten principles, it is useful to step back and look at what we are diving into. An organization refers to a wide range of groups where people work together to collectively achieve given objectives – from national and international agencies to nongovernmental organizations (NGOs), volunteer organizations, private organizations, and community-based organizations (CBOs).

In its broadest sense, capacity can be understood as “the ability of people, organizations and society as a whole to manage their affairs successfully” (OECD, 2006, p.12). Building these capacities involves improving the ability of people to solve problems, achieve objectives, and perform better over time. For evaluation capacity development (ECD), it is useful to revisit one of the earliest and most cited definitions from a volume of the journal, New Directions for Evaluation, devoted to the topic:

“ECB is a context-dependent, intentional action system of guided processes and practices for bringing about and sustaining a state of affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing practices within and/or between one or more organizations/programs/sites,” (Stockdill, Baizerman, & Compton, 2002, p. 8).

[ECD is typically used in international development settings, but Evaluation Capacity Building (ECB) is also used in academic journals and other contexts. For our purposes, we use ECD to encompass ECD (e.g., ECDG, 2014; Horton et. al., 2003; Segone & Rugh, 2013; World Bank, 2014).]

Lastly, it is important to remember that evaluation is part of many interrelated process in an organization’s program management system. Evaluation professionals often find themselves supporting program design, monitoring, data management and reporting. As such, I use ECD broadly to refer to organizational capacity development that includes evaluation and other related processes, especially those involving the collection and analysis of data for reporting and use as part of a program’s overall management system.

Without any further ado, here are the top 10 tips for ECD:

Top 10 Tips for Evaluation Capacity Development (ECD) in Organizations

1.     Adopt a systemic approach to ECD

ECD does not happen in isolation, but is embedded in complex social systems. Each organization will be distinct in time and place (context), and ECD interventions should be tailored according to the unique configuration of different factors and actors that shape the supply and demand for ECD. Supply refers to the presence of evaluation capacity, (human and material), and demand refers to the motivations and incentives for evaluation use.

It is useful to consider a three-tier conceptualization of capacity building at the micro (e.g., individual), meso (e.g., organizational), and macro (e.g., societal) levels – see diagram below. This levels approach is adopted by others in the ECD and ECD literature, (for example, Fukuda-Parr, Lopes, & Malik, 2002; Hieder, 2011; Horton et. al., 2003; OECD, 2006; Segone & Rugh, 2013).

Although such conceptual diagrams are limited in their representation of reality, they underscore that ECD is embedded in and interdependent on a larger, dynamic system and should thus be planned accordingly. Rather than approaching ECD as simply building individual knowledge and skills, a levels analysis broadens the lens of capacity building as nested within systems with cascading effects down the various levels of analysis. For example, government policies for greater accountability can have a profound impact on organizational demand and support for ECD.

2.     Plan, deliver and follow-up ECD with attention to transfer

If ECD is to make a difference, it is not enough to ensure learning occurs; targeted learners need to apply their learning. As Hallie Preskill and Shanelle Boyle (2008, p. 453) state, “Unless people are willing and able to apply their evaluation knowledge, skills, and attitudes [KSA] toward effective evaluation practice, there is little chance for evaluation practice to be sustained.” This vision for transfer needs to be clearly understood by key stakeholders, and should inform the design, deliver and evaluation of organizational ECD programs.

3. Meaningfully engage stakeholders in the ECD process

ECD will be more effective when it is done with rather than to organizational stakeholders. Stakeholder participation is more than just consultation, but direct opportunity for people to provide input and become involved in various aspects of ECD. Meaningful engagement helps build ownership to sustain ECD implementation and use. It is especially important to identify and capitalize on ECD champions, and mitigate adversaries who can block ECD and its uptake. This stakeholder analysis is an important part of the overall systems analysis in which ECD is to be provided.

4. Systematically approach organizational ECD, but remain flexible and adaptable to changing needs

ECD is intentional and conducted to meet specific needs. Therefore, it should be orderly planned to gather information and analyze training demand, needs and resources, identify relevant objectives, and design interventions to realistically achieve and evaluate these objectives. However, a systematic approach does not mean a rigid blueprint that is blindly followed. This can inhibit experimentation and the ability of ECD to adapt and respond to changing needs and unanticipated outcomes – whether positive opportunities or challenging obstacles. A flexible approach to ECD recognizes the dynamic nature of the system in which an organization is embedded, which will vary and change over time.

5. Align and pursue ECD in relation organizational objectives

A systemic approach to ECD includes careful attention to other organizational objectives and capacity building interventions. ECD does not exist for its own sake, but as a means to an end relative to the organization’s overall mission and strategic objectives.  It should not be “silo-ed,” nor should it duplicate or compete with other capacity building efforts. ECD is ideally planned as part of a coherent strategy that complements and reinforces overall capacity building in an organization. For example, rather than a stand-alone training on monitoring and evaluation (M&E), explore ways to ‘blend’ it into existing program management training.

6.  Ensure your ECD strategy is practical and realistic to organizational capacities 

ECD should be realistic given the available time, budget, expertise and other resources. Such considerations includes the organizational context, number of staff, their location, their availability, their current level of evaluation understanding, existing training materials and facilities, and the  number and experience of people required to support ECD. Just as when planning an evaluation, this entails careful context analysis to ensure ECD objectives are feasible given these real-world practicalities, and candid stakeholder communication to frame realistic expectations.

7. Identify & capitalize on existing sources for ECD

There are a multiplicity of resources for and approaches to ECD, ranging from face-to-face training, online webinars, communities of practice, discussion boards, self-paced reading, and blogs like this. These resources can be used in solo or blended as part of a capacity building program that supports different learning styles and needs. Indeed, it is important not to ‘reinvent the wheel’ if it can be ‘recycled.’ This can save time and money. However, do not fall into the trap of adopting just because it is available—ensure that ECD resources are relevant for the desired capacity building objectives, or can be modified accordingly. The table below summarizes different examples of outlets and mediums to deliver ECD.

 8. Design and deliver learning grounded on adult learning principles

Adults learn better when it is delivered in an enjoyable and meaningful way. Adults are self-directed learners that bring to training past experiences, values, opinions, expectations and priorities that shape why and how they learn. Principles for adult learning stress a learner-centered approach that is applied, experiential, participatory and builds upon prior experience. They also include other good practices for learning regardless of the age, such as the use of mixed-methods, structured progression, repetition, and feedback. Below we summarize 14 key principles for adult learning identified in our book, M&E Training: A Systematic Approach, and which you can read more about here.

Key Adult Learning Principles for M&E Training

1. Establish a safe and respectful climate – Adults learn better when they feel safe and respected.

2. Respond to the “need to know” (NTK) – Adults prefer to know what,  why, and how they are learning.

3. Provide a structured yet flexible progression – Adults prefer learning that is well-organized.

4. Empower with genuine participation – Adults want to share full responsibility for their learning.

5. Incorporate past experience – Adults prefer learning that builds upon their prior experience.

6. Keep it relevant & meaningful – Adults prefer practical learning that meets their needs.

7. Provide direct experience – Adults learn best by doing.

8. Make it active, fun and challenging – Adults learn more when it is engaging and enjoyable.

9. Use mixed/multisensory methods – Adult learners require a mixture of learning approaches.

10. Differentiate instruction – Adults learning is more effective when instruction is tailored to different learners’ needs.

11. Utilize collaborative, peer learning – Adults effectively learn from each other.

12. Include practice and repetition – Adult learning is enhanced by repetition.

13. Design for primacy and recency – Adults remember best what they learn first and last in sequence.

14. Provide feedback & positive reinforcement – Adults want to know if they are learning, and to be encouraged in the process.

9.      Uphold professional standards, principles and ethics

An essential aspect of capacity building it to instill an understanding of and appreciation for ethical conduct and other standards for good practice. Specific guidelines and principles will vary according to context - sometimes specific to the organization itself, other times adopted from industry standards, such as the. AEA’s Guiding Principles For Evaluators and Statement on Cultural Competence in Evaluation, and the JCSEE’s Program Evaluation Standards Statements. Whatever the source, it is critical that ECD is informed by and reinforces professional standards, principles and ethics.

10.     Monitor and evaluate ECD efforts to learn and adapt

ECD should practice what it preaches and track and assess ECD efforts to adapt, improve and be accountable to ECD objectives and stakeholders. This begins at the design stage, when identifying ECD objectives to be assessed, so resources can be allocated and steps taken to monitor and evaluate ECD moving forward. And it is important to remember that monitoring and evaluation not only focus on identified ECD objectives, but it should remain alert to unanticipated consequences, whether positive or negative, and other contextual cues, so the ECD strategy can be revised and adapted accordingly – which brings us full circle to tip #1 on adopting a systems perspective for ECD.

(11.    Ancillary Reminder)

The above top 10 list is far from exhaustive, and as they are about human organizations and behavior, they are not absolute.

Citations

• • • Chaplowe, S, & Cousin, J B. (2016). M&E Training: A Systematic Approach. Thousand Oaks, California: Sage Publications

    • Compton, D., Baizerman, M., & Stockdill, S. H. (Eds.). (2002). New Directions for Evaluation Series: Vol. 93. The art, craft and science of evaluation capacity building (pp. 1–120). San Francisco: Jossey Bass

    • ECDG (Evaluation Capacity Development Group). (2015). Retrieved from http://www.ecdg.net

    • Fukuda-Parr, S., Lopes, C., & Malik, K. (Eds.). (2002). Capacity for development: new solutions to old problems. New York: United Nations Development Programme (UNDP) with Earthscan Publishing.

    • Hieder, C. (2011). Conceptual framework for developing evaluation capacities: Building on good practices in evaluation and capacity development. In R. Rist, M.-H. Boily, & F. Martin (Eds.), Influencing change: Evaluation and capacity building (pp. 85–110). Washington, DC: The World Bank.

    • Horton, D., Alexaki, A., Bennett- Lartey, S., Brice, K. N., Campilan, D., Carden, F., . . . Watts, J. ( (2003). Evaluating capacity development: Experiences from research and development organizations around the world. The Netherlands: International Service for National Agricultural Research (ISNAR); Canada: International Development Research Centre (IDRC); the Netherlands: ACP-EU Technical Centre for Agricultural and Rural Cooperation (CTA).

    • OECD (Organisation for Economic Co-operation and Development). (2006). The challenge of capacity development: Working towards good practice. Paris: OECD.

    • Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459.

    • Segone, M., & Rugh, J. (Eds.). (2013). Evaluation and civil society. Stakeholders’ perspectives on national evaluation capacity development. UNICEF, IOCE, and EvalPartners.

    • World Bank. (2014). Evaluation capacity development (ECD). Independent Evaluation Group and the World Bank Group. https://ieg.worldbankgroup.org/topic/evaluation-capacity-development