Scott Chaplowe. April, 2017

Introduction

Effective evaluation capacity development (ECD) requires a realistic understanding of the context in which ECD capacities are to be developed and used.  This includes the organizational structure and culture in which capacity development occurs, as well as the broader operational context that can hinder or enable ECD capacity development. This post presents a checklist of key considerations to inform context analysis for ECD as well as monitoring and evaluation (M&E) capacity development.

A context analysis for ECD examines the system-wide components of an organization and its operational context to determine the recommended approach to capacity development.  It is important to determine not only the organizational need, but also its readiness for ECD.

Organizational contexts are diverse, ranging from public national and international agencies to civic and private organizations. Particular attention should be given to the political climate in an organization, identifying potential ‘roadblocks’ (resistance) to ECD capacity development, as well as potential champions (key supporters) that shape the demand and can help steer resources and support for the uptake of ECD efforts.

It is also essential to identify specific actors and factors in the external environment that can influence ECD and its transfer into practice. This includes broader social, political and economic forces, such as the demand for accountability, ECD expectations and standards, donor ECD and reporting requirements (format and frequency), funding availability and other resources for ECD. In addition, there are practical considerations, such as the public infrastructure, facilities, transportation, and access to internet, which can have a pervasive impact on ECD capacity development.

There are a range of secondary and primary sources of data that can be consulted for the context analysis. Potential stakeholders include people in an organization, community members, partner organizations, donors, government agencies and officials. Particular attention to any capacity assessment data and related background documentation already available; it is important to avoid  duplication of analysis, which can result in analysis fatigue and undermine support for ECD capacity development. When prior analysis data is available, it is important to understand who made the analysis and how it was conducted.

A final reminder before introducing the checklist: ECD should be aligned and pursued in relation to other organizational objectives and capacity development interventions, (see separate post, Top Ten Tips for Evaluation Capacity Building in Organizations). ECD does not exist for its own sake, but as a means to an end relative to an organization’s overall mission and strategic objectives.  It should not be “silo-ed,” nor should it duplicate or compete with other capacity development efforts. ECD is ideally planned as part of a coherent strategy that complements and reinforces overall organizational capacity development.

Ultimately, who and what is consulted for a context analysis for ECD will depend on the specific organization, stakeholders and needs. Therefore, the following checklist is far from exhaustive, but is illustrative to be adapted according to need and context.

Context Analysis Checklist for Evaluation Capacity Development (ECD)

Source: Chaplowe, S., & Cousins, B. 2016. Monitoring & Evaluation Training: A Systematic Approach. Thousand Oaks, CA: Sage Publications

Organizational Factors

1.     Management structure & processes 

• What is the overall management structure (organogram), and where do ECD functions sit?

• What are the organization’s strategic objectives, and how do they relate to ECD and its capacity development?

• How might organizational policies, procedures, and other administrative processes affect ECD practice?

• Are there well-defined roles and responsibilities for ECD—who is accountable for what?

• How is ECD reflected in job descriptions and staff recruitment?

• What incentives are used to promote ECD (carrots to encourage and reward ECD practice; sticks to require ECD and penalize poor practice; sermons that endorse or advocate the value of ECD)?

• Are individual assessments conducted for ECD and related skills sets?

• Is ECD included in performance appraisals and supported by career development—for example, coaching or mentoring? 

2.     ECD resources & practice 

• What is the current ECD capacity of the organization—for example, is ECD information timely, reliable, and useful for the organization?

• Has an organizational assessment been conducted for ECD (separately or as part of an overall organizational assessment)?

• What are the gaps in the current ECD system that need to be addressed—for example, is there an adequate analytical capacity for decision-makers to interpret and use evidence?

• Is there a strategic plan or objectives for ECD and its capacity building, and to what degree is it followed, monitored, and evaluated?

• What existing guidelines, tools, templates and other resources are available for ECD functions?

• What is the current level of human, financial, physical, and technology resources for ECD?

• Is ECD specifically itemized in program budgets, or is it subsumed (assumed) as part of the overall operational budget?

• What are the funding prospects for sustaining or improving the ECD system? 

3.       Capacity development resources & practice 

• What is the current “capacity for capacity development” within the organization? How often does it occur, and what is its uptake?

• What ECD-dedicated training, resources, and other capacity development activities are currently available (either internal or external to the organization)? 

• What ECD-dedicated training, resources, and other capacity development activities have been available in the past? Have they been effective and why?

• What non-ECD learning initiatives/training already exist? What is their timing and frequency? Do these trainings incorporate any ECD content?

• How might ECD be complemented and supported by existing capacity development activities and systems?

• What is the availability of funding for capacity development and specifically for the requested ECD capacity development?

• What is the availability of human resources for ECD – for example, for training development, facilitation, follow-up, and administration? (Note: consider not only their presence but availability for supporting capacity development).

• What material/physical resources are available for the requested ECD – for example, existing ECD training materials (curriculums and content), equipment and technology, and training? 

4.     Networking & linkages

• What systems are established for advocacy, resource mobilization, and partnership engagement for ECD?

• Are there potential public agencies, VOPEs, research centers, universities, and so forth that could serve as reliable partners for ECD capacity objectives?

• How well does the organization identify and leverage autonomy in its partnerships for ECD and its capacity development?

• Are there exchange visits, communities of practice, or other opportunities for ECD sharing and capacity development that can complement and should be planned with ECD training?

5.     Organizational culture & demand  

• What is driving the demand for ECD in the organization, e.g. internal or externally (donor) driven?

• What is the overall climate or perception of ECD capacity development? Is it perceived as top-down and mandated, or is there buy-in and understanding of its value within the organization?

• Is there leadership/senior management support for ECD within the organization?

• Who, specifically, is requesting ECD and why?

• What is the specific commitment for (or resistance to) the proposed ECD capacity development?

• How will the transfer (use of) and return on investment from ECD be sustained?

• Who are the key decision-makers, potential champions, as well as roadblocks for ECD capacity development?

• Are there trusting relationships, teamwork, and collective responsibility in the organization/program?

• What are the pressures and responsibilities faced by managers and staff that affect ECD functions within the organization?

• What incentives and disincentives are there for ECD at the organizational level? 

External Environment Factors

6.     Sociocultural setting 

• What historical, demographic, and other cultural factors are relevant for ECD capacity development—for example, mores, values, and ways of knowing as well as holidays and customs that can affect capacity development delivery and scheduling? 

7.     Political and economic trends 

• What political and economic factors affect the priorities and funding for accountable ECD—for example, stability, crisis, corruption, changing political leadership, and agendas? 

8.     Regulatory environment 

• What policy, laws, and administrative and bureaucratic systems affect the operational environment for organizations?

• Are there ECD standards, norms and requirements established and promoted through accrediting and public agencies and VOPEs?

• Are ECD standards, norms and requirements upheld? Are they externally monitored and assessed? 

9.   Demand and funding 

• What funding as well as other resources are available from public, international, and private donors for ECD and the specific training being considered?

• To what degree does external funding dictate ECD and overall programing priorities within the recipient organization?

• What are donor/partner ECD expectations and requirements for ECD and reporting format, content, and frequency?

• What degree are external incentives commonly employed for ECD? 

10.   External support/resources 

• What is the availability of external research and technical support for ECD – for example, from research institutes, universities, think tanks, VOPES and other professional organizations?

• What is the availability and reliability of public statistical information and the management systems to access and use ECD – for example, from government ministries or international organizations such as UN agencies?

• What is the public infrastructure and technological options available for ECD – for example, Internet access in a less versus more developed country?

• Are there mechanisms to support communication, coordination, and collaboration across organizations (civil, public, or private) for ECD?