OVC Support Toolkit  >  Running a programme  >  Improving the Work 

Monitoring and Evaluation

This section looks at the monitoring and evaluation of programmes. This is an essential part of improving work. Related sections include working with others and setting standards.

Key points about monitoring and evaluation are:

1. Both monitoring and evaluation are processes used to assess project progress.

2. 'Monitoring' refers to an ongoing system which routinely collects data and is used to keep the project 'on track'.

3. 'Evaluation' refers to a one-off event conducted to account for resources used and/or to document lessons learned.

4. There are two main approaches to evaluations. Scientific approaches emphasise the importance of objective facts/evidence. Interpretive approaches emphasis the views and perspectives of people affected by/involved with the project. These people are termed 'stakeholders'.

5. Many approaches to evaluation now emphasise the active participation of stakeholders, particularly children and young people. Different stakeholders may have very different levels of power within a project.

6. Indicators are things which are used to measure or assess progress made by a project. They may be expressed as numbers (quantitative) or descriptive words (qualitative). They may be internationally or locally-defined and can be used to measure project activities at different levels, for example processes/activities and outcomes/impacts

7. Good monitoring systems and evaluation approaches collect and compare information from a variety of different sources using different methods. This is termed triangulation.

Defining Monitoring and Evaluation

Resources in this section contain a number of definitions of the terms 'monitoring' and 'evaluation'. They differ from each other in four main ways:

1. Scope : Some authors use the term monitoring for the tracking of information related to the national HIV/AIDS response and the term evaluation for the gathering of information related to a particular programme or project.
2. Nature : The term monitoring is used to describe a routine system of collecting and analyzing information about the work of the project. On the other hand, the term evaluation is usually used to describe a specific event.
3. Timing : Monitoring is a regular and ongoing process which takes place throughout the life of a project. An evaluation usually occurs at a particular time, for example midway through a project or at the end.
4. Purpose: The purpose of monitoring is relatively specific, in that it usually focuses on keeping a project 'on track'. This involves routinely measuring what the project has done and comparing this with its plans. Evaluation usually has broader purposes, for example, assessing what has been learned as a result of project activities.

Project Monitoring and Evaluation Compared to Other Types of Monitoring and Evaluation

This section focuses largely on the monitoring and evaluation of a specific project or programme. Examples of evaluations of projects, such as the evaluation of the Families, Orphans and Children Under Stress (FOCUS) programme in Mutare, Zimbabwe may be helpful for others seeking to evaluate their activities.

However, monitoring and evaluation can also be carried out at a national level. It is important that any system for monitoring and evaluating a project or programme should be consistent with the national system. This fits with the UNAIDS 'three ones' strategy which calls for one national coordinating body for HIV/AIDS, one national strategic framework and one national M&E system.

Analysing a situation before activities start is sometimes called formative evaluation. This is covered in more detail in the section on situation analysis.

Purpose of Evaluation

Two main purposes can be identified for conducting an evaluation. First, an evaluation can be conducted to hold a project accountable for what it has done and achieved. In such cases, actual project activities and achievements will be compared with what was planned. Such evaluations are often required by donor organizations that provide funds for a project. The receipt of further funds may depend on achieving a satisfactory outcome to such an evaluation. This approach is sometimes referred to as 'performance-based funding'. It is used by some major donors, including the Global Fund.

Secondly, an evaluation may seek to learn lessons from project activities. This is likely to include an assessment of what worked well and what didn't. Things that have worked well may be referred to as 'good', 'effective' or 'best' practice. One reason for seeking to learn lessons is to improve the activities and services of a project. The experience of the Community Counselling Aides (CCA) Project in Uganda shows that even simple monitoring processes can improve practice and can result in attracting additional support for the project.

There is a tension between these two purposes. For example, this may explain the reason why 'negative' findings, that is what didn't work, are rarely recorded in evaluation reports. Although such findings are very useful for the purpose of learning, they are problematic in terms of accountability, because donors are unlikely to be willing to fund activities shown to be ineffective. Consequently, organizations may not wish to publicise findings of this nature.

Ways of Conducting Evaluations

There are a variety of ways of conducting evaluations. These can be divided into two main types. The 'scientific' approach seeks to compare a group of people receiving a particular intervention or service with a comparable group who do not receive it. There are various types of scientific methods that can be used. The purest form of this is referred to as a randomized, controlled trial. It places strong emphasis on objective facts/evidence. On the other hand, the 'interpretive' approach places much greater emphasis on trying to understand the views and perspectives of different individuals and groups associated with the project. Both the scientific and interpretive approach can produce important information for a programme.

Stakeholders

People associated with the project can be referred to as 'stakeholders'. Many different stakeholder groups can be identified for a particular project, including the children and young people who are intended to benefit from it. These stakeholders may have very different expectations of a project evaluation. Two key issues relating to stakeholders and evaluation are:

1. Power : There may be power imbalances between different groups of stakeholders. For example, donor views may be given greater weight within an evaluation because they control project funds.

2. Participation: Interpretive approaches to evaluation place great emphasis on the active participation of project stakeholders, particularly children and young people, in an evaluation. A number of questions have been identified relating to the participation of children and young people in activities and their evaluation. These are:

• Have children and young people, and their families, been involved in deciding the measure for the evaluation?
• Can children and young people benefit from the evaluation, and how?
• What have been the risks and costs for children of their involvement?
• Have confidentiality and privacy been respected at all times?
• Do the children involved know that they are free to refuse, or withdraw at any stage, and that this will not be held against them?
• If certain children have been excluded from participating, can their exclusion be justified?
• Have the children concerned and/or their carers, helped to plan, implement, analyse and evaluate the activity
• Are the children concerned aware of the purpose and nature of their participation, methods, timings, benefits, consequences and outcomes?
• What have the project workers learnt from the participation of children and young people?
• Will the children, young people and their families be told of the main findings?
• Apart from the effects of the activity on participants, how might the conclusions affect other children and young people?
• What are the constraints on the participation of girls, disabled children, orphans or children living with HIV/AIDS? How can these constraints be overcome?

Participation of children and young people in evaluations may lead to issues being identified which might otherwise be overlooked. For example, in the evaluation of the Families Orphans and Children Under Stress (FOCUS) programme in Zimbabwe, children and young people identified stigma and sexual abuse as major issues.

If people are to participate in monitoring and evaluation processes, they may require training. Experience from the Community-Based Options for Protection and Empowerment (COPE) project in Malawi concluded that many community committee members did not understand the data being collected as they had not received adequate training. The same review also concluded that data collection processes were more likely to be successful if government employees were involved.

Indicators

Indicators are used in both monitoring and evaluation. Essentially, they are things which can be measured or assessed to see the progress being made by a project. They may be expressed in numbers (quantitative) or through descriptive words (qualitative). They may form part of an international set of core indicators or may be developed locally for a specific project. For example, a number of core indicators have been agreed for use at national level.

Indicators may measure different 'levels' of a project. These levels include:

• Inputs - these are the things needed for the project to occur. They are like the raw ingredients needed to cook a meal. Examples include people to do the work and money.
• Processes - these are the activities of a project. In general, these are relatively easy to measure and process indicators often form the bulk of monitoring systems. An example of a process would be training sessions held for staff. Various indicators may be used to measure these processes such as number of training sessions held and number of people trained.
• Outputs - these are things produced by the project. Examples include new or improved services, trained staff, information materials etc.
• Outcomes - these are the positive short-term effects of a programme. Examples include improved child nutritional status and improved school enrolment.
• Impact - this is the longer-term change which arises as a result of several positive outcomes. Common measures of impact of the response to HIV are reduction in new HIV infections and increased survival time of people living with HIV.

However, different authors use these terms in slightly different ways. For example, some people regard the number of people trained as an input while others see it as an output. This can be confusing. As a result, some people have argued for a simpler system which measures the services a project delivers and the effect that it has. Examples of these indicators are presented in the table below.

A service delivery indicator may be expressed as an absolute number or as a percentage. In order to express a service delivery indicator as a percentage, it is necessary to know the number of people who received a service or benefited from an activity. This is the number which will be divided and is called the numerator. Then, it is necessary to estimate the number of people who needed the service. This is the number that the numerator will be divided by and is called the denominator.

For example, if a particular project reached 500 orphans and other vulnerable children with services in an area where there were 1000 orphans and other vulnerable children in need of services, the numerator would be 500 and the denominator 1000. The percentage would be 500 divided by 1000 multiplied by 100, that is 50%. This percentage is sometimes called the 'coverage' of a particular service.

Indicators may focus on children as individuals or on the environments in which they live, particularly their families. Experience from the work of the National Community of Women Living with AIDS (NACWOLA) in Uganda emphasises the latter approach. It focuses on promoting dialogue between women and their children and the passing on of vital information to children before their mother dies. Indicators of success for such processes might include:

• Percentage of mothers disclosing HIV status to their children
• Percentage of mothers who have prepared a will
• Percentage of mothers who have prepared a memory book
• Percentage of children who have been consulted in fostering arrangements
• Percentage of children trained in how to care for sick parents
• Income-generating activities of mother successfully continued by children after her death
• Incidence of property grabbing
• Proportion of wills respected and inheritance rights fulfilled

Monitoring and Evaluation Methods

Monitoring and evaluation may use a variety or different methods for collecting information. Primary methods are those used by the people doing the monitoring/evaluation. Primary methods may include questionnaires, surveys, focus group discussions, direct observation and interviews with key informants. Secondary methods involve looking at work done by other people, usually through reviewing project and other documents. Good quality monitoring and evaluation uses information from a variety of different sources, collected using different methods to cross-check its validity. This is called triangulation.

• Send to a friend
• Printer-friendly
• Introductions to themes (pdf)