Other evaluations

Alliance publications evaluation (in progress)

The main emphasis of this evaluation is to enable the Alliance publications department (and others involved in the commissioning of published material) to improve the quality, reach, use and impact of current and future publications – and so increase demand for more material. The final evaluation will generate a series of recommendations based on a comprehensive review of Alliance publications. It will also lead to the development of a simple but effective in-house publications monitoring system.

Earlier in 2004, a process evaluation was completed for the resource pack Building Blocks: Africa-wide briefing notes' (see below) – a set of locally adaptable resources to help communities and local organisations in Africa support children orphaned and made vulnerable by HIV. The remit of the process evaluation focused on how the resource pack had been developed. However, its findings highlighted the need for a separate study to determine the use, reach, effectiveness and impact of the material. The need for this had been written into the project design in 2001. The following scope of work will address this for a selection of key Alliance publications in addition to 'Building Blocks'.

Broad objectives for publications evaluation

  • to determine the reach, use and effectiveness of a selection of major Alliance-published resources
  • to identify good practice in dissemination and awareness-raising of published materials, and recommend key lessons to be incorporated into future publication planning and development
  • to compare relative use and cost of a range of Alliance publication formats/types and designs
  • to identify, recommend and suggest a simple monitoring system (with accompanying cost estimate) to track the use and reach of published resources (including website effectiveness).
Specific objectives for the Building Blocks evaluation
  • to incorporate observations and recommendations from the process evaluation report regarding follow-on impact work (including unexpected impacts)
  • to identify how people are using the resource and the reported impact it is having on their work
  • to recommend any key changes in material content, style or format to inform 'Building Blocks' revision
  • to compare and contrast differences in attitudes to, and perceived usefulness of, material by end-users – including those involved in the 'Building Blocks' development process as well as those not involved in the process.

Building Blocks process evaluation

Overview

The 'Building Blocks' project began in September 2001 as part of the United States Agency for International Development (USAID) Africa Bureau’s response to the effect of the HIV pandemic on children. It was designed to produce community-adapted tools and briefing notes for individuals and organisations assisting children affected by HIV in Africa. The project has resulted in the production and dissemination of 'Building Blocks: Africa-wide briefing notes' – a set of locally adaptable resources to help communities and local organisations in Africa support children orphaned and made vulnerable by HIV.

The actual process of producing the resources was a central component of this project. The idea from the outset was that the briefing notes would be developed through a highly participatory process. Using workshops, country-level reviews, meetings and independent evaluation, it was envisaged that 500–600 adults and young people would become involved with the development of the products during this project. This process has now resulted in greater ownership and more relevant products, together with effective and targeted distribution.

Process evaluation

The Research and Evaluation Unit undertook an evaluation of the 'Building Blocks' development process so that lessons could be learnt, and to determine whether the process had met its objectives. The scope of work did not include an evaluation of the reach, use and impact of the resources. This will be conducted at a later date to allow enough time for dissemination and uptake.

Ukraine evaluation

The R&E Unit led an evaluation of the Alliance programme in Ukraine. This consisted of five elements:

  • analysis of the Alliance programme at the national level. This was informed by key informant interviews in Ukraine using a semi-structured questionnaire specifically developed for the evaluation. The key informants interviewed included international donor organisations working in HIV in Ukraine, and government officials
  • interviews in Ukraine with the main users of published materials developed by the Alliance Information Centre, to find out how relevant the materials are to them
  • analysis of how adequately the Alliance website is being used for dissemination
  • analysis and evaluation of the operations of the non-governmental organisations supported by the Alliance and the impact of their work, based on interviews of the main stakeholders at regional and local levels in Ukraine
  • review of the published materials on the Alliance Ukraine programmes and their impact on the general population.

Africa regional strategy internal evaluation

The Alliance commissioned an internal evaluation of its Africa regional strategy covering the period 2002–2004. The purpose, objectives and scope of work were agreed between the field programmes department and research and evaluation team at the Alliance secretariat. The review then took place during August and September 2004. The two main objectives of the evaluation were to:

  • generate information and examples of good practice to inform the next Africa regional strategy (for 2005–2007)
  • identify the challenges faced and the added-value of a regional approach in programmatic and management terms.

The evaluation, conducted as a desk review, used a mixture of small-group discussions and one-to-one and telephone interviews with Alliance staff at the secretariat and partners. There was also a documentation review.

Key evaluation questions were formed and the results analysed using an adapted version of the business-quality model for performance improvement and learning – the European Foundation for Quality Management. This identifies key learning achieved and improvement areas required for a project or programme based on a set of seven enablers (or driving forces) and results achieved.