ICAO Evaluation Policy

​THE ICAO EVALUATION POLICY
 
“How do we know if the United Nations is doing the right things? And how do we know we are doing these things right? These are the questions that evaluation is supposed to answer. Evaluation is not easy. Nor is it popular. But it is essential.”
 
United Nations Secretary-General, Ban Ki-moon, UN Evaluation Group’s High-Level Panel on “UN Results – Are we achieving them? How do we know?” (16 April 2013)
 
1.   Introduction
 
1.1. The ICAO Evaluation Policy (“the Policy”) is part of the institutional framework for evaluation at ICAO and complements the existing The Evaluation and Internal Audit Office (EAO) Charter.
 
1.2. The Policy aims to foster a common understanding of evaluation and reinforce the utility that evaluations can provide to stakeholders at ICAO. The Policy outlines the aims of evaluation, the guiding principles, the  types  of  evaluation  that  ICAO  conducts,  the  evaluation  criteria,  planning  and  budgeting  for evaluations,  the  elements  of  quality  assurance,  the  roles  and  responsibilities,  the  dissemination  of evaluation reports and follow up to evaluation recommendations.
 
1.3. The Policy shall be periodically reviewed and updated, ensuring alignment with the EAO Charter. EAO will consult with the Evaluation and Audit Advisory Committee (EAAC) and the Secretary General on any subsequent changes to the Policy, which are subject to approval by the Council.
 
2.   Aims and Guiding Principles of Evaluation
 
2.1. Consistent with the United Nations Evaluation Group’s (UNEG) Norms and Standards, evaluation at ICAO aims to contribute primarily to organizational learning and decision-making by providing lessons learned and recommendations for improvement; as well as accountability by providing evidence of institutional performance. Other aims of evaluation at ICAO include knowledge management and capacity development.
 
2.2. In addition to the general professional standards of EAO as defined in its Charter and in line with the
UNEG norms and standards, the following are the principles that guide evaluation at ICAO:
 
(a) independence and credibility,
(b) utility and timeliness,
(c) collaboration with internal audit, and
(d) managing for results 1.
 
3.   Definitions
 
3.1. Evaluation is an assessment of the relevance, effectiveness, efficiency, impact and sustainability of an activity, project or programme2. Evaluation is an important source of evidence of the achievement of results and institutional performance, it builds knowledge and contributes to organizational learning and it is an important agent of change. The types of evaluations carried out by EAO include ex-ante, formative, developmental, summative/ex-post and impact evaluations.
 
3.2. Evaluations conducted and commissioned directly by management and functional units3, without the involvement of EAO are generally termed “decentralized evaluations”4. Decentralized evaluations may bemanaged and conducted outside the control of EAO; however, Programme Managers and the Senior Management of the Secretariat are required to notify EAO of such evaluations. EAO will maintain a register of decentralized evaluations, which will be updated annually.
 
3.3. The following evaluation criteria are commonly used at ICAO when conducting evaluations of an activity,  project  or  programme:  relevance,  effectiveness,  efficiency,  impact  and  sustainability.  Other criteria may be used if deemed relevant. The selected evaluation criteria provide a basis for formulating the key evaluation questions and are part of the evaluation framework and terms of reference.
 
4.   Evaluation Planning
 
4.1. Each year EAO prepares a Work Plan in consultation with the Senior Management of the Secretariat, EAAC and the Secretary General, which is presented to the Council. EAO proposes, prioritizes and finalizes evaluation topics for the Work Plan through management consultations and consideration of current and emerging organizational issues.
 
4.2. When developing the evaluation plan, due regard will be given to the cost of conducting an evaluation in relation to its potential benefit, as well as EAO’s capacity to conduct the evaluation with available human and financial resources. A key consideration is the extent to which the evaluation results could be reliably used for decision-making.
 
4.3. EAO  should  have  access  to  all  necessary  organizational  data  and  information  in  order  to  conduct evaluation planning and analysis.
 
5.   Quality Assurance
 
5.1. The quality of the evaluation process and results at ICAO is ensured through conformity to the UNEG norms and standards; the hiring of Evaluation Officers with sound evaluation competencies, which are maintained and further developed through relevant learning opportunities; the use of the Evaluation Manual and associated tools and templates; the conduct of evaluability assessments and internal reviews of evaluation reports; use of the evaluation criteria; and validation of evaluation methodology, data and results. When evaluation consultants are hired, the ICAO consultant performance evaluation is used to inform decisions on potential future collaborations.
 
5.2. Quality of the evaluation function is ensured through engagement with UNEG activities in order to keep updated with best practices in UN evaluations, by conducting periodic self-assessments and taking part in independent external assessments of the ICAO evaluation function, such as the UNEG peer review mechanism or the Joint Inspection Unit (JIU).
 
6.   Roles and Responsibilities
 
6.1. EAO is the custodian of the evaluation function at ICAO. The main role and responsibilities of EAO with respect to the centralized evaluation function are to:
 
(a) Develop a triennial and annual evaluation plan and programme;
(b) Develop and establish effective guidelines, tools and templates for centralized evaluation at ICAO, including the Evaluation Manual, which cover planning, methodology, quality assurance, management responses and knowledge management;
(c) Provide an annual activity report to the Council;
(d) Present evaluation results to the Senior Management, the EAAC and the Council;
(e) Promote the use of evaluation through consultations with the President of the Council and Senior
Management for the purposes of evaluation planning and follow-up to recommendations and through the evaluation process and use;
(f)  Participate  in  relevant  UNEG  and  JIU  events  to  represent  the  interests  of  ICAO  on  evaluation oversight issues.
 
6.2. The main role and responsibilities of EAO with respect to the decentralized evaluation function are to:
 
(a) Develop and establish effective guidelines, tools and templates to support decentralized evaluation at ICAO;
(a) Provide methodological and quality assurance support to decentralized evaluations; and
(b) Maintain a register of decentralized evaluations conducted at ICAO.
 
6.3. Evaluation Officers may offer advisory services upon request that provide guidance intended to inform decision-making of ICAO Senior Management, ensuring that such services do not assume a management role. Evaluation Officers may also conduct periodic awareness raising and training sessions on the evaluation function, policy and guidelines aimed at the Senior Management, Programme and Project Managers.
 
6.4. Senior Management, Programme Managers and other ICAO evaluation stakeholders have a role and responsibilities in supporting the evaluation function, for example, to:
 
(a) Propose relevant evaluation topics for the annual evaluation work plan;
(b) Review evaluation terms of reference;
(c) Facilitate  access  to  data  and  information  on  activities,  projects  and  programmes,  including performance results;
(d) Ensure data quality, consistency and elimination or minimization of data bias;
(e) Engage relevant Bureaus in the evaluation process and encourage their participation;
(f)  Share evaluation results with their staff;
(g) Prepare management responses to evaluation recommendations; (h) Implement evaluation action plans;
(i)  Inform EAO when planning for a decentralized evaluation; and
(j)  Use the Evaluation Manual when conducting decentralized evaluations.
 
6.5. The role of the Council in supporting the evaluation function and using evaluation for governance includes to take note of and provide its views on:
 
(a)  the annual EAO work programme that includes the evaluation work programme;
(b) the  evaluation  results,  recommendations  and  the  management  responses  and  action  plan  to evaluation recommendations, when presented as Information Papers to the Council; and
(c)  the annual report of the Evaluation and Audit Advisory Committee in reference to the evaluation function and performance.
 
7.   Dissemination of evaluation reports and follow up to evaluation recommendations
 
7.1. Evaluation draft reports are shared with the primary evaluation stakeholders5. A summary version of the report is then presented to the Council which, if considered required, will refer it to relevant Committees in the form of a Council Working Paper (C-WP). The full evaluation reports, including the executive summaries and the management response to recommendations are disseminated on the Council and EAO websites. EAO’s annual activity reports summarizing the evaluations conducted are published on ICAO’s public website.
 
7.2. The management response to recommendations outlines the decisions by Senior Management concerning each evaluation recommendation and specifically whether they accept, partially accept or reject each recommendation. EAO reserves the right to keep any recommendation that may be rejected by Senior Management  if  a  valid  justification  is  not  provided.  Subsequently,  the  Council  is  informed  of  theManagement  responses  to  the  evaluation  recommendations  and  may  overrule  the  rejection  of  an evaluation recommendation by the Secretariat, if the rejection is deemed unjustified by the Council. For each accepted or partially accepted recommendation,  managers  are required to clearly articulate the aspects of the partially accepted recommendations that are accepted and those that are not, formulate action(s) specifying how the recommendations will be implemented, by whom and the implementation timeline (Evaluation Action Plan). Senior Management ensure that accepted evaluation recommendations are implemented and monitored in their areas of responsibility.
 
8.   Resourcing of evaluation
 
8.1. The Regular Programme Budget allocated to EAO covers staff and non-staff costs, such as consultancy, travel and training costs. External evaluation consultants are hired to support evaluation, as needed, for example when part of the evaluation scope is outside the expertise of the Evaluation Officer.
 
8.2. In addition to the Regular Programme, extra-budgetary activities6  will also be evaluated by EAO. The funding of such evaluations may be covered by earmarked extra-budgetary resources. The Evaluation Manual will provide the indicative level of resources for funding evaluations of extra-budgetary activities.
 

1 In line with the Corporate Performance Management framework for implementing results based management at ICAO.

2 EAO may also evaluate other entities, such as policies, strategies or samples of evaluations with a common theme (meta-evaluations).

3 Centralized (managed by EAO) and decentralized evaluation units (managed by offices other than EAO) may outsource either part or all of the evaluation to external organizations or evaluation consultants.

4 Decentralized evaluations may also be referred to as “self-assessments”, “assessments” or “reviews”.

5 The primary stakeholders include the main internal client of the evaluation and all other internal stakeholders who participated in the evaluation.

6 Activities which are not funded by the Regular Programme Budget

Share this page: