Neutral Assessment of the Evaluation Function

Date : January 2014

PDF Version (325 Kb, 29 Pages)

Table of contents

This report sets out the results from the neutral assessment of the evaluation function at Aboriginal Affairs and Northern Development Canada (AANDC or the Department).

1.0 Background

The evaluation function is one of several senior governance functions at AANDC. Its role is to provide an evidence-based, neutral assessment of the relevance and performance of its programs. Treasury Board of Canada Policy sets out the expectations for a federal department’s or agency’s evaluation function in the Policy on Evaluation, April 1, 2009. One of the Policy’s requirements is that deputy heads ensure that a neutral assessment of their departmental evaluation function is conducted at a minimum of once every five years. This is the first neutral assessment of AANDC’s evaluation function.

Treasury Board (TB) is not prescriptive in terms of the conduct or content of a neutral assessment. To conduct this neutral assessment, AANDC’s Head of EvaluationFootnote 1 (HoE) engaged a senior evaluation consultant who had not performed any evaluation work for AANDC in recent years. The HoE and the consultant agreed that the objectives of the neutral assessment would be to address the following questions:

  1. Does the evaluation function conform to Treasury Board’s expectations?
  2. Is the evaluation organization well-managed?
  3. Is the function adding value to AANDC?

2.0 Scope and Approach

This neutral assessment exercise was conducted over a three month period from November 2013 to January 2014. It provides findings and conclusions about the state of the function at the time of the assessment. Findings and conclusions related to the evaluation function relate to the practices at the department level; findings and conclusions that relate to the evaluation organization relate to the practices within the HoE’s organization – the Evaluation, Performance Measurement and Review Branch (EPMRB).

To conclude on project-level matters, the exercise reviewed documentation from six evaluation projects completed between April 1, 2009 and March 31, 2013. To conclude on organization- and function-level matters, the exercise examined practices in place at the time of the assessment.

The framework used for this neutral assessment includes three distinct components, focusing on:

The framework includes criteria that were derived from policy, standards, and directives associated with the practice of evaluation. International standards on evaluation were also considered in developing the framework. Additional criteria were included that reflected sound management practices that were applicable to an evaluation function. The framework criteria were reviewed and agreed to by the Head of Evaluation. They are included as an appendix.

The neutral assessment was based on an analysis of documentation and interviews with members of AANDC senior management and the departmental evaluation committee.

As requested by the HoE, the analysis was conducted to an audit level of assurance. In addition to his experience in evaluation in the Canadian federal government, the consultant conducting the exercise is a Certified Internal Auditor.

3.0 Conclusions from the Neutral Assessment

The conclusions address the three assessment areas of governance; professional practice; and monitoring and performance reporting. Findings that support these conclusions are provided in Section 4.0 of this report.

The neutral assessment determined that AANDC’s evaluation function:

The findings indicate that AANDC’s evaluation function is meeting expectations. The findings also indicate that the evaluation function is adding value to department, and that the evaluation organization is well-managed.

The neutral assessment identified process improvements related to evaluation planning. There are no other recommended actions required to improve conformance.

The findings that support these conclusions are presented below.

_____________________________

Phil Carr, Certified Internal Auditor

4.0 Findings from the Neutral Assessment

The findings presented in this section inform the conclusions on whether the evaluation function either conforms to expectations, or its conformance to expectations is at risk.

Conformance indicates that processes and practices in place are judged to be in accordance with expectations set out in policy, standards, and directives. Where conformance is at risk, the processes and practices in place have been judged to fall short of expectations, and they negatively impact the evaluation function's capability to perform its responsibilities in a cost-effective manner.

The summary findings are presented below according to the structure of the neutral assessment framework agreed to at the outset of the project.

4.1 Governance of the Evaluation Function

The neutral assessment's governance criteria address values and ethics, as well as the responsibilities of the Deputy Minister with respect to the evaluation function, the departmental evaluation committee, and the HoE.

4.1.1 Deputy Head Responsibilities: Conforms to Expectations

The deputy head is fulfilling his responsibilities related to governance. The Deputy Minister has established an evaluation function, appointed a departmental evaluation committee and an HoE at the EX-03 level, and has approved a five-year Evaluation Plan. The Deputy has approved in a timely manner the reports of completed evaluations, including associated management action plans.

4.1.2 Evaluation Committee Responsibilities: Conforms to Expectations

The departmental evaluation committee is fulfilling the responsibilities set out in the Policy on Evaluation. It reviews and approves evaluation-related matters; it meets policy expectations in terms of its composition; and its stated roles and responsibilities meet TB expectations.

4.1.3 Head of Evaluation Responsibilities: Conforms to Expectations

The Head of Evaluation is fulfilling the HoE responsibilities related to governance. EPMRB conducts its work in a neutral, cost-effective manner; it supports the departmental evaluation committee; and evaluators adhere to the TB evaluation policy suite.

4.1.4 Values and Ethics: Conforms to Expectations

Evaluators are involved in a wide range of activities that require them to act in a professional and ethical manner; for example, they interact with groups all across the Department, they interact with AANDC's outside stakeholders, they initiate and manage contracts, and they have access to privileged information.

Interviews and reviewed documents indicate that EPMRB behaves professionally and in a manner consistent with federal government values and ethics.

4.2 The Evaluation Function's Professional Practice

The neutral assessment's professional practice criteria address: planning and conducting work at the function level; planning, conducting and reporting at the project level; and managing the evaluation function. The summary findings are presented below.

4.2.1 Planning and Conducting at the Function Level

4.2.1.1 Evaluation Planning at the Function Level: Conforms to Expectations

Measures are in place related to the evaluation plan. Documentation indicates that:

  • Adequate measures are in place for the development of the Department's five-year evaluation plan.
  • The five-year evaluation plan is approved and communicated.
  • The most recent evaluation plan presents a five-year strategy for 88 percent coverage over the next five years. The coverage decisions are suitably explained.
  • It would be useful for the plan to identify how evaluations completed in recent years have contributed to coverage. This perspective would provide the Deputy Minister with an understanding of the likelihood of achieving full coverage over a five year period.
4.2.1.2 Roles Related to Evaluation and Performance Measurement: Conforms to Expectations

The assessment analysis determined that the evaluation function is meeting expectations related to conducting evaluations. The evaluation projects address core issues and they count towards coverage requirements. Information on coverage is provided to Treasury Board Secretariat (TBS) as part of the annual Management Accountability Framework (MAF) exercise. The TBS Centre of Excellence for Evaluation (CEE) reviews evaluation project reports and other products, and has indicated to EPMRB that the reports reviewed meet expectations. CEE also monitors the extent to which completed evaluations contribute to coverage. This provides assurance on the quality of evaluations and the extent to which coverage is being achieved.

The assessment analysis determined that the evaluation organization is meeting expectations related to departmental performance measurement. EPMRB is participating actively in a department-wide initiative to improve AANDC's practices and processes related to performance measurement and reporting. As set out in TB expectations, the HoE comments on performance measurement strategies, reviews Cabinet documents, and provides advice on departmental performance measurement strategies and frameworks. A process is in place for the HoE to receive all TB submissions and Cabinet documents for review.

With respect to performance measurement at the function level, deputy heads are to ensure that that ongoing performance measurement is implemented throughout the Department so that sufficient performance information is available to effectively support the evaluation of programs. Reviewed evaluation projects indicated that performance information was available to support the majority of those projects. Also, EPMRB is leading a department-wide Performance Measurement Strategy Action Plan that is intended to, among other objectives, identify and address gaps in the performance information needed to support evaluations.

4.2.2 Planning, Conduct, and Reporting at the Project Level

To address this sub-section of the neutral assessment, the evaluation organization was asked to identify what they considered to be their best completed evaluation projects since the implementation of the 2009 Policy on Evaluation. The following assessment is based on the analysis of documentation provided on six projects: four selected from the best projects identified by EPMRB, and two additional projects chosen at random. The projects were:

  • Impact Evaluation of the ecoENERGY for Aboriginal and Northern Communities
  • Impact Evaluation of Treaty Related Measures in British Columbia
  • Evaluation of the Family Violence Prevention Program
  • Summative Evaluation of the Elementary/Secondary Education Program on Reserve
  • Impact Evaluation of the Urban Aboriginal Strategy
  • Evaluation of the Government of Canada Program for International Polar Year

The neutral assessment analysis at the project level includes: consultation, project planning, conduct, reporting, and use.

4.2.2.1 Consultation on Evaluation Projects: Conforms to Expectations

Consultations took place during the design and implementation of evaluation projects. These consultations involved program personnel in the planning and scoping of evaluations, and in developing methodologies that would be used in the evaluation research.

4.2.2.2 Evaluation Project Planning : Generally Conforms to Expectations

According to the TB Standard on Evaluation for the Government of Canada, plans should discuss approaches, methodologies, roles and responsibilities, and project costing.

The analysis of project information determined that formal project plans (e.g., evaluation frameworks) were done for every project, and that important information on approaches and methodologies is included in these plans. However, information on project budgeting and team roles and responsibilities could be provided more fully and consistently in project plans.

4.2.2.3 Conduct of Evaluation Projects: Conforms to Expectations

The conduct of evaluation projects includes project leadership, stakeholder consultation, project management and control, use of evidence, and information control.

Evaluation projects are led by AANDC's HoE, and stakeholders are consulted as part of the research methodologies. Evaluation reports indicate that conclusions are based on evidence.

Practices are in place related to project management and control, and information control.

4.2.2.4 Evaluation Project Reporting: Conforms to Expectations

Project reporting includes the completeness and communication of evaluation reports, and their accessibility to the public.

The reports reviewed are clear, meet the requirements of the Standard on Evaluation, include management responses and action plans, and have been submitted to TBS.

Completed evaluation reports are posted on AANDC's website.

4.2.2.5 Use of Evaluations: Conforms to Expectations

This requirement includes use of evaluation findings by the deputy head and the Department, and availability of evaluation information to key stakeholders such as the Minister, Parliament, and Canadians.

Completed evaluation studies are posted on AANDC's website and, therefore, available to all.

Interviews indicated that the Deputy Minister and many Assistant Deputy Ministers (ADMs) have used evaluation information in program decision-making. Some ADMs could cite specific examples of their use of evaluation findings. Further, ADMs who were less familiar with evaluations also indicated that they believe the function adds significant value to the Department. Some senior managers indicated that the evaluation analyses "could have gone further", suggesting that they value evaluation, but still had unmet analytical needs, which they were pursuing within their own organizations.

The ADMs who had worked with the HoE emphasized the considerable value added that she brought to the evaluation function. They highlighted that their positive relationship with the HoE contributed significantly to the usefulness of evaluation studies.

Overall, interviewees indicated that they value the evaluation function (including those who are still learning about it) and anticipate that evaluation studies will be useful to them.

4.2.3 Evaluation Competencies: Conforms to Expectations

The evaluation projects examined were conducted by EPMRB evaluation personnel and consulting firms that had demonstrated their competencies in the area of evaluation. Therefore, projects were undertaken by competent personnel.

4.2.4 Improvement to Products and Services: Conforms to Expectations

The reviewed projects indicate that the EPMRB has well-established management practices. Interviews with ADMs indicated that EPMRB representatives have met with them to consult about the evaluation plan, to discuss the role of evaluation, and discuss areas where they could add value to departmental senior managers and their organizations.

4.3 Monitoring and Performance Reporting of the Evaluation Function

The neutral assessment's Monitoring and Performance Reporting criteria address annual and ongoing reporting obligations.

4.3.1 Annual Report on the State of Performance Measurement: Conforms to Expectations

These reports have been produced for the 2009/10, 2010/11 and 2012/13 fiscal years.

4.3.2 Monitoring and Performance Reporting by the Deputy Heads: Conforms to Expectations

The Deputy Minister has indicated that he has been well served by the evaluation function, and that it has considerable influence in the Department. This neutral assessment is a component of the performance reporting activity. Also, the HoE provides TBS with information annually on behalf of the deputy head.

5.0 Recommended Actions

The neutral assessment identified process improvements related to evaluation planning. There are no other recommended actions required to improve conformance.

Interviewees indicated that their most significant current challenge is for their organizations to find the time necessary to support an evaluation study. They indicated that, with recent resource reductions, it has become even more important for evaluation studies to focus on addressing the highest value-added topics and issues, to obtain the highest return on the time and effort invested in an evaluation.

Further, interviews indicated that evaluation analyses would be even more comprehensive and useful if they relied on a broader diversity of information sources. As an example, AANDC’s longstanding evaluation practice of relying heavily on case studies could be complemented by in-depth literature reviews. These and other methods would help to enhance the discussion of the technical and conceptual issues that affect evaluation findings and conclusions.

Appendix

Aboriginal Affairs and Northern Development Canada

Framework for the Neutral Assessment of the Evaluation Function

Objectives of the Neutral Assessment

To address the following questions:

  1. Does the evaluation function conform to Treasury Board’s expectations?
  2. Is the evaluation organization well-managed?
  3. Is the function adding value to AANDC?
A Governance

There is conformance with the TB Policy on Evaluation requirements regarding Governance
A.1 Values and Ethics

There is conformance with the TB Standard on Evaluation for the Government of Canada requirements regarding Values and Ethics
A.1.a The head of evaluation ensures that evaluations are conducted in a professional and ethical manner
1 Evaluators behave in a manner that is consistent with the "Values and Ethics Code for the Public Service." S 6.3.1.1
2 Any contract by which an external evaluator (i.e. consultant) is engaged to provide evaluation services is to include a requirement that the "Values and Ethics Code for the Public Service" be complied with. S 6.3.1.1
3 Individuals performing evaluation work act in a neutral manner and with integrity in their relationships with stakeholders. S 6.3.1.2
4 Any conflict of interest is dealt with openly, honestly, and promptly so that it does not compromise evaluation processes or results. S 6.3.1.3
A.2 Deputy Head

There is conformance with the TB Policy on Evaluation requirements regarding the responsibilities of the Deputy Head
A.2.a The deputy head has fulfilled his/her responsibilities in regard to the evaluation function
1 Deputy heads are responsible for establishing a robust, neutral evaluation function in their department and for ensuring that their department adheres to this policy and to its supporting directive and standard. P 6.1
A.2.b The deputy head has fulfilled his/her responsibilities in regard to the Departmental Evaluation Committee
1 Deputy heads ensure that a committee of senior departmental officials (referred to as the Departmental Evaluation Committee) is assigned the responsibility for advising the deputy head on all evaluation and evaluation-related activities of the Department. P 6.1.3
2 Deputy heads ensure that the departmental evaluation committee has full access to information and documentation needed or requested to fulfill its responsibilities. P 6.1.9
A.2.c The deputy head has fulfilled his/her responsibilities in regard to the head of evaluation
1 Deputy heads designate a head of evaluation at an appropriate level as the lead for the evaluation function in the Department. P 6.1.1
2 Heads of evaluation are required to meet competency requirements, as specified by the Treasury Board of Canada Secretariat. D 6.1.1.1
3 Deputy heads ensure that the head of evaluation has direct and unencumbered access to the deputy head, as required. P 6.1.2
4 Deputy heads ensure that (the departmental evaluation committee and – addressed in A2b2) the head of evaluation have full access to information and documentation needed or requested to fulfill their responsibilities. P 6.1.9
A.2.d The deputy head has fulfilled his/her responsibilities in regard to the operations of the evaluation function
1 Deputy heads approve for annual submission to the Treasury Board of Canada Secretariat, a rolling five-year departmental evaluation plan. P 6.1.7
2 Deputy heads confirm that the departmental evaluation plan:
  1. Aligns with and supports the departmental Management, Resources and Results Structure;
  2. Supports the requirements of the Expenditure Management System, including strategic reviews; and,
  3. Includes all ongoing programs of grants and contributions as required by Section 42.1 of the Financial Administration Act.
P 6.1.7
3 Deputy heads approve evaluation reports, management responses and action plans in a timely manner. P 6.1.4
A.3 Departmental Evaluation Committee

There is conformance with the TB Policy on Evaluation requirements regarding the departmental evaluation committee
A.3.a The departmental evaluation committee has fulfilled its structural responsibilities
1 Deputy head ensures that the departmental evaluation committee is:
  1. Chaired by the deputy head or a senior level designate
  2. Supported by the head of evaluation on evaluation matters
  3. Structured with specific roles and responsibilities, as per Annex B
P 6.1.3
A.3.b The departmental evaluation committee has fulfilled its role and responsibilities
1 Role of the Departmental Evaluation Committee

This committee serves as an advisory body to the deputy head related to the departmental evaluation plan, resourcing, and final evaluation reports and may also serve as the decision-making body on other evaluation and evaluation-related activities of the Department
P Annex B
2 Responsibilities of the Departmental Evaluation Committee
  1. Reviews the adequacy of evaluation coverage, as expressed in a departmental evaluation plan developed by the head of evaluation, as well as the risk-based approach used for determining the evaluation approach and level of effort to be applied to the individual evaluations comprised in the plan, and recommends the plan for approval by the deputy head;
  2. If requested by the head of evaluation:
    1. Reviews and recommends approval of key elements of an evaluation, such as the terms of reference, and
    2. Considers and responds to key evaluation-related issues;
  3. Reviews final evaluation reports, including management responses and action plans, and recommends approval to the deputy head;
  4. Ensures follow-up to action plans approved by the deputy head;
  5. Reviews the adequacy of resources allocated to the evaluation function and recommends to the deputy head an adequate level of resources consistent with the departmental evaluation plan;
  6. Reviews the adequacy of resources allocated to performance measurement activities as they relate to evaluation, and recommends to the deputy head an adequate level of resources for these activities; and,
  7. Reviews the performance of the evaluation function, and recommends action to address any weaknesses.
P Annex B
A.4 Head of Evaluation

There is conformance with the TB Policy on Evaluation requirements regarding the head of evaluation
A.4.a The head of evaluation has fulfilled his/her responsibilities in regard to governance
1 The head of evaluation is responsible for managing an efficient and effective evaluation unit. D 6.1.2.1
2 The head of evaluation ensures that the evaluation unit conducts evaluations in a neutral, cost-effective manner. D 6.1.2.1
3 The head of evaluation is responsible for supporting the Departmental Evaluation Committee, that is assigned the responsibility for guiding and overseeing the evaluation function. D 6.1.2.4
4 The head of evaluation is responsible for ensuring that evaluation work and persons involved in designing, conducting and managing evaluations are informed of and adhere to the Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada. D 6.1.2.2
B Professional Practice

There is conformance with the TB Policy on evaluation requirements regarding Professional Practice
B.1 Departmental Evaluation Plan

There is conformance with the TB Policy on Evaluation requirements regarding the Departmental Evaluation Plan
B.1.1 Evaluation Plan

There is conformance with the TB Policy on evaluation requirements regarding the evaluation plan
B.1.1.a Adequate measures are in place for the development of the evaluation plan pending the Department’s full implementation of the Policy on Evaluation
1 Pending the full implementation of Section 6.1.8 (a)Footnote 2, which can be no later than March 31, 2013, deputy heads will ensure that approved departmental evaluation plans demonstrate progress towards achieving coverage of all departmental direct program spending (excluding ongoing programs of grants and contributions) over five years P 1.3.1.a
2 Pending the full implementation of Section 6.1.8 (a), which can be no later than March 31, 2013, deputy heads will ensure that departmental evaluation plans that they approve for submission to the Treasury Board of Canada Secretariat as per Section 6.1.7Footnote 3 of this policy, and that do not demonstrate evaluation coverage of all direct program spending over the ensuing five-year period, use a risk-based approach to planning coverage of direct program spending (excluding ongoing programs of grants and contributions.) P 1.3.1.b
3 Pending the full implementation of Section 6.1.8 (a), which can be no later than March 31, 2013, deputy heads will ensure that … Departmental evaluation plans that use a risk-based approach to planning coverage of direct program spending will include a written, risk-based rationale to explain the Department's evaluation coverage and non-coverage choices. P 1.3.1.b
B.1.1.b Adequate measures are in place for the development of the evaluation plan
1 The Deputy head ensures that the following evaluation coverage requirements are met and reflected in the departmental evaluation plan:
  1. All direct program spending, excluding grants and contributions, is evaluated every five years;
  2. All ongoing programs of grants and contributions are evaluated every five years, as required by Section 42.1 of the Financial Administration Act;
  3. The administrative aspect of major statutory spending is evaluated every five years;
  4. Programs that are set to terminate automatically over a specified period of time, if requested by the Secretary of the Treasury Board following consultation with the affected deputy head; and
  5. Specific evaluations, if requested by the Secretary of the Treasury Board following consultation with the affected deputy head.
P 6.1.8
2 The head of evaluation is responsible for developing a rolling five-year departmental evaluation plan (hereafter referred to as the departmental evaluation plan), and updating the plan annually. D 6.1.3.a
3 The head of evaluation is responsible for ensuring that their departmental evaluation plans:
  1. Align with and support the departmental Management, Resources and Results Structure;
  2. Support the requirements of the Expenditure Management System, including strategic reviews;
  3. Include all direct program spending, excluding grants and contributions;
  4. Include all ongoing grant and contribution programs for which their department is responsible, as required under Section 42.1 of the Financial Administration Act;
  5. Include the administrative aspect of all major statutory spending;
  6. Include programs that are set to terminate automatically after a specified period of time, if requested by the Secretary of the Treasury Board following consultation with the affected deputy head; and,
  7. Include specific evaluations, if requested by the Secretary of the Treasury Board following consultation with the affected deputy head.
D 6.1.3.b
4 The head of evaluation is responsible for identifying and recommending to the deputy head and the Departmental Evaluation Committee a risk-based approach for determining the evaluation approach and level of effort to be applied to individual evaluations comprised in the five-year departmental evaluation plan, and the appropriate level of resources required to conduct individual evaluations included in the plan. D 6.1.3.c
B.1.1.c The evaluation plan is approved and communicated
1 The head of evaluation is responsible for: submitting the departmental evaluation plan, as approved by the deputy head, to the Treasury Board of Canada Secretariat at the beginning of each fiscal year. D 6.1.3.d
2 The head of evaluation is responsible for submitting the … confirmation from the deputy head as required by Section 6.1.7 of the Policy on Evaluation. D 6.1.3.d
B.2 Nature of Evaluation

There is conformance with the TB Policy on Evaluation requirements regarding the Nature of Evaluation
B.2.1 Evaluation Roles and Responsibilities

There is conformance with the TB Policy on Evaluation requirements regarding the evaluation function’s role and responsibilities
B.2.1.a The evaluation function has fulfilled its responsibilities in regard to conducting evaluations
1 The head of evaluation is responsible for: implementing the approved departmental evaluation plan while ensuring the timely completion of individual evaluations, in accordance with the Standard on Evaluation for the Government of Canada D 6.1.3.e
2 To address value for money, evaluations will be required to assess all core issues identified below:
  • Continued Need for Program
  • Alignment with Government Priorities
  • Alignment with Federal Roles and Responsibilities
  • Achievement of Expected Outcomes
  • Demonstration of Efficiency and Economy
D Annex A
3 The head of evaluation is responsible for ensuring that all evaluations that are intended to count toward the coverage requirements of subsections "a", "b", or "c" of Section 6.1.8 of the Policy on Evaluation, include clear and valid conclusions about the relevance and performance of programs by addressing all core issues outlined in Annex A. D 6.1.3.f
B.2.1.b The evaluation function has fulfilled its responsibilities in regard to departmental performance measurement
1 The deputy head ensures that the head of evaluation is consulted on the performance measurement strategies embedded in the Department's Management, Resources and Results Structure. P 6.1.11
2 The head of evaluation is responsible for reviewing and providing advice on the performance measurement strategies for all new and ongoing direct program spending, including all ongoing programs of grants and contributions, to ensure that they effectively support an evaluation of relevance and performance. D 6.1.4
3 The head of evaluation is responsible for reviewing and providing advice on the accountability and performance provisions to be included in Cabinet documents (Memoranda to Cabinet, Treasury Board submissions). D 6.1.4
4 The head of evaluation is responsible for reviewing and providing advice on the performance measurement framework embedded in the organization's Management, Resources and Results Structure. D 6.1.4
B.3 Evaluation Project Process

There is conformance with the TB Policy on Evaluation requirements regarding the Evaluation project process
B.3.1 Managing the Evaluation Function

There is conformance with the TB Policy on Evaluation requirements regarding managing the evaluation function
B.3.1.a Appropriate consultation is taking place
1 In managing the evaluation function in departments, the head of evaluation consults appropriately with program managers, stakeholders, and peer review or advisory committees during evaluation project design and implementation. D 6.2.1
2 Peer review, advisory, or steering committee groups are used where appropriate to input to evaluation planning and processes and to the review of evaluation products to improve their quality. The head of evaluation or the evaluation manager directs these committees. S 6.1.2.2
B.3.1.b Adherence to the Policy, Directive and Standard
1 The head of evaluation is responsible for ensuring that evaluation work and persons involved in designing, conducting and managing evaluations are informed of and adhere to the Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada. D 6.1.2.2
B.3.2 Evaluation Project Planning Phase

There is conformance with the TB Policy on Evaluation requirements regarding the Planning Phase
B.3.2.a Evaluation project plans are developed and recorded
1 Evaluation planning and design requirements ensure that the basic elements of a good evaluation are in place at the outset of an evaluation. S 6.2
Evaluation procedures should be practical and responsive to the way the program operates. JCSEE F2Footnote 4
2 Evaluations are cost-effective and risk-based:
  1. Purpose and objectives are clearly stated at the outset of the evaluation project.
  2. Risks associated with the program being evaluated are clearly articulated at the outset of the evaluation.
  3. The complexity of the evaluation (design, methods and associated costs) is linked with the risks associated with the program being evaluated, the information needs of the deputy head, and the availability of reliable data and information.
S 6.2.1
3 Standard on Measurement and Analysis
  1. The evaluation approach, and associated design and related methods, are identified during the planning stage of the evaluation.
  2. The evaluation methods and associated limitations are clearly described and documented so that the data collected can be reliably and adequately assessed.
  3. Multiple lines of evidence, including evidence produced from both quantitative and qualitative analysis, are used to ensure reliable and sufficient evidence.
S 6.2.2
Evaluations should employ technically adequate designs and analyses that are appropriate for the evaluation purposes. JCSEE A6
4 Roles and responsibilities of project team members involved in specific evaluations are articulated in writing and agreed upon at the outset of the evaluation. S 6.1.1.2
5 In planning and conducting the evaluation, those involved in the evaluation comply with government policies related to information collection, use, preservation and dissemination. S 6.3.4.1
Evaluations should employ systematic information collection, review, verification, and storage methods. JCSEE A5
B.3.2.b Evaluation projects conform with the Policy on Government of Canada Communications
1 The head of evaluation is responsible for consulting with the departmental head of communications on evaluation work that could potentially be considered "public opinion research" as defined in the Policy on Government of Canada Communications. D 6.1.2.3
B.3.3 Conduct of Evaluation Project Phase

There is conformance with the TB Policy on Evaluation requirements regarding the conduct of evaluations
B.3.3.a Leading evaluation projects
1 The head of evaluation is responsible for directing evaluation projects. S 6.1.1.1
B.3.3.b Stakeholder involvement in evaluation projects
1 Evaluations incorporate sufficient and appropriate consultation, including with major stakeholders, and, where appropriate, apply the advice and guidance of specialists and other knowledgeable persons. S 6.1.2
B.3.3.c Evaluation projects are managed and controlled
Sound fiscal decision-making is employed in the management of evaluation projects so that expenditures are accounted for, and the Government of Canada receives good value for its money. S 6.3.3.1
Evaluations should use effective project management strategies. JCSEE F1
Evaluations should use resources effectively and efficiently. JCSEE F4
Evaluations should account for all expended resources and comply with sound fiscal procedures and processes. JCSEE P7
When engaging external resources, evaluation projects comply with Treasury Board contracting policies. S 6.3.3.2
When engaging external resources, contracts by which an external evaluator is engaged to provide evaluation services include a requirement to comply with this standard. S 6.3.3.3
B.3.3.d There is sufficient and appropriate evidence to achieve the evaluation project objectives and to support conclusions and engagement results
1 The deputy head ensures that ongoing performance measurement is implemented throughout the Department so that sufficient performance information is available to effectively support the evaluation of programs. P 6.1.10
2 Evaluators must identify, analyze, evaluate, and document sufficient information to achieve the project’s objectives. Based on
IIAFootnote 5 2300
3 Evaluators must identify sufficient, reliable, relevant, and useful information to achieve the engagement’s objectives. Based on
IIA 2310
4 Evaluators must base conclusions and project results on appropriate analyses. Based on
IIA 2320
B.3.3.e Relevant information is documented, maintained and controlled
1 Stakeholders and participants in an evaluation are informed of the level of confidentiality and privacy that is afforded them under the Privacy Act. S 6.3.4.2
2 Evaluators must document relevant information to support the conclusions and engagement results. Based on
IIA 2330
3 The Head of Evaluation must develop retention requirements for engagement records, regardless of the medium in which each record is stored. These retention requirements must be consistent with the organization’s guidelines and any pertinent regulatory or other requirements. Based on
IIA 2330.A2
B.3.4 Reporting Phase

There is conformance with the TB Policy on Evaluation requirements regarding the Reporting Phase
B.3.4.a Evaluation reports are complete and appropriately communicated
1 The head of evaluation is responsible for issuing evaluation reports (and other evaluation products, as appropriate) directly to the deputy head and the Departmental Evaluation Committee in a timely manner. D 1.2.5.6
2 Evaluation reporting requirements ensure that reports present the findings, conclusions and recommendations in a clear and neutral manner. S 6.4
Evaluations should document programs and their contexts with appropriate detail and scope for the evaluation purposes. JCSEE A4
3 Evaluation reports are written so that senior managers and external readers can readily focus on and understand the important issues being reported. Evaluation reports:
  1. Are concise and clearly written;
  2. Include only information that is needed for a proper understanding of the findings, conclusions and recommendations;
  3. Include an accurate assessment of the results achieved by the program being evaluated as well as clear conclusions on their relevance and performance;
  4. Present the conclusions and recommendations so that they flow logically from evaluation findings;
  5. Clearly expose the constraints of the evaluation with respect to the design, methods and consultations;
  6. Satisfy, where applicable, Cabinet, Treasury Board Submission or external reporting requirements;
  7. Provide the reader with appropriate context by describing the objectives and timing of the work, the program evaluated, how it fits into the overall operations of the organization, and its rationale;
  8. Contain clear and actionable recommendations and timing for the action plan; and,
  9. Provide relevant analysis and explanation of the exposure to risks for any significant problems identified.
S 6.4.1
4 The head of evaluation is required to submit complete evaluation reports, management responses and action plans (in electronic format) to the Treasury Board of Canada Secretariat immediately upon approval of these reports by the deputy head. D 6.1.5.1
B.3.4.b Evaluation reports are accessible to the public
1 The deputy head ensures that complete, approved evaluation reports along with management responses and action plans are made easily available to Canadians in a timely manner while ensuring that the sharing of reports respects the Access to Information Act, Privacy Act, and the Government Security Policy. P 6.1.6D
The head of evaluation is required to: make approved evaluation reports, along with the management responses and action plans, available to the public with minimal formality and post them on departmental websites in both official languages in a timely manner following their approval by the deputy head. 6.1.5.2
The deputy head ensures that evaluation reports posted on departmental websites respect the Access to Information Act, Privacy Act, and the Government Security Policy. D 6.1.5.3
B.3.5 Use of Evaluations

There is conformance with the TB Policy on Evaluation requirements regarding the use of evaluation information
B.3.5.a Deputy heads use evaluation results
1 The deputy head uses evaluation findings to inform program, policy, resource allocation and reallocation decisions. D 6.1.1.7
2 Impacts of evaluations, including savings or improvements produced, are reported to the deputy head. S 6.1.3.1
B.3.5.b Evaluation information is available for use by others
1 Evaluation information on the ongoing relevance and performance of direct program spending is available to ministers, departments and central agencies and used to support evidence-based decision-making on policy, expenditure management and program improvements. S 6.4.2.1
2 Evaluation information on the ongoing relevance and performance of direct program spending is available to Parliament and Canadians to support government accountability for results achieved by policies and programs. S 6.4.2.2
B.4 Evaluation Competencies

There is conformance with the TB Policy on Evaluation requirements regarding the proficiency and professional care
B.4.a The head of evaluation ensures that evaluation resources have the training, competencies and experience to perform their responsibilities
1 The head of evaluation is required to ensure that the person or persons involved in designing, conducting or managing evaluations has, or collectively have the appropriate training, required competencies and experience in evaluation. D 6.1.1.2
The person or persons carrying out the evaluation or evaluation-related work possesses or collectively possess the knowledge and competence to carry out the evaluation. S 6.3.2.1
Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context. JCSEE U1
B.5 Quality Assurance and Improvement

There is conformance with the TB Policy on Evaluation requirements regarding Quality Assurance and Improvement
B.5.a The head of evaluation ensures the development and maintenance of the quality assurance and improvement program
1 Ongoing, systematic inquiry is employed to identify future improvements to evaluation products and services. D 6.1.3.2
HoEs are responsible for developing and maintaining a quality assurance and improvement program that covers all aspects of the Evaluation function and continuously monitors its effectiveness. Based on
IIA 1300
Periodic reviews performed through self-assessment or by other persons within the organization with sufficient knowledge of evaluation practices. Based on
IIA 1311
Evaluators should use these and other applicable standards to examine the accountability of the evaluation design, procedures employed, information collected, and outcomes. JCSEE E2
C Monitoring and Performance Reporting

There is conformance with the TB Policy on Evaluation requirements regarding Monitoring and Performance Reporting
C.1 Head of Evaluation’s Annual Report
C.1.a The head of evaluation has submitted his/her annual reports to the departmental evaluation committee
1 The head of evaluation is responsible for submitting to the Departmental Evaluation Committee an annual report on the state of performance measurement of programs in support of evaluation. D 6.1.4.4
C2 Deputy Head

There is conformance with the TB Policy on Evaluation requirements regarding Monitoring and Performance Reporting
C.2.a The deputy head has fulfilled his/her responsibilities in regard to monitoring and performance reporting
1 The deputy head is responsible for monitoring compliance with this policy in their departments to ensure its effective implementation. P 7.1
2 The deputy head is responsible for ensuring that a neutral assessment of their departmental evaluation function is conducted at a minimum of once every five years. P 7.1
3 The deputy head is responsible for addressing issues that arise regarding compliance with this policy, and with its associated directive and standard, and ensuring that appropriate remedial actions are taken to address these issues. P 7.2
C.2.b The deputy head has the information to respond to the Secretary of the Treasury Board
1 The Secretary of the Treasury Board … may require the deputy head to provide information on the following:
  1. The use of evaluation in support of the requirements of the Expenditure Management System, including the extent to which evaluation information is used within the Department:
    1. to manage for results and inform resource reallocation decisions;
    2. to underpin proposals to Cabinet, including Treasury Board submissions and memoranda to Cabinet;
    3. to provide evidence for use in strategic reviews of program spending;
    4. to support accountability to Parliament, including through public reporting.
  2. The quality and timeliness of evaluations;
  3. Actual evaluation coverage of direct program spending and the administrative aspect of major statutory spending;
  4. Actual evaluation coverage of ongoing programs of grants and contributions; and,
  5. The capacity and competence of evaluation units.
P 7.3
Date modified: