Evaluation of the First Nations Water and Wastewater Action Plan

Final Report
Date : September 2013
Revised November 2013
Project Number: 1570-7/12026

PDF Version (289 Kb, 64 Pages)

Table of contents

List of Acronyms

AANDC
Aboriginal Affairs and Northern Development Canada
CBWM
Community-based Drinking Water Quality Monitor
CWQI
Canadian Water Quality Index
DWA
Drinking Water Advisory
EPB
Employee Benefit Program
EPMRB
Evaluation, Performance Measurement and Review Branch
EPMRC
Evaluation, Performance Measurement and Review Committee
FNWWAP
First Nations Water and Wastewater Action Plan
O&M
Operating and Maintenance

Executive Summary

The Evaluation, Performance Measurement and Review Branch at Aboriginal Affairs and Northern Development Canada (AANDC) and the Evaluation Directorate at Health Canada-Public Health Agency of Canada undertook a horizontal evaluation of the First Nations Water and Wastewater Action Plan (FNWWAP). The FNWWAP supports AANDC's Strategic Outcome, The Land and Economy: full participation of First Nations, Inuit and Métis individuals and communities in the economy. It also supports Health Canada's Strategic Outcome of First Nations and Inuit Health Programming and Services: better health outcomes and reduction of health inequalities between First Nations and Inuit and other Canadians.

From 2008 to 2012, the Government of Canada invested nearly $556 million in the FNWWAP, and a total of nearly $1.4 billion on all water and wastewater activities. The objective of the FNWWAP, according to the program's originating documents, is to support First Nation communities on reserve in bringing their drinking water and wastewater services to a level and quality of service comparable to those enjoyed by other Canadians living in communities of similar size and location. The FNWWAP was approved from 2008 to 2010 and renewed for the period from 2010 to 2012 in order to:

The current evaluation examined the relevance and performance of the FNWWAP from 2008-09 to 2011-12. While the authorities for FNWWAP have been renewed and are slated to continue to March 2014, this evaluation is intended to inform program design elements in the interim, and inform decisions on program renewal prior to the 2014 expiry. The evaluation included a series of key-informant interviews and case studies (conducted by the Institute on Governance), a survey of First Nation water and wastewater operators (conducted by Harris-Decima), and a comprehensive internal review of literature, documentation and databases.

The evaluation has made ten key findings. With respect to relevance:

  1. While there is a demonstrable need for continued investment in water and wastewater initiatives to achieve major improvements in water and infrastructure quality and risk reduction, short-term action plans may not address the more pervasive issues and a shift to longer-term planning is needed;
  2. The outcomes of the FNWWAP are clearly aligned with government priorities and strategic objectives; and
  3. The specific roles and responsibilities of AANDC are seen as ambiguous and somewhat contradictory with regard to accountability and ownership. While its role as funder is seen as appropriate, there are mixed opinions on the appropriate degree of oversight, which may be partially attributable to a low level of awareness amongst First Nations of ownership responsibilities.

    With respect to performance:
  4. The capacity of First Nation communities to address potential water quality problems has shown considerable improvements since the introduction of the FNWWAP, particularly with respect to numbers of community-based drinking water quality monitors. However, significant limitations remain, primarily respecting the proportions of communities completing water quality testing, and the numbers of qualified water system operators;
  5. While the ability to detect, monitor and react to health risks has shown improvement since the introduction of FNWWAP, there is no evidence to broadly suggest that risks specifically associated with the quality and supply have decreased, partially due to performance measures having not been adequately defined. Where reductions in risk were noted, they were largely attributed to infrastructure upgrades and the presence of trained operators;
  6. It is too soon to tell whether ongoing investments in water and wastewater systems, along with the ability of the federal government to enact Regulations stemming from the Safe Drinking Water for First Nations Act, will result in community systems meeting federal standards. There is concern that focussing on enforceable standards without ensuring First Nations have adequate infrastructure and capacity to meet those standards may overlook the core limitations facing First Nations water and wastewater systems;
  7. There have been noticeable but relatively modest improvements in confidence in drinking water in First Nation communities since the introduction of FNWWAP;
  8. There is a need for a longer-term strategy for water and wastewater issues that increases specific emphasis on recruitment, training and retention of system operators; and that is flexible in customising its approach to individual community needs;
  9. While reductions in health risks are covered by Health Canada activities and measures for system risk and capacity have been operationalised at AANDC, measures of increased confidence and comparability with other Canadian municipalities have not been articulated or operationalised; and
  10. The operating expenditures for the FNWWAP and water and wastewater in general comprise a small proportion of the total, indicating that the program is operating efficiently from an internal operations point of view. Without a thorough understanding of the value and longevity of infrastructure projects funded, however, it is not possible to measure efficiency of the vast majority of the program's expenditures. There are indications that investments in new technologies may lead to increased efficiency, particularly among smaller communities.

The evaluation thus recommends that AANDC:

  1. Work with First Nations and Health Canada to develop a long-term strategy for investments in water and wastewater infrastructure and maintenance in order to address the pervasive and longstanding issues of water and infrastructure quality and maintenance;
  2. Ensure that the Regulations ensuing from the Safe Drinking Water for First Nations Act are developed with the engagement of First Nations, and that roles and responsibilities, both current and subsequent to the ensuing Regulations, are clearly understood and communicated;
  3. Engage First Nations to develop a concrete plan to address issues of recruitment, retention and capacity development of trained and skilled operators;
  4. Engage First Nations to facilitate their readiness to comply with Regulations ensuing from the Safe Drinking Water for First Nations Act;
  5. Plan future investments in infrastructure with an equal emphasis on investing in the capacity to operate and maintain new and existing systems long term, including program flexibility to outsource water and wastewater servicing where community-level capacity is not practical; and
  6. Work with Health Canada to develop a robust Performance Measurement methodology that allows for the reliable periodic reporting of the stated outcomes of the FNWWAP, including efficiency and comparability.

Health Canada has further recommended that it:

  1. Continue to work with First Nations to build the capacity to monitor drinking water quality on reserve in order to increase the proportion of communities completing water quality testing according to the Guidelines for Canadian Drinking Water Quality.

Management Response / Action Plan

The First Nations Water and Wastewater Action Plan (FNWWAP) was introduced in 2008 and provided $330 million in water and wastewater funding over two years for treatment facility construction and renovation, operation and maintenance of facilities, training of operators and related public health activities on reserve. Budgets 2010 and 2012 saw the renewal of FNWWAP at the same levels over the same period of time.

Through the FNWWAP, the Government of Canada is improving the health and quality of life of residents of First Nation communities by assisting First Nations to provide better water and wastewater services. AANDC and Health Canada are jointly responsible for FNWWAP.

The Government of Canada believes that First Nations are entitled to the same health and safety protections for drinking water as other Canadians. FNWWAP investments endorsed this principle through facilitating the provision of safe drinking water, the effective treatment of wastewater, and the protection of sources of drinking water on First Nation lands. Further, the Safe Drinking Water for First Nations Act (the Act), which received Royal Assent on June 19, 2013, and came into force on November 1, 2013, allows the federal government to develop a regulatory regime to this end. Regulations to be developed under the Act will use existing and relevant provincial and territorial regulations as a base, with adaptations to address realities on First Nation lands.

Regulations will be developed on a regional basis with First Nations and other stakeholders. Regulations will be phased in to allow the federal government and First Nations time to bring infrastructure and capacity to the level required to meet these regulations.

FNWWAP has enabled AANDC to provide funding to First Nations to support the acquisition, design, construction, operation, and maintenance of on-reserve water and wastewater systems, and for the training and certification of water and wastewater system operators. It has also enabled Health Canada to assist First Nations communities south of 60° to monitor drinking water quality, to provide public health advice to First Nations on reserve, as well as to fund and train Community-based Drinking Water Quality Monitors.

AANDC and Health Canada are generally in agreement with the recommendations identified through this evaluation. The Action Plan below presents how these recommendations are being, or have already been, addressed by AANDC and Health Canada. The key findings of this evaluation support the long-term strategy to improve on-reserve water and wastewater, based on a four pillar approach: 1) enhanced capacity building and operator training; 2) enforceable standards and protocols; 3) infrastructure investments; and, 4) protection of public health.

Both AANDC and Health Canada are committed to addressing the recommendations found within the evaluation as set out in the Action Plan on the following pages.

Recommendations Actions Responsible Manager (Title / Sector) Planned Implementation and Completion Dates
1. Work with First Nations and Health Canada to develop a long-term strategy for investments in water and wastewater  in order to address pervasive and longstanding issues with water quality and quality and maintenance of infrastructure on reserve; In response to the 2009-2011 National Assessment of First Nations Water and Wastewater Systems, AANDC worked with First Nations and other stakeholders to develop a long-term strategy to improve on-reserve water and wastewater, resulting in a three pillar approach: 1) enhanced capacity building and operator training; 2) enforceable standards and protocols; and, 3) infrastructure investments. These priorities will continue under any future strategyFootnote 1.

As FNWWAP ends on March 31, 2014, AANDC and Health Canada are actively engaged in exploring options for investments in water and wastewater to guide activities beyond FNWWAP.
Senior Assistant Deputy Minister of Regional Operations
 
 
Manager, Drinking Water Program, Environmental Public Health Division, Interprofessional Advisory and Program Support Directorate, First Nations and Inuit Health Branch, Health Canada
Start Date:
May 2013

Completion:
March 31, 2014
 
 
Completion:
March 2014
2. Ensure that the Regulations ensuing from the Safe Drinking Water for First Nations Act are developed with the engagement of First Nations, and that roles and responsibilities, both current and subsequent to the ensuing Regulations, are clearly understood and communicated; The Department is committed to working with First Nation organizations, provincial, and territorial governments to develop regulations, region by region, to support communities in bringing drinking water and wastewater systems into regulatory compliance. Support will be provided to First Nation organizations to develop regulatory proposals based on existing provincial and territorial regulations, with adaptations as appropriate. Senior Assistant Deputy Minister of Regional Operations Start Date:
Following Royal Assent June 19, 2013

Develop all regulations by fall 2015.
3. Engage First Nations to develop a concrete plan to address issues of recruitment, retention and capacity development of trained and skilled operators; As part of the response to the 2009-2011 National Assessment of First Nations Water and Wastewater Systems, AANDC developed a strategy with First Nations to reduce system risks. The top priority of that ongoing strategy is enhanced capacity building and operator training. AANDC agreed that trained and certified operators are key to reducing risk and helping to ensure safe drinking water in First Nation communities. To support First Nations in developing and retaining the capacity to operate and maintain water and wastewater systems, AANDC will provide funding for operator training and the Circuit Rider Training Program. AANDC will work with First Nations and First Nations technical organizations to identify barriers to develop measures to improve recruitment, retention, and capacity development of operators. Senior Assistant Deputy Minister of Regional Operations Start Date:
April 1, 2012
Plan for measures to improve recruitment, retention, and capacity development created by March 31, 2014.
4. Engage First Nations to facilitate their readiness to comply with Regulations ensuing from the Safe Drinking Water for First Nations Act; The Department will work with First Nations to develop regulations region by region. Phased regulation development will provide time for the Government and First Nations to bring infrastructure, capacity and oversight to the level required to comply.     Senior Assistant Deputy Minister of Regional Operations Start Date:
Following Royal Assent June 19, 2013

Develop all regulations by fall 2015.
5. Plan future investments in infrastructure with an equal emphasis on investing in the capacity to operate and maintain new and existing systems long-term, including program flexibility to outsource water and wastewater servicing where community-level capacity is not practical; and As part of the response to the 2009-2011 National Assessment of First Nations Water and Wastewater Systems, AANDC developed a strategy with First Nations to reduce system risks. The top priority of that ongoing strategy is enhanced capacity building and operator training. AANDC provides operation and maintenance funding for community infrastructure assets, including water and wastewater infrastructure, in accordance with the Capital Facility and Maintenance Program Operation and Maintenance Policy. Maintaining existing assets remains a higher priority than construction of new infrastructure. AANDC is updating the tools used to calculate operation and maintenance funding requirements and reforming the approach to operation and maintenance funding and minor capital into a coherent asset management approach. The operation and maintenance funding changes are being implemented in Ontario region in 2013-14 and will be rolled out to other regions over the next three years. Senior Assistant Deputy Minister of Regional Operations Complete: Planning priority for capacity implemented April 1, 2012.

Asset management approach start date:
April 1, 2013

Completion: 1 additional region in 2014-15, 3 additional regions in 2015-16, 3 final regions in 2016-17.
6. Work with Health Canada to develop a robust Performance Measurement methodology that allows for the reliable periodic reporting of the stated outcomes of the FNWWAP, including efficiency and comparability. AANDC will work with Health Canada to identify ways to improve performance measurement and to incorporate efficiency and comparability indicators for any future strategy for investments in water and wastewater on reserve.

AANDC will review performance measures for the Capital Facilities and Maintenance Program activities to support First Nations water and wastewater infrastructure, and explore ways to improve them.
Senior Assistant Deputy Minister of Regional Operations Comparability –
November 2013: Explore use of existing data to measure comparability as part of the process for updating the program's Performance Measurement Strategy. The updated strategy will be completed by March 2014 in accordance with the Department's Performance Measurement Strategy Action Plan

Efficiency – This activity will be captured under the FNWWAP efficiency indicator currently under development in accordance with Treasury Board guidelines.  The development of these indicators is in the testing phase and is expected to be completed for the 2015-16 reporting cycle.
Health Canada will strengthen the tracking of water and wastewater programming expenditures to better demonstrate utilisation of financial resources. Specifically, Health Canada will develop, in collaboration with regional offices, an action plan addressing weaknesses in tracking program expenditures. Manager, Drinking Water Program, Environmental Public Health Division, Interprofessional Advisory and Program Support Directorate, First Nations and Inuit Health Branch, Health Canada Completion of action plan: September 2014 Implementation of the action plan: March 2015 
7. Health Canada should continue to work with First Nations to build the capacity to monitor drinking water quality on reserve in order to increase the proportion of communities completing water quality testing according to the Guidelines for Canadian Drinking Water Quality. Management agrees with the recommendation to continue to work with First Nations to build capacity to monitor drinking water quality on reserve in order to increase the proportion of communities completing water quality testing according to the Guidelines for Canadian Drinking Water Quality and to maintain full access to trained monitoring personnel. Specifics actions are below. Manager, Drinking Water Program, Environmental Public Health Division, Interprofessional Advisory and Program Support Directorate, First Nations and Inuit Health Branch, Health Canada March 2015
  • Health Canada will conduct an analysis of the relevance and effectiveness of the Community-based Drinking Water Quality Monitor Program.
December 2014
  • Health Canada will review and complete the implementation of the National Training Program for Community-based Drinking Water Quality Monitors. This training program has been piloted since 2011 in Health Canada's First Nations and Inuit Health Branch's regions.
  • Health Canada will implement the revised quality assurance practices for microbiological monitoring.
March 2015

I recommend this Management Response and Action Plan for approval by the Evaluation, Performance Measurement and Review Committee 

Original signed on February 5, 2014, by:

Michel Burrowes
Director, Evaluation, Performance Measurement and Review Branch

I approve the above Management Response / Action Plan

Original signed on February 5, 2014, by:

Scott Stevenson
Senior Assistant Deputy Minister, Regional Operations

The Management Response / Action Plan for the Evaluation of the First Nations Water and Wastewater Action Plan were approved by the Evaluation, Performance Measurement and Review Committee. 

1. Introduction

1.1 Overview

The Evaluation, Performance Measurement and Review Branch (EPMRB) at Aboriginal Affairs and Northern Development Canada (AANDC) and the Evaluation Directorate at Health Canada-Public Health Agency of Canada undertook a horizontal evaluation of the First Nations Water and Wastewater Action Plan (FNWWAP). The FNWWAP supports AANDC's Strategic Outcome, The Land and Economy: full participation of First Nations, Inuit and Métis individuals and communities in the economy. It also supports Health Canada's Strategic Outcome of First Nations and Inuit Health Programming and Services: better health outcomes and reduction of health inequalities between First Nations and Inuit and other Canadians.

In line with the Treasury Board Policy on Evaluation (2009), the evaluation provides a neutral and independent analysis of the relevance (continued need, alignment with government priorities, and alignment with government roles and responsibilities) and performance (effectiveness, efficiency and economy) of the FNWWAP, while providing specific analysis of the current design and implementation. Evaluation findings were based on the triangulation of document and literature reviews, key informant interviews, a survey and case studies. The evaluation has generated ten key findings and six recommendations.

1.2 Program Profile

1.2.1 Background and Description

Access to safe drinking water, the effective treatment of wastewater and source protection for First Nations and on First Nation lands is a Government of Canada priority. The FNWWAP is designed to improve the health and quality of life of people in First Nation communities by assisting First Nations in providing better water and wastewater services to their communities.

The FNWWAP was introduced in 2008 as a horizontal initiative with Health Canada as part of government commitments in the 2007 Speech from the Throne, and further reinforced in Budget 2008, Budget 2010 and Budget 2012 to support First Nations' access to safe drinking water. It is a successor to the joint First Nations Water Management Strategy (2003-2008) and the AANDC Plan of Action for Drinking Water (2006-2008). While the current evaluation covers the period up to the expiration of authorities in March 2012, the FNWWAP has since been renewed and current authorities expire March 31, 2014.

From 2008 to 2012, the Government of Canada invested nearly $556 million in the FNWWAP, and a total of nearly $1.4 billion on all water and wastewater activities.

Standards and Protocols

At the time of the evaluation, there were no legally-enforceable safe drinking water standards for First Nation communities;Footnote 2 only federal Protocols for Safe Drinking Water in First Nation Communities. These include:

  • The Protocol for Centralized Drinking Water Systems in First Nations Communities;
  • The Protocol for Centralized Wastewater Systems in First Nations Communities; and
  • The Protocol for Decentralized Water and Wastewater Systems in First Nations Communities.

The protocols set out clear standards for the design, operation and maintenance of drinking water and wastewater systems. However, they do not have enforceable provisions for ensuring compliance.

In addition to the protocols, Health Canada developed the Procedure Manual for Safe Drinking Water in First Nations Communities South of 60, which is intended for Environmental Health Officers to better assist First Nation communities. It provides a common reference and consistent set of national approaches for monitoring drinking water quality in First Nation communities.

To address risks to drinking water, Health Canada worked with the provincial and territorial governments to develop the Guidelines for Canadian Drinking Water Quality that set out the maximum acceptable concentrations of contaminants in drinking water. With respect to wastewater, Environment Canada developed standards, guidelines and/or protocols for wastewater systems on federal and Aboriginal lands as articulated in Environment Canada's Wastewater Systems Effluent Regulations Reporting.Footnote 3

Compliance to the standards set out above is a program requirement and a condition of receiving funding.

1.2.2 Program Objectives and Expected Outcomes

According to its originating policy documents, the objective of the FNWWAP is to support First Nation communities on reserves in bringing their drinking water and wastewater services to a level and quality of service comparable to those enjoyed by other Canadians living in communities of similar size and location. The FNWWAP was approved from 2008 to 2010 and renewed for the period from 2010 to 2012 in order to:

  • Continue investments in the capital construction, operation and maintenance of water and wastewater systems, as well as investments in operator oversight, including drinking water quality monitoring initiatives by Heath Canada's Drinking Water Program;
  • Develop a legislative framework for First Nations aimed at protecting water quality; and
  • Implement initiatives to enhance the efficiency of existing investments in water and wastewater infrastructure.

The stated outcomes of the FNWWAP were to ensure that:

  • First Nation communities have an increased capacity to address potential water problems;
  • Health risks associated with water quality and supply are minimized;
  • All First Nation community water and wastewater facilities meet federal standards; and
  • First Nation communities have increased confidence in their drinking water.

To meet these objectives, FNWWAP invested in five key activity areas: infrastructure investments, operations and maintenance, training, monitoring and awareness, and standards.

1.2.3  Program Management, Key Stakeholders and Beneficiaries

AANDC and Health Canada are the federal departments that are primarily responsible for funding the delivery of safe drinking water on reserves.

AANDC

AANDC provides funding and advice to First Nations for the management and operation of related to the design, construction, operation and maintenance of water and wastewater systems. It also provides funding for the training and certification of water system operators. A Memorandum of Understanding has been in place between AANDC and Health Canada since 2005 regarding data sharing related to drinking water. AANDC activities include:

  • investments in infrastructure projects to address water and wastewater needs and to maintain existing systems;
  • investments in the ongoing operations and maintenance of water and wastewater systems;
  • funding for the hands-on training of treatment plant operators to increase the number of certified water treatment system operators; and
  • funding for third-party water and wastewater systems operation under the Safe Water Operations Program, when required.

AANDC is also the lead on legislative aspects related to drinking water.

Health Canada

Health Canada works with First Nation communities south of 60° to identify potential drinking water quality problems. In so doing, Health Canada:

  • ensures that monitoring programs for drinking water quality at tap are in place on reserve south of 60° (north of 60°, the territorial governments are responsible for ensuring safe drinking water in all communities in their territories, including First Nations). Health Canada works with First Nations to measure total coliformsFootnote 4 and E-coli, free and total chlorine residualsFootnote 5, baseline and routine chemical concentrations, and disinfection by-productsFootnote 6. The test results are stored in databases that can be extracted, listing individual testing results for each of the dissolved concentrations of various chemicals and elements, largely detailed by date and the concentration in volume per litre;
  • assists First Nation communities in building capacity to verify the overall quality of drinking water at tap and reviews, interprets and disseminate results to First Nations;
  • provides advice, guidance and recommendations to First Nation communities about drinking water safety and safe disposal of on-site domestic sewage;
  • reviews First Nation water and wastewater infrastructure project proposals from a public health perspective;
  • assists in the development of legislation and the supporting regulations from the public health aspects as related to the Minister of Health authorities; and
  • provides training programs and develops various public awareness materials and resource tools.

Directors general and assistant deputy ministers from Health Canada and AANDC meet when needed to exchange and coordinate action on all relevant issues related to the FNWWAP.

Environment Canada

Environment Canada, while not part of the FNWWAP, provides advice and guidance regarding source water protection and sustainable water use. It also regulates the treatment of wastewater discharged to receiving waters.

All three departments developed a National Framework for the Review of Water and Wastewater Infrastructure Project Proposals in First Nations Communities to ensure that reviews of water and wastewater infrastructure projects in First Nations communities are coordinated and that all applicable standards and requirements are met.

First Nations

First Nations are owners and operators of the water and wastewater systems on reserve and are responsible for the daily operation and management of their systems, which includes the design, construction, operation, maintenance, and monitoring (i.e. sampling and testing) of their water systems. First Nations authorities issue Drinking Water Advisories in First Nation communities, usually on the recommendation of Health Canada, or on their own initiative in emergency situations.

1.2.4  Program Resources

From 2008-09 to 2011-12, FNWWAP (investments to all departments and Grants and Contributions) invested $556 million to improve water and wastewater services on reserve. In total, AANDC spent $1.389 billion on water and sewage infrastructure from 2008-09 to 2011-12. FNWWAP funding represents 47.35 percent of AANDC's total investments. Health Canada spent $68.7 million on FNWWAP from 2008-09 to 2011-12. Original proposed expenditures specific to FNWWAP are detailed in Table 1 and actual reported expenditures for all water and wastewater activities reported by AANDC's Office of the Chief Financial Officer are detailed in Table 2. Expenditures provided from the financial system in Health Canada are provided in Table 3.

Table 1: Original Expenditure Estimates for FNWWAP
Fiscal Year
2008-2009 2009-2010 2010-2011 2011-2012 Total
New Funding from the Department of Aboriginal Affairs and Northern Development Canada
Vote 1. Operation Expenditures
Personnel 2,330,132 2,330,132 4,637,152 4,637,152 13,934,568
O&M 12,249,068 10,702,588 5,446,617 5,702,457 34,100,730
EBP @20% 466,026 466,026 927,430 927,430 2,786,912
Total Vote 1 15,045,226 13,498,746 11,011,199 11,267,039 50,822,210
Vote 10. Grant and Contribution (Gs&Cs) 122,650,000 124,200,000 126,385,971 126,130,131 499,366,102
Total Votes 137,695,226 137,698,746 137,397,170 137,397,170 550,188,312
Accommodation 302,917 302,917 602,830 602,830 1,811,494
Total Funds from AANDC 137,998,143 138,001,663 138,000,000 138,000,000 551,999,806
New Funds from Health Canada
Vote 1. Operating Expenditures
Personnel 5,004,508 5,250,077 5,451,207 5,476,544 21,182,336
O&M 15,147,004 14,820,398 14,891,695 14,857,996 59,717,093
EBP @ 20% 1,000,902 1,050,015 1,090,241 1,095,309 4,236,467
Total Vote 1 21,152,414 21,120,490 21,433,143 21,429,849 85,135,896
Vote 10 Gs&Cs 5,517,000 5,517,000 5,258,200 5,258,200 21,550,400
Total Votes 26,669,414 26,637,490 26,691,343 26,688,049 106,686,296
Accommodation 650,586 682,510 708,657 711,951 2,753,704
Total Funds from Health Canada 27,320,000 27,320,000 27,400,000 27,400,000 109,440,000
Total Funds from all departments 165,318,143 165,321,663 165,400,000 165,400,000 661,439,806
Existing Funding from 2008 Treasury Board Submission
Existing funding from Aboriginal Affairs and Northern Development Canada
Vote 10 Gs&Cs 197,500,000 197,500,000 395,000,000
Existing funding from Health Canada
Vote 1 Operating Expenditures 5,000,000 5,000,000 5,000,000 5,000,000 20,000,000
Total existing Funding all departments 202,500,000 202,500,000 5,000,000 5,000,000 415,000,000
Grand Total 367,818,143 367,821,663 170,400,000 170,400,000 1,076,439,806
Table 2: Actual Spending on all Water and Wastewater Activities at AANDC from 2008-09 to 2011-12
Actual 08/09 09/10 10/11 11/12 Total
Vote 1 Op. Exp. Acquisition and Construction of Water and Wastewater Infrastructure 0 0 156,680 5,997 162,677
Operation and Maintenance of Water and Wastewater Infrastructure 3,657,695 3,017,407 6,387,059 2,530,251 15,592,411
Water A-Base 3,657,695 3,017,407 6,543,739 2,536,248 15,755,088
First Nations Water and Wastewater Action Plan 6,029,351 9,570,330 10,064,571 5,476,408 22,140,660
First Nations Water Management Strategy 75,217 180,981 0 0 256,198
Budget 2006 Safe Drinking Water 0 0 0 0 0
Water Targeted 6,104,568 9,751,311 10,064,571 5,476,408 31,396,858
Total Vote 1 9,762,253 12,768 718 16,608,310 8,012,689 47,151,946
Vote 10Gs&Cs AFA Block/Core Funding – Water & Sewer Infrastructure Facilities 21,997,930 21,854,331 21,777,532 22,139,982 87,769,775
Acquisition and Construction of Infrastructure Assets and Facilities 40,040,428 42,383,761 16,573,003 37,345,061 136,342,252
Operation and Maintenance of Infrastructure Assets and Facilities 55,512,213 57,675,530 59,080,151 57,418,573 229,686,466
Gathering Strength – Infrastructure Facilities 48,141,093 44,153,784 42,608,724 32,176,856 167,080,456
Water A-Base 165,691,664 166,067,405 140,039,409 149,080,472 620,878,950
First Nations Water and Wastewater Action Plan 132,325,617 131,567,078 117,452,683 152,328,007 533,673,383
Canada Economic Action Plan – Water and Wastewater 0 68,576,380 119,108,716 0 187,685,096
First Nations Water Management Strategy 0 0 0 0 0
Budget 2006 Safe Drinking Water 0 200 000 200 000
Water Targeted 132,325,617 200,343,458 236,561,399 152,328,007 721,558,479
Total Vote 10 298,017,281 366,410,863 376,600,807 301,408,478 1,342,437,429
Grand Total 307,779,544 379,179,581 393,209,117 309,421,134 1,389,589,375
Table 3: Actual Spending on FNWWAP in Health Canada (based on financial system-SAP) from 2008-09 to 2011-12
Actual Spending (SAP) 2008-2009 2009-2010 2010-2011 2011-2012 Total
Regional Environmental Health Managers indicated that operational spending may have been higher than reported in the financial systems due to a number of factors, including:
  1. the transfer of O&M to capital funding (for the purchase of vehicles, equipment, etc.) was not tracked;
  2. it is not possible to clearly separate expenses that were shared among other environmental public health programs (e.g., travel, gas, vehicle maintenance);
  3. FNWWAP funding was diverted to address priorities, such as public health related emergencies and responses (e.g., H1N1) and zoonotic activities that were not coded to water; and
  4. regions did not track the A-base funding for water activities.
Vote 1.
Personnel 3,029,700 3,270,311 4,155,550 3,886,530 14,342,091
O&M 4,554,532 4,839,457 5,020,520 4,452,202 18,866,711
SUB TOTAL VOTE 1 7,584,232 8,109,768 9,176,070 8,338,732 33,208,802
Capital 25,145 86,076 12,400 123,621
Net Vote Revenue 58,402 58,402
Vote 10.
Vote 10 6,901,420 8,296,442 8,998,297 11,239,674 35,435,833
GRAND TOTAL 14,485,652 16,431,355 18,260,443 19,532,404 68,709,854
TOTAL 14,485,652 16,431,355 18,260,443 19,532,404 68,709,854

1.3 Current Evaluation

AANDC and Health Canada previously evaluated the First Nations Water Management StrategyFootnote 7 in 2007. That evaluation found an urgent need to improve the quality of on-reserve water and wastewater systems since they continued to be inadequate and posed health risks. FNWWAP replaced the Strategy in 2008. The current evaluation also considers the results of the 2009-2011 National AssessmentFootnote 8 as its results are a major driver for most of the current initiatives aiming to reduce systems risks on reserve.

The current evaluation examines the relevance and performance of the FNWWAP from 2008-09 to 2011-12. While the authorities for FNWWAP have been renewed and are slated to continue to March 2014, this evaluation is intended to inform program design elements in the interim, and inform decisions on program renewal prior to the 2014 expiry.

Governance

AANDC was the lead on this evaluation. A Memorandum of Understanding was established between AANDC and Health Canada to outline the conduct of the evaluation, underscoring AANDC's role as principally responsible for data collection and generating a final report. Health Canada was responsible for conducting additional analysis to inform some of the key evaluation questions, reporting on the achievement of outcomes relative to Health Canada's responsibilities and contributing to the final report. Health Canada-Public Health Agency of Canada's Evaluation Directorate and the Health Canada's First Nations and Inuit Health Branch's Performance Measurement Unit and program staff reviewed and provided input to the evaluation report.

To better ensure quality and transparency, the evaluation included an Advisory Committee, whose mandate was to provide insights on evaluation tools, findings and recommendations. It was composed of representatives from the Assembly of First Nations, AANDC and Health Canada. Environment Canada was not explicitly involved in this evaluation, as it is not specifically involved in the FNWWAP.

The Evaluation, Performance Measurement and Review Committee (EPMRC) acts as a review and advisory body for AANDC evaluations and only approves evaluations that have successfully fulfilled the requirements and needs of both AANDC and Treasury Board Secretariat. Health Canada's Executive Committee acts as the review and approval body for all Health Canada evaluations.

2. Methodology

2.1 Evaluation Scope and Timing

The Terms of Reference for this evaluation was approved by EPMRC on June 22, 2012. The evaluation was led by EPMRB in AANDC, which developed the methodology and tools, conducted extensive literature, file and document reviews, conducted the analysis of interview and survey data, and authored the final report. Health Canada conducted a file, document and database review on its components of the FNWWAP, and contributed to the final report. Additionally, the Institute on Governance was contracted to conduct all in-person and telephone-based key-informant interviews, and a series of case studies. Harris/Decima was contracted to manage the collection of survey data. Primary and secondary data collection and analyses took place for AANDC between February 2013 and August 2013.

The objective of the evaluation was principally to examine the relevance and performance of the activities specific to the FNWWAP; however, in examining the resulting outcomes, the analysis inevitably includes the impacts of all water and wastewater-related infrastructure expenditures and the Government of Canada's general approach to water and wastewater issues, as these all combine to generate impacts on water quality and infrastructure sustainability. While the scope of the evaluation primarily includes activities undertaken between 2008-09 and 2011-12, where applicable, some new initiatives, approaches and results post 2011-12 are also discussed.

2.2 Evaluation Issues and Questions

The evaluation focused on the following issues:

Relevance

Continued Need

  • Is there a continued need forthe investment in water and wastewater initiatives?
  • Can the FNWWAP be reasonably expected to reach its stated objectives?

Alignment with federal government priorities and departmental strategic outcomes

  • To what extent is the FNWWAPconsistent with:
    • Federal government priorities; and
    • Departmental Strategic Outcomes.

Alignment with roles and responsibilities

  • Are the current roles of the federal government and the roles of the departments involved appropriate?

Performance

Effectiveness

  • Has the FNWWAP been achieving its intended outcomes? Have there been improvements to water and wastewater management and capacity?
  • Have there been positive or negative unintended outcomes? If so, were any actions taken?

Efficiency

  • How has the FNWWAP optimized its processes and the quantity/quality of services to achieve expected outcomes?
  • How can the FNWWAP's efficiency be improved?

Economy

  • Are there opportunities to achieve the intended results of the FNWWAP with fewer resources?

Design and Delivery

  • Has the FNWWAP's design and the means at its disposal contributed to the achievement of the intended outcomes?
  • Is the FNWWAP implemented as planned? If not, why?
  • Are the governance structures and roles and responsibilities clearly understood?
  • Are they effective in achieving the expected results?
  • Could they be improved?
  • Is a system in place to identify, collect and report on:
    • Risk?
    • Performance measurement data?
    • Is the performance measurement data collected used in decision making?
    • Is it working well? If not, could it be improved?

Other Issues

Lessons learned/Best Practices

  • Did the FNWWAP take into account the lessons learned from the previous evaluation recommendations for both the First Nations Water Management System and the National Assessment at the design and delivery stages in order to maximize the success of the investment?
  • Are there any lessons learned/best practices that could be used from the FNWWAP?

Alternatives

  • Are there other means by which the program can achieve the same results more efficiently and/or economically?

2.3 Evaluation Methodology

2.3.1 Data sources

The evaluation's findings, conclusions and recommendations are based on the analysis and triangulation of five lines of evidence: document and file review, literature review, key informant interviews, surveys and case studies.

Document and file review

EPMRB undertook a comprehensive data and file review that included: AANDC, Health Canada, Treasury Board Secretariat, Department of Justice, Canada and Office of the Auditor General reports; the National Assessment of First Nations Water and Wastewater Systems (2011); reports from the Standing Senate Committee on Aboriginal Peoples; previous evaluations; Speeches from the Throne; Federal Budgets (covering the period of study from 2008 to 2012); the FNWWAP Action Plan Progress Report; Canada's Economic Action Plan; and international and national water and wastewater initiatives from Australia, Germany and Ontario. These documents were analysed in order to inform the evaluation questions and to provide an understanding of the Government of Canada's activities pertaining to water and wastewater on reserve.

Health Canada undertook a data and file review to inform the core evaluation issues of alignment with government priorities; achievement of expected outcomes; and efficiency and economy. This review included analysis of existing databases, documents and public opinion research, an assessment of resource allocation and utilization, and validation of findings with program staff.

Literature review

A review of relevant literature over the past twelve years was undertaken by EPMRB in order to acquire insight into water and wastewater issues on and off reserve. The literature from academic and research organisations, Aboriginal organizations (including the Assembly of First Nations), and the United Nations discussed international approaches to safe drinking water, national standards, recommendations for improvement, issues of access to and confidence in safe drinking water, fiduciary responsibilities, key challenges (i.e. design of wastewater systems, capacity) and best practices.

Key informant interviews

The purpose of the interviews was to gain further insights on the priorities, objectives and performance from individuals with direct experience and expertise with FNWWAP and with water and wastewater services on reserve. Institute on Governance completed a total of 20 one-on-one interviews, including: five First Nation water and wastewater organisations; representatives from AANDC (four at Headquarters; four in regions); and Health Canada (seven from headquarters and regions). Best efforts were made to have as many interviews with key informants as possible without reaching a point of saturation. The key-informant interview guide is contained in Appendix A.

Survey

A survey of water and wastewater managers on reserve was conducted in an effort to gain insights into their water and wastewater servicing realities and the impact that FNWWAP has had on the communities. EPMRB drafted the survey and contacted all First Nation communities (approximately 600) by telephone to inform them of the evaluation and to ask for their participation. EPMRB obtained consent to send the survey to 420 communities; they were given the option of completing it online or by telephone. Harris/Decima was contracted to manage the implementation of the survey instrument. The survey tool is contained in Appendix B.

One hundred and three surveys were received (74 online and 29 by telephone). The response rate was 24.5 percent of the sample size, representing 17 percent of the total target population of potential respondents. Responses were tabulated using SPSS/PASW statistical software and were analysed by EPMRB for trends and key qualitative insights which are noted in the relevant evaluation findings sections.

The regional distribution of respondents is detailed in Table 4.

Table 4: Regional Distribution of Survey Respondents
NL NS PEI QC ON MB SK AB BC TOTAL
1 4 1 7 21 8 10 3 48 103

Approximately 75 percent of respondents were either infrastructure managers or water / wastewater operators. The remaining 25 percent included band managers, various directors of public works and lands, and various supervisors of infrastructure-related activities. A quarter of respondents had less than five years experience in their current roles; more than half had five or more years experience and about 20 percent had more than a dozen years experience. While 78 percent of participants indicated that had been formally trained in their current role, 22 percent said they had not.

Case studies

The Institute on Governance conducted six case studies in order to garner specific insights from individuals and organisations with expertise and experience with FNWWAP and water and wastewater issues on reserve. The case studies covered urban, rural and remote First Nation communities of very small (under 100) to large (more than 2000) populations in British Columbia, Alberta, Manitoba, Ontario and Quebec. Five of the case studies included site visits to the communities, which were undertaken by representatives from the Institute on Governance and EPMRB. One case study was completed by telephone.

For each case study, the following individuals for each community were contacted for interviews:

  • Community leaders who could speak to AANDC reporting, capital and Operations and Maintenance budgeting (i.e. Band Manager or the Chief Financial Officer);
  • Community leaders with specific knowledge of the water and wastewater systems (i.e. Director of Public Works);
  • Community Elders or any other community resident who wanted to speak to water and wastewater;
  • Political community leaders (i.e. the Chief and/or the Council);
  • Water plant operators;
  • Circuit Rider Trainers;
  • Health Canada Environmental Health Officers; and
  • AANDC regional officers.

Document reviews were also conducted during site visits. Documents requested ranged from the communities' capital or strategic plans, funding requests, etc., to AANDC and Health Canada reports on water quality and contribution agreements.

2.3.2 Considerations, Strengths and Limitations

Considerations

The evaluation was conducted at a time when the federal government was reviewing Bill S-8 (the Safe Drinking Water for First Nations Act). It received Royal Assent in June 2013, near the completion of the evaluation. Therefore, this evaluation references the non-enforceable federal Protocols for Safe Drinking Water in First Nation Communities.

Gender-Based analysis: Broadly speaking this study did not explicitly examine gender issues due to the scope of study primarily pertaining to water and wastewater infrastructure projects.

Strengths

The use of multiple lines of evidence maximized the reliability and validity of the results and conclusions. In addition, collaboration with Health Canada and the Assembly of First Nations benefited the evaluation since they helped provide more robust evaluation tools and acted as sources for advice, direction and review. Furthermore, Health Canada validated preliminary findings with program staff and filled data gaps with regional program input.

Limitations

While the evaluation faced five key limitations, EPMRB and Health Canada tried to mitigate them to the extent possible and reasonable. The limitations include the following:

Surveys: While EPMRB contacted each First Nation community up to three times by telephone, the desired sample size was not reached. Of the 600 communities that were contacted, 420 agreed to participate and only 103 surveys were completed. The survey is not necessarily representative and the results are not interpreted outside of the context of the broader evaluation findings insofar as implications for conclusions and policy are concerned.

Key informant interviews: Multiple attempts were made to have a more equal representation of First Nations views, but despite best efforts, First Nation experts are underrepresented.

Case studies: While case study selection was based on obtaining appropriate regional representation, including rural, urban and remote communities of various population sizes, the consultants did not receive responses from multiple First Nation communities to take part in the case studies. However, the Institute on Governance in partnership with the Assembly of First Nations and EPMRB was able to gain the support from other communities. It should be noted that case studies are intended to be illustrative in nature, and are not intended to imply generalisation.

Attribution: AANDC and Health Canada do not track specific details on how Grants and Contributions money was spent pertaining to FNWWAP as such details were not reporting requirements in the Contribution Agreements. Therefore, it is difficult to assess the incremental impact of FNWWAP versus other AANDC and Health Canada water and wastewater initiatives.

Performance measurement: AANDC does not systematically collect data related to all of its key outcomes related to the FNWWAP. Therefore, it was difficult to measure the extent of FNWWAP's outcomes without key information (see Section 5.2.2). Health Canada did not always have baseline data available for comparison. Efforts were made to use 2007-2008 data to the degree possible, although in some cases 2008-2009 was used as the baseline. Although revisions made to the internal data collection tool at Health Canada were aimed at improving the quality (reliability) of data, evidence suggested that the revised definitions have not been implemented consistently.

2.4 Roles, Responsibilities and Quality Assurance

As the lead, the evaluation was managed by EPMRB in line with EPMRB's Engagement Policy and Quality Control Process.

Quality assurance was provided through multiple sources. EPMRB was responsible for overseeing all data collection (including products delivered by consultants), their analyses and review in order to ensure quality and accuracy. EPMRB was also responsible for preparing the final report and recommendations, with input from Health Canada.

Three governing bodies (see Section 2.3) (an Advisory Committee, the EPMRC and a peer review) and Health Canada also provided quality assurance by providing strategic direction, as well as the review of evaluation tools and  draft reports.

3. Evaluation Findings: Relevance

3.1 Continued Need

While there is a demonstrable need for continued investment in water and wastewater initiatives to achieve major improvements in water and infrastructure quality and risk reduction, short-term action plans may not address the more pervasive issues and a shift to longer-term planning is needed.

There is a clear and demonstrable need for investment in water and wastewater initiatives on reserve. Communities on reserve face considerable challenges providing safe drinking water, and these challenges are similar to those faced by other small, remote or isolated communities. These include difficulties finding and retaining qualified water treatment plant operators. Upgrades to, or replacements of, water facilities may thus take a long time during which drinking water advisories may remain in effect.Footnote 9

Before the implementation of the FNWWAP, according to a 2005 report of the Commissioner of the Environment and Sustainable Development to the House of Commons,Footnote 10 there were deficiencies in design, construction, operation and maintenance of many water systems and management of these systems was often weak. There was also inconsistent technical help available to First Nations to support and develop capacity to deliver safe drinking water. Additionally, long-term drinking water advisories had been in effect in some communities for several months or over a year,Footnote 11 suggesting that while systems were in place to identify risks, significant issues with the capital infrastructure and maintenance persisted.

The 2011 National Assessment of First Nations Water and Wastewater SystemsFootnote 12 highlighted serious issues with Operations and Maintenance, certification and monitoring. Only 67 percent of water systems in the more accessible communities, and 26 percent of water systems in the most remote communities, had a certified primary operator. Similarly, 51 percent of wastewater systems in the more accessible communities, and 25 percent in the most remote communities, had a certified primary operator. In relative terms, it is difficult to determine whether this is an optimal proportion of communities with certified operators given the high degree of variance in each province in requirements for accreditation of primary operators or laboratories. As of August 31, 2013, there were 178 water systems in 122 First Nation communities under a drinking water advisory.

It was observed from the case studies that often water issues both past and present were caused by a combination of poor or old infrastructure; improper maintenance; climate and geography; and socio-economic issues. Where there was clear success in water safety and maintenance, it was attributed to direct capital investments and increased monitoring, training and maintenance funding provided through water and wastewater investments, including FNWWAP.

The need for continued investments and ongoing support from the Government of Canada is clear. Infrastructure is not meeting its life cycle expectancy and the frequent need for repairs and retrofits suggests the need to strategically invest in and support both not only infrastructure and long-term operations and maintenance, but also the best value approaches for longevity and efficiency.

Health Canada and AANDC sought renewal of the authorities for FNWWAP from 2010-11 with the stated objective to provide First Nation residents with continued access to safe drinking water and wastewater services. Specifically, AANDC identified the need for new construction projects for the most urgent health and safety priorities and Health Canada identified a need to increase the frequency of drinking water quality monitoring. The renewed FNWWAP was to be used to address the most immediate of these priorities. At the time of the FNWWAP renewal, the Government of Canada had spent over $1.6 billion over the course of five years on First Nations water and wastewater infrastructure, including A-baseFootnote 13 funding. The renewal sought $330.8 million over two years above the approximately $203 million annually in A-base targeted to water and wastewater.

Most of the supplementary investments have taken the form of strategies and action plans, including $217 million for the First Nations Water Management Strategy (2003-2006, 2006-2008); $60 million for the Plan of Action for Drinking Water in First Nation Communities (2006-2008); $330.8 million for the first iteration of the FNWWAP (2008-2010); and $183 million through Canada's Economic Action Plan (2009-2011). Given the historic and current state of water and wastewater systems on reserve, and the projected need to overcome these issues moving forward, it is not clear that these initiatives are going to result in long-term sustainability and quality of infrastructure systems and drinking water safety, as there are pervasive issues of operations and maintenance, capacity and retention that are still highly problematic for many reserves (see Section 5 for discussion).

Many interviewees, and the vast majority of survey respondents, noted that real progress requires sustained and longer-term planning and development to allow sufficient timing and resources to achieve results, as opposed to fragmented and unpredictable short-term funding influxes such as FNWWAP.

3.2 Alignment with Government Priorities

The outcomes of the FNWWAP are clearly aligned with government priorities and strategic objectives.

AANDC includes in its mandate supporting Aboriginal peoples and Northerners to develop healthier and more sustainable communities. As such, activities related to improving the state of water quality and wastewater infrastructure are very much a strong priority of the Government of Canada. In recent years, the Government of Canada has reinforced its intention to promote access to clean water;Footnote 14 continue with another $330.8 million over two years effective 2012Footnote 15 to build and renovate water infrastructure and develop of a long-term strategy to improve water quality; and have recently adopted the Safe Drinking Water for First Nations Act.

FNWWAP supports AANDC's strategic outcome of "The Land and The Economy: Full participation of First Nations, Inuit, and Métis individuals and communities in the economy" and Health Canada's strategic outcome of "better health outcomes and a reduction of health inequalities between First Nations and Inuit and other Canadians." While the performance of the FNWWAP is officially measured through the Performance Measurement Strategy for the Capital Facilities and Maintenance Program, the original stated outcomes specifically for FNWWAP included:

  • First Nation communities have an increased capacity to address potential water quality programs;
  • Reduction in health risks associated with water quality and supply;
  • All First Nation community water and wastewater facilities meet federal standards; and
  • First Nation communities have increased confidence in their drinking water.

The first three of these outcomes are currently regarded as preconditions to reliable access to safe drinking water. In turn, access to safe drinking water is a basic component of ensuring the health of First Nation people – an obvious precondition of being able to participate in the economy. Irrespective of any issues with the specific design and implementation of FNWWAP, its outcomes are clearly aligned with government priorities and strategic outcomes.

For Health Canada, while the wording has varied slightly over the years, departmental priorities have continued to refer to improving health outcomes for First Nations and Inuit populations. Health Canada's activities funded under the FNWWAP are in line with these departmental priorities. As it pertains to Health Canada's strategic outcomes, previous and current iterations encompass a number of departmental initiatives but clearly Health Canada's activities funded under the FNWWAP are aligned with and contribute to departmental strategic outcomes (e.g., the 2011-12 strategic outcome: 'First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status').

3.3 Alignment with Federal Roles and Responsibilities

The specific roles and responsibilities of AANDC are seen as ambiguous and somewhat contradictory with regard to accountability and ownership. While its role as funder is seen as appropriate, there are mixed opinions on the appropriate degree of oversight, which may be partially attributable to a low level of awareness amongst First Nations of ownership responsibilities.

A key issue of contention with respect to roles and responsibilities is the ownership and responsibility for maintenance of systems. While the Protocol for Decentralised Water and Wastewater Systems in First Nation Communities, and the Protocol for Centralised Wastewater Systems in First Nation Communities, state that communities themselves have ownership of the systems with Band Councils responsible "for ensuring that wastewater systems are designed, constructed, and upgraded" and system operators responsible for their operation and maintenance, there is a sentiment that government acts as owner of the systems yet First Nations are ultimately responsible for their maintenance. Critically, however, only 24 percent of survey respondents were fully familiar with the Decentralised systems protocol and only 31 percent were fully familiar with Centralised wastewater protocol.

There is significant divergence of opinion respecting the appropriateness of the current role of AANDC. For example, many of those interviewed felt that as communities have ownership of the systems, it was inappropriate for AANDC to make unilateral decisions on the approval of systems in any given community. Additionally, it was seen as a contradiction that while First Nations are owners of the systems, AANDC will often deny funding for projects in communities that are not deemed "high-risk" (and thus high priority), thus, potentially impacting the community's ability to continue to meet standards in the long term as their ability to proactively invest in improvements is limited. The majority of survey respondents, which primarily included water and wastewater operators and infrastructure managers, also indicated that there was a need for more involvement from AANDC in order for the Department to ensure it has a proper understanding of community and system-specific realities necessary to make informed decisions on resource allocation.

It was acknowledged among interviewees, however, that AANDC's current role via FNWWAP has improved its accountability measures by ear-marking specific projects for funding, while at the same time having increased openness to innovation. Some First Nation participants took issue with the notion that while reinforcing the First Nation ownership, AANDC will exercise significant authority, for example, on reviewing projects out for tender.

Generally speaking, Health Canada's role was seen by interviewees as appropriate in monitoring, training and building inspection capacities from a public health perspective.

The main ambiguity was seen between the roles of AANDC and First Nation communities. Some interviewees felt that the Safe Drinking Water for First Nations Act (2013)Footnote 16 (the Act had not received Royal Assent at the time of the interviews) should clarify some of this ambiguity. However, the potential to clarify this ambiguity is not immediately apparent in the Act itself, as it mainly enacts Regulations specifying the Governor in Council's ability to make regulations governing the provision of drinking water and the disposal of waste water on First Nation lands; respecting standards for the quality of drinking water on First Nation lands; and respecting monitoring, remediation and emergency measures. AANDC will need to engage First Nations in the development of Regulations to ensure roles and responsibilities are clearly laid out and communicated.

4. Evaluation Findings: Performance

4.1 Achievement of Expected Outcomes

4.1.1 Outcome 1: First Nation communities have an increased capacity to address potential water quality problems

The capacity of First Nation communities to address potential water quality problems has shown considerable improvements since the introduction of the FNWWAP, particularly with respect to the number of Community-based Drinking Water Quality Monitors. However, significant limitations remain, primarily respecting the proportion of communities completing water quality testing and the numbers of qualified water system operators.

Health Canada assessed progress toward this outcome by looking at the increases in the number of Community-based Drinking Water Quality Monitors (CBWMs); the level of community access to on-site water test kits; the frequency of water monitoring at tap; and the number of water and wastewater servicing projects reviews completed.

The analysis noted continuous increases in the number of CWBMs or Environmental Health Officers (where no CBWM is present). Access to water testing kits for bacteriologic parameters was also noted as a positive step as the proportion of communities with access to these kits increased from 92 percent to 100 percent since 2008.

In terms of water monitoring, Health Canada recommends weekly water quality monitoring of community water systems, and data analysis from 2007 to 2011 indicated that the percentage of communities meeting this target had increased from 44 percent to 61 percent. For routine chemicals, Health Canada recommends annual monitoring. Data analysis indicated that the proportion of communities meeting this target was static at 86 percent between 2007 and 2011. For bacteriological parameters, Health Canada recommends quarterly monitoring of semi-public water systems. Data analysis indicated that the percentage of communities meeting this target had decreased slightly from 69 percent to 62 percent over this time period; however, this was at least partially attributed to a change in methodology for data collection. The data further showed that with respect to cisterns (with guidance for monitoring released in 2012), 52 percent of communities were meeting the recommended annual monitoring.

In 2010, Health Canada introduced a Policy on Individual WellsFootnote 17 offering public awareness materials to prevent contamination, including educational materials for visually inspecting and maintaining wells; and on-request bacteriological sampling and testing services. The analysis in 2011-12 showed that 20 percent of individual wells were monitored for bacteriological parameters. Although this is an on-request service and not part of the performance measurement of the FNWWAP, monitoring wells plays a role in drinking water safety on reserve.

Health Canada has developed a variety of public awareness materials, guidance documents and procedures since 2008 to promote drinking water safety. Public opinion research indicates that receiving information about drinking water monitoring procedures and results can provide reassurance about the safety of tap water.

Between 2007-08 and 2011-12, all 240 water and wastewater servicing projects proposals received from AANDC were reviewed by Health Canada from a public health perspective. The overall objective of the review was to protect public health by ensuring that proposed water and wastewater systems include adequate protection measures that will minimize potential public health risks by addressing potential corrections to proposed designs. This approach is intended to help ensure that new or upgraded facilities operate as per standards and requirements.

Interview respondents all agreed that capacity to manage and monitor water and wastewater infrastructure has improved markedly since the introduction of FNWWAP. It was also widely acknowledged that AANDC has become more proactive in supporting communities that have limited local capacity. As of 2012, according to the most recent Annual Performance Inspections, 60 percent of water system operators and 54 percent of wastewater system operators had been certified to the level of their system – an increase of nine percent and 12 percent, respectively. However, only approximately one third of survey respondents felt that there were sufficient numbers of trained water and wastewater system operators in their communities, relative to their level of need. Most respondents did indicate, however, that the operators they had were certified to an appropriate level. When asked to comment, survey respondents largely cited issues of limited numbers of operators; extreme difficulties with recruitment and retention; and significant staff turnover as their main capacity barriers. The same one third of participants felt that there were sufficient numbers of individuals trained via the Community-based Drinking Water Quality Monitor Training Program or the Circuit Rider Training Program.

On Health Canada's side, CBWMs and Environmental Health Officers are the primary service providers with respect to drinking water quality monitoring at tap in First Nation communities. CBWMs are community members, funded by Health Canada, to sample and test drinking water for potential bacteriological contamination, while Environmental Health Officers train and assist CBWMs with their responsibilities. If a community does not have a CBWM, an Environmental Health Officer monitors the drinking water with permission from the community. Although most First Nation communities prefer to use the services of CBWMs for bacteriological monitoring, some prefer to only use the services of an Environmental Health Officer. In 2008-09 and 2011-12, all First Nation communities had access to either a CBWM or an Environmental Health Officer (or to both) to monitor drinking water quality at tap. Almost all CBWMs (96 percent in 2008-09 and 98 percent in 2011-12) involved in drinking water monitoring were trained. Although retention was not explicitly part of the program mandate, it was anticipated that the national training program developed by Health Canada would help increase the stability of the CBWM workforce, potentially by broadening appreciation of their importance in ensuring drinking water safety in their communities.

As well, Health Canada has made improvements to quality assurance/quality control requirements and monitoring procedures for safe drinking water in First Nations communities. The revised Quality Assurance Program for microbiological monitoring was developed by the National Drinking Water Quality Safety Quality Assurance Working Group. The revised Quality Assurance Program for microbiological monitoring includes a combination of quality control procedures and quality assessment activities. It states that, when off-reserve laboratories are used for analysis of microbiological samples, these must be accredited. For samples analyzed on-site, third party assessments of on-site water testing facility practices, including CBWM techniques are recommended every two years and the outcomes reviewed by the Environmental Health Officer. For quality control of on-site water testing facilities, a commercially-prepared positive and negative control must be analysed a minimum of monthly (ideally in conjunction with each sampling set). The revised Quality Assurance Program for microbiological monitoring is being piloted.

Emergency response procedures have also been developed to provide a standard response to potential drinking water emergencies. In 2011, Health Canada released the Procedural Guidelines for Waterborne Disease Events in First Nations Communities South of 60°. These initiatives, once fully implemented, will support First Nation communities' capacity to address potential water quality problems.

Despite the recent approval of the Safe Drinking for First Nations Water ActFootnote 18 there is concern that an improvement in drinking water safety is not likely with the implementation of a regulatory regime where there is limited local capacity. For example, as suggested by the 2011 Expert Panel on Safe Drinking Water for First NationsFootnote 19, a serious investment in training and operational support is what is required to provide a safe drinking water supply, more so than complex equipment. Further, monitoring the compliance of maximum acceptable concentrations of pollutants may sidestep the actual causes of unsafe drinking water. This point is further reinforced by research on source water protection,Footnote 20 suggesting that complex treatment technologies and facilities would not eliminate the risks of unskilled or untrained monitors.

While according to interviewees and many survey respondents, developing capacity is a cornerstone of the Circuit Rider Training Program, capacity may mean different things depending on a community's circumstances. For example, key-informants cited circumstances where local capacity could actually entail the means to reliably outsource water and wastewater projects and monitoring or operations and maintenance, particularly in circumstances where sufficient local capacity may not be feasible or realistic because of a lack of qualified or interested individuals. In many cases, communities simply have too few system operators and the existing operators are overworked. It was further emphasised by survey participants that their primary concern was a lack of backup operators and/or an inability to fill positions where operators vacate their positions or are otherwise unavailable.

4.1.2 Outcome 2: Reduction in health risks associated with water quality and supply

While the ability to detect, monitor and react to health risks has shown improvement since the introduction of FNWWAP, there is no evidence to suggest broadly that risks specifically associated with the quality and supply have decreased, partially due to performance measures having not been adequately defined. Where reductions in risk were noted, they were largely attributed to infrastructure upgrades and the presence of trained operators.

Results of the National Assessment have shown that of 807 water systems examined in 2011, 39 percent (314) were classified as having a high overall system management risk, 34 percent classified as medium system management risk, and 25 percent classified as low system management risk.

The 2011-12 Annual Performance Inspection results showed some improvements, with 28 percent classified as high; 36 percent classified as medium and 36 percent classified as low. More specifically, 29 percent of communities decreased their risk rating, while 14 percent increased and the remainder were unchanged. Consistent with observations in Section 5.1.1, operator qualifications and record-keeping account for 60 percent of the measured risk, underscoring the importance of having certified and trained operators. Additionally, the majority of high risk systems serve a small population. The perceptions of survey respondents generally align with the official data on risks; specifically with 24 percent suggesting that they have seen a reduction in the number of health risks since 2008, 62 percent saying they have seen no change, and 14 percent saying they have seen an increase. When asked about the number of drinking water advisories communities have had since 2008, about 45 percent said they have had between one and three; whereas nine percent say they have had more than ten, and 28 percent say they have had none. In terms of waterborne diseases and outbreaks, 12 percent say they have had between one and three, whereas three percent say they have had more than ten, and 84 percent say they have had none. However, despite these survey responses, Health Canada data indicated that there were no confirmed cases of waterborne disease, and no identification or confirmation was made by a laboratory of any outbreak of water borne disease from 2007-08 to 2011-12.

The National Assessment further observed that, while not necessarily high risk, 161 water systems in 116 First Nation communities were under Drinking Water Advisory (DWAs) as of February 2011. These DWAs may be impacting up to 18,900 people, or approximately 3.9 %of the total on-reserve population (cited as 484,321 in the National Roll-up. Further, 17 percent of water systems and 21 percent of wastewater systems are operating either at or beyond water treatment capacity. As reported by Health Canada, drinking water advisories have increased by about 35 percent, from 269 to 363 between 2007 and 2011. Drinking water advisories can be broken down between boil water advisories (from 253 to 355) and "do not consume"Footnote 21 advisories or orders (which decreased from 16 to 7). The number of drinking water advisories may be increasing because more water systems are being monitored more frequently. As such, the increase in drinking water advisories can be viewed as a success insofar as the ability to detect potential health threats (particularly considering the absence of confirmed waterborne illnesses). However, the increase in drinking water advisories can also be a concern given the high frequency of risks detected, which suggests infrastructure or maintenance problems are common.

Although drinking water advisories have increased, most are revoked on average within a year of being issued with short-term boil water advisories being revoked on average within three months. The most common reasons for issuing drinking water advisories are inadequate disinfection or disinfectant residuals, unacceptable microbiological quality, equipment malfunction during treatment or distribution. Water systems under drinking water advisories for equipment malfunction during treatment or distribution appear to increase the average duration of advisories likely because these reasons are more difficult to address and required more resources. As such, the number of long-term drinking water advisories is also increasing.

Case study participants indicated that boil water advisories are most often implemented because of freezing pipes in winter, flooding, unpredictable water sources (i.e., lakes and rivers), infrastructure malfunctioning or systems operating beyond capacity. Communities where new plants had been constructed indicated that the number of advisories had reduced subsequent to construction, suggesting that new infrastructure does play a role. Given the relative newness of these infrastructure projects, however, it is not possible to tell whether the improvements with respect to boil advisories was because of equipment that was simply new and thus, less apt to fail, or because the equipment was customized to the unique needs of the community (i.e., source water, population, climate and geography) and was being operated and maintained properly.

About 45 percent of survey respondents stated their water and wastewater infrastructure was good or very good, while 35 percent said it was satisfactory and 20 percent said it was poor or very poor. About 37 percent of respondents said that it has improved since 2008, whereas 46 percent said it has remained unchanged and 17 percent said it has worsened. Participants' ranking of whether or not the infrastructure has improved was highly dependentFootnote 22 on their ranking of whether or not the water quality has improved. Most comments among those saying the infrastructure has become worse related to issues with equipment and aging infrastructure. Similarly, those who indicated it has improved cited specific infrastructure projects and investments as contributing to improvements.

Importantly, the measures above do not adequately speak to water source quality and risk. As discussed in more detail in Section 4.2.2 in the sub-section on "comparability", the measures currently collected have yet to yield the kind of data necessary to analyse water quality and risk improvements beyond simply counting the frequency and duration of DWAs. It was noted in Part 2 of the Report of the Walkerton InquiryFootnote 23 that the first barrier to the contamination of drinking water involves protecting the sources of drinking water. In that report it was recommended that the province of Ontario adopt a watershed-based planning process.Footnote 24

4.1.3 Outcome 3: All First Nation community water and wastewater systems meet federal standards

It is too soon to tell whether ongoing investments in water and wastewater systems, along with the ability of the federal government to enact Regulations stemming from the Safe Drinking Water for First Nations Act, will result in community systems meeting federal standards. There is concern that focussing on enforceable standards without ensuring First Nations have adequate infrastructure and capacity to meet those standards may overlook the core limitations facing First Nations water and wastewater systems.

Following the completion of the National Assessment, AANDC committed to identify concrete actions in the areas of capacity building and training, enforceable standards and protocols, and infrastructure investments. In 2011, AANDC followed up with inspections through the department's Annual Performance Inspections cycle, which assessed water and wastewater systems in order to determine their risk management levels and progress since the National Assessment. Over the course of the second two years of the FNWWAP, AANDC contributed $424.7 million in capital funding and $253.4 million in operations and maintenance funding for water and wastewater projects. Projects funded included the construction of new treatment facilities and water and wastewater storage facilities, the expansion of existing systems, and the servicing of lots for new home construction. Over the same period, 48 major water and wastewater infrastructure projects were completed. Additionally, new Circuit Rider Training Program guidelines were developed to help standardise the program and to support system operators in improving the management of operations and maintenance of their systems. The Annual Performance Inspection results (2011–2012) indicated that 60.1 percent of water treatment systems and 53.9 percent of wastewater systems are now managed by operators certified to the level of the system.

Having in place effective standards and technology was another recommendation of the Report of the Walkerton Inquiry, as well as adopting a government-wide drinking water policy and Act (for Ontario). Respecting First Nations, the Government of Canada has moved forward with the Safe Drinking Water for First Nations Act, which now allows the Governor in Council, upon the recommendation of the Minister of Aboriginal Affairs and Northern Development, to make regulations governing the provision of drinking water and the disposal of wastewater on First Nation lands (although the latter was already in place via the 2012 Water Systems Effluent RegulationsFootnote 25). This is the first step toward what could be considered "federal standards" in the sense that this is the first stage of developing a federal standard or compliance mechanism that can now be articulated clearly and enforced. With respect to water and wastewater systems specifically, the Act allows for the creation of regulations for the location, design, construction, modification, maintenance, operation and decommissioning of drinking water and wastewater systems, as well as the distribution of drinking water by truck.

While the outcome of meeting federal standards may seem reasonable, there is concern that the focus on the authority for legal enforcement, as opposed to operational competence and knowledge transfer, may be overlooking key limitations with the current system.Footnote 26 The biggest successes noted by survey, interview and case study participants related to major infrastructure upgrades, and the biggest limitations related to human resources and limited funding for Operations and Maintenance – there was no discussion on major limitations being the result of a lack of federal regulations. The current Act does not necessarily imply funding obligations, and there is no reason to assume that enforceable regulations will help enable communities to meet federal standards without adequate capacity and infrastructure investments. Interviewees expressed some anxiety that the Act may represent a regulatory framework for the federal government with significant financial obligations for First Nations, and those financial obligations may or may not be adequately supported by the federal government.

Ultimately, the investments in infrastructure and capacity as well as major capital investments are ongoing (currently with FNWWAP authorities in effect until 2014), and the Act allowing for federal standards only came into effect in June 2013. It is thus too soon to tell whether these initiatives will enable communities to meet federal standards as they have not yet been fully articulated, and there is concern that the creation of enforceable standards will not address the real issues facing First Nation water and wastewater systems. The regulations are intended, however, to clarify the standards to be met, and provide the means to measure whether or not they are actually being met within any given system.

4.1.4 Outcome 4: First Nation communities have increased confidence in their drinking water

There have been noticeable but relatively modest improvements in confidence in drinking water in First Nation communities since the introduction of FNWWAP.

Perceptions of water quality and safety are difficult to measure and do not necessarily equate to actual quality and safety; however, one can assume that if there are consistent indications of improvements in safety and quality, particularly to a point where it is consistent with other Canadian municipalities, then confidence will increase.

Among survey respondents, 62 percent generally agreed that the quality of their drinking water was comparable to that enjoyed by other Canadians in communities of similar size and isolation. About 15 percent were not sure, and 23 percent did not agree. Whether their community had a water treatment facility or not seemed to have no bearing on this estimate.Footnote 27,Footnote 28 Further, 71 percent stated that the quality of their drinking water was good or very good; while 16 percent stated it was satisfactory and 13 percent stated it was poor or very poor. Thirty-nine percent said that their water quality has improved while about seven percent said that it had worsened, with 54 percent suggesting there has been no change. When asked to rank what they felt would be their community's level of confidence in the drinking water, 70 percent generally agreed that their communities have confidence in their drinking water, while 14 percent were unsure, and 16 percent disagreed. Among those disagreeing, there was an equal mix of those viewing this lack of confidence as based on actual risks and those viewing it based on a lack of information and trust issues with the water supply (often due to past problems or frequent advisories).

Opinion polling conducted by EKOS Research Associates in 2007, 2009 and 2011Footnote 29 showed some modest improvements in confidence over time. The proportion of respondents ranking their water quality as "bad" decreased from 33 percent to 25 percent from 2007 to 2011, and those ranking it as "good" increased from 44 percent to 49 percent. Rankings of the safety of their tap water also increased steadily from 62 percent ranking it as somewhat or very safe in 2007, to 71 percent in 2011. Critically, however, there was a sharp divide between First Nation communities and other small communities, where 21 percent of First Nation respondents suggested their treatment procedures or facilities were outdated or unsafe, versus seven percent of non-First Nation respondents. The EKOS survey also indicated that 68 percent of First Nation respondents used bottled water, compared to 40 percent of non-First Nation respondents. Generally, results of the EKOS survey were consistent with those of the AANDC evaluation survey.

4.2 Efficiency and Economy

4.2.1 Program Design

There is a need for a longer-term strategy for water and wastewater issues that increases specific emphasis on recruitment, training and retention of system operators; and that is flexible in customising its approach to individual community needs.

As discussed in Section 5.1.1, First Nation communities are facing significant challenges acquiring, training and retaining certified operators. Interviewees, survey respondents and case study participants highlighted turnover due to more competitive wages being offered by non-First Nation communities as one of the central reasons, as well as being able to afford, acquire, train and retain backup operators - all cited as major limitations facing First Nation operators in minimising risk. Importantly, however, several case study and interview participants noted the need for flexibility and customisation of approach. For example, there may be instances where a community's size and ability to acquire the capacity for water and wastewater management would implicate the need for contracting out services as opposed to building local capacity, and it was said that AANDC practices should be flexible and support this approach where appropriate. The Report of the Walkerton Inquiry had made a similar recommendation regarding special cases, and specifically First Nations, in that variances from regulatory standards should be allowed, but only where the owner demonstrates that safety will not be compromised, and never for cost purposes alone.

The available evidence in this evaluation emphasises the need to match investments in capital facilities with investments in local recruiting and training for long-term success. There are significant risks in investing in complex and expensive equipment where it is well-known that there is a high unemployment rate or a low likelihood of being able to recruit operators. It has been said that it would be better to have limited infrastructure with a highly competent operator than the best infrastructure with an inadequate operator,Footnote 30 and there was sentiment among interview participants that the FNWWAP focuses more attention on the infrastructure projects without adequate attention to the needs for sufficient numbers of qualified operators.

Case study and survey participants noted that communities are limited in their ability to attract qualified staff and have them sufficiently certified, at least partially because of a lack of available individuals having completed high school, and thus, this is a key limitation when resourcing a community for water system operators. Case study participants also discussed the need for incentives to attract or retain operators, such as housing funding, and the funds to allow for a full-time backup operator.

Other issues included:

  • Cash flow issues such as requiring payment from the band up-front for training;
  • The need for better communication, particularly when policy changes are in the works; and
  • Communicating and discussing the specific reasons for not approving proposals.

Additionally, whereas AANDC normally promotes the use of decentralised systems, there may be instances where centralised systems are more efficient, and it was recognised by key informants that AANDC could be more cognizant of and flexible toward water system needs. It was also noted that better communication was required around specifying such needs. For example, where certain types of systems are optimal, AANDC could improve its communications as to why, in order to build better trust and understanding among First Nation water system operators and managers.

The protocols for the management of water and wastewater systems are said to be useful when understood and applied, but difficult to enforce and there is evidence that they are not as well known as would be expected for a program of such high risk. Only 39 percent of system operators surveyed were generally or completely familiar with the "Protocol for the Centralised Drinking Water Systems in First Nation Communities." Seventeen percent were somewhat familiar, and 44 percent were not familiar with it. Only 31 percent were generally or very familiar with the "Protocol for Centralised Wastewater Systems in First Nation Communities", with 21 percent saying they were somewhat familiar, and 48 percent saying they were unfamiliar with it. Only 24 percent were familiar with the "Protocol for Decentralised Water and Wastewater Systems in First Nation Communities", with 20 percent saying they were somewhat familiar, and 55 percent saying they were unfamiliar.  Knowledge of each of these protocols was highly intercorrelated.

There is little recourse for not meeting standards or protocols, although it was widely acknowledged among interviewees that the Safe Drinking Water for First Nations Act is the first step to addressing this issue. It was emphasised that the success of water systems stemming from the development of clear regulations would depend on the Government of Canada matching capital investments to the degree of upgrades required to meet regulations; on First Nation readiness for devolution of authorities related to water and wastewater activities where applicable; and on a very clear articulation of responsibilities between Health Canada, AANDC and First Nations.

Finally, as discussed in Section 4.1, the Government of Canada's approach to water and wastewater likely needs a longer-term focus on operations and maintenance that moves beyond short-term funding influxes for infrastructure projects. Considering the amount of additional infrastructure projects that can be expected in the future, the limitations in having qualified operators, and the relatively slow pace of improvement that can be expected with such limitations, a longer-term strategy may be needed. Specifically, it is difficult for both the Government and First Nations to dedicate to long-term and sustainable approaches to safe water and wastewater management if funding is variable, short-term and unpredictable.

4.2.2 Performance Measurement

While measures of reductions in health risks are covered by Health Canada activities and measures for system risk and capacity have been operationalised at AANDC, measures of increased confidence and comparability with other Canadian municipalities have not been articulated or operationalised.

The performance measurement aspects of FNWWAP are held within the broader performance measurement strategy for the Capital Facilities and Maintenance Program. Issues specific to water and wastewater within this performance measurement strategy primarily relate to risk levels and communities meeting the standards outlined in each of the protocols.

While measures of reductions in health risks are covered by Health Canada activities and measures for system risk and capacity have been operationalised at AANDC, measures of increased confidence and comparability with other Canadian municipalities have not been articulated or operationalised. Additionally, while a key outcome is that systems meet federal standards, measurement has not been articulated or systematised, and there are questions regarding the utility of measuring against standards that are not part of any regulatory framework. Regarding the latter point, regulations stemming from the Safe Drinking Water for First Nations Act should poise AANDC, in consultation with Health Canada and First Nations, to develop clear performance measures against the standards to be set in these upcoming regulations.

Finally, as discussed below, AANDC and Health Canada do not manage their financial information with respect to the projects funded under FNWWAP in a way that would allow for an effective assessment of efficiencies.

Capacity

The assessments of the capacity are assessed via the Annual Performance Inspection's measure of the certification of the system operator. As mentioned in Section 5.2.1, however, optimal capacity may not always entail community members being the ones to manage water and wastewater systems; rather, the optimal state is the ability of a community to manage water and wastewater systems with minimal government intervention, and in a way that is most effective and efficient for the community. This means that in order to assess capacity, AANDC needs to assess the extent to which communities can set priorities; develop short- and long-term strategies; acquire human resources where necessary (including procurement); manage their assets; and ultimately be accountable for system maintenance, risk and performance. In other words, in addition to supporting operator capacity, capacity support from the Department should also include support to community managers and decision makers. Finally, as mentioned above, survey respondents cited capacity issues as more of a product of insufficient numbers of operators, rather than the presence of a certified operator.

Risk Reduction

Risk is also measured via AANDC's Annual Performance Inspections, which examine specific elements of water or wastewater systems, including water source (or effluent receivers for wastewater systems); system design; system operation and maintenance; operator training and certification; and record keeping and reporting. Each element is then assigned a risk score categorized as low (1.0 to 4.0), medium (4.1 to 7.0) and high (7.1 to 10). It is important to note that these risk numbers are only a measure of the overall system management risk and not a measure of water safety or quality. In other words, they reflect the risk that, in the event of a problem, a system would fail to produce safe water.

According to AANDC guidelines, deficiencies in high risk systems could result in advisories against drinking the water (such as drinking water advisories) or inadequate water supplies. Once systems are classified under this category, regions and First Nations then take immediate corrective action to minimize or eliminate deficiencies.

The overall risk for a system is also ranked using the same categories and scale. However, an overall system rank is not an average of the scores from the five categories; rather it is calculated using a weighted value for each category as follows:

  • water source and the wastewater effluent receiver (10 percent)
  • system's design (30 percent)
  • operation and maintenance (30 percent)
  •  the level of training and certification of its operator (20 percent)
  • reporting and record keeping (10 percent)

The increases in drinking water advisories shown in the National Assessment have been interpreted by some interviewees as a sign of improved capacity for detecting, and taking steps to reduce, risks to health and safety. Critically, however, from a risk reduction point of view, this is only half of the equation, as frequent or long-lasting boil water advisories are reflective of significant problems with infrastructure, maintenance, or source water. Therefore, while a first step may be ensuring the ability to identify risk to address immediate health concerns – and this has been well captured by Health Canada through drinking water advisories – equal consideration should be given to the assessment of risk from a source water and infrastructure quality and sustainability point of view, which incidentally is how this outcome is stated. The assessment of risk as discussed above appears to be able to address this sort of risk. Additionally, Asset Condition Reporting System inspections are completed every three years, which report on the system structure conditions.

Confidence

While there are proxy measures for confidence in drinking water systems largely based on public opinion research, if the AANDC and Health Canada continue to consider confidence in drinking water systems as a necessary indicator, it is necessary to plan for systematic measurement. Importantly, however, it is essential to develop indicators of confidence that truly reflect anxieties about the safety and quality of the drinking water, and to be cautious about what could be confounding variables such as taste and a general tendency in the population to prefer bottled water.

Related to confidence is the reduction in risks. As discussed above, risk reduction as measured through boil water advisories may actually have the reverse effect on confidence, given that the public may often associate boil water advisories with increased risk and poor quality. One of the best ways to avoid confounding confidence with other variables is to measure it against the frequency of advisories or other issues over time. This should give an adequate portrayal of the degree to which confidence is associated with perceived risks.

Comparability

Drinking Water Advisories

Table 4 below compares the "current" (as of May 7, 2013) raw numbers of DWAs by region between on- and off-reserve communities. The challenge with using drinking water advisories as a proxy for comparability is three-fold: 1) there is a challenge in collecting the right data from "comparable" communities with respect to numbers and durations of advisories; 2) it is difficult to conceptualise DWAs as a measure of success or failure of a system, as in a sense it could be seen as both; and, 3) DWAs are preventive measures and there exist many triggers leading to the issuance of DWAs. Drinking water quality is one of the triggers but not the only one. An additional difficulty is that an appropriate comparison would require comparing the proportion of systems under a DWA, and thus, would require an accurate knowledge of the total number of water systems in each province, as well as the ability to compare systems via the population covered. This would entail comprehensive research. Another option is to benchmark performance by setting more specific targets with respect to water quality and infrastructure stability and sustainability, as well as risk.

Beyond DWAs, comparability as it is articulated in AANDC documents is largely subjective insofar as it relates to infrastructure and water quality. There are multiple dimensions of comparability – most notably the suitability of the systems to the community given its needs, population, growth and geography; the quality of water; the stability and sustainability of the system; and the ability to maintain and monitor the system. AANDC has not articulated what aspects should be comparable and has not operationalised measures of comparability. At a minimum, the system risk indicators could be compared with geographically similar non-First Nation communities of approximately the same size. Comparing the quality of the water, however, is much more complex as there are so many measures, indices and variability in reporting. Federally, while standards for drinking water quality are within the purview of Health Canada, there is no systematic federal approach to measuring water quality off reserve as this is managed by municipalities and largely subject to provincial guidelines. (Although, it should be noted that Health Canada monitors drinking water quality on reserve at a comparable level with provincial guidelines).

One way to approach comparability would be through direct comparisons between one community's water quality and another via periodic comparisons of pH, toxins, clarity, bacteria, and biological sampling. As discussed in Section 1.2.3, currently Health Canada works with First Nations to measure total coliforms and E-coli, free and total chlorine residuals, baseline and routine chemical concentrations, and disinfection by-products. The test results are stored in databases that can be extracted, listing individual testing results for each of the dissolved concentrations of various chemicals and elements, largely detailed by date and the concentration in volume per litre. Similar data are collected by municipalities.

Table 4: Number of Existing Drinking Water Advisories as of May 7, 2013, including Precautionary, Boil, and Do Not Consume Advisories
Region First NationsFootnote 31 Municipal/Provincial Comments regarding Municipal/Provincial findings
Atlantic 9 257 (does not include Prince Edward Island - PEI) Nova Scotia – 35 Boil water advisories
Newfoundland/Labrador – 222 Boil water advisories
New Brunswick – no current Drinking Water Advisories
PEI – no information available
Quebec 2 173 Boil water and do not consume advisories for municipal and non-municipal systems from 17 administrative regions.
Ontario 75 N/A No list available.
Manitoba 2 124 Boil water, drinking water avoidance and water quality advisories for public, semi-public and private systems.
Saskatchewan 13 244 Includes Precautionary Drinking Water Advisories and Emergency Boil Water Advisories reported by Ministry of the Environment or the Health Region in relation to public and private systems.
Alberta 26 0 DWAs included as "Active Health Advisories". Alberta reporting no current Active Health Advisories related to drinking water.
British Columbia 31 590 Boil water advisories related to public and private systems from five regional health authorities.
Yukon N/A N/A Outside Health Canada mandate, not provided publicly by territory.

Similar data are collected by municipalities. As an example, the evaluators extracted data for the First Nation community of Constance Lake. In determining a comparable community, the evaluators isolated communities within a 500 km radius for similar geography, and narrowed the selection to communities of similar size. Using these parameters the most similar community was Hornepayne Ontario. While the variables collected for the two communities are similar (with slightly different intervals for testing), in both cases, the extracted reports are several hundred pages with specific volume testing on specific dates.

The Canadian Water Quality Index

Many municipalities have, however, employed the use of the Canadian Water Quality Index (CWQI) endorsed by the Canadian Council of Ministers for the Environment.Footnote 32 This index was introduced by the Water Quality Guidelines Task Group of the Canadian Council of Ministers for the Environment,Footnote 33 and was a modification of the original British Columbia Water Quality Index.

Conceptually, the CWQI comprises three factors. F1 (scope) assesses the extent of water quality guideline non-compliance over time, and is calculated:

Text description of F1

F1, representing scope, is equal to the number of failed variables divided by the total number of variables, multiplied by 100.

F2 (frequency) assesses how often the observed value was off acceptable limits, representing the percentage of individual tests that do not meet the objectives, and is calculated:

Text description of F2

F2, representing frequency, is equal to the number of failed tests divided by the total number of variables, multiplied by 100.

F3 (amplitude) assesses the amount by which the failed test values do not meet their objectives, resulting an "excursioni" variableFootnote 34, which is summed and divided by the number of tests to yield a sum of excursions, or "nse", which is calculated:

Text description of NSE

NSE is equal to the sum of excursion, divided by the number of tests.

To yield a value between 0 and 100Footnote 35, the formula is normalised as:

Text description of F3

F3, representing amplitude, is equal to NSE divided by 0.01NSE plus 0.01.

The CWQI is then calculated as:

Text description of CWQI

The CWQI is calculated as follows: CWQI is equal to 100 minus the square root of F1 squared plus F2 squared plus F3 squared, divided by 1.732.

The factor of 1.732 is used to scale the index from 0 to 100. Using this approach, the water quality would then be ranked in one of five categories:

  1. Excellent (95-100)
  2. Good (80-94)
  3. Fair (60-79)
  4. Marginal (45-59)
  5. Poor (0-44)

The Canadian Council of Ministers for the Environment has released guidelinesFootnote 36 on site-specific application of the CWQI. While data compiled for the formula for specific sites may be complex, the output of an easy-to-understand index that can be compared between "like" communities has the advantage of providing the means to measure comparability and give First Nations and federal departments a better measure of progress toward this goal. AANDC and Health Canada should be poised to engage water technicians and municipalities in an effort to gather data in a fashion necessary to measure comparability using this index.

System Type

Another measure of comparability would be simply to compare the types of water systems in place for communities of similar size and geography. While these data exist in AANDC database extracts, the comparison may not always be straightforward, as 1) the naming conventions of system types may not be the same (water system expertise would be required); and 2) it may not be appropriate to assume that the comparison community off reserve has the appropriate system in place. Asset and risk variables with AANDC's databases are comprehensive; however it would be difficult to acquire comparable information for similar communities off reserve. The more optimal approach to comparability may be to assess the cost of operations and maintenance off reserve and compare it to the investments in operations and maintenance levels on reserve. Given much of the evidence in this evaluation suggests that much of the risk lies in the maintenance and life cycle of systems, this comparison could inform the Government of Canada with respect to reasonable expectations on the level of investment needed in operations and maintenance to reduce risks and meet life cycle expectancy.

Ultimately, from a point of view of fairness and equity, it is reasonable to expect that individuals living on reserve would have the same sense of trust and safety using their water at tap as other Canadians, and certainly perceptions of water quality are a valid indicator.

Data on wastewater effluents are now collected by Environment Canada via the Government's recent Wastewater System Effluent RegulationsFootnote 37 The indicators collected as part of the Government of Canada's Effluent Regulatory Reporting Information System should enable the comparison of First Nation and non-First Nation communities with respect to meeting these wastewater regulations. AANDC would need to work with Environment Canada to assess the degree to which this comparison can be made by community.

Should AANDC decide that comparability continues to be a viable and relevant indicator of the success of investments in water infrastructure and capacity building, this concept will need to be better articulated and operationalised with the guidance of water system specialists.

Measurement of Efficiency

Data as they are currently collected from recipient reporting do not sufficiently allow for assessments of efficiency. The current system tracks information on the number of homes in various circumstances, such as type of water and wastewater systems, and there are no detailed financial reporting parameters in Contribution Agreements. In the most recent Performance Measurement Strategy for the Capital Facilities and Maintenance Program (2009), the only water and wastewater indicator related to efficiency was "the percentage of water/wastewater systems meeting general life expectancy." There were notable limitations with respect to the ability to collect data on this indicator, and it was acknowledged that extensive research would need to be done. As an indicator, this item was never fully developed; however, data collected via Asset Condition Reporting System should be able to flush out this indicator as a proxy for life expectancy. Additionally, comparisons of costs of investments in water systems by type and population covered, as well as costs of operations and maintenance, have the potential to benchmark the relative appropriateness of investments so that the Government of Canada can assess value for money.

4.2.3 Efficiency

The operating expenditures for the FNWWAP and water and wastewater in general comprise a small proportion of the total, indicating that the program is operating efficiently from an internal operations point of view. Without a thorough understanding of the value and longevity of infrastructure projects funded, however, it is not possible to measure efficiency of the vast majority of the program's expenditures. There are indications that investments in new technologies may lead to increased efficiency, particularly among smaller communities.

As discussed above, Health Canada and AANDC currently lack systematic data for measuring efficiencies in the FNWWAP. Additionally, FNWWAP as an expenditure, acts largely as a top-up to existing operating expenditures and Grants and Contributions related to water and wastewater, thus, it is not possible to separate the incremental results of FNWWAP from the broader water and wastewater spending.

It is clear, however, that while the ideal proportion of total expenditures used for operating expenditures (versus Grants and Contributions) have not really been benchmarked for government programming, in AANDC this proportion for FNWWAP was four percent in 2008, seven percent in 2009, eight percent in 2010 and 3.5 percent in 2011 (see Figure 1).

Over the four years, the financial system indicated that Health Canada spent 62 percent of all the resources available, with the percentage spent increasing over time (from 55 percent in 2008-09 to 70 percent in 2011-12) and with full utilization of Grants and Contributions. Key informants noted that not all relevant operating expenditures may be properly coded to this program area, thus total spending in financial reports is likely an underestimation.

Figure 1: FNWWAP Operating Expenditures as a Proportion of FNWWAP Total Expenditures
FNWWAP Operating Expenditures as a Proportion of FNWWAP Total Expenditures
Text description of Figure 1

Figure 1, a line graph, is titled as and depicts the "First Nations Water and Wastewater Action Plan (FNWWAP) Operating Expenditures as a Proportion of FNWWAP Total Expenditures." The x-axis displays the fiscal year, with a range from 2008-09 to 2011-12. The y-axis depicts the dollar value representing program expenditures. Expenditures range from $0 to $160,000,000. Three lines can be seen on the graph to represent: 1) Total FNWWAP Expenditures; 2) FNWWAP Operating expenditures (versus Grants and Contributions); and 3) Operating Expenditures.

The line depicting total FNWWAP expenditures remains more or less constant from 2008-09, hovering between $120 million and $140 million, and dips below $120 million for fiscal year 2010-11. Between 2010-11 and 2011-12, total expenditures peaked from below $120 million to hover between $140 million and $160 million. Essentially, the line representing total FNWWAP expenditures suggests a funding decrease for fiscal year 2010-11, followed by an increase in expenditures for fiscal year 2011-12.

The line depicting FNWWAP operating expenditures, as well as the line depicting FNWWAP operating expenditures as a percentage of total, closely resemble one another inasmuch as they both hover between the zero dollar mark and the $20 million mark. However, where FNWWAP operating expenditures remain more or less constant, "operating expenditures as a percentage of total" increases marginally between 2008-09 and 2010-11, before falling to 2008-09 levels in fiscal year 2011-12.

The operating expenditures for the FNWWAP and water and wastewater in general comprise a small proportion of the total, indicating that the program is operating efficiently from an internal operations point of view.

The increase in operating expenditures in 2009-10 and 2011-12 are largely due to the $9 million spent on the National Assessment. Proportionally speaking, AANDC operational expenditures appear to be reasonably low, suggesting that the internal administration of the FNWWAP accounts for a small proportion of the total, which can be considered a good indicator of efficiency. A similar trend can be observed in examining the total of all AANDC water and wastewater-related expenditures, with operating expenditures representing 3.1 percent, 3.4 percent, 4.2 percent and 2.6 percent of the total expenditures respectively from 2008-09 to 2011-12. With a recent up-tick in total expenditures, operating expenditures did not increase, suggesting that the vast majority of expenses are directed to projects while keeping internal costs relatively low.

While there appear to be some reasonable indicators of efficiency in terms of operating expenditures, the efficiencies of the activities funded within the Grants and Contributions componentFootnote 38 of the FNWWAP are much less clear. While it is possible to examine the projects and resulting infrastructure by cost (although this would have to be done looking at the total water and wastewater funding, as the incremental contribution of FNWWAP would likely not be obvious given that communities use the summation of their Grants and Contributions to fund various projects), the efficiency and the "cost-effectiveness" would be much less clear without understanding the value of the system in terms of its sustainability, longevity and impacts on health and safety. The infrastructure projects funded through FNWWAP, for example, are relatively new and it is difficult to know at this point their value relative to their cost, as new systems likely are operating at full functionality.

AANDC produced its National First Nations Infrastructure Investment Plan in 2012, which outlined plans for life cycle management of AANDC-funded community infrastructure, a resource allocation process and an investment planning process. The priority areas included the protection of health and safety as well as assets; health and safety improvements; recapitalisation/major maintenance; and anticipated growth. Regional First Nations Infrastructure Investment Plans are informed by a national priority ranking framework, which are intended to provide a consistent and transparent priority ranking of infrastructure investments. 

AANDC also promotes the use of full life-cycle costing analysis when approving capital construction projects. This type of analysis allows for the consideration of all the associated costs (e.g. for design, construction, operations and maintenance, insurance, and major renovations) of the proposed infrastructure throughout its entire life cycle. This type of information is used to make comparisons between proposed options (e.g. building an expensive installation with low operating and maintenance costs versus a less-expensive facility with higher operating and maintenance costs) and to determine the most cost-effective option in the short, medium, and long term. This exercise is also useful in determining the funding levels required from the First Nation in future years for the operations and maintenance of the proposed infrastructure.

Key to ensuring efficiency includes investing in the proper maintenance of systems, and ensuring the most appropriate systems are in place relative to the capacity needs (referring to water or wastewater system capacity) of the communities. There are also indications that innovative technology may reduce both risk and costs where more than one community shares operator hubs using broad-band for the detection of water issues.

While there is limited extractable data on the uptake of new technologies from recipient databases, one project of note was the Alberta First Nation Technical Services Advisory Group Remote Monitoring Project. Technical Services Advisory Group, which is responsible for the delivery of the Circuit Rider Training Program in Alberta, developed the initiative in direct response to a number of the reoccurring issues discussed above with respect to water treatment plants on reserve, including high staff turnover, improper maintenance and repair, and the troubleshooting and response time. The initiative consisted of the installation of remote water monitoring technology into all the water treatment plants across the province, at a total cost of $4.3 million. These systems are all supported by Technical Services Advisory Group Circuit Rider Program and have been integrated into a province-wide network that links all First Nation schools, administration buildings, health centers, and water treatment plants to the "Alberta SuperNet." Technical Services Advisory Group has also created an in-house Network Operations Center and Help Desk to allow First Nations to have network access with dedicated support.

Currently, plant operators are still responsible for the testing of water quality at the plant along with testing at tap by Health Canada Environmental Health Officers, as per the Guidelines for Canadian Drinking Water Quality. Health Canada provides on-site testing facilities for remote (and the majority of) First Nation communities. The added benefit of the Remote Water Monitoring system is that it would immediately help identify a potential problem and automatically contact the designated personnel.

As another example, the Williams Lake Water Improvement Project was developed in British Columbia at a total project cost of approximately $4.9 million. The water system had been inadequate both in terms of the quantity and quality needed to meet the needs of the approximately 300 people living in the community. The project included the tie-in of two recently drilled wells, the construction of a new water treatment plant, and a new reservoir supply main. The new treatment plant will be equipped with a Supervisory Control and Data Acquisition monitoring system, which will allow remote monitoring and troubleshooting of any issue, including water quality. The community has been on a drinking water advisory since 2006 and it is anticipated the new system will significantly improve the conditions of the drinking water and accommodate continued population growth.

5. Conclusion and Recommendations

Water and wastewater issues are among the most pervasive concerns on reserve, and it is clear that continued government support and investments are essential. There have been clear improvements to infrastructure and risk management; however, additional attention is needed with respect to ongoing maintenance and sustainability of systems, and ensuring First Nations have at their disposal the means to ensure water safety and infrastructure quality in the long term. Specifically, it is difficult for both the Government of Canada and First Nations to be dedicated to long term and sustainable approaches for safe water and wastewater management if funding is variable, short term and unpredictable.

While the development of the Safe Drinking Water for First Nations Act is an essential first step in ensuring there is a measured standard around safety of drinking water, it is essential that First Nations are fully engaged in the development of Regulations; that roles and responsibilities are clearly defined; and that First Nations actually have the capacity and means to meet the Regulations against which they will be held.

Currently, First Nations face issues with recruitment and retention of water system operators, and it was noted that while operators that are present are generally qualified, there are risks associated with turnover and a lack of means to acquire back-up operators. It is essential that infrastructure investments are made alongside proportional investments in recruitment, training and retention of qualified individuals, and that appropriate technologies exist where there is significant benefit to be gained from hub-based water and wastewater management models.

AANDC and Health Canada do not systematically collect all of the information needed to adequately measure performance against the outcomes stated as part of the FNWWAP. Most critically, there is a lack of a systematic approach to measuring comparability and efficiency.

It is therefore recommended that AANDC:

  1. Work with First Nations and Health Canada to develop a long-term strategy for investments in water and wastewater infrastructure and maintenance in order to address the pervasive and longstanding issues of water and infrastructure quality and maintenance;
  2. Ensure that the Regulations ensuing from the Safe Drinking Water for First Nations Act are developed with the engagement of First Nations, and that roles and responsibilities, both current and subsequent to the ensuing Regulations, are clearly understood and communicated;
  3. Engage First Nations to develop a concrete plan to address issues of recruitment, retention and capacity development of trained and skilled operators;
  4. Engage First Nations to facilitate their readiness to comply with Regulations ensuing from the Safe Drinking Water for First Nations Act;
  5. Plan future investments in infrastructure with an equal emphasis on investing in the capacity to operate and maintain new and existing systems long term, including program flexibility to outsource water and wastewater servicing where community-level capacity is not practical; and
  6. Work with Health Canada to develop a robust Performance Measurement methodology that allows for the reliable periodic reporting of the stated outcomes of the FNWWAP including efficiency and comparability.

Health Canada has further recommended that it:

  1. Continue to work with First Nations to build the capacity to monitor drinking water quality on-reserve in order to increase the proportion of communities completing water quality testing according to the Guidelines for Canadian Drinking Water Quality.

Appendix A – Interview Guide

Interview Guide for the evaluation of FNWWAP

Date:

Name:

Position:

The Evaluation, Performance Measurement and Review Branch (EPMRB), Audit and Evaluation Sector of Aboriginal Affairs and Northern Development Canada (AANDC) and Health Canada are conducting an evaluation of the First Nations Water and Wastewater Action Plan (FNWWAP).

The evaluation will focus on the relevance, design and implementation and the performance of the FNWWAP and its progress towards achieving outcomes, while looking at economy and efficiency. As part of the evaluation, we will be conducting interviews with a range of key informants from the various groups involved with FNWWAP.

The information we gather will be summarized in aggregate form; interview notes will not be shared outside of AANDC's EPMRB or Health Canada.

We realize that you may not be in a position to address some of the questions. If you cannot answer a question, please let us know. Additionally, your participation in this interview is voluntary and you may withdraw from the study at any time.

Introduction

  1. Can you briefly describe your involvement with the FNWWAP (i.e., how long, role and responsibilities, specific projects)?
    • Can you briefly describe your involvement in issues generally related to water and wastewater in First Nation communities?
  2. Is there a need for active support in matters pertaining to water and wastewater in First Nation communities by the Government of Canada?
    • If so, do you believe that the FNWWAP is the best approach?
  3. Do you believe First Nation communities and the Government of Canada share the same objectives when it comes to water and wastewater projects and/or improvements? To meet those objectives, do you believe that First Nation communities and the Government of Canada share the same understanding of their roles and responsibilities? Can you elaborate?
  4. Do you believe the role of AANDC is appropriate with respect to its activities related to the FNWWAP? The role of Health Canada? The role of First Nations?
    • Is there anything you believe should change?
    • Do you believe there's overlap or duplication with respect to these activities or responsibilities?
    • Are you familiar with the governance structure respecting the FNWWAP? If so, in what ways have the governance structure, roles and responsibilities with respect to the FNWWAP facilitated the delivery of its activities? In what ways has it impeded it?
    • Do these practices foster value for money? Do they foster sustainable development and contribute to meet the life expectancy of the infrastructure provided under FNWWAP?
  5. Were the activities of the FNWWAPimplemented as planned? What changed and why?
    • Does the planning cycle under FNWWAP enable First Nations to meet their water and wastewater infrastructure needs?
    • What activities were undertaken to implement the key components (capital investments, Operation and Maintenance investments and training) of FNWWAP? Were those activities implemented in a timely manner?
    • While currently guided by a series of protocols, there had been a stated objective to develop a legislative framework giving First Nations regulatory regimes, enforceable and compatible with provincial/territorial regimes. Is this still a relevant priority? Why/Why not?
  6. Generally speaking, do you believe AANDC collects the appropriate and necessary information (data) for managing the FNWWAP? What about Health Canada?
    • Which performance indicators are used to measure AANDC's performance/management of FNWWAP? Which performance indicators inform the level of compliance of First Nations with applicable standards? Are they appropriate?
    • Is there sufficient information to identify and prevent public health risks pertaining to drinking water and wastewater? Are risks being effectively identified and prevented?
    • With respect to collecting and analysing information to meet reporting expectations, what do you believe needs to change? How would these changes implicate stakeholders?
    • Do you believe reporting expectations need to change? Please explain.
  7. Do you believe the activities of the FNWWAP have resulted in First Nation communities having reliable access to safe drinking water? To the treatment of wastewater? To safe drinking water infrastructure? To safe wastewater infrastructure? If not, do you believe it is a reasonable expectation that the specific activities of the FNWWAP can result in such outcomes?
  8. Do you believe the activities of the FNWWAP have made sufficient progress in bringing the drinking water and wastewater services to a level and quality of service enjoyed by other Canadians? If not, do you believe this is a reasonable expectation that the activities of FNWWAP can achieve this?
  9. Has there been an increase in drinking water quality monitoring in communities since 2008? If not, why not? Is this still a relevant objective?
    • In your view, is it now easier to identify problems or risks related to drinking water than it was in 2008?
  10. Are you familiar with the National Assessment? If so, do you believe it has adequately quantified the gap between the state of infrastructure and existing provincial regulations and AANDC standards? If not, why not? What has been the impact subsequent to the National Assessment?
  11. a) Do First Nation communities have sufficient capacity to manage and monitor on-reserve infrastructure related to water and wastewater?    b) Is AANDC effectively managing risks associated with providing contribution funding for major capital projects, while respecting First Nations' rights to manage their own infrastructure?
  12. Have there been any unforeseen outcomes (positive or negative) resulting from the current design of the FNWWAP?
  13. What has been done since 2008 to enhance efficiency of the existing investments?
    • Improved targeting of Operations and Maintenance funding? If so, what has been the impact?
    • Development of standards and guidance on small systems (wells and septic systems)? If so, what has been in the impact?
    • A detailed engineering assessment of water and wastewater needs in each community, including recommendations? If so, what has been the impact?
    • Innovative use of technology, such as broad-band remote monitoring?
    • The development of procedures to detect waterborne illnesses?
  14. Can you provide any examples of lessons learned/best practices with respect to the FNWWAP?
    • Has the department acted on these?
    • Were there lessons and/or best practices from previous evaluations or assessments (the First Nations Water Management System and the National Assessment) used? If yes, which ones? Could you please explain?
  15. Do you have any other comments or observations?

Thank you for your participation.

Appendix B – Survey Tool

Preliminary Questions Response Options Comments
A. What is your role/position in the community with respect to water and wastewater?
  • Infrastructure Manager
  • Engineer
  • Foreman
  • Operator
  • Technician
  • Other
Please describe _____.
B. How long have you had this role/position?
  • ___ Years ___ Months
 
C. Describe your responsibilities (check as many that apply).
  • Regularly monitoring & reporting on water quality
  • Regularly monitoring & reporting on water and wastewater infrastructure
  • Provide training
  • Other
Please describe ______.
D. Have you been trained to undertake this role/position?
  • Yes
  • No
 
E. If so, by whom?
  • Informally trained myself
  • Informally trained by predecessor/colleague
  • Circuit Rider Training Program
  • Community-Based Drinking Water Quality Monitor program
  • Trade school
  • University
  • Other
Please describe _____.
F. Please estimate the proportion (%) of homes with each type of drinking water delivery services.
  • ______% piped
  • ______% individual wells
  • ______% truck delivery
  • ______% no service
Please describe _____.
G. Please estimate the proportion (%) of homes with each type of wastewater services.
  • ______% piped
  • ______% individual system (i.e. septic tank & shoot-out)
  • ______% truck haul
  • ______% no service
Please describe _____.
H. I am familiar with the First Nations Water and Wastewater Action Plan (FNWWAP).
  • Yes
  • No
Please describe _____.
I. I am familiar with Health Canada's Water Advisory Toolkit for First Nations.
  • Yes
  • No
Please describe _____.
J. I am familiar with Health Canada's Toolkit for Individual Wells for First Nations.
  • Yes
  • No
Please describe _____.
Evaluation Questions
Relevance Questions to ask respondent Answers
1. Is there a continued need for the FNWWAP? Does the FNWWAP address a demonstrable need in Aboriginal communities? 1. The community's drinking water is currently to a level and quality of service comparable to that enjoyed by other Canadians living in communities of similar size and location.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
2. The community's wastewater services are currently to a level and quality of service comparable to that enjoyed by other Canadians living in communities of similar size and location.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
3. Does the community have a water treatment facility?
  • Yes
  • No
A. If yes: Please rank the quality of the treatment facility.
  • Very poor
  • Poor
  • Satisfactory
  • Good
  • Very good
  • Please describe_____.
B. If yes: The community has received appropriate advice and assistance prior to the construction of the water and wastewater facility.
  • Agree
  • Disagree
  • Not sure
  • N/A Please describe_____.
C. If no: Does the community need a water treatment facility?
  • Yes
  • No
  • Please describe_____.
4. Since 2008, the FNWWAP has provided the necessary resources for the improvement, operation and maintenance of the community's water and wastewater facilities.
  • Agree
  • Disagree
  • Not sure
  • Please describe _____.
5. My community has trained operators to ensure safe and healthy drinking water and wastewater.
  • Agree
  • Disagree
  • Not sure
  • Please describe ____.
6. Since 2008, the quality the community's drinking water has on average been ___________.
  • Very poor
  • Poor
  • Satisfactory
  • Good
  • Very good
7. The quality of the community's drinking water has ______ since 2008.
  • Improved
  • Worsened
  • Unchanged
  • Please describe_____.
8. Since 2008 , the quality of the community's water and wastewater infrastructure has on average been ________.
  • Very poor
  • Poor
  • Satisfactory
  • Good
  • Very good
9. The quality of the community's water and wastewater infrastructure has _______ since 2008.
  • Improved
  • Worsened
  • Unchanged
  • Please describe_____.
2. To what extent are the objectives of the FNWWAP aligned withAANDC strategic outcomes and federal government priorities? 10. I support the objectives of the FNWWAP (bringing the community's drinking water and wastewater services to a level and quality of service comparable to that enjoyed by other Canadians of similar size and location)?
  • Agree
  • Disagree
  • Don't know
  • Please describe _____.
3. To what extent are the objectives of the FNWWAP aligned with federal roles and responsibilities? 11. The Government of Canada should have ______ involvement as it currently does with respect to the FNWWAP.
  • Less
  • The same
  • More
Design & Delivery Questions to ask respondent Answers
4. Has the FNWWAP's design and the means at its disposal contributed to the achievement of the intended outcomes? 12. I am familiar with the following Protocols and Procedure Manual:
A. Protocol for Centralised Drinking Water Systems in First Nations Communities
  • Very familiar
  • Somewhat familiar
  • Not familiar
  • Unaware
B. Protocol for Centralised Wastewater Systems in First Nations Communities
  • Very familiar
  • Somewhat familiar
  • Not familiar
  • Unaware
C. Protocol for Decentralised Water and Wastewater Systems in First Nations Communities
  • Very familiar
  • Somewhat familiar
  • Not familiar
  • Unaware
D.Procedure Manual for Safe Drinking Water in First Nations Communities South of 60°
  • Very familiar
  • Somewhat familiar
  • Not familiar
  • Unaware
13. My community meets standards as described in the following Protocols and Procedure Manual:
  • Agree
  • Disagree
  • Don't know
A. Protocol for Centralised Drinking Water Systems in First Nations Communities
  • Agree
  • Disagree
  • Don't know
B. Protocol for Centralised Wastewater Systems in First Nations Communities
  • Agree
  • Disagree
  • Don't know
C. Protocol for Decentralised Water and Wastewater Systems in First Nations Communities
  • Agree
  • Disagree
  • Don't know
D.Procedure Manual for Safe Drinking Water in First Nations Communities South of 60°
  • Agree
  • Disagree
  • Don't know
14. There are a sufficient number of trained water and wastewater operators in the community.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
15. The community's water and waste water facility is operated by an operator certified to the appropriate level.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
16. There are a sufficient number of people trained from the Community-based drinking water quality monitor (CBWM) recruitment and retention strategy.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
17. There are a sufficient number of people trained from the Circuit Rider Training Program.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
18. The frequency of testing drinking water for quality is appropriate.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
19. the FNWWAP has introduced appropriate quality assurance/quality control requirements in order to ensure safe and healthy drinking water and wastewater.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
20. The community's water and wastewater system has been properly situated and installed.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
21. The community has sufficient capacity to identify water quality problems and potential waterborne diseases.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
22. The community has sufficient capacity to address potential water problems.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
23. The community has seen _______of the number of health risks than before 2008.
  • A reduction
  • No change
  • An increase
  • Please describe_____.
24. Since 2008, the community has had_ (#)_ drinking water advisories.
  • If unsure of exact number, please estimate____.
25. Since 2008, the community has had _(#)_waterborne disease cases and outbreaks.
  • If unsure of exact number, please estimate ____.
26. The community currently has confidence in their drinking water.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.
27. Existing equipment maintenance measures are sufficient.
  • Agree
  • Disagree
  • Not sure
  • Please describe_____.

Performance

Effectiveness (Outcomes)
Outputs (Efficiency) Questions to ask respondent Answers
5. Has the FNWWAP optimized its processes and the quantity/quality of services to achieve expected outcomes? 28. What can you suggest are some areas for improvement to the management of water and wastewater issues?
  • Please describe _____.
Outputs (Economy) Questions to ask respondent Answers
6. Are there opportunities to achieve the intended results of the FNWWAP with fewer resources? 29. Do you have any suggestions to improve efficiency (achieving outcomes with fewer resources or making better use of existing resources)?
  • Please describe_____.
Concluding questions Questions to ask respondent Answers
  30. Are there comments you would like to add before concluding the survey? Comments _______.
Date modified: