ARCHIVED - Implementation Evaluation of the Enhanced Prevention Focused Approach in Alberta for the First Nations Child and Family Services Program

Archived information

This Web page has been archived on the Web. Archived information is provided for reference, research or record keeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Date: September 2010
Project Number: 1570-7/09059

PDF Version (312 Kb, 64 Pages)

 

 

Table of Contents




List of Acronyms

ACYS
Alberta Children and Youth Services

CFA
Comprehensive Funding Agreement

CFNFA
Canada/First Nations Funding Agreement

CFSA
Child and Family Service Authority

EPFA
Enhanced Prevention Focused Approach

EPMRB
Evaluation, Performance Measurement and Review Branch

FNCFS
First Nations Child and Family Services

FTP
Flexible Transfer Payment

FY
Fiscal Year

HQ
Headquarters

INAC
Indian and Northern Affairs Canada

ISIS
Intervention Services Information System

OAG
Office of the Auditor General

RMAF
Results Based Management and Accountability Framework

TOR
Terms of Reference

Return to Table of Contents





Glossary of Selected Terms

Flexible Transfer Payment (FTP): A conditional transfer payment funding mechanism made for a specified purpose where a fixed amount of funds is allocated to an eligible recipient, usually on the basis of a pre-determined formula. Recipients are expected to manage within the fixed amount. Recipients may keep unspent balances as long as the program terms and conditions have been fulfilled. Surpluses must be spent in an approved manner. Deficits are the responsibility of the recipient.

Incremental Investments: Refers to new funding to implement new measures in Alberta. Further incremental investments for other provinces/territories (P/Ts) are subject to future approval.

Maintenance: Consists of expenses for child placements out of the parental home, including, but not limited to, a foster home, group home, institution, kinship care and adoption, where authorized under provincial legislation.

Operations: Ongoing operating and infrastructure costs of eligible recipients. Funding generally covers the salary, travel, accommodation and related costs for the administration, limited prevention programming, and protection casework to run an agency.

Ordinarily resident: Means that an individual usually lives at a civic address on reserve; or, in the case of children in joint custody, lives more than 50 percent of the time on reserve; or stays on the reserve and has no usual home elsewhere. Students registered full-time in a post-secondary education or training program and who are in receipt of federal, band or Aboriginal organization education/training support funding continue to be considered ordinarily resident on reserve if: they maintain a residence on reserve, they are a member of a family that maintains a residence on reserve; they return to live on reserve with parents, guardians, caregivers or maintainers during the year, even if they live elsewhere while attending a school or working at a temporary job. The residence of a child who comes into the core of a mandated child welfare authority is derived from the residency of the child's parent or guardian at the time that child is taken into care. Individuals who are off reserve for the purpose of obtaining care not available on reserve, or who are off reserve for the primary purpose of accessing social services because there is no reasonably comparable service available on reserve continue to be considered ordinarily resident.

Prevention / Least Disruptive Measures: The existing national operations formula contains provisions for prevention / least disruptive measures, which include non-medical services and supports (time-limited services such as respite care, counseling, homemaker and parent aid services) normally of a temporary duration. These measures are designed to keep families together and children in their own homes. Commencing in Alberta, prevention/least disruptive measures is funded as a separate stream under the Enhanced Prevention Focused Approach. This approach is based upon qualified front-line social worker practitioners providing prevention and early intervention services to families to prevent children from coming into care.

Enhanced Prevention Focused Approach: Three targeted streams of investment - fixed amounts for maintenance, operations, and prevention/least disruptive measures - that will be funded as an FTP. Though funds are only eligible for use for Child and Family Services, agencies have the ability to move money between the three streams to better meet their needs. In Alberta, INAC has implemented the Targeted First Nations Child and Family Services (FNCFS) Funding Approach. This is being implemented on a province-by-province basis and the willingness of P/T and First Nation partners. This new funding is available to eligible recipients who produce business plans in line with the Tripartite Accountability Framework.

Tripartite Accountability Framework: Between P/Ts, First Nations and Canada to establish formal partnerships towards better outcomes for children and families on reserve. In the Province of Alberta, FNCFS agencies that opt into the Tripartite Accountability Frameworks are funded through the Targeted FNCFS Funding Approach

Return to Table of Contents





Executive Summary

This evaluation report, completed by DPRA on behalf of Indian and Northern Affairs Canada (INAC), presents the findings of the implementation evaluation of the Enhanced Prevention Focused Approach (EPFA) of the First Nations Child and Family Services (FNCFS) Program in Alberta for the period fiscal year (FY) 2007/08 to FY 2009/10. The total estimated funding level for the national FNCFS program for these years is $481M, $523M and $537M, respectively. During the same period in Alberta, the estimated funding level is $120M, $125M and $126M, respectively. The purpose of the evaluation has been to assess the implementation of the EPFA in Alberta to date, in terms of its relevance, effectiveness, the degree to which the approach is efficient and economical; and other issues that were identified through the evaluation process.

INAC's FNCFS program, recognizing that a prevention approach to child welfare may represent a way to improve child and family outcomes, and in the long term, reduce escalating maintenance costs, [Not 1] devised a new, more flexible approach to funding a prevention approach for child welfare services on reserve, based on a prevention model developed and implemented for a number of years in Alberta. INAC subsequently secured approval for program authority and new investments for the implementation of the new approach, first in Alberta, then in some additional regions. [Not 2] The intent is to ultimately roll out the new approach in all regions.

Program authorities reside in "Payments to support Indians, Inuit and Innu for the purpose of supplying public services in social development (support culturally appropriate prevention and protection services for Indian children and families resident on reserve)" [Not 3] derivefrom the Department of Indian Affairs and Northern Development Act, R.S.C. 1985, c. I-6, s.4 and subsequent policy proposals.

Key findings/conclusions of the evaluation are as follows:

Relevance

The need for an approach that enhances the safety and well-being of children and families on reserve is well documented, based on the over-representation of First Nations children in care, and the reality that their numbers are growing, with associated escalation of program costs. There was evidence in the literature review, as well as unanimous agreement among interview and focus group respondents that the prevention approach is needed in Alberta because it is more responsive to community needs and a more culturally appropriate model.

The prevention approach not only aligns with current best practices in child welfare internationally, but represents a potentially more cost-effective option in the long term. The prevention approach is more effective at addressing the root causes of high numbers of children in care, which include poverty, parental addictions, and parents who are younger than their non-First Nations counterparts. This is consistent with the Government of Canada objective of achieving healthy, safe and sustainable communities.

Design and Delivery

The evaluation found that the design of the EPFA is largely regarded as appropriate for meeting its intended outcomes. While there are some early indicators of success, some challenges will need to be addressed as the model moves forward into other jurisdictions.

In particular, the evaluation found that issues around timing, provincial requirements, human resource shortages, salary, support from government/agency management, community linkages, training, and geographical isolation were of greatest concern to FNCFS agencies and were considered to be essential to the successful implementation of the approach.

Effectiveness (Performance/Success)

The research is inconclusive regarding the extent to which prevention programming has been effective to date. As the model has been implemented between one and two years (depending on when each agency began implementation), it is still too early to assess performance adequately. While an increasing number of First Nations children and youth experienced permanent placements in 2007-2008 and more First Nations children in foster care or kinship care were placed with First Nations families since 2005-06, more children entered child protection between FY 2005/06 - FY 2008/09, with percentages of children entering via the family enhancement program decreasing.

Movement towards the EPFA is slowly occurring, as more culturally‑appropriate placements are taking place and knowledge of the program is spreading among families and caseworkers. A mechanism for agencies to share best practices and lessons learned was noted as being an important issue moving forward. Communication issues surrounding roles and responsibilities and intended outcomes were also noted between the tripartite partners and will need to be addressed in order for agencies to maximize the potential of the approach.

Furthermore, several jurisdictional issues were identified as challenging the effectiveness of service delivery. Of key importance are the interpretation of on-reserve and off-reserve residents; the availability and access to supportive services for prevention; and determination of who is responsible for computer hardware/technical support/IT training to build and support this capacity in First Nations agencies. An additional challenge is ensuring that reliable data is collected, which will allow for accurate performance measurement and some comparability of prevention services.

Effectiveness (Efficiency/Economy)

It is acknowledged in the literature that cost‑effectiveness of prevention approaches can only be assessed after a lengthy period of time. Less than three years into the implementation of the model (and less than two years for some agencies), it is too soon to say whether the EPFA is achieving cost savings.

Nevertheless, there are signs that a small number of agencies have reduced the numbers of children in care, thereby reducing maintenance costs. The evaluation was unable to determine whether these reductions in protection cases are a direct result of prevention activities; however, in the agencies that showed these reductions, there were associated increases in prevention spending.

There is also evidence that some FNCFS agencies are achieving cost savings by integrating programming, cost-sharing, and other creative ways to get the most from available prevention funds. Some examples include using their own resources for community-strengthening activities and skills training; cost-sharing activities with community programs; as well as integrating community and medical services to enhance their work.

While the evaluation team was able to access some provincial data on Family Enhancement caseloads and children in care, it was determined that the numbers provided were not sufficiently reliable to be included in the evaluation findings. [Not 4] Thus, the evaluation was not able to adequately assess the comparability of prevention services between First Nations agencies and provincial child welfare services. In order to obtain the most pertinent information available for decision-making, the program should collaborate with the provinces to share relevant data.

It is recommended that INAC:

  1. Revisit the EPFA funding model within the next year to learn from the past two years of implementation and incorporate resources for things such as a remoteness factor that may address some of the issues faced by rural and remote communities. As part of this review, INAC should also determine if the calculations that are based on an assumed population of children in care are relevant in achieving desired outcomes.
  2. In partnership with the province and other federal agencies, if applicable, maximize the use of scarce capacity building funds to strategically invest in developing the capacity of FNCFS agencies, bearing in mind that some agencies have higher capacity needs than others.
  3. Ensure that First Nations agencies are sufficiently supported in the development of their business plans, including establishing baseline data on common indicators for prevention in the start-up phase; that business plans are updated annually and adjusted based on the feasibility of indicators and targets; in collecting ongoing longitudinal research to report on effectiveness; and in reporting the results through one common format to the province and INAC.
  4. Improve communication and awareness of the purpose, processes, procedures and inherent flexibility of the EPFA to FNCFS agencies to facilitate implementation and administration of the EPFA moving forward.
  5. INAC and the province discuss, clarify and formally agree (i.e. in the Tripartite Accountability Framework) to their respective roles and responsibilities with regard to training and technical support (particularly computer/IT training) for FNCFS agency staff, as well as on data collection requirements.
Return to Table of Contents





Management Response / Action Plan

Project Title: Evaluation of the Enhanced Prevention Focused Approach in Alberta for the Child and Family Services Program
Project #: 1570-7/09059
Recommendations Actions Responsible Manager (Title / Sector) Planned Implementation and Completion Dates
1. Revisit the EPFA funding model within the next year to learn from the past two years of implementation and incorporate resources for things, such as a remoteness factor that may address some of the issues faced by rural and remote communities. As part of this review, INAC should also determine if the calculations that are based on an assumed population of children in care are relevant in achieving desired outcomes. Revisiting the EPFA funding model:

As the department transitions to the Enhanced Prevention Focused Approach province-by-province, costs such as remoteness are being incorporated into the funding formula (e.g. Saskatchewan, N.S., PEI all have remoteness as a factor, as will all remaining provinces). In the case of Alberta, First Nations and the Province did not flag remoteness during the discussion to develop the formula; therefore, it was not included.

The introduction and implementation of an EPFA to child welfare on-reserve will not necessarily result in an overnight decrease of children in care. In fact, initially, the children in care rate will continue to grow until the full implementation of this model takes effect, which will probably be after a few years. Alberta found that, with implementation of the provincial response model, it was able to better contain costs over a number of years. It is therefore anticipated in FNCFS that there will be fewer children and youth requiring costly out of home placements in institutions and group homes, thereby reducing federal costs of maintaining children out of their parental home, while allowing for better outcomes from FN children, youth and families.

Based on this recommendation, INAC will determine if any modifications to the new model are required.

Director Social Program Reform April 2012
2. In partnership with the province and other federal agencies, if applicable, maximize the use of scarce capacity building funds to strategically invest in developing the capacity of FNCFS agencies, bearing in mind that some agencies have higher capacity needs than others. Assessing the use of scarce capacity resources:

INAC will review its distribution of capacity resources region-by-region within the FNCFS program.

Director Social Program Reform March 2011
3. Ensure that First Nations agencies are sufficiently supported in the development of their business plans including establishing baseline data on common indicators for prevention in the start-up phase; that business plans are updated annually and adjusted based on the feasibility of indicators and targets; in collecting ongoing longitudinal research to report on effectiveness, and in reporting the results through one common format to the province and INAC. Indicator and Business Plan development:

Efforts are currently underway to further strengthen performance measures and collection of data through the development of a comprehensive Performance Measurement Strategy for the FNCFS Program. This Strategy will be informed by the National Child Welfare Outcomes Indicator Matrix (NOM), which is being adopted by provinces. As in the past, this work will continue to be undertaken in collaboration with key program stakeholders. This Strategy will ensure alignment with agency business plans, the policy on Evaluation and the Policy on Transfer Payments and is anticipated to be completed December 2010. INAC will support agencies in updating their business plans each year in collaboration with the province and will review progress in implementing their plans through meetings during the year.

Director Social Program Reform December 2010
4. Improve communication and awareness of the purpose, processes, procedures and inherent flexibility of the EPFA to FNCFS agencies to facilitate implementation and administration of the EPFA moving forward. Improved communication and awareness

INAC is aware that new program requirements need to be clearly articulated to all parties. INAC is currently updating existing material (program manual, reporting guides) and is expected to make these materials publically available by the end of fiscal year 2010-2011; it will be posted on the INAC website.

Director Social Program Reform March 2011
5. INAC and the province discuss, clarify and formally agree (i.e. in the Tripartite Accountability Framework) to their respective roles and responsibilities with regard to training and technical support (particularly computer/IT training) for FNCFS agency staff, as well as data collection requirements. Roles and Responsibilities

INAC, as a funder, will continue to work with the provinces to establish roles and responsibilities with the remaining provinces that will transition to EPFA. INAC will continue to build relationships via tripartite tables and will assess the feasibility of entering into more formal arrangements with the provinces as well as identify other mechanisms to ensure that roles and responsibilities are clearly outlined including data collection requirements.

In all future development of Tripartite Frameworks, INAC will ensure that roles and responsibilities are included.

Director, Child and Family Services, Alberta
Director Social Program Reform
Alberta and other jurisdictions under EPFA: March 2011
 

The Management Response and Action Plan for the Implementation Evaluation of the Enhanced Prevention Focused Approach of the First Nations Child and Family Services Program in Alberta were approved by the Evaluation, Performance Measurement and Review Committee on September 24, 2010.

Return to Table of Contents





1. Introduction

1.1 Overview

This evaluation report presents the findings of the implementation evaluation of the Enhanced Prevention Focused Approach (EPFA) of the First Nations Child and Family Services (FNCFS) Program in Alberta for fiscal years (FY) 2007/08 – FY 2009/10.

The purpose of the evaluation has been to assess the implementation of the EPFA in Alberta to date, in terms of its relevance, effectiveness, the degree to which the approach represents an efficient and economical approach; and other issues that were identified through the evaluation process.

The evaluation scope covers the initial design and delivery of the EPFA, and the time period between initial implementation in 2007 to the end of FY 2009/10. The lessons learned from this implementation evaluation can inform the planned future implementation of this model in other agencies and jurisdictions.

The report is structured as follows:

1.2 Program Profile

1.2.1 Background and Description

Program Rationale

The overall framework for the delivery of child welfare services for First Nations children on reserve is governed by the constitutional arrangement whereby the provinces legislate and govern child welfare services, but the Department of Indian and Northern Affairs Canada (INAC), through the FNCFS program, funds the services for First Nations children through transfer payments in the form of contributions. In most cases, First Nations have assumed the delivery of child welfare services; however, the provinces retain jurisdiction, and delegate the authority to FNCFS agencies. INAC's approach to social development policy is guided by the general principles of reasonable comparability to the reference province or territory, and that recipients be "ordinarily resident on reserve." [Not 5]

In 2002, the Alberta government introduced the Alberta Response Model, also known as the "differential model," which represented a refocusing of child welfare to a "family-centred practice with children-centred outcomes." [Not 6] The new model included a range of potential response options to families in need of child welfare interventions, whereby an initial assessment process would determine the type of intervention required. Families and/or children would then be directed to either prevention or protection services. The model, to be successful, relies on more intensive involvement of social workers, and collaboration of community social services, as well as the voluntary and private sectors, in addressing child welfare needs. Such an approach is meant to ensure that families receive needed support before they reach crisis; that they receive them in a timely manner; that services are more tailored to family needs; and that the family is more involved in planning their service needs. [Not 7] The Alberta child welfare services model has evolved, and is now called "Family Enhancement;" the model in recent years has incorporated a casework practice focus and outcomes-based service delivery.

The model was acknowledged as an innovative approach to child welfare; one that demonstrated that early intervention, prevention and permanent placement results in positive outcomes. [Not 8]

The Alberta Child, Youth and Family Enhancement Act was passed in November 2004 as the legislative framework for the new model. [Not 9] As outlined in S.122 of the Act, the Province of Alberta and the Minister of Children's Services entered into agreements with FNCFS agencies for the purposes of providing services under the Act, on reserve. The agreements with the agencies specify that: INAC provides the funding; FNCFS agencies deliver the services; and Alberta Children and Youth Services (ACYS) delegates the FNCFS agencies, provides orientation and mandatory training, provides the software and required training, and performs quality assurance activities. In the early implementation period of the new model, which included implementation n some delegated First Nations, child intervention caseloads decreased by 22 percent in the province. [Not 10], [Not 11]

INAC, recognizing that the Alberta Response Model may represent a way to improve child and family outcomes, and in the long term, reduce escalating maintenance costs, [Not 12] devised a new, more flexible approach to funding child welfare services on reserve, and secured new investments for the implementation of the new approach, first in Alberta, then in additional regions. [Not 13] The intent is to roll out the new approach in all regions.

Program authorities: "Payments to support Indians, Inuit and Innu for the purpose of supplying public services in social development (support culturally appropriate prevention and protection services for Indian children and families resident on reserve) [Not 14] derive from Department of Indian Affairs and Northern Development Act, R.S.C. 1985, c. I-6, s.4 and subsequent policy proposals. [Not 15]

1.2.2 Program Objectives and Expected Outcomes

Program Objectives

The EPFA is designed to be a more flexible approach, which will help ensure that more First Nation children and parents in Alberta get the help they need to prevent the types of crises that lead to intervention and family breakdown. [Not 16]

The overall objective of the FNCFS program is to ensure the safety and well-being of First Nations children on reserve by supporting culturally appropriate prevention and protection services for First Nations children and families, in accordance with the legislation and standards of the province or territory of residence. The anticipated result is to have a more secure and stable family environment and improved outcomes for children ordinarily resident on reserve. The EPFA is intended to provide a better framework of protection and prevention services through increased, flexible and targeted funding for Prevention Services/Least Disruptive Measures as a new stream of funding.

Program Outcomes of the Prevention Approach:

The outcomes of the FNCFS program are illustrated in the logic model shown below. [Not 17] It is anticipated that the EPFA will better enable FNCFS agencies to deliver child and family services in accordance with provincial legislation.

The anticipated immediate outcome is increased access to services that protect children and families at risk at a standard reasonably comparable to non-First Nations communities in similar circumstances. Social workers are expected to be able to strengthen partnerships through horizontal integration with other community services/organizations for better case management to improve service delivery, and provide integrated responses to meet the real needs of First Nation children and families on reserve. The anticipated intermediate outcomes include a more secure family environment, reduced need for the removal of children from parental homes, reduced incidents of abuse, and overall improvement in child well-being. [Not 18]

Links to INAC's Program Activity Architecture and Strategic Outcomes

The FNCFS program is guided by a program logic model (refer to Figure 1). The model identifies the linkages between the program activities and the final outcomes of the program. This in turn directly supports the INAC Strategic Outcome: The People: building healthy, safe and sustainable communities.

Figure 1: FNCFS Program Logic Model

[Source: INAC, 2007. RMAF for the First Nations Child and Family Services Program]

Figure 1: Implementation Evaluation of the EPFA of the FNCFS Program

This image demonstrates the logic model for the First Nations Child and Family Services Program. The logic model shows the connection of every aspect of the programming, from inputs to final outcomes. Downward-facing arrows demonstrate how each level feeds into the other.

Performance Measurement Strategy

As part of implementing the new approach, FNCFS agencies in Alberta were required to submit and have approved (by the province and INAC) business plans for the EPFA that outlined performance indicators and targets for a number of expected outcomes. These business plans were required prior to receiving new funding and are required to be updated annually. Performance indicators are intended to be aligned with provincial and INAC data requirements, and include numbers and rates of children in care; numbers of families who no longer require intervention after a specified period; numbers of families accessing prevention through enhancement agreements and other prevention activities; and children in care by type of care (i.e. foster, kinship, institution). The development of meaningful indicators of performance that will be consistent across all agencies is still ongoing. Agencies submit annual program reports to INAC that report on progress toward targets, as well as year-end financial reports as outlined in the First Nations National Reporting Guide and the Program Manual.

1.2.3 Program Management, Key Stakeholders and Beneficiaries

Roles and Responsibilities

INAC provides funds for the delivery of child welfare services, services provided by the provinces/territories (P/T) or FNCFS agencies through flexible transfer payments. To provide a mechanism for coordinated administration and accountability of the new approach in Alberta, INAC entered into a Tripartite Accountability Framework with Alberta and the FNCFS agencies to transition to the EPFA on reserve. The framework highlights the accountability relationships, goals, strategies and targets for achieving provincial child welfare goals; and the reporting on results and outcomes. There are currently 18 FNCFS agencies in funding agreements, representing 39 of the 45 First Nations in Alberta. There are six First Nations receiving child welfare services directly from the province.

INAC Headquarters (HQ) establishes, on a national basis, the program guidelines and the terms and conditions that must be included in each funding arrangement, as well as the policy related to monitoring and compliance activities. The national office also provides oversight on program issues and monitoring of consistency between regional and national program manuals.

INAC regions, under direction of the Regional Director General, are responsible for implementing and administering the social development program (of which FNCFS is a part) in accordance with program authorities.

The province ensures compliance with provincial legislation and standards; delegates First Nations agencies; provides orientation and mandatory training; provides required computer software and the accompanying training; and participates in tripartite negotiations/meetings.

FNCFS agencies are responsible for delivering the EPFA in accordance with provincial legislation and standards while adhering to the terms and conditions of their funding agreement, and reporting to INAC to demonstrate compliance. The beneficiaries of the services, provided by the P/T and FNCFS agencies, are eligible children and families "ordinarily resident on reserve."

1.2.4 Program Resources

The total estimated funding level for the national FNCFS program during the evaluation timeframe (2007-08 to 2009-10) is $481M, $523M and $537M, respectively, or $1.5B in total. The EPFA funding is a top-up to existing funding, which provided for annual funding to Alberta agencies in the amounts of $120M, $125M and $126M, respectively.

The following table provides the amount of resources for the Enhanced Prevention Focused Approach in Alberta from 2007-08 to 2011-12.

Fiscal Year Prevention
2007-2008 $15,100,000
2008-2009 $18,300,000
2009-2010 $21,200,000
2010-2011 $21,200,000
2011-2012 & ongoing $20,200,000
 

INAC uses two main types of funding arrangements to provide funding to FNCFS agencies: Comprehensive Funding Arrangements (CFA) and Canada/First Nations Funding Agreements (CFNFA). The targeted prevention funding applies to the five-year CFNFA. INAC uses the CFA to transfer annual funding to recipients for program delivery, which is structured to include Grants, Contributions and Flexible Transfer Payments (FTP). In Alberta, all funding provided for child and family services is provided as a FTP.

Funding in Alberta is based on a provincially comparable costing model, which includes three streams: operations, maintenance and prevention.

Operations – funding supports administration, protection casework and limited prevention measures. Funding for this activity is based on a formula. The formula outlines funding for core operations (e.g., director salary, insurance, finance and human resources support, etc.), for protection (e.g., caseworker salaries, training, travel, etc), and for prevention (e.g., enhancement worker salaries, brief services).

Maintenance – funding to support (per diem and special needs) costs for Aboriginal children ordinarily resident on reserve taken into care by the agency and placed in an alternate situation outside of the parental home (ie. foster home, group homes or institutions). Placements can occur on or off reserve, but the foster home or facility must be licensed by the provincial or territorial government with jurisdiction. The Alberta Region determines the base amount of funding for maintenance each fiscal year. In 2009/10, the regional program staff reviewed individual agency requirements. This information was presented to senior management for review and approval prior to the funding arrangements being completed.

Prevention – Alberta FNCFS agencies were the first to be eligible to receive funding in this stream. As outlined previously, the amount of prevention funding is determined by formula. The formula outlines funding for core operations, protection and prevention (e.g., enhancement worker salaries, brief services). Eligible expenditures may include non-medical services designed to keep families together and children in their own homes (ie. homemaker and parent aid services, mentoring services for children, home management, non-medical counseling services not covered by other funding sources).






2. Evaluation Methodology

2.1 Evaluation Scope and Timing

The evaluation, conducted by DPRA, examined the implementation of the FNCFS Enhanced Prevention Approach implemented between FY 2007/08 – FY 2009/10 in Alberta. Terms of References (TOR) were approved by INAC's Evaluation, Performance Measurement and Review Committee in September 2009. Field work (key informant interviews and focus groups) was conducted between January 8, 2010, and March 11, 2010.

2.2 Evaluation Issues and Questions

The evaluation was guided by the following overarching themes and questions: the Evaluation Matrix illustrating which lines of evidence were used to answer the evaluation questions is presented in Appendix B.

Relevance

Design and Delivery

Effectiveness (Performance/Success)

Effectiveness (Efficiency and Economy)

2.3 Evaluation Method

2.3.1 Data Sources

The evaluation's findings and conclusions are based on the analysis and triangulation of the following multiple lines of evidence:

2.3.2 Considerations, Strengths and Limitations

Limitations to the Study

2.4 Roles, Responsibilities and Quality Assurance

The Evaluation, Performance Measurement and Review Branch (EPMRB) was the contracting authority for this evaluation and managed the evaluation in line with EPMRB's Engagement Policy and Quality Control Process. Evaluation research for this project was carried out by external consultants from DPRA.

The evaluation deliverables were reviewed by an Advisory Committee made up of representatives from INAC (HQ and regional), the Alberta government and FNCFS agencies. Preliminary findings were presented at a validation session in May 2010. Moreover, the methodology report and final report were peer reviewed by members of EPMRB who were not associated with the project.






3. Evaluation Findings - Relevance

Overview

A literature review, document and file review, key informant interviews and focus groups were employed to examine the degree to which the prevention approach addresses a demonstrable need and is responsive to the needs of First Nations children and families living on reserve.

Conclusions

Overall, the EPFA model is seen to be a move in the right direction with potential for positive outcomes. The need for an approach that enhances the safety and well-being of children and families on reserve is well documented. The need for a different approach was recognized, in light of the overrepresentation of Aboriginal children in care, and the reality that their numbers are growing; the increased number of children in care resulted in steep rises in program costs; [Not 19] and prevention represents a more cost-effective option in the long term.

The prevention approach is a more holistic response to the root causes of high numbers of children in care, which include poverty, parental addictions, and parents who are younger than their non-Aboriginal counterparts.

There was unanimous agreement among interview and focus group respondents, as well as evidence in the literature, that the prevention approach is needed in Alberta because it is more responsive to community needs and a more culturally appropriate model. While this was noted by respondents, they also noted that to be effective, the prevention approach needs more resources, more coordination of services at the community level, increased awareness among reserve populations, and a long enough time period for agencies and communities to become accustomed to the new approach.

3.1 Need for the Prevention Approach

First Nations populations are younger than the Canadian population overall and have higher birth rates than other populations in Canada. Aboriginal children are also more likely to be raised by younger parents than non-Aboriginal children. [Not 20]

Aboriginal children represent 30-40 percent of all child welfare placements nationally. [Not 21] Provincially, although Aboriginal children represent only nine percent of the Alberta child population, they represent 63 percent of children and youth in care in the province. [Not 22] Data rom the First Nations Canadian Incidence Study 2003 indicate that while rates of physical, sexual and emotional abuse are lower for First Nations children than for their non-Aboriginal counterparts, rates of neglect are dramatically higher (17 per 1,000 First Nations children compared to five per 1,000 non-Aboriginals). [Not 23] The finding that neglect is the primary type of child maltreatment experienced by First Nations children calls for a reorientation of child welfare research, policy and practice to develop culturally sensitive and effective responses, according to leading child welfare researchers. [Not 24]

The significant overrepresentation of First Nations children in substantiated child investigations and referrals to child welfare placement can clearly be related to the high level of caregiver, household and community risk factors, according to child welfare researchers. Researchers have found that higher rates of maltreatment substantiation and out-of-home placement appear to be related to the disproportionate presence of structural factors contributing to child maltreatment among First Nations children, such as poverty, poor housing and parental substance misuse. Effecting change to the current status calls for a much greater emphasis by child protection authorities on addressing such risk factors among Aboriginal families. [Not 25] According to Statistics Canada, Aboriginal women are more likely than non-Aboriginal women to be single parents. In Canada, 27 percent of Aboriginal families are headed by single mothers; 40 percent of these families earn less than $12,000/year. [Not 26]

The ACYS business plan acknowledges that "A persistent challenge facing First Nations children, youth and families living on reserve is the limited availability of, and access to, services and supports." The document expresses the expectation that the INAC‑funded prevention approach would "…enable Alberta's FNCFS agencies to provide enhanced services earlier to children, youth and families who are at risk." [Not 27]

A new model of funding FNCFS was underscored by the federal Standing Committee on Public Accounts, who, in their review of the FNCFS program, note that the new prevention funding approach is intended to be an improvement over the other existing funding formula (known as Directive 20-1), which has seen little modification since its implementation in 1988. The drawbacks to the old formula were noted as:

There was unanimous agreement among the First Nations agency and non-Aboriginal agency representatives interviewed and in focus groups conducted for this evaluation, that there is an ongoing need for the prevention approach to be employed in Alberta because it is seen to be more responsive to community needs and more culturally appropriate.

3.3 Meeting the Needs of First Nations

Interviewees and focus group participants agreed that some needs of First Nations were being met in implementing the EPFA model through activities to help strengthen families and the placement of children with family or community members. The interviewees and participants did, however, qualify these statements by stating that the extent to which the needs were met depended on the community and the agency capacity.

The majority of interviewees felt confident that a community-based approach to prevention was working, by shifting control to local entities and keeping more children in their homes. The additional funding stream for prevention was reported to be helpful to agencies in implementing the prevention approach. The majority of First Nation agency representatives interviewed, for example, indicated that the approach was addressing needs, through activities to help strengthen families, such as providing counselling, helping mothers with parenting skills, identifying other supportive resources, and holding workshops and conferences. Interviewees highlighted the importance of strengthening families rather than placing children into care, suggesting that culturally appropriate services and preventive services can have positive long term impacts on families and communities.

Focus group participants from all three Treaty areas indicated that the EPFA is less intrusive than the previous approach, allows more flexibility in adapting the approach to the community, and is community-based and culturally appropriate (i.e. in keeping children with families and in communities). Participants noted that as a result of an enhancement approach, relationships with clients are stronger through building trust over time.

However, some of the FNCFS agency interviewees believe that there are insufficient resources for prevention outreach, such as local workshops and community-based programs. In addition, respondents emphasized the need for consultation between the province, INAC and FNCFS agencies on the way in which cultural principles of child welfare could be integrated into practice.

Prevention Approach as Best Practice

Prevention child welfare models are now widely implemented internationally. British researchers characterize this recent shift in child welfare policy and practice as comprising "…flexible forms of service provision that aim to overcome vulnerability through interagency or multi-agency working, paying attention to the engaged participation of users, and the building of community capacity to counteract weak forms of social capital." [Not 29] Similar models are also cited as best practices in the United States [Not 30] and New Zealand. [Not 31]

In a First Nation context, one report articulates that it is crucial for the well-being of indigenous children, families and communities to preserve the culture and identity of indigenous children and that practices that encourage extended family care and community connection are more relevant in working with indigenous children and families. The report also states that culturally sensitive care is seen as a preventative strategy for early childhood professionals to use in order to keep children rooted in their culture and attached to their families. [Not 32]






4. Evaluation Findings - Design and Delivery

Overview

The evaluation examined the extent to which the design and delivery of the prevention approach facilitated the achievement of expected outcomes and identified factors affecting the agencies' ability to implement the approach as originally planned.

Conclusions

The evaluation found that the design of the EPFA is largely regarded as appropriate for meeting its intended outcomes. While there are some early indicators of success, some challenges will need to be addressed as the model moves forward into other jurisdictions.

In particular, the evaluation found that issues around timing, provincial requirements, human resource shortages, salary, support from government/agency management, community linkages, training, and geographical isolation were of greatest concern to FNCFS agencies and were considered to be essential to the successful implementation of the approach.

4.1 Program Design

The design of the EPFA is intended to provide individual agencies with more flexibility to shift their resources to priority areas that match their needs. The model also attempts to align FNCFS programs to those provided by the province, and provide necessary resources to offer a wider variety of culturally appropriate child and family services.

The design of the prevention approach was seen by 91.6 percent of First Nation agency interviewees as appropriate for meeting the intended outcomes, either totally (50 percent) or to some extent (41.6 percent). Among the INAC and provincial government representatives interviewed, 75 percent found the design appropriate, either totally (50 percent) or partially (25 percent).

The agencies interviewed indicated that as a result of the EPFA, funding was now provided for prevention services, enabling FNCFS agencies to hire enhancement prevention workers whose work with families can potentially keep children from being taken into the protection stream. In some cases, FNCFS agencies reported that they had been providing prevention services prior to the EPFA being implemented, using Support Agreement funding, and some interviewees indicated the funding now enabled them to hire qualified staff for prevention work.

According to interview respondents, the outcomes for the EPFA were able to be met through the following:

A common implementation challenge for FNCFS agencies is the need for specialized services at the community level (e.g. Fetal Alcohol Spectrum Disorder (FASD) assessments, medical assistance, therapy, counselling, addictions support). These services are not available through INAC funding, though they are provided by other government departments and programs either on reserve or off reserve. Thus, the agencies must spend time creating appropriate linkages to ensure that children and families are able to receive the services they require. Interview respondents and focus group participants noted that alignment of FNCFS on reserve with off reserve services in the province is an objective that was only partially achieved.

Other challenges identified for agencies include insufficient timelines to build rapport between front-line workers and families, the need for appropriate staff and funds for full implementation; as well as changing negative community/parent perceptions of prevention workers. Furthermore, the evaluation found a lack of clarity around the definition of "ordinarily resident," as some agencies were unclear on when it was appropriate to fund community members who moved off reserve.

4.2 Implementation of the Approach

When asked whether they thought the EPFA was being implemented as originally designed, the evaluation received varied responses from interview respondents. Approximately half of the FNCFS agencies interviewed felt that their business plans were effective in helping them achieve their goals related to prevention staffing and activities, though others felt that various factors were hindering their ability to fully implement the approach as initially anticipated.

Thirty-three percent of interviewees felt the business plans were not supporting effectiveness to the degree possible, [Not 33] and described challenges they faced in trying to meet their stated goals/targets, such as goals that were overly optimistic for the time period available for implementation; the need for more qualitative performance indicators that reflect the incremental nature of prevention work; and that outcome measurement should reflect the length of time it takes in the initial stages to build awareness and educate communities regarding the prevention approach.

While the business plans were developed by the First Nation child and family agencies, including target setting, they were originally done prior to implementation of the EPFA and are in the process of being updated.

Other key factors affecting the delivery of the approach, as perceived by interviewees and focus group participants, include:






5. Evaluation Findings - Effectiveness (Performance / Success)

Overview

The evaluation examined the extent to which the EPFA progressed towards its intended outcomes. The results of document/data research, key contact interviews and focus groups were used to assess the overall performance of the EPFA; examine performance measures; and identify indicators of success as well as examples of best practices. Because of the short time period of implementation being reviewed, the evaluation is limited in the extent to which effectiveness can be assessed.

Conclusions

The research is inconclusive regarding the extent to which prevention programming has been effective to date. As the model has been implemented between one and two years (depending on when each agency began implementation), it is still too early to assess performance adequately. While an increasing number of First Nations children and youth experienced permanent placements in 2007-2008 and more First Nations children in foster care or kinship care were placed with First Nations families since 2005-06, more children entered child protection between FY 2005/06 - FY 2008/09, with percentages of children entering via the family enhancement program decreasing.

Movement towards the EPFA is slowly occurring, as more culturally‑appropriate placements are taking place and knowledge of the program is spreading among families and caseworkers. A mechanism for agencies to share best practices and lessons learned was noted as being an important issue moving forward. Communication issues regarding roles and responsibilities and intended outcomes of the approach were noted between the tripartite partners and will need to be addressed so that agencies can maximize the potential of the approach.

Furthermore, several jurisdictional issues were identified as challenging the effectiveness of service delivery. Of key importance are the interpretation of on reserve and off reserve residents; the availability and access to supportive services for prevention; and determination of who is responsible for computer hardware/technical support/Information Technology (IT) training to build and support this capacity in First Nations agencies. An additional challenge is ensuring that reliable data is collected that will allow for reasonable comparability of prevention services data across provincial jurisdictions and with FNCFS agencies.

5.1 Overall Performance

Numbers of average care days, (a standard measurement of child welfare program performance), are increasing on average. Figure 5.1 below illustrates the averages for 2007/08 and 08/09. The graph shows the average number of care days (based on all 18 FNCFS agencies in the province), showing a 21.3 percent overall increase from 2007/2008 to 2008/2009.

Figure 5.1: Average Number of Care Days

Source: Indian and Northern Affairs Canada. (2010). Implementation Evaluation of the Enhanced Prevention Focused Approach in Alberta for the First Nations child and Family Services Program. Project 1570-7/09059, #3047437. Evaluation, Performance Measurement, and Review Branch. Audit and Evaluation Sector, p. 19

Figure 5.1 shows the average number of care days provided to Alberta's children and youth by all 18 FNCFS agencies in Alberta for the years 2007/2008 and 2008/2009, respectively. The y-axis indicates the average number of care days, while the x-axis show the exact number of average care days provided by the 18 FNCFS agencies. The graph shows that the number of care days has, on average, increased from 2007/2008 to 2008/2009. In 2007/2008, the 18 FNCFS agencies in Alberta provided an average of 2017.61 care days. In 2008.2009, an average of 2447.28 days was provided.

Figure 5.1 shows the average number of care days provided to Alberta's children and youth by all 18 FNCFS agencies in Alberta for the years 2007/2008 and 2008/2009, respectively. The y-axis indicates the average number of care days, while the x-axis show the exact number of average care days provided by the 18 FNCFS agencies.

The graph shows that the number of care days has, on average, increased from 2007/2008 to 2008/2009. In 2007/2008, the 18 FNCFS agencies in Alberta provided an average of 2017.61 care days. In 2008.2009, an average of 2447.28 days was provided.

As Table 5.1 below illustrates, however, an average calculation fails to illustrate wide variations in this rate from one agency to another. Table 5.1 shows the changes in average care days from FY 2007/08 to FY 2008/09 by agency. There are many variables affecting the numbers of children in care, including population influxes, and the availability of support services in communities. Some agencies are notable for significant changes year over year, either large increases or significant decreases. Of the 18 FNCFS agencies, five (27 percent) have shown decreases in the Average Number of Care Days between the 2007/08 and 2008/09 periods.

Figure 5.2: Average Number of Care Days (2007/08 – 2008/09)
by FNCFS agency
[Not 34]

Source: Indian and Northern Affairs Canada. (2010). Implementation Evaluation of the Enhanced Prevention Focused Approach in Alberta for the First Nations child and Family Services Program. Project 1570-7/09059, #3047437. Evaluation, Performance Measurement, and Review Branch. Audit and Evaluation Sector, p. 19

Figure 5.2 shows the changes in the average number of care days from 2007/08 to 2008/09 by agency. Each of the 18 FNCFS agencies are represented by a set of two bars on the x-axis, with each bar indicating the average number of care days for the year 2007/08 and 2008/09, respectively. The average number of care days is measured on the y-axis. The figure illustrates that five agencies show decrease in the average care days from 2007/08 to 2008/09, while thirteen agencies show increases over the same period.

The average number of care days in the AKO agency totalled 1564 in 2007/08, and 1147 in 2008/09. The average number of care days in the ATC agency totalled 646 in 2007/08, and 465 in 2008/09. The average number of care days in the Bigstone agency totalled 1527 in 2007/08, and 589 in 2008/09. The average number of care days in the Blood Tribe agency totalled 2013 in 2007/08, and 2666 in 2008/09. The average number of care days in the KCWS agency totalled 4967 in 2007/08, and 6696 in 2008/09. The average number of care days in the KTC agency totalled 982 in 2007/08, and 1426 in 2008/09. The average number of care days in the LRRCN agency totalled 2053 in 2007/08, and 1891 in 2008/09. The average number of care days in the NPTC agency totalled 1342 in 2007/08, and 2077 in 2008/09. The average number of care days in the Piikani agency totalled 683 in 2007/08, and 961 in 2008/09. The average number of care days in the Saddle Lake agency totalled 1436 in 2007/08, and 3255 in 2008/09. The average number of care days in the Siksika agency totalled 3456 in 2007/08, and 3937 in 2008/09. The average number of care days in the Stoney agency totalled 7922 in 2007/08, and 9610 in 2008/09. The average number of care days in the TCCFS- East agency totalled 1064 in 2007/08, and 1209 in 2008/09. The average number of care days in the TCCFS- West agency totalled 777 in 2007/08, and 1054 in 2008/09. The average number of care days in the Tsuu T'ina agency totalled 1091 in 2007/08, and 1705 in 2008/09. The average number of care days in the WCTC agency totalled 496 in 2007/08, and 1054 in 2008/09. The average number of care days in the YTSA agency totalled 3543 in 2007/08, and 3069 in 2008/09.

Table 5.1: Average Care Days (% change, 2007/08 - 2008/09)
Agency Avg. # of Care Days 2007/2008 Avg. # of Care Days 2008/2009 % Change
Source: EDMONTON-559870-CFS FUNDING CALCULATIONS - PROT + ENH + PROV.XLS
NB: 2008/2009 formula = Monthly Avg. # of Care Days for Foster Care (FC), Based on Highest # of FC children * 31 days. 2007/2008 numbers are based on Avg. # of Care Days.
Agency #1 1436 3255 126.62%
Agency #2 496 1054 112.71%
Agency #3 755 1240 64.31%
Agency #4 1091 1705 56.23%
Agency #5 1342 2077 54.78%
Agency #6 982 1426 45.20%
Agency #7 683 961 40.72%
Agency #8 777 1054 35.59%
Agency #9 4967 6696 34.80%
Agency #10 2013 2666 32.46%
Agency #11 7922 9610 21.31%
Agency #12 3456 3937 13.92%
Agency #13 1064 1209 13.67%
Agency #14 2053 1891 -7.91%
Agency #15 3543 3069 -13.38%
Agency #16 1564 1147 -26.65%
Agency #17 646 465 -27.96%
Agency #18 1527 589 -61.43%
 

Without statistics regarding numbers of children in care, and complete data on performance reporting on FNCFS indicators, such as percentage decrease in case loads based on the prevention approach, it is not possible to say with confidence whether prevention outcomes are being reached.

Progress and final reports for the review period do show preliminary signs of positive changes, as evidenced through the following community-level best practices carried out by some agencies; however, attribution to the prevention approach is challenging:

According to interviewees, change towards the EPFA is slowly occurring, with more culturally‑appropriate placements taking place and with knowledge of the program spreading among families and caseworkers. Some First Nations agency interviewees stressed the importance of having a means to share knowledge and experience of best practices and challenges with other agencies.

Challenges Identified

A number of challenges for the implementation of the EPFA were also raised by interviewees and focus group participants. While prevention workers are intended to build partnerships and work toward integrating supports for families at risk, they often find that the necessary supports are not present in their community. Interviewees and focus group respondents believed that there is the need for, and frequent lack of, supportive program resources at the community level to offer support and counselling to families; provide training for families and youth; and offer specialized services. Specialized services can include FASD assessments, medical assistance, therapy, access to youth workers, psychological counselling, and addictions support.

Staff find they must dedicate considerable time and effort building awareness among community members. The need for this initial awareness-raising, and that fact that staff time has been dedicated to this in many cases, is a factor to take into account in measuring the effectiveness of the implementation over a two year period.

Strengthening linkages and support among INAC, the province and FNCFS agencies were noted by non-Aboriginal interviewees as important for enhancing the success of the EPFA. For example, tripartite meetings were to be held quarterly to review business plans, progress achieved and corrective actions required, but did not always occur. First Nation agency interviewees indicated that agencies were subject to many changes since the introduction of the preventative approach and felt that more meetings, interactions, shared experiences and successes would be of benefit for the expected outcomes. Some interviewees ascribed this to staff shortages at the INAC regional office.

The evaluation also found that there are different perceptions between INAC, the Alberta government and FNCFS agencies of what the EPFA is, how it can be implemented and how it was designed to meet First Nations needs within a provincial context. In order for FNCFS agencies to fully tap into the potential of the approach, INAC must clearly communicate roles and responsibilities and intended outcomes so that all parties have the same understanding moving forward.

Unintended Impacts

When asked what unintended impacts the implementation of the EPFA was having in communities, interviewees and focus group participants spoke mainly of improved community relationships. They noted that in some cases, parents were becoming less fearful of child welfare agencies, and were self-identifying for family enhancement programming. The new funding for EPFA has allowed some agencies to organize community outreach activities, which was mentioned as one of the ways agencies were able to change negative attitudes towards their line of work. In other cases, interviewees and focus group participants observed that not enough time has passed for this change in attitude to have occurred in their communities.

Moreover, some interviewees felt that the implementation happened too quickly, and that there was not enough time to consider all the associated costs and impacts of implementing a new approach. Moving forward, adjustments will need to be made through the evolution of business plans and strengthened partnerships between agencies, the province and INAC to increase positive outcomes for children and families.

5.2 Performance Measurement

Reporting for the FNCFS program includes annual progress reports and final reports, as well as quarterly maintenance reports, and when necessary, ad-hoc plans for using cumulative surpluses and/or deferred revenue. In addition to the program reporting, there are various case work reports that are required at the provincial level and through the provincial CYIM/ISIS system, such as ongoing case analysis reports and detailed analysis reports, as well as reports required from the agencies' Boards of Directors. FNCFS agency interviewees noted that all this reporting required significant staff time and capacity.

INAC is in the process of revising its FNCFS compliance directive, with anticipated completion by December 2010. Training sessions on the National Program manual and other additional reporting requirement changes are also expected to be completed by December 2010. A national FNCFS Information System is in preliminary approval stages and could also add to the ability of the region to better manage the program. [Not 37]

The business plans developed at the start of the implementation process are a key tool in performance measurement, as they require the establishment of baseline data and the setting of performance targets. Business plans reviewed for the evaluation showed that a number of agencies had not established baseline measures.

There is not a consistently used set of performance indicators used by all agencies, although there is considerable overlap, based on indicators and performance measurement guidelines that are part of the business plan templates used by most agencies to develop their business plans.

More data is needed at the FNCFS agency level in order to determine the effectiveness of the EPFA performance by tracking caseload trends against prevention activities. Although a drop in protection cases could not definitively be attributed to the EPFA, [Not 38] it is accepted as an indicator of program success overall.

Success Indicators

Interviewees from FNCFS agencies mentioned a number of commonly used indicators of success, such as fewer children going into care, more children leaving care (and not requiring intervention again for a specified period of time), reduced caseload numbers, more children in permanent placement in the community, and fewer days in care. Respondents generally felt that these measures were an incomplete record of levels of success in that they did not measure the subtleties of ongoing preventive work with families and communities to keep children in their homes, such as a trusting relationship between a troubled family and a supportive worker. Some interviewees and focus group participants suggested that each community should develop its own measures of success, as "people are benefitting in ways that can't always be measured."

In non-Aboriginal agency interviews, respondents listed the following performance indicators as among those commonly used:

Both the literature and some non-Aboriginal respondents felt that standardized measures on a national basis were needed, and that, beyond this, that culturally appropriate measures should be adopted, which measure the impacts that are not easily quantifiable.

5.3 Jurisdictional Issues

The evaluation found that interviewees frequently identified jurisdictional issues as challenging the effectiveness of service delivery. The evaluation concluded that not all the jurisdictional issues identified by respondents were specific to the implementation of the EPFA, and the evaluation has focused only on those that are so. However, as jurisdictional issues are clearly significant in achieving alignment, these issues should be included in the planned strategic evaluation of the FNCFS program.

Those jurisdictional issues that the evaluation considers of direct importance to the implementation of the EPFA approach include: the interpretation of on-reserve and off-reserve residents; the availability and access to supportive services for prevention; and determination of who is responsible for computer hardware/technical support/IT training to build and support this capacity in First Nations agencies.

There is a lack of clarity related to the interpretation of and funding for on and off reserve residents. For example, the movement of individuals ordinarily resident on reserve to off reserve urban centres and other locations creates jurisdictional (funding) issues. In another example, non‑Aboriginal populations move to the reserve, but these children and families are not counted in the funding allocations. Should these children and families require the services of a FNCFS agency, the agency is required to provide the services and absorb the costs. [Not 39] Similarly, FNCFS agencies do not get funded for non-status children on reserve, but also provide services for them.

Some FNCFS agencies located in close proximity to Child and Family Service Authority (CFSAs) and urban centres have established partnerships for referrals and joint program delivery with CFSAs and voluntary sector organizations; however, other agencies further from urban centres lack such access, and have little of the necessary supportive programming for full implementation of the prevention approach. These agencies need support and increased access to services; however, it is presently unclear what role the province and/or INAC should play in addressing the issue as many of the services are funded by other government departments.

Another jurisdictional issue that came to light after the negotiation of the Tripartite Agreement relates to responsibility over computer hardware, IT training and support. It is unclear to what extent either INAC or the province has responsibility over these issues. The province continues to provide training for data entry into its new data system (ISIS), but many computers require upgrades, and ongoing support services will be required to ensure accurate data collection.

Moreover, interviewees from FNCFS agencies, the provincial government and INAC have all noted the importance of strengthening relationships and working collaboratively to achieve the expected outcomes of the EPFA. A significant jurisdictional hurdle relates to the importance of data collection and the sharing of relevant information. As discussed under Section 6.3 of this report, the evaluation was not able to use data that was provided by the province to compare Family Enhancement services or the number of children in care in CFSAs and FNCFS agencies.

This inability to compare services speaks to the complex nature of funding federal programs that are provincially legislated, and to reconcile, this will require a considerable amount of partnership building between the federal and provincial governments as well as FNCFS agencies. However, as the immediate expected outcome of the program is to "increase access to services that protect children and families at risk at a standard reasonably comparable to non-First Nations communities in similar circumstances," the program will need to develop a method to effectively measure this outcome, particularly as the approach rolls out into other jurisdictions.






6. Evaluation Findings - Effectiveness (Efficiency and Economy)

Overview

The section below outlines the findings triangulated from the various lines of evidence in response to the question of whether there are more cost-effective ways of implementing the Enhanced Prevention Focused Approach of the FNCFS program in Alberta.

The constraints on determining cost-effectiveness of a prevention approach are clearly acknowledged in the literature. INAC indicated that cost savings would likely only begin to be evident at year five of implementation (i.e. 2012); and furthermore, that the costs of providing child welfare services in First Nations should be expected to be higher, because of a higher level of need. [Not 40] Because of the short period of implementation of the EPFA in Alberta, it is too early to be able to assess the cost-effectiveness of the model; although some agencies have reduced child protection rates in the period, thereby reducing maintenance costs.

Conclusions

It is acknowledged in the literature that cost‑effectiveness of prevention approaches can only be assessed after a lengthy period of time. Less than three years into the implementation of the model (and less than two years for some agencies), it is too soon to say whether the EPFA is achieving cost savings.

Nevertheless, there are signs that a small number of agencies have reduced the numbers of children in care, thereby reducing maintenance costs. The evaluation was unable to determine whether these reductions in protection cases are a direct result of prevention activities; however, in the agencies that showed these reductions, there were associated increases in prevention spending.

There is also evidence that some FNCFS agencies are achieving cost savings by integrating programming, cost-sharing, and other creative ways to get the most from available prevention funds. Some examples include using their own resources for community-strengthening activities and skills training; cost-sharing activities with community programs; as well as integrating community and medical services to enhance their work.

While the evaluation team was able to access some provincial data on Family Enhancement caseloads and children in care, it was determined that the numbers provided were not sufficiently reliable to be included in the evaluation findings. Thus, the evaluation was not able to adequately assess the comparability of prevention services between First Nations agencies and provincial child welfare services. In order to obtain the most pertinent information available for decision making, the program should collaborate with the provinces to share relevant data.

6.1 Cost-Effectiveness in Prevention Approaches

While initially more costly, in the long term, a prevention approach (broadly defined) is designed to deliver cost savings. In addition to the evidence provided by Alberta's early achievements in reducing numbers of children in care through employing the Alberta Response Model, the literature also cites the use of Alternative Response [Not 41] models in the United States. The use of preventive approaches that invest in strengthening families is noted as not only saving child protection costs in the short term, but other social program costs in the long term, such as youth and adult corrections. [Not 42]

Evaluations of programs in the United States showed 17:1 returns on early childhood development programs. Best practice in a prevention approach is represented by comprehensive, integrated programming that takes into account the needs of the whole family, and beyond that, to the community context for healthy and stable family life. [Not 43] A First Nations example of a prevention approach put in place by a child and family services program in Manitoba shows significant savings in the long run on many fronts, including preventing large numbers of medium-to-high risk children from entering formal care. [Not 44]

The cost of providing child welfare services should be expected to be higher, as the needs dictate higher rates of intervention. [Not 45] In addition, it is acknowledged that it is unlikely that strong evidence of cost‑effectiveness will be available until longitudinal studies can demonstrate this over a longer period of time.

6.2 Funding

Maintenance funding, as it is largely based on historical trends of actual costs, showed a range of change year over year; however, maintenance funding increased from FY 2007/08 to FY 2008/09 for all agencies, ranging from a low of .78 percent to a high of 4.75 percent. Maintenance funding allocations for FY 2007/08 and FY 2008/09 were based on 2006/2007 actual maintenance expenditures and adjusted for growth based on youth population for children ages 0-18. The design of the model provides the opportunity for agencies who achieve maintenance surpluses to reinvest these dollars in prevention. Those agencies that have reduced maintenance costs show an associated increase in enhancement funds. [Not 46]

The majority of respondents (75 percent of FNCFS agencies and 63.5 percent of non-FNCFS agency respondents) felt that funding for the EPFA is not sufficient to achieve intended outcomes, nor is the funding model, as currently designed, flexible enough to accommodate the varying needs of the agencies, who range from small and remote to large and close to urban areas.

INAC's capacity funding is determined on a per capita basis, [Not 47] with the result that smaller communities that often have the most capacity needs receive less funding for capacity building. Furthermore, the funds are accessed through proposals, again giving the advantage to agencies with more capacity. The capacity factor is one of the key variables in the extent to which FNCFS agencies are able to implement the EPFA model. For example, one agency that serves three First Nations had one prevention worker, whose role is intended to include collaboration with other community family service agencies for an integrated approach to prevention, in addition to working directly with families in all three communities.

Moreover, as EPFA funding is based on an annual allocation for most aspects with some pieces being determined by a formula, there is not the flexibility to respond quickly to changes in provincial policy or other external drivers, despite being a more malleable approach overall. This lack of flexibility for quick response to externalities is common to INAC programs that adhere to provincial legislation and was noted in the program Results-based Accountability Framework as an in-built risk to the program. [Not 48]

Notwithstanding the issues described above, some FNCFS agencies reported that they were able to maximize the allocated funding they received though a combination of:

6.3 Comparability to Provincial Services

The evaluation was not able to adequately assess the alignment of prevention services provided by First Nations agencies and provincial child welfare agencies, due to issues of reliability with the provincial data and a shortage of comparable First Nations data.

Data

The provincial data received by the evaluation team included statistics on Family Enhancement caseloads and the number of children in care. The main concern regarding the Family Enhancement section is that the number of average monthly caseloads is not reflective of the total number of children who received Family Enhancement services in any of the years considered (2006-07 to 2009-10). As for the number of children in care, the total was derived using the number of children who received Family Enhancement services as well as child protection services. Where a child was subject to both types of services, they were counted twice, leading to an over-inflation of the number of children in care.

What is more, the basis of comparison for the First Nation and provincial populations would have had to come from the Indian Registry and the 2006 Census, respectively, which would also have yielded methodological anomalies, particularly when trying to compare trends over time.

Services

Interview and focus group respondents noted that alignment with provincial off‑reserve services in the province was an objective their agencies were only partially able to achieve. The FNCFS agencies are challenged in implementing the EPFA because they do not always have the same access to other/linked services for the additional supports that families need. As an example, it was noted that small, rural, non-Aboriginal communities were able to access contract service providers from a provincial list, but the FNCFS agencies could not. Other services more readily accessed off reserve but are not part of FNCFS programming at INAC were mental health specialists and volunteer services.

It was noted in interviews that alignment of services between the province and FNCFS agencies has historically been as issue but that there have been improvements, with more consistency across the board. One continuing need noted by respondents was that infrastructure needs that would support prevention, such as office space for prevention workers must be developed on reserve by FNCFS agencies. Funding for on‑reserve infrastructure is available under INAC's Capital Facilities and Maintenance Program.






7. Conclusions and Recommendations

The evaluation was focused on assessing the early implementation of the EPFA in Alberta, as a way of learning lessons that can be applied on an ongoing basis in Alberta, and as the model is rolled out in other jurisdictions.

The evaluation supports the following conclusions regarding the relevance, design and delivery, performance and effectiveness of the program under the EPFA.

Relevance

The need for an approach that enhances the safety and well-being of children and families on reserve is well documented, based on the over-representation of First Nations children in care, and the reality that their numbers are growing, with associated escalation of program costs. There was evidence in the literature review, as well as unanimous agreement among interview and focus group respondents that the prevention approach is needed in Alberta because it is more responsive to community needs and a more culturally appropriate model.

The prevention approach not only aligns with current best practices in child welfare internationally, but represents a potentially more cost-effective option in the long term. The prevention approach is more effective at addressing the root causes of high numbers of children in care, which include poverty, parental addictions, and parents who are younger than their non‑First Nations counterparts. This is consistent with the Government of Canada objective of achieving healthy, safe and sustainable communities.

Design and Delivery

The evaluation found that the design of the EPFA is largely regarded as appropriate for meeting its intended outcomes. While there are some early indicators of success, some challenges will need to be addressed as the model moves forward into other jurisdictions.

In particular, the evaluation found that issues around timing, provincial requirements, human resource shortages, salary, support from government/agency management, community linkages, training, and geographical isolation were of greatest concern to FNCFS agencies and were considered to be essential to the successful implementation of the approach.

Effectiveness (Performance/Success)

The research is inconclusive regarding the extent to which prevention programming has been effective to date. As the model has been implemented between one and two years (depending on when each agency began implementation), it is still too early to assess performance adequately. While an increasing number of First Nations children and youth experienced permanent placements in 2007-2008 and more First Nations children in foster care or kinship care were placed with First Nations families since 2005-06, more children entered child protection between FY 2005/06 - FY 2008/09, with percentages of children entering via the family enhancement program decreasing.

Movement towards the EPFA is slowly occurring, as more culturally‑appropriate placements are taking place and knowledge of the program is spreading among families and caseworkers. A mechanism for agencies to share best practices and lessons learned was noted as being an important issue moving forward. Communication issues surrounding roles and responsibilities and intended outcomes were also noted between the tripartite partners and will need to be addressed in order for agencies to maximize the potential of the approach.

Furthermore, several jurisdictional issues were identified as challenging the effectiveness of service delivery. Of key importance are the interpretation of on-reserve and off-reserve residents; the availability and access to supportive services for prevention; and determination of who is responsible for computer hardware/technical support/IT training to build and support this capacity in First Nations agencies. Challenges with the data also exist; specifically reliability issues for the provincial data and lack of comparable data for First Nations.

Effectiveness (Efficiency/Economy)

It is acknowledged in the literature that cost‑effectiveness of prevention approaches can only be assessed after a lengthy period of time. Less than three years into the implementation of the model (and less than two years for some agencies), it is too soon to say whether the EPFA is achieving cost savings.

Nevertheless, there are signs that a small number of agencies have reduced the numbers of children in care, thereby reducing maintenance costs. The evaluation was unable to determine whether these reductions in protection cases are a direct result of prevention activities; however, in the agencies that showed these reductions, there were associated increases in prevention spending.

There is also evidence that some FNCFS agencies are achieving cost savings by integrating programming, cost-sharing, and other creative ways to get the most from available prevention funds. Some examples include using their own resources for community-strengthening activities and skills training; cost-sharing activities with community programs; as well as integrating community and medical services to enhance their work.

While the evaluation team was able to access some provincial data on Family Enhancement caseloads and children in care, it was determined that the numbers provided were not sufficiently reliable to be included in the evaluation findings. [Not 49] Thus, the evaluation was not able to adequately assess the comparability of prevention services between First Nations agencies and provincial child welfare services. In order to obtain the most pertinent information available for decision making, the program should collaborate with the provinces to share relevant data.

7.2 Recommendations

It is recommended that INAC:

  1. Revisit the EPFA funding model within the next year to learn from the past two years of implementation and incorporate resources for things such as a remoteness factor that may address some of the issues faced by rural and remote communities. As part of this review, INAC should also determine if the calculations that are based on an assumed population of children in care are relevant in achieving desired outcomes.
  2. In partnership with the province and other federal agencies, if applicable, maximize the use of scarce capacity building funds to strategically invest in developing the capacity of FNCFS agencies, bearing in mind that some agencies have higher capacity needs than others.
  3. Ensure that First Nations agencies are sufficiently supported in the development of their business plans, including establishing baseline data on common indicators for prevention in the start-up phase; that business plans are updated annually and adjusted based on the feasibility of indicators and targets; in collecting ongoing longitudinal research to report on effectiveness, and in reporting the results through one common format to the province and INAC.
  4. Improve communication and awareness of the purpose, processes, procedures and inherent flexibility of the EPFA to FNCFS agencies to facilitate implementation and administration of the EPFA moving forward.
  5. INAC and the province discuss, clarify and formally agree (i.e. in the Tripartite Accountability Framework) to their respective roles and responsibilities with regard to training and technical support (particularly computer/IT training) for FNCFS agency staff, as well as on data collection requirements.





Appendix A - Evaluation Terms of Reference

1. Overview

The implementation evaluation of the Enhanced Prevention Focused Approach (EPFA) for the First Nations Child and Family Services Program in Alberta is being launched in the fall of 2009-10 and will be completed in Spring 2010. This will be followed by a summative evaluation of the First Nations Child and Family Services Program (FNCFS) in 2010-11.

In May 2008, the Office of the Auditor General (OAG) presented findings of their audit of the FNCFS program. The audit concluded that INAC does not have assurance that the program provides on-reserve First Nations children and families with child welfare services that are culturally appropriate, reasonably comparable with those provided off-reserve in similar circumstances, and delivered in accordance with relevant provincial legislation and standards.

Furthermore, INAC appeared before the Standing Committee on Public Accounts in February 2009, to provide an update on how the department is moving forward on the issues raised by the OAG. The Committee then issued a report with recommendations and proposals for change. INAC tabled their response to the report in Parliament in August 2009.

2. Program Description

2.1 Background and Scope/Activities

In Alberta, there are 18 delegated First Nations Child and Family Services (FNCFS) agencies delivering services to 40 of the 47 First Nations. S. 122 (2) of the Child, Youth and Family Enhancement Act (which replaced the Child Welfare Act in 2004)allows the Province of Alberta and the Minister of Children's Services to enter into an agreement, for the purposes of providing services under the Act, on a reserve. These agreements must be in place in order for INAC to provide funding.

Under the Agreements, FNCFS agencies are required to follow provincial policy and meet the Child Intervention Standards. The Agreements clearly identify roles and responsibilities of all parties (INAC, Alberta and FNCFS agencies). INAC provides the funding; FNCFS agencies deliver the services; Alberta Children's Services delegates the FNCFS agencies, provides orientation and mandatory training, provides the software and training to use the provincial Child Youth Information Module (CYIM) and performs quality assurance activities. Since INAC is not required to be included in these agreements, there are also a number of bilateral agreements between the province and the FNCFS agencies. However, INAC receives a copy of the agreement to ensure the agency is eligible for funding.

Since the introduction of the Alberta Response Model in 2001 and the new provincial legislation in November 2004, Alberta's child intervention caseloads in provincial Child and Family Services Agencies decreased by 22 percent. During the same period, Treaty 6, 7 & 8 First Nation Child and Family Services Agencies total caseloads increased by 4 percent. From 2002– 2006, the number of on reserve out of home care days increased by 23 percent. Additionally, this has resulted in a significant increase in regional program costs which is largely attributable to high cost out of home placements in group homes and institutions.

In May 2006, the Minister of Indian and Northern Affairs Canada met with Alberta's the Minister of Children's Services who indicated that a differential response appeared to be effective in reducing child intervention caseloads off reserve. INAC expressed a strong interest in working with interested FNCFS agencies on a more flexible, Child, Youth & Family Enhancement (early intervention and prevention) focused approach. A joint working group was then established with the goal of developing a family enhancement approach suited to First Nation children and families on-reserve and engaging the principles and policies of the Child, Youth and Family Enhancement Act. In 2007, INAC entered into a Tripartite Accountability Framework with the province of Alberta, and First Nation Child and Family Service Agencies, and received funding to support the transition to an enhanced prevention focused approach on reserve. It is expected that by fiscal year 2012-13, most, if not all jurisdictions will have signed on to the prevention-focused approach on reserve.

INAC's FNCFS programming is funded through the following authority: Payments to support Indians, Inuit and Innu for the purpose of supplying public services in social development (support culturally appropriate prevention and protection services for Indian children and families resident on reserve).

2.2 Program Objectives and Expected Outcomes

The objective of the FNCFS program is to ensure the safety and well-being of First Nations children on-reserve by supporting culturally appropriate prevention and protection services for First Nations children and families, in accordance with the legislation and standards of the province or territory of residence with an anticipated result of having a more secure and stable family environment and improved outcomes for children on-reserve.

The FNCFS program is identified in INAC's Program Activity Architecture under the strategic outcome of "The People," and immediate, intermediate and long-term expected results are to contribute to the strategic outcome of building healthy, safe and sustainable communities.

2.3 Program Management, Key Stakeholders and Beneficiaries

Headquarters establishes on a national basis, the program guidelines, the terms and conditions that must be included in each funding arrangement, as well as the policy related to monitoring and compliance activities. The specific role of headquarters is to:

With the support of regional staff, the Regional Director General (RDG) in each region is responsible for implementing and administering the social development program in accordance with the guidelines issued by the program managers at headquarters, which includes, for example, assessing the eligibility of recipient applications, entering into financial arrangements with approved recipients in accordance with the transfer payment program terms and conditions, and monitoring, collecting and assessing both the financial and program performance results of individual recipients, and taking appropriate remedial action.

FNCFS falls within provincial jurisdiction. It is the role of the province or territory to:

FNCFS agencies are responsible for delivering the FNCFS program in accordance with provincial legislation and standards while adhering to the terms and conditions of their funding agreement. FNCFS agencies and service providers include First Nations (as represented by Chiefs and Councils); their organizations such as Tribal Councils or agencies (such as Child and Family Services agencies in various communities).

INAC does not fund the NWT or Nunavut. Transfer payments to the NWT and Nunavut include funding for First Nations. The delivery of FNCFS varies from province to province according to the specific needs of the First Nations communities within a particular region.

Eligible recipients for FNCFS funding are:

Self-government First Nations that have included child and family services in their self-government agreements are not eligible recipients.

Beneficiaries of this program include at-risk First Nations children and their families on-reserve that require access to child prevention/least disruptive measures services and/or child placement out of the parental home.

2.4 Program Resources

The total estimated funding level for the FNCFS program in 2009-10 is $537M.

The following table provides the amount of resources for the Enhanced Prevention Focused Approach in Alberta from 2007-08 to 2011-12.

Fiscal Year Prevention
2007-2008 $15,100,000
2008-2009 $18,300,000
2009-2010 $21,200,000
2010-2011 $21,200,000
2011-2012 & ongoing $20,200,000
 

INAC uses two main types of funding arrangements to provide funding to FNCFS agencies: Comprehensive Funding Arrangements (CFA) and Canada/First Nations Funding Agreements (CFNFA). The targeted prevention funding applies to the 5-year CFNFA. INAC uses the CFA to transfer annual funding to recipients for program delivery which is structured to include Grants, Contributions and Flexible Transfer Payments (FTP). In Alberta, all funding provided for child and family services is provided as a FTP.

Funding in Alberta is based on a provincially comparable costing model which includes three streams: operations, maintenance and prevention.

Operations – funding supports administration, protection casework and limited prevention measures. Funding for this activity is based on a formula. The formula outlines funding for core operations (e.g., Director salary, insurance, finance and HR support, etc.), for protection (e.g., caseworker salaries, training, travel, etc) and for prevention (e.g., enhancement worker salaries, brief services).

Maintenance – funding to support (per diem and special needs) costs for Aboriginal children ordinarily resident on reserve taken into care by the agency and placed in an alternate situation outside of the parental home (ie. foster home, group homes or institutions). Placements can occur on or off reserve, but the foster home or facility must be licensed by the provincial or territorial government with jurisdiction. The Alberta Region determines the base amount of funding for maintenance each fiscal year. In 2009/10, the Regional Program staff reviewed individual agency requirements. This information was presented to senior management for review and approval prior to the funding arrangements being completed.

Prevention – Alberta FNCFS agencies were the first to be eligible to receive funding in this stream. As outlined previously, the amount of prevention funding is determined by formula. The formula outlines funding for core operations, protection and prevention (e.g., enhancement worker salaries, brief services). Eligible expenditures may include non-medical services designed to keep families together and children in their own homes (ie. homemaker and parent aid services, mentoring services for children, home management, non-medical counseling services not covered by other funding sources).

3. Evaluation Methodology

3.1 Evaluation Scope and Duration

The evaluation will examine the initial design and delivery of the approach, relevance, performance to date as well as cost-effectiveness. Timelines for the evaluation are from 2007-08 to 2009-10.

3.2 Evaluation Issues

The evaluation will focus on the following issues: design and delivery, relevance, success and cost-effectiveness. Specific evaluation questions will be identified in the development of the project's detailed methodology report and work plan. Other evaluation issues may also be identified during this period.

  • Design and Delivery

    The evaluation will look at the design of the EPFA in Alberta and whether the program is well-developed to achieve expected results. The evaluation will look at whether the program has been delivered as planned and whether the proper resources are in place to implement the approach. It will assess the performance data collected and whether INAC is properly reporting and measuring outcomes. Finally, the evaluation will consider best practices as well as limitations in the design and delivery of the program.
  • Relevance

    The evaluation will assess the relevance of the Enhanced Prevention Focused Approach and examine the extent to which it realistically addresses the existing needs and priorities at various levels.
  • Performance

    The evaluation will look at the expected and unexpected results that have taken place to date and consider whether these could be applicable in other jurisdictions. Performance measurement data will be examined to determine where there can be comparability at a national level. The extent to which FNCFS agencies are meeting provincial service standards and legislation will be reviewed, as well as whether the proper measures/tools are in place to provide a comprehensive comparison of FNCFS funding with provincial funding for similar agencies. Furthermore, the evaluation will examine whether gender-specific initiatives are being offered to children and families receiving services.
  • Cost-Effectiveness

    The evaluation will look at the cost of preventative services for children and families on reserve in Alberta, and attempt to compare the funding of FNCFS agencies and provincial funding for children and families living in similar circumstances.

3. 3 Evaluation Method

3.3.1 Data Sources

Subject to further development in the detailed methodology and work plan, the evaluation findings and conclusions will be based on the analysis and triangulation of the following lines of evidence.

Literature/Document/Data/File Review:

This line of evidence will inform the evaluation findings and assist in the development of program profiles, contextual background and case studies. The study will comprise an initial review and synthesis of works identified by EPMRB. Among the key documents and types of documents to be included in this review are:

  • Proceedings / Report of the Standing Committee on Public Accounts;
  • Policy Documents;
  • CFA / CFNFAs / FFOMs;
  • Business Plans from FNCFS agencies
  • Provincial/Territorial/Aboriginal policies, programs, plans, reports, strategies and initiatives;
  • Tripartite Accountability Framework;
  • Previous evaluation and audits;
  • Program reports (and recipient reporting guides);
  • Terms and Conditions;
  • National and regional program manuals;
  • Databases: provincial, departmental and national (for comparability);
  • Program and project documents (e.g.: operational plans, strategic plans, RMAFs, performance measurement strategies, among others); and
  • Office of the Auditor General Reports.

If necessary this review may be supplemented by additional research.

Key informant interviews:

It is expected that approximately 30 interviews will be conducted with INAC officials, representatives from First Nations Child and Family Services organizations, Provincial/Territorial governments, Aboriginal Organizations, as well as subject matter experts (academic).

Case Studies:

Three case studies will be conducted – one for each Treaty Area (6, 7, 8). Case studies will be conducted to provide an in-depth look at the implementation of the approach in different parts of the region, to identify factors which have facilitated or hindered the implementation of the approach, as well as to examine best practices and lessons learned from front-line, provincial and regional perspectives.

Communities will be selected in line with the following criteria:

  • Regional representation (three (3) Treaty Areas);
  • Examples of 'best cases' as well as 'lessons learned;'
  • Community size (population);
  • Proximity to/distance from major urban centres; and
  • Number of children and families accessing FNCFS in order to get a comprehensive understanding of the need on reserve.

The Case Studies will include:

  • Interviews with representatives from INAC (Alberta region), provincial government, Treaty Areas and FNCFS agencies;
  • Community visits with two (2) communities per Treaty Area, including a visit to their FNCFS agencies and/or other relevant community facilities; and
  • Focus groups with FNCFS agencies' front-line staff.
3.3.2 Considerations, Strengths and Limitations

This evaluation is intended to be an in-depth analysis of the Alberta region's implementation of the Enhanced Prevention Focused Approach. Its applicability to other regions will be in the lessons learned and best practices as opposed to the consideration of a national strategy.

Findings from this evaluation will be considered in next year's national First Nations Child and Family Services evaluation.

Due to the collaborative nature of the new approach, the evaluation team will work closely with provincial staff, Aboriginal organizations and front-line workers to ensure the most comprehensive and valuable information is collected within the timeframe permitted.

Due to the especially sensitive nature of the evaluation issues, all precautions to ensure the privacy and confidentiality of evaluation participants, children and families will be taken.

This evaluation will consider INAC's Sustainable Development Strategy's objective of enhancing social and economic capacity in Aboriginal communities through social programming. The evaluation will furthermore adhere to INAC's Policy on Gender Based Analysis.

4. Project Management and Quality Control

The Evaluation, Performance Measurement and Review Branch (EPMRB) of INAC's Audit and Evaluation Sector (AES) will be the project authority for the EPFA evaluation, and will manage the evaluation in line with EPMRB's Engagement Policy and Quality Control Process.

EPMRB will request input from the Social Policy and Programs Branch to ensure development of the proper questions and methodology to guide the evaluation. A Working Group comprised of departmental officials and external stakeholders will also support this evaluation.

A significant portion of the work for this evaluation will be completed by a consultant. Oversight of daily activities will be responsibility of the EPMRB evaluation team, headed by a Senior Evaluation Manager. The EPMRB evaluation team will be responsible for identifying key documents, providing a number of documentation, data for the study, as well as names and contact information of INAC resource persons at headquarters and regional offices. The team will furthermore expeditiously review, comment on and approve the products delivered by the consultant.

The Department of Indian and Northern Affairs has determined that any Intellectual Property arising from the performance of the work under the contract will vest in Canada for the following reason: the main purpose of the contract is the generation of knowledge and information for public dissemination.

5. Evaluation Resources and Timeline

The estimated cost of the evaluation is $100,000. Funds to cover the cost of the evaluation will be transferred to the Evaluation, Performance Measurement and Review Branch by the Social Policy and Programs Branch, Social Programs Reform Directorate.

Subject to verification in the detailed methodology report and work plan, the evaluation is expected to be completed by April 2010.






Appendix B - Evaluation Matrix

Table 1: Evaluation Issues, Questions and Data Collection Methodologies
Theme Areas Literature/Documents/ Data/Files Key Informant Interviews Focus Groups
Relevance
To what extent does the EPFA address a demonstrable need?
To what extent is the EPFA responsive to the needs of First Nation children and families living on reserve?
Design & Delivery
To what extent does the design of the Approach allow for progress towards the achievement of expected outcomes?  
To what extent is the EPFA being implemented/delivered as originally planned?  
Performance
To what extent has progress toward intended outcomes been achieved as a result of implementing the EPFA?
Have there been any unintended impacts (positive, negative) associated with the implementation?
What performance measurement information, linked to intended outcomes, is being collected?
What are the best practices and lessons learned?
Cost-effectiveness
Are there more cost effective ways of implementing the Approach?  
How comparable is EPFA funding to provincial funding for non-First Nation communities in similar circumstances?  
 





Appendix C - Data Collection Tools

Key Informant Interview Questions - INAC Officials


Interview #:

Date, Time of Interview: _____________________________

Name of Interviewer: ___________________________

Role of Interviewee: ______________________________


This interview is part of an implementation evaluation being carried on behalf of Indian and Northern Affairs Canada on the Enhanced Prevention Focused Approach in Alberta for the First Nations Child and Family Services Program. The evaluation is being conducted to examine program design and delivery, performance, cost-effectiveness, relevance as well as, best practices and lessons that may impact future programming.

DPRA Canada and T.K. Gussman Associates Inc., the firms contracted to do the evaluation, are interviewing INAC officials, regional FNCFS agency representatives, individuals from the provincial and territorial governments, representatives from Treaty Areas 6, 7 and 8, Aboriginal organizations and individuals with expertise in the subject area (e.g., child welfare). We very much appreciate your participation in the evaluation.

The information you provide is for research purposes only. It will not be used for any purposes other than the evaluation and will be administered, retained and disposed of in accordance with the Privacy Act. Your specific responses will not be attributed to you in any evaluation report resulting from this study.

We are grateful for the time and effort you are willing to put into this research.


Overview:

  1. What is your involvement in the First Nation Child and Family Services (FNCFS) Program? How long have you been involved?
  2. How does the Family Enhancement Approach compare to previous approaches implemented by the FNCFS Program in Alberta? (prompts: culturally appropriate, comparability with services offered reserve in similar circumstances, timeliness of services, linkages between agencies, proactive approach)

    Theme 1: Design and Delivery
  3. Can the design of the Approach be reasonably expected to achieve the intended outcomes of the program? How so/why not?
  4. Has the Approach been implemented/delivered in the manner originally planned? If not, in what ways does it differ and why did the departure from the original plan occur?
  5. What key factors have helped/hindered the implementation of the Family Enhancement Approach? (prompts: human resources, financial resources, buy-in by staff, partnerships, community awareness, existence of clear roles and responsibilities, dispute resolution process)
  6. What are the service delivery gaps?
  7. Have there been any unintended impacts - positive or negative - associated with the implementation of this approach?
  8. Are there any linkages/partnerships that would enhance the implementation/coordination of the Approach?
  9. What gender-specific initiatives are being offered to children and families receiving services? If none, is there a need to consider offering gender-specific services?
  10. What are the administrative requirements (e.g., applications and reporting) required for this approach (e.g. provincial, federal, band, etc)? What impact, if any, have these had?

    Theme 2: Relevance
  11. Is there a continued need for child and family services in Alberta to be implemented using this Approach? Please explain your response.
  12. Is the Approach addressing the current needs of First Nation children and families? If yes, what specific needs (provincial/community) has it met? If not, what specific needs still need to be met?

    Theme 3: Performance
  13. How do you measure the success of the implementation of this approach in Alberta (i.e. what indicators do you use to measure success)? Is there anything that should be collected/measured that isn't currently being collected/measured?
  14. What key factors are helping in the achievement of program outcomes?
  15. What key factors are challenging the achieving program outcomes?
  16. Are there any lessons learned or best practices that could be passed on to other provinces implementing this Approach?

    Theme 4: Cost Effectiveness
  17. Is the current method for allocating resources for the Approach - specifically the funding formulas - appropriate, equitable and effective?
  18. To what extent does the enhanced funding allow the FNCFS agencies to deliver reasonably comparable services to children and their families living in similar circumstances?
  19. Can you identify more cost effective ways of implementing the Approach?

    Other:
  20. Can you direct us to any additional reports or documents that we should be aware of to help us better understand the FNCFS Program and more specifically, the Enhanced Approach?
  21. Do you have any other comments you would like to make about the Family Enhancement Approach?


Key Informant Interview Questions - Provincial Government Representatives


Interview #:

Date, Time of Interview: _____________________________

Name of Interviewer: ___________________________

Role of Interviewee: ______________________________


This interview is part of an implementation evaluation being carried on behalf of Indian and Northern Affairs Canada on the Enhanced Prevention Focused Approach in Alberta for the First Nations Child and Family Services Program. The evaluation is being conducted to examine program design and delivery, performance, cost-effectiveness, relevance as well as, best practices and lessons that may impact future programming.

DPRA Canada and T.K. Gussman Associates Inc., the firms contracted to do the evaluation, are interviewing INAC officials, regional FNCFS agency representatives, individuals from the provincial and territorial governments, representatives from Treaty Areas 6, 7 and 8, Aboriginal organizations and individuals with expertise in the subject area (e.g., child welfare). We very much appreciate your participation in the evaluation.

The information you provide is for research purposes only. It will not be used for any purposes other than the evaluation and will be administered, retained and disposed of in accordance with the Privacy Act. Your specific responses will not be attributed to you in any evaluation report resulting from this study.

We are grateful for the time and effort you are willing to put into this research.


Overview:

  1. What is your involvement in the First Nation Child and Family Services (FNCFS) Program? How long have you been involved?
  2. How does the Family Enhancement Approach compare to previous approaches implemented by the FNCFS Program in Alberta? (prompts: culturally appropriate, comparability with services offered reserve in similar circumstances, timeliness of services, linkages between agencies, proactive approach)

    Theme 1: Design and Delivery
  3. Can the design of the Approach be reasonably expected to achieve the intended outcomes of the program? How so/why not?
  4. Has the Approach been implemented/delivered in the manner originally planned? If not, in what ways does it differ and why did the departure from the original plan occur?
  5. What key factors have helped/hindered the implementation of the Family Enhancement Approach? (prompts: human resources, financial resources, buy-in by staff, partnerships, community awareness, existence of clear roles and responsibilities, dispute resolution process)
  6. What are the service delivery gaps?
  7. Have there been any unintended impacts - positive or negative - associated with the implementation of this approach?
  8. Are there any linkages/partnerships that would enhance the implementation/coordination of the Approach?
  9. What gender-specific initiatives are being offered to children and families receiving services? If none, is there a need to consider offering gender-specific services?
  10. What are the administrative requirements (e.g., applications and reporting) required for this approach (e.g. provincial, federal, band, etc)? What impact, if any, have these had?

    Theme 2: Relevance
  11. Is there a continued need for child and family services in Alberta to be implemented using this Approach? Please explain your response.
  12. Is the Approach addressing the current needs of First Nation children and families? If yes, what specific needs (provincial/community) has it met? If not, what specific needs still need to be met?

    Theme 3: Performance
  13. How do you measure the success of the implementation of this approach in Alberta (i.e. what indicators do you use to measure success)? Is there anything that should be collected/measured that isn't currently being collected/measured?
  14. What key factors are helping in the achievement of program outcomes?
  15. What key factors are challenging the achieving program outcomes?
  16. Are there any lessons learned or best practices that could be passed on to other provinces implementing this Approach?

    Theme 4: Cost Effectiveness
  17. Is the current method for allocating resources for the Approach - specifically the funding formulas - appropriate, equitable and effective?
  18. To what extent does the enhanced funding allow the FNCFS agencies to deliver reasonably comparable services to children and their families living in similar circumstances?
  19. Can you identify more cost effective ways of implementing the Approach?

    Other:
  20. Can you direct us to any additional reports or documents that we should be aware of to help us better understand the FNCFS Program and more specifically, the Enhanced Approach?
  21. Do you have any other comments you would like to make about the Family Enhancement Approach?


First Nations Interview Questions - FNCFS Agencies, Treaty Areas 6, 7 and 8, Aboriginal Organizations


Interview #:

Treaty Area (if applicable): __________________________

Date, Time of Interview: _____________________________

Name of Interviewer: ___________________________

Role of Interviewee: ______________________________


This interview is part of an implementation evaluation being carried out on behalf of Indian and Northern Affairs Canada on the Enhanced Prevention Focused Approach in Alberta for the First Nations Child and Family Services Program. The evaluation is being conducted to examine program design and delivery, performance, cost-effectiveness, relevance as well as, best practices and lessons that may impact future programming.

DPRA Canada and T.K. Gussman Associates Inc., the firms contracted to do the evaluation, are interviewing regional FNCFS agency representatives, INAC officials, individuals from the provincial and territorial governments, representatives from Treaty Areas 6, 7 and 8, and individuals from various Aboriginal organizations. We very much appreciate your participation in the evaluation.

The information you provide is for research purposes only. It will not be used for any purposes other than the evaluation and will be administered, retained and disposed of in accordance with the Privacy Act. Your specific responses will not be attributed to you in any evaluation report resulting from this study.

We are grateful for the time and effort you are willing to put into this research.


Overview:

  1. What is your involvement in the First Nation Child and Family Services (FNCFS) Program? How long have you been involved?
  2. How does the Family Enhancement Approach compare to previous approaches implemented by the FNCFS Program in Alberta? (prompts: culturally appropriate, comparability with services offered reserve in similar circumstances, timeliness of services, linkages between agencies, proactive approach)

    Theme 1: Design and Delivery
  3. Are you familiar with your organization's business plan for the Family Enhancement Approach? If so, can you comment on any of the following: goals, priorities, challenges, opportunities, and performance measurements identified in the plan.
  4. Can the design of the Approach be reasonably expected to achieve the intended outcomes of the program? How so/why not? (prompts: program flexibility to align resources to priority areas, provincial comparability, availability of a wider variety of services)
  5. Has the Approach been implemented/delivered in the manner originally planned? If not, in what ways does it differ and why did the departure from the original plan occur?
  6. What key factors have helped/hindered the implementation of the Family Enhancement Approach? (prompts: human resources, buy-in by staff, partnerships, community awareness, existence of clear roles and responsibilities, dispute resolution process).
  7. What are the service delivery gaps?
  8. Have there been any unintended impacts - positive or negative - associated with the implementation of this approach?
  9. Have you encountered any issues of resource constraint and if so, how have you dealt with them?
  10. Has the relationship with other community-based service delivery agencies changed as a result of the Family Enhancement Approach? Are there other linkages/partnerships that would assist with the implementation/coordination of the Approach? (prompt: community awareness of new scope)
  11. What gender-specific initiatives are being offered to children and families receiving services? If none, is there a need to consider offering gender-specific services?
  12. What are the administrative requirements (e.g., applications and reporting) required for this approach (e.g., provincial, federal, band, etc)? What impact, if any, has this had?

    Theme 2: Relevance
  13. Is there a continued need for child and family services in Alberta to be implemented using this approach? Please explain your response.
  14. Is the approach addressing the current needs of First Nation children and families that you serve? If yes, what specific needs (provincial/community) has it met? If not, what specific needs still need to be met?
  15. Are the articulated goals in the business plan conducive to achieving the intended outcomes?

    Theme 3: Performance
  16. How do you measure the success of the implementation of this Approach in Alberta (i.e. what indicators do you use to measure success)? Is there anything that should be collected/measured that isn't currently being collected/measured?
  17. What key factors are helping you move towards the achievement of program outcomes?
  18. What key factors are challenging you from achieving program outcomes?
  19. Are there any lessons learned or best practices that could be passed on to other provinces implementing this approach?

    Theme 4: Cost Effectiveness
  20. How is the current method for allocating resources for the Approach working?
  21. To what extent does the enhanced funding allow the FNCFS agencies to deliver reasonably comparable services to children and their families living in similar circumstances?
  22. How have you maximized the funding you were allocated through this approach? Can you provide examples of how you are being efficient and economical in implementing the Approach?

    Other:
  23. Can you direct us to any additional reports or documents that we should be aware of to help us better understand the FNCFS Program and more specifically, the Enhanced Approach?
  24. Do you have any other comments you would like to make about the Family Enhancement Approach?


Focus Group Questions

  1. From your perspective as front-line staff, can you please describe the Enhanced Prevention Focused Approach (EPFA) and explain how it plays out at the community level?
  2. How does this Approach differ from other approaches that you have taken to deliver child and family services?
  3. Can you discuss the following topics in relation to this Approach:

    1. Is this Approach allowing you to reach all of the children and families you need to in a timely fashion?
    2. Is this Approach culturally appropriate? If so, how? If not, what improvements could be made?
    3. What partnerships/linkages do you have with other organizations?
    4. How do you measure success?
  4. From a community perspective, can you please discuss the successes associated with implementing this Approach?

    1. Can you describe any specific success stories?
  5. From a community perspective, can you please discuss the challenges associated with implementing this Approach?

    1. Can you think of any ways to overcome these challenges?
  6. This Approach is currently being introduced in other provinces. What best practices and/or lessons learned that you would like to pass on to First Nation Child and Family Service Agency front-line staff that you feel would make the transition easier?







Footnotes:

  1. INAC, 2007. A Departmental Audit (2007) also noted that program expenditures more than doubled between 1996 and 2006. (return to source paragraph)
  2. The regions were Saskatchewan, Nova Scotia, PEI and Quebec. (return to source paragraph)
  3. INAC, 2009, Terms of Reference for the Implementation Evaluation of the Enhanced Prevention Focused Approach in Alberta for the First Nations Child and Family Services Program. (return to source paragraph)
  4. Of main concern for the Family Enhancement section is that the number of average monthly caseloads did not appropriately correspond with the total number of children who received Family Enhancement services in any of the years considered (2006-07 to 2009-10). As for the number of children in care, the total was derived using the number of children who received Family Enhancement services as well as child protection services. Where a child was subject to both types of services, they were counted twice, leading to an over-inflation of the number of children in care. Furthermore, the basis of comparison for the First Nation and provincial populations would have had to come from the Indian Registry and the 2006 Census, respectively, which would also have yielded methodological anomalies, particularly when trying to compare trends over time. (return to source paragraph)
  5. INAC, 2007. RMAF for the First Nations Child and Family Services Program. (return to source paragraph)
  6. Government of Alberta. Alberta response Model: Building on Successful Practice and Transforming Outcomes for Children. Information Sheet. (return to source paragraph)
  7. INAC. Backgrounder - INAC, Alberta and First Nations Join Forces to Improve Child Welfare Services for First Nations Children. Date Modified: 2008-10-16. (return to source paragraph)
  8. INAC. Canada's New Government, Treaty 6, Treaty 7 and treaty 8 First Nations and Alberta Embark on a New Approach to Child Welfare on Reserve, April 27, 2007. (return to source paragraph)
  9. INAC. Backgrounder - INAC, Alberta and First Nations Join Forces to Improve Child Welfare Services for First Nations Children. Date Modified: 2008-10-16. (return to source paragraph)
  10. INAC. Canada's New Government, Treaty 6, Treaty 7 and treaty 8 First Nations and Alberta Embark on a New Approach to Child Welfare on Reserve, April 27, 2007. (return to source paragraph)
  11. INAC, 2007. RMAF for The First Nations Child and Family Services Program - Appendix B. (return to source paragraph)
  12. INAC, 2007. A Departmental Audit (2007) also noted that program expenditures more than doubled between 1996 and 2006. (return to source paragraph)
  13. The regions were Saskatchewan, Nova Scotia, PEI and Quebec. (return to source paragraph)
  14. INAC, 2009, Terms of Reference for the Implementation Evaluation of the Enhanced Prevention Focused Approach in Alberta for the First Nations Child and Family Services Program. (return to source paragraph)
  15. INAC, 2007. RMAF for the First Nations Child and Family Services Program - Appendix B. (return to source paragraph)
  16. Indian and Northern Affairs Canada, First Nation Child and Family Services Program. (return to source paragraph)
  17. INAC, 2007. RMAF for the First Nations Child and Family Services Program - Appendix B. (return to source paragraph)
  18. Ibid. (return to source paragraph)
  19. INAC, 2009, Response to the Seventh Report of the Standing Committee on Public Accounts, Chapter 4, FNCFS Program - INAC, of the May 2008 Report of the Auditor General. (return to source paragraph)
  20. Statistics Canada. 2006. Aboriginal Children's Survey, 2006. Family, Community and Child Care. Social and Aboriginal Statistics Division. (return to source paragraph)
  21. Farris-Manning, Cheryl and Marietta Zandstra. 2003. Children in Care in Canada: A Summary of Current Issues and Trends with Recommendations for Future Research. Child Welfare League of Canada. (return to source paragraph)
  22. Alberta Children and Youth Services. (2010). Children and Youth Services Business Plan 2010-13 (January 25, 2010). (return to source paragraph)
  23. First Nations Canadian Incidence Study 2003. (return to source paragraph)
  24. Trocmé, Nico, Bruce MacLaurin, Barbara Fallon, Della Knoke, Lisa Pitman & Megan McCormack. 2006. Understanding the Overrepresentation of First Nations Children in Canada's Child Welfare System: An Analysis of the Canadian Incidence Study of Reported Child Abuse and Neglect (CIS-2003). Centre of Excellence for Child Welfare. First Nations Child and Family Caring Society of Canada. (return to source paragraph)
  25. MacLaurin, B. (2009). Using National and Provincial Data to Inform Outcomes for Child Welfare. Provincial Forum on Improving Outcomes for Children, Youth & Families; Edmonton, Alberta; May 27th, 2009. Faculty of Social Work, University of Calgary. (return to source paragraph)
  26. Native Women's Association of Canada. (2007). Strategies to Address Child Welfare: An Issue Paper. (return to source paragraph)
  27. Alberta Children and Youth Services. (2008). Children and Youth Services Business Plan 2008-11 (April 4, 2008). Pg 48.(return to source paragraph)
  28. House of Commons. 2009. Government of Canada response to the Report of the Standing Committee on Public Accounts on Chapter 4, First Nations Child and Family Program, Indian and Northern Affairs Canada of the May 2008 Report of the Auditor General. (return to source paragraph)
  29. Artaraz, K., Thurston, M., & Davies, S. (2007). Understanding family support provision within the context of prevention: a critical analysis of a local voluntary sector project. Child & Family Social Work, Vol. 12(4), 306-315. (return to source paragraph)
  30. Libesman, T. 2004. Child welfare approaches for Indigenous communities: International perspectives. National Child Protection Clearinghouse Issues, 20. (return to source paragraph)
  31. Lubin, J. 2009. Are we Really Looking out for the Best Interests of the Child? Applying the New Zealand Model of Family Group Conferences to Cases of Child Neglect in the United States. Family Court Review, Vol. 47(1), 129-147. (return to source paragraph)
  32. Carriere, J., & Richardson, C. (2009) From Longing to Belonging: Attachment Theory, Connectedness, and Indigenous Children in Canada. In S. McKay, D. Fuchs & I Brown (Eds.) Passion for Action in Child and Family Services: Voices from the Prairies (pp. 49-67). Regina , SK: Canadian Plains Research Center. (return to source paragraph)
  33. Two First Nations agency respondents did not answer the question. (return to source paragraph)
  34. Figure 5.2 illustrates average care days for each FNCFS agency, with five agencies showing decreases and 13 showing increases, year-over-year. Available data is unable to demonstrate what the drivers are behind these increases in care days, nor to allow conclusions as to the relationship between these numbers and the efficacy of the EPFA model. (return to source paragraph)
  35. One agency reported that they are using these standards, but it is unclear whether they are referring to those developed in B.C. (see Literature Review on Culturally Appropriate Standards for further details). (return to source paragraph)
  36. INAC. (2008). Child and Family Services Progress Report for an Enhanced Prevention Focused Approach. (return to source paragraph)
  37. FNCFS Audit, March 2009. (return to source paragraph)
  38. For example, such a trend could be explained by effective mental health and addictions programming in the community; or other community healing and capacity-building activity. Ultimately, in an integrated and collaborative model, all such programs would be linked and oriented toward the same overall goal of family and community health and healing. (return to source paragraph)
  39. Up to a maximum of five percent of service delivery costs. Costs over and above the five percent are paid by the province. (return to source paragraph)
  40. The document was citing the Wen:De: report (see citation below). (return to source paragraph)
  41. Also referred to as "differential response," the term refers to preventive alternatives to child protection, when appropriate. (return to source paragraph)
  42. Children's Advocate Manitoba. September 29, 2006. Strengthen the commitment. An External Review of the Child Welfare system. Pg.27. (return to source paragraph)
  43. McCain, M., J.F. Mustard & S. Shanker, 2007, Early Years Study 2: Putting Science into Action. Council for Early Child Development, Toronto, pg 137. (return to source paragraph)
  44. First Nations Child and Family Caring Society of Canada. (2005). Wen De: We are coming to the light of day. (return to source paragraph)
  45. Indian and Northern Affairs Canada. (2007). Audit of the First Nations Child and Family Services Program Departmental Audit and Evaluation Branch. (return to source paragraph)
  46. The formula is variable as to which year's actual figures are used for re-basing maintenance budgets. (return to source paragraph)
  47. INAC document. Capacity Dollars for 2009-10, by agency. (return to source paragraph)
  48. INAC, 2007. RMAF for the First Nations Child and Family Services Program. (return to source paragraph)
  49. Of main concern for the Family Enhancement section is that the number of average monthly caseloads did not appropriately correspond with the total number of children who received Family Enhancement services in any of the years considered (2006-07 to 2009-10). As for the number of children in care, the total was derived using the number of children who received Family Enhancement services as well as child protection services. Where a child was subject to both types of services, they were counted twice, leading to an over-inflation of the number of children in care. Furthermore, the basis of comparison for the First Nation and provincial populations would have had to come from the Indian Registry and the 2006 Census, respectively, which would also have yielded methodological anomalies, particularly when trying to compare trends over time. (return to source paragraph)
 
 
Date modified: