ARCHIVED - Evaluation of the Advocacy and Public Information Program (APIP)

Archived information

This Web page has been archived on the Web. Archived information is provided for reference, research or record keeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Date: February 13, 2009

PDF Version   (134 Kb, 42 pages)

 

 

Table of Contents


Initialisms and Abbreviations

AES Audit and Evaluation Sector (INAC)
AFN Assembly of First Nations
AHF Aboriginal Healing Foundation
APIP Advocacy and Public Information Program
CEP Common Experience Payment
EPMRB Evaluation, Performance Measurement and Review Branch
IAP Independent Assessment Process
INAC Indian and Northern Affairs Canada
IRSRC Indian Residential Schools Resolution Canada
PMF Performance Measurement Framework
RIAS Resolution and Individual Affairs Sector (INAC)
RBAF Risk-Based Audit Framework
RMAF Results-based Management and Accountability Framework
TORs Terms of Reference
TRC Truth and Reconciliation Commission
 
Return to Table of Contents





Executive Summary

This report presents the findings of the evaluation of the Advocacy and Public Information program, undertaken by the Evaluation, Performance Measurement and Review Branch (EPMRB) of Indian and Northern Affairs Canada (INAC). The evaluation sought to assess the Program's relevance, program design and delivery, effectiveness and success and impact on end users.

Context

The Indian Residential School Settlement Agreement came into effect on September 19, 2007. The Advocacy and Public Information program was established to inform former students and the general Canadian population about the Settlement Agreement and the impact of the Indian Residential Schools legacy on Aboriginal communities. These activities were to be undertaken as part of a larger communications strategy to disseminate information and educate these target audiences.

There are six key components of the Settlement Agreement that have related programs that the Advocacy and Public Information program disseminates information to former students about benefits available. These include:

Each of these Programs, along with the Courts and Indian and Northern Affairs Canada's (formerly Indian Residential Schools Resolution Canada) Communications Branch plays a complementary role in disseminating information. The Program Interlocutors are responsible for communications on their area of responsibility, while the others share information on the overall Settlement Agreement and all related programs. The Advocacy and Public Information program was the mechanism by which grassroots organizations were engaged to support outreach activities.

Program Objectives and Expected Results

The primary objectives of APIP are to encourage information sharing on the Settlement Agreement and the benefits for former students, their families and their communities; to support opportunities to enhance service delivery; and to help sensitize Canadians to the impact of the legacy of Indian Residential Schools on Aboriginal Communities.

The expected result was to help ensure former students of Indian Residential Schools have complete access and equal participation in the Settlement Agreement. As per the overall ambition of the Settlement Agreement, to bring about a lasting resolution to the legacy of Indian Residential Schools, it was anticipated that such a result would in turn contribute towards reconciliation between Residential School survivors and the Government of Canada.

Program Activities and Contribution Recipients

APIP was allocated $10 million, under a 2-year allotment ending March 31, 2009. The program is seeking an extension for a further four years. The 2007-08 fiscal year funding allocation, of $6M, was fully invested to support 27 initiatives, while the 2008-09 allocation of $4M has been committed to 15 projects, with an additional 7 projects still in negotiation. Between April 2007 and November 2008, 42 contribution agreements were signed with 25 organizations, predominantly Aboriginal. Contributions ranged from $25,000 to $603,240 with the average being $200,866.

Contribution agreements reflect a balance of demographic and geographic representation of intended audiences. These included First Nations, Inuit, and Métis; on- and off-reserve, rural and urban communities; and the general Canadian population.

Projects in each fiscal year, focused on disseminating information pertaining to programs related to the Settlement Agreement that were being rolled out within that respective year. In fiscal year 2007-08, projects focused on disseminating information on the Common Experience Payment, while in 2008-09 projects were to focus on the Truth and Reconciliation Commission, the Independent Assessment Process and Commemoration.

Evaluation Methodology and Limitations

The data collection strategy was comprised of multiple lines of evidence that were triangulated where possible. Research included document and literature review; a review of administrative data and information; review of secondary data sources; key informant interviews with contribution recipients, departmental officials and former students; an electronic survey of contribution recipients; and four case studies.

Initially, a survey of former students was planned. Upon further consideration of the sensitivity of the subject matter and the availability of secondary data sources, it was determined that the use of existing data would support evaluation interests.

The lack of a documented performance measurement strategy including baseline data, a program specific logic model, targets, performance indicators and results focused reporting limited the capacity of the evaluation to draw substantive conclusions on program success and cost-effectiveness.

Additionally, scheduling challenges limited the scope of the evaluation's fieldwork. Given the diversity of parties engaged in disseminating information on the Settlement Agreement, the evaluation could not draw conclusive findings regarding attribution of results achieved to the Advocacy and Public Information program. Lack of performance information also contributed to this challenge.

Conclusions

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority. With current delays in the implementation of the Settlement Agreement, there continues to be a need for communication activities directed to survivors, their families and communities and the broader Canadian population.

APIP's strategy of engaging credible and trusted Aboriginal organizations was found to be appropriate given the sensitivity of the information to be shared. It is also an appropriate response to the urgency of disseminating information about the Settlement Agreement. One challenge, however, is that no contingency plan is in place to inform individuals and communities when capable and willing organizations at the community level cannot be identified in a timely manner.

The lack of clarity in the program's expected results and performance measurement gaps severely inhibit the Program's capacity to monitor performance, measure the achievement of expected results, or identify issues or factors which may differentially affect access to information to and participation in benefits from the Settlement Agreement.

In addition to performance measurement gaps, the short time frame in which APIP has been operating also limited the potential for assessing impacts at this time. A number of promising practices were identified in terms of maximizing the reach of communication activities. Moreover, according to an analysis of departmental data, APIP project funding seems to be distributed in proportion to regional needs, but no strong correlation was found between funding and uptake. The evidence suggests that future initiatives should take additional steps to ensure clarity in information disseminated and ensure mechanisms are in place to identify communication related issues affecting applications.

Efforts are being taken to reduce overlaps and gaps and to enhance coordination in order to enhance the results of Residential School related communications. With more Settlement Components coming on-line, and some potential opportunities for overlap being noted, now is the time to ensure the roles and responsibilities are clear and complementary, and structured or streamlined in such a manner so as to best facilitate uptake.

Recommendations

The evaluation findings support the following recommendations:

1. In light of the revised implementation schedule for various Settlement Components, INAC should consider continuing support for communication activities, based in part on the APIP model, to assist in informing former students and Canadians about the Settlement Agreement;

2. Clarify roles and responsibilities of the various parties involved in disseminating information about the Settlement Agreement so as to ensure cost-effectiveness in the delivery of information and reduce the potential for duplication and overlaps; and

3. Depending on the approach adopted, develop a results based management tools and performance measurement strategy, that is mindful of reporting burden, yet includes:

Return to Table of Contents





1.0 Introduction

1.1 Introduction

This report presents the findings for the evaluation of the Advocacy and Public Information program (APIP), undertaken by the Evaluation, Performance Measurement and Review Branch (EPMRB) of Indian and Northern Affairs Canada (INAC). The evaluation focuses on the Program's relevance, success and effectiveness and is intended to inform future programming.

The Advocacy and Public Information program, established in 2007, was designed to support the implementation and results of the Indian Residential Schools Settlement Agreement by informing former students and Canadians about the Agreement and the impacts of the Indian Residential Schools legacy.

1.2 Context: The Settlement Agreement [Note 2]

Indian residential schools began operating in Canada around 1800 and were largely operated by Anglican, Roman Catholic, Methodist and Presbyterian churches. As early as 1874, the federal government was involved in their development and administration in line with requirements of the Indian Act. Over time, 130 schools were distributed across the country, with the exception of Newfoundland, New Brunswick and Prince Edward Island. On April 1, 1969, the Government assumed full responsibility for the schools. While most had ceased to operate by mid-1970, the last school closed in Saskatchewan in 1996.

Throughout 1996, the Royal Commission on Aboriginal Peoples highlighted the negative impacts that the residential schools had on individuals and communities. Abuse, separation from families, and the impacts on Aboriginal languages and culture are believed to have contributed to the family violence and substance abuse facing many First Nations, Métis and Inuit communities today.

The Indian Residential Schools Settlement Agreement, which came into effect on September 19, 2007, represents the culmination of the efforts of its signatories to work towards a fair, comprehensive and lasting resolution of the legacy of schools and the promotion of healing, education, truth and reconciliation and commemoration [Note 3].

Representing the consensus reached in the discussions between the Government of Canada, legal counsel for former students, the Churches, the Assembly of First Nations and other Aboriginal organizations, the out-of court settlement is recognized to be the largest class action settlement in Canadian history (A background study commissioned by Indian Residential Schools Resolution Canada estimated that there were some 80,000 former students still living in 2005) [Note 4].

According to the integrated RMAF-RBAF, approved in 2008 by Treasury Board, and soon to be reviewed as part of an evaluation framework exercise [Note 5], the Settlement agreement has two objectives:

Key Components of the Settlement Agreement

The Settlement Agreement includes both financial benefits for former students and non-compensatory initiatives directed to the benefit of former students and others, including family members and the general Canadian public, as described below:

Common Experience Payment

Commemoration

Independent Assessment Process

Truth and Reconciliation Commission

Healing

An important part of the Agreement's purpose involves working closely with former students, their families, and communities in support of projects that promote healing. This is done primarily through programs offered by the Aboriginal Healing Foundation and Health Canada:

Healing Fund

Mental Health and Emotional Support Services

Administrative Context

The Office of Indian Residential Schools Resolution of Canada was established as a department (by Order-in-Council) in 2001, to centralize and focus federal efforts to resolve claims associated with the operation of the former Indian Residential school system. Prior to that, a unit within Indian and Northern Affairs Canada was responsible for managing litigation, which was related to the operation of Indian Residential Schools.

On June 1, 2008, responsibility for the oversight of the timely and effective implementation of the Settlement Agreement was transferred to INAC's Resolution and Individual Affairs Sector. To this end, RIAS liaises with all other Program Interlocutors and the Courts to share information about efforts being made to implement the Settlement Agreement.

As discussed in the next section RIAS is responsible for the Advocacy and Public Information program, as well as for Commemoration, and is working with Service Canada for the delivery of Common Experience Payment process. The remaining components of the Settlement Agreement are administered by other government and non-governmental organizations:

Return to Table of Contents





2.0 Program Profile

2.1 Overview

There are a number of parties engaged in disseminating information about the Indian Residential Schools Agreement. Each of the Settlement Agreement component programs have responsibilities for informing former students and/or the general Canadian population about their own respective activities. Additionally, the Courts have a role to play in disseminating information on the Settlement Agreement in general and INAC's Communication Branch, as part of its responsibilities for informing Canadians about INAC's programs and services, also plays a role in disseminating information.

As described below, the Advocacy and Public Information program, now managed by INAC's Resolution and Individual Affairs Sector, was designed to complement and broaden the communication activities of other stakeholders and parties.

2.2 Program Objectives and Expected Results

According to APIP's Terms and Conditions (2006), the program's objectives are to:

APIP's expected outcomes, also expressed in its Terms and Conditions (2006), are to:

2.3 Program Activities and Recipients

APIP is a proposal driven program which supports the following types of activities

Eligible recipients include, among others, former students, Aboriginal organizations, public or private policy organizations, institutions or individuals with expertise on issues of interest, Tribal Councils, Aboriginal owned or controlled entities, organizations mandated to act on behalf of former students, and religious organizations and institutions.

Between April 2007 and November 2008, 42 contribution agreements were signed with 25 organizations, with an additional seven projects still in negotiation. Contributions ranged from $25,000 to $603,240 with the average being $200,866 (median $169,134, mode $25,000).

Contribution recipients to date have been predominantly Aboriginal organizations active at the regional and national levels. Projects (approved and in negotiation) include 19 national initiatives (9 focused on disseminating information on the settlement agreement and 12 on public education); and 30 regional initiatives (25 focused on disseminating information about the settlement agreement and 7 on public education) [Note 6].

The projects focus on a cross-section of different audiences, including First Nations, Inuit, and Métis, on- and off-reserve, rural and urban communities, and the general Canadian population. In fiscal year 2007-08, projects focused on disseminating information on the Common Experience Payment, while in 2008-09 projects are focusing on the Truth and Reconciliation Commission, the Independent Assessment Process and Commemoration.

2.4 Program Budget

APIP was allocated $10 million, under a 2-year allotment ending March 31, 2009. The program is seeking an extension for a further four years. The 2007-08 fiscal year funding allocation, of $6M, supported 27 initiatives, while the 2008-09 allocation of $4M has been committed to 15 projects, with an additional 7 projects still in negotiation.

Eligible expenditures include, in general terms, project manager wages; travel and accommodation for managers, staff and participants, elder honoraria, expert consultations and advice, instructional materials, and administration costs.[Note 7]

Return to Table of Contents





3.0 Evaluation Methodology

3.1 Evaluation Issues and Questions [Note 8]

This evaluation examined APIP's:

Relevance

Design and delivery

Success and impact on end users

Effectiveness, modifications and alternatives

3.2 Data Collection Strategy

The data collection strategy comprised multiple lines of evidence, including:

The evaluation Terms of Reference anticipated that a survey of survivors would also be conducted in order to examine the Program's performance. The survey was not conducted, in part because short time frame in which APIP has been operating, recognition of the sensitivities identified around Settlement agreement activities, as well as the identification of other sources of information on recipients and program uptake (e.g., including the 2008 baseline study conducted by Environics mentioned above, and program uptake data).

3.3 Research Limits

Return to Table of Contents





4.0 Findings: Relevance

The questions addressed for this evaluation issue were:

4.1 Key Findings

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority.

The current government took significant steps to re-assert the priority placed by the government on recognizing the importance of the legacy of the Residential Schools and the implementation of the Settlement Agreement when, in June 2008, the Prime Minister issued a Statement of Apology to former students of Indian Residential Schools, on behalf of the Government of Canada [Note 9].

There is a continuous need for communications activities which will enhance uptake of the Settlement Agreement programs and increase awareness about the Residential experience.

Settlement Agreement Information
APIP was intended to support the initial launch of Settlement Agreement's financial and non-financial benefit programs. However, the Program has not been able to fulfill its mandate because, while the Settlement Agreement came into effect September 19, 2007, some activities have been delayed and some not yet launched, as shown below:

Table 1 Implementation Dates of Settlement Agreement Programs
Settlement Agreement benefits / related programs Implementation / Status
Common Experience Payment September 2007
Reconsideration Process March 2008
Aboriginal Healing Foundation September 2007
Indian Residential Schools Resolution Health Support Program September 2007
Independent Assessment Process March 2008
Truth and Reconciliation Commission (Activities not yet initiated)
Commemoration (Not yet launched)
 

According to Program officials, priority has been placed on disseminating information about activities which were expected to be delivered or initiated during each of the two fiscal years APIP has been operating. In 2007-08, for example, projects focused on the Common Experience Payment, while in 2008-09, priority was placed on the Truth and Reconciliation Commission and Commemoration (neither of which in fact have yet initiated activities).

Regardless of delays relating to implementations, many key informants interviewed stated that the need for information on the Settlement Agreement and its legacy has not been completely met. This appreciation is supported by evidence from an analysis of departmental data and secondary sources, including surveys conducted before and after the Government of Canada's statement of apology.

The evaluation's analysis of departmental data, circa January 2009, found, for example, that while some regions are reporting Common Experience Payment applications far in excess of estimated students, others are reporting far less than would be expected (Appendix A). While it was beyond the scope of this evaluation to clearly identify the reasons behind these patterns, it does suggest that not all potential applicants are receiving information, or appropriate information, or have chosen not to participate in the process.

Supporting the possibility that access to information is likely one factor is the finding that just prior to the Prime Minister's statement of apology [Note 10], about 20% of the on-reserve population and more than one-third of the off-reserve First Nation population did not know about the common experience payment and about two-thirds of the Aboriginal population overall did not know of upcoming commemoration activities as noted below. If eligible recipients - former students, families, communities - are not aware of the supports available to them, they will not access them or otherwise participate fully in the resolution process.

Table 2 Level of Awareness of the Settlement Agreement Programs (2008)
Details General Population
n = 1,503
Aboriginal
On Reserve
n = 155
Off Reserve
n = 150
A "common experience payment" to be paid to all eligible former students 38% 81% 71%
Independent Assessment Process for claims of sexual/serious physical abuse 37% 61% 63%
A Truth and Reconciliation Commission 21% 24% 26%
Measures to support healing by supporting the AFH 17% 45% 41%
Support for event/memorials to commemorate the legacy of IRS 13% 29% 28%

Source: Environics Research Group, (2008). The general population sample was stratified regionally to provide for reliable analysis by region (Atlantic Canada, Quebec, Ontario, Western provinces, and the North). Oversamples of the Aboriginal population were distributed proportionately across the country.


Public Education
In terms of public education requirements, the 2008 National Benchmark Survey also revealed that overall, half (51%) of Canadians report that they had heard or read something on the subject of Indian Residential Schools, as compared to eight in ten Aboriginal Canadians. (Environics Research Group, 2008: 13):

Among Canadians generally, awareness of Indian residential schools is highest in the North (86%) and in Western Canada (72%). About half of Ontarians (49%) and Atlantic Canadians (45%) are aware of them, while only 27% of Quebecers could recall seeing or hearing anything about these schools [Note 11].

Figure 1 Awareness of Indian residential schools [Note 12] (May 2008)

Figure 1 Awareness of Indian residential schools (May 2008)

Note: Amounts noted in figure above are percentages of overall response rate of respective sample population.

As seen in Figure 1, overall, half (51%) of Canadians report that they have heard or read something on the subject of Indian residential schools. This proportion rises to 80% among Aboriginal Canadians living on reserve and 79% living off reserve.

Given the extensive media coverage of the Prime Minister's Statement of Apology regarding Indian Residential Schools, on June 11, 2008, it is fair to say that the National Benchmark Study may understate the level of awareness of the general Canadian population. An Angus Reid Strategies poll, released August 11, 2008, revealed that 67% of respondents strongly or moderately agreed with the Apology. Conversely, 17% strongly or moderately disagreed while 16% were not sure [Note 13]. While the survey question does not directly deal with perceptions on the Residential Schools legacy, it does suggest that public education and awareness may have increased some after the statement of apology but there are still gaps in information and awareness.

4.2 Summary / Conclusions

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority, and with current delays in implementation of the Settlement Agreement, there continues to be a need for communication activities.

Among these needs, the evidence suggests that former students continue to require information on ongoing initiatives and will require information on activities yet to be launched. According to program recipients and secondary survey evidence, former students are fairly well informed about the financial compensation related to the Settlement Agreement, however they are predominantly unaware of other benefits available. There also continues to be a need for public education to sensitize the general Canadian population on the legacy of the Indian Residential Schools experience on Aboriginal communities.

Return to Table of Contents





5.0 Findings: Design and Delivery

The questions addressed for this evaluation issue were:

5.1 Key Findings

Performance Measurement

APIP does not have a clearly articulated results based management or performance measurement framework.

Gaps identified through the evaluation research include:

Table 3 Comparison of stated APIP Expected Objectives and Outcomes in Terms and Conditions and Integrated RMAF/RBAF
Terms and Conditions (2006) IRSRC Integrated RMAF/RBAF (2008)
Objectives Purpose
  • Inform and encourage the sharing of diverse viewpoints on a range of Settlement issues, policies or programs that allow the Government to respond to the widest source of information possible and thus improve the quality of Departmental decisions;
  • Identify where changes may be necessary to enhance service delivery;
  • Ensure the Aboriginal community is informed of the benefits available through the IRSSA; and to
  • Help Canadians to understand the Settlement Agreement and the impact that the legacy of Indian Residential Schools has had on Aboriginal communities.
  • To provide[e] advocacy and public education on a diverse range of issues related to the Indian Residential Schools settlement agreement issues, policies and programs, allowing the Government to respond to the widest source of information possible to

    • improve the quality of departmental decisions,
    • identify where changes may be necessary to enhance service delivery, and
    • ensure the Aboriginal community is informed of the benefits available through the Settlement Agreement.
Expected outcomes
  • Final Outcome: Contribute to reconciliation between Residential School survivors and the Government of Canada;
  • Intermediate Outcome: Improved community confidence and trust, access to quality well coordinated Residential School programs and services and improved credibility for Residential School programs and services and the Government; and
  • Expected Result: Former students of Residential schools will have complete access and equal participation in the Settlement Agreement.
  • To improve:

    • community confidence and trust;
    • access to quality well coordinated Indian Residential School programs and services; and
    • improved credibility for Indian Residential School programs and services and the Government
 

Delivery Approach

APIP's strategy was found to be appropriate given the sensitivity of the information to be shared.

The Settlement Agreement takes the sensitivity of residential school issues seriously. All Settlement Agreement program websites, for example, have a disclaimer advising that the subject matter may be disturbing to some visitors and to Survivors of the residential school system and offers information for individuals to seek support if necessary (i.e. 1-800 telephone number).

The Official Settlement Notice Plan on the Indian Residential Schools Settlement Agreement observed that in-person dissemination of information was critical to reach as many former students and their families as possible:

"These grass-root efforts, to be designed chiefly by the AFN and various Inuit organizations, and possibly others, will provide additional notice exposure beyond that which will be provided through mailings and paid measurable media, and will allow for face-to-face explanations of notices and answers to basic questions regarding the Settlement and Class members' rights and options" [Note 15].

In recognition of the sensitivities involved, APIP opted to provide support to existing, credible, organizations, primarily Aboriginal, to approach former students, family and community members and to encourage face-to-face communications (e.g., as noted in Section 3, fundable activities include workshops, conferences and gatherings as well as other activities to support face-to-face meetings with former students).

A significant degree of evidence was found to support APIP's approach, for example:

The evidence also suggests that this strategy was an appropriate response to the Program's two year time frame and the urgency of disseminating information about the Settlement Agreement.

Case studies, key informant interviews, document and administrative data review revealed the engagement of former students, their families and communities in activities. By entering into Contribution Agreements with primarily regional and national Aboriginal organizations, the Advocacy and Public Information program benefited from their experience and knowledge of cultural and community context. According to Survivors interviewed, this approach is both credible and effective in achieving results.

In light of the projects initial two year time frame, Program officials chose to work primarily with Aboriginal organizations with existing capacity in order to achieve efficiencies and quickly distribute funds for communications of the Settlement Agreement.

The one challenge which the evaluation found associated with APIP's approach is that there appears to be no contingency plans in place for cases in which no capable or willing organizations have been identified. At the time of the evaluation, there continue to be seven projects under negotiation, and presumably, target audiences without access to the services provided through the program.

5.2 Summary / Conclusions

The lack of clarity in the program's expected results and performance measurement gaps severely inhibit the Program's capacity to monitor performance, measure the achievement of expected results, or identify issues or factors which may differentially affect access to information and participation in benefits from the Settlement Agreement. These gaps have also severely limited the evaluation's capacity to assess expected results and cost-effectiveness.

APIP's strategy of engaging credible and trusted Aboriginal organizations was found to be appropriate given the sensitivity of the information to be shared. It is also an appropriate response to the urgency of disseminating information about the Settlement Agreement. One challenge, however, is that no contingency plans are place to inform individuals and communities when capable and willing organizations at the community level cannot be identified in a timely manner.

Return to Table of Contents





6.0 Findings: Success and Impacts

The questions addressed for this evaluation issue were:

6.1 Key Findings

The evaluation was unable to assess to which degree the Program's expected results were being achieved.

As discussed previously, the following variables have severely limited the evaluation's ability to assess achievement of expected results: the lack of a program specific logic model, significant gaps in baseline data, performance information and the absence of targets.

Another factor with significant limitations in the assessment of the evaluation was the short timeframe that has lapsed since APIP began implementation. The first set of contribution agreements were only signed in April 2007. Furthermore, the potential to assess outcomes was limited by the fact that due to delays in Settlement Agreement activities APIP has not yet disseminated information on all planned Settlement Agreement activities (see Section 4.1).

The evaluation did identify a number of promising practices towards meeting the needs of former students.

APIP has made some contribution toward sensitizing Canadians.

The Legacy of Hope Foundation (LHF) is a national, charitable organization whose purpose is to educate and create awareness and understanding about the legacy of residential schools and the effects and intergenerational impacts on First Nations, Inuit and Métis peoples and to continue to support the ongoing healing processes [Note 21]. The following two LHF projects funded by APIP are examples of some promising practices:

Overall, the evidence suggests that, while there have been notable achievements in reaching out to the general Canadian population, APIP's main focus has been on former students, their families and their communities.

While national projects tended to focus on public information, regional level projects have tended to focus on disseminating information regarding the provisions of the Settlement Agreement, as well as its associated benefits and programs. Bearing in mind that projects may undertake activities related to the dissemination of information and public education, 12 of 19 national level projects focused on public education, and 9 on disseminating information to potential beneficiaries or service agencies. 25 of the 30 regional projects focussed on the Settlement Agreement; only 7 focused on public education.

The evidence is mixed on the extent to which APIP has assisted former students to gain access to and equal participation in the Settlement Agreement programs.

Table 4, below, uses Departmental estimates of Residential School Attendees as a proxy for baseline data on information gaps and needs by region as of September 2007 (the implementation date of the Settlement Agreement). It is important to note that this analysis does not aim to draw a correlation between APIP funding and Settlement Program uptake, which is partially due to challenges with attribution given other parties involved, and gaps in performance measurement. Rather the analysis is intended to demonstrate the potential of using such information to identify where the information needs might be most required and to compare those needs with APIP expenditures.

In general, APIP project funding seems to be proportional to regional needs (comparison of columns B and F). In some areas, where the funding was somewhat less, such as Alberta and Saskatchewan, this did not seem to have a bearing on uptake. 105% and 115% of Estimated Residential School Attendees in these regions were issued a Common Experience Payment, exceeding estimations. This suggests that information and supports in these regions are helping former students access Common Experience Payments.

Quebec is the only region that received proportionately less funding compared to estimated baseline need, while percentage of payments is significantly less than initial estimations. Within the Quebec case study, which was conducted for the evaluation, one challenge that was identified was accessibility to information in French, which may offer a partial explanation for the results achieved.

Table 4: Comparative Analysis of Common Experience Payments Issued as percentage of Estimated Residential School Attendees and Proportion of overall APIP project funding by Region
  A
Estimated Residential School Attendees (2005)*
B
Region as % of overall estimated attendees
C
Payments issued**
D
Payments as % of estimated attendees
E
APIP (07-09) Projects Funding (in thousands)
F
Proportion of overall APIP project funding by Region
BC   14,879 17% 10,233 90% $849 16%
AB   11,224 13% 8,754 105% $401 7%
SK   15,559 18% 14,362 115% $418 8%
MN   8,952 11% 6,550 95% $779 14%
NT Territories 6,442 8% 4,231 142% $734 13%
NU 1,977 $310 6%
YK 1,142 $412 8%
ON   10,427 12% 6,275 74% $426 8%
QC   11,107 14% 4,766 49% $561 10%
NL Atlantic Region 1,277 2% 16 46%    
NB 101 $555 10%
NS 298
PEI 35
OTHER   371 5% 16      
TOTAL   80,238 100% 58,756   $5,445 100%

(Source: See Appendix A)

Based on further analysis of departmental data (see Appendix A), the median of payments made on applications was 80% while the average was 75% (see Appendix A). This offers an illustration that on average, approximately one in five applicants were either deemed ineligible for payment or their applications were put forward for reconsideration. This offers an indication which information that was available to individuals related to accessing programs, was insufficient for effective participation and/or that people were unclear about the parameters of the Settlement Agreement (Residential Schools vs. Day Schools).

It is worth noting that the North and Saskatchewan had significantly higher rates of payments issued in relation to estimations, 142% and 115% respectively. his suggests that the mechanisms for disseminating information within these regions may have contributed to the overall success of reaching intended audiences. Again these results cannot solely be attributed to APIP, but APIP's efforts may have some implications for understanding regional variances. One factor which may have contributed to high uptake in the North is the attention paid by the recipient organization on the importance of addressing linguistic diversity. The Dene Nation, for example, ensured that conference communications were conducted in five local languages.

The evaluation evidence collected through key informant interviews, case studies and the document review also suggests that clarity and quality of information provided may be an issue affecting uptake and participation rates. Steps were put in place at the onset of the implementation of the Settlement Agreement to ensure consistency in the information imparted by differing parties [Note 22]. However, key informant interviews and case studies indicated repeated cases of individuals who did not understand why the school they attended was not included in the Settlement Agreement (i.e. day schools) and that application forms were difficult to complete. This raises questions about the clarity of approved communications materials for intended audiences and whether some revisions may be necessary.

6.2 Summary / Conclusions

While a number of promising practices were identified, the evaluation was unable to assess the degree to which the Program's expected results are being achieved due to the weaknesses in performance measurement, and the short time frame in which the program has been operating.

According to an analysis of departmental data, APIP project funding seems to be distributed in proportion to regional needs, but no simple correlation was found between funding and uptake. The evidence suggests that future initiatives should take additional steps to ensure the clarity of the information disseminated and ensure mechanisms are in place to identify communication related issues affecting applications.

Return to Table of Contents





7.0 Findings: Effectiveness

The evaluation questions addressed for this issue were:

7.1 Key Findings

Gaps in performance data and the short-time frame in which projects have operated negated the potential to evaluate the program's cost-effectiveness.

As discussed previously, the lack of a program specific logic model, significant gaps in baseline data, performance information and the absence of targets severely limited the evaluation's ability to assess key evaluation questions, including, in this case, cost effectiveness.

Efforts are being taken to reduce overlaps and gaps and to enhance coordination among Program Interlocutors and Contribution Recipients to strengthen implementation of Settlement Agreement.

The evidence suggests a promising informal division of labour between Settlement Agreement programs and APIP supported organizations has emerged, as has an informal network of these parties. According to Key Informants and case studies, Contribution Recipients are offering former students general information about the Settlement Agreement and related programs, and Program Interlocutors are responding to more technical aspects related to their areas of responsibility.

Additionally, it was noted that the practice of having Program Interlocutors attend APIP supported workshops is proving effective, both in transmitting information and as a forum for raising issues and concerns. Gaps in performance data do not allow an assessment of the degree to which the respective programs have been able to act on information provided.

An example of this is one concern that was raised repeatedly in the key informant interviews and case studies. Here it was noted that the multiple 1-800 telephone numbers related to different programs and supports served to complicate matters as stakeholders felt it was hard to discern which one to contact for what purpose. Informants reported that when people did call they were often forwarded from one to another, yet this still did not help to answer their questions. Specific examples of what questions went unanswered were not provided). Additionally, a few key informants noted concerns about the cultural sensitivity of telephone operators in responding to former students.

Program officials admit that they have facilitated teleconferences which bring together Contribution Recipients to share information, but that the calls do not take place on a consistent basis. Most recently, however, they have established an informal network of Contribution Recipients and Program Interlocutors in the interest of sharing information amongst parties and as a means of fostering two way communications between recipients and interlocutors.

This networking approach is consistent with the RIAS' broader partnership activities that support horizontal cooperation, such as the Community Impacts Working Group. RIAS facilitates discussions among this group, which brings together other government departments and key stakeholders to share information about local, regional and national efforts to implement the Settlement Agreement. At present this group does not convene frequently, however RIAS has made a recent commitment to utilize this resource more effectively. This could present an opportunity to serve as a mechanism to help coordinate communications activities to support consistency of information dissemination and to identify areas for improvement in service delivery and avoid duplication of efforts.

As useful as the above efforts may be for reducing the potential of overlap and for harmonizing communication activities, the current Settlement Agreement Structure lends itself to potential overlaps which may increase as differing components come on line. As earlier indicated, while APIP is the only program currently working with Aboriginal organizations for the purpose of communications:

7.2 Summary / Conclusions

With more Settlement Components coming on-line, and some potential opportunities for overlap being noted now is the time to ensure the roles and responsibilities are clear and complementary, and structured or streamlined in such a manner so as to best facilitate uptake.

As in other areas of investigation for this evaluation, efforts to examine cost-effectiveness were negated by the lack of a program specific logic model, significant gaps in baseline data, performance information as well as the absence of targets.

Return to Table of Contents





8.0 Conclusions and Recommendations

The evaluation supports the following conclusions on relevance, design and delivery, success and effectiveness.

8.1 Conclusions

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority. With current delays in the implementation of the Settlement Agreement, there continues to be a need for communication activities directed to survivors, their families and communities and the broader Canadian population.

APIP's strategy of engaging credible and trusted Aboriginal organizations was found to be appropriate given the sensitivity of the information to be shared. It is also an appropriate response to the urgency of disseminating information about the Settlement Agreement. One challenge, however, is that no evidence emerged over the course of the evaluation that a formal contingency plan is in place to inform individuals and communities when capable and willing organizations at the community level cannot be identified in a timely manner.

The lack of clarity in the program's expected results and performance measurement gaps severely inhibit the Program's capacity to monitor performance, measure the achievement of expected results, or identify issues or factors which may differentially affect access to information to and participation in benefits from the Settlement Agreement. These gaps have also severely limited the evaluation's capacity to assess expected results and cost-effectiveness.

In addition to performance measurement gaps, the short time frame in which APIP has been operating also limited the potential for assessing impacts at this time. A number of promising practices were identified in terms of maximizing the reach of communication activities. Moreover, according to an analysis of departmental data, APIP project funding seems to be distributed in proportion to regional needs, but no strong correlation was found between funding and uptake. The evidence suggests that future initiatives should take additional steps to ensure clarity in information disseminated and ensure mechanisms are in place to identify communication related issues affecting applications.

Efforts are being taken to reduce overlaps and gaps and to enhance coordination in order to enhance the results of Residential School related communications. With more Settlement Components coming on-line, and some potential opportunities for overlap being noted, now is the time to ensure the roles and responsibilities are clear and complementary, and structured or streamlined in such a manner so as to best facilitate uptake.

8.2 Recommendations

  1. In light of the revised implementation schedule for various Settlement Components, INAC should consider continuing support for communication activities, based in part on the APIP model, to assist in informing former students and Canadians about the Settlement Agreement;
  2. Clarify roles and responsibilities of the various parties involved in disseminating information about the Settlement Agreement so as to ensure cost-effectiveness in the delivery of information and reduce the potential for duplication and overlaps over time; and
  3. Depending on the approach adopted, develop a results based management tools and performance measurement strategy, that is mindful of reporting burden, yet includes:

    • a program specific logic model demonstrating links to the Settlement Agreement objectives and INAC's program activity architecture;
    • a needs assessment exercise which draws upon existing data (e.g., baseline study, program uptake rates) disaggregated by location, age and gender;
    • performance measurement indicators and targets;
    • formalized recipient selection and assessment criteria which are clearly aligned with program objectives (particularly with respect to reach)
    • results-based reporting requirements; and
    • a client survey to be applied for all activities undertaken (i.e. clarity and utility of information provided, change in understanding, and ability to act based on information provided).
Return to Table of Contents





Appendix A:

Comparative Analysis of Estimate Residential School Attendees and Common Experience Payment

  BC AB SK MN NT NU YK ON QC NL NB NS PEI OTHER TOTAL
          Territories     Atlantic Region    
Estimated Residential School Attendees (2005)* 13,947 10,651 14,586 8,490 6,119 9,851 10,891 1,214 4,263 80,012
Region as % of overall estimated attendees 17% 13% 18% 11% 8% 12% 14% 2% 5% 100%
CEP Applications received** 16,302 16,041 22,072 13,043 6,100 3,051 1,567 9,152 6,577 74 148 426 47   94,600
          10,718     695    
% Applications vs Estimated Attendees 117% 151% 151% 154% 175% 93% 60% 57%    
Payments issued** 12,584 11,199 16,827 8,039 5,113 2,220 1,372 7,303 5,359 20 129 365 42 828 71,400
Payments as of % of applications 77% 70% 76% 62% 84% 73% 88% 80% 81% 27% 87% 86% 89%    
Payments as % of estimated attendees 90% 105% 115% 95% 142% 74% 49% 46%    
Applications not eligible for payment** 3,293 4,051 3,892 4,354 859 694 161 1,432 915 52 16 48 4 66 19,837
Ineligible for payment as % of applications 20% 25% 18% 33% 14% 23% 10% 16% 14% 70% 11% 11% 9%    
Reconsideration** 3,398 3,977 5,366 3,121 1243 527 319 2,090 1,538 10 35 105 9 153 21,891
Reconsideration as % of applications 21% 25% 24% 24% 20% 17% 20% 23% 23% 14% 24% 25% 19%    
APIP (07-09) Projects Funding (in thousands) $849 $401 $418 $779 $734 $310 $412 $426 $561   $555   $5,445
Regional proportion of total APIP funding 16% 7% 8% 14% 13% 6% 8% 8% 10%   10%   100%
 
Return to Table of Contents





Appendix B: Management Response / Action Plan

Recommendations Actions Responsible Manager (Title) Planned Implementation and Completion Dates
1. In light of the revised implementation schedule for various Settlement Components, INAC should consider continuing support for communication activities, based in part on the APIP model, to assist in informing former students and Canadians about the Settlement Agreement; RIAS agrees with the recommendation and has sought and obtained the necessary authorities to continue APIP through 2009/2010 to 2012/13. Funding beyond 2009/10 is still to be identified. Director General Policy, Partnerships and Communications Implementation:
April 1, 2009

Completion:
March 31, 2013
2. Clarify roles and responsibilities, identify means for continuing information sharing of the various parties involved in disseminating information about the Settlement Agreement so as to ensure cost-effectiveness in the delivery of information and reduce the potential for duplication and overlaps over time; and RIAS agrees with the recommendation and has begun the process of developing a strategic outreach approach which will clarify roles and responsibilities, ensure cost-effectiveness in the delivery of information (and reduce the potential for gaps and overlaps) as well as ensure regular communications between APIP Contribution Recipients and Program Interlocutors (within INAC and with specialists from the various areas of the Settlement Agreement) and coordination of the development and dissemination of accurate and up to date information relevant, i.e., Common Experience Payment (CEP), Reconsideration, Independent Assessment Process (IAP), Commemoration and the Truth and Reconciliation Commission (TRC) to survivors, their families and communities.

In addition, the Community Impacts Working Group (CIWG), a network of interdepartmental representatives and Aboriginal organizations, will be reconstituted this spring and one of its goals will be to improve coordination of communications locally, nationally and regionally.
Director General Policy, Partnerships and Communications Implementation:
On-going

Completion:
December 31, 2009
3. Depending on the approach adopted, develop a results based management tools and performance measurement strategy, that is mindful of reporting burden, yet includes:
  • a program specific logic model demonstrating links to the Settlement Agreement objectives and INAC's program activity architecture;
  • an analysis of information gaps which draws upon existing data (e.g., baseline study, program uptake rates and other relevant evidence) disaggregated by location, age and gender;
  • performance measurement indicators and targets;
  • formalized recipient selection and assessment criteria which are clearly aligned with program objectives (particularly with respect to reach)
  • results-based reporting requirements; and
  • a client survey to be applied for all activities undertaken (i.e. clarity and utility of information provided, change in understanding, and ability to act based on information provided).
RIAS will develop a performance measurement strategy, that will include the recommended elements (at left), including a logic model, development of indicators and targets. RIAS's approach will include a reporting template which focuses on results and the collection of performance data, including feedback from participants at specific activities.

An analysis of information gaps and recipient reach will be developed and reviewed on a regular basis to assess communication needs and priorities, activities and targets and will inform funding decisions.

Applications for APIP support will be requested and assessed in accordance with identified communication requirements and capacity to achieve objectives and performance measurement requirements. No further formalized selection and assessment criteria are envisaged at this time.

A contingency plan will be developed to address gaps in coverage should capable and willing recipients not be identified in a timely manner.
Director General Policy, Partnerships and Communications Implementation:
On-going

Completion:
December 31, 2009
 
Return to Table of Contents





Footnotes

  1. This component of the Settlement Agreement is delivered by Health Canada through the Indian Residential Schools Resolution Health Support Program. (return to source paragraph)
  2. Key sources: IRSRC, (2008: 5), Integrated Results-based Management and Accountability Framework and Risk-Based Audit Framework (RMAF/RBAF): IRSRC, Departmental Performance Report, (2007 – 2008), and Statistics Canada, (2003: 22), Aboriginal Peoples Survey 2001 - Initial findings: Well-being of the non-reserve Aboriginal population, (Catalogue 89-589). (return to source paragraph)
  3. Indian Residential Schools Settlement Agreement, Preamble, B and C, (2007: 6). (return to source paragraph)
  4. See: IRSRC, Departmental Performance Report (2007-08); Siggner and Associates Inc., Estimating the Residential School Attendee Population – For the Years 2001, 2005 and 2006. (return to source paragraph)
  5. EPMRB expects to begin work the first quarter of 2009 on the development of an Evaluation Framework for Settlement Agreement programs administered by RIAS (the Common Experience Payment, the Independent Assessment Process, and the Commemoration Program). (return to source paragraph)
  6. The number of projects focused on disseminating information on the Settlement Agreement and public education exceeds the total number of projects because some projects address both types of activities. (return to source paragraph)
  7. Indian and Northern Affairs Canada (return to source paragraph)
  8. The ordering and wording of the questions have been slightly revised from the Terms of Reference approved by the Audit and Evaluation Committee in June 2008. To note: the final question in the list above, related to program improvements, is largely addressed in the evaluation conclusions and recommendations (Section 8). (return to source paragraph)
  9. 39th Parliament, 2nd Session, Edited Hansard, Number 110, June 2008. (return to source paragraph)
  10. Environics Research Group, (2008), National Benchmark Survey, May 2008. The survey measured the level of awareness of the general Canadian population,, with an oversampling of Aboriginal people living on and off reserve, on the Settlement Agreement and the legacy of Indian Residential Schools. (return to source paragraph)
  11. Ibid, page 14. (return to source paragraph)
  12. Ibid. (return to source paragraph)
  13. Angus Reid Strategies, (2008: 3), Canadians Agree with Prime Minister's Apology to Aboriginal Community, August 2008. (return to source paragraph)
  14. id="note14">The three indicators noted in the IRSRC Integrated RMAF/RBAF are: # conferences, community workshops, information sessions attended and presentations made by IRSRC staff directly to former students, their families, and front-line support workers; feedback from Community Impacts Working Group [Note this acronym is not defined or mentioned elsewhere in the document); and of communities visited by project staff. (return to source paragraph)
  15. Hilsoft Notifications, (2006: 53), In re Residential Schools Class Action Litigation Settlement Notice Plan, February 2006. This plan, approved as part of the Indian Residential Settlement Agreement, was designed with input from Aboriginal people and groups, lawyers for the parties, the Government and with direction from the Courts. The focus of the Plan was on opt-out (of the Settlement Agreement) and advance payments and had objectives in this context to notify the greatest number of former residential school students and their family members, and provide them with opportunities to see, read, or hear notice, understand their rights, and respond if they choose. (return to source paragraph)
  16. Fred Kaufman, (2002: 21)l Searching for Justice: An Independent Review off Nova Scotia's Response to Reports of Institutional Abuse, Chapter XVII - Report of the Law Commission of Canada, Province of Nova Scotia, January 2002. (return to source paragraph)
  17. Solicitor General Canada, (1994: 27), Victims Of Crime and the Justice System In Ontario: An Issues Paper. (return to source paragraph)
  18. Ibid. p. 31. (return to source paragraph)
  19. Ibid. (return to source paragraph)
  20. Indian Residential School Survivors, (2007: 2), National Conference and Workshops - Final Report: Preparing Survivors and Communities to Move Forward, Winnipeg, 2007. (return to source paragraph)
  21. Legacy of Hope. (return to source paragraph)
  22. The Document Review indicated that Program officials advised contribution agreement holders upfront that communications were to be consistent with the formal court Notice Plan that all communications materials must be consistent with that Notice process that was approved by the National Certification Committee. All workshops conducted with former students of IRS were to offer products approved by the NCC and use the official court notices. (return to source paragraph)
 
 
Date modified: