ARCHIVED - Evaluation of the Comprehensive Community-based Planning Pilot Project in Saskatchewan

Archived information

This Web page has been archived on the Web. Archived information is provided for reference, research or record keeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Date: June 2011
Project Number: 1570-7/10031

PDF Version (278 Kb, 67 Pages)




Table of Contents




List of Acronyms

BC
British Columbia

CEU
Cities and Environment Unit

CCBP
Comprehensive Community-Based Planning

EPMRB
Evaluation, Performance Measurement and Review Branch

AANDC
Aboriginal Affairs and Northern Development Canada

YTC
Yorkton Tribal Council

Return to Table of Contents





Executive Summary>

Aboriginal Affairs and Northern Development Canada (AANDC) conducted an Evaluation of the Comprehensive Community-Based Planning Pilot Project in Saskatchewan to provide evidence-based conclusions regarding the project's relevance and performance. The evaluation was requested by AANDC Saskatchewan region to assess the results achieved by the pilot project, capture lessons and provide recommendations to inform future support.

The $5 million ($4.55 million AANDC, $450,000 Health Canada) pilot project was designed to raise awareness and increase the capabilities of First Nations communities to engage in the development and use of comprehensive community-based plans [Note 1]. Between 2005-2006 and 2010‑2011, AANDC provided funding for the Cities and Environment Unit (CEU), Dalhousie University [Note 2] to work with 11 Saskatchewan First Nations and seven tribal councils to develop and implement community plans. CEU delivered the project in three phases with four First Nations communities involved in Phase 1 starting in 2005-2006. Another four First Nations joined the project 2007-2008 as part of Phase 2, and the final three communities joined the project in 2008‑2009.

CEU facilitated the planning process through a series of workshops that involved community members in identifying strengths and issues affecting the community, establishing a vision, long‑term goals and priorities for community action. It worked collaboratively with communities providing training, technical planning support, and on-going coaching.

The Evaluation, Performance Measurement and Review Branch conducted the evaluation with the assistance of Stiles Associates Inc. Research for the evaluation was conducted between February and May 2011. The evaluation methodology included a document and literature review, interviews with 23 key informants and five case studies developed through field visits to pilot project communities. Given that most of the communities selected to participate in the pilot project had good financial and human resource capacity, the findings on the development of this planning model may be limited to such communities.

The key findings and conclusions of the evaluation are as follows:

Relevance

Findings from the evaluation show that comprehensive community-based planning is consistent with federal roles and responsibilities as it aligns with AANDC and government priorities. All stakeholders interviewed, especially First Nations involved in the pilot project, believe there is a need for comprehensive planning that facilitates community engagement and ownership. Some of the literature and some respondents see comprehensive planning contributing to improved governance and self-reliance in First Nations communities. All lines of evidence point to the need for AANDC to move from "silo," program-based planning to a model that is both comprehensive (holistically examining community needs and priorities) and community-based (allowing for broad-based engagement of community members in the planning process).

Performance

Design and Delivery
The evaluation found that while most pilot communities understood the project objectives, some had a different understanding that led to uncertainty about AANDC's intentions. The project design by the CEU was appropriate and provided an effective, interactive planning model that, with the assistance of a paid Plan Champion, facilitated broad-based community engagement. However, some communities expressed concern that they did not have a say in the project design, that planning was limited to the boundaries of the reserve, and that it focused too much on physical infrastructure and not enough on broader social and economic issues. Neither AANDC Saskatchewan nor the CEU planning model recognized the need to address gender equity issues, such as the differing needs of women and men in planning and gender impacts in implementation.

Evidence showed that the CEU fulfilled the terms of its contracts, smoothly rolling out the seven-stage model by establishing strong relationships with communities. All stakeholders agreed that the Unit was flexible in adapting the project to the local context in different communities. There appeared to have been sufficient resources for planning, but many communities expressed concern about the lack of financial support from AANDC Saskatchewan region to implement the projects and priorities identified in the plans. Other delivery challenges included a lack of engagement by AANDC Saskatchewan staff, the distance between Saskatchewan pilot communities and CEU's offices in Halifax, and insufficient time for the Phase 3 communities that joined the project in 2008-2009 to complete the planning and implementation process.

Effectiveness
Overall, the evaluation found the project achieved its expected outcomes. It raised awareness of the value of comprehensive community-based planning in pilot communities and within tribal councils and AANDC Saskatchewan region, increased the capabilities of pilot communities to develop and use community plans, and led to improved planning practice.

The project built cohesion and trust as community members came together to document their history, articulate their values and develop a joint vision for the future. The majority of communities are using their plans and almost all have implemented at least one of the initiatives identified in the plan. As a result of the plans, some communities have developed new community infrastructure such as gardens or walkways, initiated new partnerships and leveraged funding, or made changes to community governance. In many communities, the planning process created increased expectations that community political leaders would inform and consult with community members. The Plan Champions hired in each community to assist the planning and implementation process built new skills in community facilitation and engagement and increased their self-confidence, which allowed most to go on to other jobs.

However, there is little evidence that the comprehensive community-based plans are being used within AANDC Saskatchewan region beyond informing the recent General Assessment process. [Note 3] There appeared to be little awareness of the plans among current front-line funding services officers and there is no evidence the plans have been used to inform AANDC Saskatchewan region funding priorities for the pilot communities.

The evaluation found that some of the benefits from the planning process will be sustained. In some pilot communities, the plans have provided stability of vision through several elections for Chief and Council. However, progress on initiatives identified in the plans, which often fall outside or across responsibilities in the Band administration, will likely slow without a Plan Champion within the community to move them forward. As part of the pilot project, the CEU worked with five of the pilot communities to design and build a new structure in the community, such as a community market or outdoor classroom, in order to provide tangible evidence of progress on the plan. While communities found this to be a positive process, some of these structures have seen limited use.

The project has not been successful so far to institutionalize the capacity for comprehensive community-based planning in Saskatchewan.

Economy and Efficiency
The planning process was funded at an average cost of about $450,000 per community. The fees paid to the CEU appeared reasonable for the services delivered when compared to the rates charged by professional planning consultants. However, the cost of travel between CEU's offices in Halifax and Saskatchewan, as well as expenses associated with Joint Steering Committee meetings added to the project costs.

Building technical capacity to support First Nations comprehensive community-based planning in Saskatchewan would both increase efficiency and reduce costs over the long-term. Working with geographic clusters of communities could result in cost efficiencies.

Lessons

The evaluation found limited evidence that mechanisms were in place to capture lessons and use them to facilitate reflection and on-going learning at the community level. The CEU worked with pilot communities to develop indicators to track progress in meeting community goals. However, neither it nor AANDC Saskatchewan region demonstrated to communities how monitoring data could be collected and used to inform on-going planning and program improvement.

The evaluation identified a number of lessons:

The evaluation also identified a number of factors contributing to the success of comprehensive community-based planning in Saskatchewan, including:

It is recommended that AANDC, to support comprehensive community-based planning:

  1. Consider ways to institutionalize long-term national support for such planning in First Nation communities.

  2. Review its approach to:
    1. better integrate First Nations input on the design and approach used to develop community-based plans;

    2. involve other levels of government; and

    3. integrate gender-based analysis.

  3. Consider means to increase capacity (inside and outside the Department) to support comprehensive community-based planning.

  4. Commit to using comprehensive community-based planning to inform program and funding priorities.
Return to Table of Contents





Management Response / Action Plan

Project Title: Evaluation of the Comprehensive Community-based Planning Pilot Project in Saskatchewan
Project #: 10031

1. Management Response

Recent work on a number of key initiatives at AANDC – the Community Development Framework (CDF), the Capacity Development Partnerships Program (CDPP), a new Default Prevention and Management Policy, implementation of the 2008 Policy on Transfer Payments and a reduction in reporting burden (harmonized, longer or more flexible funding agreements) – has generated renewed interest in Comprehensive Community Planning. These initiatives seek to strategically invest the Department's capacity resources in a way that responds to gaps in community capacity and community-identified needs, and strategic community planning is key to this approach. These initiatives have created an opportunity for the Department to leverage departmental resources and programs to effectively support community planning and some regions have already been leveraging these opportunities into action on a regional level.

The CDF is an approach that recognizes that Aboriginal communities are at differing stages of development along a continuum. For better outcomes to be achieved, the reality of these differentiated "development stages" and current needs must be respected in policy and program development, as well as in implementation because national one-size-fits-all approaches are not effective. Generally speaking, current programs are "output" oriented. Communities that have completed strategic plans generally have articulated clear, integrated directions that cannot readily accommodate an outside funder's ‘output-driven' approach. Investing in community strategic planning is one way of focusing on outcomes rather than activities and, because communities will vary in their pace of change, our commitment to sustain attention and engagement in this area will be important to the success of initiatives that support community planning.

A large body of literature confirms that community-based solutions are the only ones capable of supporting progress in a community's development. Federal policy for decades has been supportive of promoting healthy, thriving communities. We know that lack of capacity is a barrier to progress in many communities and that community-based planning initiatives are a proven means of addressing capacity challenges. The evaluation notes the significant role AANDC could play in supporting this approach.

In general terms, the evaluation findings support comprehensive community-based planning and there is a wide variety of models and approaches available to communities embarking upon the community planning process. As currently proposed, the CDPP would provide Aboriginal and northern communities access to workshops on community planning and funds to improve planning capacity may also be available under this new program.

2. Action Plan

Recommendations Actions Responsible Manager
(Title/Sector)
Planned Start and
Completion Dates
1. Consider to institutionalize long-term national support for such planning in First Nation communities. We do concur.
(do, do not, partially)
Juliet Balfour, Director, Sustainable Communities

Chris Rainer, Director, Professional & Institutional Development
Start Date:

FY 2011-2012
To the extent that institutionalization means a concept the Department supports and not a specific program to be established. The objective is to support communities in their use of planning to fully engage community members in charting a path forward. The objective is not to create a program necessarily or promote a defined model or approach. We do, agree that we should use all means possible to promote and support good planning practice in communities. The Community Development Framework (CDF) will elaborate on the Department's role in this regard.

While the Capacity Development Partnerships Program (CDPP) will provide some support for community planning, greater and sustained financial support will be needed to realize significant change across all Aboriginal and northern communities.
Completion:
Ongoing
2. Review its approach to:
a. better integrate First Nations input on the design and approach used, to develop community-based plans;

b.< involve other levels of government; and

c. integrate gender-based analysis.
We do concur.
(do, do not, partially)
Juliet Balfour, Director, Sustainable Communities Start Date:

FY 2011-2012
In keeping with (1) above, the Department will not control the content or form of the planning process. However, we will attempt to support and respond to leadership at the community level. We concur that AANDC has a role to play in bringing other levels of government to the table. We do concur that we can offer tools or expertise to conduct gender-based or any other analysis on a community that has engaged in a community planning process. Completion:

Ongoing
3. Consider means to increase capacity (inside and outside the Department) to support community-based planning. We do concur.
(do, do not, partially)
Chris Rainer, Director, Professional & Institutional Development

Juliet Balfour
Director, Sustainable Communities
Start Date:

FY 2011-2012
The CDPP will provide some support for community planning and may be able to fund initiatives that increase planning capacity in communities, however, greater and sustained financial support will be needed to realize significant change across all Aboriginal and northern communities.

The CDF will elaborate on the role Government needs to play to better support community-based planning and on any capacity or other impediments to that role being played.
Completion:

Ongoing
4. Commit to using community-based planning information to inform program and funding priorities. We do concur.
(do, do not, partially)
Juliet Balfour, Director, Sustainable Communities Start Date:

FY 2011-2012
The CDF will recommend that programs look to community-based planning information to inform program and funding priorities. Completion

Ongoing:


I recommend this Management Response and Action Plan for approval by the Evaluation, Performance Measurement and Review Committee

 

Original signed on June 14, 2011 by:

Judith Moe
A/Director, Evaluation, Performance Measurement and Review Branch

 

I approve the above Management Response and Action Plan

 

Original signed on June 15, 2011 by:

Gina Wilson
Sr ADM, Regional Operations Sector

 

The Management Response / Action Plan for the Evaluation of the Comprehensive Community-based Planning Pilot Project in Saskatchewan were approved by the Evaluation, Performance Measurement and Review Committee on June 20, 2011. 

Return to Table of Contents





1. Introduction

1.1 Overview

The Evaluation, Performance Measurement and Review Branch (EPRMB) of Aboriginal Affairs and Northern Development Canada (AANDC) conducted this Evaluation of the Comprehensive Community-based Planning Pilot Project in Saskatchewan to provide evidence-based conclusions regarding the project's relevance and performance.

AANDC Saskatchewan region requested an independent evaluation to assess the results achieved by the pilot project, capture lessons and provide recommendations to inform future support for comprehensive community-based planning. The report includes a program profile, evaluation methodology, evaluation findings relating to relevance, performance and lessons, and conclusions and recommendations.

1.2 Program Profile

1.2.1 Background and Description

In 2005-06, AANDC's Saskatchewan region undertook a pilot project to examine how Comprehensive Community-based Planning could assist in responding to the urgent issues and needs of First Nations communities. The pilot project was intended to overcome the lack of:

The Saskatchewan regional office through Yorkton Tribal Council (YTC) contracted the Cities and Environment Unit (CEU) of the Faculty of Architecture and Planning at Dalhousie University in Halifax to engage four Saskatchewan First Nation communities in Comprehensive Community-based Planning. This type of planning is a process through which a First Nation establishes a direction for the future and determines specific actions to realize that direction. It is comprehensive because it encompasses all aspects of community life, such as infrastructure development, governance, land and resources, health, culture, social issues and the economy. It is community-based because it employs extensive community involvement throughout the development and the implementation of the plan.

AANDC Saskatchewan region selected the first four communities to participate in the pilot project on the basis of six principles.

  1. Pilot communities, through their council, must confirm a willingness and desire to advance Comprehensive Community-based Planning.

  2. Communities from the North and South should be represented.

  3. A pilot community must possess demonstrated financial management capacity and be committed to principles of accountability.

  4. Technical expertise from tribal council advisory staff and planning consultants will be essential in the initial pilots to build comprehensive planning capacity.

  5. Initial pilot communities must be affiliated with a tribal council to ensure linkages are developed and planning is supported over the long-term.

  6. Pilot communities must have demonstrated a commitment to community engagement as part of their day-to-day operations.

In 2007-2008, CEU won a tendered contract to expand the pilot project and four more communities were added in Phase 2. In 2008-2009, three communities entered the pre‑planning stage of the process as part of Phase 3.

PHASE 1
COMMUNITIES
PHASE 2
COMMUNITIES
PHASE 3
COMMUNITIES
1. Flying Dust First Nation 1. Cowessess First Nation 1. Big River First Nation
2. George Gordon First Nation 2. Kinistin Saulteaux Nation 2. Lac La Ronge Indian Band
3. Shoal Lake Cree Nation 3. Muskoday First Nation* 3. Standing Buffalo Dakota Nation
4. Kahkewistahaw First Nation 4. Pasqua First Nation

*Muskoday First Nation withdrew from the pilot project in April 2010.

At the outset, Phase 1 communities were to receive three years of support. However, these First Nations successfully lobbied for continued funding to support the implementation of their plans and as subsequent First Nations were added to the project, communities from the earlier phases continued to receive support and acted as mentors for the newer communities. Out of 11 communities that participated in the pilot project, 10 were still involved when it ended in March 2011.

Over the course of the project, the CEU submitted two proposals to AANDC Saskatchewan region for funding to establish a diploma program in First Nations community planning, first with the First Nations University in 2006 and then with the First Nations University and the University of Saskatchewan in 2010. Such a program was to provide on-going access to local planning expertise in Saskatchewan. While AANDC Saskatchewan region set aside funding in 2010-2011 to support such an initiative, it has yet to become a reality.

The First Nations Community Planning Model (2003, 2nd Ed.), as developed by the CEU, uses a seven-stage process for developing a comprehensive community-based plan.

  1. Stage 1: Gathering Background Information - Collect basic facts and perceptions about the community so that aspects which stand out as high and low points can be identified.

  2. Stage 2: Identifying Strengths & Issues - Concentrate on recording and understanding problems that need to be dealt with and opportunities that can be built on.

  3. Stage 3: Researching Root Causes - Explore strengths and issues to reveal the root causes and the consequences of no action.

  4. Stage 4: Establishing a Vision - Establish a long-term, ambitious and appropriate direction for the community.

  5. Stage 5: Building a Framework - Translate the vision, issues and values into a blueprint for concerted action in terms of policies, administration, priority action areas and physical improvements.

  6. Stage 6: Developing an Implementation Strategy - Develop an implementation strategy and determine priority projects. Organize the necessary resources to realize the projects.

  7. Stage 7: Monitoring the Plan & Projects - Evaluate the impact of projects individually, the effects of planning as a whole, and revise the plan on an on-going basis.

1.2.2 Objectives and Expected Outcomes

The overall objective of Comprehensive Community-based Planning pilot project is to make community-based planning (developing comprehensive local, long-term, and action-oriented plans) an on-going practice for First Nations. Since no formal logic model or performance measurement framework was developed for the pilot project, the evaluators developed a logic model for the project (see Appendix C). This logic model was validated by AANDC Saskatchewan region, the CEU, and YTC, which administered the contract with the CEU on AANDC's behalf.

The logic model articulates the following outcomes:

1.2.3 Program Management, Key Stakeholders and Beneficiaries

AANDC Saskatchewan region established a Joint Steering Committee to manage and oversee the pilot project. This committee included representatives of AANDC Saskatchewan region, the CEU, participating First Nations Communities and their corresponding tribal councils, and Health Canada. While Health Canada contributed to funding for the pilot project, the Department's involvement was largely confined to attending Joint Steering Committee meetings.

The key beneficiaries of the pilot project were the participating First Nations. Within each community, AANDC Saskatchewan region provided funding for the First Nation to hire a Plan Champion to guide the planning process, facilitate community engagement, ensure that all information gathered and created was shared with the entire community, and report on progress to the Chief and Council.

A Community Contact, usually a member of the band administration, supervised the pilot project. A Planning Work Group comprised of a cross-section of community members worked with the Plan Champion to ensure broad-based community involvement in, and support for, plan development and implementation.

The tribal councils associated with the pilot project communities were to provide technical and professional support for the development and implementation of the community plans as they already played a role in assisting First Nations to fulfill AANDC planning requirements.

1.2.4 Program Resources

An examination of the fees charged by the Unit shows that they appear reasonable for the services delivered as they were lower than rates normally charged by professional planning consultants. However, the distance between Halifax and Saskatchewan added to the cost of the project. While the total travel costs are not broken down in CEU contracts, budget documents show that airfare alone comprised nearly $150,000, or six percent of Dalhousie's costs. The Joint Steering Committee meetings were also costly, bringing together between 15 and 50 participants from around the province for one- or two-day meetings several times a year. The meetings appear to have cost about $260,000 and comprised about five percent of the project.

  2005-2006 2006-2007 2007-2008 2008-2009 2009-2010 2010-2011 Total
Category Amount % of
Total
Amount % of
Total
Amount % of
Total
Amount % of
Total
Amount % of
Total
Amount % of
Total
Amount % of
Total
CEU, Dalhousie 6,000 4.7 235,060 47.9 389,397 48.0 491,850 48.0 588,000 44.5 524,253 42.8 2,234,560 44.7
                             
First Nations/
Tribal Councils
110,000 86.6 236,940 48.3 396,720 48.9 508,150 49.6 707,500 53.6 575,082 47.0 2,534,392 50.7
                             
Joint Steering
Committee
Administration*.
11,000 8.7 19,000 3.9 25,000 3.1 25,000 2.4 25,000 1.9 25,000 2.0 130,000 2.6
                             
Other
(Diploma program)
                   

100,000

8.2

100,000 2
Totals 127,000 100.0 491,000 100.0 811,117 100.0 1,025,000 100.0 1,320,500 100.0 1,224,335 100.0 4,998,952 100.0

(Source: AANDC Saskatchewan region)
* it represents only the amount spent by AANDC Saskatchewan region and not the total cost.






2. Evaluation Methodology

2.1 Evaluation Scope and Timing

The evaluation examined the Comprehensive Community-based Planning pilot project activities from its start in 2005-2006 through to its completion in 2010-2011. Research for the evaluation was conducted between February and May 2011.

2.2 Evaluation Issues and Questions

The evaluation focused on the following issues: (The full evaluation matrix can be found in Appendix B.)

  • Relevance
    • Continued Need
      Does comprehensive community-based planning address a demonstrable need in Aboriginal communities?

    • Alignment with Government Priorities
      Are the objectives of comprehensive community-based planning consistent with departmental and government-wide priorities? Is it aligned with federal roles and responsibilities?

    • Alignment with Federal Roles and Responsibilities
      Does the Comprehensive Community-based Planning pilot project duplicate or overlap with other departments or jurisdictions?

  • Performance
    • Design
      To what extent were the project objectives/outcomes clear to all major stakeholders and shared?
      To what extent was the project design appropriate?
      To what extent was the governance structure appropriate?

    • Delivery
      Were there sufficient and appropriate resources and support to implement the project?
      Was there sufficient flexibility to adapt to changing conditions/local contexts (language and culture)?

    • Effectiveness
      What did the project achieve?
      To what extent is there increased awareness (on the part of pilot communities, AANDC Saskatchewan region and tribal councils) of the importance of a comprehensive community-based plan?
      To what extent have individual and community stakeholders increased their capabilities regarding comprehensive community-based planning?
      To what extent are the community plans being used by communities and AANDC Saskatchewan region?

    • Sustainability
      To what extent will the benefits of comprehensive community-based planning continue after the pilot projects end?
      Have the pilots had unexpected outcomes, positive or negative?

    • Economy and Efficiency
      Are the most appropriate and efficient means being used to develop community plans?
      Are there alternative models that could improve comprehensive community-based planning or reduce costs?

    • Lessons
      What mechanisms are in place to capture lessons and facilitate learning from the pilot initiatives?
      Are there lessons or alternatives that have emerged that could contribute to improved community planning?

2.3 Evaluation Methodology

2.3.1 Data Sources

The evaluation findings and conclusions are based on the analysis and triangulation of the following lines of evidence:

Literature Review:

The evaluators conducted a review of relevant regional, national and international literature related to community-based planning.

Document review:

The evaluators reviewed project documentation, including management information, contracts, project reporting, newsletters, budget information, community plans, design-build project booklets, and other related information.

Key informant interviews:

The evaluators conducted interviews with 23 key informants, including:

  • Representatives of four pilot communities (6)

  • Cities and Environment Unit (3)

  • Tribal councils (3)

  • AANDC staff in Saskatchewan region, British Columbia region and Headquarters (10)

  • Health Canada representative (1)

Case Studies:

The evaluators visited five of the pilot communities over two weeks in early March 2011 to collect detailed qualitative data on the project's outcomes. In each of the communities, the evaluators conducted interviews and/or focus groups with the Plan Champion, Community Contact, Planning Work Group members, and community leadership. All five case studies incorporated information from the document review, including the community plan, project reporting and community profile data. The case studies examined best practices, lessons, key successes and how they were achieved, as well as factors that were important in facilitating or hampering success.

2.3.2 Considerations, Strengths and Limitations

There were few limitations to the evaluation. Evaluators received excellent cooperation from project stakeholders and the pilot communities, with representatives of nine of the eleven pilot communities participating in the evaluation. However, most of the communities selected to participate in the pilot project had good financial and human resource capacity, a characteristic that may limit the applicability of findings from this evaluation to similar communities, i.e. the findings cannot necessarily be generalized to communities with lower levels of financial and human resource capacity.

Since the project lacked a logic model, the evaluators created one and validated it with three members of the Joint Steering Committee representing AANDC Saskatchewan region, with the CEU, and with YTC. The project also lacked a performance measurement framework and reporting was activity-based. Project outcomes were not tracked or reported systematically. Despite this, the evaluators were able to obtain sufficient evidence to triangulate the evaluation findings.

2.4 Roles, Responsibilities and Quality Assurance

The evaluation was performed by EPMRB with the assistance of Stiles Associates Inc. Three members of the Joint Steering Committee representing AANDC Saskatchewan region, Yorkton Tribal Council and the CEU were consulted in the preparation of the evaluation methodology. They also validated the preliminary findings of the evaluation and reviewed the draft evaluation report.

2.5 Data Analysis

When presenting qualitative data in this report, certain terms are used to indicate the proportion of respondents or case studies to which the finding refers. Other terms indicate the frequency with which respondents expressed a particular view. These terms are roughly equivalent to the following percentages:

Proportional Term Frequency Term Percentage
All Always 100
Almost all Almost always 80-99
Many Often, usually 50-79
Some Sometimes 20-49
Few Seldom 10-19
Almost none Almost never 1-9
None Never 0

Findings may not apply to certain respondents or case studies because no response was provided or no response could be inferred from other comments; or because a different response was provided. Where possible, alternative views as well as the views of the majority are presented.






3. Evaluation Findings - Relevance

The evaluation looked for evidence that the Comprehensive Community-based Planning pilot project is consistent with federal roles and responsibilities, responds to the needs and priorities of Saskatchewan First Nations communities, and does not duplicate or overlap with other departments or jurisdictions.

Findings from the evaluation show that comprehensive community-based planning is consistent with federal roles and responsibilities as it aligns with AANDC and government priorities. All stakeholders interviewed, especially First Nations involved in the pilot project, believe there is a need for comprehensive planning that facilitates community engagement and ownership. Some of the literature and some respondents see comprehensive planning contributing to improved governance and self-reliance in First Nations communities. All lines of evidence point to the need for AANDC to move from "silo," program-based planning to a model that is both comprehensive (holistically examining community needs and priorities) and community-based (allowing for broad-based engagement of community members in the planning process). The evaluation found no evidence of duplication or overlap between the pilot project and planning efforts by AANDC or other jurisdictions.

3.1 Consistency with Government Priorities

The Comprehensive Community-based Planning pilot project remains highly consistent with AANDC's mandate to "develop, healthier more sustainable communities". The pilot project delivered on a commitment made in AANDC's Sustainable Development Strategy 2007-2010 to support First Nations community planning. By enabling communities to articulate and document their long-term vision, needs and priorities, comprehensive community-based planning supports the requirements contained in the federal government's Policy on Transfer Payments that AANDC take a more citizen-focused, risk-based approach to how it manages transfer payments to Aboriginal communities.

For the period 2011-2014, AANDC has three departmental planning priorities: 1) transforming for improved results; 2) improving partnerships and relationships; and 3) managing resources effectively. To support the second priority, the Department is developing a new Community Development Framework that is more community-driven and more responsive to community needs than before. Community-level strategic planning has been identified as one of the potential tools supporting such a shift.

Comprehensive community-based planning supports the federal government's commitment to reduce the reporting burden facing Aboriginal communities by modelling a more comprehensive approach. The Auditor General of Canada identified this need in 2002 as did the Blue Ribbon Panel on Grants and Contributions Programs in 2006. Comprehensive community-based plans could potentially be used to assist AANDC in streamlining programming, thus, supporting the Department's priority of "managing resources effectively."

3.2 Need

Numerous sources, including the Royal Commission on Aboriginal Peoples, have documented the persistent challenges confronting First Nations communities such as high rates of unemployment, poverty, health problems and rapid population growth. There is growing realization that the multiplicity of nationally designed and individually administered programs dealing with specific issues is inefficient and often ineffective. A fragmented approach to planning and programming fails to examine the interconnections among the challenges facing First Nations communities and puts pressure on the limited human resources capacity available in small First Nations, some of which may have fewer than 1,000 members.

All lines of evidence point to the need for AANDC to move from "silo," program-based planning to comprehensive planning across all areas of importance to First Nations using processes that allow community members to take an active role in articulating community needs and priorities. Canadian and international literature from community planning (CEU 2010, Cook 2008, Copet 2003, Harivel and Anderson 2008, Public Works and Government Services Canada 2004, Wesley-Esquimaux and Calliou 2010, Wolfe 1989), environmental health (Lee 2002), Aboriginal economic development (Cornell and Kalt 1998), psychology (Thurman et al. 2003), and international development (World Bank 2009, Mathie and Cunningham 2002) support comprehensive, community-driven development approaches. In the late 1960s, 1970s and early 1980s, AANDC supported a number of comprehensive community planning initiatives. However, the success of those initiatives was limited due to a range of problems, including low levels of funding, incomplete implementation strategies, bureaucracy that hindered program effectiveness and the use of external consultants with limited understanding of Aboriginal cultures. (Copet 2003, Wolfe 1989)

All of those interviewed for this evaluation agreed there is a need for a planning model that is comprehensive and that facilitates community engagement and ownership. The case studies provide further evidence, as do AANDC evaluations of similar comprehensive planning initiatives in the Atlantic region and British Columbia. Some of the literature (Copet 2003, Wolfe 1989, CEU 2010) and some respondents from First Nations, tribal councils and Government spoke of the contribution comprehensive community-based planning can make to improved community governance and self-reliance.

Furthermore, several respondents and some of the literature regard comprehensive community‑based planning as culturally appropriate because it supports the type of consensus approach to decision making that is practiced in many Aboriginal communities.

3.3 Duplication

The evaluation found no evidence of duplication or overlap between the pilot project and planning efforts by AANDC Saskatchewan region or other jurisdictions. First Nations, AANDC and other government departments produce a vast array of plans, including capital plans, infrastructure plans, housing plans, education plans and health plans. But none of these take both a holistic approach to examining community needs and engage community members in articulating their priorities.

Flowing from the Policy on Transfer Payments, AANDC now conducts annual General Assessments that identify the strengths and emerging risks of Aboriginal communities receiving AANDC funding. [Note 4] While these assessments do try to take a comprehensive view, they are an AANDC requirement and do not flow from a community-driven process to articulate the needs, vision and priorities of community members.






4. Evaluation Findings - Design and Delivery

The evaluation looked for evidence that the project objectives and outcomes were clear to all stakeholders, the project design was appropriate, there were sufficient and appropriate resources and support to implement the project and there was adequate flexibility to allow the project to adapt to local contexts.

The evaluation found that while most pilot communities understood the project objectives, some had a different understanding that led to uncertainty about AANDC Saskatchewan region's intentions. The project design by the CEU was appropriate and provided an effective, interactive planning model that, with the assistance of a paid Plan Champion, facilitated broad-based community engagement. However, some communities expressed concern that they had little say in the project design, that planning was limited to the boundaries of the reserve, and that it focused too much on physical planning and not enough on broader social and economic issues. While there was good representation of women and men in the planning process, the project failed to incorporate attention to gender equity issues, such as the need to explore the differing needs of women and men in planning and gender impacts in implementation.

Evidence showed that the CEU fulfilled the terms of its contracts. It smoothly rolled out the seven-stage model by establishing strong relationships with communities. All stakeholders agreed that the Unit was flexible in adapting the project to the local context in different communities. There appear to have been sufficient resources for planning, but many communities expressed concern about the lack of support from AANDC Saskatchewan region to implement the community projects and priorities identified in the plans. Other delivery challenges included a lack of engagement by AANDC Saskatchewan staff, the distance between Saskatchewan pilot communities and CEU's offices in Halifax and insufficient time for the Phase 3 communities that joined the project in 2008-2009 to complete the planning and implementation process.

4.1 Design

Evidence from the document review, case studies and interviews showed that most of those involved with the pilot project held a shared view of project objectives, which was to involve communities in comprehensive, interactive planning that engages community members and sets a vision for the future. However, this understanding was not shared by all pilot communities. The lack of a logic model meant outcomes were not clearly articulated and this, along with the pilot nature of the project, created some uncertainty about AANDC Saskatchewan region's intentions. Some First Nations were unclear about how the community plans would be used by AANDC Saskatchewan region and were concerned that such planning would become a departmental requirement for future funding. However, others felt that community plans should be used by AANDC Saskatchewan region to determine funding requirements.

Project stakeholders found that the seven-stage planning model used by the CEU was effective in facilitating community engagement and producing high-quality community plans that take a comprehensive view of the community, including health and wellness, education, governance and communication, infrastructure and housing, economics, justice and safety, and the environment. Community members described the process as "user-friendly", "hands-on" and "interactive".

The case studies, interviews and document review showed that hiring a community member as a paid Plan Champion was key to fostering broad-based engagement in the planning process. In each of the five case studies, the evaluators found evidence that a wide cross-section of community members took part in planning, including women, men, elders, youth, community leadership, and band members who would not usually be involved in such an initiative. The Plan Champion organized community events, produced newsletters and other information materials, and, in some cases, paid individual visits to each home on the reserve to explain the planning process and get band members involved.

One gap in the project was the lack of attention to gender equality issues as required under AANDC's Gender-based Analysis Policy. AANDC Saskatchewan did not include gender equality considerations in its request for proposals. While there was good representation of women and men on the Planning Work Group and in the planning process, neither CEU's planning model nor the actual planning recognized the necessity to explore the differing needs of women and men and the differential gender impacts of the plans developed. 

For Phase 2 and Phase 3 communities, Plan Champions from earlier phases acted as mentors to the Plan Champions in the newer communities. The case studies and interviews showed this peer-to-peer learning process worked well in providing ideas and support in cases where the mentors had been effective Plan Champions in their own communities.

While the pilot project was set up to explore a community-based planning model, the evaluators found the project was not entirely community-driven. With the assistance of a Technical Advisory Group made up of representatives of First Nations and tribal councils, AANDC Saskatchewan region selected the CEU to carry out the initial phase of the pilot project. According to an AANDC Saskatchewan official, the CEU was selected because it had a documented model for comprehensive community-based planning and a track record in First Nation communities in the Atlantic region. While the Technical Advisory Group helped establish the criteria for selecting pilot communities to participate, AANDC Saskatchewan region selected the communities and invited them to participate.

Community members in some pilot communities said they would have liked to have had a greater say at the outset in how the project was designed. They were concerned that planning was limited to the physical boundaries of the reserve. While the CEU emphasized that communities had the opportunity to determine the extent of their plans, some community members said they would have liked to "think outside the box" to include traditional lands and new land acquisitions, consider members living off reserve and engage provincial and municipal governments in the planning process.

The CEU model tends to emphasize physical planning solutions – buildings and infrastructure as a tangible starting point for the process of community change. However, some community members and other stakeholders would have liked to have seen more emphasis on health and social issues, and the economic issues underpinning community challenges.

The communities invited to participate were diverse in size and geographically distributed across the province, but all had comparatively strong financial and human resource capacity (with the exception of one community). A more representative or stratified sample of Saskatchewan First Nation communities would have provided more meaningful information for learning, particularly in relation to the model's reliability for broader application.

According to its terms of reference, the Joint Steering Committee was to manage and oversee the pilot project, but in practice it acted as an information, knowledge exchange, and training forum rather than as a governance body. Most participants found it useful for sharing and learning from the experiences of others. It also served as a forum to brainstorm on future support to comprehensive community-based planning such as the outlines of a proposed community planning network. In Phase 1, the project organized Federal Days during Joint Steering Committee meetings, which raised awareness about comprehensive planning beyond AANDC Saskatchewan representatives and allowed communities to make contact and search out funding and partnerships with other departments to collaborate on specific projects. Representatives of Phase 1 pilot communities saw the Federal Days as the most useful aspect of the Steering Committee. But community representatives from later phases were disappointed that they did not have the opportunity to participate in the Federal Days because the event was eventually wound down due to lack of engagement from federal departments.

4.2 Delivery

Evidence from the case studies, interviews and project documentation show the CEU fulfilled the terms of its contract. CEU staff rolled out the seven-stage model smoothly. They established strong relationships with community members that facilitated community engagement. Case studies and interviews showed that the CEU was able to adapt to the local culture and differing conditions in pilot communities. Community members and leadership spoke highly of the approach taken by the Unit staff, describing them as "open", "approachable" and "respectful."

However, many pilot communities would have liked to have had the project delivered by Aboriginal consultants or consultants with a stronger foundation in local culture. Some First Nations respondents said the time and physical distance between Saskatchewan and the CEU offices in Halifax diminished opportunities for on-going interaction and created logistical challenges. There were times when community stakeholders had difficulty contacting Unit staff and the project faced challenges rescheduling workshops that had to be postponed due to unexpected events in communities, such as funerals.

There was broad agreement that the project provided sufficient human and financial resources to develop the plans. However, many community stakeholders were critical of the lack of AANDC Saskatchewan region funding and support for implementation of the community projects and priorities identified in the plans. Phase 3 communities felt they had insufficient time to complete the planning process and implementation process, and two of thesecommunities have yet to publish their plans. At the outset of the project, Phase 1 communities were to receive three years of support. However,these communities successfully lobbied AANDC for further support for plan implementation and by the end of the project, AANDC Saskatchewan region provided up to six years of support for Phase 1 Plan Champions while Phase 3 communities received two years or less.

While there was strong engagement of AANDC Saskatchewan staff at the beginning of the project, it declined over time and, for the most part, the Department's participation was limited to attending Steering Committee meetings. Staff turnover at AANDC Saskatchewan region affected momentum and engagement with three different managers being responsible for the pilot over its six years of operation. While many AANDC staff were involved in Phase 1 of the pilot project, with one exception, current funding services officers, AANDC Saskatchewan region's front-line staff, had little awareness of the planning process. Given that funding for the project was not A-based but pulled together from the AANDC Saskatchewan region's Professional and Institutional Development Program, the Capital Program and Community Economic Development Program and from Health Canada [Note 5], there was uncertainty about funding from year to year. In at least one case, this led to the loss of an experienced Plan Champion who left to take on a more secure position.






5. Evaluation Findings - Effectiveness

The evaluation examined the expected and unexpected results the project achieved and whether the benefits would be sustained beyond the end of the pilot project.

Overall, the evaluation found the project achieved its expected outcomes. It raised awareness of the value of comprehensive community-based planning in pilot communities, with tribal councils and AANDC Saskatchewan region, increased the capabilities of pilot communities to engage in the development and use of community plans, and led to improved planning.

The project built cohesion and trust as community members came together to document their history, articulate their values and develop a joint vision for the future. The majority of communities are using their plans and almost all have implemented at least one of the initiatives identified in the plan. Some have developed new community infrastructure such as gardens or walkways, initiated new partnerships or made changes to community governance. The planning process also created greater expectations that community political leaders will inform and consult with community members. Plan Champions built new skills in community facilitation and engagement and increased their self-confidence, which helped most to move on to other jobs.

However, there is little evidence that the comprehensive community-based plans are being used within AANDC Saskatchewan region. There appeared to be little awareness of the plans among current front-line funding services officers and there is no evidence that the plans have been used to inform AANDC Saskatchewan region funding priorities for the pilot communities.

The pilot project produced a number of unexpected results, including economic benefits and the revival of cultural traditions.

The evaluation found that some of the benefits from the planning process will be sustained. In some pilot communities, the plans have provided stability of vision through several elections for Chief and Council. However, progress on initiatives identified in the plans, which often fall outside or across responsibilities in the band administration, will likely slow down without a Plan Champion within the community to move them forward. Furthermore, the technical expertise for the pilot project came from outside the province and the project has not been successful to institutionalize the capacity for comprehensive community-based planning in Saskatchewan.

5.1 Achievement of Outcomes

The evaluation found that the pilot project achieved significant results, including all of its expected outcomes. Evidence from the case studies, interviews and documentation showed that broad-based community engagement in pilot communities increased awareness of the importance of planning among community members, staff and elected leaders. Through the planning process, communities were able to come together and document their history, articulate their values and develop a common vision for the first time.

In many communities, the planning process increased community cohesion and built trust. For example, in one community, discussing strengths and weaknesses, identifying the root causes of community challenges, and developing the community vision helped it to deal openly with long-standing divisions and work toward unity.

Evidence shows that the project also raised awareness of the value of comprehensive community-based planning among the staff of AANDC Saskatchewan region, Health Canada and tribal councils who participated in the process. It also created interest among other First Nations in the province, some of which are eager to undertake their own planning process. Many stakeholders, inside and outside of AANDC, praised AANDC Saskatchewan region for having the foresight to support a pilot project that incorporates internationally established good practice in community development.

The project increased the capabilities of community members to engage in and use comprehensive community-based plans. Community members told the evaluators that involvement in both the planning process and implementation created a sense of pride and belonging that helped band members feel better about themselves and their communities. Communities are proud of their plans, which are professionally printed, easy to read and illustrated with maps and photos. While each plan is unique, all follow a standard template developed by the CEU that includes three parts: context on where the community is now, vision about where the community is going, and action areas for shaping the future.

Three communities failed either to publish their plans or to move forward with implementation. In two communities, this inaction was largely because the political leadership did not agree with the proposed direction outlined in the plans. The third community was slow to get started and felt it was rushed through the planning process. Despite this, community members told evaluators that bringing the community together through the planning process still provided benefits in connecting the community and demonstrating the role community members could play in planning.

Plan Champions built new skills in community facilitation and engagement and increased their self-confidence. Many of the Plan Champions were women and most were able to move on to other jobs following their work with the project. One told evaluators that she is contemplating running for Band Council, something she would have never considered before her work on the community plan.

The evaluation found that almost all of the pilot communities have used their plans to some extent and many communities have made progress implementing them. In five of the eleven communities, the CEU (in four cases with the help of Dalhousie architecture students), worked with community members to design and build a new structure in the community. These structures included a pow-wow arbour, a green-shed, an environmental pavilion, an outdoor classroom and a community market. The projects provided communities experience in using local resources, training people, increasing ownership, being creative about funding support and partnerships, such as donated or recycled materials and volunteer labour to implement a project.

Through their own efforts some communities have gone much further. For example, one community developed a walking path from the village to the school that is lit with streetlights so that children do not have to walk to school along a busy highway. Furthermore, it organized a tree planting day, and obtained funding to build a veteran's memorial and buy new equipment for its senior's centre.

According to the interviews and case studies, comprehensive community-based planning has allowed some communities to challenge the current federal investment model whereby First Nations respond to centrally designed government programs. Some communities are using their plans to communicate their own priorities and tell potential funders where investments need to be made in their communities. In this way, the pilot project helped some communities to develop new partnerships and leverage funding. For example, in one community, the band used the community plan to support a successful proposal to the Saskatchewan Government for funding to construct a $1.6 million Centre of Excellence for Business Development that will provide business support and skills training. Another community developed partnerships with an international non-profit organization and the Saskatchewan Department of Agriculture to set up an organic community garden.

The planning process allowed communities to openly discuss governance issues such as lack of transparency and nepotism, and several communities have made changes to governance as a result. Two communities have developed and ratified new election acts, and one of those has also developed a new housing policy and financial management act. In another community, the band administration has tried to improve its communication with members through newsletters and Internet and is in the process of developing a community radio station. The same community is also working to update its bylaws.

In many communities, the evaluators were told that the planning process has increased expectations that the leadership will inform and consult with community members. One tribal council provided support for the political leadership of a pilot community to attend nation building training to assist the Chief and Council in learning how to respond to these expectations.

While most pilot communities are using their plans, there has been little use of the plans by AANDC Saskatchewan region. The one exception was in the development of the General Assessments that are now required for all communities receiving AANDC funding. When the evaluators spoke to current funding services officers who deal directly with the pilot project communities, they learned that the majority had never seen or used the community plans. Furthermore, there was no evidence that AANDC Saskatchewan region is using the community plans to inform departmental funding priorities.

The approach in AANDC Saskatchewan region contrasts with that in AANDC's British Columbia (BC) region, which provides support both to communities in developing comprehensive community plans and to AANDC staff in responding to those plans. In BC region's Strategic Planning and Communications Directorate, one and a half staff positions are devoted to providing guidance, support and advice related to comprehensive community planning. Part of the work of the staff members includes pulling together teams from different AANDC BC programs to examine how to support the priorities identified in community plans. The Directorate also conducts regular training sessions on comprehensive community planning for funding services officers.

A 2010 evaluation of Comprehensive Community Planning in the BC region (KTA Inc. and Naut'sa mawt Resources Group 2010) showed that it had accomplished three goals – increased awareness, increased capacity, and strengthened relationships between First Nations and AANDC BC. The Saskatchewan pilot project also increased awareness and capacity. However, evaluators only found evidence of a strengthened relationship between AANDC Saskatchewan region and a First Nation in one pilot community. In that case, the funding services officer was directly involved in the planning process and the community leadership said that the process had facilitated a more open and positive relationship with the Department.

Tribal councils are tasked with providing advisory and support services to member bands and council representatives. They were included in the pilot project as one means of building awareness and capacity for comprehensive community-based planning within the province. While some tribal council representatives were actively engaged in the planning process, others felt there was insufficient funding to allow them to participate in more than the occasional Steering Committee meeting. One tribal council representative is using elements of the comprehensive community-based planning process in his work with communities. However, given the high demand for planning specialists in the province, others were sceptical about the ability of tribal councils to recruit planning specialists with the technical capacity to support comprehensive community planning efforts.

5.2 Unexpected Results

The pilot project produced a number of unexpected results – both positive and negative. By facilitating community participation, the process allowed community members who would not normally participate in planning to get involved. Community members got excited about potential ideas and, in some cases, moved forward on their own to put those in place. For example, seniors in one community came together to set up a restaurant and gathering place in an unused space in the community mall.

The planning process produced a number of economic and cultural benefits. Through the community garden project mentioned above, one community was able to get eight people off social assistance for two years and provide them with training in organic gardening. Another community was inspired to revitalize its annual pow-wow after a 20-year lapse. That pow-wow is now an established annual event on the pow-wow trail in Western Canada.

The community plans have won two awards of excellence from the Canadian Institute of Planners, including an award for rural and small town planning in 2008 and another for social planning in 2011. The CEU has showcased the plans internationally and an educator from one pilot project community shared his community's plan with indigenous education officials in Australia.

An unexpected negative result identified by the case studies and interviews was the creation of expectations for quick action on the projects identified in the plans – expectations that were not fulfilled. This outcome was most prevalent in communities that have yet to follow through with plan implementation. While AANDC Saskatchewan region was clear at the outset that the pilot project would not include funding for community projects, the very act of funding the planning process created expectations that the Department would respond to the priorities identified in the plans. Furthermore, some community leaders questioned the feasibility of some of the projects identified in the plans, given local circumstances and the limited availability of funding. Despite the creation of expectations, some stakeholders in this project regard it as an inevitable, and not necessarily negative, outcome of a process that invites community members to define their vision of a better community.

5.3 Sustainability

Evidence from the case studies, interviews and document review shows that some of the benefits of the pilot project will be sustained. Most communities are still using their plan and some have integrated it into the work of their band administration. One community prominently displays the value statements from their plan in the band office, community school and adult education centre. For at least four of the pilot communities, the community plan has provided a stable vision that has continued to be supported through several elections.

The evaluators found that three of the communities that made the most progress in implementing their plans were among those in Phase 1 of the pilot project. However, it is difficult to say whether this is due to the sustained support these communities received over six years or whether their uptake of the plans grew out of their strong human and financial resource capacity that was there before the pilot project.

The broader sustainability of the planning process is, however, in some doubt because AANDC Saskatchewan region has yet to follow through in using the plans to inform programming priorities. The CEU process envisions that First Nations will, over time, begin to change their governance structures to align with the priorities identified in their community plans. However, in the meantime, band administrations remain largely partitioned in health, education and other compartments to align with federal government funding programs. Some stakeholders emphasized how profoundly different the comprehensive, community-based model of planning is from typical strategic planning exercises that focus only on a specific sector. They also underlined that moving toward comprehensive, community-based planning is a major change in approach and thinking for federal departments used to the narrower strategic planning model.

The evidence suggests that there would be a loss of momentum on plan priorities in communities once the funding for the Plan Champion ends. Since the action areas identified in the plans often fall outside or across responsibilities in the band administration, community stakeholders believe a Plan Champion is vital to drafting the funding applications or to organizing the community members needed to move those priorities forward.

The design-build projects that formed part of the CEU model to kick-start plan implementation provided positive learning experiences that created pride and a sense of accomplishment. However, in some communities, the structures have yet to see much use by community members and in one community, the structure had to be dismantled because it was vandalized. The design‑build projects appear to have been selected quickly by communities without comprehensive needs assessments. Overall, projects such as the community walkways that the communities themselves initiated and developed appear to have greater sustained use.

CEU received funding as part of the final phase of the project to work toward establishing a First Nations Planning Network among the participating communities and tribal councils. However, there was no indication that the network would continue once project funding ended since neither the communities nor the tribal councils possess the technical expertise or resources to sustain such a network.

Over the course of the project, the CEU submitted two proposals to AANDC Saskatchewan region for funding to establish a diploma program in First Nations community planning, first with the First Nations University in 2006 and then with the First Nations University and the University of Saskatchewan in 2010. Such a program was to provide on-going access to local planning expertise in Saskatchewan. While AANDC Saskatchewan region set aside funding in 2010-2011 to support such an initiative, it has yet to become a reality.






6. Evaluation Findings - Economy and Efficiency

The evaluation sought to determine whether the most appropriate and efficient means were used to develop the community plans and whether alternative models could improve the process or reduce the cost.

Overall, the planning process was funded at an average cost of about $450,000 per community. The fees paid to the CEU appear reasonable for the services delivered when compared to rates charged by planning consultants. However, the travel between the CEU office in Halifax and Saskatchewan added to the project costs, as did expenses associated with Joint Steering Committee meetings.

Building technical capacity to support First Nations comprehensive community-based planning in Saskatchewan would both increase efficiency and reduce costs over the long-term. Alternative delivery models and approaches for comprehensive community-based planning and working with geographic clusters of communities could result in cost savings.

6.1 Economy

AANDC Saskatchewan region and Health Canada provided significant resources for the pilot project – on average about $450,000 per community when the $5 million cost is divided by the 11 participating communities. With this funding, the project was able to hire a Plan Champion in each community, take the time and effort needed to facilitate broad-based community involvement and in many communities support plan implementation.

The funding for the pilot project was roughly divided between support to First Nations (50.7 percent to communities and tribal councils) and the contract with the CEU (44.7 percent), which is in line with CEU's view that the level of effort and activity was equally distributed between CEU and First Nations. However, some First Nations expressed concern about the proportion of funding that went to the CEU.

An examination of the fees charged by the Unit shows that they appear reasonable for the services delivered as they were lower than rates normally charged by professional planning consultants. However, the distance between Halifax and Saskatchewan added to the cost of the project. While the total travel costs are not broken down in CEU contracts, budget documents show that airfare alone comprised nearly $150,000, or six percent of Dalhousie's costs.

The Joint Steering Committee meetings were also costly, bringing together between 15 and 50 participants from around the province for one- or two-day meetings several times a year. The meetings appear to have cost about $260,000 and comprised about five percent of the project budget. The evaluators found the benefits of these meetings difficult to measure. However, most participants found it useful for sharing and learning from the experiences of others. It also served as a forum to brainstorm on future support to comprehensive community-based planning such as the outlines of a proposed community planning network. In Phase 1, the project organized Federal Days during Joint Steering Committee meetings that raised awareness about comprehensive planning beyond AANDC Saskatchewan representatives, and allowed communities to make contact and search out funding and partnerships with other departments to collaborate on specific projects. Representatives of Phase 1 pilot communities saw the Federal Days as the most useful aspect of the Steering Committee.

One of the lessons from international development is that there are no shortcuts to effective community engagement and it can, therefore, be costly to undertake. (Mansuri and Rao 2004) However, when done well, the benefits in program sustainability outweigh the costs.

6.2 Efficiency

Representatives of communities, tribal councils and AANDC all agreed that there is a need to build technical capacity for First Nations comprehensive community-based planning within Saskatchewan. Access to local participatory planning expertise would reduce the travel costs and increase ease of access. Developing that technical capacity will require a long-term funding commitment. Many stakeholders emphasized the need for such a commitment to allow comprehensive community planning to move beyond pilots to being accessible to all First Nations in the province.

Some of those interviewed also suggested that AANDC Saskatchewan needs to explore alternative models or approaches beyond the CEU model for delivery of comprehensive community-based planning. Some stakeholders suggested that the proposed diploma and planning network initiatives may contribute to mitigating the cost of producing community plans. Another idea in this respect is to examine AANDC BC's approach, whereby, the Department provides funding, tools and support while communities wishing to undertake comprehensive community planning identify the model and technical expertise required to fit their needs.

When questioned about the pilot project's delivery over three phases, most community stakeholders thought it would have been preferable to proceed with all 11 communities from the start of the project. This would have given all communities sufficient time to complete the planning and implementation process. Some suggested it would have been more efficient to work with clusters of communities, for example, several communities within a tribal council rather than separate communities scattered around the province. Such an approach would allow for greater information sharing and support between communities. Further, community members suggested that involving other levels of government in the planning process could assist in developing partnerships.






7. Evaluation Findings - Learning and Lessons

The evaluation looked for evidence that the pilot project had mechanisms in place to capture lessons and facilitate learning and examined lessons that could contribute to improved community planning.

The evaluation found limited evidence that mechanisms were in place to capture lessons and use them to facilitate reflection and on-going learning at the community level. The CEU worked with pilot communities to develop indicators to track progress in meeting community goals. However, neither it nor AANDC Saskatchewan region demonstrated to communities how monitoring data could be collected and used to inform on-going planning and program improvement. The evaluation identified a number of lessons and success factors that are outlined below.

7.1 Learning

The evaluation found limited evidence that mechanisms were in place to capture lessons and use them to facilitate on-going learning at the community level. The lack of a project logic model and performance measurement framework meant that the CEU did not develop indicators against which it could collect data, track progress and make adjustments to the planning process as the project progressed. AANDC Saskatchewan region only required activity-based reporting and this meant that data such as the level of community participation and the changes resulting from the planning process were not tracked systematically, reported or examined to see what could be learned.

As part of its work with pilot communities, the CEU did discuss the need for on-going monitoring of plan implementation to track progress in meeting community goals. It also worked with communities to develop indicators of success that were to be tracked annually. However, these monitoring sections often lacked causal linkages to the actual projects that communities undertook and there is no indication that communities were using the indicators to monitor the implementation of their plans. Both AANDC Saskatchewan and CEU missed an opportunity to demonstrate to communities how monitoring could be used for on-going learning and improvement.

Facilitating learning requires more than the collection of data; it requires a reflective practice that sets time aside to reflect on progress, identify lessons and apply them, usually on an annual basis. Although CEU uses reflective practice in its own assessment of the project (CEU 2010), such practice does not appear to have been integrated into its work during the project at the community level.

Recent literature on comprehensive community initiatives that deal with complex and interconnected problems, such as those facing First Nations communities, point to the contributions real-time monitoring and evaluation can bring to those processes. (Gardner 2011) According to the literature, there is a need to build evaluation and learning into such processes as an enabler of innovation, continuous service improvement and community learning, rather than only as a means to meet narrow accountability requirements. As comprehensive community based planning progresses, it will be important to work with communities to develop meaningful and culturally appropriate monitoring and evaluation tools that allow communities to assess their own progress and make improvements.

7.2 Lessons

The case studies, interviews, and project documentation provided a number of lessons related to comprehensive community-based planning in Saskatchewan that are also supported by the literature review.

  • Planning must be community-based

It is only by engaging community members in meaningful dialogue and decision making that planning can be successful in moving a community forward. Engagement creates awareness, builds capacity, fosters cohesion, and raises expectations.

  • Planning must be comprehensive

Planning must view the community as a whole, taking into consideration the interconnections between health, social, economic, land use, infrastructure and environmental issues. This implies moving from a program-based approach to a community development approach.

  • Both AANDC and communities must make changes to develop a new relationship

Both AANDC and First Nations communities must make changes if comprehensive community‑based planning is to achieve its potential. A plan becomes a tool for crafting a new relationship when the Department responds to community priorities. Such a shift from directive to responsive programming requires a fundamental change in the way the Department conducts its business.

  • Comprehensive community-based planning takes time and requires long-term support

This type of planning is part of a long-term process of community development. It requires sustained support to build awareness, capacity and community engagement. Not all communities will proceed at the same pace, as the pilot initiative has shown.

  • Comprehensive community-based planning is costly, but potentially cost-effective

While this type of planning is costly to do well, it is potentially cost‑effective since it can help direct spending where it will be most effective in responding to community needs.

7.3 Success Factors

Over the course of their research, the evaluators identified a number of factors that contributed to the results achieved by the comprehensive community-based planning project in Saskatchewan.

  • A paid Plan Champion

Hiring a community member to facilitate the plan development is key to providing momentum, engaging community members and moving the process forward.

  • Planning expertise combined with good process and skilled facilitation

Supporting the development of comprehensive community-based plans requires a strong grounding in community planning along with a good process and excellent facilitation skills to engage community members, and allow them to openly deal with difficult issues and envision a way forward.

  • Starting from strengths

An asset-based approach that starts from community strengths rather than from problems helps people to open up and reduces negativity – all of which puts the community in a better position to move forward.

  • Engaging elders and youth

Planning allows elders an opportunity to contribute and helps ground the process in the culture. Engaging youth is crucial, but finding ways to do so requires innovation.

  • Ensuring political leadership is on-side

Chief and Council have to understand from the outset the implications of comprehensive community-based planning and be involved throughout, without controlling the process.

  • Providing support for leaders

If comprehensive community-based planning is to be successful, there may be a need to assist community political leaders in understanding the changes required in community governance in order to move forward with the priorities outlined in the plans.

  • Building relationships and partnerships

Building new relationships and partnerships beyond AANDC and the federal government with the private sector, non-governmental organizations, provincial and municipal governments is key to meeting community needs.

  • Mentoring

Peer-to-peer learning can be an effective and potentially cost-effective way to share expertise between communities.






8. Conclusions and Recommendations

8.1 Conclusions

The evaluation set out to provide evidence-based conclusions regarding the relevance and performance of the Comprehensive Community-based Planning pilot project in Saskatchewan between 2005-2006 and 2010-2011. Given that most of the communities selected to participate in the pilot project had good financial and human resource capacity, the findings on the development of this planning model may be limited to such communities.

The evaluation findings support the following conclusions.

Relevance

Findings from the evaluation show that comprehensive community-based planning is consistent with federal roles and responsibilities as it aligns with AANDC and government priorities. All stakeholders interviewed, especially First Nations involved in the pilot project, believe there is a need for comprehensive planning that facilitates community engagement and ownership. Some of the literature and some respondents see comprehensive planning contributing to improved governance and self-reliance in First Nations communities. All lines of evidence point to the need for AANDC to move from "silo," program-based planning to a model that is both comprehensive (holistically examining community needs and priorities) and community-based (allowing for broad-based engagement of community members in the planning process). The evaluation found no evidence of duplication or overlap between the pilot project and planning efforts by AANDC other jurisdictions.

Performance

Design and Delivery
The evaluation found that the project design by the CEU was appropriate and provided an effective, interactive planning model that, along with the assistance of a paid Plan Champion, facilitated broad-based community engagement. However, some communities expressed concern that they did not have a say in the project design, and that planning was limited to the physical boundaries of the reserve. Further, AANDC Saskatchewan region failed to ensure that the project incorporated attention to gender equity issues, as required by AANDC's Gender-based Analysis Policy.

The CEU successfully delivered the pilot project and adapted it to the local context in different communities. There appears to have been sufficient resources for planning and implementation, but many communities were concerned about the lack of financial support for the projects and priorities identified in their plans.

Effectiveness
The evaluation found that the project achieved its expected outcomes. It raised awareness of the value of comprehensive community-based planning in pilot communities, with tribal councils and AANDC Saskatchewan region, increased the capabilities of pilot communities to engage in the development and use of community plans, and led to improved planning.

The project built cohesion and trust as community members came together to document their history, articulate their values and develop a joint vision for the future. The majority of communities are using their plans and almost all have implemented at least one of their planned initiatives. Some have developed new community infrastructure such as gardens or walkways, begun new partnerships or made changes to community governance. Plan Champions built new skills in community facilitation and engagement and increased their self-confidence. However, there is little evidence that AANDC Saskatchewan region is using the comprehensive community‑based plans.

The evaluation found that some of the benefits from the planning process will be sustained. In several pilot communities, the plans have provided stability of vision through several elections. While the building of structures through the design-build process was a positive experience for communities, some structures have seen limited use. The technical expertise for the pilot project came from outside the province, and the project has not been successful to institutionalize the capacity for comprehensive community-based planning in Saskatchewan.

Efficiency and Economy
Overall, the planning process was funded at an average cost of about $450,000 per community. The fees paid to the CEU appeared reasonable for the services delivered when compared to the rates charged by professional planning consultants. However, travel between CEU's offices in Halifax and Saskatchewan added to project costs, as did the expenses associated with Joint Steering Committee meetings.

Building technical capacity to support First Nations comprehensive community-based planning in Saskatchewan would increase efficiency and reduce costs over the long-term. Working with geographic clusters of communities might also deliver cost-efficiencies.

Learning and Lessons
The evaluation found limited evidence that mechanisms were in place to capture lessons and use them to facilitate reflection and on-going learning at the community level. The CEU worked with pilot communities to develop indicators to track progress in meeting community goals, it did not demonstrate effective monitoring practice. Research for the evaluation identified a number of general lessons and success factors related to the experience with the Saskatchewan pilot project.

8.2 Recommendations

It is recommended that AANDC, to support comprehensive community-based planning:

  1. Consider ways to institutionalize long-term national support for such planning in First Nation communities.
  2. Review its approach to:

    1. better integrate First Nations input on the design and approach used to develop community-based plans;
    2. involve other levels of government; and

    3. integrate gender-based analysis.

  3. Consider means to increase capacity (inside and outside the Department) to support comprehensive community-based planning.

  4. Commit to using comprehensive community-based planning to inform program and funding priorities.





9. References

Cities & Environment Unit. (2010). Learning from Saskatchewan: Charting a course for community planning in Canada. Halifax: Dalhousie University.

Cities and Environment Unit and Wagmatcook First Nation. (2003) First Nations community planning model, 2nd edition. Halifax: Dalhousie University.

Cook, J. (2008). Building on traditions of the past: The rise and resurgence of First Nations CCP. Plan Canada, 48, 13-16.

Copet, Wayne (2003). An approach to community planning in Aboriginal settlements. Canadian Journal of Native Studies, 25(2) 37-50.

Cornell, S. and Kalt, J. (1998) "Sovereignty and Nation-Building: The Development Challenge in Indian Country Today" American Indian Culture and Research Journal. Vol. 22 (3) 187-214.

Gardner, B, (2011) Comprehensive Community Initiatives: Promising Directions for "Wicked" Problems. Horizons. Policy Research Initiative. Retrieved March 2, 2011.

Harivel, C. and Anderson, C. (2008). First Nations comprehensive community planning – A good investment for Canada. Plan Canada, 48, 29-31.

KTA Inc. and Naut'sa mawt Resources Group. (2010). Indian and Northern Affairs Canada: Evaluation of aspects of comprehensive community planning in the BC region.

Lane, Marcus (2001). Indigenous land and community security: A radical planning agenda. Land Tenure Center, Working Paper No. 25. Madison: University of Wisconsin.

Lee, C. (2002). Environmental justice: Building a unified vision of health and the environment. Environmental Health Perspectives, 110 (April), 141-144.

Mansuri, G. and Rao, V. (2004) Community-based and -driven development: A critical review. World Bank Research Observer, 19(1) 1-39.

Mathie, Alison & Gord Cunningham. (2002), From Clients to Citizens: Asset-Based Community Development as a Strategy for Community-Driven Development. Occasional Paper, Coady International Institute, St. Francis Xavier University.

Public Works and Government Services Canada (2004). Sharing the story – comprehensive community planning – experiences in First Nations, Inuit and northern communities.

Thurman, P. et al. (2003) Community readiness: The journey to community healing. Journal of Psychoactive Drugs, 35(1), 27-31.

Wesley-Esquimaux, C and Calliou, B. (2010) Best practices in Aboriginal community development: A literature review and wise practices approach. The Banff Centre.

Wolfe, J. (1989) Approaches to planning in native Canadian communities: A review and commentary on settlement problems and the effectiveness of planning practice. Plan Canada, 29 (2), 63-79.

World Bank. (2009). IDA at work: Community-driven development: Delivering the results people need  . Retrieved March 27, 2011.






Appendix A: Methodology Report

EVALUATION OF THE COMPREHENSIVE COMMUNITY-BASED PLANNING PILOT PROJECT IN SASKATCHEWAN

Methodology Report

February 20, 2011 version

Submitted by:
Stiles Associates Inc
305 Riverdale Avenue,
Ottawa, ON
K1S 1R4
613-730-0287

1. Introduction

Indian and Northern Affairs Canada (INAC) is conducting an evaluation of the Comprehensive Community-Based Planning (CCBP) Pilot Project in Saskatchewan. This evaluation will provide evidence-based conclusions regarding the project's relevance and performance.

The purpose of the evaluation and use of the data it generates relates to a request from INAC Saskatchewan region for an independent evaluation to assess the results achieved by the pilot project, capture lessons and provide recommendations to inform future support for CCBP.

2. Pilot Project Description

Scope of Community Planning

Community planning is a process through which a community establishes a direction for the future and determines specific actions to realize that direction. [Note 6] This is accomplished in a way that is widely understood, accepted and appropriated by many members of the community. The resulting plan allows the community to make informed decisions about where to seek funds, how to spend limited resources, how to react to requests for action, how to protect the environment and how to provide new opportunities for residents; all factors that create a sustainable community.

The Need for Comprehensive Community-based Planning (CCBP)

Planning, as a tool and in the context of First Nation (FN) communities is about the future and relies on information from both the past and the present. This information is to help understand current circumstances and to collectively determine the need for change. FN communities face many challenges, including social, economic, environmental, and governance issues. One of the ways to meet these challenges is through CCBP.

CCBP is a community planning tool that mobilizes communities through group processes (e.g. workshops) to identify strengths and issues affecting the community. CCBP helps the community to establish a vision, long-term goals, priorities and an action plan. CCBP is accomplished in a way that is widely understood, accepted and appropriated by many members in the particular community. Ideally, the process is community-driven and locally owned and controlled. Implementation and accountability for results rest with the community.

CCBP is expected to encourage communities to examine the consequence of doing nothing or of taking deliberate steps in a particular direction, thus helping to avoid crises before they occur. CCBP is recognized as an instrument that has the potential to help FN communities set the stage to guide individual decisions, plan long-term, focus on how and where the community should grow while serving as a means of transparency and accountability. The success of community plans rely on changing attitudes and approaches at the local and regional levels with respect to FN community-management.

2.1 Project history

The CCBP pilot project in Saskatchewan represents a commitment to rethinking FN community development and responding to urgent issues and needs of FN communities. It is intended to overcome the following barriers:

  • Lack of local examples illustrating that community-based planning makes a difference;

  • Lack of local expertise to initiate, guide and sustain the planning effort;

  • Lack of resources and technological capacity at the local level; and

  • Lack of awareness of what planning is.

INAC, through its Sustainable Development Strategy, established a commitment to support the introduction of CCBP within FN communities. This comprehensive approach is intended to facilitate a shift from a program-based service delivery model where, in many instances, external priorities have been placed upon FN communities, to a process more consistent with FN communities' goals and aspirations as articulated through their membership.

In 2005-06, the Saskatchewan region contracted with the Cities and Environment Unit (CEU), a research-based community planning action group of the Faculty of Architecture and Planning, Dalhousie University, Halifax, Nova Scotia, to engage four FN communities in CCBP. The selected communities were the Flying Dust FN, Gordon FN, Kahkewistahaw FN and Shoal Lake Cree Nation, with support from their respectiveTribal Councils. These four communities constituted Phase 1 of a pilot project.

In October 2005, a CCBP Joint Steering Committee (JSC) was established as a decision-making body to oversee and coordinate CCBP activities within the selected pilot FN communities; it included representatives from the selected four pilot communities, Tribal Councils, Health Canada, INAC and the Federation of Saskatchewan Indian Nations (FSIN).

In 2007-08 Dalhousie won a tendered contract to continue and four more communities were added. These Phase 2 communities were Cowessess FN, Kinistin Saulteaux Nation, Muskoday FN and Pasqua FN.

In 2008-09, Phase 3 communities – Big River First Nation, Lac La Ronge Indian Band and Standing Buffalo Dakota Nation entered the pre-planning stage of the process.

Through consultations with FN and Tribal Councils, community development and planning was established as a long-term strategic priority for INAC Saskatchewan region. The pilot project was one of the key activities supporting this priority. At present there are 10 FN communities involved in CCBP. These are:

PHASE 1
COMMUNITIES
PHASE 2
COMMUNITIES
PHASE 3
COMMUNITIES
1. Flying Dust FN 1. Cowessess FN 1. Big River FN
2. George Gordon FN 2. Kinistin Saulteaux Nation 2. Lac La Ronge Indian Band
3. Shoal Lake Cree Nation 3. Muskoday FN* 3. Standing Buffalo Dakota Nation
4. Kahkewistahaw FN 4. Pasqua FN

*Muskoday FN, a Phase 2 community, withdrew from the pilot project in April 2010.

All Phase 1 and 2 communities have published their plans and are in various stages of implementing or kick-starting other projects identified in the plans. Examples of community kick-start and action projects range from walking trails to market gardens to a coffee shop, construction of a pow wow arbour and an environmental research pavilion.So far, one of the Phase 3 communities has published its community plan.

2.2 Project Profile

Principles for the selection of First Nations

In April 2005, the inaugural meeting of the Community Development and Planning Technical Advisory Group (TAG) was held in Saskatoon, Saskatchewan. TAG , which was established to provide recommendations relating to community planning and development within the region, reviewed opportunities for collaboration between INAC, FN and Tribal Councils.

TAG was then tasked with the responsibility of establishing principles to be considered in selecting FN communities to pilot comprehensive planning theory. TAG established the following six principles:

  1. Pilot communities, through their council, must confirm a willingness and desire to advance CCBP.

  2. Communities from the North and South should be represented.

  3. A pilot community must possess demonstrated financial management capacity and be committed to the principles of accountability.

  4. Technical expertise from Tribal Council advisory staff and planning consultants will be essential in the initial pilots to build comprehensive planning capacity.

  5. Initial pilot communities must be affiliated with a Tribal Council to ensure linkages are developed and planning is supported in the long-term.

  6. Pilot communities must have demonstrated a commitment to community engagement as part of their day-to-day operations.

Based on the foregoing, the four FN communities, previously mentioned, were recommended for the 2005-2006 fiscal year.

2.2.1 Approach

The CCBP approach centers on a philosophy that encourages and employs extensive community involvement throughout the development and the implementation of a community plan. This approach ensures that community members and band administration are fully aware of the planning process, have avenues for meaningful input and participate in implementation of projects. The three fundamental conditions of CCBP are that:

  1. The Plan comes from the community.

  2. The Plan is appropriated by the community.

  3. The Plan inspires and motivates the community.

In addition, the CCBP approach is based on the following three major points:

  1. A team-centred approach: This approach, leading to the creation of a local planning work group builds on individual strengths and expands local capacity while maintaining the project's collective goal and mandate.

  2. Comprehensive view: In this fundamental approach, the CEU, Dalhousie University, assists communities in the development of holistic community plans that considers land use and physical economic development as well as incorporates social development and environment sustainability.

  3. An action-oriented plan: This approach translates visions, ideas and priorities into tangible projects and includes concrete and specific actions needed to achieve them.
2.2.2. Planning Framework

The CCBP model, as developed by the CEU (Dalhousie University), establishes a conceptual framework and approach to change centred on a comprehensive and community-based philosophy. The model outlines a seven-stage process with distinct products at the end of each stage that together constitute the Community Plan. The approach is intended to develop relevant community plans with First Nations that help them drive immediate action based on a long-term vision.

  1. Stage 1: Gathering Background Information - Collect basic facts and perceptions about the community so that aspects that stand out as high and low points can be identified.

  2. Stage 2: Identifying Strengths & Issues - Concentrate on recording and understanding problems that need to be dealt with and opportunities that can be built on.

  3. Stage 3: Researching Root Causes. - Explore strengths and issues to reveal the root causes and the consequences of no action.

  4. Stage 4: Establishing a Vision - Establish a long-term, ambitious and appropriate direction for the community.

  5. Stage 5: Building a Framework - Translate the vision, issues and values into a blueprint for concerted action in terms of policies, administration, priority action areas and physical improvements.

  6. Stage 6: Developing an Implementation Strategy - Develop an implementation strategy and determine priority projects. Organize the necessary resources to realize the projects.

  7. Stage 7: Monitoring the Plan & Projects - Evaluate the impact of projects individually, the effects of planning as a whole, and revise the Plan on an on-going basis.

2.3 Project Objectives and Expected Outcomes

The overall objective of CCBP pilot project is to make community-based planning (developing comprehensive local, long-term, and action-oriented plans) an on-going routine for FN.

The CCBP project is intended to engage community members in each stage of the planning process. It does so by increasing awareness among the various stakeholders including Tribal Councils, communities, and the federal government. It also seeks to increase the community's capacity to engage in planning. It is expected that this will position communities to take on challenges, explore opportunities and succeed in meeting their goals.

The First Nations Community Planning Model, published by the CEU and Wagmatcook First Nation (Nova Scotia), defines products, outcomes and expectations for planning that were followed in the CCBP project. However, since no formal logic model or performance measurement framework was developed for this pilot project, a logic model developed for the purposes of the evaluation appears in Appendix C.

2.4 Project Governance (Management), Key Stakeholders and Beneficiaries

Project Management and Key Stakeholders

Joint Steering Committee

  • Manages and oversees the pilot project

  • Comprised of representatives of First Nation communities (Plan Champions), Tribal Councils, INAC, Health Canada, and the Cities and Environment Unit CEU (Dalhousie University)

  • Supports and facilitates the development of strategies, work plans, and resources

  • Extends project work to other FN communities

INAC, Saskatchewan Region:

  • Federal lead in providing resources (technical and other), to support the CCBP process

  • Chair of the Joint Steering Committee (JSC), responsible for all meeting logistics and secretarial support including arranging meetings and maintaining and facilitating contact among members

  • Facilitates and supports partnership development initiatives between academic, technical and other organizations (i.e. Dalhousie and First Nations University)

  • Pursues linkages to inter/intra departmental federal government programming

  • Coordinates initiatives supporting professional and capacity development

  • Monitors CCBP Terms and Conditions

  • Liaison between INAC Regional and Headquarters CCBP National Working Group

  • Based on recommendations from the JSC, maintains responsibility for resource management and budget allocation decisions

  • Advocates for change within the federal system to ensure CCBP is supported and appropriately resourced on a long-term basis

  • Coordinates and administers funding arrangements with the pilot communities and affiliated Tribal Councils

The Yorkton Tribal Council:

  • Provides INAC an updated assessment of the current planning and technical advisory services provided to member FN engaged in the CCBP initiative

  • Ensures all financial transactions relating to CCBP support and capacity development are disclosed in a separate schedule of Revenues and Expenditures

  • Administers contract with CEU

  • Supports the community engagement, planning and reporting activities of FN

  • Serves as the liaison between the respective Tribal Council executive committees and the regional JSC

Cities and Environment Unit (Dalhousie University):

  • Provides training in community-based planning for community Plan Champions / Tribal Council employees through scheduled workshops in pilot communities and training sessions

  • Provides training in the field through work in the pilot communities (learning by doing)

  • Builds capacity and awareness in each community for the Plan Champion, associated Tribal Council, band administration, chief and council, and Planning Work Group

  • Builds awareness of community-based planning at all levels: individual, community, Tribal Council and Federal Government through meetings, project newsletters and project website

  • Identifies technological needs for planning at the local level

  • Provides professional planning expertise, advice and guidance to the JSC and member FN communities

  • Works alongside FN communities as active participants in plan development as well as implementation.

Federation of Saskatchewan Indian Nations (FSIN):

  • In partnership with INAC Saskatchewan region, ensures coordination with existing and future governance capacity building processes (post‑FMM; Treaty Governance)

  • Provides a forum through the Saskatchewan Legislative Assembly to create awareness and provide progress reports and communicate successes of CCBP initiative

  • Appoints an FSIN representative to the JSC
Beneficiaries

The key beneficiaries of the CCBP are the FN of Saskatchewan. While the following fall under beneficiaries, they are also key stakeholders and project managers:

Tribal Councils:

  • Responsible for the building of local CCBP capacity

  • Designates employee to be intricately involved in the development of the plan

  • Provides primary service to member FN seeking technical/ professional advice & assistance relating to the comprehensive planning process

  • Advocates for change to ensure CCBP is supported with respect to the Tribal Council's annual budget and planning process

  • Responsible for technical support to member FN engaged in the CCBP initiative

  • Appoints Tribal Council representative to JSC

  • Ensures adherence to the CCBP Terms and Conditions

The Community Contact

  • Supervises and oversees the CCBP project, attends workshops, training sessions

  • Develops the Plan with the Champion and ensures community participation

The Plan Champion

  • Guides the planning process and builds momentum for the project through leading public participation events

  • Works with the planning work group and analyzes information

  • Ensures that work is completed in sequence and that all information gathered and created is shared with the entire community

  • Reports project progress to Chief and Council

The Planning Work Group

  • Gathers, assembles and presents information together with the Plan Champion,

  • Ensures a broad base of involvement and support among community members in plan development and implementation,

  • Works on the plan by attending meetings, creating maps, building models of the community and advancing the Plan ideas.

2.5 Program Resources

The CCBP pilot project has no A-base budget and is funded by allocations from other programs and agencies. Contributions come from the following INAC sources: the Professional and Institutional Development (P&ID) program, Capital program, Community Economic Development program and for the last three years, Health Canada.

  2005-2006 2006-2007 2007-2008 2008-2009 2009-2010 2010-2011
Category Amount % of
Total
Amount % of
Total
Amount % of
Total
Amount % of
Total
Amount % of
Total
Amount % of
Total
CEU, Dalhousie 6,000 4.7 235,060 47.9 389,397 48.0 491,850 48.0 588,000 44.5 524,253 42.8
                         
First Nations/
Tribal Councils
110,000 86.6 236,940 48.3 396,720 48.9 508,150 49.6 707,500 53.6 575,082 47.0
                         
Joint Steering
Committee
Administration*.
11,000 8.7 19,000 3.9 25,000 3.1 25,000 2.4 25,000 1.9 25,000 2.0
                         
Other
(Diploma program)
                   

100,000

8.2

Totals 127,000 100.0 491,000 100.0 811,117 100.0 1,025,000 100.0 1,320,500 100.0 1,224,335 100.0

(Source: INAC Saskatchewan region)

 

3. Evaluation Methodology

The following section focuses on the methodology that will be used to respond to the main questions the evaluation will strive to answer.

3.1 Evaluation Objectives

The evaluation will assess the extent to which the CCBP demonstrates relevance and performance in compliance with the Treasury Board Policy on Evaluation. The evaluation will also capture lessons and provide evidence-based recommendations with respect to CCBP.

3.2 Evaluation Scope

The evaluation will examine CCBP activities undertaken since its implementation in 2005-2006 up to 2010-2011.

3.3 Evaluation Issues

The evaluation will focus on the following issues:

Relevance

The evaluation will examine the relevance of the CCBP pilot project, including the extent to which it is consistent with federal roles and responsibilities and the needs and priorities of Saskatchewan FN communities.

Design

The evaluation will look at the extent to which the design of the CCBP project is appropriate in relation to the project's objectives, resources and context. It will also look at the adequacy and effectiveness of the governance structure.

Delivery

The evaluation will examine how CCBP was implemented, assessing such issues as the extent to which it was delivered in a timely, rational and efficient manner.

Effectiveness (Results/Success)

The evaluation will determine the extent to which the project's objectives have been achieved and identify the factors that have facilitated and/or limited the project's outcomes. It will also look at unintended results, both positive and negative, and the project's cumulative effects on individuals and participating communities. The evaluators will identify what worked well, what did not and why. In doing so, they will attempt to draw lessons that can be generalized across the region and across Canada.

Efficiency and Economy

Demonstration of efficiency and economy refers to the assessment of resource utilization (inputs) in relation to the production of outputs. It will examine whether the project used the most efficient means to obtain outcomes and whether there are practical alternatives.

3.4 Methodology

3.4.1 Data Sources

The evaluation findings and conclusions will be based on the analysis and triangulation of the following lines of evidence to respond objectively to the identified issues:

Literature Review:

The evaluators will conduct a review of relevant regional, national and international literature about the subject.

Document review:

Project documentation will be reviewed, including management information, contracts, project reporting and newsletters, budget information, community plans, CEU design-build booklets, and other related information.

Key informant interviews:

It is expected that a minimum of 24 and a maximum of 36 key informant interviews will be conducted, including, but not limited to:

  • CEU (3)

  • Yorkton Tribal Council (1)

  • INAC Saskatchewan region managers, and officers (5)

  • INAC headquarters (1)

  • The Community Contacts (for each community, minimum 3, maximum 7)

  • The Plan Champions (for each project, minimum 3, maximum 7)

  • The Planning Work Group members (for each project, minimum 3, maximum 7)

  • Chiefs and council members (3)

  • Health Canada representative (1)

  • FSIN representative (1)

Case Studies:

Four communities will serve as case studies for the collection of detailed qualitative and quantitative data on the project's outcomes on individuals and communities. The case studies will highlight best practices, lessons learned, key successes and how they were achieved as well as factors that were important in facilitating or hampering success.

Each of the case studies will incorporate information from the following lines of evidence:

  • Document review, which will include, for example, the project proposal, the community plan, community profile data, CCBP project reporting

  • Key informant interviews and focus groups, including with the Community Contacts, Plan Champions, Planning Work Group members, community members and leadership, and Tribal Council representatives.

3.5 Considerations, Strengths and Limitations of the Evaluation

As this is an evaluation requested by the Saskatchewan region, it is expected that cooperation between all the players will be high in relation to the provision of information required to perform the evaluation.

A limitation is the lack of a logic model and a performance measurement strategy. As a result, the available performance data are limited. The evaluators have drafted a logic model to assist with the evaluation and will attempt to reconstruct a baseline from available information.

4. Evaluation Matrix

The following Evaluation Matrix consists of the Evaluation Issues, Questions and Data Collection Methodologies.

Evaluation Matrix - Comprehensive Community-Based Planning (CCBP) Pilot Project, Saskatchewan

Issue/Research Questions Indicators Data/Methodology Data Source
Relevance: Continued Need
1. Does the CCBP address a demonstrable need in Aboriginal communities?
  • Evidence of need for CCBP

  • Degree to which CCBP is supporting FN community development (economic, social, politics)
  • Literature review

  • Document review

  • Key informant interviews

  • Case studies
  • Internet, government policy documents, CCBP annual reports, academic papers

  • Interviews with the community contacts, plan champions, planning work group members, Tribal Council representatives, CEU staff, INAC staff
Relevance: Alignment with government priorities
2. Are the objectives of CCBP consistent with departmental and government-wide priorities?
  • Is it aligned with federal roles and responsibilities?

  • Is this bottom-up, comprehensive planning model consistent with INAC policies and strategy?
  • Extent to which CCBP objectives are consistent with government and departmental priorities including INAC's Gender-based Analysis Policy, Sustainable Development Strategy, and new Community Development Framework
  • Literature review

  • Document review

  • Key informant interviews
  • Speech from the Throne, Budgets, Departmental Program Activity Architecture, Ministerial speeches and announcements, Gender-based Analysis Policy, Sustainable Development Strategy and draft documentation on Community Development Framework

  • Interviews with INAC representatives (headquarters and Saskatchewan)
Relevance: Consistency with Federal roles and responsibilities
3. Does the CCBP duplicate or overlap with other departments or jurisdictions?
  • Information indicating duplication or overlap with other departments or jurisdictions
  • Document review

  • Key informant interviews
  • Departmental mandate, policies, and other documents that discuss jurisdictional responsibilities

  • Interviews with INAC; Health Canada, Tribal Council representatives
PERFORMANCE: Design
4.1 To what extent were the project objectives/outcomes clear to all major stakeholders and shared?
  • Were there consistent, realistic expectations?
  • Extent to which project objectives are clearly outlined in project documentation

  • Extent to which stakeholders had shared views of objectives and expectations
  • Document review

  • Key informant interviews

  • Case studies
  • Project documents, steering committee minutes

  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, CEU staff
4.2 To what extent was the project design appropriate?
  • Did the pilot project design help to improve planning processes?

  • Was the model (stages) effective?

  • Was there sufficient time for capacity development?

  • Was there an equitable system in place for determining priorities and selecting projects at the community level?
  • Extent to which communities increased their planning capacity

  • Perceptions of stakeholders as to the effectiveness of the staged approach to plan development implemented over three phases

  • Perceptions of stakeholders regarding adequacy of the time available to develop planning capacity

  • Extent to which the pilot design allowed for broad-based community involvement, including women, men, youth, elders & disadvantaged

  • Extent to which the project design fostered equitable decision making
  • Literature review

  • Document review

  • Key informant interview

  • Case studies
  • Evaluations

  • Community plans, project reports, funding agreements
  • Interviews with INAC staff, community contacts, plan champions, planning working group members, community leadership, CEU staff
4.3 To what extent was the governance structure appropriate?
  • Extent to which the JSC, Yorkton Tribal Council provided adequate direction and guidance

  • Extent to JSC, Yorkton Tribal Council contributed to the achievement of results
  • Document review

  • Key informant interviews

  • Case studies
  • Project documents, steering committee minutes

  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, CEU staff
PERFORMANCE: Delivery
5.1 Were there sufficient and appropriate resources and support to implement the project?
  • Level of INAC support

  • Key informant perspectives on the sufficiency resources and INAC engagement
  • Literature review

  • Document review

  • Key informant interview

  • Case studies
  • Evaluations

  • Project reports, budget information

  • Interviews with INAC staff, community contacts, plan champions, planning work group members, community leadership, CEU staff
5.2 Was there sufficient flexibility to allow the project to adapt to changing conditions/local contexts (language & culture)?
  • Evidence of adaptation
  • Document review

  • Key informant interviews

  • Case studies
  • Project reports, community plans

  • Interviews with INAC staff, community contacts, plan champions, planning work group members, community leadership, CEU staff
PERFORMANCE: Effectiveness - Achievement of expected outcomes
6. What did the project achieve (expected and unexpected outcomes)?

6.1 To what extent is there increased awareness (on the part of pilot communities, INAC & Tribal Councils) of the importance of a comprehensive community-based plan?
  • # of stakeholders who indicate they have greater awareness of the importance of comprehensive community based planning
  • Document Review

  • Key informant interviews

  • Case Studies.
  • Project reports, community plans

  • Interviews with INAC staff, Health Canada staff, community contacts, plan champions, planning working group members, community leadership and members, CEU staff
6.2 To what extent have individual and community stakeholders increased their capabilities regarding comprehensive community-based planning?
  • # of community members who actively participated in preparing the plans with no previous related experience

  • Perceptions of stakeholders as to their increased capabilities
  • Document review

  • Interviews

  • Case Studies.
  • Project reports, community plans

  • Interviews with INAC staff, Health Canada staff, Tribal Council representatives, community contacts, plan champions, planning work group members, community members and leadership, CEU staff
6.3 To what extent are the community plans being used by communities and INAC?
  • number of projects identified by the plans that have been implemented

  • Extent to which communities are using the plans to set priorities/ allocate resources/ lever other programs/partnerships

  • Perceptions of benefits to communities from implemented projects

  • Extent to which the plans are being used by INAC and other government departments to inform program planning and funding decisions
  • Document review

  • Key informant interviews

  • Case studies
  • Project reports, community plans

  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning work group members, community members and leadership, CEU staff
PERFORMANCE: Sustainability
7. To what extent will the benefits of CCBP continue after the pilot projects end?
  • To what extent have communities taken ownership of their plans?

  • Are there plans and resources in place to scale up this pilot initiative?

  • To what extent, if at all, has INAC's planning approach and culture changed or been influenced by the CCBP?
  • # of pilot communities that have updated

  • # of pilot communities that intend to update their plans independently

  • Level of available resources

  • Perceptions of stakeholders regarding the extent to which CCBP has led to change in approach/culture/ reduction in planning duplication
  • Key informant interviews

  • Case studies
  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, community leadership, CEU staff
8. Have the CCBP pilots had unexpected outcomes, positive or negative?
  • Unexpected changes attributed by stakeholders to the pilot projects (e.g. changes in relationships within the community, with INAC, other programs and partners)
  • Key informant interviews

  • Case Studies

  • Document review
  • Project reports

  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, community members and leadership, CEU staff
PERFORMANCE: Lessons
9. What mechanisms are in place to capture lessons and facilitate learning from the pilot initiatives?
  • Evidence of data collection data analysis and good reflective processes
  • Evidence of information sharing with other communities
  • Document review

  • Key informant interviews

  • Case studies
  • Project reports

  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, CEU staff
10. Are there lessons or alternatives that have emerged that could contribute to improved community planning?
  • Updates to guidance documents, procedures

  • Scans of research, academic literature

  • Opinions of project stakeholders
  • Key Informant Interviews

  • Case studies
  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, CEU staff
PERFORMANCE: Economy and Efficiency - Resource utilization in relation to outputs
11. Are the most appropriate and efficient means being used to develop community plans
  • Cost per community
  • Cost per community compared to other provinces
  • Literature review

  • Document review

  • Key informant interviews
  • Evaluations

  • Financial data. project reports, INAC evaluations from other provinces

  • Interviews with INAC BC officials, INAC Saskatchewan staff
12. Are there alternative models that could improve CCBP or reduce the cost?
  • Suggestions for alternative models/cost reductions
  • Literature review
  • Key informant interviews
  • Case studies
  • Academic articles

  • Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, CEU staff

5. Project Management and Quality Control

The Evaluation, Performance Measurement and Review Branch (EPMRB) of INAC's Audit and Evaluation Sector (AES) will be the project authority for the CCBP pilot project evaluation. The evaluation will be performed by EPMRB with the assistance of Stiles Associates Inc.

Three members of the JSC representing INAC, Yorkton Tribal Council and the CEU were consulted in the preparation of this evaluation methodology report. They will review this draft report and once data collection is complete, they will participate in a presentation to validate the preliminary findings. They will also review the draft evaluation report.

6. Timeline

Task Target Completion Date
1. Methodology Report Mid-February 2011
2. Data collection
Field research for case studies
March – April 2011
February 28-March 11, 2011
3. Presentation of Preliminary Findings Mid-April
4. Draft Evaluation Report Early May 2011
5. Final Report May 31, 2011







Appendix B: Interview Guide

Evaluation of the Comprehensive Community-based Planning Pilot Project

Indian and Northern Affairs Canada (INAC) is conducting an evaluation of the Comprehensive Community-Based Planning (CCBP) Pilot Project in Saskatchewan. This evaluation will provide evidence-based conclusions regarding the project's relevance and performance.

INAC Saskatchewan region requested an independent evaluation to assess the results achieved, capture lessons and provide recommendations to inform future support for CCBP.

The evaluation will encompass the work of the pilot project from 2006 to the present. It will involve INAC staff, participating communities, Tribal Councils, and the Cities and Environment Unit from Dalhousie University.

Within the evaluation report all results will appear in aggregate. While quotes may be used in the report, no individuals will be identified. The evaluation is expected to be completed by the summer of 2011 and the results will be posted on the INAC web site following approval by the Evaluation, Performance Measurement and Review Committee.

The following questions provide a framework for discussion with evaluation specialists.

Not all questions will apply to every interview.

Introduction

1. Could you briefly describe your involvement with the CCBP pilot project?

Section 1 - Relevance

2. Is there a need for CCBP in Aboriginal communities? Why or why not?

Probe:

  • How does CCBP support FN community development (economic, social, political)?

3. To what extent are the objectives of CCBP consistent with INAC and the Government of Canada priorities? (INAC only)

Probe:

  • Is it aligned with federal government roles and responsibilities?
  • Is this bottom-up strategy consistent with INAC policies?
  • How is it consistent with INAC's Gender-based Analysis Policy, Sustainable Development Strategy and the Capacity Development Framework currently being developed?

4. Are you aware of any duplication or overlap between CCBP and the work of other departments and jurisdictions?

Section 2 - Design

5. How useful was the CCBP model used for this project?

Probe:

  • Was the staged approach effective?
  • Did it increase planning capacity?
  • Did the design allow for broad-based community involvement, including women, men, youth, elders and disadvantaged?
  • Were these groups involved in decision making?
  • Did the design allow sufficient time for capacity development?
  • Was it useful to implement the project over three phases?

6. How well did the governance structure for CCBP work?

Probe:

  • Did the JSC provide adequate direction and guidance?

7. What did you see as the objectives of the CCBP pilot project?

Probe:

  • Were those objectives and expectations shared by other stakeholders?

Section 3 - Delivery

8. Did INAC provide sufficient resources and support to this pilot project? Why, or why not?

Probe:

  • Was there sufficient engagement by INAC staff?

9. Was there enough flexibility to allow the project to adapt to changing conditions and local contexts (language & culture)? Why or why not?

Section 4 - Effectiveness

10. In your view, what did the project achieve?

11. To what extent is there increased awareness among pilot communities, INAC & Tribal Councils of the importance of a comprehensive community-based plan?

12. To what extent has this project built new capabilities in comprehensive community-based planning?

Probe:

  • What are the increased capabilities for individuals and communities?
  • How many community members with no previous planning experience actively participated in preparing the plans?
  • How are these new capabilities being used?

13. To what extent are the community plans being used by communities and INAC?

Probe:

  • How many of the projects identified by the plans have been implemented?
  • To what extent has the community benefited from those projects?
  • To what extent are communities using the plan to set priorities/ allocate resources/ lever other programs and partnerships?
  • To what extent are INAC and other government departments using the plans to inform program planning and funding decisions?

Section 5 - Sustainability

14. Do you think the benefits of CCBP will continue after the pilot projects end?

Probe:

  • Have communities updated their plans?
  • What are the plans and resources in place to scale up this pilot initiative?

15. To what extent has INAC's planning approach and culture been influenced by the CCBP?

Probe:

  • Has there been a reduction in planning duplication?
  • # of plans requested by INAC in pilot communities before and after CCBP

16. Have the CCBP pilots had unexpected outcomes, positive or negative?

Probe:

  • Have there been changes in relationships within communities, with INAC or other programs and partners?

Section 6 - Lessons

17. What processes are in place, if any, to capture lessons and facilitate learning from the pilot initiatives?

Probe:

  • Is data collected and analyzed, are there good reflective practices?

18. Can you think of any lessons from this project that could contribute to improved community planning?

Probe:

  • Have guidance documents or procedures been updated?

Section 7 - Efficiency and economy of resource utilization

19. Do you think the approach used by this project is the most efficient way to introduce comprehensive community-based planning? Why or why not?

Probe:

  • Are the outcomes achieved worth the cost?

20. Do you have any suggestions for alternative models that could improve the effectiveness of CCBP or reduce the cost?

Do you have any questions or additional comments you would like included in the evaluation?

Thank you for your time.






Appendix C: Logic Model

Logic Model

Logic model description

The logic model contains six cells: five horizontal, and one vertical. The vertical cell contains the titles of the five horizontal cells and is not related to these cells. From the bottom up, they are: activities, outputs, immediate outcomes, intermediate outcomes and final outcome.

The horizontal cells, from the bottom up, are: Cell 1 – Workshops, training sessions, implementation of projects associated with community plans; Cell 2 – Comprehensive community plan containing background, vision and monitoring measures, and community projects achieved; Cell 3 – Community members are better equipped to participate in the development and use of comprehensive community plans; and, better understanding of the importance of comprehensive community planning by communities that participated in the pilot project and by AANDC; Cell 4 – Better planning by communities that participated in the pilot project; and, Cell 5 – Better quality of life (social, economic).

Arrows link four of the five horizontal cells. From the bottom up, one arrow links the last cell with the second-to-last cell. Three arrows link the second-to-last cell and the third cell, and three arrows link the third cell and the second cell. There are no arrows between the second cell and the first.

 






Footnotes:

  1. Community-based planning is a process through which a community establishes a direction for the future and determine specific actions to realize that direction. This is accomplished in a way that is widely understood, accepted and appropriated by many members in the community. The resultant plan allows the community to make informed decisions about where to seek funds, how to spend limited resources, how to react to request for action, how to protect the environment and how to provide new opportunities for residents - all are factors that create a sustainable community. (return to source paragraph)

  2. Technical expertise for the pilot project came from outside the province through a contracting process that was tendered following Phase 1 of the project. (return to source paragraph)

  3. The General Assessment is a tool to support the management of AANDC funding agreements. It works by taking an annual "snapshot" of the funding recipient's past performance and identifies strengths and emerging risks that may have an impact on its future performance. Each funding recipient is assessed and rated as to having either a "low", "medium" or "high" level of risk. Once through the process, each recipient receives a rating as well as an explanation for the rating and, where needed, recommendations for managing significant risk. (return to source paragraph)

  4. The General Assessment is a tool to support the management of AANDC funding agreements. It works by taking an annual snapshot of the funding recipient's past performance. Each funding recipient is assessed and rated as to having either a low, medium" or high level of risk. Once through the process, each recipient receives a rating along with reasons for the rating and recommendations, where needed, for managing significant risk. (return to source paragraph)

  5. As mentioned earlier, Health Canada's role in the project was limited largely to providing funding and attending Joint Steering Committee meetings. (return to source paragraph)

  6. Cities and Environment Unit, Dalhousie University. (2006) First Nation Community Planning: Saskatchewan Pilot Projects Terms of Reference. (return to source paragraph)

 
 
Date modified: