Urban Aboriginal Strategy Pilot Projects Formative Evaluation - Final Report

Date: (March 2005)

PDF Version   (357 Kb, 76 pages)

 


Table of Contents


1. Introduction

In 1998, the Government of Canada established the Urban Aboriginal Strategy (UAS) to work with partners to better address the serious socio-economic needs of Canada's urban Aboriginal population. It was part of Gathering Strength: Canada's Aboriginal Action Plan, a response to the Royal Commission on Aboriginal Peoples.

A strategy targeting urban Aboriginal communities was viewed as critical given demographic and socio-economic trends that were becoming apparent. In the 2001 Census of Canada, almost one million people identified themselves as Aboriginal (Inuit, Métis and First Nations - status and non-status - peoples). Approximately 50 percent of those live in urban centres, with one quarter of Aboriginal people living in just 10 Canadian cities: Winnipeg; Edmonton; Vancouver, Calgary, Toronto; Saskatoon; Regina; Ottawa-Gatineau; Montreal and Victoria.

Urban Aboriginal Youth: An Action Plan for Change, [1] a report of the Standing Senate Committee on Aboriginal Peoples tabled in October 2003, noted that "not only do Aboriginal people constitute a significant percentage of urban populations, especially in the western provinces, but on the whole they have higher rates of joblessness, less formal education, more contact with the justice system, and are in poorer health than their non-Aboriginal counterparts." It noted that these demographic indicators suggest that the well-being of Aboriginal people in cities has a direct impact on the wellbeing of the cities themselves, most especially in western Canada, where a substantial number of Aboriginal people reside.

The UAS seeks to enhance coordination, improve horizontal linkages and policy integration within the federal government and partner with other stakeholders to better address the needs of urban Aboriginal people. In its 2003 Budget, the Government of Canada (GOC) announced that it was allocating $25 million over three years through the Office of the Federal Interlocutor for Métis and Non-Status Indians (OFI) to support the UAS. The bulk of UAS funding is being used to support pilot projects in eight priority urban centres: Vancouver, Calgary, Edmonton, Saskatoon, Regina, Winnipeg, Toronto and Thunder Bay. The GOC is working with selected provincial and municipal governments and community members to plan, fund and help implement the projects, and has assigned UAS coordinators in each of those designated communities.

In late 2004 the federal government announced a further expansion of funding for the UAS to a total of $50 million, to enable the currently participating communities to carry their work forward and to allow several new communities to participate.

1.1 The Purpose of the Formative Evaluation

Implementation of the pilot project phase of the UAS is well underway, with communities planning their longer-term approaches to using the UAS as a vehicle for change, and working to make best use of the available project funding. In August, 2004 the Office of the Federal Interlocutor (OFI) (until recently at the Privy Council Office but now relocated within Indian and Northern Affairs Canada (INAC)) initiated a formative evaluation of the UAS pilot project program in order to assess early progress, ensure that performance measurement strategies and activities are on track, support a summative evaluation scheduled for 2007, and identify any improvements that should be made on the basis of the experience thus far.

The UAS Pilot Project Initiative was expanded through Budget 2004. This formative evaluation did not study additional initiatives undertaken through this expansion. See 2.6 for more details on the UAS renewal and expansion.

This is the final report of the UAS formative evaluation. It includes a description of the initiative and the identification of its objectives and associated activities; a section detailing the methods used for the conduct of the evaluation; a section providing detailed evaluation findings organized according to the major issue areas the study addressed; a set of conclusions about progress to date based on the findings; and, a set of recommendations to help guide future efforts under the UAS.



2. Program Description

This section of the report provides a description of the objectives of the UAS, the delivery approach, activities being undertaken at the national and local levels, and resource allocations.

2.1 Objectives

The ultimate objective of the UAS is to narrow the socio-economic gap (also often referred to as "closing the gap in life chances") between urban Aboriginal people and the mainstream population. The pilot project phase of the UAS continues earlier efforts in the eight designated communities, enhanced by project funding.

The objectives of the pilot project phase of the UAS are to:

  1. Build organizational capacity within urban Aboriginal organizations, groups and communities at the local level in order to enhance community leadership;

  2. Develop partnerships with provincial and municipal governments, urban Aboriginal organizations, groups and communities in order to engage in sustainable community development; and,

  3. Coordinate federal government resources across departments in the Pilot Project cities in order to focus efforts on addressing the disparity between urban Aboriginal and non-Aboriginal people.

It was recognized that these objectives would not be fully achievable in the 3-year funding period for the pilot projects, but that progress in each area was possible. It was believed that by 2006, the UAS Pilot Projects could work to achieve the following specific outcomes:

  • strategic management of urban Aboriginal issues on a region-wide basis;
  • increased federal responsiveness to community needs;
  • strong policy recognition of urban Aboriginal issues;
  • more academic research and interest in urban Aboriginal work;
  • more publicly available statistics and research on urban Aboriginal issues;
  • an enhancement of stakeholders and resources brought to bear on urban Aboriginal issues;
  • identification of best practices as identified in each pilot project city;
  • improved employment capability and job readiness for urban Aboriginal people; and
  • increased skills and abilities of urban Aboriginal people.

2.2 UAS Activities

The logic model for the UAS [2] identifies four major sets of activities.

2.2.1 Capacity building

Capacity building is one of the primary activities of the UAS. It is viewed as a process by which organizations increase their abilities to: set objectives and priorities, perform functions, solve problems and achieve solutions; and, understand and deal with urban Aboriginal needs in a broad, strategic context and a sustainable manner.

At the national level the UAS is working to help build community capacity to address urban Aboriginal issues by helping to build partnerships within the federal sphere and with provincial and municipal governments, by actively promoting awareness of urban Aboriginal issues within government and in other sectors, and by supporting, compiling and disseminating research. At the local level, UAS coordinators and staff are working to support Aboriginal community mechanisms to identify needs, set priorities and develop strategic approaches to address urban Aboriginal issues. In addition, they are working to establish better partnerships within government and between government and Aboriginal communities. Finally, they are providing support for the project funding component of the UAS, to help build service delivery capacity.

2.2.2 Horizontal co-ordination

Horizontal co-ordination is the other primary activity of the UAS. The UAS Pilot Projects are seen as opportunities to test new and innovative ways for the federal government to work co-operatively in the regions and respond better to community needs. One way in which this is envisioned is via the horizontal terms and conditions of the Pilot Projects. A horizontal mechanism was set up that allows for co-ordination of federal efforts in urban areas, and that sets out when it would be appropriate for federal departments to implement these terms and conditions in the context of a UAS pilot project. The use of the UAS terms and conditions by other departments can be done where the UAS pilot project is: community-based; beyond the parameters of any one federal department (but could be carried out if two or more of the departments participate on the project); consistent with the mandate(s) of the departments and consistent with the federal UAS objectives

In addition to the use of the flexible UAS terms and conditions and the idea of a "single window" for funding of UAS projects, UAS coordinators have established local federal committees to share information and coordinate activities related to urban Aboriginal issues.

Under the 2003 Treasury Board Submission, the following departments have agreed to participate and use the UAS horizontal terms and conditions when they participate in a pilot project under the UAS: Western Economic Diversification, Human Resource Development Canada, Department of Justice, Health Canada, and Canadian Heritage. Partnerships continue to be developed with other federal departments and agencies with mandates relating to urban Aboriginal issues.

2.2.3 Research

The Pilot Project phase includes funds available at the national level to conduct or to compile research into issues that affect urban Aboriginal people. The research is intended to assist federal officials and policy makers with future issues related to urban Aboriginal people.

2.2.4 Advocacy

The fourth important element of the UAS is to advance awareness and understanding of the urgency and nature of urban Aboriginal issues. This is being undertaken through conferences, publications, speeches and the use of a UAS website.

2.3 The Pilot Projects

This phase of the UAS incorporated the availability of project funding in the eight participating communities. By working with selected provincial and municipal governments (and others), the Government of Canada intended to implement pilot projects to test innovative solutions to address local priorities that could not be undertaken through existing programs. In recognition that this initiative is dependent on the experimental nature of the pilot projects and that the UAS is meant to be largely community-driven, it was considered premature to identify the specific types of projects that would be undertaken in each of the eight UAS priority cities.

Early consultations did, however, provide some indications. For example, some pilot projects were seen as likely to focus on improving educational outcomes for urban Aboriginal children by increasing parental and community involvement in the delivery of the core curriculum in inner city schools; improving feelings of inclusiveness and cultural pride among Aboriginal students so as to increase their self-reported interest in the core curriculum; and ultimately aiming to remove barriers to high school completion by Aboriginal students. Another type of project that was being considered in some of the UAS sites was an innovative approach for funding community priorities by way of a collaborative granting process.

Other projects might focus on restoring vitality to disadvantaged neighbourhoods through improvements in community safety and crime prevention, housing, education, business community partnerships and supports and community service centres, developing transitional centres that provide Aboriginal people with comprehensive information and services for making an easy transition to major urban centres, including shelter for short periods of time. Mobile services may also be considered to communities from which Aboriginal people are moving from so that they have information before they arrive in the urban centre. [3]

2.4 Resource Allocations

The Federal Interlocutor received a total resource allocation of $25 million from 2003-2006 to undertake the Pilot Project phase of the UAS. Of that total, $18M or $6M annually has been allocated to the participating cities for the funding of pilot projects [4]; this funding has been delivered through contribution agreements. In addition, regional offices of the lead federal departments in each province represented in the UAS have been allocated an amount to enhance federal horizontality and Aboriginal engagement and to conduct or compile research under the UAS. In British Columbia, Alberta and Manitoba the lead department is Western Economic Diversification (WD). In Saskatchewan and Ontario the lead department is Human Resources and Skills Development Canada (HRSDC).

Urban Aboriginal Strategy - Enhanced Funding ( $25M Over 3 years)
Purpose of Funding 2003-04 2004-05 2005-06
Pilot Projects  [5] $6.0M $6.0M $6.0M
Federal engagement  [7] Aboriginal capacity $2.5M  [6] $1.5M [6] $1.5M [6]
Research  [8]  $300k $250k $250k
PCO support $200k $250k $250k
Total $9.0M $8.0M $8.0M

2.5 Governance

As the lead for the UAS nationally, the OFI is responsible for the co-ordination and management of an inter-departmental committee structure that oversees the implementation of the UAS. The structure developed at the outset consisted of three groups: a National Committee of Federal Officials to provide strategic advice and guidance; an interdepartmental working group of Headquarters' officials to ensure policy and program co-ordination; and, regional committees of federal officials to coordinate on-the-ground implementation of the UAS from a federal perspective.

In each participating community, steering committees or other bodies have been formed, typically with membership from the Aboriginal community, the provincial and municipal governments and federal department representatives, to guide the implementation of the Strategy, set priorities and directions, make recommendations about the pilot projects to be funded, and otherwise oversee the implementation of the UAS within the terms and conditions of the Strategy.

Some consultations took place with provincial and municipal governments regarding the selection of the initial eight communities, and they were invited to recommend potential communities for participation under the 2004 expansion. The OFI made the final selection decisions based on those recommendations and on an assessment of statistics on the socio-economic circumstances of Aboriginal people in the communities and the communities' apparent readiness to take on a UAS initiative.

Horizontal initiatives require effective communications between all participants to ensure the expected outcomes are on track to be met. The inter-departmental committee structure was intended to serve this purpose and to ensure that all departmental stakeholders had access to the same information and opportunities to learn from one another. The committee structure supports the sharing and integration of common performance measurement data.

2.6 Recent Renewal and Expansion

Budget 2004 delivered on the federal government's commitment to cities and their Aboriginal residents by doubling the government's investment in the UAS, from $25M to $50M, and by extending the UAS funding to four years (to 2006-07) so that current projects with promising results could be expanded and, in partnership with willing provincial and municipal governments, more communities could participate.

Expansion of the UAS provides the federal government with the opportunity to re-adjust its course slightly to clarify and strengthen the UAS Terms and Conditions, to have additional federal departments participate in UAS pilot projects (the Aboriginal Policing Directorate at Public Safety and Emergency Preparedness Canada and Industry Canada - Aboriginal Business Canada), and to consider potential federal roles in urban Aboriginal issues such as health, economic development, education, housing and labour market development.

A significant portion of the funding provided for the UAS in Budget 2004 will be allocated to increase funding to the current pilot project communities in Vancouver, Edmonton, Calgary, Saskatoon, Regina, Winnipeg, Thunder Bay, and Toronto. Other funding will go to develop the UAS in additional communities, which are being identified in consultation between the federal, provincial, and municipal governments. At present, four additional communities have been identified to receive pilot project funding. They are: Prince George, British Columbia; Lethbridge, Alberta; Prince Albert, Saskatchewan; and Thompson, Manitoba.



3. Methodology

This section of the report describes the methods used to conduct the formative evaluation, and an assessment of the limitations to the quality of the findings.

3.1 Development of Methodologies

The Terms of Reference for the project and the results of the first meeting with the project authority established seven areas of inquiry that the formative evaluation needed to address:

  1. Whether the UAS pilot projects, and the UAS initiative as a whole, were proceeding as intended, and if there were any problems with implementation that needed to be addressed;

  2. Whether the existing approach to performance measurement and the collection of information for that purpose was sufficient;

  3. Whether there were any gaps in the types of measurements being applied or the information being collected, that presented a risk to the strength of a future summative evaluation and that required changes to the performance measurement strategy;

  4. Whether the pilot projects, and the initiative as a whole, were making progress toward the achievement of the desired outcomes;

  5. Whether there were areas where UAS activities overlapped with other government programs and services inappropriately;

  6. Whether there were indications that UAS is succeeding in leveraging partner resources and whether the size and distribution of UAS project and administrative allocations had enhanced/hindered initiative objectives; and,

  7. What lessons could be learned from the formative evaluation that offer opportunities to improve the Strategy.

It was decided at the initial meeting that there had already been sufficient work done by OFI in reviewing the literature on urban Aboriginal issues, identifying the needs of urban Aboriginal communities, and establishing the rationale for, and relevance of, the UAS. It was decided that the project would not devote additional resources to this element. Instead, it was decided that the emphasis would be placed on issues associated with the implementation of the UAS: delivery models; activities; progress to date; what can be done better; whether different approaches work better in different circumstances. It was recognized that the study would be largely exploratory in nature, given the qualitative nature of the main evaluation issues and the lack of available data on outcomes to date.

3.2 Methods Used

3.2.1 Document review and analysis

A review of documents was undertaken at a number of points in the study, for a variety of purposes:

  1. Review of UAS pilot project founding documents and planning documents, to assess priorities and the relationship between the initiative and broader government and OFI priorities;

  2. Review of project descriptions and proposals, to understand the nature of the projects being funded;

  3. Review of descriptions of other related government programs and services, to understand the opportunities for collaboration and any risks of duplication;

  4. Review of UAS budget documents and administrative data and reports, to assess the extent to which the initiative and its constituent elements are able to function effectively with existing resources.

3.2.2 Review of UAS outcomes and performance indicators

In order for the formative evaluation to have a sound analytic framework for assessing implementation progress, and to ensure that the information/data collection mechanisms are in place for a future summative evaluation, it was critical that there was agreement on a set of clearly defined outcomes and performance indicators for the Initiative. The outcomes need to reflect clearly the objectives of the UAS, and the performance indicators establish in a realistic way how progress (and later, success) will be measured.

Discussions at the first project meeting indicated that some time needed to be devoted to this aspect of the initiative. The evaluation included several steps in this regard:

  1. An initial assessment by the consultants of the UAS pilot project Results-based Management and Accountability Framework (RMAF) and the outcomes and indicators it contained;

  2. A meeting with UAS managers/staff in Ottawa to review the findings of the assessment, and agree on an approach to making any required adjustments;

  3. Development of a draft set of indicators, and their dissemination to the regional UAS coordinators and other Evaluation Steering Committee members for feedback;

  4. Development of a UAS evaluation strategy incorporating the refined outcomes and indicators;

  5. Based on this evaluation strategy, development of a set of issues, indicators and methods for the current formative evaluation.

The last item, the guide for the formative evaluation, reflects closely the original terms and conditions for the evaluation but also takes into account the decisions made at the early meetings about areas of emphasis, and the thinking that went into the development of clear and well-defined outcomes and indicators for the UAS as a whole. The table of issues, indicators and methods for this formative evaluation are presented in Appendix B, along with the similar table for the future summative evaluation of the UAS (which is a product of the formative evaluation).

3.2.3 Interviews in the National Capital Region

Interviews were conducted with OFI managers and staff and officials from federal partner departments and agencies at a national level, to address issues related to the implementation of the initiative, the delivery model, horizontality and working relationships between the OFI and regional UAS coordinators and community leaders.

The interviews were open-ended in nature, based on an interview guide listing the issues and specific questions to be addressed. The interview guides (tailored to the individuals being interviewed but with a common core set of questions), were developed in collaboration with OFI and a team member from Consulting and Audit Canada. [9] Interviews were conducted with four OFI officials (including one who was at the Cities Secretariat at the time of the interview but has since returned to the OFI), and Headquarters officials at Human Resources and Skills Development Canada, the National Secretariat on Homelessness, and Western Economic Diversification.

3.2.4 Community case studies

The community case studies were the core work of the formative evaluation. They took place in all eight designated communities (Vancouver, Edmonton, Calgary, Regina, Saskatoon, Winnipeg, Thunder Bay and Toronto). They included the following elements:

  1. Individual and group interviews with regional UAS coordinators, UAS community planning leaders and committee members, officials from partner federal departments participating in the UAS at the local level, provincial and municipal government officials in departments relevant to the UAS, and managers of a small number of UAS-funded projects. The interviews addressed issues related to:

    • the design of the initiative and the new funding mechanisms;
    • implementation to date and factors facilitating or inhibiting progress;
    • evidence of immediate outcomes and preliminary medium-term outcomes;
    • horizontality and coordination;
    • budget issues; and other issues that arose in the course of the interviews.

    The interviews were based on standardized interview guides tailored to the types of respondents being interviewed. They were primarily open-ended in nature, lasting anywhere from 45 minutes to several hours depending on the availability of the respondent and the amount of involvement they had had in the initiative.
The table below shows the number and types of interviews conducted in each community.
Community # of Federal Government Interviews # of Provincial and Municipal Government Interviews # of Interviews with Aboriginal Community Members Total # of Interviews
Vancouverr 6 4 8 18
Edmonton 6 3 8 17
Calgary 4 3 8 15
Regina 8 2 7 17
Saskatoon 7 2 8 17
Winnipeg 6 2 9 17
Thunder Bay 4 2 5 11
Toronto 7 2 10 19
Total 48 20 63 131
  1. Document review (as noted above) where local community planning and project documents were available.

  2. Community-specific assessments of implementation to date (based on interviews, document reviews and on-site observation), including:

    • structures and procedures in place;
    • horizontal mechanisms to enhance coordination;
    • partnerships across levels of government and with community groups;
    • inclusion in planning and decision-making; and
    • community capacity building.

3.2.5 Survey of Aboriginal stakeholder organizations

The formative evaluation included an internet-based survey of Aboriginal organizations that are stakeholders in the UAS process in the participating communities, in order to expand the reach of the evaluation beyond what would be possible through interviews, and in order to address certain issues that were amenable to a more quantitative approach. The survey addressed issues including:

  1. Awareness of the UAS;
  2. Experiences with the UAS to date;
  3. Aboriginal community involvement in UAS planning, decision-making and fund allocation processes;
  4. The representativeness and inclusiveness of those processes;
  5. Experiences (if applicable) with the processes; and
  6. How the initiative might be improved.

The survey was targeted to all Aboriginal organizations involved in any capacity in organizing around urban Aboriginal issues, service delivery, research, advocacy or political organization. The population of organizations was identified by the UAS coordinator in each participating city. Lists were provided to the evaluators based on criteria provided to the coordinators. In many cases the lists did not include e-mail addresses, so the evaluators contacted those organizations by telephone to obtain an e-mail address, or to arrange another mode of delivery. It was also found that many of the existing e-mail addresses were no longer in service, and these cases were followed up as well, to maximize the number of organizations receiving notice of the survey.

Organizations on the eight lists were sent an e-mail (or a fax or letter in some cases) that described the purpose of the survey and the evaluation, and directed them to the internet web-site where the survey was located. The contacts were directed to the executive director or president of the organizations, to be completed for the organization (as opposed to the individual respondent). It was left up to them to decide how to proceed internally with completion of the survey.

The survey itself provided some background information of a general nature about the UAS and some community-specific information to take into account local UAS implementation and local terminology being used.  [10] The survey was designed to guide respondents easily from one question to the next, with drop-down boxes or choices that required only a click of the mouse on the appropriate response. It also allowed for respondents to add additional comments.

Ultimately, 402 Aboriginal organizations were informed directly about the survey and invited to go to the website to complete it  [11]. Reminders were sent twice in the course of the 3-week period of the survey. The table below provides a breakdown of the number of organizations identified in each community, and the number of organizations that responded to the survey. It shows that the response rate for the survey was about 29%. This was less than what was hoped for but did not present a major surprise to the evaluators for several reasons:

  • The evaluators had no way of determining the quality of the lists of organizations provided by the UAS coordinators. The lists were based on lists the coordinators had available to them for other purposes and were not developed specifically for the survey at the time of the survey. Thus, evaluators were aware that the lists might be out of date.

  • In establishing the criteria for the lists evaluators decided, in collaboration with OFI, to be as inclusive as possible, so as to avoid any inappropriate assumptions about the kinds of organizations that might have an interest in urban Aboriginal issues. Discussions with regional coordinators confirmed the importance of this approach, and led, for example, to our including cultural organizations because Aboriginal communities do not separate cultural and political or socio-economic issues the way mainstream culture often does. While the evaluators are satisfied that this approach was appropriate, it meant that there was a risk that some organizations included on the lists would be less likely than others to be aware of the UAS or choose to participate in a survey on the UAS.

  • Evaluators were aware at the time of the survey that communication efforts about the UAS had not been a high priority in most of the eight communities in the early stages of the Pilot Project phase, and this meant that awareness about the UAS might not be as broad in the Aboriginal communities as they might otherwise have been. This was recognized as presenting a risk that the response rate might be low.

  • The survey was viewed as a reasonably cost-effective way of communicating with the broader Aboriginal communities in the eight cities, and the internet-based approach, while designed to be as accessible as possible, was known to present a possible risk that some potential respondents would not be comfortable. Efforts were undertaken to provide the survey via fax in cases where internet was not available to a responding organization In general, it was understood that most written survey approaches would present a similar risk, as opposed to an in-person approach that was not considered feasible.
Survey of Aboriginal Organizations in UAS Communities
Community # of Organizations
Informed of the Survey(N)
% of Total N Responses (# and %)
Vancouver 116 29% 24 21%
Edmonton 48 12% 21 44%
Calgary 13 3% 2 15%
Saskatoon 30 7% 5 17%
Regina 41 10% 14 34%
Winnipeg 55 14% 13 24%
Thunder Bay 24 6% 9 38%
Toronto 75 19% 23 31%
Community unknown     6 100%
Total 402 100% 117 29%

3.3 Methodological Limitations

The formative evaluation was exploratory in most respects because the pilot project phase of the UAS was relatively new, and it was unknown how the eight communities had each decided to implement this phase, according to their own circumstances. The methods used focused to a large extent on key informant interviews, with supporting information from the review of documents and the survey of Aboriginal organizations.

In terms of the active participants in the UAS in each community, and in terms of coverage of the range of types of participants (federal lead departments, partner departments and agencies, provincial and municipal officials and representatives from the Aboriginal communities), the key informant interviews are considered to have provided a solid basis for the findings. With few exceptions interviews were conducted with all individuals identified by government and Aboriginal community leaders as being important to consult with. The interviews were lengthy (often up to two hours in length and rarely less than one hour), and covered all the key evaluation issues and other issues raised by respondents.

That being said, there are inherent limitations to a case study approach based primarily on key informant interviews. Interviews capture perceptions that can be influenced by many factors including pre-existing ideas or opinions, the kind of information that informants have been exposed to, and alliances that form during initiatives that help determine points of view. Interviews can also be used by respondents to promote an agenda. In short, interviews do not represent a reliable source of objective information about what has transpired. With careful planning, selection of respondents and crafting of interview guides, experienced interviewers can minimize the limiting effect of those factors. However, where circumstances permit and resources are available, it is preferable to have additional supportive sources of information.

In this case there were some documents available to support descriptive aspects of the interview findings. The fact that a large number of interviews were conducted with key informants in all eight communities allowed us to have confidence in findings where perspectives emerged consistently. The survey of Aboriginal organizations was conducted in an effort to expand the scope of the study to include community members who were not necessarily active in the UAS.

However, the survey itself presented limitations. Because of the methods used for establishing the population of organizations for the survey and the low response rate, it is clear that the results cannot be seen as representative of the views of the Aboriginal communities as a whole, in the eight cities. Even though response rates for internet-based surveys are often lower than they would be for mail or telephone surveys (reportedly often in the 20% range for large commercial surveys and in the 30-40% range for smaller, more targeted surveys), the fact remains that a response rate of at least 50% and preferably close to 60% is desirable in order to make inferences about the broader populations represented by respondents. However, the responses at a national level, and in cities where the response rate was relatively higher, offer some indication of community awareness of the UAS and community views on some other broad issues, given that the population the survey was provided to were identified as the most likely to be interested. Findings from the survey are included in the analysis only when they indicate a strong tendency across most communities.

Because of this limitation to the survey, the evaluation did not succeed in obtaining reliable information on the views of the Aboriginal communities as a whole, on a range of issues that proved important. These include the representativeness of the current steering committees, the view of the broader communities on how they could be represented effectively in a UAS process, views about priorities for the communities relative to priorities being established under the UAS, and views about the way the UAS has been implemented. There were good reasons for the methodological choices made. It was not considered feasible with the time frame and budget for the evaluation to undertake a more comprehensive approach to canvassing the broader Aboriginal communities. As well, it was considered premature, given that little was known at the planning stages about what issues might need to be raised within the broader communities.

The evaluation has identified findings related to these issues, but they are based primarily on the perceptions of key informants and to a limited extent the responses to certain survey questions.



4. Findings

This section of the report presents the findings from the research described above. The findings are organized around issue areas identified in the evaluation terms of reference and in the initial review process. In each issue area the source of the findings is identified and findings are presented. Although there are references to specific communities when they exemplify a certain trend or offer a relevant alternative from the norm, in general the findings are national in scope. Participating communities may be conducting their own evaluations to complement this national evaluation.

The findings are generalized when they represent a clear majority of the views of relevant informants and are supported, or at least not contradicted, by any available alternative lines of evidence. Where an alternative perspective was found in three or four communities or at least one-quarter of the relevant informants interviewed, this is reported. Also, where one or more communities presented a distinctly different finding from the norm, this is reported and the reasons are examined.

4.1 Aboriginal Involvement/Inclusiveness

Evaluation Questions:

4.1.1 To what extent have urban Aboriginal communities been involved in planning and implementation of the initiative?

Finding: Members of the Aboriginal communities have been integral to the implementation of the UAS in the great majority of participating cities, primarily through steering committees that are setting priorities and making project funding decisions. However, the Aboriginal communities as a whole do not consider themselves necessarily represented by the existing steering committee members, and there are challenges to current member selection approaches from the broader communities and from Aboriginal political organizations.

4.1.2 Is UAS planning and decision-making and project funding accessible to all interested Aboriginal stakeholders?

Finding: Lead federal departments made initial efforts to engage the Aboriginal communities as broadly as possible, but since steering committee memberships were established there has been little communication with the broader communities, and this has effectively limited access to planning and decision-making. Calls for proposals for project funding have been advertised widely, but lack of communication has led to a wide perception of limited access to funding.

Source:

Findings in relation to these questions are based on interviews in all eight communities with the Aboriginal representatives sitting on UAS steering committees  [12] or otherwise directly involved in the initiative at the local level, interviews with Aboriginal political leaders not involved directly in the UAS, interviews with federal, provincial and municipal government officials involved in the UAS, a review of relevant documents, and a survey of Aboriginal organizations in the eight communities.

Discussion:

The UAS was conceived as an Aboriginal community-driven initiative. The expectation was that Aboriginal communities would set priorities, determine how best to organize the long-term strategic planning inherent in the initiative, and make decisions about how to target expenditures, subject to UAS and other program terms and conditions.

Aboriginal representatives in seven of the eight communities have been actively involved in UAS planning and decision-making. With the exception of one community in which the planning process has not proceeded to date, Aboriginal members comprise at least 50% of steering committee membership and in most cases they have a strong voting power. In one of those communities the committee process was Aboriginal-based. However, the committee has recently been disbanded by HRSDC in Ontario because disagreements within the committee had proven extremely difficult to resolve, with the result that the committee was not able to put forward plans or project proposals that had broad support. The decision was made to disband the existing committee, establish clearer terms of reference and decision-making procedures, seek to broaden the base of Aboriginal membership to better represent the Toronto Aboriginal population, and establish a new committee under those new arrangements.

This issue of Aboriginal involvement is greatly complicated by the question of what constitutes representation or inclusion of the Aboriginal communities. The process for selecting representatives from the Aboriginal community has varied greatly. In general, community-wide meetings were held at the beginning of the pilot project phase of the UAS. Invitations were extended to all Aboriginal organizations that the local federal officials were aware of, and the meetings were advertised in the community in order to encourage as wide participation as possible. In some cases the meetings resulted in a vote for community representatives for a steering committee. In the majority of cases participants in the general meetings were asked if they would like to sit on a steering committee, and some lead members of the community stepped forward without being chosen through a formal process.

An additional element in the selection of representatives for the UAS was the role of the formal Aboriginal political organizations such as the provincial affiliates of the Métis National Council and the relevant Treaty organizations at the local level, and the Assembly of First Nations and their regional Chiefs, and the Métis National Council at the national and regional levels. Prior to the pilot project phase the major political organizations in each community had been engaged in a process of identifying Aboriginal community priorities and discussing longer-term strategies. However, in most communities that process did not progress to the extent that had been intended, in part because of differing views as to how to implement the UAS, and in part because the lack of dedicated resources for the initiative at the community level discouraged active and ongoing participation. It was noted in most interviews at the community level that there are a limited number of Aboriginal leaders in each community with the skills and experience to guide the UAS process, and that these leaders are already extremely busy with their full-time positions and other volunteer work. This means that participation in new initiatives can be difficult unless there is a clear benefit.

The political organizations, and in particular the Métis National Council through its provincial affiliates, have taken the position that they represent the Métis people in the participating communities, and should be given the authority and a proportion of the funds to implement the UAS on their behalf. This view is linked to the broader movement toward Aboriginal self-government. There is a concern among some members of the Aboriginal community that were interviewed, that the UAS could lead to the withdrawal of funds that are now provided to Aboriginal organizations in areas such as employment and training, in the interests of reviewing overall funding approaches. This is viewed by some as an intrusion by the federal government into areas that Aboriginal organizations are capable of planning and delivering themselves.

According to a large majority of the Aboriginal community members interviewed for the evaluation, this perspective is not widely shared among service providing agencies and among some local chapters of the political organizations. The common perspective reported was that urban Aboriginal populations are diverse and often are not affiliated with any Aboriginal political organizations. The service providers report being focused on providing better quality service to all Aboriginal people regardless of their origins, and do not see any political organization as representing Aboriginal interests in a broad way.

That being said, there is by no means a consensus that the existing UAS steering committees are representative of the communities, and there were a number of concerns raised by the people interviewed, and similar concerns expressed in the survey responses.

First, there is a perception in several communities that steering committee members were "handpicked" by federal officials, and that the government was driving the process more than it should be. Where Aboriginal representation is seen as resulting from selection by the lead federal department, Aboriginal community members who otherwise support the idea of the UAS are dissatisfied with this approach, and would prefer an Aboriginal community selection process, even if it requires more time to establish.

A second concern is that the terms of reference for the steering committees have typically been insufficiently developed, so community members are unclear about how the committees function, what the criteria are for membership, what provisions there are for turn-over of membership, and how decisions are made. Even where members of the committee were voted by the community at large, there is now dissatisfaction in the broader community (as reflected in interviews and the survey responses) because of this lack of clarity. In Toronto in particular, the lack of sufficient terms of reference has been a major factor in the disbanding of the committee. There does not appear to be a clear line of responsibility nationally for developing such terms and conditions and other related tools, and communities have undertaken to develop them to varying degrees.

Related to community perceptions of how they are being represented on the UAS is the issue of communications. It is acknowledged in the large majority of UAS communities that communication with the Aboriginal community at large has been lacking. Community awareness of the UAS was reported by Aboriginal respondents to be low, and this is corroborated by findings from the survey that while about 70% of respondents had heard of the UAS, about half had had no involvement at all, and those same organizations reported being dissatisfied with the amount of Aboriginal involvement in the initiative. Certainly a lack of communication is seen as impeding wider participation in the initiative. Community UAS leaders including federal and other government officials agreed that little emphasis had been placed on developing an ongoing feedback mechanism with the broader community or any systematic communication, and that this would have to be a high priority in the coming months.

Survey responses and interviews with community members outside the steering committee membership indicated a serious concern that UAS funds were being distributed primarily to the agencies represented on the steering committees, and that the member agencies constituted something of a closed club that excluded other agencies from a real opportunity to obtain project funding. This was expressed as a perception, rather than actual knowledge of how the funding process functions, and it may be that the lack of public communication and transparency has engendered this perception. Evaluators did not examine the awarding of project funds to determine the fairness or openness of the process. Where an open request for proposal (RFP) process has been used for funding allocation, the calls for proposals have been disseminated widely. As well, federal officials responsible for the UAS at the community level, and Aboriginal participants in the funding process, indicated that formal decision-making processes were established and that conflict of interest guidelines were used to avoid direct conflict, but they acknowledged that committee members might have an advantage by being more knowledgeable about the UAS and its priorities. As well, they noted that the individuals active on steering committees are typically from organizations that are active and well-established in the delivery of services to the community, so it would be expected that they would have an interest in applying for available funding.

4.2 Federal, Inter-government and Overall Collaboration/Partnership Building

Evaluation Questions:

4.2.1 How effectively have federal and provincial governments, Aboriginal organizations and other stakeholders worked together to implement the UAS? What are the factors influencing effective collaboration?

Finding: Collaboration among all participating parties has been generally positive and constructive, and there is a reportedly high level of trust among Aboriginal participants in most communities that government participants understand the nature and extent of the problems in Aboriginal communities and are genuinely committed to a process for making improvements. However, relationships between the federal government and Aboriginal communities are still fragile due to:

  • Disagreement by some Aboriginal community members on the issues of representation and Aboriginal self-government
  • Previous strained relations/distrust
  • Lack of clarity about the future of the UAS
  • Tensions between the strategic focus of the UAS and spending pressures

4.2.2 Have federal departments succeeded in coordinating their efforts in relation to urban Aboriginal issues?

Finding: Collaboration among federal departments and agencies is generally constructive and positive, but limited greatly by:

  • A shortage of uncommitted funds at the local federal level
  • Well-entrenched policy and program instruments
  • A lack of impetus from senior management in partner departments/agencies
  • A major focus on project expenditures as opposed to the longer-term strategic focus
  • A lack of clarity as to how the partner departments/agencies are intended to contribute

4.2.3 To what extent has the federal capacity to respond in a coordinated way to community urban Aboriginal needs increased as a result of the initiative?

Finding: Federal government capacity is enhanced because of the development of UAS steering committees in most communities that provide a focus for setting priorities and developing approaches to address urban Aboriginal issues. Capacity to respond effectively to address those issues is still limited substantially by the above-noted factors.

4.2.4 Do federal departments participating in the UAS have a common understanding of roles and responsibilities?

Finding: Roles and responsibilities on a day-to-day basis under the UAS are clear. The lead department is responsible for implementation of the UAS. Other partner departments/agencies attend federal committee meetings or conference calls to share information and coordinate activities related to the UAS where possible, and may attend steering committee meetings. What is unclear to the partner departments/agencies is how they are intended to contribute in a more meaningful way in the longer term.

Source:

Findings in this area are based on a review of local UAS documents describing the structures and processes in place and any planning that has taken place, and on interviews with federal, provincial and municipal government officials, Aboriginal members of steering committees, and representatives of other Aboriginal organizations in the communities.

Discussion:

In most communities the participating government departments from all levels, and Aboriginal representatives, have developed collaborative and constructive working relationships. They meet regularly for steering committee meetings and less formally to deal with issues and projects that require attention. The great majority of Aboriginal members of steering committees report that the federal lead departments in particular, and government members in general, appear to understand the nature and extent of urban Aboriginal issues and be making a genuine effort to move the initiative forward. In addition, they report that the local government participants appear to be serious about allowing the Aboriginal community to lead the initiative (within the confines of the terms and conditions of the initiative, which requires an oversight role for the federal government in terms of expenditures and in the broad direction of the Strategy).

There is one UAS community, Toronto, in which the above findings do not apply. In that city there has been considerable acrimony between various Aboriginal groups and the lead federal officials, and among various Aboriginal participants. There appear to be as many versions of events in Toronto as there are people to describe the events, but it is clear that the UAS has not been successful to date in establishing positive working relationships between the federal government and the Aboriginal community or among the various Aboriginal interests that have come forward to work on the UAS. There is a considerable lack of trust that the federal government is genuinely interested in allowing the Aboriginal community to lead the Strategy in any meaningful way, and from the perspective of federal officials there has not been a real effort on the part of participating Aboriginal organizations to set aside agency interests and work collaboratively.

Aside from Toronto, the findings with regard to the development of partnerships and collaboration are encouraging. However, at this early stage of the pilot project phase these relationships are still very much works in progress, and are seen as somewhat fragile due to a number of factors.

As noted earlier there are organizations within the Aboriginal communities that do not share the vision of the UAS as a collaborative strategy between governments and Aboriginal communities and believe that federal funds to address Aboriginal issues should be devolved to representative Aboriginal political organizations. There continues to be pressure from these organizations to link the UAS to broader issues of Aboriginal self-government and to deliver it through the established political organizations. The federal government's position on this is that the federal government already has a number of programs and initiatives that flow directly through established Aboriginal political organizations. In the urban context, there are a large number of Aboriginal organizations serving the needs of Aboriginal residents, and representing to various degrees the diverse Aboriginal communities in the cities. Therefore, the UAS is designed to be community-based to allow the diversity of interests to work together collaboratively among themselves and with governments.

In some communities there are members of Aboriginal political organizations sitting on steering committees and these members are participating under the existing model, but the tensions between these two visions are reported to be detrimental to the process at times.

The working relationships are also strained at times because of long-standing strained relations between government and Aboriginal organizations. Aboriginal interview respondents almost invariably pointed to the historic lack of trust that Aboriginal people feel toward mainstream government, and also to frustration with traditional government approaches to delivering Aboriginal programs and services that are heavily bureaucratic, and that maintain control and decision-making authority in the hands of non-Aboriginals, often with little transparency. Most respondents expressed concern that the UAS may turn into "just another government funding program" with all the hoops to go through and restrictions that prevent them from making a real difference in the Aboriginal community.

This concern is supported somewhat by the short-term nature of the UAS and the lack of knowledge of whether the UAS will carry forward, and in what form. There is an apparent contradiction between the intended strategic focus of the UAS on the one hand, and the pressure exerted by federal officials to ensure that the funding allocations are spent in the fiscal years for which they are allocated. This contradiction places federal officials in a difficult position in that they are seen on the one hand to be encouraging a long-term strategic, collaborative approach while on the other hand they are focusing to a large extent on the project funding element, and imposing the usual budgetary restrictions and pressures. Most Aboriginal members recognize the reasons for the contradiction and do not view the local federal officials as being duplicitous, but it places a negative light on the Strategy as a whole, and heightens suspicion that there may not be a genuine federal government commitment to a long-term, strategic and innovative approach to urban Aboriginal issues. In practical terms it also means that the scarce available time of Aboriginal members is often used in relation to project funding instead of strategic development.

Government Collaboration

Collaboration within the federal "family" of departments and with other levels of government has shown some similarities to the government-Aboriginal relationships. In general there is a positive, collaborative approach to the UAS on the part of participating departments at all levels of government. Program officers who work with the Aboriginal communities in whatever capacity their department's mandate determines are uniformly clear about the depth of need in urban Aboriginal communities, and the need for governments to work more collaboratively and effectively. Representatives of the federal departments that have the most direct program relevance to the UAS (Western Economic Diversification, Health Canada, Human Resources and Skill Development Canada and Social Development Canada, Heritage Canada and in some cases Justice Canada and Indian and Northern Affairs Canada) attend steering committee meetings or internal federal meetings on the UAS regularly, and support the Strategy. Other departments have not been involved to date. Provincial and municipal officials attend meetings regularly as well in most communities. Where specific project ideas are raised that relate to their areas of interest they participate in planning and try to find ways to contribute either with funding or in-kind assistance.

It is worth noting that provincial representatives interviewed for this evaluation were extremely positive about the idea of the UAS as a new, strategic, collaborative approach that offers an opportunity to break through some of the long-standing barriers to improving conditions for urban Aboriginal people. They expressed frustration at times at the pace of progress and the level of bureaucracy that exists with federal programs, but they were unequivocal in their support for the Strategy and its basic concepts. This stands out as important because it differs from the experience with some other federal initiatives at the community level, where there is a perception that the federal government initiates a program, brings short-term funds to communities, heightens community expectations in areas of shared jurisdiction, and then changes priorities and leaves provincial governments under pressure to continue funding projects that have been initiated. Provincial officials have become wary of new federal initiatives in areas touching on their areas of jurisdiction for this reason.

There is little such apparent wariness about the UAS on the part of the provincial officials interviewed. In part this is because the project funding component of the UAS is relatively small, and does not present a large risk that projects will be initiated that will have major implications for provincial funding. As well, however, it is reportedly because the need for improvement in urban Aboriginal communities is so clear, the implications for provincial and municipal economies is now recognized as being significant, and the status quo in terms of government action is recognized as being insufficient. Urban Aboriginal affairs have fallen between the cracks of acknowledged federal and provincial responsibility for many years, and provincial officials see the UAS as a vehicle to break that deadlock and bring about some meaningful change.

In short, then, there is considerable support at the working level in all levels of government for the UAS, and some valuable collaboration taking place. The collaboration is still developing, however, and as with the Aboriginal community there are real barriers to truly effective collaboration to address urban Aboriginal issues.

Barriers to Government Collaboration

One of those barriers is a lack of uncommitted funds at the local federal level and among provincial and municipal departments to allocate to the UAS. Since the UAS is intended to be primarily a strategic initiative and has a relatively small funding component, the leveraging of non-UAS funds is a critical element of any longer-term success. Even for the smaller UAS pilot projects being considered under the current phase, there are few resources available within existing federal, provincial or municipal program or operating budgets to contribute. Funds have been allocated already, often through multi-year agreements with third parties or programs with long-standing partnerships and funding procedures.

Beyond the question of contributions to UAS pilot projects, provincial and federal officials pointed to the existence of strong, well-entrenched program and policy instruments among participating government departments as presenting a major challenge to innovate ways of addressing urban Aboriginal issues. In a sense, UAS committees are forced to "think small" because there is no confidence that the status quo of fund allocations in major policy and program areas can be breached significantly. For example, in the area of education, funding is typically committed to an existing complexity of school boards, schools and training institutions, or targeted programs. Consideration under the UAS of what to do about urban Aboriginal education tends to be focused on the enhancement of existing Aboriginal student support programs, or consideration of a dedicated Aboriginal school, as opposed to a major rethinking of Aboriginal education in the urban setting that may involve complex changes to current delivery models, such as a revamping of curricula or specialized early childhood education for Aboriginal children.  [13]

A second barrier is the lack of impetus from senior management at the national level and at times at the regional level as well, with the federal government. As a rule the participating departments have not received directives to commit time and resources to the UAS. For the most part, federal regional action (aside from the UAS lead departments in each community) derives from on-the-ground knowledge of the extent of need in the urban Aboriginal community, and the commitment of individual officers, as opposed to direction from senior management. This reported lack of impetus from senior management limits the time available for the UAS, dictates that representation at UAS meetings tends to be by less senior officials, and means that action on the UAS tends to happen around the margins of department activities instead of being integrated in any substantial way.

One possible reason for the lack of senior management commitment among participating federal departments, and a factor in how local officials participate as well, is a lack of clarity about how they could contribute. It was noted earlier that there has tended to be a focus on the project-funding component of the Strategy because of pressures to ensure that the funds are spent in the fiscal year that they are allocated for, and the considerable amount of work involved in planning, decision-making and administration of project funds. One result of this is that the participating partner departments come to view the UAS as focusing largely on the project funding. When they examine how they might contribute, therefore, there is a tendency to focus on how the projects being considered fit into their mandate or specific programs, and whether there might be an opportunity for some additional funding or partnerships with existing projects. These opportunities are limited, as we have seen.

What appears to be missing at the officer level and, apparently, at a senior management level, is a clear understanding of the long-term, strategic nature of the UAS, and what that means for their departments. This message, if it is clear to UAS managers, does not appear to have been communicated effectively to partner departments.

The final factor influencing the effectiveness of collaborations and partnerships on the UAS relates directly to the communication aspect. The interviews in all participating communities pointed to insufficient strategic direction from the OFI on an ongoing basis. This is in part because of the pilot project focus that has tended to drive the initiative, in part because of the desire to maintain the community-driven approach to the UAS, and also because the UAS represents a departure from traditional government practice and is potentially very complex in its implications. Participants expressed the need to clarify exactly what it is they are trying to accomplish, what the possible mechanisms are, and what the full scope of the UAS might be. Better national-level communication is seen as needed to help maintain a clear vision for the UAS, and to continue to inform government and community participants as new members take part.

4.3 Coordination With Other Related Initiatives/Programs

Evaluation Question:

4.3.1 Has there been coordination with other initiatives for urban Aboriginal people?

Finding: There is coordination with other initiatives such as the National Homelessness Initiative and various Heritage Canada and Health Canada programs, but coordination is ad hoc and dependent largely on cross-over in committee memberships. The few attempts to institute more formal linkages with other initiatives have met resistance from the Aboriginal community.

4.3.2 Are there areas where UAS activities overlap with other government programs and services?

Finding: Overlap and duplication with other government programs does not appear to be a problem in the case of the UAS, in part because the UAS, by its nature, is a strategy to identify opportunities for more effective and coordinated use of public resources. Pilot project expenditures are by design targeted to innovative approaches that do not fit the funding streams of existing government programs. The UAS can be seen as a vehicle to identify overlap and duplication among government programs and determine the most effective ways to use those available resources in a coordinated way. The summative evaluation could identify cases of pilot project-specific overlap and duplication with other government programs.

Source:

Findings in this area are based on a review of local UAS documents describing the structures and processes in place and any planning that has taken place, and on interviews with federal, provincial and municipal government officials, and Aboriginal members of steering committees.

Discussion:

To date the major focus of the UAS in most communities has been on establishing steering committees and decision-making processes, and identifying projects on which to spend available UAS funds. Coordination with other initiatives has for the most part occurred through an overlap in committee memberships, and the informal shared communications that result from those overlaps. To the extent that other initiatives have representatives on UAS committees or are otherwise participating in the UAS, those other initiatives will have some knowledge and awareness of the UAS. This is occurring to varying degrees across the eight UAS communities, but it is fair to say from the interviews conducted for the evaluation that participants in major urban Aboriginal initiatives are likely to be aware of the UAS and have some knowledge of what it is about. These include in particular the Urban Aboriginal Homelessness (UAH) component of the National Homelessness Initiative (NHI), but also community renewal projects, education projects and training and skills development programs.

In addition, UAS project funding decisions will in many cases be made with an awareness of projects being considered under other initiatives, and some coordination is occurring in this area to ensure that the most appropriate initiative is used as the funding vehicle. The coordination is reactive rather than planned, but it nevertheless results in the avoidance of overlap and duplication and better overall planning and use of federal resources.

Efforts were made in several communities to establish a more formal link between the UAS and the Urban Aboriginal Homelessness initiative, but these were not successful. In Vancouver, for example, a well-established structure was already in place with representatives of Aboriginal service-providing agencies to establish community priorities related to housing and homelessness, and to decide on project funding. There was initially an interest on the part of federal officials in linking the UAS to this structure in order to avoid duplicating what already existed and duplicating the burden on Aboriginal members. However, Aboriginal community leaders were against the idea because they had experienced some difficulties with the bureaucratic requirements of the NHI, and preferred to deal with the UAS lead department in that community, Western Economic Diversification, which has less stringent requirements and more decentralized signing authorities than Human Resources and Development Canada (HRDC). They also wanted to be sure that they could control how the funds were allocated, and believed that this was more likely under the auspices of the UAS.

In Toronto, both the UAS and the UAH are administered by HRSDC, and the UAH officials were interested in establishing formal linkages or even administering the UAS together with the UAH. However, UAS officials in that department preferred to maintain a separate administration, and no significant collaboration took place. Because the UAS in Toronto never reached the point of funding projects before the steering committee process was disbanded, the issue of coordination of projects did not arise.

In Edmonton the Joint Planning Committee on Housing has an Aboriginal sub-committee that provides guidance on Aboriginal housing issues, and that sub-committee was originally considered as a possible committee for the UAS. However, that has not taken place, and there is now a newly formed (as of December, 2004) Aboriginal Review Committee functioning under the auspices of the Edmonton Housing Trust Fund (EHTF), which is charged with administering the UAS project funds. This committee is limited in function to the funding decisions, however, and work continues on the establishment of a steering committee with a broader mandate to address strategic considerations under the UAS.

The EHTF does provide a link between the UAS and the NHI because it is the community entity for the implementation of the NHI in the city. The link is thus established in the delivery mechanism and in community planning that goes beyond homelessness to look at a range of issues affecting Aboriginal people in the city.

The evaluation examined the issue of potential overlap and duplication between the UAS and other government programs directed to urban Aboriginal people. Through interviews with the full range of participants it was determined that the nature of the UAS minimizes greatly any risk of such overlap and duplication. The UAS is a strategy to identify opportunities for more effective and more coordinated application of public resources. It is not a program, and does not have a focused area of interest such as health, education or Aboriginal justice issues. Any pilot project expenditures are by design targeted to innovative approaches that do not fit the funding streams of existing government programs. If anything, the UAS can be seen as a vehicle to identify overlap and duplication among government programs and determine the most effective ways to use those available resources in a coordinated way.

There is the possibility that some UAS project funds could be allocated to projects that could have been funded under other existing government programs. The summative evaluation will want to review project expenditures to identify the extent to which the pilot projects resulted in innovate approaches or initiatives that could not be funded from other sources. Any potential project-specific overlap and duplication could be identified through such a review.

4.4 Single Window Approach, UAS Terms and Conditions

Evaluation Questions:

4.4.1 Have single contribution agreements and common terms and conditions for pilot projects brought efficiencies for community participants, and if so, what are the main areas of efficiency?

Finding: The UAS terms and conditions are viewed as flexible and sufficiently broad to allow most types of required expenditures, but the single-window approach itself does not appear to be bringing noticeable efficiencies to date. Delays in the final sign-off on project funding, in particular, are reported to be a problem.

4.4.2 Are participating federal departments satisfied that their information and accountability needs are being met through this "single window" approach?

Finding: There were initial concerns in participating departments that the use of the UAS terms and conditions by other departments, or even other programs within the lead department, would not meet accountability requirements. Some of these have been alleviated through consultation and clarification, but there remain some doubts, and few examples to date of the single window approach being used.

Source:

Findings in this area are based on interviews with federal government officials, and the survey of Aboriginal organizations.

Discussion:

At the outset of the initiative there was some optimism that the UAS terms and conditions, and the agreement with partner departments to allow for the application of UAS terms and conditions to funding from other programs, would reduce the level of red tape and reporting requirements that typically accompanies federal project funding. This was part of the overall effort to use the UAS as a test for streamlining the bureaucracy associated with horizontal, cross-departmental initiatives. It would mean that a project proponent could apply for UAS funding and funding from other federal programs under a single proposal, and have only one set of terms and conditions and reporting requirements to deal with.

Early indications are that while the terms and conditions of the UAS are viewed as quite flexible as far as the kinds of projects that can be funded, the benefits of the "single window" approach in relation to the terms and conditions are limited because the requirements of any participating departments, or programs within departments, must be built into any project involving joint funding. The funding recipient does not have to deal with multiple departments so there is a reported efficiency there, but proposal and reporting requirements may not allow for the kinds of efficiencies that were anticipated.

Thus far there have reportedly been few opportunities to test the use of the common terms and conditions, because there are few examples of other federal departments contributing to UAS projects. However, it is notable that even within HRSDC, efforts to use the common terms and conditions across programs were not seen as beneficial because the other program saw the need to maintain its own terms and reporting standards, and financial officers were reluctant to use the UAS terms and conditions exclusively because they had not been given instructions on how to do so.

In several communities it was noted that one positive result of the "single window" approach was that financial and program officers across departments had entered into a dialogue with a view to finding a way to make the idea work, so there is optimism in those communities that some benefits may be derived in the future.

The survey of Aboriginal organizations found that while specific elements of the project proposal process were seen as adequate (in the mid-range from positive to negative) in terms of fairness, timeliness, and the nature and extent of requirements for proposals, respondents found the overall UAS proposal process "about the same" or "worse" than other project proposal experiences. Additional comments provided by some survey respondents, and comments from Aboriginal respondents and local government respondents, indicate that the reason for this negative overall perception may be the reportedly lengthy delays in gaining approvals for project funding, once the projects were selected by the steering committees. It is acknowledged by OFI managers that the review and approval process in Ottawa has been a significant factor in those delays, and efforts are now being made to reduce the project approval turn-around time.

4.5 Resourcing of the UAS Relative to Requirements

Evaluation Questions:

4.5.1 Have federal government supports and resources assisted communities in urban Aboriginal planning and decision making?

Finding: Steering committees in all but one community are highly satisfied with levels of support provided by lead departments. However, the provision of adequate supports to UAS processes has required supplemental contributions, at times substantial, from lead departments. In addition, it was found that while the available resources were usually sufficient for broad planning and project funding purposes, they were not sufficient to allow for the level of community outreach, project development, relationship building and community development that would further the intended longer-term, strategic focus of the UAS. At least initially, a lack of program delivery expertise and resources at OFI caused confusion and delays at the community level in implementing the initiative. At OFI, administrative responsibilities were found to have drawn resources away from broader strategic, partnership-building functions.

4.5.2 Have federal government supports and resources assisted project proponents?

Finding: Supports provided by lead departments for project funding were found to be sufficient in terms of processing proposals in a fair and timely manner. However, resources have not typically been available for a more extensive project and proposal development function that may have strengthened projects and placed a greater emphasis on linking proposals to the longer-term strategic approach of the UAS.

Source:

Findings in this area are based on a review of local UAS documents, interviews with OFI and local federal officials, and resource allocation information provided by UAS regional coordinators.

Discussion:

The formative evaluation was asked to consider whether the resources allocated to the implementation of the UAS (as opposed to the project funding itself) was sufficient to enable communities to advance the initiative as intended. The focus of the inquiry was on the impact of resource levels on results, as opposed to an examination of resource allocations within communities and how the funds have been spent.

The inquiry has found that federal lead departments have been able to provide the supports necessary to establish the basic elements of the initiative such as initial community consultations, development and functioning of steering committees, federal and inter-government liaison, and project administration. However, to do so the lead departments have invariably had to supplement UAS budgets with contributions from their own departmental operating budgets, over and above what was allocated through the UAS.

The UAS allocations for the implementation of the initiative (not including pilot project funding), in the range of $300,000 per year for each province, cover the costs of salaries, travel and some expenses including training and meeting costs. Lead departments have contributed an estimated additional 20% to cover additional costs including staff time for related functions such as finance, administration and communications (or portions of the cost of UAS-funded staff), and other miscellaneous expenses. In Alberta and Saskatchewan the resource issue is a particular challenge because the provinces were allocated the same amount as the other provinces but they each have two participating communities. Saskatchewan in particular has experienced problems in this regard. That province reports salary expenditure close to double the UAS allocation for Regina and Saskatoon, without taking into account travel, meeting expenses and other costs associated with the UAS.

In the majority of communities the federal support that has been provided has been very well received and seen as an important contribution. Either directly or through a secretariat employed with UAS operating funds, lead departments prepare all the background documents for steering committee meetings and other community meetings, host the meetings (in most cases) and cover all expenses, attend the meetings and provide guidance and information as required, maintain records of proceedings, communicate with committee members and assist with any activities that result from decisions. In the case of the pilot projects, the lead departments (again either directly or through a third party) assist in developing the decision-making processes, assist in implementing the process, write up the project agreements and administer the contribution.

One area consistently seen as lacking is support for community development types of functions. There is a recognized need for more ongoing communication and consultation in the communities, and for the provision of greater guidance and expertise in the development of projects in keeping with community priorities (as opposed to simply responding to whatever project proposals come forward), in order to establish a more strategic direction for the UAS. These functions are not resourced at present.

In addition, the strategic elements of the UAS appear to be underdeveloped. This is in part due to time constraints and the focus on the project funding component. However, the resources available for the initiative at the community level are reported to have an impact here as well. Developing a long-term, strategic approach to addressing urban Aboriginal issues requires different sets of activities and different organizational structures and processes than are required for short-term community planning and project funding functions. It also may require different participants at the government and community levels to bring substantive expertise in priority areas. Certainly, it requires time and effort to develop these different functions, sell the strategy to potential contributors including senior officials in relevant government departments, and coordinate the implementation of any substantive initiatives that emerge.

The UAS is currently resourced, albeit insufficiently according to some coordinators, to implement the pilot project element of the UAS, but does not appear to be sufficiently resourced to tackle both the project funding and the broader strategic elements that are the core of the Strategy.

There are also resource-related findings at the national level. The evaluation did not examine OFI functions or resourcing, but interviews with federal officials at OFI and in the participating communities pointed to difficulties in Ottawa coping with the volume of work associated with the early implementation of the pilot project phase of the Strategy, and with the administration of pilot projects. Two officers were assigned to the UAS besides senior managers with responsibilities for the UAS and other files. Both were policy officers with little experience in program operations, and while the UAS was exclusively a policy initiative this was sufficient, but the OFI reportedly lacked the expertise and resources to handle the operational side of the Strategy as the pilot project phase was implemented. This caused delays and some confusion at the community level as officers there attempted to establish procedures and fit the UAS terms and conditions with their own departmental requirements.

At the national level, a repercussion of this resource deficit was that the strategic functions dependent on OFI did not receive sufficient attention. Little effort was devoted at the officer level to establish and develop working relationships with the partner federal departments at national headquarters, or with other levels of government and other interested national organizations. Periodic consultations took place at the senior management level, but this is acknowledged at OFI as having been insufficient to champion the UAS vision and develop the broader federal government engagement that was intended. OFI has begun staffing in key administrative and communications areas to enable the initiative to resume these functions.

As noted in section 4.4, the survey of Aboriginal organizations found that supports for the project proposal process were seen as adequate in terms of fairness, timeliness, and the nature and extent of requirements for proposals.

4.6 Capacity Building

Evaluation Questions:

4.6.1 To what extent has the capacity of the Aboriginal communities in participating communities to address urban Aboriginal issues been increased as a result of the UAS?

Finding: The main capacity building to date has been in developing trust and working relationships among those government and Aboriginal representatives involved in UAS committees. This "asset" is still a work in progress and fragile, but is viewed widely as a necessary step in what the UAS is trying to accomplish. In some communities capacity has also been developed through a planning process, enabling community leaders to identify priorities and begin to establish longer-term community assets. There is a recognized need to increase substantially the pool of Aboriginal community members with the experience and expertise to participate effectively in the kind of long-term, strategic process envisioned by the UAS, and to contribute to the development of new and innovative approaches to service delivery and government support on priority urban Aboriginal issues. This capacity issue is not being addressed significantly at present.

4.6.2 To what extent has the capacity of individual Aboriginal organizations in participating communities been increased as a result of the UAS?

Finding: With the relatively small number of projects funded to date that are directed to individual service agencies or service agency partnerships, service delivery capacity cannot be said to have been enhanced appreciably thus far. For the most part the UAS has not offered resources for capacity building in individual organizations, such as project and proposal development. Services have been provided in some communities to help refine project proposals that have been formally submitted and have been approved in principle.

4.6.3 Are there indications that the UAS has succeeded in leveraging partner resources?

Finding: Data on project expenditures were only partially available at the time of this report. Those partial figures indicate that the UAS was succeeding in attracting some investments from other sources. However, it is not clear from this evaluation whether the UAS leveraged these other investments, or if they may have taken place in any case without the UAS.

Source:

Findings in this area are based on a review of local UAS documents describing the structures and processes in place and any planning that has taken place, and on interviews with federal, provincial and municipal government officials, Aboriginal members of steering committees, and representatives of other Aboriginal organizations in the communities.

Discussion:

Capacity building is a concept that is used to describe a variety of different aspects of community and individual development. For the UAS, capacity building is an objective that takes a broad view—to help provide Aboriginal communities with the tools they need to flourish and ultimately to close the gap between themselves and the non-Aboriginal population. Within that broad framework there are elements of capacity building that could include:

  • Developing community consultation and decision-making structures and procedures in whatever form, to enable Aboriginal communities to plan for their future and take action based on a long-term vision and well-founded strategic planning;

  • Developing the planning and consultative expertise of individual community leaders, so there is a wider pool of individuals available in the communities;

  • Expanding project development skills in the community so initiatives can move forward with a solid financial and management foundation and the required linkages with other related initiatives;

  • Developing service delivery expertise;

  • Developing research and analytic skills; and,

  • Building appropriate community assets (facilities, services, infrastructure, foundations, ceremonies).

The evaluation has found that the main capacity building to date has been in the first area listed above. As noted earlier, there has been notable progress in all but two communities in establishing steering committees to oversee the UAS and in developing trust and working relationships among those government and Aboriginal representatives involved in the UAS committees. There is also widespread (but not unanimous) support among both government and Aboriginal participants for the idea of the UAS, and an apparent willingness to deal with the complexities and bureaucratic hurdles and continue to move forward.

This is no small achievement given the strong perceptions in the Aboriginal community that governments have not had Aboriginal interests at heart in previous initiatives, and the abiding lack of trust that pervades the relationship. The time and effort associated with building this aspect of capacity can be considerable, especially considering that most participants, including Aboriginal representatives, are volunteering their time over and above their other responsibilities.

In several of the UAS communities capacity has been expanded as well through a community planning process associated with the UAS. Most communities already had an Aboriginal community plan that had been developed through the UAH, but these plans are typically targeted to housing and other homelessness related issues, whereas the UAS is broader in scope. Where communities have succeeded in developing a broad urban Aboriginal community plan, this can be seen as enhancing the community's capacity to address urban Aboriginal issues.

In Calgary and to some extent in Winnipeg as well, UAS planning is being undertaken in a context beyond the UAS itself. The Calgary Urban Aboriginal Initiative (CUAI) has established a model that includes a large committee of Aboriginal service providers representing different substantive "domains", representatives of Aboriginal political organizations, an elder, an Aboriginal youth member, members from the three levels of government and other members at large from the community. A smaller executive committee oversees day-to-day operations. The detailed planning and recommendations for action (projects, initiatives, research) takes place in the individual domains (e.g. health, education, justice, housing, employment), where organizations and individuals who are active in the area and have expertise to contribute meet to identify priority issues and develop new approaches to addressing those issues. The UAS is one funding program that contributes to projects emerging from these domains. While the UAS cannot take full credit for these developments in Calgary, efforts in that city to implement the UAS since 1998 did contribute directly to the CUAI. The CUAI can certainly be seen as an important community asset and an example of enhanced community capacity arising in part from the UAS.

In Winnipeg the UAS is also subsumed under a broader urban development strategy for the city. An Aboriginal Partnership Committee has been formed under the umbrella of the Winnipeg Urban Development Agreement, to recommend strategies and projects to address Aboriginal issues in Winnipeg, and to ensure that the Aboriginal community has input into broader initiatives in the community. The Committee is still at an early stage. But there is optimism among members that it will provide a basis for a planned approach to addressing urban Aboriginal issues.

The other communities have also undertaken smaller scale planning processes in order to set priorities as a basis for selecting projects to fund under the UAS. In Vancouver, that process led to a decision to forego a second RFP call, and instead develop four initiatives designed to build sustainable capacity and further community development. Working groups have been struck to implement these initiatives. In Thunder Bay the UAS committee identified the alleviation of child poverty as their community priority. It is a single, multi-dimensional project to test new approaches to service and program delivery to address Aboriginal poverty issues in Thunder Bay.

In a number of communities the Tamarack Institute was engaged to lead community sessions on capacity building. These sessions were viewed positively where they took place, and were reported to have contributed to some of the early work in bringing Aboriginal members together to establish committees, identify community priorities and otherwise move the UAS process forward.

Other capacity building initiatives have also been undertaken, including a workshop in Vancouver to assist community agencies to develop project proposals, and one-day sessions in Saskatchewan to assist agencies in conducting project evaluations for their own purposes and to meet evaluation requirements in their contribution agreements.

These are examples of one type of capacity that the UAS has contributed to. In all participating communities, though, it is recognized that other aspects of capacity are low and in need of attention. In terms of continuing to develop the UAS as a strategic initiative, there is a concern that the individuals currently sitting on the UAS committees will not be able to continue to participate indefinitely, and that there are few other community members with the skills and experience to take their places, much less to contribute to an expansion of the UAS into substantive working groups or other such mechanisms to make advances in specific issue areas. This aspect of capacity has yet to be addressed in any direct way and is seen as a potential barrier to progress.

In terms of enhancements to services and facilities in the participating communities, it is too early in the pilot project phase to determine the extent to which sustainable assets are being put in place with the assistance of the UAS. The purpose of UAS funding was, after all, to enable communities to test innovative approaches to addressing urban Aboriginal needs, and not to fund new services or facilities. As the projects progress and are completed, there will be an opportunity to assess the extent to which they are resulting in approaches that can make a sustained contribution.

Beyond the pilot projects themselves, the UAS has the potential to make a much greater tangible contribution to community assets through its strategic focus. If this approach can lead to effective collaboration between Aboriginal communities and all levels of government in specific service areas, there may be considerable gains in community capacity. There is no evidence to date of any such gains.

Figures to date on project expenditures are only partially available. Figures for projects receiving multi-year projects are included but the figures do not include new projects that have been or will be identified for fiscal year 2005/06. These figures do not offer evidence of actual gains for communities because it is too early to assess results. However, they indicate that investments are being made through the UAS that offer opportunities for capacity building. They also indicate that UAS projects are attracting investments from other sources, but this evaluation did not examine the extent to which the UAS actually leveraged investments that would not otherwise have taken place, or simply contributed to some projects that were already being planned. The summative evaluation will need to examine this issue.

Urban Aboriginal Strategy Pilot Projects Approved (fiscal year 2003-2006)
Community Number of UAS-Funded Projects Total Cost of All Projects Total UAS Contribution Average UAS Contribution Per Project
Vancouver 18 $3,090,414 $2,814,991 $156,388
Edmonton 2 $2,577,832* $2,280,000 $1,140,000
Calgary 7 $5,124,886 $1,349,647 $192,807
Regina 8 $1,642,335 $1,087,682 $135,960
Saskatoon 10 $922,082 $847,082 $84,708
Winnipeg 39 $8,598,577 $3,039,496 $77,936
Thunder Bay 2 $1,600,000 $750,000 $375,000
Toronto 7 $467,660 $359,660 $51,380
Total 93 $24,534,564 $12,860,884 $806,459

* includes $297,832 in WD funding that was provided under UAS Terms and Conditions

4.7 Enhancements for Urban Aboriginal People in the Eight Communities

Evaluation Questions:

4.7.1 To what extent has the UAS resulted in new or enhanced services for urban Aboriginal people in participating communities?

Finding: It is premature to expect measurable results in this area. These results are expected to take place to a limited extent through innovations developed in pilot projects, but primarily through the broad strategic UAS approach and ensuing actions in specific service areas. Such results will be evident only in the medium to long-term.

4.7.2 To what extent has the UAS narrowed the gap between urban Aboriginal people and the mainstream population?

Finding: As above, there is no expectation that the UAS will have led to a narrowing of the gap at this early juncture. The evaluation did note, however, that there appears to be a misalignment between apparent expectations (as delineated in initiating documents and the UAS RMAF) and realistic opportunities to affect change in the short to medium-term. There appears to be insufficient recognition of the long-term nature of the UAS building process, as a necessary precursor to meaningful and sustainable change in service delivery and in life opportunities and circumstances for urban Aboriginal people.

Source:

Findings in this area are based on a review of project expenditure data and interviews with federal, provincial and municipal government officials, Aboriginal members of steering committees, and representatives of other Aboriginal organizations in the communities.

Discussion:

As noted in the previous section it is too early in the pilot phase to assess the results of the projects being funded or longer-term results such as improvements in the circumstances of urban Aboriginal people. While projects terms of reference require reporting on results in broad terms, project reports are not yet available to provide a basis for assessing whether they contain sufficient information for project evaluation purposes.

There are, however, two findings that are worth noting that relate to the longer-term objectives of the UAS. While the logic model for the UAS identifies narrowing the gap between urban Aboriginal people and the mainstream population as an expected outcome, few people involved in the initiative expect to see significant measurable results of that sort in the current funding period. That may be obvious given the complex and intransigent nature of many of the issues facing Aboriginal people in Canada, but it is worth raising here because it has a bearing on the question of what are realistic expectations for the UAS, and what time frames might be appropriate to use in mapping out the future of the Strategy.

The finding of the evaluation is that there do not appear to be any commonly shared ideas about what are reasonable expectations for the UAS to accomplish in the current funding period, or about a timeframe for different stages of development. This finding emerged from discussions during interviews in response to questions about the progress that had been made to date.

A related finding is that the design for the current funding period may have underestimated the time required, and the work involved, in establishing even the most basic components of this strategic initiative–the community-based steering committees that link Aboriginal communities with government partners to address urban Aboriginal issues. We have identified in this report some of the modest successes that have been achieved, and also the substantial barriers that still exist. In the view of many federal officials involved in the UAS at the community level, the expectations from Ottawa may be unrealistic. Those expectations are reflected in part in the RMAF and other descriptive documents, but perhaps more importantly in the pressures to spend annual funding allocations whether or not the structures and procedures are in place to do so in a planned and deliberate manner in keeping with the strategic focus of the UAS. In several communities there has been a deliberate decision to not fund projects even if it meant leaving some of their allocation unspent, in order to maintain the focus on establishing effective bodies and mechanisms to advance the Strategy.

4.8 Overall Assessment of UAS Model

Evaluation Questions:

4.8.1 What are the strengths and weaknesses of the UAS model?

Finding: The evaluation noted the following strengths and weaknesses of the UAS model:

Strengths:

  • Flexibility to adapt to local circumstances–even to the extent of being subsumed under a larger community initiative (Calgary and Winnipeg);

  • Flexibility of terms and conditions;

  • Community-based approach (limited still in some communities, but recognized as the best way to proceed by most Aboriginal leaders and all levels of government);

  • Strong provincial support aided by small project funding budget;

  • Information-sharing and a degree of coordination among federal departments;

  • Appreciation in Aboriginal communities of what the UAS represents in theory–a recognition that existing government approaches are insufficient, and the start of a new way of approaching urban Aboriginal issues involving better collaboration and self-identification of priorities and actions; and

  • The idea of the UAS as a focal point for collaboration and long-term planning on urban Aboriginal issues is seen as important and a real innovation in thinking

Weaknesses:

  • Annual budget pressures and short-term horizon of UAS funding;

  • Contrast between supposed strategic focus and the reality of project funding pressures;

  • Lack of effective communications with the Aboriginal communities at large, and lack of an ongoing, two-way feedback mechanism with the broader communities

    • Weak visibility of the UAS
    • harm to public perceptions;

  • Lack of clear terms of reference for make-up and functioning of steering committees;

  • Lack of ongoing strategic direction from OFI beyond the initial meetings and documents

    • Variation in interpretations of purpose, scope, operations of the UAS
    • Lack of common vision among participating government and community representatives as to what the UAS is, or could be;

  • Almost complete lack of direction from National Headquarters (HQ) senior managers in participating federal departments, and no indication that the UAS is viewed as a priority;

  • Lack of flexibility in the resourcing of urban Aboriginal related programs and policies by other federal and provincial departments; and

  • Lack of a commonly understood longer-term vision for the UAS based on a realistic assessment of the building blocks that need to be put in place, and appropriate timeframes for those building blocks.

4.8.2 What challenges have been encountered in implementing the UAS, and how have these been addressed to date?

Finding: Findings on the challenges faced in implementing the UAS have been discussed already in the context of findings in other issue areas. They indicate that while there have been some encouraging developments in all but one of the eight communities, progress has been hampered by the complexity of the task, the challenges of building new working relationships, and systemic and bureaucratic barriers.

4.8.3 Are there alternative delivery approaches that might work better?

Finding: The strengths of the UAS are in fact primarily in the delivery model itself. Implementing the model as it was intended has proven to be a challenge, but there is no evidence thus far that the model is unrealistic because much of it remains untested. The evaluation has found a number of areas requiring remedial action, but these are directed to specific elements or activities of the Strategy, rather than the overall delivery model.

Source:

Findings in this area are based on the full range of evaluation methods.

Discussion:

There is wide (although not unanimous) agreement among the full range of people interviewed and surveyed for the evaluation that the current UAS model is, potentially, a very effective model with which to address urban Aboriginal issues. The findings listed above have been discussed already in the context of findings in other issue areas. They indicate that while there have been some encouraging developments in all but one of the eight communities, progress has been hampered by the complexity of the task, the challenges of building new working relationships, and systemic and bureaucratic barriers. The strengths of the UAS are in fact primarily in the model itself, and in the wide support it is receiving in government and in Aboriginal communities (with the exception of some Aboriginal political organizations, as discussed earlier). Implementing the model as it was intended has proven to be a challenge, but there is no evidence thus far that the model is unrealistic because much of it remains untested. If more effective alternative approaches are found at this early stage of the UAS, they are most likely to reflect individual elements of the initiative that may be developed in specific communities that may be applicable in other participating communities, rather than a different overall approach.

4.9 Data Collection for a Future Summative Evaluation

Evaluation Questions:

4.9.1 What information/data on UAS planning and projects is currently being collected in communities for UAS reporting purposes?

Finding: Lead departments are maintaining documentation relating to the structures and processes that have been put in place, the implementation of those structures and processes (including minutes of meetings and all resulting reports and decisions), the projects being funded, and all financial transactions related to the UAS.

4.9.2 Referring to the summative evaluation strategy, which evaluation questions and indicators does the information/data that is currently being collected pertain to, and is it sufficient for evaluation purposes in terms of validity and reliability? Are there evaluation questions/indicators that will not be addressed sufficiently through current information/data collection mechanisms, and if so, is it feasible for communities to collect that information/data?

Finding: The great majority of administrative information/data for the summative evaluation that relates to the early outcomes of the UAS is currently available in a form that will be sufficient for the evaluation. There are two exceptions. First, information on non-UAS public expenditures on urban Aboriginal issues at all levels of government is not currently available in any systematic or comprehensive form. If it is deemed necessary for the summative evaluation, the collection of this information will need to be undertaken prior to the evaluation itself, since it will require some time to compile. Second, the summative evaluation will require a full accounting of the resources used for the UAS and how they have been used, including resources provided under the UAS budget and resources contributed by the lead departments in the participating communities. If this is deemed important for the summative evaluation a systematic approach and common categories of expenditures will need to be identified and put in place.

As is to be expected, data/information for the medium and longer-term outcomes will derive primarily from methods such as interviews, surveys, case studies and expert panel reviews. There are some performance indicators in the current evaluation strategy relating to these longer-term outcomes that are not currently being tracked. A decision will have to be made regarding realistic expectations for the UAS in the current funding period, the appropriate performance indicators to apply, and therefore what data collection mechanisms will need to be put in place in preparation for the summative evaluation.

Source:

Findings for this section are based on the draft UAS Evaluation Strategy document developed in the early stages of the formative evaluation, interviews with UAS coordinators and staff, and an e-mail request for specific information about data availability sent to the UAS coordinators.

Discussion:

The review of data for evaluation indicators began with an initial assessment of each performance indicator in the Table of Issues, Questions, Indicators and Methods presented in Appendix A of this report. Each indicator and method was examined from the point of view of whether the required information would be readily available through existing sources, would be available but require collection and compiling, or would not be available unless a recording mechanism was put in place.

In some cases it was clear that the information/data would be readily available, especially for the outputs and early outcomes of the initiative. This is true, for example, of Aboriginal policy documents produced by OFI and partner departments, data on UAS project expenditures, copies of any community plans and other planning documents and reports produced under the UAS at the community level, membership lists of UAS committees, international literature on urban Aboriginal issues, and individual project budgets and types of projects funded.

In other cases it was clear that the information/data would not be readily available because it required evaluation methods that would not be expected to take place in the normal course of the ongoing performance management of the initiative. These include information to be derived from interviews with the full range of participants, any surveys to be conducted, case studies and expert panel reviews. The sources of information for these methods will be readily available, and while there may be barriers to the effective implementation of some methods, there is no apparent need for early preparation of data collection mechanisms prior to the summative evaluation.

From the initial review a number of specific types of information/data were identified for which the availability was in question. These items were pursued during interviews and included in a request for information from the UAS coordinators. They are identified and discussed below.

  1. One of the issues of interest to OFI is the extent to which non-UAS funds are being used to help fund UAS projects. This information will indicate the extent of partnering that is occurring on UAS projects, the extent of engagement with other federal departments, other levels of government and other sectors, and the possibility that the UAS is succeeding in leveraging funds from other sources. For the summative evaluation it will be useful to have information, for each UAS project, on the other funders that are contributing, and how much they are contributing. The finding is that this information will be readily available in all communities.

  2. It is known that public funds, including federal government funds, are being directed to facilities and services for urban Aboriginal people through existing programs at all government levels. There is an interest in understanding the nature and extent of these investments, and the potential pool of resources that could potentially be reallocated based on a better coordinated strategy such as could emerge from the UAS. For the summative evaluation we need to know whether there is information available for the participating communities about the overall amounts of government (federal, provincial, municipal) funding being directed to urban Aboriginal issues, and what the sources of investments are.

    Feedback from the eight participating communities indicates that such information is not currently available in any systematic or comprehensive form. Efforts are underway in several communities to pull together some information in this regard for planning purposes, but if it is deemed necessary for the summative evaluation the collection of this information will need to be undertaken prior to the evaluation itself, since it will require some time to compile through communication with potential sources, and with the work of establishing the Aboriginal portions of programs that are not exclusively for Aboriginal people.

  3. The summative evaluation will require a full accounting of the resources used for the UAS and how they have been used. This would include resources provided under the UAS budget and resources contributed by the lead departments in the participating communities. The purpose would be to assess the adequacy of resource levels and the allocation of resources in relation to the results of the initiative. At present some of that information is readily available, but some elements are available by estimate only (for example, figures on non-salary expenses that have been covered by a mix of UAS and lead department operating budgets). If this is deemed important for the summative evaluation a systematic approach and common categories of expenditures will need to be identified and put in place.

  4. Several of the proposed summative evaluation methods involve a review of information on the outcomes of UAS-funded projects. This is of interest especially because the pilot projects are intended to test innovative approaches to addressing urban Aboriginal issues. There is an interest, therefore, in having available any information such as project reports, research reports, evaluations, guidebooks and manuals, conference reports or any other such information that derive from the UAS projects. Feedback from the lead departments of the participating communities indicates that all such information will be readily available through the lead departments as a standard expectation under the project funding agreements.

Data/information for the medium and longer-term outcomes will derive primarily from methods such as interviews, surveys, case studies and expert panel reviews. There are some performance indicators in the current evaluation strategy relating to these longer-term outcomes that are not currently being tracked. These indicators include:

  • The extent to which more training/skills development programs are in place as a result of the UAS;
  • increased satisfaction among Aboriginal organizations with the quality of government services in key service areas;
  • increased satisfaction among individual clients with government services to Aboriginal people, in key service areas;
  • the extent to which service delivery managers and staff report improvements in the quality of services, that are attributable at least in part to improved coordination through the UAS;
  • the extent to which clients of UAS-funded services have improved their living circumstances in ways related to the services being received;
  • the extent to which Aboriginal organizations report that the UAS has contributed to overall improvements in the life circumstances of Aboriginal people in participating communities, relative to the mainstream population.

The first of these indicators would be relatively easy for UAS managers to monitor through an ongoing tracking mechanism, integrated into the effort to track non-UAS expenditures on urban Aboriginal issues. The remaining indicators, however, relate to more complex issues and would require some concerted effort to define terms and more specific outcome elements, establish populations and possible sampling frames, and design research strategies. They assume that there will have been some considerable progress in improvements to service delivery through the UAS, and the findings of this formative evaluation suggest that this assumption may be optimistic. A decision will have to be reached about what are realistic expectations for the UAS in the time period leading up to the summative evaluation, and what indicators will therefore be appropriate. On that basis, decisions can be made about any required data collection strategies prior to the evaluation itself. In addition, there will need to be a review of project reports to assess the extent to which they will contribute to an evaluation of the success of the projects in meeting UAS objectives.



5. Conclusions

Implementation of the pilot project phase of the UAS began in 2003, with communities planning their longer-term approaches to using the UAS as a vehicle for change, and working to make best use of the available project funding. This formative evaluation was conducted to assess early progress, ensure that performance measurement strategies and activities are on track, and identify any improvements that should be made on the basis of the experience thus far.

The previous section presented the findings of the evaluation in each of the issue areas in its terms of reference. Here, the report draws those findings together into a set of conclusions about UAS progress to date and areas that require remedial action.

The main focus of the evaluation was in assessing whether the pilot projects, and the pilot project phase of the UAS as a whole, is proceeding as intended, and identifying any problems with implementation that need to be addressed. There are a number of conclusions in this regard.

  1. There is widespread support for the UAS in government and among most Aboriginal participants, although some Aboriginal political organizations take exception to the UAS model because it does not devolve control of the Strategy and the funds to what they see as representative Aboriginal organizations. The support is based on the strategic and innovative aspects of the UAS, the fact that it is community-driven, and the fact that it offers an opportunity for improved collaboration with government.

  2. The implementation of the Strategy presents some serious challenges, and continued support for the initiative will depend on the actions taken to respond to the barriers identified in this report.

  3. At the national level the UAS has not adequately fulfilled the need to "sell" the initiative on an ongoing basis to partner federal departments and agencies, one result being that the Strategy has not benefited from senior management support from those departments and agencies.

  4. Also at the national level, the UAS was very successful in the initial stages in garnering support for the concept and the delivery model, but this has since diminished because the vision of the UAS has not been reinforced, and partners have not received sufficient guidance as to how they can participate effectively. There is a lack of a commonly understood long-term vision for the UAS among participating governments and Aboriginal representatives, and a lack of practical guidance for implementing the strategic vision and what the appropriate time frames might be.

  5. At the community level, the strategic vision of the UAS has faded into the background somewhat under the pressures to spend the available pilot project funds in the fiscal years to which they were allocated, and due to the heavy workload involved in the initial implementation.

  6. Challenges in establishing Aboriginal steering committees that adequately represent the communities, in developing terms of reference and guidelines for the committees, and in maintaining strong linkages with the broader communities, have resulted in a lack of engagement with the broader Aboriginal communities, and this poses a threat to the success of the UAS.

  7. Part of the strategic vision of the UAS is to develop innovative approaches to address specific urban Aboriginal issues, and then to leverage resources and even rethink entirely how specific sectors are served by government at all levels. There is support in theory for this approach at least at the officer level within government and among most Aboriginal observers, but are major practical barriers to this vision. Existing program and policy regimes in federal and provincial government departments are usually well-established and have a lengthy history in terms of their internal practices and working relationships with clients. Very few resources are currently unallocated and therefore available to contribute to UAS-based initiatives. There will be considerable bureaucratic and administrative resistance to face in trying to extract resources from the dedicated budgets of well-established programs, but substantial change is unlikely to occur unless this happens.

A second area of inquiry was with regard to the UAS approach to performance measurement and the collection of information for that purpose.

  1. Aside from a small number of specific information requirements, the UAS is well-situated in terms of available administrative information/data to contribute to a summative evaluation. There remain important decisions to be made about what are realistic expectations for the UAS in terms of its longer-term outcomes in the current funding period. Once this is done, appropriate performance indicators will need to be confirmed, and mechanisms put in place to collect and data/information that is required as a basis for the summative evaluation. In addition, pilot project reports will need to be reviewed to assess the extent to which they provide sufficient information to enable evaluators to assess their contribution to UAS outcomes.

The formative evaluation also looked at whether the pilot projects, and the initiative as a whole, were making progress toward the achievement of desired outcomes.

  1. The evaluation found that encouraging progress has been made in establishing working partnerships with members of Aboriginal communities in most of the participating cities, and in establishing collaborative working relationships across levels of government and within the federal family of departments and agencies. However, these partnerships are still in development and are fragile in some cases.

  2. There are historic barriers to the further development of collaboration with Aboriginal communities that need to be overcome, and some bureaucratic barriers that inhibit creative development of the strategies that the UAS envisions.

  3. There are also a number of systemic barriers within the federal government that will need to be overcome if the UAS is to advance to the extent envisioned.

  4. The evaluation found that it was premature at this point to assess progress toward the longer-term outcomes of the UAS because of the time required to develop the initial stages of the initiative and the lack of completed projects upon which to base an examination of results.

The evaluation also examined the issue of overlap and duplication of the UAS with other government programs.

  1. Overlap and duplication with other government programs does not appear to be a problem in the case of the UAS because the UAS, by its nature, is a strategy to identify opportunities for more effective and coordinated use of public resources and because the pilot project expenditures are by design targeted to innovative approaches that do not fit the funding streams of existing government programs. The UAS can be seen as a vehicle to identify overlap and duplication among government programs and determine the most effective ways to use those available resources in a coordinated way.

  2. The summative evaluation should include a mechanism identify cases of pilot project-specific overlap and duplication with other government programs.

Finally, the formative evaluation examined the resourcing of the UAS, and whether the UAS had leveraged partner resources.

  1. The evaluation found that while the available resources were usually sufficient for broad planning and project funding purposes, they were not sufficient to allow for the level of community outreach, project development, relationship building and community development that would further the intended longer-term, strategic focus of the UAS.

  2. Steering committees in all but one community are highly satisfied with levels of support provided by lead departments. However, the provision of adequate supports to UAS processes has required supplemental contributions, at times substantial, from lead departments.

  3. At least initially, a lack of program delivery expertise and resources at OFI caused confusion and delays at the community level in implementing the initiative. At OFI, administrative responsibilities were found to have drawn resources away from broader strategic, partnership-building functions.

  4. Data on project expenditures were only partially available at the time of this report. Those partial figures indicate that the UAS was succeeding in attracting some investments from other sources. However, it is not clear from this evaluation whether the UAS leveraged these other investments, or if they may have taken place in any case without the UAS.

These conclusions summarize the gains that have been made to date under the UAS, and the strengths of the existing model. They also point to some significant barriers to success that will need to be addressed. The findings suggest that there is an opportunity for the UAS to offer an important, innovative approach to resolving problems in urban Aboriginal communities, problems that to date have appeared to be intractable. They also suggest that failure to address the barriers will greatly limit the value of the UAS, and may undermine the support the Strategy has received thus far. The next section sets out recommendations to address those barriers, for the consideration of UAS managers.



6. Recommendations

Based on the conclusions presented above, evaluators offer the following recommendations for consideration by UAS managers. The recommendations do not imply that the suggested actions should necessarily be undertaken with existing resources. Some of the recommendations may require resources beyond what has currently been allocated for specific functions.

6.1 Recommendations Related to Strategic Direction

  • OFI should reaffirm the purpose and scope of the UAS, and the long-term vision for the Strategy.

The reaffirmation should reflect the fact that the UAS is primarily a strategy, and that the legacy that the UAS wants to leave is a capacity for Aboriginal communities and governments to work together in a strategic, collaborative way to address urban Aboriginal issues. The reaffirmation should also make reference to the role of the pilot project funding, and offer clearer guidelines about how the projects are intended to feed into the broader strategic vision.

  • OFI should, as soon as possible and to the extent possible, issue a clear statement of long-term federal government commitment to address urban Aboriginal issues.

Ideally, the statement would embody ongoing support for Aboriginal community strategic processes at the broad level such as the current steering committees, and just as important at the "sector working group" level, where the specific, targeted strategic planning work will likely take place.

  • OFI should undertake to establish a well-defined set of expectations for the UAS including specific time frames, and a set of guidelines for implementing the strategic vision according to those timeframes.

The guidelines should not unduly restrict communities in the paths they choose to implement the Strategy, but they should establish clearly the kinds of directions that are in keeping with the UAS vision, and the kinds of mechanisms that are seen as most likely to succeed. The guidelines should include recognition that the collaborative, Aboriginal community-driven committee process that is being fostered in the communities is a critical stepping stone that should be allowed to develop at a moderate pace if that is necessary.

6.2 Recommendations Related to Communications

  • OFI should develop a national communications strategy to convey the above-described messages effectively.

Part of the strategy should be internal to UAS participants, directed to UAS coordinators, government partners at the national and local levels, and steering committee members. A second part should be targeted externally, mainly to the broader Aboriginal communities but also to the general public. The established communication approach should include on-going delivery of those messages throughout the initiative to counteract the forces that will tend to steer the initiative in other directions, such as an undue focus on project funding or approaches that accept the barriers identified in this report as being insurmountable.

  • OFI and local UAS coordinators should establish guidelines and suggested approaches for local communications strategies in participating communities.

The main purpose of the local communication strategies would be to engage the broader Aboriginal communities through improved awareness and opportunities for participation, and to make the existing planning and decision-making structures and processes much more transparent than they are at present.

6.3 Recommendations Related to Barriers

  • OFI should work to strengthen HQ and Regional Senior Management responsibility and accountability for the UAS among partner federal departments and agencies.

This work should rely heavily on the renewed vision and the reaffirmation referred to above. It should include clear guidance in practical terms about ways in which partners can contribute nationally and at the community level. As specific strategic initiatives are developed at the community level, senior managers of relevant departments should be informed of those developments and provided with guidance about how best to contribute.

  • OFI and regional coordinators should continue to work with Aboriginal political organizations to encourage their participation in the Strategy and to maximize the integration of UAS initiatives with broader initiatives in Aboriginal communities including Reserve-based ones.

  • OFI should work with regional coordinators to develop tools to assist community steering committees.

The tools could include standard terms of reference, guides for procedures, conflict of interest guidelines, guides for strategic planning, and other tools identified by regional coordinators. The tools should be designed so as to encourage a standard approach to the extent possible, while allowing sufficient flexibility to be useful in all participating communities.

  • Regional coordinators should work with local steering committees to emphasize the strategic nature of the UAS, and to make strategic planning and the setting of priorities the focus of their efforts.

  • OFI, in partnership with senior management at partner federal and provincial departments and agencies, should establish an approach to identify and address systemic barriers to the successful advancement of the UAS.

In particular, OFI should develop an approach to address:

  • resistance to change in programming and fund allocation approaches in well-established policy and program areas;
  • Treasury Board mandated financial restrictions on how UAS funds can be allocated across fiscal years. This applies to agency-specific reallocation for UAS projects and year-over-year allocations of UAS project funding.

6.4 Recommendations in Preparation for a Summative Evaluation

The current plan is for the conduct of a summative evaluation of the UAS by early 2006, in order to provide managers with information to support an anticipated Cabinet Submission later that year. However, the findings of this formative evaluation suggest that most participating communities may not be at a stage where a summative evaluation will appropriate, because few projects will have been completed sufficiently early to allow for the assessment of results, and because the strategic elements of the UAS are still very much under development. The inclusion of four new communities in 2004-2005 has further complicated planning for the summative evaluation. Given these realities, evaluators make the following recommendations for planning a summative evaluation.

  • OFI should consider delaying the conduct of a full summative evaluation for several years.

The delay would be in order to have sufficiently broad information on the results of projects, to give communities more time to develop their strategic approach, and to allow for development in the four new communities.

  • In the mean time, OFI should conduct an interim evaluation.

The interim evaluation should examine results to date (including project results where they are available), assess progress in the four new communities, and examine the extent to which the findings and recommendations of this formative evaluation have been acted upon, and to what effect.

  • For the interim evaluation, OFI should develop a Framework that sets out the issues to be addressed, the methods to be used, and a suitable time frame to ensure that managers have information for any upcoming Cabinet Submission, and a budget.

It should use as a starting point the draft Evaluation Strategy developed at the outset of this formative evaluation, and integrate the relevant information from this formative evaluation.

  • UAS managers should review the current evaluation strategy in light of the findings in this report, and establish realistic expectations for the achievement of UAS outcomes in the current funding period. Based on that, managers should confirm the appropriate performance indicators, and put in place mechanisms to collect whatever information/data will be required.

The review should include an examination of pilot project reports to assess the extent to which they provide the required information for evaluation purposes.

  • To the extent possible national evaluations should be coordinated with community-level evaluations

The purpose of the coordination would be to ensure that common measures are used, and to avoid any unnecessary duplication of effort and expense. Consideration should be given to working with regional coordinators to develop common approaches to evaluation and to providing resources for training in evaluation where it is required. Consideration should also be given to providing one-day seminars on project evaluation to project proponents, as is the case in Saskatchewan.

  • OFI, with regional coordinators, should establish a mechanism for the collection of information on current non-UAS investments on urban Aboriginal issues from governments and other sources.

Most participating communities do not have reliable information in this area as a basis for furthering the strategic elements of the initiative and as a basis for assessing the contribution of the UAS in a future summative evaluation.

  • OFI and regional coordinators should establish a systematic approach for a full accounting of the resources used for the UAS and how they have been used.

The resources should include those provided under the UAS budget and resources contributed by the lead departments in the participating communities. The approach should include common categories of expenditures for UAS coordinators to use to allow for standardization of the process.



7. Appendix A - Logic Model for the Urban Aboriginal Strategy Pilot Projects

Logic Model - UAS Pilot Projects



8. Appendix B - Formative and Summative Evaluation Tables of Questions, Indicators and Methods

UAS Formative Evaluation Table of Issues,
Questions, Indicators and Methods
Evaluation Issues Questions Indicators Methods
1. Are the UAS pilot projects, and the pilot project initiative as a whole, proceeding as intended, and are there any problems with implementation that should be addressed?  1.1 To what extent have UA communities been involved in planning and implementation of the initiative? The extent to which Aboriginal representatives are members of UAS steering committees and other planning and decision-making bodies

The extent to which Aboriginal stakeholders report satisfaction with the level of Aboriginal involvement

Review of membership lists of UAS steering committee and other planning and decision-making bodies

Survey of Aboriginal stakeholders
1.2 Is UAS planning and decision-making and project funding accessible to all interested Aboriginal stakeholders? The extent to which community planners have encouraged participation by all interested stakeholders

The extent to which stakeholders report satisfaction with accessibility of the planning and decision-making processes

Review of documents describing communications and other activities to encourage participation

Interviews with Regional Coordinators and community planning leaders

Survey of Aboriginal stakeholders
1.3 How effectively have federal and provincial governments, Aboriginal organizations and other stakeholders worked together to implement the UAS? What are the factors influencing effective collaboration? The extent to which stakeholders have participated on UAS steering committees and other planning and decision-making bodies

The extent to which stakeholders
Review of membership lists of UAS steering committee and other planning and decision-making bodies

Interviews with the full range of stakeholders report satisfaction with the collaborative efforts

1.4 Has there been coordination with other initiatives for UA people? The extent to which UAS planning and decision-making structures and processes are linked formally to other relevant community planning and decision-making bodies

The extent to which UAS steering committee members and members of other UAS planning and decision-making bodies are also members of planning bodies for other UA initiatives

The extent to which UAS planning leaders report coordination with other UA initiatives
Interviews with Regional Coordinators and community planning leaders
1.5 Do federal departments participating in the UAS have a common understanding of roles and responsibilities? The extent to which MOU's or other written agreements are in place that outline departmental roles and responsibilities on UA issues

The extent to which local federal officials describe a common understanding of roles and responsibilities
Interviews with federal officials in participating departments, in the participating communities

Interviews with federal officials in participating departments at the national level

Review of initiative documents
1.6 Have federal departments succeeded in coordinating their efforts in relation to UA issues? The extent to which federal officials at the national level have mechanisms in place to coordinate UA-related policies and programs

The extent to which federal officials at the community level have mechanisms in place to coordinate UA-related policies and programs

The extent to which federal government "single windows" have been established in communities

The extent to which federal officials and other community stakeholders report that federal coordination is taking place
Interviews with federal officials in participating departments, in the participating communities

Interviews with federal officials in participating departments at the national level

Review of documents describing federal coordination efforts

Interviews with Regional Coordinators and community planning leaders
1.7 Has having single contribution agreements and common terms and conditions for pilot projects brought efficiencies for community participants, and if so, what are the main areas of efficiency? The extent to which project proponents report efficiencies resulting from single contribution agreements and common terms and conditions for pilot projects, as compared to their experiences under previous arrangements

The extent to which Regional Facilitators and community planning leaders report efficiencies
Interviews with UAS project proponents

Interviews with Regional Coordinators and community planning leaders
1.8 Are participating federal departments satisfied that their information and accountability needs are being met through this "single window" approach? The extent to which federal officials report satisfaction with the accountability and informational aspects of the "single window" approach Interviews with federal officials in participating departments, in the participating communities

Interviews with federal officials in participating departments at the national level
1.9 Have federal government supports and resources assisted communities in UA planning and decision making? Supports and resources made available to community UA planners

The extent to which community UA planners and decision-makers report that the supports and resources assisted planning and decisionmaking
Interviews with Regional Coordinators and community planning leaders

Review of documents describing supports and resources provided
1.10 Have federal government supports and resources assisted project proponents? Supports and resources made available to project proponents

The extent to which project proponents report that the supports and resources assisted them in preparing and implementing their projects
Interviews with Regional Coordinators and community planning leaders

Review of documents describing supports and resources provided

Interviews with project proponents
1.11 Have the UAS pilot project delivery models compared favourably to previous models for UA project funding in terms of the time and effort required by stakeholders? The extent to which community planners and decision-makers, and project proponents, report that the UAS model has resulted in savings in time and effort as compared to previous federal funding mechanisms

The extent to which the time from original proposal submission to project approval is less under the UAS as compared to previous project funding delivery models
Interviews with Regional Coordinators and community planning leaders

Interviews with project proponents

Comparative analysis of project time lines from project files for the UAS and for other similar initiatives using different delivery models
1.12 How has the UAS pilot project delivery model compared to previous models for UA project funding in terms of the time and effort required by federal officials? The extent to which federal officials report that the UAS model has resulted in savings in time and effort as compared to previous federal funding mechanisms Interviews with federal officials in participating departments, in the participating communities, including Regional Coordinators
1.14 What challenges have been encountered in implementing the UAS, and how have these been addressed to date? Challenges reported by stakeholders and participants in planning and decision-making Interviews with HQ officials of participating departments, OFI, Regional Coordinators; community planning leaders; and local federal, provincial and municipal government officials

Survey of Aboriginal stakeholders
1.15 Are there alternative delivery approaches that might work better? Views of Regional Coordinators, community planning leaders, government officials at all levels, and Aboriginal stakeholders, about alternative delivery approaches Interviews with HQ officials of participating departments, OFI, Regional Coordinators; community planning leaders; and local federal, provincial and municipal government officials

Survey of Aboriginal stakeholders
2. Is the existing approach to performance measurement and the collection of information for that purpose sufficient, and are there any gaps in the types of measurements being applied or the information being collected, that present a risk to the strength of future evaluation findings, and that require changes to the performance measurement strategy? 2.1 What information/data on UAS planning and projects is currently being collected in communities for UAS reporting purposes? Databases and other information being collected for reporting purposes Review of the UAS Summative Evaluation Strategy to identify data/information required

Review of documents describing current information/data collection requirements and systems and procedures

Inventory of actual data available at initiative, project and pilot level

Interviews with Regional Coordinators and community planning leaders
2.2 Referring to the summative evaluation strategy, which evaluation questions and indicators does the information/data that is currently being collected pertain to, and is it sufficient for evaluation purposes in terms of validity and reliability? The extent to which current collection and reporting mechanisms provide valid and reliable information/data to meet summative evaluation requirements Review of the UAS Summative Evaluation Strategy to identify data/information required

Review of information/data currently available, to assess validity and reliability
2.3 Are there evaluation questions/indicators that will not be addressed sufficiently through current information/data collection mechanisms, and if so, is it feasible for communities to collect that information/data? The extent to which current collection and reporting mechanisms provide valid and reliable information/data to meet summative evaluation requirements

The extent to which possible approaches for the collection of information/data to fill gaps can be undertaken by participating communities within their available time and budgets
Review of the UAS Summative Evaluation Strategy and available information/data to identify gaps

Development by evaluators of possible approaches to fill information/data gaps

Interviews with Regional Coordinators and community planning leaders to discuss information/data gaps and assess the feasibility of collecting any additional information/data that is required
3. Are the pilot projects, and the initiative as a whole, making progress toward the achievement of the desired outcomes? 3.1 To what extent has the UAS initiative contributed to the development of partnerships among orders of government, Aboriginal groups and communities, and other stakeholders? # of MOU's

# of projects co-located /operated with other service providers

# of new referral arrangements among service providers

the extent to which the UAS has led to an increase in the sectors, and organizations within sectors, investing (financially or in-kind) in HQ activities

the extent to which the UAS has led to the establishment of new planning and decision-making structures and processes in communities to address UA issues, that include representatives from all sectors
Review of documents describing UAS planning and decision-making structures and processes, and membership on committees and decision-making bodies

Review of funded UAS project documents describing investors and other partners

Interviews with Regional Coordinators; community planning leaders; project proponents, and federal, provincial and municipal government officials

Survey of Aboriginal stakeholders
3.2 To what extent have these partnerships resulted in improved coordination of UA related activities? The extent to which stakeholders report improved coordination

The extent to which the UAS planning and decision-making process is endorsed by stakeholders as the focal point of UA planning

The extent to which community stakeholders report a reduction in overlap and duplication in UA related programming, and government requirements for funding and reporting
Survey of Aboriginal stakeholders

Interviews with Regional Coordinators; community planning leaders; project proponents, and federal, provincial and municipal government officials
3.3 To what extent has the federal capacity to respond in a coordinated way to community UA needs increased as a result of the initiative? The extent to which a "single window" approach to UA issues has been established at the community level

The extent to which relevant federal departments are participating in UA planning at the community level

The extent to which community stakeholders report an improved federal capacity to respond to UA needs in a coordinated way
Review of documents on the federal role and functions in UA activities at the community level

Interviews with Regional Coordinators; community planning leaders; project proponents, and federal, provincial and municipal government officials

Survey of Aboriginal stakeholders
3.4 To what extent has the capacity of the Aboriginal communities in participating communities to address UA issues been increased as a result of the UAS? The extent to which UA planning and decision-making mechanisms are more widely representative of the Aboriginal community than prior to the UAS

The extent to which the UAS planning and decision-making process is endorsed by stakeholders as the focal point of UA planning

The extent to which the UAS has increased overall investments in UA issues in the communities

The extent to which the UAS has increased the diversity of investments in UA issues in the communities

The extent to which more training/skills development programs are in place as a result of the UAS 
Review of membership lists of UAS steering committee and other planning and decision-making bodies

Interviews with Regional Coordinators; community planning leaders; project proponents, and federal, provincial and municipal government officials

Review of documents describing recent community investments in UA issues

Review of UAS projects to identify training/skills development programs

Survey of Aboriginal stakeholders
3.5 To what extent has the capacity of individual Aboriginal organizations in participating communities been increased as a result of the UAS? The extent to which organizations are participants in UA planning and decision-making

The extent to which organizations have increased their funding base as a result of the UAS

The extent to which organizations have increased the diversity of their funding sources as a result of the UAS

The extent to which organizations are participating in, or providing, more training/skills development for their staff as a result of the UAS

Review of membership lists of UAS steering committee and other planning and decision-making bodies

Interviews with Regional Coordinators; community planning leaders; project proponents, and federal, provincial and municipal government officials

Review of UAS projects to identify training/skills development programs

Survey of Aboriginal stakeholders

3.6 To what extent has the UAS resulted in new or enhanced services for urban Aboriginal people in participating communities? # of new or enhanced services put in place as a result of the UAS

# of conferences, symposia, workshops, research studies, other events that have taken place as a result of the UAS

# of sustained assets arising from the conferences, symposia, research

$ value of UAS investments in the participating communities

Review of UAS administrative data

Review of project final reports

Interviews with a sample of project proponents, Regional Coordinators, community planning leaders, OFI

3.7 To what extent has the UAS initiative narrowed the gap between urban Aboriginal people and the mainstream population? The extent to which clients of UAS funded programs and services have improved their living circumstances in ways related to the services being received (ideally, want to identify some specific "life circumstances" to focus on)

The extent to which Aboriginal organizations report that the UAS has contributed to overall improvements in the life circumstances of Aboriginal people in participating communities, relative to the mainstream population

Review of program administrative data

Case study research at a sample of programs and services, to include: reviews of case files; interviews with program/service managers and staff, interviews with clients

Interviews with UAS Regional Coordinators, community planning leaders, government partners at all levels

Survey of Aboriginal stakeholders

4. Are there areas where UAS activities overlap with other government programs and services? Are there areas where UAS activities overlap with other government programs and services? The extent to which terms and conditions for federal, provincial or municipal programs targeted to urban Aboriginal communities indicate that funds can be used for the same purposes as UAS funds

The extent to which assessments by UAS community planning leaders and federal, provincial and municipal government program managers indicate a risk of overlap and duplication

Review of program documents and funding terms and conditions for other federal, provincial and municipal government programs identified by UAS Regional Coordinators, federal, provincial, municipal officials and UAS community planning leaders as being related to the UAS

Interviews with the above, in participating communities

5. Are there indications that UAS has succeeded in leveraging partner resources, and has the size and distribution of UAS project and administrative allocations enhanced or hindered initiative objectives? 5.1 Are there indications that UAS has succeeded in leveraging partner resources? The extent to which UAS projects include non-UAS investments from other sources

The extent to which UAS community planning includes investments (including in-kind investments) from other government partners

The extent to which non-UAS investments are identified by the investors as having resulted from the UAS investments

The extent to which partners have invested in other non-UAS projects as a result of participation in the UAS

Review of UAS project budgets

Review of UAS administrative budgets in participating communities

Interviews with UAS Regional Coordinators, community planning leaders, and government offices investing in UAS projects

5.1 Has the size or distribution of UAS project allocations and administrative allocations enhanced or hindered initiative objectives? The extent to which priority needs in participating communities have not been addressed due to funding limitations

The extent to which the distribution of UAS allocations for project funding and for administration are in keeping with UAS objectives

Review of project funding allocations by communities against community priorities

Review of administrative allocations to assess their contribution to the achievement of broad UAS objectives

Review of community decisionmaking processes to understand relationship between priorities and actual expenditures, and requirements for administrative funding

Interviews with UAS Regional Coordinators and community planning leaders

 

UAS Summative Evaluation Strategy Table of Issues, Questions,
Indicators and Methods
Evaluation Issues QUESTIONS Indicators Methods
Rationale and Relevance 1. To what extent do UAS related activities continue to address GoC priorities? the extent to which GoC and OFI priorities in urban Aboriginal affairs are reflected in UAS project expenditures

the extent to which GoC and OFI priorities in urban Aboriginal affairs are reflected in community priorities as documented in urban Aboriginal community plans

review of most recent policy documents produced by OFI and partner federal departments/agencies

key informant interviews with policy officials at OFI and partner departments

review of UAS project expenditures

review of UAS community plans

2. To what extent are UAS activities relevant to current and evolving urban Aboriginal needs and community needs? the extent to which UAS activities are directed to current urban Aboriginal needs as identified in the participating communities

survey of Aboriginal organizations

key informant interviews with urban Aboriginal leaders

review of available community-based documents identifying priorities, including UAS community plans

review of community consultation processes and findings

survey of UA organizations in participating communities to identify relevant federal policy and program decisions

3. To what extent are UAS pilot project activities consistent with current research and thinking about how best to address urban Aboriginal issues? the extent to which UAS-funded activities address issues faced by urban Aboriginal communities, as identified by academics and other experts

international literature review

assessment by 2 or 3 leading experts

review of documents describing the experiences of other similar community-based federal programs

Cost effectiveness 4. Are there areas where UAS activities overlap with other government programs and services? the extent to which terms and conditions for federal, provincial or municipal programs targeted to urban Aboriginal communities indicate that funds can be used for the same purposes as UAS funds

instances in which assessments by UAS community planning leaders and federal, provincial and municipal government program managers indicate overlap and duplication

the extent to which those cases of potential overlap and duplication are being addressed as opportunities for collaboration.

review of program documents and funding terms and conditions for other federal, provincial and municipal government programs identified by UAS Regional Coordinators, federal, provincial, municipal officials and UAS community planning leaders as potentially duplicating the UAS

key informant interviews with the above, in participating communities

5. Are there indications that UAS has succeeded in leveraging partner resources? the extent to which UAS projects are obtaining non-UAS investments from other sources

the extent to which UAS community planning functions are obtaining investments (including inkind investments) from other partners

the extent to which partners have invested in non-UAS projects addressing UA issues as a result of the UAS

review of UAS project budgets

review of UAS administrative budgets in participating communities

Key informant interviews with UAS Regional Coordinators, community planning leaders, and government offices investing in UAS projects

6. Has the size or distribution of UAS allocations supported or hindered initiative objectives? Evidence that priority needs in participating communities have not been addressed due to funding limitations

the extent to which the distribution of UAS allocations for project funding and for administration are in keeping with UAS objectives

the extent to which UAS administrative allocations are in keeping with amounts allocated for other similar initiatives

review of project funding allocations by communities against community priorities

review of administrative allocations to assess their contribution to the achievement of broad UAS objectives

review of community decision-making processes to understand relationship between priorities and actual expenditures, and requirements for administrative funding

key informant interviews with UAS Regional Coordinators and community planning leaders

Partnerships and Coordination

Success 7 To what extent has the UAS initiative contributed to the development of partnerships among orders of government, Aboriginal groups and communities? # of MOUs

# of projects co-located /operated with other service providers

# of new referral arrangements among service providers

the extent to which the UAS has led to an increase in the number of partnerships at the community level to address UA issues

the extent to which the UAS has led to an increase in the sectors, and organizations within sectors, investing (financially or in-kind) in UA activities

the extent to which the UAS has led to the establishment of new planning and decisionmaking structures and processes at the community level and within individual sectors to address UA issues

trend analysis of # of MOUs, joint projects, referral arrangements

review of UAS community planning documents

interviews with UAS Regional Coordinators, community planning leaders, partners in all participating sectors

survey of UA organizations in participating communities
8. To what extent have UAS partnerships resulted in a more coordinated approach to addressing UA issues? the extent to which stakeholders believe that resources for UA issues are being more effectively used

the extent to which community plans and decisionmaking are endorsed by stakeholders

the extent to which investments in UA issues at the community level are linked to community plans and priorities

the extent to which UAS planning and decisionmaking structures and processes are linked formally to other relevant community planning and decision-making bodies

the extent to which provinces and municipalities have re-aligned priorities and activities in light of UAS community plans and priorities

the extent to which the UAS pilot project horizontal Terms and Conditions have been used by signatory departments to fund UAS-related projects with funds other than those provided by PCO
Review of UAS projects and other UA-related investments in participating communities, against community plans and priorities as identified in community plans

interviews with UAS Regional Coordinators, community planning leaders, partners in all participating sectors

survey of UA organizations in participating communities
9. To what extent has the UAS initiative contributed to the establishment of better federal coordination mechanisms across departments the extent to which the UAS has led to new federal government structures (committees, decisionmaking bodies) at the national level

the extent to which the UAS has led to new federal government structures (committees, decisionmaking bodies) in participating communities

the extent to which the UAS has led to an increase in the number of federal departments and agencies participating in UA planning

the extent to which other federal initiatives are being coordinated through UAS structures/initiatives

# and type of single windows
review of UAS program documents at OFI

review of UAS community planning documents

interviews with UAS Regional Coordinators, federal partners at the national level, federal partners at the community level, community planning leaders

survey of UA organizations in participating communities to identify relevant federal policy and program decisions

review of UAS community plans to identify priorities and policy/program recommendations
10. To what extent have UAS-based federal coordination mechanisms led to improved coordination of federal activities related to UA issues? the extent to which community planning leaders report improved coordination of federal resources

the extent to which relevant federal policy and program decisions reflect community priorities as established in UAS community plans

the extent to which federal programs related to UA issues take a more holistic and integrated approach

the extent to which other relevant federal programs show linkages to the UAS 
interviews with UAS Regional Coordinators, federal partners at the national level, federal partners at the community level, community planning leaders

review of descriptions of recent, relevant federal policies and programs

interviews with provincial and municipal partners
Research and Communications

11. To what extent has UAS research (national, regional and local) resulted in new information about UA issues? # of new reports, other materials on UA issues

the extent to which this new information is being used by UAS participants to help planning and projects

review of results of UAS-funded research projects

interviews with Regional Coordinators; community planning leaders; project proponents, and federal, provincial and municipal government officials

12. To what extent have UAS research and communications activities at the national and community levels resulted in an increased interest in UA issues among academics # of new reports and articles on UA issues in academic journals

# of new conferences, symposia and other similar events focusing on UA issues

review of Canadian literature on UA issues

interviews with academics working in the area of UA issues

13. To what extent have UAS research and communications activities at the national and community levels contributed to increased investments and other activity on UA issues at the community level? the extent to which new investors in UA-related projects report being influenced by UAS communications activity

the extent to which new participants in UA planning and project development report being influenced by UAS communications activity

interviews with investors in UA projects

interviews with community planning and UAS project participants

Improved Services

14. To what extent has the capacity of the Aboriginal communities to address UA issues been increased as a result of the UAS? The extent to which UA planning and decisionmaking mechanisms are more widely inclusive of the Aboriginal community than prior to the UAS

The extent to which the UAS planning and decision-making process is endorsed by stakeholders as the focal point of UA planning

The extent to which the UAS has increased overall investments in UA issues in the communities

The extent to which the UAS has increased the diversity of investments in UA issues in the communities

The extent to which more training/skills development programs are in place as a result of the UAS

Review of membership lists of UAS steering committee and other planning and decisionmaking bodies

Interviews with Regional Coordinators; community planning leaders; project proponents, and federal, provincial and municipal government officials

Review of documents describing recent community investments in UA issues

Review of UAS projects to identify training/skills development programs

Survey of Aboriginal stakeholders

15. To what extent has the UAS resulted in new or enhanced services for urban Aboriginal people in participating communities? # of new or enhanced services put in place as a result of the UAS

# of conferences, symposia, research studies, other events that have taken place as a result of the UAS

# of sustained assets arising from the conferences, symposia, research

$ value of UAS investments in the participating communities

review of UAS administrative data

review of project final reports

key informant interviews with a sample of project proponents and with community planning leaders

16. To what extent have UAS-based improvements in coordination contributed to better government service delivery to Aboriginal communities? increased satisfaction among Aboriginal organizations with the quality of government services in key service areas

increased satisfaction among individual clients with government services to Aboriginal people, in key service areas

the extent to which service delivery managers and staff report improvements in the quality of services, that are attributable at least in part to improved coordination through the UAS

interviews with Regional Coordinators, community planning leaders and service delivery managers in key service areas, to identify perceived areas of service improvement, as a basis for assessing changes in client satisfaction

interviews with service delivery staff

survey of relevant Aboriginal organizations in participating communities

survey of clients of a sample of identified government services

in-person interviews with a sample of clients of identified government services

17. To what extent has the UAS initiative narrowed the gap between urban Aboriginal people and the mainstream population? the extent to which clients of UAS-funded services have improved their living circumstances in ways related to the services being received (ideally, want to identify some specific "life circumstances" to focus on)

the extent to which Aboriginal organizations report that the UAS has contributed to overall improvements in the life circumstances of Aboriginal people in participating communities, relative to the mainstream population

review of program administrative data

case study research at a sample of services, to include: reviews of case files; interviews with program/service managers and staff, interviews with clients

interviews with UAS Regional Coordinators, community planning leaders, government partners at all levels

survey of UA organizations in participating communities




Footnotes:

  1.  Urban Aboriginal Youth: An Action Plan for Change   (return to source paragraph)
  2. The logic model is provided in Appendix 1. (return to source paragraph)
  3. Details on the anticipated delivery approaches and procedures for pilot project funding are available in the UAS Results-Based Management and Accountability Framework. (return to source paragraph)
  4. Annual funding for the eight pilot projects is as follows: $1 million each in Winnipeg and Vancouver; $750,000 each in Calgary, Edmonton, Regina, Saskatoon and Toronto; and $250,000 in Thunder Bay. (return to source paragraph)
  5. Pilot projects will be established in the following urban centres: Vancouver ($1.0M), Edmonton ($750k), Calgary ($750k), Saskatoon ($750k), Regina ($750k), Winnipeg ($1.0M), Thunder Bay ($250k), Toronto ($750k) (return to source paragraph)
  6. UAS funding will be limited to the following provinces: British Columbia, Alberta, Saskatchewan, Manitoba, and Ontario on an equal basis, i.e., $500k in the first year and $300k in the subsequent two years (return to source paragraph)
  7. It is advised that no more than two-thirds of funding be used to support federal activities (return to source paragraph)
  8. Research activities include evaluation activities and community development training (return to source paragraph)
  9. Interview guides used in the evaluation are provided as an appendix to the report. (return to source paragraph)
  10. The logic model is provided in Appendix 1. (return to source paragraph)
  11. A copy of the survey is provided as an appendix to this report. (return to source paragraph)
  12. The eight communities use different terminology to describe the committees they have established to guide the UAS. In some cases there is both a focused UAS steering committee and a broader advisory committee with wider representation from the community at large. In all cases, though, there is a primary committee that is responsible to set priorities and make funding decisions. In this report, we use the general term "steering committee" to refer to this primary committee, whatever its name may be in a given community. (return to source paragraph)
  13. This example was raised by several interview respondents involved in Aboriginal education.(return to source paragraph)