Evaluation of the Procurement Strategy for Aboriginal Businesses

Final Report
June 2014
Project Number: 1570-7/13057

PDF Version (453 Kb, 69 Pages)

 

Table of contents


Executive Summary

The Evaluation, Performance Measurement and Review Branch (EPMRB), in compliance with the Treasury Board Policy on Evaluation, has conducted an evaluation of the Procurement Strategy for Aboriginal Business (PSAB) and its related activities, including the Aboriginal Workforce Participation Initiative. The purpose of the evaluation was to provide a neutral evidence-based assessment of program relevance and performance to inform decision making on resource allocation and reallocation, and to support policy and program improvement where required. PSAB outcomes and activities are under the Strategic Outcome The Land and Economy.

Recognizing that Aboriginal businesses are under-represented among firms seeking and winning federal government procurement opportunities, Aboriginal Affairs and Northern Development Canada (AANDC) partnered with Public Works and Government Services Canada to develop the PSAB in 1995. With its introduction, the Government of Canada recognized that, while serving its primary purpose of obtaining required goods and services at market rates, procurement is also a tool for Aboriginal businesses to grow by gaining experience, developing capacity, and forming partnerships with other businesses to compete for procurement opportunities. PSAB was intended to support and foster economic growth in the Aboriginal business sector through greater participation in government procurement. PSAB came into effect on April 1, 1996, pursuant to the Treasury Board's Contracting Policy, making it a government-wide policy, applicable to all federal departments and agencies designated as departments for the purposes of the Financial Administration Act (with the exception of the Canada Revenue Agency). PSAB is led by AANDC under the authority of Treasury Board contract policy notices.

The Aboriginal Workforce Participation Initiative carries out advocacy activities to educate and inform employers about the advantages of hiring Aboriginal people; to support partnerships with various stakeholders to increase the capacity of employers to recruit, promote and retain Aboriginal employees; and to bring all employers, existing and potential Aboriginal employees and other parties together.

The scope of this evaluation covers the period from 2007-2008 and 2012-2013. This evaluation was conducted by EPMRB with the support of Auguste Solutions and Associates Inc. conducting key informant interviews external to the federal government, and Harris Decima managing the implementation of an online and telephone-based survey. The final report was completed by EPMRB.

The evidence collected for this evaluation produced the following findings:

With respect to relevance:

  1. There is a need for continued investment in initiatives designed to strengthen viable Aboriginal businesses. Though the PSAB supports this objective, there is concern that the approach generally favours larger and more established firms over new and smaller businesses and entrepreneurs.
  2. The objectives of the PSAB are well-aligned with government priorities and departmental strategic outcomes. The approaches to meeting these objectives need to continually adjust to evolving needs and business environments to ensure optimal contribution to these objectives.
  3. The current activities under the PSAB are consistent with federal roles and responsibilities.

    With respect to performance:
  4. The PSAB is resulting in Aboriginal firms winning an increased share of contracts over time, and significant strides have been made in promoting procurement with Aboriginal firms. Data collection to date, however, does not allow for a complete analysis on whether the PSAB is truly supporting the creation and expansion of viable Aboriginal businesses, and ultimately does not allow for a complete illustration of performance.
  5. The expectations of a leverage effect resulting from PSAB were never articulated or measured, and this is no longer included as an official outcome of PSAB.
  6. Outcomes on the adoption of Aboriginal procurement strategies or participation agreements; and the outcome that Aboriginal business capacity is matched with business/procurement opportunities have only recently been established and articulated. PSAB and Aboriginal Workforce Participation initiatives have resulted in significant research related to potash, shipbuilding, and electricity sectors, and information and training sessions focused on business development competencies, procurement and business promotion and activities promoting Aboriginal women in business.
  7. The main issues affecting the success of PSAB relate to the complexity of the procurement process and challenges in securing opportunities to gain the experience necessary to be qualified for government contracts.
  8. The PSAB and its related activities are, generally speaking, an economical approach to supporting Aboriginal businesses through the Federal Framework for Aboriginal Economic Development. The significant expenditures currently invested in large conferences, however, may be better spent on providing better access to targeted training initiatives designed specifically to match businesses with economic opportunities; adequately train them on how to bid in competitive processes; and how to be more competitive and take better advantage of regional opportunities.

It is therefore recommended that AANDC:

  1. Develop an enhanced approach to the PSAB that is tailored to the different needs of different types of business, including a stronger focus on direct and regional training to support newer and smaller Aboriginal firms to navigate the increasingly complex and competitive procurement environment;
  2. Work with Public Works and Government Services Canada to ensure ongoing performance data allows for a complete capture of data on individual businesses winning procurement contracts by value and type both for set-asides and incidental contracts;
  3. Develop better accountability mechanisms for the accurate capture of whether or not bidders qualify as Aboriginal; and
  4. As part of the promotion of PSAB, work with contracting authorities to ensure the best likelihood of Aboriginal business success, including promoting the most appropriate application of the 25-day posting option.
 

 

Management Action Plan

Project Title: Evaluation of the Procurement Strategy for Aboriginal Business and Related Activities
Project #: 1570-7/13057

Recommendations Actions Responsible Manager (Title / Sector) Planned Implementation and Completion Date
1. Develop an enhanced approach to the PSAB that is tailored to the different needs of different types of business, including a stronger focus on direct and regional training to support newer and smaller Aboriginal firms to navigate the increasingly complex and competitive procurement environment Aboriginal Entrepreneurship Branch (AEB) concurs. AEB will undertake the following actions:

Action A : Distribution/ awareness of new PSAB video;

Action B : Increase outreach activities and training on federal procurement processes and how to access procurement opportunities (e.g. working with regional offices to have booths at relevant conferences, increased collaboration with the Office of Small and Medium Enterprises, building on current MOU for training, and other related opportunities);

Action C : Funding for research/ gap analysis with Canadian Council for Aboriginal Business and private sector funding to inform training needs; and

Action D: Aboriginal Financial Institution (AFI) pilots – Communication and PSAB awareness dissemination to enable Aboriginal businesses to compete on federal procurement opportunities and build on registration in the Aboriginal Business Directory (ABD).
Senior Program managers, Aboriginal Procurement and Business Promotion Directorate (Managers responsible for outreach activities, funding and partnerships with the AFIs.) Action A : Video 1 published  in July 2014. Video 2 and 3 will be published in November 2014.

Action B: Planned implementation date: Work plan will be created in August 2014-on going activity.

Action C : Planned implementation date: Discussion initiated in June 2014

Completion date: March 2015.

Action D :
Two pilot projects were initiated in 2013 and two others will be in 2014.

Completion date:  March 2015.
2. Work with PWGSC to ensure ongoing performance data allows for a complete capture of data on individual businesses winning procurement contracts by value and type both for set-asides and incidental contracts AEB concurs. AEB is committed to undertake the following actions:

Action A : Continue to work with PWGSC to explore options for replacing the Supplier Registration Information (SRI) Aboriginal indicator to enable accurate validation of Incidental Aboriginal procurement data; and

Action B: Explore a whole of government approach (DM initiative on simplifying e-services for businesses) for a common identifier that is used by all government departments and agencies. 
Senior Program manager, Aboriginal Procurement and Business Promotion Directorate (Manager responsible for the ABD and partnerships with PWGSC.) Action A : Planned implementation date: MOU with PWGSC signed in July 2014

Expected completion date: March 2016.

Action B :
Planned implementation date: Business case to implement a common identifier scheduled for completion for Fall 2014.
3. Develop better accountability mechanisms for the accurate capture of whether or not bidders qualify as Aboriginal AEB concurs. AEB is committed to undertake the following actions:

Action A : Increase audit resources through innovative approaches;

Action B : Increase numbers of pre-award audits, post-award audits and random audits;

Action C : Complete an audit toolkit for internal auditors;

Action D : Increase awareness of the auditing process within the contracting authorities' and procurement community; and

Action E: Increase the review of BuyandSell.gc.ca to monitor.
Senior Program manager, Aboriginal Procurement and Business Promotion Directorate (Manager responsible for the auditing file) Action A : Planned implementation date: A part time student hired in June 2014 and a second auditor will be hired in August 2014.

Expected completion date: March 2015

Action B : Planned implementation date: May 2014

Expected completion: March 2015

Action C: Planned implementation date: Working draft developed. completion date: September 2014

Action D: Planned implementation date: June 2014-on going activity.

Action E: Planned implementation date: June 2014-on going activity.
4. As part of the promotion of PSAB, work with contracting authorities to ensure the best likelihood of Aboriginal business success, including promoting the most appropriate application of the 25-day posting option AEB concurs. AEB will undertake the following action:

Action A: Provide 20 information sessions to the procurement community and Aboriginal businesses regarding posting options and the flexible posting options available for set-aside contracts in 2014-2015 fiscal year - to continue in future years.
Senior Program manager, Aboriginal Procurement and Business Promotion Directorate (Manager responsible for maintaining partnership with PWGSC). Action A : Planned implementation date: July 2014- Expected completion date: March 2017.
 

I recommend this Management Response and Action Plan for approval by the Evaluation, Performance Measurement and Review Committee 

Original signed by:

Michel Burrowes
Director, Evaluation, Performance Measurement and Review Branch

 
 

I approve the above Management Response / Action Plan

Original signed by:

Sheilagh Murphy
Assistant Deputy Minister, Lands and Economic Development

 

The Management Response / Action Plan for the evaluation of the Procurement Strategy for Aboriginal Business and Related Activities were approved by the Evaluation, Performance Measurement and Review Committee on June 20, 2014. 

 

 

1. Introduction

1.1 Overview

The Evaluation, Performance Measurement and Review Branch (EPMRB) at Aboriginal Affairs and Northern Development Canada (AANDC) has conducted an evaluation of the Procurement Strategy for Aboriginal Businesses (PSAB) and related activities under the former Aboriginal Workforce Participation Initiative (referred to as the Aboriginal Entrepreneurship Program as of April 2014). The objective of the program is to leverage federal government's procurement opportunities to enhance the viability of Aboriginal businesses, and thereby, increase Aboriginal economic development and employment. This evaluation fulfils the requirement of the Treasury Board Policy on Evaluation to provide a neutral and evidence-based assessment to make judgments on the relevance and performance of programming.

The PSAB was last evaluated in 2006-2007. The scope of the current evaluation covers all activities undertaken between 2007-2008 and 2012-2013. The evaluation itself was conducted between October 2013 and April 2014 by EPMRB with the assistance of two consulting firms. Under the direction of EPMRB, Auguste Solutions and Associates Inc. conducted and analysed key informant interviews with non-federal governmental stakeholders and Harris Decima implemented a survey of PSAB coordinators, material managers/contracting personnel, and some Aboriginal businesses. EPMRB led the design of all evaluation tools and methodology; conducted interviews with federal government representatives; conducted document/data and literature reviews and analysis; analysed survey and interview data; and drafted the final report.

This final report details the PSAB and all related activities and initiatives; provides an overview of methodology; details key findings and presents a set of recommendations and related responses and action plans. This evaluation also considered other pertinent evaluation issues, best practices and lessons learned.

1.2 PSAB Program Profile

Background and Description

Recognizing that Aboriginal businesses are under represented among firms seeking and winning federal government procurement opportunities, AANDC partnered with Public Works and Government Services Canada to develop the PSAB in 1995.

With its introduction, the Government of Canada recognized that, while serving its primary purpose of obtaining required goods and services at market rates, procurement is also a tool for Aboriginal businesses to grow by gaining experience, developing capacity, and forming partnerships with other businesses to compete for procurement opportunities. PSAB was intended to support and foster economic growth in the Aboriginal business sector through greater participation in government procurement. PSAB came into effect on April 1, 1996, pursuant to the Treasury Board's Contracting Policy, making it a government-wide policy,Footnote 1 applicable to all federal departments and agencies designated as departments for the purposes of the Financial Administration Act (with the exception of the Canada Revenue Agency). PSAB is led by AANDC under the authority of Treasury Board contract policy notices.

In June 2009, the Government of Canada announced the new Federal Framework for Aboriginal Economic Development in conjunction with an action plan to support its implementation. PSAB is part of the 2009 Action Plan to implement the Federal Framework for Aboriginal Economic Development. To help address challenges encountered by Aboriginal business to access federal procurement, through the Federal Framework for Aboriginal Economic Development, the Government invested $8.3 million over four years (2009-2013) in PSAB.

PSAB supports the following government priorities that were established in the 2009 Federal Framework for Aboriginal Economic DevelopmentFootnote 2:

  • Maximizing economic outcomes for Aboriginal Canadians, through the awarding of contracts, requirement on Aboriginal employees, requirements on work performed by Aboriginal employees throughout a contract, etc;
  • Support skills and training to create new opportunities and choices for Aboriginal people, by requiring Aboriginal businesses to employ (and provide the necessary skills training to) Aboriginal peoples;
  • Leveraging investment and promoting partnerships with the private sector to produce sustainable growth for Aboriginal peoples, by encouraging joint ventures between Aboriginal and non-Aboriginal firms, and contribution agreements with private firms to establish sustainable business partnership with Aboriginal businesses; and
  • Act to free businesses to grow and succeed by removing barriers, by linking Aboriginal businesses to the training and tools necessary to understand and utilize the federal tendering system to win contracts.

Furthermore, the Framework outlines a way forward to increase the ability of Aboriginal suppliers to successfully bid on contracts, increase awareness and implementation of the strategy among federal departments, and increase the number of registered firms in the Aboriginal Business Directory.

PSAB and the Federal Framework for Aboriginal Economic Development commitments formed the basis of developing a modernized approach to enhance Aboriginal procurement and business partnerships more broadly. The development of this modernized approach was part of a commitment in the 2010 Land and Economic Development strategic roadmap.

It should be noted that the North American Free Trade Agreement and the World Trade Organization Agreement on Government Procurement provide for procurements to be "set aside" for minority and small businesses. Procurements set aside for Aboriginal businesses are therefore excluded from the provisions of these two international trade agreements.

Within AANDC's Economic Development Programming, there is also the Aboriginal Workforce Participation Initiative, which is intended to benefit Aboriginal people in Canada. This component seeks increased participation by Aboriginal people in the economy through economic and employment opportunities. This initiative works to break down the barriers that may deter business opportunities and the employment of Aboriginal peoples. This was to be accomplished by raising awareness of Aboriginal employment issues; promoting and retaining Aboriginal employees; enhancing the capacity and competencies of Aboriginal businesses; establishing partnerships with Aboriginal organisations and the private sector; showcasing Aboriginal capacities; and promoting information-sharing and networking among stakeholders.

Objectives and Expected Outcomes

Program Objectives/Activities

In support of the Federal Framework for Aboriginal Economic Development's key objective, PSAB leverages federal government procurement as a tool to improve the participation of First Nations, Inuit and Métis people in the Canadian economy. The rationale behind the objective is that successful Aboriginal businesses create jobs for Aboriginal and non-Aboriginal people, and help strengthen local economies.

Within the scope of the evaluation, this objective is directly linked to AANDC's 2012-2013 Program Activity Architecture Strategic Outcome - The Land and Economy and more specifically, the Program Activity Aboriginal Economic Development and the sub-activity Strengthening Aboriginal entrepreneurship by enhancing support for Aboriginal businesses to market their goods and services. So positioned, the objective of the PSAB is to leverage the federal government's procurement opportunities to enhance the viability of Aboriginal businesses, and thereby, increasing Aboriginal economic development and employment.

As of 2014-2015, PSAB falls under AANDC's Program Alignment Architecture Strategic Outcome- The Land and Economy and more specifically the Program Activity Aboriginal Entrepreneurship and the sub-activity Business Opportunities. PSAB is positioned within the Architecture in this manner to enhance Aboriginal businesses' access to government contracting through the promotion of Aboriginal business procurement within the federal government and in other government departments.Footnote 3

PSAB objectives include:

  • Increasing the number of businesses in supplier registries;
  • Increasing Aboriginal set-asides;
  • Improving departmental objective-setting and reporting regarding procurement from Aboriginal businesses;
  • Increasing training and outreach to businesses and government departments;
  • Provide advice and guidance to federal departments, provincial/territorial governments and businesses; and
  • Researching Aboriginal benefit/participation components.

PSAB seeks to accomplish its overall goals by allowing federal departments and agencies to set aside procurement opportunities for Aboriginal-owned businesses (i.e., limit competitive bidding on the opportunity to only Aboriginal-owned businesses) with an overall aim of the strategy to increase the number of Aboriginal suppliers bidding for, and winning, federal contracts. PSAB is comprised of four components:

  • Mandatory Set-Asides: A mandatory set-aside applies to all federal contracts with a value greater than $5,000 for goods or services delivered to a primarily Aboriginal population;
  • Voluntary Set-Asides: In addition to mandatory set-asides, federal departments and agencies may elect to voluntarily set aside select procurement opportunities for Aboriginal-owned businesses;
  • Joint Ventures and Partnerships: To help with business and capacity development, the Strategy provides a framework for Aboriginal businesses to form joint ventures and partnerships with other Aboriginal or non-Aboriginal businesses and allows these partnerships to bid on opportunities that have been set aside;
  • Use of Aboriginal criteria: In the qualification and awarding of prime contracts, all departments and agencies are encouraged to request Aboriginal Business Sub-Contracting.

Activities are geared toward actively engaging the interest and participation of Aboriginal suppliers. The program also works to raise the awareness of federal government employees with regard to the capacity of Aboriginal businesses that may meet their procurement needs, and advising the private sector and different provincial governments based on the PSAB model. Different tools and approaches are used to support those activities:

  • Aboriginal Business Database identifying businesses that meet PSAB criteria to demonstrate and promote Aboriginal capacity;
  • Outreach and training activities with businesses, federal, provincial and territorial governments;
  • Working closely with businesses and stakeholders to undertake Aboriginal supplier development activities;
  • Leading a national interdepartmental PSAB Coordinator Network to help apply and support the policy;
  • Facilitating and intervening in the application of PSAB set-asides of government procurement for Aboriginal businesses and the provision of advice and guidance; and
  • Monitoring and participation to identify and negotiate set-asides and procurement opportunities.

The Aboriginal Workforce Participation Initiative carries out activities to promote business development, business promotion and literacy as well as advocacy activities to education and inform employers about the advantages of hiring Aboriginal people; to support partnerships with various stakeholders to increase Aboriginal businesses access to opportunities and increase the capacity of employers to recruit, promote and retain Aboriginal employees; and to bring Aboriginal suppliers, federal and private buyers and other parties together.

Expected Outcomes

The PSAB is expected to achieve the broader social and economic objectives of the federal government for Aboriginal economic development through the following outcomes:

  • Increased representation of Aboriginal businesses in contract awards by individual departments, commensurate with Aboriginal business presence and Aboriginal labour force presence in Canadian industry;
  • Increased representation by Aboriginal business in business development activities and in business registration inventories; and
  • Increased procurement awareness in the private sector, through collaboration with Aboriginal organizations.

The Aboriginal Workforce Participation Initiative is intended to increase participation of Aboriginal people in the economy by promoting economic and employment opportunities for Aboriginal people in all sectors. The immediate outcome is increased Aboriginal employment.

Due to the absence of a PSAB Performance Measurement Strategy at the outset of the evaluation, the program needed to clarify the outcome statements listed above prior to the commencement of an evaluation. In February 2014, a Performance Measurement Strategy for the Aboriginal Entrepreneurship Program was completed, which incorporated aspects of the Business Opportunities sub-activity and which includes PSAB and the Aboriginal Workforce Participation Initiative. This Performance Measurement Strategy will support program planning, monitoring and reporting through the identification and collection of key performance indicators that provide information for ongoing program management and decision making and will inform future evaluations. The draft logic model for the PSAB is hosted in Appendix A and the Aboriginal Entrepreneurship Activity logic model is hosted in Appendix B.

Program Management, Key Stakeholders and Beneficiaries

Program Management and Stakeholders

Aboriginal Affairs and Northern Development Canada

Aboriginal Affairs and Northern Development Canada, and more particularly the Assistant Deputy Minister of Land and Economy, is responsible for the management, monitoring and reporting of PSAB.

The management responsibility of the government-wide implementation is under the Aboriginal Procurement and Business Promotion Directorate of the Aboriginal Entrepreneurship Branch. This Directorate oversees the day to day operation and assumes advocacy roles for PSAB. AANDC is also the sponsoring department for the directive of PSAB and responsible for:

  • Collaborating with departments to set performance objectives;
  • Verifying business compliance with PSAB eligibility criteria; and
  • Providing guidance, tools and training to departments.

The Aboriginal Procurement and Business Promotion Directorate is responsible for ensuring that information about the capacity of Aboriginal suppliers of goods and services is disseminated throughout Canadian business and government communities. Aboriginal suppliers of goods and services are offered PSAB information sessions. The Aboriginal Procurement and Business Promotion Directorate team invites the Office of Small and Medium Enterprises to provide training on the federal procurement process where needed.

AANDC coordinates the implementation of PSAB on behalf of the federal government in close collaboration with Public Works and Government Services Canada and Treasury Board Secretariat.  

Treasury Board Secretariat

Based on Treasury Board policy directives, departments and agencies are to make use of approved authorities, bidding procedures and processes consistent with Government Contracts Regulations and Treasury Board Contracting Policy Notes.

Public Works and Government Services Canada

In its role as common service organization, Public Works and Government Services Canada is drawing client departments' attention to the potential for voluntary set-asides under PSAB and potential omissions regarding mandatory set-asides under PSAB.

Industry Canada

Provides a platform for AANDC's Aboriginal Business Database housed within Industry Canada's Canadian Companies Capabilities Database.

In collaboration with Industry Canada, the Aboriginal Procurement and Business Promotion Directorate developed an online Aboriginal Business Database, which contains Aboriginal business information. The Aboriginal Business Database serves as a centralized search engine that can be accessed by Aboriginal businesses, government and industry to identify Aboriginal business capacity, increase business visibility and promote procurement opportunities nationally and internationally.

Federal Departments and Agencies

Federal departments and agencies are the main authorities making the decision whether to set aside a federal contract for competition among PSAB registered Aboriginal businesses or employ other elements of the PSAB, such as the Aboriginal Business Component, requiring that a percentage of the work in a contract to be completed by an Aboriginal business.

Each department and agency with an annual contracting budget in excess of $1 million is mandated to set and report Aboriginal procurement targets and appoint a PSAB coordinator. Federal departments and agencies are encouraged to both increase their purchases from PSAB-registered Aboriginal firms and to increase the awareness about the capacity of Aboriginal firms to participate in federal contracting opportunities.

Beneficiaries

PSAB is open to all Aboriginal businesses, including sole proprietorship, limited companies, co-operatives, partnerships, and not-for-profit organizations. To be considered an Aboriginal business, the following criteria must be met:

  • at least 51 percent of the firm must be owned and controlled by Aboriginal people; and
  • if the firm has six or more full-time staff, at least one third of the employees must be Aboriginal.

If a firm is starting a joint venture, at least 51 percent of the joint venture must be owned and controlled by an Aboriginal business or businesses.

Also, a firm must demonstrate, for the duration of the contract, a level of Aboriginal content amounting to 33 percent of the value of the work performed by the Aboriginal business.

With respect to the Aboriginal Workforce Participation Initiative, eligible recipients can be:

  1. Aboriginal individuals;
  2. First Nation or Inuit communities;
  3. Municipal governments;
  4. Labour organizations;
  5. Universities and colleges;
  6. Other learning institutions such as sector councils and sector associations that can affect industry-specific learning opportunities; and
  7. Private and Aboriginal corporations

Program Resources

Table 1: Budgeted Expenditures for PSAB/Aboriginal Workforce Participation Initiative from 2009-10 to 2013-14

    2009-10 2010-11 2011-12 2012-13 2013-14
PSAB Operation and Maintenance 600,033 831,339 831,339 831,339 831,339
Salary 535,012 678,702 678,702 678,702 678,702
Grants and Contributions 225,000 500,000 500,000 500,000 500,000
Operation and Maintenance Internal Services Corporate Costs 40,666 59,398 59,398 59,398 59,398
Salary Internal Services Corporate Overhead 30,393 40,859 40,859 40,859 40,859
Treasury Board Secretariat-mandated Public Works and Government Services Canada accommodation costs (13 percent of salary costs) 73,503 93,543 93,543 93,543 93,543
Employee Benefit Plan (20 percent of salary total and corporate salary overhead) 113,081 143,912 143,912 143,912 143,912
TOTAL 1,617,687 2,347,752 2,347,752 2,347,752 2,347,752
  Grand Total 8,660,943.00
 

Table 2: Actual Expenditures for PSAB/Aboriginal Workforce Participation Initiative from 2009-10 to 2013-14

  2009-2010
Actual
2010-2011
Actual
2011-2012
Actual
2012-2013
Actual
SA1 – Indian and Northern Affairs Canada SALARY 1,088,118.90 1,019,686.50 1,028,297.52 1,330,106.66
NS1 – Indian and Northern Affairs Canada NON SALARY 468,780.07 661,219.04 550,621.41 326,404.63
V10 - GRANTS AND CONTRIBUTIONS 225,000.00 500,000.00 423,839.92 680,222.03
Total 1,781,898.97 2,180,905.54 2,002,758.85 2,336,733.32
Actual Grand total 8,302,296.68
 
 

 

2. Evaluation Methodology

2.1 Evaluation Scope and Timing

The evaluation examined program activities of PSAB and related activities under the Aboriginal Workforce Participation Initiative undertaken between 2007-2008 and 2012-2013. Terms of Reference were approved by AANDC's Evaluation, Performance Measurement and Review Committee on February 22, 2013. Field work was conducted between September 2014 and April 2014. The delay from the approval of the Terms of Reference and the commencement of field work was due to resource constraints and a high volume of evaluation studies occurring over this period.

2.2 Evaluation Issues and Questions

In line with the Terms of Reference, the evaluation of PSAB and related activities addressed core evaluation issues of relevance and performance in line with the requirements of the 2009 Policy on Evaluation. Specifically, the evaluation focussed on the following core issues and questions:  

Relevance

Continued Need

Does PSAB continue to address a demonstrable need and is it responsive to the needs of Aboriginal businesses? Does the Aboriginal Workforce Participation Initiative address a demonstrable need and is it responsive to the needs of Aboriginal peoples and First Nation and Inuit communities?

Alignment with Government Priorities

Are the objectives of the program consistent with government-wide priorities and AANDC's strategic outcomes?

Alignment with Federal Roles and Responsibilities

To what extent are the objectives of the program aligned with the role and responsibilities of the federal government? Is there duplication or overlap with other policies or initiatives?

Performance

Effectiveness (i.e. Success)

To what extent are the PSAB's expected outcomes (immediate, intermediate and long term) being achieved?

To what extent have external and internal factors influence the achievement of PSAB's expected outcomes? Are there any unexpected impacts of the PSAB, either positive or negative?

To what extent has PSAB created a leverage effect within the Federal Framework for Aboriginal Economic Development?

To what extent is PSAB programming consistent with Treasury Board policy, directives and procedures?

Have the appropriate mechanisms to deliver PSAB been put in place?

Demonstrations of Efficiency and Economy

  • Has the PSAB optimized its processes and quality of services to achieve expected outcomes? (efficiency)
  • Are there other efficient ways of delivering the initiative to achieve similar results? (efficiency)
  • Are there opportunities to achieve the intended results of PSAB using fewer resources? (economy)

Other Evaluation Issue(s)

Lessons Learned and Best Practices

What are the lessons learned and best practices that emerged from the implementation of PSAB and other similar programs from other jurisdictions, countries, or in the private sector, which may enhance PSAB effectiveness?

2.3 Evaluation Methodology

Evaluation phases

Phase 1: Preliminary Activities and Planning. Prior to the evaluation, EPMRB, in collaboration with the PSAB team, reviewed the uncompleted PSAB Performance Measurement Strategy in April 2012 to reflect new activities undertaken since the Results-based Management and Accountability Framework/Results-based Audit Framework for the Procurement Strategy for Aboriginal Business (see Appendix A), and the Performance Measurement Strategy of the New Federal Framework for Aboriginal Economic Development (see Appendix B). In December 2012, the evaluation team met the management team of the PSAB to discuss evaluation issues/questions, approach and scope.

Phase 2: Research and Analysis. The core evaluation questions identified in the planning phase of the evaluation guided all stages of data collection, analysis and reporting with the ultimate goal being to gather as many lines of evidence as possible through data triangulation in order to answer each question thoroughly.

Phase 3: Reporting. Several documents and reports were created to reflect the evidence collected from the evaluations lines of inquiry. These documents were used to influence the production of this draft evaluation report. The evaluation report includes the evidence, conclusions and recommendations related to PSAB and Aboriginal Workforce Participation Initiative where applicable.

Data Sources

Literature and Media Review

Using national and international sources, a literature review was conducted to examine core evaluation issues of need and relevance, how the program aligns with government roles and priorities, and how the program contributes to the broader issue of economic development. Peer-reviewed academic, intergovernmental, non-governmental and business literature will be collected using key words and phrases related to Aboriginal procurement and employment, including government contracts for minorities, set-asides, and Aboriginal/indigenous economic development. Provincial, territorial, and other countries' government procurement programs and policies will also be reviewed to document best practices among Procurement Set-Aside Programs and contracts for eligible groups (e.g. Aboriginal-owned business, minority-owned business and women-owned business). Finally, procurement and hiring practices in the private sector will be examined to better understand all contracting and employment opportunities for Aboriginal people in Canada.

Similarly, a media review will also be conducted to determine the prevalence of evidence supporting the performance of the PSAB as well as its effectiveness. Media scans will consist of the examination of Canadian newspapers and the AANDC internal Newsdesk. Searches will be conducted using key words and phrases related to the PSAB program such as: government procurement strategy; procurement strategy for Aboriginal business; Aboriginal workforce participation initiative; Aboriginal workforce; government contracts; supplier diversity; and set-asides. Source information will be organized based on themes, which relate to the evaluation questions. These themes are: Support for Program Relevance; Highlighting Design and Delivery Challenges / Best Practices; Supporting Performance; and Efficiency and Economy. Procurement practices in the private and public sector will be highlighted throughout the media scan where applicable.

Document Review

A document and file review will be undertaken and will include, but is not limited to, Treasury Board submissions, Speeches from the Throne, federal budgets, Memorandum to Cabinet, Office of the Auditor General of Canada reports, management plans, progress and performance reports, project files, reviews of audits, and other type of documents related to this evaluation. Reports from Aboriginal organizations and other reputable institutions were also examined. An Internet search of procurement programs from the private, international and government sectors was conducted to document alternatives and best practices. These documents will be analyzed and key points will be highlighted to address the evaluation questions presented in the Terms of Reference and the evaluation matrix.

Key Informant Interviews

It is expected that interviews and/or focus groups will be conducted to provide information on the progress of the PSAB and to validate and contextualize information obtained via other methodologies, including, but not limited to:

  • AANDC managers / regional offices, program officers (Headquarters);
  • Other federal department representatives (including Treasury Board Secretariat, Public Works and Government Services Canada);
  • PSAB coordinators;
  • Departmental material managers;
  • Aboriginal businesses representatives;
  • Provincial/territories government representatives (e.g. Ontario, Manitoba);
  • Key stakeholders and economic partners (e.g. Aboriginal Financial Institutions, Economic Development Officers);
  • Private sector representatives that implemented a Corporate Social Responsibility policy for Aboriginal procurement;
  • Aboriginal organizations and representatives; and
  • Subject matter experts.
Surveys

To collect specific information on matters directly linked to the outcomes, performance, design and delivery, stakeholders and beneficiaries, EPMRB will conduct surveys of both PSAB coordinators and departmental material managers. These surveys will be used to supplement key informant interviews as the 2007 evaluation of the PSAB used this same process but encountered a low response rate. Like the 2007 evaluation of PSAB, EPMRB will also consider a survey of businesses that received contracts under the PSAB. However, an assessment of the availability of contact information from Public Works and Government Services Canada and other departments will be required as in 2007 the evaluation team encountered challenges due to a lack of data and that a number of the businesses identified were no longer in business. If EPMRB faces the same challenge, interviews will be used.

The 2007 evaluation report recommended that a general survey of Aboriginal businesses in Canada would have been useful to assess awareness and opinion of the strategy among Aboriginal businesses in general. Discussion with the Procurement and Aboriginal Procurement and Business Promotion Directorate and the Public Opinion research will be undertaken to see the feasibility of this option. If not feasible, the evaluation will base its review on the 2011 Aboriginal Business Survey and interviews.

Data Analysis

Administrative data provided by Public Works and Government Services Canada were used to analyse trends on procurement respecting Aboriginal businesses, both within set-aside arrangements and incidentals, over the period of study.

2.4 Considerations, Strengths and Limitations

Considerations:

  1. As PSAB involves, many departments with different roles and responsibilities, collaboration and access to data of those federal departments are key to the success of this evaluation. To increase success, a communication procedure was developed in collaboration with the Procurement and Aboriginal Procurement and Business Promotion Directorate to increase participation of PSAB coordinators, departmental material managers as well as data sharing.
  2. As the Land and Economy Sector is under program renovation in fulfillment of the Federal Framework for Aboriginal Economic Development commitments, the activities undertaken by the Procurement and Aboriginal Procurement and Business Promotion Directorate may go beyond the initial reach of PSAB. The evaluation team considered those changes in the evaluation.

    Strengths:
  3. The Aboriginal Procurement and Business Promotion Directorate has recently conducted research, analysis, Business Partnership pilots and other models, including an Aboriginal Benefits Requirement to inform the efficiency. This will assist the evaluation team to assess the efficiency and economy evaluation issue.

    Limitations:
  4. A PSAB performance measurement strategy was under development for a large portion of the evaluation. EPMRB, thus, used a compilation of policy documents to articulate the intended outcomes of the strategy and measured performance against these outcomes.
  5. Due to the unavailability of a data collection system and raw data, as discussed in the previous evaluation of the PSAB in 2007, it was recommended that a data collection system for collecting performance data be implemented by March 2010. According to the March 21, 2012, Follow-up Report Status Update, an Aboriginal Business Directory and data collection systems have been developed in collaboration with Public Works and Government Services Canada, to be implemented in 2012-13. Key data that would support reporting needs on performance indicators therefore are only available as of 2012, thus preventing an analysis of the emergence and growth of specific businesses over time (discussed further in Section 3.1).
  6. A survey was implemented with the intention of capturing the insights of businesses. While businesses that were well-established and/or had established set-aside arrangements with federal departments were solicited, the primary focus of the survey was to gather insights from newer businesses and entrepreneurs, and specifically, those without knowledge of PSAB or without having established set-aside arrangements. While invitations to participate were sent to 103 businesses (with three reminder emails and one follow-up telephone call), only 15 completed the survey. EPMRB attempted to engage organizations with social media pages with a high amount of traffic among Aboriginal entrepreneurs and businesses to bolster solicitation, but no arrangement could be made within the time frame of data collection.

2.5 Roles, Responsibilities and Quality Assurance

The EPMRB of the AANDC Audit and Evaluation Sector is the project authority for the evaluation and will manage according to EPMRB's Engagement Policy and Quality Control Process, which is aligned with the Treasury Board Policy on Evaluation. Work for this evaluation will be completed by EPMRB staff and contractors under the direction of EPRMB. EPMRB, also referred to as the Project Authority, will direct and manage the evaluation. During the evaluation, EPMRB will also be responsible for liaising with an evaluation Working Group.

A Working Group has been established to ensure the quality and relevance of the evaluation approach, research instruments and the draft deliverables. The Working Group will include representatives of the various regional offices and Headquarters. Chaired by the Project Authority, the role of the Working Group will be to review documents, participate in discussion and provide the evaluation team with feedback at three key stages throughout the course of the evaluation: 1) the Methodology Report; 2) Preliminary Findings; and 3) Draft Final Report.

In line with Branch Policy, peer reviews will be conducted at two critical points in the evaluation project cycle: Methodology Reports and Draft Final Reports. Internal peer reviews are conducted by EPMRB evaluators who are not directly involved in the evaluation project. The reviewers assess the quality of the Methodology and Final Reports, and ensure they comply with relevant Treasury Board, AANDC and Branch Policy, criteria and standards. Internal peer reviewers also examine the degree to which final reports correspond with the evaluation's Terms of Reference and Methodology Report. Peer reviewers are not expected to validate factual information about programs or policies or edit reports. These tasks are the responsibility of the Project Team.  

Oversight of daily activities will be the responsibility of the PSAB evaluation team, headed by a senior evaluation manager. The Aboriginal Entrepreneurship Branch will be responsible for identifying and providing key documents and data for evaluation as well as names and contact information of department officials and key stakeholders. Other departments such as Public Works and Government Services Canada and Industry Canada will be sources for key informants and documentation.

 

 

3. Evaluation Findings - Relevance

3.1 Continued Need

Finding: There is a need for continued investment in initiatives designed to strengthen viable Aboriginal businesses. Though the PSAB supports this objective, there is concern that the approach generally favours larger and more established firms over new and smaller businesses and entrepreneurs.

Businesses operated by Aboriginal peoples face greater challenges than non-Aboriginal peoples in business start-ups and operations and maintenance. For example, concerning businesses operating on-reserve, there are regulatory and legislative obstacles stemming from Section 89 of the Indian Act, "preventing the seizure of real and personal property on-reserve."Footnote 4 As a consequence, banks and other financial institutions are hesitant to provide loans to Aboriginal people living on-reserves due to limitations on the use of Aboriginal land and homes as collateral,Footnote 5 thus, potentially limiting Aboriginal peoples' access to commercial capital, such as financing and equity partnerships.Footnote 6 These barriers to accessing financing have resulted in 55 percent of Aboriginal entrepreneurs relying on personal savings for both start-up and ongoing financing, versus 17 percent who use business loans or bank credit.Footnote 7 Banks also perceive Aboriginal businesses as being at higher risk and associated with higher service costs than non-Aboriginal businesses.Footnote 8

Additionally, businesses on-reserve also face burdensome government processes that may impede business development. Processes are extremely "slow and burdensome" when creating an application for leasing reserve lands.Footnote 9 The long process and bureaucratic delays may result in lost revenue, investment opportunities and economic development opportunities.Footnote 10 The Canadian Chamber of Commerce has also noted that their members who regularly conduct business with Aboriginal people face what they describe as "jurisdictional chaos," which results in immense confusion over the roles and responsibilities of various levels of government, which delays business processes.Footnote 11

Beyond the legislative issues associated with doing business on-reserve, persistent socioeconomic challenges faced by Aboriginal peoples both off- and on-reserve have caused significant barriers for Aboriginal people to develop businesses, and have contributed to a general under-representation of Aboriginal businesses in the Canadian business landscape.Footnote 12 Lack of capital in general, and higher rates of poverty and other socioeconomic issues have resulted in insufficient access to collateral for business financing.Footnote 13

Barriers faced by Aboriginal businesses reinforce the need for a strategy to promote Aboriginal business, and to strengthen opportunities to develop their experience and capacity. The PSAB is designed to support the participation of Aboriginal peoples and communities in the Canadian economy by leveraging federal procurement through Aboriginal set-asides, and encouraging the use of Aboriginal firms, including target-setting. Evidence in this evaluation suggests that PSAB supports Aboriginal businesses by providing a vehicle for firms to gain experience in competing for contracts and working for a large client.

In examining the need to support Aboriginal participation through business, however, it is essential to examine the significance of the main barriers facing Aboriginal businesses relative to what barriers PSAB is designed to address. Interview and survey participants supported findings in the literature, which suggest that the most significant barriers to the creation and viability of Aboriginal businesses include:

  • A lack of practical knowledge and capacity to respond to request for proposals, both in the federal government and private sector;
  • A need for coaching and mentoring;
  • Limited access to Aboriginal workers with the skills being sought;
  • Perceptions of complications in doing business with Aboriginal firms;
  • The higher cost of doing business with remote businesses; and
  • The smaller size and capacity of most Aboriginal businesses relative to the many large scale projects.

As discussed in further detail in Section 4.1, the initiatives of the Aboriginal Entrepreneurship Program (including the work directed via the former Aboriginal Workforce Participation Initiative) have prioritized activities designed to address some of these barriers. Concern was widely expressed among survey and interview participants, however, that with the focus of PSAB largely centered on bolstering procurement with Aboriginal firms, coupled with the fact that contracts with the federal government are typically large and highly competitive, PSAB may in effect be providing most of its support to larger or more established Aboriginal firms.

While limitations in procurement data (discussed in detail in Section 4.1) prevent the corroboration of this observation, the evidence suggests that focusing on Aboriginal participation in the economy via procurement may require a more modernized approach. From the literature and interviews, some key methods to approach this could include:

  • bolstering existing or creating new programs to assist Aboriginal businesses with the training and knowledge needed to obtain federal government contracts;
  • facilitating coaching and mentorships on competing for and successfully completing projects;
  • supporting or facilitating the creation of an employment/business network databases for Aboriginal people (i.e., similar to LinkedIn but specific to Aboriginal people); and
  • further strengthening the promotion of Aboriginal businesses to increase awareness of their services, and reduce existing misconceptions of increased cost and difficulty commonly associated with conducting business with Aboriginal firms.

3.2 Alignment with federal government priorities and departmental strategic outcomes

Finding: The objectives of the PSAB are well-aligned with government priorities and departmental strategic outcomes. The approaches to meeting these objectives need to continually adjust to evolving needs and business environments to ensure optimal contribution to these objectives.

Budgets 2008-2014 have emphasized the Government of Canada's commitment to improving Aboriginal workforce participation in the Canadian economy through training initiatives, the Federal Framework for Aboriginal Economic Development and the emphasis on strategic partnerships between the private, government and Aboriginal community sectors. Some specific programs that have emerged from the budgets also include the creation of the Aboriginal Skills and Employment Partnership Initiative;Footnote 14 the creation of the Aboriginal Skills and Training Strategic Investment Fund;Footnote 15 the support for partnerships between the private sector, Aboriginal groups and governments for skills and employment opportunities;Footnote 16 support for a renewed Urban Aboriginal Strategy;Footnote 17 support for Cape Breton University's Purdy Crawford Chair in Aboriginal business studies;Footnote 18 and continued support for the Skills and Partnership Fund, the Aboriginal Skills and Employment Training Strategy and provisions of the Economic Action Plan 2013, which seeks to help on-reserve First Nations people on Income Assistance find employment.Footnote 19 Speeches from the Throne and federal budgets from the past six years have dedicated millions of dollars towards increasing Aboriginal participation in the economy, consistent and complimentary with the objectives of the PSAB.

The Government of Canada has also stated that "while serving its primary purpose of obtaining required goods and services at market rates, procurement is also a tool for Aboriginal businesses to grow by gaining experience developing capacity, and forming partnerships with other businesses to compete for procurement opportunities."Footnote 20 The Government has invested $8.3 million annually from 2009-2013 in PSAB through its implementation of the Federal Framework for Aboriginal Economic Development.Footnote 21 PSAB is considered an essential element of this framework, which authorizes AANDC to strengthen its approach through initiatives such as seeking enhancements of Treasury Board policy concerning Aboriginal procurement; implementing procurement readiness training for Aboriginal businesses; conducting outreach activities; engaging with the provinces to launch similar programs; and increasing the number of set-asides.Footnote 22

The PSAB also supports AANDC's mandate to "improve social well-being and economic prosperity," create healthy Aboriginal communities, and foster Aboriginal participation in the Canadian economy.Footnote 23 Additionally, Departmental Reports on Plans and Priorities from 2009-2014 emphasized the importance of the creation of strategic partnerships between the private sector, several levels of government, Aboriginal representatives and other stakeholders; the strengthening of Aboriginal Entrepreneurship; support for Aboriginal businesses; and increasing the percentage of Aboriginal procurement relative to total federal spending to two percent.Footnote 24 In AANDC's 2014-2015 Estimates: Report on Plans and Priorities, PSAB has been recognized for supporting "the identification of other public and private-sector business opportunities and facilitates Aboriginal access to these opportunities through a variety of partnership and participation based approaches."

It is important to consider, however, the extent to which PSAB activities can be further strengthened to better support departmental strategic outcomes related to economic participation. With the key program outcome of PSAB being that Aboriginal firms win federal procurement contracts, which in turn, along with other entrepreneurship programs, is expected to contribute to the creation and/or expansion of viable Aboriginal businesses, the link between these objectives needs to continually account for evolving business and economic conditions, which may change the ways in which Aboriginal businesses are best supported in this context. For example, in considering the issues raised in Section 3.1 on whether PSAB may in fact largely be supporting larger and more established businesses, new initiatives may be necessary to further support budding and smaller Aboriginal business and new entrepreneurs. This could entail different approaches to supporting larger and more established versus smaller or newer businesses, such as a graduate approach to supporting businesses of incremental size and experience to provide different types of support as needed to different types of businesses. For example, in the United States, the participation time for set-asides for minority businesses is divided into two stages: one four-year developmental stage and another five-year transition stage. Firms and owners are only permitted to participate in the program once. This helps limit reliance on the program and encourage firms to compete in the open market.Footnote 25 This differs from PSAB, which has very few limitations once a firm has been approved for use of the program. The United States also requires that firms using the program are classified by the North American Industry Classification system as "small," as to limit large firms monopolizing the set-aside contracts.Footnote 26 PSAB does not have any business size limitations on the use of its program.

The United States program also has limits on the dollar value of the government contracts. In the program, firms may receive sole-source contracts up to US$4 million for goods and services and US$6.5 million for manufacturing contracts. The total dollar value of the sole-source contracts (assumingly excluding joint venture contracts) is $100 million over the duration of the nine year program. Suppliers also must conduct business in their commercial sector, as well as in the government program.Footnote 27 To help businesses achieve this balance, businesses must have a detailed business plan with objectives, targets and goals in the commercial and government sectors.Footnote 28 Finally, in order to ensure that businesses are progressing, the United States Small Business Administration monitors the businesses through the examinations of business planning activities, annual reviews and evaluations.Footnote 29 PSAB has no such limits, which may in part explain why nine percent of firms indicated in the last evaluation that all of their revenue was due to PSAB set-asides, suggesting that some firms relied on PSAB for survival.Footnote 30 Further details on various other approaches are included in Appendix C.

3.3 Consistency with Federal Roles and Responsibilities

Finding: The current activities under the PSAB are consistent with federal roles and responsibilities.

The Government of Canada has committed within the Federal Framework for Aboriginal Economic Development to "remove legislative and regulatory barriers that deter business development; increase access to debt and equity capital; improve procurement opportunities; strengthen capacity of entrepreneurs to succeed in business; and accommodate the real needs, conditions and opportunities facing different community in all regions of the country."Footnote 31 Within the Federal Framework for Aboriginal Economic Development, the Government is viewed as an entity that helps create the conditions for economic development, to foster the creation for partnerships and to make strategic investments in order to facilitate Aboriginal economic development. The Federal Framework for Aboriginal Economic Development has given the Government of Canada a distinct role in implementing the PSAB. An Interpretation Bulletin was released updating and clarifying the roles and responsibilities that the Government of Canada has towards procurement opportunities for Aboriginal businesses. It reiterates the responsibilities of applicable government departments and agencies towards PSAB, including the setting of set-aside targets and support for complementary measures such as the Aboriginal Business Directory.Footnote 32 Importantly, the responsibility to implement PSAB does not conflict with Canada's obligations to the North American Free Trade Agreement or World Trade Organization Agreement on Government Procurement for "non-discriminatory" competition, as the agreements are not applicable to procurement with respect to set-aside for small or minority businesses.Footnote 33 Therefore, the Government of Canada has stated that it has a role and responsibility in the implementation of the PSAB program and these obligations do not conflict with international trade agreements.

While federal government programs are generally complementary in nature to PSAB, private and Aboriginal organization procurement initiatives are similar in scope, nature and practice to PSAB. Some complementary initiatives include the National Shipbuilding Program Strategy; the Aboriginal Business Development Program; Aboriginal Financial Institutions; Canada's Economic Action Plan; the Apprenticeship Completion Grant; Aboriginal Skills and Employment Training Strategy; Aboriginal Skills and Employment Partnership Initiative; Aboriginal Skills and Training Strategic Investment Fund; and the provincial procurement initiatives in Manitoba, Ontario and Nunavut. The closest government Aboriginal procurement policy that also exists is The First Nation Tendering Policy. This policy seeks to increase the participation of qualified Aboriginal firms First Nations contracting, and encourages the development of partnerships between Aboriginal businesses and other sectors.Footnote 34

Aboriginal organizations have procurement programs and supportive initiatives that have similar objectives to PSAB. The Aboriginal Human Resource Council has an "Aboriginal Procurement Training" course, which consists of comprehensive training and advisory services to help organizations develop and deliver successful procurement programs. This training consists of workshops, advisory services and other resources.Footnote 35 The Canadian Council for Aboriginal Business also hosts an Aboriginal Business Directory, a Progressive Aboriginal Relations Program, a Certified Aboriginal Business Program, and an Aboriginal Business Mentorship Program.Footnote 36 Finally, the Canadian Aboriginal and Minority Supplier Council offers certification; an Aboriginal Minority and Supplier Database; educational seminars; training and advisory services; Corporate and Vendor Guidelines; referrals to corporate buyers; and networking opportunities.Footnote 37 As will be iterated below, there is evidence of collaboration between the PSAB and these organizations, especially with PSAB's participation in conferences.

Survey respondents suggested it was necessary for the Government to take a more active role in building awareness respecting Aboriginal businesses, and providing better and more training for both Aboriginal businesses and government sectors with a view to improving and increasing the use of Aboriginal firms in procurement. Specifically, there was a strong sentiment that there was a need to better ensure consideration of Aboriginal set-asides and Aboriginal firms generally wherever possible. Additional suggestions also included increasing the emphasis on young entrepreneurs and providing more coaching and mentorships to budding Aboriginal businesses, as well as investing in the capacity development with a view to helping Aboriginal entrepreneurs better align themselves with current industry demands.

A key objective of the PSAB is to focus the role of the federal government in supporting Aboriginal business. Most interview and survey respondents generally agree that it does this insofar as encouraging the use of Aboriginal firms, and with respect to being the primary economic driver on many reserve lands – especially those in rural areas and the territories – as well as in some sectors (e.g. social, education, health and housing). In other words, to the extent possible, PSAB contributes to the general objectives of the Federal Framework for Aboriginal Economic Development by supporting efforts to contract work to be done on-reserve to Aboriginal businesses. Further, for reserves that are not close to urban centres, there is potential for PSAB to have an impact on the viability of businesses because a large portion of the business activities on-reserve is government-based.

Key informants emphasized a need for a specific approach to procurement in the North, as the business opportunities are markedly different than in the South, and budding entrepreneurs and businesses would benefit from training and coaching in how to link with larger firms, or themselves strengthen to take on larger projects, to better respond to these economic drivers.

 

 

4. Evaluation Findings – Performance (Effectiveness / Success)

4.1 Achievement of Expected Outcomes

4.1.1 Program Outcomes: Aboriginal firms win contracts and Contribution to the Creation or Growth of Viable Aboriginal Businesses

Finding: The PSAB is resulting in Aboriginal firms winning an increased share of contracts over time, and significant strides have been made in promoting procurement with Aboriginal firms. Data collection to date, however, does not allow for a complete analysis on whether the PSAB is truly supporting the creation and expansion of viable Aboriginal businesses, and ultimately does not allow for a complete illustration of performance.

Survey and interview participants largely agreed that PSAB has had a positive impact on:

  • the value of federal contracts (both via set-asides and the general process);
  • investment in Aboriginal firms and start-ups, the capacity development of firms, and the development of partnerships between Aboriginal and non-Aboriginal businesses; and
  • building strategic partnerships between the Government of Canada, Aboriginal people, the private sector and provinces and territories.

There is much less certainty, however, as to whether PSAB has had an impact on Aboriginal entrepreneurship, partnerships that have actually maximized economic opportunities, the development of Aboriginal human capital, and focusing the role of the federal government in contributing to Aboriginal economic success.

Respecting Aboriginal entrepreneurship, it is difficult to attribute any incremental improvements over time to government procurement or policies and promotion regarding set-asides with Aboriginal firms. Improvements in Aboriginal entrepreneurship could be gauged by measuring the growth of Aboriginal firm contracts (specific to each firm) and the emergence and relative growth of new firms over time. In order to do this, there would need to be data on the value and frequency of contracts specific to each Aboriginal firm over time. Until 2012, data that would allow for the assessment of the value and frequency of contracts by firm had not been collected. Critically, the new data collection mechanisms that now include these variables only do so for set-aside arrangements and not for incidental contracts. Given that the PSAB objectives are set toward Aboriginal firms winning contracts in general and contributing to the creation or expansion of viable businesses, current data collection allows for only a portion of the performance story respecting PSAB and its potential impacts on Aboriginal entrepreneurship. The key concern respecting this issue that was raised by key informants was that PSAB may be acting as a vehicle to continue to provide existing Aboriginal businesses with contracts but may be doing little for newer businesses that have not gained the experience to date to be competitive in the bidding process, or even to qualify for a set-aside or incidental contract.

As shown in Figure 1, existing data suggest that the total value of all federal government contracting to Aboriginal firms (set-asides and incidental combined) relative to total spending on procurement has shown an increase in recent years, and well exceeds its performance management framework target of two percent by 2015. However, there are significant concerns with the reliability of data for incidental procurement contracts with Aboriginal firms. Critically, the criteria for a business identifying as "Aboriginal" is poorly defined and unverified. A firm may identify as Aboriginal for the purpose of a contract for various reasons, including that there are a number of Aboriginal people employed under the specific contract. Federal government interviewees raised concerns about this issue; specifically, that the resources originally in place to monitor compliance have been significantly reduced; there are little, if any, official verifications that Aboriginal firms are actually completing the work (particularly in the cases of partnerships); and there is concern that there may be abuse of the program in this regard.

The main issue in reporting on performance is that the data collection approach before 2012 did not allow for classification of value and frequency by business, thus preventing any proxy analysis of incidental procurement with Aboriginal firms.

Figure 1: Mean Value of Incidental Contracts by Year (left y-axis; bars) and the Mean Relative Proportion of Total Government Spending on Procurement (right y-axis; line)

Mean Value of Incidental Contracts by Year (left y-axis; bars) and the Mean Relative Proportion of Total Government Spending on Procurement (right y-axis; line
Description of Figure 1

This graph shows years along the x-axis from 1997 to 2011, the mean value of incidental contracts along the primary y-axis and the mean relative proportion (in percent) of total government spending on procurement along the secondary y-axis. The primary y-axis shows a gradual increase from about $25 million in 1997 to close to $200 million in 2002; increasing sharply to $400 million in 2003; decreasing to about $200 million in 2004, $250 million in 2005, then back up to $400 million in 2006; gradually back down to under $200 million by 2008; $400 million in 2009; over $500 million in 2010 and about $700 million by 2011. The secondary y-axis trend line follows the exact same pattern, starting at about 0.3% in 1997, reaching 3% in 2003, varying until 2008 and sharply increasing from 1% in 2008 to 4.5% in 2011.

 

In specifically examining set-asides, their total value has varied over time, as has the value as a percentage of total federal procurement (see Figure 2). Many variables can influence these figures such as one-time large projects and variability in departmental budgets.

Figure 2: Mean Value of Set-Aside Contracts by Year (left y-axis; bars) and the Mean Relative Proportion of Total Government Spending on Procurement (right y-axis; line)

Mean Value of Set-Aside Contracts by Year  (left y-axis; bars) and the Mean Relative Proportion of Total Government  Spending on Procurement (right y-axis; line)
Description of Figure 2

This graph shows years along the x-axis from 1997 to 2011, the mean value of set-aside contracts along the primary y-axis and the mean relative proportion (in percent) of spending on set-asides along the secondary y-axis. The primary y-axis shows a sharp increase from about $20 million in 1997 to a peak of $150 million in 2001; varying between $50 million and just under $100 million between 2002 and 2011, finishing at about $80 million in 2011. The secondary y-axis shows generally the same trends, starting at about 0.3% in 1997m reaching a peak of 1% in 2001, and varying from 2002 to 2011, finishing at about 0.6% in 2011.

 

It can be stated, however, that the total value of set-asides and their relative proportion of total federal government procurement had increased steadily between 2009 and 2011 (the most recent year with data organised in this fashion). Further, data collection methods effective 2012 allow for an assessment of distinct businesses and growth. For 2012 (the most recent year currently available), there were set-asides with 124 businesses, with a total of 1,424 distinct contracts totaling close to $109 million in value.

A component of the PSAB is to ensure that relevant stakeholders are familiar with and aware of the related policies and processes. As of 2012, over 120 information and outreach sessions had been held throughout Canada in order to promote Aboriginal procurement opportunities.Footnote 38 Public Works and Government Services Canada and the Office of Small and Medium Sized Enterprises have held information booths on public procurement at Aboriginal Businesses Conferences.Footnote 39 The Office of Small and Medium Enterprises has hosted seminars to help suppliers understand procurement with the federal government; to register in the database; to build relationships; to conduct procurement searches; to obtain a security clearance; and to bid on tenders.Footnote 40 Approximately 65 training activities and outreach sessions with federal, provincial and territorial governments and businesses are held per year.Footnote 41 The 2010-2011 Departmental Performance Report noted that 37 procurement readiness training sessions were delivered to Aboriginal communities about government procurement procedures and opportunities.Footnote 42 Finally, the Update on the Implementation of the Federal Framework for Aboriginal Economic Development noted that PSAB had largely met its performance targets, which included conducting over 120 information and outreach sessions across Canada to promote Aboriginal procurement opportunities and tools.Footnote 43 PSAB has reached out to Aboriginal businesses through training and outreach sessions; however, there are no reports that detail outcomes related to these sessions.

The PSAB has also included procurement training sessions with government employees and requires that departments set targets for Aboriginal procurement. In 2009-2010, AANDC provided procurement-readiness training to 43 departments and agencies reaching over 900 federal procurement officers.Footnote 44 As of 2011, AANDC had hosted a national training exhibition in Winnipeg, which reached over 150 participants, including PSAB coordinators and government procurement officers.Footnote 45 Finally, the Canada School of Public Services has a free online training tool entitled "Aboriginal Considerations in Procurement (C223), which introduces public servants to their obligations with Aboriginal procurement and comprehensive land claims agreements.Footnote 46 Thus, there have been significant strides to increase awareness of the PSAB in the federal government.

Regarding targets for set-asides, AANDC has negotiated with department contracting authorities to increase the number of Aboriginal set-asides.Footnote 47 Unfortunately, reporting of targets is not required and data are not collected. Among federal government survey respondents, about 45 percent said their departments had set targets for set-asides. Twenty-three respondents said their departments had not set targets, and 27 weren't sure. Among those who knew their department had set targets, 41 respondents said they were being met, while three said they were not and 21 weren't sure. When asked what mechanisms are most responsible for encouraging departments to meet their performance objectives respecting set-asides, they mainly attributed it to contract stipulations/Public Works and Government Services Canada tools and registration processes; awareness amongst suppliers; departmental culture; and training and courses.

Training sessions for government employees such as mandatory training for program staff under the Aboriginal Entrepreneurship sub-activity are now being implemented on an ongoing basis, and partnerships with government stakeholders have been actively pursued to build the stability of the program. AANDC has signed a Memorandum of Understanding with the Office of Small and Medium Sized Enterprises to conduct outreach sessions with Aboriginal businesses and to promote PSAB.Footnote 48 An agreement was also drafted with Industry Canada so that AANDC would be able to register businesses on the Aboriginal Business Directory and to use registration forms to gather capacity data.Footnote 49 A Memorandum of Understanding has been developed with Industry Canada to house the Aboriginal Business Directory online, which enables Aboriginal businesses to register under the PSAB. AANDC has also engaged in dialogue with Public Works and Government Services Canada to improve data collection on Aboriginal procurement.Footnote 50 Finally, AANDC has also made strides to promote the model to the provinces. Manitoba and Ontario have made Aboriginal procurement policies with the advice from the Department and work has been done to extend the model to Quebec, British Columbia and Alberta.Footnote 51

Six pilot projects in sectors, including potash and shipbuilding have been supported to focus on providing a "diagnostic on procurement policies and opportunities for Aboriginal businesses; a feasibility study on the creation of long-term procurement model, procurement and business/ financial literacy training sessions, workshops and conferences."Footnote 52 Finally, in order to address the capacity gaps of Aboriginal suppliers in numerous sectors, eight pilot projects have been developed that will focus on data analysis, networking opportunities, assessment of Aboriginal capacity, the enhancement of business and financial literacy, and opportunities for Aboriginal businesses in major projects.Footnote 53

Key informants also noted that partnerships and joint venturing with other firms provide project experience, which in turn increase the Aboriginal businesses opportunities for additional federal government contracts. Respecting partnerships, two key observations were that: 1) Industry – be it government or the private sector – is starting to see the benefits of partnering with Aboriginal businesses, which ultimately benefits all partners involved; however 2) there is a risk of "partnerships of convenience," where Aboriginal businesses may partner with larger firms so the contract is won, but ultimately, the non-Aboriginal firm conducts most of the work and the Aboriginal firm thus gains little if any experience from the project.

4.1.2 Creating a leverage effect within the Federal Framework for Aboriginal Economic Development

Finding: The expectations of a leverage effect resulting from PSAB were never articulated or measured, and this is no longer included as an official outcome of PSAB.

Since 2006, AANDC has worked with remediation teams across government to develop various approaches and provide advice on implementing and negotiating either a PSAB set-aside or Aboriginal participation component in the development of the procurement strategies for the remediation projects. Specifically, AANDC was able to negotiate set-asides through outreach, providing Aboriginal business capacity searches and providing an overview of PSAB.

For example, in early 2012, AANDC worked with Public Works and Government Services Canada to provide advice and an overview of PSAB for the Esquimalt Graving Dock Waterlot Remediation project for multiple phases of the project. The total value is approximately $85 million dollars of which a set-aside for one phase was for $6.5 million dollars on the Erosion Protection Wall project, and the next phase estimated between $2.5 to $3 million.

This example illustrates the potential leveraging of federal investment respecting set-asides; however, AANDC has not developed a clear method to measure a true leverage effect with most of its projects. The funding provided via Grants and Contributions through both the PSAB and the Aboriginal Workforce Participation Initiative is intended to stimulate business networks but the reporting on these investments does not capture performance in a way that would allow for the measurement of true leverage. Ultimately, what this leveraging would look like, and any related target-setting, was never fully articulated or developed. This outcome is not included in the 2014 Aboriginal Entrepreneurship Performance Measurement Strategy.

4.1.3 Other Outcomes of the PSAB and Aboriginal Workforce Participation Initiative

Finding: Outcomes on the adoption of Aboriginal procurement strategies or participation agreements; and the outcome that Aboriginal business capacity is matched with business/procurement opportunities have only recently been established and articulated. PSAB and Aboriginal Workforce Participation Initiative initiatives have resulted in significant research related to potash, shipbuilding, and electricity sectors, and information and training sessions focused on business development competencies, procurement and business promotion and activities promoting Aboriginal women in business.

The 2014 Aboriginal Entrepreneurship Performance Measurement Strategy includes additional outcomes related to access to business opportunities; specifically that: partners adopt Aboriginal procurement strategies/participation agreements and support access to business opportunities; and that Aboriginal business capacity is matched with business/procurement opportunities.

In 2011, AANDC funded a research study through PSAB into ways that businesses operating in the electricity and renewable sector can increase their procurement of goods and services from Aboriginal businesses. The project was designed to increase awareness for Aboriginal businesses of the various opportunities within these sectors, examine the benefits of having an Aboriginal procurement strategy, and provide the foundation and a template for businesses looking to develop an Aboriginal procurement strategy of their own.

To date, however, there are no other reports that clearly link PSAB and Aboriginal Workforce Participation Initiative Grants and Contributions, or other activities, to the above-mentioned outcomes.

There have been several initiatives, particularly since 2008, which have promoted various entrepreneurship projects, conferences and seminars, and research projects. These have included Aboriginal Entrepreneurship Conferences and Tradeshows and other seminars intended to bring together human resources and Aboriginal diversity leaders. There were also significant projects undertaken in conjunction with the 2010 Vancouver Olympic Games, including a "Vancouver Olympics High End Art Engagement" with the Inuit Tapiriit Kanatami, which celebrated Inuit art; and the design of an Aboriginal pavilion.

Other highlights include:

  • Funding research for a feasibility study by the Women's Economic Council: Regional Clustering Model for Aboriginal Women to develop a model for custom-fit programming, services, resources and networks relevant to supporting the needs of individual women and/or organizations supporting women in specific regions and/or circumstances (northern Inuit, First Nation, rural, etc.).
  • Funding the Canadian Aboriginal Arts and Communication Business Studio: An Aboriginal skills, entrepreneurship and training centre, via the Ontario Native Literacy Coalition. A business studio was developed, along with training programs, and efforts to leverage funds via multiple other organizations and programs.
  • Funding the organisation I Do Business for training master facilitators to train on the "Alexis principles." Fourteen people were trained as master facilitators, and a seminar was held with 92 participants and a planned delivery to up to 350 additional participants for 2013 (to train on event management and facilitating workshops on business development).
  • Funding the Proteus Canada Institute and the Idea Connector Network to produce a series of interviews and panel discussions on topics related to "Empowering Women"; to produce a panel discussion with the participation of a mixed audience (town-hall type setting with online live participants from across Canada) as part of the Aboriginal Entrepreneurs Conference and Tradeshow; to produce a series of interviews on business start-ups and related issues; and to promote the Idea Connector Networks' productions via traditional and social media.
  • Funding the Wachiay Friendship Centre to deliver the program "Small scale entrepreneurship for disadvantaged women: An Opportunity to break the cycle of the working poor." This project was designed to include a business incubator facility, training workshops, and one-to-one mentoring. A business model, operations manual and strategic partnerships with 14 other organizations were created.
  • Supporting the British Columbia Aboriginal Business Association in regional forums to provide knowledge-sharing and business skills development delivered by professionals from eleven corporate partners, as well as the development of a marketplace project.
  • Supporting the development of an Aboriginal Women's Entrepreneur Network – a series of activities under four pillars: establishing a National Aboriginal Women's investment fund; promoting financial knowledge, literacy and development; emphasizing asset-based development for Aboriginal women; and actively engaging young girls in future economic leadership.

4.1.4 Factors affecting the success of the PSAB

Finding: The main issues affecting the success of PSAB relate to the complexity of the procurement process and challenges in securing opportunities to gain the experience necessary to be qualified for government contracts.

Concerns stemming from the key informant interviews corroborate many of the points raised in Section 3 on whether the current approach may be the most effective way to contribute to the creation or expansion of Aboriginal business. Specifically, there is a need to further support Aboriginal businesses in light of the complexity of the procurement processes, an increase in competitiveness caused by a decrease in the number of Request for Proposals being issued by the federal government resulting in a decrease in the percentage of contracts won versus the number of proposals being produced, and a decrease in per-diems all culminating to reduce the profitability of government contracts. Ultimately, the key informants suggested that without previous federal government contracting experience gained elsewhere, new Aboriginal businesses are not actively seeking federal government service contracts due to the complexity and barriers of government procurement processes.

Based on the feedback from key informants, an Aboriginal firm that does not have the requisite project experience and staff with the required number of years of relevant experience, will likely not win a federal government contract unless it is a PSAB set-aside and the mandatory and point-rated requirements have been loosened so that a less experienced bidder can qualify and be awarded the contract.

As noted by one of the key informants:

"Our experience has been that First Nations businesses [without previous government contracting experience] need advisors or support to help them write their first few proposals. There is a general lack of understanding amongst first time proposal writers and new businesses that governments use a rigorous proposal evaluation process and that the proposals must address the requirements that are specified within the Request for Proposal; if they don't do that they don't win the job."

This issue speaks to the need to strengthen efforts focusing on strengthening viable businesses, discussed in 3.1. The work to provide set-asides and promote the use of Aboriginal firms would be significantly bolstered by further efforts to help budding Aboriginal businesses and entrepreneurs better match their skills and capacity to emerging needs, and better understand the contracting process. It was noted in Section 4.1 that there are numerous activities ongoing via both PSAB and the Office of Small and Medium Enterprises to promote PSAB and provide outreach and information to businesses; however, as mentioned, there are no measures on the effectiveness of this work and a general consensus among interviewees that this is most in need of newer and more strategic approaches.

Some key informants stated that the complexity of the procurement processes, combined with an increase in competition due to reductions in contractual spending, has resulted in a decrease in the percentage of contracts won versus the number of proposals being produced, which may in turn mean increased cost with a lower likelihood of success for any given business (Aboriginal or otherwise). It is important to consider that while reductions in contractual expenditures and increased competitiveness is a reality for all firms, when considering the objective of increasing the relative percentage of firms winning contracts that are Aboriginal, there may be an increased need to further assist Aboriginal firms with increasing their competitive advantage through direct training and mentorships.

According to interviewees, qualifying under for set-asides, particularly as laid out in Task and Solutions Professional Services, requires that the Aboriginal firm has undertaken and completed a number of contracts during the last few years. This requirement to have completed a number of projects is at cross-purposes with PSAB's goal of facilitating access to federal government contracts, as issuing a PSAB set-aside under new Task and Solutions Professional Services will likely impede an Aboriginal firm to be a first-time bidder for that potential contract. Some interviewees felt that access to government contracting is decreasing because of these new barriers and what is perceived as a more demanding and complex procurement process. Without previous federal government contracting experience gained elsewhere, new Aboriginal businesses may not actively seek federal government contracts due to the complexity and barriers of government procurement processes, and because of decreased government contracting opportunities in general, opportunities for new Aboriginal businesses are becoming more limited. Thus, as discussed in Section 3.1, without a strengthened focus on building the business capacity of newer or smaller firms so that they can become more competitive in winning contracts, there is a risk that over time, PSAB could act to sustain a small number of well-established Aboriginal firms, as opposed to contributing to an increase in Aboriginal representation in procurement overall.

An additional concern raised by key informants related to the posting period for set-asides. A 25 day posting option is available to project authorities in order to incentivize the use of set-asides. However, some key informants raised concern that in some cases, this can actually pose a problem for bidders in that the turn-around time for proposals can be unfairly short compared to the standard 40 day posting period. Ideally, the 25 day posting option would only be used in cases where the contract is simple, such as purchasing and re-selling. For more complex projects, the posting period should remain the standard 40 days.

Importantly, however, interviewees generally felt that the biggest barrier to Aboriginal representation in business is related to education and job experience. There are significant and well-documented economic challenges facing Aboriginal peoples, and it is important to consider that there are significant socio-economic forces at play that are well-beyond the scope of what PSAB and its related activities can achieve.

4.3 Efficiency and Economy

Finding: The PSAB and its related activities are generally speaking an economical approach to supporting Aboriginal businesses through the Federal Framework for Aboriginal Economic Development. The significant expenditures currently invested in large conferences, however, may be better spent on targeted training initiatives designed specifically to match businesses with economic opportunities; adequately train them on how to bid in competitive processes; and how to be more competitive and take better advantage of regional opportunities.

The value of federal investment in providing assistance to Aboriginal businesses to unlock business opportunities is significant relative to the cost of the PSAB and its related activities. The coordination of efforts by AANDC and its federal, Aboriginal, and private sector business partners has opened access to millions of dollars in benefits for Aboriginal businesses and communities. For example, the construction of a penitentiary near a reserve and the resulting linkages made via the Aboriginal Workforce Participation Initiative has allowed businesses in and around that community to take advantage of $200 million in long-term employment and services, which ultimately required very little financial investment on the part of the federal government.

Ultimately, while two-thirds of PSAB funding is for Salary and Operation and Maintenance (including employee benefits), its economy is partially demonstrated by a low operating budget relative to the steady increase in the value of contracts awarded to Aboriginal firms ($109 million in 2012).

When asked, key informants were unable to identify any areas where the implementation of PSAB and Aboriginal Workforce Participation Initiative activities could be more efficient. However, there may be an issue respecting economy with the significant amount that AANDC invests in expenditures on information sessions and conferences. Considerable internal resources are used to host information sessions, and large scale contracts, in addition to internal resources, are used for larger conferences that are intended to promote PSAB and government contracts, and link the Aboriginal business community together. The concern raised by key informants and survey respondents was regarding the overall utility of these sessions and conferences. Specifically, it was a recurrent theme that while promotion and networking are essential, large conferences may entail costs that well exceed their relative benefit with respect to the degree to which Aboriginal businesses benefit – particularly considering the cost of traveling to, and attending the events.

Typically, the cost of each of these sessions is approximately $250,000 excluding internal resources; however, while federal government representatives tended to view these as being useful and productive events, and while most participants in general agreed that these events provided an occasion for networking, non-government interviewees who attended one or more of these PSAB events were of the opinion that the events were of limited usefulness. It was suggested that regional conferences focusing more on how to win a government contract would be more beneficial and would allow more small Aboriginal businesses to participate. Arguably, the return on investment in having more direct exposure to topics of regional contracting needs and how to become more competitive, would be a better use of funds.

 

 

5. Conclusions and Recommendations

Conclusions

There is a clear need for the Government to help Aboriginal businesses gain experience through promotion and set-asides. There is a concern, however, that over time PSAB may become more of a mechanism to support already well-established Aboriginal businesses, and may have less relevance to smaller and newer businesses or budding entrepreneurs. Government procurement in general is designed to favour larger and more established businesses; particularly given the nature of most current government projects, and it is not evident that the PSAB is impacting businesses outside of a core of larger and well-established firms.

The evidence collected for this evaluation produced the following findings:

With respect to relevance:

  1. There is a need for continued investment in initiatives designed to strengthen viable Aboriginal businesses. Though the PSAB supports this objective, there is concern that the approach generally favours larger and more established firms over new and smaller businesses and entrepreneurs.
  2. The objectives of the PSAB are well-aligned with government priorities and departmental strategic outcomes. The approaches to meeting these objectives need to continually adjust to evolving needs and business environments to ensure optimal contribution to these objectives.
  3. The current activities under the PSAB are consistent with federal roles and responsibilities.

    With respect to performance:

  4. The PSAB is resulting in Aboriginal firms winning an increased share of contracts over time, and significant strides have been made in promoting procurement with Aboriginal firms. Data collection to date, however, does not allow for a complete analysis on whether the PSAB is truly supporting the creation and expansion of viable Aboriginal businesses, and ultimately does not allow for a complete illustration of performance.
  5. The expectations of a leverage effect resulting from PSAB were never articulated or measured, and this is no longer included as an official outcome of PSAB.
  6. Outcomes on the adoption of Aboriginal procurement strategies or participation agreements; and the outcome that Aboriginal business capacity is matched with business/procurement opportunities have only recently been established and articulated. PSAB and Aboriginal Workforce Participation initiatives have resulted in significant research related to potash, shipbuilding, and electricity sectors, and information and training sessions focused on business development competencies, procurement and business promotion and activities promoting Aboriginal women in business.
  7. The main issues affecting the success of PSAB relate to the complexity of the procurement process and challenges in securing opportunities to gain the experience necessary to be qualified for government contracts.
  8. The PSAB and its related activities are, generally speaking, an economical approach to supporting Aboriginal businesses through the Federal Framework for Aboriginal Economic Development. The significant expenditures currently invested in large conferences, however, may be better spent on providing better access to targeted training initiatives designed specifically to match businesses with economic opportunities; adequately train them on how to bid in competitive processes; and how to be more competitive and take better advantage of regional opportunities.

Recommendations

It is therefore recommended that AANDC:

  1. Develop an enhanced approach to the PSAB that is tailored to the different needs of different types of business, including a stronger focus on direct and regional training to support newer and smaller Aboriginal firms to navigate the increasingly complex and competitive procurement environment;
  2. Work with Public Works and Government Services Canada to ensure ongoing performance data allows for a complete capture of data on individual businesses winning procurement contracts by value and type both for set-asides and incidental contracts;
  3. Develop better accountability mechanisms for the accurate capture of whether or not bidders qualify as Aboriginal; and
  4. As part of the promotion of PSAB, work with contracting authorities to ensure the best likelihood of Aboriginal business success, including promoting the most appropriate application of the 25-day posting option.
 

 

Appendix A – PSAB Logic Model (as per 2008 RMAF-RBAF)

Strategic Priority Increased participation of Aboriginal people and Northerners in the economy
Components Procurement/ Set-asides Capacity Building
Activities Help Aboriginal firms do more contracting with all Government of Canada departments and agencies enhance the skills and know-ledge of all participants
Outputs # of contracts and subcontracts awarded to Aboriginal business # of training opportunities for Aboriginal business owners and staff
Increase in the value of contracts # of Aboriginal firms who are Pre-qualified on the Strategis Website
Immediate Outcomes Increase in contracting opportunities for Aboriginal business Aboriginal business expands due to ability to compete for federal contracts
Intermediate Outcomes Growth in Aboriginal business as a result of contracting with the federal government Aboriginal business expands its government market base to include incidental contracts
Long Term Outcome Improved socio-economic conditions in Aboriginal communities
 

 

Appendix B – Aboriginal Entrepreneurship Logic Model

Logic Model for Program 3.1 (Aboriginal Entrepreneurship) Includes
Sub-Programs 3.1.1 (Business Capital and Support Services) and 3.1.2 (Business Opportunities)

 
Logic Model for Program 3.1
Description of Logic Model for Program 3.1

The logic model is oriented from top to bottom. On the left is the title for each row. The first title on top is: Activities; the second title is: Outputs; the third title is: Immediate Outcomes; the fourth title is: Longer Term Expected Results and the last title at the bottom is: Program Expected Result.

The first row (Activities) on top is composed of five boxes. The first box on the left is: Provide funding support. The second box is: Provide operational support and facilitate capacity development. The third box is: Develop cooperation agreements with key stakeholders and partners. The fourth box is: Promotion, advocacy and training and the last box is: Identify procurement opportunities and business capacity. From the first box on the left, there is a black arrow pointing down to the first box on the left in the second arrow. From the second box, there is a red arrow pointing down, split in half and connected to the second and third box in the second row. From the third box, there is a blue arrow pointing down, split in four and connected to the first, second, third and fourth boxes in the second row. From the fourth box, there is a black arrow pointing down to the fourth box in the second row. From the last box on the right, there is a black arrow pointing down to the last box in the second row.

The second row (Outputs) is composed of five boxes. The first box on the left is: Funding agreements. The second box is: Operational support and training. The third box is: Frameworks, strategies and agreements/MOUs. The fourth box is: Promotion material, information products and training sessions and the last box is: Aboriginal business directory. From the first box on the left, there is a red arrow pointing down, split in four and connected to the first, second, fourth and fifth box in the third row. From the second box, there is a black arrow pointing down, split in half and connected to the first and second box in the third row. From the third box, there is a green arrow pointing down, split in three and connected to the third, fourth and fifth box in the third row. From the fourth box, there is a blue arrow, split in four and connected to the second, third, fourth and fifth box in the third row. From the last box on the right, there is an orange arrow pointing down and connected to the fifth box in the third row.

The third row (Immediate outcomes) is composed of five boxes. The first box on the left is: Capital pools for Aboriginal business development are established, expanded and diversified. The second box is: Aboriginal Institutions have the capacity to deliver business capital and support services. The third box is: Federal government procurement contracts are set-aside for Aboriginal businesses. The fourth box is: Partners adopt Aboriginal procurement strategies/participation agreements & support access to business opportunities and the last box is: Aboriginal business capacity matched with business/procurement opportunities. From the first and second box, there is a black arrow pointing down to the box on the left in the fourth row. From the third, fourth and fifth box, there is a black arrow pointing down to the box on the right in the fourth row.

The fourth row (Longer term expected results) is composed of two boxes. The box on the left is: A sustainable network of Aboriginal Financial Institutions. The box on the right is: Aboriginal businesses win procurement contracts. From the two boxes, there is a black arrow pointing down to the box in the fifth row.

The last row (Program expected result) is composed of one box. The box is: Creation and/or expansion of viable Aboriginal businesses.

 
 

 

Appendix C – Comparative Approaches to Procurement from Indigenous Businesses

1. International Approaches

1.1 United States

The United States Small Business Administration operates the 8(a) Business Development Program which helps small, disadvantaged businesses compete in the marketplace. The 8(a) program had a budget of $59.8 million in 2012. The 2014 Budget committed to providing $4 million for Small Business Administration to hire 32 new Procurement Center Representatives. Small Business Administration district offices monitor and measure the performance and progress through annual reviews, business planning and evaluations.

The 8(a) program elements include sole-source opportunities, competitive set-aside contract opportunities, joint-venturing, a mentor-protégé program and specialized business training. Participants in the 8(a) program are eligible for counseling, marketing assistance, and high-level executive development provided by the Small Business Administration and partners. Other benefits include access to surplus government property and supplies, Small Business Administration-guaranteed loans, and bonding assistance for being involved in the program.

Program eligibility for small disadvantaged businesses requires that enterprises be controlled and managed on a full-time basis by individual(s) meeting the Small Business Administration requirement for both social and economic disadvantage. Those requirements include disadvantaged Indian tribes. Businesses must be 51 percent owned by the eligible group. Businesses in the 8(a) program must be certified and classified as a small business according to the North American Industry Classification System standards.

In order to limit the reliance on the program and encourage firms to compete in the open market, the 8(a) program limits the amount of time that businesses can participate in the program.Footnote 54 Participation is limited to a total of nine years that are divided into a four year development stage and a five year transition stage. Once a business or individual participates in the 8(a) program once, neither the firm nor the individual are eligible to participate again.

To ensure businesses achieve success and conduct business in both the commercial and government sectors, each participant must develop a comprehensive business plan that set its business targets, objectives, and goals.Footnote 55 Meeting targets in the businesses plan is required for participation in sole-source contracts through the program. Footnote 56 Participants are limited to receive up to a maximum of $100 million in sole-source contracts.

1.2 Australia

The Indigenous Opportunities Policy is a commitment under the National Partnership Agreement on Indigenous Economic Participation to strengthen government procurement policies to maximize Indigenous employment and business opportunities. The program is managed by the Department of the Prime Minister and Cabinet. For contracts over $5 million in regions with significant Indigenous populations, tenderers must submit a plan that outlines the training and employment of Indigenous people and the use of Indigenous suppliers. Plan submission and approval is managed through an online portal.

The Western Australia Government's Department of Training and Workforce Development has set up a number of development centres throughout the region that provide support to Aboriginal individuals and businesses. The centres offer a number of services, including:

  • Aboriginal training or employment opportunities
  • Mentoring
  • Role modeling
  • Aboriginal Business Directory
  • Assistance for service providers assisting Aboriginal people with training and employment
  • Registry of Aboriginal individual based on skills and qualifications for employer matching
  • Advice to employers who want to diversify their workforce with more Aboriginal people
  • Aboriginal workforce development data

2. Provincial Approaches

2.1 Manitoba

The Manitoba Government's Procurement Services Branch operates an Aboriginal Procurement Initiative that works to increase participation of Aboriginal peoples and suppliers in providing goods and services to the Manitoba Government. The Aboriginal procurement criteria is considered when products or services are culturally specific to Aboriginal, or primarily designated for Aboriginal people. Procurement options included under the Aboriginal Procurement Initiative include Aboriginal Business Set-Aside, Mandatory Aboriginal Business Participation, Desirable Aboriginal Business Participation and Aboriginal Business Standard.

The Aboriginal Procurement Initiative program has the same criteria for participation as the PSAB. Certified businesses must be at least 51 percent Aboriginal owned and controlled by one or more Aboriginal Persons of Canada and the business has six or more full-time employees, at least one-third of those (full-time) employees must be Aboriginal Persons. Joint ventures that are controlled and owned by an Aboriginal business are also eligible.

Aboriginal Procurement Initiative also provides opportunities for Aboriginal businesses outside the province to work with the Manitoba Government with some exceptions related to the size of contract.

The Procurement Services Branch offers workshops on government contracts for Aboriginal businesses upon request, arranges meetings between suppliers and buyers, and maintains an Aboriginal Business Directory. Aboriginal Procurement Initiative is complemented by the Aboriginal Chamber of Commerce's procurement portal, a quote system for private sector organizations and all levels of government.

2.2 Ontario

Between March 2012 and March 2014, the Ontario Government ran the Aboriginal Business Procurement Pilot Program that helps connect Aboriginal businesses with opportunities to become of the governments suppliers of goods or services. The program includes an Aboriginal Business Set-Aside and Aboriginal Business Participation component. The criteria for eligibility is similar to PSAB and the Manitoba Government Aboriginal Procurement Initiative. Unlike those programs, when evaluating a supplier, departments may consider a non-Aboriginal supplier's track record of partnering with Aboriginal businesses, employing and/or training Aboriginal people.

2.3 Other Provinces and Territories

While other provinces do not necessarily have Aboriginal procurement strategies, there are a number of policies to support Aboriginal businesses. The British Colombia's Ministry of Finance has established the Aboriginal Procurement and Contract Management Guidelines, which support procurement from Aboriginal businesses when services are delivered to Aboriginal people in Aboriginal communities. It allows a direct award of a Shared Cost Arrangement, thus bypassing competition. The Nova Scotia Government has initiated Aboriginal procurement set-asides for some initiatives such as the Sydney Tar Ponds and Coke Oven cleanup.

The territories do not have Aboriginal procurement strategies, but rather policies that encourage buying local and from Aboriginal businesses. The Nunavut Government's Nunavummi Nangminiqaqtunik Ikajuuti (NNI Policy) encourages local procurement and a level of Inuit participation in the provisions of goods and services to the Government of Nunavut that reflects the Inuit proportion of the Nunavut population. The Northwest Territories Government has a Business Incentive Policy that provides an incentive toward Northwest Territories-based businesses when purchasing goods, services or construction. The Yukon Government has a Contracting and Procurement Regulation Directive that supports First Nations businesses through obligations arising out of the First Nations Final Agreement.

 

 

Appendix D – Survey Question Grid

Evaluation Questions and Issues Performance Indicator Survey Question Aboriginal Businesses Material Managers PSAB Coordinators
Demographics / Survey Screening
Are you a: Person representing an Aboriginal business; Material manager; PSAB coordinator Selected Selected Selected
How many employees does your business have?
1-5, 6-20, 21-50, 51-100, 100+
Selected    
What is your main area of business?
  1. Manufacturing
  2. Retail
  3. Consulting
  4. Construction/Repairs
  5. IT services
  6. Non-profit
  7. Shipping
  8. Food services
  9. Natural resources
  10. Legal services
  11. Health care
  12. Home care
  13. Child care
  14. Other
Selected    
How long have you been an entrepreneur / business owner/manager?
Drop down years
Selected    
How long have you been in material management or other forms of procurement handling?
Drop down years
  Selected  
How long have you been a PSAB coordinator?     Selected
How successful do you consider your business to be in terms of:
Profitability? Growth? Work that is personally rewarding? A steady client base?
[Likert – 5 pt – Not at all successful – Very successful
Selected    
In which region to you operate primarily?
Drop down regions
Selected Selected Selected
Relevance (continued need)
1. Does PSAB continue to address a demonstrable need and is it responsive to the needs of Aboriginal Businesses? PI 1.1 Evidence of challenges faced by Aboriginal businesses PI 1.1(a) What are the biggest barriers to the success of your business? [open text] Selected    
PI 1.1(b) Among the following challenges facing any business, please specify the degree to which you believe Aboriginal businesses face these barriers. For example, are they bigger barriers?  Smaller barriers? Or are they faced equally by Aboriginal and non-Aboriginal business alike?


[Likert – 5 pt – Less of a barrier for Aboriginal business; Equally a barrier for Aboriginal and non-Aboriginal business; More of a barrier for Aboriginal business]
Marketing/Advertising
Budgeting/Financing
Competition for business
Recruitment and retention
Knowing how to find clients
Making your business grow
Understanding your target markets
Getting networked into the business community
Others (please specify)
Selected Selected Selected
PI 1.1 (c) Could you / your business benefit from tools, training and resources designed to improve your:
[PROGRAM AS GRID. RANDOMIZE.]
  1. Financial literacy
  2. Business literacy
  3. Business capacity
  4. Corporate governance
Selected    
PI 1.1(d) To what extent do you feel you / your business is well-networked into the business community (i.e., you have necessary contacts and networks to strengthen your business profile)? [Likert 1-5] Selected    
PI 1.2 Evidence that the PSAB design will contribute to business success. PI 1.2(a) Have you ever delivered products or services to the Government of Canada or any federal department? ((Y/N))
(If Y) Did you deliver these products or services via the Government of Canada's set-aside for Aboriginal businesses? ((Y/N)/DK)
(If N) Do you deliver products and/or services that could be useful to the Government of Canada?  ((Y/N)/DK)
(If Y) Please describe the useful products and/or services that you deliver [open text]
Selected    
PI 1.2 (b) Under the Government of Canada's "Procurement Strategy for Aboriginal Business", contracts that serve a primarily Aboriginal population are set aside for competition among qualified Aboriginal businesses.  Using a 1 to 5 scale where 1 is low potential and 5 is high potential, to what extent do you believe this policy could have the potential to help your business succeed? (Likert 1-5) Selected    
PI 1.2 (c) To be considered an Aboriginal Business for the purpose of the Procurement Strategy for Aboriginal Business, your business must:
  1. Be at least 51 percent owned and controlled by Aboriginal people; and
  2. If you have six or more full-time staff, at least one third of the employees must be Aboriginal. 
Would your business be considered an Aboriginal business under these parameters?
Selected    
Do you believe the PSAB has the potential to meaningfully contribute to the participation of First Nations, Métis, non-status Indians and Inuit individuals and communities in the economy?   Selected Selected
Relevance (alignment of outcomes with federal priorities)
2. Are the objectives of the program consistent with government-wide priorities and AANDC's strategic outcomes? PI 2.1 Alignment of program with departmental priorities N/A      
PI 2.2 Alignment with Government of Canada priorities
Relevance (alignment of roles and responsibilities)
3. To what extent are the objectives of the program aligned with the role and the responsibilities of the federal government? PI 3.1 Alignment of program with federal government obligations and commitments  PI 3.1(a) In your opinion, what should the Government of Canada be doing (if anything) to help better position Aboriginal businesses domestically and internationally? [open text] Selected Selected Selected
4. Is there duplication or overlap with other programs, policies or initiatives? PI 4.1 Evidence of similar initiatives PI 4.1(a) Can you name or describe any programs other than PSAB that are designed to increase procurement opportunities for Aboriginal businesses? [open text] Selected Selected Selected
Performance – Effectiveness/Success (achievement of expected outcomes)
5. To what extent are the PSAB's expected outcomes (immediate, intermediate and long term) being achieved? PI 5.1 . . . Number of contracts and subcontracts awarded to Aboriginal businesses (PMS PI 5.1(a) Do you believe the PSAB is positively impacting: The number of contracts with Aboriginal firms?  The number of Aboriginal firms competing for, and winning, contracts? 
Likert 1-5
  Selected Selected
PI 5.1(b) Have you ever bid on a contract for products or services to any federal government department?
(If Y) Which department(s)? [drop down list] Select all that apply.
Can you estimate roughly how many contracts for products or services to the federal government you bid on? [pull down for numbers]
Can you estimate how many of these bids were successful (you were awarded the contract)? [pull down for numbers]
For the bids that were not successful, can you briefly describe some of the reasons you were given? [open text]
Selected    
PI 5.2 . . . Increase in the value of contracts (PMS) PI 5.2(a) Do you believe the PSAB is positively impacting: The value of federal contracts (set-aside and regular process) with Aboriginal businesses?   Selected Selected
PI 5.3 . . . Number of training opportunities for Aboriginal business owners  (PMS) PI 5.3(a) Have you heard of any training to help you / your business search for and bid on…?
[PROGRAM AS GRID]
  1. Federal government contracts ((Y/N))
  2. Other contracts ((Y/N))
(If Y) Have you attended any sessions related to procurement? ((Y/N))
(If Y) Did the training and/or information sessions assist you in obtaining contracts with: 1) federal government; 2) provincial and/or territorial governments; 3) the private sector? ((Y/N) for each)
Selected Selected Selected
PI 5.4 . . . Number of Aboriginal firms who are pre-qualified (PMS) N/A      
PI 5.5 Not identified PI 5.5(a) Do you believe the PSAB is meaningfully contributing to the full participation of First Nations, Métis, Non-Status and Inuit individuals and communities in the economy?  (Likert 1 – not at all; 5 – significantly)   Selected Selected
PI 5.5(b) Do you have any further comments or questions? [open text] Selected Selected Selected
PI 5.5.(c) Do you believe the PSAB is positively impacting: Investment in Aboriginal firms and business start-ups? The capacity of firms? Partnerships between Aboriginal and non-Aboriginal business?   Selected Selected
6. To what extent has PSAB created a leverage effect within the FFAED? PI 6.1 Evidence of partnerships with other programs of the FFAED contributing to the outcome indicators above PI 6.1(a) Have you / your business benefited in any way from partnerships between provincial/territorial, municipal, or federal governments, and other businesses or Aboriginal business associations? ((Y/N)/DK) Please elaborate [open text] Selected    
PI 6.1(b) Is the PSAB successful in building strategic partnerships between the Government of Canada and Aboriginal people, the private sector, and provinces and territories [Likert 1 – not at all necessary; 5 – completely necessary]     Selected
PI 6.1(c) Is the PSAB meaningfully contributing to: Aboriginal entrepreneurship?
Partnerships to maximise economic development opportunities?
Developing Aboriginal human capital?
Focusing the role of the Federal Government in contributing to Aboriginal economic success?
(Likert for each – 1 – Not at all; 5 – Very much so)
    Selected
7. Have appropriate mechanisms to deliver PSAB been put in place? PI 7.1 . . . Number of training sessions / outreach with Aboriginal businesses (PMS) PI 7.1(a) Before undertaking this survey, did you hear of the Government of Canada's "Procurement Strategy for Aboriginal Business?" ((Y/N))
(If Y) Do you know what it is? ((Y/N))
(If N) Are you aware of any federal government policies or guidelines respecting set-asides for Aboriginal firms?
Selected Selected  
PI 7.1(b) Do you know how to go about approaching federal departments for a potential contract? ((Y/N)) Selected    
PI 7.2 . . . Number of Aboriginal set-asides targets set by other government departments PI 7.2(a) Is your business included on an Aboriginal business set-aside for products or services to the Government of Canada? ((Y/N)/DK) Selected    
PI 7.2 (b) Does your department have targets for set-asides?  ((Y/N)/DK)
(If Y) Are they being met? ((Y/N)/DK)
  Selected Selected
PI 7.2 (c) What mechanisms encourage your department to meet its performance objectives with respect to contracting with Aboriginal suppliers?

Departmental culture? Contract stipulations/PWGSC tools/registration process? Supplier awareness? Aboriginal standing offers? Advice/training by procurement officers?  Others? [specify]
  Selected Selected
PI 7.3 . . . Evidence of consistency between PSAB programming and Treasury Board policy, directives and procedures N/A      
8. To what extent have external and internal factors influenced the achievement of PSAB's expected outcomes? PI 8.1 Evidence of decisions and recommendations that guide program direction For businesses having taken part in PSAB:
PI 8.1(a) What were the main factors in your choice to pursue contracts with the federal government?
For businesses not having taken part in PSAB:
PI 8.1(b) Why have you not participated in contracts with the federal government? [select all that apply]
Did not know about the opportunity; My business does not produce the kinds of goods or services of use to the federal government; My business does not yet have the capacity to bid on such contracts; I'm not interested in working with the federal government; Other [specify]
     
9. Are there any unexpected impacts of the PSAB, either positive or negative? PI 9.1 Evidence of gaps, unanticipated challenges or successes N/A      
Performance (Efficiency and Economy)
10. How has the PSAB optimized its processes and quality of services to achieve expected outcomes? (efficiency) PI 10.1 Evidence of ideal processes and quality of service PI 10.1(a) Are there issues with procurement policies that cause unnecessary delays or inefficiencies? If so, please describe. [open text]   Selected Selected
11. Are there other efficient ways of delivering the initiative to achieve similar results? (efficiency) PI 11.1a Evidence of comparable programs or alternatives. N/A      
PI 11.1b Analysis of potential inefficiencies or cost saving measures
12. Are there opportunities to achieve the intended results of PSAB using fewer resources? (economy) PI 12.1 Extent to which outcomes are being achieved in an efficient manner N/A      
Performance (Efficiency and Economy)
13.What are the lessons learned and best practices that emerged from the implementation of PSAB and other similar programs from other jurisdictions, countries, or in the private sector, which may enhance PSAB effectiveness? PI 13.1 Comparisons with other departments, organizations and jurisdictions with similar economic development programs N/A      
PI 13.2 [New]  Lessons learned and best practices identified by Aboriginal businesses
 
 

 

Appendix E: Interview Matrix

DRAFT INTERVIEW MATRIX
Evaluation Issues and Questions Performance Indicators Interview Questions Aboriginal Business Using PSAB Aboriginal Business Not Using PSAB Non-Aboriginal Business P/T Representatives Other Organizations Federal Representatives
Relevance (Continued Need)
1. Does PSAB continue to address a demonstrable need and is it responsive to the needs of Aboriginal Businesses? PI 1.1 Evidence of challenges faced by Aboriginal businesses KIQ 1: What are the unique challenges or barriers faced by Aboriginal Businesses wishing to obtain contract work from the federal government that are not significant challenges to non-Aboriginal Businesses? Selected Selected Selected   Selected Selected
KIQ 2: Do Aboriginal businesses wishing to obtain provincial or territorial contracts face challenges or barriers that are less significant to non-Aboriginal businesses? Selected Selected Selected Selected Selected Selected
KIQ 3: Does operating a business on Aboriginal lands create challenges or barriers to obtaining F/P/T contracts? Selected Selected   Selected Selected Selected
KIQ 4: What have you done to overcome or reduce these challenges, and were you successful? Selected     Selected Selected Selected
PI 1.2 Evidence that the PSAB design will contribute to business success. KIQ 5: Do you believe the PSAB has the potential to bolster the success of Aboriginal businesses? Selected Selected Selected Selected Selected Selected
 Relevance (Alignment)
2. Are the objectives of the program consistent with government-wide priorities and AANDC's strategic outcomes? PI 2.1 Alignment of program with departmental priorities KIQ 6: According to the Program Alignment Architecture (PAA), the strategic outcome of PSAB is to contribute to the "full participation of First Nations, Métis, non-status Indians and Inuit individuals and communities in the economy." Do you feel the activities of PSAB make sense in light of this objective? 
The ultimate goal of PSAB specifically is that Aboriginal businesses win procurement contracts. Do you feel this goal makes sense given the strategic objective?
          Selected
PI 2.2 Alignment with Government of Canada priorities KIQ 7: Do you believe these objectives are aligned with current government priorities?  How so/Why not?           Selected
Relevance (Consistency with Federal Roles and Responsibilities)
3. To what extent are the objectives of the program aligned with the role and the responsibilities of the federal government? PI 3.1 Alignment of program with federal government obligations and commitments KIQ 8: Do you feel it is an appropriate role of the Government of Canada to set parameters regarding departmental procurement respecting Aboriginal businesses? Why/why not?

KIQ 9: Do you feel it is an appropriate role of the Government of Canada to promote the participation of Aboriginal people in the workforce?  Why/why not?
      Selected Selected Selected
4. Is there duplication or overlap with other programs, policies or initiatives? PI 4.1 Evidence of similar initiatives KIQ 10: Can you name any programs other than PSAB that are designed to increase procurement opportunities for Aboriginal businesses? 

KIQ 11: Do you know of any other initiatives that are designed to promote the recruitment of Aboriginal talent?  Can you describe them?
Selected Selected Selected Selected Selected Selected
Performance – Effectiveness/Success (achievement of expected outcomes)
5. To what extent are the PSAB's expected outcomes (immediate, intermediate and long term) being achieved? PI 5.1 . . . Number of contracts and subcontracts awarded to Aboriginal businesses (PMS N/A (To be addressed by AES in the Lit/Doc review.)            
PI 5.2 . . . Increase in the value of contracts (PMS) N/A (To be addressed by AES in the Lit/Doc review.)            
PI 5.3 . . . Number of training opportunities for Aboriginal business owners  (PMS) KIQ 12: Have you attended any sessions related to procurement?  Recruitment of Aboriginal talent?  (If yes, were the sessions provided by the: 1) federal government; 2) provincial/territorial governments; 3) Aboriginal organizations; 4) the private sector; and/or 5) other sources; 6) don't know? Selected Selected Selected Selected Selected  
KIQ 13: If yes to procurement; did the training and/or information sessions assist you in obtaining contracts with: 1) federal government; 2) provincial and/or territorial governments; 3) the private sector? If yes, please describe how your business benefited.

If yes to recruiting Aboriginal talent, did the session give you any insights?  Did it help you recruit Aboriginal talent?
Selected Selected Selected   Selected  
KIQ 11: Are there other types of information or training sessions related to PSAB and/or obtaining contracts with the federal government and/or recruitment of Aboriginal talent that would benefit your business? If yes, please describe. Selected Selected Selected Selected Selected  
PI 5.4 . . . Number of Aboriginal firms who are pre-qualified (PMS) N/A (To be addressed by AES in the Lit/Doc review.)            
PI not identified KIQ 12: Generally speaking, do you feel PSAB has resulted in sufficient procurement opportunities for Aboriginal businesses to date (given the amount of resources attached to the program)?

KIQ 13: Do you believe the PSAB is meaningfully contributing to the full participation of First Nations, Métis, Non-Status and Inuit individuals and communities in the economy?  How so/why not?
          Selected
KIQ 14: What would happen in the absence of federal government parameters around the procurement of Aboriginal firms in Federal departments?

KIQ 15: What would happen in the absence of federal promotion activities of Aboriginal participation in the work force?

KIQ 16: Is the PSAB meaningfully contributing to: Aboriginal entrepreneurship?
Partnerships to maximise economic development opportunities?
Developing Aboriginal human capital?
Focusing the role of the Federal Government in contributing to Aboriginal economic success?
      Selected Selected  
6. To what extent has PSAB created a leverage effect within the FFAED? PI 6.1 Evidence of partnerships with other programs of the FFAED contributing to the outcome indicators above KIQ 17: Can you describe the relationship between PSAB and other programs designed to contribute to the FFAED

KIQ 18: Is the PSAB successful in building strategic partnerships between provincial/territorial, municipal or federal governments, and other businesses or Aboriginal associations?

KIQ 19: Is there complementarity/overlap? 
          Selected
7. Have appropriate mechanisms to deliver PSAB been put in place? PI 7.1 . . Number of training sessions / outreach with Aboriginal businesses (PMS) KIQ 20: What is the nature and reach of the training sessions / outreach to Aboriginal businesses?  What improvements could be made?           Selected
PI 7.2… Number of Aboriginal set-asides targets set by other government departments KIQ 21: Does your department have targets for set-asides?  Are they being met?

KIQ 22: Does your department have:
Systems to monitor Aboriginal procurement?
Training for contracting officers?
External orientation sessions for Aboriginal businesses?
Meetings with Aboriginal leaders (could include events and tradeshows)
      Selected   Selected
PI 7.3 . . Evidence of consistency between PSAB programming and Treasury Board policy, directives and procedures KIQ 23: Do PSAB directives and related program align with Treasury Board policies on procurement?  Are there any issues with the relationships and communication between TBS, AANDC and any of the other departments or agencies with respect to PSAB?           Selected
8. To what extent have external and internal factors influenced the achievement of PSAB's expected outcomes? PI 8.1 Evidence of decisions and recommendations that guide program direction (note that interview questions for issue 8 do not speak to this PI. This PI will be addressed via document review) KIQ 24: (For businesses having benefited from PSAB initiatives): What aspects of the government's procurement activities made you decide to approach the government for contracts?

KIQ 25: (For businesses not having benefited from PSAB initiatives): Why have you not taken advantage of the government's procurement opportunities?
Selected Selected        
KIQ 26: What are the main factors driving the success of businesses being awarded contracts with: the federal government? Provincial governments? Other firms?

KIQ 27: What are the main impediments?
Selected Selected   Selected Selected Selected
KIQ 28: What are the main factors in the success of Aboriginal people's participation in the labour market? What are the main impediments Selected Selected Selected Selected Selected Selected
KIQ 29: Do you believe PSAB eligibility criteria as articulated in guidelines are sufficiently clear?  Do believe they are well-understood by material managers? By Aboriginal businesses attempting to bid on contracts? Selected Selected Selected Selected Selected Selected
9. Are there any unexpected impacts of the PSAB, either positive or negative? PI 9.1 Evidence of gaps, unanticipated challenges or successes KIQ 30: Are there gaps or missing services or other support missing in PSAB that are making it difficult to take advantage of those opportunities that are available through PSAB? Selected Selected   Selected Selected Selected
KIQ 31: If yes to the previous question, please describe what additional services or support would be beneficial for your business? Selected Selected   Selected Selected Selected
Note: The "challenges or successes" aspect of this PI have been addressed by EQ 1. Selected Selected   Selected Selected  
Performance (Efficiency and Economy)
10. How has the PSAB optimized its processes and quality of services to achieve expected outcomes? (efficiency) PI 10.1 Evidence of ideal processes and quality of service KIQ 32: Are there issues with procurement policies that cause unnecessary delays or inefficiencies? If so, what would you suggest as alternatives? (Covers Evaluation Issue 11) What are the impacts of these issues? Selected       Selected Selected
11. Are there other efficient ways of delivering the initiative to achieve similar results? (efficiency) PI 11.1a Evidence of comparable programs or alternatives. Covered under relevance issue 4.           Selected
PI 11.1b Analysis of potential inefficiencies or cost saving measures  Covered under issue 10           Selected
12. Are there opportunities to achieve the intended results of PSAB using fewer resources? (economy) PI 12.1 Extent to which outcomes are being achieved in an efficient manner KIQ 33: Can you identify potential inefficiencies where costs can be reduced without impacting the result of the program?           Selected
Demonstration of Efficiency and Economy
13. What are the lessons learned and best practices that emerged from the implementation of PSAB and other similar programs from other jurisdictions, countries, or in the private sector, which may enhance PSAB effectiveness? PI 13.1 Comparisons with other departments, organizations and jurisdictions with similar economic development programs KIQ 34: In your dealings with the private sector, have you seen or experienced any programs focused on developing or partnering with Aboriginal businesses, other than PSAB or land claims agreements? Selected Selected     Selected  
PI 13.2 [New]  Lessons learned and best practices identified by Aboriginal businesses KIQ 35: From a lessons learned perspective, what are the two or three PSAB operational practices and processes that work best?

KIQ 36: Are there other issues impacting quality of service?
What are some key strengths in the process that maximise outcomes?
Selected Selected Selected Selected Selected  
 
 

 

Appendix F: Evidence Matrix

Issues / Questions Indicators Literature Review Key Informant Interviews Document Review / File Review Data Analysis Survey (tbc) Site Visits
Internal (e.g., Headquarters and Regions); Other Federal Government Major StakeholdersFootnote 57 Aboriginal Businesses
Relevance (Continued Need)
1. Does PSAB continue to address a demonstrable need and is it responsive to the needs of Aboriginal Businesses? Evidence of challenges faced by Aboriginal businesses

Evidence that the PSAB design will contribute to business success
Selected Selected Selected Selected Selected Selected Selected Selected
 Relevance (Alignment)
2. Are the objectives of the program consistent with government-wide priorities and AANDC's strategic outcomes? Alignment of program with Government of Canada priorities

Alignment of program with departmental priorities
Selected Selected     Selected   Selected Selected
Relevance (Consistency with Federal Roles and Responsibilities)
3. To what extent are the objectives of the program aligned with the role and the responsibilities of the federal government? Alignment of program with federal government obligations and commitments Selected Selected Selected Selected Selected Selected Selected Selected
4. Is there duplication or overlap with other programs, policies or initiatives? Evidence of similar initiatives Selected Selected Selected Selected Selected   Selected Selected
Performance (Effectiveness / Success)
5. To what extent are the PSAB's expected outcomes (immediate, intermediate and long term) being achieved? Evidence of the ability of Aboriginal businesses to indentify business opportunities (private and federal) and participate in the procurement process as indicated by:
Number of contracts and subcontracts awarded to Aboriginal businesses (PMS)
Increase in the value of contracts (PMS)
Number of training opportunities for Aboriginal business owners  (PMS)
Number of Aboriginal firms who are pre-qualified (PMS)
Selected Selected Selected Selected Selected Selected Selected Selected
6. To what extent PSAB created a leverage effect within the FFAED? Evidence of partnerships with other programs of the FFAED contributing to the outcome indicators above   Selected Selected Selected Selected Selected Selected Selected
      Selected Selected   Selected   Selected Selected
7. Have the appropriate mechanisms to deliver PSAB been put in place? Evidence of awareness among businesses (Aboriginal and non-Aboriginal) and material managers in the Federal government as indicated by:
Number of training sessions / outreach with Aboriginal businesses (PMS)

Number of Aboriginal set-asides targets set by other government departments
  Selected Selected Selected Selected Selected Selected Selected
  Evidence of consistency between PSAB programming and Treasury Board policy, directives and procedures   Selected Selected   Selected     Selected
8. To what extent have external and internal factors influenced the achievement of PSAB's expected outcomes? Evidence of decisions and recommendations that guide program direction   Selected Selected   Selected   Selected Selected
9. Are there any unexpected impacts of the PSAB, either positive or negative? Evidence of gaps, unanticipated challenges or successes   Selected Selected Selected Selected   Selected Selected
Performance (Efficiency and Economy)
10. How has the PSAB optimized its processes and quality of services to achieve expected outcomes? (efficiency) Evidence of ideal processes and quality of service   Selected Selected Selected Selected   Selected Selected
11. Are there other efficient ways of delivering the initiative to achieve similar results? (efficiency) Evidence of comparable programs or alternatives
Analysis of potential inefficiencies or cost saving measures
Selected Selected Selected Selected Selected Selected    
12. Are there opportunities to achieve the intended results of PSAB using fewer resources? (economy) Extent to which outcomes are being achieved in an efficient manner   Selected Selected Selected Selected   Selected  
[Other Evaluation Issues]
13. What are the lessons learned and best practices that emerged from the implementation of PSAB and other similar programs from other jurisdictions, countries, or in the private sector, which may enhance PSAB effectiveness? Comparisons with other departments, organizations and jurisdictions with similar economic development programs Selected Selected Selected Selected Selected     Selected
 
 
 
Date modified: