Audit of Post-Secondary Education Programs

February, 2015
Project #: 14-13

PDF Version (174 Kb, 35 Pages)

 

Table of contents

Acronyms

AANDC

Aboriginal Affairs and Northern Development Canada

DCI

Data Collection Instrument

DPR

Departmental Performance Report

EIS

Education Information System

ESDC

Employment and Social Development Canada

ESDPP

Education and Social Development Programs and Partnerships

HQ

Headquarters

NPG

National Program Guidelines

NSC

National Selection Committee

OAG

Office of the Auditor General

PMS

Performance Management Strategy

PSE

Post-Secondary Education

PSPP

Post-Secondary Partnerships Program

PSSSP

Post-Secondary Student Support Program

RDG

Regional Director General

RPP

Report on Plans and Priorities

T&C

Terms and Conditions

UCEP University and College Entrance Preparation
 

 

Executive Summary

Background

An Audit of Post-Secondary Education Programs was included in Aboriginal Affairs and Northern Development Canada's ("AANDC's" or "the Department's") 2014-2015 to 2016-2017 Risk-Based Audit Plan, approved by the Deputy Minister on February 6, 2014. The audit was identified as a priority because Post-Secondary Education (PSE) Programs are very important to both First Nations and the Department in terms of their support to capacity development and self-sufficiency. The Programs are also significant from a materiality perspective and audits performed in 2009 and 2011 identified systemic concerns related to unexpended funds and funds not being used for intended purposes.

The primary objective of AANDC's PSE Programs is to help increase access to and enable success in PSE for eligible First Nations and Inuit students. In 2013/14, AANDC provided approximately $342 million in Post-Secondary funding to recipients. This includes $5.8 million allocated to Indspire, a national non-profit organization that offers scholarships and other supports towards the education of Indigenous youth. The remaining $336.2 million is delivered through two streams of programming which are managed and coordinated at the national level by the Education Branch in the Education and Social Development Programs and Partnerships (ESDPP) Sector, and delivered at the regional level. A description of each programming stream is as follows:

The Post-Secondary Student Support Program (PSSSP) and the University and College Entrance Preparation Program (UCEP)

AANDC provides support to eligible First Nations and Inuit students for PSE through PSSSP and UCEP Program. The objectives of these programs are listed below.

  • The objective of the PSSSP is to improve the employability of First Nations and eligible Inuit students by providing them with funding to access education and skills development opportunities at the post-secondary level.
  • The objective of the UCEP Program is to provide financial support to First Nations and eligible Inuit students who are enrolled in university and college entrance preparation programs offered in Canadian post-secondary institutions, and to enable them to attain the academic level required for entrance into degree and diploma credit programs.

PSSSP and UCEP Program are the primary components of the Department's PSE programming. Spending in 2013/2014 for these programs was approximately $314 million or approximately 93% of total available PSE funding.

Funding flows from AANDC Headquarters (HQ) to the regional offices, who then allocate funding to First Nations' organizations, such as band councils, tribal councils, or representative organizations. Eligible Inuit students are funded through designated aboriginal organizations or through some AANDC regional offices. These funding recipients then administer the funds to eligible students within their respective communities. As such, the Department has limited ability to influence how funding is ultimately allocated to students, which provides a challenge to the Department's ability to directly influence Program outcomes.

The Post-Secondary Partnerships Program (PSPP)

AANDC provides support directly to post-secondary institutions through the PSPP. The objective of the PSPP is to assist Canadian post-secondary institutions in the design and delivery of college and university level courses that respond to the educational needs of First Nations and Inuit students. It is expected that the program will increase the availability of post-secondary courses and programs targeted to First Nations and Inuit students in fields of high labour market demand, either locally, regionally or nationally, and in the fields of Aboriginal governance and business development.

Planned spending in 2014/2015 for the PSPP was approximately $22 million, with $7 million allocated to support the operations of First Nations University of Canada, which is headquartered in Regina, Saskatchewan. The remaining $15 million is disbursed to fund approved projects from eligible post-secondary institutions after a proposal intake and review process. In 2014/2015, a total of 87 projects from 35 post-secondary institutions were approved.

Audit Objectives and Scope

The overall objective of this audit was to assess the adequacy, effectiveness, and efficiency of the management control framework of the Department's Post-Secondary Education Programs in relation to recent reform initiatives and controls in place to ensure compliance with applicable authorities and policy frameworks.

In addressing the audit objective, the scope included an assessment of three main areas, as follows:

  • Program Implementation – including ensuring that programs are being delivered according to program authorities and terms and conditions, and in a way that supports overarching objectives. This included compliance with national guidelines, funding allocations, funding approaches, monitoring and reporting, and program support to assist funding recipients with program delivery. 
  • Recent Reform Initiatives – including recent changes to the PSPP, specifically centralizing review and selection of funding proposals to a National Selection Committee. It also included recent changes to the PSSSP/UCEP data collection processes and the use of the Education Information System (EIS) in capturing the data. 
  • Follow-up on Previous Audit and Evaluation Recommendations – including recommendations made in the 2009 Audit of the PSE Programs, the 2012 Summative Evaluation of the PSE Programs as it relates to the objectives and scope of this audit, and the 2011 Office of the Auditor General (OAG) Audit on Programs for First Nations on Reserve as it relates to the PSE Programs. The audit focused on those observations and recommendations that continue to be relevant in 2014/15. 

The testing of transactions covered the period April 1, 2013 to March 31, 2014, and changes or reform initiatives considered the period up to the completion of the audit's conduct phase (October to December 2014). Site visits were conducted in the Yukon, British Columbia, Saskatchewan, and Ontario regions.

Statement of Conformance

This audit conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

Conclusion

The audit found that, while program management has made improvements in terms of updating its performance measurement strategy and strengthening its National Program Guidelines for PSSSP/UCEP since the audit in 2009, improvements are still required in the areas of recipient monitoring, program reporting, the approach to  funding allocation, risk management and resource planning.

In addition, there is a lack of clarity regarding the objectives and outcomes of the PSPP, and the extent to which these align with broader Departmental priorities. Also, while recent changes to PSPP have introduced greater rigour and transparency to the proposal review process, the level of effort and resources required does not appear to be commensurate with the Program's materiality and risk.

Recommendations

The audit team identified areas where management control practices and processes could be improved, resulting in the following five (5) recommendations:

  1. The Assistant Deputy Minister of ESDPP should review the appropriateness of how PSSSP/UCEP funding is allocated across regions to ensure it supports equitable access and, to the extent possible, promotes the alignment of funding practices with funding needs. Based on the outcomes of this review and overall Program objectives, the ADM, ESDPP should develop national guidance to support the consistency of funding allocation methodologies employed by regions while enabling regional differences when warranted.
  2. The Assistant Deputy Minister of ESDPP should develop a clear and formal strategy and plan as to how and when EIS reporting capabilities will be leveraged in support of the effective management and control of PSE programs (i.e. what reports, how often, to whom, etc.) as well as how training will support improved accuracy and consistency of data collection.
  3. The Assistant Deputy Minister of ESDPP should establish an implementation plan and timeline for implementing ESDPP's new compliance regime within PSE. In the interim, develop a risk-based approach to monitoring whereby a minimum standard for compliance monitoring and follow-up across all regions is established and sustained.
  4. The Assistant Deputy Minister of ESDPP should clarify and confirm PSPP program objectives in terms of their priority and alignment with broader Departmental priorities. Following this clarification and confirmation, the Program should:
    • Develop relevant performance indicators and align these with the Program DCI; and
    • Identify and implement efficiencies in proposal review process to ensure that the resources and effort spent is commensurate with Program's significance, materiality and risk.
  5. The Assistant Deputy Minister of ESDPP should strengthen key elements of control to support overall governance of the PSE Programs.  In particular, it is recommended that:
    • A Program-specific risk assessment should be completed by Education Branch on PSE programs upon which to implement risk-based controls to mitigate identified risks;
    • A resourcing plan be developed to align the required level of resources with the operational needs for program delivery and monitoring activities.

Management Response

Management is in agreement with the findings, has accepted the recommendations included in the report, and has developed a management action plan to address them. The management action plan has been integrated into this report.

 

 

1. Background

An Audit of Post-Secondary Education Programs was included in Aboriginal Affairs and Northern Development Canada's ("AANDC's" or "the Department's") 2014-2015 to 2016-2017 Risk-Based Audit Plan, approved by the Deputy Minister on February 6, 2014. The audit was identified as a priority because Post-Secondary Education (PSE) Programs are very important to both First Nations and the Department in terms of their support to capacity development and self-sufficiency. The Programs are also significant from a materiality perspective and audits performed in 2009 and 2011 identified systemic concerns related to unexpended funds and funds not being used for intended purposes.  

Since the early 1960s, AANDC has sought incremental policy authorities to support the improvement of the socio-economic conditions and overall quality of life of Aboriginal People in Canada. These activities have generally included support for First Nations and Inuit access to and participation in PSE programs, and support for the inclusion of culturally appropriate First Nations and Inuit content within the PSE environment.

AANDC's involvement in PSE is based on social policy, and is not enacted through legislation. Notwithstanding, some First Nations view PSE as being part of the larger continuum of education support, and consequently view support for PSE as being a treaty right. These fundamental differences in interpretation and expectations have created challenges for the Department in terms of the effective strategic management of PSE programming.

The primary objective of AANDC's PSE Programs is to help increase access to and enable success in PSE for eligible First Nations and Inuit students. In 2013/2014, AANDC provided approximately $342 million in Post-Secondary funding to recipients. This includes $5.8 million allocated to Indspire, a national non-profit organization that offers scholarships and other supports towards the education of Indigenous youth. The remaining $336.2 million is delivered through two streams of programming:

The programs are managed and coordinated at the national level by the Education Branch in the Education and Social Development Programs and Partnerships (ESDPP) Sector, while the programs are delivered at the regional level. In recent years, there have been a number of departmental priorities across education and social programming that have required significant effort, attention, and resources.  Recently, the Education Branch has focused much of its efforts and resources on K-12 Education reform, which consists of a budget of $1.4 billion. Similarly, regional offices have focused much of their monitoring and compliance resources on large statutory programs such as Income Assistance.

Each of the Department's PSE Programs is discussed in more detail in the following sub-sections.

1.1 Post-Secondary Student Support Program / University and College Entrance Preparation Program

AANDC provides support to eligible First Nations and Inuit students for PSE through PSSSP and UCEP Program. The objectives of these programs are listed below.

  • The objective of the PSSSP is to improve the employability of First Nations and eligible Inuit students by providing them with funding to access education and skills development opportunities at the post-secondary level.
  • The objective of the UCEP Program is to provide financial support to First Nations and eligible Inuit students who are enrolled in university and college entrance preparation programs offered in Canadian post-secondary institutions, and to enable them to attain the academic level required for entrance into degree and diploma credit programs.

PSSSP and UCEP Program are the primary components of the Department's PSE programming. Spending in 2013/2014 for these programs was approximately $314 million of the total $336.2 million allocated for all PSE Programs, or approximately 93% of total available PSE funding.

These programs have a long history, with origins dating back to the 1970's. In the early 1990's, administration of PSSSP/UCEP funding was transferred to the First Nations and other Aboriginal Organizations. Funding flows from AANDC Headquarters (HQ) to the regional offices, who then allocate funding to First Nations' organizations, such as band councils, tribal councils, or representative organizations. The First Nations then administer the funds to eligible students within their respective communities. For Inuit students who are not covered for PSE funding through their respective territorial governments, students can access the funding either directly from a AANDC regional office, or from a designated regional aboriginal organizations.

The expectations of the recipients, are outlined in the funding agreements and the PSSSP/UCEP National Program Guidelines (NPG). Specifically, recipients are expected to establish publicly available Local Operating Guidelines (LOG) that align to the requirements in the NPG, and include selection priority criteria for funding decisions and an appeals process. Recipients must also submit an annual report, using the departmentally provided data collection instrument (DCI), which provides details on funded students. While program requirements are formally defined, the Department has limited ability to influence how funding is ultimately allocated to students, which provides a challenge to the Department's ability to directly influence Program outcomes.

Eligible students for the PSSSP/UCEP are individuals who have been accepted into a post-secondary institution that is recognized by a province or territory, and who either is: 1) a treaty/registered First Nation person who has been a resident in Canada for 12 consecutive months; or 2) an Inuit person who has been a resident in Canada for 12 consecutive months but who resides outside of their territory and is no longer eligible to be funded by their territory.

Eligible expenses for PSSSP/UCEP funding include costs for tuition, examination fees, books, living allowances, travel (i.e. maximum two trips per academic year), tutorial and guidance services, and scholarships or incentives. First Nations are also able to expend up to a maximum of 10% of the total PSSSP/UCEP funding received for administration.

While the majority of PSSSP/UCEP funding is provided to First Nations as contributions, the Department also provides approximately $1 million annually directly to students as grants. These students are from First Nations who have an arrangement with their respective regional office to administer the program on their behalf or are Inuit students living in Saskatchewan or Alberta, and have resided outside of Nunavut or the Northwest Territories for 12 consecutive months (therefore not eligible for PSE funding through territorial governments). In the other AANDC regions, these Inuit students receive PSE funding and support through designated aboriginal organizations, which provide the service through a contribution agreement with AANDC.

1.2 Post-Secondary Partnerships Program

AANDC provides support directly to post-secondary institutions through the PSPP. The objective of the PSPP is to assist Canadian post-secondary institutions in the design and delivery of college and university level courses that respond to the educational needs of First Nations and Inuit students. It is expected that the program will increase the availability of post-secondary courses and programs targeted to First Nations and Inuit students in fields of high labour market demand, either locally, regionally or nationally, and in the fields of Aboriginal governance and business development.

Planned spending in 2014/2015 for the PSPP was approximately $22 million, with $7 million allocated to support the operations of First Nations University of Canada, which is located in Regina, Saskatchewan. The remaining $15 million is disbursed to fund approved projects from eligible post-secondary institutions after a proposal intake and review process. In 2014/2015, a total of 87 projects from 35 post-secondary institutions were approved.

In recent years, Education Branch has implemented significant changes to the program. The PSPP replaced the Indian Studies Support Program (ISSP) as of April 1, 2014. The redesigned program places a greater emphasis on labour market needs and the employment of Aboriginal youth. Additionally, the proposal review process was refined for 2013/2014 with the introduction of a National Selection Committee and assessment criteria to introduce more transparency and rigour into the process.

PSPP funding is disbursed to recipients through one-year set contribution agreements. Eligible recipients for PSPP funding are Canadian post-secondary institutions that are recognized by a province or territory and are degree, diploma, or certificate granting. Other post-secondary educational organizations affiliated with or operating in a formal partnership with these institutions are also eligible.

PSPP provides funding for projects that include the following activities:

  • Deliver a program of study;
  • Support for existing University and College Entrance Preparation program courses;
  • Convert existing courses to online delivery and distance education;
  • Deliver an individual course; and
  • Research and development of new courses and programs.

Eligible expenses for PSPP funding include costs for salaries and benefits, instructional materials, digitizing of instructional materials, printing and publishing, and equipment rental. Recipients are also able to expend up to a maximum of 10% of the total PSPP funding received for administration.

 

 

2. Audit Objectives and Scope

2.1 Audit Objective

The overall objective of this audit was to assess the adequacy, effectiveness, and efficiency of the management control framework of the Department’s Post-Secondary Education Programs in relation to recent reform initiatives and controls in place to ensure compliance with applicable authorities and policy frameworks.

2.2 Audit Scope

In addressing the audit objective, the scope included an assessment of three main areas, as follows:

  • Program Implementation – including ensuring that programs are being delivered according to program authorities and terms and conditions, and in a way that supports overarching objectives. This included compliance with national guidelines, funding allocations, funding approaches, monitoring and reporting, and program support to assist funding recipients with program delivery. 
  • Recent Reform Initiatives – including recent changes to the PSPP, specifically centralizing review and selection of funding proposals to a National Selection Committee. It also included recent changes to the PSSSP/UCEP data collection processes and the use of the Education Information System (EIS) in capturing the data. 
  • Follow-up on Previous Audit and Evaluation Recommendations – including recommendations made in the 2009 Audit of the PSE Programs, the 2012 Summative Evaluation of the PSE Programs as it relates to the objectives and scope of this audit, and the 2011 Office of the Auditor General (OAG) Audit on Programs for First Nations on Reserve as it relates to the PSE Programs. The audit focused on those observations and recommendations that continue to be relevant in 2014/15. 

The testing of transactions covered the period April 1, 2013 to March 31, 2014, and changes or reform initiatives considered the period up to the completion of the audit’s conduct phase (October to December 2014). Audit site visits were conducted in the Yukon, British Columbia, Saskatchewan, and Ontario regional offices.

 

 

 

3. Approach and Methodology

The audit was conducted in accordance with the requirements of the Policy on Internal Audit and followed the Internal Auditing Standards for the Government of Canada. The audit examined sufficient, relevant evidence to provide a reasonable level of assurance in support of the audit conclusion.

The planning phase of the audit involved various procedures, including a review of preliminary documentation, and interviews with management. Based on this information, a risk assessment was completed to determine the scope of the audit and to develop audit criteria. The audit criteria developed for this audit are included in Appendix A.

Based on the audit criteria, a detailed audit program including an evidence gathering strategy was developed, which drove the activities undertaken during the conduct phase. The audit activities that were undertaken included the following:

As part of the approach and methodology, four regions were selected for an on-site audit visit. Factors in the selection of the regional offices visited included the delivery of both PSSSP/UCEP and PSPP, the amount of funding received for PSE Programs, the amount of PSSSP/UCEP funding provided as grants, and the number of PSPP proposals received during the 2014/2015 intake process. Based on these factors, the Yukon, British Columbia, Saskatchewan, and Ontario regions were selected for site visits.

Based on the information gathered during the audit activities, observations and conclusions were developed against the audit criteria. These are presented in this report.

 

 

4. Conclusion

The audit found that, while program management has made improvements in terms of updating its performance measurement strategy and strengthening its National Program Guidelines for PSSSP/UCEP since the audit in 2009, improvements are still required in the areas of recipient monitoring, program reporting, the approach to  funding allocation, risk management and resource planning.

In addition, there is a lack of clarity regarding the objectives and outcomes of the PSPP, and the extent to which these align with broader Departmental priorities. Also, while recent changes to PSPP have introduced greater rigour and transparency to the proposal review process, the level of effort and resources required does not appear to be commensurate with the Program’s materiality and risk.

 

 

5. Findings and Recommendations

Based on the evidence gathered through examination of documentation, interviews and analysis, each of the audit criteria, as detailed in Appendix A, was assessed during this audit. Where a difference between the audit criterion and the observed practice was found, the risk of the gap was evaluated and used to develop the conclusion and corresponding recommendations for improvement.

This section is organized around the following three thematic areas associated with the scope and objective of audit as follows:

  1. Management and Delivery of the PSSSP and UCEP;
  2. Redesign of the PSPP; and
  3. Follow-up on Previous Audit and Evaluation Recommendations.

5.1 Management and Delivery of the PSSSP and UCEP

While authority and responsibility over the management of how PSSSP/UCEP contribution funding is distributed to eligible students is retained by recipients, the Department is responsible to ensure PSSSP/UCEP is managed and delivered in accordance with all applicable requirements and expectations. To this end, the Department has created a suite of program foundations that range from development of effective program contribution agreements to performance measurement strategies, and compliance monitoring. As they are responsible for the Department's delivery of program funds to recipients, AANDC's regional offices have a considerable role to play in ensuring PSSSP/UCEP requirements are met and program objectives are supported. Similar to other Departmental programs, regional offices are afforded a degree of flexibility in the administration and delivery of programs, as long as they align with and support applicable national guidelines or requirements and otherwise support the achievement of program objectives.

5.1.1 Policies, Procedures and Tools

As with any AANDC program, there was an expectation that Program and Regional Management develop and implement a regime of processes, procedures and supporting tools to promote, and monitor the achievement of, Program objectives as well as ensure compliance with requirements. This regime should reflect the Department's expectations for the management and control of programs, as well as reflecting the context and risk profile of the PSSSP/UCEP. As a minimum, the audit expected to confirm the existence of well-designed and effective funding agreements, performance measurement frameworks, monitoring and compliance regimes, as well as sufficient guidance, tools and training to support both the regional offices and program recipients in fulfilling their respective roles. Except for monitoring capabilities and practices, which are described in Section 5.1.4 of this report, this Section provides the results of Audit's assessment of policies, programs and tools as they apply to PSSSP/UCEP.  

The audit procedures identified that funding agreements and national program guidelines are key foundational elements supporting PSSSP/UCEP objectives. Examination of a sample of funding agreements revealed that they fully reflect AANDC's standard templates. While, as noted in Section 1, the Department has limited ability to influence how funding is ultimately allocated to students, the audit confirmed that funding agreements included Terms and Conditions (T&Cs) which promoted the use of program funds for intended purposes. Examples of T&C s that promote the appropriate use of PSSSP/UCEP Program funding include those related to: 

  • All funding agreements state that the recipient must adhere to the terms and conditions of the program as set out in the NPG (see below for further details).
  • Unlike block funding agreements, recipients in a fixed funding agreement are not allowed to shift PSE funds away from PSSSP/UCEP without an unexpended funds plan. At the end of their current agreement, any surpluses may be recovered from the recipient.  These features are aimed at encouraging the use of funds for intended purposes. 

As noted above, all funding agreements include a requirement that recipients adhere with specific T&C s that are detailed within the NPG. The program guidelines were developed by the Program to articulate a common set of expectations applied to recipients including the expectation of PSSSP/UCEP related reporting. Examination of the current national program guidelines indicated it provides detailed guidance concerning selection priority criteria for funding decisions, the appeals process, and the requirement that recipients establish publicly available Local Operating Guidelines (LOG) that align to the requirements in the NPG. The funding agreements also require that recipients provide specific financial and performance information via annual Audited Financial Statements (AFS) and the departmentally provided Data Collection Instrument (DCI) respectively on an annual basis. Audit work conducted on-site at a sample of 4 Regional offices further indicated that some Regions have developed additional tools to assist recipients in complying with the national program guidelines. For example, the Yukon Region shared a locally developed budget allocation template and its Regional Operating Guide while BC Region had developed a DVD aimed at helping First Nations address program reporting requirements.

The audit also identified that an updated Performance Management Strategy (PMS) for Education had been developed in early 2014/15.  This PMS included 6 performance indicators for PSE, each of which link to outcomes in the Education Branch's Program Logic Model.  Due to changes to the data collection processes, further described in Section 5.1.3, there have been some delays in the operationalizing of performance reporting in support of the PMS.

Finally, the audit revealed evidence that Headquarters (HQ) Program Management had taken initiatives to improve and update the tools and guidance provided to Regions and Recipients. For example, examination of the current NPG and prior program guidelines revealed evidence of updates that serve to better define eligible expenditures under PSSSP/UCEP. We also identified a recent ESDPP sector initiative to develop a Draft Program Compliance Directive and Draft Program Recipient Compliance Review Handbook, which includes guiding principles, roles and responsibilities, procedures for conducting risk assessments and developing risk-based compliance plans, and procedures for conducting compliance reviews. These tools have not been implemented for PSE yet, and are designed to be used across ESDPP programming. 

5.1.2 Approach to Program Funding

Except for a relatively small volume of grants provided by certain Regional offices directly to eligible students, recipients are responsible to fund individual students in accordance with the national program guidelines. Based on a funding mechanism, each region receives PSE funding from HQ which they, in turn, distribute to recipients. As noted earlier, regional offices are afforded a degree of flexibility in the administration and delivery of programs. Nonetheless, the audit expected to identify funding mechanisms and funding allocation methodologies that promote consistent implementation of the PSSSP/UCEP Programs across Canada and equitable access to funding by eligible students in different regions. Moreover, there was an expectation for effective mechanisms to monitor, assess and address the alignment of Program objectives with Program resource capacity, such as needs versus funding level analysis, as a means to help align funding practices with funding needs.

The program has three types of funding: block, fixed and grants. Block funding for 2013/14 was approximately $132M, fixed funding was approximately $179M, and grant funding was approximately $1M. For block funding arrangements, the Department applies a formula to apportion an amount to PSE programming out of the total block funding allocation (as set out in the block agreements) for each region. The Region then allocates to individual recipients based on their respective block agreements. PSE recipients that are funded under block arrangements are allowed to reallocate funds to other programs provided they can demonstrate progress towards achievement of PSE's Program objectives.

For fixed funding arrangements, Program funding is based on historical funding patterns subject to a nominal annual increase of approximately 2%. The Department may allocate the 2% to mandatory expenditures in other educational programs. This is consistent with the approach identified in the 2009 internal audit. This approach has not been informed by any review or analysis to ensure that HQ allocations to the regions promote the objective of equitable access to funding across all regions. As a result, there is an increased risk that achievement of this objective will be diminished over time.

The 2009 internal audit also included a recommendation for a review of the sufficiency and appropriateness of recipient funding requirements. To date there has not been a formal needs assessment to assess the sufficiency of funding to recipients relative to program objectives. There are some practical challenges associated with conducting an effective needs assessment, such as the ability to identify potential students living off-reserve. Moreover, the issue of sufficiency of funding at the student level is complicated, and made more sensitive, by a number of contextual and historical factors as described in the Background section of this Report. While defining the sufficiency of funding is an important consideration for Program management, this audit did not include a detailed assessment of such an analysis. Irrespective of funding amounts, the nominal increases to annual funding are unlikely to match the well-publicized increases in the costs of PSE across Canada and, as such, there is a risk that delivery of program objectives will be eroded over time. In addition, the variability of demographic profiles across the country can impact where funding needs are most acute.

Also identified in the 2009 Audit was the fact that funding allocation methodologies were not consistent across all regions. This audit confirmed that regions continue to exercise some flexibility in how a portion of its funding is allocated. This flexibility allows individual regions to take initiative by implementing a funding allocation approach that they consider to be in the best interests of both the Program objectives and the Recipients in their region. For example, in one region visited the allocation methodology for Fixed funding incorporated some consideration of recipient funding needs. This was accomplished by redirecting some of the overall regional funding, that is above an initial allocation amount, into pools of funds from which additional funding may be accessed by recipients through a needs-based proposal process. Recipient needs are demonstrated through planned uses of funds and student waiting lists. This process was developed in collaboration with stakeholders that included recipient representatives.  As evident from this example, small differences in funding methodologies are not inherently inappropriate.  However, inconsistencies present a risk related to the fact that recipients in one region may not have the same access to additional funds as a similar recipient in another region. Moreover, the audit found that unique methodologies employed in some of the visited regions had not been reviewed or revised for some time.

Recommendation:

1. The Assistant Deputy Minister of ESDPP should review the appropriateness of how PSSSP/UCEP funding is allocated across regions to ensure it supports equitable access and, to the extent possible, promotes the alignment of funding practices with funding needs. Based on the outcomes of this review and overall Program objectives, the ADM of ESDPP should develop national guidance to support the consistency of funding allocation methodologies employed by regions while enabling regional differences when warranted.

5.1.3 Changes to the Data Collection Processes

The collection of relevant and accurate performance information is vital for measuring how well the PSE Program is achieving intended results and planning effectively for the future. As with many other AANDC programs, the efficient and effective collection of recipient data is an ongoing challenge for PSSSP/UCEP.  Previous audits and evaluations identified the need for more rigorous collection of performance information in order to support management decision-making, planning, and monitoring of PSE programs. In 2012/13 new recipient reporting requirements for PSSSP/UCEP were introduced as part of a transition to the Department's Education Information System (EIS). The audit expected that these changes to the data collection processes would result in an improved ability to monitor PSSSP/UCEP recipients and otherwise serve to improve overall programming.

Funding agreement T&C s require that Program recipients provide two key information sources. The first source is the annual AFS which include schedules designed to provide financial details of the recipient's use of funding, including a schedule outlining the use of PSSSP/UCEP funding. During the audit, there was evidence that AFS were being monitored by regional personnel for compliance with limits for spending on program administration costs and eligible expenditures. Section 5.1.4 provides further details on the nature and effectiveness of AFS monitoring.

The other key source of information is the Program's current reporting tool, referred to as the Data Collection Instrument (DCI), which is a fillable forms that all recipients must complete annually.. The DCI facilitates collection of a variety of information from program recipients on funded students. The DCI template includes instructions for the recipient on how to complete the forms and interviews revealed that some regions have provided DCI-related training to recipients. Once completed, the Region is responsible to review the DCI and then enter/upload the data to the EIS. Audit procedures conducted in the four regional offices visited indicate that the DCIs are being reviewed to ensure they are complete and that any error reports (e.g. recipient entered or attempted to enter invalid information such as an unapproved post-secondary institutions) have been cleared. As described more fully in Section 5.1.4, the monitoring does not extend to assessing the quality of information entered into the DCI before it is uploaded to EIS.

The EIS aims to support improved accountability for education programs, inform changes to policy and program development, and improve service delivery. While EIS is being rolled out within Education Branch, it has not yet fully operational in respect of PSE programs, subject to completion of data uploading and compliance validation. There has been a backlog of data input to EIS at the regional level whereby 2012/13 data was only fully input in 2014/15 and Education Branch is currently being assessed for compliance with T&C s and NPG in preparation for reporting and analysis. Therefore, while PSE information has been collected from recipients and entered into EIS, this information is not yet leading to the generation of reports that can serve to support management in compliance monitoring, decision-making, accountability or performance measurement. Further, the lack of reporting has meant the Program has little to no ability to assess the quality of reported information. Interviews with Program management and EIS representatives indicate that the Program has not yet documented a clear plan for how and when EIS reporting will be leveraged for PSE. The information gap created by the lack of PSE reporting , as EIS is rolled out to PSE programs, has significantly impaired Management's ability to support and report on the achievement of program performance objectives and monitor the extent to which recipients are in compliance with Program requirements.

Recommendation:

2. The Assistant Deputy Minister of ESDPP should develop a clear and formal strategy and plan as to how and when EIS reporting capabilities will be leveraged in support of the effective management and control of PSE programs (i.e. what reports, how often, to whom, etc.) as well as how training will support improved accuracy and consistency of data collection.

5.1.4 Approach to Monitoring

The audit included an assessment of the extent to which monitoring and oversight activities were conducted to promote compliance with program requirements and achievement of program objectives. Further, the audit assessed those mechanisms in place to monitor, assess and address the alignment of Program objectives with Program resource capacity. To this end, the audit procedures included interviews and sample testing in four regional offices to identify and assess processes for monitoring recipient's compliance with funding agreement T&C s including the national program guidelines. We also conducted independent assessments to identify the extent to which compliance objectives were achieved. Finally, we identified and assessed the processes in place at HQ to support alignment of resources and otherwise support effective monitoring of compliance and program objectives.  

While there was some variance in the depth of monitoring, audit procedures conducted in the four (4) regions visited indicated that Regional personnel were conducting limited in-office monitoring, comprised of the review of Audited Financial Statements (AFS) and the review of the DCIs. The AFS review was conducted for purposes of examining program expenditures to determine compliance with the requirement that administration costs not exceed 10%. The DCIs were found to be tracked and reviewed for purposes of confirming receipt and completeness of the templates. 

In none of the regions visited was there evidence of comprehensive compliance monitoring against all funding agreement T&C s or the NPG. Specifically, the breadth and depth of monitoring or compliance activities undertaken would not appear sufficient to routinely identify potential compliance issues or to fully promote the achievement of Program objectives. For those limited compliance activities conducted, follow-up actions taken with recipients were also found to vary across the regions visited and such practices were not formally established. Even when monitoring activities were identified, audit procedures revealed that instances of non-compliance were not always followed-up or addressed.  For requirements that are not monitored at the regional level, such as the requirement for recipients to develop and publicize LOGs, high rates of non-compliance were identified. Of the audit sample of 45 recipients, 15 LOGs were found to be available online. These 15 were reviewed for compliance to the program guidelines, and 14 contained the basic requirements set out in the NPGs in terms of having selection criteria and an appeals process. However none of the LOGs met all the required elements of an "appeals process" as set out by the NPG. For greater certainty, there was no evidence that regions have taken a risk-based approach to monitoring PSE recipients. While the General Assessment Process is applied to all program recipients as a means to assess their capacity, this process does not provide sufficient insight to inform a risk-based approach to monitoring.  Without a comprehensive, risk-based, approach to monitoring the likelihood of not achieving program objectives is increased as is the likelihood that recipients will not be in compliance with funding agreements T&Cs.

Based on interviews with regional and HQ personnel, there was a common understanding that resource levels at the regional level are a factor in the breadth and depth of PSE-related monitoring. For example, information obtained from a region not visited during this audit provided details of a regional PSE compliance program whereby compliance with PSSSP/UCEP T&C s and program guidelines was recently assessed on a student-by-student basis. Interviews with that region revealed evidence of compliance issues which, in some instances, involved the recovery of funds.

Notwithstanding the concerns raised above, the audit identified that ESDPP is currently in process of developing Sector-wide compliance tools including a new Program Compliance Directive and Program Recipient Compliance Review Handbook which, when rolled out to PSE programs, will address the conduct of compliance reviews for PSE. These documents contain draft PSE-specific compliance checklists developed by Education Branch. The audit also identified a parallel initiative led by Regional Operations Sector to address overall resource levels for compliance. This initiative is seeking to promote consistent and efficient compliance activities that are aligned with resource capacities through strategies that include developing common structures and resource classifications While these are positive developments, the timeframe for implementation of the new Sector-wide compliance regime to PSE programs will not likely occur until 2016-17. A sound implementation plan will be necessary to assure the effectiveness/sustainability of this new compliance regime. Development of such a plan will necessarily involve engagement with regions on such matters as reporting, tools, and resourcing. In the interim, a minimum standard for compliance should be established, based on recipient risk, in order to help address the compliance gaps identified and to mitigate the risk that funds aren't being expended for intended purposes.

Recommendation:

3. The Assistant Deputy Minister of ESDPP should establish an implementation plan and timeline for implementing ESDPP's new compliance regime within PSE. In the interim, develop a risk-based approach to monitoring whereby a minimum standard for compliance monitoring and follow-up across all regions is established and sustained.

5.1.5 Management of PSSSP and UCEP Grant funding

Three (3) regional offices administer approximately $1M grant funding: Yukon ($400k), Alberta ($500k), and Saskatchewan ($60k). Yukon and Alberta administer the program on behalf of those First Nations which have made arrangements with the regional office to administer the program on their behalf. Saskatchewan administers grants to Inuit students from that have resided outside Nunavut or the Northwest Territories for 12 consecutive months and are no longer eligible for PSE funding through their respective territorial governments.

The audit site visit to Yukon revealed that the region had established a Regional Operating Guide to administer the program. This Guide functions in the same manner as a LOG developed by other program recipients. The region has also established appropriate practices and protocols for applicant student intake, documentation requirements, and funding calculations. They had also developed a compliance and assessment checklist to determine whether the student is eligible to receive funding. Audit testing completed a sample of grant files indicated that controls were sufficiently effective.

Saskatchewan was found to have funded only 4 students in 2013/14. While the audit noted that granting processes were found to ad hoc, audit testing completed on a sample of grant files indicated that controls were sufficiently effective.

  

5.2 Redesign of the PSPP

5.2.1 Program Objectives

The audit examined the process undertaken by program management to define PSPP objectives as part of its redesign. The audit team expected that a program concept and objectives for the PSPP would have been established that addresses a need as determined through consultations or other means, and that are aligned with the Departmental mandate and priorities.

The audit found that departmental senior management requested that the former ISSP program be reviewed and renewed to ensure that the management and delivery of the program was aligned to its original intent and objective, and to government and departmental priorities. As such, the program was altered to include a more rigorous and transparent proposal review process to select projects to fund (discussed in greater detail in section 5.2.3 below), and on funding projects that would contribute to meeting labour market needs (either locally or nationally) and lead to higher employment among Aboriginal youth. To assess whether projects were addressing a local or national need, the program relied on the Employment and Social Development Canada's (ESDC's) labour market needs list and the knowledge of regional staff. While these sources may provide some useful insight into assessing need, without undertaking a formal needs assessment that included stakeholder input, the extent to which the program is considering the greatest needs of First Nations' communities and students is not clear.

Additionally, the audit team was informed that program redesign was intended to introduce a greater emphasis on establishing partnerships, where proposed projects that included additional funding partners would receive preference over those without. The name of the program was also changed to include the word "partnerships" to stress this focus. The intent was that this would allow the Department, and the funding partners, to lever off the other's contribution to achieve results. Program management has embedded these new areas of focus (i.e. labour market needs and partnerships) into the rating scoring criteria used by staff to assess proposals.

The audit also reviewed Education Branch's Performance Measurement Strategy (PMS) to determine the expected results for the PSPP and defined indicators of program success. The audit team found that the PMS does not include any PSPP-specific outcomes and indicators.

Overall, the audit found that the primary objectives of the PSPP were not clearly reflected or reinforced through the program Performance Measurement Strategy and the proposal review process. While the 2014/15 Departmental Report on Plans and Priorities describes the importance of stronger relationships to labour market needs, this factor comprises only 20% of the rated scoring applied to proposals. Further, labour market alignment is not identified as a mandatory requirement. This effectively diminishes the importance of this objective as projects could still be approved without scoring well in the area identified as most closely linked to Departmental objectives. Furthermore, it is unclear how program performance will be assessed or what constitutes program success without clear program outcomes and indicators. This lack of clarity in program objectives and outcomes presents further challenges in identifying the alignment of the program with the Departmental mandate and priorities.

5.2.2 Policies, Procedures, and Tools

The audit also examined the policies, procedures, and tools that are in place to assist staff in program delivery. The audit team expected to see updated program documentation that sets out program standards, roles and responsibilities of staff, and expectations of recipients.

The audit found that program management has updated the national program guidelines to reflect the redesign of the PSPP, and has provided it to regional staff. The updated program guidelines clearly set out the program standards in terms of eligible activities, target recipients, eligible expenditures, and other key characteristics for the program. The program guidelines also define the proposal assessment criteria and weighting for each criterion.

Program management has also developed a National Selection Committee (NSC) Terms of Reference, and has provided it to regional staff. The NSC Terms of Reference clearly sets out the roles and responsibilities and steps involved in the proposal assessment process.

Expectations of PSPP recipients are set out in their funding agreement, which includes undertaking project activities in accordance with their approved proposal and the national program guidelines. It also requires recipients to submit an annual PSPP report using the departmental PSPP DCI template.

While the funding agreements set out recipient reporting expectations, the audit found that the DCI template and its data elements have not been updated after the redesign of PSPP. As such, some data elements within the DCI template do not appear to be relevant to the redesigned program. For example, the categories of expenses within the DCI include expenses for elder/guest speakers which no longer appear as eligible expenses under the PSPP program guidelines.. Furthermore, as noted earlier, the program has not yet established PSPP-specific indicators, so it is unclear whether information collected from submitted DCIs will enable an assessment of program performance.

5.2.3 Proposal Review Process

A key component of the redesign of the PSPP was a revised proposal review process. Prior to 2013/2014, the regions allocated ISSP funds to recipients. Program management has since moved this process to a national-level through the establishment of the NSC, which still includes both HQ and regional involvement. The NSC is responsible for the review and approval of PSPP proposals. In 2014/2015 the NSC members included:

  • The Assistant Deputy Minister of the ESDPP Sector (Co-Chair);
  • The Senior Assistant Deputy Minister of the Regional Operations Sector (Co-Chair);
  • The Director General of the Education Branch;
  • The Director of Education Programs; and
  • The 10 Regional Directors General (RDGs).

The proposal review process involved the regions submitting a call for proposals to potential applicants within its jurisdiction, and then receiving proposals from interested applicants. An initial review was then conducted of each proposal by the region that involved an assessment of eligibility and a scoring (out of 100) against the proposal assessment criteria set out in the national program guidelines.

Once the initial review was completed, Education Branch then assigned each proposal to two other regions (or to HQ officials within Education Branch) to conduct a second and third review (i.e. a peer review). The scores from all three reviews were then averaged for final score. Final funding decisions were then based on the final scores as well as other factors (such as whether the department had previously committed to funding a program that has existing students).

Throughout the process the NSC held regular meetings via conference call in order to discuss projects and differences in scoring. The quorum for an NSC meeting was 5 RDGs, and must include at a minimum those RDGs for which proposals are being challenged.

In total, for the 2014/2015 proposal review process, the Department received 220 PSPP proposals. Of those, 213 were subjected to three reviews, and 87 were approved for funding. Table 1 below provides a breakdown of regional and HQ involvement in the process in terms of the number of proposals received, the number of proposals not reviewed, the number of proposal reviewed, and the number of proposals approved. It highlights the unusually small number of proposals that were not subject to review (3%) based on program eligibility. It also highlights that collectively, HQ and Regions conducted 639 reviews. Interviews with regional staff indicated that the time required to complete the review of one project proposal varied based on factors such as the amount of material provided by the proponent for each project. With this variability in mind, regional staff indicated that the average time to simply review and score a single proposal was approximately 2.5 hours, not including the administrative time to manage the large volumes of information/documentation or the senior level briefings associated with the reviews. Based on 639 reviews, this suggests a total effort of over 213 person days dedicated to proposal review.

The audit noted that due to the call for proposals for 2014-15 PSPP funding not being initiated until February 2014, combined with the significant time investment associated with the NSC proposal review process, many successful proponents did not receive their funding until well after their projects began in September 2014. Program management has since addressed the timing issue for 2015-16 by issuing the call for proposals during the fall of 2014.

Table 1: Breakdown of Regional/HQ Involvement in the PSPP Proposal Review Process
  BC AB SK MB ON QC Atl NU YU NT HQ Total
Number of Proposal Received 38 40 20 14 78 15 11 3 1 0 - 220
Number of Proposals Not Reviewed 1 3 1 0 1 0 1 0 0 0 - 7
Number of Proposals Reviewed (Initial Reviews + Peer Reviews) 86 81 67 65 79 55 40 53 48 23 42 639
Number of Project Proposals Approved 9 9 11 5 42 7 2 2 0 0 - 87
 

In its examination of the proposal review process, the audit team expected that a formal process would be established and documented, guidance materials would be developed to assist staff to make assessment decisions, and an analysis would have been undertaken to determine the amount of resources required to conduct the review.

The audit found that the process is clearly described in the NSC Terms of Reference, and guidance materials in terms of scoring assessment templates were also provided to regional staff to assist in the reviews. While some regional staff raised concerns around potential subjectivity in scoring, program management is continuing to refine the templates to minimize subjectivity.

The audit also found that the proposal review process was not supported by a resource requirements analysis. Staff from all regions visited raised concerns over the significant level of effort required to complete the reviews within a tight timeframe. In some cases, the review process forced staff to drop other work for a prolonged period in order to the complete the reviews. Concerns were also raised about the amount of effort required at the senior management level (i.e. RDG). While it was regional program staff that conducted the reviews, RDGs had to be briefed on them prior to the regular NSC meetings.

Based on the audit observations, the level of effort and resources required to undertake the proposal review process appears to not be aligned with the relative materiality and risk associated with the program.

Recommendation:

4. The Assistant Deputy Minister of ESDPP should clarify and confirm PSPP Program objectives in terms of their priority and alignment with broader Departmental priorities. Following this clarification and confirmation, the Program should

  • Develop relevant performance indicators and align these with the Program DCI; and
  • Identify and implement efficiencies in proposal review process to ensure that the resources and effort spent is commensurate with Program's significance, materiality and risk.

5.3 Follow-up on Previous Audit and Evaluation Recommendations

Since 2009, 18 recommendations have been made in connection with previous PSE-related audits and evaluations. 15 of these recommendations were within the scope of this audit and therefore can be linked audit criteria as provided in Appendix A. Appendix B provides a mapping of previous audit and evaluation recommendations to this report as well as a summary of the results of this follow-up. This section contains details on those previous audit/evaluation recommendations not specifically addressed elsewhere in this report.

5.3.1 Improve the Program's performance measurement framework (2009 Internal Audit, Recommendation #4):

The audit identified that Program Management had implemented enhancements to the PMS for the PSSSP/UCEP Program which included the development of new performance indicators. Examination of these indicators demonstrated an appropriate linkage to the Program's expected outcomes. Further, as referenced in Section 5.1.3, it was determined that the information requirements within the PMS reflected alignment with the updated DCI.  

While there have been improvements to the Program's performance measurement framework the substance of the 2009 recommendation was that the Performance Measurement produces data to permit adequate and appropriate analysis evaluation and reporting. As noted in Section 5.1.3, performance reporting is not in place as the EIS system has not been fully implemented for PSE. In the absence of this reporting, the Program has yet to establish performance targets nor have performance results been reported for PSE. As part of Management's response to the recommendation in 5.1.3, there is an expectation that performance reporting will support development of performance targets.

5.3.2  Improve clarity of Program requirements and delivery standards for recipients and for AANDC personnel involved in monitoring of these requirements (2009 Internal Audit, Recommendation #6):

The follow-up of this recommendation identified that Program requirements and delivery standards are now clearly set out in the national program guidelines. These guidelines were found to be updated annually and there was evidence of periodic improvements such as the addition of limitations regarding the administrative expenditures.  The program guidelines are shared with appropriate Departmental staff and are posted to the departmental website. When requirements associated with the national program guidelines were changed, notifications are provided to recipients.  In addition, some AANDC regional offices have supplemented the program guidelines with region-specific guidance which is provided to recipients and is also updated regularly.

Based on the follow-up procedures, Management has appropriately and sufficiently addressed this 2009 recommendation.  

5.3.3 Establish a management control framework (2009 Internal Audit, Recommendation #7):

The 2009 audit recommendation indicated an expectation for a Program-specific Management Control Framework which included, as a minimum, the following:

  1. A process for regularly updating the foundations of the Program, such as the Program T&C s, the Program Policy, the PSE Guidelines, and possibly an Operations Manual, to ensure that these documents remain relevant and sufficiently comprehensive;
  2. An annual planning process that establishes objectives and targets for the Program each year, with a view to the continuous improvement of Program results (through the measuring and assessing of Program performance) and the enhancement of Program efficiencies;
  3. The provision of guidance, direction to and oversight of regional offices through the provision of support tools, regular communications, operational guidelines, and well defined and communicated roles and responsibilities;
  4. A resource planning process that considers the level of personnel needed to administer the Program, to meet Program objectives, and to ensure sufficient Program and agreement monitoring and oversight; and
  5. A risk management process that identifies Program risks on an annual basis, provides a strategy for mitigating such risks on an ongoing basis, and ensures regular monitoring and sufficient risk based compliance auditing of recipients.

Follow-up procedures on these elements indicated that the key components of the Program's  current management control framework consists of various guidance documents, such as the NPGs, some region-specific guidance, management meetings, and the provision of guidance to regional offices.  In assessing these components, the audit identified that items (i) and (iii) as described above had been adequately addressed. However, there were ongoing gaps in terms of the following:

Annual Planning Process – As noted in Section 5.3.1, performance targets were yet to be established in connection with the updated Performance Management Strategy, as such there continues to be a gap in respect of this previous audit recommendation. The recommendation provided in Section 5.1.3 addresses this item.

Resource Planning Process – Follow-up procedures did not identify evidence that the Program had developed or implemented a planning process that considered the level of personnel needed to administer the Program, to meet Program objectives, and to ensure sufficient Program and agreement monitoring and oversight. As referenced in Sections 5.1.3 and 5.2, resource alignment concerns have impacted monitoring activities and efficiency of PSPP delivery respectively.

Risk Management Process – Follow-up procedures revealed the existence of a branch wide risk assessment covering all education programs. There was, however, no PSE program-specific risk management process that would, among other things, support development of strategies for risk mitigation, including risk-based performance monitoring and compliance activities.

The existence of ongoing gaps in the management control framework supporting PSE programs not only increases the risk that program objectives will not be achieved in accordance with requirements, but also the risk that prevailing management practices are ineffective, inconsistent and/or inefficient. 

The audit team recognized that the Audit the Management Control Framework for Grants and Contributions (2013-14) was completed recently, and included a recommendation relevant to these findings. The objective of that audit was to assess the adequacy and effectiveness of departmental processes in supporting the design and approval of risk-based program control frameworks. It included an audit recommendation that the Department "…review and clarify departmental processes, governance structures, accountabilities, responsibilities and authorities for developing and approving program control frameworks…". Notwithstanding the over-arching nature of that previous audit recommendation, this audit included an assessment of the current state of the PSE Program's Management Control Framework and it is expected that the program can address the control gaps identified in the recommendation below in a timely manner, while also participating in the management actions to be implemented in response Audit the Management Control Framework for Grants and Contributions (2013-14). 

Recommendation:

5. The Assistant Deputy Minister of ESDPP should strengthen key elements of control to support overall governance of the PSE Programs. In particular, it is recommended that:

  • A Program-specific risk assessment should be completed by Education Branch on PSE programs upon which to implement risk-based controls to mitigate identified risks;
  • A resourcing plan be developed to align the required level of resources with the operational needs for program delivery and monitoring activities. 
 

 

6. Management Action Plan

Recommendations Management Response /
Actions
Responsible Manager (Title) Planned Implementation and
Completion Date
(Month & Year)
1. The Assistant Deputy Minister of ESDPP should review the appropriateness of how PSSSP/UCEP funding is allocated across regions to ensure it supports equitable access and, to the extent possible, promotes the alignment of funding practices with funding needs. Based on the outcomes of this review and overall Program objectives, the ADM, ESDPP should develop national guidance to support the consistency of funding allocation methodologies employed by regions while enabling regional differences when warranted. Regional allocations should reflect eligible program populations. Changes in funding allocations to Regions will likely have important impact on recipients. Implementation timelines should factor-in consultations and notifications, in line with the annual funding cycle. Assistant Deputy Minister, ESDPP

In consultation with the CFO and Regional Operations Sector
  • Document the current methodologies to fund the PSSSP recipients – April 2015
  • Calculate new regional allocations and analyse impact on recipients – June 2015
  • Consult with recipients 2015-2016
  • Confirm new allocations to recipients - Fall 2016
  • Implementation – April1, 2017
April 2017
As a subsequent phase to reviewing regional allocations, national guidance will be provided on the management of funds to recipients with a view to maximize program results.  June 2017
2. The Assistant Deputy Minister of ESDPP should develop a clear and formal strategy and plan as to how and when EIS reporting capabilities will be leveraged in support of the effective management and control of PSE programs (i.e. what reports, how often, to whom, etc.) as well as how training will support improved accuracy and consistency of data collection. Regional offices to continue using EIS to load proposals and reports with HQ support and training. Target dates for processing Data Collection Instrument included in Quarterly Reports. Assistant Deputy Minister, ESDPP  
  • PSSSP data for 2012-2013 available in the Education Reporting and Analysis Solution (ERAS). The ERAS produces dashboards, results by recipients and performance measures. It will be used to define national guidance in regional allocation methodology noted in action item 1 above.
Jointly with Senior Assistant Deputy Minister, Regional Operations

September 2015

  • PSSSP data for 2013-2014 available in ERAS

December 2015

  • PSSSP data for 2014-2015 available in ERAS
Spring 2016
3. The Assistant Deputy Minister of ESDPP should establish an implementation plan and timeline for implementing ESDPP’s new compliance regime within PSE. In the interim, develop a risk-based approach to monitoring whereby a minimum standard for compliance monitoring and follow-up across all regions is established and sustained. ESDPP ADM will implement the ESDPP compliance regime on April 1, 2015. ESDPP will use this regime – which is risk-based, to input into the regional compliance plans in 2015-2016. Assistant Deputy Minister, ESDPP

April 2016

The SADM of Regional Operations remains responsible for resourcing the regional offices with capacity to conduct compliance. Senior Assistant Deputy Minister, Regional Operations
The SADM of Regional Operations to fully implement the new compliance regime for PSE as per the conclusion of the RO-led project on regional compliance units. March 31, 2017
4. The Assistant Deputy Minister of ESDPP should clarify and confirm PSPP program objectives in terms of their priority and alignment with broader Departmental priorities. Following this clarification and confirmation, the Program should
  • Develop relevant performance indicators and align these with the Program DCI; and
  • Identify and implement efficiencies in proposal review process to ensure that the resources and effort spent is commensurate with Program’s significance, materiality and risk.
PSPP National Program Guidelines will be reviewed to clarify and confirm program objectives.

As National Guidelines, DCI and call for proposals are already published and finalized for 2015-2016, this will be considered for 2016-2017.
Assistant Deputy Minister, ESDPP December 2015
  • Consider new performance indicators and review DCI
National Selection Committee will review lessons learned from the 2015-2016 process to see how much reduction in resources level was achieved compared to 2014-2015. If further reductions of efforts are necessary, alternative options will be explored in the Summer of 2015.

Fall 2016

  • Publish Terms of Reference for the national selection process for PSPP
Fall 2015
5. The Assistant Deputy Minister of ESDPP should strengthen key elements of control to support overall governance of the PSE Programs. In particular, it is recommended that:
  • A Program-specific risk assessment should be completed by Education Branch on PSE programs upon which to implement risk-based controls to mitigate identified risks;
  • A resourcing plan be developed to align the required level of resources with the operational needs for program delivery and monitoring activities.
The Education Branch risk profile will be reviewed for inclusion of PSE-specific risks and include PSE-specific mitigation strategy.
  • New risk profile and mitigation strategy – April 2015
Assistant Deputy Minister, ESDPP

December 2015

SADM of Regional Operations to develop its human resources strategy and business plan for regional offices which are responsible for program delivery and monitoring. The plans should be done in consultation with ESDPP ADM who has the lead for program policy. Senior Assistant Deputy Minister, Regional Operations As per planning cycle
 
 

 

Appendix A: Audit Criteria

To ensure an appropriate level of assurance to meet the audit objective, criteria were developed to address each of the assertions included within the scope of the audit (as provided in Section 2 of this Report). The audit criteria were drawn from the Management Accountability Framework, Treasury Board Policies, prior audit and evaluation reports, as well as AANDC’s internal requirements. The criteria for the audit, in order that they appear in this Audit Report, were as follows:

Audit Criteria
Management and Delivery of PSSSP and UCEP
1.1 PSSSP and UCEP Programs have implemented appropriate processes, procedures and supporting tools to monitor and promote the achievement of Program objectives.
1.2 Mechanisms are in place to monitor, assess and address the alignment of Program objectives with Program resource capacity.
1.3 Mechanisms are in place to ensure funding allocation methodologies employed across the department promote consistent implementation of the PSSSP/UCEP Programs and equitable access to funding by eligible students in different regions.
1.4 Reporting requirements facilitate the ability to monitor PSSSP recipients and improve overall programming.
Redesign of the Post-Secondary Partnerships Program
2.1 The PSPP redesign is in alignment with departmental priorities.
2.2 Appropriate policies, procedures, and tools have been developed and are being used to facilitate the achievement of PSPP's objectives.
2.3 The PSPP has the resource levels and competencies needed to effectively manage and administer the program.
Follow-up on Previous Audit and Evaluation Recommendations
3.1 Actions taken in response to recommendations made in past audits and evaluations have been sufficient to address issues raised and related recommendations.
 
 
 

 

Appendix B: Prior Audit Recommendations

AANDC 2009 Audit Recommendations Results of Follow-up Mapping to 2014/15 Audit Criteria
1. The Director General, Education Branch should develop a process to determine actual Program spending by recipients and to incorporate this information into Program planning, results measurement, and an ongoing Program adjustment process to further the success of the Program. While the new DCI and EIS were designed to better support Program management's information needs, management reporting has not yet been fully operationalized. Moreover, there is no national process to track and quantify recipient spending and surpluses in a manner that can support management decision-making. 2.4
2. The Director General, Education Branch, in conjunction with the Transfer Payments and Financial Policy Directorate, should re-assess the funding authorities in use and the reporting needs of the Program, taking into consideration the department's obligation to account for the use of Program funds and the intended purposes of these funding authorities.  A new recipient reporting process has been developed and implemented. However, as management reporting has not yet begun is it too early to determine if the Department's obligations to account for the use of funds has been fully met. 1.1
2.4
3. The Director General, Education Branch should undertake a review of the sufficiency and appropriateness of recipient funding requirements, in light of increasing education costs and the available information on wait-listed students, to ensure that the department is able to meet the objectives of the Program. A new DCI has been developed, However management reporting has not yet begun. Further, no needs assessment has been completed to assess the sufficiency of funding in meeting program objectives. 1.2
4. The Director General, Education Branch should establish a process to enhance the performance measurement framework currently in place in support of the Program to improve the relevance and integrity of the performance data currently being captured by the Branch, in conjunction with CIMD; identify and capture new data that would enhance the existing performance measurement framework; and ensure that the Branch's new Performance Measurement System produces data to permit adequate and appropriate analysis evaluation and reporting. See section 5.3.1 of this Report. 2.4
3.1
5. The Director General, Education Branch should undertake a comprehensive review of the allocation methodology employed for the distribution of Program funds, in conjunction with Regional Directors General, to ensure eligible students in different regions have equitable access to Program funds. This review may need to consider the ability of each region to meet Program requirements within its existing core budget. There is no evidence of such a review nor has the Program implemented a needs based approach at either a regional or recipient level 1.3
6. The Director General, Education Branch should ensure that the minimum Program requirements and delivery standards (requirements) provide sufficient clarity to recipients to ensure they understand departmental expectations in operating a post-secondary education program, and to AANDC personnel to ensure they are able to assess and determine, as part of Program monitoring, whether recipients are meeting Program requirements. See section 5.3.2 of this Report 3.1
7. The Director General, Education Branch should establish a management control framework (a set of policies and procedures to ensure that results are achieved and Program objectives are met) specific to the Program that includes, but is not necessarily limited to:
  • a process for regularly updating the foundations of the Program, such as the Program T&C s, the Program Policy, the PSE Guidelines, and possibly an Operations Manual, to ensure that these documents remain relevant and sufficiently comprehensive;
  • an annual planning process that establishes objectives and targets for the Program each year, with a view to the continuous improvement of Program results (through the measuring and assessing of Program performance) and the enhancement of Program efficiencies;
  • providing guidance, direction to and oversight of regional offices through the provision of support tools, regular communications, operational guidelines, and well defined and communicated roles and responsibilities;
  • a resource planning process that considers the level of personnel needed to administer the Program, to meet Program objectives, and to ensure sufficient Program and agreement monitoring and oversight; and
  • a risk management process that identifies Program risks on an annual basis, provides a strategy for mitigating such risks on an ongoing basis, and ensures regular monitoring and sufficient risk based compliance auditing of recipients.
See section 5.3.3 of this Report 3.1
8. The Director General, Education Branch should, in conjunction with Regional Directors General, ensure that the audit review process be enhanced to ensure that non-eligible Program expenses are appropriately addressed and that Program surpluses are adequately investigated, and that, if necessary, the recipient be subject to further review or auditing activities. The review of Audited Financial Statements was found to be limited and not risk based, with little or no follow-up on anomalies or indications of non-compliance. 1.1
9. The Director General, Education Branch, in conjunction with the Transfer Payments and Financial Policy Directorate, should re-assess the financial reporting requirements of the Program, taking into consideration the department's obligation to ensure due diligence in the monitoring of Program recipients and that funds are spent for the intended purposes. The new DCI will provide more detailed information on recipient spending and information to support the PMS and program monitoring, however management reporting has not yet begun. 2.4
10. The Director General, Education Branch should conduct a thorough review of eligible Program expenses and, to the extent that it is determined that the eligible expenses in the Program T&C s are deficient, update the Program T&C s, Program Policy and the PSE Guidelines to include those expenses deemed appropriate Eligible and ineligible expenditures have been better defined in the National Program Guidelines. 1.1
11. The Director General, Education Branch should establish, in conjunction with Regional Directors General, a process for conducting appropriate recipient monitoring and compliance auditing (financial and non-financial, on a risk based approach) to ensure that recipients are adhering to Program (including ISSP) and funding agreement T&C s. Consideration should also be given to resources and capabilities within the regional offices to conduct monitoring and complete compliance audits. While introduction of new DCIs is an improvement, monitoring is limited, not risk based, and does not reflect consideration of resources and capabilities. 1.1
12. The Director General, Education Branch should, in conjunction with Regional Directors General, establish a process to be employed in the regions to ensure that the projects funded through the ISSP component of the Program adequately support all aspects and objectives of the Program. Program redesign included clarity in the processes employed in the regions. However, issues remain regarding overall clarity of program objectives. 2.2
13. The Director General, Education Branch should, in conjunction with Regional Directors General, ensure that regional offices adhere to the Program T&C s, PSE Guidelines, and Recipient Reporting Guide in regard to obtaining, reviewing, and evaluating ISSP proposals, and obtaining of annual ISSP reports from recipients A more rigorous process has been implemented to support the proposal assessment and selection process of the PSPP program 2.2
14. The Director General, Education Branch should, in conjunction with Regional Directors General, ensure that ISSP recipients provide adequate financial reporting in support of ISSP funds received and expenses incurred, and that these reports be sufficiently reviewed by regional office personnel to ensure that the expenditures are eligible. The new PSPP DCI requires a breakdown of expenses). However, these categories do not align with the program’s eligible expenses. 2.2
 
AANDC 2012 Formative Evaluation Recommendations Results of Follow-up Mapping to 2014/15 Audit Criteria
1. Develop a strategic, transparent framework to ensure clear definitions of roles, responsibilities, commitments, funding approaches and accountabilities for post-secondary education.  [Relates to reviewing options for education reform.] n/a Not in Scope
2. Further support partnership development with First Nations, education organizations and institutions to facilitate student transitions and support. [Relates to reviewing options for education reform.] n/a Not in Scope
3. Work to develop data collection methods that allow for the reliable assessment of student enrollment and success. A new DCI has been developed but management reporting has not yet begun 2.4
 
Office of the Auditor General of Canada (OAG) 2011 Audit on Programs for First Nations Mapping to 2014/15 Audit Criteria
  1. Indian and Northern Affairs Canada, in consultation with relevant federal organizations, should develop implementation plans for the recommendations on which progress to date has been unsatisfactory or incomplete. The plans should include specific goals, targets, action items, timelines for achieving results, and indicators for measuring progress.

    Previous Recommendations Followed Up by OAG Related to:
    1. Indian and Northern Affairs Canada, in consultation with First Nations, should immediately develop and implement a comprehensive strategy and action plan, with targets, to close the education gap. It should also report progress to Parliament and to First Nations on a timely basis. (Recommendation 5.33 of the 2004 November Report of the Auditor General of Canada, Chapter 5. Indian and Northern Affairs Canada—Education Program and Post-Secondary Student Support) Progress: Unsatisfactory
    2. Indian and Northern Affairs Canada should undertake to obtain reliable and consistent information on the actual costs of delivering education services on reserves and compare the costs with those of providing comparable education services in the provinces.(Recommendation 5.51 of the 2004 November Report of the Auditor General of Canada, Chapter 5)  Progress:  Satisfactory
Not in Scope
 
 

 

Appendix C: Relevant Policies/Directives

The following authoritative sources were examined and used as a basis for this audit:

Policies, Program Guidelines:

Planning, Monitoring and Reporting Tools:

 
 
Date modified: