Based on the evidence gathered through the examination of documentation, analysis and interviews, each audit criterion was assessed by the audit team and a conclusion for each was determined. Where a significant difference between the audit criterion and the observed practice was found, the risk of the gap was evaluated and used to develop a conclusion and to document recommendations for improvement.
5.1 Roles, Responsibilities and Accountabilities
Directive 410 – General Assessments (Directive 410), developed by TPCOE, which took effect on April 1, 2011,outlines certain key roles, responsibilities and accountabilities in the completion, review and approval of GAs, as well as the maintenance of the GA framework, including the development of training and working tools. The User Guide, revised in August 2011, also includes the Terms of Reference for the Regional/Sector Transfer Payment Management Committee.
Specifically, Directive 410 assigns responsibility for completing the GAs to Regional Funding Services or program equivalent with oversight provided by Regional Directors General and/or Director Generals of the program branch, the Chief Financial Officer (CFO) sector and the regional and/or sector Transfer Payment Management Committees, or equivalent. The audit observed that although the assigned roles, responsibilities and accountabilities of certain key personnel involved in the GA process are clearly defined, there was a lack of a consistent understanding of some assigned roles. The audit noted that some improvements are necessary to ensure a consistent approach to the completion of programs' components of Part A GAs and HQ sector completed Part B GAs, as well as the review and challenge of Health Canada's assessments.
5.1.1 Accountability over Programs' Components of Part A General Assessments
Through the conduct of regional site visits, the audit noted that accountabilities for the completion, review and challenge of programs' components of the Part A GAs were inconsistently assigned. Specifically, the completion of the programs' portion of the GA was, at times, assigned directly to regional program representatives with little to no challenge by the assigned FSO and at other times, completed solely by the FSO with little to no consultation with program representatives. Directive 410 does not stipulate the level of accountability programs should have in the completion of the GAs; rather, it outlines that "in Regions, the Lead Officer (FSO) will coordinate with Regional Program Responsibility Centre Manager and specialists, and where appropriate, with HQ Responsibility Centre Manager to complete the Program Management component of the assessment". Since the programs' components of the GA accounts for approximately a third of the total GA score, discrepancies in the level of engagement of both program and funding services representatives in the assessment has led to inconsistencies in both the scoring and level of justification provided for scores. For example, the audit noted some programs were providing little to no justification for the scores given, while some others were providing justifications not aligned to the risk considerations outlined in the GA Workbook.
5.1.2 Roles and Responsibilities for HQ Sector Completed Part B General Assessments
The audit noted inconsistencies in the assigned role and responsibilities within HQ sectors for the completion, review and approval of Part B GAs. Although Directive 410 andthe User Guide, do not differentiate between the role and responsibilities for the completion of Part A, Part B or Part C GAs, the audit noted that Part B GAs are at times, being reviewed, challenged and approved by a Responsibility Centre Manager, and at other times, the assessments are only subject to a peer review.
Without the consistent application of a review and approval process to all Part B GAs, which includes the review and approval of the GAs by an individual or oversight committee with sufficient authority, there is an increased risk that assessments with inconsistent scoring or inadequate justification for scoring will not be identified and amended as required.
5.1.3 Role and Responsibilities in the Review and Challenge of Health Canada's Assessments
Part A GAs developed for recipients under joint, ongoing multi-program agreements for the fiscal year 2014-2015 included an assessment of the recipients' program management activities associated with both AANDC and FNIHB programs. Health Canada was responsible for providing the assessment results for FNIHB program which feed into the overall final GA score used for decision-making by AANDC.
Through the conduct of interviews with regional representatives, the audit noted an inconsistent understanding regarding the level of review and challenge required of regional personnel over the scoring and level justification provided by HC. In one of the regions visited, the ratings provided by HC were entered directly into the Grants and Contributions Information Management System (GCIMS) with no review or challenge by AANDC; however, in another region, the assigned FSO reviewed and challenged the assessment ratings being provided by HC. Regional interviewees noted a lack of direction from TPCOE regarding regional responsibilities in reviewing and/or challenging the assessment results provided by HC.
Due to the fact that the ratings provided by Health Canada can have a significant impact on the Program Management consideration score given to a recipient with a joint, ongoing multi-program agreement, it is critical for clear expectations of regional responsibilities regarding the review and challenge of the impact of HC's assessment on the Department's overall scoring of recipients.
5.1.4 Departmental Quality Assurance Program over GAs
With the expectation that Part A GA results will be made public, it is important to ensure that regions are completing the assessments in a consistent manner, and that scoring is consistent across regions. Directive 410 outlines that the CFO Sector is responsible for "providing periodical oversight and activities to ensure compliance with the Directive". In order to meet this requirement, there is an expectation that TPCOE develop and implement a quality assurance/monitoring program across the Department. A quality assurance/monitoring program would include the review and challenge of the assessment results across regions/sectors.
Interviews with a sample of regional and TPCOE representatives, revealed that no quality assurance/monitoring program has been developed and implemented Department-wide. Rather, to date, regional offices have been responsible for developing and implementing their own quality assurance/monitoring processes to identify inconsistencies in the scoring and level of justification provided within the region; however, no holistic review regarding the consistency of scoring was performed.
Without a robust quality assurance/monitoring program which assesses the consistency of scoring across regions and challenges the level of justification provided for scores, there is an increased risk that regional differences in the completion and scoring of GAs will not be identified and addressed in a timely manner.
The audit was informed that Regional Operations is considering the implementation of a peer review process, whereby a sample of regionally completed Part A GAs would be reviewed and challenged by another region. This proposed peer review process could be a key component of a Department-wide quality assurance/monitoring program, whereby the results or issues associated with the review are escalated and addressed by TPCOE.
- For recommended actions regarding clarifying roles and responsibilities in Directive 410, pleaserefer to recommendation number four (4) bullet A on page 18.
1. The Grants and Contributions Management Oversight Committee should develop and coordinate a Department-wide quality assurance/monitoring program in order to provide the Department with a level of assurance that GAs are being completed in a consistent manner and assessment results are supported by adequate justification. The quality assurance/monitoring program should include the review and assessment of Part A, Part B and Part C GAs.
5.2 Training, Tools and Guidance
The audit team noted gaps in the level of training provided to those assigned with responsibility for completing, reviewing or approving GAs. Additionally, the audit identified opportunities for improving the objectivity of the GA process through enhancements to the GA Workbook.
TPCOE has developed and provided training to HQ sectors and regional offices on how to complete GAs using the GA Workbook and GCIMS when the GA process and tools were first rolled-out. Subsequent to that training, only online training on how to use the GA tool in GCIMS (i.e. the keystrokes required to complete the assessments) has been made available. This training does not provide guidance on how to interpret the risk considerations outlined in the Workbook nor the level of justification required for scoring. Further, through the conduct of interviews with a sample of HQ sector and regional office representatives, many key individuals responsible for completing the GAs were unaware of the availability of the online training and expressed a desire for substantive GA training.
TPCOE has recently acknowledged the need for additional training on the GA process and issued a Statement of Work to hire a contracted resource to review and revise existing training materials, as well as provide "Train the Trainer" sessions to all regional offices starting in the Fall of 2015.
5.2.2 Tools and Guidance
Those tasked with completing the GAs are required to follow the requirements outlined within the Workbook. The Workbook outlines the risk categories and considerations to be used in the assessments, as well as the benchmarks for the low, medium and high rating scales for each risk consideration. The benchmarks outline the elements that a recipient needs to have in place in order to reach that risk rating.
Through interviews with FSOs, the audit team noted that the degree to which the Workbook was actively referenced during the completion of the GA Part A varied between FSOs. Generally, newer FSOs would rely on the Workbook to a greater extent than more experienced FSOs. The risk with an inconsistent use of this tool is that FSOs may begin to stray from the key benchmarks laid out in the Workbook, and apply different assessment criteria than other FSOs. To address this risk, the audit team strongly suggests that TPCOE, as part of the "Train the Trainer" sessions, emphasize to FSOs and GA reviewers the importance of applying the benchmarks and guidance as included in the GA Workbook to ensure that consistent assessment criteria are being applied to General Assessments.
Generally, the audit team noted no concerns regarding the relevancy and appropriateness of the benchmarks established for Part B GAs, as well as the first three risk categories (Governance, Planning and Financial Management) for Part A GAs. However, certain regional program representatives noted concerns regarding the relevancy and application of the benchmarks outlined in the Workbook for Program Management risk factor and its considerations. For example, program representatives noted requiring additional guidance on: what constitutes disruptions, delays, and gaps in service/project delivery; what are the expectations in regards to program staff qualifications and capacity; and, what policies and plans are required for sound management of the program.
While the national Workbook can only provide general program guidance as it is used across multiple programs, certain program staff have found the need for more customized program specific guidance, and as such, have developed their own checklists/templates which feed into the Part A and Part B GAs. For example, in one region a checklist/template was created and is consistently used in the assessment of the Program Management risk factor for Community Infrastructure. Their checklist/template stipulates expectations for each risk consideration, such as the listing of policies required by the Capital Program for assessment against the risk consideration Service/Project Plans and Policies. The results of these checklists are incorporated into the overall GA results.
Because the development and implementation of program-specific checklists/templates is generally performed at the regional-level, the audit noted inconsistencies in the scoring against the risk considerations and the level of justification provided across the regions for the same programs.
The audit team further noted that in addition to the use of the low, medium and high risk ratings, the GA tool allows the use of the low-medium and medium-high ratings (i.e. a five point rating scale), which have not been formally defined through the establishment of benchmarks in the Workbook. Without the establishment of criteria for the low-medium and medium-high risk ratings for each risk consideration, the audit team identified inconsistencies in their use across programs, regional offices and individual FSOs.
2. The Grants and Contributions Management Oversight Committee should ensure the standardization of program-specific checklists/templates used to support the risk analysis across all regions. These checklists/templates should be assessed for their alignment to the risk considerations and benchmarks outlined in the GA Workbook prior to implementation.
- For recommended actions regarding establishing additional benchmarks in the GA Workbook, please refer to recommendation number four (4) bullet B on page 18.
5.3 GA Assessment Results
5.3.1 GA Assessment Results
Through the conduct of detailed testing on a sample of Part A, B and C GAs across the Department, the audit team noted inconsistencies in the level of justification provided for the scoring, as well as discrepancies in the scoring against the risk considerations for Part A and Part B GAs. For Part C GAs, the audit team noted a consistent and adequate level of justification for the scoring against the risk considerations.
In an attempt to re-perform the assessment for a sample of GAs using solely the information contained within the GA report and the benchmarks outlined in the Workbook, the audit team ran into difficulties arriving at the same risk score for certain risk considerations. The audit team was forced to follow-up with either a program representative or FSO in order to gain sufficient justification to support certain scores. In certain circumstances, the audit team was either unable to obtain sufficient justification to support the assessment rating assigned by the assessor, and would have assessed the recipient higher or lower based on the information provided as justification.
In the event the audit team was in disagreement with the rating provided to a recipient against certain risk considerations, the overall rating of the recipient would not have generally moved the recipient to a higher or lower overall risk rating (i.e. would have remained low, medium or high); however, for two (2) recipients (out of 30 Part A GAs reviewed), the audit team would have increased the overall risk rating of the recipient from either a low to a medium overall rating or a medium to a high overall rating.
5.3.2 Special Considerations
In addition to the above noted concerns regarding the level of justification provided for the rating, as well as the challenge with the rating assigned, the audit noted inconsistencies in how assessors were addressing specific circumstances. Specifically, the audit noted that, due to a lack of guidance/standard on how to address specific situations, scoring against certain risk considerations was inconsistent. These circumstances included:
- How to address risk considerations that require information/documentation not currently requested of recipients per the funding agreements. For example, the audit noted that, at times, when a program did not have a formal requirement for the submission of financial reports, policies or operational/strategic plans, they were not requested or reviewed. In other cases, some assessors did request and review documents which were not required by the recipient per the funding agreement but were necessary to assess the recipient against the risk consideration.
- How to address risk considerations when the delivery of a specific program usually tasked to the community has been delegated to a third party. For example, the audit noted, at times, the assessment included a recipient's ability to administer programs even if responsibility had been assigned to a third party, and at other times, the assessment was performed taking into consideration the ability of the third party to support the delivery of the program (rather than the community's ability).
- How to address the weighting of programs when a program is less "significant" than its assigned weighting. The Part A GA tool pre-populates the assigned weighting of programs for the assessment of the Program Management risk factor.The audit noted circumstances where, for example, the Education Program accounted for 30% of the overall risk rating; however, the recipient was only receiving approximately $30K in funding for students who live on reserve but attend provincial schools off-reserve (rather than delivering a full elementary/secondary education program). Any scoring assigned to this category could potentially have an inflated impact due to the weighting, although the funding provided in this category would not warrant it.
Additionally, the audit noted variations in the timeframe considered as the scope of the assessment. At times, the GA report represented solely an assessment a recipients' performance during the last fiscal year; while at other times, poor past performance, even if dating back a few years, affected the score. In one instance, the audit team noted that consideration was given for something the recipient was expected to accomplish in the future.
Lastly, the audit noted an inconsistent understanding across regions and HQ sectors on whether or not the Part A and B GAs needed to be evidenced-based, as per the GA benchmarks and criteria. Current procedural documentation does not provide clarity regarding if the assessor is responsible for not only asking if key documents such as a policies and plans exist, but also to obtain a copy of them to review for completeness, relevancy and appropriateness.
- For recommended actions regarding clarifying the treatment of these special circumstances, please refer to recommendation number four (4) bullet C on page 18.
5.4 Impact on Monitoring and Reporting Activities
Per the PTP, administrative requirements on recipients, which are required to ensure effective control, transparency and accountability, need to be proportionate to the level of risks specific to the program, the materiality of funding and the risk profile of recipients. The DTP further articulates that the level of monitoring of recipients and the reporting required from recipients should also be impacted by the same risks.
Through the conduct of interviews with representatives from a sample of regional offices and HQ sectors, the audit noted that GA scores are not being consistently used to determine the level of administrative requirements applied to recipients. More specifically, due to limitations imposed by certain programs control frameworks, the level of monitoring of, and reporting required by recipients cannot be tailored based on the results of the GA.
This observation is aligned to a finding outlined in the 2013-2014 Audit of the Management Control Framework for Grants and Contributions – Focus on Program Control Frameworks and Recipient Reporting. That audit had noted that regions and programs were generally not implementing risk-based reporting and management regimes to target limited departmental resources on projects and recipients of highest risk. Similarly, the 2012-2013 Audit of the Management Control Framework for Grants and Contributions – Funding Approaches found that the level of recipient risk was not always adequately considered in the establishment and selection of funding approaches and compliance activities within the Department.
Per Directive 410, the GA was developed to provide a standardized process for assessing a recipient for the purpose of identifying potential issues that impact delivery of AANDC-funded programs in order to provide flexibilities in funding agreement management regimes. As such, the GA should be the main tool used in determining the level of reporting and other administrative requirements imposed on recipients; facilitated by program management control frameworks that provide sufficient flexibility to allow regions to select the appropriate reporting and monitoring requirements based on the GA scores and other relevant factors.
3. The Grants and Contributions Management Oversight Committee should improve alignment of the reporting requirements and monitoring performed on recipients, as well as the flexibility in funding approaches available to recipients based on a recipient's level of risk.
5.5 Review and Approval Process
Directive 410 outlines that GA reports are subject to a multi-level review process to ensure the integrity of individual decision documents and consistency across recipients. The Directive outlines that completed GAs are first subject to a peer group review, then review by a Funding Services Manager or equivalent, and finally, the review and approval by the Chair of the Transfer Payment Management Committee or equivalent.
The GA User Guide – Annex B indicates that GA results should be approved by an established regional committee, or delegated to a sub-committee when appropriate. A risk-based approach should be conducted to determine if circumstances exist where GA approval should be escalated to a Transfer Payment Management Committee.
The audit identified a robust and consistent approach for the review and approval of Part C GAs, with all Part C GAs being reviewed and approved by an established regional oversight committee, comprised of individuals with sufficient authority. However, inconsistencies were noted in the review, challenge, and approval of Part A and Part B GAs across the sampled regional offices and HQ sectors. Of the three regional sites visited, the following was noted:
- In one region visited, no formal peer, supervisory or oversight committee review is performed on all GAs (beyond a manager approval in GCIMS). Only specific GAs that meet a pre-determined variance threshold (5% increase/ decrease against the previously completed risk rating) are reviewed and questioned by the Director of Funding Services. No Part A or B GAs are reviewed or approved by the Regional Operations Committee or a sub-committee.
- In another region, all Part A and B GAs are reviewed first by a Field Services Manager and then by the region's Band Audit and Allegations Management (BAAM) unit. GAs are reviewed and approved either at the Audit Review Committee, or through a BAAM desk review (if very low risk), and then approved in GCIMS by a member of BAAM. Medium and high risk GAs, along with a roll-up of all GA results, are discussed at the Regional Operations Committee.
- In a third region visited, the results of all Part A and B GAs were reviewed by the Funding Services Manager and the region's Risk Assessment Committee. For any Part A and B GAs assessed as Medium-High or High risk, the results were also reviewed by the Transfer Payment Management Committee.
For the three sampled HQ programs, none had established a Transfer Payment Management Committee to review the completed Part B GAs. In all programs, the GAs were reviewed and approved at a Manager-level.
Without a consistent approach for the review and approval of Part A and B GAs, there is an increased risk that unsubstantiated or incomplete GAs are not identified and addressed in a timely manner. Additionally, without a comprehensive review process, which includes the review and challenge of GAs by a committee or individual with sufficient oversight over the completion of all regional/sectoral GAs, inconsistencies in the scoring against risk considerations may not be identified.
4. The Grants and Contributions Management Oversight Committee should address the recommendations laid out below to address gaps identified with Directive 410 – General Assessments, and the GA Workbook.
- Directive 410 – General Assessments should be updated to further define:
- The role, responsibilities and accountabilities of Funding Services and program representatives in the development, review and challenge of the programs' components of a Part A GA; and
- The role, responsibilities and accountabilities of regional offices in the review and challenge of the impact of Health Canada's input into the GA scores.
- The GA Workbook should be updated to include benchmarks for the low-medium and medium-high risk ratings for each risk consideration based on relevant criteria, such as what is already included for the low, medium and high risk ratings.
- In order to ensure consistency in the scoring of GAs, the Directive 410 and GA Workbook should be updated to provide additional guidance on how special considerations, such as those identified during the audit, are to be considered/addressed when completing GAs
- Directive 410 should be updated toclarify roles and responsibilities over the review and approval of GAs, specifying the appropriate authority level for approvals for Part A and B GAs and ensure their consistent application across regions/sectors.