Based on a combination of the evidence gathered through the examination of documentation and systems, analysis and interviews, each audit criterion was assessed by the audit team and a conclusion for each audit criterion was determined. Where a significant difference between the audit criterion and the observed practice was found, the risk of the gap was evaluated and used to develop a conclusion and to document recommendations for improvement.
Observations include both management practices considered to be adequate as well as those requiring improvement. Recommendations for corrective actions accompany observations of management areas identified for improvement.
Throughout the audit fieldwork, the audit team observed examples of how controls are properly designed and are being applied effectively by NNC. This has resulted in several positive findings which are listed below:
5.2.1 Cost Containment Strategy
As with any transfer payment contribution program, it is important to ensure program financial stability, sustainability, and predictability while responding to cost pressures. Important elements of a cost containment strategy include establishing and implementing cost containment approaches, regular reporting on progress and cost pressures, and presenting key information to an oversight or governing body to allow for effective monitoring.
Given yearly cost increases under the former Food Mail Program, AANDC was required to include a Cost Containment Strategy as part of the NNC program design. In the NNC program design documentation, the Cost Containment Strategy outlines the adjustment options available to the department to contain costs within the fixed budget and a high level approach on how to use these options. The Cost Containment Strategy includes regular reporting on program costs and pressures and establishes the Oversight Committee as the approval authority for subsidy rate revisions.
NNC subsidy rates are the primary element that can be adjusted on a periodic basis to ensure NNC remains within its fixed budget. In 2011-12, as more up-to-date forecasts and information on shipping costs by community became available, there was an opportunity for NNC to adjust subsidy rates to make them fair among communities and to allow for the NNC budget to be fully utilized.
The audit found that during Oversight Committee meetings, program costs were presented and discussed; however, no decisions were taken by the Oversight Committee to adjust the subsidy rates in order to ensure program costs remained within the approved amounts. The audit also found that typical cost pressures such as inflation and increases in program popularity/demand were not addressed in the design of the program, as the program budget does not have an automatic annual budget escalator. NNC program management reported they requested at the time of development of NNC that the program design account for a growing demand for food; however, no cost of living or growth index was approved for the program's funding.
Without a proper and timely implementation of cost containment strategy measures there is increased risk that NNC will exceed its annual program budget. In addition, cost pressures such as inflation and increased demand for the program may make cost containment measures more difficult to implement without modifications to the cost containment strategy.
2. The Assistant Deputy Minister of the Northern Affairs Organization should continue to support cost containment strategy measures undertaken by the program, and monitored by the Oversight Committee, in order for the program to remain within approved budget levels.
5.2.2 Compliance reviews
For transfer payment contribution programs, it is important to establish an appropriate recipient monitoring framework to ensure that: funding recipients are in compliance with the terms and conditions of their funding agreements; adequate controls are in place; and, key program recipient risks and considerations are addressed. A recipient compliance review plan should be developed to support the allocation of resources to areas that represent the most significant priorities and highest risks. In addition, program management should retain all supporting documentation to ensure proper due diligence for any compliance review work performed.
For NNC, an independent auditor has been contracted to conduct recipient compliance reviews on behalf of NNC. Compliance reviews are conducted on behalf of NNC to ensure that recipients (i.e. retailers and suppliers) who receive subsidies through NNC are compliant with the terms and conditions of their funding agreements and to ensure the necessary controls are in place to appropriately subsidize eligible foods. To ensure NNC program objectives are adequately met, NNC program management selects recipients for compliance reviews based on program management's assessment of risks.
The audit found areas for improvement with respect to the NNC compliance review process:
- Although NNC program management has completed several compliance reviews (7 reports are published on the NNC web site for 2011-12), a compliance review plan has not been documented which adequately demonstrates the risk factors considered by NNC program management in selecting recipients for compliance review;
- While the scope and approach outlined in the compliance review statement of work address key risks and compliance concerns, the published compliance review reports do not adequately or consistently report findings on these areas. For example, the published reports provide limited conclusions on key risk such as: profit margins on subsidized versus unsubsidized products; sale and/or purchasing records and supporting documentation to verify compliance with program requirements; and, reporting and claiming systems and procedures, however there is insufficient information on the findings that lead to the conclusions; and,
- Lastly, although the statement of work used for the compliance reviews specifies that the deliverables to be provided to NNC should include all supporting compliance review working papers, the audit found that NNC program management has not requested this information and was not aware to what extent evidence was gathered or was conducted to support claims made in the published report.
Without a risk-based compliance review plan that includes documented rationale, there is a risk that compliance reviews are not conducted to mitigate the most significant priorities and risks. In addition, without proper monitoring and documented support of compliance review work performed, there is a risk that compliance reviews are not conducted in accordance with the intended approach and that key risks are not adequately addressed or reported.
3. The Assistant Deputy Minister of the Northern Affairs Organization should: document the risks and rationale used in selecting recipients for compliance reviews; create a compliance review plan based on this approach; retain the evidence gathered during the compliance reviews; and, ensure observations made in published reports specifically address the key risks and intended scope and approach.
5.2.3 Claims Processor
Verifying recipient claims includes verifying the claim for accuracy and validity (i.e. appropriate volumes are being claimed), only eligible items are being claimed, and payments are correct. Supporting documentation and analysis used to reconcile payments should also be retained with suitable quality assurance procedures and evidence of review and approval by delegated authorities.
NNC has outsourced, through a competitive procurement process, the administration and verification of subsidy claims to a claims processor (Saskatchewan Indian institute of Technologies, in partnership with Crawford & Company). Recipients submit invoices and waybills detailing shipment information such as weight by category of eligible items, destination community, and recipient (store, individual, etc.) to the claims processor. The claims processor is responsible for verifying the claim for validity and accuracy and compiling shipment data for analysis by NNC program staff. NNC program staff use this information to reconcile claims with recipient forecasts and process payments.
NNC has instructed the claims processor to perform their review of recipient claims by selecting a sample of 50 lines from the supporting documentation attached to the claim. The claim review process, therefore, entails a review of 50 lines per recipient per month. The audit found that the sample selection performed by the claims processor is done randomly (with occasional specific direction from NNC on priority areas). The audit found that the random sample selection is generally not based on a documented risk assessment which considers the risk level of both the recipient and the food item. In addition, as some claims have less than 50 lines, these claims are fully reviewed.
The audit also found that NNC has not adequately monitored all deliverable and service requirements clauses outlined in the claims processor service contract (e.g. service level agreement reports).
In instances where third party service providers are used to perform program operational activities, there are typically two options to consider in order to receive assurance over controls exercised by the third party: include a "right to audit" clause in the service contract; or, in the contract negotiation phase, require the service provider to provide a service auditor report to provide assurance that the controls they are exercising on the program's behalf are operating effectively. In the case of NNC (where the service contracts is already established), a "right to audit" clause is included in the claims processor contract; however, NNC has not elected to conduct an audit of the claims processor, nor has NNC requested information on the claims processor quality assurance approach. NNC does have claim processing process maps with key control points in the claim process, and program staff indicated that they have conducted an onsite walkthrough of SIIT and Crawford procedures. These practices can provide the program with assurance over the adequacy of the design of the Claims Processors' control framework, However, they do not provide assurance over the effectiveness of their controls (i.e. are they functioning as designed). This is achieved through the review of the reporting and results from a Quality Assurance review process/framework and/or results of SIIT/Crawford's internal audits of the process. At the time of the audit this information has not been provided to the program.
Once claims have been reviewed by the claims processor, NNC staff are responsible for performing a review and analysis of the recommended amount and preparing payments based on the amount recommended by the claims processor. The audit found that there is insufficient procedural documentation to describe the analysis and work performed by NNC in calculating the recipient payments (i.e. advance payments, corrections to support actual payment) and one instance was noted (out of 15) where evidence was missing to adequately demonstrate that approval of a claim was made by a delegated authority.
Without adequate oversight of the claims processor there is a risk that claims processor processes are not properly implemented. When there is insufficient assurance or attestation from the claims processor that appropriate controls or practices are in place, and functioning as designed, to verify the accuracy and validity of claims, there is increased risk to NNC, who is ultimately accountable, that claims are not recorded or processed accurately.
4. The Assistant Deputy Minister of the Northern Affairs Organization should request information on the claims processor quality assurance approach to determine whether an audit of the current claims processor is necessary; ensure payment review and approval procedures are adequately documented; develop a risk-based sampling approach for recipient claims; and, investigate appropriate contract requirements for future claims processor contracts (i.e. service monitoring and service auditor report).
5.2.4 Eligibility Assessment
It is essential that management have practices in place to assess program eligibility in a consistent and transparent manner. Eligibility criteria should be established to ensure that eligibility assessments reflect program requirements. As applicable, program management should be able to demonstrate that applicants submitted a complete application, met all eligibility criteria, and are able to fulfill all program requirements.
For NNC, three eligibility assessments are performed: communities; recipients (retailers and suppliers); and, food items.
- Communities: to be eligible for NNC a community must lack year-round surface transportation (i.e. no permanent road, rail or marine access), and have benefitted from the former Food Mail Program;
- Recipients: three types of businesses can apply – northern retailers, southern suppliers and northern country food processors/distributors; and,
- Food Items - eligible food items include a variety of perishable healthy foods, such as fresh fruit, frozen vegetables, bread, meat, milk and eggs. Eligible foods can be fresh, frozen, refrigerated, or have a shelf life of less than one year, and must be shipped by air. In addition, NNC provides subsidies for country foods (e.g. Arctic char, caribou, and musk-ox) harvested in eligible northern communities.
With respect to community eligibility, the audit found that only communities using the former Food Mail Program are considered eligible for NNC, and this eligibility requirement is articulated on the NNC public website. As per NNC program design documents, communities that did not use the Food Mail Program are not considered eligible for the subsidy under NNC. In 2012/13, NNC committed to define and communicate a set of community eligibility criteria; however, to date, this has not occurred. In addition, the level of eligibility (i.e. full or nominal subsidy) for communities is supposed to be reviewed annually, based on food basket prices and shipment volumes, but this review has not yet occurred. NNC program management noted that amending the list of eligible communities requires a change in program policy and approval by central agencies.
The audit also noted that the list of eligible food items was reviewed in November 2011. Although the NNC web site states that the food item list will be reviewed annually, always taking into account feedback from northerners, NNC reported that the food eligibility list remained unchanged for 2012/2013. NNC program management noted that amending the list of eligible food items also requires a change in program policy.
Without proper and thorough assessment of recipient eligibility, there is a risk that recipients are inappropriately included or excluded from the program. In addition, without a practice or process in place to ensure periodic review and policy discussion on eligible food items and communities, there is increased risk that program objectives will not be fully realized.
5. The Assistant Deputy Minister of the Northern Affairs Organization should establish a process for periodic review and policy discussion regarding food items and community eligibility.
5.3 Citizen-focused Services
Communication mechanisms should support the sharing of timely, relevant and reliable information to consumers, recipients and other stakeholders. In addition, feedback from users and other stakeholders should be used to influence strategic and operational planning.
Given that the transition from the Food Mail Program to NNC received significant media attention and NNC continues to receive media attention, the audit found that NNC has benefited from having a robust communication strategy to address stakeholder concerns in a timely manner. The NNC Communication Strategy was continuously updated based on feedback received. As part of the Communication Strategy, NNC established mechanisms to receive comments and questions from northern community members and other stakeholders. A key objective of the Communication Strategy was to educate northerners about the program, and how it works to ensure greater awareness, understanding, and potential participation. In addition, the audit noted that the Communication Branch supporting NNC has taken the steps to obtain feedback from northern retailers as input to program decisions.
Lastly, the audit noted that funds are set aside on an annual basis to promote and increase awareness of NNC through the use of signage (with translation) and updates on the NNC website.
No recommendations were identified in this area.
5.4 Results and Performance
5.4.1 Performance Measurement Strategy
It is important that performance measurement strategies be in place and linked to program objectives. Performance measures should be reasonable and demonstrate outcomes as well as accountability and transparency. Management should also monitor performance against planned results and adjust course as needed.
NNC has developed a Performance Measurement Strategy which is intended to help ensure that performance indicators and data collection methods are well defined and in place to support program monitoring. Key activities in the Performance Measurement Strategy include providing, monitoring, and verifying subsidies for eligible foods and promoting program awareness, outreach and engagement. Immediate NNC outcomes include: access to subsidized nutritious food in eligible communities; and, residents in eligible communities are informed about the program and subsidy levels. The collection of shipping and pricing data is a fundamental aspect of the NNC. Key indicators listed in the strategy include value of subsidy, weight of eligible food shipped, food basket prices, level of awareness in program and subsidy levels, estimated weight of eligible food purchased per capita (to show increase in healthy consumption).
NNC receives price and shipment data from all retailers and suppliers who are registered with the program. Several different reports are published on the NNC web site. These reports provide information on the types and amounts of products that receive a subsidy under the program. Data is reported by fiscal year, by quarter, by product category, by province or territory, or by community. The Revised Northern Food Basket (RNFB) is a performance measurement tool that AANDC currently uses to monitor trends in the cost of healthy eating in isolated northern communities eligible for the NNC program. The RNFB is a representative combination of foods that meets most nutrient requirements and food serving recommendations in Canada's Food Guide for a family of four for one week. The NNC web site also reports the cost of the RNFB and reports the cost of the RNFB in March 2012 had declined by approximately 8%, on average, as compared to when the former Food Mail Program was in place.
The audit found that while NNC has collected a significant amount of data on subsidized food, NNC has outgrown the tools used to gather and analyze the data. As a result, the audit found that NNC is not able to adequately analyze the data and identify any possible trends (e.g. retailer pricing information cannot be adequately analyzed to identify possible price anomalies). In addition, access to NNC database files is not adequately restricted, increasing the risk of unauthorized access and tampering. NNC is currently working with the Department's Information Management Branch to be able to use the Departmental data warehouse system. This will greatly increase NNC's capacity to analyze data and identify trends. The tool is expected to be fully implemented by March 2014.
The NNC Performance Measurement Strategy includes the completion of an annual survey to determine the level of awareness about NNC and subsidy levels. The audit found that NNC had not yet developed the awareness survey to obtain feedback from northern community members. Although delayed, a retail study was underway to better understand northern retailer costs.
In order to build upon current data collection and reporting practices, there is a risk that without proper tools/systems and access controls, NNC may not effectively and efficiently collect data and conduct data and trend analysis or important data may be mistakenly corrupted, altered, or destroyed. There is a risk that without consumer awareness and retail surveys to obtain feedback, NNC will not be able to establish baseline awareness data or meet program obligations outlined in the NNC Performance Measurement Strategy.
6. The Assistant Deputy Minister of the Northern Affairs Organization should implement tools/systems with access controls to effectively collect data, conduct trend analysis, and monitor changes in the price of food basket items, and complete awareness and retail surveys to obtain feedback on program progress and retailer costs.