System Under Development Audit of the Integrated Financial Management System (SAP and GCIMS)

Notice

This website will change as a result of the dissolution of Indigenous and Northern Affairs Canada, the creation of Indigenous Services Canada and the eventual creation of Crown-Indigenous Relations and Northern Affairs Canada. During this transformation, you may also wish to consult the updated Indigenous and Northern Affairs home page.

June 2014
Project #: 13-50

PDF Version (493 Kb, 42 Pages)

 

Table of contents

Acronyms

AANDC

Aboriginal Affairs and Northern Development Canada

AASB

Audit and Assurance Services Branch

AC

Audit Committee

ADM

Assistant Deputy Minister

BDS

Business Decision Support

CAMM

Corporate Accounting and Materiel Management

CanNor

Canadian Northern Economic Development Agency

CFO

Chief Financial Officer

CIDM

Comprehensive Integrated Document Management system

DG

Director General

DM

Deputy Minister

DRAP

Deficit Reduction Action Plan

FNITP

First Nations and Inuit Transfer Payment System

GCIMS

Grants and Contributions Information Management System

Gs&Cs

Grants and Contributions

HC

Health Canada

HQ

Headquarters

ICFR

Internal Controls over Financial Reporting

IFMS

Integrated Financial Management System

IT

Information Technology

OASIS

AANDC's Oracle-based legacy financial management application system

MOU

Memorandum of Understanding

PPA

Predictive Project Analytics

PHAC

Public Health Agency of Canada

PPMF

Project Portfolio Management Framework

RIMS

Resource Information Management System

SAP

Systems, Applications and Products in Data Processing

SLA

Service Level Agreement

TB

Treasury Board

TFMS Trust Fund Management System
 

 

Executive Summary

Background

Aboriginal Affairs and Northern Development Canada (AANDC) has the primary, but not exclusive, responsibility for meeting the federal government's constitutional, treaty, political, and legal responsibilities to First Nations, Inuit, Métis and Northerners. Under this mandate, AANDC is responsible for the planning, design, implementation, and assessment of policies as well as the delivery of a variety of programs and services to First Nations, Inuit, and Northern peoples and communities.

For the 2013-14 fiscal year, AANDC's actual expenditures to support the delivery of its mandate were $7.9 billion, which includes approximately $6.5 billion in transfer payments.

In an effort to gain back-end operating efficiencies, and to further improve the management of funding relationships, AANDC announced in its 2013-14 Report on Plans and Priorities that by April 2014, it would converge with Health Canada (HC) on the same financial management system and the same grants and contributions management system. Under this horizontal systems initiative, AANDC would adopt HC's financial management system (SAP), and Health Canada and the Public Health Agency of Canada (PHAC) would adopt AANDC's Grants and Contributions Information Management System (GCIMS, formerly FNITP).

The intended outcome of this partnership of reciprocal systems hosting arrangements is to create a comprehensive, common solution that manages grants and contributions in excess of $9 billion, leverages each department's corporate processes to match Government of Canada best practices, and is supported by a common, shared SAP financial management system.

To achieve these outcomes, two distinct, but interrelated, projects were undertaken concurrently by AANDC and HC/PHAC. Collectively, the following two projects are referred to as the SAP-GCIMS project:

  • The first project to migrate AANDC's financial and materiel management processes, including GCIMS, from Oracle Financials to SAP; and,
  • The second project to transform the processing of Gs&Cs at HC and PHAC by migrating to the new GCIMS solution, once AANDC made GCIMS technically compatible with the SAP financial system.

The scope of business processes and functions at AANDC altered by the implementation projects included:

  • financial management (General Ledger, Accounts Payable, Accounts Receivable, Special Ledger and commitments);
  • procurement and materiel management ("Procure-to-Pay");
  • asset tracking and accounting;
  • budgeting and salary forecasting, for all levels of the organization;
  • integration with central Government of Canada systems via system interfaces;
  • functional support to Canadian Northern Economic Development Agency (CanNor) in their migration from AANDC's OASIS financial management solution to HC's hosted SAP solution; and,
  • modification of linked systems (GCIMS, Trust Fund Management System (TFMS), Resource Information Management System (RIMS), and the Business Decision Support (BDS) reporting system) to be compatible with SAP coding.

Audit Objective and Scope

The overall objective of the audit was to assess the adequacy and effectiveness of the project management framework in place to ensure that AANDC's expected results of the systems migration projects were delivered within budget and on time, with no loss of data integrity or system functionality.

Objectives of the audit also included:

  • to provide assurance to senior management at AANDC, and to AANDC's Audit Committee (AC), that project risks related to the system implementations were mitigated to the extent possible, and that issues affecting AANDC's readiness for the "go-live" of the two system implementation projects were being addressed by the project team; and,
  • to report on leading practices and lessons learned with respect to collaboration with HC/PHAC on these system implementation projects, including the methods used to define, enable, and govern the partnership established between AANDC and HC/PHAC to oversee the ongoing operations of shared information systems.

The scope of the audit included all control elements that help ensure effective project control, risk management, stewardship and accountability for AANDC in relation to the SAP and GCIMS implementation projects. The scope of the audit also included project governance and delivery teams at both AANDC and HC to the extent that the audit was to report on leading practices and lessons learned with respect to the collaboration between the two departments, and on methods used to define, enable, and govern the ongoing operations of the partnership.

The SAP-GCIMS project can be considered to have four separate delivery teams; two technical system preparation teams (one for each of SAP and GCIMS), and two business transformation teams (one for each of AANDC and HC/PHAC). The activities of three of these four teams were largely included in the scope of the audit, as illustrated in Figure 1 below. Activities of the HC/PHAC team responsible for managing changes to grants and contributions business processes introduced by the implementation of GCIMS were not in scope, since this work was not performed by the AANDC project team, nor was the implementation of either GCIMS or SAP at AANDC dependent on the completion of these activities.

Delivery Team Overview

Figure 1: Delivery Team Overview*

* Note: The graphic in this figure is adapted from a SAP-GCIMS project briefing document.

Description of Figure 1

Figure 1, titled "Delivery Team Overview", identifies four teams responsible for the delivery (in terms of system preparation and business transformation efforts) of SAP and GCIMS.

  • With respect to SAP system preparation (which is in the scope of the audit), Health Canada's SAP Delivery Team delivers it and AANDC pays for it.
    • AANDC must ensure that the system (SAP) being delivered and supported by Health Canada is stable and available for AANDC to use.
  • With respect to SAP business transformation efforts (which are in the scope of the audit), AANDC's SAP Delivery Team undertakes these and AANDC pays for them.
    • AANDC must ensure that it has successfully migrated to SAP from a business readiness, internal control, and a technical interface perspective.
  • With respect to GCIMS system preparation (which is in the scope of the audit), AANDC's GCIMS Delivery Team delivers GCIMS and Health Canada pays for it.
    • AANDC must ensure that hosting GCIMS for use by Health Canada does not have a negative impact on AANDC operations.
  • With respect to GCIMS business transformation efforts (which is out of scope of the audit), Health Canada's GCIMS Delivery Team is responsible. HC/PHAC undertakes these efforts and pays for it.
    • Health Canada's business processes and internal controls related to the use of GCIMS are out of scope.
 

The scope of the audit included the period from August 2013 to April 2014. Audit planning was completed in November 2013; review and testing of project management processes and controls covered the period from November 2013 to April 2014. This period was chosen to align with planned implementation dates scheduled by the project team. In particular, the migration to SAP for processing financial transactions at AANDC was scheduled for April 1, 2014.  Audit fieldwork was completed on April 3, 2014. Timing of the audit relative to key project milestones is shown in Figure 2 below.

Audit Timeline

Figure 2: Audit Timeline

Description of Figure 2

Figure 2 illustrates the SAP-GCIMS project timeline and the audit timeline for the months of September 2013 to April 2014. In chronological order, dates of key project milestones and audit milestones are as follows:

  • Beginning of September 2013: The Planning Phase of the audit is underway.
  • October 1, 2013: GCIMS Release 7.0/7.1 – Multi-Department Platform
  • Late in November 2013: The Audit Planning Report is presented to the Audit Committee. The Planning Phase of the audit ends and the Conduct Phase begins.
  • December 18, 2013: SAP Wave 1 – Entry of fiscal year 2014-2015 budgets
  • January 6, 2014: GCIMS Release 7.2 – Entry of fiscal year 2014 commitments
  • Early in February 2014: Audit Interim Report #1 is presented to the Audit Committee.
  • March 1, 2014: GCIMS Release 7.3 – Processing of payments
  • April 1, 2014: SAP Wave 2 – AANDC financial processing
  • Late in April 2014: Audit Interim Report #2 is presented to the Audit Committee.
 

Statement of Conformance

This audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

Observed Strengths

Throughout the audit fieldwork, the audit team observed examples of how controls were properly designed and applied effectively by AANDC, resulting in several positive findings listed below:

  • Senior managers representing business, technical and project stakeholders at AANDC met regularly throughout the project to communicate project status, to discuss project issues and risks as they arose, and to coordinate interdepartmental activities involving HC/PHAC, PWGSC, and Shared Services Canada;
  • Project team members successfully conducted a technical re-write of the GCIMS application, which is critical to the day-to-day business of the Department, without significant error or loss of service, indicating that strong change and release management procedures are in place at AANDC;
  • Draft documents to support the provision of GCIMS services to HC/PHAC, and potential future Government of Canada partner departments, including an Operational Memorandum of Understanding and a Service Level Agreement, are aligned with Treasury Board guidelines on service agreements and service standards;
  • Initial training material received from HC for SAP did not address AANDC priorities;  the SAP-GCIMS project team at AANDC applied significant effort to customizing training materials prior to delivery of courses to users in advance of the April 2014 migration to SAP; and,
  • Significant project milestones, including the cutover to SAP, implementation of Releases 7.2 and 7.3 of GCIMS, and the processing of initial transfer payments to AANDC recipients for the fiscal year 2014-15, were completed on time, with minimal system errors or disruption reported during the audit period.

Conclusion

Generally, the project management framework at AANDC was found to be adequate, and effectively supported the management of the SAP-GCIMS project to ensure that AANDC's expected results of the system migration projects were delivered on time, with no loss of data integrity or system functionality; however, an opportunity for improvement was noted related to formal approvals of deliverables at milestone gates. 

The audit was not able to conclude on whether results were delivered within budget. As of the last AANDC/HC DG Partnership Review Committee held during the audit's conduct phase (March 31, 2014), the project status "dashboard" indicated actual expenses tracking to budget, but these results were dated November, 2013. The audit did not receive regular reports of actual versus planned expenditures in other formats.

  

With respect to project risks related to the system implementations, and issues affecting AANDC's readiness for the "go-live", findings and recommendations for addressing risks and issues were communicated by the audit to the AANDC SAP-GCIMS project team during the course of the audit. Opportunities for improvement remain regarding business requirements documentation.

  

Observations on leading practices and lessons learned with respect to collaboration on the system implementation projects with Health Canada were noted during the audit; these include opportunities for improvement with respect to project management and governance, management of project documentation and deliverables, preparation of interdepartmental agreements, and organization change management.

Finally, project management methods and controls used to define, enable, and govern the partnership established between AANDC and HC to oversee the ongoing operations of shared information systems were generally effective; however, opportunities for improvement were noted with respect to the content of master agreements and service level agreements.

Recommendations

During a system under development audit, the audit team reports findings and makes recommendations to the project team on an on-going basis so that management action can be taken immediately to address identified weaknesses as they arise. A traditional internal audit approach, where recommendations are delivered following the completion of all field work, could result in a project team receiving auditors' advice after an identified weakness had affected the project's success. Therefore, audit findings were presented to AANDC management three times during the conduct of the audit:

  • at the conclusion of the audit planning phase (December 2013);
  • following the initial release of SAP to AANDC, and the related Release 7.2 of GCIMS  (January 2014); and,
  • at the start of AANDC's fiscal year 2014-15 (April 2014), when processing of financial transactions migrated to SAP.

Audit lines of enquiry included controls and procedures related to the management of the SAP-GCIMS project, to risks and issues associated with the implementation of the software systems, and to risks and issues associated with defining, enabling, and governing the partnership established between AANDC and Health Canada to oversee the ongoing operations of shared information systems.

In all, fourteen findings were reported to AANDC management during the course of the audit. Findings and recommendations reflect work undertaken until the end of the audit's Conduct Phase (April 3, 2014). Management action plans provided to the audit team on January 20, 2014 and March 28, 2014 were reviewed by the audit team with the Project Sponsor and other members of the AANDC project team and, where appropriate, with the Chief Information Officer. The third and most recent management action plan was received following the end of the Conduct Phase on May 28, 2014.

Briefings on findings were provided to the Departmental Audit Committee (DAC) at their meetings in February and April 2014. 

It should be noted that a number of the recommendations provided to AANDC management, in particular those that addressed risks related to the system implementations and issues affecting AANDC's readiness for the "go-live" of the system implementation projects, are considered to be "point in time" recommendations, relevant to specific past phases of the project. Therefore, during the reporting phase AASB conducted an analysis of all fourteen findings, recommendations and associated management responses and determined that in several instances management has taken significant action during the course of the audit to mitigate risks. As a result, the number of recommendations that require further monitoring or follow-up has been reduced.  Section 5 of the report includes the six recommendations listed below, which should be monitored until management has implemented their proposed corrective action:

  1. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should review gate approvals for the SAP-GCIMS project and assess whether requirements specified in the Project Portfolio Management Framework have been met.
  2. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should review documented business requirements and confirm that requirements have been met by the delivered systems.

    The Chief Information Officer should consider amending the Project Portfolio Management Framework to include tracking of business requirements among its mandatory deliverables for system implementation projects.
  3. The Director General, Corporate Accounting and Materiel Management, as the SAP-GCIMS Project Sponsor, should work with Health Canada to ensure that service agreements specific to Health Canada's role as an SAP service provider are drafted and finalized.

    Furthermore, a steering committee for the ongoing partnership between the two Departments should include in its mandate the monitoring of the performance of HC as service provider to ensure that services levels are being met.
  4. The Director General, Corporate Accounting and Materiel Management should develop an operating model for a support organization for GCIMS as a hosted system for other Government of Canada departments. Service Level Agreements (SLAs) with Health Canada for the provision of GCIMS services should align with Treasury Board Secretariat guidelines on service agreements and service standards.
  5. The Director General, Corporate Accounting and Materiel Management, with assistance from the Chief Information Officer, should prepare a plan for the GCIMS technical support organization that includes technical and business analyst resources, a client service function, and other functions required to support the ongoing development, update, and deployment of the GCIMS application.
  6. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should ensure that activities described in AANDC's Business Transformation Strategy – particularly those related to the cutover to the new SAP system – are completed by the SAP project team, and results of these activities communicated to AANDC management.

Lessons Learned

An additional six observations that address the audit objective to report on leading practices and "lessons learned" with respect to collaboration with HC/PHAC, and with other Government of Canada departments on future IT implementation projects, were noted during the audit. These six observations were not included in previous reports. 

Additional lessons learned for interdepartmental system implementation projects include:

  1. A single project or program management office, reporting directly to the project steering committee, and responsible for all project streams in participating departments, should be established and assigned responsibility for supporting the steering committee in the fulfillment of its governance responsibilities through preparation of the committee's agenda, and circulation of documents for decision and/or approval.
  2. Interdepartmental projects should adopt a single project management framework for all streams of the project to provide a consistent foundation for all project deliverables, and provide a structure for a common, definitive document repository for project deliverables.
  3. Project managers should establish a common centralized repository for project documentation and deliverables that is in accordance with departmental information management requirements. The central document repository should be accessible to all project team members, in all participating departments.
  4. Business requirements should be captured early in a project, and tracked throughout to ensure that the new system adequately addresses the requirements. User acceptance testing should be performed directly by the business stakeholders to validate that business requirements have been met.
  5. Project managers should initiate preparation of agreements between departments (e.g. Memoranda of Understanding, Master Service Agreement, Service Level Agreements), including cost sharing arrangements, as early as possible in the project, to promote clear understanding of delivery responsibilities, to assist in the assessment of changes to scope as changes arise, and to assist in the planning of changes to support processes. Agreements should be consistent with Treasury Board guidelines on service agreements and service standards.
  6. Project managers should ensure that organization change management activities are included in project plans for all business transformation projects. For interdepartmental projects, organizational change management teams should be coordinated, so that change plans of departments receiving services have their plans informed by the department providing the new service. Finally, costs associated with the change effort should be considered in the overall Master Agreement between the participating departments.

Management Response

Management is in agreement with the findings, has accepted the recommendations included in the report, and has developed a management action plan to address them. The management action plan has been integrated into this report.

 

 

1. Introduction and Context

1.1 Background

Aboriginal Affairs and Northern Development Canada (AANDC) has the primary, but not exclusive, responsibility for meeting the federal government's constitutional, treaty, political, and legal responsibilities to First Nations, Inuit, Métis and Northerners. Under this mandate, AANDC is responsible for the planning, design, implementation, and assessment of policies as well as the delivery of a variety of programs and services to First Nations, Inuit, and Northern peoples and communities.

For the 2013-14 fiscal year, AANDC's actual expenditures to support the delivery of its mandate were $7.9 billion, which includes approximately $6.5 billion in transfer payments. To manage these resources, AANDC relied on the Oracle Financial System (OASIS).

OASIS is an integrated financial management system that is supported by the following five integrated sub-systems:

  • First Nations and Inuit Transfer Payment System (FNITP);Footnote 1
  • Guaranteed Loan Management Module (now incorporated into FNITP);
  • Resource Information Management System (RIMS);
  • OASIS Salary Management System/Regional Pay System (OSMS/RPS); and,
  • Trust Fund Management System (TFMS).

The First Nations and Inuit Transfer Payment System is a web-enabled grants and contributions (Gs&Cs) management system that automates the Department's transfer payment business processes, manages funding agreement information, and provides on-line access for First Nations and other funding recipients.

In an effort to gain back-end operating efficiencies, and to further improve the management of funding relationships, AANDC announced in its 2013-14 Report on Plans and Priorities that by April 2014, it would converge with Health Canada (HC) on the same financial management system and the same grants and contributions management system. Under this horizontal systems initiative, AANDC would adopt HC's financial management system (SAP) and Health Canada and the Public Health Agency of Canada (PHAC) would adopt AANDC's Grants and Contributions Information Management System (GCIMS, formerly FNITP).

Over the past few years, AANDC and HC, who together account for the vast majority of federal government funding to First Nations, have worked to align their approaches to Gs&Cs agreements to eliminate many minor differences that create additional work for recipients. The departments will take the same approach to risk assessment and funding agreements and, beginning in 2014-15, will share the same IT systems for both financial management (SAP), and for the management of grants and contributions (GCIMS).

The intended outcome of this partnership of reciprocal systems hosting arrangements is to create a comprehensive, common solution that manages grants and contributions in excess of $9 billion, leverages each department's corporate processes to match Government of Canada best practices, and is supported by a common, shared SAP financial management system.

To achieve these outcomes, two distinct, but interrelated, concurrent projects were undertaken by AANDC and HC/PHAC:

  • The first project to migrate AANDC's financial and materiel management processes, including GCIMS, from Oracle Financials to SAP; and,
  • The second project to transform the processing of Gs&Cs at HC/PHAC by migrating to the new GCIMS solution, once AANDC made GCIMS technically compatible with the SAP financial system.

The concurrent projects are referred to as the SAP-GCIMS project.

1.2 Scope of Financial System Renewal

From AANDC's perspective, financial system renewal brought about by the SAP-GCIMS project includes the following dimensions:

  • The use of Government-wide licenses for SAP: AANDC has replaced its legacy finance and materiel management systems with a solution based on the SAP ECC 6.0 software application, using the Government of Canada license;
  • Adoption of common Government of Canada business processes: Implementing an SAP solution enables AANDC to align its financial and materiel management processes with common SAP-supported processes that are vetted by the IFMS "Core" Program Office, and the Government of Canada cluster of SAP user departments and agencies;
  • Share in an established solution: Choosing a hosted SAP solution reduces substantially project risks and costs associated with AANDC implementing the system on its own, and allows AANDC to leverage established SAP system design, software configuration, and business processes; and,
  • Compatibility of linked systems with SAP: Other financial systems at AANDC, including GCIMS, are modified to be technically compatible with the SAP coding block, chart of accounts, and master data.

The scope of business processes and functions at AANDC altered by the implementation projects included:

  • financial management (General Ledger, Accounts Payable, Accounts Receivable, Special Ledger and commitments);
  • procurement and materiel management ("Procure-to-Pay");
  • asset tracking and accounting;
  • budgeting and salary forecasting, for all levels of the organization;
  • integration with central GC systems via system interfaces;
  • functional support to Canadian Northern Economic Development Agency (CanNor) in their migration from AANDC's OASIS financial management solution to HC's hosted SAP solution; and,
  • modification of linked systems (GCIMS, TFMS, RIMS, and the BDS reporting system) to be compatible with SAP coding.
 

 

2. Audit Objective and Scope

A system under development audit of the concurrent implementations of the Government of Canada SAP footprint, and AANDC's GCIMS was included in AANDC's 2013-14 to 2015-16 Risk-Based Audit Plan.  The audit was identified by the Audit and Assurance Services Branch (AASB) at AANDC as a priority engagement, since large-scale system implementation projects are typically subject to a high degree of risk in terms of loss or corruption of data, and that early audit activity can support effective project management and promote the achievement of system implementation objectives.

2.1 Audit Objective

The overall objective of the audit was to assess the adequacy and effectiveness of the project management framework in place to ensure that AANDC's expected results of the systems migration projects were delivered within budget and on time, with no loss of data integrity or system functionality.

Objectives of the audit also included:

  • to provide assurance to senior management at AANDC, and to AANDC's Audit Committee (AC), that project risks related to the system implementations were mitigated to the extent possible, and that issues affecting AANDC's readiness for the "go-live" of the two system implementation projects were being addressed by the project team; and,
  • to report on leading practices and lessons learned with respect to collaboration with HC/PHAC on these IT implementation projects, including the methods used to define, enable, and govern the partnership established between AANDC and HC/PHAC to oversee the ongoing operations of shared information systems.

2.2 Audit Scope

The scope of the audit included all control elements that help ensure effective project control, risk management, stewardship and accountability for AANDC in relation to the SAP and GCIMS implementation projects. The scope of the audit also included project governance and delivery teams at both AANDC and HC to the extent that the audit was asked to report on leading practices and lessons learned with respect to the collaboration between the two departments, and on methods used to define, enable, and govern the ongoing operations of the partnership.

The SAP-GCIMS project can be considered to have four separate delivery teams; two technical system preparation teams (one for each of SAP and GCIMS), and two business transformation teams (one for each of AANDC and HC/PHAC). The activities of three of these four teams were largely included in the scope of the audit, as illustrated in Figure 1 below. Activities of the HC/PHAC team responsible for managing changes to grants and contributions business processes introduced by the implementation of GCIMS were not in scope, since this work was not performed by the AANDC project team, nor was the implementation of either GCIMS or SAP at AANDC dependent on the completion of these activities.

Delivery Team Overview

Figure 1: Delivery Team Overview*

* Note: The graphic in this figure is adapted from a SAP-GCIMS project briefing document.

Description of Figure 1

Figure 1, titled "Delivery Team Overview", identifies four teams responsible for the delivery (in terms of system preparation and business transformation efforts) of SAP and GCIMS.

  • With respect to SAP system preparation (which is in the scope of the audit), Health Canada's SAP Delivery Team delivers it and AANDC pays for it.
    • AANDC must ensure that the system (SAP) being delivered and supported by Health Canada is stable and available for AANDC to use.
  • With respect to SAP business transformation efforts (which are in the scope of the audit), AANDC's SAP Delivery Team undertakes these and AANDC pays for them.
    • AANDC must ensure that it has successfully migrated to SAP from a business readiness, internal control, and a technical interface perspective.
  • With respect to GCIMS system preparation (which is in the scope of the audit), AANDC's GCIMS Delivery Team delivers GCIMS and Health Canada pays for it.
    • AANDC must ensure that hosting GCIMS for use by Health Canada does not have a negative impact on AANDC operations.
  • With respect to GCIMS business transformation efforts (which is out of scope of the audit), Health Canada's GCIMS Delivery Team is responsible. HC/PHAC undertakes these efforts and pays for it.
    • Health Canada's business processes and internal controls related to the use of GCIMS are out of scope.
 

The scope of the audit included the period August 2013 to April 2014. Audit planning was completed in November 2013; review and testing of project management processes and controls covered the period from November 2013 to April 2014. This period was chosen to align with planned implementation dates scheduled by the project team. In particular, the migration to SAP for processing financial transactions at AANDC was scheduled for April 1, 2014.  Audit fieldwork was completed on April 3, 2014. Timing of the audit relative to key project milestones is shown in Figure 2 below.

Audit Timeline

Figure 2: Audit Timeline

Description of Figure 2

Figure 2 illustrates the SAP-GCIMS project timeline and the audit timeline for the months of September 2013 to April 2014. In chronological order, dates of key project milestones and audit milestones are as follows:

  • Beginning of September 2013: The Planning Phase of the audit is underway.
  • October 1, 2013: GCIMS Release 7.0/7.1 – Multi-Department Platform
  • Late in November 2013: The Audit Planning Report is presented to the Audit Committee. The Planning Phase of the audit ends and the Conduct Phase begins.
  • December 18, 2013: SAP Wave 1 – Entry of fiscal year 2014-2015 budgets
  • January 6, 2014: GCIMS Release 7.2 – Entry of fiscal year 2014 commitments
  • Early in February 2014: Audit Interim Report #1 is presented to the Audit Committee.
  • March 1, 2014: GCIMS Release 7.3 – Processing of payments
  • April 1, 2014: SAP Wave 2 – AANDC financial processing
  • Late in April 2014: Audit Interim Report #2 is presented to the Audit Committee.
 

 

3. Approach and Methodology

3.1 Audit Approach

The audit was conducted in accordance with the requirements of the Treasury Board (TB) Policy on Internal Audit and followed the Internal Auditing Standards for the Government of Canada. The audit examined sufficient, relevant evidence and obtained sufficient information to provide a reasonable level of assurance in support of the audit conclusion.

The approach taken in this audit was also aligned with terminology used and advice provided by TB Chief Information Officer Branch (CIOB) in their publication The Independent Reviewer's Handbook, and with AANDC's Directive on IM/IT Project Portfolio Management. The directive describes AANDC's IM/IT Project Portfolio Management Framework (PPMF), which includes a 'gating' approach used to manage approval of projects at key milestones, and to monitor progress of IT-enabled projects against planned milestones.

The audit approach included, but was not limited to:

  • Interviews with key AANDC management personnel, project team members, and selected staff;
  • Regular attendance at steering committee meetings at all levels: the ADM Steering Committee, the DG Partnership Review Committee (both of these committees are joint committees with HC/PHAC), and the AANDC DG Steering Committee;
  • Review of relevant documentation such as:
    • Project Charter, Project Plan, and Business Case;
    • Terms of reference, meeting minutes, and records of decisions for oversight committees;
    • Supporting project planning documents, including schedules;
    • Major project deliverables and supporting documents, such as those for testing of software, training, communication of changes related to project activities, migration of data, testing of system interfaces and internal controls;
    • Agreements and supporting service level agreements that specify terms of the partnership agreements between AANDC and HC/PHAC; and,
    • Business process documentation for financial and materiel management processes.

Audit planning and fieldwork were conducted in the period August 1, 2013 through April 3, 2014.

The approach used to address the audit objectives included the development of audit criteria, against which observations and conclusions were drawn. The audit criteria developed for this audit are provided in Appendix A of this report.

Relevant policies, directives and guidelines referenced in the audit process are listed in Appendix B.

 

 

4. Conclusion

Generally, the project management framework at AANDC was found to be adequate, and effectively supported the management of the SAP-GCIMS project to ensure that AANDC's expected results of the system migration projects were delivered on time, with no loss of data integrity or system functionality; however, an opportunity for improvement was noted related to formal approvals of deliverables at milestone gates. 

The audit was not able to conclude on whether results were delivered within budget. As of the last AANDC/HC DG Partnership Review Committee held during the audit's conduct phase (March 31, 2014), the project status "dashboard" indicated actual expenses tracking to budget, but these results were dated November, 2013. The audit did not receive regular reports of actual versus planned expenditures in other formats.   

With respect to project risks related to the system implementations, and issues affecting AANDC's readiness for the "go-live", findings and recommendations for addressing risks and issues were communicated by the audit to the AANDC SAP-GCIMS project team during the course of the audit. Opportunities for improvement remain regarding business requirements documentation.  

Observations on leading practices and lessons learned with respect to collaboration on the system implementation projects with Health Canada were noted during the audit; these include opportunities for improvement with respect to project management and governance, management of project documentation and deliverables, preparation of interdepartmental agreements, and organization change management.

Finally, project management methods and controls used to define, enable, and govern the partnership established between AANDC and HC to oversee the ongoing operations of shared information systems were generally effective; however, opportunities for improvement were noted with respect to the content of master agreements and service level agreements.

 

 

5. Findings and Recommendations

Based on a combination of the evidence gathered through the examination of documentation, analysis, and interviews, each audit criterion was assessed by the audit team and a conclusion for each audit criterion was determined. Where a significant difference between the audit criterion and the observed practice was found, the risk of the gap was evaluated and used to develop a conclusion and to document recommendations for improvement. Findings and recommendations reflect work undertaken until the end of the audit's Conduct Phase (April 3, 2014).

During a system under development audit, it is important that the audit team reports findings and makes recommendations to the project team on an on-going basis so that management action can be taken immediately to address identified weaknesses as they arise. A traditional internal audit approach, where recommendations are delivered following the completion of all field work, could result in the project team receiving auditors' advice after an identified weakness had affected the project's success. Therefore, audit findings were presented to AANDC management three times during the conduct of the audit: 

Audit lines of enquiry included controls and procedures related to the management of the SAP-GCIMS project, to risks and issues associated with the implementation of the software systems, and to risks and issues associated with defining, enabling, and governing the partnership established between AANDC and Health Canada to oversee the ongoing operations of shared information systems.

In all, fourteen findings and corresponding recommendations were reported to AANDC management during the course of the audit. Management action plans provided to the audit team on January 20, 2014 and March 28, 2014 were reviewed by the audit team with the Project Sponsor and other members of the AANDC project team, and where appropriate, with the Chief Information Officer. The third and most recent management action plan was received following the end of the Conduct Phase on May 28, 2014.

Briefings on findings were provided to the Departmental Audit Committee (DAC) at their meetings in February and April 2014.

A number of recommendations provided to AANDC management, in particular those that addressed risks related to the system implementations and issues affecting AANDC's readiness for the "go-live" of the system implementation projects, are considered to be "point in time" recommendations, relevant to specific past phases of the project. Therefore, during the reporting phase AASB conducted an analysis of all fourteen findings, recommendations and associated management responses and determined that in several instances management has taken significant action during the course of the audit to mitigate risks. As a result, the number of recommendations that require further monitoring or follow-up was reduced. Implementation of the six recommendations appearing in this report will be monitored. The six findings and recommendations are categorized by the lines of enquiry listed below, consistent with the audit criteria.

Findings and recommendations are described in Sections 5.1 to 5.4 below.

 

In Section 5.5, additional observations are noted. These six findings address the objective of the audit to report on leading practices and lessons learned with respect to collaboration with HC, and other Government of Canada departments on future IT implementation projects.

 

5.1 Project Management and Governance

This line of enquiry includes control objectives related to project management and governance structures, composition and activities of project steering committees, the development and maintenance of an overall project plan, the management of resources deployed to deliver project tasks, and the management of project risks and issues.

Control objectives related to the existence and maintenance of project plans, and the deployment and management of project resources were found to be generally effective.  

The audit found that the Department's Project Portfolio Management Framework is also considered a resource for project managers; however, opportunities for improvement were noted on how the framework applied to the SAP-GCIMS project. Opportunities for improvement were also noted with respect to the management of risks and issues.

The audit found that governance and steering committees existed for the project and were generally effective, although opportunities for improvement were noted with respect to how governance committees document decisions and assign responsibility for follow-up.

5.1.1 Project Portfolio Management Framework

A project management framework is intended to assist project managers and project stakeholders by providing a structure for planning project activities to help ensure that projects achieve planned results. Project documentation requirements, and disciplined review of project status by steering committees to formally approve milestone gates, as defined in the framework, are key requirements to effective project control.

AANDC has developed a Project Portfolio Management Framework (PPMF), and issued the Directive on IM/IT Project Portfolio Management (2012) to align the Department with Treasury Board's Policy on the Management of Projects. The purpose of the PPMF is "… to ensure coherence and corporate discipline is applied to the initiation, planning, execution/control and closing of AANDC's portfolio of projects."

The Directive describes the PPMF's four key requirements for all IM/IT-enabled projects undertaken at AANDC:

  • the use of a gated review process throughout the project;
  • the participation of unbiased parties for all gate reviews;
  • status reviews and approvals at each gate based on standards developed by Treasury Board for the evaluation of mandatory deliverables; and,
  • the capture of standard and accurate information to enable timely reporting that is made available to all stakeholders.

In general, the audit found that project managers were aware of the PPMF, and that most specified requirements for project gate approvals were addressed, particularly in the early phases of the project. Application of project management processes and controls described in the PPMF, such as those related to the management of project issues and risks, is fundamental to effective project management.

Finding:

The audit found that the application of AANDC's PPMF by project managers and by governance committees was inconsistent in the SAP-GCIMS project. Exceptions were noted related to all four key requirements of the PPMF. The audit found that the project team was aware of the PPMF and its defined project gates, but the audit could not determine that all deliverables specified for Gate 5 ("Detailed Project Plan and Functional Specifications") had been formally completed and approved, nor was there a clear demarcation in project plans indicating dates on which the completion of Gate 6 ("Solution Complete") was anticipated for both the SAP and GCIMS implementations. Uncertainty surrounding formal approvals for milestone gates increases the risk that AANDC projects will implement new systems, or introduce updates to existing systems, that do not meet user requirements, contain errors, or operate without necessary security safeguards in place.

Recommendation:

1. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should review gate approvals for the SAP-GCIMS project and assess whether requirements specified in the Project Portfolio Management Framework have been met.

5.1.2  Project Governance

Governance of the project – ensuring that schedules and budgets are maintained, that issues and risks are identified and managed, and that key project milestones are supported by approved deliverables – is necessary for effective project management. Governance and oversight of the SAP-GCIMS project was provided by an executive-level steering committee within AANDC, and by two interdepartmental committees. The Project Charter for the SAP-GCIMS project describes the membership and mandate of two interdepartmental governance committees: the ADM Steering Committee and the DG Partnership Review Committee. 

Among other functions, mandates of both interdepartmental committees emphasize the role that each plays in the communication of key issues and actions related to the project, and to the partnership between departments. The mandate of the DG-level committee is "to provide a forum for the discussion and recommendation on key project activities" and "to ensure an on-going DG-level review of key project activities." Similarly, the mandate of the ADM Steering Committee is "to provide on-going guidance, review and approval for all project activities related to SAP and GCIMS, including … communication and promotion of this dual partnership within their respective departments; and discussion of recommendations to project issues with the goal of gaining consensus on resolution."

In general, the audit found that steering committees provided effective governance to the AANDC project team. In particular, senior managers representing business, technical and project stakeholders who sat on the AANDC DG Steering Committee met regularly throughout the project to communicate project status, discuss project issues and risks as they arose, and coordinate interdepartmental activities involving HC, PHAC, PWGSC, and Shared Services Canada. Interdepartmental governance committees met less frequently than the AANDC DG Steering Committee, and provided a forum for providing updates on status and issues from all project streams from AANDC and from HC/PHAC.

5.2 Functional Readiness

Functional readiness refers to project management controls related to the preparation of the IT applications for implementation. This line of enquiry includes controls related to the involvement of users in the documentation of business requirements and changes to requirements, the certification and accreditation of IT security for software applications, and several aspects of testing the new systems, including: testing application software; testing the accuracy of data migration procedures; and, testing to ensure that system interfaces operate as expected.

The audit found that controls related to the involvement of users in documenting business requirements and changes to requirements were generally effective, although business requirements documented in early phases of the project were not formally tracked through to implementation.

Generally, the audit found that business stakeholders were involved in the specification of requirements to GCIMS, although business requirements documented in early phases of the project were not formally tracked through to implementation.

Although issues were identified with respect to the communication to AANDC of SAP testing plans and results conducted by HC, controls were found to be in place over testing of the GCIMS software, including testing of interfaces between GCIMS and other systems. 

The audit noted that the AANDC project team successfully conducted a technical re-write of the GCIMS application, a system that is critical to the day-to-day business of the Department, without significant error or loss of service, indicating that strong project management, change and release management procedures were in place.

Finding:

Formal documentation and tracking of business requirements for a system implementation project provides a foundation for key project activities, including solution design, planning of test activities and development of test cases, development of go/no-go criteria for implementation of the system, and measurement of the overall success of the project. For a system implementation, tracking of business requirements throughout the project also supports the planning of future releases of the software. The audit found that business requirements documentation was initiated early in the SAP-GCIMS project, but resolutions to gaps in user requirements for the implementation of SAP at AANDC noted in early documentation were not formally documented. Without a documented record of business requirements, there is a risk that systems are developed and implemented successfully from a technical perspective, but do not meet the needs of the business users the systems are intended to support.

Recommendation

2. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should review business requirements documented in the Business Case, Blueprint documents, and SAP functional gaps document, and confirm that requirements have been met by the delivered systems. If gaps exist, temporary work-arounds should be documented, and outstanding requirements retained for consideration in future releases.

The Chief Information Officer should consider amending the Project Portfolio Management Framework to include tracking of business requirements among its mandatory deliverables for system implementation projects.

5.3 Deployment Readiness

Deployment readiness refers to project management controls related to the introduction of the prepared IT system into production service. Controls and processes in this line of enquiry are focused both on the implementation of the systems, and on ensuring that the implemented systems are operated, supported, and maintained appropriately by AANDC (for GCIMS) and HC (for SAP) going forward. Controls include those related to the existence of cutover plans, "go/no-go" criteria that must be met for software releases to be implemented into production use, and communications to user communities regarding the cutover to new systems, or new releases of existing systems.  

This line of enquiry also included a review of the service level agreements (SLAs) and related documents (e.g. memoranda of understanding, master agreements, operating models) that are fundamental to the operation of the partnership between AANDC and HC/PHAC going forward. These documents define and enable the partnership by setting out costs and terms of services, and listing specific services to be provided, including support processes, operations schedules and tasks, and performance expectations, among other items.

Minor issues noted in the review of controls related to the formal documentation of cutover plans and "go/no-go" criteria were included in the scope of recommendations made regarding the application of AANDC's PPMF to project milestone gate approvals (see Recommendation #1) and to the tracking of business requirements (see Recommendation #2).

In general, project management practices related to the communications to user communities were found to be effective, particularly for changes related to the GCIMS application, since GCIMS changes were managed within AANDC.

5.3.1 SAP

Ensuring that SAP operations adequately support AANDC's business needs includes managing the partnership between AANDC, as client, and HC, as provider of SAP system services. The success of the partnership depends on good governance, and on clear, complete service agreements that are respected by both departments, and are reviewed periodically to ensure they remain relevant.

Finding:

The audit found that documentation of the partnership between HC and AANDC for SAP services was not finalized as of April 2014 when migration to SAP occurred. Draft documents obtained by the audit indicated that operating principles have been set out that describe each department's responsibilities to the partnership. Governance principles are also described, including the Terms of Reference for three oversight committees, at the ADM, DG and Director levels. In-scope services are described at a high level in the Master Agreement, and in detail in supporting SLAs. Service performance standards are also documented. These service standards specify expected turnaround times for activities such as restoration of services after outages, and for routine service requests. Financial management and budget planning principles are included in the master agreement; however, costing details for services were not finalized in the draft document. The lack of a completed, defined SLA for SAP services being provided by HC increases the risk that the agreed upon services will not be delivered which could result in inaccurate or incomplete financial information in SAP, and therefore compromise the accuracy of AANDC's financial transactions and financial reporting.

Recommendation:

3. The Director General, Corporate Accounting and Materiel Management, as the SAP-GCIMS Project Sponsor, should work with Health Canada to ensure that service agreements specific to Health Canada's role as an SAP service provider are drafted and finalized.

Furthermore, a steering committee for the ongoing partnership between the two Departments should include in its mandate the monitoring of the performance of HC as service provider to ensure that services levels are being met.

5.3.2 GCIMS

Deployment readiness for GCIMS implies a need for AANDC to shift its resources from those required for a software development project to those required of a software service provider. This includes providing resources to continue to support the application, both at AANDC and in other "client" departments whose use of the GCIMS application is hosted by AANDC. Support in this context means much more than just performing a "break-fix" function, and typically includes: 

  • formalizing procedures for planning and releasing updated versions of the software, so that AANDC and other client departments can plan appropriately;
  • modifying existing procedures for considering and approving changes to the software to include input from client departments; and,
  • performing outreach and client service functions, to attract new clients for the software, and to maintain partnerships with existing clients.

Failure to provide efficient support for user incidents reported to help desks, an efficient change management process, and carefully planned release management processes increase the risk of damage to the reputation of AANDC and the GCIMS application, reduce its acceptance by users, and therefore fail to support the Gs&Cs business processes as intended.

Finding:

Descriptions of support services to be provided, the business processes to be used to deliver services, and performance measures to be used in periodic reviews of services are usually documented in SLAs that support the host-client relationship. In a finding submitted to management early in the conduct phase of the audit (December 2013), it was noted that the post-implementation support organization and operating model at AANDC for GCIMS had yet to be fully defined. By the end of the conduct phase (April 2014), the audit noted that draft documents to support the provision of GCIMS services to HC/PHAC, and potentially other client departments, including an Operational Memorandum of Understanding and a SLA, were aligned with Treasury Board guidelines on service agreements and service standards, and included details on support processes.

Timing of the deployment of GCIMS to most users at HC/PHAC was changed during the course of the audit; while some users at HC began to use the system at the beginning of the 2014-15 fiscal year, the majority (over 1,100) will gain access late in the fourth quarter of 2014-15, to prepare for full use of the system in the 2015-16 fiscal year.  

Without a defined operating model for the GCIMS support organization, there is increased risk that AANDC will not have appropriate human and other resources necessary to ensure an adequate support structure is in place to successfully host and support the GCIMS application.

Recommendations:

4.The Director General, Corporate Accounting and Materiel Management should develop an operating model for a support organization for GCIMS as a hosted system for other client departments. 

SLAs with HC/PHAC for the provision of GCIMS services should align with Treasury Board guidelines on service agreements and service standards. 

5. The Director General, Corporate Accounting and Materiel Management, with assistance from the Chief Information Officer, should prepare a plan for the GCIMS technical support organization that includes technical and business analyst resources, a client service function, and other functions required to support the ongoing development, update, and deployment of the GCIMS application.

5.4 Business Process Readiness

Business process readiness refers to project management processes and controls that support the preparation of users of a newly-implemented application system for its effective use in the execution of daily work tasks. These processes and controls are concerned with preparing user communities for the change to the new system, with training and other steps required to gain familiarity with changed business processes, and the effects of changes on how individuals do their jobs, and with changes to internal controls associated with business processes supported by the new application system.

This line of enquiry includes controls related to the readiness of business process controls for the implementation of new financial systems, the existence and content of user training programs for SAP, and the development and execution of an organization change management plan to manage users' adaptation to newly-implemented financial systems. 

Issues were noted with all control areas, although the audit noted that the project team at AANDC responded to a recommendation on training materials, and applied significant effort to customizing training materials received from HC by including AANDC-specific processes and data prior to delivery of courses to AANDC users in advance of the April migration to SAP.

Finding:

With respect to organizational change management, the audit found that a high-level plan for the management of AANDC's change from OASIS to SAP was prepared early in the project as part of a Business Transformation Strategy. Although some supporting material related to business changes (e.g. details on the changes to AANDC's Chart of Accounts) and on the migration to SAP, were posted on AANDC's intranet site to communicate these changes to users, deliverables referred to in the initial plan, such as a stakeholder analysis, and outcomes of an Organization Readiness Assessment, were not completed. Insufficient organizational change management efforts in a system implementation project can result in management being unaware of challenges faced by users as they adapt to changes associated with implementation activities, and thus increases the risk that system implementation results in inefficient or incorrect use of the system, confusion amongst users with respect to how and when the new system should be used, and a reduced sense of "buy-in" by AANDC staff.

Recommendation:

6. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should ensure that activities described in AANDC's Business Transformation Strategy – particularly those related to the cutover to the new SAP system - are completed by the SAP project team, and results of these activities communicated to AANDC management.

5.5 Lessons Learned for Interdepartmental Projects

An additional six observations that address the audit objective to report on leading practices and lessons learned with respect to collaboration with HC, and with other Government of Canada departments on future IT implementation projects, were noted during the audit. These six observations were not included in the audit's previous interim status reports.  

5.5.1 Project Management and Governance

Steering committees provide a governance function that should ensure that project milestones are met, risks are identified and managed by the project team, project budgets are appropriately managed, and that milestone approval decisions are supported by review of required deliverables, documentation, or evidence. In the SAP-GCIMS project, the structure and mandate of steering committees was well defined in the Project Charter; however, the audit found that these executive-level committees were frequently drawn into informal discussions on project operations, or technical details, rather than focusing on governance activities. Steering committees for projects where more than one department is participating are likely more susceptible to becoming a forum for discussion than projects undertaken within a single department. 

Project governance is also supported by a project management methodology, or framework. Interdepartmental projects can be expected to be faced with differing project frameworks from each participating department. Using more than one framework increases the risk that projects are not executed as a single program. In the SAP-GCIMS project, the configuration and testing of SAP was managed by HC as one project stream, while the implementation of changes to financial processes and preparations for people at AANDC to use SAP was managed by AANDC. HC's SAP team was not required to adopt the use of AANDC's PPMF to structure the project around specific deliverables and approvals. The AANDC team responsible for managing the migration of business processes at AANDC to SAP system was expected to adhere to the AANDC framework, but was dependent on HC for the preparation of key project deliverables required in the framework, such as user acceptance test plans, and "go/no-go" release criteria.

Similarly, changes to the GCIMS software by the AANDC development team were managed separately from business transformation activities at HC/PHAC. AANDC approached the implementation of GCIMS as a series of releases to an existing production system, whereas at HC/PHAC, the objective of the project was to implement GCIMS as a new system. 

In both cases, the separation of project activities into each department's respective domain resulted in key project milestone gate approvals from governance committees not being fully supported by deliverables as specified by the project management framework.  

Lessons Learned:

1. A single project or program management office, reporting directly to the project steering committee, and responsible for all project streams in participating departments, should be  established and assigned responsibility for supporting the steering committee in the fulfillment of its governance responsibilities through preparation of the committee's agenda, and circulation of documents for decision and/or approval.

2. Interdepartmental projects should adopt a single project management framework for all streams of the project to provide a consistent foundation for all project deliverables, and provide a structure for a common, definitive document repository for project deliverables.  

5.5.2 Functional Readiness

Project management frameworks rely on effective implementation by project managers. Implementation of a framework means setting out which key deliverables are prepared during the project, and what the content of each deliverable should be. A definitive record of project activities and deliverables implies a common, centralized repository for project documentation that is recognized by project team members to be the authoritative location for project deliverables from all streams, and from all participating departments. A common repository also reduces compartmentalization of deliverables, where project information is routed up through organizations within departments rather than made available to all project team members. The audit found that key project deliverables, such as test plans and testing results, were not made available to AANDC SAP project team members until the system was ready to be released in production. The presence of a common project repository would permit access by all project stakeholders as materials were required. 

Large IT implementation projects are often focused on deliverables related to the preparation of software, and on technical documentation supporting the software; however, such projects are also business transformation projects. The objectives of the project are based on business requirements; without clear documentation of these requirements and tracking of how the project meets these requirements, there is a risk that key deliverables in a project will be deficient. Process and software design decisions, testing strategies and the preparation of effective user acceptance test cases, and "go/no-go" implementation criteria are all supported by business requirements. Furthermore, tracking of business requirements allows project stakeholders to evaluate overall success of the project after its completion. In the SAP-GCIMS project, the audit found that AANDC's objective to implement HC's SAP system with minimal changes or customization resulted in a lack of emphasis on documenting and tracking financial business requirements, including where these requirements deviated from processes supplied by SAP at Health Canada. Although processing gaps between AANDC's legacy OASIS financial system and SAP were addressed by the project team as the project progressed, there was no formal end-to-end tracking of business requirements, resulting in an informal testing and acceptance process for SAP by AANDC business stakeholders, and an absence of formal "go/no-go" business criteria to guide decisions on implementation readiness.

Lessons Learned:

3. Project managers should establish a common centralized repository for project documentation and deliverables. The central document repository should be accessible to all project team members, in all participating departments. 

4. Business requirements should be captured early in a project, and tracked throughout to ensure that the new system adequately addresses the requirements. User acceptance testing should be performed directly by the business stakeholders to validate that business requirements have been met.

5.5.3 Deployment Readiness

Preparation for operation of shared IT systems relies on the construction of strong documents that define the partnership. The audit found that master agreements and supporting SLAs for the SAP-GCIMS project involved significant time spent drafting, negotiating and revising their contents. Early attention to this aspect of the project reduces the risk that key elements of an agreement are omitted or poorly described, reducing the likelihood of disagreements or misunderstandings later on in the project, and in the continuing operation of the partnership.

Lesson Learned:

5. Project managers should initiate preparation of agreements between departments (MOU, Master Service Agreement, SLAs), including cost sharing arrangements, as early as possible in the project, to promote clear understanding of delivery responsibilities, to assist in the assessment of changes to scope as these changes arise, and to assist in the planning of changes to support processes. Agreements should be consistent with Treasury Board guidelines on service agreements and service standards.

5.5.4 Business Process Readiness

Effects of system implementations on an organization should be assessed, and an appropriate change management program put in place, for all business transformation projects. 

As noted earlier, the audit found that business transformation elements of each of the SAP and GCIMS implementations were separated from their respective technical implementation teams, and affected by the cross-departmental characteristics of the SAP-GCIMS project, making change management activities less integrated in project plans. The audit also found that change management activities in the SAP-GCIMS project that were intended to assist AANDC business users in the transition to a new financial system were not completed as planned.

Implementing a new operational system without appropriate communications and management of the impact of changes on those affected could result in inefficient or incorrect use of the new system, confusion amongst users with respect to how and when the new system should be used, a reduced sense of "buy-in", and potentially failure of the project to recognize planned benefits. Elements expected to be found in business change management plans include identification of stakeholders affected by the system change, an assessment of the communications and assistance required by each group to manage the change, and a planned set of steps to be followed to guide stakeholders as change proceeds.  

Lesson Learned:

6. Project management should ensure that organization change management activities are included in project plans for all business transformation projects. For interdepartmental projects, organizational change management teams should be coordinated, so that change plans of departments receiving services have their plans informed by the department providing the new service. Finally, costs associated with the change effort should be considered in the overall Master Agreement between the participating departments.

 

 

6. Management Action Plan

Recommendations Management Response / Actions Responsible
Manager (Title)
Planned
Implementation
Date
Program Response
1. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should review gate approvals for the SAP-GCIMS project and assess whether requirements specified in the Project Portfolio Management Framework have been met. As of May 28, 2014:

The Project addressed standard project deliverables called for by AANDC's Project Management Framework for Gate's 1 through 4, including Business Case, Project Charter, Project Plans, High level architecture (Blueprint).

Gate 5 and 6 deliverables were tracked as follows (by system):

GCIMS:
  • Requirements were recorded within a change management control system (test track pro) for individual changes. Multiple changes went into each GCIMS release;
  • Test plans were developed for each release and a complete list of documented test scenarios was available with test scripts for each scenario;
  • Testing was centrally managed under a QA manager and test results were printed, signed and retained in a central file; and,
  • User Acceptance testing was undertaken by user business unit representatives in a formal User Acceptance Testing environment (i.e. AANDC, PHAC, SPB-HC)
SAP:
  • Requirements were documented by business analysts with the engagement of business owners. Formal audit trail of requirements included signed-off SAP blueprint and Gap documents. Detailed requirements were developed and retained by SAP business analysts.
  • Testing of SAP functionality was undertaken and confirmed through the AANDC SAP Business Analysts. Overall satisfaction with SAP functionality was a key part of approvals for SAP releases.
  • User Acceptance testing was not planned.
Gate 6 approvals for system releases, system user roll-outs and data conversions had explicit go/no-go criteria.

To date (as of April 24th), there have been 4 GCIMS enhancement releases, 2 SAP enhancement releases, 3 GCIMS User releases, 13 AANDC SAP User-wave releases, and 9 SAP Conversion releases.

Clearly documented go/no-go criteria for each Gate 6 and a precise schedule were presented in slide decks to the Joint DG committee in slides and diagrams but not documented in detailed project plans in advance. This gap in detailed plans and in the timely updating of risk logs and issues logs contributed to regular delays in establishing monthly Project Dashboards.

The gating process will be looked at from a lessons learned perspective.

As of June 25, 2014:

The Lessons Learned (included in the requirements for Gate 7) and differences between the Gate 5 and 6 approval processes for system releases, user roll-outs and conversions are scheduled to be documented by December 31, 2014.
Director General (DG), Corporate Accounting and Material Management (CAMM) December 31, 2014 PROGRAM RESPONSE :
Status: Completed
Update/Rationale:
As of March 31, 2015

GCIMS:
GCIMS completed the deliverables for Gate 5 and 6. HC now is on board with GCIMS, and is looking to implement new major enhancements to satisfy their corporate priorities.

SAP:
SAP documented the lessons learned in the Project Close out report.

 
AES: Implementation complete. Recommendation to be closed. Closed.
2. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should review business requirements documented in the Business Case, Blueprint documents, and SAP functional gaps document, and confirm that requirements have been met by the delivered systems. If gaps exist, temporary work-arounds should be documented, and outstanding requirements retained for consideration in future releases.

The Chief Information Officer should consider amending the Project Portfolio Management Framework to include tracking of business requirements among its mandatory deliverables for system implementation projects. 
As of: May 28, 2014:

The finding is correct that: documentation of (GCIMS) business requirements was managed within the software development team, consistent with previous releases of the software; however, tracking of business requirements was not formally documented by the project.

Given that changes were implemented by an established development team who, as a team, were accustomed to the established set of operational change management protocols, the documenting and approval of requirements within ADDDA's TTP system worked well. Given the aggressive timeline of this large custom system initiative, this approach served the Project outcome very well. The full audit trail of detailed requirements is available but within an operational tool rather than within Project documents. (Note: GCIMS releases 7.3.0 through to 7.4.0 constitute the Project Changes).

The SAP Fit Gap document was last updated in the Fall 2013. This document identified AANDC business requirements and whether there was an SAP solution. In some exceptions, HC made some SAP customizations for AANDC where the current SAP solution did not meet AANDC requirements and a business process work around was not possible.

For SAP changes, detailed requirements were defined and documented by individual business analysts much as they would in their operational role under HC established SAP support practices.

The documentation of requirements could certainly have been undertaken more as a project than operational releases. Greater effort could be undertaken to ensure this approach in future projects.

HC will manage and track all Partner ongoing SAP requirements under the guidance of the joint Governance Committees as defined in the SAP SLAs.

We agree with the recommendation and will take this forward in a lessons learned perspective for processes involving system implementation projects.

As of June 25, 2014:

Overall Project Lessons Learned (included in the requirements for Gate 7) are scheduled to be documented by December 31, 2014.

Greater effort will be undertaken to ensure a project approach to documentation in future projects.
DG, CAMM

Chief Information Officer
December 31, 2014 PROGRAM RESPONSE :
Status: Completed
Update/Rationale:
As of March 31, 2015

The GCIMS/SAP project is completed. It is currently at Gate 7. Proper documentation was used throughout the project using TTPRO for defects and minor change requests, word documents for testing scenarios and business requirements. GCIMS and SAP documented the lessons learned through the Project Close out Report.

AES: Implementation complete. Recommendation to be closed. Closed
3. The Director General, Corporate Accounting and Materiel Management, as the SAP-GCIMS Project Sponsor, should work with Health Canada to ensure that service agreements specific to Health Canada's role as an SAP service provider are drafted and finalized.

Furthermore, a steering committee for the ongoing partnership between the two Departments should include in its mandate the monitoring of the performance of HC as service provider to ensure that services levels are being met.
As of May 28, 2014:

At the Time of this Initial Finding:

A draft SLA is in process and is targeting completion by April 1, 2014.

Steering Committee's role and performance measures are to be addressed within the SLA including continuing after the April 1, 2014 go-live.

As of June 25, 2014:

A final draft SAP Master Agreement and SLA documents have been created that outline performance standards as well as governance structure and responsibilities. These documents are scheduled to be signed off by July 31, 2014 as they are addressing the final cost elements.
DG, CAMM Revised Date: Q3 2015-2016 Status: In Progress
4. The Director General, Corporate Accounting and Materiel Management should develop an operating model for a support organization for GCIMS as a hosted system for other GC departments. Service Level Agreements (SLAs) with Health Canada for the provision of GCIMS services should align with Treasury Board of Canada Secretariat Guidelines on Service Agreements and Service Standards.   As of May 28, 2014:

As per the recommendation to use the TBS Guidelines on Service Agreements and Service Standards, initial drafts of an MOU for GCIMS operations services as well as a detailed Service Level agreement, have been developed. The detailed SLA contains an operating model for the GCIMS post-implementation support organization.

Discussions with our partners started on February 7th on the content of the MOU and the detailed SLA.

As of March 18th, MOU discussions are complete. The draft MOU has been shared with AANDC, HC and PHAC Senior Management for their review and comments.

The SLA work sessions with HC and PHAC are complete and Draft SLAs have been shared for input/comments.

As of June 25, 2014:

The Memorandum of Understanding and Service Level Agreement for GCIMS have been revised to be separated into two distinct documents: one for PHAC and one for HC; and are under review by HC and PHAC senior management.

Review of ongoing costing requirements is being finalized by HC and PHAC.

Finalize and secure endorsement from ADMs/CFOs of both MOU and SLA by July 31, 2014.
DG, CAMM Revised Date: Q3 2015-2016 Status: In Progress
5. The Director General, Corporate Accounting and Materiel Management, with assistance from the Chief Information Officer, should prepare a plan for the GCIMS technical support organization that includes technical and business analyst resources, a client service function, and other functions required to support the ongoing development, update, and deployment of the GCIMS application. As of May 28, 2014:

As per recommendation #4, detailed SLA and changes to GCIMS support roles are scheduled for Q3 2014-15 and will be in place on time to support HC's 1,100 users. The detailed SLA will document the organization and necessary resources.

As of June 25, 2014:

Indeterminate employees and new contracted resources have been added to transition knowledge for functional, technical support and maintenance of the application. The desired team structure has been identified and a maintenance plan has been developed to assist in ensuring that the technical support has a standard change process and management structure to maintain the application.
DG, CAMM

Chief Information Officer
Revised Date: Q3 2015-2016 Status: In Progress
6. The Director General, Corporate Accounting and Materiel Management, as SAP-GCIMS Project Sponsor, should ensure that activities described in AANDC's Business Transformation Strategy  – particularly those related to the cutover to the new SAP system - are completed by the SAP project team, and results of these activities communicated to AANDC management As of May 28, 2014:

Business Process Maps were to be used in a series of impact assessment workshops and business readiness assessment. These were not accomplished as scheduled and required several changes to approach. Specifically including:  
  • Key business processes were reviewed with business owners within corporate services and hubs.
  • The updating of the full set of business processes identifying resulting process controls is scheduled for completion post-project inception.
With respect to implementing SAP to support grants and contributions business processes, changes to GCIMS have no impact on organizational structure or processes at AANDC with the exception of what is necessary to support the HC portfolio. Once finalized the results will be communicated.

As of June 25, 2014:

A completeness review of business process maps and refinements to clearly state business and system controls is scheduled to be completed by December 31, 2014.
DG, CAMM December 31, 2014 PROGRAM RESPONSE :
Status: Completed
Update/Rationale:
As of March 31, 2015

The SAP business process maps completeness review has been performed and documented.

AES: Implementation complete. Recommendation to be closed. Closed.
 

 

Appendix A: Audit Criteria

Audit criteria were developed to align with the audit objective and COBIT, an industry standard framework of Control Objectives for Information and TechnologyFootnote 2:

Audit Criteria
Project Management and Governance
1.1 Project management and governance structure has been established to provide oversight, ensure deadlines are met, ensure scope is adhered to, and ensure objectives are met.
1.2 A Steering Committee has been established that has a clear mandate and representation from key stakeholder groups.
1.3 A project plan has been established and accepted by all parties and stakeholders to assist in meeting project deadlines and objectives.
1.4 Adequate and appropriate resources have been assigned to execute the project plan.
1.5 Issues and risks are formally identified, tracked, managed, and resolved in a timely and appropriate manner.
Functional Readiness
2.1 Changes to financial business processes have been analyzed and reviewed with business users.
2.2 Key project business requirements documentation has been developed and approved by project stakeholders, including change requests.
2.3 Security requirements have been analyzed and a plan has been established to ensure adherence to departmental and GC security requirements.
2.4 A comprehensive testing strategy and plan has been developed for all business process changes and data migrations, which includes business user involvement.
2.5 Adequate testing has been planned for data migration.
2.6 Adequate testing has been planned for interfaces.
Deployment Readiness
3.1 A system cutover plan is part of the overall project plan, and includes documented conversion specifications.
3.2 For major milestones, management documents go/no-go criteria and subsequent decisions.
3.3 A communication plan informs stakeholders and management of the progress of the roll-out.
3.4 AANDC has established appropriate mechanisms with Health Canada related to its post-project role as a service provider to AANDC for SAP.
3.5 AANDC has defined its processes related to its role as a service provider of GCIMS to Health Canada.
Business Process Readiness
4.1 Business process controls are reviewed for potential changes based on changes to business processes. If there are changes, documentation is updated accordingly.
4.2 A training program has been developed, and executed prior to deployment of the new system; affected business functions are trained prior to implementation.
4.3 An organization-wide change management plan to identify and document technical and operational aspects of the system to SAP users at AANDC has been prepared and implemented.
 

 

Appendix B: Relevant Policies, Directives, and Guidelines

The following authoritative sources were examined and used as a basis for this audit:

 

 
 
 
Date modified: