Follow-Up Audit of the Education Information System

Notice

This website will change as a result of the dissolution of Indigenous and Northern Affairs Canada, and the creation of Indigenous Services Canada and the eventual creation of Crown-Indigenous Relations and Northern Affairs Canada. During this transformation, you may also wish to consult the updated Indigenous and Northern Affairs home page.

December 2015

PDF Version (183 Kb, 26 Pages)

 

Table of contents

Acronyms

DCI

Data Collection Instrument

EIS

Education Information System

EIS-AU

EIS Application Upkeep phase

EIS-NG

EIS Next-Generation Project

ERAS

Education Reporting and Analysis Solution

ESDPP

Education and Social Development Programs and Partnerships

GCIMS

Grants and Contributions Information Management System

HQ

Headquarters

INAC

Indigenous and Northern Affairs Canada

IMB

Information Management Branch

IRS

Indian Registration System

PSE

Post-Secondary Education

SSC

Shared Services Canada

TB

Treasury Board

 

 

Executive Summary

Background

An Audit of the Education Information System (EIS) was included in the Department's 2015-2016 to 2017-2018 Risk-Based Audit Plan. This audit, which is a follow-up to the 2010-2011 System under Development Audit of the EIS, was identified as a priority since the EIS is a critical tool used to support improved accountability for education programs, inform changes to policy and program development, and improve service delivery.

Education Programs

The Department's Education Programs, managed primarily within the Education and Social Development Programs and Partnerships (ESDPP) Sector, and delivered by the Regional Operations Sector, are designed to support First Nation and Inuit students in achieving educational outcomes comparable to those of other Canadians. The goal of the Department's Education Programs is to provide eligible First Nation and Inuit students with quality education and the opportunity to acquire skills that will prepare them to enter the labor market and participate in Canada's economy.

Education Information System (EIS)

The EIS is a computer database that manages the information relating to the Department's Education Programs. The system was implemented to modernize and replace paper-based processes for reporting and to streamline data collection practices. The system provides valuable additional data in a central location, which is intended to aid the Department and First Nations in strengthening accountabilities in the delivery of First Nation education by tracking performance, the movement of students, their academic achievement, and measuring success of Education Programs. The resulting information will allow the Department and First Nations to gain a better understanding of the performance of students and schools, address concerns and make enhancements where necessary, to support improved educational outcomes, in a timely manner. First Nations adoption of EIS and their use of the data will improve as the benefits of using the system are demonstrated, as capacity improves and user training is provided. All First Nations are currently using EIS in some capacity. Data entered into the EIS can be done through a Portal which is a seamless departmental gateway to a variety of systems, reporting requirements and tools including the Education Information System, the Education Reports and Analysis Solution (ERAS), and other departmental reporting systems.

EIS was designed for First Nations to log in and submit education-related information to fulfill reporting requirements and run reports for their own monitoring purposes. The rate of adoption of the system is therefore a key indicator of EIS' success.

During the audit, it was noted that a limited number of First Nations were able to use the full functionality of EIS (i.e. leveraging the Portal to enter, review and submit education-related information for proposal and reporting requirements). EIS is supported by the ERAS, which is the reporting tool used by regional and recipient users to access general and customized reports to inform timely decision-making.

Audit Objective and Scope

The objective of this audit was to assess the adequacy, effectiveness and sustainability of EIS, including supporting processes and controls designed to help ensure EIS achieves its goal of strengthening accountability and compliance within the Department's suite of Education Programs.

The scope of the audit included assessments of the adequacy and effectiveness of manual and automated controls and processes designed and operated to: (i) support an effective Education Information System; and (ii) ensure that information entered into the system is reliable and access to it is protected. The audit was focused primarily on the transition period as EIS moves from a project to maintenanceFootnote 1. This included:

  • Governance, Risk Management and Control during the Transition from Project Mode to Ongoing Upkeep of the System – The oversight mechanisms, processes, tools, resources and strategies developed to move EIS into maintenance, including but not limited to: prioritization and implementation of remaining system changes; monitoring/ managing/ reporting on risks; integration of EIS in business processes; training programs for new and experienced users; engagement with and change management strategies related to recipients for their adoption of the system; improvements to the performance measurement framework; sustainability of the system; and short-term and long-term funding to support sustainability.
  • Reporting Functionality of the System– The extent to which the reporting capabilities of EIS and ERAS are meeting and will continue to meet the information needs of stakeholders.
  • Data Integrity – The integrity (i.e. accuracy, completeness and reliability) of the information that is being entered into and maintained in EIS given that the information can be input by various methods and come from multiple sources (i.e. direct input from recipients through a Portal, direct upload from a School Information System, upload from a smart PDF and manual input within a region based on information provided by the recipients). This included the interface controls in place to enable the data integrity of the information that is transferred to/from EIS from other departmental systems and the conversion of the legacy data into EIS upon implementation.
  • Data Security and Privacy – Information that is transferred into and housed within EIS is protected from unauthorized access.

To the extent applicable, the audit followed up on the status of management's actions to address recommendations arising from the previous System Under Development audit.

The scope of the audit covered the period of September 2010 to June 30, 2015, with transactional testing focused on the 2014-2015 and 2015-2016 (to June 30, 2015) fiscal years. The systems in scope for this audit included EIS and ERAS, as well as the interfaces between EIS and other departmental systems to the extent that they impact the system.

Statement of Conformance

This audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

Conclusion

The audit found that, while EIS has attained some of its objectives, the overall project goals have not yet been fully achieved. System issues affecting the achievement of these goals includes system compatibility, performance and reporting, as well as limited training to First Nations on how to use the system. As a result, some regional representatives maintained parallel tools and processes, which combined with the system performance issues, increased their workload. However, the workload, overall, including First Nations, is less than in previous years due to the benefits of automation brought by EIS.

The audit noted that activities were ongoing during fiscal 2015-2016 in an effort to move EIS to maintenance by the planned timeline of April 1, 2016. These activities included addressing and resolving known system issues and supporting regions and First Nations in their use of the system. While these activities are ongoing, a formal transition plan was not established nor have formal monitoring of transition activities and spending taken place to continue to assess whether achievement of maintenance by the expected timeline was possible.

Recognizing that certain constraints experienced during the course of the project have impeded the achievement of some goals, the project team has not undertaken a re-establishment of the overall objectives and targets that align more closely with the current business and system realities. As a result, it is challenging to assess the effectiveness of the system's transition to maintenance and overall sustainability.

Recommendations

The audit identified areas where management control practices and processes should be improved, resulting in the following recommendations:

  1. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector jointly with the Chief Information Officer should ensure that a re-establishment of the long-term vision of EIS be undertaken to define the system's (upkeep and related business processes) maintenance as well as the outstanding activities that need to be completed to achieve this state. Once this has been defined, a formal transition plan should be established outlining how and when these activities will be completed and formal monitoring activities should take place to track the status of the achievement of the objectives.
  2. In alignment with Recommendation #1, the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector jointly with the Chief Financial Officer should ensure that once maintenance objectives, targets and key activities are established, funding requirements for these key activities be determined, approved and formally tracked and reported to governance bodies in relation to the progress of these activities.
  3. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector, in collaboration with the Senior Assistant Deputy Minister of Regional Operations, should ensure that a sufficient and reasonable training plan is developed to support the provision of EIS training to First Nations. This plan should be supported by the resources and tools necessary to enable regional representatives to carry out their training responsibilities.
  4. In conjunction with Recommendation #1, the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should re-establish and articulate the long-term vision and solution for the use of EIS/ERAS by First Nations. If maintenance includes the use of smart PDFs, USB keys and limited access to reporting tools by First Nations, the associated functionality and processes should be established to ensure consistency and the most efficient use of the system as possible. If a longer-term vision for the use of EIS by First Nations is established, the necessary objectives, funding and associated activities should be articulated to enable this to be implemented.
  5. The Chief Information Officer in collaboration with the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should ensure that system capacity reports are monitored on a periodic basis and documented evidence of this review is retained, and that ongoing efforts are made to improve the system capacity to allow more First Nations to use the system simultaneously.
  6. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector, in collaboration with the Chief Information Officer, should ensure that coding errors and defects resulting from new releases are tracked and formally communicated to regions and recipients, along with workarounds and fixes when applicable.
  7. The Chief Information Officer in collaboration with the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should ensure that the Certification and Accreditation process for EIS is completed for to meet to the Government of Canada security requirements; and that the use of test data is considered for use within the development and test environments, rather than production data, to ensure sensitive data remains secure.

Update since audit report was finalized
Since the end of the audit's conduct phase, work has continued to further improve the EIS. Regular updates are provided to oversight committee meetings and approvals were obtained for the Application Upkeep governance model. Work will progress in accordance with funding approvals and human resource capacity. The management action plan calls for a review of the vision for EIS and a training plan for First Nation users of the system to be completed by the end of the 2016-2017 fiscal year. System capacity testing has been completed and reported, and an inventory of short term system enhancements was developed. A plan to implement longer term enhancements is underway. Measures have been undertaken to track coding errors and defects and a process has been established to better communicate new system releases to regions. Regular updates on the action plans are being provided to the Departmental Audit Committee on a quarterly basis.

 

 

1. Background

An Audit of the Education Information System (EIS) was included in the Department's 2015-2016 to 2017-2018 Risk-Based Audit Plan. This audit, which is a follow-up to the 2010-2011 System under Development Audit of the EIS, was identified as a priority since the EIS is a critical tool for supporting improved accountability for education programs, inform changes to policy and program development, and improving service delivery.

Education Programs

The Department's Education Programs, managed primarily within the Education and Social Development Programs and Partnerships (ESDPP) Sector, and delivered by the Regional Operations Sector, are designed to support First Nation and Inuit students in achieving educational outcomes comparable to those of other Canadians. The goal of the Department's Education Programs is to provide eligible First Nation and Inuit students with quality education and the opportunity to acquire skills that will prepare them to enter the labor market and participate in Canada's economy.

Education Information System

EIS is a computer database that manages the information relating to the Department's Education Programs. The system was implemented to modernize and replace paper-based processes for reporting and to streamline data collection practices. The system is intended to aid the Department and First Nations in strengthening accountabilities in the delivery of First Nation education by tracking performance, the movement of students, their academic achievement, and measuring success of Education Programs. The resulting information accessible to program managers, in a central location, is of great value to the Department and  will allow the Department and First Nations to gain a better understanding of the performance of students and schools, address concerns and make enhancements where necessary, in order to support improved educational outcomes of students, in a timely manner.

EIS is supported by the Education Reporting and Analysis Solution (ERAS), which is the reporting tool used by regional and recipient users to access general and customized reports to inform timely decision-making.

 

 

2. Audit Objective and Scope

2.1 Audit Objective

The objective of this audit was to assess the adequacy, effectiveness and sustainability of EIS, including supporting processes and controls designed to help ensure that EIS achieves its goal of strengthening accountability and compliance within the Department's suite of education programs.

2.2 Audit Scope

The scope of the audit included assessments of the adequacy and effectiveness of manual and automated controls and processes designed to: (i) support an effective Education information management system; and (ii) ensure that information entered into the system is reliable and access to it is protected. The audit was focused primarily on the transition period as EIS moved from a formal project to maintenanceFootnote 2. This included:

  • Governance, Risk Management and Control during the Transition from Project Mode to Ongoing Upkeep of the System – The oversight mechanisms, processes, tools, resources and strategies developed to move EIS into maintenance, including but not limited to: prioritization and implementation of remaining system changes; monitoring/managing/ reporting on risks; integration of EIS in business processes; training programs for new and experienced users; engagement with and change management strategies related to recipients' adoption of the system; improvements to the performance measurement framework; sustainability of the system; and short-term and long-term funding to support sustainability.
  • Reporting Functionality of the System – The extent to which the reporting capabilities of EIS and ERAS are meeting and will continue to meet the information needs of stakeholders.
  • Data Integrity – The integrity (i.e. accuracy, completeness and reliability) of the information that is being entered into and maintained within EIS given that the information can be input by various methods and come from multiple sources (i.e. direct input from recipients through a Portal, direct upload from a School Information System, upload from a smart PDFFootnote 3 and manual input within a region based on information provided by the recipients). This included the interface controls in place to enable data integrity of the information that is transferred to/from EIS from other departmental systems (i.e. Indian Registration System (IRS) and the Grants and Contribution Information Management System (GCIMS)) and the conversion of the legacy data into EIS upon implementation.
  • Data Security and Privacy – Information that is transferred into and housed within EIS is protected from unauthorized access.

To the extent applicable, the audit followed up on the status of management's actions to address recommendations arising from the previous System under Development audit.

The scope of the audit covered the period of September 2010 to June 30, 2015, with transactional testing focused on the 2014-2015 and 2015-2016 (to June 30, 2015) fiscal years. The systems in scope for this audit included EIS and ERAS, as well as the interfaces between EIS and other departmental systems, to the extent that they impacted EIS.

 

 

3. Approach and Methodology

The audit was conducted in accordance with the requirements of the Treasury Board Policy on Internal Audit and followed the Internal Auditing Standards for the Government of Canada. The audit examined sufficient, relevant evidence and obtained sufficient information to provide a reasonable level of assurance in support of the audit conclusion.

The audit methodology used for this audit included performing various procedures necessary to address the audit objectives. The audit approach encompassed multiple lines of enquiry, including documentation review, interviews, walkthroughs, testing and analysis. The audit work was performed at Headquarters (HQ) and a sample of two regions was selected to be included in the conduct phase of the audit.

The approach used to address the audit objectives included the development of audit criteria against which observations, assessments and conclusions were drawn. The audit criteria developed for this audit are included in Appendix A.

 

 

4. Conclusion

The audit found that, while EIS has attained some of its objectives outlined at the outset of the project, the overall project goals have not yet been fully achieved. System issues affecting the achievement of these goals included those related to system compatibility, performance and reporting, as well as limited training to First Nations on how to use the system.

The audit noted that activities were ongoing during fiscal 2015-2016 in an effort to move EIS to maintenance by the expected date of April 1, 2016. These activities included addressing and resolving known system issues and supporting regions and First Nations in their use of the system. While these activities are ongoing, at the time of the audit a formal transition plan was not established nor had formal monitoring of transition activities and spending taken place to continue to assess whether achievement of maintenance by expected timelines was possible.

Recognizing that certain challenges experienced during the course of the project impeded the achievement of some specific goals, the project team had not undertaken a re-establishment of the overall system objectives and targets that align more closely with the current business and system realities. As a result, it has been challenging to assess the effectiveness of the system's transition to maintenance and overall sustainability.  

 

 

5. Findings and Recommendations

Based on the evidence gathered through documentation review, interviews, and analysis, each of the lines of enquiry was assessed against pre-established audit criteria, as detailed in Appendix A. Where a difference between the audit criterion and the observed practice was found, the risk of the gap was evaluated and used to develop the conclusion and corresponding recommendations for improvement. This section provides the findings of the audit, with a focus on those areas where gaps were observed and recommendations for improvement were identified.

5.1 Transition to maintenance

EIS, as it existed during the audit period, was implemented over the course of three phases:

  1. EIS Project – April 2010 to March 2014
    • The scope was defined by the original project approval documents
    • The system went live in December 2012 followed by a series of releases to address system issues up until March 2014
  2. EIS Next-Generation Project (EIS-NG) – April 2014 to June 2015
    • The scope was to address outstanding items originally included in the EIS project and to improve the existing system
    • The goal was to improve the system in one year and provide ongoing support in future years
  3. EIS Application Upkeep (EIS-AU) – April 2015 to March 2016
    • The purpose of this phase is to transition the support of EIS from the EIS-NG project to ESDPP and Information Management Branch (IMB) for ongoing support

The third phase, EIS-AU, was defined as the transition period for the EIS system to move from a project to maintenance, where it would be maintained by ESDPP and IMB. This phase was the primary focus of the audit and the following subsections detail the findings related to the system's transition to maintenance.

5.1.1 Transition Plan

Given that EIS was in its transition phase during the audit period, moving from a project to maintenance, it was communicated to the audit team that a formal Transition Plan had been established. Formal transition plans typically set out the key objectives and targets to be achieved by the end of the transition phase. In setting these objectives and targets, the project team would be defining the requirements to move EIS to maintenance. Establishing these objectives and targets would allow the project team to assess its progress throughout the year and enable management to determine whether the system has achieved its maintenance as scheduled. In addition to transition phase objectives and targets, other key elements of a formal transition plan include the associated governance structure and framework, roles and responsibilities of stakeholders, knowledge transfer and training activities, key IT processes and activities, as well as funding requirements for the transition period. Many of these elements were observed in different EIS-AU documents. However, the audit noted that a formally documented Transition Plan that outlined the objectives and targets, detailed activities and associated funding specifically for the transition phase was not established. As a result, it was challenging to determine how the project team defined the system's maintenance and what was required for the system and associated processes to achieve this result.

In lieu of formally defined objectives and targets for the transition phase and the subsequent maintenance, the audit team looked to the original project goals, objectives and critical success factors established at the beginning of the EIS project in 2010-2011. In doing so, the audit team assessed EIS against the four established critical success factors, which underpinned the project's goals, objectives and business outcomes. These factors were to be met by the end of the original EIS project completion (revised date of March 2014) in order to be assessed as providing the necessary benefits relative to the investments made. The critical success factors are highlighted in the table below along with an assessment against these factors:

Critical Success Factor Current state assessment
#1 Adoption by First Nations

While all First Nations have moved from paper-based to electronic Data Collection Instruments (DCI), a limited number are using EIS to upload their information directly to the Portal due to compatibility issues.

  • 100% of First Nation recipients are using the system for data capture, including the use of encrypted USB keys with attached smart PDFs. 
  • About 10% of the recipients are using the on-line capability of the EIS and the reporting/analytical capability of ERAS.
#2 Improved information for informed decision-making While reports are available for Elementary/Secondary education programs, during the audit period reports were not available for Post-Secondary Education (PSE).
#3 Reduced work burden for First Nations While burden is reduced as a result of automation and the reduction in the number of DCIs, there can still be ongoing back and forth between some regional representatives and First Nations to complete the DCI data validation process.
#4 Reduced manual work for INAC staff Regional staff no longer process paper-based forms and have access to online tools. However, at the time of the audit, in some regions staff still uploaded DCIs to the Portal and completed the data validation on behalf of several First Nations. This increased the work burden for some regional representatives. Due to the performance issues of the system still present at the time of the audit, some regional representatives continued to maintain parallel tools and processes.

As demonstrated in the table above, while the audit team noted that elements of these critical success factors have been attained, not all of them have been fully achieved nor are they expected to be achieved by the end of the transition phase. As a result, the overall EIS project objectives and goals have not been fully realized.

In gaining an understanding as to how and why EIS has not fully achieved its critical success factors, it was noted that the objectives for EIS and EIS-NG were not fully achieved at the time of project close, and that outstanding items were moved forward to the subsequent phase. The impact of this resulted in the transition phase primarily focusing on the implementation of key functionality to improve the system and achieve the outstanding objectives rather than the transition of the system to maintenance (i.e. ongoing system upkeep and transferring knowledge and responsibilities from contractors to staff). Secondly, it was noted that certain constraints (i.e. the Government of Canada requirement that all systems be built on an Internet Explorer platform) resulted in compatibility issues for many recipients, affecting the attainment of the system's critical success factors. As such, it was identified that over the five-year course of the system's development and implementation, several internal and external constraints affected the achievement of the original project goals.

In recognizing these constraints, the audit noted that a re-baseline exercise to re-establish system goals, objectives and critical success factors for EIS had not been undertaken to more closely align them with the current business, system and project realities. This would have allowed the team to define maintenance for EIS that reflects current constraints and position them to develop a transition plan designed to achieve maintenance.

Recommendation

1. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector jointly with the Chief Information Officer should ensure that a re-establishment of the long-term vision of EIS be undertaken to define system's (upkeep and related business processes) maintenance as well as the outstanding activities that need to be completed to achieve this state. Once this has been defined, a formal transition plan should be established outlining how and when these activities will be completed and formal monitoring activities should take place to track the status of the achievement of the objectives.

5.1.2 Funding

Funding of $27.3M was allocated for the development and implementation of the EIS at the onset of the project with an additional $450K for each subsequent year for ongoing maintenance costs (effective 2013-2014). During the project, and for each additional phase, the original EIS budget was supplemented with additional funding for ongoing support and other functionality identified by the project team. In total, at the time of the audit, it was estimated that $56.2M was to have been provided for the EIS system and its upkeep by March 2016. The table below depicts the estimated approved funding allocated for each phase of EIS:

EIS Phase Approved Funding
EIS Project
(September 2010 – March 2014)
$39.6M
(Original funding of $27.3M plus $12.3M for ongoing support and additional functionality)
EIS-NG
(April 2014 – March 2015)
$10.0M
($450K as well as $9.5M additional funding provided)
EIS-AU
(April 2015 – March 2016)
$6.6M
($450K as well as $4.9M in additional approved funding and $1.3M in operational funding allocated by the Education Branch)
Total estimated funding for EIS: $56.2M

The audit team noted that a report called the Ongoing Support Activities, Organization, and Costs Report, isa key EIS-AU document that outlined the funding required to support the transition activities in 2015-2016. Supporting this funding amount was a list of high-level activities that the funding would be allocated towards, including ongoing support to regional and First Nations users, transition activities from consultants to staff, and implementing system functionality to improve EIS. Despite the articulation of these activities, no evidence was found of reporting or monitoring activities having taken place to track these funds against the activities they were expected to be used for.

Based on the lack of reporting in relation to key activities required to achieve maintenance, it is difficult to assess whether EIS is on track to achieve maintenance within the allocated budget. Some concerns were raised in oversight committee meetings regarding the funding remaining for 2015-2016 as well as funding allocated for 2016-2017 and whether it was sufficient to carry out ongoing support activities. Given the fact that formal transition phase targets were not established and only high level activities were defined with limited monitoring of these activities against budget, it is not clear how much funding is required for EIS to achieve maintenance and whether this is within the existing funding allocation provided for EIS.

Recommendation:

2. In alignment with Recommendation #1, the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector jointly with the Chief Financial Officer should ensure that once maintenance objectives, targets and key activities are established, funding requirements for these key activities be determined, approved and formally tracked and reported to governance bodies in relation to the progress of these activities.

5.2 Training

Given the transformational nature of EIS, (moving from paper-based processes to online, electronic submissions of education-related data by recipients) training programs that include comprehensive, relevant training materials for various stakeholders are important. A formal training program, including ongoing support, would enable successful adoption of the available tools within EIS by regional and First Nations' users.

The audit team noted that HQ has one full-time resource (supported by contracted resources until March 2016) who is dedicated to training regional users on EIS. This trainer provides training sessions to all regions and is available for ongoing support and tailored training sessions, as required. Since EIS has experienced several major releases during the EIS-NG and EIS-AU phases, formal training was provided to regional representatives on the latest system functionality after each release. This training was mostly provided through online, web-based sessions, and while some regional representatives expressed challenges with the effectiveness of this training tool due to connectivity issues, the audit team noted that the online method was more cost-effective than in-person training considering the number of regional representatives to be trained, the geographic dispersion of the regional locations and the number of releases requiring training. The audit noted that training documents have been tailored for regional users to support them in carrying out their roles and responsibilities relative to EIS.

While regional representatives have received ongoing training from HQ, EIS funding allocation changes were made since the onset of the project resulting in a significant reduction to the training budget, specifically relative to First Nations' training. As a result, the revised budget for EIS did not include funding for face-to-face training for First Nations' users and it was expected that regional representatives were to fund and deliver the training of First Nations as part of their ongoing operations. Although this was the expectation from HQ, regional representatives noted that they have been challenged to dedicate funds to training First Nations given their own ongoing funding pressures. In addition, while training materials have been developed for EIS by HQ, it has been identified that they are not optimal for First Nation audiences and as such, some regions have had to dedicate additional resources to tailoring and development of training materials suitable for First Nations.

While regions have been challenged in obtaining the resources necessary to train First Nations on EIS, it was also noted that some regions have been reluctant to provide training to First Nations given EIS' performance issues. Since it is expected that system performance will continue to improve, First Nations' users will require appropriate training and user support services on the system. It is not clear how this will be achieved given the current resources available and the reliance on regions to provide this training, leveraging only their existing operational budgets.

Recommendation:

3. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector, in collaboration with the Senior Assistant Deputy Minister of Regional Operations should ensure that a sufficient and reasonable training plan is developed to support the provision of EIS training to First Nations. This plan should be supported by the resources and tools necessary to enable regional representatives to carry out their training responsibilities.

5.3 System adoption, performance and reporting

5.3.1 System adoption

The EIS system, in particular the Portal, was designed for First Nations to log in and submit education-related information to fulfill reporting requirements and run reports for their own monitoring purposes. The rate of adoption of the system is therefore a key indicator of EIS' success.

During the audit, it was noted that a limited number of First Nations were able to use the full functionality of EIS (i.e. leveraging the Portal to enter, review and submit education-related information for proposal and reporting requirements). A key reason for low adoption is system compatibility issues relative to the platform that EIS was built upon, Internet Explorer, as per the Government of Canada's requirement. EIS, the Portal and the ERAS reporting tool, have been built on and can only be accessible through the use of Internet Explorer. However, many First Nation recipients do not use Internet Explorer and as a result, these recipients cannot access EIS.

A solution that was provided to First Nations is the use of encrypted USB keys with attached smart PDFs. This has been established as an alternative solution for recipients who are in remote locations and have limited internet accessibility. This is a process whereby regions provide encrypted USB keys with attached smart PDFs to First Nation recipients who then complete the smart PDFs and save the information to the encrypted USB. The First Nation then mail the encrypted USB key back to the regional office and a regional representative uploads the PDF to EIS. The regional representative then completes the data validation process relative to the information provided by the First Nation.

The audit noted some performance issues with uploading PDFs to EIS. As further discussed in Section 5.3.2, this process can be time-consuming and system timeouts have been experienced due to the large size of some PDF files. As the regional representatives complete the data validation on behalf of the First Nation, warnings and error messages are sometimes noted which requires the regional representative to communicate with the recipients until the errors are resolved. This has resulted in an increase in workload for some regional representatives as well as delays in the overall process.

The compatibility issues experienced by many First Nations also affected their access to reporting capabilities within EIS and ERAS, which does not allow them to leverage existing reports that could help them make better informed, more timely decisions relative to education. As a key stakeholder for the system, it is important to have a relevant long-term vision on how First Nations are expected to interact and leverage the capabilities within the system.

Recommendation:

4. In conjunction with Recommendation #1, the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should re-establish and articulate the long-term vision and solution for the use of EIS/ERAS by First Nations.  If maintenance includes the use of smart PDFs, USB keys and limited access to reporting tools by First Nations, the associated functionality and processes should be established to ensure consistency and the most efficient use of the system as possible. If a longer-term vision for the use of EIS by First Nations is established, the necessary objectives, funding and associated activities should be articulated to enable this to be implemented.

5.3.2 System Performance

The audit included an assessment of the measures in place to enable adequate system performance of the EIS. As noted in Section 5.3.1, a limited number of First Nations are currently using the full functionality of EIS given compatibility and accessibility issues. The challenges with the performance of the system have affected many First Nations' ability to fully adopt the system. As a result, there was an increase in workload for some regional representatives as parallel tools and processes were maintained within the regions.

The use of electronic submission documents in the form of the smart PDFs sometimes results in large files being uploaded into EIS, particularly for Nominal Roll. The upload of these large PDF documents often resulted in system timeouts and errors. Large PDF files put significant strain on the system and can result in system failures.

System capacity is another factor that affected the performance of the EIS. If there is not sufficient capacity to maintain several users in EIS at the same time, the processing time within the system increases and the system can fail. The audit identified that the responsibility for the management of the infrastructure and environment lies with Shared Services Canada (SSC) and therefore, the Department must rely on SSC to track and monitor system capacity usage and logs. The information from the system capacity reports and logs is shared with IMB; however, the audit noted no evidence of formal, periodic monitoring of the capacity reports or logs. The audit also noted two instances of load testing that were performed over the capacity of EIS by IMB, and it was revealed that the system would consistently time-out when there was upwards of 42 users using the system simultaneously. If left unresolved, this will become a growing concern as the Department sees an increase in First Nations using EIS to submit their education-related information.

The audit also revealed that there were coding errors and defects that result from each new release of the system. In the EIS-AU phase alone, from April 2015 to March 2016, five releases are expected to be completed. Regional representatives noted that the new releases often resulted in unresolved coding errors and defects that further impeded the performance and hindered the successful use and adoption of the system.

As a result of these performance issues, regions continue to use parallel processes and tools (e.g. the use of excel spreadsheets and Crystal reports) to track and report on the collected education data from the recipients. This has created additional workload and duplication of effort for some regional staff, as they are responsible for ensuring the successful submission of education program data (in consultation with the recipients to resolve any errors and warnings resulting from the EIS data validation process), as well as maintaining separate systems to track the data already within EIS. However, overall, workload is still less than in previous years, as proposals are verified automatically and there are automated compliance reports that help reduce previous levels of work.

Recommendation:

5. The Chief Information Officer in collaboration with the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should ensure that system capacity reports are monitored on a periodic basis and documented evidence of this review is retained, and that ongoing efforts are made to improve the system capacity to allow more First Nations to use the system simultaneously.

6. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector, in collaboration with the Chief Information Officer, should ensure that coding errors and defects resulting from new releases are tracked and formally communicated to regions and recipients, along with workarounds and fixes when applicable.

5.3.3 Reporting

The audit assessed the timeliness and reliability of the reports from EIS and ERAS and the effectiveness of the reporting functionality of these systems in supporting performance measurement and timely decision-making.

The audit found that the EIS reporting tool, ERAS, has been developed with a number of general and custom reports available for both the Elementary/Secondary and PSE programs. It was noted that due to some data integrity issues with PSE data, it was not transferred to ERAS to enable the reporting capabilities for this program. As such, at the time of the audit, the Department was not yet able to generate reports, statistics or indicators for PSE from ERAS. Nominal Roll data is available in ERAS as of the year 2000, and as such, reports related to Nominal Roll are available to support departmental performance reporting for the Elementary/Secondary education programs. Education Branch representatives confirmed that PSE data was being reviewed and cleansed and would be available for reporting and analysis purposes in fiscal year 2016-2017.

Interviews with regional staff revealed that although they were provided training and access to use ERAS, the reporting tool was not yet being used in some regions. In addition to the lack of reports available for PSE , the Nominal Roll reports could only be generated from ERAS once all of the data from each region was been submitted for the fiscal year. The regions require real-time reporting data for decision-making purposes, and as a result, parallel tools and processes were still being leveraged by the regions for reporting and management decision making purposes. Also, given the accessibility challenges with EIS for First Nations, First Nations were not yet able to leverage the current reporting capabilities of the system.

5.4 Data integrity, security and privacy

To attain maintenance state, the EIS will need the required stability and security of the system, the integrity of the data as well as the effectiveness of controls in place relative to the interfaces with other departmental systems, including GCIMS and the Indian Registration System (IRS).

5.4.1 Data Integrity

A key function of EIS is the ability to validate the data against pre-defined business rules. These validation rules improve the integrity of the data within EIS, which in turn, allows for improved reporting and decision making. The audit found that adequate validation controls exist during the upload of data to EIS and in the interfaces with other systems to ensure the integrity of the data. When data is directly input into the system through the Portal, reliance is placed on the business validation rules built within EIS to ensure the accuracy and completeness of the data entered. In cases where data cannot be input directly into the Portal, the integrity of the data prior to the upload into EIS is ensured through the use of:

  • Smart PDFs, which contain mandatory field requirements, business rules and use dropdown menus with pre-defined values; and
  • XML files from a School Information System that have been tested against a test schema provided by the Department prior to submission of data.

Within EIS, there are several business validation rules that have been defined, documented and implemented by module for reports and proposals. Depending on the nature of the business rule, errors and warnings will result. Errors must be resolved before the data is submitted; however, the audit noted that the warnings will not prevent submission of the data. As an example, a key validation check prevents users from entering duplicate students into the system. This validation is performed against the student enrolment field, i.e. duplicate student enrolments within the same Nominal Roll for a given fiscal year are identified during the system validation and generate an error. In addition to the validation check, a duplicate report can also be generated to compare students against multiple Nominal Roll reports across regions.

Another example of the validation rules within EIS is the matching of the IRS number within EIS. Validation rules within EIS match the student information against the data in IRS. This includes a validation check against the IRS number, last name, date of birth and gender. Through the audit procedures, these validation rules were confirmed to be in place to demonstrate the existence of controls relative to the maintenance of data integrity in EIS and through the interfaces with other systems.

5.4.2 Data Security and Privacy

In accordance with the TB Operational Security Standard: Management of Information Technology Security (MITS), it is expected that systems are certified and accredited before they are approved for operation. Additionally, it has been established that the accreditation should be reviewed on a periodic basis in cases where the system has changed significantly or if there have been changes in the risk environment. The certification and accreditation verifies whether the established security requirements are met and that the controls and processes in place are operating effectively.

The original certification and accreditation of EIS in December 2012 resulted in EIS not meeting the mandatory security requirements, and was not approved for operational use.

A business decision was made to continue with the implementation of the system into production while working to mitigate the security concerns. In April 2014, EIS underwent another certification and accreditation process and the system was granted Interim Authority to Process. Interim Authority was granted as there were some outstanding issues surrounding deficiencies found during the architecture and code reviews and TRA. This Interim Authority expired in September 2014 and during the audit period there was no certification and accreditation for EIS despite the fact that that system continues to operate in production and ongoing changes were still being made to the system.

Given the sensitive nature of the data entered into EIS, the audit examined the segregation of production data from the development and test environments. Through examination of system screens and inquiry with representatives from HQ, it was determined that EIS is segregated into several environments (i.e. production, development, testing); however, production data is copied to these other environments on a periodic basis. This can be a concern as users in development and test environments without access to production, may have unnecessary access to the sensitive/private production data (such as student names, IRS number, date of birth and financial data). This access may not be in line with their job responsibilities or align to their current level of access to data in the production environment.

Recommendation:

7. The Chief Information Officer in collaboration with the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should ensure that the Certification and Accreditation process for EIS is completed to meet to the Government of Canada security requirements; and that the use of test data is considered for use within the development and test environments, rather than production data, to ensure sensitive data remains secure.

 

 

6. Management Action Plan

Recommendations Management Response / Actions Responsible Manager (Title) Planned Implementation Date (Month & Year) / Update/Rationale
1. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector jointly with the Chief Information Officer should ensure that a re-establishment of the long-term vision of EIS be undertaken to define the system's (upkeep and related business processes) maintenance as well as the outstanding activities that need to be completed to achieve this state. Once this has been defined, a formal transition plan should be established outlining how and when these activities will be completed and formal monitoring activities should take place to track the status of the achievement of the objectives.

EIS has already transitioned to application upkeep status.  This was done following the completion of the Effective Project Approval work in April 2014. An IMB maintenance team and Education Branch production support team were established at that time. EIS is operating under the proposed IMB application upkeep governance model.The application upkeep organization, roles and responsibilities, funding, maintenance processes, oversight mechanism and tools are outlined in the on-going support document that was shared with the audit team.

The actions established for this recommendation are done in conjunction with recommendation 4.

Director General of Education Branch  
Actions
  • Prepare a list of outstanding activities that were part of the original work package for this fiscal year and add items that arose out of normal application upkeep activities.

January 2016
Completed.

  • Report on the status of these activities to the appropriate application upkeep governance bodies.

January 2016
Completed.

  • Complete the identification and prioritization of application upkeep work for the 2016-2017 fiscal year.

January 2016
Completed.

  • Obtain approval and secure funding for the recommended work packages.

February 2016
Completed.

  • Set up a process to monitor progress against work packages.

April 2016
Completed.

2. In alignment with Recommendation #1, the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector jointly with the Chief Financial Officer should ensure that once maintenance objectives, targets and key activities are established, funding requirements for these key activities be determined, approved and formally tracked and reported to governance bodies in relation to the progress of these activities. It is the view of both IMB and Education that the Chief Financial Officer should be the lead on this recommendation supported by the ADM, ESDPP. Chief Financial Officer  
Actions
  • Obtain departmental approval for the Application Upkeep governance model proposed by IMB, recommended by ITSG/DGIOC and planned for discussion at Operations Committee in January 2016.

Q4 2015-2016
Revised to Q1 of 2016-2017
Completed.

  • Modify the application upkeep document to reflect the approved governance model and to integrate all IMB application upkeep services into a single plan/MOU for EIS.

Q4 2015-2016
Revised to Q2 of 2016-2017

  • Identify and approve the Application and database development and maintenance costing model and MOU template based on the departmentally approved MOU template and chargeback that will be defined through the IT Chargeback funding model for Application Development, Data and Database Administration (ADDDA) and Business Decision Support (BDS).

Q4 2015-2016
Revised to Q2 of 2016-2017

  • Determine and obtain approval and funding for the staffing levels in IMB, FTEs and consultants, necessary to undertake application upkeep activities in fiscal 2016-2017 and beyond.  This action item depends on decisions related to the implementation of the IMB Right Sizing exercise.

Q4 2015-2016
Revised to Q4 of 2016-2017

  • Determine and establish the tracking and reporting processes to Application Upkeep activities to effort and costs.

Q4 2015-2016
Revised to Q4 of 2016-2017

3. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector in collaboration with the Senior Assistant Deputy Minister of Regional Operations should ensure that a sufficient and reasonable training plan is developed to support the provision of EIS training to First Nations. This plan should be supported by the resources and tools necessary to enable regional representatives to carry out their training responsibilities.
  • Based on the re-established vision, determine the level of training required for First Nation users of the system in fiscal years 2017/18 and 2018/19.
Director General, Education & Regional Directors General

September 2016
Revised to December 2016

  • Determine the options for training and select the appropriate option for each region.  Options will include: types of training resources, locations, methods of training and types of training materials.

November 2016
Revised to February 2017

  • Secure funding for training options by region.

February 2017
Revised to March 2017

4. In conjunction with Recommendation #1, the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should re-establish and articulate the long-term vision and solution for the use of EIS/ERAS by First Nations.  If maintenance includes the use of smart PDFs, USB keys and limited access to reporting tools by First Nations, the associated functionality and processes should be established to ensure consistency and the most efficient use of the system as possible. If a longer-term vision for the use of EIS by First Nations is established, the necessary objectives, funding and associated activities should be articulated to enable this to be implemented. 
  • Re-establish the long-term vision for the use of EIS/ERAS by First Nations and establish a plan on implementing this vision and solution

Director General, Education
in collaboration with Regional Directors General and the CIO

September 2016
Revised to December 2016

5. The Chief Information Officer in collaboration with the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should ensure that system capacity reports are monitored on a periodic basis and documented evidence of this review is retained, and that ongoing efforts are made to improve the system capacity to allow more First Nations to use the system simultaneously.
  • Re-perform and assess the EIS system capacity load test completed in 2014.
Chief Information Officer

January 2016
Completed.

  • Determine system performance and stability enhancements for both the short and long term.

February 2016
Completed.

  • Establish a plan and secure the funding to implement the short term enhancements.

March 2016
Completed.

  • Establish a plan to implement the long term enhancements as appropriate
September 2016
  • Install second 64 bit server requested in 2014-2015.

March 2016
Revised to Q2 2016-2017

  • Establish process with Services Canada to monitor capacity and system performance and to take remedial action as necessary.

March 2016
Revised to Q3 2016-2017

  • Implement sufficient secure "drop box" capability to allow education and other departmental programs to share sensitive information with recipients.  This would replace the use of encrypted USB keys

March 2016
Revised to Q1 2016-2017
Completed.

  • Implement the capability within the department to support the use of multiple browsers including Safari, Chrome, and Firefox.

July 2016
Revised to Q4 2016-2017

6. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships, in collaboration with the Chief Information Officer, should ensure that coding errors and defects resulting from new releases are tracked and formally communicated to regions and recipients, along with workarounds and fixes when applicable. Evidence was shared with the audit team that the tracking and communication of coding errors and defects is being carried out. This is done utilizing the Test Track Pro and Remedy tools and through the release notes that were formalized this fiscal year. It is acknowledged that the timeliness of the communication to the regions needs improvement.    
Actions
  • Ensure that a process is in place to monitor and track coding errors and defects that are scheduled for implementation within new releases and hot patches.
Chief Information Officer

January 2016
Completed.

  • Establish and operationalize the process to improve communications on new system releases to regions and recipients.
Director General of Education

January 2016
Completed.

7. The Chief Information Officer in collaboration with the Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector should ensure that the Certification and Accreditation process for EIS is completed to meet to the Government of Canada security requirements; and that the use of test data is considered for use within the development and test environments, rather than production data, to ensure sensitive data remains secure.
  • Ensure that EIS has a valid security authority to operate.
Chief Information Officer

March 2016
Revised to Q2 2016-2017

  • Determine an appropriate approach to develop test data.

March 2016
Revised to October 2016

  • Secure funding and undertake to replace production data with test data in the test environments.

September 2016
Completed.

  • Complete the Certification and Accreditation for EIS.

September 2016

 

 

Appendix A: Audit Criteria

Audit Criteria

Governance, Transition and Risk Management
1. A formal governance structure has been established for the EIS steady state that includes defined roles, responsibilities and accountabilities and formal risk management processes.
Stakeholder Engagement, Change Management and Training
2. Stakeholder and IT change management practices have been established and are working effectively to support an effective transition of EIS into a steady state.
EIS Business Processes and Associated Controls
3. Business processes have been appropriately updated to enable and reflect the effective use of EIS as well as regional differences and needs.
Reporting Functionality of the System
4. Reports generated by EIS/ERAS are timely and reliable and support performance measurement and timely decision-making as EIS transitions to a steady state.
Data Integrity
5. Data integrity is maintained in EIS and through interface with other systems (i.e. GCIMS, IRS, ILRS and CIDM).
6. Appropriate measures are in place to enable adequate system performance and data backup as the system transitions to a steady state.
Data Security and Privacy
Data security and privacy controls within EIS are designed appropriately and working effectively.
 
 
Date modified: