720-913-5000 auditor@denvergov.org

Audit Report

Data Centers

This audit evaluated the two data centers operated by the City and County of Denver’s Technology Services agency and the two data centers serving Denver International Airport. The data centers were assessed for efficiency and effectiveness of operations against applicable standards to ensure the security and the operability of technology assets, such as equipment, facilities, and infrastructure.

Watch the Audit Committee presentation here when available.

Data centers house critical information technology (IT) assets, including servers, routers, and other devices. Technology Services and the airport are each responsible for a primary and a secondary data center. These four data centers use a mix of cloud-based and in-house technology solutions.

Technology Services provides all IT-related infrastructure and services to the City, including support for the data centers audited in this report. The General Services Department provides facility support, including maintenance and operation of City facilities that support Technology Services’ data centers. Denver International Airport’s Business Technologies Division is responsible for the IT infrastructure and services used by the airport, including the data centers audited.

Data Center Infrastructure Cannot Easily Adapt to Changes in Operations

  • Environmental controls at one City data center do not provide consistent temperature and humidity for the data center to function at optimal performance.
  • The existing data centers do not provide sufficient geographic diversity.
  • Technology Services cannot provide a complete and accurate inventory of data center hardware, applications, databases, and other equipment.

Data Center Operations Lack Application Reviews, Cost Data, and Collaboration between Technology Services and Business Technologies

  • No process is in place to review and approve the continuing use of software applications running in the data centers to ensure they still provide appropriate value.
  • The Technology Services agency and the Business Technologies Division do not track the costs of application services, making it impossible to truly determine the cost of providing or changing these services.
  • The Technology Services agency and the Business Technologies Division do not share their data center management tools and knowledge, so they miss out on opportunities for cost-savings and efficiency.

Technology Services and the Airport Should Employ Consistent Operational Standards at All Data Center Locations.

  • Technology Services has not adopted comprehensive data center operations and control frameworks to ensure consistent operations.
  • The City’s General Services Department operates without complete policies and procedures to provide facility services at Technology Services’ data centers.
  • Employees sometimes make unauthorized changes in Technology Services’ data centers.
    The continuity of operations plan for the airport data centers is outdated.

1.1 Relocate Data Center (Technology Services) – The Technology Services agency should complete the replacement of its secondary data center with a suitable location.

Agency Response: Agree, Implementation Date – December 2019

1.2 Address Internet Service Solution (Business Technologies) – Denver International Airport should continue implementing a more robust internet service solution as soon as possible.

Agency Response: Agree, Implementation Date – December 31, 2020

1.3 Assess Secondary Airport Data Center (Business Technologies) – After implementing Recommendation 1.2, Denver International Airport should perform an evaluation to determine if its secondary data center should be relocated or augmented.

Agency Response: Agree, Implementation Date – April 1, 2021

1.4 Develop Data Center Inventory Report (Technology Services) – The Technology Services agency should develop, as soon as possible, a readily available, easy-to-run report that provides a complete and accurate inventory of all data center hardware, software applications, databases, and underlying equipment.

Agency Response: Agree, Implementation Date – June 2019

1.5 Perform Periodic Review of Inventory (Technology Services) – The Technology Services agency should develop as soon as possible a process to periodically assess the completeness and accuracy of data center inventory.

Agency Response: Agree, Implementation Date – June 2019

2.1 Perform Periodic Application Review (Technology Services) – Working with the Mayor’s Office, the Technology Services agency should enhance their existing process, as soon as possible, for reviewing the business justification and appropriateness of software applications. This review process should occur at least every three to five years, include all applications, and be done in consultation with application owners.

Agency Response: Agree, Implementation Date – December 2019

2.2 Perform Periodic Application Review (Business Technologies) – The Denver International Airport’s Business Technologies Division should enhance their existing process, as soon as possible, for reviewing the business justification and appropriateness of software applications. This review process should occur at least every three to five years, include all applications, and be done in consultation with application owners.

Agency Response: Agree, Implementation Date – September 1, 2019

2.3 Track Data Center Costs (Technology Services) – The Technology Services agency should track costs associated with providing data center support. These tracked costs should include:
-Personnel costs – including salaries and benefits;
-Building costs – including utilities, insurance and other direct facility costs;
-IT infrastructure costs – including hardware and supporting software, such as operating systems costs; and
Application-specific costs – including licensing, specific hardware needs, and maintenance costs.

Agency Response: Agree, Implementation Date – December 2019

2.4 Track Data Center Costs (Business Technologies) – The Denver International Airport’s Business Technologies Division should track costs associated with providing data center support. These tracked costs should include:
-Personnel costs – including salaries and benefits;
-Building costs – including utilities, insurance and other direct facility costs;
-IT infrastructure costs – including hardware and supporting software, such as operating systems costs; and
Application-specific costs – including licensing, specific hardware needs, and maintenance costs.

Agency Response: Agree, Implementation Date – May 1, 2019

2.5 Collaborate to Share Tools and Knowledge – The City’s Technology Services agency and the airport’s Business Technologies Division should share knowledge, tools, and the technology they use to manage their data centers. These combined tools should provide the equivalent of a data center infrastructure management tool set.

Agency Response: Agree, Implementation Date – May 1, 2019 (Business Technologies);
June 2019 (Technology Services)

2.6 Track Energy Use (Technology Services) – The City’s Technology Services agency should set up a process as soon as possible to track, at a minimum, total energy costs for each data center. In the long term, the agency should perform an analysis of energy costs down to the device-level starting in 2020.

Agency Response: Agree, Implementation Date – December 2019

2.7 Track Energy Use (Business Technologies) – The Denver International Airport’s Business Technologies Division should set up a process as soon as possible to track, at a minimum, total energy costs for each data center. In the long term, the division should perform an analysis of energy costs down to the device-level for each data center starting in 2020.

Agency Response: Agree, Implementation Date – May 1, 2019

3.1 Adopt Data Center Standards (Technology Services) – The Technology Services agency should adopt, as soon as possible, a set of comprehensive data center operating standards, such as the Telecommunications Industry Association’s standard for data center quality, called “TIA 942.”

Agency Response: Agree, Implementation Date – December 2019

3.2 Update Existing Policies – The Technology Services agency should work with the General Services Department to update, as soon as possible, the existing IT asset management policy related to problem management and maintenance. This updated policy should be based on the National Institute of Standards and Technology’s standard 800-53 and should address environmental and security control considerations, periodic testing, and problem resolution.

Agency Response: Agree, Implementation Date – December 2019

3.3 Create Procedures – After implementing Recommendation 3.2, the Technology Services agency should work with the General Services Department to create procedures related to data center maintenance and problem management.

Agency Response: Agree, Implementation Date – December 2019

3.4 Prevent and Detect Unauthorized Changes to Systems (Technology Services) – As soon as possible, leadership of the Technology Services agency should issue a firm directive to agency personnel that no unauthorized changes to systems occur.

Agency Response: Agree, Implementation Date – June 2019

3.5 Update Continuity of Operations Plans (Business Technologies) – The Denver International Airport’s Business Technologies Division should work with Denver International Airport’s COOP (continuity of operations planning) administrator to create an annual process to review and approve continuity of operations plans for its data centers.

Agency Response: Agree, Implementation Date – May 1, 2019

Follow-up report

A follow-up report is forthcoming. 

Other related reports

None at this time.

Audit Team: Kevin Sear, Karin Doughty, Nick Jimroglou, Brian Cheli

Subscribe To Our Newsletter

Stay updated once a month with news from our office and Auditor O'Brien

Thanks for your interest in the auditor's office!