Why wait to address data centre operations? Start with the end in mind

IN UPTIME INSTITUTE’S EXPERIENCE, too many companies are forced to make significant changes to their data centres around the world within their first year of operation.

Even if the changes do not require rebuilding infrastructure, the costs can still be enormous, both in terms of rework and ongoing loss of efficiency. Most of these costs could be avoided if Operations is asked to provide critical feedback on facility maintainability during the design phase.
Often the changes seem simple but can be costly to correct, requiring major renovation or new construction.

This is because too many owners and builders don’t take the long view and instead focus on first or initial costs. The majority of these problems could be avoided if enterprises could adopt a maintenance focus.
Some avoidable problems that Uptime Institute has seen include:
Lack of space to perform preventive maintenance activities
Insufficient space for cable tray for communications
Battery racks loaded three high where it is difficult (and dangerous)
to monitor and maintain those battery strings
Door heights not coordinated so staff will have to lay down
equipment on its side to get it through the building
Not enough swing space allowed to move equipment in or out for
replacement/installs
Gauges or valves located at heights requiring a ladder

It’s easy to see how mistakes like these can happen in the rush to reduce construction costs and bring a data centre online, but the solution is equally obvious: Include operations staff at the outset of a data centre project.
Having an operations focus can:
Increase return on investment
Increase uptime
Increase data centre efficiency
Reduce costs
Reduce risks

Even though operations staff live in the data centre every day and are tasked with keeping it on-line, it is surprising how often data centre designers and owners fail to tap them as a resource, Operations should be considered the customer by the design and build teams. Only the operations team will bring the perspective that is required to make the best long-term decisions when faced with various options in the building of a data centre.
This new mindset has immense ramifications for data centre owners in Europe, the Middle East, and Africa (EMEA), but each region starts from a different level of maturity.

In more mature markets, the operations focus can immediately yield improvements in data centre design. In other markets, bringing operations in at the beginning can yield improvements in SOPs, MOPs, and EOPs so a data centre can operate at its intended Tier or reliability objective This competency has to be built and starting early to identify and train data centre operations staff is key.

In one example, Madrid-based telecommunications company Telefonica started planning its Alcala data centre in Madrid, Spain, by focusing on operations from the beginning of the project. Telefonica formulated a unique RFQ to contract for a turnkey Tier IV data centre and 15 years of operations in a unique and single contract, which meant that the winning bidder would have design, construction, and operations responsibilities.
“Telefonica established a complete requirement list and well defined layout for the data centre, mandating a Tier IV Certificate of Constructed Facility, an SLA that included all-risk maintenance on equipment, and guaranteed Power Usage Effectiveness (PUE) performance,” said Rafael Serrano, Senior Consultant at Uptime Institute. Serrano had been Global Data Centre Manager for Telefonica at the time of the project.

“Telefonica performed quite well on its Uptime Institute Pre-Operational Sustainability Certification in August 2013, just eight months after hand over to operations. A few adjustments are still needed for attaining the Tier Certification of Operational Sustainability (TCOS), but most of the critical behaviors were already in place in August,” Serrano said.

Based on its field experience and client engagements, Uptime Institute has documented the kinds of contributions operations can make in the early phases of design and even created a roadmap for organizations to follow. The roadmap highlights where Uptime Institute Tier Certifications punctuate the capital project timeline.

The roadmap shakes up the entire traditional data centre design-build process. For example, procurement departments need to change how they buy the services of data centre design and build contractors. Specifically, the RFPs should require that the winning bidders will ensure the following:
Operations must signoff on the design
Value engineering requires operations signoff
Operations program development will be conducted during
construction and all manufacturers documentation must be
provided prior to the start of construction
Procedure testing and& operations training will be conducted
during commissioning
Operations acceptance criteria must be met to turnover and
achieve substantial completion

Roadmap to Success: PRECONSTRUCTION
Enterprises can take comfort in knowing that operations experience can help inform their entire data centre strategy, not just the design and construction phases. Although the C-level executives are ultimately responsible for making the business case for an IT project, contributions from operations can bend the cost curve of operating a data centre or change TCO calculations which can enable the enterprise to defer construction, reduce footprint, or increase speed to deploy.

In particular, operations can help define measures of success, SLAs, and KPIs and determine vendor requirements, including consultants.
The potential for Operations contributions may be greatest during the design phase. Many decisions are made during the design phase that will directly affect operations and TCO after turnover and transition. At this phase Maintenance and Operations can conduct a design review that will provide invaluable insight into:
Support and specialty space
Security, access, and setbacks
Potential for incremental capacity increases
Ease of maintenance
Design concurrence
Maintenance and Operations planning can begin during the design phase too, so the enterprise can define and establish
Vendor/contractor SLAs and support contracts
Staffing levels and shift strategy
Staff qualifications and assess capabilities
Equipment maintenance plans
Operations standards consistent with site mission, reliability and
availability requirements, and industry best practices
Establish asset life-cycle analysis program

Roadmap to Success: CONSTRUCTION
Uptime Institute has identified many other areas during the construction phase where an operations focus benefitted the enterprise. These include:
Management and Operations Program Development
Develop operations procedures (SOPs, MOPs, and EOPs)
Implement systems and processes (MMS and other key operating
systems, document repository)
Develop training program
Establish minimum shift and daily inspection protocols
Develop weekly/monthly walk-through equipment checklists
Establish monitoring and controls systems reports
Develop escalation policies and protocols including contact lists
(addressing increasing levels of severity including alerts, events,
and incidents
Establish inventory of critical spare parts and consumables
Develop housekeeping policy and Critical Environment work rules
Develop Critical Environment work approval and change
management processes (normal, expedited, and emergency)
Develop Critical Environment work approval procedures and forms
Establish risk windows and allowable activities
Develop predictive maintenance program

Roadmap to Success: COMMISSIONING
My colleague, Lee Kirby, Uptime Institute’s Chief Technical Officer is surprised that more organizations do not take an operations focus during commissioning. “I think it is just commonsense that operations staff should be onsite during commissioning and get hands on as much as possible,” he said, noting that doing so will be good preparation for day one and after, when glitches can lead to downtime events or worse.

Roadmap to Success: TURNOVER
When it’s time to turn over the site to the owner, at a minimum, operations should
Review commissioning results and prioritize the punch list
Conduct operational readiness assessment
Implement operations management program
Refine operations procedures (SOPs, MOPs, and EOPs)
Exercise all procedures to ensure optimal effectiveness
Few organizations want to admit to problems in a brand new data
centre, but issues are far too common in new data centres.

Starting training during design and procurement reduces the learning curve and improves operations and security coordination training.
However, in order to effectively prepare for turnover in this manner, RFPs must require that the operations program is ready for Day 1 production with key contract points:
Operations design review and signoff
Operations program development during construction
Procedure testing and operations training during Cx
Validate with operational readiness assessment prior to turnover

However well designed a data centre appears on paper, ultimately its success stands or falls on day-to-day operations over the lifespan of the facility. Successful operations over the long term begins with having - at every stage of the planning and development process — an Operations advocate who deeply understands the integrated mesh of systems, technology, and human activity in a data centre.