Customer Connections + Corporate Services

Critical Infrastructure Protection: Lessons Learned From a Unique NERC Pilot Project

For those who have never heard of CIP V5, and just see a fancy acronym reminiscent of a high-powered car with an odd number of pistons, it represents new standards designed to keep our electric system out of the hands of the bad guys —hackers and intruders.

CIPV5 stands for the fifth version of the North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Standards. If you own bulk electric system (BES), transmission or generation assets, you should have heard about it by now. If you have facilities connected to the grid, you also are subject to it. Not only that, you will be audited for compliance by the Western Electricity Coordinating Council (WECC) and NERC after they become mandatory on April 1,2016.

Version 5 introduces a radical expansion of the CIP standards to ensure that the rules now encompass everything related to operating the BES. The BES, in general, is comprised of transmission lines, 100kV and above, and most generators. The possible exceptions are things that can be proven to have no impact. Fully discussing exceptions and inclusions would take up all the issues of this magazine for the next year.

The Sacramento Municipal Utility District (SMUD) participated in a unique NERC CIP V5 pilot program, where we learned a lot of lessons that could help other utilities and grid operators.

In September 2013, SMUD joined a group of six utilities to test drive implementing CIPV5 standards with NERC regulators. The intent was to help NERC workout the ambiguities and bugs in interpreting and applying these new rules in a live, utility setting. The Balancing Authority of Northern California (BANC) also was included in the pilot because SMUD is BANC’S delegated operator.

NERC chose SMUD because of the utility’s strong history of compliance with NERC Standards. In addition, SMUD was willing to share challenges and solutions with NERC during the 18-month pilot.

While we can’t say that all ambiguities with Version 5 Standards have been clarified, we certainly did find a lot of them and solved many (though only an audit will tell for certain).

We now have fully implemented CIP V5 at SMUD. NERC is still working to provide guidance for entities at various industry forums, and SMUD is helping where it can.

SMUD discovered that there are many ways to protect Critical Infrastructure Assets. What is important is determining if the method will pass an auditor’s review. Of course, regulators and auditors reserve the right to change their minds, develop new interpretations, and audit different methods, as the intent of the language is better understood.

No matter how many lawyers and engineers write standards and requirements, their real-world application often need tweaking.

NERC and FERC have an important mandate to protect the BES from physical and cyber intrusions; not to mention the takeover of our transmission and generation assets by the bad guys. No matter how many lawyers and engineers write standards and requirements, their real-world application often need tweaking. Ambiguity is nothing new when applying any NERC standards to an operational situation. However; to NERC’s credit, resolving complex standards applications is the key reason why it conducted the pilot program. It also is why work has begun on CIP Version 6 before CIP V5 goes into effect.

WHAT DID SMUD LEARN? What does a utility executive, manager or board member need to know?

To implement CIP V5, SMUD’s first challenge was to figure out which facilities were subject to the revamped, spruced up rules. Under old CIP standards, an electric utility alone decided what was or wasn’t critical to the BES. The goal was to protect only those few, super-critical assets from hackers and intruders. That was done by documenting that the utility met all of the old CIP standards.

Most of the new CIP standards are the same as the old. The big difference now is that all of the “transmission-level assets” will be covered by various CIP rules, depending if a particular asset is declared to be a “high, medium, or low-impact facility.” A host of new rules determine which of an entity’s transmission assets, if any, are “connected to” the newly defined BES. It doesn’t sound simple, and it isn’t.

WINTER16_BT 18

Each NERC rule has processes that each organization’s technical staff will have to evaluate and work through. We found the exercise to be more challenging than we expected — especially when it came to reaching consensus among our internal experts on classifying cyber devices and assets determined to be “high, medium or low-impact to the BES.” These classifications can make a big difference in what entities are required to do to protect the BES, and what penalties might be levied for not adhering to the rules.

One key lesson was learned while identifying the impact of our assets.

Our Distribution Control Center (DCC) became a high-impact facility because it had possible connectivity to our transmission operations computers when performing certain distribution functions (via SCADA). We didn’t see that one coming.

During the pilot study, we advocated to the NERC regulators that our DCC must be a low-impact facility, based on our reading of NERC rules. We argued that ours was only a small, inconsequential distribution control center that only handled SMUD’s residential and commercial customers. It was designed to be low impact and it could not possibly have an impact on the BES or national security. We believe we made good arguments, technically and by interpretation.

However, NERC did not budge from its interpretation of the applicable standards and made it clear that we would be in violation if we stood by our interpretation. NERC’s logic is that, while actual connectivity is not apparent, if there is the “capability of any potential connectivity” to equipment or software that involves control of the BES, then the utility’s distribution control center can have a “high impact” on the BES, and it must be identified as such. That means substantially more rigorous access controls, training and documentation.

This was a big lesson learned. We wish we could have anticipated or had been informed of this interpretation before we designed and built the new building that housed the DCC in 2014. As a result, our new building required updating to meet CI P V5, just after we occupied it. Fortunately, we were in a pilot program and could correct this otherwise-invisible security hole, instead of having it discovered during an audit.

When it comes to identifying critical 3 assets under CIP V5, we suggest that 1 organizations go through their internal identification process, and that they should consult with the WECC or NERC staff who are the assigned experts on these tasks, and to take their interpretation to heart.

Substation Access Controls Are Paramount

A second big lesson learned involved access to substation control buildings, which were deemed medium-impact facilities, subject to CIP V5 rules. It concerned our substations that for the last 60 years, were accessible to our crews and contractors without much effort, other than using a key. They just went in and did what was needed to keep the lights on. Under new CI PV5, crews now must:

  • have individual background checks and be fingerprinted,
  • complete extensive annual CIP training,
  • have personal risk assessments performed,
  • document when they enter a substation control building, and
  • understand who they can take with them into a substation control building.

This rule applies to substations identified as “medium impact,” not just those identified as “high impact.” The crews received extensive training and signed off that they knew the new rules. We thought we were good to go.

It turns out that changing human habits, not to mention communicating that our little old substations are now at the level of national security, was not that simple. Under the new rules, every movement in and out of these substation control buildings must be tracked and logged. That just did not sink in with everyone on the first pass.

Some crew members forgot their training, so we designed and built in reminders. We added tailgate alarms that buzz if two people walk in and only one had badged in. More alarms were put on control room doors. Big signs stating the new rules and processes were posted on entry doors. Keypads now have codes that need to be entered for each individual. We removed key locks and installed card readers to track entry and exit; and we regularly monitor security camera footage. Still, some folks were baffled by this “hassling.” It took some time, but it seems to have finally worked, although we’re still working on ways to simplify the process and make it obvious to all. Old habits are much harder to break than a logical set of new, CIP V5 rules would lead you to believe

Cybersecurity Upgrades

Another challenge we faced was that the new, CIP rules increased cyber security in the communications between the high- and medium-impact facilities that were identified in the pilot. SMUD installed new cyber controls to ensure that only authorized data is sent and received in all of the cyber communication coursing back and forth between our facilities’ devices. And now, we have to document that we are performing new protective measures at all times.

Firewalls that allow or block data were required at substations in order to protect them from cyber at tacks. Other interactive, remote access rules and jump host devices were needed for talking across electronic security perimeters. Just describing what those are and what these terms mean would take volumes (and it has). The technical part was challenging enough. Lots of expensive, time consuming tests were needed. Because reliability was paramount, backup plans in case devices failed were added.

A Coordinated, Multidisciplined Approach

A lesson to share involves responsibility. Figuring out which internal group owned and maintained the new, substation firewalls became key. First, there is the group who installed and maintained the firewalls. Then, we had to determine who was responsible for documenting the new tasks and performing continued maintenance. These tasks now involve several business units and skill sets, which was a change from previously, well de fined processes and responsibilities.

There are a dozen ways to approach this ownership dilemma, but we learned that it needs executive-level support. Do not assume that one group will step up and handle it all. Too many disciplines are involved in such a complex area. Some of these disciplines typically had not interfaced with others as part of their day-to-day work.

WINTER16_BT 20

We chose to set up a team with representatives across all business units. Getting executives on the same page on responsibilities is very important. Also, we empowered a dedicated, CIP pilot project manager to oversee that issues were hammered out.

Regular CIP project steering team meetings were essential to make decisions, and to resolve issues and disputes. With the aforementioned executive support, the team had representatives from telecommunications, operations, security, information technology, facilities, reliability compliance, legal, and even the budget office.

The Bottom Line

More than 10,000 person hours and $5 million later, SMUD has succeeded in the implementation of CIP V5 Standards and Requirements. SMUD’s participation in the CIP V5 Pilot Project was worth all the time and effort. We learned a lot from the NERC experts involved, and they took away first-hand experience in how their new regulations will work in real, operational situations. That is the best way to develop and implement regulations whose goal is a reliable BES.

NERC staff has applauded our efforts publicly — but, of course, only our compliance audit will tell. That comes next March. Wish us luck.


SMUD authors James Leigh-Kendall, Director of Internal Compliance and Michael Gianunzio,Chief Legislative and Regulatory Officer.

Michael Gianunzio is SMUD’s chief legislative and regulatory officer. He manages all governmental, legislative and regulatory matters affecting SMUD at the local, state and federal levels. Gianunzio also serves as SMUD’s reliability compliance officer, responsible for the implementation and compliance with all FERC/NERC reliability compliance standards and requirements. Prior to joining SMUD in 2008, Gianunzio was general counsel and chief legal officer of the Snohomish County Public Utility District. He received national and international recognition for his leadership role in uncovering tape recordings and documents of fraud and market manipulation by Enron power traders.

James Leigh-Kendall is SMUD’s director of internal compliance, dealing with all aspects of the compliance monitoring and enforcement programs administered by FERC, NERC and WECC. A registered professional engineer, he has over 25 years of experience with electric utility systems. He also serves as the compliance officer for the Balancing Authority of Northern California. Since joining SMUD in 1989, he has been responsible for ensuring that reliability criteria are being met by developing various transmission project plans and programs. Prior, he was at Western Area Power Administration offices under the Department of Energy as a transmission planning engineer.

 

About the Author

Western Energy magazine
Western Energy® is the voice of Western Energy Institute's 800+ volunteers and members, providing the latest in-depth news, best practices, solutions and conversations about pressing business and operational challenges facing electric and gas industries serving the Western United States and Canada. This quarterly publication is distributed to more than 6,500 decision-makers of public and private companies involved in electric generation and transmission, gas and electric distribution, pipeline operations, power marketing and energy-related services