It has been 16 years since Presidential Directive 63 (PDD-63) was released. The directive broadened the definition of critical infrastructure and defined what systems were “essential to the minimum operations of the economy and government”, and ultimately called for public-private partnerships to “swiftly eliminate any significant vulnerability to both physical and cyber-attacks on our critical infrastructures”. PDD-63 eventually led to the creation of the North American Electric Reliability Corporation Critical Infrastructure Protection (NERC CIP) program.
In the last 16 years the critical infrastructure security community has experienced changes including:
- The 9/11 attacks
- New regulations, such as NERC CIP, have entered the picture
- Mainstream automation system vendors have developed mitigating solutions
- Numerous start-up companies that have succeeded and many that have failed.
A few general questions to ponder: Have we made progress towards securing critical infrastructure? What’s worked and what hasn’t? Can we call the entire endeavor a success or failure?
With over 14 years’ experience securing critical infrastructure starting with pen-testing to researching to consulting and then to building products in the space, we are in a unique position to provide insight to these questions.
The Heritage Foundation recently released a paper titled “Plotting a More Confident Course: Rethinking Oversight of the Electric Sector and Critical Infrastructure Cybersecurity” which asks policy makers and the utility industry to “rethink” the oversight of the electric sector. While The Heritage Foundation asks us to rethink the oversight, we propose a very different approach: stay the course. Critical Infrastructure is a long way from utopia; however, we think we’ve made significant progress considering the constraints that come along with the critical infrastructure industries.
Sixteen Years of CIP in Review
Since PDD-63 was issued, the utility environment has certainly not grown safer. To the contrary, we have seen a sea change in the risk trends over the past 16 years. For example, very few mainstream security professionals knew much about the Modbus and DNP3 protocols—common in the utility control environment. Today we see quite a few BlackHat security talks on these topics. We have also witnessed:
- The appearance of zero day attacks targeting ICS (industrial control systems)
- Attacks directly targeting OT (operations technology) vendors and applications
- Readily available tools designed to infiltrate ICS
These trends are indicative of increasingly sophisticated threats targeting the supply chain accompanied by an increasing arsenal of easily available tools which warrant that the utility industry “must not slow down” on security.
Let’s look at what we have learned over the past 16 years about improving cyber security of the power grid and how we should apply these lessons to continual improvement, primarily:
- Regulations have helped the security posture of utility industry, even if it’s the minimum requirements
- Security solutions must make economic sense
- OT Awareness drives effective utility security, compliance and operations
- Change management and automation are essential
Regulations have helped the security posture of utility industry
Regulations in critical infrastructure industries, particularly the NERC CIP regulations for electric utilities, have helped increase the security posture of their systems. For example, back in 2000, it was very hard to push the concept of a Firewall between the control system network and the corporate network. Today, industrial protocol-aware firewalls residing inside the control network (below level 3) are common. Not too long ago, vendors sold products with little to no security controls built into products used in ICS environments. Now cybersecurity is a standard section in RFPs with line items reflecting NERC CIP requirements.
Over the long term, critical infrastructure and utility industries should control their own fate flying solo without government regulation. However, the wings of various industries—especially utility industry—need to be stronger for them to fly far. NERC CIP has been effective, but has not yet widely percolated into every aspect of the industry because of several loop holes in CIP version 3. CIP version 5 needs to be in place for a few years before the utility industry can control their fate in cyber security.
Security solutions must make economic sense
The most effective way to deploy and maintain security in a control system, according to an operator’s mindset, is to clearly communicate the operational value of the security products. Improving availability and operational efficiency is the key to justifying a security solution to the industry. For example, we could not deploy VPNs into control system environments until we showed the business value of remote monitoring of a control system environment. Today, control system professionals are able to remotely diagnose and correct problems, reducing the need for onsite personnel deployments and improving uptime.
While understanding the operational value is fundamental to making economic sense of a security solution, it is not enough to make the whole economic case. This is because, like so many other industries, the utility industry is going through a period of resource crunch. Capital budgets are getting slashed and vendors cannot sell security for security’s sake. If, as with point solutions, a product consumes excessive maintenance and management resources, or demands developer-like experience to write the rules used by the tool, these costs will quickly overwhelm the positive operational and security side of the economic equation. Thus, operational value must be accompanied by ease of use and maintenance. Solutions now must be easy to use, require little management, and provide actionable intelligence instead of raw data that needs to be analyzed.
OT Awareness drives effective utility security, compliance and operations
Major IT and IT-security companies have sensed the opportunity in the critical infrastructure industries and have deluged the market with off-the-shelf tools in spite of the fact that they lack control system experience, awareness and specificity. The utility operators have bought into the marketing themes and sales pitches, and have procured some of these IT tools over the past few years. While the ICS professionals need to observe other industries and learn from them, we have heard horror stories from the ICS operators struggling to deploy IT technologies into the OT space. We have seen:
- IT technologies causing significant downtime to ICS applications
- Excessively long deployment times and learning curves for IT technologies
- IT technology support staff lacking the intimate experience with control systems that their ICS customers would naturally expect of a technology supplier
In fact, once sold and deployed, those tools have become major impediments to the progress of securing the ICS infrastructure systems. In many cases, these tools have proven:
- Too numerous. The ICS operators are struggling with dozens of IT tools such as SIEM tools, compliance tools, reporting tools, documentation tools, ticketing tools, IDS tools, firewall management tools, performance management tools, and more.
- Too big. Each of the tools is too big for their needs and too bulky to maintain. ICS operators are desperately looking for a simple, custom technology which would perform only the needed functionality.
The technology they need must have intelligence around the control system environment, must be respectful of their applications, and must help—not impede—operations. Control system situational awareness is about bringing all the aspects of the ICS environment including events, configurations, policies, and reporting together into a single, actionable view.
Traditional technologies such as Firewalls, VPNs and SIEM tools are good, but they lack the ICS situational awareness and context of the OT applications. Understanding OT applications and their “normal vs. abnormal” context is essential for technologies that would secure control systems against increasingly sophisticated attacks while simultaneously supporting operational goals.
Change management and automation are essential
We have also found that security solutions not specifically designed for a control system environment lack the understanding of control system priorities necessary for effective change management. Fundamentally, a cyber attack changes aspects of the ICS environment such as modifying software, adding users, reconfiguring assets, manipulating files, and so forth. Effective change management and effective cyber security must go hand-in-hand. Beyond that, the solution should also reflect the context of change: the policies and regulations that changes must comply with.
We are well aware that managing change, security and compliance across an ICS environment is a daunting task, so ease of deployment, ease of use, and ease of maintenance are essential for a security solution. Traditional IT tools are not only expensive and hard to deploy in a control environment, but they often require developer-like experience to write the rules used by the tools. Thus, automation of these tasks are key: automating security event and change management monitoring and alerting with popular rules available out-of-the-box; automated CIP asset monitoring, management and reporting; and easy-to-use, intuitive policy compliance assessment and reporting features.
Security in Utilities Organization
This is an area where we have not seen as much progress as we would like. Today, the individual responsible for security in a utility is no higher up in the organization than 16 years ago. The role of a CSO in Internet companies such as Microsoft, Amazon, Google, and Facebook sits much higher in the organizational hierarchy and possesses greater influence than in critical infrastructure companies. While we have seen a few companies in which a dedicated CISO reports directly into the CEO’s inner circle, they number less than a dozen.
This represents a major impediment because of a lack of visibility. That is, without high-level visibility, it becomes much more difficult to obtain the resources required to significantly improve the security posture. Furthermore, without high-level visibility it is very hard to apply the leverage necessary to break the IT-OT barriers.
We are also seeing a lot of turnover in security staff. We understand that this is largely attributable to the burden of multiple security tools and to the comparatively weak security organization, both discussed earlier.
In keeping with their stated mission of formulating and advocating policies based on limited government, The Heritage Foundation proposes rethinking the oversight of critical infrastructure cybersecurity with a shift in focus to industry itself and state power commissions, in contrast to the current international oversight program. However, they then admit that utility industry security experts, the “boots on the ground” if you will, state that utility operations are more secure now than they would have been in the absence of NERC CIP. They also realize that the cyber security technical acumen at the state level is nowhere near what it should be to provide effective oversight.
Based on what we have seen working with utilities over the last 14 years, we do not recommend diverting focus now and throwing away all the progress made. Doing so would result in a “let up” in cyber security vigilance and a weakening of the overall critical infrastructure security posture at a time when the threats are growing ever more numerous and sophisticated.
We would, however, recommend some important adjustments that can be summarized as spending smarter by investing in security solutions that:
- Recognize that operations are key, are respectful of operations, and merge well with operations.
- Are aware of the entire operations architecture, its policies and compliance requirements.
- Focus on change management and automation.
Returning to the original question, our overall recommendation for the utility industry is unequivocal: “Stay the course! Don’t give up the CIP!”
 With apologies to Captain James Lawrence, USS Chesapeake, 1813.