In water and wastewater processing facilities, security is of the utmost importance. Communities need to have pure drinking water, and wastewater needs to be as clean as possible before being returned to the local environment.
While modern water utilities are working hard to prioritize security alongside performance, the game has recently grown tougher. Water treatment equipment and programmable logic controllers (PLCs) are now more frequent targets of cyberattacks, bringing new urgency to the task. In December 2023, the FBI reported a series of breaches1 at water treatment centers that all targeted the same industrial equipment. In a high-profile incident2 in 2021, an unauthorized individual gained remote access to a PLC in a Florida water treatment facility and raised the levels of sodium peroxide to unsafe levels. Thankfully, an employee monitoring the values caught the change before it could have any impact.
As cyber threats rise, water processing facilities may soon face more stringent regulatory pressure to shore up their defenses. In the United States, legislators have introduced a bill3 intended to address cybersecurity weaknesses in the water and wastewater sector in an effort to protect the nation’s critical infrastructure.
For water treatment facilities, addressing these new security requirements in the operational technology (OT) environment can be a tall order. Many operate multiple remote facilities beyond the main control room, such as tanks, pumphouses and lift stations. Those remote stations often run legacy components and equipment that have worked reliably for years or decades. The challenge for controls engineers now is to bring those systems up to date in line with today’s cybersecurity standards.
The good news is that as industrial automation control systems become smarter and digitized, they benefit from years of IT experience in cybersecurity best practices.
Best Practices for Water Facilities: The ISA/IEC 62443 Standard
The International Society of Automation (ISA)/International Electrotechnical Commission (IEC) 62443 standard is a consensus-based cybersecurity standard for industrial automation and control system (IACS) applications, which includes water and wastewater processing plants. It consolidates global IT cybersecurity best practices and translates them into security standards for utilities and other industrial applications. The result is a solid road map for water treatment facilities that need to shield their data and systems from breach or damage while strengthening their overall security posture.
The standard defines how networks and connections should be configured and secured for the entire life cycle of utilities applications—from design through to decommissioning. It drives a crucial point home, one learned and borrowed from IT: Security is not just implemented but needs to be continuously improved.
Strength Through Segmentation: Zones & Conduits
For water utilities, that means adopting a layered security approach. At the level of network connectivity, the ISA/IEC 62443 standard establishes requirements for dividing systems into segments as a key security measure to fortify industrial systems. To illustrate, consider the six-layer Purdue model (referenced by ISA/IEC 62443), an established blueprint for building secure control systems.
The architecture is divided into six layers: a lower “control zone,” which comprises the most critical OT functions, and an upper “enterprise security zone,” which consists of two layers for IT functions. The two areas are separated by a protective barrier, called the demilitarized zone (DMZ).
Consider a wastewater processing facility with separate stations for pumping, coagulation, sedimentation, filtration, etc., (processes A, B, C, D), each with their own input/output (I/O) devices, PLCs and variable frequency drives (VFDs). Those process systems and their control devices—VFDs, PLCs, supervisory control and data acquisition (SCADA) systems, etc.—sit at Levels 0-3. However, many water and wastewater providers maintain flat networks with no divisions between the different processes in the control zone. The problem is that a device plugged in at any of the remote stations can access every other part of the facility.
This ease of connectivity is a major security risk and can lead to intentional or unintentional compromise. An employee laptop infected with malware connected to the filtration PLC, for example, could damage the rest of the system. A hacker with malicious intent could infiltrate one network and gain easy access to the others.
A safer approach is to segment the control zone into multiple smaller containers, called zones. Since each zone has its own network, any device plugged into that zone can only access the processes and devices in that area. Devices and processing data at a lift station, for instance, can be kept separate from equipment and data in a sedimentation tank. Zones create a layered security boundary with process control and help maintain the same security level for all equipment, systems and devices within that zone.
However, zones are not isolated from one another. Communication between different zones can be enabled through conduits that control and monitor traffic in and out of individual segments.
For example, if our imaginary water facility’s Process A had a PLC that needed to communicate to a PLC in Process C, a conduit would be set up to connect Process A to Process B, and a second conduit would connect Process B to Process C. Those conduits would block or allow traffic. Done right, they would also provide visibility into what’s happening at each boundary.
The Risks of a Traditional VLAN
A conventional approach to zone segmentation in water and wastewater processing relies on virtual local area network (VLAN) technology. However, for facilities with remote stations comprising various devices and systems, VLAN can be technically challenging and resource heavy.
Each zone needs its own VLAN subnet with its own unique IP address, subnet mask and default gateway. For devices to communicate properly within and between zones, every single device in those VLAN subnets—PLCs, VFDs and I/O devices—would need to be updated with new parameters reflecting those new IP addresses. Users would also have to set up routers for those subnets. A VLAN solution would still require firewalls to block traffic ingress and egress at transport and network layers and effectively allow or disallow communication through segment boundaries.
Getting VLAN, router and firewall technology to work effectively together can be a challenging process. Every piece adds a layer of complexity, and a single change introduced to one part of the solution requires adjustments to every other device.
When a water treatment process has been running smoothly and reliably for 10 to 20 years or more, any change to that process is daunting. The local community relies on the facility for drinking water and safe wastewater treatment. The concern with VLAN technology is that what begins with a little downtime could impact water or wastewater services or jeopardize the whole process.
Solving network segmentation using VLANs requires significant time and labor investment, which many municipal utilities do not have or cannot access. On top of the work involved, the number of configuration changes required introduces significant risk. IP addresses of hundreds or even thousands of devices may need updating. A simple typing error could bring operations to a halt and necessitate a lengthy troubleshooting process to identify the error and restore the system. Key performance indicators (KPIs) including overall equipment effectiveness (OEE), productivity and quality could decline during the ensuing shutdown.
Meeting Cybersecurity Standards
An easier method is possible with a compact industrial firewall. In this case, a firewall device is installed into the industrial control panel and serves as a conduit between two zones (e.g., Line A and Line B). With this approach, rather than having a single cable running between Lines A and B, cables from Line A and Line B both plug into the firewall in the panel. The firewall acts as the conduit and provides the necessary segmentation between zones.
Some firewall-enabled units provide firewall protection as well as traffic control and monitoring functionality. In this case, they eliminate the need for VLANs and routers, which can be beneficial for organizations seeking simplicity.
In contrast to the VLAN approach, a firewall device can support zones and conduits capability and preserve network communication without requiring any time-consuming or risky changes to network devices. For organizations without a dedicated team of networking or IT security personnel, this solution can be advantageous. Some firewall devices do not expect or require special expertise, and the installation runs through a self-guided process.
Some control panel-mounted firewall devices come with configuration software enabled to automatically create a host list of devices trying to communicate with it. With this auto-generated list, the user only needs to know which hosts (and which zones) should be connected to each other and which connections should be blocked. With some firewall devices, users can easily and securely view and manage those decisions using a graphical user interface.
Advanced industrial firewalls often provide a system logging protocol among their system controls. This feature logs and stores network events involving the device. In the case of a cyber incident or attack, those network events can be collected and used to alert the IT department that something has occurred.
Bridging OT & IT for Cybersecurity Resilience
Given the mounting threat of cyberattacks on critical infrastructure, including water and wastewater processing facilities, utilities providers around the world are seeking new ways to increase the resiliency and security of their OT.
When safer, stricter access control is a priority for water treatment, large, flat networks are no longer viable options. A zones and conduits approach to network communication meets modern standards and provides more robust security and the necessary data traffic control.
Experienced controls engineers sometimes feel wet behind the ears when it comes to cybersecurity. Fortunately, cybersecurity standards for utilities applications have their best interests in mind in terms of high uptime and regulatory compliance. The tools that deliver those priorities best are those that support IT cybersecurity principles by design.
Adopting a layered security approach does not just make engineers look good, it strengthens the security posture of the organization, protects upstream and downstream partners and enhances the security of the water and wastewater
sector overall.
References
cnn.com/2023/12/01/politics/us-water-utilities-hack/index.html
cyberscoop.com/water-oldsmar-incident-cyberattack/
statescoop.com/new-cybersecurity-bill-water-wastewater-protections/