Listen to this Post
Introduction:
In the converging worlds of Operational Technology (OT) and IT, virtualized Programmable Logic Controllers (vPLCs) are becoming common, yet they introduce single points of failure that can halt industrial processes. Inspired by a recent industry discussion on minimal redundancy, this article explores a “cold standby” approach for vPLCs using heartbeat monitoring. This method provides a cybersecurity and resilience angle by ensuring that even if a primary controller fails—whether due to system crash, network fault, or resource exhaustion—a backup can assume control without the complexity of full hot-standby synchronization.
Learning Objectives:
- Understand the architecture of a minimal redundancy system for virtual PLCs using heartbeat signals.
- Learn how to implement a supervisor service to monitor vPLC health and trigger failover.
- Identify the edge cases and security considerations (state synchronization, IP takeover) in cold standby configurations.
You Should Know:
1. Architecture Overview: Primary, Backup, and the Heartbeat
The core concept, as discussed by Dan Buda, involves a Primary vPLC running the logic, a Backup vPLC in STOP mode, and a Supervisor monitoring a “heartbeat.” This is a classic watchdog pattern adapted for industrial control.
- What it does: The primary controller publishes a periodic signal (heartbeat). If the supervisor stops receiving this signal, it assumes the primary has failed and issues a start command to the backup.
- How to use it: This requires configuration on three levels: the PLC logic (to generate the heartbeat), the network (to transmit it), and the supervisor script (to listen and act).
Step‑by‑Step Guide (Conceptual & Implementation):
- Configure Primary vPLC: Add a function block to your primary PLC program (e.g., in CODESYS, Siemens TIA Portal, or Rockwell Studio 5000) that toggles a Boolean variable or increments a counter. Map this variable to a network tag.
– Example (Structured Text snippet):
PROGRAM HeartbeatGenerator; VAR HeartbeatCounter : INT := 0; TON_1s : TON; // 1-second timer END_VAR TON_1s(IN:=NOT TON_1s.Q, PT:=T1000MS); IF TON_1s.Q THEN HeartbeatCounter := HeartbeatCounter + 1; // Publish this counter value to the network (e.g., via MQTT or OPC UA) END_IF;
2. Establish Communication Protocol: Choose a protocol for the heartbeat. MQTT is lightweight and ideal for this. Configure the vPLC to publish its `HeartbeatCounter` to an MQTT broker (like Mosquitto) on a specific topic (e.g., plant/line1/plc1/heartbeat).
3. Create the Supervisor Script: This script (written in Python, for example) will subscribe to the heartbeat topic and monitor for timeouts.
– Python MQTT Supervisor Example:
import paho.mqtt.client as mqtt
import time
import os
last_heartbeat = time.time()
HEARTBEAT_TIMEOUT = 5 Seconds
BACKUP_PLC_IP = "192.168.1.102"
BACKUP_API_ENDPOINT = "http://" + BACKUP_PLC_IP + "/api/start"
def on_connect(client, userdata, flags, rc):
print("Connected to MQTT Broker")
client.subscribe("plant/line1/plc1/heartbeat")
def on_message(client, userdata, msg):
global last_heartbeat
last_heartbeat = time.time()
print(f"Heartbeat received: {msg.payload.decode()}")
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("192.168.1.50", 1883, 60) Broker IP
client.loop_start()
try:
while True:
time.sleep(1)
if time.time() - last_heartbeat > HEARTBEAT_TIMEOUT:
print("Heartbeat timeout! Triggering failover...")
Trigger backup PLC start (e.g., via REST API call)
os.system(f"curl -X POST {BACKUP_API_ENDPOINT}")
In a real scenario, you would also handle IP swapping
last_heartbeat = time.time() Reset to avoid repeated triggers
except KeyboardInterrupt:
client.loop_stop()
- Handling the Failover: From STOP to RUN and IP Takeover
As noted in the LinkedIn comments by J. Ignacio Ortega, true redundancy often involves IP swapping so that downstream devices (like HMIs and SCADA) don’t need to reconfigure. The supervisor must not only start the backup but also manage network identities.
- What it does: Upon detecting a failure, the supervisor forces the backup PLC to transition from STOP to RUN mode and re-assigns the primary’s IP address to the backup.
- How to use it: This step requires leveraging the vPLC’s vendor-specific APIs (e.g., CODESYS Automation Server API, Siemens S7-1500 SoftPLC API) or using low-level network commands on the hypervisor.
Step‑by‑Step Guide (Network & API Integration):
- API Integration for PLC Control: Extend the Python supervisor script to send commands to the backup PLC. Most virtual PLC environments offer a REST API or a command-line tool.
– Example (Using `curl` for a CODESYS Control Win V3): The supervisor could execute:
Command to start the application on the backup PLC
curl -X POST http://192.168.1.102:8080/api/plc/start -H "Content-Type: application/json" -d '{"application": "MyProject"}'
2. IP Address Takeover (Gratuitous ARP): To make the backup PLC invisible to the network until needed, keep its network interface down or assign a secondary IP. Upon failover, bring the interface up with the primary’s IP address.
– Windows (Hyper-V host) PowerShell command to reassign IP:
Remove secondary IP from failed primary (assuming it's still responsive on management)
Remove-NetIPAddress -InterfaceAlias "vEthernet (PLC Network)" -IPAddress 192.168.1.10
Add the primary IP to the backup VM
Invoke-Command -VMName "BackupPLC" -ScriptBlock { New-NetIPAddress -InterfaceAlias "Ethernet" -IPAddress 192.168.1.10 -PrefixLength 24 }
– Linux (KVM/Proxmox host) bash command:
On the Backup VM host, assign the IP
virsh domifaddr BackupPLC to find the interface
virsh qemu-agent-command BackupPLC '{"execute":"guest-exec","arguments":{"path":"/bin/ip","arg":["addr","add","192.168.1.10/24","dev","eth0"],"capture-output":true}}'
– Send a Gratuitous ARP to update network switches: `arping -U -I eth0 192.168.1.10`
3. State Synchronization: The Critical Challenge
Thiago Alves, PhD, highlighted the biggest hurdle: synchronization. If a vPLC fails in the middle of a step, the backup cannot start from a safe state unless it knows the values of timers, counters, and sequence steps.
- What it does: This step involves periodically copying the runtime data (process variables) from the primary to the backup so the backup can resume from the exact point of failure.
- How to use it: This moves the system from “cold standby” toward “warm standby.” It requires engineering the PLC code to export its variable context.
Step‑by‑Step Guide (Data Sync):
- Identify Critical Variables: In your PLC project, define a structure or a list of all variables that define the current state (e.g., current step number, timer accumulated values, counter values). Exclude derived values that can be recalculated.
- Implement Sync Logic: Create a routine on the primary that serializes this state data and publishes it (e.g., via MQTT) at a defined interval (e.g., every 500ms). The backup PLC, even in STOP mode, needs a background task (or a separate agent) that receives this data and writes it to its own memory.
- Startup State Restoration: When the supervisor triggers the backup to start, the first routine in the backup PLC should load the last received state data into its active memory before beginning logical execution. This prevents the process from resetting to zero.
4. Securing the Heartbeat and Failover Mechanism
From a cybersecurity perspective, the heartbeat and failover signals are critical control points. An attacker could spoof a heartbeat to hide a compromise or trigger a false failover to cause a denial of service.
- What it does: Implements authentication and integrity checks for all inter-PLC and supervisor communications.
- How to use it: Move from plaintext MQTT to MQTT over TLS. Use API keys for PLC control endpoints.
Step‑by‑Step Guide (Hardening):
- Enable TLS for MQTT: Configure the Mosquitto broker to use SSL/TLS certificates. Configure the PLC (if supported) and the Python supervisor to use the broker’s certificate.
– Python Client with TLS:
client.tls_set(ca_certs="ca.crt", certfile="client.crt", keyfile="client.key")
client.connect("192.168.1.50", 8883, 60) Secure port
2. Restrict API Access: The backup PLC’s start API should not be open to the entire network. Place the supervisor and PLCs in a dedicated OT VLAN with strict firewall rules. Use HTTP Basic Authentication or API tokens in the `curl` commands.
– Example with API Key: `curl -X POST https://192.168.1.102:8080/api/plc/start -H “X-API-Key: your_secure_key”`
5. Testing for Edge Cases
The LinkedIn discussion mentioned the need to test for edge cases. This includes scenarios where the heartbeat stops due to network congestion but the PLC is still running, or where the backup starts but the primary hasn’t fully relinquished control.
- What it does: A testing framework to simulate network latency, packet loss, and partial failures.
- How to use it: Use Linux traffic control (
tc) on the hypervisor to introduce network degradation.
Step‑by‑Step Guide (Simulating Failure):
- Simulate Primary Crash: Halt the primary vPLC process from the hypervisor command line.
– On Proxmox: `qm stop
– On ESXi: `vim-cmd vmsvc/power.off
2. Simulate Network Latency: Add latency to the primary’s virtual network port to see if the supervisor times out incorrectly.
– Linux host command: `tc qdisc add dev vnet0 root netem delay 3000ms` (adds 3 seconds of delay).
What Undercode Say:
- Redundancy Requires a Security Audit: A failover mechanism introduces new attack surfaces. The supervisor becomes a high-value target; securing its communication channels with TLS and API keys is not optional but mandatory for OT resilience.
- State is the New Asset: In modern virtualization, the most critical asset to protect and synchronize isn’t just the program file, but the live runtime state. Techniques for secure, real-time state replication are the true frontier of high-availability OT security.
- Simplicity vs. Complexity: The “cold standby” method is an excellent entry point for organizations daunted by full hot-standby systems. However, this analysis shows that even a simple heartbeat system quickly requires complex engineering (IP takeover, state sync) to be truly effective, blurring the lines between simple and sophisticated solutions.
Prediction:
As vPLCs become standard in Industry 4.0 and 5.0, we will see the emergence of “Redundancy-as-a-Service” built into hypervisors and container platforms. Future attacks will likely target the synchronization channel or the supervisor logic itself, aiming to desynchronize primary and backup states, creating a “split-brain” condition where both controllers assume control, leading to physical process conflicts. This will drive the development of blockchain or consensus-based verification for critical failover decisions in OT environments.
▶️ Related Video (84% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Dan Buda – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
