Infrastructure as Code

Abstract

Decoupling complexity.  Replacing Juniper SRX chassis clusters with standalone nodes and declarative Infrastructure as Code (IaC).

The Low-Down

For years high-availability firewall design was simple: take two identical SRXs and cluster them! Juniper ties them together with control and fabric links, and call it a single chassis. One management plane and stateful failover. It looks darn clean on paper.

In reality, there are a few rough edges. For example, a problem with the control plane running on the primary can take down the secondary. ISSU was sold as a system that never goes down during patching. In reality, a small IPC hiccup can turn a simple update into a pants-on-fire split-brain—or worse.

If I could do it all over again I might have chosen the boring alternative. Run independent nodes and lean hard on dynamic routing and Infrastructure as Code (IaC) methodology. Let BGP (or OSPF) handle all failover. Use OpenTofu or Terraform as the external control plane so both boxes stay in sync. No shared control plane. No reth interfaces. Maintenance becomes “patch one, watch it, patch the other.” If one node blows up during patching? So what? 🤣 The other node carries the load while you take care of the dirty work.

Standalone pair + dynamic routing

                    [ Upstream Core / WAN ]
                         /            \
              BGP/ECMP  /              \  BGP/ECMP
                       /                \
            [ SRX-1 ]                    [ SRX-2 ]
                       \                /
              BGP/ECMP  \              /  BGP/ECMP
                         \            /
                    [ Downstream Fabric ]

Live and die by dynamic routing. And why not? It’s time proven for over 30 years. It just works. Frankly, it’s intuitive. Traffic steering lives in the routing protocol. You want traffic to exit node0? Give it local preference 300. You want it to default out node1? Give node0 the default preference and node1 preference 300. Boom!

Granted, you lose stateful session connection tracking during failover. But 1997 called, and they said they want their session-dependent apps back! Most web and cookie/token-based traffic barely notices a disruption. The next transit request arrives on the next node with a fresh binding. Nearly seamless to the end user. Perhaps a 10-second delay at most.

External control plane

No cluster means no single management plane. That shared policy view is convenient while it lasts, but it only works for the two-node cluster and it still leaves every other firewall, router, and switch as its own little kingdom. Rainbows and unicorns it is not.

There is however one CLI to rule them all. Say hello to my little friend, OpenTofu (i.e., Terraform).

It is darn near the holy trinity of management. Git for version control and the audit trail. A plan step that validates changes before anything touches a node. And a distributed controller that pushes the same intent to every device you point it at. You edit the security policy once. The change lands uniformly on both SRXs. That is the single pane the chassis cluster promised, except it is not limited to two nodes and it does not share a fate with either of them.

The Git part can live wherever you are comfortable. A small server on an isolated management network works fine if you want to keep the repo off the internet. Or you can use GitHub, Azure DevOps, or whatever cloud service you already trust. Either way the workflow is the same: branch, edit, pull request. The comments on the PR become the place the team talks about the change—“is this prefix right?” “does this policy belong in the trust zone?”—before anyone runs an apply. No more “I thought you already pushed that” over chat.

A policy change in practice

Say you need to open SSH from a new jump-box subnet. You edit the HCL instead of the CLI:

variable "firewalls" {
  type = map(object({
    host_ip    = string
    untrust_ip = string
  }))
  default = {
    "srx-node1" = { host_ip = "10.10.0.11", untrust_ip = "203.0.113.10/30" }
    "srx-node2" = { host_ip = "10.10.0.12", untrust_ip = "203.0.113.14/30" }
  }
}

resource "junos_security_zone" "trust" {
  for_each         = var.firewalls
  name             = "TRUST"
  inbound_services = ["ssh", "ping"]
}

resource "junos_security_zone" "untrust" {
  for_each = var.firewalls
  name     = "UNTRUST"
}

resource "junos_security_policy" "mgmt_access" {
  for_each  = var.firewalls
  from_zone = junos_security_zone.untrust[each.key].name
  to_zone   = junos_security_zone.trust[each.key].name

  policy {
    name                      = "ALLOW-JUMPBOX"
    match_source_address      = ["10.250.0.0/24"]
    match_destination_address = ["any"]
    match_application         = ["junos-ssh"]
    then                      = "permit"
  }
}

Commit it, open a PR, and let someone else glance at the prefix and the placement in the comments before anything touches a firewall. Then:

tofu plan

You should see the identical delta for both nodes. Only after that does the apply run. OpenTofu opens NETCONF to both boxes, commits the candidate, and finishes. Both SRXs now have the same policy. If someone later hand-edits one of them, the next plan calls it out.

But wait, there’s more. The same methodology works for the rest of the fleet. All the firewalls. All the routers. All the switches. One repo, one plan, one apply. You become the white wizard. 🤣

What you actually gain

Configuration drift shows up instead of hiding until something breaks. Git becomes the audit trail.  Reverting a bad change on both firewalls is a revert plus an apply. Upgrades and hardware failures no longer risk taking the whole pair down through a shared control plane. Failover is ordinary BGP. Most of us already know how to debug that.

Honest limits

You traded firewall session synchronization for independence. Pure long-lived TCP flows can still break. Most web and cookie/token traffic barely notices—the next request just arrives on the surviving node with a fresh NAT binding and the application session stays intact. BGP convergence is a few seconds, not sub-second. The Junos Terraform provider covers a lot of the security policy and interface work, but some features are still easier with a targeted NETCONF call or a little Ansible. NAT and return-path symmetry still need deliberate design so ECMP does not surprise you.

None of that makes the pattern wrong. It just means you are choosing a different set of trade-offs: standalone hardware, routed failover, and policy consistency enforced from outside the boxes instead of from a fragile shared control plane.

If you have spent time nursing chassis clusters through ISSU or chasing reth oddities, the operational differences should feel familiar. Patch one, watch it, patch the other. If one blows up during the upgrade? The other node carries the load while you take care of the dirty work. 🤣

0 Comments:

Post a Comment

My Instagram