A few years ago, I was trying to understand why a certain profession was so reluctant to change. I read some academic papers on metathesiophobia (fear of change) to try to understand better how to address it. I wanted to understand what made people in this particular line of work so afraid to change any of their circumstances.
The profession in question?
Network engineers.
Before you throw your Cat5 cables at your screen in rage, let me walk through an example. I could have picked any of these really:
- software-defined networking (“oh no, I am going to automate myself out of a job!”),
- software-defined wide area networking (“it will never take off; the Internet is not robust enough.”)
- Voice over IP or Video over IP (same as above)
But let’s focus on an even simpler problem: network security.
I was recently skimming through the Firewall Wikipedia page and was amused by the first reported type of network firewall: the packet filter.
Have we actually moved on from that first type of firewall? It seems to me, from my daily conversations with enterprises, that network engineers still write firewall rules pretty much the same way: with a network access list.
We all agree that the essence of a firewall rule is: authorizing an entity to communicate with another entity.
But how did we identify an entity? We didn’t have a global identity store to speak of. The only thing we could identify an entity by was its IP address. And thirty-odd years on, it has led to ever increasing technical debt.
I know that from having talked to enough customers who were managing thousands of firewall rules – or rather, mismanaging.
The thing is – an entity routinely changed IP addresses but the security rule wouldn’t automatically adjust.
An entity would disappear but the security rule would stay: unfortunately, most of us network engineers just don’t have the time to maintain good security hygiene.
Worse, an entity would disappear but another one would be created and assume its IP address, which in turn means its identity and its associated security rules. Given that this new entity was unlikely to have anything to do with the original one, it would have wrong security rules associated with it; with sometimes dangerous consequences.
Think about the traditional workflow for network security engineers: an application owner or project leader would send the network team a spreadsheet such as the one below, as part of a support ticket:
| Source IP | Destination IP | Port | Action |
|---|---|---|---|
| 1.1.1.1 | 2.2.2.2 | 80 | ALLOW |
| 2.2.2.2 | 3.3.3.3 | 56 | ALLOW |
| 3.3.3.3 | 4.4.4.4 | 200 | ALLOW |
Invariably, the network team would have to justify the change through a Change Management process – many of you have probably wasted hours of your life waiting for your turn on a Change Management call, only for the decision maker to postpone your minor change by weeks because of a totally unrelated change freeze on the network.
Once the network engineer finally gets his/her opportunity to put the change in, he/she would then have to implement it; ideally without making any mistakes (a copy/paste can easily go wrong, as I know from experience).
Yes, the change would have been likely done manually – there were no APIs so no opportunity to automate security rules.
What the engineer would almost never do is maintain the table above and update it when the application changed or when it would be decommissioned: it’s unclear whose responsibility it is and anyway, life’s too short to update a boring old spreadsheet.
The network engineer would also not have the bandwidth to go and check if the security rules still in place are still relevant or if they need updating or removing. Nobody would take the risk to remove a security that might appear obsolete but might be useful to a server running a batch job once a year.
Clearly, we network engineers faced a variety of changes with dealing with security rules:
- No global identity store so the only option was to tie firewall rules to IP addresses,
- Firewall rules management was a nightmare
- There was significant change management overhead
- Keeping documentation up to date
- No API and no automation to manage security rules at scale and at speed
Fast forward to now and we should be able to do better, right?
We can do creative Firewalling based on the entity’s naming, operating system, tags, etc… We are no longer tied to IP addresses.
We’ve got REST APIs pretty much everywhere. APIs means we can automate our firewall configuration. API means we can treat our firewall config as code. Our firewall configuration can now be coded and versioned and can be used as a living documentation.
We’ve got Service Registries where we can store the identity of our applications.
Put these things together and now we can automate provisioning – creating, updating and removing – of your firewall rules from an update in your Service Registry.
We can automate most of our changes and avoid some of the tedious change management calls.
All our problems have been solved.
You would think that network engineers would embrace the change? It turns out, many of us are still reluctant to do anything different. Fear of change is real and it makes us come up with poor excuses.
“It’s too hard”.
“It’s too new”.
“It’s not my domain”.
Am I generalizing? Absolutely! Many of the network engineers I talk to via Twitter or in business conversations are embracing newer technologies.
But many are not. And you know what? I’ve been there.
While at Cisco, I was headhunted by a headhunter in 2014 for a job at VMware to work on network virtualization. I dismissed it straight-away: I wasn’t even willing to consider that a company outside of Cisco could have come up with a disruptive networking technology (ironically, I ended up working for VMware a couple of years later).
I have also found sometimes not as much a fear of change but a fear of learning. I was reluctant to go deep on Kubernetes for several years because I thought it didn’t apply to me or because it would be too difficult. In the end, I picked up Kubernetes Up and Running and realized it wasn’t that complicated.
Sometimes a new technology can appear more daunting than it actually is.
The academic papers I read on metathesiophobia provided several avenues to help folks overcome their fear of change. I won’t go full “armchair psychologist” on you but one of the recommendations was to explain the “What’s in it for me?”.
One of the primary reasons network engineers have feared change for so long was that they couldn’t really see the personal benefits of evolving.
That’s one of the reasons I covered the example above in detail: you can see the pain points (hours spent in change management calls, making manual and error-prone changes, maintaining documentation). But outside the fact a new tech can make your working life less crap, don’t you get any joy from learning and mastering a new tech? I do.
Learning and understanding technologies like NSX, SD-WAN, AWS, Python, Terraform, etc… brought me a lot of fulfilment.
Ultimately, the blame is on the tech industry and the vendors for not articulating the personal value well enough.
Perhaps this will change as vendors put a renewed focus on customer empathy.
Why is change so hard for us network engineers? Let’s discuss on Twitter.
Thanks for reading.
nicovibert.com 
One thought on “Network engineers: will you ever dare to change?”