The mundane of cybersecurity rarely makes the news. Breaches and ransomware are typical news fare.
Last week, that narrative changed. The often unnoticed, seemingly boring procedure of patching took center stage. Even people far afield from cybersecurity or IT heard about the massive outages caused by caused by the catastrophic IT outage stemming from an automated update pushed by CrowdStrike. Somewhere on the order of 8.5 million machines were affected.
The fallout was severe for individuals and organizations, with over 5000 flights canceled due to grounded aircraft, and critical sectors like banking and healthcare experiencing significant disruptions.
For those outside of cybersecurity and IT, the question doesn’t go much beyond “what happened” or “how could this happen.” For those inside the industry, this incident has reignited the debate on the safety of deploying patches.
On one hand, the hesitation is understandable; updates have been the culprits behind outages before, though not with such newsworthy reach. Any cybersecurity or IT professional has faced the pain of a misapplied or misconfigured patch at some point. The inconvenience and potential downtime for identification and remediation can be significant. “Why patch if the fix creates a bigger problem” could be a natural conclusion.
On the other hand, the urgency to patch is underscored by the alarming speed at which attackers exploit vulnerabilities—sometimes immediately after the exploit details have been published. A zero day may in fact be measured in hours or even minutes. .
We see this in both our testing and incident response. In the majority of penetration tests, we find something that is running older code that we can exploit. Very often it is a recent vulnerability. Even regular patching, if only quarterly, will leave a myriad of things to exploit. In many IR situations, somewhere unpatched systems and programs, if not the point of original entry, play some role in the escalation and persistence of the attack.
The dilemma is clear: How often should we patch? While the fear of introducing new issues is valid, the answer remains unchanged. Patch frequently and promptly, but never at the expense of safety.
Best Practices for a Patching Strategy:
• Vet Updates in a Test Environment: Before rolling out patches to production, thoroughly test them to catch any potential issues.
• Be Swift but Safe: Aim to deploy patches as quickly as possible, but not so fast that you skip essential checks.
• Stay Informed: Keep abreast of the latest threats and patch accordingly.
• Have a Rollback Plan: Always be prepared to revert changes if something goes wrong.
Of course, some patches may, as with the CrowdStrike one, be patches applied where there is not the opportunity to vet or test. In these cases, staying informed and having a rollback plan are critical.
While the CrowdStrike incident serves as a cautionary tale, it should not deter us from patching. Instead, it should reinforce the need for a robust, well-thought-out patching process that prioritizes both security and system integrity.
TL;DR: You still have to patch.
– Tanner Shinn, Principal Security Engineer
It’s almost summer, and you know what that means! Teacher appreciation weeks, Final exams, graduations, recitals, and burnout. But you know who works all year round and can’t afford to get burnout? Your school IT Team! While the rest of [...]
