As I'm sure many people have been, we've taken actions with many of our customers to remediate (or at least mitigate) from the Log4Shell Log4j 0-day. One of the things I wanted to write about are a few learnings from our experience, specifically focused on our use of AWS Web Application Firewall. I also want to touch on some of the finer points about whether network protection will mitigate the issue (and why it probably will not be enough).
I'll also preface this with a note - this is not a criticism of the product in any way, even if it sounds like it is. Sorry to the WAF team if it comes off that way.
Don't be solely reliant on your WAFs
This lesson isn't specific to AWS WAF but is incredibly important. If you're relying on a WAF to protect you, rather than ensuring you're implementing appropriate controls all the way down your stack - you're going to have a bad time. A defence in depth strategy that is applied at each and every layer of your stack is considerably more effective at providing protection.
Having a WAF apply protection for poor input validation or a lack of cross-site scripting protection, as an example, will not fully protect your application. It's analogous to the enterprise networks of old, with a hard shell but a soft gooey (easy to break) centre. Once a bad actor has gained a privileged position, your WAF will not protect you at all.
OK - what's WAF good for in this regard? I like to think of it in terms of the 80/20 Rule. Having a WAF will definitely reduce the number of bad looking requests to your app, which is less load and definitely less risk. It's certainly not going to catch all of it unless your application was designed for WAF upfront, but it's going to help more than it hinders.
Be aware of the processing order for AWS Managed Rules
One big thing we've discovered that's not documented (or, if it is, it's not remotely obvious) is that AWS Managed Rules are processed before all other rules in your V2 WAF. Yes, even if you put a higher priority on the managed rule. No, you cannot override that behaviour.
We've run into a few places where our lower priority rules should have been triggered first but were never hit. After digging with the help of AWS Support, they confirmed that because of the nature of how the AWS Managed Rules are implemented, they are always processed first.
If you're excluding rules in an AWS managed rule set and applying a similar rule, be aware that it might not function
the way you expect. The big case for this is request size restrictions, which I'll talk about in a second. If
you do exclude something like SizeRestrictions_BODY and then try and use a
Size contstraint
in your own rule, you'll want to look into a
Label match rule statement
in its place.
Request size constraints
AWS does not make a secret of the fact that WAF cannot inspect request bodies greater than 8KB. It's mentioned several times in their documentation and is bannered all over the WAF console.
For the log4j vulnerability, this is critically important. If you have an application with request sizes greater than 8KB, the AWS WAF is only inspecting parts of your request. Bypassing the protection is as simple as sending a large payload with the nasty stuff outside the first 8KB (zero padding is enough in some cases, depending on your ruleset).
If you are prepared enough, you can definitely reduce the attack surface of your application by using a WAF, but if it's not complete (as in blocking all requests that could be malicious), you're very likely to be exposed. In this case, you must consider and should be applying the log4j mitigations or remediation to your applications directly.
Network firewalls alone aren't going to protect you
I've heard this from a few teams - suggesting they're going to apply network firewall rules to protect their network. I'm confident that this isn't the only protection those teams will be using, but in my not so humble opinion, it's also still only going to provide a weak layer of protection.
Sure, you might be able to block a malicious JNDI endpoint, so you aren't at risk of running untrusted code, but I'm confident that you are still at risk of DNS exfiltration (or NTP, for that matter). My colleague James has also blogged on the subject.
Wrapping up 🎁 (because it's nearly Christmas)
Pardon the pun, but I think we all need a bad joke in light of the last few days.
If you haven't already guessed my point - it's vital to think about securing your workloads at more than just a single layer. We like to say "never waste an incident", so if you aren't currently thinking this way, now is the perfect time to start.