The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content. It's not just Cloudflare, Akamai has the same problem.
If your site discusses databases then turning on the default SQL injection attack prevention rules will break your site. And there is another ruleset for file inclusion where things like /etc/hosts and /etc/passwd get blocked.
I disagree with other posts here, it is partially a balance between security and usability. You never know what service was implemented with possible security exploits and being able to throw every WAF rule on top of your service does keep it more secure. Its just that those same rulesets are super annoying when you have a securely implemented service which needs to discuss technical concepts.
Fine tuning the rules is time consuming. You often have to just completely turn off the ruleset because when you try to keep the ruleset on and allow the use-case there are a ton of changes you need to get implemented (if its even possible). Page won't load because /etc/hosts was in a query param? Okay, now that you've fixed that, all the XHR included resources won't load because /etc/hosts is included in the referrer. Now that that's fixed things still won't work because some random JS analytics lib put the URL visited in a cookie, etc, etc... There is a temptation to just turn the rules off.
> I disagree with other posts here, it is partially a balance between security and usability.
And economics. Many people here are blaming incompetent security teams and app developers, but a lot of seemingly dumb security policies are due to insurers. If an insurer says "we're going to jack up premiums by 20% unless you force employees to change their password once every 90 days", you can argue till you're blue in the face that it's bad practice, NIST changed its policy to recommend not regularly rotating passwords over a decade ago, etc., and be totally correct... but they're still going to jack up premiums if you don't do it. So you dejectedly sigh, implement a password expiration policy, and listen to grumbling employees who call you incompetent.
It's been a while since I've been through a process like this, but given how infamous log4shell became, it wouldn't surprise me if insurers are now also making it mandatory that common "hacking strings" like /etc/hosts, /etc/passwd, jndi:, and friends must be rejected by servers.
Not just economics, audit processes also really encourage adopting large rulesets wholesale.
We're SOC2 + HIPAA compliant, which either means convincing the auditor that our in-house security rules cover 100% of the cases they care about... or we buy an off-the-shelf WAF that has already completed the compliance process, and call it a day. The CTO is going to pick the second option every time.
Yeah. SOC2 reminds me that I didn't mention sales as well, another security-as-economics feature. I've seen a lot of enterprise RFPs that mandate certain security protocols, some of which are perfectly sensible and others... not so much. Usually this is less problematic than insurance because the buyer is more flexible, but sometimes they (specifically, the buyer's company's security team, who has no interest besides covering their own ass) refuse to budge.
If your startup is on the verge of getting a 6 figure MRR deal with a company, but the company's security team mandates you put in a WAF to "protect their data"... guess you're putting in a WAF, like it or not.
>guess you're putting in a WAF, like it or not.
Install the WAF crap, and then feed every request through rot13(). Everyone is happy!
Up until you need to exercise the insurance policy and the court room "experts" come down on you like a ton of bricks.
OS-level monitoring / auditing software also never ceases to amaze me (for how awful it is). Multiple times, at multiple companies, I have seen incidents that were caused because Security installed / enabled something (AWS GuardDuty, Auditbeat, CrowdStrike…) that tanked performance. My current place has the latter two on our ProxySQL EC2 nodes. Auditbeat is consuming two logical cores on its own. I haven’t been able to yet quantify the impact of CrowdStrike, but from a recent perf report, it seemed like it was using eBPF to hook into every TCP connection, which is quite a lot for a DB connection poolers.
I understand the need for security tooling, but I don’t think companies often consider the huge performance impact these tools add.
I wish IT teams would say "sorry about the password requirement, it's required by our insurance policy". I'd feel a lot less angry about stupid password expiration rules if they told me that.
Sometime in the past few years I saw a new wrinkle: password must be changed every 90 days unless it is above a minimum length (12 or so as best I recall) in which case you only need to change it yearly. Since the industry has realized length trumps dumb "complexity" checks, it's a welcome change to see that encoded into policy.
I think I like this idea that the rotation interval could be made proportional to length, for example doubling the interval with each additional character. Security standards already now acknowledge that forced yearly rotation is a net decrease in security, so this would incentivize users to pick the longest password for which they would tolerate the rotation interval. Is yearly rotation too annoying for you? For merely the effort of going from 12 -> 14 characters, you could make it 4 years instead, or 8 years, 16, and so on.
Can confirm when I found out I'd be required to regularly change my password the security of it went down significantly. At my current job when I was a new employee I generated a secure random password and spent a week memorizing it. 6 months later when I found out I was required to change it, I reverted to a variation of the password I used to use for everything years ago with some extra characters at the end that I'll be rotating with each forced change...
I do the same but write the number at the end of the password on the laptop in sharpie. I work from home so I've been thinking about making a usb stick that simulates a keyboard with a button to enter the password.
Dangerous. You might accidentally press the button in a group chat.
They would then have an excuse to get one of those mission control button covers.
Why not make use of a password manager?
You can’t open the password manager until your computer is unlocked.
You can put the password manager on your phone or another device.
Unfortunately, lots of end users refuse to read the password policy and won't understand why their password reset interval is "random" or shorter than their colleague's.
> Sometime in the past few years I saw a new wrinkle: password must be changed every 90 days unless it is above a minimum length (12 or so as best I recall) in which case you only need to change it yearly. Since the industry has realized length trumps dumb "complexity" checks, it's a welcome change to see that encoded into policy.
This is such a bizarre hybrid policy, especially since forced password rotations at fixed intervals are already not recommended for end-user passwords as a security practice.
I think the issue is that some people don't actually understand what's going on, so in an attempt at goodwill, they try to "compromise", and "split the difference" if you will. Hell, some people will consider the windows hello pin as a password and force a regular rotation. Combined with policies coming from outside (think insurance and other compliance stuff) which try to cover as much ground as possible, you end up with half-assed implementations like these.
One discourse I hear is that "people will just use the same password everywhere". To which I'll answer, "but we have mfa". "yeah, but the insurance guys".
Having worked with PCI-DSS, some rules seem to only exist to appease insurance. When criticising decisions, you are told that passing audits to be able to claim insurance is the whole game, even when you can demonstrate how you can bypass certain rules in reality. High-level security has more to do with politics (my definition) than purely technical ability. I wouldn't go as far as to call it security theatre, there's too much good stuff there that many don't think about without having a handy list, but the game is certainly a lot bigger than just technical skills and hacker vs anti-hacker.
I still have a nervous tick from having a screen lock timeout "smaller than or equal to 30 seconds".
> but a lot of seemingly dumb security policies are due to insurers.
I keep hearing that often on HN, however I've personally never seen seen such demands from insurers. I would greatly appreciate if one share such insurance policy. Insurance policies are not trade secrets and OK to be public. I can google plenty of commercial cars insurance policies for example.
I found an example!
https://retail.direct.zurich.ch/resources/definition/product...
Questionnaire Zurich Cyber Insurance
Question 4.2: "Do you have a technically enforced password policy that ensures use of strong passwords and that passwords are changed at least quarterly?"
Since this is an insurance questionnaire, presumably your answers to that question affect the rates you get charged?
(Found that with the help of o4-mini https://chatgpt.com/share/680bc054-77d8-8006-88a1-a6928ab99a...)
We've been asked that question before on security questionnaires, and our answer has always been, "Forcing users to change passwords regularly is widely regarded as a very bad security practice, and we don't engage in bad security practices." We've never had anyone complain.
I've never had a complaint about anything I put in to a form requesting a quote for insurance. I just get the quote back. Did you write that in the comment expecting an insurance salesperson to call you up and argue passwords with you? Call their back office and say "hey this guy says our password question is crap, get our best guys on it!"?
I just cant imagine any outcome other than it was translated to just a "no" and increased your premium over what it would have otherwise been.
I've also filled out insurance quote forms several times to see the interplay of the questions and price. Quite often many of the questions do not change the quote. So the existence of the question in a form does not imply a change in price, or any true guess at the magnitude of the change at all.
Password policy is something rather common, and 'standard' firewalls. Question is in the context of of WAF as in the article. WAF requirement is something more invasive to say the least.
Directly following is question 4.3: "Are users always prevented from installing programs on end-user devices?"
Totally bonkers stuff.
A trend for corporate workstations is moving closer to a phone with a locked-down app store, with all programs from a company software repo.
Eliminating everything but a business's industry specific apps, MS Office, and some well-known productivity tools slashes support calls (no customization!) and frustrates cyberattacks to some degree when you can't deploy custom executables.
That's why this it's been a requirement for Australian government agencies for about 15 years.
In around 2011, the Defence Signals Directorate (now the Australian Signals Directorate) went through and did an analysis of all of the intrusions they had assisted with over the previous few years. It turned out that app whitelisting, patching OS vulns, patching client applications (Office, Adobe Reader, browsers), and some basis permission management would have prevented something like 90% of them.
The "Top 4" was later expanded to the Essential Eight which includes additional elements such as backups, MFA, disabling Office macros and using hardened application configs.
https://www.cyber.gov.au/resources-business-and-government/e...
Then the users start using cloud webapps to do everything. I can't install a PDF-to-excel converter, so I'll use this online service to do it.
At first glance that might seem a poor move for corporate information security. But crucially, the security of cloud webapps is not the windows sysadmins' problem - buck successfully passed.
I don’t think locking down slashes support calls because you will now receive support requests anytime someone wants to install something and actually have a good business reason to do so.
Consider the ones you don't get: ones where PCs have to be wiped from customization gone wrong, politics and productivity police calls - "Why is Bob gaming?", "Why is Alice on Discord?".
It's about the transition from artisanal hand-configuration to mass-produced fleet standards, and diverting exceptional behavior and customizations somewhere else.
Coupled with protection against executing unknown executables this also actually helps with security. It's not like (most) users know which exe is potentially a trojan.
If you don’t want exceptional behavior, that’s exactly what you’ll get. In more than one way.
Alice is on Discord because half of the products the company uses now give more or less direct access to their devs through Discord
This is standard practice for years in big corporations.
You install software via ticket requests to IT, and devs might have admin rights, but not root, and only temporary.
This is nothing new though, back in the timesharing days, where we would connect to the development server, we only got as much rights as required for the ongoing development workflows.
Hence why PCs felt so liberating.
It's a standard practice. And at $CURENT_JOB it's driven by semi-literate security folks, definitely not insurance.
Insurance and liability concerns drive the security folks.
Just wait when more countries keep adopting cybersecurity laws for companies liabilities when software doesn't behave, like in any other engineering industry.
Hello, the security folks in those companies made those up. "cyber insurance" is hogwash. That entire branch has been taken over by useless middle manager types who know to type up checklists in Word but have no understanding of anything.
As someone that happens to also be one of those clueless people when assuming DevOps roles in consulting projects, it is a very bad day when some clever user is responsible for a security breach.
A breach can turn out into enough money being lost, in credibility, canceled orders, or lawsuits, big enough to close shop, or having to fire those that thought security rules were dumb.
Also anyone with security officer title, in many countries has legal responsibilities when something goes wrong, so when they sign off software deliverables that go wrong, is their signature on the approval.
Are you arguing non technical people should have root access to company owned and managed PCs? Because I can tell you from experience, that will result in a very bad time at some point. Even if it is just for the single end user and not the wider org.
The fun part is that they don't demand anything, they just send you a worksheet that you fill out and presumably it impacts your rates. You just assume that whatever they ask about is what they want. Some of what they suggest is reasonable, like having backups that aren't stored on storage directly coupled to your main environment.
The worst part about cyber insurance, though, is that as soon as you declare an incident, your computers and cloud accounts now belong to the insurance company until they have their chosen people rummage through everything. Your restoration process is now going to run on their schedule. In other words, the reason the recovery from a crypto-locker attack takes three weeks is because of cyber insurance. And to be fair, they should only have to pay out once for a single incident, so their designated experts get to be careful and meticulous.
This is such an important comment.
Fear of a prospective expectation, compliance, requirement, etc., even when that requirement does not actually exist is so prevalent in the personality types of software developers.
It cuts both ways. I've struggled to get things like backups or multifactor authentication approved without being able to point to some force like regulation or insurance providers that can dislodge executives' inertia.
My mental model at this point says that if there's a cost to some important improvement, the politics and incentives today are such that a typical executive will only do the bare minimum required by law or some equivalent force, and not a dollar more.
You can buy insurance for just about anything, not just cars. Companies frequently buy insurance against various low-probability incidents such as loss of use, fraud, lawsuit, etc.
There should be some limits and some consequences to the insurer as well. I don't think the insurer is god and should be able to request anything no matter if it makes sense or not and have people and companies comply.
If anything, I think this attitude is part of the problem. Management, IT security, insurers, governing bodies, they all just impose rules with (sometimes, too often) zero regard for consequences to anyone else. If no pushback mechanism exists against insurer requirements, something is broken.
> There should be some limits and some consequences to the insurer as well. I don't think the insurer is god and should be able to request anything no matter if it makes sense or not and have people and companies comply.
If the insurer requested something unreasonable, you'd go to a different insurer. It's a competitive market after all. But most of the complaints about incompetent security practices boil down to minor nuisances in the grand scheme of things. Forced password changes once every 90 days is dumb and slightly annoying but doesn't significantly impact business operations. Having to run some "enterprise security tool" and go through every false positive result (of which there will be many) and provide an explanation as to why it's a false positive is incredibly annoying and doesn't help your security, but it's also something you could have a $50k/year security intern do. Turning on a WAF that happens to reject the 0.0001% of Substack articles which talk about /etc/hosts isn't going to materially change Substack's revenue this year.
The issue is that the Finance dept will show up and ask why you chose the more expensive insurance. Sure, if you're able to show how much the annoyances of the cheaper company would cost you, they'd probably shut it. But I'd argue it's not that easy. Plus, all these annoyances aren't borne by the security team, so they don't care that much in the end.
My first thought might be to put together a report showing the cost that the cheaper insurance would impose upon the organization which the more expensive up-front option is saving you. Perhaps even serve that up as a cost-savings the finance department is free to then take credit for, I'unno. :P
> Forced password changes once every 90 days is dumb and slightly annoying but doesn't significantly impact business operations.
It negatively impacts security, because users then pick simpler passwords that are easier to rotate through some simple transformation. Which is why it's considered not just useless, but an anti-pattern.
This is why everyone should have a union, including highly paid professionals. Imagine what it would be like. "No, fuck you, we're going on strike until you stop inconveniencing us to death with your braindead security theater. No more code until you give us admin on our own machines, stop wasting our time with useless Checkmarx scans, and bring the firewall down about ten notches."
> If an insurer says "we're going to jack up premiums by 20% unless you force employees to change their password once every 90 days", you can argue till you're blue in the face that it's bad practice, NIST changed its policy to recommend not regularly rotating passwords over a decade ago, etc., and be totally correct... but they're still going to jack up premiums if you don't do it.
I would argue that password policies are very context dependent. As much as I detest changing my password every 90 days, I've worked in places where the culture encouraged password sharing. That sharing creates a whole slew of problems. On top of that, removing the requirement to change passwords every 90 days would encourage very few people to select secure passwords, mostly because they prefer convenience and do not understand the risks.
If you are dealing with an externally facing service where people are willing to choose secure passwords and unwilling to share them, I would agree that regularly changing passwords creates more problems than it solves.
> removing the requirement to change passwords every 90 days would encourage very few people to select secure passwords
When you don’t require them to change it, you can just assign them a random 16 character string and tell them it’s their job to memorize it.
There's no way I will ever remember it. I will write it down. Let me choose my own password (passphrase if I need to remember it)
> jack up premiums by 20% unless you force employees to change their password once every 90 days"
Always made me judge my company's security teams as to why they enable this stupidity. Thankfully they got rid of this gradually, nearly 2 years ago now (90 days to 365 days to never). New passwords were just one key l/r/u/d on the keyboard.
Now I'm thinking maybe this is why the app for a govt savings scheme in my country won't allow password autofill at all. Imagine expecting a new password every 90 days and not allowing auto fill - that just makes passwords worse.
I believe that these kind of decisions are mostly downstream of security audits/consultants with varying level of up to date slideshows.
I believe that this is overall a reasonable approach for companies that are bigger than "the CEO knows everyone and trusted executives are also senior IT/Devs/tech experts" and smaller than "we can spin an internal security audit using in-house resources"
I'm no expert, but I did take a CISSP course a while ago. One thing I actually remember ;P, is that it recommended long passwords in in lieu of the number, special character, upper, lower ... I don't remember the exact wording of course and maybe it did recommend some of that, but it talked about having a sentence rather than all that mess in 6-8 characters, but many sites still want the short mess that I never will actually remember
While the password recommendation stuff is changing (the US government updating it guidelines last year), it’s generally best practice to not share passwords which itself implies using a password manager anyway which makes the whole “long passphrase” vs “complex” password moot - just generate 32 lowercase random characters to make it easier to type or use the autogenerated password your password manager recommends.
The long passphrase is more for the key that unlocks your password manager rather than the random passwords you use day to day.
There's also login passwords, and depending on how many systems you have to log into, these can be quite numerous. There are some attempts to address this with smartcards and FIDO tokens and so on, but it's not nearly universal yet. At least SSH keys are common for remote login nowadays, but you still need to log into some computer directly first.
I find it rare to have a huge number of machines to log into that aren't hooked up to a centralized login server. Still, nothing prevents you from having passwords for each individual machine that needs it. It's cumbersome to type it in but it works, which is why I recommended all lowercase (faster to type on a mobile device).
entropy is stronger than complexity. https://xkcd.com/936/
I wonder how many people have used Correct Horse Battery Staple as a password thanks to this comic
Worse. If you are not in the USA, i.e., if NIST is not the correct authority, that insurer might actually be enforcing what the "correct" authority believes to be right, i.e., password expiration.
Why wouldn't the IT people just tell the grumbling employees that exact explanation?
IT doesn't always hear the grumbles, hidden away as they frequently are behind a ticketing system; the help desk technicians who do hear the grumbles aren't always informed of the "why" behind certain policies, and don't have the time or inclination to go look them up if they're even documented; and it's a very unsatisfying answer even if one receives a detailed explanation.
Information loss is an inherent property of large organizations.
> Information loss is an inherent property of large organizations.
That's such an interesting axiom, I'm curious if you would want to say more about it? It feels right intuitively - complexity doesn't travel easily across contexts and reaching a common understanding is harder the more people you're talking to.
On a more micro level, I find it very hard to write good documentation. I always forget something that once pointed out seems obvious. Or worse, the reader is missing some important context that many other readers are already privy to. Not to mention, some people don't even seek out docs before acting.
I imagine this gets amplified in a large org. The docs are lacking, people might not read them anyway, and you get an explosion of people who don't understand very much but still have a job to do.
In small orgs that might happen, in large orgs it's some game of telephone where the insurance requirements are forwarded to the security team which makes the policies which are enforced by several layers of compliance which come down on the local IT department.
The underlying purpose of the rules and agency to apply the spirt rather than the letter gets lost early in the chain and trying to unwind it can be tedious.
If you've read this thread, it would appear that most people here on HN aren't actually involved with policy compliance work dictated from above. Have you ever seen a Show HN dealing with boring business decisions? No. We do, however, get https://daale.club/
I think the problem is any time we are involved with policy compliance work it’s because we get a list with inane requirements on, and nobody to challenge in regard to it.
In a lot of cases the it people are just following the rules and don’t know this.
Maybe it wouldn't make a difference, but if I was the IT person telling users they have to change their passwords every 90 days, I would 100% include a line in the email blaming the insurance company.
I'm not in an IT dept (developer instead), but I'd bet money that would get you a thorough dressing down by an executive involved with the insurance. That sort of blaming goes over well with those at the bottom of the hierarchy, and poorly with those at the top.
The insurance people are not a part of the company, so I'm not sure who would be offended.
I wouldn't be mean about it. I'm imagining adding a line to the email such as:
> (Yes, I know this is annoying, but it's required by our insurance company.)
What is the insurance company going to do, jack up our rates because we accurately stated what their policy was?
The problem is that this particular insurance company was picked by someone who does work in yours.
You would probably have no idea what the requirement actually said or where it ultimately came from.
It would've gone from the insurer to the legal team, to the GRC team, to the enterprise security team, to the IT engineering team, to the IT support team, and then to the user.
Steps #1 to #4 can (and do) introduce their own requirements, or interpret other requirements in novel ways, and you'd be #5 in the chain.
"You never know..." is the worst form of security, and makes systems less secure overall. Passwords must be changed every month, just to be safe. They must be 20 alphanumeric characters (with 5 symbols of course), just to be safe. We must pass every 3-letter compliance standard with hundreds of pages of checklists for each. The server must have WAF enabled, because one of the checklists says so.
Ask the CIO what actual threat all this is preventing, and you'll get blank stares.
As an engineer what incentive is there to put effort into knowing where each form input goes and how to sanitize it in a way that makes sense? You are getting paid to check the box and move on, and every new hire quickly realizes that. Organizations like these aren't focused on improving security, they are focused on covering their ass after the breach happens.
> Ask the CIO what actual threat all this is preventing
the CIO is securing his job.
the CIO is securing his job.
Every CIO I have worked for (where n=3) has gotten where they are because they're a good manager, even though they have near-zero current technical knowledge.
The fetishizing of "business," in part through MBAs, has been detrimental to actually getting things done.
A century ago, if someone asked you what you do and you replied, "I'm a businessman. I have a degree in business," you'd get a response somewhere between "Yeah, but what to you actually do" and outright laughter.
It's a relatively recent change, too. Transition from "the executives and managers mostly came up through 10-25 years of doing 'lower' jobs in the company, and very much know how the business actually works" to "we hire MBAs to those roles directly" was throughout the '70s-'90s.
Finance and business grads have really taken over the economy, not just through technocratic "here's how to do stuff" advice but by personally taking all the reigns of power. They're even hard at work taking over medicine and pushing doctors out of the work-social upper-middle-class. Already did it with professors. Lawyers seem safe, so far.
> Lawyers seem safe, so far.
Nope, lawyers are fucked too. It's just not as advanced yet: https://www.abajournal.com/web/article/arizona-approves-alte...
They're taking over veterinary clinics too! The biggest owner of veterinary clinics is Mars inc. the candy company!
This looks like a variation of the Scunthorpe problem[1], where a filter is applied too naively, aggressively, and in this case, to the wrong content altogether. Applying the filter to "other stuff" sent to and among the servers might make sense, but there doesn't seem to be any security benefit to filtering actual text payload that's only going to be displayed as blog content. This seems like a pretty cut and dried bug to me.
This is exactly what I was thinking as well, it's a great Scunthorpe example. Nothing from the body of a user article should ever be executed in any way. If blocking a list of strings is providing any security at all you're already in trouble because attackers will find a way around that specific block list.
> This looks like a variation of the Scunthorpe problem[1], where a filter is applied too naively
No.
> aggressively
No.
>, and in this case, to the wrong content altogether.
Yes - making it not a Scunthorpe problem.
I don't get why you'd have SQL injection filtering of input fields at the CDN level. Or any validation of input fields aside from length or maybe some simple type validation (number, date, etc). Your backend should be able to handle arbitrary byte content in input fields. Your backend shouldn't be vulnerable to SQL injection if not for a CDN layer that's doing pre-filtering.
A simple reason would be if you're just using it as a proxy signal for bad bots and you want to reduce the load on your real servers and let them get rejected at the CDN level. Obvious SQL injection attempt = must be malicious bot = I don't want my servers wasting their time
> A simple reason would be if you're just using it as a proxy signal for bad bots
Who would be that stupid?
It should be thought of as defense-in-depth only. The backend had better be immune to SQL injection, but what if someone (whether in-house or vendor) messes that up?
I do wish it were possible to write the rules in a more context-sensitive way, maybe possible with some standards around payloads (if the WAF knows that an endpoint is accepting a specific structured format, and how escapes work in that format, it could relax accordingly). But that's probably a pipe dream. Since the backend could be doing anything, paranoid rulesets have to treat even escaped data as a potential issue and it's up to users to poke holes.
Because someone said "we need security" and someone else said "what is security" and someone else said "SQL injection is security" and someone looked up SQL injections and saw the word "select" and "insert".
WAFs are always a bad idea (possible exception: in allow-but-audit mode). If you knew the vulnerabilities you'd protect against them in your application. If you don't know the vulnerabilities all you get is a fuzzy feeling that Someone Else is Taking Care of it, meanwhile the vulnerabilities are still there.
Maybe that's what companies pay for? The feeling?
WAFs can be a useful site of intervention during incidents or when high-severity vulns are first made public. It's not a replacement for fixing the vuln, that still has to happen, but it gives you a place to mitigate it that may be faster or simpler than deploying code changes.
If your clients will let you pass the buck on security like this it would be very tempting to work towards the least onerous insurance metric and no further.
Yup. Were a database company that needs to be compliant with SOC2, and I’ve had extremely long and tiring arguments with our auditor why we couldn’t adhere to some of these standard WAF rulesets because it broke our site (we allow people to spin up a demo env and trigger queries).
We changed auditors after that.
sounds like your security policy is wrong (or doesnt have a provision for exceptions managed by someone with authority to grant them), or your auditor was swerving out of his lane. As far as I've seen: SOC2 doesn't describe any hard security controls - it just asks to evaluate your policy versus your implemented controls.
You are absolutely correct, which is why we switched auditors. We use a third party to verify compliance of all our cloud resources (SecureFrame), and one of their checks is that specific AWS WAF rulesets are enabled on e.g. CloudFront endpoints. These are managed rulesets by AWS.
We disabled this check, auditor swerved out of his lane, I spent more several hours explaining things he didn’t understand, and things resolved after our CEO had a call with him (you can imagine how the discussion went).
All in all, if the auditor would have been more reasonable it wouldn’t have been an issue, but I’ve always been wary of managed firewall rulesets because of this reason.
> I disagree with other posts here, it is partially a balance between security and usability. You never know what service was implemented with possible security exploits and being able to throw every WAF rule on top of your service does keep it more secure. Its just that those same rulesets are super annoying when you have a securely implemented service which needs to discuss technical concepts.
I might be out of the loop here, but it seems to me that any WAF that's triggered when the string "/etc/hosts" is literally anywhere in the content of a requested resource, is pretty obviously broken.
I don't think so. This rule for example probably block attacks on a dozen old WordPress vulnerabilities.
And a rule that denies everything blocks all vulnerabilities entirely.
A false positive from a conservative evaluation of a query parameter or header value is one thing, conceivably understandable. A false positive due to the content of a blog post is something else altogether.
This is a strawman, especially if like the parent claims this was improving security for one of the most popular website backends ever.
Rules like this might very well have had incredible positive impact on ten of thousands of websites at the cost of some weird debugging sessions for dozens of programmers (made up numbers obviously).
> The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content
They shouldn't be doing that job at all. The content of user data is none of their business.
In my experience the pain of false positives required to outweigh the "WAF is best practice" is just very very heigh. Most big businesses would rather lose/frustrate a small percentage of customers, to be "safe".
> The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content. It's not just Cloudflare, Akamai has the same problem.
I agree. There is a business opportunity here. Right in the middle of your sentences.
Hint: Context-Aware WAF.
Many platforms have emerged in the last decade - some called it smart WAF, some called it nextgen WAF.. All vaporware garbage that consumes tons and tons of system resource and still manages to do a shit job of _actually_ WAF'ing web requests.
To be truly context-aware, you need to compute a priori about the situation - the user, the page, the interactions etc.
This is what surprises me in this story. I could not, at first glance, assume that either Substack people or Cloudflare people were incompetent.
Oh: I resisted tooth and nail about turning on a WAF at one of my gigs (there was no strict requirement for it, just cargo cult). Turns out - I was right.
> There is a temptation to just turn the rules off
Definitely, though I have seen other solutions, like inserting non-printable characters in the problematic strings (e.g. "/etc/ho<b></b>sts" or whatever, you get the idea). And honestly that seems like a reasonable, if somewhat annoying, workaround to me that still retains the protections.
I've had the issue where filling out form fields for some company website triggers a WAF and then nobody in the company is able to connect me to the responsible party who can fix the WAF rules. So I'm just stuck.
There's no "trade-off" here. Blocking IPs that send "1337 h4x0r buzzword /etc/passwd" in it is completely naive and obtrusive, which is the modus operandi of the CDN being discussed here. There are plenty of other ways of hosting a website.
100! [good] security just doesn't work as a mixing pattern... I'm not saying it's necessarily bad to use those additional protections but they come with severe limitations so the total value (as in cost/benefit) is hard to gauge.
I agree. From a product perspective, I would also support the decision. Should we make the rules more complex by default, potentially overlooking SQL injection vulnerabilities? Or should we blanket prohibit anything that even remotely resembles SQL, allowing those edge cases to figure it out?
I favor the latter approach. That group of Cloudflare users will understand the complexity of their use case accepting SQL in payloads and will be well-positioned to modify the default rules. They will know exactly where they want to allow SQL usage.
From Cloudflare’s perspective, it is virtually impossible to reliably cover every conceivable valid use of SQL, and it is likely 99% of websites won’t host SQL content.
If your web application is relying on Cloudflare filtration of input values to prevent SQL injection, your web application is vulnerable to SQL injection.
Defense in-depth. I would hope few would want a vulnerable web app and simply protect it via a WAF. But just because your web app is 'invulnerable' doesn't mean you should forgo the WAF.
But what is being defended against? This is blocking legitimate user behavior. Would it be defense in depth to also prohibit semicolons or two consecutive hyphen characters in all content? If your app is constructing paths to read from the server's filesystem based on substrings contained within client-provided field values, throwing an error if `"/etc/hosts"` appears in any input is not going to save you.
Unknown or unforeseen attacks. The WAF ruleset can be updated much faster than code. WAFs also provide flexibility in how requests are responded to, or even disallow access from IP ranges, certain browsers, etc.
WAFs do throw false positives and do require adjustments OOTB for most sites, but you’re missing the forest by focusing on this single case.
My defense in depth blocks Content-Length that's a prime number or divisible by 5. Can't be too safe!
What? If I construct my queries the right way (e.g., not concatenating strings together like it's the year 1990), then I never will want a WAF "helping" me by blocking my users because they have an apostrophe in their name.
That's a very narrow view of what a WAF does. You may want to review the OWASP ruleset at https://coreruleset.org/. However, this is just the ruleset. WAF vendors usually offer features above and beyond OWASP rule parsing.
And WAF rules can be tuned. There's no reason an apostrophe in a username or similar needs to be blocked, if it were by a rule.
Okay, I'll look at the "coreruleset" which you say is good.
Let's see what's blocked:
"Division by zero" anywhere in the response body since that's a php error. Good luck talking about math ([0] and [1])
Common substrings in webshells, all matched as strings in response bodies, rather than parsing HTML, so whatever, don't comment about webshells either [2]
Unless the body is compressed, in which case don't apply the above. Security [3].
Also, read this regex and tell me you understand what it's doing. Tell me the author of it understands what it matches: https://github.com/coreruleset/coreruleset/blob/943a6216edea...
What the coreruleset is doing here is trying to parse HTML, SQL, HTTP, and various other languages with Regular Expressions. This doesn't work. This will never give you a right result.
It's trying to keep up to date with the string representation of java and php errors, without even knowing the version of Java the server is running, and without the Java maintainers, who constantly add new errors, having any say.
The only reasons attackers aren't evading the webshell rules here trivially is because so few people use these rules in practice that they're not even worth defeating (and it is quite easy to have your php webshell generate unique html each load, which cannot be matched by a regular expression short of /.*/; html is not a regular grammar).
I was ready to see something that made WAFs feel like they did _anything_ based on your comment, but all I see is a pile of crap that I would not want anywhere near my site.
Filtering java error strings and php error strings out of my rust app's responses using regexes to parse html is just such a clown-world idea of security. Blocking the loading of web-shells until the attacker changes a single character in the 'title' block of the output html seems so dumb when my real problem is that someone could write an arbitrary executable to my server.
Every WAF ruleset I've read so far has made me sure it's a huge pile of snake-oil, and this one is no different.
[0]: https://github.com/coreruleset/coreruleset/blob/943a6216edea...
[1]: https://github.com/coreruleset/coreruleset/blob/943a6216edea...
[2]: https://github.com/coreruleset/coreruleset/blob/943a6216edea...
[3]: https://github.com/coreruleset/coreruleset/blob/943a6216edea...
The issue is you're picking out bits and pieces that /seem/ correct to you, but you don't seem to have experience defending a public website from intrusions.
These rules do in fact work. Like I've said previously, these rules require tuning for your particular website. If I'm "talking about math" then I would modify or disable that rule as needed.
I think this is the forest you're missing. WAF isn't "install it and walk away". WAF needs to be tested in conjunction with your release, like any other code would.
The WAF can and does protect against attacks your code would never think of. It also /logs requests/ in a way your web server will not, making it invaluable for auditing.
And when running 3rd party software that has a function you cannot control, but need to prevent, WAFs can do that, too. I have a particular query string that must work from an internal but not external network while external/internal users leverage the same URL -- WAF can do that with a custom rule examining the query string and denying access to the outside world.
Or if I need to prevent [AI] bot scraping. WAF can do that with a couple of clicks.
WAF also unloads the web server from malicious traffic. Instead of having to size up or out a web server, I can have a WAF appliance prevent that traffic from ever reaching the server.
> Every WAF ruleset I've read so far
You don't appear to have any experience with implementation or operation of a WAF, but are attempting to be authoritative and dismiss a WAFs utility.
Sorry, we have to reject your comment due to security. The text "Cloudflare<apostrophe>s" is a potential SQL injection.
You know, I get the spirit of this criticism. But, specially in the age of AI, we're going to get thousands of barely reviewed websites on Cloudflare.
If you know what you're doing, turn these protections off. If you don't, there's one less hole out there.
In all seriousness, I don't see the justification for blocking "/etc/hosts" but allowing "'". The latter is probably a million times more likely to trigger a vulnerability.
The problem is that people who don't know what they are doing join the cargo cult and then impose these requirements on people who do know what they are doing.
Why not just whitelist the thousand most common words? That should be good enough for 99% of approriate content, and the smelly nerds who make websites or talk about them can take their tiny market segment and get bent.