We recently configured our CDN and App Gateway for a new client using Coveo. Suddenly, Coveo stopped working. Not like “Oh man this is slow and sorta funky” but more of “Oh hey, literally nothing is working.” We were not having fun.

https://media.makeameme.org/created/sigh-youre-no-959a3b7104.jpg
Baby Yoda was sad!

Looking at Dev Tools in Firefox, pretty much every request going to the Coveo rest API (/coveo/rest) was returning a 403, Forbidden. Going to the App Service directly was fine. It had always been fine. It’s how we passed QA for crying out loud! But going to the App Gateway through the CDN.. 403-a-plenty.

Oprah handing out 403s

Obviously this wouldn’t do. After chatting with a coworker, I decided to check out the Firewall. First thing you should know about the Azure Gateway’s Firewalling. There’s a LOT of rules. You can read about every single one of them here: https://docs.microsoft.com/en-us/azure/web-application-firewall/ag/application-gateway-crs-rulegroups-rules?tabs=owasp31 Ok, so yeah, there’s a lot. Which one do you turn off? Go one at a time and see when it starts working? What if it’s multiple rules? After all, your WAF adds up infractions and then if the score goes high enough, blocks the request. While fancy, this is a royal pain in the ass. So how do you go about figuring out which rule is borking your request? Turn on Diagnostics! Luckily, this is fairly straightforward. You simply go into the Gateway and click on Diagnostics. Add in a new Diagnostic Setting:

Be sure to set your Destination to Storage. It’s a lot easier to go read that sucker directly.

Ok, now that you’ve got your logging setup, you’re going to want to wait a few minutes (give it like 10 to be sure). Now is a great time for a cold beverage of the adult nature. Or not. Maybe tea is your deal.

It kinda is, but whatever

Ok, now you’re logging is in place, you need to go trigger the error. Reload a page, start some autosuggest, whatever it takes to start seeing the 403s flood your beautiful site. Great. You’ve done something critical. You’ve failed. It’s a good fail though.

I miss XP 🙁

Now, head on over to the storage container you specified in the Diagnostic setup. You’ll see a new Container in place. Keep clicking that bugger until you get to a .json file. That file has your logs! You want this! Ok, now you’re going to want to pop that sucker open and view the JSON data. Note: each line is its own JSON object. This isn’t going to tidy nicely. So do a quick peruse over the file. You’ll see an “action” attribute in there with either a “Matched” or a “Blocked” value. Matching means it hit a criteria to accrue some points. Blocked is…you guess it: blocked. You don’t want blocked. You probably don’t even want matched. Here’s what it looks like:

Don’t hate my editor settings, folks.

Those two lines are from the same request. The Matched action (920480) gave the request enough points that it ended up getting blocked. So now that we understand the process, how do we go about tweaking things until they work? Well two things:

  1. I’m not your security team. Consult them or your client before you tweak a firewall. I’m not responsible for anything you do that lets the world hack your face off. Seriously ™.
  2. This is what worked for me. It may not work for you. Your mileage may vary.

Ok, so to turn off rules, you’ll want to go into your Firewall Profile: Click your Gateway -> Web application firewall -> Policy -> Managed Rules. Note: We are using OWASP 3.1 here. Different OWASP levels have different rules. This is what your rules screen looks like:

Not that many!

Clicking on any of those is going to show you the actual rules…

Ahhh damn that’s a lot.

Check the box of the rule you want to turn off, and then scroll up and hit “Save” Now you should be able to test whether or not things got better. Run the test again, check your logs. It’s a little bit of a trial and error. To save you some time, I’ve put the list of rules that I’ve disabled below. Hopefully this helps save you some time!

920480Restrict charset parameter within the content-type header
942200Detects MySQL comment-/space-obfuscated injections and backtick termination
942260Detects basic SQL authentication bypass attempts 2/3
942330Detects classic SQL injection probings 1/2
942340Detects basic SQL authentication bypass attempts 3/3
942370Detects classic SQL injection probings 2/2
942430Restricted SQL Character Anomaly Detection (args): # of special characters exceeded (12)
932110Remote Command Execution: Windows Command Injection
932115Remote Command Execution: Windows Command Injection
933180PHP Injection Attack: Variable Function Call Found
941160NoScript XSS InjectionChecker: HTML Injection
941200XSS using VML frames
941310US-ASCII Malformed Encoding XSS Filter – Attack Detected.
941350UTF-7 Encoding IE XSS – Attack Detected.
941330IE XSS Filters – Attack Detected.
941340IE XSS Filters – Attack Detected.
942120SQL Injection Attack: SQL Operator Detected
942210Detects chained SQL injection attempts 1/2
These worked for me. Based on your logs, you may need to tweak things

So there it is. It feels a little dirty, but we’ve already sanitized all our database inputs, right? Would hate for little Bobby Tables to come along and wreck your day. Good luck and happy Sitecore and Coveo-ing!

Facebooktwitterredditlinkedinmail