A 403 that wasn't coming from the server

Saving a page in the site builder returned a 403 and refused to write. The server wasn't the one producing it, and proving that meant getting a request past both the CDN and the load balancer sitting in front of it.

The objective

The client couldn't save edits to one page of their site. The page builder threw a 403 and wouldn't write the change.

A 403 usually means the web server turned the request down, so that's where I started looking. The obvious suspect was a web application firewall, except mod_security wasn't even installed on the machine.

What I did

I went through the server configuration and its logs first and found nothing that would throw that error.

So I sent a request straight to the server's address, past the CDN and the AWS load balancer in front of it. The save went through, which put the block at the edge rather than on the host.

I took read access to their CDN account and looked for the rule, and changed nothing, because a firewall rule on a live site isn't something you test by switching it off.

I couldn't narrow it down to a single rule, and I said so, rather than making some change and calling it the fix.

To get them unblocked, I gave them a hosts file entry pointing the domain at the server's address so they could reach the site directly and make the edit they'd been waiting on. I explained the certificate warning that comes with doing that ahead of time.

The result

The client made their edit the same day. The block was shown to be at the edge, so the rest of the search moved to their CDN and AWS setup instead of grinding on against a server that was behaving fine.