A forum going down under bot traffic

A sports community board kept falling over, and the hosting company's line was that it used too many resources. The load was actually coming from crawlers hammering search, not from members.

The objective

The board kept going down. The hosting company's answer was that it used more resources than the plan allowed, and that stayed true even after the client moved it onto a VPS.

The client wanted to know what was actually driving it before paying for a bigger server.

What I did

I set up a weekly job to clear out the old posts the client wanted gone, which brought the database size down, and I added a database backup to the same job so a deletion never runs without a copy being taken first. Both email when they run, so a silent failure gets noticed instead of discovered later.

I watched the site under load and found the requests were coming from crawlers, not members.

When the hosting company traced the load to the forum's search script, that lined up. Search is the most expensive page on a phpBB board, and crawlers ask for it constantly.

So I recommended putting the domain behind a CDN and disabling search for guests and bots, which takes the expensive page away from everything except logged-in members.

The result

The search restriction and the weekly cleanup cut down what the server had to carry. What was left was simply more traffic than the hosting could support, and I told them that instead of selling them more optimisation work on top of it.