Every WordPress site gets crawled, scraped, pinged, and probed long before a human ever loads the homepage. Some of that traffic is essential. Some of it is hostile. Most site owners react to “bots” as one undifferentiated threat and either block too aggressively (breaking SEO or monitoring) or ignore everything (letting scrapers and credential stuffers run wild). The fix is not a bigger blocklist. It is learning to read your logs and matching your response to the actual risk.
The Good Bots You Want Around
These are working for you, not against you:
- Search engine crawlers like Googlebot and Bingbot, which need regular access to index your content.
- Uptime and monitoring services that ping your site on a schedule to confirm it is alive.
- Feed readers and social preview bots that fetch a page once to generate a link preview or pull an RSS feed.
- Cloudflare’s own edge processes, which touch your origin as part of normal caching and security operations.
Blocking these by accident is a common self-inflicted wound. A site owner sees an unfamiliar user agent hitting the server repeatedly, panics, and blocks it, only to later discover it was Googlebot re-crawling after a sitemap update.
The Bad Bots Actually Worth Fighting
On the other end of the spectrum:
- Credential stuffing bots hammering
wp-login.phporxmlrpc.phpwith username and password combinations pulled from breach lists. - Content scrapers that copy your posts wholesale, often to republish on spam networks.
- Vulnerability scanners probing for outdated plugin paths, exposed config files, or known exploit endpoints.
- Comment and form spam bots that exist purely to inject links or drop malicious payloads.
These generate real cost: CPU cycles, database load, and in the worst cases, actual compromise. They deserve firm, fast responses.
The Gray Area
This is where most of the judgment calls live. SEO auditing tools (site graders, backlink checkers) crawl aggressively and can look indistinguishable from scrapers in raw logs. AI training and retrieval crawlers are a newer category; some identify themselves honestly and respect robots.txt, others do not. Aggregators and price comparison bots sit somewhere in between, useful to some site owners and unwanted by others.
The right call here depends on your business, not a universal rule. A recipe blog that depends on search discovery treats things differently than a SaaS site protecting proprietary documentation.
Reading Traffic at the Log Level
Your server and access logs are more informative than any dashboard summary. Three signals matter most:
User agent strings
Easy to fake, but still the first filter. A user agent claiming to be Googlebot from an IP that has nothing to do with Google is an immediate red flag.
Reverse DNS
Legitimate search engine bots resolve back to their own infrastructure. Run a reverse lookup on the requesting IP; if it claims to be Googlebot but resolves to a residential ISP or an unrelated hosting provider, it is spoofed.
dig -x 66.249.66.1
Request patterns
Real crawlers respect crawl-delay conventions and spread requests out. Malicious bots often hit dozens of URLs per second, ignore robots.txt entirely, or repeatedly request the same login or admin endpoint. A spike of failed login attempts against wp-login.php in a short window is one of the clearest bad-bot signatures you will see.
If you want a quick pulse check without wading through raw logs, wp-cli can help confirm what is actually happening at the application level, for example checking recent user registrations or plugin activity that a bot campaign may have triggered:
wp user list --fields=ID,user_login,user_registered --orderby=registered --order=DESC
Calibrated Responses, Not One Big Hammer
Once you can tell the difference, match your response to the threat level instead of defaulting to an outright ban:
- Allow and don’t rate limit: verified good bots (confirmed via reverse DNS) hitting normal pages at a normal pace.
- Rate limit, don’t block: gray-area crawlers that are legitimate but too aggressive. A rate limit at the edge keeps them from overwhelming your origin without shutting them out entirely.
- Challenge, don’t block: suspicious traffic that might be automated or might be a real visitor on an unusual network. A JavaScript or managed challenge filters out simple bots while letting humans through with minimal friction.
- Block outright: confirmed credential stuffing, scanning for known exploit paths, or scrapers with no legitimate purpose.
Cloudflare sitting in front of your site gives you these tools without touching WordPress itself: edge caching absorbs a lot of crawl traffic before it ever reaches your origin, the WAF can block known attack signatures, and bot mitigation features let you apply challenges tiered by risk score rather than a flat allow or deny. Keeping your origin IP unpublished also matters here, since a bad actor who cannot find your real server cannot route a scraping or DDoS campaign around your edge protections in the first place.
Watch for Second-Order Effects
Bot traffic that never triggers a security alert can still hurt performance. A crawler hammering hundreds of URLs a minute increases server load, which can push Core Web Vitals like LCP past the 2.5 second threshold for real visitors caught in the same traffic spike. If you run LiteSpeed Cache with full-page caching and a persistent object cache like Redis, most bot-driven requests get served from cache rather than hitting PHP and the database directly, which limits the blast radius considerably. That is one reason object caching pays for itself even outside of obvious security scenarios.
The Takeaway
Bot traffic is not a binary problem. Good bots keep you indexed and monitored, bad bots try to break in or steal your content, and a wide gray area needs a judgment call based on your site’s actual goals. Read your logs, verify identity with reverse DNS, and respond in proportion: allow, rate limit, challenge, or block. That calibration protects your site without accidentally locking out the traffic that helps it.