/apify-blocked-scrape-triage

name: apify-blocked-scrape-triage
description: Diagnose and recover a scrape that is blocked, throttled, or served empty or partial content, in cost order, before paying for a heavier Actor. Use when a run returns 0 items, when a site answers 403, 429 or a challenge page, when the HTML comes back empty or truncated, when a request fails with no response at all, when the page shows data in a browser that the Actor does not see, or when a scraper that worked yesterday suddenly returns nothing. Triggers - "I'm getting blocked", "403 from the site", "429 rate limited", "Cloudflare challenge", "scraper returns empty results", "connection reset", "page loads in my browser but not in the Actor", "do I need residential proxies", "should I switch to a browser", "my scraper broke overnight". Escalates in cost order - a cheaper published source first, then reachability, request rate and IP spread, request fidelity, session tokens, the site's own internal API, browser last.
author: Mikhail Koviazin
author_url: https://github.com/mikhail-koviazin
metadata:
  category: actor-development
  keywords: "blocked, 403, 429, rate-limit, anti-bot, bot-detection, cloudflare-challenge, proxy, residential-proxy, session, empty-results, troubleshooting, web-data"

Blocked scrape triage

A scrape came back wrong. Find out why before changing anything, then escalate in cost order. Most "blocks" are not bot detection, some are not even a live server, and the browser is the most expensive answer rather than the first one.

Work the steps in order. Step 0 ends more investigations than the whole ladder does.

Prerequisites

• Apify account ([sign up](https://apify.com))
• Authentication via one of:
• `apify login` (OAuth, if using the Apify CLI)
• `APIFY_TOKEN` environment variable
• Token from [Apify Console → Settings → Integrations](https://console.apify.com/settings/integrations)
• **A one-time permission approval, before the first diagnostic run.** `apify/cheerio-scraper` requires full account access and refuses to start until that is approved in the Console. Both paths refuse, with different wording: MCP `call-actor` answers `requires full access to your account. You must approve its permissions before running it`, while the CLI prints `Error: Actor apify/cheerio-scraper requires full access to your Apify account and has not been approved yet.` **on stdout with exit code 0** — so an unattended CLI flow reads it as a normal result and stops without saying why. Approve it once at the URL the message prints, before you need a diagnosis. The same applies to `apify/web-scraper` and the browser Actors in Rung 5; `apify/website-content-crawler` does not require it.

What a diagnosis is allowed to cost

**One DNS pair, one navigation per egress per host, then stop and conclude.** The unit is the host, not the organisation: one county published its records across three hostnames that refused a request in three different ways, and a budget spent on "the target" would have bought one of those three answers. The budget counts requests to hosts the target owns; looking something up on a third-party platform, a data catalogue or your own IP check is free. If you are four requests into one host and still cannot name the cause, you are no longer diagnosing, you are probing, and you should say what you know and what it would cost to learn more. One navigation means one page: a browser pulling forty subresources for that page is still one navigation, and a burst of repeated probes is not. The one exception worth spending on is the residential probe in Step 3, a single run that separates "your network" from "actually defended".

Sample before you commit to a fix, change one variable at a time, and remember that your diagnosis is traffic the target counts. Residential traffic and browser Actors are the two line items that surprise people.

**One variable includes time.** Measurements taken from one egress at 05:44 and from another at 05:57 differ by network *and* by half an hour, and a refusal that expired on its own is indistinguishable from one your second egress defeated. When you compare two egresses, interleave them inside the same minute. This bit a real investigation: three hosts looked like a clean "the network is the cause" result until the blocked egress was re-measured later and two of the three answers had changed.

Know your own vantage point first

Two requests, both free, both about you rather than the target. Skipping them is how a problem with your own address becomes a proxy purchase.

**1. What your egress looks like from outside.** Not its class in the abstract, the actual address, network and country the site sees:

curl -s "http://ip-api.com/json/?fields=query,country,as,org,hosting"

That service is free for non-commercial use and rate limited, and it tells you where you stand in its own headers: a live call returned `X-Rl: 44` (calls left in the window) with `X-Ttl: 60` (seconds until reset). Read them rather than discovering the limit as a failure, and for anything regular use a source whose terms fit your use. When you only need the address, the country and which POP you land on, an endpoint with no such conditions does that much:

curl -s "https://www.cloudflare.com/cdn-cgi/trace"

Measured side by side from one egress: the trace returned `ip=64.176.60.193`, `loc=JP` and `colo=KIX`, while the first call added what the trace does not carry, `AS20473 The Constant Company` and `"hosting": true`. The ASN and the hosting flag are the parts that predict a refusal, so the cheaper endpoint is a supplement rather than a replacement.

`"hosting": true` means every WAF you meet has classified you before reading a single header of yours, and that a `403` from a public site is the expected answer rather than a surprise. **Egress** below means exactly this: the network your request leaves from, as the site sees it. Two egresses of the same class, a hosting IP and a datace