The Failure I Was Afraid Of
Camlobe is a Django and PostgreSQL pipeline that ingests auto parts catalogs from several suppliers and matches the same part across them. One of its feed readers is an HTTP scraper for a site with a login. The failure I worry about in a scraper is not a crash, because a crash is loud. The bad one is a run that finishes green and has quietly stored less than the site showed, or stored something wrong.
Where It Stops
The scraper ends the run as partial or failed, never as a success, when a page was cut short, a table came back empty or a body is shorter than the site announced. It checks robots.txt, spaces its requests and honours Retry-After. When it meets a block or a challenge it stops and does not try to get past it.
The same rule applies one step later. If a required column, like the part number, the brand or the cost, has no header that maps to it, the run fails before a single row is written. In my sample data one supplier renames a column to something unknown partway through the month. The pipeline refuses that file with the reason on screen, and the other suppliers carry on.
Why Stopping Is Cheap
A refusal only helps if it costs nothing to recover from. Camlobe stores each raw row as it was received and never changes it, and a second run over the same source changes nothing in the catalog. So when the scraper stops, nothing needs cleaning up: the rows it read are kept, and the next run walks the site again from the first page and stores only what is new. That is what makes it safe to prefer stopping over guessing.
What I Tested It Against
Only a local stand-in shop with a login, rate limits and a challenge page, plus a local forwarding proxy. A live site will differ in its markup, its limits and its login, and so will a proxy service. The README says so in the Limits section, and it says the scraper should only be pointed at a site whose owner has allowed it.
The Same Lesson on the Search Side
A reviewer who had not seen the code sampled 400 part numbers as suppliers print them, and 125 of them found nothing in search. The query had been normalised with no brand, so one number kept a prefix it should have lost and another lost a real leading zero. Nothing errored. The search just returned fewer results.
The fix was to try a query in every form a supplier might print it, against both search backends through one function. The same 400 numbers now give no misses on either. It is the same shape as the scraper: the dangerous result is the one that looks like success, so I would rather build the check that refuses it.