Overview
Camlobe turns the messy parts feeds that auto parts suppliers send into one catalog. Each supplier sends its own shape: a CSV with pasted lines in another encoding, an XML file with renumbered parts, files with reordered columns, a SOAP service, a paged REST API, a site behind a login. The pipeline reads them all behind one interface, keeps every raw row as received, works out which rows are the same part and serves the result through REST and a Meilisearch index.
I built it on sample suppliers and part numbers, against local test servers. The README says what ran and what did not.
Key Features
- Five feed adapters behind one interface. A local file, SFTP, paged REST with rate limit and retry, SOAP, and an HTTP scraper that checks robots.txt, spaces its requests, logs in with a CSRF token, uses a proxy and stops at a challenge page instead of getting past it.
- Raw rows stored as received. A database trigger refuses UPDATE and DELETE for ordinary roles, so the history cannot be rewritten by accident.
- Part numbers normalised with recorded rules, and matching that never fuzzes. A row with no brand or a conflicting barcode goes to a review queue instead of being merged.
- EDI X12. An 850 order built from a supplier's own part numbers, and readers for 855, 856 and 810.
- Fitment and search. An ACES style fitment table, REST endpoints, and a Meilisearch index whose rebuild resumes where it stopped.
- Safe to re-run. A second run over the same source changes nothing in the catalog, and a killed run resumes at its checkpoint.
- Background runs and a live runs page. A feed run is a Celery task on Redis: a worker that loses the Postgres lock steps aside, a dropped connection is retried with backoff and a bad file fails at once. The runs table is an AG Grid page that refreshes while a run is going.
Technical Achievements
A feed that is cut inside its last row, a renamed column or a header written twice is refused or ends the run as partial, and a walk whose total moves while it runs ends as partial too, so a skipped row cannot delist a live part. A price that goes from one value to another and back stores three versions, also inside a resumed run. Secrets are removed from every stored error, source name and log line, including the pieces a library cuts a password into.
The suite is 1,025 tests with Hypothesis properties and seeded random worlds that run clean and again with injected failures. A mutation harness of 323 guards breaks one safeguard at a time and requires a test to fail; all 323 were killed in a full run. Three independent adversary rounds found 27 problems, each fixed once at its class with a generated test.
What I Learned
A fix at one site leaves its sibling open. Each round found the same kind of problem again somewhere nearby, until the fix was a test that enumerated the whole class. I also learned to say what a check cannot see: a file cut inside its last cell with every column present looks like a valid file, so that case is a written limit and not a promise.
Tech Stack
Backend: Python 3.13, Django 5.2, Django REST Framework, Celery on Redis Data: PostgreSQL 17, Meilisearch, Hypothesis Frontend: AG Grid Community (the feed runs page) Integration: paramiko (SFTP), zeep (SOAP), requests, beautifulsoup4, EDI X12 Infrastructure: Docker Compose, GitHub Actions