xerlane

Loads Primavera XER schedules, cost and commitment exports and Autodesk model data into PostgreSQL per client under forced row level security, reports earned value, and gives each client's staff a login to their own data.

953 tests, 1,757 of 1,757 mutants caught, 65,686 rows loaded in about 3 seconds

Overview

xerlane loads Primavera XER schedule exports, cost and commitment exports and Autodesk model data into PostgreSQL for a project controls team that serves several clients, reports earned value from them, and gives each client's staff a login to their own projects. The file format is a tab separated text export with its own table markers, and the loader reads it line by line, turns seven tables into typed rows and keeps the rest as raw rows.

It is tested against generated schedules with planted defects, and the parts Oracle does not publish about the format are listed in the repository instead of guessed.

Key Features

Per client from the first row. A tenant id is part of every key and every table has forced row level security, so no tenant setting means no rows.
Bad rows are set aside, never dropped. A wrong field count, a bad number or date, an unknown marker or a missing end marker becomes a named problem with its line number.
Loading twice changes nothing. One transaction per project, and a newer version of a schedule writes a change report: tasks added, removed, finish moved, float changed.
An Autodesk Platform Services connector. It pulls a translated model's object properties through the Model Derivative API into the same schema, with a change log, a quarantine, row level security and a join to the XER tasks by task code. It was written from Autodesk's documentation and tested against a stand-in server that follows it.
Cost and commitment exports. Two CSV layouts load with the same quarantine, no-op reload and reconcile, and an earned value report joins budget and actual cost to schedule progress by task, cost code and model element.
A client app with a login. A Nuxt and TypeScript app where a client's staff sign in and see only their own projects; the server takes the tenant from the session and the database's forced row level security does the separation, with administrators adding and disabling users.
A Rust tokenizer, checked against the Python parser. rust/xer-tokenizer/ is a dependency free Rust re-implementation of the XER file structure (tables, fields, rows and the structural problem codes), with 19 cargo tests, clippy with warnings denied, a format check and a CI job. A conformance script runs the Rust and Python readers over 461 files, including generated schedules, planted defects, the 6 MB benchmark file and 300 randomly damaged files, and found no differences; two deliberate breakages turned it red. On the benchmark file the Rust tokenizer takes about 60 ms against about 755 ms for the Python parse in one process, which also hashes the file and converts types, so the two are not like for like. It is a tokenizer, not a loader, and nothing here has used it on a real export.
A reconcile that can fail. It reads the file and the database separately and shows the data equals the file, and the tests damage the stored data on purpose to see it catch one row, one cent and one character.

Technical Achievements

953 tests pass across the Python suite and the web app, including property tests, and 1,757 of 1,757 mutants across eight modules were caught when each guard was broken one at a time. A 6 MB file of 65,686 rows loads in about 3 seconds. The first mutation pass left 90 survivors, and each got a test that kills it.

What I Learned

A check is only worth trusting once it has been seen to fail. The reconcile step and the isolation test both ship with a case that damages the data or turns the protection off, so a green result means something.

Tech Stack

Python, TypeScript, Rust, Nuxt, PostgreSQL with forced row level security, Docker, pytest, Vitest, Hypothesis, cargo and GitHub Actions.

xerlane

953 tests, 1,757 of 1,757 mutants caught, 65,686 rows loaded in about 3 seconds

Overview

It is tested against generated schedules with planted defects, and the parts Oracle does not publish about the format are listed in the repository instead of guessed.

Key Features

Per client from the first row. A tenant id is part of every key and every table has forced row level security, so no tenant setting means no rows.

Bad rows are set aside, never dropped. A wrong field count, a bad number or date, an unknown marker or a missing end marker becomes a named problem with its line number.

Loading twice changes nothing. One transaction per project, and a newer version of a schedule writes a change report: tasks added, removed, finish moved, float changed.

An Autodesk Platform Services connector. It pulls a translated model's object properties through the Model Derivative API into the same schema, with a change log, a quarantine, row level security and a join to the XER tasks by task code. It was written from Autodesk's documentation and tested against a stand-in server that follows it.

Cost and commitment exports. Two CSV layouts load with the same quarantine, no-op reload and reconcile, and an earned value report joins budget and actual cost to schedule progress by task, cost code and model element.

A client app with a login. A Nuxt and TypeScript app where a client's staff sign in and see only their own projects; the server takes the tenant from the session and the database's forced row level security does the separation, with administrators adding and disabling users.

A Rust tokenizer, checked against the Python parser. rust/xer-tokenizer/ is a dependency free Rust re-implementation of the XER file structure (tables, fields, rows and the structural problem codes), with 19 cargo tests, clippy with warnings denied, a format check and a CI job. A conformance script runs the Rust and Python readers over 461 files, including generated schedules, planted defects, the 6 MB benchmark file and 300 randomly damaged files, and found no differences; two deliberate breakages turned it red. On the benchmark file the Rust tokenizer takes about 60 ms against about 755 ms for the Python parse in one process, which also hashes the file and converts types, so the two are not like for like. It is a tokenizer, not a loader, and nothing here has used it on a real export.

A reconcile that can fail. It reads the file and the database separately and shows the data equals the file, and the tests damage the stored data on purpose to see it catch one row, one cent and one character.

Technical Achievements