The Challenge
A browser automation tool has to run workflows reliably, show the user what is happening while they run, and survive a page that never loads or a browser that crashes. I wanted that on free-tier infrastructure.
Architecture Decisions
A Queue Between the API and the Browsers
Running a browser inside an API request is a bad idea, because the request can take minutes and a crash takes the server with it. So the API only validates the request and puts a job on a BullMQ queue backed by Redis. A separate worker picks the job up and drives Playwright.
That gave me retries for free. A failed job is retried up to three times with exponential backoff, and the worker processes only two jobs at a time, which keeps the memory use of the browsers predictable on a small instance.
Live Monitoring over WebSocket
Users should not have to refresh a page to see a run progress, so the worker's events go out over WebSocket. Each execution has its own room, and a client only receives the events for the run it is watching.
A busy workflow can produce many log lines quickly. The server collects the log lines for an execution and sends them as one batch once a second, and it also caps the number of messages it sends per second. The browser gets a steady stream instead of a flood. The server also disconnects sockets that have been idle for half an hour.
State in the Database, Not in Memory
Each run is recorded in PostgreSQL with its status and logs, and access is limited per user with row-level security. If the worker restarts in the middle of a run, the record still says what happened.
What I Learned
- Failure handling is most of the job. A happy path is easy to build. Pages hang and browsers crash, and each of those has to end in a state the user can read.
- A queue and a WebSocket fit together well. The queue owns the work, the socket only reports on it, and neither needs to know much about the other.
- Free tiers set the design. Limited concurrency and batched messages both came from needing to run on a small instance.
AutoFlow Pro has no hosted copy any more (its free-tier backend was suspended), so the code is the thing to read: GitHub.