MetaBundle

Data analytics · Gold Coast & Brisbane · 2026

  • My own product
  • Internal tool

A quantitative pipeline over historical event-market data: ingestion, modelling, calibration and monitoring

A Dockerised Python ingester and a Next.js dashboard over licensed historical event-market data. Archives stream into Postgres, a model scores each event, and every output is checked against the real outcome before its numbers count.

This is my own R&D project, not a client engagement.

Product
Quantitative Dashboard (my own)
Industry
Data analytics
Location
Gold Coast & Brisbane
My role
Design and build
Year
2026
Status
Internal tool
Contact me

By · Updated

The brief

I had licensed historical event-market data as compressed archive files, and a single archive ran to 1.77 GB. The question was simple: does a simple model find a real signal in this data, or does it just look like it does?

Raw archive files can't answer that. They had to be parsed, joined and loaded into a database, every model output had to be scored against what actually happened, and the whole pipeline had to be watchable while it ran.

What I built, and why

  1. A streaming ingester, separate from the dashboard

    A Dockerised Python service streams .tar archives of compressed files without unpacking them to disk. For each event it reads the definition and final values, pairs the two related record sets, and writes events, model outputs and resolved outcomes to Postgres with idempotent upserts. Re-running a file changes nothing.

    It runs apart from the dashboard because heavy, long-running parsing doesn't belong inside a web host's short request timeouts. Both share one Postgres database, so historical backtests and current model outputs live in the same place.

  2. Drop a file in a folder

    Drop an archive into a watch folder and the daemon picks it up within 30 seconds. It waits 60 seconds so a half-copied file isn't read, then processes it and moves it to a done folder with a summary file, or leaves it in place with an error file.

    Retry logic handles flaky network writes, and a one-shot mode processes a single archive by hand.

  3. A dashboard that shows its working

    The dashboard shows upcoming events with the model's output and its confidence, a history page that compares past outputs with real outcomes, and a detail page for each event.

    A live ingest progress page shows an archive loading in real time, and diagnostic endpoints expose the raw API responses behind any enrichment.

  4. AI for the messy input

    Not every source offers an export. A screenshot of an event's data can be uploaded, and Claude's vision model extracts the values as structured data in a fixed shape, which feeds straight into the model.

  5. Calibration before any trust

    Every model output is logged and scored once the real outcome lands. A new model's first 100 scored outputs count as calibration: until then, its numbers are a hypothesis, not a result.

    I also keep a written record of every model variant that was tested and failed, and why, so it doesn't get rebuilt by accident.

Results

events ingested per second, with zero errors once retry logic was added
~1/s
Source: Ingester status log, first archive, 2026
  • Built in May 2026: the dashboard (34 commits from 4 May) and the ingester the same week.
  • Partway through its first 1.77 GB archive, the ingester had already written more than 1,000 events with their outcomes resolved.
  • A watch-folder daemon with a settle window, done and error folders and summary files, plus a one-shot mode.
  • Upcoming-events, history and per-event views, screenshot-to-data extraction, on-demand data enrichment, a live ingest monitor and diagnostic endpoints.

The first baseline model didn't hold up on real data, and finding that out cheaply is what the calibration step is for. The current model is still in its calibration period, so its numbers aren't reported here.

I can build something like this for your business.

Contact me

Questions about this project

Can you turn a big archive of raw data files into something I can actually query?

Yes, and that's most of the work here. The ingester streams compressed archives straight into a database without unpacking them, with each record written exactly once, so you can re-run it safely. Once the data is in Postgres, dashboards, reports and models can all read it.

How do you know a model's results are real and not luck?

You make it prove itself on data it hasn't seen, and you keep score. Every model output is logged and scored against what actually happened, a new model gets a calibration period before its numbers count, and failed variants are written down. In this project, real data showed the first baseline model didn't hold up, which saved a lot of misplaced confidence.

Why build the ingester separately from the dashboard?

They have different jobs and different limits. Heavy, long-running parsing doesn't belong in a web host's short request timeouts, so it runs in Docker on my machine and can churn through gigabytes. The dashboard stays fast and simple, and both share one database.

Can AI pull structured data out of screenshots?

Yes. Here a screenshot of an event's data is sent to Claude's vision model, which returns the values in a fixed structure that the rest of the app can calculate with. It's a practical way to get data out of systems that don't offer an export or an API.

Can a data pipeline like this run on my own computer?

Yes. This one runs as a Docker container on a home PC: you drop files into a folder and it does the rest, restarting itself after a reboot. Nothing about it needs a cloud server unless you want one.

Got data stuck in files nobody can query?

I'll build the pipeline that gets it into a database, and a dashboard that shows its working.

Contact me
See AI Assistants & Automation

Related case studies

All work