The number one thing that keeps getting hammered home to me, by experience and the folks at work I'm responsible to, is that the point of data/analytics engineering is to
…
read more »
I finally finished my albatross project: replicating all our Postgres shards to Databricks. This is great news.
In broad strokes, we export our Postgres databases daily from the backup snapshot, then ingest those
…
read more »
It's Sunday, and I was able to get some downtime at work to do some upgrades behind the scenes. I've been working on this data pipeline for months, and
…
read more »
I've been a long-time user of Pandas, since I'm a data engineer, but only recently learned about Polars: "DataFrames for the new era". [1]
Polars uses Rust
…
read more »
In my day job, I'm a data engineer at a startup. My goal for this coming year is to design and implement a unified data platform. I'm fleshing out
…
read more »
Ben Werdmuller suggests some startup ideas worth considering:
Metrics in a box. A tool that connects to your analytics, payment processor, newsletter tool, etc, and automatically gives you insights, generates actionable reports on
…
read more »