Archive

2024⁶

April²

Faster spark workloads with comet

April 7, 2024 · 2 min · Karn Wong

Slim down python docker image size with poetry and pip

April 7, 2024 · 2 min · Karn Wong

March¹

Dataframe write performance to Postgres

March 17, 2024 · 2 min · Karn Wong

February¹

How to connect to Cloud SQL from Cloud Run (no, you don’t need a VPC)

February 10, 2024 · 3 min · Karn Wong

January²

What is platform engineering?

January 21, 2024 · 2 min · Karn Wong

Collaboration model for data science projects

January 20, 2024 · 2 min · Karn Wong

2023³⁰

December³

Should data scientists deploy models to production?

December 30, 2023 · 2 min · Karn Wong

Setting up Postgres locally, what could go wrong?

December 23, 2023 · 3 min · Karn Wong

Things to watch out for GCP SSL with Cloudflare DNS

December 18, 2023 · 1 min · Karn Wong

November⁴

Serverless real-time machine learning inference with AWS

November 28, 2023 · 3 min · Karn Wong

Some problems can be solved with workflows

November 24, 2023 · 3 min · Karn Wong

Using Apache Iceberg to reduce data lake operations overhead

November 15, 2023 · 4 min · Karn Wong

Reduce operational costs with terraform

November 4, 2023 · 3 min · Karn Wong

September²

Spark on Kubernetes

September 12, 2023 · 4 min · Karn Wong

Data Engineering Resources

September 9, 2023 · 1 min · Karn Wong

August³

A Networking God Tale: All I Want is to Run a Speedtest Behind a Firewall

August 27, 2023 · 2 min · Karn Wong

Spatial data to QGIS server playbook (yes, this is for prod)

August 10, 2023 · 2 min · Karn Wong

Create Kubernetes service accounts with Terraform

August 1, 2023 · 3 min · Karn Wong

July³

pglogical setup

July 20, 2023 · 2 min · Karn Wong

Book Highlights - Build by Tony Fadell

July 6, 2023 · 4 min · Karn Wong

Book Highlights - Atlas of the Heart by Brene Brown

July 6, 2023 · 1 min · Karn Wong

June¹

Hassle-free Kubernetes monitoring with Coroot

June 9, 2023 · 2 min · Karn Wong

April⁷

Book highlights - Managing Transitions: Making the Most of Change by William Bridges

April 27, 2023 · 2 min · Karn Wong

Use SQL against CSV (or other hard files) without CLI

April 25, 2023 · 2 min · Karn Wong

DevX starts at your local machine

April 22, 2023 · 4 min · Karn Wong

The mythical ChatOps in action

April 18, 2023 · 3 min · Karn Wong

DuckDB vs Polars vs Spark!

April 7, 2023 · 3 min · Karn Wong

Kubernetes with Grafana Cloud

April 1, 2023 · 2 min · Karn Wong

Cost optimization with kubernetes

April 1, 2023 · 2 min · Karn Wong

March⁷

Bare metal works, until it doesn’t. Hello, cloud.

March 24, 2023 · 4 min · Karn Wong

Google Analytics v4 ingestion via BigQuery

March 19, 2023 · 7 min · Karn Wong

SecretOps with teller

March 19, 2023 · 2 min · Karn Wong

Data transformation - python vs sql showdown

March 18, 2023 · 1 min · Karn Wong

Terraform RDS module with DNS setup

March 17, 2023 · 2 min · Karn Wong

Book highlights - Engineering Management for the Rest of Us by Sarah Drasner

March 16, 2023 · 4 min · Karn Wong

Book highlights - The Staff Engineer’s Path by Tanya Reilly

March 15, 2023 · 12 min · Karn Wong

2022¹⁴

December²

Load balancer 101, thou shalt not be alone

December 10, 2022 · 2 min · Karn Wong

Cross-platform package (+env) management with Nix

December 3, 2022 · 4 min · Karn Wong

November⁴

Load credentials into your shell via Bitwarden CLI - Fish edition

November 29, 2022 · 1 min · Karn Wong

Cloudflare DNS management with Terraform

November 20, 2022 · 1 min · Karn Wong

Deploy more efficiently with templating

November 5, 2022 · 3 min · Karn Wong

IaC is cool, until someone specified the wrong machine size 💸

November 3, 2022 · 3 min · Karn Wong

October²

Deploy static site with branch preview via Cloudflare Pages

October 5, 2022 · 2 min · Karn Wong

Terraform with ECS task on EC2 backend

October 4, 2022 · 2 min · Karn Wong

September²

Intro to Dagster Cloud

September 27, 2022 · 3 min · Karn Wong

Intro to Pulumi

September 26, 2022 · 9 min · Karn Wong

August²

Minimal ECS task with fargate backend

August 26, 2022 · 3 min · Karn Wong

Data engineer archtypes

August 26, 2022 · 2 min · Karn Wong

May¹

What SQL can’t do for data engineering

May 15, 2022 · 3 min · Karn Wong

February¹

Use SSH key during docker build without embedding the key via ssh-agent

February 6, 2022 · 2 min · Karn Wong

2021¹⁹

December²

Use pyspark locally with docker

December 21, 2021 · 3 min · Karn Wong

Reduce docker image size with alpine

December 19, 2021 · 1 min · Karn Wong

November²

Secrets management with SOPS, AWS Secrets Manager and Terraform

November 30, 2021 · 4 min · Karn Wong

Run GitHub Actions faster with cache for pipenv and docker build

November 9, 2021 · 1 min · Karn Wong

October¹

ecs-cli snippets

October 8, 2021 · 1 min · Karn Wong

August¹

Self-hosting primer

August 22, 2021 · 2 min · Karn Wong

July¹

Python venv management

July 2, 2021 · 4 min · Karn Wong

June³

Don’t write large table to postgres with pandas

June 27, 2021 · 1 min · Karn Wong

Data engineering toolset (that I use) glossary

June 4, 2021 · 2 min · Karn Wong

Automatic scrapy deployment with GitHub actions

June 2, 2021 · 2 min · Karn Wong

May¹

Elasticsearch with custom dictionary

May 3, 2021 · 1 min · Karn Wong

April²

Shapefile to data lake

April 23, 2021 · 2 min · Karn Wong

Spark join OOM fix

April 11, 2021 · 1 min · Karn Wong

March²

Add Ghost content to Hugo

March 31, 2021 · 1 min · Karn Wong

Hello Caddy

March 7, 2021 · 1 min · Karn Wong

February²

Password auth with apache2 reverse-proxy

February 22, 2021 · 1 min · Karn Wong

Buying tea when you have Celiac

February 14, 2021 · 2 min · Karn Wong

January²

Workarounds for archiving large shapefile in data lake

January 31, 2021 · 2 min · Karn Wong

Mongodb export woes

January 27, 2021 · 2 min · Karn Wong

2020⁶

December²

CPU upgrade is a breeze, only if you know how

December 20, 2020 · 2 min · Karn Wong

There are caveats behind every statement

December 1, 2020 · 1 min · Karn Wong

September¹

Visualizing map region prefix/suffix

September 3, 2020 · 4 min · Karn Wong

May¹

Impute pipelines

May 22, 2020 · 8 min · Karn Wong

April¹

Word-based analysis with song lyrics

April 15, 2020 · 8 min · Karn Wong

2024 6

April 2

Faster spark workloads with comet

Slim down python docker image size with poetry and pip

March 1

Dataframe write performance to Postgres

February 1

How to connect to Cloud SQL from Cloud Run (no, you don’t need a VPC)

January 2

What is platform engineering?

Collaboration model for data science projects

2023 30

December 3

Should data scientists deploy models to production?

Setting up Postgres locally, what could go wrong?

Things to watch out for GCP SSL with Cloudflare DNS

November 4

Serverless real-time machine learning inference with AWS

Some problems can be solved with workflows

Using Apache Iceberg to reduce data lake operations overhead

Reduce operational costs with terraform

September 2

Spark on Kubernetes

Data Engineering Resources

August 3

A Networking God Tale: All I Want is to Run a Speedtest Behind a Firewall

Spatial data to QGIS server playbook (yes, this is for prod)

Create Kubernetes service accounts with Terraform

July 3

pglogical setup

Book Highlights - Build by Tony Fadell

Book Highlights - Atlas of the Heart by Brene Brown

June 1

Hassle-free Kubernetes monitoring with Coroot

April 7

Book highlights - Managing Transitions: Making the Most of Change by William Bridges

Use SQL against CSV (or other hard files) without CLI

DevX starts at your local machine

The mythical ChatOps in action

DuckDB vs Polars vs Spark!

Kubernetes with Grafana Cloud

Cost optimization with kubernetes

March 7

Bare metal works, until it doesn’t. Hello, cloud.

Google Analytics v4 ingestion via BigQuery

SecretOps with teller

Data transformation - python vs sql showdown

Terraform RDS module with DNS setup

Book highlights - Engineering Management for the Rest of Us by Sarah Drasner

Book highlights - The Staff Engineer’s Path by Tanya Reilly

2022 14

December 2

Load balancer 101, thou shalt not be alone

Cross-platform package (+env) management with Nix

November 4

Load credentials into your shell via Bitwarden CLI - Fish edition

Cloudflare DNS management with Terraform

Deploy more efficiently with templating

IaC is cool, until someone specified the wrong machine size 💸

October 2

Deploy static site with branch preview via Cloudflare Pages

Terraform with ECS task on EC2 backend

September 2

Intro to Dagster Cloud

Intro to Pulumi

August 2

Minimal ECS task with fargate backend

Data engineer archtypes

May 1

What SQL can’t do for data engineering

February 1

Use SSH key during docker build without embedding the key via ssh-agent

2021 19

December 2

Use pyspark locally with docker

Reduce docker image size with alpine

November 2

Secrets management with SOPS, AWS Secrets Manager and Terraform

Run GitHub Actions faster with cache for pipenv and docker build

October 1

2024⁶

April²

March¹

February¹

January²

2023³⁰

December³

November⁴

September²

August³

July³

June¹

April⁷

March⁷

2022¹⁴

December²

November⁴

October²

September²

August²

May¹

February¹

2021¹⁹

December²

November²

October¹

August¹

July¹

June³

May¹

April²

March²

February²

January²

2020⁶

December²

September¹

May¹

April¹

January¹

2019¹

November¹

2018³

October¹

July²

2017²

November¹

September¹