Setting up Postgres locally, what could go wrong?

There are multiple reasons why someone wants to set up a postgres locally. Either for learning SQL or as an application’s backend. Over the years I see people struggle with using postgres locally, so here are common use cases and possible issues, with solutions for each. For Learning SQL SQL is very common for analysts to use for accessing data from a database, because the data size outgrows Excel. However, SQL is a query language, not a database engine....

December 23, 2023 · 3 min · Karn Wong

Things to watch out for GCP SSL with Cloudflare DNS

For our production workload, we deploy the workloads on Kubernetes, in which an ingress resource is created per each deployment. Resources in ingress are GCP Load Balancer and SSL Certificate. As for DNS, we use Cloudflare since it enables CDN without extra configurations on our part. A few months after the deployment went live initially, we were informed that the website couldn’t be accessed. Turns out GCP couldn’t renew the SSL Certificate (error FAILED_NOT_VISIBLE....

December 18, 2023 · 1 min · Karn Wong

Reduce operational costs with terraform

Background Think of websites you visit each day. Most likely they are hosted on a cloud provider such as AWS, GCP, Azure. The good news is it’s very easy to create a simple deployment with a virtual machine, but for scalable and high-availability workloads, usual recommendations is to use a container-based runtime such as AWS ECS/EKS, GCP Cloud Run/GKE. These services also require more configurations than a simple VM deployment....

November 4, 2023 · 3 min · Karn Wong

Spark on Kubernetes

Background For data processing tasks, there are different ways you can go about it: using SQL to leverage a database engine to perform data transformation dataframe-based frameworks such as pandas, ray, dask, polars big data processing frameworks such as spark Check out this article for more info on polars vs spark benchmark. The problem At larger data scale, other solutions (except spark) can work, but with a lot of vertical scaling, and this can get very expensive....

September 12, 2023 · 4 min · Karn Wong

Create Kubernetes service accounts with Terraform

Sometimes you’ll have to grant other people (or entities) access to your Kubernetes cluster. Easiest is you can give them your admin credentials, but this is similar to giving your house key to a friend, when they only need access to your living room. You can give them different keys, depending on access level required. Those could be readonly access to view services status, or deploy service account that can create/update services....

August 1, 2023 · 3 min · Karn Wong