Prometheus Stack Review

operating the stateful metrics system on kubernetes

As part of my work life in the past year, a chunk of my day-to-day life has consisted of maintaining a prometheus installation on top of a sizable kubernetes cluster. My original feeling was “this is not that bad with kube-prometheus-stack”, but this sentiment has worsened somewhat with the realisation that more and more customizations and pieces were needed for large scale use. Half a year later (and 6+ charts deep), I thought I’d collect my thoughts on the ecosystem - from an operational perspective - with a rough architecture overview post.

[Read More]

A generic kubernetes client

Shaving a yak for a client-rust

It’s been about a month since we released kube, a new rust client library for kubernetes. We covered the initial release, but it was full of naive optimism and uncertainty. Would the generic setup work with native objects? How far would it extend? Non-standard objects? Patch handling? Event handling? Surely, it’d be a fools errand to write an entire client library?

With the last 0.10.0 release, it’s now clear that the generic setup extends quite far. Unfortunately, this yak is hairy, even by yak standards.

[Read More]

Kubernetes operators in rust

Writing light weight cloud services without go

When interacting with kubernetes it’s generally been standard practice to use either client-go via go, or kubectl via shell.

While these are good, non-controversial choices, the advancement of client libraries, and smarter openapi bindings, combined with the generics and procedural macros of rust-lang, it’s now quite possible to write fully fledged kube operators, using slim rust kube clients.

[Read More]

Impersonating kube service accounts

Bypassing complicated kubernetes identity providers

Authenticating with large kubernetes clusters often risks you dealing with complicated provider logic and sometimes policies outside your control.

While controllers and operators authenticate with service accounts directly, this is only true inside the cluster. That is, unless you can impersonate the service account from outside.

[Read More]

shipcat introduction

Building a secure yaml api for kubernetes

At babylon health we have a ton of microservices running on kubernetes that are, in turn, controlled by hundreds of thousands of lines of autogenerated yaml.

So for our own sanity, we built shipcat - a standardisation tool (powered by rust-lang and serde) to control the declarative format and lifecycle of every microservice.

[Read More]