A generic kubernetes client

clux June 04, 2019 [software] #rust #kubernetes

It's been about a month since we released kube, a new rust client library for kubernetes. We covered the initial release, but it was full of naive optimism and uncertainty. Would the generic setup work with native objects? How far would it extend? Non-standard objects? Patch handling? Event handling? Surely, it'd be a fools errand to write an entire client library?

With the last 0.10.0 release, it's now clear that the generic setup extends quite far. Unfortunately, this yak is hairy, even by yak standards.

Update from 2021

This post is old and many details herein are severely outdated. A few EDIT: markers have been highlighted to point out the biggest changes, but the rest of the post is left unedited for historical reasons. Consider checking for a more recent #kubernetes post.

Overview

The reason this library even works at all, is the amount of homebrew generics present in the kubernetes API.

Thanks to the hard work of many kubernetes engineers, most API returns can be serialized into some wrapper around this struct:

#[derive(Deserialize, Serialize, Clone)]
pub struct Object<T, U> where T: Clone, U: Clone
{
    #[serde(flatten)]
    pub types: TypeMeta,
    pub metadata: ObjectMeta,
    pub spec: T,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub status: Option<U>,
}

You can infer a lot of the inner api workings by looking at apimachinery/meta/types.go. Kris Nova's 2019 FOSDEM talk on [the internal clusterfuck of kubernetes] (https://fosdem.org/2019/schedule/event/kubernetesclusterfuck/) also provides a much welcome, rant-flavoured context.

By taking advantage of this, we can provide a much simpler interface to what the generated openapi bindings can provide. But it requires some other abstractions:

More object patterns

Let's compare some openapi generated structs:

All with identical contents. You could just define this generic struct:

#[derive(Deserialize)]
pub struct ObjectList<T> where
  T: Clone
{
    pub metadata: ListMeta,
    #[serde(bound(deserialize = "Vec<T>: Deserialize<'de>"))]
    pub items: Vec<T>,
}

Similarly, the query parameters optionals structs:

These are a mouthful. And again, almost all of them have the same fields. Not going to go through the whole setup here, because the TL;DR is that once you build everything with the types.go assumptions in mind, a lot just falls into place and we can write our own generic api machinery.

Api machinery

If you follow this rabbit hole, you can end up with the following type signatures:

impl<K> Api<K> where
    K: Clone + DeserializeOwned + KubeObject
{
    fn get(&self, name: &str) -> Result<K>;
    fn create(&self, pp: &PostParams, data: Vec<u8>) -> Result<K>;
    fn patch(&self, name: &str, pp: &PostParams, patch: Vec<u8>) -> Result<K>;
    fn replace(&self, name: &str, pp: &PostParams, data: Vec<u8>) -> Result<K>;
    fn watch(&self, lp: &ListParams, version: &str) -> Result<Vec<WatchEvent<P, U>>>;
    fn list(&self, lp: &ListParams) -> Result<ObjectList<K>>;
    fn delete_collection(&self, lp: &ListParams) -> Result<Either<ObjectList<K>, Status>>;
    fn delete(&self, name: &str, dp: &DeleteParams) -> Result<Either<K, Status>>;

    fn get_status(&self, name: &str) -> Result<K>;
    fn patch_status(&self, name: &str, pp: &PostParams, patch: Vec<u8>) ->Result<K>;
    fn replace_status(&self, name: &str, pp: &PostParams, data: Vec<u8>) -> Result<K>;
}

These are the main query methods on our core Api (docs / src). Observe that similar types of requests take the same *Params objects to configure queries. Return types have clear patterns, and serialization happens before entering the Api.

There's is no hidden de-multiplexing on the parsing side. When calling list, we just turbofish that type in for serde to deal with internally:

self.client.request::<ObjectList<K>>(req)

Where, typically K = Object<P, U>, but actually; K is something implementing a KubeObject trait. This is our one required trait, and you shouldn't don't have to deal with it because of an automatic blanket implementation for K = Object<P, U>.

client-go semantics

While it might not seem like it with all this talk about generics; we are actually trying to model things a little closer to client-go and internal kube apimachinery (insofar as it makes sense).

Just have a look at how client-go presents Pod objects or Deployment objects. There's already a pretty clear overlap with the above signatures.

Maybe you are in the camp with Bryan Liles, who said that "client-go is not for mortals" during his kubecon 2019 keynote. It's certainly a large library (sitting at ~80k lines of mostly go), but amongst the somewhat cruft-filled chunks, it does embed some really interesting patterns to consider.

The terminology in this library should therefore be a lot more familiar now. Not only are using ideas from client-go, our core assumptions come from api-concepts, and we otherwise try to take inspiration from frameworks such as kubebuilder. That said, we are inevitably going to hit some walls when kubernetes isn't as generic as we inadvertently promised it to be.

But delay that tale; let's first look at how to use the Api:

Api Usage

Using the Api now amounts to choosing one of the constructors for the native / custom type(s) you want and use with the verbs listed above.

For Pod objects, you can construct and use such an object like:

let pods = Api::v1Pod(client).within("kube-system");
for p in pods.list(&ListParams::default())?.items {
    println!("Got Pod: {}", p.metadata.name);
}

Here the p is an Object<PodSpec, PodStatus>. This leverages k8s-openapi for PodSpec and PodStatus as the source of these large types.

If needed, you can define these structs yourself, but as an example, let's show how that plays in with CRDs; because custom resources require you to define everything about them anyway.

#[derive(Deserialize, Serialize, Clone)]
pub struct FooSpec {
    name: String,
    info: String,
}

#[derive(Deserialize, Serialize, Clone, Debug, Default)]
pub struct FooStatus {
    isBad: bool,
}

type Foo = Object<FooSpec, FooStatus>;

This is all you need to get your "code generation". No external tools to shell out to; cargo build gives you your json serialization/deserialization, and the generic Api gives you your api machinery.

You can therefore interact with your customResource as follows:

let foos : Api<Foo> = Api::customResource(client, "foos")
    .version("v1")
    .group("clux.dev")
    .within("default");

let baz = foos.get("baz")?;
assert_eq!(baz.spec.info, "baz info");

Here we are fetching and parsing straight into the Foo object on .get().

So what about posting and patching? For brevity, let's use the serde_json macro:

let f = json!({
    "apiVersion": "clux.dev/v1",
    "kind": "Foo",
    "metadata": { "name": "baz" },
    "spec": { "name": "baz", "info": "baz info" },
});
let o = foos.create(&pp, serde_json::to_vec(&f)?)?;
assert_eq!(f["metadata"]["name"], o.metadata.name)

Easy enough, if a tad verbose. What about a patch?

let patch = json!({
    "spec": { "info": "patched baz" }
});
let o = foos.patch("baz", &pp, serde_json::to_vec(&patch)?)?;
assert_eq!(o.spec.info, "patched baz");

Here json! really shines. The macro is actually also so context-aware, that you can reference variables, and even attach structs to keys within.

Higher level abstractions

With the core api abstractions in place, an easy abstraction is Reflector<K>: an automatic resource cache for a K which - through sustained watch calls - ensures its cache reflect the etcd state. We have talked about Reflectors earlier; so let's cover Informers.

EDIT: Informers and Reflectors were deprecated as of 2020 in favour of the kube-runtime crate.

Informers

An informer for a resource is an event notifier for that resource. It calls watch when you ask it to, and it informs you of new events. In go, you attach event handler functions to it. In rust, we just pattern match our WatchEvent enum directly for a similar effect:

fn handle_nodes(ev: WatchEvent<Node>) -> Result<(), failure::Error> {
    match ev {
        WatchEvent::Added(o) => {},
        WatchEvent::Modified(o) => {},
        WatchEvent::Deleted(o) => {},
        WatchEvent::Error(e) => {}
    }
    Ok(())
}

The o being destructured here is an Object<NodeSpec, NodeStatus>. See informer examples for doing something with the objects.

To actually initialize and drive a node informer, you can do something like this:

fn main() -> Result<(), failure::Error> {
    let config = config::load_kube_config().expect("failed to load kubeconfig");
    let client = APIClient::new(config);
    let nodes = Api::v1Node(client);
    let ni = Informer::new(nodes)
        .labels("role=worker")
        .init()?;

    loop {
        ni.poll()?;

        while let Some(event) = ni.pop() {
            handle_nodes(event)?;
        }
    }
}

The harder parts typically come if you need a separate threads; like one to handle polling, one for handling events async, perhaps you are interacting with a set of threads in an tokio/actix runtime.

You should handle these cases, but it's thankfully, not hard. You can just give out a clone of your Informer to the runtime. The controller-rs example shows how trivial it is to encapsulate an informer and drive it along actix (using the 1.0.0 rc). The result is a complete example controller in a tiny alpine image.

Informer Internals

Informers are just wrappers around a watch call that keeps track of resouceVersion. There's very little inside of it:

type WatchQueue<K> = VecDeque<WatchEvent<K>>;

#[derive(Clone)]
pub struct Informer<K> where
    K: Clone + DeserializeOwned + KubeObject
{
    events: Arc<RwLock<WatchQueue<K>>>,
    version: Arc<RwLock<String>>,
    client: APIClient,
    resource: RawApi,
    params: ListParams,
}

If it wasn't for the extra internal event queue (that users are meant to consume), we could easily have built Reflector on top of Informer. The only main difference is that a Reflector uses the events to maintain an up-to-date BTreeMap rather than handing the events out.

As with Reflector, we rely on this foundational enum (now public) to encapsulate events:

#[derive(Deserialize, Serialize, Clone)]
#[serde(tag = "type", content = "object", rename_all = "UPPERCASE")]
pub enum WatchEvent<K> where
    K: Clone + KubeObject
{
    Added(K),
    Modified(K),
    Deleted(K),
    Error(ApiError),
}

You can compare with client-go's WatchEvent.

Drawbacks

So. What's awful?

Everything is camelCase!

Yeah.. #![allow(non_snake_case)]. It's arguably more helpful to be able to easily cross reference values with the main API docs using Go conventions, than to map them to rust's snake_case preference.

That said, we currently rely on k8s-openapi (and that crate maps cases..). Do people have strong feelings about this?

EDIT: This stopped being true in 2020.

Delete returns an Either

The delete verb akwardly gives you a Status object (sometimes..), so we have to maintain logic to conditionally parse those kind values (where we expect them) into an Either enum. This means users have to map_left to deal with the "it's not done yet" case, or map_right for the "it's done" case (crd example). Maybe there's a better way to do this. Maybe we need a more semantically correct enum.

Some resources are true snowflakes

While we do handle the generic subresources like Scale, some objects has a bunch of special subresources associated with them.

The most common example is v1Pod, which has pods/attach, pods/portforward, pods/eviction, pods/exec, pods/log, to name a few. Similarly, we can drain or cordon a v1Node. So we clearly have non-standard verbs and non-standard nouns.

This is probably solveable with some blunt generic_verb_noun hammer on RawApi (see #30) for our supported apis.

It clearly breaks the generic model somewhat, but thankfully only in the areas you'd expect it to break.

EDIT: This stopped being a problem in 2020.

Not everything follows the Spec + Status model

You might think these exceptions make up a short and insignificant list of legacy objects, but look at this subset:

And that was only like 20 minutes in the API docs. Long story short, we eventually stopped relying on Object<P, U> everywhere in favour of KubeObject. This meant we could deal with these special objects in mod snowflake, without feeling too dirty about it..

EDIT: This stopped being a problem in 2020.

Remaining toil

While many of the remaining tasks are not too difficult, there are quite a few of them:

The last one is a huge faff, with differences across providers, all in the name of avoiding impersonating a service accounts when developing locally.

Help

The foundation is now there, in the sense that we feel like we're covering most of the theoretical bases (..that we could think of).

Help with examples/object support/stuff listed above would be greatly appreciated at this point. Hopefully, this library will end up being useful to some. With some familiarity with rust, the generated docs + examples should get you started.

Anyway, if you do end up using this, and you work in the open, please let us link to your controllers for examples.

</🐂💈>

Index
Update from 2021
Overview
More object patterns
Api machinery
- client-go semantics
Api Usage
Higher level abstractions
- Informers
- Informer Internals
Drawbacks
- Everything is camelCase!
- Delete returns an Either
- Some resources are true snowflakes
- Not everything follows the Spec + Status model
- Remaining toil
Help