Global Owner

This is an example controller implementation using metacontroller for a Custom Resource called GlobalOwner, which is responsible for setting ownership on any kubernetes native or custom resources across the cluster.

Examples

This example presents the functionality by adopting ConfigMaps matching label selector expression and setting ownership on those resources across the cluster, but the same process could be applied to any resource kind existing in the cluster. For each resource type the permissions to own the object will be granularly adjusted.

Steps

Steps

To create an example GlobalOwner and a ConfigMap:

kubectl apply -f example

A GlobalOwner resource will be created and will adopt all resources matching the given GVKs provided in the spec.childResources field assuming they match the label selector spec.selector value.

This example shows how a GlobalOwner resource could be used to adopt all ConfigMaps that have an adopt label specified.

Example ConfigMap and a Secret resource located in example/example-resource-set.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test
  namespace: default
  labels:
    adopt: "true"
data:
  some: "value"
  other: "value"
---
apiVersion: v1
kind: Secret
metadata:
  name: test
  namespace: default
  labels:
    adopt: "true"
data: {}

After these resources are applied in the cluster, the ownership reference should be set on the resource.

$ kubectl get globalowner -o yaml

returns

apiVersion: globalowner.metacontroller.io/v1alpha1
kind: GlobalOwner
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"globalowner.metacontroller.io/v1alpha1","kind":"GlobalOwner","metadata":{"annotations":{},"name":"global-owner"},"spec":{"selector":{"matchExpressions":[{"key":"skip","operator":"DoesNotExist"}]}}}
  creationTimestamp: "2023-09-11T11:18:25Z"
  generation: 1
  name: global-owner
  resourceVersion: "3436"
  uid: 53f365d2-4c8e-469b-b09a-62994a968f8f
spec:
  childResources:
  - apiVersion: v1
    resource: secrets
    namespace: default
    names:
    - test
  - apiVersion: v1
    resource: configmaps
status:
  observedGeneration: 1

$ kubectl get cm test -n default -o yaml

will show that a ConfigMap has an OwnershipReference pointing to the GlobalOwner resource.

apiVersion: v1
data:
  other: value
  some: value
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"other":"value","some":"value"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"adopt":"true"},"name":"test","namespace":"default"}}
  creationTimestamp: "2023-09-11T12:23:41Z"
  labels:
    adopt: "true"
  name: test
  namespace: default
  ownerReferences:
  - apiVersion: globalowner.metacontroller.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: GlobalOwner
    name: global-owner
    uid: 6a3269db-f713-41cb-8111-6155f2c2b4b7
  resourceVersion: "1602"
  uid: e3aa2bc8-7615-44de-a788-e6a5296b47bb

API

This section describes APIs used to interact with the Global Owner controller.

Global Owner

This document describes the Global Owner resource.

Group Owner

This document describes the internal Group Owner resource role, resource fields and behaviour in the cluster.

Global Owner resource

This is an API specification for the GlobalOwner resource.

Resources

This is a list of structures representing the GlobalOwner resource API.

`GlobalOwner`

type GlobalOwner struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata"`

	Spec   GlobalOwnerSpec   `json:"spec"`
	Status GlobalOwnerStatus `json:"status,omitempty"`
}

`GlobalOwner` spec

type GlobalOwnerSpec struct {

	// Global label selector value. Will be applied to all child resources
	// if resource does not have its own.
	Selector       *metav1.LabelSelector `json:"selector,omitempty"`

	// A list of child resources which global owner should adopt.
	// Each child resource is passed to generated Group Owner object spec.
	//
	// The order of the resources in the list specifies removal ordering,
	// top to bottom.
	ChildResources []ChildResource       `json:"childResources"`
}

`GlobalOwner` resource rule

type ResourceRule struct {

	// Resource api version, for example: v1
	APIVersion string `json:"apiVersion"`

	// Resource plural name, for example: secrets
	Resource   string `json:"resource"`
}

`GlobalOwner` child resource

type ChildResource struct {
	ResourceRule `json:",inline"`

	// A list of unique names for the resources to adopt
	// Mutually exclusive with the selector value.
	Names     []string              `json:"names,omitempty"`

	// A namespace, where the resources should be looked up.
	Namespace string                `json:"namespace,omitempty"`

	// Label selector value for resource group.
	// Mutually exclusive with the names and namespace values.
	// Has precedence on Names/Namespaces if specified.
	Selector  *metav1.LabelSelector `json:"selector,omitempty"`
}

`GlobalOwner` status

type GlobalOwnerStatus struct {
	ObservedGeneration int `json:"observedGeneration,omitempty"`
}

Group Owner resource

This is an API specification for the GroupOwner resource.

Resources

This is a list of structures representing the GroupOwner resource API.

`GroupOwner` resource

type GroupOwner struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata"`

	Spec   GroupOwnerSpec   `json:"spec"`
	Status GroupOwnerStatus `json:"status,omitempty"`
}

`GroupOwner` spec

ChildResource

type GroupOwnerSpec struct {
	Selector      *metav1.LabelSelector `json:"selector,omitempty"`
	ChildResource ChildResource         `json:"childResource"`
}

`GroupOwner` status

type GroupOwnerStatus struct {
	OwnedResources     []OwnedResource `json:"ownedResources,omitempty"`
	ObservedGeneration int             `json:"observedGeneration,omitempty"`
}

`OwnedResource`

type OwnedResource struct {
	metav1.TypeMeta `json:",inline"`

	Name      string `json:"name,omitempty"`
	Namespace string `json:"namespace,omitempty"`
}

Concepts

This section contains the concepts and the architecture of individual components.

Terms

Bootstrap

This page describes the bootstrap phase of the controller.

Adopt

This page descibes the adoption process for the selected resources

Ownership chain

This page descibes the ownership chain process between individual group owner resources

Deletion

This page describes how the deletion process is being performed

Bootstrap

This process involves creating infrastructure resources, bound to the parent GlobalOwner object by ownership references, and allowing the child objects to be adopted by the GlobalOwner resource later.

Steps

Bootstrap process is carried by a single DecoratorController replica created during installation phase.

The process consists of 2 parts:

For each child group in the GlobalOwner resource, create a CompositeController replica.
Create an aggregated ClusterRole with permissions required to access every resource from child groups. The rules in this role will aggregate to the metacontroller replica installed with the controller.

A ClusterRole per each group has only these permissions:

get
list
watch
update - required for the created CompositeController to apply ownership reference on the resource.

Cleanup

Upon removal of the parent GlobalOwner resource, every bootstrapped component will be deleted using kubernetes garbage collection. This allows for the aggregated permissions to be scaled down upon removal, as the ClusterRole is getting removed.

Adopt

This process involves listing and watching resource groups specified under the ChildResources field in the GlobalOwner resource, and setting ownership references on each observed resource, using metacontroller adoption rules.

Process

Using collected objects from the related list collected using the customize hook semantics, an adopt CompositeController is setting a list of children resources in the sync hook response matching the namespace and name of each related resource. Returned list of children may look like:

[
    {
        "kind": "Secret",
        "apiVersion": "v1",
        "name": "owned-secret",
        "namespace": "default",
    }, {
        "kind": "ConfigMap",
        "apiVersion": "v1",
        "name": "owned-config-map",
        "namespace": "default",
    },
]

As every object in the list is already created in the cluster, the only change the metacontroller replica will do, is to apply the ownership references on the object, pointing to the parent GroupOwner resource instance.

Cleanup

Upon removal of the parent GlobalOwner resource, each GroupOwner resource and subsequentially all adopted resources will be removed in the order specified by the ChildResources content. Resource groups will be removed from top to bottom of the list.

Ownership chain

This process creates a GroupOwner resource per each resource group specified in the parent GlobalOwner resource spec.

Process

The process of setting up ownership chain consists of 2 parts.

When the bootstrap process finishes, each created CompositeController is creating a GroupOwner resource replica from the parents GlobalOwner children resource group.

Each of the created GroupOwner resources have a label matching the CompositeController label selector, for narrowing down the adopted resources to the target GroupOwner object. Secrets owned by the Secret group owner.

Once all of the GroupOwner resources are created, the controller is setting ownership references between the GroupOwner resources. This (in the best case scenario - see the issue) allows the removal process to fully rely on the kubernetes garbage collection by using foreground cascading deletion finalizer on the GroupOwner resource. More in the design document.

Deletion

This process specifies how the ordered deletion is handled, when the GlobalOwner resource is removed.

Deletion

TODO

Opt-out

As the resource is adopted using owner reference to the parent object, when the parent object is removed using foreground or background policy, the adopted resource will be removed as well. To opt out of this behavior, the metacontroller instance should be scaled down, the finalizer metacontroller.io/compositecontroller-<global-owner-name> should be removed from the resource, and then the GlobalOwner resource can be deleted with the orphan deletion policy. Adopted resources will stay untouched, however, the CompositeController and the ClusterRole created by the GlobalOwner resource will stay in the cluster, so the permission scope will not be reduced. Those will have to be cleaned up manually.

Guide

This section contains configuration and installation tips for Global Owner resource controller

Configuration

This page describes how to configure the Global Owner resource.

Helm values

Parameter	Description	Default
`namespaceOverride`	Namespace override value for installed global owner controller instance	`""`
`nameOverride`	Name override value for installed global owner controller instance	`""`
`image`	Jsonnet image to use for controller functionality	`ghcr.io/danil-grigorev/jsonnetd:v0.3.1`
`aggregationLabel`	ClusterRole aggregation label to provide granular permission scaling to the owned set of resources. Matches the metacontroller `ClusterRole` aggregation label value.	`rbac.metacontroller.k8s.io/aggregate-to-metacontroller`

Install Global Owner using Helm

This page describes how to install Global Owner controller using helm.

Building the chart from source code

The chart can be built from repository source:

git clone https://github.com/Danil-Grigorev/global-owner.git
cd global-owner
make release-chart

Installing the chart from package

helm install globalowner out/package/global-owner-v*.tgz --wait

Installing chart from ghcr.io

Charts are published as packages on ghcr.io

To install it from registry:

helm install globalowner -n metacontroller --create-namespace oci://ghcr.io/danil-grigorev/global-owner --version=v0.5.1 --wait

Installation

This page describes how to install Global Owner resource controller using kustomize.

Prerequisites

Install Metacontroller

Deploy the controller

kubectl apply -k https://github.com/danil-grigorev/global-owner/v1

or locally from the cloned repo:

kubectl apply -k v1

Design

Ordered Deletion

This is a design document for ordered resource deletion using only ownership references and native kubernetes garbage collection.

Ordered Deletion

Resources deletion is processed in the order specified on the GlobalOwner resource.

Deletion process

Blockers

Unfortunately this design does not work, because of an issue in the garbage collector implementation: https://github.com/kubernetes/kubernetes/issues/121113. Until it is fixed, this is just a concept.