Avatar of Sascha Selzer

Introduction

To empower developers to manage their applications from each commit to the repository through to the production deployment, many companies follow the idea of self-service, whereby several functions are provided that reduce the need for interaction between developers and the platform team. Part of these means is the introduction of container technologies and Kubernetes. This brings many advantages to the classic delivery approach:

With Kubernetes in place, the level of interaction shifts. There is no need for installation manuals, manually setting up servers with all required dependencies or a handover of artifacts with installation scripts.

New split of operations and responsibility
New split of operations and responsibility

The platform team can focus on hardening the platform and improving the developer experience to make it easier to use. Developer teams, on the other hand, can autonomously decide:

This is a good step towards the ability of self-management, but part of the lifecycle of an application is also external resources such as databases or caches. The platform teams still struggle deciding how to provide a way for the developers to manage resources on their own. Some of the problems they face are:

So even when everything is automated, there is still dependency on platform teams to provision infrastructure and often involves a manual process.

Infrastructure Provisioning by Process
Infrastructure Provisioning by Process

This slows down the setup and stops the developer teams from being fully autonomous.

The Operator Pattern

One of Kubernetes’ strengths is its extensibility, and this has shaped a new set of tools that leverage this possibility to move a lot of management into the cluster. These tools are generally called “operators” and typically consist of:

There are several operators already available which can also interact with external APIs on the team’s behalf.

Operator to interact with external API
Operator to interact with external API

This already solves many issues:

But some topics are still open:

What Does Crossplane Bring To The Mix?

Crossplane offers a framework to fill these shortcomings by defining easy-to-use resources which will be mapped to specific resources of the cloud provider without writing any code. Crossplane helps to create a control plane for all kinds of external and internal resources. Complicated setups are abstracted to higher-level resources to provide a simple interface for developers. Crossplane is more like a meta-framework for operators than merely an operator itself.

This sounds vague, so let’s have a look at some of the features Crossplane provides.

Provide Access To External APIs By Provider Packages

Crossplane manages the installation and configuration of operators (controllers) which can interact with external APIs in the form of so-called “providers”. Providers are Crossplane packages which include the controller and CRDs which defines external resources, i.e. “managed resources”. Crossplane already comes along with several providers, which cover several of the big cloud providers.

Crossplane Provider Packages
Crossplane Provider Packages

How provider packages are distributed and installed in a cluster is managed by Crossplane. A new provider can be installed by defining a provider resource in the cluster.

Let’s demonstrate how this works by setting up a local test cluster and installing Crossplane with the help of Helm.

kind create cluster

helm repo add crossplane-stable https://charts.crossplane.io/stable

helm repo update

helm upgrade --install \
  crossplane crossplane-stable/crossplane \
  -n crossplane-system \
  --create-namespace \
  --wait

After installation, you can see the installed Crossplane control plane and registered new CRDs (part of it is the provider resource).

kubectl -n crossplane-system get pods
NAME                                       READY   STATUS    RESTARTS   AGE
crossplane-545c58944d-gwzl9                1/1     Running   0          10m
crossplane-rbac-manager-85fd5c9f6c-4zm4m   1/1     Running   0          10m
kubectl get crds
NAME                                                       CREATED AT
compositeresourcedefinitions.apiextensions.crossplane.io   2022-06-28T13:07:13Z
compositionrevisions.apiextensions.crossplane.io           2022-06-28T13:07:13Z
compositions.apiextensions.crossplane.io                   2022-06-28T13:07:13Z
configurationrevisions.pkg.crossplane.io                   2022-06-28T13:07:13Z
configurations.pkg.crossplane.io                           2022-06-28T13:07:13Z
controllerconfigs.pkg.crossplane.io                        2022-06-28T13:07:13Z
locks.pkg.crossplane.io                                    2022-06-28T13:07:13Z
providerrevisions.pkg.crossplane.io                        2022-06-28T13:07:13Z
providers.pkg.crossplane.io                                2022-06-28T13:07:13Z
storeconfigs.secrets.crossplane.io                         2022-06-28T13:07:13Z

Now we have the infrastructure to install providers for the different cloud providers or external APIs we want to manage. An example for the AWS provider looks like this:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: "crossplane/provider-aws:master"
aws-provider.yaml

The provider links to the packages, which will then be installed by Crossplane. The CRDs which belong to the package will be registered and the controller will be installed and started.

kubectl -n crossplane-system get pods
NAME                                        READY   STATUS    RESTARTS   AGE
crossplane-545c58944d-gwzl9                 1/1     Running   0          16m
crossplane-rbac-manager-85fd5c9f6c-4zm4m    1/1     Running   0          16m
provider-aws-84fe593bd332-f6ff57c86-86qt9   1/1     Running   0          65s
the started AWS Controller
kubectl get crds
NAME                                                            CREATED AT
accesskeys.iam.aws.crossplane.io                                2022-06-28T13:22:20Z
accesspoints.efs.aws.crossplane.io                              2022-06-28T13:22:21Z
activities.sfn.aws.crossplane.io                                2022-06-28T13:22:20Z
addons.eks.aws.crossplane.io                                    2022-06-28T13:22:21Z
addresses.ec2.aws.crossplane.io                                 2022-06-28T13:22:17Z
aliases.kms.aws.crossplane.io                                   2022-06-28T13:22:18Z
apikeys.apigateway.aws.crossplane.io                            2022-06-28T13:22:23Z
apimappings.apigatewayv2.aws.crossplane.io                      2022-06-28T13:22:22Z
apis.apigatewayv2.aws.crossplane.io                             2022-06-28T13:22:18Z
authorizers.apigateway.aws.crossplane.io                        2022-06-28T13:22:24Z
authorizers.apigatewayv2.aws.crossplane.io                      2022-06-28T13:22:19Z
backups.dynamodb.aws.crossplane.io                              2022-06-28T13:22:19Z
basepathmappings.apigateway.aws.crossplane.io                   2022-06-28T13:22:22Z
brokers.mq.aws.crossplane.io                                    2022-06-28T13:22:19Z
bucketpolicies.s3.aws.crossplane.io                             2022-06-28T13:22:23Z
buckets.s3.aws.crossplane.io                                    2022-06-28T13:22:16Z
cacheclusters.cache.aws.crossplane.io                           2022-06-28T13:22:21Z
cacheparametergroups.elasticache.aws.crossplane.io              2022-06-28T13:22:22Z
cachepolicies.cloudfront.aws.crossplane.io                      2022-06-28T13:22:19Z
cachesubnetgroups.cache.aws.crossplane.io                       2022-06-28T13:22:18Z
certificateauthorities.acmpca.aws.crossplane.io                 2022-06-28T13:22:18Z
...
configurations.pkg.crossplane.io                                2022-06-28T13:07:13Z
connections.glue.aws.crossplane.io                              2022-06-28T13:22:23Z
controllerconfigs.pkg.crossplane.io                             2022-06-28T13:07:13Z
crawlers.glue.aws.crossplane.io                                 2022-06-28T13:22:17Z
databases.glue.aws.crossplane.io                                2022-06-28T13:22:22Z
dbclusterparametergroups.docdb.aws.crossplane.io                2022-06-28T13:22:22Z
dbclusterparametergroups.rds.aws.crossplane.io                  2022-06-28T13:22:18Z
dbclusters.docdb.aws.crossplane.io                              2022-06-28T13:22:18Z
dbclusters.neptune.aws.crossplane.io                            2022-06-28T13:22:17Z
dbclusters.rds.aws.crossplane.io                                2022-06-28T13:22:17Z
dbinstanceroleassociations.rds.aws.crossplane.io                2022-06-28T13:22:21Z
dbinstances.docdb.aws.crossplane.io                             2022-06-28T13:22:23Z
dbinstances.rds.aws.crossplane.io                               2022-06-28T13:22:21Z
dbparametergroups.rds.aws.crossplane.io                         2022-06-28T13:22:19Z
...
List of new CRDs coming with the AWS package

As you can see the list of new resources is quite exhaustive covering all the available resources of the provider.

Configuring The Providers

With the setup above, the developer can already interact with and e.g., create a new database resource like this:

apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: my-db
spec:
  forProvider:
    region: eu-central-1
    dbInstanceClass: db.t2.small
    masterUsername: master
    allocatedStorage: 10
    engine: postgres
    engineVersion: "12"
    skipFinalSnapshotBeforeDeletion: true
  writeConnectionSecretToRef:
    namespace: default
    name: my-db-credentials
database.yaml

When applied, we can see that it gets picked up by the provider but as the provider has no credentials to interact with the external API, it gets stuck.

kubectl apply -n crossplane-system -f database.yaml
rdsinstance.database.aws.crossplane.io/my-db created

kubectl get -n crossplane-system rdsinstance
NAME    READY   SYNCED   STATE   ENGINE     VERSION   AGE
my-db           False            postgres   12        34s

kubectl describe -n crossplane-system rdsinstances.database.aws.crossplane.io my-db

...
Events:
  Type     Reason                   Age                From                                            Message
  ----     ------                   ----               ----                                            -------
  Warning  CannotConnectToProvider  28s (x6 over 57s)  managed/rdsinstance.database.aws.crossplane.io  cannot get referenced Provider: ProviderConfig.aws.crossplane.io "default" not found

To fully work, we need to provide credentials in the form of a ProviderConfig referencing the secret. First, we need a file with the credentials. For AWS it looks like this:

[default]
aws_access_key_id=XXXXXXXXXXXXXXX
aws_secret_access_key=XXXXXXXXXXXXXXXXXXXXXXXXX
creds.conf

To create a secret in Kubernetes based on this file, the following command must be run:

kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./creds.conf

Subsequently, we can create a ProviderConfig referencing this secret and make it available to the provider.

apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds
aws-provider-config.yaml

When it’s deployed, we see that the RDSInstance will be picked up and a corresponding instance will be created in AWS.

kubectl apply -n crossplane-system -f aws-provider-config.yaml
providerconfig.aws.crossplane.io/default created

kubectl get -n crossplane-system rdsinstance
NAME    READY   SYNCED   STATE      ENGINE     VERSION   AGE
my-db   False   True     creating   postgres   12.8      21s

The Kubernetes state will now be synced to AWS.

AWS console showing the newly created DB
AWS console showing the newly created DB

After a while, the database is fully started and the corresponding secrets are available within the cluster.

kubectl get -n default secrets
NAME                TYPE                                DATA   AGE
my-db-credentials   connection.crossplane.io/v1alpha1   4      4m46s

With this setup, we have similar functionality as to that described with the operators, but with a common structure for how to package and distribute providers and overarching management of these packages with Crossplane in the cluster. As described with AWS, providers can be set up for GCP and Azure as well as other products with an API. For a full list, see the corresponding description on Crossplane’s homepage.

Creating Your Resource Abstractions And Compositions

With this setup, we only recreated the status quo with operators. It is probably better managed than with loose operators, but we still have the issue of complexity and governance. The delivered CRDs of the providers are still a one-to-one copy of the full options provided by the API.

Crossplane provides us the possibility to define our own CRDs which then will be mapped to one or many managed resource CRDs provided by the provider. The structure of these resources is completely up to us, we only need to define the resource we want and a mapping for this resource to the managed resources. These new resources are called “composite resources” (XRDs) in Crossplane.

How Composite Resources and Managed Resources are linked
How Composite Resources and Managed Resources are linked

So Crossplane needs a new CRD and a mapping. For example, we want to create a CRD for the developer with the following properties:

All other properties like dbInstanceClass, engine, or region are not configurable and will be defined by the platform team.

We can create a new resource called Database with only a limited set of configurations like this:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xdatabases.innoq.com
spec:
  group: innoq.com
  names:
    kind: XDatabase
    plural: xdatabases
  defaultCompositionRef:
    name: xdatabases.aws.innoq.com
  claimNames:
    kind: DatabaseClaim
    plural: databaseclaims
  connectionSecretKeys:
  - username
  - password
  - endpoint
  - port
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  storageGB:
                    type: integer
                  version:
                    type: string
                    enum:
                    - "11"
                    - "12"
                    default: "12"
                required:
                - storageGB
            required:
            - parameters
database-composite-resource.yaml

This creates a new CRD for us, which defines only two fields that can be set by the developer (storageGB and version). The CompositeResourceDefinition uses the Open API Schema in the same way as CRDs, so we can define constraints or defaults, as shown for the version property.

kubectl apply -f database-composite-resource.yaml
compositeresourcedefinition.apiextensions.crossplane.io/xdatabases.innoq.com created

kubectl get crds | grep innoq
databaseclaims.innoq.com                                        2022-06-29T08:32:23Z
xdatabases.innoq.com                                            2022-06-29T08:32:23Z

The second part needed is the mapping.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xdatabases.aws.innoq.com
spec:
  writeConnectionSecretsToNamespace: crossplane-system
  compositeTypeRef:
    apiVersion: innoq.com/v1alpha1
    kind: XDatabase
  resources:
  - name: rdsinstance
    base:
      apiVersion: database.aws.crossplane.io/v1beta1
      kind: RDSInstance
      spec:
        forProvider:
          region: eu-central-1
          dbInstanceClass: db.t2.small
          masterUsername: masteruser
          engine: postgres
          skipFinalSnapshotBeforeDeletion: true
          publiclyAccessible: true
        writeConnectionSecretToRef:
          namespace: crossplane-system
    patches:
    - fromFieldPath: "metadata.uid"
      toFieldPath: "spec.writeConnectionSecretToRef.name"
      transforms:
      - type: string
        string:
          fmt: "%s-postgresql"
    - fromFieldPath: "spec.parameters.storageGB"
      toFieldPath: "spec.forProvider.allocatedStorage"
    - fromFieldPath: "spec.parameters.version"
      toFieldPath: "spec.forProvider.engineVersion"
    connectionDetails:
    - fromConnectionSecretKey: username
    - fromConnectionSecretKey: password
    - fromConnectionSecretKey: endpoint
    - fromConnectionSecretKey: port
aws-postgresql-composition.yaml

The composition describes how the new CRD and its properties shall be mapped to a list of managed resources. Here it is only one resource, but we could, for example, create an IAM role, user, or an own subnet together with the RDS instance using default values set by the platform team. The final interface to the developer looks like this:

apiVersion: innoq.com/v1alpha1
kind: DatabaseClaim
metadata:
  name: my-db
  namespace: default
spec:
  parameters:
    storageGB: 20
    version: "11"
  writeConnectionSecretToRef:
    name: db-conn
my-db.yaml

The developer can only set the storage size and the engine version. All other properties are hidden and managed by the platform team thanks to the composition.

Crossplane brings all the tools with it to define new CRDs and define an easier and less error-prone interface between the platform team and developer teams just by providing configurations instead of writing our own operators.

Package And Distribute Your Own Resources

At one point we have set up a lot of our own composite resources and composition which we want to version and share between several clusters. Crossplane has the concept of packages to distribute configuration between Crossplane installations. There are two types of packages:

We already referenced a provider package when we set up the cluster.

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: "crossplane/provider-aws:master"
aws-provider.yaml

The spec.package points to the package which shall be installed for this provider. Packages are just OCI compatible container images, but instead of containing code, they contain a list of yaml files that will be read by Crossplane to register e.g. CRDs and install the corresponding operator. An example provider package can be found on the Crossplane website.

A similar approach is used to distribute configurations, i.e., our own set of composite resources and compositions. To distribute, we need to package them and upload them to a container registry. We can also define dependencies to provider packages so that Crossplane takes care of installing the corresponding provider before the configuration gets installed.

Let’s say we packaged our configuration into an image called innoq/infra-configuration. If we want Crossplane to download the configuration and register all CRDs and providers automatically, we can create a resource in our cluster like this:

apiVersion: pkg.crossplane.io/v1
kind: Configuration
metadata:
  name: innoq-infra
spec:
  package: innoq/infra-configuration:latest
  packagePullPolicy: IfNotPresent
  revisionActivationPolicy: Automatic
  revisionHistoryLimit: 1
infra-configuration.yaml

When deployed, Crossplane will:

So, it provides a simple process to share common configurations without the need for additional infrastructure. For a full description, have a look at the corresponding documentation.

Fazit

Crossplane: The missing link?

Crossplane with its framework to provide a common interface for external and internal resources and to provide a higher abstraction of resources can play an essential role on the way to a full self-service capability of development teams.

Although this article focuses on external infrastructure provisioning, Crossplane can also help with improving the developer experience in general. A complex application setup that consists of internal and external resources (like deployment, service, HPA, ingress, config map, secret, and an external DB) can be hidden and managed by the platform team e.g., by creating composite resources WebApp or BackendApp with just a set of properties needing to be provided by developers.

Kubernetes is not the end game and can be overwhelming with its operational concepts. It will be interesting to see how Crossplane will be adopted over the next years and how platform teams will leverage its features to potentially provide a better developer experience.