From c7a6ed53ca4bcc722acc4e712bbeb70027661896 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Thu, 31 Mar 2022 11:08:28 +0300 Subject: [PATCH 01/13] Add proposal for adding OCI support for Kubernetes manifests to Flux Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 140 ++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 rfcs/kubernetes-oci/README.md diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md new file mode 100644 index 00000000..65b0737e --- /dev/null +++ b/rfcs/kubernetes-oci/README.md @@ -0,0 +1,140 @@ +# RFC-xxxx Flux OCI support for Kubernetes manifests + +**Status:** provisional + +**Creation date:** 2022-03-31 + +**Last update:** 2022-03-31 + +## Summary + +Flux should be able to distribute and reconcile Kubernetes configuration packaged as OCI artifacts. + +On the client-side, the Flux CLI should offer a command for packaging Kubernetes configs into +an OCI artifact and pushing the artifact to a container registry using the Docker config file +and the Docker credential helpers for authentication. + +On the server-side, the Flux source-controller should offer a dedicated API Kind for defining +how OCI artifacts are pulled from container registries and how the artifact's authenticity can be verified. +Flux should be able to work with any type of artifact even if it's not created with the Flux CLI. + +## Motivation + +Given that OCI registries are evolving into a generic artifact storage solution, +we should extend Flux to allow fetching Kubernetes manifests and related configs +from container registries similar to how Flux works with Git and Bucket storage. + +With OCI support, Flux users can automate artifact updates to Git in the same way +they do today for container images. + +### Goals + +- Add support to the Flux CLI for packaging Kubernetes manifests and related configs into OCI artifacts. +- Add support to Flux source-controller for fetching configs stored as OCI artifacts. +- Make it easy for users to switch from Git repositories and Buckets to OCI repositories. + +### Non-Goals + +- Introduce a new OCI media type for artifacts containing Kubernetes manifests. + +## Proposal + +### Push artifacts + +Flux users should be able to package a local directory containing Kubernetes configs into a tarball +and push the archive to a container registry as an OCI artifact. + +```sh +flux push artifact docker.io/org/app-config:v1.0.0 -f ./deploy +``` + +Flux CLI with produce artifacts of type `application/vnd.oci.image.config.v1+json`. +The directory pointed to by `-f` is archived and compressed in the `tar+gzip` format +and the layer media type is set to `application/vnd.oci.image.layer.v1.tar+gzip`. + +> A proof-of-concept CLI implementation for distributing Kubernetes configs as OCI artifacts +> is available at [kustomizer.dev](https://github.com/stefanprodan/kustomizer). + +### Pull artifacts + +Flux users should be able to define a source for pulling manifests inside the cluster from an OCI repository. + +```yaml +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: OCIRepository +metadata: + name: app-config + namespace: flux-system +spec: + interval: 10m + url: docker.io/org/app-config + ref: + tag: v1.0.0 +``` + +An `OCIRepository` can refer to an artifact by tag, digest or semver range: + +```yaml +spec: + ref: + # one of + tag: "latest" + digest: "sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2" + semver: "6.0.x" +``` + +For private repositories, the credentials can be supplied with: + +```yaml +spec: + secretRef: + name: regcred +``` + +The `secretRef` points to a Kubernetes secret in the same namespace as the `OCIRepository`, +the secret type must be `kubernetes.io/dockerconfigjson`. + +To verify the authenticity of an artifact, the Sigstore cosign public key can be supplied with: + +```yaml +spec: + verify: + provider: cosign + secretRef: + name: cosign-key +``` + +### Reconcile artifacts + +The `OCIRepository` can be used as a drop-in replacement for `GitRepository` and `Bucket` sources. +For example a Flux Kustomization can refer to an `OCIRepository` and reconcile the manifests found in the OCI artifact: + +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 +kind: Kustomization +metadata: + name: app + namespace: flux-system +spec: + interval: 10m + sourceRef: + kind: OCIRepository + name: app-config + path: ./ +``` + +### User Stories + +TODO + +### Alternatives + +TODO + +## Design Details + +TODO + +### Enabling the feature + +The feature is enabled by default. From 0fc582d6fde584ec25db6080c15533f9b6594cc6 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Thu, 31 Mar 2022 11:42:58 +0300 Subject: [PATCH 02/13] Add user stories for publishing and reconciling OCI artifacts Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 91 ++++++++++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 65b0737e..d528fa1f 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -125,7 +125,96 @@ spec: ### User Stories -TODO +#### Story 1 + +> As a developer I want to publish my app Kubernetes manifests to the same GHCR registry +> where I publish my app containers. + +First login to GHCR with Docker: + +```sh +docker login ghcr.io -u ${GITHUB_USER} -p ${GITHUB_TOKEN} +``` + +Build your app container image and push it to GHCR: + +```sh +docker build -t ghcr.io/org/my-app:v1.0.0 . +docker push ghcr.io/org/my-app:v1.0.0 +``` + +Edit the app deployment manifest and set the new image tag. +Then push the Kubernetes manifests to GHCR: + +```sh +flux push artifact ghcr.io/org/my-app-config:v1.0.0 -f ./deploy +``` + +Sign the config image with cosign: + +```sh +cosign sign --key cosign.key ghcr.io/org/my-app-config:v1.0.0 +``` + +#### Story 2 + +> As a developer I want to deploy my app using Kubernetes manifests published as OCI artifacts to GHCR. + +First create a secret using a GitHub token that allows access to GHCR: + +```sh +kubectl create secret docker-registry my-app-regcred \ + --docker-server=ghcr.io \ + --docker-username=$GITHUB_USER \ + --docker-password=$GITHUB_TOKEN +``` + +Then create a secret with your cosgin public key: + +```sh +kubectl create secret generic my-app-cosgin-key \ + --from-file=cosign.pub=cosign/my-key.pub +``` + +Then define an `OCIRepository` to fetch and verify the latest app config version: + +```yaml +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: OCIRepository +metadata: + name: app-config + namespace: default +spec: + interval: 10m + url: ghcr.io/org/my-app-config + ref: + semver: "1.x" + secretRef: + name: my-app-regcred + verify: + provider: cosign + secretRef: + name: my-app-cosgin-key +``` + +And finally, create a Flux Kustomization to reconcile the app on the cluster: + +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 +kind: Kustomization +metadata: + name: app + namespace: default +spec: + interval: 10m + sourceRef: + kind: OCIRepository + name: app-config + path: ./ + prune: true + wait: true + timeout: 2m +``` ### Alternatives From 183b9a7ee013a76109fef59b2906298f35a8cf84 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Fri, 1 Apr 2022 11:12:26 +0300 Subject: [PATCH 03/13] Add auto-login feature Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 38 +++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index d528fa1f..12db4e85 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -83,7 +83,23 @@ spec: semver: "6.0.x" ``` -For private repositories, the credentials can be supplied with: +To verify the authenticity of an artifact, the Sigstore cosign public key can be supplied with: + +```yaml +spec: + verify: + provider: cosign + secretRef: + name: cosign-key +``` + +### Pull artifacts from private repositories + +For authentication purposes, Flux users can choose between supplying static credentials with Kubernetes secrets +and cloud-based OIDC using an IAM role binding to the source-controller Kubernetes service account. + +For private repositories hosted on DockerHub, GitHub, Quay, self-hosted Docker Registry and others, +the credentials can be supplied with: ```yaml spec: @@ -94,16 +110,22 @@ spec: The `secretRef` points to a Kubernetes secret in the same namespace as the `OCIRepository`, the secret type must be `kubernetes.io/dockerconfigjson`. -To verify the authenticity of an artifact, the Sigstore cosign public key can be supplied with: +When Flux runs on EKS or GKE, an IAM role (that grants read-only access to ACR, ECR or GCR) +can be used to bind the `source-controller` to the IAM role. -```yaml -spec: - verify: - provider: cosign - secretRef: - name: cosign-key +Similar to image-reflector-controller +[auto-login feature](https://fluxcd.io/docs/guides/image-update/#imagerepository-cloud-providers-authentication), +source-controller will expose dedicated flags for each cloud provider: + +```sh +--aws-autologin-for-ecr +--azure-autologin-for-acr +--gcp-autologin-for-gcr ``` +We should extract the flags and the AWS, Azure and GCP auth implementations from image-reflector-controller into +`fluxcd/pkg/oci/auth` to reuses the code in source-controller. + ### Reconcile artifacts The `OCIRepository` can be used as a drop-in replacement for `GitRepository` and `Bucket` sources. From 8a7c94180b022fe1aa49bccbab941d7bd1a1d8b6 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Tue, 12 Apr 2022 17:53:08 +0300 Subject: [PATCH 04/13] Add client certificate authentication Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 12db4e85..d96d1def 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -110,7 +110,25 @@ spec: The `secretRef` points to a Kubernetes secret in the same namespace as the `OCIRepository`, the secret type must be `kubernetes.io/dockerconfigjson`. -When Flux runs on EKS or GKE, an IAM role (that grants read-only access to ACR, ECR or GCR) +For private repositories which require a certificate to authenticate, +the client certificate, private key and the CA certificate (if self-signed), can be provided with: + +```yaml +spec: + certSecretRef: + name: regcert +``` + +The `certSecretRef` points to a Kubernetes secret in the same namespace as the `OCIRepository`: + +```shell +kubectl create secret generic regcert \ + --from-file=certFile=client.crt \ + --from-file=keyFile=client.key \ + --from-file=caFile=ca.crt +``` + +When Flux runs on AKS, EKS or GKE, an IAM role (that grants read-only access to ACR, ECR or GCR) can be used to bind the `source-controller` to the IAM role. Similar to image-reflector-controller @@ -126,6 +144,19 @@ source-controller will expose dedicated flags for each cloud provider: We should extract the flags and the AWS, Azure and GCP auth implementations from image-reflector-controller into `fluxcd/pkg/oci/auth` to reuses the code in source-controller. +### Pull artifacts from self-hosted repositories + +For self-hosted Docker registries where the API is exposed with a self-signed TLS certificate, +the CA certificate and private key can be provided with: + +```yaml +spec: + secretRef: + name: regcred + +``` + + ### Reconcile artifacts The `OCIRepository` can be used as a drop-in replacement for `GitRepository` and `Bucket` sources. From 4885278691355ec0ab141bc5c7c9c6852e8d0f68 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Wed, 13 Apr 2022 12:07:53 +0300 Subject: [PATCH 05/13] Restructure the OCI auth spec Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index d96d1def..40388236 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -4,7 +4,7 @@ **Creation date:** 2022-03-31 -**Last update:** 2022-03-31 +**Last update:** 2022-04-13 ## Summary @@ -98,6 +98,8 @@ spec: For authentication purposes, Flux users can choose between supplying static credentials with Kubernetes secrets and cloud-based OIDC using an IAM role binding to the source-controller Kubernetes service account. +#### Basic auth + For private repositories hosted on DockerHub, GitHub, Quay, self-hosted Docker Registry and others, the credentials can be supplied with: @@ -108,7 +110,16 @@ spec: ``` The `secretRef` points to a Kubernetes secret in the same namespace as the `OCIRepository`, -the secret type must be `kubernetes.io/dockerconfigjson`. +the secret type must be `kubernetes.io/dockerconfigjson`: + +```shell +kubectl create secret docker-registry regcred \ + --docker-server= \ + --docker-username= \ + --docker-password= +``` + +#### Client cert auth For private repositories which require a certificate to authenticate, the client certificate, private key and the CA certificate (if self-signed), can be provided with: @@ -128,6 +139,8 @@ kubectl create secret generic regcert \ --from-file=caFile=ca.crt ``` +#### OIDC auth + When Flux runs on AKS, EKS or GKE, an IAM role (that grants read-only access to ACR, ECR or GCR) can be used to bind the `source-controller` to the IAM role. @@ -144,19 +157,6 @@ source-controller will expose dedicated flags for each cloud provider: We should extract the flags and the AWS, Azure and GCP auth implementations from image-reflector-controller into `fluxcd/pkg/oci/auth` to reuses the code in source-controller. -### Pull artifacts from self-hosted repositories - -For self-hosted Docker registries where the API is exposed with a self-signed TLS certificate, -the CA certificate and private key can be provided with: - -```yaml -spec: - secretRef: - name: regcred - -``` - - ### Reconcile artifacts The `OCIRepository` can be used as a drop-in replacement for `GitRepository` and `Bucket` sources. From 2912d1d4375f9429ac44278198f6db1ee59e505e Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Tue, 3 May 2022 15:17:49 +0300 Subject: [PATCH 06/13] Add serviceAccountName to auth spec Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 40388236..27710d4b 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -119,6 +119,13 @@ kubectl create secret docker-registry regcred \ --docker-password= ``` +For image pull secrets attached to a service account, the account name can be specified with: + +```yaml +spec: + serviceAccountName: regsa +``` + #### Client cert auth For private repositories which require a certificate to authenticate, From 44762933b3e6ab9988d916da1a29fccec6986757 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Tue, 7 Jun 2022 11:10:56 +0300 Subject: [PATCH 07/13] Add `flux tag artifact` command Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 27710d4b..1d2ae937 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -48,6 +48,13 @@ and push the archive to a container registry as an OCI artifact. flux push artifact docker.io/org/app-config:v1.0.0 -f ./deploy ``` +To ease the promotion workflow of a specific version from one environment to another, the CLI +should offer a tagging command. + +```sh +flux tag artifact docker.io/org/app-config:v1.0.0 latest +``` + Flux CLI with produce artifacts of type `application/vnd.oci.image.config.v1+json`. The directory pointed to by `-f` is archived and compressed in the `tar+gzip` format and the layer media type is set to `application/vnd.oci.image.layer.v1.tar+gzip`. @@ -216,6 +223,12 @@ Sign the config image with cosign: cosign sign --key cosign.key ghcr.io/org/my-app-config:v1.0.0 ``` +Mark v1.0.0 as latest: + +```sh +flux tag artifact ghcr.io/org/my-app-config:v1.0.0 latest +``` + #### Story 2 > As a developer I want to deploy my app using Kubernetes manifests published as OCI artifacts to GHCR. From 34321983e7e8c6dce6108d0f0b1db35bf92fc74c Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Tue, 7 Jun 2022 11:32:55 +0300 Subject: [PATCH 08/13] Add OCI artifact type to alternatives Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 1d2ae937..ad82c7fd 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -291,7 +291,9 @@ spec: ### Alternatives -TODO +An alternative solution is to introduce an OCI artifact type especially made for Kubernetes configuration. +That is considered unpractical, as introducing an OCI type has to go through the +IANA process and Flux is not the owner of those type as Helm is for Helm artifact for example. ## Design Details From a22438b7fa85c914ea38854fa26067f486c04d37 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Wed, 22 Jun 2022 12:27:44 +0300 Subject: [PATCH 09/13] Add design details Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 121 ++++++++++++++++++++++++++++++---- 1 file changed, 110 insertions(+), 11 deletions(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index ad82c7fd..4b581923 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -4,7 +4,7 @@ **Creation date:** 2022-03-31 -**Last update:** 2022-04-13 +**Last update:** 2022-06-22 ## Summary @@ -45,19 +45,40 @@ Flux users should be able to package a local directory containing Kubernetes con and push the archive to a container registry as an OCI artifact. ```sh -flux push artifact docker.io/org/app-config:v1.0.0 -f ./deploy +flux push artifact docker.io/org/app-config:v1.0.0 \ + --path="./deploy" \ + --source="$(git config --get remote.origin.url)" \ + --revision="$(git branch --show-current)/$(git rev-parse HEAD)" +``` + +The Flux CLI with produce artifacts of type `"application/vnd.docker.distribution.manifest.v2+json` +which ensures compatibility with container registries that don't support custom OCI media types. + +The directory pointed to by `--path` is archived and compressed in the `tar+gzip` format +and the layer media type is set to `application/vnd.docker.image.rootfs.diff.tar.gzip`. + +The source URL and revision are added to the OCI artifact as annotations in the format: + +```json +{ + "schemaVersion": 2, + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "annotations": { + "source.toolkit.fluxcd.io/url": "https://github.com/org/app.git", + "source.toolkit.fluxcd.io/revision": "main/450796ddb2ab6724ee1cc32a4be56da032d1cca0" + } +} ``` To ease the promotion workflow of a specific version from one environment to another, the CLI should offer a tagging command. ```sh -flux tag artifact docker.io/org/app-config:v1.0.0 latest +flux tag artifact docker.io/org/app-config:v1.0.0 --tag=latest --tag=production ``` -Flux CLI with produce artifacts of type `application/vnd.oci.image.config.v1+json`. -The directory pointed to by `-f` is archived and compressed in the `tar+gzip` format -and the layer media type is set to `application/vnd.oci.image.layer.v1.tar+gzip`. +To help inspect artifacts, the Flux CLI will offer a `build` and a `pull` command for generating +tarballs locally and for downloading the tarballs from remote container registries. > A proof-of-concept CLI implementation for distributing Kubernetes configs as OCI artifacts > is available at [kustomizer.dev](https://github.com/stefanprodan/kustomizer). @@ -214,7 +235,7 @@ Edit the app deployment manifest and set the new image tag. Then push the Kubernetes manifests to GHCR: ```sh -flux push artifact ghcr.io/org/my-app-config:v1.0.0 -f ./deploy +flux push artifact ghcr.io/org/my-app-config:v1.0.0 --path ./deploy ``` Sign the config image with cosign: @@ -223,10 +244,10 @@ Sign the config image with cosign: cosign sign --key cosign.key ghcr.io/org/my-app-config:v1.0.0 ``` -Mark v1.0.0 as latest: +Mark `v1.0.0` as latest: ```sh -flux tag artifact ghcr.io/org/my-app-config:v1.0.0 latest +flux tag artifact ghcr.io/org/my-app-config:v1.0.0 --tag latest ``` #### Story 2 @@ -283,7 +304,7 @@ spec: sourceRef: kind: OCIRepository name: app-config - path: ./ + path: ./deploy prune: true wait: true timeout: 2m @@ -297,7 +318,85 @@ IANA process and Flux is not the owner of those type as Helm is for Helm artifac ## Design Details -TODO +Both the Flux CLI and source-controller will use the [go-containerregistry](https://github.com/google/go-containerregistry) +library for OCI operations such as push, pull, tag, list tags, etc. + +For authentication purposes, the `flux artifact` commands will use the `~/.docker/config.json` +config file and the Docker credential helpers. + +The source-controller will reuse the authentication library from +[image-reflector-controller](https://github.com/fluxcd/image-reflector-controller). + +The Flux CLI will produce OCI artifacts with the following format: + +```json +{ + "schemaVersion": 2, + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "config": { + "mediaType": "application/vnd.docker.container.image.v1+json", + "size": 233, + "digest": "sha256:e7c52109f8e375176a888fd571dc0e0b40ed8a80d9301208474a2a906b0a2dcc" + }, + "layers": [ + { + "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", + "size": 1091, + "digest": "sha256:ad804afeae14a8a5c9a45b29f4931104a887844691d040c8737ee3cce6fd6735" + } + ], + "annotations": { + "source.toolkit.fluxcd.io/revision": "6.1.6/450796ddb2ab6724ee1cc32a4be56da032d1cca0", + "source.toolkit.fluxcd.io/url": "https://github.com/stefanprodan/podinfo.git" + } +} +``` + +The source-controller will extract the first layer from the OCI artifact, and will repackage it +as an internal `sourcev1.Artifact`. The internal artifact revision will be set to the OCI SHA256 digest: + +```yaml +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: OCIRepository +metadata: + creationTimestamp: "2022-06-22T09:14:19Z" + finalizers: + - finalizers.fluxcd.io + generation: 1 + name: podinfo + namespace: oci + resourceVersion: "6603" + uid: 42e0b9f0-021c-476d-86c7-2cd20747bfff +spec: + interval: 10m + ref: + tag: 6.1.6 + timeout: 60s + url: ghcr.io/stefanprodan/manifests/podinfo +status: + artifact: + checksum: d7e924b4882e55b97627355c7b3d2e711e9b54303afa2f50c25377f4df66a83b + lastUpdateTime: "2022-06-22T09:14:21Z" + path: ocirepository/oci/podinfo/3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de.tar.gz + revision: 3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de + size: 1105 + url: http://source-controller.flux-system.svc.cluster.local./ocirepository/oci/podinfo/3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de.tar.gz + conditions: + - lastTransitionTime: "2022-06-22T09:14:21Z" + message: stored artifact for revision '3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de' + observedGeneration: 1 + reason: Succeeded + status: "True" + type: Ready + - lastTransitionTime: "2022-06-22T09:14:21Z" + message: stored artifact for revision '3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de' + observedGeneration: 1 + reason: Succeeded + status: "True" + type: ArtifactInStorage + observedGeneration: 1 + url: http://source-controller.flux-system.svc.cluster.local./ocirepository/oci/podinfo/latest.tar.gz +``` ### Enabling the feature From 143609b9fb8f18553f1d6bd3427114509117f284 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Mon, 27 Jun 2022 16:26:07 +0300 Subject: [PATCH 10/13] Add specification for `spec.url` Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 4b581923..321d4ce3 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -51,7 +51,7 @@ flux push artifact docker.io/org/app-config:v1.0.0 \ --revision="$(git branch --show-current)/$(git rev-parse HEAD)" ``` -The Flux CLI with produce artifacts of type `"application/vnd.docker.distribution.manifest.v2+json` +The Flux CLI will produce artifacts of type `application/vnd.docker.distribution.manifest.v2+json` which ensures compatibility with container registries that don't support custom OCI media types. The directory pointed to by `--path` is archived and compressed in the `tar+gzip` format @@ -100,6 +100,10 @@ spec: tag: v1.0.0 ``` +The `spec.url` field points to the container image repository in the format `://`. +Note that specifying a tag or digest is not in accepted for this field. The `spec.url` value is used by the controller +to fetch the list of tags from the remote OCI repository. + An `OCIRepository` can refer to an artifact by tag, digest or semver range: ```yaml From e060873246f583341d08352ea1cbdaae807d8076 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Tue, 28 Jun 2022 17:26:01 +0300 Subject: [PATCH 11/13] Add examples for flux build, push and list commands Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 321d4ce3..839f0174 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -46,9 +46,9 @@ and push the archive to a container registry as an OCI artifact. ```sh flux push artifact docker.io/org/app-config:v1.0.0 \ - --path="./deploy" \ --source="$(git config --get remote.origin.url)" \ - --revision="$(git branch --show-current)/$(git rev-parse HEAD)" + --revision="$(git branch --show-current)/$(git rev-parse HEAD)" \ + --path="./deploy" ``` The Flux CLI will produce artifacts of type `application/vnd.docker.distribution.manifest.v2+json` @@ -77,11 +77,20 @@ should offer a tagging command. flux tag artifact docker.io/org/app-config:v1.0.0 --tag=latest --tag=production ``` +To view all the available artifacts in a repository and their metadata, the CLI should +offer a list command. + +```sh +flux list artifacts docker.io/org/app-config +``` + To help inspect artifacts, the Flux CLI will offer a `build` and a `pull` command for generating tarballs locally and for downloading the tarballs from remote container registries. -> A proof-of-concept CLI implementation for distributing Kubernetes configs as OCI artifacts -> is available at [kustomizer.dev](https://github.com/stefanprodan/kustomizer). +```sh +flux build artifact --path ./deploy --output tmp/artifact.tgz +flux pull artifact docker.io/org/app-config:v1.0.0 --output ./manifests +``` ### Pull artifacts @@ -239,7 +248,10 @@ Edit the app deployment manifest and set the new image tag. Then push the Kubernetes manifests to GHCR: ```sh -flux push artifact ghcr.io/org/my-app-config:v1.0.0 --path ./deploy +flux push artifact ghcr.io/org/my-app-config:v1.0.0 \ + --source="$(git config --get remote.origin.url)" \ + --revision="$(git tag --points-at HEAD)/$(git rev-parse HEAD)"\ + --path="./deploy" ``` Sign the config image with cosign: @@ -254,6 +266,15 @@ Mark `v1.0.0` as latest: flux tag artifact ghcr.io/org/my-app-config:v1.0.0 --tag latest ``` +List the artifacts and their metadata with: + +```console +$ flux list artifacts ghcr.io/org/my-app-config +ARTIFACT DIGEST SOURCE REVISION +ghcr.io/org/my-app-config:latest sha256:45b95019d30af335137977a369ad56e9ea9e9c75bb01afb081a629ba789b890c https://github.com/org/my-app-config.git v1.0.0/20b3a674391df53f05e59a33554973d1cbd4d549 +ghcr.io/org/my-app-config:v1.0.0 sha256:45b95019d30af335137977a369ad56e9ea9e9c75bb01afb081a629ba789b890c https://github.com/org/my-app-config.git v1.0.0/3f45e72f0d3457e91e3c530c346d86969f9f4034 +``` + #### Story 2 > As a developer I want to deploy my app using Kubernetes manifests published as OCI artifacts to GHCR. From 6b9c0a5e4897c46500753076a7d71946b52fefd1 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Fri, 1 Jul 2022 15:49:26 +0300 Subject: [PATCH 12/13] Add `oci://` proto to the spec Signed-off-by: Stefan Prodan --- rfcs/kubernetes-oci/README.md | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/kubernetes-oci/README.md index 839f0174..90fd9137 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/kubernetes-oci/README.md @@ -45,7 +45,7 @@ Flux users should be able to package a local directory containing Kubernetes con and push the archive to a container registry as an OCI artifact. ```sh -flux push artifact docker.io/org/app-config:v1.0.0 \ +flux push artifact oci://docker.io/org/app-config:v1.0.0 \ --source="$(git config --get remote.origin.url)" \ --revision="$(git branch --show-current)/$(git rev-parse HEAD)" \ --path="./deploy" @@ -74,14 +74,14 @@ To ease the promotion workflow of a specific version from one environment to ano should offer a tagging command. ```sh -flux tag artifact docker.io/org/app-config:v1.0.0 --tag=latest --tag=production +flux tag artifact oci://docker.io/org/app-config:v1.0.0 --tag=latest --tag=production ``` To view all the available artifacts in a repository and their metadata, the CLI should offer a list command. ```sh -flux list artifacts docker.io/org/app-config +flux list artifacts oci://docker.io/org/app-config ``` To help inspect artifacts, the Flux CLI will offer a `build` and a `pull` command for generating @@ -89,7 +89,7 @@ tarballs locally and for downloading the tarballs from remote container registri ```sh flux build artifact --path ./deploy --output tmp/artifact.tgz -flux pull artifact docker.io/org/app-config:v1.0.0 --output ./manifests +flux pull artifact oci://docker.io/org/app-config:v1.0.0 --output ./manifests ``` ### Pull artifacts @@ -104,12 +104,12 @@ metadata: namespace: flux-system spec: interval: 10m - url: docker.io/org/app-config + url: oci://docker.io/org/app-config ref: tag: v1.0.0 ``` -The `spec.url` field points to the container image repository in the format `://`. +The `spec.url` field points to the container image repository in the format `oci://://`. Note that specifying a tag or digest is not in accepted for this field. The `spec.url` value is used by the controller to fetch the list of tags from the remote OCI repository. @@ -202,13 +202,10 @@ source-controller will expose dedicated flags for each cloud provider: --gcp-autologin-for-gcr ``` -We should extract the flags and the AWS, Azure and GCP auth implementations from image-reflector-controller into -`fluxcd/pkg/oci/auth` to reuses the code in source-controller. - ### Reconcile artifacts The `OCIRepository` can be used as a drop-in replacement for `GitRepository` and `Bucket` sources. -For example a Flux Kustomization can refer to an `OCIRepository` and reconcile the manifests found in the OCI artifact: +For example, a Flux Kustomization can refer to an `OCIRepository` and reconcile the manifests found in the OCI artifact: ```yaml apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 @@ -248,7 +245,7 @@ Edit the app deployment manifest and set the new image tag. Then push the Kubernetes manifests to GHCR: ```sh -flux push artifact ghcr.io/org/my-app-config:v1.0.0 \ +flux push artifact oci://ghcr.io/org/my-app-config:v1.0.0 \ --source="$(git config --get remote.origin.url)" \ --revision="$(git tag --points-at HEAD)/$(git rev-parse HEAD)"\ --path="./deploy" @@ -263,13 +260,13 @@ cosign sign --key cosign.key ghcr.io/org/my-app-config:v1.0.0 Mark `v1.0.0` as latest: ```sh -flux tag artifact ghcr.io/org/my-app-config:v1.0.0 --tag latest +flux tag artifact oci://ghcr.io/org/my-app-config:v1.0.0 --tag latest ``` List the artifacts and their metadata with: ```console -$ flux list artifacts ghcr.io/org/my-app-config +$ flux list artifacts oci://ghcr.io/org/my-app-config ARTIFACT DIGEST SOURCE REVISION ghcr.io/org/my-app-config:latest sha256:45b95019d30af335137977a369ad56e9ea9e9c75bb01afb081a629ba789b890c https://github.com/org/my-app-config.git v1.0.0/20b3a674391df53f05e59a33554973d1cbd4d549 ghcr.io/org/my-app-config:v1.0.0 sha256:45b95019d30af335137977a369ad56e9ea9e9c75bb01afb081a629ba789b890c https://github.com/org/my-app-config.git v1.0.0/3f45e72f0d3457e91e3c530c346d86969f9f4034 @@ -305,7 +302,7 @@ metadata: namespace: default spec: interval: 10m - url: ghcr.io/org/my-app-config + url: oci://ghcr.io/org/my-app-config ref: semver: "1.x" secretRef: @@ -397,7 +394,7 @@ spec: ref: tag: 6.1.6 timeout: 60s - url: ghcr.io/stefanprodan/manifests/podinfo + url: oci://ghcr.io/stefanprodan/manifests/podinfo status: artifact: checksum: d7e924b4882e55b97627355c7b3d2e711e9b54303afa2f50c25377f4df66a83b From 4a55b828b1b7ebe962319705c8b167c7acdceaab Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Wed, 6 Jul 2022 17:02:19 +0300 Subject: [PATCH 13/13] Mark RFC-0003 as implementable Signed-off-by: Stefan Prodan --- rfcs/{kubernetes-oci => 0003-kubernetes-oci}/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename rfcs/{kubernetes-oci => 0003-kubernetes-oci}/README.md (99%) diff --git a/rfcs/kubernetes-oci/README.md b/rfcs/0003-kubernetes-oci/README.md similarity index 99% rename from rfcs/kubernetes-oci/README.md rename to rfcs/0003-kubernetes-oci/README.md index 90fd9137..4494b195 100644 --- a/rfcs/kubernetes-oci/README.md +++ b/rfcs/0003-kubernetes-oci/README.md @@ -1,10 +1,10 @@ -# RFC-xxxx Flux OCI support for Kubernetes manifests +# RFC-0003 Flux OCI support for Kubernetes manifests -**Status:** provisional +**Status:** implementable **Creation date:** 2022-03-31 -**Last update:** 2022-06-22 +**Last update:** 2022-07-06 ## Summary