actions/fluxcd-flux2
3
0
Fork 0
mirror of https://github.com/fluxcd/flux2.git synced 2026-04-14 15:56:52 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

Merge branch 'main' into rfc0005

Browse source
This commit is contained in:
gregoireW 2024-06-13 17:33:46 +02:00 • committed by GitHub
commit c9361ca55f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
564 changed files with 22937 additions and 11844 deletions
Download patch file Download diff file Expand all files Collapse all files

4
rfcs/0001-authorization/README.md
View file

@ -109,8 +109,8 @@ fields that are not restricted to the namespace of the containing object, listed
| | `.spec.targetNamespace` | This gives the namespace into which a Helm chart is installed (note: using impersonation) |
| | `.spec.storageNamespace` | This gives the namespace in which the record of a Helm install is created (note: using impersonation) |
| | `.spec.chart.spec.sourceRef` | This is a reference (in the created `HelmChart` object) that can include a namespace |
| **`alerts.notification.toolkit.fluxcd.io/v1beta1`** | `.spec.eventSources` | Items are references that can include a namespace |
| **`receivers.notification.toolkit.fluxcd.io/v1beta1`** | `.spec.resources` | Items in this field are references that can include a namespace |
| **`alerts.notification.toolkit.fluxcd.io/v1beta2`** | `.spec.eventSources` | Items are references that can include a namespace |
| **`receivers.notification.toolkit.fluxcd.io/v1beta2`** | `.spec.resources` | Items in this field are references that can include a namespace |
| **`imagepolicies.image.toolkit.fluxcd.io/v1beta1`** | `.spec.imageRepositoryRef` | This reference can include a namespace[1] |
[1] This particular cross-namespace reference is subject to additional access control; see "Access

282
rfcs/0002-helm-oci/README.md Normal file
View file

@ -0,0 +1,282 @@
# RFC-0002 Flux OCI support for Helm
**Status:** implemented (partially)
**Creation date:** 2022-03-30
**Last update:** 2023-11-28
## Summary
Given that Helm v3.8 supports [OCI](https://helm.sh/docs/topics/registries/) for package distribution,
we should extend the Flux Source API to allow fetching Helm charts from container registries.
## Motivation
Helm OCI support is one of the most requested feature in Flux
as seen on this [issue](https://github.com/fluxcd/source-controller/issues/124).
With OCI support, Flux users can automate chart updates to Git in the same way
they do today for container images.
### Goals
- Add support for fetching Helm charts stored as OCI artifacts with minimal API changes to Flux.
- Add support for verifying the authenticity of Helm OCI charts signed with Cosign.
- Make it easy for users to switch from [HTTP/S Helm repositories](https://github.com/helm/helm-www/blob/416fabea6ffab8dc156b6a0c5eb5e8df5f5ef7dc/content/en/docs/topics/chart_repository.md)
to OCI repositories.
### Non-Goals
- Introduce a new API kind for referencing charts stored as OCI artifacts.
## Proposal
Introduce an optional field called `type` to the `HelmRepository` spec.
When not specified, the `spec.type` field defaults to `default` which preserve the current `HelmRepository` API behaviour.
When the `spec.type` field is set to `oci`, the `spec.url` field must be prefixed with `oci://` (to follow the Helm conventions).
For `oci://` URLs, source-controller will use the Helm SDK and the `oras` library to connect to the OCI remote storage.
Introduce an optional field called `provider` for
[context-based authorization](https://fluxcd.io/flux/security/contextual-authorization/)
to AWS, Azure and Google Cloud. The `spec.provider` is ignored when `spec.type` is set to `default`.
### Pull charts from private repositories
#### Basic auth
For private repositories hosted on GitHub, Quay, self-hosted Docker Registry and others,
the credentials can be supplied with:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: <repo-name>
spec:
type: oci
secretRef:
name: regcred
```
The `secretRef` points to a Kubernetes secret in the same namespace as the `HelmRepository`.
The [secret type](https://kubernetes.io/docs/concepts/configuration/secret/#secret-types)
must be `kubernetes.io/dockerconfigjson`:
```shell
kubectl create secret docker-registry regcred \
--docker-server=<your-registry-server> \
--docker-username=<your-name> \
--docker-password=<your-pword>
```
#### OIDC auth
When Flux runs on AKS, EKS or GKE, an IAM role (that grants read-only access to ACR, ECR or GCR)
can be used to bind the `source-controller` to the IAM role.
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: <repo-name>
spec:
type: oci
provider: azure
```
The provider accepts the following values: `generic`, `aws`, `azure` and `gcp`. When the provider is
not specified, it defaults to `generic`. When the provider is set to `aws`, `azure` or `gcp`, the
controller will use a specific cloud SDK for authentication purposes.
If both `spec.secretRef` and a non-generic provider are present in the definition,
the controller will use the static credentials from the referenced secret.
### Verify Helm charts
To verify the authenticity of the Helm OCI charts, Flux will use the Sigstore Go SDK and implement verification
for artifacts which were either signed with keys generated by Cosign or signed using the Cosign
[keyless method](https://github.com/sigstore/cosign/blob/main/KEYLESS.md).
To enable signature verification, the Cosign public keys can be supplied with:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmChart
metadata:
name: <chart-name>
spec:
verify:
provider: cosign
secretRef:
name: cosign-public-keys
```
Note that the Kubernetes secret containing the Cosign public keys, must use `.pub` extension:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: cosign-public-keys
type: Opaque
stringData:
key1.pub: <pub-key-1>
key2.pub: <pub-key-2>
```
For verifying public Helm charts which are signed using the keyless method,
the `spec.verify.secretRef` field must be omitted:
```yaml
spec:
verify:
provider: cosign
```
When using the keyless method, Flux will verify the signatures in the Rekor
transparency log instance hosted at [rekor.sigstore.dev](https://rekor.sigstore.dev/).
### User Stories
#### Story 1
> As a developer I want to use Flux `HelmReleases` that refer to Helm charts stored
> as OCI artifacts in GitHub Container Registry.
First create a secret using a GitHub token that allows access to GHCR:
```sh
kubectl create secret docker-registry ghcr-charts \
--docker-server=ghcr.io \
--docker-username=$GITHUB_USER \
--docker-password=$GITHUB_TOKEN
```
Then define a `HelmRepository` of type `oci` and reference the `dockerconfig` secret:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: ghcr-charts
namespace: default
spec:
type: oci
url: oci://ghcr.io/my-org/charts/
secretRef:
name: ghcr-charts
```
And finally in Flux `HelmReleases`, refer to the ghcr-charts `HelmRepository`:
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: my-app
namespace: default
spec:
interval: 60m
chart:
spec:
chart: my-app
version: '1.0.x'
sourceRef:
kind: HelmRepository
name: ghcr-charts
interval: 1m # check for new OCI artifacts every minute
```
#### Story 2
> As a platform admin I want to automate Helm chart updates based on a semver ranges.
> When a new patch version is available in the container registry, I want Flux to open a PR
> with the version set in the `HelmRelease` manifests.
Given that charts are stored in container registries, you can use Flux image automation
and patch the chart version in Git, in the same way Flux works for updating container image tags.
Define an image registry and a policy for the chart artifact:
```yaml
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
name: my-app
namespace: default
spec:
image: ghcr.io/my-org/charts/my-app
interval: 1m0s
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
name: my-app
namespace: default
spec:
imageRepositoryRef:
name: my-app
policy:
semver:
range: 1.0.x
```
Then add the policy marker to the `HelmRelease` manifests in Git:
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: my-app
namespace: default
spec:
interval: 60m
chart:
spec:
chart: my-app
version: 1.0.0 # {"$imagepolicy": "default:my-app:tag"}
sourceRef:
kind: HelmRepository
name: ghcr-charts
interval: 1m
```
### Alternatives
We could introduce a new API type e.g. `HelmRegistry` to hold the reference to auth secret,
as proposed in [#2573](https://github.com/fluxcd/flux2/pull/2573).
That is considered unpractical, as there is no benefit for users in having a dedicated kind instead of
a `type` field in the current `HelmRepository` API. Adding a `type` field to the spec follows the Flux
Bucket API design, where the same Kind servers different implementations: AWS S3 vs Azure Blob vs Google Storage.
## Design Details
Unlike the default `HelmRepository`, the OCI `HelmRepository` does not need to
download any repository index file. The associated HelmChart can pull the chart
directly from the OCI registry based on the registry information in the
`HelmRepository` object. This makes the `HelmRepository` of type `oci` static,
not backed by a reconciler to move to a desired state. It becomes a data
container with information about the OCI registry.
In source-controller, the `HelmRepositoryReconciler` will be updated to check
the `.spec.type` field of `HelmRepository` and do nothing if it is `oci`.
The current `HelmChartReconciler` will be adapted to handle both types.
### Enabling the feature
The feature is enabled by default.
## Implementation History
* **2022-05-19** Partially implemented by [source-controller#690](https://github.com/fluxcd/source-controller/pull/690)
* **2022-06-06** First implementation released with [flux2 v0.31.0](https://github.com/fluxcd/flux2/releases/tag/v0.31.0)
* **2022-08-11** Resolve chart dependencies from OCI released with [flux2 v0.32.0](https://github.com/fluxcd/flux2/releases/tag/v0.32.0)
* **2022-08-29** Contextual login for AWS, Azure and GCP released with [flux2 v0.33.0](https://github.com/fluxcd/flux2/releases/tag/v0.33.0)
* **2022-10-21** Verifying Helm charts with Cosign released with [flux2 v0.36.0](https://github.com/fluxcd/flux2/releases/tag/v0.36.0)
* **2023-11-28** Update the design of HelmRepository of type OCI to be static object [flux2 v2.2.0](https://github.com/fluxcd/flux2/releases/tag/v2.2.0)
### TODOs
* [Add support for container registries with self-signed TLS certs](https://github.com/fluxcd/source-controller/issues/723)

486
rfcs/0003-kubernetes-oci/README.md Normal file
View file

@ -0,0 +1,486 @@
# RFC-0003 Flux OCI support for Kubernetes manifests
**Status:** implemented
**Creation date:** 2022-03-31
**Last update:** 2023-11-07
## Summary
Flux should be able to distribute and reconcile Kubernetes configuration packaged as OCI artifacts.
On the client-side, the Flux CLI should offer a command for packaging Kubernetes configs into
an OCI artifact and pushing the artifact to a container registry using the Docker config file
and the Docker credential helpers for authentication.
On the server-side, the Flux source-controller should offer a dedicated API Kind for defining
how OCI artifacts are pulled from container registries and how the artifact's authenticity can be verified.
Flux should be able to work with any type of artifact even if it's not created with the Flux CLI.
## Motivation
Given that OCI registries are evolving into a generic artifact storage solution,
we should extend Flux to allow fetching Kubernetes manifests and related configs
from container registries similar to how Flux works with Git and Bucket storage.
With OCI support, Flux users can automate artifact updates to Git in the same way
they do today for container images.
### Goals
- Add support to the Flux CLI for packaging Kubernetes manifests and related configs into OCI artifacts.
- Add support to Flux source-controller for fetching configs stored as OCI artifacts.
- Make it easy for users to switch from Git repositories and Buckets to OCI repositories.
### Non-Goals
- Enforce a specific OCI media type for artifacts containing Kubernetes manifests or any other configs.
## Proposal
### Push artifacts
Flux users should be able to package a local directory containing Kubernetes configs into a tarball
and push the archive to a container registry as an OCI artifact.
```sh
flux push artifact oci://docker.io/org/app-config:v1.0.0 \
--source="$(git config --get remote.origin.url)" \
--revision="sha1:$(git rev-parse HEAD)" \
--path="./deploy"
```
The Flux CLI will produce OCI artifacts by setting the config layer
media type to `application/vnd.cncf.flux.config.v1+json`.
The directory pointed to by `--path` is archived and compressed in the `tar+gzip` format
and the layer media type is set to `application/vnd.cncf.flux.content.v1.tar+gzip`.
The source and revision are added to the OCI artifact as Open Containers standard annotations:
```json
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"annotations": {
"org.opencontainers.image.created": "2023-02-10T09:06:09Z",
"org.opencontainers.image.revision": "sha1:6ea3e5b4da159fcb4a1288f072d34c3315644bcc",
"org.opencontainers.image.source": "https://github.com/fluxcd/flux2"
}
}
```
To ease the promotion workflow of a specific version from one environment to another, the CLI
should offer a tagging command.
```sh
flux tag artifact oci://docker.io/org/app-config:v1.0.0 --tag=latest --tag=production
```
To view all the available artifacts in a repository and their metadata, the CLI should
offer a list command.
```sh
flux list artifacts oci://docker.io/org/app-config
```
To help inspect artifacts, the Flux CLI will offer a `build` and a `pull` command for generating
tarballs locally and for downloading the tarballs from remote container registries.
```sh
flux build artifact --path ./deploy --output tmp/artifact.tgz
flux pull artifact oci://docker.io/org/app-config:v1.0.0 --output ./manifests
```
### Pull artifacts
Flux users should be able to define a source for pulling manifests inside the cluster from an OCI repository.
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: app-config
namespace: flux-system
spec:
interval: 10m
url: oci://docker.io/org/app-config
ref:
tag: v1.0.0
```
The `spec.url` field points to the container image repository in the format `oci://<host>:<port>/<org-name>/<repo-name>`.
Note that specifying a tag or digest is not in accepted for this field. The `spec.url` value is used by the controller
to fetch the list of tags from the remote OCI repository.
An `OCIRepository` can refer to an artifact by tag, digest or semver range:
```yaml
spec:
ref:
# one of
tag: "latest"
digest: "sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2"
semver: "6.0.x"
```
### Layer selection
By default, Flux assumes that the first layer of the OCI artifact contains the Kubernetes configuration.
For multi-layer artifacts created by other tools than Flux CLI
(e.g. [oras](https://github.com/oras-project/oras),
[crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane)),
users can specify the [media type](https://github.com/opencontainers/image-spec/blob/v1.0.2/media-types.md) of the layer
which contains the tarball with Kubernetes manifests.
```yaml
spec:
layerSelector:
mediaType: "application/vnd.cncf.flux.content.v1.tar+gzip"
```
If the layer selector matches more than one layer,
the first layer matching the specified media type will be used.
Note that Flux requires that the OCI layer is
[compressed](https://github.com/opencontainers/image-spec/blob/v1.0.2/layer.md#gzip-media-types)
in the `tar+gzip` format.
### Pull artifacts from private repositories
For authentication purposes, Flux users can choose between supplying static credentials with Kubernetes secrets
and cloud-based OIDC using an IAM role binding to the source-controller Kubernetes service account.
#### Basic auth
For private repositories hosted on DockerHub, GitHub, Quay, self-hosted Docker Registry and others,
the credentials can be supplied with:
```yaml
spec:
secretRef:
name: regcred
```
The `secretRef` points to a Kubernetes secret in the same namespace as the `OCIRepository`,
the secret type must be `kubernetes.io/dockerconfigjson`:
```shell
kubectl create secret docker-registry regcred \
--docker-server=<your-registry-server> \
--docker-username=<your-name> \
--docker-password=<your-pword>
```
For image pull secrets attached to a service account, the account name can be specified with:
```yaml
spec:
serviceAccountName: regsa
```
#### Client cert auth
For private repositories which require a certificate to authenticate,
the client certificate, private key and the CA certificate (if self-signed), can be provided with:
```yaml
spec:
certSecretRef:
name: regcert
```
The `certSecretRef` points to a Kubernetes secret in the same namespace as the `OCIRepository`:
```shell
kubectl create secret generic regcert \
--from-file=certFile=client.crt \
--from-file=keyFile=client.key \
--from-file=caFile=ca.crt
```
#### OIDC auth
When Flux runs on AKS, EKS or GKE, an IAM role (that grants read-only access to ACR, ECR or GCR)
can be used to bind the `source-controller` to the IAM role.
```yaml
spec:
provider: aws
```
The provider accepts the following values: `generic`, `aws`, `azure` and `gcp`. When the provider is
not specified, it defaults to `generic`. When the provider is set to `aws`, `azure` or `gcp`, the
controller will use a specific cloud SDK for authentication purposes. If both `spec.secretRef` and
a non-generic provider are present in the definition, the controller will use the static credentials
from the referenced secret.
### Verify artifacts
To verify the authenticity of the OCI artifacts, Flux will use the Sigstore Go SDK and implement verification
for artifacts which were either signed with keys generated by Cosign or signed using the Cosign
[keyless method](https://github.com/sigstore/cosign/blob/main/KEYLESS.md).
To enable signature verification, the Cosign public key can be supplied with:
```yaml
spec:
verify:
provider: cosign
secretRef:
name: cosign-key
```
For verifying public artifacts which are signed using the keyless method,
the `.spec.verify.matchOIDCIdentity` field must be used instead of
`spec.verify.secretRef`.
```yaml
spec:
verify:
provider: cosign
matchOIDCIdentity:
- issuer: "^https://token.actions.githubusercontent.com$"
subject: "^https://github.com/org/app-repository.*$"
```
The `matchOIDCIdentity` entries must contain the following fields:
- `.issuer`, regexp that matches against the OIDC issuer.
- `.subject`, regexp that matches against the subject identity in the certificate.
The entries are evaluated in an OR fashion, i.e. the identity is deemed to be
verified if any one entry successfully matches against the identity.
When using the keyless method, Flux will verify the signatures in the Rekor
transparency log instance hosted at [rekor.sigstore.dev](https://rekor.sigstore.dev/).
### Reconcile artifacts
The `OCIRepository` can be used as a drop-in replacement for `GitRepository` and `Bucket` sources.
For example, a Flux Kustomization can refer to an `OCIRepository` and reconcile the manifests found in the OCI artifact:
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: app
namespace: flux-system
spec:
interval: 10m
sourceRef:
kind: OCIRepository
name: app-config
path: ./
```
### User Stories
#### Story 1
> As a developer I want to publish my app Kubernetes manifests to the same GHCR registry
> where I publish my app containers.
First login to GHCR with Docker:
```sh
docker login ghcr.io -u ${GITHUB_USER} -p ${GITHUB_TOKEN}
```
Build your app container image and push it to GHCR:
```sh
docker build -t ghcr.io/org/my-app:v1.0.0 .
docker push ghcr.io/org/my-app:v1.0.0
```
Edit the app deployment manifest and set the new image tag.
Then push the Kubernetes manifests to GHCR:
```sh
flux push artifact oci://ghcr.io/org/my-app-config:v1.0.0 \
--source="$(git config --get remote.origin.url)" \
--revision="sha1:$(git rev-parse HEAD)"\
--path="./deploy"
```
Sign the config image with cosign:
```sh
cosign sign --key cosign.key ghcr.io/org/my-app-config:v1.0.0
```
Mark `v1.0.0` as latest:
```sh
flux tag artifact oci://ghcr.io/org/my-app-config:v1.0.0 --tag latest
```
List the artifacts and their metadata with:
```console
$ flux list artifacts oci://ghcr.io/org/my-app-config
ARTIFACT DIGEST SOURCE REVISION
ghcr.io/org/my-app-config:latest sha256:45b95019d30af335137977a369ad56e9ea9e9c75bb01afb081a629ba789b890c https://github.com/org/my-app-config.git sha1:20b3a674391df53f05e59a33554973d1cbd4d549
ghcr.io/org/my-app-config:v1.0.0 sha256:45b95019d30af335137977a369ad56e9ea9e9c75bb01afb081a629ba789b890c https://github.com/org/my-app-config.git sha1:3f45e72f0d3457e91e3c530c346d86969f9f4034
```
#### Story 2
> As a developer I want to deploy my app using Kubernetes manifests published as OCI artifacts to GHCR.
First create a secret using a GitHub token that allows access to GHCR:
```sh
kubectl create secret docker-registry my-app-regcred \
--docker-server=ghcr.io \
--docker-username=$GITHUB_USER \
--docker-password=$GITHUB_TOKEN
```
Then create a secret with your cosgin public key:
```sh
kubectl create secret generic my-app-cosgin-key \
--from-file=cosign.pub=cosign/my-key.pub
```
Then define an `OCIRepository` to fetch and verify the latest app config version:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: app-config
namespace: default
spec:
interval: 10m
url: oci://ghcr.io/org/my-app-config
ref:
semver: "1.x"
secretRef:
name: my-app-regcred
verify:
provider: cosign
secretRef:
name: my-app-cosgin-key
```
And finally, create a Flux Kustomization to reconcile the app on the cluster:
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: app
namespace: default
spec:
interval: 10m
sourceRef:
kind: OCIRepository
name: app-config
path: ./deploy
prune: true
wait: true
timeout: 2m
```
## Design Details
The Flux controllers and CLI will use the [fluxcd/pkg/oci](https://github.com/fluxcd/pkg/tree/main/oci)
library for OCI operations such as push, pull, tag, list tags, etc.
For authentication purposes, the `flux <verb> artifact` commands will use the `~/.docker/config.json`
config file and the Docker credential helpers. On Cloud VMs without Docker installed, the CLI will
use context-based authorization for AWS, Azure and GCP.
The Flux CLI will produce OCI artifacts with the following format:
```json
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.cncf.flux.config.v1+json",
"size": 233,
"digest": "sha256:1b80ecb1c04d4a9718a6094a00ed17b76ea8ff2bb846695fa38e7492d34f505c"
},
"layers": [
{
"mediaType": "application/vnd.cncf.flux.content.v1.tar+gzip",
"size": 19081,
"digest": "sha256:46c2b334705cd08db1a6fa46f860cd3364fc1a3636eea37a9b35537549086a1c"
}
],
"annotations": {
"org.opencontainers.image.created": "2023-02-10T09:06:09Z",
"org.opencontainers.image.revision": "sha1:6ea3e5b4da159fcb4a1288f072d34c3315644bcc",
"org.opencontainers.image.source": "https://github.com/fluxcd/flux2"
}
}
```
The source-controller will extract the first layer from the OCI artifact, and will repackage it
as an internal `sourcev1.Artifact`. The internal artifact revision will be set to the OCI SHA256 digest
and the OpenContainers annotation will be copied to the internal artifact metadata:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
creationTimestamp: "2022-06-22T09:14:19Z"
finalizers:
- finalizers.fluxcd.io
generation: 1
name: podinfo
namespace: oci
resourceVersion: "6603"
uid: 42e0b9f0-021c-476d-86c7-2cd20747bfff
spec:
interval: 10m
ref:
tag: 6.1.6
timeout: 60s
url: oci://ghcr.io/stefanprodan/manifests/podinfo
status:
artifact:
checksum: d7e924b4882e55b97627355c7b3d2e711e9b54303afa2f50c25377f4df66a83b
lastUpdateTime: "2022-06-22T09:14:21Z"
metadata:
org.opencontainers.image.created: "2023-02-10T09:06:09Z"
org.opencontainers.image.revision: sha1:b3b00fe35424a45d373bf4c7214178bc36fd7872
org.opencontainers.image.source: https://github.com/stefanprodan/podinfo.git
path: ocirepository/oci/podinfo/3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de.tar.gz
revision: sha256:3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de
size: 1105
url: http://source-controller.flux-system.svc.cluster.local./ocirepository/oci/podinfo/3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de.tar.gz
conditions:
- lastTransitionTime: "2022-06-22T09:14:21Z"
message: stored artifact for revision 'sha256:3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de'
observedGeneration: 1
reason: Succeeded
status: "True"
type: Ready
- lastTransitionTime: "2022-06-22T09:14:21Z"
message: stored artifact for revision 'sha256:3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de'
observedGeneration: 1
reason: Succeeded
status: "True"
type: ArtifactInStorage
observedGeneration: 1
url: http://source-controller.flux-system.svc.cluster.local./ocirepository/oci/podinfo/latest.tar.gz
```
### Enabling the feature
The feature is enabled by default.
## Implementation History
* **2022-08-08** Partially implemented by [source-controller#788](https://github.com/fluxcd/source-controller/pull/788)
* **2022-08-11** First implementation released with [flux2 v0.32.0](https://github.com/fluxcd/flux2/releases/tag/v0.32.0)
* **2022-08-29** Select layer by OCI media type released with [flux2 v0.33.0](https://github.com/fluxcd/flux2/releases/tag/v0.33.0)
* **2022-09-29** Verifying OCI artifacts with Cosign released with [flux2 v0.35.0](https://github.com/fluxcd/flux2/releases/tag/v0.35.0)
* **2023-02-20** Custom OCI media types released with [flux2 v0.40.0](https://github.com/fluxcd/flux2/releases/tag/v0.40.0)
* **2023-10-31** OIDC identity verification implemented in
[source-controller#1250](https://github.com/fluxcd/source-controller/pull/1250)

230
rfcs/0004-insecure-http/README.md Normal file
View file

@ -0,0 +1,230 @@
# RFC-0004 Block insecure HTTP connections across Flux
**Status:** implementable
**Creation Date:** 2022-09-08
**Last update:** 2023-07-26
## Summary
Flux should have a consistent way of disabling insecure HTTP connections.
At the controller level, a flag should be present which would disable all outgoing HTTP connections.
At the object level, a field should be provided which would enable the use of non-TLS endpoints.
If the use of a non-TLS endpoint is not supported, reconciliation will fail and the object will be marked
as stalled, signalling that human intervention is required.
## Motivation
Today the use of non-TLS based connections is inconsistent across Flux controllers.
Controllers that deal only with `http` and `https` schemes have no way to block use of the `http` scheme at controller-level.
Some Flux objects provide a `.spec.insecure` field to enable the use of non-TLS based endpoints, but they don't clearly notify
users when the option is not supported (e.g. Azure/GCP Buckets).
### Goals
* Provide a flag across relevant Flux controllers which disables all outgoing HTTP connections.
* Add a field which enables the use of non-TLS endpoints to appropriate Flux objects.
* Provide a way for users to be made aware that their use of non-TLS endpoints is not supported if that is the case.
### Non-Goals
* Break Flux's current behavior of allowing HTTP connections.
* Change in behavior of communication between Flux components.
## Proposal
### Controllers
Flux users should be able to enforce that controllers are using HTTPS connections only.
This shall be enabled by adding a new boolean flag `--insecure-allow-http` to the following controllers:
* source-controller
* notification-controller
* image-automation-controller
* image-reflector-controller
The default value of this flag shall be `true`. This would ensure that there is no breaking change with controllers
still being able to access non-TLS endpoints. To disable this behavior and enforce the use of HTTPS connections, users would
have to explicitly pass the flag to the controller:
```yaml
spec:
template:
spec:
containers:
- name: manager
image: fluxcd/source-controller
args:
- --watch-all-namespaces
- --log-level=info
- --log-encoding=json
- --enable-leader-election
- --storage-path=/data
- --storage-adv-addr=source-controller.$(RUNTIME_NAMESPACE).svc.cluster.local.
- --insecure-allow-http=false
```
**Note:** The flag shall not be added to the following controllers:
* kustomize-controller: This flag is excluded from this controller, as the upstream `kubenetes-sigs/kustomize` project
does not support disabling HTTP connections while fetching resources from remote bases. We can revisit this if the
upstream project adds support for this at a later point in time.
* helm-controller: This flag does not serve a purpose in this controller, as the controller does not make any HTTP calls.
Furthermore although both controllers can also do remote applies, serving `kube-apiserver` over plain
HTTP is disabled by default. While technically this can be enabled, the option for this configuration was also disabled
quite a while back (ref: https://github.com/kubernetes/kubernetes/pull/65830/).
### Objects
Some Flux objects, like `GitRepository`, provide a field for specifying a URL, and the URL would contain the scheme.
In such cases, the scheme can be used for inferring the transport type of the connection and consequently,
whether to use HTTP or HTTPS connections for that object.
But there are a few objects that don't allow such behavior, for example:
* `ImageRepository`: It provides a field, `.spec.image`, which is used for specifying the address of the image present on
a container registry. But any address containing a scheme is considered invalid and HTTPS is the default transport used.
This prevents users from using images present on insecure registries.
* OCI `HelmRepository`: When using an OCI registry as a Helm repository, the `.spec.url` is expected to begin with `oci://`.
Since the scheme part of the URL is used to specify the type of `HelmRepository`, there is no way for users to specify
that the registry is hosted at a non-TLS endpoint.
For such objects, we shall introduce a new boolean field `.spec.insecure`, which shall be `false` by default. Users that
need their object to point to an HTTP endpoint, can set this to `true`.
### CLI
The Flux CLI offers several commands for creating Flux specific resources. Some of these commands may involve specifying
an endpoint such as creating an `OCIRepository`:
```sh
flux create source oci podinfo \
--url=oci://ghcr.io/stefanprodan/manifests/podinfo \
--tag=6.1.6 \
--interval=10m
```
Since these commands essentially create object definitions, the CLI should offer a boolean flag `--insecure`
for the required commands, which will be used for specifying the value of `.spec.insecure` of such objects.
> Note: This flag should not be confused with `--insecure-skip-tls-verify` which is meant to skip TLS verification
> when using an HTTPS connection.
### Proxy
The flag shall also apply to all possible proxy configurations. If the flag `--insecure-allow-http` is set to
`false`, then specifying the `HTTP_PROXY` environment variable to the controller will lead to the controller
exiting with a failure on startup. This also applies for when the `HTTPS_PROXY` enviornment variable's value is
a URL that has `http` as its scheme.
Similarly, if a proxy is specified using the object's API, such as through `.spec.secretRef` in `Provider` in the
`notification.toolkit.fluxcd.io` API group and the proxy URL has `http` as its scheme, the reconciler will fail and
return an error, which can be viewed in the controller logs and the object's events.
### Precedence & Validity
Objects with `.spec.insecure` as `true` will only be allowed if HTTP connections are allowed at the controller level.
Similarly, an object can have `.spec.insecure` as `true` only if the Saas/Cloud provider allows HTTP connections.
For example, using a `Bucket` with its `.spec.provider` set to `azure` would be invalid since Azure doesn't allow
HTTP connections.
### User Stories
#### Story 1
> As a cluster admin of a multi-tenant cluster, I want to ensure all controllers access endpoints using only HTTPS
> regardless of tenants' object definitions.
Apply a `kustomize` patch which prevents the use of HTTP connections:
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- patch: |
- op: add
path: /spec/template/spec/containers/0/args/-
value: --insecure-allow-http=false
target:
kind: Deployment
name: "(source-controller|notification-controller|image-reflector-controller|image-automation-controller)"
# Since the above flag is not available in kustomize-controller for reasons explained in a previous section,
# we disable Kustomize remote builds by disallowing use of remote bases. This ensures that kustomize-controller
# won't initiate any plain HTTP connections.
- patch: |
- op: add
path: /spec/template/spec/containers/0/args/-
value: --no-remote-bases=true
target:
kind: Deployment
name: kustomize-controller
```
#### Story 2
> As an application developer, I'm trying to debug a new image pushed to my local registry which
> is not served over HTTPS.
Modify the object spec to use HTTP connections explicitly:
```yaml
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
name: podinfo
namespace: flux-system
spec:
image: kind-registry:5000/stefanprodan/podinfo
interval: 1m0s
insecure: true
```
### Alternatives
Instead of adding a flag, we can instruct users to make use of Kyverno policies to enforce that
all objects have `.spec.insecure` as `false` and any URLs present in the definition don't have `http`
as the scheme. This is less attractive, as this would ask users to install another software and prevent
Flux multi-tenancy from being standalone.
## Design Details
If a controller is started with `--insecure-allow-http=false`, any URL in a Flux object which has `http`
as the scheme will result in an unsuccessful reconciliation and the following condition will be added to the object's
`.status.conditions`:
```yaml
status:
conditions:
- lastTransitionTime: "2022-09-06T09:14:21Z"
message: "Use of insecure HTTP connections isn't allowed for this controller"
observedGeneration: 1
reason: InsecureConnectionsDisallowed
status: "True"
type: Stalled
```
Similarly, if an object has `.spec.insecure` as `true` but the Cloud provider doesn't allow HTTP connections,
the reconciliation will fail and the following condition will be added to the object's `.status.conditions`:
```yaml
status:
conditions:
- lastTransitionTime: "2022-09-06T09:14:21Z"
message: "Use of insecure HTTP connections isn't allowed for Azure Storage"
observedGeneration: 1
reason: UnsupportedConnectionType
status: "True"
type: Stalled
```
If an object has `.spec.insecure` as `true`, the registry client or bucket client shall be created with the use
of HTTP connections enabled explicitly.
## Implementation History
**2022-08-12** Allow defining OCI sources for non-TLS container registries with `flux create source oci --insecure`
released with [flux2 v0.34.0](https://github.com/fluxcd/flux2/releases/tag/v0.34.0)

357
rfcs/0005-artifact-revision-and-digest/README.md Normal file
View file

@ -0,0 +1,357 @@
# RFC-0005 Artifact `Revision` format and introduction of `Digest`
**Status:** implemented
**Creation date:** 2022-10-20
**Last update:** 2023-02-20
## Summary
This RFC proposes to establish a canonical `Revision` format for an `Artifact`
which points to a specific revision represented as a checksum (e.g. an OCI
manifest digest or Git commit SHA) of a named pointer (e.g. an OCI repository
name or Git tag). In addition, it proposes to include the algorithm name (e.g.
`sha256`) as a prefix to an advertised checksum for an `Artifact` and
further referring to it as a `Digest`, deprecating the `Checksum` field.
## Motivation
The current `Artifact` type's `Revision` field format is not "officially"
standardized (albeit assumed throughout our code bases), and has mostly been
derived from `GitRepository` which uses `/` as a separator between the named
pointer (a Git branch or tag) and a specific (SHA-1, or theoretical SHA-256)
revision.
Since the introduction of `OCIRepository` and with the recent changes around
`HelmChart` objects to allow the consumption of charts from OCI registries,
this could be seen as outdated or confusing due to the format differing from
the canonical format used by OCI, which is `<name>@<algo>:<checksum>` (the
part after `@` formally known as a ["digest"][digest-spec]) to refer to a
specific version of an OCI manifest.
While also taking note that Git does not have an official canonical format for
e.g. branch references at a specific commit, and `/` has less of a symbolic
meaning than `@`, which could be interpreted as "`<branch>` _at_
`<commit SHA>`".
In addition, with the introduction of algorithm prefixes for an `Artifact`'s
checksum, it would be possible to add support and allow user configuration of
other algorithms than SHA-256. For example SHA-384 and SHA-512, or the more
performant (parallelizable) [BLAKE3][].
Besides this, it would make it easier to implement a client that can verify the
checksum without having to resort to an assumed format or guessing
method based on the length of it, and allows for a more robust solution in
which it can continue to calculate against the algorithm of a previous
configuration.
The inclusion of the `Artifact`'s algorithm prefix has been proposed before in
[source-controller#855](https://github.com/fluxcd/source-controller/issues/855),
with supportive response from Core Maintainers.
### Goals
- Establish a canonical format to refer to an `Artifact`'s `Revision` field
which consists of a named pointer and a specific checksum reference.
- Allow easier verification of the `Artifact`'s checksum by including an
alias for the algorithm.
- Deprecate the `Artifact`'s `Checksum` field in favor of the `Digest` field.
- Allow configuration of the algorithm used to calculate the checksum of an
`Artifact`.
- Allow configuration of algorithms other than SHA-256 to calculate the
`Digest` of an `Artifact`.
- Allow compatibility with SemVer name references which might contain an `@`
symbol already (e.g. `package@v1.0.0@sha256:...`, as opposed to OCI's
`name:v1.0.0@sha256:...`).
### Non-Goals
- Define a canonical format for an `Artifact`'s `Revision` field which contains
a named pointer and a different reference than a checksum.
## Proposal
### Establish an Artifact Revision format
Change the format of the `Revision` field of the `source.toolkit.fluxcd.io`
Group's `Artifact` type across all `Source` kinds to contain an `@` separator
opposed to `/`, and include the algorithm alias as a prefix to the checksum
(creating a "digest").
```text
[ <named pointer> ] [ [ "@" ] <algo> ":" <checksum> ]
```
Where `<named pointer>` is the name of e.g. a Git branch or OCI repository
name, `<checksum>` is the exact revision (e.g. a Git commit SHA or OCI manifest
digest), and `<algo>` is the alias of the algorithm used to calculate the
checksum (e.g. `sha256`). In case only a named pointer or digest is advertised,
the `@` is omitted.
For a `GitRepository`'s `Artifact` pointing towards an SHA-1 Git commit on
branch `main`, the `Revision` field value would become:
```text
main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f
```
For a `GitRepository`'s `Artifact` pointing towards a specific SHA-1 Git commit
without a defined branch or tag, the `Revision` field value would become:
```text
sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f
```
For a `Bucket`'s `Artifact` with a revision based on an SHA-256 calculation of
a list of object keys and their etags, the `Revision` field value would become:
```text
sha256:8fb62a09c9e48ace5463bf940dc15e85f525be4f230e223bbceef6e13024110c
```
For a `HelmChart`'s `Artifact` pointing towards a Helm chart version, the
`Revision` field value would become:
```text
1.2.3
```
### Introduce a `Digest` field
Introduce a new field to the `source.toolkit.fluxcd.io` Group's `Artifact` type
across all `Source` kinds called `Digest`, containing the checksum of the file
advertised in the `Path`, and alias of the algorithm used to calculate it
(creating a ["digest"][digest-spec]).
```text
<algo> ":" <checksum>
```
For a `GitRepository` `Artifact`'s checksum calculated using SHA-256, the
`Digest` field value would become:
```text
sha256:1111f92aba67995f108b3ee3ffdc00edcfe206b11fbbb459c8ef4c4a8209fca8
```
#### Deprecate the `Checksum` field
In favor of the `Digest` field, the `Checksum` field of the `source.toolkit.fluxcd.io`
Group's `Artifact` type across all `Source` kinds is deprecated, and removed in
a future version.
### User Stories
#### Artifact revision verification
> As a user of the source-controller, I want to be able to see the exact
> revision of an Artifact that is being used, so that I can verify that it
> matches the expected revision at a remote source.
For a Source kind that has an `Artifact` with a `Revision` which contains a
checksum, the field value can be retrieved using the Kubernetes API. For
example:
```console
$ kubectl get gitrepository -o jsonpath='{.status.artifact.revision}' <name>
main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f
```
#### Artifact checksum verification
> As a user of the source-controller, I want to be able to verify the checksum
> of an Artifact.
For a Source kind with an `Artifact` the digest consisting of the algorithm
alias and checksum is advertised in the `Digest` field, and can be retrieved
using the Kubernetes API. For example:
```console
$ kubectl get gitrepository -o jsonpath='{.status.artifact.digest}' <name>
sha256:1111f92aba67995f108b3ee3ffdc00edcfe206b11fbbb459c8ef4c4a8209fca8
```
#### Artifact checksum algorithm configuration
> As a user of the source-controller, I want to be able to configure the
> algorithm used to calculate the checksum of an Artifact.
The source-controller binary accepts a `--artifact-digest-algo` flag which
configures the algorithm used to calculate the checksum of an `Artifact`.
The default value is `sha256`, but can be changed to `sha384`, `sha512`
or `blake3`.
When set, newly advertised `Artifact`'s `Digest` fields will be calculated
using the configured algorithm. For previous `Artifact`'s that were set using
a previous configuration, the `Artifact`'s `Digest` field will be recalculated
using the advertised algorithm.
#### Artifact revisions in notifications
> As a user of the notification-controller, I want to be able to see the
> exact revision a notification is referring to.
The notification-controller can use the revision for a Source's `Artifact`
attached as an annotation to an `Event`, and correctly parses the value field
when attempting to extract e.g. a Git commit digest from an event for a
`GitRepository`. As currently already applicable for the `/` separator.
> As a user of the notification-controller, I want to be able to observe what
> commit has been applied on my (supported) Git provider.
The notification-controller can use the revision attached as an annotation to
an `Event`, and is capable of extracting the correct reference for a Git
provider integration (e.g. GitHub, GitLab) to construct a payload. For example,
extracting `1eabc9a41ca088515cab83f1cce49eb43e84b67f` from
`main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f`.
#### Artifact revisions in listed views
> As a Flux CLI user, I want to see the current revision of my Source in a
> listed overview.
By running `flux get source <kind>`, the listed view of Sources would show a
truncated version of the checksum in the `Revision` field.
```console
$ flux get source gitrepository
NAME REVISION SUSPENDED READY MESSAGE
flux-monitoring main@sha1:1eabc9a4 False True stored artifact for revision 'main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f'
$ flux get source oci
NAME REVISION SUSPENDED READY MESSAGE
apps-source local@sha256:e5fa481b False True stored artifact for digest 'local@sha256:e5fa481bb17327bd269927d0a223862d243d76c89fe697ea8c9adefc47c47e17'
$ flux get source bucket
NAME REVISION SUSPENDED READY MESSAGE
apps-source sha256:e3b0c442 False True stored artifact for revision 'sha256:8fb62a09c9e48ace5463bf940dc15e85f525be4f230e223bbceef6e13024110c'
```
### Alternatives
The two main alternatives around the `Revision` parts in this RFC are to either
keep the current field value formats as is, or to invent another format. Given
the [motivation](#motivation) for this RFC outlines the reasoning for not
keeping the current `Revision` format, and the proposed is a commonly known
format. Neither strike as a better alternative.
For the changes related to `Checksum` and `Digest`, the alternative is to keep
the current field name as is, and only change the field value format. However,
given the naming of the field is more accurate with the introduction of the
algorithm alias, and now is the time to make last (breaking) changes to the
API. This does not strike as a better alternative.
## Design Details
### Artifact Revision format
For an `Artifact`'s `Revision` which contains a checksum referring to an exact
revision, the checksum within the value MUST be appended with an alias for the
algorithm separated by `:` (e.g. `sha256:...`), further referred to as a
"digest". The algorithm alias and checksum of the digest MUST be lowercase and
alphanumeric.
For an `Artifact`'s `Revision` which contains a digest and a named pointer,
it MUST be prefixed with `@`, and appended at the end of the `Revision` value.
The named pointer MAY contain arbitrary characters, including but not limited
to `/` and `@`.
#### Format
```text
[ <named pointer> ] [ [ "@" ] <algo> ":" <checksum> ]
```
Where `[ ]` indicates an optional element, `" "` a literal string, and `< >`
a variable.
#### Parsing
When parsing the `Revision` field value of an `Artifact` to extract the digest,
the value after the last `@` is considered to be the digest. The remaining
value on the left side is considered to be the named pointer, which MAY contain
an additional `@` separator if applicable for the domain of the `Source`
implementation.
#### Truncation
When truncating the `Revision` field value of an `Artifact` to display in a
view with limited space, the `<checksum>` of the digest MAY be truncated to
7 or more characters. The `<algo>` of the digest MUST NOT be truncated.
In addition, a digest MUST always contain the full length checksum for the
algorithm.
#### Backwards compatibility
To allow backwards compatibility in the notification-controller, Flux CLI and
other applicable components, the `Revision` new field value format could be
detected by the presence of the `@` or `:` characters. Falling back to their
current behaviour if not present, phasing out the old format in a future
release.
### Artifact Digest
The `Artifact`'s `Digest` field advertises the checksum of the file in the
`URL`. The checksum within the value MUST be appended with an alias for the
algorithm separated by `:` (e.g. `sha256:...`). This follows the
[digest format][go-digest] of OCI.
#### Format
```text
<algo> ":" <checksum>
```
Where `" "` indicates a literal string, and `< >` a variable.
#### Library
The library used for calculating the `Digest` field value is
`github.com/opencontainers/go-digest`. This library is stable and extensible,
and used by various OCI libraries which we already depend on.
#### Calculation
The checksum in the `Digest` field value MUST be calculated using the canonical
algorithm [set at runtime](#configuration).
#### Configuration
The algorithm used for calculating the `Digest` field value MAY be configured
using the `--artifact-digest-algo` flag of the source-controller binary. The
default value is `sha256`, but can be changed to `sha384`, `sha512` or
`blake3`.
**Note:** availability of BLAKE3 is at present dependent on an explicit import
of `github.com/opencontainers/go-digest/blake3`.
When the provided algorithm is NOT supported, the source-controller MUST
fail to start.
When the configured algorithm changes, the `Digest` MAY be recalculated to
update the value.
#### Verification
The checksum of a downloaded artifact MUST be verified against the `Digest`
field value. If the checksum does not match, the verification MUST fail.
### Deprecation of Checksum
The `Artifact`'s `Checksum` field is deprecated and MUST be removed in a
future release. The `Digest` field MUST be used instead.
#### Backwards compatibility
To allow backwards compatibility, the source-controller could continue
to advertise the checksum part of a `Digest` in the `Checksum` field until
the field is removed.
## Implementation History
* **2023-02-20** First implementation released with [flux2 v0.40.0](https://github.com/fluxcd/flux2/releases/tag/v0.40.0)
[BLAKE3]: https://github.com/BLAKE3-team/BLAKE3
[digest-spec]: https://github.com/opencontainers/image-spec/blob/main/descriptor.md#digests
[go-digest]: https://pkg.go.dev/github.com/opencontainers/go-digest#hdr-Basics

BIN
rfcs/0006-cdevents/CDEvents-Flux-RFC-Adapter.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 177 KiB

BIN
rfcs/0006-cdevents/Flux-CDEvents-RFC.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 39 KiB

109
rfcs/0006-cdevents/README.md Normal file
View file

@ -0,0 +1,109 @@
# RFC-0006 Flux CDEvents Receiver
**Status:** implementable
<!--
Status represents the current state of the RFC.
Must be one of `provisional`, `implementable`, `implemented`, `deferred`, `rejected`, `withdrawn`, or `replaced`.
-->
**Creation date:** 2023-12-08
**Last update:** 2024-03-13
## Summary
This RFC proposes to add a `Receiver` type to the Flux notification-controller API
for handling [CDEvents](https://cdevents.dev/).
For `Receiver` objects configured to accept CDEvents,
notification-controller will verify the events sent to the receiver's webhook URL,
check that their type matches the expected type, and trigger the reconciliation
of the configured resources.
## Motivation
CDEvents enables interoperability between CI/CD tools in a workflow, and Flux is a
very popular continuous delivery tool, and consequently the CDF team received many questions
about integrating CDEvents with Flux.
### Goals
Allow Flux to receive CDEvents and trigger the reconciliation of resources based on the received events.
### Non-Goals
Make the Flux controllers emit CDEvents.
## Proposal
Add CDEvents to the list of available receivers in Flux notification-controller.
Similar to other receivers such as GitHub, Flux users will be able to use `spec.events`
in order to specify which event types the receiver will allow.
The receiver will also verify using the [CDEvents Go SDK](https://github.com/cdevents/sdk-go) that the
payload sent to the webhook URL is a valid CDEvent.
### User Stories
Users of multiple CI/CD tools such as Tekton and Flux
could use CDEvents as a way to enable interoperability.
For example, a user may want a Flux resource to reconcile as part of a Tekton `pipeline`.
The Tekton `pipeline` will fire off a CDEvent to the CloudEvents Broker.
A subscription that the user will have set up externally, e.g. with the [knative broker](https://knative.dev/docs/eventing/brokers/), will then
send a relevant CDEvent to the Flux webhook receiver endpoint.
![usecase](cdevents-flux-tekton.png)
### Alternatives
Certain use cases for CDEvents could be done alternatively using
available receivers such as the generic webhook.
## Design Details
Adding a Flux `Receiver` for CDEvents that works much like the other event-based receivers already implemented.
The user will be able to define a Flux `Receiver` custom resource and deploy it to their cluster.
The receiver takes the payload sent to the webhook URL by an external events broker,
checks the headers for the event type, and filters out events based on the user-defined
list of events in `spec.events`. If left empty, it will act on all valid CDEvents.
It then validates the payload body using the [CDEvents Go SDK](https://github.com/cdevents/sdk-go).
Valid events will then trigger the reconciliation of all Flux objects specified in `.spec.resources`.
The CDEvents broker is not a part of this design and is left to the users to set up however they wish.
Example Receiver:
```yaml
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
name: cdevents-receiver
namespace: flux-system
spec:
type: cdevents
events:
- "dev.cdevents.change.merged"
secretRef:
name: receiver-token
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: webapp
namespace: flux-system
```
![User Flowchart](Flux-CDEvents-RFC.png)
![Adapter](CDEvents-Flux-RFC-Adapter.png)
## Implementation History
<!--
Major milestones in the lifecycle of the RFC such as:
- The first Flux release where an initial version of the RFC was available.
- The version of Flux where the RFC graduated to general availability.
- The version of Flux where the RFC was retired or superseded.
-->

BIN
rfcs/0006-cdevents/cdevents-flux-tekton.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 649 KiB

404
rfcs/0007-git-repo-passwordless-auth/README.md Normal file
View file

@ -0,0 +1,404 @@
# RFC-0007 Passwordless authentication for Git repositories
**Status:** implementable
**Creation date:** 2023-31-07
**Last update:** 2024-06-12
## Summary
Flux should provide a mechanism to authenticate against Git repositories without
the use of passwords. This RFC proposes the use of alternative authentication
methods like OIDC, OAuth2 and IAM to access Git repositories hosted on specific
Git SaaS platforms and cloud providers.
## Motivation
At the moment, Flux supports HTTP basic and bearer authentication. Users are
required to create a Secret containing the username and the password/bearer
token, which is then referred to in the GitRepository using `.spec.secretRef`.
While this works fine, it has a couple of drawbacks:
* Scalability: Each new GitRepository potentially warrants another credentials
pair, which doesn't scale well in big organizations with hundreds of
repositories with different owners, increasing the risk of mismanagement and
leaks.
* Identity: A username is associated with an actual human. But often, the
repository belongs to a team of 2 or more people. This leads to a problem where
teams have to decide whose credentials should Flux use for authentication.
These problems exist not due to flaws in Flux, but because of the inherent
nature of password based authentication.
With support for OIDC, OAuth2 and IAM based authentication, we can eliminate
these problems:
* Scalability: Since OIDC is fully handled by the cloud provider, it eliminates
any user involvement in managing credentials. For OAuth2 and IAM, users do need
to provide certain information like the ID of the resource, private key, etc.
but these are still a better alternative to passwords since the same resource
can be reused by multiple teams with different members.
* Identity: Since all the above authentication methods are associated with a
virtual resource independent of a user, it solves the problem of a single person
being tied to automation that several people are involved in.
### Goals
* Integrate with major cloud providers' OIDC and IAM offerings to provide a
seamless way of Git repository authentication.
* Integrate with major Git SaaS providers to support their app based OAuth2
mechanism.
### Non-Goals
* Replace the existing basic and bearer authentication API.
## Proposal
A new string field `.spec.provider` shall be added to the `GitRepository` API.
The field will be an enum with the following variants:
* `generic`
* `aws`
* `azure`
* `gcp`
* `github`
* `gitlab`
`.spec.provider` will be an optional field which defaults to `generic` indicating
that the user wants to authenticate via HTTP basic/bearer auth or SSH by providing
the existing `.spec.secretRef` field. The sections below define the behavior when
`.spec.provider` is set to one of the other providers.
### AWS
Git repositories hosted on AWS CodeCommit can be accessed by Flux via [IAM roles
for service accounts
(IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)
and
[git-remote-codecommit (GRC)](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-git-remote-codecommit.html)
signed URLs.
The IAM role associated with service account used in Flux can be granted access
to the CodeCommit repository. The Flux service account can be patched with the
name of the IAM role to be assumed as an annotation. The CodeCommit HTTPS (GRC)
repository URL is of the format `codecommit::<region>://<repo-name>`. This can
be converted to a signed URL before performing a go-git Git operation.
The following patch can be used to add the IAM role name to Flux service accounts:
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- patch: |
apiVersion: v1
kind: ServiceAccount
metadata:
name: source-controller
annotations:
eks.amazonaws.com/role-arn: <role arn>
target:
kind: ServiceAccount
name: source-controller
```
Example of using AWS CodeCommit with `aws` provider:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: aws-repo
spec:
interval: 1m
url: codecommit::<region>://<repository>
ref:
branch: master
provider: aws
```
### Azure
Git repositories hosted on Azure Devops can be accessed using [managed
identity](https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/service-principal-managed-identity?view=azure-devops).
Seamless access from Flux to Azure devops repository can be achieved through
[Workload
Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet).
The user creates a managed identity and establishes a federated identity between
Flux service account and the managed identity. Flux service account is patched
to add an annotation specifying the client id of the managed identity. Flux
service account and deployments are patched with labels to use workload
identity. The managed identity must have sufficient permissions to be able to
access Azure Devops resources. This enables Flux pod to access the Git
repository without the need for any credentials.
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- patch: |-
apiVersion: v1
kind: ServiceAccount
metadata:
name: source-controller
namespace: flux-system
annotations:
azure.workload.identity/client-id: <AZURE_CLIENT_ID>
labels:
azure.workload.identity/use: "true"
- patch: |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: source-controller
namespace: flux-system
labels:
azure.workload.identity/use: "true"
spec:
template:
metadata:
labels:
azure.workload.identity/use: "true"
```
Example of using an Azure Devops repository with `azure` provider:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: azure-devops
spec:
interval: 1m
url: https://dev.azure.com/<org>/<project>/_git/<repository>
ref:
branch: master
# notice the lack of secretRef
provider: azure
```
### GCP
Git repositories hosted on Google Cloud Source Repositories can be accessed by
Flux via a [GCP Service Account](https://cloud.google.com/iam/docs/service-account-overview).
Workload Identity Federation for GKE is [unsupported](https://cloud.google.com/iam/docs/federated-identity-supported-services)
for Cloud Source Repositories. The user must instead create the GCP Service Account and
link it to the Flux service account in order to enable workload identity.
In order to link the GCP Service Account to the Flux service
account, the following patch must be applied:
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- patch: |
apiVersion: v1
kind: ServiceAccount
metadata:
name: source-controller
annotations:
iam.gke.io/gcp-service-account: <identity-name>
target:
kind: ServiceAccount
name: source-controller
```
The Service Account must have sufficient permissions to be able to access Google
Cloud Source Repositories. The Cloud Source Repositories uses the `source.repos.get`
permission to access the repository, which is under the `roles/source.reader` role.
Take a look at [this guide](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
for more information about setting up GKE Workload Identity.
Example of using a Google Cloud Source Repository with `gcp` provider:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: gcp-repo
spec:
interval: 1m
url: https://source.developers.google.com/p/<project>/r/<repository>
ref:
branch: master
provider: gcp
```
### GitHub
Git repositories hosted on GitHub can be accessed via [GitHub Apps](https://docs.github.com/en/apps/overview).
This allows users to create a single resource from which they can access all
their GitHub repositories. The app must have sufficient permissions to be able
to access repositories. The app's ID, private key and installation ID should
be mentioned in the Secret referred to by `.spec.secretRef`. GitHub Enterprise
users will also need to mention their GitHub API URL in the Secret.
Example of using a Github repository with `github` provider:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: github-repo
spec:
interval: 1m
url: https://github.com/<org>/<repository>
ref:
branch: master
provider: github
secretRef:
name: github-app
---
kind: Secret
metadata:
name: github-sa
stringData:
githubAppID: <app-id>
githubInstallationID: <installation-id>
githubPrivateKey: |
<PEM-private-key>
githubApiURl: <github-enterprise-api-url> #optional, required only for GitHub Enterprise users
```
### Gitlab
Git repositories hosted on Gitlab can be accessed via OAuth2 Gitlab Applications
created from the
[UI](https://docs.gitlab.com/ee/integration/oauth_provider.html) or using
[API](https://docs.gitlab.com/ee/api/applications.html). The Gitlab Oauth2
application must be created with the required scope to access gitlab
repositories. The application's `application_id`, `secret` and `redirect_uri`
are used to request an [access
token](https://docs.gitlab.com/ee/api/oauth2.html#authorization-code-flow).
These parameters are configured in the secret referred to by `.spec.secretRef`.
Example of using gitlab repository with `gitlab` provider:
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: gitlab-repo
spec:
interval: 1m
url: https://gitlab.com/<org>/<repository>
ref:
branch: main
provider: gitlab
secretRef:
name: gitlab-app
---
kind: Secret
metadata:
name: gitlab-app
stringData:
gitlabAppID: <app-id>
gitlabAppSecret: <app-secret>
gitlabAppRedirectUrl: <app-redirect-url>
```
### User Stories
#### User Story 1
> As a user running flux controllers, deployed from a private repository in
> a cloud provider that supports context-based authentication, I want to securely
> authenticate to the repository without setting up secrets and having to manage
> authentication tokens (refreshing, rotating, etc.).
To enable this scenario, the user would enable context-based authentication in
their cloud provider and integrate it with their kubernetes cluster. For example,
in Azure, using AKS and Azure Devops, the user would create a managed identity and
establish a federated identity between Flux service account and the managed identity.
Flux would then be able to access the Git repository by requesting a token from the
Azure service. The user would not need to create a secret or manage any tokens.
#### User Story 2
> As a user running flux controllers, deployed from a private repository, I want
> to configure authentication to the repository that is not associated to a
> personal account and does not expire.
To enable this scenario, the user would either enable context-based authentication
in their cloud provider and integrate it with their kubernetes cluster, or set
up an OAuth2 application in their Git SaaS provider and provide the OAuth2 application
details (application ID, secret, redirect URL) in a kubernetes secret.
Flux would then be able to access the Git repository by requesting a token from the
cloud provider or Git SaaS provider. The user would not need to create any credentials
tied to a personal account.
## Design Details
Flux source controller uses `GitRepository` API to define a source to produce an
Artifact for a Git repository revision. Flux image automation controller updates
YAML files when new images are available and commits changes to a given Git
repository. The `ImageUpdateAutomation` API defines an automation process that
updates the Git repository referenced in it's `.spec.sourceRef`. If the new
optional string field `.spec.provider` is specified in the `GitRepository` API,
the respective provider is used to configure the authentication to check out the
source for flux controllers.
### AWS
If `.spec.provider` is set to `aws`, Flux controllers will use the aws-sdk-go-v2
to assume the role of the IAM role associated with the pod service account and
obtain a short-lived [Security Token Service
(STS)](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html)
credential. This credential will then be used to create a signed HTTP URL to the
CodeCommit repository, similar to what git-remote-codecommit (GRC) does in
python using the boto library, see
[here](https://github.com/aws/git-remote-codecommit/blob/1.17/git_remote_codecommit/__init__.py#L176-L194).
For example, the GRC URL `codecommit::us-east-1://test-repo-1` results in a
typical Git HTTP repository address `https://AKIAYKF23ZCZFAVYGOEX:20240607T151729Zf17c9b36ba154efc81adf3df9dc3253de52e0a1ab6c81c00a5f9a26b06a103df@git-codecommit.us-east-1.amazonaws.com/v1/repos/test-repo-1`.
This URL contains a basic auth credential. This can be passed to go-git to
perform HTTP Git operations.
### Azure
If `.spec.provider` is set to `azure`, Flux controllers will use
[DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential)
to build the workload identity credential. This credential type uses the
environment variables injected by the Azure Workload Identity mutating webhook.
The [access token from the credential will be then used as a bearer
token](https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/service-principal-managed-identity?view=azure-devops#q-can-i-use-a-service-principal-to-do-git-operations-like-clone-a-repo)
to perform HTTP bearer authentication.
### GCP
If `.spec.provider` is set to `gcp`, Flux source controller will fetch the access token
from the [GKE metadata server](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#metadata_server).
The GKE metadata server runs as a DaemonSet, with one Pod on every Linux node or
a native Windows service on every Windows node in the cluster. The metadata server
intercepts HTTP requests to `http://metadata.google.internal`.
The source controller will use the url `http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token`
to retrieve a token for the IAM service account that the Pod is configured to impersonate.
This access token will be then used to perform HTTP basic authentication.
### GitHub
If `.spec.provider` is set to `github`, Flux controllers will get the app
details from the specified Secret and use it to [generate an app installation
token](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-an-installation-access-token-for-a-github-app).
This token is then used as the password and [`x-access-token` as the username](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/choosing-permissions-for-a-github-app#choosing-permissions-for-git-access)
to perform HTTP basic authentication.
### Gitlab
If `.spec.provider` is set to `gitlab`, Flux controllers will use the
application_id, secret and redirect_url specified in `.spec.secret` to generate
an access token. The git repository can then be accessed by specifying [oauth2
as the username and the access token as the
password](https://docs.gitlab.com/ee/api/oauth2.html#access-git-over-https-with-access-token)
to perform HTTP basic authentication.

2
rfcs/RFC-0000/README.md
View file

@ -51,7 +51,7 @@ you're proposing, but should not include things like API designs or
implementation.
If the RFC goal is to document best practices,
then this section can be replaced with the the actual documentation.
then this section can be replaced with the actual documentation.
-->
### User Stories