From fa217b8775ce695eb4129b9c4f43db32cdbd72ba Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Thu, 20 Oct 2022 19:12:22 +0000 Subject: [PATCH 01/15] RFC: draft proposal for artifact revision fmt Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 175 +++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100644 rfcs/XXXX-artifact-revision-format/README.md diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md new file mode 100644 index 00000000..e9e15763 --- /dev/null +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -0,0 +1,175 @@ +# RFC-NNNN Artifact Revision format and introduction of digests + +**Status:** provisional + + + +**Creation date:** 2022-10-20 + +**Last update:** 2022-10-20 + +## Summary + +This RFC proposes to establish a canonical format for an `Artifact` which +points to a specific checksum (e.g. an OCI manifest digest or Git commit SHA) +of a named pointer (e.g. an OCI image tag or Git tag). In addition, it proposes +to include the algorithm name (e.g. `sha256`) as a prefix to any advertised +checksum in an `Artifact` and further referring to it as a `Digest` opposed to +a `Checksum`. + +## Motivation + +The current `Artifact` type's `Revision` field format is not "officially" +standardized (albeit assumed throughout our code bases), and has mostly been +derived from `GitRepository` which uses `/` as a delimiter between the named +pointer (a Git branch or tag) and a specific (SHA-1, or theoretical SHA-256) +revision. + +Since the introduction of `OCIRepository` and with the recent changes around +`HelmChart` objects to allow the consumption of charts from OCI registries. +This could be seen as outdated or confusing due to the format differing from +the canonical format used by OCI, which is `@:` (the +part after `@` formally known as a "digest") to refer to a specific version +of a tagged OCI manifest. + +While also taking note that Git does not have an official canonical format for +e.g. branch references at a specific commit, and `/` has less of a symbolic +meaning than `@`. Which could be interpreted as "`` _at_ +``". + +In addition, with the introduction of algorithm prefixes for an `Artifact`'s +checksum. It would be possible to add support and allow user configuration of +other algorithms than SHA-256. For example SHA-384 and SHA-512, or the more +performant (parallelizable) [BLAKE3][]. + +Besides this, it would make it easier to implement a client that can verify the +checksum without having to resort to an assumed format or guessing +method based on the length of it, and allows for a more robust solution in +which it can continue to calculate against the algorithm of a previous +configuration. + +The inclusion of the `Artifact`'s algorithm prefix has been proposed before in +[source-controller#855](https://github.com/fluxcd/source-controller/issues/855), +with supportive response from Core Maintainers. + +### Goals + +- Establish a canonical format to refer to an `Artifact`'s `Revision` field + which consists of a named pointer and a specific checksum reference. +- Allow easier verification of the `Artifact`'s checksum by including an + alias for the algorithm. +- Allow configuration of the algorithm used to calculate the checksum of an + `Artifact`. +- Allow configuration of [BLAKE3][] as an alternative to SHA for calculating + checksums. This has promising performance improvements over SHA-256, which + could allow for performance improvements in large scale environments. +- Allow compatability with SemVer name references which might contain an `@` + symbol already (e.g. `package@v1.0.0@sha256:...`, opposed to OCI's + `tag:v1.0.0@sha256:...`). + +### Non-Goals + +- Define a canonical format for an `Artifact`'s `Revision` field which contains + a named pointer and a different reference than a checksum. + +## Proposal + +### Establish an Artifact Revision format + +Change the format of the `Revision` field of the `source.toolkit.fluxcd.io` +Group's `Artifact` type across all `Source` kinds to contain an `@` delimiter +opposed to `/`, and include the algorithm alias as a prefix to the checksum +(creating a "digest"). + +```text +[ ] [ [ "@" ] ":" ] +``` + +Where `` is the name of e.g. a Git branch or OCI tag, +`` is the exact revision (e.g. a Git commit SHA or OCI manifest +digest), and `` is the alias of the algorithm used to calculate the +checksum (e.g. `sha256`). In case only a named pointer or digest is advertised, +the `@` is omitted. + +For a `GitRepository`'s `Artifact` pointing towards an SHA-256 Git commit on +branch `main`, the `Revision` field value would become: + +```text +main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +``` + +For a `GitRepository`'s `Artifact` pointing towards a specific SHA-1 Git commit +without a defined branch or tag, the `Revision` field value would become: + +```text +sha1:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +``` + +### Change the Artifact Checksum to a Digest + +Change the format of the `Checksum` field of the `source.toolkit.fluxcd.io` +Group's `Artifact` to `Digest`, and include the alias of the algorithm used to +calculate the Artifact checksum as a prefix (creating a "digest"). + +```text + ":" +``` + +For a `GitRepository` `Artifact`'s checksum calculated using SHA-256, the +`Digest` field value would become: + +```text +sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +``` + +### User Stories + +> As a user of the source-controller, I want to be able to see the exact +> revision of an Artifact that is being used, so that I can verify that it +> matches the expected revision. + +> As a user of the notification-controller, I want to be able to see the +> exact revision a notification is referring to. + +> As a Flux CLI user, I want to see the current revision of my Source in a +> listed overview. + + + +### Alternatives + + + +## Design Details + + + +## Implementation History + + + +[BLAKE3]: https://github.com/BLAKE3-team/BLAKE3 \ No newline at end of file From 2c12385344b4b749df755d48a775131fd6a8f418 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 21 Oct 2022 07:49:41 +0000 Subject: [PATCH 02/15] RFC: add revision format example for Buckets Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index e9e15763..de988481 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -9,7 +9,7 @@ Must be one of `provisional`, `implementable`, `implemented`, `deferred`, `rejec **Creation date:** 2022-10-20 -**Last update:** 2022-10-20 +**Last update:** 2022-10-21 ## Summary @@ -17,8 +17,8 @@ This RFC proposes to establish a canonical format for an `Artifact` which points to a specific checksum (e.g. an OCI manifest digest or Git commit SHA) of a named pointer (e.g. an OCI image tag or Git tag). In addition, it proposes to include the algorithm name (e.g. `sha256`) as a prefix to any advertised -checksum in an `Artifact` and further referring to it as a `Digest` opposed to -a `Checksum`. +checksum in an `Artifact` and further referring to it as a `Digest` as opposed +to a `Checksum`. ## Motivation @@ -67,7 +67,7 @@ with supportive response from Core Maintainers. checksums. This has promising performance improvements over SHA-256, which could allow for performance improvements in large scale environments. - Allow compatability with SemVer name references which might contain an `@` - symbol already (e.g. `package@v1.0.0@sha256:...`, opposed to OCI's + symbol already (e.g. `package@v1.0.0@sha256:...`, as opposed to OCI's `tag:v1.0.0@sha256:...`). ### Non-Goals @@ -108,6 +108,13 @@ without a defined branch or tag, the `Revision` field value would become: sha1:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ``` +For a `Bucket`'s `Artifact` with a revision based on an SHA-256 calculation of +a list of object keys and their etags, the `Revision` field value would become: + +```text +sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +``` + ### Change the Artifact Checksum to a Digest Change the format of the `Checksum` field of the `source.toolkit.fluxcd.io` From cac36365ae54e5a65329e34a56c7e534c2c2f509 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 21 Oct 2022 08:38:12 +0000 Subject: [PATCH 03/15] RFC: add additional user story about algo config Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index de988481..b159aa93 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -138,6 +138,9 @@ sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 > revision of an Artifact that is being used, so that I can verify that it > matches the expected revision. +> As a user of the source-controller, I want to be able to configure the +> algorithm used to calculate the checksum of an Artifact. + > As a user of the notification-controller, I want to be able to see the > exact revision a notification is referring to. From 879041677ce164e7b12e260f48ab72aa78057fb9 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 21 Oct 2022 10:47:57 +0000 Subject: [PATCH 04/15] RFC: write down user stories Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 76 ++++++++++++++++++-- 1 file changed, 72 insertions(+), 4 deletions(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index b159aa93..7f66de2d 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -134,22 +134,90 @@ sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ### User Stories +#### Artifact revision verification + > As a user of the source-controller, I want to be able to see the exact > revision of an Artifact that is being used, so that I can verify that it -> matches the expected revision. +> matches the expected revision at a remote source. + +For a Source kind that has an `Artifact` with a `Revision` which contains a +checksum, the field value can be retrieved using the Kubernetes API. For +example: + +```console +$ kubectl get gitrepository -o jsonpath='{.status.artifact.revision}' +main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +``` + +#### Artifact checksum verification + +> As a user of the source-controller, I want to be able to verify the checksum +> of an Artifact. + +For a Source kind with an `Aritfact` the digest consisting of the algorithm +alias and checksum is advertised in the `Digest` field, and can be retrieved +using the Kubernetes API. For example: + +```console +$ kubectl get gitrepository -o jsonpath='{.status.artifact.digest}' +sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +``` + +#### Artifact checksum algorithm configuration > As a user of the source-controller, I want to be able to configure the > algorithm used to calculate the checksum of an Artifact. +The source-controller binary accepts a `--artifact-digest-algo` flag which +configures the algorithm used to calculate the checksum of an `Artifact`. +The default value is `sha256`, but can be changed to `sha384`, `sha512` +or `blake3`. + +When set, newly advertised `Artifact`'s `Digest` fields will be calculated +using the configured algorithm. For previous `Artifact`'s that were set using +a previous configuration, the `Artifact`'s `Digest` field will be calculated +using the advertised algorithm. + +#### Artifact revisions in notifications + > As a user of the notification-controller, I want to be able to see the > exact revision a notification is referring to. +The notification-controller can use the revision for a Source's `Artifact` +attached as an annotation to an `Event`, and correctly parses the value field +when attempting to extract e.g. a Git commit digest from an event for a +`GitRepository`. As currently already applicable for the `/` delimiter. + +> As a user of the notification-controller, I want to be able to observe what +> commit has been applied on my (supported) Git provider. + +The notification-controller can use the revision attached as an annotation to +an `Event`, and is capable of extracting the correct reference for a Git +provider integration (e.g. GitHub, GitLab) to construct a payload. For example, +extracting `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` +from `main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`. + +#### Artifact revisions in listed views + > As a Flux CLI user, I want to see the current revision of my Source in a > listed overview. - +By running `flux get source `, the listed view of Sources would show a +truncated version of the checksum in the `Revision` field. + +```console +$ flux get source gitrepository +NAME REVISION SUSPENDED READY MESSAGE +flux-monitoring main@sha1:6f6c0979 False True stored artifact for revision 'main@sha1:6f6c0979809c12ce4aa687fb42be913f5dc78a75' + +$ flux get source oci +NAME REVISION SUSPENDED READY MESSAGE +apps-source local@sha256:b1ad9be6 False True stored artifact for digest 'local@sha256:b1ad9be6fe5fefc76a93f462ef2be1295fa6693d57e9d783780af99cd7234dc8' + +$ flux get source bucket +NAME REVISION SUSPENDED READY MESSAGE +apps-source sha256:e3b0c442 False True stored artifact for revision 'sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' +``` ### Alternatives From cf8ac4dd0e4258c2dfc0ba8d6dd48aaaf2ae5ea4 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 21 Oct 2022 11:07:04 +0000 Subject: [PATCH 05/15] RFC: document alternatives Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index 7f66de2d..5a220cc3 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -221,11 +221,17 @@ apps-source sha256:e3b0c442 False True stored artifact for rev ### Alternatives - +For the changes related to `Checksum` and `Digest`, the alternative is to keep +the current field name as is, and only change the field value format. However, +given the naming of the field is more correct with the introduction of the +algorithm alias, and now is the time to make last (breaking) changes to the +API. This does not strike as a better alternative. ## Design Details From 76c31c6303eb595c006a4a090c6f8516aa9f3f5b Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 21 Oct 2022 11:07:18 +0000 Subject: [PATCH 06/15] RFC: outline design details Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 26 ++++++++++++-------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index 5a220cc3..99201a4e 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -235,17 +235,23 @@ API. This does not strike as a better alternative. ## Design Details - +#### Format + +#### Parsing the Revision field + +### Artifact Digest + +#### Library + +#### Configuration + +#### Calculation + +#### Verification + +#### Deprecation of Checksum ## Implementation History From f92d708051fccabf5a09dd2c299cb84e32d2c13c Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 21 Oct 2022 13:38:28 +0000 Subject: [PATCH 07/15] RFC: add design details artifact revision Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 44 +++++++++++++++++++- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index 99201a4e..9b6706f0 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -66,7 +66,7 @@ with supportive response from Core Maintainers. - Allow configuration of [BLAKE3][] as an alternative to SHA for calculating checksums. This has promising performance improvements over SHA-256, which could allow for performance improvements in large scale environments. -- Allow compatability with SemVer name references which might contain an `@` +- Allow compatibility with SemVer name references which might contain an `@` symbol already (e.g. `package@v1.0.0@sha256:...`, as opposed to OCI's `tag:v1.0.0@sha256:...`). @@ -237,9 +237,49 @@ API. This does not strike as a better alternative. ### Artifact Revision format +For an `Artifact`'s `Revision` which contains a checksum referring to an exact +revision, the checksum within the value MUST be appended with an alias for the +algorithm separated by `:` (e.g. `sha256:...`), further referred to as a +"digest". The algorithm alias and checksum of the digest MUST be lowercase and +alphanumeric. + +For an `Artifact`'s `Revision` which contains a digest and a named pointer, +it MUST be prefixed with `@`, and appended at the end of the `Revision` value. +The named pointer MAY contain arbitrary characters, including but not limited +to `/` and `@`. + #### Format -#### Parsing the Revision field +```text +[ ] [ [ "@" ] ":" ] +``` + +Where `[ ]` indicates an optional element, `" "` a literal string, and `< >` +a variable. + +#### Parsing + +When parsing the `Revision` field value of an `Artifact` to extract the digest, +the value after the last `@` is considered to be the digest. The remaining +value on the left side is considered to be the named pointer, which MAY contain +an additional `@` delimiter if applicable for the domain of the `Source` +implementation. + +#### Truncation + +When truncating the `Revision` field value of an `Artifact` to display in a +view with limited space, the `` of the digest MAY be truncated to +7 or more characters. The `` of the digest MUST NOT be truncated. +In addition, a digest MUST always contain the full length checksum for the +algorithm. + +#### Backwards compatibility + +To allow backwards compatability in the notification-controller, Flux CLI and +other applicable components. The `Revision` new field value format could be +detected by the presence of the `@` or `:` characters. Falling back to their +current behaviour if not present, phasing out the old format in a future +release. ### Artifact Digest From 5faf6ebadc2ddec1bef799eceed54d870cabe5e3 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 21 Oct 2022 15:09:03 +0000 Subject: [PATCH 08/15] RFC: add design details artifact digest Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 50 +++++++++++++++++++- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index 9b6706f0..636aea6e 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -283,15 +283,60 @@ release. ### Artifact Digest +The `Artifact`'s `Digest` field advertises the checksum of the file in the +`URL`. The checksum within the value MUST be appended with an alias for the +algorithm separated by `:` (e.g. `sha256:...`). This follows the +[digest format][go-digest] of OCI. + +#### Format + +```text + ":" +``` + +Where `" "` indicates a literal string, and `< >` a variable. + #### Library +The library used for calculating the `Digest` field value is +`github.com/opencontainers/go-digest`. This library is used by various +OCI libraries which we already depend on, stable and extensible. + #### Configuration +The checksum MUST be calculated using the algorithm configured in the +`--artifact-digest-algo` flag of the source-controller binary. The default +value is `sha256`, but can be changed to `sha384`, `sha512` or `blake3`. + +**Note:** availability of BLAKE3 is at present dependent on an explicit import +of `github.com/opencontainers/go-digest/blake3`. + +When the provided algorithm is NOT supported, the source-controller MUST +fail to start. + +When the configured algorithm changes, the `Digest` MAY be recalculated to +update the value. + #### Calculation +The checksum MUST be calculated using the algorithm configured in the +`--artifact-digest-algo` flag of the source-controller binary. + #### Verification -#### Deprecation of Checksum +The checksum of a downloaded artifact MUST be verified against the `Digest` +field value. If the checksum does not match, the verification MUST fail. + +### Deprecation of Checksum + +The `Artifact`'s `Checksum` field is deprecated and MUST be removed in a +future release. The `Digest` field MUST be used instead. + +#### Backwards compatibility + +To allow backwards compatability, the source-controller could continue +to advertise the checksum part of a `Digest` in the `Checksum` field until +the field is removed. ## Implementation History @@ -302,4 +347,5 @@ Major milestones in the lifecycle of the RFC such as: - The version of Flux where the RFC was retired or superseded. --> -[BLAKE3]: https://github.com/BLAKE3-team/BLAKE3 \ No newline at end of file +[BLAKE3]: https://github.com/BLAKE3-team/BLAKE3 +[go-digest]: https://pkg.go.dev/github.com/opencontainers/go-digest#hdr-Basics \ No newline at end of file From 8801029d95e0ab12a59c540f7b79db0583d5232b Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Mon, 24 Oct 2022 09:32:29 +0000 Subject: [PATCH 09/15] RFC: avoid overlap between calculation and config Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index 636aea6e..0f3b7eed 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -9,7 +9,7 @@ Must be one of `provisional`, `implementable`, `implemented`, `deferred`, `rejec **Creation date:** 2022-10-20 -**Last update:** 2022-10-21 +**Last update:** 2022-10-24 ## Summary @@ -302,11 +302,17 @@ The library used for calculating the `Digest` field value is `github.com/opencontainers/go-digest`. This library is used by various OCI libraries which we already depend on, stable and extensible. +#### Calculation + +The checksum in the `Digest` field value MUST be calculated using the canonical +algorithm [set at runtime](#configuration). + #### Configuration -The checksum MUST be calculated using the algorithm configured in the -`--artifact-digest-algo` flag of the source-controller binary. The default -value is `sha256`, but can be changed to `sha384`, `sha512` or `blake3`. +The algorithm used for calculating the `Digest` field value MAY be configured +using the `--artifact-digest-algo` flag of the source-controller binary. The +default value is `sha256`, but can be changed to `sha384`, `sha512` or +`blake3`. **Note:** availability of BLAKE3 is at present dependent on an explicit import of `github.com/opencontainers/go-digest/blake3`. @@ -317,11 +323,6 @@ fail to start. When the configured algorithm changes, the `Digest` MAY be recalculated to update the value. -#### Calculation - -The checksum MUST be calculated using the algorithm configured in the -`--artifact-digest-algo` flag of the source-controller binary. - #### Verification The checksum of a downloaded artifact MUST be verified against the `Digest` From 6ccdfa074fb34a74e18e99d098cab42e2f558b30 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Mon, 24 Oct 2022 09:55:07 +0000 Subject: [PATCH 10/15] RFC: wording nit Signed-off-by: Hidde Beydals --- rfcs/XXXX-artifact-revision-format/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/XXXX-artifact-revision-format/README.md index 0f3b7eed..88cf3480 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/XXXX-artifact-revision-format/README.md @@ -229,7 +229,7 @@ format. Neither strike as a better alternative. For the changes related to `Checksum` and `Digest`, the alternative is to keep the current field name as is, and only change the field value format. However, -given the naming of the field is more correct with the introduction of the +given the naming of the field is more accurate with the introduction of the algorithm alias, and now is the time to make last (breaking) changes to the API. This does not strike as a better alternative. From c6be0b9389ecef0090837a91a7be958103ab236b Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Mon, 24 Oct 2022 09:59:04 +0000 Subject: [PATCH 11/15] RFC: add assigned reference number (0005) Signed-off-by: Hidde Beydals --- .../README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename rfcs/{XXXX-artifact-revision-format => 0005-artifact-revision-and-digest}/README.md (99%) diff --git a/rfcs/XXXX-artifact-revision-format/README.md b/rfcs/0005-artifact-revision-and-digest/README.md similarity index 99% rename from rfcs/XXXX-artifact-revision-format/README.md rename to rfcs/0005-artifact-revision-and-digest/README.md index 88cf3480..c900f704 100644 --- a/rfcs/XXXX-artifact-revision-format/README.md +++ b/rfcs/0005-artifact-revision-and-digest/README.md @@ -1,4 +1,4 @@ -# RFC-NNNN Artifact Revision format and introduction of digests +# RFC-0005 Artifact `Revision` format and introduction of `Digest` **Status:** provisional From 4b2af2ede25a195f2a6ddaa70f20aa179bde4b46 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Wed, 2 Nov 2022 15:21:50 +0000 Subject: [PATCH 12/15] RFC: address review nits Various typo and structural fixes. Signed-off-by: Hidde Beydals --- .../README.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/rfcs/0005-artifact-revision-and-digest/README.md b/rfcs/0005-artifact-revision-and-digest/README.md index c900f704..41d08bca 100644 --- a/rfcs/0005-artifact-revision-and-digest/README.md +++ b/rfcs/0005-artifact-revision-and-digest/README.md @@ -9,7 +9,7 @@ Must be one of `provisional`, `implementable`, `implemented`, `deferred`, `rejec **Creation date:** 2022-10-20 -**Last update:** 2022-10-24 +**Last update:** 2022-11-02 ## Summary @@ -24,24 +24,24 @@ to a `Checksum`. The current `Artifact` type's `Revision` field format is not "officially" standardized (albeit assumed throughout our code bases), and has mostly been -derived from `GitRepository` which uses `/` as a delimiter between the named +derived from `GitRepository` which uses `/` as a separator between the named pointer (a Git branch or tag) and a specific (SHA-1, or theoretical SHA-256) revision. Since the introduction of `OCIRepository` and with the recent changes around -`HelmChart` objects to allow the consumption of charts from OCI registries. -This could be seen as outdated or confusing due to the format differing from +`HelmChart` objects to allow the consumption of charts from OCI registries, +this could be seen as outdated or confusing due to the format differing from the canonical format used by OCI, which is `@:` (the part after `@` formally known as a "digest") to refer to a specific version of a tagged OCI manifest. While also taking note that Git does not have an official canonical format for e.g. branch references at a specific commit, and `/` has less of a symbolic -meaning than `@`. Which could be interpreted as "`` _at_ +meaning than `@`, which could be interpreted as "`` _at_ ``". In addition, with the introduction of algorithm prefixes for an `Artifact`'s -checksum. It would be possible to add support and allow user configuration of +checksum, it would be possible to add support and allow user configuration of other algorithms than SHA-256. For example SHA-384 and SHA-512, or the more performant (parallelizable) [BLAKE3][]. @@ -80,7 +80,7 @@ with supportive response from Core Maintainers. ### Establish an Artifact Revision format Change the format of the `Revision` field of the `source.toolkit.fluxcd.io` -Group's `Artifact` type across all `Source` kinds to contain an `@` delimiter +Group's `Artifact` type across all `Source` kinds to contain an `@` separator opposed to `/`, and include the algorithm alias as a prefix to the checksum (creating a "digest"). @@ -154,7 +154,7 @@ main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 > As a user of the source-controller, I want to be able to verify the checksum > of an Artifact. -For a Source kind with an `Aritfact` the digest consisting of the algorithm +For a Source kind with an `Artifact` the digest consisting of the algorithm alias and checksum is advertised in the `Digest` field, and can be retrieved using the Kubernetes API. For example: @@ -175,7 +175,7 @@ or `blake3`. When set, newly advertised `Artifact`'s `Digest` fields will be calculated using the configured algorithm. For previous `Artifact`'s that were set using -a previous configuration, the `Artifact`'s `Digest` field will be calculated +a previous configuration, the `Artifact`'s `Digest` field will be recalculated using the advertised algorithm. #### Artifact revisions in notifications @@ -186,7 +186,7 @@ using the advertised algorithm. The notification-controller can use the revision for a Source's `Artifact` attached as an annotation to an `Event`, and correctly parses the value field when attempting to extract e.g. a Git commit digest from an event for a -`GitRepository`. As currently already applicable for the `/` delimiter. +`GitRepository`. As currently already applicable for the `/` separator. > As a user of the notification-controller, I want to be able to observe what > commit has been applied on my (supported) Git provider. @@ -262,7 +262,7 @@ a variable. When parsing the `Revision` field value of an `Artifact` to extract the digest, the value after the last `@` is considered to be the digest. The remaining value on the left side is considered to be the named pointer, which MAY contain -an additional `@` delimiter if applicable for the domain of the `Source` +an additional `@` separator if applicable for the domain of the `Source` implementation. #### Truncation @@ -276,7 +276,7 @@ algorithm. #### Backwards compatibility To allow backwards compatability in the notification-controller, Flux CLI and -other applicable components. The `Revision` new field value format could be +other applicable components, the `Revision` new field value format could be detected by the presence of the `@` or `:` characters. Falling back to their current behaviour if not present, phasing out the old format in a future release. From fd08bae1c7ee70f2d9eceda6b1080b3a1b296a99 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Wed, 2 Nov 2022 15:46:33 +0000 Subject: [PATCH 13/15] RFC: reword summary Signed-off-by: Hidde Beydals --- rfcs/0005-artifact-revision-and-digest/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/rfcs/0005-artifact-revision-and-digest/README.md b/rfcs/0005-artifact-revision-and-digest/README.md index 41d08bca..6e148681 100644 --- a/rfcs/0005-artifact-revision-and-digest/README.md +++ b/rfcs/0005-artifact-revision-and-digest/README.md @@ -13,12 +13,12 @@ Must be one of `provisional`, `implementable`, `implemented`, `deferred`, `rejec ## Summary -This RFC proposes to establish a canonical format for an `Artifact` which -points to a specific checksum (e.g. an OCI manifest digest or Git commit SHA) -of a named pointer (e.g. an OCI image tag or Git tag). In addition, it proposes -to include the algorithm name (e.g. `sha256`) as a prefix to any advertised -checksum in an `Artifact` and further referring to it as a `Digest` as opposed -to a `Checksum`. +This RFC proposes to establish a canonical `Revision` format for an `Artifact` +which points to a specific revision represented as a checksum (e.g. an OCI +manifest digest or Git commit SHA) of a named pointer (e.g. an OCI image tag or +Git tag). In addition, it proposes to include the algorithm name (e.g. +`sha256`) as a prefix to an advertised checksum for an `Artifact` and +further referring to it as a `Digest`, deprecating the `Checksum` field. ## Motivation From d9102150cf60f6baad85ae6baf43b5402ad96e27 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 4 Nov 2022 08:51:08 +0000 Subject: [PATCH 14/15] RFC: address more nits - Properly refer to OCI repository name - Ensure checksum examples are unique to help distinguish difference between Revision and Digest - Slightly change proposal wordings to explicitly mention deprecation of `Checksum` field - Add reference to OCI digests spec Signed-off-by: Hidde Beydals --- .../README.md | 75 +++++++++++-------- 1 file changed, 45 insertions(+), 30 deletions(-) diff --git a/rfcs/0005-artifact-revision-and-digest/README.md b/rfcs/0005-artifact-revision-and-digest/README.md index 6e148681..c3a2d586 100644 --- a/rfcs/0005-artifact-revision-and-digest/README.md +++ b/rfcs/0005-artifact-revision-and-digest/README.md @@ -9,14 +9,14 @@ Must be one of `provisional`, `implementable`, `implemented`, `deferred`, `rejec **Creation date:** 2022-10-20 -**Last update:** 2022-11-02 +**Last update:** 2022-11-04 ## Summary This RFC proposes to establish a canonical `Revision` format for an `Artifact` which points to a specific revision represented as a checksum (e.g. an OCI -manifest digest or Git commit SHA) of a named pointer (e.g. an OCI image tag or -Git tag). In addition, it proposes to include the algorithm name (e.g. +manifest digest or Git commit SHA) of a named pointer (e.g. an OCI repository +name or Git tag). In addition, it proposes to include the algorithm name (e.g. `sha256`) as a prefix to an advertised checksum for an `Artifact` and further referring to it as a `Digest`, deprecating the `Checksum` field. @@ -31,9 +31,9 @@ revision. Since the introduction of `OCIRepository` and with the recent changes around `HelmChart` objects to allow the consumption of charts from OCI registries, this could be seen as outdated or confusing due to the format differing from -the canonical format used by OCI, which is `@:` (the -part after `@` formally known as a "digest") to refer to a specific version -of a tagged OCI manifest. +the canonical format used by OCI, which is `@:` (the +part after `@` formally known as a ["digest"][digest-spec]) to refer to a +specific version of an OCI manifest. While also taking note that Git does not have an official canonical format for e.g. branch references at a specific commit, and `/` has less of a symbolic @@ -61,14 +61,14 @@ with supportive response from Core Maintainers. which consists of a named pointer and a specific checksum reference. - Allow easier verification of the `Artifact`'s checksum by including an alias for the algorithm. +- Deprecate the `Artifact`'s `Checksum` field in favor of the `Digest` field. - Allow configuration of the algorithm used to calculate the checksum of an `Artifact`. -- Allow configuration of [BLAKE3][] as an alternative to SHA for calculating - checksums. This has promising performance improvements over SHA-256, which - could allow for performance improvements in large scale environments. +- Allow configuration of algorithms other than SHA-256 to calculate the + `Digest` of an `Artifact`. - Allow compatibility with SemVer name references which might contain an `@` symbol already (e.g. `package@v1.0.0@sha256:...`, as opposed to OCI's - `tag:v1.0.0@sha256:...`). + `name:v1.0.0@sha256:...`). ### Non-Goals @@ -88,38 +88,46 @@ opposed to `/`, and include the algorithm alias as a prefix to the checksum [ ] [ [ "@" ] ":" ] ``` -Where `` is the name of e.g. a Git branch or OCI tag, -`` is the exact revision (e.g. a Git commit SHA or OCI manifest +Where `` is the name of e.g. a Git branch or OCI repository +name, `` is the exact revision (e.g. a Git commit SHA or OCI manifest digest), and `` is the alias of the algorithm used to calculate the checksum (e.g. `sha256`). In case only a named pointer or digest is advertised, the `@` is omitted. -For a `GitRepository`'s `Artifact` pointing towards an SHA-256 Git commit on +For a `GitRepository`'s `Artifact` pointing towards an SHA-1 Git commit on branch `main`, the `Revision` field value would become: ```text -main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f ``` For a `GitRepository`'s `Artifact` pointing towards a specific SHA-1 Git commit without a defined branch or tag, the `Revision` field value would become: ```text -sha1:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f ``` For a `Bucket`'s `Artifact` with a revision based on an SHA-256 calculation of a list of object keys and their etags, the `Revision` field value would become: ```text -sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +sha256:8fb62a09c9e48ace5463bf940dc15e85f525be4f230e223bbceef6e13024110c ``` -### Change the Artifact Checksum to a Digest +For a `HelmChart`'s `Artifact` pointing towards a Helm chart version, the +`Revision` field value would become: -Change the format of the `Checksum` field of the `source.toolkit.fluxcd.io` -Group's `Artifact` to `Digest`, and include the alias of the algorithm used to -calculate the Artifact checksum as a prefix (creating a "digest"). +```text +1.2.3 +``` + +### Introduce a `Digest` field + +Introduce a new field to the `source.toolkit.fluxcd.io` Group's `Artifact` type +across all `Source` kinds called `Digest`, containing the checksum of the file +advertised in the `Path`, and alias of the algorithm used to calculate it +(creating a ["digest"][digest-spec]). ```text ":" @@ -129,9 +137,15 @@ For a `GitRepository` `Artifact`'s checksum calculated using SHA-256, the `Digest` field value would become: ```text -sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +sha256:1111f92aba67995f108b3ee3ffdc00edcfe206b11fbbb459c8ef4c4a8209fca8 ``` +#### Deprecate the `Checksum` field + +In favor of the `Digest` field, the `Checksum` field of the `source.toolkit.fluxcd.io` +Group's `Artifact` type across all `Source` kinds is deprecated, and removed in +a future version. + ### User Stories #### Artifact revision verification @@ -146,7 +160,7 @@ example: ```console $ kubectl get gitrepository -o jsonpath='{.status.artifact.revision}' -main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f ``` #### Artifact checksum verification @@ -160,7 +174,7 @@ using the Kubernetes API. For example: ```console $ kubectl get gitrepository -o jsonpath='{.status.artifact.digest}' -sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +sha256:1111f92aba67995f108b3ee3ffdc00edcfe206b11fbbb459c8ef4c4a8209fca8 ``` #### Artifact checksum algorithm configuration @@ -194,8 +208,8 @@ when attempting to extract e.g. a Git commit digest from an event for a The notification-controller can use the revision attached as an annotation to an `Event`, and is capable of extracting the correct reference for a Git provider integration (e.g. GitHub, GitLab) to construct a payload. For example, -extracting `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` -from `main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`. +extracting `1eabc9a41ca088515cab83f1cce49eb43e84b67f` from +`main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f`. #### Artifact revisions in listed views @@ -208,15 +222,15 @@ truncated version of the checksum in the `Revision` field. ```console $ flux get source gitrepository NAME REVISION SUSPENDED READY MESSAGE -flux-monitoring main@sha1:6f6c0979 False True stored artifact for revision 'main@sha1:6f6c0979809c12ce4aa687fb42be913f5dc78a75' +flux-monitoring main@sha1:1eabc9a4 False True stored artifact for revision 'main@sha1:1eabc9a41ca088515cab83f1cce49eb43e84b67f' $ flux get source oci NAME REVISION SUSPENDED READY MESSAGE -apps-source local@sha256:b1ad9be6 False True stored artifact for digest 'local@sha256:b1ad9be6fe5fefc76a93f462ef2be1295fa6693d57e9d783780af99cd7234dc8' +apps-source local@sha256:e5fa481b False True stored artifact for digest 'local@sha256:e5fa481bb17327bd269927d0a223862d243d76c89fe697ea8c9adefc47c47e17' $ flux get source bucket NAME REVISION SUSPENDED READY MESSAGE -apps-source sha256:e3b0c442 False True stored artifact for revision 'sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' +apps-source sha256:e3b0c442 False True stored artifact for revision 'sha256:8fb62a09c9e48ace5463bf940dc15e85f525be4f230e223bbceef6e13024110c' ``` ### Alternatives @@ -299,8 +313,8 @@ Where `" "` indicates a literal string, and `< >` a variable. #### Library The library used for calculating the `Digest` field value is -`github.com/opencontainers/go-digest`. This library is used by various -OCI libraries which we already depend on, stable and extensible. +`github.com/opencontainers/go-digest`. This library is stable and extensible, +and used by various OCI libraries which we already depend on. #### Calculation @@ -349,4 +363,5 @@ Major milestones in the lifecycle of the RFC such as: --> [BLAKE3]: https://github.com/BLAKE3-team/BLAKE3 +[digest-spec]: https://github.com/opencontainers/image-spec/blob/main/descriptor.md#digests [go-digest]: https://pkg.go.dev/github.com/opencontainers/go-digest#hdr-Basics \ No newline at end of file From 86e39919981876b58f0b89884e3a3c8de5f56385 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Wed, 16 Nov 2022 15:22:12 +0000 Subject: [PATCH 15/15] RFC: change 0005 status to implementable Signed-off-by: Hidde Beydals --- rfcs/0005-artifact-revision-and-digest/README.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/rfcs/0005-artifact-revision-and-digest/README.md b/rfcs/0005-artifact-revision-and-digest/README.md index c3a2d586..8036fbce 100644 --- a/rfcs/0005-artifact-revision-and-digest/README.md +++ b/rfcs/0005-artifact-revision-and-digest/README.md @@ -1,15 +1,10 @@ # RFC-0005 Artifact `Revision` format and introduction of `Digest` -**Status:** provisional - - +**Status:** implementable **Creation date:** 2022-10-20 -**Last update:** 2022-11-04 +**Last update:** 2022-11-16 ## Summary