schwarzit.stackit-public/audit-go
1
0
Fork 0
mirror of https://dev.azure.com/schwarzit/schwarzit.stackit-public/_git/audit-go synced 2026-02-17 13:21:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update README.md

Browse source
This commit is contained in:
Christian Schaible 2024-08-29 12:59:46 +02:00
parent 0a8516f4a3
commit eb22259346
1 changed files with 47 additions and 168 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

231
README.md
View file

@ -1,113 +1,18 @@
## Audit Log API ## audit-go
The audit log api is a library that simplifies sending audit logs to the audit log system.
### Logging API The audit-go library is the core library for validation and sending of audit events.
The audit api provides two ways to log audit events: ### API Documentation
- **Direct logging** The api documentation can be found [here]().
Logs are directly validated, serialized and sent via solace.
It's the responsibility of the sending service to handle errors and to retry in case of an error.
```
Log(
ctx context.Context,
event *auditV1.AuditEvent,
visibility auditV1.Visibility,
routingIdentifier *RoutingIdentifier,
objectIdentifier *auditV1.ObjectIdentifier,
) error
```
- **Transactional safe logging (transactional outbox pattern)** TODO update the link
Functionality is split into two steps:
- Serialization + Validation
The `ValidateAndSerialize` method can be called to validate and serialize the event data.
The serialized data can be stored in an outbox table / collection in
the database.
```
ValidateAndSerialize(
event *auditV1.AuditEvent,
visibility auditV1.Visibility,
routingIdentifier *RoutingIdentifier,
objectIdentifier *auditV1.ObjectIdentifier,
) (SerializedPayload, error)
```
- Sending data to solace ### Supported data types for routing
The `Send` method can be used to send the previously serialized data.
It is the responsibility of the sending service to ensure that the
method is called (repetitive) until no error is reported.
```
Send(
ctx context.Context,
routingIdentifier *RoutingIdentifier,
serializedPayload *SerializedPayload,
) error
```
### Usage The following data types are currently supported for routing.
1. Create Solace client / acl rules There are `SingularType` and `PluralType` representations in the code for them.
The topic names have to be compliant to the format as specified
[here](https://async-api.stackit.schwarz/core-platform/asyncapi.html#operation-publish-stackit-platform/t/swz/audit-log/{region}/{version}/{eventSource}/{additionalParts}).
A solace client has to be provisioned manually with terraform (for
[dev](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/00_dev/essolske01/stackit/client_usernames.tf) /
[qa](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/02_qs/qssolske01/stackit/client_usernames.tf) /
[prod](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/03_prod/pssolske01/stackit/client_usernames.tf)).
The permission to write to the topic needs to be specified in the terraform configuration (for
[dev](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/00_dev/essolske01/stackit/acl_profiles.tf) /
[qa](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/02_qs/qssolske01/stackit/acl_profiles.tf) /
[prod](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/03_prod/pssolske01/stackit/acl_profiles.tf)).
2. Instantiation of the messaging API
The audit log api uses solace as messaging system. The actual implementation for the
connection to the messaging system is implemented by specifying an abstraction and providing a
concrete implementation for AMQP/Solace.
```go
messagingApi, err := NewAmqpMessagingApi(AmqpConfig{URL: "...", User: "", Password: ""})
if err != nil { ... }
```
3. Instantiation of the audit log api
There will be a new Audit API in the future as audit logs will be routed dynamically
to different data sinks in the future.
The plan though is to keep the actual methods that (validates, serializes and) sends the log statements as stable
as possible only replacing the API instantiation part later.
The instantiation of the legacy api is shown below:
```
validator, err := protovalidate.New()
if err != nil { ... }
auditApi, err := NewLegacyAuditApi(
messagingApi,
LegacyTopicNameConfig{TopicName: topicName},
validator,
)
```
For each Solace topic data should be sent to, a new object instance of the legacy API is needed.
The `messagingAPI` object can be shared across the audit api object instances.
4. Logging audit events
As described in [Logging API](#logging-api) messages can be validated, serialized and sent
through the API.
It is important to retry as long as errors are returned to ensure that the message is really sent.
### Examples
The test code can be taken as an inspiration how to use the API.
- [api_legacy_test.go](./api_legacy_test.go)
### Tests
For users of the API who integrate the library there is an additional implementation available for
testing purpose. The `MockAuditApi` does not require a connection to a messaging system (therefore,
does not send any data via messaging). It only serializes and validates data.
It can be instantiated as follows:
```
auditApi, err := NewMockAuditApi()
if err != nil { ... }
```
### Supported data types
| Singular-Type | Plural-Type | Routable to customer | Description | | Singular-Type | Plural-Type | Routable to customer | Description |
|---------------|---------------|----------------------|----------------------| |---------------|---------------|----------------------|----------------------|
| system | system | no | The STACKIT system | | system | system | no | The STACKIT system |
@ -115,61 +20,8 @@ if err != nil { ... }
| organization | organizations | yes | STACKIT organization | | organization | organizations | yes | STACKIT organization |
| folder | folders | yes | STACKIT folder | | folder | folders | yes | STACKIT folder |
### Customization ### Additional API implementations
#### os.Stdout logging
The audit log api uses `slog` to print messages on `os.Stdout` as it is part
of the standard library (Go >= 1.21).
To configure slog to print json instead of plain text, the default
logger has to be configured before using it.
```go
import "log/slog"
func main() {
slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil)))
}
```
### Development
#### Build
Before the library can be used, modified or if tests should be executed it's required to generate protobuf files
from the schema definition.
##### Generate Protobuf files
```bash
# Create the gen/java directory manually or rerun the buf generate command
mkdir -p gen/java
cd proto
# --include-imports is required for python code generation
buf generate --include-imports
```
##### Build library
```bash
go mod download && go mod tidy && go get ./... && go fmt ./... && go vet ./... && golangci-lint run && go build ./... && go test ./...
```
#### Register buf validation schema in IntelliJ / Goland
The schema files use `buf` protobuf extensions for validation of constraints.
To register the schema in IntelliJ / Goland clone the repo and add the import path:
```bash
git clone https://github.com/bufbuild/protovalidate.git
```
IntelliJ/Goland > Settings > Languages & Frameworks > Protocol Buffers > Import Paths > + (Add Path) > …/protovalidate/proto/protovalidate
#### Testcontainers
To run the tests **Docker** is needed as [Testcontainers](https://testcontainers.com/)
is used to run integration tests using a solace docker container.
#### Additional API implementations
There's already an implementation draft of the api for the new dynamically routing There's already an implementation draft of the api for the new dynamically routing
audit log solution. As the implementation of the system has not officially been audit log solution. As the implementation of the system has not officially been
started yet, it's only a draft with integration tests. started yet, it's only a draft with integration tests.
@ -179,25 +31,52 @@ is ready to be used.
The code can be found in the [api_routable.go](./api_routable.go) and The code can be found in the [api_routable.go](./api_routable.go) and
[api_routable_test.go](./api_routable_test.go) files. [api_routable_test.go](./api_routable_test.go) files.
### Java and Python ### Development
The repo currently contains additional configuration for Java and Python code generation.
For Python there's an example file showcasing the instantiation and validation, #### Schema Generation
that needs to be copied to the `gen/python` directory after generating the python-files
from the `.proto` files.
For Python, it's additionally required to install `protovalidate` on the system as
described [here](https://github.com/bufbuild/protovalidate-python) or by
running `pip3 install -r requirements.txt`.
In the future related code and configurations for both additional languages (Java and Go structs are generated from Protobuf schema by using [Buf](https://buf.build).
Python) will be extracted into separate repositories. The schema generator also generates code to validate constraints specified
in the schema.
### Open Issues It may be required to install required plugins manually by running:
- Check TODOs in the code
- Finalizing messaging schema ```shell
- Check if fields and constraints are correct and compatible with expected use cases go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
- Check that routing api specific parameters are independent of AuditEventLog go install github.com/envoyproxy/protoc-gen-validate@v1.1.0
- Clarify if `client.go` file can be used for licence / legal reasons ```
- Extraction of python / java configurations and code
- Clean up repo (delete main.go, etc. files) Please check that the version matches the version in the *go.mod* file.
- Update dependencies
Then the schema can be generated:
```bash
cd proto
buf generate
```
#### Build
The library can be built by executing the following commands:
```bash
go mod download && go mod tidy && go get ./... && go fmt ./... && go vet ./... && golangci-lint run && go build ./... && go test ./...
```
##### Testcontainers
To run the tests **Docker** is needed as [Testcontainers](https://testcontainers.com/)
is used to run integration tests using a solace docker container.
#### Register buf validation schema in IntelliJ / Goland
The schema files use `Buf` protobuf extensions for validation of constraints.
To register the schema in IntelliJ / Goland clone the repo and add the import path:
```bash
git clone https://github.com/bufbuild/protovalidate.git
```
IntelliJ/Goland > Settings > Languages & Frameworks > Protocol Buffers > Import Paths > +
(Add Path) > …/protovalidate/proto/protovalidate