schwarzit.stackit-public/audit-go
1
0
Fork 0
mirror of https://dev.azure.com/schwarzit/schwarzit.stackit-public/_git/audit-go synced 2026-02-07 16:47:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update README.md

Browse source
This commit is contained in:
Christian Schaible 2024-08-29 12:59:46 +02:00
parent 0a8516f4a3
commit eb22259346
1 changed files with 47 additions and 168 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

215
README.md
View file

@ -1,113 +1,18 @@
## Audit Log API
The audit log api is a library that simplifies sending audit logs to the audit log system.
## audit-go
### Logging API
The audit-go library is the core library for validation and sending of audit events.
The audit api provides two ways to log audit events:
### API Documentation
- **Direct logging**
Logs are directly validated, serialized and sent via solace.
It's the responsibility of the sending service to handle errors and to retry in case of an error.
```
Log(
ctx context.Context,
event *auditV1.AuditEvent,
visibility auditV1.Visibility,
routingIdentifier *RoutingIdentifier,
objectIdentifier *auditV1.ObjectIdentifier,
) error
```
The api documentation can be found [here]().
- **Transactional safe logging (transactional outbox pattern)**
Functionality is split into two steps:
- Serialization + Validation
The `ValidateAndSerialize` method can be called to validate and serialize the event data.
The serialized data can be stored in an outbox table / collection in
the database.
```
ValidateAndSerialize(
event *auditV1.AuditEvent,
visibility auditV1.Visibility,
routingIdentifier *RoutingIdentifier,
objectIdentifier *auditV1.ObjectIdentifier,
) (SerializedPayload, error)
```
- Sending data to solace
The `Send` method can be used to send the previously serialized data.
It is the responsibility of the sending service to ensure that the
method is called (repetitive) until no error is reported.
```
Send(
ctx context.Context,
routingIdentifier *RoutingIdentifier,
serializedPayload *SerializedPayload,
) error
```
TODO update the link
### Usage
1. Create Solace client / acl rules
The topic names have to be compliant to the format as specified
[here](https://async-api.stackit.schwarz/core-platform/asyncapi.html#operation-publish-stackit-platform/t/swz/audit-log/{region}/{version}/{eventSource}/{additionalParts}).
A solace client has to be provisioned manually with terraform (for
[dev](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/00_dev/essolske01/stackit/client_usernames.tf) /
[qa](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/02_qs/qssolske01/stackit/client_usernames.tf) /
[prod](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/03_prod/pssolske01/stackit/client_usernames.tf)).
The permission to write to the topic needs to be specified in the terraform configuration (for
[dev](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/00_dev/essolske01/stackit/acl_profiles.tf) /
[qa](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/02_qs/qssolske01/stackit/acl_profiles.tf) /
[prod](https://dev.azure.com/schwarzit/schwarzit.esb/_git/QaaS-Terraform?path=/03_prod/pssolske01/stackit/acl_profiles.tf)).
### Supported data types for routing
2. Instantiation of the messaging API
The audit log api uses solace as messaging system. The actual implementation for the
connection to the messaging system is implemented by specifying an abstraction and providing a
concrete implementation for AMQP/Solace.
```go
messagingApi, err := NewAmqpMessagingApi(AmqpConfig{URL: "...", User: "", Password: ""})
if err != nil { ... }
```
The following data types are currently supported for routing.
There are `SingularType` and `PluralType` representations in the code for them.
3. Instantiation of the audit log api
There will be a new Audit API in the future as audit logs will be routed dynamically
to different data sinks in the future.
The plan though is to keep the actual methods that (validates, serializes and) sends the log statements as stable
as possible only replacing the API instantiation part later.
The instantiation of the legacy api is shown below:
```
validator, err := protovalidate.New()
if err != nil { ... }
auditApi, err := NewLegacyAuditApi(
messagingApi,
LegacyTopicNameConfig{TopicName: topicName},
validator,
)
```
For each Solace topic data should be sent to, a new object instance of the legacy API is needed.
The `messagingAPI` object can be shared across the audit api object instances.
4. Logging audit events
As described in [Logging API](#logging-api) messages can be validated, serialized and sent
through the API.
It is important to retry as long as errors are returned to ensure that the message is really sent.
### Examples
The test code can be taken as an inspiration how to use the API.
- [api_legacy_test.go](./api_legacy_test.go)
### Tests
For users of the API who integrate the library there is an additional implementation available for
testing purpose. The `MockAuditApi` does not require a connection to a messaging system (therefore,
does not send any data via messaging). It only serializes and validates data.
It can be instantiated as follows:
```
auditApi, err := NewMockAuditApi()
if err != nil { ... }
```
### Supported data types
| Singular-Type | Plural-Type | Routable to customer | Description |
|---------------|---------------|----------------------|----------------------|
| system | system | no | The STACKIT system |
@ -115,89 +20,63 @@ if err != nil { ... }
| organization | organizations | yes | STACKIT organization |
| folder | folders | yes | STACKIT folder |
### Customization
### Additional API implementations
#### os.Stdout logging
There's already an implementation draft of the api for the new dynamically routing
audit log solution. As the implementation of the system has not officially been
started yet, it's only a draft with integration tests.
The API code is private to not confuse users or loose data until the new system
is ready to be used.
The audit log api uses `slog` to print messages on `os.Stdout` as it is part
of the standard library (Go >= 1.21).
To configure slog to print json instead of plain text, the default
logger has to be configured before using it.
```go
import "log/slog"
func main() {
slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil)))
}
```
The code can be found in the [api_routable.go](./api_routable.go) and
[api_routable_test.go](./api_routable_test.go) files.
### Development
#### Build
Before the library can be used, modified or if tests should be executed it's required to generate protobuf files
from the schema definition.
#### Schema Generation
##### Generate Protobuf files
Go structs are generated from Protobuf schema by using [Buf](https://buf.build).
The schema generator also generates code to validate constraints specified
in the schema.
```bash
# Create the gen/java directory manually or rerun the buf generate command
mkdir -p gen/java
It may be required to install required plugins manually by running:
cd proto
# --include-imports is required for python code generation
buf generate --include-imports
```shell
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
go install github.com/envoyproxy/protoc-gen-validate@v1.1.0
```
##### Build library
Please check that the version matches the version in the *go.mod* file.
Then the schema can be generated:
```bash
cd proto
buf generate
```
#### Build
The library can be built by executing the following commands:
```bash
go mod download && go mod tidy && go get ./... && go fmt ./... && go vet ./... && golangci-lint run && go build ./... && go test ./...
```
##### Testcontainers
To run the tests **Docker** is needed as [Testcontainers](https://testcontainers.com/)
is used to run integration tests using a solace docker container.
#### Register buf validation schema in IntelliJ / Goland
The schema files use `buf` protobuf extensions for validation of constraints.
The schema files use `Buf` protobuf extensions for validation of constraints.
To register the schema in IntelliJ / Goland clone the repo and add the import path:
```bash
git clone https://github.com/bufbuild/protovalidate.git
```
IntelliJ/Goland > Settings > Languages & Frameworks > Protocol Buffers > Import Paths > + (Add Path) > …/protovalidate/proto/protovalidate
#### Testcontainers
To run the tests **Docker** is needed as [Testcontainers](https://testcontainers.com/)
is used to run integration tests using a solace docker container.
#### Additional API implementations
There's already an implementation draft of the api for the new dynamically routing
audit log solution. As the implementation of the system has not officially been
started yet, it's only a draft with integration tests.
The API code is private to not confuse users or loose data until the new system
is ready to be used.
The code can be found in the [api_routable.go](./api_routable.go) and
[api_routable_test.go](./api_routable_test.go) files.
### Java and Python
The repo currently contains additional configuration for Java and Python code generation.
For Python there's an example file showcasing the instantiation and validation,
that needs to be copied to the `gen/python` directory after generating the python-files
from the `.proto` files.
For Python, it's additionally required to install `protovalidate` on the system as
described [here](https://github.com/bufbuild/protovalidate-python) or by
running `pip3 install -r requirements.txt`.
In the future related code and configurations for both additional languages (Java and
Python) will be extracted into separate repositories.
### Open Issues
- Check TODOs in the code
- Finalizing messaging schema
- Check if fields and constraints are correct and compatible with expected use cases
- Check that routing api specific parameters are independent of AuditEventLog
- Clarify if `client.go` file can be used for licence / legal reasons
- Extraction of python / java configurations and code
- Clean up repo (delete main.go, etc. files)
- Update dependencies
IntelliJ/Goland > Settings > Languages & Frameworks > Protocol Buffers > Import Paths > +
(Add Path) > …/protovalidate/proto/protovalidate