Audit Log API

The audit log api is a library that simplifies sending audit logs to the audit log system.

Logging API

The audit api provides two ways to log audit events:

• Direct logging
  Logs are directly validated, serialized and sent via solace.
  It's the responsibility of the sending service to handle errors and to retry in case of an error.

  Log(
    ctx context.Context,
    event *auditV1.AuditEvent,
    visibility auditV1.Visibility,
    routingIdentifier *RoutingIdentifier,
    objectIdentifier *auditV1.ObjectIdentifier,
  ) error
  

• Transactional safe logging (transactional outbox pattern)
  Functionality is split into two steps:

  • Serialization + Validation
    The ValidateAndSerialize method can be called to validate and serialize the event data. The serialized data can be stored in an outbox table / collection in the database.

    ValidateAndSerialize(
      event *auditV1.AuditEvent, 
      visibility auditV1.Visibility, 
      routingIdentifier *RoutingIdentifier, 
      objectIdentifier *auditV1.ObjectIdentifier,
    ) (SerializedPayload, error)
    

  • Sending data to solace
    The Send method can be used to send the previously serialized data. It is the responsibility of the sending service to ensure that the method is called (repetitive) until no error is reported.

    Send(
      ctx context.Context, 
      routingIdentifier *RoutingIdentifier, 
      serializedPayload *SerializedPayload,
    ) error
    

Usage

1. Create Solace client / acl rules The topic names have to be compliant to the format as specified here. A solace client has to be provisioned manually with terraform (for dev / qa / prod).
   The permission to write to the topic needs to be specified in the terraform configuration (for dev / qa / prod).

2. Instantiation of the messaging API
   The audit log api uses solace as messaging system. The actual implementation for the connection to the messaging system is implemented by specifying an abstraction and providing a concrete implementation for AMQP/Solace.

   messagingApi, err := NewAmqpMessagingApi(AmqpConfig{URL: "...", User: "", Password: ""})
   if err != nil { ... }
   

3. Instantiation of the audit log api
   There will be a new Audit API in the future as audit logs will be routed dynamically to different data sinks in the future.
   The plan though is to keep the actual methods that (validates, serializes and) sends the log statements as stable as possible only replacing the API instantiation part later. The instantiation of the legacy api is shown below:

   validator, err := protovalidate.New()
   if err != nil { ... }
   
   auditApi, err := NewLegacyAuditApi(
       messagingApi, 
       LegacyTopicNameConfig{TopicName: topicName}, 
       validator,
   )
   

   For each Solace topic data should be sent to, a new object instance of the legacy API is needed. The messagingAPI object can be shared across the audit api object instances.

4. Logging audit events
   As described in Logging API messages can be validated, serialized and sent through the API. It is important to retry as long as errors are returned to ensure that the message is really sent.

Examples

The test code can be taken as an inspiration how to use the API.

• api_legacy_test.go

Tests

For users of the API who integrate the library there is an additional implementation available for testing purpose. The MockAuditApi does not require a connection to a messaging system (therefore, does not send any data via messaging). It only serializes and validates data. It can be instantiated as follows:

auditApi, err := NewMockAuditApi()
if err != nil { ... }

Customization

os.Stdout logging

The audit log api uses slog to print messages on os.Stdout as it is part of the standard library (Go >= 1.21).

To configure slog to print json instead of plain text, the default logger has to be configured before using it.

import "log/slog"

func main() {
    slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil)))
}

Development

Build

Before the library can be used, modified or if tests should be executed it's required to generate protobuf files from the schema definition.

Generate Protobuf files

# Create the gen/java directory manually or rerun the buf generate command
mkdir -p gen/java

cd proto
# --include-imports is required for python code generation
buf generate --include-imports

Build library

go mod download && go mod tidy && go get ./... && go fmt ./... && go vet ./... && go build ./... && go test ./...

Register buf validation schema in IntelliJ / Goland

The schema files use buf protobuf extensions for validation of constraints.

To register the schema in IntelliJ / Goland clone the repo and add the import path:

git clone https://github.com/bufbuild/protovalidate.git

IntelliJ/Goland > Settings > Languages & Frameworks > Protocol Buffers > Import Paths > + (Add Path) > …/protovalidate/proto/protovalidate

Testcontainers

To run the tests Docker is needed as Testcontainers is used to run integration tests using a solace docker container.

Additional API implementations

There's already an implementation draft of the api for the new dynamically routing audit log solution. As the implementation of the system has not officially been started yet, it's only a draft with integration tests. The API code is private to not confuse users or loose data until the new system is ready to be used.

The code can be found in the api_routable.go and api_routable_test.go files.

Java and Python

The repo currently contains additional configuration for Java and Python code generation.

For Python there's an example file showcasing the instantiation and validation, that needs to be copied to the gen/python directory after generating the python-files from the .proto files. For Python, it's additionally required to install protovalidate on the system as described here or by running pip3 install -r requirements.txt.

In the future related code and configurations for both additional languages (Java and Python) will be extracted into separate repositories.

Open Issues

• Finalizing messaging schema
• Extraction of python / java configurations and code
• Clarify if client.go file can be used for licence / legal reasons
• Clean up repo (delete main.go, etc. files)
• Finalizing API Design
  • Decision whether serialized payload should be replaced with []byte or base64 encoded string