Audit logging

Open Forms has audit logging. The goals of the audit logging setup are:

  • (functional) administrators can reconstruct what happened in and to a form submission

  • GDPR (Dutch: AVG) questions like “who looked at this sensitive data” can be answered

  • logs can help in debugging issues

  • being easy to use for developers

Architecture

Audit logs are produced by using the openforms_audit logger. Logs can be produced with any level ranging from DEBUG to CRITICAL - none will be dropped because of their log level being too low.

Audit logs are routed to two destinations:

  • the standard log output destination, where non-audit-logs also go. Normally this is stdout/stderr, but configuration allows for writing to log files as an alternative.

  • the database, in particular the generic log entry table from django-timeline-logger.

Logs that are routed to the database get processed by our adapter openforms.logging.adapter.from_structlog(), which takes the event dict produced by our logger and builds the necessary context to save into the openforms.logging.models.TimelineLogProxy model, such as:

  • the event that triggered the log and associated display template

  • the user that performed the action that was logged

  • the object on which the action was performed

  • additional metadata about the action, typically accessed in the event-specific display template

The log handler from django-timeline-logger does this in a way that’s both performant and robust - audit logs will still be written to the database even if the DB transaction in the main thread rolls back.

Additionally, the stdout/stderr destination also receives all log events and is even more reliable. DevOps roles should set up log aggregation with tools like Loki and Grafana (see also: observability (logging)). When conflicting information originates from logging to both destinations, you should favour the stdout/stderr logs.

Usage and conventions

Developers can start audit logging right away with a couple of lines of code, for example:

 1# views.py
 2
 3from openforms.logging import audit_logger
 4
 5def my_view(request: HttpRequest):
 6    ...
 7    audit_logger.info("my_view_accessed", user=request.user.username)
 8    ...
 9    return HttpResponse(...)
10
11# src/openforms/logging/adapter.py
12
13def from_structlog(event_dict: EventDict) -> EventDetails:
14    from openforms.accounts.models import User
15
16    ...
17
18    match event_dict:
19        ...
20
21        case {
22            "event": "my_view_accessed" as event,
23            "user": str(username),
24        }:
25            user = User.objects.get(username=username)
26            return EventDetails(
27                event=event,
28                instance=user,
29                user=user,
30                tags=[TimelineLogTags.avg],
31                extra_data={"context": "docs-example"},
32            )
33
34        ...

Note that:

  • audit_logger can be called directly with the event and additional context

  • you can use any log level that’s best suited for the situation being logged

  • any additional keyword arguments like user are passed in the event dict, and you can/should match against them

Conventions

  • At the time of writing, we opt to only pass primitives like strings, numbers, booleans… as additional context in logging calls. When the logs are formatted for the console output, the string representation is taken of each member, and for model instances or even UUID instances, that leads to hard-to-scan/read output.

    Note

    This could be addressed in the future with a custom structlog processor.

  • When referring to a submission, use the submission_uuid=str(submission.uuid) kwarg.

  • When referring to a Django user, use the username=user.username kwarg. If this is ambiguous (see the hijack audit logs), use the username as value, but pick a more appropriate keyword argument name.

  • If you’re unsure whether a convention exists or not, scan openforms.logging.adapter.from_structlog() for pre-existing patterns.

When to audit log?

  • Important steps/phases in the lifecycle of a submission. Make sure to log start, (possible) skip, done and error events. Submission audit logs are displayed in the admin page when viewing a submission - functional administrators use this as a debug tool.

  • Anything that can be privacy-sensitive. Make sure to log who accessed information from where or what, and if known, what the reason was.

See also

Logging documentation.