Error Handling

Overview

Errors can occur at two points in the system: during upstream sync and during client requests. Each is handled differently.

Upstream Sync Errors

Upstream API Failures

If any of the three upstream API calls fail (non-2xx status code), HttpRequestException is thrown and the entire sync aborts.

What happens:

• No version increment occurs.
• No data is written to Postgres.
• The exception propagates to ASP.NET Core's default exception handler, which returns 500 Internal Server Error. In Development mode, the response includes a developer exception page (HTML). In Production mode, the response body is empty.
• Existing data in Postgres remains untouched — clients continue to receive the last successfully synced data.

Recovery: The CronJob runs with backoffLimit: 0, so a failed job is not retried — it is marked as failed and can be investigated via kubectl get jobs. The next scheduled run (03:00) recovers the data since the sync is idempotent. For immediate recovery, trigger a sync manually via POST /api/internal/sync or by running the image with --sync.

Partial Sync Failures

If upstream fetches succeed but a Postgres write fails mid-sync:

• The version counter has already been incremented.
• Some raw FHIR or output records may have been upserted with the new version number.
• Soft-deletes may not have been applied.
• The SyncHistoryEntry for this run will not have been written, so the run is missing from SyncHistory.

Impact: Clients using ?since-version=N may see some but not all changes from this sync run.

Recovery: Trigger a new sync. The next sync is idempotent — content-hash change detection means already-correct records are left alone, while inconsistent records are reconciled. No manual intervention is needed.

Timeout

The sync (HTTP endpoint or --sync) has no explicit timeout configured. The CronJob should set an activeDeadlineSeconds to prevent runaway syncs:

jobTemplate:
  spec:
    activeDeadlineSeconds: 300  # Kill after 5 minutes

Client Request Errors

Authentication Errors

The error message is intentionally identical for both cases to avoid leaking information about valid keys.

Rate Limiting

Limits are enforced per API key per replica using an in-memory sliding window (RateLimiting config section, overridable via RateLimiting__* env vars; RateLimiting__Enabled=false disables limiting entirely). Requests without a valid API key share a single anonymous bucket, so the number of limiter partitions stays bounded. The health and Swagger endpoints are not rate limited. Rejections are logged at Warning level with the source IP only — API keys are never logged.

Invalid since-version

Postgres Connection Failures

If Postgres is unreachable when a client makes a request:

• The controller throws an unhandled NpgsqlException.
• ASP.NET Core returns 500 Internal Server Error.
• The error is logged at Error level.

Monitoring: Watch for 500 responses on the output endpoints. Postgres connection issues typically resolve once the database is reachable again — no data loss occurs.

Logging

All log output goes to stdout (Serilog console). In test/prod each event is a single compact JSON object (one line), so fields are directly searchable in Splunk — NHN k8s ships pod stdout to Splunk via Fluent Bit (default index kube_MILJØ, e.g. kube_prod). In Development the console uses a readable text template instead.

Every event is enriched with Service (nompd-api), Version (the APP_VERSION environment variable, falling back to the assembly version), MachineName, ThreadId, and — for request-scoped events — RequestId (the correlation ID).

Request Logging

• Every non-health request is logged with method, path, status code, and duration. /api/internal/health is excluded (Kubernetes probes).
• v1 requests are logged as {Method} {Path} {StatusCode} in {Elapsed}ms.
• v2 requests additionally include the query string, the client IP, and the consumer — the client_id claim on the HelseID access token. The query string (e.g. since-version=N) reveals how consumers poll for updated data.

Correlation

• Every response includes an X-Correlation-Id header. Clients may send their own X-Correlation-Id request header; it is reused, otherwise a new ID is generated.
• The ID is available as the RequestId property on all log events for that request, and in the correlationId field of 500 error bodies.

Key Log Messages

# Startup (Program)
"Starting nompd-api: Environment={Environment}, Version={Version}, UpstreamBaseUrl={UpstreamBaseUrl}, Postgres={Postgres}"

# Sync lifecycle (SyncService)
"Starting sync"
"Sync completed in {Duration}ms: {Added} added, {Updated} updated, {Unchanged} unchanged, {Deleted} deleted (version {Version})"

# Upstream fetch (UpstreamApiClient)
"Fetched {Count} items from {Path}"

For per-collection detail (including the business keys of added/updated/deleted records), query the SyncHistory collection — every sync run writes one entry there.

Structured Logging

All log messages use structured logging with named parameters, making them field-searchable in Splunk (e.g. Properties.consumer:"my-client" or Properties.StatusCode:500).

Monitoring Recommendations

Failure Modes Summary