---
title: Logging & Monitoring - Amazon Bedrock API Observability
description: Production-grade observability for stdapi.ai with CloudWatch, OpenTelemetry.
  Track API performance, monitor costs, debug issues, and ensure compliance.
keywords: AWS CloudWatch logs, API monitoring, OpenTelemetry, API observability, cost
  monitoring AWS, performance tracking, compliance logging, validation error log,
  rejected request body not logged, error_detail field path, LOG_REQUEST_PARAMS 400
---

# :material-chart-line: Logging & Monitoring

stdapi.ai provides production-grade observability out of the box. Track request performance, debug issues, monitor costs, and ensure compliance with structured JSON logging and OpenTelemetry integration.

**Why this matters:**

- **Troubleshoot issues fast** - Structured logs with request IDs, timings, and error details
- **Monitor costs** - Track model usage, request volume, and performance by endpoint
- **Ensure compliance** - Full audit trail with client IPs, user IDs, and request/response logging
- **Performance optimization** - Identify slow requests, high-latency endpoints, bottlenecks
- **AWS integration** - Native CloudWatch Logs, X-Ray, and service-level metrics

stdapi.ai emits structured JSON logs for every request, stream, and background task, and integrates with OpenTelemetry (OTel) for traces and metrics. This guide shows how to enable observability, read the logs, and correlate signals across systems.

<div class="grid cards" markdown>

- :material-clipboard-text-outline: __Structured JSON Logging__
  <br>JSON logs to STDOUT—perfect for AWS CloudWatch Logs. One event per line with all request context.

- :material-identifier: __Request Correlation__
  <br>All events for a request share the same `id` (returned as `x-request-id`). Track full request lifecycle across logs and traces.

- :material-aws: __AWS-Native Integration__
  <br>Works with CloudWatch Logs, X-Ray, and service metrics. ECS auto-forwards STDOUT to CloudWatch.

- :material-graphql: __OpenTelemetry Tracing__
  <br>Enable `OTEL_ENABLED=true` to export spans to AWS X-Ray, Jaeger, Tempo, or any OTLP-compatible backend.

- :material-chart-line: __Performance Insights__
  <br>Track execution times, model usage, endpoint latency. Includes ready-to-use CloudWatch Logs Insights queries.

- :material-alert-decagram-outline: __Debug Mode__
  <br>Enable `LOG_REQUEST_PARAMS=true` for full request/response logging when troubleshooting issues.

</div>

---

## :material-rocket-launch: Quick Start

Set these environment variables, then restart the service (see the [Configuration Guide](operations_configuration.md) for details):

```bash
# Set minimum log level (optional, defaults to "info")
# Options: info, warning, error, critical, disabled
export LOG_LEVEL=warning

# Enable OpenTelemetry tracing
export OTEL_ENABLED=true
export OTEL_SERVICE_NAME=stdapi
# 0.0–1.0 (10% example)
export OTEL_SAMPLE_RATE=0.1

# Include request/response payloads in logs (for debugging ONLY)
export LOG_REQUEST_PARAMS=true

# Log client IP addresses (requires ENABLE_PROXY_HEADERS for real client IPs)
export LOG_CLIENT_IP=true
export ENABLE_PROXY_HEADERS=true  # When behind ALB/CloudFront
```

!!! warning "Sensitive data and cost impact"
    Enabling `LOG_REQUEST_PARAMS` may expose sensitive content in logs. Use only in development or during targeted troubleshooting. Redact secrets before sharing logs externally.

    Additionally, logging full request/response payloads can dramatically increase log volume and costs, especially for large LLM prompts, tool calls, and generated outputs. In AWS CloudWatch Logs, ingestion and storage costs scale with log size. Prefer short retention, targeted sampling, and temporary enablement only when needed.

!!! info "Client IP Logging"
    When `LOG_CLIENT_IP=true`:

    - The `client_ip` field is added to request logs
    - The client IP is added as `client.address` attribute to OpenTelemetry spans (when `OTEL_ENABLED=true`)

    To log the real client IP address (instead of the proxy IP), also enable `ENABLE_PROXY_HEADERS=true` when running behind AWS ALB, CloudFront, or other reverse proxies. See the [Configuration Guide](operations_configuration_observability.md#client-ip-logging) for details.

!!! tip "CloudWatch best practice"
    JSON to STDOUT is optimal for CloudWatch Logs Insights. In AWS ECS, the task's log driver forwards container STDOUT to CloudWatch Logs automatically.

---

## :material-format-list-bulleted: Event Types

stdapi.ai emits five kinds of JSON events (one per line):

| Event            | Description                                                                         |
|:-----------------|:------------------------------------------------------------------------------------|
| `start`          | Emitted once at server startup. Includes startup metadata and warnings.             |
| `stop`           | Emitted on graceful shutdown. Includes uptime.                                      |
| `request`        | One per HTTP request. Method, path, status, timings, and optional request/response. |
| `request_stream` | Streaming segments (SSE/audio). Indicates streaming activity and duration.          |
| `background`     | Background tasks correlated to the parent request.                                  |

---

## :material-table-column: Common Fields

Each event shares core fields and may add type‑specific ones.

|                                     Field | Applies to                          | Description                                                                                 |
|------------------------------------------:|:------------------------------------|:--------------------------------------------------------------------------------------------|
|                                    `type` | all                                 | One of `start`, `stop`, `request`, `request_stream`, `background`                           |
|                                   `level` | all                                 | `info`, `warning`, `error`, `critical` (controlled by `LOG_LEVEL`)                          |
|                                    `date` | all                                 | RFC3339, timezone‑aware timestamp                                                           |
|                               `server_id` | all                                 | Instance identifier — on ECS: `task_id-container_name-uuid`; elsewhere: `hostname-pid-uuid` |
|                          `server_version` | all                                 | Application version string                                                                  |
|                            `error_detail` | all                                 | Optional list of formatted exception strings                                                |
|                                      `id` | request, request_stream, background | Correlation ID (also returned as `x-request-id`)                                            |
|                       `execution_time_ms` | request, request_stream, background | Duration of the handled block                                                               |
|                                  `method` | request                             | HTTP method                                                                                 |
|                                    `path` | request                             | Request path                                                                                |
|                             `status_code` | request                             | Final HTTP status code                                                                      |
|                               `client_ip` | request                             | Client IP address (if `LOG_CLIENT_IP=true`)                                                 |
|                       `client_user_agent` | request                             | When provided by client                                                                     |
|                                `model_id` | request                             | Targeted model (if applicable)                                                              |
|                                `voice_id` | request                             | TTS voice (if applicable)                                                                   |
|                           `model_regions` | request                             | AWS region(s) that handled the request; may contain multiple values when failover occurred  |
|       `request_user_id`, `request_org_id` | request                             | Propagated identifiers (if applicable)                                                      |
|                  `aws_role_session_name` | request                             | Session the request's model usage was billed under, with [per-user cost attribution](operations_cost_management.md#per-user-attribution) enabled; the same value AWS reports as the caller identity |
|                     `aws_tenant_key_id` | request                             | Key ID of the tenant whose own AWS account the model invocations were billed to, with [tenant AWS credentials](operations_authentication_security.md#tenant-aws-credentials); pair it with the `billed_to` usage field to reconcile against that tenant's bill |
|                          `request_params` | request                             | Sanitized request payload (if `LOG_REQUEST_PARAMS=true`)                                    |
|                        `request_response` | request                             | Sanitized response payload (if `LOG_REQUEST_PARAMS=true`)                                   |
| `amzn_trace_id`, `apigw_request_id`, `cloudfront_request_id` | request          | Edge correlation IDs copied from the `X-Amzn-Trace-Id`, `x-amz-apigw-id` and `X-Amz-Cf-Id` request headers (when present) |
|                            `aws_requests` | request, request_stream, background | AWS-side request IDs of downstream AWS API calls (see [AWS Service Correlation Metadata](#aws-service-correlation-metadata)) |
|                                   `event` | background                          | Background operation name                                                                   |
| `server_start_time_ms`, `server_warnings` | start                               | Startup metrics and warnings                                                                |
|                       `region_latencies` | start                               | `latency_ms` and `stddev_ms` per Bedrock region, lowest first — the startup probe the [`lowest_latency`](operations_resilience.md#lowest-latency) routing strategy ordered the regions by. Present only under that strategy |
|                        `server_uptime_ms` | stop                                | Uptime at shutdown                                                                          |
|             `abandoned_background_tasks` | stop                                | Background tasks cancelled unfinished at shutdown, counted per kind; present only when some were, and the event is then `warning` (see [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration_server.md#shutdown-drain-timeout)) |

!!! note "Understanding warnings and errors"
    - For `request` events, default log levels are derived from the final HTTP status: 4xx → `warning`, 5xx → `error`. Unexpected server crashes (like HTTP 500) may appear as `critical`.
    - Authentication/authorization: For security, client responses for `401` and `403` include only generic messages. Full diagnostic details are captured in server logs under `error_detail` and can be correlated via `id` (see `x-request-id`).
    - `server_warnings` (on the `start` event) often highlights missing configuration and features that have been disabled as a result (for example, no S3 bucket configured disables certain image/audio features).
    - `error_detail` (on any event) contains formatted exception traces and diagnostic hints, which frequently point to missing configuration, unavailable dependencies, or disabled features.

---

## :material-cash-multiple: Usage Metrics Fields

Usage is reported as a nested `usage` list on `request` / `request_stream` events. Each entry represents billed quantities for a specific **(service, model, operation, region, tier, routing, context)** combination. Zero/empty fields are omitted.

### Entry Structure

| Field                       | Type | Description                                                                                            |
|:----------------------------|:-----|:-------------------------------------------------------------------------------------------------------|
| `service`                   | str | AWS service/API: `bedrock-runtime`, `polly`, `transcribe`, `translate`, `comprehend`                  |
| `model`                     | str | Model identifier (e.g., `amazon.titan-embed-text-v2:0`, `amazon.transcribe`)                                 |
| `operation`                 | str | Request route path (e.g., `/v1/chat/completions`, `/v1/embeddings`, `/v1/audio/speech`)              |
| `region`                    | str | AWS region that served the request (part of the aggregation key)                                       |
| `tier`                      | str | Service tier that actually served the call (AWS-reported when available, else as requested): `standard`, `flex`, `priority`, `batch` |
| `routing`                   | str | Serving profile, present as `"global"` (cross-region global routing) or `"latency"` (latency-optimized) |
| `context`                   | str | Present as `"long"` only when the call's prompt (input + cache read/write tokens) exceeded the boundary that model publishes — 200K tokens for most models, 272K for the OpenAI frontier models (GPT-5.4 and later); billed at the [long-context rate](operations_cost_management.md#long-context-pricing) where AWS publishes one |
| `billed_to`                 | str | Present as `"tenant"` only when AWS billed another account for the call — a tenant that registered [its own AWS credential](operations_authentication_security.md#tenant-aws-credentials). Such entries carry quantities but **no `cost`**, and are excluded from the request total and the `Cost` metric |
| `cost`                      | str | Full-precision cost as plain-decimal text (no exponent, no trailing zeros, e.g. `"0.000015"`)          |
| `currency`                  | str | ISO currency code for `cost`                                                                            |
| `costs`                     | dict| Per-currency full-precision cost text, replacing `cost`/`currency` when dimensions span multiple currencies |
| `requests`                  | int | Number of billed backend invocations aggregated into this entry                                        |
| `input_tokens`              | int | Real input token count (Converse, InvokeModel, embeddings)                                            |
| `input_tokens_by_spec`      | dict| Input tokens keyed by modality bucket (e.g., `{"speech": 1200}`), for models AWS charges per modality  |
| `output_tokens`             | int | Real output token count (Converse, InvokeModel)                                                        |
| `output_tokens_by_spec`     | dict| Output tokens keyed by modality bucket (e.g., `{"speech": 800}`), for models AWS charges per modality  |
| `total_tokens`              | int | Real total tokens (Converse provides this directly)                                                    |
| `cached_tokens`             | int | Real cached read tokens from Converse prompt caching                                                   |
| `cache_write_tokens`        | int | Real cache write tokens from Converse prompt caching                                                   |
| `cache_write_tokens_by_ttl` | dict| Cache write tokens by TTL, from Converse `cacheDetails` (e.g., `{"1h": 700, "5m": 100}`)            |
| `output_images`             | int | Real output image count from image generation models                                                   |
| `output_images_by_spec`     | dict| Output image count keyed by `"<resolution>:<quality>"`, for models priced per resolution/quality       |
| `output_seconds`            | int | Billed output media duration in seconds (video generation, recorded at job submission)                |
| `output_seconds_by_spec`    | dict| Output seconds keyed by rate bucket (e.g., `{"hd": 5}`), for models priced per resolution              |
| `input_seconds`             | int | Real input media duration in seconds (Transcribe audio, rounded up to the second; audio/video embedding inputs) |
| `input_seconds_by_spec`     | dict| Input seconds keyed by modality (e.g., `{"audio": 42}`), for models priced per media type; `{"streaming": 12}` for streamed transcription, priced at its own rate |
| `input_images`              | int | Input image count (multimodal embeddings, billed per image)                                            |
| `input_images_by_spec`      | dict| Input image count keyed by rate bucket (e.g., `{"document": 1}`)                                       |
| `input_characters`          | int | Real input character count (Polly `RequestCharacters`, Translate source text)                         |
| `comprehend_units`          | int | Real Comprehend units (100-char units, 3-unit minimum per call)                                       |
| `grounding_requests`        | int | Built-in grounding tool invocations (e.g., Amazon Nova Grounding `web_search`, billed per request)     |
| `search_units`              | int | Rerank search units (one per rerank query)                                                             |
| `text_units`                | int | Bedrock Guardrails text units (1,000 characters each), as reported by ApplyGuardrail, per applied policy |

!!! warning "Guardrail cost coverage"
    Guardrail units are recorded only on routes that enforce the guardrail through the ApplyGuardrail API, which reports the units each policy consumed. Chat routes use Bedrock's native guardrail integration, whose responses carry no unit counts, so their guardrail cost is absent from these entries even though AWS charges it — see [route coverage](operations_configuration_bedrock.md#route-coverage).

!!! note "Usage field placement"
    - **Non-streaming requests**: Usage appears on the `request` event
    - **Streaming requests**: Usage appears on the `request_stream` event (not `request`) — this includes streaming speech-to-text
    - **Polly / Translate**: Always on `request` event
    - Cost dashboards must aggregate **both** entry types by request `id`

!!! note "Zero values are omitted"
    Fields are only populated when non-zero. Query for present fields rather than filtering by `> 0`. Missing fields = zero.

!!! note "Client disconnect behavior"
    - **Chat (Converse) streams**: if the client disconnects before the trailing `metadata.usage` event arrives, the wrapper keeps draining the already-open Bedrock stream (bounded) looking for it, so the usage already billed by AWS is still recorded.
    - **InvokeModel streams and image generation jobs**: if a stream/job is abandoned before its final `invocationMetrics` chunk (or job completion) arrives, no usage is produced and logged.

!!! note "No estimates"
    All values are real AWS-billed quantities. No estimation is performed — if AWS doesn't return a count, the field is absent (zero).

---

## :material-chart-line: CloudWatch Metrics (EMF)

When `CLOUDWATCH_METRICS=true` is enabled, usage is emitted as CloudWatch Embedded Metric Format (EMF) lines to stdout. These are automatically extracted as CloudWatch Metrics on ECS with no extra IAM/API calls.

### Configuration

| Setting                          | Default  | Description                               |
|:----------------------------------|:---------|:-------------------------------------------|
| `CLOUDWATCH_METRICS`             | `false`  | Enable/disable EMF emission               |
| `CLOUDWATCH_METRICS_NAMESPACE`   | `stdapi` | CloudWatch namespace for the metrics      |

### EMF Line Structure

Each billed usage entry generates one EMF JSON line:

```json
{
  "_aws": {
    "Timestamp": 1748716800000,
    "CloudWatchMetrics": [{
      "Namespace": "stdapi",
      "Dimensions": [["Model"]],
      "Metrics": [
        {"Name": "Requests", "Unit": "Count"},
        {"Name": "InputTokens", "Unit": "Count"},
        {"Name": "OutputTokens", "Unit": "Count"}
      ]
    }]
  },
  "Model": "anthropic.claude-sonnet-5",
  "operation": "/v1/chat/completions",
  "Requests": 1,
  "InputTokens": 1500,
  "OutputTokens": 450
}
```

### Metric Details

- **Single dimension**: `Model` — provides low cardinality for cost control. A second `[Model, Currency]` dimension set is added for the `Cost` metric — see [Cost Management](operations_cost_management.md#cost-tracking-real-time-aws-pricing) for why cost needs its own dimension set.
- **`Requests`**: Emitted on every usage record — the number of billed backend invocations the record aggregates. It is what the usage API reports as `num_model_requests`.
- **`operation`**: Included as a queryable field (not a dimension)
- **Metric units**: `Count` for all token/image/character counts and for `Requests`, `Seconds` for audio duration

#### Dimension Sets Added by the Usage API { #usage-api-dimension-sets }

With [`USAGE_API`](operations_configuration_observability.md#usage-api) enabled, every metric above is published a second time under a `[Model, Operation]` dimension set, **alongside** the `Model` roll-up rather than in place of it — that per-endpoint breakdown is what the [organization usage endpoints](api_openai_organization_usage.md) read. Two more sets follow the caller:

- Where the request carried a tenant API key, the quantities are also published under `[Model, Operation, ApiKey]`, and `Cost` additionally under `[Model, Currency, ApiKey]`.
- With [`CLOUDWATCH_METRICS_USER_DIMENSION`](operations_configuration_observability.md#cloudwatch-metrics-user-dimension) enabled, the quantities are also published under `[Model, Operation, User]`.

`Operation` takes one of a fixed set of values, never a caller-supplied string:

`chat.completions` · `completions` · `responses` · `messages` · `realtime` · `embeddings` · `rerank` · `moderations` · `audio.speech` · `audio.transcriptions` · `audio.translations` · `images.generations` · `images.edits` · `images.variations` · `videos` · `vector_stores.search`

A route outside that set publishes no `Operation` dimension.

!!! warning "Each dimension set is a metric series of its own"
    Amazon CloudWatch stores — and charges for — every distinct combination of dimensions separately, so these sets are **additional custom metrics** on top of the `Model` roll-up, and reading them back is billed per metric as well. Budget for both before enabling the usage API: see [Usage API cost](operations_cost_management.md#usage-api-cost).

!!! tip "Querying EMF metrics"
    Use CloudWatch Logs Insights to search EMF lines:
    ```sql
    fields @timestamp, Model, InputTokens, OutputTokens
    | filter _aws.CloudWatchMetrics is not null
    | sort @timestamp desc
    | limit 20
    ```

!!! note "Performance"
    EMF lines bypass log-level filtering and write directly to stdout. This ensures metrics are always available on ECS where CloudWatch Agent scrapes stdout.

---

## :material-currency-usd: Cost Tracking and Attribution { #cost-tracking-real-time-aws-pricing }

Pricing each request from published AWS prices is covered on the [Cost Management](operations_cost_management.md) page: what `COST_TRACKING` (off by default) adds to the request log line and to the `Cost` EMF metric, where the price catalog comes from and how far an estimate can drift from your bill, the regional and multi-currency fallbacks, and how to attribute spend on the AWS side — by IAM principal or per end user. Go there to answer "what did this request cost?" rather than "what did this request do?".

[:octicons-arrow-right-24: Cost Management](operations_cost_management.md)

---

## :material-link-variant: Correlating Logs and Traces

- Group events by `id` to reconstruct a full request lifecycle (request → stream(s) → background).
- The `x-request-id` response header exposes the same value so external systems can propagate correlation.
- With OTel enabled, a root span named like `POST /v1/...` is created and carries attributes: `http.method`, `http.url`, `http.user_agent`, `request.id`, `server.id`, `http.status_code`, and `duration_ms`.

!!! note "Anthropic-compatible routes use a different header name"
    On `/anthropic/...` routes, the same correlation value is returned as `request-id` instead of `x-request-id`, following Anthropic's own convention.

!!! tip "Do and Don't for correlation"
    - Do propagate `x-request-id` across client → service → downstreams when possible.
    - Do use `request_stream` durations to account for total user‑perceived latency.
    - Don't generate your own request IDs for the same hop; prefer the provided one.

---

## :material-magnify: Reading the Logs

- High latency: Inspect `execution_time_ms` on the `request` event. If the response was streamed, also sum `request_stream` durations. Combine with OTel spans to locate downstream delays (model provider, S3, etc.).
- Errors: Look for `level=critical` and `error_detail` (formatted exceptions). With OTel, the span is marked error with attributes `error=true` and `error.message`.
- Rejected requests: A `400` from request validation lists the fields that failed and why, as `<field path>: <message>` entries in `error_detail` — the first 20 distinct faults, then a count of the rest. The values themselves are never logged, so a credential or an attachment a rejected body carries stays out of the log; the caller is answered in the wording of the API it called (see [Using stdapi.ai](api_overview.md#using-stdapiai)).
- Payload issues: Temporarily enable `LOG_REQUEST_PARAMS=true` to validate requests/responses, then disable. It covers requests that pass validation — a request rejected before it reaches its endpoint is reported by field path only.
- Client identification: `client_user_agent` and optional `request_user_id` / `request_org_id` help tie requests to users.
- Routing confirmation: `model_id` and `voice_id` confirm which provider/model/voice handled the request.

!!! warning "When to open a GitHub issue"
    If you encounter `level=critical` events, capture representative JSON log lines (redacting sensitive data) and open an issue on [GitHub](https://github.com/stdapi-ai/stdapi.ai/issues). Include information about the failing request to help reproduce the issue.

---

## :material-filter: Controlling Log Verbosity

The `LOG_LEVEL` environment variable controls which log events are written to STDOUT. Set it to filter out lower-severity events. For detailed configuration options, see the [Logging Level](operations_configuration_observability.md#logging-level) section in the Configuration Guide.

- **`info`** (default): All events are logged (info, warning, error, critical)
- **`warning`**: Only warnings and higher severity (warning, error, critical) - recommended for production
- **`error`**: Only errors and critical events
- **`critical`**: Only critical events
- **`disabled`**: No log output (not recommended)

```bash
# Production example: reduce log volume while maintaining visibility
export LOG_LEVEL=warning
```

!!! tip "Reducing CloudWatch Costs"
    In high-traffic production environments, setting `LOG_LEVEL=warning` or `LOG_LEVEL=error` can significantly reduce CloudWatch Logs ingestion and storage costs by filtering out routine `info`-level events. This is especially effective when combined with appropriate retention policies.

    Additionally, infrastructure routes are automatically excluded from logging to reduce noise: `/`, `/docs`, `/favicon.ico`, `/health`, `/openapi.json`, `/ping`, `/redoc`, `/robots.txt`, `/.well-known/api-catalog`, `/.well-known/mcp/server-card.json`, and `/.well-known/oauth-protected-resource`. The browser assets the documentation pages load, served under `/docs-assets/`, are excluded on the same grounds.

---

## :material-graphql: OpenTelemetry Integration

When `OTEL_ENABLED=true`:

- A span is created per request and for streaming/background blocks.
- Spans carry `request.id` and `server.id` for correlation.
- 4xx/5xx `status_code` marks the span with an error status.
- Sampling is controlled via `OTEL_SAMPLE_RATE`.

For exporters and advanced setup, rely on standard OTel environment variables supported by your exporter/backend.

!!! note "No X-Ray IAM permissions needed"
    Export is push-based OTLP/HTTP to the endpoint you configure — the application makes no direct AWS X-Ray API calls and needs no extra IAM permissions for tracing. To forward traces to AWS X-Ray, point the OTLP endpoint at an [ADOT collector](https://aws-otel.github.io/docs/introduction), which handles the X-Ray upload using its own credentials.

---

## :material-code-json: Example Events

__Example — Request with payload logging enabled__

```json
{
  "type": "request",
  "level": "info",
  "date": "2025-01-01T12:00:00Z",
  "server_id": "stdapi-1",
  "id": "a1b2c3d4",
  "method": "POST",
  "path": "/v1/chat/completions",
  "status_code": 200,
  "model_id": "anthropic.claude-sonnet-5",
  "execution_time_ms": 842,
  "request_params": {"messages": [{"role": "user", "content": "..."}]},
  "request_response": {"id": "cmpl_...", "choices": [...], "usage": {...}}
}
```

__Example — Streaming segment (SSE/audio)__

```json
{
  "type": "request_stream",
  "level": "info",
  "date": "2025-01-01T12:00:01Z",
  "server_id": "stdapi-1",
  "id": "a1b2c3d4",
  "execution_time_ms": 1234
}
```

__Example — Background work correlated to a request__

```json
{
  "type": "background",
  "level": "info",
  "date": "2025-01-01T12:00:02Z",
  "server_id": "stdapi-1",
  "id": "a1b2c3d4",
  "event": "image-upload-s3",
  "execution_time_ms": 97
}
```

__Example — Error with captured details__

```json
{
  "type": "request",
  "level": "critical",
  "date": "2025-01-01T12:00:05Z",
  "server_id": "stdapi-1",
  "id": "e9f0a1b2",
  "method": "POST",
  "path": "/v1/images/edits",
  "status_code": 500,
  "error_detail": ["Traceback (most recent call last): ..."],
  "execution_time_ms": 12
}
```

---

## :material-text-search: CloudWatch Logs Insights Queries

These examples assume JSON logs in CloudWatch Logs (default with ECS awslogs/awsfirelens). Adjust the log group and time range.

### 1) Follow a specific request across request/stream/background

```sql
fields @timestamp, type, level, path, event, status_code, execution_time_ms
| filter id = "<paste-request-id>"
| sort @timestamp asc
```

!!! tip
    Copy the request ID from the `x-request-id` response header or any `request` log line. Expect one `request`, optional `request_stream` entries, and `background` entries.

### 2) Find recent errors with context

```sql
fields @timestamp, level, type, path, status_code, id, error_detail
| filter level in ["error", "critical"]
| sort @timestamp desc
| limit 100
```

### 3) High-latency endpoints (P95/P99)

```sql
fields path, execution_time_ms
| filter type = "request" and ispresent(execution_time_ms)
| stats pct(execution_time_ms, 95) as p95_ms, pct(execution_time_ms, 99) as p99_ms, avg(execution_time_ms) as avg_ms by path
| sort p95_ms desc
```

---

## :material-aws: AWS Service-Level Logs and Metrics

Beyond stdapi.ai logs and OTel traces,
use AWS-native signals from the underlying AI services to validate provider behavior,
monitor throttling/latency, and audit access.
Enable only what you need: some options can capture content and increase costs.
For details, refer to the official AWS documentation.

- CloudWatch Metrics: Throughput, latency, throttling, and error rates per service/region.
- CloudTrail: Control-plane auditing of API calls (who did what, when, from where).
- Content/Invocation logging: Optional features that may record inputs/outputs. Use with caution and encryption/retention controls.
- Correlation: Service logs won't include stdapi.ai's `x-request-id`. Correlate by time window, region, model/voice/job identifiers, and volume. Use stdapi.ai's `model_id`, `voice_id`, and `execution_time_ms` to narrow windows.
- Amazon Bedrock Invocation logging (optional): Export invocation metadata and, if enabled, content to CloudWatch Logs/S3/Firehose. Treat prompts/completions as sensitive; manage retention and KMS.

### AWS Service Correlation Metadata

stdapi.ai attaches correlation identifiers to outgoing service calls, allowing you to trace a stdapi.ai request back to the corresponding AWS service invocation.

| Key                    | Description                                                                                                                      |
|:-----------------------|:-----------------------------------------------------------------------------------------------------------------------------------|
| `stdapi-ai.request_id` | Matches the `id` field in stdapi.ai logs and the `x-request-id` response header                                                  |
| `stdapi-ai.server_id`  | Matches the `server_id` field in stdapi.ai logs                                                                                  |
| `stdapi-ai.user_id`    | Present only when the client supplies a user identifier — the `safety_identifier` field (or the deprecated `user` alias) in OpenAI requests, `metadata.user_id` in Anthropic requests |

Coverage and how to use it varies by service:

- **Bedrock — synchronous inference** (every chat, embedding, image and audio route served by `bedrock-runtime`): identifiers are embedded in the invocation request — both the `Converse` and the `InvokeModel` families, streaming included — and appear in Bedrock invocation log records when [model invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) is enabled. Filter by `requestMetadata.stdapi-ai\.request_id` in CloudWatch Logs Insights to join a Bedrock record with its stdapi.ai event.
- **Bedrock — asynchronous batch jobs**: identifiers are attached as resource tags on the job, visible in the AWS Console and searchable via the CLI/API.
- **Bedrock Mantle**: the Mantle endpoint accepts no request metadata. Attribute those requests with a [Project/Workspace](operations_configuration_aws.md#bedrock-mantle-project) instead.
- **Transcribe — audio transcription**: identifiers are attached as job tags. stdapi.ai deletes completed transcription jobs automatically, so tags are only available while the job is still running.

Correlation also works in the other direction — from a stdapi.ai log event to the exact AWS API calls it made:

- **`aws_requests`** (on `request`, `request_stream` and `background` events): one entry per downstream AWS API call, with `service`, `operation`, `request_id` and — for failed calls — the AWS `error` code. The `request_id` value is the AWS-side request ID (`ResponseMetadata.RequestId` / `x-amzn-requestid` header): the identifier AWS Support asks for, and the `requestID` field of CloudTrail records. Each event keeps at most the 50 most recent calls.
- **Edge headers** (on the `request` event, when present): `amzn_trace_id` (`X-Amzn-Trace-Id`, injected by ALB and used by X-Ray), `apigw_request_id` (`x-amz-apigw-id`, API Gateway) and `cloudfront_request_id` (`X-Amz-Cf-Id`, CloudFront) tie the gateway's `id` to the edge service's own access logs. They are recorded as-is for correlation and never replace the gateway-generated request ID.

!!! note "Security"
    Any `stdapi-ai.*` key supplied by the client is silently dropped before the service call. Only values injected by stdapi.ai itself are forwarded under that prefix.

---

## :material-cloud-check: Infrastructure Observability

!!! info "Terraform Module"
    The Terraform module automatically configures production-ready observability for all infrastructure components — no manual setup required.

    **Application Logs (ECS)**

    Container STDOUT is forwarded to a dedicated CloudWatch Logs log group. Log retention defaults to **365 days** (`cloudwatch_logs_retention_in_days`). All log groups are **KMS-encrypted**.

    **Container Insights**

    ECS Container Insights is enabled by default (`container_insight = "enabled"`), providing CPU, memory, network, and storage metrics per task. Set `container_insight = "enhanced"` to enable enhanced observability with additional OS-level and application performance metrics.

    **CloudWatch Alarms**

    The module can trigger a CloudWatch alarm whenever an `error` or `critical` log event is detected. Enable with `alarms_enabled = true` and provide `sns_topic_arn` to receive notifications via SNS (email, Slack, PagerDuty, etc.). When enabled, a metric filter scans the ECS log group for any line containing `error` or `critical` and fires the alarm as soon as the count exceeds zero in a 1-minute period, evaluated over a single period.

    **ALB and WAF Logs**

    ALB access logs and WAF logs are stored in dedicated S3 buckets, **KMS-encrypted**. These capture all HTTP requests at the infrastructure level — use them to audit traffic patterns and investigate security events before they reach the application.

---

## :material-wrench: Troubleshooting Checklist

- No logs visible: Ensure you are reading container STDOUT. On ECS/Kubernetes, verify the log driver and retention.
- Missing `request_params`: Confirm `LOG_REQUEST_PARAMS=true` and restart after changing environment variables.
- No traces: Verify `OTEL_ENABLED=true` and that exporters are configured and reachable.
- Correlation missed: Ensure clients read and propagate `x-request-id` for multi‑hop requests.

---

## :material-arrow-right: Next Steps

<div class="grid cards" markdown>

- :material-cash-multiple: [**Cost Management**](operations_cost_management.md) — Cost tracking, Marketplace model billing, and cost attribution
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Complete list of environment variables including logging options
- :material-chart-bar: [**Data Sovereignty & Compliance**](operations_compliance.md) — Compliance-focused logging and audit requirements
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — Production deployment with observability stack

</div>
