Diagnostic commands

# Container status
podman ps -a | grep automation-metrics

# Systemd unit status (user scope)
systemctl --user status automation-metrics-web.service
systemctl --user status automation-metrics-tasks.service
systemctl --user status automation-metrics-scheduler.service

# Recent logs
podman logs automation-metrics-web --tail 100
podman logs automation-metrics-tasks --tail 100
podman logs automation-metrics-scheduler --tail 100

# Init container logs (for migration issues)
podman logs automation-metrics-init

# Health endpoint check (via nginx)
# Returns HTTP 2** (e.g., 200) if ok, 50* (e.g., 503) if degraded
curl https://localhost:8450/health/ ( http://localhost:8087/health if HTTPS is disabled).

# Database connectivity
psql -h localhost -U metrics_service -d metrics_service -c "SELECT 1;"
psql -h localhost -U ms_awx_readonly -d awx -c "SELECT COUNT(*) FROM main_job LIMIT 1;"

# Verify Podman secrets (should show 4 secrets)
podman secret ls | grep automationmetrics

Common issues

Service containers fail to start with validation errors

New in Ansible Automation Platform 2.7: Metrics service validates required production environment variables at startup to prevent misconfiguration.

Symptom: Service containers fail to start with log messages indicating missing or invalid environment variables.

Cause: Required production environment variables are not set or are incorrectly configured.

Solution:

1. Check container logs for validation error messages:

   podman logs automation-metrics-web --tail 50 | grep -i "validation\|error\|required"

2. Verify required environment variables are set correctly:

   podman exec automation-metrics-web env | grep METRICS_SERVICE

3. Common validation failures:

   Validation Error	Required Variable	Solution
   Database host validation failed	METRICS_SERVICE_DB_HOST	Verifyautomationmetrics_pg_host is set in inventory
   Controller database validation failed	METRICS_SERVICE_CONTROLLER_DB_HOST	Verifyautomationmetrics_controller_read_pg_host is set
   Secret key validation failed	METRICS_SERVICE_SECRET_KEY	Verify Podman secretautomationmetrics_secret_key exists

4. After fixing environment variables, restart the service:

   systemctl --user restart automation-metrics-web.service

Rollup processing troubleshooting

Metrics service uses a daily rollup process to aggregate hourly metrics data. The following scenarios describe common rollup issues and solutions.

Scenario 1: Rollup task failing

Symptom: daily_metrics_rollup task fails or shows errors in automation-metrics-scheduler logs.

Cause: Hourly collection tasks did not complete successfully before rollup runs at 2:00 AM.

Explanation: The rollup task depends on hourly data being available. If hourly collectors (collect_hourly_metrics) fail or are delayed, the rollup cannot aggregate incomplete data.

Solution:

1. Verify hourly collection tasks completed successfully:

   podman exec automation-metrics-web \
     psql -h localhost -U metrics_service -d metrics_service -c \
     "SELECT name, status, started, finished FROM dynamic_tasks_taskexecution
      WHERE name LIKE 'collect_hourly_metrics%'
      ORDER BY started DESC LIMIT 24;"

   Expected output: All hourly tasks should show status = 'success' for the past 24 hours.

2. Check scheduler logs for task execution order:

   podman logs automation-metrics-scheduler --tail 100 | grep -E "collect_hourly_metrics|daily_metrics_rollup"

3. If hourly tasks are failing, investigate root cause (database connectivity, controller database access, query timeouts).
4. After fixing hourly collection, manually trigger rollup:

   podman exec automation-metrics-web metrics-service tasks create \
     --name "manual-rollup" \
     --function daily_metrics_rollup

Scenario 2: cleanup_metrics_data running before rollup completes

Symptom: Rollup produces incomplete results or missing data for recent periods.

Cause: Data cleanup task (cleanup_metrics_data at 4:00 AM) runs before rollup completes, purging hourly data that rollup still needs.

Explanation: This should not happen given the schedule (rollup at 2:00 AM, cleanup at 4:00 AM), but can occur if rollup is delayed or takes longer than 2 hours due to large data volumes.

Solution:

1. Check rollup task duration:

   podman exec automation-metrics-web \
     psql -h localhost -U metrics_service -d metrics_service -c \
     "SELECT name, started, finished, (finished - started) AS duration
      FROM dynamic_tasks_taskexecution
      WHERE name = 'daily_metrics_rollup'
      ORDER BY started DESC LIMIT 7;"

2. If rollup consistently takes >2 hours, adjust cleanup schedule by setting later execution time (contact Red Hat Support for guidance on modifying task schedules).
3. Verify data retention settings allow sufficient overlap:

   podman exec automation-metrics-web env | grep RETENTION

   Expected: Hourly data retained for 7 days (default), giving rollup sufficient time to process even if delayed.

Scenario 3: Stuck task auto-detection and reset (NEW GA feature)

New in Ansible Automation Platform 2.7: The scheduler automatically detects and resets tasks stuck in running state beyond their timeout.

Feature behavior:

• Scheduler periodically checks for tasks in running state longer than configured timeout
• Automatically resets stuck tasks to allow retry on next schedule
• Logs stuck task detection and reset actions in automation-metrics-scheduler container logs

How to monitor automatic stuck task recovery:

1. Check scheduler logs for stuck task detection:

   podman logs automation-metrics-scheduler | grep -i "stuck\|timeout\|reset"

   Expected output: Log messages indicating stuck task detection and automatic reset.

2. Query task execution history to see reset tasks:

   podman exec automation-metrics-web \
     psql -h localhost -U metrics_service -d metrics_service -c \
     "SELECT name, status, started, finished
      FROM dynamic_tasks_taskexecution
      WHERE status = 'timeout' OR status = 'reset'
      ORDER BY started DESC LIMIT 10;"

When manual intervention is still needed:

Automatic recovery handles transient issues (network timeouts, temporary database locks). Manual intervention is required when:

• Same task repeatedly gets stuck (indicates underlying infrastructure issue)
• Task execution observability metrics show declining success rates
• Database queries from metrics service are slow (check controller database performance)

Scenario 4: Task execution observability

New in Ansible Automation Platform 2.7: collect_daily_metrics task with task_executions service provides pipeline health metrics.

Purpose: Monitor the health and performance of the metrics collection pipeline itself (meta-metrics).

What is collected:

• Task success rates for all collectors (hourly, daily, rollup, anonymization)
• Task execution timing and duration
• Task error counts and types
• Collection throughput metrics

How to access pipeline health metrics:

1. Query task execution observability data:

   podman exec automation-metrics-web \
     psql -h localhost -U metrics_service -d metrics_service -c \
     "SELECT * FROM task_executions ORDER BY collected_at DESC LIMIT 10;"

2. Identify declining success rates:

   podman exec automation-metrics-web \
     psql -h localhost -U metrics_service -d metrics_service -c \
     "SELECT name,
             COUNT(*) FILTER (WHERE status = 'success') AS success_count,
             COUNT(*) FILTER (WHERE status = 'failure') AS failure_count,
             ROUND(100.0 * COUNT(*) FILTER (WHERE status = 'success') / COUNT(*), 2) AS success_rate
      FROM dynamic_tasks_taskexecution
      WHERE started > NOW() - INTERVAL '7 days'
      GROUP BY name
      ORDER BY success_rate ASC;"

   Expected: Success rates >95% for all tasks. Rates <90% indicate infrastructure issues requiring investigation.

3. Monitor task duration trends:

   podman exec automation-metrics-web \
     psql -h localhost -U metrics_service -d metrics_service -c \
     "SELECT name,
             AVG(EXTRACT(EPOCH FROM (finished - started))) AS avg_duration_seconds,
             MAX(EXTRACT(EPOCH FROM (finished - started))) AS max_duration_seconds
      FROM dynamic_tasks_taskexecution
      WHERE started > NOW() - INTERVAL '7 days' AND status = 'success'
      GROUP BY name
      ORDER BY avg_duration_seconds DESC;"

Use pipeline health metrics to:

• Detect performance degradation early (increasing task durations)
• Identify which collectors are failing most frequently
• Validate infrastructure changes (e.g., database upgrades) haven't impacted collection

Troubleshoot events collector

Copy link

Events collector timeout or slow performance

Symptom: The main_jobevent_service task shows failure status or takes longer than expected in the /api/v1/tasks/ output.

Cause: Large deployments with high job volumes can exceed the 3600-second task timeout when processing job events. The stuck-task detector marks timed-out tasks as failed.

Resolution:

• Check the current limit settings:
  • JOBEVENT_ROW_LIMIT (default: 200,000) - controls maximum rows processed per cycle
  • JOBEVENT_JOB_LIMIT (default: 1,000) - controls maximum jobs processed per cycle
• Reduce the limits to lower values to decrease processing time. The collector has a built-in time-budget check that stops processing early and marks the collection as partial (not failed) if it risks exceeding the timeout.
• Verify the collector completes within the hourly window by querying the tasks API:

curl
https://<gateway-route>/api/metrics/v1/tasks/?name=main_jobevent_services&limit=5 \
    -H "Authorization: Bearer <token>"

Partial event data collection

Copy link

Symptom: Collection/module usage statistics appear incomplete or show fewer entries than expected.

Cause: When JOBEVENT_ROW_LIMIT, JOBEVENT_JOB_LIMIT, or built-in time-budget cap is reached, the collector stops processing and marks the collection as partial. The HourlyMetricsCollection record is stored with whatever data was collected. Uncollected events from that cycle are skipped permanently - partial collections are NOT retried for the same time window. The next hourly run collects events only from jobs that finished during the new hourly window.

Resolution: If more complete per-cycle collection is required, increase the limit values. Partial collections are included in the daily rollup with whatever data was collected, so no data gaps occur in the daily summary - but per-cycle granularity may be reduced.

Cannot disable events collector individually

Copy link

Symptom: You want to disable only the events collector without disabling other metrics collection.

Cause: There is no dedicated feature flag for the events collector. It runs as part of the METRICS_COLLECTION task group.

Workaround: Set both JOBEVENT_ROW_LIMIT and JOBEVENT_JOB_LIMIT to 1. This effectively disables event collection by limiting processing to a negligible amount, while leaving all other metrics collection active.

Troubleshoot indirect node collection

Copy link

Use these procedures when indirect node counts are missing, incomplete, delayed, or absent from the metrics report. Indirect node counting depends on a query file, either an embedded extensions/audit/event_query.yml or a supported external query file such as redhat.indirect_accounting.

Automation controller must record the audit data before metrics service can collect it. First, check the automation controller table main_indirectmanagednodeaudit for the period in question. If the records are not there, the cause is on the controller side, most often a missing or incomplete query file or indirect node counting turned off, and no metrics service configuration will produce counts. If the records are there but the counts are missing from metrics service or from the metrics utility report, the cause is on the metrics service side. Continue with the issues that follow.

Symptom: Automation you run against a network platform (for example, cisco.ios) does not appear in indirect node counts, or a collection shows fewer nodes than expected.

Cause: Only collections covered by a query file report indirect nodes. Some collections have no coverage at all; others are covered for only part of their modules.

Resolution:

Confirm the collection is in the supported list. See the supported-collections snapshot in Understand indirect node counting and collection. If the collection is not supported, no counts are produced for it. This is expected, not a fault. To request or add coverage for a new collection, see the next issue. If the collection is listed as partial, expect counts only for the covered modules.

Symptom: You want indirect node counts for a collection that does not currently report them.

Cause:

The collection has no embedded extensions/audit/event_query.yml query file and is not covered by an external query file such as redhat.indirect_accounting, or the query file does not cover the modules you use.

Resolution:

Query file authoring is documented in the Red Hat Knowledgebase article How to add indirect node counting support to an Ansible collection. Partners certifying a collection follow the Ansible Partner Connect certification guide. Metrics service does not need to be modified: once a collection ships a valid query file and that collection is used, its indirect nodes are counted automatically.

Symptom: Automation you just ran against indirect nodes does not appear in counts immediately.

Cause: The automation controller processes indirect node audit records shortly after the corresponding job events are processed, normally within minutes. An hourly fallback job is a safety net that picks up any jobs missed by the immediate path; it is not the normal processing path.

Resolution:

Allow a few minutes after the job completes for the controller to record the audit data. Remember that metrics service collects indirect node audit data once per day, at 1:55 AM UTC. Counts in the anonymized payload and in the metrics utility report therefore update on the daily cycle, not immediately after each job. If data is still missing after the next daily collection, verify that indirect node counting is enabled on automation controller and that the audit records were not purged before collection.

Symptom: Indirect node counts are lower than expected or missing for periods when the metrics utility gather runs infrequently.

Cause: The automation controller retains indirect node audit records for 7 days by default (INDIRECT_HOST_AUDIT_RECORD_MAX_AGE_DAYS). Records older than the retention window are purged and cannot be recovered.

Resolution:

Run the metrics utility gather within the retention window, or increase INDIRECT_HOST_AUDIT_RECORD_MAX_AGE_DAYS to cover the full interval between gathers. For example, if you gather monthly, set the retention to at least 31 days. This is an automation controller setting, not a metrics service inventory variable.

Symptom: The indirectly_managed_nodes report sheet (CCSP or CCSPv2) is empty.

Cause:

Indirect node counting is disabled on automation controller (run awx-manage set_indirect_node_counting. Enable to turn it back on). No automation has run against indirect nodes during the reporting window. The collections in use are not covered by a query file, so no indirect nodes are counted. Audit records were purged before collection because the retention period is shorter than the gather interval.

Resolution:

Verify that indirect node counting is enabled on automation controller. Confirm at least one supported collection has been used to manage indirect nodes during the window. Confirm the retention period covers the interval between gathers. Re-run the metrics utility gather, then regenerate the report.