Collect a troubleshooting bundle
Collect a troubleshooting bundle from an Infrahub deployment and share it with OpsMill support. Collection is read-only and safe to run at any time, including while the instance is degraded. See Infrahub Collect for what a bundle contains and how secrets are masked. The same command produces the same bundle on Docker Compose and Kubernetes.
Prerequisites
Before collecting a bundle:
- Docker Compose
- Kubernetes
- The
infrahub-collectbinary is installed (installation guide) - Docker and Docker Compose are available to your user
- The Infrahub Compose project is running (fully or partially)
- The
infrahub-collectbinary is installed (installation guide) kubectlis configured with access to the cluster running Infrahub- Your role can read pod logs and execute commands in pods (
pods/logandpods/exec); no write or scale permissions are needed
The binary is self-contained and collection works offline by default: it uses your existing Docker or kubectl access and performs no network access beyond your deployment itself. The one exception is the opt-in --benchmark, which downloads a benchmark image.
Step 1: Verify environment detection
Confirm the tool can find your Infrahub deployment:
infrahub-collect environment detect
The tool auto-detects Docker Compose projects and Kubernetes namespaces containing Infrahub. To list every deployment it detects:
infrahub-collect environment list
Step 2: Create the bundle
- Docker Compose
- Kubernetes
For a single Infrahub project, run:
infrahub-collect create
If several Infrahub projects run on the same host, target one explicitly:
infrahub-collect create --project=infrahub-production
If a single namespace contains Infrahub, run:
infrahub-collect create
To target a specific namespace:
infrahub-collect create --k8s-namespace=infrahub-prod
On Kubernetes, the bundle includes logs from every replica of each service, and for pods that crashed and restarted it also captures the previous container's logs — often exactly where the error is.
By default the bundle is written to ./infrahub_bundles. Use --output-dir to choose another location:
infrahub-collect create --output-dir=/tmp/support
Step 3: Monitor collection progress
The command streams progress as each collector runs:
INFO[0000] Detecting deployment environment...
INFO[0000] Kubernetes environment detected (namespace: infrahub-prod)
INFO[0000] Creating collection ID: 20260702_141530
INFO[0001] Collecting logs: infrahub-server (2 replicas)
INFO[0004] Collecting logs: task-worker (3 replicas, 1 restarted - including previous logs)
INFO[0009] Collecting database logs...
INFO[0012] Collecting message-queue status...
INFO[0013] Collecting cache status...
INFO[0014] Collecting task-manager state...
INFO[0016] Collecting server info...
INFO[0017] Collecting container metrics...
INFO[0018] Creating compressed archive...
INFO[0021] Bundle created: infrahub_bundles/support_bundle_20260702_141530.tar.gz
If a service is down or unreachable, collection continues with the remaining services and records the failure:
WARN[0013] Collector failed: cache status (container not running) - continuing
A partial bundle is still a useful bundle — a degraded instance is precisely when you need one.
Step 4: Verify the bundle
Check the archive and review what was collected:
# Verify archive integrity
tar -tzf infrahub_bundles/support_bundle_20260702_141530.tar.gz > /dev/null && echo "Archive is valid"
# View the collection manifest
tar -xzOf infrahub_bundles/support_bundle_20260702_141530.tar.gz bundle/bundle_information.json | jq '.'
The manifest records what was attempted and what succeeded:
{
"manifest_version": 2026070200,
"collect_id": "20260702_141530",
"created_at": "2026-07-02T14:15:30Z",
"tool_version": "1.2.0",
"infrahub_version": "1.5.2",
"environment": "kubernetes",
"edition": "enterprise",
"helm": {
"release_name": "infrahub",
"chart": "infrahub-enterprise",
"chart_version": "1.2.3"
},
"log_lines": 100000,
"collectors": [
{"name": "logs/infrahub-server", "status": "success"},
{"name": "logs/task-worker", "status": "success"},
{"name": "cache-status", "status": "failed", "reason": "container not running"},
{"name": "benchmark", "status": "skipped", "reason": "not requested"}
]
}
The edition field reports whether the deployment runs Infrahub Community or Enterprise. On Docker Compose it is classified from the infrahub-server container image, and on Kubernetes from the Helm chart name. It is omitted when the edition cannot be determined.
The bundle layout is identical on Docker Compose and Kubernetes. Inside the archive, all files live under a top-level bundle/ directory:
bundle_information.json- Collection manifestlogs/<service>/- Container logs, one file per replica (plus*.previous.logfor restarted pods)database/- Neo4j server logsmessage-queue/- RabbitMQ queues, exchanges, connections, and statuscache/- Redis info, clients, configuration, and slow logtask-worker/- Task worker state, one directory per replicatask-manager/- Work pools, work queues, recent flow runs, the pending and running flow and task runs, events, and automationstask-manager-db/- Task manager database and table sizes (storage.json), and the storage breakdown (storage-breakdown.json) when you runcreatewith--include-task-manager-storageserver/- Version, installed packages, API information, configuration, schema, worker schema-sync status (infrahub_status.json), environment variablesmetrics/- Container resource metrics
Each file holds the output of the command that produced it and nothing else, so a .json file always parses. When a command fails, its file is absent and a sibling <name>.err.txt records what ran, how it failed, and what the command reported — for example task-manager/flow-runs.err.txt in place of flow-runs.json. The manifest lists the same failure as a reason on the collector.
Masking is based on key names: environment variables and configuration keys containing password, secret, token, or key are replaced with ******** before they are written to the bundle. Secrets stored under other key names are not detected. Service logs are collected as-is, and database query logs (--include-queries) can contain customer data. Review the bundle contents before sharing.
Step 5: Share the bundle with support
Send the archive to OpsMill support through your usual support channel. The bundle is a plain local file, so environments with restricted egress can transfer it through whatever mechanism their security policy allows.
Advanced usage
Include a backup for issue reproduction
Support often asks for a backup alongside logs so they can reproduce your issue locally. Create both in one run:
infrahub-collect create --include-backup
This runs the standard infrahub-backup logic and records the backup in the bundle manifest.
Unlike log collection, the backup step follows the normal backup behavior for your deployment, which can include stopping application containers while the snapshot is taken. Plan for a short interruption, or skip --include-backup and create the backup separately during a maintenance window.
Include database query logs
Query logs can help diagnose slow or failing database operations, but they may contain customer data, so they are excluded by default:
infrahub-collect create --include-queries
Adjust the log volume limit
By default, collection captures the most recent 100,000 log lines per container. Raise or lower the limit as needed:
# Collect more history for a long-running issue
infrahub-collect create --log-lines=500000
# Or via environment variable
export INFRAHUB_LOG_LINES=500000
infrahub-collect create
The effective limit is recorded in the manifest, so support knows whether logs were truncated.
Run a performance benchmark
For performance investigations, support may ask you to include a benchmark run:
infrahub-collect create --benchmark
The benchmark requires downloading the OpsMill benchmark container image and measures whether the host meets Infrahub's resource requirements — disk IOPS, memory, and single-core CPU — rather than generating load against your instance. If the image cannot be pulled — for example in an air-gapped environment — the benchmark is skipped with a warning and the rest of the collection completes normally.
Include the storage breakdown of the task manager database
Every bundle holds the size of the task manager database and of each of its tables, in task-manager-db/storage.json. If support asks which kinds of records use the space in the run, state, log, event and artifact tables, collect a new bundle with the storage breakdown:
infrahub-collect create --include-task-manager-storage
# Or via environment variable
export INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE=true
infrahub-collect create
The breakdown reads every row of seven Prefect tables (events, flow_run, task_run, flow_run_state, task_run_state, log, and artifact), and writes the exact row count and the oldest and newest timestamps of each group of records to task-manager-db/storage-breakdown.json. Because the breakdown reads every row, it is not part of a default bundle. PostgreSQL stops the query on each table after 5 minutes, so the breakdown can take up to 5 minutes for each of the seven tables. The progress output names each table when its query starts, and reports the result when the query ends:
INFO[0021] Collecting storage breakdown of table events (time limit 300s)
INFO[0093] Table events: 5000000 rows in 42 groups (1m12s)
INFO[0093] Collecting storage breakdown of table flow_run (time limit 300s)
A table that reaches the time limit is recorded as failed in storage-breakdown.json, and the other tables are still collected. The manifest then records the task-manager-db-storage-breakdown collector as failed, with a reason that names each failed table, for example partial storage breakdown: log: timed out after 300s. A table that does not exist in the deployed Prefect version is recorded as absent, not as a failure.
Do not use --include-task-manager-storage while the task manager is being upgraded. Each breakdown query takes an ACCESS SHARE lock on the table it reads and on the tables it joins, and holds these locks until the query ends:
- The query on
flow_runalso locksflow. - The queries on
task_runandlogalso lockflow_runandflow.
A schema change on one of these tables waits until the query ends, which can take up to 5 minutes. While the schema change waits, PostgreSQL also makes every later query on that table wait behind it, including the queries of the task manager.
A schema change that already holds its lock when the breakdown starts makes each breakdown query that reads or joins its table wait as well. For example, while a schema change holds a lock on flow_run, the queries on flow_run, task_run, and log all wait, and each query that waits for 5 minutes is recorded as timed out after 300s.
During an upgrade, the figures in storage.json can be missing as well, even without the flag. To measure the size of a table, the storage query takes an ACCESS SHARE lock on it. It waits for this lock only while another session holds an ACCESS EXCLUSIVE lock on the table, the one lock mode that conflicts with ACCESS SHARE. Many forms of ALTER TABLE take an ACCESS EXCLUSIVE lock, and so do VACUUM FULL and TRUNCATE (see table-level locks in the PostgreSQL documentation). While another session holds such a lock on a table, the storage query waits and can reach its 60-second limit. The bundle then holds task-manager-db/storage.err.txt in place of storage.json, and the manifest records the task-manager-db-storage collector as failed with the reason timed out after 60s. If support needs these figures, collect a new bundle after the upgrade.
Stop the infrahub-collect queries in the task manager database
Stopping infrahub-collect, for example with Ctrl+C, does not stop a query that is already running in the database: PostgreSQL stops that query only when it reaches its time limit. Every query that infrahub-collect runs in the task manager database uses a read-only session with the PostgreSQL application name infrahub-collect. Because these sessions only read data, stopping them does not change any task manager data.
To stop these queries at once, stop infrahub-collect first, then run this statement in the task manager database:
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name = 'infrahub-collect';
Stop infrahub-collect before you run the statement, because while infrahub-collect runs, it records the table whose query you stopped as failed and starts the query on the next table.
Run the statement as a superuser, or as the role that infrahub-collect logged in with. PostgreSQL lets a role end the session of another role only in these cases (see server signaling functions in the PostgreSQL documentation):
- The role is a superuser.
- The role is a member of the role that owns the session.
- The role has the privileges of the
pg_signal_backendrole.
Only a superuser can end the session of a superuser.
infrahub-collect logs in with the user of the Prefect database connection URL (PREFECT_SERVER_DATABASE_CONNECTION_URL or PREFECT_API_DATABASE_CONNECTION_URL) in the environment where you run it, or else in the task-manager container. Without such a URL, it logs in with the POSTGRES_USER of task-manager-db, and without that variable, with the user postgres.
In the commands below, replace <user> with one of these roles and <database> with the task manager database:
- Docker Compose
- Kubernetes
docker compose -p <project> exec task-manager-db psql -U <user> -d <database> \
-c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name = 'infrahub-collect';"
In the Infrahub Docker Compose deployment, task-manager-db runs the official postgres image. In that image, the user that POSTGRES_USER names is a superuser, and connections from inside the container need no password, so this command does not ask for one. To read that user, run docker compose -p <project> exec task-manager-db printenv POSTGRES_USER.
kubectl exec -it -n <namespace> <task-manager-db-pod> -- psql -U <user> -d <database> \
-c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name = 'infrahub-collect';"
With -it, psql can ask for the password of <user> if the PostgreSQL server requires one.
On a CloudNativePG cluster, infrahub-collect runs its queries in the primary pod, so run the statement there. To find the primary pod, run kubectl get pods -n <namespace> -l cnpg.io/instanceRole=primary.
Troubleshoot collection
No deployment detected
If infrahub-collect create reports that no environment was found, list what the tool can see:
infrahub-collect environment list
On Docker Compose, confirm the project is running with docker compose ls; the project name must contain infrahub, or you must name it with --project. On Kubernetes, the namespace is discovered through the app.kubernetes.io/name=infrahub label; for deployments with custom labels, name the namespace with --k8s-namespace.
Permission denied on Kubernetes
Log collection needs the pods/log permission and the status collectors need pods/exec. If your role lacks one of them, the affected collectors are recorded as failed in the manifest. Ask your cluster administrator for read access to both — collection needs no write or scale permissions.
Docker or kubectl not found
The tool depends on the Docker CLI or kubectl being installed and configured on the machine where it runs. When neither is usable, the command fails immediately and the error names the missing CLI.
Individual collectors failed
Collector failures on a degraded deployment are expected — a stopped cache container cannot answer status queries. The command still exits successfully, and the manifest records each failure with a reason. Send the partial bundle as-is; support reads the manifest first.
Validation
Confirm your collection works:
-
infrahub-collect environment detectfinds your deployment - Bundle archive is created and passes the integrity check
- Manifest lists an explicit outcome for every collector
- Logs are present for every service replica
- Masked environment output contains no plaintext secrets