Multi-source setup

If traffic comes in through more than one reverse proxy or host, GeoMetrikks can run as one full instance plus lightweight agents instead of one instance tailing everything remotely. Each agent runs next to its own access logs (same host or Docker network as the proxy it tails) and does the ingestion locally: tail, parse, geolocate, write, publish. One full instance owns everything else: the UI, the API, database migrations, the scheduler, and CrowdSec. Every agent and the full instance share one TimescaleDB. The live map stays in sync whichever process ingested a request, because every writer publishes committed events over PostgreSQL LISTEN/NOTIFY and the full instance’s /ws/live feed relays all of them.

Every request records the server that ingested it. LOGPARSER_SERVER_NAME sets that name. It is a name you choose for this server, not your nginx server_name, and it defaults to the machine’s hostname. In Docker that is the container’s hostname, which the compose hostname: key sets. Access Logs, Geo Logs and the map filter by server.

Older releases called the server a hostname. The old variable names still work until 1.0:

Variable	Old name
LOGPARSER_SERVER_NAME	LOGPARSER_HOST_NAME
MAP_SERVER_LOCATIONS	MAP_HOME_LOCATIONS
MAP_SERVER_LATITUDE	MAP_HOME_LATITUDE
MAP_SERVER_LONGITUDE	MAP_HOME_LONGITUDE
MAP_AUTO_DETECT_SERVER_LOCATION	MAP_AUTO_DETECT_HOME
MAP_SERVER_LOCATION_REFRESH_HOURS	MAP_HOME_REFRESH_HOURS

At startup, a process that reads any old name logs one legacy_env_names_in_use event listing them, and its /health carries a legacy-env-names advisory naming each replacement. Each process reports only its own settings. The full instance’s Settings > Status shows the old names the full instance reads. Settings > Agents shows the advisory on the row of each agent that reads an old name. If an old and a new name are set to different values, startup fails.

Agents are not the only shape. If the log files already reach one machine (a shared mount, rsyslog, log shipping), a single full instance can tail them all and keep them apart by giving LOGPARSER_SERVER_NAME a JSON list matched positionally to LOGPARSER_LOG_PATHS:

LOGPARSER_LOG_PATHS=["/var/log/access/edge-01.log", "/var/log/access/edge-02.log"]
LOGPARSER_SERVER_NAME=["edge-01", "edge-02"]

Everything downstream treats those files like traffic from separate agents. Each one is its own server in the Server filters and gets its own server location on the map.

If the logs already go to Grafana Loki, one full instance can read every proxy’s lines from there instead, with one LogQL selector and one server name per proxy. See Loki setup.

An agent needs only APP_MODE=agent, database credentials for the shared instance, GeoIP credentials, and its own log mount:

services:
  agent:
    image: ghcr.io/gilbn/geometrikks:0.21.0   # same tag as the full instance
    restart: unless-stopped
    stop_grace_period: 20s
    environment:
      APP_MODE: agent
      APP_INSTANCE_NAME: edge-01
      DB_HOST: timescale.example.internal
      DB_PORT: "5432"
      DB_USER: geouser
      DB_PASSWORD: ${DB_PASSWORD}
      DB_DATABASE: geometrikks
      MAXMINDDB_USER_ID: ${MAXMINDDB_USER_ID}
      MAXMINDDB_LICENSE_KEY: ${MAXMINDDB_LICENSE_KEY}
      LOGPARSER_LOG_PATHS: '["/var/log/access/access.log"]'
      LOGPARSER_SERVER_NAME: edge-01
    volumes:
      - geoip_data:/app/data/geoip
      - /var/log/nginx:/var/log/access:ro
    healthcheck:
      # The image's own healthcheck probes /health, which answers 200 for
      # the whole schema wait. /health/ready is the one that reports 503
      # while the agent waits for the primary to migrate.
      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health/ready', timeout=5).read()\" || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      # The schema gate runs inside ASGI startup, so the port does not
      # accept at all for up to 120s. Without this grace window the first
      # probes fail a perfectly healthy cold start.
      start_period: 150s

volumes:
  geoip_data:

APP_MODE=agent is a headless process: it tails, geolocates, writes and publishes like a full instance, but serves only /health and /health/ready. No UI, no API, no OpenAPI schema, no session auth, so it needs no APP_ADMIN_PASSWORD. It downloads and refreshes its own GeoLite2 database, so it needs MaxMind credentials and a geoip volume. It never runs migrations or creates TimescaleDB objects; its writes are events and its own server_locations row. At startup it waits for the shared database’s schema to reach the revision it was built against. If that wait times out, the agent stays up in degraded mode rather than exiting, with /health answering 200 and /health/ready answering 503. An orchestrator with a readiness probe restarts it into a fresh wait; without one, restart the agent container yourself once the full instance has finished migrating.

Compose does not act on health status, so on plain Docker the probe above shows a stuck agent as unhealthy in docker ps and gates any depends_on: condition: service_healthy you add. Turning that signal into a restart takes Swarm, Kubernetes, or an autoheal sidecar. restart: unless-stopped will not do it: it reacts to a container exiting, and a waiting agent stays up.

Agent health. Every agent and the full instance write a heartbeat to the shared database every 30 seconds. Settings > Agents on the full instance lists them, the full instance first as the head. Each row shows the version, server names, lines and failures in the last 5 minutes, and when it last checked in. Click a row to see the rest of what that agent reports: ingestion counters, GeoIP databases, scheduled jobs, each log source with its own light, server locations and the last error. An agent that has not checked in for 3 minutes turns red, and only then does its row get a Remove button. If that agent is in fact still running, its next heartbeat brings the row back.

You don’t have to keep the Agents page open to notice a problem. Each agent that needs attention gets one card at the top of Settings > Status, red or amber by its worst problem, linking to its row on the Agents page. An Agents card further down sums them up, for example “3 agents, 1 out of date, 1 not checking in”. The sidebar status dot turns amber too. With no agents, the Status page looks the same as before.

Set APP_INSTANCE_NAME on every agent. The heartbeat row is keyed by that name, so it is how the Agents page tells agents apart. Without it, an agent goes by agent and the full instance by head. Two unnamed agents then write to the same agent row and overwrite each other’s status, so the page shows whichever one wrote last. The full instance flags an agent running under the default name, and flags a name more than one process is using. The fix for both is a unique name per agent. APP_INSTANCE_NAME names the process, while LOGPARSER_SERVER_NAME names the server on each request it records, so one agent tailing two servers’ logs is still one row.

An agent from a release before heartbeats never shows up on the Agents page. It also predates the rename of the hostname columns to server, so every insert it makes fails. Upgrade it to the full instance’s release.

CDN peer advisories for an agent’s tailed sources reach the head’s Settings > Status page. The head scans the shared database’s last hour of access-log rows every 5 minutes, covering only sources with LOGPARSER_SEND_LOGS=true. Private-peer lines are never stored, so that scan can’t find them. Each agent checks its own lines instead and forwards the private-peer advisory in its heartbeat, so it shows on that agent’s row on the Agents page as well as in the agent’s own /health. See docs/proxy-setup.md for details.

The reverse case works too. To keep a full instance’s UI and API without it tailing local files (a machine that only hosts the app, with all traffic ingested by agents elsewhere), set:

LOGPARSER_ENABLED=false

It still serves the UI, API, migrations, scheduler and CrowdSec; it never tails a log file itself.

Server locations. Live map routes fly to the location of the server that recorded them, one beacon per server. Each ingesting instance detects its own public-IP location and re-checks it every MAP_SERVER_LOCATION_REFRESH_HOURS (default 24h). When detection is wrong for a server (CGNAT, a VPN egress, or logs shipped from another machine), pin that server’s location on the full instance with MAP_SERVER_LOCATIONS, for example MAP_SERVER_LOCATIONS={"edge-01": [60.39, 5.32]}. Overrides win over detection, and removing one restores it. The Server locations card on Settings > Status lists each server’s location and whether it came from detection or an override.

[!WARNING] Trust model. Agents authenticate with the database using ordinary database credentials, and the app does not care where data comes from. Anyone who can run an agent has full read/write access to the entire database: all traffic history from every source, any server, no tenant isolation, no per-agent identity, and no revocation short of rotating the shared password. Sharing one instance across parties works as long as everyone understands they share everything. Run agent connections over a VPN or tailnet, not the open internet.

Keep versions aligned. Run the same image tag on every agent and the full instance. An agent tolerates the shared database running slightly ahead of its own bundled schema (a full instance mid rolling-restart) by logging a warning and proceeding rather than refusing to start. That is an allowance for a brief mismatch, not a reason to run agents and the full instance on different versions.

Upgrade every instance together to the release that renames hostnames to servers. The full instance renames the database columns and tables on its first start, without rewriting any data. An agent from an older release still starts against the renamed database, then fails every insert.

CrowdSec. Point CROWDSEC_LAPI_URL and the bouncer/machine credentials (see CrowdSec integration) at the central LAPI on the full instance only; per-machine CrowdSec agents keep reporting to that same LAPI as usual. GeoMetrikks agents ignore CROWDSEC_* settings; ban visibility and management stay with the full instance.