import-loki
Import a time range of each Loki selector into the database, and resume it when it stops partway.
What it does
Reading from Grafana Loki starts when the app starts, so the history before that stays in Loki. import-loki reads a time range of every LOKI_QUERIES selector through the same pipeline as live reading:
docker compose exec -u geometrikks app litestar import-loki --since 2026-09-01It uses the Loki settings of the app: LOKI_URL, the tenant and credentials, and per selector the format from LOKI_FORMATS and the server name from LOKI_SERVER_NAMES. A selector without a server name gets the first LOGPARSER_SERVER_NAME. Without LOKI_URL or LOKI_QUERIES it stops with an error. Each row gets the time written in its log line. The command reads Loki one hour at a time and prints its progress once per day of the range. At the end it refreshes the continuous aggregates for the imported range.
The GeoLite2 City database must be in place, as for import-logs. Without the ASN database the rows have no ASN data until you run backfill-asn.
Flags
| Flag | Default | What it does |
|---|---|---|
--since | required | Start of the range, inclusive. |
--before | where stored data begins | End of the range, exclusive. |
--query N | every selector | Only the Nth selector in LOKI_QUERIES, counting from 1. A number past the end is refused with the selector count. |
--force | off | Skips the overlap check described below. |
--batch-size | 500 | Records per database commit. |
--since and --before take a date such as 2026-09-01 or a datetime such as 2026-10-01T14:03:12+02:00. Without an offset the value is UTC. A --since that isn't earlier than --before is refused.
Without --before, each selector stops where the data already in the database begins. On the first run that is the oldest access-log row for the selector's server, which is where live reading started. After an earlier import-loki run it is the start of that run, unless the server has access-log rows older than the ones that run wrote. The command prints the end it picked and the --since and --before pair that repeats the same range.
A selector is not imported when its --since isn't earlier than that end, or when nothing is stored for its server yet and there is no end to take. The command prints the reason, lists the selector at the end and exits with an error. The other selectors still import. Pass --before in the second case.
The command picks every selector's end before any of them imports.
How resume works
Before the first selector starts, the command records each selector's range in the database. It then reads every range one hour at a time. After an hour's rows commit, it marks that hour as done.
If Loki fails, a database write fails, or you stop the command, rerun it with the same --since and --before. It prints resumed from and picks up at the start of the interrupted hour. An interrupted run keeps its own end, so the same --since without --before resumes it too.
Rows the command had already written from the interrupted hour are written again, so they end up twice. The rest of the range is not repeated.
A failure in one selector doesn't stop the others. At the end the command lists the selectors that failed and exits with an error.
Gotchas
- The boundary is off by the shipper's delay. When the default end is the oldest access-log row, it is the time written in the log line. Loki filters by the time the shipper stamped the entry, and the two differ by the shipper's delay, so lines near that boundary can be read twice or missed. An earlier import's start and an explicit
--beforeare Loki times and exact. Pass--beforewhen a tailed file shares the server name, because the oldest row may then come from the file. - Selectors that share a server go in one run. If several selectors use the same server name, import them in one command without
--query. Every selector's end is picked before any import starts, so one selector's backfill can't become another's end. Run them one at a time and the second one stops where the first one's import begins. - Overlap detection only knows this command's own imports. The command refuses a range that overlaps an earlier
import-lokirange for the same selector, and names that range. Rows from live reading, fromimport-logs, or from another selector that matches the same streams don't count, so pick--sinceand--beforearound them yourself.--forceskips the check. An unfinished run of the same range still resumes, but a finished range is imported again and the rows from the first import stay, so they end up twice. - A resume repeats one hour. The rows already written from the interrupted hour end up twice. How resume works has the details.
- At most 1000 lines per timestamp. Loki can't page inside a single timestamp. With second-precision entry timestamps, from a pipeline stage that sets the timestamp from the line, the command reads at most 1000 lines from any one second and drops the rest, as live reading does, and logs a warning. Shipping requirements has the stage to remove.
- A wrong format writes nothing. The command detects the format from the first lines it reads, or uses the one pinned in
LOKI_FORMATS. If those lines match no format, or not the pinned one, it rejects the selector before writing anything. - Retention limits both ends. The retention policy deletes imported rows older than
ANALYTICS_RAW_RETENTION_DAYS, like any other raw rows. Access Logs loses them, while the daily aggregates keep their counts. The app applies the setting only when it starts, so raise it in.envand rundocker compose up -dbefore the import. Loki's own retention limits how far back there is anything to read.