Keep DuckDB request timing separate from query time
Use DuckDB HTTP timing alongside query profiling without confusing request duration, cache effects or retries with proof of a network performance bottleneck.
By 
Paul. Architecture and infrastructure
A slow query over remote files leaves several plausible explanations. The query may spend time waiting for remote data, decoding it or doing local computation. DuckDB's HTTP request timing helps narrow that investigation. Its value depends on keeping request measurements separate from the elapsed time of the complete query. A visible remote request is evidence of activity, not proof that networking dominates the delay.
DuckDB 1.4.2 LTS was announced on 12 November 2025 with opt-in logging and profiling improvements. Its release article demonstrates HTTP request durations and access to profiler output through the logger. The release announcement establishes release context; its example timings describe the authors' example, not a general performance promise.
The request timing change has its own chronology. Pull request #19691 was opened on 7 November and merged on 8 November 2025. The demonstration there exposes request type, URL, start time and duration. Development, merge and release are different events. None supplies measurements of the workload you are trying to improve.
Start with the question the measurement can answer
Before enabling more logging, state the uncertainty. If the question is whether a report waits on remote reads, request timing is relevant. If the question is why a local aggregation uses excessive CPU, HTTP events may add little. Choosing the observation from the uncertainty keeps an investigation smaller and makes its result easier to interpret.
For a remote query, record what the complete operation includes. Application elapsed time may cover connection setup, query submission and result handling. A query profiler has its own measurement boundaries. An HTTP duration covers the request scope implemented by that logger. These intervals can be related without being interchangeable. Label them before presenting them in the same diagnosis.
A request duration should not be renamed pure network transfer time. Remote service behaviour and client-side request handling can be relevant to the observed interval. The useful initial statement is that a particular HTTP request took a particular amount of time in the recorded run. Naming its cause requires another step of evidence.
Request counts and elapsed time tell different stories
One query can produce several remote requests. DuckDB's release example includes both metadata and data requests. The published example illustrates why the unit of observation matters: an individual request is smaller than the query whose input it helps obtain.
Do not subtract the sum of request durations from query elapsed time and call the remainder computation. Concurrent requests can overlap, and local work may overlap with remote activity. Even with serial requests, the measurement boundaries need checking. A plausible-looking subtraction can create a false explanation while every original log entry remains accurate.
Retries need similar care. Repeated requests can reflect a retry, but repetition alone does not establish its reason. Query structure, metadata access and repeated file access can also produce multiple events. Preserve enough context to distinguish attempts and outcomes where available. When the log does not provide that distinction, keep it as an unresolved question rather than filling the gap with a story.
Correlation matters before aggregation. If several queries run together, a busy interval can contain HTTP activity from unrelated work. A useful investigation connects the request records to the relevant execution as far as the available identifiers and timing allow. If that connection is uncertain, the conclusion should remain uncertain too.
Cache state can reverse an apparent improvement
A hypothetical comparison might run one query, change its SQL and run it again. If the second run reads data already available through caches, its shorter duration cannot be attributed solely to the SQL change. This is a measurement risk, not a claim that any named DuckDB cache always behaves in a particular way.
Record whether a comparison is intended to represent an initial read or a repeated read. Keep the input files and relevant configuration stable enough to answer that question. A warm repeated query can be the correct workload when that matches real use. It is the silent change of conditions that weakens the comparison, not warmth itself.
Profiler output can then add local execution evidence. The release describes logging profiler information when both facilities are enabled. That makes it possible to examine another scope of work alongside HTTP events. It does not make every profiler metric an elapsed duration or turn a CPU metric into a complete account of waiting.
Observation has its own cost
The strongest reason to resist extra instrumentation is that it can change the workload being observed. Logging produces records; profiling collects additional information. For a short operation, the additional work may matter. An investigation should choose the smallest useful observation and keep the instrumentation setting visible in its comparison.
There are also cases where no HTTP timing is needed. A consistently local workload with a clearly identified expensive operator may be better served by targeted query analysis. Enabling remote-request logs simply because they are available would add material to inspect without reducing uncertainty. Diagnostic tools earn their place through the question they answer.
Where remote latency remains plausible, gather a bounded sample with the relevant conditions recorded. Look for a repeated relationship between delayed requests and slow complete queries. Then choose a change that tests that relationship. A faster result after a controlled change would be evidence to evaluate; the presence of timing fields alone is not such a result.
This approach can guide software maintenance without committing to a rewrite before the bottleneck is understood. Keep the original question, measurement scope and remaining uncertainty together. Request timing is most useful when it reduces the space of explanations enough to justify the next investigation, rather than supplying a premature verdict.