How to warm the weather cache for an offline run¶

Some environments cannot reach the internet when they run: a hardened CI runner, an air-gapped cluster, a container whose egress is closed by policy. idfkit is built to work in them, but weather files are the one thing it cannot invent, so they have to be put on disk while a network is still available.

This guide covers the build-time warm-up: what to pre-download, where to put it, and the two settings that decide whether the isolated run is silent or spends five minutes discovering it is offline.

Everything here is Python. There is no equivalent procedure in JavaScript because there is nothing there to warm: that library keeps no cache directory, so its offline story is decided at install time rather than at build time.

Python only, permanently

Caching retrieved weather files on disk belongs to Python alone. JavaScript is not waiting on a port and will not gain a counterpart: this is a permanent boundary, so no issue tracks it, and moving the capability out of that state takes a constitutional amendment rather than a ledger edit.

FR-031, stated as a decision rather than a backlog item: the on-disk cache for retrieved weather and design-day files is Python's, and JavaScript MUST NOT grow one. Moving this entry out of never needs a constitutional amendment, not an edit here.

What Python has. WeatherDownloader owns a cache directory, per platform by default and moved with cache_dir or IDFKIT_CACHE_DIR. download() looks in it first and reaches the network only on a miss or past max_age, so the second call for a station costs nothing; clear_cache empties it. Everything downstream follows: the record it returns is paths rather than text, zip_path addresses the archive it kept, and selective extraction through only= is meaningful because a suffix skipped this time may already be sitting there from an earlier call.

Why JavaScript will not have one. @idfkit/weather targets a browser, a worker, and an edge runtime as well as Node. Two of those have no directory it could own, and picking one in Node would make the package behave differently depending on where it runs, which is the one thing a portable package must not do. So it fetches every time and returns text the caller keeps. saveWeatherFiles in @idfkit/weather/node writes a retrieved bundle where the caller says, and that is a caller-directed write rather than a cache: it consults nothing on the way in and remembers nothing on the way out.

This is not a gap in JavaScript, and it is why several one-sided names exist on each side. The station index is shipped, not cached, in both languages (FR-043, FR-075), so nothing here changes how either library finds a station: that is weather-index. Retrieval itself is present in both languages: that is weather-download.

The full entry, including the vocabulary this capability owns, is on the capability parity page.

What needs warming, and what does not¶

Only the weather files need warming. The station index does not.

Asset	Ships with the package	Needs the network
Station index (about 70,000 stations)	yes, as `stations.json.gz`	no
EPW and DDY files	no	yes, once per station
IP geolocation (`ipgeo.json`)	no	yes, and do not warm it: see below
Schemas, validation, introspection, documentation URLs	yes	no

StationIndex.load() reads the bundled index straight off disk. Searching it, resolving a station, and reading its metadata are all local. Only WeatherDownloader.download() reaches for anything.

StationIndex.refresh() is optional

refresh() re-downloads the upstream KML indexes and rebuilds the cached index. It is a freshness update, not a required fetch. A run that never calls it still gets a full station index, just the one that shipped with the installed version of idfkit. Do not put refresh() in a warm-up script unless you specifically want a newer index than the release carries.

1. Choose a cache location the isolated run can read¶

Set IDFKIT_CACHE_DIR to a path you control, in both the warming environment and the isolated one. Without it, idfkit uses a platform location that a build container and its runtime almost never share.

export IDFKIT_CACHE_DIR=/opt/idfkit-cache

An explicit IDFKIT_CACHE_DIR is never silently relocated. If the path turns out to be unwritable, the write fails and names the path you chose, rather than quietly moving your downloads somewhere the later run will not look.

2. Pre-download the stations the run will need¶

from idfkit.weather import StationIndex, WeatherDownloader

# Every station the isolated run will later ask for.
QUERIES = ("chicago ohare", "denver intl ap", "singapore changi")

index = StationIndex.load()
downloader = WeatherDownloader()

for query in QUERIES:
    results = index.search(query)
    if not results:
        # Fail here, where there is still a network to fix the problem with.
        msg = f"no weather station matched {query!r}"
        raise LookupError(msg)
    station = results[0].station
    files = downloader.download(station, only={".epw", ".ddy"})
    print(f"warmed {station.display_name}: {files.epw}")

only={".epw", ".ddy"} skips extracting the STAT, CLM, WEA, PVSYST, and RAIN members of the upstream bundle, none of which a simulation reads. It saves disk, not bandwidth: the archive is downloaded whole either way and kept, so the transfer is identical. For Chicago O'Hare the full extraction occupies 2.9 MB against 2.1 MB for the EPW and DDY alone, the 384 KB archive included in both. Worth having across a few hundred stations, not worth contorting a build for.

Fail the build on a station that does not resolve. A warm-up that quietly skips a miss produces a cache that looks complete and fails hours later, in the one environment that cannot fix it.

3. Turn off the freshness nudge in the isolated run¶

This is the step that is easy to miss, and warming the cache alone does not cover it.

StationIndex.load() fires a throttled freshness check: at most once every 24 hours it sends a HEAD request for each of the 10 upstream index files, warns if the loaded index is behind, and records the check under the cache directory. Offline, every one of those requests fails and is swallowed, so the call still returns a complete index and nothing breaks. What it costs is time. Each request carries a 30 second timeout, so on a network that drops packets rather than refusing them, the first StationIndex.load() of the day can block for up to five minutes before returning a result it already had on disk.

export IDFKIT_NO_WEATHER_UPDATE_CHECK=1

Measured with every socket call failing the way an offline machine fails, so the counts are attempts made rather than requests that happened to succeed:

Configuration	`StationIndex.load()` network attempts
Cold writable cache, nudge on	10
Cold writable cache, `IDFKIT_NO_WEATHER_UPDATE_CHECK=1`	0
Warm cache checked within the last 24 hours	0
Read-only cache	0

The nudge is a convenience for interactive use, where a stale index is worth a warning. In an environment that cannot act on the warning, it buys nothing.

4. Mount the cache read-only, if you like¶

A warm cache mounted read-only is a supported configuration and needs no extra setting. Reads do not require write permission, and the freshness nudge stops on its own: it cannot write its throttle timestamp, so it gives up before reaching for the network. That is why the read-only row above reads 0 without IDFKIT_NO_WEATHER_UPDATE_CHECK being set at all.

Set IDFKIT_NO_WEATHER_UPDATE_CHECK anyway. It makes the intent explicit rather than relying on a permission bit to suppress a network call.

5. Run offline¶

from idfkit.weather import StationIndex, WeatherDownloader

index = StationIndex.load()  # the bundled index, read from disk
station = index.search("chicago ohare")[0].station

files = WeatherDownloader().download(station)  # served from the warm cache
print(files.epw)

A worked example: warming an image at build time¶

The whole procedure in one container. The build has a network, the run does not.

FROM python:3.12-slim

ENV IDFKIT_CACHE_DIR=/opt/idfkit-cache \
    IDFKIT_NO_WEATHER_UPDATE_CHECK=1

RUN pip install --no-cache-dir idfkit

# Warm the cache while the build still has a network.
COPY warm_up.py /tmp/
RUN python /tmp/warm_up.py

COPY run.py /tmp/
CMD ["python", "/tmp/run.py"]

docker build -f Dockerfile.warmup -t idfkit-warmup .
docker run --rm --network none idfkit-warmup

Set both variables in the image rather than at docker run time. They have to apply to the build stage that warms the cache and to the run stage that reads it, and a cache warmed at one path and read from another is the single most common way this goes wrong.

The cache does not need to be writable at run time. Running the same image as a user who cannot write to /opt/idfkit-cache works unchanged, which is what makes a read-only mount or a non-root runtime a supported configuration rather than a thing to work around.

Verify the warm-up before you rely on it¶

Run this while the network is still available. It exercises the same calls the isolated run will make, and a warm cache turns every download() into a disk read.

from pathlib import Path

from idfkit.weather import StationIndex, WeatherDownloader
from idfkit.weather.index import default_cache_dir

QUERIES = ("chicago ohare", "denver intl ap", "singapore changi")

cache: Path = default_cache_dir()
index = StationIndex.load()
downloader = WeatherDownloader()

missing: list[str] = []
for query in QUERIES:
    station = index.search(query)[0].station
    # download() returns from disk when the bundle is already cached, so a warm
    # cache makes this a no-op. Run it while the network is still available and
    # a miss is repairable, never as the isolated run's first act.
    files = downloader.download(station, only={".epw", ".ddy"})
    if files.epw is None or files.ddy is None:
        missing.append(station.display_name)

if missing:
    msg = f"cache at {cache} is missing EPW or DDY for: {', '.join(missing)}"
    raise SystemExit(msg)
print(f"cache at {cache} covers all {len(QUERIES)} stations")

To prove the offline claim rather than assume it, block the network inside the process instead of switching off an interface. Raising from the socket layer shows that no request was even attempted, which is a stronger result than a run that merely happened to succeed:

import socket


def refuse(*args: object, **kwargs: object) -> object:
    raise socket.gaierror(8, "offline")


socket.socket.connect = refuse  # type: ignore[method-assign]
socket.create_connection = refuse  # type: ignore[assignment]
socket.getaddrinfo = refuse  # type: ignore[assignment]

# Everything below this line must still work.

What still needs the network¶

These are the only calls that retrieve, and they fail loudly offline rather than returning something empty:

Call	Offline behaviour
`WeatherDownloader.download()` for a station not in the cache	raises `RuntimeError` naming the URL it could not reach
`StationIndex.refresh()`	raises `RuntimeError` naming the index file it could not fetch
`StationIndex.check_for_updates()`	returns `False`, by design, because a freshness check that cannot reach upstream has learned nothing
`geocode()` and `detect_location()`	raise `GeocodingError`

Geocoding is worth calling out separately. geocode() has no cache, so an isolated run has to resolve its addresses to coordinates before it loses the network, or work from coordinates directly.

Do not warm detect_location()

detect_location() does keep a disk cache, ipgeo.json, alongside the weather files, with a one hour default expiry. Warming it is worse than leaving it cold: it records the build machine's approximate location from its public IP, so a warmed entry would hand the isolated run the coordinates of your CI runner and be believed. If a run needs to know where it is, pass the coordinates in.