Skip to content
4 changes: 2 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1418,11 +1418,11 @@ nuget.org reports 1900-01-01 as the publish time of an unlisted version. For a d

Helm uses each release's `created` time from the repository's `index.yaml`. Releases in cooldown are left out of the index the proxy serves, and their archives aren't served; a release without `created` is let through. Charts pulled over OCI have no publish time and aren't held back.

Cargo, RubyGems, Hex, pub.dev, Conda and Helm check artifact downloads against cooldown, including files cached before the policy changed. When a publish-time lookup fails, an already-cached file is still served; an uncached download fails with 502, or 503 with `Retry-After` when the rate-limit gate holds the request. Conda checks timestamps per file because builds of one version can have different publish times. As in conda, a repodata `timestamp` is read as milliseconds when it is above 253402300799 (the end of year 9999 in seconds) and as seconds otherwise, since some older entries use seconds.
Cargo, RubyGems, Hex, pub.dev, Conda and Helm check artifact downloads against cooldown, including files cached before the policy changed. When a publish-time lookup fails, an already-cached file is still served; an uncached download fails with 502, or 503 with `Retry-After` when the rate-limit gate holds the request. Metadata cached within `metadata_ttl`, or served stale, can predate a release; a download of a version (or, for Conda, a file) it doesn't list fetches it again, as for npm and PyPI, and is decided on what the upstream lists now. If it can't be fetched, the lookup has failed. Helm charts are downloaded by digest and aren't covered by this yet. Conda checks timestamps per file because builds of one version can have different publish times. As in conda, a repodata `timestamp` is read as milliseconds when it is above 253402300799 (the end of year 9999 in seconds) and as seconds otherwise, since some older entries use seconds.

Conda repodata and Helm indexes are parsed once per cached document for download checks. With `cache_metadata` off, the parsed index is kept for one minute, including the fetch, so an install can reuse it across files. Conda streams repodata when its compressed copy exceeds `metadata_max_size`, under the same decompressed size bound as the metadata filter.

RubyGems, Hex and PyPI keep their publish times outside the metadata clients read: the RubyGems versions API, the Hex HTTP API (`upstream.hex_api`) and the PyPI JSON API. The proxy fetches them alongside each compact index, package or simple page it filters, and for PyPI downloads. Cooldown fails closed: when it applies to a package and those times can't be fetched or parsed, or the metadata can't be rewritten, the request fails with 502, or with 503 and `Retry-After` while the upstream is rate limiting the proxy, instead of serving the metadata unfiltered. The same goes for npm, Composer and pub.dev metadata that can't be rewritten, and for npm tarball downloads whose publish time can't be read. A PyPI file or npm tarball that is already cached is still served when its publish time can't be read; only a download that would have to be fetched fails. A PyPI release's time is the upload time of its newest file. The proxy stores it when a download looks it up, and serves files of that release that are already cached against the stored time without reading the JSON API again. A file that isn't cached yet is always looked up, since it may have been added to the release since, and the newer time is stored. Packages exempted with a duration of `0` are served as before and don't need the times. A PyPI package the JSON API returns 404 for has no times to go by and is served unfiltered.
RubyGems, Hex and PyPI keep their publish times outside the metadata clients read: the RubyGems versions API, the Hex HTTP API (`upstream.hex_api`) and the PyPI JSON API. The proxy fetches them alongside each compact index, package or simple page it filters, and for PyPI downloads. Cooldown fails closed: when it applies to a package and those times can't be fetched or parsed, or the metadata can't be rewritten, the request fails with 502, or with 503 and `Retry-After` while the upstream is rate limiting the proxy, instead of serving the metadata unfiltered. The same goes for npm, Composer and pub.dev metadata that can't be rewritten, and for npm tarball downloads whose publish time can't be read. A PyPI file or npm tarball that is already cached is still served when its publish time can't be read; only a download that would have to be fetched fails. A PyPI release's time is the upload time of its newest file. The proxy stores it when a download looks it up, and serves files of that release that are already cached against the stored time without reading the JSON API again. A file that isn't cached yet is always looked up, since it may have been added to the release since, and the newer time is stored. A JSON API document cached before the release doesn't list it, and is fetched again. Packages exempted with a duration of `0` are served as before and don't need the times. A PyPI package the JSON API returns 404 for has no times to go by and is served unfiltered.

For RubyGems and Hex, both documents go through the metadata cache. With [`cache_metadata`](#metadata-caching) they are served from it within `metadata_ttl` without asking the upstream, revalidated with their ETags after it, and filtered from the last cached copies, marked stale, while the upstream is down; without it they are still fetched once for concurrent requests. Each document has its own TTL, so the compact index or Hex package can be newer than the cached times and list a version they don't have. The times are then fetched again before filtering, and if that fails, such versions are left out until the times can be read. The filtered index or package carries an ETag of its own, so Bundler and Mix revalidate it with `304 Not Modified`. Publish times are compared with the current time on each request, and the filtered result is kept until the first version it held back leaves cooldown (see [rewritten metadata cache](#rewritten-metadata-cache)); a newly published version shows up within `metadata_ttl`, as with other cached metadata. Upstream errors are answered as for other cached metadata (404, or 502) rather than relayed.

Expand Down
41 changes: 21 additions & 20 deletions internal/handler/cargo.go
Original file line number Diff line number Diff line change
Expand Up @@ -194,28 +194,29 @@ func (h *CargoHandler) applyCooldownFiltering(downstreamResponse http.ResponseWr

// publishTime reads a crate version's pubtime from the sparse index, sharing
// the index cache with handleIndex. Entries published before crates.io
// recorded pubtime have none.
// recorded pubtime have none. A cached index that doesn't list the version
// is fetched again (readListed).
func (h *CargoHandler) publishTime(r *http.Request, name, version string) (time.Time, bool, error) {
upstreamURL := fmt.Sprintf("%s/%s", h.indexURL, h.buildIndexPath(name))
body, _, err := h.proxy.FetchOrCacheMetadata(r.Context(), "cargo", name, upstreamURL, "text/plain")
if err != nil {
return time.Time{}, false, err
}
for line := range bytes.SplitSeq(body, []byte("\n")) {
if len(bytes.TrimSpace(line)) == 0 {
continue
}
var entry crateIndexEntry
if err := json.Unmarshal(line, &entry); err != nil {
return time.Time{}, false, err
}
if entry.Version != version {
continue
}
published, err := time.Parse(time.RFC3339, entry.PublishTime)
return published, err == nil, nil
}
return time.Time{}, false, nil
listed, err := readListed(r.Context(), h.proxy, "cargo", name, upstreamURL, "text/plain", "",
func(res metadataResult) (listedTime, bool, error) {
for line := range bytes.SplitSeq(res.body, []byte("\n")) {
if len(bytes.TrimSpace(line)) == 0 {
continue
}
var entry crateIndexEntry
if err := json.Unmarshal(line, &entry); err != nil {
return listedTime{}, false, err
}
if entry.Version != version {
continue
}
published, err := time.Parse(time.RFC3339, entry.PublishTime)
return listedTime{published, err == nil}, true, nil
}
return listedTime{}, false, nil
})
return listed.published, listed.ok, err
}

// buildIndexPath builds the sparse index path for a crate name.
Expand Down
72 changes: 62 additions & 10 deletions internal/handler/conda.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"bufio"
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
Expand Down Expand Up @@ -552,25 +553,72 @@ func (h *CondaHandler) metadataCacheKey(urlPath string) string {
}

// publishTime streams compressed repodata when it exceeds metadata_max_size.
// Repodata that doesn't list the file can predate it, so it is fetched
// again and the file looked up in what upstream lists now (readListed).
func (h *CondaHandler) publishTime(r *http.Request, channel, arch, filename string) (time.Time, bool, error) {
repodataPath := "/" + channel + "/" + arch + "/repodata.json"
cacheKey := h.metadataCacheKey(repodataPath)
index, err := publishIndex(h.proxy, "conda\x00"+cacheKey, func() metadataResult {
return h.proxy.lookupMetadata(r.Context(), "conda", cacheKey, h.upstreamURL+repodataPath, "*/*", "gzip", nil)
}, func(res metadataResult) (map[string]time.Time, error) {
upstreamURL := h.upstreamURL + repodataPath
// The index is built once for the downloads joining it, so a client
// that hangs up doesn't fail the others.
ctx := context.WithoutCancel(r.Context())
build := func(res metadataResult) (map[string]time.Time, error) {
if errors.Is(res.err, ErrMetadataTooLarge) {
return h.streamRepodataIndex(r, h.upstreamURL+repodataPath)
return h.streamRepodataIndex(ctx, upstreamURL)
}
if res.err != nil {
return nil, res.err
}
return h.indexRepodata(res.body, res.contentEncoding)
}
// loaded is what the index was built from, unless an index built
// earlier was reused.
var loaded *metadataResult
indexKey := "conda\x00" + cacheKey
index, err := publishIndex(h.proxy, indexKey, func() metadataResult {
res := h.proxy.lookupMetadata(ctx, "conda", cacheKey, upstreamURL, "*/*", "gzip", nil)
loaded = &res
return res
}, build)
if err != nil {
return time.Time{}, false, err
}
published, listed := index[filename]
current := loaded != nil && !loaded.fromCache && !loaded.stale
if !listed && !current {
// A concurrent refetch may have replaced the index read above, so
// the file is looked up in the refetched one.
if recent, ok := recentRefetch(h.proxy, indexKey); ok {
published, listed = recent.(map[string]time.Time)[filename]
current = true
}
}
if listed || current {
// Listed, just read from upstream (streamed past metadata_max_size
// included), or refetched for a missing file within the last
// minute, without a digest to check it is current.
return published, listed && !published.IsZero(), nil
}
// Downloads that found the file missing from the same index share one
// refetch and build, which for repodata past metadata_max_size streams
// it. Later downloads use the refetched index.
refetched, err, _ := h.proxy.sharedProxy().publishIndexes.group.Do(indexKey+"\x00refetch", func() (any, error) {
res := h.proxy.refetchLacking(ctx, "conda", cacheKey, upstreamURL, "*/*", "gzip", nil)
if index, ok := keptPublishIndex(h.proxy, indexKey, res); ok {
return index, nil
}
index, err := build(res)
if err != nil {
return nil, err
}
storePublishIndex(h.proxy, indexKey, res, index, true)
return index, nil
})
if err != nil {
return time.Time{}, false, err
}
published, ok := index[filename]
return published, ok, nil
published = refetched.(map[string]time.Time)[filename]
return published, !published.IsZero(), nil
}

// indexRepodata indexes a cached repodata.json body.
Expand All @@ -587,8 +635,8 @@ func (h *CondaHandler) indexRepodata(body []byte, encoding string) (map[string]t
}

// streamRepodataIndex indexes repodata.json as it streams from upstream.
func (h *CondaHandler) streamRepodataIndex(r *http.Request, upstreamURL string) (map[string]time.Time, error) {
req, err := http.NewRequestWithContext(r.Context(), http.MethodGet, upstreamURL, nil)
func (h *CondaHandler) streamRepodataIndex(ctx context.Context, upstreamURL string) (map[string]time.Time, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, upstreamURL, nil)
if err != nil {
return nil, err
}
Expand All @@ -614,7 +662,7 @@ func (h *CondaHandler) streamRepodataIndex(r *http.Request, upstreamURL string)
}

// readRepodataIndex maps each package file in a repodata.json to its
// timestamp, leaving out files without one. It reads the document as it
// timestamp, or the zero time for a file without one. It reads the document as it
// streams, under the same bound as filterRepodata, so the decompressed size
// is not limited by metadata_max_size.
func readRepodataIndex(r io.Reader) (map[string]time.Time, error) {
Expand All @@ -629,7 +677,11 @@ func readRepodataIndex(r io.Reader) (map[string]time.Time, error) {
return err
}
var e repodataEntry
if json.Unmarshal(raw, &e) == nil && e.Timestamp != 0 {
if json.Unmarshal(raw, &e) != nil {
return nil
}
index[entry.Key] = time.Time{}
if e.Timestamp != 0 {
index[entry.Key] = condaTime(e.Timestamp)
}
return nil
Expand Down
43 changes: 35 additions & 8 deletions internal/handler/cooldown_times.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,24 +17,39 @@ import (
// through.
type publishTimes map[string]time.Time

// lookupPublishTime reads version's publish time for a download. A
// document clients read, cached within metadata_ttl or served stale, that
// doesn't list the version can predate it, so both documents are fetched
// again, as readListed does. One that lists it brought the times document up
// to date already (fetchTimedMetadata).
func (p *Proxy) lookupPublishTime(ctx context.Context, t timedMetadata, version string, listed func(metadataResult) ([]string, error)) (time.Time, bool, bool, error) {
published, ok, withholdUnknown, docs, err := p.readPublishTime(ctx, t, version, listed, false)
if err != nil || ok || withholdUnknown || slices.Contains(docs.versions, version) || !docs.doc.fromCache && !docs.doc.stale {
return published, ok, withholdUnknown, err
}
published, ok, withholdUnknown, _, err = p.readPublishTime(ctx, t, version, listed, true)
return published, ok, withholdUnknown, err
}

func (p *Proxy) readPublishTime(ctx context.Context, t timedMetadata, version string, listed func(metadataResult) ([]string, error), refetch bool) (time.Time, bool, bool, timedDocuments, error) {
var versions []string
docs, err := p.fetchTimedMetadata(ctx, t, func(doc metadataResult) ([]string, error) {
docs, err := p.fetchTimedMetadataFor(ctx, t, func(doc metadataResult) ([]string, error) {
var err error
versions, err = listed(doc)
return versions, err
})
}, refetch)
docs.versions = versions
switch {
case docs.doc.err != nil:
return time.Time{}, false, false, docs.doc.err
return time.Time{}, false, false, docs, docs.doc.err
case docs.times.err != nil:
return time.Time{}, false, false, docs.times.err
return time.Time{}, false, false, docs, docs.times.err
case err != nil:
return time.Time{}, false, false, err
return time.Time{}, false, false, docs, err
}
published, ok := docs.published[version]
withholdUnknown := !ok && docs.withholdUnknown && slices.Contains(versions, version)
return published, ok && !published.IsZero(), withholdUnknown, nil
return published, ok && !published.IsZero(), withholdUnknown, docs, nil
}

// timedMetadata describes metadata whose publish times live in a second
Expand Down Expand Up @@ -62,6 +77,8 @@ type timedDocuments struct {
// doesn't, and that document is a stale copy: those versions are newer
// than the copy, so they are left out rather than let through.
withholdUnknown bool
// versions are the versions doc lists, as readPublishTime read them.
versions []string
}

// fetchTimedMetadata fetches both documents concurrently. The document
Expand All @@ -75,6 +92,12 @@ type timedDocuments struct {
// from a times document read from the cache, the times document is fetched
// again before filtering.
func (p *Proxy) fetchTimedMetadata(ctx context.Context, t timedMetadata, listed func(metadataResult) ([]string, error)) (timedDocuments, error) {
return p.fetchTimedMetadataFor(ctx, t, listed, false)
}

// fetchTimedMetadataFor is fetchTimedMetadata, fetching both documents again
// from upstream when refetch is set (refetchLacking).
func (p *Proxy) fetchTimedMetadataFor(ctx context.Context, t timedMetadata, listed func(metadataResult) ([]string, error), refetch bool) (timedDocuments, error) {
var res timedDocuments
var published publishTimes
validate := func(body []byte) error {
Expand All @@ -86,11 +109,15 @@ func (p *Proxy) fetchTimedMetadata(ctx context.Context, t timedMetadata, listed
}

done := make(chan struct{})
lookup := p.lookupMetadata
if refetch {
lookup = p.refetchLacking
}
go func() {
defer close(done)
res.times = p.lookupMetadata(ctx, t.ecosystem, t.timesKey, t.timesURL, t.timesAccept, "", validate)
res.times = lookup(ctx, t.ecosystem, t.timesKey, t.timesURL, t.timesAccept, "", validate)
}()
res.doc = p.lookupMetadata(ctx, t.ecosystem, t.docKey, t.docURL, "*/*", "identity", nil)
res.doc = lookup(ctx, t.ecosystem, t.docKey, t.docURL, "*/*", "identity", nil)
<-done

if res.doc.err != nil || res.times.err != nil {
Expand Down
Loading
Loading