Dependency cooldowns turn you into a free-rider

🔥 Check out this insightful post from Hacker News 📖

📂 **Category**:

✅ **What You’ll Learn**:

Against dependency cooldowns as a response to supply chain attacks

a small house under snow

Dependency cooldowns are
suddenly
in
vogue.
They have quite quickly become widely recommended and it looks like they’re
going to get fast-tracked into the basket of “industry standard best
practices”.

The hope is that by just waiting N days after release before adopting a new
version – instead of adopting it immediately – any surreptitiously inserted
hacks will have been discovered by someone else and the bad release “yanked”
(or removed). And this does look, on the surface, like an effective approach:
most supply-chain attacks are indeed detected within a few days.

But while dependency cooldowns are individually modestly – and only modestly –
beneficial for those observing them, they place substantial costs onto everyone
else. And they don’t address the core issue: publishing and distribution are
different things and it’s not clear why they have to be coupled together.

Dependency cooldowns – the weakness of individual action

Frankly, dependency cooldowns work by free-riding on the pain and suffering of
others. Fundamental in the dependency cooldown plan is the hope that other
people – those who weren’t smart enough to configure a cooldown – serve as
unpaid, inadvertent beta testers for newly released packages. If there’s a
problem, those poor saps get hacked, everyone notices that they got hacked, and
the problematic package/executable is yanked before the dependency cooldowners’
thresholds are reached.

Even if this worked for individuals – I think it’s impossible to sustain it as
a sensible or moral system for the entire ecosystem to observe.

Another issue is that dependency cooldowns require a lot of different people to
do work. Python has multiple package managers at this point (how many now?
8?). All must implement dependency cooldowns. And every project ever created
has to configure the cooldown – which often isn’t particularly easy or clear
given that package managers often choose completely different ways to do
it.

But in fact, even the cooldowners suffer. It’s extremely easy to accidentally
circumvent the cooldown that you have sagely configured in your project file.
In Python, a single, personal, pip install litellm outside your project
config would have recently gotten you
hacked. So the cooldown
approach is not actually complete, nor particularly safe.

At some point, probably through either LLM use or old-fashioned copypasta – for
that is how the majority of project configurations are created – a
“responsible” cooldown then becomes the de facto default. And, to mangle
Greenspun, any sufficiently widespread dependency cooldown becomes an ad-hoc,
informally specified, hole-ridden, slow implementation of an upload queue.

Upload queues – the many upsides of central action

The obvious alternative is that instead of everyone configuring a cooldown
– over and over, again and again – in different package managers and projects
that instead we just do it once, a single time, in the central dependency
server. An “upload queue”. Make new packages wait some period of time after
they are published, before they are distributed.

(I use “publication” here to mean sending the release (tarball, whl, gem,
whatever) to the central index (npm, pypi, rubygems). Conversely,
“Distribution” is when the central index starts serving the release to the
public.)

During the time after publication but prior to distribution, you can run
internal lint tools, make the package available to external automated security
scanners (whose brand names are to be displayed prominently if they find
anything), display a public diff of the changes in the built package and even
make the queued releases available for intentional, explicitly volunteering
beta testers to try out.

Upload queues have precedent. The Debian project uses an upload queue.
Packages are uploaded to the repository and then face a minimum wait of 2-10
days prior to making it into the “testing” distribution. An upload queue
separates package publication and package distribution.

Publishing is making a package and posting it to the repository. Distribution
is when the package is made available to the public. There is no special
reason why these two activities need to happen at the same time – it’s just a
historical accident of how language-specific package indexes grew up.

Upload queues achieve the same goal as dependency cooldowns, but without any of
the problems. Upload queues resolve the free-rider problem: no longer are the
configurationally-challenged used as free guinea pigs for the cooldowners.
Package managers need not implement anything. Projects do not need to add yet
another config option. Security linters do not need to flag it. And even when
accidentally installing a one-off package on a developer laptop without the
configuration, you remain protected.

Removing the element of surprise

Upload queues have another substantial benefit. In the majority of supply
chain attacks it is not just the inserted hacks that were unauthorised – the
whole release was unauthorised. Making published packages sit around for a few
days dramatically reduces the power of release credentials. That’s important
because making something less powerful is a good second to securing it better.

Making published releases sit around for a few days prior to distribution also
removes the entirely unnecessary element of surprise when a new release
appears. It gives users advance notice that a new release is coming and
foreknowledge of exactly when it will be available.

And it’s not just users who need advance knowledge. The upload queue period is
also a good time to notify maintainers, to make sure that they are all indeed
aware of the forthcoming release. “Notification: Release 2.4.1 has entered the
upload queue” would be just the wake-up call required to avert a supply chain
attack getting rolled out in many cases.

This applies more than double for AI

People rarely say it so explicitly but, LLMs mean that markdown is now an
executable file format. Whether that is exciting or terrifying (or both!)
probably comes down to personal taste. But it’s a simple fact; and after all
that’s how Agent Skills work: you download some
markdown file, and now your LLM has a new 3rd party dependency.

The first big supply-chain attack on LLMs is doubtless just a matter of time –
someone inserting their “Disregard
that!” into some popular markdown
file.

I’ve been thinking about this problem for a side project of mine which is a
public memory system for AI agents called
“Soapstones”. It’s a place for them
(meaning AI agents) to record how to do things; like searching for HN posts,
getting the current weather, looking up exchange rates, etc. Soapstones is
effectively a package manager for markdown files. And, because it is, the
supply chain attack issue exists.

In fact it applies double. Not only might people poison the markdown files
with “Disregard that!”‘s but LLMs could foolishly upload secret information to
the system – such as their API keys.

And the solution here, as well, is an upload queue. In fact, a double upload
queue. Moderators review each upload for supply-chain attacks. And the owners
of the agents review each upload to make sure they approve of it too.

I think an upload queue is the only reasonable option here – I can’t just ask
LLMs to “be careful downloading recently uploaded stuff” – that would be
madness.

Is funding a problem? Not obviously

One likely retort is “who will pay for this?”. Firstly, it’s not clear that
huge funding is required. The Debian project has maintained an upload queue
for decades and has a security team who
expedite exceptions for security reasons.

Secondly, it’s not all that clear that every important package index is
actually strapped for cash. NPM, Inc was a venture-funded startup and is now a
wholly owned subsidiary of Microsoft. The Python Software Foundation, who run
PyPI, already have a laundry list of corporate
sponsors many of whom would benefit from
an upload queue. And the PSF even recently took in $1.5m from
Anthropic
for, among other things: supply-chain security.

But another option is to provide expedited security reviews for commercial
projects as a paid service.

Commercial entities are often in a hurry to get a new version out ASAP – to fix
some (non-security) bug that is affecting a customer, to make a big
announcement or just to bring forward some important server-side deprecation.
Simply charge them for expedited review as a paid service.

Expedited review would not be an opt-out from process. It wouldn’t mean the
release gets distributed instantly as soon as the company hits “publish” on
their side – all the automated stuff should still run to completion. Rather, a
manual check simply just means wall-clock time can be reduced substantially for
a specific release.

Package indexes already need security response teams: yanking releases,
maintaining embargoes, dealing with typosquatting and coordinating 0days.
Charging commercial projects for expedited review is a good way to cross-fund
that. Corporate urgency subsidises the security apparatus that serves the
whole ecosystem.

Individually rational, collectively bonkers

Dependency cooldowns are one of those things that serve you somewhat well when
you do it yourself. And we all do it to some extent. For some things, I don’t
want to be the first to upgrade: the family TV set-top box is mission critical
infra in my household.

But it’s qualitatively different to take personal, subjective responses to each
upgrade situation and fossilise them into the community best practices. I
don’t want my security to depend on someone else getting hacked first.


Contact/etc

Please write to me at
cal@calpaterson.com about this
article, especially if you disagreed with it. Conversely, if you liked
it, it helps me a lot if you share it (a friend, a discord, a
subreddit, etc).

See other things I’ve written or learn more about me on my about page.

Get an alert when I write something new by RSS.

I used to have a mailing list, but EmailOctopus incorrectly deleted it. They apologised, but don’t have backups.

I am on:


Notes

Debian stable effectively is an upload queue – the whole point is that it’s
made up of older releases that have already been subject to a QA process. The
upload queue for language-oriented package managers need not be so
comprehensive – but I do think there is something to take from the Debian
example.

One thing that is under-discussed is how good the automated scanners are
getting. One of the key elements of that is that they examine built artefacts
rather than just the upstream source code – which allows them to notice cases
where they differ. Giving scanners more time to run (whether by cooldowns or
by upload queues) appears to be one of the low hanging fruit.

And another is how dangerous Github actions appears to be. A significant
majority of supply-chain attacks appear to rest on exploits to GHA, especially
for open source projects where members of the public are able to open PRs.

If you are interested in Soapstones there is a Discord server you can
join to talk about it.

⚡ **What’s your take?**
Share your thoughts in the comments below!

#️⃣ **#Dependency #cooldowns #turn #freerider**

🕒 **Posted on**: 1776228229

🌟 **Want more?** Click here for more info! 🌟

By

Leave a Reply

Your email address will not be published. Required fields are marked *