ProTip: use PNPM, not NPM.
PNPM 10.x shutdown a lot of these attack vectors.
1. Does not default to running post-install scripts (must manually approve each)
2. Let's you set a min age for new releases before `pnpm install` will pull them in - e.g. 4 days - so publishers have time to cleanup.
NPM is too insecure for production CLI usage.
And of course make a very limited scope publisher key, bind it to specific packages (e.g. workflow A can only publish pkg A), and IP bound it to your self hosted CI/CD runners. No one should have publish keys on their local, and even if they got the publish keys, they couldn't publish from local.
(Granted, GHA fans can use OIDC Trusted Publishers as well, but tokens done well are just as secure)
Npm is what happens when you let tech debt stack up for years too far. It took them five attempts to get lock files to actually behave the way lock files are supposed to behave (lockfile version 3, + at least 2 unversioned attempts before that).
It’s clear from the structure and commit history they’ve been working their asses off to make it better, but when you’re standing at the bottom of a well of suck it takes that much work just to see daylight.
The last time I chimed in on this I hypothesized that there must have been a change in management on the npm team but someone countered that several of the maintainers were the originals. So I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying. There’s just too much stupid there to make it easy.
I’m pretty sure it still cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache. Which means people with shit internet connections and large projects basically waste hours several times a week doing updates that fail.
> I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying.
If they were trying, they'd stop doubling down on sunk costs and instead publicly concede that lock files and how npm-the-tool uses them to attempt to ensure the integrity of packages fetched from npm-the-registry is just a poor substitute for content-based addressing that ye olde DVCS would otherwise be doing when told to fetch designated shared objects from the code repo—to be accompanied by a formal deprecation of npm-install for use in build pipelines, i.e. all the associated user guides and documentation and everything else pushing it as best practice.
npm-install has exactly one good use case: probing the registry to look up a package by name to be fetched by the author (not collaborators or people downstream who are repackaging e.g. for a particular distribution) at the time of development (i.e. neither run time nor build time but at the time that author is introducing the dependency into their codebase). Every aspect of version control should otherwise be left up to the underlying SCM/VCS.
> cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache.
I wonder what circumstances led to saying “this is okay we’ll ship it like that”
I think we can blame the IO streaming API in NodeJS on this. It’s a callback and you just know you got another block. My guess is chunked mode and not checking whether the bytes expected and the bytes received matched.
Not to diminish the facepalm but the blame can be at least partially shared.
Our UI lead was getting the worst of this during Covid. I set up an nginx forward proxy mostly for him to tone this down a notch (fixed a separate issue but helped here a bit as well) so he could get work done on his shitty ISP.
True, and things that manifest only on old/slow hardware or on bad internet are the worst kind for this, since 100% of developers who have any say in the matter would never accept such circumstances at all, so they’re always approaching every issue with multi-gigabit speeds, zero latency, and this year’s $3,000 Mac. “What do you mean the page loads slowly?”
A customer of mine has the implementation of several API endpoints starting with a simulate_slow_connection method that basically sleeps for a random amount of ms. I think it sleeps 0 ms when running tests and definitely sleeps 0 ms in production. So it's never super fast even on $3,000 Macs.
but this stuff is basically solved. We have enough history with languages and distribution of packages, repositories, linux, public trust, signing, maintainers, etc.
One key shift is there is no packager anymore. Its just - trust the publisher.
Any language as big as Node should hire a handful of old unix wizards to teach them the way the truth and the life.
Likely they wouldn’t listen. Modern languages and environments seem intent on reinventing bad solutions to solved problems. I get it if it’s a bunch of kids that have never seen anything better but there is no excuse these days not to have at least a passing knowledge of older systems if you’ve been around a while.
there's certainly a piece of it. Also most seasoned people are not very interested in new languages and environments, and most languages are not 'spec built' by experts like Rob Pike building Go who explicitly set out to solve a lot of his problems, but are more naturally grown and born.
> And of course make a very limited scope publisher key, bind it to specific packages (e.g. workflow A can only publish pkg A), and IP bound it to your self hosted CI/CD runners. No one should have publish keys on their local, and even if they got the publish keys, they couldn't publish from local.
I've by now grown to like Hashicorp Vaults/OpenBao's dynamic secret management for this. It's a bit complicated to understand and get to work at first, but it's powerful:
You mirror/model the lifetime of a secret user as a lease. For example, a nomad allocation/kubernetes pod gets a lease when it is started and the lease gets revoked immediately after it is stopped. We're kinda discussing if we could have this in CI as well - create a lease for a build, destroy the lease once the build is over. This also supports ttl, ttl-refreshes and enforced max-ttls for leases.
With that in place, you can tie dynamically issued secrets to this lease and the secrets are revoked as soon as the lease is terminated or expires. This has confused developers with questionable practices a lot. You can print database credentials in your production job, run that into a local database client, but as soon as you deploy a new version, those secrets are deleted. It also gives you automated, forced database credential rotation for free through the max_ttl, including a full audit log of all credential accesses and refreshes.
I know that would be a lot of infrastructure for a FOSS project by Bob from Novi Zagreb. But with some plugin-work, for a company, it should be possible to hide long-term access credentials in Vault and supply CI builds with dropped, enforced, short-lived tokens only.
As much as I hate running after these attacks, they are spurring interesting security discussions at work, which can create actual security -- not just checkbox-theatre.
I would love to use this (for homelab stuff currently) but I would love a way to have vault/openbao be fully configuration-as-code and version controlled, and only have the actual secret values (those that would not be dynamic) in persistent storage.
Or just 'npm ci' so you install exactly what's in your package-lock.json instead of the latest version bumps of those packages. This "automatic updating" is a big factor in why these attacks are working in the first place. Make package updating deliberate instead of instant or on an arbitrary lag.
> Does not default to running post-install scripts (must manually approve each)
To get equivalent protection, use `--only-binary=:all:` when running `pip install` (or `uv pip install`). This prevents installing source distributions entirely, using exclusively pre-built wheels. (Note that this may limit version ability or even make your installation impossible.) Python source packages are built by following instructions provided with the package (specifying a build system which may then in turn be configured in an idiosyncratic way; the default Setuptools is configured using a Python script). As such, they effectively run a post-install script.
(For PAPER, long-term I intend to design a radically different UI, where you can choose a named "source" for each package or use the default; and sources are described in config files that explain the entire strategy for whether to use source packages, which indexes to check etc.)
> Let's you set a min age for new releases before `pnpm install` will pull them in - e.g. 4 days - so publishers have time to cleanup.
Pip does not support this; with uv, use `--exclude-newer`. This appears to require a timestamp; so if you always want things up to X days old you'll have to recalculate.
> Pip does not support this; with uv, use `--exclude-newer`. This appears to require a timestamp; so if you always want things up to X days old you'll have to recalculate.
I do this by having my shell init do this:
export UV_EXCLUDE_NEWER=$(date -Iu -d "14 days ago")
That’s easy to override if you need to but otherwise seamless.
FWIW, I'd like if these tools had an option to prefer the oldest version satisfying the given constraints (rather than the newest, as it is now — probably still a better default).
There were some recent posts I saw about "dependency cooldowns", which seem to be what you're referring to in item 2. The idea really resonated with me.
That said, I hard pin all our dependencies and get dependabot alerts and then look into updates manually. Not sure if I'm a rube or if that's good practice.
That's good practice. God knows how many times I've been bitten by npm packages breaking on minor or even patch version changes, even when proudly proclaiming to use semver
I'm struggling to understand why Trusted Publishers is any better.
Let's say you have a limited life, package specific scoped, IP CIDR bound publishing key, running on a private GH workflow runner. That key only exists in a trusted clouds secret store (e.g. no one will have access it from their laptop).
Now let's say you're a "trusted" publisher, running on a specific GitHub workflow, and GitHub Org, that has been configured with OIDC on the NPM side. By virtue of simply existing in that workflow, you're now a NPM publisher (run any publish commands you like). No need to have a secret passed into your workflow scope.
If someone is taking over GitHub CI/CD workflows by running `npm i` at the start of their workflow, how does the "Trusted Publisher" find themselves any more secure than the secure, very limited scope token?
Good point, but until many popular packages stop requiring install.sh to operate, you'll still need to allowlist some of them. That is built into the PNPM tooling, luckily :)
Reading through the post it looks like this infects via preinstall?
> The new versions of these packages published to the NPM registry falsely purported to introduce the Bun runtime, adding the script preinstall: node setup_bun.js along with an obfuscated bun_environment.js file.
Pnpm cannot be built without an existing pnpm binary meaning there is no way to bootstrap it from audited source code. Perfect trusting trust attack situation.
Full source bootstrapped NPM with manually reviewed dependencies is the only reasonably secure way to use NodeJS right now.
Bun disables post-install scripts by default and one can explicitly opt-in to trusting dependencies in the package.json file. One can also delay installing updated dependencies through keys like `minimumReleaseAge`. Bun is a drop-in replacement for the npm CLI and, unlike pnpm, has goals beyond performance and storage efficiency.
yes bun does both of the things mentioned in the parent comment:
> Unlike other npm clients, Bun does not execute arbitrary lifecycle scripts like postinstall for installed dependencies. Executing arbitrary scripts represents a potential security risk.
> To protect against supply chain attacks where malicious packages are quickly published, you can configure a minimum age requirement for npm packages. Package versions published more recently than the specified threshold (in seconds) will be filtered out during installation.
As far as I can understand from the documentation, that doesn't actually specify in that config that one of them is required, does it? That is, if they _all_ fail to install as far as the system is concerned there's nothing wrong? There will be runtime errors of course, but that's sort of disappointing…
Do keep in mind that the binaries are still binaries. Even if your installation process doesn't run any untrusted code from the package, you can't audit the binaries like you might the .js files prior to first run.
NPM was never "too insecure" and remains not "too insecure" today.
This is not an issue with npm, JavaScript, NodeJS, the NodeJS foundation or anything else but the consumer of these libraries pulling in code from 3rd parties and pushing it to production environments without a single review. How this still fly today, and have been since the inception of public "easy to publish" repositories remains a mystery to me even today.
If you're maintaining a platform like Zapier, which gets hacked because none of your software engineers actually review the code that ends up in your production environment (yes, that includes 3rd party dependencies, no matter where they come from), I'm not sure you even have any business writing software.
The internet been a hostile place for so long, that most of us "web masters" are used to it today. Yet it seems developers of all ages fall into the "what's the worst that can happen?" trap when pulling in either one dependency with 10K LoC without any review, or 1000s of dependencies with 10 lines each.
Until you fix your processes and workflows, this will continue to happen, even if you use pnpm. You NEED to be responsible for the code you ship, regardless of who wrote it.
They didn't deploy the code. That's not how this exploit works. They _downloaded_ the code to their machine. And npm's behavior is to implicitly run arbitrary code as part of the download - including, in this case, a script to harvest credentials and propagate the worm. That part has everything to do with npm behavior and nothing to do with how much anybody reviewed 3P deps. For all we know they downloaded the new version of the affected package to review it!
If people stop running install scripts, isn't Shai-Hulud 3: Electric Boogaloo just going to be designed to run its obfuscated malware at runtime rather than install time? Who manually reviews new versions of their project dependencies after installing them but before running them?
GP is correct. This is a workflow issue. Without a review process for dependencies, literally every package manager I know of is vulnerable to this. (Yes, even Maven.)
> If people stop running install scripts, isn't Shai-Hulud 3: Electric Boogaloo just going to be designed to run its obfuscated malware at runtime rather than install time?
Many such forms of malware have already been published and detected.
> Who manually reviews new versions of their project dependencies after installing them but before running them?
One person putting in this effort can protect everyone thereafter.
The PyPI website has a "Report project as malware" button on each project page for this purpose.
But yes, this is the world we live in. Without this particular form of insecurity, there is no "ecosystem" at all.
wait, I short-circuited here. wasn't the very concept of "libraries" created to *not* have to think about what exactly the code does?
imagine reviewing every React update. yes, some do that (Obsidian claims to review every dependency, whether new or an update), but that's due to flaws of the ecosystem.
take a look at Maven Central. it's harder to get into, but that's the price of security. you have to verify the namespace so that no one will publish under e.g. `io.gitlab.bpavuk.` namespace unless they have access to the `bpavuk` GitLab group or user, or `org.jetbrains.` unless they prove the ownership of the jetbrains.com domain.
Go is also nice in that regard - you are depending on Git repositories directly, so you have to hijack into the Git repo permissions and spoil the source code there.
> Go is also nice in that regard - you are depending on Git repositories directly, so you have to hijack into the Git repo permissions and spoil the source code there.
That in itself is scary because Git refs are mutable. Even with compromised credentials, no one can replace artifacts already deployed to Maven Central, because they simply don't allow it. There is nothing stopping someone from replacing a Git tag with one that points to compromised code.
The surface area is smaller because Go does locking via go.sum, but I could certainly see a tired developer regenerating it over the most strenuous of on-screen objections from the go CLI.
I do `cargo vendor` sometimes, but that's mostly to enable offline work and use the debugger inside some vague crates (Rust's libraries-but-not-really), and usually I gitignore the `vendor`'ed crates away.
> wasn't the very concept of "libraries" created to not have to think about what exactly the code does?
Let's say you need a FFT implementation. You can write that from scratch, or you can use a library. In both cases you should use tests to verify that the code calculates the FFT correctly, and in the library case you should read the code to make sure that it works correctly and does not omit and edge cases (e.g.).
> wasn't the very concept of "libraries" created to not have to think about what exactly the code does?
If you care about security, you only have to care once, during the audit. And you can find a pretty high percentage of malware in practice without actually having a detailed understanding of the non-malicious workings of the code.
Libraries allow you to not think about what the code does at development time, which in general is much more significant than audit time. Also, importantly, they allow you not to have to design and write that part of the code.
well, if you talk about private SSH keys for Git operations and SSH/GPG keys for signing, then you'd better set up a passphrase on them, which GitHub strongly recommends. the passphrase will make it significantly harder to use the keys. so, as usual, It Depends (c)
“Personally, I never wear a seatbelt because all drivers on the road should just follow the road rules instead and drive carefully.”
I don’t control all the drivers on the road, and a company can’t magically turn all employees into perfect developers. Get off your high horse and accept practical solutions.
> and a company can’t magically turn all employees into perfect developers
Sure, agree, that's why professionals have processes and workflows, everyone working together to build the greatest stuff you can.
But when not a single person in the entire company reviews the code that gets deployed and run by users, you have to start asking what kind of culture the company has, it's borderline irresponsible I'd say.
Or they have different priority structures where this isn’t the developers job. Also things get missed sometimes no matter how good your processes are. But improving those processes could involve something like avoiding npm over pnpm.
The "use cooldown" [0] blog post looks particularly relevant today.
I'd argue automated dependency updates pose a greater risk than one-day exploits, though I don't have data to back that up. That's harder to undo a compromised package already in thousands of lock files, than to manually patch a already exploited vulnerability in your dependencies.
Why not take it further and not update dependencies at all until you need to because of some missing feature or systems compatibility you need? If it works it works.
The arguments for doing frequent releases partially apply to upgrading dependencies. Upgrading gets harder the longer you put it off. It’s better to do it on a regular schedule, so there are fewer changes at once and it preserves knowledge about how to do it.
There's another variable, though, which is how valuable "engineering time now" is vs. "engineering time later."
Certainly, having a regular/automated update schedule may take less clock time in total (due to preserved knowledge etc.), and incur less long-term risk, than deferring updates until a giant, risky multi-version multi-dependency bump months or years down the road.
But if you have limited engineering resources (especially for a bootstrapped or cost-conscious company), or if the risks of outages now are much greater than the risks of outages later (say, once you're 5 years in and have much broader knowledge on your engineering team), then the calculus may very well shift towards freezing now, upgrading later.
And in a world where supply chain attacks will get far more subtle than Shai-Hulud, especially with AI-generated payloads that can evolve as worms spread to avoid detection, and may not require build-time scripting but defer their behavior to when called by your code - macro-level slowness isn't necessarily a bad thing.
(It should go without saying that if you choose to freeze things, you should subscribe to security notification services that can tell you when a security update does release for a core server-side library, particularly for things like SQL injection vulnerabilities, and that your team needs the discipline to prioritize these alerts.)
At the same time, unplanned engineering time is almost always more expensive than planned engineering time. I'd rather have some regular, expected, upgrade work, than all of a sudden having to scramble because I need something at a moment when I didn't plan for that.
When you do regular updates, they are quick. But also, you can timebox it and then back out and plan a harder than expected update.
Also, it keeps you in touch with your deps so you can consider if it’s even worth it. My favorite update was removing the dep (or starting a plan to remove it because it was interfering with regular updates)
There is a Goldilocks effect. Dependency just came out a few minutes ago? There is no time for the community to catch the vulnerability, no real coverage from dependency scans, and it's a risk. Dependency came out a few months ago? It likely has a large number of known vulns
That is indeed what one should do IMO. We've known for a long time now in the ops world that keeping versions stable is a good way to reduce issues, and it seems to me that the same principle applies quite well to software dev. I've never found the "but then upgrading is more of a pain" argument to be persuasive, as it seems to be equally a pain to upgrade whether you do it once every six months or once every six years.
The 'pain' comes from breaking changes, at worst if you delay you're going to ingest the same quantity of changes, and at best you might skip some short-lived ideas.
> Why not take it further and not update dependencies at all until you need to because of some missing feature or systems compatibility you need? If it works it works.
Indeed there are people doing that and communities with a consensus such approach makes sense, or at least is not frowned upon. (Hi, Gophers)
This only makes sense for vulnerabilities that can actually be exploited in your particular use-case and configuration of the library. A lot of vulns might be just noise and not exploitable so no need to patch.
Problem is code bases are continuously evolving. A safe decision now, might not be a safe decision in the future. It's very easy to accidentally introduce a new code path that does make you vulnerable.
Because updates don't just include new features but also bug and security fixes. As always, it probably depends on the context how relevant this is to you. I agree that cooldown is a good idea though.
> Because updates don't just include new features but also bug and security fixes.
This practice needs to change, although it will be almost impossible to get a whole ecosystem to adopt. You shouldn’t have to take new features (and associated new problems) just to get bug fixes and security updates. They should be offered in parallel. We need to get comfortable again with parallel maintenance branches for each major feature branch, and comfortable with backporting fixes to older releases.
I maintain both commercial and open source libs. This is a non starter in both cases. It would easily double if not triple the workload.
For open source, well these are volunteer projects on my own time, you are always welcome to fork a given version and backport any fixes that land on main/master.
For commercial libs, our users are not willing to pay extra for this service, so we don't provide it. They would rather stay on an old version and update the entire code base at given intervals. Even when we do release patch versions, there is surprisingly little uptake.
Semver doesn't help. The primary issue is effort. If it's an open source project with 1-2 devs, they probably won't be able to handle supporting multiple branches unless they're being paid to do this.
Of course it doesn't provide backports by itself, it's a versioning system. But version number changes with SemVer are meant to indicate whether an update includes new fearhews or not (minor bump means new features, patch bump means bugfixes only).
Of course, the actual issue is that maintaining backports isn't free, so expecting it from random single-person projects is a little unrealistic. Bug fixes in new code often need to be rewritten to work on old code. I do maintain old release branches for some projects and backporting single patches can cause whole new bugs that were never present in the main branch quite easily.
IMO for “boring software” you usually want to be on the oldest supported main/minor version, keeping an eye on the newest point version. That will have all the security patches. But you don't need to take every bug fix blindly.
CI fights this. But that’s peanuts compared to feature branches and nothing compared to lack of a monolith.
We had so many distinct packages on my last project that I had to massively upgrade a tool a coworker started to track the dependency tree so people stopped being afraid of the release process.
I could not think of any way to make lock files not be the absolute worst thing about our entire dev and release process, so the handful of deployables had a lockfile each that was only utilized to do hotfix releases without changing the dep tree out from underneath us. Artifactory helps only a little here.
Just make sure to update when new CVEs are revealed.
Also, some software are always buggy and every version is a mixed bag of new features, bugs and regressions. It could be due to the complexity of the problem the software is trying to solve, or because it's just not written well.
But even then you are still depending on others to catch the bugs for you and it doesn't scale: if everybody did the cooldown thing you'd be right back where you started.
I don't think that this Kantian argument is relevant in tech. We've had LTS versions of software for decades and it's not like every single person in the industry is just waiting for code to hit LTS before trying it. There are a lot of people and (mostly smaller) companies who pride themselves on being close to the "bleeding edge", where they're participating more fully in discovering issues and steering the direction.
The assumption in the post is that scanners are effective at detecting attacks within the cooldown period, not that end-device exploitation is necessary for detection.
(This may end up not being true, in which case a lot of people are paying security vendors a lot of money to essentially regurgitate vulnerability feeds at them.)
To find a vulnerability, one does not necessarily deploy a vulnerable version to prod. It would be wise to run a separate CI job that tries to upgrade to the latest versions of everything, run tests, watch network traffic, and otherwise look for suspicions activity. This can be done relatively economically, and the responsibility could be reasonably distributed across the community of users.
It does scale against this form of attack.
This attack propagates by injecting itself into the packages you host. If you pull only 7d after release you are infected 7d later. If your customers then also only pull 7d later they are pulling 14d after the attack has launched, giving defenders a much longer window by slowing down the propagation of the worm.
That worried me too, a sort of inverse tragedy of the commons. I'll use a weeklong cooldown, _someone else_ will find the issue...
Until no-one does, for a week. To stretch the original metaphor, instead of an overgrazed pasture, we grow a communally untended thicket which may or may not have snakes when we finally enter.
I don't buy this line of reasoning. There are zero/one day vulnerabilities that will get extra time to spread. Also, if everyone switches to the same cooldown, wouldn't this just postpone the discovery of future Shai-Huluds?
I guess the latter point depends on how are Shai-Huluds detected. If they are discovered by downstreams of libraries, or worse users, then it will do nothing.
There are companies like Helix Guard scanning registries. They advertise static analysis / LLM analysis, but honeypot instances can also install packages & detect certain files like cloud configs being accessed
Your line of reasoning only makes sense if literally almost all developers in the world adopt cooldowns, and adopt the same cooldown.
That would be a level of mass participation yet unseen by mankind (in anything, much less something as subjective as software development). I think we're fine.
For zero/one days, the trick is that you'd pair dependency cooldowns with automatic scanning for vulnerable dependencies.
And in the cases where you have vulnerable dependencies, you'd force update them before the cooldown period had expired, while leaving everything else you can in place.
> Note that previous stable versions will not be suggested. The package will be completely ignored if its latest published version is within the cooldown period.
I could see it being a good feature. If there have been two versions published within the last week or two, then there are reasonable odds that the previous one had a bug.
co-founder of PostHog here. We were a victim of this attack. We had a bunch of packages published a couple of hours ago. The main packages/versions affected were:
- posthog-node 4.18.1, 5.13.3 and 5.11.3
- posthog-js 1.297.3
- posthog-react-native 4.11.1
- posthog-docusaurus 2.0.6
We've rotated keys and passwords, unpublished all affected packages and have pushed new versions, so make sure you're on the latest version of our SDKs.
We're still figuring out how this key got compromised, and we'll follow up with a post-mortem. We'll update status.posthog.com with more updates as well.
You're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run.
Or require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing.
The basic issue with manual interaction is a question of authority: a pretty common problem for companies (and open source groups) is when $EARLY_EMPLOYEE/$EARLY_CONTRIBUTOR creates and owns the entire publishing process for a key package, and then leaves without performing a proper transfer of responsibility. This essentially locks the company/group out of its own work, and increases support load on community maintained indices to essentially adjudicate rightful ownership of the package name.
(There are a variety of ways to solve this, but the one I like best is automated publishing a la Trusted Publishing with environment mediated manual signoffs. GitHub and other CI/CD providers enable this.)
I don’t buy this. I mean, I’m sympathetic to the issue. It’s easy for things to get jumbled when companies are young. But we’re talking about libraries that are used by your customers. To many, this is the externally visible interface to the company.
What you describe sounds like a process problem to me. If an $EARLY_EMPLOYEE if the only one with the deploy keys for what is a product of the company, then that’s a problem. If a deployment of that key library can be made without anyone approving it, that’s also a problem. But those are both people problems… and you can’t solve a people problem with a technical solution.
> But those are both people problems… and you can’t solve a people problem with a technical solution.
I don’t think it is a people problem in this case: the only reason there’s a person involved at all is because we’ve decided to introduce one as an intermediating party. A more misuse-resistant scheme disintermediates the human, because the human was never actually a mandatory part of the scheme.
The person who intermediates the trust relationship between the index and the source repository. There’s no reason for the credential that links those two parties to be intermediated by a human; they’re two machine services talking.
(You obviously can’t disintermediate the human from maintenance or development!)
You’re saying that whatever is in the source repository should be uploaded in the npm index, right? If the code is tagged as release, the built artifact is automatically uploaded to npm. Is that what you’re proposing?
That exactly what got PostHog into this position. The keys to publish to npm were available to an engineer or GitHub to push a malware build into npm automatically. This isn’t a technical issue… it’s a process issue. I don’t see the problem as that the keys were misused. I see the problem as that it was possible to misuse the keys at all. Why do you need that process to be automatic? How often are you pushing new updates?
I would argue that those npm assets/libraries are your work product. That is what your customer needs to use your service. It is a published product from your company. It is too important to allow a new version to be published out to the public without a human in the loop to approve it.
When you have a fully automatic publishing cycle, you’re trading security for convenience. It’s all about how much risk you’re willing to accept. For me, that’s too much of a risk to the reputation to the company. I also think the balance shifts if you’re talking about purely internal assets, having completely automatic ci/cd makes perfect sense for most companies. For me, it is about who is hurt if there is an issue (and you should expect for there to be an issue).
Putting a person in the loop for releasing a product is one way to solve this. It’s not perfect, but at the moment, I think it’s the most secure (for the public).
> You’re saying that whatever is in the source repository should be uploaded in the npm index, right? If the code is tagged as release, the built artifact is automatically uploaded to npm. Is that what you’re proposing?
No, I'm saying that the source repository should act as an authentication principal itself. A human should still initiate the release process, but the authentication process that connects the source repository (more precisely CI/CD) to the index should not involve a credential that's implicitly bound to a human identity (because the human's role within a project or company is ephemeral).
As far as I can tell, what got PostHog into this situation wasn't a fully automated release process (very few companies/groups have fully automated processes), but the fact that they had user-created long-lived credentials that an attacker could store and weaponize at a time most convenient to them. That's a problem regardless of whether there's normally a human in the loop or not, because the long-lived credential itself was sufficient for publishing.
(In other words, we basically agree about human approval being good; what I'm saying is that we should formalize human approval without making the authentication scheme explicitly require an intermediating partner who doesn't inherently represent the actual principal, i.e. the source repository.)
I think we agree more than we don’t and the rest are personal preferences and policy differences. But we largely agree in principle.
I like the idea of having a person whose job is approving releases. Kind of like a QC tag — this release was approved by XX. I saw the issue as PostHog having a credential available to the CI/CD that had the authority to push releases automatically. When a new GitHub action was added, that credential was abused to push a bad update to npm. I might be wrong, I don’t deal with npm that much.
You can't "require" manual intervention. Sure you can say that the keys stays on say 2 developers laptops, but personal devices have even more surface area for key leak than CI/CD pipeline. It wouldn't have prevented attacks like this issue in any case where the binary just searched for keys across the system.
One alternative is to do the signing on airlocked system stored in physically safe but accessible location, but I guess that's just way too much inconvenience.
This is orthogonal to the issue at hand. The problem is a malicious actor cutting a release outside of the normal release process. It doesn't matter if the normal process is automated or manual.
It could have eliminated an attack surface where they steal the credentials from the CI/CD...
...But then you if I understand NPM publishing well, you would still have the credentials on someone's computer laying around? I guess you could always revoke the tokens after publishing? It's all balancing convenience and security, with some options being bad at both?
Did the client side JS being infected produce any issues which would have affected end users? As in if a web owner were on an affected version and deployed during the window would the end user of their site have had any negative impact?
No, just the host that was running the package (the exploit was pretty generic and not targeted at PostHog specifically). In fact, so far we think there were 0 production deployments of PostHog because the package was only live for a little bit.
If anything people should use an older version of the packages. Your newest versions had just been compromised, why should anyone believe this time and next time it will be different?!
OIDC is not a silver bullet either and has its own set of vectors to consider too. If it works for your org model then great, but it doesn't solve every common scenario.
Trusted Publishing addresses the vector here, which is arbitrary persistence and delayed use of credentials by attackers. You're right that it's not a silver bullet (anything claiming to be one is almost certainly a financially induced lie), but it eliminates/foreshortens the attack staging window significantly.
It's not a silver bullet in that it can open up other vectors of much more probable attack.
If I have a FOSS project where there are other contributors with write access but not publish access (very common) OIDC now allows them to publish via workflows, opening you up to Jia Tan type attacks.
Like I said, if it works for your org, great. Otherwise it can cause bigger problems.
Sorry, what's the threat model in which you're giving people write access to your repository but assuming that they can't publish?
To establish a baseline here: the assumption with Trusted Publishing is that people are already using CI/CD to publish to places like PyPI. All Trusted Publishing does is replace the user-managed credential with a machine-intermediated one.
In other words: if someone had write access to your repository before, Trusted Publishing does not change their privilege, since they could always push a new CI/CD workflow that uses your manual publishing secret directly. All Trusted Publishing does is formalize the treatment of the source repository as a source of truth, and introduce some misuse-resistance properties (short-lived credentials, minimal self-scoping).
If this isn't your user model, that's perfectly fine. But in that case you shouldn't be using CI/CD to publish at all (which is also fine); it's not an issue with Trusted Publishing per se.
> Sorry, what's the threat model in which you're giving people write access to your repository but assuming that they can't publish?
So that I'm the arbiter of what people download, and can review locally what changes were made prior to pushing it out to everyone.
> To establish a baseline here: the assumption with Trusted Publishing is that people are already using CI/CD to publish to places like PyPI.
That's not the narrative. People have been saying to switch to trusted publishing from local publishes. They're not 1:1 compatible. I agree that if you already publish via CD, then it probably already worked for your org and thus is a no brainer.
> If this isn't your user model, that's perfectly fine. But in that case you shouldn't be using CI/CD to publish at all (which is also fine); it's not an issue with Trusted Publishing per se.
As a user of Posthog, this statement is absurd:
> Or safer again not to use software this vulnerable.
Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it.
Popularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive?
Glad you updated on this front-page post. Your Twitter post is buried on p3 for me right now. Good luck on the recovery and hopefully this helps someone.
Serious question: should someone develop new technologies using Node any more?
A short time ago, I started a frontend in Astro for a SaaS startup I'm building with a friend. Astro is beautiful. But it's build on Node. And every time I update the versions of my dependencies I feel terrified I am bringing something into my server I don't know about.
I just keep reading more and more stories about dangerous npm packages, and get this sense that npm has absolutely no safety at all.
It's not "node" or "Javascript" the problem, it's this convenient packaging model.
This is gonna ruffle some feathers, but it's only a matter of time until it'll happen on the Rust ecosystem which loves to depend on a billion subpackages, and it won't be fault of the language itself.
The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.
I've worried about this for a while with Rust packages. The total size of a "big" Rust project's dependency graph is pretty similar to a lot of JS projects. E.g. Tauri, last I checked, introduces about 600 dependencies just on its own.
Like another commenter said, I do think it's partially just because dependency management is so easy in Rust compared to e.g. C or C++, but I also suspect that it has to do with the size of the standard library. Rust and JS are both famous for having minimal standard libraries, and what do you know, they tend to have crazy-deep dependency graphs. On the other hand, Python is famous for being "batteries included", and if you look at Python project dependency graphs, they're much less crazy than JS or Rust. E.g. even a higher-level framework like FastAPI, that itself depends on lower-level frameworks, has only a dozen or so dependencies. A Python app that I maintain for work, which has over 20 top-level dependencies, only expands to ~100 once those 20 are fully resolved. I really think a lot of it comes down to the standard library backstopping the most common things that everybody needs.
So maybe it would improve the situation to just expand the standard library a bit? Maybe this would be hiding the problem more than solving it, since all that code would still have to be maintained and would still be vulnerable to getting pwned, but other languages manage somehow.
I wouldn't call the Rust stdlib "small". "Limited" I could agree with.
On the topics it does cover, Rust's stdlib offers a lot. At least on the same level as Python, at times surpassing it. But because the stdlib isn't versioned it stays away from everything that isn't considered "settled", especially in matters where the best interface isn't clear yet. So no http library, no date handling, no helpers for writing macros, etc.
You can absolutely write pretty substantial zero-dependency rust if you stay away from the network and async
Whether that's a good tradeoff is an open question. None of the options look really great
It seems like the normal and rational choice to not include logging to me.
Java included 'java.util.logging'. No one uses it because it's bad, everyone just uses slf4j as the generic facade instead, and then some underlying implementation.
Golang included the "log" package, which was also a disaster. No one used it so they introduced log/slog recently, and still no one uses that one, everyone is still using zap, logrus, etc.
Golang is also a disaster in that the stdlib has "log/slog" as the approved logging method, but i.e. the 'http.Server.ErrorLog' is the old 'log.Logger' you're not supposed to use, and for backwards compatibility reasons it's stuck that way. The go stdlib is full of awkward warts because it maintains backwards compatibility, and also included a ton of extra knobs that they got wrong.
The argument of structured vs unstructured logging, arguments around log levels (glog style numeric levels, trace/debug/info/warn/error/critical or some subset, etc), how to handle lazy logging or lazy computations in logging, etc etc, it's all enough that the stdlib can't easily make a choice that will make everyone happy.
Rust as a language has provided enough that the community can make a sl4j like facade (conveniently, the clearly named 'log' crate seems to be the winner there), so there's no reason it has to be in the stdlib.
> Golang included the "log" package, which was also a disaster. No one used it so they introduced log/slog recently, and still no one uses that one, everyone is still using zap, logrus, etc.
I honestly don't know how common it is to use zap, logrus and co on new projects anymore. I know I always go for slog.
> Golang is also a disaster in that the stdlib has "log/slog" as the approved logging method, but i.e. the 'http.Server.ErrorLog' is the old 'log.Logger' you're not supposed to use, and for backwards compatibility reasons it's stuck that way. The go stdlib is full of awkward warts because it maintains backwards compatibility, and also included a ton of extra knobs that they got wrong.
This isn't a practical problem at all. First of all, the stdlib shouldn't make backwards-incompatible changes just for the sake of logging. Second, slog.NewLogLogger()[0] exists as a way to get log.Loggers to be used with things that rely on the old log.Logger.
Python includes logging, and packages generally use it. It's far from perfect, but it's important to have at least a standard interface so that all components use it in concert. When you have a proliferation of logging libraries with no standard, other packages either don't do any logging at all, or do it in idiosyncratic ways that are hard (or impossible) to integrate.
Rust's standard library hasn't received any major additions since 1.0 in 2015, back when nobody was writing web services in Rust so no one needed logging.
Clap went through some major redesigns with the 4.0 release just three years ago. That wouldn't have been possible if clap 2.0 or 3.0 had been added to the stdlib. It's almost a poster child for things where libraries where being outside the stdlib allows interface improvements (date/time handling would be the other obvious example).
Rand has the issue of platform support for securely seeding a secure rng, and having just an unsecure rng might cause people to use it when they really shouldn't. And serde is near-universal but has some very vocal opponents because it's such a heavy library. I have however often wished that num_traits would be in the stdlib, it really feels like something that belongs in there.
FWIW, there is an accepted proposal (https://github.com/rust-lang/libs-team/issues/394) to add random number generation to std, and adding traits like in `num-traits` is wanted, but blocked on inherent traits.
The std has stability promises, so it's prudent to not add things prematurely.
Go has the official "flag" package as part of the stdlib, and it's so absolutely terrible that everyone uses pflag, cobra, or urfave/cli instead.
Go's stdlib is a wonderful example of why you shouldn't add things willy-nilly to the stdlib since it's full of weird warts and things you simply shouldn't use.
> and it's so absolutely terrible that everyone uses pflag, ../
This is just social media speak for inconvenient in some cases. I have used flag package in lot of applications. It gets job done and I have had no problem with it.
> since it's full of weird warts and things you simply shouldn't use.
The only software that does not have problem is thats not written yet. This is the standard one should follow then.
That's pulling on a dependency. An ugly case, but a very specific one relating to a tricky backwards-incompatible change that the Go team fudged on back in the pre-module times.
That story doesn't generally speak to the reality that most Go packages have rather small dependency graphs.
Clap is enormous and seems way too clever for everything I do. Last I looked it added 10+ seconds to compile time and hundreds of kbs to binary size. Maybe something like ffmpeg requires that complexity, but if I am writing a CLI that takes three arguments, it is a heavy cost.
I honestly feel like that's one of Rust's biggest failings. In my ideal world libstd would be versioned, and done in such a way that different dependencies could call different versions of libstd, and all (sound/secure) versions would always be provided. E.g. reserve the "std" module prefix (and "core", and "alloc"), have `cargo new` default to adding the current std version in `cargo.toml`, have the prelude import that current std version, and make the module name explicitly versioned a la `std1::fs::File`, `std2::fs::File`. Then you'd be able to type `use std1::fs::File` like normal, but if you wanted a different version you could explicitly qualify it or add a different `use` statement. And older libraries would be using older versions, so no conflicts.
I'm afraid it won't work. The point of std lib is to be universal connection for all the libraries. But with versioned std I just can't see how can you have DateTime in std1, DateTime in std2 and use them interchangeably, for example being able to pass std2::DateTime to library depending on std1 etc. Maybe conversion methods, but it get really complicated really quickly
My personal experience (YMMV): Rust code takes 2x or 3x longer to write than what came before it (C in my case), but in the end you usually get something much more likely to work, so overall it's kind of a wash, and the product you get is better for customers - you basically front load the cost of development.
This is terrible for people working in commercial projects that are obsessed with time to market.
Rust developers on commercial projects are under incredible schedule pressure from day 0, where they are compared to expectations from their previous projects, and are strongly motivated to pull in anything and everything they can to save time, because re-rolling anything themselves is so damn expensive.
I think they were using "writing Rust" in the most strict sense: the part of the development cycle that involves typing the majority of the code, before you really start debugging in earnest and really make things work.
But their point is that "developing Rust" (as in, the entire process) ends up being a similar total effort to C, only with more up front "writing" and less work on the debugging phase.
Thank you for the clarification, that's exactly what I was trying to say :).
Perhaps another way to phrase this: in Rust, you spend more time telling the compiler how your code is expected to work (making the borrow checker happy, adding sync traits on objects you "know" are thread safe because of how you use them or assurances the underlying hardware provides, etc etc etc). In return, the compiler does a lot of work to make sure the code will actually work how you think it's going to work.
A good example is a simple producer-consumer problem. Make a ring buffer. Push the current sys clk tick count register to the ring buffer every time there's a rising edge interrupt on a GPIO (e.x. hook up a button or something to it). Poll the ring buffer in another thread and as soon as there is a timestamp in the buffer, pop it and log it to a UART.
Compare writing this in C vs Rust. In the text book implementation of a producer-consumer ring buffer, you don't need locks.
In C, this is a problem I'd expect a senior-level candidate to be able to knock out in a 60 minute interview.
(aside: I'd never actually ask a question like this in an interview, because it heavily favors people who just happen to be familiar with the algorithm, which doesn't give me enough signal about the candidate. This is borderline like asking someone to implement a sort algorithm in an interview - it just tells you they know how to google or memorize things. /aside).
(aside 2: If I did ask this, I'd be more interested how they design the thing - I'd be looking for questions like "how many events per second? How should I size the ring buffer? Wait, you want me to hook up an IRQ to a button? Is there a risk of interrupt storm from bounce or someone being malicious? Do you want me to add a cooldown timer between events? If we overflow the ring buffer, what should the behavior be? How fast can the UART go on this system - can it even keep up with the input?" - I'd be far more interested in that conversation than actually seeing them write code for this. /aside2).
In Rust, it's a bit more tricky. You'll need to give the compiler hints (in the form of Sync traits that are no-ops) to tell it you know what you're doing is thread safe. It's not rocket science, but the syntax is kind of weird and it will take some putzing around or aid from your favorite AI to get it right.
All of Rust ends up like this - you must be more verbose telling the compiler your intent. In exchange, it verifies the code matches your intent. So the up front cost is higher.
I suspect a lot of people will just pull a ring buffer off crates.io instead of figuring out the right incantations to make the compiler happy.
¯\_(ツ)_/¯ I'd like to lean into that "YMMV" in my post, I'm coming from low level C that interacts with hardware, can't really speak to higher-level C++.
Some things in Rust just don't translate to the way you'd do them in C - e.x. using different types to say if a GPIO pin is input or output adds a ton of boiler plate, but lets the compiler assure you don't mistakenly try and use a pin configured as an input for output.
In general, the whole zero sized types paradigm in Rust leads to way more lines of code to accomplish the same thing (it all ends up compiled out in the end though).
For embedded, I'll stand by what I said: it takes longer to write idiomatic Rust but you are more likely to get functionally correct code in the end.
ZSTs are a choice, if you're not benefiting from them in development time or correctness, why are you making that choice?
Rust isn't just C++ with static analysis. The safety is only about 1/3 of what makes me enjoy working with it. Even if I just consider the effort taken before I present something to a compiler, the trait system means I don't spend anywhere near as much time writing copy/copy assignment/move/move assignment constructors.
Some of that 2/3 will be me making choices like ZSTs, but I believe that's a result of it being necessary to grok the whole language and style and the idioms. If I just try to write C in Rust, it will be bad Rust and I'll have a bad time - but the same goes for just writing C++ as C with classes. I deliberately didn't say 'C/C++' in my ancestor post, because I think it's terminology that undermines claims of authority.
I do happen to have embedded experience, but avoid drawing conclusions from it because it is so different to more 'standard' systems development. If you can forgive a tangent: I avoid the whole domain now, because it feels like a ghetto. Standards are poor, tools are poor, pay is seriously poor. If you want to write embedded software, you probably have to work for a hardware company with hardware company's engineering culture and revenue model. That's a pretty crap place to write software in the west. I can't comment on comparing what like to do embedded development in Rust. Frankly, I hope I never can.
Good on you for recognizing what you enjoy and going for it.
I like hardware, always have :).
FWIW, I’ve done embedded at multiple mag 7 companies, and if levels.fyi is to be believed I’m getting paid the same as my peers doing other types of software. There are certainly places that don’t do this though (I hear the defense industry is notorious for this).
The point about the tooling being ghetto is spot on though, that’s part of why Rust is so exciting for us. A package manager?! What is this magic!!!
My experience in low-level stuff has been a mirror of the Linux kernel trying to take up Rust. Some people love it and fight hard for it, some hate it and fight hard against it, but most just see it as another tool in the toolbox and don't have strong feelings.
> Rust and JS are both famous for having minimal standard libraries
I'm all in favor of embiggening the Rust stdlib, but Rust and JS aren't remotely in the same ballpark when it comes to stdlib size. Rust's stdlib is decidedly not minimal; it's narrow, but very deep for what it provides.
C standard library is also very small. The issue is not the standard library. The issue is adding libraries for snippets of code, and in the name of convenience, let those libraries run code on the dev machine.
The issue is that our machines run 1970s OSes with a very basic security model, and are themselves so complex that they’re likely loaded with local privilege escalation attack vectors.
Doing dev in a VM can help, but isn’t totally foolproof.
It’s a good security model because everyone has the decency to follow a pull model. Like “hey, I have this thing, you can get it if you’re interested”. You decide the amount of trust you give to someone.
But NPM is more like “you’ve added me to your contact list, then it’s totally fine for me to enter your bedroom at night and wear your lingerie because we’re already BFF”. It’s “I’m doing whatever I want on your computer because I know best and you’re dumb” mentality that is very prevalent.
It’s like how zed (the editor) wants to install node.js and whatever just because they want to enable LSP. The sensible approach would have been to have a default config that relies on $PATH to find the language server.
> “you’ve added me to your contact list, then it’s totally fine for me to enter your bedroom at night and wear your lingerie because we’re already BFF”
I don't know if everyone will appreciate this, but I am in stitches right now... lol
Having worked on four different enterprise grade C# codebases, they most certainly have plenty of 3rd party dependencies. It would absolutely be the exception to not have 3rd party dependencies.
Yes, but the 3rd party dependencies tend to be conveniences rather than foundational. Easier mapping, easier mocking, easier test assertions, so a more security minded company can very easily just disallow their use without major impact. If it's something foundational to your project then what you're doing is probably somewhat niche. Most of the time there's some dependency from Microsoft that's rarely worse enough to justify using the 3rd party one.
I dunno, there's definitely stuff that's convenience related, sure, and then there's stuff that's less so. Things like MediatR, Dapper, Serilog, Refit are all pretty common in .NET projects. There's usually always some library for generating PDFs for reporting purposes etc. The ecosystem of NuGet packages is pretty large. It's definitely not all just Microsoft dependencies, nor is that usually how developers in the ecosystem think in my experience.
OP might be remembering how things were 10 years ago or so, before .NET Core became the .NET. Although there are still many companies with legacy codebases running on .NET 4.
That aside, dependency graphs in .NET land still tend to be much smaller.
This definitely not why enterprise "chooses" C# and neither of these were design decisions like implied. MS would have loved to have the explosive, viral ecosystem of Node earlier in .NET's life. Regardless a lot of companies using C# still use node-based solutions on the web so a insular development environment for one tier doesn't protect them.
I am not so sure about that. .net core is the moment they opened up, making it cross platform, going against the grain of owning it as a platform.
If they see a gap in .net, which is filled in by a third party, they would have no problem qualms about implementing their own solution in .net that meets their quality requirements. And to be fair, .net delivers on that.
This might anger some, but the philosophy is that it should be a batteries included one-stop shop, maybe driven by the culture of quite some ms shops that wouldn't eat anything unless ms feeds it them.
This has a consequence that the third-party ecosystem is a lot smaller, but I doubt MS regrets that.
If you compare that to F#, things are quite different wrt filling in the gaps, as MS does not focus on F#. A lot of good stuff for F# comes from the community.
They actually had a pretty active community on CodePlex - I used and contributed to many projects there... they killed that in ... checks the web... 2017, replaced with GitHub, and it just isn't the same...
And yet of course the world and their spouse import requests to fetch a URL and view the body of the response.
It would be lovely if Python shipped with even more things built in. I’d like cryptography, tabulate/rich, and some more featureful datetime bells and whistles a la arrow. And of course the reason why requests is so popular is that it does actually have a few more things and ergonomic improvements over the builtin HTTP machinery.
Something like a Debian Project model would have been cool: third party projects get adopted into the main software product by a sworn-in project member who who acts as quality control / a release manager. Each piece of software stays up to date but also doesn’t just get its main branch upstreamed directly onto everyone’s laps without a second pair of eyes going over what changed. The downside is it slows everything down, but that’s a side-effect of, or rather a synonym for stability, which is the problem we have with npm. (This looks sort of like what HelixGuard do, in the original article, though I’ve not heard of them before today.)
Requests is a great example of my point, actually. Creating a brand-new Python venv and running `uv add requests` tells me that a total of 5 packages were added. By contrast, creating a new Rust project and running `cargo add reqwest` (which is morally equivalent to Python's `requests`) results in adding 160 packages, literally 30x as many.
I don't think languages should try to include _everything_ in their stdlib, and indeed trying to do so tends to result in a lot of legacy cruft clogging up the stdlib. But I think there's a sweet spot between having a _very narrow_ stdlib and having to depend on 160 different 3rd-party packages just to make a HTTP request, and having a stdlib with 10 different ways of doing everything because it took a bunch of tries to get it right. (cf. PHP and hacks like `mysql_real_escape_string`, for example.)
Maybe Python also has a historical advantage here. Since the Internet was still pretty nascent when Python got its start, it wasn't the default solution any time you needed a bit of code to solve a well-known problem (I imagine, at least; I was barely alive at that point). So Python could afford to wait and see what would actually make good additions to the stdlib before implementing them.
Compare to Rust which _immediately_ had to run gauntles like "what to do about async", with thousands of people clamoring for a solution _right now_ because they wanted to do async Rust. I can definitely sympathize with Rust's leadership wanted to do the absolute minimum required for async support while they waited for the paradigm to stabilize. And even so, they still get a lot of flak for the design being rushed, e.g. with `Pin`.
So it's obviously a difficult balance to strike, and maybe the solution isn't as simple as "do more in the stdlib". But I'd be curious to see it tried, at least.
IMHO, the ideal for package management in a programming language ecosystem might recognise multiple levels of “standardisation”.
At the top, you have the true standard library for the language. This has very strong stability guarantees. Its purpose is twofold: to provide universal implementations of essentials and to define standard/baseline interfaces for common needs like abstract data types, relational databases, networking and filesystems to encourage compatibility and portability.
Next, you have a tier of recognised but not yet fully standardised libraries. These might be contributed by third parties, but they have requirements for identifying maintainers, appropriate licensing and mandatory peer review of all contributions. They have a clear versioning policy and can make breaking changes in new major releases, but they also provide some stability guarantees along the lines of semver and older releases are normally available indefinitely. The purpose of this tier is to provide a wider range of functionality and/or alternative implementations, but in a relatively stable way and implementing standard interfaces where applicable to improve portability.
Finally, you have the free-for-all, anyone-can-contribute tier. This should still have a sane security model where people can’t just upload malware scripts that run automatically just because someone installed a package. However, it comes with few guarantees about stability or compatibility, except that releases of published packages will be available indefinitely unless there’s a very good reason to pull them where you obviously wouldn’t want to use one anyway. A package you like might be written by a single contributor who no longer maintains it, but if someone does write something useful that simply doesn’t need any further maintenance once it’s finished and does its job, there is still a place to share it.
I agree with your general idea. I'd add it also looks very similar to what typical GNU/Linux have in practice: blessed packages from the distros' repositories, and third-party repos for those who want them.
Debian also has something 'in the middle' with additional repositories that aren't part of the main distribution and/or contain proprietary software.
Or maybe just get comfortable with adding versions and deprecation. eg optparse to argparse (though tbf, I would have just preferred it was optparse2). Or maybe the problem is excessive stability commitments. I think I prefer languages that realize things can improve and are willing to say if you want to run 10 year old code, use a 10 year old compiler/runtime.
I think I prefer languages that realize things can improve and are willing to say if you want to run 10 year old code, use a 10 year old compiler/runtime.
IMHO, the trouble with that stance is that it leaves no path to incrementally update a long-lived system to benefit from any of those improvements.
Suppose we have an application that runs on 2025’s most popular platform and in ten years we’re porting it to whatever new platform is popular in 2035. Personally, I’d like to know that all the business logic and database queries and UI structure and whatever else we wrote that was working before will still be working on the new platform, to whatever extent that makes sense. I’d like to make only some reasonably necessary set of changes for things that are actually different between the two platforms.
If we can’t do that, our only other option is a big rewrite. That is how you get a Python 2 to Python 3 situation. And that, in turn, is how you get a lot of systems stuck on the older version for years, despite all the advantages any later versions might offer.
Sure it does? Or do I not understand? It pushes the work to the application, not the language.
And it works fine on rails. If you want to stay on an EOL rails, you're on your own, or you can buy ongoing security backports from at least 2 (that I know of, maybe more) vendors. LTS lives roughly 2 years and then you either upgrade or deal with eol yourself.
That's not an apple-to-apple comparison, since Rust is a low-level language, and also because `reqwest` builds on top of `tokio`, an async runtime, and `hyper`, which is also a HTTP server, not just a HTTP client. If you check `ureq`, a synchronous HTTP client, it only adds 43 packages. Still more, but much less.
And in Go I can build a production-ready HTTPS (not just HTTP) server with just the standard library and a few lines of code. (0 packages).
That Rust does not have standard implementations of commonly-used features (such as an async runtime) is problematic for supply chain security, since then everyone is pulling in dozens (or hundreds) of fragmented 3rd-party packages instead of working with a bulletproof standard library.
And this is exactly why Go is winning: because it's actually rather easy to write "pure Go" utilities (no dependencies outside the standard library), which statically compile to boot (avoiding shared libraries).
> (cf. PHP and hacks like `mysql_real_escape_string`, for example.)
PHP is a fantastic resource to learn how to do proper backward compatibility and package management. By doing the exact opposite of whatever PHP does, mostly.
It might solve the problem, in as much as the problem is that not only can it be done, but it’s profitable to do so. This is why there’s no Rust problem (yet).
Most rust programmers are mediocre at best and really need the memory safety training wheels that rust provides. Years of nodejs mindrot has somehow made pulling into random dependencies irregular release schedules to become the norm for these people. They'll just shrug it off come up with some "security initiative* and continue the madness
I only personally know one Rust programmer (works in scientific HPC) and he’s fantastic, but in general I do get the sense that most Rust devs migrated from JS and are just now figuring out “omg strong typing and compiled code native to the client hardware is really nice!” and think it’s a ground breaking revelation.
Saying this as someone who is cautiously optimistic about Rust for my own work.
I agree partly. I love cargo and can’t understand why certain things like package namespaces and proof of ownership isn’t added at a minimum. I was mega annoyed when I had to move all our Java packages from jcenter, which was a mega easy setup and forget affair, to maven central. There I suddenly needed to register a group name (namespace mostly reverse domain) and proof that with a DNS entry. Then all packages have to be signed etc. In the end it was for this time way ahead. I know that these measures won’t help for all cases. But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming. Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning.
But I don’t want to go away from package managers or easy to use/sharable packages either.
> But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming.
Since your comment starts with commentary on crates.io, I'll note that this has never been possible crates.io.
> Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning.
I don't think this has ever been true. AFAIK crates.io has always prevented registering two different crates whose names differ only in the use of dashes vs underscores.
Didn’t know this term. After reading I wonder why short lived tokens get this monocle. But yeah I prefer OIDC over token based access as well. Only small downside I see is the setup needed for a custom OIDC provider. Don’t know the right terms out of my head but we had quite the fun to register our internal Jenkins to become a create valid oidc tokens for AWS. GitHub and GitHub Actions come with batteries included. I mean the downside that a huge vendor can easily provide this and a custom rolled CI needs extra steps / infrastructure.
I'm a huge Go proponent but I don't know if I can see much about Go's module system which would really prevent supply-chain attacks in practice. The Go maintainers point [1] at the strong dependency pinning approach, the sumdb system and the module proxy as mitigations, and yes, those are good. However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users.
The lack of package install hooks does feel somewhat effective, but what's really to stop an attacker putting their malicious code in `func init() {}`? Compromising a popular and important project in this way would likely be noticed pretty quickly. But compromising something widely-used but boring? I feel like attackers would get away with that for a period of time that could be weeks.
This isn't really a criticism of Go so much as an observation that depending on random strangers for code (and code updates) is fundamentally risky. Anyone got any good strategies for enforcing dependency cooldown?
A big thing is that Go does not install the latest version of transitive dependencies. Instead it uses Minimal version selection (MVS), see https://go.dev/ref/mod#minimal-version-selection. I highly recommend reading the article by Russ Cox mentioned in the ref. This greatly decreases your chances of being hit by malware released after a package is taken over.
In Go, access to the os and exec require certain imports, imports that must occur at the beginning of the file, this helps when scanning for malicious code. Compare this JavaScript where one could require("child_process") or import() at any time.
Personally, I started to vendor my dependencies using go mod vendor and diff after dependency updates. In the end, you are responsible for the effect of your dependencies.
In Go you know exactly what code you’re building thanks to gosum, and it’s much easier to audit changed code after upgrading - just create vendor dirs before and after updating packages and diff them; send to AI for basic screening if the diff is >100k loc and/or review manually. My projects are massive codebases with 1000s of deps and >200MB stripped binaries of literally just code, and this is perfectly feasible. (And yes I do catch stuff occasionally, tho nothing actively adversarial so far)
Aside from other security features already mentioned Go also doesn't execute code at compile time by design.
There is no airtight technical solution, for any language, for preventing malicious dependencies making it into your application. You can have manual or automated review using heuristics but things will still slip through. Malicious code doesn't necessarily look obvious, like decoding some base64 and piping it into bash, it can be an extremely subtle vulnerability sprinkled in that nobody will find until it's too late.
RE dependency cooldowns I'm hoping Go will get support for this. There's a project called Athens for running your own Go module proxy - maybe it could be implemented there.
> However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users.
You can't, really, aside from full on code audits. By definition, if you trust a maintainer and they get compromised, you get compromised too.
Requiring GPG signing of releases (even by just git commit signing) would help but that's more work for people to distribute their stuff, and inevitably someone will make insecure but convenient way to automate that away from the developer
Historically, arguments of "it's popular so that's why it's attacked" have not held up. Notable among them was addressing Windows desktop security vulnerabilities. As Linux and Mac machines became more popular, not to mention Android, the security vulnerabilities in those burgeoning platforms never manifested to the extent that they were in Windows. Nor does cargo or pip seem to be infected with these problems to the extent that npm is.
> Nor does cargo or pip seem to be infected with these problems to the extent that npm is.
Easy reason. The target for malware injections is almost always cryptocurrency wallets and cloud credentials (again, mostly to mine cryptocurrencies). And the utter utter majority of stuff interacting with crypto and cloud, combined with a lot of inexperienced juniors who likely won't have the skill to spot they got compromised, is written in NodeJS.
Compared to the JS ecosystem and number of users both Python and Rust are puny, also the the NPM ecosystem also allowed by default for a lot of post-install actions since they wanted to enable a smooth experience with compiling and installing native modules (Not entirely sure how Cargo and PIP handles native library dependencies).
As for Windows vs the other OS's, yes even the Windows NT family grew out of DOS and Win9x and tried to maintain compatiblity for users over security up until it became untenable. So yes, the base _was_ bad when Windows was dominant but it's far less bad today (why people target high value targets via NPM,etc since it's an easier entry-point).
Android/iOS is young enough that they did have plenty of hindsight when it comes to security and could make better decisions (Remember that MS tried to move to UWP/Appx distribution but the ecosystem was too reliant on newer features for it to displace the regular ecosystem).
Remember that we've had plenty of annoyed discourse about "Apple locking down computers" here and on other tech forums when they've pushed notarization.
I guess my point is that, people love to bash on MS but at the same time complain about how security is affecting their "freedoms" when it comes to other systems (and partly MS), MS is better at the basics today than they were 20-25 years ago and we should be happy about that.
This comment seems to address users intentionally installing malware. I mean to address cracking, the situation where an attacker gains root or installs software that the user does not know about.
Preventing the user from installing something that they want to install is another issue completely. I'm hesitant to call it exactly security, though I agree that it falls under the auspices of security.
Cracking is a term related to removing copy-protections. Rooting or privilege escalation is better terms for what you're mentioning.
As for "users intentionally installing malware", Windows in the early 00s had a bunch of fundamentally insecure deployment models like ActiveX controls and browsers (IE especially) were more or less swiss cheese in terms of security even outside the ActiveX controls.
Visiting the wrong webpage was often enough to get crap on your computer.
My view is that once you have bad native code running on your computer there's a large chance that it's game-over (the modern sandboxes like WASM were designed to enforce a probably safe subset where regular kernel mistakes are shielded by another layer of abstraction that needs to be broken).
Even Linux has had privilege escalations every year as far as I know. Notarization/stores is just a way to try to keep check on what code runs on end-user computers that isn't sandboxed (and allow for revoking that code if found to be malicious), maybe Linux is slightly safer still but that's probably due to less older features in the Kernel, but Windows has for example recently gotten a rewritten font-parser in Rust (the previous font parser was a common exploitation point that was placed with a too high privilegie).
No. By NPM not allowing any package to run code on the developer's machine. I can trust npm (the software), but not the library. It's a very weird choice to just allow any package to run post install script. Especially when there's little to none verification done on npmjs side.
Developers can feel free to not secure their computer or sell their keys. But that not means npm should allow straight code push from their computers to everyone that has downloaded their library.
> Compared to the JS ecosystem and number of users both Python and Rust are puny
This is just plainly false in case of Python.
> also the the NPM ecosystem also allowed by default for a lot of post-install actions since they wanted to enable a smooth experience with compiling and installing native modules (Not entirely sure how Cargo and PIP handles native library dependencies).
Rust is already "native" so Cargo doesn't need to do anything.
Python has the logic to do native builds baked into pip and friends, so a Python package can just specify what to build in the manifest. But it also allows for precompiled wheels, and most popular packages do them for all major OSes, so users rarely need to actually build stuff in practice.
> It's not "node" or "Javascript" the problem, it's this convenient packaging model.
That and the package runtime runs with all the same privileges and capabilities as the thing you're building, which is pretty insane when you think about it. Why should npm know anything outside of the project root even exists, or be given the full set of environment variables without so much as a deny list, let alone an allow list? Of course if such restrictions are available, why limit them to npm?
The real problem is that the security model hasn't moved substantially since 1970. We already have all the tools to make things better, but they're still unportable and cumbersome to use, so hardly anything does.
pnpm (maybe yarn too?) requires explicit allowlisting of build scripts, hopefully npm will do the same eventually
> security model
yep, some kind of seccomp or other kind of permission system for modules would help a lot. (eg. if the 3rd party library is parsing something and its API only requires a Buffer as input and returns some object then it could be marked "pure", if it supports logging then that could be also specified, and so on)
For all the other things I like about yarn, it still executes build scripts willy-nilly, so I am looking at switching to pnpm. I'm sure my $work is going to love me changing up the build toolchain again... PHP's composer on the other hand requires an allowlist in the project's composer.json. I never would have thought PHP would be the one to be getting stuff like this right.
Still, I think the "allow-scripts" section or whatever it's called should be named "allow-unrestricted-access-to-everything". Or maybe just stick "dangerously-" in front, I dunno, and drop it when the mechanism is capable of fine-grained privileges.
Every time I look at a new project, my face falls when it's written in Rust. I simply don't trust a system that pulls in gigabytes of god-knows-what off the cloud, and compiles it on my box. It's a real barrier to entry, for me.
When I download a C project, I know that it only depends on my system libraries - which I trust because I trust my distro. Rust seems to expect me to take a leap in the dark, trusting hundreds of packagers and their developers. That might be fine if you're already familiar with the Rust ecosystem, but for someone who just wants to try out a new program - it's intimidating.
On Debian you can use the local registry for Rust which is backed by packages.
Though I will say, even as someone who works at a company that sells Linux distributions (SUSE), while the fact we have an additional review step is nice, I think the actual auditing you get in practice is quite minimal.
For instance, quite recently[1] the Debian package for a StarDict plugin was configured automatically upload all text selected in X11 to some Chinese servers if you installed it. This is the kind of thing you'd hope distro maintainers to catch.
Though, having build scripts be executed in distribution infrastructure and shipped to everyone mitigates the risk of targeted and "dumb" attacks. C build scripts can attack your system just as easily as Rust or JavaScript ones can (in fact it's probably even easier -- look at how the xz backdoor took advantage of the inscrutability of autoconf).
Well, yes I do know when a C project only depends on my system libraries, because otherwise it won't compile. That's the point.
Furthermore, for the purposes of this discussion, it really doesn't matter what code there is in the C project. What's there has been put there by the people who run the project. If they are malicious, then at least I know who they are. With Rust, I'm downloading and compiling code from many, many third parties. I have no idea who they are. The potential for one of them to be malicious is much, much higher.
I think this is right about Rust and Cargo, but I would say that Rust has a major advantage in that it implements frozen + offline mode really well (which if you use, obviously significantly decreases the risks).
Any time I ever did the equivalent with NPM/node world it was basically unusable or completely impractical
Because it doesn't perform as advertised: wild amounts of inconsistencies in behavior (within and between versions), performance issues (pnpm exec adds 15s to all shebang'd execution time over npm/yarn/bun/etc.), etc. Version-to-version stability has been traditionally bad - it's half-baked software.
Claude doesn't know this, of course, because it can only read superficial summaries posted on the internet and has zero real experience actually using this dumpster fire.
There are ecosystems that have package managers but also well developed first party packages.
In .NET you can cover a lot of use cases simply using Microsoft libraries and even a lot of OSS not directly a part of Microsoft org maintained by Microsoft employees.
2020 State of the Octoverse security report showed that .NET ecosystem has on average the lowest number of transitive dependencies. Big part of that is the breadth and depth of the BCL, standard libraries, and first party libraries.
The .NET ecosystem has been moving towards a higher number of dependencies since the introduction of .NET Core. Though many of them are still maintained by Microsoft.
The "SDK project model" did a lot to reduce that back down. They did break the BCL up into a lot of smaller packages to make .NET 4.x maintenance/compatibility easier, and if you are still supporting .NET 4.x (and/or .NET Standard), for whatever reason, your dependency list (esp. transitive dependencies) is huge, but if you are targeting .NET 5+ only that list shrinks back down and the BCL doesn't show up in your dependency lists again.
Even some of the Microsoft.* namespaces have properly moved into the BCL SDKs and no longer show up in dependency lists, even though Microsoft.* namespaces originally meant non-BCL first-party.
I think first-party Microsoft packages ought to be a separate category that is more like BCL in terms of risk. The main reason why they split them out is so that they can be versioned separately from .NET proper.
Supply chain attacks are scary because you do everything "right", but the ecosystem still compromises you.
But realistically, I think the sum total of compromises via package managers attacks is much smaller than the sum total of compromises caused by people rolling their own libraries in C and C++.
It's hard to separate from C/C++'s lack of memory safety, which causes a lot of attacks, but the fact that code reuse is harder is a real source of vulnerabilities.
Maybe if you're Firefox/Chromium, and you have a huge team and invest massive efforts to be safe, you're better off with the low-dependency model. But for the median project? Rolling your own is much more dangerous than NPM/Cargo.
Rust (and really, any but JS) ecosystem have a bit more "due dilligence" applied everywhere; I don't doubt someone will try to namesquat but chance of success are far smaller
> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise.
There was no decision in case of C/C++; it was just not a thing languages had at the time so the language itself (especially C) isn't written in a way to accommodate it nicely
> Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.
Any code you download and compile is running code on dev machine; and Go does have tools to do that in compile process too.
I do however like the by default namespacing by domain, there is no central repository to compromise, and forks of any defunct libs are easier to manage.
> Rust (and really, any but JS) ecosystem have a bit more "due dilligence" applied everywhere; I don't doubt someone will try to namesquat but chance of success are far smaller
I really agree, and I feel like it's a culture difference. Javascript was (and remains) an appealing programming language for tinkerers and hobbyists, people who don't really have a lot of engineering experience. Node and npm rose to prominence as a wild west with lots of new developers unfamiliar with good practices, stuck with a programming environment that had few "batteries included," and at a time when supply chain attacks weren't yet on everybody's minds. The barriers to entry were low and, well, the ecosystem sort of reflected that. You can't wash that legacy away overnight.
Rust in contrast attracts a different audience because of the language's own design objectives.
Obviously none of this makes it immune, and you can YOLO install random dependencies in any programming language, but I don't think any language is ever going to suffer from this in quite the same way and to the same extent that JS has simply due to when and how the ecosystem evolved.
And really, even JS today is not JS of yesteryear. Sure there are lots of bad actors and these bad NPM packages sneak in, but also... how widely are all of them used? The maturation of and standardization on certain "batteries included" frameworks rather than ad hoc piecing stuff together has reduced the liklihood of going astray.
I have a similar opinion but I think Java's model with maven and friends hits the sweet spot:
- Packages are always namespaced, so typosquating is harder
- Registries like Sonatype require you to validate your domain
- Versions are usually locked by default
My professional life has been tied to JVM languages, though, so I might be a bit biased.
I get that there are some issues with the model, especially when it comes to eviction, but it has been "good enough" for me.
Maven does not support "scripts" as NPM does, such as the pre-install script used for this exploit. With scripts enabled, the mere act of downloading a dependency requires a high degree of trust in it.
While I agree that dependency tree size can be sometimes a problem in Rust, I think it often gets overblown. Sure, having hundreds of dependencies in a "simple" project can be scary, but:
1) No one forces you to use dependencies with large number of transitive dependencies. For example, feel free to use `ureq` instead of `reqwest` pulling the async kitchen sink with it. If you see an unnecessary dependency, you could also ask maintainers to potentially remove it.
2) Are you sure that your project is as simple as you think?
3) What matters is not number of dependencies, but number of groups who maintain them.
On the last point, if your dependency tree has 20 dependencies maintained by the Rust lang team (such as `serde` or `libc`), your supply chain risks are not multiplied by 20, they stay at one and almost the same as using just `std`.
On your last note, I wish they would get on that signed crate subset. Having the same dependency tree as cargo, clippy, and rustc isn't increasing my risk.
Rust has already had a supply chain attack propagating via build.rs some years ago. It was noticed quickly, so staying pinned to the oldest thing that worked and had no cve pop in cargo audit is a decent strategy. The remaining risk is that some more niche dependency you use is and always has been compromised.
> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise.
The safest code is the code that is not run. There is no lack of attacks targeting C/C++ code, and odin is just a hobby language for now.
What is worse between writing potentially vulnerable code yourself and having too many dependencies.
Finding vulnerabilities and writing exploits is costly, and hackers will most likely target popular libraries over your particular software, much higher impact, and it pays better. Dependencies also tend to do more than you need, increasing the attack surface.
So your C code may be worse in theory, but it is a smaller, thus harder to hit target. It is probably an advantage against undiscriminating attacks like bots and a downside against targeted attacks by motivated groups.
Using C++ daily, whenever I do js/ts are some javascript variant, since I don't use it daily, and update becomes a very complex task. frameworks and deps change APIs very frequently.
It's also very confusing (and I think those attack vectors benefit exactly from that), since you have a dependency but the dep itself dependent on another dep version.
Building basic CapacitorJS / Svelte app as an example, results many deps.
It might be a newbie question, but,
Is there any solution or workflow where you don't end up with this dependency hell?
Don't use a framework? Loading a JS script on a page that says "when a update b" hasn't changed much in about 20 years.
Maybe I'm being a bit trite but the world of JavaScript is not some mysterious place separate from all other web programming, you can make bad decisions on either side of the stack. These comments always read like devs suddenly realizing the world of user interactions is more complicated and has more edge cases than they think.
Not knowing that much about apt, isn't _any_ package system vulnerable, and purely a question of what guards are in place and what rights are software given upon install?
It's not the packaging tech. Apt will typically mean a Debian-based distro. That means the packages are chosen by the maintainers and updated only during specific time periods and tested before release. Even if the underlying software gets owned and replaced, the distro package is very unlikely to be affected. (Unless someone spent months building trust, like xz)
But the basic takeover... no, it usually won't affect any Debian style distro package, due to the release process.
Given the years (or decades) it takes updates to happen in Debian stable, it’s immune to supply chain attacks. You do get to enjoy vulnerabilities that have been out for years, though.
Out of the 789 npm packages in this incident, only 4 were ever used in any dependency tree of any Linux operating system (including Homebrew). Not in the affected versions, but ever.
If your Rust software observes a big enough chunk of the computer fever dream you are likely to end up with 2-3 digit amount of Rust dependencies, but they are probably all going to be high profile ones (tokio, anyhow, reqwest, the hyper crates, ...), instead of niche ones that never make it into any operating system.
This is not a silver bullet of course, but there seems to be an inverse correlation between "is part of any operating system dependency tree" and "gets compromised in an npm-like incident".
> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.
The alternative that C/C++/Java end up with is that each and every project brings in their own Util, StringUtil, Helper or whatever class that acts as a "de-facto" standard library. I personally had the misfortune of having to deal with MySQL [1], Commons [2], Spring [3] and indirectly also ATG's [4] variants. One particularly unpleasant project I came across utilized all four of them, on top of the project's own "Utils" class that got copy-and-paste'd from the last project and extended for this project's needs.
And of course each of these Utils classes has their own semantics, their own methods, their own edge cases and, for the "organically grown" domestic class that barely had tests, bugs.
So it's either a billion "small gear" packages with dependency hell and supply chain issues, or it's an amalgamation of many many different "big gear" libraries that make updating them truly a hell on its own.
That is true, but the hand-rolled StringUtil won't steal your credentials and infect your machine, which is the problem here.
And what is wrong with writing your own util library that fits your use case anyway? In C/C++ world, if it takes less than a couple hours to write, you might as well do it yourself rather than introduce a new dependency. No one sane will add a third-party git submodule, wire it to the main Makefile, just to left-pad a string.
> That is true, but the hand-rolled StringUtil won't steal your credentials and infect your machine, which is the problem here.
Yeah, that's why I said that this is the other end of the pendulum.
> In C/C++ world, if it takes less than a couple hours to write, you might as well do it yourself rather than introduce a new dependency.
Oh I'm aware of that. My point still stands - that comes at a serious maintenance cost as well, and I'd also say a safety cost because you're probably not wrapping your homebrew StringUtils with a bunch of sanity checks and asserts, meaning there will be an opportunity for someone looking for a cheap source of exploits.
> how hard is it to add a few methods that split a string or pad it?
In full generality, pretty hard. If you're just dealing with ASCII or Latin-1, no problem. Then add basic Unicode. Then combining characters. Then emojis. It won't be trivial anymore.
Full generality is not a practical target. You select your subset of the problem and you solve it. Supporting everything in a project is usually a fever dream.
This kind of thing is precisely the stuff that needs to be in stdlib.
C++ is hardly a good example to follow here, but then again it does have Boost which is the de facto "extended standard library" in practice, and gets treated differently from other dependencies by many.
> how hard is it to add a few methods that split a string or pad it?
Well, if you're in C/C++, you always risk dealing with null pointers, buffer overruns, or you end up with use-after-free issues. Particularly everything working with strings is nasty and error-prone if one does not take care of proper testing - which many "homegrown" libraries don't.
And that's before taking the subtleties of character set encodings between platforms into account. Or locale. Or any other of the myriad ways that C/C++ and even Java offer you to shoot yourself in the foot with a shotgun.
And no, hoping for the best and saying "my users won't ever use Unicode" or similar falls apart on the first person copying something from Outlook into a multi-line paste box. Or someone typing in their non-Latin name. Oh, and right-to-left languages, don't forget about these. What does "pad from left" even mean there? Is the intent of the user still "at the beginning of the string itself?" Or does the user rather want "pad at the beginning of the word/sentence", which in turn means padding at the end of the string?
There's so much stuff that can go horribly horribly wrong when dealing with strings, and I've seen more than my fair share just reading e-mail templates from supposed "enterprise" software.
My feeling is that languages with other packing models are merely less convenient,
and there is no actual tangible difference security-wise. Just take C and replace "look for writable repositories". It just takes more work and is less uniform to say write a worm that looks for writable cmake/autoconf and replicate that way.
What would actually stop this is writing compilers and build systems in a way that isolates builds from one another. It's kind of stupid that all a compiler really needs is an input file, a list of dependencies, and an output file. Yet they all make it easy to root around, replicate and exfiltrate. It can be both convenient and not suffer from these style of attacks.
Not really. cmake and automake are for compiling the library, not for the downloading it. The gap between the two is what's get erased from npm. And it made worse because of the auto update set by default when `npm install` is run.
Not having a convenient package manager doesn't mean you don't need the functionality that's otherwise offered by third-party packages, it just means that you either need other means to obtain those third-party packages (usually reducing the visibility this dependency!) or implement them yourself (sometimes this is good, but sometimes this can also be very bad for security. Your DYI code won't get as many eyes and audits as the popular third party package!).
Indeed, Rust's supply chains story is an absolute horror, and there are countless articles explaining what should be done instead (e.g. https://kerkour.com/rust-stdx)
TL;DR: ditch crates.io and copy Go with decentralized packages based directly on and an extended standard library.
Centralized package managers only add a layer of obfuscation that attackers can use to their advantage.
On the other hand, C / C++ style dependency management is even worse than Rust's... Both in terms of development velocity and dependencies that never get updated.
> Centralized package managers only add a layer of obfuscation that attackers can use to their advantage.
They add a layer of convenience. C/C++ are missing that convenience because they aren't as composable and have a long tail of pre-package manager projects.
Java didn't start with packages, but today we have packages. Same with JS, etc.
Yeah, but it's not immune to supply chain attacks. Counting on maintainers of dpkg is not that different from counting on maintainers of random crate package.
I believe you, in that package management with dependencies without security mitigation is both convenient and dangerous. And I certainly agree this could happen for other package managers as well.
My real worry, for myself re the parent comment is, it's just a web frontend. There are a million other ways to develop it. Sober, cold risk assessment is: should we, or should we have, and should anyone else, choose something npm-based for new development?
Ie not a question about potential risk for other technologies, but a question about risk and impact for this specific technology.
Every time I fire up "cmake" I chant a little spell that protects me from the goblins that live on the other side of FetchContent to promise to the Gods of the Repo that I will, eventually, review everything to make sure I'm not shipping poop nuggets .. just as soon as I get the build done, tested .. and shipped, of course .. but I never, ever do.
Surely in this case the problem is a technical one, and with more work towards a better security model and practices we can have the best of both worlds, no?
Agreed, rust's cargo model is basically the worst part of that ecosystem right now. I've had developers submit pretty simple cli tools with hundreds and hundreds of dependencies. I guess there wasn't any lessons learned from the state of NPM.
Just a last month someone was trying to figure the cargo tree on which Rust package got imported implicitly via which package. This will totally happen in rust as well as long as you use some kind of package manager. Go for zero or less decencies.
It already did happen. It propogated via build.rs as well. But as I said elsewhere, ut doesn't help you to forgo dependencies part of rust tooling itself.
It’ll probably happen eventually with Rust, but ecosystem volume and informal packaging processes / a low barrier to entry seem to be significant driver in the npm world.
(These are arguably good things in other contexts.)
I installed the package, obviously I intend to run it. How does getting pwned once I run it manually differ from getting pwned once I install it? I’m still getting pwned
NPM default installation method does not really lock down you dependencies. It allows for update when the patch number (semver) is increased. Which is why those malware bump it up. Anyone who then run `npm install` will get it and will run the code.
In the early days the Node ecosystem adopted (from Unix) the notion that everything has to be its own micro package. Not only was there a failure to understand what it was actually talking about, but it was never a good fit for package management to begin with.
I understand that there's been some course correction recently (zero dependency and minimal dependency libs), but there are still many devs who think that the only answer to their problem is another package, or that they have to split a perfectly fine package into five more. You don't find this pattern of behavior outside of Node.
> In the early days the Node ecosystem adopted (from Unix) the notion that everything has to be its own micro package.
The medium is the message. If a language creates a very convenient package manager that completely eliminates the friction of sharing code, practically any permutation of code will be shared as a library. As productivity is the most important metric for most companies, devs will prefer the conveniently-shared third-party library instead of implementing something from scratch. And this is the result.
I don't believe you can have packaging convenience and avoiding dependency hell. You need some amount of friction.
It’s not even the convenience. It’s about trust. Npm makes it so that as soon as you add something to the dependency list, you trust the third party so completely you’re willing to run their code on your system as soon as they push an update.
I hate to be the guy saying AI will solve it, but this is a case where AI can help. I think in the next couple of years we’ll see people writing small functions with Claude/codex/whatever instead of pulling in a dependency. We might or might not like the quality of software we see, but it will be more resistant to supply chain attacks.
How is this going to solve the supply chain attack problem at all though? It just obfuscates things even more, because once an LLM gets "infected" with malicious code, it'll become much more difficult to trace where it came from.
If anything, blind reliance on LLMs will make this problem much worse.
When there's a depedency, it's typically not for a small function. If you want to replace a full dependency package by your own generated code, you'll need to review hundreds of even thousands of line of code.
Now will you trust that AI didn't include its own set of security issues and will you have the ability to review so much code?
I wonder what the actual result will be. LLMs can generate functions quickly, but they're also keen to include packages without asking. I've had to add a "don't add new dependencies unless explicitly asked" to a few project configs.
For sure. I don't think the software ecosystem has come to terms with how things are going to change.
Libraries will be providing raw tools like - Sockets, Regex Engine, Cryptography, Syscalls, specific file format libraries
LLMs will be building the next layer.
I have build successful running projects now in Erlang, Scheme, Rust - I know the basic syntax of two of those but I couldn't write my deployed software in any of them in the couple of hours of prompting.
The scheme it had to do a lot of code from first principles and warned me how laborious it would be - "I don't care, you are doing it."
I have tools now I could not have imagined I could build in a reasonable time.
An approach I learnt from a talk posted to HN (I forget the talk, not the lesson) is to not depend on the outside project for its code, just lift that code directly in to your project, but to rely on it for the tests, requiring/importing it etc when running your own tests. That protects you from a lot of things (this kind of attack was not mentioned, afaic recall) but doesn’t allow bugs found by the other project to be missed either.
I don’t think I’ll live long enough to trust AI coding assistants with something like schema validation, just to name one thing I use dependencies for.
I read your comment as in, comparing to node, my bad.
With regards to Rust, crates are packages which may include opaque binaries, e.g. serde_derive, and the stdlib is weak, so imports of thousands of lines of code are basically necessary for otherwise fundamental features like async.
It's probably easier to add dependencies in Go, but in the end people/projects don't.
> but it's only a matter of time until it'll happen on the Rust ecosystem
Totally 100% agree, though tools like cargo tree make it more of a tractable problem, and running vendored dependencies is first class at least.
The one I am genuinely most concerned of is Golang. The way Dependencies are handled leaves much to be desired, I'm really surprised that there haven't been issues honestly.
The problem isn't specific to node. NPM is just the most popular repo so the most value for attacks. The same thing could happen on RubyGems, Cargo, or any of the other package managers.
Been a while since I looked into this, but afaik Maven Central is run by Sonatype, which happens to be one of the major players for systems related to Supply Chain Security.
From what I remember (a few years old, things may have changed) they required devs to stage packages to a specific test env, packages were inspected not only for malware but also vulnerabilities before being released to the public.
NPM on the other hand... Write a package -> publish. Npm might scan for malware, they might do a few additional checks, but at least back when I looked into it nothing happened proactively.
Make no mistake, Maven Central does get multiple malware components uploaded each year, though not nearly to the same extent as npm or pypi. Sonatype (my former employer) just doesn't report on these publicly each time it happens. It's not an isolated problem but certainly harder to do with maven.
I assume you're talking about malware uploaded to new artifact coordinates (possibly named so as to try to confuse users), not hijacking of existing artifact coordinates (group ID, artifact ID)?
generally yes, although hijacking can and has happened on Central with expired maintainer domains reclaimed by threat actor who can then republish malicious versions of a previously legit group/artifact ID. there's also the problem of mirrors or copies of hijacked npm being replicated on Central -https://x.com/SocketSecurity/status/1993389518247149907
no, because if you used dependency cooldown you wouldn't be using the latest version when you start your project, you would be using the one that is <cooldown period> days/versions old
edit: but if that's also compromised earlier... \o/
Value is one thing but the average user (by virtue of being popular) will be just less clued in on any security practices that could mitigate the problem.
I've started to feel it is much more an npm problem than a node problem. One of the things I've started leaning on more is prioritizing packages from JSR [0]. JSR is a part of Deno's efforts, so is often easiest to use in Deno packages, but most of the things with high scores on JSR get cross-published to npm and the few that prefer JSR only there's an alright JSR bridge to npm.
Of course using more JSR packages does start to add more reason to prefer Deno to Node. Also, there are still some packages that are deno.land/x/ only (sort of the first version of JSR, but no npm cross-compatibility) worth checking out. For instance, I've been impressed with Lume [1], a thoughtful SSG that's sort of the opposite of Astro in that it iterates at a slow, measured pace, and doesn't try to be a kitchen sink but more of workbench with a lot of tools easy to find. It's deno.land/x/ only for now for reasons I don't entirely agree with but I can't deny that JSR can be quite a step up in publishing complexity for not exactly obvious gain.
Node is fine, the issue lies in its package model and culture:
* Many dependencies, so much you don't know (and stop caring) what is being used.
* Automatic and regular updates, new patch versions for minor changes, and a generally accepted best practice of staying up to date on the latest versions of things, due to trauma from old security breaches or big migrations after not updating for a while.
* No review, trust based self-publishing of packages and instant availability
* untransparent pre/postinstall scripts
The fix is both cultural and technological:
* Stop releasing for every fart; once a week is enough, only exception being critical security reasons.
* Stop updating immediately whenever there's an update; once a week is enough.
* Review your updates
* Pay for a package repository that actually reviews changes before making them widely available. Actually I think the organization between NPM should set that up, there's trillion dollar companies using the Node ecosystem who would be willing and able to pay for some security guarantees.
I’m not a node/js apologist, but every time there is a vulnerability in NPM package, this opinion is voiced.
But in reality it has nothing to do with node/js. It’s just because it’s the most used ecosystem. So I really don’t understand the argument of not using node. Just be mindful of your dependencies and avoid updating every day.
It has everything to do with node/js. Because the community believes in tiny dependencies that must be updated as often as possible and the tooling reflects that belief.
We chose to write our platform for product security analytics (1) with PHP, primarily because it still allows us to create a platform without bringing in over 100 dependencies just to render one page.
I know this is a controversial approach, but it still works well in our case.
Not sure what the language has anything to do with it, we've built JavaScript applications within pulling in 100s of NPM packages before NPM was a thing, people and organizations can still do so today, without having to switch language, if they don't want to.
Does it require disciple and a project not run by developers who just learned program? You betcha.
I might say that every interpreter has a different minimum dependency level just to create a simple application. If we're talking about Node.js, there's a long list of dependencies by default.
So yes, in comparison, modern vanilla PHP with some level of developer discipline (as you mentioned) is actually quite suitable, but unfortunately not popular, for low-dependency development of web applications.
The language and capabilities of the platform indeed have a lot of influence on how many packages the average project depends on.
With Swift on iOS/macOS for instance it’s not strange at all for an app to have a dependency tree consisting of only 5-10 third party packages total, and with a little discipline one can often get that number down to <5. Why? Because between the language itself, UIKit/AppKit, and SwiftUI, nearly all needs are pretty well covered.
I think it’s time to beef up both JavaScript itself as well as the platforms where it’s run (such as the browser and Node), so people don’t feel nearly as much of a need to pull in tons of dependencies.
> If we're talking about Node.js, there's a long list of dependencies by default.
But that's not true? I initialize a project locally, there is zero dependencies by default, and like I did five years ago, I can still build backend/frontend projects with minimal set of dependencies.
What changed is what people are willing/OK with doing. Yes, it'll require more effort, obviously, but if you want things to be built properly, it usually takes more effort.
7.0 added scalar type declarations and a mechanism for strong typing. PHP 8.0 added union types and mixed types. PHP enforces types at runtime, Javascript/Typescript do not. PHP typesystem is built into the language, with Js u either need jsdoc or Typescript both of which wont enforce runtime type checks, Typescript even adds a buildstep. php-fpm allows u to not care about concurrency too much because of an isolated process execution model, with js based apps you need to be extremely careful about concurrency because of how easy you can create and access global stuff.
PHP also added a lot of syntax sugar over the time especially with 8.5 my beloved pipe operator.
And the ecosystem is not as fragile as Javascripts.
Just keep the number of packages you use to a minimum. If some package itself has like 200 deps uninstall that and look for an alternative with less deps or think if you really need said package.
I also switched to Phoenix using Js only when absolutely necessary. Would do the same on Laravel at work if switching to SSR would be feasible...
Oh that's great news I will have to look at it again then. That was a huge turn-off for me, to take one of the most well respected and reliable eco systems and then to pull in one of the worst as a dependency. Thank you for clearing that up.
Node doesn't have any particular relation to NPM? You don't have to download 1000 other people's code. Writing your own code is a thing that you are legally allowed to do, even if you're writing in Javascript.
Yes, and you can code in assembly as well if you want it. But: that's not how 99% of the people using node is using it so that it is theoretically possible to code up every last bit yourself is true but it does not contribute to the discussion at all.
An eco-system, if it insists on slapping on a package manager (see also: Rust, Go) should always properly evaluate the resulting risks and put proper safeguards in place or you're going to end up with a massive supply chain headache.
Writing code yourself so as not to cultivate 1000 dependencies you can't possibly ensure the security of is not the same as writing assembly. That you even reach for that comparison is indicative of the deep rot in Javascript culture. Writing your own code is perceived as a completely unreasonable thing to be doing to 99% of JS-devs and that's why the web performs like trash and has breaches every other day, but it's actually a very reasonable thing to be doing and people who write most any other language typically engage in the writing of own code on a daily basis. At any rate, JS the language itself is fine, Node is fine, and it is possible to adopt better practices without forsaking the language/ecosystem completely.
> That you even reach for that comparison is indicative of the deep rot in Javascript culture.
Sorry?
No, I'm the guy that does write all of his code from scratch so you're entirely barking up the wrong tree here. I am just realistic in seeing that people are not going to write more code than they strictly speaking have to because that is the whole point of using Node in the first place.
The Assembly language example is just to point out the fact that you could plug in at a lower level of abstraction but you are not going to because of convenience, and the people using Node.js see it no different.
JS is a perfectly horrible little language that is now being pushed into domains where it has absolutely no business being used (I guess you would object to running energy infrastructure on Node.js and please don't say nobody would be stupid enough to do that).
Node isn't fine it needs a serious reconsideration of the responsibilities of the eco-system maintainers. See also: Linux, the BSDs and other large projects for examples of how this can be done properly.
I feel like there are merits to your argument but that you have a larger anti-JS bias that's leaking through. Not that there aren't problems with Node itself, but as many people have pointed out, there are plenty of organizations writing in Node that aren't pwn'd by these sorts of attacks because we don't blindly update deps.
Perfect is the enemy of good; dependency cooldown etc is enough to mitigate the majority of these risks.
I think JS is great. It's simple, anybody can use it.
TypeScript is excellent too. The structural type system is very convenient.
It's not going to replace Rust in cases where performance is essential or where you want strict runtime type checking or whatever, but for general use and graphical applications JS seems like a great pick.
I often hear people complain about JS, but really, how is it any worse than say Python?
I tell people this over and over and over: every time you use a third party dependency, especially an ongoing one, you should consider that you are adding the developers to your team and importing their prior decisions and their biases. You add them to your circle of trust.
You can't just scale out a team without assessing who you are adding to it: what is their reputation? where did they learn?
It's not quite the same questions when picking a library but it is the same process. Who wrote it? What else did they write? Does the code look like we could manage it if the developer quits, etc.
Nobody's saying you shouldn't use third party dependency. But nobody benefits if we pretend that adding a dependency isn't a lot like adding a person.
So yeah, if you need all of posthog without adding posthog's team to yours, you're going to have to write it yourself.
> I tell people this over and over and over: every time you use a third party dependency, especially an ongoing one, you should consider that you are adding the developers to your team and importing their prior decisions and their biases. You add them to your circle of trust.
Yes. If your shop is serious about security, it is in no way unreasonable to be building out tools like that in-house, or else paying a real vendor with real security practices for their product. If you're an independent developer, the entirety of Posthog is overkill, and you can instead write the specific features you need yourself.
We had created a sort of Posthog, but for product security analytics (1), and after 4 years of development I can confirm it's not something that you can easily create in-house.
If they have a HTTP API using standard authentication methods it's not that difficult to create a simple wrapper. Granted a bit more work if you want to do things like input/output validation too, but there's a trade-off between ownership there and avoiding these kinds of supply-chain attacks.
If you aim for 100% coverage of the API you're integrating with, sure. But for most applications you're going to only be touching a small surface area, so you can validate paths you know you'll hit. Most of the time you probably don't need 100% parity, you need Just Enough for your use-case.
To my understanding, there's less surface area for problems if I have a wrapper over the one or two endpoints some API provides, which I've written and maintain myself, over importing some library that wraps all 100 endpoints the API provides, but which is too large for me to fully audit.
Professionally I am a fulltime FE Dev using Typescript+React. My Backends for my side projects are all done in C#, even so I'd be fluent in node+typescript for that very reason. In a current side project, my backend only has 3 external package dependencies, 2 of which are SQLite+ORM related. The frontend for that sideproject has over 50 (React/Typescript/MaterialUI/NextJS/NX etc.)
.NET being so batteries-included is one of its best features. And when vulnerabilities do creep in, it's nice to know that Microsoft will fix it rather than hoping a random open source project will.
This a common refrain on HN, frequently used to dismiss what may be perfectly legitimate concerns.
It also ignores the central question of whether NPM is more vulnerable to these attacks than other package managers, and should therefore be considered an unreasonable security risk.
It's not just npm, you should also not trust pypi, rubygems, cargo and all the other programming language package managers.
They are built for programmers, not users. They are designed to allow any random untrusted person to push packages with no oversight whatsoever. You just make an account and push stuff. I have no doubt you can even buy accounts if you're malicious enough.
Users are much better served by the Linux distribution model which has proper maintainers. They take responsibility for the packages they maintain. They go so far as to meet each other in person so they can establish decentralized root of trust via PGP.
Working with the distributions is hard though. Forming relationships with people. Participating in a community. Establishing trust. Working together. Following packaging rules. Integrating with a greater dynamic ecosystem instead of shipping everything as a bloated container whose only purpose is to statically link dynamic libraries. Developers don't want to do any of that.
Too bad. They should have to. Because the npm clusterfuck is what you get when you start using software shipped by totally untrusted randoms nobody cares to know about much less verify.
Using npm is equivalent to installing stuff from the Arch User Repository while deliberately ignoring all the warnings. Malware's been found there as well, to the surprise of absolutely no one.
You can go very far with just node alone (accepts typescript without tsc, has testing framework,...). Include pg library that has no dependencies. Build a thin layer above node and you can have pretty stable setup. I got burnt so many times that I think it is simply impossible to build something that won't break within 3 months if you start including batteries.
When it comes to frontend, well I don't have answers yet.
You can write simple front-end without reactive components. Most pages are not full blown apps and they were fine for a very long time with jQuery, whose features have been largely absorbed into plain js/dom/CSS.
Node itself is still fine and you can do a lot these days without needing tons of library. No need for axios when we have fetch, there's a built-in test runner and assertion library.
There are some things that kind of suck (working with time - will be fixed by the Temporal API eventually), but you can get a lot done without needing lots of dependencies.
> Serious question: should someone develop new technologies using Node any more?
Serious answer: no.
I think I'm going to just use a static site generator, maybe add some WASM modules built with a language that has a sane package manager and enjoy my life instead of getting involved with this cluster of a show.
You need standalone dependencies, like Tailwind offers with its standalone CLI. Predators go where there prey is. NPM is a monoculture. It's like running Windows in the 90's; you're just asking for viruses. But 90% of frontend teams will still use NPM because they can't figure anything else out.
Yes, it's gonna be heuristics all way down. This problem isn't solved formally but the ecosystem(s) having these issues are too big to be discarded "just" because of that.
You have this issue with ALL external code though. npm/node and javascript overall may exacerbate this problem, but you have it with any other remote repository too - often without even noticing it unless you pay close attention; see the xz-utils backdoor, it took a while before someone noticed the sneaky payload. So I don't think this works as a selective filter against using node, if you have a use case for it.
Take ruby - even before when a certain corporation effectively took over RubyCentral and rubygems.org, almost two years ago they also added a 100.000 download limit. That is, after that threshold was passed, the original author was deprived of the ability to remove the project again - unless the author resigns from rubygems.org. Which I promptly did. I could not accept any corporation trying to force me into maintaining old projects (I tend to remove old projects quickly; the licence allows people to fork it, so they can maintain it if they want to, but my name can not be associated with outdated projects I already abandoned, since newer releases were available. The new corporate overlords running rubygems.org, who keep on lying about "they serve the community", refused to accept this explanation, so my time came to a natural end at rubygems.org. Of course this year it would be even easier since they changed the rules to satisfy their new corporate overlords anyway: https://blog.rubygems.org/2025/07/08/policies-live.html)
You forget to account for the fact that the xz-utils backdoor was extremely high effort. Literally a high skilled person building trust over time. While it's obviously possible and problematic, it's still a scaling/time issue.
Npm has been a shit show from day 1. Unfortunately, Industry momentum and vc funded "fail fast, fail often" is a hell of a drug.
EDIT: Coffee hasnt kicked in yet, that was harsher than I intended. For what it's worth, it's not specifically/solely NPM/nodes fault, more of a convergance of the above and the ecosystem/users just as much as any of the Node/NPM devs/maintainers in combination with it having such a large attack cross section. Even if it had a reputation for being bulletproof and secure as fuck there's still such a large userbase with huge potential if exploited that'd it'd almost assuredly inevitably be compromised from time to time regardless.
While I feel we could use a whole lot less javascript on the web (client and server side both), without a competitor or something, it's shear size ensures any such expliot/issue gets amplified 1000x versus nearly any other project save for maybe major OS's and Browsers themselves.
Node the technology can be used without blindly relying on the update features of npm. Vet your dependency trees, lock your dependency versions at patch level and use dependency cooldown.
This is something you also need to do with package managers in other languages, mind you.
> People use Node because of the availability of the packages, not the other way around.
That is not why I use Node. Incidentally, I also use Bun.js, and pnpm for most package management operations. I also use Typescript instead of raw JS.
I use Node and these related tools fundamentally because:
- I like the isomorphism of the code I write (same language for server and client)
- JS may have many warts, but IMO it has many advantages many other languages lack, it is rapidly improving, and TS makes it even more powerful and the bad part parts manageable. One ting that has stuck with me over the many years of using JS/TS is just how direct and free-of-ceremony everything is. Want a functional style? It supports it to some extent without much fuss. Want something akin to OOP? You can object literal with method-style function, "constructors" that are regular functions, even no-fuss prototypical inheritance, if you want to go that far. Also, no need for any complicated dependency injection (DI), you can just implement pure DI with regular functions, etc. I don't get why you hate JS/TS so much.
- I use Bun.js as an alternative to Node that has more batteries included, so that I can limit my exposure to too many external packages. I add packages only if I absolutely need them, and I audit them thoroughly. So, no, although I may use some packages, I am not on the Node ecosystem just because I want to go on a package consumption spree.
- I use pnpm for installing and managing package, and it by default prevents packages from taking any actions during installation; I just get their code.
That’s not a very good analogy. Doing what I suggested is not illegal and doesn’t prevent you from using packages from npm. It’s more akin to due diligence: before driving, you check that your car is safe to drive. At the gas and service station, you choose the proper fuel, proper lubricants and spare parts from a reputable vendor which are appropriate for your car.
Nobody - and I mean absolutely nobody - using Node.js has fully audited all of the dependencies they use and if we find somewhere in a cave a person that did that they are definitely not going to do it all over again when something updates.
I can guarantee that any financial institution which has standard auditing requirements and is using Node.js has fully audited all of the dependencies they use.
I should know, I check those companies for a living. This is one of the most often flagged issues: unaudited Node.js dependencies. "Oh but we don't have the manpower to do that, think about how much code that is".
I know what the standard requires. I also know what happens in practice and typically the auditors are understaffed, overworked and their technical expertise is lower than it should be. As a result a lot of stuff slips through the cracks.
What does get flagged though is not getting employee permission for putting photos on the 'team' page. It's good they flag that. I'd rather they also went in much deeper on tech issues.
I've reviewed 270 companies to date. I have yet to find a single one that had audited the source code they imported. It's not untypical to find an installation that automatically updates a whole raft of dependencies during the build phase. And absolutely nobody looks at that code until something breaks.
In my experience, most devs and companies don't consider the dependencies they load 'their' code.
They only look at the code they write, not everything they deploy.
The list of affected packages are all under namespaces pretty much nobody uses or are subdependencies of junk libraries nobody should be using if they're serious about writing production code.
I'm getting tired of the anti-Node.js narrative that keeps going around as if other package repos aren't the same or worse.
You need to explain how one is supposed to distinguish and exclude "namespaces pretty much nobody uses" when writing code in this ecosystem. My understanding is that a typical Node developer pretty much has no control over what gets pulled in if they want to get anything done at all. If that's the case, then you don't have an argument. If a developer genuinely has no control, then the point is moot.
How is this situation any different from any other ecosystem? I think you don't have an argument here other than that npm is a relatively large public repository. Bad actors and ignorant developers are everywhere else too.
There are plenty of npm features to help assess packages and prevent unintended updates, but nothing replaces due diligence.
All of them. The issue at hand is not limited to a specific language or tool or ecosystem, rather it is fundamental to using a package manager to install and update 3rd party libraries.
I never, ever, do development outside of a podman container these days.
Basically if I am going to run some code from somewhere and I haven't read it, it goes in a container.
I know its not foolproof, but I can't believe how often people run code they haven't read where it can make a huge mess, steal secrets, etc. I'll probably get owned someday, I'm sure, but this feels like a bare minimum.
Probably because it’s fine 99.99% of the time and humans aren’t intuitively good at handling risk that functions like that. Besides, security is something handed off to specialists to free the devs up to focus on building things in most companies. We’re not going to change that no matter how much it represents some ideal.
You are just reducing the blast radius with use of podman; you will likely need secrets for your app to work, which will be exposed regardless of the podman approach.
If you're developing in a container then you would have to be doing it without doing something like say, mounting your home directory into it.
The reality here is this is the sort of attack SELinux should be good at stopping (it's not because no one uses SELinux, the policies most commonly used don't confine the user profile in a useful way, and a whole bunch of tools love ambient credentials in environment variables).
>You have to make sure you're not putting any secrets in the container environment.
How does this work exactly?
containers still need env vars and access to databases and cloud environments. Without these the container is just useless isolated pod.
Not who you asked, but I have a similar setup. I can run everything I need for local development in that image (db, message queue emulator, cache, other services). So, setting things like environment variables or running postgres work the same as they do outside the container.
The image itself isn't the same image that the app gets deployed in, but is a portable dev environment with everything needed to build and run my apps baked in.
This comes with some nice side effects like being able to instantly spin up clean work environments on my laptop, someone elses, or a remote vm.
This really depends on your setup. If possible, I have local development containers as much as possible. nginx, postgres, redis, etc. I have several containers, each only has access to what it needs. We have an isolated cloud environment for development, in its own aws account.
Its not going to stop attacks, but it will limit blast radius a lot.
What do you mean? You can drop into bash in a container and run any arbitrary command, so `npm install foo` works just fine. Why would posthog's SDK be a special case?
I think the issue is more about what else has to go into or be connected to that container. Posthog isn't really useful if it's air-gapped. You're going to give it keys to access all kinds of juicy databases and analytics, and those NPM tokens, AWS/GCP/Azure credentials, and environment variables are exactly what it exfiltrates.
I don't run much on the root OS of my dev machine, basically everything is in a container or VM of some kind, but that's more so that I can reproduce my environment by copying a VMDK than in an effort to limit what the container can do to itself and data it has access to. Yeah, even with root access to a VM guest, an attacker they won't get my password manager, personal credit card, socials, etc. that I only use from the host OS... But they'll get everything that the container contains or has access to, which is often a lot of data!
You're severely limiting the blast radius. This malware works by exfiltrating secrets during installation, if I understood it correctly. If you would properly containerize your app and limit permissions to what is absolutely required, you could be compromised and still suffer little to no consequences.
Of course, this is not a real defense on its own, its just good practice to limit blast radius, much like not giving everybody admin rights.
> Upon execution, the malware downloads and runs TruffleHog to scan the local machine, stealing sensitive information such as NPM Tokens, AWS/GCP/Azure credentials, and environment variables.
Even a properly containerized app will still have these things, because you need things like environment variables (that contain passwords, api keys, etc) for your app to function.
You could create/run thin proxies for every external service that handle the auth side, and run each in a separate container. Orchestrate everything with docker-compose. Need to connect to cloud services for local development? Have a container with a proxy that transparently handles authentication. Now only that container has the secrets for talking to that service.
That's a lot of work though, and increases the difference between your local dev environment and prod.
Sure, but only the container is affected and it is always your responsibility to grant as little access as possible to the various credentials you may need to supply that environment. AFAICT with this worm, if you don't supply write-level GitHub credentials to the container (and you shouldn't!) and you install infected packages, the exploit goes no further.
Because you need your application code to interact with Posthog's code. But if they're running in separate containers...how are you doing that. Surely you are not writing an api layer for every npm dependency you use.
This is smart and a good first step. Everyone can't be trusted to do the security dance flawlessly, though. We need sane defaults. Least privilege by default for 3rd-party code. Deno's headed in the right direction with this. But I think the solution needs to exist deeper in the stack. The surge in popularity of `curl -fsSL https://my-cool-ai-starup.ai/install.sh | bash` style installers is particularly concerning to me in this regard.
Using Podman over Docker is probably an even safer bet in that regard. But QEMU or something for an extra layer of safety and paranoia is probably the next best thing.
Why the biggest package mess is always with the Node ecosystem?
Why in particular this community still insists on preemptively updating all deps always, on running complicated extra hooks together with package installation and pretending this all is good engineering practices? ("Look, we have so plenty of things and are so busy, thus it must be good")
Why certain kind of mindset is typical to this community?
Why the Node creator abandoned his creation years ago?
Node is the new PHP: massive ecosystem, enormous mind share, and made up for a large part by programming newbies for whom JS is their first language and have no serious understanding of engineering practices, security and quite eager to use a library rather than wasting hours figuring out how to pad a string.
Because it is not a serious ecosystem run by serious people. Do you know what serious people do? They have package repositories with people called "maintainers", who are, crucially, trusted members of a community who don't write the software they package. "Oh but that's GATEKEEPING!", they screech. Yes, that's the entire point. Gatekeeping prevents shit like this from happening. There's a reason why this doesn't happen to Debian, but JavaScript developers get defensive and mean when you suggest that maybe the equivalent of a public S3 bucket isn't the best way to host a package repository.
Feels good, just for a second, to type pretence you're above everyone doesn't it? Just for those few seconds, you're better than a big whole arbitrary collection of people, and for those few seconds you have relief from the reality of your life.
The website is a mess (broken links, broken UI elements, no about section)
There is no history on webarchive. There is no information outside of this website and their "customers" are crypto exchanges and some japanese payment provider.
This seems a bit fishy to me - or am I too paranoid?
GitLab's write-up mentions a dead man's switch where "The malware continuously monitors its access to GitHub (for exfiltration) and npm (for propagation). If an infected system loses access to both channels simultaneously, it triggers immediate data destruction on the compromised machine. "
Because PostHog's "Talk to a human" chat instead gets a grumpy gatekeeping robot (which also doesn't know how to get you to a working urgent support link), and there's nothing prominently on their home page or github about this:
Ahhh, TBH I didn't look on the right. I dug through the menu on the left, thinking the right hand bar (which has the rotated labels) was all getting-started/docs related things. In my defense I have a fairly wide monitor and tend to full-screen the browser.
Your status page isn't clear, but are all versions between the compromised and "safe to install" versions compromised or just the ones listed?
For example I installed `posthog-react-native` version `4.12.4` which is between the `4.11.1` version which is compromised and the safe to install version `4.13.0`. Is that version compromised or not?
Have a slack channel with them, these are the versions they mentioned:
posthog-node 4.18.1
posthog-js 1.297.3
posthog-react-native 4.11.1
posthog-docusaurus 2.0.6
Parent comment is an indirect reference to US mass shootings:.
> "'No Way to Prevent This,' Says Only Nation Where This Regularly Happens" is the recurring headline of articles published by the American news satire organization The Onion after mass shootings in the United States.
I would say that npm likely has easier solutions here compared to Cargo.
Well before the npm attacks were a thing, we within the Rust project, have discussed a lot of using wasm sandboxing for build-time code execution (and also precompiled wasm for procedural macros, but that's its own thing.) However the way build scripts are used in the Rust ecosystem makes it quite difficult enforce sandbox while also enabling packages to build foreign code (C, C++ invoke make, cmake, etc.) The sandbox could still expose methods to e.g. "run the C compiler" to the build scripts, but once that's done they have an arbitrary access to a very non-trivial piece of code running in a privileged environment.
Whereas for Javascript rarely does a package invoke anything but other javascript code during the build time. Introduce a stringent sandbox for that code (kinda deno style perhaps?) and a large majority of the packages are suddenly safe by default.
This is a cultural problem created through a fundamental misunderstanding (and mis-application) of Unix philosophy. As far as I'm aware the Rust ecosystem doesn't have a problem appropriately sizing packages which in turn reduces the overall attack surface of dependencies.
It's kinda funny because "Unix philosophy" was never a coherent thing in the first place. Arguably Plan 9 came the closest to that in practice, but, well, you might note it didn't exactly have a strong uptake. Unix itself is a pile of hacks though, and it's both sad and amusing to watch people trying to divine some kind of methodology out of that.
This has nothing to do with package sizes. Cargo was just hit with a phishing campaign not too long ago, and does still use tokens for auth. NPM just has a wider surface area.
It's not just the popularity, it's partly the update mechanism and partly the culture. In what sane world would you always pull in all the newest things, regardless or whether you need them or not? This is a default at build time for so many setups. If you absolutely must use that package manager, at least lock down your versions, and update selectively. I don't even know if that's possible to do with the dependencies' dependencies (and so on), or are people forced to just pull in whatever, every time.
An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series", as is the de-facto standard in NPM dependencies. Therefore, it's up to each dependency-user to validate newer versions of dependencies before publishing a new version of their own package. Lots of manual attention needed, so a slower pace of releases. This is a good thing!
Another example: all Debian packages are published to unstable, but cannot enter testing for at least 2-10 days, and also have to meet a slew of conditions, including that they can be and are built for all supported architectures, and that they don't cause themselves or anything else to become uninstallable. This allows for the most egregious bugs to be spotted before anyone not directly developing Debian starts using it.
You forgot to mention it is also tied to provable namespaces. People keep saying that NPM is just the biggest target...
Hate to break it to you but from targeting enterprises, java maven artifacts would be a MASSIVE target. It is just harder to compromise because NPM is such shit.
Maven Central verifies the domain used for the package namespace, too. You need to create a DNS TXT entry with a key.
This adds a bit more overhead to typo squatting, and a paper trail, since a domain registrar can have identity/billing information subpoenaed. Versus changing a config file and running a publish command...
Maven central also requires package signing. You're not stealing my signing key. It's on a yubikey. Game over, you can't publish malware in my name using my key.
> An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series"
You can definitely do this.
To be honest, you just end up with the same thing via dependabot/renovate.
You can specify a dependency version range in Maven artifacts. But the Maven community culture and default tooling behaviour is to specify exact versions.
You can specify an exact dependency version in npm packages. But the npm community culture and default tooling behaviour is to specify version ranges.
Even if a maintainer uses a bot to bump dependency versions, most typically they will test if their package works before publishing an updated version, and also because this release work is manual (even if the bot helps out), it takes some time after the dependency is released for upstream consumers of it to endorse and use it. Therefore, nobody consuming foo 1.0.4 will use dependency bar 2.3.5 until foo 1.0.5 is released... whereas an npm foo 1.0.4 with bar dependency "^2.3.0" will give its users bar 2.3.6 from the very moment bar 2.3.6 is released, even without a foo 1.0.5 release.
Other languages seem to publish dependencies as self-contained packages whose installation does not require running arbitrary shell scripts.
This does not prevent said package from shipping with malware built in, but it does prevent arbitrary shell execution on install and therefore automated worm-like propagation.
You have separate people called "maintainers", and they're the ones who build and upload packages to the repository. Crucially, they're not the people who write the software. You know, like Linux has been doing since forever. https://wiki.debian.org/DebianMaintainer Instead of treating your package repository like a trash can at a music festival, you can treat it more like a museum, curated by experts. Unfortunately, this isn't quite the devil-may-care attitude the Node ecosystem is so accustomed to, and will be met with a lot of whining, so it never happens. See y'all in two weeks when this happens again.
That literally makes no difference at all. You’ll just vendor the malicious versions. No, a lock file with only exact versions is the safe path here. We haven’t seen a compromise to existing versions that I know of, only patch/minor updates with new malicious code.
I maintain that the flexibility in npm package versions is the main issue here.
You are using the word differently than everyone else I think. I’ve never heard someone using that word to mean maintain private forks. Then again, even private forks don’t protect you much more than package lock files and they are way more overhead IMHO.
You still need some out-of-band process to pull upstream updates and aside from a built-in “cool down” (until you merge changes) I see that method as having a huge amount of downside.
Yes, you sidestep malicious versions pushed to npm but now you own the build process for all your dependencies and you have to find time to update (and fix builds if they break) all your dependencies.
Locking to a specific version and waiting some period of time (cool down) before updating is way easier and jus as safe IMHO.
Vendoring literally just means grabbing the source code from origin and commit it to your repo after a review.
The expectation that every repo has important regular updates for you is pure FOMO. And if I don't do random updates for fun, nothing will every break.
> Version locking wont help you all the time, i.e. if you build fresh envs from scratch.
I'm confused on this. I would imagine it would protect/help you as long as releases are immutable which they are for most package managers (like npm).
> Vendoring literally just means grabbing the source code from origin and commit it to your repo after a review.
Hmm, I don't think it always necessarily means grabbing the source, it can also mean grabbing the built artifacts in my experience.
My biggest issue with vendoring dependencies is it allows for editing of said dependencies. Almost everywhere I've worked that vendored dependencies (copied source or built versions in and committed them) felt the siren song of modifying said dependencies which is hell to deal with later.
You are right about version locking, bullshit on my side, not sure what I was thinking.
I personally don't have a problem with the general ability to change vendor code. The question is whether you want it in an specific case or not. If you update frequently then certainly not. But that decision should be deliberate team policy.
> I personally don't have a problem with the general ability to change vendor code. The question is whether you want it in an specific case or not. If you update frequently then certainly not. But that decision should be deliberate team policy.
Fair, in the instances I ran into it was code that was downloaded and unzipped into a "js-library-name" folder but then the code was edited, even worse, the `.min.js` version wasn't touched, just the original one which led to some fun when someone "helpfully" switched to the min versions that didn't have the edit. IMHO, if you want to edit a library then you should be forking and/or making it super obvious that this is not "stock" library X. We also ran into issues with "just updated library Y" and only later realizing someone had modified the older version.
But yes, if it's deliberate and obvious then I don't have an issue modifying, just as long as the team understands it's "their" code now and there is no clean upgrade path in the future.
I'd require vendor code being committed to git and integrated into the CI/CD pipeline. It should be treated as if you own it, just with a policy whether you want to change it or not.
The difficulty to make changes obvious is same for forks and vendored commits, imho. You can write big warnings in commit messages, that's it I guess. Which kind of boils down to deliberate team policy again. But I generally prefer monorepos for various reasons.
I think some system would need to dynamically analyze the code (as it runs) and record what it does. Even then, that may not catch all malicious activity. It's sort of hard to define what malicious activity is. Any file read or network conn could, in theory, be malicious.
As a SW developer, you may be able to limit the damage from these attacks by using a MAC (like SELinux or Tomoyo) to ensure that your node app cannot read secrets that it is not intended to read, conns that it should not make, etc. and log attempts to do those things.
You could also reduce your use of external packages. Until slowly, over time you have very little external dependencies.
Hire an antivirus company to provide a safe and verified feed of packages. Use ML and automatic scanners to send packages to manual review. While Halting problem prevents us from 100% reliably detecting malware, at least we can block everything suspicious.
The list of packages looks like these are not just tiny solo-person dependencies-of-dependencies. I see AsyncAPI and Zapier there. Am I right that this seems quite a significant event?
AsyncAPI is used as the example in the post. It says the Github repo was not affected, but NPM was.
What I don't understand from the article is how this happened. Were the credentials for each project leaked? Given the wide range of packages, was it a hack on npm? Or...?
> it modifies package.json based on the current environment's npm configuration, injects [malicious] setup_bun.js and bun_environment.js, repacks the component, and executes npm publish using stolen tokens, thereby achieving worm-like propagation.
This is the second time an attack like this happens, others may be familiar with this context already and share fewer details and explanations than usual.
I don't get this explanation. How does it force you to run the infection code?
Yes, if you depend on an infected package, sure. But then I'd expect not just a list, but a graph outlining which package infected which other package. Overall I don't understand this at all.
I still don't get it. Like, I understand that if you apply the diff you get infected. But... why would you apply the diff? How would you trick me to apply that diff to my package?
Someone could be tricked into giving their npm credentials to the attacker (e.g. via a phishing email), and then the attacker publishes new versions of their packages with the malicious diff. Then when the infected packages are installed, npm runs the malicious preinstall script which harvests secrets from the new machine, and if these include an npm token the worm can see which packages it has access to publish, and infect them too to continue spreading.
For anyone publishing packages for others to use: please don't pin exact dependency versions. Doing so requires all your users to set "overrides" in their own package.json when your dependencies have vulnerabilities.
Most of the best practices can be translated to python ecosystem. It’s not exact 1:1 mapping but change few key terms and tools, the underlying practices should be the same.
Or copy that repo’s markdown into an llm and ask it to map to the pip ecosystem
Because install scripts are being actively exploited, so blocking them will reduce your exposure. Install scripts will also run anywhere that runs npm ci, npm install, etc., including build pipelines.
> Can't they just jam the malware into the package itself
Yes. Disabling install scripts won't safeguard you from all attack vectors.
Yes, if the malware is injected in the application code this doesn’t prevent it.
But in some cases it could help for that. For instance, if the package runs in the browser and the payload requires file-system access, etc., then the attack can’t execute in the browser. And if in addition it was added to a life-cycle script, it would be mitigated.
At any rate, it’s worth having `ignore-scripts=true` because NPM life-cycle scripts are a common target (e.g., this one targets `preinstall`).
It gets tricky with private dependencies, then you have to pass some sort of token into the container to authenticate with the host when installing dependencies.
The solutions that are effective also involve actually doing work, as developers, library authors, and package managers. But no, we want as much "convenience" as possible, so the issues continue.
Developers and package authors should use a lockfile, pin their dependencies, be frugal about adding dependencies, and put any dependencies they do add through a basic inspection at least, checking what dependencies they also use, their code and tests quality, etc.
Package managers should enforce namespacing for ALL packages, should improve their publishing security, and should probably have an opt-in verified program for the most important packages.
Doing these will go a long way to ameliorate these supply chain attacks.
There absolutely is an easy solution to these problems, and Linux has been doing it forever: package maintainers. Don't treat your repository like a superfund site, and it won't fill up with garbage.
I think if you generally depend on npm packages, being frugal is hard, because every random package works against you.
Last time my perception was also that publishing sec is a weak point. If at least heavily used packages would be forced to do manual security steps for publishing, it would help quite a bit as long the measures a safe.
The e18e community are reducing dependencies in popular libraries and building tools to prevent and reduce the impact of such attacks. Join if you want to help out! https://e18e.dev/
This is a good sign that it's time to get packages off of NPM and come up with an alternative. For those who haven't heard of or tried Verdaccio [1], it may be an option. Relatively easy to point at your own server via NPM once you set it up.
I've had decent luck running it locally, but claude keeps screwing up the cool-down settings in my monorepo.
This is probably a common problem. Has anyone gotten verdaccio to enforce cool-down policies?
I also waste a ton of time because post-install scripts are disabled. Being able to cut them off from network access, and just run a local server with 2-4 week cool-down would help me sleep better at night + simplify the hell out of my build.
I always (very naively, I fully get it) wonder if someone at GitHub could take a minute and check the logs (if there are any at this level) from a week ago or so and scan them for patterns? The code seems to grab a few files off of GitHub, use Github actions, etc. -- perhaps there's a pattern in there that shows the attacker experimenting and preparing for this? I assume most people at this level have VPNs and so forth, but I'd never underestimate the amount of bad luck even those folks can have. Would be interesting, I know I'd have a look, if those logs existed.
Some MFA requirement to publish a new version of the package would be a good idea. In me experience releasing a new version of software is a big enough deal that the product owner is on hand to authorize the release via a separate device no matter how automated the pipeline is.
Why does every major Javascript vulnerability come off as something that would be easily avoided by not doing obviously stupid things (in this case automatically updating packages with no authentication, testing or oversight)?
I looked through some of the GH repositories and - dear god - there are some crazy sensitive secrets in there. AWS Prod database credentials, various API keys (stripe, google, apple store, ...), passwords for databases, encryption keys, ssh keys, ...
I think hijacked NPM packages are just the tip of the ice berg.
MSFT can't even sort Windows out, if they can't charge for it in an Enterprise license, it gets left behind. Which is why W11 is a marketing grab bag of trash.
Does NPM use any automatic scanners? Just scanning for eval/new Function/base64 and other tokens often used by malware, and requiring a manual review, could already help.
Static scanning won't help. You can write this["eval"]() instead of eval(), therefore you can write this["e" + "v" + "a" + "l"](), and you can substitute (!![]+[])[!+[]+!+[]+!+[]] for "e", (![]+[])[+!+[]] for "a" (and so on: https://jsfuck.com/)
In this Turing-equivalent world, you can only know what actually executes (e.g. eval, fetch) by actually executing all code in the package and then see what functions got executed. Then the problem is the same as virus analysis; the virus can be written to only act under certain conditions, it will probe (e.g. look at what intepreter fingerprints, get the time of day, try to look at innocuous places in filesystem or network, measure network connection times, etc), so that it can determine it is in a VM being scanned, and go dormant for that time.
So the only thing that actually works is if node and other JS evaluators have a perfect sandbox, where nothing in a module is allowed (no network, no filesystem) except to explicit locations declared in the module's manifest, and this is perfectly tracked by the language, so if the module hands back a function for some other code to run, that function doesn't inherit the other code's network/fs access permissions. This means that, if a location is not declared, the code can't get to it at scanning time nor install time nor any time in the future.
This still leaves open the door for things like a module defining GetGoogleAnalyticsURL(params) that occasionally returns "https://badsite.com/copyandredirect?ga=...", to get some other module to eventually make a credential-exfiltrating network call, even if it's banned from making it directly or indirectly...
Well, writing obfuscated code like ["e" + "v" + "a" + "l"]() is already a huge red flag for sending the package to manual review. While it might be impossible to detect all methods of obfuscation, we could start with known methods.
Also, detecting obfuscated code sounds like an interesting and challenging task.
Deciding to put your resources into something that only a really stupid criminal would be caught by gives you a false sense of security.
Literally scanning for just "eval(" is entirely insufficient. You have to execute the code. Therefore you have to demand module authors describe how to execute code, e.g. provide a test suite, which is invoked by the scanner, and require the tests to exercise all lines of code. Provide facilities to control the behaviour of functions outside the module so that this is feasible.
This is a lot of work, so nobody wants to do it, so they palm you off with the laziest possible solution - such as literally checking for "eval(" text in the code - which then catches zero malware authors and wastes resources providing help to developers caught as a false positive, meanwhile the malware attacks continue unabated because no effective mechanism to stop them has been put in place.
Reminds me of the fraudster who sold fake bomb detectors to people who had a real need to stop bomb attacks. His detectors stopped zero bomb attacks. https://www.bbc.co.uk/news/uk-29459896
> Deciding to put your resources into something that only a really stupid criminal would be caught by gives you a false sense of security.
Interestingly enough, this is the premise for a lot of security in the physical world. Broken windows theory, door locks as a form of security in the first place, crimes of opportunity, etc.
But one should consider that in tech, the barrier to entry is a little higher and so maybe there are less 'dumb' criminals (or they don't get very far).
a concern i have is that it's only a matter of time before a similar attack is done to electron based apps (which also have packages installed using npm). probably worse because it's installed in your computer and can potentially get any information especially given admin privileges.
I’m starting an electronjs project in a few weeks and have been reading up on it. They make a big deal about the difference between the main and renderer processes and security implications. The docs are there and the advice given but it’s up to the developers to follow them.
That leads me to another point. Devs have to take responsibility for their code/projects. Everyone wants to blame npm or something else but, as software developers, you have to take responsibility for the systems you build. This means, among may other things, vetting code your code depends on and protecting the system from randomly updating itself with code you haven’t even heard about.
I was looking at the packages affected and there are some recognizable names.
The ones that got my attentions are the @ensdomains/*, that are the legit packages and are probably in every Ethereum/EVM/blockchain related apps for the resolution of decentralized domain names.
A quick search shows those Ledger hardware wallet use those libs too [0]
I cannot believe how long it's taking github to shut this down, it's still going, and all they need to do is block a 18 char alphanumeric repo name from creation for this specific wave.
So github has some tools available to mitigate some of the problems tied to it. Probably not perfect for all use cases. But considering the current scale, it doesn't seem to have any effect, as enough publishers seem not to care.
I think npm should force higher standards on popular packages.
I don’t feel confident there is a way to use npm safely. The basic problem is that of curation, and there not being any except for what you do yourself. Every day brings new surprises, and osv.dev’s npm feed is a continuous horror show.
I would love to see the equivalent of a linux distro, a curated set of packages and package versions that are known to be compatible and safe. If someone offered this as a paid product businesses would pay for it.
> Upon execution, the malware downloads and runs TruffleHog to scan the local machine, stealing sensitive information such as NPM Tokens, AWS/GCP/Azure credentials, and environment variables.
That's a wake up call to harden your operations. NPM Tokens, AWS/GCP/Azure credentials have no reason to be available in environments where packages may be installed. The same goes for sensitive environment variables.
I agree it's hard. But it's actually easier in professional settings. There are funds and you don't have an excuse to be lazy.
At minimum whatever you are working on should be built in docker. The package installation then would happen during the image build step. Yes it's easy to break out of the isolation environment but i am betting this malware does not.
NPM tokens should exist in some configuration/secret management solution not on your home directory. Devs have no business holding the NPM tokens. Same goes for sensitive environment variables they have no business existing on dev laptops or even the pipeline build steps (where package installation should happen).
AWS etc credentials / tokens are harder to secure since there are legit reasons for existing in dev laptops.
Docker is also not a silver bullet. Again, what you're claiming to be easy is often times exceedingly difficult or frictional, especially on established teams. I don't disagree that comparmentalization is important but security solutions are only as effective as their practical feasibility.
My code editor works in a sandbox. It's difficult because Linux doesn't provide it and one has to write it manually using shell scripts, random utilities. For example, I had also to write a limited FUSE emulation of /proc to allow code editor work without access to real /proc which contains lot of unnecessary information.
And if it's a "professional" setting, the company could hire a part-time developer for writing the sandbox.
I am using bwrap - it allows blocking network, creating a new process namespace (so that sandboxed program cannot mess with host processes) and creating a new filesystem, where you can mount host directories. I only pass through minimal set of directories like /usr and /lib, and only selected files from /etc, as they can contain valuable information. I also pass through several device files like /dev/null, but not the whole /dev.
I do not pass through /proc, as it contains not only information about processes (which is not a concern), but a lot of identifying information about a machine. However, lot of modern software due to poor coding and not following standards crashes or exhibits various bugs without /proc, so I wrote a minimal Python /proc emulator using FUSE: https://pastebin.com/s49V5nVL . /proc doesn't allow bind mounting its subdirectories somewhere else, so the only option was FUSE (or writing a kernel module). As you see, with Linux you learn something new every day, like it or not. The emulator is not perfect - it doesn't expose thread information, and, for example, Grand Central Dispatch library [1] terminates the application because of this. Telegram is using the library and crashes randomly because of this. I expected better quality from Apple and Swift.
Most importantly, I run the app as a different user (Android-style), because historically Linux was built around idea of protecting users from each other, and I don't trust namespaces as much. I clear the environment variables as well. Ideally, I should also be closing file descriptors to prevent leaks (for example, you can easily leak a file descriptor to your Wayland server, Dbus or something else).
To use Xorg, I grant access to the app's user. X server has no issues with sharing access to it. I am not sure if one can use Wayland with different users (is uses unix sockets). Pipewire doesn't allow its use by different users, so I have to add a pipewire-pulse plugin listening on localhost, and configure ALSA inside sandbox to connect to that pulseaudio server. Obviously, the app in sandbox has no direct access to audio devices in /dev.
Dbus is also necessary to display tray icons, but you should never grant direct access to it because it is almost like giving the root password. I use xdg-dbus-proxy [2] which I had to patch because originally it only supported running the sandboxed app under the same user. You can still find my patch in open pull requests.
I also did some experimentation with passing through a camera through a fake video4linux device, so that the app cannot know the real camera model, but it is not working great, for example, Gnome Cheese app cannot see it, because it needs not only a kernel device, but something more to detect cameras.
The sandbox cannot use hardware acceleration. It would be not easy to add because every GPU vendor uses its own directories and device files, /sys entries to interact with the hardware and I think you can get the full list only by reverse engineering. Furthermore, I am not sure if it is safe.
This setup is not perfect, and looks like a bunch of random utilities glued with shell scripts (which is not uncommon in Linux world). If I had more time, I would write something better, that can dynamically add and revoke privileges, can provide fake serial numbers and even emulate a working network which sadly loses 100% of sent packets.
I looked into using Flatpak, but they seem to grant full access to /sys, /proc and generally I don't feel that they care much about protection from malware and fingerprinting by legitimate software. Also, as I understand, they run the app as the same user which is not secure. Probably their primary goal is making installation easier but not making the use of software safer. I am not going to use Docker because it officially says that it is not a security tool.
Sure it is: don't do it. It's not like there isn't automated tooling for this, but you can also like... I don't know, look at your diffs and not commit secrets? I've never committed a secret before, and I've been working with AWS for ten years now. But I don't "git commit -a" and I triple-check my diffs.
Upgrading once a month is insane at any rate, I could see the point in upgrading maybe once a year. For stable projects, you're very much fine upgrading only when there's a vulnerability or you need something from a newer release. Upgrade when you actually need to and use stable versions that have been out for a while, no need to hamster wheel it.
When I worked in commercial aerospace, before we even shipped live there was an incident with a CERT advisory against the XML package we were using. But the fix was only added to the current major version and we were stuck one behind. It took ~3 of our best problem solvers about a week to get that damned thing upgraded. Which put us behind on our schedule.
This made some of my more forward thinking coworkers nervous because what if this happened after we went live? So we started a repeating story called “upgrade dependencies” and assigned it round robin once a month to someone on each application. Every time someone got it the first time they would ask me, “but upgrade what?” Whatever you want, but preferable something that hasn’t been in a while.
For IP and security reasons we were already on vendored dependencies, so it was pretty straightforward to tell what was old. But that made “upgrade immediately” problematic if fixes weren’t back ported far enough and we didn’t want that live.
Whats the most full proof way of defending ourselves from such attacks? My opinion is that the applications should never deal with credentials at all. Sidecars can be run which can inject credentials in real time. These sidecars can be under tight surveillance against such attacks. After all, application code is the most volatile in an organization.
I think we shouldn't be using package repositories in this way at all, shouldn't it be much better to have a package system like golang has where you directly import the sourcecode from github? You get around an entire class of problems. At least now you can only be compromised if the github source code itself is compromised, not any part of some build pipeline or a tool like npm or an npm registry. That means to vendor everything and only upgrade if you need to upgrade, treat all the code like you are responsible for it all because you are. The entire concept of relying on builds of other people is part of the problem, it's bad enough that we rely on source code of other people but that goes with the territory. Relying on their build systems is not as mandatory.
pnpm’s minimumReleaseAge can help a ton with this. There’s a tricky balance, because allowing your dependencies to get stale makes you inherently more vulnerable to vulnerabilities in your packages. And, critically, fixing a vulnerability in an urgent situation (i.e. you were compromised) gets increasingly harder to address the more stale your dependencies are.
minimumReleaseAge strikes a good balance between protecting yourself against emerging threats like Shai-Hulud and keeping your dependencies up-to-date.
Because you asked: you can get another layer of protection through Socket Firewall Free (sfw), which prevents dependencies known to be malicious from being installed. Socket typically identifies malware very soon after its is published. Disclaimer: I’m the lead dev on the project, so obviously biased — YMMV.
I was working with the assumption in this model the attestation is signed by ephemeral keys (OIDC) which would reveal the bad actor or give breadcrumbs. Enough to reduce incentives to hijack packages.
I'm guessing no one yet wants to spend the money it takes for centralized, trusted testing where the test harnesses employ sandboxing and default-deny installs, Deterministic Simulated Testing (DST), or other techniques. And the sheer scale of NPM package modifications per week makes human in the loop-based defense daunting, to the point that only a small "gold standard" subset of packages that has a more reasonable volume of changes might be the only palatable alternative.
What are the thoughts of those deep inside the intersection of NPM and cybersecurity?
It will keep happening until someone takes responsibility and starts maintaining the whole of the node eco system. This is probably a viable start-up idea: Node but audited.
You don't even need to enshittify Yet Another Service, you just need package maintainers. Debian manages to do this, and I'm guessing they get paid nothing (although, yeah, Amazon and The Goog really ought to chip in a few bucks, considering their respective empires). Unfortunately, it means you can't just YOLO your code into other people's programs anymore.
Oh, agreed 100%. I find it endlessly frustrating that these same conversations happen every single time there's a supply chain attack like this, because nobody wants an _actual_ solution, they want an _easy_ solution that doesn't involve changing anything about how they work. So we just get 500 comments asking if we can solve the Halting Problem, and then everyone forgets until the next breach. It was ever thus.
Nah - dependency cooldown is all the rage but it’s only effective if you have some noncompliant canary users. Once everyone is using it it will cease to be effective because nobody will be taking the first step/risk until everybody does.
The point of the cooldown is to allow time for vendor scans to complete and for compromised packages to be pulled. It's not about waiting for an end user to notice they've been compromised.
> Meanwhile, the aforementioned vendors are scanning public indices as well as customer repositories for signs of compromise, and provide alerts upstream (e.g. to PyPI).
Depending on “security vendors” to do scans of every single update seems naive and over optimistic to me, but hey - everyone’s jumping on the bandwagon regardless of what I think so I guess we’ll see soon.
99% of releases do NOT fix zero-days. But 100% of releases have a small risk of introducing a backdoored build-script.
There's nothing wrong with pinning dependencies and only updating when you know for sure they're fixing a zero-day (as it will be public at that point).
Or an old enough version. For one of the most damaging zero-day vulnerabilities in the Java ecosystem (log4shell), you were vulnerable if you were in the latest version, but not vulnerable if you were using an old enough version.
Zero-day on frontend has not really a y effect, except on one user at a time. Zero-day on a server though ... perhaps we arrive at the conclusion to not use the JS ecosystem on the server side.
Not sure if you're serious, but if so I agree that people should take the time to set up their own package mirrors. Not just for npm but all other package managers as well.
This is why it's so important to get to know what you're actually building instead of just "vibing" all the time. Before all the AI slop of this decade we just called it being responsible.
Exactly, there is no easy solution to these problems.
The solutions that are effective also involve actually doing work, as developers, library authors, and package managers. But no, we want as much "convenience" as possible, so the issues will continue.
Developers and package authors should use a lockfile, pin their dependencies, be frugal about adding dependencies, and put any dependencies they do add through a basic inspection at least, checking what dependencies they also use, their code and tests quality, etc.
Package managers should enforce namespacing for ALL packages, should improve their publishing security, and should probably have an opt-in verified program for the most important packages.
Doing these will go a long way to ameliorate these supply chain attacks
Maintaining a package mirror is a shared responsibility beyond just the software dev teams. The packages and their publishers need to be approved to be added to the mirror, testing needs to occur, and updates are delayed until the newer version is added to the mirror. The network team would block npm and force all machines to use this mirror.
All this would have mitigated this incident in the event that an npm install was done during the window of this update being rolled out and unpatched. The npm install would continue as normal on the last known good version and the newer vulnerable version would simply not exist on the mirror.
"Our security engineering team is investigating the matter and thus far has concluded that while some public Postman NPM packages were infected, (1) Postman as an app is not compromised, and (2) our production cloud services are also not compromised."
I’ve been tracking these supply-chain incidents as well, so I built a small
real-time scanner that looks for suspicious SHA-1 patterns inside repositories.
It’s basically a lightweight CLI tool you can run directly inside any local
project:
It’s not meant to be a full security product — just a simple “first-pass”
detector that helps catch unexpected checksum strings or injected artifacts
before they slip into CI. Feedback and contributions are welcome!
Made a package (that I needed personally), to easily reinstall all dependencies (using the same versions) in a project and check them using Aikido's safe chain for malware (supported npm, pnpm, bun, and yarn). It also easily switches a project's package manager to another. https://www.npmjs.com/package/eazypm
Yep. This is what I do. I edit and run my code in a container. That container cannot access my ssh keys or publish to GitHub. I review all changes, and manually commit / publish on my host. It’s not perfect, but that plus vendoring my dependencies goes a long way towards mitigating these kinds of things.
Ok, I think the verdict on the "JavaScript for everything" experiment is in. It was already resolved long ago (in my opinion), but this should convince any stragglers. Let's accept that the one thing JS is really great for is DOM patching, and move on.
Going forward, use WASM if you really want to make an SPA (and think about that choice), where the source language is not something that ties into the JS dependency ecosystem. Ban it and burn it with fire for anything on the backend, for christ.
How do you test your projects if there are any infected/affected dependencies used? As i understand it could also be a dependency of a dependency ... that could be affected?
What is going on with this website though? It gives cursor stutter and slow scrolling. It seems like we now need an insane amount of CPU to read static text. What a regression.
as someone that likes the npm package ecosystem but is not fond of malware - ended up building a CLI wrapper around npm to only install packages older than a configurable amount of days.
I am amazed at the dates in this article. This compromise appears to have been discovered literally this morning. Incredibly fast turnaround on this article.
Whats the most full proof way of defending ourselves from such attacks? My opinion is that the applications should never deal with credentials at all. Sidecars can be run which can inject credentials in real time. These sidecars can be under tight surveillance against such attacks. After all, application code is the most volatile in an organization.
To me this is asking the question of "what's the safest way to drink from a polluted river".
The answer is really, don't.
NPM and the JS eco-system has really gone down a path of zero security and they're paying the price for it.
If you really need libraries from NPM and whatnot, vendorize them so you're relying on known-safe files and don't arbitrarily update them without re-verification.
This is why I am not a huge fan of separate package managers for libraries, such as in the case of rust, or node. The C style of sharing deps. couldn't really be simpler as just including the headers in your Makefile.
We really don't need more package managers other than the ones provided by your operating system, but I dunno maybe its just me.
That ship has sailed, traveled around the world, and docked in a foreign port at this point.
Including headers isn't remotely "simple". There's so many considerations in linking, .SO version compatibility, architecture and instruction set issues, building against multiple versions on the same system. Or if you want to feel frustrated in a single word: GDAL (IYKYK)
And that's only where #include is even applicable. That is not gonna fly for any interpreted language - JS in this case, but also python, ruby, php.
> Including headers isn't remotely "simple". There's so many considerations in linking, .SO version compatibility, architecture and instruction set issues, building against multiple versions on the same system. Or if you want to feel frustrated in a single word: GDAL (IYKYK)
I'd say for issues with instruction sets is a minor edge case, you could argue that having layers of abstraction gives even more room for fault.
That said I am not familiar with GDAL, but from google-fu it seems relatively heavy. Being said, I don't have much familiarity with compiling CXX programs as I do with programming C programs, and C with it's smaller footprint tends to not give me as many problems.
The JS ecosystem in particular, it really seems like it was built by people hell-bent on reinventing the wheel and making all the mistakes / paying all the costs along the way. It's a pretty octagonal wheel so far, but maybe they'll get there eventually.
Ecosystems aren’t built by any homogeneous group of people. They’re a sum of their parts. It’s not like there was a committee and that committee decided how things should work wrt wheel reinvention. People publish packages, and the result is something we call an ecosystem.
Seems to me the root problem here is poor security posture from the package maintainers. We need to start including information about publisher chain of custody into package meta data, that way we can recursively audit packages that don't have a secure deployment process.
Is there a terminal AI assistant that doesn't have heaps of depenedancies and preferably no node?
Claude and codex both require node. I'm a fan of the lightweight octofriend. But also node.
I do not like installing node on systems that otherwise would not require it.
You can install codex without npm if you build it yourself, they have migrated to rust in June and npm is just a convenient install wrapper it seems.
Just `git clone git@github.com:openai/codex.git`, `cd codex-rs`, `cargo build --release` (If you have many cores and not much RAM, use `-j n`, where n is 1 to 4 to decrease RAM requirements)
Does it have a terminal assistant that I have not heard of? Otherwise, the parent asks about an assistant that is able to run various tools and stuff, not just talk.
why don't web devs just learn html and css properly, and maybe xslt for the really complex transformations then use vanilla js only when it's truly necessary?
instead we've got this absolute mess of bloated, over-engineered junk code and ridiculously complicated module systems.
For context: ramada 0.32.0 isn't a concrete thing, in the sense that glibc 2.35 is. It really means "the latest ramada code because if you were to pin on this version it'll at some point stop working". glibc 2.35 never stops working.
me too but a lot of people see it as massive overhead they don't want to deal with.
personally i pin all mine because if you don't a version could be deployed during a pipeline and this makes your local version not the same as the one in docker etc.
pinning versions is the only way to be sure that the version I am running is the same as everyone elses
- There is a single root dependency somewhere which gets overtaken
- A new version of this dependency is published
- A CI somewhere of another NPM package uses this new version dependency in a build, which trigger propagation by creating a new modified version of this dependency?
I think so. It’s that third step that I can’t figure out. Build systems are configured to pull the latest version of a dep automatically, without review, and then publish. It seems the poorly configured pipelines are what enable these attacks. Fix your pipelines
Here's the underlying problem: let's imagine someone very smart. They figure out a way to solve this problem. They are not going to make any money by doing so. That's why we have this problem.
Maybe, we have to rethink depencies from the ground up.
Implementing everything yourself probably won't cut it.
Copying a dependency into your code base and maintaining it yourself probably won't yield much better results.
However, if a dependency would be part of the version control, depends could at least do a code review before installing an update.
That wouldn't help with new dependencies, that come in with issues right from.the start, but it could help preventing new malware from slipping in later.
A setup like that could benefit from a crowd-sourced review process, similar to Wikipedia.
I think, Nimble, the package manager of Nim, uses a decentralised registry approach based on Git repos. Something like that could be a good start.
I find it unbelievable that npm still doesn't upgrade integrity entries to SHA512 across the board. This seems like such a simple hole to plug. What gives?
Vendoring wouldn't really affect this at all. If anything it would keep you vulnerable for longer because your vendored copy keeps "working" after the bad package got removed upstream. There's a tiny chance that somebody would've caught the 10MB file added in review but that's already too late - the exploit happened on download, before the vendored copy got sent for review.
My motto wrt language choices: "It's the standard lib, stupid!"
My ultra hot take: there are only¹ two² programming ecosystems suitable for serious³ work:
- .net (either run on CLR or compile as an AOT standalone binary)
- jvm
The reason why is because they have a vast and vetted std lib. A good standard lib is a bigger boost then any other syntactic niceties.
__
1. I don't want other programming languages to die, so I am happy if you disagree with me. Other valid objection: some problems are better served by niche languages. Still, both .net and java support a plethora of niche languages.
2. Shades of gr[e|a]y, some languages are more complete out of the box than others.
3. cf «pick boring tools»
Arguably both Go and Python also have great stdlibs. The only advantage that JVM and .NET have is a default GUI package. Which is fair, but keeps getting less and less relevant as people rely more on web UIs.
Respectfully disagree. Python and Go std lib do not even play in the same league. I had to help someone with datetime¹ handling in Python a while back. The stdlib is so poor, you have to reach out for a thirdparty lib for even the most basic of tasks².
Don't take my word for it, take a dive. You wouldn't be the first to have adjust their view.
1. This might be a poor example as .net has NodaTime and the jvm has YodaTime as 3rd-party libs, for if one has really strict needs. Still, the builtin DateTime constructs offer way more than what Python had to offer.
2. Don't get me started on the ORM side of things. I know, you don't have to use one, but if you do, it better does a great job. And I wouldn't bat an eye if the ORM is not in the standard, but boy was I disappointed in Python's ecosystem. EF Core come batteries included and is so much better, it isn't fun anymore.
That one stumped me. Why not just encrypt with a hardcoded public key, then only the attacker can get the creds.
The simple B64 encoding didn't hide these creds from anyone, so every vendor out there's security team can collect them (e.g. thinking big clouds, GitHub, etc) and disable them.
If you did a simple encryption pass, no one but you would know what was stolen, or could abuse/sell it. My best guess is that calling node encryption libs might trigger code scanners, or EDRs, or maybe they just didn't care.
They surely seemed to be smart enough to choose encryption over encoding.
Hard to believe encryption would be the one thing that would trigger code scanners.
Also it’s not just every vendor, also every bad actor could’ve scraped the keys. I wonder if they’ve set up the infrastructure to handle all these thousands of keys…
Like what do you even do with most of it on scale?
Can you turn Cloud, AWS , AI api keys to money on a black market?
My understanding is that docker escapes are not all that difficult, and your aliases really aren’t doing much to harden the container. but I am not an expert on the matter. I’m sure there is plenty of info online
pnpm is the better comparison maybe in this context. Most of Deno's approach to security is focussed on whole program policies which doesn't do much in this context. Just like pnpm and others, they do have opt-in for install scripts though. The npm CLI is an outlier there by now.
the left-pad fiasco seems to have been the only time npm changed a policy and reacted to a security problem, since then it seems that supply chain attacks just belong to the npm-eco-system
> bun_environment.js is a highly obfuscated malicious JavaScript file. It is over 10MB in size and contains a significant amount of built-in logic for information theft.
That seems a bit silly. Even on the beefy boi I used to work on a 10MB hiccup in deployable size would have been sufficient to make me look.
I released one of the packages I work on last night so of course this drew my eye. I assume checking the unpacked size hasn’t gotten ridiculous confirms that your code is not infected yeah? And looks like it’s past time for me to set up a separate account for release management.
Why? Everyone won't use cooldowns, but the key is to have just enough people running brand new to set off a warning/have systems that check dependencies scan and find vulns go off and the packages get pulled before production builds them.
Monocultures where everyone pulls and builds with every brand new thing for the most minor changes is dangerous.
I wish everyone here would read https://en.wikipedia.org/wiki/Capability-based_security and then realize that maybe, JUST MAYBE, THE PROGRAMMING LANGUAGES WE USE SHOULD NOT ALLOW IMPORTED PACKAGES TO ACCESS EVERYTHING, AND THEIR LACK OF SECURITY GUARANTEES AND ACCESS RESTRICTION MECHANISMS MAKES THEM DANGEROUS!
The number and range of affected devices may be reduced with any number of package manager level workarounds, but NOT the impact of attacks once any succeeds. For this, you NEED the above.
You don't provide any more information, and are promoting your own site here without even saying so despite your name being on the About page. This felt like clickbait.
ProTip: use PNPM, not NPM. PNPM 10.x shutdown a lot of these attack vectors.
1. Does not default to running post-install scripts (must manually approve each)
2. Let's you set a min age for new releases before `pnpm install` will pull them in - e.g. 4 days - so publishers have time to cleanup.
NPM is too insecure for production CLI usage.
And of course make a very limited scope publisher key, bind it to specific packages (e.g. workflow A can only publish pkg A), and IP bound it to your self hosted CI/CD runners. No one should have publish keys on their local, and even if they got the publish keys, they couldn't publish from local. (Granted, GHA fans can use OIDC Trusted Publishers as well, but tokens done well are just as secure)
Npm is what happens when you let tech debt stack up for years too far. It took them five attempts to get lock files to actually behave the way lock files are supposed to behave (lockfile version 3, + at least 2 unversioned attempts before that).
It’s clear from the structure and commit history they’ve been working their asses off to make it better, but when you’re standing at the bottom of a well of suck it takes that much work just to see daylight.
The last time I chimed in on this I hypothesized that there must have been a change in management on the npm team but someone countered that several of the maintainers were the originals. So I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying. There’s just too much stupid there to make it easy.
I’m pretty sure it still cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache. Which means people with shit internet connections and large projects basically waste hours several times a week doing updates that fail.
> I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying.
If they were trying, they'd stop doubling down on sunk costs and instead publicly concede that lock files and how npm-the-tool uses them to attempt to ensure the integrity of packages fetched from npm-the-registry is just a poor substitute for content-based addressing that ye olde DVCS would otherwise be doing when told to fetch designated shared objects from the code repo—to be accompanied by a formal deprecation of npm-install for use in build pipelines, i.e. all the associated user guides and documentation and everything else pushing it as best practice.
npm-install has exactly one good use case: probing the registry to look up a package by name to be fetched by the author (not collaborators or people downstream who are repackaging e.g. for a particular distribution) at the time of development (i.e. neither run time nor build time but at the time that author is introducing the dependency into their codebase). Every aspect of version control should otherwise be left up to the underlying SCM/VCS.
> cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache.
I wonder what circumstances led to saying “this is okay we’ll ship it like that”
I think we can blame the IO streaming API in NodeJS on this. It’s a callback and you just know you got another block. My guess is chunked mode and not checking whether the bytes expected and the bytes received matched.
Not to diminish the facepalm but the blame can be at least partially shared.
Our UI lead was getting the worst of this during Covid. I set up an nginx forward proxy mostly for him to tone this down a notch (fixed a separate issue but helped here a bit as well) so he could get work done on his shitty ISP.
Ignorance. Most programmers in open source operate on the "works on my machine"
True, and things that manifest only on old/slow hardware or on bad internet are the worst kind for this, since 100% of developers who have any say in the matter would never accept such circumstances at all, so they’re always approaching every issue with multi-gigabit speeds, zero latency, and this year’s $3,000 Mac. “What do you mean the page loads slowly?”
A customer of mine has the implementation of several API endpoints starting with a simulate_slow_connection method that basically sleeps for a random amount of ms. I think it sleeps 0 ms when running tests and definitely sleeps 0 ms in production. So it's never super fast even on $3,000 Macs.
but this stuff is basically solved. We have enough history with languages and distribution of packages, repositories, linux, public trust, signing, maintainers, etc.
One key shift is there is no packager anymore. Its just - trust the publisher.
Any language as big as Node should hire a handful of old unix wizards to teach them the way the truth and the life.
Likely they wouldn’t listen. Modern languages and environments seem intent on reinventing bad solutions to solved problems. I get it if it’s a bunch of kids that have never seen anything better but there is no excuse these days not to have at least a passing knowledge of older systems if you’ve been around a while.
there's certainly a piece of it. Also most seasoned people are not very interested in new languages and environments, and most languages are not 'spec built' by experts like Rob Pike building Go who explicitly set out to solve a lot of his problems, but are more naturally grown and born.
> One key shift is there is no packager anymore. Its just - trust the publisher.
Repositories like NPM's, and PyPI, contain many more packages than any Linux distro. And the Linux Foundation actually gets funded.
NPM isn't a package repository it's more akin a code repository.
There's a reason why most distributions dont ship upstream (except basically Arch)
[dead]
> And of course make a very limited scope publisher key, bind it to specific packages (e.g. workflow A can only publish pkg A), and IP bound it to your self hosted CI/CD runners. No one should have publish keys on their local, and even if they got the publish keys, they couldn't publish from local.
I've by now grown to like Hashicorp Vaults/OpenBao's dynamic secret management for this. It's a bit complicated to understand and get to work at first, but it's powerful:
You mirror/model the lifetime of a secret user as a lease. For example, a nomad allocation/kubernetes pod gets a lease when it is started and the lease gets revoked immediately after it is stopped. We're kinda discussing if we could have this in CI as well - create a lease for a build, destroy the lease once the build is over. This also supports ttl, ttl-refreshes and enforced max-ttls for leases.
With that in place, you can tie dynamically issued secrets to this lease and the secrets are revoked as soon as the lease is terminated or expires. This has confused developers with questionable practices a lot. You can print database credentials in your production job, run that into a local database client, but as soon as you deploy a new version, those secrets are deleted. It also gives you automated, forced database credential rotation for free through the max_ttl, including a full audit log of all credential accesses and refreshes.
I know that would be a lot of infrastructure for a FOSS project by Bob from Novi Zagreb. But with some plugin-work, for a company, it should be possible to hide long-term access credentials in Vault and supply CI builds with dropped, enforced, short-lived tokens only.
As much as I hate running after these attacks, they are spurring interesting security discussions at work, which can create actual security -- not just checkbox-theatre.
I would love to use this (for homelab stuff currently) but I would love a way to have vault/openbao be fully configuration-as-code and version controlled, and only have the actual secret values (those that would not be dynamic) in persistent storage.
Definitely curious if you've come up with a way to give each build a short lived vault approle somehow in any CI system.
We do that in our github runners with oidc integration. Works well
Or just 'npm ci' so you install exactly what's in your package-lock.json instead of the latest version bumps of those packages. This "automatic updating" is a big factor in why these attacks are working in the first place. Make package updating deliberate instead of instant or on an arbitrary lag.
You'd be surprised how many people run 'npm i' in their CI. I've seen this on multiple occasions.
'npm ci' is some mitigation, but doesn't protect against getting hit when running 'npm i(nstall)' during development.
Update your knowledge. ”npm install” hasn’t done auto-upgrades for years now.
”npm install” does not auto-upgrade packages, it installs them according to the lockfile.
It didn’t always work like that, but it has worked like that for many years.
For Python ecosystem people:
> Does not default to running post-install scripts (must manually approve each)
To get equivalent protection, use `--only-binary=:all:` when running `pip install` (or `uv pip install`). This prevents installing source distributions entirely, using exclusively pre-built wheels. (Note that this may limit version ability or even make your installation impossible.) Python source packages are built by following instructions provided with the package (specifying a build system which may then in turn be configured in an idiosyncratic way; the default Setuptools is configured using a Python script). As such, they effectively run a post-install script.
(For PAPER, long-term I intend to design a radically different UI, where you can choose a named "source" for each package or use the default; and sources are described in config files that explain the entire strategy for whether to use source packages, which indexes to check etc.)
> Let's you set a min age for new releases before `pnpm install` will pull them in - e.g. 4 days - so publishers have time to cleanup.
Pip does not support this; with uv, use `--exclude-newer`. This appears to require a timestamp; so if you always want things up to X days old you'll have to recalculate.
> Pip does not support this; with uv, use `--exclude-newer`. This appears to require a timestamp; so if you always want things up to X days old you'll have to recalculate.
I do this by having my shell init do this:
That’s easy to override if you need to but otherwise seamless.FWIW, I'd like if these tools had an option to prefer the oldest version satisfying the given constraints (rather than the newest, as it is now — probably still a better default).
> prefer the oldest version satisfying the given constraints
The problem is there's no metadata for which versions fix security bugs, and therefore which previous versions are now insecure.
There were some recent posts I saw about "dependency cooldowns", which seem to be what you're referring to in item 2. The idea really resonated with me.
That said, I hard pin all our dependencies and get dependabot alerts and then look into updates manually. Not sure if I'm a rube or if that's good practice.
That's good practice. God knows how many times I've been bitten by npm packages breaking on minor or even patch version changes, even when proudly proclaiming to use semver
You shouldn't have any keys anywhere at all. Use OIDC https://docs.npmjs.com/trusted-publishers
Unfortunately you need to `npm login` with username and password in order to publish the very first version of a package to set up OIDC.
I'm struggling to understand why Trusted Publishers is any better.
Let's say you have a limited life, package specific scoped, IP CIDR bound publishing key, running on a private GH workflow runner. That key only exists in a trusted clouds secret store (e.g. no one will have access it from their laptop).
Now let's say you're a "trusted" publisher, running on a specific GitHub workflow, and GitHub Org, that has been configured with OIDC on the NPM side. By virtue of simply existing in that workflow, you're now a NPM publisher (run any publish commands you like). No need to have a secret passed into your workflow scope.
If someone is taking over GitHub CI/CD workflows by running `npm i` at the start of their workflow, how does the "Trusted Publisher" find themselves any more secure than the secure, very limited scope token?
A whole single supported CI partner outside their own corporate family. They really planned this out well.
Both NPM and Yarn have a way to disable install scripts which everyone should do if at all possible.
Good point, but until many popular packages stop requiring install.sh to operate, you'll still need to allowlist some of them. That is built into the PNPM tooling, luckily :)
Reading through the post it looks like this infects via preinstall?
> The new versions of these packages published to the NPM registry falsely purported to introduce the Bun runtime, adding the script preinstall: node setup_bun.js along with an obfuscated bun_environment.js file.
You're right. PNPM disables all install scripts by default. I was just noting one example.
Is there a way to set a minimum release age globally for my pnpm installation? I was only able to find a way to set it for each individual project.
Did you try putting it in your global config file?
Windows: ~/AppData/Local/pnpm/config/rc
macOS: ~/Library/Preferences/pnpm/rc
Linux: ~/.config/pnpm/rc
I think it's a `pnpm-workspace.yaml` setting, for now, but PNPM has been pretty aggressive with expanding this feature set [1].
[1] https://pnpm.io/supply-chain-security
Pnpm cannot be built without an existing pnpm binary meaning there is no way to bootstrap it from audited source code. Perfect trusting trust attack situation.
Full source bootstrapped NPM with manually reviewed dependencies is the only reasonably secure way to use NodeJS right now.
ProTip: `use bun`
Funny that this is getting downvoted, but it installs dependencies super fast, and has the same approval feature as pnmp, all in a simple binary.
This is like saying "use MacOS and you won't get viruses" in the 2000s
Bun disables post-install scripts by default and one can explicitly opt-in to trusting dependencies in the package.json file. One can also delay installing updated dependencies through keys like `minimumReleaseAge`. Bun is a drop-in replacement for the npm CLI and, unlike pnpm, has goals beyond performance and storage efficiency.
Not sure what your analogy is trying to imply.
Which was for the most part true.
The suggestion was to use pnpm, and I'm suggesting something I prefer more than pnpm.
Except trying it out takes a minute and costs nothing.
"Rewrite it in rust"
How does bun compare? Does it have similar features as well?
yes bun does both of the things mentioned in the parent comment:
> Unlike other npm clients, Bun does not execute arbitrary lifecycle scripts like postinstall for installed dependencies. Executing arbitrary scripts represents a potential security risk.
https://bun.com/docs/pm/cli/install#lifecycle-scripts
> To protect against supply chain attacks where malicious packages are quickly published, you can configure a minimum age requirement for npm packages. Package versions published more recently than the specified threshold (in seconds) will be filtered out during installation.
https://bun.com/docs/pm/cli/install#minimum-release-age
What does it do with packages that download binaries for specific architecture in the post script?
You don't need post-install scripts for this. Use optionalDependencies instead https://github.com/nrwl/nx/blob/master/packages/nx/package.j...
Each of those deps contains a constraint installing only for the relevant platform.
As far as I can understand from the documentation, that doesn't actually specify in that config that one of them is required, does it? That is, if they _all_ fail to install as far as the system is concerned there's nothing wrong? There will be runtime errors of course, but that's sort of disappointing…
That’s cool, now I wish all libraries that need binaries would opt to use that instead of post script
Do keep in mind that the binaries are still binaries. Even if your installation process doesn't run any untrusted code from the package, you can't audit the binaries like you might the .js files prior to first run.
As stated, you manually approve them.
pnpm on the backend, frontend use nobuild.
> NPM is too insecure for production CLI usage.
NPM was never "too insecure" and remains not "too insecure" today.
This is not an issue with npm, JavaScript, NodeJS, the NodeJS foundation or anything else but the consumer of these libraries pulling in code from 3rd parties and pushing it to production environments without a single review. How this still fly today, and have been since the inception of public "easy to publish" repositories remains a mystery to me even today.
If you're maintaining a platform like Zapier, which gets hacked because none of your software engineers actually review the code that ends up in your production environment (yes, that includes 3rd party dependencies, no matter where they come from), I'm not sure you even have any business writing software.
The internet been a hostile place for so long, that most of us "web masters" are used to it today. Yet it seems developers of all ages fall into the "what's the worst that can happen?" trap when pulling in either one dependency with 10K LoC without any review, or 1000s of dependencies with 10 lines each.
Until you fix your processes and workflows, this will continue to happen, even if you use pnpm. You NEED to be responsible for the code you ship, regardless of who wrote it.
They didn't deploy the code. That's not how this exploit works. They _downloaded_ the code to their machine. And npm's behavior is to implicitly run arbitrary code as part of the download - including, in this case, a script to harvest credentials and propagate the worm. That part has everything to do with npm behavior and nothing to do with how much anybody reviewed 3P deps. For all we know they downloaded the new version of the affected package to review it!
If people stop running install scripts, isn't Shai-Hulud 3: Electric Boogaloo just going to be designed to run its obfuscated malware at runtime rather than install time? Who manually reviews new versions of their project dependencies after installing them but before running them?
GP is correct. This is a workflow issue. Without a review process for dependencies, literally every package manager I know of is vulnerable to this. (Yes, even Maven.)
> If people stop running install scripts, isn't Shai-Hulud 3: Electric Boogaloo just going to be designed to run its obfuscated malware at runtime rather than install time?
Many such forms of malware have already been published and detected.
> Who manually reviews new versions of their project dependencies after installing them but before running them?
One person putting in this effort can protect everyone thereafter.
The PyPI website has a "Report project as malware" button on each project page for this purpose.
But yes, this is the world we live in. Without this particular form of insecurity, there is no "ecosystem" at all.
Thank you.
wait, I short-circuited here. wasn't the very concept of "libraries" created to *not* have to think about what exactly the code does?
imagine reviewing every React update. yes, some do that (Obsidian claims to review every dependency, whether new or an update), but that's due to flaws of the ecosystem.
take a look at Maven Central. it's harder to get into, but that's the price of security. you have to verify the namespace so that no one will publish under e.g. `io.gitlab.bpavuk.` namespace unless they have access to the `bpavuk` GitLab group or user, or `org.jetbrains.` unless they prove the ownership of the jetbrains.com domain.
Go is also nice in that regard - you are depending on Git repositories directly, so you have to hijack into the Git repo permissions and spoil the source code there.
> Go is also nice in that regard - you are depending on Git repositories directly, so you have to hijack into the Git repo permissions and spoil the source code there.
That in itself is scary because Git refs are mutable. Even with compromised credentials, no one can replace artifacts already deployed to Maven Central, because they simply don't allow it. There is nothing stopping someone from replacing a Git tag with one that points to compromised code.
The surface area is smaller because Go does locking via go.sum, but I could certainly see a tired developer regenerating it over the most strenuous of on-screen objections from the go CLI.
Go also includes a database of known package hashes so altering git tag to point to another commit will be detected.
I don't know if it's a common or even a good practice, but I like to go mod vendor and add the result to my repo.
I do `cargo vendor` sometimes, but that's mostly to enable offline work and use the debugger inside some vague crates (Rust's libraries-but-not-really), and usually I gitignore the `vendor`'ed crates away.
> wasn't the very concept of "libraries" created to not have to think about what exactly the code does?
Let's say you need a FFT implementation. You can write that from scratch, or you can use a library. In both cases you should use tests to verify that the code calculates the FFT correctly, and in the library case you should read the code to make sure that it works correctly and does not omit and edge cases (e.g.).
> wasn't the very concept of "libraries" created to not have to think about what exactly the code does?
If you care about security, you only have to care once, during the audit. And you can find a pretty high percentage of malware in practice without actually having a detailed understanding of the non-malicious workings of the code.
Libraries allow you to not think about what the code does at development time, which in general is much more significant than audit time. Also, importantly, they allow you not to have to design and write that part of the code.
Are GitHub creds any harder for malware to steal than NPM creds? I don't see how that helps at all.
well, if you talk about private SSH keys for Git operations and SSH/GPG keys for signing, then you'd better set up a passphrase on them, which GitHub strongly recommends. the passphrase will make it significantly harder to use the keys. so, as usual, It Depends (c)
“Personally, I never wear a seatbelt because all drivers on the road should just follow the road rules instead and drive carefully.”
I don’t control all the drivers on the road, and a company can’t magically turn all employees into perfect developers. Get off your high horse and accept practical solutions.
> and a company can’t magically turn all employees into perfect developers
Sure, agree, that's why professionals have processes and workflows, everyone working together to build the greatest stuff you can.
But when not a single person in the entire company reviews the code that gets deployed and run by users, you have to start asking what kind of culture the company has, it's borderline irresponsible I'd say.
Or they have different priority structures where this isn’t the developers job. Also things get missed sometimes no matter how good your processes are. But improving those processes could involve something like avoiding npm over pnpm.
The "use cooldown" [0] blog post looks particularly relevant today.
I'd argue automated dependency updates pose a greater risk than one-day exploits, though I don't have data to back that up. That's harder to undo a compromised package already in thousands of lock files, than to manually patch a already exploited vulnerability in your dependencies.
[0] https://blog.yossarian.net/2025/11/21/We-should-all-be-using...
Why not take it further and not update dependencies at all until you need to because of some missing feature or systems compatibility you need? If it works it works.
The arguments for doing frequent releases partially apply to upgrading dependencies. Upgrading gets harder the longer you put it off. It’s better to do it on a regular schedule, so there are fewer changes at once and it preserves knowledge about how to do it.
A cooldown is a good idea, though.
There's another variable, though, which is how valuable "engineering time now" is vs. "engineering time later."
Certainly, having a regular/automated update schedule may take less clock time in total (due to preserved knowledge etc.), and incur less long-term risk, than deferring updates until a giant, risky multi-version multi-dependency bump months or years down the road.
But if you have limited engineering resources (especially for a bootstrapped or cost-conscious company), or if the risks of outages now are much greater than the risks of outages later (say, once you're 5 years in and have much broader knowledge on your engineering team), then the calculus may very well shift towards freezing now, upgrading later.
And in a world where supply chain attacks will get far more subtle than Shai-Hulud, especially with AI-generated payloads that can evolve as worms spread to avoid detection, and may not require build-time scripting but defer their behavior to when called by your code - macro-level slowness isn't necessarily a bad thing.
(It should go without saying that if you choose to freeze things, you should subscribe to security notification services that can tell you when a security update does release for a core server-side library, particularly for things like SQL injection vulnerabilities, and that your team needs the discipline to prioritize these alerts.)
At the same time, unplanned engineering time is almost always more expensive than planned engineering time. I'd rather have some regular, expected, upgrade work, than all of a sudden having to scramble because I need something at a moment when I didn't plan for that.
When you do regular updates, they are quick. But also, you can timebox it and then back out and plan a harder than expected update.
Also, it keeps you in touch with your deps so you can consider if it’s even worth it. My favorite update was removing the dep (or starting a plan to remove it because it was interfering with regular updates)
> Upgrading gets harder the longer you put it off.
This is only true if you install dependencies that break backwards compatibility.
Personally, I avoid this as much as possible.
There is a Goldilocks effect. Dependency just came out a few minutes ago? There is no time for the community to catch the vulnerability, no real coverage from dependency scans, and it's a risk. Dependency came out a few months ago? It likely has a large number of known vulns
That is indeed what one should do IMO. We've known for a long time now in the ops world that keeping versions stable is a good way to reduce issues, and it seems to me that the same principle applies quite well to software dev. I've never found the "but then upgrading is more of a pain" argument to be persuasive, as it seems to be equally a pain to upgrade whether you do it once every six months or once every six years.
The 'pain' comes from breaking changes, at worst if you delay you're going to ingest the same quantity of changes, and at best you might skip some short-lived ideas.
> Why not take it further and not update dependencies at all until you need to because of some missing feature or systems compatibility you need? If it works it works.
Indeed there are people doing that and communities with a consensus such approach makes sense, or at least is not frowned upon. (Hi, Gophers)
This works until you consider regular security vulnerability patching (which we have compliance/contractual obligations for).
This only makes sense for vulnerabilities that can actually be exploited in your particular use-case and configuration of the library. A lot of vulns might be just noise and not exploitable so no need to patch.
Yes and no.
Problem is code bases are continuously evolving. A safe decision now, might not be a safe decision in the future. It's very easy to accidentally introduce a new code path that does make you vulnerable.
Because updates don't just include new features but also bug and security fixes. As always, it probably depends on the context how relevant this is to you. I agree that cooldown is a good idea though.
> Because updates don't just include new features but also bug and security fixes.
This practice needs to change, although it will be almost impossible to get a whole ecosystem to adopt. You shouldn’t have to take new features (and associated new problems) just to get bug fixes and security updates. They should be offered in parallel. We need to get comfortable again with parallel maintenance branches for each major feature branch, and comfortable with backporting fixes to older releases.
I maintain both commercial and open source libs. This is a non starter in both cases. It would easily double if not triple the workload.
For open source, well these are volunteer projects on my own time, you are always welcome to fork a given version and backport any fixes that land on main/master.
For commercial libs, our users are not willing to pay extra for this service, so we don't provide it. They would rather stay on an old version and update the entire code base at given intervals. Even when we do release patch versions, there is surprisingly little uptake.
Are you just referring to backporting?
Semver was invented to facilitate that. Only if everyone adhered to it.
Semver doesn't help. The primary issue is effort. If it's an open source project with 1-2 devs, they probably won't be able to handle supporting multiple branches unless they're being paid to do this.
> Semver was invented to facilitate that
First time I've heard that. How does semver facilitate backporting?
Of course it doesn't provide backports by itself, it's a versioning system. But version number changes with SemVer are meant to indicate whether an update includes new fearhews or not (minor bump means new features, patch bump means bugfixes only).
Of course, the actual issue is that maintaining backports isn't free, so expecting it from random single-person projects is a little unrealistic. Bug fixes in new code often need to be rewritten to work on old code. I do maintain old release branches for some projects and backporting single patches can cause whole new bugs that were never present in the main branch quite easily.
IMO for “boring software” you usually want to be on the oldest supported main/minor version, keeping an eye on the newest point version. That will have all the security patches. But you don't need to take every bug fix blindly.
[dead]
For any update:
- it usually contains improvements to security
- except when it quietly introduces security defects which are discovered months later, often in a major rev bump
- but every once in a while it degrades security spectacularly and immediately, published as a minor rev
CI fights this. But that’s peanuts compared to feature branches and nothing compared to lack of a monolith.
We had so many distinct packages on my last project that I had to massively upgrade a tool a coworker started to track the dependency tree so people stopped being afraid of the release process.
I could not think of any way to make lock files not be the absolute worst thing about our entire dev and release process, so the handful of deployables had a lockfile each that was only utilized to do hotfix releases without changing the dep tree out from underneath us. Artifactory helps only a little here.
Just make sure to update when new CVEs are revealed.
Also, some software are always buggy and every version is a mixed bag of new features, bugs and regressions. It could be due to the complexity of the problem the software is trying to solve, or because it's just not written well.
Because if you're too far behind, when you "need" takes days instead of hours.
Because AppSec requires us to adhere to strict vulnerability SLA guidelines and that's further reinforced by similar demands from our customers.
But even then you are still depending on others to catch the bugs for you and it doesn't scale: if everybody did the cooldown thing you'd be right back where you started.
I don't think that this Kantian argument is relevant in tech. We've had LTS versions of software for decades and it's not like every single person in the industry is just waiting for code to hit LTS before trying it. There are a lot of people and (mostly smaller) companies who pride themselves on being close to the "bleeding edge", where they're participating more fully in discovering issues and steering the direction.
The assumption in the post is that scanners are effective at detecting attacks within the cooldown period, not that end-device exploitation is necessary for detection.
(This may end up not being true, in which case a lot of people are paying security vendors a lot of money to essentially regurgitate vulnerability feeds at them.)
To find a vulnerability, one does not necessarily deploy a vulnerable version to prod. It would be wise to run a separate CI job that tries to upgrade to the latest versions of everything, run tests, watch network traffic, and otherwise look for suspicions activity. This can be done relatively economically, and the responsibility could be reasonably distributed across the community of users.
It does scale against this form of attack. This attack propagates by injecting itself into the packages you host. If you pull only 7d after release you are infected 7d later. If your customers then also only pull 7d later they are pulling 14d after the attack has launched, giving defenders a much longer window by slowing down the propagation of the worm.
That worried me too, a sort of inverse tragedy of the commons. I'll use a weeklong cooldown, _someone else_ will find the issue...
Until no-one does, for a week. To stretch the original metaphor, instead of an overgrazed pasture, we grow a communally untended thicket which may or may not have snakes when we finally enter.
That is statistically not possible, unless you are dealing with very small sample size.
The "until no one does" is not something that can happen in something like npm ecosystem, or even among the specific user of "left-pad".
I don't buy this line of reasoning. There are zero/one day vulnerabilities that will get extra time to spread. Also, if everyone switches to the same cooldown, wouldn't this just postpone the discovery of future Shai-Huluds?
I guess the latter point depends on how are Shai-Huluds detected. If they are discovered by downstreams of libraries, or worse users, then it will do nothing.
There are companies like Helix Guard scanning registries. They advertise static analysis / LLM analysis, but honeypot instances can also install packages & detect certain files like cloud configs being accessed
But relying on the goodwill of commercial sec vendors is it's own infrastructure risk.
So don't rely on their goodwill? Instead, pay them, under a contract.. or do it yourself.
You can also pay a commercial sec vendor if you don't want to rely on their goodwill.
Your line of reasoning only makes sense if literally almost all developers in the world adopt cooldowns, and adopt the same cooldown.
That would be a level of mass participation yet unseen by mankind (in anything, much less something as subjective as software development). I think we're fine.
I don't think so. What fraction of developers would even notice the malware? Some malware seems to barely be noticed.
For zero/one days, the trick is that you'd pair dependency cooldowns with automatic scanning for vulnerable dependencies.
And in the cases where you have vulnerable dependencies, you'd force update them before the cooldown period had expired, while leaving everything else you can in place.
For Python's uv, I think the closest thing to a cooldown is something like:
uv is considering a native relative date:https://github.com/astral-sh/uv/issues/14992
Pretty easy to do using npm-check-update:
https://www.npmjs.com/package/npm-check-updates#cooldown
In one command:
The docs list this caveat:
> Note that previous stable versions will not be suggested. The package will be completely ignored if its latest published version is within the cooldown period.
Seems like a big drawback to this approach.
I could see it being a good feature. If there have been two versions published within the last week or two, then there are reasonable odds that the previous one had a bug.
some lib literally publish a new package at every PR merged, so multiple times a day.
co-founder of PostHog here. We were a victim of this attack. We had a bunch of packages published a couple of hours ago. The main packages/versions affected were:
- posthog-node 4.18.1, 5.13.3 and 5.11.3
- posthog-js 1.297.3
- posthog-react-native 4.11.1
- posthog-docusaurus 2.0.6
We've rotated keys and passwords, unpublished all affected packages and have pushed new versions, so make sure you're on the latest version of our SDKs.
We're still figuring out how this key got compromised, and we'll follow up with a post-mortem. We'll update status.posthog.com with more updates as well.
You're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run.
Or require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing.
The basic issue with manual interaction is a question of authority: a pretty common problem for companies (and open source groups) is when $EARLY_EMPLOYEE/$EARLY_CONTRIBUTOR creates and owns the entire publishing process for a key package, and then leaves without performing a proper transfer of responsibility. This essentially locks the company/group out of its own work, and increases support load on community maintained indices to essentially adjudicate rightful ownership of the package name.
(There are a variety of ways to solve this, but the one I like best is automated publishing a la Trusted Publishing with environment mediated manual signoffs. GitHub and other CI/CD providers enable this.)
I don’t buy this. I mean, I’m sympathetic to the issue. It’s easy for things to get jumbled when companies are young. But we’re talking about libraries that are used by your customers. To many, this is the externally visible interface to the company.
What you describe sounds like a process problem to me. If an $EARLY_EMPLOYEE if the only one with the deploy keys for what is a product of the company, then that’s a problem. If a deployment of that key library can be made without anyone approving it, that’s also a problem. But those are both people problems… and you can’t solve a people problem with a technical solution.
> But those are both people problems… and you can’t solve a people problem with a technical solution.
I don’t think it is a people problem in this case: the only reason there’s a person involved at all is because we’ve decided to introduce one as an intermediating party. A more misuse-resistant scheme disintermediates the human, because the human was never actually a mandatory part of the scheme.
What person do you want to remove from the process?
The person who intermediates the trust relationship between the index and the source repository. There’s no reason for the credential that links those two parties to be intermediated by a human; they’re two machine services talking.
(You obviously can’t disintermediate the human from maintenance or development!)
But that’s the person I think is mandatory.
You’re saying that whatever is in the source repository should be uploaded in the npm index, right? If the code is tagged as release, the built artifact is automatically uploaded to npm. Is that what you’re proposing?
That exactly what got PostHog into this position. The keys to publish to npm were available to an engineer or GitHub to push a malware build into npm automatically. This isn’t a technical issue… it’s a process issue. I don’t see the problem as that the keys were misused. I see the problem as that it was possible to misuse the keys at all. Why do you need that process to be automatic? How often are you pushing new updates?
I would argue that those npm assets/libraries are your work product. That is what your customer needs to use your service. It is a published product from your company. It is too important to allow a new version to be published out to the public without a human in the loop to approve it.
When you have a fully automatic publishing cycle, you’re trading security for convenience. It’s all about how much risk you’re willing to accept. For me, that’s too much of a risk to the reputation to the company. I also think the balance shifts if you’re talking about purely internal assets, having completely automatic ci/cd makes perfect sense for most companies. For me, it is about who is hurt if there is an issue (and you should expect for there to be an issue).
Putting a person in the loop for releasing a product is one way to solve this. It’s not perfect, but at the moment, I think it’s the most secure (for the public).
> You’re saying that whatever is in the source repository should be uploaded in the npm index, right? If the code is tagged as release, the built artifact is automatically uploaded to npm. Is that what you’re proposing?
No, I'm saying that the source repository should act as an authentication principal itself. A human should still initiate the release process, but the authentication process that connects the source repository (more precisely CI/CD) to the index should not involve a credential that's implicitly bound to a human identity (because the human's role within a project or company is ephemeral).
As far as I can tell, what got PostHog into this situation wasn't a fully automated release process (very few companies/groups have fully automated processes), but the fact that they had user-created long-lived credentials that an attacker could store and weaponize at a time most convenient to them. That's a problem regardless of whether there's normally a human in the loop or not, because the long-lived credential itself was sufficient for publishing.
(In other words, we basically agree about human approval being good; what I'm saying is that we should formalize human approval without making the authentication scheme explicitly require an intermediating partner who doesn't inherently represent the actual principal, i.e. the source repository.)
I think we agree more than we don’t and the rest are personal preferences and policy differences. But we largely agree in principle.
I like the idea of having a person whose job is approving releases. Kind of like a QC tag — this release was approved by XX. I saw the issue as PostHog having a credential available to the CI/CD that had the authority to push releases automatically. When a new GitHub action was added, that credential was abused to push a bad update to npm. I might be wrong, I don’t deal with npm that much.
There are many ways to fix this.
You can't "require" manual intervention. Sure you can say that the keys stays on say 2 developers laptops, but personal devices have even more surface area for key leak than CI/CD pipeline. It wouldn't have prevented attacks like this issue in any case where the binary just searched for keys across the system.
One alternative is to do the signing on airlocked system stored in physically safe but accessible location, but I guess that's just way too much inconvenience.
As someone else mentioned, the easiest way would be to have some kind of MFA in the loop. It’s not perfect, but better than what we have now.
I get that it can be useful sometimes. But requiring physical MFA to make a package available to the general public seems like a no-brainer to me.
Users who really want to could opt in to the bleeding edge.
This is orthogonal to the issue at hand. The problem is a malicious actor cutting a release outside of the normal release process. It doesn't matter if the normal process is automated or manual.
It could have eliminated an attack surface where they steal the credentials from the CI/CD...
...But then you if I understand NPM publishing well, you would still have the credentials on someone's computer laying around? I guess you could always revoke the tokens after publishing? It's all balancing convenience and security, with some options being bad at both?
This is built in NPM. You can get an email on every pkg publishing.
Sure, it might be a little bit of noise, but if you get a notice @ 3am of an unexpected publishing, you can jump on unpublishing it.
Very nice way of putting it, kudos!
Did the client side JS being infected produce any issues which would have affected end users? As in if a web owner were on an affected version and deployed during the window would the end user of their site have had any negative impact?
No, just the host that was running the package (the exploit was pretty generic and not targeted at PostHog specifically). In fact, so far we think there were 0 production deployments of PostHog because the package was only live for a little bit.
Glad to hear the impact was so muted. Thank you for the response!
If we don't know how it got compromised, chances are this attack is still spreading?
If anything people should use an older version of the packages. Your newest versions had just been compromised, why should anyone believe this time and next time it will be different?!
The packages were published using a compromised key directly, not through our ci/cd. We rolled the key, and published a new clean version from our repo through our CI/CD: https://github.com/PostHog/posthog-js/actions/runs/196303581...
Why do you keep using token auth? This is unacceptable negligence these days.
NPM supports GitHub workflow OIDC and you can make that required, disabling all token access.
Yep, we are moving to workflow OIDC as the next step in recovery.
OIDC is not a silver bullet either and has its own set of vectors to consider too. If it works for your org model then great, but it doesn't solve every common scenario.
Trusted Publishing addresses the vector here, which is arbitrary persistence and delayed use of credentials by attackers. You're right that it's not a silver bullet (anything claiming to be one is almost certainly a financially induced lie), but it eliminates/foreshortens the attack staging window significantly.
It's not a silver bullet in that it can open up other vectors of much more probable attack.
If I have a FOSS project where there are other contributors with write access but not publish access (very common) OIDC now allows them to publish via workflows, opening you up to Jia Tan type attacks.
Like I said, if it works for your org, great. Otherwise it can cause bigger problems.
Sorry, what's the threat model in which you're giving people write access to your repository but assuming that they can't publish?
To establish a baseline here: the assumption with Trusted Publishing is that people are already using CI/CD to publish to places like PyPI. All Trusted Publishing does is replace the user-managed credential with a machine-intermediated one.
In other words: if someone had write access to your repository before, Trusted Publishing does not change their privilege, since they could always push a new CI/CD workflow that uses your manual publishing secret directly. All Trusted Publishing does is formalize the treatment of the source repository as a source of truth, and introduce some misuse-resistance properties (short-lived credentials, minimal self-scoping).
If this isn't your user model, that's perfectly fine. But in that case you shouldn't be using CI/CD to publish at all (which is also fine); it's not an issue with Trusted Publishing per se.
> Sorry, what's the threat model in which you're giving people write access to your repository but assuming that they can't publish?
So that I'm the arbiter of what people download, and can review locally what changes were made prior to pushing it out to everyone.
> To establish a baseline here: the assumption with Trusted Publishing is that people are already using CI/CD to publish to places like PyPI.
That's not the narrative. People have been saying to switch to trusted publishing from local publishes. They're not 1:1 compatible. I agree that if you already publish via CD, then it probably already worked for your org and thus is a no brainer.
> If this isn't your user model, that's perfectly fine. But in that case you shouldn't be using CI/CD to publish at all (which is also fine); it's not an issue with Trusted Publishing per se.
That's exactly the point I'm making.
[dead]
> so make sure you're on the latest version of our SDKs.
Probably even safer to not have been on the latest version in the first place.
Or safer again not to use software this vulnerable.
As a user of Posthog, this statement is absurd: > Or safer again not to use software this vulnerable.
Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it.
Popularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive?
Glad you updated on this front-page post. Your Twitter post is buried on p3 for me right now. Good luck on the recovery and hopefully this helps someone.
Serious question: should someone develop new technologies using Node any more?
A short time ago, I started a frontend in Astro for a SaaS startup I'm building with a friend. Astro is beautiful. But it's build on Node. And every time I update the versions of my dependencies I feel terrified I am bringing something into my server I don't know about.
I just keep reading more and more stories about dangerous npm packages, and get this sense that npm has absolutely no safety at all.
It's not "node" or "Javascript" the problem, it's this convenient packaging model.
This is gonna ruffle some feathers, but it's only a matter of time until it'll happen on the Rust ecosystem which loves to depend on a billion subpackages, and it won't be fault of the language itself.
The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.
I've worried about this for a while with Rust packages. The total size of a "big" Rust project's dependency graph is pretty similar to a lot of JS projects. E.g. Tauri, last I checked, introduces about 600 dependencies just on its own.
Like another commenter said, I do think it's partially just because dependency management is so easy in Rust compared to e.g. C or C++, but I also suspect that it has to do with the size of the standard library. Rust and JS are both famous for having minimal standard libraries, and what do you know, they tend to have crazy-deep dependency graphs. On the other hand, Python is famous for being "batteries included", and if you look at Python project dependency graphs, they're much less crazy than JS or Rust. E.g. even a higher-level framework like FastAPI, that itself depends on lower-level frameworks, has only a dozen or so dependencies. A Python app that I maintain for work, which has over 20 top-level dependencies, only expands to ~100 once those 20 are fully resolved. I really think a lot of it comes down to the standard library backstopping the most common things that everybody needs.
So maybe it would improve the situation to just expand the standard library a bit? Maybe this would be hiding the problem more than solving it, since all that code would still have to be maintained and would still be vulnerable to getting pwned, but other languages manage somehow.
I wouldn't call the Rust stdlib "small". "Limited" I could agree with.
On the topics it does cover, Rust's stdlib offers a lot. At least on the same level as Python, at times surpassing it. But because the stdlib isn't versioned it stays away from everything that isn't considered "settled", especially in matters where the best interface isn't clear yet. So no http library, no date handling, no helpers for writing macros, etc.
You can absolutely write pretty substantial zero-dependency rust if you stay away from the network and async
Whether that's a good tradeoff is an open question. None of the options look really great
Rand, uuid, and no built in logging implementation are three examples that require crates but probably shouldn’t.
No built in logging seems pretty crazy. Is there a story behind that?
It seems like the normal and rational choice to not include logging to me.
Java included 'java.util.logging'. No one uses it because it's bad, everyone just uses slf4j as the generic facade instead, and then some underlying implementation.
Golang included the "log" package, which was also a disaster. No one used it so they introduced log/slog recently, and still no one uses that one, everyone is still using zap, logrus, etc.
Golang is also a disaster in that the stdlib has "log/slog" as the approved logging method, but i.e. the 'http.Server.ErrorLog' is the old 'log.Logger' you're not supposed to use, and for backwards compatibility reasons it's stuck that way. The go stdlib is full of awkward warts because it maintains backwards compatibility, and also included a ton of extra knobs that they got wrong.
The argument of structured vs unstructured logging, arguments around log levels (glog style numeric levels, trace/debug/info/warn/error/critical or some subset, etc), how to handle lazy logging or lazy computations in logging, etc etc, it's all enough that the stdlib can't easily make a choice that will make everyone happy.
Rust as a language has provided enough that the community can make a sl4j like facade (conveniently, the clearly named 'log' crate seems to be the winner there), so there's no reason it has to be in the stdlib.
> Golang included the "log" package, which was also a disaster. No one used it so they introduced log/slog recently, and still no one uses that one, everyone is still using zap, logrus, etc.
I honestly don't know how common it is to use zap, logrus and co on new projects anymore. I know I always go for slog.
> Golang is also a disaster in that the stdlib has "log/slog" as the approved logging method, but i.e. the 'http.Server.ErrorLog' is the old 'log.Logger' you're not supposed to use, and for backwards compatibility reasons it's stuck that way. The go stdlib is full of awkward warts because it maintains backwards compatibility, and also included a ton of extra knobs that they got wrong.
This isn't a practical problem at all. First of all, the stdlib shouldn't make backwards-incompatible changes just for the sake of logging. Second, slog.NewLogLogger()[0] exists as a way to get log.Loggers to be used with things that rely on the old log.Logger.
[0] https://pkg.go.dev/log/slog#NewLogLogger
Python includes logging, and packages generally use it. It's far from perfect, but it's important to have at least a standard interface so that all components use it in concert. When you have a proliferation of logging libraries with no standard, other packages either don't do any logging at all, or do it in idiosyncratic ways that are hard (or impossible) to integrate.
Rust's standard library hasn't received any major additions since 1.0 in 2015, back when nobody was writing web services in Rust so no one needed logging.
This is patently false. The majority of the "features" that get added to Rust in every release are additions to the standard library.
Depends on your definition of "major". Small utility functions are added on nearly every release, but they definitely aren't major.
println!() exists but there are more fancy crates like http://lib.rs/tracing and https://lib.rs/crates/log
Network without async works fine in std. However, rand, serde, and num_traits always seem to be present. Not sure why clap isn't std at this point.
Clap went through some major redesigns with the 4.0 release just three years ago. That wouldn't have been possible if clap 2.0 or 3.0 had been added to the stdlib. It's almost a poster child for things where libraries where being outside the stdlib allows interface improvements (date/time handling would be the other obvious example).
Rand has the issue of platform support for securely seeding a secure rng, and having just an unsecure rng might cause people to use it when they really shouldn't. And serde is near-universal but has some very vocal opponents because it's such a heavy library. I have however often wished that num_traits would be in the stdlib, it really feels like something that belongs in there.
FWIW, there is an accepted proposal (https://github.com/rust-lang/libs-team/issues/394) to add random number generation to std, and adding traits like in `num-traits` is wanted, but blocked on inherent traits.
> Not sure why clap isn't std at this point.
The std has stability promises, so it's prudent to not add things prematurely.
Go has the official "flag" package as part of the stdlib, and it's so absolutely terrible that everyone uses pflag, cobra, or urfave/cli instead.
Go's stdlib is a wonderful example of why you shouldn't add things willy-nilly to the stdlib since it's full of weird warts and things you simply shouldn't use.
> and it's so absolutely terrible that everyone uses pflag, ../
This is just social media speak for inconvenient in some cases. I have used flag package in lot of applications. It gets job done and I have had no problem with it.
> since it's full of weird warts and things you simply shouldn't use.
The only software that does not have problem is thats not written yet. This is the standard one should follow then.
Go is also famous for encouraging a culture of keeping down dependency count while exposing a simple to use package manager and ecosystem.
https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-... this article made the rounds here after the author pulled the thread on dependencies in Go
That's pulling on a dependency. An ugly case, but a very specific one relating to a tricky backwards-incompatible change that the Go team fudged on back in the pre-module times.
That story doesn't generally speak to the reality that most Go packages have rather small dependency graphs.
> why clap isn't std at this point.
Too big for many cases, there is also a lot of discussion around whether to use clap, or something smaller.
Clap is enormous and seems way too clever for everything I do. Last I looked it added 10+ seconds to compile time and hundreds of kbs to binary size. Maybe something like ffmpeg requires that complexity, but if I am writing a CLI that takes three arguments, it is a heavy cost.
> But because the stdlib isn't versioned
I honestly feel like that's one of Rust's biggest failings. In my ideal world libstd would be versioned, and done in such a way that different dependencies could call different versions of libstd, and all (sound/secure) versions would always be provided. E.g. reserve the "std" module prefix (and "core", and "alloc"), have `cargo new` default to adding the current std version in `cargo.toml`, have the prelude import that current std version, and make the module name explicitly versioned a la `std1::fs::File`, `std2::fs::File`. Then you'd be able to type `use std1::fs::File` like normal, but if you wanted a different version you could explicitly qualify it or add a different `use` statement. And older libraries would be using older versions, so no conflicts.
I'm afraid it won't work. The point of std lib is to be universal connection for all the libraries. But with versioned std I just can't see how can you have DateTime in std1, DateTime in std2 and use them interchangeably, for example being able to pass std2::DateTime to library depending on std1 etc. Maybe conversion methods, but it get really complicated really quickly
> On the topics it does cover, Rust's stdlib offers a lot. At least on the same level as Python, at times surpassing it.
Curious, do you have specific examples of that?
> if you stay away from the network and async
That's some "small print" right there.
It's already happening: https://cyberpress.org/malicious-rust-packages/
My personal experience (YMMV): Rust code takes 2x or 3x longer to write than what came before it (C in my case), but in the end you usually get something much more likely to work, so overall it's kind of a wash, and the product you get is better for customers - you basically front load the cost of development.
This is terrible for people working in commercial projects that are obsessed with time to market.
Rust developers on commercial projects are under incredible schedule pressure from day 0, where they are compared to expectations from their previous projects, and are strongly motivated to pull in anything and everything they can to save time, because re-rolling anything themselves is so damn expensive.
In my experience Rust development is no slower than C development (in a different environment) or C++ development (in a comparable project)
I think they were using "writing Rust" in the most strict sense: the part of the development cycle that involves typing the majority of the code, before you really start debugging in earnest and really make things work.
But their point is that "developing Rust" (as in, the entire process) ends up being a similar total effort to C, only with more up front "writing" and less work on the debugging phase.
And I'm saying it isn't any slower.
Thank you for the clarification, that's exactly what I was trying to say :).
Perhaps another way to phrase this: in Rust, you spend more time telling the compiler how your code is expected to work (making the borrow checker happy, adding sync traits on objects you "know" are thread safe because of how you use them or assurances the underlying hardware provides, etc etc etc). In return, the compiler does a lot of work to make sure the code will actually work how you think it's going to work.
A good example is a simple producer-consumer problem. Make a ring buffer. Push the current sys clk tick count register to the ring buffer every time there's a rising edge interrupt on a GPIO (e.x. hook up a button or something to it). Poll the ring buffer in another thread and as soon as there is a timestamp in the buffer, pop it and log it to a UART.
Compare writing this in C vs Rust. In the text book implementation of a producer-consumer ring buffer, you don't need locks.
In C, this is a problem I'd expect a senior-level candidate to be able to knock out in a 60 minute interview.
(aside: I'd never actually ask a question like this in an interview, because it heavily favors people who just happen to be familiar with the algorithm, which doesn't give me enough signal about the candidate. This is borderline like asking someone to implement a sort algorithm in an interview - it just tells you they know how to google or memorize things. /aside).
(aside 2: If I did ask this, I'd be more interested how they design the thing - I'd be looking for questions like "how many events per second? How should I size the ring buffer? Wait, you want me to hook up an IRQ to a button? Is there a risk of interrupt storm from bounce or someone being malicious? Do you want me to add a cooldown timer between events? If we overflow the ring buffer, what should the behavior be? How fast can the UART go on this system - can it even keep up with the input?" - I'd be far more interested in that conversation than actually seeing them write code for this. /aside2).
In Rust, it's a bit more tricky. You'll need to give the compiler hints (in the form of Sync traits that are no-ops) to tell it you know what you're doing is thread safe. It's not rocket science, but the syntax is kind of weird and it will take some putzing around or aid from your favorite AI to get it right.
All of Rust ends up like this - you must be more verbose telling the compiler your intent. In exchange, it verifies the code matches your intent. So the up front cost is higher.
I suspect a lot of people will just pull a ring buffer off crates.io instead of figuring out the right incantations to make the compiler happy.
I don't spend significant time making the borrow checker happy, because I learned how to write C++ that works.
¯\_(ツ)_/¯ I'd like to lean into that "YMMV" in my post, I'm coming from low level C that interacts with hardware, can't really speak to higher-level C++.
Some things in Rust just don't translate to the way you'd do them in C - e.x. using different types to say if a GPIO pin is input or output adds a ton of boiler plate, but lets the compiler assure you don't mistakenly try and use a pin configured as an input for output.
In general, the whole zero sized types paradigm in Rust leads to way more lines of code to accomplish the same thing (it all ends up compiled out in the end though).
For embedded, I'll stand by what I said: it takes longer to write idiomatic Rust but you are more likely to get functionally correct code in the end.
ZSTs are a choice, if you're not benefiting from them in development time or correctness, why are you making that choice?
Rust isn't just C++ with static analysis. The safety is only about 1/3 of what makes me enjoy working with it. Even if I just consider the effort taken before I present something to a compiler, the trait system means I don't spend anywhere near as much time writing copy/copy assignment/move/move assignment constructors.
Some of that 2/3 will be me making choices like ZSTs, but I believe that's a result of it being necessary to grok the whole language and style and the idioms. If I just try to write C in Rust, it will be bad Rust and I'll have a bad time - but the same goes for just writing C++ as C with classes. I deliberately didn't say 'C/C++' in my ancestor post, because I think it's terminology that undermines claims of authority.
I do happen to have embedded experience, but avoid drawing conclusions from it because it is so different to more 'standard' systems development. If you can forgive a tangent: I avoid the whole domain now, because it feels like a ghetto. Standards are poor, tools are poor, pay is seriously poor. If you want to write embedded software, you probably have to work for a hardware company with hardware company's engineering culture and revenue model. That's a pretty crap place to write software in the west. I can't comment on comparing what like to do embedded development in Rust. Frankly, I hope I never can.
Good on you for recognizing what you enjoy and going for it.
I like hardware, always have :).
FWIW, I’ve done embedded at multiple mag 7 companies, and if levels.fyi is to be believed I’m getting paid the same as my peers doing other types of software. There are certainly places that don’t do this though (I hear the defense industry is notorious for this).
The point about the tooling being ghetto is spot on though, that’s part of why Rust is so exciting for us. A package manager?! What is this magic!!!
Remind me again, how the”sufficiently senior dev” approach is panning out for places that are swapping off C/C++?
Because from here, it seems like they’re having a great time, which is extremely at-odds with your comment.
My experience in low-level stuff has been a mirror of the Linux kernel trying to take up Rust. Some people love it and fight hard for it, some hate it and fight hard against it, but most just see it as another tool in the toolbox and don't have strong feelings.
> Rust and JS are both famous for having minimal standard libraries
I'm all in favor of embiggening the Rust stdlib, but Rust and JS aren't remotely in the same ballpark when it comes to stdlib size. Rust's stdlib is decidedly not minimal; it's narrow, but very deep for what it provides.
C standard library is also very small. The issue is not the standard library. The issue is adding libraries for snippets of code, and in the name of convenience, let those libraries run code on the dev machine.
The issue is that our machines run 1970s OSes with a very basic security model, and are themselves so complex that they’re likely loaded with local privilege escalation attack vectors.
Doing dev in a VM can help, but isn’t totally foolproof.
It’s a good security model because everyone has the decency to follow a pull model. Like “hey, I have this thing, you can get it if you’re interested”. You decide the amount of trust you give to someone.
But NPM is more like “you’ve added me to your contact list, then it’s totally fine for me to enter your bedroom at night and wear your lingerie because we’re already BFF”. It’s “I’m doing whatever I want on your computer because I know best and you’re dumb” mentality that is very prevalent.
It’s like how zed (the editor) wants to install node.js and whatever just because they want to enable LSP. The sensible approach would have been to have a default config that relies on $PATH to find the language server.
> “you’ve added me to your contact list, then it’s totally fine for me to enter your bedroom at night and wear your lingerie because we’re already BFF”
I don't know if everyone will appreciate this, but I am in stitches right now... lol
This is a reason why so many enterprises use C#. Most of the time you just use Microsoft made libraries and rarely brings in 3rd party.
Having worked on four different enterprise grade C# codebases, they most certainly have plenty of 3rd party dependencies. It would absolutely be the exception to not have 3rd party dependencies.
Yes, but the 3rd party dependencies tend to be conveniences rather than foundational. Easier mapping, easier mocking, easier test assertions, so a more security minded company can very easily just disallow their use without major impact. If it's something foundational to your project then what you're doing is probably somewhat niche. Most of the time there's some dependency from Microsoft that's rarely worse enough to justify using the 3rd party one.
I dunno, there's definitely stuff that's convenience related, sure, and then there's stuff that's less so. Things like MediatR, Dapper, Serilog, Refit are all pretty common in .NET projects. There's usually always some library for generating PDFs for reporting purposes etc. The ecosystem of NuGet packages is pretty large. It's definitely not all just Microsoft dependencies, nor is that usually how developers in the ecosystem think in my experience.
OP might be remembering how things were 10 years ago or so, before .NET Core became the .NET. Although there are still many companies with legacy codebases running on .NET 4.
That aside, dependency graphs in .NET land still tend to be much smaller.
Or purchase third party libraries. This does two things - limits what you drag in and also if you drag it in you can sue someone for errors.
This definitely not why enterprise "chooses" C# and neither of these were design decisions like implied. MS would have loved to have the explosive, viral ecosystem of Node earlier in .NET's life. Regardless a lot of companies using C# still use node-based solutions on the web so a insular development environment for one tier doesn't protect them.
I am not so sure about that. .net core is the moment they opened up, making it cross platform, going against the grain of owning it as a platform.
If they see a gap in .net, which is filled in by a third party, they would have no problem qualms about implementing their own solution in .net that meets their quality requirements. And to be fair, .net delivers on that. This might anger some, but the philosophy is that it should be a batteries included one-stop shop, maybe driven by the culture of quite some ms shops that wouldn't eat anything unless ms feeds it them.
This has a consequence that the third-party ecosystem is a lot smaller, but I doubt MS regrets that. If you compare that to F#, things are quite different wrt filling in the gaps, as MS does not focus on F#. A lot of good stuff for F# comes from the community.
They actually had a pretty active community on CodePlex - I used and contributed to many projects there... they killed that in ... checks the web... 2017, replaced with GitHub, and it just isn't the same...
And yet of course the world and their spouse import requests to fetch a URL and view the body of the response.
It would be lovely if Python shipped with even more things built in. I’d like cryptography, tabulate/rich, and some more featureful datetime bells and whistles a la arrow. And of course the reason why requests is so popular is that it does actually have a few more things and ergonomic improvements over the builtin HTTP machinery.
Something like a Debian Project model would have been cool: third party projects get adopted into the main software product by a sworn-in project member who who acts as quality control / a release manager. Each piece of software stays up to date but also doesn’t just get its main branch upstreamed directly onto everyone’s laps without a second pair of eyes going over what changed. The downside is it slows everything down, but that’s a side-effect of, or rather a synonym for stability, which is the problem we have with npm. (This looks sort of like what HelixGuard do, in the original article, though I’ve not heard of them before today.)
Requests is a great example of my point, actually. Creating a brand-new Python venv and running `uv add requests` tells me that a total of 5 packages were added. By contrast, creating a new Rust project and running `cargo add reqwest` (which is morally equivalent to Python's `requests`) results in adding 160 packages, literally 30x as many.
I don't think languages should try to include _everything_ in their stdlib, and indeed trying to do so tends to result in a lot of legacy cruft clogging up the stdlib. But I think there's a sweet spot between having a _very narrow_ stdlib and having to depend on 160 different 3rd-party packages just to make a HTTP request, and having a stdlib with 10 different ways of doing everything because it took a bunch of tries to get it right. (cf. PHP and hacks like `mysql_real_escape_string`, for example.)
Maybe Python also has a historical advantage here. Since the Internet was still pretty nascent when Python got its start, it wasn't the default solution any time you needed a bit of code to solve a well-known problem (I imagine, at least; I was barely alive at that point). So Python could afford to wait and see what would actually make good additions to the stdlib before implementing them.
Compare to Rust which _immediately_ had to run gauntles like "what to do about async", with thousands of people clamoring for a solution _right now_ because they wanted to do async Rust. I can definitely sympathize with Rust's leadership wanted to do the absolute minimum required for async support while they waited for the paradigm to stabilize. And even so, they still get a lot of flak for the design being rushed, e.g. with `Pin`.
So it's obviously a difficult balance to strike, and maybe the solution isn't as simple as "do more in the stdlib". But I'd be curious to see it tried, at least.
IMHO, the ideal for package management in a programming language ecosystem might recognise multiple levels of “standardisation”.
At the top, you have the true standard library for the language. This has very strong stability guarantees. Its purpose is twofold: to provide universal implementations of essentials and to define standard/baseline interfaces for common needs like abstract data types, relational databases, networking and filesystems to encourage compatibility and portability.
Next, you have a tier of recognised but not yet fully standardised libraries. These might be contributed by third parties, but they have requirements for identifying maintainers, appropriate licensing and mandatory peer review of all contributions. They have a clear versioning policy and can make breaking changes in new major releases, but they also provide some stability guarantees along the lines of semver and older releases are normally available indefinitely. The purpose of this tier is to provide a wider range of functionality and/or alternative implementations, but in a relatively stable way and implementing standard interfaces where applicable to improve portability.
Finally, you have the free-for-all, anyone-can-contribute tier. This should still have a sane security model where people can’t just upload malware scripts that run automatically just because someone installed a package. However, it comes with few guarantees about stability or compatibility, except that releases of published packages will be available indefinitely unless there’s a very good reason to pull them where you obviously wouldn’t want to use one anyway. A package you like might be written by a single contributor who no longer maintains it, but if someone does write something useful that simply doesn’t need any further maintenance once it’s finished and does its job, there is still a place to share it.
I agree with your general idea. I'd add it also looks very similar to what typical GNU/Linux have in practice: blessed packages from the distros' repositories, and third-party repos for those who want them.
Debian also has something 'in the middle' with additional repositories that aren't part of the main distribution and/or contain proprietary software.
Or maybe just get comfortable with adding versions and deprecation. eg optparse to argparse (though tbf, I would have just preferred it was optparse2). Or maybe the problem is excessive stability commitments. I think I prefer languages that realize things can improve and are willing to say if you want to run 10 year old code, use a 10 year old compiler/runtime.
I think I prefer languages that realize things can improve and are willing to say if you want to run 10 year old code, use a 10 year old compiler/runtime.
IMHO, the trouble with that stance is that it leaves no path to incrementally update a long-lived system to benefit from any of those improvements.
Suppose we have an application that runs on 2025’s most popular platform and in ten years we’re porting it to whatever new platform is popular in 2035. Personally, I’d like to know that all the business logic and database queries and UI structure and whatever else we wrote that was working before will still be working on the new platform, to whatever extent that makes sense. I’d like to make only some reasonably necessary set of changes for things that are actually different between the two platforms.
If we can’t do that, our only other option is a big rewrite. That is how you get a Python 2 to Python 3 situation. And that, in turn, is how you get a lot of systems stuck on the older version for years, despite all the advantages any later versions might offer.
> leaves no path to incrementally update
Sure it does? Or do I not understand? It pushes the work to the application, not the language.
And it works fine on rails. If you want to stay on an EOL rails, you're on your own, or you can buy ongoing security backports from at least 2 (that I know of, maybe more) vendors. LTS lives roughly 2 years and then you either upgrade or deal with eol yourself.
That's not an apple-to-apple comparison, since Rust is a low-level language, and also because `reqwest` builds on top of `tokio`, an async runtime, and `hyper`, which is also a HTTP server, not just a HTTP client. If you check `ureq`, a synchronous HTTP client, it only adds 43 packages. Still more, but much less.
And in Go I can build a production-ready HTTPS (not just HTTP) server with just the standard library and a few lines of code. (0 packages).
That Rust does not have standard implementations of commonly-used features (such as an async runtime) is problematic for supply chain security, since then everyone is pulling in dozens (or hundreds) of fragmented 3rd-party packages instead of working with a bulletproof standard library.
And this is exactly why Go is winning: because it's actually rather easy to write "pure Go" utilities (no dependencies outside the standard library), which statically compile to boot (avoiding shared libraries).
> (cf. PHP and hacks like `mysql_real_escape_string`, for example.)
PHP is a fantastic resource to learn how to do proper backward compatibility and package management. By doing the exact opposite of whatever PHP does, mostly.
It might solve the problem, in as much as the problem is that not only can it be done, but it’s profitable to do so. This is why there’s no Rust problem (yet).
It won't, it's a culture issue
Most rust programmers are mediocre at best and really need the memory safety training wheels that rust provides. Years of nodejs mindrot has somehow made pulling into random dependencies irregular release schedules to become the norm for these people. They'll just shrug it off come up with some "security initiative* and continue the madness
I only personally know one Rust programmer (works in scientific HPC) and he’s fantastic, but in general I do get the sense that most Rust devs migrated from JS and are just now figuring out “omg strong typing and compiled code native to the client hardware is really nice!” and think it’s a ground breaking revelation.
Saying this as someone who is cautiously optimistic about Rust for my own work.
I agree partly. I love cargo and can’t understand why certain things like package namespaces and proof of ownership isn’t added at a minimum. I was mega annoyed when I had to move all our Java packages from jcenter, which was a mega easy setup and forget affair, to maven central. There I suddenly needed to register a group name (namespace mostly reverse domain) and proof that with a DNS entry. Then all packages have to be signed etc. In the end it was for this time way ahead. I know that these measures won’t help for all cases. But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming. Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning. But I don’t want to go away from package managers or easy to use/sharable packages either.
> But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming.
Since your comment starts with commentary on crates.io, I'll note that this has never been possible crates.io.
> Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning.
I don't think this has ever been true. AFAIK crates.io has always prevented registering two different crates whose names differ only in the use of dashes vs underscores.
> package namespaces
See https://github.com/rust-lang/rust/issues/122349
> proof of ownership
See https://github.com/rust-lang/rfcs/pull/3724 and https://blog.rust-lang.org/2025/07/11/crates-io-development-...
You are right. I remembered it wrong.
https://rust-lang.github.io/rfcs/0940-hyphens-considered-har...
Was from 2015 and the other discussions I remember were around default style and that cargo already blocks a crate when normalized name is equal.
The trusted publishing is rather new or? Awesome to see that they implemented it. Just saying that maven central required it already years ago.
Maven Central does not currently support OIDC-based authentication (commonly called "Trusted Publishing").
Didn’t know this term. After reading I wonder why short lived tokens get this monocle. But yeah I prefer OIDC over token based access as well. Only small downside I see is the setup needed for a custom OIDC provider. Don’t know the right terms out of my head but we had quite the fun to register our internal Jenkins to become a create valid oidc tokens for AWS. GitHub and GitHub Actions come with batteries included. I mean the downside that a huge vendor can easily provide this and a custom rolled CI needs extra steps / infrastructure.
I'm a huge Go proponent but I don't know if I can see much about Go's module system which would really prevent supply-chain attacks in practice. The Go maintainers point [1] at the strong dependency pinning approach, the sumdb system and the module proxy as mitigations, and yes, those are good. However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users.
The lack of package install hooks does feel somewhat effective, but what's really to stop an attacker putting their malicious code in `func init() {}`? Compromising a popular and important project in this way would likely be noticed pretty quickly. But compromising something widely-used but boring? I feel like attackers would get away with that for a period of time that could be weeks.
This isn't really a criticism of Go so much as an observation that depending on random strangers for code (and code updates) is fundamentally risky. Anyone got any good strategies for enforcing dependency cooldown?
[1] https://go.dev/blog/supply-chain
A big thing is that Go does not install the latest version of transitive dependencies. Instead it uses Minimal version selection (MVS), see https://go.dev/ref/mod#minimal-version-selection. I highly recommend reading the article by Russ Cox mentioned in the ref. This greatly decreases your chances of being hit by malware released after a package is taken over.
In Go, access to the os and exec require certain imports, imports that must occur at the beginning of the file, this helps when scanning for malicious code. Compare this JavaScript where one could require("child_process") or import() at any time.
Personally, I started to vendor my dependencies using go mod vendor and diff after dependency updates. In the end, you are responsible for the effect of your dependencies.
In Go you know exactly what code you’re building thanks to gosum, and it’s much easier to audit changed code after upgrading - just create vendor dirs before and after updating packages and diff them; send to AI for basic screening if the diff is >100k loc and/or review manually. My projects are massive codebases with 1000s of deps and >200MB stripped binaries of literally just code, and this is perfectly feasible. (And yes I do catch stuff occasionally, tho nothing actively adversarial so far)
I don’t believe I can do the same with Rust.
You absolutely can, both systems are practically identical in this respect.
> In Go you know exactly what code you’re building thanks to gosum
Cargo.lock
> just create vendor dirs before and after updating packages and diff them [...] I don’t believe I can do the same with Rust.
cargo vendor
cargo vendor
The Go standard library is a lot more comprehensive and usable than Node, so you need less dependencies to begin with.
Aside from other security features already mentioned Go also doesn't execute code at compile time by design.
There is no airtight technical solution, for any language, for preventing malicious dependencies making it into your application. You can have manual or automated review using heuristics but things will still slip through. Malicious code doesn't necessarily look obvious, like decoding some base64 and piping it into bash, it can be an extremely subtle vulnerability sprinkled in that nobody will find until it's too late.
RE dependency cooldowns I'm hoping Go will get support for this. There's a project called Athens for running your own Go module proxy - maybe it could be implemented there.
> However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users.
You can't, really, aside from full on code audits. By definition, if you trust a maintainer and they get compromised, you get compromised too.
Requiring GPG signing of releases (even by just git commit signing) would help but that's more work for people to distribute their stuff, and inevitably someone will make insecure but convenient way to automate that away from the developer
Historically, arguments of "it's popular so that's why it's attacked" have not held up. Notable among them was addressing Windows desktop security vulnerabilities. As Linux and Mac machines became more popular, not to mention Android, the security vulnerabilities in those burgeoning platforms never manifested to the extent that they were in Windows. Nor does cargo or pip seem to be infected with these problems to the extent that npm is.
> Nor does cargo or pip seem to be infected with these problems to the extent that npm is.
Easy reason. The target for malware injections is almost always cryptocurrency wallets and cloud credentials (again, mostly to mine cryptocurrencies). And the utter utter majority of stuff interacting with crypto and cloud, combined with a lot of inexperienced juniors who likely won't have the skill to spot they got compromised, is written in NodeJS.
Compared to the JS ecosystem and number of users both Python and Rust are puny, also the the NPM ecosystem also allowed by default for a lot of post-install actions since they wanted to enable a smooth experience with compiling and installing native modules (Not entirely sure how Cargo and PIP handles native library dependencies).
As for Windows vs the other OS's, yes even the Windows NT family grew out of DOS and Win9x and tried to maintain compatiblity for users over security up until it became untenable. So yes, the base _was_ bad when Windows was dominant but it's far less bad today (why people target high value targets via NPM,etc since it's an easier entry-point).
Android/iOS is young enough that they did have plenty of hindsight when it comes to security and could make better decisions (Remember that MS tried to move to UWP/Appx distribution but the ecosystem was too reliant on newer features for it to displace the regular ecosystem).
Remember that we've had plenty of annoyed discourse about "Apple locking down computers" here and on other tech forums when they've pushed notarization.
I guess my point is that, people love to bash on MS but at the same time complain about how security is affecting their "freedoms" when it comes to other systems (and partly MS), MS is better at the basics today than they were 20-25 years ago and we should be happy about that.
This comment seems to address users intentionally installing malware. I mean to address cracking, the situation where an attacker gains root or installs software that the user does not know about.
Preventing the user from installing something that they want to install is another issue completely. I'm hesitant to call it exactly security, though I agree that it falls under the auspices of security.
Cracking is a term related to removing copy-protections. Rooting or privilege escalation is better terms for what you're mentioning.
As for "users intentionally installing malware", Windows in the early 00s had a bunch of fundamentally insecure deployment models like ActiveX controls and browsers (IE especially) were more or less swiss cheese in terms of security even outside the ActiveX controls.
Visiting the wrong webpage was often enough to get crap on your computer.
My view is that once you have bad native code running on your computer there's a large chance that it's game-over (the modern sandboxes like WASM were designed to enforce a probably safe subset where regular kernel mistakes are shielded by another layer of abstraction that needs to be broken).
Even Linux has had privilege escalations every year as far as I know. Notarization/stores is just a way to try to keep check on what code runs on end-user computers that isn't sandboxed (and allow for revoking that code if found to be malicious), maybe Linux is slightly safer still but that's probably due to less older features in the Kernel, but Windows has for example recently gotten a rewritten font-parser in Rust (the previous font parser was a common exploitation point that was placed with a too high privilegie).
You can have security without having a walled garden. By trusting the user with the key of their own property.
You mean like the developers holding the npm-publishing keys that just allowed a worm to spread?
No. By NPM not allowing any package to run code on the developer's machine. I can trust npm (the software), but not the library. It's a very weird choice to just allow any package to run post install script. Especially when there's little to none verification done on npmjs side.
Developers can feel free to not secure their computer or sell their keys. But that not means npm should allow straight code push from their computers to everyone that has downloaded their library.
> Compared to the JS ecosystem and number of users both Python and Rust are puny
This is just plainly false in case of Python.
> also the the NPM ecosystem also allowed by default for a lot of post-install actions since they wanted to enable a smooth experience with compiling and installing native modules (Not entirely sure how Cargo and PIP handles native library dependencies).
Rust is already "native" so Cargo doesn't need to do anything.
Python has the logic to do native builds baked into pip and friends, so a Python package can just specify what to build in the manifest. But it also allows for precompiled wheels, and most popular packages do them for all major OSes, so users rarely need to actually build stuff in practice.
> It's not "node" or "Javascript" the problem, it's this convenient packaging model.
That and the package runtime runs with all the same privileges and capabilities as the thing you're building, which is pretty insane when you think about it. Why should npm know anything outside of the project root even exists, or be given the full set of environment variables without so much as a deny list, let alone an allow list? Of course if such restrictions are available, why limit them to npm?
The real problem is that the security model hasn't moved substantially since 1970. We already have all the tools to make things better, but they're still unportable and cumbersome to use, so hardly anything does.
pnpm (maybe yarn too?) requires explicit allowlisting of build scripts, hopefully npm will do the same eventually
> security model
yep, some kind of seccomp or other kind of permission system for modules would help a lot. (eg. if the 3rd party library is parsing something and its API only requires a Buffer as input and returns some object then it could be marked "pure", if it supports logging then that could be also specified, and so on)
For all the other things I like about yarn, it still executes build scripts willy-nilly, so I am looking at switching to pnpm. I'm sure my $work is going to love me changing up the build toolchain again... PHP's composer on the other hand requires an allowlist in the project's composer.json. I never would have thought PHP would be the one to be getting stuff like this right.
Still, I think the "allow-scripts" section or whatever it's called should be named "allow-unrestricted-access-to-everything". Or maybe just stick "dangerously-" in front, I dunno, and drop it when the mechanism is capable of fine-grained privileges.
Deno also requires allowlisting npm scripts. It also has a deeper permissions model in general.
Every time I look at a new project, my face falls when it's written in Rust. I simply don't trust a system that pulls in gigabytes of god-knows-what off the cloud, and compiles it on my box. It's a real barrier to entry, for me.
When I download a C project, I know that it only depends on my system libraries - which I trust because I trust my distro. Rust seems to expect me to take a leap in the dark, trusting hundreds of packagers and their developers. That might be fine if you're already familiar with the Rust ecosystem, but for someone who just wants to try out a new program - it's intimidating.
On Debian you can use the local registry for Rust which is backed by packages.
Though I will say, even as someone who works at a company that sells Linux distributions (SUSE), while the fact we have an additional review step is nice, I think the actual auditing you get in practice is quite minimal.
For instance, quite recently[1] the Debian package for a StarDict plugin was configured automatically upload all text selected in X11 to some Chinese servers if you installed it. This is the kind of thing you'd hope distro maintainers to catch.
Though, having build scripts be executed in distribution infrastructure and shipped to everyone mitigates the risk of targeted and "dumb" attacks. C build scripts can attack your system just as easily as Rust or JavaScript ones can (in fact it's probably even easier -- look at how the xz backdoor took advantage of the inscrutability of autoconf).
[1]: https://www.openwall.com/lists/oss-security/2025/08/04/1
You don't know that about a C project. And you still don't know what lurks in its 1000th reimplementation of http header parsing.
Well, yes I do know when a C project only depends on my system libraries, because otherwise it won't compile. That's the point.
Furthermore, for the purposes of this discussion, it really doesn't matter what code there is in the C project. What's there has been put there by the people who run the project. If they are malicious, then at least I know who they are. With Rust, I'm downloading and compiling code from many, many third parties. I have no idea who they are. The potential for one of them to be malicious is much, much higher.
You do not know if developers are malicious when there are many contributors.
Now we are talking about NPM other time about: https://en.wikipedia.org/wiki/XZ_Utils_backdoor
I think this is right about Rust and Cargo, but I would say that Rust has a major advantage in that it implements frozen + offline mode really well (which if you use, obviously significantly decreases the risks).
Any time I ever did the equivalent with NPM/node world it was basically unusable or completely impractical
Pnpm (a very popular npm replacement) makes completely locked packages easy and natural and ultra fast:
https://pnpm.io/cli/install
Benchmarks:
https://pnpm.io/benchmarks
pnpm is so laughably terrible compared to Cargo it's not even comparable in the same breath.
Why specifically? Your comment isn't very informative.
Anyhow, here a Claude.ai comparison: https://claude.ai/share/72d2c34c-2c86-44c4-99ec-2a638f10e3f0
Because it doesn't perform as advertised: wild amounts of inconsistencies in behavior (within and between versions), performance issues (pnpm exec adds 15s to all shebang'd execution time over npm/yarn/bun/etc.), etc. Version-to-version stability has been traditionally bad - it's half-baked software.
Claude doesn't know this, of course, because it can only read superficial summaries posted on the internet and has zero real experience actually using this dumpster fire.
There are ecosystems that have package managers but also well developed first party packages.
In .NET you can cover a lot of use cases simply using Microsoft libraries and even a lot of OSS not directly a part of Microsoft org maintained by Microsoft employees.
2020 State of the Octoverse security report showed that .NET ecosystem has on average the lowest number of transitive dependencies. Big part of that is the breadth and depth of the BCL, standard libraries, and first party libraries.
The .NET ecosystem has been moving towards a higher number of dependencies since the introduction of .NET Core. Though many of them are still maintained by Microsoft.
The "SDK project model" did a lot to reduce that back down. They did break the BCL up into a lot of smaller packages to make .NET 4.x maintenance/compatibility easier, and if you are still supporting .NET 4.x (and/or .NET Standard), for whatever reason, your dependency list (esp. transitive dependencies) is huge, but if you are targeting .NET 5+ only that list shrinks back down and the BCL doesn't show up in your dependency lists again.
Even some of the Microsoft.* namespaces have properly moved into the BCL SDKs and no longer show up in dependency lists, even though Microsoft.* namespaces originally meant non-BCL first-party.
I think first-party Microsoft packages ought to be a separate category that is more like BCL in terms of risk. The main reason why they split them out is so that they can be versioned separately from .NET proper.
Sounds to me like another example of embrace - extend - extinguish
What ? This is their project from day one.
Supply chain attacks are scary because you do everything "right", but the ecosystem still compromises you.
But realistically, I think the sum total of compromises via package managers attacks is much smaller than the sum total of compromises caused by people rolling their own libraries in C and C++.
It's hard to separate from C/C++'s lack of memory safety, which causes a lot of attacks, but the fact that code reuse is harder is a real source of vulnerabilities.
Maybe if you're Firefox/Chromium, and you have a huge team and invest massive efforts to be safe, you're better off with the low-dependency model. But for the median project? Rolling your own is much more dangerous than NPM/Cargo.
Rust (and really, any but JS) ecosystem have a bit more "due dilligence" applied everywhere; I don't doubt someone will try to namesquat but chance of success are far smaller
> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise.
There was no decision in case of C/C++; it was just not a thing languages had at the time so the language itself (especially C) isn't written in a way to accommodate it nicely
> Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.
Any code you download and compile is running code on dev machine; and Go does have tools to do that in compile process too.
I do however like the by default namespacing by domain, there is no central repository to compromise, and forks of any defunct libs are easier to manage.
> Rust (and really, any but JS) ecosystem have a bit more "due dilligence" applied everywhere; I don't doubt someone will try to namesquat but chance of success are far smaller
I really agree, and I feel like it's a culture difference. Javascript was (and remains) an appealing programming language for tinkerers and hobbyists, people who don't really have a lot of engineering experience. Node and npm rose to prominence as a wild west with lots of new developers unfamiliar with good practices, stuck with a programming environment that had few "batteries included," and at a time when supply chain attacks weren't yet on everybody's minds. The barriers to entry were low and, well, the ecosystem sort of reflected that. You can't wash that legacy away overnight.
Rust in contrast attracts a different audience because of the language's own design objectives.
Obviously none of this makes it immune, and you can YOLO install random dependencies in any programming language, but I don't think any language is ever going to suffer from this in quite the same way and to the same extent that JS has simply due to when and how the ecosystem evolved.
And really, even JS today is not JS of yesteryear. Sure there are lots of bad actors and these bad NPM packages sneak in, but also... how widely are all of them used? The maturation of and standardization on certain "batteries included" frameworks rather than ad hoc piecing stuff together has reduced the liklihood of going astray.
I have a similar opinion but I think Java's model with maven and friends hits the sweet spot:
- Packages are always namespaced, so typosquating is harder - Registries like Sonatype require you to validate your domain - Versions are usually locked by default
My professional life has been tied to JVM languages, though, so I might be a bit biased.
I get that there are some issues with the model, especially when it comes to eviction, but it has been "good enough" for me.
Curious on what other people think about it.
Maven does not support "scripts" as NPM does, such as the pre-install script used for this exploit. With scripts enabled, the mere act of downloading a dependency requires a high degree of trust in it.
Downloading a dependency also requires a high degree of trust in whatever transitive dependencies that a trusted dependency decides to pull in.
While I agree that dependency tree size can be sometimes a problem in Rust, I think it often gets overblown. Sure, having hundreds of dependencies in a "simple" project can be scary, but:
1) No one forces you to use dependencies with large number of transitive dependencies. For example, feel free to use `ureq` instead of `reqwest` pulling the async kitchen sink with it. If you see an unnecessary dependency, you could also ask maintainers to potentially remove it.
2) Are you sure that your project is as simple as you think?
3) What matters is not number of dependencies, but number of groups who maintain them.
On the last point, if your dependency tree has 20 dependencies maintained by the Rust lang team (such as `serde` or `libc`), your supply chain risks are not multiplied by 20, they stay at one and almost the same as using just `std`.
On your last note, I wish they would get on that signed crate subset. Having the same dependency tree as cargo, clippy, and rustc isn't increasing my risk.
Rust has already had a supply chain attack propagating via build.rs some years ago. It was noticed quickly, so staying pinned to the oldest thing that worked and had no cve pop in cargo audit is a decent strategy. The remaining risk is that some more niche dependency you use is and always has been compromised.
Is serde maintained by the Rust team? I thought it was basically a one-man show owned by dtolnay
> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise.
The safest code is the code that is not run. There is no lack of attacks targeting C/C++ code, and odin is just a hobby language for now.
Don't worry about C or C++, we create the vulnerabilities ourselves !
I get the joke, but that makes me think.
What is worse between writing potentially vulnerable code yourself and having too many dependencies.
Finding vulnerabilities and writing exploits is costly, and hackers will most likely target popular libraries over your particular software, much higher impact, and it pays better. Dependencies also tend to do more than you need, increasing the attack surface.
So your C code may be worse in theory, but it is a smaller, thus harder to hit target. It is probably an advantage against undiscriminating attacks like bots and a downside against targeted attacks by motivated groups.
Using C++ daily, whenever I do js/ts are some javascript variant, since I don't use it daily, and update becomes a very complex task. frameworks and deps change APIs very frequently.
It's also very confusing (and I think those attack vectors benefit exactly from that), since you have a dependency but the dep itself dependent on another dep version.
Building basic CapacitorJS / Svelte app as an example, results many deps.
It might be a newbie question, but, Is there any solution or workflow where you don't end up with this dependency hell?
Don't use a framework? Loading a JS script on a page that says "when a update b" hasn't changed much in about 20 years.
Maybe I'm being a bit trite but the world of JavaScript is not some mysterious place separate from all other web programming, you can make bad decisions on either side of the stack. These comments always read like devs suddenly realizing the world of user interactions is more complicated and has more edge cases than they think.
There's no solution. The JS world is just nonstop build and dependency hell.
Being incredibly strict with TS compiler and linter helps a bit.
Not knowing that much about apt, isn't _any_ package system vulnerable, and purely a question of what guards are in place and what rights are software given upon install?
It's not the packaging tech. Apt will typically mean a Debian-based distro. That means the packages are chosen by the maintainers and updated only during specific time periods and tested before release. Even if the underlying software gets owned and replaced, the distro package is very unlikely to be affected. (Unless someone spent months building trust, like xz)
But the basic takeover... no, it usually won't affect any Debian style distro package, due to the release process.
Given the years (or decades) it takes updates to happen in Debian stable, it’s immune to supply chain attacks. You do get to enjoy vulnerabilities that have been out for years, though.
> it’s immune to supply chain attacks
Thats a strong statement that I can see aging very badly.
Security updates are basically immediate, even on stable flavors
Agreed with the first half, but giving up on convenient packaging isn't the answer.
Things like cargo-vet help as does enforcing non-token auth, scanning and required cooldown periods.
Out of the 789 npm packages in this incident, only 4 were ever used in any dependency tree of any Linux operating system (including Homebrew). Not in the affected versions, but ever.
If your Rust software observes a big enough chunk of the computer fever dream you are likely to end up with 2-3 digit amount of Rust dependencies, but they are probably all going to be high profile ones (tokio, anyhow, reqwest, the hyper crates, ...), instead of niche ones that never make it into any operating system.
This is not a silver bullet of course, but there seems to be an inverse correlation between "is part of any operating system dependency tree" and "gets compromised in an npm-like incident".
> The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine.
The alternative that C/C++/Java end up with is that each and every project brings in their own Util, StringUtil, Helper or whatever class that acts as a "de-facto" standard library. I personally had the misfortune of having to deal with MySQL [1], Commons [2], Spring [3] and indirectly also ATG's [4] variants. One particularly unpleasant project I came across utilized all four of them, on top of the project's own "Utils" class that got copy-and-paste'd from the last project and extended for this project's needs.
And of course each of these Utils classes has their own semantics, their own methods, their own edge cases and, for the "organically grown" domestic class that barely had tests, bugs.
So it's either a billion "small gear" packages with dependency hell and supply chain issues, or it's an amalgamation of many many different "big gear" libraries that make updating them truly a hell on its own.
[1] https://jar-download.com/artifacts/mysql/mysql-connector-jav...
[2] https://commons.apache.org/proper/commons-lang/apidocs/org/a...
[3] https://docs.spring.io/spring-framework/docs/current/javadoc...
[4] https://docs.oracle.com/cd/E55783_02/Platform.11-2/apidoc/at...
That is true, but the hand-rolled StringUtil won't steal your credentials and infect your machine, which is the problem here.
And what is wrong with writing your own util library that fits your use case anyway? In C/C++ world, if it takes less than a couple hours to write, you might as well do it yourself rather than introduce a new dependency. No one sane will add a third-party git submodule, wire it to the main Makefile, just to left-pad a string.
> That is true, but the hand-rolled StringUtil won't steal your credentials and infect your machine, which is the problem here.
Yeah, that's why I said that this is the other end of the pendulum.
> In C/C++ world, if it takes less than a couple hours to write, you might as well do it yourself rather than introduce a new dependency.
Oh I'm aware of that. My point still stands - that comes at a serious maintenance cost as well, and I'd also say a safety cost because you're probably not wrapping your homebrew StringUtils with a bunch of sanity checks and asserts, meaning there will be an opportunity for someone looking for a cheap source of exploits.
Wait what? That’s just fearmongering, how hard is it to add a few methods that split a string or pad it? It’s not rocket science.
> how hard is it to add a few methods that split a string or pad it?
In full generality, pretty hard. If you're just dealing with ASCII or Latin-1, no problem. Then add basic Unicode. Then combining characters. Then emojis. It won't be trivial anymore.
Full generality is not a practical target. You select your subset of the problem and you solve it. Supporting everything in a project is usually a fever dream.
This kind of thing is precisely the stuff that needs to be in stdlib.
C++ is hardly a good example to follow here, but then again it does have Boost which is the de facto "extended standard library" in practice, and gets treated differently from other dependencies by many.
> how hard is it to add a few methods that split a string or pad it?
Well, if you're in C/C++, you always risk dealing with null pointers, buffer overruns, or you end up with use-after-free issues. Particularly everything working with strings is nasty and error-prone if one does not take care of proper testing - which many "homegrown" libraries don't.
And that's before taking the subtleties of character set encodings between platforms into account. Or locale. Or any other of the myriad ways that C/C++ and even Java offer you to shoot yourself in the foot with a shotgun.
And no, hoping for the best and saying "my users won't ever use Unicode" or similar falls apart on the first person copying something from Outlook into a multi-line paste box. Or someone typing in their non-Latin name. Oh, and right-to-left languages, don't forget about these. What does "pad from left" even mean there? Is the intent of the user still "at the beginning of the string itself?" Or does the user rather want "pad at the beginning of the word/sentence", which in turn means padding at the end of the string?
There's so much stuff that can go horribly horribly wrong when dealing with strings, and I've seen more than my fair share just reading e-mail templates from supposed "enterprise" software.
My feeling is that languages with other packing models are merely less convenient, and there is no actual tangible difference security-wise. Just take C and replace "look for writable repositories". It just takes more work and is less uniform to say write a worm that looks for writable cmake/autoconf and replicate that way.
What would actually stop this is writing compilers and build systems in a way that isolates builds from one another. It's kind of stupid that all a compiler really needs is an input file, a list of dependencies, and an output file. Yet they all make it easy to root around, replicate and exfiltrate. It can be both convenient and not suffer from these style of attacks.
Not really. cmake and automake are for compiling the library, not for the downloading it. The gap between the two is what's get erased from npm. And it made worse because of the auto update set by default when `npm install` is run.
Not having a convenient package manager doesn't mean you don't need the functionality that's otherwise offered by third-party packages, it just means that you either need other means to obtain those third-party packages (usually reducing the visibility this dependency!) or implement them yourself (sometimes this is good, but sometimes this can also be very bad for security. Your DYI code won't get as many eyes and audits as the popular third party package!).
Must read: https://wiki.alopex.li/LetsBeRealAboutDependencies
Indeed, Rust's supply chains story is an absolute horror, and there are countless articles explaining what should be done instead (e.g. https://kerkour.com/rust-stdx)
TL;DR: ditch crates.io and copy Go with decentralized packages based directly on and an extended standard library.
Centralized package managers only add a layer of obfuscation that attackers can use to their advantage.
On the other hand, C / C++ style dependency management is even worse than Rust's... Both in terms of development velocity and dependencies that never get updated.
> countless articles explaining what should be done instead (e.g. https://kerkour.com/rust-stdx)
Don't make me tap the sign: https://news.ycombinator.com/item?id=41727085#41727410
> Centralized package managers only add a layer of obfuscation that attackers can use to their advantage.
They add a layer of convenience. C/C++ are missing that convenience because they aren't as composable and have a long tail of pre-package manager projects.
Java didn't start with packages, but today we have packages. Same with JS, etc.
Or from another angle, dpkg/apt is the package manager for C/C++ ...
Yeah, but it's not immune to supply chain attacks. Counting on maintainers of dpkg is not that different from counting on maintainers of random crate package.
I believe you, in that package management with dependencies without security mitigation is both convenient and dangerous. And I certainly agree this could happen for other package managers as well.
My real worry, for myself re the parent comment is, it's just a web frontend. There are a million other ways to develop it. Sober, cold risk assessment is: should we, or should we have, and should anyone else, choose something npm-based for new development?
Ie not a question about potential risk for other technologies, but a question about risk and impact for this specific technology.
> C/C++ .. a convenient package manager
Every time I fire up "cmake" I chant a little spell that protects me from the goblins that live on the other side of FetchContent to promise to the Gods of the Repo that I will, eventually, review everything to make sure I'm not shipping poop nuggets .. just as soon as I get the build done, tested .. and shipped, of course .. but I never, ever do.
Surely in this case the problem is a technical one, and with more work towards a better security model and practices we can have the best of both worlds, no?
Agreed, rust's cargo model is basically the worst part of that ecosystem right now. I've had developers submit pretty simple cli tools with hundreds and hundreds of dependencies. I guess there wasn't any lessons learned from the state of NPM.
Just a last month someone was trying to figure the cargo tree on which Rust package got imported implicitly via which package. This will totally happen in rust as well as long as you use some kind of package manager. Go for zero or less decencies.
It already did happen. It propogated via build.rs as well. But as I said elsewhere, ut doesn't help you to forgo dependencies part of rust tooling itself.
less?
Roll your own standard library - or go without one entirely
`#![no_std]`
Make it so others depend on you? :)
decencies?
It’ll probably happen eventually with Rust, but ecosystem volume and informal packaging processes / a low barrier to entry seem to be significant driver in the npm world.
(These are arguably good things in other contexts.)
I don’t get this
I installed the package, obviously I intend to run it. How does getting pwned once I run it manually differ from getting pwned once I install it? I’m still getting pwned
NPM default installation method does not really lock down you dependencies. It allows for update when the patch number (semver) is increased. Which is why those malware bump it up. Anyone who then run `npm install` will get it and will run the code.
In the early days the Node ecosystem adopted (from Unix) the notion that everything has to be its own micro package. Not only was there a failure to understand what it was actually talking about, but it was never a good fit for package management to begin with.
I understand that there's been some course correction recently (zero dependency and minimal dependency libs), but there are still many devs who think that the only answer to their problem is another package, or that they have to split a perfectly fine package into five more. You don't find this pattern of behavior outside of Node.
> In the early days the Node ecosystem adopted (from Unix) the notion that everything has to be its own micro package.
The medium is the message. If a language creates a very convenient package manager that completely eliminates the friction of sharing code, practically any permutation of code will be shared as a library. As productivity is the most important metric for most companies, devs will prefer the conveniently-shared third-party library instead of implementing something from scratch. And this is the result.
I don't believe you can have packaging convenience and avoiding dependency hell. You need some amount of friction.
It’s not even the convenience. It’s about trust. Npm makes it so that as soon as you add something to the dependency list, you trust the third party so completely you’re willing to run their code on your system as soon as they push an update.
It’s essentially remote execution a la carte.
Node is the embodiment of move and break things. Probably will not build anything that should last more than a few months on node.
Why the word "semblance" with regard to Go modules? Are you trying to say this system is lacking something?
An open question is why PyPI doesn’t have the same problem.
PyPI is also subject to supply chain attacks. What do you mean?
maybe the solution is what linux & co used for many years: have a team of people who vet and package dependencies.
do they follow the same process ? or is it harder to submit a package and vet it on rust/cargo ?
I hate to be the guy saying AI will solve it, but this is a case where AI can help. I think in the next couple of years we’ll see people writing small functions with Claude/codex/whatever instead of pulling in a dependency. We might or might not like the quality of software we see, but it will be more resistant to supply chain attacks.
How is this going to solve the supply chain attack problem at all though? It just obfuscates things even more, because once an LLM gets "infected" with malicious code, it'll become much more difficult to trace where it came from.
If anything, blind reliance on LLMs will make this problem much worse.
When there's a depedency, it's typically not for a small function. If you want to replace a full dependency package by your own generated code, you'll need to review hundreds of even thousands of line of code.
Now will you trust that AI didn't include its own set of security issues and will you have the ability to review so much code?
I wonder what the actual result will be. LLMs can generate functions quickly, but they're also keen to include packages without asking. I've had to add a "don't add new dependencies unless explicitly asked" to a few project configs.
For sure. I don't think the software ecosystem has come to terms with how things are going to change.
Libraries will be providing raw tools like - Sockets, Regex Engine, Cryptography, Syscalls, specific file format libraries
LLMs will be building the next layer.
I have build successful running projects now in Erlang, Scheme, Rust - I know the basic syntax of two of those but I couldn't write my deployed software in any of them in the couple of hours of prompting.
The scheme it had to do a lot of code from first principles and warned me how laborious it would be - "I don't care, you are doing it."
I have tools now I could not have imagined I could build in a reasonable time.
An approach I learnt from a talk posted to HN (I forget the talk, not the lesson) is to not depend on the outside project for its code, just lift that code directly in to your project, but to rely on it for the tests, requiring/importing it etc when running your own tests. That protects you from a lot of things (this kind of attack was not mentioned, afaic recall) but doesn’t allow bugs found by the other project to be missed either.
Then your dependency will be "AI getting it right every single time".
I don’t think I’ll live long enough to trust AI coding assistants with something like schema validation, just to name one thing I use dependencies for.
Go is just as bad.
Nope. Know the difference.
How is Go any worse or better than Rust when it comes to the ease of adding dependencies?
I read your comment as in, comparing to node, my bad.
With regards to Rust, crates are packages which may include opaque binaries, e.g. serde_derive, and the stdlib is weak, so imports of thousands of lines of code are basically necessary for otherwise fundamental features like async.
It's probably easier to add dependencies in Go, but in the end people/projects don't.
> but it's only a matter of time until it'll happen on the Rust ecosystem
Totally 100% agree, though tools like cargo tree make it more of a tractable problem, and running vendored dependencies is first class at least.
The one I am genuinely most concerned of is Golang. The way Dependencies are handled leaves much to be desired, I'm really surprised that there haven't been issues honestly.
The problem isn't specific to node. NPM is just the most popular repo so the most value for attacks. The same thing could happen on RubyGems, Cargo, or any of the other package managers.
NPM has about 4 million packages, Maven Central has about 3 million packages.
If this were true, wouldn't there have been at least one Maven attack by now, considering the number of NPM attacks that we've seen?
Been a while since I looked into this, but afaik Maven Central is run by Sonatype, which happens to be one of the major players for systems related to Supply Chain Security.
From what I remember (a few years old, things may have changed) they required devs to stage packages to a specific test env, packages were inspected not only for malware but also vulnerabilities before being released to the public.
NPM on the other hand... Write a package -> publish. Npm might scan for malware, they might do a few additional checks, but at least back when I looked into it nothing happened proactively.
npm is run by github / microsoft now, which also sells security products...
There were. They're just not as popular here. For example https://www.sonatype.com/blog/malware-removed-from-maven-cen...
Maven is also a bit more complex than npm and had an issue in the system itself https://arxiv.org/html/2407.18760v4
As of 2024, Maven had 1.5 trillion requests annually vs npm's 4.5 trillion - regardless of package count, 3x more downloads in total does make it a very big target (numbers from https://www.sonatype.com/state-of-the-software-supply-chain/...).
No. Having many packages might not be the only reason to start an attack. This post shows it is/was possible in the Maven ecosystem: https://blog.oversecured.com/Introducing-MavenGate-a-supply-...
One speculation would be is that most Java apps in the wild use way older Java versions (say 17/11, while the latest will LTS is 21).
Okay then, explain to me why this is only possible with NPM? Does it have a hidden "pwn" button that I don't know about?
https://docs.npmjs.com/cli/v8/using-npm/scripts
>Does it have a hidden "pwn" button that I don't know about?
Perhaps its package owners do.
NPM executes packages as you download them.
Make no mistake, Maven Central does get multiple malware components uploaded each year, though not nearly to the same extent as npm or pypi. Sonatype (my former employer) just doesn't report on these publicly each time it happens. It's not an isolated problem but certainly harder to do with maven.
I assume you're talking about malware uploaded to new artifact coordinates (possibly named so as to try to confuse users), not hijacking of existing artifact coordinates (group ID, artifact ID)?
generally yes, although hijacking can and has happened on Central with expired maintainer domains reclaimed by threat actor who can then republish malicious versions of a previously legit group/artifact ID. there's also the problem of mirrors or copies of hijacked npm being replicated on Central -https://x.com/SocketSecurity/status/1993389518247149907
Hoe many daily downloads does Maven have?
The concern is not 'could' happen, but _does_ happen. I know this could occur in many places. But where it seems highly prevalent is NPM.
And I am genuinely thinking to myself, is this making using npm a risk?
Just use dependency cooldown. It will mitigate a lot of risk.
If you started your Node project yesterday, wouldn't that mean you'd get the fix later?
no, because if you used dependency cooldown you wouldn't be using the latest version when you start your project, you would be using the one that is <cooldown period> days/versions old
edit: but if that's also compromised earlier... \o/
Obviously you bypass the cooldown to fix critical issues.
NPM is the largest possible target for such an attack.
Attack an important package, and you can get into the Node and Electron ecosystem. That's a huge prize.
Value is one thing but the average user (by virtue of being popular) will be just less clued in on any security practices that could mitigate the problem.
I've started to feel it is much more an npm problem than a node problem. One of the things I've started leaning on more is prioritizing packages from JSR [0]. JSR is a part of Deno's efforts, so is often easiest to use in Deno packages, but most of the things with high scores on JSR get cross-published to npm and the few that prefer JSR only there's an alright JSR bridge to npm.
Of course using more JSR packages does start to add more reason to prefer Deno to Node. Also, there are still some packages that are deno.land/x/ only (sort of the first version of JSR, but no npm cross-compatibility) worth checking out. For instance, I've been impressed with Lume [1], a thoughtful SSG that's sort of the opposite of Astro in that it iterates at a slow, measured pace, and doesn't try to be a kitchen sink but more of workbench with a lot of tools easy to find. It's deno.land/x/ only for now for reasons I don't entirely agree with but I can't deny that JSR can be quite a step up in publishing complexity for not exactly obvious gain.
[0] https://jsr.io/
[1] https://lume.land/
Node is fine, the issue lies in its package model and culture:
* Many dependencies, so much you don't know (and stop caring) what is being used.
* Automatic and regular updates, new patch versions for minor changes, and a generally accepted best practice of staying up to date on the latest versions of things, due to trauma from old security breaches or big migrations after not updating for a while.
* No review, trust based self-publishing of packages and instant availability
* untransparent pre/postinstall scripts
The fix is both cultural and technological:
* Stop releasing for every fart; once a week is enough, only exception being critical security reasons.
* Stop updating immediately whenever there's an update; once a week is enough.
* Review your updates
* Pay for a package repository that actually reviews changes before making them widely available. Actually I think the organization between NPM should set that up, there's trillion dollar companies using the Node ecosystem who would be willing and able to pay for some security guarantees.
Microsoft owns npmjs.com. They could pay for AI analysis of published version deltas, looking for backdoors and malware.
I’m not a node/js apologist, but every time there is a vulnerability in NPM package, this opinion is voiced.
But in reality it has nothing to do with node/js. It’s just because it’s the most used ecosystem. So I really don’t understand the argument of not using node. Just be mindful of your dependencies and avoid updating every day.
It has everything to do with node/js. Because the community believes in tiny dependencies that must be updated as often as possible and the tooling reflects that belief.
it's interesting that staying up to date with your dependencies is considered a vulnerability in Node
Having a cooldown is different from never updating. I don’t think waiting a few days is a bad security practice in any environment, node or otherwise.
But only if most of everyone else doesn't do so.
People who live on the edge of updates always risk vulnerabilities and incompatibility issues. It’s not about node, but anything software related.
We chose to write our platform for product security analytics (1) with PHP, primarily because it still allows us to create a platform without bringing in over 100 dependencies just to render one page.
I know this is a controversial approach, but it still works well in our case.
"require": { "php": ">=8.0",
1. https://github.com/tirrenotechnologies/tirrenoNot sure what the language has anything to do with it, we've built JavaScript applications within pulling in 100s of NPM packages before NPM was a thing, people and organizations can still do so today, without having to switch language, if they don't want to.
Does it require disciple and a project not run by developers who just learned program? You betcha.
I might say that every interpreter has a different minimum dependency level just to create a simple application. If we're talking about Node.js, there's a long list of dependencies by default.
So yes, in comparison, modern vanilla PHP with some level of developer discipline (as you mentioned) is actually quite suitable, but unfortunately not popular, for low-dependency development of web applications.
The language and capabilities of the platform indeed have a lot of influence on how many packages the average project depends on.
With Swift on iOS/macOS for instance it’s not strange at all for an app to have a dependency tree consisting of only 5-10 third party packages total, and with a little discipline one can often get that number down to <5. Why? Because between the language itself, UIKit/AppKit, and SwiftUI, nearly all needs are pretty well covered.
I think it’s time to beef up both JavaScript itself as well as the platforms where it’s run (such as the browser and Node), so people don’t feel nearly as much of a need to pull in tons of dependencies.
You can do that with node.js too. It’s the libraries themselves that tries to bring in the whole world. It’s a matter of culture.
> If we're talking about Node.js, there's a long list of dependencies by default.
But that's not true? I initialize a project locally, there is zero dependencies by default, and like I did five years ago, I can still build backend/frontend projects with minimal set of dependencies.
What changed is what people are willing/OK with doing. Yes, it'll require more effort, obviously, but if you want things to be built properly, it usually takes more effort.
Perhaps, the right wording here might be that Node.js encourages the use of npm packages even for simple tasks.
I agree that in any case, it's the courage/discipline that comes before the language choice when creating low-dependency applications.
Ah yes PHP, the language known for its strong security...
Oh yes, let's remember PHP 4.3 and all the nostalgic baggage from that era.
Modern PHP is leagues above Javascript
That's not a high bar to clear
How so?
7.0 added scalar type declarations and a mechanism for strong typing. PHP 8.0 added union types and mixed types. PHP enforces types at runtime, Javascript/Typescript do not. PHP typesystem is built into the language, with Js u either need jsdoc or Typescript both of which wont enforce runtime type checks, Typescript even adds a buildstep. php-fpm allows u to not care about concurrency too much because of an isolated process execution model, with js based apps you need to be extremely careful about concurrency because of how easy you can create and access global stuff. PHP also added a lot of syntax sugar over the time especially with 8.5 my beloved pipe operator. And the ecosystem is not as fragile as Javascripts.
Just keep the number of packages you use to a minimum. If some package itself has like 200 deps uninstall that and look for an alternative with less deps or think if you really need said package.
I also switched to Phoenix using Js only when absolutely necessary. Would do the same on Laravel at work if switching to SSR would be feasible...
I do not trust the whole js ecosystem anymore.
Did Phoenix not require npm at some point or is that not true?
At the beginning, but not anymore. You still have the option to pull libraries and packages but is not really required by default.
Oh that's great news I will have to look at it again then. That was a huge turn-off for me, to take one of the most well respected and reliable eco systems and then to pull in one of the worst as a dependency. Thank you for clearing that up.
Node doesn't have any particular relation to NPM? You don't have to download 1000 other people's code. Writing your own code is a thing that you are legally allowed to do, even if you're writing in Javascript.
Yes, and you can code in assembly as well if you want it. But: that's not how 99% of the people using node is using it so that it is theoretically possible to code up every last bit yourself is true but it does not contribute to the discussion at all.
An eco-system, if it insists on slapping on a package manager (see also: Rust, Go) should always properly evaluate the resulting risks and put proper safeguards in place or you're going to end up with a massive supply chain headache.
Writing code yourself so as not to cultivate 1000 dependencies you can't possibly ensure the security of is not the same as writing assembly. That you even reach for that comparison is indicative of the deep rot in Javascript culture. Writing your own code is perceived as a completely unreasonable thing to be doing to 99% of JS-devs and that's why the web performs like trash and has breaches every other day, but it's actually a very reasonable thing to be doing and people who write most any other language typically engage in the writing of own code on a daily basis. At any rate, JS the language itself is fine, Node is fine, and it is possible to adopt better practices without forsaking the language/ecosystem completely.
> That you even reach for that comparison is indicative of the deep rot in Javascript culture.
Sorry?
No, I'm the guy that does write all of his code from scratch so you're entirely barking up the wrong tree here. I am just realistic in seeing that people are not going to write more code than they strictly speaking have to because that is the whole point of using Node in the first place.
The Assembly language example is just to point out the fact that you could plug in at a lower level of abstraction but you are not going to because of convenience, and the people using Node.js see it no different.
JS is a perfectly horrible little language that is now being pushed into domains where it has absolutely no business being used (I guess you would object to running energy infrastructure on Node.js and please don't say nobody would be stupid enough to do that).
Node isn't fine it needs a serious reconsideration of the responsibilities of the eco-system maintainers. See also: Linux, the BSDs and other large projects for examples of how this can be done properly.
I feel like there are merits to your argument but that you have a larger anti-JS bias that's leaking through. Not that there aren't problems with Node itself, but as many people have pointed out, there are plenty of organizations writing in Node that aren't pwn'd by these sorts of attacks because we don't blindly update deps.
Perfect is the enemy of good; dependency cooldown etc is enough to mitigate the majority of these risks.
> I feel like there are merits to your argument but that you have a larger anti-JS bias that's leaking through.
Familiarity breeds contempt.
What's the problem?
I think JS is great. It's simple, anybody can use it.
TypeScript is excellent too. The structural type system is very convenient.
It's not going to replace Rust in cases where performance is essential or where you want strict runtime type checking or whatever, but for general use and graphical applications JS seems like a great pick.
I often hear people complain about JS, but really, how is it any worse than say Python?
> I often hear people complain about JS, but really, how is it any worse than say Python?
That's not the flex you think it is.
So no JS, no Python - tell us where it's at then. Rust, Go, Kotlin, Swift, C#?
The truth is typically somewhere in the middle. I feel you though. I'm that way with Ruby/Bundler.
Reality has an anti-JS bias.
So your supposed to write your own posthog? be serious
I tell people this over and over and over: every time you use a third party dependency, especially an ongoing one, you should consider that you are adding the developers to your team and importing their prior decisions and their biases. You add them to your circle of trust.
You can't just scale out a team without assessing who you are adding to it: what is their reputation? where did they learn?
It's not quite the same questions when picking a library but it is the same process. Who wrote it? What else did they write? Does the code look like we could manage it if the developer quits, etc.
Nobody's saying you shouldn't use third party dependency. But nobody benefits if we pretend that adding a dependency isn't a lot like adding a person.
So yeah, if you need all of posthog without adding posthog's team to yours, you're going to have to write it yourself.
> I tell people this over and over and over: every time you use a third party dependency, especially an ongoing one, you should consider that you are adding the developers to your team and importing their prior decisions and their biases. You add them to your circle of trust.
Thanks! Now, I will also tell this to developers.
Yes. If your shop is serious about security, it is in no way unreasonable to be building out tools like that in-house, or else paying a real vendor with real security practices for their product. If you're an independent developer, the entirety of Posthog is overkill, and you can instead write the specific features you need yourself.
We had created a sort of Posthog, but for product security analytics (1), and after 4 years of development I can confirm it's not something that you can easily create in-house.
1. https://github.com/tirrenotechnologies/tirreno
If they have a HTTP API using standard authentication methods it's not that difficult to create a simple wrapper. Granted a bit more work if you want to do things like input/output validation too, but there's a trade-off between ownership there and avoiding these kinds of supply-chain attacks.
> Granted a bit more work if you want to do things like input/output validation too,
A bit? A proper input validator is a lot of work.
If you aim for 100% coverage of the API you're integrating with, sure. But for most applications you're going to only be touching a small surface area, so you can validate paths you know you'll hit. Most of the time you probably don't need 100% parity, you need Just Enough for your use-case.
That's an excellent way to get bitten.
I'm not sure how you mean.
To my understanding, there's less surface area for problems if I have a wrapper over the one or two endpoints some API provides, which I've written and maintain myself, over importing some library that wraps all 100 endpoints the API provides, but which is too large for me to fully audit.
npm has been the official package manager for node since forever (0.8 or earlier iirc). I think even before the io.js fork and merge.
Professionally I am a fulltime FE Dev using Typescript+React. My Backends for my side projects are all done in C#, even so I'd be fluent in node+typescript for that very reason. In a current side project, my backend only has 3 external package dependencies, 2 of which are SQLite+ORM related. The frontend for that sideproject has over 50 (React/Typescript/MaterialUI/NextJS/NX etc.)
.NET being so batteries-included is one of its best features. And when vulnerabilities do creep in, it's nice to know that Microsoft will fix it rather than hoping a random open source project will.
There's only two kind of technologies.
The ones that most people use and some people complain about, and the ones that nobody uses and people keep advocating for.
This a common refrain on HN, frequently used to dismiss what may be perfectly legitimate concerns.
It also ignores the central question of whether NPM is more vulnerable to these attacks than other package managers, and should therefore be considered an unreasonable security risk.
It's not just npm, you should also not trust pypi, rubygems, cargo and all the other programming language package managers.
They are built for programmers, not users. They are designed to allow any random untrusted person to push packages with no oversight whatsoever. You just make an account and push stuff. I have no doubt you can even buy accounts if you're malicious enough.
Users are much better served by the Linux distribution model which has proper maintainers. They take responsibility for the packages they maintain. They go so far as to meet each other in person so they can establish decentralized root of trust via PGP.
Working with the distributions is hard though. Forming relationships with people. Participating in a community. Establishing trust. Working together. Following packaging rules. Integrating with a greater dynamic ecosystem instead of shipping everything as a bloated container whose only purpose is to statically link dynamic libraries. Developers don't want to do any of that.
Too bad. They should have to. Because the npm clusterfuck is what you get when you start using software shipped by totally untrusted randoms nobody cares to know about much less verify.
Using npm is equivalent to installing stuff from the Arch User Repository while deliberately ignoring all the warnings. Malware's been found there as well, to the surprise of absolutely no one.
There are far too many languages and many packages for each of them for this (good) idea to be practicable.
You can go very far with just node alone (accepts typescript without tsc, has testing framework,...). Include pg library that has no dependencies. Build a thin layer above node and you can have pretty stable setup. I got burnt so many times that I think it is simply impossible to build something that won't break within 3 months if you start including batteries.
When it comes to frontend, well I don't have answers yet.
You can write simple front-end without reactive components. Most pages are not full blown apps and they were fine for a very long time with jQuery, whose features have been largely absorbed into plain js/dom/CSS.
> Serious question: should someone develop new technologies using Node any more?
I think we have given the Typescript / Javascript communities enough time. These sort of problems will continue to happen regardless of the runtime.
Adding one more library increases the risk of a supply-chain attack like this.
As long as you're using npm or any npm-compatible runtime, then it remains to be an unsolved recurring issue in the npm ecosystem.
Node itself is still fine and you can do a lot these days without needing tons of library. No need for axios when we have fetch, there's a built-in test runner and assertion library.
There are some things that kind of suck (working with time - will be fixed by the Temporal API eventually), but you can get a lot done without needing lots of dependencies.
> Serious question: should someone develop new technologies using Node any more?
Serious answer: no.
I think I'm going to just use a static site generator, maybe add some WASM modules built with a language that has a sane package manager and enjoy my life instead of getting involved with this cluster of a show.
Hell no.
You need standalone dependencies, like Tailwind offers with its standalone CLI. Predators go where there prey is. NPM is a monoculture. It's like running Windows in the 90's; you're just asking for viruses. But 90% of frontend teams will still use NPM because they can't figure anything else out.
Building websites =/= Developing new technologies.
Yup! No new technologies have been invented or discovered thru building websites since CSS 1.0 in 1996.
Even worse! We lost <FRAME> along the way.
Just lock your packages to patch versions, make sure to use versions that are at least a week old.
And maybe don't update your dependencies very often.
If I had to bet, the most likely and pragmatic solution will be to have dependencies cooldown and that's it
If everyone does it, then it becomes less effective, because there'd be fewer early testers to experience and report issues, no?
Yes, it's gonna be heuristics all way down. This problem isn't solved formally but the ecosystem(s) having these issues are too big to be discarded "just" because of that.
Node and npm are not the same things. I'm not even a developer. You're seriously a developer?
If you're looking for practical recommendations how to work with npm maintaining reasonable safety expectations, my post here mostly covers it: https://worklifenotes.com/2025/09/24/npm-has-become-a-russia...
You have this issue with ALL external code though. npm/node and javascript overall may exacerbate this problem, but you have it with any other remote repository too - often without even noticing it unless you pay close attention; see the xz-utils backdoor, it took a while before someone noticed the sneaky payload. So I don't think this works as a selective filter against using node, if you have a use case for it.
Take ruby - even before when a certain corporation effectively took over RubyCentral and rubygems.org, almost two years ago they also added a 100.000 download limit. That is, after that threshold was passed, the original author was deprived of the ability to remove the project again - unless the author resigns from rubygems.org. Which I promptly did. I could not accept any corporation trying to force me into maintaining old projects (I tend to remove old projects quickly; the licence allows people to fork it, so they can maintain it if they want to, but my name can not be associated with outdated projects I already abandoned, since newer releases were available. The new corporate overlords running rubygems.org, who keep on lying about "they serve the community", refused to accept this explanation, so my time came to a natural end at rubygems.org. Of course this year it would be even easier since they changed the rules to satisfy their new corporate overlords anyway: https://blog.rubygems.org/2025/07/08/policies-live.html)
You forget to account for the fact that the xz-utils backdoor was extremely high effort. Literally a high skilled person building trust over time. While it's obviously possible and problematic, it's still a scaling/time issue.
Npm has been a shit show from day 1. Unfortunately, Industry momentum and vc funded "fail fast, fail often" is a hell of a drug.
EDIT: Coffee hasnt kicked in yet, that was harsher than I intended. For what it's worth, it's not specifically/solely NPM/nodes fault, more of a convergance of the above and the ecosystem/users just as much as any of the Node/NPM devs/maintainers in combination with it having such a large attack cross section. Even if it had a reputation for being bulletproof and secure as fuck there's still such a large userbase with huge potential if exploited that'd it'd almost assuredly inevitably be compromised from time to time regardless.
While I feel we could use a whole lot less javascript on the web (client and server side both), without a competitor or something, it's shear size ensures any such expliot/issue gets amplified 1000x versus nearly any other project save for maybe major OS's and Browsers themselves.
Serious answer: no.
Couldn't similar issues happen with Rust, Python, Dart, C#, Java, Ruby? Supply chain attacks are not unique to Node / NPM.
I'm sure the list of available attacks are somewhat different, but you can get pwned in all of these ecosystems.
> Serious question: should someone develop new technologies using Node any more?
Please, no.
It is an absolutely terrible eco system. The layercake of dependencies is just insane.
Node the technology can be used without blindly relying on the update features of npm. Vet your dependency trees, lock your dependency versions at patch level and use dependency cooldown.
This is something you also need to do with package managers in other languages, mind you.
If everybody in your country drives on the right side of the road you could theoretically drive on the left. But you won't get very far like that.
People use Node because of the availability of the packages, not the other way around.
> People use Node because of the availability of the packages, not the other way around.
That is not why I use Node. Incidentally, I also use Bun.js, and pnpm for most package management operations. I also use Typescript instead of raw JS.
I use Node and these related tools fundamentally because:
- I like the isomorphism of the code I write (same language for server and client)
- JS may have many warts, but IMO it has many advantages many other languages lack, it is rapidly improving, and TS makes it even more powerful and the bad part parts manageable. One ting that has stuck with me over the many years of using JS/TS is just how direct and free-of-ceremony everything is. Want a functional style? It supports it to some extent without much fuss. Want something akin to OOP? You can object literal with method-style function, "constructors" that are regular functions, even no-fuss prototypical inheritance, if you want to go that far. Also, no need for any complicated dependency injection (DI), you can just implement pure DI with regular functions, etc. I don't get why you hate JS/TS so much.
- I use Bun.js as an alternative to Node that has more batteries included, so that I can limit my exposure to too many external packages. I add packages only if I absolutely need them, and I audit them thoroughly. So, no, although I may use some packages, I am not on the Node ecosystem just because I want to go on a package consumption spree.
- I use pnpm for installing and managing package, and it by default prevents packages from taking any actions during installation; I just get their code.
Would you consider your use cases typical for the average Node.js ecosystem denizen?
That’s not a very good analogy. Doing what I suggested is not illegal and doesn’t prevent you from using packages from npm. It’s more akin to due diligence: before driving, you check that your car is safe to drive. At the gas and service station, you choose the proper fuel, proper lubricants and spare parts from a reputable vendor which are appropriate for your car.
Nobody - and I mean absolutely nobody - using Node.js has fully audited all of the dependencies they use and if we find somewhere in a cave a person that did that they are definitely not going to do it all over again when something updates.
I can guarantee that any financial institution which has standard auditing requirements and is using Node.js has fully audited all of the dependencies they use.
Outside that, the issue is not unique to Node.js.
Sorry, but that had me laughing out loud.
No, they haven't.
I should know, I check those companies for a living. This is one of the most often flagged issues: unaudited Node.js dependencies. "Oh but we don't have the manpower to do that, think about how much code that is".
When I last looked (as a consulting dev in a bank or three, horrified) absolutely they had not!
If this was in the US, all financial institutions need to audit their code to comply with NIST SP 800-53.
If they haven’t, it would be ethically dubious for you to not report it.
In theory there is no difference between theory and practice, but in practice there is.
> If they haven’t, it would be ethically dubious for you to not report it.
I can report all I want, someone needs to act on that report for it to have an effect.
There are people out there who think that some static analysis tool plugged into their CI/CD pipeline is the equivalent of a code audit.
But the aforementioned NIST standard requires a lot more from auditing and operations.
I know what the standard requires. I also know what happens in practice and typically the auditors are understaffed, overworked and their technical expertise is lower than it should be. As a result a lot of stuff slips through the cracks.
What does get flagged though is not getting employee permission for putting photos on the 'team' page. It's good they flag that. I'd rather they also went in much deeper on tech issues.
I've reviewed 270 companies to date. I have yet to find a single one that had audited the source code they imported. It's not untypical to find an installation that automatically updates a whole raft of dependencies during the build phase. And absolutely nobody looks at that code until something breaks.
In my experience, most devs and companies don't consider the dependencies they load 'their' code. They only look at the code they write, not everything they deploy.
These were all multinationals, with very significant US presence.
The list of affected packages are all under namespaces pretty much nobody uses or are subdependencies of junk libraries nobody should be using if they're serious about writing production code.
I'm getting tired of the anti-Node.js narrative that keeps going around as if other package repos aren't the same or worse.
You need to explain how one is supposed to distinguish and exclude "namespaces pretty much nobody uses" when writing code in this ecosystem. My understanding is that a typical Node developer pretty much has no control over what gets pulled in if they want to get anything done at all. If that's the case, then you don't have an argument. If a developer genuinely has no control, then the point is moot.
How is this situation any different from any other ecosystem? I think you don't have an argument here other than that npm is a relatively large public repository. Bad actors and ignorant developers are everywhere else too.
There are plenty of npm features to help assess packages and prevent unintended updates, but nothing replaces due diligence.
Debian does not excuse security compromises by saying "oh nobody uses that far corner of our repository anyway".
The only way a worm like this spreads is usage of the affected packages. The proliferation itself is clear evidence of use.
Ok, I'll bite; which package repos are "the same or worse" than those of nodejs?
All of them. The issue at hand is not limited to a specific language or tool or ecosystem, rather it is fundamental to using a package manager to install and update 3rd party libraries.
I see a bunch under major SaaS vendor namespaces that have millions of weekly downloads…?
Popular junk is still junk
I never, ever, do development outside of a podman container these days. Basically if I am going to run some code from somewhere and I haven't read it, it goes in a container.
I know its not foolproof, but I can't believe how often people run code they haven't read where it can make a huge mess, steal secrets, etc. I'll probably get owned someday, I'm sure, but this feels like a bare minimum.
Probably because it’s fine 99.99% of the time and humans aren’t intuitively good at handling risk that functions like that. Besides, security is something handed off to specialists to free the devs up to focus on building things in most companies. We’re not going to change that no matter how much it represents some ideal.
It's like seatbelts/car seats.
Indeed!
> if I am going to run some code from somewhere and I haven't read it, it goes in a container
How does this work? Every single npm package has tons of dependency tree nodes
Everything runs in the container and cannot escape it. Its like a sandbox.
You have to make sure you're not putting any secrets in the container environment.
You are just reducing the blast radius with use of podman; you will likely need secrets for your app to work, which will be exposed regardless of the podman approach.
Most people don’t have NPM keys in their application containers.
If you're developing in a container then you would have to be doing it without doing something like say, mounting your home directory into it.
The reality here is this is the sort of attack SELinux should be good at stopping (it's not because no one uses SELinux, the policies most commonly used don't confine the user profile in a useful way, and a whole bunch of tools love ambient credentials in environment variables).
>You have to make sure you're not putting any secrets in the container environment.
How does this work exactly? containers still need env vars and access to databases and cloud environments. Without these the container is just useless isolated pod.
Not who you asked, but I have a similar setup. I can run everything I need for local development in that image (db, message queue emulator, cache, other services). So, setting things like environment variables or running postgres work the same as they do outside the container.
The image itself isn't the same image that the app gets deployed in, but is a portable dev environment with everything needed to build and run my apps baked in.
This comes with some nice side effects like being able to instantly spin up clean work environments on my laptop, someone elses, or a remote vm.
This really depends on your setup. If possible, I have local development containers as much as possible. nginx, postgres, redis, etc. I have several containers, each only has access to what it needs. We have an isolated cloud environment for development, in its own aws account.
Its not going to stop attacks, but it will limit blast radius a lot.
Maybe don't use JavaScript on the backend.
All right then, keep your secrets.
No it is not.
I didn't read this as separate containers.
How are you doing this in practice? These are npm packages. I don't see how could reasonably pull in Posthog's SDK in a container.
What do you mean? You can drop into bash in a container and run any arbitrary command, so `npm install foo` works just fine. Why would posthog's SDK be a special case?
I think the issue is more about what else has to go into or be connected to that container. Posthog isn't really useful if it's air-gapped. You're going to give it keys to access all kinds of juicy databases and analytics, and those NPM tokens, AWS/GCP/Azure credentials, and environment variables are exactly what it exfiltrates.
I don't run much on the root OS of my dev machine, basically everything is in a container or VM of some kind, but that's more so that I can reproduce my environment by copying a VMDK than in an effort to limit what the container can do to itself and data it has access to. Yeah, even with root access to a VM guest, an attacker they won't get my password manager, personal credit card, socials, etc. that I only use from the host OS... But they'll get everything that the container contains or has access to, which is often a lot of data!
You're severely limiting the blast radius. This malware works by exfiltrating secrets during installation, if I understood it correctly. If you would properly containerize your app and limit permissions to what is absolutely required, you could be compromised and still suffer little to no consequences.
Of course, this is not a real defense on its own, its just good practice to limit blast radius, much like not giving everybody admin rights.
> Upon execution, the malware downloads and runs TruffleHog to scan the local machine, stealing sensitive information such as NPM Tokens, AWS/GCP/Azure credentials, and environment variables.
Even a properly containerized app will still have these things, because you need things like environment variables (that contain passwords, api keys, etc) for your app to function.
You could create/run thin proxies for every external service that handle the auth side, and run each in a separate container. Orchestrate everything with docker-compose. Need to connect to cloud services for local development? Have a container with a proxy that transparently handles authentication. Now only that container has the secrets for talking to that service.
That's a lot of work though, and increases the difference between your local dev environment and prod.
Sure, but only the container is affected and it is always your responsibility to grant as little access as possible to the various credentials you may need to supply that environment. AFAICT with this worm, if you don't supply write-level GitHub credentials to the container (and you shouldn't!) and you install infected packages, the exploit goes no further.
> if you don't supply write-level GitHub credentials to the container (and you shouldn't!)
Really? What if your application needs to write to Github?
Because you need your application code to interact with Posthog's code. But if they're running in separate containers...how are you doing that. Surely you are not writing an api layer for every npm dependency you use.
This is smart and a good first step. Everyone can't be trusted to do the security dance flawlessly, though. We need sane defaults. Least privilege by default for 3rd-party code. Deno's headed in the right direction with this. But I think the solution needs to exist deeper in the stack. The surge in popularity of `curl -fsSL https://my-cool-ai-starup.ai/install.sh | bash` style installers is particularly concerning to me in this regard.
I ssh into a second local user and do development there instead with tmux.
I send mail to a demon which runs MsBuild and mails the output back to me.
You could still leak API keys
The same podman that had three new CVE breakouts not even two weeks ago?
Containers do not contain.
Security is oft done in layers. Podman is just another layer that might fail. There's no such thing as the perfect solution.
Another effective strategy I learned of recently that seems like it would have avoided this is to wait months before using new versions of packages.
Most attacks on popular packages last at most a few months before detection.
Using Podman over Docker is probably an even safer bet in that regard. But QEMU or something for an extra layer of safety and paranoia is probably the next best thing.
Would it have prevented this attack? It would still have published the secrets from your container to github.
the whole point of doing such things in ephemeral containers is to not have your secrets accessible from said container
We're monitoring this activity as well and updating the list of affected packages here: https://www.wiz.io/blog/shai-hulud-2-0-ongoing-supply-chain-...
Currently reverse engineering the malicious payload and will share our findings within the next few hours.
Why the biggest package mess is always with the Node ecosystem?
Why in particular this community still insists on preemptively updating all deps always, on running complicated extra hooks together with package installation and pretending this all is good engineering practices? ("Look, we have so plenty of things and are so busy, thus it must be good")
Why certain kind of mindset is typical to this community?
Why the Node creator abandoned his creation years ago?
Why, oh why?
Node is the new PHP: massive ecosystem, enormous mind share, and made up for a large part by programming newbies for whom JS is their first language and have no serious understanding of engineering practices, security and quite eager to use a library rather than wasting hours figuring out how to pad a string.
Because it is not a serious ecosystem run by serious people. Do you know what serious people do? They have package repositories with people called "maintainers", who are, crucially, trusted members of a community who don't write the software they package. "Oh but that's GATEKEEPING!", they screech. Yes, that's the entire point. Gatekeeping prevents shit like this from happening. There's a reason why this doesn't happen to Debian, but JavaScript developers get defensive and mean when you suggest that maybe the equivalent of a public S3 bucket isn't the best way to host a package repository.
Agreed that some level of gatekeeping and some level of friction to protect quality are useful things.
[flagged]
Feels good, just for a second, to type pretence you're above everyone doesn't it? Just for those few seconds, you're better than a big whole arbitrary collection of people, and for those few seconds you have relief from the reality of your life.
No. I am tired of seeing people dragged into poor quality environments for the wrong reasons. These people would do much better using other tools.
Your attempt to make it personal does not compute.
Well it certainly doesn't feel good to get your shit pwned every two weeks.
The reality of my life is that I have to write code targeting Node because of choices that other people made for their libraries and products.
I didn't hate JS & npm with a passion before. I do now, and it hasn't even been two years.
Slightly OT, but who is HelixGuard?
The website is a mess (broken links, broken UI elements, no about section)
There is no history on webarchive. There is no information outside of this website and their "customers" are crypto exchanges and some japanese payment provider.
This seems a bit fishy to me - or am I too paranoid?
Based in Singapore / Japan according to X: https://x.com/HelixGuard_ai
GitLab's write-up mentions a dead man's switch where "The malware continuously monitors its access to GitHub (for exfiltration) and npm (for propagation). If an infected system loses access to both channels simultaneously, it triggers immediate data destruction on the compromised machine. "
https://about.gitlab.com/blog/gitlab-discovers-widespread-np...
Because PostHog's "Talk to a human" chat instead gets a grumpy gatekeeping robot (which also doesn't know how to get you to a working urgent support link), and there's nothing prominently on their home page or github about this:
Hey PostHog! What version do we need to avoid?
co-founder here. We mentioned it in the main thread about this: https://news.ycombinator.com/item?id=46032650 and on status.posthog.com
- posthog-node 4.18.1, 5.13.3 and 5.11.3
- posthog-js 1.297.3
- posthog-react-native 4.11.1
- posthog-docusaurus 2.0.6
If you make sure you're on the latest version you should be good.
Thanks. Also - maybe change "talk to a human" to "talk to a grumpy robot" :)
Hm did you click on "help" (on the right side) -> "Email our support engineer" when logged in?
Ahhh, TBH I didn't look on the right. I dug through the menu on the left, thinking the right hand bar (which has the rotated labels) was all getting-started/docs related things. In my defense I have a fairly wide monitor and tend to full-screen the browser.
Your status page isn't clear, but are all versions between the compromised and "safe to install" versions compromised or just the ones listed?
For example I installed `posthog-react-native` version `4.12.4` which is between the `4.11.1` version which is compromised and the safe to install version `4.13.0`. Is that version compromised or not?
The only compromised versions are the ones listed. Any other versions are fine.
Thank you for the confirmation. I have updated to 4.13.0 anyway.
This is now the main thread. Though dang likes to merge dupes.
Have a slack channel with them, these are the versions they mentioned: posthog-node 4.18.1 posthog-js 1.297.3 posthog-react-native 4.11.1 posthog-docusaurus 2.0.6
"No Way To Prevent This" Says Only Package Manager Where This Regularly Happens
Parent comment is an indirect reference to US mass shootings:.
> "'No Way to Prevent This,' Says Only Nation Where This Regularly Happens" is the recurring headline of articles published by the American news satire organization The Onion after mass shootings in the United States.
Source: https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_...
See also Xe Iaso's posts about CVEs in the C ecosystem (https://xeiaso.net/shitposts/no-way-to-prevent-this/CVE-2025...)
There's nothing technically different between NPM and, say, Cargo, here that would save Cargo, is there?
I would say that npm likely has easier solutions here compared to Cargo.
Well before the npm attacks were a thing, we within the Rust project, have discussed a lot of using wasm sandboxing for build-time code execution (and also precompiled wasm for procedural macros, but that's its own thing.) However the way build scripts are used in the Rust ecosystem makes it quite difficult enforce sandbox while also enabling packages to build foreign code (C, C++ invoke make, cmake, etc.) The sandbox could still expose methods to e.g. "run the C compiler" to the build scripts, but once that's done they have an arbitrary access to a very non-trivial piece of code running in a privileged environment.
Whereas for Javascript rarely does a package invoke anything but other javascript code during the build time. Introduce a stringent sandbox for that code (kinda deno style perhaps?) and a large majority of the packages are suddenly safe by default.
This is a cultural problem created through a fundamental misunderstanding (and mis-application) of Unix philosophy. As far as I'm aware the Rust ecosystem doesn't have a problem appropriately sizing packages which in turn reduces the overall attack surface of dependencies.
It's kinda funny because "Unix philosophy" was never a coherent thing in the first place. Arguably Plan 9 came the closest to that in practice, but, well, you might note it didn't exactly have a strong uptake. Unix itself is a pile of hacks though, and it's both sad and amusing to watch people trying to divine some kind of methodology out of that.
This has nothing to do with package sizes. Cargo was just hit with a phishing campaign not too long ago, and does still use tokens for auth. NPM just has a wider surface area.
I agree, but imo the Rust ecosystem has the same problem. Not to the extent of NPM, but worse than C/C++.
No Preventative Measures (NPM)
You can host your own NPM reg, and examine every package, but your manager probably is NOT going to go for that.
Sounds like something a union should enforce as part of a drive to protect programmer professionalism.
The circumstances for this are not unique to NPM. The popularity is what makes it so susceptible to these attacks.
It's not just the popularity, it's partly the update mechanism and partly the culture. In what sane world would you always pull in all the newest things, regardless or whether you need them or not? This is a default at build time for so many setups. If you absolutely must use that package manager, at least lock down your versions, and update selectively. I don't even know if that's possible to do with the dependencies' dependencies (and so on), or are people forced to just pull in whatever, every time.
Okay then, tell me a way to prevent this.
An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series", as is the de-facto standard in NPM dependencies. Therefore, it's up to each dependency-user to validate newer versions of dependencies before publishing a new version of their own package. Lots of manual attention needed, so a slower pace of releases. This is a good thing!
Another example: all Debian packages are published to unstable, but cannot enter testing for at least 2-10 days, and also have to meet a slew of conditions, including that they can be and are built for all supported architectures, and that they don't cause themselves or anything else to become uninstallable. This allows for the most egregious bugs to be spotted before anyone not directly developing Debian starts using it.
You forgot to mention it is also tied to provable namespaces. People keep saying that NPM is just the biggest target...
Hate to break it to you but from targeting enterprises, java maven artifacts would be a MASSIVE target. It is just harder to compromise because NPM is such shit.
Maven Central verifies the domain used for the package namespace, too. You need to create a DNS TXT entry with a key.
This adds a bit more overhead to typo squatting, and a paper trail, since a domain registrar can have identity/billing information subpoenaed. Versus changing a config file and running a publish command...
Maven central also requires package signing. You're not stealing my signing key. It's on a yubikey. Game over, you can't publish malware in my name using my key.
> An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series"
You can definitely do this.
To be honest, you just end up with the same thing via dependabot/renovate.
Yes, that's why I said "typically" and "rarely".
You can specify a dependency version range in Maven artifacts. But the Maven community culture and default tooling behaviour is to specify exact versions.
You can specify an exact dependency version in npm packages. But the npm community culture and default tooling behaviour is to specify version ranges.
Even if a maintainer uses a bot to bump dependency versions, most typically they will test if their package works before publishing an updated version, and also because this release work is manual (even if the bot helps out), it takes some time after the dependency is released for upstream consumers of it to endorse and use it. Therefore, nobody consuming foo 1.0.4 will use dependency bar 2.3.5 until foo 1.0.5 is released... whereas an npm foo 1.0.4 with bar dependency "^2.3.0" will give its users bar 2.3.6 from the very moment bar 2.3.6 is released, even without a foo 1.0.5 release.
Other languages seem to publish dependencies as self-contained packages whose installation does not require running arbitrary shell scripts.
This does not prevent said package from shipping with malware built in, but it does prevent arbitrary shell execution on install and therefore automated worm-like propagation.
You have separate people called "maintainers", and they're the ones who build and upload packages to the repository. Crucially, they're not the people who write the software. You know, like Linux has been doing since forever. https://wiki.debian.org/DebianMaintainer Instead of treating your package repository like a trash can at a music festival, you can treat it more like a museum, curated by experts. Unfortunately, this isn't quite the devil-may-care attitude the Node ecosystem is so accustomed to, and will be met with a lot of whining, so it never happens. See y'all in two weeks when this happens again.
Build packages from source without any binaries (all the way down) and socially audit the source before building.
https://bootstrappable.org/ https://reproducible-builds.org/ https://github.com/crev-dev
The same way it always has been done - vendor your deps.
That literally makes no difference at all. You’ll just vendor the malicious versions. No, a lock file with only exact versions is the safe path here. We haven’t seen a compromise to existing versions that I know of, only patch/minor updates with new malicious code.
I maintain that the flexibility in npm package versions is the main issue here.
You are using the word "vendoring" differently than i do, i mean some kind of private fork of the repository.
You are using the word differently than everyone else I think. I’ve never heard someone using that word to mean maintain private forks. Then again, even private forks don’t protect you much more than package lock files and they are way more overhead IMHO.
You still need some out-of-band process to pull upstream updates and aside from a built-in “cool down” (until you merge changes) I see that method as having a huge amount of downside.
Yes, you sidestep malicious versions pushed to npm but now you own the build process for all your dependencies and you have to find time to update (and fix builds if they break) all your dependencies.
Locking to a specific version and waiting some period of time (cool down) before updating is way easier and jus as safe IMHO.
Vendoring literally just means grabbing the source code from origin and commit it to your repo after a review. The expectation that every repo has important regular updates for you is pure FOMO. And if I don't do random updates for fun, nothing will every break.
[redacted bullshit!]
> Version locking wont help you all the time, i.e. if you build fresh envs from scratch.
I'm confused on this. I would imagine it would protect/help you as long as releases are immutable which they are for most package managers (like npm).
> Vendoring literally just means grabbing the source code from origin and commit it to your repo after a review.
Hmm, I don't think it always necessarily means grabbing the source, it can also mean grabbing the built artifacts in my experience.
My biggest issue with vendoring dependencies is it allows for editing of said dependencies. Almost everywhere I've worked that vendored dependencies (copied source or built versions in and committed them) felt the siren song of modifying said dependencies which is hell to deal with later.
You are right about version locking, bullshit on my side, not sure what I was thinking.
I personally don't have a problem with the general ability to change vendor code. The question is whether you want it in an specific case or not. If you update frequently then certainly not. But that decision should be deliberate team policy.
> I personally don't have a problem with the general ability to change vendor code. The question is whether you want it in an specific case or not. If you update frequently then certainly not. But that decision should be deliberate team policy.
Fair, in the instances I ran into it was code that was downloaded and unzipped into a "js-library-name" folder but then the code was edited, even worse, the `.min.js` version wasn't touched, just the original one which led to some fun when someone "helpfully" switched to the min versions that didn't have the edit. IMHO, if you want to edit a library then you should be forking and/or making it super obvious that this is not "stock" library X. We also ran into issues with "just updated library Y" and only later realizing someone had modified the older version.
But yes, if it's deliberate and obvious then I don't have an issue modifying, just as long as the team understands it's "their" code now and there is no clean upgrade path in the future.
I'd require vendor code being committed to git and integrated into the CI/CD pipeline. It should be treated as if you own it, just with a policy whether you want to change it or not.
The difficulty to make changes obvious is same for forks and vendored commits, imho. You can write big warnings in commit messages, that's it I guess. Which kind of boils down to deliberate team policy again. But I generally prefer monorepos for various reasons.
To be fair this does only work in ecosystems where libraries are stable and don't break every 3 months as it often happens on the JS world.
You can vendor your left-pad, but good luck doing that with a third-party SDK.
... you vendor the third-party SDK? Nobody worth working with is breaking their SaaS APIs with that cadence.
that's what I do whenever feasible. Which is often
I think some system would need to dynamically analyze the code (as it runs) and record what it does. Even then, that may not catch all malicious activity. It's sort of hard to define what malicious activity is. Any file read or network conn could, in theory, be malicious.
As a SW developer, you may be able to limit the damage from these attacks by using a MAC (like SELinux or Tomoyo) to ensure that your node app cannot read secrets that it is not intended to read, conns that it should not make, etc. and log attempts to do those things.
You could also reduce your use of external packages. Until slowly, over time you have very little external dependencies.
Other than general security practices, here are few NPM ecosystem specific ones: https://github.com/bodadotsh/npm-security-best-practices
Hire an antivirus company to provide a safe and verified feed of packages. Use ML and automatic scanners to send packages to manual review. While Halting problem prevents us from 100% reliably detecting malware, at least we can block everything suspicious.
The list of packages looks like these are not just tiny solo-person dependencies-of-dependencies. I see AsyncAPI and Zapier there. Am I right that this seems quite a significant event?
AsyncAPI is used as the example in the post. It says the Github repo was not affected, but NPM was.
What I don't understand from the article is how this happened. Were the credentials for each project leaked? Given the wide range of packages, was it a hack on npm? Or...?
There is an explanation in the article:
> it modifies package.json based on the current environment's npm configuration, injects [malicious] setup_bun.js and bun_environment.js, repacks the component, and executes npm publish using stolen tokens, thereby achieving worm-like propagation.
This is the second time an attack like this happens, others may be familiar with this context already and share fewer details and explanations than usual.
Previous discussions: https://news.ycombinator.com/item?id=45260741
I don't get this explanation. How does it force you to run the infection code?
Yes, if you depend on an infected package, sure. But then I'd expect not just a list, but a graph outlining which package infected which other package. Overall I don't understand this at all.
Look at the diff in the article, it shows the “inject” part: the malicious file is added to the “preinstall” attribute in the package.json.
I still don't get it. Like, I understand that if you apply the diff you get infected. But... why would you apply the diff? How would you trick me to apply that diff to my package?
Someone could be tricked into giving their npm credentials to the attacker (e.g. via a phishing email), and then the attacker publishes new versions of their packages with the malicious diff. Then when the infected packages are installed, npm runs the malicious preinstall script which harvests secrets from the new machine, and if these include an npm token the worm can see which packages it has access to publish, and infect them too to continue spreading.
Thanks. I saw that sentence but somehow didn't parse it. Need a coffee :/
My understanding is, it's a worm that injects itself into the current package and publishes infected code to npm.
I compiled a list of NPM best practices one can adopt to reduce supply chain attack risks (even if there's no perfect security preventions, _always_): https://github.com/bodadotsh/npm-security-best-practices
Discussion on HN last time: https://news.ycombinator.com/item?id=45326754
For anyone publishing packages for others to use: please don't pin exact dependency versions. Doing so requires all your users to set "overrides" in their own package.json when your dependencies have vulnerabilities.
I have a shorter list of NPM best practices:
1. Don't
Do you know of anything similar for pip?
No.1: Run untrusted code in a sandbox! https://github.com/sandbox-utils/sandbox-venv
Most of the best practices can be translated to python ecosystem. It’s not exact 1:1 mapping but change few key terms and tools, the underlying practices should be the same.
Or copy that repo’s markdown into an llm and ask it to map to the pip ecosystem
Yeah, was mostly interested in the tooling.
Mitigate this attack vector by adding:
to your .npmrchttps://blog.uxtly.com/getting-rid-of-npm-scripts
Also add it to ~/.npmrc!
So, I do this because it's universally recommended, but why does it help?
Can't they just jam the malware into the package itself? It runs with the same permissions on my machine (in unit tests, node servers, etc).
> why does it help?
Because install scripts are being actively exploited, so blocking them will reduce your exposure. Install scripts will also run anywhere that runs npm ci, npm install, etc., including build pipelines.
> Can't they just jam the malware into the package itself
Yes. Disabling install scripts won't safeguard you from all attack vectors.
Yes, if the malware is injected in the application code this doesn’t prevent it.
But in some cases it could help for that. For instance, if the package runs in the browser and the payload requires file-system access, etc., then the attack can’t execute in the browser. And if in addition it was added to a life-cycle script, it would be mitigated.
At any rate, it’s worth having `ignore-scripts=true` because NPM life-cycle scripts are a common target (e.g., this one targets `preinstall`).
Run npm and yarn inside docker [1].
Infact, do this for all risky tools[2]
1 - https://github.com/ashishb/dotfiles/blob/067de6f90c72f0cf849...
2 - https://ashishb.net/programming/run-tools-inside-docker/
I think it's better to not run npm as root user on container. I would suggest adding --user 1000 to your docker run command.
> I think it's better to not run npm as root user on container. I would suggest adding --user 1000 to your docker run command.
Good point. Here's the improvement that work for me
https://github.com/ashishb/dotfiles/commit/fe4fb15fe867bf77a...
It gets tricky with private dependencies, then you have to pass some sort of token into the container to authenticate with the host when installing dependencies.
Definitely.
Would you prefer doing those tricks or exposing everything on your machine to random npm packages?
There is no easy solution to these problems.
The solutions that are effective also involve actually doing work, as developers, library authors, and package managers. But no, we want as much "convenience" as possible, so the issues continue.
Developers and package authors should use a lockfile, pin their dependencies, be frugal about adding dependencies, and put any dependencies they do add through a basic inspection at least, checking what dependencies they also use, their code and tests quality, etc.
Package managers should enforce namespacing for ALL packages, should improve their publishing security, and should probably have an opt-in verified program for the most important packages.
Doing these will go a long way to ameliorate these supply chain attacks.
There absolutely is an easy solution to these problems, and Linux has been doing it forever: package maintainers. Don't treat your repository like a superfund site, and it won't fill up with garbage.
That solution is not easy, and as stated it takes work. And there is a serious problem of funding people to do that work.
And it has not yet been demonstrated at PyPI/NPM scale, either.
I think if you generally depend on npm packages, being frugal is hard, because every random package works against you.
Last time my perception was also that publishing sec is a weak point. If at least heavily used packages would be forced to do manual security steps for publishing, it would help quite a bit as long the measures a safe.
Hundreds of people had access to publish the Zapier SDK, so it's little surprise they were eventually compromised! (https://bsky.app/profile/benmccann.com/post/3m6fdecsbdk2u)
The e18e community are reducing dependencies in popular libraries and building tools to prevent and reduce the impact of such attacks. Join if you want to help out! https://e18e.dev/
Just this morning, after trying to make the case over the past year, we had a change landed to remove more than a dozen dependencies from typescript-eslint! https://bsky.app/profile/benmccann.com/post/3m6fcjax7ec2h
FYI your first link is the same as your third link. It's correct as the third link, so the Zapier one is missing.
fixed!
>e18e
Yay!
>Discord
...ew.
This is a good sign that it's time to get packages off of NPM and come up with an alternative. For those who haven't heard of or tried Verdaccio [1], it may be an option. Relatively easy to point at your own server via NPM once you set it up.
[1] https://verdaccio.org/
I've had decent luck running it locally, but claude keeps screwing up the cool-down settings in my monorepo.
This is probably a common problem. Has anyone gotten verdaccio to enforce cool-down policies?
I also waste a ton of time because post-install scripts are disabled. Being able to cut them off from network access, and just run a local server with 2-4 week cool-down would help me sleep better at night + simplify the hell out of my build.
Shai-Hulud is the best thing to happen for npm.
It's much easier to demonstrate a problem (twice!) than to convince a herd that there is a problem.
I hope that other languages with similar package manager (looking at you, cargo) take note.
I always (very naively, I fully get it) wonder if someone at GitHub could take a minute and check the logs (if there are any at this level) from a week ago or so and scan them for patterns? The code seems to grab a few files off of GitHub, use Github actions, etc. -- perhaps there's a pattern in there that shows the attacker experimenting and preparing for this? I assume most people at this level have VPNs and so forth, but I'd never underestimate the amount of bad luck even those folks can have. Would be interesting, I know I'd have a look, if those logs existed.
I have first hand knowledge that they do, or at least that the data exists and can be queried in that way, but it’s a game of cat and mouse.
That's usually what those security companies do, they monitor all those repositories and look for patterns, then investigate anything suspicious.
Small plug for my open source project: you can use Cartography [https://github.com/cartography-cncf/cartography], to map your infra and then run this query (https://gist.github.com/achantavy/2cc7cc49919a8f761fea5e2d75...) to see if you're affected
Could npm adopt a reverse domain naming system similar to Java's for Maven libraries?
com.foo.bar
That would require domain verification, but it would add significant developer friction.
Also mandatory Dune reference:
"Bless the maker and his water"
I don't see how this solves the problem?
Some MFA requirement to publish a new version of the package would be a good idea. In me experience releasing a new version of software is a big enough deal that the product owner is on hand to authorize the release via a separate device no matter how automated the pipeline is.
I was thinking something similar to cargo-audit, because domain names don't really fix anything here
Why does every major Javascript vulnerability come off as something that would be easily avoided by not doing obviously stupid things (in this case automatically updating packages with no authentication, testing or oversight)?
Coding boot camps.
I looked through some of the GH repositories and - dear god - there are some crazy sensitive secrets in there. AWS Prod database credentials, various API keys (stripe, google, apple store, ...), passwords for databases, encryption keys, ssh keys, ...
I think hijacked NPM packages are just the tip of the ice berg.
I remember when I was young...
12 years ago NPM went down due to a very issue and getting stuff working again wasn't easy.
Someone got on GitHub and said something along the lines of "Get it together or get forked. "
I asked my manager if this was real or just a random guy talking. My manager, correctly said, it's just talk.
But it's different now. NPM is owned by Microsoft. One of the world's biggest companies should be able to sort things out.
Ohh well. Can't fix it.
MSFT can't even sort Windows out, if they can't charge for it in an Enterprise license, it gets left behind. Which is why W11 is a marketing grab bag of trash.
Postman getting hit is scary. For many teams, it's effectively an unmanaged password manager for API keys.
No one should have sensitive/production keys in Postman. That's a huge security lapse in the first place.
My devs don't have access to production keys at all (and would never need them).
I was going to tell my team that we didn't hear about a NPM supply chain attack for about 2 weeks and that we should celebrate.
I guess it's not going to happen...
Does NPM use any automatic scanners? Just scanning for eval/new Function/base64 and other tokens often used by malware, and requiring a manual review, could already help.
Also package manager should not run scripts.
Static scanning won't help. You can write this["eval"]() instead of eval(), therefore you can write this["e" + "v" + "a" + "l"](), and you can substitute (!![]+[])[!+[]+!+[]+!+[]] for "e", (![]+[])[+!+[]] for "a" (and so on: https://jsfuck.com/)
In this Turing-equivalent world, you can only know what actually executes (e.g. eval, fetch) by actually executing all code in the package and then see what functions got executed. Then the problem is the same as virus analysis; the virus can be written to only act under certain conditions, it will probe (e.g. look at what intepreter fingerprints, get the time of day, try to look at innocuous places in filesystem or network, measure network connection times, etc), so that it can determine it is in a VM being scanned, and go dormant for that time.
So the only thing that actually works is if node and other JS evaluators have a perfect sandbox, where nothing in a module is allowed (no network, no filesystem) except to explicit locations declared in the module's manifest, and this is perfectly tracked by the language, so if the module hands back a function for some other code to run, that function doesn't inherit the other code's network/fs access permissions. This means that, if a location is not declared, the code can't get to it at scanning time nor install time nor any time in the future.
This still leaves open the door for things like a module defining GetGoogleAnalyticsURL(params) that occasionally returns "https://badsite.com/copyandredirect?ga=...", to get some other module to eventually make a credential-exfiltrating network call, even if it's banned from making it directly or indirectly...
Well, writing obfuscated code like ["e" + "v" + "a" + "l"]() is already a huge red flag for sending the package to manual review. While it might be impossible to detect all methods of obfuscation, we could start with known methods.
Also, detecting obfuscated code sounds like an interesting and challenging task.
There's always some mathematician who tries to prove that locks on your doors "won't help" because the universe is infinite. Narrator: it is not
Deciding to put your resources into something that only a really stupid criminal would be caught by gives you a false sense of security.
Literally scanning for just "eval(" is entirely insufficient. You have to execute the code. Therefore you have to demand module authors describe how to execute code, e.g. provide a test suite, which is invoked by the scanner, and require the tests to exercise all lines of code. Provide facilities to control the behaviour of functions outside the module so that this is feasible.
This is a lot of work, so nobody wants to do it, so they palm you off with the laziest possible solution - such as literally checking for "eval(" text in the code - which then catches zero malware authors and wastes resources providing help to developers caught as a false positive, meanwhile the malware attacks continue unabated because no effective mechanism to stop them has been put in place.
Reminds me of the fraudster who sold fake bomb detectors to people who had a real need to stop bomb attacks. His detectors stopped zero bomb attacks. https://www.bbc.co.uk/news/uk-29459896
> Deciding to put your resources into something that only a really stupid criminal would be caught by gives you a false sense of security.
Interestingly enough, this is the premise for a lot of security in the physical world. Broken windows theory, door locks as a form of security in the first place, crimes of opportunity, etc.
But one should consider that in tech, the barrier to entry is a little higher and so maybe there are less 'dumb' criminals (or they don't get very far).
Which brings up a good point... is any company doing dynamic evaluation of the package updates to see what they are actually doing?
"Hey, we've figured out how to detect security vulnerabilities! We just need to solve the Halting Problem!"
Not aware of any NPM native ways but here are few community tools:
- https://socket.dev/blog/introducing-socket-firewall - https://github.com/lirantal/npq - https://bun.com/docs/pm/security-scanner-api
source: https://github.com/bodadotsh/npm-security-best-practices?tab...
Used the following script to see if I had any affected packages:
https://gist.github.com/considine/2098a0426b212f27feb6fb3b4d...
It checks yarn.lock for any of the above. Maybe needs a tweak or two but you should be able to run from a directory with yarn.lock
a concern i have is that it's only a matter of time before a similar attack is done to electron based apps (which also have packages installed using npm). probably worse because it's installed in your computer and can potentially get any information especially given admin privileges.
I’m starting an electronjs project in a few weeks and have been reading up on it. They make a big deal about the difference between the main and renderer processes and security implications. The docs are there and the advice given but it’s up to the developers to follow them.
That leads me to another point. Devs have to take responsibility for their code/projects. Everyone wants to blame npm or something else but, as software developers, you have to take responsibility for the systems you build. This means, among may other things, vetting code your code depends on and protecting the system from randomly updating itself with code you haven’t even heard about.
I'd really like to know how signal deals with this. It's supposedly super secure + stuff, but it's built on top of this ecosystem.
They probably don't willy-nilly install every new patch that comes down the pike?
I just cannot believe what a cluster of a show this is. Microsoft, you own NPM, sort it out. It's ridiculous.
I was looking at the packages affected and there are some recognizable names.
The ones that got my attentions are the @ensdomains/*, that are the legit packages and are probably in every Ethereum/EVM/blockchain related apps for the resolution of decentralized domain names.
A quick search shows those Ledger hardware wallet use those libs too [0]
So I guess they weren't just after API keys.
- [0] https://github.com/search?q=org%3ALedgerHQ%20%40ensdomain&ty...
I cannot believe how long it's taking github to shut this down, it's still going, and all they need to do is block a 18 char alphanumeric repo name from creation for this specific wave.
([0-9a-z]{18})
https://github.blog/security/supply-chain-security/our-plan-...
So github has some tools available to mitigate some of the problems tied to it. Probably not perfect for all use cases. But considering the current scale, it doesn't seem to have any effect, as enough publishers seem not to care.
I think npm should force higher standards on popular packages.
I don’t feel confident there is a way to use npm safely. The basic problem is that of curation, and there not being any except for what you do yourself. Every day brings new surprises, and osv.dev’s npm feed is a continuous horror show.
I would love to see the equivalent of a linux distro, a curated set of packages and package versions that are known to be compatible and safe. If someone offered this as a paid product businesses would pay for it.
They'll use AI to do it and it will be no better than what we have today. You will pay for it anyway.
`--ignore-scripts` should be the default behavior.
> Upon execution, the malware downloads and runs TruffleHog to scan the local machine, stealing sensitive information such as NPM Tokens, AWS/GCP/Azure credentials, and environment variables.
That's a wake up call to harden your operations. NPM Tokens, AWS/GCP/Azure credentials have no reason to be available in environments where packages may be installed. The same goes for sensitive environment variables.
That's the goal, but it's not feasible in e.g. professional settings. Much easier said than done, unfortunately.
I agree it's hard. But it's actually easier in professional settings. There are funds and you don't have an excuse to be lazy.
At minimum whatever you are working on should be built in docker. The package installation then would happen during the image build step. Yes it's easy to break out of the isolation environment but i am betting this malware does not.
NPM tokens should exist in some configuration/secret management solution not on your home directory. Devs have no business holding the NPM tokens. Same goes for sensitive environment variables they have no business existing on dev laptops or even the pipeline build steps (where package installation should happen).
AWS etc credentials / tokens are harder to secure since there are legit reasons for existing in dev laptops.
Docker is also not a silver bullet. Again, what you're claiming to be easy is often times exceedingly difficult or frictional, especially on established teams. I don't disagree that comparmentalization is important but security solutions are only as effective as their practical feasibility.
My code editor works in a sandbox. It's difficult because Linux doesn't provide it and one has to write it manually using shell scripts, random utilities. For example, I had also to write a limited FUSE emulation of /proc to allow code editor work without access to real /proc which contains lot of unnecessary information.
And if it's a "professional" setting, the company could hire a part-time developer for writing the sandbox.
could you share with us those utilities? I've tried doing the same with AppArmor, but I ended up having endless warnings and weird bugs.
I am using bwrap - it allows blocking network, creating a new process namespace (so that sandboxed program cannot mess with host processes) and creating a new filesystem, where you can mount host directories. I only pass through minimal set of directories like /usr and /lib, and only selected files from /etc, as they can contain valuable information. I also pass through several device files like /dev/null, but not the whole /dev.
I do not pass through /proc, as it contains not only information about processes (which is not a concern), but a lot of identifying information about a machine. However, lot of modern software due to poor coding and not following standards crashes or exhibits various bugs without /proc, so I wrote a minimal Python /proc emulator using FUSE: https://pastebin.com/s49V5nVL . /proc doesn't allow bind mounting its subdirectories somewhere else, so the only option was FUSE (or writing a kernel module). As you see, with Linux you learn something new every day, like it or not. The emulator is not perfect - it doesn't expose thread information, and, for example, Grand Central Dispatch library [1] terminates the application because of this. Telegram is using the library and crashes randomly because of this. I expected better quality from Apple and Swift.
Most importantly, I run the app as a different user (Android-style), because historically Linux was built around idea of protecting users from each other, and I don't trust namespaces as much. I clear the environment variables as well. Ideally, I should also be closing file descriptors to prevent leaks (for example, you can easily leak a file descriptor to your Wayland server, Dbus or something else).
To use Xorg, I grant access to the app's user. X server has no issues with sharing access to it. I am not sure if one can use Wayland with different users (is uses unix sockets). Pipewire doesn't allow its use by different users, so I have to add a pipewire-pulse plugin listening on localhost, and configure ALSA inside sandbox to connect to that pulseaudio server. Obviously, the app in sandbox has no direct access to audio devices in /dev.
Dbus is also necessary to display tray icons, but you should never grant direct access to it because it is almost like giving the root password. I use xdg-dbus-proxy [2] which I had to patch because originally it only supported running the sandboxed app under the same user. You can still find my patch in open pull requests.
I also did some experimentation with passing through a camera through a fake video4linux device, so that the app cannot know the real camera model, but it is not working great, for example, Gnome Cheese app cannot see it, because it needs not only a kernel device, but something more to detect cameras.
The sandbox cannot use hardware acceleration. It would be not easy to add because every GPU vendor uses its own directories and device files, /sys entries to interact with the hardware and I think you can get the full list only by reverse engineering. Furthermore, I am not sure if it is safe.
This setup is not perfect, and looks like a bunch of random utilities glued with shell scripts (which is not uncommon in Linux world). If I had more time, I would write something better, that can dynamically add and revoke privileges, can provide fake serial numbers and even emulate a working network which sadly loses 100% of sent packets.
I looked into using Flatpak, but they seem to grant full access to /sys, /proc and generally I don't feel that they care much about protection from malware and fingerprinting by legitimate software. Also, as I understand, they run the app as the same user which is not secure. Probably their primary goal is making installation easier but not making the use of software safer. I am not going to use Docker because it officially says that it is not a security tool.
[1] https://github.com/swiftlang/swift-corelibs-libdispatch
[2] https://github.com/flatpak/xdg-dbus-proxy
Good luck selling that to thousands of managers. That's my point. It's easy to list things that should be done. It's harder to get them greenlit.
Sure it is: don't do it. It's not like there isn't automated tooling for this, but you can also like... I don't know, look at your diffs and not commit secrets? I've never committed a secret before, and I've been working with AWS for ten years now. But I don't "git commit -a" and I triple-check my diffs.
Shai Hulud isn't about accidentally committing secrets, nobody is suggesting that's what's happening. Not sure which comment thread you're reading.
Perhaps it's time to organize a curated "stable" stream for npm packages.
If I want more stability for my OS I can choose Debian-stable rather than Ubuntu-nightly.
But for npm, there doesn't seem to be the same choice available. Either I sign up to the fire-hose or I don't.
I can choose to only upgrade once a month, but there's a chance I'm still getting a package that dropped 5 minutes before.
Upgrading once a month is insane at any rate, I could see the point in upgrading maybe once a year. For stable projects, you're very much fine upgrading only when there's a vulnerability or you need something from a newer release. Upgrade when you actually need to and use stable versions that have been out for a while, no need to hamster wheel it.
When I worked in commercial aerospace, before we even shipped live there was an incident with a CERT advisory against the XML package we were using. But the fix was only added to the current major version and we were stuck one behind. It took ~3 of our best problem solvers about a week to get that damned thing upgraded. Which put us behind on our schedule.
This made some of my more forward thinking coworkers nervous because what if this happened after we went live? So we started a repeating story called “upgrade dependencies” and assigned it round robin once a month to someone on each application. Every time someone got it the first time they would ask me, “but upgrade what?” Whatever you want, but preferable something that hasn’t been in a while.
For IP and security reasons we were already on vendored dependencies, so it was pretty straightforward to tell what was old. But that made “upgrade immediately” problematic if fixes weren’t back ported far enough and we didn’t want that live.
pnpm
Whats the most full proof way of defending ourselves from such attacks? My opinion is that the applications should never deal with credentials at all. Sidecars can be run which can inject credentials in real time. These sidecars can be under tight surveillance against such attacks. After all, application code is the most volatile in an organization.
I think we shouldn't be using package repositories in this way at all, shouldn't it be much better to have a package system like golang has where you directly import the sourcecode from github? You get around an entire class of problems. At least now you can only be compromised if the github source code itself is compromised, not any part of some build pipeline or a tool like npm or an npm registry. That means to vendor everything and only upgrade if you need to upgrade, treat all the code like you are responsible for it all because you are. The entire concept of relying on builds of other people is part of the problem, it's bad enough that we rely on source code of other people but that goes with the territory. Relying on their build systems is not as mandatory.
To not use npm. Or create a package manager like npm. Or believe in philosophy that we should have as many small dependencies as possible.
If you must use npm, containerize/VM it? treat it as if you're observing malware.
pnpm’s minimumReleaseAge can help a ton with this. There’s a tricky balance, because allowing your dependencies to get stale makes you inherently more vulnerable to vulnerabilities in your packages. And, critically, fixing a vulnerability in an urgent situation (i.e. you were compromised) gets increasingly harder to address the more stale your dependencies are.
minimumReleaseAge strikes a good balance between protecting yourself against emerging threats like Shai-Hulud and keeping your dependencies up-to-date.
Because you asked: you can get another layer of protection through Socket Firewall Free (sfw), which prevents dependencies known to be malicious from being installed. Socket typically identifies malware very soon after its is published. Disclaimer: I’m the lead dev on the project, so obviously biased — YMMV.
Why can't package managers enforce attestations backed by a transparent log for each commit made to a public repository?
They can, but what does it solve? If a malicious package gets pushed, who or what is the equivalent of the CA that you are you going to nuke?
I was working with the assumption in this model the attestation is signed by ephemeral keys (OIDC) which would reveal the bad actor or give breadcrumbs. Enough to reduce incentives to hijack packages.
They can but that wasn't done in this case and isn't commonly done for various reasons.
GitHub back in September already published their roadmap of mitigations to NPM supply chain attacks:
https://github.blog/security/supply-chain-security/our-plan-...
I'm guessing no one yet wants to spend the money it takes for centralized, trusted testing where the test harnesses employ sandboxing and default-deny installs, Deterministic Simulated Testing (DST), or other techniques. And the sheer scale of NPM package modifications per week makes human in the loop-based defense daunting, to the point that only a small "gold standard" subset of packages that has a more reasonable volume of changes might be the only palatable alternative.
What are the thoughts of those deep inside the intersection of NPM and cybersecurity?
You would need to hear the thoughts of those deep inside the intersection of money and money.
Very concerning, so that was what the "impending disaster" was as I first noted. [0] Quite worrying that this happened again to the NPM ecosystem.
Really looking forward to a deeper post-mortem on this.
[0] https://news.ycombinator.com/item?id=46031864
It will keep happening until someone takes responsibility and starts maintaining the whole of the node eco system. This is probably a viable start-up idea: Node but audited.
Maybe we can convince Shopify to hijack NPM too while they're at it.
You don't even need to enshittify Yet Another Service, you just need package maintainers. Debian manages to do this, and I'm guessing they get paid nothing (although, yeah, Amazon and The Goog really ought to chip in a few bucks, considering their respective empires). Unfortunately, it means you can't just YOLO your code into other people's programs anymore.
> Unfortunately, it means you can't just YOLO your code into other people's programs anymore.
That's a good thing, in my book.
Oh, agreed 100%. I find it endlessly frustrating that these same conversations happen every single time there's a supply chain attack like this, because nobody wants an _actual_ solution, they want an _easy_ solution that doesn't involve changing anything about how they work. So we just get 500 comments asking if we can solve the Halting Problem, and then everyone forgets until the next breach. It was ever thus.
I guess you should never use the latest versions of libraries.
Everyone needs to switch to pnpm and enable https://pnpm.io/settings#minimumreleaseage
Pnpm also blocks preinstall scripts by default.
Nah - dependency cooldown is all the rage but it’s only effective if you have some noncompliant canary users. Once everyone is using it it will cease to be effective because nobody will be taking the first step/risk until everybody does.
The point of the cooldown is to allow time for vendor scans to complete and for compromised packages to be pulled. It's not about waiting for an end user to notice they've been compromised.
> Meanwhile, the aforementioned vendors are scanning public indices as well as customer repositories for signs of compromise, and provide alerts upstream (e.g. to PyPI).
https://blog.yossarian.net/2025/11/21/We-should-all-be-using...
Depending on “security vendors” to do scans of every single update seems naive and over optimistic to me, but hey - everyone’s jumping on the bandwagon regardless of what I think so I guess we’ll see soon.
Don't "security venders" detect and report most of these types of attacks already today?
Do they? :)
What's the alternative?
Or bun
But you also need the latest versions to avoid zero-day attacks.
99% of releases do NOT fix zero-days. But 100% of releases have a small risk of introducing a backdoored build-script.
There's nothing wrong with pinning dependencies and only updating when you know for sure they're fixing a zero-day (as it will be public at that point).
Or an old enough version. For one of the most damaging zero-day vulnerabilities in the Java ecosystem (log4shell), you were vulnerable if you were in the latest version, but not vulnerable if you were using an old enough version.
Zero-day on frontend has not really a y effect, except on one user at a time. Zero-day on a server though ... perhaps we arrive at the conclusion to not use the JS ecosystem on the server side.
do zero-days even care about versions?
They care how long you take to patch your versions. Delaying patching by 7 days will affect it.
Not sure if you're serious, but if so I agree that people should take the time to set up their own package mirrors. Not just for npm but all other package managers as well.
This is why it's so important to get to know what you're actually building instead of just "vibing" all the time. Before all the AI slop of this decade we just called it being responsible.
Exactly, there is no easy solution to these problems.
The solutions that are effective also involve actually doing work, as developers, library authors, and package managers. But no, we want as much "convenience" as possible, so the issues will continue.
Developers and package authors should use a lockfile, pin their dependencies, be frugal about adding dependencies, and put any dependencies they do add through a basic inspection at least, checking what dependencies they also use, their code and tests quality, etc.
Package managers should enforce namespacing for ALL packages, should improve their publishing security, and should probably have an opt-in verified program for the most important packages.
Doing these will go a long way to ameliorate these supply chain attacks
How does having a mirror help?
Maintaining a package mirror is a shared responsibility beyond just the software dev teams. The packages and their publishers need to be approved to be added to the mirror, testing needs to occur, and updates are delayed until the newer version is added to the mirror. The network team would block npm and force all machines to use this mirror.
All this would have mitigated this incident in the event that an npm install was done during the window of this update being rolled out and unpatched. The npm install would continue as normal on the last known good version and the newer vulnerable version would simply not exist on the mirror.
I see a bunch of postman packages vulnerable. Does that mean the desktop application is compromised (oof)?
Postman posted a blog entry about the event: https://blog.postman.com/engineering/shai-hulud-2-0-npm-supp...
"Our security engineering team is investigating the matter and thus far has concluded that while some public Postman NPM packages were infected, (1) Postman as an app is not compromised, and (2) our production cloud services are also not compromised."
If the JS ecosystem continues like this, we're Duned.
I’ve been tracking these supply-chain incidents as well, so I built a small real-time scanner that looks for suspicious SHA-1 patterns inside repositories.
It’s basically a lightweight CLI tool you can run directly inside any local project:
Repo is here: https://github.com/developerjhp/sha1-hulud-scannerIt’s not meant to be a full security product — just a simple “first-pass” detector that helps catch unexpected checksum strings or injected artifacts before they slip into CI. Feedback and contributions are welcome!
Made a package (that I needed personally), to easily reinstall all dependencies (using the same versions) in a project and check them using Aikido's safe chain for malware (supported npm, pnpm, bun, and yarn). It also easily switches a project's package manager to another. https://www.npmjs.com/package/eazypm
Documenting technical details and payload analysis here: https://safedep.io/shai-hulud-second-coming-supply-chain-att...
Like previous variant, it has credential harvesting, self-replication and GitHub public repository based exfiltration.
Double base64 encoded credentials being exposed using GitHub repositories: https://github.com/search?q=%22Sha1-Hulud%3A%20The%20Second%...
containerize all the things...Nix, Podman, Docker. It's not a big hassle once you get through the initial steps.
Would be good to see projects (like those recently effected) nudging devs to do this via install instructions.
Yep. This is what I do. I edit and run my code in a container. That container cannot access my ssh keys or publish to GitHub. I review all changes, and manually commit / publish on my host. It’s not perfect, but that plus vendoring my dependencies goes a long way towards mitigating these kinds of things.
Ok, I think the verdict on the "JavaScript for everything" experiment is in. It was already resolved long ago (in my opinion), but this should convince any stragglers. Let's accept that the one thing JS is really great for is DOM patching, and move on.
Going forward, use WASM if you really want to make an SPA (and think about that choice), where the source language is not something that ties into the JS dependency ecosystem. Ban it and burn it with fire for anything on the backend, for christ.
How do you test your projects if there are any infected/affected dependencies used? As i understand it could also be a dependency of a dependency ... that could be affected?
What is going on with this website though? It gives cursor stutter and slow scrolling. It seems like we now need an insane amount of CPU to read static text. What a regression.
as someone that likes the npm package ecosystem but is not fond of malware - ended up building a CLI wrapper around npm to only install packages older than a configurable amount of days.
in case folks find it helpful: https://github.com/kevinslin/safe-npm
I am amazed at the dates in this article. This compromise appears to have been discovered literally this morning. Incredibly fast turnaround on this article.
Whats the most full proof way of defending ourselves from such attacks? My opinion is that the applications should never deal with credentials at all. Sidecars can be run which can inject credentials in real time. These sidecars can be under tight surveillance against such attacks. After all, application code is the most volatile in an organization.
To me this is asking the question of "what's the safest way to drink from a polluted river".
The answer is really, don't.
NPM and the JS eco-system has really gone down a path of zero security and they're paying the price for it.
If you really need libraries from NPM and whatnot, vendorize them so you're relying on known-safe files and don't arbitrarily update them without re-verification.
This is true. Today its npm, tomorrow it could be some other language. Shouldnt we focus on solving it at the root?
Some of us need to drink from the river to eat :(
This is why I am not a huge fan of separate package managers for libraries, such as in the case of rust, or node. The C style of sharing deps. couldn't really be simpler as just including the headers in your Makefile.
We really don't need more package managers other than the ones provided by your operating system, but I dunno maybe its just me.
That ship has sailed, traveled around the world, and docked in a foreign port at this point.
Including headers isn't remotely "simple". There's so many considerations in linking, .SO version compatibility, architecture and instruction set issues, building against multiple versions on the same system. Or if you want to feel frustrated in a single word: GDAL (IYKYK)
And that's only where #include is even applicable. That is not gonna fly for any interpreted language - JS in this case, but also python, ruby, php.
> Including headers isn't remotely "simple". There's so many considerations in linking, .SO version compatibility, architecture and instruction set issues, building against multiple versions on the same system. Or if you want to feel frustrated in a single word: GDAL (IYKYK)
I'd say for issues with instruction sets is a minor edge case, you could argue that having layers of abstraction gives even more room for fault.
That said I am not familiar with GDAL, but from google-fu it seems relatively heavy. Being said, I don't have much familiarity with compiling CXX programs as I do with programming C programs, and C with it's smaller footprint tends to not give me as many problems.
Again, this is from personal experience.
My guess is “go download a library” and it brought back memories of using Linux on dialup circa 2001
[dead]
The JS ecosystem in particular, it really seems like it was built by people hell-bent on reinventing the wheel and making all the mistakes / paying all the costs along the way. It's a pretty octagonal wheel so far, but maybe they'll get there eventually.
Ecosystems aren’t built by any homogeneous group of people. They’re a sum of their parts. It’s not like there was a committee and that committee decided how things should work wrt wheel reinvention. People publish packages, and the result is something we call an ecosystem.
Both of these attacks have used trufflehog. Is there an out of the box way to block that executable by name or signature?
I'd say an alternative question is "how can we stop storing secrets in source control" so then tools like Trufflehog can't find them :)
Seems to me the root problem here is poor security posture from the package maintainers. We need to start including information about publisher chain of custody into package meta data, that way we can recursively audit packages that don't have a secure deployment process.
Will the list of affected packages expand? How were these specific packages compromised in the first place?
More:
SHA1-Hulud the Second Comming – Postman, Zapier, PostHog All Compromised via NPM
https://www.aikido.dev/blog/shai-hulud-strikes-again-hitting...
Is there a terminal AI assistant that doesn't have heaps of depenedancies and preferably no node? Claude and codex both require node. I'm a fan of the lightweight octofriend. But also node. I do not like installing node on systems that otherwise would not require it.
You can install codex without npm if you build it yourself, they have migrated to rust in June and npm is just a convenient install wrapper it seems.
Just `git clone git@github.com:openai/codex.git`, `cd codex-rs`, `cargo build --release` (If you have many cores and not much RAM, use `-j n`, where n is 1 to 4 to decrease RAM requirements)
llama.cpp?
Does it have a terminal assistant that I have not heard of? Otherwise, the parent asks about an assistant that is able to run various tools and stuff, not just talk.
why don't web devs just learn html and css properly, and maybe xslt for the really complex transformations then use vanilla js only when it's truly necessary?
instead we've got this absolute mess of bloated, over-engineered junk code and ridiculously complicated module systems.
the issue is not that devs don't know what they are its that they don't pin packages
if you run `npm i ramda` it will set this to "ramda": "^0.32.0" (as of comment)
that ^ means install any version that is a feature or patch.
so when a package is released with malware they bump version 0.32.1 and everyone just installs it on next npm i.
pinning your deps "ramda": "0.32.0" completely removes the risk assuming the version you listed is not infected.
the trade off is you don't get new features/patches without manually changing the version bump.
For context: ramada 0.32.0 isn't a concrete thing, in the sense that glibc 2.35 is. It really means "the latest ramada code because if you were to pin on this version it'll at some point stop working". glibc 2.35 never stops working.
> the trade off
I see that as a desirable feature. I don’t want new functionality suddenly popping into my codebase without one of my team intending it.
me too but a lot of people see it as massive overhead they don't want to deal with.
personally i pin all mine because if you don't a version could be deployed during a pipeline and this makes your local version not the same as the one in docker etc.
pinning versions is the only way to be sure that the version I am running is the same as everyone elses
Because then how would they pay their inflated Bay Area rent?
Good luck with the XSLT going forward what with Google trying to remove it from the internet.
- There is a single root dependency somewhere which gets overtaken
- A new version of this dependency is published
- A CI somewhere of another NPM package uses this new version dependency in a build, which trigger propagation by creating a new modified version of this dependency?
- And so on...
Am I getting this right?
I think so. It’s that third step that I can’t figure out. Build systems are configured to pull the latest version of a dep automatically, without review, and then publish. It seems the poorly configured pipelines are what enable these attacks. Fix your pipelines
I use the following to at least sandbox things in containers with an easy to use develop experience.
https://github.com/jrz/container-shell
0.2.4 had an issue btw
Here's the underlying problem: let's imagine someone very smart. They figure out a way to solve this problem. They are not going to make any money by doing so. That's why we have this problem.
You're right, that is the underlying problem. Not only of this, but of our entire economy.
Maybe, we have to rethink depencies from the ground up.
Implementing everything yourself probably won't cut it.
Copying a dependency into your code base and maintaining it yourself probably won't yield much better results.
However, if a dependency would be part of the version control, depends could at least do a code review before installing an update.
That wouldn't help with new dependencies, that come in with issues right from.the start, but it could help preventing new malware from slipping in later.
A setup like that could benefit from a crowd-sourced review process, similar to Wikipedia.
I think, Nimble, the package manager of Nim, uses a decentralised registry approach based on Git repos. Something like that could be a good start.
I find it unbelievable that npm still doesn't upgrade integrity entries to SHA512 across the board. This seems like such a simple hole to plug. What gives?
And this, kids, is why you should vendor your dependencies
Vendoring wouldn't really affect this at all. If anything it would keep you vulnerable for longer because your vendored copy keeps "working" after the bad package got removed upstream. There's a tiny chance that somebody would've caught the 10MB file added in review but that's already too late - the exploit happened on download, before the vendored copy got sent for review.
But you would have code reviewed it
I use pnpm, but even so: thankfully naming things is hard, and all my env variable names are very_convuluted_non_standard_names for things lol.
As a marketer who relies on tools like Zapier and PostHog and n8n, etc - terrifing. Since it means the dev/security teams may take our toys away :(
Is this a problem with Python package management too? If not, what distinguishes Python from NPM or makes it more resistant etc?
Yes. But Python is less popular so it isn't targeted as often.
"This appears to have limited the imapct of the attack at this time."
typo after the listed affected packages
My motto wrt language choices: "It's the standard lib, stupid!"
My ultra hot take: there are only¹ two² programming ecosystems suitable for serious³ work:
The reason why is because they have a vast and vetted std lib. A good standard lib is a bigger boost then any other syntactic niceties.Arguably both Go and Python also have great stdlibs. The only advantage that JVM and .NET have is a default GUI package. Which is fair, but keeps getting less and less relevant as people rely more on web UIs.
.Net has Blazor for WebUI.
I don't ask you to judge if you like it, I'm just saying that you can totally make a professional WebUI within the dotnet stdlib.
Respectfully disagree. Python and Go std lib do not even play in the same league. I had to help someone with datetime¹ handling in Python a while back. The stdlib is so poor, you have to reach out for a thirdparty lib for even the most basic of tasks².
Don't take my word for it, take a dive. You wouldn't be the first to have adjust their view.
For example, this section is just about the built-in web framework asp.net: https://learn.microsoft.com/en-us/aspnet/core
______
1. This might be a poor example as .net has NodaTime and the jvm has YodaTime as 3rd-party libs, for if one has really strict needs. Still, the builtin DateTime constructs offer way more than what Python had to offer.
2. Don't get me started on the ORM side of things. I know, you don't have to use one, but if you do, it better does a great job. And I wouldn't bat an eye if the ORM is not in the standard, but boy was I disappointed in Python's ecosystem. EF Core come batteries included and is so much better, it isn't fun anymore.
You helped someone with Python, and what evidence do you have justifying your claims about alleged Go stdlib narrowness?
Online documentation: https://pkg.go.dev/std
Let me know if I look at the wrong place.
Why do the attackers publicly share the keys they found? Like what’s their mission?
My guess would be so they don't have to embed an IP address or hostname in the malware to send secrets to, which could then be blocked or taken down.
But they could encrypt it, its just double b64 encoded, everybody can read it.
That one stumped me. Why not just encrypt with a hardcoded public key, then only the attacker can get the creds.
The simple B64 encoding didn't hide these creds from anyone, so every vendor out there's security team can collect them (e.g. thinking big clouds, GitHub, etc) and disable them.
If you did a simple encryption pass, no one but you would know what was stolen, or could abuse/sell it. My best guess is that calling node encryption libs might trigger code scanners, or EDRs, or maybe they just didn't care.
Or they just wanted to prove a point.
They surely seemed to be smart enough to choose encryption over encoding.
Hard to believe encryption would be the one thing that would trigger code scanners.
Also it’s not just every vendor, also every bad actor could’ve scraped the keys. I wonder if they’ve set up the infrastructure to handle all these thousands of keys…
Like what do you even do with most of it on scale?
Can you turn Cloud, AWS , AI api keys to money on a black market?
I’m looking for info on whether MalwareBytes would catch this and not finding anything.
So people using only distro provided dependencies were indeed right all along. ;-)
If you always run npm inside of docker does that pretty much prevent attacks like this?
Docker is not a sandbox. There is some work that can be done to harden it, but you're better off looking at genuinely sandboxing your dev environment
What is genuine sandboxing? Everyone waives there hands by saying this
Good question with a lot of possible answers. You can take sandboxing as far as you want, really. I typically just use bubblewrap (linux)
I have a perfect set up in inside docker that works.
I would love to know why bubblewrap is a superior alternative.
Here's mine https://github.com/ashishb/dotfiles/blob/067de6f90c72f0cf849...
My understanding is that docker escapes are not all that difficult, and your aliases really aren’t doing much to harden the container. but I am not an expert on the matter. I’m sure there is plenty of info online
> My understanding is that docker escapes are not all that difficult,
Funny coincidence reading this while in the middle of rewatching Dune 2 on Netflix
My guy what are you doing on HN. Put down the phone and watch the movie.
Second screen experience
Would the adoption of a Deno-like security posture in NPM have mitigated this?
pnpm is the better comparison maybe in this context. Most of Deno's approach to security is focussed on whole program policies which doesn't do much in this context. Just like pnpm and others, they do have opt-in for install scripts though. The npm CLI is an outlier there by now.
the left-pad fiasco seems to have been the only time npm changed a policy and reacted to a security problem, since then it seems that supply chain attacks just belong to the npm-eco-system
How was the attack detected in the first place?
I have never tried NPM. Somehow I can still find the software I want, or write it myself. I think I would find Javascript too slow
> bun_environment.js is a highly obfuscated malicious JavaScript file. It is over 10MB in size and contains a significant amount of built-in logic for information theft.
That seems a bit silly. Even on the beefy boi I used to work on a 10MB hiccup in deployable size would have been sufficient to make me look.
I released one of the packages I work on last night so of course this drew my eye. I assume checking the unpacked size hasn’t gotten ridiculous confirms that your code is not infected yeah? And looks like it’s past time for me to set up a separate account for release management.
This is still happening -- github should implement a hotfix to disallow creation of repos with this exact name and structure...
Thought this was about the band ...
Friends don’t let friends NPM.
Once again, you cannot ask the open source world to provide you with free dependencies and security.
At some point, someone has to pay for an organisation whose job will be to review the contents of all of these modules.
Maybe one could split the ecosystem into "validated" and "non validated" stacks ? much like we have stable and dev branches ?
The people validating would of course give their own identity to build trust. And so, companies (moral person) should do that.
See also: https://news.ycombinator.com/item?id=46005111
As it arguably would have reduced impact
(I'm one of the Renovate maintainers and have recently pushed for this to be more of a widely used feature)
I think everyone just gets hit after 7 days frankly.
Why? Everyone won't use cooldowns, but the key is to have just enough people running brand new to set off a warning/have systems that check dependencies scan and find vulns go off and the packages get pulled before production builds them.
Monocultures where everyone pulls and builds with every brand new thing for the most minor changes is dangerous.
The list of affected packages is concerning - indeed.
these packages stood out for me
shinhan is a large korean bank and this admin area geo json util seems to be embedded in many korean gov services.
shinhan-limit-scrap
korea-administrative-area-geo-json-util
[dead]
[dead]
[dead]
[dead]
I wish everyone here would read https://en.wikipedia.org/wiki/Capability-based_security and then realize that maybe, JUST MAYBE, THE PROGRAMMING LANGUAGES WE USE SHOULD NOT ALLOW IMPORTED PACKAGES TO ACCESS EVERYTHING, AND THEIR LACK OF SECURITY GUARANTEES AND ACCESS RESTRICTION MECHANISMS MAKES THEM DANGEROUS!
The number and range of affected devices may be reduced with any number of package manager level workarounds, but NOT the impact of attacks once any succeeds. For this, you NEED the above.
What we need is an open source industry standard stdlib equiv developed and maintained by MS, GOOG and some open source players.
All libraries should strive to have dependency only on it.
There are actually hundreds more NPM packages infected, see here: https://www.koi.ai/incident/live-updates-sha1-hulud-the-seco...
You don't provide any more information, and are promoting your own site here without even saying so despite your name being on the About page. This felt like clickbait.