While I hate defending GHA, the docs do include this:
- Using the commit SHA of a released action version is the safest for stability and security.
- If the action publishes major version tags, you should expect to receive critical fixes and security patches while still retaining compatibility. Note that this behavior is at the discretion of the action's author.
So you can basically implement your own lock file, although it doesn't work for transitive deps unless those are specified by SHA as well, which is out of your control. And there is an inherent trade-off in terms of having to keep abreast if critical security fixes and updating your hashes, which might count as a charitable explanation for why using hashes is less prevalent.
On the other hand, this issue has been known to GitHub since shortly after Actions’ release[0]. They added some cya verbiage to their docs, but they never followed up by making version pinning meaningful.
Sure you can implement it yourself for direct dependencies and decide to only use direct dependencies that also use commit sha pinning, but most users don’t even realize it’s a problem to begin with. The users who know often don’t bother to use shas anyway.
Or GitHub could spend a little engineer time on a feasible lock file solution.
I say this as somebody who actually likes GitHub Actions and maintains a couple of somewhat well-used actions in my free time. I use sha pinning in my composite actions and encourage users to do the same when using them, but when I look at public repos using my actions it’s probably 90% using @v1, 9% @v1.2 and 1% using commit shas.
[0] Actions was the first Microsoft-led project at GitHub — from before the acquisition was even announced. It was a sign of things to come that something as basic as this was either not understood or swept under the rug to hit a deadline.
There's a repository setting you can enable to prevent actions from running unless they have their version pinned to a SHA digest. This setting applies transitively, so while you can't force your dependencies to use SHA pinning for their dependencies, you can block any workflow from running if it doesn't.
Using an SHA is an anti-pattern for me. Because by using one, you kind of modeled "I am getting this fixed/static thing"; when in reality, it is very far from that. I got bitten by it twice that I learned that you either have a lock file or you don't.
- Using the commit SHA of a released action version is the safest for stability and security.
This is not true for stability in practice: the action often depends on a specific Node version (which may not be supported by the runner at some point) and/or a versioned API that becomes unsupported. I've had better luck with @main.
Depends what you mean by stability. The post is complaining about the lack of lockfiles, and the problem you describe would also be an issue with lockfiles.
What’s more, GitHub has basically stopped maintaining their own actions, pushing people to sketchy forks to do basic things. Their entire ecosystem is basically held up with duct tape and gets very little investment.
They had working infra and a great case for keeping fairly "close to the metal". Complicated files-heavy workload that needs tons of clever caching to perform well, lots of writes, lots of non-HTTP TCP traffic.
Retrofitting that into "cloud" bullshit is such a bad idea.
Using bare-metal requires competent Unix admins, and Actions team is full of javascript clowns (see: decision to use dashes in environment variable; lack of any sort of shell quoting support in templates; keeping logs next to binaries in self-hosted runners). Perhaps they would be better off using infra someone else maintains.
They barely maintain Azure pipeline tasks / actions as well.
We had a critical outage because they deprecated Windows 2019 agents a month earlier than scheduled. MS support had the gall to both blame us for not migrating sooner, and refuse to escalate for 36 hours!
An interesting things is that GitHub is an expensive service and my guess would be that MS makes good money on it. Our small company paid about 200+ USD monthly for GitHub, much larger cumulative cost than Windows licenses. My believe was that Windows is getting worse, because it is considered legacy business by MS in favor of new offerings such as GitHub subscriptions.
GitHub also runs a free tier with significant usage.
There are ~1.4b paid instances of Windows 10/11 desktop; and ~150m Monthly active accounts on GitHub, of which only a fraction are paid users.
Windows is generating something in the region of $30b/yr for MS, and GitHub is around $2b/yr.
MS have called out that Copilot is responsible for 40% of revenue growth in GitHub.
Windows isn't what developers buy, but it is what end users buy. There are a lot more end users than developers. Developers are also famously stingy. However, in both products the margin is in the new tech.
github value maybe as not apparent as other product
but github is pair well with MS other core product like Azure and VS/VSC department
MS has a good chance to have vertical integration on how software get written from scratch to production, if they can somehow bundle everything to all in one membership like Google one subs, I think they have a good chance
hey there! blacksmith solutions engineer here :) love to hear we've helped speed up your tests and docker builds!!
could you shoot me your GH org so I can apply your startup discount? feel free to reach out to support@blacksmith.sh and I'll get back to you asap.
thanks for using blacksmith!
Thank you for the kind shout out! Always happy to see comments like this. If anyone is looking for a better GitHub or GitHub Actions experience, feel free to reach out anytime.
Founder of Depot here. We provide faster and more reliable GitHub Actions runners (as well as other build performance services) at half the cost of GitHub [0]
Is there a write up on the security of actions or equivalent that explains how they are secure both with direct and transitive dependencies? If this applies to Depot.
The legacy business usually explains why there are no new features, only minor maintenance, it doesn't explain why there is a lot of investment into work that makes it worse
It's not really that expensive. GitHub Enterprise is like $21/month/user while GitLab Ultimate was $100/month/user the last time GitLab published prices. These days GitLab Ultimate is "contact us for pricing" while the cheaper GitLab Premium is $29/month/user.
I guess Bitbucket is cheaper but you'll lose the savings in your employees bitching about Bitbucket to each other on Slack.
Do we work in the same company? That said, I really don't understand why everyone hates on Bitbucket. I really thought it was _fine_ from a user perspective. Now we're on GHE and I find it a sidegrade at best.
Now for the people who were operating Bitbucket, I'm sure it's a relief.
I believe the original GitHub Actions was in Go - it used HCL which was at that point only really implemented in Go. Quite the move backwards to switch to YAML.
There's direct evidence that GitHub Actions was the rewrite of Azure Pipelines that was originally planned to finish 5 years ago and got "stuck" (because all their resources moved to GitHub). For a while you could find 2020 roadmap repositories (on GitHub) for AzDO talking up a Pipelines rewrite bringing a lot more features (including better Docker alignment versus Pipelines' much more complex "runner skills") that instead showed up in the first version of GitHub Actions.
Microsoft claims Azure DevOps still has a roadmap, but it's hard to imagine that the real roadmap isn't simply "Wait for more VPs in North Carolina to retire before finally killing the brand".
github doesn't pay microsoft for the azure runners. that's why they came up with actions at all. microsoft gets streetcreds for stable runners, github could replace travis and appveyor.
The quality of setup-* actions has definitely gone down and there are a lot of strange decisions being made. I assume the original authors of these actions have long left the company.
Thank you for your interest in this GitHub repo, however, right now we are not taking contributions.
We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time. The GitHub public roadmap is the best place to follow along for any updates on features we’re working on and what stage they’re in.
What they are really saying is they don't want third party contributions. They don't have anyone triaging Issues or PRs so don't send them.
They will occasionally make changes if it aligns with a new product effort driven from within the org.
Saying they're dropping support is a stretch esp as very few people actually pay for their Support package anyway..... (Yes they do offer it as a paid option to Enterprise customers)
> Instead of writing bespoke scripts that operate over GitHub using the GitHub API, you describe the desired behavior in plain language. This is converted into an executable GitHub Actions workflow that runs on GitHub using an agentic "engine" such as Claude Code or Open AI Codex. It's a GitHub Action, but the "source code" is natural language in a markdown file.
This seems like a real headache to me. I understand the value proposition of LLMs in the development cycle, but CI/CD is probably the last place where I want any degree of nondeterminism.
This looks like backwards. I would understand using a LLM to generate a GitHub Actions YAML, but always running your action from a Markdown file seems extremely wasteful in terms of resources.
Edit: ok, looking at example it makes more sense. The idea is to run specific actions that are probably not well automated, like generating and keeping documentation up-to-date. I hope people don't use it to automate things like CI runs though.
Because they know most abusive business relationship partners don't leave (see also Oracle). No matter how many bruises, CIO's are not going to get fired for buying Big Blue or whoever is the current abusive standard.
The funny thing about the last one is that those actions ultimately boil down to invoking their CLI tool (which is pre-installed on the runners) with "gh release create ...", so you can just do that yourself and ignore the third-party actions and the issues that come with them. Invoking an action isn't really any easier than invoking the CLI tool.
Yeah, what really needs to happen with that repo is to put that in the README to use the gh CLI instead of pointing to the third-party action with questionable security policies. If they were accepting PRs for that repo, it would be an easy PR to make.
That issue with their own small private forks has actually raised its head while testing out the AI slop generator thing it has, making anything it produces for you not self hoatable unless you rewrite a lot of basic functions. Sweet irony.
With AI you won't need CI anymore, it's all going straight to prod anyway /s
Actions is one thing, but after all these years where the new finegrained access tokens aren't still supported across all the product endpoints (and the wack granularity) is more telling about their lack of investment in maintenance.
I never used any actions and never understood why would I need to. I just wrote bash script to build my project and that's about it. This modern tendency to add dependencies for trivial things baffles me. You don't need "action" to do `git clone`.
I’d appreciate not being called lazy for mentioning a lack of investment on Microsoft’s side to secure their paid and fairly lucrative service that they bought a popular code hosting platform to integrate with.
Can someone explain what this somewhat recent phenomenon is where people feel the need to defend the worlds biggest billion dollar businesses, that are also often subsidized by tax payer money in weird ways?
How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
> How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
Honestly I think the problem is more a rosy view of the past versus any actual change in behavior. There have always been defenders of such companies.
> How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
They hired a ton of people on very very good salaries
The original comment said to stop giving money to these companies if they are not giving you a satisfactory service.
The opposite, to be lazy and to continue giving them money whilst being unhappy with what you get in return, would actually be more like defending the companies.
The original comment actually criticized Microsoft for a lack of investment to secure their paid and fairly lucrative service that they bought a popular code hosting platform to integrate with.
The opposite we see here: to not criticize them; to blame Microsoft's failure on the critics; and even to discourage any such criticism, are actually more like defending large companies.
I won't "defend" Microsoft in this case, but I am always annoyed by phrases like "world's biggest billion-dollar businesses... bablah".
Their size or past misbehaviors shouldn't be relevant to this discussion. Bringing those up feels a bit like an ad hominem. Whether criticism is valid should depend entirely on how GitHub Actions actually works and how it compares to similar services.
There is a massive problem in open source where some people equate pointing out a problem with being too lazy to solve it — when in reality this just stifles the conversation. Especially when a prerequisite to any group project accomplishing anything is to first discuss the problem to be solved.
No that's actually a completely different issue. You're talking about volunteers working on side projects that are sometimes foundational to the way the internet works and then people feel entitled to tell them what to do without contributing.
Here we are talking about one of the worlds most valuable companies that gets all sorts of perks, benefits and preferential treatment from various entities and governments on the globe and somehow we have to be grateful when they deliver garbage while milking the business they bought.
No, that's actually the same issue. "Entitled to tell them what to do without contributing" is not a problem. Let them tell whoever what to do, the response is always the same: "patches welcome," or if that isn't even true (which it doesn't have to be), "feel free to fork."
don't confuse 'receiving something you did not pay for' with 'being allowed to feel entitled to anything' is all. 'open source' is just that, nothing more. if you want a service with your source, be prepared to sponsor it.
I still think people should want things and be vocal about what they want. This is the natural way for people to know what needs to be built. It is different from demanding something.
And besides that, a lot of people on here do pay for Github in the first place.
Well, actually, no, not everyone is free to use alternatives. Anyone using CI for "Trusted Publishing" of packages to PyPI or npm needs to use GitHub Actions or GitLab CI/CD. CircleCI and Travis CI are not supported. So many big open source projects for the two most popular languages in the world are now locked out of the alternatives you propose.
(I find it extremely sketchy from a competition law perspective that Microsoft, as the owner of npm, has implemented a policy banning npm publishers from publishing via competitors to GitHub Actions - a product that Microsoft also owns. But they have; that is the reality right now, whether it's legal or not.)
I was never convinced that trusted publishing solves any security problem, other than letting pypi eventually solve the problem of banning russian/iranian/whatever people just by relying on github doing it for them.
Trusted Publishing on PyPI supports Google Cloud and ActiveState as well. It’s not tied to GitHub or GitLab. To my recollection I looked at CircleCI support a while back, and ran into limitations on the claims they exposed.
(It can also be extended to arbitrary third party IdPs, although the benefit of that is dependent on usage. But if you have another CI/CD provider that you’d like to integrate into PyPI, you should definitely flag it on the issue tracker.)
> unless they put the money where their mouth is, it's just noise
I used to work for a Japanese company, and one of their core philosophies was “Don’t complain, unless you have a solution.” In my experience, this did not always have optimal outcomes: https://littlegreenviper.com/problems-and-solutions/
I don’t make the purchasing decision for my employer, but I certainly have to deal with their fallout, so I’ll keep complaining if that’s okay with you.
I've used CircleCI quite a bit in the past; it was pretty good. Feels tough for them to compete with GHA though when you're getting GHA credits for free with your code hosting.
I used Travis rather longer ago, it was not great. Circle was a massive step forward. I don't know if they have improved it since but it only felt useful for very simplistic workflows, as soon as you needed anything complex (including any software that didn't come out of the box) you were in a really awkward place.
CircleCI made great steps the last few years, f.e. to better support proper DRY working, supporting OPA policies-as-code, VSCode extensions with "dry-run" options.
To be clear, I do think CircleCI is a better product than GHA. I just think there's a lot of air sucked out of the room by GHA being available 'for free' and out of the box.
Also, honestly, I don't care about any of those features. The main thing I want is a CI system that is fast and customisable and that I don't have to spend a lot of time debugging. I think CircleCI is pretty decent in that regard (the "rerun with SSH" thing is way better than anything else I've seen) but it doesn't seem to be getting any better over time (e.g. caching is still very primitive and coarse-grained).
I had a considerably better time with CircleCI in the past than with Github Actions currently. It feels much more like a complete product rather than a tacked on mess, I hate how disproportionately we count running cost just because we have numbers for it (vs. DX and velocity which are hard to measure and impossible to predict)
> Anyone can complain as much as they want, but unless they put the money where their mouth is, it's just noise from lazy people.
Once I'm encharged of budge decisions of my company I'll make sure that none will go to any MS and Atlassian product. Until then I'll keep complaining.
buildkite is leaps and bounds above the others. especially if you need to really tailor your workloads to the change diff (say in a monorepo), the dynamic pipeline support is superb.
really surprised there are no others though. dagger.io was in the space but the level of complexity is an order of magnitude higher
What that type of section usually means is "there's someone from Microsoft that signed up for our service using his work account", sometimes it means "there's some tiny team within Microsoft that uses our product", but it very rarely (if ever) means "the entire company is completely reliant on our product".
Normally I’d say stop kicking the dead horse, but GHA deserves all the complaints it gets and then some. It’s the embodiment of everything that’s bad in ‘less is more’.
My biggest concern with it is that it’s somehow the de facto industry standard. You could do so much better with relatively small investments, but MS went full IE6 with it… and now there’s a whole generation of young engineers who don’t know how short their end of the stick actually is since they never get to compare it to anything.
It's funny that absolutely everything about GHA fucking sucks, and everyone agrees about this. BUT, the fact that it's free compute, and it's "right there"... means it's very very difficult to say no to!
Personally I've just retired a laptop and I'm planning to turn it into a little home server. I think I'm gonna try spinning up Woodpecker on there, I'm curious to see what a CI system people don't hate is like to live with!
I can already tell by their example that I don't like it. I've worked with a bunch of different container-based CI systems and I'm getting a little tired seeing the same approach by done slightly differently.
steps:
- name: backend
image: golang
commands:
- go build
- go test
- name: frontend
image: node
commands:
- npm install
- npm run test
- npm run build
Yes, it's easy to read and understand and it's container based, so it's easy to extend. I could probably intuitively add on to this. I can't say the same for GitHub, so it has that going for it.
But the moment things start to get a little complex then that's when the waste starts happening. Eventually you're going to want to _do_ something with the artifacts being built, right? So what does that look like?
Immediately that's when problems start showing up...
- You'll probably need a separate workflow that defines the same thing, but again, only this time combining them into a Docker image or a package.
- I am only now realizing that woodpecker is a fork of Drone. This was a huuuge issue in Drone. We ended up using Starlark to generate our drone yaml because it lacked any kind of reusability and that was a big headche.
- If I were to only change a `frontend` file or a `backend` file, then I'm probably going to end up wasting time and compute rebuilding the same artifacts over and over.
- GitHub's free component honestly hurts itself here. I don't have to care about waste if it's mostly free anyways.
- Running locally using the local backend... looks like a huge chore. In Drone this was basically impossible.
I really wish someone would take a step back and really think about the problems being solved here and where the current tooling fails us. I don't see much effort being put into the things that really suck about github actions (at least for me): legibility, waste, and the feedback loop.
By adding one file to your git repo, you get cross-platform build & test of your software that can run on every PR. If your code is open source, it's free(ish) too.
It feels like a weekend project that a couple people threw together and then has been held together by hope and prayers with more focus on scaling it than making it well designed.
I will say that SourceSafe had one advantage: You could create "composite" proxy workspaces.
You could add one or two files from one workspace, and a few from another, etc. The resulting "avatar" workspace would act like they were all in the same workspace. It was cool.
Don't remember exactly. If I think about it, it could be quite complex.
Git has the concept of "atomic repos." Repos are a single unit, including all files, branches, tags, etc.
Older systems basically had a single repo, with "lenses" into sections of the repo (usually called "workspaces," or somesuch. VSS called them something else, but I can't remember).
I find the atomic repo thing awkward; especially wrt libraries. If I include a package, I get the whole kit & kaboodle; including test harnesses and whatnot. My libraries thend to have a lot more testing code than library code.
Also, I would love to create a "dependency repo," that aggregates the exported parts of the libraries that I'm including into my project, pinned at the required versions. I guess you could say package managers are that, but they are kind of a blunt instrument. Since I eat my own dog food, I'd like to be able to write changes into the dependency, and have them propagate back to their home repo, which I can sort of do now, if I make it a point to find the dependency checkout, make a change, then push that change, but it's awkward.
But that seems crazy complex (and dangerous), so I'm OK with the way things work now.
Your workflow is fascinating! What languages do you work in, if you don’t mind me asking?
Both git and jj have sparse checkouts these days, it sounds like you’d be into that
Do you vendor the libraries you use? Python packages typically don’t include the testing or docs in wheels uploaded to PyPI, for instance
These days in Pythonland, it’s typical to use a package manager with a lockfile that enforces build reproducibility and SHA signatures for package attestation. If you haven’t worked with tools like uv, you might like their concepts (or you might be immediately put off by their idea of hermetically isolated environments idk)
a repo is a repo - you're describing what is nowadays known as a 'monorepo' and it's a perfectly reasonable and desirable even exactly for the reasons you mention, except the 'distributed' part makes it very inconvenient to handle on dev boxes if it grows a lot.
in a centralized VCS there are viable CICD options like 'check the compiler binaries in' or even 'check the whole builder OS image in' which git is simply not able to handle by design and needs extensions to work around deficiencies. git winning the mindshare battle made these a bit forgotten, but they were industry standard a couple decades ago.
No, it let you continue to follow the main branch for most files, while files you edited would have their changes saved to a different location. And was just about as horrible as you might imagine.
We moved from VSS to SVN, and it took a little encouraging for the person who had set up our branching workflow using that VSS feature to be happy losing it if that freed us from VSS.
This is making me feel quietly vindicated in pushing back on migrating our Jenkins/Ansible setup to GHA simply because corporate wanted the new shiny thing. Fortunately the "this will be a lot of work, i.e. cost" argument won.
Mind you, CI does always involve a surprising amount of maintenance. Update churn is real. And Macs still are very much more fiddly to treat as "cattle" machines.
Pleased this is being discussed somewhere as it’s something that has troubled me for a while.
There are so many third party actions where the docs or example reference the master branch. A quick malicious push and they can presumably exfiltrate data from a ton of repositories
(Even an explicit tag is vulnerable because it can just be moved still, but master branch feels like not even trying)
> The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to secrets.
Why do CI/CD systems need access to secrets? I would argue need access to APIs and they need privileges to perform specific API calls. But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
This all seems right, but the reality is that people will put secrets into CI/CD, and so the platform should provide an at least passably secure mechanism for them.
(A key example being open source: people want to publish from CI, and they’re not going to set up additional infrastructure when the point of using third-party CI is to avoid that setup.)
Fine. Then let people set up a little WASM app that gets access to the secret and has an API callable by the workflow. And make that app be configured as part of the secret’s configuration, not as a file in the repository.
“Good CI systems shouldn’t support secrets, at most there should be [the most complicated secret support ever]”
Let’s just call it secret support.
I agree with your suggestion that capabilities-based APIs are better, but CI/CD needs to meet customers where they’re at currently, not where they should be. Most customers need secrets.
With a secure enclave or an HSM, there's a secret, but the users do not have access to the secret. So, if you have a workflow that needs to, say, sign with a given private key, you would get an API that signs for you. If you need to open a TLS connection with a client certificate, you get a proxy that authenticates for you.
I suppose I would make an exception for license keys. Those have minimal blast radii if they leak.
> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
CI/CD does not exist in the vacuum. If you had CI/CD entirely integrated with the rest of the infrastructure it might be possible to do say an app deploy without passing creds to user code (say have the platform APIs that it can call to do the deployment instead of typical "install the client, get the creds, run k8s/ssh/whatever else needed for deploy").
But that's a high level of integration that's very environment specific, and without all that many positives (so what you don't need creds, you still have permission to do a lot of mess if it gets hijacked), and a lot, lot more code to write vs "run a container and pass it some env vars" that had become a standard
You seem to be talking mostly about the CD part. Some thoughts:
On the one hand, CD workflows are less exposed than CI workflows. You only deploy code that has made it through your review and CI processes. In a non-continuous deployment model, you only deploy code when you decide to. You are not running your CD workflow on a third-party pull request.
On the other hand, the actual CD permission is a big deal. If you leak a credential that can deploy to your k8s cluster, you are very, very pwned. Possibly in a manner that is extremely complex to recover from.
I also admit that I find it rather surprising that so many workflows have a push model of deployment like this. My intuition for how to design a CD-style system would be:
1. A release is tagged in source control.
2. Something consumes that release tag and produces a production artifact. This might be some sort of runner that checks out the tagged release, builds it, and produces a ghcr image. Bonus points if that process is cleanly reproducible and more bonus points if there's also an attestation that the release artifact matches the specified tag and all the build environment inputs. (I think that GitHub Actions can do this, other than the bonus points, without any secrets.)
3. Something tells production to update to the new artifact. Ideally this would trigger some kind of staged deployment. Maybe it's continuous, maybe it needs manual triggering. I think that, in many production systems, this could be a message from the earlier stages that tells an agent with production privileges to download and update. It really shouldn't be that hard to make a little agent in k8s or whatever that listens to an API call from a system like GitHub Actions, authenticates it using OIDC, and follows its deployment instructions.
P.S. An attested-reproducible CD build system might be an interesting startup idea.
Well, in my mind the build system should build an artifact (a container, or a .deb package), and then the separate system should deploy it (with smaller amount of permitted people), and have option to roll it back. So in principle I agree on that .
...but I saw that anti-pattern of "just add a step that does the deploy after CI in same" often enough that I think it might be the most common way to do it.
As mentioned by sibling comments, these two parts are very different. GHA and alternatives are primarily for CI; many people choose to use it solely for CI. For these reasons I did ignore CD.
We're iterating towards GHA for CI, AWS CodeBuild for the CD. At least on AWS projects. Mainly because managing IAM permissions to permit the github runner to do everything the deployment wants is an astonishingly large waste of time. But you need a secret to trigger one from the other.
Technically yes. It depends on whether you consider the account ID to be a secret or not (AWS say "sensitive but not secret" which doesn't help much). But also it can make sense to treat all environment variables as secrets by default just so you don't accidentally end up putting something somewhere that turns out to have been Wrong.
And even better can scope assuming an AWS IAM role to a specific branch name & workflow filename so only code/workflows that have been through review have access to CD secrets/prod infra.
IE no prod access by editing the workflow definition and pushing it to a branch.
How do you e.g. validate that a database product works with all the different cloud databases? Every time you change up SQL generation you're going to want to make sure the SQL parses and evaluates as expected on all supported platforms.
Those tests will need creds to access third party database endpoints.
You're missing that the D in CI/CD means deployment; be that packaging on pushing tags and publishing to a registry, or building images, or packaging github releases.
CircleCI and I believe GHA support injecting signed JWTs you can use to bootstrap identity be it an IAM role or some other platform where you can trust an OIDC issuer
It is. Just wanted to point out these flows are also possible on CI. In my other comment, I think it's more fair to differentiate long lived vs short lived secrets
The secret is held by the metadata server that the CI instance has access to
Or: the deployment service knows the identity of the instance, so its secret is its private key
Or, how PyPI does it: the deployment service coordinates with the trusted CI/CD service to learn the identity of the machine (like its IP address, or a trusted assertion of which repository it’s running on), so the secret is handled in however that out-of-band verification step happens. (PyPI communicates with Github Actions about which pipeline from which repository is doing the deployment, for example)
> The secret is held by the metadata server that the CI instance has access to
But how does the metadata server know that the CI instance is allowed to access the secret? Especially when the CI/CD system is hosted at a 3rd. party. It needs to present some form of credentials. The CI system may also need permission or credentials for a private repository of packages or artifacts needed in the build process.
For me, a CI/CD system needs two things: Secret management and the ability to run Bash.
Yeah I was confused about that bit too. AWS and GCP's metadata servers know which instances were deployed, so they presumably have some way of verifying the instance's identity out-of-band, such as being tagged by an internal job or machine identifier.
1. Certain CI services (like GitHub Actions) are OIDC identity providers, meaning that they can issue short-lived credentials ("OIDC tokens") that a third party can strongly verify came from the CI service (as well as which user, repository, etc. actually executed);
2. Projects on PyPI can be configured to trust a particular configuration on a particular CI service, making that configuration an OIDC publisher for that project;
3. Release automation (like GitHub Actions) can submit an OIDC token to PyPI. The token will be matched against configurations trusted by different projects; if any projects trust the token's configuration, then PyPI will mint a short-lived API token for those projects and return it;
4. The short-lived API token behaves exactly like a normal project-scoped API token, except that it's only valid for 15 minutes from time of creation (enough time for the CI to use it to upload packages).
You have to add your github repository as a "trusted pulbisher" to your PyPI packages.
Honetsly the whole workflow bothers me -- how can PyPI be sure it's talking to github? what if an attacker could mess with PyPI's DNS? -- but it's how it's done.
PyPI is sure that it’s talking to GitHub because it establishes trust in the GitHub’s IdP public keys over HTTPS. I guess you could then question the security of HTTPS, but that seems like a significant rabbit hole to jump down given that OAuth, etc. all depend on the same basic scheme.
It would be good if it could also scan build output like code coverage and test results. But that’s about all it should do.
I keep meaning to write a partially federated CI tool that uses Prometheus for all of its telemetry data but never get around to it. I ended up carving out a couple other things I’d like to be part of the process as a separate app because I was still getting panopticon vibes and some data should just be private.
While good in theory, in practice secrets are used to validate those privileges have been assigned. Even in schemes like metadata servers, you still use a secret.
Pedantically I'd say maybe it's more fair to say they shouldn't have access to long lived secrets and should only use short lived values.
The "I" stands for Integration so it's inevitable CI needs to talk to multiple things--at the very least a git repo which most cases requires a secret to pull.
You might want (or _need_) to sign your binary, for example. Or you might want to trigger a deployment.
Github actually is doing something right here. You can set it up as a trusted identity provider in AWS, and then use Github to assume a role in your AWS account. And from there, you can get access to credentials stored in Secret Manager or SSM.
Yes, their oidc setup was probably their last good feature back when they were actually delivering features back in 2020ish. Everyone else copied it within a few months though.
Maybe in some other form, but the current style of injecting an oidc token was definitely in github actions first. Here is the gitlab issue tracking the final bit of it's implementation directly mentioning github: https://gitlab.com/gitlab-org/gitlab/-/issues/356986
This is the more flexible and secure (specific aud) replacement for CI_JOB_JWT which has been there since at least 2017, if not before. Functionally it was exactly the same, a JWT token per pipeline allowing you to authenticate to third parties that support OIDC/JWT Auth.
Exactly. This workflow step takes a rather important secret and sticks it on a VM where any insufficiently sandboxed step before or after it can exfiltrate it.
GitHub should instead let you store that key as a different type of secret such that a specific workflow step can sign with it. Then a compromised runner VM could possibly sign something that shouldn’t be signed but could not exfiltrate it.
Even better would be to be able to have a policy that the only thing that can be signed is something with a version that matches the immutable release that’s being built.
I agree 100% with what I think is the key phrase, viz. "the results can change without any modification to your code".
I maintain an R package that is quite stable and is widely used. But every month or so, the GHA on one of the R testing machines will report an error. The messages being quite opaque, I typically spend a half hour trying to see if my code is doing something wrong. And then I simply make a calendar item to recheck it each day for a while. Sure enough, the problems always go away after a few days.
TFA mentions this option and then goes on at some length to explain that this doesn't help for transitive dependencies, which is how these attacks usually work.
I've also found many Actions that do other dodgy stuff, like pulling and executing unpinned scripts from external websites, or installing unpinned binaries from GitHub releases. Pinning an Action isn't enough, you have to audit it.
When you have a multi-platform image the actual per-platforms are usually not tagged. No point.
But that doesn't mean that they are untagged.
So on GitHub Actions when you upload a multi-platform image the per-platform show up in the untagged list. And you can delete them, breaking the multi-platform image, as now it points to blobs that don't exist anymore.
Yep. I'm switching our workflows to instead use regular utilities running inside a Docker container.
This works well for _most_ things. There are some issues with doing docker-in-docker for volume mapping, but they're mostly trivial. We're using taskfiles to run tasks, so I can just rely on it for that. It also has a built-in support for nice output grouping ( https://taskfile.dev/docs/reference/schema#output ) that Github actions can parse.
Pros:
1. Ability to run things in parallel.
2. Ability to run things _locally_ in a completely identical environment.
3. It's actually faster!
4. No vendor lock-in. Offramp to github runners and eventually local runners?
Cons:
It often takes quite a while to understand how actions work when you want to run them in your own environment. For example, how do you get credentials to access the Github Actions cache and then pass them to Docker? Most of documentation just tells: "Use this Github Action and stop worrying your pretty little head about it".
Do you have a write up about this? Actions are great, but my #2 gripe with actions, after the tenuous security posture, is that the default practice is not to run/validate actions locally.
I agree that github actions has a horrible syntax and can do much less than travis, appveyor, circle, ... I'd really have to resort to yaml references to do anything elegant. But yaml references aren't elegant at all.
But it still is free and vastly outperforms the other free CI's.
I just converted our old parrot travis runners to github actions. There I had constant troubles with travis timeouts of 15m 10 years ago. With the new github actions I can run the full tests (which was not possible with travis) in 3 minutes. About 8x faster hardware.
A lot of the actions people tend to use are just unnecessary. They're simple wrappers around real tools. In those cases, use mise-en-place. It's a single action that installs all relevant tools (and keeps your local dev env in check), and it supports lock files.
This right here - I am migrating all of our GHA to use the mise action. Makes keeping the version of Go, linters, formatters etc. for the project so much easier. Haven't added the mise.lock yet, but on the list. Now getting my small team of devs to try using mise is much harder.
I'm assuming the lockfile should be checked into the repo itself, which presents a bootstrapping problem if you have to run an action to create the lockfile in the first place. They may need to build proper support for running actions locally -- there is the third-party https://github.com/nektos/act tool which might be a starting point, but that's mostly designed so you can debug actions without having to repeatedly push and rerun. Probably they'll need a separate mechanism to statically analyze actions without running them.
I committed the project I maintain to GitHub Actions when Actions first came out, and I'm really starting to regret it.
The main problem, which this article touches, is that GHA adds a whole new dimension of dependency treadmill. You now have a new set of upstreams that you have to keep up to date along with your actual deployment upstreams.
After having to deal with shell rc files causing chaos and unpredictability, I can appreciate your top level code fundamental not being random code but something that is parsable and not subject to the halting problem.
> I've not understood the propensity for using yaml for CI pipelines and workflows in general. A decent programming language would be a big improvement.
Because it's clear to write and read. You don't want your CI/CD logic to end up being spaghetti because a super ninja engineer decided they can do crazy stuff just because they can. Same reason why it's a bad idea to create your infrastructure directly in a programming language (unless creating infrastructure is a core part of your software).
> Why not just build the workflows themselves as docker images? I guess running other docker images in the workflow would then become a problem.
That's how Drone CI handled it. GitLab kind of does the same, where you always start as a docker image, and thus if you have a custom one with an entrypoint, it does whatever you need it to.
IME on a Pulumi for IaC team, writing infra in a real language (TypeScript) is MILES better than- you can do conditions, have typed outputs, etc and not have it be a bastardized imperative YAML mess.
YAML is fine for data, but inevitably stuff like workflows end up tacking on imperative features to a declarative language.
> IME on a Pulumi for IaC team, writing infra in a real language (TypeScript) is MILES better than- you can do conditions, have typed outputs, etc and not have it be a bastardized imperative YAML mess.
You can have conditions and types without having the full flexibility allowing madness of a full language with HCL.
I enjoy it, but I also think my platform team has set it up in the best possible way. YMMV if it’s done differently- I am sometimes uncertain if I love pulumi, or how we do pulumi.
I really really want to use dagger, but I don’t think there’s organizational interest in it.
it's wild I can wiz through a ton of code for hours on end but seeing a yaml file for something like a CI pipeline actually makes my brain eject i dunno why. my brain has some sort of proverbial capacity limit with how many different configuration-file looking things I can tolerate in a day, and the prospect of becoming intimately familiar with what is effectively an auto integration presented to me as some sort of config makes me completely unjustifiably butthurt for no reason. have i not suffered enough needless and often times limiting abstractions already
Has anyone built a “deep fork” tool that lets you make a private fork of all dependencies, then modifies their transitive dependencies to point to private forks, and so on? Ideally in a way where updates can be pulled in manually? Seems feasible.
> Has anyone built a “deep fork” tool that lets you make a private fork of all dependencies, then modifies their transitive dependencies to point to private forks, and so on? Ideally in a way where updates can be pulled in manually? Seems feasible.
If you do, please submit a "show HN." I'd love to use it.
I checked out the linked GitHub repo https://github.com/ecosyste-ms/package-manager-resolvers and it appears to be just a README.md that collects summaries of different package managers? How do I know these weren't just LLM-generated?
I feel it's not that controversial to treat LLM output as unreliable without proper verification. So even if the output were accurate, if it's LLM-generated without proper verification I don't trust it.
> You trust GitHub to give you the right code for a SHA.
The vast majority of users use GitHub-hosted runners. If you don't trust GitHub, you have bigger problems than whether the correct code for an action is downloaded.
I assume that git (not Github) verifies that, if you checkout a hash, the contents of the commit match the hash.
Anyway, software is so complicated that at some level, you need to trust something because it's impossible to personally comprehend and audit all code.
So, you still need to trust git. You still need to trust your OS. You still need to trust the hardware. You just don't have enough minutes in your life to go down through all those levels and understand it well enough to know that there's nothing malicious in there.
> Some teams vendor actions into their own repos. zizmor is excellent at scanning workflows and finding security issues. But these are workarounds for a system that lacks the basics.
Harsh given GitHub makes it very easy to setup attestations for Artifact (like build & sbom) provenances.
That said, Zizmor (static analyser for GitHub Actions) with Step Security's Harden Runner (a runtime analyser) [0] pair nicely, even if the latter is a bit of an involved setup.
I’d say that GitHub has done an admirable job making attestations more accessible, but that “easy” is still a stretch of a characterization: it’s still not the default, and the error/configuration states are somewhat opaque (e.g. around OIDC permissions, unprivileged triggers, what constitutes a signing identity in a reusable workflow context, etc.). Some of these are latent complexities that GitHub can’t be blamed for, but some are certainly made worse by architectural decisions in GitHub Actions.
We are currently using GitHub Actions for all our CI tasks and I hate it. Yes, the marketplace is nice and there are a lot of utility actions which make life easier, but they all come with the issues the post highlights. Additionally, testing Actions locally is a nightmare. I know that act exists but for us it wasn't working most of the time. Also the whole environment management is kinda odd to me and the fact, that when using an environment (which then allows to access secrets set in that environment) it always creates a new deployment is just annoying [1]
I guess the best solution is to just write custom scripts in whatever language one prefers and just call those from the CI runner. Probably missing out on some fancy user interfaces but at least we'd no longer be completely locked into GHA...
It is concerning that GitHub hosts the majority of open-source software, while actively locking its users into a platform that is based on closed source for eerything except Git itself.
This issue with Actions shows how maintaining proprietary software inevitably ends up rather low on the priority list. Adding new features is much more marketable, just like for any other software product. Enshittification ensues.
AFAIK because "act", the tool to run github actions locally, was there and there was no need to create something else. Also makes it easier for people to switch from github.
Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/
Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.
If I write actions/setup-python@v1, I'm expecting the action to run with the v1 tag of that repository. If I rerun it, I expect it to run with the v1 tag of that repository...which I'm aware may not be the same if the tag was updated.
If I instead use actions/setup-python@27b31702a0e7fc50959f5ad993c78deac1bdfc29 then I'm expecting the action to run with that specific commit. And if I run it again it will run with the same commit.
So, whether you choose the tag or the commit depends on whether you trust the repository or not, and if you want automatic updates. The option is there...isn't it?
You specifying the top level hash doesn't do anything to pin transitive dependencies, and as the article points out, transitive dependencies - especially dependencies common to a lot of actions - would be the juciest target for a supply chain attack.
> which I'm aware may not be the same if the tag was updated.
That's the mistake that breaks the following. People don't usually expect that it's an arbitrary modifiable reference, but instead that it's the same version they've picked when they created the file (ie a tag is just a human friendly name for a commit)
While I hate defending GHA, the docs do include this:
- Using the commit SHA of a released action version is the safest for stability and security.
- If the action publishes major version tags, you should expect to receive critical fixes and security patches while still retaining compatibility. Note that this behavior is at the discretion of the action's author.
So you can basically implement your own lock file, although it doesn't work for transitive deps unless those are specified by SHA as well, which is out of your control. And there is an inherent trade-off in terms of having to keep abreast if critical security fixes and updating your hashes, which might count as a charitable explanation for why using hashes is less prevalent.
On the other hand, this issue has been known to GitHub since shortly after Actions’ release[0]. They added some cya verbiage to their docs, but they never followed up by making version pinning meaningful.
Sure you can implement it yourself for direct dependencies and decide to only use direct dependencies that also use commit sha pinning, but most users don’t even realize it’s a problem to begin with. The users who know often don’t bother to use shas anyway.
Or GitHub could spend a little engineer time on a feasible lock file solution.
I say this as somebody who actually likes GitHub Actions and maintains a couple of somewhat well-used actions in my free time. I use sha pinning in my composite actions and encourage users to do the same when using them, but when I look at public repos using my actions it’s probably 90% using @v1, 9% @v1.2 and 1% using commit shas.
[0] Actions was the first Microsoft-led project at GitHub — from before the acquisition was even announced. It was a sign of things to come that something as basic as this was either not understood or swept under the rug to hit a deadline.
> it doesn't work for transitive deps unless those are specified by SHA as well, which is out of your control
So in other words the strategy in the docs doesn't actually address the issue
There's a repository setting you can enable to prevent actions from running unless they have their version pinned to a SHA digest. This setting applies transitively, so while you can't force your dependencies to use SHA pinning for their dependencies, you can block any workflow from running if it doesn't.
A lockfile would address this issue, with the added benefit that it would work
Using an SHA is an anti-pattern for me. Because by using one, you kind of modeled "I am getting this fixed/static thing"; when in reality, it is very far from that. I got bitten by it twice that I learned that you either have a lock file or you don't.
Wait, can you explain how you _don't_ get a static thing from a SHA?
- Using the commit SHA of a released action version is the safest for stability and security.
This is not true for stability in practice: the action often depends on a specific Node version (which may not be supported by the runner at some point) and/or a versioned API that becomes unsupported. I've had better luck with @main.
Depends what you mean by stability. The post is complaining about the lack of lockfiles, and the problem you describe would also be an issue with lockfiles.
The underlying problem is that you can't keep using the same version, and one way it fails ruins the workaround for a different failure.
What’s more, GitHub has basically stopped maintaining their own actions, pushing people to sketchy forks to do basic things. Their entire ecosystem is basically held up with duct tape and gets very little investment.
> Their entire ecosystem is basically held up with duct tape and gets very little investment.
That isn't gonna get better anytime soon.
"GitHub Will Prioritize Migrating to Azure Over Feature Development" [1]
[1] https://thenewstack.io/github-will-prioritize-migrating-to-a...
Hey at least we can all expect lots of extra days off because "GitHub is down" once they're done with that migration!
They had working infra and a great case for keeping fairly "close to the metal". Complicated files-heavy workload that needs tons of clever caching to perform well, lots of writes, lots of non-HTTP TCP traffic.
Retrofitting that into "cloud" bullshit is such a bad idea.
meh, I dunno.
Using bare-metal requires competent Unix admins, and Actions team is full of javascript clowns (see: decision to use dashes in environment variable; lack of any sort of shell quoting support in templates; keeping logs next to binaries in self-hosted runners). Perhaps they would be better off using infra someone else maintains.
> Using bare-metal requires competent Unix admins
So does running VMs in a cloud provider.
Except now we call them "DevOps" or "SRE" and pay them 1.5-2x.
(as a former SRE myself, I'm not complaining).
They barely maintain Azure pipeline tasks / actions as well.
We had a critical outage because they deprecated Windows 2019 agents a month earlier than scheduled. MS support had the gall to both blame us for not migrating sooner, and refuse to escalate for 36 hours!
Except actions/ai-task. Im sure that one will come.
An interesting things is that GitHub is an expensive service and my guess would be that MS makes good money on it. Our small company paid about 200+ USD monthly for GitHub, much larger cumulative cost than Windows licenses. My believe was that Windows is getting worse, because it is considered legacy business by MS in favor of new offerings such as GitHub subscriptions.
Very many more people use Windows to GitHub.
GitHub also runs a free tier with significant usage.
There are ~1.4b paid instances of Windows 10/11 desktop; and ~150m Monthly active accounts on GitHub, of which only a fraction are paid users.
Windows is generating something in the region of $30b/yr for MS, and GitHub is around $2b/yr.
MS have called out that Copilot is responsible for 40% of revenue growth in GitHub.
Windows isn't what developers buy, but it is what end users buy. There are a lot more end users than developers. Developers are also famously stingy. However, in both products the margin is in the new tech.
github value maybe as not apparent as other product
but github is pair well with MS other core product like Azure and VS/VSC department
MS has a good chance to have vertical integration on how software get written from scratch to production, if they can somehow bundle everything to all in one membership like Google one subs, I think they have a good chance
I was surprised to learn that Depot runners, which are much faster, are also much cheaper. Would highly recommend them for anyone trapped on GitHub.
Blacksmith.sh has been great for us. Massively sped up tests and a huge improvement for Docker builds over both Actions and Google Cloud Build.
Only downside is they never got back to us about their startup discount.
hey there! blacksmith solutions engineer here :) love to hear we've helped speed up your tests and docker builds!!
could you shoot me your GH org so I can apply your startup discount? feel free to reach out to support@blacksmith.sh and I'll get back to you asap. thanks for using blacksmith!
Thank you! We've loved it! Looks like you found me, thank you :)
Yeah, but I have to set that up.
GitHub actions more or less just work for what most people need. If you have a complex setup, use a real CI/CD system.
I haven’t use depot but I’m pretty sure the setup is literally just switching out the runs-on value in your workflows
Such as?
Jenkins is open source and very well documented.
GitHub Actions are really for just short scripts. Don't take your Miata off road.
LOL, I worked on the Jenkins project paid for three years. Even they use actions to build Jenkins.
https://github.com/jenkinsci/jenkins/tree/master/.github/wor...
Jenkins! For the love of god don’t listen to this.
Always open to learning, what's wrong with Jenkins?
It's a bit bloated, but it's free and works.
Why is gha just for short scripts, out of interest?
It's just short on features.
I get the vibe it was never intended to seriously compete with real CI/CD systems.
But then people started using it as such, thus this thread is full of complaints.
What features is it missing that you would like to see it implement?
death before Jenkins
Depot.dev is great.
Thank you! Really appreciate the support.
Thank you for the kind shout out! Always happy to see comments like this. If anyone is looking for a better GitHub or GitHub Actions experience, feel free to reach out anytime.
What are Depot runners?
Founder of Depot here. We provide faster and more reliable GitHub Actions runners (as well as other build performance services) at half the cost of GitHub [0]
[0] https://depot.dev/
Is there a write up on the security of actions or equivalent that explains how they are secure both with direct and transitive dependencies? If this applies to Depot.
Ah got it, thanks. I thought there was another kind of GitHub runner (like their "large" runners) that I hadn't heard of.
The legacy business usually explains why there are no new features, only minor maintenance, it doesn't explain why there is a lot of investment into work that makes it worse
It's not really that expensive. GitHub Enterprise is like $21/month/user while GitLab Ultimate was $100/month/user the last time GitLab published prices. These days GitLab Ultimate is "contact us for pricing" while the cheaper GitLab Premium is $29/month/user.
I guess Bitbucket is cheaper but you'll lose the savings in your employees bitching about Bitbucket to each other on Slack.
When’s the last time you looked at or used Bitbucket?
Like a week ago? We’re currently migrating away from it as everyone hated it.
Do we work in the same company? That said, I really don't understand why everyone hates on Bitbucket. I really thought it was _fine_ from a user perspective. Now we're on GHE and I find it a sidegrade at best.
Now for the people who were operating Bitbucket, I'm sure it's a relief.
> My believe was that Windows is getting worse, because it is considered legacy business by MS in favor of new offerings such as GitHub subscriptions.
What if GH actions is considered legacy business in favour of LLMs?
I wouldn't be surprised if there isn't some plan to make all of GitHub's backend "legacy"
and switch everyone to the dumpster fire that is Azure DevOps
and if you thought GitHub Actions was bad...
When Microsoft bought GitHub they cancelled GitHubs own early CI effort and rebranded the existing Azure DevOps as GitHub Actions.
The GitHub Actions runner source code is all dotnet. GitHub was a Ruby shop.
I believe the original GitHub Actions was in Go - it used HCL which was at that point only really implemented in Go. Quite the move backwards to switch to YAML.
IIRC Azure DevOps was the “dead one”, all new development only takes place on GitHub.
From my perspective, Azure Pipelines is largely the same as GitHub Actions. I abhor this concept of having abstract and opaque “tasks”.
There's direct evidence that GitHub Actions was the rewrite of Azure Pipelines that was originally planned to finish 5 years ago and got "stuck" (because all their resources moved to GitHub). For a while you could find 2020 roadmap repositories (on GitHub) for AzDO talking up a Pipelines rewrite bringing a lot more features (including better Docker alignment versus Pipelines' much more complex "runner skills") that instead showed up in the first version of GitHub Actions.
Microsoft claims Azure DevOps still has a roadmap, but it's hard to imagine that the real roadmap isn't simply "Wait for more VPs in North Carolina to retire before finally killing the brand".
> I wouldn't be surprised if there isn't some plan to make all of GitHub's backend "legacy"
> and switch everyone to the dumpster fire that is Azure DevOps
The other way around. Azure DevOps is 1/2 a backend for Github these days. Github re-uses a lot of Azure Devops' infrastructure.
github doesn't pay microsoft for the azure runners. that's why they came up with actions at all. microsoft gets streetcreds for stable runners, github could replace travis and appveyor.
The quality of setup-* actions has definitely gone down and there are a lot of strange decisions being made. I assume the original authors of these actions have long left the company.
This is the first time I've heard of this, do you happen to have an example?
https://github.com/search?q=org%3Aactions+%22we+are+allocati...
i.e. from https://github.com/actions/cache/?tab=readme-ov-file#note
That's insane, so they are basically dropping support on a core feature of GH Actions?
What they are really saying is they don't want third party contributions. They don't have anyone triaging Issues or PRs so don't send them.
They will occasionally make changes if it aligns with a new product effort driven from within the org.
Saying they're dropping support is a stretch esp as very few people actually pay for their Support package anyway..... (Yes they do offer it as a paid option to Enterprise customers)
This is on the checkout action too, by the way. You know, the very first thing people put in their CI pipeline.
Wow, Microsoft just can't stop taking a dump on their users
they probably have a half-assed plan to push some sort of checkout action copilot button instead of dependable scripts/actions.
https://githubnext.com/projects/agentic-workflows/
> Instead of writing bespoke scripts that operate over GitHub using the GitHub API, you describe the desired behavior in plain language. This is converted into an executable GitHub Actions workflow that runs on GitHub using an agentic "engine" such as Claude Code or Open AI Codex. It's a GitHub Action, but the "source code" is natural language in a markdown file.
This seems like a real headache to me. I understand the value proposition of LLMs in the development cycle, but CI/CD is probably the last place where I want any degree of nondeterminism.
This looks like backwards. I would understand using a LLM to generate a GitHub Actions YAML, but always running your action from a Markdown file seems extremely wasteful in terms of resources.
Edit: ok, looking at example it makes more sense. The idea is to run specific actions that are probably not well automated, like generating and keeping documentation up-to-date. I hope people don't use it to automate things like CI runs though.
Because they know most abusive business relationship partners don't leave (see also Oracle). No matter how many bruises, CIO's are not going to get fired for buying Big Blue or whoever is the current abusive standard.
https://github.com/orgs/actions/repositories?language=&q=&so...
https://github.com/actions/create-release
The funny thing about the last one is that those actions ultimately boil down to invoking their CLI tool (which is pre-installed on the runners) with "gh release create ...", so you can just do that yourself and ignore the third-party actions and the issues that come with them. Invoking an action isn't really any easier than invoking the CLI tool.
Yeah, what really needs to happen with that repo is to put that in the README to use the gh CLI instead of pointing to the third-party action with questionable security policies. If they were accepting PRs for that repo, it would be an easy PR to make.
Didn't they announce to slow down development for AI?
you just described every microsoft business model.
That issue with their own small private forks has actually raised its head while testing out the AI slop generator thing it has, making anything it produces for you not self hoatable unless you rewrite a lot of basic functions. Sweet irony.
Which is strange because they have infinite Microsoft money and can print more if they get it into enterprises.
(we run a private gitlab instance and a merge request can spawn hundreds of jobs, that's a lot of potential Gitlab credits)
With AI you won't need CI anymore, it's all going straight to prod anyway /s
Actions is one thing, but after all these years where the new finegrained access tokens aren't still supported across all the product endpoints (and the wack granularity) is more telling about their lack of investment in maintenance.
I never used any actions and never understood why would I need to. I just wrote bash script to build my project and that's about it. This modern tendency to add dependencies for trivial things baffles me. You don't need "action" to do `git clone`.
bash scripts are as inscrutable as any GHA.
They are perfectly supportable and auditable as you're writing them yourself. It's just ordinary shell commands, there's nothing inscrutable about it.
Everyone is free to use alternative CI/CD workflow pipelines. These are often better than Github Actions.
These include
- https://circleci.com/
- https://www.travis-ci.com/
- Gitlab
Open source:
- https://concourse-ci.org/ (discussed in the context of Radicle here https://news.ycombinator.com/item?id=44658820 )
- Jenkins
-etc.
Anyone can complain as much as they want, but unless they put the money where their mouth is, it's just noise from lazy people.
I’d appreciate not being called lazy for mentioning a lack of investment on Microsoft’s side to secure their paid and fairly lucrative service that they bought a popular code hosting platform to integrate with.
Can someone explain what this somewhat recent phenomenon is where people feel the need to defend the worlds biggest billion dollar businesses, that are also often subsidized by tax payer money in weird ways?
How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
> How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
Honestly I think the problem is more a rosy view of the past versus any actual change in behavior. There have always been defenders of such companies.
> How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
They hired a ton of people on very very good salaries
I think big tech being so big now that these "issue" is too small for their priority is saying something
You better thank god for MS for being lazy and incompetent, the last thing we want for big tech is being innovative and have a stronger monopoly
The original comment said to stop giving money to these companies if they are not giving you a satisfactory service.
The opposite, to be lazy and to continue giving them money whilst being unhappy with what you get in return, would actually be more like defending the companies.
The original comment actually criticized Microsoft for a lack of investment to secure their paid and fairly lucrative service that they bought a popular code hosting platform to integrate with.
The opposite we see here: to not criticize them; to blame Microsoft's failure on the critics; and even to discourage any such criticism, are actually more like defending large companies.
It is a lucrative service just because people are lazy and keep buying from Microsoft. Otherwise, they would migrate to better alternatives.
This especially includes governments and other institutional buyers.
I won't "defend" Microsoft in this case, but I am always annoyed by phrases like "world's biggest billion-dollar businesses... bablah".
Their size or past misbehaviors shouldn't be relevant to this discussion. Bringing those up feels a bit like an ad hominem. Whether criticism is valid should depend entirely on how GitHub Actions actually works and how it compares to similar services.
Ad hominem applies to people. Corporations aren’t people, and ICs aren’t corporations.
> Their size or past misbehaviors shouldn't be relevant to this discussion.
If the past misbehaviours are exactly the same shape, there's not all that much point re-hashing the same discussion with the nouns renamed.
Microsoft's past behavior _may_ explain *why* there is a lack of investment in Github Actions; so yes, TheFeelz are relevant.
Then I agree with this. But still feel their size is irrelevant.
Their size is relevant in so far as it allows them to make really any investment they want to in GHA without it causing a cash flow problem.
There is a massive problem in open source where some people equate pointing out a problem with being too lazy to solve it — when in reality this just stifles the conversation. Especially when a prerequisite to any group project accomplishing anything is to first discuss the problem to be solved.
No that's actually a completely different issue. You're talking about volunteers working on side projects that are sometimes foundational to the way the internet works and then people feel entitled to tell them what to do without contributing.
Here we are talking about one of the worlds most valuable companies that gets all sorts of perks, benefits and preferential treatment from various entities and governments on the globe and somehow we have to be grateful when they deliver garbage while milking the business they bought.
No, that's actually the same issue. "Entitled to tell them what to do without contributing" is not a problem. Let them tell whoever what to do, the response is always the same: "patches welcome," or if that isn't even true (which it doesn't have to be), "feel free to fork."
OTOH if you didn't pay for support you shouldn't expect support. 'patches welcome' is a very valid response.
Is not the whole FOSS movement about receiving something you did not pay for? Going as far as to say that’s even what users deserve?
don't confuse 'receiving something you did not pay for' with 'being allowed to feel entitled to anything' is all. 'open source' is just that, nothing more. if you want a service with your source, be prepared to sponsor it.
I still think people should want things and be vocal about what they want. This is the natural way for people to know what needs to be built. It is different from demanding something.
And besides that, a lot of people on here do pay for Github in the first place.
Well, actually, no, not everyone is free to use alternatives. Anyone using CI for "Trusted Publishing" of packages to PyPI or npm needs to use GitHub Actions or GitLab CI/CD. CircleCI and Travis CI are not supported. So many big open source projects for the two most popular languages in the world are now locked out of the alternatives you propose.
(I find it extremely sketchy from a competition law perspective that Microsoft, as the owner of npm, has implemented a policy banning npm publishers from publishing via competitors to GitHub Actions - a product that Microsoft also owns. But they have; that is the reality right now, whether it's legal or not.)
I was never convinced that trusted publishing solves any security problem, other than letting pypi eventually solve the problem of banning russian/iranian/whatever people just by relying on github doing it for them.
Trusted Publishing on PyPI supports Google Cloud and ActiveState as well. It’s not tied to GitHub or GitLab. To my recollection I looked at CircleCI support a while back, and ran into limitations on the claims they exposed.
(It can also be extended to arbitrary third party IdPs, although the benefit of that is dependent on usage. But if you have another CI/CD provider that you’d like to integrate into PyPI, you should definitely flag it on the issue tracker.)
> unless they put the money where their mouth is, it's just noise
I used to work for a Japanese company, and one of their core philosophies was “Don’t complain, unless you have a solution.” In my experience, this did not always have optimal outcomes: https://littlegreenviper.com/problems-and-solutions/
My favorite retort to that is, "I don't have to know how to fix it to know my arm is broken."
Stealing it...
I don’t make the purchasing decision for my employer, but I certainly have to deal with their fallout, so I’ll keep complaining if that’s okay with you.
If you are not part of the solution, then you are part of the problem.
If you’re so lucky that your employer lets you use any $TOOL you want, great, but I _have_ to use GitHub actions.
So I’m part of the problem? Me specifically?
What is it that you actually want me to do here?
Just refuse to do my job because I think the tools suck?
I've used CircleCI quite a bit in the past; it was pretty good. Feels tough for them to compete with GHA though when you're getting GHA credits for free with your code hosting.
I used Travis rather longer ago, it was not great. Circle was a massive step forward. I don't know if they have improved it since but it only felt useful for very simplistic workflows, as soon as you needed anything complex (including any software that didn't come out of the box) you were in a really awkward place.
CircleCI made great steps the last few years, f.e. to better support proper DRY working, supporting OPA policies-as-code, VSCode extensions with "dry-run" options.
For some examples of more advanced usecases take a look: https://circleci.com/blog/platform-toolkit/
Disclaimer: i work for CircleCI.
To be clear, I do think CircleCI is a better product than GHA. I just think there's a lot of air sucked out of the room by GHA being available 'for free' and out of the box.
Also, honestly, I don't care about any of those features. The main thing I want is a CI system that is fast and customisable and that I don't have to spend a lot of time debugging. I think CircleCI is pretty decent in that regard (the "rerun with SSH" thing is way better than anything else I've seen) but it doesn't seem to be getting any better over time (e.g. caching is still very primitive and coarse-grained).
I had a considerably better time with CircleCI in the past than with Github Actions currently. It feels much more like a complete product rather than a tacked on mess, I hate how disproportionately we count running cost just because we have numbers for it (vs. DX and velocity which are hard to measure and impossible to predict)
I mean, they do have a free plan with 6,000 minutes
> Anyone can complain as much as they want, but unless they put the money where their mouth is, it's just noise from lazy people.
Once I'm encharged of budge decisions of my company I'll make sure that none will go to any MS and Atlassian product. Until then I'll keep complaining.
It should be highlighted that Gitlab CI/CD (self-hostable runner and GitLab itself) is also OSS.
I'm a huge fan of: https://onedev.io/ it might not the best, but it's fast it and does it's job!
Or roll your own
I tried to use CircleCI and I gotta say, it is absolutely not better than GitHub Actions…
I have also used Travis. Ditto.
Github Actions is actually one of the better CI options out there, even if on an absolute scale it is still pretty bad.
As far as I can tell nobody has made a CI system that is actually good.
buildkite is leaps and bounds above the others. especially if you need to really tailor your workloads to the change diff (say in a monorepo), the dynamic pipeline support is superb.
really surprised there are no others though. dagger.io was in the space but the level of complexity is an order of magnitude higher
GitLab CI is pretty close to being actually good. Certainly less brittle than GitHub Actions from the looks of it...
It's really not. I've used Gitlab CI extensively and the number of issues we had...
JetBrains TeamCity is pretty good IMHO
I quite liked Bitrise for mobile apps when I used that.
CircleCI is 100% trash.
Don't waste your time
It sounds like you've never worked in a large org before.
according to travis-ci, Microsoft uses that? Lol
You're falling for a marketing trick.
What that type of section usually means is "there's someone from Microsoft that signed up for our service using his work account", sometimes it means "there's some tiny team within Microsoft that uses our product", but it very rarely (if ever) means "the entire company is completely reliant on our product".
Yes and no. Generally logo usage requires permission. While the usage isn’t the whole company, it’s enough to justify some sort of logo usage.
Normally I’d say stop kicking the dead horse, but GHA deserves all the complaints it gets and then some. It’s the embodiment of everything that’s bad in ‘less is more’.
My biggest concern with it is that it’s somehow the de facto industry standard. You could do so much better with relatively small investments, but MS went full IE6 with it… and now there’s a whole generation of young engineers who don’t know how short their end of the stick actually is since they never get to compare it to anything.
It's funny that absolutely everything about GHA fucking sucks, and everyone agrees about this. BUT, the fact that it's free compute, and it's "right there"... means it's very very difficult to say no to!
Personally I've just retired a laptop and I'm planning to turn it into a little home server. I think I'm gonna try spinning up Woodpecker on there, I'm curious to see what a CI system people don't hate is like to live with!
I can already tell by their example that I don't like it. I've worked with a bunch of different container-based CI systems and I'm getting a little tired seeing the same approach by done slightly differently.
Yes, it's easy to read and understand and it's container based, so it's easy to extend. I could probably intuitively add on to this. I can't say the same for GitHub, so it has that going for it.But the moment things start to get a little complex then that's when the waste starts happening. Eventually you're going to want to _do_ something with the artifacts being built, right? So what does that look like?
Immediately that's when problems start showing up...
- You'll probably need a separate workflow that defines the same thing, but again, only this time combining them into a Docker image or a package.
- If I were to only change a `frontend` file or a `backend` file, then I'm probably going to end up wasting time and compute rebuilding the same artifacts over and over. - Running locally using the local backend... looks like a huge chore. In Drone this was basically impossible.I really wish someone would take a step back and really think about the problems being solved here and where the current tooling fails us. I don't see much effort being put into the things that really suck about github actions (at least for me): legibility, waste, and the feedback loop.
> absolutely everything about GHA fucking sucks
By adding one file to your git repo, you get cross-platform build & test of your software that can run on every PR. If your code is open source, it's free(ish) too.
It feels like a weekend project that a couple people threw together and then has been held together by hope and prayers with more focus on scaling it than making it well designed.
> there’s a whole generation of young engineers who don’t know how short their end of the stick actually is
I'm from a generation who had to use VSS for a few years. The sticks are pretty long these days, even the ones you get from github.
> VSS
I just had trauma!
I will say that SourceSafe had one advantage: You could create "composite" proxy workspaces.
You could add one or two files from one workspace, and a few from another, etc. The resulting "avatar" workspace would act like they were all in the same workspace. It was cool.
However, absolutely everything else sucked.
I don't miss it.
So it’s a workspace that includes changes from multiple branches at once, like `jj new revset-1 revset-2 revset-3 …` ?
(Git has octopus merges, jj just calls them “merge commits” even though they may have more than two parents)
Don't remember exactly. If I think about it, it could be quite complex.
Git has the concept of "atomic repos." Repos are a single unit, including all files, branches, tags, etc.
Older systems basically had a single repo, with "lenses" into sections of the repo (usually called "workspaces," or somesuch. VSS called them something else, but I can't remember).
I find the atomic repo thing awkward; especially wrt libraries. If I include a package, I get the whole kit & kaboodle; including test harnesses and whatnot. My libraries thend to have a lot more testing code than library code.
Also, I would love to create a "dependency repo," that aggregates the exported parts of the libraries that I'm including into my project, pinned at the required versions. I guess you could say package managers are that, but they are kind of a blunt instrument. Since I eat my own dog food, I'd like to be able to write changes into the dependency, and have them propagate back to their home repo, which I can sort of do now, if I make it a point to find the dependency checkout, make a change, then push that change, but it's awkward.
But that seems crazy complex (and dangerous), so I'm OK with the way things work now.
Your workflow is fascinating! What languages do you work in, if you don’t mind me asking?
Both git and jj have sparse checkouts these days, it sounds like you’d be into that
Do you vendor the libraries you use? Python packages typically don’t include the testing or docs in wheels uploaded to PyPI, for instance
These days in Pythonland, it’s typical to use a package manager with a lockfile that enforces build reproducibility and SHA signatures for package attestation. If you haven’t worked with tools like uv, you might like their concepts (or you might be immediately put off by their idea of hermetically isolated environments idk)
I work mostly in Swift (native Apple apps). Most of my libraries are Swift Package Manager modules.
You can see most of my stuff in GH. You need to look at the organizations, as opposed to my personal repos: https://github.com/ChrisMarshallNY#browse-away
Thanks for the heads-up. I'll give it a gander.
a repo is a repo - you're describing what is nowadays known as a 'monorepo' and it's a perfectly reasonable and desirable even exactly for the reasons you mention, except the 'distributed' part makes it very inconvenient to handle on dev boxes if it grows a lot.
in a centralized VCS there are viable CICD options like 'check the compiler binaries in' or even 'check the whole builder OS image in' which git is simply not able to handle by design and needs extensions to work around deficiencies. git winning the mindshare battle made these a bit forgotten, but they were industry standard a couple decades ago.
[dead]
No, it let you continue to follow the main branch for most files, while files you edited would have their changes saved to a different location. And was just about as horrible as you might imagine.
We moved from VSS to SVN, and it took a little encouraging for the person who had set up our branching workflow using that VSS feature to be happy losing it if that freed us from VSS.
I'm accustomed to just doing as much as possible locally. GHA doesn't even seem like a value-add over that for me.
This is making me feel quietly vindicated in pushing back on migrating our Jenkins/Ansible setup to GHA simply because corporate wanted the new shiny thing. Fortunately the "this will be a lot of work, i.e. cost" argument won.
Mind you, CI does always involve a surprising amount of maintenance. Update churn is real. And Macs still are very much more fiddly to treat as "cattle" machines.
Lol at my first job I migrated our setup from GHA to Jenkins. I think they eventually went back to GHA
Current job is using blacksmith to save on costs, but the reality of it is that this caching layer only adds costs in some of our projects
Pleased this is being discussed somewhere as it’s something that has troubled me for a while.
There are so many third party actions where the docs or example reference the master branch. A quick malicious push and they can presumably exfiltrate data from a ton of repositories
(Even an explicit tag is vulnerable because it can just be moved still, but master branch feels like not even trying)
What if GitHub Actions were local-first and built using Nix (proper locking)?
https://github.com/cachix/cloud.devenv.sh
Hosted code on GitHub no less
> The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to secrets.
Why do CI/CD systems need access to secrets? I would argue need access to APIs and they need privileges to perform specific API calls. But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
This all seems right, but the reality is that people will put secrets into CI/CD, and so the platform should provide an at least passably secure mechanism for them.
(A key example being open source: people want to publish from CI, and they’re not going to set up additional infrastructure when the point of using third-party CI is to avoid that setup.)
Fine. Then let people set up a little WASM app that gets access to the secret and has an API callable by the workflow. And make that app be configured as part of the secret’s configuration, not as a file in the repository.
“Good CI systems shouldn’t support secrets, at most there should be [the most complicated secret support ever]”
Let’s just call it secret support.
I agree with your suggestion that capabilities-based APIs are better, but CI/CD needs to meet customers where they’re at currently, not where they should be. Most customers need secrets.
We use proprietary tools (QNX compiler, Coverity static analysis, ...) and those require access to a license server which requires some secret.
I don't really understand what you mean by "secure enclave style"? How would that be different?
With a secure enclave or an HSM, there's a secret, but the users do not have access to the secret. So, if you have a workflow that needs to, say, sign with a given private key, you would get an API that signs for you. If you need to open a TLS connection with a client certificate, you get a proxy that authenticates for you.
I suppose I would make an exception for license keys. Those have minimal blast radii if they leak.
And how is that exposed to the CI/CD? An environment variable? Some bespoke tool that the CI runs to read the secret from the Secure Enclave?
Your approach boils down to “lets give each step its own access to its own hardware-protected secrets, but developers shouldn’t otherwise have access”
Which is a great way to “support secrets,” just like the article says.
> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
CI/CD does not exist in the vacuum. If you had CI/CD entirely integrated with the rest of the infrastructure it might be possible to do say an app deploy without passing creds to user code (say have the platform APIs that it can call to do the deployment instead of typical "install the client, get the creds, run k8s/ssh/whatever else needed for deploy").
But that's a high level of integration that's very environment specific, and without all that many positives (so what you don't need creds, you still have permission to do a lot of mess if it gets hijacked), and a lot, lot more code to write vs "run a container and pass it some env vars" that had become a standard
You seem to be talking mostly about the CD part. Some thoughts:
On the one hand, CD workflows are less exposed than CI workflows. You only deploy code that has made it through your review and CI processes. In a non-continuous deployment model, you only deploy code when you decide to. You are not running your CD workflow on a third-party pull request.
On the other hand, the actual CD permission is a big deal. If you leak a credential that can deploy to your k8s cluster, you are very, very pwned. Possibly in a manner that is extremely complex to recover from.
I also admit that I find it rather surprising that so many workflows have a push model of deployment like this. My intuition for how to design a CD-style system would be:
1. A release is tagged in source control.
2. Something consumes that release tag and produces a production artifact. This might be some sort of runner that checks out the tagged release, builds it, and produces a ghcr image. Bonus points if that process is cleanly reproducible and more bonus points if there's also an attestation that the release artifact matches the specified tag and all the build environment inputs. (I think that GitHub Actions can do this, other than the bonus points, without any secrets.)
3. Something tells production to update to the new artifact. Ideally this would trigger some kind of staged deployment. Maybe it's continuous, maybe it needs manual triggering. I think that, in many production systems, this could be a message from the earlier stages that tells an agent with production privileges to download and update. It really shouldn't be that hard to make a little agent in k8s or whatever that listens to an API call from a system like GitHub Actions, authenticates it using OIDC, and follows its deployment instructions.
P.S. An attested-reproducible CD build system might be an interesting startup idea.
Well, in my mind the build system should build an artifact (a container, or a .deb package), and then the separate system should deploy it (with smaller amount of permitted people), and have option to roll it back. So in principle I agree on that .
...but I saw that anti-pattern of "just add a step that does the deploy after CI in same" often enough that I think it might be the most common way to do it.
CI shouldn't do deployments, deployment pipelines should run separately when a new release passes CI
Of course the general purpose task runner that both run on does need to support secrets
you conveniently ignored the "CD" part of CI/CD
As mentioned by sibling comments, these two parts are very different. GHA and alternatives are primarily for CI; many people choose to use it solely for CI. For these reasons I did ignore CD.
Hmm, I have long assumed that a perfectly executed CI/CD setup would be based on a generic task runner... But maybe not?
Only the CI part needs to build; it needs little else and it's the only part of a coherent setup that needs to build.
We're iterating towards GHA for CI, AWS CodeBuild for the CD. At least on AWS projects. Mainly because managing IAM permissions to permit the github runner to do everything the deployment wants is an astonishingly large waste of time. But you need a secret to trigger one from the other.
You actually don’t need (long-lived / hard-coded) secrets in this scenario if you use OIDC:
https://docs.github.com/en/actions/how-tos/secure-your-work/...
Technically yes. It depends on whether you consider the account ID to be a secret or not (AWS say "sensitive but not secret" which doesn't help much). But also it can make sense to treat all environment variables as secrets by default just so you don't accidentally end up putting something somewhere that turns out to have been Wrong.
GP is saying that GHA would need zero information about AWS if CodeBuild used a Github token and listened for GHA runs.
That may be true, but it's not what the link describes.
Fair!
And even better can scope assuming an AWS IAM role to a specific branch name & workflow filename so only code/workflows that have been through review have access to CD secrets/prod infra.
IE no prod access by editing the workflow definition and pushing it to a branch.
How do you e.g. validate that a database product works with all the different cloud databases? Every time you change up SQL generation you're going to want to make sure the SQL parses and evaluates as expected on all supported platforms.
Those tests will need creds to access third party database endpoints.
You're missing that the D in CI/CD means deployment; be that packaging on pushing tags and publishing to a registry, or building images, or packaging github releases.
CI is arguable, but how do you intend to do deployments with no secrets?
AWS is great for this. IAM policies can allow IP Addresses or more safely just named EC2 instances. Our deploy server requires nothing.
CircleCI and I believe GHA support injecting signed JWTs you can use to bootstrap identity be it an IAM role or some other platform where you can trust an OIDC issuer
> injecting signed JWTs
How is that not secrets management?
It is. Just wanted to point out these flows are also possible on CI. In my other comment, I think it's more fair to differentiate long lived vs short lived secrets
The secret is held by the metadata server that the CI instance has access to
Or: the deployment service knows the identity of the instance, so its secret is its private key
Or, how PyPI does it: the deployment service coordinates with the trusted CI/CD service to learn the identity of the machine (like its IP address, or a trusted assertion of which repository it’s running on), so the secret is handled in however that out-of-band verification step happens. (PyPI communicates with Github Actions about which pipeline from which repository is doing the deployment, for example)
It’s still just secrets all the way down
> The secret is held by the metadata server that the CI instance has access to
But how does the metadata server know that the CI instance is allowed to access the secret? Especially when the CI/CD system is hosted at a 3rd. party. It needs to present some form of credentials. The CI system may also need permission or credentials for a private repository of packages or artifacts needed in the build process.
For me, a CI/CD system needs two things: Secret management and the ability to run Bash.
Yeah I was confused about that bit too. AWS and GCP's metadata servers know which instances were deployed, so they presumably have some way of verifying the instance's identity out-of-band, such as being tagged by an internal job or machine identifier.
As for deploying from a trusted service without managing credentials, PyPI calls this "trusted publishing": https://docs.pypi.org/trusted-publishers/
From the docs:
1. Certain CI services (like GitHub Actions) are OIDC identity providers, meaning that they can issue short-lived credentials ("OIDC tokens") that a third party can strongly verify came from the CI service (as well as which user, repository, etc. actually executed);
2. Projects on PyPI can be configured to trust a particular configuration on a particular CI service, making that configuration an OIDC publisher for that project;
3. Release automation (like GitHub Actions) can submit an OIDC token to PyPI. The token will be matched against configurations trusted by different projects; if any projects trust the token's configuration, then PyPI will mint a short-lived API token for those projects and return it;
4. The short-lived API token behaves exactly like a normal project-scoped API token, except that it's only valid for 15 minutes from time of creation (enough time for the CI to use it to upload packages).
You have to add your github repository as a "trusted pulbisher" to your PyPI packages.
Honetsly the whole workflow bothers me -- how can PyPI be sure it's talking to github? what if an attacker could mess with PyPI's DNS? -- but it's how it's done.
PyPI is sure that it’s talking to GitHub because it establishes trust in the GitHub’s IdP public keys over HTTPS. I guess you could then question the security of HTTPS, but that seems like a significant rabbit hole to jump down given that OAuth, etc. all depend on the same basic scheme.
It would be good if it could also scan build output like code coverage and test results. But that’s about all it should do.
I keep meaning to write a partially federated CI tool that uses Prometheus for all of its telemetry data but never get around to it. I ended up carving out a couple other things I’d like to be part of the process as a separate app because I was still getting panopticon vibes and some data should just be private.
That is secret management.
Yes, that's what I'm saying. I'm agreeing with your response to amluto.
> But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
There is if you pay for API access, surely?
While good in theory, in practice secrets are used to validate those privileges have been assigned. Even in schemes like metadata servers, you still use a secret.
Pedantically I'd say maybe it's more fair to say they shouldn't have access to long lived secrets and should only use short lived values.
The "I" stands for Integration so it's inevitable CI needs to talk to multiple things--at the very least a git repo which most cases requires a secret to pull.
You might want (or _need_) to sign your binary, for example. Or you might want to trigger a deployment.
Github actually is doing something right here. You can set it up as a trusted identity provider in AWS, and then use Github to assume a role in your AWS account. And from there, you can get access to credentials stored in Secret Manager or SSM.
Yes, their oidc setup was probably their last good feature back when they were actually delivering features back in 2020ish. Everyone else copied it within a few months though.
Who is everyone? As with pretty much every GitHub feature, GitLab had them beat by a few years.
Maybe in some other form, but the current style of injecting an oidc token was definitely in github actions first. Here is the gitlab issue tracking the final bit of it's implementation directly mentioning github: https://gitlab.com/gitlab-org/gitlab/-/issues/356986
This is the more flexible and secure (specific aud) replacement for CI_JOB_JWT which has been there since at least 2017, if not before. Functionally it was exactly the same, a JWT token per pipeline allowing you to authenticate to third parties that support OIDC/JWT Auth.
Yeah I sign my project APKs so people can install them from the action's artefact
Exactly. This workflow step takes a rather important secret and sticks it on a VM where any insufficiently sandboxed step before or after it can exfiltrate it.
GitHub should instead let you store that key as a different type of secret such that a specific workflow step can sign with it. Then a compromised runner VM could possibly sign something that shouldn’t be signed but could not exfiltrate it.
Even better would be to be able to have a policy that the only thing that can be signed is something with a version that matches the immutable release that’s being built.
> Why do CI/CD systems need access to secrets?
Because you need to be able to sign/notarize with private keys and deploy to cloud environments. Both of these require secrets known to the runner.
Because for some reason they use the same system to do releases and sign them and publish them.
I agree 100% with what I think is the key phrase, viz. "the results can change without any modification to your code".
I maintain an R package that is quite stable and is widely used. But every month or so, the GHA on one of the R testing machines will report an error. The messages being quite opaque, I typically spend a half hour trying to see if my code is doing something wrong. And then I simply make a calendar item to recheck it each day for a while. Sure enough, the problems always go away after a few days.
This might be specific to R, though.
To get something of a lockfile you can use the hash of the version you want to pin your dependencies:
> actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744
TFA mentions this option and then goes on at some length to explain that this doesn't help for transitive dependencies, which is how these attacks usually work.
Transitive dependencies?
Yeah, only works if all used Actions would use SHAs too, which is not the case.
Positive example: https://github.com/codecov/codecov-action/blob/96b38e9e60ee6... Negative example: https://github.com/armbian/build/blob/54808ecff253fb71615161...
I've also found many Actions that do other dodgy stuff, like pulling and executing unpinned scripts from external websites, or installing unpinned binaries from GitHub releases. Pinning an Action isn't enough, you have to audit it.
The container manager is horrible.
When you have a multi-platform image the actual per-platforms are usually not tagged. No point.
But that doesn't mean that they are untagged.
So on GitHub Actions when you upload a multi-platform image the per-platform show up in the untagged list. And you can delete them, breaking the multi-platform image, as now it points to blobs that don't exist anymore.
Yep. I'm switching our workflows to instead use regular utilities running inside a Docker container.
This works well for _most_ things. There are some issues with doing docker-in-docker for volume mapping, but they're mostly trivial. We're using taskfiles to run tasks, so I can just rely on it for that. It also has a built-in support for nice output grouping ( https://taskfile.dev/docs/reference/schema#output ) that Github actions can parse.
Pros:
1. Ability to run things in parallel.
2. Ability to run things _locally_ in a completely identical environment.
3. It's actually faster!
4. No vendor lock-in. Offramp to github runners and eventually local runners?
Cons:
It often takes quite a while to understand how actions work when you want to run them in your own environment. For example, how do you get credentials to access the Github Actions cache and then pass them to Docker? Most of documentation just tells: "Use this Github Action and stop worrying your pretty little head about it".
Do you have a write up about this? Actions are great, but my #2 gripe with actions, after the tenuous security posture, is that the default practice is not to run/validate actions locally.
I don't. Will do soon.
I agree that github actions has a horrible syntax and can do much less than travis, appveyor, circle, ... I'd really have to resort to yaml references to do anything elegant. But yaml references aren't elegant at all. But it still is free and vastly outperforms the other free CI's.
I just converted our old parrot travis runners to github actions. There I had constant troubles with travis timeouts of 15m 10 years ago. With the new github actions I can run the full tests (which was not possible with travis) in 3 minutes. About 8x faster hardware.
A lot of the actions people tend to use are just unnecessary. They're simple wrappers around real tools. In those cases, use mise-en-place. It's a single action that installs all relevant tools (and keeps your local dev env in check), and it supports lock files.
This right here - I am migrating all of our GHA to use the mise action. Makes keeping the version of Go, linters, formatters etc. for the project so much easier. Haven't added the mise.lock yet, but on the list. Now getting my small team of devs to try using mise is much harder.
And if not mise, just a Makefile, shell scripts or custom docker images. Then you can run and develop them locally.
GitHub actions has some rough edges around caching, but all the packaging is totally unimportant and best avoided.
I'm assuming the lockfile should be checked into the repo itself, which presents a bootstrapping problem if you have to run an action to create the lockfile in the first place. They may need to build proper support for running actions locally -- there is the third-party https://github.com/nektos/act tool which might be a starting point, but that's mostly designed so you can debug actions without having to repeatedly push and rerun. Probably they'll need a separate mechanism to statically analyze actions without running them.
I committed the project I maintain to GitHub Actions when Actions first came out, and I'm really starting to regret it.
The main problem, which this article touches, is that GHA adds a whole new dimension of dependency treadmill. You now have a new set of upstreams that you have to keep up to date along with your actual deployment upstreams.
renovatebot and dependabot can take care of that. Setting up CI is progress, nothing to regret.
I've not understood the propensity for using yaml for CI pipelines and workflows in general. A decent programming language would be a big improvement.
Why not just build the workflows themselves as docker images? I guess running other docker images in the workflow would then become a problem.
After having to deal with shell rc files causing chaos and unpredictability, I can appreciate your top level code fundamental not being random code but something that is parsable and not subject to the halting problem.
> I've not understood the propensity for using yaml for CI pipelines and workflows in general. A decent programming language would be a big improvement.
Because it's clear to write and read. You don't want your CI/CD logic to end up being spaghetti because a super ninja engineer decided they can do crazy stuff just because they can. Same reason why it's a bad idea to create your infrastructure directly in a programming language (unless creating infrastructure is a core part of your software).
> Why not just build the workflows themselves as docker images? I guess running other docker images in the workflow would then become a problem.
That's how Drone CI handled it. GitLab kind of does the same, where you always start as a docker image, and thus if you have a custom one with an entrypoint, it does whatever you need it to.
IME on a Pulumi for IaC team, writing infra in a real language (TypeScript) is MILES better than- you can do conditions, have typed outputs, etc and not have it be a bastardized imperative YAML mess.
YAML is fine for data, but inevitably stuff like workflows end up tacking on imperative features to a declarative language.
> IME on a Pulumi for IaC team, writing infra in a real language (TypeScript) is MILES better than- you can do conditions, have typed outputs, etc and not have it be a bastardized imperative YAML mess.
You can have conditions and types without having the full flexibility allowing madness of a full language with HCL.
I've thought about using Pulumi many times. Sounds like dagger.io can be a good complement for the workflow use case?
I enjoy it, but I also think my platform team has set it up in the best possible way. YMMV if it’s done differently- I am sometimes uncertain if I love pulumi, or how we do pulumi.
I really really want to use dagger, but I don’t think there’s organizational interest in it.
Dagger does what you're describing: https://dagger.io/
Neat! Sounds like Pulumi for workflows.
Also, using the dagger github action should make the transition easier I suppose: https://github.com/dagger/dagger-for-github
it's wild I can wiz through a ton of code for hours on end but seeing a yaml file for something like a CI pipeline actually makes my brain eject i dunno why. my brain has some sort of proverbial capacity limit with how many different configuration-file looking things I can tolerate in a day, and the prospect of becoming intimately familiar with what is effectively an auto integration presented to me as some sort of config makes me completely unjustifiably butthurt for no reason. have i not suffered enough needless and often times limiting abstractions already
Has anyone built a “deep fork” tool that lets you make a private fork of all dependencies, then modifies their transitive dependencies to point to private forks, and so on? Ideally in a way where updates can be pulled in manually? Seems feasible.
> Has anyone built a “deep fork” tool that lets you make a private fork of all dependencies, then modifies their transitive dependencies to point to private forks, and so on? Ideally in a way where updates can be pulled in manually? Seems feasible.
If you do, please submit a "show HN." I'd love to use it.
I checked out the linked GitHub repo https://github.com/ecosyste-ms/package-manager-resolvers and it appears to be just a README.md that collects summaries of different package managers? How do I know these weren't just LLM-generated?
You don't, but that's the wrong question. How do you know they're accurate?
I feel it's not that controversial to treat LLM output as unreliable without proper verification. So even if the output were accurate, if it's LLM-generated without proper verification I don't trust it.
GitHub has been showing its limitations for quite some time now, and this gives a chance to alternative code forges to rise to prominence.
I hope that Codeberg will become more mainstream for FOSS projects.
I hope another competent player, beside GitLab and Bitbucket, will emerge in the corporate paid space.
Run Nix atop of Actions, minimize the amount of actions you depend on. That works. As a bonus, you now can run your flows locally too.
I have a little launcher for that which helps: https://github.com/7mind/mudyla
> You trust GitHub to give you the right code for a SHA.
The vast majority of users use GitHub-hosted runners. If you don't trust GitHub, you have bigger problems than whether the correct code for an action is downloaded.
I assume that git (not Github) verifies that, if you checkout a hash, the contents of the commit match the hash.
Anyway, software is so complicated that at some level, you need to trust something because it's impossible to personally comprehend and audit all code.
So, you still need to trust git. You still need to trust your OS. You still need to trust the hardware. You just don't have enough minutes in your life to go down through all those levels and understand it well enough to know that there's nothing malicious in there.
I was just writing about how crazy it is to use the third-party ssh tool
https://broderic.blog/post/moving-away-from-netlify/
> Some teams vendor actions into their own repos. zizmor is excellent at scanning workflows and finding security issues. But these are workarounds for a system that lacks the basics.
Harsh given GitHub makes it very easy to setup attestations for Artifact (like build & sbom) provenances.
That said, Zizmor (static analyser for GitHub Actions) with Step Security's Harden Runner (a runtime analyser) [0] pair nicely, even if the latter is a bit of an involved setup.
[0] https://github.com/step-security/harden-runner
> The fix is a lockfile.
Hopefully, SLSA drafts in Hermetic build process as a requirement: https://slsa.dev/spec/v1.2/future-directions
I’d say that GitHub has done an admirable job making attestations more accessible, but that “easy” is still a stretch of a characterization: it’s still not the default, and the error/configuration states are somewhat opaque (e.g. around OIDC permissions, unprivileged triggers, what constitutes a signing identity in a reusable workflow context, etc.). Some of these are latent complexities that GitHub can’t be blamed for, but some are certainly made worse by architectural decisions in GitHub Actions.
The lack of lockfiles is wild. Every other package manager figured this out years ago.
Has anyone been bitten by a breaking change from an action update mid-pipeline?
Mid-pipeline? No, but midday, oh yeah.
See also this excellent video essay by fasterthanlime: GitHub Actions Feels Bad[1].
I'm pretty sure it contains the exact line of it being "deeply confused about being a package manager".
[1]: https://www.youtube.com/watch?v=9qljpi5jiMQ
We are currently using GitHub Actions for all our CI tasks and I hate it. Yes, the marketplace is nice and there are a lot of utility actions which make life easier, but they all come with the issues the post highlights. Additionally, testing Actions locally is a nightmare. I know that act exists but for us it wasn't working most of the time. Also the whole environment management is kinda odd to me and the fact, that when using an environment (which then allows to access secrets set in that environment) it always creates a new deployment is just annoying [1]
I guess the best solution is to just write custom scripts in whatever language one prefers and just call those from the CI runner. Probably missing out on some fancy user interfaces but at least we'd no longer be completely locked into GHA...
[1] https://github.com/orgs/community/discussions/36919
It is concerning that GitHub hosts the majority of open-source software, while actively locking its users into a platform that is based on closed source for eerything except Git itself. This issue with Actions shows how maintaining proprietary software inevitably ends up rather low on the priority list. Adding new features is much more marketable, just like for any other software product. Enshittification ensues.
For those who can still escape the lock-in, this is probably a good occasion to point to Forgejo, an open-source alternative that also has CI actions: https://forgejo.org/2023-02-27-forgejo-actions/ It is used by Codeberg: https://codeberg.org/
On the plus side, Forgejo Action's implementation is still actively improving, where it seems that for GitHub if it's not AI, it's not being touched.
However, as noted in the article, Forgejo's implementation currently has all the same "package manager" problems.
Good point, also to illustrate that open-source is not a panacea. It merely holds a higher potential for certain issues to be fixed/improved than.
I don't know why they basically copied GHA.
AFAIK because "act", the tool to run github actions locally, was there and there was no need to create something else. Also makes it easier for people to switch from github.
Also see https://docs.gitea.com/usage/actions/design
> The core problem is the lack of a lockfile. Every other package manager figured this out decades ago
Well... not Pip!
We're getting there https://pip.pypa.io/en/stable/cli/pip_lock/ !
Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/
Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.
I would recommend you to try uv for this.
Really compelling article! What do folks use instead in 2025?
Has anyone used ArgoCD as a CI pipeline?
microsoft never changes
Complaining about GHA is like complaining that dog poop smells.
Like, what did one expect?
Looks like removing the "meritocracy" doormat, hiring Coraline Ada Ehmke, and changing "master" to "main" paid off in spades!
I'm not sure I follow.
If I write actions/setup-python@v1, I'm expecting the action to run with the v1 tag of that repository. If I rerun it, I expect it to run with the v1 tag of that repository...which I'm aware may not be the same if the tag was updated.
If I instead use actions/setup-python@27b31702a0e7fc50959f5ad993c78deac1bdfc29 then I'm expecting the action to run with that specific commit. And if I run it again it will run with the same commit.
So, whether you choose the tag or the commit depends on whether you trust the repository or not, and if you want automatic updates. The option is there...isn't it?
You specifying the top level hash doesn't do anything to pin transitive dependencies, and as the article points out, transitive dependencies - especially dependencies common to a lot of actions - would be the juciest target for a supply chain attack.
Ah, I see it now. Thanks!
> which I'm aware may not be the same if the tag was updated.
That's the mistake that breaks the following. People don't usually expect that it's an arbitrary modifiable reference, but instead that it's the same version they've picked when they created the file (ie a tag is just a human friendly name for a commit)