It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features. If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level. There is no software development process which reliably produces software that works at scale without doing it small, and medium sized, first, and fixing what goes wrong before you go big.
> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level.
At a large box retail chain (15 states, ~300 stores) I worked on a project to replace the POS system.
The original plan had us getting everything working (Ha!) and then deploying it out to stores and then ending up with the two oddball "stores". The company cafeteria and surplus store were technically stores in that they had all the same setup and processes but were odd.
When the team that I was on was brought into this project, we flipped that around and first deployed to those two several months ahead of the schedule to deploy to the regular stores.
In particular, the surplus store had a few dozen transactions a day. If anything broke, you could do reconciliation by hand. The cafeteria had single register transaction volume that surpassed a surplus store on most any other day. Furthermore, all of its transactions were payroll deductions (swipe your badge rather than credit card or cash). This meant that if anything went wrong there we weren't in trouble with PCI and could debit and credit accounts.
Ultimately, we made our deadline to get things out to stores. We did have one nasty bug that showed up in late October (or was it early November?) with repackaging counts (if a box of 6 was $24 and if purchased as a single item it was $4.50 ... but if you bought 6 single items it was "repackaged" to cost $24 rather than $27) which interacted with a BOGO sale. That bug resulted in absurd receipts with sales and discounts (the receipt showed you spent $10,000 but were discounted $9,976 ... and then the GMs got alerts that the store was not able to make payroll because of a $9,976 discount ... one of the devs pulled an all nighter to fix that one and it got pushed to the stores ).
I shudder to think about what would have happened if we had tried to push the POS system out to customer facing stores where the performance issues in the cafeteria where worked out first or if we had to reconcile transactions to hunt down incorrect tax calculations.
You could have, in principle, implemented the new system to be able to run in "dummy mode" alongside the existing system at regular stores, so that you see that it produces the 'same' results in terms of what the existing system is able to provide.
Which is to say, there is more than one approach to gradual deployment.
Things like the credit card reader (and magnetic ink reader for checks), different input device (sending the barcode scanner two two different systems), keyboard input (completely different screens and keyed entry) would have made those hardware problems also things that needed to be solved.
The old system was a DOS based one where a given set of Fkeys were used to switch between screens on a . Need to do hand entry of a SKU? That was F4 and then type the number. Need to do a search for the description of an item? That was F5. The keyboard was particular to that register setup and used an old school XT (5 pin DIN) plug. The new systems were much more modern linux boxes that used USB plugs. The mag strip reader was flashed to new screens (and the old ones were replaced).
For this situation, it wasn't something that we could send keyboard, scanner, and credit card events to another register.
One of the most important things in there is you don't mix dev and production. The idea of putting a development box next to a production box that runs the same transactions... that just doesn't happen.
Failing a PCI DSS audit means hefty fines and increases of transaction fees (paying 1% more on each transaction done with a credit card can make a $10k/month - $100k/month fine a rounding error) to a "no, you can't process credit cards" which would mean... well... shutting down the company (that wouldn't be a first offense - its still not something you want to have a chat about with accounting about why everything costs 1% more now). Those are things that you don't want to deal with as a developer.
So, no. There is no development configuration in production, or mirroring of a point of sales terminal to another system that's running development code.
Development code doesn't touch other people's money. We had enough side eyes looking at the raw data for our manager's payment card on development systems because only people that banked at that local bank occasionally experienced a problem with their visa check card... https://en.wikipedia.org/wiki/Digital_card#Financial_cards - when it says "generally '^'" it means it can be some other character... and it was... and this wasn't a problem for most people, but it turned out that the non-standard separator (that we only found after reading the card's raw data) and a space in the surname would result in misparsing of the track and giving an error - but none of our other cards used a separator that didn't match the "generally").
So, being able to generate real production load (in the cafeteria) without using Visa, Mastercard, etc... was important. As was being able to fall back to using the nearly antique credit card imprinter ( https://en.wikipedia.org/wiki/Credit_card_imprinter ) for the store that was lucky to get a dozen transactions a day.
> So, no. There is no development configuration in production, or mirroring of a point of sales terminal to another system that's running development code.
This is a misreading of the suggestion, I think. My reading of the suggestion is to run a production "dry run" parallel code path, which you can reconcile with the existing system's work for a period of time, before you cut over.
This is not an issue precluded by PCI; it is exactly the method a team I led used to verify a rewrite of and migration to a "new system" handling over a billion dollars of recurring billing transactions annually: write the new thing with all your normal testing etc, then deploy it alongside in a "just tell us what you would do" mode, then verify its operation for specific case classes and then roll progressively over to using it for real.
edit: I don't mean to suggest this is a trivial thing to do, especially in the context you mentioned with many elements of hardware and likely odd deployment of updates, etc.
Our reading of PCI DSS was that there was no development code in a production build. Having a --dry-run flag would have meant doing that.
You could do "here is the list of skus for transaction 12120112340112345 - run this through the system and see what you get" on our dev boxes hooked up to QA store 2 (and an old device in the lab hooked up to QA store 1). That's not a problem.
Sending the scanner reads to the current production and a dev box in production would have been a hardware challenge. Not completely insurmountable but very difficult.
Sending the keyboard entry to both devices would be a problem. The screens were different and you can hand enter credit card numbers. So keyboard entry is potentially PCI data.
The backend store server would also have been difficult. There were updates to the store server (QA store 1 vs QA store 2 running simultaneously) that were needed too.
This wasn't something that we could progressively roll out to a store. When a store was to get the new terminals, they got a new hardware box, ingenicos were swapped with epson, old epson were replaced with new (same device but the screens had to be changed to match a different workflow - they were reprogrammable, but that was something that stores didn't have the setup to do), and a new build was pushed to the store server. You couldn't run register 1 with the old device and register 2 with a new one.
Fetching a list of SKUs, printing up a page of barcodes and running it was something we could do (and did) in the office. Trying to run a new POS system in a non-production mode next to production and mirroring it (with reconciling end of day runs) wasn't feasible for hardware, software, and PCI reasons that were exacerbated by the hardware and software issues.
Online this is potentially easier to do with sending a shopping cart to two different price calculators and logging if the new one matches the old one. With a POS terminal, this would be more akin to hooking the same keyboard and mouse up to a windows machine and a linux machine. The Windows machine is running MS Word and the linux is running Open office and checking to see that after five minutes of use of the windows machine that the Linux machine had the same text entered into OpenOffice. Of course they aren't - the keyboard entry commands are different, the windows are different sizes, the menus have things in different places in different drop downs... similarly, trying to do this with the two POS systems would be a challenge. And to top it off sometimes the digits typed are hand keyed credit card numbers when the MSR couldn't get a read - and make sure those don't show up on the linux machine.
I do realize this is reminiscent of business giving a poorly spec'ed thing and each time someone says "what about..." they come up with another reason it wouldn't work. This was a system that I worked on for a long while (a decade and a half ago) and could spend hours drawing and explaining diagrams of system architecture and issues that we had. Anecdotes of how something worked in a 4M Sloc system are inherently incomplete.
Neat! Yeah, that's a pretty complex context and I completely see what you mean about the new hardware being part of the rollout and necessarily meaning that you can't just run both systems. My comment is more of a strategy for just a backend or online processing system change than a physical brick and mortar swap out.
In my note about misreading the suggestion, I was thinking generally. I do believe that there is no reason from a PCI perspective why a given production system cannot process a transaction live and also in a dry mode on a new code path that's being verified, but if the difference isn't just code paths on a device, and instead involves hardware and process changes, your point about needing to deploy a dev box and that being a PCI issue totally makes sense, plus the bit about it being a bad test anyway because of the differences in actions taken or outputs.
The example you gave originally, of shipping to the lower stake exceptional stores first and then working out issues with them before you tried to scale out to everywhere, sounded to me like a very solid approach to mitigating risk while shipping early.
The original register was a custom written C program running in DOS. It was getting harder and harder to find C programmers. The consultancy that had part of the maintenance contract with it was also having that difficulty and between raising the rates and deprioritizing the work items because their senior people (the ones who still knew how to sling C and fit it into computers with 4 MB of memory that you couldn't get replacement parts for anymore) were on other (higher paying) contracts.
So the company I worked at made the decision to switch from that program to a new one. They bought and licensed the full source to a Java POS system (and I've seen the same interface at other big retail companies too) and replace all the hardware in all the stores... ballpark 5000 POS systems.
The professional services consultancy was originally brought in (I recall it being underway when I started at there in 2010). They missed deadlines and updates and I'm sure legal got in there with failure to deliver on contract. I think it was late 2011 that the company pulled the top devs from each team and set us to working on making this ready in all stores by October 2012 (side note: tossing two senior devs from four different teams into a new team results in some challenging personality situations). And that's when we (the devs) flipped the schedule around and instead of March 2013 for the cafeteria and surplus store (because they were the odd ones), we were going to get them in place in March of 2012 so that we could have low risk production environments while we worked out issues (so many race conditions and graphical event issues hanging with old school AWT).
---
... personality clash memory... it was on some point of architecture and code and our voices were getting louder. Bullpen work environment, (a bunch of unsaid backstory here) but the director was in the cube on the other side of the bullpen from us. The director "suggested" that we take our discussion to a meeting room... so we packed up a computer (we needed it to talk about code), all of the POS devices that we needed, put it on a cart, pushed the cart down the hall into a free conference room (there were two conference rooms on that floor - no, this wasn't a building designed for development teams) and set up and went back to loudly discussing. However, we didn't schedule or reserve the room... and the director that kicked us out of the bullpen had reserved the room that we had been kicked into shortly after we got there. "We're still discussing the topic, that will probably be another 5-10 minutes from now... and it will take us another 5 minutes pack the computer back up and take it back to the bullpen. Your cube with extra chairs in it should be available for your meeting and it's quiet there now without our discussions going on."
Dev: "Could we pull an export of relevant historical data and get some time to write code to safely anonymize that, and stand up a parallel production system using just the anonymized data and replicate our deploy there, so we can safely test on real-ish stuff at scale?"
Lead: "I'll think about it. In the meantime, please just build the features I asked you to. We gotta hustle on this one."
I'm not arguing with this hypothetical exchange that it's infeasible or even a bad idea to do exactly what you suggested, but attempting to justify an upfront engineering cost that isn't directly finishing the job is a difficult thing to win in most contexts.
It's very common to use identical systems but anonymised data shipped back to test environments in such cases. There are certain test card numbers that always fail or always succeed against otherwise-real infrastructure on the card provider's side.
Absolutely, I agree that it's a useful pattern. I've personally typed 4111 1111 1111 1111 into a stripe form more times than I want to even think about.
My point above was that it's not necessarily easy to convince the operators of a business that it's a justifiable engineering expense to set up a new "prodlike but with anonymized data" environment from scratch, because it's not a trivial thing to make and maintain.
I do think it's pretty easy to convince operators of a business to adopt the other strategy suggested in a sibling thread: run a dry mode parallel code path, verify its results, and cut over when you have confidence. This shouldn't really be an alternative to a test environment, but they can both achieve similar stuff.
> I do think it's pretty easy to convince operators of a business to adopt the other strategy suggested in a sibling thread: run a dry mode parallel code path, verify its results, and cut over when you have confidence. This shouldn't really be an alternative to a test environment, but they can both achieve similar stuff.
I agree - it's a nice low-risk way of doing things.
It is as low risk as trying to use Windows and Microsoft Word with a keyboard and mouse mirrored to a Linux machine running Open Office and expecting the same results.
You can't run the two systems side by side - different screens, different keyboard entry... and some of the keyboard entry can't touch the other system.
And this is assuming you can put a dry path into the production system. If the answer is "no", then you're putting a dev environment into a production environment... and that's certainly a "no".
We had test environments and we had a lab were we had two rows of systems where the two systems sat back to back and each row was hooked up to a different test store (not feasible in a production store environment).
surely you have logs from the production systems? just gather the logs and run them through the dev box. verify the end result matches between the two. You dont actually need the dev box to sit next to the production system.
You cannot, under any circumstances, keep a real card # and use it as test data. I think that's where this conversation is getting hung up, because the idea of running a transaction through prod and them doing the same in test to see if it matches isn't something you can do. I mean, of course you can throw the prices and UPCs at the new system and verify that the new system's math matches the old system, but that's only the most basic function of a POS system. Testing a transaction from end-to-end would have to be done with synthetic data in an isolated environment, and I'll assume that's what OP is trying to articulate.
There's all this stuff but I remember when I was a Junior freelancer I was analysing a calendar availability sync script for a small holiday bookings company (not the big one). The hosts would have a publicly accessible Google Calendar with their bookings on which the script I was fixing would pull from.
Turns out, most of the host stored their customers long cards + expiry etc in the comment field of the booking.
the reproduction is always fake to some extent, that does not matter, the point is to do as good a job as you can.
for example you can have a fake transaction server with the credit card numbers made up and mapped to fake accounts that always have enough money, unless the records show they did not.
I've also worked with payment processors a lot.
The ones I've used have test environments where you can fake payments, and some of them (Adyen does this) even give you actual test debit and credit cards, with real IBAN's and stuff like that.
Don't know anything about the OP's system, other than "POS" but the bug they experienced - and (maybe?) all the typical integration stuff like inventory management - is very complex and wouldn't manifest itself in a payment processing failure. I'm doubtful that anyone's production inventory or accounting systems allow for "fake" transactions that can be validated by an e2e test
It was a linux running on (year appropriate) https://www.hp.com/us-en/solutions/pos-systems-products.html... - and add on all the peripherals. The POS software was standalone-ish (you could, in theory, hook it up to a generator to a register and the primary store server and process cash, paper check, and likely store branded credit cards)... it wouldn't be pleasant, but it could.
The logic for discounts and sales and taxes (and if an item had sales tax in that jurisdiction) was all on register. The store server logged the transaction and handled inventory and price lookup, but didn't do price (sale, taxes) calculations itself.
At some point you start to get far away from reality though. If the cards have fake numbers then other auth information is also incorrect - e.g. the CVC won't match, the PIN won't either (depending on the format in use maybe). You can fake all that stuff too but now how much of that system are you really testing?
I mean in his example the discount bug they ran into wouldn’t have needed any card numbers that could have been discovered with fake/cloned transactions that contained no customer detail in this case it seems it would have been best to test the payment processing in personal at a single store and then also testing with sales logs from multiple other locations
yep, it sounds like the first implementation step really should have been to gather a large test set of data and develop the system with that in mind after understanding the test data, starting with making tests from the test data.
They explained the scenario though and it seems like a combination of rarer edge cases. It's great to think your awesome dev team and QA would have collected test data representing this ahead of time, but surely you've all been caught by this? I know I have; that's why we don't have flawless systems at launch.
OP said elsewhere that the specific discount that produced the $10M sales + refund combo was only active at five stores. That's the sort of edge case that you can't count on being in the training data. I'm as big a fan of real-data testing as anyone, but there's always something not represented.
Payment card industry. Credit card info (personal user data, etc). There’s a whole boatload of data privacy issues you run into if you mess that up. So compliance is essential.
From my experience a lot of the hardest problems in this space are either 1. edge cases or 2. integration-related and that makes them hard to validate across systems or draw boundaries around what's in the dummy mode. This type of parallel, live, full system integration test is hard to pull off.
In 1997 I was working on an integration between AOL and Circuit City (ha I outlived them both) to enable free AOL accounts for people buying PCs or some such; about a week before launch I changed the data returned from encoding spaces as "+" to "%20" and broke their integration (perl script). Very upsetting for them, and I felt bad.
I also had some weird bug when we started registrations from German accounts and I didn't handle umlauts (or UTF-16 with nuls in the string) in passwords properly.
>> We did have one nasty bug that showed up in late October (or was it early November?)
Having worked in Ecommerce & payment processing, where this weekend is treated like the Superbowl, birth of your first child and wedding day all rolled into one, a nasty POS bug at this time of year would be incredibly stressful!
After thinking back on it, I think this was earlyish October. The code hadn't frozen yet, but it was getting increasingly difficult. We were in the "this was deployed to about 1/3 of the stores - all within an 8 drive of the general office". The go/no-go decision for the rest of the stores in October was coming up (and people were reviewing backout procedures for those 100). One of the awkward parts was that marketing had a Black Friday sale that they really wanted to do (buy X, buy Y, get Z half price) that the old registers couldn't support. They wanted to get a "is this going?" so they could start printing the advertising flyers.
Incidentally, this bug resurfaced for the next five years in a different incarnation. Because it had that this department (it was with one sku) had sold $10M this week in October, the running average sales target the next year was MEAN($24k, $25k, $26k, $25k, $10M) ... and the department heads were doing a "you want me to sell how much?!"
This bug had only affected... maybe five stores (still maybe five too many). We were in the "this is the last™ build before all store deployment next week" territory. It did mess with that a bit too as the boxed up registers came with an additional step of "make sure to reboot the register after doing initial confirmation."
The setup teams had a pallet of computers delivered to the stores that were supposed to be "remove the old registers, put these registers in, swap mag strip readers, take that laptop there and run this software to configure the devices on each register." However, the build that the registers had was the buggy build. While that build likely wouldn't hit that bug (it required a particular sale to be active which was only at a few stores and had ended) it still was another step that they had to follow.
Aside: For all its clunkiness, Java Web Start was neat. In particular, it meant that instead of trying to push software to 5k registers (how do you push to registers that are powered off?), instead we'd push to 300 stores and from there JWS would check for an update each time it started up ( https://docs.oracle.com/javase/8/docs/technotes/guides/javaw... ). So instead of pushing to 5k registers, we'd have it pull from 'posupdate' on the local network when it rebooted.
There is no solution because these projects are not failing because of technical reasons.
They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions.
Then you have "important people" who want to be important and want to have an opinion on font size and that some button should be 12px to the right because they are "important" it doesn't matter for the project but they have to assert the dominance.
You have 2 or 3 companies working on a project? Great! now they will be throwing stuff over the fence to limit their own cost and blame others while trying to get away with as least work done cashing most money possible.
That is how sausage is made. Coming up with "reasonable approach" is not the solution because as soon as you get different suppliers, different departments you end up with power/money struggle.
> They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions.
Not (necessarily) wrong, but if you start small, Important People may not want to bother with something that is Unimportant and may leave things alone so something useful and working can get going. If you starting with an Important project then Important People will start circling it right away.
Even starting small isn't a surefire way to avoid that problem. They'll just show up once the thing gets big enough.
Witness how the web was once a funny little collection of nerds sharing stuff with each other. But once it got big enough that you could start making money off it, the important people showed up and started taking over. The web still has those odd little corners, but it's largely the domain of a small number of giant powerful corporations.
I don't think there is a silver bullet for dealing with egomaniacs who want infinite power. They seem to be a part of the human condition and dealing with them is part of the ticket price for having a society.
Dunno if you listen to Ezra Klein but he had an anthropologist on once who described this tribe of humans who when someone came back having bagged big game, they had to run a gauntlet of everyone else downplaying their accomplishment like "that's not that big, your father caught bigger", and "maybe one day you'll bring down an adult deer" etc. The whole idea was like, egomaniacs are pretty bad, and they had a cultural defense against it.
I often think a weakness of liberal, western society is the insistence on rationality, that like the hunter in question could just easily put their abilities and accomplishments alongside those of others and get a pretty accurate picture. This is super untrue; we need systems to guard against our frailties, but we can't admit we have them, so we keep falling into the same ditches.
> Dunno if you listen to Ezra Klein but he had an anthropologist on once […]
Klein wrote a book a few years ago on the topic of polarization, human's inclination towards clan/tribal thinking, and how it manifests itself in (US) politics:
I guess for me important point is that it is not technical issue and we already have all technical tools/processes to do really big software projects.
Even if people dislike scrum, find Git complicated and don’t want to open up JIRA - these tools are not the problem, these tools help building loads of working software.
We as software engineers with devops can deliver great and complex projects and build great systems. Lots of businesses people don’t even understand how much in control we can be of the environments and code.
Yet developers/IT is there to be blamed. Like we should be ashamed, Uncle Bob will give lectures “how developers should be more professional”.
Yet I always find business people who are like children in the corn field.
With small difference business/sales guys are pushy and walk over engineering guys and engineers bend over and take the blame and business guys can always say “those IT kids playing with toys instead of doing real job”.
Political corruption is like environmental radiation: a viable fix is never 'just get rid of political corruption'*. It's an environmental constant that needs to be handled by an effective approach.
That said, parent's size- and scope-iterative approach also helps with corruption, because corruption metastasizes in the time between {specification} and {deliverable}.
Shrink that, by tying incremental payments to working systems at smaller scales, and you shrink the blast radius for failure.
That said, there are myriad other problems the approach creates (encouraging architectures that won't scale to the final system, promoting duct taped features on top of an existing system, vendor-to-vendor transitions if the system builder changes, etc).
But on the whole, the pros outweigh the cons... for projects controlled by a political process (either public or private).
That's why military procurement has essentially landed on spiral development (i.e. iterative demonstrated risk burn-down) as a meta-framework.
* Limit political corruption, to the extent possible in a cost efficient manner, sure
That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen.
Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.
> Gradual growth inevitably results in loads of technical debt.
Why is this stated as though it's some de facto software law? The argument is not whether it's possible to waterfall a massive software system. It clearly is possible, but the failure ratios have historically been sufficiently uncomfortable to give rise to entirely different (and evidently more successful) project development philosophies, especially when promoters were more sensitive to the massive sums involved (which in my opinion also helps explains why so many wasteful government examples). The lean startup did not appear in a vacuum. Do things that don't scale did not become a motto in these parts without reason. In case some are still confused about the historical purpose of these benign sounding advices, no, they weren't originally addressed at entrepreneurs aiming to run "lifestyle" businesses.
I think the logic is that good code is code which is maintainable and modifyable; bad code is difficult to change safely. Over time, all code is changed until it is bad code and cannot be changed more. So overtime most code is bad code which is scary to touch.
software is unique field where project can be a problem that no matter how much money you throw, there is something that we can "improve" or make it better
that's why we start something small, a scope if you want call it that way
of course start something small or dare I call it simpler would result in more technical debt because that things its not designed with scale in mind because back to the first point
Try as you might, you cannot fight entropy eternally, as mistakes in this fight will accumulate and overpower you. It's the natural process of aging we see in every lifeform.
The way life continues on despite this law is through reproduction. If you bud off independent organisms, an ecosystem can gain "eternal" life.
The cost is that you must devote much of your energy to effective reproduction.
In software, this means embracing rewrites. The people who push against rewrites and claim they're not necessary are just as delusional as those who think they can live forever.
Software is a component of a product, if not the product itself. Treating software like a product, besides being the underlying truth, also means it makes sense to manage it like one.
Technical debt isn’t usually the problem people think it is. When it does become a problem, it’s best to think of it in product-like terms. Does it make the product less useful for its intended purpose? Does it make maintenance or repair inconvenient or costly? Or does it make it more difficult or even impossible to add competitive features or improvements? Taking a product evaluation approach to the question can help you figure out what the right response is. Sometimes it’s no response at all.
The discussion is not about the product where you can just remove the stuff. The thread was testing in small setting and then moving to oddball setting. If it is required to cover oddball settings, it makes sense to know and plan for oddball setting.
Designing or intending a system to be used at massive scale is not the same as building and deploying it so that it only initially runs at that massive scale.
That's just a recipe for disaster, "We don't even know if we can handle 100 users, let's now force 1 million people to use the system simultaneously." Even WhatsApp couldn't handle hundreds of millions of users on the day it was first released, nor did it attempt to. You build out slowly and make sure things work, at least if you're competent and sane.
Sure, but if you did a good job, the gradual deployment can go relatively quickly and smoothly, which is how $FAANG roll out new features and products to very large audiences. The actual rollout is usually a bit of an implementation detail of what first needed to be architected to handle that larger scale.
The issue with FAANG is that they already have the infrastructure to make these large scale deployments. So any new system - by necessity - needs to conform to that large scale architecture.
The other nice thing about FAANG is that almost nothing they do is actually necessary. If Facebook rolls out a new feature and breaks something for a few hours, it doesn't actually matter. It's harder to move fast and break things if you're, say, a bank, and every minute of downtime is a minute where your customers can't access their money. Enough minutes go by and you may have a very, very expensive crisis on your hands.
Replying to myself in sibling: except maybe people paying for ads, which is more of a faith based action; it's well known a lot of ad traffic is fraudulent, but not which traffic. So if you pay for ads, who can tell what happened.
You get certain big pieces correct maybe but you’d be surprised how many mistakes get made. For example, I had designed the billing system for a large distributed product that the engineer ended up implementing not as described in the spec which fell down fairly quickly with even a modicum of growth.
Well, Google got good at large scale rollouts, because they are doing large scale rollouts all the time. _And_ most of the time, the system they are rolling out is a small iteration from the last system they rolled out: the new GMail servers look almost exactly like the last GMail servers, but they have on extra feature flag you can turn on (and which is disabled by default) or have one bug fixed.
That's a very different challenge from rolling out a brand new system once.
No but whatsapp was built by 2 guys that had previously worked at Yahoo, and they picked a very strong tech for the backend: erlang.
So while they probably didn't bother scaling the service to millions in the first version, they 1) knew what it would take, 2) chose already from the ground up a good technology to have a smoother transition to your "X millions users". The step "X millions to XYZ millions and then billions" required other things too.
At least they didn't have to write a php-to-C++ compiler for Php like Facebook had, given the initial design choice of Mark Zuckeberg, which shows exactly what it means to begin something already with the right tool and ideas in mind.
They took already existing protocol (XMPP) and already existing implementation in Erlang (ejabberd)—there weren't many alternatives at the time really.
Did they succeed because of Erlang or in spite of Erlang? We can't draw any reliable conclusions from a single data point. Maybe a different platform would have worked even better?
Erlang is uniquely suited to chat systems out of the box in a way that most other ecosystems aren't. Lightweight green threads via the BEAM vm, process scheduler so concurrent out of the box, immutable data structures, message passing as communication between processes.
There's nothing unique about Erlang. I have nothing against it but other companies have built messaging systems using other platforms that work as well or better than WhatsApp.
Yeah - the technology used is a seperate concern to their abilities as users (developers) of that technology and the effectiveness at handling the scale.
I, for example, have always said that I am more than capable of writing code in C that is several orders of magnitude SLOWER than what I could write in.. say Python.
My skillset would never be used as an example of the value of C for whatever
> Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts.
This isn't technical debt, necessarily. Technical debt is a specific thing. You probably mean "an underlying design that doesn't perfectly map to what ended up being the requirements". But then the world moves on (what if a regulation is added that ruins your perfect structure anyway?) and you can't just wish for perfect requirements. Or not in software that interacts directly with the real world, anyway.
Gradual growth =/= many tacked on features. Many tacked on features =/= technical debt. Technical debt =/= "everybody is afraid and frozen." Those are merely often correlated, but not required.
Whatsapp is a terrible example because it's barely a product; Whatsapp is mostly a free offering of goodwill riding on the back of actual products like Facebook Ads. A great example would be a product like Salesforce, SAP, or Microsoft Dynamics. Those products are forced to grow and change and adapt and scale, to massive numbers doing tons of work, all while being actual products and being software systems. I think such products act as stark rebukes of what you've described.
There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism.
Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine.
The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."
The dominant factor is: there is a human who understands the entire system.
That is vastly easier to achieve by making a small, successful system, which gets buy in from both users and builders to the extent that the former pay sufficient money for the latter to be invested in understanding the entire system and then growing it and keeping up with the changes.
Occasionally a moon shot program can overcome all of that inertia, but the “90% of all projects fail” is definitely overrepresented in large projects. And the Precautionary Principle says you shouldn’t because the consequences are so high.
This works for Clojure, git and even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it.
But these things are meant to be used by technical people.
The non-technical people I know might want to use Linux but stay on Windows or choose Mac OS because it's more straightforward. I use Windows+WSL at work even though I would like to use a native Linux distribution.
I know someone who created a MUD game (text online game) and said to him I wanted to make one with a browser client. He said something we could translate as "Good, you can have all the newbies." Not only was he right that a MUD should be played with a MUD client like tintin++, but making a good browser client is harder than it seems and that's time not spent making content for the game or improving the engine.
My point is that he was un uncomprimising person who refused adding layers to a project because they would come at a cost which isn't only time or dollars but also things like motivation and focus.
You’re conflating “knows the system” with benevolent dictator. It’s not the same. It’s down to whether in a planning or brainstorming session, there is anyone who can say that a plan won’t work or if there’s a better one.
Also it doesn’t have to be singular. You need at least one, in case that person leaves or becomes problematic. That dictator doesn’t always remain benevolent and they can hold a project hostage if they don’t like something that everyone else wants.
You will never get to the moon by making a faster and faster bus.
I see a lot of software with that initial small scale "baked into it" at every level of its design, from the database engine choice, schema, concurrency handling, internal architecture, and even the form design and layout.
The best-engineered software I've seen (and written) always started at the maximum scale, with at least a plan for handling future feature extensions.
As a random example, the CommVault backup software was developed in AT&T to deal with their enormous distributed scale, and it was the only decently scalable backup software I had ever used. It was a serious challenge with its competitors to run a mere report of last night's backup job status!
I also see a lot of "started small, grew too big" software make hundreds of silly little mistakes throughout, such as using drop-down controls for selecting users or groups. Works great for that mom & pop corner store customer with half a dozen accounts, fails miserably at orgs with half a million. Ripping that out and fixing it can be a decidedly non-trivial piece of work.
Similarly, cardinality in the database schema has really irritating exceptions that only turn up at the million or billion row scale and can be obscenely difficult to fix later. An example I'm familiar with is that the ISBN codes used to "uniquely" identify books are almost, but not quite unique. There are a handful of duplicates, and yes, they turn up in real libraries. This means that if you used these as a primary key somewhere... bzzt... start over from the beginning with something else!
There is no way to prepare for this if you start with indexing the book on your own bookshelf. Whatever you cook up will fail at scale and will need a rethink.
Counterpoint: the idea that your project will be the one to scale up to the millions of users/requests/etc is hubris. Odds are, your project won't scale past a scale of 10,000 to 100,000. Designing every project to scale to the millions from the beginning often leads to overengineering, adding needless complexity when a simpler solution would have worked better.
Naturally, that advice doesn't hold if you know ahead of time that the project is going to be deployed at massive scale. In which case, go ahead and implement your database replication, load balancing, and failover from the start. But if you're designing an app for internal use at your company of 500, well, feel free to just use SQLite as your database. You won't ever run into the problems of scale in this app, and single-file databases have unique advantages when your scale is small.
Basically: know when huge scale is likely, and when it's immensely UNlikely. Design accordingly.
> Odds are, your project won't scale past a scale of 10,000 to 100,000.
That may be a self-fulfilling prophecy.
I agree in general that most apps don't need fancy scaling features, but apps that can't scale... won't... and hence "don't need scaling features".
> You won't ever run into the problems of scale in this app, and single-file databases have unique advantages when your scale is small.
I saw a customer start off with essentially a single small warehouse selling I dunno... widgets or something... and then the corporation grew and grew to a multi-national shipping and logistic corporation. They were saddled with an obscure proprietary database that worked like SQLite and had incredibly difficult to overcome technical challenges. They couldn't just migrate off, because that would have needed a massive many-year long total rewrite of their app.
For one performance issue we were seriously trying to convince them to use phase-change cooling on frequency-optimized server CPUs like a gamer overclocking their rig because that was the only way to eke out just enough performance to ensure their overnight backups didn't run into the morning busy time.
That's just not an issue with SQL Server or any similar standard client-server database engine.
I think part of that thinking though is that if you do basic stuff like use a standard database engine or don't go too off the beaten path if that's what you need, it tends to be that you get the ultimately needed scale for basically free.
This is a lot of times what I see the "don't build for huge scale" to be. It's not necessarily "be proud of O(n^2) algorithms". Rather it's more "use Postgres instead of some hyperscale sharded database when you only have 10 million users" because the alternative tends to miss the forest (and oftentimes the scale, ironically) for the trees
Yes, but also I've found that using a decently scalable engine is insufficient for a good outcome without the scaled data.
The best software I've written always had > 10 GB of existing data to work with from day one. So for example the "customers" table didn't have one sample entry, it had one million real entries. The "products" table had a real history of product recalls, renames, category changes over time, special one-off products for big customers, etc...
That's how you find out the reality of the business instead of some idealised textbook scenario.
Things like: Oh, actually, 99% of our product SKUs are one-off specials, but 99% of the user interactivity and sales volume is with the generic off-the-shelf 1% of them, so the UI has to cater for this and database table needs a "tag" on these so that they can be filtered. Then, it turns out the filtering 10 million products down to 100K has non-trivial performance issues when paging through the list. Or even worse, 50% of the products are secret because their mere existence or their name is "insider info" that we don't want our own staff to see. Did I say "staff"? I meant subcontractors, partners, and resellers, all with their own access rules and column-level data masking that needs to be consistent across dozens of tables. Okay, let's start going down the rabbithole of column naming conventions and metadata...
You can't predict that stuff in a vacuum, no human has the foresight needed to "ask the right questions" to figure this all out through workshops or whatever. The ground-truth reality is the best thing, especially up-front during the early phases of development.
A lot of the above is actually easy to implement as a software developer, but hard to change half-way-through a project.
While I think this is good advice in general, I don’t think your statement that “there is no process to create scalable software” holds true.
The uk gov development service reliably implements huge systems over and over again, and those systems go out to tens of millions from day 1. As a rule of thumb, the parts of the uk govt digital suite that suck are the parts the development service haven’t been assigned to yet.
The Swift banking org launches reliable features to hundreds of millions of users.
There’s honestly loads of instances of organisations reliably implementing robust and scalable software without starting with tens of users.
The uk government development service as you call it is not a service. It’s more of a declaration of process that is adhered to across diverse departments and organisations that make up the government. It’s usually small teams that are responsible for exploring what a service is or needs and then implementing it. They are able to deliver decent services because they start small, design and user test iteratively and only when there is a really good understanding of what’s being delivered do they scale out.
The technology is the easy bit.
The UK Gov has many service and process docs [1]. It started out that way but has grown rapidly and changed. Including a library for authentication, frontend templates and libraries, custom docker images.
UK GDS is great, but the point there is that they're a crack team of project managers.
People complain about junior developers who pass a hiring screen and then can't write a single line of code. The equivalent exists for both project management and management in general, except it's much harder to spot in advance. Plus there's simply a lot of bad doctrine and "vibes management" going on.
("Vibes management": you give a prompt to your employees vaguely describing a desired outcome and then keep trying to correct it into what you actually wanted)
> and those systems go out to tens of millions from day 1
I like GDS (I even interviewed with them once and saw their dev process etc) but this isn't a great example. Technically GDS services have millions of users across decades, but people e.g. aren't constantly applying for new passports every day.
A much better example I think is Facebook's rollout of Messenger, which scaled to billions of actual users on day 1 with no issues. They did it by shipping the code early in the Facebook app, and getting it to send test messages to other apps until the infra held, and then they released Messenger after that. Great test strategy.
GDS's budget is about £90 million a year or something. There are many contracts that are still spent on digital, for example PA consulting for £60 million (over a few years) which do a lot of the gov.uk home-office stuff, and their fresh grads they hire cost more to the government than GDS's most senior staff...
SWIFT? Hold my beer. SWIFT did not launch anything substantial since its startup days in early 70-ies.
Moreover, their core tech did not evolve that far from that era, and the 70-ies tech bros are still there through their progeniture.
Here's an anecdote: The first messaging system built by SWIFT was text-based, somewhat similar to ASN.1.
The next one used XML, as it was the fad of the day. Unfortunately, neither SWIFT nor the banks could handle 2-3 orders of magnitude increase in payload size in their ancient systems. Yes, as engineers, you would think compressing XML would solve the problem and you would by right. Moreover, XML Infoset already existed, and it defined compression as a function of the XML Schema, so it was somewhat more deterministic even though not more efficient than LZMA.
But the suits decided differently. At one of the SIBOS conferences they abbreviate XML tags, and did it literally on paper and without thinking about back-and-forth translation, dupes, etc.
And this is how we landed with ISO20022 abberviations that we all know and love: Ccy for Currency, Pmt for Payment, Dt for Date, etc.
Is it? No auditor will read binary, so you already need a preprocessing step to get it to a readable format. And if you're already preprocessing then adding a decompression step is like 2 lines tops.
> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
Came here to say this. I still think that Linus Torvalds has the most profound advice to building a large, highly successful software system:
"Nobody should start to undertake a large project. You start with a small trivial project, and you should never expect it to get large. If you do, you'll just overdesign and generally think it is more important than it likely is at that stage. Or worse, you might be scared away by the sheer size of the work you envision. So start small, and think about the details. Don't think about some big picture and fancy design. If it doesn't solve some fairly immediate need, it's almost certainly over-designed. And don't expect people to jump in and help you. That's not how these things work. You need to get something half-way useful first, and then others will say "hey, that almost works for me", and they'll get involved in the project."
I don't think this applies in any way to companies contracted to build a massive system for a government with a clear need. Linus is talking about growing a greenfield open-source project, which may or may not ever be used by anyone.
In contrast, if your purpose is "we need to manage our country's accounting without pen and paper", that's a clear need for a massive system. Starting work on this by designing a system that can solve accounting for a small firm is not the right way to go. Instead, you have to design with the end-goal in mind, since that's what you were paid for. But, you don't launch your system to the entire country at once: you first use this system designed for a country in a small shop, to make sure it actually handles the small scale well, before gradually rolling out to more and more people.
There's your problem. The needs are never clear, not on massive systems. Governments will write a spec, companies will read the spec, offer to implement it as written, knowing full well that it won't work. Then they charge exorbitant fees to modify the system after launch, so that it will actually full fill business needs.
The Danish government is famous for sucking at buying massive IT systems.
* Specs for new tax system: 6000 page, tax laws not included. That's basically impossible to implement and it predictably failed. The version that worked: Implement just the basics to collect TV license fees. The build from there.
* System to calculate the value of people home, I think we're at round five (rumors has it that one system worked, but was scrapped because it showed that most home are massively overvalued and it do terrible things to the tax collection in the municipalities).
* New case management system for the police, failed, development never restarted. One suggested solution was to have the police hire a handful of the best developers in the country and have them produce smaller deliverable over a number of year. The money wasted could have funded 10 world class developers for ~30-50 years.
Building those systems is a long term project, and you have to start small with a minimum number of functions, scope creep on those initial use cases often kills these kinds of projects.
No Linus Torvalds would stand against people in projects from article, he would slam the door and quit.
Those projects that author pointed out are basically political horror stories. I can imagine how dozens of people wanted to have a cut on money in those projects or wanted to push things because “they are important people”.
There is nothing you can do technically to save such projects and it is NOT an IT failure.
A bad API can constrain your implementation and often can't be changed once it's in use by loads of users. APIs should be right from day one if possible.
I would add the nuance that the possibility of controlled migration from one versioned API to another should be right from day one, not necessarily the first API version.
> I'm afraid that the solution is: build something small, and use it in production before you add more features.
Gall's Law:
> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[8]
While I like the "start small and expand" strategy better than the "big project upfront", this trades project size for project length and often that is no better:
- It gives outside leadership types many more opportunities to add requirements later. This is nice is they are things missed in the original design, but it can also lead to massive scope creep.
- A big enough project that gets done the "start small and expand" way can easily grow into a decade-plus project. For an extreme example, see the multi-decade project by the Indian rail company to gradually replace all its railways to standard gauge. It works fine if you have the organisational backing for a long duration, but the constant knowledge leaks from people leaving, retiring, getting promoted, etc can be a real problem for a project like that. Especially in fields where the knowledge is the product, like in software.
> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level.
You could also try to buy some off-the-shelf solutions? Making payroll, even for very large organisations, isn't exactly a new problem.
As a corollary I would also suggest: subsidiarity.
> Subsidiarity is a principle of social organization that holds that social and political issues should be dealt with at the most immediate or local level that is consistent with their resolution.
I think you'll find that is exactly what people do. However, payroll solutions are highly customized for every individual company and even business unit. You don't buy a payroll software in a box, deploy it, and now you have payroll. Instead, you pay a payroll software company, they come in and get information about your payroll systems, and then they roll out their software on some of your systems and work with you to make sure their customizations worked etc. There's rarely any truly "off-the-shelf" software in B2B transactions, especially the type of end-user solutions that also interact with legal systems.
Also, governments are typically at least an order of magnitude larger than the largest companies operating in their countries, in terms of employees. So sure, the government of Liechtenstein has fewer employees than Google overall, but the US government certainly does not, and even Liechtenstein probably has way more government employees than Google employees in their country.
I work at a small shop, I'm a big advocate of giving customers the 0.1 version and then talking it out what they want. It's often not exactly what they asked for at the start ... but it often is better in the end.
Yes. Also the same applies to companies. There should not be companies that are growing to $100 million revenue while losing money on a gamble that they will eventually get big enough to succeed. Good first, big later.
$100M maybe. But pretty much all tech needs an initial investment before you can start making profit. It takes a lot of development before you can get a product that anyone would want to pay for.
>It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features.
I think that is true for a lot of projects. But I'm not sure it is realistic to incrementally develop a control system for a nuclear reactor or an air traffic control system.
Not saying you're wrong, but I wonder what is the differentiating factor for software? We can build huge things like airliners, massive bridges and buildings without starting small.
Incremental makes less sense to me when you want to go to mars. Would you propose to write the software for such a mission in an incremental fashion too?
Yet for software systems it is sometimes proposed as the best way.
> We can build huge things like airliners, massive bridges and buildings without starting small.
We did start small with all of those things. We developed rigorous disciplines around engineering, architecture, material sciences. And people died along the way in the thousands[0][1]
People are still dying from those failures; The Boeing 737 MAX 9 crash was only two years ago.
> Incremental makes less sense to me when you want to go to mars.
This is yet another reason why a manned Mars mission will be exceedingly dangerous NOT a strike against incremental development and deployment.
All of the things you mentioned are designed and tested incrementally. Furthermore software has been used on Mars missions in the past, and that software was also developed incrementally. It’s proposed as the best way because it’s a way that has a track record
> All of the things you mentioned are designed and tested incrementally.
In a different way that what is proposed in this thread.
We don't build a small bridge and grow it. We build small bridges, develop a theory for building bridges and use that to design the big bridge.
I don't know of any theory of computing that would help us design a "big" program at once.
That sounds like the way nature handles growth and complexity: slowly and over long time scales. Assume there will be failures, don't die and keep trying.
When you bite off too much complexity at once you end up not shipping anything or building something brittle.
You just need: Plan -> Implement -> Test -> Repeat
Whether you are creating software, games or whatever, these iterations are foundational. How these steps look like in detail of course depends on the project itself.
That's the ideal, but a lot of these big problems can't start small because the problem they have is already big. A lot of government IT programs are set up to replace existing software and -processes, often combining a lot of legacy software's jobs and the manual labor involved.
If you have something like a tax office or payroll, they need to integrate decades of legislation and rules. It's doable, but you need to understand the problem (which at those scales is almost impossible to fit in one person's head) and more importantly have diligent processes and architecture to slowly build up and deploy the software.
tl;dr it's hard. I have no experience in anything that scale, I've been at the edges of large organizations (e.g. consumer facing front-ends) for most of my career.
The accounting, legal and business process requirements are vastly different at different scales, different jurisdictions, different countries, etc.
There's a crazy amount of complexity and customizability in systems like ERPs for multinational corporations (SAP, Oracle).
When you start with a small town, you'll have to throw most of everything away when moving to a different scale.
That's true for software systems in general.
If major requirements are bolted on after the fact, instead of designed into the system from the beginning, you usually end up with an unmaintainable mess.
Knowing that the rules for your first small deployment are not the same as the rules for everywhere, is valuable for designing well. Trying to implement all of those sets of rules in your initial deployment, is not a good idea. There is a general principle that you shouldn't code the abstraction until you've coded for the concrete example 2 or 3 times, because otherwise you won't make the right abstraction. Looking ahead is not the same as starting with the whole enchilada for your initial deployment.
I do get concerned when the solution is to be more strict on the waterfall process.
I used to believe there were some worlds in which waterfalls are better: where requirements are well know in advance and set in stone. I’ve since come to realize neither of those assumptions is ever true.
Imagine if the only way to build a skyscraper was to start with a dollhouse and keep tacking extensions and pieces onto it until. Imagine if the only way to build a bridge across San Francisco bay was to start with pop sickle sticks.
The very specific example you chose: payroll, shows how it can be difficult to incrementally step from small to huge. As you grow from town to national, you will run into all the disadvantages without really hitting the advantages. I feel that incremental does help you move from one level to one just a few above. But only if there are enough customers at these starting levels exactly.
When developing for towns, you will have all small random subsets of the variations imposed by year after year of legal changes BUT small sales. You will have to implement niche variations in arbitrary aspects for all the towns you have to support AND you will not have the customer size on which to amortize this work. Each new customer will bring a new arbitrary set of legal aspects to be met. Each new customer may be arbitrarily difficult to support.
By the time you reach national, you will have already covered most of the historical legal quirks - but that will have been done in one kludgy manner after another - and then you will hit one more set of legal quirks at the level of national organizations (some of them will have their very own laws). You will now have a very large budget to finalize things but you will be burdened by an illogical software base.
So I agree that you will need experience and subject matter experts that have worked at the various levels. BUT, now that you have this experience you now know the degree of flexibility that is required (you know where and what needs to be variable and quirk-friendly and how far the quirks can go = "any size") as well as size-related issues (mailing, transaction, user support volume) and you can now plan for all this AS YOU restart a new development from scratch. Because at this new "master" level you need both systematic flexibility AND relience at size.
Payroll is exactly the kind of topic where "adding features" will be "fun" - I mean bewildering - while you learn, but probably economically difficult to manage, until it kills you "as you climb up"?
You will be killed by a large software project that can afford to hire out a bunch of your subject matter specialists (or hires new ones) and uses them in a "from scratch" project. If you are lucky, this large project will be from the same company but only if you are lucky.
Now. AFTER you have done the one top level project - for one country -, you will probably be in a good situation to sell service to all kinds of organizations. Because you now have a system in which you can implement ridiculous quirks without breaking everything. And if you have done the job just right, you can onboard smaller customers (towns) economically enough that they can afford your solution.
That's different from where you deploy your solution first. Sure, deploy a national-design solution first at a subset of the target employees - although that does impose more requirements still: now you need to coexist with the legacy solutions. Which would be another hard to meet handicap when developing for towns first.
I study and write quite a bit of tech history. IMHO from what I've learned over the last few years of this hobby, the primary issue is quite simple. While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning. Typically, software folks build new and every generation of software developers must relearn the same problems.
I work at $FANG, every one of our org's big projects go off the rails at the end of the project and there's always a mad rush at the end to push developers to solve all the failures of project management in their off hours before the arbitrary deadline arrives.
After every single project, the org comes together to do a retrospective and ask "What can devs do differently next time to keep this from happening again". People leading the project take no action items, management doesn't hold themselves accountable at all, nor product for late changing requirements. And so, the cycle repeats next time.
I led and effort one time, after a big bug made it to production after one of those crunches that painted the picture of the root cause being a huge complicated project being handed off to offshore junior devs with no supervision, and then the junior devs managing it being completely switched twice in the 8 month project with no handover, nor introspection by leadership. My manager's manager killed the document and wouldn't allow publication until I removed any action items that would constrain management.
And thus, the cycle continues to repeat, balanced on the backs of developers.
Of course the reason it works this way is that it works. As much as we'd like accountability to happen on the basis of principle, it actually happens on the basis of practicality. Either the engineers organize their power and demand a new relationship with management, or projects start going so poorly that necessity demands a better working relationship, or nothing changes. There is no 'things get better out from wisdom alone' option; the people who benefit from improvements have to force the hand of the people who can implement them. I don't know if this looks like a union or something else but my guess is that in large part it's something else, for instance a sophisticated attempt at building a professional organization that can spread simple standards which organizations can clearly measure themselves against.
I think the reasons this hasn't happened is (a) tech has moved too fast for anyone to actually be able to credibly say how things should be done for longer than a year or two, and (b) attempts at professional organizations borrowed too much from slower-moving physical engineering and so didn't adapt to (a). But I do think it can be done and would benefit the industry greatly (at the cost of slowing things down in the short term). It requires a very 'agile' sense of standards, though.. If standards mean imposing big constraints on development, nobody will pay attention to them.
You forgot c) we're in a culture where people jump ship every 3-5 years. There's no incentive to learn from mistakes that you don't talk about at the next company, nor any care for the long term health of the current company.
>a sophisticated attempt at building a professional organization that can spread simple standards which organizations can clearly measure themselves against.
We have that as a form of IEEE, but it really doesn't come up much if you're not already neck deep in the organization.
Speaking as someone in the US, I've worked at multiple companies (some startups, some small businesses, some larger) that have either outright imploded or had mass department-level layoffs inside that 1-2 year timeframe. Some of them I would have stayed at longer than 1-2 years if I had the choice. I'm not claiming it's universal by any means, but there is a lot of volatility at US businesses in my personal experience.
And as usual, the employees get blamed. Maybe people wouldn't jump ship if companies didn't lay people off with reckless abandon, or hire sociopathic bosses to manage people, or screw them out of raises, or overwork them, or lie during the interview process.
The little people are going to do what they need to do to survive. If these multi-billion or even multi-trillion dollar companies feel some type of way about it, well, they're the ones with all the power, not us. They're more than welcome to change the culture at any time.
It "works" only on a certain timescale. We don't have sufficient incentives and penalties to make things fail quickly. A relevant example in the tech world is data breaches. If data breaches resulted in a thorough public audit and financial/criminal penalties for the managers who pushed for speed over safety, they would no longer "work".
> If standards mean imposing big constraints on development, nobody will pay attention to them.
Unless there are penalties for not doing so.
> tech has moved too fast for anyone to actually be able to credibly say how things should be done for longer than a year or two
But that's just it. If things are moving so fast that you can't say how things should be done, then that tells you that the first thing that should be done is to slow everything way down.
I agree wholeheartedly that collective action is how we stop balancing poor management on the backs of engineers, but good luck getting other engineers to see it that way. There's heaps of propaganda out there telling engineers that if they join a union their high salary will go away, even though unions have never been shown to reduce wages.
I don't like unions because one bad hire can destroy a whole team, and the option to remove that hire is worth more than any benefit a union can give me.
I also think people here misunderstand what unions do. Unions are inherently conservative (small c) institutions that aim to protect the status quo. Improving processes is not a fundamental goal to unions. We saw this with the ILA that fought to essentially ban automation in the ports that would drastically increase efficiency because of the belief that this would reduce union jobs. It's foolish to think software unions wouldn't end up becoming like this.
> I don't like unions because one bad hire can destroy a whole team, and the option to remove that hire is worth more than any benefit a union can give me.
In MANY other countries there is already WAY more regulations regarding layoffs and firing employees that has nothing to do with unions.
In Germany there is a probationary period in which you can just fire somebody for no reason basically. That time can be like half a year (in my case) and in most cases it becomes clear if the new hire fits your team or doesn't.
All unrelated to unions though. The big unions in Germany for example have a lot of power and if you are just a simple welder for example you'd have no chance getting anything done without a union.
When your scope is Europe ... The US is not the exception in the world, it's Europe which is.
The US has a dynamic job market where it's easy to lose your job, but easy to find another one. In Europe, and that's true for most EU countries, it's really hard to lose your job, but it's also really hard to get one for the very reason it's hard to get fired - and when you get a job, you will have to compromise on compensation and other benefits. It's not black and white here. While the European market is appealing to some people, the US market is preferable to others.
> It's not black and white here. While the European market is appealing to some people, the US market is preferable to others.
I agree with that, it's a very individual topic. I'd say for high paying "high performance" jobs the US model definitely has an advantage but for low-wage jobs it's quite the opposite.
Counterpoint: Denmark has something called Flexcurity: "flexible" + "security". Basically, it means you can hire and fire more easily than traditional socialist market economies. There is a good social safety net, but it is (somewhat) time constrained to pressure people to return to work quickly.
To be clear, the modern programme started in the 1990s. That makes it about 30 years old. If it had major problems, I assume they would be fixed by now.
Re-reading your comment again, I'm not sure that I understand it: "Let's see whether Denmark remains competitive." What do you mean?
Did they vote left? And that's the same left pushing over and over the Chat Control? That's an interesting twist if it turns out it's not always the right wing trying to undermine privacy rights.
She is the European Commission president, that's unrelated.
But that made me curious, and answering my own question, it's this guy https://en.wikipedia.org/wiki/Peter_Hummelgaard who is indeed a Social Democrat .. So much about workers rights, funny ...
I'd rather have the protections for my working conditions than worrying about whether my co-workers are contributing enough to the company's bottom line but maybe that makes me an outlier here.
I'd say it does. I take pride and meaning in working. Life's too short to not care about a thing you do 8 hours a day. And bad colleagues doesn't just include people not contributing enough.
I've worked places that refuse to fire low performers and its hard for it to not be toxic. I'm not saying this outcome is a forgone conclusion of unions, but my union experience is that poor performers take even longer to get rid of and I'm not sure I would be interested in that sort of environment again. This is more of an implementation problem than philosophical, but theoretically good and practically bad is still just bad.
Until healthcare and housing aren't tied to employment, making it easier to fire people will always be the monstrous position. If you want firing people with abandon to be socially acceptable then you need a public safety net. Until that's in place, I'm always going to be on the side of labor organizations demanding dignity than the people destroying lives.
How do you feel about consensual relationships? I don't impose myself on my employer, we have a mutual and consensual relationship. I don't understand, what do you want peoples material resources tied to if not their own assets?
My relationship with my employer is coercive, not consensual. They have something I need in order to survive. If I leave it will have deleterious effects on the quality of my life. You could make the argument that they are not the ones coercing me, society is or something, but I do not associate with them freely.
This was even more apparent before the ACA passed, when insurance providers were not prevented from denying or charging more for customers with pre-existing conditions. Hate your job but have cancer? Better not leave, you might not even be able to get insured at your next job.
You don't have to fire "low performers" you just have to be realistic about their skillsets and use them that way.
If you see an engineer is out of his depth then change his position, no need to fire them since like others have pointed out, that can have severe consequences in their personal lives and most of the time they can still be useful if they get more adequate work.
My hunch is that software engineers are averse to unions because they correctly perceive that unions are a wide angle away from the type of professional organization that would be most beneficial to them. The industry is sufficiently different that the normal union model is just not very good and has a 'square leg round hole' feeling.
For instance by and large the role of organizing to not to get more money but rather to reduce indignities... Wasted work, lack of forethought, bad management, arbitrary layoffs, etc. So it is much more about governing management with good practices than about keeping wages up; at least for now wages are generally high anyway.
there are also reasons to dedend jobs/wages in the face of e.g. outsourcing... But it's almost like a separate problem. Maybe there needs to be both a union and a uncoupled professional standard or something?
what type of professional organization is most beneficial? Standards are already out there, but they need a union or government regulation to be enforced. Devs who want real change need to pick their medicine, or continue to let the industry stagnate.
>the role of organizing to not to get more money but rather to reduce indignities
agreed. And I think that's why it's going to really start taking hold as we enter year 4 of mass layoffs in the US (because outsourcing). Alongside overwork from the "survivors" and abusive PIPs to keep people on edge.
> year 4 of mass layoffs in the US (because outsourcing)
A lot of the layoffs appear to be about conserving cash for investment in AI. In many cases the jobs that are cut are not backfilled by workers in the US or abroad.
It's wild to claim that the industry is stagnating. By any objective measure the industry is larger, more influential, and more innovative than ever before. Perhaps the problems that people are complaining about here just don't matter very much?
I take it you have a good reputation and a network then. It is getting increasingly difficult to get hired lately, even as a senior. At least in the EU.
It's a shame how the EU has sabotaged their own software industry through a mix of excessive regulation, misguided labor protections, and failure to develop broad capital markets. It's like they're collectively choosing to be poor and backward. I can't understand it.
I can't understand it either. To me it seems people are excessively risk-averse. I can understand that but I don't understand the leadership not trying to stimulate a little bit of risk and growth there. It's getting to absurd levels at places.
But regardless. Not like USA companies are open to EU citizens currently. Whether it's politics, compliance / legal, or tax-related, it does not matter much. Most US job ads I've seen lately are making it super clear that they want only US citizens.
Pick any measure you like: total employment, revenue, patents, new venture formation, products shipped, etc. There are fluctuations from year to year due to the business cycle but overall the trend is up.
Guess that's why gamedev is the one region where this is really starting to gain momentum. High salaries were already not a thing, and tend to mean nothing if you're laid off after 3 years of development for the release of a new game.
Though I think Gen Z in general will be making waves in the coming years. They can't even get a foot in the door, so why should they care about "high salaries"?
People aren't going to try to wrest control from management because some project is going off the rails. No one has any particular faith their coworkers will run anything better, and the pay checks show up regardless.
A union might help improve wages and working conditions in some organizations (although I personally wouldn't want one). But there is zero chance that a union could ever achieve widespread improvement in software architecture, methodologies, or project management. We don't have much consensus on the right way to do things, and what worked well in one circumstance often causes disaster in another.
I don't think unions are the right thing here, you just need to get together as a team and talk with your higher ups, that's a far smaller scope than where unions normally come in.
But I totally agree, I think people are too compliant and fear banding together to have influence on higher ups. I'd argue in most places the engineers have far more power than they realize since they are in high demand due to shortages of qualified personnel. (at least in many countries in Europe)
There are tons of factors in play though that I believe contribute to this like some employees being friends with their higher ups not wanting to hurt their careers, not wanting the tough discussions, the repercussions if management says "no" etc..
"Just". Come on, man, you know better than that. I too like my sci-fi to be over the top unrealistic.
Truth is, nobody ever thinks about their rights before it's too late. The paycheck shows up on time every month and people just don't want to rock the boat. Not to mention all the opportunists that will tell on you immediately to gain the favor of the upper class.
These things are well-known and apparently nothing ever changes before the guillotines start working. I don't think anything will ever improve in our area. Nobody is bothered. The people who are have zero power. And so it goes.
The "just" was more in the sense that it technically is not that hard to get together, there is nothing directly preventing that.
I honestly think it's just people who don't mind these problems and are fine working under those conditions, or they just quit once they are too annoyed. Change is way harder just than just leaving, I think that's also part of the reason why this keeps going on.
Absolutely not, barely anyone is okay with it -- only the most naive and fresh of young workforce, and then not even half of them by my observations chatting with many people 20+ years younger than me. It's just that people have too much at stake to risk it. Families, mortgage, basic respectful existence even. Nobody wants to live under a bridge.
It's extremely sad. I did not want to live in such times but oh well, good thing anyone asked us, right?
How about licensure and liability instead? That’s the sword of Damocles hanging over the heads of the rest of the engineering world. Sure it’s a guild system with a new name, but if the bridge collapses, somebody is going to be in a courtroom.
Customers are free to demand that software vendors take liability for certain defects or failures as part of contract negotiation. There's no need for governments to get involved.
For software that's actually safety critical, like avionics, there are already sufficient regulatory controls.
Actually it is necessary. That’s why your bridge engineer has to satisfy a licensing board that he or she knows what they’re doing before they seal a set of plans. Clients can’t distinguish good from bad engineering. They shouldn’t have to figure out if you know what you’re doing.
Actually it isn't necessary. Unlike bridges, very little software is safety critical. Clients don't need to try to distinguish good from bad software development, they can just require a certain level of performance and reliability in the contract with clear financial penalties for failure.
Has union labor resulted in measurable improvement in production outcomes in any industry they’re found in? I don’t think going from “managers are unaccountable for failure” to “nobody is accountable for failure” is a good thing.
I think introducing more competition at higher levels may be better than eliminating it below. This should be happening because I’m pretty sure most PMs could be replaced by an LLM.
How about we transition from “managers are unaccountable for failure” to “managers are accountable for every failure by default and are sued and have houses confiscated within two months of a case being open”? That must include CEOs as well though, not just some bootlicker PMs.
Executives are generally incompetent everywhere and of course they'll introduce a reality distortion field where none of them are ever accountable. That should be obvious to anyone. Question is why do all working people keep allowing it to happen. But the answer to that is also known and quite depressing, too.
While I too doubt that unions would be a net negative and I might even suspect interference by paid malicious actors to discourage people to unionize and thus never have power, I can't agree with your skepticism that PMs cannot be replaced by LLMs.
Most PMs I've ever met had zero clue what they are doing. And no it's not only N=1 sample, same anecdote is heard from many, many other people.
But sadly enough, undeniable human incompetence there is not even the biggest problem. The "we will never make more reasonable deadline" is.
Most managers, not just PMs, are an objective net negative. As any ruling class always does, they get complacent and think that just putting their foot down is going to magically change reality.
Highly irrelevant. Not my role as an individual contributor to hone and fine-tune their job description.
Fact on the ground is that they usually optimize the entirely wrong indicators and never ever optimize for avoiding future problems.
In my consulting and contracting stints I made it a habit to write down what crisis is looming on the horizon and making bets with my wife which one is going to arrive first. A fun little game.
Watching train wrecks in slow motion stopped being fun with time.
What PM's job is is above my pay grade. I want them to enable me and the team, not be a mouth piece of people with zero understanding of product _and_ of engineering. They are just there to ask you how is stuff going.
> Fact on the ground is that they usually optimize the entirely wrong indicators and never ever optimize for avoiding future problems.
Fact on the ground is they don't optimize for _your_ indicators and have to compromise on what problems can be addressed now or later. They only appear to be optimizing to the wrong indicators because you don't have all the information.
> have to compromise on what problems can be addressed now or later.
9/10 PM's work is literally avoiding any decisions and responsibilities whatsoever.
Concerned dev/ops/po: Dear PM, feature FOO-123 will not be merged before we start activities for Gate 5.2. FOO-123 is required for Gate 6.0 and if we start with Gate 5.2 activities without it, Gate 6.0 will be delayed by at least 5 weeks. Team FOO is projecting verification of FOO-123 being done by the end of week, which would delay Gate 5.2 by a week. Shall we delay our activities until FOO-123 is merged or start regardless?
PM: Gate 5.2 is extremely important for the project timeline and no delays are acceptable.
<- few moments later ->
PM: I was informed by Compliance that FOO-123 is mandatory, does that affect timelines for Gate 6.0?
Disgruntled employee: Either we start over Gate 5.2 activities with FOO-123 included, which would delay Gate 6.0 by at least 9 weeks, or you get team FOO to backport FOO-123.
<- few moments later ->
PM: gets promoted for successful handling of stressful situation with FOO-123, limiting project delay to 15 weeks and only overrunning projected costs of Gate 6.0 by 30%.
I'm waiting for the disruptive startup that revolutionizes corporate heirarchy by getting rid of PMs since they're so useless.
Since successful businesses continue to employ them, I'm going to err on the side of they do serve a function I don't always understand rather than they must be stupid.
Let me introduce you to the glory of Valve and Lord Gaben.
On a less snarky note: Depending on organization type PMs might do a ton of paperwork on behalf of the teams and act as some sort of central point for business-product communication, however that has exactly nothing with management in any typical definition.
I so wish I would agree with you, if my career has not been a string of them _never_ compromising on anything and only pulling the blanket to their side of the bed.
Your balanced take is nice and I wish I inhabited that reality. So far I have not. Maybe it's the region market or maybe my marketing sucked.
The reality in most businesses is constant roll from fighting one immediate crisis to another and handwaving future risks on "business environment may change" and "not important right now". The more crises one "mitigates", the more competent they are seen, even if they have directly caused the crisis in question.
If you talk about an upcoming shitshow you are generally seen as not entrepreneurial, if you mitigate a risk before it becomes a crisis that crisis is not viewed as crisis. However, if you let a crisis happen and "successfully" manage it you are seen as a hero.
For one project I got so far as to include in the project proposal some outcomes that showed whether or not it was a success: quote from the PM “if it doesn’t do that then we should not have bothered building this”. They objected to even including something so obviously required in the plan.
Waste of my bloody time. Project completed, taking twice as many devs for twice as long, great success, PM promoted. Doesn’t do that basic thing that was the entire point of it. Nobody has ever cared.
Edit to explain why I care: there was a very nice third party utility/helper for our users. We built our own version because “only we can do amazing direct integration with the actual service, which will make it far more useful”. Now we have to support our worse in-house tool, but we never did any amazing direct integration and I guarantee we never will.
Glad to hear that $FANG has similar incompetency as every other mid-tier software shop I've ever worked in. Your project progression sounds like any of them. Here I was thinking that $FANG's highly-paid developers and project management processes were actually better than average.
Reminds me of the military. Senior leaders often have no real idea of what is happening on the ground because the information funneled upward doesn't fit into painting a rosy report. The middle officer ranks don't want to know the truth because it impacts their careers. How can executives even hope to lead their organizations this way?
Well the US has lost every military conflict it's entered for the past 70 years. Since there's been no internal pressure to change methodology, maybe the US military doesn't view winning as necessary.
Those past 70 years weren't about winning. It was about making sure the enemies lost more out of it. The US is large and relatively stable and hasn't had to face extended war on its soil since the Civil War 170 years ago. There's no true skin in the game for those who start these wars.
Those might be the only ones? Desert Storm being the one that I'd probably call out, Just Cause was just so small.
One minor win, every major operation being a loss doesn't change the conclusion though imo.
I think it's also instructive to look at each of these operations and note that the two that were won were small, had clear objectives, and were executed quickly to meet those objectives and had no scope creep.
Iraq had one of the largest militaries in the world at the time of Desert Storm. They had tons and tons of arms and equipment and a huge standing army to counter the persistent threat (and/or to provide their own threat) of resumed hostilities with Iran (that war was still pretty recent when Desert Storm took place)
I would agree that the US is notably terrible at occupations and getting involved in civil wars, at least since WWII, but Desert Storm was pretty much an unqualified slam-dunk take-a-victory-lap success against one of the top armies in the world that wasn’t an ally or a nuclear state—carried out on the other side of the planet from the US, to boot. Like I think Iraq was ranked top-10 at the time by many ways of reckoning military strength, and that wasn’t enough to effectively resist the US effort at all, really.
If that war seems small, it’s only possible for it to seem that way from the victor’s perspective, and only because we did such an amazingly good job of totally destroying Iraq’s substantial capacity to fight in a matter of weeks. In terms of deployed and engaged men and materiel it was really big, just fast because it was so very one-sided, and “cheap” in terms of casualties on the US side for the same reason.
You're agreeing with me, or rather I agree with you.
I consider Desert Storm an unqualified victory in an engagement that is in the same conversation as a Vietnam or Korean war, but still not quite to a WW level in scope or complexity.
Is it middle management that has no accountability, or executive? Middle and line managers are nearly as targeted by layoff culling as ICs these days in FAANG. The broad processes they're passing down to ICs generally start with someone at director level or higher.
In my experience it is the constant shifting of goal posts due to execs chasing the next shiny thing, or demanding a feature that they saw somewhere, or heard from client (singular, not plural)
I've seen plenty of incredible engineers let go because of "performance issues" that were just poor project management and goal posts that moved so fast waymo should study them to improve their self-driving capabilities.
Shit rolls downhill and there's a lot more fuss when an engineer calls out risks, piss-poor planning, etc. than any actual introspection on why the risks weren't caught sooner or why the planning was piss-poor.
> For how much power they have over team organization and processes, software middle management has nearly no accountability for outcomes.
Can we also address the fact that “software spend” is distributed disproportionately to management at all levels and people who actually write the software are nickel and dimed. You’d save billions in spend and boost productivity massively if the management is bare bones and is held accountable like the rest of the folks.
that's how the inner sanctum engineering in Apple worked, just like you proposed, at least from 15 years ago to within the last 10 years. i could have been in a very lucky time window to have had that luxury, but it had been an Apple mandate to not have deep hierarchies at least in engineering.
perhaps that was involved, but one thing clearly purposeful was people were seriously filtered for particular skills and personality (apple fit it was called back then), which created groups where individuals had unique skills and collectively the group members would naturally want to collaborate. it worked great.
(as an aside, this contrasts diametrically with Amazon, where i worked for a year for healthcare, not needing to because of Apple years' savings, but after a genomics startup i had joined ran out of funding, and wanting a new challenge; there skilled engineering types were presumed to be fungible assets for (not kidding) at least 7 layers of do-nothing bureaucrats making huge salaries...they could survive because sales on the amazon store extract something like the 30% royalty to amazon)
Yes and no. The compensation is a lot, but you're not necessarily able to just quit on a dime even if you live humbly. Interviewing takes weeks now and weeks more just to find a proper replacement. And salaries can fund you for months, bu t not years (let alone if you have a fammily)
I'll also say the obvious here in Sinclair's quote about salaries: you can indeed pay for someone's self respect.
This would imply most of these types of positions are filled with less competent people willing to package and sell their self respect alongside their time?
(Thus commanding a rate similar to a more competent person who doesn’t package it to sell.)
Joke is, most of these homes aren't worth anywhere close to their paper value.
Cy Porter's home inspection videos... jeez. How these "builders" are still in business is mind-blowing to me (as a German). Here? Some of that shit he shows would lead to criminal charges for fraud.
There is an old Russian joke, that goes like this:
A man approaches a girl and asks, "Would you sleep with me for $1 million?”
She responds, “Yes, of course!”
Excited, he then asks, “What about for $1?”
She indignantly replies, “Who do you think I am?”
To which he responds, “We already established who you are; now we’re just discussing the price.”
I think it fits there. There is surprising amount of people believing that they have some Values, but just to a point when they were offered to sell them for a right price.
There's a high switching cost with substantial information asymmetry. Good places are hard to find in some absolute sense and it's hard to evaluate whether a new team is actually going to be good or not. And even if you do find a good team, there's no guarantee it'll last past the next reorg.
serious answer - you find a team with a good direct manager who handles all the upward interactions themselves, and then you basically work for that manager, rather than for the company.
If you think our ability to measure developer productivity is bad, look into what we can do to measure manager productivity.
TL;DR your realistic options are snake oil that doesn’t work, or nothing.
Keep that in mind next time anyone’s talking about managing through metrics & data or whatever bullshit. All that stuff’s kayfabe, companies mostly run on vibes outside a very-few things.
^ This. Not at FAANG, but I am too familiar with this.
This is why software projects fail. We lowly developers always take the blame and management skates. The lack of accountability among decision makers is why things like the UK Post Office scandals happen.
Heads need to be put on pikes. Start with John Roberts, Adam Crozier, Moya Greene, and Paula Vennells.
So much of the world, especially the world we see today around corporate leadership and national politics makes much more sense once you realize this fundamental law:
People who desire infinite power only want it because it gives them the power to avoid consequences, not because they want both the power and the consequences.
The people who believe that with great power comes great consequences are exactly the people who don't want great power because they don't want the weight of those consequences. The only people who see that bargain and think "sign me up!" are the ones who intend to drop the consequences on the floor.
I was a developer for a bioinformatics software startup in which the very essential 'data import' workflow wasn't defined until the release was in the 'testing' phase.
Did they go off the rails at the end, or deadlines force acknowledging that the project is not where folks want it to be?
That said, I think I would agree with your main concern, there. If they question is "why did the devs make it so that project management didn't work?" Seems silly not to acknowledge why/how project management should have seen the evidence earlier.
Where I now work, in the government, all the devs are required to be part project managers. It’s a huge breath of fresh air. The devs are in all the customer meetings, assist in requirements gathering, and directly coach the customers as necessary to keep pushing the work towards completion.
This came about because our work isn’t too diverse but the requirements are wildly diverse and many of the customers have no idea how to achieve the proper level of readiness. I do management in an enterprise API project for a large organization.
There are many pressures and this is all about a lack of transparent honesty about what the real priorities are. Getting the project done properly may be #1 priority but there's priority 0 and 0.1 and others which are unspoken because they don't sound good.
FAANG cargo cults are all around. Too many CEOs and VCs imagine they can succeed like Bezos by the bold strategy of hiring ex-Amazon executives and imitating Amazon's culture and HR practices.
I've also considered a side-effect of this. Each generation of software engineers learns to operate on top of the stack of tech that came before them. This becomes their new operating floor. The generations before, when faced with a problem, would have generally achieved a solution "lower" down in the stack (or at their present baseline). But the generations today and in the future will seek to solve the problems they face on top of that base floor because they simply don't understand it.
This leads to higher and higher towers of abstraction that eat up resources while providing little more functionality than if it was solved lower down. This has been further enabled by a long history of rapidly increasing compute capability and vastly increasing memory and storage sizes. Because they are only interacting with these older parts of their systems at the interface level they often don't know that problems were solved years prior, or are capable of being solved efficiently.
I'm starting to see ideas that will probably form into entire pieces of software "written" on top of AI models as the new floor. Where the model basically handles all of the mainline computation, control flow, and business logic. What would have required a dozen Mhz and 4MB of RAM to run now requires TFlops and Gigabytes -- and being built from a fresh start again will fail to learn from any of the lessons learned when it was done 30 years ago and 30 layers down.
Yeah, people tend to add rather than improve. It's possible to add into lower levels without breaking things, but it's hard. Growing up as a programmer, I was taught UNUX philosophy as a golden rule, but there are sharp corners on this one:
To do a new job, build afresh rather than complicate old programs by adding new "features".
It's the "Lava Flow" antipattern [1][2] identified by the Gang of Five [3], "characterized by the lava-like 'flows' of previous developmental versions strewn about the code landscape, but now hardened into a basalt-like, immovable, generally useless mass of code which no one can remember much if anything about.... these flows are often so complicated looking and spaghetti-like that they seem important but no one can really explain what they do or why they exist."
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not
I've been managing, designing, building and implementing ERP type software for a long time and in my opinion the issue is typically not the software or tools.
The primary issue I see is lack of qualified people managing large/complex projects because it's a rare skill. To be successful requires lots of experience and the right personality (i.e. low ego, not a person that just enjoys being in charge but rather a problem solver that is constantly seeking a better understanding).
People without the proper experience won't see the landscape in front of them. They will see a nice little walking trail over some hilly terrain that extends for about a few miles.
In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.
> In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.
And boy to the people making the decisions NOT want to hear that. You'll be dismissed as a naysayer being overly conservative. If you're in a position where your words have credibility in the org, then you'll constantly be asked "what can we do to make this NOT a quest to the top of Mt Doom?" when the answer is almost always "very little".
Impossible projects with impossible deadlines seems to be the norm and even when people pull them off miraculously the lesson learned is not "ok worked this time for some reason but we should not do this again", then the next people get in and go "it was done in the past why can't we do this?"
Wow, sounds so familiar! I've once had to argue precisely against this very conclusion - "you saved us once in emergency, now you're bound to do it again".
Wrote to my management: "It is, by all means, great when a navigator is able to take over an incapacitated pilot and make an emergency landing, thus averting the fatality. But the conclusion shouldn't be that navigators expected to perform more landings or continue to be backup pilots. Neither it should be that we completely retrain navigators as pilots and vice versa. But if navigators are assigned some extra responsibility, it should be formally acknowledged by giving them appropriate training, tools and recognition. Otherwise many written-off airplanes and hospitalized personnel would ensue."
For all I know the only thing this writing might have contributed to was increased resentment by management.
> And boy to the people making the decisions NOT want to hear that.
You are 100% correct. The way I've tried to manage that is to provide info while not appearing to be the naysayer by giving some options. It makes it seem like I'm on board with crazy-ass plan and just trying to find a way to make it successful, like this:
"Ok, there are a few ways we could handle this:
Option 1 is to do ABC first which will take X amount of time and you get some value soon, then come back and do DEF later
Option 2 is to do ABC+DEF at the same time but it's much tougher and slower"
My favorite fact is that every single time an organization manages to make a functional development team that can repeatedly successfully navigate all the problems and deliver working software that adds real value, the high up decision makers always decide to scale the team next.
Working teams are good for a project only, then they are destroyed.
Jesus I just had flashbacks from my last jobs. Non-technical founder always telling me I was being pessimistic (there were no technical founders). It's just not that simple Karen!
I think part of it is that reading code isn't a skill that most people are taught.
When I was in grad school ages ago, my advisor told me to spend a week reading the source code of the system we were working with (TinyOS), and come back to him when I thought I understood enough to make changes and improvements. I also had a copy of the Linux Core Kernel with Commentary that I perused from time to time.
Being able to dive into an unknown codebase and make sense of where the pieces are put together is a very useful skill that too many people just don't have.
The successful projects I've worked on, the technical staff have been given autonomy, responsibility and full insight into the problem space. This requires managers putting a lot of trust in the engineers, but it works.
Large projects I've worked on failed simply because nobody wanted the solution in the first place.
In government I've seen many millions spent on projects that were either forgotten about or the politician that requested it lost office.
You’ll notice that more senior engineers are often much better at giving useful review comments, and they will do it faster than you, thats just a skill that seems to come with experience reading other peoples code(or your own code you wrote two years prior). It can’t be taught, only practiced, same goes for reading other types of technical/academic works.
Not GP, but the general idea is the skill to take a piece of code and understand what it does by reading the code itself (probably in an IDE that can help navigate it meaningfully), not relying on docs or explanations or anything else. Surprisingly few people are comfortable in doing this, and yet it's very common in any large software project that lots of parts of the code are undocumented and no one remembers the details of how they were written.
Precisely. I don't assume any documentation is accurate unless I've verified it matches the behavior of the code - and I've run into some doozies in my day.
The worst was a custom-built copy protection scheme that was built by a former (?) software cracker who designed it to be difficult for someone of his skills to reverse engineer. It took me a week tracing through the code to understand what it was actually doing so I could extend it to add more options.
This is one part of the issue. The other major piece of this that I've seen over more than two decades in industry is that most large projects are started by and run by (but not necessarily the same person) non-technical people who are exercising political power, rather than by technical people who can achieve the desired outcomes. When you put the nexus of power into the hands of non-technical people in a technical endeavor you end up with outcomes that don't match expectations. Larger scale projects deeply suffering from "not knowing what we don't know" at the top.
If this were true all of the time then the fix would be simple - only have technical people in charge. My experience has shown that this (only technical people in charge) doesn't solve the problem.
Success pretty much requires putting technical people in charge, but that doesn't mean putting technical people in charge is sufficient for success to happen. We have plenty of data over the last 40 years to prove my case. Furthermore, unfortunately, what it means to be a "technical person" is not so simple to define, unfortunately as the easy ways to codify it often exclude the very people who you want involved.
Suffice to say, projects are significantly more likely to succeed when the power in the project is held by people who are competent /and/ understand the systems they are working with /and/ understand the problem domain you are developing a solution in. Whether or not they have a title like "engineer" or have a technical degree, or whatever other hallmark you might choose is largely irrelevant. What matters is competency and understanding, and then ultimately accountability.
Most large projects I've been a part of or near lacked all three of these things, and thus were fundamentally doomed to failure before they ever began. The people in power lacked competency and understanding, the entire project team and the people in power lacked accountability, and competency was unevenly distributed amongst the project team.
It may feel pithy, but it really is true that in many large projects the fundamental issue that leads to failure is that the decision makers don't know what they're doing and most of the implementers are incompetent. We can always root cause further to identify the incentive structures in society, and particularly in public/government projects that lead to this being true, but the fact remains at the project level this is the largest problem in my observation.
> Success pretty much requires putting technical people in charge, but that doesn't mean putting technical people in charge is sufficient for success to happen.
It does solve the problem! That's the entire reason we have such a thing as a tech industry. Tech industry startups are often just regular industries where everyone from the board down is technical. Uber is a "tech company" and not a taxi company because of this distinction. The results speak for themselves.
Sometimes giving people what they want can be bad for them; management wants cheap compliant workers, management gets cheap compliant workers, and then the projects fall apart in easily predictable and preventable ways.
Because such failures are so common management typically isn’t punished when they do so it’s hard to keep interests inline. And because many producers are run on a cost plus basis there can be a perverse incentive to do a bad job, or at least avoid doing a good one.
I'm not entirely sure what you mean with "technical people" but it seems that you may not appreciate the problems that "non-technical people" try to tackle.
Do your two decades of experience cover both sides?
> Do your two decades of experience cover both sides?
Yes.
I appreciate both sides and have a wealth of experience in both. The challenge is that all the non-technical problems cannot be solved successfully while lacking a technical understanding. Projects generally don't fail for technical reasons, they fail because they were not set up for success, and that starts with having a clear understanding of requirements, feasibility, and a solid understanding of both the current state and the path to reach your desired outcomes, both politically/financially and technically.
I was an engineer for more than a decade, I've been in Product for nearly a decade, and I'm now a senior manager in Product. I can honestly say that I have the necessary experience to hold strong opinions here and to be correct in those opinions.
You need technical people who can also handle some of the non-technical aspects of a project with the reins of power if you want the project to succeed, otherwise it is doomed by the lack of understanding and competency of those in charge.
I have a theory that the churn in technology is by design. If a new paradigm, new language, new framework comes out every so many years, it allows the tech sector to always want to hire new graduates for lower salaries. It gives a thin veneer of we want to always hire the person who has X when really they just do not want to hire someone with 10 years of experience in tech but who may not have picked up X yet.
I do not think it is the only reason. The world is complex, but I do think it factors into why software is not treated like other engineering fields.
Constantly rewriting the same stuff in endless cycles of new frameworks and languages gives an artificial sense of productivity and justifies its own existence.
If we took the same approach to other engineering, we'd be constantly tearing down houses and rebuilding them just because we have better nails now. It sure would keep a lot of builders employed though.
> If we took the same approach to other engineering, we'd be constantly tearing down houses and rebuilding them just because we have better nails now. It sure would keep a lot of builders employed though.
This is almost exactly what happens in some countries.
Pretty common in Australia. Theres heritage laws to try to prevent replacing all the old buildings, but often they are so undesirable the owner just leaves them vacant until trespassers manage to burn it down.
Have it in Japan too. You can clearly see eras in house design. Pre 1960 almost everything is wood. Then you have wood and plaster until the 2000s or so, and after that is plastic on wood. You can see the age of a neighborhood and its residents based on what houses are made of.
If the residents die and someone new purchases the land, the old house is (generally) torn down and a new one built.
I agree. But, I think the execs just say, "How can we get the most bang for our buck? If we use X, Y, Z technologies, that are the new hotness, then we will get all the new hordes of hires out there, which will make them happy, and has the added benefit of paying them less"
The problem with that is that it would require a huge amount of coordination for it to be by design. I think it's better to look on it as systemic. Which isn't to say there aren't malign forces contributing.
I agree. Perhaps, "by design" is not the correct phrasing. Many decisions and effects go through a multi weighted graph of complexity (sort of like machine learning).
It was more tech companies in collusion than many people realize.
1) Apple and Google, (2) Apple and Adobe, (3) Apple and Pixar, (4) Google and Intel, (5) Google and Intuit, and (6) Lucasfilm and Pixar.
It was settled out of court. One of the plaintiffs was very vocal that the settlement was a travesty of justice. The companies paid less in the settlement than the amount they saved by colluding to keep wages down.
There are rational explanations for this. When software fails catastrophically, people almost never die (considering how much software crashes every day). When hardware fails catastrophically, people tend to die, or lose a lot of money.
There's also the complexity gap. I don't think giving someone access to the Internet Explorer codebase is necessarily going to help them build a better browser. With millions of moving parts it's impossible to tell what is essential, superfluous, high quality, low quality. Fully understanding that prior art would be a years long endeavor, with many insights no doubt, but dubious.
I would boil this down to something else, but possibly related: project requirements are hard. That's it.
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning.
For most IT projects, software folks generally can NOT "pull apart" old systems, even if they wanted to.
> Typically, software folks build new and every generation of software developers must relearn the same problems.
Project management has gotten way better today than it was 20 years, so there is definitely some learnings that have been passed on.
A CIO once told me with Agile we didn’t need requirements. He thought my suggestion to document the current system before modifying was a complete waste of time. Instead he made all the developers go through a customer service workshop, how to handle and communicate with customers. Cough cough… most developers do not talk with customers. Instead where we worked developers took orders from product and project people whose titles changed every year but they operated with the mindset of a drill sergeant. My way or the highway.
Business developers need to occasionally talk directly to customers. It's fine to filter most requirements through Product Managers / Product Owners. But developers who never communicate directly with customers and end users get disconnected from reality and end up acting based on mythology rather than ground truth.
The developers were not given the choice. They are literally mushrooms. No requirements. No interaction with customers. They took orders from product/ project people.
"While hardware folks study and learn from the successes and failures of past hardware, software folks do not." Couldn't be further from the truth. Software folks are obsessed with copying what has been shown to work to the point that any advance quickly becomes a cargo cult (see microservices for example).
Once you've worked in both hardware and software engineering you quickly realize that they only superficially similar. Software is fundamentally philosophy, not physics.
Hardware is constrained by real world limitations. Software isn't except in the most extreme cases. Result is that there is not a 'right' way to do any one thing that everyone can converge on. The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.
Software doesn't operate in some magical realm outside of the physical world. It very much is constrained by real world limitations. It runs on the hardware that itself is limited. I wonder if some failures are a result of thinking it doesn't have these limitations?
As the great Joe Armstrong used to say, “a lot of systems actually break the laws of physics”[1] — don’t program against the laws of physics.
> In distributed systems there is no real shared state (imagine one machine in the USA another in Sweden) where is the shared state? In the middle of the Atlantic? - shared state breaks laws of physics. State changes are propagated at the speed of light - we always know how things were at a remote site not how they are now. What we know is what they last told us. If you make a software abstraction that ignores this fact you’ll be in trouble.[2]
> It very much is constrained by real world limitations. It runs on the hardware that itself is limited
And yet we scale the shit out of it, shifting limitations further and further. On that scale different problems emerge and there is no single person or even single team that could comprehend this complexity in isolation. You start to encounter problems that have never been solved before.
You can scale it within the bounds of the physical hardware it is running on. And as you scale it you start running into all the problems brought about by distributed systems, problems which very much stem from physics.
Except that it kind of does. I can horizontally scale a distributed storage system until we run out of silicon. I cannot do the same with a cargo airplane.
> Software folks are obsessed with copying what has been shown to work to the point that any advance quickly becomes a cargo cult
Seems more accurate to say they are obsessed with copying "what sounds good". Software industry doesn't seem to copy what works, rather what sounds like it'd work, or what sounds cool.
If they copied what works software would just be faster by default, because very often big established tools are replaced by something that offers similar featurage, but offers it at a higher FPS.
I disagree. At least at the RTL level they're very similar. You don't really deal with physics there, except for timing (which is fairly analogous with software performance things like hard real-time constraints).
> Result is that there is not a 'right' way to do any one thing that everyone can converge on.
Are you trying to say there is in hardware? That must be why we have exactly one branch predictor design, lol
> The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.
"The first function call looks a whole lot like a function call today..."
> That must be why we have exactly one branch predictor design, lol
I'll be that 'well akshually' guy. IIRC the AMD and intel implementations are different enough that spectre/meltdown exploits were slightly different on each manufacturers.
What you and the GP said are not mutually exclusive. Software engineers are quick to drink every new Kool-Aid out there, which is exactly why we’re so damned blind to history and lessons learned before.
In my experience, a lot of the time the people who COULD be solving these issues are people who used to code or never have. The actual engineers who might do something like this aren't given authority or scope and you have MBAs or scrum masters in the way of actually solving problems.
I know a lot of people on here will disagree with me saying this but this is exactly how you get an ecosystem like javascript being as fragmented, insecure, and "trend prone" as the old school Wordpress days. It's the same problems over and over and every new "generation" of programmers has to relearn the lessons of old.
The difficulty lies in the fact that most software is quite cheap to generate very complex designs compared to hardware. For software designs treated similarly to hardware (such as in medical devices or at NASA), you do gain back those benefits at great expense.
I think this is too simple. First of all, hardware people have high incentive to fully replace components and systems for many reasons. Replacement is also the only way they can fix major design mistakes.
Software people constantly do fix bugs and design mistakes. There is certainly no strong culture to document or dig up former mistakes made, but it's not like they don't learn from mistakes, it's just a constant process. In contrast to hardware, there is usually no point in time to retrospect. The incentives to rejuvenate systems are low and if considered often seem expensive. Software engineers self motivation is often ill-minded, new devs feeling uncomfortable with the existing system and calling for something "modern". But if the time comes to replace the "legacy" systems, then you are right, no one looks back at the former mistakes and the devs that know them, are probably long gone.
The question is whether we should ever replace an software system or focus more on gradual and active modernization. But the latter can be very hard, in hardware everything is defined, most of the time backed by standards, in software we usually don't have that, so complex interconnected systems rarely have sane upgrade paths.
Most of the time, there's no need to study anything. Any experienced software engineer can tell you about a project they worked on with no real requirements, management constantly changing their mind, etc.
How do you study software history? Most of the lessons seem forever locked away behind corporate walls - any honest assessments made public will either end careers or start lawsuits
Some consequences of NOT learning from prior successes and failures: (a) no more training for the next generation of developers/engineers (b) fighting for the best developers, and this manifests in leetcode grinding (c) decrease in cooperation among team mates, etc.
IME, "Why systems fail" almost always boils down to a principal-agent problem. This is another way of expressing the Mungerism "show me the incentive, I'll show you the outcome".
Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives.
I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.
This is an interesting distinction, but it ignores the reasons software engineers do that.
First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc.
Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.
For most generations of hardware, you’re correct, but not all. For example, high-k was invented to mitigate tunneling. Sometimes, as geometries shrink, the physics involved does change.
I think there is a ton more nuance, but can still be explained by a simple observation, which TFA hints at: "It's the economics, stupid."
Engineering is the intersection of applied sciences, economics and business. The economics aspect is almost never recognized and explains many things. Projects of other disciplines have significantly higher costs and risks, which is why they require a lot more rigor. Taking hardware as example, one bad design decision can sink the entire company.
On the other hand, software has economics that span a much more diverse range than any other field. Consider:
- The capital costs are extremely low.
- Development can be extremely fast at the task level.
- Software, once produced, can be scaled almost limitlessly for very cheap almost instantly.
- The technology moves extremely fast. Most other engineering disciplines have not fundamentally changed in decades.
- The technology is infinitely flexible. Software for one thing can very easily be extended for an adjacent business need.
- The risks are often very low, but can be very high at the upper end. The rigor applied scales accordingly. Your LoB CRUD app going down might bother a handful of people, so who cares about tests? But your flight control software better be (and is) tested to hell and back.
- Projects vary drastically in stacks, scopes and risk profiles, but the talent pool is more or less common. This makes engineering culture absolutely critical because hiring is such a crapshoot.
- Extreme flexibility also masks the fact that complexity compounds very quickly. Abstractions enable elegant higher-level designs, but they mask internal details that almost always leak and introduce minor issues that cause compounding complexity.
- The business rules that software automates are extremely messy to begin with (80K payroll rules!) However, the combination of a) flexibility, b) speed, and c) scalability engender a false sense of confidence. Often no attempt is made at all to simplify business requirements, which is probably where the biggest wins hide. This is also what enables requirements to shift all the time, a prime cause for failures.
Worse, technical and business complexity can compound. E.g. its very easy to think "80K payroll rules linearly means O(80K) software modules" and not "wait, maybe those 80K payroll rules interact with each other, so it's probably a super-linear growth in complexity." Your architecture is then oriented towards the simplistic view, and needs hacks when business reality inevitably hits, which then start compounding complexity in the codebase.
And of course, if that's a contract up for bidding, your bid is going to be unsustainably low, which will be further depressed by the competitive bidding process.
If the true costs of a project -- which include human costs to the end users -- are not correctly evaluated, the design and rigor applied will be correspondingly out of whack.
As such I think most failures, in addition to regular old human issues like corruption, can be attributed to an insufficient appreciation of the economics involved, driven primarily by overindexing on the powers of software without an appreciation of the pitfalls.
As someone who's learning programming right now, do you have any suggestions on how one would go about finding and studying these successes and failures?
First, failures aren’t always obvious, and second, studying them isn’t either. This would likely need to be a formalized course. Still…
If people want to know why Microsoft hated DOS and wanted to kill it with Xenix, then OS/2, then Windows, and then NT it would be vital to know that it only came about as a result of IBM wanting a 16bit source-compatible CP/M which didn’t yet exist. Then, you would likely want to read Dissecting DOS to see what limits were imposed by DOS.
For other stuff, you would start backwards. Take the finished product and ask what the requirements were, then ask what the pain points are, then start digging through the source and flowcharting/mapping it. This part is a must because programs are often too difficult to really grok without some kind of map/chart.
There is likely an entire discipline to be created in this…
The things people are talking about in this thread are less to do with the practice of programming, and more to do with the difficulties of managing (and being managed, within) an engineering organization.
You'll learn all of this for yourself, on the job, just via experience.
To be cynical, what's the point? You'll get employed and forced to be a part of them by circumstances.
Your company's root priorites are probably at odds with writing good software.
One Japanese company, not going to name names, kept trying to treat software as a depreciating asset. I didn't really understand well but the long and short was that fixing things that were supposed to be "done" was bad for the accounting. New things, however were good.
How can you run a software company like that? But they did and got the kind of outcome you'd expect. Japan made the laws this way and gets software to match.
It obviously will do but with any luck you'll be so successful that you'll grow enormously and perhaps not have complete control in the end. Hopefully you'll have enough money not to mind too much!
I think this is a downstream of effect of there being no real regulation or professional designations in software which leads to every company and team being wildly different leading to no standards leaving no time for anything but crunching since there are no barriers restricting your time, so nobody spends time doing much besides shipping constantly.
Software just feels so much more ephemeral than hardware. I haven't yet met a single 'old software enthusiast' in my life, yet there are so many enthusiasts for older hardware.
I am both a hardware and software enthusiast. Tons of DOS, Windows, and OS/2 software hanging around. While I don’t use them everyday, I do use them. From pre-Microsoft Visio to WordStar and MS Works for DOS, the applications are simple, powerful, and pleasing to use. While I don’t recommend anyone pull out Zenith 8bit and fire up COBOL-80 or LISP-80, they are interesting. Testing yourself in 64k is quite a challenge.
The retro community is huge and varied. If it exists, someone is really into it.
I have a pet passion for an old simulation language called Dynamo.
I think you will find people passionate about LISP and people that care about COBOL, and C is already multiple decades old.
... are you saying that hardware projects fail less than software ones? just building a bridge is something that fails on a regular occurence all over the world. Every chip comes with list of erratas longer than my arm.
Software folks treat their output as if it's their baby or their art projects.
Hardware folks just follow best practices and physics.
They're different problem spaces though, and having done both I think HW is much simpler and easier to get right. SW is often similar if you're working on a driver or some low-level piece of code. I tried to stay in systems software throughout my career for this reason. I like doing things 'right' and don't have much need to prove to anyone how clever I am.
I've met many SW folks who insist on thinking of themselves as rock stars. I don't think I've ever met a HW engineer with that attitude.
On some of the infamous large public IT project failures, you just have to look at who gets the contract, how they work, and what their incentives are. (For example, don't hire management consulting partner smooth talkers, and their fleet of low-skilled seat-warmers, to do performative hours billing.)
It's also hard when the team actually cares, but there are skills you can learn. Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).
But once you're a bit comfortable with the art and science of those, big new challenges are more about political and environment reality. It comes down to alignment and competence of: workers, internal team leadership, partners/vendors, customers, and investors/execs.
Discussing this is a little awkward, but maybe start with alignment, since most of the competence challenges are rooted in mis-alignments: never developing nor selecting for the skills that alignment would require.
> Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).
Was there any literature or other findings that you came across that ended up clicking and working for you that you can recommend to us?
I could blather for hours around this space. A few random highlights:
* The very first thing I read about requirements was Weinberg, and it's still worth reading. (Even if you are a contracting house, with a hopeless client, and you want to go full reactive scrum participatory design, to unblock you for sprints with big blocks of billable hours, not caring how much unnecessary work you do... at least you will know what you're not doing.)
* When interviewing people about business or technical, learn to use a Data Flow Diagram. You can make it accessible to almost everyone, as you talk through it, and answer all sorts of questions, at a variety of levels. There are a bunch of other system modeling tools you can use, at times, but do not underestimate the usefulness and accessibility of a good DFD.
* If you can (or have to) plan at all, find and learn to use a serious Gantt chart centric planning tool (work breakdown, dependencies, resource allocations, milestones), and keep it up to date (which probably includes having it linked with whatever task-tracking tool you use, but you'll usually also be changing it for bigger-picture reasons too). Even if you are a hardware company, with some hard external-dependency milestones, you will be changing things around those unmoveables. And have everyone work from the same source of truth (everyone can see the same Gantt chart and the task
* Also learn some kind of Kanban-ish board for tasking, and have it be an alternative view on the same data that's behind the Gantt view and the tasks/issues that people can/should/are working on at the moment, and anything immediately getting blocked.
* In a rare disruptive startup emergency, know when to put aside Gantt, and fall back to an ad hoc text file or spreadsheet of chaos-handling prioritization that's changing multiple times per day. (But don't say that your startup is always in emergency mode and you can never plan anything, because usually there is time for a Kanban board, and usually you should all share an understanding of how those tasks fit into a larger plan, and trace back to your goals, even if it's exploratory or reactive.)
* Culture of communicating and documenting, in low-friction, high-value, accessible ways. Respect it as team-oriented and professional
* Avoid routine meetings; make it easy to get timely answers and discussion, as soon as possible. This includes reconsidering how accessible upper leadership should be: can you get closer to being responsive to the needs of the work on the project (e.g., if anyone needs a decision from the director/VP/etc., then quickly prep and ask, maybe with an async message, but don't wait for weekly status meeting or to schedule time on their calendar).
* Avoid unnecessary process. Avoid performances.
* People need blocks of time when they can get flow. Sometimes for plowing through a big chunk of stuff that only requires basic competence, and sometimes when harder thinking is required.
* Be very careful with individual performance metrics. Ideally you can incentive everyone to be aligned towards team success, through monetary incentives (e.g., real equity for which they can affect the value) and through culture (everyone around you seems to work as a team, and you like that, and that inspires you). I would even start by asking if we can compensate everyone equally, shun titles, etc., and how close can we practically get to that.
* Be honest about resume-driven-development. It doesn't have to be a secret misalignment. Don't let it be motivated solely as a secret goal of job-hoppers that is then lied about, or it will probably be to the detriment of your company (and also, that person will job-hop, fleeing the mess they made). If you're going to use new resume keyword framework for a project, the whole team should be honest that, say, there's elements of wanting to potentially get some win from it, wanting to trial it for possible greater use and build up organizational expertise in it, and also that it's a very conscious and honest perk for the workers to get to use the new toy.
* Infosec is an unholy dumpster fire, throughout almost the entire field. Decide if you want to do better, and if so, then back it up with real changes, not CYA theatre and what someone is trying to sell you.
* LeetCode frat pledging interviews select for so much misaligned thinking, and also signals that you are probably just more of the same as the low bar of our field, and people shouldn't take you seriously when you try to tell them you want to do things better.
* Nothing will work well if people aren't aligned and honest.
How do you know when to call it quits? How do you know when people are not aligned or honest, or that you are not right for the team, or when the team is not right for the client/project?
How much time is normal for a team/project to get its bearings? (It depends, I know...)
> How much time is normal for a team/project to get its bearings? (It depends, I know...)
If you mean figure out their process, this can happen incrementally, starting simply and building up as you go, with a team of people who are open to that.
It can also happen that some people come in and want to recreate the fleet of tools and best-practice processes that they know from a different company. This isn't my style, but it's not necessarily a bad idea, if everyone is onboard with that. But their prior experience might not fit the different situation (e.g., trying to do things the FAANG way in a post-ZIRP early startup).
If you mean get their bearings on what they're going to build, I don't know, but it usually starts with figuring out your users'/customers' needs, and the business constraints (e.g., resources, milestones that need to be met for revenue or to get funding, etc.).
If there's a new or unfamiliar technology, there might also be exploratory/learning time with that, in parallel (e.g., there's a bias that you will use certain emerging tech or an approach someone thought of, to solve what you initially suspect is the MVP problem, and you have to play with this a bit, and see what you can and can't do with it, in parallel with figuring out what problem you're actually going to solve for MVP).
> How do you know when people are not aligned or honest,
I'm not an expert on this, but do have some conventions and thoughts, so partial off-the-cuff answer...
Of course, at some point, actions will speak pretty clearly about alignment or honesty.
Before that, a lot of tech workers aren't deceptive by default, even if they're approaching the culture assuming a mercenary environment, and they will tell you straight what they are thinking. Maybe especially more if they think you are straight with them. I tend to think you can discuss and find common ground with these people.
Some tech workers have a "California nice" persona that can obscure a few distinct categories, some fine or innocuous, one of them non-deceptive mercenary (once you start talking with them), only one of them deceptive mercenary.
IMHO, just treat everyone honestly, and they will often meet you there, or often indicate when they aren't meeting you there. If you don't meet people honestly, some people will immediately adapt to that too.
I think if you create an alignment-nurturing culture, and communicate and demonstrate it consistently from very first contact, you will scare away a few people, and onboard a lot of people who either are looking for that, or willing to try this unusual thing.
As soon as you start introducing perverse incentives (e.g., individual performance metrics), or mercenary culture (e.g., why is this option pool so small and have worker-hostile terms), or signs of misalignment yourself, or even just sound like more of the same (e.g., "<we're-arm-fuzzy> <we're different> oh btw leetcode frat hazing bro do you even lift"), I think people will revert to the default pretty mercenary tech industry worker culture. And that's rational of them, because the worker's dominant strategy for a mutually-mercenary tech industry environment was figured out a couple decades ago, for a reason.
This is an amazing summary of good advice for software projects! I literally pasted it into my notes for reference. You should write a blog or something on this topic if you don't already.
Thanks for the very kind words. No blog yet. The closest thing to a blog was that I was writing a book, "Racket Tactical Field Manual" (RTFM), which would teach a programming language (Racket) and effective software engineering practices incidentally using Racket tooling, in one tractable, self-contained book (and beg Randall Munroe to do the cover illustration)... But then it looked like Racket was determined not to have commercial uptake, so I abandoned it.
I suppose, if I had a software engineering blog that was fortunate to be well-received, maybe I wouldn't get 90%+ interviews wanting to give me a LeetCode frat hazing. We could instead speak like colleagues, who can see that the other person has, say, written a bunch of open source code, and skip ahead to getting a feel for more important factors about how we might work together. Instead, the signal I get is that they really, really want to do the frat hazing. (Like the AI company I just got off a successful recruiter screen with, minutes ago.)
Having consulted on government projects - especially huge projects spanning dozens of government departments, what I have learnt is that the project is doomed right from the start. The specifications are written in such a way that it is impossible to get a working software which can address all of the millions (yes, literally) of specifications.
For instance, I had the opportunity to review an RFP put out by a state government for software to run a single state government. The specifications stated that a SINGLE software should be used for running the full administration of all of the departments of the government - including completely disparate things such as HR, CCTV management, AI enabled monitoring of rodents and other animals near/at warehouses, all healthcare facilities, recruitment, emergency response services etc...
ONE SOFTWARE for ALL of these!
There isn't a single company in the world who can write software to monitor rodents, hospital appointment booking, general payroll, etc. And since the integration required was so deep, it would be impossible to use existing best-of-breed software.. and everything has to be written from scratch.
How is such a software project ever going to suceeed?
This touches on the absolutely vital issue of domain knowledge. Everybody understands that you're not supposed to have the same people handle sewer maintenance and preschool teaching because these are two entirely separate skillsets. To an extent you can also treat kindergartens and treatment plants as black boxes that consume money and produce desired services.
For people who don't know much about programs it's sort of natural to assume that software engineering works the same way. Put in money and specs, get back programs. But of course it doesn't work like that, because software dev is not a single skillset. To write useful programs, you have to know how to code and understand the environment in which the program will be used.
But can this software monitor patients via CCTV and see if any of them are about to faint and call ER proactively for them? No? then your bid for the project will be discarded! :)
What about the CCTV monitoring software needing to verify if there are women in a particular room and trigger an alarm when too many men enter the area - I am not kidding, but this was really in the spec!
To be fair, that’s a rare exception. Most government tenders are quite narrow in scope.
What I have found is that they’re written by people with zero knowledge of either the solution requirements or the technology! Combine that with zero profit motive and zero personal consequences, and you can end up with total nonsense even on projects with billion dollar budgets.
A state school department here put out a tender for wiring over two thousand schools with fibre, but the way the contract was stipulated only a single applicant could win the contract and most handle every single location across a thousand miles of territory. Hence, only the largest incumbent telco could possibly win… which they did… at 15x the cost of a bunch of local contractors doing the work. This cost something like a billion dollars to taxpayers.
The excuse of the guy writing the tender was “it’s easier for me to get one contract signed than fifty.”
He’s a public servant getting paid $50K. He’s got nothing else on, no other pressing needs or distractions, but he’s too busy, you see? So much easier to waste a billion dollars to save himself a few months of effort.
Most of the examples here are big government IT projects. But it's unfair to single out software projects here. There are a lot of big government projects that fail or face long and expensive delays. A lot of public sector spending is like that. In fact, you'd be hard pressed to find examples where everything worked on time and on budget.
Mostly the issues are non technical and grounded in a lack of accountability and being too big to fail. A lot of these failures are failing top down. Unrealistic expectations, hand wavy leadership, and then that gets translated into action. Once these big projects get going and are burning big budgets and it's obvious that they aren't working, people get very creative at finding ways to tap into these budgets.
Here in Germany, the airport in Berlin was opened only a few years ago after being stuck in limbo a decade after it was supposed to open and the opening was cancelled only 2 weeks before it was supposed to happen. It was hilarious, they had signs all over town announcing how they were going to shut down the highway so the interior of the old airport could be transported to the new one. I kid you not. They were going to move all the check-in counters and other stuff over and then bang on it for a day or two and then open the airport. Politicians, project leadership, etc. kept insisting it was all fine right up until the moment they could not possibly ignore the fact that there was lots wrong with the airport and that it wasn't going to open. It then took a decade to fix all that. There's a railway station in Stuttgart that is at this point very late in opening. Nuclear plant projects tend to be very late and over budget too.
Government IT projects aren't that different than these. It's a very similar dynamic. Big budgets, decision making is highly political, a lack of accountability, lots of top down pretending it's going to be fine, big budgets and companies looking to tap into those, and a lot of wishful thinking. These are all common ingredients in big project failures.
The software methodology is the least of the challenges these projects face.
There's an obvious imminent selection bias with government projects because they're by nature subject to public scrutiny, plus the stakeholders are literally everyone. Private companies can waste billions internally and it'll never make it into the news.
In my first big job in a big legacy company, 30% of ongoing effort was "how to implement this feature which needs a database without a database".
We also paid some security company to use it as a proxy in front of our server to implement some server redirects because it was simpler than configuring our own servers. Simple one-liner conf changes were a week of emails with support staff.
I think if we look at the lack of accountability it's obvious that one major problem is that many of these projects do heavily rely on contract work.
No company or gov in the world can supply the perfect brain- and manpower necessary on day one (on a huge and complex project that requires expert knowledge). So there is an prevalent delusion that talent just spawns at project kickoff and those people even care about what they do.
Maybe this is some artifact we carried over from the industrial era. We expect that complex machinery is built by expert companies over night and they just work, with a bit of maintenance and knowledge transfer. But software doesn't work like that.
Software projects fail because humans fail. Humans are the drivers of everything in our world. All government, business, culture, etc... it's all just humans. You can have a perfect "process" or "tool" to do a thing, but if the human using it sucks, the result will suck. This means that the people involved are what determines if the thing will succeed or fail. So you have to have the best people, with the best motivations, to have a chance for success.
The only thing that seems to change this is consequences. Take a random person and just ask them to do something, and whether they do it or not is just based on what they personally want. But when there's a law that tells them to do it, and enforcement of consequences if they don't, suddenly that random person is doing what they're supposed to. A motivation to do the right thing. It's still not a guarantee, but more often than not they'll work to avoid the consequences.
Therefore if you want software projects to stop failing, create laws that enforce doing the things in the project to ensure it succeeds. Create consequences big enough that people will actually do what's necessary. Like a law, that says how to build a thing to ensure it works, and how to test it, and then an independent inspection to ensure it was done right. Do that throughout the process, and impose some kind of consequence if those things aren't done. (the more responsibility, the bigger the consequence, so there's motivation commensurate with impact)
That's how we manage other large-scale physical projects. Of course those aren't guaranteed to work; large-scale public works projects often go over-budget and over-time. But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process to encourage humans to do the right thing.
> Software projects fail because humans fail. Humans are the drivers of everything in our world.
Ah finally - I've had to scroll halfway down to find a key reason big software projects fail.
<rant>
I started programming in 1990 with PL/1 on IBM mainframes and for 35 years have dipped in and out of the software world. Every project I've seen fail was mainly down to people - egos, clashes, laziness, disinterest, inability to interact with end users, rudeness, lack of motivation, toxic team culture etc etc. It was rarely (never?) a major technical hurdle that scuppered a project. It was people and personalities, clashes and confusion.
</rant>
Of course the converse is also true - big software projects I've seen succeed were down to a few inspired leaders and/or engineers who set the tone. People with emotional intelligence, tact, clear vision, ability to really gather requirements and work with the end users. Leaders who treated their staff with dignity and respect. Of course, most of these projects were bland corporate business data ones... so not technically very challenging. But still big enough software projects.
Gez... don't know why I'm getting so emotional (!) But the hard-core sofware engineering world is all about people at the end of the day.
> big software projects I've seen succeed were down to a few inspired leaders and/or engineers who set the tone. People with emotional intelligence, tact, clear vision, ability to really gather requirements and work with the end users. Leaders who treated their staff with dignity and respect.
I completely agree. I would just like to add that this only works where the inspired leaders are properly incentivized!
> But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process
If there was sufficient consequence for this stuff, no one would ever take on any risk. No large works would ever even be started because it would be either impossible or incredibly difficult to be completely sure everything will go to plan.
So instead we take a medium amount of caution and take on projects knowing it's possible for them to not work out or to go over budget.
If software engineers want to be referred to as "engineers" then they should actually learn about engineering failures. The industry and educational pipeline (formal and informal) as a whole is far more invested in butterfly chasing. It's immature in the sense that many people with decades of experience are unwilling to adopt many proven practices in large scale engineering projects because they "get in the way" and because they hold them accountable.
Surely you mean managers, right? Most developers I interact with would love to do things the right way, but there's just no time, we have to chase this week's priority!
Fundamentally this is not a statement about programming or software. It is a statement that management at almost all companies is abysmally inept and are hardly ever held to account.
Most sizeable software projects require understanding, in detail, what is needed by the business, what is essential and what is not, and whether any of that is changing over the lifetime of the project. I don't think I've ever been on a project where any of that was known, it was all guess work.
Management is always a huge problem, but software engineers left to their own devices can be just as bad.
I very rarely hear actual technical reasons for why a decision was made. They're almost always invented after the fact to retroactive justify some tool or design pattern the developer wanted to use. Capabilities and features get tacked on just because it's something someone wanted to do, not because they solve an actual problem or can be traced back to requirements in any meaningful way.
Frankly as an industry we could learn a lot from other engineering fields, aerospace and electrical engineering in particular. They aren't perfect, but in general they're much better at keeping technical decisions tied to requirements. Their processes tend to be too slow for our industry of course, but that doesn't mean there aren't lessons to be learned.
Exactly this. Not just large software projects tend to fail often; also large architectural and infrastructure projects do.
There are loads of examples, one famous one for instance is the Berlin Airport.
Management is bad at managing large projects. Whatever those projects are. In particular when third parties are involved that have a financial interest.
This is precisely the point of the article. I mean, it's right there at the top in that weird arrow-shaped infographic. It's _almost_ always a management issue.
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not.
I guess that’s the real problem I have with SV’s endemic ageism.
I was personally offended, when I encountered it, myself, but that’s long past.
I just find it offensive, that experience is ignored, or even shunned.
I started in hardware, and we all had a reverence for our legacy. It did not prevent us from pursuing new/shiny, but we never ignored the lessons of the past.
Why do you find it offensive? It’s not personal. Someone who thought webvan was a great lesson in hubris could not have built an Instacart, right? Even evolution shuns experience, all but throwing most of it out each generation, with a scant few species as exceptions.
> Someone who thought webvan was a great lesson in hubris could not have built an Instacart, right?
Not at all. The mistake to learn from in Webvan's case was expanding too quickly and investing in expensive infrastructure all before achieving product-market fit. Not that they delivered groceries.
It was a deliberate troll. It’s kind of flattering, actually, because you’ve taken the time to get to know me, and you figured it would goad me (it didn’t, but nice try). The throwaway was a nice touch. Shows class (and intent).
I think you're mistaking the funding and starting of companies with the execution of their vision through software engineering -- the entire point of the article, and the OP.
This is a classic straw man argument, which depends on the assumption that all people of a certain age would think a certain way.
Also, your understanding of evolution is incorrect. All species on Earth are the results of an enormous amount of accumulated "experience", over periods of up to billions of years. Even the bacteria we have today took hundreds of millions of years to reach anything similar to their current form.
I have never seen an industry that works so hard to self-immolate.
right now the industry is spending billions / trillions of $ on to train A.I on badly written open source code.
in universities we teach kids - DSA - but never about to really think about scoping work nor even the unix principle of how software should compose, how to prevent errors etc. hell how many working professionals know about the 10 NASA principles & actually use them in practice ?
we encourage the bright kids to go work at places which are the cathedrals of complexity - but never seeking simple solutions. & again merchants of complexity get paid more - find it easier to find jobs etc.
the tragedy is the failures are documented, but also the fixes to failures as well.
soon enough we're gonna raise a whole generation who doesn't know how to make reliable, robust software from scratch cz of 'vibecoding'. then ultimately civilization collapses.
I think GP means NASA's 10 rules, but these are specifically for C development in mission-critical systems. There's been the odd software issue at NASA, but supposedly their 10 rules should make C projects much safer.
So, I'm not a dev nor a project manager but I found this article very enlightening. At the risk of asking a stupid question and getting a RTFM or a LMGTFY can anyone provide any simple and practical examples of software successes at a big scale. I work at a hospital so healthcare specific would be ideal but I'll take anything.
FWIW I have read The Phoenix Project and it did help me get a better understanding of "Agile" and the DevOps mindset but since it's not something I apply in my work routinely it's hard to keep it fresh.
My goal is to try and install seeds of success in the small projects I work on and eventually ask questions to get people to think in a similar perspective.
Even though there were some benefits to the modularity of Multics (apparently you could unload and replace hardware in Multics servers without reboot, which was unheard of at the time), it was also its downfall. Multics was eventually deemed over-engineered and too difficult to work with. It couldn't evolve fast enough with the changing technological landscape. Bell Labs' conclusion after the project was shelved was that OSs were too costly and too difficult to design. They told engineers that no one should work on OSs.
Ken Thompson wanted a modern OS so he disregarded these instructions. He used some of the expertise he gained while working on Multics and wrote Unix for himself (in three weeks, in assembly). People started looking over Thompson's shoulder being like "Hey what OS are you using there, can I get a copy?" and the rest is history.
Brian Kernighan described Unix as "one of" whatever Multics was "multiple of". Linux eventually adopted a similar architecture.
If you click on the link, I mention other competing attempts and architectures, like Multics, Hurd, MacOS and even early Windows that either failed or started adopting Unix patterns.
This is a noble and ambitious goal. I feel qualified to provide some pointers, not because I have been instrumental in delivering hugely successful projects, but because I have been involved, in various ways, in many, many failed projects. Take what you will from that :-)
- Define "success" early on. This usually doesn't mean meeting a deadline on time and budget. That is actually the start of the real goal. The real success should be determined months or years later, once the software and processes have been used in a production business environment.
- Pay attention to Conways Law. Fight this at your peril.
- Beware of the risk of key people. This means if there is a single person who knows everything, you have a risk if they leave or get sick. Redundancy needs to be built into the team, not just the hardware/architecture.
- No one cares about preventing fires from starting. They do care about fighting fires late in the project and looking like a hero. Sometimes you just need to let things burn.
- Be prepared to say "no", alot. (This is probably the most important one, and the hardest.)
- Define ownership early. If no one is clearly responsible for the key deliverables, you are doomed.
- Consider the human aspect as equally as the technical. People don't like change. You will be introducing alot of change. Balancing this needs to be built into the project at every stage.
- Plan for the worst, hope for the best. Don't assume things will work the way you want them to. Test _everything_, always.
>No one cares about preventing fires from starting. They do care about fighting fires late in the project and looking like a hero. Sometimes you just need to let things burn.
As a Californian, I hate this mentality so much. Why can't we just have a smooth release with minimal drama because we planned well? Maybe we could properly fix some tech debt or even polish up some features if we're not spending the last 2 months crunching on some showstopper that was pointed out a year ago.
I find it kind of hard to define success or failure. Google search and Facebook are a success right? And they were able to scale up as needed, which can be hard. But the way they started is very different from a government agency or massive corporation trying to orchestrate it from scratch. I don't know if you'd be familiar with this, but maybe healthcare.gov is a good example... it was notoriously buggy, but after some time and a lot of intense pressure it was dealt with.
The untold story is of landing software projects at Google. Google has landed countless software projects internally in order for Google.com to continue working, and the story of those will never reach the light of day, except in back room conversations never to be shared publicly. How did they go from internal platform product version one to version two? it's an amazing feat of engineering that can't be shown to the public, which is a loss for humanity, honestly, but capitalism isn't going to have it any other way.
Are you saying this from firsthand experience? Because it sounds like the sort of myth that Google would like you to believe. Much more believable is that their process is as broken and chaotic as most software projects are, they are just so big that they manage to have some successes regardless. Survivorship bias. A broken clock is still right twice a day.
I was an SRE on their Internet traffic team for three years, from 2020 til 2023. The move from Sisyphus to Legislator is something I wish the world could see documented in a museum, like the moving of the Cape Hatteras Lighthouse.
That's my entire industry, so I can believe it. I'd love to learn large scale game architecture but it simply isn't public. At best you can dig into the source available 30 year legacy code of Unreal Engine as a base. But extracting architecture from the source is like looking at a building without a schematic.
Your best bet is a 500 dollar GDC vault that offers relative scraps of a schematic and making your own from those experiences.
Have you seen the presentation from GDC 2017 on the architecture of Overwatch [0]? If you watch the video in detail -- stepping through frame-by-frame at some points -- it provides a nearly complete schematic of the game's architecture. That's probably why the video has since been made unlisted.
I don't think you should focus on successful large projects. Generally you should consider that all big successes are outliers from a myriad of attempts. They have been lucky and you can't reproduce luck.
I'd like try to correct your course a bit.
DevOps is a trash concept, that had good intentions. But today it's just an industry cheatcode to fill three dev positions with a single one that is on pager duty. The good takeaways from it: Make people care that things work end to end. If Joe isn't caring about Bob's problems, something is off. Either with the process, or with the people.
Agile is a very loose term nowadays. Broadly spoken it's the opposite of making big up front plans and implement them in a big swipe. Agile wants to start small and improve it iteratively as needed. This tends to work in the industry, but the iterative time buckets have issues, some teams can move fast in 2 week cycles, others don't. The original agile movement also wanted to give back control and autonomy back to those who actually do stuff (devs and lower management). This is very well intended and highly functional, but is often buried or ignored in corporate environments. Autonomy is extremely valuable, it motivates people and fosters personal growth, but being backed by a skilled peers also creates psychological safety. One of the major complaints I hear about agile practices is that there are too many routines, meetings and other in person tasks with low value that keep you from working. This is really bad and in my perception was never intended, but companies love that shit. This part is about communication, make it easy for people to share and engage, while also keeping their focus hours high. Problems have to bubble up quickly and everyone should be motivated and able to help solving them. If you listen to agile hardliners, they will also tell you that software can't be reliably planned, you won't make deadlines, none of them, never. That is very true, but companies are unable to deal with it.
I don't like this as a metric of success, because who came up with the budget in the first place?
If they did a good job and you're still 97% over then sure, not successful.
But if the initial budget was a dream with no basis in reality then 97% over budget may simply have been "the cost of doing business".
It's easier to say what the budget could be when you're doing something that has already been done a dozen times (as skyscraper construction used to be for New York City). It's harder when the effort is novel, as is often the case for software projects since even "do an ERP project for this organization" can be wildly different in terms of requirements and constraints.
That's why the other comment about big projects ideally being evolutions of small projects is so important. It's nearly impossible to accurately forecast a budget for something where even the basic user needs aren't yet understood, so the best way to bound the amount of budget/cost mismatch is to bound the size of the initial effort.
"Worse" won't even start to describe the economical crisis we will be in once the bubble bursts.
And although that, in itself, should be scary enough, it is nothing compared to the political tsunami and unrest it will bring in its wake.
Most of the Western world is already on shaky political ground, flirting with the extreme-right. The US is even worse, with a pathologically incompetent administration of sociopaths, fully incapable of coming up with the measures necessary to slow down the train of doom careening out of control towards the proverbial cliff of societal collapse.
If the societal tensions are already close to breaking point now, in a period of relative economical prosperity, I cannot start to imagine what they will be like once the next financial crash hits. Especially one in the multi trillion of dollars.
They say that humanity progresses through episodes of turmoil and crisis. Now that we literally have all the knowledge of the world at our fingertips, maybe it is time to progress past this inadequate primeval advancement mechanism, and to truly enter an enlightened age where progress is made from understanding, instead of crises.
Unfortunately, it looks like it's going to take monumental changes to stop the parasites and the sociopaths from making at quick buck at the expense of humanity.
Not really, by most indications AI seems to be an amplifier more than anything else. If you have strong discipline and quality control processes it amplifies your throughput, but if you don't, it amplifies your problems. (E.g. see the DORA 2025 report.)
So basically things will still go where they were always going to go, just a lot faster. That's not necessarily a bad thing.
I meant it more as an observation than an optimistic prediction, really :-)
The article is sound, but it's focus on large public failures disregards the vast, vast, vast majority of the universe of software projects that nobody really thinks about, because they mostly just work -- websites and mobile apps and games and internal LoB CRUD apps and cloud services and the huge ecosystem of open source projects and enterprise and hobby software.
Without some consideration of that, we cannot really generalize this article to reflect the "success rate" of our industry.
That said, I think the acceleration introduced by AI is overall a "Good Thing (tm)" simply because, all else being equal, it's generally better to fail faster rather than later.
"If you do everything right that you weren't doing before, but with 80% fewer people and the Lie Generator that doesn't work, then you will be successful."
Edited to add: To clarify, I meant that if an organization was going to deliver a billion-dollar boondoggle of a project, AI will not change that outcome, but it WILL help deliver that faster. Which is why I meant it's not necessarily a bad thing, because as in software, it's generally better to fail faster.
Also, anything that T. Capers Jones wrote. The most comprehensive one of these books is this:
Estimating Software Costs: Bringing Realism to Estimating
Hardcover
ISBN-13978-0071483001
Many believe the official recognition of the crisis in developing software were the two NATO conferences in 1968 and 1969.
See the Wikipedia article on the History of Software Engineering.
There have been two small scale experimental comparisons of the waterfall formal model (requirements, design, code, test) and the more prototyping and agile method. They seem to have the same productivity in terms of lines per programmer-month but the formal method tends to produce larger software.
I've started calling it EDD - Executive Driven Development.
Senior stakeholders get into a room, decide they need xyz for the project to really succeed, push this down to managers who in turn try perform miracles with what little development resource they have. Very often they also have an offshore team who is only concerned with prolonging the contract as much as possible, rather than delivering. 2 weeks later senior stakeholders get back into the room...
Oh they TRY to ... it's just that the "non-technical unqualified people" get brought to heal (usually) by regulations. I've been in the room where people have tried to force a decision and a PEng, Lawyer, or CA/CPA had to say "absolutely not". It happens all the time, which is why you NEED regulations.
>Global IT spending has more than tripled in constant 2025 dollars since 2005, from US $1.7 trillion to $5.6 trillion, and continues to rise. Despite additional spending, software success rates have not markedly improved in the past two decades.
Okay but how much more software is used ? If IT spending has tripled since 2005 but we use 10x more software I'd say the trend is good.
Success rates imply a ratio. Constant dollars are adjusted.
Yes there is a lot more spending overall. But nothing improved quality wise, despite everyone in software somehow says they "make software better". (Which is phrased by people that don't do software, but own it.)
The point is not that the growth of IT spending is bad. That was just to show the scale of spending.
The point of the article is that, a billion spent on software could well lead to a loss of hundred billion.
The failed UK NHS IT project, known as the National Programme for IT (NPfIT), cost the UK government over £10 billion and produced almost nothing of value. I'm surprised that didn't get a mention.
Again those bastards, Fujitsu, were involved. They even sued UK government and won £465 million settlement when their contract was cancelled. But, despite this and their complicity in covering up the failures of the Horizons posts office system, the UK government is still giving them fat contracts.
If senior managers can preside over a massive failure and walk away with a huge pension, there isn't much incentive for them to do better, is there?
The GOV.UK project is a rare success story for IT in the UK government. They need to take that team, give them big incentives to stay, and give them more projects. Why are we outsourcing to international companies that don’t give a shit when we have a wealth of talent at home? Why aren’t we investing in our own people?
The people who did the UK COVID app also did a good job, as far as I am aware. The lessons seems to be that it is better to employ a small, experienced and talented team and get out of their way, than outsource to a huge government contractor.
Despite its overall failure, some parts of the infrastructure and national applications, such as the Summary Care Record and the Electronic Prescriptions Service, are considered to have survived and continue to be used
The difference between success and failure of large projects comes down to technical leadership. I've seen it time and time again. Projects that are managed by external consulting companies (name brand or otherwise) have a very poor track record of delivering. An in-house technical lead that is committed to the success of the project will always do better. And yes, this technical lead must have the authority to limit the scope of the system rewrite. Endless scope creep is a recipe for failure. Outside consulting firms will never say "No" to any new request - it means more business for them - their goals are not aligned with the client.
Do non-software projects succeed at a higher rate in any industry? I get the impression that projects everywhere go over time, over budget, and frequently get canceled.
How many bridges have you used that have collapsed? How much software have you used that has been broken or otherwise not served your interests? If we built the rest of society like we build software, millions of people would be dead.
The reason bridges don't fail often is because they over-build them. There's no obvious equivalent with software. One mistake in a large code base can make it fail.
Bridges to go ridiculously over budget and schedule all the time, however.
the UK Post Office scandal would be the equivalent of the Morandi bridge collapsing - the big, catastrophic failure you hope to see few times in your lifetime.
but bridges collapsing is not the only failure mode for non software projects. I know plenty of newly built houses that had serious issues with insulation, wiring, painting, heating infrastructure, etc.
Systematic decimation of test teams, elimination of test managers, and contemptuous treatment of the role of tester over the past 40 years has not yet led to a more responsible software industry. But maybe if we started burning testers at the stake all these problems will go away?
Many specialties were eliminated / absorbed over the past few decades. I started working almost 30 years ago. Today, I rarely see dedicated testers, just like I rarely see dedicated DBAs. Sysadmins went away with the "DevOps" movement. Now they are cloud engineers who are more likely to understand a vendor-specific implementation than networking fundamentals.
Except testers are needed. Testing is not merely a technical role. It's a social role. It's a commitment to think differently from the rest of the team. We do this because that provides insurance against complacency.
But, by the nature of testing, we testers are outsiders. No one is fully comfortable with a tester in the room, unless they are more afraid of failure than irritation.
Oh, it's much more interesting than that. Phoenix started as an attempt to create a gun registry. Ottawa had a bunch of civil servants that'd be reasonably compotent at overseeing such a thing, but the government decided that it wanted to build it in Miramichi, New Brunswick. The relevant people refused to move to Miramichi, so the project was built using IBM contractors and newbies. The resulting fiasco was highly predictable.
Then when Harper came in he killed the registry mostly for ideological reasons.
But then he didn't want to destroy a bunch of jobs in Miramichi, so he gave them another project to turn into a fiasco.
Sure, that's obvious. What isn't obvious is why it was killed. It could have been killed because it was a giant cluster@#$%, that would have been reason enough. But if it would have been killed for that reason, they wouldn't have let the cluster%$## spread.
New Zealand tried to do a new payroll system for teachers called Novopay which imploded spectacularly and is still creating grief. The system is now called EdPay (the government decided to take over the privately created system). The total cost of the debacle was north of $200M NZD. Somehow they managed to fail to replace a working system!
As someone that has seen technological solutions applied when they make no sense, I think the next revolution in business processes will be de-computerization. The trend has probably already started thank to one of the major cloud outages.
> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions.
Somehow I come away skeptical of the inevitable conclusion that Phoenix was doomed to fail and instead that perhaps they were hamstrung by architecture constraints dictated by assholes.
Arbitrary payroll is absurdly complicated. The trick is to not make it arbitrary - have a limited amount of stuff you do, and always have backdoors to manually pushing data through payroll.
Sounds like they put zero effort into simplifying those rules the first time around.
Now in the new project they put together a committee to attempt it
> The main objective of this committee also includes simplifying the pay rules for public servants, in order to reduce the complexity of the development of Phoenix's replacement. This complexity of the current pay rules is a result of "negotiated rules for pay and benefits over 60 years that are specific to each of over 80 occupational groups in the public service." making it difficult to develop a single solution which can handle each occupational groups specific needs.
I have worked on government payroll systems, simplifying those rules is almost impossible from political PoV. They are generally a combo of weird laws, court cases, union contracts and more.
Any time you think about touching them, the people who get those salaries come out in droves and no one else cares so government has every incentive to leave them alone.
Committees are how you discover what the problems are and agree solutions.
No single person is going to understand all of the history and legality involved, or be able to represent the people on all sides of this mess.
Yes, this means discussion, investigation, almost certainly months of effort to find something that works, and lots of compromise. That's how adults deal with complex situations.
Because you don’t just rewrite all your payroll systems with hundreds of variations in one go. That will never work. But they keep trying it.
You update the system for one small piece, while reconciling with the larger system. Then replace other pieces over time, broadening your scope until you have improved the entire system. There is no other way to succeed without massive pain.
Software development, like most other things, are part of the same make-believe market, that we run our societies in, in most countries around the world. Lets face it, most of the big money in software is believe money, not actual proven value of a thing. The word "evaluation" sort of already tells us this. It's not fact checking "How much did they sell?" or "How many users bought access or a license?", it is "How much do we believe in the future of this thing?" and risky investment "How much could we make, if this thing takes off?".
For software, I am not sure this is helpful. Maybe we would develop way less trash software, if it was different. But then again we would probably still develop engagement farming software, because people would still use or buy that.
In the book "How Big Things Get Done" [1], Bent Flyvbjerg, among other things, identifies one common feature of the projects that do not have large outliers to go over-budget and under-deliver: modularity. Ideally, fractal modularity. His favorite examples: solar power, electric transmission, pipelines, roads. Ironically, IT/software is only slightly better than nuclear power and Olympic games [2].
> Finally, project cost-benefit justifications of software developments rarely consider the financial and emotional distress placed on end users of IT systems when something goes wrong.
Most users and most management of software projects live in denial that the norm is dystopia.
I can’t help think of any required and useful feature that has happened in computer usage since the early days.
Easier to swallow is that the user interface of desktop operating systems hasn’t changed fundamentally in many years, yet hardware requirements continue to grow.
But even the invention of a mouse requires excessive movement to move a pointer to click on something that a key combination could’ve done much more quickly. The original intention of the mouse was just as another device to use, not necessarily a primary device to direct the majority of workflow.
From a dark storage area I may someday again get out an early Sceptre gaming monitor from the DOS days.
I held on to it throughout the 1990's precisely because it was not a plug & play monitor and it was real good to install Windows with so nothing would interfere with higher resolution alternative graphics you were going to install later.
Now by the 21st century it was seldom seen but these were well-made and it still worked, however the most obsolete feature that got the most interest was the sleek aftermarket plastic accessory unit attached to the side of the monitor with those sticky 3M tacky pads that are so tenacious.
Yes, you've all seen it and remember it fondly, the mouse holder.
Kind of like a custom cup holder that fits the standard mouse perfectly, it's obviously where you keep your mouse most of the time, except for those rare occasions when you dabble in a bit of software which actually supports a mouse.
You want to keep it out of the way of your everyday desktop activities :)
Please forgive me if I get something wrong. Not a native English speaker. The article boils it down to: all is a management failure. This is also my feeling after 35 years in software development. There are no such thing than a competent middle or upper management in software development. I see sometimes even devs being promoted and in an instant forget how software is made. In the other hand I see promotion of the most stupid dev. Al this leads to massive Missmanagements and hiding of problems to the upper managers. Even worse sometimes I see the best devs promoted only to watch them break because the toxin they get from there managers kills them
>Frustratingly, the IT community stubbornly fails to learn from prior failures.
So far I believe that there has been too much emphasis in education on coding and algorithms (small scale/tactical stuff) and not enough emphasis on the engineering side of things like version control, QA, system design, management etc. I think the situation has changed (40 years ago most professional programmers didn't even know what version control was, let alone use vc systems) but the scope of the projects has increased faster than our skills and tools.
This is what I’ve been thinking about when I talk to other people in software development when they can’t stop talking about how efficient they are with AI… yet they didn’t ship anything in their startup, or side project, or in a corporate setting, the project is still bug riddled, the performance is poor, now there code quality suffers too as people barely read what Cursor (etc) are spitting out.
I have “magical moments” with these tools, sometimes they solve bugs and implement features in 5 minutes that I couldn’t do in a day… at the same time, quite often they are completely useless and cause you to waste time explaining things that you could probably just code yourself much faster.
Frederick Brooks in his essay "No Silver Bullet" (included in the collection Mythical Man Month) talked about the conventions of software development and I recall had called for taking an iterative approach to software development similar to what I had followed for the Automunge project, I went into a little more detail about that in my 2019 essay of the same name:
https://medium.com/automunge/no-silver-bullet-95c77bc4bde1
Yup, and with an equal amount of mindblowing-units-of-money spent, infrastructure projects all around me are still failing as well, or at least being modified (read: downsized), delayed and/or budget-inflated beyond recognition.
So, what's the point here, exactly? "Only licensed engineers as codified by (local!) law are allowed to do projects?" Nah, can't be it, their track record still has too many failures, sometimes even spectacularly explosive and/or implosive ones.
"Any public project should only follow Best Practices"? Sure... "And only make The People feel good"... Incoherent!
Ehhm, so, yeah, maybe things are just complicated, and we should focus more on the amount of effort we're prepared to put in, the competency (c.q. pay grade) of the staff we're willing to assign, and exactly how long we're willing to wait prior to conceding defeat?
Large scale systems tend to fail. large centralised and centrally managed systems with big budgets and large numbers of people who need to coordinate, lots of people with an interest in the project pushing and lobbying for different things.
Multiple smaller systems is usually a better approach, where possible. Not possible for things like transport infrastructure, but often possible for software.
> Not possible for things like transport infrastructure
It depends what you define as a system. Arguably a lot of transport infrastructure is a bunch of small systems linked with well-understood interfaces (e.g. everyone agrees on the gauge of rail that's going to be installed and the voltage in the wires).
Consider how construction works in practice. There are hundreds or thousands of workers working on different parts of the overall project and each of them makes small decisions as part of their work to achieve the goal. For example, the electrical wiring of a single train station is its own self-contained system. It's necessary for the station to work, but it doesn't really depend on how the electrical system is installed in the next station in the line. The electricians installing the wiring make a bunch of tiny decisions about how and where the wires are run that are beyond the ability of someone to specify centrally - but thanks to well known best practices and standards, everything works when hooked up together.
Yes, I should have said "projects" rather than "systems".
Its when you need to coordinate it lots of bits. if you can modularise it its fine. If you have lots of small parts your replicate repeatedly its fine. So Each station is separate, they you lots of common components, trains come off a production line etc.
However building a single complex system as one project is the problem. A long distance railway line, especially if going through populated areas or difficult terrain that imposes constraints, for example.
In manufacturing there are economies of scale and adding more people increases workforce, in mindfacturing there are diseconomies of scale and adding more people increases confusion, yet many managers view software with a manufacturing mindset.
Nailed it, but I fear this wisdom will be easily passed by by someone who doesn’t already intuit it from years of experience. Like the Island de la Muerta: wisdom that can only be found if you already know where it is.
I'm pretty sure that we can remove the word "software" from the article headline and it remains just as true. I don't believe that software projects are unique in this regard: big, complex projects are big and complex, and prone to unexpected issues, scope creep, etc. Throw in multiple stakeholders, ineffective management, the sunk cost fallacy etc. and it's a wonder that any large projects get finished at all.
>For the foreseeable future, there are hard limits on what AI can bring to the table in controlling and managing the myriad intersections and trade-offs
?? I don’t think if thinks stopped advancing in terms of significant model improvements that actual utility would be saturated for a while. We have barely begun to consolidate potential into the tooling, use cases, knowledge sharing and depth of that knowledge throughout the workforce on how to make best use.
If someone is looking at AI as a monolith in thing and thinking “oh, silver bullet to the problems of enterprise software etc” then, I really don’t know what to say except that’s on them, not on any true big claims being pushed unless you’re breaking out the long ladders to pick those cherries, or listening to people whose background and placement within things clearly makes them a bad messenger.
Plausible article, but it reads like a preschooler frustrated that his new toy is broken. "Fix it! Make it work!" - without ever specifying how.
Granted, this is an exceedingly hard problem, and I suppose there's some value in reminding ourselves of it - but I'd much rather read thoughts on how to do it better, not just complaints that we're doing it poorly.
Looking at other domains where companies are developing complex products in highly regulated industries there’s one thing they all share in common, they invest a lot of capital in infrastructure for testing their designs. I spent years at a company trying to convince upper management in setting up a lab where we could simulate a production environment that would allow us to do a real integration test. It’s an idea hard to sell because testing is actually part of the budget in every project, so lack of testing couldn’t be attributed to our high rate of failures (going over budget fixing bugs during commissioning). Perhaps we should stop calling unit testing, testing so that we don’t confuse people. Until you we don’t put all the pieces together and do a proper stress test under close-to-realistic production conditions, our software cannot be considered tested. I think that’s the case for 99% of software companies.
> “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
This should be a criticism of the kinds of bloated firms that take on large government projects, the kinds of people they hire, the incentives at play, the bidding processes, the corruption and all the rest. It has very little to do with software and more just organizations that don't face any pressure to deliver.
I wonder how much software project failure comes from lacking clear processes. Many teams, whether in companies or open source projects, never define step by step procedures for common tasks like feature development, bug triage, code review, or architectural decisions. When six developers each follow their own approach, even with consistent code style, the team can’t improve the system in a predictable and systematic way. Clear procedures don’t guarantee success, but without them teams often end up with chaos and inconsistent delivery. This lack of structured methodology seems far more common in software engineering than in other engineering disciplines.
> "Why worry about something that isn’t going to happen?”
Lots to break down in this article other than this initial quotation, but I find a lot of parallels in failing software projects, this attitude, and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.
It was a combination of failures like this. Why was the captain going full speed ahead into a known ice field? Well, the boat can't sink and there (may have been) organizational pressure to arrive at a certain time in new york (aka, imaginary deadline must be met). Why wasn't there enough life jackets and boats for crew and passengers? Well, the boat can't sink anyway, why worry about something that isn't going to happen? Why train crew on how to deploy the life rafts and emergency procedures properly? Same reason. Why didn't the SS Californian rescue the ship? Well, the 3rd party Titanic telegraph operators had immense pressure to send telegrams to NY, and the chatter about the ice field got on their nerves and they mostly ignored it (misaligned priorities). If even a little caution and forward thinking was used, the death toll would have been drastically lower if not nearly nonexistent. It took 2 hours to sink, which is plenty of time to evacuate a boat of that size.
Same with software projects - they often fail over a period of multiple years and if you go back and look at how they went wrong, there often are numerous points and decisions made that could have reversed course, yet, often the opposite happens - management digs in even more. Project timelines are optimistic to the point of delusion and don't build in failure/setbacks into schedules or roadmaps at all. I've had to rescue one of these projects several years ago and it took a toll on me I'm pretty sure I carry to this day, I'm wildly cynical of "project management" as it relates to IT/devops.
> and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.
But the rest of your comment reveals nothing novel other than anyone would find after watching James Cameron's movie multiple times.
I suggest you go to the original inquiries (congressional in the US, Board of trade in the UK). There is a wealth of subtle lessons there.
Hint: Look at the Admiralty Manual of Seamanship that was current at that time and their recommendations when faced with an iceberg.
Hint: Look at the Board of Trade (UK) experiments with the turning behaviour of the sister ship. In particular of interest is the engine layout of the Titanic and the attempt by the crew, inexperienced with the ship, to avoid the iceberg. This was critical to the outcome.
Hint: Look at the behaviour of Captain Rostron. Lots of lessons there.
Thanks for your feedback, I’m well aware of the inquiries and the history there. However, this post was meant to be a simple analogy that related to the broader topic, not a deep dive into the theories of how and why the titanic sank. Thanks!
The lesson from “big software projects are still failing” isn’t that we need better methodologies, better project management, or stricter controls. The lesson is "don't do big software projects".
Software is not the same as building in the physical world where we get economies of scale.
Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)
Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software.
Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale.
Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process.
Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared.
It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself.
(Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally).
We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix.
The lesson is: "never build big software".
(Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)
I agree with you on "don't do big software project" Specially do not fast scale them out to hundreds of people. You have to scale them more organically ensuring that every person added is a net gain. They think that adding more people will reduce the time.
I am surprised on the lack of creativity when doing these projects. Why don't they start 5 small projects building the same thing and let them work for a year. At the end of the year you cancel one of the projects, increasing the funding in the other four. You can do that every year based on the results. It may look like a waste but it will significantly increase your chances of succeeding.
>Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)
Build 1000 JSON parsers and tell me if the next one isn't cheaper to develop with "(we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.)"
>Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics)
Uh, maybe fewer but none is way to far. Get 2 billion integer operations per second out of a 286, the 500 mile email, big data storage, etc. Physical limits are everywhere.
>It's also the same reason why enterprise software sucks.
The reason enterprise software sucks is because the lack of introspection and learning from the garbage that went before.
Working on AI that helps to manage IT shops that learns from failure & success might be better for both results and culture than most IT management roles, a profession (painting an absurdly broad brush) that tends to attract a lot of miserable creatures.
LLMs themselves don’t learn but AI systems based around LLMs can absolutely learn! Not on their own but as part of a broader system: RLHF leveraging LoRAs that get re-incorporated as model fine tunings regularly, natural language processing for context aggregation, creative use of context retrieval with embeddings databases updated in real time, etc.
A slightly different take, its probably more of people failure, the lack of required expertise, skillset, motivation and coordination. People have motivations to do the job to make a living, success of any long term project is rarely the driving factor for most people working on it. People would know ahead of time when a project is going towards the direction of failure, its just how the things are structured.
From systems perspective, an unknown system/requirement would be a good example where you build iteratively, a known set of requirements should give good enough idea about the feasibility and rough timelines even if its complex.
I like that the author propagates software developer liability. That makes sense. Unless we introduce such system, the incentives are not there to avoid failure
there you are why its failling, the fact that these system is overly massive and complex that sometimes the original creator and architecture that design this system cant foresee the future needs and etc
You could says its incompetence but the fact that software change so much is last 20 years make it most people cant really design a "future proof" system in a way that it cant cause trouble in the future
>By then, the general public and the branch managers themselves finally joined Computer Weekly’s reporters (who had doggedly reported on Horizon’s problems since 2008) in the knowledge that there was something seriously wrong with Horizon’s software.
Computer Weekly first broke the story, for which they deserve much credit. But I believe Private Eye did much of the long term campaigning.
It's good that the author makes the distinction between developers and managers. This distinction is rarely made and most media outlets talk about the wrongdoings of developers, who are almost never the decision makers of failing projects. It's quite the opposite, they are the ones who if brave enough criticize bad management practices and the lack of proper management of the software project.
The most mind boggling number in this article to me was PeopleSoft claiming it would cost $500 million to make a payroll system for the Canadian government. That’s thousands of working years of software developers. It’s such a huge scale that it seems pretty clear the project should never start. PeopleSoft should have been dumped and the project’s scope massively reevaluated.
The main problem are incentives and risks: in most of the cases you are not incentivized to build secure and reliability SW because, most of the time, it's easy to fix it. With particular categories of SW(eg. one distributed on remote system, medical sw, military sw) or HW it's the opposite: a failure it's not so easy to fix so you are incentivized to do a better job.
Failure typically comes from two directions. Unknown and changing requirements, and management that relies on (often external) technical (engineering) leadership that is too often incompetent.
These projects are often characterized by very complex functional requirements, yet are undertaken by those who primarily only know (and endlessly argue about) non-functional requirements.
Every improvement will be moderated increased demands from management, crunch, pressure to release, "good enough", add this extra library that monetizes/spys on the customer etc
In the same way that hardware improvements are quickly gobbled up by more demanding software.
The people doing the programming will also be more removed technically.
I can do Python, Java , Kotlin. I can do a little C++ ,less C, and a lot less assembly.
An endless succession of new tools, methodologies, and roles but failure persists because success is rooted in good judgment, wisdom, and common sense.
In my very humble opinion, the impact that software has on our lives is getting to the point where software engineering should become a true profession like the other engineering branches (electrical, mechanical, etc).
There should be things like professional certifications that engineers have to maintain through continuous education, a professional code of ethics, a board of review, and other functions.
My reasoning is that we are at the point where a software "engineer" can make a mistake that can have the same impact as a civil engineer making a bad calculation and causing a bridge collapse.
There's different levels to this, of course. An app for booking restaurant reservations wouldn't need that much oversight. But we've seen some outages having massive impacts that quite frankly did not happy twenty years ago.
This has dot-com bubble written all over it. But there are some deeper issues.
First, we as a society should really be scrutinizing what we invest in. Trillions of dollars could end homelessness as a rounding error.
Second, real people are going to be punished for this as the layoffs go into overdrive, people lose their houses and people struggle to have enough to eat.
Third, the ultimate goal of all this investment is to displace people from the labor pool. People are annoying. They demand things like fair pay, safe working conditions and sick leave.
Who will buy the results of all this AI if there’s no one left with a job?
Lastly, the externalities of all this investment are indefensible. For example, air and water pollution and rising utility prices.
We’re bouldering towards a future with a few thousand wealthy people where everyone else lives in worker housing, owns nothing and is the next incarnation of brick kiln workers on wealthy estates.
A trillion in a money market fund @ 5% is 50B/year.
Over the course of a few years (so as to not drive up the price of politicians too quickly) one could buy the top N politicians from most countries. From there on out your options are many.
After a decade or so you can probably have your trillion back.
I really do like this answer, but it would seem to have the property of being anti-inductive in that the cost for current politicians is so low because nobody is doing it at scale but if someone did that would force other people to ... well, it's an interesting thought experiment at least!
So I haven't looked through the comments, and assume this has been discussed, but the simple solution is to limit contracts to, say, $4M, and pay only on successful completion. Then build a large project through a series of smaller steps.
AI will absolutely solve these problems, by inventing nimble AI native companies that disrupt these business models into the Stone Age, worker by worker, role by role. Death by a billion cuts.
Completely off topic but when fonts are the size they are in this article I can't read it, the words don't register as words above a certain size. I assume this isn't normal or it wouldn't be so common...
How much money do you need to build a skyscraper on top of a tarpit? None because it’s not possible. The whole stack has to be gutted. I can do it but no one wants to listen so I’ll do it myself.
From consulting point of view, a common joke we use to tell, because customers demand a Ferrari, but are only willing to pay for the development costs of a Fiat.
Nuclear power plants usually only cost about twice as much as projected in phase II planning. IT projects are sort of open-ended. Interestingly, the simulator I was involved with (many decades ago) at a nuclear power plant came within about 10% of initial projections. The last "scan uploads for viruses" project I worked on was about 20x - 40x more expensive than projected. (Unfortunately the person who suggested we just pay a third party for this service was fired.) The bit with projecting cost and schedules for nuke plants is to ignore any initial costing and multiply the phase II planning estimate by 2 or 4.
It's not as disasterous as you think. Most people paying for power plants expect large overruns. If you're the type of company that can sell bonds to get 5.5 billion dollars to build a nuclear power plant, you can sell bonds to get 11 billion dollars to build a nuclear power plant.
Almost nobody who works in software development is a licensed professional engineer. Many are even self-taught, and that includes both ICs and managers. I'm not saying this is direct causation but I do think it odd that we are so utterly dependent on software for so many critical things and yet we basically YOLO its development compared to what we expect of the people who design our bridges, our chemicals, our airplanes, etc.
Licensing and the perceived rigor it signifies is irrelevant to whether something can be considered "professional engineering." Engineering exists at the intersection of applied science, business and economics. So most software projects can be YOLO'd simply because the economics permit it, but there are others where the high costs necessitate more rigor.
For instance, software in safety-critical systems is highly rigorously developed. However that level of investment does not make sense for run-of-the-mill internal LOB CRUD apps which constitute the vast majority of the dark matter of the software universe.
Software engineering is also nothing special when it comes to various failure modes, because you'll find similar examples in other engineering disciplines.
No big surprise. Taking a shitty process and "digitalizing" it will lead to a shitty process just on computers in the best case, in the worst case everything collapses.
managing software requirements and the corresponding changes to user/group/process behaviors is by far the hardest part of software development, and it is a task no one knows how to scale.
absent understanding, large companies engage in cargo cult behaviors: they create a sensible org chart, produce a gannt chart, have the coders start whacking code, presumably in 9 months a baby comes out.
What a joke blaming the IT community for not doing better, when most businesses refuse to look past anything but shipping features as fast as they can. "We take security and reliability very seriously", until management gets wind of the price tag. Guess what areas always get considered last and cut first. We all know.
But sure, blame the community at large, not the execs who make bad decisions in the name of short-term profits, then fail upward with golden parachutes into their next gig.
And definitely don't blame government for not punishing egregious behavior of corporations. Don't worry, you get a year of free credit monitoring from some scummy company who's also selling your data. Oh and justice was served to the offending corp, who got a one-time fine of $300k, when they make billions every quarter.
Maybe if we just outsource everything to AI, consultants, and offshore sweat shops things will improve!
> blaming the IT community for not doing better, when most businesses refuse to look past anything but shipping features
IT != software engineering. IT is a business function that manages a company's internal information. Software engineering is a time-tested process of building software.
A lot of projects fail because management thinks that IT is a software engineering department. It is not. It never was, and it never will be. Its incentives will never be aligned such that software engineering projects are set up for success.
The success rate of implementing software outside of IT and dragging them along later is much higher than implementing it through IT from the beginning.
I understand, but also, IT is an umbrella term for a wider industry that includes your definition of IT, software, and anything adjacent. If you read the article, you'll see it's the latter being referenced, and why I chose that terminology.
> The success rate of implementing software outside of IT and dragging them along later is much higher than implementing it through IT from the beginning.
That's a pretty strong statement. Isn't that the opposite of why the devops movement started?
There is no such thing as ‘simplicity science’ that can be
directly applied when dealing with IT problems. However, many insights of complexity science are applicable to solving real world IT problems. People love simple solutions. However Simple is a scam, https://nocomplexity.com/simple-is-a-scam/
There are no generic, simple solutions for complex IT challenges. But there are ground rules for finding and implementing simple solutions. I have created a playbook to prevent IT diasasters, The art and science towards simpler IT solutions see https://nocomplexity.com/documents/reports/SimplifyIT.pdf
The concerning aspect of all of this isn't the financial cost of these blunders, and what happened in the past. It is the increasing risk to human lives, and what will happen in the future. The Boeing case was only a sign of what's to come.
Take "AI", for instance. It is being adopted left and right as if it's the solution to all of our problems, and developers, managers, and executives are increasingly relying on it. Companies and governments love it because it can cut costs and potentially make us more productive. Most people are more than happy to offload their work to it, do a cursory check of its output, if at all, and ship it or publish it and claim the work as their own. After all, it can always serve as a scapegoat if things do go wrong, and its manufacturers can brush these off as user errors. Ultimately there is no accountability.
These are all components of a recipe for greater disasters. As these tools are adopted in industries where safety is paramount, in the military, etc., it's only a matter of time for more human lives to be impacted. Especially now when more egomaniacal autocrats are taking power, and surrounding themselves with yes-people. Losing face and admitting failure is not part of their playbook. We're digging ourselves into a hole we might not be able to get out of.
because most people are incompetent, produce incidental complexity to satisfy internal urge for busy work, and under-think the problem, greatly... that's why, and don't get me started on the morons who run the show
The article is kind of dumb. eg it hangs its hat on the Phoenix payroll system, which
> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. It also was attempting to implement 34 human-resource system interfaces across 101 government agencies and departments required for sharing employee data.
So basically people -- none of them in IT, but rather working for the government -- built something extraordinarily complex (80k rules!), and then are like wow, it's unforeseen that would make anything downstream at least equally as complex. And then the article blames IT in general. When this data point tells us that replacing a business process that used to require (per [1]) 2,000 pay advisors to perform will be complex. While working in an organization that has shit the bed so thoroughly that paying its employees requires 2k people. For an organization of 290k, so 0.6% of headcount is spent on paying employees!
IT is complex, but incompetent people and incompetent orgs do not magically become competent when undertaking IT projects.
Also too, making extraordinarily complex things they shouting the word "computer" at them like you're playing D&D and it's a spell does not make them simple.
Slightly related but unpopular opinion I have: I think software, broadly, today is the highest quality its ever been. People love to hate on some specific issues concerning how the Windows file explorer takes 900ms to open instead of 150ms, or how sometimes an iOS 26 liquid glass animation is a bit janky... we're complaining about so much minutia instead of seeing the whole forest.
I trust my phone to work so much that it is now the single, non-redundant source for keys to my apartment, keys to my car, and payment method. Phones could only even hope to do all of these things as of like ~4 years ago, and only as of ~this year do I feel confident enough to not even carry redundancies. My phone has never breached that trust so critically that I feel I need to.
Of course, this article talks about new software projects. And I think the truth and reason of the matter lies in this asymmetry: Android/iOS are not new. Giving an engineering team agency and a well-defined mandate that spans a long period of time oftentimes produces fantastic software. If that mandate often changes; or if it is unclear in the first place; or if there are middlemen stakeholders involved; you run the risk of things turning sideways. The failure of large software systems is, rarely, an engineering problem.
But, of course, it sometimes is. It took us ~30-40 years of abstraction/foundation building to get to the pretty darn good software we have today. It'll take another 30-40 years to add one or two more nines of reliability. And that's ok; I think we're trending in the right direction, and we're learning. Unless we start getting AI involved; then it might take 50-60 years :)
There are great big software projects and shitty ones.
IRCTC, UPI being examples of great ones.
Insurance and RTO being shitty ones.
I had an insurance deadline very near and the payment was not showing up in the insurance providers dashboard so had to do it twice and now it was still not showing up.
Also I have faced huge problems with getting the learner's licence online.
I got my name wrong in the drivers card and never went to correct it. However most of the problems the problems there were administrative not software. I agree both irctc and upi come to mind first as the successes. Insurance, could be a particular company as i never faced such problem. Websites for Tax filing and even starting an msme has been smooth.
Kind of strange take as though unique to software. Every sector that is large has issues since ambitious projects stretch what can be done by the current management and organizational practices. All software articles like these hark back to some mythical world smaller in scope/ambition/requirements. Humanity moves forward
* Construction and Engineering -- Massive cost overruns and schedule delays on large infrastructure projects (e.g., public transit systems, bridges)
* Military and Government -- Defense acquisition programs notorious for massive cost increases and years-long delays, where complex requirements and bureaucratic processes create an environment ripe for failure.
* Healthcare -- Hospital system implementations or large research projects that exceed budgets and fail to deliver intended efficiencies, often due to resistance to change and poor executive oversight.
> IT projects suffer from enough management hallucinations and delusions without AI adding to them.
Software is also incredibly hard, the human mind can understand the physical space very well but once we're deep into abstractions it simply struggles to keep up with it.
It is easier to explain how to build a house from scratch to virtually anyone than a mobile app/Excel.
I came to opposite conclusions. Technology is pretty easy, people are hard and the business culture we have fostered in the last 40 years gets in the way of success.
> We are left with only a professional and personal obligation to reemphasize the obvious: Ask what you do know, what you should know, and how big the gap is between them before embarking on creating an IT system. If no one else has ever successfully built your system with the schedule, budget, and functionality you asked for, please explain why your organization thinks it can
translation: "leave it to us professionals". Gate-keeping of this kind is exactly how computer science (the one remaining technical discipline still making reliable progress) could become like all of the other anemic, cursed fields of engineering. people thinking "hey im pretty sure I could make a better version of this" and then actually doing it is exactly how progress happens. I hope nobody reads this article and takes it seriously
There are 2 big problems with large software projects:
1. Connecting pay to work - estimates (replanning is learning, not failure)
2. Connecting work to pay - management (the world is fractal-like, scar tissue and band-aids)
I do not pre-suppose that there are definite solutions to these problems - there may be solutions, but getting there may require going far out of our way. As the old farmer said "Oh, I can tell you how to get there, but if I was you, I wouldn't start from here"
1. Pay to Work - someone is paying for the software project, and they need to know how much it will cost. Thus estimates are asked for, an architecture is asked for, and the architecture is tied to the estimates.
This is 'The Plan!'. The project administrators will pick some lifecycle paradigm to tie the architecture to the cost estimate.
The implementation team will learn as they do their work. This learning is often viewed as failure, as the team will try things that don't work.
The implementation team will learn that the architecture needs to change in some large ways and many small ways. The smallest changes are absorbed in regular work. Medium and Large changes will require more time (thus money); This request for more money will be viewed as a failure in estimation and not as learning.
2. Work to Pay - as the architecture is implemented, development tasks are completed. The Money People want Numbers, because Money People understand how they feel about Numbers. Also these Numbers will talk to other Numbers outside the company. Important Numbers with names like Share Price.
Thus many layers of management are chartered and instituted. The lowest layer of management is the self-managed software developer. The software developer will complete development tasks related to the architecture, tied to the plan, attached to the money (and the spreadsheets grew all around, all around [0]).
When the developer communicates about work, the Management Chain cares to hear about Numbers, but sometimes they must also involve themselves in failures.
It is bad to fail, especially repeated failures at the same kind of task. So managers institute rules to prevent failures. These rules are put in a virtual cabinet, or bureau. Thus we have Rules of the Bureau or Bureaucracy. These rules are not morally bad or good; not factually incorrect or correct, but whenever we notice them, they feel bad; We notice the ones that feel bad TO US. We are often in favor of rules that feel bad to SOMEONE ELSE. You are free to opt out of this system, but there is a price to doing so.
----
Too much writing, I desist from decoding verbiage:
Thus it is OK for individuals to learn many small things, but it is a failure for the organization to learn large things. Trying to avoid and prevent failure is viewed as admirable; trying to avoid learning is self-defeating.
This is a direct result of using leetcode in interviews instead of any other, more legitimate tests like winning a tekken 1v1. Have you ever seen a good developer who’s not good at real video games?
If companies had hired real developers instead of cosplayers who are stunlocked with imposter syndrome as the only candidate pool with time to memorize a rainbow table of arbitrary game trivia questions and answers, things would actually work.
The biggest reason is developer ego. Devs see their code as artwork an extension of themselves, so it's really hard to have critical conversations about small things and they erupt into holy wars. Off hand:
* Formatting
* Style
* Conventions
* Patterns
* Using the latest frameworks or whats en-vogue
I think where I've seen results delivered effectively and consistently is where there is a universal style enforced, which removes the individualism from the codebase. Some devs will not thrive in that environment, but instead it makes the code a means-to-the-end, rather than being-the-end.
As far as I can see in the modern tech industry landscape, virtually everyone has adopted style guides and automatic formatting/linting. Modern languages like Go even bake those decisions into the language itself.
I'd consider managing that stuff essentially table-stakes in big orgs these days. It doesn't stop projects from failing in highly expensive and visible ways.
Eh, you're not wrong, but management failures tend to be a bigger issue. On the hierarchy of ways software projects fail, developer ego is kind of upper-middle of the pack rather than top. Delusional, ignorant, or sadistic leadership tends to be higher.
It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features. If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level. There is no software development process which reliably produces software that works at scale without doing it small, and medium sized, first, and fixing what goes wrong before you go big.
> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level.
At a large box retail chain (15 states, ~300 stores) I worked on a project to replace the POS system.
The original plan had us getting everything working (Ha!) and then deploying it out to stores and then ending up with the two oddball "stores". The company cafeteria and surplus store were technically stores in that they had all the same setup and processes but were odd.
When the team that I was on was brought into this project, we flipped that around and first deployed to those two several months ahead of the schedule to deploy to the regular stores.
In particular, the surplus store had a few dozen transactions a day. If anything broke, you could do reconciliation by hand. The cafeteria had single register transaction volume that surpassed a surplus store on most any other day. Furthermore, all of its transactions were payroll deductions (swipe your badge rather than credit card or cash). This meant that if anything went wrong there we weren't in trouble with PCI and could debit and credit accounts.
Ultimately, we made our deadline to get things out to stores. We did have one nasty bug that showed up in late October (or was it early November?) with repackaging counts (if a box of 6 was $24 and if purchased as a single item it was $4.50 ... but if you bought 6 single items it was "repackaged" to cost $24 rather than $27) which interacted with a BOGO sale. That bug resulted in absurd receipts with sales and discounts (the receipt showed you spent $10,000 but were discounted $9,976 ... and then the GMs got alerts that the store was not able to make payroll because of a $9,976 discount ... one of the devs pulled an all nighter to fix that one and it got pushed to the stores ).
I shudder to think about what would have happened if we had tried to push the POS system out to customer facing stores where the performance issues in the cafeteria where worked out first or if we had to reconcile transactions to hunt down incorrect tax calculations.
You could have, in principle, implemented the new system to be able to run in "dummy mode" alongside the existing system at regular stores, so that you see that it produces the 'same' results in terms of what the existing system is able to provide.
Which is to say, there is more than one approach to gradual deployment.
Not easily when issues of PCI get in there.
Things like the credit card reader (and magnetic ink reader for checks), different input device (sending the barcode scanner two two different systems), keyboard input (completely different screens and keyed entry) would have made those hardware problems also things that needed to be solved.
The old system was a DOS based one where a given set of Fkeys were used to switch between screens on a . Need to do hand entry of a SKU? That was F4 and then type the number. Need to do a search for the description of an item? That was F5. The keyboard was particular to that register setup and used an old school XT (5 pin DIN) plug. The new systems were much more modern linux boxes that used USB plugs. The mag strip reader was flashed to new screens (and the old ones were replaced).
For this situation, it wasn't something that we could send keyboard, scanner, and credit card events to another register.
What's PCI?
Sorry, I'm not familiar with all the acronyms.
PCI itself is Payment Card Industry. PCI DSS as noted is the Data Security Standard.
https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Sec...
The time it was in the transition between 2.0 and 3.0 (its been refined many times since).
https://listings.pcisecuritystandards.org/documents/PCI-DSS-... is the 3.2.1 audit report template.
One of the most important things in there is you don't mix dev and production. The idea of putting a development box next to a production box that runs the same transactions... that just doesn't happen.
Failing a PCI DSS audit means hefty fines and increases of transaction fees (paying 1% more on each transaction done with a credit card can make a $10k/month - $100k/month fine a rounding error) to a "no, you can't process credit cards" which would mean... well... shutting down the company (that wouldn't be a first offense - its still not something you want to have a chat about with accounting about why everything costs 1% more now). Those are things that you don't want to deal with as a developer.
So, no. There is no development configuration in production, or mirroring of a point of sales terminal to another system that's running development code.
Development code doesn't touch other people's money. We had enough side eyes looking at the raw data for our manager's payment card on development systems because only people that banked at that local bank occasionally experienced a problem with their visa check card... https://en.wikipedia.org/wiki/Digital_card#Financial_cards - when it says "generally '^'" it means it can be some other character... and it was... and this wasn't a problem for most people, but it turned out that the non-standard separator (that we only found after reading the card's raw data) and a space in the surname would result in misparsing of the track and giving an error - but none of our other cards used a separator that didn't match the "generally").
So, being able to generate real production load (in the cafeteria) without using Visa, Mastercard, etc... was important. As was being able to fall back to using the nearly antique credit card imprinter ( https://en.wikipedia.org/wiki/Credit_card_imprinter ) for the store that was lucky to get a dozen transactions a day.
> So, no. There is no development configuration in production, or mirroring of a point of sales terminal to another system that's running development code.
This is a misreading of the suggestion, I think. My reading of the suggestion is to run a production "dry run" parallel code path, which you can reconcile with the existing system's work for a period of time, before you cut over.
This is not an issue precluded by PCI; it is exactly the method a team I led used to verify a rewrite of and migration to a "new system" handling over a billion dollars of recurring billing transactions annually: write the new thing with all your normal testing etc, then deploy it alongside in a "just tell us what you would do" mode, then verify its operation for specific case classes and then roll progressively over to using it for real.
edit: I don't mean to suggest this is a trivial thing to do, especially in the context you mentioned with many elements of hardware and likely odd deployment of updates, etc.
Our reading of PCI DSS was that there was no development code in a production build. Having a --dry-run flag would have meant doing that.
You could do "here is the list of skus for transaction 12120112340112345 - run this through the system and see what you get" on our dev boxes hooked up to QA store 2 (and an old device in the lab hooked up to QA store 1). That's not a problem.
Sending the scanner reads to the current production and a dev box in production would have been a hardware challenge. Not completely insurmountable but very difficult.
Sending the keyboard entry to both devices would be a problem. The screens were different and you can hand enter credit card numbers. So keyboard entry is potentially PCI data.
The backend store server would also have been difficult. There were updates to the store server (QA store 1 vs QA store 2 running simultaneously) that were needed too.
This wasn't something that we could progressively roll out to a store. When a store was to get the new terminals, they got a new hardware box, ingenicos were swapped with epson, old epson were replaced with new (same device but the screens had to be changed to match a different workflow - they were reprogrammable, but that was something that stores didn't have the setup to do), and a new build was pushed to the store server. You couldn't run register 1 with the old device and register 2 with a new one.
Fetching a list of SKUs, printing up a page of barcodes and running it was something we could do (and did) in the office. Trying to run a new POS system in a non-production mode next to production and mirroring it (with reconciling end of day runs) wasn't feasible for hardware, software, and PCI reasons that were exacerbated by the hardware and software issues.
Online this is potentially easier to do with sending a shopping cart to two different price calculators and logging if the new one matches the old one. With a POS terminal, this would be more akin to hooking the same keyboard and mouse up to a windows machine and a linux machine. The Windows machine is running MS Word and the linux is running Open office and checking to see that after five minutes of use of the windows machine that the Linux machine had the same text entered into OpenOffice. Of course they aren't - the keyboard entry commands are different, the windows are different sizes, the menus have things in different places in different drop downs... similarly, trying to do this with the two POS systems would be a challenge. And to top it off sometimes the digits typed are hand keyed credit card numbers when the MSR couldn't get a read - and make sure those don't show up on the linux machine.
I do realize this is reminiscent of business giving a poorly spec'ed thing and each time someone says "what about..." they come up with another reason it wouldn't work. This was a system that I worked on for a long while (a decade and a half ago) and could spend hours drawing and explaining diagrams of system architecture and issues that we had. Anecdotes of how something worked in a 4M Sloc system are inherently incomplete.
Neat! Yeah, that's a pretty complex context and I completely see what you mean about the new hardware being part of the rollout and necessarily meaning that you can't just run both systems. My comment is more of a strategy for just a backend or online processing system change than a physical brick and mortar swap out.
In my note about misreading the suggestion, I was thinking generally. I do believe that there is no reason from a PCI perspective why a given production system cannot process a transaction live and also in a dry mode on a new code path that's being verified, but if the difference isn't just code paths on a device, and instead involves hardware and process changes, your point about needing to deploy a dev box and that being a PCI issue totally makes sense, plus the bit about it being a bad test anyway because of the differences in actions taken or outputs.
The example you gave originally, of shipping to the lower stake exceptional stores first and then working out issues with them before you tried to scale out to everywhere, sounded to me like a very solid approach to mitigating risk while shipping early.
More of the background of the project.
The original register was a custom written C program running in DOS. It was getting harder and harder to find C programmers. The consultancy that had part of the maintenance contract with it was also having that difficulty and between raising the rates and deprioritizing the work items because their senior people (the ones who still knew how to sling C and fit it into computers with 4 MB of memory that you couldn't get replacement parts for anymore) were on other (higher paying) contracts.
So the company I worked at made the decision to switch from that program to a new one. They bought and licensed the full source to a Java POS system (and I've seen the same interface at other big retail companies too) and replace all the hardware in all the stores... ballpark 5000 POS systems.
The professional services consultancy was originally brought in (I recall it being underway when I started at there in 2010). They missed deadlines and updates and I'm sure legal got in there with failure to deliver on contract. I think it was late 2011 that the company pulled the top devs from each team and set us to working on making this ready in all stores by October 2012 (side note: tossing two senior devs from four different teams into a new team results in some challenging personality situations). And that's when we (the devs) flipped the schedule around and instead of March 2013 for the cafeteria and surplus store (because they were the odd ones), we were going to get them in place in March of 2012 so that we could have low risk production environments while we worked out issues (so many race conditions and graphical event issues hanging with old school AWT).
---
... personality clash memory... it was on some point of architecture and code and our voices were getting louder. Bullpen work environment, (a bunch of unsaid backstory here) but the director was in the cube on the other side of the bullpen from us. The director "suggested" that we take our discussion to a meeting room... so we packed up a computer (we needed it to talk about code), all of the POS devices that we needed, put it on a cart, pushed the cart down the hall into a free conference room (there were two conference rooms on that floor - no, this wasn't a building designed for development teams) and set up and went back to loudly discussing. However, we didn't schedule or reserve the room... and the director that kicked us out of the bullpen had reserved the room that we had been kicked into shortly after we got there. "We're still discussing the topic, that will probably be another 5-10 minutes from now... and it will take us another 5 minutes pack the computer back up and take it back to the bullpen. Your cube with extra chairs in it should be available for your meeting and it's quiet there now without our discussions going on."
Why not use aged/ anonymized data? This way you can use Prod data in Dev with custom security rules anonymizing your data and following DSS.
Lead: "We have six weeks to ship. Questions?"
Dev: "Could we pull an export of relevant historical data and get some time to write code to safely anonymize that, and stand up a parallel production system using just the anonymized data and replicate our deploy there, so we can safely test on real-ish stuff at scale?"
Lead: "I'll think about it. In the meantime, please just build the features I asked you to. We gotta hustle on this one."
I'm not arguing with this hypothetical exchange that it's infeasible or even a bad idea to do exactly what you suggested, but attempting to justify an upfront engineering cost that isn't directly finishing the job is a difficult thing to win in most contexts.
It's very common to use identical systems but anonymised data shipped back to test environments in such cases. There are certain test card numbers that always fail or always succeed against otherwise-real infrastructure on the card provider's side.
Absolutely, I agree that it's a useful pattern. I've personally typed 4111 1111 1111 1111 into a stripe form more times than I want to even think about.
My point above was that it's not necessarily easy to convince the operators of a business that it's a justifiable engineering expense to set up a new "prodlike but with anonymized data" environment from scratch, because it's not a trivial thing to make and maintain.
I do think it's pretty easy to convince operators of a business to adopt the other strategy suggested in a sibling thread: run a dry mode parallel code path, verify its results, and cut over when you have confidence. This shouldn't really be an alternative to a test environment, but they can both achieve similar stuff.
> I do think it's pretty easy to convince operators of a business to adopt the other strategy suggested in a sibling thread: run a dry mode parallel code path, verify its results, and cut over when you have confidence. This shouldn't really be an alternative to a test environment, but they can both achieve similar stuff.
I agree - it's a nice low-risk way of doing things.
Elsecomment I explained this more...
It is as low risk as trying to use Windows and Microsoft Word with a keyboard and mouse mirrored to a Linux machine running Open Office and expecting the same results.
You can't run the two systems side by side - different screens, different keyboard entry... and some of the keyboard entry can't touch the other system.
And this is assuming you can put a dry path into the production system. If the answer is "no", then you're putting a dev environment into a production environment... and that's certainly a "no".
We had test environments and we had a lab were we had two rows of systems where the two systems sat back to back and each row was hooked up to a different test store (not feasible in a production store environment).
surely you have logs from the production systems? just gather the logs and run them through the dev box. verify the end result matches between the two. You dont actually need the dev box to sit next to the production system.
You cannot, under any circumstances, keep a real card # and use it as test data. I think that's where this conversation is getting hung up, because the idea of running a transaction through prod and them doing the same in test to see if it matches isn't something you can do. I mean, of course you can throw the prices and UPCs at the new system and verify that the new system's math matches the old system, but that's only the most basic function of a POS system. Testing a transaction from end-to-end would have to be done with synthetic data in an isolated environment, and I'll assume that's what OP is trying to articulate.
There's all this stuff but I remember when I was a Junior freelancer I was analysing a calendar availability sync script for a small holiday bookings company (not the big one). The hosts would have a publicly accessible Google Calendar with their bookings on which the script I was fixing would pull from.
Turns out, most of the host stored their customers long cards + expiry etc in the comment field of the booking.
the reproduction is always fake to some extent, that does not matter, the point is to do as good a job as you can.
for example you can have a fake transaction server with the credit card numbers made up and mapped to fake accounts that always have enough money, unless the records show they did not.
I've also worked with payment processors a lot. The ones I've used have test environments where you can fake payments, and some of them (Adyen does this) even give you actual test debit and credit cards, with real IBAN's and stuff like that.
It's even public: https://docs.adyen.com/development-resources/test-cards-and-...
Don't know anything about the OP's system, other than "POS" but the bug they experienced - and (maybe?) all the typical integration stuff like inventory management - is very complex and wouldn't manifest itself in a payment processing failure. I'm doubtful that anyone's production inventory or accounting systems allow for "fake" transactions that can be validated by an e2e test
POS stands for Point Of Sales in this context.
It was a linux running on (year appropriate) https://www.hp.com/us-en/solutions/pos-systems-products.html... - and add on all the peripherals. The POS software was standalone-ish (you could, in theory, hook it up to a generator to a register and the primary store server and process cash, paper check, and likely store branded credit cards)... it wouldn't be pleasant, but it could.
The logic for discounts and sales and taxes (and if an item had sales tax in that jurisdiction) was all on register. The store server logged the transaction and handled inventory and price lookup, but didn't do price (sale, taxes) calculations itself.
At some point you start to get far away from reality though. If the cards have fake numbers then other auth information is also incorrect - e.g. the CVC won't match, the PIN won't either (depending on the format in use maybe). You can fake all that stuff too but now how much of that system are you really testing?
I mean in his example the discount bug they ran into wouldn’t have needed any card numbers that could have been discovered with fake/cloned transactions that contained no customer detail in this case it seems it would have been best to test the payment processing in personal at a single store and then also testing with sales logs from multiple other locations
yep, it sounds like the first implementation step really should have been to gather a large test set of data and develop the system with that in mind after understanding the test data, starting with making tests from the test data.
They explained the scenario though and it seems like a combination of rarer edge cases. It's great to think your awesome dev team and QA would have collected test data representing this ahead of time, but surely you've all been caught by this? I know I have; that's why we don't have flawless systems at launch.
OP said elsewhere that the specific discount that produced the $10M sales + refund combo was only active at five stores. That's the sort of edge case that you can't count on being in the training data. I'm as big a fan of real-data testing as anyone, but there's always something not represented.
PCI DSS in full. Payment Card Industry Data Security Standard. Basically a bunch of stuff you need to comply with if you're processing credit cards.
Payment card industry. Credit card info (personal user data, etc). There’s a whole boatload of data privacy issues you run into if you mess that up. So compliance is essential.
[dead]
From my experience a lot of the hardest problems in this space are either 1. edge cases or 2. integration-related and that makes them hard to validate across systems or draw boundaries around what's in the dummy mode. This type of parallel, live, full system integration test is hard to pull off.
In 1997 I was working on an integration between AOL and Circuit City (ha I outlived them both) to enable free AOL accounts for people buying PCs or some such; about a week before launch I changed the data returned from encoding spaces as "+" to "%20" and broke their integration (perl script). Very upsetting for them, and I felt bad.
I also had some weird bug when we started registrations from German accounts and I didn't handle umlauts (or UTF-16 with nuls in the string) in passwords properly.
Sounds good in theory but very few real world projects can afford to run with old system in parallel
>> We did have one nasty bug that showed up in late October (or was it early November?)
Having worked in Ecommerce & payment processing, where this weekend is treated like the Superbowl, birth of your first child and wedding day all rolled into one, a nasty POS bug at this time of year would be incredibly stressful!
After thinking back on it, I think this was earlyish October. The code hadn't frozen yet, but it was getting increasingly difficult. We were in the "this was deployed to about 1/3 of the stores - all within an 8 drive of the general office". The go/no-go decision for the rest of the stores in October was coming up (and people were reviewing backout procedures for those 100). One of the awkward parts was that marketing had a Black Friday sale that they really wanted to do (buy X, buy Y, get Z half price) that the old registers couldn't support. They wanted to get a "is this going?" so they could start printing the advertising flyers.
Incidentally, this bug resurfaced for the next five years in a different incarnation. Because it had that this department (it was with one sku) had sold $10M this week in October, the running average sales target the next year was MEAN($24k, $25k, $26k, $25k, $10M) ... and the department heads were doing a "you want me to sell how much?!"
This bug had only affected... maybe five stores (still maybe five too many). We were in the "this is the last™ build before all store deployment next week" territory. It did mess with that a bit too as the boxed up registers came with an additional step of "make sure to reboot the register after doing initial confirmation."
The setup teams had a pallet of computers delivered to the stores that were supposed to be "remove the old registers, put these registers in, swap mag strip readers, take that laptop there and run this software to configure the devices on each register." However, the build that the registers had was the buggy build. While that build likely wouldn't hit that bug (it required a particular sale to be active which was only at a few stores and had ended) it still was another step that they had to follow.
Aside: For all its clunkiness, Java Web Start was neat. In particular, it meant that instead of trying to push software to 5k registers (how do you push to registers that are powered off?), instead we'd push to 300 stores and from there JWS would check for an update each time it started up ( https://docs.oracle.com/javase/8/docs/technotes/guides/javaw... ). So instead of pushing to 5k registers, we'd have it pull from 'posupdate' on the local network when it rebooted.
There is no solution because these projects are not failing because of technical reasons.
They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions.
Then you have "important people" who want to be important and want to have an opinion on font size and that some button should be 12px to the right because they are "important" it doesn't matter for the project but they have to assert the dominance.
You have 2 or 3 companies working on a project? Great! now they will be throwing stuff over the fence to limit their own cost and blame others while trying to get away with as least work done cashing most money possible.
That is how sausage is made. Coming up with "reasonable approach" is not the solution because as soon as you get different suppliers, different departments you end up with power/money struggle.
> They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions.
Not (necessarily) wrong, but if you start small, Important People may not want to bother with something that is Unimportant and may leave things alone so something useful and working can get going. If you starting with an Important project then Important People will start circling it right away.
Even starting small isn't a surefire way to avoid that problem. They'll just show up once the thing gets big enough.
Witness how the web was once a funny little collection of nerds sharing stuff with each other. But once it got big enough that you could start making money off it, the important people showed up and started taking over. The web still has those odd little corners, but it's largely the domain of a small number of giant powerful corporations.
I don't think there is a silver bullet for dealing with egomaniacs who want infinite power. They seem to be a part of the human condition and dealing with them is part of the ticket price for having a society.
Dunno if you listen to Ezra Klein but he had an anthropologist on once who described this tribe of humans who when someone came back having bagged big game, they had to run a gauntlet of everyone else downplaying their accomplishment like "that's not that big, your father caught bigger", and "maybe one day you'll bring down an adult deer" etc. The whole idea was like, egomaniacs are pretty bad, and they had a cultural defense against it.
I often think a weakness of liberal, western society is the insistence on rationality, that like the hunter in question could just easily put their abilities and accomplishments alongside those of others and get a pretty accurate picture. This is super untrue; we need systems to guard against our frailties, but we can't admit we have them, so we keep falling into the same ditches.
> Dunno if you listen to Ezra Klein but he had an anthropologist on once […]
Klein wrote a book a few years ago on the topic of polarization, human's inclination towards clan/tribal thinking, and how it manifests itself in (US) politics:
* https://en.wikipedia.org/wiki/Why_We%27re_Polarized
I guess for me important point is that it is not technical issue and we already have all technical tools/processes to do really big software projects.
Even if people dislike scrum, find Git complicated and don’t want to open up JIRA - these tools are not the problem, these tools help building loads of working software.
We as software engineers with devops can deliver great and complex projects and build great systems. Lots of businesses people don’t even understand how much in control we can be of the environments and code.
Yet developers/IT is there to be blamed. Like we should be ashamed, Uncle Bob will give lectures “how developers should be more professional”.
Yet I always find business people who are like children in the corn field.
With small difference business/sales guys are pushy and walk over engineering guys and engineers bend over and take the blame and business guys can always say “those IT kids playing with toys instead of doing real job”.
Political corruption is like environmental radiation: a viable fix is never 'just get rid of political corruption'*. It's an environmental constant that needs to be handled by an effective approach.
That said, parent's size- and scope-iterative approach also helps with corruption, because corruption metastasizes in the time between {specification} and {deliverable}.
Shrink that, by tying incremental payments to working systems at smaller scales, and you shrink the blast radius for failure.
That said, there are myriad other problems the approach creates (encouraging architectures that won't scale to the final system, promoting duct taped features on top of an existing system, vendor-to-vendor transitions if the system builder changes, etc).
But on the whole, the pros outweigh the cons... for projects controlled by a political process (either public or private).
That's why military procurement has essentially landed on spiral development (i.e. iterative demonstrated risk burn-down) as a meta-framework.
* Limit political corruption, to the extent possible in a cost efficient manner, sure
> There is no solution because these projects are not failing because of technical reasons.
There is no technical solution. There are systems and governance solutions, if the will is there to analyze and implement them.
That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen.
Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.
> Gradual growth inevitably results in loads of technical debt.
Why is this stated as though it's some de facto software law? The argument is not whether it's possible to waterfall a massive software system. It clearly is possible, but the failure ratios have historically been sufficiently uncomfortable to give rise to entirely different (and evidently more successful) project development philosophies, especially when promoters were more sensitive to the massive sums involved (which in my opinion also helps explains why so many wasteful government examples). The lean startup did not appear in a vacuum. Do things that don't scale did not become a motto in these parts without reason. In case some are still confused about the historical purpose of these benign sounding advices, no, they weren't originally addressed at entrepreneurs aiming to run "lifestyle" businesses.
I think the logic is that good code is code which is maintainable and modifyable; bad code is difficult to change safely. Over time, all code is changed until it is bad code and cannot be changed more. So overtime most code is bad code which is scary to touch.
The Peter Principle of code.
its not the law but a cost
software is unique field where project can be a problem that no matter how much money you throw, there is something that we can "improve" or make it better
that's why we start something small, a scope if you want call it that way
of course start something small or dare I call it simpler would result in more technical debt because that things its not designed with scale in mind because back to the first point
It is a law. The law of entropy.
Try as you might, you cannot fight entropy eternally, as mistakes in this fight will accumulate and overpower you. It's the natural process of aging we see in every lifeform.
The way life continues on despite this law is through reproduction. If you bud off independent organisms, an ecosystem can gain "eternal" life.
The cost is that you must devote much of your energy to effective reproduction.
In software, this means embracing rewrites. The people who push against rewrites and claim they're not necessary are just as delusional as those who think they can live forever.
You don't understand very much about entropy. This reasoning is very, very, very sloppy.
Now I remember why I stopped commenting here.
low-effort comment with ad hominem and zero rationale. fairly toxic.
Software is a component of a product, if not the product itself. Treating software like a product, besides being the underlying truth, also means it makes sense to manage it like one.
Technical debt isn’t usually the problem people think it is. When it does become a problem, it’s best to think of it in product-like terms. Does it make the product less useful for its intended purpose? Does it make maintenance or repair inconvenient or costly? Or does it make it more difficult or even impossible to add competitive features or improvements? Taking a product evaluation approach to the question can help you figure out what the right response is. Sometimes it’s no response at all.
The discussion is not about the product where you can just remove the stuff. The thread was testing in small setting and then moving to oddball setting. If it is required to cover oddball settings, it makes sense to know and plan for oddball setting.
Took me way too long to learn this. It still makes me sad to leave projects “imperfect” and not fiddle in my free time sometimes
Designing or intending a system to be used at massive scale is not the same as building and deploying it so that it only initially runs at that massive scale.
That's just a recipe for disaster, "We don't even know if we can handle 100 users, let's now force 1 million people to use the system simultaneously." Even WhatsApp couldn't handle hundreds of millions of users on the day it was first released, nor did it attempt to. You build out slowly and make sure things work, at least if you're competent and sane.
Sure, but if you did a good job, the gradual deployment can go relatively quickly and smoothly, which is how $FAANG roll out new features and products to very large audiences. The actual rollout is usually a bit of an implementation detail of what first needed to be architected to handle that larger scale.
The issue with FAANG is that they already have the infrastructure to make these large scale deployments. So any new system - by necessity - needs to conform to that large scale architecture.
The other nice thing about FAANG is that almost nothing they do is actually necessary. If Facebook rolls out a new feature and breaks something for a few hours, it doesn't actually matter. It's harder to move fast and break things if you're, say, a bank, and every minute of downtime is a minute where your customers can't access their money. Enough minutes go by and you may have a very, very expensive crisis on your hands.
Replying to myself in sibling: except maybe people paying for ads, which is more of a faith based action; it's well known a lot of ad traffic is fraudulent, but not which traffic. So if you pay for ads, who can tell what happened.
Yeah, people get real upset about even 1 messed up money transaction.
You get certain big pieces correct maybe but you’d be surprised how many mistakes get made. For example, I had designed the billing system for a large distributed product that the engineer ended up implementing not as described in the spec which fell down fairly quickly with even a modicum of growth.
Well, Google got good at large scale rollouts, because they are doing large scale rollouts all the time. _And_ most of the time, the system they are rolling out is a small iteration from the last system they rolled out: the new GMail servers look almost exactly like the last GMail servers, but they have on extra feature flag you can turn on (and which is disabled by default) or have one bug fixed.
That's a very different challenge from rolling out a brand new system once.
FAANG tests first on test beds, and on subsets of their user base.
also, see what happened last week when Cloudflare pushed out a bad configuration without trying it on a subset
No but whatsapp was built by 2 guys that had previously worked at Yahoo, and they picked a very strong tech for the backend: erlang.
So while they probably didn't bother scaling the service to millions in the first version, they 1) knew what it would take, 2) chose already from the ground up a good technology to have a smoother transition to your "X millions users". The step "X millions to XYZ millions and then billions" required other things too.
At least they didn't have to write a php-to-C++ compiler for Php like Facebook had, given the initial design choice of Mark Zuckeberg, which shows exactly what it means to begin something already with the right tool and ideas in mind.
But this takes skills.
> No but whatsapp was built by 2 guys that had previously worked at Yahoo, and they picked a very strong tech for the backend: erlang.
https://news.ycombinator.com/item?id=44911553
Started as PHP, not as Erlang.
> 1) knew what it would take, 2) chose already from the ground up a good technology to have a smoother transition to your "X millions users".
No, as above, that was a pivot. They did not start from the ground up with Erlang or ejabberd, they adopted that later.
Thanks, somehow I remembered wrong.
They took already existing protocol (XMPP) and already existing implementation in Erlang (ejabberd)—there weren't many alternatives at the time really.
Did they succeed because of Erlang or in spite of Erlang? We can't draw any reliable conclusions from a single data point. Maybe a different platform would have worked even better?
Erlang is uniquely suited to chat systems out of the box in a way that most other ecosystems aren't. Lightweight green threads via the BEAM vm, process scheduler so concurrent out of the box, immutable data structures, message passing as communication between processes.
There's nothing unique about Erlang. I have nothing against it but other companies have built messaging systems using other platforms that work as well or better than WhatsApp.
Examples please.
Telegram
Yeah - the technology used is a seperate concern to their abilities as users (developers) of that technology and the effectiveness at handling the scale.
I, for example, have always said that I am more than capable of writing code in C that is several orders of magnitude SLOWER than what I could write in.. say Python.
My skillset would never be used as an example of the value of C for whatever
> Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts.
This isn't technical debt, necessarily. Technical debt is a specific thing. You probably mean "an underlying design that doesn't perfectly map to what ended up being the requirements". But then the world moves on (what if a regulation is added that ruins your perfect structure anyway?) and you can't just wish for perfect requirements. Or not in software that interacts directly with the real world, anyway.
You have to design for scale AND deploy gradually
Yes, absolutely. Knowing that it will need to get big eventually is important, but not at all the same as deploying at scale initially.
Yes, it can be very difficult to add “scale” after the fact, once you already have a lot of data persisted in a certain way.
Gradual growth =/= many tacked on features. Many tacked on features =/= technical debt. Technical debt =/= "everybody is afraid and frozen." Those are merely often correlated, but not required.
Whatsapp is a terrible example because it's barely a product; Whatsapp is mostly a free offering of goodwill riding on the back of actual products like Facebook Ads. A great example would be a product like Salesforce, SAP, or Microsoft Dynamics. Those products are forced to grow and change and adapt and scale, to massive numbers doing tons of work, all while being actual products and being software systems. I think such products act as stark rebukes of what you've described.
There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism.
Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine.
The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."
we get paid to add to it, we don’t get paid to take away
Now there is your problem. It is only true in the context of grave incompetence, though. I have worked on tickets with 'remove' in the title.
[dead]
The dominant factor is: there is a human who understands the entire system.
That is vastly easier to achieve by making a small, successful system, which gets buy in from both users and builders to the extent that the former pay sufficient money for the latter to be invested in understanding the entire system and then growing it and keeping up with the changes.
Occasionally a moon shot program can overcome all of that inertia, but the “90% of all projects fail” is definitely overrepresented in large projects. And the Precautionary Principle says you shouldn’t because the consequences are so high.
This works for Clojure, git and even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it. But these things are meant to be used by technical people.
The non-technical people I know might want to use Linux but stay on Windows or choose Mac OS because it's more straightforward. I use Windows+WSL at work even though I would like to use a native Linux distribution.
I know someone who created a MUD game (text online game) and said to him I wanted to make one with a browser client. He said something we could translate as "Good, you can have all the newbies." Not only was he right that a MUD should be played with a MUD client like tintin++, but making a good browser client is harder than it seems and that's time not spent making content for the game or improving the engine.
My point is that he was un uncomprimising person who refused adding layers to a project because they would come at a cost which isn't only time or dollars but also things like motivation and focus.
You’re conflating “knows the system” with benevolent dictator. It’s not the same. It’s down to whether in a planning or brainstorming session, there is anyone who can say that a plan won’t work or if there’s a better one.
Also it doesn’t have to be singular. You need at least one, in case that person leaves or becomes problematic. That dictator doesn’t always remain benevolent and they can hold a project hostage if they don’t like something that everyone else wants.
> ... even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it.
I really wonder what will happen to Linux once Linus is no longer involved.
Greg KH
You will never get to the moon by making a faster and faster bus.
I see a lot of software with that initial small scale "baked into it" at every level of its design, from the database engine choice, schema, concurrency handling, internal architecture, and even the form design and layout.
The best-engineered software I've seen (and written) always started at the maximum scale, with at least a plan for handling future feature extensions.
As a random example, the CommVault backup software was developed in AT&T to deal with their enormous distributed scale, and it was the only decently scalable backup software I had ever used. It was a serious challenge with its competitors to run a mere report of last night's backup job status!
I also see a lot of "started small, grew too big" software make hundreds of silly little mistakes throughout, such as using drop-down controls for selecting users or groups. Works great for that mom & pop corner store customer with half a dozen accounts, fails miserably at orgs with half a million. Ripping that out and fixing it can be a decidedly non-trivial piece of work.
Similarly, cardinality in the database schema has really irritating exceptions that only turn up at the million or billion row scale and can be obscenely difficult to fix later. An example I'm familiar with is that the ISBN codes used to "uniquely" identify books are almost, but not quite unique. There are a handful of duplicates, and yes, they turn up in real libraries. This means that if you used these as a primary key somewhere... bzzt... start over from the beginning with something else!
There is no way to prepare for this if you start with indexing the book on your own bookshelf. Whatever you cook up will fail at scale and will need a rethink.
Counterpoint: the idea that your project will be the one to scale up to the millions of users/requests/etc is hubris. Odds are, your project won't scale past a scale of 10,000 to 100,000. Designing every project to scale to the millions from the beginning often leads to overengineering, adding needless complexity when a simpler solution would have worked better.
Naturally, that advice doesn't hold if you know ahead of time that the project is going to be deployed at massive scale. In which case, go ahead and implement your database replication, load balancing, and failover from the start. But if you're designing an app for internal use at your company of 500, well, feel free to just use SQLite as your database. You won't ever run into the problems of scale in this app, and single-file databases have unique advantages when your scale is small.
Basically: know when huge scale is likely, and when it's immensely UNlikely. Design accordingly.
> Odds are, your project won't scale past a scale of 10,000 to 100,000.
That may be a self-fulfilling prophecy.
I agree in general that most apps don't need fancy scaling features, but apps that can't scale... won't... and hence "don't need scaling features".
> You won't ever run into the problems of scale in this app, and single-file databases have unique advantages when your scale is small.
I saw a customer start off with essentially a single small warehouse selling I dunno... widgets or something... and then the corporation grew and grew to a multi-national shipping and logistic corporation. They were saddled with an obscure proprietary database that worked like SQLite and had incredibly difficult to overcome technical challenges. They couldn't just migrate off, because that would have needed a massive many-year long total rewrite of their app.
For one performance issue we were seriously trying to convince them to use phase-change cooling on frequency-optimized server CPUs like a gamer overclocking their rig because that was the only way to eke out just enough performance to ensure their overnight backups didn't run into the morning busy time.
That's just not an issue with SQL Server or any similar standard client-server database engine.
I think part of that thinking though is that if you do basic stuff like use a standard database engine or don't go too off the beaten path if that's what you need, it tends to be that you get the ultimately needed scale for basically free.
This is a lot of times what I see the "don't build for huge scale" to be. It's not necessarily "be proud of O(n^2) algorithms". Rather it's more "use Postgres instead of some hyperscale sharded database when you only have 10 million users" because the alternative tends to miss the forest (and oftentimes the scale, ironically) for the trees
Yes, but also I've found that using a decently scalable engine is insufficient for a good outcome without the scaled data.
The best software I've written always had > 10 GB of existing data to work with from day one. So for example the "customers" table didn't have one sample entry, it had one million real entries. The "products" table had a real history of product recalls, renames, category changes over time, special one-off products for big customers, etc...
That's how you find out the reality of the business instead of some idealised textbook scenario.
Things like: Oh, actually, 99% of our product SKUs are one-off specials, but 99% of the user interactivity and sales volume is with the generic off-the-shelf 1% of them, so the UI has to cater for this and database table needs a "tag" on these so that they can be filtered. Then, it turns out the filtering 10 million products down to 100K has non-trivial performance issues when paging through the list. Or even worse, 50% of the products are secret because their mere existence or their name is "insider info" that we don't want our own staff to see. Did I say "staff"? I meant subcontractors, partners, and resellers, all with their own access rules and column-level data masking that needs to be consistent across dozens of tables. Okay, let's start going down the rabbithole of column naming conventions and metadata...
You can't predict that stuff in a vacuum, no human has the foresight needed to "ask the right questions" to figure this all out through workshops or whatever. The ground-truth reality is the best thing, especially up-front during the early phases of development.
A lot of the above is actually easy to implement as a software developer, but hard to change half-way-through a project.
You can by making a bigger and bigger rocket though.
While I think this is good advice in general, I don’t think your statement that “there is no process to create scalable software” holds true.
The uk gov development service reliably implements huge systems over and over again, and those systems go out to tens of millions from day 1. As a rule of thumb, the parts of the uk govt digital suite that suck are the parts the development service haven’t been assigned to yet.
The Swift banking org launches reliable features to hundreds of millions of users.
There’s honestly loads of instances of organisations reliably implementing robust and scalable software without starting with tens of users.
The uk government development service as you call it is not a service. It’s more of a declaration of process that is adhered to across diverse departments and organisations that make up the government. It’s usually small teams that are responsible for exploring what a service is or needs and then implementing it. They are able to deliver decent services because they start small, design and user test iteratively and only when there is a really good understanding of what’s being delivered do they scale out. The technology is the easy bit.
The UK Gov has many service and process docs [1]. It started out that way but has grown rapidly and changed. Including a library for authentication, frontend templates and libraries, custom docker images.
[1]: https://github.com/alphagov
UK GDS is great, but the point there is that they're a crack team of project managers.
People complain about junior developers who pass a hiring screen and then can't write a single line of code. The equivalent exists for both project management and management in general, except it's much harder to spot in advance. Plus there's simply a lot of bad doctrine and "vibes management" going on.
("Vibes management": you give a prompt to your employees vaguely describing a desired outcome and then keep trying to correct it into what you actually wanted)
> and those systems go out to tens of millions from day 1
I like GDS (I even interviewed with them once and saw their dev process etc) but this isn't a great example. Technically GDS services have millions of users across decades, but people e.g. aren't constantly applying for new passports every day.
A much better example I think is Facebook's rollout of Messenger, which scaled to billions of actual users on day 1 with no issues. They did it by shipping the code early in the Facebook app, and getting it to send test messages to other apps until the infra held, and then they released Messenger after that. Great test strategy.
GDS's budget is about £90 million a year or something. There are many contracts that are still spent on digital, for example PA consulting for £60 million (over a few years) which do a lot of the gov.uk home-office stuff, and their fresh grads they hire cost more to the government than GDS's most senior staff...
SWIFT? Hold my beer. SWIFT did not launch anything substantial since its startup days in early 70-ies.
Moreover, their core tech did not evolve that far from that era, and the 70-ies tech bros are still there through their progeniture.
Here's an anecdote: The first messaging system built by SWIFT was text-based, somewhat similar to ASN.1.
The next one used XML, as it was the fad of the day. Unfortunately, neither SWIFT nor the banks could handle 2-3 orders of magnitude increase in payload size in their ancient systems. Yes, as engineers, you would think compressing XML would solve the problem and you would by right. Moreover, XML Infoset already existed, and it defined compression as a function of the XML Schema, so it was somewhat more deterministic even though not more efficient than LZMA.
But the suits decided differently. At one of the SIBOS conferences they abbreviate XML tags, and did it literally on paper and without thinking about back-and-forth translation, dupes, etc.
And this is how we landed with ISO20022 abberviations that we all know and love: Ccy for Currency, Pmt for Payment, Dt for Date, etc.
Harder to audit when every payload needs to be decompressed to be inspected
Is it? No auditor will read binary, so you already need a preprocessing step to get it to a readable format. And if you're already preprocessing then adding a decompression step is like 2 lines tops.
> https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512
This is what https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 advocates too: start small, modularize, and then scale. The example of Tesla's mega factory was particular enticing.
> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
Gall’s law wins again.
Came here to say this. I still think that Linus Torvalds has the most profound advice to building a large, highly successful software system:
"Nobody should start to undertake a large project. You start with a small trivial project, and you should never expect it to get large. If you do, you'll just overdesign and generally think it is more important than it likely is at that stage. Or worse, you might be scared away by the sheer size of the work you envision. So start small, and think about the details. Don't think about some big picture and fancy design. If it doesn't solve some fairly immediate need, it's almost certainly over-designed. And don't expect people to jump in and help you. That's not how these things work. You need to get something half-way useful first, and then others will say "hey, that almost works for me", and they'll get involved in the project."
-- Linux Times, October 2004.
I don't think this applies in any way to companies contracted to build a massive system for a government with a clear need. Linus is talking about growing a greenfield open-source project, which may or may not ever be used by anyone.
In contrast, if your purpose is "we need to manage our country's accounting without pen and paper", that's a clear need for a massive system. Starting work on this by designing a system that can solve accounting for a small firm is not the right way to go. Instead, you have to design with the end-goal in mind, since that's what you were paid for. But, you don't launch your system to the entire country at once: you first use this system designed for a country in a small shop, to make sure it actually handles the small scale well, before gradually rolling out to more and more people.
> for a government with a clear need.
There's your problem. The needs are never clear, not on massive systems. Governments will write a spec, companies will read the spec, offer to implement it as written, knowing full well that it won't work. Then they charge exorbitant fees to modify the system after launch, so that it will actually full fill business needs.
The Danish government is famous for sucking at buying massive IT systems.
Building those systems is a long term project, and you have to start small with a minimum number of functions, scope creep on those initial use cases often kills these kinds of projects.
No Linus Torvalds would stand against people in projects from article, he would slam the door and quit.
Those projects that author pointed out are basically political horror stories. I can imagine how dozens of people wanted to have a cut on money in those projects or wanted to push things because “they are important people”.
There is nothing you can do technically to save such projects and it is NOT an IT failure.
This is a really dense paragraph of lifetime-accumulated wisdom in that single quote.
Works with implementations and not APIs though.
A bad API can constrain your implementation and often can't be changed once it's in use by loads of users. APIs should be right from day one if possible.
I would add the nuance that the possibility of controlled migration from one versioned API to another should be right from day one, not necessarily the first API version.
> I'm afraid that the solution is: build something small, and use it in production before you add more features.
Gall's Law:
> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[8]
* https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law
While I like the "start small and expand" strategy better than the "big project upfront", this trades project size for project length and often that is no better:
- It gives outside leadership types many more opportunities to add requirements later. This is nice is they are things missed in the original design, but it can also lead to massive scope creep.
- A big enough project that gets done the "start small and expand" way can easily grow into a decade-plus project. For an extreme example, see the multi-decade project by the Indian rail company to gradually replace all its railways to standard gauge. It works fine if you have the organisational backing for a long duration, but the constant knowledge leaks from people leaving, retiring, getting promoted, etc can be a real problem for a project like that. Especially in fields where the knowledge is the product, like in software.
- Not every project can feasibly start small.
> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level.
You could also try to buy some off-the-shelf solutions? Making payroll, even for very large organisations, isn't exactly a new problem.
As a corollary I would also suggest: subsidiarity.
> Subsidiarity is a principle of social organization that holds that social and political issues should be dealt with at the most immediate or local level that is consistent with their resolution.
(from https://en.wikipedia.org/wiki/Subsidiarity)
If you solve more problems more locally, you don't need that many people at the national level, thus making payroll there is easier.
I think you'll find that is exactly what people do. However, payroll solutions are highly customized for every individual company and even business unit. You don't buy a payroll software in a box, deploy it, and now you have payroll. Instead, you pay a payroll software company, they come in and get information about your payroll systems, and then they roll out their software on some of your systems and work with you to make sure their customizations worked etc. There's rarely any truly "off-the-shelf" software in B2B transactions, especially the type of end-user solutions that also interact with legal systems.
Also, governments are typically at least an order of magnitude larger than the largest companies operating in their countries, in terms of employees. So sure, the government of Liechtenstein has fewer employees than Google overall, but the US government certainly does not, and even Liechtenstein probably has way more government employees than Google employees in their country.
I work at a small shop, I'm a big advocate of giving customers the 0.1 version and then talking it out what they want. It's often not exactly what they asked for at the start ... but it often is better in the end.
It's hard to hit the target right the first time.
Yes. Also the same applies to companies. There should not be companies that are growing to $100 million revenue while losing money on a gamble that they will eventually get big enough to succeed. Good first, big later.
$100M maybe. But pretty much all tech needs an initial investment before you can start making profit. It takes a lot of development before you can get a product that anyone would want to pay for.
See also Gall's Law:
"All complex systems that work evolved from simpler systems that worked"
>It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features.
I think that is true for a lot of projects. But I'm not sure it is realistic to incrementally develop a control system for a nuclear reactor or an air traffic control system.
Not saying you're wrong, but I wonder what is the differentiating factor for software? We can build huge things like airliners, massive bridges and buildings without starting small.
Incremental makes less sense to me when you want to go to mars. Would you propose to write the software for such a mission in an incremental fashion too?
Yet for software systems it is sometimes proposed as the best way.
> We can build huge things like airliners, massive bridges and buildings without starting small.
We did start small with all of those things. We developed rigorous disciplines around engineering, architecture, material sciences. And people died along the way in the thousands[0][1]
People are still dying from those failures; The Boeing 737 MAX 9 crash was only two years ago.
> Incremental makes less sense to me when you want to go to mars.
This is yet another reason why a manned Mars mission will be exceedingly dangerous NOT a strike against incremental development and deployment.
[0] https://en.wikipedia.org/wiki/List_of_building_and_structure...
[1] https://en.wikipedia.org/wiki/List_of_accidents_and_incident...
All of the things you mentioned are designed and tested incrementally. Furthermore software has been used on Mars missions in the past, and that software was also developed incrementally. It’s proposed as the best way because it’s a way that has a track record
> All of the things you mentioned are designed and tested incrementally.
In a different way that what is proposed in this thread. We don't build a small bridge and grow it. We build small bridges, develop a theory for building bridges and use that to design the big bridge.
I don't know of any theory of computing that would help us design a "big" program at once.
That sounds like the way nature handles growth and complexity: slowly and over long time scales. Assume there will be failures, don't die and keep trying.
When you bite off too much complexity at once you end up not shipping anything or building something brittle.
You just need: Plan -> Implement -> Test -> Repeat
Whether you are creating software, games or whatever, these iterations are foundational. How these steps look like in detail of course depends on the project itself.
That's the ideal, but a lot of these big problems can't start small because the problem they have is already big. A lot of government IT programs are set up to replace existing software and -processes, often combining a lot of legacy software's jobs and the manual labor involved.
If you have something like a tax office or payroll, they need to integrate decades of legislation and rules. It's doable, but you need to understand the problem (which at those scales is almost impossible to fit in one person's head) and more importantly have diligent processes and architecture to slowly build up and deploy the software.
tl;dr it's hard. I have no experience in anything that scale, I've been at the edges of large organizations (e.g. consumer facing front-ends) for most of my career.
The accounting, legal and business process requirements are vastly different at different scales, different jurisdictions, different countries, etc.
There's a crazy amount of complexity and customizability in systems like ERPs for multinational corporations (SAP, Oracle).
When you start with a small town, you'll have to throw most of everything away when moving to a different scale.
That's true for software systems in general. If major requirements are bolted on after the fact, instead of designed into the system from the beginning, you usually end up with an unmaintainable mess.
Knowing that the rules for your first small deployment are not the same as the rules for everywhere, is valuable for designing well. Trying to implement all of those sets of rules in your initial deployment, is not a good idea. There is a general principle that you shouldn't code the abstraction until you've coded for the concrete example 2 or 3 times, because otherwise you won't make the right abstraction. Looking ahead is not the same as starting with the whole enchilada for your initial deployment.
I do get concerned when the solution is to be more strict on the waterfall process.
I used to believe there were some worlds in which waterfalls are better: where requirements are well know in advance and set in stone. I’ve since come to realize neither of those assumptions is ever true.
What works at small scale possibly won't work at a huge scale.
But what hasn’t even been tried at a small scale definitely won’t work at a huge scale.
Which is absolutely true, and a reason to try at medium scale second. But what doesn't work at small scale, almost certainly won't work at huge scale.
Imagine if the only way to build a skyscraper was to start with a dollhouse and keep tacking extensions and pieces onto it until. Imagine if the only way to build a bridge across San Francisco bay was to start with pop sickle sticks.
The very specific example you chose: payroll, shows how it can be difficult to incrementally step from small to huge. As you grow from town to national, you will run into all the disadvantages without really hitting the advantages. I feel that incremental does help you move from one level to one just a few above. But only if there are enough customers at these starting levels exactly.
When developing for towns, you will have all small random subsets of the variations imposed by year after year of legal changes BUT small sales. You will have to implement niche variations in arbitrary aspects for all the towns you have to support AND you will not have the customer size on which to amortize this work. Each new customer will bring a new arbitrary set of legal aspects to be met. Each new customer may be arbitrarily difficult to support.
By the time you reach national, you will have already covered most of the historical legal quirks - but that will have been done in one kludgy manner after another - and then you will hit one more set of legal quirks at the level of national organizations (some of them will have their very own laws). You will now have a very large budget to finalize things but you will be burdened by an illogical software base.
So I agree that you will need experience and subject matter experts that have worked at the various levels. BUT, now that you have this experience you now know the degree of flexibility that is required (you know where and what needs to be variable and quirk-friendly and how far the quirks can go = "any size") as well as size-related issues (mailing, transaction, user support volume) and you can now plan for all this AS YOU restart a new development from scratch. Because at this new "master" level you need both systematic flexibility AND relience at size.
Payroll is exactly the kind of topic where "adding features" will be "fun" - I mean bewildering - while you learn, but probably economically difficult to manage, until it kills you "as you climb up"?
You will be killed by a large software project that can afford to hire out a bunch of your subject matter specialists (or hires new ones) and uses them in a "from scratch" project. If you are lucky, this large project will be from the same company but only if you are lucky.
Now. AFTER you have done the one top level project - for one country -, you will probably be in a good situation to sell service to all kinds of organizations. Because you now have a system in which you can implement ridiculous quirks without breaking everything. And if you have done the job just right, you can onboard smaller customers (towns) economically enough that they can afford your solution.
That's different from where you deploy your solution first. Sure, deploy a national-design solution first at a subset of the target employees - although that does impose more requirements still: now you need to coexist with the legacy solutions. Which would be another hard to meet handicap when developing for towns first.
I study and write quite a bit of tech history. IMHO from what I've learned over the last few years of this hobby, the primary issue is quite simple. While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning. Typically, software folks build new and every generation of software developers must relearn the same problems.
I work at $FANG, every one of our org's big projects go off the rails at the end of the project and there's always a mad rush at the end to push developers to solve all the failures of project management in their off hours before the arbitrary deadline arrives.
After every single project, the org comes together to do a retrospective and ask "What can devs do differently next time to keep this from happening again". People leading the project take no action items, management doesn't hold themselves accountable at all, nor product for late changing requirements. And so, the cycle repeats next time.
I led and effort one time, after a big bug made it to production after one of those crunches that painted the picture of the root cause being a huge complicated project being handed off to offshore junior devs with no supervision, and then the junior devs managing it being completely switched twice in the 8 month project with no handover, nor introspection by leadership. My manager's manager killed the document and wouldn't allow publication until I removed any action items that would constrain management.
And thus, the cycle continues to repeat, balanced on the backs of developers.
Of course the reason it works this way is that it works. As much as we'd like accountability to happen on the basis of principle, it actually happens on the basis of practicality. Either the engineers organize their power and demand a new relationship with management, or projects start going so poorly that necessity demands a better working relationship, or nothing changes. There is no 'things get better out from wisdom alone' option; the people who benefit from improvements have to force the hand of the people who can implement them. I don't know if this looks like a union or something else but my guess is that in large part it's something else, for instance a sophisticated attempt at building a professional organization that can spread simple standards which organizations can clearly measure themselves against.
I think the reasons this hasn't happened is (a) tech has moved too fast for anyone to actually be able to credibly say how things should be done for longer than a year or two, and (b) attempts at professional organizations borrowed too much from slower-moving physical engineering and so didn't adapt to (a). But I do think it can be done and would benefit the industry greatly (at the cost of slowing things down in the short term). It requires a very 'agile' sense of standards, though.. If standards mean imposing big constraints on development, nobody will pay attention to them.
You forgot c) we're in a culture where people jump ship every 3-5 years. There's no incentive to learn from mistakes that you don't talk about at the next company, nor any care for the long term health of the current company.
>a sophisticated attempt at building a professional organization that can spread simple standards which organizations can clearly measure themselves against.
We have that as a form of IEEE, but it really doesn't come up much if you're not already neck deep in the organization.
> 3-5 years
That's maybe in Europe. Plenty of US developers those days have a litany of ~1-2 year stints at FAANGs and startups du jour in their CV.
Speaking as someone in the US, I've worked at multiple companies (some startups, some small businesses, some larger) that have either outright imploded or had mass department-level layoffs inside that 1-2 year timeframe. Some of them I would have stayed at longer than 1-2 years if I had the choice. I'm not claiming it's universal by any means, but there is a lot of volatility at US businesses in my personal experience.
And as usual, the employees get blamed. Maybe people wouldn't jump ship if companies didn't lay people off with reckless abandon, or hire sociopathic bosses to manage people, or screw them out of raises, or overwork them, or lie during the interview process.
The little people are going to do what they need to do to survive. If these multi-billion or even multi-trillion dollar companies feel some type of way about it, well, they're the ones with all the power, not us. They're more than welcome to change the culture at any time.
Jump ship every 3-5 years because if you don’t, your wages will stagnate. A prophet has honor except in his own home as it were.
True although perhaps that is part of (a), things move too fast.
It "works" only on a certain timescale. We don't have sufficient incentives and penalties to make things fail quickly. A relevant example in the tech world is data breaches. If data breaches resulted in a thorough public audit and financial/criminal penalties for the managers who pushed for speed over safety, they would no longer "work".
> If standards mean imposing big constraints on development, nobody will pay attention to them.
Unless there are penalties for not doing so.
> tech has moved too fast for anyone to actually be able to credibly say how things should be done for longer than a year or two
But that's just it. If things are moving so fast that you can't say how things should be done, then that tells you that the first thing that should be done is to slow everything way down.
I agree wholeheartedly that collective action is how we stop balancing poor management on the backs of engineers, but good luck getting other engineers to see it that way. There's heaps of propaganda out there telling engineers that if they join a union their high salary will go away, even though unions have never been shown to reduce wages.
I don't like unions because one bad hire can destroy a whole team, and the option to remove that hire is worth more than any benefit a union can give me.
I also think people here misunderstand what unions do. Unions are inherently conservative (small c) institutions that aim to protect the status quo. Improving processes is not a fundamental goal to unions. We saw this with the ILA that fought to essentially ban automation in the ports that would drastically increase efficiency because of the belief that this would reduce union jobs. It's foolish to think software unions wouldn't end up becoming like this.
> I don't like unions because one bad hire can destroy a whole team, and the option to remove that hire is worth more than any benefit a union can give me.
In MANY other countries there is already WAY more regulations regarding layoffs and firing employees that has nothing to do with unions.
In Germany there is a probationary period in which you can just fire somebody for no reason basically. That time can be like half a year (in my case) and in most cases it becomes clear if the new hire fits your team or doesn't.
All unrelated to unions though. The big unions in Germany for example have a lot of power and if you are just a simple welder for example you'd have no chance getting anything done without a union.
And this is one of the key reasons why Germany is economically stagnant, especially in the software industry.
> In MANY other countries...
When your scope is Europe ... The US is not the exception in the world, it's Europe which is.
The US has a dynamic job market where it's easy to lose your job, but easy to find another one. In Europe, and that's true for most EU countries, it's really hard to lose your job, but it's also really hard to get one for the very reason it's hard to get fired - and when you get a job, you will have to compromise on compensation and other benefits. It's not black and white here. While the European market is appealing to some people, the US market is preferable to others.
> It's not black and white here. While the European market is appealing to some people, the US market is preferable to others.
I agree with that, it's a very individual topic. I'd say for high paying "high performance" jobs the US model definitely has an advantage but for low-wage jobs it's quite the opposite.
No need to go as far as "low wage". Having strong labour protections is great if you are middle class and below.
Counterpoint: Denmark has something called Flexcurity: "flexible" + "security". Basically, it means you can hire and fire more easily than traditional socialist market economies. There is a good social safety net, but it is (somewhat) time constrained to pressure people to return to work quickly.
Good insight. Let's see whether Denmark remains competitive.
To be clear, the modern programme started in the 1990s. That makes it about 30 years old. If it had major problems, I assume they would be fixed by now.
Re-reading your comment again, I'm not sure that I understand it: "Let's see whether Denmark remains competitive." What do you mean?
Yeah and denmark finally voted left because they are finally getting tired of all this right wing shit that brings no benefit to workers.
Did they vote left? And that's the same left pushing over and over the Chat Control? That's an interesting twist if it turns out it's not always the right wing trying to undermine privacy rights.
You think Ursula von der Leyen is left wing? Like Lenin, Stalin and Ursula von der Leyen are in the same political party?
She is the European Commission president, that's unrelated.
But that made me curious, and answering my own question, it's this guy https://en.wikipedia.org/wiki/Peter_Hummelgaard who is indeed a Social Democrat .. So much about workers rights, funny ...
Well the proposal came from the european commission, doesn't seem that unrelated to me.
I'd rather have the protections for my working conditions than worrying about whether my co-workers are contributing enough to the company's bottom line but maybe that makes me an outlier here.
I'd say it does. I take pride and meaning in working. Life's too short to not care about a thing you do 8 hours a day. And bad colleagues doesn't just include people not contributing enough.
It doesn't make you an outlier. Most people work to live, not live to work.
"thanks for preserving status quo" - your boss
I've had a really shit boss who was thankfully fired and I wouldn't want him to have been in a union either.
There's trial periods… why do you notice a bad hire after 5 years?
People change, roles change, responsibilities change, motivations change.
You mean you made them miserable at their job and now it's their fault right?
???
I've worked places that refuse to fire low performers and its hard for it to not be toxic. I'm not saying this outcome is a forgone conclusion of unions, but my union experience is that poor performers take even longer to get rid of and I'm not sure I would be interested in that sort of environment again. This is more of an implementation problem than philosophical, but theoretically good and practically bad is still just bad.
Until healthcare and housing aren't tied to employment, making it easier to fire people will always be the monstrous position. If you want firing people with abandon to be socially acceptable then you need a public safety net. Until that's in place, I'm always going to be on the side of labor organizations demanding dignity than the people destroying lives.
How do you feel about consensual relationships? I don't impose myself on my employer, we have a mutual and consensual relationship. I don't understand, what do you want peoples material resources tied to if not their own assets?
My relationship with my employer is coercive, not consensual. They have something I need in order to survive. If I leave it will have deleterious effects on the quality of my life. You could make the argument that they are not the ones coercing me, society is or something, but I do not associate with them freely.
This was even more apparent before the ACA passed, when insurance providers were not prevented from denying or charging more for customers with pre-existing conditions. Hate your job but have cancer? Better not leave, you might not even be able to get insured at your next job.
You don't have to fire "low performers" you just have to be realistic about their skillsets and use them that way.
If you see an engineer is out of his depth then change his position, no need to fire them since like others have pointed out, that can have severe consequences in their personal lives and most of the time they can still be useful if they get more adequate work.
My hunch is that software engineers are averse to unions because they correctly perceive that unions are a wide angle away from the type of professional organization that would be most beneficial to them. The industry is sufficiently different that the normal union model is just not very good and has a 'square leg round hole' feeling.
For instance by and large the role of organizing to not to get more money but rather to reduce indignities... Wasted work, lack of forethought, bad management, arbitrary layoffs, etc. So it is much more about governing management with good practices than about keeping wages up; at least for now wages are generally high anyway.
there are also reasons to dedend jobs/wages in the face of e.g. outsourcing... But it's almost like a separate problem. Maybe there needs to be both a union and a uncoupled professional standard or something?
what type of professional organization is most beneficial? Standards are already out there, but they need a union or government regulation to be enforced. Devs who want real change need to pick their medicine, or continue to let the industry stagnate.
>the role of organizing to not to get more money but rather to reduce indignities
agreed. And I think that's why it's going to really start taking hold as we enter year 4 of mass layoffs in the US (because outsourcing). Alongside overwork from the "survivors" and abusive PIPs to keep people on edge.
> year 4 of mass layoffs in the US (because outsourcing)
A lot of the layoffs appear to be about conserving cash for investment in AI. In many cases the jobs that are cut are not backfilled by workers in the US or abroad.
It's wild to claim that the industry is stagnating. By any objective measure the industry is larger, more influential, and more innovative than ever before. Perhaps the problems that people are complaining about here just don't matter very much?
More innovative? Is that why none of my stuff works anymore and is crammed full of crapware I don't want that spies on me?
Oh, you meant innovative for the shareholders. Got it.
I don't know, maybe you're buying junk? I don't have that problem.
When was the last time you had to look for a job?
2024
I take it you have a good reputation and a network then. It is getting increasingly difficult to get hired lately, even as a senior. At least in the EU.
It's a shame how the EU has sabotaged their own software industry through a mix of excessive regulation, misguided labor protections, and failure to develop broad capital markets. It's like they're collectively choosing to be poor and backward. I can't understand it.
I can't understand it either. To me it seems people are excessively risk-averse. I can understand that but I don't understand the leadership not trying to stimulate a little bit of risk and growth there. It's getting to absurd levels at places.
But regardless. Not like USA companies are open to EU citizens currently. Whether it's politics, compliance / legal, or tax-related, it does not matter much. Most US job ads I've seen lately are making it super clear that they want only US citizens.
> By any objective measure the industry is larger, more influential, and more innovative than ever before
What objective measures would you use?
Pick any measure you like: total employment, revenue, patents, new venture formation, products shipped, etc. There are fluctuations from year to year due to the business cycle but overall the trend is up.
Guess that's why gamedev is the one region where this is really starting to gain momentum. High salaries were already not a thing, and tend to mean nothing if you're laid off after 3 years of development for the release of a new game.
Though I think Gen Z in general will be making waves in the coming years. They can't even get a foot in the door, so why should they care about "high salaries"?
People aren't going to try to wrest control from management because some project is going off the rails. No one has any particular faith their coworkers will run anything better, and the pay checks show up regardless.
A union might help improve wages and working conditions in some organizations (although I personally wouldn't want one). But there is zero chance that a union could ever achieve widespread improvement in software architecture, methodologies, or project management. We don't have much consensus on the right way to do things, and what worked well in one circumstance often causes disaster in another.
I don't think unions are the right thing here, you just need to get together as a team and talk with your higher ups, that's a far smaller scope than where unions normally come in.
But I totally agree, I think people are too compliant and fear banding together to have influence on higher ups. I'd argue in most places the engineers have far more power than they realize since they are in high demand due to shortages of qualified personnel. (at least in many countries in Europe)
There are tons of factors in play though that I believe contribute to this like some employees being friends with their higher ups not wanting to hurt their careers, not wanting the tough discussions, the repercussions if management says "no" etc..
> you just need to get together as a team
"Just". Come on, man, you know better than that. I too like my sci-fi to be over the top unrealistic.
Truth is, nobody ever thinks about their rights before it's too late. The paycheck shows up on time every month and people just don't want to rock the boat. Not to mention all the opportunists that will tell on you immediately to gain the favor of the upper class.
These things are well-known and apparently nothing ever changes before the guillotines start working. I don't think anything will ever improve in our area. Nobody is bothered. The people who are have zero power. And so it goes.
The "just" was more in the sense that it technically is not that hard to get together, there is nothing directly preventing that.
I honestly think it's just people who don't mind these problems and are fine working under those conditions, or they just quit once they are too annoyed. Change is way harder just than just leaving, I think that's also part of the reason why this keeps going on.
Absolutely not, barely anyone is okay with it -- only the most naive and fresh of young workforce, and then not even half of them by my observations chatting with many people 20+ years younger than me. It's just that people have too much at stake to risk it. Families, mortgage, basic respectful existence even. Nobody wants to live under a bridge.
It's extremely sad. I did not want to live in such times but oh well, good thing anyone asked us, right?
How about licensure and liability instead? That’s the sword of Damocles hanging over the heads of the rest of the engineering world. Sure it’s a guild system with a new name, but if the bridge collapses, somebody is going to be in a courtroom.
Customers are free to demand that software vendors take liability for certain defects or failures as part of contract negotiation. There's no need for governments to get involved.
For software that's actually safety critical, like avionics, there are already sufficient regulatory controls.
Actually it is necessary. That’s why your bridge engineer has to satisfy a licensing board that he or she knows what they’re doing before they seal a set of plans. Clients can’t distinguish good from bad engineering. They shouldn’t have to figure out if you know what you’re doing.
Actually it isn't necessary. Unlike bridges, very little software is safety critical. Clients don't need to try to distinguish good from bad software development, they can just require a certain level of performance and reliability in the contract with clear financial penalties for failure.
There's a better sense of safety if you do so from within, but we can't agree on what best standards are as an industry, let alone enforce them.
And the one in the courtoom is the head of the engineering firm, not the low level guy :)
Has union labor resulted in measurable improvement in production outcomes in any industry they’re found in? I don’t think going from “managers are unaccountable for failure” to “nobody is accountable for failure” is a good thing.
I think introducing more competition at higher levels may be better than eliminating it below. This should be happening because I’m pretty sure most PMs could be replaced by an LLM.
How about we transition from “managers are unaccountable for failure” to “managers are accountable for every failure by default and are sued and have houses confiscated within two months of a case being open”? That must include CEOs as well though, not just some bootlicker PMs.
Executives are generally incompetent everywhere and of course they'll introduce a reality distortion field where none of them are ever accountable. That should be obvious to anyone. Question is why do all working people keep allowing it to happen. But the answer to that is also known and quite depressing, too.
In a world like that, I hope it becomes standard practice to shoot at and bomb the repo men.
Of course every manager would want that. Who wants responsibility and taking care of their reports anyway.
Have you done any research into this or are you just assuming unions lead to bad outcomes because you've been propagandized for decades about it?
> This should be happening because I’m pretty sure most PMs could be replaced by an LLM
Says a lot about your understanding of these things.
While I too doubt that unions would be a net negative and I might even suspect interference by paid malicious actors to discourage people to unionize and thus never have power, I can't agree with your skepticism that PMs cannot be replaced by LLMs.
Most PMs I've ever met had zero clue what they are doing. And no it's not only N=1 sample, same anecdote is heard from many, many other people.
But sadly enough, undeniable human incompetence there is not even the biggest problem. The "we will never make more reasonable deadline" is.
Most managers, not just PMs, are an objective net negative. As any ruling class always does, they get complacent and think that just putting their foot down is going to magically change reality.
> Most PMs I've ever met had zero clue what they are doing. And no it's not only N=1 sample, same anecdote is heard from many, many other people.
Most people say that because they have no idea what a PM's job is and are upset that the PM isn't doing what they want.
Highly irrelevant. Not my role as an individual contributor to hone and fine-tune their job description.
Fact on the ground is that they usually optimize the entirely wrong indicators and never ever optimize for avoiding future problems.
In my consulting and contracting stints I made it a habit to write down what crisis is looming on the horizon and making bets with my wife which one is going to arrive first. A fun little game.
Watching train wrecks in slow motion stopped being fun with time.
What PM's job is is above my pay grade. I want them to enable me and the team, not be a mouth piece of people with zero understanding of product _and_ of engineering. They are just there to ask you how is stuff going.
> Fact on the ground is that they usually optimize the entirely wrong indicators and never ever optimize for avoiding future problems.
Fact on the ground is they don't optimize for _your_ indicators and have to compromise on what problems can be addressed now or later. They only appear to be optimizing to the wrong indicators because you don't have all the information.
> have to compromise on what problems can be addressed now or later.
9/10 PM's work is literally avoiding any decisions and responsibilities whatsoever.
Concerned dev/ops/po: Dear PM, feature FOO-123 will not be merged before we start activities for Gate 5.2. FOO-123 is required for Gate 6.0 and if we start with Gate 5.2 activities without it, Gate 6.0 will be delayed by at least 5 weeks. Team FOO is projecting verification of FOO-123 being done by the end of week, which would delay Gate 5.2 by a week. Shall we delay our activities until FOO-123 is merged or start regardless?
PM: Gate 5.2 is extremely important for the project timeline and no delays are acceptable.
<- few moments later ->
PM: I was informed by Compliance that FOO-123 is mandatory, does that affect timelines for Gate 6.0?
Disgruntled employee: Either we start over Gate 5.2 activities with FOO-123 included, which would delay Gate 6.0 by at least 9 weeks, or you get team FOO to backport FOO-123.
<- few moments later ->
PM: gets promoted for successful handling of stressful situation with FOO-123, limiting project delay to 15 weeks and only overrunning projected costs of Gate 6.0 by 30%.
I'm waiting for the disruptive startup that revolutionizes corporate heirarchy by getting rid of PMs since they're so useless.
Since successful businesses continue to employ them, I'm going to err on the side of they do serve a function I don't always understand rather than they must be stupid.
Startups cannot disrupt human hierarchies rooted in deep instincts built by millennia of serfdom, sadly.
And yes most PMs are useless, regardless of how snarkily you attempt to push that element aside.
Let me introduce you to the glory of Valve and Lord Gaben.
On a less snarky note: Depending on organization type PMs might do a ton of paperwork on behalf of the teams and act as some sort of central point for business-product communication, however that has exactly nothing with management in any typical definition.
I so wish I would agree with you, if my career has not been a string of them _never_ compromising on anything and only pulling the blanket to their side of the bed.
Your balanced take is nice and I wish I inhabited that reality. So far I have not. Maybe it's the region market or maybe my marketing sucked.
The reality in most businesses is constant roll from fighting one immediate crisis to another and handwaving future risks on "business environment may change" and "not important right now". The more crises one "mitigates", the more competent they are seen, even if they have directly caused the crisis in question.
If you talk about an upcoming shitshow you are generally seen as not entrepreneurial, if you mitigate a risk before it becomes a crisis that crisis is not viewed as crisis. However, if you let a crisis happen and "successfully" manage it you are seen as a hero.
Yes, sadly are quite right. I want off this toxic train. Are any aliens in your vicinity hiring? Tell them I am interested.
For one project I got so far as to include in the project proposal some outcomes that showed whether or not it was a success: quote from the PM “if it doesn’t do that then we should not have bothered building this”. They objected to even including something so obviously required in the plan.
Waste of my bloody time. Project completed, taking twice as many devs for twice as long, great success, PM promoted. Doesn’t do that basic thing that was the entire point of it. Nobody has ever cared.
Edit to explain why I care: there was a very nice third party utility/helper for our users. We built our own version because “only we can do amazing direct integration with the actual service, which will make it far more useful”. Now we have to support our worse in-house tool, but we never did any amazing direct integration and I guarantee we never will.
Glad to hear that $FANG has similar incompetency as every other mid-tier software shop I've ever worked in. Your project progression sounds like any of them. Here I was thinking that $FANG's highly-paid developers and project management processes were actually better than average.
Same, I think this one post may have cured me of a life long (unrealized) obsession with working at FANG.
They can afford to try a lot, why try better?
Those processes take longer, and waste more money. At no point will I believe they don’t waste it in the first place.
Reminds me of the military. Senior leaders often have no real idea of what is happening on the ground because the information funneled upward doesn't fit into painting a rosy report. The middle officer ranks don't want to know the truth because it impacts their careers. How can executives even hope to lead their organizations this way?
By not relying on direct reports for all their information.
Well the US has lost every military conflict it's entered for the past 70 years. Since there's been no internal pressure to change methodology, maybe the US military doesn't view winning as necessary.
Those past 70 years weren't about winning. It was about making sure the enemies lost more out of it. The US is large and relatively stable and hasn't had to face extended war on its soil since the Civil War 170 years ago. There's no true skin in the game for those who start these wars.
Which is a good strategy, but do you think the afghans lost more than 2 trillion dollars?
"The war began on April 12, 1861, when the Confederacy bombarded Fort Sumter in South Carolina"
170 years ago is 1855.
> Well the US has lost every military conflict it's entered for the past 70 years.
Operation Just Cause? Desert Storm?
And, depending on how you look at it, the US won the war in Afghanistan and Irak, but lost the peace afterwards.
Those might be the only ones? Desert Storm being the one that I'd probably call out, Just Cause was just so small.
One minor win, every major operation being a loss doesn't change the conclusion though imo.
I think it's also instructive to look at each of these operations and note that the two that were won were small, had clear objectives, and were executed quickly to meet those objectives and had no scope creep.
Iraq had one of the largest militaries in the world at the time of Desert Storm. They had tons and tons of arms and equipment and a huge standing army to counter the persistent threat (and/or to provide their own threat) of resumed hostilities with Iran (that war was still pretty recent when Desert Storm took place)
I would agree that the US is notably terrible at occupations and getting involved in civil wars, at least since WWII, but Desert Storm was pretty much an unqualified slam-dunk take-a-victory-lap success against one of the top armies in the world that wasn’t an ally or a nuclear state—carried out on the other side of the planet from the US, to boot. Like I think Iraq was ranked top-10 at the time by many ways of reckoning military strength, and that wasn’t enough to effectively resist the US effort at all, really.
If that war seems small, it’s only possible for it to seem that way from the victor’s perspective, and only because we did such an amazingly good job of totally destroying Iraq’s substantial capacity to fight in a matter of weeks. In terms of deployed and engaged men and materiel it was really big, just fast because it was so very one-sided, and “cheap” in terms of casualties on the US side for the same reason.
You're agreeing with me, or rather I agree with you.
I consider Desert Storm an unqualified victory in an engagement that is in the same conversation as a Vietnam or Korean war, but still not quite to a WW level in scope or complexity.
For how much power they have over team organization and processes, software middle management has nearly no accountability for outcomes.
Is it middle management that has no accountability, or executive? Middle and line managers are nearly as targeted by layoff culling as ICs these days in FAANG. The broad processes they're passing down to ICs generally start with someone at director level or higher.
In my experience it is the constant shifting of goal posts due to execs chasing the next shiny thing, or demanding a feature that they saw somewhere, or heard from client (singular, not plural)
I've seen plenty of incredible engineers let go because of "performance issues" that were just poor project management and goal posts that moved so fast waymo should study them to improve their self-driving capabilities.
Shit rolls downhill and there's a lot more fuss when an engineer calls out risks, piss-poor planning, etc. than any actual introspection on why the risks weren't caught sooner or why the planning was piss-poor.
> For how much power they have over team organization and processes, software middle management has nearly no accountability for outcomes.
Can we also address the fact that “software spend” is distributed disproportionately to management at all levels and people who actually write the software are nickel and dimed. You’d save billions in spend and boost productivity massively if the management is bare bones and is held accountable like the rest of the folks.
that's how the inner sanctum engineering in Apple worked, just like you proposed, at least from 15 years ago to within the last 10 years. i could have been in a very lucky time window to have had that luxury, but it had been an Apple mandate to not have deep hierarchies at least in engineering.
Maybe is because of what Steve Jobs mentioned about talented programmers having more power than CEOs as they can easily switch jobs.
perhaps that was involved, but one thing clearly purposeful was people were seriously filtered for particular skills and personality (apple fit it was called back then), which created groups where individuals had unique skills and collectively the group members would naturally want to collaborate. it worked great.
(as an aside, this contrasts diametrically with Amazon, where i worked for a year for healthcare, not needing to because of Apple years' savings, but after a genomics startup i had joined ran out of funding, and wanting a new challenge; there skilled engineering types were presumed to be fungible assets for (not kidding) at least 7 layers of do-nothing bureaucrats making huge salaries...they could survive because sales on the amazon store extract something like the 30% royalty to amazon)
The real question is why would smart competent people continue working under management with blatant ulterior motives that negatively affect them?
Why let their own credibility get dragged down for a second time, third time, fourth time, etc…?
The first time is understandable but not afterwards.
Astronomical salaries probably has something to do with it.
Yeah that could convince smart competent people to grind their teeth and take a second chance under the same management.
But I don’t think a self respecting person would do that over and over.
When people live in multi million dollar homes, self-respect doesn't pay monthly mortgage.
So it's really not the astronomical salary, it's the astronomical debt.
Yes and no. The compensation is a lot, but you're not necessarily able to just quit on a dime even if you live humbly. Interviewing takes weeks now and weeks more just to find a proper replacement. And salaries can fund you for months, bu t not years (let alone if you have a fammily)
I'll also say the obvious here in Sinclair's quote about salaries: you can indeed pay for someone's self respect.
This would imply most of these types of positions are filled with less competent people willing to package and sell their self respect alongside their time?
(Thus commanding a rate similar to a more competent person who doesn’t package it to sell.)
Astronomical debt caused by astronomical real estate inflation.
Joke is, most of these homes aren't worth anywhere close to their paper value.
Cy Porter's home inspection videos... jeez. How these "builders" are still in business is mind-blowing to me (as a German). Here? Some of that shit he shows would lead to criminal charges for fraud.
The land is worth more than the structure in these areas.
You may be over-estimating how many people are self-respecting?
Depends on the paycheck.
People will do crazy things for just $100. Including literally get fucked in the ass by a stranger.
7 figures? Ho boy. They’ll use way fancier words though for that.
There is an old Russian joke, that goes like this:
A man approaches a girl and asks, "Would you sleep with me for $1 million?”
She responds, “Yes, of course!”
Excited, he then asks, “What about for $1?”
She indignantly replies, “Who do you think I am?”
To which he responds, “We already established who you are; now we’re just discussing the price.”
I think it fits there. There is surprising amount of people believing that they have some Values, but just to a point when they were offered to sell them for a right price.
There's a high switching cost with substantial information asymmetry. Good places are hard to find in some absolute sense and it's hard to evaluate whether a new team is actually going to be good or not. And even if you do find a good team, there's no guarantee it'll last past the next reorg.
In today’s market it’s mostly because of the lack of other options to earn a livelihood
serious answer - you find a team with a good direct manager who handles all the upward interactions themselves, and then you basically work for that manager, rather than for the company.
Rent wont pay itself. Switching jobs has costs.
If you think our ability to measure developer productivity is bad, look into what we can do to measure manager productivity.
TL;DR your realistic options are snake oil that doesn’t work, or nothing.
Keep that in mind next time anyone’s talking about managing through metrics & data or whatever bullshit. All that stuff’s kayfabe, companies mostly run on vibes outside a very-few things.
^ This. Not at FAANG, but I am too familiar with this.
This is why software projects fail. We lowly developers always take the blame and management skates. The lack of accountability among decision makers is why things like the UK Post Office scandals happen.
Heads need to be put on pikes. Start with John Roberts, Adam Crozier, Moya Greene, and Paula Vennells.
So much of the world, especially the world we see today around corporate leadership and national politics makes much more sense once you realize this fundamental law:
People who desire infinite power only want it because it gives them the power to avoid consequences, not because they want both the power and the consequences.
The people who believe that with great power comes great consequences are exactly the people who don't want great power because they don't want the weight of those consequences. The only people who see that bargain and think "sign me up!" are the ones who intend to drop the consequences on the floor.
I was a developer for a bioinformatics software startup in which the very essential 'data import' workflow wasn't defined until the release was in the 'testing' phase.
> wouldn't allow publication until I removed any action items that would constrain management.
Thats what we call blameless culture lol
“I love deadlines. I love the whooshing noise they make as they go by.” ― Douglas Adams
Did they go off the rails at the end, or deadlines force acknowledging that the project is not where folks want it to be?
That said, I think I would agree with your main concern, there. If they question is "why did the devs make it so that project management didn't work?" Seems silly not to acknowledge why/how project management should have seen the evidence earlier.
Where I now work, in the government, all the devs are required to be part project managers. It’s a huge breath of fresh air. The devs are in all the customer meetings, assist in requirements gathering, and directly coach the customers as necessary to keep pushing the work towards completion.
This came about because our work isn’t too diverse but the requirements are wildly diverse and many of the customers have no idea how to achieve the proper level of readiness. I do management in an enterprise API project for a large organization.
Both happy and sad to know that the sh*t show is pretty much the same in FAANG as any regular corporate environment.
There are many pressures and this is all about a lack of transparent honesty about what the real priorities are. Getting the project done properly may be #1 priority but there's priority 0 and 0.1 and others which are unspoken because they don't sound good.
Obviously you work at AMZN. This is the most Amazonish HN comment I’ve ever seen.
FAANG cargo cults are all around. Too many CEOs and VCs imagine they can succeed like Bezos by the bold strategy of hiring ex-Amazon executives and imitating Amazon's culture and HR practices.
[dead]
I've also considered a side-effect of this. Each generation of software engineers learns to operate on top of the stack of tech that came before them. This becomes their new operating floor. The generations before, when faced with a problem, would have generally achieved a solution "lower" down in the stack (or at their present baseline). But the generations today and in the future will seek to solve the problems they face on top of that base floor because they simply don't understand it.
This leads to higher and higher towers of abstraction that eat up resources while providing little more functionality than if it was solved lower down. This has been further enabled by a long history of rapidly increasing compute capability and vastly increasing memory and storage sizes. Because they are only interacting with these older parts of their systems at the interface level they often don't know that problems were solved years prior, or are capable of being solved efficiently.
I'm starting to see ideas that will probably form into entire pieces of software "written" on top of AI models as the new floor. Where the model basically handles all of the mainline computation, control flow, and business logic. What would have required a dozen Mhz and 4MB of RAM to run now requires TFlops and Gigabytes -- and being built from a fresh start again will fail to learn from any of the lessons learned when it was done 30 years ago and 30 layers down.
Yeah, people tend to add rather than improve. It's possible to add into lower levels without breaking things, but it's hard. Growing up as a programmer, I was taught UNUX philosophy as a golden rule, but there are sharp corners on this one:
To do a new job, build afresh rather than complicate old programs by adding new "features".
Was not it: "do one thing, do it well"?
There's a bit more to it https://en.wikipedia.org/wiki/Unix_philosophy
It's the "Lava Flow" antipattern [1][2] identified by the Gang of Five [3], "characterized by the lava-like 'flows' of previous developmental versions strewn about the code landscape, but now hardened into a basalt-like, immovable, generally useless mass of code which no one can remember much if anything about.... these flows are often so complicated looking and spaghetti-like that they seem important but no one can really explain what they do or why they exist."
[1] http://antipatterns.com/lavaflow.htm
[2] https://en.wikipedia.org/wiki/Lava_flow_(programming)
[3] http://antipatterns.com/
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not
I've been managing, designing, building and implementing ERP type software for a long time and in my opinion the issue is typically not the software or tools.
The primary issue I see is lack of qualified people managing large/complex projects because it's a rare skill. To be successful requires lots of experience and the right personality (i.e. low ego, not a person that just enjoys being in charge but rather a problem solver that is constantly seeking a better understanding).
People without the proper experience won't see the landscape in front of them. They will see a nice little walking trail over some hilly terrain that extends for about a few miles.
In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.
> In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.
And boy to the people making the decisions NOT want to hear that. You'll be dismissed as a naysayer being overly conservative. If you're in a position where your words have credibility in the org, then you'll constantly be asked "what can we do to make this NOT a quest to the top of Mt Doom?" when the answer is almost always "very little".
Impossible projects with impossible deadlines seems to be the norm and even when people pull them off miraculously the lesson learned is not "ok worked this time for some reason but we should not do this again", then the next people get in and go "it was done in the past why can't we do this?"
Wow, sounds so familiar! I've once had to argue precisely against this very conclusion - "you saved us once in emergency, now you're bound to do it again".
Wrote to my management: "It is, by all means, great when a navigator is able to take over an incapacitated pilot and make an emergency landing, thus averting the fatality. But the conclusion shouldn't be that navigators expected to perform more landings or continue to be backup pilots. Neither it should be that we completely retrain navigators as pilots and vice versa. But if navigators are assigned some extra responsibility, it should be formally acknowledged by giving them appropriate training, tools and recognition. Otherwise many written-off airplanes and hospitalized personnel would ensue."
For all I know the only thing this writing might have contributed to was increased resentment by management.
> And boy to the people making the decisions NOT want to hear that.
You are 100% correct. The way I've tried to manage that is to provide info while not appearing to be the naysayer by giving some options. It makes it seem like I'm on board with crazy-ass plan and just trying to find a way to make it successful, like this:
"Ok, there are a few ways we could handle this:
Option 1 is to do ABC first which will take X amount of time and you get some value soon, then come back and do DEF later
Option 2 is to do ABC+DEF at the same time but it's much tougher and slower"
My favorite fact is that every single time an organization manages to make a functional development team that can repeatedly successfully navigate all the problems and deliver working software that adds real value, the high up decision makers always decide to scale the team next.
Working teams are good for a project only, then they are destroyed.
Jesus I just had flashbacks from my last jobs. Non-technical founder always telling me I was being pessimistic (there were no technical founders). It's just not that simple Karen!
I think part of it is that reading code isn't a skill that most people are taught.
When I was in grad school ages ago, my advisor told me to spend a week reading the source code of the system we were working with (TinyOS), and come back to him when I thought I understood enough to make changes and improvements. I also had a copy of the Linux Core Kernel with Commentary that I perused from time to time.
Being able to dive into an unknown codebase and make sense of where the pieces are put together is a very useful skill that too many people just don't have.
Being good at reading code isn't a skill that helps large software projects stay on rails.
It's more about being good at juggling 1000 balls at the same time. It's 99.9% of the time a management problem, not a software problem.
The successful projects I've worked on, the technical staff have been given autonomy, responsibility and full insight into the problem space. This requires managers putting a lot of trust in the engineers, but it works.
Large projects I've worked on failed simply because nobody wanted the solution in the first place.
In government I've seen many millions spent on projects that were either forgotten about or the politician that requested it lost office.
Reading (someone else's) code is a whole lot harder than writing it. Which is unfortunate because I do an awful lot of it at work.
I'm curious, what does "read code" mean to you? What does that skill look like and how is it taught?
You’ll notice that more senior engineers are often much better at giving useful review comments, and they will do it faster than you, thats just a skill that seems to come with experience reading other peoples code(or your own code you wrote two years prior). It can’t be taught, only practiced, same goes for reading other types of technical/academic works.
Not GP, but the general idea is the skill to take a piece of code and understand what it does by reading the code itself (probably in an IDE that can help navigate it meaningfully), not relying on docs or explanations or anything else. Surprisingly few people are comfortable in doing this, and yet it's very common in any large software project that lots of parts of the code are undocumented and no one remembers the details of how they were written.
Precisely. I don't assume any documentation is accurate unless I've verified it matches the behavior of the code - and I've run into some doozies in my day.
The worst was a custom-built copy protection scheme that was built by a former (?) software cracker who designed it to be difficult for someone of his skills to reverse engineer. It took me a week tracing through the code to understand what it was actually doing so I could extend it to add more options.
Oh TinyOS! Did my thsis on that!
This is one part of the issue. The other major piece of this that I've seen over more than two decades in industry is that most large projects are started by and run by (but not necessarily the same person) non-technical people who are exercising political power, rather than by technical people who can achieve the desired outcomes. When you put the nexus of power into the hands of non-technical people in a technical endeavor you end up with outcomes that don't match expectations. Larger scale projects deeply suffering from "not knowing what we don't know" at the top.
If this were true all of the time then the fix would be simple - only have technical people in charge. My experience has shown that this (only technical people in charge) doesn't solve the problem.
Success pretty much requires putting technical people in charge, but that doesn't mean putting technical people in charge is sufficient for success to happen. We have plenty of data over the last 40 years to prove my case. Furthermore, unfortunately, what it means to be a "technical person" is not so simple to define, unfortunately as the easy ways to codify it often exclude the very people who you want involved.
Suffice to say, projects are significantly more likely to succeed when the power in the project is held by people who are competent /and/ understand the systems they are working with /and/ understand the problem domain you are developing a solution in. Whether or not they have a title like "engineer" or have a technical degree, or whatever other hallmark you might choose is largely irrelevant. What matters is competency and understanding, and then ultimately accountability.
Most large projects I've been a part of or near lacked all three of these things, and thus were fundamentally doomed to failure before they ever began. The people in power lacked competency and understanding, the entire project team and the people in power lacked accountability, and competency was unevenly distributed amongst the project team.
It may feel pithy, but it really is true that in many large projects the fundamental issue that leads to failure is that the decision makers don't know what they're doing and most of the implementers are incompetent. We can always root cause further to identify the incentive structures in society, and particularly in public/government projects that lead to this being true, but the fact remains at the project level this is the largest problem in my observation.
> Success pretty much requires putting technical people in charge, but that doesn't mean putting technical people in charge is sufficient for success to happen.
This statement is terribly incongruent.
> This statement is terribly incongruent
In what way? A condition being required but not sufficient for success is a perfectly logical statement.
It does solve the problem! That's the entire reason we have such a thing as a tech industry. Tech industry startups are often just regular industries where everyone from the board down is technical. Uber is a "tech company" and not a taxi company because of this distinction. The results speak for themselves.
If people didn’t work, maybe we should put an LLM in charge instead.
Boeing has entered the chat
Sometimes giving people what they want can be bad for them; management wants cheap compliant workers, management gets cheap compliant workers, and then the projects fall apart in easily predictable and preventable ways.
Because such failures are so common management typically isn’t punished when they do so it’s hard to keep interests inline. And because many producers are run on a cost plus basis there can be a perverse incentive to do a bad job, or at least avoid doing a good one.
I'm not entirely sure what you mean with "technical people" but it seems that you may not appreciate the problems that "non-technical people" try to tackle.
Do your two decades of experience cover both sides?
> Do your two decades of experience cover both sides?
Yes.
I appreciate both sides and have a wealth of experience in both. The challenge is that all the non-technical problems cannot be solved successfully while lacking a technical understanding. Projects generally don't fail for technical reasons, they fail because they were not set up for success, and that starts with having a clear understanding of requirements, feasibility, and a solid understanding of both the current state and the path to reach your desired outcomes, both politically/financially and technically.
I was an engineer for more than a decade, I've been in Product for nearly a decade, and I'm now a senior manager in Product. I can honestly say that I have the necessary experience to hold strong opinions here and to be correct in those opinions.
You need technical people who can also handle some of the non-technical aspects of a project with the reins of power if you want the project to succeed, otherwise it is doomed by the lack of understanding and competency of those in charge.
"you may not appreciate the problems that 'non-technical people' try to tackle."
Do you mean the problem of wanting to build something without knowing how/having the skills, to build something?
I have a theory that the churn in technology is by design. If a new paradigm, new language, new framework comes out every so many years, it allows the tech sector to always want to hire new graduates for lower salaries. It gives a thin veneer of we want to always hire the person who has X when really they just do not want to hire someone with 10 years of experience in tech but who may not have picked up X yet.
I do not think it is the only reason. The world is complex, but I do think it factors into why software is not treated like other engineering fields.
Constantly rewriting the same stuff in endless cycles of new frameworks and languages gives an artificial sense of productivity and justifies its own existence.
If we took the same approach to other engineering, we'd be constantly tearing down houses and rebuilding them just because we have better nails now. It sure would keep a lot of builders employed though.
We do take down a lot of old buildings (or renovate them thoroughly) cause the old buildings contain asbestos, are not properly isolated, ...
> If we took the same approach to other engineering, we'd be constantly tearing down houses and rebuilding them just because we have better nails now. It sure would keep a lot of builders employed though.
This is almost exactly what happens in some countries.
which one(s)?
Pretty common in Australia. Theres heritage laws to try to prevent replacing all the old buildings, but often they are so undesirable the owner just leaves them vacant until trespassers manage to burn it down.
Have it in Japan too. You can clearly see eras in house design. Pre 1960 almost everything is wood. Then you have wood and plaster until the 2000s or so, and after that is plastic on wood. You can see the age of a neighborhood and its residents based on what houses are made of.
If the residents die and someone new purchases the land, the old house is (generally) torn down and a new one built.
Japan, famously. Oddly enough it actually works very well in keeping buildings cheap.
I agree. But, I think the execs just say, "How can we get the most bang for our buck? If we use X, Y, Z technologies, that are the new hotness, then we will get all the new hordes of hires out there, which will make them happy, and has the added benefit of paying them less"
The problem with that is that it would require a huge amount of coordination for it to be by design. I think it's better to look on it as systemic. Which isn't to say there aren't malign forces contributing.
I agree. Perhaps, "by design" is not the correct phrasing. Many decisions and effects go through a multi weighted graph of complexity (sort of like machine learning).
Indeed. How does that saying go? Don’t attribute to malice what can be explained by stupidity?
On the other hand Microsoft and taceboook did collude to keep salaries low. So who knows.
Anyone in tech should read up on https://en.wikipedia.org/wiki/High-Tech_Employee_Antitrust_L...
It was more tech companies in collusion than many people realize. 1) Apple and Google, (2) Apple and Adobe, (3) Apple and Pixar, (4) Google and Intel, (5) Google and Intuit, and (6) Lucasfilm and Pixar.
It was settled out of court. One of the plaintiffs was very vocal that the settlement was a travesty of justice. The companies paid less in the settlement than the amount they saved by colluding to keep wages down.
https://www.mercurynews.com/2014/06/19/judge-questions-settl...
There are rational explanations for this. When software fails catastrophically, people almost never die (considering how much software crashes every day). When hardware fails catastrophically, people tend to die, or lose a lot of money.
There's also the complexity gap. I don't think giving someone access to the Internet Explorer codebase is necessarily going to help them build a better browser. With millions of moving parts it's impossible to tell what is essential, superfluous, high quality, low quality. Fully understanding that prior art would be a years long endeavor, with many insights no doubt, but dubious.
I would boil this down to something else, but possibly related: project requirements are hard. That's it.
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning.
For most IT projects, software folks generally can NOT "pull apart" old systems, even if they wanted to.
> Typically, software folks build new and every generation of software developers must relearn the same problems.
Project management has gotten way better today than it was 20 years, so there is definitely some learnings that have been passed on.
A CIO once told me with Agile we didn’t need requirements. He thought my suggestion to document the current system before modifying was a complete waste of time. Instead he made all the developers go through a customer service workshop, how to handle and communicate with customers. Cough cough… most developers do not talk with customers. Instead where we worked developers took orders from product and project people whose titles changed every year but they operated with the mindset of a drill sergeant. My way or the highway.
Business developers need to occasionally talk directly to customers. It's fine to filter most requirements through Product Managers / Product Owners. But developers who never communicate directly with customers and end users get disconnected from reality and end up acting based on mythology rather than ground truth.
The developers were not given the choice. They are literally mushrooms. No requirements. No interaction with customers. They took orders from product/ project people.
"While hardware folks study and learn from the successes and failures of past hardware, software folks do not." Couldn't be further from the truth. Software folks are obsessed with copying what has been shown to work to the point that any advance quickly becomes a cargo cult (see microservices for example).
Once you've worked in both hardware and software engineering you quickly realize that they only superficially similar. Software is fundamentally philosophy, not physics.
Hardware is constrained by real world limitations. Software isn't except in the most extreme cases. Result is that there is not a 'right' way to do any one thing that everyone can converge on. The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.
Software doesn't operate in some magical realm outside of the physical world. It very much is constrained by real world limitations. It runs on the hardware that itself is limited. I wonder if some failures are a result of thinking it doesn't have these limitations?
As the great Joe Armstrong used to say, “a lot of systems actually break the laws of physics”[1] — don’t program against the laws of physics.
> In distributed systems there is no real shared state (imagine one machine in the USA another in Sweden) where is the shared state? In the middle of the Atlantic? - shared state breaks laws of physics. State changes are propagated at the speed of light - we always know how things were at a remote site not how they are now. What we know is what they last told us. If you make a software abstraction that ignores this fact you’ll be in trouble.[2]
[1]: “The Mess We’re In”, 2014 https://www.youtube.com/watch?v=lKXe3HUG2l4
[2]: https://news.ycombinator.com/item?id=19708900
> It very much is constrained by real world limitations. It runs on the hardware that itself is limited
And yet we scale the shit out of it, shifting limitations further and further. On that scale different problems emerge and there is no single person or even single team that could comprehend this complexity in isolation. You start to encounter problems that have never been solved before.
You can scale it within the bounds of the physical hardware it is running on. And as you scale it you start running into all the problems brought about by distributed systems, problems which very much stem from physics.
Except that it kind of does. I can horizontally scale a distributed storage system until we run out of silicon. I cannot do the same with a cargo airplane.
> Software folks are obsessed with copying what has been shown to work to the point that any advance quickly becomes a cargo cult
Seems more accurate to say they are obsessed with copying "what sounds good". Software industry doesn't seem to copy what works, rather what sounds like it'd work, or what sounds cool.
If they copied what works software would just be faster by default, because very often big established tools are replaced by something that offers similar featurage, but offers it at a higher FPS.
I disagree. At least at the RTL level they're very similar. You don't really deal with physics there, except for timing (which is fairly analogous with software performance things like hard real-time constraints).
> Result is that there is not a 'right' way to do any one thing that everyone can converge on.
Are you trying to say there is in hardware? That must be why we have exactly one branch predictor design, lol
> The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.
"The first function call looks a whole lot like a function call today..."
> That must be why we have exactly one branch predictor design, lol
I'll be that 'well akshually' guy. IIRC the AMD and intel implementations are different enough that spectre/meltdown exploits were slightly different on each manufacturers.
Source: wrote exploits.
It was sarcasm. There are loads of branch predictor designs.
Sorry, I had missed that. I'll go back to my cave.
> "The first function call looks a whole lot like a function call today..."
Only superficially. What's actually happening varies radically by language. See for instance tail call optimization.
What you and the GP said are not mutually exclusive. Software engineers are quick to drink every new Kool-Aid out there, which is exactly why we’re so damned blind to history and lessons learned before.
In my experience, a lot of the time the people who COULD be solving these issues are people who used to code or never have. The actual engineers who might do something like this aren't given authority or scope and you have MBAs or scrum masters in the way of actually solving problems.
Agree 100%.
I know a lot of people on here will disagree with me saying this but this is exactly how you get an ecosystem like javascript being as fragmented, insecure, and "trend prone" as the old school Wordpress days. It's the same problems over and over and every new "generation" of programmers has to relearn the lessons of old.
The difficulty lies in the fact that most software is quite cheap to generate very complex designs compared to hardware. For software designs treated similarly to hardware (such as in medical devices or at NASA), you do gain back those benefits at great expense.
I think this is too simple. First of all, hardware people have high incentive to fully replace components and systems for many reasons. Replacement is also the only way they can fix major design mistakes. Software people constantly do fix bugs and design mistakes. There is certainly no strong culture to document or dig up former mistakes made, but it's not like they don't learn from mistakes, it's just a constant process. In contrast to hardware, there is usually no point in time to retrospect. The incentives to rejuvenate systems are low and if considered often seem expensive. Software engineers self motivation is often ill-minded, new devs feeling uncomfortable with the existing system and calling for something "modern". But if the time comes to replace the "legacy" systems, then you are right, no one looks back at the former mistakes and the devs that know them, are probably long gone. The question is whether we should ever replace an software system or focus more on gradual and active modernization. But the latter can be very hard, in hardware everything is defined, most of the time backed by standards, in software we usually don't have that, so complex interconnected systems rarely have sane upgrade paths.
Most of the time, there's no need to study anything. Any experienced software engineer can tell you about a project they worked on with no real requirements, management constantly changing their mind, etc.
How do you study software history? Most of the lessons seem forever locked away behind corporate walls - any honest assessments made public will either end careers or start lawsuits
Some consequences of NOT learning from prior successes and failures: (a) no more training for the next generation of developers/engineers (b) fighting for the best developers, and this manifests in leetcode grinding (c) decrease in cooperation among team mates, etc.
IME, "Why systems fail" almost always boils down to a principal-agent problem. This is another way of expressing the Mungerism "show me the incentive, I'll show you the outcome".
Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives.
I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.
This is an interesting distinction, but it ignores the reasons software engineers do that.
First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc.
Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.
For most generations of hardware, you’re correct, but not all. For example, high-k was invented to mitigate tunneling. Sometimes, as geometries shrink, the physics involved does change.
I think there is a ton more nuance, but can still be explained by a simple observation, which TFA hints at: "It's the economics, stupid."
Engineering is the intersection of applied sciences, economics and business. The economics aspect is almost never recognized and explains many things. Projects of other disciplines have significantly higher costs and risks, which is why they require a lot more rigor. Taking hardware as example, one bad design decision can sink the entire company.
On the other hand, software has economics that span a much more diverse range than any other field. Consider:
- The capital costs are extremely low.
- Development can be extremely fast at the task level.
- Software, once produced, can be scaled almost limitlessly for very cheap almost instantly.
- The technology moves extremely fast. Most other engineering disciplines have not fundamentally changed in decades.
- The technology is infinitely flexible. Software for one thing can very easily be extended for an adjacent business need.
- The risks are often very low, but can be very high at the upper end. The rigor applied scales accordingly. Your LoB CRUD app going down might bother a handful of people, so who cares about tests? But your flight control software better be (and is) tested to hell and back.
- Projects vary drastically in stacks, scopes and risk profiles, but the talent pool is more or less common. This makes engineering culture absolutely critical because hiring is such a crapshoot.
- Extreme flexibility also masks the fact that complexity compounds very quickly. Abstractions enable elegant higher-level designs, but they mask internal details that almost always leak and introduce minor issues that cause compounding complexity.
- The business rules that software automates are extremely messy to begin with (80K payroll rules!) However, the combination of a) flexibility, b) speed, and c) scalability engender a false sense of confidence. Often no attempt is made at all to simplify business requirements, which is probably where the biggest wins hide. This is also what enables requirements to shift all the time, a prime cause for failures.
Worse, technical and business complexity can compound. E.g. its very easy to think "80K payroll rules linearly means O(80K) software modules" and not "wait, maybe those 80K payroll rules interact with each other, so it's probably a super-linear growth in complexity." Your architecture is then oriented towards the simplistic view, and needs hacks when business reality inevitably hits, which then start compounding complexity in the codebase.
And of course, if that's a contract up for bidding, your bid is going to be unsustainably low, which will be further depressed by the competitive bidding process.
If the true costs of a project -- which include human costs to the end users -- are not correctly evaluated, the design and rigor applied will be correspondingly out of whack.
As such I think most failures, in addition to regular old human issues like corruption, can be attributed to an insufficient appreciation of the economics involved, driven primarily by overindexing on the powers of software without an appreciation of the pitfalls.
As someone who's learning programming right now, do you have any suggestions on how one would go about finding and studying these successes and failures?
First, failures aren’t always obvious, and second, studying them isn’t either. This would likely need to be a formalized course. Still…
If people want to know why Microsoft hated DOS and wanted to kill it with Xenix, then OS/2, then Windows, and then NT it would be vital to know that it only came about as a result of IBM wanting a 16bit source-compatible CP/M which didn’t yet exist. Then, you would likely want to read Dissecting DOS to see what limits were imposed by DOS.
For other stuff, you would start backwards. Take the finished product and ask what the requirements were, then ask what the pain points are, then start digging through the source and flowcharting/mapping it. This part is a must because programs are often too difficult to really grok without some kind of map/chart.
There is likely an entire discipline to be created in this…
The things people are talking about in this thread are less to do with the practice of programming, and more to do with the difficulties of managing (and being managed, within) an engineering organization.
You'll learn all of this for yourself, on the job, just via experience.
One place to look is https://thedailywtf.com/
To be cynical, what's the point? You'll get employed and forced to be a part of them by circumstances.
Your company's root priorites are probably at odds with writing good software.
One Japanese company, not going to name names, kept trying to treat software as a depreciating asset. I didn't really understand well but the long and short was that fixing things that were supposed to be "done" was bad for the accounting. New things, however were good.
How can you run a software company like that? But they did and got the kind of outcome you'd expect. Japan made the laws this way and gets software to match.
I'm hoping to do a startup in molecular simulation. Granted, a startup has its own set of headaches, but it does dodge that particular issue :)
It obviously will do but with any luck you'll be so successful that you'll grow enormously and perhaps not have complete control in the end. Hopefully you'll have enough money not to mind too much!
I think this is a downstream of effect of there being no real regulation or professional designations in software which leads to every company and team being wildly different leading to no standards leaving no time for anything but crunching since there are no barriers restricting your time, so nobody spends time doing much besides shipping constantly.
Indeed.
That's why we see every now and then "new" programming paradigms which were once obsolete.
I was so annoyed when I found out the OTP library and realized we’ve been reinventing things for 20+ years
I’ve read one tech history book and I really enjoyed it. any you recommend?
Hardcore Software, Fire in the Valley, Life Under the Sun… there are many.
Software just feels so much more ephemeral than hardware. I haven't yet met a single 'old software enthusiast' in my life, yet there are so many enthusiasts for older hardware.
I am both a hardware and software enthusiast. Tons of DOS, Windows, and OS/2 software hanging around. While I don’t use them everyday, I do use them. From pre-Microsoft Visio to WordStar and MS Works for DOS, the applications are simple, powerful, and pleasing to use. While I don’t recommend anyone pull out Zenith 8bit and fire up COBOL-80 or LISP-80, they are interesting. Testing yourself in 64k is quite a challenge.
The retro community is huge and varied. If it exists, someone is really into it.
I have a pet passion for an old simulation language called Dynamo. I think you will find people passionate about LISP and people that care about COBOL, and C is already multiple decades old.
Doesn't retro gaming count?
Yes, and it's because there aren't very many textbook ways to do software engineering, because it's evolving too fast to reach consensus.
... are you saying that hardware projects fail less than software ones? just building a bridge is something that fails on a regular occurence all over the world. Every chip comes with list of erratas longer than my arm.
Software folks treat their output as if it's their baby or their art projects.
Hardware folks just follow best practices and physics.
They're different problem spaces though, and having done both I think HW is much simpler and easier to get right. SW is often similar if you're working on a driver or some low-level piece of code. I tried to stay in systems software throughout my career for this reason. I like doing things 'right' and don't have much need to prove to anyone how clever I am.
I've met many SW folks who insist on thinking of themselves as rock stars. I don't think I've ever met a HW engineer with that attitude.
Because the software market is bigger and more competitive; hardware is mature.
What are the silver bullets... I mean, best practices that keep getting ignored?
On some of the infamous large public IT project failures, you just have to look at who gets the contract, how they work, and what their incentives are. (For example, don't hire management consulting partner smooth talkers, and their fleet of low-skilled seat-warmers, to do performative hours billing.)
It's also hard when the team actually cares, but there are skills you can learn. Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).
But once you're a bit comfortable with the art and science of those, big new challenges are more about political and environment reality. It comes down to alignment and competence of: workers, internal team leadership, partners/vendors, customers, and investors/execs.
Discussing this is a little awkward, but maybe start with alignment, since most of the competence challenges are rooted in mis-alignments: never developing nor selecting for the skills that alignment would require.
Right, it's largely politically and ego driven; a people not a software problem.
Large-scale software is always a people problem. The hard part in software is communication, not typing the code.
> Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).
Was there any literature or other findings that you came across that ended up clicking and working for you that you can recommend to us?
I could blather for hours around this space. A few random highlights:
* The very first thing I read about requirements was Weinberg, and it's still worth reading. (Even if you are a contracting house, with a hopeless client, and you want to go full reactive scrum participatory design, to unblock you for sprints with big blocks of billable hours, not caring how much unnecessary work you do... at least you will know what you're not doing.)
* When interviewing people about business or technical, learn to use a Data Flow Diagram. You can make it accessible to almost everyone, as you talk through it, and answer all sorts of questions, at a variety of levels. There are a bunch of other system modeling tools you can use, at times, but do not underestimate the usefulness and accessibility of a good DFD.
* If you can (or have to) plan at all, find and learn to use a serious Gantt chart centric planning tool (work breakdown, dependencies, resource allocations, milestones), and keep it up to date (which probably includes having it linked with whatever task-tracking tool you use, but you'll usually also be changing it for bigger-picture reasons too). Even if you are a hardware company, with some hard external-dependency milestones, you will be changing things around those unmoveables. And have everyone work from the same source of truth (everyone can see the same Gantt chart and the task
* Also learn some kind of Kanban-ish board for tasking, and have it be an alternative view on the same data that's behind the Gantt view and the tasks/issues that people can/should/are working on at the moment, and anything immediately getting blocked.
* In a rare disruptive startup emergency, know when to put aside Gantt, and fall back to an ad hoc text file or spreadsheet of chaos-handling prioritization that's changing multiple times per day. (But don't say that your startup is always in emergency mode and you can never plan anything, because usually there is time for a Kanban board, and usually you should all share an understanding of how those tasks fit into a larger plan, and trace back to your goals, even if it's exploratory or reactive.)
* Culture of communicating and documenting, in low-friction, high-value, accessible ways. Respect it as team-oriented and professional
* Avoid routine meetings; make it easy to get timely answers and discussion, as soon as possible. This includes reconsidering how accessible upper leadership should be: can you get closer to being responsive to the needs of the work on the project (e.g., if anyone needs a decision from the director/VP/etc., then quickly prep and ask, maybe with an async message, but don't wait for weekly status meeting or to schedule time on their calendar).
* Avoid unnecessary process. Avoid performances.
* People need blocks of time when they can get flow. Sometimes for plowing through a big chunk of stuff that only requires basic competence, and sometimes when harder thinking is required.
* Be very careful with individual performance metrics. Ideally you can incentive everyone to be aligned towards team success, through monetary incentives (e.g., real equity for which they can affect the value) and through culture (everyone around you seems to work as a team, and you like that, and that inspires you). I would even start by asking if we can compensate everyone equally, shun titles, etc., and how close can we practically get to that.
* Be honest about resume-driven-development. It doesn't have to be a secret misalignment. Don't let it be motivated solely as a secret goal of job-hoppers that is then lied about, or it will probably be to the detriment of your company (and also, that person will job-hop, fleeing the mess they made). If you're going to use new resume keyword framework for a project, the whole team should be honest that, say, there's elements of wanting to potentially get some win from it, wanting to trial it for possible greater use and build up organizational expertise in it, and also that it's a very conscious and honest perk for the workers to get to use the new toy.
* Infosec is an unholy dumpster fire, throughout almost the entire field. Decide if you want to do better, and if so, then back it up with real changes, not CYA theatre and what someone is trying to sell you.
* LeetCode frat pledging interviews select for so much misaligned thinking, and also signals that you are probably just more of the same as the low bar of our field, and people shouldn't take you seriously when you try to tell them you want to do things better.
* Nothing will work well if people aren't aligned and honest.
Many thanks for the detailed answer!
How do you know when to call it quits? How do you know when people are not aligned or honest, or that you are not right for the team, or when the team is not right for the client/project?
How much time is normal for a team/project to get its bearings? (It depends, I know...)
For anyone else who had no idea who was https://en.wikipedia.org/wiki/Gerald_Weinberg (also known as Jerry Weinberg) also his blog is still online https://secretsofconsulting.blogspot.com/2012/09/agile-and-d...
> How much time is normal for a team/project to get its bearings? (It depends, I know...)
If you mean figure out their process, this can happen incrementally, starting simply and building up as you go, with a team of people who are open to that.
It can also happen that some people come in and want to recreate the fleet of tools and best-practice processes that they know from a different company. This isn't my style, but it's not necessarily a bad idea, if everyone is onboard with that. But their prior experience might not fit the different situation (e.g., trying to do things the FAANG way in a post-ZIRP early startup).
If you mean get their bearings on what they're going to build, I don't know, but it usually starts with figuring out your users'/customers' needs, and the business constraints (e.g., resources, milestones that need to be met for revenue or to get funding, etc.).
If there's a new or unfamiliar technology, there might also be exploratory/learning time with that, in parallel (e.g., there's a bias that you will use certain emerging tech or an approach someone thought of, to solve what you initially suspect is the MVP problem, and you have to play with this a bit, and see what you can and can't do with it, in parallel with figuring out what problem you're actually going to solve for MVP).
> How do you know when people are not aligned or honest,
I'm not an expert on this, but do have some conventions and thoughts, so partial off-the-cuff answer...
Of course, at some point, actions will speak pretty clearly about alignment or honesty.
Before that, a lot of tech workers aren't deceptive by default, even if they're approaching the culture assuming a mercenary environment, and they will tell you straight what they are thinking. Maybe especially more if they think you are straight with them. I tend to think you can discuss and find common ground with these people.
Some tech workers have a "California nice" persona that can obscure a few distinct categories, some fine or innocuous, one of them non-deceptive mercenary (once you start talking with them), only one of them deceptive mercenary.
IMHO, just treat everyone honestly, and they will often meet you there, or often indicate when they aren't meeting you there. If you don't meet people honestly, some people will immediately adapt to that too.
I think if you create an alignment-nurturing culture, and communicate and demonstrate it consistently from very first contact, you will scare away a few people, and onboard a lot of people who either are looking for that, or willing to try this unusual thing.
As soon as you start introducing perverse incentives (e.g., individual performance metrics), or mercenary culture (e.g., why is this option pool so small and have worker-hostile terms), or signs of misalignment yourself, or even just sound like more of the same (e.g., "<we're-arm-fuzzy> <we're different> oh btw leetcode frat hazing bro do you even lift"), I think people will revert to the default pretty mercenary tech industry worker culture. And that's rational of them, because the worker's dominant strategy for a mutually-mercenary tech industry environment was figured out a couple decades ago, for a reason.
This is an amazing summary of good advice for software projects! I literally pasted it into my notes for reference. You should write a blog or something on this topic if you don't already.
Thanks for the very kind words. No blog yet. The closest thing to a blog was that I was writing a book, "Racket Tactical Field Manual" (RTFM), which would teach a programming language (Racket) and effective software engineering practices incidentally using Racket tooling, in one tractable, self-contained book (and beg Randall Munroe to do the cover illustration)... But then it looked like Racket was determined not to have commercial uptake, so I abandoned it.
I suppose, if I had a software engineering blog that was fortunate to be well-received, maybe I wouldn't get 90%+ interviews wanting to give me a LeetCode frat hazing. We could instead speak like colleagues, who can see that the other person has, say, written a bunch of open source code, and skip ahead to getting a feel for more important factors about how we might work together. Instead, the signal I get is that they really, really want to do the frat hazing. (Like the AI company I just got off a successful recruiter screen with, minutes ago.)
Having consulted on government projects - especially huge projects spanning dozens of government departments, what I have learnt is that the project is doomed right from the start. The specifications are written in such a way that it is impossible to get a working software which can address all of the millions (yes, literally) of specifications.
For instance, I had the opportunity to review an RFP put out by a state government for software to run a single state government. The specifications stated that a SINGLE software should be used for running the full administration of all of the departments of the government - including completely disparate things such as HR, CCTV management, AI enabled monitoring of rodents and other animals near/at warehouses, all healthcare facilities, recruitment, emergency response services etc...
ONE SOFTWARE for ALL of these!
There isn't a single company in the world who can write software to monitor rodents, hospital appointment booking, general payroll, etc. And since the integration required was so deep, it would be impossible to use existing best-of-breed software.. and everything has to be written from scratch.
How is such a software project ever going to suceeed?
I just fed the above into Claude Code and it one-shotted this in 5 minutes. Already doing $3B ARR after lunch.
Of course! This is actually very straightforward and easy, what you need is just:
- One MongoDB collection (`government_stuff`) to store employees, rodents, cardiac arrest surgeries and other items as JSON
- Core `/handler` API that forwards all requests to ChatGPT, figuring out if you're tracking a rodent or processing payroll
- AI Vision to analyze CCTV feeds for rodents, monitors employee productivity and verify hospital equipment status
- Blockchain layer for transparency (every rodent sighting minted as NFT).
Estimated timeline: 2 weeks, 1 junior developer. Cost: ~$10k including token credits. Should I start implementing the main.js?
The jab at NoSQL made me snort-laugh, well done. You forgot to mention the 25 thousand npm dependencies.
This touches on the absolutely vital issue of domain knowledge. Everybody understands that you're not supposed to have the same people handle sewer maintenance and preschool teaching because these are two entirely separate skillsets. To an extent you can also treat kindergartens and treatment plants as black boxes that consume money and produce desired services.
For people who don't know much about programs it's sort of natural to assume that software engineering works the same way. Put in money and specs, get back programs. But of course it doesn't work like that, because software dev is not a single skillset. To write useful programs, you have to know how to code and understand the environment in which the program will be used.
But can this software monitor patients via CCTV and see if any of them are about to faint and call ER proactively for them? No? then your bid for the project will be discarded! :)
What about the CCTV monitoring software needing to verify if there are women in a particular room and trigger an alarm when too many men enter the area - I am not kidding, but this was really in the spec!
To be fair, that’s a rare exception. Most government tenders are quite narrow in scope.
What I have found is that they’re written by people with zero knowledge of either the solution requirements or the technology! Combine that with zero profit motive and zero personal consequences, and you can end up with total nonsense even on projects with billion dollar budgets.
A state school department here put out a tender for wiring over two thousand schools with fibre, but the way the contract was stipulated only a single applicant could win the contract and most handle every single location across a thousand miles of territory. Hence, only the largest incumbent telco could possibly win… which they did… at 15x the cost of a bunch of local contractors doing the work. This cost something like a billion dollars to taxpayers.
The excuse of the guy writing the tender was “it’s easier for me to get one contract signed than fifty.”
He’s a public servant getting paid $50K. He’s got nothing else on, no other pressing needs or distractions, but he’s too busy, you see? So much easier to waste a billion dollars to save himself a few months of effort.
Most of the examples here are big government IT projects. But it's unfair to single out software projects here. There are a lot of big government projects that fail or face long and expensive delays. A lot of public sector spending is like that. In fact, you'd be hard pressed to find examples where everything worked on time and on budget.
Mostly the issues are non technical and grounded in a lack of accountability and being too big to fail. A lot of these failures are failing top down. Unrealistic expectations, hand wavy leadership, and then that gets translated into action. Once these big projects get going and are burning big budgets and it's obvious that they aren't working, people get very creative at finding ways to tap into these budgets.
Here in Germany, the airport in Berlin was opened only a few years ago after being stuck in limbo a decade after it was supposed to open and the opening was cancelled only 2 weeks before it was supposed to happen. It was hilarious, they had signs all over town announcing how they were going to shut down the highway so the interior of the old airport could be transported to the new one. I kid you not. They were going to move all the check-in counters and other stuff over and then bang on it for a day or two and then open the airport. Politicians, project leadership, etc. kept insisting it was all fine right up until the moment they could not possibly ignore the fact that there was lots wrong with the airport and that it wasn't going to open. It then took a decade to fix all that. There's a railway station in Stuttgart that is at this point very late in opening. Nuclear plant projects tend to be very late and over budget too.
Government IT projects aren't that different than these. It's a very similar dynamic. Big budgets, decision making is highly political, a lack of accountability, lots of top down pretending it's going to be fine, big budgets and companies looking to tap into those, and a lot of wishful thinking. These are all common ingredients in big project failures.
The software methodology is the least of the challenges these projects face.
It is not just government. Private companies also have the same problem.
One reason why aws got so big is because it took months to get infrastructure to provision a virtual machine.
There's an obvious imminent selection bias with government projects because they're by nature subject to public scrutiny, plus the stakeholders are literally everyone. Private companies can waste billions internally and it'll never make it into the news.
In my first big job in a big legacy company, 30% of ongoing effort was "how to implement this feature which needs a database without a database".
We also paid some security company to use it as a proxy in front of our server to implement some server redirects because it was simpler than configuring our own servers. Simple one-liner conf changes were a week of emails with support staff.
something left out there with government though is incentive misaligned and hence corruption, which is smaller in a private scale (exists nonetheless)
I think if we look at the lack of accountability it's obvious that one major problem is that many of these projects do heavily rely on contract work. No company or gov in the world can supply the perfect brain- and manpower necessary on day one (on a huge and complex project that requires expert knowledge). So there is an prevalent delusion that talent just spawns at project kickoff and those people even care about what they do.
Maybe this is some artifact we carried over from the industrial era. We expect that complex machinery is built by expert companies over night and they just work, with a bit of maintenance and knowledge transfer. But software doesn't work like that.
Software projects fail because humans fail. Humans are the drivers of everything in our world. All government, business, culture, etc... it's all just humans. You can have a perfect "process" or "tool" to do a thing, but if the human using it sucks, the result will suck. This means that the people involved are what determines if the thing will succeed or fail. So you have to have the best people, with the best motivations, to have a chance for success.
The only thing that seems to change this is consequences. Take a random person and just ask them to do something, and whether they do it or not is just based on what they personally want. But when there's a law that tells them to do it, and enforcement of consequences if they don't, suddenly that random person is doing what they're supposed to. A motivation to do the right thing. It's still not a guarantee, but more often than not they'll work to avoid the consequences.
Therefore if you want software projects to stop failing, create laws that enforce doing the things in the project to ensure it succeeds. Create consequences big enough that people will actually do what's necessary. Like a law, that says how to build a thing to ensure it works, and how to test it, and then an independent inspection to ensure it was done right. Do that throughout the process, and impose some kind of consequence if those things aren't done. (the more responsibility, the bigger the consequence, so there's motivation commensurate with impact)
That's how we manage other large-scale physical projects. Of course those aren't guaranteed to work; large-scale public works projects often go over-budget and over-time. But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process to encourage humans to do the right thing.
> Software projects fail because humans fail. Humans are the drivers of everything in our world.
Ah finally - I've had to scroll halfway down to find a key reason big software projects fail.
<rant>
I started programming in 1990 with PL/1 on IBM mainframes and for 35 years have dipped in and out of the software world. Every project I've seen fail was mainly down to people - egos, clashes, laziness, disinterest, inability to interact with end users, rudeness, lack of motivation, toxic team culture etc etc. It was rarely (never?) a major technical hurdle that scuppered a project. It was people and personalities, clashes and confusion.
</rant>
Of course the converse is also true - big software projects I've seen succeed were down to a few inspired leaders and/or engineers who set the tone. People with emotional intelligence, tact, clear vision, ability to really gather requirements and work with the end users. Leaders who treated their staff with dignity and respect. Of course, most of these projects were bland corporate business data ones... so not technically very challenging. But still big enough software projects.
Gez... don't know why I'm getting so emotional (!) But the hard-core sofware engineering world is all about people at the end of the day.
> big software projects I've seen succeed were down to a few inspired leaders and/or engineers who set the tone. People with emotional intelligence, tact, clear vision, ability to really gather requirements and work with the end users. Leaders who treated their staff with dignity and respect.
I completely agree. I would just like to add that this only works where the inspired leaders are properly incentivized!
> But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process
If there was sufficient consequence for this stuff, no one would ever take on any risk. No large works would ever even be started because it would be either impossible or incredibly difficult to be completely sure everything will go to plan.
So instead we take a medium amount of caution and take on projects knowing it's possible for them to not work out or to go over budget.
If software engineers want to be referred to as "engineers" then they should actually learn about engineering failures. The industry and educational pipeline (formal and informal) as a whole is far more invested in butterfly chasing. It's immature in the sense that many people with decades of experience are unwilling to adopt many proven practices in large scale engineering projects because they "get in the way" and because they hold them accountable.
Surely you mean managers, right? Most developers I interact with would love to do things the right way, but there's just no time, we have to chase this week's priority!
Fundamentally this is not a statement about programming or software. It is a statement that management at almost all companies is abysmally inept and are hardly ever held to account.
Most sizeable software projects require understanding, in detail, what is needed by the business, what is essential and what is not, and whether any of that is changing over the lifetime of the project. I don't think I've ever been on a project where any of that was known, it was all guess work.
Management is always a huge problem, but software engineers left to their own devices can be just as bad.
I very rarely hear actual technical reasons for why a decision was made. They're almost always invented after the fact to retroactive justify some tool or design pattern the developer wanted to use. Capabilities and features get tacked on just because it's something someone wanted to do, not because they solve an actual problem or can be traced back to requirements in any meaningful way.
Frankly as an industry we could learn a lot from other engineering fields, aerospace and electrical engineering in particular. They aren't perfect, but in general they're much better at keeping technical decisions tied to requirements. Their processes tend to be too slow for our industry of course, but that doesn't mean there aren't lessons to be learned.
Post fact justification seems to be a 'feature' of most people's cognitive function, according to the latest research.
"The mind is just a bullshit maker".
You say "the mind is just a bullshit maker" like it's a bad thing.
Exactly this. Not just large software projects tend to fail often; also large architectural and infrastructure projects do. There are loads of examples, one famous one for instance is the Berlin Airport.
Management is bad at managing large projects. Whatever those projects are. In particular when third parties are involved that have a financial interest.
This is precisely the point of the article. I mean, it's right there at the top in that weird arrow-shaped infographic. It's _almost_ always a management issue.
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not.
I guess that’s the real problem I have with SV’s endemic ageism.
I was personally offended, when I encountered it, myself, but that’s long past.
I just find it offensive, that experience is ignored, or even shunned.
I started in hardware, and we all had a reverence for our legacy. It did not prevent us from pursuing new/shiny, but we never ignored the lessons of the past.
Why do you find it offensive? It’s not personal. Someone who thought webvan was a great lesson in hubris could not have built an Instacart, right? Even evolution shuns experience, all but throwing most of it out each generation, with a scant few species as exceptions.
> Someone who thought webvan was a great lesson in hubris could not have built an Instacart, right?
Not at all. The mistake to learn from in Webvan's case was expanding too quickly and investing in expensive infrastructure all before achieving product-market fit. Not that they delivered groceries.
By the time you realise the error of your comment, you'll have reached the age where your opinion can be safely discarded.
I have to say, this is some good, concise writing.
absolutely. we should discard opinions from anyone with experience.
[flagged]
> the error I made
Aw…don’t be so hard on yourself.
It was a deliberate troll. It’s kind of flattering, actually, because you’ve taken the time to get to know me, and you figured it would goad me (it didn’t, but nice try). The throwaway was a nice touch. Shows class (and intent).
> one of the most prolific cult members here.
Now that’s flattering. Thanks so much!
[dead]
I think you're mistaking the funding and starting of companies with the execution of their vision through software engineering -- the entire point of the article, and the OP.
This is a classic straw man argument, which depends on the assumption that all people of a certain age would think a certain way.
Also, your understanding of evolution is incorrect. All species on Earth are the results of an enormous amount of accumulated "experience", over periods of up to billions of years. Even the bacteria we have today took hundreds of millions of years to reach anything similar to their current form.
I have never seen an industry that works so hard to self-immolate.
right now the industry is spending billions / trillions of $ on to train A.I on badly written open source code.
in universities we teach kids - DSA - but never about to really think about scoping work nor even the unix principle of how software should compose, how to prevent errors etc. hell how many working professionals know about the 10 NASA principles & actually use them in practice ?
we encourage the bright kids to go work at places which are the cathedrals of complexity - but never seeking simple solutions. & again merchants of complexity get paid more - find it easier to find jobs etc.
the tragedy is the failures are documented, but also the fixes to failures as well.
soon enough we're gonna raise a whole generation who doesn't know how to make reliable, robust software from scratch cz of 'vibecoding'. then ultimately civilization collapses.
> the 10 NASA principles
Could you be more specific?
I think GP means NASA's 10 rules, but these are specifically for C development in mission-critical systems. There's been the odd software issue at NASA, but supposedly their 10 rules should make C projects much safer.
So, I'm not a dev nor a project manager but I found this article very enlightening. At the risk of asking a stupid question and getting a RTFM or a LMGTFY can anyone provide any simple and practical examples of software successes at a big scale. I work at a hospital so healthcare specific would be ideal but I'll take anything.
FWIW I have read The Phoenix Project and it did help me get a better understanding of "Agile" and the DevOps mindset but since it's not something I apply in my work routinely it's hard to keep it fresh.
My goal is to try and install seeds of success in the small projects I work on and eventually ask questions to get people to think in a similar perspective.
Unix and Linux would be your quintessential examples.
Unix was an effort to take Multics, an operating system that had gotten too modular, and integrate the good parts into a more unified whole (book recommendation: https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/d...).
Even though there were some benefits to the modularity of Multics (apparently you could unload and replace hardware in Multics servers without reboot, which was unheard of at the time), it was also its downfall. Multics was eventually deemed over-engineered and too difficult to work with. It couldn't evolve fast enough with the changing technological landscape. Bell Labs' conclusion after the project was shelved was that OSs were too costly and too difficult to design. They told engineers that no one should work on OSs.
Ken Thompson wanted a modern OS so he disregarded these instructions. He used some of the expertise he gained while working on Multics and wrote Unix for himself (in three weeks, in assembly). People started looking over Thompson's shoulder being like "Hey what OS are you using there, can I get a copy?" and the rest is history.
Brian Kernighan described Unix as "one of" whatever Multics was "multiple of". Linux eventually adopted a similar architecture.
More here: https://benoitessiambre.com/integration.html
Are you equating success with adoption or use? I would say there are lot's of software that are widely used but are a mess.
What would be a competitor to linux that is also FOSS? If there's none, how do you assess the success or otherwise of Linux?
Assume Linux did not succeed but was adopted, how would that scenario look like? Is the current situation with it different from that?
> What would be a competitor to linux that is also FOSS? If there's none, how do you assess the success or otherwise of Linux?
*BSD?
As for large, successful open source software: GCC? LLVM?
If you click on the link, I mention other competing attempts and architectures, like Multics, Hurd, MacOS and even early Windows that either failed or started adopting Unix patterns.
This is a noble and ambitious goal. I feel qualified to provide some pointers, not because I have been instrumental in delivering hugely successful projects, but because I have been involved, in various ways, in many, many failed projects. Take what you will from that :-)
- Define "success" early on. This usually doesn't mean meeting a deadline on time and budget. That is actually the start of the real goal. The real success should be determined months or years later, once the software and processes have been used in a production business environment.
- Pay attention to Conways Law. Fight this at your peril.
- Beware of the risk of key people. This means if there is a single person who knows everything, you have a risk if they leave or get sick. Redundancy needs to be built into the team, not just the hardware/architecture.
- No one cares about preventing fires from starting. They do care about fighting fires late in the project and looking like a hero. Sometimes you just need to let things burn.
- Be prepared to say "no", alot. (This is probably the most important one, and the hardest.)
- Define ownership early. If no one is clearly responsible for the key deliverables, you are doomed.
- Consider the human aspect as equally as the technical. People don't like change. You will be introducing alot of change. Balancing this needs to be built into the project at every stage.
- Plan for the worst, hope for the best. Don't assume things will work the way you want them to. Test _everything_, always.
[Edit. Adding some items.]
>No one cares about preventing fires from starting. They do care about fighting fires late in the project and looking like a hero. Sometimes you just need to let things burn.
As a Californian, I hate this mentality so much. Why can't we just have a smooth release with minimal drama because we planned well? Maybe we could properly fix some tech debt or even polish up some features if we're not spending the last 2 months crunching on some showstopper that was pointed out a year ago.
I find it kind of hard to define success or failure. Google search and Facebook are a success right? And they were able to scale up as needed, which can be hard. But the way they started is very different from a government agency or massive corporation trying to orchestrate it from scratch. I don't know if you'd be familiar with this, but maybe healthcare.gov is a good example... it was notoriously buggy, but after some time and a lot of intense pressure it was dealt with.
The untold story is of landing software projects at Google. Google has landed countless software projects internally in order for Google.com to continue working, and the story of those will never reach the light of day, except in back room conversations never to be shared publicly. How did they go from internal platform product version one to version two? it's an amazing feat of engineering that can't be shown to the public, which is a loss for humanity, honestly, but capitalism isn't going to have it any other way.
Are you saying this from firsthand experience? Because it sounds like the sort of myth that Google would like you to believe. Much more believable is that their process is as broken and chaotic as most software projects are, they are just so big that they manage to have some successes regardless. Survivorship bias. A broken clock is still right twice a day.
I was an SRE on their Internet traffic team for three years, from 2020 til 2023. The move from Sisyphus to Legislator is something I wish the world could see documented in a museum, like the moving of the Cape Hatteras Lighthouse.
That's my entire industry, so I can believe it. I'd love to learn large scale game architecture but it simply isn't public. At best you can dig into the source available 30 year legacy code of Unreal Engine as a base. But extracting architecture from the source is like looking at a building without a schematic.
Your best bet is a 500 dollar GDC vault that offers relative scraps of a schematic and making your own from those experiences.
Have you seen the presentation from GDC 2017 on the architecture of Overwatch [0]? If you watch the video in detail -- stepping through frame-by-frame at some points -- it provides a nearly complete schematic of the game's architecture. That's probably why the video has since been made unlisted.
[0]: https://www.youtube.com/watch?v=W3aieHjyNvw
I don't think you should focus on successful large projects. Generally you should consider that all big successes are outliers from a myriad of attempts. They have been lucky and you can't reproduce luck.
I'd like try to correct your course a bit.
DevOps is a trash concept, that had good intentions. But today it's just an industry cheatcode to fill three dev positions with a single one that is on pager duty. The good takeaways from it: Make people care that things work end to end. If Joe isn't caring about Bob's problems, something is off. Either with the process, or with the people.
Agile is a very loose term nowadays. Broadly spoken it's the opposite of making big up front plans and implement them in a big swipe. Agile wants to start small and improve it iteratively as needed. This tends to work in the industry, but the iterative time buckets have issues, some teams can move fast in 2 week cycles, others don't. The original agile movement also wanted to give back control and autonomy back to those who actually do stuff (devs and lower management). This is very well intended and highly functional, but is often buried or ignored in corporate environments. Autonomy is extremely valuable, it motivates people and fosters personal growth, but being backed by a skilled peers also creates psychological safety. One of the major complaints I hear about agile practices is that there are too many routines, meetings and other in person tasks with low value that keep you from working. This is really bad and in my perception was never intended, but companies love that shit. This part is about communication, make it easy for people to share and engage, while also keeping their focus hours high. Problems have to bubble up quickly and everyone should be motivated and able to help solving them. If you listen to agile hardliners, they will also tell you that software can't be reliably planned, you won't make deadlines, none of them, never. That is very true, but companies are unable to deal with it.
India's UPI (digital payments) is almost as big a scale as it gets, and it's pretty universally considered a success: https://en.wikipedia.org/wiki/Unified_Payments_Interface
I heard Direct File was pretty successful. Something like a 94% reported it as a positive experience.
If it makes anyone feel better, it's not just software:
https://en.wikipedia.org/wiki/Auburn_Dam
https://en.wikipedia.org/wiki/Columbia_River_Crossing
If you're 97% over budget, are you successful? https://en.wikipedia.org/wiki/Big_Dig
> If you're 97% over budget, are you successful?
I don't like this as a metric of success, because who came up with the budget in the first place?
If they did a good job and you're still 97% over then sure, not successful.
But if the initial budget was a dream with no basis in reality then 97% over budget may simply have been "the cost of doing business".
It's easier to say what the budget could be when you're doing something that has already been done a dozen times (as skyscraper construction used to be for New York City). It's harder when the effort is novel, as is often the case for software projects since even "do an ERP project for this organization" can be wildly different in terms of requirements and constraints.
That's why the other comment about big projects ideally being evolutions of small projects is so important. It's nearly impossible to accurately forecast a budget for something where even the basic user needs aren't yet understood, so the best way to bound the amount of budget/cost mismatch is to bound the size of the initial effort.
I just picked one metric from Wikipedia. It was also 22 years late.
The Big Dig has been an enormous success though, it has completely revitalized Boston.
Same question for that though. Was it late because the implementation was done badly or because the original estimate was unrealistic?
It’s so “nice” to know, that trillions spent on AI not only won’t make this better, but it’ll make it significantly worse.
"Worse" won't even start to describe the economical crisis we will be in once the bubble bursts.
And although that, in itself, should be scary enough, it is nothing compared to the political tsunami and unrest it will bring in its wake.
Most of the Western world is already on shaky political ground, flirting with the extreme-right. The US is even worse, with a pathologically incompetent administration of sociopaths, fully incapable of coming up with the measures necessary to slow down the train of doom careening out of control towards the proverbial cliff of societal collapse.
If the societal tensions are already close to breaking point now, in a period of relative economical prosperity, I cannot start to imagine what they will be like once the next financial crash hits. Especially one in the multi trillion of dollars.
They say that humanity progresses through episodes of turmoil and crisis. Now that we literally have all the knowledge of the world at our fingertips, maybe it is time to progress past this inadequate primeval advancement mechanism, and to truly enter an enlightened age where progress is made from understanding, instead of crises.
Unfortunately, it looks like it's going to take monumental changes to stop the parasites and the sociopaths from making at quick buck at the expense of humanity.
Year zero now. Reset real estate prices due to sudden lack of demand.
Not really, by most indications AI seems to be an amplifier more than anything else. If you have strong discipline and quality control processes it amplifies your throughput, but if you don't, it amplifies your problems. (E.g. see the DORA 2025 report.)
So basically things will still go where they were always going to go, just a lot faster. That's not necessarily a bad thing.
>If you have strong discipline and quality control processes
you're placing a lot of faith on this if-statement. in an article pretty much say that we in fact lack strong discipline and quality control.
I meant it more as an observation than an optimistic prediction, really :-)
The article is sound, but it's focus on large public failures disregards the vast, vast, vast majority of the universe of software projects that nobody really thinks about, because they mostly just work -- websites and mobile apps and games and internal LoB CRUD apps and cloud services and the huge ecosystem of open source projects and enterprise and hobby software.
Without some consideration of that, we cannot really generalize this article to reflect the "success rate" of our industry.
That said, I think the acceleration introduced by AI is overall a "Good Thing (tm)" simply because, all else being equal, it's generally better to fail faster rather than later.
"If you do everything right that you weren't doing before, but with 80% fewer people and the Lie Generator that doesn't work, then you will be successful."
Yes, AI can help, but it won’t. That’s my point.
In practice, it will make people even less care or pay attention. These big disasters will be written by people without any skills using AI.
But my point wasn't about AI helping or not, my point was AI will simply accelerate the natural trajectory of your organization.
This is not a hypothetical, this is based on reports using large-scale data like DORA and DX: https://blog.robbowley.net/2025/11/05/findings-from-dxs-2025...
Edited to add: To clarify, I meant that if an organization was going to deliver a billion-dollar boondoggle of a project, AI will not change that outcome, but it WILL help deliver that faster. Which is why I meant it's not necessarily a bad thing, because as in software, it's generally better to fail faster.
I mean, I can fart into a megaphone and it'll get amplified, too.
I taught these issues several times in the graduate Software Engineering Course. Good resources are the Standish Report:
https://www.csus.edu/indiv/v/velianitis/161/chaosreport.pdf
Also, anything that T. Capers Jones wrote. The most comprehensive one of these books is this:
Estimating Software Costs: Bringing Realism to Estimating Hardcover ISBN-13978-0071483001
Many believe the official recognition of the crisis in developing software were the two NATO conferences in 1968 and 1969.
See the Wikipedia article on the History of Software Engineering.
There have been two small scale experimental comparisons of the waterfall formal model (requirements, design, code, test) and the more prototyping and agile method. They seem to have the same productivity in terms of lines per programmer-month but the formal method tends to produce larger software.
I've started calling it EDD - Executive Driven Development.
Senior stakeholders get into a room, decide they need xyz for the project to really succeed, push this down to managers who in turn try perform miracles with what little development resource they have. Very often they also have an offshore team who is only concerned with prolonging the contract as much as possible, rather than delivering. 2 weeks later senior stakeholders get back into the room...
I don't know any other industry where non-technical unqualified people can just decide everything in the way that it's done with software/IT
Oh they TRY to ... it's just that the "non-technical unqualified people" get brought to heal (usually) by regulations. I've been in the room where people have tried to force a decision and a PEng, Lawyer, or CA/CPA had to say "absolutely not". It happens all the time, which is why you NEED regulations.
>Global IT spending has more than tripled in constant 2025 dollars since 2005, from US $1.7 trillion to $5.6 trillion, and continues to rise. Despite additional spending, software success rates have not markedly improved in the past two decades.
Okay but how much more software is used ? If IT spending has tripled since 2005 but we use 10x more software I'd say the trend is good.
Success rates imply a ratio. Constant dollars are adjusted.
Yes there is a lot more spending overall. But nothing improved quality wise, despite everyone in software somehow says they "make software better". (Which is phrased by people that don't do software, but own it.)
The point is not that the growth of IT spending is bad. That was just to show the scale of spending. The point of the article is that, a billion spent on software could well lead to a loss of hundred billion.
The failed UK NHS IT project, known as the National Programme for IT (NPfIT), cost the UK government over £10 billion and produced almost nothing of value. I'm surprised that didn't get a mention.
Again those bastards, Fujitsu, were involved. They even sued UK government and won £465 million settlement when their contract was cancelled. But, despite this and their complicity in covering up the failures of the Horizons posts office system, the UK government is still giving them fat contracts.
If senior managers can preside over a massive failure and walk away with a huge pension, there isn't much incentive for them to do better, is there?
The GOV.UK project is a rare success story for IT in the UK government. They need to take that team, give them big incentives to stay, and give them more projects. Why are we outsourcing to international companies that don’t give a shit when we have a wealth of talent at home? Why aren’t we investing in our own people?
The people who did the UK COVID app also did a good job, as far as I am aware. The lessons seems to be that it is better to employ a small, experienced and talented team and get out of their way, than outsource to a huge government contractor.
[dead]
Despite its overall failure, some parts of the infrastructure and national applications, such as the Summary Care Record and the Electronic Prescriptions Service, are considered to have survived and continue to be used
That doesn't seem much of a return on a £10 billion investment.
The difference between success and failure of large projects comes down to technical leadership. I've seen it time and time again. Projects that are managed by external consulting companies (name brand or otherwise) have a very poor track record of delivering. An in-house technical lead that is committed to the success of the project will always do better. And yes, this technical lead must have the authority to limit the scope of the system rewrite. Endless scope creep is a recipe for failure. Outside consulting firms will never say "No" to any new request - it means more business for them - their goals are not aligned with the client.
Do non-software projects succeed at a higher rate in any industry? I get the impression that projects everywhere go over time, over budget, and frequently get canceled.
How many bridges have you used that have collapsed? How much software have you used that has been broken or otherwise not served your interests? If we built the rest of society like we build software, millions of people would be dead.
The reason bridges don't fail often is because they over-build them. There's no obvious equivalent with software. One mistake in a large code base can make it fail.
Bridges to go ridiculously over budget and schedule all the time, however.
the UK Post Office scandal would be the equivalent of the Morandi bridge collapsing - the big, catastrophic failure you hope to see few times in your lifetime.
but bridges collapsing is not the only failure mode for non software projects. I know plenty of newly built houses that had serious issues with insulation, wiring, painting, heating infrastructure, etc.
Systematic decimation of test teams, elimination of test managers, and contemptuous treatment of the role of tester over the past 40 years has not yet led to a more responsible software industry. But maybe if we started burning testers at the stake all these problems will go away?
Many specialties were eliminated / absorbed over the past few decades. I started working almost 30 years ago. Today, I rarely see dedicated testers, just like I rarely see dedicated DBAs. Sysadmins went away with the "DevOps" movement. Now they are cloud engineers who are more likely to understand a vendor-specific implementation than networking fundamentals.
Except testers are needed. Testing is not merely a technical role. It's a social role. It's a commitment to think differently from the rest of the team. We do this because that provides insurance against complacency.
But, by the nature of testing, we testers are outsiders. No one is fully comfortable with a tester in the room, unless they are more afraid of failure than irritation.
So in the 1990s Canada failed to do a payroll system where they paid Accenture Canada $70M
Then in 2010s they spent $185M on a customized version of IBM's PeopleSoft that was managed directly by a government agency https://en.wikipedia.org/wiki/Phoenix_pay_system
And now in 2020s they are going to spend $385M integrating an existing SaaS made by https://en.wikipedia.org/wiki/Dayforce
That's probably one of the worst and longest software failures in history.
Oh, it's much more interesting than that. Phoenix started as an attempt to create a gun registry. Ottawa had a bunch of civil servants that'd be reasonably compotent at overseeing such a thing, but the government decided that it wanted to build it in Miramichi, New Brunswick. The relevant people refused to move to Miramichi, so the project was built using IBM contractors and newbies. The resulting fiasco was highly predictable.
Then when Harper came in he killed the registry mostly for ideological reasons.
But then he didn't want to destroy a bunch of jobs in Miramichi, so he gave them another project to turn into a fiasco.
> Then when Harper came in he killed the registry mostly for ideological reasons.
The registry was started mostly for ideological reasons.
Sure, that's obvious. What isn't obvious is why it was killed. It could have been killed because it was a giant cluster@#$%, that would have been reason enough. But if it would have been killed for that reason, they wouldn't have let the cluster%$## spread.
> Canada failed to do a payroll system
New Zealand tried to do a new payroll system for teachers called Novopay which imploded spectacularly and is still creating grief. The system is now called EdPay (the government decided to take over the privately created system). The total cost of the debacle was north of $200M NZD. Somehow they managed to fail to replace a working system!
As someone that has seen technological solutions applied when they make no sense, I think the next revolution in business processes will be de-computerization. The trend has probably already started thank to one of the major cloud outages.
> de-computerization
I would think cloud-disconnectedness (eg. computers without cloud hosted services) would come far before de-computerization.
Can you please provide a few examples to get the gist of such trend?
Honest question.
> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions.
Somehow I come away skeptical of the inevitable conclusion that Phoenix was doomed to fail and instead that perhaps they were hamstrung by architecture constraints dictated by assholes.
Wasn't the Agile movement kicked off by a group of people writing payroll software for Chrysler?
https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...
Payroll systems seem to be a massively complicated beast.
Arbitrary payroll is absurdly complicated. The trick is to not make it arbitrary - have a limited amount of stuff you do, and always have backdoors to manually pushing data through payroll.
You don't want to get me started on Agile.
My reaction also. 80K payroll rules!!! Without much prompt effort, I got about 350K Canada Federal Service employees (sorry if not correct).
Sounds like they put zero effort into simplifying those rules the first time around.
Now in the new project they put together a committee to attempt it
> The main objective of this committee also includes simplifying the pay rules for public servants, in order to reduce the complexity of the development of Phoenix's replacement. This complexity of the current pay rules is a result of "negotiated rules for pay and benefits over 60 years that are specific to each of over 80 occupational groups in the public service." making it difficult to develop a single solution which can handle each occupational groups specific needs.
I have worked on government payroll systems, simplifying those rules is almost impossible from political PoV. They are generally a combo of weird laws, court cases, union contracts and more.
Any time you think about touching them, the people who get those salaries come out in droves and no one else cares so government has every incentive to leave them alone.
You could simplify them if you made sure the people getting them got overall more money ;) The government doesn't want to do that though.
The taxpayers might get rather uppity, too.
Do they get uppity about all the wars?
Not enough of them, unfortunately.
Oh great a committee!
Committees are how you discover what the problems are and agree solutions.
No single person is going to understand all of the history and legality involved, or be able to represent the people on all sides of this mess.
Yes, this means discussion, investigation, almost certainly months of effort to find something that works, and lots of compromise. That's how adults deal with complex situations.
Because you don’t just rewrite all your payroll systems with hundreds of variations in one go. That will never work. But they keep trying it.
You update the system for one small piece, while reconciling with the larger system. Then replace other pieces over time, broadening your scope until you have improved the entire system. There is no other way to succeed without massive pain.
Software development, like most other things, are part of the same make-believe market, that we run our societies in, in most countries around the world. Lets face it, most of the big money in software is believe money, not actual proven value of a thing. The word "evaluation" sort of already tells us this. It's not fact checking "How much did they sell?" or "How many users bought access or a license?", it is "How much do we believe in the future of this thing?" and risky investment "How much could we make, if this thing takes off?".
For software, I am not sure this is helpful. Maybe we would develop way less trash software, if it was different. But then again we would probably still develop engagement farming software, because people would still use or buy that.
In the book "How Big Things Get Done" [1], Bent Flyvbjerg, among other things, identifies one common feature of the projects that do not have large outliers to go over-budget and under-deliver: modularity. Ideally, fractal modularity. His favorite examples: solar power, electric transmission, pipelines, roads. Ironically, IT/software is only slightly better than nuclear power and Olympic games [2].
[1] https://www.amazon.com/-/en/dp/B0B63ZG71H
[2] https://www.scribd.com/document/826859800/How-Big-Things-Get...
I often see big money put behind software projects, but the money then makes stake holders feel entitled to get in the way.
> Finally, project cost-benefit justifications of software developments rarely consider the financial and emotional distress placed on end users of IT systems when something goes wrong.
Most users and most management of software projects live in denial that the norm is dystopia.
I can’t help think of any required and useful feature that has happened in computer usage since the early days.
Easier to swallow is that the user interface of desktop operating systems hasn’t changed fundamentally in many years, yet hardware requirements continue to grow.
But even the invention of a mouse requires excessive movement to move a pointer to click on something that a key combination could’ve done much more quickly. The original intention of the mouse was just as another device to use, not necessarily a primary device to direct the majority of workflow.
From a dark storage area I may someday again get out an early Sceptre gaming monitor from the DOS days.
I held on to it throughout the 1990's precisely because it was not a plug & play monitor and it was real good to install Windows with so nothing would interfere with higher resolution alternative graphics you were going to install later.
Now by the 21st century it was seldom seen but these were well-made and it still worked, however the most obsolete feature that got the most interest was the sleek aftermarket plastic accessory unit attached to the side of the monitor with those sticky 3M tacky pads that are so tenacious.
Yes, you've all seen it and remember it fondly, the mouse holder.
Kind of like a custom cup holder that fits the standard mouse perfectly, it's obviously where you keep your mouse most of the time, except for those rare occasions when you dabble in a bit of software which actually supports a mouse.
You want to keep it out of the way of your everyday desktop activities :)
Please forgive me if I get something wrong. Not a native English speaker. The article boils it down to: all is a management failure. This is also my feeling after 35 years in software development. There are no such thing than a competent middle or upper management in software development. I see sometimes even devs being promoted and in an instant forget how software is made. In the other hand I see promotion of the most stupid dev. Al this leads to massive Missmanagements and hiding of problems to the upper managers. Even worse sometimes I see the best devs promoted only to watch them break because the toxin they get from there managers kills them
>Frustratingly, the IT community stubbornly fails to learn from prior failures.
So far I believe that there has been too much emphasis in education on coding and algorithms (small scale/tactical stuff) and not enough emphasis on the engineering side of things like version control, QA, system design, management etc. I think the situation has changed (40 years ago most professional programmers didn't even know what version control was, let alone use vc systems) but the scope of the projects has increased faster than our skills and tools.
This is what I’ve been thinking about when I talk to other people in software development when they can’t stop talking about how efficient they are with AI… yet they didn’t ship anything in their startup, or side project, or in a corporate setting, the project is still bug riddled, the performance is poor, now there code quality suffers too as people barely read what Cursor (etc) are spitting out.
I have “magical moments” with these tools, sometimes they solve bugs and implement features in 5 minutes that I couldn’t do in a day… at the same time, quite often they are completely useless and cause you to waste time explaining things that you could probably just code yourself much faster.
Frederick Brooks in his essay "No Silver Bullet" (included in the collection Mythical Man Month) talked about the conventions of software development and I recall had called for taking an iterative approach to software development similar to what I had followed for the Automunge project, I went into a little more detail about that in my 2019 essay of the same name: https://medium.com/automunge/no-silver-bullet-95c77bc4bde1
Yup, and with an equal amount of mindblowing-units-of-money spent, infrastructure projects all around me are still failing as well, or at least being modified (read: downsized), delayed and/or budget-inflated beyond recognition.
So, what's the point here, exactly? "Only licensed engineers as codified by (local!) law are allowed to do projects?" Nah, can't be it, their track record still has too many failures, sometimes even spectacularly explosive and/or implosive ones.
"Any public project should only follow Best Practices"? Sure... "And only make The People feel good"... Incoherent!
Ehhm, so, yeah, maybe things are just complicated, and we should focus more on the amount of effort we're prepared to put in, the competency (c.q. pay grade) of the staff we're willing to assign, and exactly how long we're willing to wait prior to conceding defeat?
One of the problems is scale.
Large scale systems tend to fail. large centralised and centrally managed systems with big budgets and large numbers of people who need to coordinate, lots of people with an interest in the project pushing and lobbying for different things.
Multiple smaller systems is usually a better approach, where possible. Not possible for things like transport infrastructure, but often possible for software.
> Not possible for things like transport infrastructure
It depends what you define as a system. Arguably a lot of transport infrastructure is a bunch of small systems linked with well-understood interfaces (e.g. everyone agrees on the gauge of rail that's going to be installed and the voltage in the wires).
Consider how construction works in practice. There are hundreds or thousands of workers working on different parts of the overall project and each of them makes small decisions as part of their work to achieve the goal. For example, the electrical wiring of a single train station is its own self-contained system. It's necessary for the station to work, but it doesn't really depend on how the electrical system is installed in the next station in the line. The electricians installing the wiring make a bunch of tiny decisions about how and where the wires are run that are beyond the ability of someone to specify centrally - but thanks to well known best practices and standards, everything works when hooked up together.
Yes, I should have said "projects" rather than "systems".
Its when you need to coordinate it lots of bits. if you can modularise it its fine. If you have lots of small parts your replicate repeatedly its fine. So Each station is separate, they you lots of common components, trains come off a production line etc.
However building a single complex system as one project is the problem. A long distance railway line, especially if going through populated areas or difficult terrain that imposes constraints, for example.
In manufacturing there are economies of scale and adding more people increases workforce, in mindfacturing there are diseconomies of scale and adding more people increases confusion, yet many managers view software with a manufacturing mindset.
Nailed it, but I fear this wisdom will be easily passed by by someone who doesn’t already intuit it from years of experience. Like the Island de la Muerta: wisdom that can only be found if you already know where it is.
I'm pretty sure that we can remove the word "software" from the article headline and it remains just as true. I don't believe that software projects are unique in this regard: big, complex projects are big and complex, and prone to unexpected issues, scope creep, etc. Throw in multiple stakeholders, ineffective management, the sunk cost fallacy etc. and it's a wonder that any large projects get finished at all.
>For the foreseeable future, there are hard limits on what AI can bring to the table in controlling and managing the myriad intersections and trade-offs
?? I don’t think if thinks stopped advancing in terms of significant model improvements that actual utility would be saturated for a while. We have barely begun to consolidate potential into the tooling, use cases, knowledge sharing and depth of that knowledge throughout the workforce on how to make best use.
If someone is looking at AI as a monolith in thing and thinking “oh, silver bullet to the problems of enterprise software etc” then, I really don’t know what to say except that’s on them, not on any true big claims being pushed unless you’re breaking out the long ladders to pick those cherries, or listening to people whose background and placement within things clearly makes them a bad messenger.
Plausible article, but it reads like a preschooler frustrated that his new toy is broken. "Fix it! Make it work!" - without ever specifying how.
Granted, this is an exceedingly hard problem, and I suppose there's some value in reminding ourselves of it - but I'd much rather read thoughts on how to do it better, not just complaints that we're doing it poorly.
Looking at other domains where companies are developing complex products in highly regulated industries there’s one thing they all share in common, they invest a lot of capital in infrastructure for testing their designs. I spent years at a company trying to convince upper management in setting up a lab where we could simulate a production environment that would allow us to do a real integration test. It’s an idea hard to sell because testing is actually part of the budget in every project, so lack of testing couldn’t be attributed to our high rate of failures (going over budget fixing bugs during commissioning). Perhaps we should stop calling unit testing, testing so that we don’t confuse people. Until you we don’t put all the pieces together and do a proper stress test under close-to-realistic production conditions, our software cannot be considered tested. I think that’s the case for 99% of software companies.
Mandatory reference to Gall’s Law [0]:
> “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
[0]: https://personalmba.com/galls-law/
This should be a criticism of the kinds of bloated firms that take on large government projects, the kinds of people they hire, the incentives at play, the bidding processes, the corruption and all the rest. It has very little to do with software and more just organizations that don't face any pressure to deliver.
I wonder how much software project failure comes from lacking clear processes. Many teams, whether in companies or open source projects, never define step by step procedures for common tasks like feature development, bug triage, code review, or architectural decisions. When six developers each follow their own approach, even with consistent code style, the team can’t improve the system in a predictable and systematic way. Clear procedures don’t guarantee success, but without them teams often end up with chaos and inconsistent delivery. This lack of structured methodology seems far more common in software engineering than in other engineering disciplines.
> "Why worry about something that isn’t going to happen?”
Lots to break down in this article other than this initial quotation, but I find a lot of parallels in failing software projects, this attitude, and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.
It was a combination of failures like this. Why was the captain going full speed ahead into a known ice field? Well, the boat can't sink and there (may have been) organizational pressure to arrive at a certain time in new york (aka, imaginary deadline must be met). Why wasn't there enough life jackets and boats for crew and passengers? Well, the boat can't sink anyway, why worry about something that isn't going to happen? Why train crew on how to deploy the life rafts and emergency procedures properly? Same reason. Why didn't the SS Californian rescue the ship? Well, the 3rd party Titanic telegraph operators had immense pressure to send telegrams to NY, and the chatter about the ice field got on their nerves and they mostly ignored it (misaligned priorities). If even a little caution and forward thinking was used, the death toll would have been drastically lower if not nearly nonexistent. It took 2 hours to sink, which is plenty of time to evacuate a boat of that size.
Same with software projects - they often fail over a period of multiple years and if you go back and look at how they went wrong, there often are numerous points and decisions made that could have reversed course, yet, often the opposite happens - management digs in even more. Project timelines are optimistic to the point of delusion and don't build in failure/setbacks into schedules or roadmaps at all. I've had to rescue one of these projects several years ago and it took a toll on me I'm pretty sure I carry to this day, I'm wildly cynical of "project management" as it relates to IT/devops.
> and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.
But the rest of your comment reveals nothing novel other than anyone would find after watching James Cameron's movie multiple times.
I suggest you go to the original inquiries (congressional in the US, Board of trade in the UK). There is a wealth of subtle lessons there.
Hint: Look at the Admiralty Manual of Seamanship that was current at that time and their recommendations when faced with an iceberg.
Hint: Look at the Board of Trade (UK) experiments with the turning behaviour of the sister ship. In particular of interest is the engine layout of the Titanic and the attempt by the crew, inexperienced with the ship, to avoid the iceberg. This was critical to the outcome.
Hint: Look at the behaviour of Captain Rostron. Lots of lessons there.
Thanks for your feedback, I’m well aware of the inquiries and the history there. However, this post was meant to be a simple analogy that related to the broader topic, not a deep dive into the theories of how and why the titanic sank. Thanks!
Got it. Thanks.
The lesson from “big software projects are still failing” isn’t that we need better methodologies, better project management, or stricter controls. The lesson is "don't do big software projects".
Software is not the same as building in the physical world where we get economies of scale.
Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)
Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software.
Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale.
Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process.
Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared.
It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself.
(Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally).
We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix.
The lesson is: "never build big software".
(Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)
I agree with you on "don't do big software project" Specially do not fast scale them out to hundreds of people. You have to scale them more organically ensuring that every person added is a net gain. They think that adding more people will reduce the time.
I am surprised on the lack of creativity when doing these projects. Why don't they start 5 small projects building the same thing and let them work for a year. At the end of the year you cancel one of the projects, increasing the funding in the other four. You can do that every year based on the results. It may look like a waste but it will significantly increase your chances of succeeding.
>Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)
Build 1000 JSON parsers and tell me if the next one isn't cheaper to develop with "(we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.)"
>Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics)
Uh, maybe fewer but none is way to far. Get 2 billion integer operations per second out of a 286, the 500 mile email, big data storage, etc. Physical limits are everywhere.
>It's also the same reason why enterprise software sucks.
The reason enterprise software sucks is because the lack of introspection and learning from the garbage that went before.
You have to be able to turn away unsuitable customers.
Working on AI that helps to manage IT shops that learns from failure & success might be better for both results and culture than most IT management roles, a profession (painting an absurdly broad brush) that tends to attract a lot of miserable creatures.
... If this happens, the next hacks will be context poisoning. A whole cottage industry will pop around preserving and restoring context.
Sounds miserable.
Also, LLMs don't learn. :)
LLMs themselves don’t learn but AI systems based around LLMs can absolutely learn! Not on their own but as part of a broader system: RLHF leveraging LoRAs that get re-incorporated as model fine tunings regularly, natural language processing for context aggregation, creative use of context retrieval with embeddings databases updated in real time, etc.
A slightly different take, its probably more of people failure, the lack of required expertise, skillset, motivation and coordination. People have motivations to do the job to make a living, success of any long term project is rarely the driving factor for most people working on it. People would know ahead of time when a project is going towards the direction of failure, its just how the things are structured. From systems perspective, an unknown system/requirement would be a good example where you build iteratively, a known set of requirements should give good enough idea about the feasibility and rough timelines even if its complex.
I like that the author propagates software developer liability. That makes sense. Unless we introduce such system, the incentives are not there to avoid failure
https://queue.acm.org/detail.cfm?id=3489045
https://therecord.media/cybersecurity-software-liability-sta...
While I don't disagree, it would increase the cost of software by an extraordinary amount.
"big software project"
there you are why its failling, the fact that these system is overly massive and complex that sometimes the original creator and architecture that design this system cant foresee the future needs and etc
You could says its incompetence but the fact that software change so much is last 20 years make it most people cant really design a "future proof" system in a way that it cant cause trouble in the future
It's possible that most business projects fail.
Most advertising campaigns fail.
Software was failing and mismanaged.
So we added a language and cultural barrier, 12 hour offset, and thousands of miles of separation with outsourcing.
Software was failing and mismanaged.
So now we will take the above failures, and now tack on an AI "prompt engineering" barrier (done by the above outsourced labor).
And on top of that, all engineers that know what they are doing are devalued from the market, all the newer engineers will be AI braindead.
Everything will be fixed!
>By then, the general public and the branch managers themselves finally joined Computer Weekly’s reporters (who had doggedly reported on Horizon’s problems since 2008) in the knowledge that there was something seriously wrong with Horizon’s software.
Computer Weekly first broke the story, for which they deserve much credit. But I believe Private Eye did much of the long term campaigning.
It's good that the author makes the distinction between developers and managers. This distinction is rarely made and most media outlets talk about the wrongdoings of developers, who are almost never the decision makers of failing projects. It's quite the opposite, they are the ones who if brave enough criticize bad management practices and the lack of proper management of the software project.
The most mind boggling number in this article to me was PeopleSoft claiming it would cost $500 million to make a payroll system for the Canadian government. That’s thousands of working years of software developers. It’s such a huge scale that it seems pretty clear the project should never start. PeopleSoft should have been dumped and the project’s scope massively reevaluated.
The main problem are incentives and risks: in most of the cases you are not incentivized to build secure and reliability SW because, most of the time, it's easy to fix it. With particular categories of SW(eg. one distributed on remote system, medical sw, military sw) or HW it's the opposite: a failure it's not so easy to fix so you are incentivized to do a better job.
The second problem are big con.
Failure typically comes from two directions. Unknown and changing requirements, and management that relies on (often external) technical (engineering) leadership that is too often incompetent.
These projects are often characterized by very complex functional requirements, yet are undertaken by those who primarily only know (and endlessly argue about) non-functional requirements.
Every improvement will be moderated increased demands from management, crunch, pressure to release, "good enough", add this extra library that monetizes/spys on the customer etc
In the same way that hardware improvements are quickly gobbled up by more demanding software.
The people doing the programming will also be more removed technically. I can do Python, Java , Kotlin. I can do a little C++ ,less C, and a lot less assembly.
will be moderated by* increased demands.
An endless succession of new tools, methodologies, and roles but failure persists because success is rooted in good judgment, wisdom, and common sense.
In my very humble opinion, the impact that software has on our lives is getting to the point where software engineering should become a true profession like the other engineering branches (electrical, mechanical, etc).
There should be things like professional certifications that engineers have to maintain through continuous education, a professional code of ethics, a board of review, and other functions.
My reasoning is that we are at the point where a software "engineer" can make a mistake that can have the same impact as a civil engineer making a bad calculation and causing a bridge collapse.
There's different levels to this, of course. An app for booking restaurant reservations wouldn't need that much oversight. But we've seen some outages having massive impacts that quite frankly did not happy twenty years ago.
This has dot-com bubble written all over it. But there are some deeper issues.
First, we as a society should really be scrutinizing what we invest in. Trillions of dollars could end homelessness as a rounding error.
Second, real people are going to be punished for this as the layoffs go into overdrive, people lose their houses and people struggle to have enough to eat.
Third, the ultimate goal of all this investment is to displace people from the labor pool. People are annoying. They demand things like fair pay, safe working conditions and sick leave.
Who will buy the results of all this AI if there’s no one left with a job?
Lastly, the externalities of all this investment are indefensible. For example, air and water pollution and rising utility prices.
We’re bouldering towards a future with a few thousand wealthy people where everyone else lives in worker housing, owns nothing and is the next incarnation of brick kiln workers on wealthy estates.
Systemically, how would you solve homelessness, if I gave you a trillion dollars?
A trillion in a money market fund @ 5% is 50B/year.
Over the course of a few years (so as to not drive up the price of politicians too quickly) one could buy the top N politicians from most countries. From there on out your options are many.
After a decade or so you can probably have your trillion back.
I really do like this answer, but it would seem to have the property of being anti-inductive in that the cost for current politicians is so low because nobody is doing it at scale but if someone did that would force other people to ... well, it's an interesting thought experiment at least!
The article isn't really about AI (for a change).
So I haven't looked through the comments, and assume this has been discussed, but the simple solution is to limit contracts to, say, $4M, and pay only on successful completion. Then build a large project through a series of smaller steps.
Is it a failure if we ship the project a year late? What if everyone involved would have predicted exactly that outcome
Hot take: It's not technical problems causing these projects to fail.
It's leadership and accountability (well, the lack of them).
And that often takes a particular form: The requirements never converge, or at least never converge on anything realistically buildable.
I spent way less - and they still fail!
AI will absolutely solve these problems, by inventing nimble AI native companies that disrupt these business models into the Stone Age, worker by worker, role by role. Death by a billion cuts.
Completely off topic but when fonts are the size they are in this article I can't read it, the words don't register as words above a certain size. I assume this isn't normal or it wouldn't be so common...
Worth a view also. Is software engineering still an oxymoron?
https://youtu.be/D43PlUr1x_E?si=em2nNYuI8WDvtP21
How much money do you need to build a skyscraper on top of a tarpit? None because it’s not possible. The whole stack has to be gutted. I can do it but no one wants to listen so I’ll do it myself.
From consulting point of view, a common joke we use to tell, because customers demand a Ferrari, but are only willing to pay for the development costs of a Fiat.
>not only are IT projects risky, they are the riskiest from a cost perspective.
Nuclear reactor projects seem to be regularly delivered massively late and over budget.
Nuclear power plants usually only cost about twice as much as projected in phase II planning. IT projects are sort of open-ended. Interestingly, the simulator I was involved with (many decades ago) at a nuclear power plant came within about 10% of initial projections. The last "scan uploads for viruses" project I worked on was about 20x - 40x more expensive than projected. (Unfortunately the person who suggested we just pay a third party for this service was fired.) The bit with projecting cost and schedules for nuke plants is to ignore any initial costing and multiply the phase II planning estimate by 2 or 4.
>20x - 40x more expensive than projected
That is an impressive cost overrun!
I guess the issue with nuclear is that they are so expensive that even a x2 overrun is disastrous.
It's not as disasterous as you think. Most people paying for power plants expect large overruns. If you're the type of company that can sell bonds to get 5.5 billion dollars to build a nuclear power plant, you can sell bonds to get 11 billion dollars to build a nuclear power plant.
Almost nobody who works in software development is a licensed professional engineer. Many are even self-taught, and that includes both ICs and managers. I'm not saying this is direct causation but I do think it odd that we are so utterly dependent on software for so many critical things and yet we basically YOLO its development compared to what we expect of the people who design our bridges, our chemicals, our airplanes, etc.
Licensing and the perceived rigor it signifies is irrelevant to whether something can be considered "professional engineering." Engineering exists at the intersection of applied science, business and economics. So most software projects can be YOLO'd simply because the economics permit it, but there are others where the high costs necessitate more rigor.
For instance, software in safety-critical systems is highly rigorously developed. However that level of investment does not make sense for run-of-the-mill internal LOB CRUD apps which constitute the vast majority of the dark matter of the software universe.
Software engineering is also nothing special when it comes to various failure modes, because you'll find similar examples in other engineering disciplines.
I commented about this at length a few days ago: https://news.ycombinator.com/item?id=45849304
Change your idea of success to being "Propping up the consultant market", and by that definition these projects are smashing it out of the park.
No big surprise. Taking a shitty process and "digitalizing" it will lead to a shitty process just on computers in the best case, in the worst case everything collapses.
https://en.wikipedia.org/wiki/Productivity_paradox
managing software requirements and the corresponding changes to user/group/process behaviors is by far the hardest part of software development, and it is a task no one knows how to scale.
absent understanding, large companies engage in cargo cult behaviors: they create a sensible org chart, produce a gannt chart, have the coders start whacking code, presumably in 9 months a baby comes out.
every time, ugly baby
What a joke blaming the IT community for not doing better, when most businesses refuse to look past anything but shipping features as fast as they can. "We take security and reliability very seriously", until management gets wind of the price tag. Guess what areas always get considered last and cut first. We all know.
But sure, blame the community at large, not the execs who make bad decisions in the name of short-term profits, then fail upward with golden parachutes into their next gig.
And definitely don't blame government for not punishing egregious behavior of corporations. Don't worry, you get a year of free credit monitoring from some scummy company who's also selling your data. Oh and justice was served to the offending corp, who got a one-time fine of $300k, when they make billions every quarter.
Maybe if we just outsource everything to AI, consultants, and offshore sweat shops things will improve!
Okay cool, good article.
> blaming the IT community for not doing better, when most businesses refuse to look past anything but shipping features
IT != software engineering. IT is a business function that manages a company's internal information. Software engineering is a time-tested process of building software.
A lot of projects fail because management thinks that IT is a software engineering department. It is not. It never was, and it never will be. Its incentives will never be aligned such that software engineering projects are set up for success.
The success rate of implementing software outside of IT and dragging them along later is much higher than implementing it through IT from the beginning.
I understand, but also, IT is an umbrella term for a wider industry that includes your definition of IT, software, and anything adjacent. If you read the article, you'll see it's the latter being referenced, and why I chose that terminology.
> The success rate of implementing software outside of IT and dragging them along later is much higher than implementing it through IT from the beginning.
That's a pretty strong statement. Isn't that the opposite of why the devops movement started?
There is no such thing as ‘simplicity science’ that can be directly applied when dealing with IT problems. However, many insights of complexity science are applicable to solving real world IT problems. People love simple solutions. However Simple is a scam, https://nocomplexity.com/simple-is-a-scam/
There are no generic, simple solutions for complex IT challenges. But there are ground rules for finding and implementing simple solutions. I have created a playbook to prevent IT diasasters, The art and science towards simpler IT solutions see https://nocomplexity.com/documents/reports/SimplifyIT.pdf
Interesting article ... with the wisdom of software engineering being forgotten, it will unfortunately get worse ...
Projects don’t fail, people do. If projects fail it means the wrong people are hired for them.
The concerning aspect of all of this isn't the financial cost of these blunders, and what happened in the past. It is the increasing risk to human lives, and what will happen in the future. The Boeing case was only a sign of what's to come.
Take "AI", for instance. It is being adopted left and right as if it's the solution to all of our problems, and developers, managers, and executives are increasingly relying on it. Companies and governments love it because it can cut costs and potentially make us more productive. Most people are more than happy to offload their work to it, do a cursory check of its output, if at all, and ship it or publish it and claim the work as their own. After all, it can always serve as a scapegoat if things do go wrong, and its manufacturers can brush these off as user errors. Ultimately there is no accountability.
These are all components of a recipe for greater disasters. As these tools are adopted in industries where safety is paramount, in the military, etc., it's only a matter of time for more human lives to be impacted. Especially now when more egomaniacal autocrats are taking power, and surrounding themselves with yes-people. Losing face and admitting failure is not part of their playbook. We're digging ourselves into a hole we might not be able to get out of.
because most people are incompetent, produce incidental complexity to satisfy internal urge for busy work, and under-think the problem, greatly... that's why, and don't get me started on the morons who run the show
'Managers' aren't really getting any better as time goes on...
The article is kind of dumb. eg it hangs its hat on the Phoenix payroll system, which
> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. It also was attempting to implement 34 human-resource system interfaces across 101 government agencies and departments required for sharing employee data.
So basically people -- none of them in IT, but rather working for the government -- built something extraordinarily complex (80k rules!), and then are like wow, it's unforeseen that would make anything downstream at least equally as complex. And then the article blames IT in general. When this data point tells us that replacing a business process that used to require (per [1]) 2,000 pay advisors to perform will be complex. While working in an organization that has shit the bed so thoroughly that paying its employees requires 2k people. For an organization of 290k, so 0.6% of headcount is spent on paying employees!
IT is complex, but incompetent people and incompetent orgs do not magically become competent when undertaking IT projects.
Also too, making extraordinarily complex things they shouting the word "computer" at them like you're playing D&D and it's a spell does not make them simple.
[1] https://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_0...
To stop failing we could use AI to replace managers not software developers.
No need to waste GPUs, a simple bash script that alternates between asking for status updates and randomly changing requirements would do
We are seeking improvements, not the status quo.
Slightly related but unpopular opinion I have: I think software, broadly, today is the highest quality its ever been. People love to hate on some specific issues concerning how the Windows file explorer takes 900ms to open instead of 150ms, or how sometimes an iOS 26 liquid glass animation is a bit janky... we're complaining about so much minutia instead of seeing the whole forest.
I trust my phone to work so much that it is now the single, non-redundant source for keys to my apartment, keys to my car, and payment method. Phones could only even hope to do all of these things as of like ~4 years ago, and only as of ~this year do I feel confident enough to not even carry redundancies. My phone has never breached that trust so critically that I feel I need to.
Of course, this article talks about new software projects. And I think the truth and reason of the matter lies in this asymmetry: Android/iOS are not new. Giving an engineering team agency and a well-defined mandate that spans a long period of time oftentimes produces fantastic software. If that mandate often changes; or if it is unclear in the first place; or if there are middlemen stakeholders involved; you run the risk of things turning sideways. The failure of large software systems is, rarely, an engineering problem.
But, of course, it sometimes is. It took us ~30-40 years of abstraction/foundation building to get to the pretty darn good software we have today. It'll take another 30-40 years to add one or two more nines of reliability. And that's ok; I think we're trending in the right direction, and we're learning. Unless we start getting AI involved; then it might take 50-60 years :)
Why don’t i hear news of such failures from India and China?
There are great big software projects and shitty ones. IRCTC, UPI being examples of great ones.
Insurance and RTO being shitty ones.
I had an insurance deadline very near and the payment was not showing up in the insurance providers dashboard so had to do it twice and now it was still not showing up.
Also I have faced huge problems with getting the learner's licence online.
Do let me know if you've faced anything same.
I got my name wrong in the drivers card and never went to correct it. However most of the problems the problems there were administrative not software. I agree both irctc and upi come to mind first as the successes. Insurance, could be a particular company as i never faced such problem. Websites for Tax filing and even starting an msme has been smooth.
Fujitsu has never been served justice for Horizon.
Kind of strange take as though unique to software. Every sector that is large has issues since ambitious projects stretch what can be done by the current management and organizational practices. All software articles like these hark back to some mythical world smaller in scope/ambition/requirements. Humanity moves forward
* Construction and Engineering -- Massive cost overruns and schedule delays on large infrastructure projects (e.g., public transit systems, bridges)
* Military and Government -- Defense acquisition programs notorious for massive cost increases and years-long delays, where complex requirements and bureaucratic processes create an environment ripe for failure.
* Healthcare -- Hospital system implementations or large research projects that exceed budgets and fail to deliver intended efficiencies, often due to resistance to change and poor executive oversight.
Throwing money at a problem never works and never will!
> IT projects suffer from enough management hallucinations and delusions without AI adding to them.
Software is also incredibly hard, the human mind can understand the physical space very well but once we're deep into abstractions it simply struggles to keep up with it.
It is easier to explain how to build a house from scratch to virtually anyone than a mobile app/Excel.
I came to opposite conclusions. Technology is pretty easy, people are hard and the business culture we have fostered in the last 40 years gets in the way of success.
Easy, just imagine a 1GB array as a 2.5mm long square in RAM (assuming a DRAM cell is 10nm). Now it's physical.
> We are left with only a professional and personal obligation to reemphasize the obvious: Ask what you do know, what you should know, and how big the gap is between them before embarking on creating an IT system. If no one else has ever successfully built your system with the schedule, budget, and functionality you asked for, please explain why your organization thinks it can
translation: "leave it to us professionals". Gate-keeping of this kind is exactly how computer science (the one remaining technical discipline still making reliable progress) could become like all of the other anemic, cursed fields of engineering. people thinking "hey im pretty sure I could make a better version of this" and then actually doing it is exactly how progress happens. I hope nobody reads this article and takes it seriously
People concerned about small benefits from AI should consider that IT and the internet is failing upwards for more than three decades now.
The purpose of a system is what it does.
1. Enable grift to cronies
2. Promo-driven culture
3. Resume-oriented software architecture
There are 2 big problems with large software projects:
1. Connecting pay to work - estimates (replanning is learning, not failure)
2. Connecting work to pay - management (the world is fractal-like, scar tissue and band-aids)
I do not pre-suppose that there are definite solutions to these problems - there may be solutions, but getting there may require going far out of our way. As the old farmer said "Oh, I can tell you how to get there, but if I was you, I wouldn't start from here"
1. Pay to Work - someone is paying for the software project, and they need to know how much it will cost. Thus estimates are asked for, an architecture is asked for, and the architecture is tied to the estimates.
This is 'The Plan!'. The project administrators will pick some lifecycle paradigm to tie the architecture to the cost estimate.
The implementation team will learn as they do their work. This learning is often viewed as failure, as the team will try things that don't work.
The implementation team will learn that the architecture needs to change in some large ways and many small ways. The smallest changes are absorbed in regular work. Medium and Large changes will require more time (thus money); This request for more money will be viewed as a failure in estimation and not as learning.
2. Work to Pay - as the architecture is implemented, development tasks are completed. The Money People want Numbers, because Money People understand how they feel about Numbers. Also these Numbers will talk to other Numbers outside the company. Important Numbers with names like Share Price.
Thus many layers of management are chartered and instituted. The lowest layer of management is the self-managed software developer. The software developer will complete development tasks related to the architecture, tied to the plan, attached to the money (and the spreadsheets grew all around, all around [0]).
When the developer communicates about work, the Management Chain cares to hear about Numbers, but sometimes they must also involve themselves in failures.
It is bad to fail, especially repeated failures at the same kind of task. So managers institute rules to prevent failures. These rules are put in a virtual cabinet, or bureau. Thus we have Rules of the Bureau or Bureaucracy. These rules are not morally bad or good; not factually incorrect or correct, but whenever we notice them, they feel bad; We notice the ones that feel bad TO US. We are often in favor of rules that feel bad to SOMEONE ELSE. You are free to opt out of this system, but there is a price to doing so.
----
Too much writing, I desist from decoding verbiage:
Thus it is OK for individuals to learn many small things, but it is a failure for the organization to learn large things. Trying to avoid and prevent failure is viewed as admirable; trying to avoid learning is self-defeating.
----
0. https://www.google.com/search?q=the+green+grass+grew+all+aro...
> git commit -am "decomposing recapitulating and recontextualizing software development bureaucracy" && git push
Bureaucracy is: scar tissue, someone else's moat, someone else's data model
AI will fix this
[dead]
[dead]
[dead]
People are _so amazingly close_ to realizing what is wrong with this entire industry. So close.
Enlighten us.
This is a direct result of using leetcode in interviews instead of any other, more legitimate tests like winning a tekken 1v1. Have you ever seen a good developer who’s not good at real video games?
If companies had hired real developers instead of cosplayers who are stunlocked with imposter syndrome as the only candidate pool with time to memorize a rainbow table of arbitrary game trivia questions and answers, things would actually work.
The biggest reason is developer ego. Devs see their code as artwork an extension of themselves, so it's really hard to have critical conversations about small things and they erupt into holy wars. Off hand:
* Formatting
* Style
* Conventions
* Patterns
* Using the latest frameworks or whats en-vogue
I think where I've seen results delivered effectively and consistently is where there is a universal style enforced, which removes the individualism from the codebase. Some devs will not thrive in that environment, but instead it makes the code a means-to-the-end, rather than being-the-end.
As far as I can see in the modern tech industry landscape, virtually everyone has adopted style guides and automatic formatting/linting. Modern languages like Go even bake those decisions into the language itself.
I'd consider managing that stuff essentially table-stakes in big orgs these days. It doesn't stop projects from failing in highly expensive and visible ways.
> in the modern tech industry landscape, virtually everyone has adopted style guides and automatic formatting/linting
the modern tech industry landscape is in absolute terms is small compared to the wider tech industry landscape afaik
The UK Post Office lied and made people kill themselves ... because of dev ego?
To me it screams more like an organization not wanting to assume blame and risk paying for their errors.
Ironically, the downvotes pretty much prove this is exactly correct.
Eh, you're not wrong, but management failures tend to be a bigger issue. On the hierarchy of ways software projects fail, developer ego is kind of upper-middle of the pack rather than top. Delusional, ignorant, or sadistic leadership tends to be higher.