I'm pretty certain that the one factor above all others that makes software delivery slow down is doing too many things at once. You should aim to be working on one project at a time in your team.
Taking on another project, or workstream, or idea, or investigation in a team is pretty much entirely downside.
- more context switching, which means people get cranky faster
- lower bus factor for any piece of work, until its 1 dev per thing and that things stops of the dev isn't there
- lower people count on work means everything takes how long 1 person can do it (2 people is twice as fast for parallizable work)
- slower dependency resolution until something as basic as getting a PR reviewed is a big deal
- fewer people seeing the problem means less experience brought to bear, which reduces speed gains from having seen this stuff before
And more besides. For every project the goal should be maximum parallelism.
I've never really understood this opinion. For me there's plenty of times that some problem needs to be solved and simply banging my head against it isn't going to make it happen any faster. Having some easy low priority stuff to do for a bit (or a high priority short task that interrupts) seems to let my subconscious solve the issue in the background and keeps me happier and more motivated.
If it was two difficult tasks that were supposed to be done in parallel maybe that would be different, but I've never seen anyone asking for that
> low priority stuff to do for a bit (or a high priority short task that interrupts) seems to let my subconscious solve the issue in the background and keeps me happier and more motivated.
For a bit is ok. I had one month task, that was interrupted daily for over a year, that's how I got demotivated and burned out so much that I needed neurologist.
> two difficult tasks that were supposed to be done in parallel maybe that would be different, but I've never seen anyone asking for that.
Probably that's why you say interruptions are good, no experience of how bad can interruptions be.
But if that person has rare skills where at least 25% of their output makes another team of 5 100% more effective (200%), than than loss of 50% output is actually a 33% gain in output!
Say you have a team of 6 (5+1 DevOps Wizard), and a team of 5. so Call it 5x200 +5 x100% output, 1500 output total. Split our wizard in half and you get 5x200 + 5x200 = 2000 output total!
At a certain minimum level the effectiveness inverts because people are waiting on wizard magic and everyone is less effective, but there is benefit to having specialists be split.
> You should aim to be working on one project at a time in your team.
You're swimming against the tide with this. AI pilled management is driving teams to deliver more parallel workstreams, not fewer, with fewer, AI-augmented devs taking them on.
Context switching isn't seen as a cost. AI is seen as the solution to the bus factor and experience problem. Same goes with the issue of dev capacity.
IMO it's deeply misguided but it's crystal clear where the winds are blowing.
It's funny, because LLMs have context switching costs too! Cached tokens cost less than non-cached. There's no reason to think that doesn't apply to humans.
I’ve genuinely found that context switching is a skill like any other. When I first started using LLMs and context switching doing 5-10 different work streams at once I was utterly, utterly exhausted at the end of each day.
Now many months in, it doesn’t feel nearly as taxing.
Maybe this is in no small part due to LLMs improving enough you don’t have to spend as much mental energy reviewing their output and them being better at managing worktrees and other tedium, but the idea of doing lots of things at once feels like a mental muscle that was previously weak and has gained in strength in ways I never though possible for me.
5-10 feels like a lot. I can reliably maintain hands on 2-4 separate work streams. I still need to maintain a pretty close eye on changes to our system, as I’m one of 2 devs. I’ve tried doing 5+ things at once and I just cannot get it to work. I think I move faster when I’m more involved anyhow, since I have adequate “mini-context” of what each operation is.
Our org has a really nicely set up review bot which has access to a repo of really strict standards to check new code against. It means that the reviews are much more useful and catch a lot of things that a plain bot wouldn’t. I think that’s a huge factor in being able to trust the reviews to catch more. You can also check the repo out locally.
And this is why software consultancies always recommend a “Digital Transformation Project” to hire as many remote people as possible, all being paid peanuts and charged out at $1500+ per day, the added bonus being the project will take an infinite amount of time!
I struggle to believe people above me where I work refer to these leaches as “our partners” it seems mad to me.
One project at a time? How on earth can that possibly work? I always ask to have multiple projects to work on, because getting blocked is a real thing, and having other work you can still pursue will ensure you keep things moving! Perhaps my definition of project is very different from yours?
I am friendly with a project manager at work - he's overseeing a fairly big modernisation project with lots of stakeholders, lots of moving parts, lots of direct reports, and while he seems to have a lot of long lunches, he makes sure that no one working on the project is blocked - he'll do everything he needs to to unblock people - regular meetings to hear about blockers, chase down approvers, just tell his direct reports to do something without approval (as long as it's not something bad) and he'll deal with the consequences. But most people are not this dedicated unfortunately.
Single Piece Flow fits with other Toyota ideas of how to get high quality work done. If something is frequently blocking the line, the solution is to put resources toward the bottleneck, not spin up more work in parallel.
Building software isn't the same as building cars. Some ideas needs months or years to be realized, and there's only so fast you can realistically move.
Projects can be broken down into parallel pieces of work, so you should always be able to pick up another part of the project rather than switch to a different project if you're blocked. As the other poster says, the goal is to not be blocked though. The leadership team need to support the delivery team to get the work done as a focused flow of items with as little blocking and waiting time as possible. If you're getting blocked, that's a signal that the org is failing.
Admittedly this is a little idealistic, but if it's not the goal then the reality is always far worse.
For us, review latency was obviously a problem so we eliminated it (for most tickets, anyway) by making the handover synchronous. Get on a call with the reviewer when it goes into the review column. The win from eliminating review latency massively outweighs the context switch cost for the reviewer. All the more so when the patch is small enough to actually read through on the call, so if there is anything immediate to address you can ping-pong it without anything waiting.
Over the last 16 years all I've seen is people implying (heavily) that they can replace QA with "good enough unit tests"..
Which then inevitably leads to bugs, and performance regressions and developers doing their own QA.
I honestly think that all of this is a psyop to get devs to do more, that and "devops" which is either rebranded sysadmins, or developers working as yaml monkeys between code reviews and feature development.
Yeah I've seen both approaches. Either teams do their own QA or we have a dedicated function. Depends on the domain to some extent. Both can work. But I don't hear people saying stupid shit anymore like "QA is a cost center". When we have it, everyone understands the value.
I worked in a team with no QAs for a while - testing was done by developers and there were a lot end to end tests. It worked pretty well - I think a couple of small bugs got through, but whether a QA would have found the issues is debatable. It was for an internal system too so the cost of a defect was low anyway. I've also worked a company with no QAs in some pretty core teams and they had almost no defects either.
Currently working on some legacy stuff with no proper end to end tests and I'm very grateful for competent testers right now.
In my experience, things head south when managers see “feature X requires 4 person-months effort” and then assign 4 people to work on it, with 2-3 being junior, and the remaining multiplexed with other tasks. They think this will allow them to meet their goal of being feature complete in a month.
There ends up being no shared vision of how it all works. Disparate pieces are developed and some poor sap ends up gluing them together in a painful manner, 2-3 months later. QA finds some edge cases, driving last minute fixes and delaying release.
Ultimately more difficult bugs escaped QA and go on to result in customer escalations.
Eventually, the feature gets cleaned up a year or two later, after another month or two of development effort.
Customers just see a broken feature for 2-3 years and give up on it.
So. Much. Waste.
It’s not about having a Chief Navel-Gazer Architect. Development on a shared code base is just not as parallelizable as one would desire.
One has to thoroughly understand exactly what they are doing and know all the relevant considerations. It isn’t practical for a group to all be capable of such for every feature. And nobody is going to take the time to write a detailed internal API spec that would even allow such to be possible…
> "When coupling getting higher, considerable amount of time is wasted in heavy coordination, alignment, and shared strategy. The other side of the same coin, considerable amount of time is lost when one component breaks for everyone"
Is this slop, or poor english skills? Is there any peer review barrier to being published here? Because this kind of looks like a low-effort 4-page survey of other people's data with no original empirical evidence. In other words, probably should've been a blog.
It's an article in an official publication. It's like a "letter to the editor" of a scientific journal, and has always been published alongside the more fact based research reports. People were posting "blog posts" in journals long before "blogs" were a thing. Part of the idea of a journal is exactly, like a newspaper, to exercise editorial control over what sorts of discussions they decide to include in each issue.
Science isn't just cold reports on aimless facts. You're supposed to discuss with your peers.
> Organizations needs to prioritize measuring for these inefficiencies that lowers the quality bar, churn software engineers, and slow down delivery process. Once measured, organizations should add mechanism, both technical and procedural to limit the waste
Organizations can't even track and improve on sprint performance. They aren't gonna track and improve inefficiencies either, because they have no motivation to improve. Even if they're losing money and customers due to their product sucking, they still won't see those inefficiencies as the cause of their problems. They'll instead imagine that what they really need is to push more sales/increase initial subscriptions, or add a killer new feature they have no time to implement, or pivot to a different business model. But their lack of rigor is only partly due to executive stupidity; the other part is mediocrity. If you have lazy, ignorant devs, you're gonna have lazy, ignorant process, leading to all the waste.
It's hard to believe something when I saw the opposite work in practice. Removing QA makes developers care about production. You can say now they are wasting part of their time doing what QA did but to me it's not wasted time. Incidents went down both times I saw it happen, and stayed down.
> Removing QA makes developers care about production
And removing Product Managers makes devs care about the product. Therefore, we should strip away all roles apart from devs..?
These roles exist because the skill set is sufficiently large to justify a dedicated role. If you're a startup, sure - maybe it's better to be lean - but in a larger org, QA folks are worth the investment and can definitely help to spot things that devs may not otherwise catch themselves since they know the product intimately and likely learn of common pitfalls. The few QA folks I've worked with have been worth their weight in gold.
Just to offer a different perspective, my last job removed QA and it was a nightmare. Feature delivery slowed way down and incidents and bugs went way up to the point where we started losing customers.
In theory, I agree with what you're saying, but I think a codebase needs to be in a certain state and a company's leadership needs to be in a certain mindset to make it work. Our codebase was far too large and interconnected for any developer to fully understand all of it because of years of building with unclear boundaries between teams. QA's value wasn't really the testing they did. The value was that we had a dedicated team who's job it was to see the big picture and keep up with the changes 15 different teams were making on the same product. Management wanted the benefits of removing QA without the pain of paying down all the organizational and tech debt that made them valuable.
Again, I agree with your sentiment, but also, I think that most large organizations are not willing to do the necessary work to get a place where QA can be removed.
Sounds just like every regular software project. That’s why cutting QA is a bad idea for every regular software project. Same goes for product management, unless you are Google and can afford building products people don’t need.
We’ve never had a QA because we are on a tight budget but our PM is acting as a gatekeeper before declaring something as done.
However, there were too many cases that things were delivered as “ready” when they were not. PR review was not enough as everyone were checking diffs from GitHub only.
I implemented an intermediate step where the responsible engineer must post a screenshot or a short video of the ticket before passing to the PM and that alone resolved a lot of issues before they materialized on customers screens, just because you must take the extra effort to work on a proof of work.
People are good at different things. Usually they like the things they are good at and vice versa. Different tasks require different mind sets and dev, QA and ops are vastly different in that regard.
That is why we have assembly line production in basically every industry except software.
Writing automated end-to-end tests? Those are very often best done by those writing the software, with assistance from experts who know how to effectively build test infrastructure. I've very rarely seen them done effectively by QA or "test engineering" folks.
Running manual test scripts and torture testing the UI to break it in new and unusual ways? Definitely get dedicated testing / QA folks. It's very easy to be blind to how others might use the software, especially if you wrote it.
> Writing automated end-to-end tests? Those are very often best done by those writing the software
This so much - but most of the places I've worked at have insisted that manual testers with zero programming experience should build and maintain the test frameworks. Talk about trying to push a square peg into a round hole. Granted some of the testers have embraced it and not done a terrible job but by and large most of the testers have had no idea and done very little.
Building high-quality test infrastructure often requires more skill than building the software it supports. Some of the most impactful work I've done in staff/principal roles has been test infra.
Yep, very very true. Unfortunately some of the places I've been at more recently have believed that mock servers are the best way to do end to end testing. It was a battle I couldn't win, but no one was willing to see that testing critical UI flows against something that spits back canned JSON could hardly be considered thorough testing.
I'm pretty certain that the one factor above all others that makes software delivery slow down is doing too many things at once. You should aim to be working on one project at a time in your team.
Taking on another project, or workstream, or idea, or investigation in a team is pretty much entirely downside.
- more context switching, which means people get cranky faster
- lower bus factor for any piece of work, until its 1 dev per thing and that things stops of the dev isn't there
- lower people count on work means everything takes how long 1 person can do it (2 people is twice as fast for parallizable work)
- slower dependency resolution until something as basic as getting a PR reviewed is a big deal
- fewer people seeing the problem means less experience brought to bear, which reduces speed gains from having seen this stuff before
And more besides. For every project the goal should be maximum parallelism.
I've never really understood this opinion. For me there's plenty of times that some problem needs to be solved and simply banging my head against it isn't going to make it happen any faster. Having some easy low priority stuff to do for a bit (or a high priority short task that interrupts) seems to let my subconscious solve the issue in the background and keeps me happier and more motivated. If it was two difficult tasks that were supposed to be done in parallel maybe that would be different, but I've never seen anyone asking for that
> low priority stuff to do for a bit (or a high priority short task that interrupts) seems to let my subconscious solve the issue in the background and keeps me happier and more motivated.
For a bit is ok. I had one month task, that was interrupted daily for over a year, that's how I got demotivated and burned out so much that I needed neurologist.
> two difficult tasks that were supposed to be done in parallel maybe that would be different, but I've never seen anyone asking for that.
Probably that's why you say interruptions are good, no experience of how bad can interruptions be.
My hand wavy math.
One person dedicated to a single project / task baseline 100% total output.
One person split between two tasks / projects that require context switching, 70% total output at best, perhaps 50% total output, a loss of 50%.
One person working on a single task that they’re passionate about / intrinsically motivated to complete: 150-500% output.
When you have one task / project to think about and focus on, it engages your brain: sleeping, showering, walking, your subconscious is processing.
Adding just one extra project / task that requires a context switch mostly kills this subconscious processing mechanism.
But if that person has rare skills where at least 25% of their output makes another team of 5 100% more effective (200%), than than loss of 50% output is actually a 33% gain in output!
Say you have a team of 6 (5+1 DevOps Wizard), and a team of 5. so Call it 5x200 +5 x100% output, 1500 output total. Split our wizard in half and you get 5x200 + 5x200 = 2000 output total!
At a certain minimum level the effectiveness inverts because people are waiting on wizard magic and everyone is less effective, but there is benefit to having specialists be split.
> You should aim to be working on one project at a time in your team.
You're swimming against the tide with this. AI pilled management is driving teams to deliver more parallel workstreams, not fewer, with fewer, AI-augmented devs taking them on.
Context switching isn't seen as a cost. AI is seen as the solution to the bus factor and experience problem. Same goes with the issue of dev capacity.
IMO it's deeply misguided but it's crystal clear where the winds are blowing.
It's funny, because LLMs have context switching costs too! Cached tokens cost less than non-cached. There's no reason to think that doesn't apply to humans.
I’ve genuinely found that context switching is a skill like any other. When I first started using LLMs and context switching doing 5-10 different work streams at once I was utterly, utterly exhausted at the end of each day.
Now many months in, it doesn’t feel nearly as taxing.
Maybe this is in no small part due to LLMs improving enough you don’t have to spend as much mental energy reviewing their output and them being better at managing worktrees and other tedium, but the idea of doing lots of things at once feels like a mental muscle that was previously weak and has gained in strength in ways I never though possible for me.
5-10 feels like a lot. I can reliably maintain hands on 2-4 separate work streams. I still need to maintain a pretty close eye on changes to our system, as I’m one of 2 devs. I’ve tried doing 5+ things at once and I just cannot get it to work. I think I move faster when I’m more involved anyhow, since I have adequate “mini-context” of what each operation is.
Our org has a really nicely set up review bot which has access to a repo of really strict standards to check new code against. It means that the reviews are much more useful and catch a lot of things that a plain bot wouldn’t. I think that’s a huge factor in being able to trust the reviews to catch more. You can also check the repo out locally.
And this is why software consultancies always recommend a “Digital Transformation Project” to hire as many remote people as possible, all being paid peanuts and charged out at $1500+ per day, the added bonus being the project will take an infinite amount of time!
I struggle to believe people above me where I work refer to these leaches as “our partners” it seems mad to me.
One project at a time? How on earth can that possibly work? I always ask to have multiple projects to work on, because getting blocked is a real thing, and having other work you can still pursue will ensure you keep things moving! Perhaps my definition of project is very different from yours?
I am friendly with a project manager at work - he's overseeing a fairly big modernisation project with lots of stakeholders, lots of moving parts, lots of direct reports, and while he seems to have a lot of long lunches, he makes sure that no one working on the project is blocked - he'll do everything he needs to to unblock people - regular meetings to hear about blockers, chase down approvers, just tell his direct reports to do something without approval (as long as it's not something bad) and he'll deal with the consequences. But most people are not this dedicated unfortunately.
Single Piece Flow fits with other Toyota ideas of how to get high quality work done. If something is frequently blocking the line, the solution is to put resources toward the bottleneck, not spin up more work in parallel.
Building software isn't the same as building cars. Some ideas needs months or years to be realized, and there's only so fast you can realistically move.
Projects can be broken down into parallel pieces of work, so you should always be able to pick up another part of the project rather than switch to a different project if you're blocked. As the other poster says, the goal is to not be blocked though. The leadership team need to support the delivery team to get the work done as a focused flow of items with as little blocking and waiting time as possible. If you're getting blocked, that's a signal that the org is failing.
Admittedly this is a little idealistic, but if it's not the goal then the reality is always far worse.
For us, review latency was obviously a problem so we eliminated it (for most tickets, anyway) by making the handover synchronous. Get on a call with the reviewer when it goes into the review column. The win from eliminating review latency massively outweighs the context switch cost for the reviewer. All the more so when the patch is small enough to actually read through on the call, so if there is anything immediate to address you can ping-pong it without anything waiting.
> Get on a call with the reviewer when it goes into the review column.
The context switch cost is simply moved to the reviewer, who presumably was working on something. It’s net the same result.
"Stop blaming QA.."
What year is it?! Does anyone still blame QA?! If so, please consult the calendar and enter the 21st century!
Do people still hire QA?
Over the last 16 years all I've seen is people implying (heavily) that they can replace QA with "good enough unit tests"..
Which then inevitably leads to bugs, and performance regressions and developers doing their own QA.
I honestly think that all of this is a psyop to get devs to do more, that and "devops" which is either rebranded sysadmins, or developers working as yaml monkeys between code reviews and feature development.
Yeah I've seen both approaches. Either teams do their own QA or we have a dedicated function. Depends on the domain to some extent. Both can work. But I don't hear people saying stupid shit anymore like "QA is a cost center". When we have it, everyone understands the value.
I worked in a team with no QAs for a while - testing was done by developers and there were a lot end to end tests. It worked pretty well - I think a couple of small bugs got through, but whether a QA would have found the issues is debatable. It was for an internal system too so the cost of a defect was low anyway. I've also worked a company with no QAs in some pretty core teams and they had almost no defects either.
Currently working on some legacy stuff with no proper end to end tests and I'm very grateful for competent testers right now.
> "good enough unit tests"
Good enough end to end tests then maybe.
In my experience, things head south when managers see “feature X requires 4 person-months effort” and then assign 4 people to work on it, with 2-3 being junior, and the remaining multiplexed with other tasks. They think this will allow them to meet their goal of being feature complete in a month.
There ends up being no shared vision of how it all works. Disparate pieces are developed and some poor sap ends up gluing them together in a painful manner, 2-3 months later. QA finds some edge cases, driving last minute fixes and delaying release.
Ultimately more difficult bugs escaped QA and go on to result in customer escalations.
Eventually, the feature gets cleaned up a year or two later, after another month or two of development effort.
Customers just see a broken feature for 2-3 years and give up on it.
So. Much. Waste.
It’s not about having a Chief Navel-Gazer Architect. Development on a shared code base is just not as parallelizable as one would desire.
One has to thoroughly understand exactly what they are doing and know all the relevant considerations. It isn’t practical for a group to all be capable of such for every feature. And nobody is going to take the time to write a detailed internal API spec that would even allow such to be possible…
Layer on the merge conflicts resulting from this fun scenario...
> "When coupling getting higher, considerable amount of time is wasted in heavy coordination, alignment, and shared strategy. The other side of the same coin, considerable amount of time is lost when one component breaks for everyone"
Is this slop, or poor english skills? Is there any peer review barrier to being published here? Because this kind of looks like a low-effort 4-page survey of other people's data with no original empirical evidence. In other words, probably should've been a blog.
I was quite shocked to witness a watchmojo Top 5 drivers behind software delivery inefficiencies kind of research paper
It's an article in an official publication. It's like a "letter to the editor" of a scientific journal, and has always been published alongside the more fact based research reports. People were posting "blog posts" in journals long before "blogs" were a thing. Part of the idea of a journal is exactly, like a newspaper, to exercise editorial control over what sorts of discussions they decide to include in each issue.
Science isn't just cold reports on aimless facts. You're supposed to discuss with your peers.
Great read. Learned the hard way that we need QA roles.
Great paper. QA's role in software is underrated.
> Organizations needs to prioritize measuring for these inefficiencies that lowers the quality bar, churn software engineers, and slow down delivery process. Once measured, organizations should add mechanism, both technical and procedural to limit the waste
Organizations can't even track and improve on sprint performance. They aren't gonna track and improve inefficiencies either, because they have no motivation to improve. Even if they're losing money and customers due to their product sucking, they still won't see those inefficiencies as the cause of their problems. They'll instead imagine that what they really need is to push more sales/increase initial subscriptions, or add a killer new feature they have no time to implement, or pivot to a different business model. But their lack of rigor is only partly due to executive stupidity; the other part is mediocrity. If you have lazy, ignorant devs, you're gonna have lazy, ignorant process, leading to all the waste.
It's hard to believe something when I saw the opposite work in practice. Removing QA makes developers care about production. You can say now they are wasting part of their time doing what QA did but to me it's not wasted time. Incidents went down both times I saw it happen, and stayed down.
> Removing QA makes developers care about production
And removing Product Managers makes devs care about the product. Therefore, we should strip away all roles apart from devs..?
These roles exist because the skill set is sufficiently large to justify a dedicated role. If you're a startup, sure - maybe it's better to be lean - but in a larger org, QA folks are worth the investment and can definitely help to spot things that devs may not otherwise catch themselves since they know the product intimately and likely learn of common pitfalls. The few QA folks I've worked with have been worth their weight in gold.
Just to offer a different perspective, my last job removed QA and it was a nightmare. Feature delivery slowed way down and incidents and bugs went way up to the point where we started losing customers.
In theory, I agree with what you're saying, but I think a codebase needs to be in a certain state and a company's leadership needs to be in a certain mindset to make it work. Our codebase was far too large and interconnected for any developer to fully understand all of it because of years of building with unclear boundaries between teams. QA's value wasn't really the testing they did. The value was that we had a dedicated team who's job it was to see the big picture and keep up with the changes 15 different teams were making on the same product. Management wanted the benefits of removing QA without the pain of paying down all the organizational and tech debt that made them valuable.
Again, I agree with your sentiment, but also, I think that most large organizations are not willing to do the necessary work to get a place where QA can be removed.
Sounds just like every regular software project. That’s why cutting QA is a bad idea for every regular software project. Same goes for product management, unless you are Google and can afford building products people don’t need.
We’ve never had a QA because we are on a tight budget but our PM is acting as a gatekeeper before declaring something as done.
However, there were too many cases that things were delivered as “ready” when they were not. PR review was not enough as everyone were checking diffs from GitHub only.
I implemented an intermediate step where the responsible engineer must post a screenshot or a short video of the ticket before passing to the PM and that alone resolved a lot of issues before they materialized on customers screens, just because you must take the extra effort to work on a proof of work.
People are good at different things. Usually they like the things they are good at and vice versa. Different tasks require different mind sets and dev, QA and ops are vastly different in that regard. That is why we have assembly line production in basically every industry except software.
I agree. I saw the same.
Hold up... We saw it at the same place :D
Really depends on what you're having QA folks do.
Writing automated end-to-end tests? Those are very often best done by those writing the software, with assistance from experts who know how to effectively build test infrastructure. I've very rarely seen them done effectively by QA or "test engineering" folks.
Running manual test scripts and torture testing the UI to break it in new and unusual ways? Definitely get dedicated testing / QA folks. It's very easy to be blind to how others might use the software, especially if you wrote it.
> Writing automated end-to-end tests? Those are very often best done by those writing the software
This so much - but most of the places I've worked at have insisted that manual testers with zero programming experience should build and maintain the test frameworks. Talk about trying to push a square peg into a round hole. Granted some of the testers have embraced it and not done a terrible job but by and large most of the testers have had no idea and done very little.
Building high-quality test infrastructure often requires more skill than building the software it supports. Some of the most impactful work I've done in staff/principal roles has been test infra.
Yep, very very true. Unfortunately some of the places I've been at more recently have believed that mock servers are the best way to do end to end testing. It was a battle I couldn't win, but no one was willing to see that testing critical UI flows against something that spits back canned JSON could hardly be considered thorough testing.
Entirely dependent on what you're shipping.
If there is a significant amount of user interaction to test, QA is going to do a better job than devs.
If there's a lot of integration behind the scenes with a ton of edge cases, devs will do a better job.
If you have both (probably), you need both teams to pitch in more than upper management wants to believe and they'll just cut you instead! :D