> What code is truly about is precision: code is unambiguous, even when it’s abstract. It’s easy to conflate ambiguity and abstraction—both refer to “a single statement that could have to multiple meanings.” But the meanings of an ambiguous statement are entirely unconstrained.
I used to believe this, but after working at a successful SaaS I have come to believe that correctness and unambiguity are not entirely necessary for successful software products.
It was a very sad realization that systems can be flaky if there is enough support people to solve edge case problems. That features can be delivered while breaking other features as long as enough users don't run into the middle of that venn diagram, etc.
Fact is it always comes down to economics, your software can afford to be as broken and unpredictable as your users will still be willing to pay money for it.
Right. One thing I learned over the years is that you can support arbitrarily high level of tech debt and still be able to effectively maintain and enrich a successful software system, as long as you throw enough warm bodies at it.
The overhead will get absurd, you'll end up with 10x or more increase in engineers working on the system, all of them making slow progress while spending 90% time debugging, researching, or writing docs and holding meetings to work out this week's shared understanding of underlying domain semantics - but progress they will make, system will be kept running, and new features will be added.
If the system is valuable enough for the company, the economic math actually adds up. I've seen at least one such case personally - a system that survived decades and multiple attempts at redoing it from scratch, and keeps going strong, fueled by massive amount of people-hours spent on meetings.
Adding AI to the mix today mostly just shifts individual time balance towards more meetings (Amdahl's Law meets Parkinson's law). But ironically, the existence of such systems, and the points made in the article, actually reinforce the point of AI being key to, if not improving it, then at least keeping this going: it'll help shorten the time to re-establish consensus on current global semantics, and update code at scale to stay consistent.
This comment seems off-topic, but just to exemplify:
In your example even as the interface for those products is unstable (UI that changes all the time, slightly broken API), those products are coded in a language like C++ or Java, which benefit from compiler error checking. The seams where it connects with other systems is where they're unstable. That's the point of this blog post.
This is the realization that pretty much all engineers go through as they become more senior. The areas where the business really can’t afford issues is very small. Usually only accounting / billing and then that’s solved not by great code / design but just audit it once and never touch it again.
In the end most challenges for a business holding them back to better code quality are organizational, not technical.
> In the end most challenges for a business holding them back to better code quality are organizational, not technical.
This is true. And I get sad every time it is used as an argument not to improve tooling. It feels like sort of a self-fulfilling prophecy: an organizational problem that prevents us from investing into technical improvements... is indeed an organizational problem.
Yeah that's true, a product can be successful with truly bad code, but it also makes developers lives miserable each time they need to add a new feature, solve a bug, or simply understand how that entangled mess works.
Management and sales may not appreciate good software design and good code, the next developer that has to work on system will.
Not sure if tools and technologies can solve accidental complexity.
In my opinion, a system that has been stable for years isn't 'mature' in a good sense. An exceptional system is one that can still change after many years in production.
I believe this is almost impossible to achieve for enterprise software, because nobody has incentive to make the (huge) investment into longterm maintainability and changeability.
For me, consistent systematic naming and prefixes/suffixes to make names unique are a hint that a person is thinking about this or has experience with maintaining old systems. This has a huge effect on how well you can search, analyze, find usages, understand, replace, change.
> Not sure if tools and technologies can solve accidental complexity.
... and then say
> For me, consistent systematic naming and prefixes/suffixes to make names unique are a hint that a person is thinking about this or has experience with maintaining old systems. This has a huge effect on how well you can search, analyze, find usages, understand, replace, change.
I have battle scars from refactoring legacy systems where my predecessors did _not_ consistently or uniquely name things and I would not have seen it through without my sidekick, the type checker!
This is light on specifics, but is still directionally the closest anyone has come to describing my ideal future data platform. The founder Dan Sotolongo's work at Snowflake included Dynamic Tables which makes me think the proposal is basically this: a hybrid transactional/analytical database, with declarative domain-oriented schemas, with smart incrementally calculated updates, and a la carte performance characteristics (i.e. opt into row or columnar storage, or whatever indexing you like). Nathan Marz's Rama is also close to this vision but perhaps a little less accessible for many enterprises as it's more purely developer focused. The win here is that you have a single system with all your data, all the time, and can build on it however you like. Piecing complex new services together across multiple microservices is deeply painful - if you can give me a single smart platform for all my data, and make it work operationally, that feels like a big win.
Obviously a lot of this you can piece together today, in fact Snowflake itself does a lot of it. But the other part of the article makes me think they understand the even harder part of the problem in modern enterprises, which is that nobody has a clear view of the model they're operating under, and how it interacts with parts of the business. It takes insane foresight and discipline to keep these things coherent, and the moment you are trying to integrate new acquisitions with different models you're in a world of pain. If you can create a layer to make all of this explicit - the models, the responsibilities, the interactions, and the incompatibilities that may already exist, then mediate the chaos with some sort of AI handholding layer (because domain experts and disciplined engineers aren't always going to be around to resolve ambiguities), then you can solve both a huge technical problem but a much more complicated ecological one.
Anyway, whatever they're working on, I think this is the exact area you should focus on if you want to transform modern enterprise data stacks. Throwing AI at existing heterogenous systems and complex tech stacks might work, but building from scratch on a system that enforces cohesion while maintaining agility feels like it's going to win out in the end. Excited to see what they come up with!
Sotolongo's lineage is Twitter observability -> Google streaming -> Snowflake Dynamic Tables, which is a declarative, relational, query-optimizer-centric tradition. Marz's lineage is Storm -> Trident -> Rama, which is a procedural-dataflow, programmer-controls-the-plan, event-sourcing-centric tradition. Both are trying to unify OLTP + OLAP + application logic + reactivity into a coherent substrate, but they're coming at it from opposite epistemological poles. Rama says "give the programmer fine-grained control over partitioning, indexing, and dataflow, and trust them to design the right physical representation for their queries." Cambra, if your inference about Dynamic Tables is right, will almost certainly say "let the programmer describe the domain model declaratively and let the system figure out the physical representation." This is the classic Codd-vs-Codasyl split, recapitulated forty years later with much more sophisticated machinery on both sides.
If this is the right framing, then the two systems aren't really competitors despite solving the same problem--they're going to appeal to fundamentally different developer sensibilities. Rama is for people who want to think like Jay Kreps or Martin Kleppmann: the event log is sacred, physical data layout is a first-class design decision, and the programmer earns the performance benefits by understanding the system deeply. Cambra (if these assumptions hold) will be for people who want to think like database users: describe what you want, let the optimizer figure out how, intervene only when necessary. These are both defensible positions and both have historical track records of working. SQL's history shows the declarative camp has ecosystem advantages once the optimizer is good enough; Kafka/Rama's history shows the log-centric camp has correctness and observability advantages for event-heavy domains.
Ok this was a nice read. Very expressive and I agreed with the majority of the points.
I'm not going to lie, I expected "and this is why we built <insert generic gpt wrapper>", but it never came.
Which leads me to my question, and forgive me for my impatience, but _what_ are you selling? Is it a platform? Ide? Books? So many words and I ended with more questions than I started.
On any who, good read indeed.
> I'm not going to lie, I expected "and this is why we built <insert generic gpt wrapper>", but it never came.
Uh...
> Implementing it is more than I can do alone, which is why my cofounders, Daniel Mills and Skylar Cook, and I are starting Cambra. We are developing a new kind of programming system that rethinks the traditional internet software stack on the basis of a new model.
This article reminded me a lot of a previous HN post[0] which hinted at choreographic programming[1]. While this post is light on details, I think this is an interesting space and deserves much more attention than it gets today. I'm excited to watch what comes from this space.
I've also felt that composition is unnecessarily hard in web & data infrastructure. I do game development as a hobby, and web + data technology is in the stone age compared to game engines when it comes to composability.
For example, say I develop some object (scene) in Godot Engine. It interacts with the environment using physics simulation, renders 3D graphics to the screen with some shaders and textures, and plays back audio from its 3D location.
I can send this scene to some other user of Godot, and it will naturally just work in their project, including colliding with their objects, rendering in their viewport (including lighting and effects), and the player will hear the location of the object spatially.
Of course there is much more you can do in Godot, too: network requests, event-driven updates, localization, cross-platform inputs, the list goes on. And all of these compose and scale in a manageable way as projects grow.
Plus the engine provides a common data and behavior backbone that makes it possible for a single project to have code in C++, C#, GDScript, and even other languages simultaneously (all of these languages talk with the engine, and the engine exposes state and behaviors to each language's APIs).
In fact, I've been thinking about making a Godot-inspired (or perhaps even powered) business application framework because it's just such a productive way of building complex logic and behavior in a way that is still easy to change and maintain.
So I imagine if Cambra can bring a similar level of composability for web & data software, it could dramatically improve the development speed and quality of complex applications.
Back in the days we had this in programming as well with palettes of drag & drop components.
It is kind of broken now, much thanks to using web applications (and applications that are basically just wrappers for web applications), but I don't know I if want to go back.
On one side it was much easier when I could hack together a program that was good enough (since everything was the same bland grey).
On the other hand some programs certainly looks nicer today.
And it has become easier to compose logic with solutions like Maven, Nuget and the various frontend package managers.
But yes, we lost drag and drop UI development, we lost consistency and we lost a lot of UX (at least temporarily).
I think there's a lot of potential for some more old-school UI in business software. People using points of sale, CRM/ERP/PLM/etc. systems, intranet portals, and so on don't really care about how nice it looks. Efficiency is more important.
Especially if it can be easy for non-technical people to build efficient UIs and databases (so they don't have to resort to spreadsheet contraptions), I think there's an opportunity here...
I've heard of people using Godot for non-game applications and I'm tempted to try something similar in the (near?) future because supposedly Low Processor Usage Mode makes it kinda feasible/sensible if you develop with that in mind from the start.
And while web pages can masquerade as desktop and mobile apps why wouldn't games be allowed to do the same? Godot for example can do desktop multi-window while something like Flutter (which is amazing in its own right) can't do.
But yeah, someone needs to spend time and build out UI toolkits for Godot and sadly that's not really a long weekend undertaking.
Still! It's nice to dream from time to time and imagine a reality where we can either do some generic cookie cutter UI because it's meant to get things done without much ceremony or we can pull out all the stops and plop a 3D scene to walk around the file system and shot files to delete them. And yeah, I'm aware someone did a thing like that in VS Code with Three.js (I think?)[0] and for Flutter you can do something similar in a webview inside the app proper.
Yet somehow I would rather do those things inside Godot for reasons unknown to me.
Your Godot scene doesn't work in Unreal Engine. I always felt that composition is unnecessarily hard in game engines infrastructure. I do web development as a hobby, and any .js file targeting the DOM works on any browser. I can't even run GDScript in Unity!
You're just comparing the wrong things. Yes, when you're locked into one environment, everything works together well. The moment you interact with outside systems, all hell breaks loose. If anything, what you're saying is just that platforms should have a much larger stdlib, or abstract platform differences properly (hint: this is only doable if you're a game engine and can afford to absolutely ignore _everything_ the OS does and just concern yourself with reinventing every wheel).
Not to say there's nothing good in the games side of things: a bunch of software could benefit from accepting that some systems like a big fat central message bus and singletons can be good when handled well.
This resonates a lot if you’ve ever worked in system integration.
In practice, most of the complexity comes exactly from what’s described here: every system has a rich internal model, but the moment data crosses a boundary, everything degrades into strings, schemas, and implicit contracts.
You end up rebuilding semantics over and over again (validation, mapping, enrichment), and a lot of failures only show up at runtime.
I’m skeptical about “one model to rule them all”, but I strongly agree that losing semantics at system boundaries is the core problem.
> You end up rebuilding semantics over and over again (validation, mapping, enrichment), and a lot of failures only show up at runtime.
I think die-hard fans of static typing mostly fail to acknowledge this objective reality and its implications. Every time they encounter this problem again and again, they approach it as if nobody thought of this before, and didn’t develop reliable abstractions to productively work in these environments.
Maybe this unfairly off topic, but in reading the intro, my thought was not that things become brittle and hard to change because of the inelegance of the abstractions, but because of the banal dynamics of work. Mostly, people act quickly towards short term goals, without full understanding. They accept technical debt, they pass things between people, and the numbers to avoid making things kinda gross just never work out.
Good. The article makes sense to me, it gives names to certain things I've already sort of internalized intuitively and not only does it not end with a proposal to use AI to fix everything, it actually explains how AI (or at least today's LLMs) fails at the same boundaries as we do, which is exactly what I've been seeing so far.
I'm looking forward to whatever these people come up with, because I believe they do understand the problem, which is the best starting position you can have.
I've always found system administration much harder than "coding". You get parser and compiler errors for a lot of your mistakes, but administering a server you're testing it all at runtime.
Em dash , corny sub headings, numbered lists for no reason. I think some of this may be human written or edited, but it isn't passing my smell test. Sorry if this is mostly or all human written, but you have a very AI style.
If the article is a product of a LLM, I am willing to pay good money to be able to use that exact LLM for development. Because it seems to understand many things that current LLMs and their loudest proponents don't seem to understand.
That's an interesting post, most of which I agree with. I think the problems it describes were mostly solved (and then rejected by the industry) in the 2010s with stuff like .Net MVC & WebForms (Java had similar frameworks). You'd still have a relational database at the bottom, but everything else was written in dotnet/C# under the MVC or WebForm model. Monoliths were the norm, and deployments were "just spin up a second server, and change the IP address when it's ready to go".
It was a very productive way to produce most software. But as soon as you want to do something off-piste, you pay the entire productivity penalty.
Once you have centralization, "composition" is not so hard. You get to define all your edge cases, define how you see the real world. Everybody doesn't have their own way of doing things, you have only one way of doing things.
Of course, then comes the extension of the software. People will see the world differently. And we have not algorithmically figured out how domains themselves evolve. The centralization abstraction breaks because people disagree and have different use cases.
I don't see how you get around this fundamental limitation. Are you going to impose yet another secret standard on everybody to get the interoperability you want? If you had full control over the world, yes, things are easy.
I'm not saying this as a diss. I truly do believe centralization works. AWS? Palantir? Building the largest centralized platforms in history and having everybody go through your tooling, when executed carefully, is a dummy effective strategy. In the past, monopolies were effectively this too (though I'd say buying steel is much different than "buying" arbitrary turing-complete services to help deal with a wide variety of semantic issues, and that's what precisely makes the 'monopoly' model break in the 21st century). And hey, at least AWS is a pretty good service, insofar that it makes certain things braindead easy. Is it a "good" service, intrinsically or whatever? I don't know.
No I mean like, centralization is unfortunately the thing that just works.
I work at a company that thinks extremely deeply about interoperability issues and everybody is on the opposite side: it can be said that we were made as a response to xkcd 927, to try and solve the issue.
I think the company is right in that semantic decentralization with interoperability would be a good end goal, but I think just plain darwinism explains the necessity of the opposite.
> Although the council had planned to implement Oracle "out-of-the-box," it created several customizations including a banking reconciliation system that failed to function properly. The council struggled to understand its cash position and was unable to produce auditable accounts. It has spent more than £5 million on manual workaround labor.
Not a great example of a single centralised system. The errors came from trying to write custom reconciliation code between two systems, the ERP and the bank - perfect example of the problems OP raises.
Fair point but AWS is also highly extensible, and i'm not sure about Palantir but i guess it must be too to a point? Maybe it's a classic case of good abstractions vs bad ones
I agree that this is our profession's Achilles heel. But I see one more cause: binaries and processes are black boxes unless they deliberately implement an interconnect feature. Try to extract the list of playlists from a Spotify client, for example.
We lucked into filesystems that have open structures (even if the data is opaque). Perhaps we should be pushing for "in-memory filesystems" as a default way of storing runtime data, for example.
I started getting into webdev using PHP almost 30 years ago. So I'm probably biased. But when you're developing on just one machine in one language and you can do most of the stuff you need to do within that one system, you can make progress very fast, and the system can support you coding fast (I'm not proud of it but I was live patching production code via SSH and refreshing a web page as fast as humanly possible to make sure it didn't break).
I believe there are several ways achieve that analogy today, even though the technology we have access to (and our own demands) has exponentially grown in complexity. I am happy to see more people thinking about it.
[Side track: I am personally not a fan of "break it up into many tiny systems" (microservices, etc) since it removes that agility of logic/state moving around the system. I just see an attempt to codify the analog of a very large human organization.]
Now that AI lets a single person (and in some cases, no person at all!) write several orders of magnitude more code than they would possibly have been able to, the requirements of our systems will change too, and our old ways of working is cracking at the seams. In a way we're perhaps building up a whole new foundation, sending our AIs to run 50-year-old terminal commands. Maybe that's all we needed all along, but I do find it strange that AI is forced to work within a highly fragmented system, where 95%, if not 99%, of all startups that write code with AI while hiding it from the user, are essentially following the recipe of: (1) launch VM (2) tell AI to install Next.js and good luck.
I too have a horse in this race and have come to similar conclusions as the article: there is a way to create primitives on top of bare metal that work really well for small and large applications alike, and let you express what you really wanted across compute/memory/network. And I believe that with AI we can go back to first principles and rethink how we do things, because this time the technology is not just for groups of humans. I find this really exciting!
Sounds like the typical "Distributed Monolith" antipattern. And we will see more of that because "software is cheap" and everybody will built microservices and find themselves in hell because nobody stopped to think if their domain context is well contained.
> What is Rama?
Rama is a platform for building distributed backends as single programs. Instead of stitching together databases, queues, caches, and stream processors, you write one application that handles event ingestion, processing, and storage.
> I think I’ve found a model that can break out of this tradeoff. Implementing it is more than I can do alone
I think anything that can change this has to be simple enough that it'd be more effective to just explain the system and implement it, than wax about the general outline of part of the problem. Especially since the real target audience for an initial release by necessity needs to understand it.
There are some big leaps we could make with having code be more flat. Things like having the frontend and backend handler in the same file under the same compiler/type checker. But somebody will want to interact with a system outside of the 'known-world' and then you're writing bindings and https://xkcd.com/927/
At the end of the day I think the core tension is that once the speed of light is noticeable to your usecase things become distributed, which creates the desire for separate rate-of-change. I'm not sure what would 'solve' that.
AI will be a plus, for the fact that a single team can be in charge of more of the parts leading to a more coherent whole.
Hope OP builds some nice tools, but I've seen too many of these attempts fail to get excited about "i think we found it".
There's a contradiction here that needs to be untangled:
There are many examples of models that enable coherent systems within specific domains:
- Type systems in programming languages catch many logic errors and interface misuses
- The relational model in databases enables programmers to access incredible scale and performance with minimal effort.
[...]
So coherent systems are great: everyone should just buy into whatever model will most effectively do the job. Right? Unfortunately, the listed models are all domain-specific–they don’t generalize to other contexts. And most modern internet software is not domain-specific. Modern applications typically span a wide variety of domains, including web and API serving, transaction processing, background processing, analytical processing, and telemetry. That means that trying to keep a system coherent limits what that system can ultimately do. As one implements more capabilities, application requirements push us outside of a single domain, forcing us to reach for components with a different internal model. So, bit by bit, our system fragments.
The problem of course is that type systems and databases are not meaningfully "domain-specific." They aren't technical magic bullets but they separately provide real value for the use cases of "web and API serving, transaction processing, background processing, analytical processing, and telemetry." So then why hasn't the industry settled on a specific type system? Why do database vendors (and the SQL standard) keep breaking the relational model in favor of something ad hoc and irritating?
I believe the real problem is that software is symbolic and the problems it solves usually aren't. Writing an application means committing to a certain set of symbolic axioms and derivation schemas, and these are never going to encapsulate the complexity of the real world. This relates to Greenspun's 10th rule:
Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
Or in a modern context, C++/C# and managing a huge amount of configuration data with a janky JSON/XML parser, often gussied up as an "entity component system" in game development, or a "DSL" in enterprise. The entirely equivalent alternative is a huge amount of (deterministic!) compile-time code generation. Any specific symbolic system small enough to be useful to humans is eventually going to go "out of sync" with the real world. The authors hint at this with the discrepancy between SQL's type system and that of most programming languages, but this is a historical artifact. The real problem is that language designers make different tradeoffs when designing their type system, and I believe this tradeoff is essentially fundamental. Lisp is a dynamically-typed s-expression parser and Lisp programs benefit from being able to quickly and easily deal with an arbitrary tree of whatever objects. In C#/C++ you would either have to do some painful generics boilerplate (likely codegen with C#) or box everything as System.Object / void pointer and actually lose some of the type safety that Lisp provides. OTOH Idris and Lean can do heterogeneous lists and trees a little more easily, but that cost is badly paid for in compilation times, and AFAICT it'll still demand irritating "mother may I?" boilerplate to please the typechecker. There is a fundamental tradeoff that seems innate to the idea of communicating with relatively short strings of relatively few symbols.
This sounds like Godel incompleteness, and it's a related idea. But this has more to do with cognition and linguistics. I wish I was able to write a little more coherently about this... I guess I should collect some references and put together a blog at some point.
> The problem of course is that type systems and databases are not meaningfully "domain-specific." They aren't technical magic bullets but they separately provide real value for the use cases of "web and API serving, transaction processing, background processing, analytical processing, and telemetry." So then why hasn't the industry settled on a specific type system? Why do database vendors (and the SQL standard) keep breaking the relational model in favor of something ad hoc and irritating?
I'm not sure what point you're trying to make here. The list you're referring to is definitely a bit hand-wavy, but it also makes sense to me to read it as, for example, "today's relational databases (software) are almost perfectly aligned to the domain of relational databases (concept)". As in, MariaDB running on my Mac wraps an insane amount of complexity and smarts in a very coherent system that only exposes a handful of general concepts.
The concepts don't match what I'd like to work with in my Rails app, which makes the combination of both a "fragmented system", as the article calls it, but the database itself, the columns, tables, rows and SQL above it all, that's coherent and very powerful.
It depends on what you mean by "almost perfectly aligned to the domain of relational databases" but by my standards I can't think of a single production database where that's true, in large part because it's not true for SQL itself.
- Tables are not relations. Tables are multisets, allowing duplicate rows, whereas relations always have a de facto primary key. SQL is fundamentally a table language, not a relational language.
- NULL values are not allowed in relations, but they are in SQL. In particular, there's nothing relational about an outer join.
In both cases they are basically unscientific kludges imposed by the demands of real databases in real problems. "NULL" points to the absence of a coherent answer to a symbolic rule, requiring ad hoc domain-specific handling. So this isn't a pedantic point: most people wouldn't want to use a database that didn't allow duplicate rows (the SQL standard committee mentioned a cash register receipt with multiple entries that don't need to be distinguished, just counted). Nullable operations are obviously practical even if they're obviously messy. Sometimes you just want the vague structure of a table, a theory that's entire structural and has no semantics whatsoever. But doing so severely complicates the nice symbolic theory of relational algebra.
That's the point I'm getting at: there isn't really a "domain" limitation for relational algebra, it's more that there's a fundamental tradeoff between "formal symbolic completeness" and "practical ability to deal with real problems." Eventually when you're dealing with real problems, practicality demands kludges.
> "we believe advances in programming language theory and database systems have opened a path that wasn’t available before"
Which is tantamount to waving one's hands about and saying there's "New magic!(tm)"
... while standing next to a pile of discarded old magic that didn't work out.
This blog post says nothing about what makes Cambra's approach unique and likely to succeed; it is just a list of (valid) complaints about the status quo.
I'm guessing they want to build a "cathedral" instead of the current "bazaar" of components, perhaps like Heroku or Terraform, but "better"? I wish them luck! They're going to need it...
> What code is truly about is precision: code is unambiguous, even when it’s abstract. It’s easy to conflate ambiguity and abstraction—both refer to “a single statement that could have to multiple meanings.” But the meanings of an ambiguous statement are entirely unconstrained.
I used to believe this, but after working at a successful SaaS I have come to believe that correctness and unambiguity are not entirely necessary for successful software products.
It was a very sad realization that systems can be flaky if there is enough support people to solve edge case problems. That features can be delivered while breaking other features as long as enough users don't run into the middle of that venn diagram, etc.
Fact is it always comes down to economics, your software can afford to be as broken and unpredictable as your users will still be willing to pay money for it.
A decision can be right even if the outcome doesn’t work out and a decision can be wrong even if the outcome works out.
Example: Gambling is wrong but you can win big money.
Right. One thing I learned over the years is that you can support arbitrarily high level of tech debt and still be able to effectively maintain and enrich a successful software system, as long as you throw enough warm bodies at it.
The overhead will get absurd, you'll end up with 10x or more increase in engineers working on the system, all of them making slow progress while spending 90% time debugging, researching, or writing docs and holding meetings to work out this week's shared understanding of underlying domain semantics - but progress they will make, system will be kept running, and new features will be added.
If the system is valuable enough for the company, the economic math actually adds up. I've seen at least one such case personally - a system that survived decades and multiple attempts at redoing it from scratch, and keeps going strong, fueled by massive amount of people-hours spent on meetings.
Adding AI to the mix today mostly just shifts individual time balance towards more meetings (Amdahl's Law meets Parkinson's law). But ironically, the existence of such systems, and the points made in the article, actually reinforce the point of AI being key to, if not improving it, then at least keeping this going: it'll help shorten the time to re-establish consensus on current global semantics, and update code at scale to stay consistent.
The classical case of this are banks that tend to keep ancient systems alive for a very, very long time.
> Amdahl's Law meets Parkinson's Law
[Infinite screaming]
This comment seems off-topic, but just to exemplify:
In your example even as the interface for those products is unstable (UI that changes all the time, slightly broken API), those products are coded in a language like C++ or Java, which benefit from compiler error checking. The seams where it connects with other systems is where they're unstable. That's the point of this blog post.
This is the realization that pretty much all engineers go through as they become more senior. The areas where the business really can’t afford issues is very small. Usually only accounting / billing and then that’s solved not by great code / design but just audit it once and never touch it again.
In the end most challenges for a business holding them back to better code quality are organizational, not technical.
> In the end most challenges for a business holding them back to better code quality are organizational, not technical.
This is true. And I get sad every time it is used as an argument not to improve tooling. It feels like sort of a self-fulfilling prophecy: an organizational problem that prevents us from investing into technical improvements... is indeed an organizational problem.
Yeah that's true, a product can be successful with truly bad code, but it also makes developers lives miserable each time they need to add a new feature, solve a bug, or simply understand how that entangled mess works.
Management and sales may not appreciate good software design and good code, the next developer that has to work on system will.
Not sure if tools and technologies can solve accidental complexity.
In my opinion, a system that has been stable for years isn't 'mature' in a good sense. An exceptional system is one that can still change after many years in production.
I believe this is almost impossible to achieve for enterprise software, because nobody has incentive to make the (huge) investment into longterm maintainability and changeability.
For me, consistent systematic naming and prefixes/suffixes to make names unique are a hint that a person is thinking about this or has experience with maintaining old systems. This has a huge effect on how well you can search, analyze, find usages, understand, replace, change.
You can't say...
> Not sure if tools and technologies can solve accidental complexity.
... and then say
> For me, consistent systematic naming and prefixes/suffixes to make names unique are a hint that a person is thinking about this or has experience with maintaining old systems. This has a huge effect on how well you can search, analyze, find usages, understand, replace, change.
I have battle scars from refactoring legacy systems where my predecessors did _not_ consistently or uniquely name things and I would not have seen it through without my sidekick, the type checker!
This is light on specifics, but is still directionally the closest anyone has come to describing my ideal future data platform. The founder Dan Sotolongo's work at Snowflake included Dynamic Tables which makes me think the proposal is basically this: a hybrid transactional/analytical database, with declarative domain-oriented schemas, with smart incrementally calculated updates, and a la carte performance characteristics (i.e. opt into row or columnar storage, or whatever indexing you like). Nathan Marz's Rama is also close to this vision but perhaps a little less accessible for many enterprises as it's more purely developer focused. The win here is that you have a single system with all your data, all the time, and can build on it however you like. Piecing complex new services together across multiple microservices is deeply painful - if you can give me a single smart platform for all my data, and make it work operationally, that feels like a big win.
Obviously a lot of this you can piece together today, in fact Snowflake itself does a lot of it. But the other part of the article makes me think they understand the even harder part of the problem in modern enterprises, which is that nobody has a clear view of the model they're operating under, and how it interacts with parts of the business. It takes insane foresight and discipline to keep these things coherent, and the moment you are trying to integrate new acquisitions with different models you're in a world of pain. If you can create a layer to make all of this explicit - the models, the responsibilities, the interactions, and the incompatibilities that may already exist, then mediate the chaos with some sort of AI handholding layer (because domain experts and disciplined engineers aren't always going to be around to resolve ambiguities), then you can solve both a huge technical problem but a much more complicated ecological one.
Anyway, whatever they're working on, I think this is the exact area you should focus on if you want to transform modern enterprise data stacks. Throwing AI at existing heterogenous systems and complex tech stacks might work, but building from scratch on a system that enforces cohesion while maintaining agility feels like it's going to win out in the end. Excited to see what they come up with!
Sotolongo's lineage is Twitter observability -> Google streaming -> Snowflake Dynamic Tables, which is a declarative, relational, query-optimizer-centric tradition. Marz's lineage is Storm -> Trident -> Rama, which is a procedural-dataflow, programmer-controls-the-plan, event-sourcing-centric tradition. Both are trying to unify OLTP + OLAP + application logic + reactivity into a coherent substrate, but they're coming at it from opposite epistemological poles. Rama says "give the programmer fine-grained control over partitioning, indexing, and dataflow, and trust them to design the right physical representation for their queries." Cambra, if your inference about Dynamic Tables is right, will almost certainly say "let the programmer describe the domain model declaratively and let the system figure out the physical representation." This is the classic Codd-vs-Codasyl split, recapitulated forty years later with much more sophisticated machinery on both sides.
If this is the right framing, then the two systems aren't really competitors despite solving the same problem--they're going to appeal to fundamentally different developer sensibilities. Rama is for people who want to think like Jay Kreps or Martin Kleppmann: the event log is sacred, physical data layout is a first-class design decision, and the programmer earns the performance benefits by understanding the system deeply. Cambra (if these assumptions hold) will be for people who want to think like database users: describe what you want, let the optimizer figure out how, intervene only when necessary. These are both defensible positions and both have historical track records of working. SQL's history shows the declarative camp has ecosystem advantages once the optimizer is good enough; Kafka/Rama's history shows the log-centric camp has correctness and observability advantages for event-heavy domains.
Ok this was a nice read. Very expressive and I agreed with the majority of the points. I'm not going to lie, I expected "and this is why we built <insert generic gpt wrapper>", but it never came. Which leads me to my question, and forgive me for my impatience, but _what_ are you selling? Is it a platform? Ide? Books? So many words and I ended with more questions than I started. On any who, good read indeed.
> I'm not going to lie, I expected "and this is why we built <insert generic gpt wrapper>", but it never came.
Uh...
> Implementing it is more than I can do alone, which is why my cofounders, Daniel Mills and Skylar Cook, and I are starting Cambra. We are developing a new kind of programming system that rethinks the traditional internet software stack on the basis of a new model.
That answers nothing. What is a "programming system"? If it were a programming language, they'd say that.
I think you mixed up AI model with just generally a programming model
Huh I thought they meant like a new paradigm.
This article reminded me a lot of a previous HN post[0] which hinted at choreographic programming[1]. While this post is light on details, I think this is an interesting space and deserves much more attention than it gets today. I'm excited to watch what comes from this space.
[0] Distributed Systems Programming Has Stalled: https://news.ycombinator.com/item?id=43195702
[1] Choreographic Programming: https://en.wikipedia.org/wiki/Choreographic_programming
I've also felt that composition is unnecessarily hard in web & data infrastructure. I do game development as a hobby, and web + data technology is in the stone age compared to game engines when it comes to composability.
For example, say I develop some object (scene) in Godot Engine. It interacts with the environment using physics simulation, renders 3D graphics to the screen with some shaders and textures, and plays back audio from its 3D location.
I can send this scene to some other user of Godot, and it will naturally just work in their project, including colliding with their objects, rendering in their viewport (including lighting and effects), and the player will hear the location of the object spatially.
Of course there is much more you can do in Godot, too: network requests, event-driven updates, localization, cross-platform inputs, the list goes on. And all of these compose and scale in a manageable way as projects grow.
Plus the engine provides a common data and behavior backbone that makes it possible for a single project to have code in C++, C#, GDScript, and even other languages simultaneously (all of these languages talk with the engine, and the engine exposes state and behaviors to each language's APIs).
In fact, I've been thinking about making a Godot-inspired (or perhaps even powered) business application framework because it's just such a productive way of building complex logic and behavior in a way that is still easy to change and maintain.
So I imagine if Cambra can bring a similar level of composability for web & data software, it could dramatically improve the development speed and quality of complex applications.
Back in the days we had this in programming as well with palettes of drag & drop components.
It is kind of broken now, much thanks to using web applications (and applications that are basically just wrappers for web applications), but I don't know I if want to go back.
On one side it was much easier when I could hack together a program that was good enough (since everything was the same bland grey).
On the other hand some programs certainly looks nicer today.
And it has become easier to compose logic with solutions like Maven, Nuget and the various frontend package managers.
But yes, we lost drag and drop UI development, we lost consistency and we lost a lot of UX (at least temporarily).
I think there's a lot of potential for some more old-school UI in business software. People using points of sale, CRM/ERP/PLM/etc. systems, intranet portals, and so on don't really care about how nice it looks. Efficiency is more important.
Especially if it can be easy for non-technical people to build efficient UIs and databases (so they don't have to resort to spreadsheet contraptions), I think there's an opportunity here...
I've heard of people using Godot for non-game applications and I'm tempted to try something similar in the (near?) future because supposedly Low Processor Usage Mode makes it kinda feasible/sensible if you develop with that in mind from the start.
And while web pages can masquerade as desktop and mobile apps why wouldn't games be allowed to do the same? Godot for example can do desktop multi-window while something like Flutter (which is amazing in its own right) can't do.
But yeah, someone needs to spend time and build out UI toolkits for Godot and sadly that's not really a long weekend undertaking.
Still! It's nice to dream from time to time and imagine a reality where we can either do some generic cookie cutter UI because it's meant to get things done without much ceremony or we can pull out all the stops and plop a 3D scene to walk around the file system and shot files to delete them. And yeah, I'm aware someone did a thing like that in VS Code with Three.js (I think?)[0] and for Flutter you can do something similar in a webview inside the app proper.
Yet somehow I would rather do those things inside Godot for reasons unknown to me.
[0] Found it: https://marketplace.visualstudio.com/items?itemName=brian-nj...
Your Godot scene doesn't work in Unreal Engine. I always felt that composition is unnecessarily hard in game engines infrastructure. I do web development as a hobby, and any .js file targeting the DOM works on any browser. I can't even run GDScript in Unity!
You're just comparing the wrong things. Yes, when you're locked into one environment, everything works together well. The moment you interact with outside systems, all hell breaks loose. If anything, what you're saying is just that platforms should have a much larger stdlib, or abstract platform differences properly (hint: this is only doable if you're a game engine and can afford to absolutely ignore _everything_ the OS does and just concern yourself with reinventing every wheel).
Not to say there's nothing good in the games side of things: a bunch of software could benefit from accepting that some systems like a big fat central message bus and singletons can be good when handled well.
This resonates a lot if you’ve ever worked in system integration.
In practice, most of the complexity comes exactly from what’s described here: every system has a rich internal model, but the moment data crosses a boundary, everything degrades into strings, schemas, and implicit contracts.
You end up rebuilding semantics over and over again (validation, mapping, enrichment), and a lot of failures only show up at runtime.
I’m skeptical about “one model to rule them all”, but I strongly agree that losing semantics at system boundaries is the core problem.
It’s one of the reasons AWS ended up happening, when you make REST APIs the only allowed contract.
> You end up rebuilding semantics over and over again (validation, mapping, enrichment), and a lot of failures only show up at runtime.
I think die-hard fans of static typing mostly fail to acknowledge this objective reality and its implications. Every time they encounter this problem again and again, they approach it as if nobody thought of this before, and didn’t develop reliable abstractions to productively work in these environments.
I'm genuinely interested in examples of these reliable abstractions, or at least in some keywords I can look up.
Maybe this unfairly off topic, but in reading the intro, my thought was not that things become brittle and hard to change because of the inelegance of the abstractions, but because of the banal dynamics of work. Mostly, people act quickly towards short term goals, without full understanding. They accept technical debt, they pass things between people, and the numbers to avoid making things kinda gross just never work out.
Good. The article makes sense to me, it gives names to certain things I've already sort of internalized intuitively and not only does it not end with a proposal to use AI to fix everything, it actually explains how AI (or at least today's LLMs) fails at the same boundaries as we do, which is exactly what I've been seeing so far.
I'm looking forward to whatever these people come up with, because I believe they do understand the problem, which is the best starting position you can have.
I've always found system administration much harder than "coding". You get parser and compiler errors for a lot of your mistakes, but administering a server you're testing it all at runtime.
If code > infra (wrt 'elegance' or whatever we're talking about), then just use code. I.e. build a monolith (or monoliths) as far as is practical.
right, I think DHH's "Majestic Monolith" is another good approach here
Em dash , corny sub headings, numbered lists for no reason. I think some of this may be human written or edited, but it isn't passing my smell test. Sorry if this is mostly or all human written, but you have a very AI style.
If the article is a product of a LLM, I am willing to pay good money to be able to use that exact LLM for development. Because it seems to understand many things that current LLMs and their loudest proponents don't seem to understand.
Have you ever asked an LLM if it understands this stuff?
So this is basically Pick (https://en.wikipedia.org/wiki/Pick_operating_system) for the modern world. Which is a good thing, its about time we resurrected this idea. I wish them the best.
That's an interesting post, most of which I agree with. I think the problems it describes were mostly solved (and then rejected by the industry) in the 2010s with stuff like .Net MVC & WebForms (Java had similar frameworks). You'd still have a relational database at the bottom, but everything else was written in dotnet/C# under the MVC or WebForm model. Monoliths were the norm, and deployments were "just spin up a second server, and change the IP address when it's ready to go".
It was a very productive way to produce most software. But as soon as you want to do something off-piste, you pay the entire productivity penalty.
The proposition reminds me of Rama by Red Planet Labs: https://redplanetlabs.com/
This sounds a lot like the original vision of darklang: https://darklang.com
Yeah exactly, came to the comments just for this check
Centralization is the #1 solution. It works. It's "ugly", but it works.
You see even on this thread people begging for one single standard.
What actually happens with that one single standard?
- Behind it, you have a shittload of people implicitly optimizing for the general use case and hiding all the said complexity for you
- No need to worry about [semantic conflict](https://www.sigbus.info/worse-is-better)
Once you have centralization, "composition" is not so hard. You get to define all your edge cases, define how you see the real world. Everybody doesn't have their own way of doing things, you have only one way of doing things.
Of course, then comes the extension of the software. People will see the world differently. And we have not algorithmically figured out how domains themselves evolve. The centralization abstraction breaks because people disagree and have different use cases.
I don't see how you get around this fundamental limitation. Are you going to impose yet another secret standard on everybody to get the interoperability you want? If you had full control over the world, yes, things are easy.
I'm not saying this as a diss. I truly do believe centralization works. AWS? Palantir? Building the largest centralized platforms in history and having everybody go through your tooling, when executed carefully, is a dummy effective strategy. In the past, monopolies were effectively this too (though I'd say buying steel is much different than "buying" arbitrary turing-complete services to help deal with a wide variety of semantic issues, and that's what precisely makes the 'monopoly' model break in the 21st century). And hey, at least AWS is a pretty good service, insofar that it makes certain things braindead easy. Is it a "good" service, intrinsically or whatever? I don't know.
I'm not disagreeing but I was reminded of a counterexample: https://www.theregister.com/2026/01/29/birmingham_oracle_lat...
No I mean like, centralization is unfortunately the thing that just works.
I work at a company that thinks extremely deeply about interoperability issues and everybody is on the opposite side: it can be said that we were made as a response to xkcd 927, to try and solve the issue.
I think the company is right in that semantic decentralization with interoperability would be a good end goal, but I think just plain darwinism explains the necessity of the opposite.
> Although the council had planned to implement Oracle "out-of-the-box," it created several customizations including a banking reconciliation system that failed to function properly. The council struggled to understand its cash position and was unable to produce auditable accounts. It has spent more than £5 million on manual workaround labor.
Not a great example of a single centralised system. The errors came from trying to write custom reconciliation code between two systems, the ERP and the bank - perfect example of the problems OP raises.
Fair point but AWS is also highly extensible, and i'm not sure about Palantir but i guess it must be too to a point? Maybe it's a classic case of good abstractions vs bad ones
Which is why Clojure advocates for simple data structures. They compose easier than code itself.
I agree that this is our profession's Achilles heel. But I see one more cause: binaries and processes are black boxes unless they deliberately implement an interconnect feature. Try to extract the list of playlists from a Spotify client, for example.
We lucked into filesystems that have open structures (even if the data is opaque). Perhaps we should be pushing for "in-memory filesystems" as a default way of storing runtime data, for example.
I started getting into webdev using PHP almost 30 years ago. So I'm probably biased. But when you're developing on just one machine in one language and you can do most of the stuff you need to do within that one system, you can make progress very fast, and the system can support you coding fast (I'm not proud of it but I was live patching production code via SSH and refreshing a web page as fast as humanly possible to make sure it didn't break).
I believe there are several ways achieve that analogy today, even though the technology we have access to (and our own demands) has exponentially grown in complexity. I am happy to see more people thinking about it.
[Side track: I am personally not a fan of "break it up into many tiny systems" (microservices, etc) since it removes that agility of logic/state moving around the system. I just see an attempt to codify the analog of a very large human organization.]
Now that AI lets a single person (and in some cases, no person at all!) write several orders of magnitude more code than they would possibly have been able to, the requirements of our systems will change too, and our old ways of working is cracking at the seams. In a way we're perhaps building up a whole new foundation, sending our AIs to run 50-year-old terminal commands. Maybe that's all we needed all along, but I do find it strange that AI is forced to work within a highly fragmented system, where 95%, if not 99%, of all startups that write code with AI while hiding it from the user, are essentially following the recipe of: (1) launch VM (2) tell AI to install Next.js and good luck.
I too have a horse in this race and have come to similar conclusions as the article: there is a way to create primitives on top of bare metal that work really well for small and large applications alike, and let you express what you really wanted across compute/memory/network. And I believe that with AI we can go back to first principles and rethink how we do things, because this time the technology is not just for groups of humans. I find this really exciting!
Sounds like the typical "Distributed Monolith" antipattern. And we will see more of that because "software is cheap" and everybody will built microservices and find themselves in hell because nobody stopped to think if their domain context is well contained.
Great idea, though maybe you meant "standards" not "models"? https://xkcd.com/927/
This reminds me of Rama [1] from Red Planet Labs
[1] https://redplanetlabs.com/programming-model
> What is Rama? Rama is a platform for building distributed backends as single programs. Instead of stitching together databases, queues, caches, and stream processors, you write one application that handles event ingestion, processing, and storage.
> I think I’ve found a model that can break out of this tradeoff. Implementing it is more than I can do alone
I think anything that can change this has to be simple enough that it'd be more effective to just explain the system and implement it, than wax about the general outline of part of the problem. Especially since the real target audience for an initial release by necessity needs to understand it.
There are some big leaps we could make with having code be more flat. Things like having the frontend and backend handler in the same file under the same compiler/type checker. But somebody will want to interact with a system outside of the 'known-world' and then you're writing bindings and https://xkcd.com/927/
At the end of the day I think the core tension is that once the speed of light is noticeable to your usecase things become distributed, which creates the desire for separate rate-of-change. I'm not sure what would 'solve' that.
AI will be a plus, for the fact that a single team can be in charge of more of the parts leading to a more coherent whole.
Hope OP builds some nice tools, but I've seen too many of these attempts fail to get excited about "i think we found it".
Can someone give a tl;dr? Feels like a whole lot of preamble in the article
There's a contradiction here that needs to be untangled:
The problem of course is that type systems and databases are not meaningfully "domain-specific." They aren't technical magic bullets but they separately provide real value for the use cases of "web and API serving, transaction processing, background processing, analytical processing, and telemetry." So then why hasn't the industry settled on a specific type system? Why do database vendors (and the SQL standard) keep breaking the relational model in favor of something ad hoc and irritating?I believe the real problem is that software is symbolic and the problems it solves usually aren't. Writing an application means committing to a certain set of symbolic axioms and derivation schemas, and these are never going to encapsulate the complexity of the real world. This relates to Greenspun's 10th rule:
Or in a modern context, C++/C# and managing a huge amount of configuration data with a janky JSON/XML parser, often gussied up as an "entity component system" in game development, or a "DSL" in enterprise. The entirely equivalent alternative is a huge amount of (deterministic!) compile-time code generation. Any specific symbolic system small enough to be useful to humans is eventually going to go "out of sync" with the real world. The authors hint at this with the discrepancy between SQL's type system and that of most programming languages, but this is a historical artifact. The real problem is that language designers make different tradeoffs when designing their type system, and I believe this tradeoff is essentially fundamental. Lisp is a dynamically-typed s-expression parser and Lisp programs benefit from being able to quickly and easily deal with an arbitrary tree of whatever objects. In C#/C++ you would either have to do some painful generics boilerplate (likely codegen with C#) or box everything as System.Object / void pointer and actually lose some of the type safety that Lisp provides. OTOH Idris and Lean can do heterogeneous lists and trees a little more easily, but that cost is badly paid for in compilation times, and AFAICT it'll still demand irritating "mother may I?" boilerplate to please the typechecker. There is a fundamental tradeoff that seems innate to the idea of communicating with relatively short strings of relatively few symbols.This sounds like Godel incompleteness, and it's a related idea. But this has more to do with cognition and linguistics. I wish I was able to write a little more coherently about this... I guess I should collect some references and put together a blog at some point.
> The problem of course is that type systems and databases are not meaningfully "domain-specific." They aren't technical magic bullets but they separately provide real value for the use cases of "web and API serving, transaction processing, background processing, analytical processing, and telemetry." So then why hasn't the industry settled on a specific type system? Why do database vendors (and the SQL standard) keep breaking the relational model in favor of something ad hoc and irritating?
I'm not sure what point you're trying to make here. The list you're referring to is definitely a bit hand-wavy, but it also makes sense to me to read it as, for example, "today's relational databases (software) are almost perfectly aligned to the domain of relational databases (concept)". As in, MariaDB running on my Mac wraps an insane amount of complexity and smarts in a very coherent system that only exposes a handful of general concepts.
The concepts don't match what I'd like to work with in my Rails app, which makes the combination of both a "fragmented system", as the article calls it, but the database itself, the columns, tables, rows and SQL above it all, that's coherent and very powerful.
It depends on what you mean by "almost perfectly aligned to the domain of relational databases" but by my standards I can't think of a single production database where that's true, in large part because it's not true for SQL itself.
- Tables are not relations. Tables are multisets, allowing duplicate rows, whereas relations always have a de facto primary key. SQL is fundamentally a table language, not a relational language.
- NULL values are not allowed in relations, but they are in SQL. In particular, there's nothing relational about an outer join.
In both cases they are basically unscientific kludges imposed by the demands of real databases in real problems. "NULL" points to the absence of a coherent answer to a symbolic rule, requiring ad hoc domain-specific handling. So this isn't a pedantic point: most people wouldn't want to use a database that didn't allow duplicate rows (the SQL standard committee mentioned a cash register receipt with multiple entries that don't need to be distinguished, just counted). Nullable operations are obviously practical even if they're obviously messy. Sometimes you just want the vague structure of a table, a theory that's entire structural and has no semantics whatsoever. But doing so severely complicates the nice symbolic theory of relational algebra.
That's the point I'm getting at: there isn't really a "domain" limitation for relational algebra, it's more that there's a fundamental tradeoff between "formal symbolic completeness" and "practical ability to deal with real problems." Eventually when you're dealing with real problems, practicality demands kludges.
Oh good
Obligatory XKCD: https://xkcd.com/927/
> "we believe advances in programming language theory and database systems have opened a path that wasn’t available before"
Which is tantamount to waving one's hands about and saying there's "New magic!(tm)"
... while standing next to a pile of discarded old magic that didn't work out.
This blog post says nothing about what makes Cambra's approach unique and likely to succeed; it is just a list of (valid) complaints about the status quo.
I'm guessing they want to build a "cathedral" instead of the current "bazaar" of components, perhaps like Heroku or Terraform, but "better"? I wish them luck! They're going to need it...