Same setup here since 2017. Since then, RAM usage decreased by 60%.
The admin panel is not something I'd need but it would be a nice-to-have.
Started with postgres as I wouldn't go for anything else if I wanna use it for decades. It has 2.5GBs for 10users and I don't mind if it takes 10 or 20, that's something I expected. Never did a cleanup of anything, I just dumped the db and moved to OVH recently onto a new VPS with NVMe SSD, it flies.
The fact that I cannot delete attachments that users delete is certainly my biggest irritation, 50GBs of stuff I am not sure if I can or cannot delete, but considering the size, I am just gonna bite the bullet, couple terabytes should not be a problem in 25 years. But this is def something I would love to see addressed sooner rather than later. It must be a pain even for the matrix.org server team.
After moving to a better server I do not have issues with slow notification unless the phone is sleeping for longer period of time which is an android optimization (I'd assume). It is more reliable than teams at this point. One of my friends had issues but removing 15 old devices fixed the issue.
As for element-x, I did call out "the another rewrite" issue especially with android and I do think it makes things worse. I still do not know how am I supposed to fix calling and video between old and new clients. For now I don't bother with new clients and everyone is using old ones, but it starts to become an issue as classic clients are in maintenance only mode
I ran my own matrix server for a number of years, then learned that any matrix homeserver would happily serve up any media from matrix without authentication. I shut it down the next day because I have no desire to operate a proxy for downloading csam.
A url like https://my.domain.com/_matrix/media/<some.other.domain>/<some-bad-media-id> would happily proxy the bad media through my server.
I think they've fixed this, but it's not worth the risk to me.
Whatever is going on in Matrix land isn't stable enough for most people to switch.
I gave up after they broke their calling system after changing something to this livekit system. It doesn't work, my existing TURN server became useless, and Matrix was left as a very slow chat application.
I don't plan to give up just yet. Switching is way too difficult and I don't see many platforms having both solid desktop and mobile versions with e2ee.
But if they dare rewriting anything again from scratch I am leaving. They MUST stick with what they have and make it good at this point.
None of my friends ever migrated from Discord to my server for over a year, so not much is lost. I'll join more actively maintained instances to stick around the communities and chats I found for myself.
I gave up after I upgraded my server from a SQLite backend to a PostgreSQL one with their conversion script which introduced errors into my DB.
Maybe one day I'll dig into it and see if I can fix the DB by extracting whatever data in it that's causing the errors, but like, is it really worth it at this point?
> What does running a matrix server get me in 2025?
Is there any other FOSS, self-hosted, decentralised platform with E2EE chat & E2EE group VoIP - i.e. the equivalent of Signal, but without depending on a centralised service?
From my pov (which is biased, as Matrix project lead), the downsides are:
* We still expose too much metadata to the server. Work is afoot to fix this, though - e.g. https://youtu.be/Q6NSmptZIS4?t=933 for encrypted state events.
* Synapse is still uses waaaaay too much database. I proposed a solution here: https://youtu.be/D5zAgVYBuGk?t=1852 but it needs to get properly implemented.
* Element's transition from the legacy apps to Element X has not been smooth, causing much of the gnashing of teeth in this thread (e.g. lack of interop between 1:1 voip and group-e2ee voip, or teething problems in the new apps).
* Post Quantum Encryption hasn't landed yet; it's been painful to get funding together for it.
I would say the biggest issue is not any of those things, it's that issues of this magnitude (including many of the ones fixed in the last year) are still being identified and requiring things like a "transition from legacy apps", breaking server changes, etc. In other words, it is not stable.
Haven't been on the main matrix.org server for many years, but they used to have a lot of issues with latency - it wasn't really instant messaging but more like email. When I felt it was the worst, I decided on self hosting it. That improved things significantly - it started to behave like an instant msg platform. Now with OVH migration I am even able to scroll through hundreds of attachments super fast and everything feels snappy.
And I also know there is no online status on the main server.
Other than that, no idea
Honestly this doesn't fit my usecase. If I understand it correctly, it is just like any other retention policy meaning it is not related to the fact that the attachment was deleted by user, but it just deletes everything after a certain point.
I don't mind storing things that people can access for many years - if I wanna see that funny picture from 10 years ago, I should be able to find it (even though GUI in Element sucks for that now). But what I do care about are things they uploaded accidentally or that they wanted to delete for whatever reason and it stayed on my server.
>this also creates a situation where anything said across federation cannot be unsaid, which is an ironic situation for a protocol/system that often comes up when talking about privacy.
How is it ironic? No protocol in the world can force anyone to delete anything from their own device. Chat apps that implement this function are either proprietary (so you cannot control what they can do) or, if OSS, do it on a pinky-promise-basis.
> No protocol in the world can force anyone to delete anything from their own device.
But they either do not sign the messages or allow repudiating the signatures. Matrix signs all events forever.
Matrix also makes the entire event history (minus message content depending on room configuration) available to servers on join, even if that server's users are not allowed to see it.
These are distinctions without a difference. Events replicated across several independent Matrix servers are not meaningfully different than events broadcast across independent clients in terms of observability or repudiation.
A protocol can mandate forced deletion. A particular client implementation may ignore it, or some users may circumvent it, so it would be a weaker kind of feature, but still a feature. And depending on circumstances it can be quite useful.
An open protocol can mandate indeed, but that is still in the realm of pinky promise security. A better design for a privacy-friendly chat protocol is to not write a lot of stuff on a lot of different remote servers when that's not necessary IMHO. One of matrix's selling points is to be censorship-proof though; in that case copying stuff as much as possible makes a lot more sense.
You are right, though I still prefer "weak feature" as a term :) There's enough value in such things. Cryptography crowd is concentrated on omnipotent Eve breaking ciphers, and that wrench from xkcd, but I dare to claim that majority of both commercial and private leaks happen just because well-intentioned users don't have enough capacity to keep track of all the things, and proverbially think twice. Features like "unsend", or timed deletion are indeed laughable on their purely technical merits, but do wonders saving users from grave mistakes anyway.
It's hard to explain to a non technical user. Something like "We tried to delete the message, but some of the people who received your message might still have a copy." Does not sound great and is going to be hard for a non technical user to understand and hard to implement in a way that a non technical user will find satisfying.
So if I was a dev on matrix/element and this feature came across my plate I would have to weigh it against features that I know can be implemented in a way which make technical and non technical people feel satisfied and better about the application.
A protocol can only support, never mandate. If I send you "DELETE MSG #4829" and you do nothing and reply with "200 OK; DELETE MSG #4829", nobody observing the protocol's messages will ever know what happened. Sure, an omniscent being could say "but he internally broke protocol, he didn't delete the message!", but by definition if something cannot be verified inside the protocol, it is outside of protocol.
In practice, in federated networks bad actors end up being blacklisted. It does not provide any "formal" guarantee, but… it tends to work fine enough. For this specific "deletion request" feature, of course it should always be seen as a convenience thing, and absolutely not about security.
As with many engineering things, it's tradeoffs all the way down. For instant messaging, a federated approach, using open protocols, offers what I value most: decentralisation, hackability, autonomy, open source. My options in this space are Matrix or XMPP. I have not attempted to self-host a matrix server, but have been very happy with my [prosody](https://prosody.im/) instance for almost a decade now.
I don't know what's wrong with XMPP other than the network effect collapsed when the GMail chat thing was killed, while the mobile client options were poor for a very long time.
Matrix has the appearance of being a drop in replacement for Slack or Discord, but the design decisions seem so compromised that the only explanation is they did manage to establish a (somewhat weak) network effect? It certainly is not a good look for an open source project to be running on Slack or Discord (free/cheap plans rugpulled or to be soon.) Then that leaves IRC, which has a network effect collapsing at a much slower pace.
I never got far enough to try hosting a matrix server, but reading the linked post -- Matrix definitely is not GDPR compliant. The combination of whatever end form of ChatControl the EU gets along with possibly hundreds of other laws across the world and individual US states makes me think the days of a public facing non-profit or small startup running a project like this are over. (Or maybe the future of open source is funding lawyers while the development is all done for pennies by AI?)
The GDPR is being neutered anyway because the EU caved in to Trump.
Not being chatcontrol compliant? That's a feature not a bug. Nobody wants that anyway. Just another stupid US lobby (Thorn).
A big organisation won't be able to run matrix for everyone no but that's the cool thing about it. People can run their own for smaller groups of people.
I don't know such definition frankly. And to the best of my knowledge there are plenty of things which people call "protocols" strongly prescribing actions non-verifiable in the very sense you used. That said I'm not here for a terminological discussion. We may call it green cheese, but it's still a useful feature.
Nobody claimed it isn't a useful feature. The only claim I made is that it cannot be mandated with an open protocol, so if you expect 100% adherence in the name of privacy, you're setting yourself up for disappointment.
People should related to anything federated like email. If you send something it is in someone else's computer now. With matrix or any e2ee protocols it is depending on pinky promise of the client to modify it. I thought the whole Snapchat fiasco already taught us that. Did we forget?
There's a difference between "I have an active adversarial actor" as a security model and "sometimes I send something I don't want to and want to delete it, the people watching are friends and acquaintances and are not deliberately preprepared to collect kompromat".
When I delete a message off Signal chat, the expectation is that the chat members are agreeing by social contract to abide by that.
There are many cases in between, though, in particular those where you're using messaging for "serious" purposes like business, but not so serious that the other users are your enemies. Sometimes for instance you want to let people edit/delete recent messages but not really old ones (to prevent someone trying to "rewrite history").
> How is it ironic? No protocol in the world can force anyone to delete anything from their own device.
You may have noticed the constant pushing to remove the user's access to their "own" device.
Forcing people to delete things from their own device is the whole concept of the Snapchat protocol, for example. Snapchat fortunately doesn't offer an OS and can't meaningfully be part of this push, but they make a convenient illustration.
You can check out Snapchat's bug bounty policy here: https://hackerone.com/snapchat . On the list of ineligible vulnerabilities is "screenshot detection avoidance". That's not a bug (because there's nothing they can do about it), even though it is their product's selling point.
Sometimes stronger companies want similar things, and they try to do something about it.
Right, but we did have efforts to take over hardware security enclaves to deliver user data, instead of copyrighted company data, to user devices.
Tim Berners-Lee tries to make the internet a place where you can choose, what it "forgets". At least that were the news I got from the 2010s and early 2020s. As for how: DRM-like tech in the hands of users should allow for that.
So having privacy by design would be nice, and e.g. many messengers try to do "it is inconvenient to copy a message that someone send you that is marked as view-only-once-or-up-to-a-timespan, but of course, you can use an external camera, i.e. make more low-fidelity copies or even exfiltrate data".
Even F/LOS software can use/would be forced to use these proprietary enclaves or at least non-user accessible key stores. (As far as I understand hardware level DRM.)
>Tim Berners-Lee tries to make the internet a place where you can choose, what it "forgets". At least that were the news I got from the 2010s and early 2020s.
Tim Berners-Lee created the web, not the internet, which is what chat apps use. Also, unless you can provide some direct quotes about it being designed for "forgetting" stuff, I have no idea where these "news" you got came from.
>As for how: DRM-like tech in the hands of users should allow for that.
If it's in the hands of the users, i.e. open source, it can be disabled at any moment, which is exactly what my reply already addressed.
>> As for how: DRM-like tech in the hands of users should allow for that.
> If it's in the hands of the users, i.e. open source, it can be disabled at any moment, which is exactly what my reply already addressed.
The point is that with the help of hardware-backed DRM on the client, the Matrix server could send data only to unmodified clients. You modified your client in a way that does not match what the Matrix server expects? No data for you.
Yeah I thought this was a weird take too. Too often people take privacy for "I can do what I like". IMO deleting something you've sent to someone else is not a privacy concern at all.
IIRC it is possible to have some clever encryption so that the person you sent your message to can prove to their own satisfaction that it came from you, but they cannot prove to anyone else that it came from you. Which gives you plausible deniability; you can always claim that your contact forged the message.
No particular name. Just deniability. I personally like to call this particular scheme, deniability through claimed forgery. Not particularly clever. You just provide your correspondent with what they need to forge your messages after the end of the session.
I don't know if it actually could work in practice:
Isn't the scheme simply agreeing in a shared key and both using it? I'll know that the message is from you if it's signed with that key and is not from me and vice versa, but neither of us can prove who created the message.
Just one example, but trying to get that revenge porn off the web, can be seen as an attempt to restore ones privacy. Where others should not have the right to continue to peek into ones private life.
That's not quite the kind of thing I was talking about. I think that is generally already covered by current laws in most places?
The right-to-be-forgotten advocates argue that everyone should have the right to demand that any trace of their previous online existence be deleted. On social media of course, but also independent web forums, chat logs, git commits, etc.
Even if it were a privacy issue, it would be impossible to enforce it technologically via FOSS software, because, by definition, the user at the other end could run a forked version with remote deletion disabled.
Ran a homeserver for 5 years on a minimal VPS and it worked fine. Upsides - works everywhere, self hosted, feature complete. Client software in the ecosystem mostly felt bloated, with the exception of NeoChat. By 2022 the clients could no longer call each other. Decommissioned it this year in favor of traditional XMPP which works fine and it's nice that notifications are appropriately processed, finally.
Our team highly appreciates the work done in Matrix it's just unfortunate that the elephant in the room was never addressed at the start of the project, which is the need for a -simple- first-party administrative dashboard or tool to manage users, storage, and configuration. Without that core component, then you've got a layer of complexity between an admin and an audit which will increase likelihood of misconfiguration or resource management issues.
In terms of VoIP interop - yes, one of the worst bits of Matrix is that the legacy 1:1 VoIP calling is not interoperable with MatrixRTC-based (multiparty) VoIP calling, but we ran out of time/cash to implement interop and instead focused on making MatrixRTC work well. (Does XMPP give you E2EE multiparty calling ooi?)
Thank you Aaron for the direct response. Glad to see the central interface for administrative use. Haven't had the need to use the calling feature in a while. One of the hurdles with the Rust client on Fdroid was it was too new for the OS on the device it needed to be used on, but on iOS it was a performance improvement.
One of the genius ideas behind Element was the ecosystem of clients services that attach to rooms, and in some cases XMPP does not reach those expectations. I do plan on a redeployment soon after creating a new technical scope for its use.
I like what I can observe of the new admin interface. I hope it can come to include security and configuration check guidances to provide tools for admins of all skillsets to properly configure their servers, and related TURN-or-not and storage.
Matrix is bigtime like XMPP - I don't think it's going anywhere anytime soon. Thankful for all that it has provided to our organizations.
Another cashflow idea, as if you don't think of many yourself: A white-label version of Element (called something else) that can be deployed as scoped to an organization and based on an LTS release schedule. We'd gladly pay for this.
Looks excellent, sharp website - and I'll admit I hadn't looked for the white label app before talking shit. That offering looks excellent - I see an opportunity for a white label tier between community and enterprise for small/medium orgs with pricing listed on the website.
Well, I can't check if it is still a thing today, but Element's managed servers used to allow you to do that, at least for the web app (though I had the idea they also did that for the old mobile apps?)
Thanks for mentioning XMPP. I never ran a server, but as a user I've been enjoying it. Also has its problems - for example, I picked some server to create my first account, and the other day the server disappeared for 6 hours. What do normal users (i.e. non-newbies like me) do in practice in these situations?
That said, I would love to hear your experience running an XMPP server. Do you still run it?
Trusting your chat activity and availability to any XMPP server on the internet that allows free public registration is not ideal. If you want reliability then pay for it or use what you perceive as a highly credible free server. I do not run any public facing XMPP services so I do not have any input on that and as with Matrix I wouldn't roll your own unless you know exactly what you're doing due to inherit risks associated with these platforms in certain states of configuration or exposure.
As a former user I felt these pain points trying to do nothing more than have a very active one-on-one chat with a good friend. Tens of messages an hour, maybe 2 years running. Using matrix.org and the pre-X clients. It's fine for group chat (IRC style) but that's not a high bar.
(a) the encryption between using a mobile and the webapp desyncs/breaks all the time, it just sucks. I mean you'll get "cannot decrypt" a lot, have to bounce back and forth and generally try and force it to re-sync properly again. Sometimes never worked at all. Lots of issues on GH over the years.
(b) as mentioned in this article, insane delays on new message notif and sending and receiving. Just logging in on the webapp every morning took minutes of some sort of mysterious sync process, often the mobile app had the same problems. The X stuff may fix this, we were pre-X.
(c) cleanup. There's no message retention set on matrix.org, when I wanted to extract and remove our past chats the process and experience was excruciatingly bad. It took tens of hours over several weekends of the webapp (mobile completely non-op in practice for this) polling and loading old content, just so I could select 100 at a time to delete and then it took an hour. Once I started culling back over a year or so, the loading got longer and longer and longer, until eventually it 100% stopped working at all to load old messages.
Signal and DeltaChat are far, far better experiences for one-on-one chats with friends & family. The Delta client is a bit UI/UX behind but not horrible; e.g. you can't correct a typo in a sent message in Delta, unlike Signal - because each msg is a unique gpg-encrypted "email" rather than a database object that can be re-manipulated.
a) As of when? I had a "cannot decrypt" room failure on matrix.org a year ago.
b) Unfortunately, X breaks other important things, like audio/video calls. It currently feels like an alpha-quality release: buggy and lots of missing features. Not ready for widespread use.
I felt bad making a long thread once I opened Element X and saw it didn't have support for threads.
Someone let me know later that threads are hidden behind a Labs setting, but it only allows the client to reply to threads, but still exposes the entire thread inline for the channel which sucks up all the air in the chat.
I am describing what I see on Element X on iOS today. If someone replies to threads, it is not hidden and navigable on this client. Even after switching the labs feature on, the app was showing the thread interspersed with the rest of the conversation.
Uninstalling the app, reinstalling, and making sure the labs option is turned on before navigating to a room with threads, is behaving how I expect.
So maybe, its a bug that the room does not re-render after toggling that setting.
I'm sorry you had a crap time, and agreed that in the past things were not great. But in defence of the Matrix team, we fixed almost all known encryption in ~Sept 2024, and the new generation of clients fixes the slow sync issues.
Hi Arathorn, I recognize your name as the core developer lead. :) Alas, when I was trying to cull my old messages I tried that (and a lot of other API type hacks I found floating around) and none of them, at that time, worked.
In short: the complete lack of user-facing easy to use controls for data retention and culling in the Matrix (Element) clients is a deal killer for me. That painful experience taught me a lesson - now when I test something new, one of the first things I look for is "how do I extract from this thing?". I never want to go through my Matrix extraction pain again, so a personal life lesson was learned.
> The only thing that I don't really understand is the decision on data replication. If a user on server A joins a room on server B, recent room data is copied from server B to server A and then kept in sync on both servers.
The idea here is that rooms are abstracted from servers and sort-of exist ephemerally. This has the advantage/disadvantage of making it hard for the underlying infrastructure to exert control over the hosted communities, and seems to have become a distinguishing feature of federation.
My experience of Matrix as a possible replacement for Discord has led me to believe it's mostly a disadvantage since it leads to gross misalignments between the communities in top and the infrastructure providers underneath. I consider e.g. Discourse to be much healthier (although I would like to see an app for Discourse so that my Discourse communities behave more like Discord/Slack servers) and it's frustrating to me that there hasn't been a clear "Discourse for chat" emerge to replace Discord.
> it leads to gross misalignments between the communities in top and the infrastructure providers underneath
Yeah, this is a great way to put it. We see this a lot with the mod experience. So much of the mod tooling is "run this bot on your server" or "configure synapse like this". But those kinds of things are inaccessible to people who aren't able to run any server of any kind. Of course someone else can run the mod bot for them, but then you still have the same problem where it introduces a layer of friction between the person who needs to perform mod actions and the person who controls the mechanism for doing so.
Until there's a robust and full-featured mod toolkit built in at the protocol/client level, it's a dicey situation for people who want to use Matrix to host a community. The insidious part is that everything may seem fine until suddenly your room is flooded with images of gore or worse.
(By the way, we still miss you in the Python room on Matrix! :-)
You could also try Movim. You could have a decentralized XMPP server with a client that support group calls as well as having posts like a forum folks can comment on.
There are a bunch of options out there (though I've never seen Movim - thanks, I'll check it out) but most communities seem either to be on Discord or Matrix (with a few still hanging on to IRC and a few others on Slack) - Discord being by far the best UX of the lot.
> The idea here is that rooms are abstracted from servers and sort-of exist ephemerally
No, that's not even remotely true. In fact the opposite is true. The domain name of the server used to create the room is perpetually and permanently embedded in the room name and can't be changed, ever.
No, not at all. Posession of a CA-signed TLS certificate for that domain, and the corresponding private key, is basically the root of trust for room operations.
I've been using Matrix for several years as a user. It works great. The problems decrypting messages have gone. X is becoming a good client.
I'm deleting my whatsapp and télégram accounts in a few weeks after a painful week-long backup...
Edit : I wonder how easy it is to backup a Matrix accounts's data. Conversations and files.
> I wonder how easy it is to backup a Matrix accounts's data. Conversations and files.
Not as easy as you might hope. The Element client has an export feature, but you have to manually activate it on each room/chat, and the export has a size cap so it may not work if you have lots of files you want to save. It's also pretty slow if the room has a lot of history. You could also try using something like Matrix Commander (a command-line client), but I couldn't get that to work fully either.
I’ve been running a Matrix server for about two years on a Proxmox host in a colo I rent for the purpose (plus some other hobby stuff, but mainly because I just think it’s cool). This playbook is awesome and it’s pretty easy to set up and keep running:
https://github.com/spantaleev/matrix-docker-ansible-deploy
Regarding the "Requires federation" section, that is not true. I've been running a small family-only homeserver for several years now, and had federation disabled on it from the very beginning, and there have been exactly zero issues related to (lack of) federation with it.
Same here, though you do still have to expose it to the internet unless you use a VPN. I'd prefer something with less of an attack service especially because bridges don't currently encrypt, but I host behind vpn now.
That's not a fair criticism. Of course you need to expose the server to the network clients are going to be connecting from. Whether that is Internet or just a LAN, that's orthogonal to the protocol.
I've also had an intra-company Matrix server running completely on a company internal LAN, with no Internet access, so there is no inherent need for it to be on The Internet.
I've installed fresh matrix about 8 times or so, got it to work 3 of those times. I started looking for how to do a fresh install earlier this year (matrix/synapse element) and I have no idea which docs to use,
It was bad enough before that I had to detour to random places to find workarounds for things like how nginx does it's thing differently than the matrix docs I was reading at the time.. but now with the newer element that makes it easy to not find a self hosting way - to main docs being listed as archived or old -
I'd enjoy installing it fresh with someone from matrix watching the process just to see what a slog it is - and then watch someone try to get some addins running like admin and moderation.
wordpress is a one click install via softalicious and similar on most cpanel 10$ and up hosts.. you can one click add a decent free chat plugin if you are on a VPS.
If matrix was that simple to setup, lots of people would be using it.
> While technically, Synapse can work with a sqlite database (and which at first seems like an OK choice for having <10 users on the server), it WILL become corrupted.
Does anyone have any more information on this? Running Postgres is not a big deal, but I would expect SQLite to be fine given how well it works in my experience.
I've not seen many corrupted sqlite databases, but because Synapse is not currently storage-efficient you can end up with some terrifying large (100GB+) sqlite databases which might as well be corrupted.
We only ever supported sqlite for ease of tinkering; it was never intended to be used in production, and in retrospect supporting it at all was a mistake.
I use sqlite for my 2-user server and haven't had any issues in the several years I've been running it.
It's possible to corrupt a sqlite database file, but generally it shouldn't happen unless you're doing something weird with it. https://www.sqlite.org/howtocorrupt.html
I've been running unfederated Matrix instance with sqlite for 10 years for 10-20 people some very active. In WAL mode sqlite is fine i am not sure why it would get corrupted.
Few years ago we did a data/history wipe (i managed to migrate accounts) because we switched to conduit.rs with sqlite. I very much prefer conduit to synapse.
I never had issue with sqlite it was mostly Synapse. With recent changes i think they are completely dropping the ball on small instances and it feels like the ecosystem is splitting. Element X is just not very good. It seems to exist to kill classic element and since for element x you need some server features that not all alternative synapse servers support.
> While technically, Synapse can work with a sqlite database (and which at first seems like an OK choice for having <10 users on the server), it WILL become corrupted
I want to hear more about this. Is this because Synapse’s SQLite support is half-baked? What sort of corruption are we taking about?
I use sqlite for synapse (since ~2 years) and I haven't noticed any corruption at all. I am in lot of rooms (20-30) with 1k-2k users in a lot of them and the db size is 8.8G currently
I've tried off and on to actually use Matrix. I was a bit of a loud supporter in the early days. Unfortunately, it looks like it still hasn't grown past the fundamental issues I was having then. It might be time to try something else
As someone who has looked into forking Matrix for a new type of chat service, I'm grateful to see a more in-depth look at running it behind the scenes. Thank you.
I've been developing open source code for over 25 years. i've deployed hugely complicated systems like hadoop. I've never seen anything as hard to run as this + the bridges.
OT: I have some very big groups in Telegram (7 years or more, with a lot of pictures). Can Matrix (Rocketchat or alternatives) have similar storage features (with some migration scripts)? Thanks
To add another data point, I've been hosting a (tiny) matrix server for a few months. I'm pretty comfortable with self-hosting using docker, so I opted not to use the ansible scripts in the hope that it'd keep my setup simpler and more maintainable. Somehow I didn't find any mentions of ESS until Synapse was already up and running, but Kubernetes would have been a dealbreaker for similar reasons.
In this short time I've run a database migration (sqlite is the default, but MAS requires postgres), tried and failed to migrate to MAS (required to use Element X) and have lost a couple of days messing around with coturn and eturnal with nothing to show for it -- my calls still don't connect when NAT is involved. I have to tell new users to ignore the recommendations to install Element X until I get MAS working.
There's a lot of room for foundational improvements here, even updating docs to point would-be server admins to the recommended setup du jour would help.
TLDR Self-hosting isn’t dying because people stopped caring. It’s dying because the complexity has gotten out of hand.
This post highlights how something that used to be a fun, lightweight hobby has turned into a full-time maintenance burden. Systems like Matrix are powerful, but they’ve become so intricate that even skilled engineers struggle to run them reliably. The result is a slow drift back toward centralized platforms, not out of preference, but because convenience keeps winning over autonomy. It’s a reminder of the growing gap between the ideal of a user-owned internet and the realities of modern software.
I would say it became much easier in recent years. docker-compose became defacto server "app install", any linux supports it. That includes GUI options like Truenas/Unraid and very nice admins like Dockge exist.
The company behind matrix is aiming at huge scale servers but if you care about unfederated private instances you will find there are few much simpler "one binary" projects that can even use file based sqlite/rocksdb. Hosting those couldn't be simpler. You actually don't even need docker just systemd service and switch binary when updating.
The reason Element ships ESS Community as kubernetes helm charts is simply so that the officially supported distro is as close as possible to the thing we have to support for the big govtech customers, where docker compose simply isn't going to cut it. Perversely, that means that ESS Community is pretty bulletproof (even if the scaling stuff is missing), but comes at the expense of having to run k8s.
Yeah, I think this is true and it's unfortunate. I think it's one of the areas where what seemed like the plan for Matrix hasn't totally worked out in practice. And it's even just the intricacy of managing it; some of the problems are deeper in the design. When I started using it several years ago, it seemed their vision was a lot of people running a lot of small servers. But the full-replication nature of the protocol, plus the resource demands of the server software, make that kind of impractical. Even tech-oriented people may shy away if they find out the database could grow to tens of gigabytes, and Synapse is not exactly light on RAM or CPU either.
Perhaps in the future if implementations improve some of this may get better, and it will become more feasible for small operators to run their own servers. But by that time it will be harder to build trust because too many people will have written it off as bloated or unstable. I think it would have been better to start lean and keep the system in more of a nerd niche until that process of evolution reached a later stage.
NixOS makes this rather easy too. There exist modules for synapse, livekit, etc (MAS module is still a Pull Request, though) and the setup is quite doable: https://wiki.nixos.org/wiki/Matrix
However, you still need to know what you are doing (the manual helps) and connect the pieces (in theory there could be a nixpkgs module that does this for you but apparently nobody did bother). Once its done you can lean back.
I've been running my homeserver happily for > 5 years and it was fairly straightforward.
Yeah, but then you have to run NixOS, which besides the sheer difficulty of being Linux, for the average user, stacks Nix on top, which comes with its own learning curve.
Also, based on your description, you're a techie. I highly doubt the average city hall clerk will set up their on Matrix server.
I've been running synapse on a small VPS for the past few years. I got some of the bridges working too. Def bumps along the way, but its still the daily driver for me and about five other people.
Recently I spun up the new ESS Community Edition on a new VPS. Much easier to get up and running. I was delightfully surprised. although since that one uses kubernetes and other things I'm not familiar with getting bridges and other things I've become used to is going to require more learning on my part. Since ESS is so new, not a lot of newbie friendly howtos yet out there.
XMPP felt great when compared to Matrix. Matrix was in a bad state some years ago when I hosted it for a while, and seems like it still is the same messy state. I avoid it as much as possible but for some reason there are communities using it.
At this point it should just die so people would be motivated to replace it with something better.
Which perfectly fits the Matrix ethos of everything constantly being rewritten or replaced with something new that only implements half the features before the original thing is ever finished.
I've had only one issue with ejabberd (didn't read the docs close enough and had to move from mnesia to sql when I hit the size limit) in over a decade of self-hosting it.
Knowing which XEPs you actually want is a bit of a pain, but has gotten easier. Now I usually just periodically look at the current prosody example[1] which lists extensions by "essential" "recommended" "nice to have" and "other" and it's easy enough to find the matching modules for ejabberd.
One of my favourite small features we added in the latest release is a 'prosodyctl check features' command which will validate that your configuration is up to date with current best practices in various ways.
Although we like to curate the default configuration with care, people tend to keep their existing configuration file when they upgrade (we generally recommend this, as we aim for backwards compatibility). The new command makes it easier for folk who upgrade to ensure they are still getting the expected experience as the recommendations evolve.
Obviously that has to be somewhat opinionated. One thing many people appreciate about Prosody is that you can customize it however you want to, even if it's not following the mainstream. E.g. if you really don't care about synchronizing to multiple devices, you can totally disable that message cache. Don't need audio/video calls? Fine!
I have been hosting synapse for 2 years now and it's been a smooth sail. I don't recall having any major breaking changes, most updates are smooth. Element client itself is definitely PITA but it's getting better
I set up Whatsapp and Telegram bridges with Tuwunel today. The config file for mautrix-whatsapp indicates Dendrite should work as well so it seems just about every server implementation supports bridges?
Former Dendrite user: Yes and no. Mautrix stuff does work with Dendrite, technically, but only partially. For Mautrix-Discord, I was able to get DMs mostly working, but only DMs: bridging servers just never really worked at all. I'm honestly not 100% sure why this is the case, but it's not worth worrying about Dendrite anymore. It is a dead-end server that never worked very well. Up until shortly before it was discontinued, none of the Mautrix bridges worked well enough to actually use.
Been selfhosting synapse for about 1.5 years in a docker compose setup using bunkerweb (formerly "bunkerized nginx", which better explains it premise) reverse proxy, eturnal for TURN and postgres, also recently added livekit and MAS for element call and element X compatibility. All that runs on a small 2vcore/4gb VPS, and it runs pretty good, I experience a server crash every half a month, but that may be caused by the fact that bunkerweb isn't the most lightweight solution (they actually require 8GB RAM minimum, so I'm already beneath the limit), and also because I run some other software (mailserver, ebook server, plex, etc..).
My experience as a administrator has been pretty good, perhaps it's because from the beginning I was optimistic, it suited my needs as I wanted a selfhosted, modern and fairly convenient communication platform. From what I recall, most problems during configuration were caused mostly by bunkerweb (or rather my inability to correctly set it up to proxy requests correctly and not hijack the 4xx and 5xx HTTP codes). Synapse itself has been a pleasure to maintain, but also bear in mind that I did not tinker with with it, I basically set it up and let it run for about a year and then added MAS and livekit.
Yeah, disk usage sucks, for about 5-10 active users and 1.5 year usage my postgres "schemas" folder clocks at 10Gibs. It doesn't include the media_store catalog where synapse keeps media (images, videos). The homeserver is federated and I joined a couple of big rooms in the past. Mechanics mentioned in the links below do help though:
Clientwise also sucks, but I think enough has already been said on this matter. But it's good enough to keep my nontechnical friends using it. They do hate it, but not enough to kick me in the arse. Would love to say that this proves that element clientside is usable, but I also have to admit that my friends are just hella good guys who would even write pigeon mail to me if I stopped using anything else for communication :) for me as a techie, element is obviously alright. Clunky, but works. I think clients simply need more time.
What irritates me is the Matrix authentication service (MAS), it's kind of a separate service for matrix homeservers that handles logins specifficaly. You can't use element X without it. However when it's enabled, you cannot log in from your client, instead a web browser opens and shows the login panel where you have to authorize, and then it should return to the client. Except in my case it simply doesn't :( I observed that for some reason chromium based browsers won't redirect back to the element app, and it doesn't know that the authorization has been granted. I managed to bypass it by copying the URL and opening it on firefox, but in one instance even that didn't work.
But other than that MAS problems everything has been fine from administration standpoint. I think it simply needs more time, as it already has traction, I see that a lot of new projects seem to include a matrix room in their social/communication channels, frequently it's the only option besides the bugtracker. And I'm willing to wait patiently :)
edit: added links for people who also struggle with disk space usage
Ran synapse for a few months, figured out all the tui clients are either abandonware or broken (originally thought i could use bitlbee, and did the install before realizing it was unusable).
Looked at current tui offerings now some years later, situation seems to be unchanged, the only client that ran back then was gomuks, and that has received a rewrite that hasn't reached feature parity yet.
I am probably the type of person referred to in the last part of xkcd 1782.
I've mentioned this here before, but it bears repeating. A couple years ago or so, I made the catastrophic choice to use Dendrite as my homeserver software. It seemed like a safe bet: it was supposedly lighter weight than Synapse, being written in Go instead of Python and with everything reengineered from the ground up. It didn't support everything, but nothing in the disclaimers made it seem like it was about to abruptly become defunded and essentially unmaintained. Alas, that's exactly what happened not too long after I made that choice. Despite showing no interest in maintaining it, New Vector still found it necessary to relicense their abandonware under the more restrictive AGPL license. Good priorities. Then, when a security patch was needed, a new release was rushed out that included not just a security fix, but also a bug that caused Dendrite to completely stop processing messages for minutes at a time multiple times a day. (This only got fixed months later by a volunteer.) Joining large federated rooms on Dendrite took so long that I thought it was just broken; it could take hours to days to complete the operation! There was even a brief period when Element actually didn't even support Dendrite, leaving everyone locked out. Dendrite has never supported Element X, and the old sliding sync proxy was never updated to support the new simplified sliding sync either, which means you're stuck with the old slow sync and no support for things that require it. Also, most appservers still don't work right on Dendrite either. I got Mautrix-Discord working, but only for DMs.
I legitimately could go on and I'm sure I've forgotten things. It's amazing how quickly my experience with Dendrite went from pretty good to nightmare.
I realized that nobody in charge at the Matrix Foundation or New Vector really cared enough about leaving people stranded on a completely broken server to actually do anything about it (and trust me, I'm not alone. In every single federated room I've ever been in, I've always seen hostnames with dendrite subdomains. I could see them pass by in the logs while joining servers was taking hours.) I honestly considered just leaving the Matrix ecosystem, but I wasn't alone on my home server, so I decided to do my best to fix the problem. I wrote a tool that attempts to migrate the data from Dendrite to Synapse. This is a complicated operation that really took a huge amount of effort to get working, but after a couple of months of failed attempts, I had a test where I was able to seamlessly perform the migration and have clients continue to work and stay in federated rooms. So after getting it "close enough", I went ahead and gave the migration a shot in production and of course, it didn't work very well. All of the user accounts were intact, but a lot of stuff was broken. People indeed stayed in federated rooms, but my room state migration was definitely not 100% correct. Despite this, though, after manually cleaning up the database a bit more, hackishly while live, it was mostly a success. I believe I am probably the first person to directly move from Dendrite to Synapse.
So now that I am on Synapse, have my thoughts on Matrix changed? Yes. It's significantly better using Synapse, without question. The ecosystem is still a mess, but everything about Synapse is less broken than Dendrite. There are so many features Dendrite just doesn't do, like URL previews.
Why not contribute to Dendrite? Honestly, I don't want to. Their CLA sucks and they're not going to change it for me, and I don't think they're really going to spend time reviewing PRs given the circumstances. If I'm going to contribute to a project without retaining my rights I'd prefer to be on payroll. That's not something you should get from a community member. Either change the CLA to guarantee the project must stay open, or don't expect any free contributions.
Why not post my migration tool? Well honestly, for starters, it's not a very high quality tool. I could probably do some good for the Matrix ecosystem if I could get this tool in much better shape and have it migrating complex room states correctly, but I don't even know if I want to help anyways. This should've never been my problem. I will fully admit that it was my bad choice that got me here, but I really think it can be forgiven: nothing I saw suggested to me that Dendrite was on the way out. On the contrary, everything suggested it was the future, and just simply not ready for large scale usage yet. I'm bitter. I spent a lot of hours on this problem and I feel like hours spent on the Matrix ecosystem won't be repaid.
I hate to be this cynical, but it's just how it is. It's a mess. I didn't bother going into the other messes that still exist when using Synapse, like the seemingly many different ways that VoIP can work in Element and Element X, and the fact that Element X seems to only support a newer VoIP protocol that Element on desktop does not. (Surprise! There is no Element X on desktop...)
Matrix has some other downsides, that I think are tolerable but definitely make me a bit bummed. It leaks quite a lot of metadata to the homeservers, which is kind of alright, but I do think it's a bit sad; even room names are not encrypted, clearly it would be possible to do better. The ecosystem of clients is sad; Element is the only one that is feature complete and while I think it has improved quite a lot I still would prefer a native application over a web view. (You kind of need a webview if you want feature parity though, since group A/V in Element desktop seems to just use a Jitsi iframe...)
The upside is that it is federated and at least messages and files are E2EE in DMs and optionally for groups. I do like that.
Personally though the federation thing feels a bit off to me. I know it's a pipe dream, but it just feels like 1-on-1 and small group DMs should be roughly peer-to-peer. Servers should be for chatrooms and relays. The problems I see are mobile notifications, offline messaging, and discovery. I have wondered if a model like AT proto could get you there for DMs. I would like to try to prototype something some day, but I know that at this point the XKCD 927 count for IM software is pretty insane, so if I'm going to throw my hat into the ring it better be worth it.
Maybe some day I will be less bitter. I mean, Matrix is free, how much can I really be angry if it didn't work how I wanted it to. But, it's hard. I tried to buy in hard, and wound up making a lot of trouble for me and a small group of people that I inadvertently looped into my own mess. I trusted Matrix because it seemed to be the leading option, but definitely I will now be much, much more careful before adopting an ecosystem into my life and the life of people around me. For example, I still have no ActivityPub server... Maybe it's better if I just wait and see what happens there before jumping in, if I'm going to.
This doesn't really excuse it, but to give you some idea of why Element haven't spent any time on dendrite right now, here's a sample of active servers on the federation as of today:
To add on to that, none of their customers use it, and there's no real community around it, unlike the independent Conduit-family servers that make up that remaining 11%.
Dendrite was a bet for two things: Could they make synapse faster? Yes. Synapse is faster now, and Synapse Pro is apparently even faster. And, can we make Matrix peer to peer? They ran out of money, and didn't have any customers who would fund it themselves. They're left with this project that is idling without a community, a customer, or a reason to exist.
Apparently, they have some funding to do work on the foundational parts of p2p now, but that will take a long time, and Dendrite is unlikely to be a part of that for a while, possibly at all (the Rust ecosystem seems to be where Element invested their time on the client, while Beeper invested in Go).
In the end, a lot of Element's pain is because of ambitious technical decisions made without a way to back them up practically or business wise. Things would be amazing if everyone working on Matrix had infinite time and money. Unfortunately, they don't, and Element is eating the consequences for acting like they did for a while. They seem to be better now, though.
I honestly don't blame them that Dendrite didn't work out, I blame them for leaving us all stranded without any sort of migration path. I don't personally care if my homeserver is written in Python or Go or Rust, at least not very much. Rust may very well be a better choice than Go in the long run. I only chose Dendrite because when I chose it I felt like it was in better shape than Conduit (which at the time wasn't forked off, wasn't well-supported by app services either, and didn't handle federated presence) and lighter weight than Synapse.
Is my homeserver even counted in the 369 active Dendrite servers anyway? I mean, I guess it definitely isn't in that count since it is now an active Synapse server instead, as of late November. However for what it's worth, I don't think I actually connect to Matrix.org at all. Maybe it still counts if you're in a room with Matrix.org users?
But assuming it is basically just 369 Dendrite servers, well, they're all stuck there. How hard could it have been to provide a migration path? I dunno. My attempt is definitely incomplete and possibly quite ill-advised. I truthfully don't understand Dendrite or Synapse well enough to design a migration tool that works correctly. Did I calculate forward extremities correctly? Maybe. Are my state groups actually all correct? Probably not. How the hell did I end up with rooms that have multiple auth events? I have no idea, I just wound up destroying the corrupted rooms I found. I can only imagine that whoever developed the Dendrite database structure also had knowledge of how the Synapse database structure works and would've been able to do all of this in many less hours than I was.
After all is said and done I'm sure the balance sheets will look much better but the damage done to the community side of things will linger. I won't feel bad if Element fails and Matrix winds up replaced by a different standard later.
FTR the server stats are from https://matrixrooms.info/stats. It's not a full view of the network but it's a reasonable sampling, based on servers discovered through public room aliases and servers that use it as a notary (trusted key server).
> Maybe some day I will be less bitter. I mean, Matrix is free, how much can I really be angry if it didn't work how I wanted it to. But, it's hard. I tried to buy in hard, and wound up making a lot of trouble for me and a small group of people that I inadvertently looped into my own mess. I trusted Matrix because it seemed to be the leading option, but definitely I will now be much, much more careful before adopting an ecosystem into my life and the life of people around me. For example, I still have no ActivityPub server... Maybe it's better if I just wait and see what happens there before jumping in, if I'm going to.
Sorry this is your experience - my only small comment on this last part is here: if there's no financial incentive to keep things running then it is very possible that they won't. Money isn't some evil thing we should alternatives to. It's how we value relatively our time and effort vs other people's time and effort.
In a healthier ecosystem with many stakeholders and sponsors this is less of a concern. However, instead of Matrix Foundation gaining more independence, it seems like almost everything moved out of the Matrix Foundation and into a for-profit company. Clearly, things didn't go according to plan.
Most of us just running homeservers are just random people who have no power over this situation and not enough money to make a dent in their budget needs. For us, Matrix was sold as an open E2EE chat system, and we bought in. Does us buying in do anything for Matrix Foundation/Element? I dunno. Maybe it helps sell Matrix to get broader adoption. Either way, we took a gamble on it, potentially instead of other solutions.
Now the free and open source Matrix homeserver has to compete with a closed source commercial one. I sort of hope there's a happy ending on this one, but it sure looks dicey to me right now.
I get the frustration that Dendrite dev has stalled, but the fault here is not of an evil take-over by Element (given Element wrote both Dendrite and Synapse in the first place) but simply neither Element and Matrix Foundation having the $ to do be able to maintain two homeservers simultaneously.
> Now the free and open source Matrix homeserver has to compete with a closed source commercial one
This isn't really the case - normal open source Synapse is the core focus for Element, and the vast majority of work is going into that, including performance work. Synapse Pro is "just" for scalability and HA for big government-scale deployments - https://element.io/blog/scaling-to-millions-of-users-require... etc. But all the schema work etc which supports that goes into FOSS Synapse.
> No image captions
>
> This is silly, but while (official?) bridges support image captions, official Element app does not.
Element Web & X support captions. (Element Web doesn't currently support authoring them, but can display them - obviuosly this is on the todo list though).
Element Classic has basically not been touched in 2.5 years, unfortunately, and so yes - doesn't support them. But at this point the old app is just not being developed; we don't have bandwidth to do both.
> Slow notifications
This sounds like it might be overloaded server problems ftr.
> Element X is Slower
> Somehow, it is slower. Clicking on a conversation takes 0.5-1.0 seconds to load it, compared to almost instant load on Classic.
I think author uses regular synapse install and there is no apt package for element-admin, manual build would be required with no update process.
> Element Web & X support captions - i think this is honestly the worst of both worlds
if app_old has features A,B,C,D and app_new has A,C,D,E it frustrates both groups of users. Which app are you actually supposed to use? Having feature parity with the old app is essential before moving on with newer features. And I am still waiting for that to tell people to switch.
> Sorry, but you have to run an auth server (matrix-authentication-service) if you want Element X to work.
This is a bit outrageous IMO. Actually breaking and deprecating the classic auth and requiring a new server component to keep the only actively supported client (which still can't properly manage keys or sessions on its own, like classic Element can, even as non-verified sessions are being disabled) is a bit rich.
> if i've got any of the above wrong please just say, rather than downvote :D
Maybe it's not necessarily that any of those are wrong per se as that they just don't address several other important issues raised in the article, such as (copied from headings in the article):
* Needs constant cleanup
* Database grows out of control
* Element X is now recommended as the new and better client. It is not.
* Calls are not backward compatible
Also saying the info is "outdated" because a bug was fixed two weeks ago (and you're not even sure if it made it into a built yet!) seems pretty disingenuous. More generally, I think the timeline on which info becomes "outdated" is maybe different for you than for actual users. Actual users expect to install it and have everything work. If there are major bugs, protocol changes, and app-throwaway-and-rewrites happening on a timescale of 2-3 years, that's still too fast for most people to consider their info outdated. If everything were going swimmingly with none of these problems for like 5 years, then yeah, okay, but that's not the situation.
god if there was a comprehensive chat protocol without a loser ceo i'd move to it tomorrow. matrix org staff don't know how to interact with other human beings.
Same setup here since 2017. Since then, RAM usage decreased by 60%. The admin panel is not something I'd need but it would be a nice-to-have. Started with postgres as I wouldn't go for anything else if I wanna use it for decades. It has 2.5GBs for 10users and I don't mind if it takes 10 or 20, that's something I expected. Never did a cleanup of anything, I just dumped the db and moved to OVH recently onto a new VPS with NVMe SSD, it flies.
The fact that I cannot delete attachments that users delete is certainly my biggest irritation, 50GBs of stuff I am not sure if I can or cannot delete, but considering the size, I am just gonna bite the bullet, couple terabytes should not be a problem in 25 years. But this is def something I would love to see addressed sooner rather than later. It must be a pain even for the matrix.org server team.
After moving to a better server I do not have issues with slow notification unless the phone is sleeping for longer period of time which is an android optimization (I'd assume). It is more reliable than teams at this point. One of my friends had issues but removing 15 old devices fixed the issue.
As for element-x, I did call out "the another rewrite" issue especially with android and I do think it makes things worse. I still do not know how am I supposed to fix calling and video between old and new clients. For now I don't bother with new clients and everyone is using old ones, but it starts to become an issue as classic clients are in maintenance only mode
I ran my own matrix server for a number of years, then learned that any matrix homeserver would happily serve up any media from matrix without authentication. I shut it down the next day because I have no desire to operate a proxy for downloading csam.
A url like https://my.domain.com/_matrix/media/<some.other.domain>/<some-bad-media-id> would happily proxy the bad media through my server.
I think they've fixed this, but it's not worth the risk to me.
Yes they fixed that about a year ago. It was a breaking change and required everyone to update their server software config.
Whatever is going on in Matrix land isn't stable enough for most people to switch. I gave up after they broke their calling system after changing something to this livekit system. It doesn't work, my existing TURN server became useless, and Matrix was left as a very slow chat application.
I'm over it.
I don't plan to give up just yet. Switching is way too difficult and I don't see many platforms having both solid desktop and mobile versions with e2ee.
But if they dare rewriting anything again from scratch I am leaving. They MUST stick with what they have and make it good at this point.
None of my friends ever migrated from Discord to my server for over a year, so not much is lost. I'll join more actively maintained instances to stick around the communities and chats I found for myself.
90% of my friends migrated, so I do have a problem on my hands.
https://m.youtube.com/watch?v=z0ULOptq2vk&pp=0gcJCR4Bo7VqN5t...
They seem to plan to rewrite the desktop client next.
I gave up after I upgraded my server from a SQLite backend to a PostgreSQL one with their conversion script which introduced errors into my DB.
Maybe one day I'll dig into it and see if I can fix the DB by extracting whatever data in it that's causing the errors, but like, is it really worth it at this point?
What does running a matrix server get me in 2025?
> What does running a matrix server get me in 2025?
Is there any other FOSS, self-hosted, decentralised platform with E2EE chat & E2EE group VoIP - i.e. the equivalent of Signal, but without depending on a centralised service?
From my pov (which is biased, as Matrix project lead), the downsides are:
* We still expose too much metadata to the server. Work is afoot to fix this, though - e.g. https://youtu.be/Q6NSmptZIS4?t=933 for encrypted state events.
* Synapse is still uses waaaaay too much database. I proposed a solution here: https://youtu.be/D5zAgVYBuGk?t=1852 but it needs to get properly implemented.
* Element's transition from the legacy apps to Element X has not been smooth, causing much of the gnashing of teeth in this thread (e.g. lack of interop between 1:1 voip and group-e2ee voip, or teething problems in the new apps).
* Post Quantum Encryption hasn't landed yet; it's been painful to get funding together for it.
I would say the biggest issue is not any of those things, it's that issues of this magnitude (including many of the ones fixed in the last year) are still being identified and requiring things like a "transition from legacy apps", breaking server changes, etc. In other words, it is not stable.
Haven't been on the main matrix.org server for many years, but they used to have a lot of issues with latency - it wasn't really instant messaging but more like email. When I felt it was the worst, I decided on self hosting it. That improved things significantly - it started to behave like an instant msg platform. Now with OVH migration I am even able to scroll through hundreds of attachments super fast and everything feels snappy.
And I also know there is no online status on the main server. Other than that, no idea
Conduit [1] has retention policies [2] for media and attachments!
[1] https://conduit.rs/ and https://gitlab.com/famedly/conduit [2] https://gitlab.com/famedly/conduit/-/blob/next/docs/configur...
Honestly this doesn't fit my usecase. If I understand it correctly, it is just like any other retention policy meaning it is not related to the fact that the attachment was deleted by user, but it just deletes everything after a certain point.
I don't mind storing things that people can access for many years - if I wanna see that funny picture from 10 years ago, I should be able to find it (even though GUI in Element sucks for that now). But what I do care about are things they uploaded accidentally or that they wanted to delete for whatever reason and it stayed on my server.
>this also creates a situation where anything said across federation cannot be unsaid, which is an ironic situation for a protocol/system that often comes up when talking about privacy.
How is it ironic? No protocol in the world can force anyone to delete anything from their own device. Chat apps that implement this function are either proprietary (so you cannot control what they can do) or, if OSS, do it on a pinky-promise-basis.
> No protocol in the world can force anyone to delete anything from their own device.
But they either do not sign the messages or allow repudiating the signatures. Matrix signs all events forever.
Matrix also makes the entire event history (minus message content depending on room configuration) available to servers on join, even if that server's users are not allowed to see it.
> Matrix also makes the entire event history (minus message content depending on room configuration) available to servers on join
The important bit is the bit in brackets: as you say, historical message content is not shared if the room's is configured not to share history.
These are distinctions without a difference. Events replicated across several independent Matrix servers are not meaningfully different than events broadcast across independent clients in terms of observability or repudiation.
But normally when you join a conversation and are not allowed to see previous messages, you don't see anything about them. A matrix server does.
A protocol can mandate forced deletion. A particular client implementation may ignore it, or some users may circumvent it, so it would be a weaker kind of feature, but still a feature. And depending on circumstances it can be quite useful.
An open protocol can mandate indeed, but that is still in the realm of pinky promise security. A better design for a privacy-friendly chat protocol is to not write a lot of stuff on a lot of different remote servers when that's not necessary IMHO. One of matrix's selling points is to be censorship-proof though; in that case copying stuff as much as possible makes a lot more sense.
>pinky promise security
You are right, though I still prefer "weak feature" as a term :) There's enough value in such things. Cryptography crowd is concentrated on omnipotent Eve breaking ciphers, and that wrench from xkcd, but I dare to claim that majority of both commercial and private leaks happen just because well-intentioned users don't have enough capacity to keep track of all the things, and proverbially think twice. Features like "unsend", or timed deletion are indeed laughable on their purely technical merits, but do wonders saving users from grave mistakes anyway.
It's hard to explain to a non technical user. Something like "We tried to delete the message, but some of the people who received your message might still have a copy." Does not sound great and is going to be hard for a non technical user to understand and hard to implement in a way that a non technical user will find satisfying.
So if I was a dev on matrix/element and this feature came across my plate I would have to weigh it against features that I know can be implemented in a way which make technical and non technical people feel satisfied and better about the application.
That is exactly what happens in WhatsApp though. Maybe the message isn't there anymore but it used to say pretty much exactly that.
A protocol can only support, never mandate. If I send you "DELETE MSG #4829" and you do nothing and reply with "200 OK; DELETE MSG #4829", nobody observing the protocol's messages will ever know what happened. Sure, an omniscent being could say "but he internally broke protocol, he didn't delete the message!", but by definition if something cannot be verified inside the protocol, it is outside of protocol.
Sure.
In practice, in federated networks bad actors end up being blacklisted. It does not provide any "formal" guarantee, but… it tends to work fine enough. For this specific "deletion request" feature, of course it should always be seen as a convenience thing, and absolutely not about security.
As with many engineering things, it's tradeoffs all the way down. For instant messaging, a federated approach, using open protocols, offers what I value most: decentralisation, hackability, autonomy, open source. My options in this space are Matrix or XMPP. I have not attempted to self-host a matrix server, but have been very happy with my [prosody](https://prosody.im/) instance for almost a decade now.
I don't know what's wrong with XMPP other than the network effect collapsed when the GMail chat thing was killed, while the mobile client options were poor for a very long time.
Matrix has the appearance of being a drop in replacement for Slack or Discord, but the design decisions seem so compromised that the only explanation is they did manage to establish a (somewhat weak) network effect? It certainly is not a good look for an open source project to be running on Slack or Discord (free/cheap plans rugpulled or to be soon.) Then that leaves IRC, which has a network effect collapsing at a much slower pace.
I never got far enough to try hosting a matrix server, but reading the linked post -- Matrix definitely is not GDPR compliant. The combination of whatever end form of ChatControl the EU gets along with possibly hundreds of other laws across the world and individual US states makes me think the days of a public facing non-profit or small startup running a project like this are over. (Or maybe the future of open source is funding lawyers while the development is all done for pennies by AI?)
In what way do you think it's not compliant with GDPR?
The GDPR is being neutered anyway because the EU caved in to Trump.
Not being chatcontrol compliant? That's a feature not a bug. Nobody wants that anyway. Just another stupid US lobby (Thorn).
A big organisation won't be able to run matrix for everyone no but that's the cool thing about it. People can run their own for smaller groups of people.
I don't know such definition frankly. And to the best of my knowledge there are plenty of things which people call "protocols" strongly prescribing actions non-verifiable in the very sense you used. That said I'm not here for a terminological discussion. We may call it green cheese, but it's still a useful feature.
Nobody claimed it isn't a useful feature. The only claim I made is that it cannot be mandated with an open protocol, so if you expect 100% adherence in the name of privacy, you're setting yourself up for disappointment.
It can be mandated, it's just that the mandate can't be enforced.
Good, nobody claimed any expectation of 100% adherence as well!
True, and Matrix has the weaker version of the feature: https://spec.matrix.org/v1.16/client-server-api/#redactions It should absolutely work in normal situations across all servers and most all clients.
People should related to anything federated like email. If you send something it is in someone else's computer now. With matrix or any e2ee protocols it is depending on pinky promise of the client to modify it. I thought the whole Snapchat fiasco already taught us that. Did we forget?
There's a difference between "I have an active adversarial actor" as a security model and "sometimes I send something I don't want to and want to delete it, the people watching are friends and acquaintances and are not deliberately preprepared to collect kompromat".
When I delete a message off Signal chat, the expectation is that the chat members are agreeing by social contract to abide by that.
There are many cases in between, though, in particular those where you're using messaging for "serious" purposes like business, but not so serious that the other users are your enemies. Sometimes for instance you want to let people edit/delete recent messages but not really old ones (to prevent someone trying to "rewrite history").
> How is it ironic? No protocol in the world can force anyone to delete anything from their own device.
You may have noticed the constant pushing to remove the user's access to their "own" device.
Forcing people to delete things from their own device is the whole concept of the Snapchat protocol, for example. Snapchat fortunately doesn't offer an OS and can't meaningfully be part of this push, but they make a convenient illustration.
You can check out Snapchat's bug bounty policy here: https://hackerone.com/snapchat . On the list of ineligible vulnerabilities is "screenshot detection avoidance". That's not a bug (because there's nothing they can do about it), even though it is their product's selling point.
Sometimes stronger companies want similar things, and they try to do something about it.
Right, but we did have efforts to take over hardware security enclaves to deliver user data, instead of copyrighted company data, to user devices.
Tim Berners-Lee tries to make the internet a place where you can choose, what it "forgets". At least that were the news I got from the 2010s and early 2020s. As for how: DRM-like tech in the hands of users should allow for that.
So having privacy by design would be nice, and e.g. many messengers try to do "it is inconvenient to copy a message that someone send you that is marked as view-only-once-or-up-to-a-timespan, but of course, you can use an external camera, i.e. make more low-fidelity copies or even exfiltrate data".
Even F/LOS software can use/would be forced to use these proprietary enclaves or at least non-user accessible key stores. (As far as I understand hardware level DRM.)
>Tim Berners-Lee tries to make the internet a place where you can choose, what it "forgets". At least that were the news I got from the 2010s and early 2020s.
Tim Berners-Lee created the web, not the internet, which is what chat apps use. Also, unless you can provide some direct quotes about it being designed for "forgetting" stuff, I have no idea where these "news" you got came from.
>As for how: DRM-like tech in the hands of users should allow for that.
If it's in the hands of the users, i.e. open source, it can be disabled at any moment, which is exactly what my reply already addressed.
I think they're talking about Solid, Tim Berners-Lee's newer venture: https://en.wikipedia.org/wiki/Solid_(web_decentralization_pr...
>> As for how: DRM-like tech in the hands of users should allow for that.
> If it's in the hands of the users, i.e. open source, it can be disabled at any moment, which is exactly what my reply already addressed.
The point is that with the help of hardware-backed DRM on the client, the Matrix server could send data only to unmodified clients. You modified your client in a way that does not match what the Matrix server expects? No data for you.
You're repeating the same thing with more words. If you cannot control your hardware-backed DRM, then Matrix requires proprietary blobs to work.
Yeah I thought this was a weird take too. Too often people take privacy for "I can do what I like". IMO deleting something you've sent to someone else is not a privacy concern at all.
IIRC it is possible to have some clever encryption so that the person you sent your message to can prove to their own satisfaction that it came from you, but they cannot prove to anyone else that it came from you. Which gives you plausible deniability; you can always claim that your contact forged the message.
Can't remember what the algorithm is called.
No particular name. Just deniability. I personally like to call this particular scheme, deniability through claimed forgery. Not particularly clever. You just provide your correspondent with what they need to forge your messages after the end of the session.
I don't know if it actually could work in practice:
https://articles.59.ca/doku.php?id=pgpfan:repudiability
Isn't the scheme simply agreeing in a shared key and both using it? I'll know that the message is from you if it's signed with that key and is not from me and vice versa, but neither of us can prove who created the message.
Off The Record chat did this.
https://en.wikipedia.org/wiki/Off-the-record_messaging
I don't agree with it myself, but there are people who seem to want to frame "the right to be forgotten" as a privacy issue.
Just one example, but trying to get that revenge porn off the web, can be seen as an attempt to restore ones privacy. Where others should not have the right to continue to peek into ones private life.
That's not quite the kind of thing I was talking about. I think that is generally already covered by current laws in most places?
The right-to-be-forgotten advocates argue that everyone should have the right to demand that any trace of their previous online existence be deleted. On social media of course, but also independent web forums, chat logs, git commits, etc.
Even if it were a privacy issue, it would be impossible to enforce it technologically via FOSS software, because, by definition, the user at the other end could run a forked version with remote deletion disabled.
Ran a homeserver for 5 years on a minimal VPS and it worked fine. Upsides - works everywhere, self hosted, feature complete. Client software in the ecosystem mostly felt bloated, with the exception of NeoChat. By 2022 the clients could no longer call each other. Decommissioned it this year in favor of traditional XMPP which works fine and it's nice that notifications are appropriately processed, finally.
Our team highly appreciates the work done in Matrix it's just unfortunate that the elephant in the room was never addressed at the start of the project, which is the need for a -simple- first-party administrative dashboard or tool to manage users, storage, and configuration. Without that core component, then you've got a layer of complexity between an admin and an audit which will increase likelihood of misconfiguration or resource management issues.
We did address the elephant eventually with https://github.com/element-hq/element-admin a few months ago.
In terms of VoIP interop - yes, one of the worst bits of Matrix is that the legacy 1:1 VoIP calling is not interoperable with MatrixRTC-based (multiparty) VoIP calling, but we ran out of time/cash to implement interop and instead focused on making MatrixRTC work well. (Does XMPP give you E2EE multiparty calling ooi?)
Thank you Aaron for the direct response. Glad to see the central interface for administrative use. Haven't had the need to use the calling feature in a while. One of the hurdles with the Rust client on Fdroid was it was too new for the OS on the device it needed to be used on, but on iOS it was a performance improvement.
One of the genius ideas behind Element was the ecosystem of clients services that attach to rooms, and in some cases XMPP does not reach those expectations. I do plan on a redeployment soon after creating a new technical scope for its use.
I like what I can observe of the new admin interface. I hope it can come to include security and configuration check guidances to provide tools for admins of all skillsets to properly configure their servers, and related TURN-or-not and storage.
Matrix is bigtime like XMPP - I don't think it's going anywhere anytime soon. Thankful for all that it has provided to our organizations.
Another cashflow idea, as if you don't think of many yourself: A white-label version of Element (called something else) that can be deployed as scoped to an organization and based on an LTS release schedule. We'd gladly pay for this.
That is Element’s current business model: https://element.io/en/pro-app and https://element.io/server-suite/pro.
Looks excellent, sharp website - and I'll admit I hadn't looked for the white label app before talking shit. That offering looks excellent - I see an opportunity for a white label tier between community and enterprise for small/medium orgs with pricing listed on the website.
Well, I can't check if it is still a thing today, but Element's managed servers used to allow you to do that, at least for the web app (though I had the idea they also did that for the old mobile apps?)
Here's a couple of links:
https://element.io/blog/custom-branding/
https://element.io/blog/a-white-label-messaging-app-to-creat...
They've been on their game with this product, which hadn't been on my radar until Aaron's response above.
Thanks for mentioning XMPP. I never ran a server, but as a user I've been enjoying it. Also has its problems - for example, I picked some server to create my first account, and the other day the server disappeared for 6 hours. What do normal users (i.e. non-newbies like me) do in practice in these situations?
That said, I would love to hear your experience running an XMPP server. Do you still run it?
Trusting your chat activity and availability to any XMPP server on the internet that allows free public registration is not ideal. If you want reliability then pay for it or use what you perceive as a highly credible free server. I do not run any public facing XMPP services so I do not have any input on that and as with Matrix I wouldn't roll your own unless you know exactly what you're doing due to inherit risks associated with these platforms in certain states of configuration or exposure.
I want to give credit, and that is the early rust iOS version of Element X I tried was very slick and quick, but I didn't try it for long.
As a former user I felt these pain points trying to do nothing more than have a very active one-on-one chat with a good friend. Tens of messages an hour, maybe 2 years running. Using matrix.org and the pre-X clients. It's fine for group chat (IRC style) but that's not a high bar.
(a) the encryption between using a mobile and the webapp desyncs/breaks all the time, it just sucks. I mean you'll get "cannot decrypt" a lot, have to bounce back and forth and generally try and force it to re-sync properly again. Sometimes never worked at all. Lots of issues on GH over the years.
(b) as mentioned in this article, insane delays on new message notif and sending and receiving. Just logging in on the webapp every morning took minutes of some sort of mysterious sync process, often the mobile app had the same problems. The X stuff may fix this, we were pre-X.
(c) cleanup. There's no message retention set on matrix.org, when I wanted to extract and remove our past chats the process and experience was excruciatingly bad. It took tens of hours over several weekends of the webapp (mobile completely non-op in practice for this) polling and loading old content, just so I could select 100 at a time to delete and then it took an hour. Once I started culling back over a year or so, the loading got longer and longer and longer, until eventually it 100% stopped working at all to load old messages.
Signal and DeltaChat are far, far better experiences for one-on-one chats with friends & family. The Delta client is a bit UI/UX behind but not horrible; e.g. you can't correct a typo in a sent message in Delta, unlike Signal - because each msg is a unique gpg-encrypted "email" rather than a database object that can be re-manipulated.
Thanks for mentioning Delta - https://delta.chat/ and implicitly https://chatmail.at/
> you can't correct a typo in a sent message in Delta
At least on the iOS app you can, just tested it. I run my own postfix/dovecot, so shouldn’t require any esoteric configurations.
Good to know, thank you - those of us currently using it are on Androids so we've not had a friend on iOS join yet. We're using a chatmail relay.
a) is really not a big issue any more
b) yeah, X solves it (via sliding sync)
a) As of when? I had a "cannot decrypt" room failure on matrix.org a year ago.
b) Unfortunately, X breaks other important things, like audio/video calls. It currently feels like an alpha-quality release: buggy and lots of missing features. Not ready for widespread use.
I felt bad making a long thread once I opened Element X and saw it didn't have support for threads.
Someone let me know later that threads are hidden behind a Labs setting, but it only allows the client to reply to threads, but still exposes the entire thread inline for the channel which sucks up all the air in the chat.
this is wrong - the threads support in labs lets you create threads and navigate into them as you expect.
what you are describing is the old threads workaround which predates the proper solution in labs which should exit labs asap.
I am describing what I see on Element X on iOS today. If someone replies to threads, it is not hidden and navigable on this client. Even after switching the labs feature on, the app was showing the thread interspersed with the rest of the conversation.
Uninstalling the app, reinstalling, and making sure the labs option is turned on before navigating to a room with threads, is behaving how I expect.
So maybe, its a bug that the room does not re-render after toggling that setting.
ah right - that sounds very plausible; sorry for the confusion. have filed it as a likely bug at https://github.com/element-hq/element-x-ios/issues/4810 - thanks for the report :)
I'm sorry you had a crap time, and agreed that in the past things were not great. But in defence of the Matrix team, we fixed almost all known encryption in ~Sept 2024, and the new generation of clients fixes the slow sync issues.
In terms of message retention: https://element-hq.github.io/synapse/latest/message_retentio... is how you should have/could have cleaned up those rooms. (It's not exposed in Element's UI yet as we've been prioritising more fundamental stuff).
Hi Arathorn, I recognize your name as the core developer lead. :) Alas, when I was trying to cull my old messages I tried that (and a lot of other API type hacks I found floating around) and none of them, at that time, worked.
In short: the complete lack of user-facing easy to use controls for data retention and culling in the Matrix (Element) clients is a deal killer for me. That painful experience taught me a lesson - now when I test something new, one of the first things I look for is "how do I extract from this thing?". I never want to go through my Matrix extraction pain again, so a personal life lesson was learned.
> The only thing that I don't really understand is the decision on data replication. If a user on server A joins a room on server B, recent room data is copied from server B to server A and then kept in sync on both servers.
The idea here is that rooms are abstracted from servers and sort-of exist ephemerally. This has the advantage/disadvantage of making it hard for the underlying infrastructure to exert control over the hosted communities, and seems to have become a distinguishing feature of federation.
My experience of Matrix as a possible replacement for Discord has led me to believe it's mostly a disadvantage since it leads to gross misalignments between the communities in top and the infrastructure providers underneath. I consider e.g. Discourse to be much healthier (although I would like to see an app for Discourse so that my Discourse communities behave more like Discord/Slack servers) and it's frustrating to me that there hasn't been a clear "Discourse for chat" emerge to replace Discord.
> it leads to gross misalignments between the communities in top and the infrastructure providers underneath
Yeah, this is a great way to put it. We see this a lot with the mod experience. So much of the mod tooling is "run this bot on your server" or "configure synapse like this". But those kinds of things are inaccessible to people who aren't able to run any server of any kind. Of course someone else can run the mod bot for them, but then you still have the same problem where it introduces a layer of friction between the person who needs to perform mod actions and the person who controls the mechanism for doing so.
Until there's a robust and full-featured mod toolkit built in at the protocol/client level, it's a dicey situation for people who want to use Matrix to host a community. The insidious part is that everything may seem fine until suddenly your room is flooded with images of gore or worse.
(By the way, we still miss you in the Python room on Matrix! :-)
You could also try Movim. You could have a decentralized XMPP server with a client that support group calls as well as having posts like a forum folks can comment on.
There are a bunch of options out there (though I've never seen Movim - thanks, I'll check it out) but most communities seem either to be on Discord or Matrix (with a few still hanging on to IRC and a few others on Slack) - Discord being by far the best UX of the lot.
> The idea here is that rooms are abstracted from servers and sort-of exist ephemerally
No, that's not even remotely true. In fact the opposite is true. The domain name of the server used to create the room is perpetually and permanently embedded in the room name and can't be changed, ever.
That doesn't have any practical effect, though. It's just part of the name and the room can still be used even if that server disappears forever.
No, not at all. Posession of a CA-signed TLS certificate for that domain, and the corresponding private key, is basically the root of trust for room operations.
I think their point is that the room doesn't depend on that server's continued availability.
I've been using Matrix for several years as a user. It works great. The problems decrypting messages have gone. X is becoming a good client. I'm deleting my whatsapp and télégram accounts in a few weeks after a painful week-long backup...
Edit : I wonder how easy it is to backup a Matrix accounts's data. Conversations and files.
> I wonder how easy it is to backup a Matrix accounts's data. Conversations and files.
Not as easy as you might hope. The Element client has an export feature, but you have to manually activate it on each room/chat, and the export has a size cap so it may not work if you have lots of files you want to save. It's also pretty slow if the room has a lot of history. You could also try using something like Matrix Commander (a command-line client), but I couldn't get that to work fully either.
I’ve been running a Matrix server for about two years on a Proxmox host in a colo I rent for the purpose (plus some other hobby stuff, but mainly because I just think it’s cool). This playbook is awesome and it’s pretty easy to set up and keep running: https://github.com/spantaleev/matrix-docker-ansible-deploy
Regarding the "Requires federation" section, that is not true. I've been running a small family-only homeserver for several years now, and had federation disabled on it from the very beginning, and there have been exactly zero issues related to (lack of) federation with it.
Same here, though you do still have to expose it to the internet unless you use a VPN. I'd prefer something with less of an attack service especially because bridges don't currently encrypt, but I host behind vpn now.
That's not a fair criticism. Of course you need to expose the server to the network clients are going to be connecting from. Whether that is Internet or just a LAN, that's orthogonal to the protocol.
I've also had an intra-company Matrix server running completely on a company internal LAN, with no Internet access, so there is no inherent need for it to be on The Internet.
Yes but even for a web service the attack surface is pretty big. Lots of obscure URLs and the code isn't that clean either.
I've installed fresh matrix about 8 times or so, got it to work 3 of those times. I started looking for how to do a fresh install earlier this year (matrix/synapse element) and I have no idea which docs to use,
It was bad enough before that I had to detour to random places to find workarounds for things like how nginx does it's thing differently than the matrix docs I was reading at the time.. but now with the newer element that makes it easy to not find a self hosting way - to main docs being listed as archived or old -
I'd enjoy installing it fresh with someone from matrix watching the process just to see what a slog it is - and then watch someone try to get some addins running like admin and moderation.
wordpress is a one click install via softalicious and similar on most cpanel 10$ and up hosts.. you can one click add a decent free chat plugin if you are on a VPS.
If matrix was that simple to setup, lots of people would be using it.
https://github.com/element-hq/ess-helm?tab=readme-ov-file#el... is meant to make it pretty easy. I sped-run an install from scratch in about 90 seconds during the Element keynote for the Matrix conference this year: https://youtu.be/TZgcdgv2NXk?list=PLl5dnxRMP1hUgnYEbpEsEEhIq...
Clearly we need to do a better job of pointing folks there.
> While technically, Synapse can work with a sqlite database (and which at first seems like an OK choice for having <10 users on the server), it WILL become corrupted.
Does anyone have any more information on this? Running Postgres is not a big deal, but I would expect SQLite to be fine given how well it works in my experience.
I've not seen many corrupted sqlite databases, but because Synapse is not currently storage-efficient you can end up with some terrifying large (100GB+) sqlite databases which might as well be corrupted.
We only ever supported sqlite for ease of tinkering; it was never intended to be used in production, and in retrospect supporting it at all was a mistake.
In terms of synapse storage efficiency and how to improve it, folks may be interested in https://www.youtube.com/watch?v=D5zAgVYBuGk&t=1851s
I use sqlite for my 2-user server and haven't had any issues in the several years I've been running it.
It's possible to corrupt a sqlite database file, but generally it shouldn't happen unless you're doing something weird with it. https://www.sqlite.org/howtocorrupt.html
I've been running unfederated Matrix instance with sqlite for 10 years for 10-20 people some very active. In WAL mode sqlite is fine i am not sure why it would get corrupted.
Few years ago we did a data/history wipe (i managed to migrate accounts) because we switched to conduit.rs with sqlite. I very much prefer conduit to synapse.
I never had issue with sqlite it was mostly Synapse. With recent changes i think they are completely dropping the ball on small instances and it feels like the ecosystem is splitting. Element X is just not very good. It seems to exist to kill classic element and since for element x you need some server features that not all alternative synapse servers support.
> While technically, Synapse can work with a sqlite database (and which at first seems like an OK choice for having <10 users on the server), it WILL become corrupted
I want to hear more about this. Is this because Synapse’s SQLite support is half-baked? What sort of corruption are we taking about?
I use sqlite for synapse (since ~2 years) and I haven't noticed any corruption at all. I am in lot of rooms (20-30) with 1k-2k users in a lot of them and the db size is 8.8G currently
I've tried off and on to actually use Matrix. I was a bit of a loud supporter in the early days. Unfortunately, it looks like it still hasn't grown past the fundamental issues I was having then. It might be time to try something else
As someone who has looked into forking Matrix for a new type of chat service, I'm grateful to see a more in-depth look at running it behind the scenes. Thank you.
I've been developing open source code for over 25 years. i've deployed hugely complicated systems like hadoop. I've never seen anything as hard to run as this + the bridges.
OT: I have some very big groups in Telegram (7 years or more, with a lot of pictures). Can Matrix (Rocketchat or alternatives) have similar storage features (with some migration scripts)? Thanks
To add another data point, I've been hosting a (tiny) matrix server for a few months. I'm pretty comfortable with self-hosting using docker, so I opted not to use the ansible scripts in the hope that it'd keep my setup simpler and more maintainable. Somehow I didn't find any mentions of ESS until Synapse was already up and running, but Kubernetes would have been a dealbreaker for similar reasons.
In this short time I've run a database migration (sqlite is the default, but MAS requires postgres), tried and failed to migrate to MAS (required to use Element X) and have lost a couple of days messing around with coturn and eturnal with nothing to show for it -- my calls still don't connect when NAT is involved. I have to tell new users to ignore the recommendations to install Element X until I get MAS working.
There's a lot of room for foundational improvements here, even updating docs to point would-be server admins to the recommended setup du jour would help.
TLDR Self-hosting isn’t dying because people stopped caring. It’s dying because the complexity has gotten out of hand.
This post highlights how something that used to be a fun, lightweight hobby has turned into a full-time maintenance burden. Systems like Matrix are powerful, but they’ve become so intricate that even skilled engineers struggle to run them reliably. The result is a slow drift back toward centralized platforms, not out of preference, but because convenience keeps winning over autonomy. It’s a reminder of the growing gap between the ideal of a user-owned internet and the realities of modern software.
I would say it became much easier in recent years. docker-compose became defacto server "app install", any linux supports it. That includes GUI options like Truenas/Unraid and very nice admins like Dockge exist.
The company behind matrix is aiming at huge scale servers but if you care about unfederated private instances you will find there are few much simpler "one binary" projects that can even use file based sqlite/rocksdb. Hosting those couldn't be simpler. You actually don't even need docker just systemd service and switch binary when updating.
If you want a simple docker-compose for Matrix, you can always use https://github.com/element-hq/element-docker-demo.
The reason Element ships ESS Community as kubernetes helm charts is simply so that the officially supported distro is as close as possible to the thing we have to support for the big govtech customers, where docker compose simply isn't going to cut it. Perversely, that means that ESS Community is pretty bulletproof (even if the scaling stuff is missing), but comes at the expense of having to run k8s.
Yeah, I think this is true and it's unfortunate. I think it's one of the areas where what seemed like the plan for Matrix hasn't totally worked out in practice. And it's even just the intricacy of managing it; some of the problems are deeper in the design. When I started using it several years ago, it seemed their vision was a lot of people running a lot of small servers. But the full-replication nature of the protocol, plus the resource demands of the server software, make that kind of impractical. Even tech-oriented people may shy away if they find out the database could grow to tens of gigabytes, and Synapse is not exactly light on RAM or CPU either.
Perhaps in the future if implementations improve some of this may get better, and it will become more feasible for small operators to run their own servers. But by that time it will be harder to build trust because too many people will have written it off as bloated or unstable. I think it would have been better to start lean and keep the system in more of a nerd niche until that process of evolution reached a later stage.
NixOS makes this rather easy too. There exist modules for synapse, livekit, etc (MAS module is still a Pull Request, though) and the setup is quite doable: https://wiki.nixos.org/wiki/Matrix
However, you still need to know what you are doing (the manual helps) and connect the pieces (in theory there could be a nixpkgs module that does this for you but apparently nobody did bother). Once its done you can lean back.
I've been running my homeserver happily for > 5 years and it was fairly straightforward.
Yeah, but then you have to run NixOS, which besides the sheer difficulty of being Linux, for the average user, stacks Nix on top, which comes with its own learning curve.
Also, based on your description, you're a techie. I highly doubt the average city hall clerk will set up their on Matrix server.
I've been running synapse on a small VPS for the past few years. I got some of the bridges working too. Def bumps along the way, but its still the daily driver for me and about five other people.
Recently I spun up the new ESS Community Edition on a new VPS. Much easier to get up and running. I was delightfully surprised. although since that one uses kubernetes and other things I'm not familiar with getting bridges and other things I've become used to is going to require more learning on my part. Since ESS is so new, not a lot of newbie friendly howtos yet out there.
I remain optimistic.
And I thought that XMPP felt broken...
XMPP felt great when compared to Matrix. Matrix was in a bad state some years ago when I hosted it for a while, and seems like it still is the same messy state. I avoid it as much as possible but for some reason there are communities using it.
At this point it should just die so people would be motivated to replace it with something better.
> I avoid it as much as possible but for some reason there are communities using it.
The "for some reason", at least for my group, is because it's rock solid for us. We have no reason to want to change, so of course we are using it.
People are motivated to replace it with something better, it's called Matrix 2.0.
Which perfectly fits the Matrix ethos of everything constantly being rewritten or replaced with something new that only implements half the features before the original thing is ever finished.
https://www.jwz.org/doc/cadt.html
Copy-paste the link, otherwise the Easter egg might be offensive to some :-D
I like this more than sticking to things you know you won't be happy with.
Do they plan to fix the fundamental issues of Matrix in 2.0 or should I wait for 3.0?
you could be less vague
I could, that's why the link has a few more words about the issues with matrix.
Great, even less vague nw - will get you an answer, not.
I tried it a year ago when it was introduced, it did not work. Is it stable now?
Matrix 2.0?
Not quite yet, but it was promised that the next release will be Matrix 2.0
I guess it was the Element X client that was advertised as Matrix 2.0 client and that did not work.
It does (now), been using it daily for months now
I've had only one issue with ejabberd (didn't read the docs close enough and had to move from mnesia to sql when I hit the size limit) in over a decade of self-hosting it.
Knowing which XEPs you actually want is a bit of a pain, but has gotten easier. Now I usually just periodically look at the current prosody example[1] which lists extensions by "essential" "recommended" "nice to have" and "other" and it's easy enough to find the matching modules for ejabberd.
1: https://prosody.im/doc/example_config
Prosody dev here. Good to know this is useful :)
One of my favourite small features we added in the latest release is a 'prosodyctl check features' command which will validate that your configuration is up to date with current best practices in various ways.
Although we like to curate the default configuration with care, people tend to keep their existing configuration file when they upgrade (we generally recommend this, as we aim for backwards compatibility). The new command makes it easier for folk who upgrade to ensure they are still getting the expected experience as the recommendations evolve.
Obviously that has to be somewhat opinionated. One thing many people appreciate about Prosody is that you can customize it however you want to, even if it's not following the mainstream. E.g. if you really don't care about synchronizing to multiple devices, you can totally disable that message cache. Don't need audio/video calls? Fine!
Now you can feel twice as broken. That is what new, modern standards deliver. Is this part of XCD #927 or another one, too?
XMPP is 26 years old, Matrix is 11 years old.
I have been hosting synapse for 2 years now and it's been a smooth sail. I don't recall having any major breaking changes, most updates are smooth. Element client itself is definitely PITA but it's getting better
> Synapse is the only choice that supports bridges
This is not true, at least today. Continuwuity, which is an alternative server implementation, and its predecessors support bridges very well.
I set up Whatsapp and Telegram bridges with Tuwunel today. The config file for mautrix-whatsapp indicates Dendrite should work as well so it seems just about every server implementation supports bridges?
Former Dendrite user: Yes and no. Mautrix stuff does work with Dendrite, technically, but only partially. For Mautrix-Discord, I was able to get DMs mostly working, but only DMs: bridging servers just never really worked at all. I'm honestly not 100% sure why this is the case, but it's not worth worrying about Dendrite anymore. It is a dead-end server that never worked very well. Up until shortly before it was discontinued, none of the Mautrix bridges worked well enough to actually use.
Been selfhosting synapse for about 1.5 years in a docker compose setup using bunkerweb (formerly "bunkerized nginx", which better explains it premise) reverse proxy, eturnal for TURN and postgres, also recently added livekit and MAS for element call and element X compatibility. All that runs on a small 2vcore/4gb VPS, and it runs pretty good, I experience a server crash every half a month, but that may be caused by the fact that bunkerweb isn't the most lightweight solution (they actually require 8GB RAM minimum, so I'm already beneath the limit), and also because I run some other software (mailserver, ebook server, plex, etc..).
My experience as a administrator has been pretty good, perhaps it's because from the beginning I was optimistic, it suited my needs as I wanted a selfhosted, modern and fairly convenient communication platform. From what I recall, most problems during configuration were caused mostly by bunkerweb (or rather my inability to correctly set it up to proxy requests correctly and not hijack the 4xx and 5xx HTTP codes). Synapse itself has been a pleasure to maintain, but also bear in mind that I did not tinker with with it, I basically set it up and let it run for about a year and then added MAS and livekit.
Yeah, disk usage sucks, for about 5-10 active users and 1.5 year usage my postgres "schemas" folder clocks at 10Gibs. It doesn't include the media_store catalog where synapse keeps media (images, videos). The homeserver is federated and I joined a couple of big rooms in the past. Mechanics mentioned in the links below do help though:
https://matrix-org.github.io/synapse/v1.40/admin_api/purge_h...
https://github.com/matrix-org/rust-synapse-compress-state
Clientwise also sucks, but I think enough has already been said on this matter. But it's good enough to keep my nontechnical friends using it. They do hate it, but not enough to kick me in the arse. Would love to say that this proves that element clientside is usable, but I also have to admit that my friends are just hella good guys who would even write pigeon mail to me if I stopped using anything else for communication :) for me as a techie, element is obviously alright. Clunky, but works. I think clients simply need more time.
What irritates me is the Matrix authentication service (MAS), it's kind of a separate service for matrix homeservers that handles logins specifficaly. You can't use element X without it. However when it's enabled, you cannot log in from your client, instead a web browser opens and shows the login panel where you have to authorize, and then it should return to the client. Except in my case it simply doesn't :( I observed that for some reason chromium based browsers won't redirect back to the element app, and it doesn't know that the authorization has been granted. I managed to bypass it by copying the URL and opening it on firefox, but in one instance even that didn't work.
But other than that MAS problems everything has been fine from administration standpoint. I think it simply needs more time, as it already has traction, I see that a lot of new projects seem to include a matrix room in their social/communication channels, frequently it's the only option besides the bugtracker. And I'm willing to wait patiently :)
edit: added links for people who also struggle with disk space usage
> it runs pretty good, I experience a server crash every half a month
I don't... what? How can you... arrrgh!
Ran synapse for a few months, figured out all the tui clients are either abandonware or broken (originally thought i could use bitlbee, and did the install before realizing it was unusable).
Looked at current tui offerings now some years later, situation seems to be unchanged, the only client that ran back then was gomuks, and that has received a rewrite that hasn't reached feature parity yet.
I am probably the type of person referred to in the last part of xkcd 1782.
iamb is pretty good and not abandonware, fwiw: https://github.com/ulyssa/iamb
Wouldn't this just be called hosting?
I've mentioned this here before, but it bears repeating. A couple years ago or so, I made the catastrophic choice to use Dendrite as my homeserver software. It seemed like a safe bet: it was supposedly lighter weight than Synapse, being written in Go instead of Python and with everything reengineered from the ground up. It didn't support everything, but nothing in the disclaimers made it seem like it was about to abruptly become defunded and essentially unmaintained. Alas, that's exactly what happened not too long after I made that choice. Despite showing no interest in maintaining it, New Vector still found it necessary to relicense their abandonware under the more restrictive AGPL license. Good priorities. Then, when a security patch was needed, a new release was rushed out that included not just a security fix, but also a bug that caused Dendrite to completely stop processing messages for minutes at a time multiple times a day. (This only got fixed months later by a volunteer.) Joining large federated rooms on Dendrite took so long that I thought it was just broken; it could take hours to days to complete the operation! There was even a brief period when Element actually didn't even support Dendrite, leaving everyone locked out. Dendrite has never supported Element X, and the old sliding sync proxy was never updated to support the new simplified sliding sync either, which means you're stuck with the old slow sync and no support for things that require it. Also, most appservers still don't work right on Dendrite either. I got Mautrix-Discord working, but only for DMs.
I legitimately could go on and I'm sure I've forgotten things. It's amazing how quickly my experience with Dendrite went from pretty good to nightmare.
I realized that nobody in charge at the Matrix Foundation or New Vector really cared enough about leaving people stranded on a completely broken server to actually do anything about it (and trust me, I'm not alone. In every single federated room I've ever been in, I've always seen hostnames with dendrite subdomains. I could see them pass by in the logs while joining servers was taking hours.) I honestly considered just leaving the Matrix ecosystem, but I wasn't alone on my home server, so I decided to do my best to fix the problem. I wrote a tool that attempts to migrate the data from Dendrite to Synapse. This is a complicated operation that really took a huge amount of effort to get working, but after a couple of months of failed attempts, I had a test where I was able to seamlessly perform the migration and have clients continue to work and stay in federated rooms. So after getting it "close enough", I went ahead and gave the migration a shot in production and of course, it didn't work very well. All of the user accounts were intact, but a lot of stuff was broken. People indeed stayed in federated rooms, but my room state migration was definitely not 100% correct. Despite this, though, after manually cleaning up the database a bit more, hackishly while live, it was mostly a success. I believe I am probably the first person to directly move from Dendrite to Synapse.
So now that I am on Synapse, have my thoughts on Matrix changed? Yes. It's significantly better using Synapse, without question. The ecosystem is still a mess, but everything about Synapse is less broken than Dendrite. There are so many features Dendrite just doesn't do, like URL previews.
Why not contribute to Dendrite? Honestly, I don't want to. Their CLA sucks and they're not going to change it for me, and I don't think they're really going to spend time reviewing PRs given the circumstances. If I'm going to contribute to a project without retaining my rights I'd prefer to be on payroll. That's not something you should get from a community member. Either change the CLA to guarantee the project must stay open, or don't expect any free contributions.
Why not post my migration tool? Well honestly, for starters, it's not a very high quality tool. I could probably do some good for the Matrix ecosystem if I could get this tool in much better shape and have it migrating complex room states correctly, but I don't even know if I want to help anyways. This should've never been my problem. I will fully admit that it was my bad choice that got me here, but I really think it can be forgiven: nothing I saw suggested to me that Dendrite was on the way out. On the contrary, everything suggested it was the future, and just simply not ready for large scale usage yet. I'm bitter. I spent a lot of hours on this problem and I feel like hours spent on the Matrix ecosystem won't be repaid.
I hate to be this cynical, but it's just how it is. It's a mess. I didn't bother going into the other messes that still exist when using Synapse, like the seemingly many different ways that VoIP can work in Element and Element X, and the fact that Element X seems to only support a newer VoIP protocol that Element on desktop does not. (Surprise! There is no Element X on desktop...)
Matrix has some other downsides, that I think are tolerable but definitely make me a bit bummed. It leaks quite a lot of metadata to the homeservers, which is kind of alright, but I do think it's a bit sad; even room names are not encrypted, clearly it would be possible to do better. The ecosystem of clients is sad; Element is the only one that is feature complete and while I think it has improved quite a lot I still would prefer a native application over a web view. (You kind of need a webview if you want feature parity though, since group A/V in Element desktop seems to just use a Jitsi iframe...)
The upside is that it is federated and at least messages and files are E2EE in DMs and optionally for groups. I do like that.
Personally though the federation thing feels a bit off to me. I know it's a pipe dream, but it just feels like 1-on-1 and small group DMs should be roughly peer-to-peer. Servers should be for chatrooms and relays. The problems I see are mobile notifications, offline messaging, and discovery. I have wondered if a model like AT proto could get you there for DMs. I would like to try to prototype something some day, but I know that at this point the XKCD 927 count for IM software is pretty insane, so if I'm going to throw my hat into the ring it better be worth it.
Maybe some day I will be less bitter. I mean, Matrix is free, how much can I really be angry if it didn't work how I wanted it to. But, it's hard. I tried to buy in hard, and wound up making a lot of trouble for me and a small group of people that I inadvertently looped into my own mess. I trusted Matrix because it seemed to be the leading option, but definitely I will now be much, much more careful before adopting an ecosystem into my life and the life of people around me. For example, I still have no ActivityPub server... Maybe it's better if I just wait and see what happens there before jumping in, if I'm going to.
This doesn't really excuse it, but to give you some idea of why Element haven't spent any time on dendrite right now, here's a sample of active servers on the federation as of today:
To add on to that, none of their customers use it, and there's no real community around it, unlike the independent Conduit-family servers that make up that remaining 11%.Dendrite was a bet for two things: Could they make synapse faster? Yes. Synapse is faster now, and Synapse Pro is apparently even faster. And, can we make Matrix peer to peer? They ran out of money, and didn't have any customers who would fund it themselves. They're left with this project that is idling without a community, a customer, or a reason to exist.
Apparently, they have some funding to do work on the foundational parts of p2p now, but that will take a long time, and Dendrite is unlikely to be a part of that for a while, possibly at all (the Rust ecosystem seems to be where Element invested their time on the client, while Beeper invested in Go).
In the end, a lot of Element's pain is because of ambitious technical decisions made without a way to back them up practically or business wise. Things would be amazing if everyone working on Matrix had infinite time and money. Unfortunately, they don't, and Element is eating the consequences for acting like they did for a while. They seem to be better now, though.
I honestly don't blame them that Dendrite didn't work out, I blame them for leaving us all stranded without any sort of migration path. I don't personally care if my homeserver is written in Python or Go or Rust, at least not very much. Rust may very well be a better choice than Go in the long run. I only chose Dendrite because when I chose it I felt like it was in better shape than Conduit (which at the time wasn't forked off, wasn't well-supported by app services either, and didn't handle federated presence) and lighter weight than Synapse.
Is my homeserver even counted in the 369 active Dendrite servers anyway? I mean, I guess it definitely isn't in that count since it is now an active Synapse server instead, as of late November. However for what it's worth, I don't think I actually connect to Matrix.org at all. Maybe it still counts if you're in a room with Matrix.org users?
But assuming it is basically just 369 Dendrite servers, well, they're all stuck there. How hard could it have been to provide a migration path? I dunno. My attempt is definitely incomplete and possibly quite ill-advised. I truthfully don't understand Dendrite or Synapse well enough to design a migration tool that works correctly. Did I calculate forward extremities correctly? Maybe. Are my state groups actually all correct? Probably not. How the hell did I end up with rooms that have multiple auth events? I have no idea, I just wound up destroying the corrupted rooms I found. I can only imagine that whoever developed the Dendrite database structure also had knowledge of how the Synapse database structure works and would've been able to do all of this in many less hours than I was.
After all is said and done I'm sure the balance sheets will look much better but the damage done to the community side of things will linger. I won't feel bad if Element fails and Matrix winds up replaced by a different standard later.
FTR the server stats are from https://matrixrooms.info/stats. It's not a full view of the network but it's a reasonable sampling, based on servers discovered through public room aliases and servers that use it as a notary (trusted key server).
> Maybe some day I will be less bitter. I mean, Matrix is free, how much can I really be angry if it didn't work how I wanted it to. But, it's hard. I tried to buy in hard, and wound up making a lot of trouble for me and a small group of people that I inadvertently looped into my own mess. I trusted Matrix because it seemed to be the leading option, but definitely I will now be much, much more careful before adopting an ecosystem into my life and the life of people around me. For example, I still have no ActivityPub server... Maybe it's better if I just wait and see what happens there before jumping in, if I'm going to.
Sorry this is your experience - my only small comment on this last part is here: if there's no financial incentive to keep things running then it is very possible that they won't. Money isn't some evil thing we should alternatives to. It's how we value relatively our time and effort vs other people's time and effort.
In a healthier ecosystem with many stakeholders and sponsors this is less of a concern. However, instead of Matrix Foundation gaining more independence, it seems like almost everything moved out of the Matrix Foundation and into a for-profit company. Clearly, things didn't go according to plan.
Most of us just running homeservers are just random people who have no power over this situation and not enough money to make a dent in their budget needs. For us, Matrix was sold as an open E2EE chat system, and we bought in. Does us buying in do anything for Matrix Foundation/Element? I dunno. Maybe it helps sell Matrix to get broader adoption. Either way, we took a gamble on it, potentially instead of other solutions.
Now the free and open source Matrix homeserver has to compete with a closed source commercial one. I sort of hope there's a happy ending on this one, but it sure looks dicey to me right now.
I get the frustration that Dendrite dev has stalled, but the fault here is not of an evil take-over by Element (given Element wrote both Dendrite and Synapse in the first place) but simply neither Element and Matrix Foundation having the $ to do be able to maintain two homeservers simultaneously.
> Now the free and open source Matrix homeserver has to compete with a closed source commercial one
This isn't really the case - normal open source Synapse is the core focus for Element, and the vast majority of work is going into that, including performance work. Synapse Pro is "just" for scalability and HA for big government-scale deployments - https://element.io/blog/scaling-to-millions-of-users-require... etc. But all the schema work etc which supports that goes into FOSS Synapse.
[dead]
There's a bunch of outdated info in here, unfortunately:
> Does not have an admin panel
The admin panel is at https://github.com/element-hq/element-admin (but it's relatively new, so many folks haven't noticed it exists)
> No image captions > > This is silly, but while (official?) bridges support image captions, official Element app does not.
Element Web & X support captions. (Element Web doesn't currently support authoring them, but can display them - obviuosly this is on the todo list though).
Element Classic has basically not been touched in 2.5 years, unfortunately, and so yes - doesn't support them. But at this point the old app is just not being developed; we don't have bandwidth to do both.
> Slow notifications
This sounds like it might be overloaded server problems ftr.
> Element X is Slower > Somehow, it is slower. Clicking on a conversation takes 0.5-1.0 seconds to load it, compared to almost instant load on Classic.
This was an Android specific bug which was fixed a few weeks ago; EXA should now be instant as it should be, at least on nightlies: it was stuff surrounding https://github.com/matrix-org/matrix-rust-sdk/pull/5841 and https://github.com/matrix-org/matrix-rust-sdk/pull/5854. Unsure if the fix has made it into a build yet.
EDIT: the fix shipped in stable Element X Android a few weeks ago.
> Conversations are sorted by... who knows. It is not recent nor alphabetical.
It should be by recency.
> Onboarding is bad
Sorry, but you have to run an auth server (matrix-authentication-service) if you want Element X to work.
EDIT: if i've got any of the above wrong please just say, rather than downvote :D
> The admin panel is at https://github.com/element-hq/element-admin
I think author uses regular synapse install and there is no apt package for element-admin, manual build would be required with no update process.
> Element Web & X support captions - i think this is honestly the worst of both worlds
if app_old has features A,B,C,D and app_new has A,C,D,E it frustrates both groups of users. Which app are you actually supposed to use? Having feature parity with the old app is essential before moving on with newer features. And I am still waiting for that to tell people to switch.
You claim:
> There's a bunch of outdated info in here
Then link to a 3 week old issue and a repo that didn't meaningfully exist until two months ago? Wow.
If you consider that to be outdated, then that reinforces the OP's point about Matrix being difficult to host.
> at this point the old app is just not being developed; we don't have bandwidth to do both.
You're wording that as if it's a good excuse for providing a poor user experience. It's not.
> > Onboarding is bad
> Sorry, but you have to run an auth server (matrix-authentication-service) if you want Element X to work.
This is a bit outrageous IMO. Actually breaking and deprecating the classic auth and requiring a new server component to keep the only actively supported client (which still can't properly manage keys or sessions on its own, like classic Element can, even as non-verified sessions are being disabled) is a bit rich.
> if i've got any of the above wrong please just say, rather than downvote :D
Maybe it's not necessarily that any of those are wrong per se as that they just don't address several other important issues raised in the article, such as (copied from headings in the article):
* Needs constant cleanup
* Database grows out of control
* Element X is now recommended as the new and better client. It is not.
* Calls are not backward compatible
Also saying the info is "outdated" because a bug was fixed two weeks ago (and you're not even sure if it made it into a built yet!) seems pretty disingenuous. More generally, I think the timeline on which info becomes "outdated" is maybe different for you than for actual users. Actual users expect to install it and have everything work. If there are major bugs, protocol changes, and app-throwaway-and-rewrites happening on a timescale of 2-3 years, that's still too fast for most people to consider their info outdated. If everything were going swimmingly with none of these problems for like 5 years, then yeah, okay, but that's not the situation.
god if there was a comprehensive chat protocol without a loser ceo i'd move to it tomorrow. matrix org staff don't know how to interact with other human beings.