Nice trick. Just a heads up that I had to whitelist your domain as NextDNS blocked it for being newly registered.
Given this thread will probably attract other Unifi users... has anyone had success migrating from MongoDB to something like FerretDB?
I played around with getting this to work a few weeks ago and found that day-to-day it works without issue, but restoring a backup will error since it relies on some unsupported Mongo semantics (renaming collections iirc).
If you subscribe to the mindset of "new domains are likely to be bad" you just deal with a steady stream of allowlist requests from your users until the end of time. There will be new domains until the end of time, and site owners shouldn't be doing anything extra (imo) to justify their existence to admins. If you use a firewall voluntarily and that firewall blocks sites that are legitimate, that's on you, not the site owner.
We get this a lot at my job, where many customers' admins block s3 buckets by default. We give our customers a list of hostnames to allowlist and if they can't figure it out, that's on them.
Sounds like a massive waste of your time for NextDNS admins and a poor UX for end users. If your security relies on trusting old domains, then you need to rethink your security. Also, I bet it's just as easy for you to accidentally whitelist a bad actors as to blacklist a good one. What am I missing here?
>If you subscribe to the mindset of "new domains are likely to be bad" you just deal with a steady stream of allowlist requests from your users until the end of time.
Newly-registered domains are not generally an issue with enterprise users. However, they are overrepresented in malicious traffic due to domain-generation algorithms (DGAs).
> Newly-registered domains are not generally an issue with enterprise users.
I take it this means enterprise users are not generally needing to do anything legit-for-work on a newly registered domain.
Enterprise clicks on newly registered domains tend to be (a) being phished or smished or cryptomined or whatever, or (b) someone reading X or Bsky or HN or ProductHunt's vibe code of the date -- things the enterprise would also like to have blocked.
Consider the CloudFlare/Proofpoint/NextDNS/etc. domain block on new domains much like updating one's HN home page to https://news.ycombinator.com/classic …
It seems like a pretty tall order, but I really want an open source access point controller daemon that knows how to provision and manage a wide variety of APs from different manufacturers.
So you'd have one services that can provision Ubiquity, MikroTik, TPLink and other APs and manage the clients.
I'm glad the payload was usable and the author has fixed their problem, it's an interesting challenge.
However, there are other approaches. A public IP per client isn't going to be nearly as expensive as a VM per client, and lets you route your clients by target. Or you could route by source IP: either by having the client register their IPs, or with some combination with seeing where folk log in from.
Neither is necessary, though, given inspection does appear to work.
The author framed his issue as a choice between separate VMs (with high cost) per user or decoding the messages. As he, you, and I all say: what he's got does work. I'm absolutely not saying that now he's solved the problem he should do something else. But the choice wasn't between only those two extremes.
This protocol was amenable to inspection, the next might not be.
I use NextDNS, one of the features it provides is letting you register a source IP so requests from your network "just work". It might not be a mainstream consumer feature, but neither NextDNS nor managed Unifi controllers are mainstream consumer products.
Yeah IPv6 would solve this because we wouldn't have to go through the proxy at all, since each VM could get their own. Unfortunately IPv6 isn't quite as supported as we'd like.
Do I miss something? How do you adopt the device in the first place? If you have to SSH into the device and set the inform URL manually could't you just route the request based on the request hostname?
Yep, once you set-inform the host header handles the routing. This in particular is most useful for things like DHCP Option 43, where devices only get an IP.
We support two approaches, you can either pre-register MAC-Addresses or you can add source IP's to assist with that mapping. There is more information in our docs about this: https://tamarack.cloud/docs/migration
I wonder if there's a way to control routing client side and remove the list of mac addresses. Eg manage DNS for customers (upsell ad blocking!) and CNAME the unifi entry to a customer specific vhost.
Thank you! DNS-based adoption works well for this. You point the unifi hostname at the tenant's subdomain and the Host header handles routing from there. We also have a DHCP Option 43 generator for setups where DNS isn't practical.
ubnt has been the ubiquiti default login at least back to 2010 when I started using their products, before UniFi was a brand. I always assumed it was short for Ubiquiti Networks.
A lot of companies in that space did then. I was at a robotics company at the time and we experimented with mikrotik routerboards + the various long-range Ubiquiti wifi modules, some of which are even still listed on the website: https://techspecs.ui.com/uisp/accessory-tech/xr (though not the 900 MHz XR9, which was arguably one of the most interesting for long range comms)
Bit of a thread-jack, but has anyone reverse-engineered the UniFi camera adoption protocol? I was surprised to discover that, unlike the APs, the cameras can't be adopted through the Unifi Software Controller that you can just throw into a Docker container. You're supposed to do that through their NVR appliance (Unifi Protect). I was hoping to just use them with my open-source NVR. They seem to be about the only option for a reasonably priced, larger image sensor camera that is not made by a company participating in the Uyghur genocide (Hikvision, Dahua, Univision, Huawei).
I found https://community.home-assistant.io/t/unifi-cameras-without-... in which someone sshed in, edited some config files by hand, and got streaming to work for the current boot. One could probably take that a bit further and, you know, save the config to flash. But it'd be nice to just do it the way their controller does and know it's going to work for future firmware updates and such.
They also stream by connecting to your NVR with modified version of flv, rather than you connecting to them with RTSP, which is annoying but can be worked around.
If you want to bypass Unifi Protect, what sort of "adoption" are you thinking of? AFAIK, "adoption" is a Unifi Protect thing. Otherwise it's just a device on your network that you can configure Frigate etc. to connect to and pull streams.
Changing the credentials for web access (firmware upgrade, janky jpeg-based live stream, etc.) and ssh access from the default ubnt:ubnt. Surprisingly, I don't see a page for this in the web UI, and the `password` command in the CLI is ineffective. I haven't looked around the filesystem.
Setting where it sends the video stream.
Configuring video settings, zone detections, etc. I found a video going through them here: <https://youtu.be/URam5XSFzuM?si=8WK4Yghh9kidZe6c&t=279> Just about any other camera lets you change this stuff through the camera's built-in web interface and/or ONVIF. Ubiquitis apparently don't.
> Otherwise it's just a device on your network that you can configure Frigate etc. to connect to and pull streams.
Unifi cams don't stream RTSP, they stream FLV v1 (FlashVideo) on 3 streams over plain TCP on port 7550, one per quality channel. And yes, they stream that TO the NVR who adopted them only... then the NVR recodes and sends RTSP (configurable).
For the adoption stage, UniFi cameras broadcast on UDP port 10001 using a proprietary TLV (Type-Length-Value) protocol. The Protect console listens on this port and picks up new cameras immediately. 4 bytes `\x01\x00\x00\x00` sent as UDP broadcast to `255.255.255.255:10001`
The response then contains these fields:
| Hex Code | Field | Data |
|----------|-------|------|
| `0x01` | MAC Address | 6-byte hardware address |
| `0x02` | MAC + IP | Combined MAC and IPv4 address |
| `0x03` | Firmware Version | String |
| `0x0B` | Hostname | String |
| `0x0C` | Platform (Short Model) | String |
| `0x0A` | Uptime | 64-bit integer |
| `0x13` | Serial | String |
| `0x14` | Model (Full) | String |
| `0x17` | Is Default | Boolean (adopted vs unmanaged) |
After discovery, the Protect console:
1. Connects to the camera via SSH (default credentials)
2. Configures the Inform URL (TCP 8080)
3. Camera registers with the controller
So conceivably at step 2 you could use your own modified URL to point to your own NVR and then grab the FLV streams from there.
> 1. Connects to the camera via SSH (default credentials) 2. Configures the Inform URL (TCP 8080)
Not what I expected but okay. Looks like there's a `set-inform` command. It looks like it opens a TLS connection, doesn't check the certificate, and tries to opens a websocket:
I might try accepting the websocket but I have a feeling I'll get stuck about there without knowing what the server is supposed to send over it. I'm debating if I'm willing to buy a Unifi Protect device or not.
...then again I did a search for a couple strings and ran across https://github.com/keshavdv/unifi-cam-proxy . It's the opposite direction of what I want (makes a standard camera work with Unifi Protect) but maybe contains the protocol details I'm looking for...
> ...then again I did a search for a couple strings and ran across https://github.com/keshavdv/unifi-cam-proxy . It's the opposite direction of what I want (makes a standard camera work with Unifi Protect) but maybe contains the protocol details I'm looking for...
Actually, yes. I got lazy and just asked Claude Code to write a server, using that as a reference...and it worked. It was able to change the password and have it start streaming flv video. Not exactly a production-quality implementation but as a proof-of-concept it's quite successful.
I hear you, but on the other hand, I'd take a bit of interop pain over supporting genocide any day. It looks like with the hints from moonlighter and from https://github.com/keshavdv/unifi-cam-proxy I'll be able to get this to work.
Honestly it might be less work than some other cameras that (allegedly) speak RTSP. You'd be shocked how low-quality these implementations are. Never advancing timestamps, setting the RTP MARK bit arbitrarily, writing uninitialized memory framed as audio packets (on cameras that don't have microphones), closing file descriptors then writing data to them anyway (and so having it show up on the next accepted connection to be assigned that fd even pre-auth), etc.
It would definitely be simpler, however the routing issue still stands. You would need to have a public IP for every VM, which is getting less practical. The MAC-based proxy makes it so we only need one IP and we can worry about the routing within our platform instead.
You can after the initial discovery step, the article mentions this. The MAC routing is for the first step where the device is reaching out to try and find a controller and signal it's available for adoption, which uses an IP address at least in the scheme that is relevant for hosted operators. After that initial channel is established, the controller uses it to tell the device what its hostname is, and you can switch to more normal HTTP proxy routing thereafter.
You definitely can once the device has a hostname set in set-inform. The MAC proxy is just for initial adoption where a device may only know the IP. Such as when you use DHCP Option 43.
Is it just me that pretty much cannot read most of the text in the "Reading the MAC" code block? I don't know if it's because I use dark mode, but some of the text is #24292E on top of #141A16, which for me at least is practically invisible
Nice trick. Just a heads up that I had to whitelist your domain as NextDNS blocked it for being newly registered.
Given this thread will probably attract other Unifi users... has anyone had success migrating from MongoDB to something like FerretDB?
I played around with getting this to work a few weeks ago and found that day-to-day it works without issue, but restoring a backup will error since it relies on some unsupported Mongo semantics (renaming collections iirc).
How are you performing backup of FerratDB? Are you using MongoDB tools, or are you using PostgreSQL-specific tools?
What does an admin do about NextDNS blocks?
If you subscribe to the mindset of "new domains are likely to be bad" you just deal with a steady stream of allowlist requests from your users until the end of time. There will be new domains until the end of time, and site owners shouldn't be doing anything extra (imo) to justify their existence to admins. If you use a firewall voluntarily and that firewall blocks sites that are legitimate, that's on you, not the site owner.
We get this a lot at my job, where many customers' admins block s3 buckets by default. We give our customers a list of hostnames to allowlist and if they can't figure it out, that's on them.
Sounds like a massive waste of your time for NextDNS admins and a poor UX for end users. If your security relies on trusting old domains, then you need to rethink your security. Also, I bet it's just as easy for you to accidentally whitelist a bad actors as to blacklist a good one. What am I missing here?
I don't disagree. The idea seems to be that newly registered domains are far more likely to be malicious (and not present on domain blocklists yet).
>If you subscribe to the mindset of "new domains are likely to be bad" you just deal with a steady stream of allowlist requests from your users until the end of time.
Newly-registered domains are not generally an issue with enterprise users. However, they are overrepresented in malicious traffic due to domain-generation algorithms (DGAs).
> Newly-registered domains are not generally an issue with enterprise users.
I take it this means enterprise users are not generally needing to do anything legit-for-work on a newly registered domain.
Enterprise clicks on newly registered domains tend to be (a) being phished or smished or cryptomined or whatever, or (b) someone reading X or Bsky or HN or ProductHunt's vibe code of the date -- things the enterprise would also like to have blocked.
Consider the CloudFlare/Proofpoint/NextDNS/etc. domain block on new domains much like updating one's HN home page to https://news.ycombinator.com/classic …
It seems like a pretty tall order, but I really want an open source access point controller daemon that knows how to provision and manage a wide variety of APs from different manufacturers.
So you'd have one services that can provision Ubiquity, MikroTik, TPLink and other APs and manage the clients.
Alternately, run OpenWRT on the APs themselves, and then you just need one provisioning protocol.
Does it support seamless roaming of clients between group of APs?
Last time I've tried, it was not supported by any open source solution.
Well AFAIK the core seamless roaming in Unifi is using hostapd, which is the same AP software you use on OpenWrt. See 802.11r Fast Transition.
I think it should even be possible to get seamless roaming between Unifi and OpenWrt with correct configuration of hostapd.
Now that would be interesting! Multi-vendor support is on the radar, but haven't started looking into it much yet.
I'm glad the payload was usable and the author has fixed their problem, it's an interesting challenge.
However, there are other approaches. A public IP per client isn't going to be nearly as expensive as a VM per client, and lets you route your clients by target. Or you could route by source IP: either by having the client register their IPs, or with some combination with seeing where folk log in from.
Neither is necessary, though, given inspection does appear to work.
I don’t get this comment. Inspection does work but the suggested alternatives don’t.
Having the client register their IPs isn’t tenable for most folks. What’s my IP at the shop? (No idea) Will it change? (Yes) now it’s broken.
Seeing where folks log in from isn’t nearly the same as where their UniFi networks are located. (Store vs home.) Broken.
So neither of the those are robust approaches whereas the author’s solution is bulletproof and simply works in all cases.
No offense, but why suggest “other approaches” that have such major holes? Why not just cheer on the solution that works all the time?
The author framed his issue as a choice between separate VMs (with high cost) per user or decoding the messages. As he, you, and I all say: what he's got does work. I'm absolutely not saying that now he's solved the problem he should do something else. But the choice wasn't between only those two extremes.
This protocol was amenable to inspection, the next might not be.
I use NextDNS, one of the features it provides is letting you register a source IP so requests from your network "just work". It might not be a mainstream consumer feature, but neither NextDNS nor managed Unifi controllers are mainstream consumer products.
Nice hack! Would hosting controllers on individual IPv6 addresses on the same VM solve this too?
Yeah IPv6 would solve this because we wouldn't have to go through the proxy at all, since each VM could get their own. Unfortunately IPv6 isn't quite as supported as we'd like.
Ubiquity in general has had garbage support for IPv6 since the beginning.
They claimed most of their customers aren't asking for it when I pushed them on the issue years ago.
Do I miss something? How do you adopt the device in the first place? If you have to SSH into the device and set the inform URL manually could't you just route the request based on the request hostname?
Yep, once you set-inform the host header handles the routing. This in particular is most useful for things like DHCP Option 43, where devices only get an IP.
But if you only got that IP and a MAC-Address - how do you know which tenant is supposed to adopt the device?
We support two approaches, you can either pre-register MAC-Addresses or you can add source IP's to assist with that mapping. There is more information in our docs about this: https://tamarack.cloud/docs/migration
that's the key information to understand the whole thing.
it would be great if that could be in the article in the first place. (I'm assuming you are the author)
Good call, I should have made that more clear. Article updated!
This is very cool.
I wonder if there's a way to control routing client side and remove the list of mac addresses. Eg manage DNS for customers (upsell ad blocking!) and CNAME the unifi entry to a customer specific vhost.
Thank you! DNS-based adoption works well for this. You point the unifi hostname at the tenant's subdomain and the Host header handles routing from there. We also have a DHCP Option 43 generator for setups where DNS isn't practical.
> ("TNBU" is "UNBT" backwards, presumably UniFi Broadcast Technology.)
This seems like an odd misunderstanding, especially because the correct inversion “UBNT” is the default login name for most UniFi web UIs.
You might have a bit of dyslexia, OP!
You might be onto something there! But yes, good catch, I'll get that updated.
ubnt has been the ubiquiti default login at least back to 2010 when I started using their products, before UniFi was a brand. I always assumed it was short for Ubiquiti Networks.
Sure, but the parent was saying this part was odd:
> "TNBU" is "UNBT" backwards
TNBU is clearly NOT uNbt backwards.
Using the network byte ordering (big endian) of UBNT as the magic number in the protocol is a nice touch.
I believe they used MIPS processors in their early gear, so that makes sense.
A lot of companies in that space did then. I was at a robotics company at the time and we experimented with mikrotik routerboards + the various long-range Ubiquiti wifi modules, some of which are even still listed on the website: https://techspecs.ui.com/uisp/accessory-tech/xr (though not the 900 MHz XR9, which was arguably one of the most interesting for long range comms)
Bit of a thread-jack, but has anyone reverse-engineered the UniFi camera adoption protocol? I was surprised to discover that, unlike the APs, the cameras can't be adopted through the Unifi Software Controller that you can just throw into a Docker container. You're supposed to do that through their NVR appliance (Unifi Protect). I was hoping to just use them with my open-source NVR. They seem to be about the only option for a reasonably priced, larger image sensor camera that is not made by a company participating in the Uyghur genocide (Hikvision, Dahua, Univision, Huawei).
I found https://community.home-assistant.io/t/unifi-cameras-without-... in which someone sshed in, edited some config files by hand, and got streaming to work for the current boot. One could probably take that a bit further and, you know, save the config to flash. But it'd be nice to just do it the way their controller does and know it's going to work for future firmware updates and such.
They also stream by connecting to your NVR with modified version of flv, rather than you connecting to them with RTSP, which is annoying but can be worked around.
If you want to bypass Unifi Protect, what sort of "adoption" are you thinking of? AFAIK, "adoption" is a Unifi Protect thing. Otherwise it's just a device on your network that you can configure Frigate etc. to connect to and pull streams.
Changing the credentials for web access (firmware upgrade, janky jpeg-based live stream, etc.) and ssh access from the default ubnt:ubnt. Surprisingly, I don't see a page for this in the web UI, and the `password` command in the CLI is ineffective. I haven't looked around the filesystem.
Setting where it sends the video stream.
Configuring video settings, zone detections, etc. I found a video going through them here: <https://youtu.be/URam5XSFzuM?si=8WK4Yghh9kidZe6c&t=279> Just about any other camera lets you change this stuff through the camera's built-in web interface and/or ONVIF. Ubiquitis apparently don't.
> Otherwise it's just a device on your network that you can configure Frigate etc. to connect to and pull streams.
No, it connects to you!
Im using a g3 flex with frigate without use of unifi protect, works fine.
I think newer models like g4 flex dont support this thou.
Here's that YouTube link without the creepy Google tracking component:
https://youtu.be/URam5XSFzuM?t=279
You want to change the credentials of the camera, so Frigate can log into it while it is connected to your Unifi network?
I did that for 5 different cameras yesterday, you're saying Unifi's cameras doesn't allow user management? That sucks!
> No, it connects to you!
I thought frigate connects to the camera's RTSP stream (maybe with ONVIF in the mix)?
Unifi cams don't stream RTSP, they stream FLV v1 (FlashVideo) on 3 streams over plain TCP on port 7550, one per quality channel. And yes, they stream that TO the NVR who adopted them only... then the NVR recodes and sends RTSP (configurable).
For the adoption stage, UniFi cameras broadcast on UDP port 10001 using a proprietary TLV (Type-Length-Value) protocol. The Protect console listens on this port and picks up new cameras immediately. 4 bytes `\x01\x00\x00\x00` sent as UDP broadcast to `255.255.255.255:10001`
The response then contains these fields:
After discovery, the Protect console: 1. Connects to the camera via SSH (default credentials) 2. Configures the Inform URL (TCP 8080) 3. Camera registers with the controllerSo conceivably at step 2 you could use your own modified URL to point to your own NVR and then grab the FLV streams from there.
Thanks!
> 1. Connects to the camera via SSH (default credentials) 2. Configures the Inform URL (TCP 8080)
Not what I expected but okay. Looks like there's a `set-inform` command. It looks like it opens a TLS connection, doesn't check the certificate, and tries to opens a websocket:
I might try accepting the websocket but I have a feeling I'll get stuck about there without knowing what the server is supposed to send over it. I'm debating if I'm willing to buy a Unifi Protect device or not....then again I did a search for a couple strings and ran across https://github.com/keshavdv/unifi-cam-proxy . It's the opposite direction of what I want (makes a standard camera work with Unifi Protect) but maybe contains the protocol details I'm looking for...
> ...then again I did a search for a couple strings and ran across https://github.com/keshavdv/unifi-cam-proxy . It's the opposite direction of what I want (makes a standard camera work with Unifi Protect) but maybe contains the protocol details I'm looking for...
Actually, yes. I got lazy and just asked Claude Code to write a server, using that as a reference...and it worked. It was able to change the password and have it start streaming flv video. Not exactly a production-quality implementation but as a proof-of-concept it's quite successful.
> I thought frigate connects to the camera's RTSP stream (maybe with ONVIF in the mix)?
Right, that's the expectation of Frigate, my own Moonfire NVR, and basically every other NVR out there. Ubiquiti decided to think different.
Well thanks for the heads-up to avoid their cameras.
I hear you, but on the other hand, I'd take a bit of interop pain over supporting genocide any day. It looks like with the hints from moonlighter and from https://github.com/keshavdv/unifi-cam-proxy I'll be able to get this to work.
Honestly it might be less work than some other cameras that (allegedly) speak RTSP. You'd be shocked how low-quality these implementations are. Never advancing timestamps, setting the RTP MARK bit arbitrarily, writing uninitialized memory framed as audio packets (on cameras that don't have microphones), closing file descriptors then writing data to them anyway (and so having it show up on the next accepted connection to be assigned that fd even pre-auth), etc.
FWIW, there are multiple other camera brands who don't manufacture in Xinjiang (or China for that matter), like Axis or Vivotek.
I don't quite get the reason for sniffing the packets. Wouldn't it be simpler to just run multiple VMs on one host to be multi tenant?
It would definitely be simpler, however the routing issue still stands. You would need to have a public IP for every VM, which is getting less practical. The MAC-based proxy makes it so we only need one IP and we can worry about the routing within our platform instead.
but it is http! why can't you do virtual hosting on a reverse proxy?
You can after the initial discovery step, the article mentions this. The MAC routing is for the first step where the device is reaching out to try and find a controller and signal it's available for adoption, which uses an IP address at least in the scheme that is relevant for hosted operators. After that initial channel is established, the controller uses it to tell the device what its hostname is, and you can switch to more normal HTTP proxy routing thereafter.
Yep, that's exactly right!
You definitely can once the device has a hostname set in set-inform. The MAC proxy is just for initial adoption where a device may only know the IP. Such as when you use DHCP Option 43.
Is it just me that pretty much cannot read most of the text in the "Reading the MAC" code block? I don't know if it's because I use dark mode, but some of the text is #24292E on top of #141A16, which for me at least is practically invisible
Sorry about that, I typically use light mode, fixed and deployed!
A million times better, thanks =)
Thanks for calling it out!
Controller uses way to much RAM compared to OpenWISP and good luck if a device is EOL. Lots of $10 USG-3P's out there.
FWIW, the USG-3P is listed as supported by OpenWRT.