You know I was actually really curious about this so I went back to the HTML and URL W3C standards and surprisingly they don't actually have any definitions of format other than being percent encoded. One might conflate query strings with "form-urlencoded"[0] query strings, which is one potential interoperability format, but in general a queries string is just any percent encoded string following a "?" in a url[1], and just another property in the "URL" HTML object that can be used in the generation of a response. While additionally there is a URLSearchParams object that is the result of parsing the query string with the form-urlencoded parser, this is simply an interoperability layer for JavaScript.
I'm going to be honest, I was pretty geared up to have a contrarian opinion until I looked at the standards but they're actually pretty clear, a 404 could be a proper response to unexpected query string; query string is as much part of the URL API as the path is and I think pretty much everyone can acknowledge that just tacking random stuff onto the path would be ill advised and undefined behavior.
Back in the day it was reasonably common for CMSs and forums to only have an index.php, and routing entirely by query string (in form-urlencoded form, people were not savages). So you would have index.php?p=home and index.php?p=shop. Or index.php?action=showthread&forum=42&thread=17976. It should be immediately obvious that in that scheme 404 is indeed the correct answer to unknown query parameters
In fact lots of sites still work like that, they just hide it behind a couple rewrite rules in apache/nginx for SEO reasons
On the other hand, if it's a CRUD app and you're filtering a list of entities by various field values? Returning that no items matched your selection (or an empty list, if an API) makes more sense than a 404, which would more appropriate for an attempt to pull up a nonexistent entity URI.
A response code of 204 seems more appropriate but the problem is you're not allowed to send further information, which would make that descriptive response... not descriptive enough.
The point was that returning a 404 for unexpected query strings doesn’t just happen to okay per the specs, but that there is significant historical precedent for doing so based on application design that was common in the past.
Libraries that automatically throw errors for status codes in the 400 and 500 ranges are pretty obnoxious (looking at you, axios). It adds unnecessary overhead, complexity, and bad ergonomics by hijacking control flow from the app.
Responses with status codes in the 400 range are client errors, so the client shouldn't retry the same request. So a 404 is appropriate despite how annoying a library might be at handling it. Depending on which language/ecosystem you are using, there are likely more sane alternatives.
> It should be immediately obvious that in that scheme 404 is indeed the correct answer to unknown query parameters
That's not obvious at all. If I receive json data that contains a property I'm not aware of, i don't reject the entire document for that reason. In the case of query strings, extra query parameters might be used by other parts of the stack besides yours, so rejecting the entire document because someone somewhere else is trying to pass information to itself is the wrong approach.
As a web developer, you’re the like the guy standing with a clipboard outside a fancy club checking if people requesting entry are allowed or not. Basically, level 1 security.
If someone is not on the list, your job is to default to declining them access, not granting them access assuming level 2 security will handle them at a deeper layer.
It’s possible that the teams you work with expect fuzzy behaviour from the website but that’s a choice, not a practice.
Standards are just commonly accepted behaviour that somebody chose to write down somewhere. There are a great number of commonly accepted behaviours that nobody's ever bothered to encode into a formal standard, but where failure to follow the accepted practice will result in widespread breakage. There are also a great many "standards" that you would be a fool to follow to the letter. In the OP case, the only thing that will break is people trying to visit their site, who will presumably simply press the back button on their browser and go about their day. They can decide for themselves if that is an acceptable casualty. But it isn't definitionally acceptable because no standard says it isn't (nor would is suddenly become unacceptable because a standard said it was...)
> I was pretty geared up to have a contrarian opinion until I looked at the standards but they're actually pretty clear, a 404 could be a proper response to unexpected query string; query string is as much part of the URL API as the path is and I think pretty much everyone can acknowledge that just tacking random stuff onto the path would be ill advised and undefined behavior.
This feels like a technically correct is the best kind of correct situation. Like technically, yeah web servers may respond 404 if they dont understand a query parameter, but in practise that is not how urls are conceptualized normally.
Interestingly, quite a few places that should treat query strings transparently make a lot of assumptions about their structure. We ran into that when picking a new CDN, some providers didn't handle repeat parameters (?a=1&a=2) correctly.
Incorrectly would be processing the query string and deduping keys. Correctly would be passing it through as-is, or at least only lightly processing it, like normalizing escaping or such.
Wait until you realize that the difference between path and query string is entirely arbitrary and decided by the server. Query strings should never have existed. They are an implementation detail of CGI webservers that leaked all over everything and now smells really bad.
I dunno, it seems like the fact that we arrived at a fairly standard structure for URL paths that works pretty well is not a bad outcome.
Seems a lot better than the other potential world we could lived in, where paths were a black box and every web server/framework invented their own structure for them.
In my current project I use URIs to refer to absolutely any entity in a git(-ish) repo. Files, branches, revisions, diffs, anything. URI turns out to be a really good addressing scheme for everything. Surprise. But the most used and abused element is always the path. Query takes a lot of that mess away. Might have been unmanageable otherwise.
Grouping data by user is common and normal in computing: /home laid precedent decades ago.
Project directories are an extremely common grouping within a user’s work sets. Yeah, some of us just dump random files in $HOME, but this is still a sensible tier two path component.
The choice to make ‘view metadata-wrapped content in browser HTML output’ the default rather than ‘view raw file contents’ the default is legitimate for their usage. One could argue that using custom http headers would be preferable to a path element (to the exclusion of JavaScript being able to access them, iirc?) or that the path element blob should be moved into the domain component or should prefix rather than suffix the operands; all valid choices, but none implicitly better or worse here.
Object hash is obviously mandatory for git permalinks, and is perhaps the only mandatory component here. (But notably, that’s not the same as a commit hash.) However, such paths could arguably be interpreted as maximally user-hostile.
File path, interestingly enough, is completely disposable if one refers to a specific result object hash within a commit, but if the prior object hash was required to be a commit, then this is a valid unique identifier for the filesystem-tree contents of that commit. You could use the object hash instead of the full path within the commit hash, but that’s a pretty user-hostile way to go about this.
So, then, which part of the ordering and path selections do you consider indiscriminate, and why?
actually, instead of the object hash, you could also use the commit-hash. then the filename would be mandatory, but the url would be more readable and usable: give me the file VERBS.md as it is at commit <hash>
what would be a better way of doing that? i am not disagreeing, but i just can't think of any way to improve on this. put everything into the query part? i prefer to use the query only for optional arguments. in this example the blob argument is the only thing that doesn't fit in my opinion.
Every object in git (commit, tree, revision of a single file) has a hash that is guaranteed unique within a repository (otherwise many more things than a web UI would break) and likely also globally. I can understand wanting to isolate repositories to prevent hash collisions from causing problems, but within a repo everything has a universally unique ID.
edit: for instance, that specific VERBS.md is represented by the blob 3b9a46854589abb305ea33360f6f6d8634649108.
They are following the /key/value/key/value pattern, but the first two pairs in a GitHub URL are fixed to user and project, which lets them omit the key names. I could see them not being willing to hardcode the third pair to blob.
Back when GitHub URLs were kind of cool, github.com/user/gritzko/project/beagle would have been much less cool than just github.com/gritzko/beagle.
Of course there's nothing to stop you using URIs like this (I think Angular does, or did at one point?) but I don't think the rules for relative matrix URIs were ever figured out and standardised, so browsers don't do anything useful with them.
Not entirely arbitrary - forms that use the GET method instead of POST will append form values as query params.
For sites without Javascript, it's great for things like search boxes, tables with sorting/filtering, etc. instead of POST, since it preserves your query in the URL.
It has always amazed me how much trouble the SPA folks are willing to go to in order to slowly rebuild just normal boring URLs with querystrings because users demand deep linking and back buttons and the like.
Or you could accept that you're probably going to need a round trip to the server and use a normal URL and it's fine.
For all but the absolute biggest websites in the world, anyhow. At Facebook or Google scale yeah it's needed.
Query strings existed before CGI did, and the way they're defined to be filled in from web forms is quite useful; I wouldn't want to need Javascript to fit that into path format. There's nothing wrong about having things decided by the server; I don't get that part of your argument at all.
Maybe dumb question: how does the server “decide” anything other than what file to serve? Today we have many choices but back in the day CGI was the first standard way to do it.
So yes query parameters existed before CGI but to use them you had to hack your server to do something with them (iirc NCSA web servers had some magic hacks for queries). CGI drove standardization.
It's arbitrary to a degree like the difference between using an attribute or child element in XML, but it's not entirely arbitrary. If you want to include data in the URL that's not part of the hierarchy of the path, query strings are good for that.
> It is a small, decentralised, self-hosted web console that lets visitors to your website explore interesting websites and pages recommended by a community of independent personal website owners.
Back in the Stone Age, we called these “Webrings,” but they weren’t as fancy.
One of the issues that I faced, while developing an open-source application framework, was that hosting that used FastCGI, would not honor Auth headers, so I was forced to pass the tokens in the query. It sucked, because that makes copy/paste of the Web address a real problem. It would often contain tokens. I guess maybe this has been fixed?
In the backends that I control, and aren’t required to make available to any and all, I use headers.
> an open-source application framework, was that hosting that used FastCGI, would not honor Auth headers
So you were writing your application as a fcgi-app, and (e.g.) Apache was bungling Auth headers? Can you expand on this? Curious about the technical detail of (I guess) PARAM records not actually giving you what you expect?
I don’t remember, exactly. Long time ago (I stepped away from that project many years ago).
I just remember the auth headers never showing up in the $_SERVER global (it was a PHP app). This was what I was told was the issue. They made it sound like it was well-known.
Query strings do have uses (such as for searching files and some other kind of dynamic files), but you shouldn't add them to URLs that should not expect them. So, I agree that they would be right to refuse requests with UTM and other stuff added like that.
I think 404 probably makes the most sense as the response if a query string is not expected but is present anyways, although 400 might also be suitable.
>So I’ve decided to try a blanket ban for this site: no unauthorised query strings.
His site returns (I think incorrectly) a 414 if a request includes a query string. If this protest is meant to advocate for the user, who presumably wasn't able to manage that string in the first place, why would you penalize them for it being there?
Why not just use it as a cue to tell users how they can make this decision themselves (e.g. through browser tools)?
"You could argue that I’m abusing 414 URI Too Long. I respond that it’s funnier this way. Other options I considered were:
400 Bad Request, the generic client error code, which is correct but boring;
402 Payment Required, and honestly if you want to pay me to make a particular URL with query string work, I’m open to it;
404 Not Found, but it’s too likely to have side effects, and it doesn’t convey the idea that the request was malformed, which is what I’m going for; and
303 See Other with no Location header, which is extremely uncommon these days but legitimate. Or at least it was in RFC 2616 (“The different URI SHOULD be given by the Location field in the response”), but it was reworded in 7231 and 9110 in a way that assumes the presence of a Location header (“… as indicated by a URI in the Location header field”), while 301, 302, 307 and 308 say “the server SHOULD generate a Location header field”. Well, I reckon See Other with no Location header is fair enough. But URI Too Long was funnier."
I don't think it's an abuse, RFC9110 defines 414 as a response for "refusing to service the request because the target URI is longer than the server is willing to interpret". Since adding a query string involves only adding characters, this seems fine; there's no stipulation as far as I can tell that all pages a server hosts must adhere to the same length. I'd be curious if any well-known clients interpret it that way though, and make caching decisions based on it. As far as I know, they shouldn't.
Obviously it's against the spirit of the thing, but I don't think it's wrong per-se.
Opinions vary on how good an idea the robustness principle is. That is why, for example, the XML standard requires a conforming validator to throw an error on invalid XML.
In our modern world, the robustness principle has become an invitation to security bugs, and vendor lock-in. Edge cases snuck through one system on robustness, then trigger unfortunate behavior when they hit a different system. Two systems tried to do something reasonable on an ambiguous case, but did it differently, leading to software that works on one, failing to work on the other.
I generally agree, but I don't think XML is the best example. Getting HTML out of XML is considered to have been the right move isn't it? I was pro-XHTML2 at the time but in retrospect, have we suffered much for not sending webpage validation errors to end users?
Once people have gotten used to not having to conform, forcing them to conform is an uphill battle. Doubly so when, as happened with Microsoft and IE, the vendors would like to encourage vendor lock-in. The only time to reasonably do it is at the start.
That said, we are paying a huge complexity cost due to our efforts to allow nonconforming pages. This complexity is widely abused by malicious actors. See, for instance, https://cheatsheetseries.owasp.org/cheatsheets/XSS_Filter_Ev... for ways in which attackers try to bypass security filters. A lot of it is only possible because of this unnecessary complexity.
But, this is robust? I mean it's pretty clearly stating that you are visiting an unsupported URL. It provides direction on what to do about it to the user. It does not crash the browser or the server. In pretty much every dimension this is highly robust.
The robustness principle is itself bad manners, in plenty of contexts. If I deliver packages by throwing them at the customer, I really want a customer to tell me "hey, don't throw packages at me!" before I attempt to lob something fragile and breakable, or something heavy at someone fragile and breakable. Otherwise, how am I supposed to learn that I'm doing anything wrong?
It's been years but I seem to remember there was a version of PLSQL server pages that would return 500 if you tried to pass in an unknown query string.
Just straight "400" ("Bad Request") or "403" ("Forbidden") would also probably be defensible. Odd that there aren't any error response codes specific to URI parameters.
Several options which seem like they might be appropriate aren't on close examination:
- "406" ("Not Acceptable") which is based on content-negotiation headers.
- "409" ("Conflict") which is largely for WebDAV requests.
- Others such as 411, 422, and 431 are also for specific conditions which aren't relevant here.
- 300 or 500 errors are inappropriate as this isn't a relocation or server-side failure, it's a client-side request problem.
I think either 400 or 404 would be fine. 400 because the request isn't in the expected format, 404 because a resource with that query string doesn't exist.
Of course they do. For example you can lower a string from the top to query the fill level. Or you can wrap a string around the pot to query the circumference.
The tone of this and Chris's post gives me the impression that it's harmful to include these query parameters, but I don't understand how. Could someone elucidate me? I understand it can mangle some URLs and that's good enough reason not do it, but even then it seems like a minor incovenience.
There are a lot of reasons I might not want a site to know where I came from to get to their site. It is basically sharing your browsing history with the site you are visiting.
Because of this, there have been a lot of updates to the http referer header, with restrictions on when it is sent, and an ability to opt out of the feature entirely.
Adding a url parameter with the same information bypasses any of these existing rules and ability to opt out. They should just use the standard.
If I release a video and send an email newsletter at the same time, which one caused the traffic increase? Should I invest in making more videos of sending more emails?
If you insist on knowing, include a different url in both that goes to the same place and use your damn server logs. You don’t need google analytics and whatever.
presumably you control the urls you are sending in the email. As a result if you want to use query strings that's fine. The issue only arises when you use query strings to implement tracking on someone else's site instead.
What's interesting is that none of these sites have a "search" feature. Which is an important accessibility feature and a clear and legitimate use case for a query string.
Oh, I have a couple - the users did not agree on being tracked (these query params are tracking information), and the site administrator does not want incoming traffic to be tracked. I know the latter can be hard to understand, but I for example sure as hell do not want to have any info in my logs that can be used to harm my users.
On a more personal note, I hate it when I go to copy a link to send via a message, and the tracking code glued onto it is twice as long as original URL... I either have to fiddle around with it to clean it up or leave the person I sent it to to wonder wtf am I on about with a screenful of random characters...
So it's violating users' privacy, it's shit UX, and on top of that, nobody asked for it...
Query strings are useful for way more than just tracking. Saving and servicing search queries is a way more common use case. So assuming it's only useful for tracking is very misleading.
Query strings are probably the least invasive tracking. They are transparent, obvious, and anonymous. Users are free to strip out and edit query strings if they don't want them.
More to the point, I can essentially do the same thing with HTTP routing - create an infinite number of unique URLs for tracking purposes. In that regard calling out query strings specifically for essentially the same thing but more transparently seems like splitting hairs.
Yeah I do not see an alternative way to easily copy paste links with things like filter settings saved.
Filters especially make sense as query params as they are non sequential but still visually readable as to what they do.
URL slugs make sense for sequential pages that are hierarchical but make no sense for non hierarchical data/routes.
Services can force tracking into links by encoding the whole url into a shortlink that makes it impossible to just remove the tracking alone as everything is encoded into a shorter non editable string.
Thank you for explaining to me that query parameters can be used for other purposes apart from tracking. The articles in question though, are railing against query parameters being abused for tracking purposes - passing referers (sic) and UTM by adding them to URLs of sites that neither process them, nor want them.
Referral query strings are not for tracking though. The person putting them on the links gets nothing out of them. There is no PII being shared. They are purely added out of courtesy.
If I am handing out maps to your address, letting people know who is publishing the map is generally a good thing.
This is like saying having a return to sender address on mail is an invasion of privacy.
So my understanding is, he is annoyed that other website adds a query string such as "?ref=origin.com" to links pointing to authors website.
How does this benefit the other website? How does this hurt the authors website?
I am completely confused about the behavior of both side here.
I get that when I run an ad-campaing I want google to add a utm-query string, so I can track which campaign users arrived from - but then the origin and the destination are working together. Here the origin just adds stuff for no reason. Why?
Maybe an alternative would be to inconvenience people following such links still, but somewhat less.
Instead of responding with an error, give a page that states “The link you followed to get here appears to have had some tracking gubbins added, in case you are a bot following arbitrary links, and/or using random URL additions to look like a more organic visit, please wait while we run a little PoW automaton deterrent before passing you on to the page you are looking for.” then do a little busy work (perhaps a real PoW thingy) before redirecting. Or maybe don't redirect directly, just output the unadorned URL for the user to click (and pass on to others). This won't stop the extra gubbins being added of course, but neither will the error and this inconveniences potential readers less.
"I don’t like people adding tracking stuff to URLs" and "You abuse your users by adding that to the link" and "no unauthorised query strings" and "At present I don’t use any query strings" but for some reason ?igsh, which i'm pretty sure is an instagram tracking parameter, is allowed. weird
"wander console" sounds like they're just web rings re-invented. In the era of forced feeds by giant corporations which consist of the things they want you to see, I've wondered if this old idea would make a comeback. Human curated content from trusted people seems like the only way forward.
I'll start with the clarification that the moderators have changed the URL of the original post from <https://susam.net/no-query-strings.html> to <https://chrismorgan.info/no-query-strings>. Hopefully, this will prevent any confusion about why we are discussing random walks in a post about query strings. Now let me answer your question.
> Is it not a random walk? Might sound pedantic but if there is graph structure I am interested.
The network is a directed graph. Every Wander Console declares a few other consoles as its neighbours. The person setting up the console decides who they want to list as their neighbours. So if we call the network graph X, then the set of vertices is:
V(X) = the set of all URLs that point to Wander Consoles
and the set of directed edges is:
E(X) = {(u, v) in V(X) : u declares v as its neighbour}
The traversal between consoles is not strictly a random walk. If I could call it something, I would call it randomised graph exploration with frontier expansion. On each click of the 'Wander' button, the tool picks one console at random from the set of discovered consoles and visits that console. It then fetches the neighbours declared by that console and adds any newly discovered consoles to the set.
The difference from a random walk is that the next console is not chosen from the neighbours of the last visited console. It is chosen from the whole set of consoles discovered so far. In other words, each click expands the known part of the graph, but the console used for that expansion is selected randomly from all discovered consoles, not just from the last console visited.
This is not the first site to do so. A few years back, scarygoround.com started blocking query strings, although it seems to have stopped doing so now. Back then, Facebook had started to add ?fbclid=... to every outgoing link.
Trying to boostrap some taboo against novel unpermissioned URL munging is silly prudishness.
Ensuring both sides of a hyperlink agree/consent was a design flaw that limited the uptake of pre-web hypertext systems. The web's laissez-faire approach demonstrated a looser coupling was far better for users, despite all the new failure modes.
Of course any site/server has the practical power free to treat inbound requests as rigorously (or harshly) as they want. But by the web's essential nature, it is equally part of the inherent range-of-freedom of outlink authors to craft their URLs (and thus the resulting requests) however they want. URLs are permissionless hyperlanguage, not the intellectual property of entities named therein.
Plenty of sites welcome such extra info, and those that don't want it can ignore it easily enough – including by just not caring enough about the undefined behavior/failures to do nothing.
Though, when a web publisher has naively deployed a system that's fragile with respect to unexpected query-string values, they should want to upgrade their thinking for robustness, via either conscious strictness or conscious permissiveness. Thereafter, their work will be ready for the real web, not a just some idealized sandbox where scolding unwanted behavior makes sense.
There’s nothing ruder in hypertext etiquette than giving someone a link to navigate to someone else’s HTTP server, where you have manipulated that URL in some way unsanctioned by the server you are sending them to.
You can’t just send arbitrary query string parameters to a server and assume they will just ignore them. Just like you can’t just remove query string parameters and assume the URL will work.
In fact, you usually can just send arbitrary query string parameters to a server - that's why the behavior is so common, and often useful.
Most sites don't mind or break, some sites get value from the behavior in ways hard to replicate in other ways – and those sites that don't like such additions can easily ignore them. And a few lines of code will work better than ineffectually appealing to manners, when the freedom of the web's form of hypertext, and protocols, gives the outlink authors full freedom to craft URLs (and thus requests) however they like.
Crafting outbound links with your own additions and handing them out to visitors to your site is similar to the practice of writing someone’s phone number on the door of a bathroom cubicle with ‘for a good time call:’ written above it.
You’re handing out someone elses’s contact details, but giving the person you hand them to a completely fabricated expectation for how the interaction will go.
Adding query strings is one of those things that I think a lot of sites could get away with more easily if they were reasonable about it.
A link that is "https:// web.site" is fine.
A link that is "https:// web.site?via=another.site" is fine.
A link that is "https:// web.site?fbm=avddjur5rdcbbdehy63edjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63edaaaddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednzzddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63edn"
is annoying as shit and I need to literally apologize to people after sending it if I forget to manually redact the query string. Don't abuse this.
>Want to share an amazon product on a chat to discuss about it. I would have liked a nice short url that I can copy, instead I get a monstrosity, it forces me to manually select only the id portion of it if I want to share it.
Just referrer policy of strict origin when cross origin gives host level referer (sic) header in most mainstream browsers unless user has configured otherwise right? That’s usually enough for web authors to know what audience they’re appealing to and privacy-maximizers can turn off that header sending.
YouTube is also quite famous with their source identifiers, especially with the short urls, the tracking part is longer then the url I’m trying to share.
I was just wondering if I should do something like this. I use a couple query string values and I validate them and issue a 40x if the value is invalid. So, I was wondering if I should issue a 40x for an unused query string val.
Query strings break unpredictably, and that alone is enough to ban them by third parties, especially for something as minor as referral tracking.
Example: The Browser is a well known link aggregation paid periodical. I subscribe, and every 1 in 10 or 20 links I clicked, it'd just break outright and I'd have to tediously edit the URL to fix it (assuming the website didn't do a silent ninja URL edit and make it impossible for me to remember what URL I opened possibly days or weeks ago in a tab and potentially fix it). This was annoying enough to bother me regularly, but not enough to figure out a workaround.
Why? ...Because TB was injecting a '?referrer=The_Browser' or something, and the receiving website server got confused by an invalid query and errored out. 'Wow, how careless of The Browser! Are they really so incompetent as to not even check their URLs before mailing an issue out to paying subscribers?'
I wondered the same thing, and I eventually complained to them. It turns out, they did check all their URLs carefully before emailing them out... emphasis on 'before', which meant that they were checking the query-string-free versions, which of course worked fine. (This is a good example of a testing failure due to not testing end-to-end or integration testing: they should have been testing draft emails sent to a testing account, to check for all possible issues like MIME mangling, not just query string shenanigans.)
After that they fixed it by making sure they injected the query string before they checked the URLs. (I suggested not injecting it at all, but they said that for business reasons, it was too valuable to show receiving websites exactly how much traffic TB was driving to them on net, because referrers are typically stripped from emails and reshares and just in general - this, BTW, is why the OP suggestion of 'just set a HTTP referrer header!' is naive and limited to very narrow niches where you can be sure that you can, in fact, just set the referrer header.)
But this error was affecting them for god knows how long and how many readers and how many clicks, and they didn't know. Because why would they? The most important thing any programmer or web dev should know about users is that "they may never tell you": https://pointersgonewild.com/2019/11/02/they-might-never-tel... (excerpts & more examples: https://gwern.net/ref/chevalier-boisvert-2019 ). No matter how badly broken a feature or service or URL may be, the odds are good that no user will ever tell you that. Laziness, public goods, learned helplessness / low standards, I don't know what it is, but never assume that you are aware of severe breakage (or vice-versa, as a user, never assume the creator is aware of even the most extreme problem or error).
Even the biggest businesses.... I was watching a friend the other day try to set up a bank account in Central America, and clicking on one of the few banks' websites to download the forms on their main web page. None of the form PDF download links worked. "That's not a good sign", they said. No, but also not as surprising as you might think - the bank might have no idea that some server config tweak broke their form links. After all, at least while I was watching, my friend didn't tell them about their problem either!
I don't see how your example, The Browser (thebrowser.com), supports your argument that ad-hoc query-string additions are so prone-to-breaking that 3rd parties should ban them.
In fact, the example seems to suggest the opposite: a 17+ year successful paid subscription business – to which you appear to be a generally-satisfied customer! – receives enough "business value" from the practice, despite its failure modes, they don't want to stop. Improving their probe of the risk-of-failure was enough.
Seemingly, the practice works often enough, pleasing more destination sites than it angers, that "referral tracking" is not something "so minor".
> Improving their probe of the risk-of-failure was enough.
The point was it was dangerous in a way they didn't even realize was an issue, for a thin business rationale. Unless you are going to do thorough tests and understand the risk you are taking (which they did not, as evidenced by screwing it up systematically at scale for years), you should not be doing it.
And it's not obvious that they are correct in their tightened-up testing, because even if a link is correct at the time they test it, it could break at any time thereafter.
> to which you appear to be a generally-satisfied customer!
No matter what _X_ is, _X_ would have to be a pretty epic screwup to make a customer unsubscribe solely over that! I never claimed it was such a major epic screwup that it could do that. So that is an unreasonable criterion: "well, you didn't outright quit, so I guess it can't be that bad." Indeed, but I never said it was, and somewhat bad is still bad; I was in fact fairly annoyed by the random breakage, and at the margin, everything matters. If TB did a few other things, in sum, they could potentially convince me to let my subscription lapse. An annoyance here, a papercut here, and pretty soon a generally-satisfied customer is no longer so satisfied...
Referrer is sometimes nice to know. If your site gets a traffic spike from an email newsletter that traffic won't correctly identify the source in the http headers.
It's just a string though. A project that I'll never get to is a custom webserver so that QR codes can use the smaller characterset, so it can link to a URL with parameters without forcing the larger character set.
Query strings are awesome. Especially for one-page applications.
I build a lot of internal applications, and one of my golden UI rules is that a user should be able to share their URL and other users should be able to see exactly what the sender did.
So if you have a dashboard or visualization where the user can add filters or configurations, I have all of their settings saved automatically in the URL. It's visible, it's obvious, it's easy, it's convenient.
>There is also a moral question here about whether it is okay to modify a given URL on behalf of the user in order to insert a referral query string into it. I think it isn't.
These dogmatic technical screeds are all so weird to me. They usually reveal more about the authors lack of experience or imagination than provide a useful truism.
> If I wanted to know I’d look at the Referer header; and if it isn’t there, it’s probably for a good reason. You abuse your users by adding that to the link.
The reason is that the referrer headers are a usability and privacy nightmare. It's weird for the author to jump to such a conclusion.
This referral information is being done purely as a courtesy to the webhost. If we imagined a world in which ChatGPT or Wikipedia launched massive hugs of death on referral links without attributing themselves, that is a much, much worse outcome.
There's a referrer header, if the client wishes to send it. If they don't, the "courtesy to the web host" is done at the expense of the client. This particular web host takes umbrage at other sites taking advantage of their clients that way, which seems reasonable to me.
A relatively minor impact concern is that query strings create a new cache entry both in the browser and typically on server-side caches unless configured otherwise, so you might want to use URL fragment parameters if the parameters are only used by clientside JavaScript but the server response is the same.
It also makes me wonder what other noxious online behaviours might be addressed through ... creative ... client-side responses similar to this.
We've already seen, for years, sites attempting to socially-condition people over the use of ad-blockers and Javascript disablers. No reason why the Other Side can't fight back as well.
Running your own small website is a constant battle against grifters and bad online etiquette. When people hotlink images, I usually make a point of having some personal fun with mod_rewrite.
> After I implemented that feature, a page from one of my favourite websites refused to load in the console... the third URL returns an HTTP 404 error page. The website uses the query string to determine which one of its several font collections to show.
Yes, let's unilaterally decide that query strings are bad because one website (ab)uses query strings to load different fonts.
It's the query strings that are the problem, not the website!
jfc.
Look, I'm against utm fragments as much as the next guy, but let's not throw away a perfectly good thing because tracking is evil.
Adding your own garbage to someone else's URLs is in fact the problem. Could they handle your garbage better? Sure. Is your garbage still a problem? Yes.
Postel's law worked OK when people operated in good faith. But today the internet is full of abusers. Rejecting requests that aren't exactly what they should be is probably the best policy now.
Postel's law is typically stated as "be conservative in what you do, be liberal in what you accept from others". It's unfortunately common for people to ignore the first half and hallucinate a third clause demanding that the recipient stay silent about the errors they receive.
That website is not abusing query strings, though, its usage of query strings is perfectly cromulent. And tfa is not saying not to use query strings, but not to append random garbage to other people's URLs.
The website uses the feature for its intended purpose. Adding random trash to the query string of another website assuming it'll ignore it is in fact a bad idea, always, even if you can usually get away with it.
I mean…the site that broke should know what to do with arbitrary query strings. If your site breaks when someone puts in an invalid query string, that’s on you?
That does not make a lot of sense. Yes, you can do what you want
with your website, but query-string is a way for users to query
for additional information or wants or needs. I use them on my own
websites to have more flexibility. For instance:
foobar.com/ducks?pdf
That will download the website content as a formatted .pdf file.
I can give many more examples here. The "query strings are horrible"
I can not agree with at all. His websites don't allow for query
strings? That's fine. But in no way does this mean query strings
are useless. Besides, what does it mean to "ban" it? You simply
don't respond to query strings you don't want to handle. We do
so via general routing in web-applications these days.
This isn't relevant when talking about links to his site. This is relevant when talking about links to your site.
> Besides, what does it mean to "ban" it? You simply don't respond to query strings you don't want to handle.
It means that you're going to get some sort of 400 error when you follow a link to his site with a query string attached to it. He simply will not respond to query strings that he doesn't want to handle, which is all of them.
You know I was actually really curious about this so I went back to the HTML and URL W3C standards and surprisingly they don't actually have any definitions of format other than being percent encoded. One might conflate query strings with "form-urlencoded"[0] query strings, which is one potential interoperability format, but in general a queries string is just any percent encoded string following a "?" in a url[1], and just another property in the "URL" HTML object that can be used in the generation of a response. While additionally there is a URLSearchParams object that is the result of parsing the query string with the form-urlencoded parser, this is simply an interoperability layer for JavaScript.
I'm going to be honest, I was pretty geared up to have a contrarian opinion until I looked at the standards but they're actually pretty clear, a 404 could be a proper response to unexpected query string; query string is as much part of the URL API as the path is and I think pretty much everyone can acknowledge that just tacking random stuff onto the path would be ill advised and undefined behavior.
[0]: https://url.spec.whatwg.org/#application/x-www-form-urlencod...
[1]: https://url.spec.whatwg.org/#url-class
Back in the day it was reasonably common for CMSs and forums to only have an index.php, and routing entirely by query string (in form-urlencoded form, people were not savages). So you would have index.php?p=home and index.php?p=shop. Or index.php?action=showthread&forum=42&thread=17976. It should be immediately obvious that in that scheme 404 is indeed the correct answer to unknown query parameters
In fact lots of sites still work like that, they just hide it behind a couple rewrite rules in apache/nginx for SEO reasons
If you're routing like it's 1999, sure, 404.
On the other hand, if it's a CRUD app and you're filtering a list of entities by various field values? Returning that no items matched your selection (or an empty list, if an API) makes more sense than a 404, which would more appropriate for an attempt to pull up a nonexistent entity URI.
There is no reason you can return that "no items matched your selection" with a 404 HTTP response code instead of a 200.
A response code of 204 seems more appropriate but the problem is you're not allowed to send further information, which would make that descriptive response... not descriptive enough.
The point was that returning a 404 for unexpected query strings doesn’t just happen to okay per the specs, but that there is significant historical precedent for doing so based on application design that was common in the past.
Yea, empty response at a valid path. Isn’t 204 the code for it?
Lots of REST libraries that I’ve used treat any 400 response as an error so generating a 404 when for an empty list would just create more headaches.
Libraries that automatically throw errors for status codes in the 400 and 500 ranges are pretty obnoxious (looking at you, axios). It adds unnecessary overhead, complexity, and bad ergonomics by hijacking control flow from the app.
Responses with status codes in the 400 range are client errors, so the client shouldn't retry the same request. So a 404 is appropriate despite how annoying a library might be at handling it. Depending on which language/ecosystem you are using, there are likely more sane alternatives.
204 might be acceptable if you aren’t returning an entity body to describe what is missing, but do wish to indicate the request was successful.
I think the author is comfortable creating headaches for people tacking query strings onto URLs
> It should be immediately obvious that in that scheme 404 is indeed the correct answer to unknown query parameters
That's not obvious at all. If I receive json data that contains a property I'm not aware of, i don't reject the entire document for that reason. In the case of query strings, extra query parameters might be used by other parts of the stack besides yours, so rejecting the entire document because someone somewhere else is trying to pass information to itself is the wrong approach.
> other parts of the stack
As a web developer, you’re the like the guy standing with a clipboard outside a fancy club checking if people requesting entry are allowed or not. Basically, level 1 security.
If someone is not on the list, your job is to default to declining them access, not granting them access assuming level 2 security will handle them at a deeper layer.
It’s possible that the teams you work with expect fuzzy behaviour from the website but that’s a choice, not a practice.
Oh no, looks like my old forum software urls.
watch?v=oHg5SJYRHA0
Standards are just commonly accepted behaviour that somebody chose to write down somewhere. There are a great number of commonly accepted behaviours that nobody's ever bothered to encode into a formal standard, but where failure to follow the accepted practice will result in widespread breakage. There are also a great many "standards" that you would be a fool to follow to the letter. In the OP case, the only thing that will break is people trying to visit their site, who will presumably simply press the back button on their browser and go about their day. They can decide for themselves if that is an acceptable casualty. But it isn't definitionally acceptable because no standard says it isn't (nor would is suddenly become unacceptable because a standard said it was...)
> I was pretty geared up to have a contrarian opinion until I looked at the standards but they're actually pretty clear, a 404 could be a proper response to unexpected query string; query string is as much part of the URL API as the path is and I think pretty much everyone can acknowledge that just tacking random stuff onto the path would be ill advised and undefined behavior.
This feels like a technically correct is the best kind of correct situation. Like technically, yeah web servers may respond 404 if they dont understand a query parameter, but in practise that is not how urls are conceptualized normally.
Interestingly, quite a few places that should treat query strings transparently make a lot of assumptions about their structure. We ran into that when picking a new CDN, some providers didn't handle repeat parameters (?a=1&a=2) correctly.
What’s do you mean by correctly?
Incorrectly would be processing the query string and deduping keys. Correctly would be passing it through as-is, or at least only lightly processing it, like normalizing escaping or such.
Indeed I would expect pass through with no changes.
Though there are “smart” CDNs that will resize images etc. all beats are off for those.
Something I discovered looking back at some old sites: "pages" defined by URL params don't always make it into the Wayback Machine.
Whatwg is for html, try the IEEE http rfcs
Wait until you realize that the difference between path and query string is entirely arbitrary and decided by the server. Query strings should never have existed. They are an implementation detail of CGI webservers that leaked all over everything and now smells really bad.
I dunno, it seems like the fact that we arrived at a fairly standard structure for URL paths that works pretty well is not a bad outcome.
Seems a lot better than the other potential world we could lived in, where paths were a black box and every web server/framework invented their own structure for them.
My next website is going to have the path portion of the URL be a base64 encoded ASN.1 blob.
In my current project I use URIs to refer to absolutely any entity in a git(-ish) repo. Files, branches, revisions, diffs, anything. URI turns out to be a really good addressing scheme for everything. Surprise. But the most used and abused element is always the path. Query takes a lot of that mess away. Might have been unmanageable otherwise.
https://github.com/gritzko/beagle
In fact, GitHub URIs are a good example of overusing paths: https://github.com/gritzko/beagle/blob/a7e17290a39250092055f...
... all concatenated indiscriminately.That’s not an indiscriminate hierarchy.
Grouping data by user is common and normal in computing: /home laid precedent decades ago.
Project directories are an extremely common grouping within a user’s work sets. Yeah, some of us just dump random files in $HOME, but this is still a sensible tier two path component.
The choice to make ‘view metadata-wrapped content in browser HTML output’ the default rather than ‘view raw file contents’ the default is legitimate for their usage. One could argue that using custom http headers would be preferable to a path element (to the exclusion of JavaScript being able to access them, iirc?) or that the path element blob should be moved into the domain component or should prefix rather than suffix the operands; all valid choices, but none implicitly better or worse here.
Object hash is obviously mandatory for git permalinks, and is perhaps the only mandatory component here. (But notably, that’s not the same as a commit hash.) However, such paths could arguably be interpreted as maximally user-hostile.
File path, interestingly enough, is completely disposable if one refers to a specific result object hash within a commit, but if the prior object hash was required to be a commit, then this is a valid unique identifier for the filesystem-tree contents of that commit. You could use the object hash instead of the full path within the commit hash, but that’s a pretty user-hostile way to go about this.
So, then, which part of the ordering and path selections do you consider indiscriminate, and why?
actually, instead of the object hash, you could also use the commit-hash. then the filename would be mandatory, but the url would be more readable and usable: give me the file VERBS.md as it is at commit <hash>
what would be a better way of doing that? i am not disagreeing, but i just can't think of any way to improve on this. put everything into the query part? i prefer to use the query only for optional arguments. in this example the blob argument is the only thing that doesn't fit in my opinion.
Every object in git (commit, tree, revision of a single file) has a hash that is guaranteed unique within a repository (otherwise many more things than a web UI would break) and likely also globally. I can understand wanting to isolate repositories to prevent hash collisions from causing problems, but within a repo everything has a universally unique ID.
edit: for instance, that specific VERBS.md is represented by the blob 3b9a46854589abb305ea33360f6f6d8634649108.
that's not what i meant. i was trying to suggest that the string "blob" does not fit. why is it there? why is it needed?
this should be sufficient to represent the file."blob" is like a descriptor of the value that follows. it would be like doing this:
this actually irks me every time i see it in a github urlThey are following the /key/value/key/value pattern, but the first two pairs in a GitHub URL are fixed to user and project, which lets them omit the key names. I could see them not being willing to hardcode the third pair to blob.
Back when GitHub URLs were kind of cool, github.com/user/gritzko/project/beagle would have been much less cool than just github.com/gritzko/beagle.
Back in the day there was an attempt to introduce "matrix URIs" as a more structured alternative to query strings: https://www.w3.org/DesignIssues/MatrixURIs.html
Of course there's nothing to stop you using URIs like this (I think Angular does, or did at one point?) but I don't think the rules for relative matrix URIs were ever figured out and standardised, so browsers don't do anything useful with them.
Nothing you said here is correct. Paths, query strings, and fragments are all well defined entities. https://datatracker.ietf.org/doc/html/rfc3986#section-3.3
It’s a string between ? and # isn’t well defined. Or it is and it says very little.
Not entirely arbitrary - forms that use the GET method instead of POST will append form values as query params.
For sites without Javascript, it's great for things like search boxes, tables with sorting/filtering, etc. instead of POST, since it preserves your query in the URL.
https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...
It has always amazed me how much trouble the SPA folks are willing to go to in order to slowly rebuild just normal boring URLs with querystrings because users demand deep linking and back buttons and the like.
Or you could accept that you're probably going to need a round trip to the server and use a normal URL and it's fine.
For all but the absolute biggest websites in the world, anyhow. At Facebook or Google scale yeah it's needed.
Query strings existed before CGI did, and the way they're defined to be filled in from web forms is quite useful; I wouldn't want to need Javascript to fit that into path format. There's nothing wrong about having things decided by the server; I don't get that part of your argument at all.
Maybe dumb question: how does the server “decide” anything other than what file to serve? Today we have many choices but back in the day CGI was the first standard way to do it.
So yes query parameters existed before CGI but to use them you had to hack your server to do something with them (iirc NCSA web servers had some magic hacks for queries). CGI drove standardization.
Forgive the phone-posting.
and what server software is running this code in 1995?
CL-HTTP or AOLserver
sure looks like VB there, what’s the plugin? Didn’t see anything like that before.
It's arbitrary to a degree like the difference between using an attribute or child element in XML, but it's not entirely arbitrary. If you want to include data in the URL that's not part of the hierarchy of the path, query strings are good for that.
How do you figure?
Paths are hierarchical; query strings are name/value.
(Note I speak of common usage.)
You can create a different convention, but that one is pretty dang useful.
> It is a small, decentralised, self-hosted web console that lets visitors to your website explore interesting websites and pages recommended by a community of independent personal website owners.
Back in the Stone Age, we called these “Webrings,” but they weren’t as fancy.
One of the issues that I faced, while developing an open-source application framework, was that hosting that used FastCGI, would not honor Auth headers, so I was forced to pass the tokens in the query. It sucked, because that makes copy/paste of the Web address a real problem. It would often contain tokens. I guess maybe this has been fixed?
In the backends that I control, and aren’t required to make available to any and all, I use headers.
> an open-source application framework, was that hosting that used FastCGI, would not honor Auth headers
So you were writing your application as a fcgi-app, and (e.g.) Apache was bungling Auth headers? Can you expand on this? Curious about the technical detail of (I guess) PARAM records not actually giving you what you expect?
I don’t remember, exactly. Long time ago (I stepped away from that project many years ago).
I just remember the auth headers never showing up in the $_SERVER global (it was a PHP app). This was what I was told was the issue. They made it sound like it was well-known.
Query strings do have uses (such as for searching files and some other kind of dynamic files), but you shouldn't add them to URLs that should not expect them. So, I agree that they would be right to refuse requests with UTM and other stuff added like that.
I think 404 probably makes the most sense as the response if a query string is not expected but is present anyways, although 400 might also be suitable.
>So I’ve decided to try a blanket ban for this site: no unauthorised query strings.
His site returns (I think incorrectly) a 414 if a request includes a query string. If this protest is meant to advocate for the user, who presumably wasn't able to manage that string in the first place, why would you penalize them for it being there?
Why not just use it as a cue to tell users how they can make this decision themselves (e.g. through browser tools)?
"You could argue that I’m abusing 414 URI Too Long. I respond that it’s funnier this way. Other options I considered were:
https://chrismorgan.info/no-query-strings?fooI don't think it's an abuse, RFC9110 defines 414 as a response for "refusing to service the request because the target URI is longer than the server is willing to interpret". Since adding a query string involves only adding characters, this seems fine; there's no stipulation as far as I can tell that all pages a server hosts must adhere to the same length. I'd be curious if any well-known clients interpret it that way though, and make caching decisions based on it. As far as I know, they shouldn't.
Obviously it's against the spirit of the thing, but I don't think it's wrong per-se.
If the goal is to be misleading, but technically correct, it hits the bullseye
When the goal is "the funniest way", I think that's a hit :)
You could also redirect to the url with the query string dropped.
Also from the 414 page:
>Complain to whoever gave you the bad link, and ask them to stop modifying URLs, because it’s bad manners.
It's ironic that an error response so blatantly violating the robustness principle is throwing shade about bad manners.
Opinions vary on how good an idea the robustness principle is. That is why, for example, the XML standard requires a conforming validator to throw an error on invalid XML.
In our modern world, the robustness principle has become an invitation to security bugs, and vendor lock-in. Edge cases snuck through one system on robustness, then trigger unfortunate behavior when they hit a different system. Two systems tried to do something reasonable on an ambiguous case, but did it differently, leading to software that works on one, failing to work on the other.
I generally agree, but I don't think XML is the best example. Getting HTML out of XML is considered to have been the right move isn't it? I was pro-XHTML2 at the time but in retrospect, have we suffered much for not sending webpage validation errors to end users?
Once people have gotten used to not having to conform, forcing them to conform is an uphill battle. Doubly so when, as happened with Microsoft and IE, the vendors would like to encourage vendor lock-in. The only time to reasonably do it is at the start.
That said, we are paying a huge complexity cost due to our efforts to allow nonconforming pages. This complexity is widely abused by malicious actors. See, for instance, https://cheatsheetseries.owasp.org/cheatsheets/XSS_Filter_Ev... for ways in which attackers try to bypass security filters. A lot of it is only possible because of this unnecessary complexity.
But, this is robust? I mean it's pretty clearly stating that you are visiting an unsupported URL. It provides direction on what to do about it to the user. It does not crash the browser or the server. In pretty much every dimension this is highly robust.
The robustness principle is itself bad manners, in plenty of contexts. If I deliver packages by throwing them at the customer, I really want a customer to tell me "hey, don't throw packages at me!" before I attempt to lob something fragile and breakable, or something heavy at someone fragile and breakable. Otherwise, how am I supposed to learn that I'm doing anything wrong?
This is the point this has delved into internet crankery.
No, it's a fairly uncontroversial take. See https://en.wikipedia.org/wiki/Robustness_principle#Criticism and the "See also" section.
It's been years but I seem to remember there was a version of PLSQL server pages that would return 500 if you tried to pass in an unknown query string.
> You could argue that I’m abusing 414 URI Too Long. I respond that it’s funnier this way. Other options I considered were:
Another option to consider is "418 I'm a teapot": teapots usually also don't support query strings
Just straight "400" ("Bad Request") or "403" ("Forbidden") would also probably be defensible. Odd that there aren't any error response codes specific to URI parameters.
Several options which seem like they might be appropriate aren't on close examination:
- "406" ("Not Acceptable") which is based on content-negotiation headers.
- "409" ("Conflict") which is largely for WebDAV requests.
- Others such as 411, 422, and 431 are also for specific conditions which aren't relevant here.
- 300 or 500 errors are inappropriate as this isn't a relocation or server-side failure, it's a client-side request problem.
Teapot or too long seem best bets.
I think either 400 or 404 would be fine. 400 because the request isn't in the expected format, 404 because a resource with that query string doesn't exist.
> 409" ("Conflict") which is largely for WebDAV requests.
I’ve always used them in API servers when a client was POSTing to create a duplicate of a unique item.
I'm willing to pay them $1 for a contact guaranteeing that they won't service such requests. That would make 451 the most appropriate.
Just fire off a 200 OK with text body of "499 Bad query string"
Im not making this up btw. A old NOC I woeked at emitted every error as 200 OK with the body message with the real error. They were a real shitshow.
Of course they do. For example you can lower a string from the top to query the fill level. Or you can wrap a string around the pot to query the circumference.
The tone of this and Chris's post gives me the impression that it's harmful to include these query parameters, but I don't understand how. Could someone elucidate me? I understand it can mangle some URLs and that's good enough reason not do it, but even then it seems like a minor incovenience.
You can read some of the issues people have had with this by reading up on the http referer header: https://en.wikipedia.org/wiki/HTTP_referer
There are a lot of reasons I might not want a site to know where I came from to get to their site. It is basically sharing your browsing history with the site you are visiting.
Because of this, there have been a lot of updates to the http referer header, with restrictions on when it is sent, and an ability to opt out of the feature entirely.
Adding a url parameter with the same information bypasses any of these existing rules and ability to opt out. They should just use the standard.
If I send out an email campaign, I can't use custom http headers to know that a user arrived from the newsletter.
If you are sending out an email, you can use whatever url form you like?
This is talking about links to third party sites, not your own.
Do you really need to? Basic statistics will tell you if the email campaign had any significant effect on site visits.
If I release a video and send an email newsletter at the same time, which one caused the traffic increase? Should I invest in making more videos of sending more emails?
If you insist on knowing, include a different url in both that goes to the same place and use your damn server logs. You don’t need google analytics and whatever.
Isn't putting in a different query string "including a different url that goes to the same place"?
Isn't this functionally the exact same?
presumably you control the urls you are sending in the email. As a result if you want to use query strings that's fine. The issue only arises when you use query strings to implement tracking on someone else's site instead.
use a unique url for each email
As your reader, I might not actually want you to know.
What's interesting is that none of these sites have a "search" feature. Which is an important accessibility feature and a clear and legitimate use case for a query string.
> If I ever start using any query strings, I’ll allow only known parameters.
They aren't saying the concept of query strings are bad, They're saying unsolicited query strings during referal are the issue.
My website has search without a query string: https://www.jeremykun.com/
There is no reason at all.
You could simply throw the information away.
It's a ridiculously extreme stance and lacks proper explanation how this will lead to a better web.
It may be ridiculous to some, but Chris is totally right - it’s his website and he can do what he wants with it!
Oh, I have a couple - the users did not agree on being tracked (these query params are tracking information), and the site administrator does not want incoming traffic to be tracked. I know the latter can be hard to understand, but I for example sure as hell do not want to have any info in my logs that can be used to harm my users.
On a more personal note, I hate it when I go to copy a link to send via a message, and the tracking code glued onto it is twice as long as original URL... I either have to fiddle around with it to clean it up or leave the person I sent it to to wonder wtf am I on about with a screenful of random characters...
So it's violating users' privacy, it's shit UX, and on top of that, nobody asked for it...
>(these query params are tracking information)
Query strings are useful for way more than just tracking. Saving and servicing search queries is a way more common use case. So assuming it's only useful for tracking is very misleading.
Query strings are probably the least invasive tracking. They are transparent, obvious, and anonymous. Users are free to strip out and edit query strings if they don't want them.
More to the point, I can essentially do the same thing with HTTP routing - create an infinite number of unique URLs for tracking purposes. In that regard calling out query strings specifically for essentially the same thing but more transparently seems like splitting hairs.
Yeah I do not see an alternative way to easily copy paste links with things like filter settings saved.
Filters especially make sense as query params as they are non sequential but still visually readable as to what they do.
URL slugs make sense for sequential pages that are hierarchical but make no sense for non hierarchical data/routes.
Services can force tracking into links by encoding the whole url into a shortlink that makes it impossible to just remove the tracking alone as everything is encoded into a shorter non editable string.
Thank you for explaining to me that query parameters can be used for other purposes apart from tracking. The articles in question though, are railing against query parameters being abused for tracking purposes - passing referers (sic) and UTM by adding them to URLs of sites that neither process them, nor want them.
Referral query strings are not for tracking though. The person putting them on the links gets nothing out of them. There is no PII being shared. They are purely added out of courtesy.
If I am handing out maps to your address, letting people know who is publishing the map is generally a good thing.
This is like saying having a return to sender address on mail is an invasion of privacy.
So my understanding is, he is annoyed that other website adds a query string such as "?ref=origin.com" to links pointing to authors website.
How does this benefit the other website? How does this hurt the authors website?
I am completely confused about the behavior of both side here.
I get that when I run an ad-campaing I want google to add a utm-query string, so I can track which campaign users arrived from - but then the origin and the destination are working together. Here the origin just adds stuff for no reason. Why?
Since the original source hadn't had a discussion on HN yet, I've put that link (https://chrismorgan.info/no-query-strings) at the top and moved the response link (https://susam.net/no-query-strings.html) to the toptext.
Both are good but it seems fair to give priority to the original.
Maybe an alternative would be to inconvenience people following such links still, but somewhat less.
Instead of responding with an error, give a page that states “The link you followed to get here appears to have had some tracking gubbins added, in case you are a bot following arbitrary links, and/or using random URL additions to look like a more organic visit, please wait while we run a little PoW automaton deterrent before passing you on to the page you are looking for.” then do a little busy work (perhaps a real PoW thingy) before redirecting. Or maybe don't redirect directly, just output the unadorned URL for the user to click (and pass on to others). This won't stop the extra gubbins being added of course, but neither will the error and this inconveniences potential readers less.
While I don't take the author's hard stance, I do hate gratuitous query params that result in links that are thousands of characters long.
I use this bookmarklet to strip query params before sharing a link:
edit: not true https://news.ycombinator.com/item?id=48077990
"I don’t like people adding tracking stuff to URLs" and "You abuse your users by adding that to the link" and "no unauthorised query strings" and "At present I don’t use any query strings" but for some reason ?igsh, which i'm pretty sure is an instagram tracking parameter, is allowed. weird
?igsh doesn't get through for me, and I don't see any links on the page including it.
oh, it's uBlock Origin (non-lite) removing it without telling me at all. retracted
"wander console" sounds like they're just web rings re-invented. In the era of forced feeds by giant corporations which consist of the things they want you to see, I've wondered if this old idea would make a comeback. Human curated content from trusted people seems like the only way forward.
FTA: It is also a bit like web rings except that the community network is not restricted to being a cycle; it is a graph and it is flexible.
Is it not a random walk? Might sound pedantic but if there is graph structure I am interested.
I'll start with the clarification that the moderators have changed the URL of the original post from <https://susam.net/no-query-strings.html> to <https://chrismorgan.info/no-query-strings>. Hopefully, this will prevent any confusion about why we are discussing random walks in a post about query strings. Now let me answer your question.
> Is it not a random walk? Might sound pedantic but if there is graph structure I am interested.
The network is a directed graph. Every Wander Console declares a few other consoles as its neighbours. The person setting up the console decides who they want to list as their neighbours. So if we call the network graph X, then the set of vertices is:
and the set of directed edges is: The traversal between consoles is not strictly a random walk. If I could call it something, I would call it randomised graph exploration with frontier expansion. On each click of the 'Wander' button, the tool picks one console at random from the set of discovered consoles and visits that console. It then fetches the neighbours declared by that console and adds any newly discovered consoles to the set.The difference from a random walk is that the next console is not chosen from the neighbours of the last visited console. It is chosen from the whole set of consoles discovered so far. In other words, each click expands the known part of the graph, but the console used for that expansion is selected randomly from all discovered consoles, not just from the last console visited.
This url worked fine:
https://chrismorgan.info/no-query-strings#:~:text=So%20I%E2%...
but this one was too long:
https://chrismorgan.info/no-query-strings?a=1
Doh! The part past the # does not go to the sever, so that wasn't a longer URL. How about:
https://chrismorgan.info/%6e%6f-%71%75%65%72%79-%73%74%72%69...
This is not the first site to do so. A few years back, scarygoround.com started blocking query strings, although it seems to have stopped doing so now. Back then, Facebook had started to add ?fbclid=... to every outgoing link.
Trying to boostrap some taboo against novel unpermissioned URL munging is silly prudishness.
Ensuring both sides of a hyperlink agree/consent was a design flaw that limited the uptake of pre-web hypertext systems. The web's laissez-faire approach demonstrated a looser coupling was far better for users, despite all the new failure modes.
Of course any site/server has the practical power free to treat inbound requests as rigorously (or harshly) as they want. But by the web's essential nature, it is equally part of the inherent range-of-freedom of outlink authors to craft their URLs (and thus the resulting requests) however they want. URLs are permissionless hyperlanguage, not the intellectual property of entities named therein.
Plenty of sites welcome such extra info, and those that don't want it can ignore it easily enough – including by just not caring enough about the undefined behavior/failures to do nothing.
Though, when a web publisher has naively deployed a system that's fragile with respect to unexpected query-string values, they should want to upgrade their thinking for robustness, via either conscious strictness or conscious permissiveness. Thereafter, their work will be ready for the real web, not a just some idealized sandbox where scolding unwanted behavior makes sense.
There’s nothing ruder in hypertext etiquette than giving someone a link to navigate to someone else’s HTTP server, where you have manipulated that URL in some way unsanctioned by the server you are sending them to.
You can’t just send arbitrary query string parameters to a server and assume they will just ignore them. Just like you can’t just remove query string parameters and assume the URL will work.
In fact, you usually can just send arbitrary query string parameters to a server - that's why the behavior is so common, and often useful.
Most sites don't mind or break, some sites get value from the behavior in ways hard to replicate in other ways – and those sites that don't like such additions can easily ignore them. And a few lines of code will work better than ineffectually appealing to manners, when the freedom of the web's form of hypertext, and protocols, gives the outlink authors full freedom to craft URLs (and thus requests) however they like.
Crafting outbound links with your own additions and handing them out to visitors to your site is similar to the practice of writing someone’s phone number on the door of a bathroom cubicle with ‘for a good time call:’ written above it.
You’re handing out someone elses’s contact details, but giving the person you hand them to a completely fabricated expectation for how the interaction will go.
My use case for this is making separate bookmarks in different folders for a single URL:
Example.com/interesting -> bookmark folder one
Example.com/interesting?dummy=t -> bookmark folder two
Use #fragment identifiers then
Adding query strings is one of those things that I think a lot of sites could get away with more easily if they were reasonable about it.
A link that is "https:// web.site" is fine.
A link that is "https:// web.site?via=another.site" is fine.
A link that is "https:// web.site?fbm=avddjur5rdcbbdehy63edjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63edaaaddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednzzddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63ednddjur5rdcbbdehy63edn"
is annoying as shit and I need to literally apologize to people after sending it if I forget to manually redact the query string. Don't abuse this.
There are addons to remove unnecessary params from the worst offending sites:
https://www.google.com/search?q=clearurls+addon
Newer versions of Firefox also have a "Copy Clean Link" option in the right-click menu.
https://support.mozilla.org/en-US/kb/enhanced-tracking-prote...
Thanks for removing the rest on that google link, the one I get after switching to "images" and back to "web" is this monstrosity:
https://www.google.com/search?newwindow=1&sca_esv=8061bd9cb1...
Edit: which luckily and sensibly Hacker News cuts short since it's 463 characters
You can post the string as text (indent by 2+ spaces) to avoid that trimming.
Since the purpose is to show the full URL with trackers and other cruft, that's sensible here:
And yeah, that's pretty awful.In conclusion, Google must be destroyed.
Yeah I removed the rest of the link as an example of how much cleaner the urls can be with that addon haha. I was being meta
>Right click a youtube video from the results to copy the URL. I would have liked a short URL ready to share with people in chats, but no, I get: https://www.youtube.com/watch?v=IFfLCuHSZ-U&pp=ygUNcmF0Ym95I...
>Want to share an amazon product on a chat to discuss about it. I would have liked a nice short url that I can copy, instead I get a monstrosity, it forces me to manually select only the id portion of it if I want to share it.
A neat and funny idea - but in the end it is hostile to the users who don’t always control what’s added to links.
Just referrer policy of strict origin when cross origin gives host level referer (sic) header in most mainstream browsers unless user has configured otherwise right? That’s usually enough for web authors to know what audience they’re appealing to and privacy-maximizers can turn off that header sending.
YouTube is also quite famous with their source identifiers, especially with the short urls, the tracking part is longer then the url I’m trying to share.
This is cool and creative!
It uses 4xx, but not just 400 :)
https://chrismorgan.info/no-query-strings?why=unknown
I was just wondering if I should do something like this. I use a couple query string values and I validate them and issue a 40x if the value is invalid. So, I was wondering if I should issue a 40x for an unused query string val.
Query strings break unpredictably, and that alone is enough to ban them by third parties, especially for something as minor as referral tracking.
Example: The Browser is a well known link aggregation paid periodical. I subscribe, and every 1 in 10 or 20 links I clicked, it'd just break outright and I'd have to tediously edit the URL to fix it (assuming the website didn't do a silent ninja URL edit and make it impossible for me to remember what URL I opened possibly days or weeks ago in a tab and potentially fix it). This was annoying enough to bother me regularly, but not enough to figure out a workaround.
Why? ...Because TB was injecting a '?referrer=The_Browser' or something, and the receiving website server got confused by an invalid query and errored out. 'Wow, how careless of The Browser! Are they really so incompetent as to not even check their URLs before mailing an issue out to paying subscribers?'
I wondered the same thing, and I eventually complained to them. It turns out, they did check all their URLs carefully before emailing them out... emphasis on 'before', which meant that they were checking the query-string-free versions, which of course worked fine. (This is a good example of a testing failure due to not testing end-to-end or integration testing: they should have been testing draft emails sent to a testing account, to check for all possible issues like MIME mangling, not just query string shenanigans.)
After that they fixed it by making sure they injected the query string before they checked the URLs. (I suggested not injecting it at all, but they said that for business reasons, it was too valuable to show receiving websites exactly how much traffic TB was driving to them on net, because referrers are typically stripped from emails and reshares and just in general - this, BTW, is why the OP suggestion of 'just set a HTTP referrer header!' is naive and limited to very narrow niches where you can be sure that you can, in fact, just set the referrer header.)
But this error was affecting them for god knows how long and how many readers and how many clicks, and they didn't know. Because why would they? The most important thing any programmer or web dev should know about users is that "they may never tell you": https://pointersgonewild.com/2019/11/02/they-might-never-tel... (excerpts & more examples: https://gwern.net/ref/chevalier-boisvert-2019 ). No matter how badly broken a feature or service or URL may be, the odds are good that no user will ever tell you that. Laziness, public goods, learned helplessness / low standards, I don't know what it is, but never assume that you are aware of severe breakage (or vice-versa, as a user, never assume the creator is aware of even the most extreme problem or error).
Even the biggest businesses.... I was watching a friend the other day try to set up a bank account in Central America, and clicking on one of the few banks' websites to download the forms on their main web page. None of the form PDF download links worked. "That's not a good sign", they said. No, but also not as surprising as you might think - the bank might have no idea that some server config tweak broke their form links. After all, at least while I was watching, my friend didn't tell them about their problem either!
I don't see how your example, The Browser (thebrowser.com), supports your argument that ad-hoc query-string additions are so prone-to-breaking that 3rd parties should ban them.
In fact, the example seems to suggest the opposite: a 17+ year successful paid subscription business – to which you appear to be a generally-satisfied customer! – receives enough "business value" from the practice, despite its failure modes, they don't want to stop. Improving their probe of the risk-of-failure was enough.
Seemingly, the practice works often enough, pleasing more destination sites than it angers, that "referral tracking" is not something "so minor".
> Improving their probe of the risk-of-failure was enough.
The point was it was dangerous in a way they didn't even realize was an issue, for a thin business rationale. Unless you are going to do thorough tests and understand the risk you are taking (which they did not, as evidenced by screwing it up systematically at scale for years), you should not be doing it.
And it's not obvious that they are correct in their tightened-up testing, because even if a link is correct at the time they test it, it could break at any time thereafter.
> to which you appear to be a generally-satisfied customer!
No matter what _X_ is, _X_ would have to be a pretty epic screwup to make a customer unsubscribe solely over that! I never claimed it was such a major epic screwup that it could do that. So that is an unreasonable criterion: "well, you didn't outright quit, so I guess it can't be that bad." Indeed, but I never said it was, and somewhat bad is still bad; I was in fact fairly annoyed by the random breakage, and at the margin, everything matters. If TB did a few other things, in sum, they could potentially convince me to let my subscription lapse. An annoyance here, a papercut here, and pretty soon a generally-satisfied customer is no longer so satisfied...
Referrer is sometimes nice to know. If your site gets a traffic spike from an email newsletter that traffic won't correctly identify the source in the http headers.
No qualms with OP, your site your rules.
It's just a string though. A project that I'll never get to is a custom webserver so that QR codes can use the smaller characterset, so it can link to a URL with parameters without forcing the larger character set.
Query strings are awesome. Especially for one-page applications.
I build a lot of internal applications, and one of my golden UI rules is that a user should be able to share their URL and other users should be able to see exactly what the sender did.
So if you have a dashboard or visualization where the user can add filters or configurations, I have all of their settings saved automatically in the URL. It's visible, it's obvious, it's easy, it's convenient.
>There is also a moral question here about whether it is okay to modify a given URL on behalf of the user in order to insert a referral query string into it. I think it isn't.
These dogmatic technical screeds are all so weird to me. They usually reveal more about the authors lack of experience or imagination than provide a useful truism.
Yes, query strings often enable useful features! But Chris's post, "no unauthorised query strings", is only regarding third parties adding them.
But... like... that's a weird hill to die on.
> If I wanted to know I’d look at the Referer header; and if it isn’t there, it’s probably for a good reason. You abuse your users by adding that to the link.
The reason is that the referrer headers are a usability and privacy nightmare. It's weird for the author to jump to such a conclusion.
This referral information is being done purely as a courtesy to the webhost. If we imagined a world in which ChatGPT or Wikipedia launched massive hugs of death on referral links without attributing themselves, that is a much, much worse outcome.
There's a referrer header, if the client wishes to send it. If they don't, the "courtesy to the web host" is done at the expense of the client. This particular web host takes umbrage at other sites taking advantage of their clients that way, which seems reasonable to me.
A relatively minor impact concern is that query strings create a new cache entry both in the browser and typically on server-side caches unless configured otherwise, so you might want to use URL fragment parameters if the parameters are only used by clientside JavaScript but the server response is the same.
This is genius, kudos Chris.
It also makes me wonder what other noxious online behaviours might be addressed through ... creative ... client-side responses similar to this.
We've already seen, for years, sites attempting to socially-condition people over the use of ad-blockers and Javascript disablers. No reason why the Other Side can't fight back as well.
But ?fbclid is not banned?
It is. Your ad blocker is removing it, so it doesn't trigger the error page.
Running your own small website is a constant battle against grifters and bad online etiquette. When people hotlink images, I usually make a point of having some personal fun with mod_rewrite.
> After I implemented that feature, a page from one of my favourite websites refused to load in the console... the third URL returns an HTTP 404 error page. The website uses the query string to determine which one of its several font collections to show.
Yes, let's unilaterally decide that query strings are bad because one website (ab)uses query strings to load different fonts.
It's the query strings that are the problem, not the website!
jfc.
Look, I'm against utm fragments as much as the next guy, but let's not throw away a perfectly good thing because tracking is evil.
Adding your own garbage to someone else's URLs is in fact the problem. Could they handle your garbage better? Sure. Is your garbage still a problem? Yes.
Postel's law worked OK when people operated in good faith. But today the internet is full of abusers. Rejecting requests that aren't exactly what they should be is probably the best policy now.
Postel's law is typically stated as "be conservative in what you do, be liberal in what you accept from others". It's unfortunately common for people to ignore the first half and hallucinate a third clause demanding that the recipient stay silent about the errors they receive.
That website is not abusing query strings, though, its usage of query strings is perfectly cromulent. And tfa is not saying not to use query strings, but not to append random garbage to other people's URLs.
The website uses the feature for its intended purpose. Adding random trash to the query string of another website assuming it'll ignore it is in fact a bad idea, always, even if you can usually get away with it.
The problem is adding query strings to the URLs of others. It's peak entitlement to think that's proper
> one website (ab)uses query strings
Really not abusing abusing query strings from a standards perspective, a 404 is not an improper response to an unexpected query string
This is really cool. My site is hosted by cloudflare, so I guess I could do the same with a cloudflare worker... maybe?
I mean…the site that broke should know what to do with arbitrary query strings. If your site breaks when someone puts in an invalid query string, that’s on you?
This. Query strings are a standard feature and have many more purposes beyond tracking.
Yes, and if the site actually used query strings, then it would of course accept them. Why does it have any reason to accept invalid query strings?
> It’s my website: I can do what I want with it.
> And you can do what you want with yours!
That does not make a lot of sense. Yes, you can do what you want with your website, but query-string is a way for users to query for additional information or wants or needs. I use them on my own websites to have more flexibility. For instance:
That will download the website content as a formatted .pdf file.I can give many more examples here. The "query strings are horrible" I can not agree with at all. His websites don't allow for query strings? That's fine. But in no way does this mean query strings are useless. Besides, what does it mean to "ban" it? You simply don't respond to query strings you don't want to handle. We do so via general routing in web-applications these days.
> "query strings are horrible"
That's not at all what the article says. You're responding to a weird strawman that doesn't resemble the article's actual point.
> foobar.com/ducks?pdf
This isn't relevant when talking about links to his site. This is relevant when talking about links to your site.
> Besides, what does it mean to "ban" it? You simply don't respond to query strings you don't want to handle.
It means that you're going to get some sort of 400 error when you follow a link to his site with a query string attached to it. He simply will not respond to query strings that he doesn't want to handle, which is all of them.