DST was a mistake and it needs to die. It solves no problems and only creates them. And not even the type of problem that someone could profit from, just plain old complete waste of time for everyone problems.
Just stick to winter time (because that's the correct one) and adjust all working hours once and for all to be like summer time, for e.g. instead of business opening at 7AM and closing at 10PM they can open at 6AM and close a 9PM. There will be a period of adjustment, but we have a period of adjustment twice a year already, so nothing is lost.
Why match working hours to summer time? Because I want to have more sunlight by the time I'm done with work. Especially during winter. We have it completely backwards, if we are going to adjust our clocks we should adjust them in a way that gives us more sunlight during winter, not less. I don't care if it's dawning by the time I'm going to work where I will be stuck indoors for nine hours, I want sunlight when I'm free again.
Everybody think DST is the worst and needs to be replaced in some way or another. Most people don't realize that their proposed "solution" has already been tried. Your particular solution of "year-round DST" was tried in America in 1973-1975. I'll just quote Wikipedia:
> During the 1973 oil embargo by the Organization of Arab Petroleum Exporting Countries (OAPEC), in an effort to conserve fuel, Congress enacted a trial period of year-round DST (P.L. 93-182), beginning January 6, 1974, and ending April 27, 1975.[17] The trial was hotly debated. Those in favor pointed to increased daylight hours in the summer evening: more time for recreation, reduced lighting and heating demands, reduced crime, and reduced automobile accidents. The opposition was concerned about children leaving for school in the dark and the construction industry was concerned about morning accidents.[18] After several morning traffic accidents involving schoolchildren in Florida, including eight children who were killed, Governor Reubin Askew asked for the year-round law to be repealed.[19]
> Over three months from December to March, public support dropped from 79% to 42%.[19] Some schools moved their start times later.[19] Shortly after the end of the Watergate scandal caused a change of administration, the act was amended in October 1974 (P.L. 93-434) to return to standard time for four months, beginning October 27, 1974, and ending February 23, 1975, when DST resumed.[18][20] When the trial ended in October 1975, the country returned to observing summer DST (with the aforementioned exceptions).[12]
Outdoor lighting is a lot cheaper now than it was in the 1970s. I think we can give it another shot after 50 years. And it's worth pointing out that Arizona has gone without DST for the last 50 years and seems to be doing fine.
Interestingly part of the UK approach then was to make street lighting more efficient, around that time a lot of low-pressure sodium lamps were installed. They used so little energy they were only beaten for efficiency by LEDs in this decade, but the monochromatic yellow light was seen as unacceptable by some countries which continued to use inefficient high-pressure mercury then later high-pressure sodium.
I miss the humble SOX lamp to be honest, they made night look like night rather than a poor approximation of day. They also had benefits for wildlife, much of which is insensitive to the 589 nm wavelength as well as astronomy where the light is easily filtered out.
France has year round DST and an hour on top during summer since 1940 and far less car accidents than the USA both in the morning and in the evening. So does Spain and Portugal but I'm too lazy to check since when. I don't think automobile accidents is a very good metric to evaluate the interest.
It's basically a trade off between light in the morning and the evening. When Britain tried, they saw that it was mainly impactless with regards to the total number of accidents. They still reverted.
> France has year round DST and an hour on top during summer since 1940 and far less car accidents than the USA both in the morning and in the evening.
In 2019, the European Sleep Research Society (ESRS), the European Biological Rhythms Society (EBRS), and the Society for Research on Biological Rhythms (SRBR) wrote a joint statement to the European Commission advocating for permanent establishment of a more natural time.
Actually, the document only argues for permanent CET and ending DST. It does not mention change of time zones.
The logic would be sound however. Social Jetlag is real. I for one loath DST and switching back to CET noticably improves my sleep and overall well being.
Have we tried my solution of having a continuous curve of time adjustments instead of one big discrete jump? I think that would have been awful in the day of manually set, analog clocks, but surely in 2025 when everything's digital and many things are connected it's totally possible, no?
My fist reaction was that it would be immediately rejected, but the more I thought about it, it seems like it could work if you moved clocks BACK 5 minutes at the START of each MONTH from July 1st to December 1st, and then FORWARD 5 minutes starting from January 1st to June 1st.
No, the more I think about this, PROGRAMMERS will HATE this.
I’m also not resetting the manual clocks around my home and vehicle every damned month.
I genuinely dont care if we pickstandard time” or “savings time” … i just want my year-round circadian rhythm to not get fucked up twice a year - it takes me so long to get used to the new one and then it gets rug pulled again.
It's still awful. The only devices I own that would give me a reliable time, would then be my phone, laptop, desktop, and maybe my TV. My microwave, oven, thermostat, alarm clock, car, watch, grandfather clock, etc would all be wrong.
You would wreak unmeasurable havoc across the target country.
Eh, it's like 5 minutes a month they would be off by. Eventually all those devices will be Internet connected anyways and we'll all have something else new to rage over.
The only "advantage" to that is that companies would sell a lot more devices as people end up realising that their phone/central heating/doorbell/dashcam etc doesn't get any OTA upgrades and is now almost always showing the wrong time.
So you propose we stop shifting the clock but shift every opening and closing times instead producing exactly the same effect with basically no difference.
And you want to abolish DST but the end of your comment is that you actually want even more shift including in winter.
I fear I have trouble following your reasoning and understanding what you actually want.
Yes shift the opening and closing times of everything. Get everyone to coordinate, so that your kids' school time doesn't shift relative to when you need to be at work, not to mention everything else where two different organizations have to do something that regularly happens at a particular time. Give up eliding this whole problem so that some programmers have an easier time scheduling backups or database compaction.
In fact, as some have suggested, we should also give up time zones and just use UTC for everything. That way when you have to schedule a Zoom call a few thousand miles away, there is no confusion as to the time. Some idiot might retort that since people like to sleep when it's dark, that instead of looking up the time zone difference, you'd have to look up what time the sun rises where they live.
I say screw them. Half of the world should just learn to live with sleeping during the "day" and go to work at "night." We've got electric lights now. It's a small price to pay so that programmers don't have to do time zone conversions.
If we're talking about radical time-related ideas, mine is to shift midnight to be the current 4AM. The reasoning is that many events last until past midnight, while 4AM is more or less the time when both the party and the work people are asleep.
also obviously remove support for 12hour clocks from everything, who the duck thought having two different 4 O'Clock times every day makes sense. And what the insanity is the nonsense that write 12 mean 0 a.m. I.e. going from 11:59am goes to 12pm, like seriously who thought clocks braking basic math is a good ideaf!?!? Like the rang [11:59am;12am] is 12 hour and 1 minute wide instead of just 1 minute even through it uses the same unit.
Whilst we're at it, let's have 100 seconds per minute and 100 minutes per hour. I'm not sure whether I'd prefer 10 or 100 hours per day, though maybe 10 hours would be sufficient.
> So you propose we stop shifting the clock but shift every opening and closing times instead producing exactly the same effect with basically no difference.
Lol no, that would be indeed even worse. I want to do the shift once and forever. Well, first and foremost I want to get rid of the twice-a-year switch. The question that usually arises then is "should we be permanently on standard time or on DST", to which my proposal is standard time (because it's correct) but shift working hours once and forever so we have more sunlight all year around. During summer it would be the same as it already is and during winter we have one more hour of sunshine, which is the time of the year when I want sunshine the most.
There is no correct or wrong time, it's conventions that do not apply generally. Some countries have some timezones where summer time is more "correct" than winter time.
Therefore why not just stick to summertime and skip the whole "change every opening hours" thingy ?
I think PC is proposing using the natural time zone, and adjusting hours if you need sunlight to operate. Summer time is awful because different latitudes have different hours of sunlight, yet all are forced to the least common denominator.
Cities closer to the poles might want to adjust more than one hour, or none at all since they see little sunlight for months anyway. Cities near the equator might not need to adjust at all. Businesses that need to be synchronized could still coordinate their operating hours. It's the most natural approach is DST had never existed.
A lot of the conflict over which of daylight and standard time is better would be solved by allowing people more freedom to choose their own working hours. A lot of the rhetoric around it seems to be about what the optimal sleep-wake schedule is, which strangely tends to match up with the speaker's own sleep-wake schedule.
I don't want to be petty, but there is no 'winter time', only standard time and summer time or 'daylight saving time'.
I first heard the term 'winter time' when it was discussed to decide weather to keep DST permanently or if people would like to keep 'winter time' always. And of course who would want to have winter-something always. ;)
If we're being really pedantic, in the U.S. context the zones observe "standard time" all year. "Standard time" refers to the standardization across the zone, and the practice of advancing the clock during daylight-saving time changes each zone's standard time. The Unix-style usage of "EST" vs. "EDT" isn't pedantically correct (e.g., New York observes "eastern standard time" even in summer).
> and the practice of advancing the clock during daylight-saving time changes each zone's standard time.
Even more pedantically, “standard time” is not necessarily consistent across each zone (particularly, during the period for which in parts of the zone it is advanced by an hour) since "standard time” only advances for those states, or parts of states, for which an exemption is not in place.
So, the Unix-y convention of using PST for "Pacific Standard Time without advancement", and PDT for "Pacific Standard Time with advancement" is the simplest way of getting meaningful concise labels out of the US legal scheme. (This is only a theoretical issue for some US timezones, but it is a concrete one for at least the Pacific and Hawaii-Aleutian Time Zones.)
> Even more pedantically, “standard time” is not necessarily consistent across each zone (particularly, during the period for which in parts of the zone it is advanced by an hour) since "standard time” only advances for those states, or parts of states, for which an exemption is not in place.
I can't find a source (including 15 U.S.C. § 260a) that supports this reading, although I agree it's a little ambiguous. The law suggests that a region that doesn't observe DST is observing "the standard time otherwise applicable during that period" and is exempt from the provisions regarding advancement, not that "Pacific standard time" depends on where you are (see 15 U.S.C. § 263).
> So, the Unix-y convention [] is the simplest way
DST is just a hack to make the time roughly correspond to when the sun rises, instead of when noon is. It solves the problem that people want their day to start when the sun is up, not before, and not after. The problems it creates are mostly for programmers who can't be bothered to write good software. Well I say tough cookies, you'll have to handle the edge cases so the rest of us have comfortable mornings.
> The problems it creates are mostly for programmers who can't be bothered to write good software
Myopic and ill informed.
> A 2017 study in the American Economic Journal: Applied Economics estimated that "the transition into DST caused over 30 deaths at a social cost of $275 million annually", primarily by increasing sleep deprivation.[128]
> An LSE study found that DST transition increases people's feeling of being rushed for time, and the number of hours spent on leisure decreases by roughly 10 minutes following the transition and more specifically the spring transition into DST decreases life satisfaction by around 1.44 per cent.
I think this summarizes it. The negative effects are concentrated around the transition. The positive effects are spread throughout the year. You lose 1.44% life satisfaction for a few days, and gain immeasurably more over the course of the year.
I don’t give a crap about the day starting when the sun comes up. I just want to go home at 5PM in winter and it not be dark already. There are plenty of studies about how this increases depression in the winter. The fact we keep persisting with this perverse nonsense infuriates me.
I’d much rather go to work in the dark and have sunlight after work. The only people that it’s better for are children who would have to wait for the school bus outside in the dark, though in my experience growing up school started early enough that we were still waiting in the dark anyway.
DST wasn't a mistake and did make a lot of time (at least if you are far enough away from the equator). Heck if you go back far enough local time might have varied by session/be relative to sun rise, and that did make sense in the past.
What is the mistake is keeping it.
Through changes need to be coordinated (e.g. in the EU across _all_ EU member states), which makes it an annoying topic to takle.
That people tent to say "keep winter time" but it directly conflict with what sleep scientist tend to recommend is not helping either. In the end normal time is summer time and it is better for most peoples sleep rithm that is what we should go to.
Oh also don't shift work start/end times, the whole point is to not shift when you need to wake up/go to bed but can have consistent healthy sleep.
I think you have this backward. Standard time is during winter. Daylight savings is during summer.
That being said, I used to hate the switch, like you. Now, I'm convinced it's the best of only bad options. Just look at the history of DST. We've tried it all. We've tried year-round standard time. We've tried year-round DST. We've tried switching at all sorts of different dates. Every system has it's problems and it's advantages. The current system is basically the one that actually pissed people off the least.
oh, yes, I got that mixed up. That also mean permanent winter time would be better for the sleep not permanent summer times (it was whatever the "standard" time was as far as I remember). Either way people cant decide on what time they prefer makes any change harder.
> I'm convinced it's the best of only bad options.
idk. the time switch every year comes with an increase in traffic accidents for up to a weak after it happens. While most jobs today either aren't overly dependent on sunlight anymore or don't sync up with e.g. start of school or similar anyway (e.g. field work). I guess that argument does differ for any area widely dominated by farming and limited farming automation.
PS: on the other hand the affect on sleep might not matter (as in there are many far large effects). I live north enough for the number of sunlight hours and sunrise/set in summer and winter to massively differ, with the sky never getting fully dark during the few longest days.
Everyone carries a GPS in their pocket; why not use it for good? We can have people's clocks continually update so that 12:00 is solar noon at their current location each day, and avoid the jarring transitions into and out of DST.
Don't ask what people living in the high altitudes will do - I'm still working on that.
Here in Bristol, UK, we've got an old Corn Exchange that has a clock with two minute hands to show London time (GMT) and Bristol time which was about 10 minutes different.
Continuously updating clocks is an interesting idea, but it isn't solar noon that you want to to anchor it to. That doesn't really accomplish anything biologically useful.
It is sunrise you want to anchor it to, because it is sunrise that our circadian rhythms sync to.
Forget different towns being 5 minutes ahead, from the other linked qntm post [0]:
> At the equator, the position directly underneath the mean Sun travels west at about 463 metres per second. That means a standard rack unit is about one millisecond wide. At latitudes closer to the poles, the effect is amplified, although but not by more than an order of magnitude in the realistically habitable parts of the world.
Even a row of servers would all have different times.
that is quite a terrible idea IMHO (at least if done internationally)
sure they make international meetings/events harder, but for most people most of their live is bound to local time meanings even if you travel to another country. 7am is in the morning 7pm in the evening 11:59am is mid day etc. If you remove time zone then for some people 7am is morning for other it's mid day and for other noon. So creating a lot of issues for everyone just to make international remote meetings and events easier seems like a bad idea, not even considering the absurde level of practical issues a switch like that would cause for any country not around UTC+0. Even more so remote meetings and events normally involve software and and time zone confusions are very solvable there (display time dynamically in the time zone device currently displaying it (but with time zone suffix, and some considerations about storage wrt. changing time zones, also maybe add an option to display in home instead of current time zone)). Annoying is how few programs do that properly.
Except even locally time is different for different people. People work nights, mornings, afternoons. Some shops open at 6am, others at 9am. Some people get up at 5, others and noon.
You still learn that x time is your time to do y, and people will quickly adapt to whatever number x happens to have.
We only associate 9am with anything because we learned to. We could just as easily learn a different number, and people with different work schedules do just that.
Noon should be a 12 o'clock. So whichever of the two times is closer to 12 o'clock during noon in the middle of the time zone should be the one time. I don't really care if it's DST or standard time, just pick one and if necessary adjust working hours so we get that one extra hour of sunlight during winter.
Sunrise varies greatly. In my country, it’s anywhere as early as 5am and as late as 10am (9am with DST)
That’s quite a range to sync to. At least syncing to noon, you don’t get much shifting throughout the year since sunset and sunrise shift roughly equally.
The thing I don't like with these kinds of articles is rather than list potential pros/cons they make a wholly one sided story everyone is supposed to agree with the whole way and say "oh wow, yeah" at the end. Reality is it breaks right at the start - you don't really know when a good time to call someone is by the sun. You know because of when they wake up, when they go to bed, what hours they work, what you're calling them about, when they like to eat meals, etc. All of that varies by many hours within a timezone based on culture or individual, so it derails that build up pitch. It's a given the author isn't particularly swayed by that, but that they don't even talk about the detail and just move on spoils the rest of the (well put together) list IMO.
One way or the other I don't think we'll make a big shift in timekeeping until/if we ever have a sizable population off Earth. Of course, that introduces its own time problems we don't have to deal with as much all being so close together :).
I don’t personally see a lot of difference in consulting a chart of “what time is it in x country” vs a chart of say “the time business starts in country x”.
They’d be the same exact list. “offset +9 hours”- the only semantic difference would be that clocks don’t change.
I should mention that I spent a little bit of time in Saudi Arabia and expecting them to be out and about at 7pm like in Western Europe and the USA is crazy, they seem to get up later and keep going until 3am. I’ve never seen rush hour at 3am until I spent time in Riyadh. It’s a false construct we’ve decided on: that everyone follows the same time pattern anyway.
Why do we believe the world needs to wake up at 7am? If nothing else its so incredibly arbitrary to begin with.
I never found those arguments for keeping time zones particularly compelling. Everyone has their own schedule, trying to standardise time to sync people up is silly, you sync by talking to them and asking them what times they are available. The number on the clock doesn’t actually matter.
But DST affects us exactly because people tried to standardise on time AND then shift that meaning twice a year. If we ditched timezones, then businesses could say ok we will work from 1700 until until 0100 or whatever, based on time relative to sunrise, it it would be consistent all year round.
The thing is, regardless of timezones, you have to ask people what times they work or are available or whatever. Also consider that timezones are arbitrary and made up anyway: china physically spans the space of 3 timezones yet the entire country uses one timezone.
One thing I hear people say in places DST was abolished is that the late sunrises in winter are similarly depressing, and that this is something not really appreciated by those who want to abolish DST by having it be summer time year round
Same, I hate this last weekend of October when sunset suddenly shifts from 17.00 to 16.00, it makes the winter darkness so much worse... It'll be dark before I start and end work anyway (on solstice it's 8.30-14.30), at least let me have sunlight a little later in the day so I won't feel as miserable.
In Superman III, the eponymous man straightens the leaning tower of Pisa, and in Superman II he spins the earth backwards to reverse time.
A better idea would be to un-tilt the earth’s axis, thus getting rid of variable day lengths, annual seasons, the need for DST etc, just one enjoyable spring day, every where, all year round.
So much of the discussion that happens around DST always end up coming across to me like they're based on people's subjective experiences that they seem to assume apply universally (or close enough to it to structure society around, with the assumption that only a small fraction of people don't experience things similarly). I don't think things are nearly so simple, and while having issues strong enough to be diagnosed with a condition like the ones I mentioned above is probably not super common, it seems pretty plausible to me that there's a lot more of a spectrum than just "people whose circadian rhythm perfectly matches the sun" and "outliers whose circadian rhythms are designed misaligned with the first group". It just doesn't seem obvious to me that the issue couldn't be that some people have different enough experiences in terms of their circadian rhythms that their optimal clock scheme might be an hour earlier or later than someone else, or that one person might be greatly affected by the total length of the sunlight in the day and another might be completely indifferent to it.
There are a number of disorders[1][2][3][4] that relate to circadian rhythm, and I don't think it's that much of a stretch to imagine that even outside the bounds of a diagnosable condition people's experiences might be a spectrum rather than a binary of "majority of people who would all have the optimal experience with the exact same scheme for how clock time should work" and "small fraction of outliers who wouldn't agree with the obvious best solution that everyone else would". It's absolutely wild to me how many people feel so strongly about the "correct" way to handle this that when examined basically boil down to "well, this is what would make me happy". I don't doubt that people probably are correct about what would work better for them, but I also don't think it's that crazy to think that it might not be that uncommon for two arbitrary people to have optimal clock schemes that differ by an hour or more, or that whether or not their optimal clock scheme might vary direct by the time of year, especially given that the length of sunlight in a day isn't at all uniform at different latitudes.
I don't pretend to know for sure whether DST is right or not, or what the correct hours to pick in the absence of it would be, but I can't help but be skeptical of an argument that it "needs to die and solves no problems" and that a certain time is the "correct" one because "I want <something>". It's totally valid to want and even argue for something, but that's not at all the same as anything else being automatically incorrect when your way of getting it is literally to enforce how time works for millions of people. Phrasing it like that just doesn't make it sound like your arguments are going to be very convincing.
Um, no? Using standard time in summer (assuming northern hemisphere) means that the time will be earlier when the sun goes up and under. So it will be darker at 9 PM and easier to put your children to bed. They may get up earlier though.
The early rising part of the population would love to pull that off, just like it actually happened for centuries now. The problem is that late rising part of the population has never disappeared and does have a voice. Late risers can't do anything with people collectively deciding to open work time at 7 or whatever. But they can and will voice their opinion when a public discussion about time zone will inevitably arise. Surprise, surprise, late risers exist, and they are also human.
PS: by late risers I of course mean people with shifted circadian clock, and not "lazy" people, like they are sometimes presented.
Also anyone living at higher latitudes won't be thrilled by having the sun up even earlier. I don't live that high up compared to quite a lot of people in Europe, and during summer the sun already starts shining from below the horizon at 3 in the morning
In theory this is nice assuming you push the opening hours back but the cost of it would be significantly more than the developer time wasted by a mile.
Binding times to timezones to geographic locations (which have different sunrise and sunset times) to opening hours to social stigmas (you wake up after 11am? Tisk tisk) is downright silly, but humans don’t like change so fixing it will never happen.
I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year.
I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
Whenever you are unsure whether to use a clever solution or follow the globally accepted standard in your work as a DevOps or Software engineer, always choose the standard.
Among other things, in an incident, people’s brains aren’t working - the more they have to remember about your particular system, the more likely they are to forget something.
it was already a 40 year old standard at the time you're talking about.
awareness of UTC being the correct choice has definitely increased over time, but UTC being the correct choice has not changed.
you say reddit servers use UTC now, which implies there was a cutover at some point. were you still at reddit when that happened? were you still hands-on with server maintenance? any anecdotes or war stories from that switchover you want to share?
because I can easily imagine parts of the system taking a subtle dependency on Arizona being Reddit Standard Time, and the transition to UTC causing headaches when that assumption was broken. your memory of this "clever" trick might be different if you had to clean up the eventual mess as well.
Hold on, I'm not a sysadmin guy. Are you folks saying the server should not know what part of the world its in, that basically it should think it's in Greenwitch?
I would have thought you configure the server to know where it is have it clock set correctly for the local time zone, and the software running on the server should operate on UTC.
From a logging perspective, there is a time when an event happens. The timestamp for that should be absolute. Then there's the interaction with the viewer of the event, the person looking at the log, and where he is. If the timestamp is absolute, the event can be translated to the viewer at his local time. If the event happens in a a different TZ, for example a sysadmin sitting in PST looking at a box at EST, it's easier to translate the sysadmin TZ env, and any other sysadmin's TZ anywhere in the world, than to fiddle with the timestamp of the original event. It's a minor irritation if you run your server in UTC, and you had to add or subtract the offset, eg. if you want your cron to run at 6PM EDT, you have to write the cron for 0 22 * * *. You also had to do this mental arithmetic when you look at your local system logs, activities at 22:00:00 seem suspicious, but are they really? Avoid the headaches and set all your systems to UTC, and throw the logs into a tool that does the time translation for you.
The server does not "know" anything about the time, that is, it's really about the sysadmin knowing what happened and when.
1) Most software gets its timestamps from the system clock
2) If you have a mismatch between the system time and the application time, then you just have log timestamps that don't match up; it's a nightmare - even more so around DST/ST transitions
you've got it backwards - the server clock should be in UTC, and if an individual piece of software needs to know the location, that should be provided to it separately.
for example, I've got a server in my garage that runs Home Assistant. the overall server timezone is set to UTC, but I've configured Home Assistant with my "real" timezone so that I can define automation rules based on my local time.
Home Assistant also knows my GPS coordinates so that it can fetch weather, fire automation rules based on sunrise/sunset, etc. that wouldn't be possible with only the timezone.
Windows assumes computer clocks are local time. It can be configured to assume UTC. Other operating systems assume computer clocks are UTC. Many log tools are not time zone aware.
that's the difference between "aware" and "naive" timestamps. Python has a section explaining it in their docs (though the concept applies to any language):
Would he by any chance refer to it as Zulu or Zebra time? The Z-suffix shorthand for UTC/GMT standardisation has nautical roots IIRC and the nomenclature was adopted in civil aviation also. I sometimes say Zulu time and my own dad, whose naval aspirations were crushed by poor eyesight, is amongst the few that don’t double-take.
While I agree on this particular instance, there are two types of things future engineers have to clean up after: Their predecessors thinking too small (and picking the easy route) or too big (and adding needless complexity).
One is not necessarily and in all instances less of a mess to clean up behind than the other.
I can’t quantify how much time my team wasted in diagnosing production glitches on checking the wrong time offsets but it was substantial. One of our systems wasn’t using UTC, and given enough time the fact that Slack wasn’t using it either does become an issue. When an outage transitions to All Hands on Deck everyone needs to get caught up to what’s going on preferably under their own power so you don’t suffer the Adding Resources to a Late Project problem.
So that first alert that came in ?? minutes ago you need to align with the telemetry and logs in order to see what the servers were doing right before everything went to shit.
What if it's your personal machine? I'm thinking about jobs I've set up... thing is, I actually do want those to align to DST in most cases. For example, ZFS scrub should start after I leave for work so that it has the greatest chance of being done by the time I get home. (It's too loud to run overnight.)
This shouldn’t be hard to deal with if the timestamp is always serialized with the offset: I’m much more picky about always persisting the offset than about always persisting UTC
That's still a bit risky as Arizona might just change its time zone definition on a whim. I'm an engineer on one of the big calendaring applications, and it's mind-boggling how often stuff like this happens world-wide, sometimes short-notice (a few weeks in advance). It regularly gives us headaches.
Agreed, watching the rate of changes to timezone databases will rapidly disabuse you of the notion of any constant. It's rare that a day goes by without an update to some definition somewhere, which is astounding.
Closely related pet peeve: Log display web UIs that allow selecting display timezones including UTC absolutely insist on deriving my preferred time format (12/24 hours) from my browser language preference.
If nothing else, selecting UTC should be a very strong hint to any UI that I am capable of parsing YYYY-MM-DD and 24 hour times.
If you use <input type=time>, the browser uses locale or user regional preferences… even if the app is in an application domain (e.g. medicine, military, science) where 24h is preferred even in 12h-predominant locales. There is no way for the app to say to the browser “this time field should be 24h in all locales”, the only option is to build a custom time field
Some browser APIs respect `LC_TIME`. Others say "fuck standards, I'm using `LC_MESSAGES`".
This means that if those locales differ about say, y/m/d ordering, it is quite possible for the way the browser formats a date to be different than the way it parses the date.
Operating systems generally allow user override of locale settings, and browsers generally respect that; I use a locale which officially has 12 hour time as its standard (Australian English), and I always override it to 24 hour time in user preferences (although Australia is rather inconsistent, e.g. in Sydney, train services use 24 hour time; in Melbourne, the metro trains use 12 hour time but the regional services use 24 hour time)
Interesting, so you're saying that that OS-level time preference is available via JavaScript? I wasn't able to figure out how to query that in a little bit of trying, so I assumed there was no API for it.
If there actually is, I'm now even more upset at that log web UI.
For me, the first returns 24 hour time and the second returns 12 hour time. Because 12 hour time is default for my primary locale (en-AU), but I override it to 24 hour time in my macOS settings.
I know the same works on Windows, and I’m sure it works on Linux too, just the way you configure it is different for each OS.
I would especially like to call out the Scandic Hotels chain for this behaviour as well. Booking a hotel room should not involve me wondering if I booked it for the wrong day when I'm not in an European time zone.
That's interesting because I strongly prefer that timestamps are stored as UTC and converted to my timezone on the fly for easier debugging. I dunno about using the browser language choice (do sites really do this)? Usually a simple conversion with javascript is fine (javascript knows the local timezone).
That is the correct and only sane way to determine date format, timezone is not the same as formatting preference. But yeah it sucks. Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.
In firefox you can try it out in about:preferences by setting en-CA as the topmost option in "Choose your preferred language for displaying pages"
And I just checked, firefox is capable of understanding system-wide split locale settings, if you only want LC_TIME
> That is the correct and only sane way to determine date format, timezone is not the same as formatting preference.
That's unfortunate, but then wouldn't the sane way for localization-aware software be to ask the user and not make any such assumptions?
> Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.
Thanks, I'll give that a try tomorrow! If my log exploring UI of, well, not quite choice actually respects that, I'll be very grateful :)
In a way it is asking the user. Date formatting rules are traditionally tied to locale, so the user picks their locale and their expected date formatting rules are derived from that.
On Linux you can mix and match via LC_*= environmental variables, but that appears to be complexity the browser vendors didn't expose in their UI.
Java allows setting the default timezone at jvm level and everyone in our org were setting their favourite TZ which was mostly PST somewhere in the code.
We had application logs and system level logs with different timestamp and someone decided a certain db column has to be a string of the format 'yyyy-mm-dd hh:mm:ss". You can imagine the confusion and the "fun" time we had while debugging logs from multiple systems at specific times.
Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
> Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
That seems to me like a really bad mandate. If you are going to mandate something, mandate UTC. If someone forced me to change a system from UTC to something else, I’d not be very happy. It’s the kind of decision which makes you seriously question if you want to work there any more.
Does that mean you ran PST during daylight savings time and half the year your logs were “off” by an hour? Or did you actually run Pacific Time and deal with the clock changes twice a year?
> I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
Also important here is that the date stays the same! Even though I've gotten used to instinctively decoding UTC, that part is still frustrating sometimes.
There are many solutions, but when you're actually running a site with millions of daily visitors and one person focused on ops, you do the easy thing, not the right thing, unless those happen to coincide.
If this is a complexity battle, I'd rather that sysadmins have to use a tool to read logs vs. figuring out why the server is melting down because all the jobs are running 2x or answering questions about why jobs didn't run at all.
In my old gig, jobs ran for many many HOURS and consumed most of the IO on the server (processing reams of data), which meant that when you got them running 2x because of overlap, it was a trainwreck.
In my experience, it's really not pleasant having to work with logs that require a dedicated viewer compared to regular old text files I can use Unix commands available everywhere (`tail`, `head`, `wc` etc), with, i.e. not just on the server producing the logs, but also the device I'm viewing them on.
That said, I absolutely prefer UTC times in logfiles. I'd rather do some math every time than make wrong assumptions about the local timezone of the file even once. (Even correctly indicated local times are more annoying, since I have the math for the offset of my timezone to UTC memorized, but not that of every possible server time to mine.)
Yup. There's always some kind of tool anyway so why not. I mean, even if you read the logs as they come out of a screaming line printer, the tool is the thing that takes log messages and spits them out the printer port.
How hard can it be to write a log cat utility in awk?
Nooo... we have a bunch of metrics and logs reporting systems at my company. Some of them are in UTC, some of them display my local time, some of them display China Time, and I'm trying to collaborate with colleagues in London and Australia who get data displayed in their local times as well. When I'm working to address an incident and combing through multiple systems to try to correlate events, it's a pain in the ass having to constantly double-check which time zone this data is in.
In my experience, the hard part is getting everybody else to do that. And then also getting them to actually include the timezone in their communication with you.
I'd rather the logs be consistent, and not rely on every single person who ever looks at the system logs make sure to use the correct tool the correct way.
The problem is a lot of times when you are looking at logs you are already very far off the happy path of things you look at often and want to invest resources into displaying well.
This is not smart, this is just surprising to the next person who is going to maintain your ”smart” tricks. Thank god they switched to utc, that is what everyone expect.
Actually, back then, 18 years ago, most people expected your servers to be in Pacific or Eastern time, depends on where your company was headquartered, because none of us really had global technical workforces back then. We all pretty much worked in one office and used the local time zone, because often our servers were in the building with us or in a datacenter nearby.
Case in point, before reddit I was at eBay, and we kept all those servers in Pacific time, since the entire technical workforce was in Pacific time, as well as all of the servers.
Making blanket statements like that without considering the context of the time is usually not a good idea. ;)
> most people expected your servers to be in Pacific or Eastern time
I was there back then, working for shops people have heard of, and I honestly don't know where you're getting this idea from. Some places did things wild and wacky when they were wee small, but most of us quickly learned that such shenanigans (like fun server naming conventions) start to fall apart and maybe we should do things differently.
Ah you think UTC is your ally? You merely adopted UTC. I was born in it, molded by it. I didn't see DST until I was already a man, by then it was nothing to me but blinding!
In the 1980s, PT and ET were common. I was working at Bell Labs then, and one of my jobs was to change the time zone (back then it was two words) on the testing machines, as needed. This is stuck in my memory since to change the timezone, you needed to edit the Unix kernel source code and recompile it!
2000 for me. That was the first time I had users from outside my own time zone, so I figured it was better to just use UTC for everything and just convert depending on the user's settings. I think I just applied the thinking to the whole server.
It was probably the smartest option at the time, given the context. As long as it was documented properly, no one on such a small team would have been surprised. Spend more than the 30 minutes you've spent here on HN so far, and you may learn a thing or two about what is "smart", and how that is inextricably linked to the given situation.
If you're like 5 people, each with a list of TODOs that doesn't fit on one screen, it's pretty smart to just do something quick and good enough, then move on, revisit it in the future.
I worked at a place that had set the servers to BST not UTC and used cron jobs extensively for reporting, which caused chaos twice a year and twice a year they had to explain to the business again why it caused chaos and ask for the time to fix it.
Company went bust before it got fixed.
Just use UTC folks unless you have a really good idea why you shouldn't.
$ zdump -Vc 2025,2026 America/New_York
America/New_York Sun Mar 9 06:59:59 2025 UT = Sun Mar 9 01:59:59 2025 EST isdst=0 gmtoff=-18000
America/New_York Sun Mar 9 07:00:00 2025 UT = Sun Mar 9 03:00:00 2025 EDT isdst=1 gmtoff=-14400
America/New_York Sun Nov 2 05:59:59 2025 UT = Sun Nov 2 01:59:59 2025 EDT isdst=1 gmtoff=-14400
America/New_York Sun Nov 2 06:00:00 2025 UT = Sun Nov 2 01:00:00 2025 EST isdst=0 gmtoff=-18000
$ zdump -Vc 2025,2026 Europe/London
Europe/London Sun Mar 30 00:59:59 2025 UT = Sun Mar 30 00:59:59 2025 GMT isdst=0 gmtoff=0
Europe/London Sun Mar 30 01:00:00 2025 UT = Sun Mar 30 02:00:00 2025 BST isdst=1 gmtoff=3600
Europe/London Sun Oct 26 00:59:59 2025 UT = Sun Oct 26 01:59:59 2025 BST isdst=1 gmtoff=3600
Europe/London Sun Oct 26 01:00:00 2025 UT = Sun Oct 26 01:00:00 2025 GMT isdst=0 gmtoff=0
"Our customers aren't in UTC" is often a compelling reason.
People generally expect things like usage limits to reset overnight, not at UTC midnight, and these are often implicitly tied to (the result of) batch jobs.
Financial jobs, including billing, is another big category.
Having tried many times to use local time instead of UTC to make life simpler for myself, I really don't think it's ever a good idea.
Better to localize times for users on the client-side based on their local settings, IMHO
For quotas and usage limit resets it can be a little harder, but if it's a cron job anyway, my initial approach would be to just schedule it for a time that is close to midnight for the area your customers are in.
That does not require setting the server to local time. Run the cron job every hour. Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. This also happens to be more resilient to server downtime around midnight.
> Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit.
That sounds a bit like reimplementing a scheduler inside the scheduled task (which is ok if you don't trust your scheduler, but I'd avoid it if at all possible).
> Hopefully just a datetime recent enough that the job does not currently need to be (re-)run.
I can think of several ways to mess that seemingly trivial calculation up just from the top of my head. (Consider changing timezones, increasing the interval the job is running to within half a day of scheduling period etc.)
> "Our customers aren't in UTC" is often a compelling reason.
This doesn’t work as soon as you become a global business… because if you tell a customer in Asia that it resets at midnight in some US timezone, what are they going to think?
Many people will accept UTC because it is the international standard. Everyone will accept “let each customer pick the timezone that works best for them”. Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
Really depends on your market and customers. Especially in the financial industry, local time zones (often of a given currency's central bank's main branch, a primary regulator etc.) are often very, very entrenched.
> Many people will accept UTC because it is the international standard.
I really wish that were true universally...
And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more.
> Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" one for resetting quotas etc. has some merit as well, if only because it makes the concepts of "today" and "tomorrow" work a bit better. (UTC midnight might be very unintuitive to both you and your customers.)
> Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical"
What happens when you move your head office to a different time zone? e.g. Oracle moved their HQ from California to Texas (and have announced a further move to Tennessee, although last I heard that still hasn’t happened)
What happens when an M&A happens and you are now part of a much larger enterprise with a different HQ timezone, and systems with wildly inconsistent configured timezones due to having acquired multiple companies which had HQs in different time zones and all of which configured everything to use their HQ time?
> UTC midnight might be very unintuitive to both you and your customers
I live in eastern Australia, UTC+11 at the moment (UTC+10 during Southern Hemisphere winter). Right now, resetting stuff at UTC midnight means at 11am my time, resetting stuff at US Eastern midnight means at 3pm my time - roughly equal in potential inconvenience, but I’ll be much more forgiving of the first than the second.
> And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more.
For me, add/subtract 10 is pretty easy… and then just one more during DST. Does that make UTC “Australia-centric”? Or Guam-centric or Vladivostok-centric? (both of which are also UTC+10)
Do you actually care if the requests were made on some specific calendar day? Or do you just want to make sure that heavy users are paying more and/or people aren't abusing your system?
Instead of tying your quotas to calendar days in some specific time zone, tie them to a rolling 24 hour usage period. Even better: use a rolling hour.
The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time. This is exactly what I said above.
> Generate before and send to everyone based on timezone.
How are you going to do this? Clearly cron (with timezone set to UTC) doesn't work for this so you'll need another system. In that case why not have that multi-time zone scheduling system run your script?
I absolutely want some daily business reports to cover 25 hours on one day of the year and 23 on another, since that's how long a calendar day is in local time in every place that has DST shifts.
> That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes".
How hard is to convert the UTC time to user's local time and perform quota reset?
I mean you should be doing this anyway. Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
>Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
1. allowing users to change their timezone and tying it to when their quota reset sounds like a great way to add more edge cases and complexity. For instance, does messing with your time zone mean you can get your quota reset multiple times a day?
2. Not everyone operates some sort of global b2c SaaS that's timezone agnostic. For many enterprise backoffice tasks "6am HQ time" is far more reasonable than "6am/7am, dunno depends on whether daylight saving time is on or not". In this case having a server set to the HQ's timezone makes far more sense than doing UTC and trying to work around daylight saving issues.
> How hard is to convert the UTC time to user's local time and perform quota reset?
Probably roughly as hard as writing a timezone-aware scheduler that considers all edge cases around daylight savings time, i.e. probably possible but I'd try fairly hard to see if I can get around it.
One way of getting around it is to run your batch jobs in your "primary business timezone". Sure, you might outgrow that concept eventually, but many companies never do, and in that sense "running on UTC" seems similarly aspirational in spirit (albeit at a much smaller scale) to starting with high scalability.
You could have the reset script check the timezone of the user before actually performing the reset, and have this script run at least once an hour to make sure you get close to accurate for most people (some timezones only differ by less than an hour though!).
But also, if the user can change the timezone themselves, now you have a condition where the user could keep their quota at bay for a very long time (if not forever) if they keep changing their timezone accordingly.
Also don't use cron full stop. Build (or use an off the shelf) scheduler in your app that's more sophisticated, has access to data and client preferences, etc.
If I want a report of what happened at a specific time I need that report at local time
I get a daily status report of various things from our 24 hour operational management team which runs at 8am UK time every day. That means last week it ran at 0700UTC and this week at 0800UTC
This is built around operational events, shift changes, etc.
I've got another system which is in operation in Sydney from 0630 to 1630 local time, this means that maintence windows which overlap with UK shift patterns depend on the week but mean the system is operating 2130-0730 UK time at some times, 2030-0630 UK at others, and 1930-0530 at others.
UTC is not "the answer". Sometimes you want things running at a UTC time, sometimes you want them running at local time.
I have a regular meeting at 10am London time on Tuesday and Thursday. That can't be stored in UTC as it varies depending on the time of the year. It has to have the timezone stored and actioned.
It certainly is, but that's a business question, not a technical question
If the business requires it to run at 01:30 and there are two per day, or zero per day, then the business rules needs to define what happens. You can't solve this by running it at UTC.
Technically correct, but the business needs to be aware and a solution needs to be proposed beforehand IME. UTC is such a solution, imperfect, but usually workable. Business rules are sometimes unexpectedly elastic when confronted with a particularly explosive mix of logic and legislation.
How about I use some form of library to do it. I tell it I want to run at 0800 London time, and it runs at 0800 London time
If I tell it I want to run at 0130 London time (or 0330 Athens time) I still have a problem -- do I run it twice when the clocks go back, do I skip it when clocks go forward?
But that's a business logic problem, and defining it as UTC and having another job to update the time twice a year doesn't actually solve the question of "what do I do at this point".
You don't have any choice in the matter. If you control the time you want the jobs to run externally from the server, then you can have control over it. Otherwise, you're getting jobs that run 2x or not at all.
No matter where you run the job from, you run it in the required timezone, not UTC, and you still have the issue with what to do with the change in the local timezone.
This isn't a problem that can be solved with a single technical solution.
Generally, yes but I there's a surprising amount of cases when it is important, which makes it difficult to generalise
e.g. Huge amounts of the financial sector cares because of market times or regulatory reasons.
I use UTC for all public/production servers, but for my homelab servers in my closet rack they all use local time. Makes grokking times in homelab servers much easier. The exception is database insert/update date/times which are always stored UTC.
At Marin Software, they ran each customer on their own set of servers and set the servers to the customers defined TZ. It was an endless cronjob nightmare. Now they are also bankrupt.
Don't forget changing the day 1/3 of the time. But more importantly, if I'm scrambling to solve an incident, the last thing I want to do is deal with time conversions in my head.
You just do it once and think in UTC going forward. But as I said to the other person, if you're only dealing with one timezone it's fine, just when you have multiple it's a lot easier to just deal with one time and let everyone convert.
I'd expect everyone who works in computer related jobs to know their UTC offset.
I'm able to subtract 8, but if I'm scanning logs it's one more thing to process.
If it's local time I know instantly when something happened, without having to do mental math.
Is there anything wrong with ISO8601 (including timezone offset) for storing times? They're in my local time, but they can be accurately converted to any other timezone.
No problem as long as it's all local, but it's a big annoyance to the other teams if I'm trying to coordinate with the West US team who're on PT, the East coast on ET, Europe on CET, India on IST, Australia on BBQ...
It's just easier for everyone if we agree on UTC and everyone knows their own offset.
Yeah. I log in json + unix timestamp nanoseconds, and just convert it to something human-readable on display (github.com/jrockway/json-logs). I am not sure why logs need to be "pretty" at time of logging when they can be made pretty at time of display. Doing it at time of display means you can just use local time, and then nobody is confused.
Wait, are there timezones where the DST change doesn't happen at midnight? That's news to me.
Now this article makes sense to me; I was wondering what made 02:00 and 03:00 special, since the DST changes would be from 00:00 to 01:00 and from 00:00 to 23:00, as I'm used to since childhood here in Brazil. Perhaps some other countries change DST from 02:00 to 03:00 and vice versa?
In the UK we move forward at 1am and they go backward at 2am. Doing it at midnight adds the extra complexity that now the day is different. Doing it in the early morning doesn't change the day.
My guess is that in the US they do the same but shifted by one.
Most timezones change time when there's a minimal amount of people working, as these people would have to work an extra hour, and doing it at around 3am is the most reasonable from that point of view.
Better to think of it that it changes at 01:00 UTC, which takes care of the parts of Europe that are 2 or 3 hours ahead (instead of CET's 1 or 2), and the UK going "forward at 1 and back at 2"
"Europe DST changes at 01:00 UTC" - much simpler ;)
Brazil not only had DST at midnight, but until 2008 they also had no standard rule for when DST would begin and end, setting the dates by decree often just a few weeks in advance.
Redundantly running cron jobs are the least of our problems. Daylight savings adjustment day causes surge in deaths and ER visits[1]. Daylight savings needs to go.
This was worked around ages ago in OpenBSD, and the workaround was already included in Debian (and by extension, Ubuntu) when I started maintaining it in 2010 (I no longer do).
Here's a link to a patch [1] from a version from when the package still kept standalone patches.
It's been a long long time and I honestly don't remember the details, but Debian's cron(8) still says [2]: Special considerations exist when the clock is changed by less than 3 hours, for example at the beginning and end of daylight savings time. If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run.
Edit: According to this bug report [3], this workaround first entered Debian in 1999.
Most likely, yes. The author mentions vixie-cron, which was the name of the project before Paul Pixie joined/founded(?) ISC, and it was released as ISC from after.
Debian's fork is still based on vixie-cron, but it couldn't have been the one because of the aforementioned patch.
In general I try to use odd times for cronjobs. Backups starting at XX:13, analysis scripts at XX:23, data exports at XX:42 and so on. This simplifies the question of "Why does my system go whack at 23:23? Oh, wait".
This is a great tip. One of our QA engineers noticed recently that a particularly difficult-to-reproduce bug was affecting records saved at :21, which is when a particular cronjob hits. Without that extra info we’d probably still be scratching our heads over it.
The details I never really thought about until I encountered timezone issues was:
There are points in time that don't exist, and there are points in time that are ambiguous.
If we adjust the time forward by 1 hour, then then 2:30am doesn't exist. If we adjust time backward, then 2:30am is ambiguous.
2am-3am is only safe for certain countries.
America/Santiago (Chile), for example, adjusts their daylight saving on midnight. On a certain day, midnight doesn't exist because it immediately jumps to 1am. If you are building an analytics chart by day, you will encounter this issue. I wrote about several weird time zone edgecases here: https://tanin.nanakorn.com/edgecases-for-timezones/
To add more confusion, IIRC Java and Ruby handle these cases differently where Ruby raises exceptions (e.g. InvalidTime and AmbiguousTime) but Java doesn't.
I caused 3 incidents in a row at Stripe in my first month because of this. I thought I was gonna be fired. Luckily I wasn't...
While not always possible, in general, any mission critical scheduled job should be idempotent. This avoids so much headache, especially when there's a queue involved somewhere.
I try to avoid crons at the top of the hour, partly because of this... but also because (in shared / serverless infrastructure) I assume a lot more people are setting their crons for 'on the hour' so there's more resource contention... I also aim for 'after 4am' where I can as well, or 'before midnight', to avoid this whole range.
I prefer to combine this with FixedRandomDelay=true. FixedRandomDelay ensures that the randomized delay is an arbitrary number up to RandomizedDelaySec, but it is deterministic per server and timer. I find this useful because this means the timer will always run at XX:12:45 on server01, always run on XX:06:23 on server02 and so on.
This combines very simple configuration, while being predictable and spreading out timers well.
I like cron jobs that run during the work day. If something goes wrong I would prefer to handle it at 10am rather than 10pm. Why make life unpleasant?
If it really needs to run during a low load period I line it up for 6am work time. Again so that if something goes wrong it leads into the work day rather than disrupts the on-call too badly. 6am also provides a daylight savings buffer either way so you don't have to care. Recent work has all been on 24/7 operations though.
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all.
Is this what people actually want, though?
I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input).
Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March.
The article is mostly talking about using the `cron` scheduler and running things at 2am on Sunday because there's especially low traffic. These cron jobs are "I don't care when it runs, but it should have minimal chance of causing problems."
Your use-case is totally different: "I want this job to run at this time" so the only lesson that applies to you is that the `cron` utility might behave weird during DST switches. No idea if it underlies your cloud provider, so it may be completely irrelevant.
> Everybody wants the job to run four hours before the work day starts
recurring events like this - ones that take place relative to humans in a given location - are the most common snag related to "just use UTC everywhere" advice. but they're not actually in conflict at all. they just require an extra layer of indirection.
let's say this is a factory that starts early in the morning, 06:30 local time, because then "four hours before" falls into the window affected by DST.
"run at 02:30 local time" is what you want to avoid, because on the "spring forward" day, 02:30 simply does not exist. the clocks jump from 01:59:59 to 03:00:00. [0]
likewise, "run at 01:30 local time" is to be avoided, because on the "fall back", that occurs twice.
ideally, you want to think of the recurrence as being associated with a given location, and that recurrence generates individual events, which happen in UTC time.
for example, if the factory is in San Francisco, the recurrence might specify "once a day, at 02:30 SF time", which would generate a series of daily events in UTC time. during the summer those events are at 09:30Z, during the winter they're at 10:30Z.
you also want the scheduler to be smart enough to understand the "once a day" constraint - so that a "once a day, at 01:30 local time" task generates only one daily event, rather than two, even though 01:30 local time happens twice in the same day.
and then, because the individual events are all in UTC, and the system clock is as well, there's no ambiguity in the actual execution time of the event. the complexity of the recurrence is handled at a higher level.
if you want an even harder version of this problem, imagine a team in SF and a team in London, that have a Zoom meeting every Monday at 8am (SF) / 4pm (London). it's necessary for one of those locations to "own" the recurrence, because the UK does their time changes 1-3 weeks offset from the US [1], creating weeks where it isn't a straightforward 8 hour time difference.
The more I read this thread, the more I think this is just a bug in Cron. It should really only operate on UTC, regardless of the local time set on the server.
with systemd timers [0], explicitly specifying the timezone is an option:
> systemd-analyze calendar '02:30'
Original form: 02:30
Normalized form: *-*-* 02:30:00
Next elapse: Tue 2025-10-28 02:30:00 PDT
(in UTC): Tue 2025-10-28 09:30:00 UTC
From now: 10h left
> systemd-analyze calendar '02:30 UTC'
Original form: 02:30 UTC
Normalized form: *-*-* 02:30:00 UTC
Next elapse: Mon 2025-10-27 19:30:00 PDT
(in UTC): Tue 2025-10-28 02:30:00 UTC
From now: 3h 34min left
however, making it disregard the server timezone and use UTC by default isn't really an option. it would violate the Principle of Least Surprise, as well as being a hugely backwards-incompatible breaking change.
I read this a while ago and was looking forward to the mess that would happen with these default Alpine configurations (after setting non-UTC localtime)
"BusyBox’s crond is a lightweight implementation designed for embedded systems, so it doesn’t have all the timezone/DST logic of full vixie-cron or cronie. It just reads /etc/TZ or the system timezone at startup and then wakes up every minute to check if something should run."
Seems to me the real problem here is not the timezone (there's legitimate business needs to run something daily at a specific localtime...) but having multiple instances of a cron job that overlap, in which case it should wait until the previous is done or not start at all. At least prefix a job with "flock -n" if it doesn't/can't handle that.
For the most part of Europe and USA the day is either "quite" long (summer) or quite short (winter) due to the curve of the duration. Having light for long summer evenings and dark for long winter mornings is bad for many reasons, one of witch is seasonal depression, with which the Scandinavians suffer and fight. I was for the abolishment of DST until COVID cancelled it in Europe but now I am thinking it again and probably I am against.
anacron is worth a mention here for sometimes-on computers like desktops, laptops, or personal servers. It's great for ensuring something runs at the set interval, or just whenever the machine comes back online. Minimum anacron interval is one day, so this can't simply replace cron. cron might even be the most sensible thing to use to kick off anacron on your system, and it might already be in use (see if your /etc/crontab fires off anacron).
anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.
2 am is mid afternoon for the Pune office. If you’re running a 24/7 operation and all of your employees are in PST you’re gonna have a bad time.
But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic.
The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration.
It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.
I run a cronjob at 2:30AM every day that uses dirvish (a perl wrapper around rsync) to perform backups of several computers. In the fall, at 2AM the clock is rolled back to 1AM. The cronjob runs at 2:30AM, which is 25 hours after the previous day's run. That's not a problem.
However, in the spring, at 2AM the clock jumps to 3AM. Vixie cron seems to be smart enough to run all the cron jobs that were scheduled between 2M and 3AM at that point. However, dirvish is started but exits without running the backups! After some troubleshooting I concluded that dirvish does not make the backups because the previous day's run happened less that 24 hours ago (because that day was shortened by the change to DST) so it sees no need to run a backup yet. This problem would persist even if I moved the cronjob to a different time because it has to do with the day having only 23 hours. I solved this by adding a second entry to the crontab that runs at 3:45AM only on the 2nd Sunday in March.
The topic was to prevent overlapping runs. In case of power failures etc all your server processes have this problem. That's why, for example, /var/run exists and files/directories are deleted from /var/run on reboot?
The problem is not so much which timezone you pick, the problem is that cron uses 'naive' datetimes. If the timezone is explicit then you don't have duplicate hours because one is 02:00 and one is 02:00DST.
And then you can translate that time to any other timezone with ease, knowing that you are referencing a specific hour in physical time that only ever happened once.
I'll give you leap seconds, but at least in Python historical timezone changes are recorded in the datetime library, so the timezone at any particular time in the past has the timezone offset that was active then.
Which is to say I wish cron had the TZ variable that Jenkins uses in it's scheduling system. And the H value to easily spread multiple jobs over the hour/day/month etc.
Not a crown job, but I configured my lights to turn on progressively at my scheduled wake up time, using Home Assistant.
I figured that scheduling lights to turn on at <0 AM + wake up time> was a good approach... We got a surprise early wake up last year, so I ended up changing the formula to <4 AM - 4 hours + wake-up time>.
I avoid cron job at this time for this reason and for many other random problem because 3:00 am seem to be often choose for a maintenance like automatic update, server reboot, database backup..
Can be tricky: an old project had a job that ran every 15 minutes to poll data from a popular smart thermostat company. I always knew when DST was starting or ending because that job would throw an error email at 2/3am on those days.
UTC also solves plotting daily time plots in zones with time changes (twice a year when the plots make no sense). But there still isn't no good fix for leap year plots!
You’re signing up for a world of pain in the real world, where every other system is UTC. Just use UT1 or smear the leap second if you really can’t stand the risk.
2-3am is bad due to DST but there is another reason to also avoid 3-5am or so
around 3am is the time where "in average" human attention is the worst as its where "in average" their most relevant deep sleep phases are
now I guess the average over HN readers has a good chance to be somewhat later
but either way if the job goes wrong and triggers alarms 2-5am it very bad for the the health and sleep rithm of whoever needs to fix it and the most likely time they make mistakes
it sadly is also the time where non-international interruptions are least likely to matter
if you are an international company having international teams can be an solution
and if you are a very small team adopting job times to an overlap of low usage and your admins sleep habits can be an option
but if you need to pick a generic time and don't provide for logistics or other "start work at 4/5am" companies probably 5-6am is a better time to run your batch jobs
If you want to avoid overlapping runs simply simply have the task check for the existence of a lockfile, if not create a lockfile containing the start timestamp, run the task and delete the lockfile at exit or error.
The timestamp gives you a chance to recognize stale/orphraned lock files in case of crashes based on their age.
I have a job that deliberately runs at 2am and 3am... to update the time in a bunch of really old PLCs for DST. And check that every other device on my telemetry network has correctly updated its time.
Here's my radical time idea: instead of keeping the same schedule year round, we just incrementally chop parts of the day off (we just put that time in a "bucket" and pretend it doesn't exist). Then at the summer equinox we empty the time bucket into a 6 day bender of a party. /s
This isn't a problem with actual cron and crontab. It is a problem with the systemd-timers shim "crontab" which doesn't work the same in many corner cases and often has weird bugs.
> until one of them achieves cron’s level of ubiquity, we have to live with cron at least some places and sometimes
systemd could arguably be described as close to (maybe behind, maybe ahead of since it's the default for the most popular Linux distros) cron's level of ubiquity, and doesn't have this bug as far as we know.
What do you consider "actual cron"? Because the post specifically says Vixie cron, which has been the basis for most versions of cron on Linux systems.
DST was a mistake and it needs to die. It solves no problems and only creates them. And not even the type of problem that someone could profit from, just plain old complete waste of time for everyone problems.
Just stick to winter time (because that's the correct one) and adjust all working hours once and for all to be like summer time, for e.g. instead of business opening at 7AM and closing at 10PM they can open at 6AM and close a 9PM. There will be a period of adjustment, but we have a period of adjustment twice a year already, so nothing is lost.
Why match working hours to summer time? Because I want to have more sunlight by the time I'm done with work. Especially during winter. We have it completely backwards, if we are going to adjust our clocks we should adjust them in a way that gives us more sunlight during winter, not less. I don't care if it's dawning by the time I'm going to work where I will be stuck indoors for nine hours, I want sunlight when I'm free again.
Everybody think DST is the worst and needs to be replaced in some way or another. Most people don't realize that their proposed "solution" has already been tried. Your particular solution of "year-round DST" was tried in America in 1973-1975. I'll just quote Wikipedia:
> During the 1973 oil embargo by the Organization of Arab Petroleum Exporting Countries (OAPEC), in an effort to conserve fuel, Congress enacted a trial period of year-round DST (P.L. 93-182), beginning January 6, 1974, and ending April 27, 1975.[17] The trial was hotly debated. Those in favor pointed to increased daylight hours in the summer evening: more time for recreation, reduced lighting and heating demands, reduced crime, and reduced automobile accidents. The opposition was concerned about children leaving for school in the dark and the construction industry was concerned about morning accidents.[18] After several morning traffic accidents involving schoolchildren in Florida, including eight children who were killed, Governor Reubin Askew asked for the year-round law to be repealed.[19]
> Over three months from December to March, public support dropped from 79% to 42%.[19] Some schools moved their start times later.[19] Shortly after the end of the Watergate scandal caused a change of administration, the act was amended in October 1974 (P.L. 93-434) to return to standard time for four months, beginning October 27, 1974, and ending February 23, 1975, when DST resumed.[18][20] When the trial ended in October 1975, the country returned to observing summer DST (with the aforementioned exceptions).[12]
The world extends beyond the US and many countries have successfully abolished DST. But maybe America is exceptional in some way, sure.
Possibly a relaxed attitude towards driving standards combined with a complete reliance on cars?
Outdoor lighting is a lot cheaper now than it was in the 1970s. I think we can give it another shot after 50 years. And it's worth pointing out that Arizona has gone without DST for the last 50 years and seems to be doing fine.
Interestingly part of the UK approach then was to make street lighting more efficient, around that time a lot of low-pressure sodium lamps were installed. They used so little energy they were only beaten for efficiency by LEDs in this decade, but the monochromatic yellow light was seen as unacceptable by some countries which continued to use inefficient high-pressure mercury then later high-pressure sodium.
I miss the humble SOX lamp to be honest, they made night look like night rather than a poor approximation of day. They also had benefits for wildlife, much of which is insensitive to the 589 nm wavelength as well as astronomy where the light is easily filtered out.
> And it's worth pointing out that Arizona has gone without DST for the last 50 years and seems to be doing fine.
Arizona observes year-round Standard Time:
* https://en.wikipedia.org/wiki/Time_in_Arizona
Most legislation seems to be proposing year-round Daylight Saving Time, e.g.,
* https://en.wikipedia.org/wiki/Sunshine_Protection_Act
Thats only because it’s so hot in Arizona they want to sun to set earlier so it’s cooler in summer evenings.
Arizona is permanent standard time rather than permanent DST, and is thus unaffected by the permanent-DST winter mornings issue.
France has year round DST and an hour on top during summer since 1940 and far less car accidents than the USA both in the morning and in the evening. So does Spain and Portugal but I'm too lazy to check since when. I don't think automobile accidents is a very good metric to evaluate the interest.
It's basically a trade off between light in the morning and the evening. When Britain tried, they saw that it was mainly impactless with regards to the total number of accidents. They still reverted.
> France has year round DST and an hour on top during summer since 1940 and far less car accidents than the USA both in the morning and in the evening.
In 2019, the European Sleep Research Society (ESRS), the European Biological Rhythms Society (EBRS), and the Society for Research on Biological Rhythms (SRBR) wrote a joint statement to the European Commission advocating for permanent establishment of a more natural time.
* https://esrs.eu/wp-content/uploads/2019/03/To_the_EU_Commiss...
This would mean France and Spain being in UTC/GMT, and (most of) Portugal being in UTC-1.
Actually, the document only argues for permanent CET and ending DST. It does not mention change of time zones.
The logic would be sound however. Social Jetlag is real. I for one loath DST and switching back to CET noticably improves my sleep and overall well being.
Have we tried my solution of having a continuous curve of time adjustments instead of one big discrete jump? I think that would have been awful in the day of manually set, analog clocks, but surely in 2025 when everything's digital and many things are connected it's totally possible, no?
My fist reaction was that it would be immediately rejected, but the more I thought about it, it seems like it could work if you moved clocks BACK 5 minutes at the START of each MONTH from July 1st to December 1st, and then FORWARD 5 minutes starting from January 1st to June 1st.
No, the more I think about this, PROGRAMMERS will HATE this.
I’m also not resetting the manual clocks around my home and vehicle every damned month.
I genuinely dont care if we pickstandard time” or “savings time” … i just want my year-round circadian rhythm to not get fucked up twice a year - it takes me so long to get used to the new one and then it gets rug pulled again.
It's still awful. The only devices I own that would give me a reliable time, would then be my phone, laptop, desktop, and maybe my TV. My microwave, oven, thermostat, alarm clock, car, watch, grandfather clock, etc would all be wrong.
You would wreak unmeasurable havoc across the target country.
Yeah at that point I think we'd be better off if everything was just UTC and dealt with locally
Eh, it's like 5 minutes a month they would be off by. Eventually all those devices will be Internet connected anyways and we'll all have something else new to rage over.
The only "advantage" to that is that companies would sell a lot more devices as people end up realising that their phone/central heating/doorbell/dashcam etc doesn't get any OTA upgrades and is now almost always showing the wrong time.
So you propose we stop shifting the clock but shift every opening and closing times instead producing exactly the same effect with basically no difference.
And you want to abolish DST but the end of your comment is that you actually want even more shift including in winter.
I fear I have trouble following your reasoning and understanding what you actually want.
Yes shift the opening and closing times of everything. Get everyone to coordinate, so that your kids' school time doesn't shift relative to when you need to be at work, not to mention everything else where two different organizations have to do something that regularly happens at a particular time. Give up eliding this whole problem so that some programmers have an easier time scheduling backups or database compaction.
In fact, as some have suggested, we should also give up time zones and just use UTC for everything. That way when you have to schedule a Zoom call a few thousand miles away, there is no confusion as to the time. Some idiot might retort that since people like to sleep when it's dark, that instead of looking up the time zone difference, you'd have to look up what time the sun rises where they live.
I say screw them. Half of the world should just learn to live with sleeping during the "day" and go to work at "night." We've got electric lights now. It's a small price to pay so that programmers don't have to do time zone conversions.
> In fact, as some have suggested, we should also give up time zones and just use UTC for everything.
Obligatory link to 2015's "So You Want To Abolish Time Zones":
* https://qntm.org/abolish
* https://news.ycombinator.com/item?id=39692011
* https://news.ycombinator.com/item?id=26416337
* https://news.ycombinator.com/item?id=8902765
> Get everyone to coordinate
Lol. Lmao, even.
No further comment needed.
If we're talking about radical time-related ideas, mine is to shift midnight to be the current 4AM. The reasoning is that many events last until past midnight, while 4AM is more or less the time when both the party and the work people are asleep.
While we're at it. I'd like for 9am to be 5pm, and for 12 to be 4:20.
oh please, 9am should be whenever I log into my work computer.
also obviously remove support for 12hour clocks from everything, who the duck thought having two different 4 O'Clock times every day makes sense. And what the insanity is the nonsense that write 12 mean 0 a.m. I.e. going from 11:59am goes to 12pm, like seriously who thought clocks braking basic math is a good ideaf!?!? Like the rang [11:59am;12am] is 12 hour and 1 minute wide instead of just 1 minute even through it uses the same unit.
I say keep the 12 hour clock, it is too ingrained, instead let's have twice as many days. April 42nd anyone?
Whilst we're at it, let's have 100 seconds per minute and 100 minutes per hour. I'm not sure whether I'd prefer 10 or 100 hours per day, though maybe 10 hours would be sufficient.
20 hours per day, 10 in the morning and 10 in the afternoon
Mine is to indicate time of day by what longitude the sun is at meridian.
> So you propose we stop shifting the clock but shift every opening and closing times instead producing exactly the same effect with basically no difference.
Lol no, that would be indeed even worse. I want to do the shift once and forever. Well, first and foremost I want to get rid of the twice-a-year switch. The question that usually arises then is "should we be permanently on standard time or on DST", to which my proposal is standard time (because it's correct) but shift working hours once and forever so we have more sunlight all year around. During summer it would be the same as it already is and during winter we have one more hour of sunshine, which is the time of the year when I want sunshine the most.
It could be better for YOU but you seem to assume it's a universal desire.
many people prefer to have the daylight at the start of their day.
There is no correct or wrong time, it's conventions that do not apply generally. Some countries have some timezones where summer time is more "correct" than winter time.
Therefore why not just stick to summertime and skip the whole "change every opening hours" thingy ?
If I’m reading the parent correctly, they Want to switch to non-summer time and adjust opening hours to be effectively like summer time, forever.
I think PC is proposing using the natural time zone, and adjusting hours if you need sunlight to operate. Summer time is awful because different latitudes have different hours of sunlight, yet all are forced to the least common denominator.
Cities closer to the poles might want to adjust more than one hour, or none at all since they see little sunlight for months anyway. Cities near the equator might not need to adjust at all. Businesses that need to be synchronized could still coordinate their operating hours. It's the most natural approach is DST had never existed.
I wonder what the earliest sunrise time would be as a result, 2am?
The earliest sunrise is that the sun does not set, which happens every summer for all the cities and towns north of the polar circle
Stores already do this. Check your hours at your local mall, Kroger or Lowes.
A lot of the conflict over which of daylight and standard time is better would be solved by allowing people more freedom to choose their own working hours. A lot of the rhetoric around it seems to be about what the optimal sleep-wake schedule is, which strangely tends to match up with the speaker's own sleep-wake schedule.
I don't want to be petty, but there is no 'winter time', only standard time and summer time or 'daylight saving time'.
I first heard the term 'winter time' when it was discussed to decide weather to keep DST permanently or if people would like to keep 'winter time' always. And of course who would want to have winter-something always. ;)
If we're being really pedantic, in the U.S. context the zones observe "standard time" all year. "Standard time" refers to the standardization across the zone, and the practice of advancing the clock during daylight-saving time changes each zone's standard time. The Unix-style usage of "EST" vs. "EDT" isn't pedantically correct (e.g., New York observes "eastern standard time" even in summer).
See 15 U.S.C. §§ 260a & 263 (https://www.law.cornell.edu/uscode/text/15/chapter-6/subchap...).
> and the practice of advancing the clock during daylight-saving time changes each zone's standard time.
Even more pedantically, “standard time” is not necessarily consistent across each zone (particularly, during the period for which in parts of the zone it is advanced by an hour) since "standard time” only advances for those states, or parts of states, for which an exemption is not in place.
So, the Unix-y convention of using PST for "Pacific Standard Time without advancement", and PDT for "Pacific Standard Time with advancement" is the simplest way of getting meaningful concise labels out of the US legal scheme. (This is only a theoretical issue for some US timezones, but it is a concrete one for at least the Pacific and Hawaii-Aleutian Time Zones.)
> Even more pedantically, “standard time” is not necessarily consistent across each zone (particularly, during the period for which in parts of the zone it is advanced by an hour) since "standard time” only advances for those states, or parts of states, for which an exemption is not in place.
I can't find a source (including 15 U.S.C. § 260a) that supports this reading, although I agree it's a little ambiguous. The law suggests that a region that doesn't observe DST is observing "the standard time otherwise applicable during that period" and is exempt from the provisions regarding advancement, not that "Pacific standard time" depends on where you are (see 15 U.S.C. § 263).
> So, the Unix-y convention [] is the simplest way
No argument there!
DST is just a hack to make the time roughly correspond to when the sun rises, instead of when noon is. It solves the problem that people want their day to start when the sun is up, not before, and not after. The problems it creates are mostly for programmers who can't be bothered to write good software. Well I say tough cookies, you'll have to handle the edge cases so the rest of us have comfortable mornings.
> The problems it creates are mostly for programmers who can't be bothered to write good software
Myopic and ill informed.
> A 2017 study in the American Economic Journal: Applied Economics estimated that "the transition into DST caused over 30 deaths at a social cost of $275 million annually", primarily by increasing sleep deprivation.[128]
From https://en.wikipedia.org/wiki/Daylight_saving_time, Effects on Health section. That section mentions many more negative effects of DST.
> An LSE study found that DST transition increases people's feeling of being rushed for time, and the number of hours spent on leisure decreases by roughly 10 minutes following the transition and more specifically the spring transition into DST decreases life satisfaction by around 1.44 per cent.
I think this summarizes it. The negative effects are concentrated around the transition. The positive effects are spread throughout the year. You lose 1.44% life satisfaction for a few days, and gain immeasurably more over the course of the year.
I don’t give a crap about the day starting when the sun comes up. I just want to go home at 5PM in winter and it not be dark already. There are plenty of studies about how this increases depression in the winter. The fact we keep persisting with this perverse nonsense infuriates me.
You presumably also want to go to work and have it not still be dark. You can't have it both ways.
I’d much rather go to work in the dark and have sunlight after work. The only people that it’s better for are children who would have to wait for the school bus outside in the dark, though in my experience growing up school started early enough that we were still waiting in the dark anyway.
DST wasn't a mistake and did make a lot of time (at least if you are far enough away from the equator). Heck if you go back far enough local time might have varied by session/be relative to sun rise, and that did make sense in the past.
What is the mistake is keeping it.
Through changes need to be coordinated (e.g. in the EU across _all_ EU member states), which makes it an annoying topic to takle.
That people tent to say "keep winter time" but it directly conflict with what sleep scientist tend to recommend is not helping either. In the end normal time is summer time and it is better for most peoples sleep rithm that is what we should go to.
Oh also don't shift work start/end times, the whole point is to not shift when you need to wake up/go to bed but can have consistent healthy sleep.
> In the end normal time is summer time
I think you have this backward. Standard time is during winter. Daylight savings is during summer.
That being said, I used to hate the switch, like you. Now, I'm convinced it's the best of only bad options. Just look at the history of DST. We've tried it all. We've tried year-round standard time. We've tried year-round DST. We've tried switching at all sorts of different dates. Every system has it's problems and it's advantages. The current system is basically the one that actually pissed people off the least.
> I think you have this backward.
oh, yes, I got that mixed up. That also mean permanent winter time would be better for the sleep not permanent summer times (it was whatever the "standard" time was as far as I remember). Either way people cant decide on what time they prefer makes any change harder.
> I'm convinced it's the best of only bad options.
idk. the time switch every year comes with an increase in traffic accidents for up to a weak after it happens. While most jobs today either aren't overly dependent on sunlight anymore or don't sync up with e.g. start of school or similar anyway (e.g. field work). I guess that argument does differ for any area widely dominated by farming and limited farming automation.
Farmers don’t care what a clock says at all. They start when the sunlight says they can.
It’s never been about farmers and never will be.
PS: on the other hand the affect on sleep might not matter (as in there are many far large effects). I live north enough for the number of sunlight hours and sunrise/set in summer and winter to massively differ, with the sky never getting fully dark during the few longest days.
Everyone carries a GPS in their pocket; why not use it for good? We can have people's clocks continually update so that 12:00 is solar noon at their current location each day, and avoid the jarring transitions into and out of DST.
Don't ask what people living in the high altitudes will do - I'm still working on that.
That's what people did before time zones were a thing. Is caused a major hassle for railway schedules when those started to be all the rage.
Here in Bristol, UK, we've got an old Corn Exchange that has a clock with two minute hands to show London time (GMT) and Bristol time which was about 10 minutes different.
https://en.wikipedia.org/wiki/The_Exchange,_Bristol
'Oxford Time' is still used for a few things in Oxford, the clock at Christ Church reads five minutes slow with respect to GMT for example.
Continuously updating clocks is an interesting idea, but it isn't solar noon that you want to to anchor it to. That doesn't really accomplish anything biologically useful.
It is sunrise you want to anchor it to, because it is sunrise that our circadian rhythms sync to.
What definition of sunrise? I live near a mountain range, so sunrise changes depending on what part of the mountain the sun aligns with that day.
please don't ditch timezones.
https://qntm.org/continuous
”Lets meet at the coffeeshop in town X at 5pm” becomes a real difficult thing to orchestrate.
It's only actually an issue at the poles. Even 10 meters away from the pole, there's technically going to be a solar noon.
> Don't ask what people living in the high altitudes will do - I'm still working on that.
You meant latitudes?
Maybe he means astronauts on the International Space Station, who will now have a 90 minute long day.
Why will they have a 90 minute long day? It's probably obvious I'm just not clocking(tee hee) it.
Do you not appreciate how incredibly annoying it would be if the clocks in the next town over were 5 minutes ahead of yours?
Forget different towns being 5 minutes ahead, from the other linked qntm post [0]:
> At the equator, the position directly underneath the mean Sun travels west at about 463 metres per second. That means a standard rack unit is about one millisecond wide. At latitudes closer to the poles, the effect is amplified, although but not by more than an order of magnitude in the realistically habitable parts of the world.
Even a row of servers would all have different times.
[0] https://qntm.org/continuous
> because that's the correct one
No it’s not. We assigned the meaning of time, we can assign it one hour shifted. Or better yet, just ditch timezones altogether.
Summer time is much better where I am, winter days wouldn’t get dark quite so stupid early.
> Or better yet, just ditch timezones altogether.
that is quite a terrible idea IMHO (at least if done internationally)
sure they make international meetings/events harder, but for most people most of their live is bound to local time meanings even if you travel to another country. 7am is in the morning 7pm in the evening 11:59am is mid day etc. If you remove time zone then for some people 7am is morning for other it's mid day and for other noon. So creating a lot of issues for everyone just to make international remote meetings and events easier seems like a bad idea, not even considering the absurde level of practical issues a switch like that would cause for any country not around UTC+0. Even more so remote meetings and events normally involve software and and time zone confusions are very solvable there (display time dynamically in the time zone device currently displaying it (but with time zone suffix, and some considerations about storage wrt. changing time zones, also maybe add an option to display in home instead of current time zone)). Annoying is how few programs do that properly.
Except even locally time is different for different people. People work nights, mornings, afternoons. Some shops open at 6am, others at 9am. Some people get up at 5, others and noon.
You still learn that x time is your time to do y, and people will quickly adapt to whatever number x happens to have.
We only associate 9am with anything because we learned to. We could just as easily learn a different number, and people with different work schedules do just that.
Noon should be a 12 o'clock. So whichever of the two times is closer to 12 o'clock during noon in the middle of the time zone should be the one time. I don't really care if it's DST or standard time, just pick one and if necessary adjust working hours so we get that one extra hour of sunlight during winter.
As mentioned previously on other comments, it is much more important for human activity to sync everything to sunrise instead of noon.
Sunrise varies greatly. In my country, it’s anywhere as early as 5am and as late as 10am (9am with DST)
That’s quite a range to sync to. At least syncing to noon, you don’t get much shifting throughout the year since sunset and sunrise shift roughly equally.
Oh the halcyon days of Swatch Internet Time measured in .beats (1000 .beats to a day)
https://en.wikipedia.org/wiki/Swatch_Internet_Time
I love that it's metric. One would think they'd at least give homage to 1024, but it's Switzerland so they gotta shove mod10 in our faces.
I’m obviously not seriously pushing to abolish time zones, just stating my personal opinion.
However, if it were somehow up to me, I’d would prefer a single global metric time.
Yes, about 14 years ago I made http://the.endoftimezones.com as a lark. its probably the thing I've made that's been in prod the longest.
please don't ditch timezones.
https://qntm.org/abolish
The thing I don't like with these kinds of articles is rather than list potential pros/cons they make a wholly one sided story everyone is supposed to agree with the whole way and say "oh wow, yeah" at the end. Reality is it breaks right at the start - you don't really know when a good time to call someone is by the sun. You know because of when they wake up, when they go to bed, what hours they work, what you're calling them about, when they like to eat meals, etc. All of that varies by many hours within a timezone based on culture or individual, so it derails that build up pitch. It's a given the author isn't particularly swayed by that, but that they don't even talk about the detail and just move on spoils the rest of the (well put together) list IMO.
One way or the other I don't think we'll make a big shift in timekeeping until/if we ever have a sizable population off Earth. Of course, that introduces its own time problems we don't have to deal with as much all being so close together :).
I don’t personally see a lot of difference in consulting a chart of “what time is it in x country” vs a chart of say “the time business starts in country x”.
They’d be the same exact list. “offset +9 hours”- the only semantic difference would be that clocks don’t change.
I should mention that I spent a little bit of time in Saudi Arabia and expecting them to be out and about at 7pm like in Western Europe and the USA is crazy, they seem to get up later and keep going until 3am. I’ve never seen rush hour at 3am until I spent time in Riyadh. It’s a false construct we’ve decided on: that everyone follows the same time pattern anyway.
Why do we believe the world needs to wake up at 7am? If nothing else its so incredibly arbitrary to begin with.
I never found those arguments for keeping time zones particularly compelling. Everyone has their own schedule, trying to standardise time to sync people up is silly, you sync by talking to them and asking them what times they are available. The number on the clock doesn’t actually matter.
But DST affects us exactly because people tried to standardise on time AND then shift that meaning twice a year. If we ditched timezones, then businesses could say ok we will work from 1700 until until 0100 or whatever, based on time relative to sunrise, it it would be consistent all year round.
The thing is, regardless of timezones, you have to ask people what times they work or are available or whatever. Also consider that timezones are arbitrary and made up anyway: china physically spans the space of 3 timezones yet the entire country uses one timezone.
Summer time definitely "feels" better to me. The early sunsets in winter are super depressing.
One thing I hear people say in places DST was abolished is that the late sunrises in winter are similarly depressing, and that this is something not really appreciated by those who want to abolish DST by having it be summer time year round
It’s already late though. If you work in an office, you don’t see either the sunrise or sunset: it’s dark when I get to work and dark when I leave.
Same, I hate this last weekend of October when sunset suddenly shifts from 17.00 to 16.00, it makes the winter darkness so much worse... It'll be dark before I start and end work anyway (on solstice it's 8.30-14.30), at least let me have sunlight a little later in the day so I won't feel as miserable.
In Superman III, the eponymous man straightens the leaning tower of Pisa, and in Superman II he spins the earth backwards to reverse time.
A better idea would be to un-tilt the earth’s axis, thus getting rid of variable day lengths, annual seasons, the need for DST etc, just one enjoyable spring day, every where, all year round.
NC has a house bill that has not passed committee to abolish DST, and a senate bill to only have DST.
So much of the discussion that happens around DST always end up coming across to me like they're based on people's subjective experiences that they seem to assume apply universally (or close enough to it to structure society around, with the assumption that only a small fraction of people don't experience things similarly). I don't think things are nearly so simple, and while having issues strong enough to be diagnosed with a condition like the ones I mentioned above is probably not super common, it seems pretty plausible to me that there's a lot more of a spectrum than just "people whose circadian rhythm perfectly matches the sun" and "outliers whose circadian rhythms are designed misaligned with the first group". It just doesn't seem obvious to me that the issue couldn't be that some people have different enough experiences in terms of their circadian rhythms that their optimal clock scheme might be an hour earlier or later than someone else, or that one person might be greatly affected by the total length of the sunlight in the day and another might be completely indifferent to it.
There are a number of disorders[1][2][3][4] that relate to circadian rhythm, and I don't think it's that much of a stretch to imagine that even outside the bounds of a diagnosable condition people's experiences might be a spectrum rather than a binary of "majority of people who would all have the optimal experience with the exact same scheme for how clock time should work" and "small fraction of outliers who wouldn't agree with the obvious best solution that everyone else would". It's absolutely wild to me how many people feel so strongly about the "correct" way to handle this that when examined basically boil down to "well, this is what would make me happy". I don't doubt that people probably are correct about what would work better for them, but I also don't think it's that crazy to think that it might not be that uncommon for two arbitrary people to have optimal clock schemes that differ by an hour or more, or that whether or not their optimal clock scheme might vary direct by the time of year, especially given that the length of sunlight in a day isn't at all uniform at different latitudes.
I don't pretend to know for sure whether DST is right or not, or what the correct hours to pick in the absence of it would be, but I can't help but be skeptical of an argument that it "needs to die and solves no problems" and that a certain time is the "correct" one because "I want <something>". It's totally valid to want and even argue for something, but that's not at all the same as anything else being automatically incorrect when your way of getting it is literally to enforce how time works for millions of people. Phrasing it like that just doesn't make it sound like your arguments are going to be very convincing.
[1]: https://en.wikipedia.org/wiki/Circadian_rhythm_sleep_disorde...
I'm all for abolishing DST, but I'm not sure which one I'd choose, both have trade offs.
Um, no? Using standard time in summer (assuming northern hemisphere) means that the time will be earlier when the sun goes up and under. So it will be darker at 9 PM and easier to put your children to bed. They may get up earlier though.
The early rising part of the population would love to pull that off, just like it actually happened for centuries now. The problem is that late rising part of the population has never disappeared and does have a voice. Late risers can't do anything with people collectively deciding to open work time at 7 or whatever. But they can and will voice their opinion when a public discussion about time zone will inevitably arise. Surprise, surprise, late risers exist, and they are also human.
PS: by late risers I of course mean people with shifted circadian clock, and not "lazy" people, like they are sometimes presented.
Also anyone living at higher latitudes won't be thrilled by having the sun up even earlier. I don't live that high up compared to quite a lot of people in Europe, and during summer the sun already starts shining from below the horizon at 3 in the morning
I agree with you except I think we should stick to summer time because it's obviously the correct one. And hence the whole debate.
How about we cut the baby in half and just shift back by 30 minutes.
That's a great idea. The concept of "any two places in US or EU are a full number of hours apart" is overrated.
The Newfoundlanders were on to something
In theory this is nice assuming you push the opening hours back but the cost of it would be significantly more than the developer time wasted by a mile.
Binding times to timezones to geographic locations (which have different sunrise and sunset times) to opening hours to social stigmas (you wake up after 11am? Tisk tisk) is downright silly, but humans don’t like change so fixing it will never happen.
I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year.
I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
I've had to try to clean up after a similar early decision and it was very painful.
Please stick with utc across the board people, someone someday may have to clean up your mess.
This can also be expressed as general advice.
Whenever you are unsure whether to use a clever solution or follow the globally accepted standard in your work as a DevOps or Software engineer, always choose the standard.
The Principle of Least Surprise.
Among other things, in an incident, people’s brains aren’t working - the more they have to remember about your particular system, the more likely they are to forget something.
Nowadays I'd agree with you, UTC is probably the best bet. But back then, it wasn't.
> But back then, it wasn't.
UTC was standardized in 1963 [0]
it was already a 40 year old standard at the time you're talking about.
awareness of UTC being the correct choice has definitely increased over time, but UTC being the correct choice has not changed.
you say reddit servers use UTC now, which implies there was a cutover at some point. were you still at reddit when that happened? were you still hands-on with server maintenance? any anecdotes or war stories from that switchover you want to share?
because I can easily imagine parts of the system taking a subtle dependency on Arizona being Reddit Standard Time, and the transition to UTC causing headaches when that assumption was broken. your memory of this "clever" trick might be different if you had to clean up the eventual mess as well.
0: https://en.wikipedia.org/wiki/Coordinated_Universal_Time
using utc on servers was very common in 2005
I’d say it was common enough but not universally, given the number of arguments I had from 2005 to 2015 about this exact issue.
Hold on, I'm not a sysadmin guy. Are you folks saying the server should not know what part of the world its in, that basically it should think it's in Greenwitch?
I would have thought you configure the server to know where it is have it clock set correctly for the local time zone, and the software running on the server should operate on UTC.
From a logging perspective, there is a time when an event happens. The timestamp for that should be absolute. Then there's the interaction with the viewer of the event, the person looking at the log, and where he is. If the timestamp is absolute, the event can be translated to the viewer at his local time. If the event happens in a a different TZ, for example a sysadmin sitting in PST looking at a box at EST, it's easier to translate the sysadmin TZ env, and any other sysadmin's TZ anywhere in the world, than to fiddle with the timestamp of the original event. It's a minor irritation if you run your server in UTC, and you had to add or subtract the offset, eg. if you want your cron to run at 6PM EDT, you have to write the cron for 0 22 * * *. You also had to do this mental arithmetic when you look at your local system logs, activities at 22:00:00 seem suspicious, but are they really? Avoid the headaches and set all your systems to UTC, and throw the logs into a tool that does the time translation for you.
The server does not "know" anything about the time, that is, it's really about the sysadmin knowing what happened and when.
1) Most software gets its timestamps from the system clock 2) If you have a mismatch between the system time and the application time, then you just have log timestamps that don't match up; it's a nightmare - even more so around DST/ST transitions
you've got it backwards - the server clock should be in UTC, and if an individual piece of software needs to know the location, that should be provided to it separately.
for example, I've got a server in my garage that runs Home Assistant. the overall server timezone is set to UTC, but I've configured Home Assistant with my "real" timezone so that I can define automation rules based on my local time.
Home Assistant also knows my GPS coordinates so that it can fetch weather, fire automation rules based on sunrise/sunset, etc. that wouldn't be possible with only the timezone.
I kind of assumed all computer clocks were UTC, but that you also specified a location, and when asked what time it is, it did the math for you.
Windows assumes computer clocks are local time. It can be configured to assume UTC. Other operating systems assume computer clocks are UTC. Many log tools are not time zone aware.
Computer clock is just counter, if you set the start counting point to UTC, then it's UTC, you set it to local time, then it's local time.
that's the difference between "aware" and "naive" timestamps. Python has a section explaining it in their docs (though the concept applies to any language):
https://docs.python.org/3/library/datetime.html#aware-and-na...
AKA time zones
A server doesn't need to "know" where in the world it is (unless it needs to know the position in the sun in the sky for some reason).
A server doesn't "think" and the timezone has no relevance to where it is located physically.
Common, but not universal - from 2005 to as late as 2014 I worked for companies that used Pacific time on their servers.
the standard for service providers was UTC in 1995
I have photos showing that my dad (born 1949, never in the military) kept his watch on UTC in the early 70s.
Would he by any chance refer to it as Zulu or Zebra time? The Z-suffix shorthand for UTC/GMT standardisation has nautical roots IIRC and the nomenclature was adopted in civil aviation also. I sometimes say Zulu time and my own dad, whose naval aspirations were crushed by poor eyesight, is amongst the few that don’t double-take.
While I agree on this particular instance, there are two types of things future engineers have to clean up after: Their predecessors thinking too small (and picking the easy route) or too big (and adding needless complexity).
One is not necessarily and in all instances less of a mess to clean up behind than the other.
I can’t quantify how much time my team wasted in diagnosing production glitches on checking the wrong time offsets but it was substantial. One of our systems wasn’t using UTC, and given enough time the fact that Slack wasn’t using it either does become an issue. When an outage transitions to All Hands on Deck everyone needs to get caught up to what’s going on preferably under their own power so you don’t suffer the Adding Resources to a Late Project problem.
So that first alert that came in ?? minutes ago you need to align with the telemetry and logs in order to see what the servers were doing right before everything went to shit.
What if it's your personal machine? I'm thinking about jobs I've set up... thing is, I actually do want those to align to DST in most cases. For example, ZFS scrub should start after I leave for work so that it has the greatest chance of being done by the time I get home. (It's too loud to run overnight.)
This shouldn’t be hard to deal with if the timestamp is always serialized with the offset: I’m much more picky about always persisting the offset than about always persisting UTC
That's still a bit risky as Arizona might just change its time zone definition on a whim. I'm an engineer on one of the big calendaring applications, and it's mind-boggling how often stuff like this happens world-wide, sometimes short-notice (a few weeks in advance). It regularly gives us headaches.
Agreed, watching the rate of changes to timezone databases will rapidly disabuse you of the notion of any constant. It's rare that a day goes by without an update to some definition somewhere, which is astounding.
It might be instructive for PHBs to have a 'hasatimezonechanged.com' website counting the days since the last timezone DB change...
Closely related pet peeve: Log display web UIs that allow selecting display timezones including UTC absolutely insist on deriving my preferred time format (12/24 hours) from my browser language preference.
If nothing else, selecting UTC should be a very strong hint to any UI that I am capable of parsing YYYY-MM-DD and 24 hour times.
If you use <input type=time>, the browser uses locale or user regional preferences… even if the app is in an application domain (e.g. medicine, military, science) where 24h is preferred even in 12h-predominant locales. There is no way for the app to say to the browser “this time field should be 24h in all locales”, the only option is to build a custom time field
I asked the HTML spec editors to fix this, but thus far they haven’t: https://github.com/whatwg/html/issues/6698
It gets worse.
Some browser APIs respect `LC_TIME`. Others say "fuck standards, I'm using `LC_MESSAGES`".
This means that if those locales differ about say, y/m/d ordering, it is quite possible for the way the browser formats a date to be different than the way it parses the date.
My issue is more with outputs than inputs (although the latter are also annoying to me in 12-hour format).
Operating systems generally allow user override of locale settings, and browsers generally respect that; I use a locale which officially has 12 hour time as its standard (Australian English), and I always override it to 24 hour time in user preferences (although Australia is rather inconsistent, e.g. in Sydney, train services use 24 hour time; in Melbourne, the metro trains use 12 hour time but the regional services use 24 hour time)
Interesting, so you're saying that that OS-level time preference is available via JavaScript? I wasn't able to figure out how to query that in a little bit of trying, so I assumed there was no API for it.
If there actually is, I'm now even more upset at that log web UI.
Well, if you run this Javascript:
It should return the current time, formatted according to your locale and user preferences.By contrast, this should return it formatted with the defaults for your locale, ignoring your user preferences:
For me, the first returns 24 hour time and the second returns 12 hour time. Because 12 hour time is default for my primary locale (en-AU), but I override it to 24 hour time in my macOS settings.I know the same works on Windows, and I’m sure it works on Linux too, just the way you configure it is different for each OS.
I would especially like to call out the Scandic Hotels chain for this behaviour as well. Booking a hotel room should not involve me wondering if I booked it for the wrong day when I'm not in an European time zone.
My favorite example of that so far is the company that managed to shift my birthday by one day between their mobile and web apps.
That's interesting because I strongly prefer that timestamps are stored as UTC and converted to my timezone on the fly for easier debugging. I dunno about using the browser language choice (do sites really do this)? Usually a simple conversion with javascript is fine (javascript knows the local timezone).
A plus would be showing both.
You can blame the browsers on that one. There is now an API to determine hour cycles, but it's not even supported in all browser yet:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
That is the correct and only sane way to determine date format, timezone is not the same as formatting preference. But yeah it sucks. Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.
In firefox you can try it out in about:preferences by setting en-CA as the topmost option in "Choose your preferred language for displaying pages"
And I just checked, firefox is capable of understanding system-wide split locale settings, if you only want LC_TIME
and dates will format, but currencies, addresses, etc will not> That is the correct and only sane way to determine date format, timezone is not the same as formatting preference.
That's unfortunate, but then wouldn't the sane way for localization-aware software be to ask the user and not make any such assumptions?
> Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.
Thanks, I'll give that a try tomorrow! If my log exploring UI of, well, not quite choice actually respects that, I'll be very grateful :)
In a way it is asking the user. Date formatting rules are traditionally tied to locale, so the user picks their locale and their expected date formatting rules are derived from that.
On Linux you can mix and match via LC_*= environmental variables, but that appears to be complexity the browser vendors didn't expose in their UI.
Just a strong hint your fellow techdorks were being at work. Oh so clever, aren't they, always?
Java allows setting the default timezone at jvm level and everyone in our org were setting their favourite TZ which was mostly PST somewhere in the code.
We had application logs and system level logs with different timestamp and someone decided a certain db column has to be a string of the format 'yyyy-mm-dd hh:mm:ss". You can imagine the confusion and the "fun" time we had while debugging logs from multiple systems at specific times.
Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
> Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
That seems to me like a really bad mandate. If you are going to mandate something, mandate UTC. If someone forced me to change a system from UTC to something else, I’d not be very happy. It’s the kind of decision which makes you seriously question if you want to work there any more.
Does that mean you ran PST during daylight savings time and half the year your logs were “off” by an hour? Or did you actually run Pacific Time and deal with the clock changes twice a year?
Doesn't the JVM handle this when you set the tz? Otherwise...how is it different than just setting a clock?
They're two different timezones. One is a UTC offset and one is a dynamic timezone (DST).
> I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
Also important here is that the date stays the same! Even though I've gotten used to instinctively decoding UTC, that part is still frustrating sometimes.
What about that one hour in the day when the date isn't the same?
You probably aren't working that hour. (also way better than 8).
And the computer isn't doing anything interesting during that time? You're reading logs, not using the computer as a clock.
Computers can do things. Build a UX to read the logs and have it automatically parse/convert the logs to show whatever is local time.
https://unix.stackexchange.com/questions/274816/how-to-conve...
There are many solutions, but when you're actually running a site with millions of daily visitors and one person focused on ops, you do the easy thing, not the right thing, unless those happen to coincide.
Sure... "I'm too busy fighting fires to come up with a solution" is always a valid answer, but that's not what I was replying to.
Even with infinite resources (love to see that in real companies!), any such solution adds complexity, which comes at a cost.
Naively storing timezones also adds complexity, just later and probably someone else's problem.
Timezones are naturally complex. They just are. If you're not handling it, then your stuff is broken.
If this is a complexity battle, I'd rather that sysadmins have to use a tool to read logs vs. figuring out why the server is melting down because all the jobs are running 2x or answering questions about why jobs didn't run at all.
In my old gig, jobs ran for many many HOURS and consumed most of the IO on the server (processing reams of data), which meant that when you got them running 2x because of overlap, it was a trainwreck.
In my experience, it's really not pleasant having to work with logs that require a dedicated viewer compared to regular old text files I can use Unix commands available everywhere (`tail`, `head`, `wc` etc), with, i.e. not just on the server producing the logs, but also the device I'm viewing them on.
That said, I absolutely prefer UTC times in logfiles. I'd rather do some math every time than make wrong assumptions about the local timezone of the file even once. (Even correctly indicated local times are more annoying, since I have the math for the offset of my timezone to UTC memorized, but not that of every possible server time to mine.)
alias tail="my fancy log viewer tail wrapper"
https://thecasualcoder.github.io/tztail/
That's exactly the type of thing I like to avoid having to do on some remote server, inside a container's minimal shell etc.
If you're at that level, I'd be exporting the logs anyway.
Yup. There's always some kind of tool anyway so why not. I mean, even if you read the logs as they come out of a screaming line printer, the tool is the thing that takes log messages and spits them out the printer port.
How hard can it be to write a log cat utility in awk?
Harder than not having to, in any case.
I mean, if this doesn't depict modern devops, I don't know what does. Unsung heroes, honestly.
Nooo... we have a bunch of metrics and logs reporting systems at my company. Some of them are in UTC, some of them display my local time, some of them display China Time, and I'm trying to collaborate with colleagues in London and Australia who get data displayed in their local times as well. When I'm working to address an incident and combing through multiple systems to try to correlate events, it's a pain in the ass having to constantly double-check which time zone this data is in.
In my experience, the hard part is getting everybody else to do that. And then also getting them to actually include the timezone in their communication with you.
I'd rather the logs be consistent, and not rely on every single person who ever looks at the system logs make sure to use the correct tool the correct way.
If you'd rather the logs be consistent, then log to UTC.
The problem is a lot of times when you are looking at logs you are already very far off the happy path of things you look at often and want to invest resources into displaying well.
> Computers can do things. Build a UX [...]
That sounds like a job for a human (at least the review part), not only a computer.
This is not smart, this is just surprising to the next person who is going to maintain your ”smart” tricks. Thank god they switched to utc, that is what everyone expect.
Actually, back then, 18 years ago, most people expected your servers to be in Pacific or Eastern time, depends on where your company was headquartered, because none of us really had global technical workforces back then. We all pretty much worked in one office and used the local time zone, because often our servers were in the building with us or in a datacenter nearby.
Case in point, before reddit I was at eBay, and we kept all those servers in Pacific time, since the entire technical workforce was in Pacific time, as well as all of the servers.
Making blanket statements like that without considering the context of the time is usually not a good idea. ;)
> most people expected your servers to be in Pacific or Eastern time
I was there back then, working for shops people have heard of, and I honestly don't know where you're getting this idea from. Some places did things wild and wacky when they were wee small, but most of us quickly learned that such shenanigans (like fun server naming conventions) start to fall apart and maybe we should do things differently.
Using UTC for servers was standard when I entered the field in 2005.
I was setting them to UTC in 1995.
Ah you think UTC is your ally? You merely adopted UTC. I was born in it, molded by it. I didn't see DST until I was already a man, by then it was nothing to me but blinding!
In the 1980s, PT and ET were common. I was working at Bell Labs then, and one of my jobs was to change the time zone (back then it was two words) on the testing machines, as needed. This is stuck in my memory since to change the timezone, you needed to edit the Unix kernel source code and recompile it!
2000 for me. That was the first time I had users from outside my own time zone, so I figured it was better to just use UTC for everything and just convert depending on the user's settings. I think I just applied the thinking to the whole server.
This is the way.
Yelp servers were set to Pacific time when I started in 2009, probably a decision from 2004
I run into this a lot when working with legacy code. The first reaction most teams have is to mock it, not understand it.
Everybody's dunking on you here but yeah, circa 18 years ago I remember that setting servers to local time was still pretty common.
It only matters if the servers were running cron jobs where it mattered if they ran "not at all or 2x."
Logs with weird dates on high demand production servers... less important.
Can you please make your substantive points without swipes? This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.
I believe it was an anecdote and not a recommendation.
It was probably the smartest option at the time, given the context. As long as it was documented properly, no one on such a small team would have been surprised. Spend more than the 30 minutes you've spent here on HN so far, and you may learn a thing or two about what is "smart", and how that is inextricably linked to the given situation.
Sounds like it was smart at the time, and then eventually outgrew the solution.
If you're like 5 people, each with a list of TODOs that doesn't fit on one screen, it's pretty smart to just do something quick and good enough, then move on, revisit it in the future.
"revisit it in the future" is exactly the reason your TODO list keeps growing rather than shrinking
Nice when the company lives long enough for that to be an issue
I worked at a place that had set the servers to BST not UTC and used cron jobs extensively for reporting, which caused chaos twice a year and twice a year they had to explain to the business again why it caused chaos and ask for the time to fix it.
Company went bust before it got fixed.
Just use UTC folks unless you have a really good idea why you shouldn't.
I put a little analog clock on my desk that's set to UTC time. It helps a lot with converting logs to get a quick reference.
I have a piece of paper on my desk, each side has the time zone, utc offset, and date when DST will change. Twice a year I flip over the paper.
Right now it reads "EDT, UTC-4, until: Sun, Nov 2"
Going to add a clock next to it now, that's a great idea
And then with distributed team you realize DST change dates are different around the globe.
Desks around the globe can have different pieces of paper.
Analog clock is a great idea. I just use the Windows multiple timezone clocks feature, but I can see the usefulness of being able to glance at UTC.
I did this back when I used Windows. I suppose I'll have to add multiple clock widgets to my XFCE panels to get the same effect.
Expense a Rolex GMT Master or Sky Dweller. Tell your boss I said it was necessary!
Perhaps a shell alias for:
TZ="Etc/UTC" xclock -analog -update 1 -norender -hl grey -fg grey -bg black
(or similar) would also suffice.
I do the same! I am in PDT and keep a UTC and EDT clock (most of my team is east coast)
One of the great things about living at 0 latitude is it's UTC half the year
Did you mean longitude?
Yes, thanks. Having a baby will play havoc with your brain.
I really love this idea!
"Our customers aren't in UTC" is often a compelling reason.
People generally expect things like usage limits to reset overnight, not at UTC midnight, and these are often implicitly tied to (the result of) batch jobs.
Financial jobs, including billing, is another big category.
Having tried many times to use local time instead of UTC to make life simpler for myself, I really don't think it's ever a good idea.
Better to localize times for users on the client-side based on their local settings, IMHO
For quotas and usage limit resets it can be a little harder, but if it's a cron job anyway, my initial approach would be to just schedule it for a time that is close to midnight for the area your customers are in.
That does not require setting the server to local time. Run the cron job every hour. Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. This also happens to be more resilient to server downtime around midnight.
> Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit.
That sounds a bit like reimplementing a scheduler inside the scheduled task (which is ok if you don't trust your scheduler, but I'd avoid it if at all possible).
Except downtime can start during execution. So for many types of jobs...
- Check if run today, exiting on "yes"
- Run
- Update the last-run-on date
What is "today"?
Hopefully just a datetime recent enough that the job does not currently need to be (re-)run.
But as soon as one job needs another to complete first, or two jobs shouldn't be run at the same time - things start getting messy.
> Hopefully just a datetime recent enough that the job does not currently need to be (re-)run.
I can think of several ways to mess that seemingly trivial calculation up just from the top of my head. (Consider changing timezones, increasing the interval the job is running to within half a day of scheduling period etc.)
> "Our customers aren't in UTC" is often a compelling reason.
This doesn’t work as soon as you become a global business… because if you tell a customer in Asia that it resets at midnight in some US timezone, what are they going to think?
Many people will accept UTC because it is the international standard. Everyone will accept “let each customer pick the timezone that works best for them”. Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
Really depends on your market and customers. Especially in the financial industry, local time zones (often of a given currency's central bank's main branch, a primary regulator etc.) are often very, very entrenched.
> Many people will accept UTC because it is the international standard.
I really wish that were true universally...
And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more.
> Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" one for resetting quotas etc. has some merit as well, if only because it makes the concepts of "today" and "tomorrow" work a bit better. (UTC midnight might be very unintuitive to both you and your customers.)
> Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical"
What happens when you move your head office to a different time zone? e.g. Oracle moved their HQ from California to Texas (and have announced a further move to Tennessee, although last I heard that still hasn’t happened)
What happens when an M&A happens and you are now part of a much larger enterprise with a different HQ timezone, and systems with wildly inconsistent configured timezones due to having acquired multiple companies which had HQs in different time zones and all of which configured everything to use their HQ time?
> UTC midnight might be very unintuitive to both you and your customers
I live in eastern Australia, UTC+11 at the moment (UTC+10 during Southern Hemisphere winter). Right now, resetting stuff at UTC midnight means at 11am my time, resetting stuff at US Eastern midnight means at 3pm my time - roughly equal in potential inconvenience, but I’ll be much more forgiving of the first than the second.
> And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more.
For me, add/subtract 10 is pretty easy… and then just one more during DST. Does that make UTC “Australia-centric”? Or Guam-centric or Vladivostok-centric? (both of which are also UTC+10)
Always set the servers to UTC. Convert the time to local time for display, but store in UTC.
The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time.
That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes".
Do you actually care if the requests were made on some specific calendar day? Or do you just want to make sure that heavy users are paying more and/or people aren't abusing your system?
Instead of tying your quotas to calendar days in some specific time zone, tie them to a rolling 24 hour usage period. Even better: use a rolling hour.
The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time. This is exactly what I said above.
That works if all your consumers of the report are indifferent to it arriving at a different time half of the year.
All of yours might well be, but you can't generalize that assumption to every domain and scenario.
If you want an email of the report to send out at 9am locally. Generate before and send to everyone based on timezone.
> Generate before and send to everyone based on timezone.
How are you going to do this? Clearly cron (with timezone set to UTC) doesn't work for this so you'll need another system. In that case why not have that multi-time zone scheduling system run your script?
You don’t want a report to include 25h or 23h twice a year just to be able to send it at some distinct local time.
I absolutely want some daily business reports to cover 25 hours on one day of the year and 23 on another, since that's how long a calendar day is in local time in every place that has DST shifts.
Yes, that's exactly what I want. Everything shifts with dst - business processes, people's working hours, ach windows, ...
The report can run at whatever time you want it to run at. But this solution solves the issue of the report running 2x or not at all.
> That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes".
How hard is to convert the UTC time to user's local time and perform quota reset?
I mean you should be doing this anyway. Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
>Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
1. allowing users to change their timezone and tying it to when their quota reset sounds like a great way to add more edge cases and complexity. For instance, does messing with your time zone mean you can get your quota reset multiple times a day?
2. Not everyone operates some sort of global b2c SaaS that's timezone agnostic. For many enterprise backoffice tasks "6am HQ time" is far more reasonable than "6am/7am, dunno depends on whether daylight saving time is on or not". In this case having a server set to the HQ's timezone makes far more sense than doing UTC and trying to work around daylight saving issues.
> How hard is to convert the UTC time to user's local time and perform quota reset?
Probably roughly as hard as writing a timezone-aware scheduler that considers all edge cases around daylight savings time, i.e. probably possible but I'd try fairly hard to see if I can get around it.
One way of getting around it is to run your batch jobs in your "primary business timezone". Sure, you might outgrow that concept eventually, but many companies never do, and in that sense "running on UTC" seems similarly aspirational in spirit (albeit at a much smaller scale) to starting with high scalability.
You could have the reset script check the timezone of the user before actually performing the reset, and have this script run at least once an hour to make sure you get close to accurate for most people (some timezones only differ by less than an hour though!).
But also, if the user can change the timezone themselves, now you have a condition where the user could keep their quota at bay for a very long time (if not forever) if they keep changing their timezone accordingly.
Also don't use cron full stop. Build (or use an off the shelf) scheduler in your app that's more sophisticated, has access to data and client preferences, etc.
If I want a report of what happened at a specific time I need that report at local time
I get a daily status report of various things from our 24 hour operational management team which runs at 8am UK time every day. That means last week it ran at 0700UTC and this week at 0800UTC
This is built around operational events, shift changes, etc.
I've got another system which is in operation in Sydney from 0630 to 1630 local time, this means that maintence windows which overlap with UK shift patterns depend on the week but mean the system is operating 2130-0730 UK time at some times, 2030-0630 UK at others, and 1930-0530 at others.
UTC is not "the answer". Sometimes you want things running at a UTC time, sometimes you want them running at local time.
I have a regular meeting at 10am London time on Tuesday and Thursday. That can't be stored in UTC as it varies depending on the time of the year. It has to have the timezone stored and actioned.
I agree with most everything except:
> If I want a report of what happened at a specific time I need that report at local time
this is hard when a particular hour can happen twice in a day right?
It certainly is, but that's a business question, not a technical question
If the business requires it to run at 01:30 and there are two per day, or zero per day, then the business rules needs to define what happens. You can't solve this by running it at UTC.
Technically correct, but the business needs to be aware and a solution needs to be proposed beforehand IME. UTC is such a solution, imperfect, but usually workable. Business rules are sometimes unexpectedly elastic when confronted with a particularly explosive mix of logic and legislation.
You can tell the job when to run in UTC time and change that time depending on what the current DST rules are.
But if the job is set to run in UK time and the system clock jumps around because of TZ changes, it will run twice or not at all.
Having a stable zero based time that doesn't move around by external forces that you can't control is "the answer."
So I have to manually update the job
How about I use some form of library to do it. I tell it I want to run at 0800 London time, and it runs at 0800 London time
If I tell it I want to run at 0130 London time (or 0330 Athens time) I still have a problem -- do I run it twice when the clocks go back, do I skip it when clocks go forward?
But that's a business logic problem, and defining it as UTC and having another job to update the time twice a year doesn't actually solve the question of "what do I do at this point".
Generate earlier and send to various timezones when needed.
You don't have any choice in the matter. If you control the time you want the jobs to run externally from the server, then you can have control over it. Otherwise, you're getting jobs that run 2x or not at all.
No matter where you run the job from, you run it in the required timezone, not UTC, and you still have the issue with what to do with the change in the local timezone.
This isn't a problem that can be solved with a single technical solution.
Generally, yes but I there's a surprising amount of cases when it is important, which makes it difficult to generalise e.g. Huge amounts of the financial sector cares because of market times or regulatory reasons.
I use UTC for all public/production servers, but for my homelab servers in my closet rack they all use local time. Makes grokking times in homelab servers much easier. The exception is database insert/update date/times which are always stored UTC.
At Marin Software, they ran each customer on their own set of servers and set the servers to the customers defined TZ. It was an endless cronjob nightmare. Now they are also bankrupt.
Just found out some guy decided to try to restart the company. Good luck. https://x.com/Austen/status/1981904435539280324
> Just use UTC folks unless you have a really good idea why you shouldn't.
If my whole team is based in Pacific Time, I'm gonna use something close to PT so I don't have to do hard math when reading raw logs.
If we're all US based, I still want something in the USA. The math is easier.
`date -u` gives you current UTC time when the arithmetic trips you up.
Hard math. Ah right, I think it's not till the third year of a mathematics degree that you deal with "subtracting 8"! :)
Don't forget changing the day 1/3 of the time. But more importantly, if I'm scrambling to solve an incident, the last thing I want to do is deal with time conversions in my head.
You just do it once and think in UTC going forward. But as I said to the other person, if you're only dealing with one timezone it's fine, just when you have multiple it's a lot easier to just deal with one time and let everyone convert.
I'd expect everyone who works in computer related jobs to know their UTC offset.
I'm able to subtract 8, but if I'm scanning logs it's one more thing to process.
If it's local time I know instantly when something happened, without having to do mental math.
Is there anything wrong with ISO8601 (including timezone offset) for storing times? They're in my local time, but they can be accurately converted to any other timezone.
No problem as long as it's all local, but it's a big annoyance to the other teams if I'm trying to coordinate with the West US team who're on PT, the East coast on ET, Europe on CET, India on IST, Australia on BBQ...
It's just easier for everyone if we agree on UTC and everyone knows their own offset.
Technical debt. Please don’t.
Yeah. I log in json + unix timestamp nanoseconds, and just convert it to something human-readable on display (github.com/jrockway/json-logs). I am not sure why logs need to be "pretty" at time of logging when they can be made pretty at time of display. Doing it at time of display means you can just use local time, and then nobody is confused.
Why not just store everything utc but add a local time to the logging along with it?
Nowadays you could probably pull that off, most tools support quickly changing the time on the fly.
Apparently there are some timezones (Cuba, Egypt, Lebanon) where DST change happens at midnight, so, also watch out for that!
https://mas.to/@mpirnat/115395859892135002
Wait, are there timezones where the DST change doesn't happen at midnight? That's news to me.
Now this article makes sense to me; I was wondering what made 02:00 and 03:00 special, since the DST changes would be from 00:00 to 01:00 and from 00:00 to 23:00, as I'm used to since childhood here in Brazil. Perhaps some other countries change DST from 02:00 to 03:00 and vice versa?
In the UK we move forward at 1am and they go backward at 2am. Doing it at midnight adds the extra complexity that now the day is different. Doing it in the early morning doesn't change the day.
My guess is that in the US they do the same but shifted by one.
Most timezones change time when there's a minimal amount of people working, as these people would have to work an extra hour, and doing it at around 3am is the most reasonable from that point of view.
In the United States, we go from 1:59am to 3:00am in March, and from 1:59am to 1:00am in November for our time change.
All of Europe changes from 02:00 to 03:00
Better to think of it that it changes at 01:00 UTC, which takes care of the parts of Europe that are 2 or 3 hours ahead (instead of CET's 1 or 2), and the UK going "forward at 1 and back at 2"
"Europe DST changes at 01:00 UTC" - much simpler ;)
There are TZ rules that do all sorts of wild things.
Leading to my favourite web comic of all time.
https://www.monkeyuser.com/2018/going-global/
Monkey User is always entertaining.
Is there any other person other than Arthur David Olson who needed an RFC written to cover their retirement?
https://www.rfc-editor.org/rfc/rfc6557.html
Jon Postel, the original RFC Editor and IANA: https://www.rfc-editor.org/rfc/rfc2468
Brazil not only had DST at midnight, but until 2008 they also had no standard rule for when DST would begin and end, setting the dates by decree often just a few weeks in advance.
https://www.timeanddate.com/time/time-zones-interesting.html
I wonder how on earth do you deal with a 30 or 45 minute offset in real life
Redundantly running cron jobs are the least of our problems. Daylight savings adjustment day causes surge in deaths and ER visits[1]. Daylight savings needs to go.
https://www.sciencealert.com/daylight-savings-time-change-ki...
This was worked around ages ago in OpenBSD, and the workaround was already included in Debian (and by extension, Ubuntu) when I started maintaining it in 2010 (I no longer do).
Here's a link to a patch [1] from a version from when the package still kept standalone patches.
It's been a long long time and I honestly don't remember the details, but Debian's cron(8) still says [2]: Special considerations exist when the clock is changed by less than 3 hours, for example at the beginning and end of daylight savings time. If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run.
Edit: According to this bug report [3], this workaround first entered Debian in 1999.
[1]: https://sources.debian.org/src/cron/3.0pl1-137/debian/patche...
[2]: https://manpages.debian.org/unstable/cron/cron.8.en.html
[3]: https://bugs.debian.org/8499
So we can assume the author was not running a distro that had "fixed" the issue?
Most likely, yes. The author mentions vixie-cron, which was the name of the project before Paul Pixie joined/founded(?) ISC, and it was released as ISC from after.
Debian's fork is still based on vixie-cron, but it couldn't have been the one because of the aforementioned patch.
Also, never set jobs to run at midnight (00:00) as nobody will be able to tell what day it is. Set it to 00:01 or something.
And tbh, don't run jobs on the hour anyway. Everything kicks off then. Set stuff to run at 01:45 or 02:15 or something instead.
(None of these suggestions are time change related)
In general I try to use odd times for cronjobs. Backups starting at XX:13, analysis scripts at XX:23, data exports at XX:42 and so on. This simplifies the question of "Why does my system go whack at 23:23? Oh, wait".
That's a good idea.
With enough precision in your timestamp, you can base-10 encode arbitrary binary data and shove it in your nanos for debugging.
This is a great tip. One of our QA engineers noticed recently that a particularly difficult-to-reproduce bug was affecting records saved at :21, which is when a particular cronjob hits. Without that extra info we’d probably still be scratching our heads over it.
Honestly haven’t seen 00:00 cause confusion outside of “dog ate my homework” scenarios. If you’re using 12h clock then clearly it’s a bigger issue.
The details I never really thought about until I encountered timezone issues was:
There are points in time that don't exist, and there are points in time that are ambiguous.
If we adjust the time forward by 1 hour, then then 2:30am doesn't exist. If we adjust time backward, then 2:30am is ambiguous.
2am-3am is only safe for certain countries.
America/Santiago (Chile), for example, adjusts their daylight saving on midnight. On a certain day, midnight doesn't exist because it immediately jumps to 1am. If you are building an analytics chart by day, you will encounter this issue. I wrote about several weird time zone edgecases here: https://tanin.nanakorn.com/edgecases-for-timezones/
To add more confusion, IIRC Java and Ruby handle these cases differently where Ruby raises exceptions (e.g. InvalidTime and AmbiguousTime) but Java doesn't.
I caused 3 incidents in a row at Stripe in my first month because of this. I thought I was gonna be fired. Luckily I wasn't...
While not always possible, in general, any mission critical scheduled job should be idempotent. This avoids so much headache, especially when there's a queue involved somewhere.
I try to avoid crons at the top of the hour, partly because of this... but also because (in shared / serverless infrastructure) I assume a lot more people are setting their crons for 'on the hour' so there's more resource contention... I also aim for 'after 4am' where I can as well, or 'before midnight', to avoid this whole range.
To incrementally improve that tactic, systemd has RandomizedDelaySec, which is a convenient way to reduce the possibility of scheduling conflicts.
I prefer to combine this with FixedRandomDelay=true. FixedRandomDelay ensures that the randomized delay is an arbitrary number up to RandomizedDelaySec, but it is deterministic per server and timer. I find this useful because this means the timer will always run at XX:12:45 on server01, always run on XX:06:23 on server02 and so on.
This combines very simple configuration, while being predictable and spreading out timers well.
One trick for cron is to prepend the actual command in the crontab with something like
which will delay it by 0 to 59 minutes at random.I like cron jobs that run during the work day. If something goes wrong I would prefer to handle it at 10am rather than 10pm. Why make life unpleasant?
If it really needs to run during a low load period I line it up for 6am work time. Again so that if something goes wrong it leads into the work day rather than disrupts the on-call too badly. 6am also provides a daylight savings buffer either way so you don't have to care. Recent work has all been on 24/7 operations though.
Doesn't systemd timers fix this? AFAIK, a job set to run once a day shouldn't run a second time because the time rolled back.
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all.
Is this what people actually want, though?
I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input).
Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March.
Am I missing something?
The article is mostly talking about using the `cron` scheduler and running things at 2am on Sunday because there's especially low traffic. These cron jobs are "I don't care when it runs, but it should have minimal chance of causing problems."
Your use-case is totally different: "I want this job to run at this time" so the only lesson that applies to you is that the `cron` utility might behave weird during DST switches. No idea if it underlies your cloud provider, so it may be completely irrelevant.
> Everybody wants the job to run four hours before the work day starts
recurring events like this - ones that take place relative to humans in a given location - are the most common snag related to "just use UTC everywhere" advice. but they're not actually in conflict at all. they just require an extra layer of indirection.
let's say this is a factory that starts early in the morning, 06:30 local time, because then "four hours before" falls into the window affected by DST.
"run at 02:30 local time" is what you want to avoid, because on the "spring forward" day, 02:30 simply does not exist. the clocks jump from 01:59:59 to 03:00:00. [0]
likewise, "run at 01:30 local time" is to be avoided, because on the "fall back", that occurs twice.
ideally, you want to think of the recurrence as being associated with a given location, and that recurrence generates individual events, which happen in UTC time.
for example, if the factory is in San Francisco, the recurrence might specify "once a day, at 02:30 SF time", which would generate a series of daily events in UTC time. during the summer those events are at 09:30Z, during the winter they're at 10:30Z.
you also want the scheduler to be smart enough to understand the "once a day" constraint - so that a "once a day, at 01:30 local time" task generates only one daily event, rather than two, even though 01:30 local time happens twice in the same day.
and then, because the individual events are all in UTC, and the system clock is as well, there's no ambiguity in the actual execution time of the event. the complexity of the recurrence is handled at a higher level.
if you want an even harder version of this problem, imagine a team in SF and a team in London, that have a Zoom meeting every Monday at 8am (SF) / 4pm (London). it's necessary for one of those locations to "own" the recurrence, because the UK does their time changes 1-3 weeks offset from the US [1], creating weeks where it isn't a straightforward 8 hour time difference.
0: https://www.timeanddate.com/time/change/usa
1: https://www.timeanddate.com/time/change/uk
The more I read this thread, the more I think this is just a bug in Cron. It should really only operate on UTC, regardless of the local time set on the server.
with systemd timers [0], explicitly specifying the timezone is an option:
however, making it disregard the server timezone and use UTC by default isn't really an option. it would violate the Principle of Least Surprise, as well as being a hugely backwards-incompatible breaking change.0: https://www.freedesktop.org/software/systemd/man/latest/syst...
I call this "rake stomping", as long as you're running back and forth in the same places every time, you'll never notice.
I read this a while ago and was looking forward to the mess that would happen with these default Alpine configurations (after setting non-UTC localtime)
It appears that busybox cron is smart enough to handle it and only ran the jobs once.Google's AI and ChatGPT answer disagrees.
https://www.google.com/search?q=busybox+cron+dst
"BusyBox’s crond is a lightweight implementation designed for embedded systems, so it doesn’t have all the timezone/DST logic of full vixie-cron or cronie. It just reads /etc/TZ or the system timezone at startup and then wakes up every minute to check if something should run."
Please don't do this. AI is not a source and is frequently wrong (as it is in this case).
https://github.com/mirror/busybox/blob/master/miscutils/cron...
Seems to me the real problem here is not the timezone (there's legitimate business needs to run something daily at a specific localtime...) but having multiple instances of a cron job that overlap, in which case it should wait until the previous is done or not start at all. At least prefix a job with "flock -n" if it doesn't/can't handle that.
For the most part of Europe and USA the day is either "quite" long (summer) or quite short (winter) due to the curve of the duration. Having light for long summer evenings and dark for long winter mornings is bad for many reasons, one of witch is seasonal depression, with which the Scandinavians suffer and fight. I was for the abolishment of DST until COVID cancelled it in Europe but now I am thinking it again and probably I am against.
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all
this is the way
Use TAI
anacron is worth a mention here for sometimes-on computers like desktops, laptops, or personal servers. It's great for ensuring something runs at the set interval, or just whenever the machine comes back online. Minimum anacron interval is one day, so this can't simply replace cron. cron might even be the most sensible thing to use to kick off anacron on your system, and it might already be in use (see if your /etc/crontab fires off anacron).
anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.
2 am is mid afternoon for the Pune office. If you’re running a 24/7 operation and all of your employees are in PST you’re gonna have a bad time.
But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic.
The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration.
It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.
I run a cronjob at 2:30AM every day that uses dirvish (a perl wrapper around rsync) to perform backups of several computers. In the fall, at 2AM the clock is rolled back to 1AM. The cronjob runs at 2:30AM, which is 25 hours after the previous day's run. That's not a problem. However, in the spring, at 2AM the clock jumps to 3AM. Vixie cron seems to be smart enough to run all the cron jobs that were scheduled between 2M and 3AM at that point. However, dirvish is started but exits without running the backups! After some troubleshooting I concluded that dirvish does not make the backups because the previous day's run happened less that 24 hours ago (because that day was shortened by the change to DST) so it sees no need to run a backup yet. This problem would persist even if I moved the cronjob to a different time because it has to do with the day having only 23 hours. I solved this by adding a second entry to the crontab that runs at 3:45AM only on the 2nd Sunday in March.
> better open source job scheduler [...] one that will ensure no overlapping runs
Or learn to use lock files, which would make sure that your jobs don't run when another one is already running?
Unix shells offer the trap command, among other things, to cleanup, even in the case of errors, e.g.:
https://en.wikipedia.org/wiki/Semaphore_%28programming%29Tell me how that goes after a server malfunction or power failure. Where is your trap god then?
The topic was to prevent overlapping runs. In case of power failures etc all your server processes have this problem. That's why, for example, /var/run exists and files/directories are deleted from /var/run on reboot?
The problem is not so much which timezone you pick, the problem is that cron uses 'naive' datetimes. If the timezone is explicit then you don't have duplicate hours because one is 02:00 and one is 02:00DST.
And then you can translate that time to any other timezone with ease, knowing that you are referencing a specific hour in physical time that only ever happened once.
> knowing that you are referencing a specific hour in physical time that only ever happened once
"Falsehoods programmers believe about time"
This assumption can break with:
* Leap seconds
* Calendar and time zone changes
* (In exotic circumstances) Relativity
I'll give you leap seconds, but at least in Python historical timezone changes are recorded in the datetime library, so the timezone at any particular time in the past has the timezone offset that was active then.
Which is to say I wish cron had the TZ variable that Jenkins uses in it's scheduling system. And the H value to easily spread multiple jobs over the hour/day/month etc.
Not a crown job, but I configured my lights to turn on progressively at my scheduled wake up time, using Home Assistant.
I figured that scheduling lights to turn on at <0 AM + wake up time> was a good approach... We got a surprise early wake up last year, so I ended up changing the formula to <4 AM - 4 hours + wake-up time>.
I avoid cron job at this time for this reason and for many other random problem because 3:00 am seem to be often choose for a maintenance like automatic update, server reboot, database backup..
Can be tricky: an old project had a job that ran every 15 minutes to poll data from a popular smart thermostat company. I always knew when DST was starting or ending because that job would throw an error email at 2/3am on those days.
Great to read about something that's a solved problem now. It's been a very long time since I used a server that wasn't set to UTC.
I've seen this same problem under UTC. Gotta use TAI
UTC also solves plotting daily time plots in zones with time changes (twice a year when the plots make no sense). But there still isn't no good fix for leap year plots!
Just use TAI?
Or just use UTC and stop using timezones in code. I get it’s not that simple for some use cases but UTC exists for a reason.
Use TAI.
UTC has most of the same wacky timezone problems, just less often.
You’re signing up for a world of pain in the real world, where every other system is UTC. Just use UT1 or smear the leap second if you really can’t stand the risk.
2-3am is bad due to DST but there is another reason to also avoid 3-5am or so
around 3am is the time where "in average" human attention is the worst as its where "in average" their most relevant deep sleep phases are
now I guess the average over HN readers has a good chance to be somewhat later
but either way if the job goes wrong and triggers alarms 2-5am it very bad for the the health and sleep rithm of whoever needs to fix it and the most likely time they make mistakes
it sadly is also the time where non-international interruptions are least likely to matter
if you are an international company having international teams can be an solution
and if you are a very small team adopting job times to an overlap of low usage and your admins sleep habits can be an option
but if you need to pick a generic time and don't provide for logistics or other "start work at 4/5am" companies probably 5-6am is a better time to run your batch jobs
anyway just some random thoughts
It depends on tz. 4 am in some countries too
DST is such a dumb mechanism anyway
just use UTC
If you want to avoid overlapping runs simply simply have the task check for the existence of a lockfile, if not create a lockfile containing the start timestamp, run the task and delete the lockfile at exit or error.
The timestamp gives you a chance to recognize stale/orphraned lock files in case of crashes based on their age.
I have a job that deliberately runs at 2am and 3am... to update the time in a bunch of really old PLCs for DST. And check that every other device on my telemetry network has correctly updated its time.
bruh, just use UTC
Ugh gross. Same non monotonic time problems just not every year.
TAI4LYFE!
Here's my radical time idea: instead of keeping the same schedule year round, we just incrementally chop parts of the day off (we just put that time in a "bucket" and pretend it doesn't exist). Then at the summer equinox we empty the time bucket into a 6 day bender of a party. /s
This isn't a problem with actual cron and crontab. It is a problem with the systemd-timers shim "crontab" which doesn't work the same in many corner cases and often has weird bugs.
This post is literally about an issue observed in vixie-cron (as included in some distro c. 2013), not about any systemd implementation.
Exactly! The article makes the point that:
> until one of them achieves cron’s level of ubiquity, we have to live with cron at least some places and sometimes
systemd could arguably be described as close to (maybe behind, maybe ahead of since it's the default for the most popular Linux distros) cron's level of ubiquity, and doesn't have this bug as far as we know.
Which cron is actual cron? There are tons of implementations out there.
This post describes vixie-cron, not systemd-timers.
What do you consider "actual cron"? Because the post specifically says Vixie cron, which has been the basis for most versions of cron on Linux systems.
How does actual cron handle this situation?
"Not a problem" as in these only run these cron job once per day, irrespective of DST changes making a given "hour happen twice"?