Perspective from the trenches: I teach at a university that uses Canvas. We are in our final exams period right now.
We got our first email (from Academic Affairs) notifying us that it was down at 5:17pm EDT this afternoon, with little info; followup emails were sent at 6:24 and 6:57 with more info, but mostly about how we would be compensating for it and not about what actually was going on (other than, "nationwide shutdown" and "cybersecurity attacks", no further detail). I don't get a sense that they know much more than that, not that I would expect them to.
A perhaps telling detail: they're instructing us to have students email us directly with any work that had been submitted via Canvas. That suggests that they have no particular confidence that it will come back up soon.
I personally am only slightly affected; as a CS professor a lot of my students' work is done on department machines, and submitted that way, and I do the actual exams on paper. More importantly, I've never liked or trusted Canvas's gradebook, and so although I do upload grades to Canvas so students can see them, my primary gradebook is always a spreadsheet I maintain locally.
But I have a lot of colleagues for whom this is catastrophic at a level of "the whole building burnt down with all my exams and gradebooks in it"---even many of those that teach 100% in person have shifted much or all of their assessment into Canvas (using the Canvas "quiz" feature for everything up to and including final exams), and use the Canvas gradebook as their source-of-truth record. We've been encouraged to do so by our administration ("it makes submitting grades easier"). For faculty in that situation, they have few or zero artifacts that the students have produced, the students themselves don't have the artifacts to resubmit via email because they were done in Canvas in the first place, and they have no record of student grades or even attendance (because they managed that all inside Canvas). I guess they have access to the advisory midterm grades from March, if they submitted them (most do, some don't), but that might be it.
My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers), or weeks (they don't). Very little in-between. And if that's true and we wake up tomorrow with this unresolved, I really have no idea what a lot of professors at my university and across the country are going to do to submit grades that are fair and reasonable. In the extreme case, they may have to revert to something we did in the pandemic semester (and before that, at my school, in the semester that two major academic buildings actually did burn to the ground a week before finals): let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
(Well, one thing you can do is not put your eggs all in one basket, and not trust "the cloud" quite so much, but that ship's already sailed. I do wonder if in the longer term, anybody learns any lessons from this....)
UPDATE: As of 11:45pm EDT, my university's canvas instance is up and running! Here's hoping it stays (but I'll be downloading some stuff just in case...)
> the students themselves don't have the artifacts to resubmit via email because they were done in Canvas
It’s so simple to send an e-mail to the student with relevant records on completion of a quiz or whatnot. They don’t do it, because they want to control the data. (And universities don’t insist on it for who knows what reason.)
I've never used Canvas before, but all the LMSes that I've used allow students to enable emails whenever anything is updated, including when grades are posted. This is off by default because it's often 10+ emails a day, because many teachers post notes once a day, and with 5 classes, that adds up pretty quick. I personally have it enabled because it's pretty manageable with some custom Outlook rules, but setting this up is well beyond the capabilities of most students.
I understood your comment perfectly fine. I'm asking which graduates of which colleges you were referring to. It looked like you were generalizing about US HS and colleges. If so, plenty of other countries' HS and college education systems work better.
Going by a certain story 2 years ago, their concern should be that they're overqualified for Meta.
It doesn't help that gmail, which is the only serious direct competition to outlook, straight up doesn't do "folders" and instead goes with markers. So you can't really just put a filter that drags all the 100 low-priority alerts in what would count as a first degree abstraction of "place where things are sorted into". No, there are two layers of abstraction between point A and B of things, sorter and sorted things. The result? Muggles can't recognize the heck you're describing and refuse to even acknowledge the possibility.
> It doesn't help that gmail, which is the only serious direct competition to outlook, straight up doesn't do "folders" and instead goes with markers.
While true, unless I'm mistaken, markers (I assume you're referring to tags) can be nested to provide a pseudo-folder hierarchy, and with proper filters you can remove the "inbox" tag and have the mail only show up under the specific tag.
TBH I don't fully mind it, it lets you classify an email in multiple ways (eg "See Later" as well as "Work related").
Tags are great but I still want my folders. Also doesn't help that the way google describes some things is unnecessarily complex or confusing.
For example, removing an email from the inbox requires archiving it. In most other applications (WhatsApp, Signal, Outlook, etc) archiving usually results in the email being placed in a specific archive folder that isn't readily accessible through the UI. At least not to the same level that normal emails are.
I partially solve this by using Thunderbird on my laptop. When I get emails on my smartphone (on the Gmail app), they unfortunately all go to the inbox. But the moment I open Thunderbird, it nicely organizes them for me.
If a CS graduate can't figure out some simple gmail labels and filters then they should not be awarded that degree. Plain and simple. It's not rocket science.
And there are no other students at any college other than CS students? I'm not sure why a biologist or a literature student would need to be au fait with Google's admittedly fairly unfriendly email management setup.
Digital literacy is important to every field. Email filters are not some arcane computer science concept, they are the modern equivalent of filing physical mail into the right folder/pidgeon hole/inbox/whatever.
Biology is a great example because of just how important digital record management is to experimentation in the field.
I'd hope/assume that any Computer Science students would be able to do this, but most Biology/Education/English/Art students probably couldn't.
I mean, anyone smart enough to attend university could probably figure it out if they really wanted to, but there are hundreds of other useful things that they could learn too. There are only so many hours in the day, and given that most students don't get that many emails, I can hardly blame them for not wanting to prioritize learning how to filter emails.
(I personally have over a hundred lines of Sieve filters, but I'm definitely not a typical student)
Most managers I've met, struggle with setting up email filters, and have to ask tech support to do it for them. These students will be qualified just fine.
>Setting up custom email filters is beyond the capabilities of most students?
Yes. And most of the general population. They can do it once they know it exists, most people just are not aware it is a thing at all.
>What are they learning?
Here, their "major" as you say in the US. Someone in econ, biology or even CS is not going to learn Outlook rules. Maybe IT or business will have a sentence on it.
Exactly what is in their field of study, nothing more. That's a huge part of the problems created by treating academia as a degree mill mandatory to get a job able to feed yourself instead of a place only for those truly interested in actually studying a subject.
Students having records of what their score was doesn't prove to the professor / university what score they received. "FWD: Exam 1 Results" is not especially auditable.
Canvas is built to automatically export its gradebook to an external system. It will do that automatically every day if you want it to. Teachers or others can manually export to the configured foreign system on demand. So if you grade something and want it to show up in the foreign gradebook without waiting for the daily export, you can just press the button to make it happen right away.
i cannot believe how much benefit of the doubt people are giving canvas
ed tech is the WORST performing VC sector
the ONLY game in that town is vendor lock-in! are people joking?
c'mon, canvas is a huge piece of shit. the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first, rather than universities writing an open alternative they share with each other for free.
Canvas is AGPL licensed. Moodle is GPL. Universities or anyone else can already contribute to big name LMS.
Canvas is used by Harvard, MIT, Stanford, Carnegie Mellon, CalTech, etc. If they each paid 10 FTE, they could set up a foundation that could govern the development of a top-tier LMS. Every tier-1 state institution could contribute 5 FTE. Even little JuCos could chip in an employee here and there. You'd pick up hundreds of capable employees at a fraction of what those schools currently pay to Instructure.
On paper your idea seems obvious. You take a bunch of institutions that actually teach students how to program and have them cooperate to build an open LMS that benefits them all.
In reality, universities always spin off anything that looks like it could generate revenue. It is very telling that you can't even get your college transcript from your college. You have to go to (and pay) some third party to get it. Some universities even outsource their "classes" like elderhostel to cruise lines and travel companies.
> rather than universities writing an open alternative they share with each other for free
That already exists [0], and is actually reasonably popular.
> the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first
I doubt it, because enterprise sales has nothing to do with how good your product is, how expensive it is, how easy it is to administer, how secure it is, etc.; it only depends on how good you are at enterprise sales. I mean, my university is Oracle-based, and I'm pretty sure that you could get 3 random undergraduates to write something better, so I don't think that LLMs writing better/cheaper software will make any difference here.
Presumably the system will be back up eventually, so there's not much benefit to lying here, since at best you'll raise your grade in a few classes for a couple months, while taking on a pretty big risk of getting caught.
Exactly this, when was the last time a HN user had to interact with the prototypical 60-year-old set-in-their-ways professor?
Extremely non-tech savvy, hates computers, and is gonna grumble "What the hell is a PGP? Better not be another one of those phone code things." as you try to pitch this highly-technological solution to a largely niche problem domain.
I mean a cloud based learning management system also seems to be a very technological solution to the very old problem of checks notes grading quizzes?
They don’t even need to not be tech savvy. This stuff just registers as “hassle” to most people so they do the bare minimum or search for ways to not deal with it at all. It’s easy to “tut tut” at them but ultimately we need to accept reality: privacy, security, these things take extra effort that isn’t strictly necessary for people to go about their daily lives even though the stakes can be super high. It’s not a problem until it is, so they aren’t really barriers that require people to do the work. It’s like convincing someone who just simply doesn’t want to go out and buy/install a lock on their door to go do it, except it’s not even a one-time thing. Their door works fine. They can come and go as they please. It’s not until something happens that they maybe change their tune (and even then!)
Hell just getting people to do secure passwords is a whole thing.
Makes me glad I've always avoided doing my work on web platforms. When we used to have to make presentations in Google Slides I used to do them in Org-mode, then export to Sheets. I still have all those assignments sitting on my disk. Sure, there's versions of them on Google Drive, but I always make sure that the canonical version is the one on my disk.
Just to add one more data point, we also use Canvas at my university. The deadline for submitting who are eligible (i.e. passed compulsory assignments and labs) to take the exam was yesterday, and I couldn’t meet that deadline because Canvas went down. I usually do corrections offline so I have backups of my own evaluations, but these are courses with many teachers and many TAs, so Canvas is the way we sync our assessments.
I work in the Education sector as IT. We don't know much else either.
Everything we know has come from reddit threads / hackernews threads. There has been 0 official communication today indicating this was an attack, yet the login page was defaced by ShinyHunters.
All these articles listing the American schools affected, "nationwide" outage reported, meanwhile hundreds of millions in the rest of the world affected.
I don't understand what's the panic and doomerism about. Any competent IT team has backups and will be up and running as they go back to a state before the breach. This is HN. I'm disappointed that everyone is talking about losing grades and going back to pen and paper. I don't see how that could happen in 2026.
And from the hacker's message itself, it's clear they want money in exchange for not releasing private info, not for the data itself.
Do we live in a fear based culture? Why the panic? Even if everything was hosted on Instructure's infrastructure, it's all AWS. I'd be VERY surprised if there aren't multiple way to go back to a previous state.
Most of the work and delay is to make sure they figure out where the breach occurred.
I'm sure you're right. Across tens (hundreds?) of thousands of institutions worldwide, each one is exercising its well-written incident runbook that not only gets updated regularly but also is rehearsed constantly, just in case something like this happens. After all, what university IT department DOESN'T prepare obsessively for the moment when they need to restore all grades on all assignments for all courses from backup and fall over to the backup system for final exam administration in any required format specified by any professor, in the second week of May, on a non-negotiable schedule? There's absolutely nothing to worry about here.
Here in the Netherlands a data center's power source (not even the machines) burnt down, data center is offline and University of Utrecht, one of the biggest universities here, is closed. Access passes don't work, work from home environment doesn't work, student information system is down, system for grading doesn't work. No failover for any of them (or maybe it was in the same DC?)
Backups can be sabotaged (turned off or schedules manipulated) or compromised (say, by lateral movement).
> Even if everything was hosted on Instructure's infrastructure, it's all AWS.
AWS Backup isn't foolproof. Get your hands on administrator credentials as an attacker and suddenly the only thing between everything being gone for good and unrecoverable even for AWS is remembering to have put a permanent deletion protection on all resources in AWS Backup.
Most jobs I've had didn't care about a transcript in the slightest. It matters for future education and a small selection of jobs, and even them a few pass/fail courses won't cause any issues. It's not great if important, major-specific coursework is pass/fail, but usually you're not allowed to do that, so when it does come up you'll just have somebody ask what absurd situation (like this canvas thing) caused it.
To my European ears this just sounds like a disaster like this waiting to happen. God bless the annoying privacy OSS advocates and bureaucrats, I guess.
Reminds me of the incident last year when a South Korean government's server room caught fire, which contained the government equivalent of Google Drive, and the only backup was in the same room, and they all burnt down together.
Some data was permanently lost, and then officers told reporters that multi-regional backup was not yet built because it was too hard at such a massive scale... of 858 TB.
Backups are definitely helpful in ransomwares, but before systems can be restored and brought back online, victim organizations still need to assess the scope of the breach, find the initial access vector, identify compromised accounts, and evict the threat actor. That can take time.
I’m not certain, but it appears you’re giving Instructure a pass here, as if this is the first time they were hacked. But, it’s the second, by the same group.
As a parent of kids who are impacted by this, I’m not super concerned about the data being held for ransom, but I sure as fuck am concerned about how much it’s going to cost the district to move to another provider.
Not at all; standard IR procedure is scope -> containment -> eradication -> recovery. There is a fog right now; we don't know all the details. It seems to me that it's just as likely they weren't fully kicked out before or that the initial vulnerability wasn't remediated. You can't recover until the threat actor has been removed.
> let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
Schedule a single exam and that's your grade for that subject? That's how it should work anyway, credits for work during semester (or worse attendance) are not needed to evaluate if someone learned the material, give them an exam and done.
That's just bad outdated practice. It leads to cramming and less remembering than of the demand is for students to do work and show learning and effort throughout the year.
Most courses I've taken have obligatory assignments that are pass/fail, and you have to pass a certain amount during the semester to take the final exam. But the grade is determined entirely of the final exam.
Which to me seems the best way, you still have to learn throughout the year. Especially to avoid cheating this works nice. And as an aside, most people I know that did a year abroad in the US got 1-2 grades higher, as it was quite easy to just farm extra credits.
I'm surprised how few comments there are on this thread. This is probably affecting millions of students at the most stressful time of the year.
Incidentally I've always hated Canvas and probably every other LMS provider, but what is particularly amusing about this current outage is that it is occurring at exactly the time when universities are demanding that all professors put all of their materials on Canvas, without exception, due to ADA compliance regulations. It is explicitly forbidden for professors to, e.g., refer to pdfs posted on a personal website.
Other commentators here seem not to understand that many faculty also do not enjoy being forced to use Canvas.
They have not succeeded in forcing me, yet. But it's sad how many computing faculty apparently can't operate the basic online infrastructure needed to support their courses. Not that universities make it easy for us.
And of course the other serious concern I have with Canvas is that they are likely using all the materials faculty upload to train their AI replacements. Many of my colleagues engage in dark humor about this but I haven't noticed much action.
Former Instructure engineer here. Ive been gone almost 10 years at this point, but some of the best engineers I've ever worked with were at INST.
I'm not sure where your stereotype even comes from, because Canvas is not trivial software. You can see for yourself as it's AGPL and I assume you looked at the code before criticizing it because any good engineer would do that.
If they're at the level you say, they just might install some AI gizmo like the Vercel employee was accused of, but really let it run amok with write permissions.
Can you explain for the billions of the rest of us why this is the "most stressful time of the year" for the group you're referencing? I assume that's American students and/or teachers?
Live streaming of class through Canvas is very popular. Quite a few people just watch from their dorms. So maybe people will have to come back to class, that will be entertaining. The class rooms are almost standing room only (sometimes they are) on the first day of class and then gradually thin out. Sometimes 10 or so people show up out of a class of 100. If Canvas is not back up soon I think it could actually be disruptive for that reason also.
This is awful to hear. The idea that students are just half assedly streaming the lectures is really just ruining things in the long run. This is a bit old manny, but showing up to lectures is good. You go to class, you get face time with professors, you can ask impromptu questions, you rub elbows with classmates, you talk on the walk between classes, you maybe run into a cute girl. Friction like walking to class and finding a nook in that annoying hour gap you have, are the things that make life enjoyable.
Not much overlap between students and HN these days, though? I’m an extremely rare outlier afaik :)
The administration has so far opened with one “Canvas said” and then an hour later one “Canvas is down indefinitely” email noting that they’re aware it’s serious.
(Canvas is a glorified wiki for teaching students, with quizzes and such, for those unaware.)
I'm an undergrad student in computer science and I come here regularly. Many of my friends do the same. Of course, that can't be extrapolated to students globally, but students who love what they do are not extinct!
FWIW, I'm a student, so there are at least a few still here. Feel free to ask me any questions (either via email or via replies to this post) and I'll try to answer them.
Many of my sisters friends do everything entirely via tiktok. They look at what trends are popular and they target that fully on platform. This is for stuff like building niche targeted apps, selling beauty products/clothing brands, restaurants.
If my peers are any indication, a whole lot of TikTok, Reels, Twitter, Discord, and other such mind-numbing platforms.
The types of platforms I would consider 'substantive' (or, at least, more substantive than those platforms) are definitely on the way out.
The few times friends have seen me browsing Hacker News or a certain Mongolian basket weaving form, the first thing they comment on is how confusing the interface is, and how old the site looks.
I truly don't understand the mentality, but if your site doesn't take three seconds to buffer a simple text drop down menu, and have JavaScript elements load in mid-scroll that bump elements around the page making you just barely miss that button you were trying to click, then your site is seen as 'inferior' or 'sketchy'.
Perhaps I've just had a bad sample, but I've experienced a variety of different environments by this point, and by and large, I've seen more people in my generation act in that manner than not.
This is actually reassuring. We don't need all your peers! We just need you and whatever smart cohort you're bonded with.
It's true that HN looks old - it looked old before you were born, probably - but (a) I have no idea how to change it, and (b) the whole of HN is a long bet on plain text. If the smartest young people lose interest in reading, I'm ok with HN dying for that reason. I just don't want it to die for any cheaper reason.
I would like to offer some additional reassurance: I send my friends articles I see on HN that might interest them. A (in my view) very good litmus test is when someone asks where I saw it, because this demonstrates some desire for continual learning. I find that anyone that asks that question seemingly trusts an interface like HN more because of it. My suspicion is that this is probably because at a certain point you see stuff like Agner Fog's work, LWN, or a number of other minimalist websites and realize that a website that is popular despite the lack of overindulgence in UI must be popular because of the content. It doesn't hurt that the best courses in my university experience have had websites that have not changed much since the late 1990s (one did change the lime green text on turquoise background on their page after the recession to a color scheme that didn't cause headaches in students).
I do find that my peers that now read HN used to be judicial about curating a Reddit feed and mostly otherwise limited on other sources. Short-form content is addictive and as nearly as unavoidable as sugar, but many of my brighter peers work on reducing that intake. Long-form YouTube is also something I find to be a marker of someone who is seeking knowledge. Many of my peers do scroll Twitter and TikTok all day, but I find that those who are easiest to chat with are those who have already scrolled HN today and want to discuss a particular article they know I would have seen. I've had conversations that start with "Did you see that TikTok?" and conversations that start with "Did you see that article on HN?" and the latter is always more engaging.
What? What makes Canvas accessible in a way that HTML and PDF files are not? It's true that PDF readers aren't the best for screenreaders, but surely you can just upload a .html copy as well.
Canvas has an easy way of checking if a pdf or other course material is accessible, so many universities are forcing faculty to put all their materials on Canvas. That way if a pdf or powerpoint is not compliant it is immediately flagged. The goal is to reach a "100% accessible" metric.
Note that little of this really helps the students that it is supposed to help, because as you wisely point out, raw HTML is almost by definition extremely accessible. I work in a field that uses Latex and the source code of Latex should also be considered more accessible than the compiled pdf. But for university administrators the only important thing is that the accessibility metric that appears (or used to appear, before today!) on Canvas shows 100% accessible.
That really sucks. I'm visually impaired and many members of my family are/were blind. I think accessibility is really important, but it's so painful to me to feel like people's limited energy is being directed towards performative measures, useless rituals, vanity metrics, etc.
Nobody has infinite energy, and disabled people don't have infinite social capital. It's a shame when energy from that shared pool gets spent on things that don't really impact meeting people's access needs.
And the other thing is that everyone's access needs are different. It can certainly be useful to try to set a baseline or propagate common guidance. But the most important thing, especially in a university setting, is for instructors to be flexible and responsive and for classes (and non-teaching workloads) to be structured in a way (e.g., small enough) that supports that.
I think metrics like "100% accessible" might even be dangerous. It makes it easy for able-bodied people who aren't in direct contact with disabled stakeholders to pat themselves on the back without actually knowing what's going on.
Bleh. Good luck doing right by your disabled students and disabled colleagues, and good luck resisting the bullshit.
I was only a lowly TA so I saw these issues from afar, but I would add that, on a more optimistic note, I don't think I've ever met an instructor who wouldn't do whatever he or she had to do to support someone with special needs. As you suggested, metrics do not tell the whole story and certainly metrics for the sake of metrics are not helpful and may in fact be dangerous.
That said there is certainly a lot more work that needs to be done in this area. Hopefully these regulations over time bring out practical positive change. Time will tell.
Putting aside the "So you hate waffles?" non-sequitur, surely the entire topic of this thread should be a bit of a hint that this misguided policy has not, in fact, "[made sure] courses are fully accessible".
Accessibility regulations, implemented with feedback from faculty and with the support of university resources, are certainly a good thing. But that is not what is happening in my experience.
1. It should be illegal for any company to pay ransomware attacks. Period. No pay out ever.
2. The penalty for being the attacker should be linked to the system they violated. If you do this to a hospital and someone dies you are life in prison / chair. The minimum sentence should be so painful that it deters the attack.
No this will not stop this and companies need to be held accountable for their lack of security investment. Every attack should be investigate if the company met an agreed industry standards best practices and staffing, etc. The penalties for not meeting the requirements should be punitive.
It should be illegal to host insecure services, especially when you're dealing with PII. Breaches keep happening and nobody gives a fuck, because the worst that'll happen is you might lose a handful of customers and buy some "credit monitoring".
Incidents like this should be followed by an audit and charges being laid. Send corp officers to jail for negligent security failures. If you can go to jail for accounting fraud, you should be able to go to jail for cybersecurity-promises-fraud.
They claim to be compliant with a number of security standards [1]. I would love to see a postmortem audit of how much of this they actually implemented.
I don't think that criminal negligence is the most helpful legal tool for incentivizing improved security. It's too hard to prove negligence.
Instead, there should be standard civil penalties for leaking various degrees of PII paid as restitution to the affected individual. Importantly, this must be applied REGARDLESS of "certification" or whether any security practices were "incorrect" or "insufficient". Even if there's a zero-day exploit and you did everything right, you pay. That's the cost of storing people's secrets.
This would make operating services whose whole "thing" is storing a bunch of information about individuals (like Canvas) much more expensive. Good! It's far to cheap to stockpile a ticking time bomb of private info and then walk away paying no damages just because you complied with some out-of-date list of rules or got the stamp of approval from a certification org that's incentivized to give out stamps of approval.
And this strict liability will come with an expectation of insurance. The insurance policies will necessitate audits, which will actually improve security.
It's not a popular opinion but I agree. I live in a country that has a very extensive principle of public records, and often times these leaks disclose much less than you would get by simply calling the authorities and ask. Now, whether that's good or bad is a different story.
This is a solved problem in pretty much every other domain of life - if you are following best practises but something that wasn't reasonably forseeable happens, then you're fine, but if the bad thing happens as a result of negligence then you are in trouble.
Criminal law isn't about making things alright for the victim. That's what insurance is for.
Even if you leave your door unlocked, if someone walks in and steals your stuff, it's a crime. The state has an interest in prosecuting crimes even if the victim didn't do everything they could to prevent it.
The company is not the victim here. Its users are. [I suppose my previous comment was a bit ambigious - i meant something bad happens to someone else not to yourself]
A better version of your analogy would be if your landlord failed to repair your front door in a reasonable period of time and as a result soneone walked in and stole your stuff. Yes the theif is the primary responsible party, but the landlords negligence in maintaining the property probably also exposes them to some liability.
P.s. This is neither here nor there, but restitution is a part of criminal law.
"Best practice" in cybersecurity is largely vendor-driven with little to no independent empirical validation.
That standard is likely to lock people into buying some pretty bad software, but it does little to ensure that they're running reasonably secure systems.
I like to relate it to operating an automobile. You can follow every traffic law and still be liable in an accident, because you owned the vehicle that caused the damage. This is why you have insurance.
The equivalent analogy is charging lock/door/drywall/timber makers and suppliers for lapses if a thief entered the house by picking a lock or drilling/sawing through the wall.
If Boeing claimed a plane was airworthy, but it crashed because basic engineering controls were skipped, we have collectively put our faith in the NTSB to preserve evidence, run an independent technical investigation, etc. There is no such authority for software - most security auditors (SOC2, HITRUST, etc) are just looking at self-reported data.
Just take a look at the recent Epic vs. Health Gorilla lawsuit to see how nonexistent the protection is around exchanging your medical records, one of the most sensitive types of PII.
People who haven’t been hacked just haven’t been looked at. If someone wants to hack you, they will hack you. It’s really unfortunate that people have this level of confidence in their ability.
Shouldn’t we be focusing on making it harder to pay overseas criminals in the first place? /ahem/ crypto platforms facilitating transfers to bad actors /ahem/
One of those eye opening moments for me was learning about how these criminals work on trust. They need to be trusted to not release the data or to unencrypt when paid, and by and large they do.
One way to weaken any group that works on trust would be to make them less trustworthy. That way victims wouldn't be as confident paying the criminals and thereby making the effort by the criminals less attractive.
interestingly, having actually done the law enforcement side of these investigations, 50% of them are local. And I understand that this is not 100% solution, but neither is any form of law enforcement, but that doesn't mean we should fail to attempt it.
Kids from the local uni having a lark, stalkers, vindictive ex employees, local gangs, criminals who understand their victims because they hail from the same community. These are your local hackers. Sift them from the nation states and international crime groups, then deal with the International as a matter of diplomacy. Because we do this so poorly locally, we have little ammunition to when it comes to diplomacy. "reduce attacks by your crime groups and we buy your natural gas, seel you wheat etc"
Want more motivation?- 75% of the local attacks by volume send funds back to terrorist or separatist organizations.
It is not an in-soluble problem. Sentences are a fraction of the answer, effective and receptive reporting processes are more important, then government backing for investigation and enforcement, then policy around home-team activities (ie don't do the bad things yourselves Mr Gov). Deterrence comes after all that.
One tech ransom case I know of was an inside job. It definitely happens.
There are already significant penalties for doing anything like this. The guy involved is in prison for a very long time. I don’t recall the exact number of years but I do remember it was so long that he wasn’t going to see his kids grow up.
I don’t think anyone who puts a little thought into a crime like this doesn’t understand that the penalties are already very huge. You don’t get a slap on the wrist for extorting a company (or person, for that matter)
To the country or an ally of the country they are targeting, duh. it doesn't matter if you believe it, it's been the truth for over a decade. Heck, Sh1nyHunt3rs people were arrested in the UK recently.
Yeah, they identified themselves as ShinyHunters, and the IP they've put on the demonstration page is geocoded to Russia. Notice this is the same group responsible for the Infinite Campus hack last year.
Really, though, if you want someone to blame, Instructure is not a particularly compelling target. Let's review:
1. Iran is intentionally targeting infrastructure due to a war started by the current administration.
2. China is actively seeking corporate secrets to steal and commercialize for themselves, spurred by extreme protectionism and retaliatory tariffs.
3. North Korea is doing anything they can -- including just taking a remote job by proxy -- in order to extract any money.
4. And Russia is working with and aiding all of them, after everything else going on has forced the embargo to break.
5. All of this while completely alienating every single one of the United States' allies.
6. Meanwhile, the American DHS is currently shut down.
7. And this is after Trump cut funding and personnel for CISA severely enough they've had to end the contract with MS-ISAC, meaning all state and local entities can only remain in the organization if they foot the bill for it directly and CISA and other agencies responsible for cybersecurity are more thinly staffed than they have been in decades.
In short, the current administration systematically disassembled all the protections we have built over the last 100 years, and then placed infrastructure -- schools, in this case, but also power companies, water treatment facilities, communications companies, local governments, hospitals, food producers -- directly on the front lines of the modern geopolitical conflict.
That vast ocean that has kept us safe historically is a poor moat in the modern era.
Having an IP in Russia means about zero regarding their location. Literally anyone doing anything like this is going to get a Chinese or a Russian IP for obvious reasons. Mostly decoy and people like you.
Complete internet blockage of nations allowing the attacks. If foreign governments are you can always execute them. We are living in a different world where this is no longer a zero probability occurrence.
When will countries start treating cyberattacks as an act of war? If the North Korean military came to America and robbed fort Knox of $200M in gold there would be retribution. But hack an American company for the same amount and the feds do nothing.
Ok, so we treat it as an act of war. Now what? Attack North Korea? Great, the entire city of Seoul gets shelled within five minutes of your attack and hundreds of thousands of innocent people die.
It's very easy to play with lives that aren't yours.
Never retaliating is a great way to get people to attack you. Of course escalating to all-out war provokes the same in response, but there does need to be a proportionate response, because it needs to be stupid to hurt us, not good business. t’s a significant failure of the US government when half the world freely loots US citizens and businesses.
You would be surprised how many people naively think "Why doesn't my country just open a war on X country and this Y problem will be solved forever" in their head they think war is just a flurry of bombardments and the other side (not theirs) is just destroyed to rubble and their country will have only minimal losses
Exactly. This is the "Declare fentanyl a WMD" of solutions to ransomware. Sounds kinda badass as long as you don't spend too long thinking about it but has no practical relevance to actual enforcement challenges.
It's a familiar example of the perennial "[THING] could be solved overnight if [PERSON_OR_GROUP] would just start taking [THING] seriously" trope.
How do you know which country to blame? It is standard practice for foreign actors (or just hackers in general) to use proxies around the world to misdirect and insert false clues as to their origin. It could be an American teenager proxying through North Korea, and it could be a North Korean proxying through another American teenager's residential connection, there's no way to know.
They already do. This is what asymmetric warfare looks like, your weakest links will break in a time of crisis. Focusing on retribution for the Dunder Mifflin cyberattack is pointless, the adversarial motivation is purely to disrupt and extort.
The best response to a cyberattack on critical systems is to take security seriously. Document the offense, avoid the same mistakes and invest in penetration testing. Of course, nobody is incentivized to do that until they're attacked, so the cycle perpetuates itself.
If someone robs a bank and someone inside dies of a heart attack, thats felony murder. I would be happy if the same applied to ransom attacks or other blackmail/leaking of info. If someone commits suicide because of it, its murder.
felony murder is pretty widely regarded as a leading factor in incredibly unjust prosecutions and sentencing decisions. perhaps not the best concept to build your ideas on top of.
> If you do this to a hospital and someone dies you are life in prison / chair.
If you're going to get the chair you might as well murder some witnesses or destroy some systems to hide the fact you got hacked. "Hack? What hack? Our servers all burned down in an arson attack".
We could also throw the CEOs of companies who don't properly secure their infrastructure and pay their security engineers enough in jail. A little justice on both ends.
Uh, who determines that the infrastructure wasn't properly secured? Who is willing to risk prison because some intern accidentally committed an API key or made a dumb mistake. Conversely, what's the chances that no one actually gets prosecuted regardless of how sloppy their security practices are?
> who determines that the infrastructure wasn't properly secured
An investigative body, the same kind that determines the who, the why, and the how when an airliner crashes or a bridge collapses. Obviously a lot of work needs to be done to get from point A to point B, and it won't happen overnight, but software development is currently a deeply unserious profession and at some point a genuine software engineering practice needs to be developed.
I am, perhaps naively, slightly hopeful that the LLM bullshit plaguing our industry will be the gust of wind needed for the house of cards to collapse and governments to realise that allowing the entire world to be vibe coded is not sustainable.
Pretty famously, aviation incident investigations are almost always not done with prosecutorial intent, and more about truth finding. It leads to people involved being cooperative to prevent future problems instead of ass covering to prevent jail.
In a darker reading; strong aviation safety is mostly motivated by not killing customers. An airline or plane maker who kills more customers than others will rapidly bleed those same customers and lose them to less lethal competitors. If no one cared about dying people I imagine aviation safety wouldn’t be so impressive.
As someone else here said, software, for the most part, is a deeply unserious industry. The stakes are so comparatively low and the consequences less obvious that it’s a lot easier for companies like intuit to maintain their supremacy simply by being entrenched, having strong sales teams, and the hearts & minds of non-technical managers.
In recent times it seems Boeing has been flirting with enshitification and half-assery but critics are not quiet and not falling on deaf ears
Sure, fatal stuff is bad for the bottom line, but that is a vanishing minority of what gets investigated.
You may not be aware, but there are thousands of non fatal incidents reported per year that just don't make the news.
There is a strong culture of self reporting instilled right from basic flight training, even when there is no damage or injuries, and even when the incident would have never been noticed by the authorities. You are almost guaranteed not to face consequences if you are open and honest about an incident. The FAA openly says that they would much rather educate than punish, and they tend to do that with pilots who own their mistakes. As long as there is no intent behind the fuckup, pilots are unlikely to lose their job, let alone their license.
Ideally the chances are high to certain they get prosecuted for sloppy security practices. It's part of the gig of being a CEO, if you imagine you are such a visionary/ideas guy/leader/whatever, risk taker (always a risk taker) then you can gamble spending 20 to life because you weren't actually as good as you thought.
Failure to protect computer system from forseen failure should result passing corporate veil and resulting all stock holders and managers/leadership of funds to be jailed for same period as perpetrator. It is only way to ensure that these things are taken seriously and enough pressure is put on leadership of companies.
> No this will not stop this and companies need to be held accountable for their lack of security investment.
I think in principle, its sound. Im also just baffled hearing anecdotes from friends that are in big corp world and hearing the type of incidents they have, and how they respond to it.. It makes me wonder if there is enough capable talent to go around for the "boring corp" crowd.
Hint: I don't think there is nearly enough talent to go round, but for these companies, its either that they think they have solid experts (and didn't), OR its not a real priority until you get hit.
Or maybe it should be mandatory for all companies to pay ransomware attackers. Think of it as an involuntary bounty program. Now they get to just say 'sorry (for your hurt feelings)' and suffer no consequences.
Apart from the 4% of the total worldwide annual turnover fine that theoretically could be levied under GDPR, but has never been imposed in full.
My kids are in the middle of their finals week. What a mess. Universities know nothing, Canvas claims to be in a "scheduled maintenance", and one Prof claims to "not have any copies of material offline" which seems pretty negligent. Sounds like one section of a popular class will be doing paper exams while other sections had Canvas-based "half points for 2nd attempt"-type exams earlier today. How soon before names & grades appear in data dumps?
This would be like TurboTax "scheduling maintenance" on April 14th in the US.
The "Scheduled Maintenance" is just total B.S. and just honestly makes them look worse. Apparently according to their status pages this is what 99.996% uptime looks like. Pay attention lol.
It has been over 5 hours now and there has not been any communication about this being an attack, despite many of us seeing the ShinyHunters message on the login page.
There is a lot of people who likely are unaware the latest outage is because they were compromised again.
Them marking the incident as 'Under Maintenance' means the status page isn't reporting this as an outage and adding to downtime%.
The incident yesterday was technically from April 28th, with most communications coming out on the 2nd and 3rd, with it being "Resolved" yesterday.
This incident is the second attack, because they failed to secure their infra again. Everything being reported is a bit delayed, which makes it seem like this is a single attack, not technically two instances.
I was going to make a joke that they should have just taken a page from the military and said “Rapid Unscheduled Maintenance”, but I guess that’s actually the phrase for it.
Once again, an example of why corporations should not have free speech. Corporate statements that are transparent lies should be criminally actionable.
A friend who teaches at MIT said they were hit by this. I found it ironic and a little sad that a place like MIT doesn't have an IT staff that can maintain their own on-prem solutions for things like this.
But it turns out that MIT used to have their own homegrown system, and recently switched to Canvas. Bet they're regretting that now.
The build vs. buy decision seems to have swung very hard toward buy in the last decade, and I think that's a shame. Yes, orgs need to focus on their core competency, and sometimes that means outsourcing things that aren't core competencies to third parties. But there are always downsides.
Homegrown systems are expensive to maintain and usually still fail to match up to the commercial options available at this point. LMS's are also just really complicated pieces of software. I worked on my university's own version as an undergrad.
There is no need to reinvent any wheels by making a homegrown LMS. Moodle exists and is completely open source. Lots of large institutions use it. Even in the case that you need to do something really weird with it that isn't solved by one of the many plugins that exist, you're already 90% of the way there with its base platform, and only 10% remaining for DIY software development.
My highschool, for a while, had a website, which was eventually replaces by a large corporate CMS. Was the website as complicated or complex as the CMS? No, you would have needed to know HTML to publish to it. The CMS was no doubt "more user friendly", I suppose.
But … the original site had a soul. It was unique to the school. There was a student directory! All lost, because the CMS meant utter standardization between all the schools using it (their pages were all identical, except for each got like a different picture of the school as the banner at the top) and the CMS did not do directory anything.
Of course, the directory largely didn't matter in the end. (This was when you needed people's landlines! Quite laughable nowadays…) But it was still sad to see it lost, and several of us students worked on it, which provided us with some early real-world experience.
A large number of my college professors published their own sites, too, where they'd put their lecture notes, homework, etc. I loved those far more than I loved "Canvas" or whatever the ugly LMS we used was.
MIT has an incredible IT staff and they do some cool stuff. Every time I interact with any other organizations IT stuff I find it inferior. They just aren’t super big from what I gathered and probably don’t want to do the incredibly boring work of an LMS.
The one they had before Canvas was very very inadequate.
edit: also some of the more popular cs classes have custom websites and don’t really use canvas, but that isn’t the centralized IT department’s doing.
I started my tech career in EDU. I’m not at all surprised.
IT staff who are ambitious and talented don’t last long in education. The pay is very low compared to industry. Where I worked, you could retire with a comfortable pension after a number of service years, so the IT staff outsourced as much as possible so they needed to take zero risks to their nest egg. Blame all the problems on the consultants and do as little as possible.
It’s literally where dreams go to die.
MIT is known for the brilliant professors and students but at the end of the day, running a university is pretty standard stuff. They don’t need a genius rockstar to admin the courseware servers.
I'm a student at Stanford — this is hitting the whole school hard. Unlike a lot of schools on the east coast that are affected (Brown, Harvard, MIT) we are on the quarter system so we're just ending Midterms right now. We're also lucky enough to have our CS department entirely independent from Canvas, but most of my humanities classes are not so lucky. One art history class is having us submit our midterm papers by uploading to a google drive folder—another is pausing weekly quizzes. The main thing this has revealed is just how dependent students and teachers are on Canvas... I hope that this re-prompts discussions about moving off of a platform that was already (from a student perspective) not very good.
Canvas is handling this terrible. No communication, no status updates, etc. Also looks pretty bad their whole platform was compromised and not a single real report for the breach that already had happened. Wonder how long it will take for SLA violations and lawsuits to manifest, especially with most U.S. schooling having finals right now.
Yeah like their page says "Scheduled Maintenance" which is total B.S. Talking to people at my university's IT side of things Canvas has said nothing to any clients.
The "scheduled maintenance" thing is likely just because that's the easiest maintenance page to throw up site wide, or at least it was back when I was on the Canvas deploy rotation back at Instructure ~10 years ago.
That doesn't excuse any of their other messaging though.
So many universities used to run homegrown or on-prem student systems. This is the downside of consolidating in the cloud. If the infrastructure is compromised, it affects everyone, not just isolated or single installations. I wonder how they are feeling about that decision now? I guess they can say "not our fault" so they might be feeling better than if it was a vulnerability in their own system.
If an exploit is found in the software, hackers will often be able to attack hundreds of separate institutional installations in an automated way just as easily. And depending on the exploit, potentially more easily if on-prem admins fail to take all recommended security steps.
I'm actually much more interested if there is any financial liability for Instructure here? It's interesting that it's the universities being ransomed, while the technical failure was Instructure's. We're used to uptime SLA's -- what about security breach SLA's?
> It's interesting that it's the universities being ransomed, while the technical failure was Instructure's.
My guess would be they get likelihood of getting paid when blackmailing 9,000 schools (at least a few would pay up) than blackmailing Canvas/Instructure.
I don't think any SLA/terms would change who gets to feel the pain.
My guess is that they believe by maximizing their attack coverage, the odds are greatest that some of the institutions will pay up. And otherwise, they can still make a bit of money by selling the data.
Yeah, if they had spent the time and money to roll their own that got hacked, they'd be responsible. Now, they can just clap their hands and show them palms up to you like a black jack dealer and walk away from the table with no responsibility. Probably one of the biggest benefits of using a product instead of building your own.
You’d think this is how it works but universities and schools will still end up holding the bag at the end of the day, irrespective of who is responsible.
It's annoying that this is how internal politics usually works. Decision-makers at an org should be considered just as responsible when a third-party choice goes bad as when an internal tool goes bad.
Running on prem or homegrown systems used to be considered a core competency of having a computer science department and a campus-wide IT/networking staff at a university. In the environment that exists today in academia, for instance, BSD would never be created because somebody could just pay a third party external vendor for some packaged product. What happened in the past 20 years to change that? I really wonder.
I remember when I was in high school (2016? 2017?), I found a super simple XSS in the assignment submission form and told the programming teacher. Canvas then proceeded to lock my account and got me my first (only?) detention. Good times.
Somewhat similar vein, the school's blocking software would block YouTube and embeds unless they came from Canvas. They were smart enough to disable the HTML editor for posting discussion comments, but forgot that since it was a rich text editor, you could just copy-paste in an embed by putting the code in data:text/html, then copying the element as formatted html.
I also ran the entire DOMPurify sample XSS and managed to find one way to download custom content onto someone's computer.
Goddammit. Anyone in the know, know if Parchment was also impacted by this potentially? They were acquired by Instructure a few years ago, and deal with a LOT of transcripts.
Edit: https://status.parchment.com/ says "While Canvas, Canvas Beta and Canvas test are currently unavailable, we are simultaneously monitoring all of our other product environments, including Parchment. We continue to see no reason to believe any Parchment resources have been impacted."
I remember circa 2010 a friend of mine at college was like “blackboard sucks, let’s build something new”. At the time I poo pood the idea and lo and behold canvas came out a year later. Outside looking in, they been crushing it.
One of my mentors created Blackboard. It used to be very very good, but he sold it to private equity, and they immediately fired all of the customer support and developers, 3xd prices overnight leading to the 'blackboard sucks' problem. This gave the opening for Canvas to eventually come on to the scene and dominate.
My wife and I each have to use it as we're both following an online master's at the same university... it's definitely gone downhill (compared to the days where I originally used it ~20 yrs ago in college; tracker-riddled, slow); surprisingly, a recent change made it so that you can only attend online lessons in Chrome (haven't had time to see if this is just a user-agent thing).
I worked in a college IT department around that time and the common belief was that all LMSes suck. There are just too many different ways that too many different people want to do things that it's just bound to be hated. Kind of like Jira / Asana for software dev project management.
I used both and could not tell you the major differences. I feel like they are equivalent in the bread and butter features. Most people don't use 99% of the functions they bake into these. Just use it to hold the syllabus, maybe hold the slides, submit assignments, and spreadsheet for grades. All stuff you can do with email + spreadsheet already. Maybe throw in a shared drive for larger files, which every university in the country already pays for.
As someone who has used both as a student and a TA I find blackboard miles better, much easier to find what i'm looking for and my professors seem to have better luck laying out their course on blackboard than canvas.
I actually disagree, based on my time using Blackboard as an admin, student, and teacher. Although my experience is a few years out of date, I found the interface cumbersome and the performance slow.
It depends on what vintage of Blackboard your IT team has installed. We moved from a circa 2011 BB instance to Canvas in 2022, and it was hands down superior. A different university is running the most recent BB and it’s similar to Canvas.
They are definitely crushing it on sales. The actual product is a radioactive dumpster fire that is simultaneously hostile to students, teachers, and parents.
The Canvas instance at the nearby university is now down (May 7, 4 PM Eastern), but was briefly displaying the message in this screenshot (1). The ransom message implies that today's problem is the second wave in an attack on Instructure after ignoring their first breach in recent days.
We received communication that Canvas is down for "Under Maintenance" although it seems ShineyHunters have compromised Canvas again with that message you posted.
Yes, I work for an Australian online school. We’re down “for scheduled maintenance” (I question how “scheduled” it was given this is within school hours on a school day), but we’re not on the list published by ShinyHunters.
our instance went from [insert hacker leet text] to "down for scheduled maintenance" and myself and other faculty are just having the darkest humor about this.
My wife is in grad school at a major university and is dealing with this right now the week of midterms for spring quarter.
I totally understand why a university wouldn’t want to bake their own learning portals but just feels like such a single point of risk to use third party solutions for something like this.
Back in my day… all we had was a school email via on-premise services. I guess we registered for classes in a web portal but that’s about it. The idea of online class was entirely foreign at the time. Ain’t nobody hacking a blue book.
It’s wild to me that people in this comment section are suggesting that schools should improve their security by rolling their own platform, which is bound to be filled with security holes, instead of using a popular, maintained, open source option.
To be fair to the idea, though, while this would make individual instances less secure, it would drastically decrease the leverage for the work bad actors put in.
There is a saying in the software security industry that (I'm paraphrasing from rusty memories) a system is secure if the cost of hacking it is higher than the value it protects.
Each system being completely distinct from another means that the cost of hacking the average student goes up by 9000 (from the article, Canvas is used by 9000 schools).
Still not saying that rolling out your own is the preferred solution, but the idea is not as ludicrous as it would seem, and should definitely be entertained and discussed, at least.
Maybe. I still remember the Drupal community sneering at the New York Times when they unveiled their homegrown online news platform bitd. After 15 years of recursively scraping ad-hoc porn sites off of server hard drives when clients dragged their feet on migrating to latest versions I 'm less certain the assumption that homegrown == less secure is as valid as it sounds.
Universities used to do this sort of stuff themselves. Then it became a business handled by purchasing rather than needs met by the department themselves.
Tech requirements are the same as they always were. One needs to ask whether they need so many frameworks to host some files on the internet and submit some files and perform spreadsheet calculations. We still used one of those First Age 1990s websites for sort of pre lab quizzes this one class when I was going through it, and it might have looked a little "old" but I mean it did the thing and worked for years and will continue to do the thing and work for years.
You're being deliberately obtuse. Canvas has many many features. Wikis and discussion boards and quizzes (with some anticheat) and groups and the list goes on and on. Furthermore, while it was never the flashiest thing, it did it better than many of its predecessors. Yes, an individual class may not use all of these features, and yes canvas has suffered feature creep even over my time as a student and yes canvas is not doing anything technically challenging, but there is enough of it that each school rolling their own everything would be a drastic waste of everybody's time and money.
Because faculty didn’t want to do it anymore. They want it handled by others but also they want oversight and veto power but also they don’t want to be bothered. But it better always work, and if they make a mistake the software is broken because don’t tell them it’s a user error they used to write Fortran.
As a faculty member at a large university…I have a deep respect for the impossible job of university IT departments.
We originally rolled our on LMS decades ago. When we switched to canvas we kept the home brew running for five years past its expiration date because faculty refused to remove their files. Finally each one was manually moved by IT for the recalcitrant old faculty.
It is kind of funny when these LMS tools with 100+ functions are being used for little more than what email, a grades spreadsheet, and maybe a shared drive would do. University might even ask for the final grades in spreadsheet format by the end of the term anyhow, so data goes into the LMS just to come back out again.
I totally understand why a university wouldn’t want to bake their own learning portals
They used to, in the pre-cloud/SaaS era; and they were much simpler and better UX than the slop that they're renting today, because the actual users were not far from the developers.
Counterpoint: I was a PhD student in 2004 and on the universities board* which oversaw the roll-out of the campus management system. It cost > 10m EUR to implement a shitty system with the worst UX and years of stabilizing to make it somewhat work.
The amount of corner cases and performance requirements during rush times (semester start) made it really infeasible for a university to roll their own.
* German universities have this funny system where 51% of such boards are controlled by the professors and the rest is made up of other employees/staff and students. They call it academic participation.
Ongoing. It is not "down" but purposefully offline for "maintenance." Main status does show the LMS (all the course stuff) down, and my instance shows "up" but that's because (I assume) you can reach it and the maintenance page. But that's not useful, if technically not "down."
Canvas LMS is the core service that universities rely on. I assume they're trying to develop a fix and that's why the service is labeled "Under Maintenance". I'm a Berkeley student and can confirm that our instance (bcourses.berkeley.edu) is still down.
The boy is a biochem PhD student at UIUC and reports that all their finals are now cancelled. "Is this good news?" I ask. "Yes. Everything coming up Milhouse."
Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to.
Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other.
It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.)
---
Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this...
---
(Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
It's possible that Instructure's servers got compromised:
dig canvas.ucdavis.edu
[...]
;; ANSWER SECTION:
canvas.ucdavis.edu. 1974 IN CNAME ucdavis-vanity.instructure.com.
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.125
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.103
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.15
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.18
dig canvas.duke.edu
;; ANSWER SECTION:
canvas.duke.edu. 300 IN CNAME duke-vanity.instructure.com.
duke-vanity.instructure.com. 60 IN A 18.173.121.125
duke-vanity.instructure.com. 60 IN A 18.173.121.18
duke-vanity.instructure.com. 60 IN A 18.173.121.103
duke-vanity.instructure.com. 60 IN A 18.173.121.15
I'm shocked universities don't host their own LMS? At least large universities have the IT departments to do this. They host compute clusters, so they can certainly host an LMS.
The same reason hospitals don't have their own Patient Information System but all use Epic. The amount of customization you need and continuous churn due to changing curricula and regulatory requirements makes it hard to keep up without scale.
Hugs going out to the teams at Instructure working to fix this. I've been through a similar Ransomware attack (national news stories, lots of customers dead in the water, etc.), and it's about as bad a situation you can wind up in.
I used canvas for some Harvard extension classes 10 to 5ish years ago. It worked Ok. Work distributed, grades posted. I didn't realized so many schools used it, or that it was all schools on one instance, which seems kind of nuts.
I lost access when I left as it was tied to my work email. I downloaded a lot, but there was still some useful stuff on the boards.
I wonder what the havkers found out about me. Perhaps the class notes will be lifted to train AI, higher quality than a lot thats on the internet anyway.
I discovered one of my old school assignments ended up on some homework help website. I had never posted this document publicly and had only uploaded it to the schools work submission page. Presumably at that point it was shared with multiple third parties for plagiarism checking and such. And then was exposed to a data breach years later and ended up on the public internet.
One thing I remember from my days in the LMS world is that obfuscated copies of prod tenants were used for testing. Almost every dev had at least one tenant from prod on their local computer. So with some de-obfuscation at least some of the data is plausibly retrievable. Whether that data is also public depends on how the negotiations go.
It is how classes (even in person ones) are organized. Assignments, quizzes, links to online textbooks, discussion boards, student/teacher messaging, student group messaging, etc. From the teacher side, I'm not sure if there is a backup copy for things like grades outside of Canvas. It's that pervasive.
Everything from middle school up to grad school.
It's a particularly interesting time to have this happen too -- many finals going on now.
It’s a “learning management system.” It replaces a course website in most instances. It’s also used for course grades and you can submit assignments or take quizzes.
Our whole testing center is down. This is inconvenient, but mainly it's amusing. I swear strangers are talking to each other more. I'm noticing people just sitting in the sun and relaxing. Nature is healing.
(Of course, plenty of people have also just finished their exams, so it's hard to know the cause.)
Any idea what data Instructure-and-also-now-ShinyHunters even purport to have beyond names, profile photos, pronouns, homework assignments, school communications, phone numbers, and email addresses?
i.e. What makes this threat so different from what any old data brokers have already scraped?
What leverage besides aura farming do the ShinyHunters really have?
All I can think of that's really valuable is passwords. And private communications in Canvas DMs. But if you're being at all intimate over your school email, that's kinda on you.
Anyway surely Instructure only stores user public keys or something?
Alternate history question: If they just sold the data, never revealed the hack, and didn't make a scene, from a customer perspective, how different would this be from business as usual?
I use Canvas for some postgraduate studies, and my teenage daughter uses it at her high school.
We already bond over how awful the Canvas UX is (and she has a bunch of Chrome extensions to improve it.) Now we’ve got something else to gripe over together.
I vibecoded a pretty extensive CLI for Canvas and using it is very pleasant. Joyful, even, when combined with an LLM. Especially when compared to the developer hostile Blackboard Ultra.
I wonder when the public is going to start calling for corporate liability for malpractice in software development and corporate liability for malpractice in IT deployments. Even if the tech industry fights it, it probably won't be that much longer.
I'll never understand this point of view. If someone would please explain how to create perfectly secure software, I will gladly start writing perfectly secure software. Only after, if it's clear I ignored obviously correct advice, should there be malpractice penalties.
Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths. Malpractice insurance is high. Litigation is constant. And patients still die on the operating table. It's unclear what all the malpractice tort law actually gets you in the end.
> "Consider surgery instead of software development."
Is that really the analogy you want to use the bolster your argument? Licensing was forced on the medical profession because of rampant quackery causing a large number of deaths. Some of the horrors that went on before enforced medical licensing are well-nigh unbelievable, e.g. https://en.wikipedia.org/wiki/John_R._Brinkley
> Only after, if it's clear I ignored obviously correct advice, should there be malpractice penalties.
In most of these cases, the companies involved did NOT follow standard security practices.
I am pretty sure that is what people mean when they say "held responsible", they mean "held responsible for failing to follow standard security practices", not for the actual act of getting hacked.
> Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths.
I like this analogy, but deaths shouldn't be the leading indicator just an indicator. Family member had a surgery with well known procedures, say removing a gall bladder. Unfortunately, this surgeon skipped a step in lieu of setting a record for fastest procedure. Because steps were skipped, the gall bladder was not scooped into a net to avoid spilled gall stones which resulted stones spilling into the abdominal cavity requiring numerous follow up surgeries to remove the spilled stones as they made themselves known. So clearly not following accepted procedures should be a clear win in a malpractice case, yeah? Wrong. No doctor would testify against the surgeon and the case was dismissed. I feel like this is exactly how it would work in software security incidents as well.
I agree that even if companies do everything right, they can still get popped. But most companies do not do everything right, and they should be legally responsible for those things.
But even if they do everything right, is it really fair to let the companies just shrug their shoulders and say "it happens"? While their users are the ones who really get hurt.
I do wonder if that won't just end up INCREASING ransom-type attacks, though?
If we increase the penalties for a company being hacked, you create even MORE incentive for hackers to try to break in, because if they succeed, they have a pretty big stick to threaten companies with when demanding a random payment - not only will the company have the negative effect of the data being leaked and the PR that accompanies it, they now know that if they don't pay and the attack becomes public knowledge, they face a big fine or other punishment.
A company is much more likely to pay a big ransom if they know they are just going to end up paying that much or more in fines if they refuse the ransom and report the hack instead.
If you take this route, and increase punishment for being hacked, you are making a pretty big bet that the main reason companies are hacked is because of poor security practices. I am not sure if that is true or not.
Tbh this is extremely annoying for high school/college students too. High schools are in the middle of AP tests, and many universities have yet to finalize grades, so overall this is a terrible time for this to happen. After the first issue a few weeks ago Canvas should have upped their security and prepared for another attack. They also should provide better communication. If Canvas is down for more than a few days, many schools and universities will have a lot of trouble when it comes time to publish course grades.
I bet it depends on the institution and the IT team behind said institution, but at least for my university we apparently don't delete old course shells or anything.
I'm friends with a professor who complained to me a couple times about how sometimes he will need to scroll through pages and pages of courses he taught in the past. He also mentioned that profs aren't able to delete their own course shells either.
It wouldn't surprise me if most of it is still around. The amounts of data are probably fairly small, and thus unless intentionally deleted, it's probably still there (maybe unis in Europe are more likely to bother to click the relevant buttons as to comply with the GDPR?). I can't imagine storage becoming an issue unless you've got a huge uni or classes that deal with video (and even then, those probably end up on Youtube as private videos, or only as really small clips).
Grades, records, etc I would assume. Someone else pointed out that they recently acquired https://www.parchment.com/ so they may have also been able to scoop up those records too
Great. More data gone astray. Given Canvas' handling of the situation, I doubt they're going to learn much.
The timing probably isn't a coincidence. Great time to stress out students and staff alike. Hopefully it doesn't affect them too much in the end, but I imagine it will.
My wife’s grades are due tomorrow. She was in the middle of finishing exams when it happened. She can’t even access the exams to grade by hand. Total mess.
i work tech at a university that's impacted by this. while it doesn't impact me directly, many many other staff and instructors i know are heavily affected by this outage. the students are absolutely outraged, mostly because the university hasn't been providing updates as quickly as they'd like, but since the staff/admin are waiting on word from instructure -- and there hasn't been a lot from them, it just generally sucks for all of us.
this is really, really, REALLY bad. it's not great that names/emails/etc will potentially be leaked, but also private messages between students and instructors. and since many of the campus systems rely on canvas integration, things have pretty much ground to a halt a week before finals.
after they were breached on the 1st of this month, instructure had an announcement yesterday that "everything is great! we're good! hackers are gone! we've rotated our keys!".
It is absolute chaos at my institution. This is the last day of finals and grades are due Monday morning. Most faculty are spending today, tomorrow, and through the weekend finalizing grades.
What we don't have access to includes:
* Already graded work
* Ungraded work
* overall adn assignment grades
* lists of students and student emails from the course
* messages from students that are often sent through gradescope
Nothing to see here folks. Just another predicable data breach from allowing companies to do whatever the hell they want with sensitive personal information.
This will keep happening, more and more, and never stop, until we create a software building code and legally require it for all online businesses.
Universities, Parents: ya'll actually have the political and economic power to get a software building code passed. This incident isn't the last.
Terrible that this affects children and that their information may be ultimately leaked. They need to be greater consequences in the law for security breaches.
Do you remember how Canvas was a gigantic improvement over Blackboard?
And GitHub doesn't provide a way to record grades that remain private per student last I checked, much less sync them to the university, or 99% of other things Canvas does.
I don't love Canvas, but it's far, far preferable to a world without it.
I've written a bunch of LMS integrations so I've had the opportunity to use all of the major LMSs. Basically, all LMS systems are rather user unfriendly and complicated with a ton of customization options hidden under layers of sub-menus/configuration settings. At their core they provide a grade book, student management tools, and some basic CMS type functionality for posting class messages/coursework/etc. They've all adopted a standard for interacting with external tools (LTI).
Canvas generally is the 'easiest' to use, and the 'cleanest' looking one although D2L Brightspace is pretty good too. Moodle out of the box is pretty confusing and ugly, but I've seen some heavily customized instances that look a lot better. Blackboard is the worst of the bunch IMO.
I saw this happen to my Canvas account today. At first I thought it was a prank from the school or Instructure. The message was sent to students which makes no sense. Second, the message that was sent basically implies that ShinyHunter is actively getting patched out, and no one is ever going to give into their demands. They're basically saying that they're done and desperate. It's a strange message for ShinyHunter to send, but I think they were trying to pull off a psyop / FUD.
Looking into the payload they sent me this is how they hijacked the screen. Everything in the payload is unchanged except for one line of code:
body::after {
content:
"\A\A"
"S H I N Y H U N T E R S"
"\A"
"rooting your systems since '19 ;)"
"\A\A\A"
"ShinyHunters has breached Instructure (again)."
"\A"
"Instead of contacting us to resolve it they"
"\A"
"ignored us and did some \201Csecurity patches\201D."
"\A\A"
"\26A0 W A R N I N G"
"\A\A"
"If any of the schools in the affected list are"
"\A"
"interested in preventing the release of their"
"\A"
"data, please consult with a cyber advisory firm"
"\A"
"and contact us privately at TOX to negotiate a"
"\A"
"settlement. You have till the end of the day by"
"\A"
"12 May 2026 before everything is leaked."
"\A\A"
"Instructure still has until EOD 12 May 2026"
"\A"
"to contact us."
"\A\A"
" \25BC DOWNLOAD AFFECTED_SCHOOLS.TXT \25BC"
"\A"
"91.215.85.103/pay_or_leak/"
"\A"
"instructure_affected_schools_list.txt"
"\A\A"
"visit us: shnyhntww34phqoa6dcgnvps2yu7dlwzmy5"
"\A"
"lkvejwjdo6z7bmgshzayd.onion" !important;
At the same time,
Aussie tech giant pauses work, devotes entire week to AI
Design software giant Canva has halted normal operations across its 5300-strong global workforce for five days of nothing but AI learning and hackathons, bucking the global wave of technology giants that have slashed jobs, citing the technology.
https://www.smh.com.au/technology/aussie-tech-giant-pauses-w...
Perspective from the trenches: I teach at a university that uses Canvas. We are in our final exams period right now.
We got our first email (from Academic Affairs) notifying us that it was down at 5:17pm EDT this afternoon, with little info; followup emails were sent at 6:24 and 6:57 with more info, but mostly about how we would be compensating for it and not about what actually was going on (other than, "nationwide shutdown" and "cybersecurity attacks", no further detail). I don't get a sense that they know much more than that, not that I would expect them to.
A perhaps telling detail: they're instructing us to have students email us directly with any work that had been submitted via Canvas. That suggests that they have no particular confidence that it will come back up soon.
I personally am only slightly affected; as a CS professor a lot of my students' work is done on department machines, and submitted that way, and I do the actual exams on paper. More importantly, I've never liked or trusted Canvas's gradebook, and so although I do upload grades to Canvas so students can see them, my primary gradebook is always a spreadsheet I maintain locally.
But I have a lot of colleagues for whom this is catastrophic at a level of "the whole building burnt down with all my exams and gradebooks in it"---even many of those that teach 100% in person have shifted much or all of their assessment into Canvas (using the Canvas "quiz" feature for everything up to and including final exams), and use the Canvas gradebook as their source-of-truth record. We've been encouraged to do so by our administration ("it makes submitting grades easier"). For faculty in that situation, they have few or zero artifacts that the students have produced, the students themselves don't have the artifacts to resubmit via email because they were done in Canvas in the first place, and they have no record of student grades or even attendance (because they managed that all inside Canvas). I guess they have access to the advisory midterm grades from March, if they submitted them (most do, some don't), but that might be it.
My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers), or weeks (they don't). Very little in-between. And if that's true and we wake up tomorrow with this unresolved, I really have no idea what a lot of professors at my university and across the country are going to do to submit grades that are fair and reasonable. In the extreme case, they may have to revert to something we did in the pandemic semester (and before that, at my school, in the semester that two major academic buildings actually did burn to the ground a week before finals): let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
(Well, one thing you can do is not put your eggs all in one basket, and not trust "the cloud" quite so much, but that ship's already sailed. I do wonder if in the longer term, anybody learns any lessons from this....)
UPDATE: As of 11:45pm EDT, my university's canvas instance is up and running! Here's hoping it stays (but I'll be downloading some stuff just in case...)
> the students themselves don't have the artifacts to resubmit via email because they were done in Canvas
It’s so simple to send an e-mail to the student with relevant records on completion of a quiz or whatnot. They don’t do it, because they want to control the data. (And universities don’t insist on it for who knows what reason.)
I've never used Canvas before, but all the LMSes that I've used allow students to enable emails whenever anything is updated, including when grades are posted. This is off by default because it's often 10+ emails a day, because many teachers post notes once a day, and with 5 classes, that adds up pretty quick. I personally have it enabled because it's pretty manageable with some custom Outlook rules, but setting this up is well beyond the capabilities of most students.
Most graduates aren't really qualified to work anywhere that they couldn't have worked before going to college in the first place.
You mean graduates of US colleges? Not colleges in general. Or non-technical graduates of US colleges?
I think they point weird-eye-issue wants to make is: Students attend college to become qualified to work.
I think you completely misread my comment.
I understood your comment perfectly fine. I'm asking which graduates of which colleges you were referring to. It looked like you were generalizing about US HS and colleges. If so, plenty of other countries' HS and college education systems work better.
> Where will they be qualified to work?
Going by a certain story 2 years ago, their concern should be that they're overqualified for Meta.
It doesn't help that gmail, which is the only serious direct competition to outlook, straight up doesn't do "folders" and instead goes with markers. So you can't really just put a filter that drags all the 100 low-priority alerts in what would count as a first degree abstraction of "place where things are sorted into". No, there are two layers of abstraction between point A and B of things, sorter and sorted things. The result? Muggles can't recognize the heck you're describing and refuse to even acknowledge the possibility.
> It doesn't help that gmail, which is the only serious direct competition to outlook, straight up doesn't do "folders" and instead goes with markers.
While true, unless I'm mistaken, markers (I assume you're referring to tags) can be nested to provide a pseudo-folder hierarchy, and with proper filters you can remove the "inbox" tag and have the mail only show up under the specific tag.
TBH I don't fully mind it, it lets you classify an email in multiple ways (eg "See Later" as well as "Work related").
Tags are great but I still want my folders. Also doesn't help that the way google describes some things is unnecessarily complex or confusing. For example, removing an email from the inbox requires archiving it. In most other applications (WhatsApp, Signal, Outlook, etc) archiving usually results in the email being placed in a specific archive folder that isn't readily accessible through the UI. At least not to the same level that normal emails are.
I partially solve this by using Thunderbird on my laptop. When I get emails on my smartphone (on the Gmail app), they unfortunately all go to the inbox. But the moment I open Thunderbird, it nicely organizes them for me.
I use Thunderbird on both the desktop and Android. Love it.
Perhaps Outlook is difficult to configure. Thunderbird is intuitive.
If a CS graduate can't figure out some simple gmail labels and filters then they should not be awarded that degree. Plain and simple. It's not rocket science.
And there are no other students at any college other than CS students? I'm not sure why a biologist or a literature student would need to be au fait with Google's admittedly fairly unfriendly email management setup.
Digital literacy is important to every field. Email filters are not some arcane computer science concept, they are the modern equivalent of filing physical mail into the right folder/pidgeon hole/inbox/whatever.
Biology is a great example because of just how important digital record management is to experimentation in the field.
I have been using email for as long as email was a thing and I still managed to blackhole important emails with filters not too long ago.
In my experience, it’s hard enough to make students check their school email in the first place. Let alone filter it.
I'd hope/assume that any Computer Science students would be able to do this, but most Biology/Education/English/Art students probably couldn't.
I mean, anyone smart enough to attend university could probably figure it out if they really wanted to, but there are hundreds of other useful things that they could learn too. There are only so many hours in the day, and given that most students don't get that many emails, I can hardly blame them for not wanting to prioritize learning how to filter emails.
(I personally have over a hundred lines of Sieve filters, but I'm definitely not a typical student)
Most managers I've met, struggle with setting up email filters, and have to ask tech support to do it for them. These students will be qualified just fine.
it's MS software, i think it's inanely difficult
Most people who have office jobs don't know how to do this either
>Setting up custom email filters is beyond the capabilities of most students?
Yes. And most of the general population. They can do it once they know it exists, most people just are not aware it is a thing at all.
>What are they learning?
Here, their "major" as you say in the US. Someone in econ, biology or even CS is not going to learn Outlook rules. Maybe IT or business will have a sentence on it.
>Where will they be qualified to work?
Any office job. Any job really.
> What are they learning?
Exactly what is in their field of study, nothing more. That's a huge part of the problems created by treating academia as a degree mill mandatory to get a job able to feed yourself instead of a place only for those truly interested in actually studying a subject.
Students having records of what their score was doesn't prove to the professor / university what score they received. "FWD: Exam 1 Results" is not especially auditable.
If only we had some way of signing messages
> Students having records of what their score was doesn't prove to the professor / university what score they received
It's better than nothing. (And good training for the real world.)
Also, most universities (and many schools now) issue academic e-mail addresses to students. In those cases, the email is definitive proof.
DKIM signature could be used to verify that Canvas' server sent the email with the given content
And who exactly do you think is going to verify 100s of thousands of emails this way dude?
A computer?
As opposed to a screenshot of a website? Presumably the professor has a spreadsheet of all assignment grades that is submitted to the school?
> Presumably the professor has a spreadsheet of all assignment grades that is submitted to the school?
This would undermine Canvas's lock-in.
Canvas is built to automatically export its gradebook to an external system. It will do that automatically every day if you want it to. Teachers or others can manually export to the configured foreign system on demand. So if you grade something and want it to show up in the foreign gradebook without waiting for the daily export, you can just press the button to make it happen right away.
i cannot believe how much benefit of the doubt people are giving canvas
ed tech is the WORST performing VC sector
the ONLY game in that town is vendor lock-in! are people joking?
c'mon, canvas is a huge piece of shit. the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first, rather than universities writing an open alternative they share with each other for free.
Canvas is AGPL licensed. Moodle is GPL. Universities or anyone else can already contribute to big name LMS.
Canvas is used by Harvard, MIT, Stanford, Carnegie Mellon, CalTech, etc. If they each paid 10 FTE, they could set up a foundation that could govern the development of a top-tier LMS. Every tier-1 state institution could contribute 5 FTE. Even little JuCos could chip in an employee here and there. You'd pick up hundreds of capable employees at a fraction of what those schools currently pay to Instructure.
How well has this worked for Open edX?
Why do they all pay for it then? Seems pretty universal in the UK too. Is it having the benefit of someone to blame when things go wrong?
On paper your idea seems obvious. You take a bunch of institutions that actually teach students how to program and have them cooperate to build an open LMS that benefits them all.
In reality, universities always spin off anything that looks like it could generate revenue. It is very telling that you can't even get your college transcript from your college. You have to go to (and pay) some third party to get it. Some universities even outsource their "classes" like elderhostel to cruise lines and travel companies.
> rather than universities writing an open alternative they share with each other for free
That already exists [0], and is actually reasonably popular.
> the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first
I doubt it, because enterprise sales has nothing to do with how good your product is, how expensive it is, how easy it is to administer, how secure it is, etc.; it only depends on how good you are at enterprise sales. I mean, my university is Oracle-based, and I'm pretty sure that you could get 3 random undergraduates to write something better, so I don't think that LLMs writing better/cheaper software will make any difference here.
[0]: https://moodle.org/
Nope! We're encouraged to keep all that exclusively in canvas. (As noted, I have my own spreadsheet. But I'm an outlier.)
Presumably the system will be back up eventually, so there's not much benefit to lying here, since at best you'll raise your grade in a few classes for a couple months, while taking on a pretty big risk of getting caught.
You forget things can be signed, with the key owned by the school. It can be done.
Does signing really make this easily auditable from the professor’s perspective?
Exactly this, when was the last time a HN user had to interact with the prototypical 60-year-old set-in-their-ways professor?
Extremely non-tech savvy, hates computers, and is gonna grumble "What the hell is a PGP? Better not be another one of those phone code things." as you try to pitch this highly-technological solution to a largely niche problem domain.
I mean a cloud based learning management system also seems to be a very technological solution to the very old problem of checks notes grading quizzes?
They don’t even need to not be tech savvy. This stuff just registers as “hassle” to most people so they do the bare minimum or search for ways to not deal with it at all. It’s easy to “tut tut” at them but ultimately we need to accept reality: privacy, security, these things take extra effort that isn’t strictly necessary for people to go about their daily lives even though the stakes can be super high. It’s not a problem until it is, so they aren’t really barriers that require people to do the work. It’s like convincing someone who just simply doesn’t want to go out and buy/install a lock on their door to go do it, except it’s not even a one-time thing. Their door works fine. They can come and go as they please. It’s not until something happens that they maybe change their tune (and even then!)
Hell just getting people to do secure passwords is a whole thing.
Makes me glad I've always avoided doing my work on web platforms. When we used to have to make presentations in Google Slides I used to do them in Org-mode, then export to Sheets. I still have all those assignments sitting on my disk. Sure, there's versions of them on Google Drive, but I always make sure that the canonical version is the one on my disk.
>It’s so simple to send an e-mail to the student ...
What seems easy on hobby projects gets way more difficult at scale. Source: experience.
For what they charge for these LMSs, they should definitely be able to sent some emails.
Just to add one more data point, we also use Canvas at my university. The deadline for submitting who are eligible (i.e. passed compulsory assignments and labs) to take the exam was yesterday, and I couldn’t meet that deadline because Canvas went down. I usually do corrections offline so I have backups of my own evaluations, but these are courses with many teachers and many TAs, so Canvas is the way we sync our assessments.
I guess what surprises me the most is that it’s even legal for schools to outsource the core of what they do to some random tech company.
Either way, they were under no obligation to adopt this garbage technology regardless of whether it’s available, so this is 110% on them.
The alternative would be that each school develop their own platform for this, which also isn't very good use of their time and money?
They do not need to develop it, but host an existing software on their infrastructure maybe...
The alternative is FOSS.
Seems like instructure canvas is FOSS: https://github.com/instructure/canvas-lms/tree/master
I work in the Education sector as IT. We don't know much else either.
Everything we know has come from reddit threads / hackernews threads. There has been 0 official communication today indicating this was an attack, yet the login page was defaced by ShinyHunters.
All these articles listing the American schools affected, "nationwide" outage reported, meanwhile hundreds of millions in the rest of the world affected.
Does anyone have a list of affected schools?
I don't have a list, but I can tell you the University of Iceland is affected.
I don't understand what's the panic and doomerism about. Any competent IT team has backups and will be up and running as they go back to a state before the breach. This is HN. I'm disappointed that everyone is talking about losing grades and going back to pen and paper. I don't see how that could happen in 2026.
And from the hacker's message itself, it's clear they want money in exchange for not releasing private info, not for the data itself.
Do we live in a fear based culture? Why the panic? Even if everything was hosted on Instructure's infrastructure, it's all AWS. I'd be VERY surprised if there aren't multiple way to go back to a previous state.
Most of the work and delay is to make sure they figure out where the breach occurred.
I'm sure you're right. Across tens (hundreds?) of thousands of institutions worldwide, each one is exercising its well-written incident runbook that not only gets updated regularly but also is rehearsed constantly, just in case something like this happens. After all, what university IT department DOESN'T prepare obsessively for the moment when they need to restore all grades on all assignments for all courses from backup and fall over to the backup system for final exam administration in any required format specified by any professor, in the second week of May, on a non-negotiable schedule? There's absolutely nothing to worry about here.
Schools don't have competent IT teams.
Here in the Netherlands a data center's power source (not even the machines) burnt down, data center is offline and University of Utrecht, one of the biggest universities here, is closed. Access passes don't work, work from home environment doesn't work, student information system is down, system for grading doesn't work. No failover for any of them (or maybe it was in the same DC?)
https://nos.nl/artikel/2613485-storingen-in-hele-land-door-b...
Sometimes it is very hard to recover from the offlining of essential systems: https://www.bbc.co.uk/news/articles/cy9pdld4y81o (Jaguar Land Rover, estimated cost in the billions)
> Any competent IT team has backups
Backups can be sabotaged (turned off or schedules manipulated) or compromised (say, by lateral movement).
> Even if everything was hosted on Instructure's infrastructure, it's all AWS.
AWS Backup isn't foolproof. Get your hands on administrator credentials as an attacker and suddenly the only thing between everything being gone for good and unrecoverable even for AWS is remembering to have put a permanent deletion protection on all resources in AWS Backup.
Maybe a hybrid approach. Scramble to create a final exam/project and give them the option to do pass/fail or a real grade, their choice.
And then wish for the death of saas and a day where you can deploy your own software you can control and modify as you need.
What is the strategic response then? Assuming I'm a student and my grades are gone, and I want to graduate, shouldn't I pick pass/fail?
Does a future employer look at pass/fail vs the grade? do they care? Are there even jobs that matter enough to care out there for them?
This seems like, solving the problem but without actually seeing the broader goal or trajectory education is supposed to follow.
Most jobs I've had didn't care about a transcript in the slightest. It matters for future education and a small selection of jobs, and even them a few pass/fail courses won't cause any issues. It's not great if important, major-specific coursework is pass/fail, but usually you're not allowed to do that, so when it does come up you'll just have somebody ask what absurd situation (like this canvas thing) caused it.
> day where you can deploy your own software you can control and modify as you need.
Canvas is mostly FOSS
https://github.com/instructure/canvas-lms
To my European ears this just sounds like a disaster like this waiting to happen. God bless the annoying privacy OSS advocates and bureaucrats, I guess.
> they have airgapped backups and can be working as soon as they can spin up new servers
... and assuming they have a documented, tested, and trusted restore process.
Reminds me of the incident last year when a South Korean government's server room caught fire, which contained the government equivalent of Google Drive, and the only backup was in the same room, and they all burnt down together.
Some data was permanently lost, and then officers told reporters that multi-regional backup was not yet built because it was too hard at such a massive scale... of 858 TB.
> it was too hard at such a massive scale... of 858 TB
There are probably many S3 buckets in existence that are bigger than that.
Not saying that they should've used S3, but it's definitely possible configure multi-regional backup (and a government can afford it).
My home theater setup has more storage than that.
Ah yes the “recovery” part of the continuity plan. We tested that right? Right?
Backups are definitely helpful in ransomwares, but before systems can be restored and brought back online, victim organizations still need to assess the scope of the breach, find the initial access vector, identify compromised accounts, and evict the threat actor. That can take time.
I’m not certain, but it appears you’re giving Instructure a pass here, as if this is the first time they were hacked. But, it’s the second, by the same group.
As a parent of kids who are impacted by this, I’m not super concerned about the data being held for ransom, but I sure as fuck am concerned about how much it’s going to cost the district to move to another provider.
> I sure as fuck am concerned about how much it’s going to cost the district to move to another provider
Does Canvas have cybersecurity insurance?
Not at all; standard IR procedure is scope -> containment -> eradication -> recovery. There is a fog right now; we don't know all the details. It seems to me that it's just as likely they weren't fully kicked out before or that the initial vulnerability wasn't remediated. You can't recover until the threat actor has been removed.
> let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
Schedule a single exam and that's your grade for that subject? That's how it should work anyway, credits for work during semester (or worse attendance) are not needed to evaluate if someone learned the material, give them an exam and done.
That's just bad outdated practice. It leads to cramming and less remembering than of the demand is for students to do work and show learning and effort throughout the year.
Most courses I've taken have obligatory assignments that are pass/fail, and you have to pass a certain amount during the semester to take the final exam. But the grade is determined entirely of the final exam.
Which to me seems the best way, you still have to learn throughout the year. Especially to avoid cheating this works nice. And as an aside, most people I know that did a year abroad in the US got 1-2 grades higher, as it was quite easy to just farm extra credits.
It has been my observation that most of the better students were the ones who would not put in work during the semester/year and cram at the end.
That's maybe something a school can do if exams are next week, or after.
At my school, tomorrow is the last day of exams. Most of the students have left campus. There's no time or mechanism to schedule an(other) exam.
Then you're testing how good someone is at exams as much as anything
Exams have performance variance. Otherwise you're only getting a pass/fall signal in any case.
Exams are the only fair way to evaluate if someone knows something (written or oral, in person). Take homes and attendance are just window dressing.
I'm surprised how few comments there are on this thread. This is probably affecting millions of students at the most stressful time of the year.
Incidentally I've always hated Canvas and probably every other LMS provider, but what is particularly amusing about this current outage is that it is occurring at exactly the time when universities are demanding that all professors put all of their materials on Canvas, without exception, due to ADA compliance regulations. It is explicitly forbidden for professors to, e.g., refer to pdfs posted on a personal website.
Other commentators here seem not to understand that many faculty also do not enjoy being forced to use Canvas.
They have not succeeded in forcing me, yet. But it's sad how many computing faculty apparently can't operate the basic online infrastructure needed to support their courses. Not that universities make it easy for us.
And of course the other serious concern I have with Canvas is that they are likely using all the materials faculty upload to train their AI replacements. Many of my colleagues engage in dark humor about this but I haven't noticed much action.
> they are likely using all the materials faculty upload to train their AI replacements
Instructure (Canvas's developer) partnered with OpenAI last year [1], about a year after KKR and Dragoneer (PE firms) acquired it [2].
[1] https://www.forbes.com/sites/rayravaglia/2025/07/23/instruct...
[2] https://www.pehub.com/kkr-and-dragoneer-complete-4-8bn-take-...
instructure/canvas-lms is open-source -- is there anything preventing universities from hosting it themselves?
Money, skill, liability.
That calculus is about to shift.
I'm sure the engineers at instructure are not capable of building systems that can do that. You give them too much credit.
Former Instructure engineer here. Ive been gone almost 10 years at this point, but some of the best engineers I've ever worked with were at INST.
I'm not sure where your stereotype even comes from, because Canvas is not trivial software. You can see for yourself as it's AGPL and I assume you looked at the code before criticizing it because any good engineer would do that.
If they're at the level you say, they just might install some AI gizmo like the Vercel employee was accused of, but really let it run amok with write permissions.
Can you explain for the billions of the rest of us why this is the "most stressful time of the year" for the group you're referencing? I assume that's American students and/or teachers?
Final exam season, and it's ongoing in Iceland too, so not just American students.
Live streaming of class through Canvas is very popular. Quite a few people just watch from their dorms. So maybe people will have to come back to class, that will be entertaining. The class rooms are almost standing room only (sometimes they are) on the first day of class and then gradually thin out. Sometimes 10 or so people show up out of a class of 100. If Canvas is not back up soon I think it could actually be disruptive for that reason also.
This is awful to hear. The idea that students are just half assedly streaming the lectures is really just ruining things in the long run. This is a bit old manny, but showing up to lectures is good. You go to class, you get face time with professors, you can ask impromptu questions, you rub elbows with classmates, you talk on the walk between classes, you maybe run into a cute girl. Friction like walking to class and finding a nook in that annoying hour gap you have, are the things that make life enjoyable.
Not much overlap between students and HN these days, though? I’m an extremely rare outlier afaik :)
The administration has so far opened with one “Canvas said” and then an hour later one “Canvas is down indefinitely” email noting that they’re aware it’s serious.
(Canvas is a glorified wiki for teaching students, with quizzes and such, for those unaware.)
> Not much overlap between students and HN these days, though?
That's my biggest fear.
I'm an undergrad student in computer science and I come here regularly. Many of my friends do the same. Of course, that can't be extrapolated to students globally, but students who love what they do are not extinct!
FWIW, I'm a student, so there are at least a few still here. Feel free to ask me any questions (either via email or via replies to this post) and I'll try to answer them.
Is there any internal data on where students are going instead?
Not much, but I do ask the youngest founders what their friends read if they don't read HN, and the only consistent answer I hear is Twitter.
(and btw, they do say "twitter")
Many of my sisters friends do everything entirely via tiktok. They look at what trends are popular and they target that fully on platform. This is for stuff like building niche targeted apps, selling beauty products/clothing brands, restaurants.
You honestly don't wanna know
If my peers are any indication, a whole lot of TikTok, Reels, Twitter, Discord, and other such mind-numbing platforms.
The types of platforms I would consider 'substantive' (or, at least, more substantive than those platforms) are definitely on the way out.
The few times friends have seen me browsing Hacker News or a certain Mongolian basket weaving form, the first thing they comment on is how confusing the interface is, and how old the site looks.
I truly don't understand the mentality, but if your site doesn't take three seconds to buffer a simple text drop down menu, and have JavaScript elements load in mid-scroll that bump elements around the page making you just barely miss that button you were trying to click, then your site is seen as 'inferior' or 'sketchy'.
Perhaps I've just had a bad sample, but I've experienced a variety of different environments by this point, and by and large, I've seen more people in my generation act in that manner than not.
This is actually reassuring. We don't need all your peers! We just need you and whatever smart cohort you're bonded with.
It's true that HN looks old - it looked old before you were born, probably - but (a) I have no idea how to change it, and (b) the whole of HN is a long bet on plain text. If the smartest young people lose interest in reading, I'm ok with HN dying for that reason. I just don't want it to die for any cheaper reason.
I would like to offer some additional reassurance: I send my friends articles I see on HN that might interest them. A (in my view) very good litmus test is when someone asks where I saw it, because this demonstrates some desire for continual learning. I find that anyone that asks that question seemingly trusts an interface like HN more because of it. My suspicion is that this is probably because at a certain point you see stuff like Agner Fog's work, LWN, or a number of other minimalist websites and realize that a website that is popular despite the lack of overindulgence in UI must be popular because of the content. It doesn't hurt that the best courses in my university experience have had websites that have not changed much since the late 1990s (one did change the lime green text on turquoise background on their page after the recession to a color scheme that didn't cause headaches in students).
I do find that my peers that now read HN used to be judicial about curating a Reddit feed and mostly otherwise limited on other sources. Short-form content is addictive and as nearly as unavoidable as sugar, but many of my brighter peers work on reducing that intake. Long-form YouTube is also something I find to be a marker of someone who is seeking knowledge. Many of my peers do scroll Twitter and TikTok all day, but I find that those who are easiest to chat with are those who have already scrolled HN today and want to discuss a particular article they know I would have seen. I've had conversations that start with "Did you see that TikTok?" and conversations that start with "Did you see that article on HN?" and the latter is always more engaging.
Discord is just chat, I wouldn't call it mind-numbing, reminds me perfectly of IRC from a utility perspective.
That said, it's a commercial closed-source single point of failure.
How is Discord mind-numbing?
Perhaps some interest-related Discord servers. Tragically, Discord is just another locked down silo without publicly accessible front on the web.
Drop me an email if you like — it’s not really topical to Canvas but I’m happy to discuss further.
(Comments were split across multiple threads and we've since merged them.)
Definitely not a criticism of your (hard) work here. Thank you!
Thanks! I just added that bit to pre-empt confusion - context-switches like this are one of those rug-pulling moments (https://news.ycombinator.com/item?id=48041875).
We all appreciate the work you do! Thank you!
Replace your material content with lorem ipsum or garbage LLM content and upload it to Canvas to test the accessibility of your documents if required.
What? What makes Canvas accessible in a way that HTML and PDF files are not? It's true that PDF readers aren't the best for screenreaders, but surely you can just upload a .html copy as well.
Canvas has an easy way of checking if a pdf or other course material is accessible, so many universities are forcing faculty to put all their materials on Canvas. That way if a pdf or powerpoint is not compliant it is immediately flagged. The goal is to reach a "100% accessible" metric.
Note that little of this really helps the students that it is supposed to help, because as you wisely point out, raw HTML is almost by definition extremely accessible. I work in a field that uses Latex and the source code of Latex should also be considered more accessible than the compiled pdf. But for university administrators the only important thing is that the accessibility metric that appears (or used to appear, before today!) on Canvas shows 100% accessible.
That really sucks. I'm visually impaired and many members of my family are/were blind. I think accessibility is really important, but it's so painful to me to feel like people's limited energy is being directed towards performative measures, useless rituals, vanity metrics, etc.
Nobody has infinite energy, and disabled people don't have infinite social capital. It's a shame when energy from that shared pool gets spent on things that don't really impact meeting people's access needs.
And the other thing is that everyone's access needs are different. It can certainly be useful to try to set a baseline or propagate common guidance. But the most important thing, especially in a university setting, is for instructors to be flexible and responsive and for classes (and non-teaching workloads) to be structured in a way (e.g., small enough) that supports that.
I think metrics like "100% accessible" might even be dangerous. It makes it easy for able-bodied people who aren't in direct contact with disabled stakeholders to pat themselves on the back without actually knowing what's going on.
Bleh. Good luck doing right by your disabled students and disabled colleagues, and good luck resisting the bullshit.
I was only a lowly TA so I saw these issues from afar, but I would add that, on a more optimistic note, I don't think I've ever met an instructor who wouldn't do whatever he or she had to do to support someone with special needs. As you suggested, metrics do not tell the whole story and certainly metrics for the sake of metrics are not helpful and may in fact be dangerous.
That said there is certainly a lot more work that needs to be done in this area. Hopefully these regulations over time bring out practical positive change. Time will tell.
Are you saying that making sure your courses are fully accessible to your students by following disability regulations is a bad thing?
Putting aside the "So you hate waffles?" non-sequitur, surely the entire topic of this thread should be a bit of a hint that this misguided policy has not, in fact, "[made sure] courses are fully accessible".
Well, to be fair, it has made every course hosted on Canvas equally accessible to everyone. ;)
Not GP, Incompetent policy makers are the bad thing.
Accessibility regulations, implemented with feedback from faculty and with the support of university resources, are certainly a good thing. But that is not what is happening in my experience.
1. It should be illegal for any company to pay ransomware attacks. Period. No pay out ever. 2. The penalty for being the attacker should be linked to the system they violated. If you do this to a hospital and someone dies you are life in prison / chair. The minimum sentence should be so painful that it deters the attack.
No this will not stop this and companies need to be held accountable for their lack of security investment. Every attack should be investigate if the company met an agreed industry standards best practices and staffing, etc. The penalties for not meeting the requirements should be punitive.
> It should be illegal
It should be illegal to host insecure services, especially when you're dealing with PII. Breaches keep happening and nobody gives a fuck, because the worst that'll happen is you might lose a handful of customers and buy some "credit monitoring".
Incidents like this should be followed by an audit and charges being laid. Send corp officers to jail for negligent security failures. If you can go to jail for accounting fraud, you should be able to go to jail for cybersecurity-promises-fraud.
They claim to be compliant with a number of security standards [1]. I would love to see a postmortem audit of how much of this they actually implemented.
[1] https://www.instructure.com/en-au/trust-center/compliance
I don't think that criminal negligence is the most helpful legal tool for incentivizing improved security. It's too hard to prove negligence.
Instead, there should be standard civil penalties for leaking various degrees of PII paid as restitution to the affected individual. Importantly, this must be applied REGARDLESS of "certification" or whether any security practices were "incorrect" or "insufficient". Even if there's a zero-day exploit and you did everything right, you pay. That's the cost of storing people's secrets.
This would make operating services whose whole "thing" is storing a bunch of information about individuals (like Canvas) much more expensive. Good! It's far to cheap to stockpile a ticking time bomb of private info and then walk away paying no damages just because you complied with some out-of-date list of rules or got the stamp of approval from a certification org that's incentivized to give out stamps of approval.
And this strict liability will come with an expectation of insurance. The insurance policies will necessitate audits, which will actually improve security.
I feel like there’s a tendency here to seriously overestimate how damaging these leaks are to individuals.
For most individuals impacted by these hacks, appropriate restitution would be $0. Anything more than that would go beyond making them whole.
It's not a popular opinion but I agree. I live in a country that has a very extensive principle of public records, and often times these leaks disclose much less than you would get by simply calling the authorities and ask. Now, whether that's good or bad is a different story.
The only right answer.
Let's do this.
How could you possibly make it illegal to host insecure services? Is any service 100% secure? And if it were how would we know?
I do agree with the audit and punishments for clear failure to adhere to established standards.
This is a solved problem in pretty much every other domain of life - if you are following best practises but something that wasn't reasonably forseeable happens, then you're fine, but if the bad thing happens as a result of negligence then you are in trouble.
Criminal law isn't about making things alright for the victim. That's what insurance is for.
Even if you leave your door unlocked, if someone walks in and steals your stuff, it's a crime. The state has an interest in prosecuting crimes even if the victim didn't do everything they could to prevent it.
> Criminal law isn't about making things alright for the victim
Restitution and retribution are the components of justice [1] entirely about "making things alright for the victim."
[1] https://www.unodc.org/e4j/en/crime-prevention-criminal-justi...
The company is not the victim here. Its users are. [I suppose my previous comment was a bit ambigious - i meant something bad happens to someone else not to yourself]
A better version of your analogy would be if your landlord failed to repair your front door in a reasonable period of time and as a result soneone walked in and stole your stuff. Yes the theif is the primary responsible party, but the landlords negligence in maintaining the property probably also exposes them to some liability.
P.s. This is neither here nor there, but restitution is a part of criminal law.
"Best practice" in cybersecurity is largely vendor-driven with little to no independent empirical validation.
That standard is likely to lock people into buying some pretty bad software, but it does little to ensure that they're running reasonably secure systems.
I like to relate it to operating an automobile. You can follow every traffic law and still be liable in an accident, because you owned the vehicle that caused the damage. This is why you have insurance.
In civil law maybe, but you aren’t allowed to blame a rape victim for choosing to walk down rape alley…
No building has a 100% chance of not caving in, yet somehow I think charges would be laid if a skyscraper caved in.
The equivalent analogy is charging lock/door/drywall/timber makers and suppliers for lapses if a thief entered the house by picking a lock or drilling/sawing through the wall.
This analogy seems to be portraying 'ransomware hackers' as an unstoppable force of nature akin to gravity.
I'm not sure that's a fair analogy.
The other side of that spectrum portrays the service providers as pure, negligence-free victims. The truth is probably somewhere in the middle.
If Boeing claimed a plane was airworthy, but it crashed because basic engineering controls were skipped, we have collectively put our faith in the NTSB to preserve evidence, run an independent technical investigation, etc. There is no such authority for software - most security auditors (SOC2, HITRUST, etc) are just looking at self-reported data.
Just take a look at the recent Epic vs. Health Gorilla lawsuit to see how nonexistent the protection is around exchanging your medical records, one of the most sensitive types of PII.
Edit: I was incorrect / non-American, I was thinking of your FAA.
People who haven’t been hacked just haven’t been looked at. If someone wants to hack you, they will hack you. It’s really unfortunate that people have this level of confidence in their ability.
Here’s an example. https://hacks.mozilla.org/2026/05/behind-the-scenes-hardenin...
Has a corporate officer ever gone to jail or been meaningfully fined for a data breach?
> Incidents like this should be followed by an audit and charges being laid
What? Why? Who died? This whole thing is perfectly dealt with through civil process.
Shouldn’t we be focusing on making it harder to pay overseas criminals in the first place? /ahem/ crypto platforms facilitating transfers to bad actors /ahem/
Criminals should focus on proven methods, like Steam Gift cards.
But, then, how would Trump’s family and cronies get paid?
One of those eye opening moments for me was learning about how these criminals work on trust. They need to be trusted to not release the data or to unencrypt when paid, and by and large they do.
One way to weaken any group that works on trust would be to make them less trustworthy. That way victims wouldn't be as confident paying the criminals and thereby making the effort by the criminals less attractive.
Your "minimum sentence so painful" will certainly dissuade foreign nationals, even foreign governments.
interestingly, having actually done the law enforcement side of these investigations, 50% of them are local. And I understand that this is not 100% solution, but neither is any form of law enforcement, but that doesn't mean we should fail to attempt it.
Kids from the local uni having a lark, stalkers, vindictive ex employees, local gangs, criminals who understand their victims because they hail from the same community. These are your local hackers. Sift them from the nation states and international crime groups, then deal with the International as a matter of diplomacy. Because we do this so poorly locally, we have little ammunition to when it comes to diplomacy. "reduce attacks by your crime groups and we buy your natural gas, seel you wheat etc"
Want more motivation?- 75% of the local attacks by volume send funds back to terrorist or separatist organizations.
It is not an in-soluble problem. Sentences are a fraction of the answer, effective and receptive reporting processes are more important, then government backing for investigation and enforcement, then policy around home-team activities (ie don't do the bad things yourselves Mr Gov). Deterrence comes after all that.
One tech ransom case I know of was an inside job. It definitely happens.
There are already significant penalties for doing anything like this. The guy involved is in prison for a very long time. I don’t recall the exact number of years but I do remember it was so long that he wasn’t going to see his kids grow up.
I don’t think anyone who puts a little thought into a crime like this doesn’t understand that the penalties are already very huge. You don’t get a slap on the wrist for extorting a company (or person, for that matter)
50% of ransomware attacks are local to where? You’ll need to cite some sources because I don’t believe that is possible.
To the country or an ally of the country they are targeting, duh. it doesn't matter if you believe it, it's been the truth for over a decade. Heck, Sh1nyHunt3rs people were arrested in the UK recently.
Yeah, they identified themselves as ShinyHunters, and the IP they've put on the demonstration page is geocoded to Russia. Notice this is the same group responsible for the Infinite Campus hack last year.
Really, though, if you want someone to blame, Instructure is not a particularly compelling target. Let's review:
1. Iran is intentionally targeting infrastructure due to a war started by the current administration.
2. China is actively seeking corporate secrets to steal and commercialize for themselves, spurred by extreme protectionism and retaliatory tariffs.
3. North Korea is doing anything they can -- including just taking a remote job by proxy -- in order to extract any money.
4. And Russia is working with and aiding all of them, after everything else going on has forced the embargo to break.
5. All of this while completely alienating every single one of the United States' allies.
6. Meanwhile, the American DHS is currently shut down.
7. And this is after Trump cut funding and personnel for CISA severely enough they've had to end the contract with MS-ISAC, meaning all state and local entities can only remain in the organization if they foot the bill for it directly and CISA and other agencies responsible for cybersecurity are more thinly staffed than they have been in decades.
In short, the current administration systematically disassembled all the protections we have built over the last 100 years, and then placed infrastructure -- schools, in this case, but also power companies, water treatment facilities, communications companies, local governments, hospitals, food producers -- directly on the front lines of the modern geopolitical conflict.
That vast ocean that has kept us safe historically is a poor moat in the modern era.
Having an IP in Russia means about zero regarding their location. Literally anyone doing anything like this is going to get a Chinese or a Russian IP for obvious reasons. Mostly decoy and people like you.
Complete internet blockage of nations allowing the attacks. If foreign governments are you can always execute them. We are living in a different world where this is no longer a zero probability occurrence.
When will countries start treating cyberattacks as an act of war? If the North Korean military came to America and robbed fort Knox of $200M in gold there would be retribution. But hack an American company for the same amount and the feds do nothing.
Ok, so we treat it as an act of war. Now what? Attack North Korea? Great, the entire city of Seoul gets shelled within five minutes of your attack and hundreds of thousands of innocent people die.
It's very easy to play with lives that aren't yours.
Never retaliating is a great way to get people to attack you. Of course escalating to all-out war provokes the same in response, but there does need to be a proportionate response, because it needs to be stupid to hurt us, not good business. t’s a significant failure of the US government when half the world freely loots US citizens and businesses.
You would be surprised how many people naively think "Why doesn't my country just open a war on X country and this Y problem will be solved forever" in their head they think war is just a flurry of bombardments and the other side (not theirs) is just destroyed to rubble and their country will have only minimal losses
Many country leaders also clearly think the same
Exactly. This is the "Declare fentanyl a WMD" of solutions to ransomware. Sounds kinda badass as long as you don't spend too long thinking about it but has no practical relevance to actual enforcement challenges.
It's a familiar example of the perennial "[THING] could be solved overnight if [PERSON_OR_GROUP] would just start taking [THING] seriously" trope.
How do you know which country to blame? It is standard practice for foreign actors (or just hackers in general) to use proxies around the world to misdirect and insert false clues as to their origin. It could be an American teenager proxying through North Korea, and it could be a North Korean proxying through another American teenager's residential connection, there's no way to know.
They already do. This is what asymmetric warfare looks like, your weakest links will break in a time of crisis. Focusing on retribution for the Dunder Mifflin cyberattack is pointless, the adversarial motivation is purely to disrupt and extort.
The best response to a cyberattack on critical systems is to take security seriously. Document the offense, avoid the same mistakes and invest in penetration testing. Of course, nobody is incentivized to do that until they're attacked, so the cycle perpetuates itself.
> When will countries start treating cyberattacks as an act of war?
When appropriate. I.e. never.
If someone robs a bank and someone inside dies of a heart attack, thats felony murder. I would be happy if the same applied to ransom attacks or other blackmail/leaking of info. If someone commits suicide because of it, its murder.
felony murder is pretty widely regarded as a leading factor in incredibly unjust prosecutions and sentencing decisions. perhaps not the best concept to build your ideas on top of.
> If you do this to a hospital and someone dies you are life in prison / chair.
If you're going to get the chair you might as well murder some witnesses or destroy some systems to hide the fact you got hacked. "Hack? What hack? Our servers all burned down in an arson attack".
We could also throw the CEOs of companies who don't properly secure their infrastructure and pay their security engineers enough in jail. A little justice on both ends.
Uh, who determines that the infrastructure wasn't properly secured? Who is willing to risk prison because some intern accidentally committed an API key or made a dumb mistake. Conversely, what's the chances that no one actually gets prosecuted regardless of how sloppy their security practices are?
> who determines that the infrastructure wasn't properly secured
An investigative body, the same kind that determines the who, the why, and the how when an airliner crashes or a bridge collapses. Obviously a lot of work needs to be done to get from point A to point B, and it won't happen overnight, but software development is currently a deeply unserious profession and at some point a genuine software engineering practice needs to be developed.
I am, perhaps naively, slightly hopeful that the LLM bullshit plaguing our industry will be the gust of wind needed for the house of cards to collapse and governments to realise that allowing the entire world to be vibe coded is not sustainable.
Pretty famously, aviation incident investigations are almost always not done with prosecutorial intent, and more about truth finding. It leads to people involved being cooperative to prevent future problems instead of ass covering to prevent jail.
Aviation’s safety record is not coincidental.
In a darker reading; strong aviation safety is mostly motivated by not killing customers. An airline or plane maker who kills more customers than others will rapidly bleed those same customers and lose them to less lethal competitors. If no one cared about dying people I imagine aviation safety wouldn’t be so impressive.
As someone else here said, software, for the most part, is a deeply unserious industry. The stakes are so comparatively low and the consequences less obvious that it’s a lot easier for companies like intuit to maintain their supremacy simply by being entrenched, having strong sales teams, and the hearts & minds of non-technical managers.
In recent times it seems Boeing has been flirting with enshitification and half-assery but critics are not quiet and not falling on deaf ears
Sure, fatal stuff is bad for the bottom line, but that is a vanishing minority of what gets investigated.
You may not be aware, but there are thousands of non fatal incidents reported per year that just don't make the news.
There is a strong culture of self reporting instilled right from basic flight training, even when there is no damage or injuries, and even when the incident would have never been noticed by the authorities. You are almost guaranteed not to face consequences if you are open and honest about an incident. The FAA openly says that they would much rather educate than punish, and they tend to do that with pilots who own their mistakes. As long as there is no intent behind the fuckup, pilots are unlikely to lose their job, let alone their license.
> An investigative body
This just in: Anthropic, Harvard and Jimmy Kimmel have been investigated and found guilty of not securing their infrastructure.
When a great product is built it was the leadership and when a mistake was made it was always the employee that did it. Cool!
Ideally the chances are high to certain they get prosecuted for sloppy security practices. It's part of the gig of being a CEO, if you imagine you are such a visionary/ideas guy/leader/whatever, risk taker (always a risk taker) then you can gamble spending 20 to life because you weren't actually as good as you thought.
> Uh, who determines that the infrastructure wasn't properly secured?
ShinyHackers, obviously.
The only way to prevent terrorism is to never meet terrorists' demands.
Failure to protect computer system from forseen failure should result passing corporate veil and resulting all stock holders and managers/leadership of funds to be jailed for same period as perpetrator. It is only way to ensure that these things are taken seriously and enough pressure is put on leadership of companies.
> No this will not stop this and companies need to be held accountable for their lack of security investment.
I think in principle, its sound. Im also just baffled hearing anecdotes from friends that are in big corp world and hearing the type of incidents they have, and how they respond to it.. It makes me wonder if there is enough capable talent to go around for the "boring corp" crowd.
Hint: I don't think there is nearly enough talent to go round, but for these companies, its either that they think they have solid experts (and didn't), OR its not a real priority until you get hit.
> It should be illegal for any company to pay ransomware attacks. Period.
That makes as much sense as illegal to give your wallet to a mugger.
I.e. no sense.
Or maybe it should be mandatory for all companies to pay ransomware attackers. Think of it as an involuntary bounty program. Now they get to just say 'sorry (for your hurt feelings)' and suffer no consequences.
Apart from the 4% of the total worldwide annual turnover fine that theoretically could be levied under GDPR, but has never been imposed in full.
1. It should be illegal to run insecure services. Massive Fines.
2. The payout to the hackers should form part, but not all of the penalties. Pay those guys for their great service to humanity they earned it.
My kids are in the middle of their finals week. What a mess. Universities know nothing, Canvas claims to be in a "scheduled maintenance", and one Prof claims to "not have any copies of material offline" which seems pretty negligent. Sounds like one section of a popular class will be doing paper exams while other sections had Canvas-based "half points for 2nd attempt"-type exams earlier today. How soon before names & grades appear in data dumps?
This would be like TurboTax "scheduling maintenance" on April 14th in the US.
The "Scheduled Maintenance" is just total B.S. and just honestly makes them look worse. Apparently according to their status pages this is what 99.996% uptime looks like. Pay attention lol.
It has been over 5 hours now and there has not been any communication about this being an attack, despite many of us seeing the ShinyHunters message on the login page.
There is a lot of people who likely are unaware the latest outage is because they were compromised again.
Them marking the incident as 'Under Maintenance' means the status page isn't reporting this as an outage and adding to downtime%.
Once we hit 8h 45m SLA has been broken. https://uptime.is/99.9 https://www.instructure.com/trust-center/availability
Compromised again? This is a separate in ident to the one seen yesterday?
Correct.
The incident yesterday was technically from April 28th, with most communications coming out on the 2nd and 3rd, with it being "Resolved" yesterday.
This incident is the second attack, because they failed to secure their infra again. Everything being reported is a bit delayed, which makes it seem like this is a single attack, not technically two instances.
I was going to make a joke that they should have just taken a page from the military and said “Rapid Unscheduled Maintenance”, but I guess that’s actually the phrase for it.
Once again, an example of why corporations should not have free speech. Corporate statements that are transparent lies should be criminally actionable.
Crazy that kids data are getting leaked before they even had a chance to properly understand the consequences and consent to it being used
A friend who teaches at MIT said they were hit by this. I found it ironic and a little sad that a place like MIT doesn't have an IT staff that can maintain their own on-prem solutions for things like this.
But it turns out that MIT used to have their own homegrown system, and recently switched to Canvas. Bet they're regretting that now.
The build vs. buy decision seems to have swung very hard toward buy in the last decade, and I think that's a shame. Yes, orgs need to focus on their core competency, and sometimes that means outsourcing things that aren't core competencies to third parties. But there are always downsides.
Homegrown systems are expensive to maintain and usually still fail to match up to the commercial options available at this point. LMS's are also just really complicated pieces of software. I worked on my university's own version as an undergrad.
There is no need to reinvent any wheels by making a homegrown LMS. Moodle exists and is completely open source. Lots of large institutions use it. Even in the case that you need to do something really weird with it that isn't solved by one of the many plugins that exist, you're already 90% of the way there with its base platform, and only 10% remaining for DIY software development.
> LMS's are also just really complicated pieces of software
it's MIT.
But it’s not like MIT gains anything from rolling their own LMS.
You don’t need to roll your own LMS—you can self-host Canvas: https://github.com/instructure/canvas-lms/wiki/Production-St...
Maintaining an LMS doesn't seem like a good use of time. You should almost always outsource pieces that aren't your core business.
Computer science != software engineering.
… so?
My highschool, for a while, had a website, which was eventually replaces by a large corporate CMS. Was the website as complicated or complex as the CMS? No, you would have needed to know HTML to publish to it. The CMS was no doubt "more user friendly", I suppose.
But … the original site had a soul. It was unique to the school. There was a student directory! All lost, because the CMS meant utter standardization between all the schools using it (their pages were all identical, except for each got like a different picture of the school as the banner at the top) and the CMS did not do directory anything.
Of course, the directory largely didn't matter in the end. (This was when you needed people's landlines! Quite laughable nowadays…) But it was still sad to see it lost, and several of us students worked on it, which provided us with some early real-world experience.
A large number of my college professors published their own sites, too, where they'd put their lecture notes, homework, etc. I loved those far more than I loved "Canvas" or whatever the ugly LMS we used was.
MIT has an incredible IT staff and they do some cool stuff. Every time I interact with any other organizations IT stuff I find it inferior. They just aren’t super big from what I gathered and probably don’t want to do the incredibly boring work of an LMS.
The one they had before Canvas was very very inadequate.
edit: also some of the more popular cs classes have custom websites and don’t really use canvas, but that isn’t the centralized IT department’s doing.
I started my tech career in EDU. I’m not at all surprised.
IT staff who are ambitious and talented don’t last long in education. The pay is very low compared to industry. Where I worked, you could retire with a comfortable pension after a number of service years, so the IT staff outsourced as much as possible so they needed to take zero risks to their nest egg. Blame all the problems on the consultants and do as little as possible.
It’s literally where dreams go to die.
MIT is known for the brilliant professors and students but at the end of the day, running a university is pretty standard stuff. They don’t need a genius rockstar to admin the courseware servers.
CYA is a powerful drug for the C Suite
I'm a student at Stanford — this is hitting the whole school hard. Unlike a lot of schools on the east coast that are affected (Brown, Harvard, MIT) we are on the quarter system so we're just ending Midterms right now. We're also lucky enough to have our CS department entirely independent from Canvas, but most of my humanities classes are not so lucky. One art history class is having us submit our midterm papers by uploading to a google drive folder—another is pausing weekly quizzes. The main thing this has revealed is just how dependent students and teachers are on Canvas... I hope that this re-prompts discussions about moving off of a platform that was already (from a student perspective) not very good.
I really feel like SH fucked up by sinking this low hitting students and Americas young minds like this....
One thing to target coroporations but leave the students alone....
It's not so bad, I'd say the Christmas PS3 hack was worse
You don't care that students are impacted but your ps3 not being playable for a short period was more important.
Heard you loud and clear sheesh
And what's your opinion on the em dash?
Canvas is handling this terrible. No communication, no status updates, etc. Also looks pretty bad their whole platform was compromised and not a single real report for the breach that already had happened. Wonder how long it will take for SLA violations and lawsuits to manifest, especially with most U.S. schooling having finals right now.
Lot of experience dealing with Canvas/Instructure. Tech is o-k. Culture seems to be full of themselves due to market position.
Yeah like their page says "Scheduled Maintenance" which is total B.S. Talking to people at my university's IT side of things Canvas has said nothing to any clients.
The "scheduled maintenance" thing is likely just because that's the easiest maintenance page to throw up site wide, or at least it was back when I was on the Canvas deploy rotation back at Instructure ~10 years ago.
That doesn't excuse any of their other messaging though.
So many universities used to run homegrown or on-prem student systems. This is the downside of consolidating in the cloud. If the infrastructure is compromised, it affects everyone, not just isolated or single installations. I wonder how they are feeling about that decision now? I guess they can say "not our fault" so they might be feeling better than if it was a vulnerability in their own system.
If an exploit is found in the software, hackers will often be able to attack hundreds of separate institutional installations in an automated way just as easily. And depending on the exploit, potentially more easily if on-prem admins fail to take all recommended security steps.
I'm actually much more interested if there is any financial liability for Instructure here? It's interesting that it's the universities being ransomed, while the technical failure was Instructure's. We're used to uptime SLA's -- what about security breach SLA's?
> It's interesting that it's the universities being ransomed, while the technical failure was Instructure's.
My guess would be they get likelihood of getting paid when blackmailing 9,000 schools (at least a few would pay up) than blackmailing Canvas/Instructure.
I don't think any SLA/terms would change who gets to feel the pain.
My guess is that they believe by maximizing their attack coverage, the odds are greatest that some of the institutions will pay up. And otherwise, they can still make a bit of money by selling the data.
Don't ransom all your eggs in one basket
Yeah, if they had spent the time and money to roll their own that got hacked, they'd be responsible. Now, they can just clap their hands and show them palms up to you like a black jack dealer and walk away from the table with no responsibility. Probably one of the biggest benefits of using a product instead of building your own.
You’d think this is how it works but universities and schools will still end up holding the bag at the end of the day, irrespective of who is responsible.
It's annoying that this is how internal politics usually works. Decision-makers at an org should be considered just as responsible when a third-party choice goes bad as when an internal tool goes bad.
Running on prem or homegrown systems used to be considered a core competency of having a computer science department and a campus-wide IT/networking staff at a university. In the environment that exists today in academia, for instance, BSD would never be created because somebody could just pay a third party external vendor for some packaged product. What happened in the past 20 years to change that? I really wonder.
It's still more secure this way, especially with AI hacking making it harder to rely on obscurity.
Also yeah there is value in being able to blame another party, and also being down when everyone else is down.
Is there a good self-hostable FOSS version of Canvas/Blackboard?
Canvas is open-source and can be self-hosted.
I remember when I was in high school (2016? 2017?), I found a super simple XSS in the assignment submission form and told the programming teacher. Canvas then proceeded to lock my account and got me my first (only?) detention. Good times.
Somewhat similar vein, the school's blocking software would block YouTube and embeds unless they came from Canvas. They were smart enough to disable the HTML editor for posting discussion comments, but forgot that since it was a rich text editor, you could just copy-paste in an embed by putting the code in data:text/html, then copying the element as formatted html.
I also ran the entire DOMPurify sample XSS and managed to find one way to download custom content onto someone's computer.
Uh, did you tell the teacher by exploiting the vuln?
Goddammit. Anyone in the know, know if Parchment was also impacted by this potentially? They were acquired by Instructure a few years ago, and deal with a LOT of transcripts.
Edit: https://status.parchment.com/ says "While Canvas, Canvas Beta and Canvas test are currently unavailable, we are simultaneously monitoring all of our other product environments, including Parchment. We continue to see no reason to believe any Parchment resources have been impacted."
I remember circa 2010 a friend of mine at college was like “blackboard sucks, let’s build something new”. At the time I poo pood the idea and lo and behold canvas came out a year later. Outside looking in, they been crushing it.
One of my mentors created Blackboard. It used to be very very good, but he sold it to private equity, and they immediately fired all of the customer support and developers, 3xd prices overnight leading to the 'blackboard sucks' problem. This gave the opening for Canvas to eventually come on to the scene and dominate.
I believe Canvas was also sold to private equity pretty recently too. https://www.instructure.com/press-release/instructure-to-be-...
canvas was bought by PE for the first time in 2020 https://www.thomabravo.com/portfolio/instructure
My wife and I each have to use it as we're both following an online master's at the same university... it's definitely gone downhill (compared to the days where I originally used it ~20 yrs ago in college; tracker-riddled, slow); surprisingly, a recent change made it so that you can only attend online lessons in Chrome (haven't had time to see if this is just a user-agent thing).
..and be acquired by PE so the cycle can continue.. https://www.instructure.com/press-release/instructure-to-be-... sigh. Barbarians at the gate probably didn't double down on security
I worked in a college IT department around that time and the common belief was that all LMSes suck. There are just too many different ways that too many different people want to do things that it's just bound to be hated. Kind of like Jira / Asana for software dev project management.
LMS’s are a lot like programming languages. There’s the ones people complain about and the ones no one uses.
I used both and could not tell you the major differences. I feel like they are equivalent in the bread and butter features. Most people don't use 99% of the functions they bake into these. Just use it to hold the syllabus, maybe hold the slides, submit assignments, and spreadsheet for grades. All stuff you can do with email + spreadsheet already. Maybe throw in a shared drive for larger files, which every university in the country already pays for.
"Equivocal describes something ambiguous, uncertain, or open to multiple interpretations, often used to intentionally mislead or evade."
do you mean equivalent ?.
yes
Blackboard got a lot better in response to the flood of customers heading to canvas.
As someone who has used both as a student and a TA I find blackboard miles better, much easier to find what i'm looking for and my professors seem to have better luck laying out their course on blackboard than canvas.
I actually disagree, based on my time using Blackboard as an admin, student, and teacher. Although my experience is a few years out of date, I found the interface cumbersome and the performance slow.
It depends on what vintage of Blackboard your IT team has installed. We moved from a circa 2011 BB instance to Canvas in 2022, and it was hands down superior. A different university is running the most recent BB and it’s similar to Canvas.
> circa 2010
Instructure, "the developer and publisher of Canvas," was founded in 2008 [1].
[1] https://en.wikipedia.org/wiki/Instructure
That sounds like “circa 2010” to me. And Canvas was launched in 2011, according to the article you linked.
Blackboard, the Canvas predecessor, was so unstable that we called it BlackOutBoard
How does canvas compare to Brightspace?
Maybe schools should be self-hosting something like Sakai instead.
They are definitely crushing it on sales. The actual product is a radioactive dumpster fire that is simultaneously hostile to students, teachers, and parents.
Yeah but the customer is the administrators who never have to make contact with the real world
The Canvas instance at the nearby university is now down (May 7, 4 PM Eastern), but was briefly displaying the message in this screenshot (1). The ransom message implies that today's problem is the second wave in an attack on Instructure after ignoring their first breach in recent days.
1: https://ibb.co/r29RjdnH
Yeah, this is ongoing.
We received communication that Canvas is down for "Under Maintenance" although it seems ShineyHunters have compromised Canvas again with that message you posted.
We do not see that message anymore, although all instrucuture.com URLs are down. The list of schools in the ShinyHunters publication can be found here: https://web.archive.org/web/20260507042014/http://91.215.85....
https://web.archive.org/web/20260507042014fw_/http://91.215....
Original now shows 404.
Seems like Canvas instances of schools not listed are also down (at least my alma mater is)
Yes, I work for an Australian online school. We’re down “for scheduled maintenance” (I question how “scheduled” it was given this is within school hours on a school day), but we’re not on the list published by ShinyHunters.
our instance went from [insert hacker leet text] to "down for scheduled maintenance" and myself and other faculty are just having the darkest humor about this.
My wife is in grad school at a major university and is dealing with this right now the week of midterms for spring quarter.
I totally understand why a university wouldn’t want to bake their own learning portals but just feels like such a single point of risk to use third party solutions for something like this.
Back in my day… all we had was a school email via on-premise services. I guess we registered for classes in a web portal but that’s about it. The idea of online class was entirely foreign at the time. Ain’t nobody hacking a blue book.
It’s wild to me that people in this comment section are suggesting that schools should improve their security by rolling their own platform, which is bound to be filled with security holes, instead of using a popular, maintained, open source option.
To be fair to the idea, though, while this would make individual instances less secure, it would drastically decrease the leverage for the work bad actors put in.
There is a saying in the software security industry that (I'm paraphrasing from rusty memories) a system is secure if the cost of hacking it is higher than the value it protects.
Each system being completely distinct from another means that the cost of hacking the average student goes up by 9000 (from the article, Canvas is used by 9000 schools).
Still not saying that rolling out your own is the preferred solution, but the idea is not as ludicrous as it would seem, and should definitely be entertained and discussed, at least.
Maybe. I still remember the Drupal community sneering at the New York Times when they unveiled their homegrown online news platform bitd. After 15 years of recursively scraping ad-hoc porn sites off of server hard drives when clients dragged their feet on migrating to latest versions I 'm less certain the assumption that homegrown == less secure is as valid as it sounds.
Could you explain the last sentence a bit more? I don’t follow
Universities used to do this sort of stuff themselves. Then it became a business handled by purchasing rather than needs met by the department themselves.
In fairness in the era where universities did it themselves the tech requirements and expectations were dramatically lower.
Tech requirements are the same as they always were. One needs to ask whether they need so many frameworks to host some files on the internet and submit some files and perform spreadsheet calculations. We still used one of those First Age 1990s websites for sort of pre lab quizzes this one class when I was going through it, and it might have looked a little "old" but I mean it did the thing and worked for years and will continue to do the thing and work for years.
You're being deliberately obtuse. Canvas has many many features. Wikis and discussion boards and quizzes (with some anticheat) and groups and the list goes on and on. Furthermore, while it was never the flashiest thing, it did it better than many of its predecessors. Yes, an individual class may not use all of these features, and yes canvas has suffered feature creep even over my time as a student and yes canvas is not doing anything technically challenging, but there is enough of it that each school rolling their own everything would be a drastic waste of everybody's time and money.
Have these dramatically higher tech requirements and expectations improved the quality of education whatsoever?
Because faculty didn’t want to do it anymore. They want it handled by others but also they want oversight and veto power but also they don’t want to be bothered. But it better always work, and if they make a mistake the software is broken because don’t tell them it’s a user error they used to write Fortran.
As a faculty member at a large university…I have a deep respect for the impossible job of university IT departments.
We originally rolled our on LMS decades ago. When we switched to canvas we kept the home brew running for five years past its expiration date because faculty refused to remove their files. Finally each one was manually moved by IT for the recalcitrant old faculty.
It is kind of funny when these LMS tools with 100+ functions are being used for little more than what email, a grades spreadsheet, and maybe a shared drive would do. University might even ask for the final grades in spreadsheet format by the end of the term anyhow, so data goes into the LMS just to come back out again.
In a sense you aren’t wrong but those analogies fail at scale. It’s like saying you could replace all hr functions with a spreadsheet.
They are large databases yes but they do a lot of small and large things that that analogy glosses over
> Ain’t nobody hacking a blue book.
Well not with that attitude
A university doesn't need to bake its own learning portal, Moodle exists and is used by a lot of large schools.
Moodle is an open-source LMS that can be self-hosted.
https://moodle.org/
Another open-source LMS that can be self-hosted is... Canvas.
Almost no one does
Didn't realize that. Thanks for the info!
I totally understand why a university wouldn’t want to bake their own learning portals
They used to, in the pre-cloud/SaaS era; and they were much simpler and better UX than the slop that they're renting today, because the actual users were not far from the developers.
Counterpoint: I was a PhD student in 2004 and on the universities board* which oversaw the roll-out of the campus management system. It cost > 10m EUR to implement a shitty system with the worst UX and years of stabilizing to make it somewhat work.
The amount of corner cases and performance requirements during rush times (semester start) made it really infeasible for a university to roll their own.
* German universities have this funny system where 51% of such boards are controlled by the professors and the rest is made up of other employees/staff and students. They call it academic participation.
Respect to Canvas sales team, its like microsoft level platform lock-in into low sec infra
Also here: https://news.ycombinator.com/item?id=48054386
It looks like Instructure has been removed from the ShinyHunters website. Both the entry and the list of schools has been removed.
Look for large BTC moves recently?
Ransom paid?
https://status.instructure.com/ implies Canvas became available again about thirty minutes ago from the time of this post.
Is this accurate? Or is this still an ongoing issue?
Ongoing. It is not "down" but purposefully offline for "maintenance." Main status does show the LMS (all the course stuff) down, and my instance shows "up" but that's because (I assume) you can reach it and the maintenance page. But that's not useful, if technically not "down."
Thanks
Canvas LMS is the core service that universities rely on. I assume they're trying to develop a fix and that's why the service is labeled "Under Maintenance". I'm a Berkeley student and can confirm that our instance (bcourses.berkeley.edu) is still down.
Federated logins appear to now be broken for the campus I’m affiliated with. So more action is needed.
> Canvas is currently undergoing scheduled maintenance
doesn't seem that scheduled to me
ex-Instructure employee here (though it's been about 10 years since I worked for them).
That's just the quickest page/status update to throw up; it was a one-liner to push it live back when I was on the deploy rotation.
I'd hazard a guess they have more important things to worry about right now than exact status page messaging ;)
> That's just the quickest page/status update to throw up
Funny how a lie is always quicker than the truth...
I thought the same. The "scheduled" part of the message is gone now, at least on the instance I use.
Well, scheduled by whom? :)
Whoever it is, is likely defended by Cloudflare. They seem to like the booters.
https://news.ycombinator.com/item?id=48025001
The boy is a biochem PhD student at UIUC and reports that all their finals are now cancelled. "Is this good news?" I ask. "Yes. Everything coming up Milhouse."
yep, i work for a major university and our canvas instance is down. this is really, really bad.
edit: here's the list of impacted universities (unsure if they all have their canvas instances offline, but i'd be surprised if not): http://91.215.85.103/pay_or_leak/instructure_affected_school...
The source txtfile has since either been dos'd or deleted (at least it was when I tried to access)
Someone dumped the content into a google doc on reddit[1] if anyone's interested.
[1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...
> The source txtfile has since either been dos'd or deleted (at least it was when I tried to access)
> Someone dumped the content into a google doc on reddit[1] if anyone's interested.
> [1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...
Thanks for linking this. Ended up finding my kids school district on the list unfortunately.
tbh this has me wondering if canvas "instances" are actually as isolated and segregated from each other as they're supposed to be.
Define "as they're supposed to be".
Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to.
Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other.
It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.)
---
Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this...
---
(Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
It's possible that Instructure's servers got compromised:
dig canvas.ucdavis.edu
dig canvas.duke.eduthat's what the screenshot says. They rooted Instructure servers.
It depends on what you pay for. If you need FedRamp or IL4+ compliance you are likely on dedicated infrastructure. Everyone else uses multi tenancy.
https://web.archive.org/web/20260507042014fw_/http://91.215....
Here's an archive https://archive.is/eB2hE
I'm shocked universities don't host their own LMS? At least large universities have the IT departments to do this. They host compute clusters, so they can certainly host an LMS.
The same reason hospitals don't have their own Patient Information System but all use Epic. The amount of customization you need and continuous churn due to changing curricula and regulatory requirements makes it hard to keep up without scale.
A post on the official Canvas forum: https://community.instructure.com/en/discussion/666027/ranso...
Hugs going out to the teams at Instructure working to fix this. I've been through a similar Ransomware attack (national news stories, lots of customers dead in the water, etc.), and it's about as bad a situation you can wind up in.
My son was in the middle of an exam and then his screen went black and it showed the message from ShinyHunters. Hasn’t been able to get back in since.
Somehow I have less distaste for ShinyHunters than I do for the companies who don't secure user data
When you picture the attacker, don't picture a bored nerdy teanager. Picture a selfish, $$ motivated psychopath.
Let's not side with the parasites.
And lets not side with Canvas PR.
Just learned the defacement page was hosted from instructure's own aws bucket so seems pretty bad.
https://archive.is/5v693
What are we even coming to when even internet blogs are paywalled. Verge? Next thing Gizmodo is gonna be paywalled.
I used canvas for some Harvard extension classes 10 to 5ish years ago. It worked Ok. Work distributed, grades posted. I didn't realized so many schools used it, or that it was all schools on one instance, which seems kind of nuts.
I lost access when I left as it was tied to my work email. I downloaded a lot, but there was still some useful stuff on the boards.
I wonder what the havkers found out about me. Perhaps the class notes will be lifted to train AI, higher quality than a lot thats on the internet anyway.
I discovered one of my old school assignments ended up on some homework help website. I had never posted this document publicly and had only uploaded it to the schools work submission page. Presumably at that point it was shared with multiple third parties for plagiarism checking and such. And then was exposed to a data breach years later and ended up on the public internet.
A college student I know just sent me a screenshot, he can't access canvas for his school at all
Same, my daughter just sent a screenshot, she was trying to study for finals.
One thing I remember from my days in the LMS world is that obfuscated copies of prod tenants were used for testing. Almost every dev had at least one tenant from prod on their local computer. So with some de-obfuscation at least some of the data is plausibly retrievable. Whether that data is also public depends on how the negotiations go.
It's been a long time since I was in school. What does this software do?
It is how classes (even in person ones) are organized. Assignments, quizzes, links to online textbooks, discussion boards, student/teacher messaging, student group messaging, etc. From the teacher side, I'm not sure if there is a backup copy for things like grades outside of Canvas. It's that pervasive.
Everything from middle school up to grad school.
It's a particularly interesting time to have this happen too -- many finals going on now.
It’s a “learning management system.” It replaces a course website in most instances. It’s also used for course grades and you can submit assignments or take quizzes.
Grades, lessons, quizzes, exams, homework submission, rosters, messaging platform. Lots of things.
If you’re a student or teacher: nearly everything that matters. Homework, materials, lectures, grades. It’s all on canvas.
Qld, Australia was also affected: https://www.itnews.com.au/news/qld-gov-says-students-staff-c...
QLD Government vendor selection is always terrible.
Student at an impacted university here.
Our whole testing center is down. This is inconvenient, but mainly it's amusing. I swear strangers are talking to each other more. I'm noticing people just sitting in the sun and relaxing. Nature is healing.
(Of course, plenty of people have also just finished their exams, so it's hard to know the cause.)
Any idea what data Instructure-and-also-now-ShinyHunters even purport to have beyond names, profile photos, pronouns, homework assignments, school communications, phone numbers, and email addresses?
i.e. What makes this threat so different from what any old data brokers have already scraped?
What leverage besides aura farming do the ShinyHunters really have?
All I can think of that's really valuable is passwords. And private communications in Canvas DMs. But if you're being at all intimate over your school email, that's kinda on you.
Anyway surely Instructure only stores user public keys or something?
Alternate history question: If they just sold the data, never revealed the hack, and didn't make a scene, from a customer perspective, how different would this be from business as usual?
I use Canvas for some postgraduate studies, and my teenage daughter uses it at her high school.
We already bond over how awful the Canvas UX is (and she has a bunch of Chrome extensions to improve it.) Now we’ve got something else to gripe over together.
It is open source, so you could send pull requests with improvements: https://github.com/instructure/canvas-lms
https://github.com/instructure/canvas-lms/pulls?q=is%3Apr+is...
haha i went to go check and they haven't merged a PR since 2017
Look by is:closed instead. They don't merge the PR directly.
I vibecoded a pretty extensive CLI for Canvas and using it is very pleasant. Joyful, even, when combined with an LLM. Especially when compared to the developer hostile Blackboard Ultra.
Canvas seems like it’s not that great. But if you then use Blackboard Ultra it makes canvas look amazing.
I wonder when the public is going to start calling for corporate liability for malpractice in software development and corporate liability for malpractice in IT deployments. Even if the tech industry fights it, it probably won't be that much longer.
I'll never understand this point of view. If someone would please explain how to create perfectly secure software, I will gladly start writing perfectly secure software. Only after, if it's clear I ignored obviously correct advice, should there be malpractice penalties.
Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths. Malpractice insurance is high. Litigation is constant. And patients still die on the operating table. It's unclear what all the malpractice tort law actually gets you in the end.
> "Consider surgery instead of software development."
Is that really the analogy you want to use the bolster your argument? Licensing was forced on the medical profession because of rampant quackery causing a large number of deaths. Some of the horrors that went on before enforced medical licensing are well-nigh unbelievable, e.g. https://en.wikipedia.org/wiki/John_R._Brinkley
> Only after, if it's clear I ignored obviously correct advice, should there be malpractice penalties.
In most of these cases, the companies involved did NOT follow standard security practices.
I am pretty sure that is what people mean when they say "held responsible", they mean "held responsible for failing to follow standard security practices", not for the actual act of getting hacked.
> Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths.
I like this analogy, but deaths shouldn't be the leading indicator just an indicator. Family member had a surgery with well known procedures, say removing a gall bladder. Unfortunately, this surgeon skipped a step in lieu of setting a record for fastest procedure. Because steps were skipped, the gall bladder was not scooped into a net to avoid spilled gall stones which resulted stones spilling into the abdominal cavity requiring numerous follow up surgeries to remove the spilled stones as they made themselves known. So clearly not following accepted procedures should be a clear win in a malpractice case, yeah? Wrong. No doctor would testify against the surgeon and the case was dismissed. I feel like this is exactly how it would work in software security incidents as well.
> this surgeon skipped a step
That was the foundational premise of Dr. Atul Gawande's book The Checklist Manifesto, an expansion of his article The Checklist in The New Yorker [0]
[0] https://www.newyorker.com/magazine/2007/12/10/the-checklist
I agree that even if companies do everything right, they can still get popped. But most companies do not do everything right, and they should be legally responsible for those things.
But even if they do everything right, is it really fair to let the companies just shrug their shoulders and say "it happens"? While their users are the ones who really get hurt.
Well, you don't know how many more would have died if doctors and hospital didn't care about their insurance going higher???
I do wonder if that won't just end up INCREASING ransom-type attacks, though?
If we increase the penalties for a company being hacked, you create even MORE incentive for hackers to try to break in, because if they succeed, they have a pretty big stick to threaten companies with when demanding a random payment - not only will the company have the negative effect of the data being leaked and the PR that accompanies it, they now know that if they don't pay and the attack becomes public knowledge, they face a big fine or other punishment.
A company is much more likely to pay a big ransom if they know they are just going to end up paying that much or more in fines if they refuse the ransom and report the hack instead.
If you take this route, and increase punishment for being hacked, you are making a pretty big bet that the main reason companies are hacked is because of poor security practices. I am not sure if that is true or not.
There's precedent for simply making it illegal to pay the ransom, e.g. https://www.reuters.com/world/uk/uk-plans-ban-public-sector-...
That is already happening in the EU [1][2]. Most of the world will catch up soon I suspect, with some notable exceptions.
[1] https://digital-strategy.ec.europa.eu/en/policies/cyber-resi... [2] https://ec.europa.eu/commission/presscorner/detail/en/ip_22_...
Tbh this is extremely annoying for high school/college students too. High schools are in the middle of AP tests, and many universities have yet to finalize grades, so overall this is a terrible time for this to happen. After the first issue a few weeks ago Canvas should have upped their security and prepared for another attack. They also should provide better communication. If Canvas is down for more than a few days, many schools and universities will have a lot of trouble when it comes time to publish course grades.
I wonder how much old data Canvas keeps around? Are students who graduated in 2016 going to be at risk of having their academic data leaked?
I bet it depends on the institution and the IT team behind said institution, but at least for my university we apparently don't delete old course shells or anything.
I'm friends with a professor who complained to me a couple times about how sometimes he will need to scroll through pages and pages of courses he taught in the past. He also mentioned that profs aren't able to delete their own course shells either.
It wouldn't surprise me if most of it is still around. The amounts of data are probably fairly small, and thus unless intentionally deleted, it's probably still there (maybe unis in Europe are more likely to bother to click the relevant buttons as to comply with the GDPR?). I can't imagine storage becoming an issue unless you've got a huge uni or classes that deal with video (and even then, those probably end up on Youtube as private videos, or only as really small clips).
Our public school system here in Maryland got hit, ransom screen.
Down for all students at my University… it’s going to be a headache for all professors to deal with extending due assignments.
My daughter says that Northeastern is also affected. Is it more widespread? Did they infect all SaaS Canvas universities?
Yes, all 8000+ institutions that use Canvas.
Pretty cruel to do this right around finals.
That's exactly the point, I'm sure.
Even more incentive to pay up. I wonder if the timing was intentional or just coincidental.
That is the point. Get an extra million or two $ in btc from Instructure.
What's in the files they've already released? Some of them are > 800GB.
Where are you getting that information from?
I'm under the impression files are getting released 12th May. I don't see any reporting on 800GB?
Grades, records, etc I would assume. Someone else pointed out that they recently acquired https://www.parchment.com/ so they may have also been able to scoop up those records too
Also discussions between students and teaching staff.
I'm guessing loads of student work? If so, it'll be great for anyone who wants to research AI usage in papers.
I’m not surprised. Canvas kind of sucks. And their development is slow. And they are poor at communicating during mundane events.
They're also apparently poor at communication during highly interesting events as well
I remember this group did something else a while back too.
Great. More data gone astray. Given Canvas' handling of the situation, I doubt they're going to learn much.
The timing probably isn't a coincidence. Great time to stress out students and staff alike. Hopefully it doesn't affect them too much in the end, but I imagine it will.
It looks like every CSU System is on the list (California State University). Surprised this hasn't hit the front page yet.
Possibly because they haven't released the data yet?
I'm honestly surprised more people aren't talking about this.
And grades are due in the next week or so for many of these (usually a quick deadline at the end of the semester due to graduation happening)...
My wife’s grades are due tomorrow. She was in the middle of finishing exams when it happened. She can’t even access the exams to grade by hand. Total mess.
Graduation is just a ceremony. The actual credential award depends on whether you finished all your coursework and is not time-boxed by that event.
Of course if you can't complete your exams because of this, that's more of an issue!
Some instances seem to be recovering. I wonder if a ransom was paid.
It looks like Instructure has been removed from the ShinyHunters website. Both the entry and the list of schools has been removed.
i work tech at a university that's impacted by this. while it doesn't impact me directly, many many other staff and instructors i know are heavily affected by this outage. the students are absolutely outraged, mostly because the university hasn't been providing updates as quickly as they'd like, but since the staff/admin are waiting on word from instructure -- and there hasn't been a lot from them, it just generally sucks for all of us.
this is really, really, REALLY bad. it's not great that names/emails/etc will potentially be leaked, but also private messages between students and instructors. and since many of the campus systems rely on canvas integration, things have pretty much ground to a halt a week before finals.
after they were breached on the 1st of this month, instructure had an announcement yesterday that "everything is great! we're good! hackers are gone! we've rotated our keys!".
no. nothing is great. we are not good.
It is absolute chaos at my institution. This is the last day of finals and grades are due Monday morning. Most faculty are spending today, tomorrow, and through the weekend finalizing grades.
What we don't have access to includes:
* Already graded work
* Ungraded work
* overall adn assignment grades
* lists of students and student emails from the course
* messages from students that are often sent through gradescope
Just...complete implosion.
What happens if the system isn't back up in time for grades to be submitted? Just a delay?
Nothing to see here folks. Just another predicable data breach from allowing companies to do whatever the hell they want with sensitive personal information.
This will keep happening, more and more, and never stop, until we create a software building code and legally require it for all online businesses.
Universities, Parents: ya'll actually have the political and economic power to get a software building code passed. This incident isn't the last.
Eek I bet there are a few people at Instructure who won't be getting much sleep tonight!
going after systems that affect students is beyond bad taste
You learn all the technical details only to harm people like that instead of making a modest and honest living.
Shame on your existence basically.
Damn, all schools in our district in Washington moved to Instructure last year.
They moved away from Teams because it objectively sucked, but I haven't heard of widespread compromises like this in Microsoft's systems so...
Well instructure is slightly better than the somehow legal torture of having to use the "product" Microsoft Teams
Canvas shouldn't exist in its current form, and neither should have Blackboard.
It's always been as stupid as requiring that your chalkboard, chalk, chairs, bluebooks, pens, paper, gradebook etc etc all come from the same company.
I, for one, am very much looking forward to my IT Gov council meeting tomorrow.
Terrible that this affects children and that their information may be ultimately leaked. They need to be greater consequences in the law for security breaches.
I hate Canvas. I would rather run a course on GitHub. But our university forces it on us. And now this.
Do you remember how Canvas was a gigantic improvement over Blackboard?
And GitHub doesn't provide a way to record grades that remain private per student last I checked, much less sync them to the university, or 99% of other things Canvas does.
I don't love Canvas, but it's far, far preferable to a world without it.
It is really convenient and stays out of the way. As much as I'm enjoying the mess, I am forced to appreciate its value.
> remain private per student last I checked
last I checked it appears grades remain private per planet or so ...
How does Canvas compare to things like Moodle?
Or is it an entirely different class of beast?
I've written a bunch of LMS integrations so I've had the opportunity to use all of the major LMSs. Basically, all LMS systems are rather user unfriendly and complicated with a ton of customization options hidden under layers of sub-menus/configuration settings. At their core they provide a grade book, student management tools, and some basic CMS type functionality for posting class messages/coursework/etc. They've all adopted a standard for interacting with external tools (LTI).
Canvas generally is the 'easiest' to use, and the 'cleanest' looking one although D2L Brightspace is pretty good too. Moodle out of the box is pretty confusing and ugly, but I've seen some heavily customized instances that look a lot better. Blackboard is the worst of the bunch IMO.
Wow, I last used Moodle in 7th grade, 2008. It seemed like a similar thing.
I saw this happen to my Canvas account today. At first I thought it was a prank from the school or Instructure. The message was sent to students which makes no sense. Second, the message that was sent basically implies that ShinyHunter is actively getting patched out, and no one is ever going to give into their demands. They're basically saying that they're done and desperate. It's a strange message for ShinyHunter to send, but I think they were trying to pull off a psyop / FUD.
Looking into the payload they sent me this is how they hijacked the screen. Everything in the payload is unchanged except for one line of code:
<link rel="stylesheet" href="https://instructure-uploads.s3.amazonaws.com/account_9363000..." media="all"/>
This links to the following styling sheet:
@import url('https://fonts.googleapis.com/css2?family=Orbitron:wght@500;7...');
html, body { height: 100% !important; overflow: hidden !important; margin: 0 !important; padding: 0 !important; }
body > * { display: none !important; }
body { display: flex !important; align-items: center !important; justify-content: center !important; background: #07080c !important; }
body::before { content: "" !important; position: fixed !important; inset: 0 !important; z-index: 999998 !important; background: radial-gradient(ellipse at 50% 20%, rgba(255,59,59,.06), transparent 55%), radial-gradient(ellipse at 50% 85%, rgba(125,70,152,.04), transparent 45%), repeating-linear-gradient(0deg, rgba(255,255,255,.035), rgba(255,255,255,.035) 1px, transparent 1px, transparent 3px), #07080c !important; pointer-events: none !important; }
body::after { content: "\A\A" "S H I N Y H U N T E R S" "\A" "rooting your systems since '19 ;)" "\A\A\A" "ShinyHunters has breached Instructure (again)." "\A" "Instead of contacting us to resolve it they" "\A" "ignored us and did some \201Csecurity patches\201D." "\A\A" "\26A0 W A R N I N G" "\A\A" "If any of the schools in the affected list are" "\A" "interested in preventing the release of their" "\A" "data, please consult with a cyber advisory firm" "\A" "and contact us privately at TOX to negotiate a" "\A" "settlement. You have till the end of the day by" "\A" "12 May 2026 before everything is leaked." "\A\A" "Instructure still has until EOD 12 May 2026" "\A" "to contact us." "\A\A" " \25BC DOWNLOAD AFFECTED_SCHOOLS.TXT \25BC" "\A" "91.215.85.103/pay_or_leak/" "\A" "instructure_affected_schools_list.txt" "\A\A" "visit us: shnyhntww34phqoa6dcgnvps2yu7dlwzmy5" "\A" "lkvejwjdo6z7bmgshzayd.onion" !important;
}@keyframes pulseWarn { 0% { box-shadow: 0 0 20px rgba(255,59,59,.15), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } 50% { box-shadow: 0 0 55px rgba(255,59,59,.4), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } 100% { box-shadow: 0 0 20px rgba(255,59,59,.15), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } }
The hack is crude, and it seems unlikely that they have any access to Instructure's developer tools.
At the same time, Aussie tech giant pauses work, devotes entire week to AI Design software giant Canva has halted normal operations across its 5300-strong global workforce for five days of nothing but AI learning and hackathons, bucking the global wave of technology giants that have slashed jobs, citing the technology. https://www.smh.com.au/technology/aussie-tech-giant-pauses-w...
Canvas is not Canva. Is this a bot reply