We have successfully replaced thousands of complicated deep net time series based anomaly detectors at a FANG with statistical (nonparametric, semiparametric) process control ones.
They use 3 to 4 orders lower number of trained parameters and have just enough complexity that a team of 3 or four can handle several thousands of such streams.
The amount of baby sitting that deep net models needed was astronomical, debugging and understanding what has happened quite opaque.
For small teams, with limited resources I would still heavily recommend stats based models for time series anomaly detection.
May not be your best career move right now for political reasons. Those making massive bets do not like to confront that some of their bets might not have been well placed. They may try to make it difficult for contrary evidence to become too visible.
This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.
You'd think that rational businesses would take the low-risk snooze-fest high-margin option any day instead of unintelligible and unreliable options that demand a lot of resources, and yet...
>This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.
In 2013 my statistics professor warned that once we are in the real world, "people will come up to you trying to sell fancy machine learning models for big money, though the simple truth is that many problems can be solved better by applying straightforward statistical methods".
There has always been the ML hype, but the last couple years are a whole different level.
Say you have bet billions as a CEO, CTO, CFO. The decision has already been made. Such a steep price had to come at the cost of many groups and teams and projects in the company.
Now is not a time to water plants that offer alternatives. You will have a smoother ride choosing tools that justifies that billion dollar bet.
Decision-making in organizations is definitely a hard problem.
I think an uncomfortable reality is that a lot of decisions (technology, strategy, etc.) are not optimal or even rational, but more just an outcome of personal preferences.
Even data-driven approaches aren't immune since they depend on the analysis and interpretation of the data (which is subjective).
Data informed is good. Purely data driven is a bad idea.
After all even in Physics big advances came from thought experiments. Data is one way to reason about a decision, logic and knowledgebase is another way. Both can be very powerful if one retains the humility of fallibility.
In organizations one common failure mode is that the organisational level at which decisions are made are not the same levels where the decisions are going to have their effects felt.
It's a really difficult problem to solve. Too much decentralisation is also a bad idea. You get the mess of unplanned congested cities.
For a while now, I've been summarizing the ease with which everything turns into a "Humanity Complete" problem via: "Delegation affords so much, but trust sure is tricky."
At least one aspect of all of it is that trust is a little like money/capital and "faking it" is a bit like "stealing". The game theory of it is that since faking is virtually always vastly cheaper there are (eventually) huge incentives to do so, at some point by someone(s). So, almost any kind of trust/delegation structure has a strong pull toward "decay", from knock-off brands to whatever. It just takes a sadly small fraction of Prisoner's Dilemma defectors to ruin things/systems thereof. 2nd law of thermo makes order cost energy and this decay feels like almost an isomorphic (maybe even the same..?) thing. It's not just product/tech enshittification, but that might be yet another special case/example.
Anyway, I have no great answers or as some responder to me a while back said, if I did, I'd "have a Nobel and possibly be the first president of the united planet".
- Instead of trying to get LLMs to answer user questions, write better FAQs informed by reviewing tickets submitted by customers
- Instead of RAG for anything involving business data, have some DBA write a bunch of reports that answer specific business questions
- Instead of putting some copilot chat into tools and telling users to ask it to e.g. "explain recent sales trends", make task-focused wizards and visualizations so users can answer these with hard numbers
- Instead of generating code with LLMs, write more expressive frameworks and libraries that don't require so much plumbing and boilerplate
Of course, maybe there is something I am missing, but these are just my personal observations!
I agree, however, I've seen first hand how the AI fever and mandate from the top has finally busted enough information silos that previously 'have some DBA write a bunch of reports that answer specific business questions' just wasn't feasible in the first place, and now is.
Yeah, I will say it provides a good excuse to leadership & external stakeholders for fixing technical debt and working cross-functionally! Making infrastructure "AI-ready" and all that.
I imagine more of the benefit of AI will come from companies prioritizing infrastructure and effective organization than from the technology itself.
With all due respect, all of those examples are the examples of "yesterday" ... that's how we have been bringing money to businesses for decades, no? Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned.
Businesses are incentivized to be more productive and cost-effective since they are solely profit-driven so they naturally see this as an opportunity to make more money by hiring less people while keeping the amount of work done roughly the same or even more.
So "classical" approach to many of the problems is I think the thing of a past already.
> Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned.
We really don't. There are demos that look cool onstage, but there is a big difference between "in store good" and "at home good" in the sense that products aren't living up to their marketing during actual use.
IMO there is a lot of room to grow within the traditional approaches of "yesterday" - The problem is that large orgs get bogged down in legacy + bureaucracy, and most startups don't understand the business problems well enough to make a better solution. And I don't think that there is any technical silver bullet that can solve either of these problems (AI or otherwise)
I am wondering how often do you use AI models? Because I do it on a daily basis, and as much as they have limitations, I find them to be performing incredibly well. It's far very far from being a demo - last time it was a demo that looked "cool" was around 2020/21 when they were cool for spitting out the haiku poetry, and perhaps 2022 when capabilities were not as good. But today? Completely mind-blowing.
If you're not convinced, I suggest you to search for the law firms, hospitals, and laboratories ... all of which are using AI models as of today to do both the research and boiler-plate work. Creative industries are being literally erased by the generative AI as we are speaking. What will happen with the Photoshop and other similar tools when I can create whatever I want using the free AI model in literally 2 seconds without prior knowledge? What will happen with majority of movie effect makers when single guy will be able to do the work of 5 people at the same time? Or interior designers? The heck, what will happen with the Google search - I anticipate nobody will be using it in a year or two. I already don't because it's a massive sink of time compared to what I can do with perplexity for example.
There's many many examples. You just need to have your mind open to see it.
You're making a ridiculously overconfident statement.
* Show me a discrete manufacturing company using AI models for statistical process control or quality reporting
* Show me a pharmaceutical company using AI models for safety data analysis
* Show me an engineering company using AI models for structural design
The list goes on and on. There are precious few industries or companies that have replaced traditional analysis & prediction with AI. Why? Because one of two things are true: 1) their data is already in highly structured relational stores that have long legacies of SQL-based extraction and analysis, 2) they're in regulated industries and have to have audit-proof, explainable reporting, or 3) they need evidence-based design and analysis that has a key component coming from real people observing real processes in action.
For all the hyped "AI Automation" you read about, there are 100 other things that aren't, or where firms don't believe they can be, or where they'll struggle to for [reasons].
Right right, I get it. Pharma, structural engineering, discrete manufacturing, ..., all of the industries which are "too hard" to be conquered by some stupid statistical parrot. You're being delusional my friend but I am not going to be the one trying to persuade you to believe otherwise. I am here for sharing experiences and interesting discussions from which I can learn and I am not here for combating triggered and defensive strangers on the internet. And FWIW both of your conclusion and premise, and interpretation of my comment is wrong.
I try them from time to time, but I have yet to see AI models produce a useful output for my work. The problems I work on are not well-represented in training data and internet-based resources, and correctness matters far more than speed.
For my work, it's important to form strong and correct mental models of complex systems so I can reason about them well. It's more about thinking and writing clearly than anything else. LLMs tend to include subtle mistakes or even completely incorrect information (and reasoning!) which disrupts this process.
On the creative industry side...well, you can produce some results that look fine by themselves, but producing large-scale cohesive artwork (games, movies, etc.)? It's mostly human elbow-grease for the foreseeable future.
> LLMs tend to include subtle mistakes or even completely incorrect information (and reasoning!) which disrupts this process.
You see, so many humans do that as well but yet we make it believe that LLMs are somehow special here. Yes, they make mistakes, we make mistakes, I make mistakes, your colleague makes mistakes, but that's not the point of this discussion at all and it's a form of confirmation bias.
Why this reflex gag happens, I think, is because people are biased to catch somebody in making the mistake so that they can feel superior and irreplaceable. We do that to keep our position strong (in society, work), and this is completely natural - it's called the survival instinct, and it is present in our species regardless of LLMs. LLMs are just one of the ways we can obviously trigger this condition.
So, your response is no more special than other people combating the AI but take into account that "The problems I work on are not well-represented in training data" or "correctness matters far more than speed" or "it's important to form strong and correct mental models of complex systems so I can reason about them well" makes a great deal of strong assumptions. Almost any domain can literally copy-paste this into their defense, and this is also something interesting I observed through the course of years - every domain I worked in, and there were plenty, each domain thought that it is their domain which is the "hardest". Vanity.
High quality writing (e.g. MDN Web Docs, Go Documentation, internal docs) does not have a tendency to include mistakes or incorrect reasoning because it comes from a place of clear thinking and goes through a continuous process of peer review and improvement.
LLMs outputs are, at best, first drafts that have not been reviewed or revised, and they certainly do not have analytical reasoning behind their content.
I am not claiming that my domain is uniquely challenging, but the statements I made are factual for my work (and likely many adjacent fields, too).
Would you trust your life to a pacemaker or aircraft control system designed and manufactured quickly by people without a correct understanding of what they are developing, working purely off of information from the Internet and other semi-public sources? I wouldn't. But maybe you are braver than me!
Nobody here is talking about the pacemakers nor am I suggesting that AI will be replacing 100% of working force.
While we are at writing documentation, did you hear about the latest layoffs from MySQL which, among others, impacted heavily the documentation team as well cutting their size from 8 to 3 people?
I mean, the software I develop is the system of record for the engineering and development of medical devices (and aerospace, automotive, industrial, etc. systems). So pacemakers and aircraft are relevant examples.
I hadn't heard about the MySQL layoffs. I hope the affected people find good new opportunities.
I tried to give a counter-example of people being replaced in an extremely complex piece of software as we speak. This may or may not be related to AI but I am inclined to think that there is some correlation - yesterday 8 people in the team, today 3, tomorrow who knows, maybe only a single guy.
In my domain, I see lots of people reaching immediately for "AI" techniques to solve sensor fusion and state estimation problems where a traditional Kalman filter type solution would be faster and much more interpretable.
Incidentally, I worked on the exact same thing - Kalman filtering for tracking objects in hard real-time systems. And it is not quite as simple as one would think - developing mathematical models for all kinds of different objects that one might wanna track is far from trivial, and it was difficult to model the real-world with more or less simplistic discrete equations. And it didn't work completely reliably so we needed an extra layer of confidence - I don't remember what we used back then but it was yet another algorithm with yet another source of data.
There's a lot of situations where you genuinely need more sophisticated algorithms. I'm talking about people reaching for those kinds of things before they determine the traditional algorithms won't work.
I thought the OCR was one of the obvious examples where we have a classical technology that is already working very well but in the long-run I don't see it surviving. _Generic_ AI models already can do the OCR kinda good but they are not even trained for that purpose, it's almost incidental - they've never been trained to extract the, let's say name/surname from some sort of a document with a completely unfamiliar structure, but the crazy thing is that it does work somehow! I think that once somebody finetunes the AI model only for this purpose I think there's a good chance it will outperform classical approach in terms of precision and scalability.
Yes, that's correct about OCR. I work as a machine vision engineer in the semiconductor industry, where each wafer usually has both OCR text and machine-readable codes such as barcodes, QR codes, or data matrix codes. The OCR typically uses the SEMI font standard.
To achieve accurate OCR results, I need to preprocess the image by isolating each character, sorting them from left to right, and using regular expressions (regex) to verify the output. However, I prefer machine-readable codes because they are simpler to use, feature built-in error detection, and are much more reliable. While deep-learning OCR solutions often perform well, they cannot guarantee the 100 percent accuracy required in our applications.
This approach is similar to how e-wallet payments use cameras to scan QR codes instead of OCR text, as QR codes provide greater reliability and accuracy.
In general I agree. For OCR I agree vehemently. Part of the reason is the structure of the solution (convolutions) match the space so well.
The failure cases are those where AI solutions have to stay in a continuous debug, train, update mode. Then you have to think about the resources you need, both in terms of people as well as compute to maintain such a solution.
Because of the way the world works, it's endemic nonstationarity, the debug-retrain-update is a common state of affairs even in traditional stats and ML.
I see. Let's take another example here, I hope I understood you - imagine you have an AI model which is connected to all of your company's in-house data generation sources such as wiki, chat, jira, emails, merge requests, excel sheets, etc. Basically everything that can be deemed useful to query or to create business inteligence on top of. These data sources are continously generating more and more data every day, and given their nature they are more or less unstructured.
Yet, we have such systems in place where we don't have to retrain the model with ever-growing data. This is one example I could think of but it kinda suggests that models, at least for some purposes, don't have to be retrained continuously to keep them running well.
I also use a technique of explaining something to the AI model he has not seen before (according to the wrong answer I got from it previously), and it manages to evolve the steps, whatever they are, so that it gives me the correct answer in the end. This also suggests that capacity of the models is larger than what they have been trained on.
Data science solutions are different in the sense they rarely ever get done and dusted in a sense a sorting library might.
There's almost always something or the other breaking. Did the nature of data change. Did my upstream data feed change. Why are these small set of examples not working for this high paying customer.
You would need resources to understand and fix these problems quarter after quarter.
A rich network of data dependencies can be a double edged sword. Rarely are upstream code and data changes benign to the output of the layer you own.
There are two cases where AI solutions are perfect. They are so good that they are fire and forget. The second is that your customer is a farmer not a gardener. Individual failing saplings mean little to him.
If a single misbehaving plant can cause commercially significant damage then when choosing opaque tools you must consider the maintenance cost you may be signing up for.
Say I have a ton of historical data that is being continuously added to. It's a real temptation to replace the raw data with a model that uses less number of parameters than the raw data. In a sense lossy compression. Can be a very bad idea. Data instances where the model does not fit well may be the most important pieces of art information. Model paints with a broad brush stroke. If you are hunting faults, you have been aware that a lossy compression can paper them away. You are also potentially harming a future model that could have been trained but you have thrown away a decade of useful data because storage costs were running so high.
No easy solution. General recommendation would be to compress but losslessly simply because you know not what may be valuable in the future. If it's impossible, then so be it, you have to eat that opportunity cost in the future, but you did your best.
It’s not fundamentally different from cattle vs pets, but I like how it shapes the human agency. There is a problem in your cow/wheat/server -what do *you* do? Can you intentionally ignore it by virtue of large numbers, or stop everything to resolve?
I've seen a lot of uses for SAT solvers, but what do you use them for in data science? I can't find many references to people using them in that context.
My first big career move was similar. Employer had all these complicated ML models that played great at conferences, but operators who had to use them said they were inaccurate and indecipherable. I asked the operators what they used to sort real vs false alarms, and the answers were very simple mathematical relationships. Ripped out the ML, coded up the operators versions with some slightly tuned up statistics and got dramatically better results.
Thank you for your perspective. As a machine vision engineer in the semiconductor industry, I have seen a lot of hype around deep learning and AI for vision applications. From my experience, deep learning works well for OCR but less so for classification tasks.
I often achieve better results by focusing on good lighting and using classical computer vision techniques.
I agree with your point about the politics of technology adoption. To protect my career, I usually promote hybrid approaches that combine deep learning and traditional computer vision methods. In reality, many deep learning solutions still rely heavily on classical techniques. Your comments on political challenges and decision-making in technology are very relevant to my experience.
What confuses me about deep nets is that there's rarely enough signal to be able to meaningfully train a large number of parameters. Surely 99 % of those parameters are either (a) incredibly unstable, or (b) correlate perfectly with other parameters?
They do. There are enormous redundancies. There's a manifold over which the parameters can vary wildly yet do zilch to the output. The nonlinear analogue of a null space.
Parameter instability does not worry a machine learner as much as it worries a statistician. ML folks worry about output instabilities.
The current understanding goes that this overparameterization makes reaching good configurations easier while keeping the search algorithm as simple as stochastic gradient descent.
Huh, I didn't know that! Are there efforts to automatically reduce the number of parameters once the model is trained? Or do the relationships between parameters end up too complicated to do that? I would assume such a reduction would be useful for explainability.
(Asking specifically about time series models and such.)
Selecting which weights to discard seems as hard as the original problem. But random decimation, sometimes barely informed decimation have been observed to be effective.
On the theory side now it's understood that in the thicket of weights, lurk a much much smaller subset that can have nearly the same output.
These observations are for DNNs in general. For time series specifically I don't know what the state of the art is. In general NNs are still catching up with traditional stats approaches in this domain. There are a few examples where traditional approaches have been beaten, but only a few.
One good source to watch are the M series of competitions.
It's fairly standard to prune the hell out of a model for deployment, because many of the parameters end up being close to zero. This doesn't really help with explainability of the parameters, because (imo) that's a dead end. You assume that the data is iid and a representative sample of whatever god-given function generated it, and you throw a universal approximator at it because it's impossible to come up with some a priori function that models the data in the first place.
Latent space clustering is about as good as it gets imo, and in my experience, that's fairly stable for individual implementations (but not necessarily across implementations for the same model, for various reasons), but it doesn't tell you anything about the meaning of the parameters themselves. If the model is well calibrated, you can validate its performance and it becomes explainable as a unit.
If you started with a deep neural network, one can't really use pruning to go all the way down to a parameter count that is directly intepretable (say under 100). One would at least have to try some techniques to get more disentangled representations. But local surrogate models are popular for explainability, see Shap and LIME.
For interpretable time series I would encourage to construct features and transformations the old fashioned way, and then learn it all end to end as a differentiable program. Then you can get the best of both worlds.
> We have successfully replaced thousands of complicated deep net time series based anomaly detectors at a FANG with statistical (nonparametric, semiparametric) process control ones.
Interesting.
Were you using things like Matrix Profile too ? And if so, have those been replaced too ?
Fwiw, I have a masters in operations research as a focus area within an industrial engineering degree, and spent 15 years working in manufacturing systems with a focus on test automation & quality. Traditional SPC/SQC analysis is, and will remain, king -- at least for some time. That can potentially evolve on high-vol/low-mix scenarios that lend themselves more easily to training models on anomaly detection, but especially for complex product manufacturing in high-mix factories that's not the case. It's far better to let your test/quality engineers do their jobs and figure out statistical controls on their own.
Among other reasons, this is largely true because acceptable ranges for different anomaly & defect types can vary significantly for different revs of a single product, or even sub-revs (things that are tied to an ECO but don't result in incrementing the product rev), or -- more crucially -- the line the product is manufactured on. One thing that's notoriously tricky to troubleshoot without being physically onsite is whether a defect is because of a machine, because of a person, or because of faulty piece parts/material.
Understanding and knowing how to apply traditional statistical analysis to these problems -- and also designing useful data structures to store all the data you're collecting -- is far more valuable right now than trying to shoehorn in an AI model to do this work.
Can you be more specific about what SPC algorithm you moved to? Did you trade off prediction quality for complexity, increasing the number of false alarms?
We generally targeted specific statistics of derived/processed streams. For some such streams we cared if the mean changed. In others if the spread changed in a way that was unusual for the time of day. In yet others if some percentile changed that was unusual for the time of day. Sometimes it will be more than one of such statistics.
Then we would track an online estimator of that measure with an SPC chart. The thresholds would be set based on our appetite for false alarms. We did not fit or use properties of parametric distributions that standard SPC charts use. So no 3-sigma business. In our case convergence to Gaussian would often be not fast enough for such techniques to be useful.
Also the original streams were far from IID, temporal dependencies were strong. So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly. This was the most important bit.
The next key aspect was to keep the alerting thresholds as untarnished and unaffected as possible from the outliers that would unavoidably occur. Getting this to work without additional human supervisory labels was the next most important part.
Make this part too robust to outliers then the system would not automatically adapt to a new normal. Make it too sensitive and we would get overwhelmed by false positives.
> So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly
Could you expand on that (or at least point me towards some keywords to read up on)?
I think I understand conceptually what this means (network latency increases at noon CDT each day because YouTube load increases during the lunch hour, as an example) but I'm wondering how you normalize data streams for temporal dependencies with unknown frequencies (nominal change #1 happens each Sunday, while nominal change #2 happens each day at noon).
That's where you need time series models. Take a look at linear filters and how to handle seasonal variations.
It gets interesting when there are different frequencies at play simultaneously, like in your example -- day of the week and time of day effects. If their periods are mutually prime that's more convenient to handle as they are orthogonal in the Fourier space.
Would rather not. Just to be in the compliant zone legally and also to stay somewhat anonymous. Sincerely sorry to disappoint. But let me assure you it was nothing exotic.
Fair. Not sure what it’s like getting tech talks approved through comms these days, but this would be fascinating to hear about at a SF or SouthBay Systems meetup.
Can't for obvious reasons. But no specialized libraries used. The usual Python stack that comes packaged for any respectable OS distribution these days, mixed in with other close-to-the-metal languages for performance or API compatibility reasons.
Look up nonparametric statistical process control and you will find useful papers. The algorithms are actually quite simple to implement. If the algorithms are not simple then probably they are not worth your time. The analysis in the paper might be complicated, don't worry about that, look for simplicity of the algorithms.
Assume you have signal from one IoT device, say a sensor reading. Anomalies are sudden changes in the value of the signal. Define sudden (using the time delta between observations and your other domain knowledge); let's say the sensor reports 1x/second and sudden means 1-3 minutes.
Simple options:
rolling mean last 3 values/rolling mean last 60 values. If this value is over a threshold, alert
Complexer but still simple options. Say you have IoT sensors over a larger area, and an anomaly is one sensor which is higher than others. Run roughly the same analysis as above, but on the correlation matrix of all the sensors. Look for rapidly changing correlations.
example: temperature detectors in each room of your house, and your kid opens the front door to go play in the snow. The entry hall cools down while the rest of the house's temp stays roughly stable. You can picture what that does to the correlation matrix.
It was little more complicated to remove temporal dependencies from the original streams and we could not rely on Gaussian behaviour. Other than that, it's pretty much the same, barring an effort to keep the alerting thresholds unaffected by recent anomalies.
I'm old enough that I got my Lean Six Sigma Green Belt certification and used Minitab (!!!) to do a statistical process control project on some investment banking back office process that was being automated.
Does anyone here even remember Minitab? You kids and your newfangled Python!
I got a six sigma green belt using minitab in ~2018. It's still in use and getting updates, for example they now provide an interface to python: https://pypi.org/project/mtbpy/
For the longest time, open source solutions were incomplete in the sense that all of them did x-bar/S/R and then usually never got to the more esoteric but handy stuff. Multivariate, even less support.
Classical stats is still bread and butter for lot of smallish dataset in clinical datasets.
It is hard to do machine learning or even regression on some very preliminary data. Metadata is tough to collect and harmonize so it becomes hard to integrate specially with human studies with rare diseases.
In the real world, data is never this clean. Majority of the time is data quality work because you will see outliers that might be due to measurement error, calibration, process changes. It requires familiarity with the process and having an understanding and intuition for why the shape of the data distribution in a process is the way it is. Because ad-hoc data visualization and exploration is critical here, enterprise requirements need mature tools that can be used quickly. BYOT and DIY code for an SPC Cpk chart is not what you want to be doing.
Fun memories.
We have successfully replaced thousands of complicated deep net time series based anomaly detectors at a FANG with statistical (nonparametric, semiparametric) process control ones.
They use 3 to 4 orders lower number of trained parameters and have just enough complexity that a team of 3 or four can handle several thousands of such streams.
The amount of baby sitting that deep net models needed was astronomical, debugging and understanding what has happened quite opaque.
For small teams, with limited resources I would still heavily recommend stats based models for time series anomaly detection.
May not be your best career move right now for political reasons. Those making massive bets do not like to confront that some of their bets might not have been well placed. They may try to make it difficult for contrary evidence to become too visible.
Super cool, thanks for sharing!
This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.
You'd think that rational businesses would take the low-risk snooze-fest high-margin option any day instead of unintelligible and unreliable options that demand a lot of resources, and yet...
>This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.
In 2013 my statistics professor warned that once we are in the real world, "people will come up to you trying to sell fancy machine learning models for big money, though the simple truth is that many problems can be solved better by applying straightforward statistical methods".
There has always been the ML hype, but the last couple years are a whole different level.
It does not work that way in the short term.
Say you have bet billions as a CEO, CTO, CFO. The decision has already been made. Such a steep price had to come at the cost of many groups and teams and projects in the company.
Now is not a time to water plants that offer alternatives. You will have a smoother ride choosing tools that justifies that billion dollar bet.
Decision-making in organizations is definitely a hard problem.
I think an uncomfortable reality is that a lot of decisions (technology, strategy, etc.) are not optimal or even rational, but more just an outcome of personal preferences.
Even data-driven approaches aren't immune since they depend on the analysis and interpretation of the data (which is subjective).
Data informed is good. Purely data driven is a bad idea.
After all even in Physics big advances came from thought experiments. Data is one way to reason about a decision, logic and knowledgebase is another way. Both can be very powerful if one retains the humility of fallibility.
In organizations one common failure mode is that the organisational level at which decisions are made are not the same levels where the decisions are going to have their effects felt.
It's a really difficult problem to solve. Too much decentralisation is also a bad idea. You get the mess of unplanned congested cities.
For a while now, I've been summarizing the ease with which everything turns into a "Humanity Complete" problem via: "Delegation affords so much, but trust sure is tricky."
This has been observed forever in various forms/contexts. Planning & policy people call them "Wicked Problems" (https://en.wikipedia.org/wiki/Wicked_problem). The Philosophy of Science one goes by the Demarcation Problem (https://en.wikipedia.org/wiki/Demarcation_problem) { roughly, in the sense that the really hard nugget connects to "trust" }.
At least one aspect of all of it is that trust is a little like money/capital and "faking it" is a bit like "stealing". The game theory of it is that since faking is virtually always vastly cheaper there are (eventually) huge incentives to do so, at some point by someone(s). So, almost any kind of trust/delegation structure has a strong pull toward "decay", from knock-off brands to whatever. It just takes a sadly small fraction of Prisoner's Dilemma defectors to ruin things/systems thereof. 2nd law of thermo makes order cost energy and this decay feels like almost an isomorphic (maybe even the same..?) thing. It's not just product/tech enshittification, but that might be yet another special case/example.
Anyway, I have no great answers or as some responder to me a while back said, if I did, I'd "have a Nobel and possibly be the first president of the united planet".
> There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.
I know some examples but not too many. Care to share more examples?
Some off the top of my head...
- Instead of trying to get LLMs to answer user questions, write better FAQs informed by reviewing tickets submitted by customers
- Instead of RAG for anything involving business data, have some DBA write a bunch of reports that answer specific business questions
- Instead of putting some copilot chat into tools and telling users to ask it to e.g. "explain recent sales trends", make task-focused wizards and visualizations so users can answer these with hard numbers
- Instead of generating code with LLMs, write more expressive frameworks and libraries that don't require so much plumbing and boilerplate
Of course, maybe there is something I am missing, but these are just my personal observations!
I agree, however, I've seen first hand how the AI fever and mandate from the top has finally busted enough information silos that previously 'have some DBA write a bunch of reports that answer specific business questions' just wasn't feasible in the first place, and now is.
Yeah, I will say it provides a good excuse to leadership & external stakeholders for fixing technical debt and working cross-functionally! Making infrastructure "AI-ready" and all that.
I imagine more of the benefit of AI will come from companies prioritizing infrastructure and effective organization than from the technology itself.
With all due respect, all of those examples are the examples of "yesterday" ... that's how we have been bringing money to businesses for decades, no? Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned.
Businesses are incentivized to be more productive and cost-effective since they are solely profit-driven so they naturally see this as an opportunity to make more money by hiring less people while keeping the amount of work done roughly the same or even more.
So "classical" approach to many of the problems is I think the thing of a past already.
> Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned.
We really don't. There are demos that look cool onstage, but there is a big difference between "in store good" and "at home good" in the sense that products aren't living up to their marketing during actual use.
IMO there is a lot of room to grow within the traditional approaches of "yesterday" - The problem is that large orgs get bogged down in legacy + bureaucracy, and most startups don't understand the business problems well enough to make a better solution. And I don't think that there is any technical silver bullet that can solve either of these problems (AI or otherwise)
I am wondering how often do you use AI models? Because I do it on a daily basis, and as much as they have limitations, I find them to be performing incredibly well. It's far very far from being a demo - last time it was a demo that looked "cool" was around 2020/21 when they were cool for spitting out the haiku poetry, and perhaps 2022 when capabilities were not as good. But today? Completely mind-blowing.
If you're not convinced, I suggest you to search for the law firms, hospitals, and laboratories ... all of which are using AI models as of today to do both the research and boiler-plate work. Creative industries are being literally erased by the generative AI as we are speaking. What will happen with the Photoshop and other similar tools when I can create whatever I want using the free AI model in literally 2 seconds without prior knowledge? What will happen with majority of movie effect makers when single guy will be able to do the work of 5 people at the same time? Or interior designers? The heck, what will happen with the Google search - I anticipate nobody will be using it in a year or two. I already don't because it's a massive sink of time compared to what I can do with perplexity for example.
There's many many examples. You just need to have your mind open to see it.
You're making a ridiculously overconfident statement.
* Show me a discrete manufacturing company using AI models for statistical process control or quality reporting
* Show me a pharmaceutical company using AI models for safety data analysis
* Show me an engineering company using AI models for structural design
The list goes on and on. There are precious few industries or companies that have replaced traditional analysis & prediction with AI. Why? Because one of two things are true: 1) their data is already in highly structured relational stores that have long legacies of SQL-based extraction and analysis, 2) they're in regulated industries and have to have audit-proof, explainable reporting, or 3) they need evidence-based design and analysis that has a key component coming from real people observing real processes in action.
For all the hyped "AI Automation" you read about, there are 100 other things that aren't, or where firms don't believe they can be, or where they'll struggle to for [reasons].
Right right, I get it. Pharma, structural engineering, discrete manufacturing, ..., all of the industries which are "too hard" to be conquered by some stupid statistical parrot. You're being delusional my friend but I am not going to be the one trying to persuade you to believe otherwise. I am here for sharing experiences and interesting discussions from which I can learn and I am not here for combating triggered and defensive strangers on the internet. And FWIW both of your conclusion and premise, and interpretation of my comment is wrong.
I try them from time to time, but I have yet to see AI models produce a useful output for my work. The problems I work on are not well-represented in training data and internet-based resources, and correctness matters far more than speed.
For my work, it's important to form strong and correct mental models of complex systems so I can reason about them well. It's more about thinking and writing clearly than anything else. LLMs tend to include subtle mistakes or even completely incorrect information (and reasoning!) which disrupts this process.
On the creative industry side...well, you can produce some results that look fine by themselves, but producing large-scale cohesive artwork (games, movies, etc.)? It's mostly human elbow-grease for the foreseeable future.
> LLMs tend to include subtle mistakes or even completely incorrect information (and reasoning!) which disrupts this process.
You see, so many humans do that as well but yet we make it believe that LLMs are somehow special here. Yes, they make mistakes, we make mistakes, I make mistakes, your colleague makes mistakes, but that's not the point of this discussion at all and it's a form of confirmation bias.
Why this reflex gag happens, I think, is because people are biased to catch somebody in making the mistake so that they can feel superior and irreplaceable. We do that to keep our position strong (in society, work), and this is completely natural - it's called the survival instinct, and it is present in our species regardless of LLMs. LLMs are just one of the ways we can obviously trigger this condition.
So, your response is no more special than other people combating the AI but take into account that "The problems I work on are not well-represented in training data" or "correctness matters far more than speed" or "it's important to form strong and correct mental models of complex systems so I can reason about them well" makes a great deal of strong assumptions. Almost any domain can literally copy-paste this into their defense, and this is also something interesting I observed through the course of years - every domain I worked in, and there were plenty, each domain thought that it is their domain which is the "hardest". Vanity.
High quality writing (e.g. MDN Web Docs, Go Documentation, internal docs) does not have a tendency to include mistakes or incorrect reasoning because it comes from a place of clear thinking and goes through a continuous process of peer review and improvement.
LLMs outputs are, at best, first drafts that have not been reviewed or revised, and they certainly do not have analytical reasoning behind their content.
I am not claiming that my domain is uniquely challenging, but the statements I made are factual for my work (and likely many adjacent fields, too).
Would you trust your life to a pacemaker or aircraft control system designed and manufactured quickly by people without a correct understanding of what they are developing, working purely off of information from the Internet and other semi-public sources? I wouldn't. But maybe you are braver than me!
Nobody here is talking about the pacemakers nor am I suggesting that AI will be replacing 100% of working force.
While we are at writing documentation, did you hear about the latest layoffs from MySQL which, among others, impacted heavily the documentation team as well cutting their size from 8 to 3 people?
I mean, the software I develop is the system of record for the engineering and development of medical devices (and aerospace, automotive, industrial, etc. systems). So pacemakers and aircraft are relevant examples.
I hadn't heard about the MySQL layoffs. I hope the affected people find good new opportunities.
I tried to give a counter-example of people being replaced in an extremely complex piece of software as we speak. This may or may not be related to AI but I am inclined to think that there is some correlation - yesterday 8 people in the team, today 3, tomorrow who knows, maybe only a single guy.
In my domain, I see lots of people reaching immediately for "AI" techniques to solve sensor fusion and state estimation problems where a traditional Kalman filter type solution would be faster and much more interpretable.
Incidentally, I worked on the exact same thing - Kalman filtering for tracking objects in hard real-time systems. And it is not quite as simple as one would think - developing mathematical models for all kinds of different objects that one might wanna track is far from trivial, and it was difficult to model the real-world with more or less simplistic discrete equations. And it didn't work completely reliably so we needed an extra layer of confidence - I don't remember what we used back then but it was yet another algorithm with yet another source of data.
There's a lot of situations where you genuinely need more sophisticated algorithms. I'm talking about people reaching for those kinds of things before they determine the traditional algorithms won't work.
Sigma point filters ? The escalation ladder is usually KF, EKF, unscented KF, sigma point ...
I believe it was random forest that we combined with EKF.
In the realm of data science, Linear models and SAT solvers used cleverly will get you a surprisingly long way.
I thought the OCR was one of the obvious examples where we have a classical technology that is already working very well but in the long-run I don't see it surviving. _Generic_ AI models already can do the OCR kinda good but they are not even trained for that purpose, it's almost incidental - they've never been trained to extract the, let's say name/surname from some sort of a document with a completely unfamiliar structure, but the crazy thing is that it does work somehow! I think that once somebody finetunes the AI model only for this purpose I think there's a good chance it will outperform classical approach in terms of precision and scalability.
Yes, that's correct about OCR. I work as a machine vision engineer in the semiconductor industry, where each wafer usually has both OCR text and machine-readable codes such as barcodes, QR codes, or data matrix codes. The OCR typically uses the SEMI font standard.
To achieve accurate OCR results, I need to preprocess the image by isolating each character, sorting them from left to right, and using regular expressions (regex) to verify the output. However, I prefer machine-readable codes because they are simpler to use, feature built-in error detection, and are much more reliable. While deep-learning OCR solutions often perform well, they cannot guarantee the 100 percent accuracy required in our applications.
This approach is similar to how e-wallet payments use cameras to scan QR codes instead of OCR text, as QR codes provide greater reliability and accuracy.
In general I agree. For OCR I agree vehemently. Part of the reason is the structure of the solution (convolutions) match the space so well.
The failure cases are those where AI solutions have to stay in a continuous debug, train, update mode. Then you have to think about the resources you need, both in terms of people as well as compute to maintain such a solution.
Because of the way the world works, it's endemic nonstationarity, the debug-retrain-update is a common state of affairs even in traditional stats and ML.
I see. Let's take another example here, I hope I understood you - imagine you have an AI model which is connected to all of your company's in-house data generation sources such as wiki, chat, jira, emails, merge requests, excel sheets, etc. Basically everything that can be deemed useful to query or to create business inteligence on top of. These data sources are continously generating more and more data every day, and given their nature they are more or less unstructured.
Yet, we have such systems in place where we don't have to retrain the model with ever-growing data. This is one example I could think of but it kinda suggests that models, at least for some purposes, don't have to be retrained continuously to keep them running well.
I also use a technique of explaining something to the AI model he has not seen before (according to the wrong answer I got from it previously), and it manages to evolve the steps, whatever they are, so that it gives me the correct answer in the end. This also suggests that capacity of the models is larger than what they have been trained on.
Data science solutions are different in the sense they rarely ever get done and dusted in a sense a sorting library might.
There's almost always something or the other breaking. Did the nature of data change. Did my upstream data feed change. Why are these small set of examples not working for this high paying customer.
You would need resources to understand and fix these problems quarter after quarter.
A rich network of data dependencies can be a double edged sword. Rarely are upstream code and data changes benign to the output of the layer you own.
There are two cases where AI solutions are perfect. They are so good that they are fire and forget. The second is that your customer is a farmer not a gardener. Individual failing saplings mean little to him.
If a single misbehaving plant can cause commercially significant damage then when choosing opaque tools you must consider the maintenance cost you may be signing up for.
Say I have a ton of historical data that is being continuously added to. It's a real temptation to replace the raw data with a model that uses less number of parameters than the raw data. In a sense lossy compression. Can be a very bad idea. Data instances where the model does not fit well may be the most important pieces of art information. Model paints with a broad brush stroke. If you are hunting faults, you have been aware that a lossy compression can paper them away. You are also potentially harming a future model that could have been trained but you have thrown away a decade of useful data because storage costs were running so high.
No easy solution. General recommendation would be to compress but losslessly simply because you know not what may be valuable in the future. If it's impossible, then so be it, you have to eat that opportunity cost in the future, but you did your best.
Never heard the farmer vs gardener framing before, but I love it. Can classify so many business problems like this.
Oh well, was just dipping into the delights of some old Martin Gardner. Amazing how the human brain works.
It’s not fundamentally different from cattle vs pets, but I like how it shapes the human agency. There is a problem in your cow/wheat/server -what do *you* do? Can you intentionally ignore it by virtue of large numbers, or stop everything to resolve?
I've seen a lot of uses for SAT solvers, but what do you use them for in data science? I can't find many references to people using them in that context.
Root causing from symptoms is one case where SAT or their ML analogue -- graphical models are quite useful.
>unintelligible and unreliable options that demand a lot of resources
Some options have more persuasive salesmen than others.
My first big career move was similar. Employer had all these complicated ML models that played great at conferences, but operators who had to use them said they were inaccurate and indecipherable. I asked the operators what they used to sort real vs false alarms, and the answers were very simple mathematical relationships. Ripped out the ML, coded up the operators versions with some slightly tuned up statistics and got dramatically better results.
Thank you for your perspective. As a machine vision engineer in the semiconductor industry, I have seen a lot of hype around deep learning and AI for vision applications. From my experience, deep learning works well for OCR but less so for classification tasks.
I often achieve better results by focusing on good lighting and using classical computer vision techniques.
I agree with your point about the politics of technology adoption. To protect my career, I usually promote hybrid approaches that combine deep learning and traditional computer vision methods. In reality, many deep learning solutions still rely heavily on classical techniques. Your comments on political challenges and decision-making in technology are very relevant to my experience.
Yes hybrid and ensembles would be a good way to handle this.
What confuses me about deep nets is that there's rarely enough signal to be able to meaningfully train a large number of parameters. Surely 99 % of those parameters are either (a) incredibly unstable, or (b) correlate perfectly with other parameters?
They do. There are enormous redundancies. There's a manifold over which the parameters can vary wildly yet do zilch to the output. The nonlinear analogue of a null space.
Parameter instability does not worry a machine learner as much as it worries a statistician. ML folks worry about output instabilities.
The current understanding goes that this overparameterization makes reaching good configurations easier while keeping the search algorithm as simple as stochastic gradient descent.
Huh, I didn't know that! Are there efforts to automatically reduce the number of parameters once the model is trained? Or do the relationships between parameters end up too complicated to do that? I would assume such a reduction would be useful for explainability.
(Asking specifically about time series models and such.)
What you are looking for is the lottery ticket hypothesis for neural networks. Hit a search engine with those words you will find examples.
https://arxiv.org/abs/1803.03635 ( you can follow up on semantic scholar for more)
Selecting which weights to discard seems as hard as the original problem. But random decimation, sometimes barely informed decimation have been observed to be effective.
On the theory side now it's understood that in the thicket of weights, lurk a much much smaller subset that can have nearly the same output.
These observations are for DNNs in general. For time series specifically I don't know what the state of the art is. In general NNs are still catching up with traditional stats approaches in this domain. There are a few examples where traditional approaches have been beaten, but only a few.
One good source to watch are the M series of competitions.
It's fairly standard to prune the hell out of a model for deployment, because many of the parameters end up being close to zero. This doesn't really help with explainability of the parameters, because (imo) that's a dead end. You assume that the data is iid and a representative sample of whatever god-given function generated it, and you throw a universal approximator at it because it's impossible to come up with some a priori function that models the data in the first place.
Latent space clustering is about as good as it gets imo, and in my experience, that's fairly stable for individual implementations (but not necessarily across implementations for the same model, for various reasons), but it doesn't tell you anything about the meaning of the parameters themselves. If the model is well calibrated, you can validate its performance and it becomes explainable as a unit.
If you started with a deep neural network, one can't really use pruning to go all the way down to a parameter count that is directly intepretable (say under 100). One would at least have to try some techniques to get more disentangled representations. But local surrogate models are popular for explainability, see Shap and LIME. For interpretable time series I would encourage to construct features and transformations the old fashioned way, and then learn it all end to end as a differentiable program. Then you can get the best of both worlds.
> We have successfully replaced thousands of complicated deep net time series based anomaly detectors at a FANG with statistical (nonparametric, semiparametric) process control ones.
Interesting.
Were you using things like Matrix Profile too ? And if so, have those been replaced too ?
Fwiw, I have a masters in operations research as a focus area within an industrial engineering degree, and spent 15 years working in manufacturing systems with a focus on test automation & quality. Traditional SPC/SQC analysis is, and will remain, king -- at least for some time. That can potentially evolve on high-vol/low-mix scenarios that lend themselves more easily to training models on anomaly detection, but especially for complex product manufacturing in high-mix factories that's not the case. It's far better to let your test/quality engineers do their jobs and figure out statistical controls on their own.
Among other reasons, this is largely true because acceptable ranges for different anomaly & defect types can vary significantly for different revs of a single product, or even sub-revs (things that are tied to an ECO but don't result in incrementing the product rev), or -- more crucially -- the line the product is manufactured on. One thing that's notoriously tricky to troubleshoot without being physically onsite is whether a defect is because of a machine, because of a person, or because of faulty piece parts/material.
Understanding and knowing how to apply traditional statistical analysis to these problems -- and also designing useful data structures to store all the data you're collecting -- is far more valuable right now than trying to shoehorn in an AI model to do this work.
In this specific project no, but in others a very emphatic yes.
Can you be more specific about what SPC algorithm you moved to? Did you trade off prediction quality for complexity, increasing the number of false alarms?
We generally targeted specific statistics of derived/processed streams. For some such streams we cared if the mean changed. In others if the spread changed in a way that was unusual for the time of day. In yet others if some percentile changed that was unusual for the time of day. Sometimes it will be more than one of such statistics.
Then we would track an online estimator of that measure with an SPC chart. The thresholds would be set based on our appetite for false alarms. We did not fit or use properties of parametric distributions that standard SPC charts use. So no 3-sigma business. In our case convergence to Gaussian would often be not fast enough for such techniques to be useful.
Also the original streams were far from IID, temporal dependencies were strong. So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly. This was the most important bit.
The next key aspect was to keep the alerting thresholds as untarnished and unaffected as possible from the outliers that would unavoidably occur. Getting this to work without additional human supervisory labels was the next most important part.
Make this part too robust to outliers then the system would not automatically adapt to a new normal. Make it too sensitive and we would get overwhelmed by false positives.
> So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly
Could you expand on that (or at least point me towards some keywords to read up on)?
I think I understand conceptually what this means (network latency increases at noon CDT each day because YouTube load increases during the lunch hour, as an example) but I'm wondering how you normalize data streams for temporal dependencies with unknown frequencies (nominal change #1 happens each Sunday, while nominal change #2 happens each day at noon).
That's where you need time series models. Take a look at linear filters and how to handle seasonal variations.
It gets interesting when there are different frequencies at play simultaneously, like in your example -- day of the week and time of day effects. If their periods are mutually prime that's more convenient to handle as they are orthogonal in the Fourier space.
Think of these as radio AM modulation.
This sounds fascinating. Can you say anything about the application?
Autoscaling? Data center cooling and power use?
Would rather not. Just to be in the compliant zone legally and also to stay somewhat anonymous. Sincerely sorry to disappoint. But let me assure you it was nothing exotic.
Fair. Not sure what it’s like getting tech talks approved through comms these days, but this would be fascinating to hear about at a SF or SouthBay Systems meetup.
could you give a brief overview of: - what libs were you using - what kind of algos / models were most useful for what kind of data?
I have an IoT use-case, I wanted to look both at NNs and more classical stats models to see if it has value
Can't for obvious reasons. But no specialized libraries used. The usual Python stack that comes packaged for any respectable OS distribution these days, mixed in with other close-to-the-metal languages for performance or API compatibility reasons.
Look up nonparametric statistical process control and you will find useful papers. The algorithms are actually quite simple to implement. If the algorithms are not simple then probably they are not worth your time. The analysis in the paper might be complicated, don't worry about that, look for simplicity of the algorithms.
did similar work at similar scale to srean.
Assume you have signal from one IoT device, say a sensor reading. Anomalies are sudden changes in the value of the signal. Define sudden (using the time delta between observations and your other domain knowledge); let's say the sensor reports 1x/second and sudden means 1-3 minutes.
Simple options: rolling mean last 3 values/rolling mean last 60 values. If this value is over a threshold, alert
Say the readings are normally distributed, or they can be detrended/made normal via a simple 1 or 2 stage AR/MA model. Apply the https://en.wikipedia.org/wiki/Western_Electric_rules to detect anomalies.
Complexer but still simple options. Say you have IoT sensors over a larger area, and an anomaly is one sensor which is higher than others. Run roughly the same analysis as above, but on the correlation matrix of all the sensors. Look for rapidly changing correlations.
example: temperature detectors in each room of your house, and your kid opens the front door to go play in the snow. The entry hall cools down while the rest of the house's temp stays roughly stable. You can picture what that does to the correlation matrix.
Bang on.
It was little more complicated to remove temporal dependencies from the original streams and we could not rely on Gaussian behaviour. Other than that, it's pretty much the same, barring an effort to keep the alerting thresholds unaffected by recent anomalies.
I'm old enough that I got my Lean Six Sigma Green Belt certification and used Minitab (!!!) to do a statistical process control project on some investment banking back office process that was being automated.
Does anyone here even remember Minitab? You kids and your newfangled Python!
I got a six sigma green belt using minitab in ~2018. It's still in use and getting updates, for example they now provide an interface to python: https://pypi.org/project/mtbpy/
Minitab had a good SPC toolbox back in the day.
For the longest time, open source solutions were incomplete in the sense that all of them did x-bar/S/R and then usually never got to the more esoteric but handy stuff. Multivariate, even less support.
I had the privilege of working at a JMP shop! It still gives me the warm and fuzzies.
My uni had its manual in its library is all I can say :)
For beginners to SPC, I wrote a practitioner's guide a couple of years ago. Might provide meaningful context: https://entropicthoughts.com/statistical-process-control-a-p...
Classical stats is still bread and butter for lot of smallish dataset in clinical datasets. It is hard to do machine learning or even regression on some very preliminary data. Metadata is tough to collect and harmonize so it becomes hard to integrate specially with human studies with rare diseases.
In the real world, data is never this clean. Majority of the time is data quality work because you will see outliers that might be due to measurement error, calibration, process changes. It requires familiarity with the process and having an understanding and intuition for why the shape of the data distribution in a process is the way it is. Because ad-hoc data visualization and exploration is critical here, enterprise requirements need mature tools that can be used quickly. BYOT and DIY code for an SPC Cpk chart is not what you want to be doing.
SPC is so great! Far simpler than many other toolsets that don’t work well.
Many type errors
On a sidenote, love the look and feel of your page!!