"For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful.
This can’t be said enough. “Good for Agents” just means self-documenting, obvious ergonomics, save defaults, succinct (or controllable) output, programable interfaces, … all of which support human users too!
anything more than their own handpicked examples and you're better off using d3 or yfiles. layering, clustering, boundaries, rearranging are all basic needs for text to diagrams. None support it.
Both suck at being any good for rendering diagrams from readable structured text. There is a gap to be addressed.
yes and that led me to a more fundamental question : is it even possible for an easy way to do fine grained adjustments. The finer granularity brings complexity. that makes it unreadable (especially for the untrained) and hence set aside and forgotten. The other way is to narrow down and focus on a well defined subset of problems.
either way the number of people willing or compelled to learn it will be tiny. and it hence becomes a niche, perfect for a long-term side project but with no real return.
my conclusion was to stop looking and use D3 or custom code ( good looking charts for humans).
this project is trying to do the same, balance verbosity in text with granularity in the charts with a narrowish usecase : agent consumption.
That's the best option for now. But it can also be frustrating to ask AI to do small edits on D3 just to fix some idioms (like switching order etc) and they kept messing up with other stuff accidentally. Thus I still believe have a language with native representation for these diagram concepts would be helpful.
Good news is that AI do make new language a bit more accessible than before! If your agents can use it well and you can steer easily, it will naturally be good adoption.
Why can’t AIs generate for the “existing, mature ones?” Like the other commenter said, I’m not sure I get the “this is totally for AI” marketing. Why can’t AI use the existing ones and why can’t humans use this?
the design here has some constructs (i.e., semantic types) that AI can use better than human at generation time; and then the generated spec can be easy for user to edit since there is no need for hard-coded low-level parameters!
Thank you for summarizing it this way. All their flowery language (esp from OP) seems like long for "we figure out how the chart should look based on its data". From their page:
> Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout, the Flint compiler derives optimized chart settings from the data, semantic types, chart type, and encodings.
I work with someone who did a lot of work with this to improve our ability to generate awesome visualizations with little thinking. It's a very powerful language but needs guardrails and guidance, particularly if you want end users to be able to produce consistent and standardized visualizations without knowing anything about it.
When I first saw Claude generating PPT decks by writing Python code instead of making the XML directly, it was sort of an "aha moment" for me. This seems to be the path for many things. It also feels slightly limiting, and like a hack LONG term, but 100% correct approach for a while.
A well designed intermediary enables both validation and control over the output independent of the AI. This changes the interaction model between human and AI from delegation to collaboration.
also user interaction afterwards -- if can be frustrating if the only way the user can interact with the chart is to chat with the agent again (simple spec allows easy UI interaction!)
I'm all in on this idea. Every piece of agentic coding I have done in the last month has been via an intermediate representation. Iteration is done in the IR layer mainly. It's remarkable how close you can get to a deterministic coding output using this methodology.
Yes, this has been the pattern for agentic systems since the beginning: permissive generation, that retries over and over until it gets the right size shape through the hole, and the input validates.
Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents.
Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.
So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.
I'm sorry, but as someone who creates data visualisation as a big part of my job, I wouldn't say the charts on the website look good. Most aren't awful either, but by no means are they an improvement over what I'd get by telling any coding agent to make a chart with Vega-Lite or Observable Plot, and probably worse than if I had some decent instructions/skills.
I don't quite get what the goal of this is other than abstracting away a little bit of the complexity at the expense of flexibility. To me, the promise of LLMs is the opposite, I can get flexibility and customisation without the cost of complexity.
Some composite charts are quite annoying to be generated well (like bullet, waterfall etc), their Vega-Lite equivalent can be quite long if just starting from scratch.
The intention here is that Flint is a simpler abstraction to get basic setups right and any followup edits can be done on top of the first compiled outputs (thus not limiting expressiveness). It also makes it easier for user to manipulate (like swapping axes, click to change something, which can be very hard if LLM generates a complex chart spec upfront).
But for many basic stuff your intuition is completely right.
That's fair, I generally make charts for publication, so I spend much more time and effort on the details. But I can understand this being useful for quick exploration for some people.
Generally speaking, I suggest anyone interested in learning to make charts get familiar with grammar of graphics [0] libraries like Vega-Lite, Observable Plot, ggplot2, Altair. There is a bit of a learning curve if you're used to selecting chart types like in Excel, but once it clicks, it gives you virtually unlimited choices in the kinds of charts you can make. And with ggplot or Observable Plot [1], it's about the same number of lines as something like Flint.
Grammar of graphics has been the foundation of a lot of stuff and definitely worth learning for everyone!
A challenge with GoG is that it assumes configurations as second-class stuff, which makes it quite difficult for users to deal with things like changing formatter, scale, annotations. Flint kinda want to hide this aways (so Flint sets them on behalf of the agent or the user). But yeah, GoG is still the foundation for expressivenss.
Can you elaborate on what you mean? Why is it more difficult to deal with formatters, scales, annotations compared with other solutions? Unless I'm misunderstanding something, the defaults are similar to what you would get in Flint, and if you want to add or customise anything, it's usually just one extra line. That's kind of the entire point of the grammar of graphics.
The "how it works" section explains a little bit of this. For example, for the heatmap example showing temporal data, for a "good-looking" chart, we need to (1) reconcile the conflict between banded discrete steps and continuous temporal axis, and it requires understanding and setting stepsize and time parser, (2) for the correlation color, we need to set domain etc under the color axis.
These are supposed to be handled automatically as system defaults, but the tricky part is that these decisions are "semantical", thus requires us to understand the data and design principles, thus existing languages won't stretch that far. And the actual good looking spec is actually over 40 lines of json spec with many low-level paramters, way beyond the simple 5 line encoding promised by GoG. Flint uses semantic type and a layout optimization algorithm to handle this, so 5 lines of encoding + data semantic types can derive rest parameters automatically.
Some examples in the gallery are more extreme: like the waterfall chart example is way over 100 lines of code, and sunbusrt, rose chart are even more since compositions are quite difficult in GoG.
Glad to have a discussion on this level! In fact, we wrote a paper about this, will be putting it online in a week or so!
The paper's line of reasoning seems to continue the endless subjective loop of assuming your viz framework has the right abstractions & defaults , which the next person will rightfully disagree with for their slightly different eval set
We found in practice:
- LLM's generate charts fine
- LLM's tweak charts fine
- LLM's take user feedback to tweak them fine
In that sense, going higher-level for abstractions, as is being argued for here, is strictly worse: it's better to give controls so the LLM can go deep and customize
In practice, we found the choice of json config language X vs json config language Y to be pretty equivalent across different charting systems (vega, plotly, perspective, etc), LLM's do them all fine
The harder parts were deciding what a good chart is (model, reasoning, context), and opposite of this approach, giving lower-level facility for doing user change requests on tweaks, interactivity, and tricky in practice, when they have a lot of data on it.
You are absolutely right. But note that we are actually on the same point here.
This is exactly why this is an intermediate language designed to get 95% stuff right easily (for expressiveness and reliability purpose), while 5% of more advanced case where the agents need to revise chart for other purpose can be done easily on top of the compiled low-level spec (low in terms of Vega-Lite etc, not SVG). We are not really designing a higher abstraction to replace existing ones.
In the past, the split is like 50% good at first run for some common stuff, all other stuff requires agent-loop or user involvement.
Our goal is to make it easy for most case, not everything needs a full multi-round trip agentic workflow to solve. :)
We are kinda all advanced users in fact, for a lot of users, they are easily get confused with the first time result if that is not as good, and the interactivity cost / multi-round isn't an option.
The interview is with Frank Elavsky who seems to be a rock star in the field (and no, I don’t know him and am not him) and made a project called Chartability relating to this which has heuristics, principles, and guidelines for a11y audits:
Thanks a lot for the suggestion, I do think we will need to work on accessibility support in Flint, this is a perfect place to centrally handle accessibility issues.
> simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability
N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?
A simpler spec can be used by a simpler agent. So, maybe that's the use-case here... use by smaller/cheaper agents that run in parallel as opposed to large models running one visualization at a time.
Or at least, maybe that's the idea?
IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.
we are considering also reliability, interactivity besides expressiveness. Simpler spec with good expressiveness comes handy when you want the agent to be reliably for non-expert users and with small models.
>Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout.
Ok, Microsoft is conflating two different things here: LLMs don't really care about code being low level and verbose, they can read things like Assembly and SPIR-V just fine: visualization is the real issue in that LLMs have no natural understanding of spatial composition through visual comparison because they literally "see" things differently than humans, so the way to get around that is provide them with "visualization" in code form that they can easily reason about and understand, so basically anything that's not deeply nested and has hidden states that they have to reason about.
Also, Flint being stringly typed in JSON is a decision that I don't think I agree with. Looking at the actual spec, this could have just been a normal, human usable TypeScript library, and it would have been 100x better. Using their own example (excuse the formatting):
type SemanticType = "Category" | "YearMonth" | "Profit";
type ChartType = "Heatmap" | "BarChart" | "LineChart" | "ScatterPlot"; // extend as needed
// The actual typed object literal:
const chartConfig: ChartConfig = {
data: {}, // replace with your actual data shape/type
semantic_types: {
game: "Category",
period: "YearMonth",
newUsers: "Profit",
},
chart_spec: {
chartType: "Heatmap",
encodings: {
x: "period",
y: "game",
color: "newUsers",
},
chartProperties: {
colorScheme: "redblue",
},
},
};
EDIT:
Went and actually looked at the source instead of just eyeballing it from the docs, and it was a lot more complete and sophisticated than my assumed mockup already.
Core complaint (string-keyed JSON vs. a real generic authoring surface) still stands, but the specific types I posted aren't what Flint has. My bad.
I do find the chartType part is not quite elegant, since templates should be more extensible. We will need to fix that.
For other parts, it's quite common in visualization and diagram etc libraries to have json, since they are easily portable in different rendering contexts.
I'm sure you know that JSON are object literals in TypeScript, there isn't really even a serialization process that's needed there. The AST/IR can still be in JSON, but the authoring surface can be a restricted subset that does not allow logic to execute, that way you can still get the type safety and functionality of TS when you need conditional/loops/logic without throwing away what TS already gives you.
I mean, it would be great if you guys would have like a "TSON" that is basically "JSON with type declaration and comments" from TS, which I think would just solve a lot of problem straight up. JSON itself is just too restrictive and comes with its own bracket verbosity tax.
sadly, I think we are stuck with JSON as the most reliable way to get data / code in and out of an LLM (could be worse, could be YAML) … I’m interested in custom DSLs that improve LLM predictability and it is quite nice to see that even the Microsoft dinosaur “gets it” … see the Contacts example at https://slangify.org/examples which does VCARD to JCARD round tripping as a way to easily roll your own DSL
I felt conflicted as well, json is portable and easy to parse / validate and edit. But many models do still struggle. There are some stuff from functional programming might be worth bringing back here.
It's often comes with missing keys, use wrong value type (e.g., list over dictionaries). Mostly a small model issue and open source models, they don't follow instructions on the structure guidance that well, and there is no easy way to do generation-time validation.
In fact, Json as a common language for human in visualization has been around for a while! The benefit of declarative grammar is that users can effective manipulate specs through UI (drag and drop, clicks).
Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.
> Json as a common language for human in visualization has been around for a while
Plant, Mermaid, Graphviz are all declarative textual representations designed for human authoring, JSON is made for tools. Its not a criticism just a statement that if interop across agent and human was intended this is not the simplest option.
I don't really understand the point of this, I feel like LLMs have been able to one-shot matplotlib since GPT 3.5. I have extensively used LLMs to do data viz and haven't run into any problems. What is a specific instance where an agent struggles to generate a visualization and Flint solves it?
This comes with a bit of last mile issue, if just chatting with GPT in the chat panel and keep steering it, it's generally fine (as a power user).
But when building it in a tool that serve end users, we are starting to see that a 80% success rate in generating good looking charts can become a big issue. We experienced this when building some data analysis system. So the reliability, expressiveness, and costs (in terms of time and tokens) are hard to achieve all together with directly generating matplotlib, vega-lite etc.
So we essentially designed the langauge as a trade-off across the three, by moving some decisions to the compiler to reduce generation cost while maintain good expressivenss.
This is a valuable method of closing the gap in making LLM results available in a good visual form. The idea of viewing the charting process as the last step—where the AI deals only with high-level semantic specifications as opposed to all the low-level visual details—makes so much sense.
I'm interested to know what the layout optimization engine does: can the developers inject the desired limitations for some cases, or is the layout design process a black box? It's good to see you've made it open-source!
I love this. I'm trying not to go down yet another rabbit hole, but I really want to be able to tell claude code to make a visualization 'like this but just the way you know I like it'.
I'm seeing more attempts at standardizing how AI displays data via presets. It's practical, I guess, and hopefully it will make things less error-prone.
Semantic types as the extra formatting factor is super useful because they are concise encodings of a lot of formatting boilerplate.[1] Do you envision the flint type registry being shared/extensible? Why not have that be a data property itself?
[1] landed on pretty much the same spec for higher level tether charting on vegalite
Expressing charts in XML, JSON, etc is a terrible idea. There are several existing examples of this.
Ppl who think this is a good idea should be compelled to maintain some M4 or other configuration "files".
These configuration files ALWAYS devolve into a full language if they are actually used. So now you have the language you wrote your app in and then you have another, shittier language for your configs ... that you have to maintain.
Please stop doing this.
What should we do? In this case just express the chart in a natural language. Where there are uncertainties, the LLM should ask questions to clarify and record the results.
I'm not sure if Flint is the right tool for me. I'd like to have a tool that expresses code in visual form for me. For example, right now I need to reverse engineer some code for debugging purposes.
I already found out there are three tasks:
* Task one fills task two's queue and waits for an event to get notified
* Task two reads from its queue, forwards elements to task three's queue.
* Task three reads from its queue and sends a success/fail message back to task two's queue
* Task two then notifies the waiting task one.
Visually it's easily expressed: 3 bubbles lined up with 2 connections between the neighboring ones.
> Chat responses can now render interactive Mermaid diagrams with the renderMermaidDiagram tool. This lets models use flowcharts, sequence diagrams, and other visualizations to visually break down complex concepts. The diagrams are interactive, so you can pan and zoom to explore them in detail, or open them in a full-sized editor for easier viewing.
This is cool to see from a research team. A few weeks ago I was exploring a similar idea with ntcharts, where a user or LLM can specify a chart in a Golang or JSON object...
and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.
But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...
This is fun! We started thinking it would just be an engineering task in the beginning, but doing a solid intermediate language turned out to be a research project (the paper will be out soon).
Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45
I find that agents can use charts pretty well though. Maybe it would be interesting to see a comparison of an agent making charts for you based on a prompt and some data directly into Javasript it looks like, versus the same agent making the charts for you in Flint.
This is pretty crazy, literally built something almost exactly like this for a project I'm working on (a local-first AI agent that does work on folders while you sleep). Basically going from JSON "Lego blocks" to full reports (including charting, though a subset of what Flint offers). And with post-generation validation and retry steps.
Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.
Second comment, having read in more depth (really love the auto-layout detail!) - the spec doesn't seem to naturally support layering (which is useful in some multi-axis automatic cases) - any plans for composability?
We made an intentional decision to make it non-compositional for now, and we are using chart-type based approach to handle more expressive charts. (e.g., CDF charts)
But! We have a new project coming in to work on annotation machanism and interactivity to make it supporting presentation tasks.
Seaborn is already a nice high level semantic plotting interface. I suppose it doesn't target HTML output in a web app, but its main API is very declarative, one big function call with detailed kwargs. A port to JS seems possible
great point, I feel both the low-level foundation language and the high-level abstraction should evolve together, just like how human programming language evolves (e.g., frontend frameworks and libraries).
Mostly for expressiveness and reliability & cost trade-off. Flint has advantage of being an intermediate language that allow agents to generate good-looking stuff without additional refinement loops, since the compiler derives lower-level geometric constraints from semantic types.
Would especially be handy when we are building some agents that produce charts that serve end users (they want faster and more reliable experiences!).
Both, it's a combination of latency / reliability / expressivenss. I think when models are more powerful, we will also more naturally want to create more expressive stuff.
Hard not to reflexively reference the XKCD here. I think the authors of most charting languages would say they look good by default. It seems more likely this is achieving a subjectively different presentation than something objectively better. Higher level implies information loss. So it can only be better if it is doing so by assuming some better defaults. But then you have to ask if it lost expressiveness.
We find an additional dimension where previous language doesn't explicitly use (semantic type) as the key part of the specification, which turned out to be the root reason of why certain low-level parameters are set.
Effectively Flint is designed to specify charts using another set of semantic parameters over geometric parameters, which happens to be something AI are pretty good at! (They can based on field names + sample values to infer the semantic types reliably than guessing geometrical params.
Btw, for others reading this comment, this is the XKCD we are talking about! https://xkcd.com/927/
It's more like a simple high-level spec to make it easier. The idea is that you don't have to fill position / axes details just to make the chart work. The compiler has a bit of magic of using semantic types to optimize what parameters will be set in ECharts.
In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!
The charts are very nice, and I think the visualisation layer for LLMs is a very interesting problem.
I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.
Can you please stop? You're crossing the line into spamming HN and we're getting complaints.
It sounds like cool work, but you shouldn't be using HN in this way. Linking to your project occasionally in a relevant context is fine, but this should be a (small) minority of what you post to HN.
From https://news.ycombinator.com/newsguidelines.html: "Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity."
I'm terrible at diagrams, so I gave GPT very generic descriptions of one of our project, to convert in to that mermaid style, then for Lucid I pasted it in there, and had a visualization of what I needed. Worked out nicely.
interesting how you don't discuss literally anything about the project actually posted and spam your thing. Not pointing you out, seen many other comments like this on HN but always felt a bit weird about them
Flint: because watching an AI agent work is like watching a Roomba navigate — you need a visualization to understand why it spent 20 minutes in the corner
more like, instead of watching the AI agent 20 minutes to create a polished chart with 100k tokens, Flint lets you get there in 20 second with 200 tokens with a 95% polished chart. :)
"For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful.
This can’t be said enough. “Good for Agents” just means self-documenting, obvious ergonomics, save defaults, succinct (or controllable) output, programable interfaces, … all of which support human users too!
The "curb cut effect", right? I'm sure there's a human-friendly interface to MCPs myself. I think we're reinventing application scripting that way.
Isn't this literally made for AI agents to be accessed through an MCP server? Seems to me the AI agents part of the marketing is important.
But why be exclusive? Why not "Chart language for computer programs to generate"?
I don't want to use an agent at all, but i wouldn't mind generating some charts with an easy-to-generate markup language...
But for that we already have mermaid.js (and its precursor Graphviz/dot).
The only reason to use this instead of existing, mature ones designed for humans is if you are an AI agent.
Graphviz and mermaid are a shitshow.
anything more than their own handpicked examples and you're better off using d3 or yfiles. layering, clustering, boundaries, rearranging are all basic needs for text to diagrams. None support it.
Both suck at being any good for rendering diagrams from readable structured text. There is a gap to be addressed.
I find it lacks some easy way to do alignments or grouping, which makes editing frustrating. Could be a good language re-design opportunity.
yes and that led me to a more fundamental question : is it even possible for an easy way to do fine grained adjustments. The finer granularity brings complexity. that makes it unreadable (especially for the untrained) and hence set aside and forgotten. The other way is to narrow down and focus on a well defined subset of problems.
either way the number of people willing or compelled to learn it will be tiny. and it hence becomes a niche, perfect for a long-term side project but with no real return.
my conclusion was to stop looking and use D3 or custom code ( good looking charts for humans).
this project is trying to do the same, balance verbosity in text with granularity in the charts with a narrowish usecase : agent consumption.
That's the best option for now. But it can also be frustrating to ask AI to do small edits on D3 just to fix some idioms (like switching order etc) and they kept messing up with other stuff accidentally. Thus I still believe have a language with native representation for these diagram concepts would be helpful.
Good news is that AI do make new language a bit more accessible than before! If your agents can use it well and you can steer easily, it will naturally be good adoption.
Why can’t AIs generate for the “existing, mature ones?” Like the other commenter said, I’m not sure I get the “this is totally for AI” marketing. Why can’t AI use the existing ones and why can’t humans use this?
the design here has some constructs (i.e., semantic types) that AI can use better than human at generation time; and then the generated spec can be easy for user to edit since there is no need for hard-coded low-level parameters!
its not marketing to you, its marketing to agents looking for tools to use
I'd ask the question in reverse too. Why not let the AI agent use mermaid.js/graphviz/dot?
Presumably Microsoft thinks this one is better. Why? And why would that answer be any different for a human vs an LLM?
Mermaid looks terrible.
It is only better than nothing for the purpose of showing it to people whether produced by an LLM, by a human, or by both.
Your employees may just accept the internal slop, but at some point, you have to show your charts to your customer.
Doesn't that depend on the style that you apply to it?
No? I don't think there is a way to meaningfully change the layout for non trivial charts?
Careful, they'll spring something XML-derived on us.
Because these are completely different requirements.
Which are what, exactly?
Thank you for summarizing it this way. All their flowery language (esp from OP) seems like long for "we figure out how the chart should look based on its data". From their page:
> Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout, the Flint compiler derives optimized chart settings from the data, semantic types, chart type, and encodings.
also worth noting that Vega-Lite is literally just fucking that and AI already does good job with producing JSONs for it
I work with someone who did a lot of work with this to improve our ability to generate awesome visualizations with little thinking. It's a very powerful language but needs guardrails and guidance, particularly if you want end users to be able to produce consistent and standardized visualizations without knowing anything about it.
Also guidelines sent to the agent may or may not get ignored if they are just part of the context :(
A little bit more than that! Here shows why a higher-level abstraction can be helpful for chart generation: https://github.com/microsoft/flint-chart#features
for agent to generate, but also easy for human to edit (especially with UI) :)
And as per packages, built on top of existing charting libraries.
Did you read the post? It directly answers why this is specifically for AI agents.
There’s an emerging pattern in agentic systems and this project is a great example.
A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.
I feel we will be seeing this more and more in the near future.
When I first saw Claude generating PPT decks by writing Python code instead of making the XML directly, it was sort of an "aha moment" for me. This seems to be the path for many things. It also feels slightly limiting, and like a hack LONG term, but 100% correct approach for a while.
Have you seen marp? https://marp.app/
Another somewhat similar concept is how you can write Quarto markdown notebooks and have them output a RevealJS slideshow
I've been using slidev recently. How does this compare?
This is awesome
Do we have more details / documentation on this capability. Can we replicate the same using a generic agent skill with custom apps / harness.
I think I saw some library named pptxjs? in the thinking traces of Claude. Or was it this: https://www.npmjs.com/package/pptxgenjs
Anthropic released their PPTX skill[0] a while ago, the relies a bit on python and pptxgenjs, depending on the usecase:
> Read/analyze content....python -m markitdown presentation.pptx > Edit or create from template....Read editing.md > Create from scratch...Read pptxgenjs.md
[0] https://github.com/anthropics/skills/blob/main/skills/pptx/S...
Thanks -- will study and adopt this. Appreciate the pointer.
can you save that python and regenrate that exact powerpoint later ? maybe check it into repo isntead of powerpoint to genrate variations.
A well designed intermediary enables both validation and control over the output independent of the AI. This changes the interaction model between human and AI from delegation to collaboration.
also user interaction afterwards -- if can be frustrating if the only way the user can interact with the chart is to chat with the agent again (simple spec allows easy UI interaction!)
I'm all in on this idea. Every piece of agentic coding I have done in the last month has been via an intermediate representation. Iteration is done in the IR layer mainly. It's remarkable how close you can get to a deterministic coding output using this methodology.
Yes, this has been the pattern for agentic systems since the beginning: permissive generation, that retries over and over until it gets the right size shape through the hole, and the input validates.
Programming is more alive than ever
Is there a specific explanation about how this is better or different than vega itself? https://vega.github.io/vega/docs/specification/
My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.
Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents.
Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.
So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.
I'm sorry, but as someone who creates data visualisation as a big part of my job, I wouldn't say the charts on the website look good. Most aren't awful either, but by no means are they an improvement over what I'd get by telling any coding agent to make a chart with Vega-Lite or Observable Plot, and probably worse than if I had some decent instructions/skills.
I don't quite get what the goal of this is other than abstracting away a little bit of the complexity at the expense of flexibility. To me, the promise of LLMs is the opposite, I can get flexibility and customisation without the cost of complexity.
Some composite charts are quite annoying to be generated well (like bullet, waterfall etc), their Vega-Lite equivalent can be quite long if just starting from scratch.
The intention here is that Flint is a simpler abstraction to get basic setups right and any followup edits can be done on top of the first compiled outputs (thus not limiting expressiveness). It also makes it easier for user to manipulate (like swapping axes, click to change something, which can be very hard if LLM generates a complex chart spec upfront).
But for many basic stuff your intuition is completely right.
That's fair, I generally make charts for publication, so I spend much more time and effort on the details. But I can understand this being useful for quick exploration for some people.
Generally speaking, I suggest anyone interested in learning to make charts get familiar with grammar of graphics [0] libraries like Vega-Lite, Observable Plot, ggplot2, Altair. There is a bit of a learning curve if you're used to selecting chart types like in Excel, but once it clicks, it gives you virtually unlimited choices in the kinds of charts you can make. And with ggplot or Observable Plot [1], it's about the same number of lines as something like Flint.
0: https://data.europa.eu/apps/data-visualisation-guide/why-you...
1: https://observablehq.com/@observablehq/plot-gallery
Grammar of graphics has been the foundation of a lot of stuff and definitely worth learning for everyone!
A challenge with GoG is that it assumes configurations as second-class stuff, which makes it quite difficult for users to deal with things like changing formatter, scale, annotations. Flint kinda want to hide this aways (so Flint sets them on behalf of the agent or the user). But yeah, GoG is still the foundation for expressivenss.
Can you elaborate on what you mean? Why is it more difficult to deal with formatters, scales, annotations compared with other solutions? Unless I'm misunderstanding something, the defaults are similar to what you would get in Flint, and if you want to add or customise anything, it's usually just one extra line. That's kind of the entire point of the grammar of graphics.
https://microsoft.github.io/flint-chart/#/
The "how it works" section explains a little bit of this. For example, for the heatmap example showing temporal data, for a "good-looking" chart, we need to (1) reconcile the conflict between banded discrete steps and continuous temporal axis, and it requires understanding and setting stepsize and time parser, (2) for the correlation color, we need to set domain etc under the color axis.
These are supposed to be handled automatically as system defaults, but the tricky part is that these decisions are "semantical", thus requires us to understand the data and design principles, thus existing languages won't stretch that far. And the actual good looking spec is actually over 40 lines of json spec with many low-level paramters, way beyond the simple 5 line encoding promised by GoG. Flint uses semantic type and a layout optimization algorithm to handle this, so 5 lines of encoding + data semantic types can derive rest parameters automatically.
Some examples in the gallery are more extreme: like the waterfall chart example is way over 100 lines of code, and sunbusrt, rose chart are even more since compositions are quite difficult in GoG.
Glad to have a discussion on this level! In fact, we wrote a paper about this, will be putting it online in a week or so!
Thanks for the amazing work! How to follow up on reading the paper when available ?
We will link it on github and the project page!
https://github.com/microsoft/flint-chart
I strongly disagree ;-)
The paper's line of reasoning seems to continue the endless subjective loop of assuming your viz framework has the right abstractions & defaults , which the next person will rightfully disagree with for their slightly different eval set
We found in practice:
- LLM's generate charts fine
- LLM's tweak charts fine
- LLM's take user feedback to tweak them fine
In that sense, going higher-level for abstractions, as is being argued for here, is strictly worse: it's better to give controls so the LLM can go deep and customize
In practice, we found the choice of json config language X vs json config language Y to be pretty equivalent across different charting systems (vega, plotly, perspective, etc), LLM's do them all fine
The harder parts were deciding what a good chart is (model, reasoning, context), and opposite of this approach, giving lower-level facility for doing user change requests on tweaks, interactivity, and tricky in practice, when they have a lot of data on it.
You are absolutely right. But note that we are actually on the same point here.
This is exactly why this is an intermediate language designed to get 95% stuff right easily (for expressiveness and reliability purpose), while 5% of more advanced case where the agents need to revise chart for other purpose can be done easily on top of the compiled low-level spec (low in terms of Vega-Lite etc, not SVG). We are not really designing a higher abstraction to replace existing ones.
In the past, the split is like 50% good at first run for some common stuff, all other stuff requires agent-loop or user involvement.
Our goal is to make it easy for most case, not everything needs a full multi-round trip agentic workflow to solve. :)
We are kinda all advanced users in fact, for a lot of users, they are easily get confused with the first time result if that is not as good, and the interactivity cost / multi-round isn't an option.
I was wondering the same for vega-lite, which is relatively high level, declarative, and looks similar to their syntax.
Since it isn’t mentioned on the page, I’ll mention that accessibility is a really important thing to design in when creating data visualizations.
This podcast (I only have a spotify link) has a really good short interview on this:
https://open.spotify.com/episode/18dHTAxCCeIaLOTch6tRld
The interview is with Frank Elavsky who seems to be a rock star in the field (and no, I don’t know him and am not him) and made a project called Chartability relating to this which has heuristics, principles, and guidelines for a11y audits:
https://chartability.github.io/POUR-CAF/
Thanks a lot for the suggestion, I do think we will need to work on accessibility support in Flint, this is a perfect place to centrally handle accessibility issues.
I added an issue to track this
https://github.com/microsoft/flint-chart/issues/48
> simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability
N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?
A simpler spec can be used by a simpler agent. So, maybe that's the use-case here... use by smaller/cheaper agents that run in parallel as opposed to large models running one visualization at a time.
Or at least, maybe that's the idea?
IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.
we are considering also reliability, interactivity besides expressiveness. Simpler spec with good expressiveness comes handy when you want the agent to be reliably for non-expert users and with small models.
>Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout.
Ok, Microsoft is conflating two different things here: LLMs don't really care about code being low level and verbose, they can read things like Assembly and SPIR-V just fine: visualization is the real issue in that LLMs have no natural understanding of spatial composition through visual comparison because they literally "see" things differently than humans, so the way to get around that is provide them with "visualization" in code form that they can easily reason about and understand, so basically anything that's not deeply nested and has hidden states that they have to reason about.
Also, Flint being stringly typed in JSON is a decision that I don't think I agree with. Looking at the actual spec, this could have just been a normal, human usable TypeScript library, and it would have been 100x better. Using their own example (excuse the formatting):
type SemanticType = "Category" | "YearMonth" | "Profit";
type ChartType = "Heatmap" | "BarChart" | "LineChart" | "ScatterPlot"; // extend as needed
interface ChartEncodings { x: string; y: string; color?: string; size?: string; tooltip?: string; }
interface ChartProperties { colorScheme: string; [key: string]: unknown; // allow other optional properties }
interface ChartSpec { chartType: ChartType; encodings: ChartEncodings; chartProperties: ChartProperties; }
type SemanticTypes = Record<string, SemanticType>;
interface ChartConfig<TData = Record<string, unknown>> { data: TData; semantic_types: SemanticTypes; chart_spec: ChartSpec; }
// The actual typed object literal: const chartConfig: ChartConfig = { data: {}, // replace with your actual data shape/type semantic_types: { game: "Category", period: "YearMonth", newUsers: "Profit", }, chart_spec: { chartType: "Heatmap", encodings: { x: "period", y: "game", color: "newUsers", }, chartProperties: { colorScheme: "redblue", }, }, };
EDIT:
Went and actually looked at the source instead of just eyeballing it from the docs, and it was a lot more complete and sophisticated than my assumed mockup already.
Core complaint (string-keyed JSON vs. a real generic authoring surface) still stands, but the specific types I posted aren't what Flint has. My bad.
I do find the chartType part is not quite elegant, since templates should be more extensible. We will need to fix that.
For other parts, it's quite common in visualization and diagram etc libraries to have json, since they are easily portable in different rendering contexts.
I'm sure you know that JSON are object literals in TypeScript, there isn't really even a serialization process that's needed there. The AST/IR can still be in JSON, but the authoring surface can be a restricted subset that does not allow logic to execute, that way you can still get the type safety and functionality of TS when you need conditional/loops/logic without throwing away what TS already gives you.
I mean, it would be great if you guys would have like a "TSON" that is basically "JSON with type declaration and comments" from TS, which I think would just solve a lot of problem straight up. JSON itself is just too restrictive and comes with its own bracket verbosity tax.
JSON schema can do it to some degree, I think?
Agreed, JSON is not the most optimal spec language choice, but is it better if different projects all started to writing their own specs?
Related: https://www.openui.com/blog/stop-making-ai-write-json
sadly, I think we are stuck with JSON as the most reliable way to get data / code in and out of an LLM (could be worse, could be YAML) … I’m interested in custom DSLs that improve LLM predictability and it is quite nice to see that even the Microsoft dinosaur “gets it” … see the Contacts example at https://slangify.org/examples which does VCARD to JCARD round tripping as a way to easily roll your own DSL
I felt conflicted as well, json is portable and easy to parse / validate and edit. But many models do still struggle. There are some stuff from functional programming might be worth bringing back here.
What is it about json that models struggle with, in your experience? Is it syntax or structure?
It's often comes with missing keys, use wrong value type (e.g., list over dictionaries). Mostly a small model issue and open source models, they don't follow instructions on the structure guidance that well, and there is no easy way to do generation-time validation.
> requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler
Isnt graphviz there for the same reason?
Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!
In fact, Json as a common language for human in visualization has been around for a while! The benefit of declarative grammar is that users can effective manipulate specs through UI (drag and drop, clicks).
Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.
> Json as a common language for human in visualization has been around for a while
Plant, Mermaid, Graphviz are all declarative textual representations designed for human authoring, JSON is made for tools. Its not a criticism just a statement that if interop across agent and human was intended this is not the simplest option.
right, in fact many small models still struggle with following json, some new forms are also needed
Absolutely. This is DOA honestly and not really better than what we had before.
Project page: https://microsoft.github.io/flint-chart/
MCP setup: https://microsoft.github.io/flint-chart/#/mcp
I don't really understand the point of this, I feel like LLMs have been able to one-shot matplotlib since GPT 3.5. I have extensively used LLMs to do data viz and haven't run into any problems. What is a specific instance where an agent struggles to generate a visualization and Flint solves it?
This comes with a bit of last mile issue, if just chatting with GPT in the chat panel and keep steering it, it's generally fine (as a power user).
But when building it in a tool that serve end users, we are starting to see that a 80% success rate in generating good looking charts can become a big issue. We experienced this when building some data analysis system. So the reliability, expressiveness, and costs (in terms of time and tokens) are hard to achieve all together with directly generating matplotlib, vega-lite etc.
So we essentially designed the langauge as a trade-off across the three, by moving some decisions to the compiler to reduce generation cost while maintain good expressivenss.
Ah got it, that makes more sense. Thanks for the info!
https://github.com/scicloj/kindly
but enterprise
This is a valuable method of closing the gap in making LLM results available in a good visual form. The idea of viewing the charting process as the last step—where the AI deals only with high-level semantic specifications as opposed to all the low-level visual details—makes so much sense. I'm interested to know what the layout optimization engine does: can the developers inject the desired limitations for some cases, or is the layout design process a black box? It's good to see you've made it open-source!
It's a white box! We built a demo here: https://microsoft.github.io/flint-chart/#/documentation/char...
We use a fun elastic algorithm to decide dimension etc within the developer's constraint.
I love this. I'm trying not to go down yet another rabbit hole, but I really want to be able to tell claude code to make a visualization 'like this but just the way you know I like it'.
I'm seeing more attempts at standardizing how AI displays data via presets. It's practical, I guess, and hopefully it will make things less error-prone.
We struggled the same, especially when deploying an agent to end users, they are even more frustrated with any mistakes from agents than us!
Semantic types as the extra formatting factor is super useful because they are concise encodings of a lot of formatting boilerplate.[1] Do you envision the flint type registry being shared/extensible? Why not have that be a data property itself?
[1] landed on pretty much the same spec for higher level tether charting on vegalite
Expressing charts in XML, JSON, etc is a terrible idea. There are several existing examples of this.
Ppl who think this is a good idea should be compelled to maintain some M4 or other configuration "files". These configuration files ALWAYS devolve into a full language if they are actually used. So now you have the language you wrote your app in and then you have another, shittier language for your configs ... that you have to maintain.
Please stop doing this.
What should we do? In this case just express the chart in a natural language. Where there are uncertainties, the LLM should ask questions to clarify and record the results.
This is a formal language that compiler can take! The natural language and clarify loop helps the LLM to generate Flint spec that can render.
Nice to stumble over this thread.
I'm not sure if Flint is the right tool for me. I'd like to have a tool that expresses code in visual form for me. For example, right now I need to reverse engineer some code for debugging purposes.
I already found out there are three tasks:
Visually it's easily expressed: 3 bubbles lined up with 2 connections between the neighboring ones.Which ML tools suited best for that?
Seems a bit more like https://mermaid.js.org use cases!
Here's the official mermaid vscode ext: MermaidChart.vscode-mermaid-chart: https://marketplace.visualstudio.com/items?itemName=MermaidC... .. src: https://github.com/Mermaid-Chart/vscode-mermaid-chart :
> @mermaid-chart
The functionality of the vscode-mermAId extension was merged into vscode FWIU? From https://code.visualstudio.com/updates/v1_109#_mermaid-diagra... :
> Mermaid diagrams in chat responses
> Chat responses can now render interactive Mermaid diagrams with the renderMermaidDiagram tool. This lets models use flowcharts, sequence diagrams, and other visualizations to visually break down complex concepts. The diagrams are interactive, so you can pan and zoom to explore them in detail, or open them in a full-sized editor for easier viewing.
This is cool to see from a research team. A few weeks ago I was exploring a similar idea with ntcharts, where a user or LLM can specify a chart in a Golang or JSON object...
and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.
But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...
https://github.com/NimbleMarkets/ntcharts/blob/spec/spec/REA...
This is fun! We started thinking it would just be an engineering task in the beginning, but doing a solid intermediate language turned out to be a research project (the paper will be out soon).
Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45
Choosing JSON spec might be a bad idea.
It's machine friendly but not exactly LLM friendly.
https://github.com/tcsenpai/flint-chart-skill
Here is a skill using that (tested)
This is awesome!
I find that agents can use charts pretty well though. Maybe it would be interesting to see a comparison of an agent making charts for you based on a prompt and some data directly into Javasript it looks like, versus the same agent making the charts for you in Flint.
For a while now, everytime I see „for AI agents” Im looking for benchmarks with comparable solutions.
The only „for AI agent” solution that will gain adoption is/should be measured by two dimensions: token usage and correctness.
If the solution doesnt use less tokens than generating chart.js code for example - why should I use it?
Same for correctness, if the generated chart spec is correct only 90% of the time - why should I use it?
It’s still early but I think this is the direction we should be thinking about „for AI agents” libraries and projects.
Yes! correctness, expressiveness, reliability and cost (token + time)
I agree this is the right time to be exploring this more.
This is pretty crazy, literally built something almost exactly like this for a project I'm working on (a local-first AI agent that does work on folders while you sleep). Basically going from JSON "Lego blocks" to full reports (including charting, though a subset of what Flint offers). And with post-generation validation and retry steps.
Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.
That's awesome!
I find that besides training better models, designing new language for agents is also a super viable paths to improve their performance!
LLMs are bad at low-level chart config. Compiling from a smaller semantic language is the right shape for this problem.
Forget AI agents, this DSL is better even for humans. Cool project!
For AI agents and Humans :)
Second comment, having read in more depth (really love the auto-layout detail!) - the spec doesn't seem to naturally support layering (which is useful in some multi-axis automatic cases) - any plans for composability?
We made an intentional decision to make it non-compositional for now, and we are using chart-type based approach to handle more expressive charts. (e.g., CDF charts)
But! We have a new project coming in to work on annotation machanism and interactivity to make it supporting presentation tasks.
Seaborn is already a nice high level semantic plotting interface. I suppose it doesn't target HTML output in a web app, but its main API is very declarative, one big function call with detailed kwargs. A port to JS seems possible
Good points. Another step to be a middleware translator between LLM abilities to specific application area.
It might lead to good or miss leading.
While LLM ability boundry extends, it might invalid for next turn.
Best Qiuwu
great point, I feel both the low-level foundation language and the high-level abstraction should evolve together, just like how human programming language evolves (e.g., frontend frameworks and libraries).
I think a lot of were already using Mermaid and/or Python/Matplotlib(etc) for this. What would be the advantages to Flint?
Mostly for expressiveness and reliability & cost trade-off. Flint has advantage of being an intermediate language that allow agents to generate good-looking stuff without additional refinement loops, since the compiler derives lower-level geometric constraints from semantic types.
Would especially be handy when we are building some agents that produce charts that serve end users (they want faster and more reliable experiences!).
Doubt this is really a language problem instead of a model-capability one. A better prompt might get you most of the way there.
Both, it's a combination of latency / reliability / expressivenss. I think when models are more powerful, we will also more naturally want to create more expressive stuff.
Can't they just release a dataset that AIs can train on to become better at writing scripts that visualize data?
I like it.Impressive variations.
Hard not to reflexively reference the XKCD here. I think the authors of most charting languages would say they look good by default. It seems more likely this is achieving a subjectively different presentation than something objectively better. Higher level implies information loss. So it can only be better if it is doing so by assuming some better defaults. But then you have to ask if it lost expressiveness.
We find an additional dimension where previous language doesn't explicitly use (semantic type) as the key part of the specification, which turned out to be the root reason of why certain low-level parameters are set.
Effectively Flint is designed to specify charts using another set of semantic parameters over geometric parameters, which happens to be something AI are pretty good at! (They can based on field names + sample values to infer the semantic types reliably than guessing geometrical params.
Btw, for others reading this comment, this is the XKCD we are talking about! https://xkcd.com/927/
Goodbye Salesforce Tableau
It compiles into Echarts, but echarts already has a JSON co figuration spec
It's more like a simple high-level spec to make it easier. The idea is that you don't have to fill position / axes details just to make the chart work. The compiler has a bit of magic of using semantic types to optimize what parameters will be set in ECharts.
In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!
can this be used for math viz? Im having claude teach me visualize math and it writes lots of low level canvas code just to make it look good.
no yet! that's a good direction!
Probability of an MS project existing in the next N days:
That’s no language, that’s a JSON schema
This happens if a company has a CEO who presumably can no longer successfully go to the toilet without AI assistance.
Agents, npm, typescript, MCP. All buzzwords are there. Will anyone look at the slop charts? Of course not, the tokens are the goal.
MSFT stock is at 2024 levels. Maybe someone should produce a flint chart and present the agentic work to Nadella. No one buys this AI slop any more.
I actually made a chart with Flint to show MSFT stock, and with sparkline chart to compare with other companies... :)
why is this better than eg matplotlib?
Tweakers.
Make something people want.
The charts are very nice, and I think the visualisation layer for LLMs is a very interesting problem.
I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.
Sample document: https://smalldocs.org/blogs/what-is-a-smalldoc
Source: https://github.com/espressoplease/smalldocs
Can you please stop? You're crossing the line into spamming HN and we're getting complaints.
It sounds like cool work, but you shouldn't be using HN in this way. Linking to your project occasionally in a relevant context is fine, but this should be a (small) minority of what you post to HN.
From https://news.ycombinator.com/newsguidelines.html: "Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity."
Sorry dang, I will stop.
> mermaid diagrams
I'm terrible at diagrams, so I gave GPT very generic descriptions of one of our project, to convert in to that mermaid style, then for Lucid I pasted it in there, and had a visualization of what I needed. Worked out nicely.
Yep I find them to be very useful for explaining a system.
They can do a lot of cool things! Mermaid gallery here: https://smalldocs.org/s/xZrc-lNW1kbXpoIuU3l_ky#k=-0ehGe2B-hR...
interesting how you don't discuss literally anything about the project actually posted and spam your thing. Not pointing you out, seen many other comments like this on HN but always felt a bit weird about them
Fair point. Sorry
Flint: because watching an AI agent work is like watching a Roomba navigate — you need a visualization to understand why it spent 20 minutes in the corner
more like, instead of watching the AI agent 20 minutes to create a polished chart with 100k tokens, Flint lets you get there in 20 second with 200 tokens with a 95% polished chart. :)