Show HN: Microsoft releases Flint, a visualization language for AI agents

"For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful.

There’s an emerging pattern in agentic systems and this project is a great example.

A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.

I feel we will be seeing this more and more in the near future.

>Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout.

Ok, Microsoft is conflating two different things here: LLMs don't really care about code being low level and verbose, they can read things like Assembly and SPIR-V just fine: visualization is the real issue in that LLMs have no natural understanding of spatial composition through visual comparison because they literally "see" things differently than humans, so the way to get around that is provide them with "visualization" in code form that they can easily reason about and understand, so basically anything that's not deeply nested and has hidden states that they have to reason about.

Also, Flint being stringly typed in JSON is a decision that I don't think I agree with. Looking at the actual spec, this could have just been a TypeScript library, and it would have been 100x better. Using their own example (excuse the formatting):

type SemanticType = "Category" | "YearMonth" | "Profit";

type ChartType = "Heatmap" | "BarChart" | "LineChart" | "ScatterPlot"; // extend as needed

interface ChartEncodings { x: string; y: string; color?: string; size?: string; tooltip?: string; }

interface ChartProperties { colorScheme: string; [key: string]: unknown; // allow other optional properties }

interface ChartSpec { chartType: ChartType; encodings: ChartEncodings; chartProperties: ChartProperties; }

type SemanticTypes = Record<string, SemanticType>;

interface ChartConfig<TData = Record<string, unknown>> { data: TData; semantic_types: SemanticTypes; chart_spec: ChartSpec; }

// The actual typed object literal: const chartConfig: ChartConfig = { data: {}, // replace with your actual data shape/type semantic_types: { game: "Category", period: "YearMonth", newUsers: "Profit", }, chart_spec: { chartType: "Heatmap", encodings: { x: "period", y: "game", color: "newUsers", }, chartProperties: { colorScheme: "redblue", }, }, };

I don't really understand the point of this, I feel like LLMs have been able to one-shot matplotlib since GPT 3.5. I have extensively used LLMs to do data viz and haven't run into any problems. What is a specific instance where an agent struggles to generate a visualization and Flint solves it?

This happens if a company has a CEO who presumably can no longer successfully go to the toilet without AI assistance.

Agents, npm, typescript, MCP. All buzzwords are there. Will anyone look at the slop charts? Of course not, the tokens are the goal.

MSFT stock is at 2024 levels. Maybe someone should produce a flint chart and present the agentic work to Nadella. No one buys this AI slop any more.

Is there a specific explanation about how this is better or different than vega itself? https://vega.github.io/vega/docs/specification/

My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.

> simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability

N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?

> requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler

Isnt graphviz there for the same reason?

Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!

Project page: https://microsoft.github.io/flint-chart/

MCP setup: https://microsoft.github.io/flint-chart/#/mcp

This is cool to see from a research team. A few weeks ago I was exploring a similar idea with ntcharts, where a user or LLM can specify a chart in a Golang or JSON object...

and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.

But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...

https://github.com/NimbleMarkets/ntcharts/blob/spec/spec/REA...

This is pretty crazy, literally built something almost exactly like this for a project I'm working on (a local-first AI agent that does work on folders while you sleep). Basically going from JSON "Lego blocks" to full reports (including charting, though a subset of what Flint offers). And with post-generation validation and retry steps.

Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.

The charts are very nice, and I think the visualisation layer for LLMs is a very interesting problem.

I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.

Sample document: https://smalldocs.org/blogs/what-is-a-smalldoc

Source: https://github.com/espressoplease/smalldocs

https://github.com/scicloj/kindly

but enterprise

Forget AI agents, this DSL is better even for humans. Cool project!

Packages show its built on:

https://www.chartjs.org/

https://echarts.apache.org/en/index.html

https://vega.github.io/vega/

.....

.....

https://github.com/microsoft/flint-chart/blob/main/packages/...

Tweakers.

Make something people want.

It compiles into Echarts, but echarts already has a JSON co figuration spec

[flagged]