Convert Figma logo to code with AI

microsoft logodata-formulator

🪄 Data Formulator is an interactive AI-powered data analysis system makes it easy to connect, explore and visualize data.

15,928
1,496
15,928
85
View on GitHub

Top Related Projects

huggingface's avatar

transformers

159,444

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

deepspeedai's avatar

DeepSpeed

42,282

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

microsoft's avatar

FLAML

4,373

A fast library for AutoML and tuning. Join our Discord: https://discord.gg/Cppx2vSPVP.

microsoft's avatar

onnxruntime

20,841

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

google-research's avatar

bert

40,008

TensorFlow code and pre-trained models for BERT

pytorch's avatar

pytorch

99,486

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Quick Overview

Data Formulator is an open-source project by Microsoft that aims to simplify the process of creating and managing data schemas for various data formats. It provides a unified approach to define, validate, and transform data structures across different platforms and languages.

Pros

  • Streamlines data schema creation and management
  • Supports multiple data formats and languages
  • Improves data consistency and interoperability
  • Reduces development time and potential errors in data handling

Cons

  • Limited documentation and examples available
  • Still in early development stages
  • May require a learning curve for users unfamiliar with schema definition concepts
  • Limited community support compared to more established data schema tools

Code Examples

# Define a simple schema
from data_formulator import Schema

user_schema = Schema({
    "name": str,
    "age": int,
    "email": str
})

# Validate data against the schema
valid_user = {"name": "John Doe", "age": 30, "email": "john@example.com"}
user_schema.validate(valid_user)  # Returns True

invalid_user = {"name": "Jane Doe", "age": "25", "email": "jane@example.com"}
user_schema.validate(invalid_user)  # Raises ValidationError

# Transform data using a schema
from data_formulator import Schema, Transform

transform_schema = Schema({
    "full_name": Transform(lambda x: x.upper()),
    "age": Transform(lambda x: x * 2)
})

data = {"full_name": "John Doe", "age": 30}
transformed_data = transform_schema.apply(data)
# Result: {"full_name": "JOHN DOE", "age": 60}

# Create a nested schema
nested_schema = Schema({
    "user": {
        "name": str,
        "address": {
            "street": str,
            "city": str,
            "country": str
        }
    },
    "orders": [int]
})

# Validate nested data
nested_data = {
    "user": {
        "name": "Alice",
        "address": {
            "street": "123 Main St",
            "city": "Anytown",
            "country": "USA"
        }
    },
    "orders": [1001, 1002, 1003]
}
nested_schema.validate(nested_data)  # Returns True

Getting Started

To get started with Data Formulator, follow these steps:

  1. Install the library:

    pip install data-formulator

  2. Import the necessary modules:

    from data_formulator import Schema, Transform

  3. Define your schema:

    my_schema = Schema({
    "name": str,
    "age": int,
    "email": str
})

  4. Use the schema to validate or transform data:

    data = {"name": "Alice", "age": 28, "email": "alice@example.com"}
my_schema.validate(data)

For more advanced usage and features, refer to the project's documentation and examples in the GitHub repository.

Competitor Comparisons

huggingface logo

transformers

159,444

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Pros of transformers

  • Extensive library of pre-trained models for various NLP tasks
  • Active community and frequent updates
  • Comprehensive documentation and examples

Cons of transformers

  • Larger library size and potential overhead for simpler projects
  • Steeper learning curve for beginners

Code comparison

data-formulator:

from data_formulator import DataFormulator

df = DataFormulator()
result = df.generate_data("Create a list of 5 fruits")
print(result)

transformers:

from transformers import pipeline

generator = pipeline('text-generation', model='gpt2')
result = generator("List 5 fruits:", max_length=50)
print(result[0]['generated_text'])

Summary

transformers is a comprehensive library for NLP tasks with a wide range of pre-trained models, while data-formulator focuses on data generation. transformers offers more flexibility and options for various NLP applications, but may be more complex for simple tasks. data-formulator provides a straightforward approach to data generation, which could be beneficial for specific use cases.

deepspeedai logo

DeepSpeed

42,282

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Pros of DeepSpeed

  • Highly optimized for distributed training of large models
  • Supports a wide range of hardware configurations and model architectures
  • Extensive documentation and active community support

Cons of DeepSpeed

  • Steeper learning curve for beginners
  • May be overkill for smaller projects or simpler models
  • Requires more setup and configuration compared to Data Formulator

Code Comparison

Data Formulator:

from data_formulator import DataFormulator

df = DataFormulator()
df.load_data("dataset.csv")
df.preprocess()
df.train_model()

DeepSpeed:

import deepspeed
import torch

model = MyModel()
engine = deepspeed.initialize(model=model, config_params=ds_config)
for batch in dataloader:
    loss = engine(batch)
    engine.backward(loss)
    engine.step()

Key Differences

  • Data Formulator focuses on simplifying data preprocessing and model training for tabular data
  • DeepSpeed is designed for large-scale distributed training of deep learning models
  • Data Formulator provides a higher-level API, while DeepSpeed offers more fine-grained control
  • DeepSpeed is better suited for advanced users and complex projects, while Data Formulator caters to beginners and simpler use cases
microsoft logo

FLAML

4,373

A fast library for AutoML and tuning. Join our Discord: https://discord.gg/Cppx2vSPVP.

Pros of FLAML

  • More comprehensive AutoML toolkit with support for various tasks (classification, regression, time series forecasting, etc.)
  • Efficient hyperparameter tuning with cost-aware search algorithms
  • Active development and regular updates

Cons of FLAML

  • Steeper learning curve due to more advanced features
  • May be overkill for simpler data processing tasks
  • Requires more computational resources for complex optimizations

Code Comparison

FLAML example:

from flaml import AutoML
automl = AutoML()
automl.fit(X_train, y_train, task="classification")
predictions = automl.predict(X_test)

Data-Formulator example:

from data_formulator import DataFormulator
df = DataFormulator(data)
df.process()
result = df.get_result()

Summary

FLAML is a more comprehensive AutoML toolkit suitable for various machine learning tasks, while Data-Formulator focuses on data processing and transformation. FLAML offers advanced features like efficient hyperparameter tuning but may have a steeper learning curve. Data-Formulator is simpler to use for basic data manipulation tasks but lacks the advanced ML capabilities of FLAML. Choose based on your specific needs and project complexity.

microsoft logo

onnxruntime

20,841

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Pros of ONNX Runtime

  • Widely adopted and supported across multiple platforms and frameworks
  • Optimized for high-performance inference on various hardware
  • Extensive documentation and community support

Cons of ONNX Runtime

  • Larger codebase and more complex setup compared to Data Formulator
  • Primarily focused on inference, not data preprocessing or transformation

Code Comparison

Data Formulator:

from data_formulator import DataFormulator

df = DataFormulator()
df.add_column("new_column", lambda row: row["existing_column"] * 2)
transformed_data = df.transform(input_data)

ONNX Runtime:

import onnxruntime as ort

session = ort.InferenceSession("model.onnx")
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
result = session.run([output_name], {input_name: input_data})[0]

Summary

ONNX Runtime is a powerful inference engine for machine learning models, while Data Formulator focuses on data preprocessing and transformation. ONNX Runtime offers broader platform support and optimization capabilities, but may be overkill for simpler data manipulation tasks. Data Formulator provides a more straightforward approach to data transformation but lacks the extensive inference capabilities of ONNX Runtime.

google-research logo

bert

40,008

TensorFlow code and pre-trained models for BERT

Pros of BERT

  • Widely adopted and extensively researched in the NLP community
  • Pre-trained models available for various languages and tasks
  • Extensive documentation and community support

Cons of BERT

  • Requires significant computational resources for training and fine-tuning
  • May be overkill for simpler NLP tasks
  • Limited flexibility for customizing the model architecture

Code Comparison

BERT example:

import tensorflow as tf
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained('bert-base-uncased')

Data Formulator example:

from data_formulator import DataFormulator

formulator = DataFormulator()
formulated_data = formulator.formulate(input_data)

While BERT focuses on natural language processing tasks, Data Formulator appears to be a tool for data manipulation and transformation. BERT provides pre-trained models for various NLP tasks, whereas Data Formulator seems to offer a more flexible approach to data formatting and preparation. The choice between the two would depend on the specific requirements of your project and the nature of the data you're working with.

pytorch logo

pytorch

99,486

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Pros of PyTorch

  • Widely adopted and supported by a large community
  • Extensive ecosystem of tools and libraries
  • Flexible and intuitive for dynamic neural networks

Cons of PyTorch

  • Steeper learning curve for beginners
  • Larger memory footprint compared to some alternatives
  • Can be slower for certain operations on CPU

Code Comparison

Data-Formulator:

from data_formulator import DataFormulator

df = DataFormulator()
df.load_data("dataset.csv")
df.preprocess()

PyTorch:

import torch
from torch.utils.data import Dataset, DataLoader

class CustomDataset(Dataset):
    def __init__(self, data):
        self.data = data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]

While Data-Formulator focuses on simplifying data preprocessing and formatting, PyTorch provides a comprehensive framework for building and training neural networks. Data-Formulator offers a more streamlined approach for data preparation, while PyTorch requires more setup but offers greater flexibility and power for complex machine learning tasks.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Data Formulator icon  Data Formulator: AI-powered Data Visualization

🪄 Explore data with visualizations, powered by AI agents.

Try Online Demo   Install Locally

arXiv  License: MIT  YouTube  build  Discord

Why Data Formulator?

Your data lives everywhere — databases, warehouses, BI tools, files. Coding agents can help, but only after someone wires them up, and answers come back as walls of code or text that are hard to follow, refine, or share.

Data Formulator makes it simple: connect any data, ask anything, get charts you can edit, branch, and share — all on one interactive, visual canvas.

  • Data & platform teams: wire up your databases, warehouses, and BI sources once, and give the whole org an AI-powered data exploration layer.
  • Analysts & users: ask, edit, branch, share. It's so easy to get insights from good-looking charts.

https://github.com/user-attachments/assets/8e4f8a08-6423-4227-a1f7-559e0126ce31

News 🔥🔥🔥

[05-28-2026] Data Formulator 0.7 — turn ANY data into insights in five easy steps:

  1. Connect. Governed, reusable connections to databases, warehouses, BI systems, object stores, and files (Superset, Kusto, Cosmos DB, MySQL, PostgreSQL, MSSQL, BigQuery, S3, Azure Blob, …). Need a custom source? Point your coding agent at the data loader plugin guide.
  2. Load. Ask the data-loading agent to find tables from connected databases, or extract data from Excel files, images, websites, and text.
  3. Explore. A unified Data Agent with thread memory inspects data, runs sandboxed code, and weaves explanation, exploration, and recommendation into one fluid conversation — grounded in your context. The Data Thread keeps questions, intermediate results, and charts navigable: revisit earlier steps, branch into alternatives, and compare side by side.
  4. Refine. 30+ chart types (area, streamgraph, candlestick, radar, maps, KPI, …) via a new semantic chart engine, plus a style-refinement agent that turns rough charts into presentation-ready visuals through natural language.
  5. Share. Build reports and export as image or PDF to tell the story.

➕ Persistent sessions & workspaces — identity-isolated, saved across restarts. Data Formulator is your de facto data analysis pane.

Multilingual UI — Data Formulator now speaks Chinese in addition to English (没错，DF现在会说中文了！). More languages on the way — contributions welcome.

Install with pip install data_formulator or run instantly with uvx data_formulator.

[!TIP] Are you a developer? Join us to shape the future of AI-powered data exploration! We're looking for help with new agents, data connectors, chart templates, and more. Check out the Developers' Guide and our open issues.

Previous Updates

Here are milestones that lead to the current design:

  • v0.7 alpha 2 (05-11-2026): Early preview of data connectors, the unified DataAgent with thread memory, persistent workspaces, the semantic chart engine, and experimental knowledge distillation.
  • v0.6 (Demo): Real-time insights from live data — connect to URLs and databases with automatic refresh
  • uv support: Faster installation with uv — uvx data_formulator or uv pip install data_formulator
  • v0.5.1 (Demo): Community data loaders, US Map & Pie Chart, editable reports, snappier UI
  • v0.5: Vibe with your data, in control — agent mode, data extraction, reports
  • v0.2.2 (Demo): Goal-driven exploration with agent recommendations and performance improvements
  • v0.2.1.3/4 (Readme | Demo): External data loaders (MySQL, PostgreSQL, MSSQL, Azure Data Explorer, S3, Azure Blob)
  • v0.2 (Demos): Large data support with DuckDB integration
  • v0.1.7 (Demos): Dataset anchoring for cleaner workflows
  • v0.1.6 (Demo): Multi-table support with automatic joins
  • Model Support: OpenAI, Azure, Ollama, Anthropic via LiteLLM (feedback)
  • Python Package: Easy local installation (try it)
  • Visualization Challenges: Test your skills (challenges)
  • Data Extraction: Parse data from images and text (demo)
  • Initial Release: Blog | Video

Overview

Data Formulator is a Microsoft Research project for data exploration with visualizations powered by AI agents. It combines UI interactions with natural language so analysts can communicate intent, branch into alternative analyses, and share results — starting from any data format (screenshot, text, CSV, or database).

Get Started

Play with Data Formulator with one of the following options.

  • Option 1: Install via uv (recommended)

    uv is an extremely fast Python package manager. If you have uv installed, you can run Data Formulator directly without any setup:

    uvx data_formulator

    Run uvx data_formulator --help to see all available options, such as custom port, sandboxing mode, and data storage location.

  • Option 2: Install via pip

    Use pip for installation (recommend: install it in a virtual environment).

    pip install data_formulator # install
python -m data_formulator # run

    Data Formulator will be automatically opened in the browser at http://localhost:5567.

  • Option 3: Run with Docker

    docker compose up --build

    Open http://localhost:5567 in your browser. To stop, press Ctrl+C or run docker compose down.

  • Option 4: Codespaces

    You can run Data Formulator in Codespaces; we have everything pre-configured. For more details, see CODESPACES.md.

    Open in GitHub Codespaces

  • Option 5: Working as developer

    You can build Data Formulator locally and develop your own version. Check out details in DEVELOPMENT.md.

Using Data Formulator

Besides uploading csv, tsv or xlsx files that contain structured data, you can ask Data Formulator to extract data from screenshots, text blocks or websites, or load data from databases use connectors. Then you are ready to explore. Ask visualizaiton questions, edit charts, or delegate some exploration tasks to agents. Then, create reports to share your insights.

https://github.com/user-attachments/assets/164aff58-9f93-4792-b8ed-9944578fbb72

Research Papers

@article{wang2024dataformulator2iteratively,
      title={Data Formulator 2: Iteratively Creating Rich Visualizations with AI}, 
      author={Chenglong Wang and Bongshin Lee and Steven Drucker and Dan Marshall and Jianfeng Gao},
      year={2024},
      booktitle={ArXiv preprint arXiv:2408.16119},
}

@article{wang2023data,
  title={Data Formulator: AI-powered Concept-driven Visualization Authoring},
  author={Wang, Chenglong and Thompson, John and Lee, Bongshin},
  journal={IEEE Transactions on Visualization and Computer Graphics},
  year={2023},
  publisher={IEEE}
}

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.