awesome-llm-json
Resource list for generating JSON using LLMs via function calling, tools, CFG. Libraries, Models, Notebooks, etc.
Top Related Projects
🦜🔗 The platform for reliable agents.
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf
Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
Examples and guides for using the OpenAI API
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
Quick Overview
The "awesome-llm-json" repository is a curated list of resources focused on JSON output from Large Language Models (LLMs). It provides a comprehensive collection of tools, libraries, and techniques for working with structured JSON data generated by LLMs, aiming to improve the reliability and usability of LLM outputs in various applications.
Pros
- Extensive collection of resources related to LLM JSON output
- Well-organized and categorized for easy navigation
- Regularly updated with new tools and techniques
- Includes both open-source and commercial solutions
Cons
- Lacks detailed explanations or tutorials for each resource
- May overwhelm beginners due to the large number of options
- Some listed resources might become outdated quickly in the fast-evolving LLM field
- Focuses primarily on English-language resources
Getting Started
To explore the awesome-llm-json repository:
- Visit the GitHub repository: https://github.com/imaurer/awesome-llm-json
- Browse through the categories to find relevant resources
- Click on the links to explore individual tools, libraries, or techniques
- Consider starring the repository to stay updated with new additions
Note: This is not a code library, so there are no code examples or specific installation instructions. The repository serves as a curated list of resources for developers and researchers working with LLM JSON outputs.
Competitor Comparisons
🦜🔗 The platform for reliable agents.
Pros of langchain
- Comprehensive framework for building LLM applications
- Active development with frequent updates and new features
- Extensive documentation and community support
Cons of langchain
- Steeper learning curve due to its complexity
- May be overkill for simple LLM-based projects
- Requires more setup and configuration
Code comparison
awesome-llm-json:
{
"name": "GPT-3",
"description": "Large language model by OpenAI",
"capabilities": ["text generation", "translation", "summarization"]
}
langchain:
from langchain import OpenAI, LLMChain
from langchain.prompts import PromptTemplate
llm = OpenAI(temperature=0.9)
prompt = PromptTemplate(input_variables=["product"], template="What is a good name for a company that makes {product}?")
chain = LLMChain(llm=llm, prompt=prompt)
Summary
awesome-llm-json is a curated list of LLM resources in JSON format, making it easy to parse and integrate into projects. It's simpler and more straightforward for quick reference.
langchain is a full-fledged framework for building LLM applications, offering more functionality and flexibility. It's better suited for complex projects but requires more time to learn and implement.
Choose awesome-llm-json for quick reference and simple integrations, or langchain for building comprehensive LLM-powered applications.
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Pros of DeepSpeed
- Focuses on optimizing deep learning training and inference
- Provides extensive documentation and examples for implementation
- Offers advanced features like ZeRO optimizer and 3D parallelism
Cons of DeepSpeed
- Steeper learning curve due to its complexity
- Primarily designed for large-scale models and may be overkill for smaller projects
- Requires more setup and configuration compared to simpler libraries
Code Comparison
DeepSpeed:
import deepspeed
model_engine, optimizer, _, _ = deepspeed.initialize(
args=args,
model=model,
model_parameters=params
)
awesome-llm-json: No direct code comparison available as it's a curated list of resources rather than a library.
Summary
DeepSpeed is a powerful library for optimizing deep learning models, offering advanced features and extensive documentation. However, it has a steeper learning curve and may be more complex than necessary for smaller projects. awesome-llm-json, on the other hand, is a curated list of resources related to LLMs and JSON, providing a valuable reference but not offering direct implementation tools.
JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf
Pros of JARVIS
- More comprehensive AI agent framework with multi-modal capabilities
- Actively maintained by Microsoft with frequent updates
- Includes pre-trained models and extensive documentation
Cons of JARVIS
- Steeper learning curve due to complexity
- Requires more computational resources
- Less focused on JSON-specific LLM interactions
Code Comparison
JARVIS example:
from jarvis.core import Agent
agent = Agent()
response = agent.run("Describe the weather today")
print(response)
awesome-llm-json example:
import json
from llm_json import parse_json
result = parse_json("Generate a weather report for today")
print(json.dumps(result, indent=2))
Key Differences
JARVIS is a full-fledged AI agent framework with broader capabilities, while awesome-llm-json focuses specifically on JSON interactions with LLMs. JARVIS offers more features but requires more setup and resources. awesome-llm-json is simpler and more targeted for JSON-related tasks.
Use Cases
Choose JARVIS for complex, multi-modal AI applications. Opt for awesome-llm-json when working primarily with JSON data and LLMs in a lightweight manner.
Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
Pros of promptflow
- Comprehensive toolkit for building AI workflows and applications
- Integrates with Azure AI services and supports multiple LLM providers
- Offers visual flow designer and debugging tools
Cons of promptflow
- Steeper learning curve due to more complex features
- Primarily focused on Microsoft ecosystem and tools
- Requires more setup and configuration
Code comparison
promptflow:
from promptflow import tool
@tool
def my_python_tool(input1: str, input2: int) -> str:
return f"Input 1: {input1}, Input 2: {input2}"
awesome-llm-json:
{
"name": "My JSON Tool",
"input": {
"input1": "string",
"input2": "integer"
},
"output": "string"
}
Summary
promptflow is a more comprehensive solution for building AI workflows, offering integration with Azure services and visual tools. However, it has a steeper learning curve and is more Microsoft-centric.
awesome-llm-json focuses on providing a curated list of JSON-based LLM tools and resources, making it simpler to use but less feature-rich compared to promptflow.
The code comparison shows that promptflow uses Python decorators for tool definition, while awesome-llm-json relies on JSON schemas to describe tools and their inputs/outputs.
Examples and guides for using the OpenAI API
Pros of openai-cookbook
- Comprehensive collection of examples and best practices for using OpenAI's API
- Regularly updated with new features and improvements
- Officially maintained by OpenAI, ensuring accuracy and relevance
Cons of openai-cookbook
- Focused solely on OpenAI's products, limiting its scope for other LLM implementations
- May not cover as many specific JSON-related use cases as awesome-llm-json
Code Comparison
openai-cookbook example:
import openai
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello!"}]
)
awesome-llm-json example:
import json
from langchain import PromptTemplate, LLMChain
from langchain.llms import OpenAI
template = """Generate a JSON object for {entity}:"""
prompt = PromptTemplate(template=template, input_variables=["entity"])
The openai-cookbook focuses on direct API usage, while awesome-llm-json emphasizes JSON-specific implementations using various LLM frameworks. The cookbook provides a broader range of examples for OpenAI's products, whereas awesome-llm-json offers more specialized JSON-related use cases across different LLM platforms.
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
Pros of transformers
- Comprehensive library with support for numerous pre-trained models
- Extensive documentation and community support
- Regularly updated with new models and features
Cons of transformers
- Larger and more complex, potentially overwhelming for beginners
- May include unnecessary features for specific use cases
- Higher resource requirements due to its extensive functionality
Code comparison
transformers:
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
awesome-llm-json:
# No direct code comparison available as awesome-llm-json is a curated list
# of resources rather than a functional library
Summary
transformers is a comprehensive library for working with various transformer models, offering extensive functionality and support. It's regularly updated but can be complex for beginners. awesome-llm-json, on the other hand, is a curated list of resources focused on LLMs and JSON, providing a more targeted approach for specific use cases. While transformers offers direct implementation, awesome-llm-json serves as a valuable reference for relevant tools and libraries.
Convert 
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Awesome LLM JSON List
This awesome list is dedicated to resources for using Large Language Models (LLMs) to generate JSON or other structured outputs.
Table of Contents
- Terminology
- Hosted Models
- Local Models
- Python Libraries
- Blog Articles
- Videos
- Jupyter Notebooks
- Leaderboards
Terminology
Unfortunately, generating JSON goes by a few different names that roughly mean the same thing:
- Structured Outputs: Using an LLM to generate any structured output including JSON, XML, or YAML regardless of technique (e.g. function calling, guided generation).
- Function Calling: Providing an LLM a hypothetical (or actual) function definition for it to "call" in it's chat or completion response. The LLM doesn't actually call the function, it just provides an indication that one should be called via a JSON message.
- JSON Mode: Specifying that an LLM must generate valid JSON. Depending on the provider, a schema may or may not be specified and the LLM may create an unexpected schema.
- Tool Usage: Giving an LLM a choice of tools such as image generation, web search, and "function calling". The functional calling parameter in the API request is now called "tools".
- Guided Generation: For constraining an LLM model to generate text that follows a prescribed specification such as a Context-Free Grammar.
- GPT Actions: ChatGPT invokes actions (i.e. API calls) based on the endpoints and parameters specified in an OpenAPI specification. Unlike the capability called "Function Calling", this capability will indeed call your function hosted by an API server.
None of these names are great, that's why I named this list just "Awesome LLM JSON".
Hosted Models
Parallel Function Calling
Below is a list of hosted API models that support multiple parallel function calls. This could include checking the weather in multiple cities or first finding the location of a hotel and then checking the weather at it's location.
- anthropic
- claude-3-opus-20240229
- claude-3-sonnet-20240229
- claude-3-haiku-20240307
- azure/openai
- gpt-4-turbo-preview
- gpt-4-1106-preview
- gpt-4-0125-preview
- gpt-3.5-turbo-1106
- gpt-3.5-turbo-0125
- cohere
- command-r
- together_ai
- Mixtral-8x7B-Instruct-v0.1
- Mistral-7B-Instruct-v0.1
- CodeLlama-34b-Instruct
Local Models
Mistral 7B Instruct v0.3 (2024-05-22, Apache 2.0) an instruct fine-tuned version of Mistral with added function calling support.
C4AI Command R+ (2024-03-20, CC-BY-NC, Cohere) is a 104B parameter multilingual model with advanced Retrieval Augmented Generation (RAG) and tool use capabilities, optimized for reasoning, summarization, and question answering across 10 languages. Supports quantization for efficient use and demonstrates unique multi-step tool integration for complex task execution.
Hermes 2 Pro - Mistral 7B (2024-03-13, Nous Research) is a 7B parameter model that excels at function calling, JSON structured outputs, and general tasks. Trained on an updated OpenHermes 2.5 Dataset and a new function calling dataset, it uses a special system prompt and multi-turn structure. Achieves 91% on function calling and 84% on JSON mode evaluations.
Gorilla OpenFunctions v2 (2024-02-27, Apache 2.0 license, Charlie Cheng-Jie Ji et al.) interprets and executes functions based on JSON Schema Objects, supporting multiple languages and detecting function relevance.
NexusRaven-V2 (2023-12-05, Nexusflow) is a 13B model outperforming GPT-4 in zero-shot function calling by up to 7%, enabling effective use of software tools. Further instruction-tuned on CodeLlama-13B-instruct.
Functionary (2023-08-04, MeetKai) interprets and executes functions based on JSON Schema Objects, supporting various compute requirements and call types. Compatible with OpenAI-python and llama-cpp-python for efficient function execution in JSON generation tasks.
Hugging Face TGI enables JSON outputs and function calling for a variety of local models.
Python Libraries
DSPy (MIT) is a framework for algorithmically optimizing LM prompts and weights. DSPy introduced typed predictor and signatures to leverage Pydantic for enforcing type constraints on inputs and outputs, improving upon string-based fields.
FuzzTypes (MIT) extends Pydantic with autocorrecting annotation types for enhanced data normalization and handling of complex types like emails, dates, and custom entities.
guidance (Apache-2.0) enables constrained generation, interleaving Python logic with LLM calls, reusable functions, and calling external tools. Optimizes prompts for faster generation.
Instructor (MIT) simplifies generating structured data from LLMs using Function Calling, Tool Calling, and constrained sampling modes. Built on Pydantic for validation and supports various LLMs.
LangChain (MIT) provides an interface for chains, integrations with other tools, and chains for applications. LangChain offers structured outputs and tool calling across models.
LiteLLM (MIT) simplifies calling 100+ LLMs in the OpenAI format, supporting function calling, tool calling, and JSON mode.
LlamaIndex (MIT) provides modules for structured outputs at different levels of abstraction, including output parsers for text completion endpoints, Pydantic programs for mapping prompts to structured outputs using function calling or output parsing, and pre-defined Pydantic programs for specific output types.
Marvin (Apache-2.0) is a lightweight toolkit for building reliable natural language interfaces with self-documenting tools for tasks like entity extraction and multi-modal support.
Outlines (Apache-2.0) facilitates structured text generation using multiple models, Jinja templating, and support for regex patterns, JSON schemas, Pydantic models, and context-free grammars.
Pydantic (MIT) simplifies working with data structures and JSON through data model definition, validation, JSON schema generation, and seamless parsing and serialization.
PydanticAI (MIT) is a Python agent framework designed to make it less painful to build production grade applications with Generative AI.
SGLang (MPL-2.0) allows specifying JSON schemas using regular expressions or Pydantic models for constrained decoding. Its high-performance runtime accelerates JSON decoding.
SynCode (MIT) is a framework for the grammar-guided generation of Large Language Models (LLMs). It supports CFG for Python, Go, Java, JSON, YAML, and many more.
Mirascope (MIT) is an LLM toolkit that supports structured extraction with an intuitive python API.
Magnetic (MIT) call LLMs from Python using 3 lines of code. Simply use the @prompt decorator to create functions that return structured output from the LLM, powered by Pydantic.
Formatron (MIT) is an efficient and scalable constrained decoding library that enables controlling over language model output format using f-string templates that support regular expressions, context-free grammars, JSON schemas, and Pydantic models. Formatron integrates seamlessly with various model inference libraries.
Transformers-cfg (MIT) extends Hugging Face Transformers with context-free grammar (CFG) support via an EBNF interface. It enables grammar-constrained generation with minimal changes to existing code of transformers and supports JSON mode and JSON Schema.
Blog Articles
How fast can grammar-structured generation be? (2024-04-12, .txt Engineering) demonstrates an almost cost-free method to generate text that follows a grammar. It is shown to outperform llama.cpp by a factor of 50x on the C grammar.
Structured Generation Improves LLM performance: GSM8K Benchmark (2024-03-15, .txt Engineering) demonstrates consistent improvements across 8 models, highlighting benefits like "prompt consistency" and "thought-control."
LoRAX + Outlines: Better JSON Extraction with Structured Generation and LoRA (2024-03-03, Predibase Blog) combines Outlines with LoRAX v0.8 to enhance extraction accuracy and schema fidelity through structured generation, fine-tuning, and LoRA adapters.
FU, Show Me The Prompt. Quickly understand inscrutable LLM frameworks by intercepting API calls (2023-02-14, Hamel Husain) provides a practical guide to intercepting API calls using mitmproxy, gaining insights into tool functionality, and assessing necessity. Emphasizes minimizing complexity and maintaining close connection with underlying LLMs.
Coalescence: making LLM inference 5x faster (2024-02-02, .txt Engineering) shows how structured generation can be made faster than unstructured generation using a technique called "coalescence", with a caveat regarding how it may affect the quality of the generation.
Why Pydantic became indispensable for LLMs (2024-01-19, Adam Azzam) explains Pydantic's emergence as a critical tool, enabling sharing data models via JSON schemas and reasoning between unstructured and structured data. Highlights the importance of quantizing the decision space and potential issues with LLMs overfitting to older schema versions.
Getting Started with Function Calling (2024-01-11, Elvis Saravia) introduces function calling for connecting LLMs with external tools and APIs, providing an example using OpenAI's API and highlighting potential applications.
Pushing ChatGPT's Structured Data Support To Its Limits (2023-12-21, Max Woolf) delves into leveraging ChatGPT's capabilities using paid API, JSON schemas, and Pydantic. Highlights techniques for improving output quality and the benefits of structured data support.
Why use Instructor? (2023-11-18, Jason Liu) explains the benefits of the library, offering a readable approach, support for partial extraction and various types, and a self-correcting mechanism. Recommends additional resources on the Instructor website.
Using grammars to constrain llama.cpp output (2023-09-06, Ian Maurer) integrates context-free grammars with llama.cpp for more accurate and schema-compliant responses, particularly for biomedical data.
Using OpenAI functions and their Python library for data extraction (2023-07-09, Simon Willison) demonstrates extracting structured data using OpenAI Python library and function calling in a single API call, with a code example and suggestions for handling streaming limitations.
Videos
GPT Extracting Unstructured Data with Datasette and GPT-4 Turbo (2024-04-09, Simon Willison) showcases the datasette-extract plugin's ability to populate database tables from unstructured text and images, leveraging GPT-4 Turbo's API for data extraction.
LLM Structured Output for Function Calling with Ollama (2024-03-25, Andrej Baranovskij) demonstrates function calling-based data extraction using Ollama, Instructor and Sparrow agent.
Hermes 2 Pro Overview (2024-03-18, Prompt Engineer) introduces Hermes 2 Pro, a 7B parameter model excelling at function calling and structured JSON output. Demonstrates 90% accuracy in function calling and 84% in JSON mode, outperforming other models.
Mistral AI Function Calling (2024-02-24, Sophia Yang) demonstrates connecting LLMs to external tools, generating function arguments, and executing functions. Could be extended to generate or manipulate JSON data.
Function Calling in Ollama vs OpenAI (2024-02-13, Matt Williams) clarifies that models generate structured output for parsing and invoking functions. Compares implementations, highlighting Ollama's simpler approach and using few-shot prompts for consistency.
LLM Engineering: Structured Outputs (2024-02-12, Jason Liu, Weights & Biases Course) offers a concise course on handling structured JSON output, function calling, and validations using Pydantic, covering essentials for robust pipelines and efficient production integration.
Pydantic is all you need (2023-10-10, Jason Liu, AI Engineer Conference) discusses the importance of Pydantic for structured prompting and output validation, introducing the Instructor library and showcasing advanced applications for reliable and maintainable LLM-powered applications.
Jupyter Notebooks
Function Calling with llama-cpp-python and OpenAI Python Client demonstrates integration, including setup using the Instructor library, with examples of retrieving weather information and extracting user details.
Function Calling with Mistral Models demonstrates connecting Mistral models with external tools through a simple example involving a payment transactions dataframe.
chatgpt-structured-data by Max Woolf provides demos showcasing ChatGPT's function calling and structured data support, covering various use cases and schemas.## Leaderboards
Leaderboards
Berkeley Function-Calling Leaderboard (BFCL) is an evaluation framework for LLMs' function-calling capabilities including over 2k question-function-answer pairs across languages like Python, Java, JavaScript, SQL, and REST API, focusing on simple, multiple, and parallel function calls, as well as function relevance detection.
Top Related Projects
🦜🔗 The platform for reliable agents.
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf
Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
Examples and guides for using the OpenAI API
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot