Convert Figma logo to code with AI

uhop logostream-json

A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint — parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.

1,205
55
1,205
2
View on GitHubView on NPM

Top Related Projects

dominictarr's avatar

JSONStream

1,928

rawStream.pipe(JSONStream.parse()).pipe(streamOfObjects)

Quick Overview

Stream-JSON is a powerful Node.js library for parsing and processing large JSON files and streams. It provides a streaming API that allows for efficient handling of JSON data, even when dealing with files that are too large to fit into memory.

Pros

  • Efficient memory usage through streaming, allowing processing of very large JSON files
  • High performance due to its streaming nature and optimized parsing
  • Flexible and composable architecture with various plugins for different use cases
  • Supports both parsing and stringifying JSON data

Cons

  • Steeper learning curve compared to simple JSON.parse() for basic use cases
  • Requires more setup and code for simple JSON processing tasks
  • Limited browser support (primarily designed for Node.js environments)
  • May be overkill for small JSON files or simple parsing needs

Code Examples

Parsing a large JSON file:

const { parser } = require('stream-json');
const fs = require('fs');

const pipeline = fs.createReadStream('large-file.json').pipe(parser());

pipeline.on('data', data => {
  console.log(data);
});

pipeline.on('end', () => {
  console.log('Parsing completed');
});

Filtering specific keys from a JSON stream:

const { chain } = require('stream-chain');
const { parser } = require('stream-json');
const { pick } = require('stream-json/filters/Pick');
const { streamValues } = require('stream-json/streamers/StreamValues');

const pipeline = chain([
  fs.createReadStream('data.json'),
  parser(),
  pick({filter: 'users'}),
  streamValues(),
  data => data.value
]);

pipeline.on('data', user => {
  console.log(user);
});

Stringifying a large object:

const { Stringifier } = require('stream-json/Stringifier');

const stringifier = new Stringifier();

stringifier.on('data', chunk => {
  console.log(chunk.toString());
});

const largeObject = { /* ... */ };
stringifier.write(largeObject);
stringifier.end();

Getting Started

To use Stream-JSON in your project, first install it via npm:

npm install stream-json

Then, import and use it in your Node.js application:

const { parser } = require('stream-json');
const fs = require('fs');

const pipeline = fs.createReadStream('data.json').pipe(parser());

pipeline.on('data', data => {
  // Process each parsed token
  console.log(data);
});

pipeline.on('end', () => {
  console.log('Parsing completed');
});

This basic setup allows you to start parsing JSON streams efficiently. Explore the library's documentation for more advanced usage and available plugins.

Competitor Comparisons

dominictarr logo

JSONStream

1,928

rawStream.pipe(JSONStream.parse()).pipe(streamOfObjects)

Pros of JSONStream

  • Simpler API with fewer options, making it easier to get started
  • Lightweight with minimal dependencies
  • Well-established project with a longer history and wider adoption

Cons of JSONStream

  • Less flexible parsing options compared to stream-json
  • Limited support for custom transformations and advanced use cases
  • Slower performance for large JSON files or complex parsing scenarios

Code Comparison

JSONStream:

const JSONStream = require('JSONStream');
const fs = require('fs');

fs.createReadStream('data.json')
  .pipe(JSONStream.parse('*.name'))
  .on('data', console.log);

stream-json:

const {parser} = require('stream-json');
const {streamValues} = require('stream-json/streamers/StreamValues');
const fs = require('fs');

fs.createReadStream('data.json')
  .pipe(parser())
  .pipe(streamValues())
  .on('data', ({value}) => console.log(value.name));

Key Differences

  • JSONStream uses a simpler, more concise syntax for basic parsing tasks
  • stream-json offers more granular control over parsing and streaming
  • stream-json provides better performance for large-scale data processing
  • JSONStream is more suitable for quick, straightforward JSON parsing needs
  • stream-json excels in scenarios requiring advanced customization and optimization

Both libraries have their strengths, and the choice between them depends on the specific requirements of your project, such as performance needs, parsing complexity, and desired level of control over the streaming process.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

stream-json NPM version

stream-json is a micro-library of components for processing JSON files and streams, with a minimal memory footprint. Point it at a document far larger than available memory and it streams straight through — you pick out only the parts you care about and handle them one at a time, instead of loading the whole thing with JSON.parse. Even individual keys, strings, and numbers can be streamed piece by piece, and a SAX-inspired event API is included.

Each component is one stage of a pipeline: the parser turns text into a token stream, filters trim and reshape that stream on the fly, and streamers assemble the surviving tokens back into JavaScript objects. They compose with each other and with your own code through stream-chain, the zero-dependency library this one is built on; TypeScript typings are bundled.

Why it might be for you:

  • Surgical. pick, ignore, replace, and filter keep just the subobjects you want out of a massive document and drop the rest — the bytes you skip are never assembled into memory.
  • Composable. Every component is an ordinary pipeline stage. Mix them with each other, with plain functions and generators, and with any Node or Web stream.
  • Performance-minded. The parser and assemblers are measured and tuned along the hot paths. Real numbers depend on your data and hardware, so benchmark on your own.
  • Solid. ESM, bundled TypeScript typings, and a broad test suite exercised across Node, Bun, Deno, and the browser.

Example

Pull one array out of a JSON document larger than memory and tally it — one record at a time, in constant memory:

import {parser} from 'stream-json';
import {pick} from 'stream-json/filters/pick.js';
import {streamArray} from 'stream-json/streamers/stream-array.js';
import chain from 'stream-chain';
import fs from 'node:fs';

// data.json: { "meta": {...}, "data": [ ...millions of records... ] }
const pipeline = chain([
  fs.createReadStream('data.json'), // a file far bigger than RAM is fine
  parser(),
  pick({filter: 'data'}), // descend into "data", ignore everything else
  streamArray() // emit one array element at a time
]);

const byDepartment = {};
pipeline.on('data', ({value}) => {
  byDepartment[value.department] = (byDepartment[value.department] ?? 0) + 1;
});
pipeline.on('end', () => console.log(byDepartment));

Each stage is a building block; stream-chain wires them into one stream and handles the streaming and backpressure. To read straight from a file you can drop createReadStream and use the Node-only parseFile(); to write a stream back to disk, use stringerToFile(). See Recipes for more.

Installation

npm install --save stream-json

ESM only; runs on actively-maintained Node, plus Bun and Deno — see Supported runtimes.

What's in the box

  • Parsers — JSON, JSONL (the fast path for line-delimited data dumps), and JSONC (JSON with comments).
  • Helpers for streamed objects — pick out the parts you want, assemble objects from tokens, output JSON back, and read or write files directly.
  • Runtimes — works out of the box on Node (Node streams); browser-safe (Web Streams) and dependency-free Core (no streams) builds are there when you need them.

Want the details? Follow the links, or browse the wiki — index or search.

Companion projects

  • stream-chain — the pipeline-composition substrate stream-json is built on (wire functions, generators, and streams into one chain); also home to streaming JSONL.
  • stream-csv-as-json — streams huge CSV files in a stream-json-compatible token format: rows as arrays of strings, or as objects when a header row is present.

License

BSD-3-Clause

Release History

  • 3.5.0 New maxDepth option on path filters. Thx ataberk.xyz.
  • 3.4.0 Deprecated stream-json/utils/{pipe,drain} — use stream-chain's generic helpers directly; the file-edge internals now delegate to stream-chain.
  • 3.3.0 File I/O components (parseFile, stringerToFile, verifyFile), faithful JSONC comma round-trip (streamCommas / useCommas), JSONL delegated to stream-chain.
  • 3.2.0 Improvements in TS typings, faster JSON parser.
  • 3.1.0 Web Streams parity sweep.
  • 3.0.0 Moved to ESM using stream-chain 4.x. See Migrating from 2.x to 3.x.
  • 2.1.0 new: jsonc/Verifier — validates JSONC text with exact error locations. Parser performance improvements (pre-allocated token singletons).
  • 2.0.0 major rewrite: functional API based on stream-chain 3.x, bundled TypeScript definitions. New: JSONC parser/stringer, FlexAssembler. See Migrating from 1.x to 2.x.

The full history is in the wiki: Release history.

NPM DownloadsLast 30 Days