Skip to content

Commit a2f4446

Browse files
author
mochatek
committed
refactor: Handle edge cases, organize files and add unit tests
1 parent 37455df commit a2f4446

File tree

14 files changed

+1010
-593
lines changed

14 files changed

+1010
-593
lines changed

‎README.md‎

Lines changed: 132 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -1,100 +1,162 @@
1-
# jsonarrayfs <img alt="NPM Version" src="https://img.shields.io/npm/v/jsonarrayfs"> <img alt="Monthly Downloads" src="https://img.shields.io/npm/dm/jsonarrayfs" /> <img alt="GitHub Actions Workflow Status" src="https://img.shields.io/github/actions/workflow/status/mochatek/jsonarrayfs/publish-to-npm.yml">
1+
# jsonarrayfs
22

3-
"jsonarrayfs" is a Node.js library crafted for robust and memory-efficient management of massive JSON array files. It enables seamless handling of JSON arrays without the need to load the entire file into memory, making it perfect for efficiently managing large datasets without overwhelming system resources.
3+
[![npm version](https://img.shields.io/npm/v/jsonarrayfs.svg)](https://www.npmjs.com/package/jsonarrayfs)
4+
[![npm downloads](https://img.shields.io/npm/dm/jsonarrayfs.svg)](https://www.npmjs.com/package/jsonarrayfs)
5+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
6+
![Coverage](https://img.shields.io/badge/coverage-96.88%25-brightgreen)
47

5-
## 🎯 Key Features
8+
Specialized Node.js library for memory-efficient operations on JSON array files. Stream individual elements from large JSON array files and append new elements without loading the entire array into memory. Perfect for processing large-scale JSON array datasets without memory limitations.
69

7-
- **Stream Processing**: Read JSON array in manageable chunks (eg: 50k elements at a time) using stream.
8-
- **On-the-Fly Filtering**: Apply filter to the stream to fetch only relevant data, reducing the data you handle even further.
9-
- **Direct Appends**: Append new elements directly to the JSON array file, avoiding unnecessary loading, modification and rewriting.
10-
- **Formatting Flexibility**: Works regardless of whether the JSON file is formatted with proper indentation or not.
11-
- **Handles Mixed Types**: Can handle JSON arrays containing both uniform and mixed types of elements.
10+
## Why Use This?
1211

13-
## 💡 Benefits
12+
- 🎯 **Specialized**: Purpose-built for JSON array files
13+
- 💾 **Memory Efficient**: Process arrays of any size without loading them entirely
14+
- ⚡ **High Performance**: Optimized streaming and batch operations
15+
- ✍️ **Direct Updates**: Append elements without rewriting the entire file
16+
- 🔄 **Format Agnostic**: Works with any valid JSON array structure
1417

15-
- **Memory Optimization**: Process JSON array files with minimal memory usage, making it ideal for resource-constrained environments.
16-
- **Handles Large Datasets**: Efficiently manage massive JSON array files without memory limitations.
17-
- **Improved Performance**: Faster processing times due to efficient streaming, filtering and appending capabilities.
18-
- **Enhanced Scalability**: Scales seamlessly with growing datasets, ensuring smooth performance.
18+
## Installation
1919

20-
## ⚙️ Installation
21-
22-
To install jsonarrayfs, use:
23-
24-
```sh
20+
```bash
2521
npm install jsonarrayfs
2622
```
2723

28-
## 🚀 Usage
24+
## Usage
25+
26+
```typescript
27+
import { JsonArrayStream, appendToJsonArrayFile } from "jsonarrayfs";
28+
import { createReadStream } from "node:fs";
29+
import { Transform } from "node:stream";
30+
31+
// Process a large application log file (10GB+ JSON array)
32+
const fileStream = createReadStream("app.log.json");
33+
const arrayStream = new JsonArrayStream("utf8");
34+
35+
// Analyze logs: Count errors and slow responses
36+
let errorCount = 0;
37+
let slowResponses = 0;
38+
39+
for await (const log of fileStream.pipe(arrayStream)) {
40+
if (log !== JsonArrayStream.NULL) {
41+
if (log.level === "error") {
42+
errorCount++;
43+
console.error(`Error in ${log.service}: ${log.message}`);
44+
}
45+
46+
if (log.responseTime > 1000) {
47+
slowResponses++;
48+
console.warn(`Slow response: ${log.path} (${log.responseTime}ms)`);
49+
}
50+
}
51+
}
2952

30-
- Stream Processing:
53+
console.log(
54+
`Analysis complete: ${errorCount} errors, ${slowResponses} slow responses`,
55+
);
3156

32-
```ts
33-
import { createReadStream } from "jsonarrayfs";
57+
// Append new log entries
58+
const newLogs = [
59+
{
60+
timestamp: Date.now(),
61+
level: "info",
62+
service: "auth",
63+
path: "/api/login",
64+
responseTime: 245,
65+
message: "User login successful",
66+
},
67+
{
68+
timestamp: Date.now(),
69+
level: "info",
70+
service: "auth",
71+
path: "/api/login",
72+
responseTime: 1245,
73+
message: "User login successful",
74+
},
75+
null,
76+
{
77+
timestamp: Date.now(),
78+
level: "error",
79+
service: "payment",
80+
path: "/api/checkout",
81+
responseTime: 1532,
82+
message: "Database connection timeout",
83+
},
84+
];
3485

35-
// Option 1: Create a streamer to read JSON array elements from a file
36-
const streamer = await createReadStream("./data.json", { encoding: "utf-8" });
86+
await appendToJsonArrayFile("app.log.json", "utf8", ...newLogs);
87+
```
3788

38-
// Option 2: Create a streamer to read JSON array elements from an existing Readable:
89+
## API
3990

40-
// From a file
41-
import fs from "fs";
42-
const readStream = fs.createReadStream("./data.json", { encoding: "utf-8" });
43-
const streamer = await createReadStream(readStream);
91+
### JsonArrayStream
4492

45-
// From an API response
46-
import { ReadableStream } from "stream/web";
47-
import { Readable } from "stream";
48-
const response = await fetch("https://www.example.com/json-data");
49-
const readableStream = Readable.fromWeb(
50-
response.body as unknown as ReadableStream,
51-
);
52-
const streamer = await createReadStream(readableStream);
93+
A transform stream that reads JSON array files and emits elements one by one for efficient processing. When processing arrays containing `null` values, it uses a special sentinel value (`JsonArrayStream.NULL`) to distinguish between JSON `null` and stream EOF.
5394

54-
// Stream JSON array elements in batches of 100
55-
for await (const elements of streamer.batch(100)) {
56-
// Your processing logic here
57-
}
95+
#### Constructor
96+
97+
```typescript
98+
new JsonArrayStream(encoding?: string)
5899
```
59100

60-
- On-the-Fly Filtering:
101+
##### Parameters
61102

62-
```ts
63-
import { createReadStream } from "jsonarrayfs";
103+
- `encoding` (string, optional): File encoding (default: 'utf8')
64104

65-
const streamer = await createReadStream<{ offer: boolean; price: number }>(
66-
"./data.json",
67-
{ encoding: "utf-8" },
68-
);
105+
#### Properties
69106

70-
// Add filter to the batch to fetch only relevant elements
71-
for await (const elements of streamer.batch(
72-
100,
73-
(element) => element.price < 500 || element.offer,
74-
)) {
75-
// Your processing logic here
76-
}
77-
```
107+
- `JsonArrayStream.NULL`: Special sentinel value to distinguish between JSON `null` and stream EOF
78108

79-
- Append data to existing JSON array:
109+
#### Events
80110

81-
```ts
82-
import { appendFile } from "jsonarrayfs";
111+
- `data`: Emitted for each array element
112+
- `error`: Emitted when parsing fails or input is invalid
83113

84-
// Simulate new data to append
85-
const newData = [
86-
{ id: 1, name: "Earth", price: 1000, offer: true },
87-
{ id: 2, name: "Moon", price: 500, offer: false },
88-
];
114+
### appendToJsonArrayFile
89115

90-
// Append new data to the existing JSON array file
91-
await appendFile("./data.json", "utf-8", ...newData);
116+
Appends elements to a JSON array file efficiently without loading the entire file into memory.
117+
118+
#### Signature
119+
120+
```typescript
121+
async function appendToJsonArrayFile(
122+
filePath: string,
123+
encoding?: string,
124+
...elements: any[]
125+
): Promise<void>;
92126
```
93127

94-
## 🤝 Contributing
128+
#### Parameters
129+
130+
- `filePath` (string): Path to the JSON array file
131+
- `encoding` (string, optional): File encoding (default: 'utf8')
132+
- `...elements` (any[]): Elements to append to the array
133+
134+
#### Returns
135+
136+
Promise that resolves when the append operation is complete.
137+
138+
## Error Handling
139+
140+
The library can throw these errors:
141+
142+
### `JsonArrayStreamError`
143+
144+
- Invalid JSON array format
145+
- Malformed array elements
146+
- Unexpected end of input
147+
148+
### `JsonArrayAppendError`
149+
150+
- Invalid JSON array format
151+
- File system errors
152+
- Permission issues
153+
- Invalid input elements
154+
155+
## Requirements
95156

96-
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
157+
- Node.js >= 16.0.0
158+
- Input file must be a valid JSON array
97159

98-
## 📜 License
160+
## License
99161

100-
[MIT License ](https://github.com/mochatek/jsonarrayfs/blob/main/LICENSE)
162+
MIT License - see the [LICENSE](LICENSE) file for details

‎package.json‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "jsonarrayfs",
33
"version": "1.2.1",
4-
"description": "Efficiently handle JSON array files in Node.js with minimal memory usage. Perfect for processing large data volumes without worrying about memory limitations.",
4+
"description": "Specialized Node.js library for memory-efficient operations on JSON array files. Stream individual elements from large JSON array files and append new elements without loading the entire array into memory. Perfect for processing large-scale JSON array datasets without memory limitations.",
55
"main": "./dist/index.js",
66
"module": "./dist/index.mjs",
77
"types": "./dist/index.d.ts",
@@ -26,13 +26,16 @@
2626
"coverage": "c8 npm test"
2727
},
2828
"keywords": [
29-
"json",
30-
"stream",
31-
"chunk",
32-
"batch",
33-
"json file handling",
34-
"json array",
35-
"json stream",
29+
"json-array",
30+
"json-array-stream",
31+
"json-array-append",
32+
"memory-efficient",
33+
"large-file",
34+
"streaming",
35+
"json-array-parser",
36+
"array-manipulation",
37+
"file-processing",
38+
"node-json",
3639
"stream filter",
3740
"json append",
3841
"stream processing",

‎src/index.ts‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
1-
import JsonArrayStreamer from "./modules/JsonArrayStreamer";
2-
import Transform from "./modules/JsonArrayTransformer";
3-
import appendFile from "./modules/JsonArrayAppend";
1+
import JsonArrayStream from "./modules/JsonArrayStream";
2+
import appendToJsonArrayFile from "./modules/JsonArrayAppend";
43

5-
const createReadStream = JsonArrayStreamer.create;
6-
7-
export { createReadStream, appendFile, Transform };
4+
export { JsonArrayStream, appendToJsonArrayFile };

0 commit comments

Comments
 (0)