Repository navigation
Proposal: Improve partial CSV processing performance by enabling buffer‑based operations #66658
Description
What is the problem this feature will solve?
Many Node.js users need to process CSV files partially — for example, reading only specific rows or columns without loading the entire file into the V8 heap.
However, most existing CSV libraries convert the entire CSV into strings or objects, which:
-increases heap usage
-triggers unnecessary garbage collection
-slows down partial data extraction
-wastes resources when only a small portion of the CSV is needed
Why this matters
For large CSV files (hundreds of MB or several GB), loading the entire file into memory is inefficient.
A buffer‑based approach allows users to extract only the required segments without converting the whole file into strings.
What is the feature you are proposing to solve the problem?
Example solution
I created an npm module called getdatafromcsvbuffer, which manipulates CSV data directly from raw Buffer operations.
Key characteristics:
operates directly on Buffer
-avoids string conversion and split()
-significantly reduces heap usage
-minimizes GC pressure
-ideal for partial CSV extraction
This approach demonstrates that CSV processing can be much faster when using Buffer‑only operations.
Reference
Package: https://www.npmjs.com/package/getdatafromcsvbuffer
Proposal
I would like to propose discussing whether Node.js could:
1.Provide official utilities for partial CSV processing using Buffer
2.Offer APIs that help developers manipulate structured text formats without converting entire files into strings
3.Consider exposing lower‑level primitives that make buffer‑based parsing easier and more ergonomic
This could benefit workloads involving:
-machine learning data preprocessing
-ETL pipelines
-log processing
-large CSV datasets
-streaming analytics
I would appreciate feedback from the Node.js team on whether such functionality aligns with Node’s direction or could be considered for future enhancements.
I will not hesitate to give you the benchmark if you need, so please ask me .
What alternatives have you considered?
No response
Activity
om0852 commentedon Oct 12, 2026
Thanks for opening this proposal, @FukuzawaTetsu!
The problem you highlighted is very real: in high-throughput data processing and large CSV/log ingestion pipelines, converting multi-megabyte or gigabyte files into V8 JavaScript strings via
.toString()or.split()introduces massive heap overhead (due to UTF-16 representation and intermediate string allocations) and heavy garbage collection pressure.To quantify the performance benefits of your proposal, I put together a benchmark comparing a 32.6 MB CSV (300,000 rows, 10 columns) targeting partial cell extraction (seeking row 150,000, column
email):Benchmark Results (Node.js with
--expose-gc)toString() + split())147.45 ms52.35 MB89.03 MB222.26 ms2.57 MB71.93 MBgetdatafromcsvbufferlogic)81.09 ms0.71 MB38.17 MBindexOf+subarray)47.82 ms1.68 MB39.12 MBKey Takeaways
Buffer.prototype.indexOfis backed by SIMD vector instructions in Node's native C++ layer, byte-level scanning is over 3x faster than V8 string splitting.Note on
getdatafromcsvbuffer(Quick Bug Fix)While benchmarking your package, Node.js threw:
FukuzawaTetsu commentedon Oct 13, 2026
@om0852
Thank you for your reply and for running the benchmark.
Could you explain how the "4. Zero-Copy Buffer Scanning (indexOf + subarray)" approach locates and extracts the target cell from the CSV buffer?
For example, if the user wants to retrieve a specific row and a specific column (identified by the column name), how is that achieved using "4. Zero-Copy Buffer Scanning (indexOf + subarray)" approach?
In getdatafromcsvbuffer, I also traverse the Buffer directly and use buffer.slice() to extract only the required portion.
As far as I understand, buffer.slice() and buffer.subarray() both create views on the same underlying memory without copying data, so I would expect similar behavior.
Therefore, I would like to better understand what accounts for the significant performance difference between:
and
If you could share additional details about the benchmark implementation, I would greatly appreciate it.