Large CSV Imports: What Breaks, and at Exactly What Number

Large CSV imports fail at four predictable points, each with a number attached. Excel stops at 1,048,576 rows. A browser cannot hold a CSV in one JavaScript string beyond roughly 536 million characters. Hosting layers cut the request off long before a large file finishes: Heroku at 30 seconds, AWS Application Load Balancer and nginx at 60, Cloudflare at 125. Sizing an import means knowing which ceiling you hit first.

All figures below were checked against vendor documentation on 11 August 2026 and are linked inline so you can verify each one.

A single Excel worksheet holds a maximum of 1,048,576 rows by 16,384 columns, and a single cell holds at most 32,767 characters, per Microsoft's published specifications.

This matters less as a technical constraint than as a diagnostic. Anyone asking how to process a 5 million line CSV has already left the spreadsheet world, and "just open it in Excel" is not an answer available to them. It also explains a common support pattern: a customer opens a large file to inspect it, Excel silently truncates at 1,048,576 rows, and then reports that rows are missing from your product when the truncation happened on their desktop.

The 32,767 character cell limit surprises people most, usually when a free-text notes field or a serialized JSON blob has been stuffed into a spreadsheet column.

If your importer reads a file into a single JavaScript string before parsing it, there is a hard cap. On current 64-bit builds of V8, the engine behind Chrome and Node.js, the maximum length of one string is 536,870,888 characters. The value is defined in V8's own header as (1 << 29) - 24 for 64-bit builds, and Node exposes it at runtime as buffer.constants.MAX_STRING_LENGTH.

Three caveats matter if you are going to rely on this number:

It is build dependent. On 32-bit builds the cap is 268,435,440 characters, and it has changed across versions. Node 12 reported a little over 1 billion before the limit dropped in Node 14. Read it at runtime rather than hardcoding it.

It counts UTF-16 code units, not bytes. Calling it "512 MiB" is only accurate for a string V8 has stored with one byte per character. A two-byte-backed string of the same length occupies roughly twice that in memory.

The practical limit is well below the theoretical one. A parsed representation is substantially larger than the raw text, because every row becomes objects and every field becomes a separate string with its own overhead. A file that fits in a string comfortably can still exhaust memory once parsed.

A separate limit applies to arrays. A JavaScript array cannot exceed 4,294,967,295 elements, which is 2^32 minus 1, as specified in ECMA-262. In practice this is almost never the binding constraint, because memory runs out long before four billion rows.

This is where most real failures happen, and almost nobody names the numbers. A large import sent as a single HTTP request has to complete before the shortest timeout in your stack fires. Those timeouts belong to infrastructure you may not have configured and may not think about.

Heroku terminates at 30 seconds. The Heroku router will terminate a request that takes longer than 30 seconds and return an H12 error. The mechanism is worth stating precisely: the 30 second countdown begins after the entire request has been sent from the router to the dyno, so upload time does not count against it. Once a first byte of response is sent, a rolling 55 second window applies to each subsequent byte. The 30 second limit is not configurable.

AWS Application Load Balancer defaults to 60 seconds. Elastic Load Balancing sets the idle timeout to 60 seconds by default, configurable from 1 to 4000 seconds. This is the Application Load Balancer connection idle timeout specifically. Network and Classic Load Balancers behave differently.

nginx defaults to 60 seconds. Both proxy_read_timeout and proxy_send_timeout default to 60 seconds. The nuance that catches people out is that this is a timeout between two successive read operations, not a deadline for the whole response. A slow but steady response survives. A response that goes quiet for 61 seconds while your application chews through rows does not.

Cloudflare returns error 524 at 125 seconds. Cloudflare returns a 524 when the origin does not respond within the default 125 second Proxy Read Timeout. Only Enterprise zones can raise it. If you have seen 100 seconds quoted, that was the historical default and it is still repeated widely in community threads.

Stack these and the effective limit is the smallest one in the path. An application on Heroku behind Cloudflare has 30 seconds, not 125.

The fourth ceiling has no single number, which is why it gets skipped. Writing rows one at a time means one round trip per row, and each round trip carries network latency, query parsing and transaction overhead. That cost is tolerable at a thousand rows and untenable at a million.

Bulk load paths exist for this. Multi-row inserts, batched transactions, and native bulk loaders such as COPY in PostgreSQL or LOAD DATA INFILE in MySQL amortize the per-statement overhead across many rows at once. The mechanism is the point rather than any throughput figure, because real throughput depends on row width, index count, whether constraints and triggers fire per row, and hardware. Anyone quoting a universal rows-per-second number is quoting a benchmark from a machine that is not yours.

The practical rule: if your import writes row by row, the database will become the ceiling somewhere in the hundreds of thousands of rows, and the fix is a change of write strategy rather than a bigger instance.

The table is about the first ceiling, not the only one. A 3 million row import that survives the timeout still has to survive the write.

A local test skips almost everything that causes production import failures. There is no load balancer, no reverse proxy and no CDN between the browser and the application, so the 30, 60 and 125 second limits above do not exist. Local disk is faster than network transfer, the database is usually smaller, and no concurrent traffic competes for connections.

The result is an import that completes locally in 40 seconds and fails in production for reasons unrelated to the code. Test large imports through the same proxy and CDN configuration production uses, or you are testing a different system.

Streaming means reading and processing the file in pieces rather than loading all of it first. A streaming parser holds roughly constant memory whether the file is 10 MB or 2 GB, because at any moment it holds only the current chunk.

Chunking changes which ceiling applies. The browser string cap stops mattering, because no single string ever holds the file. Memory stops scaling with file size. Progress becomes reportable, which matters more than it sounds: a user watching a progress indicator waits, and a user watching a frozen tab refreshes it.

Chunking does not remove the timeout ceiling by itself. A chunked parser inside one HTTP request still has to finish before the shortest timeout fires. Removing that ceiling means moving the work off the request, into a background job or into the browser.

Dromo offers three architectures, and the right one depends on file size and where the data is allowed to go.

Private Mode processes the import in the end user's browser. The file is parsed, validated and corrected client side, and Dromo does not see the end user's data. There is no upload of file contents to Dromo servers, so the request timeout ceiling does not apply to the file transfer at all.

Headless import with SFTP handles scheduled and server-side ingestion. For recurring partner feeds and nightly batches, files arrive without a browser or a human involved, which removes the interactive upload from the equation entirely. This is the appropriate path for the largest files.

Self-hosted deployment runs the whole system on your own infrastructure, for when data cannot leave your environment. Results can be written directly to Amazon S3, Google Cloud Storage, Azure Blob Storage or Dropbox.

Dromo does not publish throughput figures, processing times or a maximum file size, and this page will not invent them. The useful framing is architectural: interactive uploads suit files a person is waiting on, and scheduled server-side ingestion suits files measured in millions of rows. Plan details are on the Dromo pricing page, the technical documentation is at developer.dromo.io, and the architecture is described on the Dromo data privacy page.

Speed is only half the problem. A 2 million row import that completes and writes 40,000 malformed records has not succeeded.

Find the failing rows without reading the file. At a million rows, nobody scrolls. What is needed is a filtered view of only the failing rows, grouped by the rule they violated, so one bad date format across 12,000 rows presents as one problem rather than 12,000.

Correct in bulk, not one at a time. Most large-file errors are systematic rather than random: a single date format, a currency symbol, a trailing space. These want one correction applied across every affected row.

Decide in advance what happens to rows that cannot be fixed. There are three options: reject the whole file, import the valid rows and report the rest, or hold everything pending review. All three are defensible. Discovering that you never chose, while an import is half written, is not.

There is no single answer, and any vendor quoting one is quoting their own hardware. Time depends on row width, validation rules, write strategy and infrastructure. The more useful question is which ceiling you hit first: at 5 million rows, request timeouts and write strategy matter far more than raw parsing speed.

Because the request outlives a timeout somewhere in the stack. Heroku terminates at 30 seconds, AWS Application Load Balancer and nginx default to 60, and Cloudflare returns error 524 at 125. The effective limit is the shortest one in your path, and a single-request upload of millions of rows survives none of them.

Yes, if it streams. Reading the file into one JavaScript string caps out around 536,870,888 characters on 64-bit V8. A streaming parser that processes chunks holds roughly constant memory regardless of file size, so row count stops being the constraint and available memory for parsed data becomes it.

The CSV format itself sets no limit, since it is plain text. Limits come from whatever reads it. Excel stops at 1,048,576 rows per worksheet. A browser reading the file into a single string is capped near 536 million characters. Your database and timeouts usually bite first.

Move the work off the request. Either process the file in the browser so no long upload exists, or accept the file and hand it to a background job that returns a job ID the frontend polls for progress. For recurring large files, scheduled ingestion over SFTP or an API avoids the interactive request entirely.

Filter to only failing rows and group them by the rule they broke, so a systematic error appears once rather than thousands of times. Apply corrections in bulk across all affected rows. Decide in advance whether unfixable rows block the file, are skipped and reported, or hold the import for review.