How to open large CSV files Excel can't handle

Excel caps a sheet at 1,048,576 rows. Open a bigger file and it drops the rest, often without a clear warning. That’s worse than an error, because your numbers look fine and they’re wrong. Here’s how to work with files past that limit.

First, make sure you’re seeing every row

If a file opens in a spreadsheet and ends at exactly 1,048,576 rows, it was cut off. Check the count against the source before you conclude anything.

No row count from the source? Divide the file size by a typical row length. If you land in the same ballpark, you’re fine. If not, something got cut.

SELECT COUNT(*) AS row_count FROM orders;

Why a columnar engine copes better

A spreadsheet keeps every cell in memory along with its formatting and formula state. That’s why memory use shoots up on big files. DuckDB stores each column as a typed array and only reads the columns your query uses.

Select three columns from a fifty-column file and the other forty-seven barely cost anything. Aggregations run in a fraction of the memory.

Aggregate, don’t scroll

Nobody reads a million rows. You open a big file to shrink it: a total, a distribution, or a filtered slice small enough to look at.

Do that in the query instead of scrolling, and file size stops being a problem.

SELECT
  DATE_TRUNC('month', order_date) AS month,
  COUNT(*)                        AS orders,
  SUM(revenue)                    AS revenue
FROM orders
GROUP BY month
ORDER BY month;

Profile before you trust the numbers

Big files usually come out of an export pipeline, and pipelines leave junk behind: nulls, a header row repeated halfway down, dates in two formats. Check null counts, distinct values, and min and max for each column first. It takes a minute and keeps a bad total out of your report.

How big is too big?

In NoEgress, large CSVs stream from disk into DuckDB instead of loading into page memory, so there’s no fixed row limit. In our tests a 25-million-row, 1.7 GB file loads in under half a minute. Your laptop sets the limit, not an upload quota.

If you outgrow the browser, desktop DuckDB runs the same SQL. Every query in this guide works there unchanged.