How to remove duplicate rows from a CSV or Excel file

Duplicates creep in when exports overlap, a report runs twice, or two people paste the same rows. Removing them is one click once the file is open, and the file never leaves your computer.

The one-click way

Open NoEgress, drop your CSV, Excel, TSV, JSON or Parquet file, and the Start page shows how many repeated rows it found. Click Remove duplicates.

That keeps the first copy of every row that is identical across all columns, the same default as Excel’s Remove Duplicates, and opens the result in the Table view.

Match on a key instead of every column

Often two rows are the same record even though one column differs: the same order exported on two days with a different timestamp. In the Table view, the Remove duplicates step lets you tick the columns that identify a record, such as order_id, and choose whether to keep the first or the last copy.

Keeping the last copy is the usual choice when later rows are corrections of earlier ones.

Check what you are about to remove

Before deleting anything, look at the repeated keys and how often they repeat. A handful is normal overlap. Thousands usually means a join or an export went wrong upstream, and removing them hides the real problem.

SELECT
  order_id,
  COUNT(*) AS copies
FROM orders
GROUP BY order_id
HAVING COUNT(*) > 1
ORDER BY copies DESC;

The same thing in SQL

SELECT DISTINCT removes rows that are identical in every column. To keep one row per key, number the copies and keep the first; QUALIFY does that without a subquery.

SELECT *
FROM orders
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY order_id
  ORDER BY updated_at DESC
) = 1;

Save it and run it again next month

Click Save recipe for next time and the step is kept in your browser. When next month’s export arrives, open it and run the saved recipe from the Start page. Then download the result as CSV, Excel or Parquet.