How to Remove Duplicates From a List or Array (and Keep the Original Order)
Order-preserving and fast ways to deduplicate lists in Python, arrays in JavaScript and collections in Java, including duplicates of objects by a key — and the quadratic approach to avoid.
By Software Side Hustles Editorial Team · Published · Updated · 4 min read
Quick answer: in Python use list(dict.fromkeys(items)), in JavaScript [...new Set(items)], and in Java new ArrayList<>(new LinkedHashSet<>(items)). All three remove duplicates in linear time and keep the first occurrence of each item in its original position. Converting to a plain set() in Python is also fast, but it throws the order away.
Deduplicate by language
| Language | Keeps order | Code |
|---|---|---|
| Python | Yes | list(dict.fromkeys(items)) |
| Python | No | list(set(items)) |
| JavaScript | Yes | [...new Set(items)] |
| Java | Yes | new ArrayList<>(new LinkedHashSet<>(items)) |
| Java (streams) | Yes | items.stream().distinct().toList() |
| C# | Yes (in practice) | items.Distinct().ToList() |
| SQL | n/a | SELECT DISTINCT col FROM t |
How do I remove duplicates from a list in Python?
items = [3, 1, 3, 2, 1]
list(dict.fromkeys(items)) # [3, 1, 2] — order preservedlist(set(items)) # some order — do not rely on itsorted(set(items)) # [1, 2, 3] — when you want sorted output anywaydict.fromkeys works because dictionaries preserve insertion order (guaranteed since Python 3.7) and ignore repeated keys. A set has no defined order: small integers often look sorted, which is how this bug survives testing and then appears with strings in production.
If you need more control — for example, to skip items as you go — write the loop explicitly with a seen set:
def dedupe(items): seen = set() out = [] for x in items: if x not in seen: seen.add(x) out.append(x) return outHow do I remove duplicates from an array in JavaScript?
const items = [3, 1, 3, 2, 1];const unique = [...new Set(items)]; // [3, 1, 2]A Set remembers insertion order, so this is order-preserving. It compares values with SameValueZero: primitives work as expected, NaN equals NaN, but objects are compared by reference. Two separate { id: 1 } objects are both kept.
The filter/indexOf version you will see in older answers is correct but O(n²):
items.filter((x, i) => items.indexOf(x) === i); // fine for 50 items, slow for 50,000How do I remove duplicate objects by a key?
Most real-world deduplication is "same user ID" or "same email", not identical objects.
JavaScript — keep the first occurrence per key:
function uniqueBy(items, key) { const seen = new Set(); return items.filter((item) => { const k = key(item); if (seen.has(k)) return false; seen.add(k); return true; });}
uniqueBy(users, (u) => u.email.toLowerCase());To keep the last occurrence instead, build a Map: [...new Map(items.map((u) => [u.id, u])).values()]. Later entries overwrite earlier ones, but each key keeps the position where it first appeared.
Python — the same idea:
def unique_by(items, key): seen = set() return [x for x in items if not (key(x) in seen or seen.add(key(x)))]That one-liner is clever but hard to read; the explicit loop from the previous section is usually the better choice in shared code.
How do I remove duplicates from a List in Java?
List<Integer> items = List.of(3, 1, 3, 2, 1);List<Integer> unique = new ArrayList<>(new LinkedHashSet<>(items)); // [3, 1, 2]List<Integer> viaStream = items.stream().distinct().toList(); // [3, 1, 2]HashSet would lose the order; LinkedHashSet keeps it; TreeSet sorts. For your own classes, deduplication depends on equals() and hashCode(). Override one without the other and duplicates slip through, because hash-based collections check the hash first. Java records generate both for you.
Why the nested-loop approach is a trap
def dedupe_slow(items): out = [] for x in items: if x not in out: # 'in' on a list scans the whole list out.append(x) return outThis is correct, and that is what makes it dangerous: it passes every test with ten items. Membership on a list is O(n), so the whole function is O(n²). At 100,000 items it does billions of comparisons. Swapping the list for a set makes each check O(1) on average. If that distinction is new to you, read Big O notation explained.
What about unhashable items?
Sets and dict keys need hashable values. Lists and dicts are not hashable in Python, so set([[1], [1]]) raises a TypeError mentioning "unhashable type: 'list'". Convert each item to a hashable form first — a tuple for lists, or json.dumps(d, sort_keys=True) for dictionaries — and use that as the key while keeping the original item in the output.
Deduplicating in SQL and pandas
If the data lives in a database, deduplicate there: SELECT DISTINCT email FROM users, or ROW_NUMBER() OVER (PARTITION BY email ORDER BY created_at) to keep the earliest row per email. In pandas, df.drop_duplicates(subset=["email"], keep="first") does the same in one call.
Frequently asked questions
Does set() keep order in Python?
No. Sets are unordered. Use dict.fromkeys(items) to deduplicate while keeping the original order.
How do I find which items are duplicated rather than remove them?
Count them: [x for x, n in collections.Counter(items).items() if n > 1] in Python. In JavaScript, track a seen set and add to a dupes set whenever an item is already in seen.
Is [...new Set(arr)] fast?
Yes — linear time. It is the idiomatic choice for arrays of primitives. For objects, deduplicate by a key as shown above.
How do I remove duplicates case-insensitively?
Use a normalised key but keep the original value: dedupe by s.casefold() in Python or s.toLowerCase() in JavaScript.
Where this skill pays off
"Remove duplicates while preserving order" is a classic instruction that AI models miss — they return list(set(items)), the output happens to look sorted, and nobody notices. Our free nested loops vs Set exercise covers the performance side, and the response evaluation lab includes the order-preservation failure. Finding issues like these is the review work platforms pay developers for. Learn more in how to evaluate AI-generated code.