SoftwareSideHustles
How-To

How to Remove Duplicates From a List or Array (and Keep the Original Order)

Order-preserving and fast ways to deduplicate lists in Python, arrays in JavaScript and collections in Java, including duplicates of objects by a key — and the quadratic approach to avoid.

By Software Side Hustles Editorial Team · Published · Updated · 4 min read

Quick answer: in Python use list(dict.fromkeys(items)), in JavaScript [...new Set(items)], and in Java new ArrayList<>(new LinkedHashSet<>(items)). All three remove duplicates in linear time and keep the first occurrence of each item in its original position. Converting to a plain set() in Python is also fast, but it throws the order away.

Deduplicate by language

LanguageKeeps orderCode
PythonYeslist(dict.fromkeys(items))
PythonNolist(set(items))
JavaScriptYes[...new Set(items)]
JavaYesnew ArrayList<>(new LinkedHashSet<>(items))
Java (streams)Yesitems.stream().distinct().toList()
C#Yes (in practice)items.Distinct().ToList()
SQLn/aSELECT DISTINCT col FROM t

How do I remove duplicates from a list in Python?

python
items = [3, 1, 3, 2, 1]
list(dict.fromkeys(items)) # [3, 1, 2] — order preserved
list(set(items)) # some order — do not rely on it
sorted(set(items)) # [1, 2, 3] — when you want sorted output anyway

dict.fromkeys works because dictionaries preserve insertion order (guaranteed since Python 3.7) and ignore repeated keys. A set has no defined order: small integers often look sorted, which is how this bug survives testing and then appears with strings in production.

If you need more control — for example, to skip items as you go — write the loop explicitly with a seen set:

python
def dedupe(items):
seen = set()
out = []
for x in items:
if x not in seen:
seen.add(x)
out.append(x)
return out

How do I remove duplicates from an array in JavaScript?

javascript
const items = [3, 1, 3, 2, 1];
const unique = [...new Set(items)]; // [3, 1, 2]

A Set remembers insertion order, so this is order-preserving. It compares values with SameValueZero: primitives work as expected, NaN equals NaN, but objects are compared by reference. Two separate { id: 1 } objects are both kept.

The filter/indexOf version you will see in older answers is correct but O(n²):

javascript
items.filter((x, i) => items.indexOf(x) === i); // fine for 50 items, slow for 50,000

How do I remove duplicate objects by a key?

Most real-world deduplication is "same user ID" or "same email", not identical objects.

JavaScript — keep the first occurrence per key:

javascript
function uniqueBy(items, key) {
const seen = new Set();
return items.filter((item) => {
const k = key(item);
if (seen.has(k)) return false;
seen.add(k);
return true;
});
}
uniqueBy(users, (u) => u.email.toLowerCase());

To keep the last occurrence instead, build a Map: [...new Map(items.map((u) => [u.id, u])).values()]. Later entries overwrite earlier ones, but each key keeps the position where it first appeared.

Python — the same idea:

python
def unique_by(items, key):
seen = set()
return [x for x in items if not (key(x) in seen or seen.add(key(x)))]

That one-liner is clever but hard to read; the explicit loop from the previous section is usually the better choice in shared code.

How do I remove duplicates from a List in Java?

java
List<Integer> items = List.of(3, 1, 3, 2, 1);
List<Integer> unique = new ArrayList<>(new LinkedHashSet<>(items)); // [3, 1, 2]
List<Integer> viaStream = items.stream().distinct().toList(); // [3, 1, 2]

HashSet would lose the order; LinkedHashSet keeps it; TreeSet sorts. For your own classes, deduplication depends on equals() and hashCode(). Override one without the other and duplicates slip through, because hash-based collections check the hash first. Java records generate both for you.

Why the nested-loop approach is a trap

python
def dedupe_slow(items):
out = []
for x in items:
if x not in out: # 'in' on a list scans the whole list
out.append(x)
return out

This is correct, and that is what makes it dangerous: it passes every test with ten items. Membership on a list is O(n), so the whole function is O(n²). At 100,000 items it does billions of comparisons. Swapping the list for a set makes each check O(1) on average. If that distinction is new to you, read Big O notation explained.

What about unhashable items?

Sets and dict keys need hashable values. Lists and dicts are not hashable in Python, so set([[1], [1]]) raises a TypeError mentioning "unhashable type: 'list'". Convert each item to a hashable form first — a tuple for lists, or json.dumps(d, sort_keys=True) for dictionaries — and use that as the key while keeping the original item in the output.

Deduplicating in SQL and pandas

If the data lives in a database, deduplicate there: SELECT DISTINCT email FROM users, or ROW_NUMBER() OVER (PARTITION BY email ORDER BY created_at) to keep the earliest row per email. In pandas, df.drop_duplicates(subset=["email"], keep="first") does the same in one call.

Frequently asked questions

Does set() keep order in Python?

No. Sets are unordered. Use dict.fromkeys(items) to deduplicate while keeping the original order.

How do I find which items are duplicated rather than remove them?

Count them: [x for x, n in collections.Counter(items).items() if n > 1] in Python. In JavaScript, track a seen set and add to a dupes set whenever an item is already in seen.

Is [...new Set(arr)] fast?

Yes — linear time. It is the idiomatic choice for arrays of primitives. For objects, deduplicate by a key as shown above.

How do I remove duplicates case-insensitively?

Use a normalised key but keep the original value: dedupe by s.casefold() in Python or s.toLowerCase() in JavaScript.

Where this skill pays off

"Remove duplicates while preserving order" is a classic instruction that AI models miss — they return list(set(items)), the output happens to look sorted, and nobody notices. Our free nested loops vs Set exercise covers the performance side, and the response evaluation lab includes the order-preservation failure. Finding issues like these is the review work platforms pay developers for. Learn more in how to evaluate AI-generated code.

Cookies on this site

We use essential cookies to run the questionnaire, practice labs and sign-in. With your permission we'd also like to use analytics to understand which pages help people, and advertising cookies to measure our Google Ads. Rejecting never limits any feature. Cookie policy