Skip to content
shellhacks.dev

Sort, Dedupe and Clean Lines

Sort lines alphabetically or naturally, remove duplicates, keep only duplicated lines, trim, drop empty lines or number them.

Free · No sign-up · Your inputs stay in your browser

Input

Sort, dedupe and clean up lines

Output

Result

7 lines in, 7 out

Sorting uses Intl.Collator, so accented letters sort next to their base letter, unlike a plain byte sort such as LC_ALL=C sort.

Runs in this browser. Nothing you paste is sent to a server.

The shell equivalents

Every operation here has a one-line shell form: sort, sort -r, sort -V for natural order, sort -u or awk '!seen[$0]++' to remove duplicates while keeping order, sort | uniq -d for duplicated lines, and nl or cat -n to number them.

Locale changes the order

GNU sort follows the current locale, so the same file sorts differently under LC_ALL=C, which compares bytes, and under en_US.UTF-8, which compares letters the way a dictionary does. This tool sorts the dictionary way with Intl.Collator. When two scripts disagree about the order of a sorted file, a different locale is the usual cause.

Share

Frequently asked questions

What is a natural sort?
A sort that compares runs of digits as numbers, so web-2 comes before web-10. A plain alphabetical sort puts web-10 first because 1 sorts before 2.
Does removing duplicates keep the original order?
Yes. The first occurrence of each line stays where it was and later copies are removed, like awk '!seen[$0]++'.
Is what I paste sent anywhere?
No. The tool runs entirely in your browser with JavaScript. Nothing you type or paste is sent to our server or stored.

Go deeper

Suggested reading

Keep planning

Similar tools