How to Split a File by Lines Size or Pattern With Examples

A 40 GB log does not open in an editor and does not upload in one piece. Splitting it into smaller files by lines, size or content makes it manageable. split handles the first two, and csplit the last. The examples use GNU coreutils 9.4; the reverse operation is in merging files without losing newlines.

By lines or size

split -l 100000 -d --additional-suffix=.log access.log part-
ls part-*
split -b 100M -d big.bin chunk-
split -n l/4 data.txt quarter-

part-00.log  part-01.log  part-02.log

-l sets lines per piece, -b bytes per piece (suffixes K, M, G), and -n l/4 makes four pieces of roughly equal size without cutting a line, which -n 4 alone may do. The last argument is the prefix, and the default suffix is two letters, aa, ab, which -d changes to digits. A suffix length of two allows 676 pieces with letters or 100 with digits, and -a 4 widens it. -b splits in the middle of lines, so use it for binary files or when the pieces are reassembled with cat.

Process each piece on the fly

split -l 1000000 --filter='gzip -9 > "$FILE.gz"' big.log big-
zcat big.log.gz | split -l 1000000 -d - part-

--filter runs a command for each piece with the output name in $FILE, so compressed pieces are written directly and the uncompressed data never lands on disk. Reading from - splits standard input, which lets you split a stream from a pipe. Both save the disk space that the pieces would otherwise need, which matters when the original is already tight on space, as in no space left on device.

By pattern with csplit

csplit -z -f section- -b '%02d.txt' notes.txt '/^## /' '{*}'
ls section-*

section-00.txt  section-01.txt  section-02.txt

csplit cuts before every line matching /^## /, repeated by {*} for as many matches as exist. -z drops empty pieces, -f sets the prefix and -b a printf format for the suffix. It prints the byte count of each piece, which -s silences. A line number such as csplit file 100 200 cuts at those lines. Use %regex% to skip lines up to a match without writing them.

Keep a CSV header on each piece

head -n 1 data.csv > header.csv
tail -n +2 data.csv | split -l 50000 -d --additional-suffix=.csv - part-
for f in part-*.csv; do cat header.csv "$f" > "with-header-$f"; done

Splitting a CSV by lines puts the header only in the first piece, so strip it and prepend it to every piece. Fields that contain newlines inside quotes would be cut in two by a line split, so do not split such data by lines; use a CSV-aware tool, as in extracting CSV columns.

How split names and sizes its pieces

The command writes pieces called by a prefix and a suffix. The default suffix is two letters, aa to zz, which allows 676 pieces before it fails with "output file suffixes exhausted". For a large input with a small piece size, use -a 4 for longer suffixes, or -d for numeric ones, and add --numeric-suffixes=1 to start from a chosen number. Sizing is exact for byte splits and line-based for the others, so pieces from -b may cut a line in half, which is fine for binary data and wrong for text. With -C, the option splits by bytes but never breaks a line, putting as many complete lines in each piece as fit. For equal numbers of pieces, -n 8 makes eight pieces by bytes and -n l/8 makes eight pieces by lines.

Reassembly is a concatenation. Because the suffixes sort in order, cat part-* > whole restores the file exactly, and a checksum before and after confirms it. Store the checksum with the pieces, as a missing piece in the middle gives a shorter file and no error.

Parallel processing of the pieces

Splitting is often the first step in using several cores on one big file. Divide the input by lines into pieces about as numerous as the processors, process them in parallel with xargs -P or GNU parallel, and combine the results. Make sure the operation is one that can be combined: counting, filtering and per-line transformations are, and global sorting and deduplication need a merge step. For sorting, split, sort each piece, and merge with sort -m which reads sorted inputs and combines them in one pass, much faster than sorting the whole again. GNU parallel --pipe can do the split and the dispatch in one command without temporary files, handling block sizes with --block.

Splitting structured files

Text formats with structure need care. A JSON array cannot be split by lines, but JSON Lines can, one object per line. A CSV with quoted newlines cannot be split by lines safely, and a parser-based tool such as mlr split or xsv split handles it. SQL dumps can be split at statement boundaries, with csplit on a pattern like the start of each table, and a dump that uses extended inserts can have very long lines, so line counts mean little. Compressed archives should be split after compression for transfer, and by content before, if the pieces are to be used independently. Keep the pieces' format valid on their own, with headers or opening tags repeated where the consumer needs them.

Uploading and transferring in parts

Many services limit upload sizes, and splitting is the workaround. Choose a piece size below the limit with room for overhead, name the pieces so they sort, send a manifest listing the names, sizes and checksums, and verify on the receiving side. For object stores, a multipart upload is built in and better than manual splitting. For email attachments and chat tools, splitting a compressed archive with split -b 20M works, with instructions to join them. For resumable transfers over unstable links, rsync --partial on the whole file is simpler than managing pieces.

Cleaning up and avoiding surprises

The pieces of a huge file take as much space as the file, so check free space before splitting and remove the original only after verification. Write them to a directory of their own, because a prefix that matches other files makes the later glob catch strangers. Use --verbose to print each name as it is created, and quote the prefix if it contains special characters. In a script, test the exit status, and compare the sum of the piece sizes with the original size as a final check before deleting anything.

Where this does not apply

To split by a key, such as one file per customer, awk is more direct: awk -F, '{print > ($1 ".csv")}' data.csv, though it can run out of file descriptors with many keys, and GNU awk then manages them. Check line counts after splitting with wc -l part-* | tail -1 against the original. For huge files that must be processed in parallel, split by lines and run xargs -P on the pieces, as in running commands in parallel. See the split manual and the csplit manual.