How to Read a File Line by Line in Python, Node.js, Java, Go, C# and Bash
Memory-efficient ways to read a file one line at a time in six languages, plus the bugs that come with it — trailing newlines, Windows line endings, encodings, unclosed files and Go's token too long error.
By Software Side Hustles Editorial Team · Published · Updated · 4 min read
Quick answer: iterate over the file object and let the language stream it. Python: for line in open(path, encoding="utf-8") inside a with block. Node.js: readline with for await. Java: Files.lines(path) in try-with-resources. Go: bufio.Scanner. C#: File.ReadLines(path). Bash: while IFS= read -r line. These read one line at a time, so a 20 GB log file uses almost no memory.
How do I read a file line by line in Python?
with open("app.log", encoding="utf-8") as f: for line in f: line = line.rstrip("\n") if "ERROR" in line: print(line)Key points:
- Each line keeps its newline. Strip it with
rstrip("\n")(orrstrip()to remove trailing whitespace too). - Always pass
encoding. Without it Python uses the platform default; on older Windows setups that is not UTF-8, and a file that works on your Mac raisesUnicodeDecodeErroron a colleague's machine. - Use
with. It closes the file even if an exception is raised. f.readlines()andf.read().splitlines()load the whole file into memory. That is fine for small config files and a problem for large logs.
Want line numbers? for n, line in enumerate(f, start=1):.
How do I read a file line by line in Node.js?
import { createReadStream } from "node:fs";import { createInterface } from "node:readline";
const rl = createInterface({ input: createReadStream("app.log"), crlfDelay: Infinity });for await (const line of rl) { if (line.includes("ERROR")) console.log(line);}crlfDelay: Infinity makes Windows \r\n endings count as a single line break. Lines are delivered without their newline characters.
For small files, reading everything at once is simpler:
import { readFileSync } from "node:fs";const lines = readFileSync("config.txt", "utf8").split(/\r?\n/);Note the \r? — splitting on "\n" alone leaves \r at the end of every line of a Windows file, which then breaks string comparisons in ways that are invisible when printed.
How do I read a file line by line in Java?
try (Stream<String> lines = Files.lines(Path.of("app.log"))) { lines.filter(l -> l.contains("ERROR")).forEach(System.out::println);}
try (BufferedReader reader = Files.newBufferedReader(Path.of("app.log"))) { String line; while ((line = reader.readLine()) != null) { // process line }}Files.lines returns a lazy stream backed by an open file, so it must be closed — hence the try-with-resources. Forgetting it leaks file handles, and long-running servers eventually fail with "Too many open files". Files.readAllLines loads everything into a list. Since Java 18 the default charset is UTF-8; on older versions pass StandardCharsets.UTF_8 explicitly.
How do I read a file line by line in Go?
f, err := os.Open("app.log")if err != nil { return err}defer f.Close()
scanner := bufio.NewScanner(f)scanner.Buffer(make([]byte, 0, 64*1024), 10*1024*1024) // allow lines up to 10 MBfor scanner.Scan() { line := scanner.Text() _ = line}if err := scanner.Err(); err != nil { return err}Two Go-specific traps:
- By default a
Scannerrejects lines longer than 64 KB.Scan()simply returnsfalseand the error —bufio.Scanner: token too long— only appears if you checkscanner.Err(). Minified JSON logs hit this regularly. Raise the limit withscanner.Bufferas above, or usebufio.Reader.ReadString('\n'). - Always check
scanner.Err()after the loop. Otherwise a read error looks exactly like the end of the file.
C# and Bash
foreach (var line in File.ReadLines("app.log")) { /* streamed */ }var all = File.ReadAllLines("app.log"); // loads everythingReadLines is lazy; ReadAllLines is not. The names are close enough that people swap them by accident.
while IFS= read -r line || [ -n "$line" ]; do printf '%s\n' "$line"done < app.logIFS=keeps leading and trailing whitespace.-rstops backslashes being treated as escapes.|| [ -n "$line" ]processes a final line that has no trailing newline, whichreadwould otherwise skip.- Do not write
for line in $(cat file)— it splits on spaces, not lines, and expands wildcards.
Common problems when reading files
| Symptom | Cause | Fix |
|---|---|---|
| String comparisons fail though the text looks identical | Trailing \n or \r | Strip line endings; split on \r?\n |
UnicodeDecodeError / garbled characters | Wrong encoding | Specify UTF-8 (or the file's real encoding) |
| First line has an invisible character | UTF-8 byte order mark | Python: encoding="utf-8-sig"; strip in JavaScript |
FileNotFoundError though the file exists | Relative path resolved from a different working directory | Build paths from the script location: Path(__file__).parent / "data.txt" |
| Process slows and memory climbs | Whole file loaded into memory | Stream line by line |
| "Too many open files" | Files never closed | with, try-with-resources, defer f.Close() |
Frequently asked questions
How do I read only the last N lines of a file?
Python: collections.deque(f, maxlen=n) streams the file and keeps the last n lines. On the command line, tail -n 100 app.log.
How do I read a file in chunks instead of lines?
For binary data use fixed-size reads: while chunk := f.read(65536): in Python, a ReadableStream in Node.js, or InputStream.read(byte[]) in Java.
How do I count lines in a large file quickly?
sum(1 for _ in f) in Python, or wc -l file on Unix (which counts newline characters, so a final line without a newline is not counted).
Why does my loop skip every other line?
You are probably calling readline() inside a for line in f loop, which consumes an extra line per iteration. Use one or the other.
Where this skill pays off
"Read a large log and report the errors" is a typical prompt in AI coding evaluations — and the typical weak answer calls read() on the whole file, forgets the encoding or ignores scanner.Err(). Spotting that is a paid skill; see how to evaluate AI-generated code or practise in the free code review lab.