Why does Korean text break only in big files when you join stream chunks?

If you read a text file as a stream in Node.js and join the pieces with text += chunk, a small test file can come out fine while a large file has a few Korean characters replaced by U+FFFD (the diamond with a question mark, �). On 2026-11-09 the same Korean text was read on Node.js v24.21.0 under macOS 27.0.1, as files in five ways and as a local HTTP response in three. After this note you will be able to find the broken spot byte by byte and pick one fix that suits your code.

The same code broke only on the large file

Each line of the sample text is 66 bytes: a four-digit line number, a space, 20 Korean syllables and a newline. A Korean syllable takes 3 bytes in UTF-8, so the 20 syllables take 60 bytes. 100 such lines went into small.txt (6,600 bytes) and 3,000 into large.txt (198,000 bytes), and both were read like this.

import fs from 'node:fs';

let text = '';
for await (const chunk of fs.createReadStream('large.txt')) {
  text += chunk; // chunk is a Buffer
}

Compared with the original string, the results split. small.txt had zero U+FFFD and matched the original. large.txt had five U+FFFD and did not match. The first U+FFFD was at the last character of line 992, counting from 0. A line that should have ended with U+CD9B ended with two U+FFFD instead.

Chunks are cut by byte count, not by character

Without an encoding, fs.createReadStream hands over the data as Buffer chunks. The Node.js v24 documentation gives the default chunk size (highWaterMark) as 64 * 1024, or 65,536 bytes, and the run reported a readableHighWaterMark of 65536. small.txt is smaller than that, so it came as one chunk. large.txt came as three 65,536-byte chunks and one 1,392-byte chunk.

The trouble is text += chunk. Adding a Buffer to a string decodes that one Buffer as UTF-8 and appends it. When a chunk is cut in the middle of a character's three bytes, neither the bytes left at the end of one chunk nor the first bytes of the next are valid UTF-8 on their own, so each becomes U+FFFD.

Printing the bytes at the three boundaries showed that two of them fell inside a character. At byte 65,536, U+CD9B (ec b6 9b) was split as ec b6 | 9b and became two U+FFFD. At byte 196,608, U+B9AE (eb a6 ae) was split as eb | a6 ae, and a6 and ae in the next chunk were each replaced on their own, making three. That is five in total. The boundary at byte 131,072 happened to fall between two characters, and nothing broke there.

DEV NOTES

When a chunk boundary lands inside a character

The three boundaries when the 198,000-byte large.txt is read in default 64 KiB chunks

• Byte 65,536: split 2 | 1

  ec b6 | 9b

  • U+CD9B is ec b6 9b in UTF-8
  • End of chunk 1, ec b6 → one U+FFFD
  • Start of chunk 2, 9b → one U+FFFD
  • The last character of line 992 became two U+FFFD

• Byte 131,072: between characters by chance

  ec 8e 92 | ec 8e b7

  • Chunk 2 ends on 92, the last byte of a character
  • Chunk 3 starts with ec, the first byte of U+C3B7
  • Nothing broke at this boundary

• Byte 196,608: split 1 | 2

  eb | a6 ae

  • U+B9AE is eb a6 ae in UTF-8
  • End of chunk 3, eb → one U+FFFD
  • Start of chunk 4, a6 and ae → two U+FFFD
  • Five U+FFFD from the two boundaries

Boundary bytes printed by lab/read.mjs on Node.js v24.21.0 on 2026-11-09, and the U+FFFD count from decoding each split piece as its own Buffer. One line is 66 bytes (four digits, a space, 20 Korean syllables, a newline).

That is why this bug shows up only sometimes. A file under 65,536 bytes has no boundary at all, and a boundary that falls between characters does no harm. Letters, digits, spaces and newlines, like the line numbers in this sample, take 1 byte each, so a cut never leaves part of one behind. With the default 64 KiB chunks it shows up only in large files with multi-byte text such as Korean, and only in a few places.

Over HTTP, chunk sizes and the count could change between runs

The same thing happens with fetch. On the same Mac, a small server on 127.0.0.1:47110, used only for this test, sent large.txt; the client read the body with res.body.getReader() and called decoder.decode(value) on every chunk. The server was closed when the run finished.

Network chunks were not an even 65,536 bytes like the file's. The first run had six chunks starting 65,357, 179, 32,657 and 65,527 bytes, and ten U+FFFD. Five more runs gave 5 to 7 chunks and 6 to 10 U+FFFD, depending on the run (the same result came up twice). With the same file, the U+FFFD count can change from run to run, which makes the bug harder to reproduce from a log. A remote server or a proxy was not tried.

Decode where the end of the last chunk is remembered

Every fix works the same way: bytes that stop in the middle of a character are kept until the next chunk arrives and are decoded together with it. The three methods below each gave zero U+FFFD where they were run (file or HTTP) and matched the original.

• For a Node.js stream, give the encoding to the stream. Pass it as an option, as in fs.createReadStream(file, {encoding: 'utf8'}), or call readable.setEncoding('utf8'). The Node.js documentation says the stream then handles multi-byte characters that would be decoded wrongly if simply pulled out as Buffer objects. The chunks then arrive as strings, not Buffers.
• For fetch or a web stream, give TextDecoder {stream: true}. Create one decoder, call decoder.decode(value, {stream: true}) on each chunk, and call decoder.decode() with no arguments once more at the end. In the WHATWG Encoding Standard, stream defaults to false. A call with stream false finishes whatever is pending, and the call after it starts afresh; only with true are unfinished bytes carried over to the next call.
• If you need everything at once, decode it in one go. Collect the Buffer chunks in an array and convert them with Buffer.concat(chunks).toString('utf8'), or use await res.text() with fetch. This loads the whole content into memory, so for very large files the first two methods fit better.

const reader = res.body.getReader();
const decoder = new TextDecoder(); // one decoder for the whole body
let text = '';
for (;;) {
  const {done, value} = await reader.read();
  if (done) break;
  text += decoder.decode(value, {stream: true});
}
text += decoder.decode(); // flush what is left

DEV NOTES

Let the code that turns bytes into text remember the end of the last chunk

The file read in five ways, the HTTP response in three

• Avoid: decoding each chunk on its own

  text += chunk

  • text += chunk (each Buffer becomes a string alone)
  • decoder.decode(value) on every chunk
  • Five U+FFFD on large.txt, 6 to 10 over HTTP
  • Zero on the 6,600-byte small.txt, a single chunk

• Node.js streams: give the encoding to the stream

  {encoding: 'utf8'}

  • fs.createReadStream(file, {encoding: 'utf8'})
  • or readable.setEncoding('utf8')
  • Incomplete bytes wait for the next chunk
  • Result: zero U+FFFD, same as the original

• fetch and web streams: stream: true, then decode()

  {stream: true}

  • decoder.decode(value, {stream: true})
  • Call decoder.decode() once more at the end
  • For the whole body: res.text() or Buffer.concat
  • Zero U+FFFD with all three

Results from lab/read.mjs and lab/http.mjs (a local server on 127.0.0.1:47110) on Node.js v24.21.0 on 2026-11-09. The HTTP figures are the range over five extra runs.

The final decoder.decode() releases any unfinished bytes left at the end. Feeding the three bytes 41 ec b6 with {stream: true} returned only A, and the decode() call after it returned one U+FFFD. Leave it out and there is no trace that the data stopped in the middle of a character. Conversely, if the text is read correctly and still ends with U+FFFD, the data itself was cut, not the decoding.

Where to look in your code, and a test

Look for three things: a Buffer added straight to a string, as in data += chunk; chunk.toString() called on each chunk; and a new TextDecoder() created per chunk, or decode(value) called without {stream: true}.

A test does not need a large file. Reading a 99-byte file with 30 Korean syllables at highWaterMark: 1 gave 90 U+FFFD with text += chunk, meaning every one of the 90 Korean bytes became U+FFFD on its own, and zero with encoding: 'utf8'. With 2 and 4 it still gave 75 and 40. One test that reads with highWaterMark: 1 and counts '�' in the result will not pass just because a boundary happened to fall between characters.

Do not turn byte chunks into text chunk by chunk; leave it to one decoder that remembers the end of the previous chunk.