Read large log files with bounded Rust buffers

Reading an entire log into a string is convenient until the log is several gigabytes. A desktop preview should have an explicit byte budget and tell the reader when more data exists. This tutorial builds a bounded prefix reader in Rust; it is a starting point for a viewer, not a full search engine or live tail implementation.

Define the preview budget

log-preview/
├── preview.rs
└── README.md

The helper reads at most 64 KiB plus one extra byte. That extra byte tells us whether the preview was truncated without loading the rest of the file.

use std::io::{self, Read};
 
pub const LIMIT: usize = 64 * 1024;
 
pub fn preview<R: Read>(reader: R) -> io::Result<(Vec<u8>, bool)> {
    let mut bytes = Vec::with_capacity(LIMIT + 1);
    reader.take((LIMIT + 1) as u64).read_to_end(&mut bytes)?;
    let truncated = bytes.len() > LIMIT;
    bytes.truncate(LIMIT);
    Ok((bytes, truncated))
}

A file can be passed directly as the reader. The helper does not follow paths, open files or grant permissions; the caller is responsible for choosing an allowed file and handling open errors. Its memory use is bounded by the preview budget, not total log size.

Avoid the giant-line trap

Buffered reading reduces operating-system calls, but does not automatically bound the memory allocated for one line. A file with no newline can make a naive read_line or lines approach accumulate a huge string.

The prefix helper works on bytes, so a single enormous line is still capped. It may stop halfway through a line or a UTF-8 character. For a preview, String::from_utf8_lossy can display replacement characters while keeping the original bytes separate. Label the view as a truncated preview; do not imply that the displayed text is an exact complete export.

If you need pagination, use byte offsets and define how partial lines are carried between pages. Do not interpret an offset as a line number. A live tail also needs a policy for file truncation, rotation and replacement.

Keep the interface responsive

Open and read the file on a worker, then send a bounded result to the desktop renderer. Render text as text, not HTML. Log content can contain markup, terminal escape sequences or private data; strip or visibly represent controls according to the viewer's purpose.

Limit rendered rows as well as bytes. A virtualised list can help with many small lines, but it does not make an unbounded backend response safe. Provide a visible “first 64 KiB” message and an intentional action for another page or a full export.

Never upload log contents to a server merely to show a local preview. If a product offers sharing, make that a separate action with redaction and a clear destination.

Verify memory-related edges

The downloadable tests exercise empty input, exactly the limit, one byte over it, a very large newline-free input and an I/O error. They check both the output length and the truncation flag.

rustc --edition=2021 --test preview.rs -o preview-tests
./preview-tests

These are helper tests, not a benchmark of a complete desktop application. For integration, test a slow filesystem, permission failures, invalid UTF-8 and closing the window while a read is pending. Add cooperative cancellation where work is split into multiple reads.

See Rust buffered-reading behaviour. For a log inspector or other local developer tool, explore Rust desktop services.