File Handling

Introduction

Go's os and io packages handle file operations with explicit error checking. Open files with os.Open, always defer Close(), and use io.Copy for streaming large files without loading them into memory.

Production services read config files, write audit logs, process CSV uploads, and manage temp files for image processing. Understanding file permissions, path handling with filepath, and embedded files (embed package) covers real backend requirements.

This lesson teaches safe file I/O patterns used in log rotation, data import pipelines, and certificate loading for TLS servers.

The story

A Docker image scanner reads layer tarballs from disk, streams SHA-256 hashes without loading multi-gigabyte files into memory, and writes vulnerability reports to /tmp/scan-{id}.json. os.Open, bufio.Scanner, and io.Copy compose the pipeline — the same building blocks in Terraform's state file reader and Kubernetes' kubelet log tailer.

Always defer file.Close() and check errors from Close — deferred cleanup runs when functions return, even on error paths.

Understanding the topic

Key concepts

• os.Open returns *os.File and error; defer f.Close() immediately.
• os.ReadFile reads entire file; os.WriteFile writes bytes with perm.
• io.Copy streams between Reader and Writer efficiently.
• filepath.Join builds cross-platform paths — never string concat.
• embed.FS embeds files at compile time for static assets.
• os.Stat checks existence; os.IsNotExist(err) for missing files.

Step-by-step explanation

1. Open file with flags: os.O_RDONLY, O_WRONLY, O_CREATE, O_APPEND.
2. Read with bufio.Scanner line-by-line or io.ReadAll for small files.
3. Write with os.WriteFile or bufio.Writer for buffered output.
4. Create temp file: os.CreateTemp for secure temporary storage.
5. MkdirAll creates directory tree with permissions.
6. Rename provides atomic file replace on same filesystem.

Practical code example

Read config file, process lines, and write results atomically:

package main

import (
	"bufio"
	"fmt"
	"os"
	"path/filepath"
	"strings"
)

func processInput(path string) error {
	f, err := os.Open(path)
	if err != nil {
		return fmt.Errorf("open input: %w", err)
	}
	defer f.Close()

	outPath := filepath.Join(filepath.Dir(path), "output.txt")
	tmp, err := os.CreateTemp(filepath.Dir(path), "out-*.txt")
	if err != nil {
		return fmt.Errorf("create temp: %w", err)
	}
	defer os.Remove(tmp.Name())

	w := bufio.NewWriter(tmp)
	scanner := bufio.NewScanner(f)
	for scanner.Scan() {
		line := strings.TrimSpace(scanner.Text())
		if line == "" {
			continue
		}
		fmt.Fprintln(w, strings.ToUpper(line))
	}
	if err := scanner.Err(); err != nil {
		return fmt.Errorf("scan: %w", err)
	}
	if err := w.Flush(); err != nil {
		return err
	}
	return os.Rename(tmp.Name(), outPath)
}

func main() {
	if err := processInput("input.txt"); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	fmt.Println("done")
}

Line-by-line code explanation

• f, err := os.Open("config.yaml") opens a file for reading — returns *os.File and error.
• defer f.Close() ensures the file descriptor is released when the function exits.
• data, err := os.ReadFile("small.txt") reads entire small files in one call.
• os.WriteFile("out.json", data, 0644) writes bytes with Unix permission bits.
• scanner := bufio.NewScanner(f) reads line-by-line without loading the whole file.
• io.Copy(dst, src) streams data between files, network connections, or hashers efficiently.
• filepath.Join(dir, name) builds cross-platform paths — never string-concatenate with "/".
• os.Stat(path) checks existence and metadata before processing.

Key takeaway: Temp file + Rename gives atomic output. defer Close and Remove. filepath.Join for portable paths.

Real-world use

Where you'll use this in production

• Loading TLS certificates and keys for HTTPS servers.
• Processing CSV/JSONL batch uploads in ETL pipelines.
• Writing rotated audit logs to disk.
• Embedding HTML templates with embed.FS in single binary.

Best practices

• Always defer f.Close(); check Close error on critical files.
• Use filepath.Join, not hardcoded slashes.
• Stream large files with io.Copy — don't ReadAll multi-GB files.
• Set restrictive file permissions: 0600 for secrets.
• Atomic writes via temp + rename pattern.
• Validate paths to prevent directory traversal attacks.

Common mistakes

• Forgetting Close — file descriptor leak.
• Not checking scanner.Err() after range loop.
• Reading huge files into memory with ReadFile.
• Using relative paths without documenting working directory.
• Windows vs Unix path separators with string concat.

Advanced interview questions

Summary

os and io provide explicit, error-checked file operations. defer Close; stream large files; use filepath.Join. Atomic writes use temp file + rename pattern. embed.FS bundles static assets into the binary. Next lesson: JSON encoding and decoding.