Text

name: Text description: Transform, format, and process text with patterns for writing, data cleaning, localization, citations, and copywriting.

Quick Reference

Universal Text Rules

Encoding

• Always verify encoding first: file -bi document.txt
• Normalize line endings: tr -d '\r'
• Remove BOM if present: sed -i '1s/^\xEF\xBB\xBF//'

Whitespace

• Collapse multiple spaces: sed 's/[[:space:]]\+/ /g'
• Trim leading/trailing: sed 's/^[[:space:]]*//;s/[[:space:]]*$//'

Common Traps

• Smart quotes (" ") break parsers → normalize to "
• Em/en dashes (– —) break ASCII → normalize to -
• Zero-width chars invisible but break comparisons → strip them
• String length ≠ byte length in UTF-8 ("café" = 4 chars, 5 bytes)

Format Detection

# Detect encoding
file -I document.txt

# Detect line endings
cat -A document.txt | head -1
# ^M at end = Windows (CRLF)
# No ^M = Unix (LF)

# Detect delimiter (CSV/TSV)
head -1 file | tr -cd ',;\t|' | wc -c

Quick Transformations

Before Processing Checklist

• Encoding verified (UTF-8?)
• Line endings normalized
• Delimiter identified (for structured text)
• Target format/style defined
• Edge cases considered (empty, Unicode, special chars)