bfiles 2025-06-01
Added token counting and file chunking for LLM context window awareness
15 files · 951+ · 532-

Token counting support

instantiate behavioral
2 files changed
  • core.py 694 lines of changes adding token counting and reorganizing the bundling pipeline
  • metadata.py Added per-file and content-level token count estimates using tiktoken
  • Added per-file and content-level token count estimates to bundle output using tiktoken instantiate behavioral
    2 files
    • metadata.py
    • core.py
  • Stable implementation tracking token counts for each file and aggregate totals instantiate internal
    1 file
    • core.py
  • Token counts appear in bundle metadata headers for LLM context window planning instantiate internal
    1 file
    • metadata.py

File chunking

instantiate behavioral
1 file changed
  • config.py Added chunk_size and chunk_overlap configuration fields
  • Implemented --chunk-size and --chunk-overlap CLI options for splitting large files instantiate behavioral
    2 files
    • cli.py
    • config.py
  • Core chunking logic splits files by token count with configurable overlap between chunks instantiate internal
    1 file
    • core.py
  • Prepared infrastructure for multi-file chunked output instantiate behavioral
    1 file
    • core.py

CLI and output updates

interface behavioral
6 files changed
  • cli.py Updated CLI arguments, default output location, and added chunking-related options
  • output.py Updated output display logic
  • metadata_writer.py Updated metadata writer for token count and chunk data
  • exclusions.py Minor exclusion handling updates
  • README.md Updated documentation
  • pyproject.toml Added tiktoken dependency

Test expansion

qualify behavioral
6 files changed
  • test_cli.py 163 lines of new tests for chunking CLI options and token count display
  • test_core.py 253 lines of new tests for chunking logic and token counting
  • test_exclusions.py Updated exclusion tests
  • test_metadata.py Updated metadata tests
  • test_output.py Updated output tests
  • output.py Updated output fixtures
  • Added 163 lines of new CLI tests for chunking options and token count display qualify behavioral
    1 file
    • test_cli.py
  • Added 253 lines of new tests for chunking logic and token counting qualify internal
    1 file
    • test_core.py
  • Updated output and exclusion tests qualify behavioral
    2 files
    • test_exclusions.py
    • test_output.py

Branch merges

remediate internal
1 file changed
  • cli.py Merged fix/bfiles-import-errors branch changes
  • Merged fix/bfiles-import-errors branch (two merge commits) remediate internal
    1 file
    • cli.py
← 2025-05-31 bfiles changelog → 2025-06-27 bfiles →