tools / token

Token

engineering2 tools

Count and chunk text by LLM token budget (OpenAI tiktoken encodings).

Tools

chunk.py

● networkpython3

Split text into token-budgeted chunks using OpenAI tiktoken encodings, preferring paragraph boundaries and hard-splitting only paragraphs that exceed the budget on their own. Each chunk is introduced by a "[chunk i/n tokens=k]" marker line, and overlap carries that many tokens of tail context into the next chunk so meaning is not severed mid-thought. Companion to counter, which measures what this splits.

5 paramsdetails →

count.py

● networkpython3

Calculate BPE token counts and token statistics for text or files using OpenAI tiktoken tokenizers (cl100k_base or o200k_base).

2 paramsdetails →

Get it

This directory is the skill — SKILL.md lists the tools and the tools sit beside it. As one file, into any Agent Skills directory:

curl -fsSL https://tools.ofthemachine.com/token.tar.gz | tar xz -C ~/.claude/skills