---
type: Category
title: Token
description: Count and chunk text by LLM token budget (OpenAI tiktoken encodings).
tags:
- engineering
generated:
  by: process:tools/catalog/compile.py
  at: '2026-11-13T18:27:40Z'
---

# Token

Count and chunk text by LLM token budget (OpenAI tiktoken encodings).

- [chunk](chunk.md) — Split text into token-budgeted chunks using OpenAI tiktoken encodings, preferring paragraph boundaries and hard-splitting only paragraphs that exceed the budget on their own. Each chunk is introduced by a "[chunk i/n tokens=k]" marker line, and overlap carries that many tokens of tail context into the next chunk so meaning is not severed mid-thought. Companion to counter, which measures what this splits.
- [count](count.md) — Calculate BPE token counts and token statistics for text or files using OpenAI tiktoken tokenizers (cl100k_base or o200k_base).
