---
name: token
description: 'Count and chunk text by LLM token budget (OpenAI tiktoken encodings). Tools: chunk, count. Use when a request needs any of these; each tool''s own trigger is in SKILL.md.'
compatibility: Requires fragletc and Docker. Network access required.
metadata:
  tags: engineering
  tools: chunk.py, count.py
---

# Token

Count and chunk text by LLM token budget (OpenAI tiktoken encodings).

Every tool below is one executable file beside this `SKILL.md`, in the directory the commands below call `<skill-dir>`. Run it directly. Two host requirements: [`fragletc`](https://github.com/ofthemachine/fraglet) (`curl -fsSL https://raw.githubusercontent.com/ofthemachine/fraglet/main/install.sh | sh && fragletc version`) and Docker; the image, parameters, outputs and network reach are declared in the file itself. Before running a tool, print its contract:

```sh
<skill-dir>/chunk.py --fraglet-help
```

Parameters are passed as `-p name=value` (a `file` parameter takes a host path); declared outputs are copied out with `--output <name>[=<host path>]`. A missing required parameter fails host-side with exit 2 before any container starts; otherwise the tool's own exit code is the run's. A declared output not named with `--output` is produced and discarded (a stderr note says what you could have copied). Add `--receipt run.json` (or set `FRAGLETC_RECEIPT_DIR`) to record the run: the receipt carries the tool's `procedure_hash`, image digest, parameters, and every input and output by content hash, and `receipt/attest.py` can verify it later without re-running anything.

## Tools

### `chunk.py`

Split text into token-budgeted chunks using OpenAI tiktoken encodings, preferring paragraph boundaries and hard-splitting only paragraphs that exceed the budget on their own. Each chunk is introduced by a "[chunk i/n tokens=k]" marker line, and overlap carries that many tokens of tail context into the next chunk so meaning is not severed mid-thought. Companion to counter, which measures what this splits.

Use when text is too large for one prompt and must be split into token-budgeted pieces that respect paragraph boundaries.

```sh
<skill-dir>/chunk.py [-p text="<text>"] [-p text_source="<text_source>"] [-p max_tokens=1000] [-p overlap=100] [-p encoding=cl100k_base]
```

Parameters:
- `text` (string, optional) — Inline text to chunk (or pipe stdin)
- `text_source` (file, optional) — Host file mounted at /input/text_source
- `max_tokens` (string, optional, default `1000`) — Token budget per chunk
- `overlap` (string, optional, default `100`) — Tail tokens carried into the next chunk
- `encoding` (string, optional, default `cl100k_base`) — tiktoken encoding or model name
- stdin (optional) — piped input is read in full and hashed into the run's receipt

Outputs:
- stdout

Runs in `ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522`; network required — output depends on live external state; do not cache indefinitely.

### `count.py`

Calculate BPE token counts and token statistics for text or files using OpenAI tiktoken tokenizers (cl100k_base or o200k_base).

Use when the user asks how many tokens some text or file is, or a prompt must be checked against a token budget.

```sh
<skill-dir>/count.py [-p text="<text>"] [-p encoding=cl100k_base]
```

Parameters:
- `text` (string, optional) — Inline text to tokenize (or pipe stdin)
- `encoding` (string, optional, default `cl100k_base`) — tiktoken encoding or model name
- stdin (optional) — piped input is read in full and hashed into the run's receipt

Outputs:
- stdout

Runs in `ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522`; network required — output depends on live external state; do not cache indefinitely.
