---
type: Attested Computation
title: chunk
identifier: token/chunk
description: Split text into token-budgeted chunks using OpenAI tiktoken encodings, preferring paragraph boundaries and hard-splitting only paragraphs that exceed the budget on their own. Each chunk is introduced by a "[chunk i/n tokens=k]" marker line, and overlap carries that many tokens of tail context into the next chunk so meaning is not severed mid-thought. Companion to counter, which measures what this splits.
pack: token
tags:
- engineering
runtime: fragletc
image: ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522
network: required
stdin: buffer
execution_class: environmental
procedure_hash: sha256:6250f718ac5beb7d6ded8117d495e3f8ac7f66cd57a2733312a981704d5c3326
parameters:
- name: text
  type: string
  required: false
  description: Inline text to chunk (or pipe stdin)
- name: text_source
  type: file
  required: false
  description: Host file mounted at /input/text_source
- name: max_tokens
  type: string
  required: false
  default: '1000'
  description: Token budget per chunk
- name: overlap
  type: string
  required: false
  default: '100'
  description: Tail tokens carried into the next chunk
- name: encoding
  type: string
  required: false
  default: cl100k_base
  description: tiktoken encoding or model name
computation: https://tools.ofthemachine.com/token/chunk.py
executor:
  resource: https://github.com/ofthemachine/fraglet
  receipt:
  - schema
  - fragletc
  - procedure
  - procedure_hash
  - image
  - image_digest
  - network
  - stdin_mode
  - params
  - inputs
  - argv
  - env
  - memo_key
  - started
  - finished
  - exit_code
  - stdout
  - outputs
  format: fraglet-receipt/2 (fragletc --receipt <path>, or FRAGLETC_RECEIPT_DIR)
attester:
  resource: https://tools.ofthemachine.com/receipt/attest.py
  method: 'deterministic, no re-execution: procedure_hash equals sha256 of computation; memo_key recomputes from the receipt''s own fields; claimed params and any artifact hash match the receipt'
generated:
  by: process:tools/catalog/compile.py
  at: '2026-11-13T18:27:40Z'
---

# chunk

Split text into token-budgeted chunks using OpenAI tiktoken encodings, preferring paragraph boundaries and hard-splitting only paragraphs that exceed the budget on their own. Each chunk is introduced by a "[chunk i/n tokens=k]" marker line, and overlap carries that many tokens of tail context into the next chunk so meaning is not severed mid-thought. Companion to counter, which measures what this splits.

Use when text is too large for one prompt and must be split into token-budgeted pieces that respect paragraph boundaries.

# Schema

## Parameters
- `text` (string, optional) — Inline text to chunk (or pipe stdin)
- `text_source` (file, optional) — Host file mounted at /input/text_source
- `max_tokens` (string, optional, default `1000`) — Token budget per chunk
- `overlap` (string, optional, default `100`) — Tail tokens carried into the next chunk
- `encoding` (string, optional, default `cl100k_base`) — tiktoken encoding or model name
- stdin (optional) — piped input is read in full and hashed into the run's receipt

## Outputs
- stdout (the anonymous result)

# Execution

The computation is [`token/chunk.py`](https://tools.ofthemachine.com/token/chunk.py) (frontmatter `computation`; in `tools.tar.gz` the same file sits at `token/chunk.py` beside this bundle), run by `fragletc` in `ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522` with network `required`. Its identity is `procedure_hash` = sha256 of that file, shebang included — the same value every receipt records. Parameters are passed as `-p name=value`; `--receipt run.json` (or `FRAGLETC_RECEIPT_DIR`) writes the receipt, a `fraglet-receipt/2` JSON document with the fields listed under `executor.receipt`.

# Attestation

A run's receipt is attested with [`receipt/attest.py`](https://tools.ofthemachine.com/receipt/attest.py): the computation that ran is this script (`procedure_hash`), the receipt is internally consistent (`memo_key` recomputes from its own fields), the claimed parameters are the ones that ran, and an artifact claimed as an output hashes to what the receipt recorded. No re-execution: attestation is provenance fidelity, not reproduction.

Environmental: the tool reaches the network, so outputs depend on state outside its inputs; the receipt records exactly what ran and what came out and makes no reproduction claim.

Skill: [SKILL.md](https://tools.ofthemachine.com/token/SKILL.md)
