---
name: web
description: 'Search the web and turn a page into Markdown, PDF, a screenshot, or its metadata; find an existing archived snapshot. Tools: archive-snapshot, page-metadata, page-to-markdown, page-to-pdf, screenshot, search, wiki-summary. Use when a request needs any of these; each tool''s own trigger is in SKILL.md.'
compatibility: Requires fragletc and Docker. Network access required.
metadata:
  tags: internet, documents
  tools: archive-snapshot.py, page-metadata.py, page-to-markdown.py, page-to-pdf.py, screenshot.py, search.py, wiki-summary.py
---

# Web

Search the web and turn a page into Markdown, PDF, a screenshot, or its metadata; find an existing archived snapshot.

Every tool below is one executable file beside this `SKILL.md`, in the directory the commands below call `<skill-dir>`. Run it directly. Two host requirements: [`fragletc`](https://github.com/ofthemachine/fraglet) (`curl -fsSL https://raw.githubusercontent.com/ofthemachine/fraglet/main/install.sh | sh && fragletc version`) and Docker; the image, parameters, outputs and network reach are declared in the file itself. Before running a tool, print its contract:

```sh
<skill-dir>/archive-snapshot.py --fraglet-help
```

Parameters are passed as `-p name=value` (a `file` parameter takes a host path); declared outputs are copied out with `--output <name>[=<host path>]`. A missing required parameter fails host-side with exit 2 before any container starts; otherwise the tool's own exit code is the run's. A declared output not named with `--output` is produced and discarded (a stderr note says what you could have copied). Add `--receipt run.json` (or set `FRAGLETC_RECEIPT_DIR`) to record the run: the receipt carries the tool's `procedure_hash`, image digest, parameters, and every input and output by content hash, and `receipt/attest.py` can verify it later without re-running anything.

## Tools

### `archive-snapshot.py`

Look up whether archive.ph already has a snapshot of a URL -- GET /newest/<url> and report its Location header on a redirect. Lookup only: never submits a new snapshot, so it can't be used to force-archive something that isn't already there. Prints nothing (not an error) when no snapshot exists -- the caller falls back to the original link.

Use when a link may be paywalled or gone and the user wants an existing archive.ph snapshot of it, without creating one.

```sh
<skill-dir>/archive-snapshot.py -p url="<url>"
```

Parameters:
- `url` (string, required) — URL to look up on archive.ph

Outputs:
- stdout

Runs in `ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522`; network required — output depends on live external state; do not cache indefinitely.

### `page-metadata.py`

Fetch a URL and extract its <title> and Open Graph tags (og:title, og:description, og:image) -- stdlib html.parser only, no external service.

Use when the user wants a page's title, description, or preview image, or a link needs a caption without fetching the whole article.

```sh
<skill-dir>/page-metadata.py -p url="<url>"
```

Parameters:
- `url` (string, required) — Page URL to fetch

Outputs:
- stdout

Runs in `ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522`; network required — output depends on live external state; do not cache indefinitely.

### `page-to-markdown.py`

Extract clean, readable Markdown article text from any web page using Trafilatura, stripping ads, navigations, sidebars, and boilerplate.

Use when the user wants the article text of a web page, or a page must be read without ads, navigation, and boilerplate.

```sh
<skill-dir>/page-to-markdown.py -p url="<url>" --output article.md
```

Parameters:
- `url` (string, required) — Page URL to extract

Outputs:
- `article.md` — copy out with `--output article.md[=<host path>]`

Runs in `ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522`; network required — output depends on live external state; do not cache indefinitely.

### `page-to-pdf.py`

Render a webpage to a PDF (headless Chromium via Playwright) -- the print-to-PDF cousin of web/screenshot.py. Pass headers=<json object> to send extra request headers, as in web/screenshot.py.

Use when the user wants a PDF of a web page as the browser would print it, including layout and images.

```sh
<skill-dir>/page-to-pdf.py -p url="<url>" [-p headers="<headers>"] --output page.pdf
```

Parameters:
- `url` (string, required) — Page URL to render
- `headers` (string, optional) — JSON object of extra non-secret HTTP headers (e.g. Accept-Language); recorded in plaintext in receipts, so never credentials

Outputs:
- `page.pdf` — copy out with `--output page.pdf[=<host path>]`

Runs in `ofthemachine/headless-browser@sha256:f79c496c6737113c0f6e2d648474a67fb1a416fa34c22220d5714d0b7c6a6036`; network required — output depends on live external state; do not cache indefinitely.

### `screenshot.py`

Capture a full-page PNG screenshot of a webpage (headless Chromium via Playwright). Pass headers=<json object> to send extra request headers. settle_ms (default 2000) is an extra wait after page load, for pages whose real content only appears after their own async JS runs.

Use when the user wants a picture of a rendered page, or a visual check of a URL rather than its text.

```sh
<skill-dir>/screenshot.py -p url="<url>" [-p headers="<headers>"] [-p settle_ms=2000] --output page.png
```

Parameters:
- `url` (string, required) — Page URL to capture
- `headers` (string, optional) — JSON object of extra non-secret HTTP headers (e.g. Accept-Language); recorded in plaintext in receipts, so never credentials
- `settle_ms` (string, optional, default `2000`) — Extra wait after page load for async JS content

Outputs:
- `page.png` — copy out with `--output page.png[=<host path>]`

Runs in `ofthemachine/headless-browser@sha256:f79c496c6737113c0f6e2d648474a67fb1a416fa34c22220d5714d0b7c6a6036`; network required — output depends on live external state; do not cache indefinitely.

### `search.py`

Search the web through DuckDuckGo and print one "title<TAB>url<TAB>snippet" result per line, with no API key required. Set site to constrain results to a single domain, timelimit to d, w, m, or y for recency, and region for locale-weighted results. Pairs with page-to-markdown to turn a query into full article text.

Use when the user wants web search results, recent news about a topic or place, or pages from a specific site, and no API key is available.

```sh
<skill-dir>/search.py -p query="<query>" [-p max_results=10] [-p site="<site>"] [-p region=wt-wt] [-p timelimit="<timelimit>"]
```

Parameters:
- `query` (string, required) — Search query
- `max_results` (string, optional, default `10`) — Maximum results to print
- `site` (string, optional) — Restrict results to this domain
- `region` (string, optional, default `wt-wt`) — DuckDuckGo region code (wt-wt = worldwide)
- `timelimit` (string, optional) — Recency filter: d, w, m, or y

Outputs:
- stdout

Runs in `ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522`; network required — output depends on live external state; do not cache indefinitely.

### `wiki-summary.py`

The lead summaries of the Wikipedia articles that best match one or more topics (Wikipedia's own search, then each article's page summary), written to summary.md as each article's title, link and a few paragraphs. Topics are separated by semicolons (cat behavior; object permanence) and share the byte budget. An article whose title shares no word with its topic is not taken (a search for a sentence finds noise), so a topic may find nothing; a query nothing matches still succeeds, and summary.md says no article was found, so a workflow that cites it goes on with no evidence rather than stopping. A rate limit or a server error exits 75 (temporary: fragletc --retry). Unlike web/page-to-markdown it needs no URL, only words.

Use when a workflow step needs encyclopedic background on subjects named in a few plain words -- a topic, a dish, a person -- as a file a later step reads. Name subjects, not sentences: "cat behavior", not "cats who knock things off tables".

```sh
<skill-dir>/wiki-summary.py -p query="<query>" [-p language=en] [-p max_bytes=3000] --output summary.md
```

Parameters:
- `query` (string, required) — What to look up: a topic in a few plain words, or several separated by semicolons, e.g. cat behavior; object permanence
- `language` (string, optional, default `en`) — Wikipedia language code, e.g. en, fr, de
- `max_bytes` (string, optional, default `3000`) — At most this many bytes of summary text in all, shared by the topics

Outputs:
- `summary.md` — copy out with `--output summary.md[=<host path>]`

Runs in `ofthemachine/python3@sha256:59784cff1945b7d6093a591b5e5292b8c989f0691af0494fbbe1c6b8083a16cb`; network required — output depends on live external state; do not cache indefinitely.
