web/page-to-markdown.py
Extract clean, readable Markdown article text from any web page using Trafilatura, stripping ads, navigations, sidebars, and boilerplate.
Use when the user wants the article text of a web page, or a page must be read without ads, navigation, and boilerplate.
- the file
- web/page-to-markdown.py
- its skill
- web/SKILL.md (the whole web pack)
- as knowledge
- okf/web/page-to-markdown.md
- on GitHub
- web/page-to-markdown.py ↗
Run it
1. Once per machine: install fragletc (Docker must already be running).
curl -fsSL https://raw.githubusercontent.com/ofthemachine/fraglet/main/install.sh | sh
2. Fetch the tool and print its contract.
curl -fsSLO https://tools.ofthemachine.com/web/page-to-markdown.py && chmod +x page-to-markdown.py && ./page-to-markdown.py --fraglet-help
3. Run it.
./page-to-markdown.py -p url="<url>" --output article.md
Parameters
| name | type | requirement | description |
|---|---|---|---|
| url | string | required | Page URL to extract |
Outputs
article.md
Provenance
- image
- ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522
- reach
network=required— output depends on live external state; do not cache indefinitely.- procedure_hash
- sha256:10dc2e8d27f66a33aeb7c7bc9ad59feef1d7c0d2450bbbbffa8bcdd22a7c8e18
- receipt
- add
--receipt run.jsonto any run;receipt/attest.pyverifies it against this hash without re-running.
Source
page-to-markdown.py · 59 lines · sha256 10dc2e8d27f6…
#!/usr/bin/env -S fragletc --image ofthemachine/python3@sha256:a015b4f9f0e7c1648f8fcd4164f6a296dd3c9189404f88f333a9764d08d13522
#: d=Extract clean, readable Markdown article text from any web page using Trafilatura, stripping ads, navigations, sidebars, and boilerplate.
#: when=Use when the user wants the article text of a web page, or a page must be read without ads, navigation, and boilerplate.
#: network=required
#: stdin=none
#: param=url:required:d=Page URL to extract
#: output=article.md
import os
import sys
import urllib.request
import trafilatura
url = os.environ.get("URL", "").strip()
if not url:
print("Error: URL parameter is required", file=sys.stderr)
sys.exit(1)
req = urllib.request.Request(
url,
headers={
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
},
)
try:
with urllib.request.urlopen(req, timeout=30) as resp:
html_content = resp.read().decode("utf-8", errors="replace")
except Exception as e:
print(f"Error fetching URL '{url}': {e}", file=sys.stderr)
sys.exit(1)
extracted = trafilatura.extract(
html_content,
include_links=True,
include_images=True,
include_tables=True,
output_format="markdown",
url=url,
)
if not extracted or not extracted.strip():
# Fallback to basic text extraction if trafilatura finds no main body article
extracted = trafilatura.html2txt(html_content)
if not extracted or not extracted.strip():
extracted = f"# {url}\n\n*No readable article body could be extracted from this page.*"
# Write to output file if /output exists, otherwise write to stdout or article.md
output_path = "/output/article.md"
if not os.path.exists("/output"):
output_path = "article.md"
with open(output_path, "w", encoding="utf-8") as f:
f.write(extracted)
print(f"Extracted readable markdown ({len(extracted)} chars) -> article.md")