On Writing Faster HTML Parsers

By A. Developer · March 12, 2026

Parsing HTML dominates the CPU profile of many crawlers. The stock choice in Python — BeautifulSoup — trades speed for ergonomics. For a hot path that sees thousands of pages, this is usually the wrong trade.

Why selectolax

The selectolax library wraps lexbor, a C parser that is about 20x faster than lxml for typical DOM operations. It exposes a small, sharp API:

• HTMLParser(html) replaces BeautifulSoup(html, "lxml")
• tree.css(selector) replaces soup.select(selector)
• node.text() replaces el.get_text()

Caveats

There are a few features selectolax does not support: deep tree mutation, certain regex-driven navigation helpers. For those, keep BS4.

Right tool, right path.