Skip to content

Hacker News: browser-driven scraping

The examples/browser/hn.py example reads the Hacker News front page with a real (Playwright-driven) browser — the tool to reach for when plain HTTP won't do. There is no per-site code: the task only combines the generic browsing starter pack with the generic HTML actions (html_select, get_text_from_list), and the agent figures out the right CSS selector on its own.

The key trick is in the instructions: the agent is told to extract the stories deterministically — grab the rendered page HTML once, then select and read the titles with HTML actions — rather than eyeballing the page. That way the recipe it authors re-scrapes the live page on every replay, with no model in the loop.

The task 

from conatus import task, visible_browsing_actions
from conatus_actions.html_actions import (
    get_text_from_list,
    html_select,
)


@task(
    actions=visible_browsing_actions + html_select + get_text_from_list, # (1)!
    config={"preferred_model": "openai:gpt-5.5", "max_turns": 10},
)
def top_hacker_news_stories() -> list[str]:
    """List the current Hacker News front-page story titles.

    Steps:
        1. Open a browser and go to https://news.ycombinator.com/.
        2. EXTRACT DETERMINISTICALLY so the recipe re-scrapes: call
           `browser_page_html` to get the page HTML, then `html_select` with a
           CSS selector matching the story-title links, then
           `get_text_from_list` to read their text.
        3. `terminate` by passing the resulting list VARIABLE -- never retype
           the stories as a literal list.
    """
  1. Starter packs and individual actions compose with +. visible_browsing_actions opens a browser window you can watch; swap in browsing_actions for a headless run.

Calling the task returns the typed result, and the run carries the recipe the agent authored — printable as standalone Python:

stories = top_hacker_news_stories()

run = top_hacker_news_stories.task_runs[-1]
if run.recipe is not None:
    print(run.recipe.to_code(top_hacker_news_stories.definition))

Run it

Needs an OPENAI_API_KEY:

uv run python examples/browser/hn.py

The script prints the first ten titles and then the generated recipe — the code that re-runs the scrape with no model in the loop.

Source: examples/browser/hn.py