# Retrieval and DataFrames

Use this reference for data source queries, pagination, flattened records, and pandas output.

## Stream Pages from a Data Source

```python
for page in helper.iter_data_source_pages(data_source_id, page_size=100):
    print(page["id"])
```

Options:

- `limit`: maximum number of pages to yield.
- `page_size`: clamped to Notion's 100 item limit.
- `start_cursor`: continue from a known cursor.
- `retry_policy` and `request_timeout`: per-call overrides.

Use streaming for large data sources to avoid loading all pages into memory.

## Get Page IDs

```python
page_ids = helper.get_data_source_page_ids(data_source_id)
```

## Get JSON Properties

```python
rows = helper.get_data_source_pages_as_json(data_source_id, limit=50)
```

This returns each page's raw `properties` object.

## Flatten Records

```python
records = list(
    helper.iter_data_source_page_records(
        data_source_id,
        include_page_ids=True,
        utc=True,
    )
)
```

Flattening supports common property types: `title`, `status`, `number`, `date`, `url`, `checkbox`, `rich_text`, `email`, `select`, `people`, `phone_number`, `multi_select`, `created_time`, `created_by`, `rollup`, `relation`, `last_edited_by`, `last_edited_time`, `formula`, and `file`.

## DataFrame Output

```python
df = helper.get_data_source_pages_as_dataframe(
    data_source_id,
    limit=500,
    include_page_ids=True,
    utc=True,
)
```

The DataFrame includes `notion_page_id` by default. Dates and datetimes are normalized with `normalize_notion_date`; datetime values use UTC with trailing `Z` when `utc=True`, while date-only values remain `YYYY-MM-DD`.

## When to Keep Raw JSON

Use raw JSON instead of DataFrames when:

- You need full Notion property payloads.
- Multiple people, relation, rollup, or formula values must be preserved exactly.
- You will write data back to Notion.

Use DataFrames when:

- The task is analysis, reporting, filtering, or export.
- A simple flattened representation is acceptable.
