Metadata-Version: 2.4
Name: synthetic-surveys
Version: 0.1.1
Summary: Generate surveys, simulate synthetic respondents, and produce exploratory reports
Author: Christian Arthur
License-Expression: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.6
Requires-Dist: openai>=1.40.0
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: scipy>=1.10
Requires-Dist: scikit-learn>=1.3
Requires-Dist: requests>=2.31
Requires-Dist: Pillow>=10.0
Requires-Dist: pypdfium2<6,>=5
Requires-Dist: matplotlib>=3.7
Requires-Dist: structlog>=23.1
Requires-Dist: keyring>=25.6
Requires-Dist: prompt-toolkit<4,>=3.0.52
Provides-Extra: executable
Requires-Dist: pyinstaller<7,>=6.16; extra == "executable"
Requires-Dist: pdfplumber>=0.10; extra == "executable"
Provides-Extra: ocr
Requires-Dist: pdfplumber>=0.10; extra == "ocr"
Requires-Dist: pytesseract>=0.3.10; extra == "ocr"
Provides-Extra: nlp
Requires-Dist: sentence-transformers>=2.2.2; extra == "nlp"
Provides-Extra: report
Requires-Dist: reportlab>=4.0; extra == "report"
Provides-Extra: gui
Requires-Dist: customtkinter>=5.2; extra == "gui"
Provides-Extra: desktop
Requires-Dist: customtkinter>=5.2; extra == "desktop"
Requires-Dist: pyinstaller>=6.0; extra == "desktop"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: ty>=0.0.21; extra == "dev"
Requires-Dist: ruff>=0.5.0; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Requires-Dist: black>=23.12; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Dynamic: license-file

# Synthetic Surveys

A CLI for exploring policy and program questions with synthetic survey respondents. Generate a ten-question survey, review or edit it, simulate a panel, and save a Markdown report with results and an interpretation.

These are model-generated responses, not measured public opinion. Use them to explore questions and hypotheses before research with people.

## Install

Synthetic Surveys is available on [PyPI](https://pypi.org/project/synthetic-surveys/). With [uv](https://docs.astral.sh/uv/getting-started/installation/) installed, download or update the package and launch it in one command:

```bash
uvx --upgrade --from synthetic-surveys synthetic-surveys
```

For a persistent command available in your terminal:

```bash
uv tool install --upgrade synthetic-surveys
synthetic-surveys
```

Python 3.10 or later is required. Alternatively, use pip in a virtual environment:

```bash
python -m venv .venv
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell:
# .venv\Scripts\Activate.ps1
python -m pip install --upgrade synthetic-surveys
synthetic-surveys
```

Optional native executable previews are built separately for Windows, macOS, and Linux. Publishing a Python wheel does not rebuild those previews. See [executable builds](docs/RELEASE.md#executable-builds) and [installer preparation](docs/DISTRIBUTION.md).

## First run

The default provider is **OpenRouter**. One `OPENROUTER_API_KEY` covers survey generation, respondent simulation, interpretation, and Pro-mode embeddings. Missing credentials trigger a connection menu with browser authorization and hidden API-key entry. The CLI validates OpenRouter/OpenAI keys using an account/models request before offering to save them.

- API usage is billed to your provider account. Start with two or three respondents; Pro mode makes additional requests. Key validation does not generate model tokens and does not guarantee access to every model.
- Saving a key is optional. Credentials use macOS Keychain or the Linux desktop Secret Service through Python keyring. If no secure store is available, the app keeps the key only for the current process. Non-secret preferences remain in `~/.synthetic_surveys/settings.json`.
- Environment variables override saved settings. A bad exported key must be changed or unset; changing the saved key alone will not override it on the next launch.
- On first use, choose browser login (default), paste an API key, or authorize on another device and paste a one-time code. Browser login opens OpenRouter and returns authorization to a temporary localhost listener. Ctrl+C or a timeout returns to setup options; no separate OpenRouter CLI is required.

To replace a key or select a provider:

```bash
synthetic-surveys configure
synthetic-surveys configure --provider openai
```

Direct OpenAI configuration sets OpenAI as both the LLM and embedding provider. Remove any old OpenRouter model overrides when switching providers.

## Your first report

```bash
synthetic-surveys run --mode normal --sample-size 3
```

1. Type or paste your question into the text box. Enter adds a new line; use the arrow keys to edit. Press Tab, then Enter to continue; an arrow appears in the bracket when the continue control is selected.
2. Optionally add survey-design guidance and supporting documents.
3. Choose a sampling profile, review the draft cost, and approve generation. You can change the draft model or question before paying.
4. Review the survey, its respondent background, and any private framing concerns. Accept it, edit the background or questions for free, or approve a paid revision.
5. Choose the respondent model and count, then review an itemized estimate including the final interpretation and any Pro work. Change settings or cancel before starting.
6. Read the results, reported charges, and saved report path.

The question and survey-design context boxes preserve pasted paragraphs and blank lines. A character count shows the limit; if you exceed it, you can edit your text without starting over. Shift+Tab moves back to the text box. Ctrl+C exits. Redirected input uses ordinary line prompts; scripts can pass a multiline question with `--question`.

Reports go into `~/synthetic-surveys/reports/YYYY-MM-DD/`, with a readable survey-title-and-timestamp filename. The title comes from the generated survey, so saving does not require another model request. Choose another reports root with `--output-dir /path/to/reports` (`--dev-dir` remains an alias); date folders are created beneath that root. The CLI is interactive even when you supply a question and sample size. `generate-survey` and `sample-personas` are separate utility commands.

```bash
synthetic-surveys --help
synthetic-surveys generate-survey "Should the city pedestrianize Main Street?"
synthetic-surveys sample-personas 3
```

Normal mode directly selects Likert answers. **Pro mode** generates free-text responses and maps them through embeddings to Likert labels; it needs additional requests and fails clearly if embedding requests fail.

### Simulation size

Choose the number of synthetic respondents based on your budget and the variety of profiles you want to explore. Larger runs can reduce random fluctuations but do not guarantee more realistic answers. There is no validated sample-size recommendation or real-world margin of error for this tool; target population headcount does not determine the required simulation size. A run needs at least one respondent.

### Respondent context

The survey draft includes a neutral background paragraph for respondents to read. The original question and private design context guide survey generation; respondent models receive the approved background, survey items, persona, and supporting materials. Private framing notes remain available for your review. Supporting materials are passed through without being replaced by the background. Original research inputs and private context can still appear in your saved report metadata.

### Cost transparency

The interactive survey flow reads OpenRouter's public model and embedding catalogs without sending your survey text. Estimates use current list prices, approximately three UTF-8 bytes per input token (including system prompts and JSON schemas), and explicit output allowances: 2,200 tokens for a draft, 800/1,000 per Normal/Pro respondent, 1,600 per anchor-generation call, and 400 per interpretation. The review shows approximate token counts, dated per-million-token rates, and an estimated cost for each stage. Allowances are planning assumptions, not measured usage averages. Expanded estimates, including configured output limits, are retained in report metadata; they are not spending caps.

Pro adds one anchor-generation request per survey item, then six embedding requests per item per respondent. The estimate includes repeated anchors and answers (40 tokens per anchor statement and 80 per answer). Its additional work is itemized before you run the simulation. There is no fixed or empirically validated Pro/Normal cost multiplier.

Actual charges can differ because of reasoning, output length, caching, routing, retries, or price changes. Native image/file processing, direct-provider prices, unknown models and embedding fallback costs may be unavailable; incomplete estimates are labeled explicitly. Credit-purchase fees and taxes are excluded. No price lookup makes an inference request.

OpenRouter response-reported charges and token counts are shown after the session and saved in completed reports alongside approved estimates. Missing costs are marked as incomplete, including embeddings if their response omits cost. Failed requests without usage data may still incur charges: your provider's billing activity remains authoritative. Canceling stops future steps, not charges already incurred. These checkpoints apply to the interactive CLI; utility commands and the desktop preview do not use this review flow.

## Weighting and IPUMS

| Profile | Included with the app? | Setup |
| --- | --- | --- |
| General demographic sampling (menu label: No Demographics Weights) | Built-in sampling proportions | None |
| IPUMS Population Sample | No; individual-record extract | Your own IPUMS USA access and a one-time download |

The profiles affect which personas are sampled, not post-hoc weighting of completed responses. Even the general profile has built-in demographic sampling proportions.

Select **IPUMS Population Sample**, then enter exactly one state abbreviation or DC (for example, MA or CA). Multiple states are not accepted. The app checks for a valid local extract before offering a download or a return to the profile menu. You can also prepare it explicitly:

```bash
synthetic-surveys configure --provider ipums
synthetic-surveys download-ipums --state MA
```

Use your own [IPUMS USA account](https://account.ipums.org/usa/) and API key. Existing `IPUMS_API_KEY` credentials take precedence; otherwise the app prompts with hidden entry and offers the existing optional secure-keyring storage. The default sample is `us2024a`; override it with `--ipums-sample` or `SYNTHETIC_SURVEYS_IPUMS_SAMPLE`. Preparation may take several minutes. Valid cached data needs no API key.

Extracts are stored under `~/.synthetic_surveys/data/ipums/<sample>/<STATE>.csv.gz` (or beneath `SYNTHETIC_SURVEYS_HOME` when configured). A companion JSON file records state, sample ID, extract number, download time, and checksum. Reuse verifies those details, required columns, state records, and usable `PERWT` values. Invalid or incomplete caches trigger a new download; failed downloads preserve existing data. State/sample combinations have separate caches.

Personas are drawn with replacement from the chosen state's records using `PERWT`. Each persona is also told its state of residence. Small synthetic samples can differ from the state's population proportions, and matching demographics does not validate the simulated opinions. State, sample, and extract provenance appear in reports.

The old `download-ipums-ma` command remains an alias defaulting to MA and writes the new cache layout. Programmatic sampling and the desktop runner can still read a compatible `~/.synthetic_surveys/data/ipums/massachusetts.csv.gz` when no sample override is requested and no valid modern cache exists. That legacy file is recorded as having unknown sample/extract provenance. The interactive CLI downloads a verified extract instead of assuming a legacy file matches the requested sample.

**Do not commit or redistribute IPUMS microdata.** Redistribution requires IPUMS permission outside the stated publication exception. Follow [IPUMS USA terms](https://usa.ipums.org/usa/terms.shtml), and retain the extract's citation for reports/publications. The downloader specification is in `synthetic_surveys/ipums_api.py`; no IPUMS data is bundled.

## Documents and limits

PDF, JPG, PNG, WebP, and plain-text attachments work without installing a local OCR engine. PDFs and images go to compatible model providers; text files and HTML are extracted locally. Convert Word documents to PDF first. The app checks visual-model support before paid requests and asks you to change incompatible models. Exact model IDs are required when a routing alias cannot be verified.

Models with native PDF support receive PDFs directly. For image-capable models without PDF support, the app renders every page locally and sends the pages as images in order. The renderer installs automatically with the app; no OCR engine is needed. Conversion supports up to 20 pages per PDF at 144 DPI, capped at 2,048 pixels on the longest edge. Longer or unreadable PDFs are rejected before generation; pages are never silently omitted. Rendered pages are reused in memory across calls, but image input remains billable on each request. Small text may be less legible after rendering; native PDF input is preferred.

Visual capability checks use the public OpenRouter model catalog (also as a conservative capability reference for matching direct OpenAI model IDs). If support cannot be verified, visual requests stop before inference. Provider-specific size, context, and access limits can still reject a request. See [OpenRouter PDF inputs](https://openrouter.ai/docs/guides/overview/multimodal/pdfs).

Limits: four attachments, 4 MB per file, 12 MB total, a 4,000-character question, and 2,000 characters of regeneration feedback. Questions, document content, and generated personas are sent to the selected providers. Reports contain research inputs and run metadata, so choose their storage location accordingly.

## Environment variables

All are optional unless needed by the selected feature. Environment values take precedence over saved settings.

| Variable | Purpose/default |
| --- | --- |
| `OPENROUTER_API_KEY` | Default LLM and embedding credential |
| `OPENAI_API_KEY` | Direct OpenAI credential; optional embedding fallback |
| `GEMINI_API_KEY` | Only for direct Gemini embeddings |
| `IPUMS_API_KEY` | Only for downloading an IPUMS extract |
| `SYNTHETIC_SURVEYS_HOME` | App settings/data/cache root; defaults to `~/.synthetic_surveys` |
| `SYNTHETIC_SURVEYS_LLM_PROVIDER` | `openrouter` (default), `openai`, or offline `stub` |
| `SYNTHETIC_SURVEYS_EMBEDDING_PROVIDER` | `openrouter`, `openai`, or `gemini`; CLI defaults to the selected LLM provider |
| `SYNTHETIC_SURVEYS_MODEL` | Override all LLM stages |
| `SYNTHETIC_SURVEYS_SURVEY_MODEL` | Override survey generation model |
| `SYNTHETIC_SURVEYS_SIMULATION_MODEL` | Override simulation model |
| `SYNTHETIC_SURVEYS_SUMMARY_MODEL` | Override interpretation model |
| `SYNTHETIC_SURVEYS_OPENROUTER_EMBEDDING_MODEL` | `openai/text-embedding-3-small` |
| `SYNTHETIC_SURVEYS_OPENAI_EMBEDDING_MODEL` | `text-embedding-3-small` |
| `SYNTHETIC_SURVEYS_EMBEDDING_MODEL` | Direct Gemini embedding model |
| `SYNTHETIC_SURVEYS_IPUMS_SAMPLE` | Download sample override; default `us2024a` |
| `SYNTHETIC_SURVEYS_SSR_EPSILON` | SSR bias offset; default `0.0` |
| `SYNTHETIC_SURVEYS_SSR_TEMPERATURE` | SSR temperature; default `1.0` |

The checked-in defaults use `openai/gpt-5.5` for OpenRouter survey/summary stages and `openai/gpt-5.4-mini` for simulation. Provider availability can change; use model overrides if necessary.

## Local development

Ablation is a development diagnostic, available through the hidden `--ablation` CLI flag. It is absent from the interactive menus and desktop controls:

```bash
synthetic-surveys --mode normal --ablation
```

This adds a second run without demographic, state, or personality persona information and reports distribution similarity. Shared survey content and materials remain unchanged. The control always uses Normal mode; use a Normal primary run to avoid also changing the response method. When explicitly enabled, its additional costs are included in the estimate. Similarity measures profile influence, not agreement with real people.

```bash
uv sync --extra dev
cp .env.example .env
# Edit .env locally to add your key. Never commit it.
```

The application does **not** load `.env` itself. If you use direnv, create `.envrc` containing:

```bash
dotenv_if_exists .env
if [[ -d .venv/bin ]]; then
  PATH_add .venv/bin
fi
```

Then run `direnv allow` with direnv installed and hooked into your shell. Alternatively export the variables in your shell. Both files are gitignored.

```bash
uv run synthetic-surveys
uv run ruff check .
uv run ty check
uv run pytest -q
```

For an offline demo with placeholder responses, use `uv run synthetic-surveys --provider stub --mode normal`. This is not a real simulation. Live OpenAI tests require `SYNTHETIC_SURVEYS_RUN_OPENAI_TESTS=1` and `OPENAI_API_KEY`.

See [research foundations and audit findings](docs/RESEARCH.md), [methodology](docs/METHODOLOGY.md), and [release verification](docs/RELEASE.md). See [LICENSE](LICENSE) for licensing terms.
