Metadata-Version: 2.4
Name: surveyshield-py
Version: 0.1.3
Summary: Static review of online survey instruments for resistance to AI/bot respondents
Author: Kianté Fernandez, Andrea Low, Jonathan Bogard, Craig R. Fox
License: MIT License
        
        Copyright (c) 2025 Kianté Fernandez
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/kiante-fernandez/survey-shield
Project-URL: Repository, https://github.com/kiante-fernandez/survey-shield
Project-URL: Issues, https://github.com/kiante-fernandez/survey-shield/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.27
Requires-Dist: gunicorn>=21.0
Requires-Dist: python-multipart>=0.0.7
Requires-Dist: slowapi>=0.1.9
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2.5
Requires-Dist: python-dotenv>=1.0
Requires-Dist: openai>=1.0
Requires-Dist: langchain-core>=0.3
Requires-Dist: langchain-openai>=0.2
Requires-Dist: langchain-google-genai>=2.0
Requires-Dist: typer>=0.12
Requires-Dist: httpx>=0.24
Requires-Dist: requests>=2.28
Requires-Dist: aiofiles>=23.0
Provides-Extra: live
Requires-Dist: browser-use>=0.9.5; extra == "live"
Requires-Dist: playwright>=1.40; extra == "live"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Requires-Dist: black>=24; extra == "dev"
Requires-Dist: flake8>=6; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4; extra == "dev"
Dynamic: license-file

# 🛡️ Survey Shield

[![PyPI version](https://img.shields.io/pypi/v/surveyshield-py.svg)](https://pypi.org/project/surveyshield-py/)
[![Python versions](https://img.shields.io/pypi/pyversions/surveyshield-py.svg)](https://pypi.org/project/surveyshield-py/)
[![Tests](https://github.com/kiante-fernandez/survey-shield/actions/workflows/test.yml/badge.svg)](https://github.com/kiante-fernandez/survey-shield/actions/workflows/test.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/kiante-fernandez/survey-shield/blob/main/LICENSE)

Static review of online survey instruments for resistance to AI/bot respondents — plus an optional live runtime that drives a real browser through your survey.

## What is Survey Shield?

Survey Shield gives researchers feedback on whether their survey instrument is hardened against AI respondents. Two paths:

1. **Instrument Review** *(primary, static, no browser)* — point it at a Qualtrics `.qsf` export. Multi-agent LLM reviewers fan out across bot-resistance dimensions (attention checks, identity questions, visual-perceptual traps), produce a peer-review-style verdict with verbatim-grounded findings, and render a self-contained HTML report with a copy-paste Methods statement and APA/BibTeX citation.
2. **Take Survey (live runtime)** *(optional `[live]` extra)* — drives a real browser ([browser-use](https://docs.browser-use.com)) through a live Qualtrics URL, reports detected mechanisms after the fact. Costs ~5–10 minutes and real LLM credit per run, so the hosted demo doesn't expose it; install the `[live]` extra to run it locally.

Researchers using Survey Shield mostly want Instrument Review. Reach for the live runtime when you need to exercise the survey end-to-end.

## Install

```bash
pip install surveyshield-py            # static review only — small, no browser
pip install "surveyshield-py[live]"    # adds browser-use + Playwright
playwright install chromium            # only if you installed [live]
```

The PyPI name is `surveyshield-py`; the import name is `surveyshield`.

Set an LLM provider key in your environment (or in a `.env` file in the working directory — Survey Shield loads it via `python-dotenv`):

```env
OPENAI_API_KEY=sk-...        # used unless the model name starts with "gemini"
GOOGLE_API_KEY=...           # used for Gemini models
```

## CLI

```bash
surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input

surveyshield review your_survey.qsf --output report.html --json review.json \
                                    --model gpt-4o-mini

surveyshield take https://qualtrics.com/jfe/form/SV_xxx     # requires [live]
                  --model gemini-3-flash-preview --max-steps 150

surveyshield serve --host 127.0.0.1 --port 8000
# → boots the FastAPI app + bundled React SPA
```

`surveyshield --help` lists every command and flag.

## Python API

```python
import asyncio
import surveyshield

review, parsed = asyncio.run(
    surveyshield.review_qsf(
        "your_survey.qsf",
        model="gpt-4o-mini",
        # api_key="sk-...",      # or rely on env vars
        # dimensions=["attention_checks"],   # default = all
    )
)

print(review.overall_score, review.overall_feedback.headline)

with open("report.html", "w") as f:
    f.write(surveyshield.render_html(review, parsed))
```

The `review` object is a `surveyshield.InstrumentReview` Pydantic model. Power users can compose the lower-level seams directly: `parse_qsf`, `run_review`, `drop_unverified_quotes`, `consolidate_and_summarize`, `aggregate`. See `surveyshield/__init__.py` for the public surface.

For live runtime:

```python
import asyncio, surveyshield  # surveyshield-py[live] installed

result = asyncio.run(surveyshield.take_survey(
    "https://qualtrics.com/jfe/form/SV_xxx",
    model="gemini-3-flash-preview",
    max_steps=150,
))
print(result.success_probability, [m.name for m in result.detected_mechanisms])
```

If the `[live]` extra isn't installed, `surveyshield.take_survey` resolves to `None` and the CLI's `take` command exits with a clear install hint.

## Self-host the hosted UI

```bash
git clone https://github.com/kiante-fernandez/survey-shield
cd survey-shield
./setup.sh                              # creates .conda env + writes .env stub
echo "OPENAI_API_KEY=sk-..." >> .env    # or GOOGLE_API_KEY
surveyshield serve --reload             # → http://localhost:8000
```

The `Take Survey` tab is gated on `live_take_enabled` — `GET /api/v1/survey/config` flips it to `true` once a key is detected in the env.

### Endpoints

- **Web UI**: http://localhost:8000
- **Interactive API docs**: http://localhost:8000/docs
- **Health check**: http://localhost:8000/health
- **Live-runtime config**: http://localhost:8000/api/v1/survey/config

### Models

The hosted UI does not expose a model picker — Instrument Review reviewers run on a sensible default (`gpt-4o-mini`). Self-hosters who want a different model can pass `model_name` directly to the API or CLI. The backend has no allowlist; any model name `langchain-openai`'s `ChatOpenAI` or `langchain-google-genai`'s `ChatGoogleGenerativeAI` accept will be routed by prefix:

- Names starting with `gemini` → Google (requires `GOOGLE_API_KEY`)
- Everything else → OpenAI (requires `OPENAI_API_KEY`)

## API usage (self-host)

### Instrument Review (primary)

```bash
# Submit a QSF for review
curl -F "file=@your_survey.qsf" \
     http://localhost:8000/api/v1/instrument/review
# → {"review_id": "<uuid>", "status": "queued", ...}

# Poll
curl http://localhost:8000/api/v1/instrument/status/<uuid>
# queued → running → completed (~30–90 s)

# Structured JSON
curl http://localhost:8000/api/v1/instrument/results/<uuid>

# Human-readable HTML report
curl "http://localhost:8000/api/v1/instrument/report/<uuid>"

# Download as a file
curl -OJ "http://localhost:8000/api/v1/instrument/report/<uuid>?download=1"
```

### Live runtime (self-host only)

```bash
curl -X POST "http://localhost:8000/api/v1/survey/analyze" \
     -H "Content-Type: application/json" \
     -d '{
       "survey_url": "https://example.com/survey",
       "model_name": "gpt-4o-mini",
       "max_steps": 150,
       "use_vision": true
     }'
# Then poll /api/v1/survey/status/<id> and fetch /api/v1/survey/results/<id>.
```

## What Survey Shield evaluates

### Instrument Review dimensions (primary)

The reviewer fans out across plug-in dimensions defined in [`surveyshield/review/dimensions.py`](https://github.com/kiante-fernandez/survey-shield/blob/main/surveyshield/review/dimensions.py). v1 ships three:

- **`attention_checks`** — explicit IMCs and instructional manipulation checks (Westwood, 2025; PNAS).
- **`identity_questions`** — direct LLM-resistance items (identity probes, reverse-shibboleth questions, impossible-event questions).
- **`visual_perceptual_traps`** — image- and layout-based cognitive traps that exploit vision-language model architectural constraints (Affonso, 2026; JCR).

Adding a new dimension is one entry in `dimensions.py` plus a Pydantic output schema — the reviewer fan-out, aggregator, and HTML report are all plug-in-driven.

Findings are **grounded**: every reviewer finding must cite a verbatim excerpt from the source survey. Survey Shield substring-checks each excerpt against the parsed survey and drops anything that can't be located. What reaches the report is grounded in real survey content.

The product is scoped strictly to **bot resistance**. We do not critique a survey's substantive research design, theoretical framing, or question wording — those remain the researcher's domain.

### Live-runtime mechanisms

When you run a live analysis against a real Qualtrics URL, Survey Shield's browser-use Agent navigates the survey end-to-end and inventories detected mechanisms by category — hover traps, invisible text, timing requirements, CAPTCHAs, honeypot fields, attention checks, and behavioural / mouse-tracking signals. Per-mechanism severity weights (1–10) feed into success-probability and difficulty scores. See [`surveyshield/live/analyzer.py`](https://github.com/kiante-fernandez/survey-shield/blob/main/surveyshield/live/analyzer.py) for the taxonomy and [`surveyshield/live/prompts.py`](https://github.com/kiante-fernandez/survey-shield/blob/main/surveyshield/live/prompts.py) for the agent task.

## Development

### Project structure

```
survey-shield/
├── surveyshield/                # the importable package
│   ├── __init__.py              # public API (review_qsf, render_html, take_survey, …)
│   ├── cli.py                   # Typer CLI (review / take / serve)
│   ├── models/                  # Pydantic schemas
│   ├── review/                  # static review pipeline
│   │   ├── parser.py            #   QSF → ParsedSurvey
│   │   ├── dimensions.py        #   plug-in dimension registry
│   │   ├── reviewer.py          #   fan-out, verify, consolidate, aggregate
│   │   ├── mechanism_context.py
│   │   └── templates/           #   Jinja2 self-contained HTML report
│   ├── live/                    # browser-use runtime ([live] extra)
│   │   ├── analyzer.py          #   SurveyAnalyzer / take_survey
│   │   ├── prompts.py
│   │   └── patches.py
│   └── serve/                   # FastAPI app + bundled React SPA
│       ├── app.py
│       ├── config.py
│       ├── api/{survey,instrument}.py
│       └── static/              # built React (populated by bin/build.sh)
├── frontend/                    # React/TypeScript source (CRA)
├── tests/                       # pytest suite + tiny QSF fixture
├── pyproject.toml               # canonical package metadata
├── Procfile                     # web: gunicorn surveyshield.serve.app:app
├── bin/build.sh                 # React build → surveyshield/serve/static/
└── .github/workflows/           # test.yml + release.yml (PyPI trusted publishing)
```

### Local dev

```bash
./setup.sh                                   # one-time: conda env at ../.conda
pip install -e ".[dev,live]"                 # editable install + tests + browser-use
pytest -q                                    # ~30 tests, no LLM calls
cd frontend && npx tsc --noEmit && npm run build
```

### Releasing

```bash
git tag v0.1.0 && git push --tags
# .github/workflows/release.yml builds the wheel + sdist (with the React SPA
# bundled into surveyshield/serve/static/) and publishes to PyPI via OIDC.
```

The PyPI project must be configured with this repo + `release.yml` as a Trusted Publisher before the first push.

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Submit a pull request

## License

MIT License — see [LICENSE](https://github.com/kiante-fernandez/survey-shield/blob/main/LICENSE).

## Citation

If you use Survey Shield in published work:

```bibtex
@misc{fernandez2026surveyshield,
  author = {Fernandez, K. and Low, A. and Bogard, J. and Fox, C. R.},
  title  = {Survey Shield: Static review of online survey instruments for resistance to non-human responses},
  year   = {2026},
  note   = {Manuscript in preparation},
}
```

Every report includes the same citation pre-formatted (APA + BibTeX).

## Disclaimer

Survey Shield is intended for research and testing purposes.
