Metadata-Version: 2.4
Name: surveyshield-py
Version: 0.4.1
Summary: Static review of online survey instruments for resistance to AI/bot respondents
Author: Kianté Fernandez, Andrea Low, Jonathan Bogard, Craig R. Fox
License: MIT License
        
        Copyright (c) 2025 Kianté Fernandez
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/kiante-fernandez/survey-shield
Project-URL: Repository, https://github.com/kiante-fernandez/survey-shield
Project-URL: Issues, https://github.com/kiante-fernandez/survey-shield/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.27
Requires-Dist: gunicorn>=21.0
Requires-Dist: python-multipart>=0.0.7
Requires-Dist: slowapi>=0.1.9
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2.5
Requires-Dist: python-dotenv>=1.0
Requires-Dist: openai>=1.0
Requires-Dist: langchain-core>=0.3
Requires-Dist: langchain-openai>=0.2
Requires-Dist: langchain-google-genai>=2.0
Requires-Dist: typer>=0.12
Requires-Dist: httpx>=0.24
Requires-Dist: requests>=2.28
Requires-Dist: aiofiles>=23.0
Provides-Extra: live
Requires-Dist: browser-use>=0.9.5; extra == "live"
Requires-Dist: playwright>=1.40; extra == "live"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Requires-Dist: black>=24; extra == "dev"
Requires-Dist: flake8>=6; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4; extra == "dev"
Dynamic: license-file

# 🛡️ Survey Shield

[![PyPI](https://img.shields.io/pypi/v/surveyshield-py.svg)](https://pypi.org/project/surveyshield-py/)
[![Python](https://img.shields.io/pypi/pyversions/surveyshield-py.svg)](https://pypi.org/project/surveyshield-py/)
[![Tests](https://github.com/kiante-fernandez/survey-shield/actions/workflows/test.yml/badge.svg)](https://github.com/kiante-fernandez/survey-shield/actions/workflows/test.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/kiante-fernandez/survey-shield/blob/main/LICENSE)

**Survey Shield reviews your survey for resistance to AI/bot respondents** and
hands you back a score, a list of concrete weaknesses, and a copy-paste
Methods paragraph you can drop straight into your manuscript.

You upload a Qualtrics `.qsf` file. Survey Shield returns a self-contained
HTML report you can email, print, or attach as a supplement.

## What you get

```
Instrument Review · my-survey.qsf · 2026-05-10

   Defense score                72 / 100
   Bot completion likelihood    28 / 100   (lower is better)
   Defense breadth                5 /   8   categories with a present defense

   Bot-resistance verdict
      "This instrument has a strong attention-check layer but no
       behavioral telemetry. Consider adding reCAPTCHA v3 or a
       total-survey-time gate before payment."

   Per-category review
      Logic                       85 / 100   ·  2 findings
      Visual Reasoning            60 / 100   ·  1 finding
      Traps                       95 / 100   ·  0 findings
      Open Ends                   80 / 100   ·  1 finding
      Mouse and Keyboard Input     0 / 100   ·  1 finding   ← missing
      Behavioral                   0 / 100   ·  2 findings  ← missing
      Context Awareness           40 / 100   ·  3 findings
      ECLAIR                     100 / 100   ·  0 findings

   Methods statement (copy into your paper)
      "We evaluated this instrument for resistance to non-human
       responses using Survey Shield (Fernandez et al., 2026). The
       instrument received an overall defense score of 72/100 and an
       estimated completion-likelihood for automated agents of 28/100.
       Reviews were generated using openai gpt-5.4-mini-2026-03-17 on
       2026-05-11. Review ID: …"
```

The full HTML report adds per-category cards with the quoted survey text,
a Top Recommendations section, and APA + BibTeX citation blocks with copy
buttons.

Same QSF + same model → same score, every time. The methods paragraph
above captures everything a reader needs to reproduce the run; the model
alias you pass (e.g. `gpt-5.4-mini`) is automatically pinned to a dated
snapshot (`gpt-5.4-mini-2026-03-17`) so reports stay reproducible after
the floating alias rotates.

## 60-second quickstart

```bash
pip install surveyshield-py
export OPENAI_API_KEY=sk-...
surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input — open it in a browser
```

That's the whole tool for most researchers. Cost: ~$0.05–$0.30 per review
on the default model, depending on survey size. Wall-clock: 30–90 seconds.

## What Survey Shield evaluates

Eight bot-resistance categories, grounded in Westwood et al. (PNAS 2025)
and related work on detecting automated respondents:

| Category | What it tests |
|---|---|
| **Logic** | Cognitive Reflection Test items, Sally-Anne theory-of-mind, syllogisms, impossible-event probes |
| **Visual Reasoning** | Image-based illusions, counting elements, perspective tasks |
| **Traps** | Attention checks, human-attestation oaths, invisible-text instructions |
| **Open Ends** | Knowledge-gap probes ("first paragraph of the Constitution"), reverse-shibboleths |
| **Mouse and Keyboard Input** | Map clicks, drag-and-drop, keystroke-timing tracking |
| **Behavioral** | reCAPTCHA v3, IAT latencies, total-survey-time gating |
| **Context Awareness** | "Is it raining where you are?" verified against a weather API |
| **ECLAIR** | Refusal probes — questions safety-tuned LLMs refuse but humans answer freely |

### Scope: what Survey Shield does *not* do

Survey Shield is scoped strictly to bot resistance. **It does not critique
your research design, theoretical framing, or question wording.** Those
remain your domain. Run the same QSF through Survey Shield twice and
you'll get the same score; what neither run will tell you is whether the
underlying questions are good research.

## Install

```bash
pip install surveyshield-py            # review-only — small, no browser
pip install "surveyshield-py[live]"    # adds browser-use + Playwright (live mode)
```

The PyPI name is `surveyshield-py`; the import name is `surveyshield`.

Set an LLM provider key in your environment (or a `.env` file in the working
directory — Survey Shield loads it via `python-dotenv`):

```env
OPENAI_API_KEY=sk-...        # default
GOOGLE_API_KEY=...           # for Gemini models (model name starting with "gemini")
```

## Usage

### CLI

```bash
surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input

surveyshield review your_survey.qsf --output report.html --json review.json
surveyshield review your_survey.qsf --model gpt-4o --categories logical,content-traps

surveyshield serve --host 127.0.0.1 --port 8000
# → boots the FastAPI app + web UI at http://localhost:8000
```

`surveyshield --help` lists every command and flag.

### Batch audit (many QSFs at once)

`surveyshield audit` runs `review` over a directory of QSFs (or a
manifest CSV) and emits a long-format CSV joinable with manifest metadata
(year, journal, study_id, …). One row per `(paper × rubric × category)`,
plus a per-paper HTML report + JSON under the output directory.

```bash
surveyshield audit path/to/jcr_qsfs/ \
    --rubrics traps,traps-behavioral,traps-behavioral-visual,full \
    --output-dir audit_output/

# Manifest form (preferred for the paper — extra columns pass through):
surveyshield audit manifest.csv --rubrics full
# manifest.csv columns: paper_id, qsf_path, year, journal, study_id, ...
```

The CSV is the analysis hand-off: load it into pandas / R for whatever
figures the downstream paper or report needs.

### Python

```python
import asyncio
import surveyshield

review, _parsed = asyncio.run(
    surveyshield.review_qsf(
        "your_survey.qsf",
        model="gpt-5.4-mini",
        # categories=["content-traps", "eclaire"],   # default = all 8
    )
)

print(review.overall_score, review.overall_feedback.headline)
print(review.methods_statement)

with open("report.html", "w") as f:
    f.write(surveyshield.render_html(review))
```

The `review` object is a `surveyshield.InstrumentReview` Pydantic model
with `categories`, `recommendations`, `overall_feedback`, `parameters`,
and `methods_statement` fields. See [`surveyshield/__init__.py`](https://github.com/kiante-fernandez/survey-shield/blob/main/surveyshield/__init__.py)
for the public surface.

## Optional: live runtime

The live runtime drives a real browser through your survey URL and reports
which categories actually blocked an AI agent (vs. just being present in the
QSF). It costs ~$0.20 and 5–10 minutes per run, so the hosted demo doesn't
expose it.

```bash
pip install "surveyshield-py[live]"
playwright install chromium

surveyshield take https://qualtrics.com/jfe/form/SV_xxx --max-steps 150
```

```python
result = asyncio.run(surveyshield.take_survey(
    "https://qualtrics.com/jfe/form/SV_xxx",
    model="gpt-4o-mini",
    max_steps=150,
))
print(result.defense_score, result.bot_completion_likelihood)
```

Live and static reviews share the same 8-category rubric and produce the
same shape of result — the difference is what they evaluate. Static review
asks *"is this defense implemented in the QSF?"*; live review asks *"did
this defense actually work when an agent ran the survey?"*.

## Self-host the web UI

```bash
git clone https://github.com/kiante-fernandez/survey-shield
cd survey-shield
./setup.sh                              # creates .conda env + .env stub
echo "OPENAI_API_KEY=sk-..." >> .env
surveyshield serve --reload             # → http://localhost:8000
```

Endpoints once it's running:

- Web UI: <http://localhost:8000>
- Interactive API docs: <http://localhost:8000/docs>
- Health: <http://localhost:8000/health>

<details>
<summary>HTTP API (for integration into other tools)</summary>

```bash
# Submit a QSF
curl -F "file=@your_survey.qsf" http://localhost:8000/api/v1/instrument/review
# → {"review_id": "<uuid>", "status": "queued", ...}

# Poll until complete (~30–90 s)
curl http://localhost:8000/api/v1/instrument/status/<uuid>

# Fetch the structured JSON result
curl http://localhost:8000/api/v1/instrument/results/<uuid>

# Or the HTML report
curl http://localhost:8000/api/v1/instrument/report/<uuid>
curl -OJ "http://localhost:8000/api/v1/instrument/report/<uuid>?download=1"
```

The live-runtime endpoint mirrors this shape at `/api/v1/survey/*` and
is only enabled when an LLM API key is set in the env.

</details>

## Citation

If you use Survey Shield in published work:

```bibtex
@misc{fernandez2026surveyshield,
  author = {Fernandez, K. and Low, A. and Bogard, J. and Fox, C. R.},
  title  = {Survey Shield: Static review of online survey instruments for resistance to non-human responses},
  year   = {2026},
  note   = {Manuscript in preparation},
}
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for the local dev setup, test
suite layout, and pull-request workflow. Bug reports + category-rubric
suggestions are very welcome.

## License

MIT — see [LICENSE](https://github.com/kiante-fernandez/survey-shield/blob/main/LICENSE).
