Metadata-Version: 2.5
Name: occu-choice-pf
Version: 0.1.0
Summary: Tables for a survey experiment on perceived wage differences and job search, run on Statistics Denmark microdata
Author-email: AskerNC <hms467@econ.ku.dk>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: loguru>=0.7
Requires-Dist: numpy>=2.3
Requires-Dist: pandas>=2.2
Requires-Dist: polars>=1.43
Requires-Dist: pyfixest>=0.60.0
Requires-Dist: pytest>=9.1.1
Requires-Dist: scipy>=1.14
Description-Content-Type: text/markdown

# occu-choice-pf

Tables for the survey experiment of *Perceived Wage Differences and Early
Career Job Search Choices*, run on the Statistics Denmark (DST) microdata.
The functions are written and tested locally against simulated data, then
`pip install`ed on the server, where only loading, cleaning and the calls are
typed by hand.

All numbers in this package come from simulated data.

## Use

```python
from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.regression_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.regression_table(long, method='ols'), out / 'tab_ols.tex')     # OLS
```

Tables return a bare `tabular` in the style of the paper's tables (the
`\sym` stars of esttab included); `save_tex` writes it and logs it, so it can
be read before an export request. Defaults (variables, labels) sit in
`occu_choice_pf.labels` and can be overridden per call. See the docstrings,
and `examples/` for every exhibit and for custom tables.

## The regression table

`regression_table(long)` regresses each outcome on `T_vj_vij` and `vj_vij`
with a person fixed effect, one panel per outcome, and adds controls column
by column:

| Column | Adds | Argument |
| --- | --- | --- |
| (1) | `T_vj_vij`, `vj_vij`, person fixed effect | `coefs`, `fe` |
| (2) | `vj_vij` squared and cubed | `powers` |
| (3) | pre-treatment controls times `vj_vij` | `controls` |
| (4) | education field dummies times `vj_vij` | `dummies` |
| (5) | DISCED (`disced_6d`) dummies times `vj_vij` | `dummies` |

`method='poisson'` (the default) is `pyfixest.fepois`, PPML as ppmlhdfe in
Stata; `method='ols'` is `pyfixest.feols`, as reghdfe. Standard errors are
clustered on the person (`cluster='pnr'`), CRV1 with pyfixest's small-sample
correction, and the stars (`* p<0.1, ** p<0.05, *** p<0.01`) use pyfixest's
p-values. `reg.estimate(long, ...)` returns the pyfixest fits behind a table,
`{outcome: [fit per column]}`, to look into with `.summary()` or `pf.etable`.

The arguments say which columns go in and what they are called:

- `outcomes={column: panel title}`: a count or a 0/1 dummy per person and
  job type; one panel each, lettered A), B), ... when there are several.
- `coefs={column: row label}`: the regressors of every column, all reported.
- `x='vj_vij'`: the column the powers and interactions are built from.
- `powers={label: [2, 3]}`, `controls={label: [column, ...]}`,
  `dummies={label: column}`: each entry adds one column, in that order, and
  its key labels the check-mark row. `{}` adds none. `dummies` makes a dummy
  for each value but the lowest.
- `fe={column: label}`: the one fixed effect; `cluster`: a column.
- `\n` in a label breaks it into lines.

What drops out of a column, all of it logged:

- rows with a missing (null or NaN) value in any variable of the column; a
  row with a null in a `dummies` column leaves from that table column on;
- singletons, and in the Poisson regression persons whose outcome is 0 in
  every row (they carry no information there; pyfixest logs them as removed
  "because of separation");
- in the Poisson regression, separated rows: an outcome of 0 that some
  combination of the regressors predicts exactly. They are found with an
  exact linear program (scipy) and dropped first, as ppmlhdfe does. The
  typical case is a person alone in a DISCED code whose outcome is at the
  extreme of `vj_vij`. Left in, they make pyfixest fail ("Demeaning failed");
- regressors collinear with earlier ones or with the fixed effect (some
  DISCED codes within a field). A reported coefficient that is dropped
  leaves its cells empty.

So the persons and rows below each panel can differ between panels and
columns. Cells resting on fewer than `min_n=5` persons are left empty (DST
disclosure).

Inputs are checked. The following raise an error: an outcome, coefficient or
control that is not a number or Boolean (a categorical column goes in
`dummies`), an infinite value, a `dummies` column with one value, more than
one fixed effect, and an unknown `method`.

## The long frame

One row per person and job type (`pnr` x `type_sizerank`), with the columns
of `misc_docs/all_waves_long_variable_definitions.txt`: `T`, `vj_vij`,
`T_vj_vij`, the pre-treatment variables in `labels.PRETREATMENT`,
`educ_field`, `disced_6d`, and the outcomes. **The outcome names in
`labels.OUTCOMES` (`job_j`, `appl_j`) are placeholders**; pass the server's
names in the call, or set them in `labels.py` for the next release.

## Tests on the server data

The tests are installed with the package, and pytest with them. After prep,
run them from the server script:

```python
occu_choice_pf.run_tests({'long': long})
```

It logs pytest's output as it comes, so it shows on screen and is in the log
file of `setup_log`, and returns 0 if every test passes. The frame is written
to a temporary folder, deleted afterwards, and the tests run in a new
process. Further arguments go to pytest: `run_tests(frames, '-m', 'data')`
runs the data tests only. It skips the test that compiles the tables with
latexmk, since LaTeX on the server differs.

The data tests (`tests/test_regressions_data.py`, marked `data`) check what
the table assumes of its input but cannot check itself, and build both
default tables:

- one row per person and job type;
- the outcomes, coefficients and controls are numbers or Booleans;
- `T`, the pre-treatment controls and the education columns take one value
  per person;
- `T` is 0 or 1, and `T_vj_vij` is `T * vj_vij`, null where `vj_vij` is;
- the outcomes are at least 0;
- both default tables build, keep `T_vj_vij` and `vj_vij` in every column,
  and count no more rows or persons than the frame has.

A failed data test shows the offending values. The other tests check the code
on fake data; they give the same result anywhere, so on the server they test
the environment.

## Layout

```text
src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       imported from the top: from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames
```

`simulation/`, `examples/` and `tests/` are installed with the package. To find them:

```bash
python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>
```

Every example also runs in the VS Code Interactive window, whole or cell by
cell (`# %%`).

## Adding a module

1. `functions/<name>.py`: importable at once as `occu_choice_pf.<name>`;
   arguments as `{column: label}` dicts, with defaults in `labels.py`
2. `simulation/<name>.py`: `simulate()`, `save()`, `load()`
3. `examples/<name>/code/run_<name>.py`, plus `logs/.gitkeep`. Take the
   output folder from `root = example_root('<name>')`, and split sections
   with `# %%`. `tests/test_examples.py` runs every example.
4. `tests/test_<name>.py` for the code, and `tests/test_<name>_data.py`,
   marked `data`, for what the functions assume of real frames.

## Development

```bash
uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames
```

The lower bounds in `pyproject.toml` are the server's versions for loguru,
numpy and polars (as for ad-data-pf). To test against exactly those, in a
separate environment so `uv.lock` is not rewritten:

```bash
cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock
```

GitHub Actions does both on every push (`.github/workflows/test.yml`).

## Release

Bump `version` in `pyproject.toml`, run `uv lock`, and turn the
`## Unreleased` section of [CHANGELOG.md](CHANGELOG.md) into the version's
entry. For each change to a call, give the old and the new call, since the
server script is retyped by hand. Commit, then push a tag
`occu_choice_pf-v<version>`; GitHub Actions checks that the tag matches the
version, runs the tests, builds and publishes to PyPI (trusted publishing,
`.github/workflows/publish.yml`):

```bash
git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0
```

The same tag runs `.github/workflows/publish_1.yml`, which publishes the same
code as `occu_choice_pf_1`. The DST server caches a package when it is
installed and won't update that name for 24 hours, so install an exact
version under whichever name hasn't been installed in the last 24 hours:

```bash
pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
```

Both install the import `occu_choice_pf`, and the most recent install is the
one used.

Before the first release, register both names as pending trusted publishers
on pypi.org (Account settings, Publishing): owner `AskerNC`, repository
`occu_choice_pf`, environment `pypi`, and workflow `publish.yml` for
`occu-choice-pf`, `publish_1.yml` for `occu_choice_pf_1`.
