Metadata-Version: 2.5
Name: occu_choice_pf_1
Version: 0.4.0
Summary: Tables for a survey experiment on perceived wage differences and job search, run on Statistics Denmark microdata
Author-email: AskerNC <hms467@econ.ku.dk>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: loguru>=0.7
Requires-Dist: numpy>=2.3
Requires-Dist: pandas>=2.2
Requires-Dist: polars>=1.43
Requires-Dist: pyfixest>=0.60.0
Requires-Dist: pytest>=9.1.1
Requires-Dist: scipy>=1.14
Description-Content-Type: text/markdown

# occu-choice-pf

Tables for the survey experiment of *Perceived Wage Differences and Early
Career Job Search Choices*, run on the Statistics Denmark (DST) microdata.
The functions are written and tested locally against simulated data, then
`pip install`ed on the server, where only loading, cleaning and the calls are
typed by hand.

All numbers in this package come from simulated data.

## Use

```python
from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS
```

Tables return a bare `tabular` in the style of the paper's tables (the
`\sym` stars of esttab included); `save_tex` writes it and logs it, so it can
be read before an export request. Defaults (variables, labels) sit in
`occu_choice_pf.labels` and can be overridden per call. See the docstrings,
and `examples/` for every exhibit and for custom tables.

## The table of controls

`reg_control_table(long)` regresses each outcome on `T_vj_vij_job`, one
panel per outcome, with one table column per entry of `cols_control`. Every
column has `coefs` (reported), `common_control`, and the controls of its
entry. The defaults (`labels.COLS_CONTROL`, `labels.COMMON_CONTROL`) give:

| Column | Controls | `cols_control` entry |
| --- | --- | --- |
| (1) | `vj_vij_job` and the person fixed effect (`common_control`) | `{}` |
| (2) | + `vj_vij_job` squared and cubed | `POWERS` |
| (3) | + pre-treatment controls times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS}` |
| (4) | (3) + education field dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **EDUC_FIELD}` |
| (5) | (3) + DISCED (`disced_6d`) dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **DISCED}` |

A control is `{check-mark row label: terms}`, and a term is a column
(`'age'`), a column made into a dummy for each value (`'i.age'`, as Stata's
`i.`), or a polars expression (`c.vj_vij_job ** 2`). Your own columns:

```python
from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])
```

The arguments, the check-mark rows and the input checks are in
`help(reg.reg_control_table)`; the estimation, and what drops out of a
column (all of it logged), in `help(reg)`. `reg.reg_control_fits(long, ...)`
returns the pyfixest fits behind a table, `{outcome: [fit per column]}`, to
look into with `.summary()` or `pf.etable`. `examples/` has custom calls: a
subsample, other regressors, clustering and a second fixed effect.

## The long frame

One row per person and job type (`pnr` x `type_sizerank_job`), with the columns
of `misc_docs/all_waves_long_variable_definitions.txt`: `T`, `vj_vij_job`,
`T_vj_vij_job`, the pre-treatment variables in `labels.PRETREATMENT`,
`educ_field`, `disced_6d`, and the outcomes in `labels.OUTCOMES`:
`spell_1_new_job` (1 for the job type of the first new job after the survey)
and `appl_job` (the number of applications to the job type).

## Tests on the server data

The tests are installed with the package, and pytest with them. After prep,
run them from the server script:

```python
occu_choice_pf.run_tests({'long': long})
```

It logs pytest's output as it comes, so it shows on screen and is in the log
file of `setup_log`, and returns 0 if every test passes. The frame is written
to a temporary folder, deleted afterwards, and the tests run in a new
process. Further arguments go to pytest: `run_tests(frames, '-m', 'data')`
runs the data tests only. It skips the test that compiles the tables with
latexmk, since LaTeX on the server differs.

The data tests (`tests/test_regressions_data.py`, marked `data`) check what
the table assumes of its input but cannot check itself, and build both
default tables:

- one row per person and job type;
- the outcomes, coefficients and controls are numbers or Booleans;
- `T`, the pre-treatment controls and the education columns take one value
  per person;
- `T` is 0 or 1, and `T_vj_vij_job` is `T * vj_vij_job`, missing (null or
  NaN) where `vj_vij_job` is;
- the pre-treatment controls have no missing values, so that the columns of
  a panel rest on the same rows but for what the Poisson regression drops;
- the outcomes are at least 0;
- both default tables build and keep `T_vj_vij_job` and `vj_vij_job` in
  every column, and pyfixest's estimation sample, whose persons the
  Individuals row and `min_n` count, has the fit's *N* rows (a check on the
  server's pyfixest version).

A failed data test shows the offending values. The other tests check the code
on fake data; they give the same result anywhere, so on the server they test
the environment.

## Layout

```text
src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       imported from the top: from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames
```

`simulation/`, `examples/` and `tests/` are installed with the package. To find them:

```bash
python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>
```

Every example also runs in the VS Code Interactive window, whole or cell by
cell (`# %%`).

## Adding a module

1. `functions/<name>.py`: importable at once as `occu_choice_pf.<name>`;
   arguments as `{column: label}` dicts, with defaults in `labels.py`
2. `simulation/<name>.py`: `simulate()`, `save()`, `load()`
3. `examples/<name>/code/run_<name>.py`, plus `logs/.gitkeep`. Take the
   output folder from `root = example_root('<name>')`, and split sections
   with `# %%`. `tests/test_examples.py` runs every example.
4. `tests/test_<name>.py` for the code, and `tests/test_<name>_data.py`,
   marked `data`, for what the functions assume of real frames.

## Development

```bash
uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames
```

The lower bounds in `pyproject.toml` are the server's versions for loguru,
numpy and polars (as for ad-data-pf). To test against exactly those, in a
separate environment so `uv.lock` is not rewritten:

```bash
cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock
```

GitHub Actions does both on every push (`.github/workflows/test.yml`).

## Release

Bump `version` in `pyproject.toml`, run `uv lock`, and turn the
`## Unreleased` section of [CHANGELOG.md](CHANGELOG.md) into the version's
entry. For each change to a call, give the old and the new call, since the
server script is retyped by hand. Commit, then push a tag
`occu_choice_pf-v<version>`; GitHub Actions checks that the tag matches the
version, runs the tests, builds and publishes to PyPI (trusted publishing,
`.github/workflows/publish.yml`):

```bash
git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0
```

The same tag runs `.github/workflows/publish_1.yml`, which publishes the same
code as `occu_choice_pf_1`. The DST server caches a package when it is
installed and won't update that name for 24 hours, so install an exact
version under whichever name hasn't been installed in the last 24 hours:

```bash
pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
```

Both install the import `occu_choice_pf`, and the most recent install is the
one used.

Before the first release, register both names as pending trusted publishers
on pypi.org (Account settings, Publishing): owner `AskerNC`, repository
`occu_choice_pf`, environment `pypi`, and workflow `publish.yml` for
`occu-choice-pf`, `publish_1.yml` for `occu_choice_pf_1`.
