Metadata-Version: 2.5
Name: occu_choice_pf_2
Version: 0.6.1
Summary: Tables for a survey experiment on perceived wage differences and job search, run on Statistics Denmark microdata
Author-email: AskerNC <hms467@econ.ku.dk>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: loguru>=0.7
Requires-Dist: numpy>=2.3
Requires-Dist: pandas>=2.2
Requires-Dist: polars>=1.43
Requires-Dist: pyfixest>=0.60.0
Requires-Dist: pytest>=9.1.1
Requires-Dist: scipy>=1.14
Description-Content-Type: text/markdown

# occu-choice-pf

Tables for the survey experiment of *Perceived Wage Differences and Early
Career Job Search Choices*.
The functions are written and tested locally against simulated data, then run on the actual data.
All numbers in this package come from simulated data.

## Use

```python
from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS
```

Tables return a bare `tabular` in the style of the paper's tables (the
`\sym` stars of esttab included); `save_tex` writes it and logs it, so it can
be read before an export request. Defaults (variables, labels) sit in
`occu_choice_pf.labels` and can be overridden per call. See the docstrings,
and `examples/` for every exhibit and for custom tables.

## The table of controls

`reg_control_table(long)` regresses each outcome on `T_vj_vij_job`, one
panel per outcome, with one table column per entry of `cols_control`. Every
column has `coefs` (reported), `common_control`, and the controls of its
entry. The defaults (`labels.COLS_CONTROL`, `labels.COMMON_CONTROL`) give:

| Column | Controls | `cols_control` entry |
| --- | --- | --- |
| (1) | `vj_vij_job` and the person fixed effect (`common_control`) | `{}` |
| (2) | + `vj_vij_job` squared and cubed | `POWERS` |
| (3) | + pre-treatment controls times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS}` |
| (4) | (3) + education field dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **EDUC_FIELD}` |
| (5) | (3) + DISCED (`disced_6d`) dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **DISCED}` |

A control is `{check-mark row label: terms}`, and a term is a column
(`'age'`), a column made into a dummy for each value (`'i.age'`, as Stata's
`i.`), or a polars expression (`c.vj_vij_job ** 2`). Your own columns:

```python
from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])
```

Every term of `cols_control` is times `vj_vij_job`, except one wrapped in
`reg.plain()`, which enters its column as it is; an `i.<column>` there is a
fixed effect absorbed in that column only:

```python
from occu_choice_pf.labels import JOB
reg.reg_control_table(long, cols_control=[{}, {'Job type FE': reg.plain('i.' + JOB)}])
```

The arguments, the check-mark rows and the input checks are in
`help(reg.reg_control_table)`; the estimation, and what drops out of a
column (all of it logged), in `help(reg)`. `reg.reg_control_fits(long, ...)`
returns the pyfixest fits behind a table, `{outcome: [fit per column]}`, to
look into with `.summary()` or `pf.etable`. `examples/` has custom calls: a
subsample, other regressors, clustering and a second fixed effect.

## The table of wage gap underestimation

`underestimation_gap.gap_table(rows)` is Table 3 of the paper: one row per
subsample, `rows={label: frame}`, each frame the extra-long frame or a
subsample of it, made with polars before the call.

```python
from occu_choice_pf import underestimation_gap as ug

clean = {'d_vj_vij': c.d_vj_vij_clean, 'd_Vj_Vij_rate': c.d_Vj_Vij_rate_clean}
rows = {
    'Full sample': elong,
    **{f'Actual log wage gap $>$ {x}': elong.filter(c.d_vj > x) for x in [0.05, 0.1]},
    'Full sample, correcting reporting errors': elong.with_columns(**clean),
    'Reweighted for non-participation': elong,
}
weights = {'Reweighted for non-participation': 'weight_invited'}
save_tex(ug.gap_table(rows, weights), out / 'tab_estimation_gaps.tex')
save_tex(ug.gap_table(rows, weights, bootstrap=1000), out / 'tab_estimation_gaps_se.tex')
```

The rows are numbered. `bootstrap=B` adds standard errors from B draws of
persons with replacement. `compare=[(label, label)]` puts the p-value of the
difference between two rows under them (`$p$-value of difference (1)-(2)`),
and a line under that. The rows need not be next to each other, and a row
may be in several pairs: `compare=[('Men', 'Women'), ('Men', 'Women, reweighted to (1)')]`
puts both p-values under the three rows (the rows from the first to the last
of a pair are a block, joined with the blocks it overlaps). `p_at_end=True`
puts all the p-values in the last rows of the table instead.
`share_column` sets column 7: `'underestimator'` (the default), the share of
individuals who are underestimators; `'zero_gap'`, the share of job pairs
with no perceived wage gap (`d_Vij == 0`); a list of both,
`['underestimator', 'zero_gap']`, columns 7 and 8 in that order; or `None`,
no column 7. The frames need not have a column not shown. A frame with NaN
raises, since a filter such as `c.d_vj > 0.05` keeps NaN: make it missing in
prep, `elong = elong.with_columns(pl.col(pl.Float64).fill_nan(None))`. `ps_group`
reweights one group to the other with a propensity score model estimated
anew in each draw, as Table C1 of the parental transmission paper: it adds
`ps_weight` (reweighted to the other group) and `ps_support` (1 for persons
with a weight) to every row, to name in `weights`:

```python
no, yes = elong.filter(c.par_anyjobtype == 0), elong.filter(c.par_anyjobtype == 1)
rows = {'Neither': no, 'Either': yes, 'Neither, weightable': no, 'Either, reweighted to (3)': yes}
ug.gap_table(rows, {'Neither, weightable': 'ps_support', 'Either, reweighted to (3)': 'ps_weight'},
             bootstrap=1000, ps_group='par_anyjobtype',
             compare=[('Neither', 'Either'), ('Neither, weightable', 'Either, reweighted to (3)')])
```

The model is a logit of `ps_group` on `ps_xs` with the fixed effects
`ps_fes` (majors, `disced_6d`, by default), in pyfixest; with one fixed
effect and no `ps_xs` it is the group's share in each major. The cells, the
weights and what is left out are in `help(ug.gap_table)` and
`help(ug.ps_weights)`.

## The long frame

One row per person and job type (`pnr` x `type_sizerank_job`), with the columns
of `misc_docs/all_waves_long_variable_definitions.txt`: `T`, `vj_vij_job`,
`T_vj_vij_job`, the pre-treatment variables in `labels.PRETREATMENT`,
`educ_field`, `disced_6d`, and the outcomes in `labels.OUTCOMES`:
`spell_1_new_job` (1 for the job type of the first new job after the survey)
and `appl_job` (the number of applications to the job type).

## The extra-long frame

One row per person and ordered pair of job types (`pnr` x
`type_sizerank_job` x `type_sizerank_jobj`), each pair in both orders; a
`d_` column is job j minus job j'. `gap_table` uses `d_Vj_Vij_rate`,
`d_vj_vij`, `d_v_vi_max_notmaxV_rate` (one value per person) and, to
reweight, `disced_6d`.

## Tests on the server data

The tests are installed with the package, and pytest with them. After prep,
run them from the server script:

```python
occu_choice_pf.run_tests({'long': long, 'elong': elong})
```

The data tests of a frame left out are skipped.

It logs pytest's output as it comes, so it shows on screen and is in the log
file of `setup_log`, and returns 0 if every test passes. The frame is written
to a temporary folder, deleted afterwards, and the tests run in a new
process. Further arguments go to pytest: `run_tests(frames, '-m', 'data')`
runs the data tests only. It skips the test that compiles the tables with
latexmk, since LaTeX on the server differs.

The data tests (`tests/test_regressions_data.py`, marked `data`) check what
the table assumes of its input but cannot check itself, and build both
default tables:

- one row per person and job type;
- the outcomes, coefficients and controls are numbers or Booleans;
- `T`, the pre-treatment controls and the education columns take one value
  per person;
- `T` is 0 or 1, and `T_vj_vij_job` is `T * vj_vij_job`, missing (null or
  NaN) where `vj_vij_job` is;
- the pre-treatment controls have no missing values, so that the columns of
  a panel rest on the same rows but for what the Poisson regression drops;
- the outcomes are at least 0;
- both default tables build and keep `T_vj_vij_job` and `vj_vij_job` in
  every column, and pyfixest's estimation sample, whose persons the
  Individuals row and `min_n` count, has the fit's *N* rows (a check on the
  server's pyfixest version).

A failed data test shows the offending values. The other tests check the code
on fake data; they give the same result anywhere, so on the server they test
the environment.

## Layout

```text
src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       underestimation_gap.py; imported from the top:
│                       from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames
```

`simulation/`, `examples/` and `tests/` are installed with the package. To find them:

```bash
python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>
```

Every example also runs in the VS Code Interactive window, whole or cell by
cell (`# %%`).

## Adding a module

1. `functions/<name>.py`: importable at once as `occu_choice_pf.<name>`;
   arguments as `{column: label}` dicts, with defaults in `labels.py`
2. `simulation/<name>.py`: `simulate()`, `save()`, `load()`
3. `examples/<name>/code/run_<name>.py`, plus `logs/.gitkeep`. Take the
   output folder from `root = example_root('<name>')`, and split sections
   with `# %%`. `tests/test_examples.py` runs every example.
4. `tests/test_<name>.py` for the code, and `tests/test_<name>_data.py`,
   marked `data`, for what the functions assume of real frames.

## Development

```bash
uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames
```

The lower bounds in `pyproject.toml` are the server's versions for loguru,
numpy and polars (as for ad-data-pf). To test against exactly those, in a
separate environment so `uv.lock` is not rewritten:

```bash
cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock
```

GitHub Actions does both on every push (`.github/workflows/test.yml`).

## Release

Bump `version` in `pyproject.toml`, run `uv lock`, and turn the
`## Unreleased` section of [CHANGELOG.md](CHANGELOG.md) into the version's
entry. For each change to a call, give the old and the new call, since the
server script is retyped by hand. Commit, then push a tag
`occu_choice_pf-v<version>`; GitHub Actions checks that the tag matches the
version, runs the tests, builds and publishes to PyPI (trusted publishing,
`.github/workflows/publish.yml`):

```bash
git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0
```

The same tag runs `.github/workflows/publish_1.yml` and `publish_2.yml`,
which publish the same code as `occu_choice_pf_1` and `occu_choice_pf_2`.
The DST server caches a package when it is installed and won't update that
name for 24 hours, so install an exact version under whichever name hasn't
been installed in the last 24 hours:

```bash
pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
pip install occu_choice_pf_2==X.Y.Z
```

All three install the import `occu_choice_pf`, and the most recent install is
the one used.

Before a name's first release, register it as a pending trusted publisher on
pypi.org (Account settings, Publishing): owner `AskerNC`, repository
`occu_choice_pf`, environment `pypi`, and workflow `publish.yml` for
`occu-choice-pf`, `publish_1.yml` for `occu_choice_pf_1`, `publish_2.yml` for
`occu_choice_pf_2`.
