Metadata-Version: 2.5
Name: occu_choice_pf_1
Version: 0.7.0
Summary: Tables for a survey experiment on perceived wage differences and job search, run on Statistics Denmark microdata
Author-email: AskerNC <hms467@econ.ku.dk>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: loguru>=0.7
Requires-Dist: matplotlib>=3.9
Requires-Dist: numpy>=2.3
Requires-Dist: pandas>=2.2
Requires-Dist: polars>=1.43
Requires-Dist: pyfixest>=0.60.0
Requires-Dist: pytest>=9.1.1
Requires-Dist: scipy>=1.18.1
Description-Content-Type: text/markdown

# occu-choice-pf

Tables for the survey experiment of *Perceived Wage Differences and Early
Career Job Search Choices*.
The functions are written and tested locally against simulated data, then run on the actual data.
All numbers in this package come from simulated data.

## Use

```python
from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS
```

Tables return a bare `tabular` in the style of the paper's tables (the
`\sym` stars of esttab included); `save_tex` writes it and logs it, so it can
be read before an export request. Defaults (variables, labels) sit in
`occu_choice_pf.labels` and can be overridden per call. See the docstrings,
and `examples/` for every exhibit and for custom tables.

## The table of controls

`reg_control_table(long)` regresses each outcome on `T_vj_vij_job`, one
panel per outcome, with one table column per entry of `cols_control`. Every
column has `coefs` (reported), `common_control`, and the controls of its
entry. The defaults (`labels.COLS_CONTROL`, `labels.COMMON_CONTROL`) give:

| Column | Controls | `cols_control` entry |
| --- | --- | --- |
| (1) | `vj_vij_job` and the person fixed effect (`common_control`) | `{}` |
| (2) | + `vj_vij_job` squared and cubed | `POWERS` |
| (3) | + pre-treatment controls times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS}` |
| (4) | (3) + education field dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **EDUC_FIELD}` |
| (5) | (3) + DISCED (`disced_6d`) dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **DISCED}` |

A control is `{check-mark row label: terms}`, and a term is a column
(`'age'`), a column made into a dummy for each value (`'i.age'`, as Stata's
`i.`), or a polars expression (`c.vj_vij_job ** 2`). Your own columns:

```python
from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])
```

Every term of `cols_control` is times `vj_vij_job`, except one wrapped in
`reg.plain()`, which enters its column as it is; an `i.<column>` there is a
fixed effect absorbed in that column only:

```python
from occu_choice_pf.labels import JOB
reg.reg_control_table(long, cols_control=[{}, {'Job type FE': reg.plain('i.' + JOB)}])
```

The arguments, the check-mark rows and the input checks are in
`help(reg.reg_control_table)`; the estimation, and what drops out of a
column (all of it logged), in `help(reg)`. `reg.reg_control_fits(long, ...)`
returns the pyfixest fits behind a table, `{outcome: [fit per column]}`, to
look into with `.summary()` or `pf.etable`. `examples/` has custom calls: a
subsample, other regressors, clustering and a second fixed effect.

The columns of a panel can rest on different rows: the Poisson regression
with the DISCED dummies drops the separated rows of persons alone in their
DISCED, which the other columns keep. `common_sample=True` estimates every
column on the rows all of them keep, so that the columns differ in their
controls only; `reg.reg_control_samples(long, ...)`, with the arguments of
the table, returns `long` with a Boolean column `in_<outcome>_<column>` per
regression, True in the rows it is estimated on:

```python
save_tex(reg.reg_control_table(long, common_sample=True), out / 'tab_poisson_common.tex')
s = reg.reg_control_samples(long)
s.filter(c.in_appl_job_5)                      # the rows of panel B, column (5)
```

A separated row leaves the person's other rows in: with `vj_vij_job`
winsorized, a person alone in their DISCED whose rows with applications are
tied at the bound keeps those. `drop_persons='separated'` drops a person
with a separated row whole, and `drop_persons='any'` a person who loses any
row, for a missing value, separation or pyfixest; both take the arguments
above too:

```python
save_tex(reg.reg_control_table(long, drop_persons='separated'), out / 'tab_poisson_sep.tex')
save_tex(reg.reg_control_table(long, drop_persons='any'), out / 'tab_poisson_whole.tex')
```

`share_range=(low, high)` keeps the persons of a major (`share_educ`, by
default `disced_6d`) only where its treated share among the persons of the
regression, after separation and pyfixest's drops, is from `low` to `high`.
The persons of the other majors are dropped whole and the regression refit
until every major left is in the range. Each table column and outcome has
its own sample, and so its own shares:

```python
save_tex(reg.reg_control_table(long, share_range=(0.1, 0.9)), out / 'tab_poisson_share.tex')
```

## The treated share within majors

With the major dummies times `vj_vij_job` (columns 4 and 5), the treatment
effect is estimated within majors: a major whose persons are all treated, or
all in the control group, adds next to nothing, and a major weighs more the
closer its treated share is to one half. Without them, a treated share that
differs between majors moves the estimate where the slope on `vj_vij_job`
differs between majors. `treatment_shares` shows how large the persons'
majors are and how far the treated share of their major is from what random
assignment gives (`T` permuted across the persons, 1000 draws, no strata).
Three table columns or a plot per column of `labels.EDUC`: major field and
DISCED. The functions take the persons frame, one row per person, and raise
on a person with more than one row:

```python
from occu_choice_pf import save_fig
from occu_choice_pf import treatment_shares as ts

save_tex(ts.percentile_table(persons), out / 'tab_share_percentiles.tex')
save_fig(ts.share_figure(persons), out / 'fig_share_hist.png')
ts.share_frame(persons, 'disced_6d')  # one row per major: on the server only
```

- `percentile_table`: percentiles over persons of the size of their major
  and of its treated share (data and random assignment). Each percentile is
  the mean of the 9 persons around it (`window`), never one person's value.
  Below them, the standard deviation over persons of the treated share of
  their major (data and random assignment) and its *p*-value under random
  assignment: a permutation chi-squared test of treatment by major.
- `share_figure`: a histogram of the persons by the treated share of their
  major (0, ten intervals, 1), with the mean and the 95% range of random
  assignment; a matplotlib figure, the numbers plotted in the log.
  `save_fig` writes it, a PNG at 300 dpi unless the path ends in `.pdf`.

The persons are those of the frame given, so the persons of a regression,
`persons.join(s.filter(c.in_appl_job_5), on='pnr', how='semi')` with
`s = reg.reg_control_samples(long)`, give the shares within it.

Disclosure: no number shown rests on fewer than `min_n` (5) persons, nor can
one be worked out from the others, the persons treated (in the paper's
balance table) included. A count below 5, 0 included, reads <5. Percentiles
are left empty where their 9-person windows overlap or leave 1-4 persons
outside; the shares, the test and random assignment, which depends on the
persons treated, with fewer than 5 treated or untreated. A bar of fewer than 5
persons reads <5; the bars at 0 and 1 are left out where the persons in
all-treated or all-control majors, or the treated or untreated of the
others, are 1-4; where the bars left out add up to 1-4 persons, the smallest
interval bar is left out too. A 0 can still be worked out. `share_frame`
lists every major, however small, and is for the server only.

## The table of wage gap underestimation

`underestimation_gap.gap_table(rows)` is Table 3 of the paper: one row per
subsample, `rows={label: frame}`, each frame the extra-long frame or a
subsample of it, made with polars before the call.

```python
from occu_choice_pf import underestimation_gap as ug

clean = {'d_vj_vij': c.d_vj_vij_clean, 'd_Vj_Vij_rate': c.d_Vj_Vij_rate_clean}
rows = {
    'Full sample': elong,
    **{f'Actual log wage gap $>$ {x}': elong.filter(c.d_vj > x) for x in [0.05, 0.1]},
    'Full sample, correcting reporting errors': elong.with_columns(**clean),
    'Reweighted for non-participation': elong,
}
weights = {'Reweighted for non-participation': 'weight_invited'}
save_tex(ug.gap_table(rows, weights), out / 'tab_estimation_gaps.tex')
save_tex(ug.gap_table(rows, weights, bootstrap=1000), out / 'tab_estimation_gaps_se.tex')
```

The rows are numbered. `bootstrap=B` adds standard errors from B draws of
persons with replacement. `compare=[(label, label)]` puts the p-value of the
difference between two rows under them (`$p$-value of difference (1)-(2)`),
and a line under that. The rows need not be next to each other, and a row
may be in several pairs: `compare=[('Men', 'Women'), ('Men', 'Women, reweighted to (1)')]`
puts both p-values under the three rows (the rows from the first to the last
of a pair are a block, joined with the blocks it overlaps). `p_at_end=True`
puts all the p-values in the last rows of the table instead.
`share_column` sets column 7: `'underestimator'` (the default), the share of
individuals who are underestimators; `'zero_gap'`, the share of job pairs
with no perceived wage gap (`d_Vij == 0`); a list of both,
`['underestimator', 'zero_gap']`, columns 7 and 8 in that order; or `None`,
no column 7. The frames need not have a column not shown. A frame with NaN
raises, since a filter such as `c.d_vj > 0.05` keeps NaN: make it missing in
prep, `elong = elong.with_columns(pl.col(pl.Float64).fill_nan(None))`. `ps_group`
reweights one group to the other with a propensity score model estimated
anew in each draw, as Table C1 of the parental transmission paper: it adds
`ps_weight` (reweighted to the other group) and `ps_support` (1 for persons
with a weight) to every row, to name in `weights`:

```python
no, yes = elong.filter(c.par_anyjobtype == 0), elong.filter(c.par_anyjobtype == 1)
rows = {'Neither': no, 'Either': yes, 'Neither, weightable': no, 'Either, reweighted to (3)': yes}
ug.gap_table(rows, {'Neither, weightable': 'ps_support', 'Either, reweighted to (3)': 'ps_weight'},
             bootstrap=1000, ps_group='par_anyjobtype',
             compare=[('Neither', 'Either'), ('Neither, weightable', 'Either, reweighted to (3)')])
```

The model is a logit of `ps_group` on `ps_xs` with the fixed effects
`ps_fes` (majors, `disced_6d`, by default), in pyfixest; with one fixed
effect and no `ps_xs` it is the group's share in each major. The cells, the
weights and what is left out are in `help(ug.gap_table)` and
`help(ug.ps_weights)`.

## The long frame

One row per person and job type (`pnr` x `type_sizerank_job`), with the columns
of `misc_docs/all_waves_long_variable_definitions.txt`: `T`, `vj_vij_job`,
`T_vj_vij_job`, the pre-treatment variables in `labels.PRETREATMENT`,
`educ_field`, `disced_6d`, and the outcomes in `labels.OUTCOMES`:
`spell_1_new_job` (1 for the job type of the first new job after the survey)
and `appl_job` (the number of applications to the job type).

## The persons frame

One row per person (`pnr`), with the columns of the long frame that take one
value per person: `T`, the pre-treatment variables and the education
columns. `treatment_shares` uses `T`, `educ_field` and `disced_6d`.

## The extra-long frame

One row per person and ordered pair of job types (`pnr` x
`type_sizerank_job` x `type_sizerank_jobj`), each pair in both orders; a
`d_` column is job j minus job j'. `gap_table` uses `d_Vj_Vij_rate`,
`d_vj_vij`, `d_v_vi_max_notmaxV_rate` (one value per person) and, to
reweight, `disced_6d`.

## Tests on the server data

The tests are installed with the package, and pytest with them. After prep,
run them from the server script:

```python
occu_choice_pf.run_tests({'long': long, 'persons': persons, 'elong': elong})
```

The data tests of a frame left out are skipped.

It logs pytest's output as it comes, so it shows on screen and is in the log
file of `setup_log`, and returns 0 if every test passes. The frame is written
to a temporary folder, deleted afterwards, and the tests run in a new
process. Further arguments go to pytest: `run_tests(frames, '-m', 'data')`
runs the data tests only. It skips the test that compiles the tables with
latexmk, since LaTeX on the server differs.

The data tests (`tests/test_regressions_data.py`, marked `data`) check what
the table assumes of its input but cannot check itself, and build both
default tables:

- one row per person and job type;
- the outcomes, coefficients and controls are numbers or Booleans;
- `T`, the pre-treatment controls and the education columns take one value
  per person;
- `T` is 0 or 1, and `T_vj_vij_job` is `T * vj_vij_job`, missing (null or
  NaN) where `vj_vij_job` is;
- the pre-treatment controls have no missing values, so that the columns of
  a panel rest on the same rows but for what the Poisson regression drops;
- the outcomes are at least 0;
- both default tables build and keep `T_vj_vij_job` and `vj_vij_job` in
  every column, and pyfixest's estimation sample, whose persons the
  Individuals row and `min_n` count, has the fit's *N* rows and carries the
  row numbers that `common_sample` and `reg_control_samples` read (a check
  on the server's pyfixest version).

`tests/test_treatment_shares_data.py` checks that `persons` has one row per
person, `T` and the education columns, and, with `long` given too, the same
persons as `long` with the same `T` and education (the exhibits read them
from `persons`, the regressions from `long`); and it builds the exhibits of
`treatment_shares`.

A failed data test shows the offending values. The other tests check the code
on fake data; they give the same result anywhere, so on the server they test
the environment.

## Layout

```text
src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       underestimation_gap.py, treatment_shares.py; imported from the top:
│                       from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames
```

`simulation/`, `examples/` and `tests/` are installed with the package. To find them:

```bash
python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>
```

Every example also runs in the VS Code Interactive window, whole or cell by
cell (`# %%`).

## Adding a module

1. `functions/<name>.py`: importable at once as `occu_choice_pf.<name>`;
   arguments as `{column: label}` dicts, with defaults in `labels.py`
2. `simulation/<name>.py`: `simulate()`, `save()`, `load()`
3. `examples/<name>/code/run_<name>.py`, plus `logs/.gitkeep`. Take the
   output folder from `root = example_root('<name>')`, and split sections
   with `# %%`. `tests/test_examples.py` runs every example.
4. `tests/test_<name>.py` for the code, and `tests/test_<name>_data.py`,
   marked `data`, for what the functions assume of real frames.

## Development

```bash
uv sync                                          # environment in .venv
uv run pytest -n auto                            # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames
```

`-n auto` runs the tests on all cores (pytest-xdist, in the dev group only).
The lower bounds in `pyproject.toml` are the server's versions for loguru,
numpy, polars and scipy (as for ad-data-pf). To test against exactly those,
in a separate environment so `uv.lock` is not rewritten:

```bash
cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest -n auto
cp "$TEMP/uv.lock" uv.lock
```

GitHub Actions does both on every push (`.github/workflows/test.yml`).

## Release

Bump `version` in `pyproject.toml`, run `uv lock`, and turn the
`## Unreleased` section of [CHANGELOG.md](CHANGELOG.md) into the version's
entry. For each change to a call, give the old and the new call, since the
server script is retyped by hand. Commit, then push a tag
`occu_choice_pf-v<version>`; GitHub Actions checks that the tag matches the
version, runs the tests, builds and publishes to PyPI (trusted publishing,
`.github/workflows/publish.yml`):

```bash
git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0
```

The same tag runs `.github/workflows/publish_1.yml` and `publish_2.yml`,
which publish the same code as `occu_choice_pf_1` and `occu_choice_pf_2`.
The DST server caches a package when it is installed and won't update that
name for 24 hours, so install an exact version under whichever name hasn't
been installed in the last 24 hours:

```bash
pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
pip install occu_choice_pf_2==X.Y.Z
```

All three install the import `occu_choice_pf`, and the most recent install is
the one used.

Before a name's first release, register it as a pending trusted publisher on
pypi.org (Account settings, Publishing): owner `AskerNC`, repository
`occu_choice_pf`, environment `pypi`, and workflow `publish.yml` for
`occu-choice-pf`, `publish_1.yml` for `occu_choice_pf_1`, `publish_2.yml` for
`occu_choice_pf_2`.
