Metadata-Version: 2.4
Name: pandaspro
Version: 1.12.2
Summary: Upgraded pandas package for easier dataframe operations
License: MIT
License-File: LICENSE
Author: Shiyao Wang
Author-email: sw1016@georgetown.edu
Requires-Python: >=3.8,<3.14
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: PyPDF2 (>=3.0.1)
Requires-Dist: jinja2 (>=3.1.4)
Requires-Dist: numpy (>=1.26.0)
Requires-Dist: openpyxl (>=3.1.5)
Requires-Dist: pandas (>=2.2.1)
Requires-Dist: python-dateutil (>=2.8.2)
Requires-Dist: pyyaml (>=6.0)
Requires-Dist: tabulate (>=0.9.0)
Requires-Dist: termcolor (>=2.4.0)
Requires-Dist: xlwings (>=0.31.10)
Project-URL: Homepage, https://github.com/Unicell-Organization/pandaspro
Description-Content-Type: text/markdown

# pandaspro

pandas with friendlier dataframe methods and Excel export / formatting helpers.

```
pip install pandaspro
```

See `CHANGELOG.md` for what changed in each version, and `df.cpdhelp()` for the dataframe shortcuts.

## Data assembly: cpdBaseFrame and cpdDatabaseC

The two halves of assembling a project's data:

- **cpdBaseFrame** — dated system extracts. One folder of files named by date (`SOB_20261001.csv`);
  decorate a class and load a version with `X(version="latest")`. The data is produced elsewhere
  and only read here.
- **cpdDatabaseC** — hand-maintained lists. One folder of JSON files named by list
  (`modules/clients.json`) plus a browser editor; decorate a store class and one class per list.
  The data is maintained by people in the published page.

### cpdDatabaseC

```python
from mylists import MyLists, Clients

MyLists.publish()          # creates <folder>/mylists.html and modules/*.json (data is never overwritten)
MyLists.open()             # opens the editor in Edge / Chrome; "Open folder" -> pick <folder>

Clients.frame()            # rows as a DataFrame
MyLists.list()             # every module: id, title, group, live, rows, revision, saved_at
MyLists.check()            # one line per module: filled? live? same as its old source?

CountryGroups.import_source()   # rows from the old Excel into the module (not live yet)
CountryGroups.compare()         # empty frame = module and Excel agree
CountryGroups.set_live()        # from now on read_live() returns the module
```

`mylists` is the example shipped in `pandaspro/cpddatabasec/example/`: copy `mylists/` into your
project, rename the store, keep or replace the three example lists, add your own. Declaring a
store and a list:

```python
from pandaspro import cpdDatabaseC, cpdField as field

@cpdDatabaseC(name="MyLists", package="myproject", editor="mylists.html")
class MyLists:
    @classmethod
    def get_path(cls):                       # or path="..." in the decorator
        return os.environ["MYLISTS_DIR"]     # per machine, never a hard-coded path in the repo

@MyLists.module("clients", title="Clients", group="Reference", key=["code"])
class Clients:
    code = field(required=True, help="Short code, unique")
    name = field(required=True)
    tier = field(type="select", options=["A", "B", "C"])
```

Add a list = add a decorated class. Add a field = add a line (`publish()` refreshes the
definition, rows stay). Change the page = pass `editor_source="your.html"` (start from a copy of
`pandaspro/cpddatabasec/editor.html`).

## Word reports: WordReport

Take Word reports apart into sections, paragraphs and tables, check that nothing was lost, put
several together and write one `.docx` with a consistent layout. Standard library only.

```python
import pandaspro as cpd

a = cpd.WordReport.read("part_a.docx")
b = cpd.WordReport.read("part_b.docx")

a.preview()                 # numbered outline: S1 sections, P3 paragraphs, T2 tables
a.preview_html()            # rendered page in the browser, with the check on top
print(a.check())            # characters, digits, every number, footnotes, tables vs the source

table = a.tables[0]
table.title, table.header_rows, table.notes, table.footnotes
table.frame                 # DataFrame; table.to_frame(numeric=True) parses the numbers

merged = cpd.WordReport.merge(a, b, titles=["Part A", "Part B"])
merged.to_docx("out/combined.docx")   # written, read back and compared; raises on any difference
```

What is kept: the text, bold / italic / underline / strike, colour, highlight, super and
subscript, links, footnotes and endnotes, bullets and numbering, merged cells, header rows, cell
shading, pictures and text boxes. Fonts and font sizes are dropped on purpose: the output uses one
layout. Charts and diagrams cannot be carried; a red placeholder marks their position and
`report.warnings` lists them. The table of contents, page headers / footers and comments are not read.

Table titles ("Table 1 ...", the Caption style) and notes ("Note:", "Source:", "a/ ...") are found
by rule. When a rule guesses wrong, pass `title_pattern=` / `note_pattern=` to `read()`, or move
the paragraphs yourself: `section.blocks`, `table.title` and `table.notes` are ordinary lists.

`report.to_dict()` / `to_json()` / `WordReport.from_json()` expose the whole structure.

Some Word files lay a whole page out as one table: a row of running text, empty spacer rows,
a "Table 1 ..." row, the real header and data rows, another title row. `read()` takes such
layout tables apart into paragraphs and separate tables and drops the empty rows and columns
(`unpack_layout=False` keeps them as they are; `report.unpacked` says how many were found).

### Merging

```python
merged = cpd.WordReport.merge(a, b, tables_last=True, tables_heading="Tables", number_tables=True)
```

`tables_last` puts the text of all reports first (headings and paragraphs, in order) and all
their tables after it. `number_tables` gives the table titles "Table 1.", "Table 2." ... in that
order: a number a title already has ("Table A1:") is replaced, a table without a title gets
"Table N"; nothing else in the titles changes. `report.number_tables(prefix="Table", sep=".")`
does the same on any report.

### Reviewing before writing

```python
merged.review_html()                     # opens a page: keep / drop, rename, title and subtitle
merged = merged.apply_review(r'''{...}''')   # the line the page's "Copy result" button gives
```

Every section, paragraph and table on the page has a checkbox; headings and table titles are
editable; a title and subtitle can be typed at the top. Cells and numbers cannot be changed
there. `apply_review()` takes the JSON as a dict, a string or the path of the downloaded
`review.json` and returns a new report; `check()` still passes, with the dropped text counted.

### Writing into a template

`to_docx(template=...)` writes into your own blank `.docx` / `.dotx`: its page header and footer,
page size and margins, theme and styles are kept, and the report goes under whatever its body
holds (a title line, empty lines that leave room for a banner). `keep_template_body=False` starts
from an empty body. `fill={"XXX": "Region A", "MM DD, YYYY": "September 30, 2026"}` replaces
placeholders in the template body, page headers and footers, also when Word split them over
several runs, and removes their highlight. A placeholder the template does not have gives a warning
(`fill_missing="error"` raises, `"ignore"` stays quiet). `WordReport.inspect_template(path)` lists
the template's style names, its highlighted placeholders and its body text. Headings,
captions and tables take the template's styles of the same name ("heading 1", "Table Grid" ...).
`styles=` points each part at another style or changes it:

```python
merged.to_docx(
    "out/combined.docx",
    template="blank_template.docx",
    font="Arial",                      # body font; left out, the template's stays
    styles={
        "heading1": "Report Heading",  # a style name in the template
        "heading2": {"font": "Georgia", "size": 13, "color": "1F4E79", "space_before": 18},
        "body": {"east_asia_font": "SimSun", "line_spacing": 1.15},
        "table": {"style": "Grid Table 4", "header_fill": "DDEBF7", "borders": "horizontal"},
        "table_header": {"bold": True},
    },
)
```

Roles: `body`, `heading1` … `heading9`, `title`, `subtitle`, `list`, `quote`, `caption`,
`table_title`, `table_text`, `table_header`, `table_note`, `footnote`, `endnote`, `table`.
Paragraph options: `font`, `east_asia_font`, `size`, `bold`, `italic`, `underline`, `color`,
`all_caps`, `small_caps`, `letter_spacing` (points, negative is narrower), `align`, `space_before` / `space_after` (points), `line_spacing`,
`indent` / `first_line` (inches), `keep_with_next`, `page_break_before`, and `style` to start from a
named style. Table options: `style`, `borders` (`grid`, `horizontal`, `outside`, `none`),
`border_color`, `border_size`, `header_fill` (a colour, or `none` for no shading), `header_color`
(text in header rows), `header_height` / `row_height` (inches), `align` (the table on the page),
`header_align` / `text_align` / `first_column_align` (the cells: header rows, the rest, the first
column in every row), `even_columns` (`rest`: every column but the first gets the same width;
`all`), `first_column`, `banded_rows`, `keep_together` (default on). An option
that sets `bold`, `italic` or `underline` on a role also overrides what the source runs carried. A name the
template does not have raises an error that lists the names it does have. `page=`, `landscape=`
and `margin=` override the template's page setup only when given.

A table stays on one page (it moves to the next page whole when it does not fit; no page break
is inserted). `blank_lines=True` sets paragraphs apart with an empty line (none under a heading,
inside a list or after a table); `margin_top="auto"` starts the text under the page header's
banner on every page, with `header_gap` inches of air under it (0.1 by default); `new_page=r"^annex"`
makes the headings that match begin a new page; `number_tables=True` numbers the table titles.
`preset=` takes these five and `styles` together as one dict, with `styles=` laid over it; the
role `headings` stands for `heading1` … `heading9`.
`title=` / `subtitle=` add those two lines above the report:

```python
merged.to_docx("out/brief.docx", template="blank_template.docx", preset={...},
               title="Regional Brief", subtitle="As of September 30, 2026",
               fill={"XXX": "Region A", "MM DD, YYYY": "September 30, 2026"})
```

### Profiles: a project's rules in one file

Nothing project-specific is built in. What a project knows about its own reports — which lines
are headings, what to leave out, the merge order, the fonts and table look — goes in one YAML
(or JSON) file, kept next to the project, and is passed as `profile=`:

```python
profile = cpd.WordReport.profile("project.yaml")
a = cpd.WordReport.read("Note on Region A.docx", profile=profile)   # the matching document's rules
b = cpd.WordReport.read("Outcomes note.docx", profile=profile)
merged = cpd.WordReport.merge(a, b, profile=profile)                # order, headings, tables last
merged.review_html()                                                # optional, as above
merged.to_docx("out/brief.docx", profile=profile, fill={"XXX": "Region A"})
report = profile.combine(["Note on Region A.docx", "Outcomes note.docx"])   # read + merge in one go
```

```yaml
name: Regional note
documents:                        # one entry per kind of source file
  first:
    name: First Team
    match: '^Note on\b'           # tried on the file name, then on the first lines of text
    heading: ~                    # a text here puts the whole document under that heading
    drop: ['^Summary of\b']       # paragraphs to leave out (counted as read by check())
    headings:
      - {italic_line: true, max_chars: 100, level: 1}   # a line in italics throughout
      - {match: '^ANNEX\s*:?\s*$', text: Annex, level: 1}   # rewritten as "Annex"
    table_titles: ['^(Headcount|Offices)\s*\(FY\d{2}']  # the paragraph above a table
  second:
    name: Second Team
    match: '^Outcomes\b'
    drop: ['^Confidential\b']
    headings:
      - {match: '^Outcomes and Promotion Note\b', text: Promotion, level: 1}
merge:
  order: [first, second]
  tables_last: true
  tables_heading: Annex           # a source's own emptied "Annex" section is reused
  number_tables: true
write:                            # everything to_docx takes: styles, blank_lines, margin_top,
  blank_lines: true               # new_page, number_tables, template (relative to this file),
  margin_top: auto                # font, size, page, landscape, margin, table_size, note_size,
  new_page: '^\s*(annex|appendix)\b'   # placeholders, keep_template_body, fill_missing
  number_tables: true
  styles:
    body:     {space_before: 0, space_after: 0, line_spacing: 1.0}
    title:    {font: Open Sans, size: 14, bold: true, align: center, space_after: 8}
    headings: {font: Open Sans, size: 11, align: left}
    heading1: {italic: true}
    table:    {header_fill: '156082', header_color: 'FFFFFF', header_height: 0.5}
```

`read()` picks the document whose `match` fits the file name or one of its first three lines
(`document="first"` names it outright); `report.document_name` says which one was used. The rules
turn paragraphs into headings (the heading style decides their look), leave out the ones in
`drop`, and add the document's own table title and note patterns to the defaults. `merge()` puts
the reports in `order`, under the documents' `heading` when one is given, and fills in
`tables_last`, `tables_heading` and `number_tables`; what is passed to `merge()` wins. `to_docx()`
takes the `write` section as its defaults, again under what is passed. Patterns are regular
expressions searched in the text; `match`, `drop` and `headings` are case-sensitive, table titles
and notes are not. A key the profile does not know raises an error that lists the known ones.

Per table, by its title: `tables: [{title: '^Staff\b', corner: Staff, shade: none}]` sets the
text of the top-left cell and which body rows are shaded (`shade: none`, or patterns matched on
the first cell of a row: those rows are shaded whole, the others not; `shade_fill` picks the
colour, else the one the table used).

Several files from one merged report go in `outputs:`; `profile.write_outputs(report, folder,
stem, template=..., title=...)` writes them all (`report.tables_only()` is what `content: tables`
uses):

```yaml
outputs:
  internal:
    file: '{stem} - Internal.docx'
    content: all
    stamp: {find: Strictly Confidential, text: Internal Use Only, fill: FFC000}
  tables:
    file: '{stem} - Tables.docx'
    content: tables                 # nothing but the tables, with their titles and notes
    subtitle: Tables
```

`stamp` (also `to_docx(stamp=...)`) changes a label of the template: the text, whatever its case
and however Word split it, and the background of the text box or shape that holds it (a plain
label gets the colour behind its runs). Either `text` or `fill` may be left out.

### The page

```python
cpd.WordReport.app()                 # in a notebook opened in the project folder
```

opens a local page that does the whole job without code: **Quick read** finds the profile, the
template and the source files in the folder (files are sorted to their documents by name), or
drop a profile and the files on the page; **Read** shows each document's outline and check and
the merged one; **Review** is the review page, inline; **Generate** writes the outputs and opens
them. The page talks to a small server on 127.0.0.1 inside the Python process, so the kernel has
to stay alive; nothing leaves the machine, dropped files are copied to a temporary folder.
`app(profile=..., folder=..., block=True)` for a script; the returned object has `.url` and `.stop()`.

## Versions and releases

All work happens on `master`. A version is a tag on `master` (`v1.3.1`); there are no version
branches. The version number lives in one place, `pyproject.toml`.

- While you work: add a line under **Unreleased** in `CHANGELOG.md` for anything a user can notice.
- To release: `python release.py 1.3.2` (add `--dry-run` to rehearse). It checks the tree, runs
  the tests, sets the version, dates the changelog section, commits, tags and pushes.
  The pushed tag makes GitHub Actions build the package and publish it to PyPI.
- Every push to `master` runs the tests on GitHub.
- A machine that produces reports should sit on a tag, not on every push:
  `git fetch --tags` then `git checkout v1.3.2`.
- Version numbers: last digit for fixes and small additions, middle digit for new features, first
  digit when existing calls stop working.

