Metadata-Version: 2.4
Name: itw_python_builder
Version: 0.2.16
Summary: Standardized Django deployment pipeline with Docker, testing, and SonarQube integration
Author-email: IT-Works <contact@it-works.io>
License: MIT
Project-URL: Homepage, https://git.it-works.io/
Project-URL: Repository, https://git.it-works.io/
Project-URL: Issues, https://git.it-works.io/
Keywords: django,deployment,docker,ci-cd,sonarqube
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Build Tools
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: invoke>=2.0.0
Requires-Dist: pylint>=3.0.0
Requires-Dist: pylint-django>=2.5.0
Requires-Dist: python-decouple>=3.8
Requires-Dist: requests>=2.28.0
Dynamic: license-file

# ITW Python Builder

Standardized Django and Angular deployment pipeline with Docker, testing, SonarQube integration, automatic changelog generation, GitLab issue creation, and code quality enforcement.

## Features

- Automated deployment with semantic versioning
- Docker/Podman-based build and push pipeline (backend)
- Angular build, packaging and upload to the GitLab Package Registry (frontend)
- Angular SSR scaffolding and SSR builds
- Automated testing with coverage
- SonarQube static code analysis
- Pylint linting with SonarQube integration
- Quality gates - deploy only when tests pass
- Automatic changelog generation from commit trailers
- Automatic `CHANGELOG.md` creation and propagation to `staging`/`develop`
- GitLab issue creation from a markdown file
- Cached GitLab authentication
- Support for local and production pipelines
- Global CLI tool
- Automatic venv detection per project

## Installation
```bash
pip install itw-python-builder
```

Install globally (outside any project venv). The `itw` command becomes available system-wide.

Check the installed version:
```bash
itw --version
```

## Quick Start

1. Navigate to your project directory (Django projects must have a `.venv` or `venv`):

2. Authenticate with GitLab:
```bash
itw login
```

3. Initialize versioning — run it on `staging` first, then on `master`:
```bash
git checkout staging && itw tag-init   # creates v.0.0.1-rc1
git checkout master  && itw tag-init   # promotes the latest staging RC to -release
```

4. Run local pipeline:
```bash
itw pipelinelocal
```

5. Deploy to staging:
```bash
itw incrementrc
```

6. Deploy to production:
```bash
itw incrementpatch
```

## Available Commands

### Deployment Commands

- `itw incrementpatch` — Increment patch version and deploy
- `itw incrementminor` — Increment minor version and deploy
- `itw incrementmajor` — Increment major version and deploy
- `itw incrementrc` — Increment release candidate (staging)
- `itw release` — Promote RC to stable release (master)

### Timesheet Reporting for Frontend Releases

Pass `-tm` to any deployment command to report a merge request to Timesheet before publishing:

```bash
# Find a specific MR number merged today in the current GitLab group
itw incrementpatch -tm 110

# Choose from today's merged MRs in the terminal
itw incrementrc -tm
```

`--tm` and `--tm=110` are also supported. The option is available on `incrementrc`, `incrementpatch`, `incrementminor`, `incrementmajor` and `release`, for frontend projects on their allowed `staging`/`master` branches. Backend projects reject it before authentication or release work.

The current group is the full namespace of the origin repository; the search includes its subgroups and all accessible projects, including backend repositories. It uses the existing GitLab login and matches `merged_at` against today's calendar day in the local timezone. The number is the project-local MR number shown in GitLab, such as `!110`. If that number matches several repositories, you must choose one. Without a number, you must choose from the listed MRs even when only one is available. An interactive selection requires a terminal; a unique explicit number works without one. No matching MR stops the release with a message naming the number and group.

Configure Timesheet in the builder's root `.env`:

```dotenv
TIMESHEET_REPORT_URL=https://tm.it-works.io/api/qa-issue-from-merge-request/
QA_ISSUE_API_KEY=<shared-api-key>
```

Both values are required and are read only from the `.env` in the builder's root, resolved relative to the package location. The project being released and environment variables do not override this configuration.

After the tests and analysis pass, `itw` sends one POST to `TIMESHEET_REPORT_URL`, before creating a tag or changing/publishing the version. The request headers are `X-Api-Key` (the configured API key), `Version` (for example `1.2.3`, without the tag prefix or RC suffix), `Merge-Request-Repo` (the selected MR's `group/project`) and `Merge-Request-Id` (its MR number).

An HTTP error stops the release and displays the response's `message` (or `MESSAGE`); non-JSON error responses and connection/timeout failures also stop it with their error details. Reporting is not automatically retried. `--skip-pipeline` still reports to Timesheet, and SSR or multiple production environments report once per release. If a later build or upload fails, the existing Timesheet report is not rolled back. Commands without `-tm` do not search MRs or report to Timesheet.

### Authentication Commands

- `itw login` — Capture and cache a GitLab token (backend also logs into the container registry)
- `itw logout` — Forget the cached token and log out of the container registry

### GitLab Issue Commands

- `itw task-init` — Create a `TASK.md` template in the current directory
- `itw task --file=TASK.md` — Create GitLab issues from a markdown file

### Local Development Commands

- `itw pipelinelocal` — Run full local pipeline (lint → test → analyze → build)
- `itw lintlocal` — Run pylint with human-readable output
- `itw lint` — Run pylint and generate SonarQube report files
- `itw buildlocal` — Build Docker image locally
- `itw test` — Run tests with coverage
- `itw analyze` — Run SonarQube analysis
- `itw changelog` — Generate changelog manually
- `itw tag-init` — Initialize version tagging (see below)
- `itw ssr-init` — Scaffold Angular SSR (one-time, frontend only)

### Version Initialization

`itw tag-init` is branch-aware and must be run from `staging` or `master`:

- On **staging** it creates the first release candidate, `v.0.0.1-rc1`.
- On **master** it reads the latest `-rc` tag reachable from `staging` (local or `origin/staging`, after fetching tags) and creates that same version as a release. For example `v.0.0.2-rc10` on staging becomes `v.0.0.2-release` on master.

The `VERSION` file is written with the tag that was created. The command stops with a clear error when there is no staging branch, no release candidate to promote, or when the tag already exists.

### Skip Pipeline
```bash
itw incrementpatch --skip-pipeline
```

### Run a Subset of Tests
```bash
# Django: an app or a single test module
itw test --target=users
itw test --target=users.test_views

# Angular: a directory or a spec name
itw test --target=core
itw test --target=core.service
```

### SSR Builds

On a production frontend deploy (`master`), `itw` looks in `src/environments` for `*.prod.ts` files. One file keeps the existing `build-master` script and uploads `frontend/<version>/dist.tar.gz`. Several files prompt for which one to deploy, with `all` as the last choice. Each selected environment is built with the matching `angular.json` configuration, packaged as `dist-<environment>.tar.gz`, and uploaded before the next one starts, at `frontend/<version>/dist-<environment>.tar.gz`. The version commit then includes that name, for example `version update to v.1.2.3-release (lenslight)`. Choosing `all` lists every name: `version update to v.1.2.3-release (greenterrace, lenslight)`.

Frontend projects scaffolded with `itw ssr-init` build through the SSR npm scripts when `--ssr` is passed:
```bash
itw incrementrc --ssr
itw incrementpatch --ssr
```

## Virtual Environment Detection

`itw` is installed globally but automatically detects the project's virtual environment (`.venv` or `venv`) in the current directory. Tasks that need Python/Django dependencies (test, lint, analyze) activate the venv automatically — no need to manually activate it.

If no venv is found, those tasks will error with a clear message. Tasks that only use git (like `changelog`, `tag-init`, `buildlocal`) work without a venv.

## Linting

The pipeline runs pylint automatically on every deployment and generates report files for SonarQube. The `.pylintrc` configuration is shipped with the package — no config files needed in your project.
```bash
# Review issues with human-readable output
itw lintlocal

# Generate report files for SonarQube
itw lint

# Use a custom pylint configuration
itw lint --pylintrc=.pylintrc
```

Add this to your `sonar-project.properties`:
```properties
sonar.python.pylint.reportPaths=pylint-report.txt
```

## Changelog Generation

Changelog entries are generated automatically on every deployment based on commit messages. To categorize a commit, add a `Changelog:` trailer to the commit body:
```
add user authentication

Changelog: added
```

Available categories: `added`, `fixed`, `changed`, `deprecated`, `removed`, `security`, `performance`, `tests`, `docs`, `refactor`

Commits without a trailer appear under `Other Changes`. The `CHANGELOG.md` file is committed and pushed automatically with each deployment.

If `CHANGELOG.md` does not exist in the current directory it is created automatically — including when there are no new commits since the last tag, so the deployment never fails on a missing file.

### Changelog Propagation

After a successful deployment the changelog is synced forward through a temporary git worktree:

- from `master` → `staging` and `develop`
- from `staging` → `develop`

Propagation is skipped when the working tree is not clean or when there is nothing new to sync.

## GitLab Issue Creation

Create GitLab issues from a markdown file instead of the web UI.

Generate the template:
```bash
itw task-init
```

Then fill it in and run:
```bash
itw task --file=TASK.md
```

`task-init` refuses to overwrite an existing `TASK.md`. Without `.git` in the current directory, repo and milestone stay empty. With `.git`, it reads the GitLab remote and finds the namespace's `_pm` repositories. A single repository is selected automatically; several prompt for a selection. Milestones from the selected project and its parent groups follow the same selection rule, and the chosen milestone's available start/end dates are filled in. When no milestones exist, the milestone stays empty and the date placeholders are kept.

### Template

```markdown
/glab_host 
/glab_username 
/glab_token 
/repo 
/milestone 
/milestone-start YYYY-MM-DD
/milestone-end YYYY-MM-DD
/assignee 
## Comment

## Comment
```

Add a `/title <text>` line for each issue before running `itw task`. The parser also supports the optional directives listed below, including `/label`, `/estimate`, `/due` and custom acceptance criteria. Leaving a key empty is allowed — empty keys are simply not sent to GitLab. The date placeholders use `YYYY-MM-DD` and are treated as empty until you replace them.

Any text outside the directives becomes the issue description. Create several issues from one file by separating the blocks with a line containing only `===`.

### Directives

| Directive | Description |
| --- | --- |
| `/glab_host` | GitLab host to authenticate against |
| `/glab_username` | GitLab username |
| `/glab_token` | GitLab personal access token |
| `/title` | Issue title (required) |
| `/repo` | Target project path |
| `/milestone` | Milestone title |
| `/milestone-start` | Milestone start date, used only when the milestone is created |
| `/milestone-end` | Milestone due date, used only when the milestone is created |
| `/assignee` | Username to assign the issue to |
| `/label` | Comma-separated labels |
| `/estimate` | Time estimate (e.g. `3h`) |
| `/due` | Issue due date |

### Authentication

Either provide all three of `/glab_host`, `/glab_username` and `/glab_token`, or none of them. Providing only some of them is an error.

When no credentials are given, `itw` uses the `.git` directory in the current folder to detect the GitLab host and project, and the token cached by `itw login`.

### Target Repository

The first `/repo` in the file is the file default. Every later block that leaves `/repo` empty uses that value. A block that sets its own `/repo` keeps it and does not change the default for the blocks below. Blocks above the first `/repo`, and a file with no `/repo` at all, are resolved once by walking the current repository's namespace upward looking for a `_pm` project. If several are found you are prompted to pick one, and that choice is reused for the remaining issues that still have no repo.

With `/repo`, leading and trailing slashes are stripped automatically, and full URLs or SSH remotes are accepted:

```
/repo /tools/itw_python_builder/          → tools/itw_python_builder
/repo https://git.it-works.io/tools/x.git → tools/x
/repo git@git.it-works.io:tools/x.git     → tools/x
```

### Milestones, Labels and Duplicates

Milestones, labels and issue titles are matched with a shared fuzzy comparison at 95% similarity, so small differences in spacing or casing still count as a match. Digit runs must be identical (`java 38` does not match `java 37`). The threshold is a single constant (`SIMILARITY_THRESHOLD` in `task_utils.py`) if you need to tune it.

- **Milestone** — an existing project or parent-group milestone with the same title is reused; otherwise it is created (with the start/end dates when both are given).
- **Label** — a similar existing label is linked and logged in white; a new one is created and logged in green. An empty `/label` is never sent to GitLab.
- **Duplicate issues** — before creating, the target project is searched by title. A similar title is treated as a duplicate and skipped only when the milestone and assignees also match. An empty milestone or assignee matches only an issue where that field is also empty.

### Acceptance Criteria and Comments

The `## Acceptance Criteria` section must be opened and closed with the same header. When omitted or left with empty checkboxes, a default checklist is applied.

The `## Comment` section, opened and closed the same way, is posted as a note on the issue after it is created.

## Requirements

- Python 3.10+
- Podman
- Git
- SonarQube server
- Node.js and npm (frontend projects)

## Configuration

### Environment Variables

Add to your project `.env`:
```
SONAR_HOST_URL=https://your-sonar-server
SONAR_TOKEN=your-token
GIT_DEPTH=0
```

### Project Files Required

- Git repository with `develop`, `staging`, and `master` branches
- `VERSION` file in project root (backend) or `package.json` (frontend)
- `Dockerfile` (backend)
- `sonar-project.properties`

## License

MIT
