Metadata-Version: 2.4
Name: virtuale-dl
Version: 0.1.0
Summary: Download materials from Virtuale (Moodle) courses using secure cookie handling
Author: g-mainardi
License-Expression: MIT
Project-URL: Homepage, https://github.com/g-mainardi/scraping-virtuale
Project-URL: Repository, https://github.com/g-mainardi/scraping-virtuale
Project-URL: Issues, https://github.com/g-mainardi/scraping-virtuale/issues
Keywords: moodle,virtuale,downloader,unibo
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: OS Independent
Classifier: Environment :: Console
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31.0
Requires-Dist: keyring>=24.0.0

# Virtuale Course Materials Downloader

This script downloads downloadable files from a course on Virtuale (Moodle) using your authenticated browser session cookie.

## What It Does

- Connects to a Virtuale course page
- Scans course resource/folder pages for downloadable files
- Downloads all discovered files to a target folder
- Overwrites existing files when the filename is the same

## Requirements

- Python 3.10+
- `requests` package
- `keyring` package (optional, strongly recommended for secure credential storage)

Install dependencies:

```bash
python -m pip install requests keyring
```

## Install For Other Users

### Option A: install directly from GitHub (fastest)

Recommended for end users that just want the command:

```bash
pipx install "git+https://github.com/g-mainardi/scraping-virtuale.git"
```

Or with plain pip:

```bash
python -m pip install "git+https://github.com/g-mainardi/scraping-virtuale.git"
```

Then verify:

```bash
virtuale-dl --help
```

### Option B: install from PyPI (after you publish)

```bash
pipx install virtuale-dl
# or
python -m pip install virtuale-dl
```

## Publish The Package (Maintainer)

### Option A: GitHub Actions (recommended)

This repository includes an automated workflow at:

- `.github/workflows/release.yml`

Behavior:

- Push tag `v*` (example: `v0.1.0`) -> build + publish to PyPI
- Manual run (`workflow_dispatch`) -> build + publish to TestPyPI

Required one-time setup:

1. Create package projects on TestPyPI and PyPI (`virtuale-dl`)
1. In each index, configure **Trusted Publisher** with owner `g-mainardi`, repository `scraping-virtuale`, and workflow `release.yml`
1. If you add environment restrictions in the publisher settings, create matching GitHub environments first

Release commands:

```bash
# Test release (manual from GitHub UI):
# Actions -> Release Python Package -> Run workflow

# Production release:
git tag v0.1.0
git push origin v0.1.0
```

### Option B: manual `twine` upload

One-time setup:

```bash
python -m pip install --upgrade build twine
python -m twine check --help
```

For each release:

```bash
# 1) bump version in pyproject.toml
# 2) build
python -m build

# 3) validate artifacts
python -m twine check dist/*

# 4) publish to TestPyPI first (recommended)
python -m twine upload --repository testpypi dist/*

# 5) publish to PyPI
python -m twine upload dist/*
```

After publishing, users can install with `pipx install virtuale-dl`.

## Security: Credential Storage

**IMPORTANT**: Never pass cookies on the command line to production systems or AI agents. This script uses a **secure fallback chain**:

1. **System Keyring** (Windows Credential Manager) - most secure, recommended ✅
2. `VIRTUALE_MOODLE_SESSION` environment variable
3. `--moodle-session` CLI argument (for special cases only)
4. Interactive prompt with `getpass` (no echo to terminal)

### Recommended Setup: Keyring (One-time)

Store your cookie securely using Python's keyring package:

```bash
# Get fresh cookie from browser (see "How to Get MoodleSession" below)
# Then store it securely:
python -m keyring set virtuale-moodle moodle YOUR_COOKIE_VALUE
```

This stores your cookie in Windows Credential Manager (encrypted, OS-managed).

**After one-time setup**, you can run:

```bash
python scraping_virtuale.py IRS
# No credential needed! Script retrieves it from keyring automatically.
```

**This is ideal for AI Agents** - they can invoke the tool without handling any secrets!

### Easier: Use the Helper Script

Instead of manually copying the cookie and running the keyring command, use the included helper:

```bash
./refresh-virtuale-cookie.cmd
```

This script will:

1. Guide you to DevTools & the Cookies tab
2. Ask for the cookie (hidden from terminal for security)
3. Store it securely in Windows Credential Manager
4. Clear sensitive data from memory

**First-time setup:**

```bash
./refresh-virtuale-cookie.cmd
# Follow the interactive prompts -> done!
```

**When cookie expires** (next day or after logout):

```bash
./refresh-virtuale-cookie.cmd
# Get fresh cookie -> done!
```

If automatic authentication fails (for example: `No MoodleSession found in keyring/env` or `The cookie is invalid or expired`), run `./refresh-virtuale-cookie.cmd` again and then retry `virtuale-dl`.

Then use the downloader normally:

```bash
virtuale-dl IRS
```

### Alternative setup using environment variable (less secure)

```bash
# Set environment variable (consider adding to PowerShell profile for persistence):
$env:VIRTUALE_MOODLE_SESSION = "YOUR_COOKIE_VALUE"
python scraping_virtuale.py IRS
```

## Script Files

- `scraping_virtuale.py` - main downloader
- `course_ids.json` - course code -> Moodle course ID mapping
- `refresh-virtuale-cookie.ps1` - local helper that stores MoodleSession in keyring
- `refresh-virtuale-cookie.cmd` - Windows launcher for the local PowerShell helper
- `sync-refresh-cookie-global.ps1` - copies local refresh scripts to your global bin folder

## Keep Global Script In Sync

If you edit the local repo scripts and also want your globally installed command updated, run:

```bash
powershell -NoProfile -ExecutionPolicy Bypass -File .\sync-refresh-cookie-global.ps1
```

Default source files:

- `./refresh-virtuale-cookie.ps1`
- `./refresh-virtuale-cookie.cmd`

Default global target folder:

- `%USERPROFILE%\\bin`

Example with custom destination:

```bash
powershell -NoProfile -ExecutionPolicy Bypass -File .\sync-refresh-cookie-global.ps1 -InstallDir "C:\\tools\\bin"
```

## CLI Usage

```bash
virtuale-dl SELECTED_COURSE [OUTPUT_DIR] [--moodle-session COOKIE] [--course-ids-file PATH] [--max-pages N] [--debug-scan]
```

### Positional Arguments

- `SELECTED_COURSE` (required): one of:
  - `IRS`, `ASMD`, `PM`, `SAP`, `SPE`, `DS`, `old-ASMD`, `old-PM`
- `OUTPUT_DIR` (optional): destination folder for downloaded files

If `OUTPUT_DIR` is omitted, the script uses automatic defaults:

- Standard courses: `<BASE_DIR>/<SELECTED_COURSE>/materials`
- Historical: `old-ASMD` -> `<BASE_DIR>/ASMD/old_materials`, `old-PM` -> `<BASE_DIR>/PM/old_materials`

`<BASE_DIR>` is resolved as:

1. `VIRTUALE_OUTPUT_BASE_DIR` environment variable (if set)
2. Otherwise, your user home directory

Missing folders are created automatically.

### Options

- `--moodle-session` (optional): MoodleSession cookie value. If omitted, script tries keyring → env var → prompt.
- `--course-ids-file` (optional): path to JSON file with course IDs (default: built-in course IDs, or `course_ids.json` if present next to script)
- `--max-pages` (optional, default: `300`): max number of resource/folder pages to crawl
- `--debug-scan` (optional): print crawler progress details

## Course IDs Configuration

The script reads course mappings from `course_ids.json`.

Example format:

```json
{
  "IRS": "72659",
  "ASMD": "71106",
  "old-ASMD": "59758"
}
```

Provide a different mapping with `--course-ids-file PATH`.

## How to Get `MoodleSession` Cookie

1. Log in to Virtuale in your browser
2. Open browser DevTools (F12)
3. Go to Storage/Application → Cookies → `https://virtuale.unibo.it`
4. Find `MoodleSession` cookie
5. Copy its value (long alphanumeric string)
6. Store it: `python -m keyring set virtuale-moodle moodle <VALUE>`

**Cookie expires after browser session/inactivity.** If you get auth errors, repeat the above steps to get a fresh one.

## Usage Examples

**After keyring setup** (recommended):

```bash
# Most basic - retrieves cookie from keyring automatically
virtuale-dl IRS
```

**With optional custom output:**

```bash
virtuale-dl PM ~/Downloads/PM_Materials
```

**With debug output:**

```bash
virtuale-dl SAP --max-pages 120 --debug-scan
```

**With environment variable** (if keyring not available):

```bash
$env:VIRTUALE_MOODLE_SESSION = "cookie_value"
python scraping_virtuale.py IRS
```

**With CLI argument** (not recommended, only for special cases):

```bash
python scraping_virtuale.py IRS --moodle-session "cookie_value"
```

## Typical Output

On success:

- `Authenticated for course ...`
- `Found N downloadable file link(s).`
- `Downloaded: ...`
- `Download completed. Saved N file(s) in: ...`

On invalid/expired cookie:

- `Error: The cookie is invalid or expired. You need a fresh one.`
- Fix: run `./refresh-virtuale-cookie.cmd` and then run `virtuale-dl IRS` again.

## Notes

- The script only follows Virtuale links
- It discovers files through resource and folder pages
- Non-download HTML pages are skipped during download
- If a file with the same name already exists, it is replaced.
