Metadata-Version: 2.5
Name: tutor-contrib-panorama
Version: 22.0.2
Summary: Tutor plugin for Panorama Analytics
Project-URL: Code, https://github.com/aulasneo/tutor-contrib-panorama
Project-URL: Issues, https://github.com/aulasneo/tutor-contrib-panorama/issues
Author-email: Aulasneo <andres@aulasneo.com>
License: AGPLv3
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: GNU Affero General Public License v3
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10
Requires-Dist: tutor-mfe<23.0.0,>=22.0.0
Requires-Dist: tutor<23.0.0,>=22.0.0
Provides-Extra: dev
Requires-Dist: black; extra == 'dev'
Requires-Dist: build; extra == 'dev'
Requires-Dist: isort; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pylint; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Description-Content-Type: text/markdown

# Panorama Tutor Plugin

## Introduction

[Panorama](https://www.aulasneo.com/panorama-analytics/) is the analytics
solution developed by [Aulasneo](https://www.aulasneo.com) for Open edX. It is
a complete stack that includes data extraction, load, transformation,
visualization and analysis. The data extracted is used to build a datalake
that can easily combine multiple LMS installations and even other sources of
data.

This utility is in charge of connecting to the MySQL and MongoDB tables and
extracting the most relevant tables. Then it uploads the data to the datalake
and updates all tables and partitions.

[![Linting](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/pylint-dev/pylint)

## Installation

The Tutor 22 / Verawood package is currently a prerelease. The commands below
opt into the 22.x prerelease series; use them after the package is published.
For an unpublished checkout, install with `pip install -e .` instead.

1. Install as a Tutor plugin:

   ```text
   pip install --pre 'tutor-contrib-panorama>=22,<23'
   ```

2. Enable the plugin:

   ```text
   tutor plugins enable panorama
   ```

## Panorama Modes

Starting with version 16.3.0, the Tutor Panorama Plugin now offers four modes
to use Panorama:

- `DEMO`: Full access to the standard Panorama service, with anonymized data.
- `FREE`: Hosted Panorama service for free, with limited functionalities.
- `SAAS`: Hosted Panorama service provided by Aulasneo with most typical
  features enabled.
- `CUSTOM`: Full Panorama flexibility, either in SaaS modality or self-hosted.

Since 16.3.0, the default mode of Panorama is `DEMO`.

### Panorama DEMO Mode

In DEMO mode you can try the functionality of Panorama with anonymized data.
You will be able to experiment with the power of Panorama as you would with
your own data. What you will see is the actual Panorama SaaS solution from our
production servers, showing the dashboards offered out of the box for the
`SAAS` mode.

In `DEMO` mode, Panorama will not extract any data from your server.

To activate `DEMO` mode, install the plugin, rebuild the `openedx` and `mfe`
images, and restart your deployment. No specific configuration is needed.

```text
pip install --pre 'tutor-contrib-panorama>=22,<23'
tutor plugins enable panorama
tutor images build openedx
tutor images build mfe
tutor {local|k8s} restart
```

### Panorama FREE Mode

Panorama `FREE` mode offers a basic yet powerful set of dashboards that you can
use for free. It is part of the Aulasneo SaaS offering. To get your `FREE`
credentials, please register at
[Panorama](https://www.aulasneo.com/panorama-analytics/) and send us an email
to <info@aulasneo.com>.

In `FREE` mode, only the relational and courseware data is extracted. No logs
are processed. Therefore you will not be able to get statistics based on
events, such as video views, forum activity, or PDF downloads.

The free mode is part of the SaaS offering. Please be aware that data from your
instance will be uploaded to our servers.

To activate `FREE` mode, install the plugin, rebuild the `openedx` and `mfe`
images, and restart your deployment. No specific configuration is needed.
Contact us at <info@aulasneo.com> to get the additional settings needed to
activate Panorama.

```text
pip install --pre 'tutor-contrib-panorama>=22,<23'
tutor plugins enable panorama
tutor images build openedx
tutor images build mfe
tutor {local|k8s} restart
```

### Panorama SaaS Mode

Panorama `SAAS` mode offers a full set of dashboards that you can use out of
the box. This is a paid service offered by Aulasneo to any Open edX user.

Please be aware that data from your instance will be uploaded to our servers.

To connect to Panorama SaaS, please contact us at <info@aulasneo.com> to get
instructions.

```text
pip install --pre 'tutor-contrib-panorama>=22,<23'
tutor plugins enable panorama
tutor images build openedx
tutor images build mfe
tutor {local|k8s} restart
```

### Panorama Custom Mode

The Panorama `CUSTOM` mode offers the highest flexibility to use Panorama. To
set up custom mode, you will have to deploy your own data infrastructure.

#### Setting Up the Datalake

The Panorama plugin for Tutor is configured to work with an AWS datalake.

To set up your AWS datalake, you will need to:

- Create or use an IAM user or role with permissions to access the S3 buckets,
  KMS if encrypted, Glue, and Athena.
- Create one S3 bucket to store the data, one for raw logs (optional), and
  another as the Athena query results location.
- Use encrypted buckets and strict access policies to prevent unauthorized
  access.
- Create the Panorama database in Athena with `CREATE DATABASE panorama`.
- Create the Athena workgroup `panorama` to keep the queries isolated from
  other projects.
- Set the query result location to the bucket created for this workgroup.

##### User Permissions to Work With the AWS Datalake

In order to work with an AWS datalake, you will need to create a user, for
example `panorama-elt`, and assign a policy, for example `PanoramaELT`, with at
least the following permissions.

Replace `<panorama_data_bucket>`, `<panorama_logs_bucket>`,
`<panorama_athena_bucket>`, `<region>`, and `<account_id>` with proper values.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": [
        "arn:aws:s3:::<panorama_data_bucket>/openedx/*",
        "arn:aws:s3:::<panorama_logs_bucket>/tracking_logs/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::<panorama_data_bucket>/PanoramaConnectionTest"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<panorama_athena_bucket>",
        "arn:aws:s3:::<panorama_athena_bucket>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:BatchCreatePartition",
        "glue:GetDatabase",
        "athena:StartQueryExecution",
        "glue:CreateTable",
        "athena:GetQueryExecution",
        "athena:GetQueryResults",
        "glue:GetDatabases",
        "glue:GetTable",
        "glue:DeleteTable",
        "glue:GetPartitions",
        "glue:UpdateTable"
      ],
      "Resource": [
        "arn:aws:athena:<region>:<account_id>:workgroup/panorama",
        "arn:aws:glue:<region>:<account_id>:database/panorama",
        "arn:aws:glue:<region>:<account_id>:catalog",
        "arn:aws:glue:<region>:<account_id>:table/panorama/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "*"
    }
  ]
}
```

If you have encrypted S3 buckets with KMS, you may need to add permissions to
get the KMS keys.

Additionally, the user must have Lake Formation permissions to access the data
locations and query the database and all tables.

Finally, you will have to connect QuickSight to Athena to visualize the data.

## Configuration

Tutor 22 / Verawood implementation and validation details are in
[docs/verawood-validation.md](docs/verawood-validation.md). Panorama remains a
standalone MFE; the plugin contributes native frontend-base navigation as well.
Before testing new images, set
`PANORAMA_OPENEDX_BACKEND_VERSION` to the PyPI package version (default `22.0.0`), and
`PANORAMA_MFE_REPO` / `PANORAMA_MFE_VERSION`, and `PANORAMA_ELT_REPO` /
`PANORAMA_ELT_VERSION` to your reviewed source references. The MFE defaults to
`release/verawood/v20260926`; the extractor still defaults to `v1.0.1`.
Confirm the selected backend package and source references contain the changes
being validated before building.
For standalone MFE builds Tutor also supports a mounted source build context.

`PANORAMA_MFE_ENABLED=False` disables the MFE, navigation, backend install/settings
and migrations without disabling separately configured extraction. Backend init
skips DEMO mode and otherwise only applies committed migrations. Backend
installation uses the target platform's
requirements as constraints and runs `pip check`. The install step explicitly
bind-mounts the constraints from Tutor's `edx-platform` build stage because
the mount used by Tutor's earlier install step does not persist.

Recurring jobs forbid overlap, retain one successful/two failed jobs and default to
one retry, a 3300-second active deadline and a 300-second scheduling deadline.
Adjust `PANORAMA_K8S_JOB_BACKOFF_LIMIT`, `PANORAMA_K8S_JOB_ACTIVE_DEADLINE_SECONDS`
and `PANORAMA_K8S_CRON_STARTING_DEADLINE_SECONDS` for the configured schedule.
The one-off initialization Job retains Kubernetes default retries and has no
active deadline. Both jobs use `PANORAMA_K8S_JOB_MEMORY_REQUEST` and `PANORAMA_K8S_JOB_MEMORY_LIMIT`.

Fluent Bit uses a digest-pinned 3.4.15 image (Fluent Bit 5.0.9). Enable
`PANORAMA_RUN_K8S_FLUENTBIT` in exactly one SAAS/CUSTOM site per cluster and disable
it in every other site. That site's DaemonSet reads `lms*.log` across all
namespaces, excluding `lms-worker*.log`, using the collector site's bucket/credentials.
This selects LMS pod log files, including matching sidecars; it does not filter
by container name.
RBAC names and host state are scoped to the hosting installation, not the source
sites. Offsets and S3 buffers persist under
`/var/lib/panorama-fluentbit/<collector-namespace>` on each Kubernetes node;
Compose uses Tutor's `data/panorama-fluentbit` directory. Node/disk loss is not
covered by this persistence. On Linux, Docker creates the Compose state directory
and the root-running collector writes state as root; cleanup can require `sudo`.
If running the collector as a different user, provision this directory with
ownership writable by that container UID before starting Compose.
Default S3 spool capacity is 1G (`PANORAMA_FLB_STORE_LIMIT`),
tail memory 32M (`PANORAMA_FLB_TAIL_MEMORY_LIMIT`) and line limit 1M
(`PANORAMA_FLB_LINE_MAX_SIZE`). Oversized physical lines are skipped with a warning
so subsequent records remain readable; spool exhaustion can lose data. Monitor
object arrival and disk/backlog growth rather than relying on output-success counters.
Kubernetes keys retain `/tracking_logs/<source-namespace>/...`; Compose keys retain their
existing LMS-host paths. `PANORAMA_LOGS_UPLOAD_CHUNK_SIZE` is retained as a deprecated
configuration key but is unused with PutObject. Drain existing buffers before switching
collectors and account for the new offset path when checking duplicates at cutover.

When upgrading a Kubernetes collector from the legacy global RBAC names, applying
the new manifest does not remove the old objects. Complete this migration once
per cluster after the replacement collector is running:

1. Inspect `kubectl get clusterrolebinding pod-log-crb -o yaml` and
   `kubectl get clusterrole pod-log-reader -o yaml`. Confirm these are the legacy
   Panorama resources and identify every service account relying on them.
2. Verify the new `panorama-<collector-namespace>-pod-log-crb` binding references
   `panorama-<collector-namespace>-pod-log-reader` and the active collector's
   `fluent-bit` service account. Confirm the collector can retrieve Kubernetes
   metadata. Migrate any other consumers before removing their old grants.
3. Remove the confirmed legacy binding, then the unused legacy role:

   ```sh
   kubectl delete clusterrolebinding pod-log-crb --ignore-not-found
   kubectl delete clusterrole pod-log-reader --ignore-not-found
   ```

Do not delete a role still referenced by another binding. These are cluster-wide
resources; disabling a site's collector or applying the new manifest alone does
not revoke a leftover legacy binding.

Set the following variables to configure Panorama:

| Variable | Default | Description |
| --- | --- | --- |
| `PANORAMA_BUCKET` |  | S3 bucket to store the raw data |
| `PANORAMA_MODE` | `DEMO` | Panorama mode: `DEMO`, `FREE`, `SAAS`, `CUSTOM` |
| `PANORAMA_MFE_ENABLED` | `True` | Enable the Panorama MFE |
| `PANORAMA_DEFAULT_USER_ARN` | `arn:aws:quicksight:{{ PANORAMA_REGION }}:{{ PANORAMA_AWS_ACCOUNT_ID }}:user/default/{{ LMS_HOST }}` | QuickSight user to map by default |
| `PANORAMA_ENABLE_STUDENT_VIEW` | `True` | Allow students to access the student's panel |
| `PANORAMA_MFE_PORT` | `2100` | Internal port of the Panorama MFE |
| `PANORAMA_RAW_LOGS_BUCKET` | `PANORAMA_BUCKET` | S3 bucket to store the tracking logs |
| `PANORAMA_CRONTAB` | `55 * * * *` | Crontab entry to update the datasets |
| `PANORAMA_BASE_PREFIX` | `openedx` | Directory inside the `PANORAMA_BUCKET` to store the raw data |
| `PANORAMA_REGION` | `us-east-1` | AWS default region |
| `PANORAMA_DATALAKE_DATABASE` | `panorama` | Name of the AWS Athena database |
| `PANORAMA_DATALAKE_WORKGROUP` | `panorama` | Name of the AWS Athena workgroup |
| `PANORAMA_AWS_ACCESS_KEY` | `OPENEDX_AWS_ACCESS_KEY` | AWS access key |
| `PANORAMA_AWS_SECRET_ACCESS_KEY` | `OPENEDX_AWS_SECRET_ACCESS_KEY` | AWS access secret |
| `PANORAMA_USE_SPLIT_MONGO` | `True` | Set to false for versions older than Maple |
| `PANORAMA_FLB_LOG_LEVEL` | `info` | Set the Fluent Bit logging level |
| `PANORAMA_RUN_K8S_FLUENTBIT` | `True` | Enable in exactly one SAAS/CUSTOM site per cluster; collects LMS logs from all namespaces. Set false in all other sites. |
| `PANORAMA_DEBUG` | `False` | Set to true to run Panorama ELT in verbose debug mode |
| `PANORAMA_LOGS_TOTAL_FILE_SIZE` | `50M` | Maximum size of log files before uploading to S3 |
| `PANORAMA_LOGS_UPLOAD_TIMEOUT` | `10m` | Maximum time before log files are uploaded even if they don't reach the size limit |
| `PANORAMA_LOGS_UPLOAD_CHUNK_SIZE` | `10M` | Deprecated; unused for PutObject uploads |
| `PANORAMA_K8S_JOB_BACKOFF_LIMIT` | `1` | Retry limit for recurring extraction jobs |
| `PANORAMA_K8S_JOB_ACTIVE_DEADLINE_SECONDS` | `3300` | Active deadline in seconds for recurring extraction jobs |
| `PANORAMA_K8S_CRON_STARTING_DEADLINE_SECONDS` | `300` | Maximum scheduling delay in seconds for extraction CronJobs |
| `PANORAMA_K8S_JOB_MEMORY_REQUEST` / `PANORAMA_K8S_JOB_MEMORY_LIMIT` |  | Memory request/limit for one-off and recurring jobs |
| `PANORAMA_FLB_MEM_REQUEST` | `100Mi` | Memory request for Fluentbit daemonsets in K8s. |
| `PANORAMA_FLB_CPU_REQUEST` | `100m` | CPU request for Fluentbit daemonsets in K8s. |
| `PANORAMA_FLB_MEM_LIMIT` | `256Mi` | Memory limit for Fluentbit daemonsets in K8s. |
| `PANORAMA_FLB_CPU_LIMIT` | `500m` | CPU limit for Fluentbit daemonsets in K8s. |

## Datalake Directory Structure

For each table, or for each field-based partition in each table when enabled,
one CSV file will be generated and uploaded. The file will have the same name
as the table, with a `.csv` extension.

Each CSV file will be uploaded to the following directory structure:

```text
s3://<bucket>/[<base prefix>/]<table name>/[<base partitions>/][field partitions/]<table name>.csv
```

Where:

- `bucket`: Bucket name, configured in the `panorama_raw_data_bucket` setting.
- `base prefix`: Optional subdirectory to hold tables of the same kind of
  system, for example `openedx`. It can receive files from multiple sources, as
  long as the table names are the same and share a field structure.
- `table name`: Base location of the datalake table. All text files inside this
  directory must have exactly the same column structure.
- `base partitions`: Partitions common to the same installation, in Hive
  format. These are not based on fields in the data sources, but will appear as
  fields in the datalake. For multiple Open edX installations, the default is
  to use `lms` as field name and the LMS host as the value, for example
  `lms=openedx.example.com`.
- `field partitions`: Optional partitioning for large tables. The field will be
  removed from the CSV file, but will appear as a partition field in the
  datalake. In Open edX installations, the default setting is to partition
  `courseware_studentmodule` by `course_id`.

## License

This software is licensed under the Apache 2.0 license. Please see `LICENSE`
for more details.

## Contributing

Contributions are welcome. Please submit your PR and we will review it.
For questions, please send an email to <andres@aulasneo.com>.
