Metadata-Version: 2.4
Name: msdev-kit
Version: 0.2.8
Summary: Microsoft developer toolkit: Fabric, MS Graph, and SharePoint
License: MIT
License-File: LICENSE
Author: Bernardo Rufino
Author-email: contact@bernardorufino.com
Requires-Python: >=3.10
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Dist: azure-identity (>=1.26.0,<2)
Requires-Dist: azure-kusto-data (>=6.0.4,<7.0.0)
Requires-Dist: ipykernel (>=7.3.0,<8.0.0)
Requires-Dist: numpy (==2.1.3)
Requires-Dist: openpyxl (==3.1.5)
Requires-Dist: pandas (==2.2.3)
Requires-Dist: pydantic (>=2.14.1,<3.0.0)
Requires-Dist: pyjwt (>=2.15.1,<3.0.0)
Requires-Dist: pyodbc (==5.2.0)
Requires-Dist: python-dotenv (>=1.2.4,<2)
Requires-Dist: pyyaml (>=6.0.3,<7.0.0)
Requires-Dist: requests (>=2.34.2)
Requires-Dist: sqlalchemy (==2.0.44)
Project-URL: Homepage, https://github.com/Bernardo-Rufino/msdev-kit
Project-URL: Repository, https://github.com/Bernardo-Rufino/msdev-kit
Description-Content-Type: text/markdown

# msdev-kit

Microsoft developer toolkit for Python: Fabric/Power BI, MS Graph (Entra), and SharePoint.

[![PyPI version](https://img.shields.io/pypi/v/msdev-kit)](https://pypi.org/project/msdev-kit/)
[![Python](https://img.shields.io/pypi/pyversions/msdev-kit)](https://pypi.org/project/msdev-kit/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)

> Until 2026-06-17 this package was developed in a private repository. The git
> history was squashed when the project was opened to the public, so commits
> before that date are not preserved.

## Supported OS

- Windows x64
- macOS Apple ARM
- Linux

### Unsupported OS

- Currently, because of updates to `cryptography` package (`>=50.0.0`), `Windows 32-bit` and `macOS Intel` are not supported.

## Installation

```shell
pip install msdev-kit
```

Or install from GitHub:

```shell
pip install git+https://github.com/Bernardo-Rufino/msdev-kit.git
```

For local development, use the repository instructions in [Development](docs/development.md).

**Requirements:** Python >= 3.10. Service principal authentication requires an
Azure app registration, tenant ID, client ID, and client secret. Interactive
user authentication is also supported for APIs that permit it.

---

## Quick Start

```python
from msdev_kit import Auth
from msdev_kit.fabric import Workspace
from msdev_kit.graph import GraphClient
from msdev_kit.sharepoint import SharePointClient

# authenticate
auth = Auth(tenant_id="...", client_id="...", client_secret="...")

# fabric: list workspaces
ws = Workspace(auth.get_token('pbi'))
workspaces = ws.list_workspaces_for_user()

# graph: look up a user
graph = GraphClient(auth)
user_id = graph.get_user_id('user@company.com')

# sharepoint: download a file
sp = SharePointClient(auth, sp_hostname='company', sp_site_path='sites/DataTeam')
sp.download_file('/Reports/monthly.xlsx', local_dir='./downloads')
```

---

## Authentication

`Auth` owns token acquisition. Fabric and Power BI classes receive a token string,
while Graph and SharePoint clients receive the `Auth` instance. You can use
different service principals for different services by instantiating one `Auth`
per service principal:

```python
from msdev_kit import Auth

# service principal auth
fabric_auth = Auth(tenant_id="...", client_id="spn-a", client_secret="...")
graph_auth  = Auth(tenant_id="...", client_id="spn-b", client_secret="...")
```

### Supported scopes

| Service | Scope | Usage |
|---|---|---|
| `pbi` (default) | Power BI API | `auth.get_token()` or `auth.get_token('pbi')` |
| `fabric` | Fabric API | `auth.get_token('fabric')` |
| `graph` | MS Graph API | `auth.get_token('graph')` |
| `azure` | Azure Management API | `auth.get_token('azure')` |

### Interactive user auth

For scenarios requiring user context (e.g., RLS-enabled datasets), omit the client-secret arguments to use browser authentication by default:

```python
auth = Auth()
token = auth.get_token('fabric')
```

You can also request a fresh user token from a service-principal `Auth` instance:

```python
token = auth.get_token_for_user('pbi')     # opens browser for login
token = auth.get_token_for_user('fabric')
```

If the operating system cannot provide encrypted token-cache storage, opt in
explicitly to an unencrypted cache. This can expose tokens to other users of
the same machine, so use it only when secure storage is unavailable:

```python
token = auth.get_token_for_user('fabric', allow_unencrypted_storage=True)
```

To reuse one user's interactive session across runs, configure an explicit
tenant and pass that user's Azure Identity username. The account record and
token cache are isolated by tenant and username. Omitting `account` keeps the
existing fresh interactive behavior:

```python
auth = Auth(tenant_id="your-tenant-id", client_id="spn-client-id", client_secret="...")
token = auth.get_token_for_user('fabric', account='person@example.com')
```

The first call opens a browser. Later calls reuse the saved account record and
encrypted OS token cache when available. If the signed-in username differs from
`account`, authentication fails rather than caching the wrong identity. The
account record contains identifying information but no access token and is
stored in a private directory. Use `allow_unencrypted_storage=True` only when
you accept the token exposure risk described above.

### Credentials

For the bundled examples, copy the template and set the values:

```shell
cp .env.example .env
```

`msdev-kit` does not load `.env` automatically. The scripts in `examples/` load
`.env` from the repository root. Applications may pass values directly to
`Auth`, or load environment variables with their own configuration mechanism.

---

## Fabric & Power BI

```python
from msdev_kit import Auth
from msdev_kit.fabric import Workspace, Dataset, Report, Dataflow, Pipeline

auth = Auth(tenant_id, client_id, client_secret)
```

Fabric classes take a token string — call `auth.get_token('fabric')` or `auth.get_token('pbi')` depending on the API.

- [Workspace](#workspace) — workspaces, users, permissions
- [Dataset](#dataset) — semantic models, DAX queries, permissions
- [Report](#report) — metadata, definitions, visuals, measures
- [Dataflow](#dataflow) — Gen1, Gen2, Gen2 CI/CD management
- [Pipeline](#pipeline) — Data Pipeline management
- [Other modules](#other-modules) — Capacity, Admin, KQL, Notebook, Database

### Workspace

Manage Power BI workspaces, users, and permissions.

```python
ws = Workspace(auth.get_token('pbi'))
workspaces = ws.list_workspaces_for_user()
ws.add_user('user@company.com', workspace_id, 'Member', 'User')
```

`add_user` and `update_user` accept the official Power BI principal types:
`User`, `Group`, and `App`. `add_user` updates an existing matching principal
instead of issuing a duplicate add request.

| Method | Description |
|---|---|
| `list_workspaces_for_user(...)` | List all workspaces the user has access to, with optional filters. |
| `get_workspace_details(workspace_id)` | Get details for a specific workspace. |
| `list_users(workspace_id)` | List all users in a workspace. |
| `list_reports(workspace_id)` | List all reports in a workspace. |
| `add_user(user_principal_name, workspace_id, access_right, user_type)` | Add or update a User, Group, or App in a workspace. |
| `update_user(user_principal_name, workspace_id, access_right, user_type)` | Update a User, Group, or App role in a workspace. |
| `remove_user(user_principal_name, workspace_id)` | Remove a user from a workspace. |
| `batch_update_user(user, workspaces_list)` | Batch update a user across multiple workspaces. |

### Dataset

Manage datasets (semantic models), permissions, and execute DAX queries.

```python
ds = Dataset(auth.get_token('pbi'))
result = ds.execute_query(workspace_id, dataset_id, "EVALUATE Sales")
```

| Method | Description |
|---|---|
| `list_datasets(workspace_id)` | List all datasets in a workspace. |
| `get_dataset_details(workspace_id, dataset_id)` | Get details of a specific dataset. |
| `get_dataset_name(workspace_id, dataset_id)` | Resolve the display name of a dataset. Tries PBI API first, falls back to Fabric semantic models API. |
| `execute_query(workspace_id, dataset_id, query)` | Execute a DAX query. Runs a COUNTROWS pre-check to detect truncation. |
| `list_users(workspace_id, dataset_id)` | List users with access to a dataset. |
| `add_user(user_principal_name, workspace_id, dataset_id, access_right)` | Grant a user access to a dataset. |
| `update_user(user_principal_name, workspace_id, dataset_id, access_right)` | Update a user's access to a dataset. |
| `remove_user(user_principal_name, workspace_id, dataset_id)` | Remove a user's access to a dataset. |
| `list_dataset_related_reports(workspace_id, dataset_id)` | List all reports linked to a dataset. |
| `export_dataset_related_reports(workspace_id, dataset_id)` | Export all reports linked to a dataset as `.pbix` files. |

### Report

Retrieve report metadata, definitions, visuals, and report-level measures.

```python
rpt = Report(auth.get_token('pbi'))
pages = rpt.list_report_pages(workspace_id, report_id)
```

| Method | Description |
|---|---|
| `list_reports(workspace_id)` | List all reports in a workspace. |
| `get_report_metadata(workspace_id, report_id)` | Get metadata for a specific report. |
| `get_report_name(workspace_id, report_id)` | Get a report's display name. |
| `list_report_pages(workspace_id, report_id)` | List all pages in a report. |
| `get_legacy_report_pages_and_visuals(json_data, workspace_id, report_id)` | Parse a PBIR-Legacy report JSON and extract pages/visuals into a DataFrame. |
| `get_legacy_report_json(workspace_id, report_id, operations)` | Get and decode the full report definition for PBIR-Legacy reports. |
| `export_report(workspace_id, report_id, ...)` | Export a report as a `.pbix` file. |
| `get_report_measures(workspace_id, report_id, operations)` | Extract report-level measures and generate a DAX Query View script. |
| `rebind_report(workspace_id, report_id, new_dataset_id, ...)` | Rebind a report to a new dataset and migrate Read access. |

### Dataflow

Manage Power BI and Fabric dataflows, including Gen1, Gen2, and Gen2 CI/CD.

```python
df = Dataflow(auth.get_token('pbi'))

# upgrade Gen1 to Gen2 CI/CD
result = df.upgrade_to_gen2_cicd(
    workspace_id='<workspace_id>',
    dataflow_id='<gen1_dataflow_id>',
    display_name='my_dataflow_cicd',
    source_type='gen1'
)
```

| Method | Description |
|---|---|
| `list_dataflows(workspace_id)` | List all dataflows (Gen1, Gen2, Gen2 CI/CD), merged and deduplicated. |
| `get_dataflow_details(workspace_id, dataflow_id)` | Get details of a specific dataflow. |
| `get_dataflow_name(workspace_id, dataflow_id)` | Resolve the display name of a dataflow. |
| `create_dataflow(workspace_id, dataflow_content)` | Create a new Power BI dataflow. |
| `delete_dataflow(workspace_id, dataflow_id, type='pbi')` | Delete a dataflow. Use `type='fabric'` for Fabric API. |
| `export_dataflow_json(workspace_id, dataflow_id, dataflow_name)` | Export a dataflow definition as JSON. |
| `get_dataflow_gen2_definition(workspace_id, dataflow_id)` | Get the definition of a Dataflow Gen2 CI/CD item. |
| `create_dataflow_gen2_from_definition(workspace_id, display_name, definition)` | Create a Dataflow Gen2 CI/CD from a definition. |
| `update_dataflow_gen2_from_definition(workspace_id, dataflow_id, display_name, definition)` | Update an existing Dataflow Gen2 CI/CD definition. |
| `get_data_destinations(workspace_id, dataflow_id)` | Get data destination details for each table in a dataflow. |
| `get_workspace_data_destinations(workspace_id, max_workers=4)` | Inspect every dataflow, return its destination details, and save destination-only rows as a workbook under `data/dataflows`. The inventory is paced at 200 requests per minute and retries 429 responses. |
| `change_data_destination(workspace_id, dataflow_id, destination_type, ...)` | Change data destination (Lakehouse/Warehouse). Modes: `preview`, `replace`, `create`. |
| `create_dataflow_with_new_destination(workspace_id, dataflow_id, ...)` | Create a new Gen2 CI/CD dataflow with a different data destination. |
| `upgrade_to_gen2_cicd(..., refresh=False)` | Upgrade a Gen1 or Gen2 dataflow to Gen2 CI/CD. For standard Gen2, verify data destinations and preserve bound connection IDs by default. Set `use_accessible_connections=True` to bind exact connector paths to shared connections visible to the client. Set `refresh=True` to run and wait for the new dataflow's refresh. |

For a standard Gen2 source, the upgrade keeps each destination's workspace,
item, table, and write method. It returns an error without creating a new item
if it cannot verify destination preservation. A CI/CD source is copied from its
definition without retargeting destination references. Gen1 uses its own
internal storage and has no Gen2 data destination to reuse.

For service principal creation of a standard Gen2 dataflow, use a delegated
Fabric user token for refresh. Fabric does not allow the service principal to
run the Dataflow Gen2 CI/CD refresh job. The delegated token is used only for
the refresh request and polling. Pass an SPN Power BI token separately for
reading the standard source. Connection credentials are not copied.
See the [complete setup and runnable example](docs/dataflow_gen2_cicd_upgrade.md)
before running this against an existing data destination.

```python
spn_auth = Auth(tenant_id, client_id, client_secret)
user_auth = Auth()  # interactive user login
df = Dataflow(spn_auth.get_token('fabric'))
result = df.upgrade_to_gen2_cicd(
    workspace_id=workspace_id,
    dataflow_id=standard_gen2_dataflow_id,
    source_type='gen2',
    use_accessible_connections=True,
    refresh=True,
    refresh_access_token=user_auth.get_token('fabric'),
    pbi_access_token=spn_auth.get_token('pbi'),
)
```
Refresh is opt-in because it writes to the configured destination. A failed
refresh returns its job details and the new item's ID for investigation.

#### Inventory workspace data destinations

```python
from msdev_kit import Auth
from msdev_kit.fabric import Dataflow

workspace_id = "<workspace-id>"
auth = Auth(tenant_id="<tenant-id>", client_id="<client-id>", client_secret="<client-secret>")
dataflow = Dataflow(auth.get_token("pbi"))

result = dataflow.get_workspace_data_destinations(workspace_id, max_workers=4)
```

The method inspects every dataflow so `result["content"]` also records empty
inspections and failures. Its workbook contains only table rows with a real data
destination. Table name is exported as `table_name`, not `table_table`. A Fabric
source without an API generation value is normalized to `2.1`. The progress line
is updated in place. If a 429 response occurs, the method prints the completed
count, waits, then resumes the progress line. See
[`examples/dataflow_destinations.py`](examples/dataflow_destinations.py) for a
DataFrame normalizer and a runnable placeholder.

### Pipeline

Manage Fabric Data Pipelines.

```python
pipe = Pipeline(auth.get_token('fabric'))
activities = pipe.get_pipeline_activities(workspace_id, 'My Pipeline')
all_activities = pipe.get_pipeline_activities(
    workspace_id, 'My Pipeline', recursive=True
)
```

| Method | Description |
|---|---|
| `list_pipelines(workspace_id)` | List all Fabric Data Pipelines in a workspace. |
| `get_pipeline(workspace_id, pipeline_id)` | Get the metadata of a specific pipeline. |
| `get_pipeline_definition(workspace_id, pipeline_id)` | Get the full definition of a pipeline. |
| `update_pipeline_definition(workspace_id, pipeline_id, definition)` | Update an existing pipeline definition. |
| `get_pipeline_activities(workspace_id, pipeline_id_or_name, max_workers=5, recursive=False)` | Get activities from a pipeline by ID or display name. Set `recursive=True` to include invoked pipelines across workspaces. |
| `find_pipelines_by_dataflow(workspace_id, dataflow_id_or_name)` | Find pipelines that reference one dataflow or a list of dataflows, fetching each pipeline definition once. |
| `replace_dataflow_id_in_pipeline(workspace_id, pipeline_id, old_id, new_id)` | Replace a dataflow ID in all RefreshDataflow activities. |

Recursive results are flat and include `workspace_id` on each activity. They
include activities inside control-flow containers and follow both
`InvokePipeline` and `ExecutePipeline` references. Each pipeline is scanned
once, so cycles and repeated invocations do not duplicate downstream
activities. An inaccessible child returns an error rather than a partial result.

To find pipelines for multiple dataflows, pass IDs, display names, or a mixture
of both. The method lists the pipelines once and scans each pipeline definition
once, concurrently up to `max_workers`:

```python
matches = pipe.find_pipelines_by_dataflow(
    workspace_id,
    ['sales-dataflow-id', 'Finance Dataflow'],
    max_workers=5,
)
```

For a list input, each matching pipeline includes a `dataflows` list. Every
entry identifies the matched dataflow and its matching `RefreshDataflow`
activities. Single-dataflow calls keep the existing `activities` list output.

<details>
<summary><strong>Example: replacing a dataflow destination and updating pipelines</strong></summary>

When `change_data_destination(mode='replace')` is used on a standard Gen2 dataflow, the original is deleted and a new CI/CD dataflow is created with a new ID. Pipelines referencing the old ID must be updated:

```python
from msdev_kit import Auth
from msdev_kit.fabric import Dataflow, Pipeline

auth = Auth(tenant_id, client_id, client_secret)
dataflow = Dataflow(auth.get_token('pbi'))
pipeline = Pipeline(auth.get_token('fabric'))

workspace_id = '<workspace_id>'
old_dataflow_id = '<dataflow_id>'

# 1. replace dataflow destination (creates new CI/CD, deletes original)
result = dataflow.change_data_destination(
    workspace_id=workspace_id,
    dataflow_id=old_dataflow_id,
    destination_type='Warehouse',
    destination_workspace_id=workspace_id,
    destination_item_id='<warehouse_id>',
    mode='replace'
)
new_dataflow_id = result['content']['id']

# 2. find pipelines referencing the old dataflow ID
matches = pipeline.find_pipelines_by_dataflow(workspace_id, old_dataflow_id)

# 3. update each pipeline to use the new ID
for m in matches['content']:
    pipeline.replace_dataflow_id_in_pipeline(
        workspace_id, m['pipeline_id'], old_dataflow_id, new_dataflow_id
    )
```

</details>

### Other modules

| Module | Class | Description |
|---|---|---|
| Capacity | `Capacity` | Monitor and manage Power BI and Fabric capacities. |
| Operations | `Operations` | Track long-running Fabric API operations. |
| Admin | `Admin` | Power BI Admin API operations. |
| KQL | `KQLDatabase` | Query Kusto (KQL) databases in Microsoft Fabric. |
| Notebook | `Notebook` | Manage Fabric notebooks (list, get metadata). |
| Database | `Database` | Query and write to SQL databases (Lakehouse, Warehouse) via ODBC. |

---

## MS Graph (Entra)

Manage Entra ID (Azure AD) users and groups via the MS Graph API.

```python
from msdev_kit import Auth
from msdev_kit.graph import GraphClient

auth = Auth(tenant_id, client_id, client_secret)
graph = GraphClient(auth)

# look up a user and add them to a group
user_id = graph.get_user_id('user@company.com')
group_id = graph.get_group_id('Data Team')
graph.add_group_member(group_id, user_id)

# list all members of a group
members = graph.list_group_members(group_id)
```

| Method | Description |
|---|---|
| `get_user_id(email)` | Resolve user object ID by UPN/email, with mail fallback. |
| `get_group_id(group_name)` | Resolve Entra group object ID by display name. |
| `list_group_members(group_id)` | Paginated member list (id, displayName, mail, UPN). |
| `add_group_member(group_id, user_id)` | Add user to group. Silently ignores already-member errors. |
| `remove_group_member(group_id, user_id)` | Remove user from group. Silently ignores 404/403. |

---

## SharePoint

Manage SharePoint files and folders via MS Graph API (no ACS/Office365 dependency).

```python
from msdev_kit import Auth
from msdev_kit.sharepoint import SharePointClient

auth = Auth(tenant_id, client_id, client_secret)
sp = SharePointClient(auth, sp_hostname='company', sp_site_path='sites/DataTeam')

# download a file
sp.download_file('/Reports/monthly.xlsx', local_dir='./downloads')

# upload a file (from path or bytes)
sp.upload_file('/Reports/updated.xlsx', source='./local/updated.xlsx')
sp.upload_file('/Reports/data.csv', source=csv_bytes, content_type='text/csv')

# create nested folders
sp.create_folder('/Reports/2026/Q1')
```

| Method | Description |
|---|---|
| `download_file(file_path, local_dir)` | Download a file from the default document library. Returns local file path. |
| `upload_file(remote_path, source, content_type?)` | Upload/overwrite a file up to 250 MB. `source` is a local file path or raw bytes. |
| `create_folder(folder_path)` | Create a folder and all intermediate folders. |

Hostname and site path inputs are normalized automatically:

| Input | Normalized to |
|---|---|
| `company` | `company.sharepoint.com` |
| `company.sharepoint.com` | `company.sharepoint.com` |
| `https://company.sharepoint.com` | `company.sharepoint.com` |
| `DataTeam` | `sites/DataTeam` |
| `sites/DataTeam` | `sites/DataTeam` |
| `/sites/DataTeam` | `sites/DataTeam` |

---

## Limitations

- Power BI and Fabric endpoints apply operation-specific throttling. Do not assume one global quota. `get_workspace_data_destinations` intentionally paces definition lookups at 200 requests per minute and backs off after HTTP 429.
- Not all users can be updated via the API. See Microsoft docs: [Dataset permissions](https://learn.microsoft.com/en-us/power-bi/developer/embedded/datasets-permissions#get-and-update-dataset-permissions-with-apis).
- **Dataset query limits** (executeQueries API):
  - Max **100,000 rows** or **1,000,000 values** (rows x columns) per query, whichever is hit first.
  - Max **15 MB** of data per query.
  - **120 query requests per minute** per user.
  - Only **DAX** queries are supported (no MDX, INFO functions, or DMV).
  - Datasets hosted in Azure Analysis Services or with a live connection to on-premises AAS are not supported.
  - Service Principals are not supported for datasets with RLS or SSO enabled.

---

## Examples

Runnable, placeholder-based scripts live in [`examples/`](./examples). Start
with the [examples guide](examples/README.md), then run a read-only example from
the repository root:

```bash
cp .env.example .env
python -m examples.workspaces
python -m examples.dataflow_destinations
```

Write examples never perform a mutation when run directly. Edit the placeholder
values and call their explicit helper only after reviewing the target IDs.

---

## Contributing

Read [Contributing](CONTRIBUTING.md) before opening a pull request. It links the
local setup, validation commands, issue and pull request process, branch
conventions, and release boundary.

