Metadata-Version: 2.4
Name: spfetch
Version: 0.1.1
Summary: Simple. Streaming. Resilient. MFA-ready. Fetch files from SharePoint via Microsoft Graph.
Author: João Guilherme Brandi
License: MIT License
        
        Copyright (c) 2026 João Guilherme Brandi
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/jg-ebrandi/spfetch
Project-URL: Repository, https://github.com/jg-ebrandi/spfetch.git
Project-URL: Issues, https://github.com/jg-ebrandi/spfetch/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: msal>=1.20.0
Requires-Dist: httpx>=0.24.0
Requires-Dist: fsspec>=2023.1.0
Provides-Extra: pandas
Requires-Dist: pandas>=1.5.0; extra == "pandas"
Requires-Dist: openpyxl>=3.0.0; extra == "pandas"
Provides-Extra: s3
Requires-Dist: s3fs>=2023.1.0; extra == "s3"
Provides-Extra: gcs
Requires-Dist: gcsfs>=2023.1.0; extra == "gcs"
Provides-Extra: azure
Requires-Dist: adlfs>=2023.1.0; extra == "azure"
Dynamic: license-file

# 🚀 spfetch

![spfetch_lg](https://github.com/user-attachments/assets/c66f083b-3899-4482-94da-1f85609b357e)

<p align="center">
  <b>Simple. Streaming. Resilient. MFA-ready.</b><br>
  List and fetch files from <b>SharePoint</b> via <b>Microsoft Graph</b> with clean APIs and cloud-native downloads.
</p>

---

## ✨ What is spfetch?

`spfetch` is an asynchronous Python library built for data pipelines:

- 📂 **List** SharePoint folders with structured metadata.
- ⬇️ **Stream** large files directly to Local Disk, S3, GCS, or Azure without memory crashes.
- 📊 **Load** small files directly into Pandas DataFrames.
- 🔐 **Authenticate** via MFA (Device Code) or Silent (Client Secret) flows.
- 🛡️ **Auto-Recover** from Microsoft API Throttling (HTTP 429) using Exponential Backoff.

---

## 🔐 1. Authentication

Before running any workflow, you must instantiate the client with your Microsoft Entra ID (Azure AD) credentials.

### Option A: Interactive / Local (Device Code Flow)
Ideal for local scripts. Supports MFA.

```python
from spfetch.auth import DeviceCodeAuth
from spfetch.client import SharePointClient

auth = DeviceCodeAuth(tenant_id="<YOUR_TENANT_ID>", client_id="<YOUR_CLIENT_ID>")
client = SharePointClient(auth=auth)
```

### Option B: Automated / CI-CD (Client Secret Flow)
Ideal for Airflow, Databricks, or GitHub Actions.

```python
from spfetch.auth import ClientSecretAuth
from spfetch.client import SharePointClient

auth = ClientSecretAuth(
    tenant_id="<YOUR_TENANT_ID>", 
    client_id="<YOUR_CLIENT_ID>", 
    client_secret="<YOUR_CLIENT_SECRET>"
)
client = SharePointClient(auth=auth)
```

---

## 📖 2. Exploration: Listing Folders

📦 Required Installation:

```bash
pip install spfetch
```

```python
import asyncio

async def list_files():
    items = await client.ls(
        hostname="<tenant>.sharepoint.com",
        site_path="/sites/<YourSite>",
        folder_path="/Shared Documents/General"
    )
    for item in items:
        print(item["name"], item["size"], item["is_folder"])

asyncio.run(list_files())
```

---

## 🌊 3. Ingestion Workflows

### 💻 Workflow A: Download to Local Disk

📦 Required Installation:

```bash
pip install spfetch
```

```python
from spfetch.destinations import LocalDestination
import asyncio

async def download_local():
    # 1. Setup local destination
    dest = LocalDestination()
    
    # 2. Stream to disk
    await client.download(
        hostname="<tenant>.sharepoint.com",
        site_path="/sites/<YourSite>",
        file_path="/Shared Documents/Data/file.csv",
        dest_path="./local_downloads/file.csv",  # Local file path
        destination=dest
    )

asyncio.run(download_local())
```

---

### ☁️ Workflow B: Download directly to Azure (ADLS / Blob)

📦 Required Installation:

```bash
pip install "spfetch[azure]"
```

```python
from spfetch.destinations import AzureDestination
import asyncio

async def download_to_azure():
    # 1. Setup Azure credentials
    dest = AzureDestination(
        account_name="<YOUR_STORAGE_ACCOUNT_NAME>",
        account_key="<YOUR_STORAGE_ACCOUNT_KEY>"  # Or sas_token="<YOUR_SAS_TOKEN>"
    )
    
    # 2. Stream directly to Azure (abfs://)
    await client.download(
        hostname="<tenant>.sharepoint.com",
        site_path="/sites/<YourSite>",
        file_path="/Shared Documents/Data/file.parquet",
        dest_path="abfs://<container_name>/bronze/file.parquet",
        destination=dest
    )

asyncio.run(download_to_azure())
```

---

### ☁️ Workflow C: Download directly to Amazon S3

📦 Required Installation:

```bash
pip install "spfetch[s3]"
```

```python
from spfetch.destinations import S3Destination
import asyncio

async def download_to_s3():
    # 1. Setup AWS credentials
    dest = S3Destination(
        key="<AWS_ACCESS_KEY_ID>",
        secret="<AWS_SECRET_ACCESS_KEY>"
    )
    
    # 2. Stream directly to S3 (s3://)
    await client.download(
        hostname="<tenant>.sharepoint.com",
        site_path="/sites/<YourSite>",
        file_path="/Shared Documents/Data/file.csv",
        dest_path="s3://<bucket_name>/raw/file.csv",
        destination=dest
    )

asyncio.run(download_to_s3())
```

---

### ☁️ Workflow D: Download directly to Google Cloud Storage (GCS)

📦 Required Installation:

```bash
pip install "spfetch[gcs]"
```

```python
from spfetch.destinations import GCSDestination
import asyncio

async def download_to_gcs():
    # 1. Setup GCS credentials (can use default environment or token path)
    dest = GCSDestination(
        project="<my-gcp-project-id>",
        token="google_default"  # Or path to service_account.json
    )
    
    # 2. Stream directly to GCS (gs://)
    await client.download(
        hostname="<tenant>.sharepoint.com",
        site_path="/sites/<YourSite>",
        file_path="/Shared Documents/Data/file.csv",
        dest_path="gs://<bucket_name>/raw/file.csv",
        destination=dest
    )

asyncio.run(download_to_gcs())
```

---

### 📊 Workflow E: Read directly to Pandas DataFrame

📦 Required Installation:

```bash
pip install "spfetch[pandas]"
```

Ideal for smaller files (`.csv`, `.xlsx`). This method skips saving to disk and loads the file straight into memory.

```python
import asyncio

async def read_to_memory():
    # client.read_df accepts all standard pandas kwargs (sheet_name, sep, skiprows, etc.)
    df = await client.read_df(
        hostname="<tenant>.sharepoint.com",
        site_path="/sites/<YourSite>",
        file_path="/Shared Documents/Reports/data.xlsx",
        sheet_name="Sheet1",
        skiprows=2,
        usecols="A:D"
    )
    print(df.head())

asyncio.run(read_to_memory())
```

---

## 🛡️ 4. Resilience (Handling HTTP 429)

Microsoft Graph API strictly throttles heavy requests. spfetch handles this out-of-the-box.

If a `429 Too Many Requests` occurs, the client automatically:

- Pauses execution.
- Reads the `Retry-After` header.
- Applies Exponential Backoff.
- Retries seamlessly (up to 5 times for downloads).

Your pipeline won't crash; it will simply wait and recover gracefully.

---

## 🤝 Contributing

PRs are welcome! Check our `CONTRIBUTING.md`. Ensure all tests pass via `make test`.

---

## 📄 License

MIT
