Metadata-Version: 2.5
Name: repo-exporter
Version: 2.0.0
Summary: A Python package to collect key information (Name, description, creation date, last updated date, contributors, etc.) from all repos in a GitHub or Hugging Face organization and save it to a color-coded Google Sheet file of choice.
Project-URL: Homepage, https://github.com/Imageomics/repo-exporter
Project-URL: Repository, https://github.com/Imageomics/repo-exporter.git
Project-URL: Bug Tracker, https://github.com/Imageomics/repo-exporter/issues
Author-email: "Elizabeth G. Campolongo" <e.campolongo479@gmail.com>
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: google-auth==2.41.1
Requires-Dist: gspread==6.2.1
Requires-Dist: huggingface-hub==1.2.3
Requires-Dist: pandas==2.3.3
Requires-Dist: pygithub==2.8.1
Requires-Dist: python-dotenv==1.2.2
Requires-Dist: pyyaml==6.0.3
Requires-Dist: tqdm==4.67.1
Provides-Extra: dev
Requires-Dist: pytest==9.0.3; extra == 'dev'
Provides-Extra: keywords
Requires-Dist: automation; extra == 'keywords'
Requires-Dist: github; extra == 'keywords'
Requires-Dist: google-sheets; extra == 'keywords'
Requires-Dist: huggingface; extra == 'keywords'
Requires-Dist: imageomics; extra == 'keywords'
Requires-Dist: metadata; extra == 'keywords'
Description-Content-Type: text/markdown

# Repository Exporter [![DOI](https://zenodo.org/badge/1080019710.svg)](https://doi.org/10.5281/zenodo.17835081)

Python package that gathers metadata for all repositories in a provided GitHub or Hugging Face organization and automatically exports the data into a designated Google Sheet (using a Google Cloud Console Service Account) for easy viewing and analysis.

## Contents
- [Features](#features)  
- [Usage](#usage)  
- [Set up your own GitHub Actions workflow](#set-up-your-own-github-actions-workflow)  
  - [Create a GitHub Personal Access Token](#create-a-github-personal-access-token)
  - [Create a Hugging Face Token](#create-a-hugging-face-token)   
  - [Set up Google Cloud Service Account Access](#set-up-google-cloud-service-account-access)  
- [Run repo exporter locally](#run-repo-exporter-locally)  
  - [Command-line options](#command-line-options)
- [Environment Variables](#environment-variables)  
- [Testing](#testing)

---

## Features
- Fetches all repositories in the indicated GitHub or Hugging Face organization(s)
- Collects key details:
  - Repo visibility, name (hyperlinked to repo), and description
  - Date created and last updated, with a flag for inactivity (set after a year of no commits)
  - Creator and top 4 contributors (`N/A` creator means it was either a transferred repository or a forked repository and `None (<GitHub Username>)` means there was no full name attached to their GitHub account)
  - Number of stars and number of branches for GitHub repositories, for Hugging Face repos, the analogous number of likes and open Discussions/PRs are collected
  - Standard file/metadata checks:
      - **For GitHub:** `README.md`, license, `.gitignore`, package requirements (`requirements.txt`, `environment.yaml`, etc.), `CITATION.cff`, `.zenodo.json`, `CONTRIBUTING.md`, and `AGENTS.md` files
      - **For Hugging Face:** `README.md` ([dataset or model card](https://imageomics.github.io/Collaborative-distributed-science-guide/wiki-guide/About-Templates/)/Space README) and license (read from `yaml`)
      - DOI for the repository (HF generated or from Zenodo for GitHub repos)
  - Primary Programming Language (**GitHub only**)
  - Website Reference/Homepage, Associated Dataset(s), Model(s), or Paper(s), and associated [GitHub] repo for Hugging Face repositories
  - Supports configurable worksheet names with defaults:
    - GitHub: `GH_SHEET_NAME` defaults to `GH-Repos`
    - Hugging Face: `HF_SHEET_NAME` defaults to `HF-Repos`
- Exports everything to a given Google Sheet document that it will require Editor permission to on the sheet's sharing permissions list
- For [**Standard Files**](https://imageomics.github.io/Collaborative-distributed-science-guide/wiki-guide/GitHub-Repo-Guide/#standard-files) highlights **No** data cell values with red cell colors and for [**Recommended Files**](https://imageomics.github.io/Collaborative-distributed-science-guide/wiki-guide/GitHub-Repo-Guide/#recommended-files) and **Filters** highlights **No** data cell values with orange cell colors

## Usage
The workflow runs automatically each week (9am UTC on Mondays); however, you can also run the GitHub Actions workflow manually:

   1. Go to the [Actions tab](https://github.com/Imageomics/repo-exporter/actions)
   2. Click **Update Metadata for GitHub Repository Sheet**
   3. Click **Run workflow**, with branch as **Branch: main**, with selection **all** and finally press **Run workflow**

## Set up your own GitHub Actions workflow

To use this script within your own GitHub organization, first fork this repo, then follow the setup steps below to ensure proper access.

### Create a GitHub Personal Access Token
  
  To create one with permissions for both private and public repositories (public repository read-access only is enabled by default without adminstrator approval):
   
  1. Go to [github.com/settings/personal-access-tokens](https://github.com/settings/personal-access-tokens)
  2. Click **Generate new token -> Fine-grained token**
  3. Under **Resource owner**, select the **organization** you want to access.
  4. Under **Repository access**, choose **All repositories**.
  5. Under **Permissions** select **Repositories** and set:
      - **Metadata** -> Read-only 
      - **Contents** -> Read-only
      - **Administration** -> Read-only
  6. Click **Generate token** and **copy it** (make sure to store it somewhere safe for future use).
  7. Navigate to `https://github.com/<gh-org-name>/repo-exporter/settings/secrets/actions` and click **New repository secret** and name it **GH_TOKEN** and copy paste the token into the **Secret** section and click **Add secret**
  **Note:** The token must be approved by the organization administrator before accessing private repositories.

### Create a Hugging Face Token

  To create one with permissions for both private and public repositoriesL

  1. Go to [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)
  2. Click on **New Token** and name it **repo-exporter**
  3. For permissions select **Fine-grained**:
      - Specify the desired organization (under **Org permissions**)
      - Under **Repositories**, select "Read access to contents of all repos in selected organizations"
  4. Click **Generate** and **copy it** (make sure to store it somewhere safe for future use)
  5. Navigate to `https://github.com/<gh-org-name>/repo-exporter/settings/secrets/actions` and click **New repository secret** and name it **HF_TOKEN** and copy paste the token into the **Secret** section and click **Add secret** 

### Set up Google Cloud Service Account Access

Instructions to create a Google Cloud Console Service Account and give it permission to use in the repository and in the Google sheet:

 1. Go to https://console.cloud.google.com/
 2. Under "IAM & Admin", create a **new project** and name it **inventory**
 3. Go to https://console.cloud.google.com/iam-admin/serviceaccounts, if you have multiple projects you'll need to **select the project** that you just made if it hasn't already been selected
 4. Create a **service account**, named **Imageomics**, with description: "Repo checklist automation account" and finally press **Done** (You do not need to add any Permissions or Principals with access)
 5. Click on the **service account email** -> **Keys** -> **Add key** -> **Create new key** and select **JSON** then finally click **Create**
 7. Go to `https://github.com/<gh-org-name>/repo-exporter/settings/secrets/actions` and click **New repository secret** and name it **GOOGLE_SERVICE_ACCOUNT_JSON** and copy paste the entire contents of the JSON file into the **Secret** section and click **Add secret**
 8. Go to https://console.cloud.google.com/apis/library/sheets.googleapis.com and enable the **Google Sheets API** for the project you made
 9. Go to your chosen Google Sheet and go to **Share** settings and add the new Service Account email you made and set it as an **Editor**

After cloning the repository, configure the [environment variables](#environment-variables) required for the exporter(s) you plan to run.

Once configured, the workflow can be run by following the [Usage Instructions](#usage).

## Run repo exporter locally
   
1. Create and activate a Conda environment:
   ```
   conda create -n repo-exporter python -y
   conda activate repo-exporter
   ```
    
2. [RECOMMENDED] Create a `.env` file in the root of the project to configure required [environment variables](#environment-variables). See [`.env.example`](.env.example) for an example; the default sheet names are included and can be removed. Any value in `.env` can also be overridden with a CLI flag. See [Command-line options](#command-line-options) below.


3. Install the package (this installs the `repo-exporter` CLI along with all dependencies, as defined in `pyproject.toml`):
   ```
   pip install repo-exporter
   ```
    
4. Each exporter runs individually with the `repo-exporter` command, specifying the platform as a subcommand:

    - **Run the GitHub repository exporter**
      ```
      repo-exporter github
      ```

    - **Run the Hugging Face repository exporter**
      ```
      repo-exporter huggingface
      ```

    - Check the installed version at any time with:
      ``` 
      repo-exporter --version
      ```

### Command-line options

All options fall back to the corresponding [environment variable](#environment-variables) when omitted, so you can mix and match `.env` values with one-off CLI overrides.

**`repo-exporter github [options]`**

| Flag | Overrides | Description |
|---|---|---|
| `--org` | `GH_ORG_NAME` | GitHub organization name |
| `--token` | `GH_TOKEN` | GitHub personal access token |
| `--repo-type` | `GH_REPO_TYPE` | Repo type filter: `all`, `public`, `private`, `forks`, `sources`, `member` (default: `all`) |
| `--spreadsheet-id` | `SPREADSHEET_ID` | Google Sheets spreadsheet ID |
| `--sheet-name` | `GH_SHEET_NAME` | Sheet tab name (default: `GH-Repos`) |
| `--credentials-path` | `GOOGLE_CREDENTIALS_PATH` | Path to `service_account.json` (default: `service_account.json`) |

**`repo-exporter huggingface [options]`**

| Flag | Overrides | Description |
|---|---|---|
| `--org` | `HF_ORG_NAME` | Hugging Face organization name (**case-sensitive**) |
| `--token` | `HF_TOKEN` | Hugging Face token |
| `--spreadsheet-id` | `SPREADSHEET_ID` | Google Sheets spreadsheet ID |
| `--sheet-name` | `HF_SHEET_NAME` | Sheet tab name (default: `HF-Repos`) |
| `--credentials-path` | `GOOGLE_CREDENTIALS_PATH` | Path to `service_account.json` (default: `service_account.json`) |

Run `repo-exporter github --help` or `repo-exporter huggingface --help` to see these options from the command line.

## Environment Variables

> [!NOTE]
> `SPREADSHEET_ID` and Google service account credentials are required for both GitHub and Hugging Face exports, as well as the platform-prefaced variables (e.g., `GH_ORG_NAME` and `HF_ORG_NAME`)
> * Both exporters support optional custom sheet name variables (`GH_SHEET_NAME` and `HF_SHEET_NAME`; these must be *distinct*).
> * Any of these values may instead be passed as a CLI flag. See [Command-line options](#command-line-options).

> [!WARNING]  
> Tokens and Google service account credentials must be saved as repository *secrets*. All other values may be saved as environment variables assuming your spreadsheet is appropriately protected.

### GitHub exporter

* Set `GH_ORG_NAME` to your GitHub organization name (for API calls).
* Set `SPREADSHEET_ID` to the Google Sheet ID used by the exporter.
* `GH_SHEET_NAME` is optional. If not provided, the exporter uses "GH-Repos".
* `GH_REPO_TYPE` is optional. If not provided, the exporter uses "all" (also accepts `public`, `private`, `forks`, `sources`, `member`). 
* `GH_TOKEN` is required to access GitHub repositories.

### Hugging Face exporter

* Set `HF_ORG_NAME` to your Hugging Face organization name (**case-sensitive**, for API calls).
* Set `SPREADSHEET_ID` to the Google Sheet ID used by the exporter.
* `HF_SHEET_NAME` is optional. If not provided, the exporter uses "HF-Repos".
* `HF_TOKEN` is required to access Hugging Face repositories.

### Shared configuration

* Both exporters require Google service account credentials created in the [service account setup](#set-up-google-cloud-service-account-access). Set `GOOGLE_CREDENTIALS_PATH` to the JSON key file path (defaults to `service_account.json` if omitted).
* The Google service account must have Editor access to the target spreadsheet.
* Ensure all required values are available as environment variables locally or as GitHub Actions secrets.

For example, if the spreadsheet URL is:

```text
https://docs.google.com/spreadsheets/d/fake-long-alpha-numeric-id-1a2b3c4d/edit
```

then the spreadsheet ID is:

```text
fake-long-alpha-numeric-id-1a2b3c4d
```


---

## Development

To develop the package further:

1. Clone the repository and create a branch.
2. Install the package with dev dependencies in a dedicated environment ([local install instructions](#run-repo-exporter-locally)) at the repo root:
   ```console
    pip install -e ".[dev]"
   ```

### Testing

Run the following in your `repo-exporter` environment to run all tests:
   ```console
   python -m pytest -q
   ```
