Metadata-Version: 2.4
Name: laiser
Version: 1.0.1
Summary: Leveraging Artificial Intelligence for Skills Extraction and Research
Author-email: LAiSER Team <PSCWP@gwu.edu>
License-Expression: BSD-3-Clause
Project-URL: Homepage, https://github.com/LAiSER-Software/extract-module
Project-URL: Repository, https://github.com/LAiSER-Software/extract-module
Project-URL: Documentation, https://github.com/LAiSER-Software/extract-module/blob/main/README.md
Project-URL: Bug Reports, https://github.com/LAiSER-Software/extract-module/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=1.9.0
Requires-Dist: transformers>=4.20.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: numpy>=1.21.0
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: faiss-cpu>=1.7.0
Requires-Dist: certifi>=2023.0.0
Requires-Dist: requests>=2.25.0
Requires-Dist: psutil
Requires-Dist: google-genai>=0.8.0
Provides-Extra: gpu
Requires-Dist: vllm>=0.2.0; extra == "gpu"
Requires-Dist: bitsandbytes>=0.43.1; extra == "gpu"
Requires-Dist: accelerate>=0.26.0; extra == "gpu"
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.0; extra == "dev"
Requires-Dist: pytest-check>=2.0.0; extra == "dev"
Requires-Dist: black>=22.0; extra == "dev"
Requires-Dist: flake8>=4.0; extra == "dev"
Requires-Dist: isort>=5.0; extra == "dev"
Requires-Dist: build>=0.8.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: python-dotenv<2.0.0,>=0.19.0; extra == "dev"
Dynamic: license-file

> [!CAUTION]
> <h3>LAiSER is currently in development mode, features could be experimental. Use with caution!</h3>


<div align="center">
<img src="https://i.imgur.com/XznvjNi.png" width="70%"/>
<h1>Leveraging ​Artificial ​Intelligence for ​Skill ​Extraction &​ Research (LAiSER)</h1>

<a href="https://laiser-software.github.io/extract-module/"><b>Documentation</b></a> ·
<a href="https://pypi.org/project/laiser/"><b>PyPI</b></a> ·
<a href="https://github.com/LAiSER-Software/laiser-cookbook"><b>Cookbook notebooks</b></a>
</div>

LAiSER turns free text such as job postings and course syllabi into structured skills, knowledge and tasks, each matched to an entry in ESCO, O\*NET, the UK Skills Classification or the Open Skills Network.

### Contents

- [About](#about)
- [Architecture](#architecture)
- [Requirements](#requirements)
- [Setup and Installation](#setup-and-installation)
- [Usage](#usage)
- [Try it in Google Colab](#try-it-in-google-colab)
- [Funding](#funding)
- [Authors](#authors)
- [Partners](#partners)

## About

LAiSER is a Python package for turning unstructured text about work and learning into structured, comparable skill data.

You give it a table of documents — job postings, course descriptions, syllabi. For each document, a language model extracts the skills it describes and, optionally, the knowledge and tasks behind them. LAiSER then matches every extracted phrase to its closest entries in established taxonomies using embedding similarity. The result is a table with one row per match: the phrase as written, the taxonomy entry it corresponds to, which taxonomy that entry comes from, and a similarity score.

Because results point at shared taxonomy entries rather than free-text phrases, they can be counted and compared across sources — the skills employers ask for can be set against the skills a program teaches, or tracked across thousands of postings.

The language model can be a hosted API (Gemini or OpenAI) or a model running on your own machine, including on CPU with no API key. Taxonomy alignment always runs locally against indexes prepared during setup.

## Architecture

LAiSER runs four stages for every document:

1. **Extraction** — The text is placed into a prompt for its input type and sent to the language model, which returns candidate concepts.
2. **Parsing and deduplication** — The response is parsed into phrases. Exact duplicates are dropped and near-duplicates are collapsed by embedding similarity.
3. **Taxonomy alignment** — Each phrase is embedded and searched against local FAISS indexes of taxonomy entries. Matches above a per-type similarity threshold are kept.
4. **Output** — Matches are returned as a single table, optionally with graph edges linking knowledge areas to the tasks they enable.

## Requirements

- Python `>=3.10`; CI tests 3.10 through 3.13.
- No GPU or API key is required. A GPU speeds up larger local models; hosted providers need their own API key.
- Supported model providers and their settings are listed in the [documentation](https://laiser-software.github.io/extract-module/providers/).

## Setup and Installation

- Install LAiSER from PyPI:

  ```shell
  pip install laiser
  laiser setup
  ```

- Install with GPU extras:

  ```shell
  pip install "laiser[gpu]"
  laiser setup
  ```

- Install development dependencies from source:

  ```shell
  pip install -e ".[dev]"
  ```

You can check if your machine has a GPU available with:

```shell
python -c "import torch; print(torch.cuda.is_available())"
```

## Usage

LAiSER is used as a Python package. The recommended API is `SkillExtractorRefactored`.

### Without an API key

This runs a small open model locally on CPU. The model downloads on first use.

```python
import pandas as pd

from laiser.skill_extractor_refactored import SkillExtractorRefactored

data = pd.DataFrame(
    [
        {
            "Research ID": "job-001",
            "description": "Build production machine learning systems in Python.",
        }
    ]
)

extractor = SkillExtractorRefactored(model_id="Qwen/Qwen2.5-0.5B-Instruct", use_gpu=False)

results = extractor.extract_concepts(
    data=data,
    id_column="Research ID",
    text_columns=["description"],
    input_type="job_desc",
    concepts=["skills"],
)

print(results[["Raw Concept", "Taxonomy Concept", "Taxonomy Source", "Correlation Coefficient"]])
```

Small local models are convenient for trying LAiSER out; hosted or larger models extract more reliably.

### Job description extraction with Gemini

```python
import os
import pandas as pd

from laiser.skill_extractor_refactored import SkillExtractorRefactored

data = pd.DataFrame(
    [
        {
            "Research ID": "job-001",
            "description": "Build production machine learning systems in Python.",
        }
    ]
)

extractor = SkillExtractorRefactored(
    model_id="gemini",
    api_key=os.getenv("GEMINI_API_KEY") or os.getenv("GOOGLE_API_KEY"),
    use_gpu=False,
)

results = extractor.extract_concepts(
    data=data,
    id_column="Research ID",
    text_columns=["description"],
    input_type="job_desc",
    concepts=["skills", "knowledge", "tasks"],
    allowed_sources=["esco", "onet", "osn", "ukos"],
)

print(results.head())
```

### Course syllabus extraction

```python
import os
import pandas as pd

from laiser.skill_extractor_refactored import SkillExtractorRefactored

data = pd.DataFrame(
    [
        {
            "Research ID": "course-001",
            "description": "Introduction to data visualization and exploratory analysis.",
            "learning_outcomes": "Create dashboards, explain patterns in data, and evaluate charts.",
        }
    ]
)

extractor = SkillExtractorRefactored(
    model_id="gemini",
    api_key=os.getenv("GEMINI_API_KEY") or os.getenv("GOOGLE_API_KEY"),
    use_gpu=False,
)

results = extractor.extract_concepts(
    data=data,
    id_column="Research ID",
    text_columns=["description", "learning_outcomes"],
    input_type="course_syllabi",
    concepts=["skills"],
    allowed_sources=["esco", "onet", "osn", "ukos"],
)

print(results.head())
```

### Common runtime options

| Option | Description |
|---|---|
| `model_id` | `"gemini"`, `"openai"`, or a Hugging Face model id to run locally |
| `api_key` | API key for hosted providers |
| `use_gpu` | run local models on GPU where available |
| `backend` | `"llama_cpp"` to run a local GGUF model |
| `temperature` | decoding temperature for every backend (default `0.0`, greedy) |
| `seed` | seed for backends that accept one (default `42`) |
| `concepts` | `["skills"]` (default), or add `"knowledge"` and `"tasks"` |
| `allowed_sources` | taxonomies to align against: `"esco"`, `"onet"`, `"osn"`, `"ukos"` |
| `top_k` | maximum aligned matches per concept type, per document (default 25) |
| `return_edges` | return `{nodes, edges}` instead of only the results table |
| `output_csv_path` | also write the results to this CSV file |

Every option is described in the [usage guide](https://laiser-software.github.io/extract-module/usage/), and more snippets are in [docs/examples.md](docs/examples.md).

### Reproducible results

Every backend decodes greedily by default (temperature `0.0`) and uses a fixed seed where the backend accepts one, so repeated runs over the same input give the same result. To make results reproducible for others, pin every setting that affects them:

```python
extractor = SkillExtractorRefactored(
    model_id="Qwen/Qwen2.5-0.5B-Instruct",  # pin the model
    use_gpu=False,
    temperature=0.0,  # the default: greedy decoding
    seed=42,  # the default
)

results = extractor.extract_concepts(
    data=data,
    id_column="Research ID",
    text_columns=["description"],
    input_type="job_desc",
    concepts=["skills"],
    allowed_sources=["esco", "onet", "osn", "ukos"],
    similarity_thresholds={"skill": 0.60, "knowledge": 0.50, "task": 0.50},  # the defaults
)
```

When reporting results, also record the LAiSER version: it pins the taxonomy data and the embedding model used for alignment (`sentence-transformers/all-MiniLM-L6-v2`). Hosted providers do not guarantee identical output even at temperature `0.0`. See [Reproducibility](https://laiser-software.github.io/extract-module/reproducibility/) for details.

## Try it in Google Colab

The [cookbook](https://github.com/LAiSER-Software/laiser-cookbook) has complete analyses that open directly in Colab:

| Notebook | |
|---|---|
| Job skill analysis for job seekers | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/Job_Skill_Analysis/Job_Skill_Analysis_for_Job_Seekers.ipynb) |
| University program skill analysis | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/University_Program_Skill_Analysis/University_Program_Skill_Analysis.ipynb) |
| Pay equity analysis | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/Pay_Equity_Analysis/Pay_Equity_Analysis_Based_on_Skills.ipynb) |

## Funding
<div align="center">
<img src="https://laiser.gwu.edu/sites/g/files/zaxdzs6976/files/2026-07/gwu.jpg" width="100px" alt="George Washington University"/>
<img src="https://laiser.gwu.edu/sites/g/files/zaxdzs6976/files/2026-07/gatesfoundation.jpg" width="130px" alt="Gates Foundation"/>
<img src="https://laiser.gwu.edu/sites/g/files/zaxdzs6976/files/2026-07/walmartfoundation.jpg" width="130px" alt="Walmart Foundation"/>
</div>

## Authors
<a href="https://github.com/LAiSER-Software/extract-module/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=LAiSER-Software/extract-module" />
</a>

## Partners
<div align="center">
<img src="https://i.imgur.com/hMb5n6T.png" width="120px"/>
<img src="https://i.imgur.com/dxz2Udo.png" width="70px"/>
<img src="https://i.imgur.com/5O1EuFU.png" width="100px"/>
</div>



</br>
<!-- <p align='center'> <b> Made with Passion💖, Data Science📊, and a little magic!🪄 </b></p> -->
