Metadata-Version: 2.4
Name: JobSelect
Version: 0.11.2
Summary: A CLI Based AI Skill Classifier for Job Descriptions Build by Akshay Babu
Author-email: Akshay Babu <akshaysureshbabu100@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Ak47xdd/Job-Description-Analysis
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rich==15.0.0
Requires-Dist: pyfiglet==1.0.4
Requires-Dist: scikit-learn==1.9.0
Requires-Dist: uvicorn==0.50.2
Requires-Dist: pydantic==2.13.4
Requires-Dist: requests==2.34.2
Requires-Dist: torch==2.12.1
Dynamic: license-file

# Job Description Skill Classifier [JobSelect v0.10.5 & JobAnalyze 6k v1.0] (Multi-Label)

[![Python](https://img.shields.io/badge/Python-3.8+-3776AB?style=flat&logo=python)](https://www.python.org/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.12.1-%23EE4C2C?style=flat&logo=pytorch)](https://pytorch.org/)
[![scikit-learn](https://img.shields.io/badge/scikit--learn-1.9.0-F7931E?style=flat&logo=scikit-learn)](https://scikit-learn.org/)
[![NumPy](https://img.shields.io/badge/NumPy-2.4.6-013243?style=flat&logo=numpy)](https://numpy.org/)
[![Pandas](https://img.shields.io/badge/Pandas-3.0.3-150458?style=flat&logo=pandas)](https://pandas.pydata.org/)

This project builds a lightweight text classification pipeline that predicts **multiple technical skills** from a job posting. Given a job description (optionally augmented with role and job type), the model outputs a ranked list of likely skills.

Installation:

- `pip install jobselect`

It uses:

- **TF-IDF** features over the combined text (job description + role + type)
- A **PyTorch feed-forward neural network** trained as a **multi-label** classifier
- **Per-skill thresholding** for evaluation and **top-k ranked probabilities** for inference

---

## What it does

1. **Data preparation** (`model/prep/data_prep.py`)
   - Reads cleaned job description data.
   - Normalizes/repairs common skill typos (e.g., `tesnorflow/pytorch` → `tensorflow/pytorch`).
   - Builds a **multi-hot** target vector of skills.
   - Fits a **TF-IDF** vectorizer (with n-grams) and splits into train/test.
   - Saves:
     - `model/prep/prepared_data.npz` (TF-IDF arrays + labels + indexes)
     - `model/prep/vectorizer.pkl` (fitted TF-IDF vectorizer)
     - `model/prep/label_vocab.json` (skill label vocabulary)

2. **Model training** (`model/model.py`)
   - Loads prepared TF-IDF arrays.
   - Defines a simple **MLP**:
     - Linear → ReLU → Dropout → Linear (one logit per skill)
   - Trains with `BCEWithLogitsLoss` (multi-label setting).
   - Saves:
     - `model_out/skill_classifier.pt` (model weights)
     - `model_out/training_history.json` (train/test loss curves)

3. **Evaluation** (`model/eval.py`)
   - Loads the trained model.
   - Applies a fixed sigmoid + threshold (**0.5**) to obtain binary skill predictions.
   - Reports:
     - Per-skill precision/recall/F1
     - Micro-F1 and Macro-F1
   - Compares against a simple baseline (frequency-driven / always-predict-most-frequent labels).

4. **Prediction / Inference** (`model/pred.py`)
   - Loads the TF-IDF vectorizer and trained model.
   - Creates TF-IDF features for the input text.
   - Outputs the **top-k** skills by probability.

---

## Use cases

- **Resume/job-post matching** (first-pass filtering of relevant skills)
- **Job taxonomy building** (discover recurring skills from postings)
- **Recruiting analytics** (aggregate predicted skill demand by seniority/role/type)
- **Prototyping multi-label NLP classifiers** (TF-IDF + MLP baseline)

---

## Project structure

```text
.
├─ data/
│  ├─ raw/
│  │  └─ Job_descriptions.csv                 # Raw input dataset
│  ├─ sample_data/
│  │  └─ test.txt                             # Example JD, Role and Type for testing
│  └─ clean/
│     ├─ cleaned_job_descriptions.csv         # Cleaned master CSV
│     ├─ cleaned_job_descriptions_internships.csv
│     ├─ cleaned_job_descriptions_junior.csv
│     └─ cleaned_job_descriptions_senior.csv
│
├─ model/
│  ├─ prep/
│  │  ├─ data_prep.py                         # TF-IDF + multi-hot label creation + train/test split
│  │  ├─ sym_map.py                           # Synonym/phrase normalization map used during prep
│  │  ├─ vectorizer.pkl                       # Saved TF-IDF vectorizer (generated by data_prep)
│  │  ├─ prepared_data.npz                    # Saved arrays (generated by data_prep)
│  │  └─ label_vocab.json                     # Skill label vocabulary (generated by data_prep)
│  │
│  ├─ model.py                                # PyTorch multi-label classifier training
│  ├─ eval.py                                 # Thresholded evaluation + F1 metrics + baseline comparison
│  └─ pred.py                                 # Predict top-k skills for new text
│
├─ model_out/
│  ├─ skill_classifier.pt                     # Trained model weights (generated by model.py)
│  └─ training_history.json                   # Training loss history (generated by model.py)
│
├─ cli/
│  └─ jobselect.py                              # Rich terminal CLI for interactive prediction (uses model.pred.predict)
│
├─ test/
│  └─ test_model.py                           # Pytest checks expected artifacts exist in model_out/ and model/prep/
│
├─ notebooks/
│  ├─ 01_EDA.ipynb                            # Exploratory Data Analysis
│  └─ 02_Data_Engineering.ipynb               # Data engineering / cleaning notes
│
├─ pipeline.py                                # Executes notebooks + training/eval steps in order
├─ pyproject.toml                             # Installs as a cli tool
├─ requirements.txt
└─ README.md
```

---

## Requirements

See `requirements.txt` for the exact dependencies.

---

## Getting started

### 1) Clone and Install dependencies

```bash
git clone https://github.com/Ak47xdd/Job-Description-Analysis.git
pip install -r requirements.txt
```

### 2) Run data preparation (optional)

This builds the TF-IDF features and label vocabulary from the cleaned CSV.

```bash
python model/prep/data_prep.py
```

Expected outputs:

- `model/prep/prepared_data.npz`
- `model/prep/vectorizer.pkl`
- `model/prep/label_vocab.json`

### 3) Train the model (optional)

```bash
python model/model.py
```

Expected outputs:

- `model_out/skill_classifier.pt`
- `model_out/training_history.json`

### 4) Evaluate performance (optional)

```bash
python model/eval.py
```

Outputs include:

- Per-skill metrics (precision/recall/F1)
- Micro-F1 and Macro-F1
- Baseline comparison

### 5) Predict skills for a new job description

#### Option A: Use Python function

`from model.pred import predict`

`data/sample_data/test.txt` contains an example job description inside. You can also call the `predict(job_desc, role=..., job_type=..., top_k=...)` function from Python.

#### Option B: Use the interactive CLI

```bash
python cli/jobselect.py

or

pip install jobselect
jobselect
```

The CLI prompts for job description, role, and job type, then prints the top skills with probabilities.

---

## How predictions work

- Text is concatenated as:
  `"{job_desc} {role} {job_type}"`
- TF-IDF transforms text into a fixed-size vector
- The network outputs one logit per skill
- Sigmoid converts logits → probabilities
- Skills are ranked by probability and the top-k are returned

---

## Important implementation notes

- **Multi-label learning:** Each skill is treated independently (binary relevance via sigmoid + `BCEWithLogitsLoss`).
- **Evaluation threshold:** `model/eval.py` uses a fixed threshold of **0.5**. For production use, you may want per-label thresholds tuned on a validation set.
- **Dataset size:** The included notebooks and evaluation code suggest the dataset may be small; results can be limited by label frequency and data coverage.

---

## New features / capabilities

- **Rich terminal CLI** (`cli/jobauto.py`) using `rich` + `pyfiglet` for interactive top-skill display.
- **Synonym/phrase normalization hook** (`model/prep/sym_map.py`) applied during data preparation.
- **Pipeline runner** (`pipeline.py`) to execute notebooks and training steps in sequence.

---

## Customization ideas

- Improve text cleaning and skill normalization in `data_prep.py`
- Tune TF-IDF parameters (`max_features`, `ngram_range`, `min_df`)
- Replace the simple MLP with a stronger baseline (e.g., logistic regression on TF-IDF)
- Calibrate thresholds per label using validation data
- Add a CLI or web service endpoint for prediction

---

## References / Inspiration

This repository follows a common pattern for multi-label NLP baselines:
TF-IDF features + a simple neural network + sigmoid-based multi-label outputs.
