Metadata-Version: 2.4
Name: jwebs
Version: 1.1.6
Summary: A powerful and advanced web scraping and automation library for Python
Author: J Code
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/JCode-JCode/jwebs
Project-URL: Repository, https://github.com/JCode-JCode/jwebs
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: urllib3>=1.26
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: justhtml>=0.1.0
Requires-Dist: redis>=4.5.0
Provides-Extra: sentiment
Requires-Dist: vaderSentiment; extra == "sentiment"
Provides-Extra: translation
Requires-Dist: deep-translator; extra == "translation"
Provides-Extra: brotli
Requires-Dist: brotli>=1.0.0; extra == "brotli"
Provides-Extra: http2
Requires-Dist: httpx[http2]>=0.27.0; extra == "http2"
Provides-Extra: lxml
Requires-Dist: lxml>=4.9; extra == "lxml"
Provides-Extra: all
Requires-Dist: vaderSentiment; extra == "all"
Requires-Dist: deep-translator; extra == "all"
Requires-Dist: chardet; extra == "all"
Requires-Dist: charset_normalizer; extra == "all"
Requires-Dist: brotli>=1.0.0; extra == "all"
Requires-Dist: httpx[http2]>=0.27.0; extra == "all"
Dynamic: license-file

# jwebs

[![Python Version](https://img.shields.io/pypi/pyversions/jwebs)](https://pypi.org/project/jwebs/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![PyPI version](https://img.shields.io/pypi/v/jwebs)](https://pypi.org/project/jwebs/)
[![PyPI project](https://img.shields.io/badge/PyPI-jwebs-blue)](https://pypi.org/project/jwebs/)
[![Developer](https://img.shields.io/badge/developer-J%20Code-blueviolet)](#)

<br>

> **Version bumped to 1.1.6** — a major update: new no‑compile HTML parsing core, optional success‑request logging, and several dependency/packaging fixes. See [CHANGELOG.md](CHANGELOG.md) for details.

<br>

<img src="docs/images/jwebs-logo.png" alt="jwebs logo">

<br>

**jwebs** is a complete, high‑performance library for web scraping, crawling automation, and content analysis. It supports both HTTP/1.1 and HTTP/2 (user selectable) and includes built‑in caching, rate limiting, robots.txt handling, dynamic proxy rotation, distributed crawling (via Redis), data extraction, content differencing, uptime monitoring, Sitemap/RSS generation, and optional AI‑powered extraction. HTML parsing runs on a fast, no‑compile core (a direct ctypes binding to libxml2's parser and XPath engine, ~1.7x lxml's speed) that installs cleanly on constrained platforms like Android/Termux, with automatic BeautifulSoup fallback. Tested on Python 3.9 – 3.13.

---

## Quick Start – Simple GET Request

```python
from jwebs import JWebs

j = JWebs()
resp = j.GET("https://example.com")
print(f"Status: {resp.status}")
print(f"Content length: {len(resp.text)}")
```

---

## Main Capabilities

**· HTTP** – HTTP/1.1 and HTTP/2 (user selectable), Keep‑Alive, automatic redirects, batch concurrent requests.

**· Fast HTML Parsing** – A no‑compile ctypes binding straight to libxml2's parser and XPath engine (~1.7x lxml's speed), with automatic BeautifulSoup fallback. `lxml` itself is still available as an optional accelerator (`jwebs[lxml]`) for platforms that can compile it.

**· Request Management** – Two‑layer cache (memory + SQLite), rate limiting (Token Bucket), robots.txt respect, session management.

**· Security & Flexibility** – User‑Agent rotation, dynamic proxy rotation, client certificates (mTLS), SSL and security headers checking, HTTP Basic Auth (`auth=`).

**· Crawling & Automation** – Simple crawler and distributed crawler (Redis) that can run across multiple machines.

**· Data Extraction** – Extract text, links, emails, phone numbers, prices, JSON‑LD, meta tags, images, social media links.
· Content Analysis – Sentiment analysis, automatic translation, content differencing (diff).

**· Monitoring** – Uptime monitoring, performance testing (TTFB, page size), SEO and security audits.

**· Logging** – Structured, rotating file/console logging; optionally log successful requests too, not just errors (off by default, toggle anytime with `SET_LOG_SUCCESS()`).

**· Utilities** – Sitemap.xml generator, RSS feed generator, GraphQL client, async client.

**· AI (optional)** – Intelligent data extraction via natural language instructions (DeepSeek/OpenAI) and text summarization.

---

## Installation

```bash
# Basic installation (core dependencies only, Redis client included)
pip install jwebs

# With HTTP/2 support
pip install jwebs[http2]

# With the lxml accelerator (fastest HTML parsing; needs a C compiler,
# so it may not install on Android/Termux -- the fast no-compile core
# above is used automatically wherever lxml isn't available)
pip install jwebs[lxml]

# All optional features
pip install jwebs[all]
```
## Debug

`redis` (the Python client) is installed automatically as a core dependency. You still need the Redis **server** itself running for `DistributedCrawler` -- install it with your package manager:

· Ubuntu/Debian: sudo apt install redis
· Termux (Android): pkg install redis
· macOS: brew install redis

Or download from redis.io

If `pip install jwebs[lxml]` fails to build on Android/Termux (no prebuilt wheel for `lxml` there), you don't need it -- just `pip install jwebs` and the library automatically uses its built‑in no‑compile HTML core instead, at close to the same speed.

---

## More Examples

## HTTP/2 and Caching

```python
from jwebs import JWebs

j = JWebs(http_version='2', use_cache=True)
title = j.GET_TITLE("https://http2.golang.org/")
print(f"Title: {title}")
```
By default, caching is disabled for every HTTP method, even when
`use_cache=True`. You must explicitly opt in per method via
`cache_methods`:

```python
from jwebs import JWebs

# Cache GET responses only
j = JWebs(use_cache=True, cache_methods=['GET'])

# Cache GET and POST responses
j = JWebs(use_cache=True, cache_methods=['GET', 'POST'])

# Cache every supported method
j = JWebs(use_cache=True, cache_methods='all')

# Change it later on an existing instance
j.set_cache(True, methods=['GET', 'PUT'])
```

Caching non-GET methods (POST, PUT, PATCH) can be unsafe if those
requests have side effects (e.g. creating a resource) -- only enable
it for endpoints you know are idempotent. Cache keys for these
methods include a hash of the request body, so different payloads
to the same URL are cached separately.

## Extracting Emails and Links

```python
from jwebs import JWebs

j = JWebs()
emails = j.EXTRACT_EMAILS("https://example.com")
links = j.GET_LINKS("https://example.com", internal=True)
print(f"Emails: {emails}\nInternal Links: {len(links)}")
```

## Distributed Crawling with Redis

```python
from jwebs import JWebs

j = JWebs()
crawler = j.create_distributed_crawler(redis_url="redis://localhost:6379/0")
crawler.add_seed("https://example.com", depth=0)
crawler.crawl_worker(max_pages=10, max_depth=2, strict_page_limit=True)

results = crawler.get_all_results()
for url, info in results.items():
    print(f"{url} → {info.get('title', 'no title')}")
```

## Security Audit

```python
from jwebs import JWebs

j = JWebs()
report = j.SECURITY_AUDIT("https://example.com")
print(f"SSL valid: {report.ssl_valid}")
print(f"Security grade: {report.grade}")
```

## Content Differencing

```python
from jwebs import JWebs

j = JWebs()
snap1 = j.TAKE_SNAPSHOT("version1", "Hello world")
snap2 = j.TAKE_SNAPSHOT("version2", "Hello jwebs")
diff = j.COMPARE_SNAPSHOTS(snap1, snap2)
print(f"Similarity: {j.SIMILARITY('Hello world', 'Hello jwebs')}")
```

## Uptime Monitor

```python
from jwebs import JWebs
import time

j = JWebs()
j.MONITOR_URL("https://example.com", expected_status=200)
j.START_MONITORING()
time.sleep(5)
j.STOP_MONITORING()
```

## Logging

```python
from jwebs import JWebs

j = JWebs(enable_logging=True, log_success=True)  # also logs successful requests, not just errors
resp = j.GET("https://example.com")

j.SET_LOG_SUCCESS(False)  # keep logging on, stop logging successes only
```

---

## Issues and Contributions

You can report bugs via GitHub Issues or submit fixes via pull requests.

---

## Links

**· GitHub repository:**
https://github.com/JCode-JCode/jwebs

**· PyPI page:**
https://pypi.org/project/jwebs/

---

## License

This project is licensed under the Apache License 2.0 – see the LICENSE file for details.

---

Designed and built with love by **J Code❤️**
