Metadata-Version: 2.5
Name: shadow-tools
Version: 0.8.1
Summary: A practical all-in-one Python toolkit for HTTP, WebSockets, files, data, validation, async work, and CLIs.
Project-URL: Homepage, https://github.com/WilliamLaurento/shadow-tools
Project-URL: Issues, https://github.com/WilliamLaurento/shadow-tools/issues
Author: William Laurento II
License: MIT
License-File: LICENSE
Keywords: async,cli,http,tools,utilities,websocket
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: twine>=6.0; extra == 'dev'
Provides-Extra: ml
Requires-Dist: torch>=2.2; extra == 'ml'
Provides-Extra: nlp
Requires-Dist: tokenizers>=0.20; extra == 'nlp'
Provides-Extra: websocket
Requires-Dist: websocket-client>=1.8; extra == 'websocket'
Description-Content-Type: text/markdown

# Shadow Tools

## 0.8.1 generation fix

`ask(bot, question)` returns only the generated answer and stops before a new
`Question:` or `Answer:` line. `teach()` now trains with an end token after each
answer. Retrain existing bots to learn these boundaries. `say()` returns only
new text by default; use `include_prompt=True` to include the input.

Generation accepts `repetition_penalty=1.1` (1.0 disables it). This discourages
repeated tokens; it cannot guarantee original or accurate answers. Training
on a few examples can still memorize those examples. No pretrained weights
or answer lookup are used.

## Train on regular text

```python
from shadowtools import predict_text, train_text

texts = ["I love this game", "this is terrible", "really fun game", "I hate it"]
labels = ["positive", "negative", "positive", "negative"]

model, history = train_text(texts, labels, epochs=20)
print(predict_text(model, ["this game is fun", "I really hate this"]))
# ['positive', 'negative']
```

`train_text` learns a vocabulary from your examples. Its tokenizer keeps words,
contractions, URLs, punctuation, and common two-word phrases, then converts each
text into a TF-IDF number vector before training. Use `SmartTokenizer` yourself
when you want to fit once and reuse the exact same vocabulary later.

## Train a tokenizer for your own Transformer

```python
from shadowtools import BPETokenizer

tokenizer = BPETokenizer(vocab_size=8_000).train(my_training_texts)
batch = tokenizer.batch_encode(["Hello, new world!"], max_length=32)

# Pass these directly to your Transformer.
input_ids = batch["input_ids"]
attention_mask = batch["attention_mask"]
tokenizer.save("my_tokenizer.json")
```

This is byte-level BPE: it learns reusable chunks instead of treating each full
word as a separate thing. Install it with `pip install "shadow-tools[nlp]"`.

## Train and generate text from one package

```python
from shadowtools import learn, say

bot = learn("corpus.txt", epochs=20)
print(say(bot, "The future of technology", max_tokens=80))
bot.save("my_text_model")
```

Training prints the tokenizer status plus each epoch and loss. Use
`learn("corpus.txt", epochs=20, verbose=False)` when you do not want updates.

## Train a question-and-answer bot

```python
from shadowtools import ask, teach

bot = teach({
    "What is ChatGPT?": "ChatGPT is an AI system that predicts useful next tokens to answer a prompt.",
    "How does a tokenizer work?": "A tokenizer splits text into smaller tokens a model can process.",
    "What is machine learning?": "Machine learning is when a model learns patterns from examples.",
}, epochs=20)

print(ask(bot, "What is ChatGPT?"))
```

Add many more real Q&A pairs for better answers.

Install it with `pip install "shadow-tools[ml,nlp]"`. The package trains the
tokenizer and Transformer together, then converts predicted token IDs back to
normal text for you.

`shadow-tools` is a lightweight Python toolbox for everyday apps and scripts. It keeps the basics in one clean package without trying to replace specialized libraries.

## Install

```bash
pip install shadow-tools
pip install "shadow-tools[websocket]"  # optional WebSocket client
```

## What is included

- HTTP requests with no required third-party dependency
- WebSocket connections through an optional extra
- Async execution for blocking jobs
- File and directory helpers
- JSON and CSV reading/writing
- Email, URL, and required-key checks
- DNS lookup, subdomain parsing, and documentation-only test IPs
- Provider-neutral AI prompt and tool-schema helpers
- File and text hashing
- UUIDs, secure tokens, and timestamp IDs
- Slugs, snake/camel case, chunks, environment variables, and retry logic
- Defensive security checks, secret redaction, password generation, and file verification
- Optional PyTorch training for new MLPs or supplied local models
- `shadow` command-line tool

## Quick start

```python
from shadowtools import get, write_json, is_email

response = get("https://api.github.com")
print(response.status, response.json())
write_json("output.json", {"ready": True})
assert is_email("person@example.com")
```

## DNS, subdomains, and test IPs

```python
from shadowtools import generate_random_ips, generate_test_ips, make_subdomain, resolve_dns

print(make_subdomain("api", "example.com"))  # api.example.com
print(resolve_dns("example.com"))
print(generate_test_ips(2))  # safe documentation-only addresses
print(generate_random_ips(5))  # globally routable public IPv4 addresses only
print(generate_random_ips(3, family=6))
print(generate_random_ips(3, include_private=True, include_special=True))
```

Reverse-DNS lookup for IP addresses:

```python
from shadowtools import generate_random_ips, reverse_dns

for result in reverse_dns(generate_random_ips(5)):
    print(result)  # only IPs with hostnames are shown

# Or look up every generated IP at once.
print(reverse_dns(generate_random_ips(5)))
print(reverse_dns(generate_random_ips(5), only_hostnames=False))  # include misses
```

## AI app helpers

```python
from shadowtools.ai import prompt, tool_schema

message = prompt("Explain {topic} in {words} words.", topic="DNS", words=50)
search_tool = tool_schema(
    "search_docs", "Search internal documentation",
    {"query": {"type": "string"}}, required=["query"]
)
```

## Everyday helpers

```python
from shadowtools import hash_text, random_token, slugify, to_bool, uuid4

print(hash_text("hello"))
print(random_token())
print(uuid4())
print(slugify("Shadow Tools: Version 3!"))  # shadow-tools-version-3
print(to_bool("yes"))  # True
```

```bash
shadow hash settings.json
shadow uuid
```

## Security and training

```python
from shadowtools.security import audit_headers, redact_secrets

print(redact_secrets("Authorization: Bearer secret-value"))
print(audit_headers({"X-Content-Type-Options": "nosniff"}))
```

Install ML support, then train a small classifier from scratch:

```bash
pip install "shadow-tools[ml]"
```

```python
from shadowtools.training import Trainer, TrainingConfig

config = TrainingConfig(
    mode="from_scratch",
    architecture={"type": "mlp", "input_size": 8, "hidden_sizes": [32, 16], "num_classes": 2},
    epochs=5,
)
trainer = Trainer(config)
history = trainer.fit(train_loader, validation_loader)
```

For normal lists, use the short root-level API—no PyTorch imports or loaders:

```python
from shadowtools import train

model, history = train(
    features=[[0, 1], [1, 1], [0, 0], [1, 0]],
    labels=[1, 1, 0, 0],
    epochs=10,
)
print(history)
```

```bash
shadow get https://api.github.com --json
shadow json-format settings.json
```

## WebSockets

```python
from shadowtools.websocket import connect

socket = connect("wss://example.com/socket")
socket.send("hello")
print(socket.recv())
socket.close()
```

## Release

Build locally with `python -m build`, then publish only after reviewing the generated distributions and confirming the PyPI project name is available.

## License

MIT
