Metadata-Version: 2.4
Name: yta-fastapi-docker-llamacpp
Version: 0.0.4
Summary: Youtube Autonomous FastAPI Docker Llama.cpp Module
License-File: LICENSE
Author: danialcala94
Author-email: danielalcalavalera@gmail.com
Requires-Python: >=3.10,<3.14
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: fastapi (>=0.0.1,<9999.0.0)
Requires-Dist: uvicorn (>=0.0.1,<9999.0.0)
Requires-Dist: yta_fastapi_docker_pydantic_models (>=0.0.5,<1.0.0)
Requires-Dist: yta_httpx (>=0.0.27,<1.0.0)
Requires-Dist: yta_programming_env (>=0.3.2,<1.0.0)
Description-Content-Type: text/markdown

# Youtube Autonomous FastAPI Docker Llama.cpp Module

The module that is providing the functionality related to the Llama.cpp models hub (having the models and using them) through a FastAPI that is included and isolated in a Docker container.

This module is meant to be exposed as a container inside the internal network, to be connected with its own FastAPI that is exposing the functionality outside.

### Endpoints

#### GET
No endpoints by now.

#### POST
No endpoints by now.

## Instructions
I've followed these steps to make `llama.cpp` available in my laptop as a container running with cuda, and I've adapted this workflow to this project so its done automatically:

1. I will use a `gemma4` image that is built for CUDA, so I will make sure that I have CUDA properly installed and detected by docker. Execute the following command and it will show your Nvidia GPU if installed and detected: `$docker run --rm --gpus all nvidia/cuda:12.9.1-runtime-ubuntu24.04 nvidia-smi`. If the output is positive, you can go on and use your GPU. Once you've confirmed that you can or cannot, delete the image that has been downloaded with the command. You don't need it.


1. Create a `models` folder in which we will store the models we will use (I will use an external SSD to save space). My folder is `D:/llama/models`.

2. Download the GGUF model that we need (check the specifications of your PC and choose the right one), in the cmd inside the `models` folder you created in the 2nd step:
`$huggingface-cli download unsloth/gemma-4-E2B-it-GGUF gemma-4-E2B-it-UD-Q4_K_XL.gguf --local-dir ./`

1. Download the `llamacpp` version that is using CUDA, being in the cmd inside the `models`:
`$docker run --rm --gpus all -p 8080:8080 -v "${PWD}:/models" ghcr.io/ggml-org/llama.cpp:server-cuda -m  --host 0.0.0.0 -ngl 999`


## Extra
If you download and execute `llamacpp` by itself, you'll a web client like with ChatGPT in which you can have a chat with your agent (it should probably be accesible through the `http://localhost:8080` url). I'm not using it like this, so I wan't it accessible just by my custom endpoints through my APIs.
