Metadata-Version: 2.4
Name: langchain-nvidia-ai-endpoints
Version: 1.5.0
Summary: An integration package connecting NVIDIA AI Endpoints and LangChain
License: MIT
License-File: LICENSE
Requires-Python: >=3.10.0,<4.0.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: aiohttp (>=3.13.3,<3.15.0)
Requires-Dist: langchain-core (>=1.4.7,<2.0.0)
Requires-Dist: requests (>=2.28.0)
Project-URL: Repository, https://github.com/langchain-ai/langchain-nvidia
Project-URL: Source Code, https://github.com/langchain-ai/langchain-nvidia/tree/main/libs/ai-endpoints
Description-Content-Type: text/markdown

# LangChain NVIDIA AI Endpoints

The `langchain-nvidia-ai-endpoints` package contains LangChain integrations for chat models and embeddings powered by [NVIDIA AI Foundation Models](https://www.nvidia.com/en-us/ai-data-science/foundation-models/), and hosted on the [NVIDIA API Catalog](https://build.nvidia.com/).

A strong starting point is [Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/), NVIDIA's open model family built for agentic AI. Nemotron models use a hybrid Mamba-Transformer mixture-of-experts architecture that delivers leading benchmark accuracy with high throughput and up to 1M token context windows. Model weights, training data, and implementation recipes are published openly under the NVIDIA Open Model License.

NVIDIA AI Foundation models are community- and NVIDIA-built models that are optimized to deliver the best performance on NVIDIA-accelerated infrastructure. 
You can use the API to query live endpoints that are available on the NVIDIA API Catalog to get quick results from a DGX-hosted cloud compute environment. 
or you can download models from NVIDIA's API catalog with NVIDIA NIM, which is included with the NVIDIA AI Enterprise license. 
The ability to run models on-premises gives your enterprise ownership of your customizations and full control of your IP and AI application. 

NIM microservices are packaged as container images on a per model/model family basis 
and are distributed as NGC container images through the [NVIDIA NGC Catalog](https://catalog.ngc.nvidia.com/). 
At their core, NIM microservices are containers that provide interactive APIs for running inference on an AI Model. 

Use this documentation to learn how to install the `langchain-nvidia-ai-endpoints` package 
and use it for some common functionality for text-generative and embedding models.


## Get Started

### Install langchain-nvidia-ai-endpoints

To install the `langchain-nvidia-ai-endpoints` package, use the following code.

```python
%pip install -U --quiet langchain-nvidia-ai-endpoints
```


### Get Access to the NVIDIA API Catalog

To get access to the NVIDIA API Catalog, do the following:

1. Create a free account on the [NVIDIA API Catalog](https://build.nvidia.com/) and log in.
2. Click your profile icon, and then click **API Keys**. The **API Keys** page appears.
3. Click **Generate API Key**. The **Generate API Key** window appears.
4. Click **Generate Key**.  You should see **API Key Granted**, and your key appears.
5. Copy and save the key as `NVIDIA_API_KEY`.
6. To verify your key, use the following code.

    ```python
    import getpass
    import os

    if not os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
        nvidia_api_key = getpass.getpass("Enter your NVIDIA API key: ")
        assert nvidia_api_key.startswith("nvapi-"), f"{nvidia_api_key[:5]}... is not a valid key"
        os.environ["NVIDIA_API_KEY"] = nvidia_api_key
    ```

You can now use your key to access endpoints on the NVIDIA API Catalog.

### Choose a deployment and check it before inference

Hosted NVIDIA API Catalog calls default to `https://integrate.api.nvidia.com/v1`
and **require** `NVIDIA_API_KEY`. Set it in your environment (do not put it in
source code or a URL). A self-hosted NIM can use its deployment root
(`http://localhost:8000`) or its OpenAI-compatible API base
(`http://localhost:8000/v1`); both resolve to `/v1`. Set `NVIDIA_BASE_URL`
or pass `base_url=` to `ChatNVIDIA`, `NVIDIAEmbeddings`, `NVIDIA` (completions),
or `NVIDIARerank`. An explicit constructor `base_url` takes precedence over
`NVIDIA_BASE_URL`. Custom reverse-proxy prefixes are retained (for example,
`https://proxy.example/nim` becomes `https://proxy.example/nim/v1`).
Do **not** pass `/models`, `/chat/completions`, `/embeddings`, `/completions`,
`/ranking`, `/rankings`, or `/reranking` as the base URL: those are endpoint
paths, not deployment roots. The connector uses `GET <base>/models` for
discovery and appends `/chat/completions`, `/embeddings`, `/completions`, or
`/ranking` for the corresponding client. Self-hosted deployments may accept
anonymous access, but set `NVIDIA_API_KEY` if yours requires bearer auth.
Hosted catalog models with dedicated endpoints may not appear in `/models`;
their connector routing depends on the known hosted-model registry.

Run the read-only Doctor preflight before sending prompts:

```bash
python -m langchain_nvidia_ai_endpoints doctor --model nvidia/nemotron-3-super-120b-a12b --capability chat
# Self-hosted NIM; no API key unless your deployment requires one
python -m langchain_nvidia_ai_endpoints doctor --base-url http://localhost:8000/v1 --model my-model --capability embeddings
```

Doctor uses `NVIDIA_BASE_URL` (or the hosted default), `NVIDIA_API_KEY`, and
`GET /v1/models`; it sends **no inference request or usage telemetry**.
`--capability` accepts `chat`, `embeddings`, `completions`, or `ranking`.
It checks `/models` model type metadata where available; unknown type is
reported as **UNKNOWN**, not verified by a generated response. It exits 0
when connectivity/auth/model listing checks pass and nonzero for missing
hosted auth, rejected keys, bad URLs, unavailable `/models`, missing models,
incompatible declared model types, network failures, or timeouts. Check
`--help` for options; use `--timeout 10` for a slow deployment. A preflight
pass is not proof that inference succeeds or that a framework is BCB-compatible.

Optional BCB compatibility evidence is available from NIM OSS Manager's
**authenticated internal-alpha** API (not a public anonymous service).
It is never queried by default. If your team has approved, scoped read-only
Manager consumer access, set `NIM_OSS_MANAGER_TOKEN` in the environment and opt in explicitly:

```bash
python -m langchain_nvidia_ai_endpoints doctor --model nvidia/nemotron-3-super-120b-a12b --capability chat --manager-url https://your-manager-host
```

Doctor requests the Manager's read-only `bcb-public-v1alpha1` compatibility
summary for `langchain-nvidia` and that model. It does not transmit your
NVIDIA API key to Manager. Missing access, unknown responses, and stale
evidence are labeled **unavailable**, **unknown**, or **stale**, never
invented as compatible. An unavailable optional Manager lookup does not
turn a working endpoint into a failed connectivity check. Do not put
Manager tokens in command-line arguments or URLs.

For a maintainer-reviewed, framework-level compatibility snapshot, see
[NIM compatibility evidence](docs/compatibility.md). The optional BCB
review artifact separates verified framework status from unknown individual
capabilities; it is not a live release approval or a public API.

For hosted-versus-local setup, streaming/tools, embeddings retrieval and
migration from generic OpenAI-compatible clients, see the
[adoption guide](docs/adoption_guide.md), including its scoped hosted recipe
smoke results. Capabilities and compatibility still require deployment-specific evidence.

For the documented gaps in OCR, page-elements, table-structure, and offline
ASR support (API shapes, proposed abstractions, and future validation cases),
see the [non-text NIM modality guidance](docs/non_text_nim_modalities.md).
This is a maintainer proposal, not implemented modality support.

Connector maintainers: the [reliability queue](docs/reliability_backlog.md)
groups open issues by root cause, evidence, owner role, and next release or
backlog decision; pending fixes are not claimed as released.


## Invoke the Core Chat Interface

Use the following code to invoke the core chat interface.

```python
## Core LC Chat Interface
from langchain_nvidia_ai_endpoints import ChatNVIDIA

llm = ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b")
result = llm.invoke("Write a ballad about LangChain.")
print(result.content)
```


## Use Stream, Batch, and Async

The models exposed by the NVIDIA API natively support streaming, and they expose a batch method to handle concurrent requests, as well as async methods for invoke, stream, and batch. 

The following examples demonstrate how to use batch and stream, and their async versions.


```python
# Batch example
print(llm.batch(["What's 2*3?", "What's 2*6?"]))

# Batch example (async)
await llm.abatch(["What's 2*3?", "What's 2*6?"])

# Stream example
for chunk in llm.stream("How far can a seagull fly in one day?"):
    # Show the token separations
    print(chunk.content, end="|")

# Stream example (async)
async for chunk in llm.astream("How long does it take for monarch butterflies to migrate?"):
    print(chunk.content, end="|")
```


## Get a List of Supported Models

You can query `available_models` to get a list of the models that you can access with your API credentials. 
Use the following code.

```python
[model.id for model in llm.available_models if model.model_type]
```

You should see output similar to the following.

```python
#[
# ...
# 'databricks/dbrx-instruct',
# 'google/codegemma-7b',
# 'google/gemma-2b',
# 'google/gemma-7b',
# 'google/recurrentgemma-2b',
# 'meta/codellama-70b',
# 'meta/llama2-70b',
# 'meta/llama3-70b-instruct',
# 'meta/llama3-8b-instruct',
# 'microsoft/phi-3-mini-128k-instruct',
# 'mistralai/mistral-7b-instruct-v0.2',
# 'mistralai/mistral-large',
# 'mistralai/mixtral-8x22b-instruct-v0.1',
# 'mistralai/mixtral-8x7b-instruct-v0.1',
# 'snowflake/arctic',
# ...
#]
```

> [!TIP]
> To find out more about a specific model, on the [Models page](https://build.nvidia.com/models), search for the name of the model (without the company and `/`), click the model, and then click **Model Card**.



## Work With Different Model Types

Some model types support unique prompting techniques and chat messages. 
Use this section to learn about a few examples.


### Nemotron

[Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) is NVIDIA's open model family optimized for agentic AI. The models deliver strong reasoning and tool-calling performance with high throughput. `nvidia/nemotron-3-super-120b-a12b` is a recommended starting point: a powerful model with 12B active parameters and 120B parameters in total, with a 1M token context window, available on the NVIDIA API Catalog and as a self-hosted NIM.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA

llm = ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b")
result = llm.invoke("Plan a three-step agentic workflow for competitive research.")
print(result.content)
```


### Nemotron 3.5 Super VL

Nemotron 3.5 Super VL accepts text and images through `ChatNVIDIA`.

The connector defaults this model to 8,192 completion tokens. Set a thinking
preference as needed; pass `max_completion_tokens` to override the default:

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA

super_model = ChatNVIDIA(
    model="nvidia/nemotron-3.5-super-vl-120b-a12b",
    chat_template_kwargs={"enable_thinking": True},
)
result = super_model.invoke("Plan a three-step agentic workflow for competitive research.")
print(result.content_blocks)
```

To describe an image, replace `photo.png` with a local PNG and pass a standard
image content block:

```python
import base64
from pathlib import Path

from langchain_core.messages import HumanMessage

image = base64.b64encode(Path("photo.png").read_bytes()).decode("ascii")
response = super_model.invoke([
    HumanMessage(content=[
        {"type": "text", "text": "Describe this image."},
        {"type": "image", "base64": image, "mime_type": "image/png"},
    ])
])
print(response.content_blocks)
```

### General Chat

Models such as `nvidia/nemotron-3-super-120b-a12b` and `mistralai/mixtral-8x22b-instruct-v0.1`
are good all-around models that you can use for any LangChain chat messages.

The following example generates a simple chat response.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful AI assistant named Fred."),
        ("user", "{input}")
    ]
)
chain = (
    prompt
    | ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b")
    | StrOutputParser()
)

for txt in chain.stream({"input": "What's your name?"}):
    print(txt, end="")
```


### Code Generation

Code generation models, such as `meta/codellama-70b` and `google/codegemma-7b`, 
tend to perform better on code-generation and structured code tasks. 

The following example generates python code to solve a problem.

```python
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are an expert coding AI. Respond only in valid python; no narration whatsoever."),
        ("user", "{input}")
    ]
)
chain = (
    prompt
    | ChatNVIDIA(model="meta/codellama-70b", max_completion_tokens=419)
    | StrOutputParser()
)

for txt in chain.stream({"input": "How do I solve this fizz buzz problem?"}):
    print(txt, end="")
```


### Multimodal Support

NVIDIA also has models that support multimodal inputs, such as `nvidia/neva-22b`. 
You can provide both images and text for the model to reason over. 
These models accept LangChain's standard image formats. 
Local file paths are no longer accepted directly. If you are upgrading code that
passed local paths, read trusted local files explicitly and pass them as
`data:image/...;base64,...` URIs.

The following example asks the model to describe an image.

```python
import requests
import base64

# Initialize the image content
image_url = "https://picsum.photos/seed/kitten/300/200"
image_content = requests.get(image_url).content

# Initialize the model
from langchain_nvidia_ai_endpoints import ChatNVIDIA
llm = ChatNVIDIA(model="nvidia/neva-22b")

# Pass an image as a URL
from langchain_core.messages import HumanMessage

llm.invoke(
    [
        HumanMessage(content=[
            {"type": "text", "text": "Describe this image:"},
            {"type": "image_url", "image_url": {"url": image_url}},
        ])
    ])

# Pass an image as a base64 encoded string
b64_string = base64.b64encode(image_content).decode('utf-8')
llm.invoke(
    [
        HumanMessage(content=[
            {"type": "text", "text": "Describe this image:"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_string}"}},
        ])
    ])
```


The NVIDIA API uniquely accepts images as base64 images within HTML `img` tags. 
While this isn't interoperable with other LLMs, you can prompt the model accordingly 
as shown in the following example.

```python
base64_with_mime_type = f"data:image/png;base64,{b64_string}"
llm.invoke(
    f"What's in this image?\n<img src=\"{base64_with_mime_type}\" />"
)
```


### Completions

You can work with models that support the Completions API. 
These models accept a `prompt` instead of `messages`.

The following example gets a list of models that support the Completions API.

```python
completions_llm = NVIDIA().bind(max_tokens=512)
[model.id for model in completions_llm.get_available_models()]

# [
#   ...
#   'bigcode/starcoder2-7b',
#   'bigcode/starcoder2-15b',
#   ...
# ]
```

The following example uses the Completions API to generate a code example.

```python
prompt = "# Function that does quicksort written in Rust without comments:"
for chunk in completions_llm.stream(prompt):
    print(chunk, end="", flush=True)
```


### Dynamo KV Cache Optimization

`ChatNVIDIADynamo` is a drop-in replacement for `ChatNVIDIA` that injects NVIDIA Dynamo KV cache routing hints into requests. Use it with Dynamo-enabled deployments to improve inference scheduling.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIADynamo

llm = ChatNVIDIADynamo(model="nvidia/nemotron-3-super-120b-a12b", osl=1024, iat=100)
result = llm.invoke("Write a short poem.")
```


### Embeddings

The following example connects to an embeddings model.

```python
from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings

embedder = NVIDIAEmbeddings(model="NV-Embed-QA")
embedder.embed_query("What's the temperature today?")
embedder.embed_documents([
    "The temperature is 42 degrees.",
    "Class is dismissed at 9 PM."
])
```

### Ranking

The following example connects to a ranking model.

```python
from langchain_nvidia_ai_endpoints import NVIDIARerank
from langchain_core.documents import Document

query = "What is the GPU memory bandwidth of H100 SXM?"
passages = [
    "The Hopper GPU is paired with the Grace CPU using NVIDIA's ultra-fast chip-to-chip interconnect, delivering 900GB/s of bandwidth, 7X faster than PCIe Gen5. This innovative design will deliver up to 30X higher aggregate system memory bandwidth to the GPU compared to today's fastest servers and up to 10X higher performance for applications running terabytes of data.",
    "A100 provides up to 20X higher performance over the prior generation and can be partitioned into seven GPU instances to dynamically adjust to shifting demands. The A100 80GB debuts the world's fastest memory bandwidth at over 2 terabytes per second (TB/s) to run the largest models and datasets.",
    "Accelerated servers with H100 deliver the compute power—along with 3 terabytes per second (TB/s) of memory bandwidth per GPU and scalability with NVLink and NVSwitch™.",
]

client = NVIDIARerank(model="nvidia/llama-3.2-nv-rerankqa-1b-v1")

response = client.compress_documents(
  query=query,
  documents=[Document(page_content=passage) for passage in passages]
)

print(f"Most relevant: {response[0].page_content}\nLeast relevant: {response[-1].page_content}")
```

Reranked results are separate `Document` objects with `metadata["relevance_score"]`.
The input documents and earlier result scores remain unchanged across subsequent
sync or async calls, so callers can safely reuse the same source documents.

### Ranking with a Vision-Language Rerank Model

`NVIDIARerank` can also rerank documents that carry an image alongside their text using a ranking VLM such as
[nvidia/llama-nemotron-rerank-vl-1b-v2](https://build.nvidia.com/nvidia/llama-nemotron-rerank-vl-1b-v2).

The following example connects to a ranking VLM.

```python
from langchain_nvidia_ai_endpoints import NVIDIARerank
from langchain_core.documents import Document

query = "Show me a picture of a cat"
image = "data:image/png;base64,<base64-encoded-image>"
documents = [
    Document(
        page_content="The picture of a cat.",
        metadata={"image": image},
    ),
    Document(
        page_content="",
        metadata={"image": image},
    ),
    Document(page_content="The weather today is sunny."),
]

ranker = NVIDIARerank(model="nvidia/llama-nemotron-rerank-vl-1b-v2")
response = ranker.compress_documents(query=query, documents=documents)
```

### Retrieval

The following example connects to a running NVIDIA RAG Blueprint server to retrieve relevant documents via the `/v1/search` endpoint.

```python
from langchain_nvidia_ai_endpoints import NVIDIARetriever

retriever = NVIDIARetriever(base_url="http://localhost:8081", k=4)
docs = retriever.invoke("What is machine learning?")
```


## Self-host with NVIDIA NIM Microservices

When you are ready to deploy your AI application, you can self-host models with NVIDIA NIM. 
For more information, refer to [NVIDIA NIM Microservices](https://www.nvidia.com/en-us/ai-data-science/products/nim-microservices/).

The following code connects to locally hosted NIM Microservices.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank

# Connect to an chat NIM running at localhost:8000, and specify a model
llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="nvidia/nemotron-3-super-120b-a12b")

# Connect to an embedding NIM running at localhost:8080
embedder = NVIDIAEmbeddings(base_url="http://localhost:8080/v1")

# Connect to a reranking NIM running at localhost:2016
ranker = NVIDIARerank(base_url="http://localhost:2016/v1")
```

If your deployment exposes a direct inference URL instead of a `/v1/models`
listing plus a standard inference path, register it as a model endpoint rather
than passing the inference URL as `base_url`. A 404 from `base_url` plus
`/embeddings` usually means the deployment URL is already the full inference
endpoint.

```python
import os

from langchain_nvidia_ai_endpoints import Model, NVIDIAEmbeddings, register_model

register_model(
    Model(
        id="nvidia/llama-3.2-nv-embedqa-1b-v2",
        model_type="embedding",
        client="NVIDIAEmbeddings",
        endpoint="https://CUSTOM_DEPLOYMENT_LINK_WITHOUT_TRAILING_EMBEDDINGS",
    )
)

embedder = NVIDIAEmbeddings(
    model="nvidia/llama-3.2-nv-embedqa-1b-v2",
    api_key=os.environ.get("NVIDIA_API_KEY"),
)
```

---

## Telemetry & Privacy

`langchain-nvidia-ai-endpoints` includes content-free, aggregate telemetry to
help NVIDIA understand adoption and reliability of supported NVIDIA NIM
integrations. Telemetry is enabled by default for supported NVIDIA-hosted NIM
endpoint usage and can be disabled at any time.

Telemetry is limited to aggregate operational metrics such as connector and
framework version, operation type, recognized model family, request and outcome
counts, token counts, latency buckets, and coarse error categories. These totals
are best-effort and are not billing-grade.

The telemetry is aggregate-only and does not include request or response content,
credentials, endpoint URLs, or persistent user, device, or session identifiers.
It is not used to track individual users, for billing, or for precise capacity
accounting.

Telemetry is aggregated in memory and emitted only for supported NVIDIA-hosted
NIM endpoints. Self-hosted endpoints do not emit this telemetry.

You may opt out of telemetry collection at any time. Opting out applies only to
telemetry collection by the `langchain-nvidia-ai-endpoints` connector itself.
To disable telemetry for the current shell, set
`NVIDIA_USAGE_TELEMETRY_ENABLED=false` before running your application:

```bash
export NVIDIA_USAGE_TELEMETRY_ENABLED=false
```

You can also disable telemetry for a single client instance:

```python
llm = ChatNVIDIA(
    model="nvidia/nemotron-3-super-120b-a12b",
    usage_telemetry_enabled=False,
)
```

If telemetry is disabled, the connector does not send telemetry requests and
telemetry has no effect on connector operation.

**Use of third-party endpoints, including NVIDIA Build:** `langchain-nvidia-ai-endpoints`
can be configured to use various inference endpoints, including
[build.nvidia.com](https://build.nvidia.com/) (NVIDIA Build). If you choose to
use NVIDIA Build or any other third-party endpoint, that endpoint's own terms of
service and privacy practices apply independently of this library. Any opt-out
you exercise within `langchain-nvidia-ai-endpoints` does not extend to data
collection by your chosen endpoint. NVIDIA Build is intended for evaluation and
testing purposes only and may not be used in production environments. Do not
submit confidential information or personal data when using NVIDIA Build.

