Sources

Sources #

Source nodes represent external tables in a database or data warehouse and make up the foundational layer on which other nodes are built upon.

Attribute Description Type
name Unique name used by other nodes to select from this node string
description A human readable description of the node string
mode published or draft (see Node Mode) string
catalog The name of the external catalog string
schema_ The name of the external schema string
table The name of the external table string
columns A map of the external table’s column names and types map

Creating Source Nodes #

Source nodes can be created by making a POST request to /nodes/source/.

curl -X POST http://localhost:8000/nodes/source/ \
-H 'Content-Type: application/json' \
-d '{
    "name": "repair_orders",
    "description": "Repair orders",
    "mode": "published",
    "catalog": "default",
    "schema_": "roads",
    "table": "repair_orders",
    "columns": {
        "repair_order_id": {"type": "int"},
        "municipality_id": {"type": "string"},
        "hard_hat_id": {"type": "int"},
        "order_date": {"type": "timestamp"},
        "required_date": {"type": "timestamp"},
        "dispatched_date": {"type": "timestamp"},
        "dispatcher_id": {"type": "int"}
    }
}'
from djclient import DJ, Source

client = DJ("http://localhost:8000/")
client.push(
    Source(
        name="repair_orders",
        description="Repair orders",
        mode="published",
        catalog="default",
        schema_="roads",
        table="repair_orders",
        columns={
            "repair_order_id": {"type": "int"},
            "municipality_id": {"type": "string"},
            "hard_hat_id": {"type": "int"},
            "order_date": {"type": "timestamp"},
            "required_date": {"type": "timestamp"},
            "dispatched_date": {"type": "timestamp"},
            "dispatcher_id": {"type": "int"}
        },
    )
)

Automatic Column Reflection Using DJRS #

To make creating and maintaining source nodes easier, you can optionally run a DataJunction Reflection Service (DJRS). A reflection service automatically adds the column names and types to all source nodes and updates them whenever the external table changes. This allows you to exclude the columns when adding a source node.

curl -X POST http://localhost:8000/nodes/source/ \
-H 'Content-Type: application/json' \
-d '{
    "name": "repair_orders",
    "description": "Repair orders",
    "mode": "published",
    "catalog": "default",
    "schema_": "roads",
    "table": "repair_orders"
}'
from djclient import DJ, Source

client = DJ("http://localhost:8000/")
client.push(
    Source(
        name="repair_orders",
        description="Repair orders",
        mode="published",
        catalog="default",
        schema_="roads",
        table="repair_orders",
    )
)

To read more about DJRS, see the Table Reflection page.