


    2. Python SDK (Community)

  The https://github.com/DaSenf1860/ms-fabric-sdk-core package wraps the REST API:

  from msfabricpysdkcore import FabricClientCore

  fc = FabricClientCore()
  fc.create_shortcut(
      workspace_id="workspace_id",
      item_id="lakehouse_id",
      path="Tables",
      name="lineitem",
      target={
          "oneLake": {
              "workspaceId": "workspace_id",
              "itemId": "lakehouse_id",
              "path": "Files/tpch_1_incremental/lineitem"
          }
      }
  )

  Important Notes

  - Tables folder: Shortcuts only at top level (no subdirectories)
  - Files folder: Shortcuts at any level
  - For parquet files to appear as Delta tables, they need to be in Delta format or use the Files folder

  Would you like me to add a shortcut-based publisher to LakeXpress that uses this API instead of CREATE TABLE?

  Sources:
  - https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/create-shortcut
  - https://github.com/DaSenf1860/ms-fabric-sdk-core
  - https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts




Incremental
===========


  Test Workflow

  Step 1: Initial Setup (create schema, load initial data)

  python tests/incremental_test_setup.py \
      --credentials data/ds_credentials.json \
      --source_auth ds_04_pg \
      --phase init

  This will:
  - Create schema tpch_1_incremental
  - Copy all 8 TPC-H tables
  - For lineitem and orders: only load data where date < 1995-01-01
  - For other tables: full copy

  Step 2: Generate LakeXpress Config

  python tests/incremental_test_setup.py --phase config

  This prints the lxpress.py config create command with correct --incremental_table args.

  Step 3: Create LakeXpress Sync and Run First Export

  # Create config (adjust storage/publish targets as needed)
  python lxpress.py config create \
      -a data/ds_credentials.json \
      --log_db_auth_id log_db_ms_02 \
      --source_db_auth_id ds_04_pg \
      --source_db_name tpch \
      --source_schema_name tpch_1_incremental \
      --fastbcp_dir_path /home/francois/Workspace/FastBCP_exe/latest/ \
      --target_storage_id <your_s3_storage> \
      --publish_target_id <your_glue_target> \
      --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date:i" \
      --incremental_table "tpch_1_incremental.orders:o_orderdate:date:i" \
      --generate_metadata

  # Run first sync
  python lxpress.py sync run --sync_id <sync_id>

  Step 4: Add Incremental Data

  python tests/incremental_test_setup.py \
      --credentials data/ds_credentials.json \
      --source_auth ds_04_pg \
      --phase increment

  This adds data where date is between 1995-01-01 and 1996-01-01.

  Step 5: Run Second Sync (Incremental)

  python lxpress.py sync run --sync_id <sync_id>

  Only the new rows should be exported for lineitem/orders.

  Step 6: Verify Results

  python tests/incremental_test_setup.py \
      --credentials data/ds_credentials.json \
      --source_auth ds_04_pg \
      --log_db_auth log_db_ms_02 \
      --sync_id <sync_id> \
      --phase verify

  Check Current Stats

  python tests/incremental_test_setup.py \
      --credentials data/ds_credentials.json \
      --source_auth ds_04_pg \
      --phase stats



--------------------------------------------------------------------------------


python lxpress.py logdb drop --auth_file data/ds_credentials.json --log_db_auth_id log_db_ms_02 --confirm
python lxpress.py logdb init --auth_file data/ds_credentials.json --log_db_auth_id log_db_ms_02

python tests/incremental_test_setup.py \
       --credentials data/ds_credentials.json \
       --source_auth ds_04_pg \
       --phase init

python lxpress.py config create \
       -a data/ds_credentials.json \
       --log_db_auth_id log_db_ms_02 \
       --source_db_auth_id ds_04_pg \
       --source_db_name tpch \
       --source_schema_name tpch_1_incremental \
       --fastbcp_dir_path /home/francois/Workspace/FastBCP_exe/latest/ \
       --target_storage_id aws_s3_01 \
       --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date" \
       --incremental_table "tpch_1_incremental.orders:o_orderdate:date" \
       --generate_metadata

python lxpress.py sync

python tests/incremental_test_setup.py \
       --credentials data/ds_credentials.json \
       --source_auth ds_04_pg \
       --phase increment

python lxpress.py sync


python tests/incremental_test_setup.py \
      --credentials data/ds_credentials.json \
      --source_auth ds_04_pg \
      --log_db_auth log_db_ms_02 \
      --sync_id sync-20260108-41eab0 \
      --phase verify

python lxpress.py config delete --sync_id sync-20260108-41eab0 \
       -a data/ds_credentials.json \
       --log_db_auth_id log_db_ms_02 --confirm

