Metadata-Version: 2.4
Name: arcane-bigquery-storage
Version: 1.3.2
Summary: A package to use the BigQuery Storage Write API
Author: Arcane
Author-email: product@wearcane.com
Requires-Python: >=3.9,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: backoff (>=1.10.0)
Requires-Dist: google-cloud-bigquery-storage (>=2.28.0,<3.0.0)
Description-Content-Type: text/markdown

# Arcane bigquery-storage

A thin wrapper around the [BigQuery Storage Write API](https://cloud.google.com/bigquery/docs/write-api).
The Storage Write API uses [protocol buffers](https://protobuf.dev/) to
describe and serialize rows, so to write to a table you bring a compiled
protobuf message class that mirrors the table schema.

The flow is always the same:

1. **Write a `.proto`** that mirrors your BigQuery table schema.
2. **Compile it** with `protoc` to get a `*_pb2.py` module.
3. **Use it** with `Client.create_proto_descriptor` / `Client.create_row` /
   `Client.write_rows`.

The next three sections walk through each step.

## 1. Write a `.proto`

Create a file like `my_table.proto` next to your code. Each field must match
a column in your BigQuery table by name; the proto type must be compatible
with the BigQuery column type (see the
[protobuf → BigQuery type mapping](https://cloud.google.com/bigquery/docs/write-api#data_type_conversions)).

```proto
syntax = "proto2";

message MyRow {
    optional string name  = 1;
    optional int64  value = 2;
}
```

A couple of rules worth knowing:

- Field **names** in the proto must equal column names in BigQuery.
- Field **numbers** (the `= 1`, `= 2` tags) identify fields on the wire.
  Never re-use a number for a different field — that breaks compatibility
  with any rows already written.
- `proto2` makes every field `optional` by default, which mirrors BigQuery's
  NULLABLE columns nicely. `proto3` works too; just be aware of its
  default-value semantics.

## 2. Compile the `.proto`

You only need to do this once per message type — commit the generated
`*_pb2.py` to your repo.

### Install the compiler

Check [the protoc downloads page](https://protobuf.dev/downloads/) for the
current release and the [Python compatibility matrix](https://protobuf.dev/support/version-support/#python).
At the time of writing, protoc 21.12 pairs with protobuf 3.20.3.

On macOS:

```sh
PB_REL="https://github.com/protocolbuffers/protobuf/releases"
curl -LO $PB_REL/download/v21.12/protoc-21.12-osx-universal_binary.zip
unzip protoc-21.12-osx-universal_binary.zip -d protoc-21.12
sudo mv protoc-21.12/bin/protoc /usr/local/bin/
sudo cp -r protoc-21.12/include/* /usr/local/include/
protoc --version
```

### Run the compiler

From the directory containing `my_table.proto`:

```sh
protoc -I. -I/usr/local/include --python_out=. --pyi_out=. my_table.proto
```

This generates `my_table_pb2.py` (and a `.pyi` stub). Commit both.

## 3. Write rows to BigQuery

```python
from arcane.bigquery_storage.client import Client
from my_package.proto import my_table_pb2  # the file you just compiled

client = Client()  # picks up Application Default Credentials

# Get a stream to write to.
# Default stream = at-least-once, no per-hour limit. For exactly-once,
# use client.create_application_stream(...) instead.
stream_name = client.create_default_stream_name(
    project_id="my-project",
    dataset_id="my_dataset",
    table_id="my_table",
)

# Build a descriptor of your message type.
descriptor = Client.create_proto_descriptor(my_table_pb2.MyRow)

# Serialize each row as bytes.
rows = [
    Client.create_row(my_table_pb2.MyRow, {"name": "alice", "value": 1}),
    Client.create_row(my_table_pb2.MyRow, {"name": "bob",   "value": 2}),
]

# Append.
client.write_rows(stream_name, descriptor, rows)
```

Keys in the `dict` passed to `create_row` must match the field names in your
`.proto` exactly; unknown keys raise `ValueError` from the protobuf runtime.

See the [stream-type docs](https://cloud.google.com/bigquery/docs/write-api#application-created_streams)
for the trade-offs between the default stream and application-created
streams.

## Legacy helpers (deprecated)

Earlier versions of this package shipped table-specific helpers
(`create_feed_boost_result_proto_descriptor`,
`create_feed_boost_statistic_proto_descriptor`,
`create_feed_boost_result_row`, `create_feed_boost_statistic_row`) and
bundled the matching `.proto` files. They still work but now emit a
`DeprecationWarning` and will be removed in a future major release. New code
should use `Client.create_proto_descriptor` and `Client.create_row` with its
own compiled protobufs.

## Release history

See [CHANGELOG.md](./CHANGELOG.md).

