{# Spec 151 - per-table layout recommendation detail. Renders: - Header (table name, region, size, window cost) - Current vs proposed layout (side-by-side) - Ranked candidate columns (table) - Apply SQL (copy panel) - Rollback SQL (copy panel) - Caveats list Empty state (no recommendation in latest scan) renders a single explanation card with a "Re-run scan" link instead of 404ing. #} {% extends "layout.html" %} {% block content %} {% from "components/_page_header.html" import back_link, header_card %} {# Spec 151 US3 - small "Deep analyse" entry point per candidate column. Renders as either a cached-result chip (when the user has already run deep analysis on that column) or a tiny form button that POSTs to the /deep/preview endpoint to get a cost estimate before the chargeable run. The route accepts ``column_name`` as form data; the table_full_name is on the URL. #} {%- macro _deep_button(table_full_name, column_name, deep_results_by_column) -%} {%- set _deep = (deep_results_by_column or {}).get(column_name) -%} {%- if _deep -%} ~{{ "{:,}".format(_deep.approx_distinct_count or 0) }} distinct {%- else -%}
{%- endif -%} {%- endmacro %}
{{ back_link("/recommendations/layout", "Partition & Cluster") }} {% set _path_parts = (table_full_name or '').split('.') %} {% set _dataset = _path_parts[1] if _path_parts | length >= 2 else '' %} {% set _table = _path_parts[2] if _path_parts | length >= 3 else (table_full_name or '') %} {% set _title %}{% if _dataset %}{{ _dataset }}.{% endif %}{{ _table }}{% endset %} {{ header_card(_title, caption="Partition & Cluster recommendation") }} {# Recommendation source / provenance - make it unmistakable whether this pick comes from Governor-audit, BigQuery's own Recommender, or both (and whether the two methodologies agree). #} {% if recommendation %} {% set _src_has_audit_pick = recommendation.recommended_partition_column or cluster_candidates %}
Recommended by: {% if _src_has_audit_pick %} Governor-audit {% endif %} {% if bq_second_opinion %} {% if _src_has_audit_pick and bq_second_opinion.agrees_with_audit_top_pick %} BigQuery Recommender ✓ both methodologies agree {% elif not _src_has_audit_pick %} BigQuery Recommender only audit didn't pick this table {% else %} BigQuery Recommender different top pick - see second opinion {% endif %} {% else %} BigQuery second opinion not enabled {% endif %}
{% endif %} {% if not recommendation %}

No recommendation in the latest scan for this table. The most common reasons:

  • The table is below 10 GB - clustering's benefit doesn't outweigh metadata cost.
  • Consumer queries don't filter / join / group on enough common columns to give a confident signal.
  • The table is already optimally laid out for the workload we see.

Recommendations are derived from the workload the latest scan observed. Run another scan if you suspect this changed recently.

{% else %} {# Three-card stat strip - reframed to answer "is this worth doing?": 1. Table size - magnitude check (is this a big table?) 2. Cost in scan window - what we'd potentially reduce 3. Affected downstreams - count of consumer models with active issues (the actionable signal - "Contributing jobs" was just evidence density and not user-meaningful) #} {# Downstream consumers split: all (for the lineage graph) and the issue-only subset (for the "Affected downstreams" stat + the "how it helps" copy). The lineage now shows every downstream, so these can no longer be the same count. #} {% set _all_consumers = (lineage_payload.nodes | rejectattr('kind', 'equalto', 'target') | list) if lineage_payload else [] %} {% set _issue_consumers = _all_consumers | selectattr('issue_count') | list %} {% set _impacted_count = _issue_consumers | length %} {% from "components/_stat_card.html" import stat_card %}
{{ stat_card("Table size", recommendation.table_total_bytes | format_bytes) }} {{ stat_card( "Cost in scan window", recommendation.table_total_cost_in_window_usd | format_usd, meta="across " ~ recommendation.contributing_job_count ~ " consumer job" ~ ("s" if recommendation.contributing_job_count != 1 else "") ) }} {% if _impacted_count > 0 %} {{ stat_card( "Affected downstreams", _impacted_count, meta="model" ~ ("s" if _impacted_count != 1 else "") ~ " with active issues", accent="rose" ) }} {% else %} {{ stat_card("Affected downstreams", 0, meta="no flagged downstreams - preventative", accent="emerald") }} {% endif %}
{# US2 - BigQuery second-opinion panel. Renders when a LayoutBqSecondOpinion row exists for this recommendation; otherwise renders a banner explaining the IAM grant. #} {% if bq_second_opinion %} {# "Audit has no pick" detection - empty audit fields tell us the row was created from Google's recommendation alone (BQ-only row, not analyzed by the audit's heuristic). Used to suppress the misleading "Different top pick" badge and to swap the Audit-picks panel for a clear "audit didn't analyze" note. #} {% set _audit_has_pick = recommendation.recommended_partition_column or cluster_candidates %}

BigQuery's recommendation (second opinion)

{% if not _audit_has_pick %} BigQuery only - audit didn't analyze this table {% elif bq_second_opinion.agrees_with_audit_top_pick %} Agreement on top column {% else %} Different top pick {% endif %}

BigQuery picks

{% if bq_second_opinion.bq_recommended_columns %}
{% for col in bq_second_opinion.bq_recommended_columns %} {{ col }} {% endfor %}
{% else %}

No specific columns surfaced

{% endif %}

Methodology: BigQuery's ML-based recommender - slot-time + scan-pattern weighting on the recent workload.

Audit picks

{% if _audit_has_pick %}
{% if recommendation.recommended_partition_column %} {{ recommendation.recommended_partition_column }} {% endif %} {% for c in cluster_candidates %} {{ c.column_name }} {% endfor %}

Methodology: cost-weighted filter / join / group-by frequency across {{ recommendation.contributing_job_count }} cached queries.

{% else %}

No audit pick - the table didn't meet the audit's size gates (≥ 10 GB cluster / ≥ 100 GB partition) or had too few consumer queries in the scan window to converge on a column. The audit's recommendations require both.

Methodology: cost-weighted filter / join / group-by frequency across cached consumer queries - needs a populated window to fire.

{% endif %}
{% if bq_second_opinion.bq_description %}

{{ bq_second_opinion.bq_description }}

{% endif %}
{% else %}

BigQuery second opinion

Add roles/recommender.bigqueryPartitionClusterViewer to the principal that runs the scan to surface BigQuery's own recommendation alongside the audit's. The page works without it - this enables a second-opinion panel that increases confidence when both methodologies agree.

{% endif %} {# Why this recommendation - rationale panel. Explains the reasoning behind the picked partition / cluster columns and how a layout change ripples into the downstream issues shown in the lineage below. Replaces the deleted "Ranked candidates" table - same evidence, surfaced in plain English. #} {% set _part_pick = partition_candidate %} {% set _cluster_pick = (cluster_candidates | first) if cluster_candidates else None %} {% set _consumers = (lineage_payload.nodes | rejectattr('kind', 'equalto', 'target') | list) if lineage_payload else [] %} {# Single card: the picked columns, then the explanation of how the layout change helps downstream issues + the pick methodology. #}

Why this recommendation

Picked columns

    {% if _part_pick %}
  • PARTITION BY {{ _part_pick.column_name }} cost share {{ "%.0f%%" | format([_part_pick.cost_share * 100, 100.0] | min) }} - score {{ "%.2f" | format(_part_pick.score) }} {{ _deep_button(recommendation.table_full_name, _part_pick.column_name, deep_results_by_column) }}
  • {% endif %} {% for c in cluster_candidates %}
  • CLUSTER #{{ c.cluster_rank }} {{ c.column_name }} {{ c.usage_type | replace('_', ' ') }} cost share {{ "%.0f%%" | format([c.cost_share * 100, 100.0] | min) }} - score {{ "%.2f" | format(c.score) }} {{ _deep_button(recommendation.table_full_name, c.column_name, deep_results_by_column) }}
  • {% endfor %} {% if not _part_pick and not cluster_candidates %}
  • No picks - workload signal was below the partition / cluster gates.
  • {% endif %}
{# Spec 151 US3 entry point. Each candidate column above gets a small "Deep analyse" button that POSTs to the preview endpoint - backend dry-runs and returns the cost estimate the user must acknowledge before the chargeable run. Cached results show inline so the user sees the approx distinct count. #}

How it helps the downstream issues

{% if _issue_consumers | length > 0 %} {{ _issue_consumers | length }} downstream model{{ 's' if _issue_consumers | length != 1 else '' }} reading from this table carr{{ 'y' if _issue_consumers | length != 1 else 'ies' }} active issues today. {% if _cluster_pick %} Clustering by {{ _cluster_pick.column_name }} reorders the table's micro-files so reads filtered on it scan fewer bytes - the same scans that drive slot-contention and shuffle-spill on those downstreams. The lineage below names each impacted model (red). {% elif _part_pick %} Partitioning by {{ _part_pick.column_name }} lets BigQuery prune partitions at read time, reducing the byte volume that drives slot-contention and shuffle-spill on those downstreams. {% endif %} {% else %} No downstream models with active issues today. The recommendation is preventative - it locks in efficient layout before issues appear at scale. {% endif %}

Pick methodology: cost-weighted filter / join / group-by frequency across {{ recommendation.contributing_job_count }} cached consumer job{{ 's' if recommendation.contributing_job_count != 1 else '' }}, intersected with the table's schema. Columns must clear the materiality (≥ 1 GB), partition (DATE / TIMESTAMP, 70 % filter), and cluster (50 % combined) gates.

{# Downstream lineage - pure HTML/SVG dbt-style chip layout. Target chip on the left, every consumer model with active issues stacked on the right; curved SVG connectors drawn between them. Pure CSS + tiny inline SVG so it always renders (the previous ECharts attempt failed silently on the user's data). #}

Downstream lineage (all consumers)

Every consumer model that reads from this table. Red carries active issues, blue is clean. Click a chip to open its detail.

this table has issues clean
{% if _consumers | length == 0 %}
No downstream consumers read from this table in the scan window - this layout change is purely preventative.
{% else %} {# Cap visible at 12 so the lineage stays a sensible height even when a base table fans out to 30+ downstreams. Top rows are picked by issue_count, then cost. #} {% set _max_visible = 12 %} {% set _sorted_consumers = _consumers | sort(attribute='total_cost_usd', reverse=true) | sort(attribute='issue_count', reverse=true) %} {% set _visible = _sorted_consumers[:_max_visible] %} {% set _hidden_count = (_consumers | length) - (_visible | length) %} {# ECharts ``tree`` series renders a clean dbt-style left-to- right hierarchical layout for free: chip-shaped nodes, smooth bezier edges, automatic spacing. We pre-compute the tree payload server-side and pass it via ``data-payload``; the script below feeds it straight into ECharts. #}
{% if _hidden_count > 0 %}

+ {{ _hidden_count }} more downstream model{{ 's' if _hidden_count != 1 else '' }} - view all

{% endif %} {% endif %}
{# Current / Proposed layout panel removed - the picks are already surfaced in the "Picked columns" sub-panel of the rationale, and the exact ALTER SQL is in the Apply panel below. Two restatements were redundant. #} {# Apply / Rollback SQL panels. #} {% from "components/_copy_pre.html" import copy_pre %} {% if recommendation.apply_sql %} {{ copy_pre("apply-sql", "Apply", recommendation.apply_sql) }} {% endif %} {% if recommendation.rollback_sql %} {{ copy_pre("rollback-sql", "Rollback", recommendation.rollback_sql) }} {% endif %} {# Caveats - use render_caveats (route-computed) so per-request warnings like PARTITION_CARDINALITY_EXCEEDED show alongside the persisted caveats. #} {% set _caveats = render_caveats if render_caveats is defined else recommendation.caveats %} {% if _caveats %}

Caveats

    {% for caveat in _caveats %}
  • {{ caveat.code }} {{ caveat.message }}
  • {% endfor %}
{% endif %} {% endif %}
{% endblock %}