Skip to content
Storage

BigQuery policy tags

Keep BigQuery column-level access control on the copies Keboola makes of a protected table within your BigQuery backend.

BigQuery policy tags (opens in a new tab) restrict who can read a column. You attach them in BigQuery.

Without this feature, Keboola copies a tagged column within your backend without its tag. The copy is unprotected, and the only trace is a storage.policyTagsNotPropagated event in the project (see Events). This holds only when the project’s service account can read the column: on a taxonomy that enforces fine-grained access, a project without the Fine-Grained Reader gets storage.backend.policyTagAccessDenied instead, with or without the feature.

With policy-tag support turned on, Keboola does three things:

  • It keeps the policy tag when it pulls a protected table into a development branch, loads it into a workspace (clone or copy, including from a linked bucket), or creates a table from it with the Storage API. A Storage load into an existing tagged table keeps that table’s tags. Other paths, such as transformation outputs, are listed in What this feature does not cover.
  • When it cannot keep the tag on one of those copies, it refuses to make the copy.
  • It reports each column’s protection state in the table detail.

Keboola keeps tags that a source already carries. It never adds a tag to a table whose source has none.

The feature is available only on your own BigQuery backend (BYODB). It is off by default. See Enable BigQuery policy tags.

The feature is switched on at two levels, and Keboola support handles both on request:

  • The backend. You grant one custom role on your taxonomy, and Keboola verifies the grant and enables the backend. This is done once per backend.
  • Each project. Tags are carried over only in projects where Keboola also turns on the project feature bigquery-policy-tags. The other projects on the backend keep the previous behavior: a copy loses its tag, and a storage.policyTagsNotPropagated event names the exposed columns, provided the project can read them (see above). You can start with one project and add more later; adding a project needs nothing on your side beyond asking for it.

For a linked bucket, the project that decides is the one that makes the copy, not the one that shares the data. Turning the feature on in a project affects the next copy. Tables already copied without tags are not re-tagged, except a workspace table that a later incremental load reaches (see Incremental loads into an existing workspace table).

Keboola carries the tag over on these copies within your backend:

  • pulling a table into a development branch;
  • loading a table into a workspace, as a clone or a copy, including from a linked bucket;
  • creating a table from another table with the Storage API (the tables-definition endpoint (opens in a new tab) with a source).

Other paths, such as snapshot restore, are listed in What this feature does not cover. A new destination is created already protected, before any row reaches it.

Columns are matched by name. A column renamed in an input mapping keeps its tag under the new name.

A workspace’s service account can edit the tables in its workspace, but it does not own the workspace dataset. It cannot remove a tag that Keboola carried over, or replace it with another one. Workspaces created before this role was introduced are an exception; see Known gaps.

When Keboola cannot keep the protection, the job fails and names the affected columns. Nothing is copied, and a table the job was about to replace stays as it was. The job fails as a user error, so Keboola does not retry it. Typical reasons:

  • The tag sits on a field inside a STRUCT, or a column carries more than one tag.
  • The tag comes from a taxonomy in another region, or from a taxonomy other than the one you named for the backend. BigQuery checks the project service account’s permission on that tag’s taxonomy, and the setter role is granted only on yours.
  • The source is a view that renames, computes or joins a protected column, so Keboola cannot trace where the column comes from. A view that passes a protected column through under its own name is copied with the tag. An alias of a tagged table is copied with the tag as well. Creating a table from a filtered alias of a tagged table is refused, because the filter makes the alias such a view.
  • Keboola cannot read the source, or the project’s service account lacks a grant that the copy needs.

Incremental loads into an existing workspace table

Section titled “Incremental loads into an existing workspace table”

A workspace table that existed before the feature was switched on, or before its source was tagged, has no tags. The next incremental load from a tagged source adds the source’s tags to those columns before it adds any rows, and records a storage.workspacePolicyTagsAdded event. The new rows land in a protected column, and the rows already in the table are protected from then on. The tags stay even if the load then fails, for example because the project may not read the tagged column.

Keboola never replaces an existing tag. If a column of the workspace table already carries a different tag than its source, the load fails with storage.workspace.policyTagMismatch before it writes anything, and the project records a storage.workspacePolicyTagMismatch event. This happens when the source’s tag changed after an earlier load tagged the workspace table, or when the tag was set on the workspace table directly. A workspace table that carries a tag its source does not have is left alone, and the load proceeds.

You grant one custom role on the taxonomy that holds your tags. The role holds a single permission, datacatalog.taxonomies.get, and you bind it to two principals:

PrincipalWhy
The service account your backend is registered withKeboola uses it to test the setup before it enables the backend.
principalSet://cloudresourcemanager.googleapis.com/folders/{folder}/type/ServiceAccount, the backend’s folderThe service accounts of your Keboola projects use it to attach your tags to the tables they create.

The folder principal set covers every service account in every project under the backend’s folder, including projects created later, so you make this binding once. Keboola makes no change to your taxonomy’s access policy, neither at enablement nor when projects are created or deleted, and it does not ask for the right to change that policy.

The role must be a custom role. Keboola refuses a predefined role, because the role reaches every service account in the folder. Attaching a tag also needs the BigQuery permission bigquery.tables.setCategory, which project service accounts already hold on their own data.

These grants do not let anyone read protected values. Reading a protected column is a separate grant, described in Fine-grained access.

If your taxonomy enforces fine-grained access control, BigQuery lets only principals with the Fine-Grained Reader role (roles/datacatalog.categoryFineGrainedReader) read a tagged column. Keboola’s copies run as the project’s service account, so the project needs that role to copy tagged data.

Without it, the project cannot:

  • clone or copy a table with a tagged column, pull it into a development branch, or create a table from it;
  • preview or export it;
  • load it incrementally, because the merge reads the tagged column. Only a full load succeeds.

Each of these fails with the error storage.backend.policyTagAccessDenied. The error names the policy tag, the column where BigQuery reports it, and the role to ask for. A job that fails this way is not retried.

Neither enabling the backend nor turning on the project feature grants this role. They only let Keboola attach tags. Which projects may see protected data is your decision: grant the role per project, on the taxonomy, to the principal set of the project’s data project, named by project number:

principalSet://cloudresourcemanager.googleapis.com/projects/{project number}/type/ServiceAccount

When Keboola turns on bigquery-policy-tags in a project of an enforcing taxonomy, it checks this grant and warns when it is missing. It still turns the feature on. The check reads the grant on the taxonomy’s test tag, so a role bound per tag rather than on the taxonomy can cause a warning that does not apply to your data tags. Binding on the taxonomy avoids that.

Each column of a BigQuery table reports a policyTagState in the Storage API table detail (definition.columns). The bucket listing does not carry it, and workspace tables report no state.

  • protected: Keboola observed a policy tag on the column.
  • unprotected: Keboola observed the column and it had no tag.
  • unknown: Keboola has not observed the table yet, or the column’s protection cannot be stated, for example a tag inside a STRUCT. Columns of a view are always unknown, because a view’s own schema never carries a tag.

The table’s Schema tab in the Keboola UI shows the same state.

The state reflects what Keboola last observed, not what BigQuery says right now. If you change a tag directly in BigQuery, refresh the bucket’s table information to update it. The refresh runs a small query that is billed to the project.

Keboola records an event in the project when a policy tag affects a copy within your BigQuery backend. Each event names the source table, the destination table and the columns involved.

EventWhat happenedWhat to do
storage.policyTagsNotPropagatedA protected column was copied without its tag, because Keboola could not carry the protection over on this copy. Usually the backend is not enabled, or the project does not have the feature turned on.Ask for the feature if copies must stay protected, and check who can read the copy.
storage.policyTagPropagationFailedKeboola tried to keep the tag, failed, and copied nothing.Read the error in the event, fix its cause, and run the job again.
storage.workspacePolicyTagsAddedAn incremental load added the source’s tags to columns of an existing workspace table that had no tag. The tags stay even if the load that follows fails.Nothing. The event records which columns are protected from now on.
storage.workspacePolicyTagMismatchAn incremental load was refused, because a column of the existing workspace table carries a different tag than its source. Keboola does not replace an existing tag.Ask someone with the rights to change policy tags in your Google Cloud project to change or remove that tag on the workspace table, then run the load again. The workspace’s own credentials normally cannot change a tag. See Incremental loads into an existing workspace table.

Policy-tag support keeps protection on the copies Keboola makes within your BigQuery backend. It does not cover:

  • Per-user access inside a project. BigQuery sees each project’s service account, not individual Keboola users. Every user with access to a project acts through the same account.
  • Loading into a Snowflake workspace. A protected BigQuery table loaded into a Snowflake workspace carries no tag, and Keboola records no event.
  • Snapshot restore. A table restored from a snapshot carries no tag.
  • Transformation outputs. A table that a transformation or another component writes back to Storage through output mapping carries no tag, even when a column passes through unchanged, and Keboola records no event. Writing into an existing tagged Storage table keeps that table’s tags. Tag new output tables in BigQuery yourself.
  • Data Apps. Data Apps are not covered.
  • Data leaving BigQuery. Keboola does not stop protected values from leaving through components such as the Google Sheets writer, through a Storage API table export, or through the data preview in the UI, if the project service account can read them.
  • Approved access. Granting a person access to protected data after a legal or compliance approval is not part of this feature.
  • Row-level security. BigQuery row access policies are a separate mechanism from policy tags and are outside this feature. A clone-based copy keeps them; a copy Keboola has to rebuild (a source that cannot be cloned, or a COPY load into a workspace) does not.
  • Grants removed after enablement. Keboola does not detect or prevent changes to the grants on your taxonomy. If you remove them, copies of protected columns fail.
  • An older workspace can still remove or replace a tag on its own tables. A workspace’s service account used to own its workspace dataset, which let it remove a carried-over tag or replace it with another tag from your taxonomy. New workspaces get an editor role that cannot do either. Workspaces created before that change keep the owner role until Keboola migrates them on your backend; ask Keboola support for the migration. Until then, if your taxonomy enforces fine-grained access, such a workspace can remove the tag and then read that column of its copy. This exposes nothing beyond what the project could already read when the project holds the Fine-Grained Reader role, but the copy is no longer protected, for example when it is exported.
  • Anyone allowed to read a tagged column can copy it without the tag. A policy tag restricts reading, not copying. A principal that holds the Fine-Grained Reader role, inside or outside Keboola, can write the values to a new table that carries no tag. This is how BigQuery policy tags work, and Keboola cannot prevent it.
  • One taxonomy per backend. The project service accounts hold the setter role only on the taxonomy you named, and BigQuery checks that permission on the taxonomy of each tag. A copy that Keboola rebuilds with a tag from another taxonomy therefore fails. Keep every tag on data Keboola copies in that taxonomy. Support for several taxonomies on one backend is planned.
Ask Kai

Hi, I'm Kai — Keboola's AI assistant for the docs. Ask me anything and I'll answer from the documentation and cite the pages I use.

Kai is an AI and can make mistakes. Check the sources it links.