Skip to content

Polars BigQuery

polars-bigquery is a fast I/O connector that brings Google BigQuery tables and queries directly into Polars LazyFrame and DataFrame workflows.

Under the hood, it streams Apache Arrow record batches from the BigQuery Storage Read API in Rust and hands them off to Polars via the zero-copy Arrow C Stream interface.

Key Features

  • Lazy scanning with pushdown: Client.scan_table integrates with the Polars query optimizer to push down column selections and row filter predicates to BigQuery—reducing both network transfer and BigQuery bytes scanned.
  • High-throughput Rust streaming: Reads from multiple BigQuery Storage Read API streams in parallel with LZ4 Arrow compression and zero-copy transfer into Polars.
  • GoogleSQL query execution: Client.read_query runs arbitrary GoogleSQL queries and streams the result table back into a Polars DataFrame.
  • Connection & credential reuse: Client pools gRPC and HTTP connections and authenticates using Google Cloud Application Default Credentials (ADC) via polars.CredentialProviderGCP.

Getting Started

1. Installation

Install polars-bigquery alongside polars:

pip install polars-bigquery

uv add polars-bigquery

2. Authentication

By default, polars-bigquery uses Application Default Credentials (ADC). When developing locally, authenticate with the Google Cloud CLI:

gcloud auth application-default login

You will also need a Google Cloud project ID (quota_project_id) with the BigQuery Read Session User (roles/bigquery.readSessionUser) role to bill Storage Read API sessions and query jobs.

3. Quick Example

Use Client.scan_table to lazily scan a table—Polars will push both .select() columns and .filter() predicates down to BigQuery when .collect() is called:

import polars as pl
import polars_bigquery

with polars_bigquery.Client(quota_project_id="my-gcp-project") as client:
    # Lazily scan a BigQuery table with column and predicate pushdown
    lf = client.scan_table("bigquery-public-data.usa_names.usa_1910_2013")
    df = (
        lf.filter((pl.col("state") == "WA") & (pl.col("year") >= 2000))
        .select(["name", "year", "number"])
        .group_by("name")
        .agg(pl.col("number").sum().alias("total"))
        .sort("total", descending=True)
        .head(10)
        .collect()
    )
    print(df)

Need to run a custom SQL query? Use Client.read_query:

with polars_bigquery.Client(quota_project_id="my-gcp-project") as client:
    df = client.read_query(
        """
        SELECT name, SUM(number) AS total_born
        FROM `bigquery-public-data.usa_names.usa_1910_2013`
        GROUP BY name
        ORDER BY total_born DESC
        LIMIT 10
        """
    )

For query results that are larger than memory or over 10 GB, see Scanning large query results.

Documentation

Development

Interested in contributing or running the test suite?