Using the Raster Loader CLI

Most functions of the Raster Loader are accessible through the carto command-line interface (CLI). To start the CLI, use the carto command in a terminal.

Currently, Raster Loader allows you to upload a local raster file to a BigQuery table. You can also download and inspect a raster file from a BigQuery table.

Note

Accessing BigQuery with Raster Loader requires the GOOGLE_APPLICATION_CREDENTIALS environment variable to be set to the path of a JSON file containing your BigQuery credentials. See the GCP documentation for more information.

Uploading to BigQuery

To upload a raster file to a BigQuery table, use the carto bigquery upload command.

Before you can upload a raster file, you need to have set up the following in BigQuery:

A GCP project
A BigQuery dataset

You have the option to also set up a BigQuery table and use this table to upload your data to. In case you do not specify a table name, Raster Loader will automatically generate a table name for you and create that table.

At a minimum, the carto bigquery upload command requires a file_path to a local raster file that can be read by GDAL and processed with rasterio. It also requires the project (the GCP project name) and dataset (the BigQuery dataset name) parameters. There are also additional parameters, such as table (BigQuery table name) and overwrite (to overwrite existing data). For example:

carto bigquery upload \
  --file_path /path/to/my/raster/file.tif \
  --project my-gcp-project \
  --dataset my-bigquery-dataset \
  --table my-bigquery-table \
  --overwrite

This command uploads the TIFF file from /path/to/my/raster/file.tif to a BigQuery project named my-gcp-project, a dataset named my-bigquery-dataset, and a table named my-bigquery-table. If the table already contains data, this data will be overwritten because the --overwrite flag is set.

You can also use the --output_quadbin flag to upload the raster to the BigQuery table in a quadbin format. To use this option, the input raster must be a GoogleMapsCompatible raster. You can make your raster compatible by converting it with the following command with GDAL:

gdalwarp your_raster.tif -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE your_compatible_raster.tif

Inspecting a raster file on BigQuery

You can also use Raster Loader to retrieve information about a raster file stored in a BigQuery table. This can be useful to make sure a raster file was transferred correctly or to get information about a raster file’s metadata, for example.

To access a raster file in a BigQuery table, use the carto bigquery describe command.

At a minimum, this command requires a GCP project name, a BigQuery dataset name, and a BigQuery table name. For example:

carto bigquery describe \
  --project my-gcp-project \
  --dataset my-bigquery-dataset \
  --table my-bigquery-table

CLI details

The following is a detailed overview of all of the CLI’s subcommands and options:

carto

The carto command line interface.

carto [OPTIONS] COMMAND [ARGS]...

bigquery

Manage Google BigQuery resources.

carto bigquery [OPTIONS] COMMAND [ARGS]...

describe

Load and describe a table from BigQuery

carto bigquery describe [OPTIONS]

Options

--project <project>: Required The name of the Google Cloud project.

--dataset <dataset>: Required The name of the dataset.

--table <table>: Required The name of the table.

--limit <limit>: Limit number of rows returned

upload

Upload a raster file to Google BigQuery.

carto bigquery upload [OPTIONS]

Options

--file_path <file_path>: Required The path to the raster file.

--project <project>: Required The name of the Google Cloud project.

--dataset <dataset>: Required The name of the dataset.

--table <table>: The name of the table.

--band <band>: Band within raster to upload.

--chunk_size <chunk_size>: The number of blocks to upload in each chunk.

--input_crs <input_crs>: The EPSG code of the input raster’s CRS.

--overwrite: Overwrite existing data in the table if it already exists.

--output_quadbin: Upload the raster to the BigQuery table in a quadbin format (input raster must be a GoogleMapsCompatible raster).

--test: Use Mock BigQuery Client

info

Display system information.

carto info [OPTIONS]