Cloud Storage
Destination Connector
Overview
This destination writes data to GCS bucket. Each stream is written to its own directory under the bucket.
Requirements
- Allow connections from Tarsal servers to your GCS cluster.
- A GCP bucket with credentials
(for the COPY strategy)
.
Setup guide
- Retrieve the following info to setup the configuration
- GCS Bucket Name
- See this for instructions on how to create a GCS bucket. The bucket cannot have a retention policy. Set Protection Tools to none or Object versioning.
- GCS Bucket Region
-
- HMAC Key Access ID
- See this on how to generate an access key. For more information on hmac keys please reference the GCP docs
- We recommend creating a Tarsal-specific user or service account. This user or account will require the following permissions for the bucket:
You can set those by going to the permissions tab in the GCS bucket and adding the appropriate the email address of the service account or user and adding the aforementioned permissions.storage.multipartUploads.abort storage.multipartUploads.create storage.objects.create storage.objects.delete storage.objects.get storage.objects.list
- Secret Access Key
- Corresponding key to the above access ID.
- GCS Bucket Name
Configuration
This reference table describes the portal fields required for key and role authentication types. Replace the values in brackets ({}
) with the applicable information for your google account and resources.
Parameter | Description | Example | |
---|---|---|---|
Bucket name | Required | Name of the bucket to sync data into | audit-logs |
GCS Bucket Path | Optional | Subdirectory under the above bucket to sync the data into. | tarsal-bucket-path |
GCS Region | Required | See here for all region codes. | US-EAST1 |
HMAC Key Access ID | Required | HMAC key access ID . The access ID for the GCS bucket. When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. See HMAC key for details. | GOOGTS7C7FUP3AI...TJFSA |
HMAC Key Secret | Optional | The corresponding secret for the access ID. It is a 40-character base-64 encoded string. | bGoa+V7g/yqDXvK...f9RzJ |
Output Format | Required | The delimiter in the destination data file(s) | JSONL |
Currently, only the HMAC key is supported. More credential types will be added in the future, please contact us if you'd like to request another credential type.
Updated 4 days ago