Azure Blob Storage

Overview

The Destination Connector allows you to sync data to Azure Blob Storage. Each stream is written to its own blob under the container.

Azure Blob Storage Configuration

This reference table describes the portal fields required for key and role authentication types. Replace the values in brackets ({}) with the applicable information for your Azure account and resources.

Parameter		Description	Example
`Endpoint Domain Name`	Optional	This is Azure Blob Storage endpoint domain name. If empty, will use Microsoft native one.	`blob.core.windows.net`
`Azure blob storage container (Bucket) Name`	Required	A name of the Azure blob storage container. If not exists - will be created automatically. If leave empty, then will be created automatically tarsalcontainer+timestamp.	`tarsal-audit-logs`
`Azure Blob Storage account name`	Required	The account's name of the Azure Blob Storage.	`tarsal-account`
`Azure blob storage account key`	Required	Azure blob storage account key.	`abcdefghijklmnopqrstuvwxyz/0123456789+ABCDEFGHIJKLMNOPQRSTUVWXYZ/0123456789%++sampleKey==`
`Azure Blob Storage output buffer size`	Optional	Azure Blob Storage output buffer size, in megabytes.	`5`
`Output Format`	Required	The delimiter in the destination data file(s)	`JSONL`

Add and configure the Azure Blob Storage destination connector:

Sign in to the Tarsal portal at https://app.tarsal.cloud.
Go to Configuration > Destinations from the left navigation.
Click the Add Destination button in the upper right.
Click Azure Blob Storage.
Under Metadata, enter Azure Blob Storage for Name.
Under Configuration,
1. enter your bucket name for Azure Blob Storage Container Name.
2. enter your account name for Azure Blob Storage Account Name.
3. Fill out the optional fields if applies
Under Output Format
1. Select JSONL.
Click the Save button.

The portal immediately notifies you whether the connector configuration is successful with a status banner in the lower right.

If the connector configuration fails, verify all preceding steps or contact Tarsal customer support. See the next section for testing.

Test the Connector

In the portal, hover over the icon in the Health column for the connector in the list on the Destinations page or next to the Status label on the connector’s detail page. A broken heart icon indicates failure, and the Summary widget on the Destinations page also lists destinations errors.

📘
Updating AWS Configurations
Be sure to test the connector configuration in the portal after any AWS changes to associated users, roles, policies, or regions.

Sign in to the Tarsal portal at https://app.tarsal.cloud.
Go to Configuration > Destinations from the left navigation.
In the Destinations list, click . . . (three dots) under the Actions column for the connector.
Select Test from the drop-down list.

A banner in the lower right indicates the success or failure of the connector configuration test.

Alternatively, the connector can be tested directly from the connector destination detail page using the Test button in the upper right.

Output Schema

Each stream will be outputted to its dedicated Blob according to the configuration. The complete datastore of each stream includes all the output files under that Blob. You can think of the Blob as equivalent of a Table in the database world. New output files will be added that only contain the new data.

Column	Condition	Description
`data`	Always exists	The log or event data
`_tarsal_metadata`	Always exists	Tarsal-defined column for each record; an event processing `timestamp` and `UUID`
Root-level fields	Root-level normalization (flattening)	Expanded root-level field

JSON Lines (JSONL)

JSON Lines, or newline-delimited JSON, is a text format for structured data. JSONL has one JSON per line. The output file has the following structure:

{  
  "_tarsal_metadata": "<json-metadata>"  
  "data": "<json-data-from-source>"  
}

An example of JSON source and output follows:

[
  {
    "user_id": 123,
    "name": {
      "first": "John",
      "last": "Doe"
    }
  },
  {
    "user_id": 456,
    "name": {
      "first": "Jane",
      "last": "Roe"
    }
  }
]

{ "_tarsal_metadata": {"_tarsal_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_tarsal_emitted_at": "1631948170000"}, "data": { "user_id": 123, "name": { "first": "John", "last": "Doe" } } }
{ "_tarsal_metadata": {"_tarsal_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_tarsal_emitted_at": "1631948170000"}, "data": { "user_id": 456, "name": { "first": "Jane", "last": "Roe" } } }

Overview