Azure Blob Storage
Overview
The Destination Connector allows you to sync data to Azure Blob Storage. Each stream is written to its own blob under the container.
Azure Blob Storage Configuration
This reference table describes the portal fields required for key and role authentication types. Replace the values in brackets ({}
) with the applicable information for your Azure account and resources.
Parameter | Description | Example | |
---|---|---|---|
Endpoint Domain Name | Optional | This is Azure Blob Storage endpoint domain name. If empty, will use Microsoft native one. | blob.core.windows.net |
Azure blob storage container (Bucket) Name | Required | A name of the Azure blob storage container. If not exists - will be created automatically. If leave empty, then will be created automatically tarsalcontainer+timestamp. | tarsal-audit-logs |
Azure Blob Storage account name | Required | The account's name of the Azure Blob Storage. | tarsal-account |
Azure blob storage account key | Required | Azure blob storage account key. | abcdefghijklmnopqrstuvwxyz/0123456789+ABCDEFGHIJKLMNOPQRSTUVWXYZ/0123456789%++sampleKey== |
Azure Blob Storage output buffer size | Optional | Azure Blob Storage output buffer size, in megabytes. | 5 |
Output Format | Required | The delimiter in the destination data file(s) | JSONL |
Add and configure the Azure Blob Storage destination connector:
- Sign in to the Tarsal portal at https://app.tarsal.cloud.
- Go to Configuration > Destinations from the left navigation.
- Click the Add Destination button in the upper right.
- Click Azure Blob Storage.
- Under Metadata, enter
Azure Blob Storage
for Name. - Under Configuration,
- enter your bucket name for Azure Blob Storage Container Name.
- enter your account name for Azure Blob Storage Account Name.
- Fill out the optional fields if applies
- Under Output Format
- Select
JSONL
.
- Select
- Click the Save button.
The portal immediately notifies you whether the connector configuration is successful with a status banner in the lower right.
If the connector configuration fails, verify all preceding steps or contact Tarsal customer support. See the next section for testing.
Test the Connector
In the portal, hover over the icon in the Health column for the connector in the list on the Destinations page or next to the Status label on the connector’s detail page. A broken heart icon indicates failure, and the Summary widget on the Destinations page also lists destinations errors.
Updating AWS Configurations
Be sure to test the connector configuration in the portal after any AWS changes to associated users, roles, policies, or regions.
- Sign in to the Tarsal portal at https://app.tarsal.cloud.
- Go to Configuration > Destinations from the left navigation.
- In the Destinations list, click
. . .
(three dots) under the Actions column for the connector. - Select
Test
from the drop-down list.
A banner in the lower right indicates the success or failure of the connector configuration test.
Alternatively, the connector can be tested directly from the connector destination detail page using the Test button in the upper right.
Output Schema
Each stream will be outputted to its dedicated Blob according to the configuration. The complete datastore of each stream includes all the output files under that Blob. You can think of the Blob as equivalent of a Table in the database world. New output files will be added that only contain the new data.
Column | Condition | Description |
---|---|---|
data | Always exists | The log or event data |
_tarsal_metadata | Always exists | Tarsal-defined column for each record; an event processing timestamp and UUID |
Root-level fields | Root-level normalization (flattening) | Expanded root-level field |
JSON Lines (JSONL)
JSON Lines, or newline-delimited JSON
, is a text format for structured data. JSONL
has one JSON
per line. The output file has the following structure:
{
"_tarsal_metadata": "<json-metadata>"
"data": "<json-data-from-source>"
}
An example of JSON source and output follows:
[
{
"user_id": 123,
"name": {
"first": "John",
"last": "Doe"
}
},
{
"user_id": 456,
"name": {
"first": "Jane",
"last": "Roe"
}
}
]
{ "_tarsal_metadata": {"_tarsal_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_tarsal_emitted_at": "1631948170000"}, "data": { "user_id": 123, "name": { "first": "John", "last": "Doe" } } }
{ "_tarsal_metadata": {"_tarsal_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_tarsal_emitted_at": "1631948170000"}, "data": { "user_id": 456, "name": { "first": "Jane", "last": "Roe" } } }
Updated 4 days ago