Transformations
Tarsal provides out-of-the-box transformation capabilities, allowing you to accommodate analysis tools or convert data types as needed. Transforming data usually involves altering the structure, format, and content of the data to fit specific requirements.
Example Use Cases:
Cleansing: Your data may contain unclean values or values in inconsistent formats. For example, you may want to convert all booleans (true/false) to numeric (1/0) or trim special characters or spaces from values.
Filtering: You can filter Events that you do not want to replicate to your Destination based on simple or complex logic. For example, you can drop all records where author_name = "shakespeare"
Mutations: Many mutations are available. For example, you may want to convert dates to a common UTC format, or uppercase all your keys.
Enrichment: IP address fields can be enriched with GeoIP enrichment.
List of Tarsal Transformations:
| Transformation | Type | Description |
|---|---|---|
| Base64 Encode | Encode | Encodes fields using base64 |
| Capitalize | Mutate | Convert a string to its capitalized equivalent |
| Cipher (AES-GCM) | Encode | Encrypt a field using AES-GCM |
| Coerce | Cleanse | Set a default value for empty or missing fields |
| Copy | Mutate | Copy an existing field to another field |
| Date | Mutate | Copy/Transform dates |
| Dedupe | Cleanse | Removes duplicate lines |
| Drop Fields | Mutate | |
| Filter Record | Filter | |
| Filter PII | Encode/Mutate | Mask or drop fields containing Personally Identifiable Information (PII) |
| GeoIP - ASN | Enrich | ASN Lookup |
| GeoIP - City | Enrich | City lookup using GeoIP |
| GeoIP - Country | Enrich | Country lookup using GeoIP |
| Regex Replace | Mutate | Use regex to match and replace |
| Join | Mutate | Join array / object to single string |
| Lowercase | Mutate | Convert to lowercase |
| Merge | Mutate | Merge multiple fields |
| Prune | Mutate | Prune fields using allow/block lists |
| Rename | Mutate | Rename a field |
| Replace | Mutate | Replace the value of a field |
| Sample | Sample records | |
| Split | Mutate | Split to array |
| Strip | Mutate | Remove leading or trailing spaces |
| Uppercase | Mutate | Convert to uppercase |
How it Works
Transformations occur during a Flow. When a source and a destination are configured into a flow, the source will start pulling data where the data will then be staged for transformation processing prior to being routed to the destination.
Transformations must be first drafted and then published prior to taking affect.
Transformations are broken down and configured into steps. Steps are executed serially, one after another.
How to Configure
To configure a transformation, you must first create a flow. Once a flow is created, the ability to create a new transformation draft, or edit an existing transformation draft, will be available as an action in the stream selection table.
Editing transformations is always done via a draft. This is to allow for the opportunity to configure, test, and tune the transformation steps prior to taking affect. It also means the current published transformations can continue to operate as intended while working on updates.
When creating a draft transformation, you add "transformation steps." This can be done by either clicking on a step to add to the end of the "steps list" or you can drag and drop the steps into their correct execution order.
The steps are displayed in order (in the UI from top to bottom, but also numbered) which will be the serial execution of the steps. Steps can be individually disabled and that step will not be processed. Steps can be re-ordered via drag and drop.
Once a transformation draft is considered ready to use, you simply publish the draft and that version of the transformation will go live immediately, and will take effect the next time the source pulls or receives data.
Transformation changes are recorded, and any previous version of the transformations can be restored if necessary.
Attribution
This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.
Updated about 1 month ago