# Datamart replication The datamart replication feature allows you to replicate the data ingested by mediarithmics in an external solution of your choice. For now, we are integrated with: * Google Cloud Platform - Pub/Sub * Microsoft Azure - Event Hubs (Alpha) {% hint style="info" %} This module is not included in the default plan. Contact your Account manager to activate it. {% endhint %} ## How it works ### Replication We replicate the `update` and `delete` operations from your [Datamart ](https://developer.mediarithmics.io/communities-and-organisations/datamart)for the following objects: * [UserPoint](https://developer.mediarithmics.io/user-points) * [UserAgent](https://developer.mediarithmics.io/user-points/user-identifiers/userdevicetechnicalid#user-agents-legacy) *(for datamart `user_point_system_version` before v202205)* * [UserDevicePoint](https://developer.mediarithmics.io/user-points/user-identifiers/userdevicetechnicalid#user-device-point) *(for datamart `user_point_system_version` v202205)* * [UserDeviceTechnicalId](https://developer.mediarithmics.io/user-points/user-identifiers/userdevicetechnicalid#user-device-technical-identifier) *(for datamart `user_point_system_version` v202205)* * [UserActivity](https://developer.mediarithmics.io/user-points/user-activities) * [UserSegment](https://developer.mediarithmics.io/user-points/segments) * [UserProfile](https://developer.mediarithmics.io/user-points/user-profiles) * [UserAccount](https://developer.mediarithmics.io/user-points/user-identifiers/useraccount) * [UserEmail](https://developer.mediarithmics.io/user-points/user-identifiers/useremail) * Parent UserPoint (when [UserPoint are merging](https://developer.mediarithmics.io/user-points#user-points-merges)) ### Object filtering When creating a datamart replication, you can select which object(s) to replicate. {% hint style="warning" %} Note that you cannot update the objects selection once set. You will need to create a new datamart replication in that case. {% endhint %} ## Versionning There are currently 2 versions of Datamart replications:
VersionFormatSupported destination
Version 1 (Legacy)JSON formatGoogle Cloud Platform - Pub/Sub
Microsoft Azure - Event Hubs
Version 2Avro Binary formatGoogle Cloud Platform - Pub/Sub
{% hint style="info" %} Please note that: * Any new GCP Pub/Sub replication will be created in Version 2 * You cannot upgrade a datamart replication from Version 1 to Version 2 {% endhint %} ## Replication status Your replication can be in one of the following status: * **ACTIVE**: All data processed by your datamart will be replicated to your external solution. * **PAUSED**: No data processed by your datamart will be replicated to your external solution. * **ERROR**: The system is no longer able to replicate messages. In this case, check your external solution (expired instance, invalid credentials, etc). If you can't find anything wrong, please contact your Account manager. ## Initial synchronization You can run an **initial synchronization** for one or multiple **ACTIVE** datamart replications. This operation replicates all existing documents selected in the datamart replication and stored within the **mediarithmics** platform into your cloud environment. Depending on your needs, you have several ways to trigger an initial synchronization: **For a single datamart replication:** 1. Locate the replication in the list. 2. Click the **caret-down** icon, then select **New Initial Synchronization**. **For multiple datamart replications:** 1. Click the **New Initial Synchronization** button. 2. Select the datamart replications you want to synchronize. 3. Click **Execute Initial Synchronization(s)**. All selected datamart replications will receive a set of `UPDATE` operations representing all existing elements (for example, `UserProfile`) in your datamart. {% hint style="warning" %} Please note that if you run an initial synchronization you might receive a large volume of messages. Processing them can be expensive, depending on your cloud provider. {% endhint %} ## Output messages We convert datamart operations in a standardized output format: *`operation = {ts, doc_type, doc_id, op, value}`*
FieldTypeCommentVersion availability
tsUNIX Timestamp in ms (Long)The mutation dateAll
doc_typeEnumerationThe object type :UserActivity, UserProfile, UserSegment, UserAgent, UserAccount, UserEmail, UserPoint or UserPointParentAll
ctx_idUUIDThe userpoint idVersion 2
doc_idStringThe object unique id. The format varies depending on doc_type All
opStringThe operation type UPDATE or DELETEAll
valueJSON ObjectThe object value. The format varies depending on doc_type.All
### Object formats based on `doc_type` | doc\_type | doc\_id | value | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | `UserPoint` | `{{user_point_id}}` | Empty (you already have the `user_point_id` in the `doc_id`) | | `UserAgent` | `{{user_point_id}}:{{vector_id}}` | Browser info and device info | | `UserDevicePoint` | `{{user_point_id}}:{{user_device_point_id}}` | Browser info and device info | | `UserDeviceTechnicalId` | `{{user_point_id}}:{{user_device_point_id}}:{{user_device_technical_id}}` | Empty (you already have the `user_device_technical_id` in the `doc_id`) | | `UserActivity` | `{{user_point_id}}:{{user_activity_id}}` | Detailed activity | | `UserSegment` | `{{user_point_id}}:{{segment_id}}` | Segment info | | `UserProfile` | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}` | Detailed profile | | `UserAccount` | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}` | Empty (you already have the `user_account_id` in the `doc_id`) | | `UserEmail` | `{{user_point_id}}:{{email_hash}}` | User's email hash | | `UserPointParent` |

{{user\_point\_id}}

It is the ID of the UserPoint which is merged on the oldest one (the kept one).

| Message ` merged with ` | ### Message metadata To help filtering the topic, replication adds some metadata on message (attributes in PubSub and properties in EventHub) | Metadata key | Comment | | ------------ | ---------------------- | | `doc_type` | The message `doc_type` | ### Legacy JSON Format This format is the original replication format. It was designed to work with a streaming architecture (like Dataflow or Databricks) but has some limitations with tools needing a schema (like BigQuery) #### Examples A new activity will trigger a replicated `UserActivity` operation. You will receive a similar message in your external solution as shown in this example. 
{
   "ts": 1676627112685,
   "doc_type": "UserActivity",
   "doc_id": "XXXXXXX-XXXX-XXX-XXXXXXXX:XXXXXX-XXXXX-XXXX-XXXX-XXXXXXXXXX",
   "op":" UPDATE",
   "value":{
       "$type":"SITE_VISIT",
       "$source":"XXXX",
       "etc": "etc"
   }
}
A new user agent will trigger a replicated UserAgent operation like the one bellow ```javascript { "ts":1676627112685, "doc_type":"UserAgent", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893", "op":"UPDATE", "value":{ "$os_family":"LINUX", "$brand":null, "$os_version":null, "$form_factor":"PERSONAL_COMPUTER", "$carrier":null, "$model":null, "$creation_ts":0, "$browser_family":"FIREFOX" } } ``` A new user device point will trigger a replication UserDevicePoint operation like the one bellow ```javascript { "ts":1676627112685, "doc_type":"UserDevicePoint", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893", "op":"UPDATE", "value":{ "$os_family":"LINUX", "$brand":null, "$os_version":null, "$form_factor":"PERSONAL_COMPUTER", "$carrier":null, "$model":null, "$creation_ts":0, "$browser_family":"FIREFOX" } } ``` A new user device technical id will trigger a replicated `UserDeviceTechnicalId` operation like the ones bellow ```javascript // exemple with a MumId { "ts":1676627112685, "doc_type":"UserDeviceTechnicalId", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539", "op":"UPDATE", "value":{} } // exemple with an installationId { "ts":1676627112685, "doc_type":"UserDeviceTechnicalId", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:ins:1001:aZmFhOTVlM2ItMGRhOC00NDZlLWFhODMtNjZlZGI0YjNiNTk2", "op":"UPDATE", "value":{} } ``` ### Avro Binary Format This version introduce a schema to help integration. ```json { "type": "record", "name": "OperationRecord", "namespace": "com.mediarithmics.replication.format", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "doc_type", "type": { "name": "DocumentType", "type": "enum", "symbols": [ "UserPoint", "UserActivity", "UserProfile", "UserSegment", "UserDevicePoint", "UserDeviceTechnicalId", "UserAgent", "UserAccount", "UserEmail" ] } }, { "name": "doc_id", "type": "string", "doc": "It will always start with the ctx_id (ie: user_point) followed by ':' and other internal ids. It identifies uniquely a document." }, { "name": "ctx_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "op", "type": "string", "doc": "UPDATE or DELETE" }, { "name": "value", "type": "string", "doc": "The object in JSON format" } ] } ``` This format is almost the same as Legacy one, but with Avro binary format. {% hint style="warning" %} The target topic should reference the schema and the encoding as BINARY to take full advantages of the format. {% endhint %} ## Upgrade of datamarts to **`user_point_system_version` v202205** For datamarts with `user_point_system_version` anterior to v202205, device identifiers are stored as [User Agents](https://developer.mediarithmics.io/user-points/user-identifiers/userdevicetechnicalid#user-agents-legacy), and replicated as **UserAgent operations** (doc\_id exemple: `4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893`). However for datamarts leveraging the `user_point_system_version` v202205, device identifiers are stored as [User Device Points and User Device Technical Ids](https://developer.mediarithmics.io/user-points/user-identifiers/userdevicetechnicalid), and replicated through **UserDevicePoint and UserDeviceTechnicalId operations**. In the case of a datamart that is upgraded to the`user_point_system_version` v202205: * New device identifiers are directly stored and replicated using the device point formats, * Existing device identifiers that were previously stored in the UserAgent format are progressively migrated. This migration is seemless within the datamart, however it is reflected on your datamart replication. **For each migrated device identifier, you will receive:** * **A DELETE operation with the doc\_type User Agent** * **Two UPDATE operations with doc\_type UserDevicePoint and doc\_type UserDeviceTechnicalId** {% hint style="info" %} For instance, a migration of a user agent with a doc\_id `4700c85f-17e3-4304-aa7f-dc140173b08d:vec:`7231822539 will produce * 1 DELETE operation with doc\_type UserAgent the same doc\_id * 2 UPDATE operations: * 1 with doc\_type UserDevicePoint and the following doc\_id: `4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893` * 1 with doc\_type UserDeviceTechnicalId and the following doc\_id `4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539` {% endhint %} After migration, no more UserAgent operations will be produced ## Setting up replications ### Prerequisites You need to have an instance of the external solution where you want to replicate your mediarithmics data. Depending on the external solution, you will need to fulfill some requirements. #### Google Pub/Sub You will need: * A Google Cloud Platform account * A Google Cloud Platform project: ; * An Access Control on this project: ; * To create and activate your service accounts (generate credentials file): > **TO SUM IT UP:** You can click here , create a service account, and edit it to create a Key in a JSON format (this is the credential file); Click on **Create Service Account** : ![](https://4196284719-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MMuoqM-5hJ5JY0WnAKL%2Fuploads%2FV96ZCwWFyNcNgd2XyYMQ%2Fimage.png?alt=media\&token=fc4b4b2f-1cdb-4be1-885d-6ae39651e986) Give your service account a name, select the right account access (Pub/Sub Publisher, Pub/Sub Editor) and save. ![](https://4196284719-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MMuoqM-5hJ5JY0WnAKL%2Fuploads%2Fhu7APYJybKFa4SVSrY7j%2Fimage.png?alt=media\&token=3939b2ff-ac1a-466e-a953-75c629f0dc5f) Once your Service Account is created, you can generate your key : ![](https://4196284719-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MMuoqM-5hJ5JY0WnAKL%2Fuploads%2Fz8MmgJjd6CvbDoO5isXo%2Fimage.png?alt=media\&token=26b3a184-6123-4c55-83f4-716422b060fa) ![](https://4196284719-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MMuoqM-5hJ5JY0WnAKL%2Fuploads%2FCrGfPweuDaT1bPmJKkcm%2Fimage.png?alt=media\&token=76464c66-6363-46a8-bf32-9540311d2315) ![](https://4196284719-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MMuoqM-5hJ5JY0WnAKL%2Fuploads%2FCOvJXQX79tFSpdeGLbDq%2Fimage.png?alt=media\&token=af83a5c2-368f-4d6c-8599-a379662e5b7e) credentials.json file example: ```javascript { "type": "service_account", "project_id": "xxx-xxx-xx", "private_key_id": "xxxxxxx", "private_key": "-----BEGIN PRIVATE KEY-----\n xxxxxxx \n-----END PRIVATE KEY-----\n", "client_email": "xxx@project_id.iam.gserviceaccount.com", "client_id": "xxxxxx", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/projetc_id.iam.gserviceaccount.com" } ``` * To create a Google Cloud Platform Pub/Sub instance. Pub/Sub documentation: until Quickstart setup > Create service account credentials (included) should be enough to begin. ***NOTE:*** Here is the Google Pub/Sub Pricing documentation: #### Microsoft Azure Event Hubs (Alpha) You will need: * A Microsoft Azure account * A Resource Group, an Event Hubs namespace, and an Event Hub: * A connection string over the namespace or the Event Hub: * Save your connection string in a credentials.txt file. It would be your credentials file to upload in mediarithmics platform. credentials.txt file example: ```javascript Endpoint=sb:///;SharedAccessKeyName=;SharedAccessKey= ``` ***NOTE:*** Here is the Microsoft Azure Event Hubs Pricing documentation: ### Listing your replications You can access replications in the datamart settings in your navigator application. 1. Select the **organisation** on which there is the datamart you want to replicate. 2. Click on **Settings**. 3. Click on the **Datamarts** tab and then click the **Datamart** menu entry. 4. Select the datamart you want to replicate. 5. In the **Replications** subtab, you will see a table dedicated to your Datamart **Replications**. ### Creating & starting a replication To create a new replication: 1. Go to the [**Replications**](#listing-your-replications) subtab. 2. Click **New Replication**. 3. Select a **Replication type** matching the external solution of your choice. 4. Complete configuration information (see [Prerequisites](#prerequisites) as help to configure advanced fields). 5. Click **Select a File** to upload your credentials file and click on **Update**. 6. Click **Save Replication** to create your new replication. 7. You will see your new replication in the **Replications** subtab. example for Google Pub/Sub : ![](https://4196284719-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MMuoqM-5hJ5JY0WnAKL%2Fuploads%2FoKDQiKLS6ItzdeA0quFy%2Fimage.png?alt=media\&token=b1c3322a-632d-4bda-bee3-3230fdd7b371) {% hint style="success" %} When a Replication is created, its status is automatically set to **Paused**. To start your replication, you will have to activate it. If the system can't replicate your datamart on activation, you will see an error. {% endhint %} {% hint style="info" %} When a replication can't be activated, it is usually due to an error on **credentials**, so you might want to verify your replication configuration and your credentials file first. {% endhint %} ### Activating / pausing a replication You can change the replication status using the status button. If the system is no longer able to replicate the messages, the replication status will be set to **ERROR**. In this case, check your external solution (expired instance, invalid credentials, etc). If you can't find anything wrong, please contact your Account manager. {% hint style="info" %} In case of an error with your external solution, you will need to recreate your replication to rebind it to a new working external solution with good credentials and the right specific information. {% endhint %} ### Executing an initial synchronization A dashboard listing every **Initial Synchronization** that was done on your datamart is available in the same **Replications** subtab. For now, you will have to ask your Account manager to run an initial synchronization. Later, you will be able to run an initial synchronization yourself by clicking **New Execution**. You must have at least one active replication and you can't run an initial synchronization more than once a week. {% hint style="warning" %} We replicate all operations. No filtering is possible. Please note that **you might receive a large volume of messages** while running an initial synchronization. Processing them can be expensive, depending on your cloud provider. {% endhint %} **While an initial synchronization is running, you can't change the status of your replications**. The initial synchronization will only replicate the data for active replications. The active replications are still running during initial synchronizations. Messages from the initial synchronization and live messages (tag, import, etc.) from the active replications are mixed. ## API documentation ### Create a replication `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications` #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | #### Request Body
NameTypeDescription
versionintegerVersion of the datamart replication to be created. Check #versionning for more info.
namestringThe name of the API Token
statusstringStatus of the replication. Need to be "PAUSED" at the creation
typestringGOOGLE_PUBSUB or AZURE_EVENT_HUBS
project_idstringGoogle project ID. Only for GOOGLE_PUBSUB
topic_idstringGoogle PubSub topic ID.Only for GOOGLE_PUBSUB
event_hub_namestringAzure event hub name. Only for AZURE_EVENT_HUBS
datamart_idstringAs per the Path parameter
replication_filtersarray

Optional.

List of documents to replicate. If not provided, all documents will be replicated

replication_filters > documentstringName of the document to replicate among USER_SEGMENT, USER_EMAIL, USER_ACCOUNT, USER_PROFILE, USER_DEVICE_POINT, USER_AGENT, USER_DEVICE_TECHNICAL_ID, USER_ACTIVITY, USER_POINT
replication_filters > filterstringNot used at the moment
{% tabs %} {% tab title="Request Body example " %} ``` { "status": "PAUSED", "name": , "project_id": "test", "topic_id": "test", "datamart_id": "1649", "type": "GOOGLE_PUBSUB", "replication_filters": [{ "document": "USER_SEGMENT", "filter": null, }, { "document": "USER_ACCOUNT", "filter": null, }, { … }] } ``` {% endtab %} {% endtabs %} ### Add credentials to replication `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId/credentials` #### Path Parameters | Name | Type | Description | | ------------- | ------- | ---------------------------------- | | datamartId | integer | The ID of the datamart | | replicationId | integer | The ID of the datamart replication | #### Request Body | Name | Type | Description | | ---- | ---- | ------------------------------------------------- | | file | form | The path of the file that contain the credentials | ### Curl example ```bash curl --location 'https://api.mediarithmics.com/v1/datamarts/:datamartID/replications/:replicationID/credentials' \ --header 'Authorization: api:XXX' \ --form 'file=@"/XXX/service-account.json"' ``` ### Retrieve a replication `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId` #### Path Parameters | Name | Type | Description | | ------------- | ------- | ---------------------------------- | | datamartId | integer | The ID of the datamart | | replicationId | integer | The ID of the datamart replication | {% tabs %} {% tab title="200" %} ``` { "id": "1234", "version" 2, "status": "PAUSED", "name": , "project_id": "test", "topic_id": "test", "datamart_id": "1649", "type": "GOOGLE_PUBSUB", "credentials_uri": "internal_path_to_credential_uri" "replication_filters": [{ "id": "1", "replication_id": "1234", "document": "USER_SEGMENT", "filter": null, }, { "id": "2", "replication_id": "1234", "document": "USER_ACCOUNT", "filter": null, }, { … }] } ``` {% endtab %} {% endtabs %} ### Launch initial synchronization on one replication `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replication_job_executions` #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | #### Request Body
NameTypeDescription
replication_idsintegerThe IDs of the replication
{% tabs %} {% tab title="Request Body example " %} ``` { "replication_ids": ["123", "456"] } ``` {% endtab %} {% endtabs %}