> For the complete documentation index, see [llms.txt](https://developer.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.mediarithmics.io/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication.md). # External message queues replication ## Available objects We replicate the `update` and `delete` operations from your [Datamart ](/communities-and-organisations/datamart.md)for the following objects: * [UserPoint](/user-points.md) * [UserAgent](/user-points/user-identifiers/userdevicetechnicalid.md#user-agents-legacy) *(for datamart `user_point_system_version` before v202205)* * [UserDevicePoint](/user-points/user-identifiers/userdevicetechnicalid.md#user-device-point) *(for datamart `user_point_system_version` v202205)* * [UserDeviceTechnicalId](/user-points/user-identifiers/userdevicetechnicalid.md#user-device-technical-identifier) *(for datamart `user_point_system_version` v202205)* * [UserActivity](/user-points/user-activities.md) * [UserSegment](/user-points/segments.md) * [UserProfile](/user-points/user-profiles.md) * [UserAccount](/user-points/user-identifiers/useraccount.md) * [UserEmail](/user-points/user-identifiers/useremail.md) * Parent UserPoint (when [UserPoint are merging](/user-points.md#user-points-merges)) When creating a datamart replication, you can select which object(s) to replicate. {% hint style="warning" %} Note that you cannot update the objects selection once set. You will need to create a new datamart replication in that case. {% endhint %} ## Versionning There are currently 2 versions of Datamart replications:
VersionFormatSupported destination
Version 1 (Legacy)JSON formatGoogle Cloud Platform - Pub/Sub
Microsoft Azure - Event Hubs
Version 2Avro Binary formatGoogle Cloud Platform - Pub/Sub
{% hint style="info" %} Please note that: * Any new GCP Pub/Sub replication will be created in Version 2 * You cannot upgrade a datamart replication from Version 1 to Version 2 {% endhint %} ## Output messages We convert datamart operations in a standardized output format: *`operation = {ts, doc_type, doc_id, op, value}`*
FieldTypeCommentVersion availability
tsUNIX Timestamp in ms (Long)The mutation dateAll
doc_typeEnumerationThe object type :UserActivity, UserProfile, UserSegment, UserAgent, UserAccount, UserEmail, UserPoint or UserPointParentAll
ctx_idUUIDThe userpoint idVersion 2
doc_idStringThe object unique id. The format varies depending on doc_type All
opStringThe operation type UPDATE or DELETEAll
valueJSON ObjectThe object value. The format varies depending on doc_type.All
### Object formats based on `doc_type` | doc\_type | doc\_id | value | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | `UserPoint` | `{{user_point_id}}` | Empty (you already have the `user_point_id` in the `doc_id`) | | `UserAgent` | `{{user_point_id}}:{{vector_id}}` | Browser info and device info | | `UserDevicePoint` | `{{user_point_id}}:{{user_device_point_id}}` | Browser info and device info | | `UserDeviceTechnicalId` | `{{user_point_id}}:{{user_device_point_id}}:{{user_device_technical_id}}` | Empty (you already have the `user_device_technical_id` in the `doc_id`) | | `UserActivity` | `{{user_point_id}}:{{user_activity_id}}` | Detailed activity | | `UserSegment` | `{{user_point_id}}:{{segment_id}}` | Segment info | | `UserProfile` | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}` | Detailed profile | | `UserAccount` | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}` | Empty (you already have the `user_account_id` in the `doc_id`) | | `UserEmail` | `{{user_point_id}}:{{email_hash}}` | User's email hash | | `UserPointParent` |

{{user\_point\_id}}

It is the ID of the UserPoint which is merged on the oldest one (the kept one).

| Message ` merged with ` | ### Message metadata To help filtering the topic, replication adds some metadata on message (attributes in PubSub and properties in EventHub) | Metadata key | Comment | | ------------ | ---------------------- | | `doc_type` | The message `doc_type` | ### Legacy JSON Format This format is the original replication format. It was designed to work with a streaming architecture (like Dataflow or Databricks) but has some limitations with tools needing a schema (like BigQuery) #### Examples A new activity will trigger a replicated `UserActivity` operation. You will receive a similar message in your external solution as shown in this example. 
{
   "ts": 1676627112685,
   "doc_type": "UserActivity",
   "doc_id": "XXXXXXX-XXXX-XXX-XXXXXXXX:XXXXXX-XXXXX-XXXX-XXXX-XXXXXXXXXX",
   "op":" UPDATE",
   "value":{
       "$type":"SITE_VISIT",
       "$source":"XXXX",
       "etc": "etc"
   }
}
A new user agent will trigger a replicated UserAgent operation like the one bellow ```javascript { "ts":1676627112685, "doc_type":"UserAgent", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893", "op":"UPDATE", "value":{ "$os_family":"LINUX", "$brand":null, "$os_version":null, "$form_factor":"PERSONAL_COMPUTER", "$carrier":null, "$model":null, "$creation_ts":0, "$browser_family":"FIREFOX" } } ``` A new user device point will trigger a replication UserDevicePoint operation like the one bellow ```javascript { "ts":1676627112685, "doc_type":"UserDevicePoint", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893", "op":"UPDATE", "value":{ "$os_family":"LINUX", "$brand":null, "$os_version":null, "$form_factor":"PERSONAL_COMPUTER", "$carrier":null, "$model":null, "$creation_ts":0, "$browser_family":"FIREFOX" } } ``` A new user device technical id will trigger a replicated `UserDeviceTechnicalId` operation like the ones bellow ```javascript // exemple with a MumId { "ts":1676627112685, "doc_type":"UserDeviceTechnicalId", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539", "op":"UPDATE", "value":{} } // exemple with an installationId { "ts":1676627112685, "doc_type":"UserDeviceTechnicalId", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:ins:1001:aZmFhOTVlM2ItMGRhOC00NDZlLWFhODMtNjZlZGI0YjNiNTk2", "op":"UPDATE", "value":{} } ``` ### Avro Binary Format This version introduce a schema to help integration. ```json { "type": "record", "name": "OperationRecord", "namespace": "com.mediarithmics.replication.format", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "doc_type", "type": { "name": "DocumentType", "type": "enum", "symbols": [ "UserPoint", "UserActivity", "UserProfile", "UserSegment", "UserDevicePoint", "UserDeviceTechnicalId", "UserAgent", "UserAccount", "UserEmail" ] } }, { "name": "doc_id", "type": "string", "doc": "It will always start with the ctx_id (ie: user_point) followed by ':' and other internal ids. It identifies uniquely a document." }, { "name": "ctx_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "op", "type": "string", "doc": "UPDATE or DELETE" }, { "name": "value", "type": "string", "doc": "The object in JSON format" } ] } ``` This format is almost the same as Legacy one, but with Avro binary format. {% hint style="warning" %} The target topic should reference the schema and the encoding as BINARY to take full advantages of the format. {% endhint %} ## Upgrade of datamarts to **`user_point_system_version` v202205** For datamarts with `user_point_system_version` anterior to v202205, device identifiers are stored as [User Agents](/user-points/user-identifiers/userdevicetechnicalid.md#user-agents-legacy), and replicated as **UserAgent operations** (doc\_id exemple: `4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893`). However for datamarts leveraging the `user_point_system_version` v202205, device identifiers are stored as [User Device Points and User Device Technical Ids](/user-points/user-identifiers/userdevicetechnicalid.md), and replicated through **UserDevicePoint and UserDeviceTechnicalId operations**. In the case of a datamart that is upgraded to the`user_point_system_version` v202205: * New device identifiers are directly stored and replicated using the device point formats, * Existing device identifiers that were previously stored in the UserAgent format are progressively migrated. This migration is seemless within the datamart, however it is reflected on your datamart replication. **For each migrated device identifier, you will receive:** * **A DELETE operation with the doc\_type User Agent** * **Two UPDATE operations with doc\_type UserDevicePoint and doc\_type UserDeviceTechnicalId** {% hint style="info" %} For instance, a migration of a user agent with a doc\_id `4700c85f-17e3-4304-aa7f-dc140173b08d:vec:`7231822539 will produce * 1 DELETE operation with doc\_type UserAgent the same doc\_id * 2 UPDATE operations: * 1 with doc\_type UserDevicePoint and the following doc\_id: `4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893` * 1 with doc\_type UserDeviceTechnicalId and the following doc\_id `4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539` {% endhint %} After migration, no more UserAgent operations will be produced ## Setting up replications You need to have an instance of the external solution where you want to replicate your mediarithmics data. You will need to get credentials from the external solution : * [Google Pub/Sub](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication/google-pub-sub.md) * [Microsoft Azure Event Hubs (Alpha)](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication/azure-events-hub.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://developer.mediarithmics.io/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.