External message queues replication

Available objects

We replicate the update and delete operations from your Datamart for the following objects:

• UserPoint

• UserAgent (for datamart user_point_system_version before v202205)

• UserDevicePoint (for datamart user_point_system_version v202205)

• UserDeviceTechnicalId (for datamart user_point_system_version v202205)

• UserActivity

• UserSegment

• UserProfile

• UserAccount

• UserEmail

• Parent UserPoint (when UserPoint are merging)

When creating a datamart replication, you can select which object(s) to replicate.

Note that you cannot update the objects selection once set. You will need to create a new datamart replication in that case.

Versionning

There are currently 2 versions of Datamart replications:

Version	Format	Supported destination
Version 1 (Legacy)	JSON format	Google Cloud Platform - Pub/Sub Microsoft Azure - Event Hubs
Version 2	Avro Binary format	Google Cloud Platform - Pub/Sub

Please note that:

• Any new GCP Pub/Sub replication will be created in Version 2

• You cannot upgrade a datamart replication from Version 1 to Version 2

Output messages

We convert datamart operations in a standardized output format: operation = {ts, doc_type, doc_id, op, value}

Field	Type	Comment	Version availability
ts	UNIX Timestamp in ms (Long)	The mutation date	All
doc_type	Enumeration	The object type :UserActivity, UserProfile, UserSegment, UserAgent, UserAccount, UserEmail, UserPoint or UserPointParent	All
ctx_id	UUID	The userpoint id	Version 2
doc_id	String	The object unique id. The format varies depending on doc_type	All
op	String	The operation type UPDATE or DELETE	All
value	JSON Object	The object value. The format varies depending on doc_type.	All

Object formats based on doc_type

doc_type	doc_id	value
UserPoint	{{user_point_id}}	Empty (you already have the user_point_id in the doc_id)
UserAgent	{{user_point_id}}:{{vector_id}}	Browser info and device info
UserDevicePoint	{{user_point_id}}:{{user_device_point_id}}	Browser info and device info
UserDeviceTechnicalId	{{user_point_id}}:{{user_device_point_id}}:{{user_device_technical_id}}	Empty (you already have the user_device_technical_id in the doc_id)
UserActivity	{{user_point_id}}:{{user_activity_id}}	Detailed activity
UserSegment	{{user_point_id}}:{{segment_id}}	Segment info
UserProfile	{{user_point_id}}:{{compartment_id}}:{{user_account_id}}	Detailed profile
UserAccount	{{user_point_id}}:{{compartment_id}}:{{user_account_id}}	Empty (you already have the user_account_id in the doc_id)
UserEmail	{{user_point_id}}:{{email_hash}}	User's email hash
UserPointParent	{{user_point_id}} It is the ID of the UserPoint which is merged on the oldest one (the kept one).	Message <current_user_point_id> merged with <the_kept_user_point_id>

Message metadata

To help filtering the topic, replication adds some metadata on message (attributes in PubSub and properties in EventHub)

Metadata key	Comment
doc_type	The message doc_type

Legacy JSON Format

This format is the original replication format. It was designed to work with a streaming architecture (like Dataflow or Databricks) but has some limitations with tools needing a schema (like BigQuery)

Examples

A new activity will trigger a replicated UserActivity operation. You will receive a similar message in your external solution as shown in this example.

{
   "ts": 1676627112685,
   "doc_type": "UserActivity",
   "doc_id": "XXXXXXX-XXXX-XXX-XXXXXXXX:XXXXXX-XXXXX-XXXX-XXXX-XXXXXXXXXX",
   "op":" UPDATE",
   "value":{
       "$type":"SITE_VISIT",
       "$source":"XXXX",
       "etc": "etc"
   }
}

A new user agent will trigger a replicated UserAgent operation like the one bellow

{
   "ts":1676627112685,
   "doc_type":"UserAgent",
   "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893",
   "op":"UPDATE",
   "value":{
      "$os_family":"LINUX",
      "$brand":null,
      "$os_version":null,
      "$form_factor":"PERSONAL_COMPUTER",
      "$carrier":null,
      "$model":null,
      "$creation_ts":0,
      "$browser_family":"FIREFOX"
   }
}

A new user device point will trigger a replication UserDevicePoint operation like the one bellow

{
   "ts":1676627112685,
   "doc_type":"UserDevicePoint",
   "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893",
   "op":"UPDATE",
   "value":{
      "$os_family":"LINUX",
      "$brand":null,
      "$os_version":null,
      "$form_factor":"PERSONAL_COMPUTER",
      "$carrier":null,
      "$model":null,
      "$creation_ts":0,
      "$browser_family":"FIREFOX"
   }
}

A new user device technical id will trigger a replicated UserDeviceTechnicalId operation like the ones bellow

// exemple with a MumId 
{
   "ts":1676627112685,
   "doc_type":"UserDeviceTechnicalId",
   "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539",
   "op":"UPDATE",
   "value":{}
}

// exemple with an installationId
{
   "ts":1676627112685,
   "doc_type":"UserDeviceTechnicalId",
   "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:ins:1001:aZmFhOTVlM2ItMGRhOC00NDZlLWFhODMtNjZlZGI0YjNiNTk2",
   "op":"UPDATE",
   "value":{}
}

Avro Binary Format

This version introduce a schema to help integration.

{
  "type": "record",
  "name": "OperationRecord",
  "namespace": "com.mediarithmics.replication.format",
  "fields": [
    {
      "name": "ts",
      "type": {
        "type": "long",
        "logicalType": "timestamp-micros"
      }
    },
    {
      "name": "doc_type",
      "type": {
        "name": "DocumentType",
        "type": "enum",
        "symbols": [
          "UserPoint",
          "UserActivity",
          "UserProfile",
          "UserSegment",
          "UserDevicePoint",
          "UserDeviceTechnicalId",
          "UserAgent",
          "UserAccount",
          "UserEmail"
        ]
      }
    },
    {
      "name": "doc_id",
      "type": "string",
      "doc": "It will always start with the ctx_id (ie: user_point) followed by ':' and other internal ids. It identifies uniquely a document."
    },
    {
      "name": "ctx_id",
      "type": {
        "type": "string",
        "logicalType": "uuid"
      },
      "doc": "The UserPoint id"
    },
    {
      "name": "op",
      "type": "string",
      "doc": "UPDATE or DELETE"
    },
    {
      "name": "value",
      "type": "string",
      "doc": "The object in JSON format"
    }
  ]
}

This format is almost the same as Legacy one, but with Avro binary format.

The target topic should reference the schema and the encoding as BINARY to take full advantages of the format.

Upgrade of datamarts to user_point_system_version v202205

For datamarts with user_point_system_version anterior to v202205, device identifiers are stored as User Agents, and replicated as UserAgent operations (doc_id exemple: 4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893).

However for datamarts leveraging the user_point_system_version v202205, device identifiers are stored as User Device Points and User Device Technical Ids, and replicated through UserDevicePoint and UserDeviceTechnicalId operations.

In the case of a datamart that is upgraded to theuser_point_system_version v202205:

• New device identifiers are directly stored and replicated using the device point formats,

• Existing device identifiers that were previously stored in the UserAgent format are progressively migrated.

This migration is seemless within the datamart, however it is reflected on your datamart replication. For each migrated device identifier, you will receive:

• A DELETE operation with the doc_type User Agent

• Two UPDATE operations with doc_type UserDevicePoint and doc_type UserDeviceTechnicalId

For instance, a migration of a user agent with a doc_id 4700c85f-17e3-4304-aa7f-dc140173b08d:vec:7231822539 will produce

• 1 DELETE operation with doc_type UserAgent the same doc_id

• 2 UPDATE operations:

  • 1 with doc_type UserDevicePoint and the following doc_id: 4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893

  • 1 with doc_type UserDeviceTechnicalId and the following doc_id 4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539

After migration, no more UserAgent operations will be produced

Setting up replications

You need to have an instance of the external solution where you want to replicate your mediarithmics data. You will need to get credentials from the external solution :

• Google Pub/Sub

• Microsoft Azure Event Hubs (Alpha)