> For the complete documentation index, see [llms.txt](https://developer.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.mediarithmics.io/advanced-usages/exporting-your-data/datamart-replication.md).

# Datamart replication

{% hint style="info" %}
This module is not included in the default plan. Contact your Account manager to activate it.
{% endhint %}

Datamart replication allows you to replicate the data ingested by mediarithmics in an external solution of your choice. We currently support replication to :

* [External message queues](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication.md) : Google Pub/Sub and Microsoft Azure Events Hub
* [Data warehouses ](/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication.md): Google BigQuery and Snowflake

## How it works

### Creating & starting a replication

To create a new replication:

1. Go to the Computing console>Data stream menu
2. Click **New Replication**.
3. Select a **Replication type** matching the external solution of your choice.
4. Complete configuration information.
5. Click **Save Replication** to create your new replication.
6. You will see your new replication in the **Replications** subtab.

#### Replication statuses

Your replication can be in one of the following status: 

* **ACTIVE**: All data processed by your datamart will be replicated to your external solution.
* **PAUSED**: No data processed by your datamart will be replicated to your external solution.
* **ERROR**: The system is no longer able to replicate messages. In this case, check your external solution (expired instance, invalid credentials, etc). If you can't find anything wrong, please contact your Account manager.

{% hint style="success" %}
When a Replication is created, its status is automatically set to **Paused**. To start your replication, you will have to activate it. If the system can't replicate your datamart on activation, you will see an error. 
{% endhint %}

{% hint style="info" %}
When a replication can't be activated, it is usually due to an error on **credentials**, so you might want to verify your replication configuration and your credentials file first.
{% endhint %}

### API Documention 

Creating a replication is a 2 step action : first creating the replication object, and then posting credentials attached to the replication.

* In an external message queue replication : the credentials to post are the ones carrying the access rights to the [message queue service](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication.md#setting-up-replications). 
* In a data warehouse replication : the credentials to posts are the ones carrying the access rights to the [object storage used in the replication process.](/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication.md)

#### Create a replication

<mark style="color:green;">`POST`</mark> `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications`

**Path Parameters**

| Name       | Type    | Description            |
| ---------- | ------- | ---------------------- |
| datamartId | Integer | The ID of the datamart |

**Request Body**

<table><thead><tr><th>Name</th><th width="175">Type</th><th>Description</th></tr></thead><tbody><tr><td>type</td><td>Enum </td><td><code>GOOGLE_PUBSUB</code> , <code>AZURE_EVENT_HUBS,</code> <code>DATA_WAREHOUSE</code></td></tr><tr><td>name</td><td>String</td><td>The name of the replication</td></tr><tr><td>replication_filters</td><td>Array</td><td>List of documents to replicate. Array of <code>{ "document": "" , "filter":""}</code></td></tr><tr><td>replication_filters>document</td><td>String</td><td><p><strong>For message queue replication</strong> :  <code>USER_SEGMENT</code>, <code>USER_EMAIL</code>, <code>USER_ACCOUNT</code>, <code>USER_PROFILE</code>, <code>USER_DEVICE_POINT</code>, <code>USER_DEVICE_TECHNICAL_ID</code>, <code>USER_ACTIVITY</code>, <code>USER_POINT, USER_POINT_PARENT</code>, <code>USER_AGENT</code> <em>(legacy)</em><br><strong>For data warehouse replication :</strong> </p><p> <code>USER_SEGMENT</code>, <code>USER_EMAIL</code>, <code>USER_ACCOUNT</code>, <code>USER_PROFILE</code>, <code>USER_DEVICE_POINT</code>, <code>USER_DEVICE_TECHNICAL_ID</code>, <code>USER_ACTIVITY</code>, <code>USER_EVENT, USER_COMPUTED_FIELD</code></p></td></tr><tr><td>replication_filters>filter</td><td>String</td><td>Required but unused. Should be <code>null</code></td></tr><tr><td>version</td><td>Integer (optional)</td><td>Version of the datamart replication to be created. Check <a href="/pages/dBg4QAZUCylEhsl0MsHC#versionning">versionning</a> for more info. <em>NB: "2" is the default except for</em> <code>AZURE_EVENT_HUBS</code> <em>replication</em></td></tr><tr><td>datamart_id</td><td>String (optional)</td><td>As per the Path parameter</td></tr><tr><td>status</td><td>String (optional)</td><td>Status of the replication. Default is "PAUSED"</td></tr><tr><td>project_id</td><td>String</td><td>Google project ID. Only for <code>GOOGLE_PUBSUB</code></td></tr><tr><td>topic_id</td><td>String</td><td>Google PubSub topic ID. Only for <code>GOOGLE_PUBSUB</code></td></tr><tr><td>event_hub_name</td><td>String</td><td>Azure event hub name. Only for <code>AZURE_EVENT_HUBS</code></td></tr><tr><td>data_warehouse_id</td><td>String</td><td>The id of the data warehouse in which the data will be replicated. <br>Only for <code>DATA_WAREHOUSE</code></td></tr><tr><td>schema_id</td><td>String</td><td>The id of the schema/dataset in which the data will be replicated. <br>Only for <code>DATA_WAREHOUSE</code></td></tr><tr><td>bucket_name</td><td>String</td><td>The name of the bucket in which the log files will be sent. Only for <code>DATA_WAREHOUSE</code></td></tr></tbody></table>

**Request Body example**

```json
{
	"name": "test_replication" ,
	"replication_filters": [
		{
		"document": "USER_SEGMENT",
		"filter": null,
		},
		{
		"document": "USER_ACCOUNT",
		"filter": null,
		}
	],
	"data_warehouse_id": "<data_warehouse_id>",				
	"schema_id": "<schema_id>",
	"datamart_id": "<datamart_id>",
	"bucket_name": "<bucket_name>",
	"type": "DATA_WAREHOUSE" 
}
```

#### Add credentials to replication

<mark style="color:green;">`POST`</mark> `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId/credentials`

**Path Parameters**

| Name          | Type    | Description                        |
| ------------- | ------- | ---------------------------------- |
| datamartId    | Integer | The ID of the datamart             |
| replicationId | Integer | The ID of the datamart replication |

**Content-Type**&#x20;

The endpoint accepts two body formats (Content-Type):

<table><thead><tr><th width="208.00006103515625">Content-Type</th><th>Usage</th></tr></thead><tbody><tr><td><code>application/json</code></td><td>Used for <code>DATA_WAREHOUSE</code> replication type</td></tr><tr><td><code>multipart/form-data</code></td><td>Used for <code>GOOGLE_PUBSUB</code> and <code>AZURE_EVENT_HUBS</code> replication type</td></tr></tbody></table>

**Request body**

| Name        | Type   | Description                                                     |
| ----------- | ------ | --------------------------------------------------------------- |
| type        | String | Should be `"OBJECT_STORAGE"`                                    |
| access\_key | String | Access key                                                      |
| secret\_key | String | Secret key                                                      |
| protocol    | Enum   | `"GS"` for Google Cloud Storage bucket, `"S3"` for a AWS bucket |

**Curl examples**

```bash
curl --location 'https://api.mediarithmics.com/v1/datamarts/:datamartID/replications/:replicationID/credentials' \
--header 'Authorization: api:XXX' \
--form 'file=@"/XXX/service-account.json"'
```

```bash
curl --location 'https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId/credentials' \
  --header 'Authorization: api:XXX' \
  --header 'Content-Type: application/json' \
  --data '{
    "type": "OBJECT_STORAGE",
    "access_key": "AAA...",
    "secret_key": "xxxxxxxx",
    "protocol": "S3"
```

#### Retrieve a replication

<mark style="color:blue;">`GET`</mark> `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId`

**Path Parameters**

| Name          | Type    | Description                        |
| ------------- | ------- | ---------------------------------- |
| datamartId    | integer | The ID of the datamart             |
| replicationId | integer | The ID of the datamart replication |

**Request body example**

```json
{
  "status": "ok",
  "data": {
    "type": "GOOGLE_PUBSUB",
    "id": "<replication_id>",
    "name": "<replication_name>",
    "datamart_id": "<datamart_id>",
    "credentials_uri": "<path_to_credentials_file>",
    "project_id": "<project_id>",
    "topic_id": "<topic_id>",
    "status": "PAUSED",
    "version": 2,
    "replication_filters": [
      {
        "id": "81",
        "replication_id": "<replication_id>",
        "document": "USER_DEVICE_TECHNICAL_ID",
        "filter": null,
        "schema_type_name": null
      },
      {
        "id": "82",
        "replication_id": "<replication_id>",
        "document": "USER_SEGMENT",
        "filter": null,
        "schema_type_name": null
      }
    ],
    "creation_ts": 1763384421645,
    "internal_replication": false,
    "connector_type": null
  }
}
```

### Initial synchronization

You can run an **initial synchronization** for one or multiple **ACTIVE** datamart replications. This operation replicates all existing documents selected in the datamart replication and stored within the **mediarithmics** platform into your cloud environment.

Depending on your needs, you have several ways to trigger an initial synchronization:

**For a single datamart replication:**

1. Locate the replication in the list.
2. Click the **caret-down** icon, then select **New Initial Synchronization**.

**For multiple datamart replications:**

1. Click the **New Initial Synchronization** button.
2. Select the datamart replications you want to synchronize.
3. Click **Execute Initial Synchronization(s)**.

All selected datamart replications will receive a set of `UPDATE` operations representing all existing elements (for example, `UserProfile`) in your datamart. You can't run an initial synchronization more than once a week.

{% hint style="warning" %}
Please note that if you run an initial synchronization you might receive a large volume of messages. Processing them can be expensive, depending on your cloud provider.
{% endhint %}

#### Launch initial synchronization on one replication

<mark style="color:green;">`POST`</mark> `https://api.mediarithmics.com/v1/datamarts/:datamartId/replication_job_executions`

**Path Parameters**

| Name       | Type    | Description            |
| ---------- | ------- | ---------------------- |
| datamartId | integer | The ID of the datamart |

**Request Body**

<table><thead><tr><th>Name</th><th width="175">Type</th><th>Description</th></tr></thead><tbody><tr><td>replication_ids</td><td>integer</td><td>The IDs of the replication</td></tr></tbody></table>

```json
{
	"replication_ids": ["123", "456"]
}
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://developer.mediarithmics.io/advanced-usages/exporting-your-data/datamart-replication.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.