> For the complete documentation index, see [llms.txt](https://developer.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.mediarithmics.io/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication/bigquery.md).

# BigQuery

{% hint style="info" %}
The replication to BigQuery requires an object storage to store the log files. We support only **GCS** for this use case. We recommend creating a dedicated bucket, but you can also use a existing one.
{% endhint %}

## Prerequisite

In order to set up the replication you will need :  

* To have declared [the connection to your BigQuery data warehouse](/advanced-usages/data-warehouse-management/create-a-data-warehouse/bigquery.md) in mediarithmics
  * When declaring the warehouse we expect you declare the **mics\_output dataset** in which we should have the necessary rights (<mark style="color:red;">`bigquery.dataEditor`</mark>) to write the replicated tables
* An object storage to store the AVRO log files. We only support **GCS bucket** for this use case. We recommend creating a dedicated bucket, but you can also use a existing one.

{% hint style="info" %}
**Nota bene** : the log tables are BigQuery external tables that read the GCS bucket in real time. This means that if you wish to work with log files, the log tables are always up to date with what is in the bucket.&#x20;
{% endhint %}

## Overview

We assume the prerequisite steps are completed. If so, then the steps to complete the setup of the replication are :&#x20;

1. Grant permissions on the GCS bucket to the service account used for the mediarithimics <-> BigQuery connection
2. Create an HMAC key pair associated to the service account
3. Configure the replication from the Computing Console interface

## Step #1 : manage permissions on the GCS bucket

The service account [created for the connection](https://developer.mediarithmics.io/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication/pages/D4mwgs6dcPYieN1rS8nD#id-3.-service-account-creation-and-permissions) should have the following role on the bucket :&#x20;

* <mark style="color:red;">`storage.objectAdmin`</mark>&#x20;
* <mark style="color:red;">`storage.viewer`</mark>&#x20;

To do so, in the Cloud Storage interface :&#x20;

* Go to Cloud Storage>Buckets, select the one bucket you wish to use&#x20;
* Then on the "Permissions" tab > "View by principals" > "Grant access"
* Enter the service account address, then select the two necessary roles&#x20;

{% hint style="info" %}
You will need permissions included in the <mark style="color:red;">`storage.admin`</mark> role to perform these actions
{% endhint %}

Find more information in the Cloud Storage documentation : <https://docs.cloud.google.com/storage/docs/access-control/using-iam-permissions>

{% hint style="info" %}
We assume the **same** service account will be used for querying the datasets and holding the rights on the bucket for an easier setup. If you wish to decouple these functions, you can create a new service account  for holding the editing rights on the bucket, and give viewing rights to the previously existing service account
{% endhint %}

## Step #2 : create an HMAC key pair

In the Cloud Storage interface :&#x20;

* Go to Cloud Storage>Settings, then in the "Interoperability" tab&#x20;
* Then in Access keys for service accounts > "Create a key for another service account"
* Select your service account
* Click on Create (the Secret key is only shown once at creation).
* **Keep these credentials somewhere as you will need them later**

Do not change the "Request endpoint" : keep existing request URI <https://storage.googleapis.com>

{% hint style="info" %}
You will need permissions included in the <mark style="color:red;">`storage.admin`</mark> role to perform these actions
{% endhint %}

Find more information in the Cloud Storage documentation : <https://docs.cloud.google.com/storage/docs/authentication/managing-hmackeys>

## Step #3 : Configure the replication in mediarithmics

You are now able to [configure your replication](/advanced-usages/exporting-your-data/datamart-replication.md). Just make sure to use the correct data warehouse destination, bucket, dataset with editing rights, and re-use the HMAC key-pair you generated in step #2.&#x20;

Now you should be able to [activate](/advanced-usages/exporting-your-data/datamart-replication.md#creating-and-starting-a-replication) the replication and trigger an [initial loading](/advanced-usages/exporting-your-data/datamart-replication.md#launch-initial-synchronization-on-one-replication).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://developer.mediarithmics.io/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication/bigquery.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.