# Welcome! Welcome to our documentation web site, whether you are Ad / Digital Ops, Data Engineer, Data Analyst or Data Scientist, you will find here all the information you need to make the most of mediarithmics. ## Our vision Our vision is to provide a **data-first marketing cloud** to solve the need for organisations to serve as best-in-class in a digital economy. The mediarithmics platform can be seen both as a set of **digital marketing applications** that can be used directly by marketers, or as a highly customizable **data marketing cloud infrastructure** that data engineers and data scientists can tailor to the needs of their organisation. Even if this 'Get started' section is mainly targeted at a technical audience, it is written in plain English, without reference to technical implementation details, and should be readable by anyone interested in the way mediarithmics structures the world of modern marketing. It is a good introduction to the mediarithmics platform. ## From collection to activation The purpose of a data marketing platform is to enable use of all available information to make better decisions in all areas of marketing. The mediarithmics platform can collect all data sources. Whether the data comes from the users' online activity or from backoffice management systems such as CMS', ERPs, and CRMs, all the data is stored for each user in a graph model as shown below. This universal data model allows you to connect all the information of a user, the collected information such as account identifiers, terminal identifiers, profile information, activities, and purchases, as well as data calculated on the fly, such as appetence score, age and gender predictions, and recommendations of articles or other content. These centralized data allow us to conduct multiple analyzes (visit analytics, uplift analytics, segment insights, data discovery ...) and to act when the time comes: * Either by activating audience segments via a rich ecosystem of connectors to advertising networks, communication tools (notifications, emails) or personalization tools (on-site display, A/B testing on user journey), * Either by defining orchestrations that will be triggered automatically, by aligning themselves with the highlights of the user experience (e.g., first visit, creation of an account, drop in the frequency of visits, etc.). ## For all stakeholders In many companies, the recent history of data marketing has been built by the accumulation of different solutions that stack or overlap. Whether it is for the acquisition of new customers, the improvement of the conversion rate, the retention of existing customers, or the construction of predictive models, each application in isolation makes sense and provides a service to a team. But after a while, this structuring in silos no longer makes it possible to be reactive and to innovate. The data is multiplied in multiple systems and these systems cannot speak to each other because of the "heaviness" of these huge amounts of data. The mediarithmics platform has been designed from the start to overcome these structural difficulties and introduce a new model: the data-first marketing cloud. In this model, the data gathered in a single datamart are shared by all applications and users. Both mediarithmics and partner applications access the same source of truth for analytical queries that mobilize large amounts of data, but also for all real-time personalisation micro-queries that are interested in the characteristics of a single user. ## Multi-model database technology All the user data captured or calculated by the mediarithmics platform ends up in a special database named **Datamart**. ![](/files/-MUE8QZsn7MgJMISLzUf) A mediarithmics datamart is more than a traditional database. This new generation database is leveraging a database engine specially designed to outperform in the domain of data-driven marketing applications. This database engine is the outcome of more than seven years of R\&D from mediarithmics engineering teams. It is the first database engine to provide both real-time query processing and scalability on large data volumes (see focus on Datamart). The mediarithmics datamart is a key enabler of all the use cases that can be imagined in one-to-one marketing … (real-time user experience customization, audience segmentation, funnel analytics, and campaign analytics …). ## User 360 view The information inserted in a datamart is primarily stored following a graph structure, where nodes containing pieces of information are connected through a link to represent a relationship between those pieces of information. Let's take an example. A user visits a website which implements a tracker connected to the mediarithmics platform. At the end of the visit, three nodes will be created in the datamart. * One node to represent the user * One node to represent the visit (e.g., piece of information: the web site, the date, the visit duration, pageviews, products in the basket, etc.) * One node to represent the device which was used. Notice that the node representing the user looks like a pinpoint. This node is called a UserPoint and it plays the role of a central pinpoint connecting all pieces of information which are collected about the user. ### Any kind of user identifiers In the User 360 View, it is important to distinguish the nodes representing a user identifier and the nodes representing a describing content on the user. There are three different types of [user identifiers](/user-points#user-identifiers): * The **UserAccount** Id : It represents the identifier of a registered user. The registration system can be a CRM system, a loyalty program system, or an authentication system. This registration system is external to the mediarithmics platform. The UserAccount Id is composed of a character string and a compartment id, which represents the registration system. * The **UserEmail**: It represents an identifier based on the user email. It is usually derived from the user email by using a hash function (MD5, SHA-256, …) * The **UserDeviceTechnicalId**: it represents the identifier of a UserDevice. ### Many different pieces of information attached to a user The describing content of a user 360 view is composed of different data elements corresponding to different point of view on the user: * A **UserActivity** represents an interaction with the user * A **UserProfile** represents a summary of timeless information about the user (e.g. firstname, lastname, birthdate, address, sex, age, etc.) * A **UserSegment** is here to capture that the user belongs to a particular group of users, such as an audience segment. * A **UserChoice** is here to capture that the user has given his consent for a specific type of data processing #### **UserActivity** A UserActivity represents any interaction with the user either online or offline. It can represent the summary of an online visit, the detailed content of an in-store purchase, the summary of a call to a call-center, a campaign interaction like an opened email or a click on a banner. A UserActivity can contain a list of events. An event represents the most granular level of interaction with a user. It is defined by a name and a set of custom properties. For each user, the UserActivity are ordered in a timeline. *For more detailed information on UserActivity and UserEvent, please follow the link below.* {% content-ref url="/pages/-MNSuS1g6Npcklh14qQu" %} [UserActivity & UserEvent](/user-points/user-activities) {% endcontent-ref %} **UserProfile** A UserProfile provides a summary of information on the user and can contain any kind of information. The UserProfile is usually imported from an existing information system like a CRM or login database, and collect information such as: contact details, the user preferences, the status in a loyalty program, the subscription to newsletters. #### **UserSegment** A UserSegment node represents a user who belongs to an audience segment. An Audience Segment is a group of users who share some common characteristics in the eyes of the marketer. There are several ways to define an audience segment in the platform. Whatever the type of Audience Segment, a User 360 view contains a UserSegment node for each audience segment the user belongs to. *For more detailed information on audience segments and their life-cycles, please follow the link below.* {% content-ref url="/pages/-MPoHsh4ttPjX9iyadb3" %} [UserSegment](/user-points/segments) {% endcontent-ref %} #### UserChoice A UserChoice node represents the choice of a user regarding data privacy and data processing declared by the organisation. This information is used to automatically adapt the behavior of the applications to consider the user choices about Data Privacy. #### UserScore A UserScore is a set of data calculated dynamically and providing predictive information on a user. A UserScore is connected to a Machine Learning Function (ML Function). A UserScore can, for example, predict the expected lifetime value, churn risk, age, or gender of a user. ## Plug-able Custom Logic ![](/files/-MUEAFiBWxjgymeJPnIV) All platform services are directly manageable and consumable through APIs. Beyond these standard capabilities, the platform is also a computing infrastructure which enables the hosting of custom algorithms packaged in plugins. A plugin is a bundle of code and data which defines the behavior of a live function hosted in the mediarithmics cloud. It can be developed in any programming language. Due to latency requirements, some languages may be more adaptable than others. The mediarithmics platform can integrate these plugins: * **Activity Analyzer:** to filter and enrich activities data collected through a tag or the API * **Audience Feed**: to push audience segments to third-party systems * **Machine Learning Function**: to add live predictive data to a user graph * **Display Ad Renderer**: to customize the rendering of display ads * **Email Renderer**: to customize the rendering of emails * **Attribution Processor**: to customize the attribution of a marketing conversion to different campaigns ## A large ecosystem of connectors The mediarithmics platform operates at the heart of an open ecosystem by offering connections with many partners. Our platform covers your needs, including (but not limited to): analytics tools, messaging solutions, personalization solutions, prediction tools, advertising networks, and audience monetization. ![](/files/-MUE1e8Gn4uoLLzEhNzu) # Organisations structure Please read [Communities and organisations](https://userguides.mediarithmics.io/basics/communities-and-organisations) page on User guides. # Datamart All user data captured or calculated by the mediarithmics platform ends in a special database named **datamart**. ‌ A mediarithmics datamart is more than a traditional database. This new generation database leverages a database engine specially designed to outperform in the domain of data-driven marketing applications. This database engine is the outcome of more than seven years of R\&D from mediarithmics engineering teams. Datamart is the first database engine to provide both real-time query processing and scalability on large data volumes. ![](/files/-MRKwZOTYHCjpaxXi328) A datamart is a multi-model database. Datamart displays information it manages in an **object** graph, where nodes can either be **persisted objects** or **objects computed on the fly**. Datamart then generates column-oriented tables specialized for analytics scenarios. ## Automatic identity resolution The object graph nature of a datamart is particularly adapted to capture the 'user graph' where each piece of information associated to a user is connected with the others in a local graph structure. The mediarithmics graph structure organizes a user as a node, called a **UserPoint**. The user can be identified or anonymous. Think of the UserPoint as a pin-point that a detective would use to connect various pieces of information on the board, as shown in the following diagram. ![](/files/-MUE3pwGGsiykvbo7sio) Automatic identity resolution is where the software "detective" merges two UserPoint representing the same person. All content and identifiers are regrouped under the older UserPoint. The newer UserPoint is then archived. ## Schema based The structure of a datamart is defined by a customizable schema, defined in a text file and based on the GraphQL Schema Definition Language (SDL). The GraphQL schema defines what types of data are stored in your data graph. Schemas are strongly typed, which unlocks powerful developer tooling. SDL is simple and intuitive to use, while being extremely powerful and expressive. The specification of this standard is available here: A tutorial explaining the syntax of the type system is available here: ![](/files/-MUE6t83dITlUNOra5-o) ## Computed objects User data alone is not sufficient to implement powerful personalized marketing. As in interpersonal communication, experience can help us make educated guesses. Datamart supports homogeneous management collected and automatically computed data to get the necessary insight on each user. Both types of data are declared in the same GraphQL schema. The mediarithmics datamart allows the homogeneous management of the collected data and the computed data. They are both declared in the same GraphQL schema and from the outside nothing distinguishes them. From the inside, the computed data is associated with algorithms that are loaded into the platform in the form of plugins. These algorithms can have different roles: * Machine Learning Function to calculate predictive data * UserTrait Function to calculate aggregates on user data Like all the other plugins, they are freely modifiable and customizable. ## Query languages * **GraphQL**: to allow [querying, in real time, the local graph of a single user](/querying-your-data/graphql-queries) (e.g. for real-time personalization scenarios)
* **OTQL** (Object Tree Query Language): to [query the whole object graph in a couple of seconds](/querying-your-data/otql-queries), even with billions of data points. (e.g. for audience segmentation queries)
* **SQL**: to run powerful [analysis of your behavioral data](/data-ingestion/activities-analytics) ![](/files/-MUE9BQfmvlqdT1wwXR2) # Users and roles Please read [Users and roles](https://userguides.mediarithmics.io/basics/users-and-roles) page on User guides. # UserPoint A UserPoint is **a 360° vision of a unique user on the platform**. ![](/files/-MUE3pwGGsiykvbo7sio) It is composed of different sets of data to cover all the user aspects providing a 360° view (online and offline): * [UserActivity](/user-points/user-activities) represent an interaction with the user * [UserProfile](/user-points/user-profiles) represent a summary of timeless information about the user, like first name, last name, birth date... * [UserSegment](/user-points/segments) are here to capture that the user belongs to a particular group of users * [UserChoice](/advanced-usages/data-privacy-compliance/user-choices) capture that the user has given his consent for a specific type of data processing * UserTrait represent a calculated attribute used to describe the user. ## UserPoint merges Imagine you are going to a site with firefox, and you go back to that site later on safari. At the moment, there is no way for the platform to know that all those actions are from the same user. You'll have two UserPoints: one with the UserDeviceTechnicalId from firefox and the other with the UserDeviceTechnicalId from safari. \ \ If later you log into the site with your UserAccount on both navigators, the two UserPoint will be associated with a common UserAccount ID identifier. The platform will automatically merge the two UserPoint and their activities to represent the reality that we now know those two users were the same. UserPoint merges can only be triggered by : * [UserActivity](/user-points/user-activities) updates * [User identifiers association declarations](/data-ingestion/bulk-processing/user-identifiers-association) The merge will result in a UserPoint survivor, to which we migrate all the data related to the previous two UserPoint. # User identifiers It is important to distinguish the nodes representing a user identifier and the nodes describing content on the user. Every activity or import that'll have one of those properties set will automatically be related to the correct UserPoint. Any activity using an identifier that doesn't already exist in the datamart will trigger the creation of a new UserPoint. {% hint style="info" %} You can bulk associate, dissociate or delete user identifiers using the [bulk import](/data-ingestion/bulk-processing) feature. {% endhint %} There are three different types of user identifiers: * [User accounts](/user-points/user-identifiers/useraccount) * [User emails](/user-points/user-identifiers/useremail) * [User device points and device technical ids](/user-points/user-identifiers/userdevicetechnicalid) # Networks IDs We categorise Network IDs into two groups: **Device-based Network IDs** and **User-based Network IDs**. Each category allows you to add custom identifiers or subscribe to existing ones. To integrate a Device-based or User-based Network ID, the steps are generally: 1. Create a new / Subscribe to a Device Registry or Compartment 2. Activate this Device Registry or Compartment on the desired Datamarts (only required where there is multiple datamarts in your environment) 3. Activate the Device Registry or Compartment ingestion on your Channels Don't hesitate to check the documentation of a particular Network ID below for more information and detailed procedures. # Device-based Network IDs ### Managing device registries in the mediarithmics platform **Device-based Network IDs** refer to **Device Registries** because each device is assigned a "registry" where its identifiers are stored. #### **How to active a Device Registry on a datamart** Here are common steps to activate a Device registry on a specific datamart: * **If you have a single datamart**, the Device Registry is automatically activated, and you can go to the next step. * **If you have multiple datamarts**, you must manually activate the Device Registry: 1. Go to **Navigator > Settings > Organisation > Device Registries.** 2. Select your Device Registry. 3. Click **More** button to open the menu. 4. Choose **Edit linked datamarts.** 5. **Select all datamarts** where you want to activate the Device Registry. # Custom Device ID integration A **UserPoint** can be associated with multiple **UserDevicePoint**. Each user device point can include: * Multiple device technical identifiers * One device information record ### Create a new registry 1. Go to **Navigator > Settings > Organisation > Device Registries.** 2. In the **First-party device registries** section, click on **New Registry.** 3. Specify the following details:
ParametersDescription
NameName of the Device Registry
TokenA technical identifier used to link the Device Registry with the JS Tag
Type

Must be one of the following enums:

MOBILE_VENDOR_ID ,CUSTOM_DEVICE_ID

Description(Optional) More information about the Device Registry
### Subscribe to an existing registry 1. Go to **Navigator > Settings > Organisation > Device Registries.** 2. In the **Shared device registries** section, click on **Manage subscriptions.** 3. Select **Subscribe** to the desired registry. If you have multiple datamarts, don't forget to activate your **Device Registry** on all datamarts. # ID5 For ID5**,** you need to : 1. Subscribe to ID5 2. Activate ID5 data ingestion from websites ### Subscribe to ID5 1. Go to **Navigator > Settings > Organisation > Device Registries**. 2. In the **Shared device registries** section, click on **Manage subscriptions** 3. **Subscribe** to ID5. If you have multiple datamarts, activate the ID5 subscription on all datamarts. ### Activate ID5 ingestion for your websites ID5 requires a specific technical configuration to be properly ingested from your website. Please contact your Account Manager to complete the ID5 configuration for your channel (an activity analyzer will be added to your channel to handle ID5 IDs). # First ID For First ID, you need to : 1. Subscribe to First ID 2. Activate First ID data ingestion from Websites ### **Subscribe to First ID** 1. Go to **Navigator > Settings > Organisation > Device Registries**. 2. In the **Shared device registries** section, click **Manage subscriptions**. 3. Subscribe to **First ID**. If you have multiple datamarts, don't forget to activate First ID on all datamarts. ### **Activate First ID ingestion from websites** First ID requires a modification of the **user-event-tag** to ingest the identifier. Follow these steps: 1. Head to **Navigator > Settings > Datamart > Channels**. 2. Select the site where you want to activate automated First ID capture. 3. Go to **JS Tag Configuration > Device Identification**. 4. Enable **First ID**. 5. Optionally enable **Install First ID javascript tag (SDK) on the webpage**. 6. Repeat these steps for all site where First ID data collection is expected. {% hint style="info" %} The **Install First ID javascript tag on the webpage** option allows the mediarithmics JavaScript tag to automatically install the First ID JavaScript tag (SDK) on the webpage. This option is optional. You may also choose to deploy and manage the First ID SDK independently on your websites. {% endhint %} {% hint style="warning" %} The First ID SDK may read and write information on the device (cookies and local storage) and trigger additional processing of personal data by First ID. Before enabling this option, make sure you have implemented all prerequisites required by First ID regarding user information and consent collection. {% endhint %} # IP address {% hint style="info" %} IP address automatic capture is only available for CTV channels. If you wish to discuss activating it for Site or Mobile channels please get in touch with your account manager. {% endhint %} For IP address, you have 2 options : Option 1 : Activate auto IP address capture for a CTV channel Option 2 : Create dedicated registries then activate auto capture ### Option #1 If you wish to have 2 dedicated registries for each CTV channel 1. Go to **Navigator > Settings >Channels** 2. Select an existing **CTV Channel** or create a new one 3. In the IP addresses capture section, toggle "Active" 4. Save This will automatically create dedicated registries to store IPv4 and IPv6 addresses. All captured IP addresses will then be stored in one of these two registries. If you have multiple CTV channels don't forget to do this for each one. ### Option #2 If you wish to use the same 2 registries for all your CTV channels 1. Create 2 devices registries **Navigator>Settings>Organisation>Device Registries** 1. One of type `IP_V4_ADDRESS_ID` 2. One of type `IP_V6_ADDRESS_ID` 2. Activate automatic IP capture on your CTV channel(s) : **Navigator > Settings >Channels** 3. Select an existing **CTV Channel** or create a new one 4. In the IP addresses capture section, toggle "Active" 5. Check the "Use exisiting device registries" and select the 2 registries you created 6. Save # User-based Network IDs ### Managing compartments in the mediarithmics platform **User-based Network IDs** refer to **Compartments** because each user network ID is assigned a compartment where its identifiers are stored. #### **How to activate a compartment on a datamart** Here are general steps to follow in order to activate a Compartment on a specific datamart: * **If you have a single datamart**, the Compartment is automatically activated. * **If you have multiple datamarts**, you must activate it on every datamart 1. Head to **Navigator > Settings > Datamart > Compartments.** 2. Select your Compartment. 3. Click to the **More** button to open the menu. 4. Choose **Edit linked datamarts.** 5. Select the datamart where you want to activate the Compartment. #### Activate IDs across your organisations: For community-created user IDs, activate them across organizations by selecting Enable Compartment in the Shared User Account Compartments Enabled section. # Custom User ID integration Custom User IDs are used to identify your users with your systems such as CRMs, loyalty programs, or authentication systems. ### Create a Custom User ID 1. Go to **Navigator > Settings > Organisation > Compartments**. 2. In the **First-party user account compartments** section, click on **New Compartment**. 3. Specify the following details: | Parameters | Description | | ---------- | ------------------------------------------------------------------- | | Name | Name of the Compartment | | Token | A technical identifier used to link the Compartment with the JS Tag | If you have multiple datamarts, don't forget to activate the new Compartment on all other datamarts. # UTIQ martechpass ### Understand UTIQ identifiers UTIQ proposes **two types of identifiers**: * **UTIQ martechpass (mobile)**: refers to identifiers coming from **mobile network connections** * **UTIQ martechpass (fixed)**: refers to identifiers coming from **internet box connections** ### Subscribe to UTIQ To subscribe: 1. Go to **Navigator > Settings > Organisation > Compartments**. 2. In the **Shared user account compartments** section, click on **Manage subscriptions**. 3. Subscribe to: * **UTIQ martechpass (mobile)** * **UTIQ martechpass (fixed)** {% hint style="warning" %} If you manage **multiple datamarts**, ensure UTIQ is activated on each one. {% endhint %} {% hint style="warning" %} You must **subscribe to each type individually**, depending on the identifiers you want to capture.\ The configuration steps are **the same** for both mobile and fixed identifiers. {% endhint %}
### Activate UTIQ ingestion from your websites UTIQ IDs can be ingested **natively** via the `user-event-tag` when users browse your websites. To enable automatic ingestion: 1. Go to **Navigator > Settings > Datamart > Channels**. 2. Select the site where you want to activate UTIQ capture. 3. Under **JS Tag Configuration > Device Identification**, enable: * **UTIQ martechpass (mobile)** * **UTIQ martechpass (fixed)** 4. Repeat this process for each website where UTIQ ID capture is required.
### Storage and impact on user point merges The storage method for UTIQ identifiers depends on their type, which directly impacts how UserPoint are merged: * **UTIQ martechpass (mobile)**: stored as **UserAccount** identifiers in compartment 20. The identifier value is provided in the `user_account_id` property. These act as reconciliation keys, triggering a [**User Point merge**](https://userguides.mediarithmics.io/basics/key-concepts/what-is-a-user-point/user-point-merges) when the same identifier is seen across different sessions. 
{
    "type": "USER_ACCOUNT",
    "user_account_id": "mt1-k4LwxGWEn-...........",
    "creation_ts": 1770466916312,
    "compartment_id": 20,
    "expiration_ts": 1778241090397
}
* **UTIQ martechpass (fixed)**: stored as **UserProfile** in compartment 21. The identifier value is provided in the `$utiq_martechpass_fixed` property. These are anonymous profiles (no `user_account_id` and therefore not linked to any userAccount) and **do not trigger any UserPoint merge**. ```json { "$compartment_id": "21", "$last_modified_ts": 1774049259779, "$expiration_ts": 1779898252683, "$creation_ts": 1774049244368, "$utiq_martechpass_fixed": "mt2-k4LwxGWEn-..........." } ``` # UserAccount ## Definition UserAccount identify a user registered on your different systems, like your CRM, a loyalty program or any authentication system. They have the following properties: | Property | Type | Description | | ----------------- | -------------------- | -------------------------------------------------------- | | user\_account\_id | String | Usually the same as the ID as the source, like your CRM. | | compartment\_id | String | Compartment associated with the user account | | creation\_ts | Timestamp | Account's creation timestamp | | expiration\_ts | Timestamp (optional) | Account's eventual expiration timestamp | Always use the `user_account_id` in correlation with a `compartment_id` to identify a user by its account. If you don't specify a `compartment_id`, then the default compartment will be used. A **Compartment** is a group that organizes specific user account identifiers. Each compartment has a unique ID that is used to identify the corresponding user account identifier. For instance, “UTIQ Martechpass” might have the compartment id 20. There are two categories of **Compartments**: * The **First-party user account compartments** which compartments contain user account identifiers created within your organization, such as CRM identifiers. * **Shared user account compartments** which include user account identifiers shared from another organisation. For instance, if you wish to add **UTIQ** to your datamart, first subscribe to it within your community organisation. It will then be automatically shared with your other organisations, but you’ll still need to activate it within your datamart. # UserEmail Users can have multiple UserEmail registered on the platform. They have the following properties: | Property | Type | description | | -------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hash | ID | Hashed user email. Always use the same hashing function (ex: `SHA-256`) in your datamart and all its integrations to allow proper matching between data flows. | | email | String | *Optional*. User's email, not hashed. | | creation\_ts | Timestamp | When the email was registered on the platform | | expiration\_ts | Timestamp (optional) | Email's eventual expiration timestamp | The `hash` property is mandatory and is the property used to identify a user (provided in `$hash` property in User activities request). # UserDeviceTechnicalId In each datamart, all device information is stored within a device graph: * Devices are represented by **UserDevicePoint** * Device identifiers are represented by **UserDeviceTechnicalId** A UserPoint can have multiple user device points. A user device point can have multiple user device technical identifiers, and one device info. 
UserPoint
|
|---------UserDevicePoint--------------------------DeviceInfo: PC - CHROME BROWSER
|                |---------UserDeviceTechnicalId: 1P cookie
|                |---------UserDeviceTechnicalId: Network ID
|                |---------UserDeviceTechnicalId: 3P cookie
|
|---------UserDevicePoint--------------------------DeviceInfo: ANDROID TABLET
|                |---------UserDeviceTechnicalId: Mobile advertising id
|                |---------UserDeviceTechnicalId: Mobile vendor id
## Organizing device technical identifiers within registries Device technical identifiers are related to a registry. There are 6 types of registries:
TypeDescription
INSTALLATION_IDFor each site on which the installation ID feature is activated, a registry of type INSTALLATION_ID is created.
MOBILE_ADVERTISING_IDRegistries representing mobile advertising ids that can be shared across editors are related to this type. It includes for instance Android Advertising IDs (AAID) and Apple Identifier for Advertisers (IDFA).
MOBILE_VENDOR_IDRegistries representing mobile ids that can be only be shared among apps of the same developer account are related to this type. This includes for instance Apple Identifier for Vendors (IDFV).
TV_ADVERTISING_IDIncludes all registries that refer to Smart TVs and TV boxes: AAID on Android TV and Android boxes, IDFA on Apple TV, Amazon Advertising ID on Fire TV, Tizen Advertising ID on Samsung Smart TV etc.
NETWORK_DEVICE_IDFor registries designating IDs that are device-related & managed by third parties actors. For instance ID5, First-ID etc.
⚠️Some network identifiers are directly user-related and not device related (such as the ones generated from the email)
CUSTOM_DEVICE_IDIf you want to use your own device identifier (for instance: a 1P cookie that you generate), you can create it under this type.
MUM_ID

"mediarithmics User Mapping Identifier"

This type and the single registry it contains are dedicated to hosting mediarithmics 3P cookie (vector_id) and references to partners' 3P cookies, until their deprecation

IP_V4_ADDRESS_IDRegistry used to store IPv4 addresses. Can be automatically generated when activating IP addresses auto capture on a channel
IP_V6_ADDRESS_IDRegistry used to store IPv6 addresses. Can be automatically generated when activating IP addresses auto capture on a channel
Registries are related to organisations. You can manage them by going to **Navigator** > **Settings** > **Organisation** > **Device registries**. You can create your own registries under the types `MOBILE_VENDOR_ID` and `CUSTOM_DEVICE_ID`, `IP_V4_ADDRESS_ID`, `IP_V6_ADDRESS_ID` or subscribe to existing registries under other types. For more information of IP addresses refer to [this page](/user-points/user-identifiers/networks-ids/device-based-network-ids/ip-address). Once created, you can activate them on the organisation datamarts. You can set up a expiration duration for identifiers stored in registries you created manually. This will only apply to identifiers captured after setting up the expiration duration. Registries of type `INSTALLATION_ID` are automatically created and removed by the platform. {% hint style="warning" %} Do not forget to create or subscribe to the required registries before using them to identify user data. If user data is received under unknown registries, identifiers will be removed and data may not be ingested properly. {% endhint %} {% hint style="info" %} Two web browsers on the same desktop PC are considered as two distinct agents. For example, your Chrome browser on your Windows laptop is a different device than your Firefox browser on the same laptop. On smartphones, the web browser and the phone itself are considered as two distinct agents. {% endhint %} Some device registry identifiers require **Channel** configuration updates to trigger tracking using them on your websites properly. ## Merge of device points When two device technical ids are associated, the related device points are merged. This can happen: * When capturing user activities with multiple identifiers * When using the [identifier association feature](/data-ingestion/bulk-processing/user-identifiers-association) {% hint style="info" %} If the related device points are linked to different UserPoint, a merge of device points will induce the [merge of UserPoint](/user-points#user-points-merges) as well. {% endhint %} {% hint style="info" %} For some ids that are considered as probabilistic (e.g. that can create false positive matchings of users), the merge of UserDevicePoint/UserPoint will not be allowed to avoid inconsistencies in your ID graph. You contact your Account manager for additional information {% endhint %} ## Description of device graph documents Hereafter is the description of the documents and their properties, as they can be fetched on the APIs. ### User device point | Property | Type | Description | | ---------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id | String | The document identifier. Formatted as `udp:-1234` | | type | Enum UserIdentifierType | In the case of device points, this property takes the value `"USER_DEVICE_POINT"` | | creation\_ts | Timestamp | Timestamp at which the device point was created | | last\_activity\_ts | Timestamp |

Last time an event was ingested from this device.
⚠️This property is not updated at the moment.

| | device | Object of type `DeviceInfo` | Description of the device through a set of normalized properties | | technical\_identifiers | List of objects | List of available identifiers for the device | ### `device` property of user device point | Property | Type | Description | | ---------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | brand | String | The brand of the device (ex.: "Apple") | | model | String | The model of the device (ex.: "Iphone 14") | | agent\_type | Enum UserAgentType | Indicates the type of device. Possible values: `WEB_BROWSER`, `MOBILE_APP` or `CTV_APP` | | browser\_family | Enum BrowserFamily |

If the device is a browser.

Possible values: OTHER, CHROME, IE, FIREFOX, SAFARI, OPERA, STOCK\_ANDROID, BOT, EMAIL\_CLIENT, MICROSOFT\_EDGE.

| | browser\_version | String | The version of the browser. | | form\_factor | Enum FormFactor | Indicates the format of the device. Possible values: `PERSONAL_COMPUTER`, `SMART_TV`, `GAME_CONSOLE`, `SMARTPHONE`, `TABLET`, `WEARABLE_COMPUTER`, `OTHER`. | | os\_family | Enum OperatingSystemFamily | The operating system family of the device. Possible values: `WINDOWS`, `MAC_OS`, `LINUX`, `ANDROID`, `IOS`, `TIZEN`, `WEB_OS`, `ANDROID_TV`, `GOOGLE_TV`, `TV_OS`, `FIRE_TV`, `ROKU_TV`, `TITAN_OS`, `OTHER`. | | os\_version | String | The version of the operating system (ex.: "macOS 10.15 Catalina") | | carrier | String | The service provider that ensures connectivity of the device. | ### User device technical identifier | Property | Type | Description | | ------------------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | user\_agent\_id | String |

The value of the identifier.
See format hereunder

| | registry\_id | String | The registry to which the technical identifier | | type | Enum RegistryType |

The type of the registry to which it is attached.

See section above for possible values

| | creation\_ts | Timestamp | Timestamp at which the document was created | | last\_activity\_ts | Timestamp |

Last time an event was ingested with this identifier.
⚠️This property is not updated at the moment.

| | expiration\_ts | Timestamp |

Timstamp at which this document will expire.
ℹ️For the moment, technical ids are stored for a period of 1 year after their creation

| {% hint style="info" %} The properties available in the documents might differ slightly from the ones of the runtime schema. {% endhint %} ## `user_agent_id` `user_agent_id` property allows to use device identifiers in a single property by concatenating several informations such as registry type, registry id and the id value. ### user\_agent\_id formatting - Cookie-based identifiers #### Installation ID - First-party cookie generated by mediarithmics The Installation ID can appear in different formats depending on where it is used:
LocationValue formatExample
Cookie stored in the user' browser<version_prefix><base64(value)> aMGQ0YTU4Y2EtMTRlNS0xMWVlLWJlNTYtMDI0MmFjMTIwMDAy
Used in authenticated endpointsins:<registry_id>:<value>ins:1001:0d4a58ca-14e5-11ee-be56-0242ac120002
Stored in mediarithmics & used in non authenticated endpoint ins:<registry_token>:<version_prefix><base64(value)>ins:first:aMGQ0YTU4Y2EtMTRlNS0xMWVlLWJlNTYtMDI0MmFjMTIwMDAy
{% hint style="info" %} Note that `version_prefix` is currently defined as `a`. {% endhint %} #### Vector ID - mediarithmics third-party cookie mediarithmics offers to use its third-party cookie, the vector ID. More information can be found on the [cookie documentation](https://developer.mediarithmics.io/user-points/user-identifiers/pages/-MVHeKVgEo5Qf_eNRREb#cookies-installed-on-the-third-party-domain-.mediarithmics.com). mediarithmics third-party cookie can be used as a user identifier with two different formats: * `vec:` * `mum:` Both formats are equivalent: the first one is the exact format stored in the cookie, the second one is the one stored as a technical identifier in the the datamart. Example: `vec:89998434` / `mum:89998434` For a given device point, the vector ID can be retrieved: * In the `technical_identifiers` list as a device technical id of type MUM\_ID ; in which case the identifier format will be `mum:89998434` * In the `mappings` list related to the device point #### Partners' third party cookies The formatting of advertising cookie values from Google and Xandr is a bit specific due to the historical activity of mediarithmics as a DSP provider: * `tech:goo:` for Google advertising cookies * `tech:apx:` for Xandr advertising cookies For other partners, their third-party cookie value is attached to a web domain that is defined by mediarithmics. The identifier format is as follows: * `web::` The web domain designates the `partner-domain.com` and `value` the identifier value inside the cookie on `partner-domain.com`. ### user\_agent\_id formatting - Mobile application identifiers #### Mobile advertising identifiers The generic format for mobile advertising ids is: * `mob:::`. The `os` field designates the OS of device: `and` for Android, `ios` for iOS. The `encoding` field describes how the value is encoded. Available values are: `raw` for no encoding, `sha1` for SHA1, `md5` for MD5. The `value` field contains the mobile advertising id value, encoded according to the previous field. The non-encoded ID should be in lower case for Android and in uppercase for iOS. Example: ``` mob:ios:raw:12345654-ABCD-1234-A1B2-123456789876 mob:and:raw:12345678-abcd-1234-a1b2-123456789876 ``` #### Mobile vendor identifiers The compressed format for mobile vendor ids is: * `mov:::` The `os` field designates the OS of device: `and` for Android, `ios` for iOS. The `registry_id` field designates the registry to which this device id should be linked. `value` refers to the identifier value as generated within the mobile application. ### user\_agent\_id formatting - CTV identifiers #### CTV advertising identifiers The generic format for CTV advertising ids is: * `tv::` #### IP addresses captured on CTV channel * `ipv4::` for registries of type `IPV4_ADDRESS_ID` * `ipv6::` for registries of type `IPV6_ADDRESS_ID` `value` will contain the raw IP address (no hashing or normalization) {% hint style="info" %} While there is no registry dedicated to store CTV vendor identifiers, you can still use either custom registries or Mobile vendor ones (if you want to store the value of the `os` in the identifier) {% endhint %} ### user\_agent\_id formatting - Other device identifiers * `net::` for device technical ids of type `NETWORK_ID` * `dev::` for type `CUSTOM_DEVICE_ID` * `udp:` if you want to use directly the device point identifier instead of a custom device identifier. ## User agents (legacy) {% hint style="warning" %} User agent is a legacy format that is being replaced replaced by user device points and user device technical identifiers. {% endhint %} User agents are the legacy format to store device identifiers. Several user agents can be attached to a UserPoint, and each user agent has a device info object. ``` UserPoint | |---------UserAgent: vec:1111--------------------------DeviceInfo: PC - CHROME BROWSER | |---------UserAgent: vec:2222--------------------------DeviceInfo: ANDROID TABLET ``` All user agents are identified by a vector ID. User agents have the following properties: | Property | Type | Description | | ------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | vector\_id | String | Unique ID generated by mediarithmics and associated with each agent | | device | Object | Device pieces of information such as operating system and browser | | creation\_ts | Timestamp | When the user agent was registered on the platform | | mappings | Array | Additional identifiers called the device mappings. They are cookie-based identifiers or mobile application identifiers associated with the agent. | Two web browsers on the same desktop PC are considered as two distinct agents. For example, your Chrome browser on your Windows laptop is a different device than your Firefox browser on the same laptop. On smartphones, the web browser and the phone itself are considered as two distinct agents. Agent-based operations like visiting a website or seeing an ad will generally automatically leverage the user agent to identify the user. You can use an agent to identify a user, usually with a `user_agent_id` field in the requests. 
// Browser based agent
{
    "vector_id": "vec:12345654321",
    "device": {
        "form_factor": "SMARTPHONE",
        "os_family": "IOS",
        "browser_family": "SAFARI",
        "browser_version": null,
        "brand": null,
        "model": null,
        "os_version": null,
        "carrier": null,
        "raw_value": null,
        "agent_type": "WEB_BROWSER"
    },
    "creation_ts": 1591712194234,
    "mappings": [
        {
            "user_agent_id": "tech:goo:Cazrazrkazeeza-azeree9-azezrze",
            "realm_name": "GOOGLE_OPERATOR"
        },
        {
            "user_agent_id": "tech:apx:12345654321654321",
            "realm_name": "APP_NEXUS_OPERATOR"
        }
    ]
}
The `value` field contains the string value in lower case, after the optional encoding. ```javascript // Mobile application agent { "vector_id": "vec:12345654321", "device": { "form_factor": "OTHER", "os_family": "OTHER", "browser_family": null, "browser_version": null, "brand": null, "model": null, "os_version": null, "carrier": null, "raw_value": null, "agent_type": null }, "creation_ts": 1573405364473, "mappings": [ { // IDFA user agent id with raw encoding: mob:ios:raw:6d92078a-8246-4ba4-ae5b-76104861e7dc // IDFA user agent id with SHA1 encoding: mob:ios:sha1:d520a80c026be39edeb9c6e3f37c01f2da5f5e97 // AAID user agent id with raw encoding: mob:and:raw:97987bca-ae59-4c7d-94ba-ee4f19ab8c21 // AAID user agent id with MD5 encoding: mob:and:md5:ba06c008973b8a1bff6e087c6149227f "user_agent_id": "mob:ios:raw:12345654-8246-1234-ae5b-123456454654", } ] }, ``` # UserActivity & UserEvent User activity management is at the core of audience management services. **A UserActivity is a collection of events (UserEvent) done by a single user in a given period of time**. For instance, mediarithmics considers that [a website activity is a session of no more than 30mn on a given website](/data-ingestion/real-time-user-tracking#session-aggregation). An UserActivity is also linked to its referrer (like Google, Yahoo!, ...), if an user switch of referrer an go back to the same website, it will be considered like a new activity. Events can be given any name and any properties, but some predefined names and properties can be used to trigger specific treatment on the data. **You send UserEvent** to mediarithmics, that are **aggregated and encapsulated into UserActivity**. Some UserActivity only have one UserEvent, but some can have multiple UserEvent. {% hint style="info" %} To visualize user activities for a specific user, go to the navigator > Audience > Monitoring and select a user id. You can then view the complete JSON for each activity. {% endhint %} User activities and events can be indexed and queried using APIs and/or the query engine, if their properties fit the object tree schema. When an activity is ingested via [the real time tracking pipeline](/data-ingestion/real-time-user-tracking), it will trigger some dedicated processes like [event rules](/data-ingestion/real-time-user-tracking/event-rules), [activity analyzers](/data-ingestion/real-time-user-tracking/activity-analyzers), ... ## User Activity object A minimal user activity is an object with the following properties. | field | type | description | | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $ts | Long | The timestamp of the activity. For a session, it should correspond to the start date (Unix Epoch Time in **milliseconds**) | | $type | String enum |

The activity type :
- SITE\_VISITfor activities happening on a channel of type site
- APP\_VISITfor activities happening on a channel of type app

-CTV\_VISIT for activities happening on a channel of type CTV
- DISPLAY\_AD for activities involving ad view and ad click events. See Ads exposure tracking for more information.
- EMAIL when users read a mail or click in a link inside it. For more information, see Email views and clicks.
- TOUCH for other activities

There are also activity types specific to automations:

- USER\_SCENARIO\_START when a scenario starts

- USER\_SCENARIO\_STOP when a scenario stops

- USER\_SCENARIO\_NODE\_ENTER when a new scenario node is entered

- USER\_SCENARIO\_NODE\_EXIT when a scenario node is exited

| | $session\_status | String enum | The sessions status, automatically updated by mediarithmics. `NO_SESSION`, `IN_SESSION`, `CLOSED_SESSION` | | $ttl | Integer | The Time To Live in minutes for the storage of this activity. `0` means no expiration | | $user\_agent\_info | [User Agent info](#user-agent-info) object | Optional. Information of the device | | $user\_agent\_id | String |

The identifier of the user device as provided by the user id mapping service. ex: vec:89090939434
⚠️ Legacy, please use $user\_identifiers instead

| |

$user\_account\_id

(in addition to $compartment\_id)

| String |

The user account id of the user
⚠️ Legacy, please use $user\_identifiers instead

| |

$compartment\_id

(in addition to

$user\_account\_id)

| Integer |

The ID of the compartment associated with this activity
⚠️ Legacy, please use $user\_identifiers instead

| | $email\_hash | Email Hash Object |

The email hash object { “$hash”:…, “$email”:…}
⚠️ Legacy, please use $user\_identifiers instead

| | $user\_identifiers | List of [User Identifier](#user-identifier) |

A list of all identifiers relative to the activity.
You can have many of each type.
⚠️ Use this list rather than the legacy properties ($user\_agent\_id, $user\_account\_id & $compartment\_id, $email\_hash)

| | $origin | [Activity Origin](/user-points/user-activities#activity-origin) | The activity origin | | $location | [Activity Location](/user-points/user-activities#activity-location) | The activity location | | $events | List of [User Event](/user-points/user-activities#user-events-object) | A list of user events attached to this activity | | $unique\_key | String | The unique\_key of the activity formatted as an uuid-v1. If empty, the platform generates one automatically | | \[any custom property] | Any | The value of a custom property | {% hint style="info" %} User activities could be enriched with any custom property, and specific activity types can have additional base properties. **Base properties names all begin with '$'**. That's a good way to differentiate base properties and your custom properties if you don't start your property names with that same character. {% endhint %} {% hint style="warning" %} **A few things to keep in mind :** * **Always prefer predefined properties** to custom properties when they exist, as the platform automates a lot of actions based on those properties. You could miss some critical steps in having well organized data * **Activities are limited** to 8000 characters. * For user identification you can : * **either use** $user\_identifiers; * **or use** $user\_agent\_id, $user\_account\_id, $compartment\_id, and $email\_hash {% endhint %} ### Site visits user activities Their type is SITE\_VISIT. They have those additional base properties : | field | type | description | | ------------------ | ------- | -------------------------------- | | $site\_id | String | The site ID (channel) | | $session\_duration | Integer | The session duration in seconds. | ### App visits user activities Their type is APP\_VISIT. They have those additional base properties : | field | type | description | | ------------------ | ------------------ | -------------------------------- | | $app\_id | String | The mobile app ID (channel) | | $session\_duration | Integer (Optional) | The session duration in seconds. | ### CTV visits user activities Their type is Ct\_VISIT. They have those additional base properties : | field | type | description | | ------------------ | ------------------ | -------------------------------- | | $ctv\_id | String | The CTV ID (channel) | | $session\_duration | Integer (Optional) | The session duration in seconds. | ### Channels If you have multiple sites/apps (Mobile or CTV) you can create a channel for each one of them. Each channel will have an ID, representing either a site ID or an app ID. When you attach site IDs or app IDs to your user activities, you link them to the corresponding channel. This is useful for[ attaching data privacy rules](/advanced-usages/data-privacy-compliance/user-choices) to a site or an app, or when you'll want to create queries and segments, allowing you to select only users having activities on a specific site or app. ### User Identifier A user identifier is either a user account, a user email, or a user agent. #### User Account | field | type | description | | ------------------ | -------------------- | ------------------------------------------------------- | | $type | Constant String | USER\_ACCOUNT | | $compartment\_id | Integer | The ID of the compartment associated with this activity | | $user\_account\_id | String | The user account id of the user | | $expiration\_ts | Timestamp (Optional) | The account's expiration timestamp | #### User Email | field | type | description | | --------------- | -------------------- | -------------------------------- | | $type | Constant String | USER\_EMAIL | | $hash | String | The email hash | | $email | String (Optional) | The "raw" email | | $expiration\_ts | Timestamp (Optional) | The email's expiration timestamp | #### User Agent | field | type | description | | ---------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | $type | Constant String | USER\_AGENT | | $user\_agent*\_*id | String |

The user agent id
Currently support agent type :
- vector id : vec:vector\_id ; e.g.: vec:89090939434

| | $expiration\_ts | Timestamp (Optional) | The agent's expiration timestamp | #### User Agent info | Property | Type | Description | | ----------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $initial\_ts | Timestamp | Timestamp at which device has been seen | | $form\_factor | Enum | Indicates the format of the device. Possible values: `PERSONAL_COMPUTER`, `SMART_TV`, `GAME_CONSOLE`, `SMARTPHONE`, `TABLET`, `WEARABLE_COMPUTER`, `OTHER`. | | $os\_family | Enum | The operating system family of the device. Possible values: `WINDOWS`, `MAC_OS`, `LINUX`, `ANDROID`, `IOS`, `TIZEN`, `WEB_OS`, `ANDROID_TV`, `GOOGLE_TV`, `TV_OS`, `FIRE_TV`, `ROKU_TV`, `TITAN_OS`, `OTHER`. | | $browser\_family | Enum |

If the device is a browser.

Possible values: OTHER, CHROME, IE, FIREFOX, SAFARI, OPERA, STOCK\_ANDROID, BOT, EMAIL\_CLIENT, MICROSOFT\_EDGE.

| | $browser\_version | String | The version of the browser. | | $brand | String | The brand of the device (ex.: "Apple") | | $model | String | The model of the device (ex.: "Iphone 14") | | $os\_version | String | The version of the operating system (ex.: "macOS 10.15 Catalina") | | $carrier | String | The service provider that ensures connectivity of the device (ex: "AT\&T") | | $raw\_value | String | Raw user agent value | | $agent\_type | Enum | Indicates the type of device. Possible values: `WEB_BROWSER` `MOBILE_APP` or `CTV_APP` | ### Activity Location | field | type | description | | ------------ | ---------------------- | ----------------------------------------------------------------------------------------------- | | $source | String (Optional) | The location source (`IP`, `GPS`, `OTHER`) | | $country | String (Optional) | The country’s name | | $region | String (Optional) | The region’s name | | $iso\_region | String (Optional) | The region iso (ISO 3166-2) | | $city | String (Optional) | The city’s name | | $iso\_city | String (Optional) | The city iso (UN/LOCODE) | | $zip\_code | String (Optional) | The zip code | | $latlon | Array\[Double, Double] | The latitude and longitude where the first element is the latitude and the second the longitude | ### Activity Origin In the mediarithmics vocabulary, the activity origin refers to **the last digital channel leading to user interaction**. It is key information to be used to analyze the results and the performance of marketing activities. This interaction can be a Touch (the user views a banner or an email) or a Visit (the user visits a web site or an app). In both cases, the $origin object of the [user activity](/user-points/user-activities#user-activity-object) is used to capture the information related to the originating channel. The $origin object is a customizable object with predefined properties as follows: | origin field | description | | ------------------------------- | ------------------------------------------------------------------ | | $ts | | | $channel | the communication channel. ex: cpc, newsletter, banner, video, ... | | $source | the source of the traffic. ex: google.com, news-foo.com, ... | | $campaign\_name | the campaign name | | $campaign\_technical\_name | the campaign technical name | | $campaign\_id | the campaign id | | $sub\_campaign\_technical\_name | the sub campaign (Ad Group) technical name | | $sub\_campaign\_id | the sub campaign (Ad Group) id | | $message\_id | | | $message\_technical\_name | | | $keywords | the keywords used in the search ex:sport+shoes | | $creative\_name | the creative name | | $creative\_technical\_name | the creative technical name | | $creative\_id | the creative id | | $engagement\_content\_id | | | $social\_network | the social network | | $referral\_path | the URL of the referral | | $log\_id | the custom unique identifier for the activity origin | | $gclid | the unique identifier for a Google AdWords click | #### Origin calculation The origin of user activity is calculated in different ways depending on the activity type (Touch/Visit) and source (Tag/API): * **for a visit on a site or an app**: from the analysis of the referrer and/or the query parameters provided in the destination URL (like the UTM parameters used by Google Analytics) * **for touch events generated by campaigns** delivered by the mediarithmics platform, the origin is automatically calculated from campaign information * **for touch events** (pixel events in emails or banners) the origin is calculated from the properties provided in the event (predefined properties) * **for an activity inserted through the API,** the origin fields can be directly filled with the relevant data. {% hint style="info" %} A user activity can only have one origin. If a user comes back from a different origin in a live session. The current session is closed and a new session is opened with the second origin. {% endhint %} #### Origin detection based on Google Analytics parameters (UTM) Here is the table of correspondence between mediarithmics origin fields and Google Analytics parameters: | origin field | url parameters | example / description | | --------------- | -------------- | ------------------------------------------------------------ | | $channel | utm\_medium |

utm\_medium = email or
$channel = email

| | $source | utm\_source | utm\_source = base loyalty program | | $campaign\_name | utm\_campaign | utm\_campaign = back to school | | $creative\_name | utm\_content | utm\_content = template 1 | | $keywords | utm\_term |

utm\_term = sport+shoes or
$keywords = sport+shoes

| | $gclid | gclid | Google Click Identifier | #### Origin detection based on AT Internet parameter (xtor) Here is the table of correspondence between mediarithmics origin fields and AT Internet parameter:The structure of the xtor parameter is as follows: ``` xtor=A-B-C-D-E-F-G-H ``` The xtor parameter is automatically analyzed to fill the following origin fields: | origin field | xtor field | example / description | | --------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $source | A | EPR when xtor=EPR-14234 | | $campaign\_name | B | 7880 when xtor=AD-7880 | | $creative\_name | C | [ad\_version7](https://developer.mediarithmics.com/guides/user-activities/activity-origin/) when xtor=AD-3030-[ad\_version7](https://developer.mediarithmics.com/guides/user-activities/activity-origin/) | #### Marketing Channel inference According to [AT Internet documentation about xtor parameter](https://help.atinternet-solutions.com/en/configurations/configcampaigns_en.htm#MiniTOCBookMark5), source may be related to marketing channel.Here is the table of correspondance between AT Internet source and mediarithmics channel origin field | source | channel | | ------ | ------------- | | EPR | $email | | EREC | $email | | ES | $email | | AD | $rtb\_display | | AL | $affiliation | #### Origin declaration with predefined event properties When the user event is declared through a tag it is possible to add predefined event properties to declare an activity origin. Here is the list of the predefined properties : | origin field | predefined event properties | example / description | | ------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------- | | $source | $source | $source=crm\_database | | $campaign\_name | $campaign\_name | $campaign\_name=back to school | | $campaign\_technical\_name | $ctn | $ctn=DISPLAY-BACK-TO-SCHOOL | | $campaign\_id | $caid | $caid=8989 | | $sub\_campaign\_technical\_name | $scatn | $scatn=STRATEGY-1, the sub campaign technical name is equivalent to the ad group technical name | | $sub\_campaign\_id | $scaid | $scaid=8782, sub campaign id is equivalent to the ad group id | | $creative\_name | $creative\_name | | | $creative\_technical\_name | $crtn | $crtn=special-banner | | $keywords | $keywords | $keywords = sport+shoes | | $social network | $social\_network | | | $referral path | $referral\_path | | | $log\_id | $log\_id | unique custom identifier | | $gclid | $gclid | Google Click Identifier | ### Activity unique key In mediarithmics, each unique activity is stored with 4 different accesses in this order: Datamart\_id, user\_point\_id, ts and unique\_key. It means that when those **4 parameters** are identical between 2 activities, the new one cancels and replaces the old one. If one of the parameter differs then a new activity is created. #### Unique key calculation This unique key is an uuid format. It can be calculated by the platform or the user itself. The best practice is to used uuid-v1 when generated by the user as it’s not a random value but calculated on the timestamp and a unique String. * **mediarithmics tag through user-event-front**: the unique\_key is generated by the platform. * **mediarithmics apis through datamart-front**: the unique\_key can be calculated by the user. If empty, it’s generated by the platform. * **mediartihmics document import**: the unique\_key can be calculated by the user. If empty, it’s generated by the platform. When generated manually, the user has to select a key property (such as an order\_id, a unique id generated on client size) or properties concatenation in order for the String to be unique. The uuid-v1 also need a timestamp to be generated and it's highly recommended to select the activity's one. See additional documentation ## User Events object A User Event is an object composed of an event name and a list of properties. Each property is composed of a name and a value. ```javascript { "$ts": 3489009384393, "$event_name": "$transaction_confirmed", // Conversion detected "$properties": { "$items": [ { "$id": "product_ID", // Used to filter in funnel analytics "$qty": 20, // Used for conversion amounts "$price": 102.8, // Used or conversion amounts "$brand": "Apple" // Used to filter in funnel analytics "$category1": "Category 1", // Used to filter in funnel analytics "$category2": "Category 2", // Used to filter in funnel analytics "$category3": "Category 3", // Used to filter in funnel analytics "$category4": "Category 4" // Used to filter in funnel analytics }, { "$id": "product_ID2", "$qty": 12, "$price": 3.4, "$brand": "Microsoft" } ], "$currency": "EUR" } } ``` ### **Predefined event names** Using some predefined event names will allow you to have useful adapted automatic processing on some of your events. | event name | description | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $page\_view |

The user has viewed a page.

Events with this event name only serve in session aggregation to have correct information on the user activity. They will be dropped when the session closes, and you won't see them on the platform anymore. If you wish to keep a record of the pages a user viewed in your site and create queries based on that data, you should name your event differently. You can also use the $item\_view event name if your pages show products.

| | $home\_view | The user has viewed the home page | | $item\_view | The user has viewed an item | | $item\_list\_view | The user has viewed a list of items | | $basket\_view | The user has viewed the basket | | $transaction\_confirmed | The user has completed a transaction | | $conversion | The user has completed a conversion. This event is either registered by the integrator or automatically craeted by the platform when a goal is met. | | $app\_install | The user has installed an app | | $app\_update | The user has updated an app | | $app\_open | The user has opened an app or resumed it | | $ad\_view | The user has been exposed to a display add | | $ad\_click | The user has clicked on an app | | $email\_view | The user has opened an email | | $email\_click | The user has opened a link in an email | | $set\_user\_choice | The user has given consent or objected to a [processing activity](/advanced-usages/data-privacy-compliance/user-choices#processing-activities) | {% hint style="warning" %} **Always prefer predefined event names** to custom events when they exist, as the platform automates a lot of actions based on those names. You could miss some critical steps in having well organized data. {% endhint %} ### Predefined event properties Using some predefined event properties will allow you to have useful adapted automatic processing on some of your events. | property name | description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | $items |

An array of products associated with the event. Mainly used in products tracking for e-commerce sites.
Example value : \[{"$id":"ProductID1"},{"$id":"ProductID2"}]

Each item has the $id, $ean, $qty , $price, $brand, $name, $category1, $category2, $category3 and $category4predefined properties.

| | $campaign\_technical\_name | Technical name of a campaign associated to the event when [tracking ads exposure](/data-ingestion/real-time-user-tracking/ads-exposure). | | $sub\_campaign\_technical\_name | Technical name of a sub campaign associated to the event when [tracking ads exposure](/data-ingestion/real-time-user-tracking/ads-exposure). | | $creative\_technical\_name | Technical name of a creative associated to the event | | $processing\_token | Token of the associated [processing activity](/advanced-usages/data-privacy-compliance/user-choices#processing-activities) when event name is `$set_user_choice`. | | $choice\_acceptance\_value | Acceptance value of the [user choice](/advanced-usages/data-privacy-compliance/user-choices#user-choices) when event name is `$set_user_choice`. | {% hint style="warning" %} **Always prefer predefined event properties** to custom event properties when they exist, as the platform automates a lot of actions based on those names. You could miss some critical steps in having well organized data. {% endhint %} ### User Activity JSON schema and TypeScript interfaces Below is the JSON schema for a single activity, according to the rules enacted in this documentation. {% code overflow="wrap" expandable="true" %} ```json { "$schema": "http://json-schema.org/draft-07/schema#", "$ref": "#/definitions/UserActivity", "definitions": { "UserActivity": { "oneOf": [ { "$ref": "#/definitions/GenericUserActivity" }, { "$ref": "#/definitions/SiteVisitUserActivity" }, { "$ref": "#/definitions/AppVisitUserActivity" }, { "$ref": "#/definitions/CTVVisitUserActivity" } ] }, "GenericUserActivity": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$session_status": { "$ref": "#/definitions/UserActivitySessionStatus" }, "$ttl": { "type": "number" }, "$user_agent_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$user_account_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$compartment_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$email_hash": { "$ref": "#/definitions/Nullable%3Calias-1071211137-70767-70920-1071211137-0-212510%3Cdef-interface-792792747-896-1013-792792747-0-7494%2C%22%24type%22%3E%3E" }, "$user_identifiers": { "$ref": "#/definitions/Nullable%3Cdef-alias-792792747-1226-1323-792792747-0-7494%5B%5D%3E" }, "$origin": { "$ref": "#/definitions/Nullable%3CUserActivityOrigin%3E" }, "$location": { "$ref": "#/definitions/Nullable%3CUserActivityLocation%3E" }, "$unique_key": { "$ref": "#/definitions/UUID" }, "$type": { "type": "string", "enum": [ "DISPLAY_AD", "EMAIL", "TOUCH", "USER_SCENARIO_START", "USER_SCENARIO_STOP", "USER_SCENARIO_NODE_ENTER", "USER_SCENARIO_NODE_EXIT" ] }, "$events": { "type": "array", "items": { "$ref": "#/definitions/UserActivityEvent" } } }, "required": [ "$email_hash", "$events", "$location", "$origin", "$session_status", "$ts", "$ttl", "$type" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "Timestamp": { "type": "number" }, "UserActivitySessionStatus": { "type": "string", "enum": [ "NO_SESSION", "IN_SESSION", "CLOSED_SESSION" ] }, "Nullable": { "anyOf": [ { "$ref": "#/definitions/ID" }, { "type": "null" } ] }, "ID": { "type": "string" }, "Nullable>": { "anyOf": [ { "type": "object", "properties": { "$hash": { "type": "string" }, "$email": { "$ref": "#/definitions/Nullable%3Cstring%3E" } }, "required": [ "$hash" ], "additionalProperties": false }, { "type": "null" } ] }, "Nullable": { "type": [ "string", "null" ] }, "Nullable": { "anyOf": [ { "type": "array", "items": { "$ref": "#/definitions/UserIdentifier" } }, { "type": "null" } ] }, "UserIdentifier": { "anyOf": [ { "$ref": "#/definitions/UserEmailIdentifier" }, { "$ref": "#/definitions/UserAccountIdentifier" }, { "$ref": "#/definitions/UserAgentIdentifier" } ] }, "UserEmailIdentifier": { "type": "object", "properties": { "$type": { "type": "string", "const": "USER_EMAIL" }, "$hash": { "type": "string" }, "$email": { "$ref": "#/definitions/Nullable%3Cstring%3E" } }, "required": [ "$type", "$hash" ], "additionalProperties": false }, "UserAccountIdentifier": { "type": "object", "properties": { "$type": { "type": "string", "const": "USER_ACCOUNT" }, "$user_account_id": { "$ref": "#/definitions/ID" }, "$compartment_id": { "$ref": "#/definitions/ID" } }, "required": [ "$type", "$user_account_id", "$compartment_id" ], "additionalProperties": false }, "UserAgentIdentifier": { "type": "object", "properties": { "$type": { "type": "string", "const": "USER_AGENT" }, "$user_agent_id": { "$ref": "#/definitions/ID" } }, "required": [ "$type", "$user_agent_id" ], "additionalProperties": false }, "Nullable": { "anyOf": [ { "$ref": "#/definitions/UserActivityOrigin" }, { "type": "null" } ] }, "UserActivityOrigin": { "type": "object", "properties": { "$campaign_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$campaign_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$channel": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$creative_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$creative_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$engagement_content_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$gclid": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$keywords": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$log_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$message_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$message_technical_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$referral_path": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$social_network": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$source": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$sub_campaign_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$sub_campaign_technical_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$ts": { "type": "number" } }, "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "Nullable": { "type": [ "number", "null" ] }, "JsonType": { "anyOf": [ { "$ref": "#/definitions/Nullable%3Cstring%3E" }, { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, { "type": "boolean" }, { "type": "object" }, { "type": "array", "items": {} }, { "not": {} } ] }, "Nullable": { "anyOf": [ { "$ref": "#/definitions/UserActivityLocation" }, { "type": "null" } ] }, "UserActivityLocation": { "type": "object", "properties": { "$source": { "$ref": "#/definitions/LocationSource" }, "$country": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$region": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$iso_region": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$city": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$iso_city": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$zip_code": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$latlon": { "$ref": "#/definitions/Nullable%3Cnumber%5B%5D%3E" } }, "required": [ "$latlon" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "LocationSource": { "type": "string", "enum": [ "GPS", "IP", "OTHER" ] }, "Nullable": { "anyOf": [ { "type": "array", "items": { "type": "number" } }, { "type": "null" } ] }, "UUID": { "type": "string", "pattern": "^[0-9a-fA-F]{8}\\b-[0-9a-fA-F]{4}\\b-[0-9a-fA-F]{4}\\b-[0-9a-fA-F]{4}\\b-[0-9a-fA-F]{12}$" }, "UserActivityEvent": { "anyOf": [ { "$ref": "#/definitions/GenericUserActivityEvent" }, { "$ref": "#/definitions/AdTrackingEvent" }, { "$ref": "#/definitions/SetUserChoiceEvent" }, { "$ref": "#/definitions/SetUserProfilePropertiesEvent" }, { "$ref": "#/definitions/RetailEvent" }, { "$ref": "#/definitions/ConversionEvent" } ] }, "GenericUserActivityEvent": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$expiration_ts": { "$ref": "#/definitions/Nullable%3CTimestamp%3E" }, "$event_name": { "anyOf": [ { "$ref": "#/definitions/EventName" }, { "type": "string" } ] }, "$properties": { "$ref": "#/definitions/Customizable" } }, "required": [ "$event_name", "$properties", "$ts" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "Nullable": { "anyOf": [ { "$ref": "#/definitions/Timestamp" }, { "type": "null" } ] }, "EventName": { "anyOf": [ { "$ref": "#/definitions/DefaultEventName" }, { "type": "string" } ] }, "DefaultEventName": { "type": "string", "enum": [ "$page_view", "$home_view", "$category_view", "$email_view", "$email_click", "$email_sent", "$email_delivered", "$email_soft_bounce", "$email_hard_bounce", "$email_unsubscribe", "$email_complaint", "$content_corrections" ] }, "Customizable": { "type": "object", "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "AdTrackingEvent": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$expiration_ts": { "$ref": "#/definitions/Nullable%3CTimestamp%3E" }, "$event_name": { "type": "string", "enum": [ "$ad_view", "$ad_click" ] }, "$properties": { "$ref": "#/definitions/AdTrackingEventProperties" } }, "required": [ "$event_name", "$properties", "$ts" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "AdTrackingEventProperties": { "type": "object", "properties": { "$url": { "type": "string" }, "$referrer": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$campaign_technical_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$sub_campaign_technical_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$creative_technical_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$message_technical_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$campaign_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$sub_campaign_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$message_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$creative_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" } }, "additionalProperties": { "$ref": "#/definitions/JsonType" }, "required": [ "$url" ] }, "SetUserChoiceEvent": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$expiration_ts": { "$ref": "#/definitions/Nullable%3CTimestamp%3E" }, "$event_name": { "type": "string", "const": "$set_user_choice" }, "$properties": { "$ref": "#/definitions/SetUserChoiceEventProperties" } }, "required": [ "$event_name", "$properties", "$ts" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "SetUserChoiceEventProperties": { "type": "object", "properties": { "$url": { "type": "string" }, "$referrer": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$processing_token": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$processing_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$choice_acceptance_value": { "type": "boolean" }, "$choice_source_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" } }, "required": [ "$choice_acceptance_value", "$url" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "SetUserProfilePropertiesEvent": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$expiration_ts": { "$ref": "#/definitions/Nullable%3CTimestamp%3E" }, "$event_name": { "type": "string", "const": "$set_user_profile_properties" }, "$properties": { "$ref": "#/definitions/Customizable" } }, "required": [ "$event_name", "$properties", "$ts" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "RetailEvent": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$expiration_ts": { "$ref": "#/definitions/Nullable%3CTimestamp%3E" }, "$event_name": { "$ref": "#/definitions/RetailEventName" }, "$properties": { "$ref": "#/definitions/RetailEventProperties" } }, "required": [ "$event_name", "$properties", "$ts" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "RetailEventName": { "type": "string", "enum": [ "$item_view", "$item_list_view", "$product_view", "$product_list_view", "$basket_view", "$transaction_confirmed" ] }, "RetailEventProperties": { "type": "object", "properties": { "$url": { "type": "string" }, "$referrer": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$items": { "$ref": "#/definitions/Nullable%3Cdef-interface-792792747-3521-3897-792792747-0-7494%5B%5D%3E" } }, "additionalProperties": { "$ref": "#/definitions/JsonType" }, "required": [ "$url" ] }, "Nullable": { "anyOf": [ { "type": "array", "items": { "$ref": "#/definitions/RetailEventPropertiesItem" } }, { "type": "null" } ] }, "RetailEventPropertiesItem": { "type": "object", "properties": { "$id": { "type": "string" }, "$ean": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$qty": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$price": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$brand": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$name": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$category1": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$category2": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$category3": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$category4": { "$ref": "#/definitions/Nullable%3Cstring%3E" } }, "required": [ "$id" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "ConversionEvent": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$expiration_ts": { "$ref": "#/definitions/Nullable%3CTimestamp%3E" }, "$event_name": { "type": "string", "const": "$conversion" }, "$properties": { "$ref": "#/definitions/ConversionEventProperties" } }, "required": [ "$event_name", "$properties", "$ts" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "ConversionEventProperties": { "type": "object", "properties": { "$conversion_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$goal_id": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$conversion_technical_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$goal_technical_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$conversion_value": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$log_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$conversion_external_id": { "$ref": "#/definitions/Nullable%3Cstring%3E" }, "$goal_technical_name": { "$ref": "#/definitions/Nullable%3Cstring%3E" } }, "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "SiteVisitUserActivity": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$session_status": { "$ref": "#/definitions/UserActivitySessionStatus" }, "$ttl": { "type": "number" }, "$user_agent_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$user_account_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$compartment_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$email_hash": { "$ref": "#/definitions/Nullable%3Calias-1071211137-70767-70920-1071211137-0-212510%3Cdef-interface-792792747-896-1013-792792747-0-7494%2C%22%24type%22%3E%3E" }, "$user_identifiers": { "$ref": "#/definitions/Nullable%3Cdef-alias-792792747-1226-1323-792792747-0-7494%5B%5D%3E" }, "$origin": { "$ref": "#/definitions/Nullable%3CUserActivityOrigin%3E" }, "$location": { "$ref": "#/definitions/Nullable%3CUserActivityLocation%3E" }, "$unique_key": { "$ref": "#/definitions/UUID" }, "$session_duration": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$error_analyzer_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$analyzer_errors": { "type": "array", "items": { "type": "object" } }, "$topics": { "$ref": "#/definitions/Nullable%3Calias-1071211137-70404-70537-1071211137-0-212510%3Cstring%2Calias-1071211137-70404-70537-1071211137-0-212510%3Cstring%2Cnumber%3E%3E%3E" }, "$type": { "type": "string", "const": "SITE_VISIT" }, "$events": { "type": "array", "items": { "$ref": "#/definitions/UserActivityEvent" } }, "$site_id": { "$ref": "#/definitions/ID" } }, "required": [ "$email_hash", "$events", "$location", "$origin", "$session_status", "$site_id", "$ts", "$ttl", "$type" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "Nullable>>": { "anyOf": [ { "type": "object", "additionalProperties": { "type": "object", "additionalProperties": { "type": "number" } } }, { "type": "null" } ] }, "AppVisitUserActivity": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$session_status": { "$ref": "#/definitions/UserActivitySessionStatus" }, "$ttl": { "type": "number" }, "$user_agent_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$user_account_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$compartment_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$email_hash": { "$ref": "#/definitions/Nullable%3Calias-1071211137-70767-70920-1071211137-0-212510%3Cdef-interface-792792747-896-1013-792792747-0-7494%2C%22%24type%22%3E%3E" }, "$user_identifiers": { "$ref": "#/definitions/Nullable%3Cdef-alias-792792747-1226-1323-792792747-0-7494%5B%5D%3E" }, "$origin": { "$ref": "#/definitions/Nullable%3CUserActivityOrigin%3E" }, "$location": { "$ref": "#/definitions/Nullable%3CUserActivityLocation%3E" }, "$unique_key": { "$ref": "#/definitions/UUID" }, "$session_duration": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$error_analyzer_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$analyzer_errors": { "type": "array", "items": { "type": "object" } }, "$topics": { "$ref": "#/definitions/Nullable%3Calias-1071211137-70404-70537-1071211137-0-212510%3Cstring%2Calias-1071211137-70404-70537-1071211137-0-212510%3Cstring%2Cnumber%3E%3E%3E" }, "$type": { "type": "string", "const": "APP_VISIT" }, "$events": { "type": "array", "items": { "anyOf": [ { "$ref": "#/definitions/UserActivityEvent" }, { "$ref": "#/definitions/AppActivityEvent" } ] } }, "$app_id": { "$ref": "#/definitions/ID" } }, "required": [ "$app_id", "$email_hash", "$events", "$location", "$origin", "$session_status", "$ts", "$ttl", "$type" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "CTVVisitUserActivity": { "type": "object", "properties": { "$ts": { "$ref": "#/definitions/Timestamp" }, "$session_status": { "$ref": "#/definitions/UserActivitySessionStatus" }, "$ttl": { "type": "number" }, "$user_agent_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$user_account_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$compartment_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$email_hash": { "$ref": "#/definitions/Nullable%3Calias-1071211137-70767-70920-1071211137-0-212510%3Cdef-interface-792792747-896-1013-792792747-0-7494%2C%22%24type%22%3E%3E" }, "$user_identifiers": { "$ref": "#/definitions/Nullable%3Cdef-alias-792792747-1226-1323-792792747-0-7494%5B%5D%3E" }, "$origin": { "$ref": "#/definitions/Nullable%3CUserActivityOrigin%3E" }, "$location": { "$ref": "#/definitions/Nullable%3CUserActivityLocation%3E" }, "$unique_key": { "$ref": "#/definitions/UUID" }, "$session_duration": { "$ref": "#/definitions/Nullable%3Cnumber%3E" }, "$error_analyzer_id": { "$ref": "#/definitions/Nullable%3CID%3E" }, "$analyzer_errors": { "type": "array", "items": { "type": "object" } }, "$topics": { "$ref": "#/definitions/Nullable%3Calias-1071211137-70404-70537-1071211137-0-212510%3Cstring%2Calias-1071211137-70404-70537-1071211137-0-212510%3Cstring%2Cnumber%3E%3E%3E" }, "$type": { "type": "string", "const": "CTV_VISIT" }, "$events": { "type": "array", "items": { "anyOf": [ { "$ref": "#/definitions/UserActivityEvent" }, { "$ref": "#/definitions/AppActivityEvent" } ] } }, "$ctv_id": { "$ref": "#/definitions/ID" } }, "required": [ "$ctv_id", "$email_hash", "$events", "$location", "$origin", "$session_status", "$ts", "$ttl", "$type" ], "additionalProperties": { "$ref": "#/definitions/JsonType" } }, "AppActivityEvent": { "type": "object", "properties": { "$event_name": { "type": "string", "enum": [ "$app_open", "$app_update", "$app_install" ] }, "$properties": { "$ref": "#/definitions/Customizable" } }, "required": [ "$event_name", "$properties" ], "additionalProperties": false } } } ``` {% endcode %} # Compartments A UserPoint could have multiple user accounts and user profiles on the same datamart. For example, having a profile on one of your sites and another profile on another site. Compartments are a notion created to represent those different places where you have accounts and profiles. There is always one default compartment on a datamart, and you can create more of them. Compartments are associated with user account IDs to create a user identifier. # UserProfile A UserProfile is usually [imported from an existing information system,](/data-ingestion/bulk-processing/imports/profiles) for example CRM, and login database. We usually see our customers using the profile to collect: * Contact details * User preferences * Status in a loyalty program * Subscription to a newsletter * CRM information * Scoring calculated outside the platform * Aggregated Values * *other details* Each UserProfile is associated with a UserPoint by a [user identifier](/user-points#user-identifiers). It can also be linked to a UserAccount. ## User profile object | field | type | description | | ---------------------- | ------------------ | ------------------------------------------------------------------------------------------------ | | $user\_account\_id | String (Optional) | The associated UserAccount ID. If none, the profile remains anonymous. | | $compartment\_id | Integer (Optional) | The compartment ID. If none, the profile is imported into the default compartment\_id. | | $last\_modified\_ts | Timestamp | The timestamp of the last edit operation for this user profile. Automatically set by the system. | | $creation\_ts | Timestamp | The timestamp of the creation operation for this user profile. Automatically set by the system. | | \[any custom property] | Any | The value of a custom property | {% hint style="warning" %} When importing profiles it is recommended to complete the $user\_account\_id and the $compartment\_id event if the values are the same than the ones used as the identifier as the values are not inherited from the identifiers info. You can find more information about the [user profiles import](/data-ingestion/bulk-processing/imports/profiles). {% endhint %} ### Example: ```javascript # UserProfile object { "$compartment_id": 1606, "$user_account_id": "c9879698769OIUYOIY9879879", "$last_modified_ts": 5467987654613, "$creation_ts": 6579874654654654, "firstname": "David", "lastname": "Guetta", "gender": 1, "newsletter_options": { "subscribed": true, "preferred_periods": "MONTHLY" } } ``` ```javascript # Complete payload when importing a profile { "operation": "UPSERT", "compartment_id": "1600", "user_account_id": "identifier_account_id", "force_replace": true, "user_profile": { "$compartment_id": 1606, "$user_account_id": "c9879698769OIUYOIY9879879", "$last_modified_ts": 5467987654613, "$creation_ts": 6579874654654654, "firstname": "David", "lastname": "Guetta", "gender": 1, "newsletter_options": { "subscribed": true, "preferred_periods": "MONTHLY" } } } ``` {% hint style="success" %} You can have different compartment IDs and different UserAccount between the one in the profile wrapper, as the UserPoint identifier and the one into the UserProfile object. {% endhint %} # UserSegment Audience segments represents a group of users and are central to most marketing actions. Users can be grouped by common profile characteristics, or similar behavior in their online browsing or in their purchases. Users can be grouped either using the mediarithmics platform, or externally and then imported. Here is a list of audience segment types that differ from each other by the method used to group users: * Audience segments calculated from a query (type USER QUERY).\ Several tools are provided to define this type of segment. Occasional platform users will find it easier to use the Audience Builder. More advanced users such as data admin or data analyst will find a greater wealth of expression in the Segment Builder. Finally, technical users, data engineers or integrators, can directly use a textual query language (see OTQL). * Audience segments imported from an external system (type USER LIST) * Audience segments calculated from a likeness prediction algorithm (type USER LOOKALIKE) * Audience segments associated with the events of a campaign (eg: all the people exposed to a given video campaign) (type USER ACTIVATION) * Audience segments associated with A / B tests (control group and test group) ## Audience Segment & UserSegments An audience segment represents a group of users. A UserSegment is a piece of information that is associated with a user to indicate that this user belongs to the segment. For example, if an audience segment has 10,000 users, each user has a UserSegment to signify their membership in this segment. So there are 10,000 UserSegment, one for each user in the segment. In the data schema this information element is materialized in the form of an object of type UserSegment (see standard datamart schema). This object contains the following information: * The segment identifier * The date the user entered the segment (creation date of the UserSegment object) * The expiration date at the end of which the user will exit the segment. This date is optional and is only entered for some types of segments. ## Counting and Persistence of an audience segment of type User Query When working with a User Query audience segment, it is important to understand the difference between the count of the segment and the persistence of this segment. Counting consists of calculating the number of users who verify the segment's grouping rules. That is to say the users who respond positively to the conditions and to the boolean operators present in the segment query. The counting of a segment is done in real time. It usually only takes a few seconds (2-5 sec) to get a result. However, if the count is immediate, the process of bringing all the targeted users into the segment by creating a UserSegment type record for each of them may take more time. This time depends on the size of the segment, the segment refresh policy and the calculation budget allocated to the account. # Hyper point & Quarantine ### Hyper point A **hyper point** is a preventive way to flag a UserPoint whenever there are too many identifiers linked to it. Concretely, a UserPoint becomes a hyper point when mediarithmics tries to [merge UserPoint](https://userguides.mediarithmics.io/basics/key-concepts/what-is-a-user-point/user-point-merges) and especially when at least one of the following conditions is met: * the frequency of [user identifiers](/user-points/user-identifiers) creation in this UserPoint reaches: * more than 10 **identifiers by** [**Registry ID**](https://developer.mediarithmics.io/user-points/user-identifiers/device-identifiers#user-device-technical-identifier) in 2 days, * more than 4 **Account ID** in 7 days, * more than 4 **Email hash** in 7 days, * more than 8 UserPoint were merged to that UserPoint in the last 14 days, * more than 100 user identifiers are attached to this UserPoint. This system is a preventive action of flagging a UserPoint so that the UserPoint is still actionable in mediarithimics. ### Quarantine The quarantine job is a monthly process that flags any UserPoint which has too many objects attached to it. This happens especially when at least one of the following conditions is met: * more than 10.000 [activities](/user-points/user-activities), * more than 5.000 [segments](/user-points/segments). A UserPoint in quarantine can still be looked up in Navigator but no more additions will happen to it.A UserPoint can be freed by the quarantine job if it gets below the above-mentionned limits. This particularly happens by deleting activities (through API or when cleaning rules are applied) on a quarantined UserPoint. # Data model A schema is applied to a datamart and **defines what mediarithmics should index and make available through queries**. ‌Each schema is a [Graph Query Language](https://graphql.org/) schema defining an object tree index that will allow you to run fast [Object Tree Query Language queries](https://app.writer.com/@mediarithmics/s/developer-guide/~/drafts/-MVkpG9_-bGog-3zbolA/guides/querying-your-data/otql-queries) to search users. ## Sample Schema ![](/files/-MUE5P0y1TB26WBfvuGB) This schema defines all available mediarithmics objects with the standard properties. When [defining your schema](/schema/defining-your-schema), you can start from this schema and add/remove properties based on your needs and the data you ingest into the platform. {% hint style="info" %} The number of referenced properties has an impact on query performance. It would be best only to have the properties you need to use when [defining your schema](/schema/defining-your-schema). Don't just copy the default ones. {% endhint %} {% hint style="success" %} UserPoint is the root element of any mediarithmics schema, and only one index can be created. This may change in future releases to allow you to build different indexes. {% endhint %} ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX"){ id: ID! creation_ts: Timestamp! @TreeIndex(index:"USER_INDEX") creation_date:Date! @Function(name:"ISODate", params:["creation_ts"]) # User identifiers accounts: [UserAccount!]! emails: [UserEmail!]! devices: [UserDevicePoint!]! # User content activities: [UserActivity!]! events: [UserEvent!]! profiles: [UserProfile!]! choices: [UserChoice!]! # Technical objects scenarios: [UserScenario!]! segments: [UserSegment!]! # Deprecated identifiers # agents: [UserAgent!]! } ### User identifiers type UserAccount { id:ID! @TreeIndex(index:"USER_INDEX") creation_ts: Timestamp! @TreeIndex(index:"USER_INDEX") creation_date:Date! @TreeIndex(index:"USER_INDEX") @Function(params:["creation_ts"], name:"ISODate") compartment_id: String! @TreeIndex(index:"USER_INDEX") @ReferenceTable(model_type:"COMPARTMENTS", type:"CORE_OBJECT") user_account_id: String! @TreeIndex(index:"USER_INDEX") } type UserEmail { id:ID! @TreeIndex(index:"USER_INDEX") creation_ts: Timestamp! @TreeIndex(index:"USER_INDEX") last_activity_ts: Timestamp @TreeIndex(index:"USER_INDEX") email: String @TreeIndex(index:"USER_INDEX") } type UserDevicePoint { id:ID! @TreeIndex(index:"USER_INDEX") creation_ts:Timestamp! @TreeIndex(index:"USER_INDEX") creation_date:Date! @Function(name:"ISODate", params:["creation_ts"]) device_info:DeviceInfo technical_identifiers:[UserDeviceTechnicalId!]! mappings:[UserAgentMapping!]! @Function(name:"ThirdPartyCookieMappings", params:["id"]) } type DeviceInfo { brand:String @TreeIndex(index:"USER_INDEX") browser_version:String @TreeIndex(index:"USER_INDEX") carrier:String @TreeIndex(index:"USER_INDEX") model:String @TreeIndex(index:"USER_INDEX") os_version:String @TreeIndex(index:"USER_INDEX") agent_type:UserAgentType @TreeIndex(index:"USER_INDEX") browser_family:BrowserFamily @TreeIndex(index:"USER_INDEX") form_factor:FormFactor @TreeIndex(index:"USER_INDEX") os_family:OperatingSystemFamily @TreeIndex(index:"USER_INDEX") } type UserDeviceTechnicalId { id:ID! @TreeIndex(index:"USER_INDEX") creation_ts:Timestamp! @TreeIndex(index:"USER_INDEX") expiration_ts:Timestamp! @TreeIndex(index:"USER_INDEX") last_seen_ts:Timestamp! @TreeIndex(index:"USER_INDEX") registry_id:String! @TreeIndex(index:"USER_INDEX") type:String! @TreeIndex(index:"USER_INDEX") } type UserAgentMapping { last_seen:Timestamp user_agent_id:String @TreeIndex(index:"USER_INDEX") vector_id:String } ### User content type UserActivity { id: ID! type: UserActivityType! channel_id:String @TreeIndex(index:"USER_INDEX") @ReferenceTable(type:"CORE_OBJECT", model_type:"CHANNELS") @Property(paths:["$site_id", "$app_id"]) source: UserActivitySource! ts: Timestamp! @TreeIndex(index:"USER_INDEX") duration: Int @TreeIndex(index:"USER_INDEX") events: [UserEvent!]! } type UserEvent @Mirror(object_type:"UserEvent") { id: ID! ts: Timestamp! @TreeIndex(index:"USER_INDEX") date:Date! @Function(params:["ts"], name:"ISODate") name:String! @TreeIndex(index:"USER_INDEX") channel_id:String @TreeIndex(index:"USER_INDEX") @ReferenceTable(model_type:"CHANNELS", type:"CORE_OBJECT") @Property(paths:["[parent].$site_id", "[parent].$app_id"]) url: String @TreeIndex(index:"USER_INDEX") referrer:String @TreeIndex(index:"USER_INDEX") } type UserProfile { id: ID! creation_ts: Timestamp! @TreeIndex(index:"USER_INDEX") last_modified_ts: Timestamp! @TreeIndex(index:"USER_INDEX") compartment_id: String! @TreeIndex(index:"USER_INDEX") @ReferenceTable(model_type:"COMPARTMENTS", type:"CORE_OBJECT") user_account_id: String @TreeIndex(index:"USER_INDEX") } type UserChoice { id: ID! creation_ts: Timestamp! @TreeIndex(index:"USER_INDEX") choice_ts: Timestamp! @TreeIndex(index:"USER_INDEX") processing_id: String! @TreeIndex(index:"USER_INDEX") choice_acceptance_value: Boolean! @TreeIndex(index:"USER_INDEX") user_account_id: String compartment_id: String email_hash: String user_agent_id: String channel_id: String } ### Technical objects type UserSegment { id: ID! @TreeIndex(index:"USER_INDEX") @ReferenceTable(model_type:"SEGMENTS", type:"CORE_OBJECT") creation_ts: Timestamp! @TreeIndex(index:"USER_INDEX") last_modified_ts: Timestamp! @TreeIndex(index:"USER_INDEX") expiration_ts: Timestamp @TreeIndex(index:"USER_INDEX") } type UserScenario { id: ID! @TreeIndex(index:"USER_INDEX") scenario_id: String! @TreeIndex(index:"USER_INDEX") execution_id: String! @TreeIndex(index:"USER_INDEX") node_id: String! @TreeIndex(index:"USER_INDEX") callback_ts: Timestamp @TreeIndex(index:"USER_INDEX") start_ts: Timestamp! @TreeIndex(index:"USER_INDEX") node_start_ts: Timestamp! @TreeIndex(index:"USER_INDEX") active: Boolean @TreeIndex(index:"USER_INDEX") } ### Deprecated identifiers # type UserAgent { # id:ID! # creation_ts: Timestamp! # last_activity_ts: Timestamp # user_agent_info:UserAgentInfo @Function(name:"DeviceInfo", params:["id"]) # } # type UserAgentInfo { # form_factor:FormFactor # brand:String # browser_family:BrowserFamily # browser_version:String # carrier:String # model:String # os_family:OperatingSystemFamily # os_version:String # agent_type:UserAgentType # } ``` ## Syntax highlights ### The `!` operator The `!` operator marks elements as [mandatory](https://graphql.org/learn/schema/#lists-and-non-null). That means the element is expected not to be null. ```graphql type MyType { user_account_id: String # doesn't necessarily have a user account user_account_id: String! # has a user account events: [UserEvent!]! # has a list of events, in which each event can't be null events: [UserEvent!] # doesn't necessarily have a list of events, but lists can't have null elements } ``` {% hint style="warning" %} If you add the `!` operator to a field that happens to have null values, the entire object won't be indexed. It is hard to ensure a field will always have a value in all the data you'll put into the platform, whatever the ingestion method. Therefore, **we recommend not using this operator in your schema for fields other than the predefined ones**. {% endhint %} ### The `ID` type This type is treated as a [keyword string,](/schema#string-indexed-as-keyword) but marks data that is not understandable for a user, as it is an identifier. ```graphql type UserChoice { id: ID! } ``` ### Basic types There is existing multiple native type you can use in your schema. ```graphql type UserProfile { id: ID! creation_ts: Timestamp email: String age: Int active: Boolean } ``` ### Timestamps and dates A best practice is to import objects with dates as `Timestamp` To display the value as date and time when running queries or in exports, you can use the `Date` type. ```javascript // Origin activity { ... "$ts": 1632753811859, "other_date": "2021-09-27T14:43:31.859Z", "other_ts": 1632753811859 ... } ``` ```graphql type UserActivity { ... ts: Timestamp @TreeIndex(index:"USER_INDEX") other_date: Date other_ts: Timestamp date: Date @Function(name:"ISODate", params:["ts"]) ... } ## Doing SELECT { ts other_date other_ts date } ... ## returns ## "ts": 1632753811859, ## "other_date": "2021-09-27T14:43:31.859Z", ## "other_ts": 1632753811859, ## "date": "2021-09-27T14:43:31.859Z", ``` You usually get data as `Timestamp` and generate the `Date` type from the `Timestamp` with the [ISODate](/schema#isodate) function. If not, then ensure you get data in the correct format. There is no implicit conversion between timestamps and dates. ```graphql // Origin activity { ... "other_date": 1632753811859, ... } type UserActivity { # This won't work as received data is a timestamp. other_date: Date } ## SELECT { other_date } ... ## throws an error ``` {% hint style="info" %} Both types can are compatible with [Date operators](/querying-your-data/otql-queries#date-operators) in queries. \ \ Only use one [@TreeIndex](/schema#treeindex) directive when creating a date from a timestamp : this will save space in the index and both types have the same capabilities in the query. {% endhint %} ## Directives ### @TreeIndexRoot This directive marks the root element of an Object Tree Index. The `index` property marks the name of the Object Tree Index It should always be `USER_INDEX` as multiple indexes are not currently supported. ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX"){ } ``` ### @TreeIndex This directive makes a field available in the `WHERE` clause and in [Aggregation operations](/querying-your-data/otql-queries#aggregation-operations) of your [OTQL queries](/schema). Fields that don't have this directive can't be used in the `WHERE` clause but can still be retrieved in the `SELECT` clause. ```graphql type UserEvent { id:ID! ts:Timestamp! # url and referrr properties are now available in WHERE clauses url:String @TreeIndex(index:"USER_INDEX") referrer:String @TreeIndex(index:"USER_INDEX") } ``` Don't mark every field with this directive. Some fields, like first name, last name ... will never be used in `WHERE` clauses and would only make your index larger. {% hint style="warning" %} The @TreeIndex directive is mandatory for some default properties. They already have that directive in the default schema, and you shouldn't remove it, or your schema won't be validated. The value of the index in @TreeIndex should always be USER\_INDEX. {% endhint %} When registering a `String` in a Tree Index with the directive `@TreeIndex`, you can specify how the field should be indexed, depending on how you want to use it later. Two modes are available, `text` and `keyword`. ```graphql type myType { mystring:String @TreeIndex(index:"USER_INDEX", data_type: "text") secondstring:String @TreeIndex(index:"USER_INDEX", data_type: "keyword") } ``` #### String indexed as `text` This mode is considering your value as a **set of words** (e.g. a text). For example, the value 'The QUICK brown fox JuMpS, over the Lazy doG.' will be considered as the list of: * the * quick * brown * fox * jumps * over * lazy * dog. As you can see, some transformations were done before storing the data: * all the words were put in lowercase -> **all string operators will be case insensitive** on a field indexed with data\_type: `text` * the original string was split, and the splitting characters were removed (here, it was `,` `.` and `,`) The method used to split the words together is described [in great details here.](http://unicode.org/reports/tr29/) The most common characters that trigger a split are (non-exhaustive list): * (space) * \- * " * ‘ * , * ; * ? * ! * / Note that the following characters do not trigger a split (non-exhaustive list): * . * \_ * ' * ’ The `data_type: "text"` mode should be used when you're working with: * Full sentences (ex. a Page Title) * URLs * List of keywords (separated by a splitting character as listed above) * similar text Generally, this mode is used when you don't have great control over the value being collected in this field, and you want to do "broad" queries based on it. #### String indexed as `keyword` This mode is used to consider your value as a **single word**. No transformation is done with the provided value. The data\_type: "keyword" mode should be used when you're working with: * Single values * Ids passed as text (ex: UUIDs, productId, categoryId, etc.) * Every time that you already know the values that are passed in the field (e.g. when the field data is linked to a taxonomy) * etc. Generally, this mode is used when you have great control over the value being collected in this field, and you want to do exact queries on it later by doing exact equality in queries. ### @Property By default, the path associated with each of your properties is the name of these properties. You can change this behavior with the @Property directive. ```graphql type UserEvent { id:ID! ts:Timestamp! name:String! # We are creating shortcuts to the $url, $referrer and $items properties # that are normaly in a $properties object in the user event. # This will make them easier to query url:String @Property(path:"$properties.$url") referrer:String @Property(path:"$properties.$referrer") products:[Product] @Property(path:"$properties.$items") } type Product { # Here we simply change the name into id and name instead of $id and $name id: String @TreeIndex(index:"USER_INDEX") @Property(path:"$id") name: String @TreeIndex(index:"USER_INDEX") @Property(path:"$name") } ``` All the properties in the default schema already redefine their path. For example, the `creation_ts` property in the `UserPoint` object points to the `$creation_ts` property in the stored data. The declaration should theoretically have used the @Property directive, but it is unnecessary to do the work for you. ```graphql type UserPoint { # What should have been declared creation_ts: Timestamp! @Property(path:"$creation_ts") # What is declared as a shortcut creation_ts: Timestamp! } type Product { # We do have to use the @Property directive as those properties # don't exist in the default schema for a Product object type id: String @Property(path:"$id") name: String @Property(path:"$name") } ``` #### Taking value from multiple paths You can define multiple paths to get the data from. If the first path is empty, the second one will be used and so one. In this example, user activities channel ID is either the site ID or the app ID depending on the user activity's context. ```graphql type MyType { channel_id: String @Property(paths:["$site_id", "$app_id"]) } ``` #### Available tokens You can use the \[parent] token to go up in the object tree when defining a path ```graphql type MyType { creative_id:String @Property(path:"[parent].[parent].$origin.$creative_id") } ``` ### @Mirror This directive allows you to create custom types based on predefined types. ```graphql # UserEvent type has been renamed ArticleView # Not really interesting and should be avoided type ArticleView @Mirror(object_type:"UserEvent"){} # More advanced usage : ArticleView object are UserEvents # with a name of "navigation.article" type ArticleView @Mirror(object_type:"UserEvent", filter:"name == \"navigation.article\""){} ``` #### Sample usage: custom types with filters ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX"){ ### basketviews: [BasketView] productviews: [ProductView] } type BasketView @Mirror(object_type:"UserEvent", filter:"name == \"$basket_view\""){} type ProductView @Mirror(object_type:"UserEvent", filter:"name == \"$page_view\""){} ``` ### @Function The @Function directive is used to declare a calculated field with a set of predefined functions. #### ISODate This function creates a date from a timestamp. ```graphql type MyType { # creation_date is a Date created from the timestamp creation_ts creation_date:Date! @Function(name:"ISODate", params:["creation_ts"]) } ``` #### ThirdPartyCookieMappings In order to retrieve third party cookie mappings for a given device point, the `ThirdPartyCookieMapping` function can be used: ```graphql type UserDevicePoint { id:ID! @TreeIndex(index:"USER_INDEX") ... mappings:[UserAgentMapping!]! @Function(name:"ThirdPartyCookieMappings", params:["id"]) } type UserAgentMapping { last_seen:Timestamp user_agent_id:String vector_id:String } ``` The function works on device points that have a device technical id of type MUM\_ID attached to them. It translates the MUM\_ID into a vector\_id (`mum:-1234` -> `vec:1234`) and retrieves attached partners' 3P cookies. {% hint style="info" %} See [user\_agent\_id section](/user-points/user-identifiers/userdevicetechnicalid#user_agent_id) in the device identifiers documentation for more information on the user\_agent\_id formatting (including partners' 3P cookies). {% endhint %} #### DeviceInfo (legacy) {% hint style="warning" %} This function is only used on datamarts referencing the legacy type `UserAgent`. For datamarts referencing the new type `UserDevicePoint`, we suggest to use the previous function: `ThirdPartyCookieMappings`. {% endhint %} This function extracts device information for an agent identifier. ```graphql type UserAgent { id:ID! @TreeIndex(index:"USER_INDEX") user_agent_info:UserAgentInfo @Function(name:"DeviceInfo", params:["id"]) } ``` The `UserAgentInfo` class has the following properties: ```graphql type UserAgentInfo { form_factor:FormFactor brand:String browser_family:BrowserFamily browser_version:String carrier:String model:String os_family:OperatingSystemFamily os_version:String agent_type:UserAgentType } ### The following enums are predefined. ### It is not necessary to define them enum FormFactor { WEARABLE_COMPUTER TABLET SMARTPHONE GAME_CONSOLE SMART_TV PERSONAL_COMPUTER OTHER } enum BrowserFamily { OTHER CHROME IE FIREFOX SAFARI OPERA STOCK_ANDROID BOT EMAIL_CLIENT MICROSOFT_EDGE } enum OperatingSystemFamily { OTHER WINDOWS MAC_OS LINUX ANDROID IOS } enum UserAgentType { WEB_BROWSER MOBILE_APP } ``` ### @ReferenceTable When users create their queries using your schema, they usually remember some elements they search for but don't know their identifiers. You can add the @ReferenceTable directive to fields storing channels, compartments and segment identifiers. That way, the user will have an autocomplete with the element's name instead of their identifier when creating his queries. ```graphql type UserSegment { id:ID! @ReferenceTable(type:"CORE_OBJECT", model_type:"SEGMENTS") @TreeIndex(index:"USER_INDEX") } type UserActivity { channel_id:String @ReferenceTable(model_type:"CHANNELS", type:"CORE_OBJECT") @TreeIndex(index:"USER_INDEX") @Property(paths:["$site_id", "$app_id"]) } type UserProfile { compartment_id:String! @ReferenceTable(model_type:"COMPARTMENTS", type:"CORE_OBJECT") @TreeIndex(index:"USER_INDEX") } type UserEvent { channel_id:String @ReferenceTable(model_type:"CHANNELS", type:"CORE_OBJECT") @Property(paths:["[parent].$site_id", "[parent].$app_id"]) @TreeIndex(index:"USER_INDEX") } ``` ### @EdgeAvailability This directive marks properties as usable in queries when creating [Edge segments](/schema). ```graphql type UserAccount { id:ID! # This property won't be usable in Edge segment queries compartment_id:String! # This property will be usable in Edge segment queries user_account_id:String! @TreeIndex(index:"USER_INDEX") @EdgeAvailability } ``` ### @ComputedField This directive marks properties as calculated from a [Computed Field Function](/schema/computed-fields). ```graphql type UserPoint { id: ID! accounts: [UserAccount] … rfm_score: RfmScore @ComputedField(technical_name = “RfmScore”) @TreeIndex(index:"USER_INDEX") } type RfmScore { … } ``` ### @UpdateStrategyKey This directive is used in order to make [targeted updates on objects with an ID property in a UserProfile](/data-ingestion/bulk-processing/imports/profiles) ```graphql type UserProfile { … loyalty: Loyalty } type Loyalty { cards:[LoyaltyCards] } type LoyaltyCards { card_id : String! @UpdateStrategyKey benefits : String last_visit_date : String } ``` ### @UpdateValueObject This directive is used in order to make[ targeted updates on objects that do not not have ID properties in a UserProfile](/data-ingestion/bulk-processing/imports/profiles) ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String segmentations: [Segmentation] } type Segmentation @UpdateValueObject { type: String label: String } ``` ## Best practices ### Do not index ISODate function result Do not index the output of the ISODate Function. You should index the timestamp value only. ```graphql # DO type UserAgent { creation_ts:Timestamp! @TreeIndex(index:"USER_INDEX") creation_date:Date! @Function(name:"ISODate", params:["creation_ts"]) user_agent_info:UserAgentInfo @Function(params:["id"], name:"DeviceInfo") id:ID! last_activity_ts:Timestamp } # DON'T type UserAgent { creation_ts:Timestamp! creation_date:Date! @Function(name:"ISODate", params:["creation_ts"]) @TreeIndex(index:"USER_INDEX") user_agent_info:UserAgentInfo @Function(params:["id"], name:"DeviceInfo") id:ID! last_activity_ts:Timestamp } ``` ### UserEvent indexed twice In some scenarios, you could have events directly in the UserPoint and in user activities. For example, to use frequency OTQL directives on UserEvents and build queries on several events that occurred on a single activity. In any other case, do not duplicate the UserEvent. Either use it in the UserPoint or the user activity. ```graphql # Only do if in a specific scenario requiring it type UserPoint @TreeIndexRoot(index:"USER_INDEX"){ ### activities: [UserActivity!]! events:[UserEvent!]! } type UserActivity { ### events: [UserEvent!]! } type UserEvent @Mirror(object_type:"UserEvent") { name:String! @TreeIndex(index:"USER_INDEX") id:ID! ts:Timestamp! } ``` # Defining your schema [Schemas](/schema) are associated with [datamarts](/communities-and-organisations/datamart) by managing objects with the following properties: {% hint style="success" %} Make sure you learned about [schema's concepts and how they are structured](/schema). {% endhint %} ```javascript // DRAFT Schema { "id": "1385", "datamart_id": "1509", "status": "DRAFT", "creation_date": 1609888013947, "last_modification_date": 1610443926552, "publication_date": null, "suspension_date": null } // LIVE Schema { "id": "1281", "datamart_id": "1509", "status": "LIVE", "creation_date": 1603198414771, "last_modification_date": 1603198415117, "publication_date": 1603198415733, "suspension_date": null } // ARCHIVED Schema { "id": "1276", "datamart_id": "1509", "status": "ARCHIVED", "creation_date": 1603102827888, "last_modification_date": 1603102828087, "publication_date": 1603102828479, "suspension_date": 1603102848269 } ``` ## Schema publication workflow The process for publishing a schema is as follows: 1. Create a new schema definition 2. Upload the schema associated with the definition 3. Validate the schema 4. Publish the schema ## Schema updates After updating a schema, you can immediately use all its properties into the `select` part of your [OTQL queries](/querying-your-data/otql-queries) and any operator that doesn't require indexing. If you add a new indexed property, only new elements going into your datamart will be indexed. You will be able to run `WHERE` queries and operators needing indexing, but your queries will not return values for elements already in your datamart. You can add a new indexed property by either adding a new property with [@TreeIndex directive](/schema#treeindex) or adding the @TreeIndex directive to an existing property. If you remove an indexed property, you will instantly stop being able to run `WHERE` queries and operators needing indexing for this query. {% hint style="success" %} You can ask your mediarithmics contact to start a complete reindexing of your datamart if required {% endhint %} ## Manage schemas with mics CLI You can use the [MICS CLI](/resources/tools-and-libraries/mics-cli) to power up your schema definitions workflow. ### Download a schema Retrieve the LIVE schema or the specified one for a given datamart, and save it as a `.gql` file named `schema--.gql`. Use it to download the schema you want to start with, usually the current LIVE version. ``` USAGE $ mics-cli schema:fetch DATAMARTID [SCHEMAID] ARGUMENTS DATAMARTID the ID of the datamart SCHEMAID [default: LIVE] the ID of the schema, or LIVE to get the live schema OPTIONS --stdout output the content of the file instead of saving it in a file ``` ![](/files/-MQrPw_Ghbuk-qCyjzzo) If you want to show the schema in the output, use the `--stdout` flag.
### Update a schema Create a draft, upload a file, validate it, and publish the schema with this all-in-one command. Automatically handles status and [cloning best practices](/schema/defining-your-schema#create-a-draft-by-cloning): * If a draft schema already exists, update it before publishing. * Latest LIVE schema is cloned, if available, and `--noClone` flag is not set You can keep your schema as a DRAFT and not publish it using the `--skipPublish`flag.
## Get existing schemas `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas` Returns all schemas and their status. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ``` { "status": "ok", "data": [ { "id": "1266", "datamart_id": "1509", "status": "ARCHIVED", "creation_date": 1602860201358, "last_modification_date": 1602860221562, "publication_date": 1602860275189, "suspension_date": 1602860362588 }, { "id": "1281", "datamart_id": "1509", "status": "LIVE", "creation_date": 1603198414771, "last_modification_date": 1603198415117, "publication_date": 1603198415733, "suspension_date": null }, { "id": "1385", "datamart_id": "1509", "status": "DRAFT", "creation_date": 1609888013947, "last_modification_date": 1610443926552, "publication_date": null, "suspension_date": null } ], "count": 3, "total": 3, "first_result": 0, "max_result": 2147483647, "max_results": 2147483647 } ``` {% endtab %} {% endtabs %} {% hint style="success" %} Archived schemas are displayed, meaning you can watch your history {% endhint %} ## Get a schema `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId` #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | | schemaId | integer | The ID of the schema | {% tabs %} {% tab title="200 " %} ``` { "status": "ok", "data": { "id": "1266", "datamart_id": "1509", "status": "ARCHIVED", "creation_date": 1602860201358, "last_modification_date": 1602860221562, "publication_date": 1602860275189, "suspension_date": 1602860362588 } } ``` {% endtab %} {% endtabs %} ## Get the content of a schema `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/text` Allows you to visualize how your schema is in the current version, or how it has been in archived versions #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | string | The ID of the datamart | | schemaId | integer | The ID of the schema | {% tabs %} {% tab title="200 " %} ``` ## type UserPoint @TreeIndexRoot(index:"USER_INDEX") { events:[ActivityEvent!]! creation_ts:Timestamp! @TreeIndex(index:"USER_INDEX") id:ID! } ## type ActivityEvent { referrer:String @TreeIndex(index:"USER_INDEX") @Property(path:"$properties.$referrer") url:String @TreeIndex(index:"USER_INDEX") @Property(path:"$properties.$url") date:Date! @TreeIndex(index:"USER_INDEX") @Function(params:["ts"], name:"ISODate") nature:String @Property(path:"$event_name") @TreeIndex(index:"USER_INDEX") ts:Timestamp! id:ID! } ``` {% endtab %} {% endtabs %} ## Create a DRAFT by cloning `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/clone` Creates a new DRAFT schema if there's no existing one, by cloning an existing one. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------------------------------------------------- | | schemaId | integer | The ID of the schema to clone. Usually the current LIVE version. | | datamartId | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "1395", "datamart_id": "1509", "status": "DRAFT", "creation_date": 1610449867207, "last_modification_date": 1610449867207, "publication_date": null, "suspension_date": null } } ``` {% endtab %} {% tab title="400 If you try to create a new schema while there's already one draft" %} ```javascript { "status": "error", "error": "Impossible to create a new schema, there is already one draft schema", "error_code": "CONSTRAINT_VIOLATION_EXCEPTION", "error_id": "690596a9-b0e0-43b3-88e8-b90d08b98029" } ``` {% endtab %} {% endtabs %} {% hint style="success" %} This is the preferred method to create a DRAFT schema, as it will keep all settings from the previous version, like cluster versions, and index sizes. {% endhint %} ## Create a DRAFT without cloning `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas` Creates a new DRAFT schema if there's no existing one without cloning. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "1395", "datamart_id": "1509", "status": "DRAFT", "creation_date": 1610449867207, "last_modification_date": 1610449867207, "publication_date": null, "suspension_date": null } } ``` {% endtab %} {% tab title="400 If you try to create a new schema while there's already one draft" %} ```javascript { "status": "error", "error": "Impossible to create a new schema, there is already one draft schema", "error_code": "CONSTRAINT_VIOLATION_EXCEPTION", "error_id": "690596a9-b0e0-43b3-88e8-b90d08b98029" } ``` {% endtab %} {% endtabs %} {% hint style="warning" %} For the schema to keep initial settings—for example, elastic search version, and index size—you need to clone the previous LIVE version. Only use this endpoint if you know the required settings and you can set them up before publishing. {% endhint %} ## Upload the content `PUT` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/text` Add the schema to content to the raw body of the request #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | | schemaId | integer | The ID of the schema | #### Request Body | Name | Type | Description | | ---- | ------ | -------------------- | | body | string | Raw schema to upload | {% tabs %} {% tab title="200 " %} ``` { "status": "ok" } ``` {% endtab %} {% endtabs %} ## Validate a DRAFT schema `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/validation` Tells you if the uploaded schema is valid, and shows errors if there are any. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | | schemaId | integer | The ID of the schema | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "datamart_id": "1509", "schema_id": "1266", "tree_index_operations": [ { "datamart_id": "1509", "index_selection_id": "2121", "index_name": "USER_INDEX", "init_strategy": "FORCE_NO_BUILD", "driver_version_major_number": 1, "driver_version_minor_number": 2, "current_index_id": "574", "current_index_size": "SMALL", "new_index": false, "new_index_size": "SMALL", "init_job": null, "error_code": null, "error_message": null } ], "schema_errors": [] } } ``` {% endtab %} {% tab title="400 If there are validation errors" %} ```javascript { "status": "error", "error": "2 error(s) found when validating schema : Type 'UserPoint' which is root of tree index 'USER_INDEX' requires a scalar field named 'creation_ts' to be annotated with '@TreeIndex(index:\"USER_INDEX\")' and to be typed 'Timestamp!', Type 'UserAccount' which is root of tree index 'USER_INDEX' requires a scalar field named 'compartment_id' to be annotated with '@TreeIndex(index:\"USER_INDEX\")' and to be typed 'String!'", "error_code": "BAD_REQUEST_DATA", "error_id": "6a52cea9-6de4-40f5-972e-a8480c268d19" } ``` {% endtab %} {% endtabs %} ## Publish a schema `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/publication` The selected schema goes LIVE, and the actual LIVE schema is ARCHIVED. Will show an error if you didn't validate the schema. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | | schemaId | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "datamart_id": "1509", "schema_id": "1395", "tree_indices": [ { "index_name": "USER_INDEX", "new_index": false, "index_id": "574", "index_size": "SMALL", "init_strategy": "FORCE_NO_BUILD", "driver_version_major_number": 1, "driver_version_minor_number": 2, "init_job": null } ] } } ``` {% endtab %} {% tab title="400 If the schema has not been validated" %} ```javascript { "status": "error", "error": "Impossible to publish schema, validate the schema before a new publication", "error_code": "BAD_REQUEST_DATA", "error_id": "76e70020-02cb-40ec-8109-adf204aa0a7b" } ``` {% endtab %} {% endtabs %} # Decorators Schema decorators allow you to customize how your graph appear within the mediarithmics platform interfaces, specifically the Advanced Segment Builder and the Query Tool. By uploading a specific CSV file, you can "shallow-rename" fields (change their display label) or hide specific properties from users without altering the underlying technical schema. ## Decorator CSV Format Schema decorators are defined using a CSV file. Below are the specifications for the file format. #### CSV Dialect Rules * Separator: Comma (`,`) * Quotes: Double quotes (`"`) must be used for strings that contain commas. #### Columns Specification The CSV must contain the following headers. | Header | Description | | ------------ | --------------------------------------------------------------------------------------------------------- | | OBJECT\_NAME | The name of the object (resource) where the property is located. | | FIELD\_NAME | The original technical name of the property to decorate. | | HIDDEN | `true` or `false`. Indicates if the field should be hidden from the UI. | | LABEL | The user-friendly name to display in the Segment Builder and Query Tool. | | HELP\_TEXT | A short description displayed as a tooltip when hovering over the question mark (?) icon in the platform. | | LOCALE | The locale for the label. Currently, this must be set to `en-US`. | #### Example Code snippet ``` OBJECT_NAME,FIELD_NAME,HIDDEN,LABEL,HELP_TEXT,LOCALE UserPoint,id,false,User ID,Unique identifier for the user point,en-US UserPoint,technical_hash,true,,,en-US UserEvent,subcategory,false,Subcategory,The specific sub-category of the event,en-US UserEvent,subcategory_id,true,,,en-US UserEvent,title,false,Event Title,The main title or headline of the event,en-US UserEvent,region,false,Region,The geographic region associated with the event,en-US ``` {% hint style="info" %} Note on Character Encoding You must normalize strings by removing accents. Characters such as `é`, `è`, or `à` are not supported in the CSV file and should be replaced with their non-accented counterparts. {% endhint %} ## Managing Decorators You can manage schema decorators either through the mediarithmics user interface (Navigator) or programmatically via the API. ### Via User Interface In the **Datamart > Object View Configuration** section of the Navigator Settings, you can use the buttons under the Schema to manage your CSV files directly. The available actions are: * **Upload new Decorators**: Allows you to upload a prepared CSV file to apply new labels and visibility rules. This will overwrite existing decorators for this schema. * **Download Template**: Downloads a blank CSV file containing the required headers (`OBJECT_NAME`, `FIELD_NAME`, etc.) to help you get started. * **Download Decorators**: Downloads the current active decorator CSV file. This is useful if you want to make edits to the existing configuration. * **Delete Decorators**: Removes the current decorator file. The schema will revert to displaying raw technical field names, and hidden fields will become visible again. ### Via API #### Prerequisites You will need the following values: * `DATAMART_ID`: The ID of your Datamart. * `SCHEMA_ID`: The ID of the Schema you wish to decorate. * `MICS_API_TOKEN`: Your API authentication token. #### Retrieve Current Decorators To fetch the existing decorator file for a specific schema: Bash ``` curl -H "Authorization:$MICS_API_TOKEN" \ -X GET \ --location "https://api.mediarithmics.com/v1/datamarts/$DATAMART_ID/graphdb_runtime_schemas/$SCHEMA_ID/schema_decorators" ``` #### Update Decorators To upload a new decorator CSV file (replacing the existing configuration): Bash ``` curl -H "Authorization:$MICS_API_TOKEN" \ -X PUT \ --location "https://api.mediarithmics.com/v1/datamarts/$DATAMART_ID/graphdb_runtime_schemas/$SCHEMA_ID/schema_decorators" \ --data-raw "$(cat path/to/schema_decorators.csv)" ``` # Computed fields {% hint style="info" %} This feature is in Alpha. {% endhint %} ## Definition A computed field is a dynamic field defined in the schema, linked to a script that is triggered for each new user activity or profile update and performs a computation of a function's result **on a regular basis**. It is used when the desired outcome cannot be achieved with existing directives or to simplify an OTQL query by replacing multiple directives with a single field. Those fields can be used in segment definition to improve analysis and sharing, or in the ***Experiment*** feature to enhance control groups creation. ### Examples Those examples are handled by computed fields: * Retrieve the most recent event (e.g. last visit, last transaction, etc.) * Get the total amount for a specific type of event (e.g. sum of transactions in the last 30 days, total expenditure on a particular category of product, etc.). * Weighted sum (Affinity total, etc.) * Compare multiple channels (e.g. to find the best channel for transactions). This example should be handled by a standard directive: * Get all users with at least a specific amount for one category This example can't be handled by a computed field: * Identify the top X% of a specific user group (e.g. the top 10% of buyers). ### **Usage Example** Once live, a computed field behaves like a regular field, making it transparent to the user. Here's how you can use it in practice: > Use Case: Identify Users Who Have Spent More Than 100€ on IT Products in the Last 3 Months **With Standard Directives:** ```sql // With standard directives SELECT { id } FROM UserPoint WHERE activity_events @ScoreSum(min : 100) { basket { items @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" AND date >= "now-3M/M" } } } ``` This query uses standard directives to sum the basket amounts for **IT** products over the last 3 months. **With a Computed Field:** ```sql //With a COmputed field "IT_amount_3months" SELECT { id } FROM UserPoint WHERE IT_amount_3months >= 100 ``` In this version, the computed field `IT_amount_3months` is used directly in the query. The field is calculated on a regular basis, meaning no computation is performed during query execution. This results gives faster query response times, as the value is precomputed and readily available. # Concepts ## Technical concept Computed fields are defined in the schema at the UserPoint level. The plugin behind it performs computations on a regular basis of a function's result, based on the user activities and profile updates. ## Definitions ### Computed Field A computed field is an instance of a Datamart function applied within a specific context. To use it, it must be declared in your schema. ### Computed Field Function A Computed Field Function is a [plugin](/advanced-usages/plugins) of type `COMPUTED_FIELD_FUNCTION`, which can be created via the API following the standard [process](/advanced-usages/plugins/creation) or through the computing console interface. ## Notions A computed field is defined by these elements: 1. **State**: The **data stored** to build the result. It is the history. Each new activity updates the **State**. → the state is closely related to the *Lookback Window* - i.e. the historical depth of events to be analyzed. 2. **Logic**: The logic can be very simple (*like a formula, a sum, a count…*) or more complex (*like a logical operation*). 3. **Result**: The **data calculated and stored**. It contains the score(s) to be used. The result can be either a single variable or an object, depending on the complexity of the computation. It is computed from the **State** Other interesting concepts to take into consideration: * **Input Data**: Either new userActivities, either userProfile update, either both. * **Period Update**: The regularity of updates (every x days - by default x=1). ## Lifecycle 1. **INITIAL** – The instance is created with this status. 2. **INITIAL\_LOADING** – Once declared in the schema and successfully validated, an initial loading job is triggered. 3. **ACTIVE** – The computed field becomes active if the job is completed successfully. Computed field lifecycle ## Technical flow To better understand how a computed field works, here are the technical steps and the relation with the methods: 1. **Initialization**: The computed field is declared in the schema. 2. **State Updates**: Triggered by new user activities or profile updates. 3. **Result Calculation**: The result is computed based on the current state. 4. **Storage**: The result is stored for querying. # Setup {% hint style="info" %} Computed fields are created as of plugin, requiring collaboration with mediarithmics Professional Services (PS) team. The implementation involves defining the logic, writing plugin specifications, developing the plugin, and configuring the computed fields. Please discuss your needs with your account manager. {% endhint %} 1. [Create your computed field function (plugin)](#step-1-create-your-computed-field-function-plugin) 2. [Create your computed field within the platform](#step-2-create-your-computed-field-within-the-platform) 3. [Declare your computed field in your schema](#step-3-declare-your-computed-field-in-your-schema) 4. [Wait until the end of the initial loading](#step-4-wait-until-the-end-of-the-initial-loading) 5. [Use your computed field](#step-5-use-your-computed-field) ## Step 1: Create your computed field function (plugin) Check the [following documentation to learn more about Plugin development](/advanced-usages/plugins/creation-deployment). Bellow are some precisions about specific Computed fields Function implementation. ### Develop the plugin with the SDK and create a build from it {% hint style="info" %} You are required to implement 4 key functions from the Plugin SDK. {% endhint %} #### OnUpdate functions These 3 functions are triggered during the initialization of the Computed Field Function and whenever a new activity, profile update, or computed field modification occurs, updating the `State` accordingly. [You can find the list of all the methods to be developed here](/schema/computed-fields/development#list-of-methods). {% hint style="warning" %} Although `onUpdateComputedField` has not been released yet, it is still required to be implemented in your plugin. In the meantime, simply return the `State` directly to avoid any errors. {% endhint %} #### BuildResult This function returns the result of the computed field function for a specific `State`. The `Result` will be stored for a defined duration, and querying the field will return the stored value during that period. [You can find all the details about this method here](/schema/computed-fields/development#buildresult). ### Create a plugin and a version Follow the standard procedure to [create a plugin](/advanced-usages/plugins) with the type `COMPUTED_FIELD_FUNCTION`. Once the plugin is created, you can [generate a new version](/advanced-usages/plugins#versions) from the build. ## Step 2: Create your computed field within the platform 1. In the **Navigator > Settings > Datamart**, navigate to **Computed Fields**. 2. Add a **new computed field**. 3. Select the **plugin** and **version** of the new instance you want to use. 4. Complete the necessary fields:
FieldsDescription
NameName of the computed field
Description(Optional) Describe the computed field
Technical nameName of the computed field used in the schema
Compute period (in days)Maximal duration before the periodic update of the Result in days.
Events filterGraphQL selector query to filter the activity which will trigger the State updates
This will set up the computed field linked to your computedfield function for use in the schema. ## Step 3: Declare your computed field in your schema After creating the computed field, you need to connect it to your schema: 1. Use the directive `@ComputedField(technical_name = "...")` at the appropriate level of your schema. 2. Ensure the **technical name** matches the one used when creating the computed field. This will integrate the computed field into your schema and make it available for use in queries and operations. ```graphql // Example in your schema type UserPoint { id: ID! accounts: [UserAccount] … rfm_score: RfmScore @ComputedField(technical_name = “RfmScore”) @TreeIndex(index:"USER_INDEX") } type RfmScore { … } ``` A **schema validation** will be triggered when you click **Save**. It will ensure that your Datamart has access to all the declared computed fields and verify their integration with the schema. {% hint style="warning" %} The computed field must be in the UserPoint object. {% endhint %} ## Step 4: Wait until the end of the initial loading Once your schema is validated, our service will run a job to initialize your computed field and set the state to match the current status of the timeline. For each UserPoint in your Datamart, we will process their timeline and execute the **OnUpdateXXX()** function. To monitor the progress: 1. Go to the **Computing Console** > **Computed Fields**. 2. Check if your computed field is ready or if the initial loading is still in progress. Please note, this process may take some time to complete (up to 72h) as it computes the timelines for all UserPoint. {% hint style="danger" %} Avoid using your computed field until the initial loading is complete. While the query will not return an error, the result may be inaccurate during this process. {% endhint %} ## Step 5: Use your computed field Once the initial loading is complete, you can start using your computed field just like any other standard field in mediarithmics features. For example, in **query tools** or any **OTQL** query: ```sql SELECT { rfm_score } FROM UserPoint ``` If your computed field is indexed, you can also perform queries like: ```sql SELECT { id } FROM UserPoint WHERE rfm_score = "PASSIVE" ``` This allows you to incorporate the computed field into your data analysis and queries once it's fully initialized. # Development ## List of Methods ### OnUpdate methods These functions are triggered during the initialization of the Datamart Function and whenever a new activity, profile update, or computed field modification occurs, updating the `State` accordingly. {% hint style="warning" %} These methods must be **commutative** to ensure consistency during initial loading, where event order is not guaranteed. It should always produce the same result, regardless of the sequence in which events are processed. {% endhint %} {% hint style="danger" %} To prevent system overload, the `State` size is restricted to a maximum of **1 MB**. If this limit is exceeded, the update will be discarded and not saved. {% endhint %} #### onUpdateActivity `onUpdateActivity(state: State, userActivity: UserActivity): State` * **Inputs**: * `state`: Current state object. * `userActivity`: New user activity data. * **Outputs**: Updated state object. * **What It Does**: Updates the state with new user activities and removes outdated activities. * **Example**: 
// Trigger by a new activity
onUpdateActivity(state, userActivity) {
    // Logic to update state
    return updatedState;
}
{% hint style="info" %} The activities or profiles received by the plugin are raw data, so you can't rely on the structure defined in your schema to understand its design. However, you can refer to the structure outlined on the **User Lookup** page. {% endhint %} #### onUpdateUserProfile `onUpdateUserProfile(state: State, userProfile: UserProfile, operation: core.Operation): State` * **Inputs**: * `state`: Current state object. * `userProfile`: Updated userProfile data. * `operation`: Operation type (enum: UPDATE | DELETE). * **Outputs**: Updated state object. * **What It Does**: Handles updates to userProfile. * **Example**: ```jsx // Trigger by an update on the UserProfile onUpdateUserProfile(state, userProfile, operation) { // Logic to handle UserProfile updates return updatedState; } ``` #### onUpdateComputedFields `onUpdateComputedFields(state: State | null): Result` * **Inputs**: `state`: Current state object or null. * **Outputs**: Result object. * **What It Does**: Computes the result from the current state. * **Example**: ```jsx // Trigger by the result computation of another computed field onUpdateComputedFields(state) { // Logic to compute result return state; } ``` {% hint style="danger" %} This method must be implemented, even if it not currently used. Please return the same state. {% endhint %} ### buildResult `buildResult(state: State | null): Result` * **Inputs**: `state`: Current state object or null. * **Outputs**: Result object. * **What It Does**: Computes the result from the current state. * **Example**: ```jsx buildResult(state) { // Logic to compute result return result; } ``` ## Test Before deploying the plugin, create tests to ensure the function behaves as expected. Test cases might include: * Adding new activities, and update profiles and verifying the state is updated correctly. Check that the order we send profile updates and activities has no impact. * Checking that activities which are no more relevant to compute the result are removed from the state. * Validating that the correct values are returned in the result. ## Best Practices * How to decide if we create 1 or several plugin(s)? *(example with 2 scores)* * If the required data in the state is the same for both scores → 1 plugin * If the required data in the state is different (state is polymorph) → 2 plugins * Always return the same state in case of error, or null. Otherwise, we delete the state, and so reset the result. * Maximize the number of scores calculated by a Computed Field Function to optimize performance. Therefore, the same state is used to calculate multiple scores, and we avoid storing the same state multiple times. * Ensure that the state is designed to store relevant information efficiently * Ensure that the state is commutative to maintain consistency while building the result.\ ie. state activities and profiles data updates can happend in any order * Be sure to have a cleaning rule to update the state base on the defined lookback window for both userProfile and userActivities. * In other words, when updating the state, be sure to remove the data that are no more useful to compute the result, in order to limit the state size. * When editing a live computed field, you must relaunch an Initial Loading through API. # Examples ## Context for the following example To demonstrate how the computed field works, consider this use case: > **Goal**: Calculate the sales **over 12 months** for items in the **IT category** per user. For simplicity, this example focuses on the **IT category**. However, the computed field can be scaled to compute the amount for all categories, not just IT. In this scenario, the computed field will: 1. Aggregate the order amounts over the last 12 months. 2. Filter the data based on the **IT category** (or any other category, depending on the use case). By scaling this approach, you can calculate the sales amount for multiple categories, ensuring flexibility and extensibility in your calculations. ## **Step 1: Define your context for the Datamart Function** In this use case, we need to declare the **State**, **Result**, and **UserActivity** for computing the sales amount over the last 12 months for items in the **IT category**. ### **State Declaration** The **State** stores the order amounts for each day, categorized by **IT** items, for the last 12 months. Here's how the state structure looks : ```typescript export interface State { activities_for_the_last_12 _months: { [date: number] : [{ amount: number; }] } } ``` The `activities_for_the_last_12_months` keeps the amount data for each day, where the `date` is represented by a numeric value (e.g., timestamp), and the `amount` is the total order value for that day. ### **Result Declaration** The **Result** represents the total sales amount over the last 12 months for the **IT category**. ```typescript export interface Result { IT_amount_for_the_last_12_months: number; } ``` The result will return the computed `IT_amount_for_the_last_12_months` after summing up the values stored in the state. ### **UserActivity Declaration** The **UserActivity** defines the structure of the activity that triggers the update. In this example, we focus on the items bought, particularly in the IT category. ```typescript interface Items { category: string; price: number; } interface UserActivity { items_bought: Items[]; } ``` The `items_bought` array contains details about each item, such as `category` (IT, for example) and `price`. {% hint style="info" %} To get some examples of your activity structure, you can look at the activity returned in the User lookup page in Navigator. {% endhint %} ## **Step 2: Declare your Computed Field class** With the context declared, you can implement your computed field logic. Here's how the **MyComputedField** class looks: ```typescript export class MyComputedField extends core.ComputedFieldPlugin { constructor() { super(); } // Function to Update the state onUpdateActivity(state: State, userActivity: UserActivity): State { ... } // Won't be used but need to be declared; onUpdateUserProfile(state: State, userProfile: UserProfile, operation: core.Operation): State { return state; } // Won't be used but need to be declared; onUpdateComputedField(state: State, computedField: ComputedField): State { return state; } // Function to compute the Result buildResult(state: State | null): Result { ... } } ``` ### **onUpdateActivity** * **Goal**: Update the state with new IT category purchases and remove activities older than 12 months. * It filters the UserActivity to ensure only IT items are included. * Removes old activities beyond the 12-month period. {% hint style="info" %} **Best practice:** Use the `Events filter` to exclude activities or profiles that are not relevant to your computed field, rather than adding a filter within your function. {% endhint %} ### **buildResult** * **Goal**: Sum all the basket amounts stored in the state and return the total. * It checks each stored activity date to ensure it is within the last 12 months and sums the `amount` for each IT purchase. {% hint style="warning" %} The `onUpdateActivity` function is triggered only when a new activity occurs. Therefore, you need to handle outdated activities within your function if the UserPoint does not receive a new event to update the State. {% endhint %} ## **Step 3: Test your plugin** Before deploying the plugin, create tests to ensure the function behaves as expected. Test cases might include: * Adding new activities and verifying the state is updated correctly. * Checking that old activities are purged. * Validating that the correct basket amounts are returned in the result. ## **Step 4: Declare your Computed Field in the schema** Once the computed field is implemented, you need to declare it in your schema as follows: ```graphql type UserPoint { id: ID! accounts: [UserAccount] … IT_amount_for_the_last_12_months: Int! @ComputedField(technical_name = “IT_Amount”) @TreeIndex(index:"USER_INDEX") } ``` This will link the computed field (`IT_amount_for_the_last_12_months`) to your schema, making it available for querying. *** ## **Handling Profile and Activity Information** When working with **profile** and **activity** information: Be mindful that the **computed field** needs to be **commutative** during the initial loading. For cases where UserProfile and UserActivity need to be aggregated, it might be necessary to store more information in the **State** to ensure consistency and accuracy during this phase. **Example use-case: Basket amount by fidelity card** During initial loading, if you need to track the basket amount for **current fidelity cards**, you may need to store all activities grouped by fidelity card in the state. This is because, until the initial loading completes, you may not know which fidelity cards are currently active for the user. # Data ingestion # Real time user tracking Online tracking aims at giving the ability to track unique users across digital properties. We support the following integrations in real time: * [Website tracking](/data-ingestion/real-time-user-tracking/web-events) * [Application tracking](/data-ingestion/real-time-user-tracking/application-tracking) * [API tracking](/data-ingestion/real-time-user-tracking/api) * [Conversion tracking](/data-ingestion/real-time-user-tracking/conversions) * [AMP tracking](/data-ingestion/real-time-user-tracking/amp) * [Email tracking](/data-ingestion/real-time-user-tracking/email-views-and-clicks) * [Display campaign tracking](/data-ingestion/real-time-user-tracking/ads-exposure) {% hint style="success" %} Use this feature if you want to track in real time what your users are doing on your digital touch points. {% endhint %} {% hint style="danger" %} If your need is to mass import activities, users or CRM profiles, you should consider [Bulk Imports](/data-ingestion/bulk-processing). {% endhint %} ## User activities and events With real-time user tracking, **you send user events** to mediarithmics that are **aggregated and encapsulated into** [**user activities**](/user-points/user-activities). Some user activities only have one user event, while others can have multiple events. {% content-ref url="/pages/-MNSuS1g6Npcklh14qQu" %} [UserActivity & UserEvent](/user-points/user-activities) {% endcontent-ref %} For example, when you send hits from the JS Tag on a web site, mediarithmics aggregates all the events into sessions, creating one user activity per session. ![](/files/-MPDyxIKz1SJnxObE-dL) ## Session aggregation If registering a visit as `IN_SESSION`, the visit will go through the **session aggregation** step. We aggregate visits in sessions based on the provided identifiers. We close sessions after 30 minutes of inactivity or if a new event is recorded with a referrer from a different domain than the previously recorded. Visits registered as `LIVE` don't create sessions. ## The processing pipeline When using the real-time tracking capability of mediarithmics, you enter what is called the **User Activity Processing Pipeline**. It allows mediarithmics to do some processing on your behalf. ![](/files/-MPE4kxzJamGadf4sftT) Here are the steps of processing, in order : 1. At session closing or every minute if events are `live`, creation of a user activity. 2. Execution of [**event rules**](/data-ingestion/real-time-user-tracking/event-rules). They allow you to edit the events in the activity and add new ones based on their shapes. 3. Application of the [**activity analyzer.**](/data-ingestion/real-time-user-tracking/activity-analyzers) It is a plugin allowing you to execute code to transform each activity. 4. Detection of **query-based conversions**. Conversions relying on pixels skip this step. 5. Evaluation of **automations triggers "React to an event"**. If the ongoing activity matches the trigger, the UserPoint enters the scenario. 6. [**UserChoice**](/advanced-usages/data-privacy-compliance/user-choices) management ensures the data you ingest is compatible with GDPR and other regulations. The activity is finally stored. # Website tracking The mediarithmics [JS tag](https://developer.mediarithmics.io/resources/tools-and-libraries/js-tag) is used to track the visitor's navigation on a website. It is imported in a JavaScript snippet that needs to be executed on all the pages you wish to track. {% hint style="info" %} The two common ways to integrate the tag are: * Through a universal tag container like Google Tag Manager or another similar solution. The code snippet must be copied once in a new tag and the tag container service automatically inserts the code wherever it is needed. * By inserting the code snippet in a general purpose template (ex: header template for an e-shop) which is used in all the web site pages. {% endhint %} The code of the Visit Tracking snippet is non-blocking—it does not impact on the page rendering time. The snippet can be inserted in the `` part of the web page. You can use the snippet to track in real-time, what your users are doing on your website. If you want to track users from your backend, please consider using our API. If you want to import bulk events or activities, please consider using the [Bulk Import](/data-ingestion/bulk-processing) feature. {% hint style="danger" %} Cookies used by the tag are considered as advertising cookies. **You must obtain user consent before using it.** {% endhint %} ## Implementing the snippet The mediarithmics tracking snippet is made of two parts: 1. A technical **tag** which contains JavaScript code to asynchronously load the TAG in the page. **This part should not be edited**, except when customizing the TAG name (see below). 2. The **configuration** that you should fill according to your context (site token / event name / event properties / etc.) Here is an example of the tracking snippet you should implement on every page. ```javascript ``` ### Changing the name To implement multiple tags or customize the tag for your own needs, you can do the following. Here is an example of a tag called **umbrella\_corp**. ```javascript ``` ### Changing the domain name If you want to get your own domain name, to fully remove mediarithmics from your website, contact your technical support to get your own domain name. Here is a an example of the full transparent snippet implementation. ```javascript ``` ### Pushing on events You don't have to push data on page load. You can, for example, bind an event to a button-click. ```javascript ``` ### Passing one or multiple user identifiers If you want to pass one or multiple identifiers for your user, you can leverage the `addIdentifier(type: string, identifier: object)` method for each identifier you want to pass Hereafter are the formats: * If `type == "USER_ACCOUNT"` then identifier object must have the following structure `{$user_account_id: string}`or `{$user_account_id: string, $compartment_token: string}` * If `type == "USER_EMAIL"` then identifier object must have the following structure `{$email_hash: string}`or `{$email_hash: string, $email: string}` * If `type == "USER_AGENT"` then identifier object must have the following structure `{$user_agent_id: string}` ```javascript mics.addIdentifier( "USER_ACCOUNT", { $user_account_id:"account 1", $compartment_token:"token1" } ); mics.addIdentifier( "USER_ACCOUNT", { $user_account_id:"account 2", $compartment_token:"token2" }); ); mics.addIdentifier( "USER_EMAIL", { $email_hash:"email hash", $email:"email address" } ); mics.addIdentifier( "USER_AGENT", { $user_agent_id:"user agent id" } ); ``` {% hint style="info" %} * `$email` is optional * `$compartment_token` is optional, if omitted, the datamart default compartment will be used * $user\_agent\_id is a [UserAgentId](https://developer.mediarithmics.io/user-points/user-identifiers/device-identifiers#user_agent_id). Agent type accepted are mobile advertising id, mobile vendor id, tv advertising id, and custom id. * You should not add network device id here as they are managed from the UI. If the desired network device id is not available please contact support. {% endhint %} {% hint style="warning" %} If invalid arguments are passed to `addIdentifier`, no error is thrown in the browser console {% endhint %} {% hint style="info" %} The legacy method will soon be removed : ```javascript mics.addProperty("$user_account_id", "" ) mics.addProperty("$comp_token", "" ); ``` We advise to use the new one as soon as possible {% endhint %} ## Sending the right events and properties The main purpose of the job is to send the right events with the right properties. This ensures good usability of the data for the users. {% hint style="info" %} NOTE: Sending too many properties creates confusion when the users need to decide which properties to query. Only send what you know is required. {% endhint %} ### Default tracking We provide out-of-the-box tracking that will generate a `$page_view` event. This is the behavior if you are using our snippet as is. ```javascript mics.pushDefault(); ``` {% hint style="warning" %} `$page_view` events **will be dropped when the session closes** (and disappear from the monitoring timeline)**,** so it will only be useful to test that events are correctly sent to mediarithmics. {% endhint %} ### Custom events If you need to trigger a custom event on a particular page, you can use the `mics.push` function to specify an event name and an object containing the event properties. In this example, we send the event named `view my form` with the property `form subject` set to `auto trial` : ```javascript mics.push("view my form", { "form subject": "auto trial" }); ``` {% hint style="warning" %} Refer to the list of base event names and prefer using them to custom event names, as the platform will handle some automatic processing for them. {% endhint %} ### E-commerce and products events It is recommended to implement the mediarithmics snippet at several key places, with specific event names: * Home page `$home_view` * Category or search results page `$item_list_view` * Product page `$item_view` * Basket page `$basket_view` * Transaction confirmation page `$transaction_confirmed` A list of products should be associated with any event with the [reserved property](/user-points/user-activities#predefined-event-properties) `$items`. {% hint style="info" %} If you have a unique identifier for the user, don't forget to add the `$user_account_id` property as described in this article for an optimal tracking. {% endhint %} #### Home page ```javascript // Add the user account id if available mics.addIdentifier( "USER_ACCOUNT", { $user_account_id:"", $compartment_token:"" } ); // Push the event mics.push("$home_view"); ``` #### Category or search results page When the user views a list of products in a category page or in a search results page. Only the first 3 items in the list should be declared. ```javascript // Add the user account id if available mics.addIdentifier( "USER_ACCOUNT", { $user_account_id:"", $compartment_token:"" } ); // Push the event mics.push("$item_list_view", { "$items": [ {"$id": "78798978"}, {"$id": "444444"}, {"$id": "78808900"} ] }); ``` * The $id field corresponds to the product id. #### Product page 
// Add the user account id if available
mics.addIdentifier(
    "USER_ACCOUNT",
    {
        $user_account_id:"<USER_ACCOUNT_ID>",
        $compartment_token:"<COMPARTMENT_TOKEN>"
    }
);

// Push the event
mics.push("$item_view", {
  "$items": [
    { "$id": "89999999" }
  ]
});
* The $id field corresponds to the product id. #### Basket page ```javascript // Add the user account id if available mics.addIdentifier( "USER_ACCOUNT", { $user_account_id:"", $compartment_token:"" } ); // Push the event mics.push("$basket_view", { "$items" : [ {"$id" : "78794", "$price" : 10.8, "$qty" : 1 }, {"$id" : "78677", "$price" : 56.99, "$qty" : 1 } ], "$currency" : "EUR" }); ``` * The $id field corresponds to the product id. * The $price field should contain the product price without the currency * The $qty field should contain the quantity of this product in the basket * The $currency field is optional. If there is only one catalog, the catalog currency is used by default #### Transaction confirmation page ```javascript // Add the user account id if available mics.addIdentifier( "USER_ACCOUNT", { $user_account_id:"", $compartment_token:"" } ); // Push the event mics.push("$transaction_confirmed", { "$items" : [ {"$id" : "78794", "$price" : 10.8, "$qty" : 1 }, {"$id" : "78677", "$price" : 56.99, "$qty" : 1 } ], "$transaction_id" : "transact-XYZ", "$currency" : "EUR" }); ``` * The $id field corresponds to the product id. It should be the same id as the one used in the product feed * The $price field should contain the product price without the currency * The $qty field should contain the quantity of this product in the basket * The $transaction\_id field should contain the ID of the transaction * The $currency field is optional. If there is only one catalog, the catalog currency is used by default {% hint style="info" %} You can add any custom properties available for each product in your page JavaScript. You can also use Activity Analyzers to enrich your data later using the product IDs. {% endhint %} ### UserProfile updates If the user is identified, you can register updates to its profile from the JS Snippet. It is generally used in user profile pages of your site. ```javascript // Add the user account id if available mics.addIdentifier( "USER_ACCOUNT", { $user_account_id:"", $compartment_token:"" } ); // Push the profile mics.push("$set_user_profile_properties", { gender: "male", // add any other properties you whish }); ``` Use a special event called `$set_user_profile_properties` . The properties and values associated with the event will be written as key-value pairs on the `UserProfile` . By default, the anonymous `UserProfile` associated with the default compartment of the datamart will be updated. By default, the property `force_replace` is set to false when sending an event `$set_user_profile_properties` . The following special properties are available to specify which `UserProfile` should be updated: | Property | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | $set\_user\_profile\_comp\_token | (Optional) The compartment token representing the compartment in which the profile should be written. If not provided, this will be the default compartment. | | $set\_user\_profile\_user\_account\_id | (Optional) The user account id under which the profile should be written. If not provided, it is the anonymous profile. | *Example 1*: Write the gender to the anonymous `UserProfile` of the default compartment. ```javascript umbrella_corp.push( '$set_user_profile_properties', { 'gender': 'female' } ); ``` *Example 2*: Write the gender to the anonymous `UserProfile` of a given compartment. ```javascript umbrella_corp.push( '$set_user_profile_properties', { '$set_user_profile_comp_token': 'my_compartment_token', 'gender': 'female' } ); ``` *Example 3*: Write the gender to the `UserProfile` identified by a User Account Id of a given compartment. ```javascript umbrella_corp.push( '$set_user_profile_properties', { '$set_user_profile_comp_token': 'my_compartment_token', '$set_user_profile_user_account_id': '456', 'gender': 'female' } ); ``` ## Server-side configuration of the JavaScript tag Some behaviors for the JavaScript can be set server-side, in the same way as tag managers. These configurations can be found on the related channel in Navigator settings. ### Device Identification Select the identifiers that will automatically be used by the JavaScript tag to identify devices. The possible options include: * mediarithmics first-party cookie * mediarithmics third-party cookie (vector ID) In addition, mediarithmics tag can automatically retrieve a user or device identifier from the following partners if you have previously integrated with them: * ID5 * First ID * Utiq martechpass (mobile) #### mediarithmics first-party cookies This device identification method, also referred to as **INSTALLATION\_ID**, can only be used for tracking on the specific website where it is dropped , and to a limited extent for first-party activation (acting as a PPID). As a result, it **does not enable cross-site reconciliation or cross-site targeting**. {% hint style="info" %} Note that: * This device identification method is only available on the **SITE** channel type. * It does not provide full browser coverage, as some browsers (such as Safari) enforce restrictions on first-party cookies set via JavaScript originating from third-party sources (such as our JS tag). * This mechanism is different from the replication of the mediarithmics third-party cookie onto the first-party domain (**mics\_vid**), which occurs systematically for caching purposes (see [Cookies](/advanced-usages/data-privacy-compliance/cookies#cookies-installed-on-the-first-party-domain)). {% endhint %} When you activate *mediarithmics first-party cookie*, a first-party cookie named `mics_` OR `mics_` will be dropped in the user’s browser. Refer to [UserDeviceTechnicalId](/user-points/user-identifiers/userdevicetechnicalid#installation-id-first-party-cookie-generated-by-mediarithmics) to learn more about the cookie value format. ### IAB Transparency and Consent Framework Enable this setting if you want the javascript tag to automatically consider any TCF-compliant CMP on the website before third-party cookie creation and cookie matchings. ### Cookie matching with Google and Xandr Select if you want the javascript tag to trigger cookie matchings with Google and Xandr. For other partners, please check with your support. ### Support of mediarithmics third-party cookie (vector ID) By default, mediarithmics third-party cookie (vector ID) is not generated if the browser does not support matchings with Google and Xandr.\ Use this setting to force its generation even if these matchigns do not succeed.
## JavaScript Tag API Reference {% hint style="success" %} You can check the complete reference of the JS Tag tool [here](/resources/tools-and-libraries/js-tag). {% endhint %} ## One Tag approach A One Tag approach in web tracking refers to a method of tracking website visitors and their behavior on a website using a single tracking code. The advantage of a one tag approach is that it is relatively simple to implement and maintain, as it only requires the use of a single tracking code on the website. {% hint style="info" %} It usually implies to use an activity analyzer to process the collected data on the server side. {% endhint %} The mediarithmics javascript tag can easily be used in a "one tag" approach by referencing the relevant variables in the browser Document Object Model. ### Examples: Here are several examples of a One Tag implementation with different tag management services. * By default, the mediarithmics tag already collects the url, the referrer, the user agent and the time. It is not necessary to provide these properties: ```html ``` * For Google Tag Manager, it is possible to collect all the relevant properties in the data layer: ```html ``` * For Commanders Act, all the properties are stored in the tc\_vars variable: ```html ``` # Application tracking ## Setup summary In order to track activities on your application, you'll need to keep in mind the following : * All user activities need to be sent to mediarithmics using the [**Tracking API**](/data-ingestion/real-time-user-tracking/api) * The [**Signature authentication**](https://developer.mediarithmics.io/resources/api-overview/authentication#signature-authentication) should be used to authenticate any requests between your application and mediarithmics. Don't hesitate to contact your Account manager to have more information about this. * The activity and events in the payload need to comply with the specificities described below * The user agent identifier ($user\_agent\_id) needs to be particularly formatted using guidelines described [here](/user-points/user-identifiers/userdevicetechnicalid#user_agent_id) ## Application integration options The first option is to integrate a small piece of code (approx. 100 lines) into the application to execute calls to the mediarithmics tracking API. **Sample code for iOS and Android is available in the** [**Authentication**](/resources/api-overview/authentication#examples) **section** (see illustration below).
The second option is to re-use an already existing analytics tool. It is then possible to transfer events from the analytics solution's server to the mediarithmics API.
## Predefined event names In the context of app tracking, predefined event names are available out-of-the-box to simplify and automate event processing ([full list of predefined event names](/user-points/user-activities#predefined-event-names)): * app open event (`$app_open`) corresponds to the opening of the app and app resume event (when the app becomes active again) * app install event (`$app_install`) * app update event (`$app_update`) The install and update events are automatically calculated on server side and you don't have to send them : * The install event (`$app_install`) is triggered the first time an app open event is received for a user, regardless if the user is new or existing. * The update event (`$app_update`) is triggered when the SDK version, app version or OS version changes from one open to the next. ## Mobile application tracking - UserActivity formatting In the context of mobile app tracking, you should adapt some fields of the UserActivity. Here are the fields to look out for : | Field | Type | Description | | ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $type | String enum | The activity type should only be `APP_VISIT` | | $session\_status | String enum |

The sessions status should be:
IN\_SESSION: This value should be used if you’re making one API call per tracked event (recommended way). The platform will automatically aggregate all the events of a session sent through many API calls when this value is used.
CLOSED\_SESSION: should be used only if you do a single API call per session at its end. In this case, you should provide ALL the events of the session in the $events array in the call. Each API call with this value will generate a new User Activity in the Platform.

| | $user\_agent\_id | String (Optional) | The [user agent identifier](/user-points/user-identifiers/userdevicetechnicalid#user_agent_id-formatting-mobile-application-identifiers) of the user device containing a unified representation of an identifier | | $app\_id | String | The mobile app id (previously created through Navigator / API) | ## CTV application tracking - UserActivity formatting In the context of CTV app tracking, you should adapt some fields of the UserActivity. Here are the fields to look out for : | Field | Type | Description | | ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $type | String enum | The activity type should only be `CTV_VISIT` | | $session\_status | String enum |

The sessions status should be:
IN\_SESSION: This value should be used if you’re making one API call per tracked event (recommended way). The platform will automatically aggregate all the events of a session sent through many API calls when this value is used.
CLOSED\_SESSION: should be used only if you do a single API call per session at its end. In this case, you should provide ALL the events of the session in the $events array in the call. Each API call with this value will generate a new User Activity in the Platform.

| | $user\_agent\_id | String (Optional) | The [user agent identifier](/user-points/user-identifiers/userdevicetechnicalid#user_agent_id-formatting-ctv-identifiers) of the user device containing a unified representation of an identifier. | | $ctv\_id | String | The ctv app id (previously created through Navigator / API) | {% hint style="info" %} For CTV application tracking, we allow automatic capture of IP addresses.[ Learn more](/user-points/user-identifiers/networks-ids/device-based-network-ids/ip-address)
{% endhint %}
# Ads exposure tracking You can track the exposition of users to the different kind of Ads format that exist today: * **Display Ads**: All Ads that are either static image/animations or HTML5 animated Ads * **Video Ads**: All Video Ads that the User can see through a marketing campaign All Ads tracking is done using **Pixels** and **Click Tracking URLs**. {% hint style="info" %} You should consider using this feature to get ad view and ad click directly within your datamart when using DSPs . {% endhint %} {% hint style="info" %} Activities tracked through this method will be of `DISPLAY_AD` *$type* and `NO_SESSION` *$session\_status*. See [User activity object](/user-points/user-activities#user-activity-object) for more information. {% endhint %} ## Associated user events There are two predefined [user events](/user-points/user-activities#predefined-event-names) that should be tracked during the exposition of a user to an Ad : | Event name | Tracking method | Description | | ---------- | ------------------ | --------------------------------------------------- | | $ad\_view | Pixel | The 'view/impression' of the Display Ad to the User | | $ad\_click | Click Tracking URL | The 'click' of the Ad by the User | ## Display Ad Pixel Use the following URL in your tracking pixel to send an `$ad_view` event to the platform. ```markup https://events.mediarithmics.com/v1/touches/pixel? \ $ev=$ad_view& \ $dat_token=& $catn=& $scatn=& $crtn=& $cb=& gdpr=& gdpr_consent= ... any custom property ``` | Field | Type | Description | | ------------------------ | ------------------------- | ---------------------------------------------------------------------------------------------- | | $ev | String | The event name. **$ad\_view for Display Ad impression tracking** | | $dat\_token | String | The **token** (not the ID) of the datamart in the mediarithmics platform. | | $catn | String | Campaign technical name | | $scatn | String | Sub-campaign technical name | | $crtn | String | Creative technical name | | $cb | String | The cache buster parameter. It should contain a random string. optional | | $cuid | String | User account [identifier](/user-points#user-identifiers) of the user | | $uaid | String | Mobile identifier of the user to identify the [User agent](/user-points#user-agents) | | $email\_hash | String | Email Hash [identifier](/user-points#user-identifiers) of the user | | $comp\_token | String | Compartment token (not the ID) | | $uids | JSON as string (Optional) | The list of [user identifiers](#passing-user-identifiers-in-pixel-based-tracking) of the user. | | gdpr | Number | TCF v2.2 parameter to indicate if gdpr applies or not (values: 1 or 0) | | gdpr\_consent | String | TCF v2.2 parameter containing the encoded consent string | | any custom property name | Any Type | Any custom property. optional | {% hint style="info" %} Please note that before declaring a custom property, we recommend that you inform our PS team so that it can be indexed to your data schema (and therefore used for segmentation and/or dataviz), see our [data model documentation](https://developer.mediarithmics.io/schema). {% endhint %} ## Ad Click tracking URL Use the following click-tracking URL to send an `$ad_click` event to the platform. ```markup https://events.mediarithmics.com/v1/touches/click? $ev=$ad_click& $dat_token=& $catn=& $scatn=& $crtn=& $cb=& $redirect=& gdpr=& gdpr_consent= ... any custom property ``` | Field | Type | Description | | ------------------------ | ------------------------- | ---------------------------------------------------------------------------------------------- | | $ev | String | The event name. **$ad\_click for Display Ad click tracking** | | $dat\_token | String | The id of the audience datamart in the mediarithmics platform. | | $redirect | String | The redirect url. This string should be URL Encoded. (RFC 3986) | | $catn | String | Campaign technical name | | $scatn | String | Sub-campaign technical name | | $crtn | String | Creative technical name | | $cb | String | The cache buster parameter. It should contain a random string. optional | | $cuid | String | User account ID [identifier](/user-points#user-identifiers) of the user | | $email\_hash | String | Email Hash [identifier](/user-points#user-identifiers) of the user | | $comp\_token | String | Compartment token (not the ID) | | $uids | JSON as string (Optional) | The list of [user identifiers](#passing-user-identifiers-in-pixel-based-tracking) of the user. | | gdpr | Number | TCF v2.2 parameter to indicate if gdpr applies or not (values: 1 or 0) | | gdpr\_consent | String | TCF v2.2 parameter containing the encoded consent string | | any custom property name | Any Type | Any custom property. optional | {% hint style="info" %} If you want to integrate with an external DSP / Ad Server such as Google Ad Manager or App Nexus, you can use the ids of the creative ad groups and campaign from your external DSP as technical names. See the integrations below for examples. {% endhint %} ## Campaign tracking on Google ### Display Video 360 *You can use the following macros as a minimum configuration for tracking on DV 360 (ex-Doubleclick Bid Manager):* ```markup https://events.mediarithmics.com/v1/touches/pixel? $ev=$ad_view& $dat_token=& $catn=${CAMPAIGN_ID}& $scatn=${INSERTION_ORDER_ID}& $crtn=${CREATIVE_ID}& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $cb=${CACHEBUSTER} ``` ```markup https://events.mediarithmics.com/v1/touches/click? $ev=$ad_click& $dat_token=& $catn=${CAMPAIGN_ID}& $scatn=${INSERTION_ORDER_ID}& $crtn=${CREATIVE_ID}& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $cb=${CACHEBUSTER}& $redirect=${CLICK_URL_ENC} ``` {% hint style="info" %} More info on DV360 macros available here: {% endhint %} ### Campaign Manager 360 integration *You can use the following macros as a minimum configuration for tracking on Campaign Manager 360 (ex-Doubleclick* Campaign Manager*):* ```markup https://events.mediarithmics.com/v1/touches/pixel? $ev=$ad_view& $dat_token=& $catn=%ebuy!& $scatn=%eaid!& $crtn=%ecid!& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $cb=%n ``` ```markup https://events.mediarithmics.com/v1/touches/click? $ev=$ad_click& $dat_token=& $catn=%ebuy!& $scatn=%eaid!& $crtn=%ecid!& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $cb=%n& $redirect= ``` {% hint style="info" %} More info on Google Campaign Manager 360 macros can be found here: And for TCF integration: {% endhint %} ### Ad Manager integration *You can use the following macros as a minimum configuration for tracking on Ad Manager:* ```markup https://events.mediarithmics.com/v1/touches/pixel? $ev=$ad_view& $dat_token=& $catn=%ebuy!& $scatn=%eaid!& $crtn=%ecid!& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $cb=%%CACHEBUSTER%% ``` ```markup https://events.mediarithmics.com/v1/touches/click? $ev=$ad_click& $dat_token=& $catn=%ebuy!& $scatn=%eaid!& $crtn=%ecid!& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $cb=%%CACHEBUSTER%%& $redirect= ``` {% hint style="info" %} More info on Google Ad Manager macros can be found here: {% endhint %} ## Campaign tracking on Xandr *You can use the following macros as a minimum configuration for tracking on Xandr:* ```markup https://events.mediarithmics.com/v1/touches/pixel? $ev=$ad_view& $dat_token=& $catn=${CP_CODE}& $scatn=${CPG_CODE}& $crtn=${CREATIVE_CODE}& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $cb=${CACHEBUSTER} ``` ```markup https://events.mediarithmics.com/v1/touches/click? $ev=$ad_click& $dat_token=& $catn=${CP_CODE}& $scatn=${CPG_CODE}& $crtn=${CREATIVE_CODE}& $cb=${CACHEBUSTER}& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $redirect=${CLICK_URL_ENC} ``` {% hint style="info" %} More info on Xandr supported macros can be found here: {% endhint %} ## Campaign tracking on The Trade Desk *You can use the following macros as a minimum configuration for tracking on The Trade Desk:* ```markup https://events.mediarithmics.com/v1/touches/pixel? $ev=$ad_view& $dat_token=& $catn=%%TTD_CAMPAIGNID%%& $scatn=%%TTD_ADGROUPID%%& $crtn=%%TTD_CREATIVEID%%& $cb=%%TTD_CACHEBUSTER%%& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184} ``` ```markup https://events.mediarithmics.com/v1/touches/click? $ev=$ad_click& $dat_token=& $catn=%%TTD_CAMPAIGNID%%& $scatn=%%TTD_ADGROUPID%%& $crtn=%%TTD_CREATIVEID%%& $cb=%%TTD_CACHEBUSTER%%& gdpr=${GDPR}& gdpr_consent=${GDPR_CONSENT_184}& $redirect=%%TTD_CLK_ESC%% ``` {% hint style="info" %} More info on The Trade Desk supported macros can be found here (a The Trade Desk account is requiered to access this page): {% endhint %} You can track additional properties by using custom properties such as: ``` ... domain=%%TTD_SITE%%& device=%%TTD_DEVICETYPE%%& ... ``` {% hint style="info" %} Please note that before declaring a custom property, we recommend that you inform our PS team so that it can be indexed to your data schema (and therefore used for segmentation and/or dataviz), see our [data model documentation](https://developer.mediarithmics.io/schema). {% endhint %} ## Video Ad Tracking Basic video ad tracking can be achieved by integrating the display ad pixel (with `$ad_view` events) and the click-tracking URL (with `$ad_click` events) into your video ad format. For more advanced capabilities, a specific integration can be setup based on the visit pixel (with custom completion events) and an Activity Analyzer. Please advise with your Account Representatives during the design phase. ## Passing user identifiers in pixel-based tracking The following is true for all types of pixel-based tracking (events, ads, email, conversions, ...) which use the `events.mediarithmics.com/v1/touches/pixel` API endpoint You can pass one or more user identifiers when using the `$uids` field. ``` https://events.mediarithmics.com/v1/touches/pixel? \ $ev=$ad_view& \ $dat_token=& $catn=& $scatn=& $crtn=& gdpr=& gdpr_consent=& $cb=& $uids=jso-[{"$tpe":"AG","$agid":"vec:1234"}]& ... any custom property ``` {% hint style="info" %} `jso-` prefix means that the rest of the string will be interpreted as a json object. {% endhint %} {% hint style="warning" %} Don't forget to correctly encode the URL {% endhint %} ### User Account | Field | Type | Description | | ----- | --------------- | ------------------------------- | | $tpe | Constant String | AC | | $ctok | String | The token of the compartment | | $acid | String | The user account id of the user | ### User Email | Field | Type | Description | | ----- | ----------------- | --------------- | | $tpe | Constant String | EM | | $eh | String | The email hash | | $e | String (Optional) | The "raw" email | ### User Agent | Field | Type | Description | | ----- | --------------- | -------------------------------------------------------------------------------------------------------------------------- | | $tpe | Constant String | AG | | $agid | String | The [user agent id](/user-points/user-identifiers/userdevicetechnicalid#user_agent_id-formatting-cookie-based-identifiers) | {% hint style="warning" %} Use the registry token, not the id, when formatting the user agent id. Eg: `dev::` {% endhint %} # AMP tracking For the following integration to work, you must add the [amp-analytics module](https://amp.dev/documentation/components/amp-analytics) on the page environment. Here is an example of the AMP tag: ```markup ``` **Arguments** | name | type | description | | ------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------- | | site\_token | string | **site\_token** is the token of the website (**required**). | | event\_name | string | **event\_name** is the [name of the event](/user-points/user-activities#user-events-object) (optional, default to '$page\_view'). | | any custom property | string | All **custom properties** should be added to the extraUrlParams object (prop1, prop2 in the example). | # Conversions tracking To measure and optimize the performance (i.e., cost per action) of your marketing campaigns, you need to track user conversions driven by your Marketing Campaign. On the mediarithmics platform, you create a **Goal** to define a trigger **when a user should convert**. There are two ways of configuring a Goal: 1. **Creating a Goal and using the associated Pixel.** Each user that sees the Pixel in a page will convert for this goal. This Pixel should be present on the confirmation page or dynamically loaded when the user converts (by using a Tag Manager or by including the 'pixel load' in the website logic) 2. **Writing a rule** that will be run on the Datamart data. It will generate a user conversion each time a user matches the rule. From experience option #1 is the simplest solution to configure and should be used for each 'non-recurrent' User Conversion tracking**.** The option #2 offers more flexibility but with the cost of a longer & more complex configuration. It should be used either when the: * datamart tracking JS TAGs / Pixels are already present on the website, including the page where the conversion happens * Goal to track will be used repeatedly * Goal definition requires a complex rule that can't be achieved by displaying a JS TAG / Pixel ## Tracking User Conversions by using Pixels When creating a **goal** in the navigator (Campaigns tab > Goals sub-tab): 1. Define the name of the goal. 2. Select **Trigger via Pixel**. 3. Copy/paste the generated HTML code to your technical team so that they can integrate it. 4. Don't forget to associate the correct attribution model for the attribution of the conversion to your Marketing Campaigns. 5. Save the goal. ## Tracking User Conversions by using Datamart Queries When creating a **goal** in the navigator (**Campaigns** tab > **Goals** sub-tab) : 1. Define the name of the goal. 2. Select **Trigger via Query**. 3. Define the query that should be used to tell if a User has converted or not. [Please read the documentation about Queries here](https://developer.mediarithmics.com/pages/audience/query.html) 4. Don't forget to associate the correct attribution model for the attribution of the conversion to your Marketing Campaigns 5. Save the goal. If you have issues when defining the proper Query to define your Goals, please try Option #1 or reach out to your Account Manager. # Email views and clicks You can track **read** and **click** actions in emails sent to your customers. * To track the **opening** of the emails, you have to **include a pixel** * To track the **clicks** on the links included in your emails, you have to r**eplace each of your links with a Click Tracking URL** For each event, the datamart will **identify the users** reading/clicking the email by: * **Recommended option:** using the user's Email Hash, included by you in the pixel/tracking URL * Reading their cookies if possible: * It is always working when the users are clicking (e.g. Click Tracking URL) * It is sometimes possible when the users are opening the emails, depending on their email client (e.g. Pixel) {% hint style="info" %} Activities tracked through this method will be of `EMAIL` *$type* and `NO_SESSION` *$session\_status*. See [UserActivity object](/user-points/user-activities#user-activity-object) for more information. {% endhint %} ## Pixel Touch for Email Opening Calls to those URLs generate [events](/user-points/user-activities#user-events-object). ```markup https://events.mediarithmics.com/v1/touches/pixel? $ev=$email_view& $dat_token=& $email_hash=78b04074e616166938cf672f70f41b4d& Any Custom Properties ... ``` or if your are using the `$uids` field (see [Passing user identifiers in pixel-based tracking](/data-ingestion/real-time-user-tracking/ads-exposure#passing-user-identifiers-in-pixel-based-tracking)) ``` https://events.mediarithmics.com/v1/touches/pixel? $ev=$email_view& $dat_token=& $uids=jso-[{"$tpe":"EM","$eh":"78b04074e616166938cf672f70f41b4d"}]& Any Custom Properties ... ``` or if you are using a custom domain: ```markup https://analytics.custom-domain.com/v1/touches/pixel? $ev=$email_view& $dat_token=& $email_hash=78b04074e616166938cf672f70f41b4d& Any Custom Properties ... ``` It is possible to set custom properties | Field | Type | Description | | ------------------------ | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | $ev | String | The event name. **$email\_view for email opening tracking** | | $dat\_token | String | The id of the audience datamart in the mediarithmics platform. | | $cuid | String (Optional) | The user account id. | | $email | String (Optional) | The user email. | | $email\_hash | String (Optional) | The user email hash. | | $cb | String (Optional) | The cache buster parameter. It should contain a random string. | | $uids | JSON as string (Optional) | The list of [user identifiers](/data-ingestion/real-time-user-tracking/ads-exposure#passing-user-identifiers-in-pixel-based-tracking) of the user. | | any custom property name | Any Type | Any custom property | ## Click Tracking for Emails Calls to those URLs generate click [events](/user-points/user-activities#user-events-object) ```markup https://events.mediarithmics.com/v1/touches/click? $ev=$email_click& $dat_token=& $email_hash=78b04074e616166938cf672f70f41b4d& Any Custom Properties & $redirect= ``` \ It is also possible to set custom properties. \ The $redirect parameter is used to define the destination of the url redirection. The URL put in the $redirect parameter should be [URL Encoded (RFC 3986)](https://en.wikipedia.org/wiki/Percent-encoding). | Field | Type | Description | | ------------------------ | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $ev | String | The event name. **$email\_click for email opening tracking** | | $dat\_token | String | The id of the audience datamart in the mediarithmics platform. | | $redirect | String | The redirect url. This string should url encoded. (RFC 3986). **Warning**: this parameters must be placed at the end of the URL. *Any parameters that will be placed after the $redirect parameters will not be saved.* | | $cuid | String (Optional) | The user account id. | | $email | String (Optional) | The user email. | | $email\_hash | String (Optional) | The user email hash. | | $cb | String (Optional) | The cache buster parameter. It should contain a random string. | | $uids | JSON as string (Optional) | The list of [user identifiers](/data-ingestion/real-time-user-tracking/ads-exposure#passing-user-identifiers-in-pixel-based-tracking) of the user. | | any custom property name | Any Type (Optional) | Any custom property. | # Tracking API {% hint style="warning" %} Beware of the [authentication method](/resources/api-overview/authentication) you choose, so that it is adapted to the context running API calls. {% endhint %} ## UserActivity API ### General information You can import [UserActivit](/user-points/user-activities)y using our dedicated API endpoint. ## Import a UserActivity `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_activities` The body of the request must be a UserActivity object. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ------------------------------------------------------------------- | | datamartId | integer | The ID of the datamart in which the UserActivity should be imported | #### Headers | Name | Type | Description | | ------------ | ------ | ---------------- | | Content-Type | string | application/json | #### Request Body | Name | Type | Description | | ---- | ------ | --------------------------------- | | body | object | The UserActivity object to import | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} The body must be a valid [UserActivity](/user-points/user-activities) object. Identification of the user or of the device is achieved through the `$user_identifiers` property. We encourage you to use as many identifiers as available in your environment at the time of the capture. ```json { "$ts" : 3489009384393, "$type" : "APP_VISIT", "$session_status" : "IN_SESSION", "$user_identifiers" : [{ "$type": "USER_ACCOUNT", "$compartment_id" : "", "$user_account_id" : "" }, { "$type": "USER_ACCOUNT", "$compartment_id" : "", "$user_account_id" : "" }, { "$type": "USER_AGENT", "$user_agent_id" : "" }, { "$type": "USER_EMAIL", "$hash" : "", "$email" : "" }], "$app_id" : "1023", "$events" : [ { "$ts" : 3489009384393, "$event_name" : "$app_open", "$properties" : {} }] } ``` {% hint style="info" %} It is still possible to use the identifiers properties at the activity root level (`$user_agent_id`, `$user_account_id` + `$compartment_id`, `$email_hash)`, however these are to be considered as legacy. `$user_identifiers` property is to be preferred. {% endhint %} 
{ 
	"$ts" : 3489009384393,
	"$type" : "APP_VISIT",
	"$session_status" : "IN_SESSION",
	"$user_agent_id" : "<USER_AGENT_ID>",
	"$compartment_id" : "<COMPARTMENT_ID>",
	"$user_account_id" : "<ACCOUNT_ID>",
	"$app_id" : "1023",
	"$events" : [
	{
		"$ts" : 3489009384393,
		"$event_name" : "$app_open",
		"$properties" : {}
	}]
}
### Create / Update a UserProfile from within an activity A UserProfile can be created / updated by registering a UserActivity containing a `$set_user_profile_properties` event. In that case you would need to use `$user_account_id` and `$compartment_id` inside `$properties` to identify the UserProfile to update: ```json { "$ts" : 3489009384393, "$type" : "APP_VISIT", "$session_status" : "IN_SESSION", "$user_identifiers" : [{ "$type": "USER_ACCOUNT", "$compartment_id" : "", "$user_account_id" : "" }], "$app_id" : "1023", "$events" :[{ "$ts" : 1679588413000, "$event_name" : "$set_user_profile_properties", "$properties" : { "$compartment_id" : "", "$user_account_id" : "", "gender" : "Male", "zipcode" : "78000" } }] } ``` {% hint style="info" %} Note that you can used any identifier available on the UserPoint in the UserActivity object {% endhint %} ### Create / Update a UserChoice from within an activity A UserChoice can be created / updated by registering a UserActivity containing a `$set_user_choice` event. {% hint style="success" %} This method is used when you want to achieve real-time tracking but can't use the mediarithmics JavaScript Tag. In mobile applications for example. Please note that those events will go through the [processing pipeline](/data-ingestion/real-time-user-tracking#the-processing-pipeline) before being stored as a UserChoice. You must ensure no [activity analyzers](/data-ingestion/real-time-user-tracking/activity-analyzers) is removing them during that process. {% endhint %} ```javascript // Sample UserActivity to add using the tracking API { "$user_account_id":"", "$compartment_id":, "$type":"", "$session_status":"NO_SESSION", "$ts":, "$events": [ { "$event_name":"$set_user_choice", "$ts":, "$properties":{ "$processing_id": "", // Mandatory "$choice_acceptance_value":, // Mandatory "" : "" } } ] } ``` ## UserPoint API ### UserPoint selector The attribute `:userPointSelector` is used to select the UserPoint on which apply the query. You can provide the following values : ``` // Select a UserPoint using a user_point_id /v1/datamarts//user_points//user_profiles/compartment_id=:compartmentId/user_account_id=:userAccountId /v1/datamarts//user_points/user_point_id=/user_profiles/compartment_id=:compartmentId/user_account_id=:userAccountId // Select a UserPoint using a user_agent_id /v1/datamarts//user_points/user_agent_id=/user_profiles/compartment_id=:compartmentId/user_account_id=:userAccountId // Select a UserPoint using a user_account_id + compartment_id /v1/datamarts//user_points/compartment_id=,user_account_id=/user_profiles/compartment_id=:compartmentId/user_account_id=:userAccountId // Select a UserPoint using an email_hash /v1/datamarts//user_points/email_hash=/user_profiles/compartment_id=:compartmentId/user_account_id=:userAccountId ``` ### Create / Update a UserProfile An other way to create / update a UserProfile is to use the *user\_profiles* API endpoint . Prefer this method if you are able to integrate various API endpoints and if you don't need to track the UserProfile update as an event for further retrieval. ## Create/Update a UserProfile `PUT` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_points/:userPointSelector/user_profiles/compartment_id=:compartmentId/user_account_id=:userAccount` The body of the request must be a UserProfile object. #### Path Parameters | Name | Type | Description | | ------------- | ------- | ----------------------------------------------------------------------------------------------------------------------- | | userAccount | string | The user\_account\_id linked to the user\_profile that should be imported | | compartmentId | integer | The ID of the compartment in which the UserProfile should be imported | | datamartId | integer | The ID of the datamart in which the UserProfile should be imported | | userSelector | string |

The identifier of the user for whom the UserProfile should be imported.
see the options of the user selector.

| #### Query Parameters | Name | Type | Description | | ---------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | update\_strategy | Enum (Optional) |

Values are PARTIAL\_UPDATE, PARTIAL\_DELETE, FORCE\_REPLACE
(Detailed examples)

| ***Legacy parameters (use update\_strategy instead)*** | Name | Type | Description | | ---------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *force\_replace* | *boolean (optional)* |

If true, then the UserProfile will be completely replaced by the object passed in the user\_profile field.
If false, the object passed in the user\_profile field will be merged with the existing UserProfile of the UserPoint.

| | *merge\_objects* | *boolean (optional)* |

Only considered if force\_replace is false.


Manage the comportement between two objects with a same property.


If false (default value), the new object overrides the existing one.

If true the new object is merged in deep to the existing one (see example).

| #### Headers | Name | Type | Description | | ------------ | ------ | ---------------- | | Content-Type | string | application/json | #### Request Body | Name | Type | Description | | ---- | ------ | -------------------------------- | | body | object | The UserProfile object to import | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} The body must be a valid [UserProfile](https://app.gitbook.com/@mediarithmics/s/developer-guide/~/drafts/-MiLHGsNDWXB4IJ4_i8w/getting-started/user-profiles) object. ```javascript { "$compartment_id" : ":compartment_id", "$user_account_id" : ":user_account_id", "gender" : "female", "zipcode" : "75001" } ``` {% hint style="info" %} `$compartment_id` & `$user_account_id` in the payload are not mandatory since they are already provided as query parameters. {% endhint %} {% hint style="warning" %} Beware of : * `` & `` which are used to select the UserPoint * `:compartmentId` & `:userAccountId` which are used to select the profile to update {% endhint %} ### Create / Update a UserChoice An other way to create / update a UserChoice is to use the *user\_choices* API endpoint . Prefer this method if you are able to integrate various API endpoints. ## Create/Update UserChoice `PUT` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_points/:userSelector/user_choices/processing_id=:processingId` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ------------------------------------------------------------------------ | | datamartId | integer | The datamart ID | | userSelector | integer | An identifier to the UserPoint for which the UserChoice should be added. | | processingId | integer | The ID of the associated processing | #### Request Body | Name | Type | Description | | ---- | ------ | ----------- | | Body | object | The payload | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ```json // Sample payload { "$choice_ts": "", // Mandatory "$choice_acceptance_value":, // Mandatory "" : " // Optional } ``` ## List UserChoice `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_points/:userSelector/user_choices/processing_id=:processingId` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------------------------------------------------------------- | | datamartId | integer | The datamart ID | | userSelector | integer | An identifier to the UserPoint for which the UserChoice should be added. | | processingId | integer | *Optional*. The ID of the processing for which you want to list UserChoices. | #### Request Body | Name | Type | Description | | ---- | ------ | ----------- | | Body | object | The payload | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## UserChoice history `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_points/:userSelector/user_choices/processing_id=:processingId/change_log` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ------------------------------------------------------------------------ | | datamartId | integer | The datamart ID | | userSelector | integer | An identifier to the UserPoint for which the UserChoice should be added. | | processingId | integer | The ID of the processing for which you want to get UserChoice history. | #### Request Body | Name | Type | Description | | ---- | ------ | ----------- | | Body | object | The payload | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} # Event rules **Every time an event is sent, we run the event rules associated with its channel**. They are **predefined actions** to extract properties, reshape data or identify a user from a property. {% hint style="success" %} If the action you wish to do on each event is not possible with an event rule, you should have a look at [Activity Analyzers](/data-ingestion/real-time-user-tracking/activity-analyzers). {% endhint %} ## How to add event rules You can go to a **channel's settings, edit, and then scroll down to event rules.** ![](/files/-MPiSHy2PHO3YYQduz-D) ## Available event rules ### Property To Origin Copy This Event Rule will help you take an existing property from an event and copy it into the origin of the activity. You can copy: * The URL of the Event * The Referrer of the Event * Any Event Property ### Url Match This Event Rule allows you to write a **pattern** that will be **matched against all the $url values of incoming $page\_view events**. The pattern can extract some values from the Url Path and from the Url Query String. *If you want more information about the different parts of an URL,* [*please read this article.*](https://en.wikipedia.org/wiki/URL) #### Limitations Url Match only works for $page\_view events. Any other event won't be processed by the Url Match event rule. {% hint style="warning" %} Please note that $page\_view events will be deleted at the end of the activity processing stage. {% endhint %} #### Use case Useful if for technical and/or organizational reasons, you can't customize the Tracking JS Snippet on a web page to retrieve information from the data layer. 1. Use the [default JS Snippet ](/data-ingestion/real-time-user-tracking/web-events#implementing-the-tag)that will automatically track $page\_view events with the URL of the page. 2. Use the Url Match event rule to extract properties from the page's URL and add them to the event. For example, if a $page\_view event is tracked with the URL `https://foo.bar/category/0001/article/super-awesome-article`, you would be able to generate this kind of event ```javascript { "$ts": 1568296040000, "$event_name": "article_view", "$properties": { "category": "0001", "article_id": "super-awesome-article", "source": "web" } } ``` #### How to The URL Match is taking 2 parameters: 1. The URL Pattern that will be used to match the $url value of $page\_view events 2. The event template that will be used to generate the event if the $url is matching the URL pattern **Patterns** are URL in which you can add: * A variable extraction rule for path parameters with:variableName * A wildcard with \* {% hint style="info" %} Wildcards can be placed several times in the URL pattern. For instance, you can do `https://foo.bar/*.*` which will match for a route shaped as `https://foo.bar/a/b/c.xls` {% endhint %} Example `*//foo.bar/category/:categoryId/article/:articleId` * \* at the beginning will match both URLs in and in http in the URL * :categoryId will extract the value in the URL corresponding to the categoryId. As it is a named variable, it can be used as a value in the event that will be generated. * :articleId works as:categoryId **The Query String values are automatically extracted** in variables that have the name of the parameter in the Query String. `https://foo.bar/a/b/c?var=value&var2=value2` extracts both value and value2 in variables named var and var2. The **event template** is containing the following information: * The $event\_name that will be used for the generated event as a static string * A list of properties in a key-value way that will be added in the event properties In the properties values you can either pass: * A static string directly. Ex: sport * A string in the format {{variableName}} that will be replaced by a value extracted in the URL pattern {% hint style="info" %} If a variable used in the event template is not extracted from the URL (either from the path or from the Query String), it's value will be {{variableName}} in the generated event. {% endhint %} #### Examples Let's take the following event rule: * Url pattern: `*//foo.bar/category/:categoryId/article/:articleId` * Event template: | Property | Value | | ------------------- | -------------- | | article\_id | {{articleId}} | | category\_id | {{categoryId}} | | visit\_origin | {{origin}} | | source\_event\_rule | 1 | \ $page\_view event with Url: `https://foo.bar/category/0001/article/super-awesome-article?origin=email` will generate ```javascript { "$ts": 1568296040000, "$event_name": "article_view", "$properties": { "category": "0001", "article_id": "super-awesome-article", "visit_origin": "email", "source_event_rule": "1" } } ``` \ $page\_view event with Url `http://foo.bar/category/0001/article/super-awesome-article` will generate ```javascript { "$ts": 1568296040000, "$event_name": "article_view", "$properties": { "category": "0001", "article_id": "super-awesome-article", "visit_origin": "{{origin}}", "source_event_rule": "1" } } ``` $page\_view event with Url `https://oof.bar/category/0001/article/super-awesome-article` will generate *nothing* as the Url domain (oof.bar) is not matching the pattern. page\_view event with Url `https://foo.bar/category/0001/article/super-awesome-article` will generate *nothing* as the "source" event name is not $page\_view.
### User Identifier Insertion This event rule allows you to **extract a property from an event to convert it into a user identifier,** such as an Email Hash or a User account ID. You can apply a hash on this extraction. We currently support the following hash methods: * SHA\_256 * SHA\_1 * SHA\_384 * SHA\_512 * MD5 * MD2 This event rule is useful when you don't want to pass an identifier as a global property of the activity, but rather have it computed directly by your datamart. *If several values of an identifier are found within an activity, we keep only the last one.* ```javascript { "$event_name": "PageView", "$properties": { "user_id": "" } } ``` In the scenario above, you will be able to extract the user\_id, store it as a User account ID. ### Contextual Targeting Extractor This event rule is used for Contextual targeting purpose. You can find more information about it in the [Contextual targeting > Setup](/advanced-usages/contextual-targeting/general-setup#event-rule) section # Activity analyzers An Activity Analyzer is a [Plugin](/advanced-usages/plugins) that allows you to modify an activity on the fly before storing it. It runs as a part of [the processing pipeline](/data-ingestion/real-time-user-tracking#the-processing-pipeline), for each activity of the channel it is associated with. This feature is useful for: * Reformatting data (adapting the ingestion data model to the datamart schema) * Enriching events (for instance by fetching product information based on a product id) * Improving data quality (filtering unwanted events, matching input values to standard catalogs, parsing URLs into categories etc.) {% hint style="success" %} If you don't know what a plugin is, you can find the [complete documentation in the specific section.](/advanced-usages/plugins) {% endhint %} {% hint style="info" %} An activity analyzer is only executed for activities tracked in real time, e.g. via the user\_activity API, javascript tag or pixel tracking (see [real time user tracking guide](/data-ingestion/real-time-user-tracking)). If you want to upload bulk activities, make sure they are already formatted before starting the upload as the activity analyzer won't run. The standard group ID for an activity analyzer is {domain}.{organisation}.activity-analyzer, for example com.mediarithmics.activity-analyzer {% endhint %} ## Endpoints to implement Activity analyzers have only one predefined endpoint to implement ## Process an activity `POST` `myworker/v1/activity_analysis` This entry point is called any time an activity is processed by the platform. The activity analyzer receives an activity and responds to the request by returning a new activity. It cannot modify the identifiers that are passed in the incoming activities. ### Request Body | Name | Type | Description | | ---------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------- | | activity\_analyzer\_id | string | The ID of the activity analyzer instance that should be used to process the activity. **Used to retrieve the instance properties.** | | datamart\_id | string | The ID of the datamart | | activity | object | The UserActivity Object to analyze | {% tabs %} {% tab title="200 In most cases, your plugin should respond with the modified user activity object that should be put into the platform." %} ```javascript { "status": "ok", "data": { // New UserActivity object } } ``` {% endtab %} {% tab title="302 Here is what the plugin should respond if it encounters an unexpected error." %} ```javascript { "status": "error", "error": "Your error message" } ``` {% endtab %} {% endtabs %} See [Plugin Instances ](/advanced-usages/plugins#instances)to learn why you should use the `activity_analyzer_id` parameter to retrieve the instance properties. {% hint style="info" %} If you need to create or update user profiles using the dedicated event `$set_user_profile_properties` , please refer to [Create / Update a UserProfile from within an activity](/data-ingestion/real-time-user-tracking/api#create-update-a-userprofile-from-within-an-activity). Specifically note that you must use properties `$compartment_id` and (optionnally) `$user_account_id`. Properties `$set_user_profile_comp_token` / `$set_user_profile_user_account_id` are reserved for the case when the event is [triggerred from a website via our javascript tag](/data-ingestion/real-time-user-tracking/web-events#userprofile-updates). They are translated into the above ones before the activity analyzer is called and should not be used in it. {% endhint %} ## Available outbound services The code of the activity analyzer can call the following API endpoints to retrieve[ its instance context](/advanced-usages/plugins#instances). ## Retrieve the instance `GET` `https://api.mediarithmics.com/v1/activity_analyzers/:id` Use the activity\_analyzer\_id from the incoming request to retrieve the activity analyzer instance that has been called. #### Path Parameters | Name | Type | Description | | ---- | ------ | -------------------------------------------------------------------------------------------- | | id | string | ID of the activity analyzer, typically the `activity_analyzer_id` from the incoming request. | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "1000", "name": "my analyzer", "organisation_id": "1000", "visit_analyzer_plugin_id": "1001", "group_id": "com.mediarithmics.visit-analyzer", "artifact_id": "default" } } ``` {% endtab %} {% endtabs %} ## Retrieve the instance properties `GET` `https://api.mediarithmics.com/v1/activity_analyzers/:id/properties` Get the properties associated with the activity analyzer instance #### Path Parameters | Name | Type | Description | | ---- | ------ | ------------------------------------------------------------------------------------------ | | id | string | ID of the activity analyzer, typically the`activity_analyzer_id` from the incoming request | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "technical_name": "debug", "value": { "value": false }, "property_type": "BOOLEAN", "origin": "PLUGIN", "writable": true, "deletable": false }, { "technical_name": "topic_properties", "value": { "value": "vertical" }, "property_type": "STRING", "origin": "PLUGIN", "writable": true, "deletable": false } ], "count": 2 } ``` {% endtab %} {% endtabs %} ## Creating an activity analyzer See the plugins documentation to see [how plugins are created and deployed](/advanced-usages/plugins). {% hint style="success" %} An activity analyzer has the `ACTIVITY_ANALYZER` plugin type. Its group id should be `{domain.organisation.activity-analyzer}` (for example com.mediarithmics.activity-analyzer). Its *artifact id* should be the name of the activity analyzer, ie `update-product-infos`. {% endhint %} Use our [Plugins SDK](/resources/tools-and-libraries/plugin-sdk) to create your activity analyzer in `nodejs` : the required routes are already defined and you only have to override specific functions. {% hint style="info" %} You can find a sample activity analyzer [in the examples folder of the plugins SDK](https://github.com/MEDIARITHMICS/plugins-nodejs-sdk/tree/master/examples/activity-analyzer). {% endhint %} We can provide you with a hello world project using our SDK. Please contact your Account manager in order to have access to it. The project structure and files work as [with every other plugin](/advanced-usages/plugins/creation). ### Interfaces to implement Your should extend `ActivityAnalyzerPlugin` class and implement the `instanceContextBuilder` and `onActivityAnalysis`functions from the plugins SDK. `onActivityAnalysis` function is called every time an activity runs through the activity analyzer. It is responsible for the activity transformation. The instance context built in `instanceContextBuilder` is cached to improve performances. It should retrieve and store the plugin properties and configuration files used by the code. {% hint style="warning" %} Don't forget to catch your errors. You should log / respond with the appropriate message to facilitate debugging. {% endhint %} ```javascript import { core } from "@mediarithmics/plugins-nodejs-sdk"; import { CustomInstanceContext } from "./interfaces/InstanceContextInterface"; export class ActivityAnalyzerPlugin extends core.ActivityAnalyzerPlugin { // Called to update a UserActivity // Uses the instance context built with instanceContextBuilder // to adapt to the properties and technical files protected async onActivityAnalysis( request: core.ActivityAnalyzerRequest, instanceContext: CustomInstanceContext) : Promise { try{ const updatedActivity = request.activity; // Your code to modify the activity. // Exemple adding product infos in each event // If the technical configuration allows it if (instanceContext.technicalConfig.updateActivities){ updatedActivity.$events.forEach(event => { if (event.$properties && event.$properties.$items && event.$properties.$items.length > 0) { event.$properties.$items.forEach((item: any) => { var product = Products.find(p => p.$id == item.$id); item.$name = product.$name; item.categories = product.categories; item.inStock = product.inStock; }); } }); } const response: core.ActivityAnalyzerPluginResponse = { status: "ok", data: updatedActivity }; return Promise.resolve(response); } catch (err) { const errorResponse: core.ActivityAnalyzerPluginResponse = { status: 'error', data: request.activity }; this.logger.error(`TRANSFORMATION ERROR while processing activity: ${JSON.stringify(request.activity)}`); return Promise.resolve(errorResponse) } } // Build the instance context // by fetching properties and configuration files protected async instanceContextBuilder(activityAnalyzerId: string) : Promise { const baseInstanceContext = await super.instanceContextBuilder(activityAnalyzerId); try { // Retrieve a technical configuration file const validator = new Jsonschema.Validator(); const technicalConfig: ITechnicalConfig = await this.validateJSONSchema(TECH_CONFIG_FILE, validator, technicalConfigurationSchema, activityAnalyzerId); // Retrieve a property from the plugin instance const eventExclusionList = baseInstanceContext.properties.findStringProperty("events_exclusion_list"); // Return the completed instance context const result: CustomInstanceContext = { ...baseInstanceContext, event_exclusion_list: eventExclusionList, technicalConfig: technicalConfig }; this.logger.debug(`Loaded InstanceContext with: ${JSON.stringify(result,null,4)}`); return Promise.resolve(result); } catch (err) { this.logger.error(`Something bad happened during the build of the Instance Context ${err}`); return Promise.reject(`Something bad happened during the build of the Instance Context ${err}`); } }; } ``` Your instance context interface should extend ActivityAnalyzerBaseInstanceContext ```javascript import { core } from "@mediarithmics/plugins-nodejs-sdk"; export interface CustomInstanceContext extends core.ActivityAnalyzerBaseInstanceContext { event_exclusion_list: string[]; technicalConfig: ITechnicalConfig; } ``` ## Creating an instance Like other plugins, activity analyzer need to be instantiated. To create an instance, connect to Navigator and head toward Settings > Datamart > Activity Analyzers. You will get a list of existing instances and a button to create new ones. ![](/files/CKpTr0M3CcnPPvt3GWi6) Click on **New Activity Analyzer**. ![](/files/Gvciu6kSThUuWSSpQcOj) Select the activity analyzer you want to instantiate. ![](/files/lJUmymQecc5ZANqrZwwv) Enter a **name** to easily recognize the instance, select an **Error recovery strategy** and fill **Properties** if you need to overwrite some of them. Save your modifications to create a new instance of your activity analyzer. The error recovery strategy determines how the activity is processed when the plugin fails.
error_recovery_strategyFailure reaction
STORE_WITH_ERROR_IDThe activity will be sent without any modification to the next activity analyzer.
STORE_WITH_ERROR_ID_AND_SKIP_UPCOMING_ANALYZERSThe activity will be saved without modification of the activity analyzer in failure. It doesn't be sent to the next plugin.
DROPThe activity won’t be saved
## Linking an instance to a channel Once your activity analyzer instance is created, you can link it to one or multiple channels. To do so, connect to Navigator and head toward Settings > Datamart > Channels and select the channel where you want your activity analyzer to be executed. Go to the **Activity Analyzers** category. ![](/files/kift7KJMvWRrnoFA1SW5) Click on **Add an Activity Analyzer** and select your instance. Several activity analyzers can be used on the same channel. In this case, they will process the same activity in a sequence of your choice: the second analyzer will process the activity as rendered by the first one and so on... {% hint style="info" %} Currently, you can't get more than 5 activity analyzers. If you need more, please contact your Account manager. {% endhint %} Make sure to define the right **order** and **error recovery strategies.** ## Debugging ### Plugin logs and metrics As activity analyzers are plugin, you can monitor them[ as you do with all plugins](/advanced-usages/plugins/monitoring). ### Verifying an activity {% hint style="info" %} UserActivity that run through activity analyzers are generally aggregated into sessions. You won't see your UserActivity until it has been put into a session and gone through the whole activity processing pipeline. See [how sessions are built](/data-ingestion/real-time-user-tracking#session-aggregation) to understand when you should see your activity or how you could fasten the process. {% endhint %} 1. Go to the navigator > monitoring and search for the UserPoint associated with the activity. 2. Click on the **view json** button on any activity on a timeline 3. You can check if all the properties are OK and if your activity analyzers processed the activity as expect In case of problem, you can look at two properties added to the activity. `processed_by` will tell you if the activity has been processed by your activity analyzer, and `$error_analyzer_id` will give you an error ID if the activity analyzer returned an error response. ```javascript { "processed_by": "", "$error_analyzer_id": "" } ``` # Bulk processing Bulk import aims at **giving you the ability to bulk-import data into the mediarithmics platform**. You can import: * [Offline activities](/data-ingestion/bulk-processing/imports/offline-activities) such as offline purchases and store visits * User segments such as email lists, cookies list, user accounts list, etc. * User profiles such as CRM data and scoring * User association such as CRM Onboarding * User dissociation * User suppression requests such as GDPR Suppression requests, and Opt-Out Management ## How it works You upload files associated with a document import definition: * Files represent the data. * Document imports represent what mediarithmics should do with the data. If you need to track users in real-time, you should read [the real-time tracking guide.](/data-ingestion/real-time-user-tracking) The two steps for bulk import are: 1. Create the document import definition to tell mediarithmics what you are importing 2. Upload files associated with the document import definition. Each uploaded file creates a new document import execution. {% hint style="info" %} For maximum performance: * Ensure a maximum size for each file of 100M. * Use the document import for multiple records when there will be more than 1,000 per file. {% endhint %} *How to choose between creating a new document import or adding a new file to an existing document import? Our recommendation is to create a new document import each time you have a new set of files to upload. For example, if you upload CRM profiles every night, you should create a new "User profiles from CRM - " document import every night instead of just uploading new files to a unique "User profiles from CRM" document import.* {% hint style="success" %} Each line in the uploaded file is a command to execute. Depending on the document import type, you have different commands available. {% endhint %} ## User identifiers in imports When importing data, you need to properly add [user identifiers](/user-points#user-identifiers). This will ensure your data is associated with the proper [UserPoint](/user-points). {% hint style="warning" %} Only one identifier is allowed per line. For example, you shouldn't specify the user agent ID if the Email Hash is already used in a line. However, you don't have to always use the same type of identifier in your document. For example, one line could use the user account ID while another uses the email hash. {% endhint %} ## Document import Document imports define what you are about to upload in one or multiple files. A document import object has the following properties: | field | type | description | | ------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | document\_type | Enum |

The type of data you want to import. Should be USER\_ACTIVITY, USER\_SEGMENT, USER\_PROFILE,

USER\_CHOICE,

USER\_IDENTIFIERS\_DELETION , or USER\_IDENTIFIERS\_ASSOCIATION\_DECLARATIONS

| | mime\_type | Enum | The format of the imported data. `APPLICATION_X_NDJSON`or `TEXT_CSV`It should match the file format of the upload file, e.g. `.csv` or `.ndjson`. The csv format can be chosen only for `USER_SEGMENT` imports. | | encoding | String | Encoding of the data that will be imported. Usually`utf-8` | | name | String | The name of your import. | | priority | Enum | `LOW`, `MEDIUM` or `HIGH` | | use\_processing\_pipeline | Boolean | Use this parameter if the import should go through activity analyzers or session aggregation for instance. Values are `true` or `false`. Default is `false` | | shuffle\_lines | Boolean | Will shuffle the lines of the file for better performance. Values are : `true` or `false`. Default is `true` | ```javascript // Sample document import object { "document_type": "USER_ACTIVITY", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" } ``` ## Create a document import `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports` #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------------------------------------------- | | datamartId | integer | The ID of the datamart in which your data will be imported | #### Request Body | Name | Type | Description | | ---- | ------ | --------------------------------------------- | | data | object | The document import object you wish to create | Response: {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "36271", "datafarm_key": "DF_KEY", "datamart_id": "DATAMART_ID", "document_type": "USER_PROFILE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "YOUR_DOCUMENT_IMPORT_NAME", "priority": "MEDIUM", "shuffle_lines" : true, "use_processing_pipeline" : false } } ``` {% endtab %} {% endtabs %} Here is a sample request using **curl**: ```bash curl -X POST \ "https://api.mediarithmics.com/v1/datamarts//document_imports" -H 'Authorization: ' -H 'Content-Type: application/json' -d '{ "document_type": "USER_ACTIVITY", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ## List document imports `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports` You can list all document imports for a datamart or search them with filters. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | #### Query Parameters | Name | Type | Description | | --------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | keywords | string | The keywords to match with document import names. It is case sensitive.Examples: | | mime\_type | string | Filter on a specific mime type. Supported values are `APPLICATION_X_NDJSON` or `TEXT_CSV` . | | document\_types | string | Filter on specific document types. Supported values are`USER_PROFILE`, `USER_ACTIVITY` or `USER_SEGMENT` .Multiple filters can be separated with commas.*Examples :* `&document_types=USER_PROFILE` *or* `&document_types=USER_PROFILE,USER_ACTIVITY` | | order\_by | string | ID sorts result by default, you can specify `&order_by=name` to sort them by name | The query is paginated as described in [using our API guide](/resources/api-overview). {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "id": "19538", "datafarm_key": "DF_KEY", "datamart_id": "DATAMART_ID", "document_type": "USER_PROFILE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "December 2020 user profiles", "priority": "MEDIUM", "shuffle_lines" : true, "use_processing_pipeline" : false }, { "id": "19552", "datafarm_key": "DF_KEY", "datamart_id": "DATAMART_ID", "document_type": "USER_PROFILE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "January 2021 user profiles", "priority": "MEDIUM", "shuffle_lines" : true, "use_processing_pipeline" : false }, { "id": "19553", "datafarm_key": "DF_EU_2020_02", "datamart_id": "1509", "document_type": "USER_PROFILE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "February 2021 user profiles", "priority": "MEDIUM", "shuffle_lines" : true, "use_processing_pipeline" : false } ], "count": 3, "total": 3, "first_result": 0, "max_result": 50, "max_results": 50 } ``` {% endtab %} {% endtabs %} ## Get a document import `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId` #### Path Parameters | Name | Type | Description | | ---------- | ------- | ----------------------------- | | datamartId | integer | The ID of the datamart | | importId | integer | The ID of the document import | {% tabs %} {% tab title="200 " %} ```json { "status": "ok", "data": { "id": "36271", "datafarm_key": "DF_KEY", "datamart_id": "DATAMART_ID", "document_type": "USER_PROFILE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "December 2020 user profiles", "priority": "MEDIUM", "shuffle_lines" : true, "use_processing_pipeline" : false } } ``` {% endtab %} {% endtabs %} ## Update a document import `PUT` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId` #### Path Parameters | Name | Type | Description | | ---------- | ------- | ----------------------------- | | datamartId | integer | The ID of the datamart | | importId | integer | The ID of the document import | #### Request Body | Name | Type | Description | | ---- | ------ | --------------------------------- | | data | object | The document import object to put | {% tabs %} {% tab title="200 " %} ```json { "status": "ok", "data": { "id": "36271", "datafarm_key": "DF_KEY", "datamart_id": "DATAMART_ID", "document_type": "USER_PROFILE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "YOUR_DOCUMENT_IMPORT_NAME", "priority": "MEDIUM", "shuffle_lines" : true, "use_processing_pipeline" : false } } ``` {% endtab %} {% endtabs %} ## Remove a document import `DELETE` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId` Removes a document import you don't want to see anymore in the system. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ----------------------------- | | datamartId | integer | The ID of the datamart | | importId | integer | The ID of the document import | ## File upload A file upload creates an execution. After creation, the execution is at the `PENDING` status. It goes into the `RUNNING` status when the import starts and `SUCCEEDED` status once the platform has correctly imported the file. ## Create an execution `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions` You create an execution and upload a file with this endpoint. #### Path Parameters | Name | Type | Description | | -------------------------------------------- | ------ | ----------------------------- | | datamartId\* | string | The ID of the datamart | | importId\* | string | The ID of the document import | #### Headers | Name | Type | Description | | ---------------------------------------------- | ------ | -------------------------- | | Content-Type\* | string | Your upload configuration. | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "parameters": null, "result": null, "error": null, "id": "11597785", "status": "PENDING", "creation_date": 1609410143659, "start_date": null, "duration": null, "organisation_id": "1426", "user_id": null, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "MTowOjA6NDI1MzAxMg==", "num_tasks": null, "completed_tasks": null, "erroneous_tasks": null, "retry_count": 0, "job_type": "DOCUMENT_IMPORT", "import_mode": "MANUAL_FILE", "import_type": null } } ``` {% endtab %} {% endtabs %} See an example: ``` curl --location --request POST 'https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:executionId/executions/' \ --header 'Content-Type: application/x-ndjson; \ --header 'Authorization: api:TOKEN' \ --data-binary '@/Users/username/path/to/the/file.ndjson' ``` You retrieve metadata about the created execution, notably and id property you can use to track the execution. ## List executions `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions` You can list all executions for a document, import and retrieve useful data like their status, execution time and error messages. #### Path Parameters | Name | Type | Description | | -------------------------------------------- | ------- | ------------------------- | | datamartId\* | integer | The ID of the datamart | | importId\* | integer | The ID of document import | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "parameters": { "datamart_id": 1609, "document_import_id": 19718, "mime_type": "APPLICATION_X_NDJSON", "document_type": "USER_PROFILE", "input_file_name": "requestBody9664967795462448677asRaw", "file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody9664967795462448677asRaw-2020-12-31_10.22.23-KzgivDim3y.json", "number_of_lines": 4, "segment_id": null }, "result": { "total_success": 4, "total_failure": 0, "input_file_name": "requestBody9664967795462448677asRaw", "input_file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody9664967795462448677asRaw-2020-12-31_10.22.23-KzgivDim3y.json", "error_file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody9664967795462448677asRaw-2020-12-31_10.22.23-KzgivDim3y_errors.csv", "possible_issue_on_identifiers": false, "top_identifiers": {} }, "error": null, "id": "11597785", "status": "SUCCEEDED", "creation_date": 1609410143659, "start_date": 1609410150976, "duration": 3059, "organisation_id": "1426", "user_id": null, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "MTowOjA6NDI1MzAxMg==", "num_tasks": 4, "completed_tasks": 4, "erroneous_tasks": 0, "retry_count": 0, "job_type": "DOCUMENT_IMPORT", "import_mode": "MANUAL_FILE", "import_type": null, "end_date": 1609410154035 }, { "parameters": { "datamart_id": 1609, "document_import_id": 19718, "mime_type": "APPLICATION_X_NDJSON", "document_type": "USER_PROFILE", "input_file_name": "requestBody17471990940413569967asRaw", "file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody17471990940413569967asRaw-2020-10-19_09.54.45-JvP1ssxKSu.json", "number_of_lines": 4, "segment_id": null }, "result": { "total_success": 0, "total_failure": 4, "input_file_name": "requestBody17471990940413569967asRaw", "input_file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody17471990940413569967asRaw-2020-10-19_09.54.45-JvP1ssxKSu.json", "error_file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody17471990940413569967asRaw-2020-10-19_09.54.45-JvP1ssxKSu_errors.csv", "possible_issue_on_identifiers": false, "top_identifiers": {} }, "error": { "message": "0 success, 4 failures\nSaved errors:\nNo profile id found while upserting a user profile Error id = 9d5016ea-6b7b-4c64-bc74-60ba207e3bed.\nNo profile id found while upserting a user profile Error id = 99f8d9bb-4c94-49ea-8bb2-934bc6056cac.\nNo profile id found while upserting a user profile Error id = d1216b0e-619c-4d92-9098-cc5ae4ac8e16.\nNo profile id found while upserting a user profile Error id = a92d3258-163c-4b9d-949e-94f9006cd77d.\n" }, "id": "11170897", "status": "SUCCEEDED", "creation_date": 1603101286198, "start_date": 1603101317674, "duration": 1062, "organisation_id": "1426", "user_id": null, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "MTowOjA6MzgyNjEyNA==", "num_tasks": 4, "completed_tasks": 0, "erroneous_tasks": 4, "retry_count": 0, "job_type": "DOCUMENT_IMPORT", "import_mode": "MANUAL_FILE", "import_type": null, "end_date": 1603101318736 } ], "count": 2, "total": 2, "first_result": 0, "max_result": 50, "max_results": 50 } ``` {% endtab %} {% endtabs %} ## Get an execution `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions/:executionId` Get a specific execution and retrieves useful data like its status, execution time and error messages. #### Path Parameters | Name | Type | Description | | --------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------- | | datamartId\* | integer | The ID of the datamart | | importId\* | integer | The ID of the document import | | executionId\* | integer | The ID of the execution (usually retrieved from "create execution" or "list executions" requests) | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "parameters": { "datamart_id": 1609, "document_import_id": 19718, "mime_type": "APPLICATION_X_NDJSON", "document_type": "USER_PROFILE", "input_file_name": "requestBody9664967795462448677asRaw", "file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody9664967795462448677asRaw-2020-12-31_10.22.23-KzgivDim3y.json", "number_of_lines": 4, "segment_id": null }, "result": { "total_success": 4, "total_failure": 0, "input_file_name": "requestBody9664967795462448677asRaw", "input_file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody9664967795462448677asRaw-2020-12-31_10.22.23-KzgivDim3y.json", "error_file_uri": "mics://data_file/tenants/1426/datamarts/1509/document_imports/19518/requestBody9664967795462448677asRaw-2020-12-31_10.22.23-KzgivDim3y_errors.csv", "possible_issue_on_identifiers": false, "top_identifiers": {} }, "error": null, "id": "11597785", "status": "SUCCEEDED", "creation_date": 1609410143659, "start_date": 1609410150976, "duration": 3059, "organisation_id": "1426", "user_id": null, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "MTowOjA6NDI1MzAxMg==", "num_tasks": 4, "completed_tasks": 4, "erroneous_tasks": 0, "retry_count": 0, "job_type": "DOCUMENT_IMPORT", "import_mode": "MANUAL_FILE", "import_type": null, "end_date": 1609410154035 } } ``` {% endtab %} {% endtabs %} ## Cancel an execution `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions/:executionId/action` Cancel a specific execution #### Path Parameters | Name | Type | Description | | --------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------- | | datamartId\* | string | The ID of the datamart | | importId\* | string | The ID of the document import | | executionId\* | string | The ID of the execution (usually retrieved from "create execution" or "list executions" requests) | #### Request Body | Name | Type | Description | | -------------------------------------- | ---- | ------------------------------ | | body\* | json | Must be: `{"action":"CANCEL"}` | {% tabs %} {% tab title="200: OK " %} ```javascript { "status": "ok", "data": { "parameters": null, "result": null, "error": null, "id": "22747195", "status": "CANCELED", "creation_date": 1646060596034, "start_date": null, "duration": null, "organisation_id": "1581", "user_id": null, "cancel_status": "REQUESTED", "debug": null, "is_retryable": false, "community_id": "1581", "num_tasks": null, "completed_tasks": null, "erroneous_tasks": null, "retry_count": 0, "permalink_uri": null, "job_type": "DOCUMENT_IMPORT", "import_mode": "MANUAL_FILE", "import_type": null } } ``` {% endtab %} {% endtabs %} The cancellation of an execution will only work if the status of this executions is "PENDING" ## Splitting large files If you need to import larger files than 100Mbytes, you can split them before using the upload API and call it multiple times. You can split massive files using the shell command. ```bash split -l ./your/file/path ``` # Imports # User activities import Bulk import [user activities](/user-points/user-activities#user-activity-object) to insert user activities that happened outside of [real-time tracking](/data-ingestion/real-time-user-tracking). Those activities usually are offline activities, like store visits or store purchases, but you can adapt it to your use cases. {% hint style="warning" %} Activities imported through bulk import don't go through the [activity processing pipeline](/data-ingestion/real-time-user-tracking#the-processing-pipeline). You shouldn't use this feature if you intend to do conversion detection or automation activation. Your [activity analyzers](/data-ingestion/real-time-user-tracking/activity-analyzers) also won't process those activities, and you should format exactly how you expect them to be stored by mediarithmics. {% endhint %} ## How-to Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the`USER_ACTIVITY`document type and `APPLICATION_X_NDJSON` mime type. Only `ndjson` data is supported for user activities. Then, create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your user activities formatted in `ndjson` . ### Example ```bash # Create the document import curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_ACTIVITY", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ```bash # Create the execution curl -X POST \ https://api.mediarithmics.com/v1/datamarts/1162/document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d '{ "$user_agent_id": "", "$type":"TOUCH","$session_status":"NO_SESSION","$ts":,"$events":[{"$event_name":"$email_mapping","$ts":,"$properties":{}}]}' ``` {% hint style="success" %} You can, of course, upload multiple activities at once. Note the uploaded data is in `ndjson` and not `json`. That means the different activities are not separated by commas, but by a line separator `\n` {% endhint %} # User profiles import Use this feature to UPSERT or DELETE [user profiles](/user-points/user-profiles) in your datamart. ## How-to Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the`USER_PROFILE`document type and `APPLICATION_X_NDJSON` mime type. Only `ndjson` data is supported for user profiles. {% hint style="success" %} You can, of course, upload multiple user profiles at once. Note the uploaded data is in `ndjson` and not `json`. That means the different profiles are not separated by commas, but by a line separator `\n` {% endhint %} Then, create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your user profile import commands formatted in `ndjson` . ### User profile import command Each line in the `ndjson` refers to one profile (`user_profile` property). It is wrapped in a JSON object that contains information about the UserPoint to target, the operation to perform, the update strategy etc. Selecting **a single main identifier** in the JSON wrapper is mandatory. The available ones are : * user\_account\_id and compartment\_id * email\_hash * user\_agent\_id Optionally, you can, in addition to the main identifier, also give a list of identifiers related to the same UserPoint in the user\_identifiers property. It's different from [User Activities import](/data-ingestion/bulk-processing/imports/offline-activities) where there is no wrapper, and the identifiers list is given directly in the User Activity payload. Each line in the uploaded file can have the following properties: | property | type | description | | ----------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | operation | Enum | Either `UPSERT` or `DELETE` | | user\_account\_id | String (One of. In combination with compartment\_id ) | The User Account ID, acting as main user [identifier](/user-points/user-profiles#user-profile-object) in correlation with `compartment_id` | | compartment\_id | String (One of. In combination with user\_account\_id) | The compartment ID, used when `user_account_id` is acting as main user identifier | | email\_hash | String (One of) | The Email Hash, acting as main user [identifier](/user-points#user-identifiers) | | user\_agent\_id | String (One of) | The User Agent ID, acting as main user [identifier](/user-points#user-identifiers) | | update\_strategy | Enum (Optional) |

Only considered when operation == UPSERT
Values are PARTIAL\_UPDATE, PARTIAL\_DELETE, FORCE\_REPLACE

| | user\_profile | JSON Object (Optional) |

Mandatory when operation == UPSERT.

JSON Object representing the User Profile. Please refer to the user profile object for more information.

| | user\_identifiers | Array of [User Identifiers Resource](#user-identifiers-resource) (optional) |

List of User Identifier objects

  • For UPSERT : this will run a user identifier association before doing the upsert of the profile
  • For DELETE : the search of the UserPoint will be done on all identifiers (main one AND the ones in user\_identifiers). The search must resolve to one UserPoint only otherwise it will fail
| {% hint style="warning" %} When importing profiles, only one main identifier is allowed per line ; for instance, you shouldn't specify the user agent ID if the email hash is already declared in a given line.
However, for a given line, if you use the user\_identifiers property : it will first perform a user identifier association to stitch these identifiers to the profile, and then upsert the profile {% endhint %} ### User Identifiers Resource #### User Account Resource | field | type | description | | ----------------- | -------------------- | -------------------------------------------- | | type | Enum | Must be `USER_ACCOUNT` in this case | | compartment\_id | string | Compartment associated with the user account | | user\_account\_id | string | The user\_account\_id | | expiration\_ts | Timestamp (optional) | Account's eventual expiration timestamp | #### User Email Resource | field | type | description | | -------------- | -------------------- | ------------------------------------- | | type | Enum | Must be `USER_EMAIL` in this case | | hash | string | Hashed user email | | email | string (optional) | User's email, not hashed. | | expiration\_ts | Timestamp (optional) | Email's eventual expiration timestamp | #### User Agent Resource | field | type | description | | --------------- | -------------------- | --------------------------------------------- | | type | Enum | Must be `USER_AGENT` in this case | | user\_agent\_id | string | User's user\_agent\_ id | | expiration\_ts | Timestamp (optional) | user\_agent\_id eventual expiration timestamp | ### Example First create the document import using the call below. You can also reuse a document import that was previously created ```bash curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_PROFILE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` Each user profile import you do will be a new execution of the document import created. Here is a simple example : ```bash # Here we use the combination of compartment_id and user_account_id acting as identifier curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d '{ "operation": "UPSERT", "compartment_id": "", "user_account_id": "", "user_profile": { this": "is", "a":"test" } }' ``` Here is an exemple using the user\_identifier field : ```bash # Here we use the user_agent_id acting as main identifier and a list of user_identifiers to stitch to the userpoint before upserting the porifle curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d '{ "operation": "UPSERT", "user_agent_id": "", "user_profile": { this": "is", "a":"test" } "user_identifiers": [ { "type": "USER_ACCOUNT", "compartment_id": "2785", "user_account_id": "crm_0000057" } { "type": "USER_AGENT", "user_agent_id": "vec:0198a1b2c3d4" } { "type": "USER_EMAIL", "hash": "f36897...4034a", "email": "john@doe.com" } ] }' ``` ## Available update strategies when updating existing user profiles When doing an `UPSERT` if you want to update existing profiles in your datamart you should use the `update_strategy` property. ### PARTIAL\_UPDATE and PARTIAL\_DELETE If you wish to perform targeted updates on existing profiles without overwriting the whole existing user profile object, you should use the `PARTIAL_UPDATE` or `PARTIAL_DELETE` values of the `update_strategy` property. There are 2 main usage for these strategies : 1. **Dealing with arrays of objects** If you're dealing with arrays of objects, these strategies work together with two [directives](/schema#directives) that should be defined on the schema of the datamart you are working on. In this case, you should first [update the schema](/schema/defining-your-schema) in order to include the directives. If you want to make targeted updates on a object that has "id-like" field that can be used to identify the object, use `@UpdateStrategyKey` * Mark the **field** with the directive inside the given object you would like to make updates on. The field that has the directive will serve to identify the given object based on the value of this field in the update request. * For example of where the directive is needed, see use cases #1 to #5 below. If you want to make targeted updates on a object that does not have a field that can be used to identify the object you should use `@UpdateValueObject` * Mark the **object** with the directive : send in the payload with the new value of the object and it will override the previous value. * For example of where the directive is needed, see use case #6 and #7 below. {% hint style="warning" %} **You should use only one of the directive in a given object** {% endhint %} 2. **Dealing with objects** When there are no arrays of objects involved, you can still use the `PARTIAL_UPDATE` but the no directive is necessary. For examples, check use cases #8 to #10. #### **#1 - Add an object in array of objects - PARTIAL\_UPDATE** using the `@UpdateStrategyKey` directive ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String loyalty: [Loyalty] } type Loyalty { cards : [Cards] } type Cards { card_id: String! @UpdateStrategyKey benefits: String last_visit_date: String } ``` ```json // Existing profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": 500, "last_visit_date": "2024-01-01" } ] } } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ //adding a new cards object { "card_id": "def", "benefits": 100, "last_visit_date": "2025-01-01" } ] } } //New profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ // the cards object contains both cards { "card_id": "abc", "benefits": 500, "last_visit_date": "2024-01-01" }, { "card_id": "def", "benefits": 100, "last_visit_date": "2025-01-01" } ] } } ``` #### **#2 - Update a property in an array of objects - PARTIAL\_UPDATE** using the `@UpdateStrategyKey` directive ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String loyalty: [Loyalty] } type Loyalty { cards : [Cards] } type Cards { card_id: String! @UpdateStrategyKey benefits: String last_visit_date: String } ``` ```json // Existing profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": 500, "last_visit_date": "2024-01-01" } ] } } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ //request changing the benefits value of the existing card { "card_id": "abc", // value of the field marked with @UpdateStrategyKey of the inner object to update "benefits": 100 } ] } } //New profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ // { "card_id": "abc", "benefits": 100, // value has been updated "last_visit_date": "2024-01-01" } ] } } ``` {% hint style="info" %} Remark : arrays of scalar values are treated as scalars. See example below : {% endhint %} ```json // Extract of existing profile { "compartment_id": "", "user_account_id": "", "my_array_of_scalars": [1,2,3], ... } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "my_array_of_scalars": [4,5,6], ... } //New profile { "compartment_id": "", "user_account_id": "", "my_array_of_scalars": [4,5,6], // the value of the property has been replaced by the new value ... } ``` #### **#3 - Delete a property inside an array of objects - PARTIAL\_UPDATE** using the `@UpdateStrategyKey` directive ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String loyalty: [Loyalty] } type Loyalty { cards : [Cards] } type Cards { card_id: String! @UpdateStrategyKey benefits: String last_visit_date: String } ``` For a non mandatory value you can also set a given field to `null` (but not an object directly). For instance in the previous example you could have done the following : ```json // Existing profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": 500, "last_visit_date": "2024-01-01" } ] } } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": null // set the value to null } ] } } //New profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ // benefits value has been cleared { "card_id": "abc", "last_visit_date": "2024-01-01" } ] } } ``` #### **#4 - Update object inside an array and add new object - PARTIAL\_UPDATE** using the `@UpdateStrategyKey` directive ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String loyalty: [Loyalty] } type Loyalty { cards : [Cards] } type Cards { card_id: String! @UpdateStrategyKey benefits: String last_visit_date: String } ``` ```json // Existing profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": 500, "last_visit_date": "2024-01-01" } ] } } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": 100 //updating exsisting object }, { "card_id": "def", //adding a new object "benefits": 600, "last_visit_date": "2025-01-01" } ] } } //New profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": 100, // value has been updated "last_visit_date": "2024-01-01" }, { "card_id": "def", // object has been added "benefits": 600, "last_visit_date": "2025-01-01" } ] } } ``` #### **#5 - Delete an object in an array of objects - PARTIAL\_DELETE** using the `@UpdateStrategyKey` directive ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String loyalty: [Loyalty] } type Loyalty { cards : [Cards] } type Cards { card_id: String! @UpdateStrategyKey benefits: String last_visit_date: String } ``` ```json //Existing profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc", "benefits": 100, "last_visit_date": "2024-01-01" }, { "card_id": "def", "benefits": 600, "last_visit_date": "2025-01-01" } ] } } //Profile in update payload with update_strategy == PARTIAL_DELETE { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "card_id": "abc" // value of the field marked with @UpdateStrategyKey of the inner object to delete } ] } } //New profile { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ // the card with card_id = "abc" has been deleted { "card_id": "def", "benefits": 600, "last_visit_date": "2025-01-01" } ] } } ``` {% hint style="warning" %} **If you wish to delete a specific property value inside an object, you should use PARTIAL\_UPDATE. Please refer to use case #3.** {% endhint %} #### **#6 - Override an array of objects - PARTIAL\_UPDATE** using the `@UpdateValueObject` directive ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String segmentations: [Segmentation] } type Segmentation @UpdateValueObject { type: String label: String } ``` ```json //Existing profile { "compartment_id": "", "user_account_id": "", "segmentations": [ { "type": "segRFM", "label": "seg1" }, { "type": "segRFM", "label": "seg2" } ] } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "segmentations": [ // contains the new value of the object { "type": "segOther", "label": "seg3" }, { "type": "segOther", "label": "seg4" } ] } //New profile { "compartment_id": "", "user_account_id": "", "segmentations": [ //full array of objects was replaced { "type": "segOther", "label": "seg3" }, { "type": "segOther", "label": "seg4" } ] } ``` #### **#7 - Override an object - PARTIAL\_UPDATE** using the `@UpdateValueObject` directive ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String geolocation: Geolocation } type Geolocation @UpdateValueObject { address: String city: String postal_code : String } ``` ```json //Existing profile { "compartment_id": "", "user_account_id": "", "geolocation": { "address": "1 first avenue", "city": "new york", "postal_code" :"0101" } } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "geolocation": { "address": "1 first avenue", "city": "new york" } } //New profile { "compartment_id": "", "user_account_id": "", "geolocation": { "address": "1 first avenue", "city": "new york" } } ``` #### **#8 - Update properties inside the UserProfile - PARTIAL\_UPDATE with no directive** ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String first_name: String last_name : String } ``` ```json //Existing profile { "compartment_id": "", "user_account_id": "", "first_name" : "john" "last_name" : "doe" } //Profile in update payload with update_strategy == PARTIAL_UPDATE { "compartment_id": "", "user_account_id": "", "first_name" : null "last_name" : "smith" } //New profile { "compartment_id": "", "user_account_id": "", "last_name" : "smith" } ``` #### **#9 - Update a property inside an object - PARTIAL\_UPDATE with no directive** ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String geo_location: GeoLocation } type GeoLocation { address: String postal_code: String city: String country: String } ``` 
//Existing profile

{
  "compartment_id": "<COMPARTMENT_ID>",
  "user_account_id": "<USER_ACCOUNT_ID>",
  "geo_location": {
    "address": "1 main street",
    "city": "paris",
    "postal_code" : "0001",
    "country" : "France"
  }
}

//Profile in update payload with update_strategy == PARTIAL_UPDATE
{
  "compartment_id": "<COMPARTMENT_ID>",
  "user_account_id": "<USER_ACCOUNT_ID>",
 "geo_location": {
    "address": null,
    "city": "New york",
  }
}

//New profile

{
  "compartment_id": "<COMPARTMENT_ID>",
  "user_account_id": "<USER_ACCOUNT_ID>",
  "geo_location": {
    "city": "new york",
    "postal_code" : "0001",
    "country" : "France"
  }
}
#### **#10 - Override an object - PARTIAL\_UPDATE with no directive** ```graphql # Schema extract type UserProfile { compartment_id : String! user_account_id : String geo_location: GeoLocation } type GeoLocation { address: String postal_code: String city: String country: String } ``` 
//Existing profile

{
  "compartment_id": "<COMPARTMENT_ID>",
  "user_account_id": "<USER_ACCOUNT_ID>",
  "geo_location": {
    "address": "1 main street",
    "city": "paris",
    "postal_code" : "0001",
    "country" : "France"
  }
}

//Profile in update payload with update_strategy == PARTIAL_UPDATE
{
  "compartment_id": "<COMPARTMENT_ID>",
  "user_account_id": "<USER_ACCOUNT_ID>",
 "geo_location": {
    "address": "52 fifth avenue",
    "city": "New York City",
    "postal_code" : "12345",
    "country" : "USA"
  }
}

//New profile

{
  "compartment_id": "<COMPARTMENT_ID>",
  "user_account_id": "<USER_ACCOUNT_ID>",
  "geo_location": {
    "address": "52 fifth avenue",
    "city": "New York City",
    "postal_code" : "12345",
    "country" : "USA"
  }
}
#### Constraints and limitations **Schema related constraints :** The update request **must** respect the datamart schema : * If a property is in the payload of the request but not declared in the schema, the whole request will fail. However, if a property is already present in the profile but not declared in the schema, it will not be overwritten by a partial update or delete. In fact it will not be possible to update such property unless using `FORCE_REPLACE` * If the types of the properties in the payload of the request do not respect the schema, the whole request will fail. {% hint style="info" %} * It is possible to define a hierarchy of inner objects while using `@UpdateStrategyKey` at each level * However, `@UpdateStrategyKey` should only be set to one property inside a given object {% endhint %} ```graphql #This is allowed type LoyaltyCard { card_id:String @UpdateStrategyKey benefits: Int last_visit_date: String other_information : [OtherInformation] } type OtherInformation { info_id: String @UpdateStrategyKey } ################################ #This is not allowed type LoyaltyCard { card_id:String @UpdateStrategyKey benefits: Int @UpdateStrategyKey last_visit_date: Date } ``` **Limitations :** * If a property in the schema has a directive such as `@Property(path: “[parent].property”)` (referencing \[parent] in path) : this property cannot be updated with the partial update * Updatable properties cannot be computed values such as : computed field, ML function, results of function. * `@UpdateValueObject` cannot be used with `@UpdateStrategyKey` in the same object ### FORCE\_REPLACE When importing user profiles using `UPSERT`, if you wish to update existing profiles by completely overwriting the existing profiles you should use the `FORCE_REPLACE` value of the `update_strategy` property: the user profile will be completely replaced by the object passed in the `user_profile` property. ```bash # Stored profile: { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "benefits": 200, "card_id": "abc", "last_visit_date": "2024-01-01" } ] } } # New profile in request payload curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d '{ "operation": "UPSERT", "compartment_id": "", "user_account_id": "", "update_strategy": "FORCE_REPLACE", "user_profile": { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "benefits": 500, "card_id": "xyz", "last_visit_date": "2025-01-01" } ] } } }' # New saved profile: { "compartment_id": "", "user_account_id": "", "loyalty": { "cards": [ { "benefits": 500, "card_id": "xyz", "last_visit_date": "2025-01-01" } ] } } ``` ## Legacy parameters | property | type | description | | -------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | force\_replace | Boolean (Optional) |

Mandatory when the operation is UPSERT.

If true, then the User Profile will be completely replaced by the object passed in the user\_profile field.
If false, the object passed in the user\_profile field will be merged with the existing User Profile of the UserPoint.

| | merge\_objects | Boolean (Optional) |

Only considered if force\_replace is false.


Manage the comportement between two objects with a same property.


If false (default value), the new object overrides the existing one.

If true the new object is merged in deep to the existing one (see example).

| More details on `merge_objects` behavior : ```bash # Stored profile: { "my_property_1": "value1", "my_property_2": "value1", "my_array_property": ["value1"] "my_array_object_property": [ { "my_sub_array_object_property_1": "value1", "my_sub_array_object_property_2": "value1" } ], "my_object_property": { "my_sub_object_property_1": "value1", "my_sub_object_property_2": "value1" } } # New profile in request payload curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d '{ "operation": "UPSERT", "compartment_id": "", "user_account_id": "", "force_replace": false, "merge_objects": true, "user_profile": { "my_property_2": "value2", "my_property_3": "value3", "my_array_property": ["value2"] "my_array_object_property": [ { "my_sub_array_object_property_2": "value2" "my_sub_array_object_property_3": "value3" } ], "my_object_property": { "my_sub_object_property_2": "value2" "my_sub_object_property_3": "value3" } } }' # New saved profile: { "my_property_1": "value1", "my_property_2": "value2", # override scalar property "my_property_3": "value3", "my_array_property": ["value1","value2"] # merge arrays "my_array_object_property": [ # merge arrays { "my_sub_array_object_property_1": "value1" "my_sub_array_object_property_2": "value1" }, { "my_sub_array_object_property_2": "value2" "my_sub_array_object_property_3": "value3" } ], "my_object_property": { # merge objects "my_sub_object_property_1": "value1" "my_sub_object_property_2": "value2" # override scalar property within object "my_sub_object_property_3": "value3" } } ``` # User choices import Use this feature to UPSERT or DELETE [user\_choices](/advanced-usages/data-privacy-compliance/user-choices) in your datamart. ## How-to Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the`USER_CHOICE`document type and `APPLICATION_X_NDJSON` mime type. Only `ndjson` data is supported for user choices. Then, create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your user choice import commands formatted in `ndjson` . ### User choice import command Each line in the uploaded file can have the following properties: | field | type | description | | ----------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | operation | Enum | Either `UPSERT` or `DELETE` | | compartment\_id | String (Optional) | The Compartment ID, acting as a user [identifier ](/user-points#user-identifiers)in correlation with `user_account_id` | | user\_account\_id | String (Optional) | The User Account ID, acting as an [identifier](/user-points/user-profiles#user-profile-object) in correlation with `compartment_id` | | email\_hash | String (Optional) | The Email Hash, acting as an [identifier](/user-points#user-identifiers) | | user\_agent\_id | String (Optional) | The User Agent ID, acting as an [identifier](/user-points#user-identifiers) | | force\_replace | Boolean (Optional) |

Mandatory when the operation is UPSERT.

If true, then the User Choice will be completely replaced by the object passed in the user\_choice field.


If false, the object passed in the user\_choice field will be merged with the existing User Choice of the UserPoint.

| | user\_choice | JSON Object (Optional) |

Mandatory when operation is UPSERT.

This is a JSON Object representing the User Choice.

Please refer to the User choices page for more

information.

Note that the $processing\_id field is always mandatory, and $choice\_ts is mandatory when operation is UPSERT.

| ### Example ```bash # Create the document import curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_CHOICE", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ```bash # Create the execution curl -X POST \ https://api.mediarithmics.com/v1/datamarts/1162/document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d '{ "operation": "UPSERT", "compartment_id": "", "user_account_id": "", "force_replace": true, "user_choice": { "$processing_id": "", "$choice_ts": "" } }' ``` {% hint style="success" %} You can, of course, upload multiple user choices at once. Note the uploaded data is in `ndjson` format and not `json`. That means the different choices are not separated by commas, but by a line separator `\n` {% endhint %} {% hint style="warning" %} When importing choices with identifiers, only one identifier is allowed per line. For example, you shouldn't specify the user agent ID if the Email Hash is already used in a line. {% endhint %} # Segments import Use this feature to add or remove [UserPoint](/user-points) from [segments](/user-points/segments). ## How-to 1. Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the`USER_SEGMENT`document type and `APPLICATION_X_NDJSON` or `TEXT_CSV` mime type. 2. Create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your user segment commands formatted in `ndjson` or `csv` depending on the mime type you chose. ### User segment command Each line in the uploaded file can have the following properties: | field | type | description | | ------------------------ | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | operation | Enum (Mandatory) | Either `UPDATE` or `DELETE` | | compartment\_id | String (Optional) | The Compartment ID acting as a [user identifier](/user-points#user-identifiers) in correlation with the user account ID | | user\_account\_id | String (Optional) | The User Account ID acting as a [user identifier](/user-points#user-identifiers) in correlation with the user account ID. | | email\_hash | String (Optional) | The Email Hash acting as a [user identifier](/user-points#user-identifiers). | | user\_agent\_id | String (Optional) | The User Agent ID acting as a [user identifier](/user-points#user-identifiers). | | segment\_id | String (Optional) | The Id of the segment in which the User is inserted/deleted. | | segment\_technical\_name | String (Optional) | The technical name of the segment in which the User is inserted/deleted. | | expiration\_duration | Integer (Optionnal) | The number of minutes before the user will be removed from the segment. 0 means that the User will never leave the segment | | expiration\_ts | Number (Optional) | The timestamp of the expiration date of the User in the segment. A value of 0 means that the user will never leave the segment | | data\_bag | Escaped JSON String (Optional) | The data bag associated with the user/segment relationship | {% hint style="info" %} **Information about the fields to be provided:** * At least one identifier must be provided: either `compartment_id` and `user_account_id`, `email_hash`, or `user_agent_id`. * Although `expiration_duration` and `expiration_ts` are individually optional, at least one of them must be provided. * Although `segment_id` and `segment_technical_name` are individually optional, at least one of them must be provided. {% endhint %} ### Example ```bash # Create the document import curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_SEGMENT", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ```bash # Create the execution curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d '{ "operation": "UPDATE", "expiration_duration": , "compartment_id": "", "user_account_id": "", "segment_id": "" }' ``` {% hint style="success" %} You can have UPDATE and DELETE operations in the same file upload. Please note, if not using `csv`, that the uploaded data is in `ndjson` and not `json`. That means the different profiles are not separated by commas, but by a line separator `\n` {% endhint %} {% hint style="success" %} Use segment\_ids or segment\_technical\_names if you need to handle multiple segments for a single user. {% endhint %} # Deletions # User identifiers deletion This document import allows you to mark [user identifiers](/user-points#user-identifiers) for deletion. Each line in the document represents a different object to remove from the platform. {% hint style="info" %} This is only supported for datamarts using a UserPoint system version of `v201901` or later. **It only deletes the user identifier but not the associated UserPoint.** {% endhint %} ## How-to 1. Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the`USER_IDENTIFIERS_DELETION`document type and `APPLICATION_X_NDJSON` mime type. Only `ndjson` data is supported for user activities. 2. Create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your commands formatted in `ndjson`. Each command can either be a user account deletion, a user email deletion or a user agent deletion. ### Example ```bash # Create the document import curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_IDENTIFIERS_DELETION", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ```bash # Create the execution curl -X POST \ https://api.mediarithmics.com/v1/datamarts/1162/document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d ' { "type": "USER_ACCOUNT", "compartment_id": "1000", "user_account_id": "8541254132" } { "type": "USER_EMAIL", "hash": "982f50d88d437d13bdbd541edfv4fe5176cc8d862f8cbe7ca4f0dc8ea" } { "type": "USER_AGENT", "user_agent_id": "vec:89998434" } ' ``` {% hint style="success" %} You can, of course, remove different identifier types at the same time. Please note that the uploaded data is in `ndjson` and not `json`. That means the different deletions are not separated by commas, but by a line separator `\n` {% endhint %} ## User Account deletion command | field | type | description | | ----------------- | ----------------- | ------------------------------------------------------- | | type | String | USER\_ACCOUNT | | user\_account\_id | String | The User Account Id. | | compartment\_id | Number (Optional) | The Compartment Id associated with the User Account Id. | ### **Example:** ```javascript { "type": "USER_ACCOUNT", "compartment_id": "1000", "user_account_id": "8541254132" } ``` ## User Email deletion command | field | type | description | | ----- | ------ | -------------------------- | | type | String | USER\_EMAIL | | hash | String | Hash of the Email address. | ### Example: ```javascript { "type": "USER_EMAIL", "hash": "982f50d88d437d13bdbd541edfv4fe5176cc8d862f8cbe7ca4f0dc8ea" } ``` ## User Agent deletion command | field | type | description | | --------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | type | String | USER\_AGENT | | user\_agent\_id | String | A [user agent id](https://developer.mediarithmics.io/user-points/user-identifiers/device-identifiers#user_agent_id) other than a device point id. Ex: "vec:123456" or "net:9:12345". | {% hint style="warning" %} User device point ids ("udp:123456") are not supported by this document import job and will cause the job to be rejected. Use the [Device points deletion](/data-ingestion/bulk-processing/deletions/device-points-deletion)job type if you want to delete device points along with all their associated technical ids. {% endhint %} \ Example: ```javascript { "type": "USER_AGENT", "user_agent_id": "vec:89998434" } ``` # Device points deletion This document import allows you to mark device points for deletion. Each line in the document is a user agent identifier that is linked to a device point (or is a device point id directly). For each of those identifiers, the job will find its associated device point and delete it. Note that a device point may be linked to multiple user agent identifiers, all of which will be deleted once their device point is (including user agent identifiers that may not appear in your document). {% hint style="info" %} This is only supported for datamarts using a UserPoint system version of `v202205` or later. **It only deletes the device point and its identifiers but not the associated UserPoint.** {% endhint %} ## How-to 1. Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the `USER_DEVICE_POINTS_DELETION` document type and `APPLICATION_X_NDJSON` mime type. Only `ndjson` data is supported. 2. Create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your commands formatted in `ndjson`. Each line represents a user agent. ### Example ``` # Create the document import curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_DEVICE_POINT_DELETION", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ``` # Create the execution curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d ' { "type": "USER_AGENT", "user_agent_id": "vec:89998434" } { "type": "USER_AGENT", "user_agent_id": "udp:987654" } ' ``` Please note that the uploaded data is in `ndjson` and not `json`. That means the different deletions are not separated by commas, but by a line separator. A list of possible `user_agent_id` values can be found at [UserDeviceTechnicalId](/user-points/user-identifiers/userdevicetechnicalid#user_agent_id) {% hint style="info" %} If you want to pass device point ids directly ("udp:987654") you must do so using the `USER_AGENT` type. {% endhint %} # UserPoint deletion This document import allows you to mark UserPoint for deletion. Each line in the document is a user identifier that is linked to a UserPoint (or is a UserPoint ID directly). For each of those identifiers, the job will find its associated UserPoint and delete it, along with all of its identifiers, segments, scenario and its profile . ## How-to 1. Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the `USER_POINTS_DELETION` document type and `APPLICATION_X_NDJSON` mime type. Only `ndjson` data is supported. 2. Create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your commands formatted in `ndjson`. Each line can represent either a user agent, a user email, a user account or a UserPoint ID. Their respective syntax is detailed in [User identifiers deletion](/data-ingestion/bulk-processing/deletions/identifiers). ### Example ``` # Create the document import curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_POINTS_DELETION", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ``` # Create the execution curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d ' { "type": "USER_ACCOUNT", "compartment_id": "1000", "user_account_id": "8541254132" } { "type": "USER_EMAIL", "hash": "982f50d88d437d13bdbd541edfv4fe5176cc8d862f8cbe7ca4f0dc8ea" } { "type": "USER_AGENT", "user_agent_id": "vec:89998434" } { "type": "USER_AGENT", "user_agent_id": "udp:987654" } { "type": "USER_POINT", "user_point_id": "95772db3-e762-45da-8cf1-4893debffae7" } ' ``` Please note that the uploaded data is in `ndjson` and not `json`. That means the different deletions are not separated by commas, but by a line separator. A list of possible `user_agent_id` values can be found at [UserDeviceTechnicalId](/user-points/user-identifiers/userdevicetechnicalid#user_agent_id) # User identifiers association This document import allows you to merge UserPoint by associating their [user identifiers](/user-points#user-identifiers). Each line in the document represents a different user identifiers association {% hint style="info" %} This is only supported for datamarts using a UserPoint system version of `v201812` or later. {% endhint %} ## How-to 1. Use the [bulk import ](/data-ingestion/bulk-processing)endpoints to create a [document import](/data-ingestion/bulk-processing#document-import) with the`USER_IDENTIFIERS_ASSOCIATION_DECLARATIONS`document type and `APPLICATION_X_NDJSON` mime type. Only `ndjson` data is supported for user activities. 2. Create an[ execution ](/data-ingestion/bulk-processing#create-an-execution)with your commands formatted in `ndjson` . ### User identifiers association command Each line will create/merge a UserPoint that has all the specified identifiers | field | type | description | | ----------- | ------------------------- | ------------------------------------------------ | | identifiers | UserIdentifierResource\[] | An array of User Identifier Resource of any type | User identifier resource can be of three shapes. Either email or user agent or user account id. They correspond with the different types of [user identifiers](/user-points#user-identifiers). #### Email | field | type | description | | ----- | ----------------- | ------------------------------------------------------------------------ | | type | "USER\_EMAIL" | The type of the identifier. | | hash | String | A hash of the email. The hashing function should be unique per datamart. | | email | String (optional) | the email address | #### User Agent | field | type | description | | --------------- | ------------- | --------------------------- | | type | "USER\_AGENT" | The type of the identifier. | | user\_agent\_id | String | The user agent ID | #### User Account | field | type | description | | ----------------- | ----------------- | ---------------------------------------------------------------------------------------------- | | type | "USER\_ACCOUNT" | The type of the identifier. | | user\_account\_id | String | The User Account ID | | compartment\_id | String (optional) | The Compartment ID. If you don't input the compartment id it will fall back on the default one | ### Example ```bash # Create the document import curl -X POST \ https://api.mediarithmics.com/v1/datamarts//document_imports \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "document_type": "USER_IDENTIFIERS_ASSOCIATION_DECLARATIONS", "mime_type": "APPLICATION_X_NDJSON", "encoding": "utf-8", "name": "" }' ``` ```bash # Create the execution curl -X POST \ https://api.mediarithmics.com/v1/datamarts/1162/document_imports//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/x-ndjson' \ -d ' { "identifiers":[ { "type": "USER_EMAIL", "hash":"" }, { "type": "USER_AGENT", "user_agent_id": "" }, { "type": "USER_ACCOUNT", "user_account_id": "", "compartment_id": "" } ] } ' ``` {% hint style="success" %} You can, of course, add different identifier types at the same time. Please note that the uploaded data is in `ndjson` and not `json`. That means the different additions are not separated by commas, but by a line separator `\n` {% endhint %} # Integration batch The integration batch is a plugin type used for customers' integrations. It can be periodic or non-periodic plugin. You can choose to pause a recurring plugin so that all the coming executions are canceled. {% hint style="info" %} You can check [plugins key concepts](/advanced-usages/plugins#key-concepts) to understand the hierarchy between a plugin, the versions, the instances and the executions. {% endhint %} Imagine you want to create a script that imports data every night for a customer : * You declare a new integration batch plugin called `import-data-for-customer` * You declare a first `1.0.0` version for this plugin with the code of the script and the declaration of the script parameters * Your script is now available for usage To execute the script, you can : * Create an integration batch instance that will use the code from the 1.0.0 version with specific input parameters * Either program the instance to automatically create executions at a specified cron, or manually create a new execution to start now or later. ## How-to ### Plugin creation Use the [plugin creation](https://developer.mediarithmics.com/guides/plugins/deployment#create-the-plugin-and-its-version) endpoints to create a new plugin with the plugin type as `INTEGRATION_BATCH`. Everything else remains the same. ### Example ```bash # Create the plugin definition curl -X POST \ https://api.mediarithmics.com/v1/plugins \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "organisation_id": "my_organisation_id", "plugin_type": "INTEGRATION_BATCH", "group_id": "com.my-client.integration_batch", "artifact_id": "integration-batch-my-client" }' ``` ### Plugin version creation Use the [plugin version creation](https://developer.mediarithmics.com/guides/plugins/deployment#create-the-plugin-and-its-version) endpoints to create a new plugin version with all the properties. The call and format are the same than usual. ### Example ```bash # Create the plugin version curl -X POST \ https://api.mediarithmics.com/v1/plugins//versions \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "version_id":"1.0.0", "plugin_properties":[ { "technical_name": "one_more_property", "value": { "value": "" }, "property_type": "STRING", "origin": "PLUGIN", "writable": true, "deletable": true }, { "technical_name": "provider", "value": { "value": "mediarithmics" }, "property_type": "STRING", "origin": "PLUGIN_STATIC", "writable": false, "deletable": false }, { "technical_name": "name", "value": { "value": "My Plugin Name" }, "property_type": "STRING", "origin": "PLUGIN_STATIC", "writable": false, "deletable": false } ] }' ``` ### Plugin instance creation For the integration batch plugin, the instance is called `integration_batch_instances`. There are five properties that are used for this plugin type: `cron` , `cron_status`, `ram_size`, `cpu_size` and `disk_size`. The `cron` and the `cron_status` are not mandatory as you can create non-periodic jobs. If used, you should use them together. The `ram_size` , `cpu_size` and `disk_size` are mandatory and the default values are set to `LOW`. ### Example ```bash # Create the plugin instance curl -X POST \ https://api.mediarithmics.com/v1/integration_batch_instances \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "group_id": "com.my-client.integration_batch", "artifact_id": "integration-batch-my-client", "version_id": "my-version-id", "organisation_id": "my-plugin-id", "name": "The name of my instance", "archived": false, "cron" :"* * * 7 *", "cron_status": "ACTIVE | PAUSED", "ram_size": "LOW | MEDIUM | LARGE | EXTRA_LARGE", "disk_size": "LOW | MEDIUM | LARGE | EXTRA_LARGE", "cpu_size": "LOW | MEDIUM | LARGE | EXTRA_LARGE" }' ``` You can perform the operations POST / PUT / GET and DELETE on the instances. ### Integration batch execution creation Executions can be created either automatically by the scheduler using the cron defined in the instance or manually using the API or the interface. When creating an execution you have to set the `execution_type` and `expected_start_date` properties. ``` // Create an execution curl -X POST \ https://api.mediarithmics.com/v1/integration_batch_instances//executions \ -H 'Authorization: ' \ -H 'Content-Type: application/json' \ -d '{ "parameters": { execution_type: "MANUAL | CRON", expected_start_date: 1562595783663 }, "organisation_id": "1185", "user_id": "1007", "error": null, "status": "PENDING", "external_model_id: "42", "external_model_name": "PUBLIC_INTEGRATION_BATCH", "start_date": 1562595789171, "job_type": "BATCH_INTEGRATION", } ``` You can perform the operations POST / PUT / GET and DELETE on the executions. The execution\_type can be either `MANUAL` when created using the interface or `CRON` when created by the instance using the cron value set in the instance. The expected\_start\_date is set by the timestamp chosen in the interface or by the cron set in the instance. # Activities analytics Data coming into your datamart is stored in a multi-model database, optimizing it for different usages. **To display performance analytics for elements like session duration, conversions,** and **funnel, the platform duplicates some** [**user activities**](/user-points/user-activities) **and information and optimizes them.** {% hint style="success" %} For that purpose, it is important that you use [predefined event names and properties](/user-points/user-activities#predefined-event-names) when possible. Custom events won't be taken into account when calculating metrics. For example, don't create `order` events when tracking an e-commerce site, but the [predefined](/data-ingestion/real-time-user-tracking/web-events#transaction-confirmation-page) `$transaction_confirmed` [event](/data-ingestion/real-time-user-tracking/web-events#transaction-confirmation-page). `$transaction_confirmed` events are used when calculating conversions and amounts but not `order` events. {% endhint %} Here is a sample event that can be used in analytics: ```javascript { "$ts": 3489009384393, "$event_name": "$transaction_confirmed", // Conversion detected "$properties": { "$items": [ { "$id": "product_ID", // Used to filter in funnel analytics "$qty": 20, // Used for conversion amounts "$price": 102.8, // Used or conversion amounts "$brand": "Apple" // Used to filter in funnel analytics "$category1": "Category 1", // Used to filter in funnel analytics "$category2": "Category 2", // Used to filter in funnel analytics "$category3": "Category 3", // Used to filter in funnel analytics "$category4": "Category 4" // Used to filter in funnel analytics }, { "$id": "product_ID2", "$qty": 12, "$price": 3.4, "$brand": "Microsoft" } ], "$currency": "EUR" } } ``` The list of predefined events that are used in analytics are as follows. | Event name | Usage | Important properties | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $transaction\_confirmed |

Home dashboards (E-Commerce Engagement)

Segment dashboards (E-Commerce Engagement)

Funnel Analytics

|

$items : list of products in the transaction

$items.$qty : for the conversion amounts

$items.$price: for the conversion amounts

$items.$id : for the product IDs in funnel analytics

$items.$brand: for the brand filter in funnel analytics

$items.$category1, $items.$category2, $items.$category3 and $items.$category4 : for the categorization in funnel analytics

| | $item\_view | Funnel Analytics |

$items : contains only one product

$items.$price: for the conversion amounts

$items.$id : for the product IDs in funnel analytics

$items.$brand: for the brand filter in funnel analytics

$items.$category1, $items.$category2, $items.$category3 and $items.$category4 : for the categorization in funnel analytics

| | $basket\_view | Funnel Analytics |

$items : list of products in the basket

$items.$qty : for the conversion amounts

$items.$price: for the conversion amounts

$items.$id : for the product IDs in funnel analytics

$items.$brand: for the brand filter in funnel analytics

$items.$category1, $items.$category2, $items.$category3 and $items.$category4 : for the categorization in funnel analytics

| ## Activities analytics data retention Activities analytics data is kept **4 month** in order to optimize performances. ## Transforming important events into your custom events While it is better to use predefined events when possible, it isn't always the best solution for you. To keep having analytics correctly stored, you can transform your custom events to predefined ones. {% hint style="info" %} Only events coming into the datamart after an event transformation has been defined will be transformed. Those transformations don't transform user activities and events anywhere else. You should use [activity analyzers](/data-ingestion/real-time-user-tracking/activity-analyzers) to transform them everywhere. {% endhint %} ### Event transformations An event transformation is linked to a datamart. Here is a sample transformation: ```javascript { "id": 15, // Read only "datamart_id": "3333", // Read only "created_ts": <>, // Read only "created_by": <>, // Read only "last_modified_by": <>, // Read only "last_modified_ts": <>, // Read only "channel_id": "8888", "source_event_name": "order", "target_event_name": "$transaction_confirmed", "mapping_id": 15 } ``` | Property | Type | Description | | ------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | datamart\_id | Integer | The ID of the datamart where the transformation is applied | | channel\_id | Integer | The ID of the channel where the transformation is applied | | source\_event\_name | String | The name of your custom event you wish to transform into a predefined event | | target\_event\_name | String |

The name of the predefined event it is transformed into. Allowed values are :

  • $transaction\_confirmed
  • $item\_view
  • $basket\_view
  • $list\_item\_view

To learn about predefined events, see Predefined event names

| | mapping\_id | Integer | The ID of the [Property mapping](/data-ingestion/activities-analytics#property-mapping) to apply when transforming the event | ### Property mappings Event transformations use property mappings to choose which property in your custom event becomes which property in the predefined event. Here is a sample property mapping to start with for `$transaction_confirmed` events. ```javascript // Comments are here to help you understand, // remove them before uploading mappings as they are not accepted in JSON { "id": 17, // Read only "created_by": <>, // Read only "created_ts": <>, // Read only "mapping": { "values": [ { // Maps $properties.order.order_products[] to $items[] "target_attribute_name": "$items", "source_attribute_path": { "attribute_name": "$properties", "sub_path": { "attribute_name": "order", "sub_path": { "attribute_name": "order_products" } } }, // Alternative : mapping directly $properties if there are no multiple // elements in the event. It will be treated as an array of only one item "target_attribute_name": "$items", "source_attribute_path": { "attribute_name": "$properties" }, // Maps $properties.order.order_products.xxx to $items.xxx // You can't use other target attributes than the ones // in this example. // But they are not all mandatory : a non used target // attribute will use the default value "children": [ // $properties.order.order_products.product.id // becomes $items.$id // to be used by analytics database { "target_attribute_name": "$id", "source_attribute_path": { "attribute_name": "product", "sub_path": { "attribute_name": "id" } } }, // $properties.order.order_products.qty // becomes $items.$qty // to be used by analytics database { "target_attribute_name": "$qty", "source_attribute_path": { "attribute_name": "qty" } }, { "target_attribute_name": "$price", "source_attribute_path": { "attribute_name": "price" } }, { "target_attribute_name": "$ean", "source_attribute_path": { "attribute_name": "ean", } }, { "target_attribute_name": "$brand", "source_attribute_path": { "attribute_name": "brand", } }, { "target_attribute_name": "$category1", "source_attribute_path": { "attribute_name": "cat1" } }, { // In this example $category2 will be // event.$event_name instead of // $properties.order.order_products[].$event_name // thanks to absolute_path "target_attribute_name": "$category2", "source_attribute_path": { "absolute_path": true, "attribute_name": "$event_name", } }, { "target_attribute_name": "$category3", "source_attribute_path": { "attribute_name": "cat3" } }, { "target_attribute_name": "$category4", "source_attribute_path": { "attribute_name": "cat4" } } ] } ] } } ``` {% hint style="warning" %} Map events JSON as it is stored in the database and visible in user's timelines (post [activity analyzers](/data-ingestion/real-time-user-tracking/activity-analyzers)). {% endhint %} ## Get existing transformations `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/analytics_event_transformation` #### Path Parameters | Name | Type | Description | | ---------- | ------- | --------------- | | datamartId | integer | The datamart ID | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "datamart_id": ":datamartId", "channel_id": "8888", "source_event_name": "order", "target_event_name": "$transaction_confirmed", "mapping_id": xxx }, { ... } ] } ``` {% endtab %} {% endtabs %} ## Create an event transformation `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/analytics_event_transformation` #### Path Parameters | Name | Type | Description | | ---------- | ------- | --------------- | | datamartId | integer | The datamart ID | #### Request Body | Name | Type | Description | | ---- | ------ | ------------------------------- | | Body | string | The event transformation object | {% tabs %} {% tab title="200 " %} ```javascript ``` {% endtab %} {% endtabs %} ## Update an event transformation `PUT` `https://api.mediarithmics.com/v1/datamarts/:datamartId/analytics_event_transformation/:transformationId` #### Path Parameters | Name | Type | Description | | ---------------- | ------- | --------------------- | | transformationId | integer | The transformation ID | | datamartId | integer | The datamart ID | #### Request Body | Name | Type | Description | | ---- | ------ | ------------------------------- | | Body | string | The event transformation object | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Remove an event transformation `DELETE` `https://api.mediarithmics.com/v1/datamarts/:datamartId/analytics_event_transformation/:transformationId` #### Path Parameters | Name | Type | Description | | ---------------- | ------- | --------------------- | | transformationId | integer | The transformation ID | | datamartId | integer | The datamart ID | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Get property mappings `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/analytics_mapping` #### Path Parameters | Name | Type | Description | | ---------- | ------- | --------------- | | datamartId | integer | The datamart ID | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Create property mappings `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/analytics_mapping` #### Path Parameters | Name | Type | Description | | ---------- | ------- | --------------- | | datamartId | integer | The datamart ID | #### Request Body | Name | Type | Description | | ---- | ------ | --------------------------- | | Body | object | The property mapping object | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { id: 17, created_by: <> ... } } ``` {% endtab %} {% endtabs %} ## Delete property mappings `DELETE` `https://api.mediarithmics.com/v1/datamarts/:datamartId/analytics_mapping/:mappingId` #### Path Parameters | Name | Type | Description | | ---------- | ------- | --------------- | | mappingId | integer | The mapping ID | | datamartId | integer | The datamart ID | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok" } ``` {% endtab %} {% tab title="400 " %} ```javascript { "status": "error", "error": "Deleting a property mapping used in a transformation is forbidden", "error_code": "BAD_REQUEST_DATA", "error_id": "9460be23-0f80-4acb-9a33-dd8a1c5d08a9" } ``` {% endtab %} {% endtabs %} # Data warehouse ingestion {% hint style="info" %} This feature is in Alpha, please contact your Account Manager to learn more. {% endhint %} ## Pre-requisite You first need to declare your data warehouse as a data source in mediarithmics as detailed in the [data warehouse management section](/advanced-usages/data-warehouse-management). ## Synchronization strategies When trying to ingest data from a data warehouse to mediarithmics, you will need to configure a service account with specific rights on the data you want to expose : the service account will need to at least have read access on the data. #### Working dataset/schema If you want to use synchronization strategies that can handle updates and deletes on existing data, you will need to create a **working dataset/schema**. On this dataset/schema, the service account should have editing rights : this will be used to create temporary tables and track changes on your tables. #### Available strategies * **Profile data** * *Timestamp based* : all updates in the source table (insert, update, delete) are impacted in mediarithmics. This requires a timestamp column tracking the last update of each line of the table. * *Hash table* : all updates in the source table (insert, update, delete) are impacted in mediarithmics. A temporary table is created based on a hash of columns of the source table. * *Full import* : every synchronization will import the full table. It does not handle deletes or updates, this is a append-only strategy. * **Event data** * *Timestamp based* : import all events that occurred since the last import. This requires a timestamp column tracking when each event of the table has occurred. It does not handle deletes or updates, this is a append-only strategy. {% hint style="info" %} At the time of the synchronization, if the source table for events contains data that will be impacted by active [cleaning rules](/advanced-usages/data-privacy-compliance/cleaning-rules) on events for the datamart, then these events will not be imported. {% endhint %} #### Summary Without a working dataset/schema only append-only strategies are available : timestamp based for event data, and full import for profile data.
StrategyNeeds a working dataset/schema
Timestamp basedEvents : No
Profile : Yes
Full importNo
Hash tableYes
# Querying your data You have three different ways to query your data in mediarithmics depending on what you want to achieve. GraphQL\ \ ------------------------------------------------------------------------------- The [GraphQL API](/querying-your-data/graphql-queries) gives you the ability to query all kinds of UserPoint data, as collected by the platform (profile, activities, segments, identifiers, etc.) **with a single request**. ## OTQL The [OTQL Language](/querying-your-data/otql-queries) is an extension to GraphQL. It is based on the same schema and **adds unique features to query a graph of millions/billions of objects**. ## Analytics
We provide an [analytics data cube to query user activities](/querying-your-data/activities-analytics-queries). # OTQL queries This page provides a formal description of OTQL capabilities. ## Introduction The Object Tree Query Language (OTQL) has been designed to help to search and calculate aggregates on a large collection of object trees. The object tree is defined[ in the schema](/schema) using the [@TreeIndexRoot](/schema#treeindexroot) and the [@TreeIndex ](/schema#treeindex)directives. {% hint style="success" %} OTQL queries help you : * Build segments * Explore data * Monitor data integration * Check the volumetry * Build data exports * ... {% endhint %} ![](/files/-MROjMB-0j3Dtp5MH1rD) An OTQL query looks like an SQL query. ```sql SELECT { id name } FROM Product WHERE price > 50.0 ``` It is composed of three parts: * A `SELECT` **Operation**: It gives indications on what needs to be done: extracting field values or calculating aggregates * A `FROM` **starting Object Type**: It defines the starting object type in the evaluation process * A `WHERE` **Object Tree Expression:** It defines a logical expression mixing boolean operators and field operators to connect different objects in the object tree. ```sql SELECT [Operation] FROM [Object Type] WHERE [Object Tree Expression] ``` There are two kinds of operations: * **Selection Operations** are similar to a GraphQL operation and return a list of objects containing the required fields. * **Aggregation Operations** return aggregated values (count, stats, histograms, ...) calculated on the selected objects. ![](/files/-MROk2B2t0scoP3lO8mz) {% hint style="info" %} The fields in the queries are completely related to [how you built your schema](/schema). {% endhint %} Here are some examples of requests you can do with OTQL : ```sql # Counts number of new users in the past 7 days SELECT @count{} FROM UserPoint WHERE creation_date >= "now-7d/d" # Counts the number of transactions on a specific site (channel) 7 days ago SELECT @count{} FROM UserEvent WHERE name = "$transaction_confirmed" AND date = "now-7d/d" AND channel_id = 2419 # Counts the number of profiles with female gender SELECT @count{} FROM UserProfile WHERE gender = "W" # Lists all categories from universes in events done on a specific channel SELECT { universe { category @map }} FROM UserEvent WHERE channel_id = 2417 # Lists all event names collected in the platform SELECT {name @map} FROM UserEvent # Number of users having at least 3 events related to laptops in the past 15 days SELECT @count{} FROM UserPoint WHERE activities { events @ScoreSum(min:3) { category = "Laptop" AND date >= "now-15d/d" }} # Number of transactions per site and per day SELECT { channel_id @map { date @date_histogram } } FROM UserEvent WHERE name = "$transaction_confirmed" # Number of users having an account but no emails SELECT @count{} FROM UserPoint where accounts{} and not emails{} ``` ## FROM - Starting object type Imagining the following Object Tree: ```undefined UserPoint ├── UserActivity │ └── UserEvent ├── UserEmail └── UserAccount ``` You could build queries starting from all `UserPoint`, all `UserActivity`, `UserEvent`, `UserEmail` or `UserAccount` ```sql # Selects all names from all UserPoint SELECT {name} FROM UserPoint # Selects all names from all UserActivity SELECT {name} FROM UserActivity # Equivalent of SELECT {activities { name }} FROM UserPoint # Selects all names from all UserEvent SELECT {name} FROM UserEvent # Equivalent of SELECT { activities { events { name }}} FROM UserPoint ``` {% hint style="info" %} You target object types with the FROM. In the example, the link field is activities, but we don't do SELECT (...) FROM activities. {% endhint %} ## WHERE - Object tree expressions The expression contained in the `WHERE` clause is composed of a list of predicates, separated by logical operators `AND`, `OR`, and `NOT` . Parenthesis can be used to group together two predicates with a logical operator. Examples : ```sql SELECT (...) FROM (...) WHERE (PredicateA AND PredicateB) OR PredicateC SELECT (...) FROM (...) WHERE PredicateA AND (PredicateB OR PredicateC) ``` ```sql SELECT (...) FROM (...) WHERE price > 50.0 ``` ```sql SELECT (...) FROM (...) WHERE price > 50.0 AND last_modified_date > "now-10d" ``` ### Logical operators Each predicate doesn't return directly a boolean but a score, 1 if the condition is respected else 0. At the end the score is compared to 0 and return true if it's higher than 0 else return false. These operator keep the same priority as boolean ones. The logical operators work as below : | Logical operator | Real operation | Priority | | ----------------------------- | ------------------------------ | -------- | | **NOT** PredicateA | if ( ScoreA > 0) then 0 else 1 | high | | PredicateA **AND** PredicateB | ScoreA x ScoreB | middle | | PredicateA **OR** PredicateB | max(ScoreA, ScoreB) | low | ```sql SELECT (...) FROM (...) WHERE price > 50.0 AND last_modified_date > "now-10d" # ( price > 50.0 ) x ( last_modified_date > "now-10d" ) SELECT (...) FROM (...) WHERE price > 50.0 AND last_modified_date > "now-10d" OR price > 100.0 AND last_modified_date > "now-20d" # (( price > 50.0 ) x ( last_modified_date > "now-10d" )) + (( price > 100.0 ) x ( last_modified_date > "now-20d" )) ``` ### Tree exploration As we are querying an Object Tree, and as predicates are only possible on a **leaf** (e.g. fields that only contain a scalar value), it is natural to have a way of **going from the root to each of the leaves by traversing the tree.** Braces symbols `{}` are used to traverse the tree through **link fields**. The sub-query written in the braces will be evaluated on each item in the linked list. **Let's see it in action.** Let's say we built a schema corresponding to the following Object Tree. ```graphql # Functional tree UserPoint └─ UserActivity └─ UserEvent # Associated schema type UserPoint @TreeIndexRoot(index:"USER_INDEX") { # activities is a link field to UserActivity objects activities: [UserActivity] } type UserActivity { # events is a link field to UserEvent objects events: [UserEvent] } type UserEvent { name: String @TreeIndex(index_name: "USER_INDEX") amount: Int @TreeIndex(index_name: "USER_INDEX") date: Timestamp! } # A UserPoint can have 0..n User Activity # Each UserActivity can have 0..m UserEvent # UserEvent has a "name" String field, an "amount" Int field and a "date" Timestamp field ``` The following query will return all the `UserPoint` that have at least one `UserEvent` whose name is `$transaction_confirmed` . As the root of the Object Tree is the `UserPoint` in this example, we'll need to start from there. And then follow the `activities` link to the associated `UserActivity` and then the `events` link to the associated `UserEvent`. ```sql SELECT (...) FROM UserPoint WHERE activities { events { name = "$transaction_confirmed" } } ``` ### Scoring operator The latest query returns items with at least one of the events have a `$transaction_confirmed` name. We return every user that did at least one purchase and at least one visit. If we instead want the users that bought things through at least 3 different visits (frequent buyers), we will use a **scoring operator**. Each time there is a pair of braces `{ }` and a sub-query written in the braces, there is implicitly a score calculated for the sub-query. **By default, the score will be the number of items matching the sub-query**. {% tabs %} {% tab title="@ScoreSum" %} Only returns score if the nested sub-query has a score superior or equal to min. | args | description | | ------------- | -------------------------------------------------------------- | | min : `Float` | minimum required score for the nested sub-query to be returned | ```sql # Select UserPoint that bought things through at least 3 different visits SELECT (...) FROM UserPoint WHERE activities @ScoreSum(min: 3.0){ events { name = "$transaction_confirmed" } } # Select UserPoint that have at least 1 activity that contains at least 3 $transaction_confirmed events SELECT (...) FROM UserPoint WHERE activities { events @ScoreSum(min: 3.0) { name = "$transaction_confirmed" } } ``` {% endtab %} {% tab title="@ScoreAvg" %} Calculate the average from the sub-query matching scores and returns true if it's superior or equal to min. | args | description | | ------------- | -------------------------------------------------------------- | | min : `Float` | minimum required score for the nested sub-query to be returned | {% hint style="warning" %} **By default, the score will be the number of items matching the sub-query**. So using `@ScoreMax` or `@ScoreAvg` like that is useless because each score from the sub-query will be 1. It's why you should apply a modification on the score calculation. {% endhint %} {% endtab %} {% tab title="@ScoreMax" %} Takes the maximum score from the sub-query matching scores and returns true if it's superior or equal to min. | args | description | | ------------- | -------------------------------------------------------------- | | min : `Float` | minimum required score for the nested sub-query to be returned | {% hint style="warning" %} **By default, the score will be the number of items matching the sub-query**. So using `@ScoreMax` or `@ScoreAvg` like that is useless because each score from the sub-query will be 1. It's why you should apply a modification on the score calculation. {% endhint %} {% endtab %} {% endtabs %} #### Changing the way scores are calculated As said above, by default, score values are equal to the number of items matching a sub-query when following a link. However, your Object Tree leaves have some number typed fields (`Int` or `Float`). It is possible to use those values as the score of a sub-query. {% tabs %} {% tab title="@ScoreField" %} Select a specific field in which the numeric value used as the score is stored. | args | description | | -------------- | ------------------------------ | | name: `String` | The name of the field selected | ```sql # Using @ScoreField alone is useless because it could be replace by logical operator SELECT (...) FROM UserPoint WHERE activity { events @ScoreField(name:"amount") { name = "$transaction_confirmed" } } # Can be written SELECT (...) FROM UserPoint WHERE activity { events { name = "$transaction_confirmed" } } ``` The information of which field is selected bubble up still it didn't catch by a `@ScoreSum`, `@ScoreAvg` or `@ScoreMax` . ```sql # Select UserPoint having spent at 1000 in one event SELECT (...) FROM UserPoint WHERE activity { events @ScoreField(name:"amount") @ScoreSum(min : 1000) { name = "$transaction_confirmed" } } # Select UserPoint having spent at 1000 in one activity SELECT (...) FROM UserPoint WHERE activity @ScoreSum(min : 1000) { events @ScoreField(name:"amount") { name = "$transaction_confirmed" } } ``` {% hint style="danger" %} Be sure sub-field selected in @ScoreField exist in any field. Add a condition if the query return an error ```sql SELECT (...) FROM UserPoint WHERE activity @ScoreSum(min : 1000) { events @ScoreField(name:"amount") { name = "$transaction_confirmed" } } # To be sure field "amount" exist SELECT (...) FROM UserPoint WHERE activity @ScoreSum(min : 1000) { events @ScoreField(name:"amount") { is_defined(amount) AND name = "$transaction_confirmed" } } ``` {% endhint %} {% endtab %} {% tab title="@ScoreBoost" %} Multiply the score by the factor. Can be used to boost a sub-query over another one. | args | description | | --------------- | --------------------------------------- | | factor: `Float` | Constant float which multiply the score | ```sql # Select UserPoints having spent at least 1000 orders, where IT product count twice SELECT (...) FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreBoost(factor: 2.0) @ScoreSum(result: "score_value") { category="IT" }, order_products @ScoreSum() { category!="IT" } } } ``` {% endtab %} {% endtabs %} #### Using this calculated score With the possibility to use score in a field, you may want to return the calculated score of a `@ScoreSum` and not only it a sub-query validate the condition or not. This is why we add a new parameters to `@ScoreSum` :`result`. {% tabs %} {% tab title="@ScoreSum" %} Only returns score if the nested sub-query has a score superior or equal to min. | args | description | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | min : `Float` | minimum required score for the nested sub-query to be returned | | result : `String` |

Two values possibles :

  • "boolean\_value" (by default) : reduce the returned score value to 0 or 1. It has the same comportment as the previous one explain in conditional predicate.
  • "score\_value" : return the real score value.
| ```sql # Select UserPoint having spent more than 1000€ through $transaction_confirmed events during the past year SELECT (...) FROM UserPoint WHERE activities { events @ScoreField(name: "amount") @ScoreSum(min: 1000) { name = "$transaction_confirmed" AND date >= "now-1y/y" } } # Select UserPoint having spent more than 1000€ in cross orders with at least products which cost 10€: SELECT (...) FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name: "amount") @ScoreSum(min: 10, result:"score_value") { category="IT" } } } ``` {% endtab %} {% tab title="@ScoreAvg" %} Calculate the average from the sub-query matching scores and returns it if it's superior or equal to min args | args | description | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | min : `Float` | minimum required score for the nested sub-query to be returned | | result : `String` |

Two values possibles :

  • "boolean\_value" (by default) : reduce the returned score value to 0 or 1.
  • "score\_value" : return the real score value.
| ```sql # Select user points having spent on average at least 1000 through $transaction_confirmed events during the past year SELECT (...) FROM UserPoint WHERE activities { events @ScoreField(name: "amount") @ScoreAvg(min: 1000) { name = "$transaction_confirmed" AND date >= "now-1y/y" } } # Select user points having spent in average more than 1000€ by orders with at least products which cost 10€: SELECT (...) FROM UserPoint WHERE activity_events @ScoreAvg(min : 1000) { order { order_products @ScoreField(name: "amount") @ScoreSum(min: 10, result:"score_value") { category="IT" } } } ``` {% endtab %} {% tab title="@ScoreMax" %} Takes the maximum score from the sub-query matching scores and returns it if it's superior or equal to min | args | description | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | min : `Float` | minimum required score for the nested sub-query to be returned | | result : `String` |

Two values possibles :

  • "boolean\_value" (by default) : reduce the returned score value to 0 or 1.
  • "score\_value" : return the real score value.
| ```sql # Select user points with at least 1 activity that contains at least 1 $transaction_confirmed event with amount >= 1000 during the past year SELECT (...) FROM UserPoint WHERE activities { events @ScoreField(name: "amount") @ScoreMax(min: 1000) { name = "$transaction_confirmed" AND date >= "now-1y/y" } } ``` {% endtab %} {% endtabs %} #### Go forward It is possible to use these two ways at the same time but be careful, it is currently not possible to grow up a @ScoreField after a @ScoreSum(result: "boolean\_score"). Example of possible use case : ```sql # Select UserPoint having spent in average more than 1000 in one activity of events with a amount superior than 10 SELECT (...) FROM UserPoint WHERE activity @ScoreAvg(min : 1000) { events @ScoreField(name:"amount") @ScoreSum(min 10, result:"score_value") { name = "$transaction_confirmed" } } # Select UserPoint having spent in average in one activity, more than 1000 events with a amount superior than 10 SELECT (...) FROM UserPoint WHERE activity @ScoreAvg(min : 1000) { events @ScoreField(name:"amount") @ScoreSum(min: 10, result:"boolean_value") { name = "$transaction_confirmed" } } ``` However the following use case can't be written : ```sql # We wanted : # Count UserPoint having spent more than 1000€ in cross orders with at least 10 orders more than 100€ : SELECT (...) FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order @ScoreSum(min: 10, result: "score_value") { @ScoreSum(min: 100, result: "boolean_value") { order_products @ScoreField(name: "price") @ScoreSum(result:"score_value") { category="IT" } } } } # But the query return : # Count UserPoint having spent more than 1000 orders with at least 10 orders more than 100€ ``` Using conditional and scoring ways in the same query is useful for many use case whose won't be detailed here. A specific page has been created for regroup examples of them if you want to [go forward](/querying-your-data/otql-examples#scoring-operation). ### Date operators The following operators are available to work with dates : * `>=` Greater or equal * `>` Greater * `<=` Lower or equal * `<` Lower * `= or ==` Equal * `!=` Not equal Dates can be formatted either * in **ISO8601 format** (time part is optional) `2012-09-27`, `2012-09-27T12:42:00` * in a **timestamp** in milliseconds `1549365498507` * in a **Date Math** format, defining a relative date ```sql SELECT (...) FROM UserPoint WHERE activities {creation_ts <= "2012-09-27"} SELECT (...) FROM UserPoint WHERE activities {creation_ts > "1549365498507"} SELECT (...) FROM UserPoint WHERE activities {creation_ts > "now-7d"} ``` #### Date Math format The idea of the date match syntax is to define **a relative date compared to an anchor date**. The anchor date is either `now` or a date (`ISO8601` or `timestamp` format) followed by `||`. The expression begins with the anchor date and is followed by one or more math expressions : * `+1h` adds one hour * `-1d` substracts one day * `/d` rounds down to the nearest day Example, assuming now is `2001-01-01 12:00:00` : ```javascript now+1h // Resolves to: 2001-01-01 13:00:00 now-1h // Resolves to: 2001-01-01 11:00:00 now-1h/d // Resolves to: 2001-01-01 00:00:00 2001.02.01||+1M/d // Resolves to: 2001-03-01 00:00:00 ``` The supported units are the following : | Date operator | description | | ------------- | ----------- | | y | Years | | M | Months | | w | Weeks | | d | Days | | h or H | Hours | | m | Minutes | | s | Seconds | ### String operators Only the indexed fields of type String are eligible. Depending on the specified data\_type in the schema, the String operator will behave differently. #### With `data_type: text` : All operators are **case-insensitive.** [Values are stored as a set of words, transformed](/schema#string-indexed-as-text)**.** | String operator | description | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | match(fieldName, comparisonValue, operator = "or") |

Returns true if words in the text contained in fieldName match words in comparisonValue
operator defines how words are matched. Defaults to "or".

  • "or": returns true if at least one word in fieldName matches a word in comparisonValue
  • "and": returns true only if all words in comparisonValue are found in fieldName
| | starts\_with(fieldName, comparisonValue) | Returns true if a word of the text contained in fieldName starts with one of the words contained in comparisonValue. | The same transformation is done on the text data before storage is also done on the comparison value. ```sql # Doing (...) WHERE match(url_as_text, "Hello World!") # Will search in the text values for words matching 'hello' or 'world' https://www.hello.com/ https://www.world.com/ https://www.hello.com/world/ (...) ``` {% hint style="success" %} Diacritical marks (e.g. é, è, à, ç), number/digits, and word/expression containing apostrophe are usually stored as-is which means that you will need to provide the same value in the match function. {% endhint %} Below are some examples comparing the ingested and stored values, along with a demonstration of the **match** function: | Value ingested | Values stored | Value matching | Value not matching | | ---------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------- | | Métamorphosé | métamorphosé |

Métamorphosé
métamorphosé

|

metamorphose
métamorphos
métamorphosés

| | hameçon, ligne et bouchon |

hameçon
ligne
et
bouchon

|

hameçon
hameÇon
ligne
et
bouchon

| hamecon | | aujourd'hui | aujourd'hui | aujourd'hui |

aujourd hui
aujourd
hui

| | s'inscrire | s'inscrire | s'inscrire |

s inscrire
inscrire
s

| | 100 | 100 | 100 |

1000
10

| | 100 900 km |

100
900
km

|

100 900 km
100
900
km

|

100900
100,900

| | km/h |

km
h

|

km/h
km
h

|

kmh

| | 100km/h |

100km
h

|

100km/h
100km
h
km/h

|

100
km

| | H\&M |

h
m

|

H\&M
h
m

| hm | |

£1,000
1,000
1,000+

| 1,000 |

£1,000
1,000
1,000+

|

1000
000
1.000

| | 1.000 | 1.000 | 1.000 |

1000
000
1,000

| |

chou-fleur
chou- fleur
chou fleur

|

chou
fleur

|

chou-fleur
chou- fleur
chou fleur
chou
fleur

|

chou\_fleur
choufleur

| | chou\_fleur | chou\_fleur | chou\_fleur |

chou
fleur
choufleur
chou-fleur

| | recherche.aspx | recherche.aspx | recherche.aspx |

recette
aspx

| | |

test
mics.com

|


test
mics.com

|

@
mics

| #### With `data_type: keyword` : All operators are **case-sensitive**. [Values are considered as a single word. No transformation is made](https://developer.mediarithmics.io/schema#string-indexed-as-keyword). | String operator | description | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | starts\_with(fieldName, comparisonValue) | Returns true if the **exact** value contained in fieldName starts with the **exact** value passed in comparisonValue. | | = or == | Returns true if the **exact** value contained in fieldName is equal with the **exact** value passed in comparisonValue. | ```sql (...) WHERE starts_with(mykeyword, "Hello World!") (...) WHERE mykeyword == "Hello World!" ``` ### In operator You can use the `IN` operator as a shortcut to filter on multiple values of the same field. ```sql # Total sold in events for channel IDs 2456, 5489, 1426 SELECT {events {basket { amount @sum}}} FROM UserActivity WHERE channel_id IN ["2456","5489","1426"] # Equivalent of WHERE channel_id = "2456" OR channel_id = "5489" OR channel_id = "1426" ``` {% hint style="info" %} This operator offers better performances than multiple `OR` operators. {% endhint %} ### Is\_defined operator This is used to evaluate the value of a field and check if it is defined or not. The predicate can be applied in any indexed field in the schema and return a boolean. ```sql # Return all UserPoint with a profile SELECT { id } FROM UserPoint WHERE is_defined(profiles) # Return all UserPoint with an email in their profile SELECT { id } FROM UserPoint WHERE profiles{is_defined(email)} ``` | Field Value | Return | | ---------------------- | ------ | | myField = “example” | True | | myField = \[“example”] | True | | myField = \[null] | True | | myField = "" | True | | myField = \[ ] | True | | myField = null | False | | myField = undefined | False | | (NoField) | False | \*\*\*\* ## JOIN operations You may want to add another list of predicates FROM various objects. To do so, use JOIN clause to mention another object right after the FROM/WHERE clauses. It's possible to apply multiple JOIN in the same query. ```sql # Get the activity “$transaction_confirmed” of UserPoint of the segment id “1234“ SELECT { id } FROM ActivityEvent WHERE name=="$transaction_confirmed" JOIN Userpoint WHERE segments { id="1234" } ``` ## LIMIT operations ```sql # Get only 5 or fewer activities named “$transaction_confirmed” SELECT { id } FROM ActivityEvent WHERE name=="$transaction_confirmed" LIMIT 5 ``` In the Query Tools, we return 10 elements by default but you can easily override this by using the LIMIT clause followed by the number of elements required: ```sql SELECT { id } FROM ActivityEvent WHERE name=="$transaction_confirmed" LIMIT 100 # This query returns 100 (or fewer) activities named “$transaction_confirmed” ``` {% hint style="danger" %} It's not possible to return more than 10 000 elements with `LIMIT`. The query will fail if you try it. {% endhint %} Note that the LIMIT clause will be ignored when using `@count` or `@cardinality` directives. ```sql SELECT @count{ } FROM ActivityEvent WHERE name=="$transaction_confirmed" LIMIT 5 # Return the count of all activities named "$transaction_confirmed" # Example: Return - 21,866,076 ``` {% hint style="danger" %} `LIMIT` operator isn't applied during the segment calculation. If you create a User Query Segment with a `LIMIT`, it will be ignored and return all UserPoint who respect the `WHERE` clause. {% endhint %} {% hint style="info" %} If you're using an aggregation directive such as @map and you want to extend the number of buckets returned, you need to use the limit parameter of the aggregation directive (for instance: `@map(limit:200)`) and not the `LIMIT` clause. More details available hereunder. {% endhint %} ## SELECT operations They are simply selecting fields. Every field present [in the schema](/schema) can be selected. ```sql # Select id and name in the root level SELECT { id name } FROM UserPoint # Select name in UserPoint and creation_ts and id in emails linked to the UserPoint SELECT { name emails { creation_ts id } } FROM UserPoint ``` ### Filters [As we saw earler in this documentation](#where-object-tree-expressions), the `WHERE` expression gives you the ability to filter a sublist of objects at `FROM` level. You also have the capability to filter **in** or **out** the data returned by the query using the `@filter` directive in `SELECT` : ``` { @filter(clause: " == \"\"", filter_empty: ) { } ```
OptionMandatoryUsage
clauseYesUsed to list fields names & values that you want to filter in/out
filter_emptyNoUsed to filter out empty object. Set to true by default.
#### Tips Here are some tips to properly use the `@filter` directive in your queries: * Filter multiple fields (note that you can used OR or AND between fields, based on the required filter logic): ``` @filter(clause: "category == \"CAT_1\" OR referrer == \"REF\"") ``` * Filter **in** multiple values for a given fields: ``` @filter(clause: "category == \"CAT_1\" OR category == \"CAT_2\"") ``` * Filter **out** multiple values for a given fields: ``` @filter(clause: "category != \"CAT_1\" AND category != \"CAT_2\"") ``` * Combine AND & OR filters: ``` @filter(clause: "(category == \"CAT_1\" OR category == \"CAT_2\") AND referrer == \"REF\"") ``` * Filter by a subfield: ``` @filter(clause: "events { category == \"CAT_1\" } ") ``` When `filter_empty:true` option is provided, the following elements will be **filtered out**: * An optional array which is empty * An object which is empty and either optional or inside an array * A mandatory array where the following conditions are met : * Tha array is empty * The other mandatory selections of the parent object are only empty arrays * This parent object can be filtered {% hint style="warning" %} Please note that the `@filter` directive cannot be used at the same time as aggregation operations such as @map, @date\_histogram, ... {% endhint %} #### Example 1 - Usage of clause The following query retrieves userpoints, activities, events and some of their field for each UserPoint that has an event named "display". ``` select { id { activities { id events { name score } } } } from UserPoint where { activities { events { name == "display" } } } ``` Let's assume it gives the following result : ``` [ { "id": "up1", "activities": [ { "id": "a1", "events": [ { "name": "display", "score": 123 }, { "name": "click"} ] }, { "id": "a2", "events": [ { "name": "display" } ] } ] }, { "id": "up2", "activities": [ { "id": "a3", "events": [ { "name": "click" } ] }, { "id": "a4", "events": [ { "name": "display" } ] } ] } ] ``` The user might be surprised to find "click" events in this result. However remember that the where clause only filter the roots (i.e the UserPoints). To retrieve only "display" events, the user will need to add an @filter clause as follow: ``` select { id { activities { id events @filter(clause: "name == \"display\"") { name score } } } } from UserPoint where { activities { events { name == "display" } } } ``` Assuming the same data, this query would produce the following result : ``` [ { "id": "up1", "activities": [ { "id": "a1", "events": [ { "name": "display", "score": 123 } ] }, { "id": "a2", "events": [ { "name": "display" } ] } ] }, { "id": "up2", "activities": [ { "id": "a3", "events": [] }, { "id": "a4", "events": [ { "name": "display" } ] } ] } ] ``` #### Example 2 - Usage of filter\_empty The `@filter` predicate also filters by default empty result in its scope. To illustrate this, we reduce our query to retrieve only the score fields : ``` select { { activities { events @filter(clause: "name == \"display\"") { score } } } } from UserPoint where { activities { events { name == "display" } } } ``` Still the same data, the result is the following: ``` [ { "activities": [ { "events": [ { "score": 123 } ] }, { "events": [ ] } ] }, { "activities": [ { "events": [] }, { "events": [] } ] } ] ``` You'll notice that two `display` events have disappeared. Since they don't have a score, they would be empty object. Actually, this "filter empty" behavior can be set using a second optional parameter to the `@filter` directive. If we set it to false, we will obtain the following result : ``` select { { activities { events @filter(clause: "name == \"display\"", filter_empty: false) { score } } } } from UserPoint where { activities { events { name == "display" } }} //result [ { "activities": [ { "events": [ { "score": 123 } ] }, { "events": [ {} ] } ] }, { "activities": [ { "events": [] }, { "events": [ {} ] } ] } ] ``` #### Example 3 - Usage of extra @filter One might think that the previous result still contains a lot of noise (4 events retrieved for only one score). You can add an extra `@filter` before the object name to lighten the result: ``` select @filter { { activities { events @filter(clause: "name == \"display\"") { score } } } } from UserPoint where { activities { events { name == "display" } }} ``` You will get the following result : ``` [ { "activities": [ { "events": [ { "score": 123 } ] } ] } ] ``` #### Example 3 - Usage of @filter sub-field @filter can be used to filter a field by a condition on a sub-field. ``` SELECT { activities @filter(clause: "events {is_defined(event_name) AND event_name == \"display\"}") { events { event_name } } } FROM UserPoint where { activities { events { name == "display" }}} ``` ### Aggregation Operations The aggregation operations are initiated by a directive in the `SELECT` clause. They take into account the filter defined in the `WHERE` clause, however they are not compatible with the `@filter` directive that you can use in the `SELECT` clause. #### @count This directive is used to count the number of objects verifying the query ```sql # Counts the number of UserPoint SELECT @count {} FROM UserPoint # Counts number of new users in the past 7 days SELECT @count{} FROM UserPoint WHERE creation_date >= "now-7d/d" ``` #### Metrics directives {% hint style="warning" %} Fields used with metrics directives should have the [@TreeIndex](/schema#treeindex) directive in your schema. {% endhint %} Those directives calculate a value per bucket created in the bucket directive, or with only one bucket containing all elements if you don't use bucket directives. * `@avg`: average value for a specific field (only applies to numeric values) ```sql # Average basket amount between two specific dates SELECT {basket {amount @avg}} FROM UserEvent WHERE {date >= "2020-12-01" AND date <= "2020-12-31" } ``` * `@min`: minimum value for a specific field (only applies to numeric values) ```sql # Minimum basket amount between two specific dates SELECT {basket {amount @min}} FROM UserEvent WHERE { date >= "2020-06-20" AND date <= "2020-06-25” } ``` * `@max`: maximum value for a specific field (only applies to numeric values) ```sql # Maximum basket amount between two specific dates SELECT {basket {amount @max}} FROM UserEvent WHERE { date >= "2020-06-20" AND date <= "2020-06-25”} ``` * `@sum`: sum of value for a specific field (only applies to numeric values) ```sql # Sum of basket amounts between two specific dates SELECT {order{amount @sum }} FROM ActivityEvent WHERE {date >= "2020-06-20" AND date <= "2020-06-25”} ``` * `@cardinality`: count of distinct values ```sql # Number of channels in a datamart SELECT {channel_id @cardinality} FROM ActivityEvent # Number of cookies associated with UserPoint in a specific segment SELECT {agents{id @cardinality}} FROM UserPoint WHERE segments {id=”XXXX”} ``` #### Bucket directives {% hint style="warning" %} Fields used with bucket directives should have the [@TreeIndex](/schema#treeindex) directive in your schema. {% endhint %} Those directives separate values into buckets * `@map` one bucket per field value ```sql SELECT { channel_id @map { # map the values of channel id in several buckets session_duration @avg # The average duration } } FROM UserActivity # Data # channel ID : 1234, count : 654987987987, session_duration: 100 # channel ID : 1235, count : 987987965465, session_duration: 1500 ``` {% hint style="info" %} @map does not count null values {% endhint %} {% hint style="info" %} You can use the `limit` function to extend the number or elements that can be returned by the directive (default: 50, maximum: 50 000) : `SELECT { name @map(limit:200) } FROM UserEvent` {% endhint %} * `@histogram` aggregated count on a specific field. The interval can be modified regarding the business needs ```sql SELECT { order { amount @histogram(interval:50)}} FROM UserEvent WHERE date >= "now-7d" # Data # Key: 0, count: 97681 # Key: 50, count: 50324 # Key: 100, count: 33164 # Key: 150, count: 36528 ``` * `@date_histogram` aggregated count by a period of an object associated with a date. Allowed intervals are `1M` for a month and `XXD` for a XX number of days. ```sql # Mere use of @date_histogram directive: selecting all page_view events # in the last 30 days SELECT { date @date_histogram(interval:"1d") } FROM UserEvent WHERE name = "page_view" and date >= "now-30d/d" # @date_histogram used together with @map directive with default interval (days) SELECT { channel_id @map {date @date_histogram }} FROM UserEvent WHERE name = "$transaction_confirmed" # Data # Key: 2416, count: 27563351 # 2018-01-16T00:00:00.000Z 330 # 2018-01-17T00:00:00.000Z 331 # 2018-01-18T00:00:00.000Z 3332 # ... # Key: 2417, count: 65498798 # ... # Force an interval of one month SELECT { channel_id @map {date @date_histogram(interval: "1M") } } FROM UserEvent WHERE name = "$transaction_confirmed" ``` {% hint style="info" %} Please note that you cannot use bucket directives with metrics. {% endhint %} #### Aliases It is possible to add an alias to the field expression. This alias is then used in the output to identify the field result. ```sql SELECT { numberOfChannels: channel_id @cardinality # The approximate number of distinc values averageDuration: duration @avg # The average duration mininumDuration: duration @min # map the values of channel id in several buckets } FROM UserEvent ``` ## Managing queries You usually enter OTQL queries directly in tools like the navigator. However, **they can be saved and managed by code as objects.** Some features will require you to link an object to an OTQL query, instead of just saving the query as text. {% hint style="info" %} Features asking you to save queries to reference them usually want to leverage this for performances optimisation. {% endhint %} ## Creating a query `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/queries` Create an OTQL query in the platform before creating an export based on this query #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | ---- | ------ | ----------- | | Body | object | Payload | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "50409", // ID of the query to retrieve for the next steps "datamart_id": "1509", "query_language": "OTQL", "minor_version": null, "major_version": null, "query_text": "SELECT {id} FROM UserPoint", "favorite": false } } ``` {% endtab %} {% tab title="400 If the query is invalid" %} ```javascript { "status": "error", "error": "cannot save invalid query, cause: Syntax error while parsing document \"nawak\". Invalid input 'n', expected Comments or select (line 1, column 1):\nnawak\n^", "error_code": "BAD_REQUEST_DATA", "error_id": "ef292c4e-1eab-4a7f-8fb2-77d797139be9" } ``` {% endtab %} {% endtabs %} ```javascript // Creating a query payload { "query_text": "SELECT {id} FROM UserPoint", // Your query "datamart_id": "", "query_language": "OTQL" } ``` ## Check a query `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/query_check/otql` Create an OTQL query in the platform before creating an export based on this query #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | ---- | ------ | ----------- | | Body | object | Payload | {% tabs %} {% tab title="200 " %} ```javascript // If valid { "status": "ok", "data": { "type": "VALID", "validation": { "parameters": [] }, "status": "ok" } } // If invalid { "status": "ok", "data": { "type": "PARSING_ERROR", "error": { "message": "Syntax error while parsing document \"nawak\". Invalid input 'n', expected Comments or select (line 1, column 1):\nnawak\n^", "position": { "row": 1, "col": 1 }, "error_type": "PARSING" }, "status": "error" } ``` {% endtab %} {% endtabs %} ```javascript // Checking a query payload { "query": "SELECT {id} FROM UserPoint" // Your query } ``` ## Executing queries You can execute queries in the different tools that mediarithmics offer, or using our API. ## Execute a query `POST` `https://api.mediarithmics.com/v1/datamarts/:datamart_id/query_executions/otql?use_cache=true` Executes an OTQL query on the specified datamart #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------- | | datamart\_id | integer | The ID of the datamart | #### Query Parameters | Name | Type | Description | | ---------- | ------- | ------------------------------------------- | | use\_cache | boolean | Optimize the response time using the cache. | #### Request Body | Name | Type | Description | | ---- | ------ | --------------------- | | Body | string | OTQL query to execute | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "took": 112015, "timed_out": false, "offset": null, "limit": null, "result_type": "COUNT", "precision": "FULL_PRECISION", "sampling_ratio": null, "rows": [ { "count": 80975924 } ], "cache_hit": true } } ``` {% endtab %} {% tab title="503 Retry later." %} ```javascript { "status": "error", "error": "Service Unavailable", "error_code": "SERVICE_UNAVAILABLE", "error_id": "482416e1-2d93-484a-948b-615b639b5e4f" } ``` {% endtab %} {% endtabs %} ### Query cache When setting the `use_cache` query parameter to `TRUE`, the system returns the query from the cache if available. To know if the returned value is from the cache or a new query execution, look at the `cache_hit` property from the response. Its value is `TRUE` if the response comes from the cache and `FALSE` otherwise. {% hint style="info" %} The cache expiration delay is **12 hour**. {% endhint %} When not setting the `use_cache` query parameter or setting its value to `FALSE`, the cache system is skipped. The query will be executed and its value won't be stored. You can't use this to force a cache update . Running the query `SELECT ... FROM ... WHERE ts >= "now-1h"` (with a *Date Math* format from [Date operators](/querying-your-data/otql-queries#date-operators)) will return the same result now, in five minutes and during the next hour if using the cache. ## Queries optimization Our engine tries to automatically optimize queries before running them. For example, a query with multiple `OR` operators can use the `IN` operator instead if it is better. # OTQL examples This page provides examples of OTQL queries, based on simplified schemas. The objective is to be less technical and illustrate how works our language by different use cases. ## Basic queries To begin, we'll talk about fundamentals of the syntaxe language. How it works and what it's the excepted result for each query. It will give you additional informations about OTQL. {% hint style="info" %} It's better to start reading the [Introduction to OTQL](/querying-your-data/otql-queries#introduction) queries, if you didn't to it yet, before continue. {% endhint %} ### Schema example For the following examples, we consider the runtime schema below: ```graphql type UserPoint { id : ID! activities : [UserActivity!]! profiles : [UserProfile!]! } type UserActivity { id : ID! events : [UserEvent!]! } type UserEvent { id : ID! name : String } type UserProfile { id : ID! age : String } ``` If we want to represent a userpoint, it will be object tree. For illustrate it, we could considerate it like this : ![Representation of an userpoint](/files/wj8RFEZJlkYWZhlIA7Zb) ### The different steps of an OTQL query Even if the syntaxe of an OTQL query is close to a SQL one, the execution isn't the same at all. We talk about object tree and not column. This main difference gets lots of consequences and one of them is how the query is executed. **Query resolution is a two phases process** 1. Narrow queried object mentioned in `FROM` by applying a `WHERE` clause on it or/and on any sub-object's fields ```sql // SELECT # fields returned // FROM # where the query will be executed // WHERE # filter applied SELECT { activities { events { name } } } FROM UserPoint WHERE activities { events { name = "$transaction_confirmed" } } ``` When `WHERE` clause is applied on sub-object's field, if at least one sub-object validates the condition then all parent objects validate it as well. ![In this example, the UserEvent id=3 validates the WHERE which means UserActivity id=2 is selected and therefore UserPoint id=1 is returned from the query.](/files/4t9DjNyaCxDRUKxVOOLR) 2\. Return only desired objects & fields by listing them in `SELECT` clause ![This query means : "Get all event’s name by user with at least one transaction confirmed"](/files/JpUqoC89c25nydM5iMva) Finally the query returns : ``` [ { activities : [ { events : [ { name : "$page_view" } ] }, { events : [ { name : "$page_view" }, { name : "$transaction_confirmed" } ] } ] } ] ``` As you can see, despite the `WHERE` clause on `$transaction_confirmed` events, the query returns `$page_view` events since `SELECT` is applied from **UserPoint**. #### Same explanation with multiple userpoints 1. Narrow queried object mentioned in `FROM` by applying a `WHERE` clause on it or/and on any sub-object's fields ```sql # Example : Get all event’s name by user with at least one transaction confirmed SELECT { activities { events { name } } } FROM UserPoint WHERE activities { events { name = "$transaction_confirmed" } } ``` ![Like we seen, one transaction confirmed validate all the userpoint](/files/HnA9BBdC4SMjd38wkCOS) 2\. Return only desired objects & fields by listing them in `SELECT` clause ![The SELECT only be apply in userpoint still in the list, after the WHERE filter](/files/KOb5i1YaTJ6HoKPIAsc0) Here the query returns : ``` [ { activities : [ { events : [ { name : "$page_view" }, { name : "$transaction_confirmed" } ] } ] }, { activities : [ { events : [ { name : "$page_view" }, { name : "$transaction_confirmed" } ] }, { events : [ { name : "$page_view" } ] } ] } ] ``` In this example, the **UserPoint id=3** was not picked in the `WHERE` clause since it doesn't have any `$transaction_confirmed` event attached to it. ### Change the scope of your query #### Use FROM to chose your execution context As you could see, the `WHERE` can filter only object in defined by the `FROM` . It gives you the ability to start the query where you need, it defines the scope of the query resolution and where the other operators are executed. If you want to pick only a specific event, you will have to change the context. The `FROM` allows you to do so. ```sql # Example : Get the name of each event where the name is "$trasaction_confirmed" # Here, we just want to be sure this query return only events we want SELECT { name } FROM UserEvent WHERE name = "$transaction_confirmed" ``` ![](/files/LYTIfTOpHvfpzONug7P4) Here the query returns : ``` [ { name : "$transaction_confirmed" } ] ``` However specifying a sub-object also limit the scope of the predicate. For example, use `FROM UserEvent` doesn't give you access to **UserPoint** fields (like `profiles`). **Another way to get specific fields values : use `@filter` in the `SELECT` part to only retrieve elements you need** You can also use [`@filter`](/querying-your-data/otql-queries#filters) if you don't want to change the scope of your query but you need to get only specific elements. It is completely independent of the `WHERE` clause and the filter is applied after the `WHERE` clause execution. The first step will be the same as the one mentionned previously (narrow queried object mentioned in `FROM` using the `WHERE` clause). ```sql # Example : Get events named "$page_view" by user with at least one transaction confirmed SELECT { activities { events @filter(clause:"name = \"$page_view\"") { name } } } FROM UserPoint WHERE activities { events { name = “$transaction_confirmed” } } ``` ![Apply the WHERE clause on your data](/files/Cmdhh7C9XlEkELLLBFkX) But during the second phase, the `@filter` will be apply on selected objects. ![@filter remove all the object which doesn't match with its clause](/files/hHmixv8RwwDG2W4OGbYM) Thefore, only `$page_view` are returned by the query : ``` [ { activities : [ { events : [ { name : "$page_view" } ] } ] }, { activities : [ { events : [ { name : "$page_view" } ] }, { events : [ { name : "$page_view" } ] } ] } ] ``` #### Get condition on different sub-object Tree You can add multiple conditions in the `WHERE` clause using [boolean operators](/querying-your-data/otql-queries#where-object-tree-expressions). ```sql # Example : Get event’s names by user with at least one transaction confirmed # and an age between 20 and 30 years old SELECT { activities { events { name } } } FROM UserPoint WHERE profiles { age = "20-30" } AND activities {events { name = "$transaction_confirmed" } } ``` ![In this example, the two conditions need to be validated](/files/s9aS18p1W6oz2Zr99uO9) ``` [ { activities : [ { events : [ { name : "$page_view" }, { name : "$transaction_confirmed" } ] }, { events : [ { name : "$page_view" } ] } ] } ] ``` #### Add a condition from another Object Tree As previously demonstrated, it's quite easy to add conditions in sub-objects scope (`FROM`). However if you need to execute your query in a specific scope and apply condition from an other object, you will have to use `JOIN`. ```sql # Example : Get event’s names by events named "$transaction_confirmed" # and where the user is between 20 and 30 years old SELECT { name } FROM UserEvent WHERE name = "$transaction_confirmed" JOIN UserPoint WHERE profiles { age == "20-30" } ``` *Note:* the join is automatically resolved by UserPoint, there is no need to provide join constraint. ![Step 1 : Apply the first WHERE clause ](/files/DKgVanFNX0wpcs7yGB9m) ![Step 2 : Create an intersection with the second WHERE ](/files/BV1ygYacSd9wfBDFUKuU) ![Step 3 : Execute the selection in the new scope](/files/0LxLOiNNHrkUFXJSJZWS) ``` [ { name : "$transaction_confirmed" } ] ``` By the way, it is also possible to directly make the `JOIN` in `UserProfile` to get the same result : ```sql # Example : Get event’s names by events named "$transaction_confirmed" # and where the user is between 20 and 30 years old SELECT { name } FROM UserEvent WHERE name = "$transaction_confirmed" JOIN UserProfile WHERE age == "20-30" ``` ## Scoring operator This is the runtime schema for examples below ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX") { id:ID! activity_events:[ActivityEvent!]! } type ActivityEvent @Mirror(object_type:"UserEvent") { order:Order @Property(path:"$properties.order") } type Order { order_products:[OrderProduct]! date: Timestamp! } type OrderProduct { id:String @TreeIndex(index:"USER_INDEX") price: Float @TreeIndex(index:"USER_INDEX") # in € category:String @TreeIndex(index:"USER_INDEX") # possible value : "IT" or "Book" } ``` #### Use case : Count UserPoint who bought more than X€ of product of the IT category ```sql # More than 1000€ in one order : SELECT @count{} FROM UserPoint WHERE activity_events { order { order_products @ScoreField(name: "price") @ScoreSum(min: 1000) { category="IT" } } } # More than 1000€ in cross orders (explicite): SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000, result:"boolean_value") { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" } } } # More than 1000€ in cross orders (implicite): SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") { category="IT" } } } # More than 1000€ in cross orders this last 10 days: SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") { category="IT" } AND date > "now-10d" } } # More than 1000€ in cross orders with at least products which cost 10€: SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name: "price") @ScoreSum(min: 10, result:"score_value") { category="IT" } } } # More than 1000€ in cross orders with at least 10 products : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name: "price") @ScoreSum(result:"score_value") { category="IT" } AND order_products @ScoreSum(min: 10) { category="IT" } } } # More than 1000€ in cross orders with at least one product which costs more than 10€ : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name: "price") @ScoreSum(result:"score_value") { category="IT" } AND order_products @ScoreField(name: "price") @ScoreMax(min: 10) { category="IT" } } } # WARNING : DOES NOT WORK # More than 1000€ in cross orders with at least 10 orders more than 100€ : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order @ScoreSum(min: 10, result: "score_value") { @ScoreSum(min: 100, result: "boolean_value") { order_products @ScoreField(name: "price") @ScoreSum(result:"score_value") { category="IT" } } } } ``` #### Use case : Count UserPoint who bought in average more than X€ of product of the IT category ```sql # More than 1000€ in one order : SELECT @count {} FROM UserPoint WHERE activity_events { order { order_products @ScoreField(name: "price") @ScoreAvg(min: 1000) { category="IT" } } } # More than 1000€ in cross orders : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreAvg(min : 1000) { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" } } } ``` #### Use case : Count UserPoint who bought more than X€ of product of the IT or Book category ```sql # More than 1000€ in cross orders in IT or Book category : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" OR category="BOOK" } } } # Same result : #Doesn't work yet SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" }, order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="BOOK" } } } # More than 1000€ in cross orders in only with product of IT or Book category : # maximum of separately IT products and Book products is superior to 1000€ SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" } OR order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="BOOK" } } } # More than 1000€ in cross orders of products in IT and products in Book category : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" OR category="BOOK" } } } # More than 1000€ in cross orders in IT and Book category : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="IT" } } AND activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name:"price") @ScoreSum(result:"score_value") { category="BOOK" } } } # More than 1000€ in cross orders with at least 10 IT products and 10 BOOK products : SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name: "amount") @ScoreSum(){ category="IT" OR category="BOOK" } AND order_products @ScoreSum(min: 10) { category="IT" } AND order_products @ScoreSum(min: 10) { category="BOOK" } } } ### DUPLICATE WITH PREVIOUS QUERY # More than 1000€ in cross orders in IT or Book category with at least 10€ of each in each order: SELECT @count {} FROM UserPoint WHERE activity_events @ScoreSum(min : 1000) { order { order_products @ScoreField(name: "amount") @ScoreSum(){ category="IT" OR category="BOOK" } AND order_products @Scorefield(name: "amount") @ScoreSum(min: 10) { category="IT" } AND order_products @Scorefield(name: "amount") @ScoreSum(min: 10) { category="BOOK" } } } ``` ## Date operators This is the runtime schema for examples below ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX") { id:ID! profiles:[UserProfile!]! } type UserProfile { id:ID! birth_date:Date @TreeIndex(index:"USER_INDEX") } ``` **Use case: Select all UserPoint who are celebrating their birthday today and are between 18 and 28 years old (rolling years).** 
SELECT { id } 
FROM UserPoint 
WHERE profiles { birth_date IN ["now-18y/d", "now-19y/d", "now-20y/d", "now-21y/d", 
    "now-22y/d", "now-23y/d", "now-24y/d", "now-25y/d", "now-26y/d", "now-27y/d", "now-28y/d"]
}
{% hint style="warning" %} Be aware that `now` is evaluated at the start of the segment calculation, which may result in a discrepancy between the expected and actual values. {% endhint %} **Use case: Select all UserPoint who are celebrating their birthday in the next 7 days and are between 18 and 28 years old (rolling years).** ```sql SELECT { id } FROM UserPoint WHERE profiles { ( birth_date >= "now-18y/d" AND birth_date < "now+7d-18y" ) OR ( birth_date >= "now-19y/d" AND birth_date < "now+7d-19y" ) OR ( birth_date >= "now-20y/d" AND birth_date < "now+7d-20y" ) OR ( birth_date >= "now-21y/d" AND birth_date < "now+7d-21y" ) OR ( birth_date >= "now-22y/d" AND birth_date < "now+7d-22y" ) OR ( birth_date >= "now-23y/d" AND birth_date < "now+7d-23y" ) OR ( birth_date >= "now-24y/d" AND birth_date < "now+7d-24y" ) OR ( birth_date >= "now-25y/d" AND birth_date < "now+7d-25y" ) OR ( birth_date >= "now-26y/d" AND birth_date < "now+7d-26y" ) OR ( birth_date >= "now-27y/d" AND birth_date < "now+7d-27y" ) OR ( birth_date >= "now-28y/d" AND birth_date < "now+7d-28y" ) } ``` ## Filters `@filter` can be quite hard to understand, so let's see some examples to clarify its usage. This is the runtime schema for examples below ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX") { id:ID! events:[UserEvent!]! activities:[UserActivity!]! } type UserActivity { id:ID! events:[UserEvent!]! } type UserEvent @Mirror(object_type:"UserEvent") { id:ID! event_name:String @Property(path:"$event_name") @TreeIndex(index:"USER_INDEX") } ``` **Use case: I want to retrieve, for each user, all URLs of type "newsarticle" and category "actu" that were browsed yesterday.** If I use the following query : ```sql SELECT { events @filter(clause: "page_type == \"newsarticle\" AND page_category == \"actu\""){ url } } FROM UserPoint WHERE activities { ts >= "now-1d/d" } ``` ```json // This query returns [ [ { "events": [ { "url": "xxx" } ] }, { "events": [] }, { "events": [ { "url": "xxx" }, { "url": "xxx" }, { "url": "xxx" } ] }, { "events": [] }, //... ] ] ``` This query returns many empty events, making the result unusable. Although the events are filtered, I haven’t excluded UserPoint that don’t contain any events with the clause. To fix this, I need to add a **WHERE** clause to ensure each UserPoint has at least one event which matches the clause. ```sql SELECT { events @filter(clause: "page_type == \"newsarticle\" AND page_category == \"actu\""){ page_type page _category url } } FROM UserPoint WHERE activities { ts >= "now-1d/d" } AND events { page_type == "newsarticle" AND page_category == "actu" } ``` ```json // This query returns [ [ { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] }, { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] } ] ] ``` Here, the result is an improvement over the previous one, displaying only the events with a url of a page with the type "newsarticle" and the category "actu". Now, if we want to retrieve information from activities while still filtering the events, we need to refine the query further. ```sql SELECT { activities { events @filter(clause: "page_type == \"newsarticle\" AND page_category == \"actu\""){ page_type page_category url } } } FROM UserPoint WHERE activities { ts >= "now-1d/d" } AND events { page_type == "newsarticle" AND page_category == "actu" } ``` ```json [ [ { "activities": [ { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] }, { "events": [] }, { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] } ] } ] ] ``` As you can see, some events may be empty because the filter is not applied at the `activities` level. This means the returned activities contain at least one `events` which matches the clause. If you try applying the filter at a higher level, specifically at the activity level: ```sql SELECT { activities @filter(clause: "events { page_type == \"newsarticle\" AND page_category == \"actu\"}"){ events { page_type page_category url } } FROM UserPoint WHERE activities { ts >= "now-1d/d" } AND events { page_type == "newsarticle" AND page_category == "actu" } ``` ```json [ [ { "activities": [ { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] }, { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "video", "page_category": "sport", "url": "xxx" }, { "page_type": "video", "page_category": "sport", "url": "xxx" } ] } ] ] ``` By removing the event filter, the query now returns all events from activities that contain at least one page that matches the clause, rather than only pages that match. To fix this, the first solution is to add another `@filter` to exclude unnecessary elements from both `activities` and `events`. ```sql SELECT { activities @filter(clause: "events { page_type == \"newsarticle\" AND page_category == \"actu\"}"){ events @filter(clause: "page_type == \"newsarticle\" AND page_category == \"actu\""){ page_type page_category url } } } FROM UserPoint WHERE activities { ts >= "now-1d/d" } AND events { page_type == "newsarticle" AND page_category == "actu" } ``` ```json [ [ { "activities": [ { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] }, { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] } ] ] ``` This query filters out unnecessary elements from both events and activities, but it is quite lengthy and difficult to read. The second solution is to apply an empty `@filter` at the activities level in addition to the events level one: 
SELECT @filter{ activities { events @filter(clause: "page_type == \"newsarticle\" AND page_category == \"actu\""){ page_type page_category url } } }
FROM UserPoint 
WHERE activities { ts >= "now-1d/d" } AND events { page_type == "newsarticle" AND page_category == "actu" }
```json [ [ { "activities": [ { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] }, { "events": [ { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" }, { "page_type": "newsarticle", "page_category": "actu", "url": "xxx" } ] } ] ] ``` The empty filter removes any `events` that are directly empty from the `activities`. {% hint style="warning" %} The empty filter only removes empty sub-objects. If you add another field that contains data, the filter will not remove the `events` ```sql SELECT @filter{ activities { id events @filter(clause: "page_type == \"newsarticle\" AND page_category == \"actu\""){ url } } } FROM UserPoint WHERE activities { ts >= "now-1d/d" } AND events { page_type == "newsarticle" AND page_category == "actu" } ``` 
[
  [
    {
      "activities": [
        {
          "events": [
            {
              "url": "xxx"
            },
            "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
          ]
        },
        {
          "events": [],
          "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        }
      ]
    }
  ]
]
{% endhint %} # GraphQL queries The GraphQL API gives you the ability to query all kinds of UserPoint data as collected by the platform (profile, activities, segments, identifiers, etc.) **with a single request**. Each GraphQL query is meant to fetch a **single** UserPoint data. To query multiple UserPoint in a single query, depending on your use case, please refer to the [OTQL queries](/querying-your-data/otql-queries) and/or to the [Query exports](/advanced-usages/exporting-your-data/query-exports). {% hint style="info" %} If you're not familiar with GraphQL and how to interact with such APIs, please refer to the official documentation: {% endhint %} ## GraphQL schema The schema used in the GraphQL endpoint is based on the [customer defined schema](/schema/defining-your-schema). To add/remove fields from the GraphQL API, the schema has to be updated accordingly. ## GraphQL endpoint `POST` `/v1/datamarts/:datamartId/query_executions/graphql` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ------------------------------------------------------------ | | datamartId | string | The datamartId in which the UserPoint data will be looked up | #### Request Body | Name | Type | Description | | ----- | ------ | -------------------------------------------------------------------------------------------------------------------------------------- | | query | string | This parameter should include your GraphQL query, starting with "query MyQuery {...". Do not forget to escape double quotes if needed. | {% tabs %} {% tab title="200 " %} ``` { "data": { ...GraphQL response... } } ``` {% endtab %} {% tab title="429 " %} ``` { "error": "..." } ``` {% endtab %} {% endtabs %} ## GraphQL editor To test and write GraphQL queries, we recommend you using our GraphQL editor: 1. Head to the Computing Console 2. Click on GraphQL menu under DATA STUDIO section ## Limits Through the GraphQL endpoint, only queries (reads) are supported. Mutations (writes) are not supported. This endpoint is being rate-limited and will respond with 429 HTTP status code if the QPS exceed its limits. Please discuss with your account manager to have more information about this rate limiting and to request any limit increase. {% hint style="warning" %} The user activities number retuned is also limited. A query return to **maximum 100 activities**. Be careful, there is not warning and you can't change this limit yet. Moreover, an user event is always in an activity so, this limitation is indirectly apply on the events. The query will return only the event in the hundred first activities. {% endhint %} ## Query examples ```graphql # Return the UserPoint creation_ts field query MyQuery { user_point(user_point_id: "xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx") { creation_ts } } ``` ```graphql # Return profiles, accounts, segments and scenarios data of a UserPoint # Warning: This query relies on a customer defined schema so it may not # work as-is on your datamart. query MyQuery { user_point(user_point_id: "xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx") { id creation_ts profiles { gender birth_date } accounts { compartment_id user_account_id } segments { id } scenarios { scenario_id } } } ``` ## Selecting a UserPoint You can select a UserPoint by any identifier using different functions. ```graphql # Select by UserPoint ID query MyQuery { user_point(user_point_id: "xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx") { id creation_ts } } # Select by email hash query MyQuery { user_point_by_email_hash(email_hash: "xxxxxxxxxxxxxxxxxxx") { id creation_ts } } # Select by user account # Both compartment ID and user account ID are mandatory query MyQuery { user_point_by_user_account_id( compartment_id: "XXXX", user_account_id: "xxxx-xxx-xx-xxxxx" ) { id creation_ts } } # Select by user agent ID query MyQuery { user_point_by_user_agent_id(user_agent_id: "xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"){ id creation_ts } } ``` For more information, see [User identifiers](/user-points#user-identifiers) ## Filters To only select elements matching a specific clause, use `@filter`. This clause is a WHERE object tree expressions, so you can use any thing like it was this kind of expression. ```graphql query micsQuery { user_point(user_point_id:"xxx") { activities { #only return events with the name "$home_view" events @filter(clause: "name == \"$home_view\""){ ts } } } } ``` {% hint style="warning" %} Be carful the `@filter` is applied **after the limitation** of the activities. So if your query does not return activities it only mean there isn't activities which respect your condition in the hundred first of the UserPoint select. {% endhint %} # UserPoint API The UserPoint API allows to retrieve, create, update or delete all data related to a single UserPoint. ## General description The endpoint is formatted as follows: `/v1/datamarts//user_points/`**``**`/`**``**`/`**``** where * **``** allows to select a UserPoint based on any of its identifiers Possible values are: ``` or user_point_id= user_agent_id= compartment_id=,user_account_id= email_hash= ``` * **``** designates the type of data that will be addressed Possible values are: ``` user_identifiers user_profiles user_activities user_choices user_scenarios user_segments ``` * **``** allows to select a particular document on the UserPoint As each document is identified by a different identifier, possible values depend on the type of document previously selected. It is also optional: use it if you want to get, update or delete a particular document. ``` user_profiles/compartment_id= user_profiles/compartment_id=/user_account_id= user_choices/processing_id= user_scenarios/scenario_id= user_segments/audience_segment_id= ``` # UserActivity API The **user activities** endpoint gives you the ability to retrieve & delete any activities (and attached events) on a given UserPoint. ## Retrieve activities & events from a UserPoint `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_timelines/:userpointId/user_activities?filters=:filters` #### Path Parameters | Name | Type | Description | | --------------------------------------------- | ------ | ----------------------------------------------------------- | | datamartId\* | String | The ID of the datamart want to retrieve activities for | | userpointId\* | String | The ID of the UserPoint you want to retrieve activities for | #### Query Parameters | Name | Type | Description | | ------- | ------ | ---------------------------------------------- | | filters | String | Filter(s) to select appropriat user activities | {% tabs %} {% tab title="200: OK " %} ```javascript { // Response } ``` {% endtab %} {% endtabs %} ## Delete activities & events from a UserPoint `DELETE` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_timelines/:userpointId/user_activities?filters=:filters` #### Path Parameters | Name | Type | Description | | --------------------------------------------- | ------ | ----------------------------------------------------------- | | datamartId\* | String | The ID of the datamart want to retrieve activities for | | userpointId\* | String | The ID of the UserPoint you want to retrieve activities for | #### Query Parameters | Name | Type | Description | | ------- | ------ | ---------------------------------------------- | | filters | String | Filter(s) to select appropriat user activities | {% tabs %} {% tab title="200: OK " %} ```javascript { // Response } ``` {% endtab %} {% endtabs %} ## Filters As mentionned above, filters are optional and give you the ability to narrow your query. You can apply filters on any custom fields you have in your activity object as well as the following proprietary fields: * $unique\_key * $type * $channel\_id * $ts (through start\_ts & end\_ts keywords) To apply any filter, mention the field on which you want your filter to be applied, followed by double equal (**==**), then by value(s) for the field to be matched against. Note that: * You can provide in your query multiple filters by seperating filters with a comma (**,**). In that case, it will be considered as an "AND" between those filters * A filter can take multiple values by separating values with a pipe (**|**). In that case, it will be considered as an "OR" between those values ## Query examples ``` # As explained above, user activities query is always for a particular datamart (1649) & userpoint (dff34408-6cc1-4531-86c5-bac8da9ebac9) # Retrieve all activities curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities" # Retrieve all activities for a given channel (4307) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=$channel_id==4307" # Retrieve all activities for a given activity type (SITE_VISIT) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=$type==SITE_VISIT" # Retrieve all activities for several activity types (SITE_VISIT & APP_VISIT) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=$type==SITE_VISIT|APP_VISIT" # Retrieve a given activity (17928440-e72f-11ec-aad0-d12e54ffd215) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=$unique_key==17928440-e72f-11ec-aad0-d12e54ffd215" # Retrieve all activities ingested after a particular date (14/06/2022 10:38:47) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=start_ts==1655195927000" # Retrieve all activities ingested before a particular date (13/07/2022 13:15:00) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=end_ts==1657717207000" # Retrieve all activities ingested between 2 dates (between 14/06/2022 10:38:47 & 13/07/2022 13:15:00) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=start_ts==1655195927000,end_ts==1657717207000" # Retrieve all activities ingested between 2 dates for a given channel (between 14/06/2022 10:38:47 & 13/07/2022 13:15:00 for channel 4307) curl -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=start_ts==1655195927000,end_ts==1657717207000,$channel_id==4307" # Delete a given activity (17928440-e72f-11ec-aad0-d12e54ffd215) curl -X DELETE -k -H "Authorization: " -H "content-Type: application/json" "https://api.mediarithmics.com/v1/datamarts/1649/user_timelines/dff34408-6cc1-4531-86c5-bac8da9ebac9/user_activities?filters=$unique_key==17928440-e72f-11ec-aad0-d12e54ffd215" ``` # Activities analytics queries The activity analytics endpoint has been designed as a cube to query user activities. This API gives you programmatic access to user activities as a data cube. You get metrics with dimensions, filters, and within date ranges leveraging the [Activity analytics](/data-ingestion/activities-analytics) part of our multi-model database. ![](/files/kZeCRU7t1moe3QZis7m9) With the activities analytics API, you can create reports to answer questions like: * **Number of active users.** The metric is `users` and it has no dimensions or filter. * **Number of active users per channel per day.** The metric is `users`, grouped by `channel_id` and `date_yyyymmdd` dimensions, without filters. * **Number of sessions per day for users who had an activity of type AD\_VIEW on campaign 666.** The metric is `sessions`, grouped by the `date_yyyymmdd` dimension with filter clauses `activity_type = AD_VIEW` and `origin_campaign_id = 666` * **Days with more than 200k transactions on a specific channel**. The metric is `number_of_transactions`, grouped ty the `date_yyyymmdd` dimension with filter clauses on `channel_id = 666`. Then a filter is applied on the calculated metric to only keep days with more than 200k transactions. It can also be used to build custom dashboards. For more information, see [Dashboards](/dashboards). {% hint style="info" %} * Maximum recommended **5 queries per second.** * Response time around **1 second**. * Results date range for the **4 latest months**. {% endhint %} ## Quick start guide Calling the API to get your first metrics is easy with your favorite tool that you already use to query other mediarithmics endpoints. See the [API Quickstart](/querying-your-data/activities-analytics-queries/api-quickstart) guide to get started. ## Available endpoints * [`user_activities_analytics`](/resources/data-cubes/reference#user_activities_analytics-endpoint) returns a customized report of your activities analytics data. ## Supported dimensions and metrics See [Dimensions en metrics](/querying-your-data/activities-analytics-queries/dimensions-and-metrics) for the complete list of supported dimensions and metrics. ## How data cubes work This endpoint is a mediarithmics [Data cube](/resources/data-cubes). You can find documentation on how data cubes work and which data cubes are available in the specific documentation section. # API Quickstart This page shows you how to get started using the activities analytics API to query your data in mediarithmics. ## Step 1 : Configure authentication This quickstart guide uses the [Long term access tokens](/resources/api-overview/authentication#long-term-access-tokens) authentication method. Choose and configure your own authentication method. For more information, see [Authentication](/resources/your-first-integration/make-the-first-bulk-import/api-basics/authentication). ## Step 2 : API call `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_activities_analytics` #### Query Parameters | Name | Type | Description | | -------------------------------------------- | ------ | ------------------------------- | | datamartId\* | number | The ID of the datamart to query | #### Request Body | Name | Type | Description | | ---------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | metrics\* | array | Array of [Metric](/resources/data-cubes/reference#metric) to retrieve. | | dimension\_filter\_clauses | object | Filters to apply on dimensions before calculating the metric. For more information, see [FilterClause](/resources/data-cubes/reference#filterclause). | | dimensions\* | array | [Dimensions](/resources/data-cubes/reference#dimension) to group metrics by. | | date\_ranges\* | array | Periods to analyze. Each date range is an object with a `start_date` and an `end_date`. See [DateRange](/resources/data-cubes/reference#daterange). | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "report_view": { "items_per_page": 100, "total_items": 7, "columns_headers": [ "type" ], "rows": [ [ "DISPLAY_AD" ], [ "EMAIL" ], [ "SITE_VISIT" ], [ "USER_SCENARIO_NODE_ENTER" ], [ "USER_SCENARIO_NODE_EXIT" ], [ "USER_SCENARIO_START" ], [ "USER_SCENARIO_STOP" ] ] } } } ``` {% endtab %} {% endtabs %} Here is a sample [report request](/resources/data-cubes/creating-a-report#report-request) as body payload with all the important properties ```json { // Retrieve the data in the specified date range // Mandatory. The data is only queryable for the last 4 months // Only one range is allowed now, but the API is prepared to accept // multiple ranges in the future. // Tip : you can use dates in "now-Xd/d" format as in OTQL queries "date_ranges": [ { "start_date": "2021-10-10T00:00:00", "end_date": "2021-10-25T23:59:59" } ], // List of dimensions to retrieve "dimensions": [ { "name": "date_yyyy_mm_dd" }, { "name": "channel_id" } ], // Filters on dimensions "dimension_filter_clauses": { "operator": "OR", "filters": [ { "dimension_name": "type", "operator": "EXACT", "expressions": [ "SITE_VISIT" ] } ] }, // Order by dates, beginning with the most recent "order_by": { "field_name": "-date_yyyy_mm_dd" }, // List of metrics to retrieve "metrics": [ { "expression": "users" }, { "expression": "number_of_transactions" } ] } ``` The API will answer with a [Single resource wrapper](/resources/api-overview#single-resource-wrapper) containing a [ReportView](/resources/data-cubes/reference#reportview). ```json { "status": "ok", "data": { "report_view": { // Note : pagination not implemented yet "items_per_page": 100, "total_items": 100, // To know which data is in which column "columns_headers": [ "date_yyyy_mm_dd", "channel_id", "users", "number_of_transactions" ], // Data "rows": [ [ "2021-10-10", 666, 3881, 17800.0 ], [ "2021-10-10", 555, 1838, 4200.0 ], [ "2021-10-11", 666, 532, 3900.0 ], [ "2021-10-11", 555, 8, 100.0 ] // ...[ ] } } } ``` Congratulations! You've sent your first request to the Activities analytics API. # Dimensions and metrics The dimensions and metrics allowed in the activities analytics API. ## Dimensions The following dimensions can be requested in reports ### General dimensions | Name | Title | Description | | ---------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | Activity type | See [User activity object](/user-points/user-activities#user-activity-object) for a list of all activity types. | | `date_time` | Date + time | The combined value of date and time of the activity in timestamp format | | `channel_id` | Channel ID | The ID of the channel on which the activity was registered | | `session_duration` | Session duration | Duration of the session in seconds | | `segment_id` | Segment ID |

IDs of the segments in which the user was when doing the activity.

Note : querying this dimension can throw an error if date ranges of the query are too big.

| | `date_yyyymmdd` | Date | Date in the `YYYYMMDD` format | | `date_yyyymmddhh` | Date + Hour | Date in the `YYYYMMDDHH` format | | `date_yyyy_mm_dd` | Date | Date in the `YYYY_MM_DD` format | | `date_yyyy_mm_dd_hh_mm` | Date + Hour + minutes | Date in the `YYYY_MM_DD_HH_mm` format | | `has_conversion` | Has conversion | Boolean. If a `$conversion` event happened during the activity. For more information, see [Predefined event names](/user-points/user-activities#predefined-event-names). | | `goal_id` | Goal ID | IDs of the goals triggered during the activity | | `has_bounced` | Has bounced | Boolean. If the user only visited one page during the activity. | | `transaction_amount` | Transaction amount | Amount spent by the user during the activity. | | `number_of_user_events` | Number of events |

Total number of User events the user triggered during the activity.

All events (custom and predefined) are counted in this total, except : $conversion $ad\_click $ad\_view $email\_view $email\_click $email\_sent $email\_delivered $email\_soft\_bounce $email\_hard\_bounce $email\_unsubscribe $email\_complaint $set\_user\_profile\_properties $set\_user\_consent $content\_correction $quit\_while\_running $cleaned\_referrer

| | `number_of_ad_views` | Number of `$ad_view` events | Total number of [User events](/user-points/user-activities#user-events-object) named `$ad_view` during the activity. | | `number_of_ad_clicks` | Number of `$ad_click` events | Total number of [User events](/user-points/user-activities#user-events-object) named `$ad_click` during the activity. | | `number_of_email_views` | Number of `$email_view` events | Total number of [User events](/user-points/user-activities#user-events-object) named `$email_view` during the activity. | | `number_of_email_clicks` | Number of `$email_click` events | Total number of [User events](/user-points/user-activities#user-events-object) named `$email_click` during the activity. | | `number_of_confirmed_transactions` | Number of `$transaction_confirmed` events | Total number of [User events](/user-points/user-activities#user-events-object) named `$transaction_confirmed` during the activity | ### Event dimensions You can query dimensions specific to the events that happened during each activity. {% hint style="info" %} Only specific predefined events are registered at the moment. See [Event transformations](/data-ingestion/activities-analytics#event-transformations) to integrate your events in these dimensions. {% endhint %} | `event_type` | Event type. Only `$transaction_confirmed` `$item_view` `$list_item_view` `$basket_view` events are stored at the moment | | ------------ | ----------------------------------------------------------------------------------------------------------------------- | | `brand` | Brands of items related to the events | | `category1` | Category 1 of items related to the events | | `category2` | Category 2 of items related to the events | | `category3` | Category 3 of items related to the events | | `category4` | Category 4 of items related to the events | ### Device dimensions Informations about the device used during each activity. | `device_form_factor` | Type of device : `PERSONAL_COMPUTER` `SMART_TV` `GAME_CONSOLE` `SMARTPHONE` `TABLET` `WEARABLE_COMPUTER` `OTHE` | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | `device_os_family` | OS of the device : `WINDOWS` `MAC_OS` `LINUX` `ANDROID` `IOS` `OTHER` | | `device_os_versions` | Version of the OS, for example `Windows 8` `ios10` | | `device_browser_family` | Browser used during the activity : `CHROME` `IE` `FIREFOX` `OPERA` `STOCK_ANDROID` `BOT` `EMAIL_CLIENT` `MICROSO_EDGE` `OTHER` | | `device_browser_version` | Browser's version. For example `10.3.4`, `2.2` | | `device_brand` | Device brand. For example `Acer` `Free` | | `device_model` | Device model. For example `10 plus` `4K Ultraslim` | | `device_agent_type` | Agent type : `MOBILE_APP` `WEB_BROWSER` | ### Origin dimensions The following dimensions are populated by the [Activity origin](/user-points/user-activities#activity-origin) : * `origin_campaign_name` / `origin_campaign_technical_name` / `origin_campaign_id` * `origin_sub_campaign_technical_name` / `origin_sub_campaign_id` * `origin_message_id` / `origin_message_technical_name` * `origin_keywords` * `origin_creative_name` / `origin_creative_technical_name` / `origin_creative_id` * `origin_engagement_content_id` * `origin_social_network` * `origin_referral_path` ### Location dimensions The following dimensions are populated by the [Activity location](/user-points/user-activities#activity-location) : * `location_source` * `location_country` * `location_region` * `location_iso_region` * `location_city` * `location_iso_city` * `location_latitude` * `location_longitude` ## Metrics The following metrics can be displayed in reports. | Name | Title | Description | | ------------------------------------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `users` | Active users | The number of distinct active users | | `sessions` | Activities / Sessions | The number of activities / sessions | | `conversion_rate` | Conversion rate | Calculated with (Number of activities with conversions / Total number of activities) | | `avg_number_of_sessions_per_user_point` | Average number of sessions per UserPoint |

Calculated from the number of sessions and the number of UserPoint.

Note : this metric cannot be used with other metrics

| | `avg_revenue_per_user_point` | Average revenue per UserPoint |

Revenue divided by the number of distinct UserPoint

Note : this metric cannot be used with other metrics

| | `avg_number_of_transactions_per_user_poit` | Average number of transactions per UserPoint |

Number of transactions devided by the number of distinct UserPoint

Note : this metric cannot be used with other metrics

| | `avg_session_duration` | Average session duration | Calculated by doing an average of the `session_duration` dimension. | | `revenue` | Revenue | Sum of `transaction_amount` dimension | | `avg_transaction_amount` | Average transaction amount per activity |

Sum of transaction\_amount dimension divided by the number of transactions.

Note : this metric cannot be used with other metrics

| | `avg_number_of_user_events` | Average number of user events per activity | Sum of `number_of_user_events` dimension divided by the number of activities. | | `number_of_user_events` | Total number of user events | Sum of `number_of_user_events` dimension | # Use cases This page demonstrates different useful queries that can be handled through the activities analytics API ## Number of active users {% code title="Request object" %} ``` { "date_ranges": [ { "start_date": "2021-09-26T00:00:00", "end_date": "2021-10-28T23:59:59" } ], "dimensions": [], "metrics": [ { "expression": "users" } ] } ``` {% endcode %} {% code title="Response" %} ``` "report_view": { "items_per_page": 100, "total_items": 1, "columns_headers": [ "users" ], "rows": [ [ 114353272 ] ] } ``` {% endcode %} ## Number of activities per day {% code title="Request object" %} ``` { "date_ranges": [ { "start_date": "2021-10-20T00:00:00", "end_date": "2021-10-28T23:59:59" } ], "dimensions": [ {"name": "date_yyyymmdd"} ], "metrics": [ { "expression": "sessions" } ] } ``` {% endcode %} {% code title="Response" %} ``` "report_view": { "items_per_page": 100, "total_items": 9, "columns_headers": [ "date_yyyymmdd", "sessions" ], "rows": [ [ "20211020", 1372624 ], [ "20211021", 1368085 ], ... ] } ``` {% endcode %} ## Values of a dimension {% code title="Request object" %} ``` { "date_ranges": [ { "start_date": "2021-10-20T00:00:00", "end_date": "2021-10-28T23:59:59" } ], "dimensions": [ {"name": "type"} ], "metrics": [ ] } ``` {% endcode %} {% code title="Response" %} ``` "report_view": { "items_per_page": 100, "total_items": 4, "columns_headers": [ "type" ], "rows": [ [ "DISPLAY_AD" ], [ "SITE_VISIT" ], [ "USER_SCENARIO_NODE_ENTER" ], [ "USER_SCENARIO_NODE_EXIT" ] ] } ``` {% endcode %} ## Active users and the revenue per day per device form factor on a specific channel {% code title="Request object" %} ``` { "date_ranges": [ { "start_date": "2021-10-20T00:00:00", "end_date": "2021-10-28T23:59:59" } ], "dimensions": [ {"name": "date_yyyymmdd"}, {"name": "device_form_factor"} ], "dimension_filter_clauses": { "operator": "AND", "filters": [ { "dimension_name": "channel_id", "operator": "EXACT", "not": false, "expressions": [ "666" ] } ] }, "metrics": [ {"expression": "users"}, {"expression": "revenue"} ] } ``` {% endcode %} {% code title="Response" %} ``` "report_view": { "items_per_page": 100, "total_items": 54, "columns_headers": [ "date_yyyymmdd", "device_form_factor", "users", "revenue" ], "rows": [ [ "20211020", "OTHER", 141, 222.74 ], [ "20211020", "PERSONAL_COMPUTER", 821923, 87656567.1 ], [ "20211020", "SMART_TV", 11, null ] [ "20211020", "SMARTPHONE", 1901978, 98435875.79 ], ... ] } ``` {% endcode %} ## More use cases You can create any use case you want using the list of [dimensions and metrics](/querying-your-data/activities-analytics-queries/dimensions-and-metrics) that are available. # Funnel API The **Data** **Studio** > **Funnel** page in the navigator uses an API that you can leverage to analyze funnel conversions in your own tools. For more information on the feature, see [Funnel](https://mediarithmics.gitbook.io/user-guide/data-studio/funnel). ## Retrieve dimensions values autocomplete `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_activities_analytics` Use this call to get suggestions or autocomplete values for a dimension #### Query Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | number | The ID of the datamart | #### Request Body | Name | Type | Description | | -------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | metrics | array | Empty array | | dimension\_filter\_clauses | object | Dimensions filters clause to apply. | | dimensions | array | Names of the dimensions to retrieve. Usually only one dimension.Use multiple dimensions to get possible values of a dimension if the other dimension is set. For example, using the dimensions `TYPE` and `EVENT_TYPE` we can ask for the possible values of `EVENT_TYPE` if `TYPE` is `SITE_VISIT`. | | date\_ranges | array | Periods to analyze. Each date range is an object with a `start_date` and an `end_date`. | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "report_view": { "items_per_page": 100, "total_items": 7, "columns_headers": [ "type" ], "rows": [ [ "DISPLAY_AD" ], [ "EMAIL" ], [ "SITE_VISIT" ], [ "USER_SCENARIO_NODE_ENTER" ], [ "USER_SCENARIO_NODE_EXIT" ], [ "USER_SCENARIO_START" ], [ "USER_SCENARIO_STOP" ] ] } } } ``` {% endtab %} {% endtabs %} Here is a sample body payload ```javascript { "date_ranges": [ { "start_date": "2021-04-22T00:00:00", "end_date": "2021-04-29T23:59:59" } ], "dimensions": [ { "name": "TYPE" } ], "dimension_filter_clauses": { "operator": "OR", // OR or AND "filters": [ { "dimension_name": "TYPE", "operator": "LIKE", // LIKE, EXACT or IN_LIST "expressions": [ "" ] } ] }, "metrics": [] } ``` ## Submit a funnel for results `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/user_activities_funnel` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | number | The ID of the datamart | #### Request Body | Name | Type | Description | | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | limit | number | When spliting a step on a specific field, sets the maximum values to be retrieved to optimize the query with what would be displayed.For example, set to 5 if you only show the 5 best channel IDs in the UI when splitting by channel ID to optimize the query. | | in | object | Period to query the funnel. Should be in the last 4 months maximum. | | for | object | List of steps in the funnel | {% tabs %} {% tab title="200 " %} ``` { "status": "ok", "data": { "global": { "total": 879879879, "steps": [ { "name": "Step 1", "count": 546546546, "interaction_duration": 0 }, { "name": "Step 2", "count": 897987984651, "amount": 1213213.27, "conversion": 11221, "interaction_duration": 515151 } ] }, "grouped_by": [] } } ``` {% endtab %} {% endtabs %} Here is a sample payload: ```javascript { "for": [ { "name": "Step 1", "filter_clause": { "operator": "OR", "filters": [ { "dimension_name": "TYPE", "not": false, "operator": "EXACT", "expressions": [ "DISPLAY_AD" ] } ] } }, { "name": "Step 2", "filter_clause": { "operator": "AND", "filters": [ { "dimension_name": "EVENT_TYPE", "not": false, "operator": "EXACT", "expressions": [ "$transaction_confirmed" ] }, { "dimension_name": "CHANNEL_ID", "not": false, "operator": "IN_LIST", "expressions": [ "8888", "6666" ] } ] } } ], "in": { "type": "DATES", "start_date": "2021-04-23", "end_date": "2021-05-01" }, "limit": 5 } ``` ## Dimensions You can build queries with the following dimensions: * Activity Date `DATE_TIME` * Activity Type `TYPE` * Ad Group Id `ORIGIN_SUB_CAMPAIGN_ID` * Brand `BRAND` * Channel Id `CHANNEL_ID` * Campaign Id `ORIGIN_CAMPAIGN_ID` * Category 1 `CATEGORY1` * Category 2 `CATEGORY2` * Category 3 `CATEGORY3` * Category 4 `CATEGORY4` * Creative Id `ORIGIN_CREATIVE_ID` * Device Brand `DEVICE_BRAND` * Device Browser `DEVICE_BROWSER_FAMILY` * Device Carrier `DEVICE_CARRIER` * Device Form Factor `DEVICE_FORM_FACTOR` * Device Model `DEVICE_MODEL` * Device OS `DEVICE_OS_FAMILY` * Has conversion `HAS_CONVERSION` * Has clicked `HAS_CLICKED` * Has bounced `HAS_BOUNCED` * Event type `EVENT_TYPE` * Is in segment `SEGMENT_ID` * Campaign Id `CAMPAIGN_ID` * Goal Id `GOAL_ID` * Product Id `PRODUCT_ID` ## Dimensions filters clause This object represents a group of filters to apply in a request. It has: * An `operator` field to apply either an `AND` or an `OR` between the filters * A `filters` array for the list filters to apply. For more information, see [Dimensions filters](/querying-your-data/funnel-api#dimensions-filter). ```javascript "filter_clause": { "operator": "OR", // OR or AND "filters": [ ... ] } ``` ## Dimensions filter This object represents a filter in a filters clause. It has; * A `dimensions_name` field to select the dimension it applies on. For more information, see [Dimensions](/querying-your-data/funnel-api#dimensions). * A `not` boolean field to apply boolean logic * An `operator` field to select one of the following queries: * `EXACT` will force the dimension to match the first expression set * `LIKE` will allow the dimension to only contain the first expression set * `IN_LIST` will allow the dimension to be one of the expressions set * A list of `expressions` representing the keywords to search for. ### Examples ```javascript // TYPE should be DISPLAY_AD { "dimension_name": "TYPE", "not": false, "operator": "EXACT", "expressions": [ "DISPLAY_AD" ] } // TYPE should contain SITE // SITE_VISIT activities will be used { "dimension_name": "TYPE", "not": false, "operator": "LIKE", "expressions": [ "SITE" ] } // TYPE should not contain SITE { "dimension_name": "TYPE", "not": true, "operator": "LIKE", "expressions": [ "SITE" ] } // CHANNEL_ID should be either 8888 or 6666 { "dimension_name": "CHANNEL_ID", "not": false, "operator": "IN_LIST", "expressions": [ "8888", "6666" ] } ``` # Alerting mediarithmics modules can trigger alerts to grab the attention of users/integrators on specific points to improve or fix. Alerts are displayed in the UI, but can also be accessed by API if you want to automate actions or grab them in your own reports. ## Alert types Here are the various alert types that exist:
TypeTrigger
SEGMENT_DEFINITION_ERRORError in the segment definition.
More information here
SEGMENT_VOLUME_DROP

Segment volume drops by more than a configured threshold (in percentage).

In the segment computation process, after it has been computed, we check

  • If volumes have dropped by more than XX%
  • If none of the segment labels are in the blocklist when in blocklist mode
  • If any of the segment labels are in the allowlist when in allowlist mode

More information here

INITIAL_LOADING_ERRORError during initial loading of a feed attached to a segment
More information here
SEGMENT_COMPUTATION_ERRORError during the calculation of a segment
More information here
## Alert properties Alerts have several properties associated: * `type`: The type of the alert (e.g., SEGMENT\_COMPUTATION\_ERROR, SEGMENT\_VOLUME\_DROP...) * `id`: The unique identifier of the alert * `datamart_id`: The identifier of the datamart associated with the alert * `organisation_id`: The identifier of the organisation associated with the alert * `community_id`: The identifier of the community associated with the alert * `created_ts`: The timestamp indicating when the alert was created * `archived`: A flag indicating whether the alert is closed/archived (`true`) or open (`false`) * `archived_ts`: The timestamp indicating when the alert was closed/archived * `archived_by`: The identifier of the user that closed/archived the alert * `expiration`: The expiration timestamp for the alert * `count`: The number of times the alert has been triggered * `last_count_ts`: The timestamp of the most recent trigger of the alert ### Polymorphism The alerting system supports polymorphism to accommodate specific fields for various alert types. #### For Segment alert
Alert typeProperty nameValue
-segment

{

'segment_id': 'xxx',

'segment_name': 'The segment name',

'segment_type': 'USER_QUERY',

'user_points_count': 71989,

'feeds_count': 0

}

SEGMENT_INITIAL_LOADINGalert_sub_type

sub-type of the error:

  • INITIAL_LOADING_EXECUTION_ON_ERROR
  • INITIAL_LOADING_RECORDS_ERROR
  • INITIAL_LOADING_NOT_STARTING
  • INITIAL_LOADING_RUNNING_TOO_LONG
SEGMENT_INITIAL_LOADINGfeed_idID of the feed concerned by the error
SEGMENT_VOLUME_DROPdrop_rateTotal drop (in percentage) of the segment volume since the alert was first triggered
### Open/Close To provide a familiar terminology to users, alerts can be opened or closed. However, in the system, the open/closed state is represented by the `archived` field. Opening an alert sets the `archived` field to `false`, while closing an alert sets it to `true`. The closed state implies that the alert is no longer active or visible to users. ### Preventing duplicates To avoid having multiple instances of the same alert, the system employs a prevention mechanism. Each alert has `count` and `count_last_ts` properties. When triggering a new alert, we check if there is already an active alert for the same target. If such an alert exists, the system increments the `count` property and updates the `count_last_ts` to reflect the latest trigger. This prevents the proliferation of identical alerts and ensures that only one alert remains active with an incremented counter. For example, if a segment has a query error and the issue persists without resolution, the system will increment the `count` property of the existing alert rather than creating multiple duplicate alerts. The `count_last_ts` property stores the timestamp of the most recent trigger, while the `created_ts` property stores the timestamp of the initial trigger. ### Expiration To manage the storage of alerts and ensure their relevance, the system implements an expiration mechanism. Alerts have an expiration duration associated with them. It is set to the `created_ts` + 1 month. You can't modify this behavior. A cleaning job runs regularly to identify and delete all expired alerts from the database. This prevents the accumulation of unnecessary historical data. ## API The API allows users and integrators to interact with alerts through the following functionalities:API ## Retrieve a list of alerts based on specific criteria `GET` `https://api.mediarithmics.com/v1/alerts` [Paginated API](/resources/api-overview). You have to fill in either the `organisation_id`, `datamart_id` or `community_id` parameters. Archived alerts are not returned by default. You need to ask them through the `archived` parameter. #### Query Parameters | Name | Type | Description | | ---------------- | --------- | --------------------------------------------------------------------- | | organisation\_id | Int | ID of the organisation in which to find alerts | | datamart\_id | Int | ID of the datamart in which to find alerts | | community\_id | Int | ID of the community in which to find alerts | | type | AlertType | Such as `SEGMENT_COMPUTATION_ERROR`. See [Alert types](#alert-types). | | archived | Boolean | `true` to return archived alerts. | {% tabs %} {% tab title="401: Unauthorized You don't have access to the organisation, datamart or community" %} {% endtab %} {% tab title="200: OK List of alerts" %} {% endtab %} {% endtabs %} ## Change the status of an alert from open to closed or vice versa `PUT` `https://api.mediarithmics.com/v1/alerts/:alertId` #### Path Parameters | Name | Type | Description | | ------- | ---- | ----------------------- | | alertId | Int | ID of the alert to edit | #### Query Parameters | Name | Type | Description | | ------- | ------- | --------------------------------------------- | | archive | boolean | `true` to close an alert, `false` to open it. | {% tabs %} {% tab title="401: Unauthorized You can't open/close this alert" %} {% endtab %} {% tab title="200: OK OK. Returns the new alert object." %} {% endtab %} {% tab title="400: Bad Request Missing parameter: archive" %} {% endtab %} {% endtabs %} `DELETE` `https://api.mediarithmics.com/v1/alerts/:alertId` #### Path Parameters | Name | Type | Description | | ------- | ---- | ------------------------- | | alertId | Int | ID of the alert to delete | {% tabs %} {% tab title="401: Unauthorized " %} {% endtab %} {% tab title="200: OK " %} {% endtab %} {% endtabs %} No other operations or modifications are permitted through the API. Users and integrators are restricted from editing any field other than the `archived` flag for an alert. # Alert configurations Alert Configurations allow customization of settings for different alert types within the Alerting module. Configurations are used to define specific behavior, thresholds, and rules associated with each alert type. They are dedicated resources. Each alert type can have a different set of configurations. By configuring alert types, you can tailor the behavior of the alerts to meet specific requirements. Alert configurations can be personalized for a specific organization. This means that each organization can have its own set of configuration values for the alert types. If a configuration is not set, a default value set by mediarithmics will be used. ## Allowed configurations Alert configurations are identified by combining three values: `config_key`, `organisation_id`, and `alert_type`. The `config_key` uniquely identifies a specific configuration setting, while the `organisation_id` and `alert_type` specify the organization and alert type to which the configuration belongs. Here is a list of available configuration keys and their sample values:
configconfig_valueEditable in the UI
volume_drops_segment_labels_mode:

alert_type: SEGMENT_VOLUME_DROP

Defines whether segment labels in volume_drops_segment_labels_ids config are whitelisted or blacklisted.

To edit in the UI, go to alerts on the segment list. For more information, see Using the Segments page.		blacklist | whitelist. Volume drops apply to all segments if not definedtrue
volume_drops_segment_labels_ids:

alert_type: SEGMENT_VOLUME_DROP

List of segment labels that will or won't receive volume drop alerts depending on the volume_drops_segment_labels_mode configuration

To edit in the UI, go to alerts on the segment list. For more information, see Using the Segments page.

Sample value: 1,2,3

Volume drops apply to all segments if not defined

true
volume_drops_threshold

alert_type: SEGMENT_VOLUME_DROP

Percentage of volume drop in a segment that triggers the alert.		Value is an integer such as 15. Defaults to 10false

initial_loading_not_starting_hours_threshold

alert_type: SEGMENT_INITIAL_LOADING

Threshold in hours after which an alert will be triggered if the initial loading hasn't started

Value is an integer & reprents a number of hours Defaults to 24false

initial_loading_running_too_long_hours_threshold

alert_type: SEGMENT_INITIAL_LOADING

Threshold in hours after which an alert will be triggered if the initial loading is taking more time than expected

Value is an integer & reprents a number of hours Defaults to 24false

initial_loading_records_error_threshold

alert_type: SEGMENT_INITIAL_LOADING

An alert will be triggered if the percentage of errors is above the one defined in this config.

Value is an integer. Defaults to 10false
## Accessing configurations You can list/edit configurations for your organizations by API. {% hint style="info" %} It is much easier to edit configurations through the UI when available. {% endhint %} ## Get all configurations set up for an organisation `GET` `https://api.mediarithmics.com/v1/alert_type_configs` If a configuration from the allowed list is not setup, it won't be returned by this call but fall back to the default platform value in usage. #### Query Parameters | Name | Type | Description | | -------------------------------------------------- | ---- | ---------------------- | | organisation\_id\* | Int | ID of the organisation | {% tabs %} {% tab title="200: OK " %} {% endtab %} {% tab title="401: Unauthorized " %} {% endtab %} {% endtabs %} ## Creates a new configuration entry `POST` `https://api.mediarithmics.com/v1/alert_type_configs/config_key=:configKey/organisation_id=:organisationId/alert_type=:alertType` #### Path Parameters | Name | Type | Description | | ------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------- | | configKey\* | string | Configuration key. Use the list of allowed configurations. Other keys won't have an impact. | | organisationId\* | string | ID of the organisation for which to create the configuration | | alertType\* | string | [AlertType](/alerting). Use the list of allowed configurations. | #### Request Body | Name | Type | Description | | ----------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------- | | config\_value\* | string | The value of the configuration. Use the list of allowed configurations for the correct value format, depending on your configuration. | {% tabs %} {% tab title="200: OK " %} {% endtab %} {% tab title="400: Bad Request Wrong parameter or the configuration already exists" %} {% endtab %} {% tab title="401: Unauthorized " %} {% endtab %} {% endtabs %} ## Change the value or archive a configuration `PUT` `https://api.mediarithmics.com/v1/alert_type_configs/config_key=:configKey/organisation_id=:organisationId/alert_type=:alertType` #### Path Parameters | Name | Type | Description | | ------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------- | | configKey\* | string | Configuration key. Use the list of allowed configurations. Other keys won't have an impact. | | organisationId\* | string | ID of the organisation for which to create the configuration | | alertType\* | string | [AlertType](/alerting). Use the list of allowed configurations. | #### Request Body | Name | Type | Description | | ----------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | config\_value\* | string | The value of the configuration. Use the list of allowed configurations for the correct value format, depending on your configuration. | | archived | Boolean | `true` to archive a configuration. `false` to reactivate it. For more information, see [Archived configurations](#archived-configurations). | {% tabs %} {% tab title="200: OK " %} {% endtab %} {% tab title="400: Bad Request " %} {% endtab %} {% tab title="401: Unauthorized " %} {% endtab %} {% tab title="404: Not Found " %} {% endtab %} {% endtabs %} ## Deletes a config `DELETE` `https://api.mediarithmics.com/v1/alert_type_configs/config_key=:configKey/organisation_id=:organisationId/alert_type=:alertType` #### Path Parameters | Name | Type | Description | | ------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------- | | configKey\* | string | Configuration key. Use the list of allowed configurations. Other keys won't have an impact. | | organisationId\* | string | ID of the organisation for which to create the configuration | | alertType\* | string | [AlertType](/alerting). Use the list of allowed configurations. | {% tabs %} {% tab title="200: OK " %} {% endtab %} {% tab title="404: Not Found " %} {% endtab %} {% tab title="401: Unauthorized " %} {% endtab %} {% tab title="400: Bad Request " %} {% endtab %} {% endtabs %} ## Archived configurations Configurations can be archived using the PUT request. An archived configuration is not used anymore by the platform (fallback to default value) but is easier to reactivate later. # Data visualisation {% hint style="success" %} **You can manage charts using** [**the query tool**](https://userguides.mediarithmics.io/data-studio/query-tool) **and** [**edit dashboards in the UI**](https://userguides.mediarithmics.io/basics/dashboards/custom-dashboards#editing-dashboards-in-the-ui)**.** Manipulating dashboards by API and in advanced mode can be useful in some advanced integrations, but will take longer. {% endhint %} With data visualisation, you can create dashboards : * In your datamart's home page * In your segments * In the standard segment builder
Those dashboards will answer questions like : * What is ingested in the platform? * Do we have moments where we ingest less data than on other days? * Who are my users? * How many active users do I have? On which channels? * What differentiates people in this segment from the rest of my users? * Do I have enough data to activate a segment? ## Quick start guide Create your first dashboard with our [Quickstart](/dashboards/quickstart) guide. ## Cookbook Speed up your learning curve with useful examples in our [Cookbook](#undefined). # Quickstart This page shows you how to start with API-based and advanced mode dashboard creation in mediarithmics. {% hint style="success" %} **You can manage charts using** [**the query tool**](https://userguides.mediarithmics.io/data-studio/query-tool) **and** [**edit dashboards in the UI**](https://userguides.mediarithmics.io/basics/dashboards/custom-dashboards#editing-dashboards-in-the-ui)**.** Manipulating dashboards by API and in advanced mode can be useful in some advanced integrations, but will take longer. {% endhint %} ## Advanced mode ### Full dashboard edition 1. Run your query[ in the query tool](https://userguides.mediarithmics.io/data-studio/query-tool) and save it as a technical query. Note the ID of the query. 2. In the computing console, go to **dashboards** and add/edit a dashboard 3. Choose a name and save your dashboard. 4. Switch to the **Advanced** tab. 5. Edit the JSON See the [DashboardContent](/dashboards/reference#dashboardcontent) object for a quick reference.

Dashboard advanced mode

### Chart edition 1. Run your query[ in the query tool](https://userguides.mediarithmics.io/data-studio/query-tool) and save it as a technical query. Note the ID of the query. 2. In the computing console, go to **dashboards** and add/edit a dashboard 3. Add or edit a chart and go to the **Advanced** tab 4. Edit the JSON and preview your changes. See the [Chart](/dashboards/reference#chart) object for a quick reference.

Chart advanced mode

## By API ### Step 1: Configure authentication This quickstart guide uses the [Long term access tokens](/resources/api-overview/authentication#long-term-access-tokens) authentication method. Choose and configure your own authentication method. For more information, see [Authentication](/resources/your-first-integration/make-the-first-bulk-import/api-basics/authentication). ### Step 2 : Prepare the OTQL queries you will use on your dashboard Your dashboard could use [OTQL queries](/querying-your-data/otql-queries) or [activities analytics queries](/querying-your-data/activities-analytics-queries) to retrieve data. We will use both in this tutorial, and OTQL queries need to be registered using the [Creating a query](/querying-your-data/otql-queries#managing-queries) endpoint. ## Creating a query `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/queries` Register an OTQL query in the platform We will create two OTQL queries for this tutorial. The first one counts the UserPoint in the datamart, the second one lists the devices they use. {% hint style="info" %} **For each query, note their ID.** You should adapt those queries to your data model if required. {% endhint %} {% code title="BODY payload for query 1" %} ```json5 { "datamart_id": {{datamartId}}, "query_language": "OTQL", "query_text": "select @count{} FROM UserPoint" } ``` {% endcode %} {% code title="BODY payload for query 2" %} ```json5 { "datamart_id": {{datamartId}}, "query_language": "OTQL", "query_text": "SELECT {agents {user_agent_info{form_factor @map}}} FROM UserPoint" } ``` {% endcode %} ### Step 3 : Create a dashboard registration You first create a [DashboardRegistration](/dashboards/reference#dashboardregistration) object to reference your dashboard and define where it is visible. ## Create a dashboard registration `POST` `https://api.mediarithmics.com/v1/dashboards` Here is a sample body payload for a home dashboard with all the important properties ```json { "title": "My awesome dashboards", "scopes": [ "home" ], "segment_ids": [], "builder_ids": [], "archived": false, "organisation_id": "{{orgId}}", "community_id": "{{communityId}}" } ``` ### Step 4 : Put content in your dashboard You can now upload content in your dashboard using the [DashboardContent](/dashboards/reference#dashboardcontent) object. ## Edit the content of a dashboard `PUT` `https://api.mediarithmics.com/v1/dashboards/:id/content?organisation_id=:organisation_id` Here is a sample body payload for a content using the queries we previously created. ```json { "sections": [ { "title": "", "cards": [ { "x": 0, "y": 0, "h": 3, "layout": "vertical", "w": 8, "charts": [ { "title": "User points", "type": "Metric", "dataset": { "type": "OTQL", "query_id": "{{ID of query 1}}" } }, { "title": "Device form factors", "type": "Bars", "dataset": { "type": "OTQL", "query_id": "{{ID of query 2}}" }, "options": {} } ] } ] } ] } ``` Go to your datamart's home page, and your dashboard is now displayed with the two charts we created ! ![](/files/dpfevFNxO3jO9Yiabm3O) # Dashboards Each dashboard is represented by a [DashboardRegistration](/dashboards/reference#dashboardregistration) object. It has a title, scopes, and a [DashboardContent](/dashboards/reference#dashboardcontent). Its content is composed of [sections and cards](/dashboards/sections-and-cards). Dashboards can be displayed on: * Datamart's home page with the `home` scope. * Segments page with the `segments` scope. * Standard segment builders with the `builders` scope. * A specific set of segments with the `segments` scope and segment IDs in the `segment_ids` property. * A specific set of standard segment `builders` with the builders scope and builder IDs in the `builder_ids` property. ![Multiple dashboards displayed on "home" scope](/files/5GbjLCJrMrO8cA5gNLIZ) You can have multiple dashboards at the same scope. If you have multiple dashboards to be displayed at a given scope, tabs will be created to switch between them. If you don't have multiple dashboards, it is displayed without any tab. {% hint style="success" %} It is best to have multiple dashboards than a single big one to prevent too many requests from being executed simultaneously. {% endhint %} See [REST resources](/dashboards/reference#rest-resources) for managing dashboards by API. ### Tip: Cloning a dashboard If you want to start from an existing dashboard, you can 1. Open the dashboard you want to clone on the computing console 2. Go to the Advanced tab and copy the whole JSON 3. Create a new dashboard 4. Go to the Advanced tab and paste the whole JSON 5. Start editing your new dashboard in the WISYWIG or the Advanced tab # Sections and cards A [DashboardContent](/dashboards/reference#dashboardcontent) is composed of sections and cards. * Each section has a title and cards disposed on a grid. * Each card is a white block organizing [charts](/dashboards/charts) horizontally or vertically.

Dashboard structure

The size and position of each card is defined by a 12 column grid, with as many rows as needed. Cards size and position are set with `{h,w,x,y}` properties : * `h` is the number of rows that the card takes. * `w` is the number of columns that the card takes * `{x,y}` are the coordinates of the top left corner of the card on the grid Here is a sample grid with five cards and their corresponding properties : ![A sample grid with 5 cards](https://lh5.googleusercontent.com/H6pY5bRARXZcGmoTzNq9UGfIrubGQa_J4moSjiIaeTUkxK9mheT0R2iIpBwc64dlI-EvCpDiiA5YZdKcomsyK4MAv7zARBuNaWF4Ep-n0POOJUPswlQWzxLLcZftXhwdNdKKGTDymj0) # Charts [DashboardContent](/dashboards/reference#dashboardcontent) has [sections and cards](/dashboards/sections-and-cards), which organize charts. Each chart has a title, colors (optional), a type, a dataset and options (optional). ![A Bars chart](/files/DGRwMT9XfOPqzZaJ4YAt) ![A Pie chart with legend](/files/REECrAVQOFsfKaqykOI0) Here is the JSON representation of a chart. ```json { "title": "Device form factors", // Could be an empty string "type": "Bars", // Bars || Pie || Metric || Radar "dataset": { // See Datasets and datasources page "type": "OTQL", "query_id": "50171" }, "options": {} // Options depending on the type of chart } ``` ## `Bars` charts Here are the different states of a `Bars` chart depending on its options and the dataset ![Columns and bars charts](/files/jfBziEH2jiSMIn5N5qyk) The available `options` for the `bars` chart are ```json "options": { "legend": { enabled: boolean; # Show or hide the legend. Defaults to FALSE. position: ‘bottom’ | ‘right’; # Display legend on the bottom or on the right of the chart. Defaults to bottom }, "colors": string[]; # Defaults to current theme colors "format": ‘count’ | ‘percentage’ | ‘index’; # Defaults to count. Pass percentage or index if dataset is compatible (comes from a percentage or index transformation) to automatically change labels, and tooltips and have more info. "drilldown": boolean; # Enables drill down if dataset is compatible. Defaults to FALSE "stacking": boolean; # Enables stacking if dataset is compatible. Prioritized over drilldown. Defaults to FALSE "type": ‘bar’ || ‘column’; # Set to ‘bar’ to display horizontal bars instead of columns. Defaults to ‘column’ "plot_line_value": int; # Set to draw a line at the specified value "hide_x_axis": boolean; # TRUE to hide X axis. Defaults to FALSE "hide_y_axis": boolean; # TRUE to hide Y axis. Defaults to FALSE "tooltip": {format: string} # Highcharts tooltip pointFormat. Defaults to “{point.y}” if dataset don’t have a -count field, “{point.y}% ({point.count})” otherwise. "big_bars": boolean; # TRUE displays large bars close to each other, FALSE smaller bars with more space between them. Defaults to TRUE } ``` See [Hicharts.filter](https://api.highcharts.com/highcharts/plotOptions.line.dataLabels.filter) for the `data_labels.filter` format. ## `Area` charts Here are the different states of an `Area` chart depending on its options and the dataset ![Area and line charts](/files/OXnaP1O5gjlZE2pGy5TT) The available `options` for the `area` chart are ```json "options": { "legend": { enabled: boolean; # Show or hide the legend. Defaults to FALSE. position: ‘bottom’ | ‘right’; # Display legend on the bottom or on the right of the chart. Defaults to bottom }, "colors": string[]; # Defaults to current theme colors "format": ‘count’ | ‘percentage’; # Defaults to count. Pass percentage if dataset is compatible (comes from a percentage transformation) to automatically change labels, and tooltips and have more info. "type": ‘area’ || ‘line’; # Set to ‘line’ to only display lines and no area. Defaults to ‘area’ "plot_line_value": int; # Set to draw a line at the specified value "hide_x_axis": boolean; # TRUE to hide X axis. Defaults to FALSE "hide_y_axis": boolean; # TRUE to hide Y axis. Defaults to FALSE "double_y_axis": boolean; # TRUE for each serie to have its own scale (and shows 2 vertical axis). Defaults to FALSE "tooltip": {format: string} # Highcharts tooltip pointFormat. Defaults to “{point.y}” if dataset don’t have a -count field, “{point.y}% ({point.count})” otherwise. } ``` See [Hicharts.filter](https://api.highcharts.com/highcharts/plotOptions.line.dataLabels.filter) for the `data_labels.filter` format. ## `Pie` charts Here are the different states of the `Pie` charts depending on its options and the dataset ![](/files/Gv35crLZaoOlaCJeY8Wz) The available options for the `pie` charts are ```json "options": { "legend": { "enabled": boolean; # Show or hide the legend. Defaults to FALSE. "position": ‘bottom’ | ‘right’; # Display legend on the bottom or on the right of the chart. Defaults to bottom }, "colors": string[]; # Defaults to current theme colors "drilldown": boolean; # Enables drill down if dataset is compatible. Defaults to FALSE "inner_radius": boolean; # True displays the chart as a Donut, false as a Pie. Defaults to TRUE "is_half": boolean; # True to only display half of a donut or half of a pie. Defaults to FALSE. "size": number | string | undefined; # The diameter of the pie relative to the plot area. Can be a percentage (75%) or a value. "data_labels": { "enabled": boolean; # If labels are displayed or not, default to TRUE "distance": int; # Distance between labels and the figure. Defaults to 10 if isHalf:true, else 0 "format": string; # Labels text. See Labels and string format. Defaults to “{point.percentage:.2f}%” if legend.enable:false, “{point.name} {point.percentage:.2f}%” otherwise. "filter": Highcharts.filter; # To hide labels under certain values. Defaults to undefined. }; "tooltip": {"format": string} # Highcharts tooltip pointFormat. Defaults to “{point.percentage:.2f}%” if legend.enable:false, “{point.name} {point.percentage:.2f}%” otherwise. } ``` See [Hicharts.filter](https://api.highcharts.com/highcharts/plotOptions.line.dataLabels.filter) for the `data_labels.filter` format. ## `Radar` charts Here are the different states of the `Radar` charts depending on its options and the dataset. ![](/files/M7ra4e1PgaCQW1NKbkiN) The available options for the `Radar` charts are ```json5 "options": { "legend": { "enabled": boolean; # Show or hide the legend. Defaults to FALSE. "position": ‘bottom’ | ‘right’; # Display legend on the bottom or on the right of the chart. Defaults to bottom }, "colors": string[]; # Defaults to current theme colors "format": ‘count’ | ‘percentage’; # Defaults to count. Pass percentage if dataset is compatible to automatically change labels, and tooltips. "data_labels": { "enabled": boolean; # If labels are displayed or not, default to TRUE "format": string; # Labels text. See Labels and string format. Defaults to “{point.y}” if dataset don’t have a -count field, “{point.y}%” otherwise "filter": Highcharts.filter; # To hide labels under certain values. Defaults to undefined. }; "tooltip": {"format": string} # Highcharts tooltip pointFormat. Defaults to “{point.y}” if dataset don’t have a -count field, “{point.y}% ({point.count})” otherwise. } ``` See [Hicharts.filter](https://api.highcharts.com/highcharts/plotOptions.line.dataLabels.filter) for the `data_labels.filter` format. ## `Metric` charts `Metric` charts are either displayed as percentages or as count, depending on its options. ![With percentage format](/files/p8vY1UmJQLL2SD21UfdC) ![With count format](/files/qbtjZBzgY8hPvcr4ALXe) Here are the available options for the `e` charts ``` "options": { "format": ‘count’ | ‘percentage’ | 'float'; # Defaults to count. If count, simply displays the number, else displays the number with % } ``` # Datasets and data sources A dataset is built based on at least one data source, and optional [transformations](/dashboards/transformations) and processed for visualisation in [Charts](/dashboards/charts). You can retrieve data from the following data sources : * [OTQL queries](/querying-your-data/otql-queries) * [Activities analytics](/querying-your-data/activities-analytics-queries) data cube * [Collection volumes](/advanced-usages/platform-monitoring/collection-volumes) data cube * [Resources usage](/advanced-usages/platform-monitoring/usage-reports) data cube * [Events ingestion monitoring](/advanced-usages/platform-monitoring/events-ingestion-monitoring) data cube Depending on the query you run and the transformations you apply, you can build different types of datasets. Here is a recap of which datasets are created from which data sources and transformations and the available visualisations for each. | Dataset | Created from | Compatible with | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Single number |

Queries

Transformations

None

|

Charts

Metric

Transformations

ratio to-list

| | Key / value |

Queries

Transformations

to-list to-percentages index

|

Charts

Pie Radar Bars

Transformations

to-percentages join index reduce

| | Key / value / buckets |

Queries

Transformations

None

|

Charts

Pie Bars

Transformations

to-percentagesreduce

| | Key / values |

Queries

None

Transformations

join

|

Charts

Bars Radar

Transformations

reduce

| ## Single number datasets Here is an example dataset with only one [OTQL](/querying-your-data/otql-queries) data source, that returns a number : ```json "dataset": { "type": "OTQL", "query_id": 666 // SELECT @count FROM UserPoint } ``` You can build the same kind of dataset with a different data source, like [activities analytics](/querying-your-data/activities-analytics-queries) : ```json5 "dataset": { "type": "activities_analytics", "query_json": { // This query returns the number of active users "dimensions": [], "metrics": [ { "expression": "users" } ] } } ``` Use this type of dataset in [Metric](/dashboards/charts#metric-charts) charts to display a single number. ![](/files/77uFQxVVw7T6VW1mGGoS) ## Key / value datasets Queries in the preceding paragraph were only returning numbers, but you can build key / value datasets with more complex queries like [OTQL bucket directives](/querying-your-data/otql-queries#aggregation-operations) and [activities analytics dimensions](/resources/data-cubes/creating-a-report#dimensions). ```json5 // key-value dataset built with an OTQL query "dataset": { "type": "OTQL", "query_id": 666 // SELECT {gender @map} FROM UserProfile } // key-value dataset built with an activities analytics query "dataset": { "type": "activities_analytics", "query_json": { // This query returns the number of active users per channel "dimensions": [ {"name": "channel_id"} ], "metrics": [ { "expression": "users" } ] } } ``` You can pass this kind of dataset in [Bars](/dashboards/charts#bars-charts), [Pie](/dashboards/charts#pie-charts) and [Radar](/dashboards/charts#radar-charts) charts to visualize the content. ![](/files/msPnSVXg0NdlpoZbkL9R) Key / value datasets also come from transformations like `to-list`, to create a list from multiple numbers. You can note the `series_title` property that gives you control over the title that will be displayed in tooltips and legends. ```json5 "dataset": { "type": "to-list", "sources": [ { "type": "OTQL", "query_id": "666", "series_title": "Female" }, { "type": "OTQL", "query_id": "777", "series_title": "Male" } ] } ``` ## Key / value / buckets datasets You can go further by adding up to three levels of buckets in your dataset with [multi-level bucket directives](/querying-your-data/otql-queries#aggregation-operations) and [activities analytics queries with multiple dimensions](/resources/data-cubes/creating-a-report#dimensions). ```json5 // key-value dataset built with an OTQL query "dataset": { "type": "OTQL", "query_id": 666 // SELECT {cat1 @map{cat2 @map{cat3 @map}}} FROM UserProfile } // key-value dataset built with an activities analytics query "dataset": { "type": "activities_analytics", "query_json": { // Number of active users per day per channel "dimensions": [ {"name": "date_yyyymmdd"} {"name": "channel_id"} ], "metrics": [ { "expression": "users" } ] } } ``` This can then be displayed with [Bars](/dashboards/charts#bars-charts) and [Pie](/dashboards/charts#pie-charts) charts, with drill down or multiple / stacking bars. ![](/files/zikyX7KkxC3hcATpYqLi) ## Key / values datasets The `join` transformation with multiple key / value datasets with common keys creates a single dataset with multiple values associated with each key. ```json5 "dataset": { "type": "join", "sources": [ { "type": "OTQL", "query_id": 777, // Select {interests @map} FROM UserPoint WHERE ... "series_title": "Group 1" }, { "type": "OTQL", "query_id": 666, // Select {interests @map} FROM UserPoint WHERE... "series_title": "Group 2" } ] } ``` The two groups can be displayed together in [Bars](/dashboards/charts#bars-charts) and [Radar](/dashboards/charts#radar-charts) charts to efficienly compare their data. ![](/files/KkyT8EOTftsq8Di9dWMq) ## Dataset JSON declaration A dataset is formed with a tree of data sources and transformations chained. ```json5 "dataset": { "type": "transformation-name", "sources": [ { "type": "transformation-name", "sources": [ { // OTQL data source "type": "OTQL", // ID of the OTQL query to call "query_id": Int, // Optional. Title of the series for tooltips and legends "series_title": String, // Optional. Datamart on which to run the query. // Defaults to current datamart "datamart_id": Int, // Optional. To adapt the query to the current scope // for example by adding current segment's query // when dashboard is executed on a segment // Defaults to TRUE // COMING SOON "adapt_to_scope": Boolean // Optional. To run the query in a specific precision // To be used when charts take too long to load and // a lower precision is accepted // Defaults to FULL_PRECISION "precision": "FULL_PRECISION" | "LOWER_PRECISION" | "MEDIUM_PRECISION" } ] }, { "type": "activities_analytics", // JSON representation of the activities analytics query "query_json": Object, // Optional. Title of the series for tooltips and legends "series_title": String, // Optional. Datamart on which to run the query. // Defaults to current datamart "datamart_id": Int, // Optional. To adapt the query to the current scope // for example by only selecting activities of users // that were in the segment while doing it // when dashboard is executed on a segment // Defaults to TRUE // COMING SOON "adapt_to_scope": Boolean }, { "type": "collection_volumes", // JSON representation of the activities analytics query "query_json": Object, // Optional. Title of the series for tooltips and legends "series_title": String }, { "type": "resources_usage", // JSON representation of the activities analytics query "query_json": Object, // Optional. Title of the series for tooltips and legends "series_title": String }, { "type": "data_ingestion", // JSON representation of the activities analytics query "query_json": Object, // Optional. Title of the series for tooltips and legends "series_title": String }, { "type": "data_file", // URI of the JSON data file containing data // Format "mics://data_file/tenants/1426/dashboard-1.json" "uri": String, // Path of the property in the JSON that should be used as dataset // This allows you to have multiple datasets in the same JSON file // Should use the JSONPath syntax. See https://jsonpath.com/ // For example, "$[0].components[1].component.data" "JSON_path": String, // Optional. Title of the series for tooltips and legends "series_title": String } ] } ``` {% hint style="info" %} * To learn about OTQL queries, go to [OTQL queries](/querying-your-data/otql-queries). * To learn about activities analytics queries, go to [Activities analytics queries](/querying-your-data/activities-analytics-queries). * To learn about collection volumes queries, go to [Collection volumes](/advanced-usages/platform-monitoring/collection-volumes). * For a list of available transformations, see [Transformations](/dashboards/transformations). {% endhint %} ### `series_title` property All data sources have a `series_title` property. This is useful when combining multiple sources together to set the title associated with each source. This will be reflected in tooltips and legends. Here is an example of a `Datamart` and a `Segment` data sources combined together. ```json5 "dataset": { "type": "join", "sources": [ { "type": "OTQL", "query_id": 777, // Select {interests @map} FROM UserPoint WHERE ... "series_title": "Segment" }, { "type": "OTQL", "query_id": 666, // Select {interests @map} FROM UserPoint WHERE... "series_title": "Datamart", "adapt_to_scope": false } ] } ``` ![](/files/wgYo0J9arHLIikfFttVX) ### `datamart_id` property All data sources have a `datamart_id` property allowing you to specify the datamart on which to run the query. It defaults to current datamart. This allows you to bring data for an other datamart or to create a dashboard at the community level that aggragates data from sub organisations. The user loading the dashboard should have the permissions to query the specified datamart or the chart will throw an error for this user. ### `adapt_to_scope` property By defaults, all data sources will try to adapt to the page on they are executed, with the `adapt_to_scope` property set to `TRUE`. The goal is to : * Filter data for the current segment when a dashboard is displayed on a segments page * Filter data based on the current query when a dashboard is displayed on a builder. For OTQL data sources : * On `home` scopes, nothing is changed and the query is run as is. * On `segments` scopes, the current segment's query is added at the end of the OTQL query. That means that **only OTQL queries `FROM UserPoint` will adapt to the scope**. * On `builders` scopes, the current query selected in the builder is added at the end of the OTQL query. That means that **only OTQL queries `FROM UserPoint` will adapt to the scope**. For activities analytics data sources : * On `home` and `builders` scopes, nothing changes and the query is run as is. * On `segments` scopes, activities are filters so that **only those of users that were in the segment while having the activity will be kept.** {% hint style="info" %} If the dashboard is meant to be displayed on `segments`, only build OTQL queries `FROM UserPoint` and activities analytics queries unless you want to retrieve data for the whole datamart. If the dashboard is meant to be displayed on `builders`, only build OTQL queries `FROM UserPoint` unless you want to retrieve data for the whole datamart. {% endhint %} # Using a data file data source You can use raw data stored in a file using the `data_file` datasource. It works this way : 1. You upload a data file in the platform using the `data_file` API 2. You reference this file in the dashboard as a dataset 3. This source can then be transformed and display like all other data sources. ## Upload your raw data in the platform Use the data\_file API to upload any JSON file containing your raw data. Its structure is not fixed. {% code title="Sample JSON" %} ```json5 { "id": "1", "name": "Demographics", "other_metadata_as_you_wish": "SEGMENT", "genders": [ { "key": "male", "value": 358 }, { "key": "female", "value": 66 } ], "ages": [ { "key": "18-24", "count": 277 }, { "key": "45-54", "count": 8 }, { "key": "65+", "count": 9 }, { "key": "25-34", "count": 12 }, { "key": "35-44", "count": 9 }, { "key": "55-64", "count": 3 } ], "total": 666 } ``` {% endcode %} There are two types of datasets that you can use : ### Key / value dataset A default key / value dataset is an array of key / value objects. ```json5 { "key_value_dataset": [ { "key": "Dimension 1", "value": 666 } ... { "key": "Dimension X", "value": 999 } ] } ``` {% hint style="info" %} The `value` property could be named differently, such as `count` or `value-1` by using the `series_title` property of the data source. {% endhint %} ### Number dataset ```json5 { ... "total": 666 } ``` ## Reference this file in your charts The whole structure of the dashboard is exactly the same as with other data sources. * For more information on datasets and datasources, see [Datasets and data sources](/dashboards/datasets-and-data-sources#dataset-json-declaration). * For a quick start on how to upload a dashboard, see [Quickstart](/dashboards/quickstart). The data source declaration is : ```json5 { "type": "data_file", // URI of the JSON data file containing data // Format "mics://data_file/tenants/1426/dashboard-1.json" "uri": String, // Path of the property in the JSON that should be used as dataset // This allows you to have multiple datasets in the same JSON file // Should use the JSONPath syntax. See https://jsonpath.com/ // For example, "$[0].components[1].component.data" "JSON_path": String, // Optional. Title of the series for tooltips and legends "series_title": String } ``` Here is an example with the JSON file we used previously ```json5 { "sections": [ { "title": "Section", "cards": [ { "x": 0, "charts": [ { "title": "Gender", "type": "Bars", "dataset": { "type": "data_file", "uri": "mics://data_file/tenants/XXX/dashboard-1.json", "JSON_path": "$.genders" } } ], "y": 0, "h": 3, "layout": "vertical", "w": 4 }, { "x": 4, "charts": [ { "options": { "legend": { "enabled": true, "position": "right" } }, "dataset": { "type": "data_file", "uri": "mics://data_file/tenants/XXX/dashboard-1.json", "JSON_path": "$.ages", "series_title": "count" }, "title": "Age range", "type": "Pie" } ], "y": 0, "h": 3, "layout": "vertical", "w": 5 }, { "x": 9, "charts": [ { "title": "Totals", "type": "Metric", "dataset": { "type": "data_file", "uri": "mics://data_file/tenants/XXX/dashboard-1.json", "JSON_path": "$.total" } } ], "y": 0, "h": 3, "layout": "vertical", "w": 3 } ] } ] } ``` You can use a `{SEGMENT_ID}` token in `uri` and/or `JSON_path` properties. It will be replaced by the current segment if the dashboard is loaded on a segment's page. If the dashboard is loaded at any other scope, the token will not be replaced. # Transformations Transformations process the result of the inner `sources` before it is passed to the [dataset](/dashboards/datasets-and-data-sources) or to the next transformation. ```json5 "dataset": { "type": "transformation-1", // Transformations always take a list of sources // Even if only one is used "sources": [ { // Transformations can be chained "type": "transformation-2" "sources": [ { // End the end we have one more // query data sources "type": "OTQL" ... } ] } ] } ``` ## `to-list` This puts multiple numbers from multiple queries into a [key / value dataset](/dashboards/datasets-and-data-sources#key-value-datasets). It is important to set the key of each number with the `series_title` property. It takes from one to any number of `sources`. ```json5 "dataset": { "type": "to-list", "sources": [ { "type": "OTQL", "query_id": "666", // SELECT @count{} FROM UserPoint WHERE... "series_title": "Unknown" }, { "type": "OTQL", "query_id": "777", // SELECT @count{} FROM UserPoint WHERE... "series_title": "With online account" }, { "type": "OTQL", "query_id": "888", // SELECT @count{} FROM UserPoint WHERE... "series_title": "With fidelity program" } ] } ``` ## `join` This joins two [key / value datasets](/dashboards/datasets-and-data-sources#key-value-datasets) into a [key / values](/dashboards/datasets-and-data-sources#key-values-datasets) dataset. It is important to set the name of each series with the `series_title` property. It takes from one to any number of `sources`. ```json5 "dataset": { "type": "join", "sources": [ { "type": "OTQL", "query_id": 666, // Select {interests @map} FROM UserPoint WHERE... "series_title": "Unkwnown" }, { "type": "OTQL", "query_id": 777, // Select {interests @map} FROM UserPoint WHERE... "series_title": "With fidelity program" } ] } ``` ## `to-percentages` This calculates the representation of each value in the complete dataset. Only one `sources` is accepted. You usually want to use the `format: percentage` option of the associated data visualisation to automatically change the labels and tooltips and formats to display `percentage% (count)` ```json5 "type": "Radar", "dataset": { "type": "to-percentages", "sources": [{ "type": "OTQL", "query_id": 666 // Select {interests @map} FROM UserPoint WHERE... }] }, "options": { "format": "percentage" } ``` ## `ratio` This calculates the ratio between two numbers (`source 1 / source 2 * 100)`. It only accepts two `sources` that should each return numbers. ```json5 "dataset": { "type": "ratio", "sources": [ { "type": "OTQL", "query_id": "666" // SELECT @count{} FROM UserPoint WHERE... }, // Returns 100k { "type": "OTQL", "query_id": "777" // SELECT @count{} FROM UserPoint } // Returns 200k ] } // Result is 100k/200k*100 = 50 ``` ## `index` This calculates the representation of values from a [key / value dataset](/dashboards/datasets-and-data-sources#key-value-datasets) in comparison to an other [key / value dataset](/dashboards/datasets-and-data-sources#key-value-datasets). For example, if 10% of the users in a segment viewed content associated with tag 1, while 5% of the users in the whole datamart viewed content associated with this same tag, the index of tag 1 in segment in comparison to the whole datamart is `10 / 5 * 100 = 200`. This is typically used to see which values are more/less represented in the first data source compared to the second one. An index above 100 means the value is more represented in the first data source than in the second, a value under 100 means the value is less represented in the first data source than in the second. This is usually represented in a `Bars` chart with a `plot_line_value` of 100 and an `index` format : ![](/files/PHOJxPCwEvfPnIhfaS8a) For each value in the first dataset, it automatically calculates its percentage representation in the first and the second source, then does the formula `source value (in percentages) / comparison value (in percentages) * 100` . ```json5 "type": "Bars", "dataset": { "type": "index", // Use limits like "limit:20" wisely // as the index will be calculated for each return value, then ordered. // If you only do a @map with a limit of 10 elements returned and you are asking // to show the top 10 indexes, you will have the top 10 indexes from the top 10 values // A value could be in position 20 by numbers, but in position 2 by index "sources": [ { // This query adapts to the current segment "type": "OTQL", "query_id": 666, // SELECT {interests @map} FROM UserPoint "series_title": "Segment" }, { // Same query without adapting to the current segment // and always returns data for the whole datamart "type": "OTQL", "query_id": 666, "series_title": "Datamart" "adapt_to_scope": false } ], "options": { "limit": 10 // Number of elements to display. 10 by default "order": "Ascending" | "Descending" // Descending by default // This means that indexes will only be calculated for values // representing 0.65% of values in source 1. "minimum_percentage" : 0.65 // 0 by default. Values between 0 and 100 } }, "options": { "type": "bar", "plotLineValue": 100, "format": "index" // So that the index is correctly displayed in tooltips } ``` ## `format-dates` This formats timestamps and date fields to the specified date format. Available date formats are [Moment.js date formats](https://momentjs.com/docs/#/displaying/). Use this transformation to allow the display of friendly dates to the user or to allow joining multiple data sources into the same dataset by putting returning dates in the same format. ```json5 "dataset": { "type": "format-dates", "sources": [ // Only one source allowed { "type": "OTQL", "query_id": "666" // SELECT {date @date_histogram} FROM UserEvent WHERE... }, ], "date_options": { "format": "YYYY-MM-DD" } } ``` Dates must be in the `2021-11-05T00:00:00.000Z` format or in timestamp to be formatted.\ \ Typical compatible queries are : * [OTQL queries](/querying-your-data/otql-queries#date-operators) returning timestamps or @date\_histogram. * [Activities analytics](/data-ingestion/activities-analytics) queries returning the date\_time dimension * [Collection volumes](/advanced-usages/platform-monitoring/collection-volumes) queries returning the date\_time dimension. ## `reduce` This transforms a key / value or key / values dataset into a single number to be displayed in [Metric charts](/dashboards/charts#metric-charts). * `avg` calculates the average of values * `count` calculates the number of values * `first` returns the first value * `last` returns the last value * `max` returns the maximum value * `min` returns the minimum value * `sum` returns the sum of all values ```json5 "type": "Metric", "dataset": { "type": "reduce", "sources": [{ "type": "OTQL", "query_id": 666 // Select {interests @map} FROM UserPoint WHERE... }], "reduce_options": { // avg || count || first || last || max || min || sum "type": "count" } }, ``` ## `get-decorators` This transforms identifiers such as channel IDs, compartment IDs and segment IDs into the corresponding channel names, compartment names and segment names. ```json5 // This returns channel IDs associated with the value "dataset": { "type": "OTQL", "query_id": "666" // SELECT {channel_id @map} FROM UserEvent WHERE... } } // This returns channel names associated with the value "dataset": { "type": "get-decorators", "sources": [ // Only one source allowed { "type": "OTQL", "query_id": "666" // SELECT {channel_id @map} FROM UserEvent WHERE... }, ], "decorators_options": { "model_type": "CHANNELS", // CHANNELS || COMPARTMENTS || SEGMENTS // Optional if the data source returns sub buckets, // to define the transformation for those sub buckets "buckets": { // Recursive "buckets": { "model_type": "SEGMENTS" } } } } ``` # Filters Dashboards can have filters in the top action bar.

Compartment and channel filter in the top action bar

You can set up filters only in advanced mode using the `available_filters` property of your [DashboardContent](broken://spaces/-MVWpcWULC-ugQNay59s). The user can select a value and all the queries in the dashboard adapt to the selected value. {% code title="JSON representation" %} ```json5 { // Using technical names of compartments, segments or channels // will result in IDs being automatically replaced by names in the UI "technical_name": String, "title": String, "values_retrieve_method": 'Query', // Only available value at the moment // OTQL query to retrieve list of selectable values // Use a query string, not the ID of a query "values_query": String, // How to adapt queries in the dashboard to the selected value(s) "query_fragments": [QueryFragment], "multi_select": Boolean, // If the user can select multiple values } ``` {% endcode %} A query fragment tells the dashboard how to adapt each query to the value(s) selected by the user. {% code title="JSON representation" %} ```json5 { // Any available data source such as 'activities_analytics' or 'OTQL' "type": String, // Only for OTQL type, chooses which queries should be transformed // Select 'ActivityEvent' to transform queries FROM ActivityEvent "starting_object_type": String, // The query part to add "fragment": String, } ``` {% endcode %} Here is a sample with a filter that enables the selection of compartments and an other for channels ```json5 { "available_filters": [ { "values_retrieve_method": "query", "values_query": "SELECT {compartment_id @map} FROM UserProfile", "technical_name": "compartments", "query_fragments": [ { "type": "OTQL", "starting_object_type": "UserPoint", "fragment": "profiles {compartment_id IN $values}" }, { "type": "OTQL", "starting_object_type": "UserProfile", "fragment": "compartment_id IN $values" } ], "multi_select": true, "title": "Data provider" }, { "values_retrieve_method": "query", "values_query": "SELECT {channel_id @map} FROM UserEvent", "technical_name": "channels", "query_fragments": [ { "type": "OTQL", "starting_object_type": "UserPoint", "fragment": "events {channel_id IN $values}" }, { "type": "OTQL", "starting_object_type": "UserEvent", "fragment": "channel_id IN $values" }, { "type": "activities_analytics", "fragment": [ { "dimension_name": "channel_id", "operator": "IN_LIST", "not": false, "expressions": "$values" } ] } ], "multi_select": true, "title": "Channels" } ], "sections": ... } ``` # Cookbook This page references recipes you can use to speed up your data visualization learning curve. ## Working with dates Use the [format-dates transformation](/dashboards/transformations#format-dates) to display dates in a user-friendly way. ### Single series ![](/files/5XqgzmjzOHFXwlAExccI) {% code title="Chart's JSON" %} ```json { "title": "Application events (last 6 months)", "type": "Bars", "dataset": { "type": "format-dates", "sources": [ { // @date_histogram query "type": "OTQL", "query_id": "666" } ], "date_options": { "format": "YYYY-MM-DD" } } } ``` {% endcode %} ### Multiple series You can do the same with the result of a join ![](/files/1xAZ6FGn6sVTpW9YtT16) {% code title="Chart's JSON" %} ```json5 { "title": "Montly events per channel or type-", "type": "Bars", "dataset": { "type": "format-dates", "sources": [ { // This works with a join but this can also work from a single source // without the join "type": "join", "sources": [ { "type": "OTQL", // Select {date @date_histogram } FROM UserEvent // WHERE channel_id = XXX "query_id": "666", "series_title": "Group 1" }, { "type": "OTQL", // Select {date @date_histogram } FROM UserEvent // WHERE channel_id = YYY "query_id": "777", "series_title": "Group 2" }, { "type": "OTQL", // Select {date @date_histogram } FROM UserEvent // WHERE channel_id = ZZZ "query_id": "888", "series_title": "Group 3" } ] } ], "date_options": { "format": "YYYY-MM-DD" // The date format we want to return } }, // Show the legend for a better event display "options": { "legend": { "enabled": true, "position": "bottom" }, "big_bars": false // Allow space between dates } } ``` {% endcode %} ### Combining different data sources With this technique, you can also combine data from different data sources where the date would be returned in different formats. ![](/files/vUcxRt4IDgcGvm5GDqRL) {% code title="Chart's JSON" %} ```json5 { "title": "Events", "type": "Bars", "dataset": { "type": "join", "sources": [ // Get some counts from activities analytics by month { "type": "activities_analytics", "query_json": { "dimensions": [ { "name": "date_YYYYMMDD" } ], "metrics": [ { "expression": "number_of_user_events" } ] }, "series_title": "activities_analytics" }, // Get other counts from OTQL by month with @date_histogram // and format the result in the same format as activities analytics { "type": "format-dates", "sources": [ { "type": "OTQL", "query_id": "666" } ], "series_title": "OTQL", "date_options": { "format": "YYYYMMDD" } } ] }, "options": { "hide_x_axis": true // We hide the x axis as there are a lot of values } } ``` {% endcode %} ## Collection volumes A nice way to display collection volumes is by showing the actual number of elements in the collection with a quick history of the volumes. ![](/files/x9wZml7UFDvub7Lugo1J) This can be achieved with two [Charts](/dashboards/charts) in the same [Card](/dashboards/sections-and-cards) : * A [Metric](/dashboards/charts#metric-charts) chart using an OTQL query in its [dataset](/dashboards/datasets-and-data-sources), such as `SELECT @count{} FROM UserPoint` * A [Bars](/dashboards/charts#bars-charts) chart using a Collection volumes query in its [dataset](/dashboards/datasets-and-data-sources). {% code title="Card's JSON" %} ```json5 { // Card display options. // Here we show a small card with a vertical layout "x": 0, "y": 0, "h": 2, "w": 3, "layout": "vertical", // The two charts in the card "charts": [ { // The number of UserPoint as a metric "title": "UserPoint", "type": "Metric", "dataset": { "type": "OTQL", "query_id": "666" // SELECT @count FROM UserPoint } }, { // Bars showing a history of the number of UserPoint "title": "", "type": "Bars", "dataset": { // We do use the format-dates transformation to display // friendly dates instead of timestamps "type": "format-dates", "sources": [ { "type": "collection_volumes", "query_json": { "dimensions": [ { "name": "date_time" }, { "name": "collection" } ], "dimension_filter_clauses": { "operator": "AND", "filters": [ { "dimension_name": "datamart_id", "operator": "EXACT", "expressions": [ YOUR_DATAMART_ID ] }, { "dimension_name": "collection", "operator": "EXACT", "expressions": [ "UserPoint" ] } ] }, "metrics": [ { "expression": "count" } ] } } ], "date_options": { "format": "YYYY-MM-DD HH:mm" } }, // We hide axis to have a nice little chart only showing trends // with the ability for the user to get values by hovering the bars "options": { "hide_x_axis": true, "hide_y_axis": true } } ] } ``` {% endcode %} Another tip when showing collection volumes is to replace lists of metrics with bar charts. This makes it easier to visualize proportions, especially if you have a reference number like the total number of UserPoint. ![](/files/qYfJQK4QvvHQZg6KozoV)![](/files/HucWHh52YD17bgSGxOme) {% code title="Card's JSON" %} ```json5 { "x": 0, "charts": [ { "title": "UserPoint", "type": "Metric", "dataset": { "type": "OTQL", "query_id": "666" } }, { "title": "Activability", "type": "Bars", "dataset": { "type": "to-list", "sources": [ { "type": "OTQL", "query_id": "111", "series_title": "Total UserPoint" }, { "type": "OTQL", "query_id": "222", "series_title": "With accounts" }, { "type": "OTQL", "query_id": "333", "series_title": "With emails" }, { "type": "OTQL", "query_id": "444", "series_title": "With web cookies" }, { "type": "OTQL", "query_id": "555", "series_title": "With apple Mobile ID" }, { "type": "OTQL", "query_id": "666", "series_title": "With google Mobile ID" } ] }, "options": { "type": "bar", "hide_y_axis": true, "colors": [ "#333333" ] } } ], "y": 0, "h": 5, "layout": "vertical", "w": 4 } ``` {% endcode %} ## Comparing audiences You may want to compare a particular audience you are building or that's been built to the whole datamart or to a specific reference audience. For example to answer the question *Do users in this audience have different viewing modes than all users ?* you can build a dashboard at the builders and/or the segments scope with : * The number of UserPoint visiting through each viewing mode for your audience * The number of UserPoint visiting through each viewing mode for all users * [Index](/dashboards/transformations#index) calculation to visualize which viewing modes are more/less used in your audience ![In the following example, people in the audience are more likely to get touched in LIVE events that in replays.](/files/coXlfpncGJ1SNjagd75P) ```json5 { "title": "Viewing modes", "type": "Bars", "dataset": { "type": "index", "sources": [ { "type": "OTQL", "query_id": "666", // SELECT {events {session_mode @map}} FROM UserPoint "series_title": "Segment" }, { "type": "OTQL", "query_id": "666", // SELECT {events {session_mode @map}} FROM UserPoint "series_title": "Datamart", "adapt_to_scope": false } ], "options": { "limit": 10, "minimum_percentage": 1, "sort": "Descending" } }, "options": { "type": "bar", "plot_line_value": 100, "format": "index" } } ``` ## Working with channel, compartments and segments When doing any chart that returns channels, compartments or segments, you will usually want to display names instead of IDs in the UI. For this, use the [get-decorators](/dashboards/transformations#get-decorators) transformation to replace IDs with names. ![Display channel names instead of channel IDs](/files/wy1pOHJKGYgISmdCE9QW) ```json { "title": "Data by channels", "type": "Bars", "dataset": { "type": "get-decorators", "sources": [ { "type": "to-percentages", "sources": [ { "type": "OTQL", "query_id": "666" // SELECT { channel_id @map} FROM UserEvent WHERE ... } ] } ], "decorators_options": { "model_type": "CHANNELS" } }, "options": { "format": "percentage" } } ``` ## Using the reduce transformation to display a @cardinality OTQL query as a metric [@cardinality OTQL queries](/querying-your-data/otql-queries#metrics-directives) return a key / value dataset. In lots of cases, this dataset only has one value but can't be displayed as a metric as it is not in the correct format. We can use [the reduce transformation](/dashboards/transformations#reduce) to put the dataset in the correct format. ![](/files/pQayugjGgPmQDs3qfCyi) ```json { "title": "Number of different event names retrieved", "type": "Metric", "dataset": { "type": "reduce", "sources": [ { "type": "OTQL", "query_id": "666" // SELECT {nature @cardinality} FROM ActivityEvent } ], "reduce_options": { "type": "first" } } } ``` # Reference ## REST resources ### Dashboard registration Dashboard registration endpoints let you manage dashboards and where they are displayed. Those endpoint usually take or return [DashboardRegistration](#dashboardregistration) objects. ## List all dashboard registrations for a specific organisation `GET` `https://api.mediarithmics.com/v1/dashboards?organisation_id=:organisation_id` Returns a [paginated resource list wrapper](/resources/api-overview#resource-list-wrapper-paginated-apis) of [DashboardRegistration](#dashboardregistration) objects. ## Create a dashboard registration. `POST` `https://api.mediarithmics.com/v1/dashboards` Receives a [DashboardRegistration](#dashboardregistration) object as body. ## Edit a dashboard registration `PUT` `https://api.mediarithmics.com/v1/dashboards/:id?organisation_id=:organisation_id` Receives a [DashboardRegistration](#dashboardregistration) object as body. ## Get a specific dashboard registration `GET` `https://api.mediarithmics.com/v1/dashboards/:id?organisation_id=:organisation_id` Returns a [DashboardRegistration](#dashboardregistration) object. ## Delete a dashboard registration `DELETE` `https://api.mediarithmics.com/v1/dashboards/:id?organisation_id=:organisation_id` ### Dashboard content Dashboard content endpoints let you manage the sections, cards and charts in a specific dashboard. ## Get the content of a dashboard. Returns a DashboardContentWrapper. `GET` `https://api.mediarithmics.com/v1/dashboards/:id/content?organisation_id=:organisation_id` ## Edit the content of a dashboard with the specified DashboardContent. `PUT` `https://api.mediarithmics.com/v1/dashboards/:id/content?organisation_id=:organisation_id` ## DashboardRegistration This object represents a dashboard and where it should be displayed. {% code title="JSON representation" %} ```json5 { "title": String, "scopes": [Scope] "segment_ids": [String], "builder_ids": [String], "archived": Boolean, "dashboard_content_id": String, "organisation_id": String, "community_id": String, // Readonly fields "created_ts": Timestamp, "created_by": String, "last_modified_ts": Timestamp, "last_modified_by": String } ``` {% endcode %} #### Fields `title` **string** The title of the dashboard, as displayed in the UI `scopes[]` **enum(`home`,`segments`,`builders`)** The list of scopes where the dashboard is visible. Mandatory, but can be an empty array. `segment_ids[]` **string** When `scopes` property contains `segments`, you can specify a list of segment IDs to only display the dashboard on those specific segments. Mandatory, but can be an empty array. `builder_ids[]` **string** When `scopes` property contains `builders`, you can specify a list of standard segment builder IDs to only display the dashboard on those specific builders. Mandatory, but can be an empty array. `archived` **boolean** Set to `true` to hide a dashboard from the UI without deleting it. `dashboard_content_id` **string** Identifier of the [DashboardContentWrapper](#dashboardcontentwrapper) that's been associated with the dashboard registration. `community_id` **string** ID of the community on which the dashboard is visible. `organisation_id` **string** ID of the organisation on which the dashboard is visible. Must be on the `community_id` community. `created_ts` **timestamp** When the dashboard registration was created. **ReadOnly**. `created_by` **user ID** By who the dashboard registration was created. **ReadOnly**. `last_modified_ts` **timestamp** When the dashboard registration was last modified. Not updated when dashboard content is updated as DashboardContent object has its own `created_ts` field. **ReadOnly**. `last_modified_by` **user ID** By who the dashboard was last modified. Not updated when dashboard content is updated as DashboardContent object has its own `created_by` field. **ReadOnly**. ## DashboardContentWrapper This object is returned when doing a `GET` request to get the content of a dashboard. It returns useful metadata as well as dashboard's content {% code title="JSON representation" %} ```json5 { "id": String, "content": DashboardContent "organisation_id": String, "created_ts": Timestamp, "created_by": String } ``` {% endcode %} #### Fields `id` **string** Content's identifier, used in [DashboardRegistration](#dashboardregistration) to associate a dashboard and its content. `content` **object(**[**DashboardContent**](#dashboardcontent)**)** Dashboard's JSON representation. `organisation_id` **string** ID of the organisation on which the dashboard is visible. `created_ts` **timestamp** When the dashboard content was created. **ReadOnly**. `created_by` **user ID** By who the dashboard content was created. **ReadOnly**. {% hint style="info" %} Each time you do a PUT request to update a dashboard's content, a new `DashboardContentWrapper` resource is created with a new ID, and the new resource is associated with the dashboard registration. That means the `created_ts` and `created_by` fields also represent the `last_modified_ts` and `last_modified_by` fields you are used to see. {% endhint %} ## DashboardContent This object represents the sections, cards and charts displayed in a dashboard. {% code title="JSON representation" %} ```json5 { "available_filters": [Filter] "sections": [Section] } ``` {% endcode %} `available_filters[]` **object(**[**Filter**](#filter)**)** The list of filters activated for the dashboard. `sections[]` **object(**[**Section**](#section)**)** The list of sections inside a dashboard. ## Filter A filter is displayed at the top of a dashboard. The user can select a value and all the queries in the dashboard adapt to the selected value {% code title="JSON representation" %} ```json5 { // Using technical names of compartments, segments or channels // will result in IDs being automatically replaced by names in the UI "technical_name": String, "title": String, "values_retrieve_method": 'Query', // Only available value at the moment // OTQL query to retrieve list of selectable values // Use a query string, not the ID of a query "values_query": String, // How to adapt queries in the dashboard to the selected value(s) "query_fragments": [QueryFragment], "multi_select": Boolean, // If the user can select multiple values } ``` {% endcode %} A query fragment tells the dashboard how to adapt each query to the value(s) selected by the user. {% code title="JSON representation" %} ```json5 { // Any available data source such as 'activities_analytics' or 'OTQL' "type": String, // Only for OTQL type, chooses which queries should be transformed // Select 'ActivityEvent' to transform queries FROM ActivityEvent "starting_object_type": String, // The query part to add "fragment": String, } ``` {% endcode %} ## Section A section gives you a title and a new grid to display cards. {% code title="JSON representation" %} ```json5 { "title": String, "cards": [Card], } ``` {% endcode %} `title` **string** The title of the section, displayed in the UI. `cards[]` **object(**[**Card**](#card)**)** The list of cards to display in the section. ## Card A white zone in the section, that displays and organizes charts. {% code title="JSON representation" %} ```json5 { "x": Int, "y": Int, "h": Int, "w": Int, "layout": "vertical" || "horizontal", "charts": [Chart], } ``` {% endcode %} `x,y,h,w` **int** The position of the card in the section's grid. See [Sections, cards and charts](/dashboards/sections-and-cards) for a guide on how to use it. `layout` **enum(`vertical`, `horizontal`)** Wether charts in the card will stack horizontally or vertically. `charts[]` **object(**[**Chart**](#chart)**)** The list of charts to display in the card. ## Chart A chart displayed in a card. {% code title="JSON representation" %} ```json5 { "title": String, "type": "Pie" || "Bars" || "Radar" || "Metric" || "Area", "dataset": Dataset, "options": PieOptions || BarsOptions || RadarOptions || MetricOptions || AreaOptions } ``` {% endcode %} `title` **string** The chart's title, displayed in the UI. `type` **enum(`Pie`, `Bars`, `Radar`, `Metric`)** The type of chart to display `colors[]` **string** *Optional*. You can use this property to override default chart colors, which are defined by the theme of the site. Define as many color codes (in `#FFFFFF` format) as needed by the chart. `dataset` **object(Dataset)** How to get data for the chart `options` **object(**[**`PieOptions`**](/dashboards/charts#pie-charts)**,** [**`BarsOptions`**](/dashboards/charts#bars-charts)**,** [**`RadarOptions`**](/dashboards/charts#radar-charts)**,** [**`MetricOptions`**](/dashboards/charts#metric-charts)**,** [**`AreaOptions`**](/dashboards/charts#area-charts)**)** *Optional*. Options specific to the type of chart that has been selected. {% hint style="info" %} * For more information on the different chart types and there options, see [Charts](/dashboards/charts). * For more information on how datasets are built, see [Datasets and data sources](/dashboards/datasets-and-data-sources). {% endhint %} # Audience segmentation # Audience features {% hint style="success" %} Audience features you create are available in the navigator, in **Audience** > **Builders** > **Standard**. {% endhint %} Let's take the following query as an example: ```sql SELECT @count{} FROM UserPoint WHERE events { nature = "$transaction_confirmed" and date >= $date and products {brand in $brand and name in $name} } ``` It is technically an OTQL query based on your schema in which you registered parameters**.** In this case, it represents a business feature that may be used regularly by your users: selecting UserPoint that perform a transaction in a particular date range and for specific products. Users will see this selector in the builder instead of an OTQL query. ![](/files/-MXffizH0-0WAWIV6Xxf) This audience feature will automatically be combined with other audience features, which the user can select to create segments. ## The process Users will have access to the standard segment builder if at least one audience feature is set up. But you need multiple ones to create value for users. Good knowledge of the schema and the queries that are usually created in your datamart is important for the success of this feature. Here is a sample process to follow to enable audience features and create value for your users: 1. Know your schema. What is it optimized for? Which queries are regularly created? What are your actual segments and what are their queries? You should be able to create a list of useful audience features with these pieces of information. 2. Set up audience features, in the UI and/or by script. 3. Monitor usage and update audience features regularly! ## Folders You can store audience features in folders. Any audience feature without a folder will be assigned to the root folder. Here are some examples. ## List folders `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_feature_folders` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------- | | datamart\_id | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "first_result": 0, "count": 0, "max_results": 0, "status": "ok", "data": [ { "id": "string", "children_ids": [ "string" ], "audience_features_ids": [ "string" ], "parent_id": "string", "datamart_id": "string", "name": "string" } ], "total": 0 } ``` {% endtab %} {% endtabs %} ## List one folder `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_feature_folders/{audience_feature_folders_id}` #### Path Parameters | Name | Type | Description | | ------------------------------ | ------- | ---------------------- | | audience\_feature\_folders\_id | integer | The ID of the folder | | datamart\_id | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "string", "children_ids": [ "string" ], "audience_features_ids": [ "string" ], "parent_id": "string", "datamart_id": "string", "name": "string" } } ``` {% endtab %} {% endtabs %} ## Create a folder `POST` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_feature_folders` #### Path Parameters | Name | Type | Description | | ---------------------------------------------- | ------- | ---------------------- | | datamart\_id\* | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | -------------------------------------- | ------ | ----------------------------------------------- | | children\_ids | array | IDs of folder's children | | audience\_features\_ids | array | IDs of audience features attached to the folder | | parent\_id | string | The ID of the folder's parent | | datamart\_id | string | The ID of the datamart | | name\* | string | The name of the folder | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ```javascript // Create a folder payload { "children_ids": [ "string" ], "audience_features_ids": [ "string" ], "parent_id": "string", "datamart_id": "string", "name": "string" } ``` ## Edit a folder `PUT` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_feature_folders/{audience_feature_folders}` #### Path Parameters | Name | Type | Description | | ---------------------------------------------------------------- | ------- | ------------------------------------- | | datamart\_id\* | integer | The ID of the datamart | | audience\_feature\_folders\_id\* | integer | The ID of the audience feature folder | #### Request Body | Name | Type | Description | | -------------------------------------- | ------ | ----------------------------------------------- | | children\_ids | array | IDs of folders's children | | audience\_features\_ids | array | IDs of audience features attached to the folder | | parent\_id | string | The ID of the folder's parent | | datamart\_id | string | The ID of the datamart | | name\* | string | The name of the folder | {% tabs %} {% tab title="200 " %} ``` { "status": "ok", "data": { "id": "string", "children_ids": [ "string" ], "audience_features_ids": [ "string" ], "parent_id": "string", "datamart_id": "string", "name": "string" } } ``` {% endtab %} {% endtabs %} ```javascript // Editing a folder payload { "children_ids": [ "string" ], "audience_features_ids": [ "string" ], "parent_id": "string", "datamart_id": "string", "name": "string" } ``` ## Audience features ## List audience features `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_features` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------- | | datamart\_id | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "first_result": 0, "total": 0, "count": 0, "data": [ { "object_tree_expression": "string", "description": "string", "id": "string", "variables": [ { "field_name": "string", "data_type": "string", "reference_model_type": "string", "type": "string", "parameter_name": "string", "path": [ "string" ], "reference_type": "string", "directive": "string", "container_type": "string" } ], "token": "string", "datamart_id": "string", "addressable_object": "string", "name": "string", "folder_id": "string", "creation_date": 0 } ], "max_results": 0, "status": "ok" } ``` {% endtab %} {% endtabs %} ## Get an audience feature `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_features/{audience_feature_id}` #### Path Parameters | Name | Type | Description | | --------------------- | ------- | ------------------------------ | | audience\_feature\_id | integer | The ID of the audience feature | | datamart\_id | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "object_tree_expression": "string", "description": "string", "id": "string", "variables": [ { "field_name": "string", "data_type": "string", "reference_model_type": "string", "type": "string", "parameter_name": "string", "path": [ "string" ], "reference_type": "string", "directive": "string", "container_type": "string" } ], "token": "string", "datamart_id": "string", "addressable_object": "string", "name": "string", "folder_id": "string", "creation_date": 0 } } ``` {% endtab %} {% endtabs %} ## Create an audience feature `POST` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_features` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------- | | datamart\_id | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | ------------------------ | ------ | -------------------------------------------------------------------------------------------------------------- | | object\_tree\_expression | string | The **WHERE** statement of the query associated to the audience feature | | description | string | The description of your audience feature | | datamart\_id | string | The ID of the datamart | | addressable\_object | string | The **SELECT** statement of the query associated to the audience feature. It must always set at **UserPoint**. | | name | string | The name of the audience feature | | folder\_id | string | The ID of folder where the audience feature is stored | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "object_tree_expression": "string", "description": "string", "id": "string", "variables": [ { "field_name": "string", "data_type": "string", "reference_model_type": "string", "type": "string", "parameter_name": "string", "path": [ "string" ], "reference_type": "string", "directive": "string", "container_type": "string" } ], "token": "string", "datamart_id": "string", "addressable_object": "string", "name": "string", "folder_id": "string", "creation_date": 0 } } ``` {% endtab %} {% endtabs %} ```javascript // Creating an audience feature payload { "object_tree_expression": "string", "description": "string", "datamart_id": "string", "addressable_object": "string", "name": "string", "folder_id": "string" } ``` ## Edit an audience feature `PUT` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_features/{audience_feature_id}` #### Path Parameters | Name | Type | Description | | --------------------- | ------- | -------------------------------------- | | audience\_feature\_id | integer | The ID of the audience feature to edit | | datamart\_id | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | ------------------------ | ------ | -------------------------------------------------------------------------------------------------------------- | | object\_tree\_expression | string | The **WHERE** statement of the query associated to the audience feature | | description | string | The description of your audience feature | | datamart\_id | string | The ID of the datamart | | addressable\_object | string | The **SELECT** statement of the query associated to the audience feature. It must always set at **UserPoint**. | | name | string | The name of the audience feature | | folder\_id | string | The ID of folder where the audience feature is stored | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "object_tree_expression": "string", "description": "string", "id": "string", "variables": [ { "field_name": "string", "data_type": "string", "reference_model_type": "string", "type": "string", "parameter_name": "string", "path": [ "string" ], "reference_type": "string", "directive": "string", "container_type": "string" } ], "token": "string", "datamart_id": "string", "addressable_object": "string", "name": "string", "folder_id": "string", "creation_date": 0 } } ``` {% endtab %} {% endtabs %} ```javascript // Editing an audience feature payload { "object_tree_expression": "string", "description": "string", "datamart_id": "string", "addressable_object": "string", "name": "string", "folder_id": "string" } ``` ## OTQL query rules ### Parameters Create query parameters using the `$parameter_name`syntax. Example: ```sql // The gender will be selectable by the user SELECT @count{} FROM UserPoint where profiles {gender in $gender} // The date will be selectable by the user, // but the nature will always be $transaction_confirmed SELECT @count{} FROM UserPoint where events { nature = "$transaction_confirmed" and date >= $date } ``` The field in the audience feature will have the name you enter after the `$` . Spaces in field name are not authorized. The type of selector in the audience feature is automatically chosen based on the field type If you want to be able to select several values in a field, use keyword `in` instead of the classic `==`. ```sql // User will only be able to select one gender SELECT @count{} FROM UserPoint where profiles {gender == $gender} // User will be able to select multiple gender values SELECT @count{} FROM UserPoint where profiles {gender in $gender} ``` You can create parameters for frequency requests with the `@ScoreSum` directive: ```sql // User will be able to select the minimum number of transactions SELECT @count{} FROM UserPoint WHERE events@ScoreSum(min: $frequency) { purchase in $user_purchase } ``` # Segment builders You have access to three tools to segment your audience using queries: * Leverage [audience features](/advanced-usages/audiences/audience-features) to build your queries in the standard segment builder (**Audience** > **Builders** > **Standard**). Once set up, this is the preferred solution for fast queries building and visualising the segment in a dashboard before saving it. * Drag and drop fields from your schema into a visual OTQL query builder with the advanced segment builder (**Audience** > **Builders** > **Advanced**). It doesn't require any setup but requires knowledge about the schema. May not be the best option for casual users. * Build OTQL queries directly in **Data Studio** > **Query tool.** This requires a solid knowledge of your schema and [OTQL](/querying-your-data/otql-queries). ## Standard segment builder set up You enable this feature when you set up at least: * One segment builder * One [audience feature](/advanced-usages/audiences/audience-features). You can set up multiple segment builder to create templates once you identify common segment queries that you often use. Each segment builder has a list of default audience features that are automatically used in it. You can create and edit segment builders through the UI by going to **Settings** > **Datamart** > **Your datamart** > **Segment builders.** You can also manage them by API. ## List standard segment builders `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------- | | datamart\_id | integer | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "first_result": 0, "count": 0, "max_results": 0, "status": "ok", "data": [ { "id": "string", "children_ids": [ "string" ], "audience_features_ids": [ "string" ], "parent_id": "string", "datamart_id": "string", "name": "string" } ], "total": 0 } ``` {% endtab %} {% endtabs %} ## Get a standard segment builder `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders/{audience_builders_id}` #### Path Parameters | Name | Type | Description | | ---------------------- | ------- | -------------------------------------------------- | | datamart\_id | integer | The ID of the datamart | | audience\_builders\_id | integer | The ID of standard segment builder you want to get | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Create a standard segment builder `POST` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------- | | datamart\_id | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | --------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | datamart\_id | string | The ID of the datamart | | demographics\_features\_ids | array | Array of string: the IDs of audience features you want to link to your standard segment builder. These audience features will always be selected in the builder. | | name | string | Name of the standard segment builder | {% tabs %} {% tab title="200 " %} ```javascript { "first_result": 0, "count": 0, "max_results": 0, "status": "ok", "data": [ { "id": "string", "children_ids": [ "string" ], "audience_features_ids": [ "string" ], "parent_id": "string", "datamart_id": "string", "name": "string" } ], "total": 0 } ``` {% endtab %} {% endtabs %} ## List standard segment builders `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ------------ | | datamart\_id | integer | GLLBBPYmcMbn | ## Get a standard segment builder `GET` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders/{audience_builders_id}` #### Path Parameters | Name | Type | Description | | ---------------------- | ------- | ------------ | | datamart\_id | integer | EACXzNpxesGA | | audience\_builders\_id | integer | y1iztafbxNOl | ## Create a standard segment builder `POST` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ------------ | | datamart\_id | integer | UpWZubSJk7SC | #### Request Body | Name | Type | Description | | --------------------------- | ------ | ------------ | | datamart\_id | string | ZdvK39oPj25k | | demographics\_features\_ids | array | Zu89g8D3qu6m | | name | string | bpFIx0HWRbci | ```javascript // Create a standard segment builder payload { "datamart_id": "string", "demographics_features_ids": [ "string" ], "name": "string" } ``` {% hint style="danger" %} You cannot create more than 20 standard segment builder instances per datamart. {% endhint %} ## Edit a standard segment builder `PUT` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders/{audience_builders_id}` #### Path Parameters | Name | Type | Description | | ---------------------- | ------- | ---------------------------------------------- | | audience\_builders\_id | integer | The ID of the standard segment builder to edit | | datamart\_id | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | --------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | datamart\_id | string | The ID of the datamart | | demographics\_features\_ids | string | Array of string: The IDs of audience features you want to link to your standard segment builder. These audience features will always be selected in the builder. | | name | string | The name of the standard segment builder | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Edit a standard segment builder `PUT` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/audience_builders/{audience_builders_id}` #### Path Parameters | Name | Type | Description | | ---------------------- | ------- | ------------ | | audience\_builders\_id | integer | aIhsvJpgK6ny | | datamart\_id | integer | TTSN6yxr96WI | #### Request Body | Name | Type | Description | | --------------------------- | ------ | ------------ | | datamart\_id | string | N2N4Y3WHqATi | | demographics\_features\_ids | string | 1EsZV25xtbUt | | name | string | xsAiWSvKY437 | ```javascript // Edit a standard segment builder payload { "datamart_id": "string", "demographics_features_ids": [ "string" ], "name": "string" } ``` ### Dashboards API This API helps you upload dashboards. ## List dashboards for standard segment builder `GET` `https://api.mediarithmics.com/v1/data_file/data?uri=mics://data_file/tenants/{organisation_id}/dashboards/{datamart_id}/AUDIENCE_BUILDER-{audience_builder_id}.json` #### Path Parameters | Name | Type | Description | | --------------------- | ------- | ----------------------------------------------------------------------------- | | organisation\_id | integer | The ID of the organisation | | datamart\_id | integer | The ID of the datamart | | audience\_builder\_id | integer | The ID of the standard segment builder on which you want to upload dashboards | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Create & edit dashboards for standard segment builder `PUT` `https://api.mediarithmics.com/v1/data_file/data?uri=mics://data_file/tenants/{organisation_id}/dashboards/{datamart_id}/AUDIENCE_BUILDER-{audience_builder_id}.json` #### Path Parameters | Name | Type | Description | | --------------------- | ------- | ----------------------------------------------------------------------------- | | organisation\_id | integer | The ID of the organization | | datamart\_id | integer | The ID of the datamart | | audience\_builder\_id | integer | The ID of the standard segment builder on which you want to upload dashboards | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## List dashboards for standard segment builder `GET` `https://api.mediarithmics.com/v1/data_file/data?uri=mics://data_file/tenants/{organisation_id}/dashboards/{datamart_id}/AUDIENCE_BUILDER-{audience_builder_id}.json` #### Path Parameters | Name | Type | Description | | --------------------- | ------- | ------------ | | organisation\_id | integer | xpvJF5g4SpwP | | datamart\_id | integer | uqK1nIMmTAn6 | | audience\_builder\_id | integer | cZe7evLowSbp | ## Create & edit dashboards for standard segment builder `PUT` `https://api.mediarithmics.com/v1/data_file/data?uri=mics://data_file/tenants/{organisation_id}/dashboards/{datamart_id}/AUDIENCE_BUILDER-{audience_builder_id}.json` #### Path Parameters | Name | Type | Description | | --------------------- | ------- | ------------ | | organisation\_id | integer | UFX1xf9M0BWy | | datamart\_id | integer | uhpUbCPUJ0Gm | | audience\_builder\_id | integer | 5Rm1gDwXgxFe | ```javascript // Create a dashboard payload example [ { "id": "1", "name": "Standard segment builder", "type": "AUDIENCE_BUILDER", "datamart_id": "xxxx", "components": [ { "layout": { "h": 1, "static": false, "w": 6, "x": 0, "y": 0 }, "component": { "id": 2, "component_type": "COUNT", "title": "User Profiles", "query_id": "22252" } }, { "layout": { "h": 1, "static": false, "w": 6, "x": 6, "y": 0 }, "component": { "id": 2, "component_type": "COUNT", "title": "User Cookies", "query_id": "22264" } }, { "layout": { "h": 3, "static": false, "w": 12, "x": 0, "y": 1 }, "component": { "id": 5, "component_type": "MAP_BAR_CHART", "title": "Genre", "show_legend": true, "query_id": "47031", "sortKey": "A-Z", "percentage": true, "labels": { "enable": true, "filterValue": "", "format": "{point.y}%" }, "tooltip": { "formatter": "{point.y}% ({point.count})" } } }, { "layout": { "h": 3, "static": false, "w": 12, "x": 0, "y": 4 }, "component": { "id": 5, "component_type": "COUNT_BAR_CHART", "labels_enabled": true, "plot_labels": [ "Email", "Print", "Sms", "Tel", "Web", "App" ], "title": "Contactabilité", "show_legend": false, "query_ids": [ "47033", "47034", "47035", "47036", "47037", "47038" ] } } ] } ] ``` ### Final values import In ordre to be able to select audience features thanks to final values, you should first import your final values thanks to a csv file. For more information about the search by final feature, please read the *search by final value feature guider*. #### Requirements Your csv file should have: * The following format: 1 level min and 8 levels max, final\_value, ```javascript level1,level2, ... ,final_value ``` Example ```javascript level1,level2,level3,level4,final_value activities,channel_id,,,my channel id1 segments,creation_ts,,,123 ... ``` {% hint style="info" %} Don't need to specify UserPoint for the level1, it's implicit. {% endhint %} * A maximum of 100 000 lines, each line should match your schema, * Final values' field of type `String` or `[String]`. #### API ## Upload final values csv file `POST` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/reference_table_job_executions` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ---------------------- | | datamart\_id | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | ---- | ------ | ------------------------------------------------------------------------------- | | file | string | The name of the file you want to import. Ex: "@final\_value\_file.csv**"** | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "parameters": null, "result": null, "error": null, "id": "xxxxxx", "status": "PENDING", "creation_date": 1634134417792, "start_date": null, "duration": null, "organisation_id": "xxxx", "user_id": "xxxx", "cancel_status": null, "debug": null, "is_retryable": false, "num_tasks": null, "completed_tasks": null, "erroneous_tasks": null, "retry_count": 0, "permalink_uri": null, "job_type": "REFERENCE_TABLE", "import_mode": "MANUAL_FILE", "import_type": null } } ``` {% endtab %} {% endtabs %} ## Upload final values csv file `POST` `https://api.mediarithmics.com/v1/datamarts/{datamart_id}/reference_table_job_executions` #### Path Parameters | Name | Type | Description | | ------------ | ------- | ------------ | | datamart\_id | integer | Od2HfDMVLPel | #### Request Body | Name | Type | Description | | ---- | ------ | ------------ | | file | string | kUYOfU4QZxF3 | Example ```javascript curl -k --location --request POST 'https://api.mediarithmics.com/v1/datamarts/{datamart_id}/reference_table_job_executions' \ -H 'Content-Type: text/csv' \ -H 'Authorization: TOKEN' \ --data-binary '@./final_value_file.csv' ``` # Audience segment metrics {% hint style="info" %} **Important**\ \ If you're using the new **segment metrics** system, please check the dedicated documentation page for up-to-date information: [segment metrics documentation](https://userguides.mediarithmics.io/audience/segments/deep-dive-on-the-segment-page/metrics-panel) {% endhint %} Audience segment metrics are a way to offer **custom metrics on segments** to users. They are visible on the segment listing and segment details pages. The value of each metric is calculated regularly for each segment and saved to offer a historic view of its values. ![Segment details page on a datamart with 2 custom audience segment metrics](/files/-MTKbA8sDkL5nn7OrJWV) {% hint style="success" %} The total number of UserPoint is always calculated and displayed, even if there are no custom metrics. {% endhint %} ![A segment view without custom metrics](/files/-MTKamLVw7tWnD9y8NS9) ## How to configure metrics Audience segment metrics are configured per datamart and built on top of OTQL queries. Each metric has: * An associated [OTQL Query](/querying-your-data/otql-queries#managing-queries) * A technical name, possible values being `emails`, `user_accounts`, `desktop_cookie_ids`, `mobile_cookie_ids` or `mobile_ad_ids`. You can't use a custom value, and each of these values can only be used once per datamart. * A display name shown in the UI * A status: `DRAFT`, `LIVE` or `ARCHIVED`. * An icon, from a set of possible icons. {% hint style="info" %} **The metrics calculate how many UserPoint in the segment have at least one record. It doesn't count the number of records in the segment.** If you add a metric that counts the number of cookies and a UserPoint is associated with multiple cookies in the platform, it will be counted as only a +1 and not a +2. The metrics values will always be lower or equal to the number of UserPoint in the segment. {% endhint %} A metric goes from `DRAFT` status to `LIVE` and from `LIVE` status to `ARCHIVED`. You cannot republish an `ARCHIVED` metric. You can only remove it. ## Listing existing metrics `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/audience_segment_metrics` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | string | The ID of the datamart | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "id": "1555", "datafarm_key": "DF_EU_2020_02", "datamart_id": "1509", "query_id": "50659", "technical_name": "user_accounts", "display_name": "User Profiles", "icon": "users", "status": "LIVE", "creation_date": 1613125152462, "last_modified_date": 1613125152462, "last_published_date": null }, { "id": "1558", "datafarm_key": "DF_EU_2020_02", "datamart_id": "1509", "query_id": "50659", "technical_name": "mobile_cookie_ids", "display_name": "User Profiles", "icon": "users", "status": "LIVE", "creation_date": 1613125314757, "last_modified_date": 1613125314757, "last_published_date": null }, { "id": "1566", "datafarm_key": "DF_EU_2020_02", "datamart_id": "1509", "query_id": "50659", "technical_name": "mobile_ad_ids", "display_name": "User Profiles 7", "icon": "users", "status": "ARCHIVED", "creation_date": 1613128930707, "last_modified_date": 1613128930707, "last_published_date": null }, { "id": "1569", "datafarm_key": "DF_EU_2020_02", "datamart_id": "1509", "query_id": "50659", "technical_name": "desktop_cookie_ids", "display_name": "User Profiles 4", "icon": "gears", "status": "LIVE", "creation_date": 1613129103522, "last_modified_date": 1613129103522, "last_published_date": null }, { "id": "1570", "datafarm_key": "DF_EU_2020_02", "datamart_id": "1509", "query_id": "50659", "technical_name": "desktop_cookie_ids", "display_name": "User Profiles 4", "icon": "gears", "status": "DRAFT", "creation_date": 1613129261878, "last_modified_date": 1613129261878, "last_published_date": null } ], "count": 5, "total": 5, "first_result": 0, "max_result": 50, "max_results": 50 } ``` {% endtab %} {% endtabs %} ## Create an audience metric `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/audience_segment_metrics` This creates a DRAFT metric. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | #### Request Body | Name | Type | Description | | ---- | ------ | ----------------------------- | | Body | object | The metric you wish to create | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "1571", "datafarm_key": "DF_EU_2020_02", "datamart_id": "1509", "query_id": "50659", "technical_name": "emails", "display_name": "User Profiles 7", "icon": "users", "status": "DRAFT", "creation_date": 1613130322659, "last_modified_date": 1613130322659, "last_published_date": null } } ``` {% endtab %} {% tab title="400 If values are not valid" %} ```javascript { "status": "error", "error": "Json object is not structured as expected", "error_code": "BAD_REQUEST_FORMAT", "error_id": "e18c26a1-7497-470d-8480-2bcb66fc8f16" } ``` {% endtab %} {% endtabs %} {% hint style="info" %} You must first [create an OTQL query,](/querying-your-data/otql-queries#creating-a-query) returning a number corresponding to your metric. As each datamart can have its very own schema, **those queries should be tested for each datamart (by API or in the Data Studio) and should return a number.** The configured query will be "merged" with the query of each segment to calculate the proper metric. Here are some common query examples: * `SELECT @count{id} FROM UserPoint WHERE agents{}` counts the number of UserPoint having at least 1 cookie or mobile ID (user agent) * `SELECT @count{id} FROM UserPoint WHERE profiles{}` counts the number of UserPoint having at least 1 profile * `SELECT @count{id} FROM UserPoint WHERE emails{}` counts the number of UserPoint having at least 1 email {% endhint %} Here is a sample body payload: ```javascript { "datamart_id": "<>", "query_id": "", "technical_name": "", "display_name": "User Profiles", "icon": "users" } ``` ## Activate the audience metric `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/audience_segment_metrics/:metricId/action` This action transitions the metric go from `DRAFT` to `LIVE`.status. Any existing metric in `LIVE` status with the same technical name is `ARCHIVED`. #### Path Parameters | Name | Type | Description | | ---------- | ------- | ------------------------------- | | datamartId | integer | The ID of the datamart | | metricId | integer | The ID of the metric to publish | #### Request Body | Name | Type | Description | | ---- | ------ | ----------------------- | | Body | object | { "action": "PUBLISH" } | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Remove an audience metric `DELETE` `https://api.mediarithmics.com/v1/datamarts/:datamartId/audience_segment_metrics/:metricId` #### Path Parameters | Name | Type | Description | | ---------- | ------- | ------------------------------ | | datamartId | integer | The ID of the datamart | | metricId | integer | The ID of the metric to remove | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Limitations 1. Only five custom audience segment metrics per datamart are allowed—one per available technical name. 2. [@cardinality aggregations](/querying-your-data/otql-queries#aggregation-operations) are not supported in the queries. ## Available icons Each metric is associated with an icon taken from the following catalogue. ### Most used icons * display * users * email-inverted * phone ![](/files/-MTLAE6FrxsDSyHaPGBI) ### Other icons * adGroups * ads * automation * bell * bolt * check-rounded-inverted * check-rounded * check * chevron-right * chevron * close-big * close-rounded * close * code * creative * data * delete * display * dots * download * email-inverted * email * extend * filters * full-users * file * gears * goals-rounded * goals * image * info * laptop * library * magnifier * menu-close * minus * optimization * options * partitions * pause * pen * phone * play * plus * query * question * refresh * settings * smartphone * status * tablet * user * users * user-query * user-pixel * user-list * video * warning # Audience segment feed An audience feed is a [plugin](/advanced-usages/plugins) acting like a connector that allows mediarithmics customers to push their segments to a third-party platform. It is generic: once a connector to a partner has been created, every customer can use it. Each audience feed has a specific set of options to adapt to each customer. {% hint style="info" %} For more details on how to manage consent in segments and feeds, have a look at the [User choice](/advanced-usages/data-privacy-compliance/user-choices)s section. {% endhint %} ## Key concepts An **audience external feed** is a mediarithmics [plugin](/advanced-usages/plugins) specific to a partner, but shared across customers. In the UI, it is called Server-side plugin: go to a specific segment, click **Add a feed,** and you will see a feed type called server-side. It is marketed as connectors or server-to-server connectors. It has: * Plugin definition * group\_id: `com.mediarithmics.audience.externalfeed` * artifact\_id: `[[partner]]-connector` * Plugin versions with * Deployed code * An external service referencing partner’s API * Configuration files for things like credentials, tokens, technical configurations A **feed** is an instance of an audience feed. It is specific to a segment in an organisation. It has: * Feed ID * Instance properties specified by users in the UI when adding a feed to a segment. A **feed session** is initiated whenever an external feed is activated. A new session will also be created if a feed is paused and then reactivated. This session is not visible in the UI. A **feed preset** is a template allowing users to easily create feed instances with pre-configured properties. You can, for example, create a Facebook feed preset containing your key for your organisation, and you won't have to remember it every time you set up a new feed. # Building new feeds {% hint style="success" %} mediarithmics will create new audience segment feeds for you, and they will be made available to all other customers. Please discuss your needs with your Account Manager. {% endhint %} ## Creation steps Here are our creation steps for making a new audience segment feed for you : 1. **Validation process** to ensure we need the connector. 2. **Audience segment feed specifications:**, studying how we connect to the partner and which parameters should be made available to the user. 3. **Code and validation.** 4. **Cookie matching set up.** Some partners need a cookie-matching mechanic so that we can send them identifiers that have meaning for them. 5. **Deployment.** We allow connections to the partner's API during this step. # Monitoring a feed ## Accessing the feed 1. **Go to the Segment:** Navigate to the segment where your feed is configured. 2. **Open Feed Options:** Click the menu button and choose either **Stats** or **Edit**. ## Feed details overview The feed details are divided into three sections to enhance understanding: 1. [**Stats**](#stats-high-level-overview-of-the-feed)**:** * Provides a high-level summary of the feed. * Shows the status of the feed instance. * Displays the identifiers sent to the partner for addition or deletion. 2. [**Configuration**](#configuration-plugin-properties-setup)**:** * Lists all plugin properties configured for the current instance. 3. [**Troubleshooting**](#troubleshooting-technical-details)**:** * Offers an in-depth view of processing steps, aiding in the investigation of anomalies. ## Stats: high-level overview of the feed The **Stats** tab is tailored for non-technical users to easily assess feed activity. It shows whether the feed is successfully transmitting identifiers to the destination platform. ### **Server-side feeds** #### **Feed lifecycle** 1. **Creation:** * **Waiting for Activation:** The feed card is created on the segment but remains inactive. * **Feed Activated:** The feed creates the segment on the destination platform (triggers the `onExternalSegmentCreation` function). 2. **Connection:** * The feed tests the connection with the destination platform (triggers the `onExternalSegmentConnection` function). 3. **Starting:** * The feed begins transmitting identifiers to the destination platform. 4. **Initial Loading:** * Processes all users in the segment and sends their identifiers to the destination. 5. **Live:** * Continues to send new identifiers and requests deletion of those no longer present in the segment. #### **Sending metrics** * **Successful Identifier Transactions:** number of upserts or deletions of identifiers successfully sent during the selected period. * **Identifier Coverage:** percentage of UserPoint without any identifiers sent to the destination platform. * **Daily Graph:** visual representation of upserts or deletions sent daily. ### **Client-side feed** Client-side feeds have simpler functionality but use a similar interface. #### **Feed lifecycle** * **PAUSED:** the feed card is created but not activated. * **ACTIVE:** the feed is activated on the segment and the feed is downloaded by the browser. #### **Sending metrics** * **Daily Downloads:** number of times the feed was downloaded by the browser during the selected period. ## Configuration: plugin properties setup The **Configuration** tab displays the plugin properties for the feed instance: * Plugin properties layout is determined during the creation of the plugin version. * These values can only be modified when the feed is not activated. Once activated, the configuration becomes read-only. ## Troubleshooting: technical details The **Troubleshooting** tab is designed for technical users, providing detailed insights into: * Successful and failed operations (e.g., upserts and deletions). * Errors and processing steps for investigating anomalies. ### **General information** * **Server-Side and Client-Side feeds:** * Resource details, plugin information, and plugin version information. * **Server-Side Feeds only:** * Instance details and initial loading logs. ### **Metrics for server-side feeds** #### **Standard Metrics:** 1. **API Calls to the Audience Feed Plugin:** * Displays response status for `/user_segment_update` calls. 2. **Identifiers Sent to the Destination Platform:** * Shows processed identifiers and their statuses for non-batching cases. #### **Actions and Statuses:** * **Push:** Adding a UserPoint to the segment or an identifier to a UserPoint. * **Remove:** Removing a UserPoint from the segment or an identifier from a UserPoint. * **Statuses:** * **PROCESSED:** Successful with no destination platform response (e.g., batch/file delivery). * **SUCCEEDED:** Successful with a positive response from the destination platform. * **FAILED:** Error occurred within the plugin or at the destination platform. * **NO\_ELIGIBLE\_IDENTIFIERS:** No eligible identifiers to send. #### **Batching Metrics:** * **API Calls to the Audience Feed Plugin (/batch\_update route):** * Number and status of batches created. * **Records Sent to the Destination Platform:** * Rows sent per batch (may contain multiple identifiers). #### **File Delivery Metrics:** * **Files Sent by File Delivery Service:** * Number of files sent and response statuses. * **Records Sent to the Destination Platform:** * Rows sent per file (may represent multiple identifiers). ### **Metrics for Client-Side Feeds** * Displays a daily graph of browser downloads, as shown in the **Stats** tab. # Edge segments ## How it works You can **compute segments browser-side** using our **Edge technology** and share them with Google Ad Manager. This allows you to react instantly to user behavior and trigger campaigns as users navigate your website. To do so, the mediarithmics tag: * **Collects and stores identifiers and navigation data in local storage**. When an event is pushed by the tag, it is sent both to the activity processing pipeline and saved locally. * **Computes browser-side which segments the user enters or leaves**. * When a user enters or exits one or more segments: * **Triggers the Google Ad Manager integration** * **Sends the information back to the server to maintain a server-side state of the segment**. Segment statistics are computed periodically by running a query that counts users flagged as belonging to the segment. {% hint style="info" %} Only users who navigated your site with the Edge feature activated will be in the segment. {% endhint %} ## Setup To configure the Edge segmentation feature, please follow the steps outlined below. #### 1. Prerequisites * **Update Snippet**: Ensure your site is running the latest version of the mediarithmics snippet. (Refer to [Website tracking](/data-ingestion/real-time-user-tracking/web-events#implementing-the-snippet) documentation for details). * **Feature Activation**: Contact your Account Manager to enable Edge on your specific channels. #### 2. Schema Configuration Identify the properties in your schema required for Edge-based user segmentation and mark them with the `@EdgeAvailability` flag (see [Data model @EdgeAvailability directives](/schema#edgeavailability) for more information) Note that you must meet the following requirements: * **Supported Objects**: Edge only supports properties from the following objects: * `UserDeviceTechnicalId` * `DeviceInfo` * `UserSegment` * `UserEvent` (including all sub-object properties) * **Data Integrity**: Properties must exist in the schema and be stored exactly as pushed by the tag. No transformations (e.g., event rules or visit analyzers) can be applied. * **Directives**: A property using the `@PropertyPath` directive is only compatible if the path name and property name are identical in both the schema and the data layer. {% hint style="info" %} **Tip:** We recommend validating your schema changes with our technical team before deployment. {% endhint %} #### 3. SSP / AdServer Integration Finalize the connection between mediarithmics and your delivery platforms. * Google Ad Manager (GAM): The JS snippet automatically pushes segment IDs as key-values to the ad call. However, these must be manually created within the GAM UI. * *Reference:* [GAM Key-Value Setup Guide](https://support.google.com/admanager/answer/9796369?hl=en) * Other Platforms: While there is no native "out-of-the-box" integration for other platforms yet, it can be implemented easily on your side, as segment IDs are stored and available in Local Storage. # Cohort-based Lookalike {% hint style="info" %} Please refer to our [user guide documentation](https://userguides.mediarithmics.io/audience/segments/segment-typology/user-lookalike-segment/cohort-based-lookalike) to learn more about this feature. {% endhint %} {% hint style="warning" %} To configure this feature, please follow the next steps **IN ORDER**: 1. Attributes definition 2. ML function creation 3. ML function activation 4. Schema update 5. ML function initial loading {% endhint %} ## Attributes definition ### Attributes #### Attributes selection A designated cohort is assigned to a user depending on attributes (also named features in DataScience) you have defined to characterize your users. You will need to format those attributes using JSON format (see below). {% hint style="info" %} We recommend to : * Pick attributes that can be used to segment users and that are relevant to the business * Pick attributes that are available on all your users (logged / unlogged) * Pick attributes from various typology (UserEvent, UserProfile, …) * Select between 3 & 10 attributes * Have between 50 & 300 values of attributes (from all various attributes) * Keep the default of 1024 cohorts (Cohort Id Bit Size = 10, see below for more information about this) {% endhint %} #### JSON format For instance, let's imagine that you want to create cohorts based on: * **os\_family** - defined on **UserAgentInfo** nested in **UserAgent** * **age** - defined on **UserProfile** * **city** - defned on **UserEvent** You will therefore define the following JSON: ```json [ { "type": "FREQUENCY_ENUM", "field_path": "agents.user_agent_info.os_family", "values": [ "OTHER", "WINDOWS", "MAC_OS", "LINUX", "ANDROID", "IOS" ] }, { "type": "FREQUENCY_NUMBER", "field_path": "profiles.age", "intervals": [ { "from": 0, "to": 10 }, { "from": 10, "to": 100 } ] }, { "type": "FREQUENCY_TEXT", "field_path": "events.city", "vector_size": 100 } ] ``` #### Configuration help There are 3 types of attributes available: * **FREQUENCY\_ENUM**: use this type for a finite list of values like operating systems. * **FREQUENCY\_NUMBER**: use this type for classifying number buckets like age. Using the above example: * First bucket: >= 0 & < 10 * Second bucket: >= 10 & < 100 * Third bucket: anything that didn't fell into the 2 defined buckets * **FREQUENCY\_TEXT**: use this type an infinite (or long) liste of values like keywords, cities, ... Choose wisely the **vector\_size** parameter as it will be used as a modulo on values to reduce the disparity of values to a fixed number The **field\_path** must contain the path of the attribute from the UserPoint definition (see [schema documentation](https://developer.mediarithmics.io/schema) for more info) ### **GraphQL Query** A ML function requires a query to fetch data used in its configuration. In the case of cohort-based lookalike, it requires an appropriate query to fetch fields used as attributes and specified in the JSON. Following our previous example, the graphQL query will be : ```graphql {agents {user_agent_info {os_family}} profiles {age} events{city}} ``` ## ML function creation Please follow the next steps to instantiate the ML function developed by mediarithmics to assign a cohort to your userpoints: 1. Head to Settings > Datamart > ML Functions 2. Click on **New Ml Function**, pick the datamart where to apply the ML function then choose **simhash-cohorts-calulation** 3. Enter the following information on the ML function configuration panel: * General Informations * Name: **Cohort ML Function** * Hosting Object Type: **UserPoint** * Field Type Name: **ClusteringCohort** * Field Name: **clustering\_cohort** * Query: *\* * Properties * Features: *\* * Cohort Id Bit Size: *\* 4. Click on **Save** button {% hint style="warning" %} Note that only one Cohort-based Lookalike model can be set up at a time in an organisation. {% endhint %} ## ML function activation Once the ML function has been instantiated, you will need to update **batch\_mode** parameter to **true** and **activate the ML function** by running the following API : ```json PUT https://api.mediarithmics.com/v1/ml_functions/ { "batch_mode": true, "status": "ACTIVE" } ``` ## Schema update Two changes have to be made in your runtime schema : * Add a field **clustering\_cohort** in **UserPoint** as follow : ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX") { ... clustering_cohort:ClusteringCohort ... } ``` * Create a new **ClusertingCohort** type as follow : ```graphql type ClusteringCohort { id:ID! @TreeIndex(index:"USER_INDEX") expiration_ts:Timestamp @TreeIndex(index:"USER_INDEX") cohort_id:String! @TreeIndex(index:"USER_INDEX") last_modified_ts:Timestamp! @TreeIndex(index:"USER_INDEX") } ``` {% hint style="success" %} Don't hesitate to have a look at [schema update documentation](https://developer.mediarithmics.io/schema/defining-your-schema) to learn more about how to update your schema. {% endhint %} ## ML function initial loading You can ask your Account manager to run an initial loading on your datamart to calculate cohorts on existing userpoints. # Exporting your data There are few ways to export data depending on your usage. ## Audience feeds Use [audience feeds](/advanced-usages/audiences/audience-segment-feed) if you need to export audience segments data to your partners' solutions and keep it recurring and synchronized. ## Destination file If you need a feed to deliver its data to an external system, first set up a destination file to define where and how those files are delivered. ## Query exports Create [query exports](/advanced-usages/exporting-your-data/query-exports) to create `ndjson` files corresponding to results of [OTQL queries](/querying-your-data/otql-queries). ## Datamart replication [Replicate your datamart](/advanced-usages/exporting-your-data/datamart-replication) by sending all the ingested data to Google Cloud Platform (Pub/Sub) or Microsoft Azure (Event Hubs - Alpha). # Query Exports The platform allows you to export data as `ndjson`, based on [OTQL queries](/querying-your-data/otql-queries). This action is available in the navigator or through API to automate the process. {% hint style="success" %} For punctual exports, you may want to use the navigator (in Data Studio > Exports) rather than our API for more simplicity. {% endhint %} **An export is a definition** with: * a name * an output format (always `JSON` for now) * a type (always `QUERY` for now) * an associated query * an associated [organisation](/communities-and-organisations) * an associated [datamart](/communities-and-organisations/datamart) **Each export has executions** representing its result at a given time. An execution goes through different stages: * `Pending` if the export is not started * `Running` if the export is started * `Succeeded` or `Failed` once the export has been completed The resulting file is formatted in [`ndjson`](http://ndjson.org), and contains the properties you selected in the `SELECT` part of the OTQL query. ## Listing exports `GET` `https://api.mediarithmics.com/v1/exports?organisation_id=:orgId` #### Path Parameters | Name | Type | Description | | ----- | ------- | ------------------------------------------------------------- | | orgId | integer | The ID of the organisation for which you want to list exports | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "type": "QUERY", "id": "9879", "organisation_id": "1426", "name": "2018-08-29 - Export test segments for Facebook and Adwords", "query_id": "65987", "datamart_id": "1509", "output_format": "JSON" }, { "type": "QUERY", "id": "9872", "organisation_id": "1426", "name": "export-visiteurs", "query_id": "45987", "datamart_id": "1509", "output_format": "JSON" }, ... ], "count": 50, "total": 163, "first_result": 0, "max_result": 50, "max_results": 50 } ``` {% endtab %} {% endtabs %} ## Details of an export `GET` `https://api.mediarithmics.com/v1/exports/:exportId?organisation_id=:orgId` #### Path Parameters | Name | Type | Description | | -------- | ------- | ------------------------------------------------ | | exportId | integer | The ID of the export | | orgId | integer | The ID of the organisation containing the export | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "type": "QUERY", "id": "8978", "organisation_id": "1426", "name": "2018-08-29 - Export test segments for Facebook and Adwords", "query_id": "98798", "datamart_id": "1509", "output_format": "JSON" } } ``` {% endtab %} {% endtabs %} ## Create an export You will first need to [create a query for your export](/querying-your-data/otql-queries#creating-a-query). {% hint style="info" %} You should check your query and verify its results before creating an export. It is easier to do at this step, as the export would only return an empty file and no message. {% endhint %} ## Create an export `POST` `https://api.mediarithmics.com/v1/exports?organisation_id=:orgId` #### Path Parameters | Name | Type | Description | | ----- | ------- | -------------------------- | | orgId | integer | The ID of the organisation | #### Request Body | Name | Type | Description | | ---- | ------ | ------------------- | | Body | string | the request payload | {% tabs %} {% tab title="200 " %} ```javascript { "status":"ok", "data": { "type": "QUERY", "id": "8205", "organisation_id": "1426", "name": "test", "query_id": "50409", "datamart_id": "1509", "output_format": "JSON" } } ``` {% endtab %} {% endtabs %} ```javascript // Creating an export payload { "name": "", "output_format": "JSON", "query_id": "", "type": "QUERY" } ``` ## Generate an execution `POST` `https://api.mediarithmics.com/v1/exports/:exportId/executions` #### Path Parameters | Name | Type | Description | | -------- | ------- | -------------------- | | exportId | integer | The ID of the export | #### Request Body | Name | Type | Description | | ---- | ------ | ----------------------------------- | | Body | object | Must be an empty JSON object : `{}` | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "parameters": {}, "organisation_id": "1426", "user_id": "2886", "result": null, "error": null, "id": "879879879", "status": "PENDING", "creation_date": 1612789702668, "start_date": null, "duration": null, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "MTowOjA65985NjYyOQ==", "num_tasks": null, "completed_tasks": null, "erroneous_tasks": null, "retry_count": 0, "job_type": "DATAMART_QUERY_EXPORT" } } ``` {% endtab %} {% endtabs %} ## Execution status `GET` `https://api.mediarithmics.com/v1/exports/:exportId/executions` #### Path Parameters | Name | Type | Description | | -------- | ------- | -------------------- | | exportId | integer | The ID of the export | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "parameters": {}, "organisation_id": "1426", "user_id": "8987", "result": { "output_files": [ "export" ] }, "error": null, "id": "7897897", "status": "SUCCEEDED", "creation_date": 1562595783663, "start_date": 1562595789171, "duration": 292617, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "", "num_tasks": 299808, "completed_tasks": 277105, "erroneous_tasks": 22703, "retry_count": 0, "job_type": "DATAMART_QUERY_EXPORT", "end_date": 1562596081788 }, { "parameters": {}, "organisation_id": "1426", "user_id": "8987", "result": { "output_files": [ "export" ] }, "error": null, "id": "9879879", "status": "SUCCEEDED", "creation_date": 1561655188369, "start_date": 1561655192158, "duration": 413140, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "", "num_tasks": 284602, "completed_tasks": 263822, "erroneous_tasks": 20780, "retry_count": 0, "job_type": "DATAMART_QUERY_EXPORT", "end_date": 1561655605298 }, { "parameters": {}, "organisation_id": "1426", "user_id": "9879", "result": { "output_files": [ "export" ] }, "error": null, "id": "4564564", "status": "SUCCEEDED", "creation_date": 1535555844642, "start_date": 1535555850435, "duration": 337507, "cancel_status": null, "debug": null, "is_retryable": false, "permalink_uri": "", "num_tasks": null, "completed_tasks": null, "erroneous_tasks": null, "retry_count": 0, "job_type": "DATAMART_QUERY_EXPORT", "end_date": 1535556187942 } ], "count": 3, "total": 3, "first_result": 0, "max_result": 10, "max_results": 10 } ``` {% endtab %} {% endtabs %} {% hint style="info" %} If available, the technical name of the file to download is in the `result.output_files` array {% endhint %} ## Download the result of an export The result of an export is saved as a data file in the platform. You have two methods to retrieve it : * Using the exports API (recommended) * Using the data\_file API ## Download using the exports API `GET` `https://api.mediarithmics.com/v1/exports/:exportId/executions/:executionId/files/technical_name=:technicalName` #### Path Parameters | Name | Type | Description | | ----------- | ------- | -------------------------------------------------------- | | exportId | integer | The ID of the export | | executionId | integer | The ID of the execution for which to download the report | #### Query Parameters | Name | Type | Description | | --------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | | technical\_name | string | The name of the file(s) in the `result.output_files` of the execution. Usually `export` when there is a single file. | {% tabs %} {% tab title="200 " %} ```javascript // For query SELECT {emails {id}} FROM UserPoint WHERE segments {id=""} {"emails":[{"id":"87987987985465432198"}]} {"emails":[]} {"emails":[]} {"emails":[{"id":"46546549879879845654"},{"id":"564654987987465465465"}]} ``` {% endtab %} {% tab title="404 The file does not exist or has been removed" %} ```javascript { "status": "error", "error": "Resource Not Found", "error_id": "9eedef5f-dfe3-4354-9a2f-d79760113386" } ``` {% endtab %} {% endtabs %} ## Download using the data\_file API `GET` `https://api.mediarithmics.com/v1/data_file/data?uri=mics://data_file/tenants/:organisationId/jobs/executions/:executionId/result/export.data` #### Query Parameters | Name | Type | Description | | -------------- | ------- | -------------------------- | | executionId | string | The ID of the execution | | organisationId | integer | The ID of the organisation | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} ## Download the error file `GET` `https://api.mediarithmics.com/v1/data_file/data?uri=mics://data_file/tenants/:organisationId/jobs/executions/:executionId/result/export.error` Use this endpoint in case of a `FAILED` execution status. #### Query Parameters | Name | Type | Description | | -------------- | ------- | -------------------------- | | executionId | string | The ID of the execution | | organisationId | integer | The ID of the organisation | {% tabs %} {% tab title="200 " %} ``` ``` {% endtab %} {% endtabs %} # Datamart replication {% hint style="info" %} This module is not included in the default plan. Contact your Account manager to activate it. {% endhint %} Datamart replication allows you to replicate the data ingested by mediarithmics in an external solution of your choice. We currently support replication to : * [External message queues](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication) : Google Pub/Sub and Microsoft Azure Events Hub * [Data warehouses ](/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication): Google BigQuery and Snowflake ## How it works ### Creating & starting a replication To create a new replication: 1. Go to the Computing console>Data stream menu 2. Click **New Replication**. 3. Select a **Replication type** matching the external solution of your choice. 4. Complete configuration information. 5. Click **Save Replication** to create your new replication. 6. You will see your new replication in the **Replications** subtab. #### Replication statuses Your replication can be in one of the following status: * **ACTIVE**: All data processed by your datamart will be replicated to your external solution. * **PAUSED**: No data processed by your datamart will be replicated to your external solution. * **ERROR**: The system is no longer able to replicate messages. In this case, check your external solution (expired instance, invalid credentials, etc). If you can't find anything wrong, please contact your Account manager. {% hint style="success" %} When a Replication is created, its status is automatically set to **Paused**. To start your replication, you will have to activate it. If the system can't replicate your datamart on activation, you will see an error. {% endhint %} {% hint style="info" %} When a replication can't be activated, it is usually due to an error on **credentials**, so you might want to verify your replication configuration and your credentials file first. {% endhint %} ### API Documention Creating a replication is a 2 step action : first creating the replication object, and then posting credentials attached to the replication. * In an external message queue replication : the credentials to post are the ones carrying the access rights to the [message queue service](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication#setting-up-replications). * In a data warehouse replication : the credentials to posts are the ones carrying the access rights to the [object storage used in the replication process.](/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication) #### Create a replication `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications` **Path Parameters** | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | Integer | The ID of the datamart | **Request Body**
NameTypeDescription
typeEnum GOOGLE_PUBSUB , AZURE_EVENT_HUBS, DATA_WAREHOUSE
nameStringThe name of the replication
replication_filtersArrayList of documents to replicate. Array of { "document": "" , "filter":""}
replication_filters>documentString

For message queue replication : USER_SEGMENT, USER_EMAIL, USER_ACCOUNT, USER_PROFILE, USER_DEVICE_POINT, USER_DEVICE_TECHNICAL_ID, USER_ACTIVITY, USER_POINT, USER_POINT_PARENT, USER_AGENT (legacy)
For data warehouse replication :

USER_SEGMENT, USER_EMAIL, USER_ACCOUNT, USER_PROFILE, USER_DEVICE_POINT, USER_DEVICE_TECHNICAL_ID, USER_ACTIVITY, USER_EVENT, USER_COMPUTED_FIELD

replication_filters>filterStringRequired but unused. Should be null
versionInteger (optional)Version of the datamart replication to be created. Check versionning for more info. NB: "2" is the default except for AZURE_EVENT_HUBS replication
datamart_idString (optional)As per the Path parameter
statusString (optional)Status of the replication. Default is "PAUSED"
project_idStringGoogle project ID. Only for GOOGLE_PUBSUB
topic_idStringGoogle PubSub topic ID. Only for GOOGLE_PUBSUB
event_hub_nameStringAzure event hub name. Only for AZURE_EVENT_HUBS
data_warehouse_idStringThe id of the data warehouse in which the data will be replicated.
Only for DATA_WAREHOUSE
schema_idStringThe id of the schema/dataset in which the data will be replicated.
Only for DATA_WAREHOUSE
bucket_nameStringThe name of the bucket in which the log files will be sent. Only for DATA_WAREHOUSE
**Request Body example** ```json { "name": "test_replication" , "replication_filters": [ { "document": "USER_SEGMENT", "filter": null, }, { "document": "USER_ACCOUNT", "filter": null, } ], "data_warehouse_id": "", "schema_id": "", "datamart_id": "", "bucket_name": "", "type": "DATA_WAREHOUSE" } ``` #### Add credentials to replication `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId/credentials` **Path Parameters** | Name | Type | Description | | ------------- | ------- | ---------------------------------- | | datamartId | Integer | The ID of the datamart | | replicationId | Integer | The ID of the datamart replication | **Content-Type** The endpoint accepts two body formats (Content-Type):
Content-TypeUsage
application/jsonUsed for DATA_WAREHOUSE replication type
multipart/form-dataUsed for GOOGLE_PUBSUB and AZURE_EVENT_HUBS replication type
**Request body** | Name | Type | Description | | ----------- | ------ | --------------------------------------------------------------- | | type | String | Should be `"OBJECT_STORAGE"` | | access\_key | String | Access key | | secret\_key | String | Secret key | | protocol | Enum | `"GS"` for Google Cloud Storage bucket, `"S3"` for a AWS bucket | **Curl examples** ```bash curl --location 'https://api.mediarithmics.com/v1/datamarts/:datamartID/replications/:replicationID/credentials' \ --header 'Authorization: api:XXX' \ --form 'file=@"/XXX/service-account.json"' ``` ```bash curl --location 'https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId/credentials' \ --header 'Authorization: api:XXX' \ --header 'Content-Type: application/json' \ --data '{ "type": "OBJECT_STORAGE", "access_key": "AAA...", "secret_key": "xxxxxxxx", "protocol": "S3" ``` #### Retrieve a replication `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replications/:replicationId` **Path Parameters** | Name | Type | Description | | ------------- | ------- | ---------------------------------- | | datamartId | integer | The ID of the datamart | | replicationId | integer | The ID of the datamart replication | **Request body example** ```json { "status": "ok", "data": { "type": "GOOGLE_PUBSUB", "id": "", "name": "", "datamart_id": "", "credentials_uri": "", "project_id": "", "topic_id": "", "status": "PAUSED", "version": 2, "replication_filters": [ { "id": "81", "replication_id": "", "document": "USER_DEVICE_TECHNICAL_ID", "filter": null, "schema_type_name": null }, { "id": "82", "replication_id": "", "document": "USER_SEGMENT", "filter": null, "schema_type_name": null } ], "creation_ts": 1763384421645, "internal_replication": false, "connector_type": null } } ``` ### Initial synchronization You can run an **initial synchronization** for one or multiple **ACTIVE** datamart replications. This operation replicates all existing documents selected in the datamart replication and stored within the **mediarithmics** platform into your cloud environment. Depending on your needs, you have several ways to trigger an initial synchronization: **For a single datamart replication:** 1. Locate the replication in the list. 2. Click the **caret-down** icon, then select **New Initial Synchronization**. **For multiple datamart replications:** 1. Click the **New Initial Synchronization** button. 2. Select the datamart replications you want to synchronize. 3. Click **Execute Initial Synchronization(s)**. All selected datamart replications will receive a set of `UPDATE` operations representing all existing elements (for example, `UserProfile`) in your datamart. You can't run an initial synchronization more than once a week. {% hint style="warning" %} Please note that if you run an initial synchronization you might receive a large volume of messages. Processing them can be expensive, depending on your cloud provider. {% endhint %} #### Launch initial synchronization on one replication `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/replication_job_executions` **Path Parameters** | Name | Type | Description | | ---------- | ------- | ---------------------- | | datamartId | integer | The ID of the datamart | **Request Body**
NameTypeDescription
replication_idsintegerThe IDs of the replication
```json { "replication_ids": ["123", "456"] } ``` # External message queues replication ## Available objects We replicate the `update` and `delete` operations from your [Datamart ](/communities-and-organisations/datamart)for the following objects: * [UserPoint](/user-points) * [UserAgent](/user-points/user-identifiers/userdevicetechnicalid#user-agents-legacy) *(for datamart `user_point_system_version` before v202205)* * [UserDevicePoint](/user-points/user-identifiers/userdevicetechnicalid#user-device-point) *(for datamart `user_point_system_version` v202205)* * [UserDeviceTechnicalId](/user-points/user-identifiers/userdevicetechnicalid#user-device-technical-identifier) *(for datamart `user_point_system_version` v202205)* * [UserActivity](/user-points/user-activities) * [UserSegment](/user-points/segments) * [UserProfile](/user-points/user-profiles) * [UserAccount](/user-points/user-identifiers/useraccount) * [UserEmail](/user-points/user-identifiers/useremail) * Parent UserPoint (when [UserPoint are merging](/user-points#user-points-merges)) When creating a datamart replication, you can select which object(s) to replicate. {% hint style="warning" %} Note that you cannot update the objects selection once set. You will need to create a new datamart replication in that case. {% endhint %} ## Versionning There are currently 2 versions of Datamart replications:
VersionFormatSupported destination
Version 1 (Legacy)JSON formatGoogle Cloud Platform - Pub/Sub
Microsoft Azure - Event Hubs
Version 2Avro Binary formatGoogle Cloud Platform - Pub/Sub
{% hint style="info" %} Please note that: * Any new GCP Pub/Sub replication will be created in Version 2 * You cannot upgrade a datamart replication from Version 1 to Version 2 {% endhint %} ## Output messages We convert datamart operations in a standardized output format: *`operation = {ts, doc_type, doc_id, op, value}`*
FieldTypeCommentVersion availability
tsUNIX Timestamp in ms (Long)The mutation dateAll
doc_typeEnumerationThe object type :UserActivity, UserProfile, UserSegment, UserAgent, UserAccount, UserEmail, UserPoint or UserPointParentAll
ctx_idUUIDThe userpoint idVersion 2
doc_idStringThe object unique id. The format varies depending on doc_type All
opStringThe operation type UPDATE or DELETEAll
valueJSON ObjectThe object value. The format varies depending on doc_type.All
### Object formats based on `doc_type` | doc\_type | doc\_id | value | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | `UserPoint` | `{{user_point_id}}` | Empty (you already have the `user_point_id` in the `doc_id`) | | `UserAgent` | `{{user_point_id}}:{{vector_id}}` | Browser info and device info | | `UserDevicePoint` | `{{user_point_id}}:{{user_device_point_id}}` | Browser info and device info | | `UserDeviceTechnicalId` | `{{user_point_id}}:{{user_device_point_id}}:{{user_device_technical_id}}` | Empty (you already have the `user_device_technical_id` in the `doc_id`) | | `UserActivity` | `{{user_point_id}}:{{user_activity_id}}` | Detailed activity | | `UserSegment` | `{{user_point_id}}:{{segment_id}}` | Segment info | | `UserProfile` | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}` | Detailed profile | | `UserAccount` | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}` | Empty (you already have the `user_account_id` in the `doc_id`) | | `UserEmail` | `{{user_point_id}}:{{email_hash}}` | User's email hash | | `UserPointParent` |

{{user\_point\_id}}

It is the ID of the UserPoint which is merged on the oldest one (the kept one).

| Message ` merged with ` | ### Message metadata To help filtering the topic, replication adds some metadata on message (attributes in PubSub and properties in EventHub) | Metadata key | Comment | | ------------ | ---------------------- | | `doc_type` | The message `doc_type` | ### Legacy JSON Format This format is the original replication format. It was designed to work with a streaming architecture (like Dataflow or Databricks) but has some limitations with tools needing a schema (like BigQuery) #### Examples A new activity will trigger a replicated `UserActivity` operation. You will receive a similar message in your external solution as shown in this example. 
{
   "ts": 1676627112685,
   "doc_type": "UserActivity",
   "doc_id": "XXXXXXX-XXXX-XXX-XXXXXXXX:XXXXXX-XXXXX-XXXX-XXXX-XXXXXXXXXX",
   "op":" UPDATE",
   "value":{
       "$type":"SITE_VISIT",
       "$source":"XXXX",
       "etc": "etc"
   }
}
A new user agent will trigger a replicated UserAgent operation like the one bellow ```javascript { "ts":1676627112685, "doc_type":"UserAgent", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893", "op":"UPDATE", "value":{ "$os_family":"LINUX", "$brand":null, "$os_version":null, "$form_factor":"PERSONAL_COMPUTER", "$carrier":null, "$model":null, "$creation_ts":0, "$browser_family":"FIREFOX" } } ``` A new user device point will trigger a replication UserDevicePoint operation like the one bellow ```javascript { "ts":1676627112685, "doc_type":"UserDevicePoint", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893", "op":"UPDATE", "value":{ "$os_family":"LINUX", "$brand":null, "$os_version":null, "$form_factor":"PERSONAL_COMPUTER", "$carrier":null, "$model":null, "$creation_ts":0, "$browser_family":"FIREFOX" } } ``` A new user device technical id will trigger a replicated `UserDeviceTechnicalId` operation like the ones bellow ```javascript // exemple with a MumId { "ts":1676627112685, "doc_type":"UserDeviceTechnicalId", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539", "op":"UPDATE", "value":{} } // exemple with an installationId { "ts":1676627112685, "doc_type":"UserDeviceTechnicalId", "doc_id":"4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:ins:1001:aZmFhOTVlM2ItMGRhOC00NDZlLWFhODMtNjZlZGI0YjNiNTk2", "op":"UPDATE", "value":{} } ``` ### Avro Binary Format This version introduce a schema to help integration. ```json { "type": "record", "name": "OperationRecord", "namespace": "com.mediarithmics.replication.format", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "doc_type", "type": { "name": "DocumentType", "type": "enum", "symbols": [ "UserPoint", "UserActivity", "UserProfile", "UserSegment", "UserDevicePoint", "UserDeviceTechnicalId", "UserAgent", "UserAccount", "UserEmail" ] } }, { "name": "doc_id", "type": "string", "doc": "It will always start with the ctx_id (ie: user_point) followed by ':' and other internal ids. It identifies uniquely a document." }, { "name": "ctx_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "op", "type": "string", "doc": "UPDATE or DELETE" }, { "name": "value", "type": "string", "doc": "The object in JSON format" } ] } ``` This format is almost the same as Legacy one, but with Avro binary format. {% hint style="warning" %} The target topic should reference the schema and the encoding as BINARY to take full advantages of the format. {% endhint %} ## Upgrade of datamarts to **`user_point_system_version` v202205** For datamarts with `user_point_system_version` anterior to v202205, device identifiers are stored as [User Agents](/user-points/user-identifiers/userdevicetechnicalid#user-agents-legacy), and replicated as **UserAgent operations** (doc\_id exemple: `4700c85f-17e3-4304-aa7f-dc140173b08d:vec:32453299893`). However for datamarts leveraging the `user_point_system_version` v202205, device identifiers are stored as [User Device Points and User Device Technical Ids](/user-points/user-identifiers/userdevicetechnicalid), and replicated through **UserDevicePoint and UserDeviceTechnicalId operations**. In the case of a datamart that is upgraded to the`user_point_system_version` v202205: * New device identifiers are directly stored and replicated using the device point formats, * Existing device identifiers that were previously stored in the UserAgent format are progressively migrated. This migration is seemless within the datamart, however it is reflected on your datamart replication. **For each migrated device identifier, you will receive:** * **A DELETE operation with the doc\_type User Agent** * **Two UPDATE operations with doc\_type UserDevicePoint and doc\_type UserDeviceTechnicalId** {% hint style="info" %} For instance, a migration of a user agent with a doc\_id `4700c85f-17e3-4304-aa7f-dc140173b08d:vec:`7231822539 will produce * 1 DELETE operation with doc\_type UserAgent the same doc\_id * 2 UPDATE operations: * 1 with doc\_type UserDevicePoint and the following doc\_id: `4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893` * 1 with doc\_type UserDeviceTechnicalId and the following doc\_id `4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539` {% endhint %} After migration, no more UserAgent operations will be produced ## Setting up replications You need to have an instance of the external solution where you want to replicate your mediarithmics data. You will need to get credentials from the external solution : * [Google Pub/Sub](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication/google-pub-sub) * [Microsoft Azure Event Hubs (Alpha)](/advanced-usages/exporting-your-data/datamart-replication/external-message-queues-replication/azure-events-hub) # Google Pub/Sub ## Set up You will need: * A Google Cloud Platform account * A Google Cloud Platform project: ; * An Access Control on this project: ; * To create and activate your service accounts (generate credentials file): > **TO SUM IT UP:** You can click here , create a service account, and edit it to create a Key in a JSON format (this is the credential file); Click on **Create Service Account** : ![](/files/avf2KoWc2UfaL5Kraore) Give your service account a name, select the right account access (Pub/Sub Publisher, Pub/Sub Editor) and save. ![](/files/M2yhk4vqkGLyy5Z3dTR8) Once your Service Account is created, you can generate your key : ![](/files/uu7JFFs05EknyT14jMkp) ![](/files/ruZKm7wqB85L6mAohQ5B) ![](/files/ZmZKpgTf63oa0GKm1FsA) credentials.json file example: ```javascript { "type": "service_account", "project_id": "xxx-xxx-xx", "private_key_id": "xxxxxxx", "private_key": "-----BEGIN PRIVATE KEY-----\n xxxxxxx \n-----END PRIVATE KEY-----\n", "client_email": "xxx@project_id.iam.gserviceaccount.com", "client_id": "xxxxxx", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/projetc_id.iam.gserviceaccount.com" } ``` * To create a Google Cloud Platform Pub/Sub instance. Pub/Sub documentation: until Quickstart setup > Create service account credentials (included) should be enough to begin. ***NOTE:*** Here is the Google Pub/Sub Pricing documentation: # Azure Events Hub ## Set up You will need: * A Microsoft Azure account * A Resource Group, an Event Hubs namespace, and an Event Hub: * A connection string over the namespace or the Event Hub: * Save your connection string in a credentials.txt file. It would be your credentials file to upload in mediarithmics platform. credentials.txt file example: ```javascript Endpoint=sb:///;SharedAccessKeyName=;SharedAccessKey= ``` ***NOTE:*** Here is the Microsoft Azure Event Hubs Pricing documentation: # Data warehouse replication ## Overview A Data Warehouse replication continuously delivers the data from mediarithmics into your own data warehouse, through an intermediate cloud storage bucket. ### Data flow ``` mediarithmics ─► cloud storage bucket (Avro files) ─► your warehouse (tables) ``` 1. **mediarithmics → bucket** — replicated operations are written as **Avro files** into a cloud storage bucket 2. **bucket → warehouse** — those files are exposed as **tables** in your warehouse, per document type The Avro files are loaded every 15 minutes in the bucket. Each document type has a **log** table (all operations, append-only) and a **state** table (the current, deduplicated view). A **daily job** rebuilds the state tables from the logs. ### Supported buckets per destination | Warehouse destination | Supported bucket | | --------------------- | --------------------------------------- | | BigQuery | Google Cloud Storage (GCS) | | Snowflake | Google Cloud Storage (GCS) or Amazon S3 | You provide the bucket credentials (GCS HMAC key or S3 access key) when setting up the replication. {% hint style="info" %} For more information on how to set up a connection to you data warehouse in mediarithmics see [Data warehouse management](/advanced-usages/data-warehouse-management) {% endhint %} ## Avro files description This section describes the **Avro files** that a `DATA_WAREHOUSE` datamart replication writes into the intermediate cloud storage bucket, and gives the exact record schema for **each document type** that can be replicated. Every file is a standard **Avro Object Container File** (`.avro`) with **one record type per file** — a file only ever contains one `…OperationRecord` type #### File naming and partitioning Files are laid out in a **Hive-style partitioned** directory tree. One top-level folder per document type, then partitioned by hour: ``` //type=/year=YYYY/month=MM/day=DD/hour=HH/data__offset_.avro ``` For example: ``` my-bucket/replication=12345/type=UserProfileOperationRecord/year=2026/month=07/day=13/hour=14/data_0_offset_0.avro ``` ### Document types and their records Each document you select in the replication filters maps to exactly one Avro record type, i.e. one `type=…` folder / one warehouse table.
Replication filter documentBucket folder (type=…) / warehouse table
USER_SEGMENTUserSegmentOperationRecord
USER_DEVICE_POINTUserDevicePointOperationRecord
USER_DEVICE_TECHNICAL_IDUserDeviceTechnicalIdOperationRecord
USER_ACCOUNTUserAccountOperationRecord
USER_EMAILUserEmailOperationRecord
USER_PROFILEUserProfileOperationRecord
USER_ACTIVITYUserActivityOperationRecord
USER_EVENTUserEventOperationRecord
USER_COMPUTED_FIELDUserComputedFieldOperationRecord
### Record schemas There is **one Avro record per document type**. Each `.avro` file embeds the schema of its own record. All records share the same three leading fields — `ts` (`timestamp-micros`), `op` (`OperationType` enum: `UPDATE` / `DELETE`) and `user_point_id` (`uuid`). In these schemas `data` is a JSON `string` and `user_identifiers` is an `array` — this is the raw Avro shape written on the bucket (the conversion to `VARIANT` / `JSON` happens later, in the state tables). #### `UserSegmentOperationRecord` ```json { "type": "record", "name": "UserSegmentOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "segment_id", "type": "long" }, { "name": "creation_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "expiration_ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-micros" } ] }, { "name": "last_modified_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } } ] } ``` #### `UserDevicePointOperationRecord` ```json { "type": "record", "name": "UserDevicePointOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "user_device_point_id", "type": "long" }, { "name": "creation_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "form_factor", "type": [ "null", "string" ] }, { "name": "os_family", "type": [ "null", "string" ] }, { "name": "os_version", "type": [ "null", "string" ] }, { "name": "browser_family", "type": [ "null", "string" ] }, { "name": "brand", "type": [ "null", "string" ] }, { "name": "model", "type": [ "null", "string" ] }, { "name": "carrier", "type": [ "null", "string" ] }, { "name": "agent_type", "type": [ "null", "string" ] } ] } ``` #### `UserDeviceTechnicalIdOperationRecord` ```json { "type": "record", "name": "UserDeviceTechnicalIdOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "user_device_point_id", "type": "long" }, { "name": "expiration_ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-micros" } ] }, { "name": "registry_type", "type": { "name": "RegistryType", "type": "enum", "symbols": [ "INSTALLATION_ID", "MUM_ID", "NETWORK_DEVICE_ID", "CUSTOM_DEVICE_ID", "MOBILE_ADVERTISING_ID", "MOBILE_VENDOR_ID", "TV_ADVERTISING_ID", "IP_V4_ADDRESS_ID", "IP_V6_ADDRESS_ID" ] } }, { "name": "registry_id", "type": "long" }, { "name": "user_device_technical_id_value", "type": "string" }, { "name": "last_seen_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } } ] } ``` #### `UserAccountOperationRecord` ```json { "type": "record", "name": "UserAccountOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "compartment_id", "type": "long" }, { "name": "user_account_id", "type": "string" }, { "name": "creation_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "expiration_ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-micros" } ] } ] } ``` #### `UserEmailOperationRecord` ```json { "type": "record", "name": "UserEmailOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "email_hash", "type": "string" }, { "name": "email", "type": [ "null", "string" ] }, { "name": "creation_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "expiration_ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-micros" } ] } ] } ``` #### `UserProfileOperationRecord` ```json { "type": "record", "name": "UserProfileOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "compartment_id", "type": "long" }, { "name": "user_account_id", "type": [ "null", "string" ] }, { "name": "creation_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "expiration_ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-micros" } ] }, { "name": "last_modified_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "data", "type": [ "null", "string" ], "doc": "The object in JSON format" } ] } ``` #### `UserActivityOperationRecord` ```json { "type": "record", "name": "UserActivityOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "activity_key", "type": { "type": "string", "logicalType": "uuid" } }, { "name": "channel_id", "type": [ "null", "string" ] }, { "name": "schema_id", "type": [ "null", "string" ] }, { "name": "user_identifiers", "type": { "type": "array", "items": { "name": "UserIdentifier", "type": "record", "fields": [ { "name": "type", "type": { "name": "UserIdentifierType", "type": "enum", "symbols": [ "EMAIL_HASH", "USER_ACCOUNT_ID", "USER_DEVICE_TECHNICAL_ID" ] } }, { "name": "registry_type", "type": [ "null", { "name": "RegistryType", "type": "enum", "symbols": [ "INSTALLATION_ID", "MUM_ID", "NETWORK_DEVICE_ID", "CUSTOM_DEVICE_ID", "MOBILE_ADVERTISING_ID", "MOBILE_VENDOR_ID", "TV_ADVERTISING_ID", "IP_V4_ADDRESS_ID", "IP_V6_ADDRESS_ID" ] } ], "default": null }, { "name": "registry_id", "type": [ "null", "string" ], "default": null }, { "name": "compartment_id", "type": [ "null", "string" ], "default": null }, { "name": "value", "type": "string" } ] } } }, { "name": "expiration_ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-micros" } ] }, { "name": "schema_error", "type": [ "null", "boolean" ] }, { "name": "data", "type": "string", "doc": "The object in JSON format" } ] } ``` #### `UserEventOperationRecord` ```json { "type": "record", "name": "UserEventOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "activity_key", "type": { "type": "string", "logicalType": "uuid" } }, { "name": "event_id", "type": { "type": "string", "logicalType": "uuid" } }, { "name": "channel_id", "type": [ "null", "string" ] }, { "name": "event_name", "type": [ "null", "string" ] }, { "name": "schema_id", "type": [ "null", "string" ] }, { "name": "user_identifiers", "type": { "type": "array", "items": { "name": "UserIdentifier", "type": "record", "fields": [ { "name": "type", "type": { "name": "UserIdentifierType", "type": "enum", "symbols": [ "EMAIL_HASH", "USER_ACCOUNT_ID", "USER_DEVICE_TECHNICAL_ID" ] } }, { "name": "registry_type", "type": [ "null", { "name": "RegistryType", "type": "enum", "symbols": [ "INSTALLATION_ID", "MUM_ID", "NETWORK_DEVICE_ID", "CUSTOM_DEVICE_ID", "MOBILE_ADVERTISING_ID", "MOBILE_VENDOR_ID", "TV_ADVERTISING_ID", "IP_V4_ADDRESS_ID", "IP_V6_ADDRESS_ID" ] } ], "default": null }, { "name": "registry_id", "type": [ "null", "string" ], "default": null }, { "name": "compartment_id", "type": [ "null", "string" ], "default": null }, { "name": "value", "type": "string" } ] } } }, { "name": "expiration_ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-micros" } ] }, { "name": "schema_error", "type": [ "null", "boolean" ] }, { "name": "data", "type": [ "null", "string" ], "doc": "The object in JSON format" } ] } ``` #### `UserComputedFieldOperationRecord` ```json { "type": "record", "name": "UserComputedFieldOperationRecord", "namespace": "com.mediarithmics.replication.format.tabular", "fields": [ { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "op", "type": { "name": "OperationType", "type": "enum", "symbols": [ "UPDATE", "DELETE" ] } }, { "name": "user_point_id", "type": { "type": "string", "logicalType": "uuid" }, "doc": "The UserPoint id" }, { "name": "plugin_instance_id", "type": "long" }, { "name": "last_modified_ts", "type": { "type": "long", "logicalType": "timestamp-micros" } }, { "name": "data", "type": [ "null", "string" ], "doc": "The object in JSON format" } ] } ``` ## Log & state table schemas For each document type, the warehouse holds a **log** table (`_log`, all operations) and a **state** table (``, deduplicated current state). Each column's type is given for both Snowflake and BigQuery. > On Snowflake, every **log** table also has these extra columns: `partition_date DATE`, `year`, `month`, `day`, `hour` (`NUMBER`). On BigQuery the log table is an external table over the Avro files (schema auto-detected). State tables add `partition_date DATE` on Snowflake only. ### Avro → warehouse type mapping | Avro type | Snowflake | BigQuery | | --------------------------- | --------------- | ----------------- | | `long` + `timestamp-micros` | `TIMESTAMP_NTZ` | `TIMESTAMP` | | `string` (incl. `uuid`) | `STRING` | `STRING` | | `enum` | `STRING` | `STRING` | | `long` | `NUMBER` | `INT64` | | `int` | `INT` | `INT64` | | `boolean` | `BOOLEAN` | `BOOL` | | `array` | `ARRAY` | `RECORD REPEATED` | | `record` | `OBJECT` | `RECORD` | In **state** tables, `long` columns are `BIGINT` (Snowflake) / `INT64` (BigQuery), and JSON payloads (`data`, `user_identifiers`) are stored as `VARIANT` (Snowflake) / `JSON` (BigQuery). ### `user_segment` **Log** (`user_segment_log`) | Column | Snowflake | BigQuery | | ------------------ | -------------- | --------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `segment_id` | NUMBER | INT64 | | `creation_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `expiration_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `last_modified_ts` | TIMESTAMP\_NTZ | TIMESTAMP | **State** (`user_segment`) — cluster by `segment_id` | Column | Snowflake | BigQuery | | --------------- | --------- | --------- | | `user_point_id` | STRING | STRING | | `segment_id` | BIGINT | INT64 | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_device_point` **Log** (`user_device_point_log`) | Column | Snowflake | BigQuery | | ---------------------- | -------------- | --------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `user_device_point_id` | NUMBER | INT64 | | `creation_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `form_factor` | STRING | STRING | | `os_family` | STRING | STRING | | `os_version` | STRING | STRING | | `browser_family` | STRING | STRING | | `brand` | STRING | STRING | | `model` | STRING | STRING | | `carrier` | STRING | STRING | | `agent_type` | STRING | STRING | **State** (`user_device_point`) — no clustering | Column | Snowflake | BigQuery | | ---------------------- | --------- | --------- | | `user_point_id` | STRING | STRING | | `user_device_point_id` | BIGINT | INT64 | | `brand` | STRING | STRING | | `model` | STRING | STRING | | `agent_type` | STRING | STRING | | `browser_family` | STRING | STRING | | `form_factor` | STRING | STRING | | `os_family` | STRING | STRING | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_device_technical_id` **Log** (`user_device_technical_id_log`) | Column | Snowflake | BigQuery | | -------------------------------- | -------------- | --------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `user_device_point_id` | NUMBER | INT64 | | `expiration_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `registry_type` | STRING | STRING | | `registry_id` | NUMBER | INT64 | | `user_device_technical_id_value` | STRING | STRING | | `last_seen_ts` | TIMESTAMP\_NTZ | TIMESTAMP | **State** (`user_device_technical_id`) — cluster by `registry_id` | Column | Snowflake | BigQuery | | -------------------------- | --------- | --------- | | `user_point_id` | STRING | STRING | | `user_device_technical_id` | STRING | STRING | | `registry_type` | STRING | STRING | | `registry_id` | BIGINT | INT64 | | `id` | STRING | STRING | | `user_device_point_id` | BIGINT | INT64 | | `last_seen_ts` | TIMESTAMP | TIMESTAMP | | `expiration_ts` | TIMESTAMP | TIMESTAMP | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_account` **Log** (`user_account_log`) | Column | Snowflake | BigQuery | | ----------------- | -------------- | --------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `compartment_id` | NUMBER | INT64 | | `user_account_id` | STRING | STRING | | `creation_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `expiration_ts` | TIMESTAMP\_NTZ | TIMESTAMP | **State** (`user_account`) — cluster by `compartment_id` | Column | Snowflake | BigQuery | | ----------------- | --------- | --------- | | `user_point_id` | STRING | STRING | | `compartment_id` | BIGINT | INT64 | | `user_account_id` | STRING | STRING | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_email` **Log** (`user_email_log`) | Column | Snowflake | BigQuery | | --------------- | -------------- | --------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `email_hash` | STRING | STRING | | `email` | STRING | STRING | | `creation_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `expiration_ts` | TIMESTAMP\_NTZ | TIMESTAMP | **State** (`user_email`) — no clustering | Column | Snowflake | BigQuery | | --------------- | --------- | --------- | | `user_point_id` | STRING | STRING | | `email_hash` | STRING | STRING | | `email` | STRING | STRING | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_profile` **Log** (`user_profile_log`) | Column | Snowflake | BigQuery | | ------------------ | -------------- | --------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `compartment_id` | NUMBER | INT64 | | `user_account_id` | STRING | STRING | | `creation_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `expiration_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `last_modified_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `data` | STRING | STRING | **State** (`user_profile`) — cluster by `compartment_id` | Column | Snowflake | BigQuery | | ----------------- | --------- | --------- | | `user_point_id` | STRING | STRING | | `compartment_id` | BIGINT | INT64 | | `user_account_id` | STRING | STRING | | `data` | VARIANT | JSON | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_activity` **Log** (`user_activity_log`) | Column | Snowflake | BigQuery | | ------------------ | -------------- | --------------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `activity_key` | STRING | STRING | | `channel_id` | STRING | STRING | | `schema_id` | STRING | STRING | | `user_identifiers` | ARRAY | RECORD REPEATED | | `expiration_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `schema_error` | BOOLEAN | BOOL | | `data` | STRING | STRING | **State** (`user_activity`) — no clustering | Column | Snowflake | BigQuery | | ------------------ | --------- | --------- | | `user_point_id` | STRING | STRING | | `activity_key` | STRING | STRING | | `channel_id` | STRING | STRING | | `user_identifiers` | VARIANT | JSON | | `expiration_ts` | TIMESTAMP | TIMESTAMP | | `data` | VARIANT | JSON | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_event` **Log** (`user_event_log`) | Column | Snowflake | BigQuery | | ------------------ | -------------- | --------------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `activity_key` | STRING | STRING | | `event_id` | STRING | STRING | | `channel_id` | STRING | STRING | | `event_name` | STRING | STRING | | `schema_id` | STRING | STRING | | `user_identifiers` | ARRAY | RECORD REPEATED | | `expiration_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `schema_error` | BOOLEAN | BOOL | | `data` | STRING | STRING | **State** (`user_event`) — cluster by `activity_key` | Column | Snowflake | BigQuery | | ------------------ | --------- | --------- | | `user_point_id` | STRING | STRING | | `event_id` | STRING | STRING | | `activity_key` | STRING | STRING | | `channel_id` | STRING | STRING | | `event_name` | STRING | STRING | | `user_identifiers` | VARIANT | JSON | | `expiration_ts` | TIMESTAMP | TIMESTAMP | | `data` | VARIANT | JSON | | `ts` | TIMESTAMP | TIMESTAMP | *** ### `user_computed_field` **Log** (`user_computed_field_log`) | Column | Snowflake | BigQuery | | -------------------- | -------------- | --------- | | `ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `op` | STRING | STRING | | `user_point_id` | STRING | STRING | | `plugin_instance_id` | NUMBER | INT64 | | `last_modified_ts` | TIMESTAMP\_NTZ | TIMESTAMP | | `data` | STRING | STRING | **State** (`user_computed_field`) — cluster by `plugin_instance_id` | Column | Snowflake | BigQuery | | -------------------- | --------- | --------- | | `user_point_id` | STRING | STRING | | `plugin_instance_id` | BIGINT | INT64 | | `data` | VARIANT | JSON | | `ts` | TIMESTAMP | TIMESTAMP | *** Setting up replication In order to set up a replication to your data warehouse, follow these steps * [BigQuery](/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication/bigquery) * [Snowflake](/advanced-usages/exporting-your-data/datamart-replication/data-warehouse-replication/snowflake) # BigQuery # Snowflake # Destination file ## Overview A **destination file** defines where and how data files generated by the platform are delivered (for example via SFTP or cloud storage). It is configured directly in the **computing console**. On its own, a destination file does not actively export data. It becomes effective when a **feed** (through the *stream-to-file* capability) references the destination file using its **token (auto-generated)**. In that case, the feed exports data chunks to the delivery file system, which aggregates them into a single file and delivers it to the targeted destination. Typical use cases include: * Sending audience segment's user identifiers to an external partner for activation. * Supporting custom connectors relying on file-based delivery. ### Supported protocols & object storages A destination file can target one of the following systems: * **SFTP** – Secure file transfer over SSH * **S3** – Amazon S3 buckets * **Google Cloud Storage** – mediarithmics GCS buckets The destination `type` property defines how and where the files are physically uploaded. ## Destination file setup The setup is done in the Computing console > File destinations. {% stepper %} {% step %} ### Create the destination file 1. Click on *New file delivery destination* button. 2. Fill in the required [configuration fields](#configuration-fields). 3. Save. {% endstep %} {% step %} ### Add the credentials 1. In the action menu of the created file delivery destination, select the *Credentials* option. 2. Copy the provided template (different for each storage type) and fill it in with the required fields. 3. Save. {% hint style="info" %} Once saved, you won't be able to read the credential file again. But you will be able to overwrite it. {% endhint %} {% endstep %} {% step %} ### Validate the connection In the action menu of the created file delivery destination, select the *Validate* option. {% endstep %} {% step %} ### Use it Use the token to reference the destination file. {% endstep %} {% endstepper %} {% hint style="info" %} To validate the connection, the integration user must have an Editor role on the object storage (or equivalent) with permissions to create and delete a temporary empty file for verification. {% endhint %} ## Configuration fields ### General information | Field | Description | | ------------------ | ----------------------------------------------------------------------------------------------------------------------- | | **Name** | Human-readable name used in the UI. | | **Technical name** | Internal identifier used by connectors. This value is referenced in plugin code but has no functional impact by itself. | | **Type** | Protocols or object storage type: `SFTP`, `S3`, or `GOOGLE_CLOUD_STORAGE`. | | **Deduplicate** | When enabled, removes *perfect duplicates* within a file (two rows that are strictly identical). | ### File information | Field | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **File path macro** | Path (from the root of the destination) where the file will be written. Must **not** **start with `/`** and must **end with `/`**. Supports dynamic macros. | | **File name macro** | File name to generate. Supports dynamic macros. | | **Compression** | File compression mode: `NONE`, `GZIP`, or `ZIP`. | | **Encoding** | Character encoding. Recommended and supported value: `UTF-8`. | | **File header macro** | Optional header written at the top of the file (for example column names). Can be left empty. Supports macros. | ### Upload trigger You must select one of the trigger types: 1. **Size-based**: Files are uploaded immediately when they reach the specified size limit. Otherwise, a secondary time-based interval ensures data is delivered even if the size limit is not reached. 2. **Frequency-based**: Files are uploaded based on a strict daily limit. This restricts the maximum number of files sent to the destination within a rolling 24-hour period. If you have selected Size-based, fill in the following fields: | Field | Description | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **File max size (KB)** |

As soon as the aggregated data reaches this size, the file is immediately uploaded to the destination.

Recommended value: 100,000 KB (≈100 MB).

| | **Min upload per day** | Controls the upload **frequency** (number of upload windows per rolling 24h period). Acts as a secondary trigger when file size is not reached. | If you have selected Frequency-based, fill in the following field: | Field | Description | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Max upload per day** | Divides a rolling 24-hour period into equal upload windows to cap the daily file count. Uploads are only delayed if the destination server is still processing the previous file | ## Technical appendix ### Macros and templating All macros rely on **Freemarker** templating. Macros can be used in: * File path * File name * File header They allow dynamic file organization based on time, segments, or feeds configuration. #### Commonly used variable in macros | Macro | Description | | ------------------------ | ---------------------------------------------------------------------------------------------- | | `DATE` | Evaluates the date at which the file is generated. | | `PROPERTY` | Evaluates a property defined in the plugin configuration. | | `SEGMENT.ID` | Returns the segment ID. | | `SEGMENT.TECHNICAL_NAME` | Returns the segment technical name. | | `GROUPING_KEY` | Evaluates the `grouping_key` defined in the plugin code. Used only in the context of a plugin. | | `RANDOM_UUID` | Generates a random UUID for uniqueness. | #### Date formatting example Freemarker date formatting can be used to structure folders: ``` data/partner/events/${DATE?string["yyyy/MM/dd"]}/ ``` This example generates a hierarchical path such as: ``` data/partner/events/2025/03/18/ ```
Example configuration **File path macro** ``` data/exports/${SEGMENT.TECHNICAL_NAME}/${DATE?string["yyyy/MM"]}/ ``` **File name macro** ``` events_${GROUPING_KEY}_${RANDOM_UUID}.csv ``` This setup produces compressed, monthly-partitioned files, uniquely identified per execution.
### Grouping key The **grouping key** is a concept used **exclusively in plugins** (during the `user_segment_update` phase) to control how records are aggregated. It defines **which records belong to the same logical file stream**. Records sharing the same grouping key value are grouped together and written into the same file(s). The grouping key is **defined in the** **plugin code**, not in the destination file configuration. The destination file can then reference it through the `${GROUPING_KEY}` macro. #### Common Grouping Strategies * **Grouping by `segment_id`**\ Accumulates records belonging to the same segment. * Can mix data coming from **multiple plugin instances** as long as they target the same segment. * Useful for segment-centric exports. * **Grouping by `datamart_id`**\ Accumulates records coming from all plugin instances within the same datamart. * Produces consolidated exports at the datamart level. * **Grouping by `feed_id`**\ Accumulates records coming from a **single plugin instance only**. * Ensures strict isolation between feeds. ### File size & upload frequency behavior File uploads are controlled by **two effective parameters**. A file is uploaded as soon as **one of the active conditions is met**. #### File max size (primary trigger) * The **file max size** is always the **main limiting factor**. * While writing records, as soon as the file reaches the configured size limit, it is **immediately uploaded** to the destination. * This behavior applies regardless of time-based settings. #### Min upload per day (secondary Trigger) Despite its name, this parameter should be understood as: > **Number of upload windows per rolling 24-hour period** * It defines a **time-based flush interval**. * Even if the file size limit is not reached, the file will be uploaded at the end of each interval. * The period is **rolling**, not aligned to fixed clock boundaries. The interval duration is calculated as: ``` 24 hours / min_upload_per_day ``` #### Max upload per day It divides a rolling 24-hour period into equal upload windows to cap the daily file count. The interval duration is calculated as: ``` 24 hours / max_upload_per_day ``` Then, it uploads the files to respect the maximum number of files set on the time period. If a same file (names match exactly) is present on the destination server, we a delay the upload and aggregate the coming data to send in the next file upload. #### Examples **Example 1** ``` File max size (in kB) = 100000 Min upload per day = 1 ``` Behavior: * Files are uploaded immediately when they reach **100 MB**. * If the size is not reached, the file is uploaded at the end of a **rolling 24-hour period**. This configuration: * Does **not guarantee daily delivery** (in case of no data coming from the feed). * Is suitable when file volume is unpredictable and may require more than one file per day. **Example 2** ``` File max size (in kB) = 100000 Min upload per day = 24 ``` Behavior: * Files are uploaded immediately when they reach **100 MB**. * If the size is not reached, the file is uploaded at the end of a **rolling 1-hour period**. This configuration: * Caps delivery to **at most one file per hour**. * Is **not suitable** if more than one file per hour is required. ### Key notes & best practices * Use **compression** for large exports to optimize transfer and storage costs. * Prefer date-based partitioning in paths to simplify downstream processing and retention. # Data warehouse management {% hint style="info" %} This feature is in Alpha, please contact your Account Manager to learn more. {% endhint %} If you wish to leverage data stored in your data warehouse, you can declare you own data warehouse in mediarithmics as a data source. We currently support the following warehouses : * [Google BigQuery](/advanced-usages/data-warehouse-management/create-a-data-warehouse/bigquery) * [Snowflake](/advanced-usages/data-warehouse-management/create-a-data-warehouse/snowflake) Learn more : * [How to set up a connection to your data warehouse ](/advanced-usages/data-warehouse-management/create-a-data-warehouse) * [Ingest data coming from your warehouse](/data-ingestion/data-warehouse-ingestion) --- [Next Page](/llms-full.txt/1)