# Cleaning rules Cleaning rules **define how each datamart manages the deletion of user data**. They can apply to two different user objects: [profiles](https://developer.mediarithmics.io/user-points/user-profiles) and [events](https://developer.mediarithmics.io/user-points/user-activities). When an object enters the datamart, **mediarithmics computes the expiration timestamp of this object based on the different cleaning rules that can be applied**. This expiration timestamp is part of the object properties, and can't be modified. That means **cleaning rules are non-retroactive**: changing the rules will only have an effect on future incoming events and profiles. {% hint style="success" %} 1. A UserProfile or UserEvent enters the datamart 2. Which cleaning rules apply to it? 3. Based on the cleaning rules and the priority between them, compute the expiration timestamp of the object 4. Store the expiration timestamp in the object {% endhint %} An event or profile expires if it hasn't been modified before the expiration timestamp. In details: * For profile: last modification timestamp field (`$last_modified_ts` for profile) is checked against expiration timestamp field. * For event: since an event cannot be updated, creation timestamp field (`$ts`) is checked against expiration timestamp field. **When an object expires, it does not return in queries or in the interface**. It still exists in the passive database, until it is removed by a scheduled operation. All cleaning rules have a **status** (`DRAFT`, `LIVE` or `ARCHIVED`), an **action** (`KEEP` or `DELETE`), a duration and a set of filters. `KEEP` action means the associated object should stay in the datamart for the duration.`DELETE` action means the associated object should be removed from the datamart after the duration has expired. There are two types of cleaning rules: * **Event-based cleaning rules** apply to events. They can apply to all events, or only to different channels, activity types or event names. * **Profile-based cleaning rules** apply to profiles. They can apply to all profiles or only to a compartment. {% hint style="success" %} When a UserPoint does not have any activity, nor profile, nor `USER_LIST` segment, it is removed from the platform. By cleaning activities and profiles, cleaning rules will clean your UserPoint. {% endhint %} {% hint style="info" %} Cleaning rules can be easily edited in the navigator, in your datamart settings {% endhint %} ## Rules priority `KEEP` cleaning rules have the priority on `DELETE` rules. Only `LIVE` cleaning rules are taken into account when computing the expiration timestamp of an object. ### Example 1 The following cleaning rules apply to my incoming event :\ \- KEEP for 60 days\ \- KEEP for 180 days\ \- DELETE after 150 days The event will be kept for 180 days then deleted. ### Example 2 The following cleaning rules apply to my incoming event :\ \- KEEP for 60 days\ \- DELETE after 150 days The event will be deleted after 150 days. ### Example 3 The following cleaning rules apply to my incoming profile :\ \- DELETE after 10 days\ \- DELETE after 150 days The profile will be deleted after 10 days. ## Rules lifecycle Cleaning rules are created with the `DRAFT` status. They can't be created `LIVE` or `ARCHIVED`. While `DRAFT` you can change the different properties of the rule, except its type.\ \ You then change the rule status to `LIVE` by doing an edit request. Once live, a rule can't be edited anymore and only the status can be changed to `ARCHIVED`. {% hint style="info" %} At any moment, there should always be a `LIVE` event based rule with a `DELETE` action. You'll need to create at least an other rule to be able to archive this one. {% endhint %} Once `ARCHIVED`, rules can't be changed anymore, nor deleted. You can only delete `DRAFT` cleaning rules. Once `LIVE`, they'll have to be `ARCHIVED` ## Object reference A cleaning rule has the following set of properties. | Property | Type | Description | | ---------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id | string | The cleaning rule identifier | | action | enum |

KEEP or DELETE.

KEEP means the cleaning rule will force data to be kept for the duration, while DELETE means the data should be deleted after the duration.

Only DELETE is allowed for cleaning rules of type USER\_PROFILE\_CLEANING\_RULE

| | archived | boolean | Used to make a rule disappear from the UI. You can only set it to `true` if the status is already `ARCHIVED` | | status | enum |

ARCHIVED, DRAFT or LIVE.

A cleaning rule can only go from the DRAFT status to LIVE, and can only go from LIVE to ARCHIVED.

| | datamart\_id | string | The ID of the associated datamart | | type | enum | `USER_EVENT_CLEANING_RULE` or `USER_PROFILE_CLEANING_RULE` | | life\_duration | ISO 8601 duration | The duration associated with the cleaning rule | | channel\_filter | string |

USER\_EVENT\_CLEANING\_RULE only.


Optional. If a cleaning rule only applies to a channel, specifies its ID.

channel\_filter must be compatible activity\_type\_filter if set at the same time (for example you can't target an app channel if you set the activity type to DISPLAY\_AD)

| | activity\_type\_filter | enum |

USER\_EVENT\_CLEANING\_RULE only.


Optional. If a cleaning rule only applies to a specific activity type, specifies its ID.

SITE\_VISIT, APP\_VISIT, TOUCH, DISPLAY\_ADor EMAIL .

channel\_filter must be compatible activity\_type\_filter if set at the same time (for example you can't target an app channel if you set the activity type to DISPLAY\_AD)

| | compartment\_filter | string |

USER\_PROFILE\_CLEANING\_RULE only.

Optional. If a cleaning rule only applies to a compartment, specified its ID.

| ### Content filter Cleaning rules of type `USER_EVENT_CLEANING_RULE` can be associated with a content filter : | Proprty | Type | Description | | ------------- | ------ | --------------------------------------------------------------- | | filter | string | Value of the filter | | content\_type | enum | Target of the filter. Only `EVENT_NAME_FILTER` allowed for now. | ### Samples ```javascript // KEEP UserEvent cleaning rule with channel filter and activity type filter { "id": "907", "action": "KEEP", "archived": false, "datamart_id": "1509", "channel_filter": "3844", "activity_type_filter": "SITE_VISIT", "life_duration": "P65D", "status": "DRAFT", "type": "USER_EVENT_CLEANING_RULE" } // DELETE UserEvent cleaning rule without filters { "id": "778", "action": "DELETE", "archived": false, "datamart_id": "1509", "channel_filter": null, "activity_type_filter": null, "life_duration": "P60D", "status": "LIVE", "type": "USER_EVENT_CLEANING_RULE" } // its associated content filter // DELETE UserProfile cleaning rule with compartment filter { "id": "905", "action": "DELETE", "archived": false, "datamart_id": "1509", "compartment_filter": "1606", "life_duration": "P365D", "status": "ARCHIVED", "type": "USER_PROFILE_CLEANING_RULE" } ``` ## List cleaning rules in a datamart `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | string | The ID of the datamart | #### Query Parameters | Name | Type | Description | | ---- | ------ | -------------------------------------------------------------------------------------------------------- | | type | string | `USER_EVENT_CLEANING_RULE` or `USER_PROFILE_CLEANING_RULE` to filter on a specific type of cleaning rule | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": [ { "id": "904", "action": "DELETE", "archived": false, "datamart_id": "1509", "compartment_filter": null, "life_duration": "P300D", "status": "LIVE", "type": "USER_PROFILE_CLEANING_RULE" }, { "id": "778", "action": "DELETE", "archived": false, "datamart_id": "1509", "channel_filter": null, "activity_type_filter": null, "life_duration": "P60D", "status": "LIVE", "type": "USER_EVENT_CLEANING_RULE" }, { "id": "902", "action": "KEEP", "archived": false, "datamart_id": "1509", "channel_filter": "3844", "activity_type_filter": null, "life_duration": "P250D", "status": "ARCHIVED", "type": "USER_EVENT_CLEANING_RULE" } ], "count": 3, "total": 3, "first_result": 0, "max_result": 50, "max_results": 50 } ``` {% endtab %} {% endtabs %} ## Get a cleaning rule `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId` #### Path Parameters | Name | Type | Description | | ---------- | ------ | --------------------------- | | datamartId | string | The ID of the datamart | | ruleId | string | The ID of the cleaning rule | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "778", "action": "DELETE", "archived": false, "datamart_id": "1509", "channel_filter": null, "activity_type_filter": null, "life_duration": "P60D", "status": "LIVE", "type": "USER_EVENT_CLEANING_RULE" } } ``` {% endtab %} {% endtabs %} ## Get the content filter for a rule `GET` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter` Only applies to event-based cleaning rules #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | string | The ID of the datamart | | ruleId | string | The ID of the rule | {% tabs %} {% tab title="200 " %} ```javascript { "status":"ok", "data":{ "filter":"$transaction_confirmed", "content_type":"EVENT_NAME_FILTER" } } ``` {% endtab %} {% tab title="404 If the rule doesn't have a content filter" %} ```javascript { "status": "error", "error": "user_event_cleaning_rule 778 has no content_filter", "error_id": "2e5d3338-34ec-4abd-aaff-fb370958ff2a" } ``` {% endtab %} {% endtabs %} ## Create a cleaning rule `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | string | The ID of the datamart | #### Request Body | Name | Type | Description | | ---------------------- | ------ | ------------------------------------------------------------------------------------------------------ | | activity\_type\_filter | string | Only for event-based rules. `SITE_VISIT`, `APP_VISIT`, `TOUCH`, `DISPLAY_AD`or `EMAIL` . | | channel\_filter | string | Only for event-based rules. The ID of the channel on which the rule applies | | compartment\_id | string | Only for profile-based rules. The ID of the compartment on which the rule applies. | | life\_duration | string | ISO 8601 duration. Example "P160D" | | status | string | `DRAFT` | | type | string | `USER_EVENT_CLEANING_RULE` or `USER_PROFILE_CLEANING_RULE` | | action | string |

KEEP or DELETE.

Only DELETE for event-based rules

| {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "917", "action": "KEEP", "archived": false, "datamart_id": "1509", "channel_filter": "3844", "activity_type_filter": "SITE_VISIT", "life_duration": "P70D", "status": "LIVE", "type": "USER_EVENT_CLEANING_RULE" } } ``` {% endtab %} {% endtabs %} ## Update a cleaning rule and/or change its status `PUT` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------------- | | ruleId | string | The ID of the rule to update | | datamartId | string | The ID of the datamart | #### Request Body | Name | Type | Description | | ---------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | archived | boolean | `true` to make the rule disappear from the UI. Can only be put to `true` if the status is already `ARCHIVED`. Use `false` otherwise. | | id | string | Required to set the status to `ARCHIVED`. The ID of the rule to update. | | activity\_type\_filter | string |

Only for event-based rules. SITE\_VISIT, APP\_VISIT, TOUCH, DISPLAY\_ADor EMAIL .

Can't be changed if the status is not DRAFT.

| | channel\_filter | string |

Only for event-based rules. The ID of the channel on which the rule applies.

Can't be changed if the status is not DRAFT.

| | compartment\_id | string |

Only for profile-based rules. The ID of the compartment on which the rule applies.

Can't be changed if the status is not DRAFT.

| | life\_duration | string |

ISO 8601 duration. Example "P160D".

Can't be changed if the status is not DRAFT.

| | status | string | `DRAFT` to keep it as a draft. `LIVE` to publish it from draft. `ARCHIVED` to archive it from `LIVE` | | type | string |

USER\_EVENT\_CLEANING\_RULE or USER\_PROFILE\_CLEANING\_RULE based on the existing rule type.

Can't be changed.

| | action | string |

KEEP or DELETE.

Only DELETE for event-based rules

Can't be changed if the status is not DRAFT.

| {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "id": "917", "action": "KEEP", "archived": false, "datamart_id": "1509", "channel_filter": "3844", "activity_type_filter": "SITE_VISIT", "life_duration": "P70D", "status": "LIVE", "type": "USER_EVENT_CLEANING_RULE" } } ``` {% endtab %} {% endtabs %} {% hint style="warning" %} You'll have an error if doing any update that is prohibited, like changing a property while the status is not DRAFT or {% endhint %} ## Delete a DRAFT rule `DELETE` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------------- | | datamartId | string | The ID of the datamart | | ruleId | string | The ID of the rule to delete | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok" } ``` {% endtab %} {% endtabs %} ## Add or update a content filter for a rule `POST` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | string | The ID of the datamart | | ruleId | string | The ID of the rule | #### Request Body | Name | Type | Description | | ------------ | ------ | ------------------------------------------------ | | content-type | string | `EVENT_NAME_FILTER` | | filter | string | Name of the events for which the rule will apply | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "filter": "$transaction_confirmed", "content_type": "EVENT_NAME_FILTER" } } ``` {% endtab %} {% endtabs %} ## Remove a content filter for a rule `DELETE` `https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter` #### Path Parameters | Name | Type | Description | | ---------- | ------ | ---------------------- | | datamartId | string | The ID of the datamart | | ruleId | string | The ID of the rule | {% tabs %} {% tab title="200 " %} ```javascript { "status": "ok", "data": { "filter": "$transaction_confirmed", "content_type": "EVENT_NAME_FILTER" } } ``` {% endtab %} {% endtabs %}