Cleaning rules
Cleaning rules define how each datamart manages the deletion of user data. They can apply to two different user objects: profiles and events.
When an object enters the datamart, mediarithmics computes the expiration timestamp of this object based on the different cleaning rules that can be applied. This expiration timestamp is part of the object properties, and can't be modified. That means cleaning rules are non-retroactive: changing the rules will only have an effect on future incoming events and profiles.
A user profile or user event enters the datamart
Which cleaning rules apply to it?
Based on the cleaning rules and the priority between them, compute the expiration timestamp of the object
Store the expiration timestamp in the object
An event or profile expires if it hasn't been modified before the expiration timestamp. In details:
For profile: last modification timestamp field (
$last_modified_ts
for profile) is checked against expiration timestamp field.For event: since an event cannot be updated, creation timestamp field (
$ts
) is checked against expiration timestamp field.
When an object expires, it does not return in queries or in the interface. It still exists in the passive database, until it is removed by a scheduled operation.
All cleaning rules have a status (DRAFT
, LIVE
or ARCHIVED
), an action (KEEP
or DELETE
), a duration and a set of filters.
KEEP
action means the associated object should stay in the datamart for the duration.DELETE
action means the associated object should be removed from the datamart after the duration has expired.
There are two types of cleaning rules:
Event-based cleaning rules apply to events. They can apply to all events, or only to different channels, activity types or event names.
Profile-based cleaning rules apply to profiles. They can apply to all profiles or only to a compartment.
When a user point does not have any activity, nor profile, nor USER_LIST
segment, it is removed from the platform. By cleaning activities and profiles, cleaning rules will clean your user points.
Cleaning rules can be easily edited in the navigator, in your datamart settings
Rules priority
KEEP
cleaning rules have the priority on DELETE
rules.
Only LIVE
cleaning rules are taken into account when computing the expiration timestamp of an object.
Example 1
The following cleaning rules apply to my incoming event : - KEEP for 60 days - KEEP for 180 days - DELETE after 150 days
The event will be kept for 180 days then deleted.
Example 2
The following cleaning rules apply to my incoming event : - KEEP for 60 days - DELETE after 150 days
The event will be deleted after 150 days.
Example 3
The following cleaning rules apply to my incoming profile : - DELETE after 10 days - DELETE after 150 days
The profile will be deleted after 10 days.
Rules lifecycle
Cleaning rules are created with the DRAFT
status. They can't be created LIVE
or ARCHIVED
. While DRAFT
you can change the different properties of the rule, except its type.
You then change the rule status to LIVE
by doing an edit request. Once live, a rule can't be edited anymore and only the status can be changed to ARCHIVED
.
At any moment, there should always be a LIVE
event based rule with a DELETE
action. You'll need to create at least an other rule to be able to archive this one.
Once ARCHIVED
, rules can't be changed anymore, nor deleted.
You can only delete DRAFT
cleaning rules. Once LIVE
, they'll have to be ARCHIVED
Object reference
A cleaning rule has the following set of properties.
Property | Type | Description |
id | string | The cleaning rule identifier |
action | enum |
Only |
archived | boolean | Used to make a rule disappear from the UI. You can only set it to |
status | enum |
A cleaning rule can only go from the |
datamart_id | string | The ID of the associated datamart |
type | enum |
|
life_duration | ISO 8601 duration | The duration associated with the cleaning rule |
channel_filter | string |
Optional. If a cleaning rule only applies to a channel, specifies its ID.
|
activity_type_filter | enum |
Optional. If a cleaning rule only applies to a specific activity type, specifies its ID.
|
compartment_filter | string |
Optional. If a cleaning rule only applies to a compartment, specified its ID. |
Content filter
Cleaning rules of type USER_EVENT_CLEANING_RULE
can be associated with a content filter :
Proprty | Type | Description |
filter | string | Value of the filter |
content_type | enum | Target of the filter. Only |
Samples
List cleaning rules in a datamart
GET
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules
Path Parameters
Name | Type | Description |
---|---|---|
datamartId | string | The ID of the datamart |
Query Parameters
Name | Type | Description |
---|---|---|
type | string |
|
Get a cleaning rule
GET
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId
Path Parameters
Name | Type | Description |
---|---|---|
datamartId | string | The ID of the datamart |
ruleId | string | The ID of the cleaning rule |
Get the content filter for a rule
GET
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter
Only applies to event-based cleaning rules
Path Parameters
Name | Type | Description |
---|---|---|
datamartId | string | The ID of the datamart |
ruleId | string | The ID of the rule |
Create a cleaning rule
POST
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules
Path Parameters
Name | Type | Description |
---|---|---|
datamartId | string | The ID of the datamart |
Request Body
Name | Type | Description |
---|---|---|
activity_type_filter | string | Only for event-based rules. |
channel_filter | string | Only for event-based rules. The ID of the channel on which the rule applies |
compartment_id | string | Only for profile-based rules. The ID of the compartment on which the rule applies. |
life_duration | string | ISO 8601 duration. Example "P160D" |
status | string |
|
type | string |
|
action | string |
|
Update a cleaning rule and/or change its status
PUT
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId
Path Parameters
Name | Type | Description |
---|---|---|
ruleId | string | The ID of the rule to update |
datamartId | string | The ID of the datamart |
Request Body
Name | Type | Description |
---|---|---|
archived | boolean |
|
id | string | Required to set the status to |
activity_type_filter | string | Only for event-based rules. |
channel_filter | string | Only for event-based rules. The ID of the channel on which the rule applies.
Can't be changed if the status is not |
compartment_id | string | Only for profile-based rules. The ID of the compartment on which the rule applies.
Can't be changed if the status is not |
life_duration | string | ISO 8601 duration. Example "P160D".
Can't be changed if the status is not |
status | string |
|
type | string |
|
action | string |
|
You'll have an error if doing any update that is prohibited, like changing a property while the status is not DRAFT or
Delete a DRAFT rule
DELETE
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId
Path Parameters
Name | Type | Description |
---|---|---|
datamartId | string | The ID of the datamart |
ruleId | string | The ID of the rule to delete |
Add or update a content filter for a rule
POST
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter
Path Parameters
Name | Type | Description |
---|---|---|
datamartId | string | The ID of the datamart |
ruleId | string | The ID of the rule |
Request Body
Name | Type | Description |
---|---|---|
content-type | string |
|
filter | string | Name of the events for which the rule will apply |
Remove a content filter for a rule
DELETE
https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter
Path Parameters
Name | Type | Description |
---|---|---|
datamartId | string | The ID of the datamart |
ruleId | string | The ID of the rule |
Last updated