Cleaning rules

Cleaning rules define how each datamart manages the deletion of user data. They can apply to two different user objects: profiles and events.

When an object enters the datamart, mediarithmics computes the expiration timestamp of this object based on the different cleaning rules that can be applied. This expiration timestamp is part of the object properties, and can't be modified. That means cleaning rules are non-retroactive: changing the rules will only have an effect on future incoming events and profiles.

1. A user profile or user event enters the datamart

2. Which cleaning rules apply to it?

3. Based on the cleaning rules and the priority between them, compute the expiration timestamp of the object

4. Store the expiration timestamp in the object

An event or profile expires if it hasn't been modified before the expiration timestamp. In details:

• For profile: last modification timestamp field ($last_modified_ts for profile) is checked against expiration timestamp field.

• For event: since an event cannot be updated, creation timestamp field ($ts) is checked against expiration timestamp field.

When an object expires, it does not return in queries or in the interface. It still exists in the passive database, until it is removed by a scheduled operation.

All cleaning rules have a status (DRAFT, LIVE or ARCHIVED), an action (KEEP or DELETE), a duration and a set of filters.

KEEP action means the associated object should stay in the datamart for the duration.DELETE action means the associated object should be removed from the datamart after the duration has expired.

There are two types of cleaning rules:

• Event-based cleaning rules apply to events. They can apply to all events, or only to different channels, activity types or event names.

• Profile-based cleaning rules apply to profiles. They can apply to all profiles or only to a compartment.

When a user point does not have any activity, nor profile, nor USER_LIST segment, it is removed from the platform. By cleaning activities and profiles, cleaning rules will clean your user points.

Cleaning rules can be easily edited in the navigator, in your datamart settings

Rules priority

KEEP cleaning rules have the priority on DELETE rules.

Only LIVE cleaning rules are taken into account when computing the expiration timestamp of an object.

Example 1

The following cleaning rules apply to my incoming event : - KEEP for 60 days - KEEP for 180 days - DELETE after 150 days

The event will be kept for 180 days then deleted.

Example 2

The following cleaning rules apply to my incoming event : - KEEP for 60 days - DELETE after 150 days

The event will be deleted after 150 days.

Example 3

The following cleaning rules apply to my incoming profile : - DELETE after 10 days - DELETE after 150 days

The profile will be deleted after 10 days.

Rules lifecycle

Cleaning rules are created with the DRAFT status. They can't be created LIVE or ARCHIVED. While DRAFT you can change the different properties of the rule, except its type. You then change the rule status to LIVE by doing an edit request. Once live, a rule can't be edited anymore and only the status can be changed to ARCHIVED.

At any moment, there should always be a LIVE event based rule with a DELETE action. You'll need to create at least an other rule to be able to archive this one.

Once ARCHIVED, rules can't be changed anymore, nor deleted.

You can only delete DRAFT cleaning rules. Once LIVE, they'll have to be ARCHIVED

Object reference

A cleaning rule has the following set of properties.

Property	Type	Description
id	string	The cleaning rule identifier
action	enum	KEEP or DELETE. KEEP means the cleaning rule will force data to be kept for the duration, while DELETE means the data should be deleted after the duration. Only DELETE is allowed for cleaning rules of type USER_PROFILE_CLEANING_RULE
archived	boolean	Used to make a rule disappear from the UI. You can only set it to true if the status is already ARCHIVED
status	enum	ARCHIVED, DRAFT or LIVE. A cleaning rule can only go from the DRAFT status to LIVE, and can only go from LIVE to ARCHIVED.
datamart_id	string	The ID of the associated datamart
type	enum	USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE
life_duration	ISO 8601 duration	The duration associated with the cleaning rule
channel_filter	string	USER_EVENT_CLEANING_RULE only. Optional. If a cleaning rule only applies to a channel, specifies its ID. channel_filter must be compatible activity_type_filter if set at the same time (for example you can't target an app channel if you set the activity type to DISPLAY_AD)
activity_type_filter	enum	USER_EVENT_CLEANING_RULE only. Optional. If a cleaning rule only applies to a specific activity type, specifies its ID. SITE_VISIT, APP_VISIT, TOUCH, DISPLAY_ADor EMAIL . channel_filter must be compatible activity_type_filter if set at the same time (for example you can't target an app channel if you set the activity type to DISPLAY_AD)
compartment_filter	string	USER_PROFILE_CLEANING_RULE only. Optional. If a cleaning rule only applies to a compartment, specified its ID.

Content filter

Cleaning rules of type USER_EVENT_CLEANING_RULE can be associated with a content filter :

Proprty	Type	Description
filter	string	Value of the filter
content_type	enum	Target of the filter. Only EVENT_NAME_FILTER allowed for now.

Samples

// KEEP User event cleaning rule with channel filter and activity type filter
{
    "id": "907",
    "action": "KEEP",
    "archived": false,
    "datamart_id": "1509",
    "channel_filter": "3844",
    "activity_type_filter": "SITE_VISIT",
    "life_duration": "P65D",
    "status": "DRAFT",
    "type": "USER_EVENT_CLEANING_RULE"
}

// DELETE User event cleaning rule without filters
{
    "id": "778",
    "action": "DELETE",
    "archived": false,
    "datamart_id": "1509",
    "channel_filter": null,
    "activity_type_filter": null,
    "life_duration": "P60D",
    "status": "LIVE",
    "type": "USER_EVENT_CLEANING_RULE"
}
// its associated content filter


// DELETE User profile cleaning rule with compartment filter
{
    "id": "905",
    "action": "DELETE",
    "archived": false,
    "datamart_id": "1509",
    "compartment_filter": "1606",
    "life_duration": "P365D",
    "status": "ARCHIVED",
    "type": "USER_PROFILE_CLEANING_RULE"
}

List cleaning rules in a datamart

GET https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart

Query Parameters

Name	Type	Description
type	string	USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE to filter on a specific type of cleaning rule

200

{
    "status": "ok",
    "data": [
        {
            "id": "904",
            "action": "DELETE",
            "archived": false,
            "datamart_id": "1509",
            "compartment_filter": null,
            "life_duration": "P300D",
            "status": "LIVE",
            "type": "USER_PROFILE_CLEANING_RULE"
        },
        {
            "id": "778",
            "action": "DELETE",
            "archived": false,
            "datamart_id": "1509",
            "channel_filter": null,
            "activity_type_filter": null,
            "life_duration": "P60D",
            "status": "LIVE",
            "type": "USER_EVENT_CLEANING_RULE"
        },
        {
            "id": "902",
            "action": "KEEP",
            "archived": false,
            "datamart_id": "1509",
            "channel_filter": "3844",
            "activity_type_filter": null,
            "life_duration": "P250D",
            "status": "ARCHIVED",
            "type": "USER_EVENT_CLEANING_RULE"
        }
    ],
    "count": 3,
    "total": 3,
    "first_result": 0,
    "max_result": 50,
    "max_results": 50
}

Get a cleaning rule

GET https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the cleaning rule

200

{
    "status": "ok",
    "data": {
        "id": "778",
        "action": "DELETE",
        "archived": false,
        "datamart_id": "1509",
        "channel_filter": null,
        "activity_type_filter": null,
        "life_duration": "P60D",
        "status": "LIVE",
        "type": "USER_EVENT_CLEANING_RULE"
    }
}

Get the content filter for a rule

GET https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter

Only applies to event-based cleaning rules

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule

200

{
    "status":"ok",
    "data":{
        "filter":"$transaction_confirmed",
        "content_type":"EVENT_NAME_FILTER"
    }
}

404 If the rule doesn't have a content filter

{
    "status": "error",
    "error": "user_event_cleaning_rule 778 has no content_filter",
    "error_id": "2e5d3338-34ec-4abd-aaff-fb370958ff2a"
}

Create a cleaning rule

POST https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart

Request Body

Name	Type	Description
activity_type_filter	string	Only for event-based rules. SITE_VISIT, APP_VISIT, TOUCH, DISPLAY_ADor EMAIL .
channel_filter	string	Only for event-based rules. The ID of the channel on which the rule applies
compartment_id	string	Only for profile-based rules. The ID of the compartment on which the rule applies.
life_duration	string	ISO 8601 duration. Example "P160D"
status	string	DRAFT
type	string	USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE
action	string	KEEP or DELETE. Only DELETE for event-based rules

200

{
    "status": "ok",
    "data": {
        "id": "917",
        "action": "KEEP",
        "archived": false,
        "datamart_id": "1509",
        "channel_filter": "3844",
        "activity_type_filter": "SITE_VISIT",
        "life_duration": "P70D",
        "status": "LIVE",
        "type": "USER_EVENT_CLEANING_RULE"
    }
}

Update a cleaning rule and/or change its status

PUT https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId

Path Parameters

Name	Type	Description
ruleId	string	The ID of the rule to update
datamartId	string	The ID of the datamart

Request Body

Name	Type	Description
archived	boolean	true to make the rule disappear from the UI. Can only be put to true if the status is already ARCHIVED. Use false otherwise.
id	string	Required to set the status to ARCHIVED. The ID of the rule to update.
activity_type_filter	string	Only for event-based rules. SITE_VISIT, APP_VISIT, TOUCH, DISPLAY_ADor EMAIL . Can't be changed if the status is not DRAFT.
channel_filter	string	Only for event-based rules. The ID of the channel on which the rule applies. Can't be changed if the status is not DRAFT.
compartment_id	string	Only for profile-based rules. The ID of the compartment on which the rule applies. Can't be changed if the status is not DRAFT.
life_duration	string	ISO 8601 duration. Example "P160D". Can't be changed if the status is not DRAFT.
status	string	DRAFT to keep it as a draft. LIVE to publish it from draft. ARCHIVED to archive it from LIVE
type	string	USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE based on the existing rule type. Can't be changed.
action	string	KEEP or DELETE. Only DELETE for event-based rules Can't be changed if the status is not DRAFT.

200

{
    "status": "ok",
    "data": {
        "id": "917",
        "action": "KEEP",
        "archived": false,
        "datamart_id": "1509",
        "channel_filter": "3844",
        "activity_type_filter": "SITE_VISIT",
        "life_duration": "P70D",
        "status": "LIVE",
        "type": "USER_EVENT_CLEANING_RULE"
    }
}

You'll have an error if doing any update that is prohibited, like changing a property while the status is not DRAFT or

Delete a DRAFT rule

DELETE https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule to delete

200

{
    "status": "ok"
}

Add or update a content filter for a rule

POST https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule

Request Body

Name	Type	Description
content-type	string	EVENT_NAME_FILTER
filter	string	Name of the events for which the rule will apply

200

{
    "status": "ok",
    "data": {
        "filter": "$transaction_confirmed",
        "content_type": "EVENT_NAME_FILTER"
    }
}

Remove a content filter for a rule

DELETE https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule

200

{
    "status": "ok",
    "data": {
        "filter": "$transaction_confirmed",
        "content_type": "EVENT_NAME_FILTER"
    }
}