Cleaning rules

Cleaning rules define how each datamart manages the deletion of user data. They can apply to two different user objects: profiles and events.

When an object enters the datamart, mediarithmics computes the expiration timestamp of this object based on the different cleaning rules that can be applied. This expiration timestamp is part of the object properties, and can't be modified. That means cleaning rules are non-retroactive: changing the rules will only have an effect on future incoming events and profiles.

A user profile or user event enters the datamart
Which cleaning rules apply to it?
Based on the cleaning rules and the priority between them, compute the expiration timestamp of the object
Store the expiration timestamp in the object

An event or profile expires if it hasn't been modified before the expiration timestamp. In details:

For profile: last modification timestamp field ($last_modified_ts for profile) is checked against expiration timestamp field.
For event: since an event cannot be updated, creation timestamp field ($ts) is checked against expiration timestamp field.

When an object expires, it does not return in queries or in the interface. It still exists in the passive database, until it is removed by a scheduled operation.

All cleaning rules have a status (DRAFT, LIVE or ARCHIVED), an action (KEEP or DELETE), a duration and a set of filters.

KEEP action means the associated object should stay in the datamart for the duration.DELETE action means the associated object should be removed from the datamart after the duration has expired.

There are two types of cleaning rules:

Event-based cleaning rules apply to events. They can apply to all events, or only to different channels, activity types or event names.
Profile-based cleaning rules apply to profiles. They can apply to all profiles or only to a compartment.

When a user point does not have any activity, nor profile, nor USER_LIST segment, it is removed from the platform. By cleaning activities and profiles, cleaning rules will clean your user points.

Cleaning rules can be easily edited in the navigator, in your datamart settings

Rules priority

KEEP cleaning rules have the priority on DELETE rules.

Only LIVE cleaning rules are taken into account when computing the expiration timestamp of an object.

Example 1

The following cleaning rules apply to my incoming event : - KEEP for 60 days - KEEP for 180 days - DELETE after 150 days

The event will be kept for 180 days then deleted.

Example 2

The following cleaning rules apply to my incoming event : - KEEP for 60 days - DELETE after 150 days

The event will be deleted after 150 days.

Example 3

The following cleaning rules apply to my incoming profile : - DELETE after 10 days - DELETE after 150 days

The profile will be deleted after 10 days.

Rules lifecycle

Cleaning rules are created with the DRAFT status. They can't be created LIVE or ARCHIVED. While DRAFT you can change the different properties of the rule, except its type. You then change the rule status to LIVE by doing an edit request. Once live, a rule can't be edited anymore and only the status can be changed to ARCHIVED.

At any moment, there should always be a LIVE event based rule with a DELETE action. You'll need to create at least an other rule to be able to archive this one.

Once ARCHIVED, rules can't be changed anymore, nor deleted.

You can only delete DRAFT cleaning rules. Once LIVE, they'll have to be ARCHIVED

Object reference

A cleaning rule has the following set of properties.

Property

Type

Description

string

The cleaning rule identifier

action

enum

KEEP or DELETE.

KEEP means the cleaning rule will force data to be kept for the duration, while DELETE means the data should be deleted after the duration.

Only DELETE is allowed for cleaning rules of type USER_PROFILE_CLEANING_RULE

archived

boolean

Used to make a rule disappear from the UI. You can only set it to true if the status is already ARCHIVED

status

enum

ARCHIVED, DRAFT or LIVE.

A cleaning rule can only go from the DRAFT status to LIVE, and can only go from LIVE to ARCHIVED.

datamart_id

string

The ID of the associated datamart

type

enum

USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE

life_duration

ISO 8601 duration

The duration associated with the cleaning rule

channel_filter

string

USER_EVENT_CLEANING_RULE only.

Optional. If a cleaning rule only applies to a channel, specifies its ID.

channel_filter must be compatible activity_type_filter if set at the same time (for example you can't target an app channel if you set the activity type to DISPLAY_AD)

activity_type_filter

enum

USER_EVENT_CLEANING_RULE only.

Optional. If a cleaning rule only applies to a specific activity type, specifies its ID.

SITE_VISIT, APP_VISIT, TOUCH, DISPLAY_ADor EMAIL .

channel_filter must be compatible activity_type_filter if set at the same time (for example you can't target an app channel if you set the activity type to DISPLAY_AD)

compartment_filter

string

USER_PROFILE_CLEANING_RULE only.

Optional. If a cleaning rule only applies to a compartment, specified its ID.

Content filter

Cleaning rules of type USER_EVENT_CLEANING_RULE can be associated with a content filter :

Proprty

Type

Description

filter

string

Value of the filter

content_type

enum

Target of the filter. Only EVENT_NAME_FILTER allowed for now.

Samples

// KEEP User event cleaning rule with channel filter and activity type filter
{
    "id": "907",
    "action": "KEEP",
    "archived": false,
    "datamart_id": "1509",
    "channel_filter": "3844",
    "activity_type_filter": "SITE_VISIT",
    "life_duration": "P65D",
    "status": "DRAFT",
    "type": "USER_EVENT_CLEANING_RULE"
}

// DELETE User event cleaning rule without filters
{
    "id": "778",
    "action": "DELETE",
    "archived": false,
    "datamart_id": "1509",
    "channel_filter": null,
    "activity_type_filter": null,
    "life_duration": "P60D",
    "status": "LIVE",
    "type": "USER_EVENT_CLEANING_RULE"
}
// its associated content filter


// DELETE User profile cleaning rule with compartment filter
{
    "id": "905",
    "action": "DELETE",
    "archived": false,
    "datamart_id": "1509",
    "compartment_filter": "1606",
    "life_duration": "P365D",
    "status": "ARCHIVED",
    "type": "USER_PROFILE_CLEANING_RULE"
}

List cleaning rules in a datamart

GET https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart

Name

Type

Description

datamartId

string

The ID of the datamart

Query Parameters

Name Type Description

Name	Type	Description
type	string	`USER_EVENT_CLEANING_RULE` or `USER_PROFILE_CLEANING_RULE` to filter on a specific type of cleaning rule

type

string

USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE to filter on a specific type of cleaning rule

{
    "status": "ok",
    "data": [
        {
            "id": "904",
            "action": "DELETE",
            "archived": false,
            "datamart_id": "1509",
            "compartment_filter": null,
            "life_duration": "P300D",
            "status": "LIVE",
            "type": "USER_PROFILE_CLEANING_RULE"
        },
        {
            "id": "778",
            "action": "DELETE",
            "archived": false,
            "datamart_id": "1509",
            "channel_filter": null,
            "activity_type_filter": null,
            "life_duration": "P60D",
            "status": "LIVE",
            "type": "USER_EVENT_CLEANING_RULE"
        },
        {
            "id": "902",
            "action": "KEEP",
            "archived": false,
            "datamart_id": "1509",
            "channel_filter": "3844",
            "activity_type_filter": null,
            "life_duration": "P250D",
            "status": "ARCHIVED",
            "type": "USER_EVENT_CLEANING_RULE"
        }
    ],
    "count": 3,
    "total": 3,
    "first_result": 0,
    "max_result": 50,
    "max_results": 50
}

Get a cleaning rule

GET https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the cleaning rule

Name

Type

Description

datamartId

string

The ID of the datamart

ruleId

string

The ID of the cleaning rule

{
    "status": "ok",
    "data": {
        "id": "778",
        "action": "DELETE",
        "archived": false,
        "datamart_id": "1509",
        "channel_filter": null,
        "activity_type_filter": null,
        "life_duration": "P60D",
        "status": "LIVE",
        "type": "USER_EVENT_CLEANING_RULE"
    }
}

Get the content filter for a rule

GET https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter

Only applies to event-based cleaning rules

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule

Name

Type

Description

datamartId

string

The ID of the datamart

ruleId

string

The ID of the rule

{
    "status":"ok",
    "data":{
        "filter":"$transaction_confirmed",
        "content_type":"EVENT_NAME_FILTER"
    }
}

{
    "status": "error",
    "error": "user_event_cleaning_rule 778 has no content_filter",
    "error_id": "2e5d3338-34ec-4abd-aaff-fb370958ff2a"
}

Create a cleaning rule

POST https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart

Name

Type

Description

datamartId

string

The ID of the datamart

Request Body

Name Type Description

Name	Type	Description
activity_type_filter	string	Only for event-based rules. `SITE_VISIT`, `APP_VISIT`, `TOUCH`, `DISPLAY_AD`or `EMAIL` .
channel_filter	string	Only for event-based rules. The ID of the channel on which the rule applies
compartment_id	string	Only for profile-based rules. The ID of the compartment on which the rule applies.
life_duration	string	ISO 8601 duration. Example "P160D"
status	string	`DRAFT`
type	string	`USER_EVENT_CLEANING_RULE` or `USER_PROFILE_CLEANING_RULE`
action	string	`KEEP` or `DELETE`. Only `DELETE` for event-based rules

activity_type_filter

string

Only for event-based rules. SITE_VISIT, APP_VISIT, TOUCH, DISPLAY_ADor EMAIL .

channel_filter

string

Only for event-based rules. The ID of the channel on which the rule applies

compartment_id

string

Only for profile-based rules. The ID of the compartment on which the rule applies.

life_duration

string

ISO 8601 duration. Example "P160D"

status

string

DRAFT

type

string

USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE

action

string

KEEP or DELETE. Only DELETE for event-based rules

{
    "status": "ok",
    "data": {
        "id": "917",
        "action": "KEEP",
        "archived": false,
        "datamart_id": "1509",
        "channel_filter": "3844",
        "activity_type_filter": "SITE_VISIT",
        "life_duration": "P70D",
        "status": "LIVE",
        "type": "USER_EVENT_CLEANING_RULE"
    }
}

Update a cleaning rule and/or change its status

PUT https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId

Path Parameters

Name	Type	Description
ruleId	string	The ID of the rule to update
datamartId	string	The ID of the datamart

Name

Type

Description

ruleId

string

The ID of the rule to update

datamartId

string

The ID of the datamart

Request Body

Name Type Description

Name	Type	Description
archived	boolean	`true` to make the rule disappear from the UI. Can only be put to `true` if the status is already `ARCHIVED`. Use `false` otherwise.
id	string	Required to set the status to `ARCHIVED`. The ID of the rule to update.
activity_type_filter	string	Only for event-based rules. `SITE_VISIT`, `APP_VISIT`, `TOUCH`, `DISPLAY_AD`or `EMAIL` . Can't be changed if the status is not `DRAFT`.
channel_filter	string	Only for event-based rules. The ID of the channel on which the rule applies. Can't be changed if the status is not `DRAFT`.
compartment_id	string	Only for profile-based rules. The ID of the compartment on which the rule applies. Can't be changed if the status is not `DRAFT`.
life_duration	string	ISO 8601 duration. Example "P160D". Can't be changed if the status is not `DRAFT`.
status	string	`DRAFT` to keep it as a draft. `LIVE` to publish it from draft. `ARCHIVED` to archive it from `LIVE`
type	string	`USER_EVENT_CLEANING_RULE` or `USER_PROFILE_CLEANING_RULE` based on the existing rule type. Can't be changed.
action	string	`KEEP` or `DELETE`. Only `DELETE` for event-based rules Can't be changed if the status is not `DRAFT`.

archived

boolean

true to make the rule disappear from the UI. Can only be put to true if the status is already ARCHIVED. Use false otherwise.

string

Required to set the status to ARCHIVED. The ID of the rule to update.

activity_type_filter

string

Only for event-based rules. SITE_VISIT, APP_VISIT, TOUCH, DISPLAY_ADor EMAIL . Can't be changed if the status is not DRAFT.

channel_filter

string

Only for event-based rules. The ID of the channel on which the rule applies. Can't be changed if the status is not DRAFT.

compartment_id

string

Only for profile-based rules. The ID of the compartment on which the rule applies. Can't be changed if the status is not DRAFT.

life_duration

string

ISO 8601 duration. Example "P160D". Can't be changed if the status is not DRAFT.

status

string

DRAFT to keep it as a draft. LIVE to publish it from draft. ARCHIVED to archive it from LIVE

type

string

USER_EVENT_CLEANING_RULE or USER_PROFILE_CLEANING_RULE based on the existing rule type. Can't be changed.

action

string

KEEP or DELETE. Only DELETE for event-based rules Can't be changed if the status is not DRAFT.

{
    "status": "ok",
    "data": {
        "id": "917",
        "action": "KEEP",
        "archived": false,
        "datamart_id": "1509",
        "channel_filter": "3844",
        "activity_type_filter": "SITE_VISIT",
        "life_duration": "P70D",
        "status": "LIVE",
        "type": "USER_EVENT_CLEANING_RULE"
    }
}

You'll have an error if doing any update that is prohibited, like changing a property while the status is not DRAFT or

Delete a DRAFT rule

DELETE https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule to delete

Name

Type

Description

datamartId

string

The ID of the datamart

ruleId

string

The ID of the rule to delete

{
    "status": "ok"
}

Add or update a content filter for a rule

POST https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule

Name

Type

Description

datamartId

string

The ID of the datamart

ruleId

string

The ID of the rule

Request Body

Name Type Description

Name	Type	Description
content-type	string	`EVENT_NAME_FILTER`
filter	string	Name of the events for which the rule will apply

content-type

string

EVENT_NAME_FILTER

filter

string

Name of the events for which the rule will apply

{
    "status": "ok",
    "data": {
        "filter": "$transaction_confirmed",
        "content_type": "EVENT_NAME_FILTER"
    }
}

Remove a content filter for a rule

DELETE https://api.mediarithmics.com/v1/datamarts/:datamartId/cleaning_rules/:ruleId/content_filter

Path Parameters

Name	Type	Description
datamartId	string	The ID of the datamart
ruleId	string	The ID of the rule

Name

Type

Description

datamartId

string

The ID of the datamart

ruleId

string

The ID of the rule

{
    "status": "ok",
    "data": {
        "filter": "$transaction_confirmed",
        "content_type": "EVENT_NAME_FILTER"
    }
}

PreviousUser choices NextExercise of user rights

Last updated 1 year ago