Using Insight data (developers' guide)

Less technical information is also available

This is a developers' guide to using our insight data; for a shorter, less technical introduction to insight data see our guide for non-developers.

Contents

What is insight data?
Dataset examples
What can I do with this data?
How is insight data stored?
Keys are used to refer to individual pieces of data
Data is stored against a contact or as account-scoped data
Restrictions on keys
JSON data representation
Restrictions on names
Restrictions and extensions on values
JSON stores data in schema-bound collections
The schema of a collection is defined 'on the fly'
Types are immutable
Objects are extendable
Bulk importing insight data
Plan ahead

What is insight data?

Insight data is storable information related to your customers’ purchasing histories and habits. It allows you to store collections of arbitrary data against your contacts or account via our API.

So they’re additional contact data fields, extending the contact data fields we already store?

Yes – but with the prime difference being these are transaction-based ones. Any additional keyed data you have for a contact can be stored as insight data, with very little restriction on the type of data it can be. As a broad rule, you can store anything that is serialisable to JSON.

Just as you can currently segment upon contact data fields such as gender, age and geography, insight data gives you the ability to segment upon types of items purchased, regularity of purchases and amount spent on purchases, for instance. As ever, the quality of your insight data will dictate how well you can segment address books and personalise your campaigns and offers.

Dataset examples

Here are some examples of datasets that could be stored as insight data:

  • For an auction site, a list of all bids a user has made and if they succeeded or not, including the date and time of the bid, the amount of the bid and the category and name of the item bid on
  • For a travel agent, a list of destinations a user has visited via bookings on the site, including the amount of bookings for the destination (solo, couple, family, etc.,), the amount spent and the type of accommodation
  • A list of a user’s likes and dislikes

The key point is that many different types of insight data can be stored for each contact, and structured in a way of your choosing.

What can I do with this data?

Once insight data is stored against your contacts, you can use the website to write queries to segment your contacts against this data and create new address books. This will enable you to send targeted, personalised campaigns based upon your contacts’ transactional history and habits. Using the above examples, you will be able to run segmentations based on bids made, countries visited and likes and dislikes.

For more on segmenting insight data, read our article on insight data segmentation.

How is insight data stored?

This section provides information and guidelines on storing your insight data. It does get quite technical (if it hasn’t enough already!), so if you are reading this and are not a developer then you will probably want a developer to read it instead, or at least help you with understanding it.

Keys are used to refer to individual pieces of data

At its heart, insight data is a key-value pair storage mechanism; you store a piece of data against a contact, using a key to refer to it. If you need to update or delete a piece of data at a later date, you use the key as a reference to retrieve the piece of data and alter it.

Restrictions on keys

Keys must:

  • Be unique (it is up to the user of the API to ensure uniqueness)
  • Contain no more than 255 characters

A key can contain any character you like.

Data is stored against a contact or as account-scoped data

A contact identifier (email address or ID) is submitted with each insight data document uploaded. If the identifier used is "account" rather than an email address, then the record will be stored as account-scoped insight data (AccountInsight data). This data can be used for advanced personalisation and viewed via the account menu in application.

JSON data representation

All data to be stored must be serialisable as JSON. This means complex object structures can be used to represent data. For example, you could represent the contents of a user’s shopping basket like this:

{
"basketId": 4858,
"datePurchased": "2013-03-19T17:22:54.8042",
"productPurchased": [{
"name": "a dvd",
"cost": 10.99,
"isOnSale": false
}, {
"name": "a book",
"cost": 2.99,
"isOnSale": true
}
]
}

There are a few restrictions and extensions to the JSON format as outlined below.

Restrictions on collection names

Valid collection names can only contain the following characters:

  • Alphanumeric (a-z, A-Z, 0-9)
  • Hyphen ("-")
  • Underscore ("_")

Furthermore, collection names can't:

  • Begin with a number
  • Exceed 255 characters in length

Restrictions and extensions on values

  • The null value is not allowed
  • String values are restricted to 255 characters in length.
  • Date time values are also supported via a subset of the ISO 8601 standard (e.g. 2013-03-19T17:22:55.804Z)
  • Lists must be of the same type. Lists may only contain either Booleans, strings, numbers, objects, lists, etc.

See the JSON specification for all other value types.

A note on date time formatting

Not all ISO 8601 formats are supported (for example the string "2013" would not be treated as a date). The following are valid for this implementation.

   Complete date plus hours, minutes and seconds:
      YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00)
   Complete date plus hours, minutes, seconds and a decimal fraction of a second
      YYYY-MM-DDThh:mm:ss.sTZD (eg 1997-07-16T19:20:30.45+01:00)

where:

     YYYY = four-digit year
     MM   = two-digit month (01=January, etc.)
     DD   = two-digit day of month (01 through 31)
     hh   = two digits of hour (00 through 23) (am/pm NOT allowed)
     mm   = two digits of minute (00 through 59)
     ss   = two digits of second (00 through 59)
     s    = one or more digits representing a decimal fraction of a second
     TZD  = time zone designator (Z or +hh:mm or -hh:mm)

All times without a time zone designator will be treated as UTC.

JSON stores data in schema-bound collections

While you can store any JSON structure as insight data, it is not strictly schemaless. Insight data requires that you store similarly structured data in a collection. Each collection may have a different schema, allowing you to store a variety of different data against contacts.

The schema of a collection is defined ‘on the fly’

Schemas are defined implicitly upon uploading data via the API for the first time. Furthermore, schemas are extendable but not editable.

For example, let’s say you’ve just uploaded the following object into a collection you’ve created called “preferences”:

{
"likes": {
"animals": [
{
"animal": "dogs"
},
{
"animal":"cats"
}
],
"numbers": [
{
"number": 1,
},
{
"number": 2,
},
{
"number": 7
}
]
},
"dislikes": {
"animals": [
{
"animal": "rats"
}
],
"numbers": [13]

}
}

For all proceeding uploads, the preferences collection will expect an object with two properties, “likes” and “dislikes”, each of which refer to an object where each object has a property called “animals” which is a list of strings and “numbers” which is a list of numbers.  

Types are immutable

Once you’ve defined a named property, its type can’t be changed. For example, if you upload this JSON object to define your schema for a collection:

{
"id": "1",
"name": "Tom Bloggs"
}

You couldn’t then upload the following object:

{
"id": 2,
"name": {
"first": "john",
"second": "smith"
}
}

Why? Because the “id” property has been changed from a string to a number and the “name” property has been changed from a string to an object.

Objects are extendable

The only things that are mutable are objects because they are extendable. Properties can be added to an object at any time (however they are then bound by the above rule). For example, if this object was uploaded to define a collection’s schema:

{
"fullName": "John Bloggs"
}

Then uploading this object would extend the collection's schema to include the two new properties, “firstName” and “lastName”:

{
"fullName": "John Bloggs"
"firstName": "John",
"lastName": "Bloggs"
}

Bulk importing insight data

As standard, you will be adding your data on a single-transaction-at-a-time basis – for instance, at the point of online checkout.

However, to get you up and running, it is highly likely that you’ll want to make an initial bulk import of all your historical data. This can be done using the SOAP API methods ImportTransactionalData and GetTransactionalDataImportProgress, or the REST API methods Bulk add transactional data to contacts and Get transactional data import status.

You can perform multiple bulk imports at a time for an account. Typically we'd suggest five as a maximum, for optimal speed and performance.

Plan ahead

Given the nature of Insight data storage, you will want to spend some time thinking about how your Insight data needs to work for you. A little bit of planning will go a long way! What sort of data do you want to store? How might you want to extend this data as you move forward?

The point to remember is that you don’t want to become backed into a corner by initiating a schema that will prove constrictive. For example, you will probably want to avoid entering your product ID as an integer. This won’t give you very easily identifiable information and you won’t be able to change it once it is uploaded. To change it, you would need to start again with a new schema and re-upload all of your data to conform to it. This is definitely something to be avoided! 

Have more questions? Submit a request

Comments

  • Avatar

    All very well, but how do you tag the data to a Contact?  I have assumed that the "ContactIdentifier" field is intended to be the contact's email address.  If so, why not say so - if not then what is it and how do you tag the data to the contact?

  • Avatar

    'Schemas are defined implicitly upon uploading data via the API for the first time. Furthermore, schemas are extendable but not editable.

    For example, let’s say you’ve just uploaded the following object into a collection you’ve created called “preferences”:'

    How do you add collections? I get the bit about the schema of a collection being defined by the first upload but how do you add a collection in the first place.

    Not seeing anything in either the docs OR in the admin.

     

  • Avatar

    Just to bump the above Comments, the above documentation is not very clear, are there any plans to update these pages in the future?

  • Avatar

    Hi all,

    Thanks for all of your comments added here.

    The 'contact identifier' is indeed the contact's email address. Alternatively, it can also be the contact's ID number. This is required by our API's REST operations/SOAP methods when adding or replacing pieces of transactional data connected to a contact.

    Collections are added when adding your first piece or pieces of transactional data. For instance, if you want to create a collection called 'Purchases', you'd use the API to post to:

    https://apiconnector.com/v2/contacts/transactional-data/Purchases 

    (this is assuming you're using REST, although there is of course a SOAP equivalent called AddTransactionalData)

    In your request payload, you'd include:

    • A unique identifier for the document, called the key (e.g. "P00000001") 
    • The email address (or ID) of the contact you want to add it to 
    • The JSON for the purchase that you want to add to the contact

    In which case, it would look something like this (all in JSON):

    {

    "key": "P00000001",

    "contactIdentifier": "john.smith@thatdomain.com",

    "json": "{\r\n \"PurchaseDate\": \"2015-10-05T15:03:21Z\",\r\n \"TotalExTax\": 56.0000000000,\r\n \"TotalIncTax\": 67.2000000000,\r\n \"Product\": [\r\n {\r\n \"Name\": \"Armani Jeans beanie hat\",\r\n \"Brand\": \"Armani Jeans\",\r\n \"Department\": \"Menswear\",\r\n \"Category\": \"Hat\",\r\n \"PriceExTax\": 56.0000000000,\r\n \"ProductID\": \"24938\"\r\n }\r\n ],\r\n \"SalesChannel\": \"In store\",\r\n \"SalesSubChannel\": \"Manchester\"\r\n}"

    }

     

    The above page will be updated to ensure all of this information is provided with greater clarity.

Powered by Zendesk