IoT Analytics over time with Keen and Scriptr

This is a guest post written by Ed Borden, Evangelist at Scriptr.io, VP Ads at Soofa.

A large part of Internet of Things applications typically involves management operations; you want to know what your assets are doing right now and if you need to react in some way.

I think about that ‘realtime’ domain as an area with a particular set of tools, challenges, and thought processes, and the ‘historical’ domain as another. In the historical domain of IoT, I think about what the high-value information will be in the form of questions, like:

• How long was this parking space unoccupied last week?
• Which truck in my fleet was in service the longest?
• How long was this machine in power-saving mode?
• What are the 5 best and worst performers in this group?

For these types of questions, Keen is my go-to. However, architecting the answers to these questions takes a little bit of shifting in your architecture design.

You might typically push events to Keen as they are happening, but if you are only pushing data based on the changes in state of a thing (as is the common model for asset tracking/management-type scenarios), you won’t have enough information to ask these types of questions since you need to know how long the thing has been in each state. So, when an event comes in:

1. you need to cache the timestamp and state the thing is going into, and
2. create an event based on the previous cached state that was just transitioned out of, which must include the “duration” of that state.

Once this is done, Keen really shines at the rest! You can simply do a “sum” query on the durations of events, filtering by groups of devices and timeframes.

The below snippet using Keen IO will tell you how long a parking space was occupied:

var timeOccupied = new Keen.Query("sum", {
   event_collection: "deviceUpdates",
   target_property: "duration",
   timeframe: "this_7_days",
   filters: [ 
      { operator: "eq",
        property_name: "hardwareId",
        property_value: hardwareId
      },        
      { 
        operator: "eq",
        property_name: "deviceState",
        property_value: "occupied"
      }
   ]
});

If you want to sum all of the parking spots on the street, give each event a “streetId” and filter by that instead of “hardwareId”.

The below snippet will tell you how many parking spaces were occupied longer than an hour (because street parking is limited to one hour and you want to know where the most violations are occurring):

var violationsOccurred = new Keen.Query("count", {
   event_collection: "deviceUpdates",
   target_property: "duration",
   timeframe: "this_7_days",
   filters: [ 
      { operator: "gt",
        property_name: "duration",
        property_value: 60
      },
      { 
        operator: "eq",
        property_name: "deviceState",
        property_value: "occupied"
      }
   ]
});

I could do this all day! That’s because once you have this sort of infrastructure in place, the sky really is the limit on the types of high-value information you can extract. And you did this all without managing any database infrastructure or API surface of your own?!

So, how do we implement the complete system? Here Keen can use a little help from an IoT service called Scriptr.io. Scriptr.io has a web-based IDE which lets you write some Javascript, hit “Save”, and that code instantly becomes a hosted webservice with a URL. Using Scriptr.io’s fast local storage module and Keen connector, we can do some caching and light processing on that ‘in-flight’ datastream in a simple and expressive way that ALSO requires no devops/infrastructure! A match made in #NoDevOps heaven. It would look like this:

//Any POST body to your Scriptr script's URL can be accessed 
//with the 'request' object  
var eventData = JSON.parse(request.rawBody);

//The 'storage' object is a key/value store which we access with 
//the current device's ID
var lastEventData = storage.local[eventData.hardwareId];

//In this example, we'll assume these are epoch times, otherwise we'd convert
var eventDuration = eventData.timestamp - lastEventData.timestamp; 

//Add the duration to the last event data object which we'll push to Keen
lastEventData.eventDuration = eventDuration; 

//This is the Scriptr.io -> Keen.io connector
var keenModule = require('../../modules/keenio/keenioclient'); 
var keen = new keenModule.Keenio("my_Keen_credentials");

//Next, record the Keen event
keen.recordEvent({
  collection: "deviceUpdates",
  data: lastEventData
});

//Cache the current event by the device's ID
storage.local[eventData.hardwareId] = eventData;

Below, you can see this in the Scriptr IDE:

There you go — Big IoT Data! You can learn more about the Scriptr.io platform here or the Scriptr -> Keen connector here.