Platform Stability AMA Recap

Peter, Brad, and I hosted an AMA about recent platform instability on February 27th. It was a chance for our community to ask about what’s been up with the API lately and how we’re addressing it. Here are a few highlights from the conversation, fit and trimmed for readability. Editing notes are in brackets. The AMA itself contains the full, unedited versions.

Nima: We’re also an analytics company dealing with billions of events each month, and have 3 layers of redundancy that last up to 60 days in our system – I was surprised to see that you only keep your queues for a week.

Josh: Most data is off of Kafka and persisted to Cassandra in multiple replicas, multiple data centers, within 5 seconds of receipt. When the write path gets backed up that delay can turn into hours, but a week isn’t something we anticipated because we hadn’t considered a failover scenario where data in the previous DC still needed to be ingested a week later. We haven’t decided exactly what we want to do about this. Storing Kafka logs longer is one option but that’s a difference in degree, not kind. My hunch right now is that we’ll build in warnings into the monitoring stack for this a la “Hey dummy, this data is going to be reclaimed and isn’t permanently persisted yet!”

Nima: We’re having discussions on building even more of our analytics on top of Keen (client events -> engineering dash-boarding) – but it’s hard to make that decision due to the recent events; especially when no real plan has been shared with the customers around what you’ll do to address these issues.

Josh: Here are some of the plans we have in store:

  • Index optimization, which will improve query performance and reduce Cassandra I/O generally. Expect to see a document about this on the dev group next week. We’re trying to share more of this stuff, earlier, for transparency but also feedback (we know many of y’all are pretty smart about this stuff)
  • Cassandra and Astyanax / Cassandra driver work. There are a few optimizations we’ve identified that will reduce the % of Cassandra reads & writes that have to be coordinated to nodes that don’t own the row key in question. Big drop in I/O and increase in performance to be gained.
  • Hardware. Kafka, Zookeeper, Storm, and Cassandra are continuously getting new metal.

Nima: So I guess the question is: what are the plans? How will you scale up faster than your customers are scaling?

Josh: The question of how we scale up faster than customers is favorite of mine. Fundamentally I think it comes down to people. Another kind of “monitoring” we have in place is this – “Is Keen a great place to work for distributed systems engineers?” If the answer is “Yes!” than we have the opportunity to address performance and scale challenges with the brightest minds out there. I think that gives us as good a shot as any to keep up with your (very impressive!) growth.

Jody: I’ve heard to write good software, you have to write it twice. What are some of the major things you’d do differently if you had to rebuild keen from the ground up?

Josh: I have heard that too, and I’d say at LEAST twice. Here are a few things I would have done differently [geared specifically toward our distributed Java stack]:

  • Not use Storm’s DRPC component to handle queries. It wasn’t as battle-tested as I’d hoped and I missed that. We spent a lot of time chasing down issues with it, and it’s partially responsible for queries that seem to run forever but not return anything. Thankfully due to the work of Cory et. al. we have a new service Pine that to replace DRPC and all queries are running on it now. It has queueing and more intelligent rate limiting and customer isolation.
  • Greater isolation of write and read code & operational paths. The semantics of failure are very different in writes vs. reads (you can re-try failed reads, but not un-captured writes). To be honest it took us a few bruises in 2013 & 2014 to really drill that into our design philosophy. Most of our stack is pretty well separated today, but we could have saved some headaches and customer impact by doing it earlier.
  • More monitoring, more visibility, earlier. Anything to ease the operational burden across our team during incidents and prevent stress and burnout.

Steve: I would be interested in your dev plans over the next 3-6 months especially on the query API.

Peter: Over the next 3-6 months our dev plans revolve around performance, consistency, and scalability. We intend that these architectural changes set us up for future feature expansion. Doors we’d like to open include custom secondary indices, greater filter complexity, chained queries, and increased ability to do data transformation. I’m happy to share more of my excitement around what the future holds if you’re curious :)

Rob: We’ve been using Keen for a while now and I recommend you guys to everyone I know. It looks like you’re having some success, which is awesome, but with that, issues scaling. How do you intend to improve performance even while you scale?

Brad: Thanks so much for your support Rob. There are a few things that we are doing to be able to improve performance as we scale. We are looking at making our systems more efficient, further down the road looking at new capabilities in terms of how we service queries and writes as the come into the system. We are currently on the second major iteration of our architecture which is a result of looking at how we can better serve our customers.

Nariman: Any updates on the read inconsistency issue? We first reported this in January and continue to see it in production as of yesterday. The problem hasn’t changed since it was first reported, it includes null timestamp and other null properties.

Brad: We have a number of areas which require repair and I have made it through about a quarter of the data as we ran into a few operational issues around doing the repairs. That said, we are now running full steam ahead on the repairs and monitoring them closely and making sure we don’t run them so fast they cause other issues in our infrastructure. This is my top priority and I have all the support I need to get this done.

Marios: You mention that you use Cassandra, are there scenarios where Cassandra’s conflict resolution may result in customer data loss/corruption?

Peter: In the past we ran into issues with Cassandra’s concept of Last Write Wins (“doomstones” and clock drift). Currently, the majority of our consistency issues are due to unforeseen edge cases in node failure scenarios.

Lee: I have been thinking about querying Keen overnight for consistency and replaying (when possible) if this check fails. Do you have a recommended procedure for replaying events over a questionable period? Thoughts on this?

Josh: [the first part of the answer goes into how, this is the 2nd part] Would I recommend it? It depends. If individual events have a very high impact (e.g. financial / you’re using them to bill customers) than if it were me I’d keep my own rolling record of events over some period of time. The Internet and natural disasters and the entropy of an expanding universe can never be ruled out. Most compliance directives & regulations mandate this anyway.

That said, one of our design goals for Keen is that in the general case you shouldn’t have to do that. The amount of write downtime Keen had last year can be stated in single-digit hours. We hate that we have a period with 1-2% inconsistent data, believe me, but mobile apps suffer connectivity issues, people run AdBlock on the web, batteries in sensors unexpectedly die – single-digit error percentages creep into the analytics equation way before data is sent to Keen. The question you have to ponder is a cost/benefit – the cost of building/maintaining verification vs. the cost to you / your business when more error creeps in than usual (because Keen or something else fails in an unexpected way).

There’s no one right answer. It really depends on what you’re tracking, your cost & time constraints, and in some sense your belief that the resiliency of the Internet and downstream services like Keen will get more reliable over time. I believe they (and we) will. Part of why I wanted to have this AMA, and more in the future, is to share the transparency I have into what’s going on at Keen so you have more information with which to form your own beliefs.

That’s it for the quick summaries. You can read all ~35 posts in full on the AMA itself.

I want to pre-emptively acknolwedge that we didn’t get into as much of the nitty gritty tech stuff as I had hoped. There is a still a technical postmortem on the way and I’m happy to use that as a starter for another conversation on the developer group.

A special thanks to everyone who dropped by to chat with us. We learned a lot and your participation was really inspiring.

Josh Dzielak

In-House Open Sorcerer

Net Neutrality and Startups: Talking with the FCC and Congress about an Open Internet

Yesterday marked a huge moment in the fight for net neutrality and an open internet. After years of contention and debate, the FCC voted to ensure that the internet remains an open utility uncontrolled by corporate interests. If you’re anything like us, you’re probably breathing a huge sigh of relief. 

We’re also super proud, though, that a member of the Keen team had the opportunity to speak to members of the FCC and Congress on the subject of net neutrality on the eve of this critical vote.

Two weeks ago, our very own Dan Kador joined a handful of representatives from other startups in a fly-in to Washington D.C. to offer up some perspective on how the policy could affect the startup industry. Here’s Dan with more info on exactly what went down:

How and why did you get brought into this whole thing? The FCC invited a bunch of startups to discuss net neutrality, right? For what reason? What other startups were invited along with you?

Caroline [part of our Advocacy team] sent us an email from a group called Engine. They were looking to send startup folks who are interested in net neutrality to Washington D.C. to speak with members of the FCC, House, and Senate. I connected with them, and they were eager to get a group together before the FCC announced its decision.

Washington is used to hearing from the big guys. Comcast, Verizon, even Netflix – they’re the ones spending garbage bags full of money on this. They don’t get to hear from young companies like ours, and they’re eager to hear our perspectives.

Alongside Keen, we had representation from Vimeo, Etsy, Foursquare, Union Square Ventures, Bigger Markets, Capitol Bells, and Spend Consciously.

What was the itinerary like? Where did you go? Who did you speak to? What was the format? What did they ask you? 

They call it a “fly-in”. We take one day and try to have as many meetings as possible. I flew into D.C. Wednesday night. We met as a group Thursday morning at the Mandarin Oriental, then walked over to the FCC. Our first meeting was with FCC Commissioner Rosenworcel and her staff. We introduced ourselves and then discussed why net neutrality was important to us. Rosenworcel is an ally, so we spent much of the meeting talking about how we could best use our time in DC with people who weren’t as aligned with us.

Then we met with members of FCC Chairman Wheeler’s staff. This meeting was mostly spent on digging into some of the specific details of what the FCC might rule on. Things like interconnection, paid prioritization, and zero rating. Our goal was to have the FCC write rules that were as simple and as clear as possible.

After that, we had a series of meetings with staff of various senators and representatives. In these meetings, we had a few goals. The biggest was to impress upon them that we wanted them to support the FCC’s decision. One big risk is that Congress might wade into this debate and try to legislate. Our belief is that we can get much stronger and better protections if the FCC is left to its own devices. Any legislation that passes the Republican-controlled Congress is going to be worse for net neutrality than what the FCC has now passed.

What were the points that you made in these discussions? Basically, why do you think net neutrality is important – both as an average person and as someone at a startup?

There were a couple important points we were trying to make.

First, that net neutrality is essential for entrepreneurship, especially in tech. Without net neutrality, large incumbents could easily pay to enforce their monopoly over a particular industry. Imagine if Comcast could charge its internet subscribers extra for access to the next YouTube but give access to Hulu for free? It would stifle innovation.

We also wanted to make the point that startups need simple and understandable rules for how to bring complaints. If I have a reasonable belief that I’m being made to pay more for my service’s traffic, I need a quick and easy way to bring that complaint to the FCC and have action taken quickly. Otherwise, there would be a strong stifling effect on innovation. Imagine if the process involved spending millions of dollars on lawyers and several years? No startup could grow under those circumstances.

As an average person, this is really important. One of the main reasons we’ve seen so much innovation over the last 30 years is because of the internet. If it suddenly became an uneven playground, we’d see a lot less growth. I don’t want a small group of ISPs deciding what technology wins and what technology dies. I want consumers to decide. We get that with net neutrality. 

Last, but definitely not least, I think high-speed access to the internet is a fundamental human right at this point and the FCC’s ruling helps protect that.


That’s Dan in the middle, with the bowtie!

Anyway, we don’t want to say that Keen helped definitely save the Internet or anything, but we’re super proud that we had the chance to speak on behalf of startups like us. Great job, Dan!

Alexa Meyer

Brand and behavioral design. Learner + Activator. Cheese consumer.

How I Went From 60,000 Readers to 1 in One Year

When I started writing publicly on this blog in 2012, it opened it up a whole new world to me. One of my posts was published in an O’Reilly book. Another became part of the entrepreneurship curriculum at Harvard Business School, where I was invited to guest lecture. Several of my posts got upwards of 50k views. Suddenly, people I met in San Francisco already “knew” me from the blog. Folks wrote me personal emails asking for career advice. Werner Vogels, the CTO of amazon, retweeted one of my pieces to his 65,000 followers. It was incredibly rewarding.

Then… something changed.

When our company scaled from 8 people to 15, I started a draft about how hard that was, and what worked, and what didn’t. After several weeks, half of those ideas were already being disproven. We were going so fast. Defeated, I abandoned the draft and stopped blogging completely. That was a year ago.

But, looking back, it wasn’t my writing that was blocked, it was the sharing that I’d stopped doing.

In fact, after a 10+ years hiatus, I’d started writing letters again, to my now-pregnant childhood friend, back home in Illinois. Writing to her is as effortless as talking.

I told my friend about all the biggest things that were going in my life, which to be honest were 99% company-related. I told her about the teammate we knew we needed to let go, and how we were too chicken to do it. I told her about raising our Series A funding, and my mixed feelings about what that meant to me and my husband, and our company. I confessed about how surprisingly intimate it can be to work closely with people you trust & admire, and how confusing those feelings can be sometimes.

When she told me about how she was finally getting over morning sickness, I told her I couldn’t think about having kids anymore, because I was completely emotionally maxed out. By that point the company was 30 people and I felt personally responsible for all of them. That letter is where I finally admitted to myself how ridiculously I’d over-extended myself. It was such a relief.

When I look back, I’ve learned so much over the last year, and have a lot of regrets about not publishing any of it.

The truth of the matter is, I am fucking terrified of sharing any of this type of stuff publicly. It’s a mix of Imposter Syndrome and outright Fear. 

Here are some examples of thoughts I have:

  • Why would anyone care about what you have to say? Your story isn’t special. Keep your ego in check. 
  • Your writing is mediocre at best. You just got lucky in the past because your friends up-voted your posts on Hacker News.
  • Our company has a brand now. That some people actually know about. Don’t screw it up. 
  • The only reason people liked your posts was because you are a female and they were trying to be nice.
  • People already think the only reason you have this job is because your husband is the CEO. Blabbing about stuff you don’t really know that much about on the internet is not going to help your case.
  • If you write about your company, your friends and family back home are going to think you’re bragging about your success. 

Taken one by one, these doubts are pretty easy to argue against, but the combination of them can feel like quicksand. With practice, we can all learn to quiet these voices, but it takes hard work.

Then there’s the straight-up Actual Risks category of anxieties. This is a special category that disproportionately affects women & minorities in the tech community. It’s the threat of the trolls, the misogynists, the racists, the rape threats. It’s accidentally getting your company DDOS’d because you complained at a conference. It’s the stalker-fan who sends you explicit messages through every channel, and knowing there’s nothing you can do about it except hope they don’t show up on your doorstep. This special kind of fear is a topic for entirely separate article, but I wanted to mention it briefly because it is a very real and true blocker for so many of us, particularly women in technology.

DESPITE all these doubts & fears. I still think it’s worth it to write. 

So, this is me reminding myself:

  • of all the helpful posts I’ve read and how glad I am that people shared them. To pay it forward.
  • of the thank you notes and tweets I’ve gotten, for the stuff I wrote that really helped people. 
  • that unlike a lot of other types of work, content keeps on giving long after you’ve published it, and it always seems worth it in hindsight
  • that I have a responsibility to the tech community. to do my part so that it’s not only men’s voices talking about technology and entrepreneurship. 
  • that publishing has been some of the most rewarding work I have done in my career, and that it’s incredibly fulfilling. that doing it for myself might be reason enough.

Peace & Love,

Michelle Wetzler

Chief Data Scientist, Human.

Query Performance Update

It’s been a rough couple of months for Keen IO customers – if you’ve been tracking Keen’s Status Page recently, you’ve probably noticed that we’ve reported a large number of incidents that have negatively impacted query performance for all of our customers.

We’ve been posting follow-up information for individual incidents on the Status Page, but we wanted to take some time to:

  • Acknowledge that there has been a pattern of frequent degraded query performance issues which is not acceptable to us
  • Provide you with more insight into why query performance has been suffering lately
  • Explain what we are doing to improve query performance

Why Has Query Performance Been So Variable?

Unanticipated Query Patterns
We deliberately designed Keen to be a very flexible analytics solution, and provided a set of APIs to Collect, Analyze, and Visualize data. Given the flexibility of the API and the growth of Keen IO, we’ve started to see customers using Keen and running Keen analysis queries that we didn’t anticipate and therefore we don’t handle gracefully.

Hardware/Infrastructure Issues
We’ve had a streak of bad luck with our hardware and physical infrastructure and haven’t done a great job managing their impact on our service: a network outage with our hosting provider, a hardware failure, a Cassandra node failure, and a misconfigured setting on our front-end API servers.

Increased Volume
The use of Keen IO by our customers has grown tremendously over the last several months, and we’ve run into some issues at scale that haven’t presented themselves before. A significant increase in events and queries has caused the platform to misbehave in different and unanticipated ways.

The combination of these issues has put a lot of strain on our platform, and query performance has suffered as a result.

That said, Keen should be able to handle increased customer volume, accommodate unanticipated query patterns, and keep running even when hardware/infrastructure fails. As a platform provider, it’s our job to anticipate and plan for these issues so that our customers don’t have to. We know that we need to do better. And we will!

What We’re Doing to Improve Query Performance

In the Short Term
We’ve started providing more prescriptive guidance on how to optimize queries by publishing a Query Performance Guide. We’re also working on a new version of our documentation to ensure that customers know how to optimally use Keen across the board.

We’re also currently rolling out some new internal tools to our infrastructure that will provide us much better visibility into customer query patterns, allowing us to implement better quality of service for our Analytics API and helping us harden our platform against unanticipated spikes in query volume.

Finally, we’re actively working on revising our query rate limits, and will be providing more detail on this very soon.

Longer-Term Fixes
In addition to those short-term tactical fixes, we’re currently implementing low-level changes to our Cassandra infrastructure to better identify, isolate, and remove performance bottlenecks in our platform.

It’s important to note that rolling out fixes while maintaining an active platform for customers will not be an overnight process. The “timeframe to resolution” on this should be thought of in weeks, not days. We’re shoring up our infrastructure now to harden the platform – not just for today’s requirements, but to keep us up and running well into the future.


We are really, really sorry for the hassle and inconvenience you’ve had to put up with lately. “Scaling is hard” is an explanation, not an excuse. It’s our job to ensure that what we’ve built can stand up and perform well for all of our customers, regardless of what load we put on it, and we’re committed to keep working and improving so we can provide all of our customers a rock-solid analytics platform that can be depended on.

We’ve hit some bumps in the road. We know how frustrating it is for you and your customers when things aren’t working, and we’re working incredibly hard to make things right. We’re confident things will get better and we hope you all stick with us while we’re making these improvements.

If you have any questions, please reach out to us anytime at

John Shim

I'm a tech focused "people person"

Updating Keen's API Status Codes

This week we’re pleased to ship some small changes to make our API responses a bit more precise when a problem is encountered. We’re doing this by changing a few of our HTTP status codes. We’ve gone over our SDKs to verify everything will continue to behave as expected, but we wanted to be sure and communicate these changes to the folks who’ve built such wonderful stuff on top of our API: you!

Better Than 400

Most of our errors return a status code of 400. This gets the point across, but it doesn’t necessarily identify the specific problem in such a way that an API user can react accordingly. A developer would need to examine the error message string, which is not nearly as simple as looking at the status code! To that end, here is the summary of changes:

  • Events that are too large (either single or in batch) will now return a 413, representing “request entity too large.”
  • Trying to delete a “keen” property will return a 403, representing “forbidden.”
  • API endpoints that have been blocked due to lack of payment will return a 402, representing “payment required.”

And One More Thing: Query Timeouts

There’s one other change that’s a bit bigger. Query timeouts used to return a 400, but they now return a 503, representing a “gateway timeout.” This is a more correct answer because responses in the 4XX range signal that the client has made a mistake. Queries timing out are clearly a problem with Keen, and we should be making that clear in our responses!

Anything Else?

These changes improve our SDK’s ability to take proper action on certain failures, improve our ability to monitor patterns of status codes, and all-around make our API better. We look forward to continuing to improve our API both in capabilities and correctness in the future. If you have any questions, don’t hesitate to shoot an email to Thanks for building things with Keen!

Cory Watson

Bigger than a breadbox.

7 Lessons from Heavybit’s DevGuild

I wanted to share with you my takeaways from Heavybit’s DevGuild. It was a great event with over 200 people coming together to focus on developer community-building.

One of the best parts of DevGuild was meeting so many fresh faces who are passionate about Developer Evangelism. Josh Dzielak was the emcee and Tim Falls eloquently gave a talk called, “Measuring developer evangelism…or not!?” Josh always keeps the crowd energized and Tim’s man bun was on point! You can check out all of the talks on Heavybit’s website.


So knowledgeable and full of man-bunly goodness!

Some solid learnings:

  • You don’t have to be intimidated or embarrassed about the word evangelism. This word can potentially have some negative connotations but you have control over how you represent yourself. Mainly, you have valuable information to share. Share it with the right people.
  • You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.
  • Don’t just get measurements. Ask why you’re measuring!
  • “Many of the things you can count, don’t count. Many of the things you can’t count, really count.”- Albert Einstein
  • Don’t underestimate the value of face-to-face interactions to build trust and develop relationships with your network. Building communities is an art. Connect your differences.
  • Big NO NO’s: 1. Not knowing your product very well 2. Drinking too much coffee (jitters) 3. Not buying the beer 4. Being a bridge to nowhere
  • The red sauce on the veggie platter from Bi-rite is spicy and awkwardly tangy

Plus, the event raised over $2735 for a local charity, the Canon Kip Senior Center {WAAAAPPOOWWW!!!}.

I had a blast meeting other Developer Evangelists and I talked to many people who said that they had to really fight to convince their management team that Developer Evangelism is important and should be invested in. We had some great conversations about metrics and tangible items that you can and cannot measure.

The field of Developer Evangelism is developing very quickly. It’s exciting and it makes all the difference knowing you are part of a team you can trust to represent your product in the very best way! Now go out there and share all that amazing information you have in those beautiful heads of yours.

Ricky Vidrio

empath adventurer

Top 12 new features and tools you might not know about

If you’re using Keen, you might have wondered: what’s the easiest way to build a dashboard? Or how can I get the fastest queries? Or how can I set up automatic reports? And maybe you’ve also wondered, where can I go to see all the latest features in one place? Well, wonder no more because here it is!

New API Features

  • Query caching (in private beta) - Get super-fast responses by setting preferences for when to use cached results

  • Custom intervals - Break up query results into any sub-timeframe, such as ‘every_10_minutes’

  • Funnels enhancements - Add actors and optional steps to analyze complex funnels

  • User-agent parsing - If you send Keen IO a user agent string, we can parse it out into browser, device, etc. for you

  • Extraction API enhancements - Use ‘property_names’ param to extract only the relevant properties of your events or ‘content_encoding’ param to compress your CSV extractions for faster download

  • Query Performance Guide - Tips for speeding up query time, extra useful for large data sets

New Query, Viz, and Reporting Tools

  • keen-cli - Quickly run Keen queries, add events, and run maintenance directly from the command line

  • New JavaScript library - More ways to render charts, interact with data, and show query visualizations

  • Dashboard templates for Bootstrap - Super handy when it comes time to make those beautiful dashboards

  • Pushpop - Easy way to make nightly / weekly / monthly email reports and SMS alerts based on Keen queries

New Data Collection and Streaming Tools

  • Streaming to S3 - If you need to analyze your incoming data outside of Keen, we can stream it to an S3 server for you

  • Electric Imp and Tessel Integrations - great ways for Internet of Things (IoT) developers to add analytics to all their projects

We also want to share some happy data from 2014 about our top two priorities: reliability and scalability. Over the past year, we scaled Keen volume by 12X and maintained 99.99% uptime for the year :)

Want to stay up-to-date on all our new features and announcements? Join the Keen dev group, and follow us on Twitter. Here’s to a Keenly awesome 2015!

This is the bar where we wrote this list. (Note: we later discovered we’re better at making lists than playing bar trivia.)

Kevin Wofsy

Teacher, traveler, storyteller

How To Display User Actions with LEDs

David Cook from Jut recently guest-wrote an awesome post on Segment’s blog about the benefits of creating “information radiators” to help quickly share information using visual cues and displays. In the post, he talked about how Jut built one of these radiators using Segment, Keen, and some LED lights. We thought this was so cool that we asked him to put together a more technical how-to guide for those of you who want to try this out for yourselves. Check it out!

First, What Is An Information Radiator?

Now that companies are collecting tons of data on what users are doing in their products (you are collecting user data, aren’t you?), the challenge shifts from figuring out what users are doing, to disseminating that information to the company. If you really want to be a data-driven organization, you have to make data accessible to everyone at all times. Enter the information radiator.

The most common information radiator is a wallboard – a TV that constantly displays information. However, humans have a natural instinct to tune out things that aren’t relevant. This is why it’s valuable to take advantage of alternate means of constantly communicating information to your team. My data communication tool of choice is an installation of LEDs.

Ok, So, What’s So Great About LEDs?

LEDs add depth to your data visualization arsenal. They’re more engaging than a wallboard because they’re unusual. You can configure LEDs into a variety of shapes: individual pixels, pixels arranged into a matrix, or a strip of LEDs strung together. We use LED strips because they’re easier to setup and allow you to construct a larger installation that more people are likely to see. Specifically, we use Heroic Robotics’ LED strips and control them with the PixelPusher. This combination simplifies the act of controlling individual LEDs on the strip.

How to Display User Actions with LEDs!

Start Capturing Data

While you should track everything your users do, it’s best to measure and display the moments where users derive value from your product so your team can strive to make more of those moments occur.

At Jut, the main way our users derive value from our product is by writing and running programs in our new language, Juttle, that retrieve and display data. We fire an event whenever a user does this, so we decided to send a pulse of light down an LED strip for each of these events. We use Segment to send those events to Keen.

Retrieve the Data from Keen

We’ve written a java program that runs on a server in our closet and pings Keen to get new data and displays that data on the LEDs. You can find the code here, but we’ll explain some of it below:

Every minute, we use Keen’s API to request the timestamps and status of all the Juttle programs run in the last minute. (We used Keen’s workbench to generate the URL that we hit via the API.) The java code looks like this and should be used in a thread that runs once a minute:

URL event_one = new URL("");

StringBuffer response = new StringBuffer();

try {

    HttpURLConnection con = (HttpURLConnection) url.openConnection();


    BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));

    String inputLine;

    while ((inputLine = in.readLine()) != null) {




} catch (ProtocolException e) {


} catch (IOException e) {



ArrayList eventList = new ArrayList();


Once this code runs, you have to parse the results of the Keen API call. How exactly you parse the response depends on the information Keen stores for you, so I’ll leave that up to you. However, you’ll definitely want to extract a timestamp and add a minute to it, to make it possible to play back the last minute of events at the same pace, instead of sending a bunch of pulses for each event at once.

That means, if you go to our website and run a Juttle program, one minute later a green pulse will shoot down the LED strip in our office like this:

Here’s an example of how to do this in java:

JSONArray events = new JSONObject(eventList.get(r).toString()).getJSONArray("result");

for (int x = 0; x < events(); x++) {

    String timestring = ((JSONObject)) events(x).get("keen")).getString("timestamp");

    String time = timestring.split("T")[1];

    int minute = Integer.parseInt(time.split(":")[1]) + 1;


Now that you know when to send the LED pulses, you just need to send them.

Configure the LEDs

To communicate with the LEDs, you need to use some of the methods that the PixelPusher team has provided:

if (testObserver.hasStrips) {




    List strips = registry.getStrips();


Once you have the strip, you can tell it what to do. We use a series of comparisons with the current system time to determine when to send a pulse:

long eventTime = eventList.get(x).getTimestring().getTimeInMillis();

// If the adjusted time of the event hasn't happened yet, then we haven't sent a pulse yet

if (eventTime > System.currentTimeMillis()) {

    // While the time of the event is greater than the current time, wait and keep checking to see if the time of the event ~= the current time

    while (eventTime > System.currentTimeMillis()) {

        // If the eventTime is within 10 milliseconds of system time, that's close enough, so send the pulse 

        // As soon as we start the pulse thread, we move on to the next event in the list to avoid sending multiple pulses for the same event

        if (eventTime - System.currentTimeMillis() <= 10) {

            pulseThread = new LEDPulse(pulseStrip, 0xffffff, 10);



The pulseThread class is what actually controls the turning off and on of the LEDs to make it look like a pulse of colored light is traveling down the strip. You can control the color and speed of the pulse which gives you plenty of flexibility to display all kinds of events on one LED strip. It’s pretty simple:

import devices.pixelpusher.Strip;

public class LEDPulse extends Thread {

    private Strip strip;

    private int color;

    private int speed;

    public LEDPulse(Strip strip, int color, int speed) {

        this.strip = strip;

        this.color = color;

        this.speed = speed;


    public void run() {

        try {

            for (int stripx = 0; stripx < strip.getLength(); stripx++) {

                strip.setPixel(color, stripx);

                if (stripx > 2) { // Once the pulse is 3 LEDs long, start turning the LEDs off behind the pulse

                    strip.setPixel(0, stripx - 3);




            // After the pulse has gone up the whole strip, make sure the last LEDs are off.

            for (int stripx = strip.getLength() - 4; stripx < strip.getLength(); stripx++) {

                strip.setPixel(0, stripx);


        } catch (InterruptedException e) {

            System.err.println("pulse interrupted");




Those are the key bits of the program; to see all of the code, check out the repo.

All in all, information radiators like these create a more lively atmosphere in the office. We can easily see how active our users are in almost realtime. If it seems like we’re experiencing an unusual amount of traffic, we can turn to our wallboard to see how the current level compares to historical levels.

What else do you think you could use LEDs for? Let us know on Twitter @jut_inc.

David Cook

Growth data dude @ Jut

Analytical Apps for Your Smartwatch: Getting Started with Keen + Pebble

As the market for connected devices keeps growing, there are a ton of cool possibilities for how you can connect the data you collect to these devices (and how you collect data from them). Today, I’ll show you how to build a Pebble smartwatch app with built-in data and analytics features. What do I mean by this? Well, as an example, I built a Pebble app that displays charts and graphs of API response times.

I used Keen IO to collect and query the data, and Pingpong, one of our open source projects, for monitoring API response times.

Read on to see how we put the pieces together, then get started tracking and querying in an awesome project of your own!

Download the full source code for the app here:

Things You’ll Need:

  • A free Keen IO account
  • A Pingpong instance – an open source project that uses Keen IO as a backend to monitor API response time
  • A Pebble smartwatch
  • A basic understanding of JavaScript

Connecting to Keen via Pebble.js

Our goal here is to build a simple app on the Pebble smartwatch that runs a query to get a list of all of the API checks our Pingpong instance are running, along with a more detailed summary page for each individual API check that shows the last 30 minutes of activity.

The first part is a little tricky – we need to get a list of all the checks that Pingpong makes, but Pingpong itself does not have an API end-point to pull the check information. We can get around that by knowing that the checks collection can provide that info via an API call:

runKeenQuery("average", "checks", {
    target_property: "request.duration",
    group_by: "",
    timeframe: "this_2_minutes",
    interval: "minutely"
}, function(keenRes) {

The response contains all the information we need, namely the field that gives us each of the individual checks. The runKeenQuery function is just a convenience function that wraps the ajax pebble.js function.

That runKeenQuery call is the same one that populates the more detailed view of a check, just with an additional filter applied:

runKeenQuery("average", "checks", {
  target_property: "request.duration",
  group_by: "",
  timeframe: "this_30_minutes",
  interval: "minutely",
  filters: JSON.stringify([{"property_name":"","operator":"eq","property_value":e.item.title}])
}, function(check) {

These two calls are the main crux of this application, but can serve as a basis for any Keen query integration. Once the query returns, you have easy access to the resulting JSON and can then integrate that into the application in any number of ways.

Example of Using Result Data

For this watch app, I wanted to create a little graph that showed the resulting response times. Since no charting library existed, I wrote a basic script to translate the query results into drawn rectangles.

The response JSON from the Keen query looks something like this:

      }, {...}         

To turn this into a graph, I normalized the result field, and then multiplied the normalized number by the maximum x-axis height of the graph (adding a little padding so that we next max out the height).

for (var ndx in averageTimes) {
  var height = Math.floor((averageTimes[ndx] / scalar) * maxHeight);
  var currentYPos = yPos + (maxHeight - height);

  checkCard.add(new UI.Rect({
    position: new Vector2(currentXPos, currentYPos),
    size: new Vector2(width, height),
    backgroundColor: 'black'

  currentXPos += width + 1;

This crudely draws out a rectangle for each result, making the bar graph.

This is just one basic example of using query results to display information, but there are a bunch of other possibilities – not to mention the possibility of sending data to Keen based on interactions with your app. If you have questions or ideas, feel free to shoot me an email, or tweet me @alexk_k.

Alex Kleissner

Software engineer by day, Partybot by night

That Time We Almost Became a Music Company

So, yep, we made some pretty kooky music videos as part of a team building exercise for our company holiday party. Some of us even seriously considered pivoting. If you can’t wait for the end of the story, check them out here. Otherwise, keep reading to find out how it all went down.

The build-up

It was all very hush-hush. Something was going to happen at 3pm on December 4th, but only Chelle knew exactly what. Nate knew props were needed, but with Nate, there are always props, so that didn’t really help with the mystery.

Finally the time came, and we went downstairs to the speakeasy room. Waiting for us was a man in an orange jumpsuit with a bunch of giant duffel bags. “Oh my god, we’re going skydiving,” I thought. “Skydiving with props.”

That’s me holding the coffee cup in fear and anticipation.

But actually, it was much scarier than that.

“You’re going to make music videos!” said Jumpsuit John. “In two hours!”



We were stunned. Excited, of course. But… Well, here’s the thing. At Keen we always say we’re more than just an analytics company. We’re a people development company. We give people the tools they need to create amazing things. It’s just, we didn’t realize those tools might be duffel bags full of foam wigs and inflatable saxophones.

For a few moments, we hesitated in our groups, spread out around the room, pawing through costumes. The pressure mounted. After all, we were going to be screening the videos at the holiday party in two hours.

Jumpsuit John was not impressed by our dithering and vacillation. “Go, go, go!” he said. “Pick a lead singer. Decide on your characters and your story. The clock is ticking. Go!”

Shooting the Dream

We dispersed throughout the building. Micah led my team to a floor I didn’t know existed. It was still under construction, dusty, live wires dangling from the ceiling, generally the kind of place you’re pretty sure you’re not supposed to be.

“We need a story.”

“It could be a love story.”

“Maybe a love triangle.”

“There’s a bunch of us. It should be a love rhombus.”

“A love trapezoid.”

“I vote hexagon. Love hexagon all the way!”

From there, my group kind of spitballed as we went, prodded along by Jumpsuit John’s helpful texts, such as: “Why haven’t you sent me any scenes yet?” and “You’re running out of time!” and “Start!”

Other teams had different approaches. The *NSYNC group white-boarded the scenes in advance, broken down by the number of seconds per shot, just as Justin Timberlake would have done if he were secretly a data scientist.

Amazingly, every group finished their video in time and felt pretty psyched about the results. The same people who had been scared and confused at 3pm had now become rock gods, or at least a lot closer to rock godhood than we had ever expected to get.

When the screening ended, Nate popped the cork on a giant champagne bottle that shot confetti across the room. We voted on winners, but we seriously all felt like winners. (Disclaimer: I say this as someone who did not win anything. The winners probably felt even more like winners, especially since they got cool Oscar statues and got to give speeches. Best speech award goes to Andrew Koester. It was three words long: “I was home-schooled.”)

Special Thanks

I have to end with some special thanks and some semi-maudlin reflections. First, the thanks. Chelle, thank you so much for putting together the perfect team-building event and party! Thank you for keeping it a secret, which built anticipation, maximized impact, and spared us days of anxiety about how on earth we were going to make music videos. Also, thanks for not booking a skydiving day, because that would have been seriously terrifying.

Thanks also to Nate for always adding fun and creativity to everything he touches. Props for the props.

And thanks to Jumpsuit John for being an extraordinary producer/team-builder/taskmaster flash. You gave us just the right amount of structure, encouragement, tough love, and duffel bags to pull this off.

Semi-Maudlin Reflections

This whole video business was tons of fun, but it also made me think about what we’re trying to build here as a culture and a company. At Keen, we talk a lot about how important it is for us to be able to be vulnerable with each other, to trust each other and care about each other as people. I’ve heard almost everyone here say in one way or another that this core value is what made them want to join the company. Certainly that’s the case for me. It’s what makes me want to do the best work I can, and it’s what makes me feel safe trying new things even if I might not be the very best at them the first time I try. That’s how we grow as people, it’s how we will grow as a company, and we hope our customers will see and feel that we’re a people-driven place even though we have a data-centric product.

And if we can be vulnerable with each other, we want to be vulnerable with our friends and readers and customers and onlookers, too. So we’re sharing our music videos with the world. Maybe you’ll love them. Maybe you’ll hate them. Maybe you’ll be super glad that we’re an analytics company and not a music label. Either way, Happy Holidays from Keen and enjoy the show!

If you want to make your own music videos – or if you just really want to meet Jumpsuit John and his duffel bags – go check out The Go Game. It’s seriously awesome!”

The big debut at BoxSF.

Kevin Wofsy

Teacher, traveler, storyteller