Personal Data

Combining Tasker with Elasticsearch and Kibana for Personal Dashboards

Tasker is probably my favorite Android app. It gives you total control over your device. Not to mention there’s a great community that develops all sorts of interesting tasks/integrations.

I was recently on a 10+ hour flight where the entertainment system wasn’t working and my laptop was inaccessible so I was kind of bored. But I had my phone/Tasker, and I got the idea to write a little IoT type of integration while I was on the plane. Basically, I wanted to create an interesting integration of shipping the sensors from my phone off to an Elasticsearch cluster. With this, I could:

Keep track of where I’ve been, including my run paths and any hikes
Track and visualize my pedometer over days
Remember what that song was I was listening to on Play Music/Pandora/Spotify/etc but forgot the name to
See graphs of my battery usage over days so I can see how the profile changes as I add/update/remove apps, as I tether, and as my phone ages
Not be bored for a while on the plane

Building Kibana Dashboards

Before I walk through how I created this and give you the downloadable task, I just want to show these dashboards, which I really liked:

Here, I’ve plotted the current battery % on the left and using a derivative pipeline aggregation (available in Elasticsearch and Kibana), we can see how much much the device charged or how much it drew down. We can use this to plot the charge speed of different chargers or see how much apps use. Because I also have things like the screen brightness and network information in the same index object, I can actually plot them against one another to see how much the screen on time/brightness uses battery over a multi-day cycle.

I can plot my runs and corresponding step counts:

Or where I’ve been the last 7 days:

Or even the last 90 days:

I was hoping to get a signal strength map of my cellphone signal, but it looks like Tasker may always be returning “0” even when I have full strength. I may have set that up wrong, so if you have info on how to get it to return non-zero values, let me know!

Anyway, on to the code.

Tasker Javascript Tasks: The Framework

Tasker has Javascript tasks which are quite interesting: you can access variables and you have access to things like XMLHttpRequest. That makes it pretty easy to get the Tasker variables and then send them across to a RESTful engine like Elasticsearch.

A few hundred miles across the Atlantic later, I had something like this:

var esServer = par[0];
if (esServer.substr(esServer.length - 1) !== "/") {
	esServer = esServer + "/";
}
setLocal('%esserver', esServer);
setLocal('%authinfo', par[1]);
var indexPrefix = "tasker";

var d = new Date();
var dateStr = d.toISOString().slice(0,10);

var intVars = { battery: '%BATT', cell_signal_strength: '%CELLSIG',
									  display_brightness: '%BRIGHT', light_level: '%LIGHT',
										uptime: '%UPS', free_memory: '%MEMF', pedometer: '%STEPSTAKEN'
									};
var doubleVars = { altitude: '%LOCALT', magnetic_strength: '%MFIELD', temperature: '%TEMP' };
var booleanVars = { bluetooth_on: '%BLUE', locked: '%KEYG', muted: '%MUTED', speakerphone: '%SPHONE',
									  wifi_enabled: '%WIFI', wimax_enabled: '%WIMAX', screen_on: '%SCREEN', roaming: '%ROAM',
airplane_mode: '%AIR'
									};
var keywordVars = { bluetooth_on: '%BLUE', cell_network: '%TNET', device: '%DEVID', device_id: '%DEVTID' };

postData = { timestamp: d.toISOString() };
template = { timestamp: { type: 'date' } };

for (var i in keywordVars) {
  var res = global(keywordVars[i]);
  if (typeof(res) !== 'undefined') {
  	postData[i] = res;
  }
  template[i] = { type: 'keyword' };
}

for (var i in booleanVars) {
  var res = global(booleanVars[i]);
  if (typeof(res) !== 'undefined' && res !== '') {
  	postData[i] = (res === 'true' || res === 'on');
  }
  template[i] = { type: 'boolean' };
}

for (var i in intVars) {
  var res = global(intVars[i]);
  if (typeof(res) !== 'undefined' && res !== '') {
  	if (res >= 0) {
    	postData[i] = parseInt(res);
    }
  }
  template[i] = { type: 'long' };
}

for (var i in doubleVars) {
  var res = global(doubleVars[i]);
  if (typeof(res) !== 'undefined' && res !== '') {
  	if (res !== 0) {
    	postData[i] = parseFloat(res);
    }
  }
  template[i] = { type: 'double' };
}

template['location'] = { type: 'geo_point' };
if (getLocation('any', true, 30)) {
  var loc = global('%LOC');
	var latlon = loc.split(',');
	postData['location'] = { 'lat': parseFloat(latlon[0]), 'lon': parseFloat(latlon[1]) };
}

template['song'] = { type: 'text' };
template['music_playing'] = { type: 'boolean' };
if (global('%ARTIST') !== '' && global('%TRACK') !== ''
   ) {
	  postData['music_playing'] = (global('%ISPLAYING') === 'true' || global('%ISPLAYING') === true);
	  if (postData['music_playing'] === true) {
		  postData['song'] = global('%ARTIST') + ' - ' + global('%TRACK');
		}
} else {
	if (global('%ISPLAYING') === 'false' || global('%ISPLAYING') === false) {
	  postData['music_playing'] = false;
	}
}

var wifii = global('%WIFII');
if (typeof(wifii) !== 'undefined' && res !== '') {
	var wifiarr = wifii.split("\n");
	if (wifiarr[0] === '>>> CONNECTION <<<') {
		postData['wifi_name'] = (wifiarr[2]).slice(1,-1);
	}
	template['wifi_name'] = { type: 'keyword' };
}

var jsondocstring = JSON.stringify(postData);
var indexName = indexPrefix + '-' + dateStr;
indexType = 'doc';
setLocal('%jsondocbulkheader', JSON.stringify({ "_index": indexName, "_type": indexType}));
setLocal('%jsondocstring',jsondocstring);

var xhrTemplate = new XMLHttpRequest();
xhrTemplate.open("PUT", esServer + "_template/" + indexPrefix, false);
xhrTemplate.setRequestHeader("Content-type", "application/json");
if (typeof(par[1]) !== 'undefined') {
	xhrTemplate.setRequestHeader("Authorization", "Basic " + btoa(par[1]));
}
var templateString = JSON.stringify({ template: indexPrefix + '-*', mappings: { doc: { properties: template } } });

try {
	xhrTemplate.send(templateString);
} catch (e) { }
try {
	var xhrDoc = new XMLHttpRequest();
	xhrDoc.open("POST", esServer + indexName + '/' + indexType, false);
	xhrDoc.setRequestHeader("Content-type", "application/json");
	if (typeof(par[1]) !== 'undefined') {
		xhrDoc.setRequestHeader("Authorization", "Basic " + btoa(par[1]));
	}

	xhrDoc.send(jsondocstring);
	setLocal('%sentdoc','1');
} catch (e) { }
exit();

Let’s walk through what this does.

In the beginning:

var esServer = par[0];
if (esServer.substr(esServer.length - 1) !== "/") {
	esServer = esServer + "/";
}
setLocal('%esserver', esServer);
setLocal('%authinfo', par[1]);
var indexPrefix = "tasker";

var d = new Date();
var dateStr = d.toISOString().slice(0,10);

Here, we set up an index prefix (“tasker”) and we’ll get 2 parameters passed in: the Elasticsearch server host/port (in par[0], which goes into %esserver), and any authentication in the form of username:password (in par[1], which goes into %authinfo).So reminder: when we add this Javascript task, we’ll need to pass those bits in.

var intVars = { battery: '%BATT', cell_signal_strength: '%CELLSIG',
									  display_brightness: '%BRIGHT', light_level: '%LIGHT',
										uptime: '%UPS', free_memory: '%MEMF', pedometer: '%STEPSTAKEN'
									};
var doubleVars = { altitude: '%LOCALT', magnetic_strength: '%MFIELD', temperature: '%TEMP' };
var booleanVars = { bluetooth_on: '%BLUE', locked: '%KEYG', muted: '%MUTED', speakerphone: '%SPHONE',
									  wifi_enabled: '%WIFI', wimax_enabled: '%WIMAX', screen_on: '%SCREEN', roaming: '%ROAM',
airplane_mode: '%AIR'
									};
var keywordVars = { bluetooth_on: '%BLUE', cell_network: '%TNET', device: '%DEVID', device_id: '%DEVTID' };

Here, we gather a list of hardware/software variables that Tasker defines into a variety of types. For example, we put %TEMP into a JSON object of other double variables. It’s important that we do this because we need to know what these types are so we can make some assumptions around how we process each of the variables. For example, looking at Tasker’s variables page, we can see that the variable representing if the device is in Airplane mode (%AIR) returns as “on” or “off” while we really want to process them as booleans (true or false). Many numbers in Tasker return -1 if the value is not defined or cannot be determined, where we really want that to be an empty value in storage. So we store our expected types in a JSON object to check later.

The next set of for loops does 2 things:

for (var i in keywordVars) {
  var res = global(keywordVars[i]);
  if (typeof(res) !== 'undefined') {
  	postData[i] = res;
  }
  template[i] = { type: 'keyword' };
}

for (var i in booleanVars) {
  var res = global(booleanVars[i]);
  if (typeof(res) !== 'undefined' && res !== '') {
  	postData[i] = (res === 'true' || res === 'on');
  }
  template[i] = { type: 'boolean' };
}

for (var i in intVars) {
  var res = global(intVars[i]);
  if (typeof(res) !== 'undefined' && res !== '') {
  	if (res >= 0) {
    	postData[i] = parseInt(res);
    }
  }
  template[i] = { type: 'long' };
}

for (var i in doubleVars) {
  var res = global(doubleVars[i]);
  if (typeof(res) !== 'undefined' && res !== '') {
  	if (res !== 0) {
    	postData[i] = parseFloat(res);
    }
  }
  template[i] = { type: 'double' };
}

First, it tries to convert, parse, or drop the value based upon the aforementioned logic. Next, it adds the variables it finds to an Elasticsearch template variable we use later.

Next, we have a bit of custom logic for parsing the current location (which comes in a single string of the form “Lat,Lon”), the currently playing song (which comes through Media Utilities), and the WiFi connection info to get the current network name (the %WIFII variable has a rather bizarre format: the 3rd line contains the network name if the first line is “>>> CONNECTION <<<“):

template['location'] = { type: 'geo_point' };
if (getLocation('any', true, 30)) {
  var loc = global('%LOC');
	var latlon = loc.split(',');
	postData['location'] = { 'lat': parseFloat(latlon[0]), 'lon': parseFloat(latlon[1]) };
}

template['song'] = { type: 'text' };
template['music_playing'] = { type: 'boolean' };
if (global('%ARTIST') !== '' && global('%TRACK') !== ''
   ) {
	  postData['music_playing'] = (global('%ISPLAYING') === 'true' || global('%ISPLAYING') === true);
	  if (postData['music_playing'] === true) {
		  postData['song'] = global('%ARTIST') + ' - ' + global('%TRACK');
		}
} else {
	if (global('%ISPLAYING') === 'false' || global('%ISPLAYING') === false) {
	  postData['music_playing'] = false;
	}
}

var wifii = global('%WIFII');
if (typeof(wifii) !== 'undefined' && res !== '') {
	var wifiarr = wifii.split("\n");
	if (wifiarr[0] === '>>> CONNECTION <<<') {
		postData['wifi_name'] = (wifiarr[2]).slice(1,-1);
	}
	template['wifi_name'] = { type: 'keyword' };
}

Finally:

var jsondocstring = JSON.stringify(postData);
var indexName = indexPrefix + '-' + dateStr;
indexType = 'doc';
setLocal('%jsondocbulkheader', JSON.stringify({ "_index": indexName, "_type": indexType}));
setLocal('%jsondocstring',jsondocstring);

var xhrTemplate = new XMLHttpRequest();
xhrTemplate.open("PUT", esServer + "_template/" + indexPrefix, false);
xhrTemplate.setRequestHeader("Content-type", "application/json");
if (typeof(par[1]) !== 'undefined') {
	xhrTemplate.setRequestHeader("Authorization", "Basic " + btoa(par[1]));
}
var templateString = JSON.stringify({ template: indexPrefix + '-*', mappings: { doc: { properties: template } } });

try {
	xhrTemplate.send(templateString);
} catch (e) { }
try {
	var xhrDoc = new XMLHttpRequest();
	xhrDoc.open("POST", esServer + indexName + '/' + indexType, false);
	xhrDoc.setRequestHeader("Content-type", "application/json");
	if (typeof(par[1]) !== 'undefined') {
		xhrDoc.setRequestHeader("Authorization", "Basic " + btoa(par[1]));
	}

	xhrDoc.send(jsondocstring);
	setLocal('%sentdoc','1');
} catch (e) { }
exit();

In this block, we form the document (and the index template!) into a string via JSON.stringify(), we upload them both to Elasticsearch. Tasker seemed to have some issues with the asynchronous nature of XMLHttpRequest, though it’s possible I was doing something wrong, so I converted them into synchronous calls. At the very end, we set “%sentdoc” to 1 to indicate to the rest of the task that we’ve successfully sent the document. If this isn’t set to 1, we know we didn’t have Internet access or some other barrier to uploading the metrics. In a subsequent action in the task, we’ll write out the bulk header and document to a file on disk so we can upload all the data we missed.

Putting It Into A Project

This project has 4 profiles:

Every 10 steps, run a task that sets %STEPSTAKEN to 10+%STEPSTAKEN
Every night at midnight, run a task that sets %STEPSTAKEN to 0
Every 5 minutes, run the “Update Info” task
Every time Media Utilities has a new track, run the “Update Info” task

That leaves me to the “Update Info” task. That simply grabs the current values of %ISPLAYING, %TRACK, and %ARTIST, and then runs the “Send to Elasticsearch” task.

The “Send to Elasticsearch” task is a bit more complicated. It:

Sets %sentdoc to 0 (we’ll use this later!)
Runs the Javacriptlet above
If %sentdoc isn’t set to 1 at the end of the Javascript, we write out %jsonbulkheader and then %jsondocstring to a file Tasker/tasks/taskQueue.json
Else if %WIFII contains >>> CONNECTION <<< this means we’re on a wifi network and we can bulk upload the taskQueue contents. So we read that into %jsonqueue and send it up with another (much shorter) Javascript (and then delete the taskQueue.json file so we don’t resend it):

var jsondocstring = jsonqueue;

var xhrDoc = new XMLHttpRequest();
xhrDoc.open("POST", esServer + '_bulk', false);
xhrDoc.setRequestHeader("Content-type", "application/json");
if (typeof(authinfo) !== 'undefined') {
	xhrDoc.setRequestHeader("Authorization", "Basic " + btoa(authinfo));
}

xhrDoc.send(jsondocstring);

setLocal('%jsonqueue','✓');
exit();

And that’s it!

Putting It All Together

I’ve uploaded the entire project here. You can download them directly and import the project or just have a look over the output and see how it’s tied together and make your own. It does need the %ELASTICSEARCH and %ELASTICSEARCH_AUTH variables to be set and you’ll see it needs Media Utilities to be installed as well. I had fun making this and I’m hoping to add more to it in the future!

November 15, 2017 by Shane Connelly Personal Data Software

Connecting Domoticz and Elasticsearch

I use Domoticz for my home automation needs. I’ll just start off by saying it has its pros and cons.

Pros:

It seems to have pretty wide device support
It allows for a completely offline home-automation setup (for those of us that don’t want our every home action to go through a 3rd party server, some of which have proven…problematic)
It has an API (though the output format is…not so great — more on that later)
You can create your own virtual devices (e.g. I use this to create a virtual device for when my phone connects to my home router)
It can run natively on a RaspberryPi (or a BananaPi or BananaPro if you fork the project and make some minor modifications). Together with a RazBerry, this makes for a great self-contained unit.
It’s pretty scriptable. They’ve embedded Blockly and both a Lua and Python API

Cons:

The default UI is pretty ugly
Not a lot of great mobile support
Integration with 3rd parties is limited. For example, the community has made an InfluxDB/Grafana integration, but there doesn’t seem to be any Elasticsearch integration at this time
The API is very string-y: many components return a string like “3.8A” instead of value: 3.8 + unit: “A”. This makes many things a bit challenging to parse.

This post is about trying to address a few of the cons by uploading data to Elasticsearch and using Kibana for visualizations.

Understanding the Domoticz API

First, it’s important to note the output of the Domoticz API. It’s is largely documented here, but for this demo, we’re going to focus on the type=devices component of the API.

Let’s have a look at the result:

{
   "ActTime" : 1510530122,
   "ServerTime" : "2017-11-12 15:42:02",
   "Sunrise" : "06:48",
   "Sunset" : "16:59",
   "result" : [
      {
         "AddjMulti" : 1.0,
         "AddjMulti2" : 1.0,
         "AddjValue" : 0.0,
         "AddjValue2" : 0.0,
         "BatteryLevel" : 100,
         "CustomImage" : 0,
         "Data" : "74.3 F, 41 %",
         "Description" : "",
         "DewPoint" : "49.10",
         "Favorite" : 0,
         "HardwareID" : 6,
         "HardwareName" : "RaZberry",
         "HardwareType" : "OpenZWave USB",
         "HardwareTypeVal" : 21,
         "HaveTimeout" : false,
         "Humidity" : 41,
         "HumidityStatus" : "Comfortable",
         "ID" : "0201",
         "LastUpdate" : "2017-11-12 15:41:28",
         "Name" : "Office Temperature",
         "Notifications" : "false",
         "PlanID" : "0",
         "PlanIDs" : [ 0 ],
         "Protected" : false,
         "ShowNotifications" : true,
         "SignalLevel" : "-",
         "SubType" : "WTGR800",
         "Temp" : 74.299999999999997,
         "Timers" : "false",
         "Type" : "Temp + Humidity",
         "TypeImg" : "temperature",
         "Unit" : 0,
         "Used" : 1,
         "XOffset" : "0",
         "YOffset" : "0",
         "idx" : "65"
      }
   ]
}

This gives us the basis for what we’ll need to do!

A Few Bits About the API

There are a few things that are a bit of pain with the API that you can see from the above:

Domoticz treats everything in Celsius even if the devices return in Fahrenheit. As a result, there can be multiple conversions back and forth which can result in floating-point rounding troubles, as you can see above which is 74.29999… instead of 74.3 as the device had returned. I haven’t delved into every API or component return, but I assume this is more than just Celsius/Fahrenheit but in other metric/English conversions.
The devices tend to return multiple values in the same device and those may have varying types. For example, notice the “Data” above is “74.3F, 41 %” which is Temperature + Humidity combined in a single string while we also get a separate “Humidity” value (integer) and a separate “Temp” value (floating point) and a separate “DewPoint” value (floating point, but JSON typed to string and not in the Data block). Other values you’d expect to be numbers (“SignalLevel”) may return strings (e.g. “-“) if no value is provided. “true/false” may return in strings (as you can see in “Timers”) etc.
A continuation of #2, if every type of device had all of the values it was returning in a separate JSON object that was at least standardized against itself, then this would just be mildly annoying, but alas that’s not the case. For example, energy monitoring devices only seem to return their data in the “Data” string (no “Amperage” or “Wattage” attribute)

Also, Domoticz doesn’t seem to have any internal scheduled-scripting (that I’ve found) with any frequency higher than once every minute, so we have to poll this API to get the data off the device instead of using those lovely internals. So this is what we have to work around. Let’s get started!

Using Elasticsearch’s Ingest Nodes

Elasticsearch has Ingest Nodes, which allow us to transform this type of data directly in the Elasticsearch cluster without setting up any external dependencies. First, we’ll need to create an ingest pipeline:

I set up the following:

curl -XPUT "https://MY_ES_USER:MY_ES_PASSWORD@MY_ES_HOST:MY_ES_PORT/_ingest/pipeline/domoticz" -H 'Content-Type: application/json' -d'
{
  "description": "Converts domoticz API to ES",
  "processors": [
    {
      "date": {
        "field": "LastUpdate",
        "target_field": "timestamp",
        "formats": [
          "yyyy-MM-dd HH:mm:ss"
        ],
        "timezone": "America/Los_Angeles"
      },
      "remove": {
        "field": "LastUpdate"
      },
      "set": {
        "field": "IngestTime",
        "value": "{{_ingest.timestamp}}"
      },
      "rename": {
        "field": "Data",
        "target_field": "DataString"
      },
      "script": {
        "lang": "painless",
        "source": "if (ctx.Type == \"Current\") { ctx.Current = ctx.DataString.replace(\" A\",\"\") } else if (ctx.Type == \"General\") { if (ctx.Usage != null) { ctx.Wattage = ctx.Usage.replace(\" Watt\",\"\"); } else if (ctx.CounterToday != null) { ctx.EnergyToday = ctx.CounterToday.replace(\" kWh\",\"\"); } else if (ctx.DataString.endsWith(\" SNR\")) { ctx.SNR = ctx.DataString.replace(\" SNR\",\"\"); } } else if  (ctx.Type == \"Lux\") { ctx.Lux = ctx.DataString.replace(\" Lux\",\"\") }"
      },
      "date_index_name": {
        "field": "IngestTime",
        "index_name_format": "yyyy-MM-dd",
        "index_name_prefix": "devices-",
        "date_rounding": "d"
      }
    }
  ]
}'

This does a few things: it moves the “LastUpdate” field into a “timestamp” field, sets an “IngestTime” field to the time the document was received, adds “Current,” “Wattage,” “EnergyToday,” “Lux,” and “SNR” named fields if the based on the “Type” and the Data field string, and finally sets up a date-based index for the values.

Great. Now we have an ingest pipeline that can process the results of each of the devices. This will place the data into an index named devices-<date>. We’ll also want an index template to store the data to:

curl -XPUT "https://MY_ES_USER:MY_ES_PASSWORD@MY_ES_HOST:MY_ES_PORT/_template/domoticz" -H 'Content-Type: application/json' -d'
{
  "domoticz": {
    "order": 0,
    "template": "devices-*",
    "settings": {
      "index": {
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "devices": {
        "properties": {
          "BatteryLevel": {
            "type": "scaled_float",
            "scaling_factor": 100
          },
          "DataString": {
            "type": "text"
          },
          "HardwareName": {
            "type": "keyword"
          },
          "IngestTime": {
            "type": "date"
          },
          "Level": {
            "type": "double"
          },
          "LevelInt": {
            "type": "scaled_float",
            "scaling_factor": 100
          },
          "Name": {
            "type": "text"
          },
          "Status": {
            "type": "keyword"
          },
          "Type": {
            "type": "keyword"
          },
          "Current": {
            "type": "double"
          },
          "DewPoint": {
            "type": "double"
          },
          "Gust": {
            "type": "double"
          },
          "EnergyToday": {
            "type": "double"
          },
          "Rain": {
            "type": "double"
          },
          "RainRate": {
            "type": "double"
          },
          "Speed": {
            "type": "double"
          },
          "Chill": {
            "type": "double"
          },
          "UVI": {
            "type": "double"
          },
          "Direction": {
            "type": "double"
          },
          "Lux": {
            "type": "integer"
          },
          "Wattage": {
            "type": "double"
          },
          "CounterToday": {
            "type": "text"
          },
          "HumidityStatus": {
            "type": "text"
          },
          "Usage": {
            "type": "text"
          },
          "idx": {
            "type": "integer"
          },
          "timestamp": {
            "type": "date"
          }
        }
      }
    }
  }
}'

Most of this template is not purely “necessary” as Elasticsearch would handle the data correctly in most if not all of these field values, but it sets up efficient mappings and makes sure we don’t have mistakes in our data model.

Uploading the API to Elasticsearch

We need to push the data to Elasticsearch at some regular interval. I’ve used jq to do some lightweight filtering and reformatting of the data and then ultimately push it up to Elasticsearch. We can add something like the following to the crontab to initiate it on boot.

#!/bin/bash

while true
  do curl "http://127.0.0.1:8080/json.htm?type=devices&filter=all&used=true&order=Name" | jq -c '.result[] | with_entries(select( .key as $k | $keys | index($k)))' --arg keys '["idx","HardwareName","Name","BatteryLevel","Gust","Direction","Chill","Speed","Temp","UVI","Rain","RainRate","Visibility","Radiation","Level","LevelInt","Status","Humidity","DewPoint","HumidityStatus","CounterToday","Usage","Voltage","Type","Data","LastUpdate"]' | sed -e 's/^/{ "index" : { "_index" : "devices", "_type" : "devices" } }\
/' > /tmp/devices_bulk.json && curl -H 'Content-type: application/json' -XPOST 'https://MY_ES_USER@MY_ES_PASSWORD:MY_ES_PORT/_bulk?pipeline=domoticz' --data-binary @/tmp/devices_bulk.json > /dev/null && sleep 5
done

This script does a few things: it selects the relevant pieces of information that I want indexed (Temperature, HardwareName, etc) from the response and then formats a _bulk index request to Elasticsearch and then POSTs it.

In my case, I’ve used an Elastic Cloud instance to push the data to. This allows me to externalize the data in a secure fashion and make it available no matter where I’m at, but only to me. It also lets me fire off alerts, e.g. if one of my security devices goes off and I’m not home.

Drop this bash script into /opt/domoticz/scripts/ (where I’ve installed Domoticz) and in crontab, add:

@reboot bash /opt/domoticz/scripts/uploadES.sh

Setting Up Kibana

The two biggest quirks for setting up Kibana with this data are

Domoticz reports the time as the last time the device had an update, but for most dashboards what you really want is to see the state of all devices at a given point in time. This means we really want the ingest time, not the device update time (hence adding that to the ingest template).
Using the device IDs for multi-valued documents with similar types. For example, showing all motion sensors or door sensors on the same graph. For this, there are a few approaches, but I used the sub aggregation to split the charts by filtering for specific device IDs (“idx” in the data). This looks like the following in Kibana:

Example Dashboard

Once we tie it all together, we get quite nice graphs!

This makes for a nice dashboard! We can see the inside and outside temperatures, when there was motion in various spaces (in red), when doors were opened (in red), and when devices were turned on and off. You can even see in the “office” and “living room” temperature charts when the heat kicked on in my apartment. We can fire off alerts for security (e.g. “front door is opened but nobody’s home”) or

Future Work

You’ll likely see some of the following in future posts or as future work from me:

The bottom right graph hides additional detail of something I’ve built out, which is tracking the wifi strength of connected/roaming devices. I think this is quite useful as a virtual presence detection device. I’ll add more detail in the future about how I’ve done this.
I’m adding more devices. One of the things I really like about having the data in a place like this is I can start to gather and visualize other external inputs to the system (e.g. my phone’s sensors and location). I’ve got some code written for this already.
I’d like to make a native Elasticsearch exporter and timer for Domoticz and fix some of this API weirdness, but I’m not terribly familiar with the Domoticz C++ codebase yet to do this.
I think my habits could be fairly well modeled with Elastic’s machine learning. I’d like to build/use machine learning models on top of this data to automatically flag and alert on anomalies in the data rather than hard-code rules.