I use Domoticz for my home automation needs. I’ll just start off by saying it has its pros and cons.
Pros:
- It seems to have pretty wide device support
- It allows for a completely offline home-automation setup (for those of us that don’t want our every home action to go through a 3rd party server, some of which have proven…problematic)
- It has an API (though the output format is…not so great — more on that later)
- You can create your own virtual devices (e.g. I use this to create a virtual device for when my phone connects to my home router)
- It can run natively on a RaspberryPi (or a BananaPi or BananaPro if you fork the project and make some minor modifications). Together with a RazBerry, this makes for a great self-contained unit.
- It’s pretty scriptable. They’ve embedded Blockly and both a Lua and Python API
Cons:
- The default UI is pretty ugly
- Not a lot of great mobile support
- Integration with 3rd parties is limited. For example, the community has made an InfluxDB/Grafana integration, but there doesn’t seem to be any Elasticsearch integration at this time
- The API is very string-y: many components return a string like “3.8A” instead of value: 3.8 + unit: “A”. This makes many things a bit challenging to parse.
This post is about trying to address a few of the cons by uploading data to Elasticsearch and using Kibana for visualizations.
Understanding the Domoticz API
First, it’s important to note the output of the Domoticz API. It’s is largely documented here, but for this demo, we’re going to focus on the type=devices component of the API.
Let’s have a look at the result:
{
"ActTime" : 1510530122,
"ServerTime" : "2017-11-12 15:42:02",
"Sunrise" : "06:48",
"Sunset" : "16:59",
"result" : [
{
"AddjMulti" : 1.0,
"AddjMulti2" : 1.0,
"AddjValue" : 0.0,
"AddjValue2" : 0.0,
"BatteryLevel" : 100,
"CustomImage" : 0,
"Data" : "74.3 F, 41 %",
"Description" : "",
"DewPoint" : "49.10",
"Favorite" : 0,
"HardwareID" : 6,
"HardwareName" : "RaZberry",
"HardwareType" : "OpenZWave USB",
"HardwareTypeVal" : 21,
"HaveTimeout" : false,
"Humidity" : 41,
"HumidityStatus" : "Comfortable",
"ID" : "0201",
"LastUpdate" : "2017-11-12 15:41:28",
"Name" : "Office Temperature",
"Notifications" : "false",
"PlanID" : "0",
"PlanIDs" : [ 0 ],
"Protected" : false,
"ShowNotifications" : true,
"SignalLevel" : "-",
"SubType" : "WTGR800",
"Temp" : 74.299999999999997,
"Timers" : "false",
"Type" : "Temp + Humidity",
"TypeImg" : "temperature",
"Unit" : 0,
"Used" : 1,
"XOffset" : "0",
"YOffset" : "0",
"idx" : "65"
}
]
}
This gives us the basis for what we’ll need to do!
A Few Bits About the API
There are a few things that are a bit of pain with the API that you can see from the above:
- Domoticz treats everything in Celsius even if the devices return in Fahrenheit. As a result, there can be multiple conversions back and forth which can result in floating-point rounding troubles, as you can see above which is 74.29999… instead of 74.3 as the device had returned. I haven’t delved into every API or component return, but I assume this is more than just Celsius/Fahrenheit but in other metric/English conversions.
- The devices tend to return multiple values in the same device and those may have varying types. For example, notice the “Data” above is “74.3F, 41 %” which is Temperature + Humidity combined in a single string while we also get a separate “Humidity” value (integer) and a separate “Temp” value (floating point) and a separate “DewPoint” value (floating point, but JSON typed to string and not in the Data block). Other values you’d expect to be numbers (“SignalLevel”) may return strings (e.g. “-“) if no value is provided. “true/false” may return in strings (as you can see in “Timers”) etc.
- A continuation of #2, if every type of device had all of the values it was returning in a separate JSON object that was at least standardized against itself, then this would just be mildly annoying, but alas that’s not the case. For example, energy monitoring devices only seem to return their data in the “Data” string (no “Amperage” or “Wattage” attribute)
Also, Domoticz doesn’t seem to have any internal scheduled-scripting (that I’ve found) with any frequency higher than once every minute, so we have to poll this API to get the data off the device instead of using those lovely internals. So this is what we have to work around. Let’s get started!
Using Elasticsearch’s Ingest Nodes
Elasticsearch has Ingest Nodes, which allow us to transform this type of data directly in the Elasticsearch cluster without setting up any external dependencies. First, we’ll need to create an ingest pipeline:
I set up the following:
curl -XPUT "https://MY_ES_USER:MY_ES_PASSWORD@MY_ES_HOST:MY_ES_PORT/_ingest/pipeline/domoticz" -H 'Content-Type: application/json' -d'
{
"description": "Converts domoticz API to ES",
"processors": [
{
"date": {
"field": "LastUpdate",
"target_field": "timestamp",
"formats": [
"yyyy-MM-dd HH:mm:ss"
],
"timezone": "America/Los_Angeles"
},
"remove": {
"field": "LastUpdate"
},
"set": {
"field": "IngestTime",
"value": "{{_ingest.timestamp}}"
},
"rename": {
"field": "Data",
"target_field": "DataString"
},
"script": {
"lang": "painless",
"source": "if (ctx.Type == \"Current\") { ctx.Current = ctx.DataString.replace(\" A\",\"\") } else if (ctx.Type == \"General\") { if (ctx.Usage != null) { ctx.Wattage = ctx.Usage.replace(\" Watt\",\"\"); } else if (ctx.CounterToday != null) { ctx.EnergyToday = ctx.CounterToday.replace(\" kWh\",\"\"); } else if (ctx.DataString.endsWith(\" SNR\")) { ctx.SNR = ctx.DataString.replace(\" SNR\",\"\"); } } else if (ctx.Type == \"Lux\") { ctx.Lux = ctx.DataString.replace(\" Lux\",\"\") }"
},
"date_index_name": {
"field": "IngestTime",
"index_name_format": "yyyy-MM-dd",
"index_name_prefix": "devices-",
"date_rounding": "d"
}
}
]
}'
This does a few things: it moves the “LastUpdate” field into a “timestamp” field, sets an “IngestTime” field to the time the document was received, adds “Current,” “Wattage,” “EnergyToday,” “Lux,” and “SNR” named fields if the based on the “Type” and the Data field string, and finally sets up a date-based index for the values.
Great. Now we have an ingest pipeline that can process the results of each of the devices. This will place the data into an index named devices-<date>. We’ll also want an index template to store the data to:
curl -XPUT "https://MY_ES_USER:MY_ES_PASSWORD@MY_ES_HOST:MY_ES_PORT/_template/domoticz" -H 'Content-Type: application/json' -d'
{
"domoticz": {
"order": 0,
"template": "devices-*",
"settings": {
"index": {
"number_of_shards": "1"
}
},
"mappings": {
"devices": {
"properties": {
"BatteryLevel": {
"type": "scaled_float",
"scaling_factor": 100
},
"DataString": {
"type": "text"
},
"HardwareName": {
"type": "keyword"
},
"IngestTime": {
"type": "date"
},
"Level": {
"type": "double"
},
"LevelInt": {
"type": "scaled_float",
"scaling_factor": 100
},
"Name": {
"type": "text"
},
"Status": {
"type": "keyword"
},
"Type": {
"type": "keyword"
},
"Current": {
"type": "double"
},
"DewPoint": {
"type": "double"
},
"Gust": {
"type": "double"
},
"EnergyToday": {
"type": "double"
},
"Rain": {
"type": "double"
},
"RainRate": {
"type": "double"
},
"Speed": {
"type": "double"
},
"Chill": {
"type": "double"
},
"UVI": {
"type": "double"
},
"Direction": {
"type": "double"
},
"Lux": {
"type": "integer"
},
"Wattage": {
"type": "double"
},
"CounterToday": {
"type": "text"
},
"HumidityStatus": {
"type": "text"
},
"Usage": {
"type": "text"
},
"idx": {
"type": "integer"
},
"timestamp": {
"type": "date"
}
}
}
}
}
}'
Most of this template is not purely “necessary” as Elasticsearch would handle the data correctly in most if not all of these field values, but it sets up efficient mappings and makes sure we don’t have mistakes in our data model.
Uploading the API to Elasticsearch
We need to push the data to Elasticsearch at some regular interval. I’ve used jq to do some lightweight filtering and reformatting of the data and then ultimately push it up to Elasticsearch. We can add something like the following to the crontab to initiate it on boot.
#!/bin/bash
while true
do curl "http://127.0.0.1:8080/json.htm?type=devices&filter=all&used=true&order=Name" | jq -c '.result[] | with_entries(select( .key as $k | $keys | index($k)))' --arg keys '["idx","HardwareName","Name","BatteryLevel","Gust","Direction","Chill","Speed","Temp","UVI","Rain","RainRate","Visibility","Radiation","Level","LevelInt","Status","Humidity","DewPoint","HumidityStatus","CounterToday","Usage","Voltage","Type","Data","LastUpdate"]' | sed -e 's/^/{ "index" : { "_index" : "devices", "_type" : "devices" } }\
/' > /tmp/devices_bulk.json && curl -H 'Content-type: application/json' -XPOST 'https://MY_ES_USER@MY_ES_PASSWORD:MY_ES_PORT/_bulk?pipeline=domoticz' --data-binary @/tmp/devices_bulk.json > /dev/null && sleep 5
done
This script does a few things: it selects the relevant pieces of information that I want indexed (Temperature, HardwareName, etc) from the response and then formats a _bulk index request to Elasticsearch and then POSTs it.
In my case, I’ve used an Elastic Cloud instance to push the data to. This allows me to externalize the data in a secure fashion and make it available no matter where I’m at, but only to me. It also lets me fire off alerts, e.g. if one of my security devices goes off and I’m not home.
Drop this bash script into /opt/domoticz/scripts/ (where I’ve installed Domoticz) and in crontab, add:
@reboot bash /opt/domoticz/scripts/uploadES.sh
Setting Up Kibana
The two biggest quirks for setting up Kibana with this data are
- Domoticz reports the time as the last time the device had an update, but for most dashboards what you really want is to see the state of all devices at a given point in time. This means we really want the ingest time, not the device update time (hence adding that to the ingest template).
- Using the device IDs for multi-valued documents with similar types. For example, showing all motion sensors or door sensors on the same graph. For this, there are a few approaches, but I used the sub aggregation to split the charts by filtering for specific device IDs (“idx” in the data). This looks like the following in Kibana:
Example Dashboard
Once we tie it all together, we get quite nice graphs!
This makes for a nice dashboard! We can see the inside and outside temperatures, when there was motion in various spaces (in red), when doors were opened (in red), and when devices were turned on and off. You can even see in the “office” and “living room” temperature charts when the heat kicked on in my apartment. We can fire off alerts for security (e.g. “front door is opened but nobody’s home”) or
Future Work
You’ll likely see some of the following in future posts or as future work from me:
- The bottom right graph hides additional detail of something I’ve built out, which is tracking the wifi strength of connected/roaming devices. I think this is quite useful as a virtual presence detection device. I’ll add more detail in the future about how I’ve done this.
- I’m adding more devices. One of the things I really like about having the data in a place like this is I can start to gather and visualize other external inputs to the system (e.g. my phone’s sensors and location). I’ve got some code written for this already.
- I’d like to make a native Elasticsearch exporter and timer for Domoticz and fix some of this API weirdness, but I’m not terribly familiar with the Domoticz C++ codebase yet to do this.
- I think my habits could be fairly well modeled with Elastic’s machine learning. I’d like to build/use machine learning models on top of this data to automatically flag and alert on anomalies in the data rather than hard-code rules.
