COLLECTING DATA FROM A PRIVATE LORAWAN SENSOR NETWORK INTO ELASTIC
来源:互联网 发布:c语言中char是几位 编辑:程序博客网 时间:2024/06/05 04:23
| Reply
Introduction to LoRaWAN and ELK
Why LoRaWAN, and what makes it different from other types of low power consumption, high range wireless protocols like ZigBee, Z-Wave, etc … ?
LoRa is a wireless modulation for long-range, low-power, low-data-rate applications developed by Semtech. The main features of this technology are the big amount of devices that can connect to one network and the relatively big range that can be covered with one LoRa router. One gateway can coordinate around 20’000 nodes in a range of 10–30km. It’s a very flexible protocol and allows the developers build various types of network architectures according to the demand of the client. The general description of the LoRaWAN protocol together with a small tutorial are available in myprevious post.
What is the ELK stack, and why use it with LoRaWAN?
In the figure above, you can see a simplified model of what a typical LoRaWAN network looks like.
As you can see, the data from the LoRa endpoints, has to go through several devices before it reaches the back end application. Nowadays there are a lot of tools that would allow us to gather and manipulate the data. A very good solution is the ELK stack which consists of Elasticsearch, Logstash and Kibana; these three tools allow to gather, store and analyze big amounts of data. More information and details can be found on the official website: https://www.elastic.co/.
The network architecture
Mainly, in this post I will describe how to set up a private LoRa network, and will provide details about each module between the sensor data acquisition and the data analysis in Kibana. I will describe two possible architectures: one which uses a free public cloud server, and another one which involves a private server or PC.
The hardware that I am using:
- 2x Sodaq Mbili data acquisition boards
- 2x RN2483 LoRabee modules
- ic880a-spi LoRa gateway
- Raspberry Pi 2
- PC with Ubuntu
- different types of analog sensors
Below you can see a full model of the network architecture using the public cloud server of The Things Network:
Setting up the nodes
The Sodaq Mbili with the RN2483 are the LoRa end devices, which can read sensor data and send it to the gateway. A tutorial on how to set them up is available in my previous post (link in the introduction) or on the official support page: http://support.sodaq.com/mbili/.
For this tutorial I created a demo project which reads data from an analog sensor connected to the A0 port of the Sodaq and sends the data to the RN2483 LoRaBee. The program can be uploaded using the Arduino IDE; before doing that it’s important to change the constant called devAddr[4] to your own node addresses. In order to reserve addresses for the TTN cloud, you should register your name at: http://thethingsnetwork.org/wiki/AddressSpace. Trifork Eindhoven has reserved the address range 02:D1:DC:XX . In this example I am using node address 02D1DC09
Setting up the gateway
In this example I am using the ic880a-spi LoRa gateway from IMST GmbH, which is one of the cheapest and practical solutions for setting up a private gateway. The documentation can be found at: http://www.wireless-solutions.de/products/long-range-radio/ic880a.
The first thing is to connect the Raspberry Pi to the gateway and to the power supply. In the table below you can see the pin connection:
It is recommended to use Raspbian Jessie Lite with no desktop graphical interface. The SPI interface has to be enabled; this can be done in the “raspi-config” under the “Advanced” sub menu.
The next step is to check if the connection was done correctly; Some testing scripts can be found on git at: https://github.com/LoRa-net/LoRa_gateway. Download the files on the Raspberry Pi and open the folder util_tx_test
. This script will check if the Raspberry Pi can connect to the gateway and will send test RF data packets. Run the following command in the Rpi terminal:
sudo ./util_tx_test – r 1257 -f 866.5
If the pins were connected properly, you should get the following response:
Connecting the LoRa gateway to the TTN network
The gateway can either store the data locally, send it to a private server, or send it to The Things Network (TTN) public network. First I will show how to connect to the free TTN service.
There are two ways, you can either download the basic packet forwarder and install it on the rpi:https://github.com/LoRa-net/packet_forwarder/tree/master/basic_pkt_fwd and configure it manually, or you can simply download the installer provided by TTN at: https://github.com/ttn-zh/ic880a-gateway/tree/spi. Simply run the ./install.sh api
and then follow the instructions on the terminal. Make sure that the Rpi is connected to the internet before running the installation.
After the installation the Rpi will reboot, and the gateway service will always run in the background. At this point we created a public LoRaWAN gateway, that can be used by any node in the range; later in this post will explain how to make the network private and secure.
In order to check if the gateway is online, connect the Sodaq Mbili to power and let it send data to the network. Go to http://thethingsnetwork.org/api/v0/nodes/<NODE devaddr>/?format=json where devaddr is the address that you chose when setting up the node. Here you should find the messages sent by your node, as in the example below:
If there is no data on the URL, you can try running the packet_forwarder script manually.
You can find the script under /opt/ttn-gateway/bin
. Simply run ./start.sh
Getting the data into elasticsearch
The next step it to get the JSON data from the URL and save it on a personal server or computer. First, install elasticsearch. An installation guide can be found at:https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
and kibana: https://www.elastic.co/guide/en/kibana/current/setup.html
Create a new index in elasticsearch for storing the data from the nodes:
curl -XPUT 'http://localhst:9200/LoRa_messages/'
and check if the index was created successfully with
curl 'localhost:9200/_cat/indices?v'
There are a lot of ways in which the JSON file can be imported from that URL and put into elasticsearch. For example, you can make a script like this one:
1
import
urllib, json
2
import
elasticsearch
3
import
time<
/
code>
4
5
#data = json.loads(response.read())
6
7
f
=
open
(
"ttnsample.json"
,
"r"
)
8
#data = f.read()
9
10
es
=
elasticsearch.Elasticsearch()
11
12
def
lemain():
13
14
while
True
:
15
url
=
"http://thethingsnetwork.org/api/v0/nodes/02D1DC09/"
16
response
=
urllib.urlopen(url)
17
jsonobject
=
json.loads(response.read())
18
for
x
in
range
(
0
,
50
):
19
es.index(index
=
'lora_messages'
, doc_type
=
'node_message'
,body
=
jsonobject[x])
20
time.sleep(
15
)
21
22
if
__name__
=
=
"__main__"
:
23
lemain();
As in the previous steps, make sure to use your own devaddr in the url.
Visualising the data in Kibana
The last step is to make the data readable and easy to understand. Make sure both Kibana and elasticsearch are running. Open the browser and go to 127.0.0.1:5601
. Add the new indexLoRa_message*
. Kibana should auto detect the time filed. Below you can see how should the node data look like:
Each message contains the following fields :
- “node_eui” – the node’s devaddr
- “datarate”
- “data_plain” – the data sent from the node
- “gateway_eui” – the unique mac address of the gateway
- “data_raw” – data before decryption
- “snr” – Signal to noise ratio (it indicates the quality of the signal)
- “time” – the time at which the packet was received
- “rssi” – Received signal strength indication
- “data_json” – the same data, but in a number format.
- “data” – data in chars
- “frequency”
Kibana allows to read this data in various ways. For this example I created three simple line charts that would show the sensor data, the signal to noise ration and RSSI over time.
In here you can see the data from a sound sensor in the Trifork office in Eindhoven over an hour, together with the signal to noise ratio and RSSI. The dashboard can be customized in any way possible, and multiple nodes can be added.
Setting up a personal LoRaWAN network without the open TTN cloud
The main benefit of using the free TTN cloud server is that you don’t need a personal server with static IP address that has to run 24/7. On the other hand, by using their server, the data becomes public, and can be accessed by anyone. In addition, anyone can use the same devaddr for the nodes and fill your buffer with unnecessary data.
In order to redirect your data from the gateway to a personal server you can use Logstash, which can parse and filter the messages. The network architecture in this case would look like this:
Setting up the gateway
In order to configure the gateway, you have to access the local_conf.json
file which can be fond in the /opt/ttn-gateway/bin
folder. You have to change the parameters under the index “server” to the IP address of your server’s address and the UTP ports used for sending the data to/from the gateway, as in the example below:
And the run the ./start.sh
from the same folder. Now the gateway is sending JSON packages to your server.
Setting up Logstash
The next step is to capture the data from the UDP port and add it to Elasticsearch. In the logstash-x.x.x
folder, create a config file, lora.conf
for example. In this file you have to specify where does the data come from, how should it be filtered, and where it should be sent to. The following example is the minimal setup for simply redirecting all the data from the UDP port to elasticsearch:
As you can see there is no filter applied, which means that all the information will be redirected. The gateway sends json packages about the gateway’s status, periodical acknowledgements and the data received from the nodes. More information can be found in the PROTOCOL.TXT file which can be found in the Rpi’s folder /opt/ttn_gateway/packet_forwarder
.
In order to get only the node’s data, you need to apply a filter in logstash, or choose a specific search index in Kibana. In order to visualize the data in kibana, simply choose logstash*
as the search index.
The last step is to decode the node data. In the previous setup, the data was decoded by TTN framework, but in this case you have to decode it using a base64 decoder. The encryption keys can be set up in the Arduino code by changing the variables netwskey
and appskey
. The decryption can be done with logstash using the cipher plugin.
- COLLECTING DATA FROM A PRIVATE LORAWAN SENSOR NETWORK INTO ELASTIC
- Mysql select data from one table into a new one
- How To Load CLOB Data from a File into a CLOB column using PL/SQL
- Read & Return Data From Files Into PHP
- C++ Read Data from File into Struct
- fcn——free connect your private network from anywhere
- LORAWAN A B C
- Reading Text Data Into A Unity Game
- Get IIS Errors Into Elastic
- A beginner’s guide to collecting Twitter data (and a bit of web scraping)
- Insert raw data into a binary data field with ADO
- convert the Data from Table into Insert Sentence.
- Dump data into MySQL from Excel with PHP
- HOWTO setup a small server OpenVPN (Virtual Private Network)
- Look into "A Neural Network in 11 lines of Python"
- From a compilation problem: A glance into STL
- Protect Your Windows Network : From Perimeter to Data
- Permission problem on a derived private method from public method
- ConvenientBanner
- 现有‘abcdefghijkl’12个字符,将其所有的排列按字典序进行排序,给出任意一组排列,说出这租排列在所有排列中是第几小的
- 洛谷 P1372 又是毕业季I 数论
- Python科学计算环境推荐——Anaconda
- static和final修饰的变量之间的区别
- COLLECTING DATA FROM A PRIVATE LORAWAN SENSOR NETWORK INTO ELASTIC
- 【1】-4 Semaphores
- 第九周--猴子选大王
- python2.7安装requests模块
- Leetcode 206 Reverse Linked List
- 前端知识(2)
- hdoj 1171 Big Event in HDU(母函数/多重背包)
- JS版设计模式--Singleton
- 微信公众号图片防盗链机制的应对办法^-^