Category Archives: IoT

Why not just use NiFi and MiNiFi instead of rt-ai Edge?

Any time I start a project I always wonder if I am just reinventing the wheel. After all, there is so much software out there (on GitHub and others)¬† that almost everything already exists in some form. The most obvious analog to rt-ai Edge is Apache NiFi and Apache MiNiFi. NiFi provides a very rich environment of processor blocks and great tools for joining them together to create stream processing pipelines. However, there are some characteristics of NiFi that I don’t particularly like. One is the reliance on the JVM and the consequent garbage collection issues that mess up latency guarantees. Tuning a NiFi installation can be a bit tricky – check here for example. However, many of these things are the price that is inevitably paid for having such a rich environment.

rt-ai Edge was designed to be a much simpler and lower overhead way of creating flexible stream processing pipelines in edge processors with low latency connections and no garbage collection issues. That isn’t to say that an rt-ai Edge pipeline module could not be written using a managed memory language if desired (it certainly could) but instead that the infrastructure does not suffer from this problem.

In fact, rt-ai Edge and NiFi can play together extremely well. rt-ai Edge is ideal at the edge, NiFi is ideal at the core. While MiNiFi is the NiFi solution for embedded and edge processors, rt-ai Edge can either replace or work with MiNiFi to feed into a NiFi core. So maybe it’s not a case of reinventing the wheel so much as making the wheel more effective.


rt-ai: real time stream processing and inference at the edge enables intelligent IoT

The “rt” part of rt-ai doesn’t just stand for “richardstech” for a change, it also stands for “real-time”. Real-time inference at the edge will allow decision making in the local loop with low latency and no dependence on the cloud. rt-ai includes a flexible and intuitive infrastructure for joining together stream processing pipelines in distributed, restricted processing power environments. It is very easy for anyone to add new pipeline elements that fully integrate with rt-ai pipelines. This leverages some of the concepts originally prototyped in rtndf while other parts of the rt-ai infrastructure have been in 24/7 use for several years, proving their intrinsic reliability.

Edge processing and control is essential if there is to be scalable use of intelligent IoT. I believe that dumb IoT, where everything has to be sent to a cloud service for processing, is a broken and unscalable model. The bandwidth requirements alone of sending all the data back to a central point will rapidly become unworkable. Latency guarantees are difficult to impossible in this model. Two advantages of rt-ai (keeping raw data at the edge where it belongs and only upstreaming salient information to the cloud along with minimizing required CPU cycles in power constrained environments) are the keys to scalable intelligent IoT.

DroNet – flying a drone using data from cars and bikes

Fascinating video about a system that teaches a drone to fly around urban environments using data from cars and bikes as training data. There’s a paper here and code here. It’s a great example of leveraging CNNs in embedded environments. I believe that moving AI and ML to the edge and ultimately into devices such as IoT sensors is going to be very important. Having dumb sensor networks and edge devices just means that an enormous amount of worthless data has to be transferred into the cloud for processing. Instead, if the edge devices can perform extensive semantic mining of the raw data, only highly salient information needs to be communicated back to the core, massively reducing bandwidth requirements and also allowing low latency decision making at the edge.

Take as a trivial example a system of cameras that read vehicle license plates. One solution would be to send the raw video back to somewhere for license number extraction. Alternately, if the cameras themselves could extract the data, then only the recognized numbers and letters need to be transferred, along with possibly an image of the plate. That’s a massive bandwidth saving over sending constant compressed video. Even more interesting would be edge systems that can perform unsupervised learning to optimize performance, all moving towards eliminating noise and recognizing what’s important without extensive human oversight.

The Bitcoin Lightning Network

Just started finding out about the Lightning system that is designed to avoid some of the key inefficiencies of Bitcoin. I’ve always thought the excitement about Bitcoin to be a bit strange given its seven transactions per second throughput limit, absurd mining energy requirements, the need to effectively bribe miners with a fee in order to get a transaction included in the blockchain and the ever increasing (and already quite large) blockchain itself.

As described here, Lightning helps solve the scaling problem by implementing a network of micropayment channels that bypass the blockchain. Something that I have been interested in is how to use blockchain technology in large IoT networks that can generate large amounts of data in real time. The benefit would be an immutable (append-only) record of everything that happened that could be relied upon for accuracy and not amenable to after the fact modification. This record could be of evidential quality. Whether Lightning (or similar technology) can help with this is the question.

Years ago I was working on a tamper-proof video system for surveillance cameras using steganography – basically every frame included embedded overlapping error correcting codes that could survive compression and indicate on a grid where a frame had been modified by checking the syndromes. By using the frame timestamp as the data in the code, encrypted with a private key embedded in the source camera, completeness of the video record could also be determined. What’s interesting is whether blockchain technology can be similarly leveraged to solve this and related problems.

Project Sopris

Interesting piece here about Microsoft’s Project Sopris. There’s also an interesting paper giving background to the approach. It’s become a thing when people talk about IoT to slam its security but, in the end, this is just another engineering problem to solve. If IoT class processors were available with built-in and easy to use (without being a security expert) security features, I’d like to think that people would use them and this barrier to implementation would be removed permanently.

The trouble with temperature sensors

Working with the Bosch XDK reminded me that temperature sensing seems like such an obvious concept but it is actually very tough to do and get correct results. The prototype above was something I tried to do in a startup a few years ago, back when this kind of thing was all the rage. It combined motion sensing, the usual environmental sensors including air quality and could have a webcam attached if you wanted video coverage of the space also.

In this photo of the interior you can see my attempt at getting reasonable results from the temperature sensor by keeping the power and ground planes away from the sensor – the small black chip on the right of the photo. Trouble is, the pcb’s FR-4 still conducts heat, as do the remaining copper traces to the chip. Various other attempts followed included cutting a slot through the FR-4 and isolating the air above the rest of the circuit board from the sensor. This is an example:SensorBoard1.jpegAnd this is a thermistor design (with some additional wireless hardware):


In the end, the only solution was to use a thermistor attached by wires that could be kept some distance from the main circuitry. Or, just having all the very low power sensors completely removed from the processor.

The Raspberry Pi Sense HAT suffers from this problem as it is right above the Pi’s processor, as does the Bosch XDK¬†itself. Actually I am not aware of any other really good solution apart from the one where a cable is used to separate the sensor board completely from the processor controlling it (which might work for the Sense HAT although I have not tried that.

Apache Nifi, MQTT and the Bosch XDK

Turned out to be really easy to drive Apache NiFi from the Bosch XDK sensor node via MQTT. The XDK actually has an MQTT example project that does pretty much everything for you – it’s the MQTT Paho demo on this page. I am using it with the mosquitto broker on Ubuntu.

The screen capture shows the output of a simple display application that subscribes to the MQTT topic and graphs the sensor values. This is actually a version of sensorview in the rtndf GitHub repo. It doesn’t display the gyro or magnetometer data from the XDK but it displays the rest of the data.

On the Apache NiFi side of things, I am using the ConsumeMQTT processor. This is an example of a data record recovered from the provenance data.

One day, if I am feeling really keen, I could port the old RTIMULib2 software onto the XDK and fill out the sensorpose field in the message with something other than zeroes.