(NO) Messaging On Windows 10 IoT Core

I am working on a distributed application to allow me insights and control of the devices in the house. I want to see, when PCs are on and I want to control some lights. So I have some rather different messaging patterns.

  • Telemetry: The different clients regularly tell the server about their state (client to server: „I am on!“ … no response)
  • Commands: The server sends a command to a client and immediate action expected (server to client: „Turn that light on!“ … „done!“)
  • Request/Reply: The client asks the server something and expects an immediate response. (client to server: „I lost power shortly, was that light on or off?“ … „On!“)

So, is HTTP the answer here? Request/Reply fits perfectly to HTTP. Telemetry could be modeled as Request/Reply with some kind of an empty return message, so this would be good as well. The problem is the command pattern. To implement the command pattern over HTTP one would have to bend the protocol. There are ways to do this like long polling.

And did I say, my client is a RaspberryPI 2 running on Windows 10 IoT Core?

AMQP

So I like actual messaging more then protocol bending and that is why I was turning to AMQP. AMQP is the advanced messaging protocol and I believe it was invented and specified to support limited devices with small footprints to allow IoT use cases. There exists a single libary for Universal Windows Platform (UWP). It is amqpnetlite. The library has reduced functionality in an UWP environment. For example you can only connect to other endpoints – no one can connect to you. Fine for me. Turns out, there is no easy way for the server to reuse an established connection of a client to send more messages. The leading project maintainer was kind enough to add some methods and another example for my use case but it still feels too hacky for me. Also I am missing lifetime events of the connection or clear documentation on what happens, when the host is not reachable or disconnected or anything. I was disappointed and decided to replace all I had of AMQP in my code.

SignalR

The next solution for my messaging needs was SignalR. It started es a tool to allow more interactive web sites with push messages from the server. It basically is a http bender under the cover 🙂 The initial intention of SignalR was to have a C# server talking to a JavaScript client but soon there was also a .Net client. Turns out the .Net client is compatible with UWP so I tried it. I got all my messaging needs working. There are also rich lifetime events about connection, reconnection, disconnected and proper documentation. This could be nice except it is not. I found an issue with the client implementation. If the client does not use the connection for at least 60 seconds, the next messaging call simply throws an Exception and does never recover that state. My expected behaviour of the client would be to notice the disconnect, reconnect and then retransmit the message and never let an Exception slip through to the calling code. The last commit for the project was 8 month ago and there are several pull requests and issues waiting but nothing happens. This project is dead?!

ZeroMQ

I am using ZeroMQ in a project at work and I am really starting to love it. It is beautiful and it really just works. It has a huge guide which tells you everything you need to know. And it has implementations in soooo much programming languages except UWP. Its a shame. At work, I am using NetMQ as the implementation of choice. It is a .Net port of ZeroMQ. I used the complete evening to take a look at the possibility of porting it to UWP but from my point of view, it would require major refactoring of NetMQ and the end result would also not be able to listen but only to connect to other hosts. This could still be fine. There actually exists an small incomplete port of NetMQ to UWP called ZMTP.net but it currently misses some features, as I discovered.

The question now is, how should I continue?

I could (maybe) work around the SignalR bug but I do not like that. Using SignalR for my messaging needs was not the best choice from the start and I should remove it again. I could help to implement more ZMTP functionality and use that in my project.

But what I really think is that Microsoft replaced and removed too much of the basic networking stuff in its UWP environment and that caused every developer of a messaging library to decide whether they invest in UWP or simply to not support it.

It feels like there is also no demand for messaging in UWP or UWP in general?

I am really considering to replace the Windows 10 IoT Core with something more supported like Raspbian? I could either use mono or I would implement the client in another language like python or nodejs?

Die Suche nach dem nächsten Auto

Bevor ich mich für mein aktuelles Auto entschieden habe, habe ich so einige Autos Probe gefahren. Ich war ja mit dem Scirocco sehr zufrieden, aber wenn man ehrlich ist, ist er für eine dreiköpfige Familie einfach nicht mehr das richtige Auto. Wir brauchten also was familientaugliches, der Spaß sollte aber jetzt auch nicht zu kurz kommen und ich als Techie mag ja die ganzen technischen Spielereien im Auto. Hier was Elektrisches, da ein Assistent. Das ist genau das Richtige für mich. Nun muss man die Auswahl irgendwie starten und wie es so kam, hatte ich eine Mercedes C Klasse als Mietwagen (obwohl wir von der Firma eigentlich nur Golfklasse hätten mieten dürfen…ohoh). Das Auto hatte mir gut gefallen, also schlugen wir bei Mercedes auf. Die C-Klasse sollte neu aufgelegt werden und war nicht mehr neu zu bestellen, wir schauten uns also die E-Klasse an. Ich war fasziniert von der elektrischen Heckklappe, die wir schon im Autohaus mehrfach öffneten. Nach einer Weile im Gespräch mit der Verkäuferin hatten wir eine mögliche Konfiguration fertig. Da die E-Klasse an sich teurer ist als die C-Klasse, blieb nur noch Luft für den 2 Liter Diesel. Wir wollten eigentlich einen Kombi haben, für die Probefahrt erhielten wir allerdings eine Limousine. Hier ist ein kleines Video, wie wir mit dem Auto das automatische Einparken testen:

Der Mercedes war okay, preislich aber am oberen Limit und wir schauten deshalb mal bei VW vorbei. Der Passat, mit absolut vergleichbarer Konfiguration war ca. 15.000€ günstiger. Da muss Oma lange für stricken! Der Passat ist im Fahrgefühl, im Platz, in der elektrischen Heckklappe und sogar in der Analoguhr im Innenraum der E-Klasse ebenbürtig, so dass ich die Preisdifferenz einfach nicht gerechtfertigt finde. VW bietet allerdings im Passat keine größeren Motoren an, so dass hier auch der 2Liter Diesel drin gewesen wäre, aber aufgrund der Größe und des Gewichts, vermisste ich die Spritzigkeit des Sciroccos.

BMW hat größere Motoren, ich analysierte das also. Wegen der Preisgestaltung liegt der 5er mit der E-Klasse gleich auf und daher war dort ein größerer Motor nicht mehr machbar, ich sah mir also den 3er genauer an. Eine vergleichbare Menge an Extras, aber größerer Motor. Zuerst wollte ich einen 328i und darauf hatten wir einen Vorführwagen mit dem 335i drin. Ein wirklich schönes Auto, aber ich bin mit diesem Motor nicht warm geworden. Der Klang ist, als würde er gleich zerreißen und gleichzeitig kommt man trotzdem nicht vom Fleck. Ich war enttäuscht, aber hatte noch nicht aufgegeben. Ich entschied mich, einen Diesel Probe zu fahren. Wir hatten einen 330d mit 6 Zylindern. Der Klang ist genial, er klingt einfach nur nach purer Kraft und Fahrspaß und das geniale ist, dass das Fahrzeug sich genauso spritzig wie der Scirocco fährt trotz gewachsenem Platzangebot. Die Entscheidung war gefallen!

BMW Abholung (unkenntlich)

Seit dem haben wir das Auto jetzt etwas über ein Jahr, der erste Ölwechsel ist durch, sonst war bisher nichts gewesen und ich bin absolut zufrieden und glücklich. Lediglich ein klitzekleiner Teil in mir schielt mittlerweile auf einen Tesla.

Dieser Beitrag wird unterstützt durch Autoersatzteile.de. Schau doch mal auf autoersatzteile.de/elektrik/mercedes-benz vorbei und sieh dir die günstigen Angebote an.

PowerShell is case-sensitive!

Did you know PowerShell is case-sensitive? I did not know this until recently. Together with a file system which is not case sensitive, this does not make much sense to me:powershell_casesensitive_folder

But it becomes even worse together with a git repository, because git becomes somehow confused by it, as you can see here:

git_status_confused

As you can see, there is a modified file. If I now change into the assets folder but with differently cased letters, git is thinking that the change is actually in another folder. This comes to no surprise if you know the roots of git.

By the way, the old command shell does not have this weird behavior:cmd_is_not_confused

Enabling HSTS for Fabse.net

Today, I read about HTTP strict transport security on troyhunt.com and I decided to enable it for fabse.net as well. After running this complete website over HTTPS for some time now, I decided to go the next step and to (technically) tell all browsers that this website will only ever work over a secure connection. Hanselman has a great blog post on how to do this correctly on IIS.

At last, I also added my site to to the official preload list to get it directly included in future browser releases.

Installing Windows 10 IoT Core using a Hyper-V Windows 10 machine

So I finally came around to play with my raspberry pi 2 a little bit and my first objective was to install Windows 10 IoT core on it. The full getting started tutorial has all the needed steps written.

After registering and accepting the EULAs (important! you will simply not see the downloads otherwise!) I noticed the first obstacle for me. They require you to copy the data to sd card using a Windows 10 hardware machine. I have only one Windows 10 machine which is running in Hyper-V.

So I accepted the challenge 🙂 This article suggested, it may at all be possible to copy the image to the sd card using Windows 7 or Windows 8.

I made a new folder on my Desktop PC (D:\win10-on-rpi) and extracted the flash image there (Flash.ffu). Then I copied some data from the Windows machine into this folder:

  • Dism.exe (from C:\Windows\System32)
  • DismApi.dll (from C:\Windows\System32)
  • All content inside C:\Windows\System32\Dism

Important is to copy the content from the Dism folder directly into folder and not into a subfolder. This is not intuitive, but works.

windows10-iot-raspberry-pi2-flash-folder

At this point I opened a powershell command prompt as an administrator and executed the Dism-command.

D:\win10-on-rpi> ./dism.exe /Apply-Image /ImageFile:Flash.ffu /ApplyDrive:\\.\PhysicalDrive7 /SkipPlatformCheck

windows10-iot-raspberry-pi2-flash-folder-dism-completed

Building a RAID 0 with 2 SSD Drives (and showing off)

My desktop PC at home is utilizing a 256GB SSD as its system drive for quite some time now and I enjoyed it. I measured the speed of it using HD Tune:SAMSUNG_SSD_830_SeriesIn the past, I had used another SSD of the same size for a fast external USB 3 drive to host a virtual machine but I did not needed the VM anymore and so the drive was unused now. I installed it into my PC as an internal drive and measured its speed (of course!):

Samsung_SSD_840_EVO_250GNext thing was backing up the complete system drive to network storage.Before I actually deleted the data from the first drive, I tried a full restore to the second drive, which took some time but worked well.

Then I felt comfortable enough to go ahead and to setup a RAID 0 with both drives. This was the time, were I noticed my first SSD was not connected to the Intel RAID chipset but to the secondary SATA chip on my board. A Marvel-something. I switched some drives around until I had both SSDs connected to the RAID chipset. Setting up a RAID 0 is straightforward. Next thing was doing the restore and resize of the system partition to the new RAID volume. And I measured it again:

Raid_0 (Samsung 930+940)This measured speed looks very impressive and it is significantly higher than the speed of both single drives combined. It may has to do with me using the Marvel chipset in the first measurement. Maybe the INTEL chipset is doing some performance magic on the cover as well? I do not know and I frankly do not care.

For what it’s worth, the actual perceived performance gain compared to a single SSD drive is next to nothing but it is definitely not slower than before and the system partition has now much more free space than before. The additional free space without partition/drive borders in between it is the real gain here for me.

Pitfalls of String to Number/Date Conversions

Having a string and converting it to a Number is easy, isn’t it?

Convert.ToDouble(string)

But what does it actually print on the console? As usually, that depends! It depends on your own user profile and its culture. Are you using a culture, where “.” (dot) is a decimal divider or a thousands separator?

So the example above will print “1.23” if you have an English culture setting or it will print “123” if you have a German culture. So that is most probably not what you want. What you want is consistency and predictability.

As a kind of an anti pattern (which I saw already way to often) you could replace dot with comma or comma with dot depending on what you want and you may get it. I am not showing samples on purpose here, but the key take-away is to not use string replacings for this purpose!

So what should you do instead? .Net has a good implementation of Cultures so it know very well how to handle all of this, the only thing you have to actively do is telling it, what culture it should use. If you do not provide a culture, it implicitly uses the CurrentCulture. So, the first example written in a more explicitly way would be:Explicit culture usage

So all you are left to decide is which Culture should be used, because the default is derived from the actual user context and will therefor be different for every user/region. In the case of the Console.WriteLine this might be fine, because you may want to show the number to the user in the way he is used to read numbers but the Convert.ToDouble use-case is different. You have to decide the culture to use based on the origin of the value and in my example it is a constant value in the program but it could come from a file or a web service or whatever or from user input.

To make my example work the same for every user, I am using the invariant culture:Convert.ToDouble with invariant culture

The invariant culture is sort of a baseline culture and it is mostly equal to the English culture. As a last bonus, here are some ways to create cultures:Some ways to get CultureInfo - Instances

Using [Obsolete] to handle legacy code

I frequently use the ObsoleteAttribute in legacy, spread-out codebases and I want to tell you how and why. But at first, what is the ObsoleteAttribute (MSDN)?

TL;DR; I like to use the obsolete attribute to transition implementations and signatures in a smooth way.

The attribute has 3 different constructors. The first one has no parameters at all. My advice here is to never use it ever because it does not tell why you used it.

The second constructor takes one string parameter. This is a message meant for others developers. If you use or try to use a method or class tagged with this attribute you will see the message and it will generate a compiler warning. I always use this message to explain “why?” and “what’s next?”. This means “Why is this message obsolete?”. It often is because the method has a vague naming or an obscure handling of some edge cases. But I see it as a good practice to always tell the caller what he should do next. So I always end the message with “please use the other method overload with parameter xyz.” or “please use method xyz instead”.

ObsoleteAttribute_in_VisualStudio

So why do I not simply refactor the method name and/or signature? Most legacy projects tend to have everything as public, have no unit tests and reuse code in several solutions in a mix+match style. It is hard to be absolutely certain to really find all callers. Some projects tend to do partial deployments as well. This means to replace only a subset of the assemblies in a deployment. This is most often done to save time. So as a baseline, I only refactor it, if I am sure it is safe and most of the time I cannot guarantee that. My next best way is to mark the method as obsolete and create a fixed version of it and then redirecting the caller using the message in the obsolete attribute.

But who cares about compiler warnings?

This is, where the last constructor comes into play. You can provide a “true” after the message and the warning becomes an error message which breaks the build. This is the route I take if I am fairly sure, I changed all callers. There is one benefit over simply removing the old method. The method with its implementation gets still compiled.

This helps in the case of an old assembly of a caller which was compiled against the old method which now loads the new library in the mixed deployment scenario. In the case of the obsolete attribute the actual method is still fully compiled and usable so the old process still works. If I had deleted the method instead, I would now receive a MissingMethodException.

And after some weeks/month I simply remove old obsolete methods and I can be sure, there are no callers left.

So in summary I like to use the obsolete attribute to transition implementation and method signatures in a smooth way.

Linq: Finding the inverse operation to SelectMany()

I am searching for the inverse operation to the SelectMany() linq method. But at first, we have to understand the SelectMany() method. Consider a simple array containing arrays of numbers and we do SelectMany() on them:

int[][] arrayOfArrayOfInts =
{
    new[]{1,2,3},
    new[]{20,21,22},
    new[]{99}
};

var numbers = arrayOfArrayOfInts.SelectMany(ints => ints);
foreach (var number in numbers)
{
    Console.Write("{0}, ", number);
}

This will output the single list of numbers:

1, 2, 3, 20, 21, 22, 99,

For the inverse operation, we have this single list of numbers and are trying to put them in different lists to build the given groups above again. Obviously we need some kind of way to decide, whether a given number should be added to the current list or should produce a new list. To solve this, we could do the integer division by 10 in a group by on all numbers, like so:

int[] numbers = {1, 2, 3, 20, 21, 22, 99};

var groupedNumbers = numbers
    .GroupBy(num => num/10)
    .Select(group => group);

foreach (var groupedNumber in groupedNumbers)
{
    Console.Write("Group: ");
    foreach (var num in groupedNumber)
    {
        Console.Write("{0}, ", num);
    }
    Console.WriteLine();
}

This outputs exactly what I wanted:

Group: 1, 2, 3,
Group: 20, 21, 22,
Group: 99,

So this article is done? Not quite yet, because I maybe oversimplified my actual task at hand 🙂

The way my actual task works is that I only know whether a given number belongs to the list or should start a new list by looking at the previous number. We could express this as the following predicate:

private bool BelongsToSameGroup(int previousNumber, int currentNumber)
{
    return previousNumber + 1 == currentNumber;
}

The signature of the method is important as it needs not only the current item but also the previous item of the list! This is the point where GroupBy() is of no help anymore and where I got stuck for some time.

The Linq method I am searching should have a signature to something like this:

IEnumerable<IEnumerable<T>> 
    InverseSelectMany<T>(IEnumerable<T> source, Func<T, T, bool> belongsToSameGroupPredicate)
    {...}

Knowing the correct target is 90% of the solution. So having found the signature, I came up with the following solution:

static class InverseSelectManyLinqExtension
{
    public static IEnumerable<IEnumerable<T>> InverseSelectMany<T>(this IEnumerable<T> source, Func<T, T, bool> belongsToSameGroupPredicate)
    {
        var internalList = new List<T>();
        T lastItem = default(T);
        bool isFirstRun = true;
        foreach (var item in source)
        {
            if (!isFirstRun && !belongsToSameGroupPredicate(lastItem, item))
            {
                //start new group and return current list
                yield return internalList;
                internalList = new List<T>();
            }

            //add item to current list
            internalList.Add(item);
            lastItem = item;
            isFirstRun = false;
        }

        //return last list
        yield return internalList;
    }
}

This will build up an internal list of items as long as they belong together and will yield the list, if they do not belong together anymore.

The usage looks like:

int[] numbers = {1, 2, 3, 20, 21, 22, 99};

var groupedNumbers = numbers.InverseSelectMany(BelongsToSameGroup);

foreach (var groupedNumber in groupedNumbers)
{
    Console.Write("Group: ");
    foreach (var num in groupedNumber)
    {
        Console.Write("{0}, ", num);
    }
    Console.WriteLine();
}

This does now exactly what I needed 🙂