On Wednesday, November 22, we had our second CaSA, Computer and System Architecture Unraveled, meetup. Same place in Kista as the last time, the 25th floor of the Kista Science Tower building, thanks to the kind sponsorship of Vasakronan and our collaboration with Kista Science City. This time, the theme was networking – but not at the socket level. Per Holmberg presented how his team used “micro sleep” for power management in line-rate network processing, and Hans Brandberg talked about the Precision Time Protocol. Another great event!
Energy-Efficient Packet Processing
Per Holmberg presented work he did when he was at Ericsson, published in the Ericsson Review. It was an interesting mix of networking coding and computer architecture power management.
Networking applications of the type that Per talked about typically do not use power management of any kind, even when mostly idle. The latency of waking up a processor from sleep is too high, and even adjusting clock frequencies can result in added packet latencies. Thus, normally a processor used for packet processing is kept at a steady clock rate and with no attempts made to save power.
The networking code is written using DPDK, the Data-Plane Development Kit, as a set of busy-wait poll loops in user space. This bypasses the kernel and removes kernel power management from the system since all cores are all busy all the time. However, there is a way to do power management even from user mode. Recent Intel processors feature UMONITOR/UMWAIT, a way for user-mode code to utilize “microsleep”. Essentially, the busy loops were coded so that they waited using UMWAIT, monitoring a memory location where the networking interfaces would deposit newly arrived network packets. Similar features are available on ARM and RISC-V too.
This resulted in some surprising power savings – idle power went down to about a third, and even under max network load, there were some significant savings in power. This is basically due to the margin built into the system – you do not build a networking application in such a way that maximum supported network load means 100% processor load. You need some margin to allow for hiccups and variations. User-level sleeping lets the system take advantage of that to save power.
Note that we missed a detail when announcing the event: until the day of the event, the presentation was listed as “Presentation 2”. Giving associations to a certain famous Blur song… Guess we should have played that when introducing him.
Precision Time Protocol
Hans Brandberg, now at Intel, presented the history of time synchronization and the Precision Time Protocol.
Hans presentation of the history was maybe not very technical, but quite interesting. Over time, we have got extremely good at measuring and synchronizing time. Probably more so than in many other measurement types. One aside that came up in the discussion was that time can be used as the basic “sensor” in many applications – like doing Analog-to-Digital conversions by measuring the time a capacitor charges instead of trying to directly measure voltages. Not sure I followed all of it, but it was an interesting point. He also talked about the history of the leap seconds, and the fact that something is happening today with the Earth’s rotation that might force us into a negative leap second at some future time.
The precision time protocol (IEEE Standard 1588) itself is a deceptively simple design that uses a packet exchange with time-stamped packets to set the clock of a subordinate system to that of a master system. It includes algorithms to select the “grandmaster” clock (wonderful term) and defines multiple types of nodes, etc.
Time-stamping is critical to PTP and usually you have hardware support in network interfaces to automatically time-stamp packets as they are sent. With a typical rate of 128 synchs per second the packet rate overhead does not sound too heavy. The PTP handling can be run part of the load on an average server. But then we have datacenters like those run by Facebook/Meta where you want to synchronize the time across hundreds of thousands of servers. There you really need dedicated hardware and hardware offload to processing from the servers.
One interesting aspect is that just how you adjust the time in a system is unspecified by the standard, and an area where there is room for innovation and care. Just changing the local time in a single jump is rarely a good idea. Instead, you need a good time adjustment algorithm that can adjust time smoothly to keep applications that use time happy.
Discussions
We had about the same number of attendees as last time, and just like the last time the discussions were intense with a noticeable murmur as soon as we released the audience from the talks. Some people were return attendees and some were new. Given the difference in topics from last time, that was to be expected.
For me personally, a highlight was to meet Robert Virding, one of the creators of Erlang. Back when I was a student in Uppsala, I bought the book Concurrent Programming in Erlang. I still have my copy, with lots of notes and little indexing stickers, as can be seen above.
We got into a fun argument about object-oriented programming and its relationship to the process-oriented programming in Erlang. Robert is not a fan of classic object-orientation (neither am I), and he told us that since they published the Erlang book in the mid-1990s they had to say something about object-oriented programming. So, they added what amounts to joke chapter to the book talking about how to do OOP in Erlang.
Kista at Night
We seem to have bad luck with the weather. This time, it was snowing. I hope that some day we will run a CaSA event and get the full beautiful view that the altitude should give us!
Vasakronan had also improved the show room furniture on the floor, which was much appreciated!
Future
The third event is planned for January 17, 2024. Keep an eye on the Meetup page or the Kista.com event page for details once the speakers and topic is finalized. Hope to see you there!