I have recently seen a number of examples of what Microsoft does with the user feedback data they collect from their massive installed base. I am not talking about Google-style personal information collection, but rather anonymous collection of user interface and error data in a way that is more designed to built better products than targeting ads.

The first paper is “Debugging in the (very) large: ten years of implementation and experience” by Kinshumann et al, Communications of the ACM, July 2011. This paper describes how Microsoft uses of the data they collect from Windows Error Reporting (you know, the little dialog boxes that appear every once in a while on Windows when a program has crashed or frozen, or Windows restored from a crash).

Microsoft has a number of heuristics that look at the data collected, grouping the bug reports into buckets. Ideally, each bucket corresponds to a single root cause for possibly quite different errors. They automatically analyze the errors and generate metadata about the error reports that can be used to generate statistics and allow database queries to be performed over all collected error
reports.  Heuristics include walking through chains of threads blocked on synchronization objects to determine which one is the actual cause of a hang, and finding the most likely thread and stackframe for containing the root cause of an error.  Heuristics are applied both on the client and the server, but mostly on the server. Technically very hard to do right, I can appreciate the huge amount of work that has gone into engineering this.

With this huge pile of information, a new debugging method becomes available: statistics-driven bug finding and prioritization at large scale.  The introduction to the paper puts it very well:

Beyond mere debugging from error reports, WER enables a new form of statistics-based debugging. WER gathers all error reports to a central database. In the large, programmers can mine the error report database to prioritize work, spot trends, and test hypotheses. Programmers use data from WER to prioritize debugging so that they fix the bugs that affect the most users, not just the bugs hit by the loudest customers. WER data also aids in correlating failures to co-located components. For example, WER can identify that a collection of seemingly unrelated crashes all contain the same likely culprit—say a device driver—even though its code was not running at the time of failure.

For a product manager like me, used to working with individual bug reports in bug reporting systems and trying to manually assess the importance of each error, this is nothing short of a dream.  Instead of trying to guess how many users can be impacted by a bug, Microsoft can run queries against the error report database and get a fairly accurate idea of how common a certain error is in the user base.  This has allowed them to address the most common errors first, leading to Windows and Office becoming more stable for more users in recent generations.  They can also pinpoint which device drivers are causing the most issues, and putting pressure on vendors to clean up their act.

I wonder where else you really apply this idea of statistical debugging. You need a large user base, in systems which are connected to the Internet so you can collect data, and who are comfortable with providing direct feedback to you as a vendor.  Apparently, Apple has the same kind of feature built into iOS, with more than 100 million users which seem not to be too interested in strong privacy.  Presumably, Google can do the same thing with Android, at least its use in phones. Mozilla has a crash reporter, so I guess it makes sense in the consumer space.

But when your user base counts in thousands of seats and half of these are in defense sector beyond air-gaps, it is harder to apply. Products that call home are not taken to kindly in the professional field, as secrecy and confidentiality is very important to big companies. Industrial embedded products like telecom infrastructure might have sufficient volume of code and computer hardware to form a basis for statistical reporting – as long as operators agree to provide the information to the hardware vendors.

Another example of how Microsoft makes use of their collected data is in UI design. The blog post “Improvements in Windows Explorer“, by Steven Sinofsky, from the Windows 8 blog discusses how Windows Explorer has evolved over the years, and how it is now getting a radical redesign based on usage data.  Microsoft is an enviable position here, having collected information about what millions of users are doing.  Definitely beats inspiration or trying with a few users in a classic user interface lab.

I have seen quite a few people criticize this blog post from a variety of angles – from the fact that they are not data-driven enough and keep rarely-used buttons in the ribbon to the fact that they remove somebody’s favorite function.  It is also the case that the measurements can only tell you which functions people are using from what is available today – if you want to invent new things, data like this might not be very helpful.

Fortunately, Microsoft also seems to have taken a clue from Linux and is allowing much more user customization than before. For me, this is great news, as I seem to have a user profile quite far from the mainstream.  We have not seen Windows 8 in its final form just yet, but hopefully this approach will be applied to other parts of that GUI overhaul too.  There are professional Windows users who need an OS that makes even very esoteric operations easy to access, and customizations of things like the start menu possible.  Hopefully, we do not get washed away by the flood of data from regular users.

For some reason, I feel that bug reporting is not as sensitive to the user style as GUI design – Windows and driver bugs would seem to be more evenly distributed as they depend more on hardware than on software. At least it seems to me that Windows is more stable today
than it was a couple of years ago.

There is a better way.

Just right-click on the desktop, select “View” and then “Show desktop icons”. So simple.

Should have found this years ago. Works on Windows Vista and Windows 7 as far as I know.

0 [main] us 0 init_cheap: VirtualAlloc pointer is null,
Win32 error 487 AllocationBase 0x0, BaseAddress 0x68540000,
RegionSize 0x480000, State 0x10000
c:\msysgit\bin\sh.exe:
*** Couldn't reserve space for cygwin's heap, Win32 error 0

More than mildly annoying.

I tried searching the web, and found a number of discussions on similar issues. It was not easy to find one that worked, but in the end it turns out that playing with the base address of the msys-1.0.dll file worked. The error is not really in TortoiseGit per se, but rather in msysgit (which tortoisegit relies on to actually do its work).

The magic incantation that I wound up using:

c:\msysgit\bin>rebase.exe -b 0x50000000 msys-1.0.dll

Posting it here for the benefit of any other poor soul who is hit by the same. Apparently, you might have to change to use different other addresses.

The details of my setup for reference:

• msysgit 1.7.4
• tortoisegit 1.6.5.0
• Windows 7, 64-bit

Here is the scenario.

We have a wiki system on our Intranet at Wind River, where I attached a Powerpoint 2007 presentation to a file in the wiki. This file has the extension .pptx, as they tend to do. I do this in Firefox, and I can download and save and view the files with no problems:

A colleague of mine tried to do download the presentation, but failed. The problem for her was that she was using Microsoft’s Internet Explorer, rather than Firefox. IE did not recognize .pptx, and instead did some kind of magic lookup and decided that the file is a .zip file:

So here we have a case where Firefox is able to recognize a file extension, but not Microsoft’s own web browser. The reason that IE and Firefox are using the file extension is that our wiki did not have a MIME filetype set for the .pptx files, leaving each client to figure it out on their own.

To compound the problem, the simple fix to change the file extension of the downloaded file from .zip to .pptx is not available per default on modern Windows systems. The default policy is to  hide extensions for known file types, so they are unavailable to be edited. The fix is to reveal all file extensions, which at least for me is the first thing I do when I get a new system setup. As  a programmer, I need to know exactly what things are, and quite often change their extensions.

I guess that Microsoft only tested things in a completely homogeneous system. I assume that if you use Microsoft ISS as your webserver (in a recent version), it will give you the right MIME type automatically. Testing with an Apache server or an old server version might not have been in the plans for Microsoft IE8. Once again, we see that testing requires some imagination to do well.

I would like to note that this problem is not new, and there is a writeup at http://www.pptfaq.com/FAQ00911.htm explaining the issue and some related issues as well.

Here is a list of some of the ideas I have seen floating around:

• They are going to put Windows 7 on ARM to defend against Linux in the sub-netbook segment. Certainly a useful move – but do you need an architecture license for that? It would not make much sense to put Windows on ARM in a way that requires you to have a Microsoft-designed processor core to run it.
• It is a case of Apple “A4″ envy – if Apple can build their own custom ARM chip for a consumer devices in the tablet space, so can Microsoft. I think this makes a lot of sense. But once again, what use is a license to design your own ARM core and ARM instructions? Apple does not seem to have created a custom processor core, instead using a standard ARM Cortex-A8 core in the A4 chip as far as I understand.
• Handheld gaming, building their own phone. It is essentially the A4 envy argument, and I cannot see how an architecture license will help.
• Deeply embedded devices. Microsoft wants to ride the ARM wave into things like power meters and other truly embedded systems. Once again, that might certainly be a good idea – but why an architecture license?
• As a defensive move in the Microsoft-Intel relationship. If Intel is toying with Linux, Microsoft can be toying with ARM. In that context, a few million dollars to acquire a license to do nothing much with it might make sense as negotiation leverage.

The best idea I think is the server angle. ARM is making inroads into servers where single-thread processor speed is not that important. The power consumption advantage over x86 is significant, as long as you do not need utter speed or very many cores sharing memory.

Power-efficient servers is an area where Microsoft can certainly see the potential for truly revolutionary changes in the IT field. There is no reason why x86 would be the best architecture there, and the fact that x86 is controlled by Intel and AMD makes it very hard for Microsoft to really take part in hardware-software innovation. Essentially, Intel and AMD design the processors, and the software has to adapt. This model is not optimal if you want to really see what you can do with the hardware-software boundary.

Thus, if I think Microsoft might do with the ARM license is to start tinker with the OS interface of the hardware. The beneficiary would be both Microsoft and ARM, since Microsoft innovations for ARM might well become standard in ARM land. For example, an OS might benefit tremendously from small changes in the processor in areas like:

• Memory management – a hardware scheme tailored to what Microsoft sees being done in their operating systems and their third-party server software could certainly be very beneficial to efficiency.
• Multicore and multithreading – ARM has some special support in their MPCore designs for communicating between cores in a multicore system. Such support could be extended to help the OS manage both threads and processors across multicore designs. Also, ARM might want some help from Microsoft to design a scalable OS interface that works for tens or hundreds of tightly-coupled cores (rather than the current limit of 4 in ARM Cortex-A9 MP).
• More bits – the biggest architectural problem I see with ARM in servers is that ARM is currently a 32-bit architecture. 32 bits are not enough for even medium servers today. Having Microsoft help ARM design a 64-bit version of the ARM architecture sounds far-fetched, but it is certainly something where a license to change the ARM instruction set would help.
• ARM instruction set additions to help build an emulator for x86 instructions on ARM, to allow current x86-Windows applications to run on an ARM-based device. This is an idea which has been tried in the past and which has never been very successful in the market. ARM currently has a set of instructions which help accelerate virtual machines such as .net and JVM, and I think that is as far as this idea has been proven useful.

In summary, the only reasonable use I can see that Microsoft would make of a license to build custom ARM processor cores and tweak instruction sets would be to build better servers, by improving multicore communications mechanisms, upping ARM to 64 bits, tweaking the MMU design, and possibly creating some kind of x86 emulation support.

That said, I think the most likely result is a custom ARM-based chip to power a phone or tablet or other consumer electronics device, running a Microsoft software stack (derived from Win CE/Windows Phone) on Microsoft hardware, all sold as a Microsoft product. Essentially, doing the Apple all-integrated solution.

There are two important aspects to this worm, as I see it.

First, it only works due to the fact that the software in question is running on regular Windows PCs.  An attack on a real embedded OS like Wind River VxWorks or Enea OSE would be more interesting, and much scarier since that would mean a much more devoted enemy. In this case, the attackers are opportunistic, using the window of vulnerability of a new Windows flaw to attack Windows-based plant control systems. They also use signed Windows drivers, which is apparently a new development in malware. All quite interesting in its own right, and worth reading about for those interested in security.

Second, the malware spreads using physical movement of USB memory sticks rather than attacks over the Internet or other networks. This makes the very important point that even if systems are not connected to the Internet, they can still be attacked if something crosses the “air gap” that separates them from the outside world. In this case, a plant would be infected by using social engineering to make some employee carry an infected USB stick into the plant and putting it into some internal PC. Once the infection is inside the plant, it might spread over networks or by USB sticks moving around inside the presumably protected perimeter.

The lesson I think we can draw from this is that using general-purpose desktop operating systems for critical systems is a bad idea. Using a more obscure real-time OS (or even Linux) would probably reduce the number of vulnerabilities – but more importantly, it would make it much more difficult to make an infection hop from computer to computer until it reaches its target.

In this particular case, all Windows machines are potential bearers and spreaders of the infection. An attacker can rely on that fact to seed the Windows ecosystems at some place, hoping to get the infection hopping from machine to machine until it reaches something interesting. There is no real need to seed the infection directly into the target plant. If the target systems had been running a different OS, the attackers would have had to get really close to the target, making them easier to stop.

Overall, embedded systems security is something that we need to take much more seriously going forward. As we rely on embedded control systems to run much of the modern infrastructure and economy, we really need to be concerned about how to secure these systems. Security needs to be part of the architecture of embedded systems, including their operating systems (please make them as robust as possible), application designs, and networking systems. Unfortunately, current embedded technology tends to be designed to work, with little care for how they could be broken by an intentional attack. One scary example of this was provided in the SecurityNow podcast episode #251, where a listener shows how easy it could be to take remote control over a car due to carelessly designed fleet management units.

Updated information

I found some more in-depth information on the issue at Infoworld. It notes that the software that is attacked is vulnerable since all installations use the same password to login – changing it is likely to break it. That is totally ridiculous as a security solution, period.

Turned out this is a fairly common problem, with a documented solution.

It is MS knowledge base article 211494.

For some reason, you need to go into Excel options and have it allow other programs to access Excel via DDE, as in the screenshot below:

What I ended up doing to keep using my scanner (whose hardware is still very much intact and solid) is fairly obvious: I installed my old Windows XP license on a VMWare virtual machine (I had the good luck to have a full license with physical media), and then install the Canon LIDE30 driver on that virtualized XP.

VMWare Player is sufficient to let me attach the physical scanner to the virtual machine’s USB interface, and drive it without the host Vista 64 machine being any the wiser. To get the scanned pictures out, I have to resort to drag-and-drop, as I have failed to get shared folders to work with Player for some unknown reason.

The end result can be pretty complex… To send some emails from my work computer including scans with this scanner, I had to:

• Scan on the virtual XP machine
• Drag-and-drop to the Pictures folder on my Vista 64 machine
• Use file-sharing in Windows to move to my work laptop
• Attach in Outlook

Workable. It is also a pretty good demo of the power afforded by modern consumer operating systems. Imagine trying to do that in 1995… would not have been quite as fun.

However, moving to Vista has made the printing bit harder.

In Windows XP, there were drivers available for the printer, since it was fairly recent when XP was released. In Vista, no such luck. So you have to resort to using “LPR” printing (optional install), and using the generic “Microsoft Imagesetter” as the printer profile. This, somewhat surprisingly, works pretty well.

With one exception: Acrobat.

It seems that Acrobat is trying so hard to be smart about printing that it gets confused by the Imagesetter bit, and decides that the thing on the other end is not a Postscript printer. And thus, it needs to have a 600 dpi bitmap of the page being printed sent to it. Needless to say, my old printer with its upgraded 12 MB of RAM (it came with 4, and I scavenged 8 more MB from some old dead PC that passed through my hands in the late 1990′s) usually chokes on this.

When my last XP machine was retired, this did indeed create a problem, since all official online forms tend to be Acrobat-based.

However, by accident and luck I decided to try the Foxit Reader as an Acrobat Reader alternative. This has turned out to be the perfect to solution to my printing woes. With Foxit, a PDF file prints as a small nice regular vector graphics file that my LaserWriter has little problem printing. It makes printing PDFs feasible and reliable again, and means that I do not have to go out and figure out which new printer to buy. It is kind of cool to have such a decade-old technology icon at home, and still in working order.

A final note: 12 MB in my printer. My first hard drive back in 1990 had 20 MB on it. My new desktop Core i7 machine just got upped to 9 GB of RAM. Back in 1991, I had my high school’s most powerful home computer: a Macintosh SE/30 with all of 5 MB of RAM (which cost a fortune at the time).

To get it installed, I found good instructions in the WinhelpOnline Blog. The only annoyance was that it was put on the top level of accessories folder, and not under Tablet PC… despite being a part of that toolset. It is here on my machine:

Still note that for professional heavy-duty screen dumping and capturing, nothing beats SnagIt!

In the end, it turned out it was really dead easy!

The steps are the following (please refer to this guide at iLounge for more background and details):

• On the new machine, start iTunes and authorize it using user iTunes store account.
• Make sure the folder you use as your iTunes library folder is within the Windows standard user data places, like the default location inside “Documents\Music” on XP and “C:\User\username\Music” on  Vista. It needs not be in your user’s hierarchy, I had mine in the shared hierarchy and that worked fine.
• Consolidate your iTunes library, using the iTunes 8 File-Library-Consolidate Library option.
• Copy everything in that iTunes library folder to the corresponding location on the new machine. In my case, this was the folder called “Shared Music” in the “Public” part of the file system.
• The iTunes database is separate from the storage of files. On Vista, it was located in User\Username\Music\iTunes. This is easy to find, as it is created when iTunes is started. After locating it, simply replace the iTunes folder on the new machine with the one from the old machine (without iTunes running!).
• Start iTunes on the new machine, and all your data and music should be there!

For me, this worked on the first try. The key thing that I had to do was the consolidation phase… there were some files outside of the managed iTunes folder that needed to be copied into it in order to make this work completely.

To do this, it seems that you need to have a display driver that supports rotation and installs the keyboard shortcuts for it:

• ctrl-alt-up/down/left/right arrow

It seems that children tend to hit the left or right arrow, leaving the screen rotated. This rotation survives reboots it seems,

To fix it, just press ctrl-alt-up arrow and the screen should be right fine.

I have not been able to duplicate this on any of my own machines, as that shortcut does not seem to be active on them.

So I dove into the help system, and the first hit was the right one.

A topic called, of all things, “The text of my message is larger or smaller than usual”:

If you read that closely, you realize that what we have here is a really bad case of product design and bug fixing. Essentially, they have put in a documentation fix for a bad UI problem. This text is a plain admission that people do this by mistake and have no idea what they did to cause it, or even that they are seeing a zoomed view. The root problem is that the zoom activitity can be done by mistake, by the shortcut “ctrl-scrollwheel-on-mouse”. Microsoft must have noticed this in testing, and put in a piece of help to help you out. But before you can find that, you need to figure out that you should search the help for the issue.

If you do not have a scroll wheel or don’t know this, the solution is to get to the format text tab, click zoom, and then change settings in a dialog box. Amazingly bad, consdering the effort going into Outlook:

The correct solution is NOT to document it like this, but rather to just have the same little zoom control as all other Office programs and remove the easy-to-hit-by-accident shortcut (maybe it can remain if the effect is clearly seen in the zoom control).

I have been doing both of these modes to some extent, and the main problem with them at least on a PC is that while the processors might be good at multitasking and sharing the CPU load, my IO system is annoyingly non-parallel.

At least on my laptop running Windows Vista, any task that accesses the hard drive intensely (compiling, updating source tree, indexing files) blocks any other task that would also be hard drive intense. For some reason, that includes Microsoft Powerpoint and Outlook. So being productive while my machine churns in the background is pretty hard.

In general, this means that any multicore device intended to do many different things at once will have to have IO that is similarly parallelizable. And that means working hard to avoid any choke points where one stream of data can disturb another. The typical PC of today is not that system, with lots of bottlenecks where “all” data have to pass. Most embedded multicore SoCs appear at least to have more direct connections from IO to cores, but I have not seen how well it works in practice.

This small rant was written while everything else was locked-up competing for my hard drive. I want a parallel fast disk system in my laptop! Pain!

The basic premise is definitely one that makes sense. We have seen this at Virtutech, working with stripped-down Linux images on various embedded boards. Turning on and off individual services often has a significant impact on both execution speed and memory consumption of a particular target. The gut reaction when setting up a new target is to think about what can be stripped out rather than on putting in everything that could be useful. Since that will result in a bloated image that will consume resources with little additional value.

The difference is even more obvious for Windows machines, where the speed of the same virtual machine running NT, 2000, XP, or Vista is incredibly different. With Vista and XP in particular, there is much going on in the background eating up memory and processor time. Some simple tuning tricks like turning off animations in the GUI or background indexing can have a very large impact on speed. Same thing goes for graphical desktop Linux distributions, where turning off eye candy results in significant speed increases.

So I believe that Fastscale can offer what they promise, it is a nice simple idea. But as always with a simple idea that makes for a powerful product, there is something more than just the idea. There is a somewhat tricky piece of execution.

The trick that Fastscale brings is to automate the process of creating a minimal but sufficient substrate for a particular application, given the app, target OS, and server hardware. Sounds like dependency checking once you have the data on what needs what, but finding out the particular dependence chains in any particular OS is hard work. Doing it manually is pretty painful, and the Linux kernel configurator is not always up to the task.

It is also a nice example of a case where open-source opens up for new innovation. Doing this kind of configuration to Windows is much harder in practice, if nothing else since you do not get to recompile the kernel yourself.

Also, Linux in particular has rather good support for removing non-essential components, I believe to some extent thanks to its extensive use in embedded software. Embedded people have a tradition of application-specific configurations, and here is a nice case of when that results in a better desktop/server product in the end. Who said embedded requirements were just for a niche market?