Remember the Itanic

Remember the Itanic

Intel may have had the best of intentions for the Itanium architecture. However, from the very first day I sat down and tried to use one after I joined the Windows team, I found it frustrating. It was loud, slow, and hot. Just starting the notepad was slow. It was a hot, angry pig. (And not even a fast pig like a javalina.)

Itanium was designed to be a brand new platform for high-end workstations and servers. The x86 instruction set (the “PC” all of us have been using since 1981) that Intel itself had been built upon for more than 20 years would be cast aside. Along with the x86 architecture, the firmware that the computer used to boot the operating system – the BIOS. Along with both of those, the Itanium would also not provide any x86 compatibility. This meant, among other things, that Itaniums would not run any existing legacy applications. Only applications that had been ported to (rebuilt for) the Itanium would run there.

This was not good.

Intel had publicly boasted that the Itanium would be an incredible performer. While Windows may have deserved some of the blame, that was never my experience. From the earliest prototypes we had, performance was abysmal. The initial prototypes we had were loud, hot, and slow. If they were in offices, the Itaniums were usually off unless the developer (the individual with the misfortune to need them most, to add or support some set of features) needed to be running code on it. Many developers referred to these pre-production systems as “space heaters”. But space heaters were quieter and used less energy. <rimshot/>

A story I had heard that said even key Microsoft architects were sour on Itanium from the very beginning, and didn’t mince words in sharing that with Intel. I wasn’t there, so that’s hearsay. 

This version of Windows was the first true 64-bit operating system Microsoft had ever produced, and the hope was we would be able do what no other processor architecture had been able to do before it. As I’ve noted before, Microsoft has never been able to sustain Windows on any platform besides x86 and now x64 (née AMD64.) 

MS-DOS and Windows originated on, and grew up on, the 16-bit Intel x86 architecture. Windows 95 began the consumer step towards 32-bit Intel x86 support, while Windows NT started with 32-bit support natively – but Windows NT didn’t start on the x86 processor, as I’ve noted a few times before.

At a high level, Windows NT was designed to be easily ported across architectures from the very beginning. A low-level component of the operating system, called the Hardware Abstraction Layer (HAL), was originally intended to enable Windows to move from one hardware platform to another with minimal work. Alas, it’s not always that easy, as there are always minute (or major) differences in the architectures. As a result, even when Windows NT did manage to arrive for this or that new architecture, it was always a matter that this particular feature didn’t get ported to that platform, or only x86 supports that old feature, etc. Small nuances pop up from the hardware, and expose themselves through the seams of the OS when you least expect it.

Windows NT was originally developed on the Intel i860 processor (a RISC-based chip) first, with the intention of ensuring platform portability. But the i860 was never supported by a released version of Windows. And so it was that Windows NT, with it’s portable architecture, was born with that same x86 birthmark as MS-DOS and all earlier Windows consumer OS releases.

To this point in 2000, the Windows NT platform had been ported to 5 architectures in its lifetime (8? today):

– Intel i860 – Pre Windows NT 3.1 (never released)

– Intel x86 – Windows NT 3.1-Windows 8 (1993-present)

– DEC Alpha – Windows NT 3.5-4.0 (1994-2000)

– MIPS – Windows NT 3.5-4.0 (1994-2000)

– PowerPC – Windows NT 3.51-4.0 (1995-2000)

The Itanium would mark the sixth architecture officially supported with Windows NT. Unfortunately, off these 6 platforms, only one to this point (other than x86 itself) had ever supported running x86 code on them. Even then, the past was already littered with the corpses of ports of Windows to other platforms that had largely failed due to the combination of their own cost vs. commodity x86 hardware, and the lack of software ported to run on their platforms. And even on DEC’s Alpha (natively and only 64-bit), Windows NT wasn’t ever 64-bit, it was 32-bit. The cost and value proposition never lined up.

So here we were, building another version of Windows that wouldn’t run legacy applications, and would be very expensive. 

The Itanium was to be a critically important move for Intel and HP in particular. (And oh boy were they vocal about it.) I was a bit cynical from the first time I got the architecture that it was the right direction forward. (To this day, I still wonder about how much money changed hands which way for Windows on Itanium to happen.)

But at that point, we had one goal – to put Windows on the Itanium and see if customers would buy it. To do so, we had some work to do. I mentioned how Windows was deployed to date, and that since there was no 64-bit version of MS-DOS for the Itanium, nor would there ever be one, there was a need to create WinPE for Itanium as well. For the longest time, clean installation via CD was the only way to install Windows on the Itanium. We had thought of putting support into Remote Installation Services (RIS) to deploy Itanium, but even with PXE built into it, versus duct-taped on as it was in most x86 PCs at that time, it looked like supporting RIS on Itanium would be a chore, for unknown value.

Plus I was going to have to ship RIS when we shipped Windows XP, not with Windows Whistler Server – which would ship later (much, much later, but I’ll try to discuss that another time.) So if we were going to support Itanium for deployment through RIS, we’d have to either duct-tape something to Windows 2000, as we were doing for Windows XP, or we’d have to wait and use other deployment tools like WinPE. The short answer was we focused mostly on WinPE-based deployment, though one of the developers on the kernel team was able to get setup mostly working through RIS before we shipped – though only using our easier model of installation (scripted setup), not an image.

The irony wasn’t lost on me that I was trying to decide whether we should (or could) support a certain model of setup for Itanium – the platform that had tangentially birthed WinPE by necessity – which would grow to become the standard tool used for all of Windows setup. This is the one place where I issue a thank you to the Itanium. If it had not come along, one of the most interesting, plot-twisting parts of my career would have never happened. Thanks, Itanium. Rest in peace.

This was the first time at Microsoft I had really struggled with major “cut lines” – trying to figure out what we could support in a released physical product, and what we couldn’t. Adding things cost money, and doing it wrong could cost even more.

The primary companies making Itanium servers were HP and Intel. While Dell sold Itaniums, these were all white-label Intel systems, if memory serves correctly. Regardless, most of our feature requests – and problem reports – for Itanium came from either HP or Intel (because they both hit the issue first), or one of their pilot customers – in particular a certain large manufacturer.

When I interview people, I usually ask them, “Tell me about the biggest mistake you’ve ever made in your academic or professional career.” My goal is rarely to find out their deepest darkest secrets. Rather, it is to see how good they are at reflection and introspection – and more importantly, if they can admit they make mistakes. 

For me, I often throw this next anecdote in that list when I consider how I would answer that question myself.

In 2002, an OEM asked me if it would be possible to make WinPE boot from an LS-240 drive. A magnetic drive that is basically a super-capacity floppy diskette, the LS-240 was never a huge hit in the US. However, it was included in many Itanium systems instead of a floppy (which admittedly would have been a pointless peripheral on the Itanium). 

The first thing you have to understand about Windows on the Itanium was, it was big, particularly in comparison to sibling builds of Whistler on x86. The OS took up considerably more disk space and memory than x86 Windows. As a result, while we had come up with ways to strip WinPE on x86 down to 96MB, on Itanium, that LS-240 even left almost no breathing room. But that wasn’t the worst part. If you’ve ever tried copying massive amounts of data to, or reading massive amounts of data from, a floppy diskette, you know that it takes time to do that. Now play along with me – imagine booting an entire 200MB OS off of it.

Even before we started, I thought the idea was bad. This OEM’s manufacturing team insisted… absolutely insisted they needed the feature – insisting that if we didn’t do it, they’d have to look at using Linux to deploy Windows – talk about a groin kick. I agreed that we’d take their request under advisement, and see what we could do.

After my concall, I walked down the hall to the office of one of the newer developers, who was now sharing my old office with one of the WinPE and RIS developers.

I asked him how hard it would be. He said he thought it would be feasible, but he’d need to talk to the kernel team. This dev was always one of the most easy-going on the team, and I always enjoyed his attitude. Unfortunately for both of us – he didn’t push back then harder towards me, and probably should have (not blaming him, I should have killed the request).

I could tell there was still a certain hesitance from him – his voice really seemed to have a “I’ll investigate, but it’s really not a good idea” tone to it. A day later, the dev reported back to me we could do it. It would be about 3 days of development, and two weeks of test. I talked to my manager and on up my chain, and got permission to do it. I still didn’t think it would be a good idea, but I couldn’t bear the OEM’s Linux threat coming to fruition. We really wanted to make WinPE work for any OEM we could, in whatever scenario they needed.

So it was, less than a month later, the dev and I sat down and walked through the process of how the OEM would need to create the LS-240 diskettes, and what the experience was like. 

It was awful. Horrible.

It took upwards of 5 minutes to boot, and paged to the disk constantly – because the OS thought the LS-240 was a hard disk, not a floppy diskette. Still, it did what the OEM had asked for, and what we promised them we would deliver.

A year later, my manager and I were on a concall with this same team at the same OEM again. They had feature requests again, and seemed to feel that we hadn’t done what they needed often enough. I muted the phone and asked my manager if I could ask them how the LS-240 Itanium boot feature was working for them.

Then I asked them, “I’m curious how the LS-240 boot capability we added last year for you all worked out. Did you all implement it in the factory?

There was a bit of silence, and some conversation in the conference room on the other end of the line. They eventually replied that they hadn’t ever used it.

I was bummed. Not shocked, not devastated, but bummed. Truth be told, I should have known it would fall on the cutting room floor once I gave them that capability, because the whole idea was junk to begin with. Lessons like that made me a better PM, though, as they gave me a measuring stick, and more importantly a reminder to pull that measuring stick out, when asked by a customer for a feature. 

In fact, during my time in Windows, I established a bit of a saying that I would use routinely. Customers would often ask me the question, “Would it be possible to do <insert feature request?” My standard retort to that, to this day is, “It’s software. Almost anything is possible. It’s just a question of time, money, and priorities.” Indeed, adding features without respecting priorities or cost would be a lesson I would observe the hard way when we launched into Longhorn – what eventually became Windows Vista. But I’ll save that for a bit.

For me, another example was my script to add WSH, HTA, and ADO for SQL to WinPE for Itanium. Little nits such as the directory where Windows booted from (IA64 instead of x86), and a few differences in ADO. I only lost a day or so to debugging and fixing it, and got it working.

However, the differences between the Itanium and x86 popped through several more times as we tried to build tools for OEMs and corporate customers, specifically around CD boot capability. When we first started with WinPE, booting from a CD would permanently lock the CD-ROM tray. Meaning everything you needed to install Windows had better be on that CD or on a network share you can get to. On x86 with Whistler, you could easily fit WinPE as well as a copy of Windows to install with a script. On Itanium, it wasn’t even close. You had to go to the network to install Windows (which, admittedly, did work for a lot of our customers, but not all). WinPE locked the drive because it was the boot drive ñ and it couldn’t handle it if the CD was removed. So the original WinPE dev added code (triggered at boot using a special configuration file). The feature, called “/INRAM”, took all of the dependencies loaded by WinPE as it booted, and put them in RAM – it cached them. But it didn’t load the whole disc. As a result, you could remove the disk – but then if your installation process ever tried to load something not already in memory, things would silently die, even if you put the disc back. In the end, this feature proved invaluable for our OEM partners in particular. But it was fragile to get working, and we lost a lot of time helping customers debug it when this feature went sideways during their development. I imagine Microsoft’s field employees who helped customers did too. It worked. Recall my comment about duct tape in my last post. Ingenious duct tape, mind you… but duct tape nonetheless.

Years later, working with an architect on the Windows team, we came up with another idea to let people swap WinPE CDs. When you burn a WinPE CD, you start by making an .ISO file. ISO files are just single-file CD images that follow the ISO 9660 – Compact Disc File System (CDFS) standard. Special software is able to burn that image on to a writable CD. An idea that a few of us thought up was to create a special .ISO image that would be completely loaded into RAM when you booted from a CD. Basically, WinPE would become what is commonly referred to as a RAM disk (I guess more correctly, a RAM disc). When you started the computer, a small piece of software on the actual CD would create the RAM disk, chuck the ISO image into it, and “jump” to it. Once that RAM disk was loaded, you could remove the CD from the drive as many times as you wanted, and everything just worked. 

There were only two problems with this idea.

The first problem was the RAM required. Because WinPE took up over 128MB, and you needed RAM left over to actually run the computer, you needed 256MB or more of RAM. This may seem laughable now when stripped machines ship with 4GB of RAM, but recall that Windows XP happily ran on 128MB, though we were starting to see a move towards 256MB as the low end. Outside of one OEM who was quite unhappy with this idea when I announced it to them – since they had a very popular low-end server that they shipped with only 128MB (even then I was scratching my head), it looked like we were good to go.

The second problem was… you guessed it. Itanium. It would have required considerable RAM – but still possibly fitting within the range we were seeing most Itanium systems ship with then. However, because of the way the memory manager worked on Windows for Itanium, it would have required a fairly significant overhaul of the code that loaded the RAM disk in order for it to work on Itanium. We mulled whether it would even be possible in the near term, and decided it wouldn’t. We shelved it. So only the popular x86 platform and later x64 received the far easier RAM disk boot capability while I owned WinPE.

I filed 7 patents while at Microsoft – most with co-inventors. Most of them, I feel, were for pretty innovative ideas. As I look back, the RAM disk boot (and another one having to do with helping the boot process for WinPE) are two I’m proudest of contributing to.

When we were working on USB boot (at that time solely intended for WinPE, but that’s another post for another time), we looked into making it work on Itanium as well. Because of the completely different boot process (different firmware and disk formats), we quickly dropped it for the duration of the development while I was there.

I don’t ever recall meeting a fan of the Itanium within Microsoft. It was always looked at as some sort of weird necessary evil. Even senior architects within the company were “anti-fans” of it. But the company had signed on to support it, and here we were.

Unfortunately, when we met with Intel, most of their proposals had to do with implementing PXE boot security improvements and EFI – the firmware used by Intel for Itanium systems – but by nobody else at that time. Given my cynicism about Itanium, you can likely understand why I was hesitant to meet on futures. I didn’t feel like Itanium was long for this world, even back in 2001. How could I possibly be jazzed about working on technology that was dependent on it?

If the enthusiasm that most developers on the Windows had for Itanium was a dull thud – like a child getting a filling at the dentist – AMD’s proposal for the future was an off-season, non-crowded trip to Disneyland.

I had heard secretive rumblings about another new 64-bit architecture for some time, but hadn’t nailed down specifics. Thankfully, working in setup, we heard in relatively short order as – once moved outside of a skunkworks project hand-porting it, the rest of the platform needed to be modified to support it. In particular, people were asking when we would support WinPE on AMD64.

Compared to Itanium, supporting AMD64 was a cakewalk. The machines were incredibly fast, ran either old 32-bit versions of Windows as-is, or the new 64-bit code we were working on. It was a very clear direction forward. Where Itanium required special casing things all over the OS, AMD64 just worked. Quickly, we added support for every single setup feature in AMD64.

Where Itanium was years old and had major feature gaps with x86 in setup and when installed, AMD64 actually offered a bonus feature. When you started setup from WinPE, we tweaked winnt32 setup so you could say “install 64-bit” even though it was a 32-bit WinPE. An OEM or corporate customer could easily deploy 32-bit or 64-bit versions of Windows to systems from a single set of discs or RIS server image.

Where Itanium had unique properties all the way up the stack, and only ran software compiled for it. AMD64 ran almost any existing 32-bit application and any newly compiled 64-bit application as-is. The systems were fast, ran cooler, and promised to be much, much cheaper than Itanium. Even if they fell short in the near term, I felt that the long term would surely crown AMD64 the winner. In fact, I wasn’t alone in this belief. When I first went over and talked with a peer on the kernel team who was working on porting Windows to AMD64 (or “Hammer” as it was fondly known at that time, surely a jab at what they wanted to do to Intel, though I never heard that officially) was easy, and he was astonished at how fast it ran. The first time I ran setup (even the old 32-bit setup) on an AMD64, I shared that astonishment. It was very, very, fast.

The initial AMD64 prototypes came in single, dual, and quad-processor configurations. AMD at the time apparently had a musical motif, so the systems were referred to as “Solo”, “Duet”, and “Quartet”, respectively. The dev who was building the Windows imaging (WIM) architecture and tools wanted one of these for his development and test use. I had been able to finagle a few Duets and Solos for the RIS test lab, so this dev asked me if I could find one for him to work with. I inquired, and was told one would be delivered the next day. This dev, who always tended to come in a little later than I, walked down and asked me what the heck that thing in his office was. I had no idea. But we walked down to his office, and he pointed at it. It was a Quartet, one of the four-processor boxes, larger than a mobile cabinet we used with our desks, and also on wheels. We laughed, it obviously being overkill for his needs. I swapped it with one of the smaller systems the RIS test lab had instead.

This experience exemplified the difference to me. As we worked on Windows setup for the next version of Windows (Longhorn, which became Vista), AMD64 became vital. The systems were astonishingly fast, and let us use common build hardware for 32-bit and 64-bit builds of Windows. Moreover, where Windows developers would fight you to keep an Itanium out of their office, they were fighting to get AMD64 systems.

The x64 architecture, though it does not support legacy 16-bit applications or 32-bit drivers, runs 32-bit Windows applications directly without requiring recompilation or any advanced emulation layer. There is a thin software layer that enables this, but the x64 processor runs old x86 applications natively, without emulation. ARM64 Windows 11 now does similar for 32-bit and 64-bit applications designed for Intel processors, potentially making the ARM64 platform viable (if doing so with questionable motivations).

For me, every time I had to work on an Itanium-only feature, or enhancements to EFI or other Itanium platform components, it became frustrating – because I saw the writing on the wall.

Itanium wasn’t the future. In fact, I didn’t think it ever had a future. For Windows at least, it didn’t. In 2010, Microsoft announced the end of support for Itanium, after less than 9 years on the market. Only a handful of Windows workloads had ever been supported on Itanium – those died with it.

It was interesting for me to watch AMD64 grow to become the standard. Inside Windows, we had short names that we used to identify the build of a specific product. X86 was 32-bit Windows, IA64 was Itanium, etc. Early on, since it was AMD’s reference design, the name AMD64 was chosen. Years later, when Intel elected to support (most of, and then eventually all of) AMD’s design, they went with their own names for their design, including EM64T (enhanced memory 64) and Intel 64. There was a discussion internally to Microsoft as to whether we could rename these directories, buried as they were within Windows and Windows setup for most users, from AMD64 to something more benign (or AMD-specific). Thankfully, this effort went nowhere, and the directories are still AMD64 to this day, though the architecture is now generically known in the outside world through the more benign “x64” name.

Perhaps I’m showing my bias. Indeed, it was bold of Intel, of all companies, to try and go completely clean slate with the Itanium. But the cost, abysmal performance, and never-improving lack of software support killed the Itanium.

Because of the split ship (eventually very split ship) of XP and Windows Server 2003, and the urgency of Windows XP Service Pack 2, (SP2), which had to have significant security enhancements and mitigations added to defend against a growing crop of vulnerabilities and exploits, AMD64 support didn’t ship until Windows Server 2003 Service Pack 1 (SP1), in March 2005 (almost a year after I’d left Microsoft in 2004). Given that AMD had delivered the first 64-bit Opteron chips almost two years before, in April 2003, it broke my heart a bit that it had been so delayed. There were reasons – XP needed to be shored up. Windows Server had been broken off into an entirely new branch… But I always wished that Microsoft had been able to ship AMD64 support a bit earlier. AMD had good foresight, and took a big gamble, to try stretching the legacy x86 architecture to be 64-bits. They had a head start. But our internal processes led to the operating system support coming too late to benefit them, and in fact led to plenty of runway for Intel to catch up.

Looking back, now, that legacy birthmark continues to haunt NT. x86, and its younger but fraternal twin, x64, continue to be the only processors that Windows NT has successfully thrived on to date. Time will tell if that ever changes.

Comments are closed.