Background

When I joined the Windows setup team in late 2000, just  ahead of Windows Whistler Beta 2 (the OSs you know as Windows XP and it’s belated sibling, Windows Server 2003), Windows was in this odd limbo when it came to how the OS was put down onto new or existing Windows PCs.

1. Legacy Windows setup and upgrade – how consumers and many smaller businesses would roll out releases of Windows
2. Third-party imaging – where tools would apply a preconfigured image down on every PC, often with user data migrated off the system and back around the new OS image being applied.

Windows setup for Whistler was, as my hiring group program manager called it during my internal interview, “lipstick on a pig”. Whistler was to be an incredibly rapid cycle release after Windows 2000, and was focused on creating the first consumer-centric version of Windows NT. Lots of UI and usability polish, improvements on software, game, and hardware compatibility and a new edition of the OS for consumers (Windows Whistler Personal, which became Windows XP Home edition), but no big overhauls. Although we got a lot done, it was mostly polish, not redesigning things. As Jim Allchin’s mantra for Whistler went at the time, “it just works,” which was supposed to infer that things in the OS “just worked”, when in fact in many cases they juuuuuust worked.

Legacy Windows setup was what consumers and small business customers were used to. It was an amalgam of very early Windows setup technologies, with some UI polish, bubble gum, and duct tape on top, with significant improvements to upgrade and migration – this NT-based setup experience in Whistler was getting very specific features, most of them, like OOBE (the out of box experience) were a focused effort by a peer team within my group, and were intended to make XP feel… less like NT – to add some consumer polish to a platform/codebase that had been business-centric for its entire life. (Fun fact, the much-loved Windows XP tour music was something that my group program manager [GPM] was responsible for shimming in – and that was something that made the team responsible for Windows shell look and feel less than pleased.)

So while we were polishing up setup, OEMs (original equipment manufacturers, like Dell), businesses, and larger organizations, on the other hand, had pushed Microsoft to embrace a new world of imaging. With imaging, Windows was installed, tweaked, configured, applications were installed, updates were applied, and then the whole set of files (I’m being simplistic here) would be picked up by a third-party imaging tool like Symantec Ghost or PowerQuest Drive Image, and repeatedly spackled down onto each new system/hard drive at PC manufacture time.

NT’s Setup was time consuming and sometimes fragile, but it was generally time-tested and well understood. Imaging let OEMs and corporate customers deploy more computers in less time, but not everybody was sold on the idea. While most OEMs were on board with imaging, not all were. Many corporate customers in particular were still using MS-DOS floppies to bootstrap Windows setup from DOS. Heck, I had been using it myself within Microsoft just a few years prior, when I first joined the company. (Note that Windows XP was the last Windows OS that supported starting setup from floppies before loading from the CD.)

One of my tasks as a Program Manager in Windows setup was eventually to help the development team define the tools that would help convince businesses to stop using winnt.exe (the old 16-bit DOS bootstrap for starting Windows setup) and really begin moving forward with imaging—and eventually, a newer model of setup with Longhorn->Vista. But that was years away. Since Windows Whistler was destined to support platforms other than x86 (Itanium out of the gate, but eventually AMD64), there needed to be a replacement for MS-DOS so setup could be started on a bare PC, or imaging tools could be used to apply a clean install of the OS.

When you installed from floppy or CD, Windows had a three-step process to get the OS up and running. The first phase was called, not so creatively, “textmode setup”, because there was really no way to support graphics. We had the ability to interact with devices (storage, networking, etc), interacting with some parts of Windows (video, modems, etc) was limited, if possible at all. Textmode served to get enough of the operating system down onto the hard disk to reboot and begin the next mode of setup, called GUI-mode setup. This was the prettier part of setup from Windows XP and earlier – if you could consider setup pretty. GUI-mode setup then effectively spackled together the installation of the OS you were going to run. It took the files and registry hives (hives are set collections of registry keys) that comprise a Windows install, used .INF files, and built your Windows install from scratch, using plug and play (PnP) to detect the devices installed on the computer, as well as integrating in any migration state if you were performing an upgrade.

It was a rather nasty business, and I was always amazed at how cobbled-together and repetitive Windows’ setup process was. In later years I would use the analogy that this model of setup was “Like Ford building your car in your driveway, and your neighbor’s car in his driveway, even though you both bought the same model car and they arrived at the same time.”

Enter WinPE

It was ironic in some ways, as we had managed to replace MS-DOS with… a 32-bit command prompt (or 64-bit, on the loud, hot, hulking Itanium systems), which was the main feature that DOS offered at that point.

But WinPE was tiny (about 140MB, versus 1GB or more for a full install of Windows), it ran from a CD instead of a floppy, and it offered a stable 32-bit NT-based command-prompt, with networking and mass-storage support identical to that of the Windows release it was built from. Classically to install Windows NT, you had to find at least mass-storage drivers, and often networking drivers, for both DOS and NT just to get NT installed. Finding DOS drivers for modern hardware in 2001 was becoming more and more difficult. (This was a problem that became massive during the lifecycle of Windows XP, as better storage and networking technologies arrived after the OS shipped – but we couldn’t do much about it. Sorry, Greg.)

The initial enthusiasm around Microsoft for WinPE was amazing. It was fun to be a part of a team building technologies that made life easier for so many people. More importantly, most of the OEMs were jazzed about the promise of the platform. While I would later get burned by a weird, complex feature request that they made and never used, Dell’s manufacturing team in particular was invaluable, and helped to generate the idea for one of my favorite things that occurred while I worked on Windows. Dell came to us with a list of manufacturability concerns. In particular, two things they wanted were robust scripting support and a Web browser. Another OEM requested the ability to run Microsoft Access on WinPE.

Initially taken aback by these requests when I first saw them in the spring of 2001, I came to view addressing them as a personal challenge, although what I would deliver for the second and third requests would be very different, for strategic reasons.

Years before Whistler, Microsoft had begun building a version of Windows NT – NT Embedded – for use in devices such as hotel kiosks, ATMs, etc. NT Embedded focused on making the OS lightweight, by only including the sections of the OS that were needed for the role of the device that is needed. Whistler was getting an update of this same functionality, with even more granular breakdown, which would allow Windows XP to invade embedded devices around the world. This effort, called “componentization” was an uphill battle for the Windows Embedded team because, like setup, dependencies were taken in parts of Windows with little regard for the cost; teams never designed Windows features with the idea of “what’s the bare minimum things I need for this feature to work”. Windows was Windows – you could take a dependency on anything in that version and edition of the OS, and assume that it would be there when you needed it. Since Embedded was to run on lower-cost and lower-capability hardware, Windows had to lose weight.

Dependencies are code that your code takes in order to run. For example, huge chunks of Windows took dependencies on the Windows shell (Explorer.exe, IE, and a ton of other pieces), which in turn took huge dependencies on others. It makes sense in a vacuum, because Windows is Windows. Explorer, Internet Explorer, Control Panel… these are just parts of Windows.

But if you quickly add that up, you can wind up with really weird cyclical dependencies where a tiny control panel applet (.CPL file) takes a dependency on a huge chunk of the Windows shell, and RAM, CPU, and disk requirements for the OS blow out with it.. Need that control panel applet, but don’t need or want the entire Windows shell? Tough, unless you can convince a developer to change the code.

So it was that the dependencies of Windows were beginning to be unwound – but that effort would continue and continue. (And now, it’s largely abandoned – but that’s another story for another time.)

I bring this up because componentization helped me, to a degree, to unwind the three hacks I wanted to do with WinPE.

Windows Script Host

While WinPE supported batch/cmd files out of the gate, customers immediately asked about running Windows Script Host on WinPE. Amusingly, this was near and dear to my heart, as my first job at Microsoft had been doing ASP-based web-development, and I wrote a ton of WSH scripts to do automation for our training system. So I was quite familiar with the internals and infrastructure of WSH. (Some have asked why we didn’t add PowerShell… but we shipped long before PowerShell did, and PS took a hard dependency on the .NET Framework, which did not play well with WinPE.)

Trying to find contacts to help break down the components of WSH had been easy, as there was a simple inf file (setup information file) that contained the definition of everything WSH needed on Windows to run – because WSH had originally been distributed out-of-band, not in Windows itself.

I wrote a hacky script that manually installed all of the files for WSH (amusingly it couldn’t be a WSH script, because WSH wasn’t installed yet). It worked the very first time I ran it.

Scripting… done.

HTA Applications

Then I moved onto the second wish-list item, “a web browser”.

I hemmed and hawed on this one a lot. I knew Web browsers. I knew Dell’s scenarios, and those of my corporate customers. A Web browser lets you show Web pages, but doesn’t let you build an application – more importantly, the security boundary of a Web browser doesn’t let you do much with the local computer.

I stepped back with the OEMs to find out what they wanted to do. While many said, “I want a web browser”, I knew better. WinPE was all about automation, and a web browser isn’t an automation tool. First off, IE was a pig, and the dependencies were a mess. Secondarily, what they wanted was “forms”. They wanted the ability to have users or admins type things in forms and then hit “go”, and install the OS, etc. The security model of IE would never let that work.

However, Microsoft had a tiny project that had seemingly died off, but which fit the bill perfectly. HTML Applications – or HTA files for short, were Web pages that either ran in a Window or full-screen, and once started, had free-reign over Windows. You could do anything. HTA gave you a light local browser for HTML files and did not give a crap about security. Want an HTA that nukes the hard disk, repartitions it, and installs the OS? Easy. Want an HTA that reformats the disk entirely? Easy…

I wrote a sample application “wizard” that walked through and asked you questions about creating or formatting disk partitions and pretended to do so, and showed it to Dell. Though it wasn’t a “Web browser” per se, they agreed it met their needs. Now I just had to figure out how to shim it into WinPE.

Unfortunately, HTA never shipped as its own component. It had only shipped as a feature of Internet Explorer, which in turn required the Windows shell. Like I said, it was a pig.

There was no way I could take a dependency stack that big. I had to find someone who knew the internals. I reached out to the Windows shell team, who in turn pointed me to a veteran dev on the IE team. Though he wasn’t immediately well versed in the dependencies HTA itself would require, we sat down for an afternoon and tried debugging it while running WinPE from a PC I had brought over with me. With a fair amount of work, we were able to isolate it down so we knew almost all of the dependencies to get HTA working. Was it perfect? No. Did it work and meet the basic needs? Absolutely.

“Web browser”… done.

Database Connectivity

As both Dell, other OEMs, and a few of my corporate customers in the early Joint Development Program (JDP) FOLLOWUP realized, there was one other feature that would be handy – the ability to connect out to a database. While Microsoft had a handy redistributable that included a bunch of “Microsoft Data Access Components” (MDAC), it was monolithic, because it assumed you were running on a full install of Windows, and brought in some thick dependencies.

The early requests had been to run Access on WinPE itself. I shot those down early. It’s nonsensical to run a database locally on a read-only, lightweight OS. Plus I hated Access. I’d been burned by it early on at Microsoft and swore by SQL Server instead.

So I figured that the thing I needed most was the ability to connect to a big SQL Server database using a data connectivity protocol called ActiveX Database Objects (ADO). Thinking that OEM and corporate customers would likely both want to have a sort of bill of materials (BOM) for each PC that it could retrieve from the database, I wanted to get this working.

Because MDAC shipped separately, I was able to get an idea as to what things it required to run. But I vividly recall how mind-numbingly hard it was to figure out what the dependencies actually were. The two devs now working on WinPE had come up with a way to run it from a HDD instead of just CD. This made tinkering with it faster, but gave me a false sense of confidence.

Namely, I had hacked everything together to run ADO to SQL Server from HDD, and I was able to connect to a SQL Server on the network. I made a CD image and burned a CD-R (not a fast feat at the time) and it didn’t work. It did not work at all. The same code that worked on a hard disk, failed when running from shiny media.

I couldn’t figure it out. I can vividly picture in my mind the evening (actually crazy late the next morning after tinkering all night) I got it working. I stayed up late after my wife went to bed, sitting at my home computer, beating on this to try and get it to work. At about 3 in the morning, I figured it out. Even with only a couple of hours of sleep I went in early to work the next day, elated that I had figured it out.

In hindsight it was completely obvious why ADO hadn’t worked. Like WSH and HTA before it, I was trying to “install” ADO when the WinPE CD finished booting. You run a script, register a few DLLs and run a few inf files, and boom… done.

But nobody on the MDAC team had ever thought about Windows running from read-only media. When you would try to register several of the key DLLs for ADO, it failed – because it was trying to create a temporary directory, vomit out a few installer files, and run them. But neither of those operations could work on an OS booted from CD, so… the install silently died. I figured out a way to grab the installer files, and create the directory that the installer wanted to be there… and have them all already on the CD. Once that was burned, it worked perfectly.

ADO to SQL Server… done.

This sort of all harkens back to my point earlier. What I built wasn’t elegant. It wasn’t clean. But it worked. Much of what we did with WinPE was jam tools together to the point that they worked – and it continued that way through Longhorn.

Buildoptionalcomponents.vbs

As a part of WinPE, a bunch of tools were made available for OEMs and enterprises (some) so they could craft their own WinPE build. A script took the appropriate files from a Windows CD/share, and “built” WinPE out of it, and then could write it to an ISO file to be burned to a CD.

Conceptually I guess I could have tried to make my hacks more integrated into the WinPE build process, but I didn’t. Either I didn’t think to do it, or quite possibly the idea wasn’t warmly welcomed by development.

Regardless, I came up with a solution, which was (tada!) to write another script. It did the same thing as the WinPE batch file – grabbed source files, made the installers and some samples, and threw them all into your WinPE build.

I wasn’t ever taught to write code. (Think of Ferris Bueller on the clarinet. Never had one lesson…) As a result, this thing was an absolute mess. It was long, chaotic, had weird logic, had no error handling, and couldn’t ever be localized. A year after I checked it in, I was asked to localize it and add error handling, and I honestly had no idea where to start. So when someone on Twitter the other days said, “point us to the repository with the worst code you’ve ever written”, I said they should go check this script.

One of my key regrets was honestly… not naming the dang file something shorter.

A Learning Opportunity

In the end, while I’m not proud of the script itself, I am proud of what we did with WinPE, and these three hacks I came up with. They were used frequently for easily more than a decade (heck, probably still used even though they probably shouldn’t be), and used by many teams across Microsoft. Working on Windows was a humbling experience. I learned that below the image that you’re used to seeing within Windows often hides a gory underbelly that’s not quite as refined as you might think. I worked with some absolutely wonderful and brilliant people, made some great tools, and had fun. (Also made some contacts that led to my next role after Microsoft, but that’s a topic for another day.)