22
Sep 15

You have the right… to reverse engineer

This NYTimes article about the VW diesel issue and the DMCA made me think about how, 10 years ago next month, the Digital Millennium Copyright Act (DMCA) almost kept Mark Russinovich from disclosing the Sony BMG Rootkit. While the DMCA provides exceptions for reporting security vulnerabilities, it does nothing to allow for reporting breaches of… integrity.

I believe that we need to consider an expansion of how researchers are permitted to, without question, reverse engineer certain systems. While entities need a level of protection in terms of their copyright and their ability to protect their IP, VW’s behavior highlights the risks to all of us when of commercial entities can ship black box code and ensure nobody can question it – technically or legally.

In October of 2005, Mark learned that a putting a particular Sony BMG CD in a Windows computer would result in it installing a rootkit. Simplistically, a rootkit is a piece of software – usually installed by malicious individuals – that sits at a low level within the operating system and returns forged results when a piece of software at a higher level asks the operating system to perform an action. Rootkits are usually put in place to allow malware to hide. In this case, the rootkit was being put in place to prevent CDs from being copied. Basically, a lame attempt at digital rights management (DRM) gone too far.

In late October, Mark researched this, and prepped a blog post outlining what was going on. We talked at length, as he was concerned that his debugging and disclosure of the rootkit might violate the DMCA, a piece of legislation put in place to protect copyrights and prevent reverse engineering of DRM software, among other things. So in essence, to stop exactly what Mark had done. I read over the DMCA several times during the last week of October, and although I’m not a lawyer, I was pretty satisfied that Mark’s actions fit smack dab within the part of the DMCA that was placed there to enable security professionals to diagnose and report security holes. The rootkit that Sony BMG had used to “protect” their CD media had several issues in it, and was indeed creating security holes that were endangering the integrity of Windows systems where the software had unwittingly been installed.

Mark decided to go ahead and publish the blog post announcing the rootkit on October 31, 2005 – Halloween. Within 48 hours, Mark was being pulled in on television interviews, quoted in major press publications, and was repeatedly a headline on Slashdot, the open-source focused news site over the next several months – an interesting occurrence for someone who had spent almost his entire career in the Windows realm.

The Sony BMG disclosure was very important – but it almost never happened. Exceptions that allow reverse engineering are great. But security isn’t the only kind of integrity that researchers need to diagnose today. I don’t think we should tolerate laws that keep researchers from ensuring our systems are secure, and that they operate the way that we’ve been told they do.


22
May 15

Farewell, floppy diskette

I never would have imagined myself in an arm-wrestling match with the floppy disk drive. But sitting where I did in Windows setup, that’s exactly what happened. A few times.

When I had started at Microsoft, a boot floppy was critical to setting up a new machine. Not by the time I was in setup. Since Remote Installation Services (RIS) could start with a completely blank machine, and you could now boot a system to WinPE using a CD, there were two good-sized nails in the floppy diskette’s coffin.

Windows XP was actually the first version of Windows that didn’t ship with boot floppies. It only shipped with a CD. While you could download a tool that would build boot floppies for you, most computers that XP happily ran on supported CD boot by that time. The writing was on the wall for the floppy diskette. In the months after XP released, Bill Gates made an appearance on the American television sitcom Frasier. Early in the episode, a caller asks about whether they need diskettes to install Windows XP. For those of us on the team, it was amusing. Unfortunately, the reality was that behind the scenes, there were some issues with customers whose systems didn’t boot from CD, or didn’t boot properly, anyway. We made it through most of those those birthing pains, though.

It was both a bit amusing and a bit frustrating to watch OEMs early on during the early days of Windows XP; while customers often said, “I want a legacy free system”, they didn’t know what that really meant. By “legacy free”, customers usually meant they wanted to abandon all of the legacy connectors (ports) and peripherals used on computers before USB had started to hit its stride with Windows 98.

While USB had replaced serial in terms of mice – which were at one time primarily serial – the serial port, parallel port, and floppy disk controller often came integrated together in the computer. We saw some OEMs not include a parallel port, and eventually not include a floppy diskette, but still include a serial port – at least inside – for when you needed to debug the computer. When a Windows machine has software problems, you often hook it up to a debugger, an application on another computer, where the developer can “step through” the programming code to figure out what is misbehaving. When Windows XP shipped, a serial cable connection was the primary way to debug.  Often, to make the system seem more legacy free than it actually was, this serial port was tucked inside the computer’s case – which made consumers “think” it was legacy free when it technically wasn’t. PCs often needed BIOS updates, too – and even when Windows XP shipped with them, these PCs would still usually boot to an MS-DOS diskette in order to update the BIOS.

My arrival in the Windows division was timely; when I started, USB Flash Drives (UFDs) were just beginning to catch on, but had very little storage space, and the cheapest ones were slow and unreliable. 32MB and 64MB drives were around, but still not commonplace. In early 2002, the idea of USB booting an OS began circling around the Web, and I talked with a few developers within The Firm about it. Unfortunately, there wasn’t a good understanding of what would need to happen for it to work, nor was the UFD hardware really there yet. I tabled the idea for a year, but came back to it every once in a while, trying to research the missing parts.

As I tinkered with it, I found that while many computers supported boot from USB, they only supported USB floppy drives (a ramshackle device that had come about, and largely survived for another 5-10 years, because we were unable to make key changes to Windows that would have helped killed it). I started working with a couple of people around Microsoft to try and glue the pieces together to get WinPE booting from a UFD. I was able to find a PC that would try to boot from the disk, and failed because the disk wasn’t prepared for boot as a hard disk normally would be. I worked with a developer from the Windows kernel team and one of our architects to get a disk formatted correctly. Windows didn’t like to format UFDs as bootable because they were removable drives; even Windows to Go in Windows 8.1 today boots from special UFDs which are exceptionally fast, and actually lie to the operating system about being removable disks. Finally, I worked with another developer who knew the USB stack when we hit a few issues booting. By early 2003, we had a pretty reliable prototype that worked on my Motion Computing Tablet PC.

Getting USB boot working with Windows was one of the most enjoyable features I ever worked on, although it wasn’t a formal project in my review goals (brilliant!). USB boot was even fun to talk about, amongst co-workers and Microsoft field employees. You could mention the idea to people and they just got it. We were finally killing the floppy diskette. This was going to be the new way to boot and repair a PC. Evangelists, OEM representatives, and UFD vendors came out of the woodwork to try and help us get the effort tested and working. One UFD manufacturer gave me a stash of 128MB and larger drives – very expensive at the time – to prepare and hand out to major PC OEMs. It gave us a way to test, and gave the UFD vendor some face time with the OEMs.

For a while, I had a shoebox full of UFDs in my office which were used for testing; teammates from the Windows team would often email or stop by asking to get a UFD prepped so they could boot from it. I helped field employees get it working so many times that for a while, my nickname from some in the Microsoft field was “thumbdrive”, one of the many terms used to refer to UFDs.

Though we never were able to get UFD booting locked in as an official feature until Windows Vista, OEMs used it before then, and it began to go mainstream. Today, you’d be hard pressed to find a modern PC that can’t boot from UFD, though the experience of getting there is a bit of a pain, since the PC boot experience, even with new EFI firmware, still (frankly) sucks.

Computers usually boot from their HDD all the time. But when something goes wrong, or you want to reinstall, you have to boot from something else; a UFD, CD/DVD, PXE server like RIS/WDS, or sometimes an external HDD. Telling your Windows computer what to boot from if something happens is a pain. You have to hit a certain key sequence that is often unique to each OEM. Then you often have to hit yet another key (like F12) to PXE boot. It’s a user experience only a geek could love. One of my ideas was to try and make it easier not only for Windows to update the BIOS itself, but for the user to more easily say what they wanted to boot the PC from (before they shut it down, or selecting from a pretty list of icons or a set of keys – like Macs can do). Unfortunately, this effort largely stalled out for over a decade until Microsoft delivered a better recovery, boot, and firmware experience with their Surface tablets. Time will tell whether we’re headed towards a world where this isn’t such a nuisance anymore.

It’s actually somewhat amusing how much of my work revolved around hardware even though I worked in an area of Windows which only made software. But if there was one commonly requested design change request that I wish I could have accommodated but couldn’t ever get done, it was F6 from UFD. Let me explain.

When you install Windows, it attempts to use the drivers it ships with on the CD to begin copying Windows down onto the HDD, or to connect over the network to start setup through RIS.

This approach worked alright, but it had one little problem which became significant. Not long after Windows XP shipped, new categories of networking and storage devices began arriving on high-end computers and rapidly making their way downmarket; these all required new drivers in order for Windows to work. Unfortunately, none of these drivers were “in the box” (on the Windows CD) as we liked to say. While Windows Server often needed special drivers to install on some high-end storage controllers before, this was really a new problem for the Windows consumer client. All of a sudden we didn’t have drivers on the CD for the devices that were shipping on a rapidly increasing number of new PCs.

In other words, even with a new computer and a stock Windows XP CD in your hand, you might never get it working. You needed another computer and a floppy diskette to get the ball rolling.

Early on during Windows XP’s setup, it asks you to press the keyboard’s F6 function key if you have special drivers to install. If it can’t find the network and you’re installing from CD, you’ll be okay through setup – but then you have no way to add new drivers or connect to Windows Update. If you were installing through RIS and you had no appropriate network driver, setup would fail. Similarly, if you had no driver for the storage controller on your PC, it wouldn’t ever find find a HDD where it could install Windows – so it would terminally fail too. It wasn’t pretty.

Here’s where it gets ugly. As I mentioned, we were entering an era where OEMs wanted to ship, and often were shipping, those legacy-free PCs. These computers often had no built-in floppy diskette – which was the only place we could look for F6 drivers at the time. As a result, not long after we shipped Windows XP, we got a series of design change requests (DCRs) from OEMs and large customers to make it so Windows setup could search any attached UFD for drivers as well. While this idea sounds easy, it isn’t. This meant having to add Windows USB code into the Windows kernel so it could search for the drives very early on, before Windows itself has actually loaded and started the normal USB stack. While we could consider doing this for a full release of Windows, it wasn’t something that we could easily do in a service pack – and all of this came to a head in 2002.

Dell was the first company to ever request that we add UFD F6 support. I worked with the kernel team, and we had to say no – the risk of breaking a key part of Windows setup was too great for a service pack or a hotfix, because of the complexity of the change, as I mentioned. Later, a very large bank requested it as well. We had to say no then as well. In a twist of fate, at Winternals I would later become friends with one of the people who had triggered that request, back when he was working on a project onsite at that bank.

Not adding UFD F6 support was, I believe, a mistake. I should have pushed harder, and we should have bitten the bullet in testing it. As a result of us not doing it, a weird little cottage industry of USB floppy diskette drives continued for probably a decade longer than it should have.

So it was, several years after I left, that the much maligned Windows Vista brought both USB boot of WinPE and also brought USB F6 support so you could install the operating system on hardware with drivers newer than Windows XP, and not need a floppy diskette drive to get through setup.

As I sit here writing this, it’s interesting to consider the death of CD/DVD media (“shiny media”, as I often call it) on mainstream computers today. When Apple dropped shiny media on the MacBook Air, people called them nuts – much as they did when Apple dropped the floppy diskette on the original iMac years before. As tablets and Ultrabooks have finally dropped shiny media drives, there’s an odd echo of the floppy drive from years ago. Where external floppy drives were needed for specific scenarios (recovery and deployment), external shiny media drives are still used today for movies, some storage and installation of legacy software. But in a few years, shiny media will be all but dead – replaced by ubiquitous high-speed wired and wireless networking and pervasive USB storage. Funny to see the circle completed.


06
Aug 14

My path forward

Note: I’m not leaving Seattle, or leaving Directions on Microsoft. I just thought I would share the departure email I sent in 2004. Today, August 6, 2014 marks the tenth anniversary of the day I left Microsoft and Seattle to work at Winternals in Austin. For those who don’t know – earlier that day, Steve Ballmer had sent a company-wide memo entitled “Our path forward”, hence my tongue-in cheek subject selection.

From: Wes Miller
Sent: Tuesday, July 06, 2004 2:32 PM
To: Wes Miller
Subject: My path forward

Seven years ago, when I moved up from San Jose to join Microsoft, I wondered if I was doing the right thing… Not that I was all that elated working where I was, but rather we all achieve a certain level of comfort in what we know, and we fear that which we don’t know. I look back on the last seven years and it’s been an amazing, fun, challenging, and sometimes stressful experience – experiences that I would never trade for anything.

At the same time, for family reasons and for personal reasons, I’ve had to do some soul searching that retraced the memories I have from, and steps I went through when I initially came to Microsoft, and I have accepted a position working for a small software company in Austin, TX. My last day at Microsoft will be Friday August 6, one month from today. The best way to reach me after that until my new address is set up is <redacted>. Between now and August 6th I will be doing my best to meet with any of you that need closure on deployment or LH VPC related issues before my departure. Please do let me know if you need something from me between now and then.

Many thanks to those of you who I have worked with over the years – take care of yourselves, and stay in touch.

Thanks,
Wes


27
Apr 14

Job titles are free.

“The Sunscreen song”, which is actually named “Everybody’s Free (to Wear Sunscreen)”, by Baz Luhrmann, has been a (potentially odd) source of wisdom for me since it came out in 1998, just a few years after I graduated from college. I listen to the song periodically, and try to share it with my kids who, at 9 and 13, don’t yet “get” it.

The words of the song aren’t those of the artist, and they aren’t Kurt Vonnegut’s either, regardless of what urban legend says. No, the words come from Mary Schmich’s 1997 Chicago Tribune column, “Advice, like youth, probably just wasted on the young.” Much like Desiderata, the article attempted to gently deliver nuggets of wisdom about life to a younger generation – in this case as if Mary were delivering a graduation speech.

For years, I pondered how best to share my thoughts on surviving in the work world. While college prepares us for the world by chucking text at us page by page, it often can’t show us the deeper machinations of how the work world happens.

I present to you a collection of some of my thoughts about making the most of your career.

 

Ladies and gentlemen of the incoming workforce of 2014;
Job titles are free.

It’s true. You’ll bump into all sorts of people in your career, with lots of fancy, frilly titles. Chief of this. Executive of that. Founder of something you’ve never heard of.
Remember that titles cost nothing to hand out, and business cards are cheap to print.

Every time you go in for an interview, remember you’re interviewing the job just as much as the job is interviewing you. These are the people you’ll be working with as well as the job you’ll be working at.

Always ask, “Why did the last person in this position leave?”

Don’t settle.

Salary isn’t everything, but salary isn’t unimportant. Pats on the back won’t pay the electric bill. But if you’re only working somewhere because the pay is great, you’re cheating your colleagues, your employer, and yourself.

Typecasting isn’t just for actors. Don’t sit still. Always be working to improve yourself and your skills.

An employer who doesn’t value you improving your knowledge through training and doesn’t help you grow doesn’t value you. Don’t value them.

Age doesn’t equate to wisdom, and neither do words printed on a piece of paper in a frame on the wall. Wisdom almost exclusively arrives through experience, and experience results in both failures and successes. Humility comes from living through life’s failures, life’s successes, and learning over time that both can deliver valuable lessons.

“It seemed like a good idea at the time.” Whenever you run across the bad decisions of others who preceded you, shake your head, laugh, and repeat this to yourself. Make a plan and move forward. Don’t complain.

Consider yourself lucky if you ever work somewhere that an executive steps down because they, themselves (not the board) realize that someone else could do the job better than they could.

Murder your darlings. Suffer for your art. Take criticism as sunlight and water, and let it help you grow.

Simplify.

Surround yourself with people who make you wish you were smarter. Bolt from jobs where you’re always the smartest person in the room.

Value people who say “I don’t know” and ask “what do you need?”, guard yourself from people who keep secrets and never ask for help when things are going wrong.

Hiring the right people is hard. Hiring the wrong people is harder.

Firing someone, or laying someone off, is never fun.
Getting fired, or getting laid off, is never fun.

If your product or service isn’t selling, it’s probably not the marketing. It’s probably the product.

Perhaps you’ll find yourself at a startup. In such a situation, beware of strangers offering you sweat equity. Usually you’ll sweat, and they’ll get the equity.

There is no silver bullet.

You’ll probably find several stops along the way where “outsourcing” will be tossed out as the solution to a problem. With a perfect definition of the problem, a clear budget, and good management, it can be. Lacking any one of those three steps, you’ve got two problems instead.

Features, quality, or date. Choose any two.

In your career, you will likely have a spectrum of managers. Some will micromanage you, which is usually a result of their fear of failure and your failure to communicate with them enough to make them comfortable. Other managers will be so remote that you may fear failure, and feel like they aren’t communicating with you enough to make you comfortable. Communicate and collaborate, and it’ll all be fine.

When you find problems, point them out. If others around you tell you to keep it quiet, then they’re part of the problem too. If others above you tell you to keep it quiet, then you’ve got a real problem. Matches can become bonfires if you let them burn long enough.

If you make bad decisions, take the blame. If others make bad decisions, don’t feel the need to blame them.

Always be on the lookout for your next move. You may find yourself in a role that fits you from college to retirement. You may move to a new opportunity every few years. The main thing is to be cognizant that nothing is permanent, nothing is forever, and you should know what you would do the next day if your card-key stops working to unlock the door.

Do something you’re passionate about. If you’re not passionate about the thing you’re doing, you’re probably doing the wrong thing.

Meetings. Emails. Letters. Have a point, or there isn’t one.

Brevity.

Throughout your career, you will run into people whose primary skill is peacock language. They’ll tell you about themselves, strut around trying to look important, and talk in perfectly cromulent phrases. Smile to yourself, and remember that job titles are free.

 

An amendment: ˆTwo more sentiments I regret not adding to the above:

  • The unspoken word never needs to be taken back.
  • Burned bridges are hard to walk across when you need them.

I’m kind of surprised I forgot to put the first one one. It’s one of the earliest lessons I learned about work – through my father’s experiences, specifically around things that were said when leaving a job. Hint: If you think you might regret saying something to someone later, don’t. Just a good rule of thumb for life.