28
Aug 16

It doesn’t have to be a crapfest

A  bit ago, this blog post crossed my Twitter feed. I read it, and while the schadenfreude made me smirk for a minute, it eventually made me feel bad.

The blog post purports to describe how a shitty shutdown dialog became a shitty shutdown dialog. But instead, it documents something I like to call “too many puppies” syndrome. If you are working on high visibility areas of a product – like the Windows Shell – like Explorer in particular, everybody has an belief that their opinion is the right direction. It’s like dogs and a fire hydrant. My point really isn’t to be derisive here, but to point out that the failure of that project does not seem to be due to any other teams. Instead, it seems to have been due to some combination of unclear goals and a fair amount of the team he was on being lost in the wilderness.

I mentioned on Twitter that, if you are familiar with the organizational structure of Windows, that you can see the cut lines of those teams in the UI. A reply to that mentioned Conway’s law – which I was unfamiliar with, but basically states that as a general concept, a system designed by an organization will reflect the structure of that organization.

But not every project is doomed to live inside its own silo. In fact, some of my favorite projects that I worked on while I was at The Firm were ones that fought the silo, and the user won. Unfortunately, this was novel then, and still feels novel now.

During the development of Windows Server 2003, Bill Veghte, a relatively new VP on the product, led a series of reviews where he had program managers (PMs) across the product walk through their feature area/user scenario, to see how it worked, didn’t work, and how things could perhaps be improved. Owning the enterprise deployment experience for Windows at the time, I had the (mis?)fortune of walking Bill through the setup and configuration experience with a bunch of people from the Windows Server team.

When I had joined the Windows “Whistler” team just before beta 2, the OS that became Windows XP was described by a teammate as a “lipstick on a chicken” release was already solidifying, and while we had big dreams of future releases like “Blackcomb” (never happened), Whistler was limited largely by time to the goal of shipping the first NT-based OS to both replace ME and the 9X family for consumers, and Windows 2000 in business.

Windows Server, on the other hand, was to ship later. (In reality, much, much later, on a branched source tree, due to the need to <ahem/> revisit XP a few times after we shipped it.) This meant that the Windows Server team could think a bit bigger about shipping the best product for their customers. These scenario reviews, which I really enjoyed attending at the time, were intended to shake out the rattles in the product and figure out how to make it better.

During my scenario review, we walked through the entire setup experience – from booting the CD to configuring the server. If you recall, this meant walking through some really ugly bits of Windows. Text-mode setup. F5 and F6 function keys to install a custom HAL or mass-storage controller drivers during text-mode setup. Formatting a disk in text-mode setup. GUI-mode setup. Fun, fun stuff.

Also, some forget, but this was the first time that Windows Server was likely to ship with different branding from the client OS. Yet the Windows client branding was… everywhere. Setup “billboards” touting OS features that were irrelevant in a server, wizards, help files, even the fact that setup was loading drivers for PCMCIA cards and other peripherals that a server would never need or use in the real world, or verbs on the shutdown menu that made no sense on a server, like standby or hibernate.

A small team of individuals on the server team owned the resulting output from these walkthroughs, which went far beyond setup, and resulted in a bunch of changes to how Windows Server was configured, managed, and more. In terms of my role, I wound up being their liaison for design change requests (DCRs) on the Windows setup team.

There were a bunch of things that were no-brainers – fixing Windows Setup to be branded with Windows Server branding, for example. And there were a ton of changes that, while good ideas, were just too invasive to change, given the timeframe that Windows Server was expected to ship in, (and that it was still tethered to XP’s codebase at that time, IIRC). So lots of things were punted out to Blackcomb, etc.

One of my favorite topics of discussion, however, became the Start menu. While Windows XP shipped with a bunch of consumer items in the Start menu, almost everything it put there was… less than optimal on a server. IE, Outlook Express, and… Movie Maker? Heck, the last DCR I had to say no to for XP was a very major customer telling us they didn’t even want movie maker in Windows XP Pro! It had no place on servers – nor did Solitaire or the Windows XP tour.

So it became a small thing that David, my peer on the server team, and I tinkered with. I threw together a mockup and sent it to him. (It looked a lot like the finished product you see in this article.) No consumer gunk. But tools that a server administrator might use regularly. David ran this and a bunch of other ideas by some MVPs at an event on campus, and even received applause for their work.

As I recall, I introduced David to Raymond Chen, the guru of all things Windows shell, and Raymond and David wound up working together to resolve several requests that the Windows Server team had in the user interface realm. In the end, Windows Server 2003 (and Server SP1, which brought x64 support) wound up being really important releases to the company, and I think they reflected the beginning of a new maturity at Microsoft on building a server product that really felt… like a server.

The important thing to remember is that there wasn’t really any sort of vehicle to reflect cross-team collaboration within the company then. (I don’t know if there is today.) It generally wasn’t in your review goals (those all usually reflected features in your team’s immediate areas), and compensation surely didn’t reflect it. I sat down with David this week, having not talked for some time, and told him how most of my favorite memories of Microsoft were working on cross-team projects where I helped other teams deliver better experiences by refining where their product/feature crossed over into our area, and sometimes beyond.

I think that if you can look deeply in a product or service that you’re building, and see Conway’s law in action, you need to take a step back. Because you’re building a product for yourself, not for your customers. Building products and services that serve your entire customer base means always collaborating, and stretching the boundaries of what defines “your team”. I believe the project cited in the original blog post I referenced above failed both because there were too many cooks, but also because it would seem that anyone with any power to control the conversation actually forgot what they were cooking.

 

 


22
May 15

Farewell, floppy diskette

I never would have imagined myself in an arm-wrestling match with the floppy disk drive. But sitting where I did in Windows setup, that’s exactly what happened. A few times.

When I had started at Microsoft, a boot floppy was critical to setting up a new machine. Not by the time I was in setup. Since Remote Installation Services (RIS) could start with a completely blank machine, and you could now boot a system to WinPE using a CD, there were two good-sized nails in the floppy diskette’s coffin.

Windows XP was actually the first version of Windows that didn’t ship with boot floppies. It only shipped with a CD. While you could download a tool that would build boot floppies for you, most computers that XP happily ran on supported CD boot by that time. The writing was on the wall for the floppy diskette. In the months after XP released, Bill Gates made an appearance on the American television sitcom Frasier. Early in the episode, a caller asks about whether they need diskettes to install Windows XP. For those of us on the team, it was amusing. Unfortunately, the reality was that behind the scenes, there were some issues with customers whose systems didn’t boot from CD, or didn’t boot properly, anyway. We made it through most of those those birthing pains, though.

It was both a bit amusing and a bit frustrating to watch OEMs early on during the early days of Windows XP; while customers often said, “I want a legacy free system”, they didn’t know what that really meant. By “legacy free”, customers usually meant they wanted to abandon all of the legacy connectors (ports) and peripherals used on computers before USB had started to hit its stride with Windows 98.

While USB had replaced serial in terms of mice – which were at one time primarily serial – the serial port, parallel port, and floppy disk controller often came integrated together in the computer. We saw some OEMs not include a parallel port, and eventually not include a floppy diskette, but still include a serial port – at least inside – for when you needed to debug the computer. When a Windows machine has software problems, you often hook it up to a debugger, an application on another computer, where the developer can “step through” the programming code to figure out what is misbehaving. When Windows XP shipped, a serial cable connection was the primary way to debug.  Often, to make the system seem more legacy free than it actually was, this serial port was tucked inside the computer’s case – which made consumers “think” it was legacy free when it technically wasn’t. PCs often needed BIOS updates, too – and even when Windows XP shipped with them, these PCs would still usually boot to an MS-DOS diskette in order to update the BIOS.

My arrival in the Windows division was timely; when I started, USB Flash Drives (UFDs) were just beginning to catch on, but had very little storage space, and the cheapest ones were slow and unreliable. 32MB and 64MB drives were around, but still not commonplace. In early 2002, the idea of USB booting an OS began circling around the Web, and I talked with a few developers within The Firm about it. Unfortunately, there wasn’t a good understanding of what would need to happen for it to work, nor was the UFD hardware really there yet. I tabled the idea for a year, but came back to it every once in a while, trying to research the missing parts.

As I tinkered with it, I found that while many computers supported boot from USB, they only supported USB floppy drives (a ramshackle device that had come about, and largely survived for another 5-10 years, because we were unable to make key changes to Windows that would have helped killed it). I started working with a couple of people around Microsoft to try and glue the pieces together to get WinPE booting from a UFD. I was able to find a PC that would try to boot from the disk, and failed because the disk wasn’t prepared for boot as a hard disk normally would be. I worked with a developer from the Windows kernel team and one of our architects to get a disk formatted correctly. Windows didn’t like to format UFDs as bootable because they were removable drives; even Windows to Go in Windows 8.1 today boots from special UFDs which are exceptionally fast, and actually lie to the operating system about being removable disks. Finally, I worked with another developer who knew the USB stack when we hit a few issues booting. By early 2003, we had a pretty reliable prototype that worked on my Motion Computing Tablet PC.

Getting USB boot working with Windows was one of the most enjoyable features I ever worked on, although it wasn’t a formal project in my review goals (brilliant!). USB boot was even fun to talk about, amongst co-workers and Microsoft field employees. You could mention the idea to people and they just got it. We were finally killing the floppy diskette. This was going to be the new way to boot and repair a PC. Evangelists, OEM representatives, and UFD vendors came out of the woodwork to try and help us get the effort tested and working. One UFD manufacturer gave me a stash of 128MB and larger drives – very expensive at the time – to prepare and hand out to major PC OEMs. It gave us a way to test, and gave the UFD vendor some face time with the OEMs.

For a while, I had a shoebox full of UFDs in my office which were used for testing; teammates from the Windows team would often email or stop by asking to get a UFD prepped so they could boot from it. I helped field employees get it working so many times that for a while, my nickname from some in the Microsoft field was “thumbdrive”, one of the many terms used to refer to UFDs.

Though we never were able to get UFD booting locked in as an official feature until Windows Vista, OEMs used it before then, and it began to go mainstream. Today, you’d be hard pressed to find a modern PC that can’t boot from UFD, though the experience of getting there is a bit of a pain, since the PC boot experience, even with new EFI firmware, still (frankly) sucks.

Computers usually boot from their HDD all the time. But when something goes wrong, or you want to reinstall, you have to boot from something else; a UFD, CD/DVD, PXE server like RIS/WDS, or sometimes an external HDD. Telling your Windows computer what to boot from if something happens is a pain. You have to hit a certain key sequence that is often unique to each OEM. Then you often have to hit yet another key (like F12) to PXE boot. It’s a user experience only a geek could love. One of my ideas was to try and make it easier not only for Windows to update the BIOS itself, but for the user to more easily say what they wanted to boot the PC from (before they shut it down, or selecting from a pretty list of icons or a set of keys – like Macs can do). Unfortunately, this effort largely stalled out for over a decade until Microsoft delivered a better recovery, boot, and firmware experience with their Surface tablets. Time will tell whether we’re headed towards a world where this isn’t such a nuisance anymore.

It’s actually somewhat amusing how much of my work revolved around hardware even though I worked in an area of Windows which only made software. But if there was one commonly requested design change request that I wish I could have accommodated but couldn’t ever get done, it was F6 from UFD. Let me explain.

When you install Windows, it attempts to use the drivers it ships with on the CD to begin copying Windows down onto the HDD, or to connect over the network to start setup through RIS.

This approach worked alright, but it had one little problem which became significant. Not long after Windows XP shipped, new categories of networking and storage devices began arriving on high-end computers and rapidly making their way downmarket; these all required new drivers in order for Windows to work. Unfortunately, none of these drivers were “in the box” (on the Windows CD) as we liked to say. While Windows Server often needed special drivers to install on some high-end storage controllers before, this was really a new problem for the Windows consumer client. All of a sudden we didn’t have drivers on the CD for the devices that were shipping on a rapidly increasing number of new PCs.

In other words, even with a new computer and a stock Windows XP CD in your hand, you might never get it working. You needed another computer and a floppy diskette to get the ball rolling.

Early on during Windows XP’s setup, it asks you to press the keyboard’s F6 function key if you have special drivers to install. If it can’t find the network and you’re installing from CD, you’ll be okay through setup – but then you have no way to add new drivers or connect to Windows Update. If you were installing through RIS and you had no appropriate network driver, setup would fail. Similarly, if you had no driver for the storage controller on your PC, it wouldn’t ever find find a HDD where it could install Windows – so it would terminally fail too. It wasn’t pretty.

Here’s where it gets ugly. As I mentioned, we were entering an era where OEMs wanted to ship, and often were shipping, those legacy-free PCs. These computers often had no built-in floppy diskette – which was the only place we could look for F6 drivers at the time. As a result, not long after we shipped Windows XP, we got a series of design change requests (DCRs) from OEMs and large customers to make it so Windows setup could search any attached UFD for drivers as well. While this idea sounds easy, it isn’t. This meant having to add Windows USB code into the Windows kernel so it could search for the drives very early on, before Windows itself has actually loaded and started the normal USB stack. While we could consider doing this for a full release of Windows, it wasn’t something that we could easily do in a service pack – and all of this came to a head in 2002.

Dell was the first company to ever request that we add UFD F6 support. I worked with the kernel team, and we had to say no – the risk of breaking a key part of Windows setup was too great for a service pack or a hotfix, because of the complexity of the change, as I mentioned. Later, a very large bank requested it as well. We had to say no then as well. In a twist of fate, at Winternals I would later become friends with one of the people who had triggered that request, back when he was working on a project onsite at that bank.

Not adding UFD F6 support was, I believe, a mistake. I should have pushed harder, and we should have bitten the bullet in testing it. As a result of us not doing it, a weird little cottage industry of USB floppy diskette drives continued for probably a decade longer than it should have.

So it was, several years after I left, that the much maligned Windows Vista brought both USB boot of WinPE and also brought USB F6 support so you could install the operating system on hardware with drivers newer than Windows XP, and not need a floppy diskette drive to get through setup.

As I sit here writing this, it’s interesting to consider the death of CD/DVD media (“shiny media”, as I often call it) on mainstream computers today. When Apple dropped shiny media on the MacBook Air, people called them nuts – much as they did when Apple dropped the floppy diskette on the original iMac years before. As tablets and Ultrabooks have finally dropped shiny media drives, there’s an odd echo of the floppy drive from years ago. Where external floppy drives were needed for specific scenarios (recovery and deployment), external shiny media drives are still used today for movies, some storage and installation of legacy software. But in a few years, shiny media will be all but dead – replaced by ubiquitous high-speed wired and wireless networking and pervasive USB storage. Funny to see the circle completed.


07
Apr 14

The end is near here!

Imagine I handed you a Twinkie (or your favorite shelf-stable food item), and asked you to hold on to it for almost 13 years, and then eat it.

Aw, c’mon. Why the revulsion?

It’s been hard for me to watch the excited countdown to the demise of Windows XP. Though I did help ship Windows Server 2003 as well, no one product (or service) that I’ve ever worked on became so popular, for so long – by any stretch of the imagination – as Windows XP did.

Yet, here we are, reading articles discussing the topic of what country or what company is now shelling out $M to get support coverage for Windows XP for the next 1, 2, or 3 years (getting financially more painful as the year count goes up). It’s important to note that this is no “get out of jail free” card. Nope. This is just life support for an OS that has terminal zero-day. These organizations still have to plan and execute a migration to a newer version of Windows that isn’t on borrowed time.

Why didn’t these governments and companies execute an XP evacuation plan? That’s a very good question. Putting aside the full blame for a second, there’s a bigger issue to consider.

Go back and think of that Twinkie. Contrary to popular opinion, Twinkies don’t last forever (most sources say it’s about 25 days). Regardless, you get the idea that for most normal things, even shelf-stable isn’t shelf-stable forever. Heck, even most MRE‘s need to be stored at a reasonable temperature and will taste suboptimal after 5 or more years.

While I can perhaps excuse consumers who decide to hang on to an operating system past it’s expiration date, I have a harder time understanding how organizations and governments with any long-term focus sat by and let XP sour on them. It would be one thing if XP systems were all standalone and not connected to the Internet. Perhaps then we could turn a blind eye to it. But that’s not usually the case; XP systems in business environments, which lack most of the security protections delivered later for Windows Vista, 7, and 8.x, are largely defenseless, and will be standing there waiting to get pwned as the vulnerabilities stack up after tomorrow. In my mind, the most dangerous thing is security vendors claiming to be able to protect the OS after April 8. In most cases, that’s an all but impossible feat, and instills a false sense of confidence in XP users and administrators.

The key concern I have is that people are looking at Windows XP as if software dying is a new thing, or something unusual. It isn’t. In fact, tomorrow, the entire spectrum of Office 2003 software (the Office productivity suite, SharePoint, Exchange, and more) also leave support and could have their own set of security compromises down the road. But as I said, this isn’t the first time software has entered an unsupportable realm, and it won’t be the last. It’s just a unique combination as we get the perfect storm of XP’s pervasiveness, the ubiquity of the Internet, and the increasing willingness of bad people to do bad things to computers for money. Windows Server 2003 (and 2003 R2) are next, coming up in July of 2015.

People across the board seem to have this odd belief that when they buy a perpetual license to software, it can be used forever (versus Office 365, which people more clearly understand as a subscription that expires if not paid in an ongoing manner). But no software, even if “perpetually licensed”, is actually perpetual. Like that Twinkie I’ve mentioned a few times, even good software goes bad. As an industry, we need to start getting customers throughout the world to understand that, and get more organizations to begin planning software deployments as an ongoing lifecycle, rather than a one-time expense that is ignored until it goes terminal.


17
Jan 14

Running Windows XP after April? A couple of suggestions for you

Yesterday on Twitter, I said the following:

Suggestion… If you have an XP system that you ABSOLUTELY must run after April, I’d remove all JREs, as well as Acrobat Reader and Flash.

This was inspired by an inquiry from a customer about Windows XP support that arrived earlier in the day.

As a result of that tweet, three things have happened.

  1. Many people replied “unplug it from the network!” 1
  2. Several people asked me why I suggested doing these steps.
  3. I’ve begun working on a more comprehensive set of recommendations, to be available shortly. 2

First off… Yes, it’d be ideal if we could just retire all of these XP systems on a dime. But that’s not going to happen. If it was easy (or free), businesses and consumers wouldn’t have waited until the last second to retire these systems. But there’s a reason why they haven’t. Medical/dental practices have practice management or other proprietary software that isn’t tested/supported on anything newer, custom point of sale software from vendors that disappeared, were acquired, or simply never brought that version of their software… There’s a multitude of reasons, and these systems aren’t all going to disappear or be shut off by April. It’s not going to happen. It’s unfortunate, but there are a lot of Windows XP systems that will be used for many years still in many places that we’d all rather not see happen. There’s no silver bullet for that. Hence, my off the cuff recommendations over Twitter.

Second, there’s a reason why I called out these three pieces of software. If you aren’t familiar with the history, I’d encourage you to go Bing (or Google, or…) the three following searches:

  1. zero day java vulnerability
  2. zero day Flash vulnerability
  3. zero day Acrobat vulnerability

Now if you looked carefully, each one of those, at least on Bing, returned well over 1M results, many (most?) of them from the last three years. In telling me that these XP systems should be disconnected from the Web, many people missed the point I was making.

PCs don’t get infected from the inside out. They get infected from the outside in. When Microsoft had the “Security Push” over ten years ago that forced us to reconsider how we designed, built and tested software, it involved stopping where we were, and completely thinking about how Windows was built. Threat models replaced ridiculous statements like, “We have the very best xx encryption, so we’re ‘secure'”. While Windows XP may be more porous than Vista and later are (because the company was able to implement foundational security even more deeply, and engineer protections deeply into IE, for example, as well as implement primordial UAC), Windows XPSP2 and later are far less of a threat vector than XPSP1 and earlier were. So if you’re a bad guy and you want to get bad things to happen on a PC today, who do you go after? It isn’t Windows binaries themselves, or even IE. You go next for the application runtimes that are nearly as pervasive. Java, Flash, and Acrobat. Arguably, Acrobat may or may not be a runtime, depending on your POV. But the threat is still there, especially if you haven’t been maintaining these as they’ve been updated over the last few years.

As hard as Adobe and Oracle may try to keep these three patched, these three codebases have significant vulnerabilities that are found far too often. Those vulnerabilities, if not patched by vendors and updated by system owners incredibly quickly, become the primary vector of infecting both Windows and OS X systems by executing shellcode.

After April, Windows XP is expected to get no updates. Got that? NO UPDATES. NONE. Nada. Zippo. Zilch. So while you may get antivirus updates from Microsoft and third parties, but at that point you honestly have a rotting wooden boat. I say this in the nicest way possible. I was on the team shipping Windows XP, and it saddens me to throw it under the bus, but I don’t think people get the threat here. Antivirus simply cannot protect you from every kind of attack. Windows XP and the versions of IE (6-8) have still regularly received patches almost every month for the past several years. So Windows XP isn’t “war hardened”, it is brittle. So after April, you won’t even get those patches trying to spackle over newly found vulnerabilities in the OS and IE. Instead, these will become exploit vectors ready to be hit by shellcode coming in off of the Internet (or even the local network) and turned into opportunistic infections.

Disclaimer: This is absolutely NOT a guarantee that systems won’t get infected, and you should NOT remove these or any piece of Microsoft or third-party software if a business-critical application actually depends on them or if you do not understand the dependencies of the applications in use on a particular PC or set of PCs! 

So what is a business or consumer to do? Jettison, baby. Jettison. If you can’t retire the entire Windows XP system, retire every single piece of software on that system that you can, beginning with the three I mentioned above. Those are key connection points of any system to the Web/Internet. Remove them and there is a good likelihood of lessening the infection vector.   But it is a recommendation to make jetsam of any software on those XP systems that you really don’t need. Think of this as not traveling to a country where a specific disease is breaking out until the threat has passed. In the same vein, I’d say blocking Web browsers and removing email clients coming in a close second, since they’re such a great vector for social engineering-based infections today.

Finally, as I mentioned earlier, I am working on an even more comprehensive set of recommendations to come in a more comprehensive report to be published for work, in our next issue, which should be live on the Web during the last week of January. My first recommendation would of course be to, if at all possible, retire your Windows XP systems as soon as possible. But I hope that this set of recommendations, while absolutely not a guarantee, can help some people as they move away, or finally consider how to move away, from Windows XP.

Footnotes

  1. Or unplug the power, or blow it up with explosives, or…
  2. These recommendations will be included in the next issue of Update.

20
Dec 13

Security and Usability – Yes, you read that right.

I want you to think for a second about the key you use most. Whether it’s for your house, your apartment, your car, or your office, just think about it for a moment.

Now, this key you’re thinking of is going to have a few basic properties. It consists of metal, has a blade extending out of it that has grooves along one or both sides, and either a single set of teeth cut into the bottom, or two sets of identical teeth cut into both the top and bottom.

If it is a car key, it might be slightly different; as car theft has increased, car keys have gotten more complex, so you might be thinking about a car key that is just a wireless fob that unlocks and or starts the car based on proximity, or it might be an inner-cut key as is common with many Asian and European cars today.

Aside from the description I just gave you, when was the last time you thought about that key? When did you actually last look at the ridges on it?

It’s been a while, hasn’t it? That’s because that key and the lock it works with provide the level of security you feel that you need to protect that place or car, yet it doesn’t get in your way, as long as the key and the lock are behaving properly.

Earlier this week, I was on a chat on Twitter, and we were discussing aspects of security as they relate to mobile devices. In particular, the question was asked, “Why do users elect to not put a pin/passcode/password on their mobile devices?” While I’ve mocked the idea of considering security and usability in the same sentence, let alone the same train of thought while developing technology, I was wrong. Yes, I said it. I was wrong. Truth be told, Apple’s Touch ID is what finally schooled me on it. Security and usability should be peers today.

When Apple shipped the iPhone 5s and added the Touch ID fingerprint sensor, it was derided by some as not secure enough, not well designed, not a 100% replacement for the passcode, or simply too easy to defeat. But Touch ID does what it needs to do. It works with the user’s existing passcode – which Apple wisely tries to coax users into setting up on iOS 7, regardless of whether they have a 5s or not – to make day to day use of the device easier while living with a modicum of security, and a better approach to securing the data, the device, and the credentials stored in it and iCloud in a better way than most users had prior to their 5s.

That last part is important. When we shipped Windows XP, I like to think we tried to build security into it to begin with. But the reality is, security wasn’t pervasive. It took setting aside a lot of dedicated time (two solid months of security training, threat modeling, and standing down on new feature work) for the Windows Security Push. We had to completely shift our internal mindset to think about security from end to end. Unlike the way we had lived before, security wasn’t to be a checkbox, it wasn’t a developer saying, “I used the latest cryptographic APIs”, and it wasn’t something added on at the last minute.

Security is like yeast in bread. If you add it when you’re done, you simply don’t have bread – well, at least you don’t have leavened bread. So it took us shipping Windows XP SP2 – an OS update so big and so significant many people said it should have been called a new OS release – before we ever shipped a Windows release where security was baked in from the beginning of the project, across the entirety of the project.

When it comes to design, I’ve mentioned this video before, but I think two of Jonathan Ives’ quotes in it are really important to have in your mind here. Firstly:

“A lot of what we seem to be doing in a product like that (the iPhone) is getting design out of the way.”

and secondarily:

“It’s really important in a product to have a sense of the hierarchy of what’s important and what’s not important by removing those things that are all vying for your attention.”

I believe that this model of thought is critical to have in mind when considering usability, and in particular where security runs smack dab into usability (or more often, un-usability). I’ve said for a long time that solutions like two-factor security won’t take off until they’re approachable by, and effectively invisible to, normal people. Heck, too much of the world didn’t set ever set their VCR clocks for the better part of a decade because it was too hard, and it was a pain in the ass to do it again every time the power went out. You really don’t understand why they don’t set a good pin, let alone a good passcode, on their phone?

What I’m about to say isn’t meant to infer that usability isn’t important to many companies, including Microsoft, but I believe many companies run, and many software, hardware or technology projects are started, run, and finished, where usability is still just a checkbox. As security is today at Microsoft, usability should be embraced, taught, and rewarded across the organization.

One can imagine an alternate universe where a software project the world uses was stopped in it’s tracks for months, redesigned, and updated around the world because a user interface element was so poorly designed for mortals that they made a bad security decision. But this alternate universe is just that, an alternate universe. As you’re reading the above, it sounds wacky to you – but it shouldn’t! As technologists, it is our duty to build hardware, software, and devices where the experience, including the approach to security, works with the user, not against them. Any move that takes the status quo of “security that users self-select to opt into” and moves it forward a notch is a positive move. But any move here also has to just work. You can’t implement nerd porn like facial recognition if it doesn’t work all of the time or provide an alternative for when it fails.

Projects that build innovative solutions where usability and security intersect should be rewarded by technologists. Sure, they should be critiqued and criticized, especially if designing in a usable approach really compromises the security fundamentals of the – ideally threat-modeled – implementation. But critics should also understand where their criticism falls down in light of the practical security choices most end users make in daily life.

Touch ID,  with as much poking, prodding, questioning, and hacking as it received when it was announced, is a very good thing. It’s not perfect, and I’m sure it’ll get better in future iterations of the software and hardware, and perhaps as competitors come up with alternatives or better implementations, Apple will have to make it ever more reliable. But a solution that allows that bar to be moved forward, from a place where most users don’t elect to set a pin or passcode to a place where they do? That’s a net positive, in my book.

As Internet-borne exploits continue to grow in both intensity and severity, it is so critical that we all start taking the usability of security implementations by normal people seriously. If you make bad design decisions about the intersection where security and usability collide, your end users will find their own desire path through the mayhem, likely making the easiest, and not usually the best, security decisions.

 


04
Nov 13

Plan on profiting off of Windows XP holdouts? There’s no gold left in them thar hills.

A few times over the last year, I’ve had conversations with people about Windows XP holdouts. That is, that as Windows XP’s impending doom rapidly approaches next April, businesses and consumers holding out on Windows XP will readily flock to something new, such as – ideally for Microsoft, Windows 8.1 – or Windows 7.

I’m not so sure.

To start, let’s consider why a business or consumer would still be running Windows XP today. Most likely, it’s a combination of all of the following:

  1. It’s paid for (the OS and hardware)
  2. It runs on the hardware they have
  3. Applications they have won’t run, or aren’t supported, on anything newer
  4. It requires no user retraining
  5. They don’t see  a compelling reason to move beyond XP
  6. They don’t realize the risks of sticking with XP after next April.

You can split those reasons into two categories. Items 1-3 are largely due to financial impediment, while 4-6 are generally due to “static quo” – XP meets their business needs and 8.1 or 7 doesn’t provide the necessary pull to motivate them to move off of XP.

It’s not that 7, 8, or 8.1 did anything wrong, necessarily. I was there when XP shipped, and I’ll tell you I heard many business customers complain about numerous things. They hated the theme and felt it was toy-like. They wanted to be able to take off Movie Maker, Internet Mail & News, or other consumer niblets of the OS, but couldn’t. Frankly, some of them just felt it was a warmed over Windows 2000 (obviously none of those customers had ever tried to undock a hibernated Windows 2000 laptop). For many customers, it took until XP had been effectively re-released as XPSP2 for them to really fall for it. When Vista shipped, reasons 4 & 5 above largely doomed it. Vista had a completely nebulous value proposition for most consumers and almost every business, leading to Windows XP becoming even more deeply engrained into many businesses.

Many people describe Windows 7 as “a better Windows XP”, which I think is actually an insult to both operating systems. But frankly, unless a business understands item number 6 (which Microsoft just grabbed a drum and started beating really hard – albeit very, very late), the rest don’t matter.

I’ve talked to several businesses about Windows XP over the past several years. For better or worse, most of them are happy with the hardware and software investments they made over the last 12 years, and many don’t feel like spending money for new hardware (especially new touch-centric form factors with value that they don’t see clearly yet).

Even more important though, is the number of times we have run into businesses – especially small businesses like dental, medical, or other independent practices – which during the past decade either bought commercial software packages or hired consultants to build them custom software. As a result, many of them hit item number 3 – “Applications they have won’t run, or aren’t supported, on anything newer”. I kid you not, there are a lot of small businesses with a lot of applications that honestly have no path forward. They cannot stay on XP – they cannot be secure. They cannot move off, as they either cannot find a replacement of one or more of their key applications, cannot move that key existing application, or in some cases, simply cannot afford to move to a replacement (in case you haven’t noticed, we’re still not in a great economic climate). They are stuck between a rock and a hard place. Move off of XP and throw away working systems your employees already know for new systems with unknown features or functionality. To boot, any of these new solutions are primarily still targeting Windows 7 (the desktop), not the Windows 8+ “modern UI” – diminishing some of the key value in acquiring tablet or touch-centric devices running 8, if the system is, for the time being (and likely for the foreseeable future) stuck on the desktop. Since Windows 8+ doesn’t include Windows XP Mode, unless a customer has appropriate enterprise licensing with Microsoft, they can’t even run Windows XP in a VM on Windows 8+ (and I have a hard time believing that customers who spend that kind of money are the kind of customers who are holding out).

There are still many organizations that are using XP (and likely Office 2003) and appear to have no exit strategy or plan to leave XP behind. It appears a lot of organizations don’t realize (or don’t care) now porous Windows XP will become after it ceases being patched in April. It isn’t a war-hardened OS, as some customers believe. It’s a U.S.S. Constitution in an era of metal battleships. I hate to sound like a shill, but XP systems will be ripe for an ass-kicking beginning next spring, and they can, and will, be taken advantage of. I also don’t believe Microsoft will do any favors for businesses that stay on XP (and don’t pay the hefty costs for custom support agreements with a locked and loaded exit plan in place).

XP is dying. But I believe lots of organizations are simply unclear about what kind of threat it poses to them. As a result, they’re sitting on the investment they’ve already made.

I also think that a lot of organizations that are still sitting on XP today may even be aware of (some of the) potential risks that XP poses to their organization after April, but simply don’t have the budget to escape in time, even if they had the motivation (which lots of them don’t appear to have). Even if a company pulls the trigger today, if they have any significant number of XP systems and XP dependent applications, they’ll be lucky to be off of XP by April.

There’s a belief that a lot of these customers had budget sitting there, had no app blockers, and might have even wanted to go to 7, 8 or just “something new”, but were just lackadaisical and for some reason will now get a fire lit under them, generating a windfall of sorts for Microsoft, PC OEMs, and partners over the next 5 months. Instead of that easy opportunity, I believe where you run across XP in the majority of organizations at this point, a better analogy is a set of four fully impacted wisdom teeth in a patient with no dental coverage.


10
Apr 13

Windows XP – Hitting the Wall

Just under one year from now, on April 8, 2014, Windows XP leaves Extended Support.

There are three key questions I’ve been asked a lot during the past week, related to this milestone:

  1. What even happens when Windows XP leaves Extended Support?
  2. Will Microsoft balk, and continue to support Windows XP after that date?
  3. What will happen to systems running Windows XP after that date?

All important questions.

The first question can be exceedingly complex to answer. But for all intents and purposes, the end of Extended Support means that you will receive absolutely no updates – including security updates – after that date. While there are some paid support options for Windows XP after 4/8/2014, however as we understand it they will be very tightly time limited, very expensive, and implemented with a contractual, date-driven expectation for a retirement of the organization’s remaining Windows XP desktops. There’s no “get out of jail” card, let alone a “get out of jail free” card. If you have Windows XP desktops today, you have work to do, and it will cost you money to migrate away.

If you want to look for yourself, you can go to Microsoft’s downloads site and look – but Windows XP still receives patches for both Windows itself or Internet Explorer (generally 6,7, and 8 all get patched) every month. From April 2012 to April 2013, every month saw security updates to either Windows XP or IE on it – and 8 of the 13 months saw both. Many of these are not pretty vulnerabilities, and if left unpatched, could leave targeted organizations exceedingly vulnerable after that date.

This leads us to the second question. In a game of chicken, will Microsoft turn and offer support after 4/8/2013?

Why are you asking? Seriously. Why? I was on the team that shipped Windows XP. I wish that, like a work of art, Windows XP could be timeless and run forever. But it can’t (honestly, that theme is starting to get rather long in the tooth too). It’s a piece of machinery – and machinery needs maintenance (and after a time, it usually needs replacement). Windows 2000 received it’s last patch the month before it left Extended Support. So, while 4/8/2014 is technically a Patch Tuesday, and Microsoft might give you one last free cup of joe, I’d put a good wager down that if you want patches after that day, you’d better plan your migration, get on the phone to Microsoft relatively soon, get a paid support contract in place, and be prepared to pay for the privilege of support while you migrate away.

Companies that are running Windows XP today – especially in any sort of mission critical or infrastructure scenario – especially if connected to the Internet, need to have a migration plan away to a supported operating system.

At a security startup I used to work at (not that long ago), it shocked me how many of our prospects had Windows 2000, Windows NT, or even older versions of NT or 9x, in production (and often connected to networks or the Internet. Even more terrifying, many of these were mission critical systems.

And this segues us to the third question. What happens to systems running after 4/8/2014? You can quote Clint Eastwood’s “Dirty Harry” character, “Do I feel lucky? Well do you?” It’s not a good bet to make. Again, we’ve seen some nasty bugs patched in IE 6,7, and 8, and Windows XP itself over the last year. While one would hope an OS 12 years out would be battle-hardened to the point of being bulletproof, that is not the case. Windows XP isn’t bulletproof. It’s weary. It’s ready to be retired. Organizations with critical infrastructure roles still running Windows XP will have giant targets on them after next April, and no way to defend those systems.

A common thread I’ve also seen is a belief that a wave of Windows XP migrations over the next 12 months will mean anything, economically. It really isn’t likely to. While we will likely see a good chunk of organizations move away from Windows XP over the next year, doing so may mean finding budget to replace 5+ year old PCs, and patch, update, or purchase replacement Windows, Java, and Web applications that can run on newer operating systems. Most of the easy lifting has already been done. The last customers remaining are likely extremely hard, extremely “financially challenged”, or both. It may be unfortunate, but this time next year (and likely the year after that, and years after that), there will still be Windows XP systems out there, some of them running in highly critical infrastructure. Dangerous, but unfortunately, likely to be the case.


30
Mar 13

The mythology of MinWin, MinKernel, and BaseFS

Beginning with Windows NT Embedded, an effort was started to refactor the Windows operating system in such a way that embedded device manufacturers could tweak the operating system to only include the parts that they needed to run their application or workload. Though it wasn’t completely decomposed/recomposable, it was a huge step in the right direction. By removing components not necessary to run a certain workload (say a point of sale terminal or a control system for manufacturing), the security footprint, as well as the overall footprint of the OS (sometimes CPU, but more often HDD/RAM and video) could be constrained, resulting in a device that was better at what it was supposed to do, while minimizing security and cost risks. Take a look at the claims on that Windows NT Embedded page – 9MB for a standalone OS runtime that didn’t include the Windows shell or networking.

This effort kept progressing throughout Windows, and the next major release, Windows XP Embedded, took the effort, referred to as componentization, even further. This effort required the XP Embedded (XPE) team to go out across Windows and evangelize componentization, and convince teams to help them break down their dependencies up and down the stack. You would be shocked to learn about how intertwined pieces of Windows were with each other. Nobody had, for the life of NT, ensured that components at the top levels of Windows had logically defined or clarified their dependencies down the stack. As a result, if you added one component of the Windows shell, you could wind up bringing in a gigantic stack of components. Add a certain runtime, and you needed to add a control panel (.cpl) file, because… well, it made sense at the time to put that library inside the cpl file instead of another DLL.

At that same time during Windows XP (Windows Whistler) development, I was on the setup and deployment team, which was separate from Windows Embedded (but would later merge). I was the second program manager (I  took the specs from the customers to the developers) for WinPE, also known as Windows Preinstallation Environment. If you’ve never heard of Windows PE, imagine a tiny OS runtime, about 120MB, that could run in 96MB or less of RAM. It featured TCP/IP networking, NTFS and FAT disk access, and cmd.exe as it’s shell. Most importantly, WinPE booted from CD, and was intended to replace MS-DOS for the pre-installation needs of Windows, whether you needed to kick off a scripted install of setup, or throw down an image to disk. WinPE was created primarily because the new Itanium platform had no deployment OS to help OEMs deploy Windows to their hardware. While Windows on Itanium died, WinPE became the underpinning of effectively every deployment and recovery tool in Windows.

I mention WinPE for one reason – I explicitly lived the XPE team’s pain trying to understand dependencies. Not long after we revealed WinPE to OEMs, they came back with wish lists of things they wanted added – including a Web browser. Eventually, I was able to talk them down from a Web browser, because they really wanted HTML Applications (HTA). I also hacked together Windows Script Host support and ADO connectivity to SQL Server. You cannot imagine the work I did to find the dependencies for those three components – which had not been defined individually in a way I could just grab and reuse. Finding someone who knew IE internals to the level I needed was almost impossible.

So what does this have to do with MinWin? Not long after Windows XP wrapped, there was a goal to come up with a minimal version of Windows (MinWin) that could serve as the centerpiece of every version of Windows. Whether you were building Windows Server or a Windows client release, or even WinPE, MinWin was the idea of a bootable version of Windows that featured the core components of the OS. I guess you could perhaps think of it as the quark of Windows. A common misstatement I’ve heard is that “MinWin is the minimal Windows kernel” or anything relating just to the kernel. It is more than just the kernel. MinWin, like the development of Windows on ARM that also occurred while I was at Microsoft, was never officially discussed outside, other than a few leaks. As a result, there was always much confusion about it. Most importantly, MinWin was never “a thing”. You never would be able to buy MinWin by itself, it was always intended to be a core of every version of Windows. In essence, it wasn’t about making a thing, it was about refactoring what we already had, to make it more nimble, both for continuing and growing embedded scenarios for Windows, but also to create products like Windows Server Core is today, as well as the current version of WinPE.

I encourage you to keep that in mind as you read this piece on MinKernel and BaseFS. While I have not talked with anyone inside Microsoft about either of these projects, I can only assume that they are the continuation of the refactoring of Windows. The Windows kernel was, for a long time, much more regulated than most of Windows. However, it still grew “big-boned” over time as things like Win32, networking, video,  .NET, WinSxS, and now WinRT required the kernel to be constantly updated – in a completely monolithic form. That would be fine, if every device required everything that was in the kernel. But it doesn’t. Windows Phone 8 likely includes at least some aspects of Win32, even though it will never run a full legacy Win32 application. So here again, my assumption is that the MinKernel project isn’t anything new per se, rather it is a modular, rather than monolithic, approach to building the Windows kernel. The result would be a layer of native-mode pieces that can be combined as needed for numerous platforms (Windows Phone 8+, Windows 8+, Windows RT+, Windows Azure, and even potentially the next generation Xbox and other hardware we aren’t aware of yet). The same could exist in the BaseFS realm. NTFS includes a ton of baggage, much of which is used by a small number of customers. Many of the NTFS features I’m alluding to as baggage didn’t even make it into the first public iteration of ReFS, which attempts to replicate much – but not all – of the functionality of NTFS. So what could BaseFS be? Likely the minimal NTFS, ReFS, or perhaps a shared driver for both, that implements a minimal set of filesystem functionality.

One can imagine this kind of refactoring of both to result in not only a well-factored and malleable kernel and filesystem driver, but also the kind of thin working set that might be ideal for, perhaps, running Windows Azure or Windows Hyper-V Server on, where the role of the server is solely as a host. I’m guessing here, and time will tell if I’m right or if I’ve missed some greater scenario. But by and large, much as MinWin was never a tangible thing, I’m not clear at this point that BaseFS or MinKernel will be much that even power users will ever get to see, directly.


27
Mar 13

The Stigma of Mac Shaming

I recall hearing a story of a co-worker at Microsoft, who was a technical assistant to an executive, who had a Mac. It wouldn’t normally be a big deal, except he worked directly for an executive. As a result, this Mac was seen in many meetings across campus – it’s distinct aluminum body and fruity ghost shining through the lid a constant reminder that this was one less PC sold (even if it ran Windows through Boot Camp or virtualization software. Throughout most of Microsoft, there was a strange culture of “eww, a Mac”. Bring a Mac or an iPod to work, feel like an outcast. This was my first exposure to Mac Shaming.

I left Microsoft in 2004, to work at Winternals in Austin (where I had the last PC I ever really loved – a Toshiba Tecra A6). In 2006, on the day Apple announced Boot Camp, I placed an order for a white Intel iMac. This was just over three months before Winternals was acquired by Microsoft (but SHH… I wasn’t supposed to know that yet). This was my first Mac. Ever.

Even though I had worked at Microsoft for over 7 years, and was still writing for Microsoft’s TechNet Magazine as a monthly Contributing Editor, I was frustrated. My main Windows PC at home was an HP Windows XP Media Center PC. Words cannot express my frustration at this PC. It “worked” as I originally received it – but almost every time it was updated, something broke. All I wanted was a computer that worked like an appliance. I was tired of pulling and pushing software and hardware to try and get it to work reliably. I saw Windows Vista on the horizon and… I saw little hope for me coming to terms with using Windows much at home. It was a perfect storm – me being extreme underwhelmed with Windows Vista, and the Mac supporting Windows so I could dual-boot Windows as I needed to in order to write. And so it began.

Writing on the Mac was fine – I used Word, and it worked well enough. Running Windows was fine (I always used VMware Fusion), and eventually I came to terms with most of the quirks of the Mac. I still try to cut and paste with the Ctrl key sometimes, but I’m getting better.

I year later, I flipped from a horrible Windows CE “smartish” phone from HTC on the day that Apple dropped the price of the original iPhone to $399. Through two startups – one a Windows security startup, the other a Web startup, I used two 15″ MacBook Pros as my primary work computer – first the old stamped MBP, then the early unibody.

For the last two years, I’ve brought an iPad with me to most of the conferences I’ve gone to – even Build 2011, Build 2012, and the SharePoint Conference in 2012. There’s a reason for that. Most PCs can’t get you on a wireless network and keep you connected all day, writing, without needing to plug in (time to plug in, or plugs to use, being a rarity at conferences). Every time I whipped out my iPad and it’s keyboard stand with the Apple Bluetooth keyboard, people would look at me curiously. But quite often, as I’d look around, I’d see many journalists or analysts in the crowd also using Macs or iPads. The truth is, tons of journalists use Macs. Tons of analysts and journalists that cover Microsoft even use Macs – many as their primary device. But there still seems to be this weird ethos that you should use Windows as your primary device if you’re going to talk about Windows. If you are a journalist and you come to a Microsoft meeting or conference with a Mac, there’s all but guaranteed to be a bit of an awkward conversation if you bring it out.

I’m intimately familiar with Windows. I know it quite well. Perhaps a little too well. Windows 8 and I? We’re kind of going in different directions right now. I’m not a big fan of touch. I’m a big fan of a kick-ass desktop experience that works with me.

Last week, my ThinkPad died. This was a week after my iMac had suffered the same fate, and I had recovered it through Time Machine. Both died of a dead Seagate HDD. I believe that there is something deeper going on with the ThinkPad, as it was crashing regularly. While it was running Windows 8, I believe it was the hardware failing, not the operating system, that led to this pain. In general, I had come to terms with Windows 8. Because my ThinkPad was touch, it didn’t work great for me, but worked alright – though I really wasn’t using the “WinRT side” of Windows 8 at all, I had every app I used daily pinned to the Taskbar instead. Even with the Logitech t650, I struggled with the WinRT side of Windows 8.

So here, let me break this awkward silence. I bought another Mac, to use as my primary writing machine. A 13″ Retina MacBook Pro. Shun me. Look down upon me. Shake your head in disbelief. Welcome to Mac shaming. The machine is beautiful, and has a build quality that is really unmatched by any other OEM. A colleague has a new Lenovo Yoga, and I have to admit, it is a very interesting machine – likely one of the few that’s out there that I’d really consider – but it’s just not for me. I also need a great keyboard. The selection of Windows 8 slates with compromised keyboards in order to be tablets is long. I had contemplated getting a Mac for myself for some time. I still have a Windows 8 slate (the Samsung), and will likely end up virtualizing workloads I really need in order to evaluate things.

My first impression is that, as an iPad power user (I use iOS gestures a lot) it’s frighteningly eerie how powerful that makes one on a MBP with Mountain Lion and fullscreen apps. But I’ll talk about that later.

I went through a bit of a dilemma about whether to even post this or not, due to the backlash I expect. Post your thoughts below All I request? I invoke Wheaton’s Law at this point.


25
Mar 13

The care and feeding of software

App hoarding. The dark, unspoken secret. We’ve all done it. I logged on to a Windows 8 tablet I hadn’t used for quite some time, and I was so ashamed of myself. So much junk, so many free apps I downloaded, tried, and abandoned. Only recently have I begun steadfastly maintaining a “two screen” limit on iOS to try and keep the applications on my devices solely to those that I use regularly.

This isn’t new, mind you. Enterprises have been doing this for years. Sometimes the “application” is an Excel spreadsheet. Sometimes it’s an old database application, or some other piece of old code, owned by a developer who long since ran from the organization.

For a long time, like Microsoft and comprehensive security ahead of the Windows security push, customers could turn a blind eye to application proliferation. Like feral rabbits, one will lead to many, and if you don’t manage or cut them back, they get out of control. Unfortunately, many enterprise applications are borne out of short-term necessity, without a great period of design forethought. Just as unfortunately, nobody goes around every year and does an “application census” in most organizations to figure out which applications are dead, abandoned, unused, or worst of all – insecure or unsecurable.

A colleague today was telling me about an antivirus application from a major vendor that relies on Java. Terrifying. But that’s nothing. Java is still supported (how well supported is arguable). If your organization is of any significant size, you’ve got applications based around ancient versions of Microsoft Office, SQL Server, or other products and platforms that are likely long past their expiration date. No updates, no patches, nothing. Yet your organization depends on them, and likely has no security mitigation or migration story in play. With the current crop of vulnerabilities we’ve seen recently in Java, Flash, and Acrobat Reader, I’ve been growing increasingly concerned with how dependent so many organizations are upon all three, yet how laissez-faire they seem to be about eliminating or at least reducing their dependency upon all three.

On a similar note, as someone who helped ship Windows XP, I love how well it has stood the test of time. But it was not engineered for today’s world – from an always-on connection to the Internet to the threat vectors being thrown at it and the software running on it.

It concerns me that so many organizations aren’t cognizant of what software (operating systems, platforms, and applications) are running in their organizations. They talk big of cloud, and how it’s better that they run the software on their premises. Yet they’re running old, unpatched software, often with known, never-to-be-patched vulnerabilities, and no plan to consolidate applications and remove dead, unsupported operating systems, platforms, and applications. It’s the equivalent of every enterprise having a bunch of storage units full of random crap you keep around because “someone might need it someday”.

Microsoft has been beating a drum about Windows XP – if you look at it closely, it sounds more like a marketing message. But whether you view it as that or not, and whether Windows 7, Windows 8, or something else entirely is in the cards for you, your business has barely one year to get off of Windows XP (April 8, 2014). We’ve heard from some customers that they have heard of custom support options after that time, but they are on a per-desktop basis, and the adage, “if you have to ask, you probably can’t afford it” appears to apply quite well. Windows XP (officially at death’s door) and Office 2003, also very widely used still, both pass into the great beyond on that same day.

Whether it is Windows XP, Office 2003, more porous (hard or impossible to patch) platform components, or custom applications on top of them, it’s imperative that organizations start managing and monitoring – and deprecating/discontinuing – applications that rely on dead software to exist. They’re putting your organization at risk. For me, there are two takes to this – cut back the applications you already have, and more importantly, carefully regulate how you build and deploy new ones, with a keen eye on the support lifecycle – and the patchability/supportability – of the OS, runtimes, and applications that you build upon. Applications can seem quick and easy to build on a whim. But like a puppy, or perhaps even more like a parrot, applications aren’t free to build or maintain. They are a long-term commitment.