069. Mega-Scale, Mega-Complexity [Ch. X]
“So now, when we face a choice between adding features and resolving security issues, we need to choose security.” —BillG in “Trustworthy Computing”
Welcome to Chapter X!
The turn of a new century and survival of “Y2K” begins a massive expansion of the Office product line, a dramatic change in the products we built, and a reinvention of how we market Office, while simultaneously tripling and quadrupling down on enterprise customers. The result of this transformation sets the stage for the most formative years culturally for the Microsoft that takes us to today’s products (365, Azure), but unknowingly constrained us going forward.
Back to 068. The XP eXPerience
One early company meeting as the CFO Mike Brown (MikeBro) was going over the numbers he had a chart of the Fortune 500 showing that Microsoft finally made it into the rankings. He showed the top 5 companies or so, which dwarfed Microsoft then just barely in the top half. Sitting next to my first Microsoft friend KirkG, I recall asking him if he thought there would ever be a software company the size of AT&T or General Motors. His answer, “hell yes.”
Though Microsoft was number 84 by sales, it had achieved the unimaginable as the largest company by market capitalization. Thinking about that even today is still difficult to fathom—bigger than the oil companies, bigger than the entire auto industry, bigger than the phone company. It was obviously more than humbling. It was terrifying.
There was, nevertheless, a swagger across the company. Sure, the trial and settlement were still going on but the worst seemed over. There was such an incredible wave of success from the new enterprise business, both the licensing of enterprise agreements and the product offerings on the Windows/Office 2000 platform, along with Exchange and several other key products. For product people these were the magical product releases. There were “investments” as SteveB called them across every imaginable product category. The combination of Microsoft pivoting to internet technologies, the dot com crash, and simply winning against competitors is what I believe really caused a change in how we thought about ourselves, by rising to the moment and vanquishing competitors: Netscape, Sun, and Novel on servers, Apple on desktops, Borland on tools, Lotus on email, and so many other smaller companies. We saw success in everything we were doing. It became difficult even to find competitive products to get the team sincerely fired up. The biggest competitor was no one company but alliances such as anyone using Java or open source, but these were diffused and fragmented battles. Microsoft’s oxygen came from competing and winning and we’d sucked all the oxygen out of the market by winning so soundly. The market capitalization was a realization of that.
It would be a mistake to think success would not change how we built products. When you chase a competitor, even if you’ve got better ideas for how to express, implement, or sell a product as we did for Excel versus 1-2-3, the idea for the product in the first place came from the competitor. Microsoft didn’t invent from whole cloth many products. Most every product has a lineage of some form, so this is never an entirely fair way to judge. Most every successful product could be traced to a direct competitor with scale. On the other hand, many of our biggest failures were products that lacked a competitor where we would often get lost on a meandering journey trying to figure out what to even build. The Blackbird internet authoring tool previously described was a good example of this.
We had competitors of course, and some were spectacularly good. Oracle would forever remain the top SQL database company. Linux and open source were clearly a huge issue, but as should have been anticipated they would not win head-to-head but would only win when a paradigm shift—cloud and mobile—enabled them to flourish in business software. SAP and the whole space of software that ran the back office of enterprise and CRM software escaped us even with their deep connections to Office and email.
Increasingly Microsoft’s strategies became so daunting and all-encompassing it was nearly impossible to imagine a single competitor that could offer anything close to our strategy slides, and equally impossible for any single person to track. We were in a constant state of tuning the message and product line. Platforms lurched from Distributed interNet Architecture (DNA), to Next Generation Windows Services (NGWS), to .NET to .NET MyServices (aka Hailstorm), with stops along the way such as putting “+” on the name of every new endeavor, for example COM+, or adding XML to expand an existing strategy. At each iteration, everything was rebranded, expanded, and re-factored. An area like accessing data stored in a database cycled through acronyms representing increased capability and complexity, and varying compatibility and tools support: ODBC, OLEDB, DAO, ADO, ADO.NET, RDO, and more. The complexity of naming was only matched by the increasing complexity of software. The original Windows 3.0 APIs numbering about 350 had expanded to a literally uncountable breadth of platform services. No single developer could comprehend this. Definitely no one in Office and we began to feel distant from the very platform strategy we depended upon. Office had historically been the source of killer apps for platform strategies, but now the platform team looked to the biggest consumer web sites and largest commercial web applications as the desktop and client were deprioritized.
The evolution of the much beloved Visual Basic provided a lesson in this divergence. Office had just managed to add VB to all the products with solid performance, a major cross-company success. VB was in the process of being updated, really replaced, by a newly rearchitected product called Visual Basic .NET or VB.NET, which gained synergy with the .NET strategy and new iteration of the language for use on servers. This product was so different from the classic and wildly popular Visual Basic that one of Microsoft’s own and much-loved MVPs (the leaders in the community carefully selected to offer the best knowledge and support for products) dubbed the new product Visual Fred in an infamous blog post that rallied the community but divided it from Microsoft. Other posts began to meticulously track the differences in the new product and the time and effort required to migrate existing projects.
Office could not possibly retrofit this incompatible tool into the product—customers would have lost their minds as we had just completed migrating from the legacy programming languages—leaving the carefully crafted cross-group collaboration in somewhat of a Cold War. The VB team’s best response to the trade press asking about VB.NET was that we had “some difficult choices to make, as it's tough to move from the PC-centric computing model to the Web-centric .Net model.”
Were we losing touch or were we trying to do bigger and better things that would naturally and unfortunately alienate our best fans? Were we growing our ability to create strategic enterprise products or were we over-reaching on what we could (or even should) deliver? Was disruptive innovation happening in real time like it should, or were we just being disruptive?
Broadly, we began the decade far more inwardly focused than ever before. An expression that was often used was “smoking our own supply” or I might have said, “Microsoft was creating our own bubble”. Everything we did was relative to everything else we did which was relative to the feedback we got from the enterprise customers we already won who were champions for anything we did. I thought often about the very first technical conference I attended which happened to be the last global DECWorld on a cruise ship in Boston in 1987. Digital Equipment Corporation, DEC, was the famed makers of the VAX and VMS operating systems I used in college. I was a new graduate student and drove to Boston for the day.
The conference felt like the most grownup event I’d ever seen—everyone was twice my age. It seemed like everyone wore a suit and all they talked about was DEC. What I recall vividly, however, was that the sessions I went to were so advanced and so deep into the specifics of DEC and never mentioned anything else—in particular, I didn’t understand how they only talked about VMS and not Ultrix (DECs Unix variant) which was what we were moving to at UMass—I went to learn about Ultrix. I realize now both how naïve I was to the idea of an industry conference, but also how much of a bubble that conference was. Not long after, DEC began a rapid decline and about 10 years later was no longer a standalone company. It was no surprise that shortly after DEC collapsed the book DEC Is Dead, Long Live DEC: The Lasting Legacy of Digital Equipment Corporation, was widely read across Microsoft. Being on top of the world in technology can be so fleeting and the descent so rapid, even if the products are loved—if only regulators could understand that point. BillG knew that. The competitive nature at the core of Microsoft existed for that reason. But what happens without a competitor? That’s how DEC acted—it built products as though there were no competitors, or more precisely that the customer saw the world only through DEC products.
A company that was on top of the world in 2000 was Sun Microsystems, but it was also starting to struggle post the dot com crash that impacted so many of their customers. There was no cloud computing, so starting a company meant buying a lot of Sun computers, but the crash stifled the creation of new companies and those purchases. Scott McNealy, the outspoken cofounder and CEO had said that people bought more Sun computers when the economy was good so they could grow and more when the economy was bad so they could be more efficient. He did not anticipate what Windows NT would do to those expensive computers though. McNealy never held back in his commentary on Microsoft. He reserved his harshest critiques for BillG or SteveB. McNealy’s rivalry with SteveB seemed a bit personal, perhaps because they attended rival private schools in the suburbs of Detroit. As Office XP was making its way to customers, McNealy unloaded on Office, taking aim at the ever-present topic of bloat when it came to, apparently, banning the use of Office at Sun.
Why did we ban it? Let me put it this way: If I want to tell my forty thousand employees to attack, the word “attack” in ASCII is forty-eight bits. As a Microsoft Word document, it’s 90,112 bits. Put that same word in a PowerPoint slide and it becomes 458,048 bits. That’s a pig through the python when you try to send it over the Net.
Nevertheless Sun continued to use Office in finance, presentations, and more. Sun was still a strong company, though that would change in due time, coincidently as their use of Office declined. McNealy held an influential position, particularly when it came to internet technologies. Like Microsoft, Linux also took its toll on Sun.
To bolster his anti-Office stance, in 1999 Sun acquired a maker of a clone of Microsoft Office, a German company, Star Division GmbH. The rumors were that it was cheaper to buy the company and attempt to standardize on its software than it was to buy Microsoft Office. Star Office was working to be fully compatible with Office and available on all platforms, especially Sun’s Java which had consistently shown that it was not up to the task.
The engineers on the product were adamant that it was a “perfect copy” of Office, copyrights, trademarks, and patents aside. At a trade show, when they were still an independent company, I saw a demonstration at their booth. I was younger then and not concerned about the ramifications of obscuring my badge and affiliation. The demonstrator said they were a member of the engineering team while describing the keen attention to cloning Microsoft Office, “even the toolbars”, they said. I pointed out some, um deficiencies. He launched Microsoft Office on their demo machine and shockingly I was correct. He summoned a coworker and after exchanging some thoughts in German told me to come back later in the day to see that they corrected the mistake. I came back and sure enough they did, and I properly introduced myself. Amazing.
With Sun’s ownership and financial support, Star moved from a low-priced product to a free and open-source product. In keeping with the apparent desire to needle SteveB as much as possible, McNealy and team created OpenOffice.org, a philanthropic-sounding consortium to support open source contributions and distributions of an Office competitor. What started off as an odd and failing strategy turned into an annoyance.
Highlighting Star Office as an alternative to Office was one way enterprise customers expressed increasing frustration with bloat. Depending on who you asked the products indeed had many flaws. What we began to consider was that we were now being evaluated on and held to an absolute scale. It was no longer enough to be better than a competitor we vanquished or a new competitor like Star. We needed to be great on our own, an absolute standard.
Despite the flaws, customers continued to buy, and continued to sign up for multiyear agreements. It was difficult to be a serious global company and not be using the full Microsoft platform for PCs, document creation, email, and enterprise infrastructure. Office and Windows were the very definition of product-market fit—we just could not lose a deal. Even in markets where software piracy was the norm, Office and Windows still won against free (and legal) alternatives. This created a disconnect that was difficult for product groups to fully grok.
Winning didn’t necessarily mean we had built the best product. Winning is a combination of product, price, place, and promotion and can come from any or all of them. At least for Word, Excel, and PowerPoint it wasn’t simply that our products were the most satisfactory sold by Microsoft, dominated industry reviews, and consistently won against competitors. Microsoft’s global sales force, product support, and complete product line were all part of a winning equation. Winning led to more winning when it came to familiarity, training, and cross-organization standards. Whatever products do win are by definition the best product.
Microsoft was a big place, and as we grew there were more and more people with ideas who were organizationally disconnected from execution. It was not difficult to find someone putting forth a proposition suggesting this or that risked ending a franchise: the browser, Java, open source, Linux, competitors using one or more of those, or customers rejecting Office due to bloat, poor quality, or complexity and cost in the enterprise. It was too easy to cast aspersions at the parts of Microsoft executing on what it takes to sell billions of dollars of software. These cross-group dynamics between the big money-making products and the new products or the groups simply incubating ideas with ample time to criticize introduced a new kind of tension in the company—one between those that were shipping and those that weren’t (yet).
In the world at large, Windows and Office were becoming listed resume skills and that enabled them to become the punchline to jokes on late night TV, sitcoms, and a constant stream of syndicated cartoons such as Dilbert. Earlier I described when late-night TV host Conan O’Brien did a funny number on Clippy. “Come on, Bill, Microsoft got off easy compared to what the government did to Clippy, that annoying icon that pops up in Word all the time.” (Followed by an animated gunshot and Clippy’s demise.)
We laughed—how could we not? Perhaps it was a rationalization to say that people always hated the tools foisted on them at work. I certainly remember how much people hated all the IBM mainframe software in use at my summer aerospace job and then the MS-DOS software used throughout Cornell. People loved the Mac, until it ate their file at midnight.
Our answer (and my answer) to all of this—the flaws, the increasing expectations, the late products, the quality issues, the jokes, and more—was to execute. Thinking back to that 1:1 when SteveB became president, ever present in my thoughts was what happens when a large development team loses focus and spins out of control. Considering how large our team had become with the addition of SharePoint, my concerns grew deeper. I had no idea just how real this concern was about to become.
As we were planning the follow-on to Office XP, the Windows team was starting to lose control after so carefully maintaining it. Windows XP shipped on August 24, 2001, within weeks of the original goal and six years after Windows 95, and then sketched out a grand vision for the next two releases. The first was code-named Longhorn after a bar in Canada between Blackcomb, which was the code name for the next release, and Whistler, the original code name for XP. Longhorn was planned to be a scaled back version of Blackcomb available sooner, which was being planned simultaneously as Windows traditionally did. Don’t worry, I couldn’t keep track either. There was so much excitement about the plans for Blackcomb that BillG kept pressing and the team was receptive to accelerating the long term work for Blackcomb into the near term Longhorn. It was typical to attempt this when planning two releases in parallel as the second release always appeared more exciting.
Nevertheless Longhorn was slated to be finished in a reasonable timeframe. Windows generally did not have specific completion dates as much as ranges though there were clear dates for early milestones. The release had a set of four strategic initiatives and grand long-term aspirations. These four initiatives were BillG’s self-declared main projects for the next 5 years (two releases) and included major advances in storage (code name WinFS), user interface platform and graphics (code name Avalon), networking (code name Indigo), and a major set of new developer APIs (code name WinFX). Longhorn would embody the biggest and broadest platform strategy ever attempted by Microsoft. I began to share copies of my favorite book on massive engineering, IBM's 360 and Early 370 Systems by Emerson Pugh, detailing the history of the biggest and most enduring computing platform ever built. Could we top that? I probably should have noted more of the history of the project that came before the 360, code-named Project Stretch, which went so poorly the project leader was ostracized within IBM (only to find redemption with the 360).
A really (really) big problem brewing was that Windows XP was struggling in the market, having received muted reviews it often proved difficult for enthusiasts to upgrade and required a significant uptick in PC hardware from OEMs. More alarming though, the virus and malware criminals and troublemakers that attacked Office moved their focus to Windows XP and Windows Server. Those products with significantly more surface area were under assault. The Windows team was scrambling to patch a relentless onslaught of bugs. There was ample consternation over the rationalization that the product was behaving as was intended while also deep concerns about breaking third-party software. If this challenge sounds familiar it is because it is exactly the situation Office was in years earlier.
Windows developed a plan to implement a fast-turnaround service pack that addressed the major holes in the product and to complete it in six to nine months. From my vantage point, having a 50 percent error rate on the estimated schedule completion of a short-term project was already a sign of a team that was not operating in control. The scope of the work, the resources, and the schedule were not aligned. The update to Windows XP was going to need more time, more people, and more work.
There was some good news in how the update progressed. Windows XP as it released was outfitted with Watson technology from Office. For the first time a Windows release was getting real-time information on crashes. The Office team continued to run the Watson service while Windows was able to isolate and fix a very large number of common crashes, as Office did while shipping Office XP.
Six months turned into twelve. Then more. More and more of the team, especially management, were being pulled into security challenges. An annoyance erupted into an existential threat to Windows. Even more importantly, the security issues threatened to prevent Microsoft’s .NET strategy from taking hold and earning product-market fit before even reaching the market. All around the world, the value proposition of only a browser and web pages with code on the server—the strategy espoused by Sun’s McNealy and Oracle’s Ellison—was looking more attractive. At stake was Microsoft’s reputation with enterprise customers.
A favorite saying in the Office hallways was that it is not enough for the leading product to drop the proverbial ball, but someone had to be there to pick it up and run with it. While there were many challenges in market with Windows XP, the real concern was that it was increasingly apparent that the network computer and browser were there to pick up the dropped ball.
There were endless debates over how far to go. I had a long email thread with a VP of Windows sharing my experience asking if “fixing” Windows was even possible? No matter how much was “broken” the architecture was fundamentally open and extensible and thus subject to ongoing assault. I had my doubts this problem was solvable based on my experience in Office. I would return to this topic in just a few years when I moved to Windows.
The definition of product-market fit ultimately prevailed—Windows was the winning product and the market could not get enough of it, even with security issues and incompatibilities. All Microsoft needed was to respond. The company needed to show it was taking the problem seriously.
Following the September 11, 2001 tragedy, Craig Mundie (CraigMu) led an effort to cement Microsoft’s response to security threats. CraigMu, from his position as chief research officer, went around the world representing Microsoft to governments, universities, and enterprise customers. He was deeply in touch with the public sector perception of products and the nature of the existential threat. Working with BillG, Craig and his team authored a memo called Trustworthy Computing (TwC), released in early 2002 and dictated a new set of priorities and new way to develop products. In addition, CraigMu further developed Microsoft’s Security and Response Center and led it to first-class citizenship in the world of cyber defense.
Often the press and outside world extend too much credit to BillG with something big like TwC. In this case, enough credit cannot go to Craig. He was early to this challenge and brought together the product groups and technologists in Washington, DC and around the world, academics, and other domain experts. He navigated these communities and found a way to frame the problem they were expressing so it could be addressed by the disparate organizations at Microsoft including engineering, sales, legal, product support, and more. Even the phrase trustworthy computing was no doubt influenced by the government commissioned report of that same name, which included participation from members of Microsoft Research and Craig’s advanced products group. Bridging the regulatory and technical gap became Craig’s specialty and proved enormously transformative for Microsoft.
TwC brought increased attention to cybersecurity as a boardroom issue for companies, beyond the damage done by viruses and malware. This was something existential to all companies and their customers, not just technology providers. Microsoft’s Executive Briefing Center added sessions on TwC which served to further entrench Microsoft as a thought-leader with enterprise customers. This was a significant turning point in the establishment of deep customer relationships.
The TwC memo also saw broad external distribution (and would be celebrated at decade milestones). Along with establishing the center, mandatory security training for engineers, and a host of commitments to enterprise customers, we also made security a first priority in everything we did. While many offered input and additions to the memo, I was always proud of pushing what I learned from responding to the Word and Outlook viruses. The January 2002 memo prioritized security over features as we did for Office years before—a strong signal to enterprise customers that we would be, essentially, making incompatible changes.
So now, when we face a choice between adding features and resolving security issues, we need to choose security. Our products should emphasize security right out of the box, and we must constantly refine and improve that security as threats evolve. A good example of this is the changes we made in Outlook to avoid e-mail-borne viruses. If we discover a risk that a feature could compromise someone’s privacy, that problem gets solved first. If there is any way we can better protect important data and minimize downtime, we should focus on this. These principles should apply at every stage of the development cycle of every kind of software we create, from operating systems and desktop applications to global Web services.
Message received. We were going to break a lot of stuff. Customers loved the TwC message, but the compatibility concerns would become a constant source of frustration for decades to follow.
Unfortunately executing the product changes to secure Windows XP turned into a 36-month journey (including an interim Service Pack 1), releasing Windows XP Service Pack 2, XP SP2, on August 24, 2004, three years after RTM and much longer than a quick turnaround. There were several major security incidents over the course of this, which either motivated more changes or slowed down releasing broad product changes, depending on perspective. When people say the regulatory climate distracted Microsoft and slowed execution, all I can think about is how much more responding to security did. While not everyone was working on compliance, every single group with code in Windows, Server, Office was making changes, fixing bugs, or investigating potentially risky areas to improve security of products while continuing to function correctly with the changes Windows was making. PC OEMs and independent hardware vendors (IHVs) contributed immensely by updating all the software installed on new PCs and required by hardware devices.
While the first couple of years were rather difficult with Windows XP, it emerged to become a deeply loved and fixture of a PC operating system. When I was working on Windows, we ended up extending official support for the product to nearly 13 years, three years longer than any other product.
Office was in a different place. We did not face the security challenges to the same degree but faced the needling of Scott McNealy, softening demand for new Office features, and high-friction upgrades from enterprise customers, and the ever-present risks of browser-computing.
More importantly, we were on the verge of understanding how Office could be a full participant in “services”. We began crafting our plans to finally deliver on those offsites from almost a decade ago where we were asked the question of how to turn software into an “annuity” business.
We faced the challenge of forging into new areas and doing new things. Everybody had ideas, which meant saying “no” became a big part of the process. But who would say no and to what and why?
On to 070. Office.NOT
The Windows Scalability slide brings back a lot of memories. :-)
WinXP was the first OS with Watson. As such, a lot of low-lying fruit was there to be picked; many bugs (in Windows itself) got reported and fixed. But alas, the fixes were being held up, because Win management really really wanted to simul-ship the Windows XP Service Pack with Windows Server. Lots of good reasons for this, but there's a limit!
Watson has a feature where we can tag buckets with a link; if you hit Send on that bucket, we bring up the browser, typically saying "we fixed this, go <here> for the update". But since the SP hadn't shipped, the page (for a zillion buckets) was authored to say (in effect) "hey we've fixed this bug, really we have, and you'll get the fix really soon, but not just yet". Needless to say, this made not a lick of sense to our customers.
Win XP probably ran on 1000x as many machines as Server. No one was clamoring for the Server release, but all my XP peeps were being denied their fixes, and it just kept dragging on. So I wrote to whatever Windows Veep ran the thing, and told him to get off the pot, sever the connection, ditch the simship, deliver my SP. To his credit, they did so, and I was happy.
And then, twenty years later at a party, somehow the subject came up, and some guy I'd never met before said "that was you?", and was ready to punch me out over it. I guess he'd been square in the simship clan, and I'd shoved him aside.