003. Klunder College
003. Klunder College
Writing Bulletproof Code
Go back to 002. SteveSi
ADC, Apps Developer College, was more a curriculum than a college, run by Dan Newell (DanN) and Doug Klunder (DougK). It was called Klunder College because Doug created it. ADC was basically a couple of three-ring binders of assorted documentation and memos and a bunch of self-paced coding exercises that constituted a new and unique approach to on-boarding at Microsoft at a time when most everyone who programmed was self-taught. The idea of a programming orientation or bootcamp seemed unnecessary, perhaps even insulting. I would take away much more than I could really understand at such an early juncture in my career as I immersed myself in my first lessons in culture.
Most teaching was done during a meeting with Dan, and usually by stopping by or hovering at his office. Even though we had offices, there was a constant roaming of the hallways and stopping by unscheduled to see people. This, I would soon learn, was the Microsoft management and learning culture—self-sufficient, informal, and interrupt-driven (a specific computer term that became one of my first Microsoft-isms—“What’s the best way to meet them? … Oh, they are interrupt-driven.”) It was a big change from the structure of universities but also consistent with how most everyone there had learned PC programming in the early days.
DanN’s office was filled with vinyl records, a stacked stereo system, and a few early ’80s music posters. He was an experienced Microsoft SDE and was half the leadership of ADC. A few doors down was DougK. In contrast to Dan’s office, Doug’s office was completely spartan, as though he had only recently moved in. Doug looked like a member of the Doobie Brothers, with a long beard, flannel shirt, cords, and no shoes. He was exactly what my mother had warned me about.
Doug was a programming legend at Microsoft. After graduation from MIT he joined Microsoft as the first college hire and subsequently an informal leader in the quest to hire directly from college, especially in Apps. He was one of the earliest Apps SDEs and had written much of the code in an early spreadsheet for MS-DOS that Microsoft released as Multiplan but called Electronic Paper, or EP, while under development.
Dan told me that BillG decided the company’s future was on graphical user interface (GUI) like OS/2 and Macintosh so the company chose not to bring an updated MS-DOS (CUI, or character user interface) spreadsheet to market. Doug was so frustrated by this decision that he quit Microsoft and went off to work on a farm in California. Doug’s innovative work was critical to Mac Excel 1.0, which ultimately shipped for the original Macintosh. Later, he returned to help finish Mac Excel 1.0 and contribute broadly to Apps in the transition to GUI. Doug was the ideal person to indoctrinate us into the ways of Microsoft Apps.
My first day had been a success, or at least not a failure.
After a few weeks of ADC, I finally received an email from ScottRa. He suggested we meet the next day first thing. “How about 11 a.m.?”
Microsoft SDEs bordered on nocturnal in those days. This was consistent with how college programmers coped with the scarcity of computers. It was always best to work late at night when fewer people were trying to get to terminals and, if on a shared mainframe, slowing it down. Everyone was working nights at the office back then. There was no way to even do email from home and certainly not any coding. The old Xenix email system made it easy to see if a person was logged on, and rumor was that BillG was always checking in on key people to see if they were connected. These were all traits of the original hacker ethos that had worried my mother.
When asked if Microsoft had “flex time” (an ’80s buzzword) by prospective college hires, we always said, “Yes. You can choose to work whichever 80 hours of the week you want to work.” That was, essentially, true for the ’90s. Our views later matured as did the company, much to the chagrin of new old-timers like me. Mostly we were in our 20s and loved what we were doing.
Scott explained what was in store for me for the next few months. Before programming anything I needed to learn the unique dialect Microsoft used. While I knew the programming language, Microsoft had a unique style called Hungarian, named for Charles Simonyi (CharlesS), one of the rarified level 14 architects in the company and the only one in Apps. CharlesS was recruited by BillG from Xerox PARC where he had built the first GUI word processor. Hungarian was the secret handshake used between programmers at Microsoft and it was unlike anything I’d ever seen. In college programming or in books, one might use a name in code such as FormatLine. In Hungarian, we used names like FFormatLineFspec, which were chosen to make code more manageable for large teams.
I also needed to learn the tools used to build Excel and Word. There was a proprietary programming language called CSL, also named after CharlesS. This language, based on C, had a virtual machine, which made it easier to run on other operating systems (in theory) and also had a good debugger—attributes that were lacking in the relatively immature C product from Microsoft. I also learned RAID, the database tool that the product groups used to track bugs in products (get it, complete with backronym of Reporting and Incidents Database). And, most importantly, I learned SLM, pronounced slime, the source code tracking tool (like GitHub much later). Through this I used shipping code and coded up features and fixed bugs as exercises, never checking them into production. It sounded pretty cool. It was pretty cool.
I loved talking with Doug. He was not like anyone I had spent a lot of time with—he was truly the hippie/hacker Time described. As I got to know him, I learned of some of his relatively extreme perspectives. He was obsessed with privacy. He didn’t have a driver’s license, bank account, telephone, or anything, but still lived among us. I often saw him outside of work. We lived in the same neighborhood (Capitol Hill, soon the heart of grunge music) where he always paid cash at one of the restaurants in the neighborhood. This dedication to privacy proved even more ironic, and prescient, as he went on to build Microsoft Money, which he used to track his cash transactions. In his post-Microsoft career, Doug became an attorney and, admirably, went on to work for the ACLU on issues such as privacy. He was ahead of his time.
Importantly, for my own future, Doug instilled in me a sense of principled product development. Throughout my time at ADC Doug shared his account of the decisions around building Mac Excel instead of a new MS-DOS spreadsheet, including a famous Red Lion Inn offsite where the strategic decision was made to bet on the graphical interface. BillG insisted on prioritizing GUI over CUI even with a potentially killer MS-DOS spreadsheet close to complete, causing DougK to ultimately quit out of principle. Doug’s principled lessons followed me through my career.
Doug was an incredible programmer, the first of many amazing ones I met. He had a full map of an entire body of product code in his head and a deep understanding of the data structures and code paths—more than encyclopedic but organic as though his brain had merged with the code. In hindsight, I came to understand that great programmers have the same relationship to code and products as great writers do to words and complete works, or filmmakers do to camera shots and complete films. Every line of code, every data structure, was not only deliberate but deeply related to the other choices one makes.
One issue we spent an enormous amount of time discussing was memory management, which was an acute problem in PCs those days—how to fit more information in less space. This was not simply about data but also the amount of code in an application—how to do more with less. Doug recounted how he had developed minimal recalc for Excel (originally for the unreleased spreadsheet product), which was a way of only recalculating the cells in a spreadsheet that needed to be calculated when something changed. Previously every change in a spreadsheet recalculated the entire sheet, which was slow and memory intensive. It is difficult to overstate the brilliance in Doug’s approaches. It would be equally fair to say that Doug wrote code only Doug could understand, very much how great products were built then.
Many of the conventions or techniques used in Apps were pioneered by Doug and taught or, more correctly, transfused to us in ADC. Even though software products were often late (very late) and quality was spotty (very spotty), that was only in hindsight. In general, Microsoft’s Apps, particularly MS-DOS Word and Excel, were generally among the most robust and highest quality. DougK and DanN instilled several key lessons about being an Apps software design engineer:
Scheduling and estimating work. Individuals needed to be really good at scheduling their own work and coming up with estimates that were not only precise to the day but accurate as well.
Bulletproof code. Developers were responsible for building code beyond their features, including the code that ran in “debug mode” that was constantly checking the integrity of the data structures and ensuring that assumptions made by programmers were validated. During any code review in ADC, the code was sent back if it was not fully defined by what were called debug ASSERTs, or code that was run to verify the values of variables and integrity of data structures while running a special build of the product for this purpose.
Self-documenting code. Schools taught that good code had comments or annotations that were not code but English that explained the code. Doug hated comments and insisted that code should be the ultimate comment itself. This was a bit ironic because he was known for developing some insanely efficient code that was also difficult to read and would have benefitted from comments. His view was that comments were always out of date and far less precise than the code itself. I really had to unlearn commenting, a practice I had embraced.
Performance in speed and minimal memory. CPUs were slow and memory was extremely scarce, so there was a big focus on writing efficient code. This was rooted in BillG and was an important part of Microsoft’s early developer culture and key contributor to success. Code written this way was hardcore.
During ADC, I built a memory allocator, code that handled requests from a program to provide memory to store data and later free memory. Early programs, especially written in C or CSL, were notorious for mismanaging memory, which caused program crashes and in turn crashed the entire computer along with it.
Doug had developed a set of ideas for building a bulletproof memory allocator, one that tracked allocations and reallocations and made sure memory was properly initialized before it was used in the code. These were super common bugs in PC software.
It was a huge learning experience. But I had also been working on modern, automatic memory allocation called garbage collection (GC) in graduate school. After a few days of hacking away on my memory allocator project for Doug, I worked up the guts to approach him and ask why Microsoft did not use GC.
It didn’t go so well. Doug literally laughed at me. Still, after I persisted, he suggested I go meet Jon DeVaan (JonDe) on the Excel team who could go through every bug in Excel (it was up to version 2.x and ran on Windows) and explain how many came from bad memory management code. Doug knew that his memory allocator, also my programming exercise, was a significant barrier to creating memory allocation bugs in the first place. He wanted me to see for myself.
JonDe didn’t laugh at me like Doug had, rather he listened then indulged me by going through 30 minutes of RAID searches looking at memory management bugs to see if GC fixed them. GC prevented perhaps eight of 7,500 in Excel. That made for a convincing argument that GC was not remotely worth the tradeoff in memory and performance. GC would eventually become standard practice on the web and on Apple’s iPhone1, but I was a decade too optimistic.
JonDe was among the best Microsoft had ever produced, in addition to being one of the best engineering managers to have ever worked at the company. We worked together for the remainder of my time at Microsoft, each with unique strengths, making each other better at what we did.
I spent an inordinate amount of time installing Microsoft products and using them. I developed my own method for using a network boot disk and getting a “clean” PC up and running. This skill seemed to be both a necessity and something each person reinvented on his or her own. PCs still barely worked.
One product I spent a lot of time on that was still under development, Windows 3.0, a year away from finishing but was already the buzz of the developers running it. At the time, the mainstream SDE machine was running OS/2 because it was able to use more memory and handle more programs gracefully, but it also had a lot of limitations, such as a lack of apps and the inability to print. There was always much cafeteria discussion about how lame OS/2 was, wondering what was going on over in Systems with our partnership with IBM.
Windows 2.x was already in market and had some early pioneers supporting it, including Ray Ozzie at Lotus, who had developed the innovative Notes product. The biggest supporter of Windows 2.x was Excel, which was also developing version 3 of Excel to be on both Windows and Mac (and OS/2) at the same time, using an innovative cross-platform layer of software.
Back then, buyers did not get a PC with Windows. PCs came with MS-DOS. Using Excel meant buying Excel, and it came with Windows. According to branding and packaging, Windows was an operating environment, or more like an app on top of MS-DOS. This became the subject of antitrust consternation years later. As innovative and successful as Excel was on the Mac, Excel on Windows was not competitive with the MS-DOS Lotus 1-2-3. Microsoft’s older CP/M and later MS-DOS spreadsheet Multiplan was a distant number two.
Windows 3 was a bit of a skunkworks project and was a bet on a new architecture called protected mode, which enabled multiple programs to operate at the same time and share a larger amount of memory. In many ways this was also what OS/2 was supposed to do, but OS/2 had much grander visions as well as a difficult engineering partner in IBM (or IBM might say it was Microsoft that was the difficult partner). Windows 3 was beginning to look more like an operating system and less like an add-on to MS-DOS. It was interesting to install and play around with, which all of us did. Actively under development were versions of Word and Excel and several other products. We spent the summer trying everything.
DanN shared with me what he was most excited about—one of the key “secrets” of Windows 3, which was how the product enabled protected mode but could also remain compatible with old MS-DOS programs. David Weise (DavidW) and Murray Sargent (MurrayS) had invented some novel uses of the Intel chipset that even Intel did not anticipate, which enabled these efforts. One programming trick called PrestoChangeoSelector was a key “hack” they developed and later became an absurd symbol of “secret” application programmable interfaces (APIs) in Windows (absurd, because it was supposedly secret, when in fact it was right there to see). Dan told this story with great Microsoft pride, as the development of Windows 3 and these techniques represented much of what Microsoft did so well in those days. Hack.
I admired Dan. What Doug brought to ADC in insights for coding, Dan brought in big picture about how products were built and team dynamics. Dan knew everyone at the company and was definitely a cool kid. Because of Dan I got to meet quite a few of the senior people across both Apps and Systems. Systems was pretty okay to Apps people and vice versa, but there was clearly both an organizational and cultural divide between the two—a theme that I would experience for many years to come and also be the subject of many “battles”.
A few months in, Scott came by and said he was anxious for me to join the team. My time in ADC was abbreviated so I could get cracking on our project. Unbeknownst to me, I was joining a newly formed “tiger team” created to build an entirely new product to streamline building applications.
Object-oriented, GUI, and Microsoft’s next killer product (killer was Microsoft jargon), plus Scott told me our project was “super important to BillG.”
On to 004. Everything is Buggy
Some readers have said that the iPhone does not support “garbage collection” because it only supports a technique known as “reference counting”. In casual use and in particular in debates about the future of iPhone’s language at the time ,Objective-C, the phrase “garbage collection” referred to an advanced technique “tracing garbage collection”. Specifically within computer science and also contemporaneously, these were both forms of the general technique of automatic memory management or garbage collection. From Wikipedia, “Tracing garbage collection is the most common type of garbage collection – so much so that "garbage collection" often refers to tracing garbage collection, rather than other methods such as reference counting – and there are a large number of algorithms used in implementation.” Certainly in my grad school lab, reference counting (as we used in Smalltalk) was garbage collection. (Fetched at time of this post)
I recall the release of Excel that installed enough of the not-yet-released Windows to run as a GUI app.
Later I would be at an event in New York City where Microsoft and IBM personnel attempted to explain why there would still be Windows (at 2.03 or maybe 2.10) at a client level and that OS/2 would provide a higher-level arrangement.
In July 1992 I was in Palo Alto doing some house-hunting as I prepared to transfer to XSoft (situated adjacent to PARC). Since I was in town, I went up to San Francisco for a Microsoft event in the Moscone Center. It was the reveal of Windows/NT and that was going to be that.
I got to meet Dave Cutler at that event and he reminisced about his love for the Univac 110x at Dupont, before moving on to DEC in his career. The key take-away from conversations around Dave as a system architect concerned choosing the invariants to control and what others to leave loosely observed. It was no surprise to learn, years later, about Dave taking charge of the integration and shaming NT build-breakers.
By then the World was on Wiindows 3.10 and 95 was still to come.
Before I came to Silicon Valley in 1992 I was in a self-designated skunk works project to build a multifunction digital scanner/copier/printer with a PC as the processing unit. It was a pilot effort and some units would end up in Hallmark shops for producing customized greeting cards. We were using beta releases of Windows 3 and also fussing with co-processor boards to cope with the scanner. When the Windows 3 launch happened and swamped the computer stores the WTF moment was, "Wait, why had we not bought Microsoft stock?"
There were a lot of bad bets. It seemed that Lotus had concluded to do 1-2-3 on OS/2 first, probably well-incentivized, but it left Excel in an unassailable position.
At Xerox, the introduction of the Apple Lisa was taken as validating their march to commercialize GUI-based workstations. There was an effort to make developer-oriented releases of the workstation and provide the developer tools (i.e., the Mesa language compiler and other tools). It was expensive, and the sales folk saw it as an opportunity to sell laser printers, also not cheap. The product required a network (XNS protocols at that point). I doubt they sold any.
At PARC there was much Apple envy, and I think they wanted to be Apple. Microsoft was viewed with disdain. Xerox had introduced some consumer level computers, starting with a CP/M-80 all-in-one based on the same motherboard as the Kaypro. When the PC hit, an Olivetti clone (also adopted by AT&T) was marketed. There were even short-lived Xerox stores. At one point, Xerox even did an employee buy-back effort to get more of those discontinued units for use in the business. The roll-out of Xerox workstations had not happened yet.
There was great confusion in Palo Alto on how Xerox was fumbling the future as it was viewed from there. Fundamentally, there was not going to be such a pivot for an $18 billion manufacturing company built around a direct sales and leasing business.
More fundamentally, there was a problem about the perfect killing off the good. As may be typical of companies that exploded around a killer product (the original Xerox hallway copier), there was an intense craving to be the next trick. Xerox had quality erections around laser imaging, fonts, digital formats (not Postscript) and production printing. Meanwhile, Hewlett-Packard, with its Canon engine, not only captured the commodity laser-printer market, it also determined what people would see as good printing. In that context, Xerox write-white technology, adapted from optical copiers, suddenly didn't look right, no matter what the test-print quality metric was. The more-telling case was with inkjet though. HP inkjet technology was considered not good enough and Xerox engineers wanted to kill it with quality. Timing is everything in more than comedy.
It was also a problem for Xerox that it manufactured its own processors and chips. The processors were important for embedded digital processing in copiers and multifunction units, but they could not get to commodity scale and it impacted all products. Engineers preferred Motorola and PCI boards and did not chase Intel. Still, big products depended on the company's own foundry.
This section is just wonderful and brings back lots of memories, so weird that we were padding around the same buildings and we never met. Doug Klunder was just my everything at Microsoft, he and Jabe Blumenthal and I did so many things together. Doug was brilliant and also the kindest. sweetest, funniest person you would ever meet. When Jabe designed Excel 1.0 and Doug wrote (most) of Excel 1.0 and I introduced Excel 1.0, well, that was one of the two most fun years I've ever experienced in my career (the other was conceiving of and launching what became ESPN.com in 1995). The Windows/OS/2 conundrum was just a classic case of shit happening when good people work hard, kind of the opposite of IBM which was filled with mediocre people not working very hard. Great to hear from Jon DeVaan, he had the world's most epic Star Trek calendar in his humble office when I first met him in 1984.