003. Klunder College
Writing Bulletproof Code
Go back to 002. SteveSi
ADC, Apps Developer College, was more a curriculum than a college, run by Dan Newell (DanN) and Doug Klunder (DougK). It was called Klunder College because Doug created it. ADC was basically a couple of three-ring binders of assorted documentation and memos and a bunch of self-paced coding exercises that constituted a new and unique approach to on-boarding at Microsoft at a time when most everyone who programmed was self-taught. The idea of a programming orientation or bootcamp seemed unnecessary, perhaps even insulting. I would take away much more than I could really understand at such an early juncture in my career as I immersed myself in my first lessons in culture.
Most teaching was done during a meeting with Dan, and usually by stopping by or hovering at his office. Even though we had offices, there was a constant roaming of the hallways and stopping by unscheduled to see people. This, I would soon learn, was the Microsoft management and learning culture—self-sufficient, informal, and interrupt-driven (a specific computer term that became one of my first Microsoft-isms—“What’s the best way to meet them? … Oh, they are interrupt-driven.”) It was a big change from the structure of universities but also consistent with how most everyone there had learned PC programming in the early days.
DanN’s office was filled with vinyl records, a stacked stereo system, and a few early ’80s music posters. He was an experienced Microsoft SDE and was half the leadership of ADC. A few doors down was DougK. In contrast to Dan’s office, Doug’s office was completely spartan, as though he had only recently moved in. Doug looked like a member of the Doobie Brothers, with a long beard, flannel shirt, cords, and no shoes. He was exactly what my mother had warned me about.
Doug was a programming legend at Microsoft. After graduation from MIT he joined Microsoft as the first college hire and subsequently an informal leader in the quest to hire directly from college, especially in Apps. He was one of the earliest Apps SDEs and had written much of the code in an early spreadsheet for MS-DOS that Microsoft released as Multiplan but called Electronic Paper, or EP, while under development.
Dan told me that BillG decided the company’s future was on graphical user interface (GUI) like OS/2 and Macintosh so the company chose not to bring an updated MS-DOS (CUI, or character user interface) spreadsheet to market. Doug was so frustrated by this decision that he quit Microsoft and went off to work on a farm in California. Doug’s innovative work was critical to Mac Excel 1.0, which ultimately shipped for the original Macintosh. Later, he returned to help finish Mac Excel 1.0 and contribute broadly to Apps in the transition to GUI. Doug was the ideal person to indoctrinate us into the ways of Microsoft Apps.
My first day had been a success, or at least not a failure.
After a few weeks of ADC, I finally received an email from ScottRa. He suggested we meet the next day first thing. “How about 11 a.m.?”
Microsoft SDEs bordered on nocturnal in those days. This was consistent with how college programmers coped with the scarcity of computers. It was always best to work late at night when fewer people were trying to get to terminals and, if on a shared mainframe, slowing it down. Everyone was working nights at the office back then. There was no way to even do email from home and certainly not any coding. The old Xenix email system made it easy to see if a person was logged on, and rumor was that BillG was always checking in on key people to see if they were connected. These were all traits of the original hacker ethos that had worried my mother.
When asked if Microsoft had “flex time” (an ’80s buzzword) by prospective college hires, we always said, “Yes. You can choose to work whichever 80 hours of the week you want to work.” That was, essentially, true for the ’90s. Our views later matured as did the company, much to the chagrin of new old-timers like me. Mostly we were in our 20s and loved what we were doing.
Scott explained what was in store for me for the next few months. Before programming anything I needed to learn the unique dialect Microsoft used. While I knew the programming language, Microsoft had a unique style called Hungarian, named for Charles Simonyi (CharlesS), one of the rarified level 14 architects in the company and the only one in Apps. CharlesS was recruited by BillG from Xerox PARC where he had built the first GUI word processor. Hungarian was the secret handshake used between programmers at Microsoft and it was unlike anything I’d ever seen. In college programming or in books, one might use a name in code such as FormatLine. In Hungarian, we used names like FFormatLineFspec, which were chosen to make code more manageable for large teams.
I also needed to learn the tools used to build Excel and Word. There was a proprietary programming language called CSL, also named after CharlesS. This language, based on C, had a virtual machine, which made it easier to run on other operating systems (in theory) and also had a good debugger—attributes that were lacking in the relatively immature C product from Microsoft. I also learned RAID, the database tool that the product groups used to track bugs in products (get it, complete with backronym of Reporting and Incidents Database). And, most importantly, I learned SLM, pronounced slime, the source code tracking tool (like GitHub much later). Through this I used shipping code and coded up features and fixed bugs as exercises, never checking them into production. It sounded pretty cool. It was pretty cool.
I loved talking with Doug. He was not like anyone I had spent a lot of time with—he was truly the hippie/hacker Time described. As I got to know him, I learned of some of his relatively extreme perspectives. He was obsessed with privacy. He didn’t have a driver’s license, bank account, telephone, or anything, but still lived among us. I often saw him outside of work. We lived in the same neighborhood (Capitol Hill, soon the heart of grunge music) where he always paid cash at one of the restaurants in the neighborhood. This dedication to privacy proved even more ironic, and prescient, as he went on to build Microsoft Money, which he used to track his cash transactions. In his post-Microsoft career, Doug became an attorney and, admirably, went on to work for the ACLU on issues such as privacy. He was ahead of his time.
Importantly, for my own future, Doug instilled in me a sense of principled product development. Throughout my time at ADC Doug shared his account of the decisions around building Mac Excel instead of a new MS-DOS spreadsheet, including a famous Red Lion Inn offsite where the strategic decision was made to bet on the graphical interface. BillG insisted on prioritizing GUI over CUI even with a potentially killer MS-DOS spreadsheet close to complete, causing DougK to ultimately quit out of principle. Doug’s principled lessons followed me through my career.
Doug was an incredible programmer, the first of many amazing ones I met. He had a full map of an entire body of product code in his head and a deep understanding of the data structures and code paths—more than encyclopedic but organic as though his brain had merged with the code. In hindsight, I came to understand that great programmers have the same relationship to code and products as great writers do to words and complete works, or filmmakers do to camera shots and complete films. Every line of code, every data structure, was not only deliberate but deeply related to the other choices one makes.
One issue we spent an enormous amount of time discussing was memory management, which was an acute problem in PCs those days—how to fit more information in less space. This was not simply about data but also the amount of code in an application—how to do more with less. Doug recounted how he had developed minimal recalc for Excel (originally for the unreleased spreadsheet product), which was a way of only recalculating the cells in a spreadsheet that needed to be calculated when something changed. Previously every change in a spreadsheet recalculated the entire sheet, which was slow and memory intensive. It is difficult to overstate the brilliance in Doug’s approaches. It would be equally fair to say that Doug wrote code only Doug could understand, very much how great products were built then.
Many of the conventions or techniques used in Apps were pioneered by Doug and taught or, more correctly, transfused to us in ADC. Even though software products were often late (very late) and quality was spotty (very spotty), that was only in hindsight. In general, Microsoft’s Apps, particularly MS-DOS Word and Excel, were generally among the most robust and highest quality. DougK and DanN instilled several key lessons about being an Apps software design engineer:
Scheduling and estimating work. Individuals needed to be really good at scheduling their own work and coming up with estimates that were not only precise to the day but accurate as well.
Bulletproof code. Developers were responsible for building code beyond their features, including the code that ran in “debug mode” that was constantly checking the integrity of the data structures and ensuring that assumptions made by programmers were validated. During any code review in ADC, the code was sent back if it was not fully defined by what were called debug ASSERTs, or code that was run to verify the values of variables and integrity of data structures while running a special build of the product for this purpose.
Self-documenting code. Schools taught that good code had comments or annotations that were not code but English that explained the code. Doug hated comments and insisted that code should be the ultimate comment itself. This was a bit ironic because he was known for developing some insanely efficient code that was also difficult to read and would have benefitted from comments. His view was that comments were always out of date and far less precise than the code itself. I really had to unlearn commenting, a practice I had embraced.
Performance in speed and minimal memory. CPUs were slow and memory was extremely scarce, so there was a big focus on writing efficient code. This was rooted in BillG and was an important part of Microsoft’s early developer culture and key contributor to success. Code written this way was hardcore.
During ADC, I built a memory allocator, code that handled requests from a program to provide memory to store data and later free memory. Early programs, especially written in C or CSL, were notorious for mismanaging memory, which caused program crashes and in turn crashed the entire computer along with it.
Doug had developed a set of ideas for building a bulletproof memory allocator, one that tracked allocations and reallocations and made sure memory was properly initialized before it was used in the code. These were super common bugs in PC software.
It was a huge learning experience. But I had also been working on modern, automatic memory allocation called garbage collection (GC) in graduate school. After a few days of hacking away on my memory allocator project for Doug, I worked up the guts to approach him and ask why Microsoft did not use GC.
It didn’t go so well. Doug literally laughed at me. Still, after I persisted, he suggested I go meet Jon DeVaan (JonDe) on the Excel team who could go through every bug in Excel (it was up to version 2.x and ran on Windows) and explain how many came from bad memory management code. Doug knew that his memory allocator, also my programming exercise, was a significant barrier to creating memory allocation bugs in the first place. He wanted me to see for myself.
JonDe didn’t laugh at me like Doug had, rather he listened then indulged me by going through 30 minutes of RAID searches looking at memory management bugs to see if GC fixed them. GC prevented perhaps eight of 7,500 in Excel. That made for a convincing argument that GC was not remotely worth the tradeoff in memory and performance. GC would eventually become standard practice on the web and on Apple’s iPhone1, but I was a decade too optimistic.
JonDe was among the best Microsoft had ever produced, in addition to being one of the best engineering managers to have ever worked at the company. We worked together for the remainder of my time at Microsoft, each with unique strengths, making each other better at what we did.
I spent an inordinate amount of time installing Microsoft products and using them. I developed my own method for using a network boot disk and getting a “clean” PC up and running. This skill seemed to be both a necessity and something each person reinvented on his or her own. PCs still barely worked.
One product I spent a lot of time on that was still under development, Windows 3.0, a year away from finishing but was already the buzz of the developers running it. At the time, the mainstream SDE machine was running OS/2 because it was able to use more memory and handle more programs gracefully, but it also had a lot of limitations, such as a lack of apps and the inability to print. There was always much cafeteria discussion about how lame OS/2 was, wondering what was going on over in Systems with our partnership with IBM.
Windows 2.x was already in market and had some early pioneers supporting it, including Ray Ozzie at Lotus, who had developed the innovative Notes product. The biggest supporter of Windows 2.x was Excel, which was also developing version 3 of Excel to be on both Windows and Mac (and OS/2) at the same time, using an innovative cross-platform layer of software.
Back then, buyers did not get a PC with Windows. PCs came with MS-DOS. Using Excel meant buying Excel, and it came with Windows. According to branding and packaging, Windows was an operating environment, or more like an app on top of MS-DOS. This became the subject of antitrust consternation years later. As innovative and successful as Excel was on the Mac, Excel on Windows was not competitive with the MS-DOS Lotus 1-2-3. Microsoft’s older CP/M and later MS-DOS spreadsheet Multiplan was a distant number two.
Windows 3 was a bit of a skunkworks project and was a bet on a new architecture called protected mode, which enabled multiple programs to operate at the same time and share a larger amount of memory. In many ways this was also what OS/2 was supposed to do, but OS/2 had much grander visions as well as a difficult engineering partner in IBM (or IBM might say it was Microsoft that was the difficult partner). Windows 3 was beginning to look more like an operating system and less like an add-on to MS-DOS. It was interesting to install and play around with, which all of us did. Actively under development were versions of Word and Excel and several other products. We spent the summer trying everything.
DanN shared with me what he was most excited about—one of the key “secrets” of Windows 3, which was how the product enabled protected mode but could also remain compatible with old MS-DOS programs. David Weise (DavidW) and Murray Sargent (MurrayS) had invented some novel uses of the Intel chipset that even Intel did not anticipate, which enabled these efforts. One programming trick called PrestoChangeoSelector was a key “hack” they developed and later became an absurd symbol of “secret” application programmable interfaces (APIs) in Windows (absurd, because it was supposedly secret, when in fact it was right there to see). Dan told this story with great Microsoft pride, as the development of Windows 3 and these techniques represented much of what Microsoft did so well in those days. Hack.
I admired Dan. What Doug brought to ADC in insights for coding, Dan brought in big picture about how products were built and team dynamics. Dan knew everyone at the company and was definitely a cool kid. Because of Dan I got to meet quite a few of the senior people across both Apps and Systems. Systems was pretty okay to Apps people and vice versa, but there was clearly both an organizational and cultural divide between the two—a theme that I would experience for many years to come and also be the subject of many “battles”.
A few months in, Scott came by and said he was anxious for me to join the team. My time in ADC was abbreviated so I could get cracking on our project. Unbeknownst to me, I was joining a newly formed “tiger team” created to build an entirely new product to streamline building applications.
Object-oriented, GUI, and Microsoft’s next killer product (killer was Microsoft jargon), plus Scott told me our project was “super important to BillG.”
On to 004. Everything is Buggy
Some readers have said that the iPhone does not support “garbage collection” because it only supports a technique known as “reference counting”. In casual use and in particular in debates about the future of iPhone’s language at the time ,Objective-C, the phrase “garbage collection” referred to an advanced technique “tracing garbage collection”. Specifically within computer science and also contemporaneously, these were both forms of the general technique of automatic memory management or garbage collection. From Wikipedia, “Tracing garbage collection is the most common type of garbage collection – so much so that "garbage collection" often refers to tracing garbage collection, rather than other methods such as reference counting – and there are a large number of algorithms used in implementation.” Certainly in my grad school lab, reference counting (as we used in Smalltalk) was garbage collection. (Fetched at time of this post)
There is so much in this section, but the number one take away has to be the immense historical chance involved in Windows being successful. Literally two people, banking on a prior relationship, with DavidW working in his spare time, or perhaps also ignoring his day job completely. Let's be clear, that work was not on strategy, it was not sanctioned by management, and yet it changed the trajectory of Microsoft and the entire PC Revolution (as you call it).
The Hungarian naming convention sounds crazy, and the Systems group at Microsoft was never fully convinced of its benefit, but the benefit was real. The reason? Start with DougK’s maxims of debugging data not code and writing self-documenting code.
At its base, program code contains instructions that transform the program data. The code has no meaning without its data. So it makes perfect sense to make the data structures a primary element of the symbolic names used in the code. This allows any human code reader to see what data transformations are happening in every line of code. The benefit is many common programming errors are immediately visible.
Compiler theory was way ahead of reality in these early days. Modern programmers take type checking in structured languages for granted. But due to RAM limitations, the early compilers did not actually do type checking. Hungarian reduced mismatched type errors to nearly zero. Eventually there was proof.
Later compilers did type checking, but many teams working on old code bases never turned it on because it created a huge backlog of work. Teams that used Hungarian found turning on type checking relatively easy. Much later, when Microsoft Research was developing its first static analysis code review tools, the rigorously Hungarian code bases had many fewer type based errors than the others.