I work in the Linux Production Kernel (ProdKernel) Memory Management group. My current projects are:

Kernel Metrics Technical Debt Google generates a variety of telemetry and metrics using an internal debugfs filesystem, which is a source of technical debt. This prevents some subsystems (like Memory Management) from adding many statistics, since the cost of rebasing is too high. I'm working on a plan to reduce this by working with upstream statistics and metrics projects to add a similar filesystem to the upstream kernel, so that we can switch to that.

Upstream A/B performance testing Helping maintain and extend an existing single-machine benchmarking system to work on newer upstream kernels. The goal is to allow us to both contribute performance data to the upstream community on hardware we care about (and find regressions earlier), and enable our kernel team to more easily measure and share performance results on patches being sent upstream. I'm supporting the MM-related benchmarks, and helping our team and our collaborators get quick turn-around on their benchmarks.

ProdKernel Rotation I'm a member of the "ProdKernel Rotation", where I spend one week out of every 8-12 being the kernel representative for on-call issues that have stumped SRE. This is a working-hours-only on-call shift, and (since such issues are rare) it mostly involves triage of incoming syzkaller bugs and handing off those issues to the relevant teams.

Previously, I've worked on:

Far Memory Investigations I worked with the Borglet (Google's distributed system monitor) and Hardware Platform teams to investigate integrating 3D Xpoint and other "distant" memory (cheaper than DRAM but slower) into our production fleet. I investigated of PMU-based read/write sample data using Python and pandas to see if we could use it to watch access patterns to drive data motion between DRAM and Far Memory, but it didn't pan out.

Investigating L1TF mitigations I spent a couple months working to investigate directions for Address-space-isolation-based mitigations for the L1TF Intel vulnerability

Co-hosted two interns With another engineer, I hosted two rising-Junior summer interns, who investigated alternative compression & allocation algorithms for our production zswap stack. Introduced them to and mentored them in professional engineering, linux development, cherry-picking patches, kernel testing, crash dumps, and kernel debugging.

RDBMS Data Layer Group   From September 2017 – April 2018 I was in the Oracle RDBMs Data Layer group, working on a project to apply persistent memory (3D XPoint, etc.) to the database, and doing some cleanups to familiarize myself with the codebase. I continued to consult with the Solaris team when issues came up.

Oracle's acquisition of Sun Microsystems completed in February 2010. From February 2010 – September 2017, I worked on the Virtual Memory team of the Solaris Kernel Group doing:

VM 2.0 Phase 1   I spent two and a half years as one of the primary engineers on the first phase of VM 2.0, which re-implemented the "physical side" of the Solaris VM subsystem, which was ~30 years old. The new version bakes in large pages, the physical partitioning of user and kernel memory, and Non-Uniform Memory Access (NUMA) into the basic design. Phase 1 was a >100k-line project built by a team of 5 engineers consisting of 10-11 intertwined sub-projects.

I pair-programmed with another engineer (the technical lead), and we built around 7 or 8 of those sub-projects, including the basic page state tracking structure and memory freelists, dynamic addition and removal of memory, memory power management support, memory allocation credits, queues for waiting for memory availability, and debugging support for all of our new data structures. I was also responsible for maintaining the project repository, merging with ongoing development work, and managing the code review process. Phase 1 shipped as part of Solaris 11.1.

VM / IO interface restructuring   I refactored the interface between the Virtual Memory subsystem and the I/O subsystem, so that the changes to how physical memory is locked down for I/O done in VM2 Phase 2 would be behind an interface barrier. This was a 10k+ line project spanning a wide swatch of the Solaris Kernel, including rewriting four separate variants of the DMA setup code. The new interfaces made the resulting code both shorter and easier to understand, and integrated into Solaris 11.2.

Virtual Memory, Phase 2   I was the primary engineer for Phase 2 of the VM 2.0 project (the process of rewriting Solaris Virtual Memory from the ground up). Phase 2 covered re-implementing the "Intimate Shared Memory" used by databases for their main shared memory segment in terms of the new VM 2.0 data structures, some introduced by Phase 1, and the rest new to Phase 2. The work improved the anonymous memory initialization part of database startup time by ~5-10%, and turned the database's teardown of the memory segment from a tens-to-hundreds of seconds operation (scaling with segment size) into an almost constant-time millisecond cleanup. Phase 2 integrated into Solaris 11.3, and was a ~35,000 line project.

Async write acquire for kernel rwlocks   I extended the kernel readers/writer lock primitive (krwlock) to allow a thread to request asynchronous write acquisitions of normal kernel rwlocks. This required minor reworking of the main krwlock codepaths, as well as adding a new interface to the underlying priority-inheritance turnstile implementation. It was integrated as part of the Phase 2 project.

Virtual Memory Technical Lead   From early 2015-September 2017, I was Technical Lead for the VM2 team, overseeing the work of 5-7 engineers to extended and maintain the virtual memory system. This involved mentoring the junior group members, scoping projects, working with the engineers on design and implementation issues, coordinating work in progress with other groups who depended upon it, organizing bug triage, and presenting schedules and progress to the larger Solaris group. I also worked on various VM projects, including re-working the swap and reserved memory accounting to give better statistics to the new Solaris Analytics project.

Other wider duties I had included:

Crash Dump Analysis   I have a lot of experience analyzing Solaris crash dumps, and often helped the Solaris technical leads and other engineers analyze difficult crash dumps. I also maintained documentation for our debugging tools and contributed to internal mailing lists about them.

Patent Review Committee   From July 2013 – April 2018, I was on the Software & Operating Systems Patent Review Committee, which reviews potentially patentable inventions submitted by engineers throughout the Solaris, Oracle Labs, and Oracle Hardware divisions. It is responsible for evaluating and triaging the submitted ideas, and either rejecting them, requesting further information from the submitters, or green-lighting them to go through the full patenting process.

Change Review Team   From 2004 – 2006 and 2008 – 2017, I was on the Solaris "Change Review Team" (CRT), whose job was to evaluate the technical risk and merit of proposed changes to the Solaris source base, so that its level of quality is maintained. The CRT was the final gate between code review and the integration of a project or bugfix.

Worked as a senior engineer in the Solaris development organization, focusing on Solaris Kernel Development. In addition to resuming my Change Review Team and Crash Dump Analysis duties (see above), I worked on:

ZFS   Investigated and addressed performance issues in the Zettabyte File System (ZFS). Worked on analyzing checksum algorithm performance and implementation. Improved fault management information for ZFS checksum errors.

System Duty Cycle scheduling class   As part of my ZFS performance investigation, I worked on fixing issues with ZFS I/O processing threads (responsible for doing work like compressing/decompressing and checksumming data on its way to/from disks) preventing other threads from making progress. With one other engineer, I designed and implemented a new System scheduling class for kernel threads which use large amounts of CPU time. This scheduling class has since been applied to many other threads in the kernel.

Debugging support   I've done various work improving Solaris debugging support in the Solaris modular debugger (mdb), including making object identification pluggable, making searching kernel memory for a pointer efficient, summarizing thread stack traces, improving printing of enum values, etc.

I worked in a two-person startup designing and implementing "Coyotos", a high-reliability, high-security microkernel targeting the embedded space. Coyotos was a from-scratch successor to the EROS kernel I previously worked on at University of Pennsylvania. The kernel was written in C, the build tools and user programs in C and C++. The initial kernel rans on x86, and we did most of our development testing under qemu. I worked in the following areas:

Business   I collaborated on several SBIR (Small Business Innovative Research) Phase I proposals and two BAA (Broad Agency Announcement) proposals for government funding.

Build Tools   I modified the Interface Description Language (IDL) code generation tool to process and understand doxygen-style documentation comments, for automatically generated architecture documentation.

I wrote address space tree builders in the system image generation tool, for loading ELF binaries into the bootable image.

Kernel   I wrote most of the kernel's Object handling code, including the system image loader, the object hash (manages locating objects by ID), the capability preperation logic, and the generic address space walker. I also wrote the x86 page fault handler and page-table invalidation logic.

I assisted with the architecture, design, and implementation of the kernel's invocation path, which was used for all extra-Process communication, both to the kernel and to other processes. The IPC design went through several revisions.

User-land   I designed and implemented the bulk of the extant user-level support and basic utility programs in Coyotos, including the "Space Bank" (which provides all user-related objects and memory), the low-level C runtime initialization code, the "spawn"ing process, a copy-on-write address space manager, and the "elf" program address space manager.

By the end of my time at Sun, my projects and responsibilities included the Change Review Team and Crash Dump Analysis duties mentioned above, and:

Userland Filesystem Framework   My final project involved designing and implementing a userland-daemon backed filesystem framework based on the Solaris Doors IPC mechanism. I did initial prototyping and scoping of the work, but left before it was integrated.

OpenSolaris Sponsor   In addition to being on the Change Review Team (see above), I was also a OpenSolaris Sponsor, which meant I took contributions from the OpenSolaris community and worked to integrate them into the main Solaris release.

Product Training   I presented training presentations to groups of either Sun Employees or customers about specific subsystems of Solaris about once every month or two. These presentations mainly involved training on the design and use of the Solaris Management Framework and the Dynamic Tracing Framework (Dtrace). I've presented to groups as small as 10 and as large as 300 people.

My past projects and responsibilities included:

Completing libumem(3lib)   My first project after joining Sun was to "productize" the port of the kernel memory allocator (kmem) I'd done as an intern. libumem(3lib) became a major competitive feature for Solaris; its scalability provided a major advantage. Internally, hundreds of bugs have been found, tracked down, and fixed using libumem's debugging features.

Solaris Management Framework   One of the major projects for Solaris 10 was re-architecting the way software services are managed. The Solaris Management Framework (smf) was the end result; I designed and implemented the programmatic interface and underlying daemon for the smf repository, a transactional store of service configuration and state. The overall project was reported in the 2005 LISA conference.

Platinum Beta   The Solaris Platinum Beta program is for bleeding-edge customers who are willing to run Solaris beta bits in a production environment. I was a front-line engineer for three customers, and acted as their first point-of-contact for all Solaris issues.

Doors IPC Subsystem   I discovered a series of very serious security holes in a Solaris-specific IPC subsystem called Doors, did a pinpoint fix for the version of Solaris about to be released, and re-wrote it in the following Solaris release, closing the remaining holes and improving the code's maintainability.

Debugging Infrastructure   Solaris 10 introduced an in-situ kernel debugger, kmdb, which had substantial memory constraints. I re-worked many of the memory-related debugger commands to use substantially less memory, which allowed them to function in the kmdb environment, including our memory leak detector.

Tool Building and Support   I did several build and source code tool improvements (increasing build parallelism, improving the coding-style checker, improving the source-code searcher, etc.)

I interned in the Solaris Kernel Group. I investigated a syscall performance degredation. I ported the Solaris kernel memory allocator (including advanced debugging support and memory leak detection) to a user-space library which was reported in the 2001 USENIX proceedings. I fixed multi-threaded performance in the existing 'MT-hot' malloc implementation, making it scale linearly up to the number of available processors, and decreased its fragmentation issues.

I was responsible for software architecture and significant implementation of data analysis and trading system for the Options Market. I supervised two other software developers, and wrote a data analysis tool using the Statistical Analysis System to analyze the trade data and create Postscript pictures of the results.

I worked with Jonathan Shapiro on the "Extremely Reliable Operating System" (EROS), a capability-based research OS. I was the principal designer and implementor of the EROS "Space Bank", which was responsible for disk storage allocation, placement, and locality. EROS is a single-level store, so the Space Bank design had to consider application-level usage patterns, including heap, stack, and bulk data allocation. This work was reported in the 2002 Usenix proceedings. I also designed, implemented, and tested a component that manages a set of capabilities, and a line discipline implementation.

Filoli was a Palo Alto start-up which designed and implemented a NextStep-based paperless office system for a property-casualty insurance company. I began as a part-time employee and became a full-time employee in less than a year.

I initially functioned as a general troubleshooter, and did a wide variety of projects, including tool building, application programming, and setting up communications hardware and software. I subsequently designed and implemented tool to facilitate development in a daily-build environment which was used daily by more than 30 programmers. I designed and did initial implementation of integrated computerized telephony application. I designed and implemented an integrated check writing system which included security and journaling.

Jonathan Shapiro and Jonathan Adams
Design Evolution of the EROS Single-Level Store
Proceedings of the 2002 Usenix Conference

Jeff Bonwick and Jonathan Adams
Magazines and vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources
Proceedings of the 2001 Usenix Conference