Principal Engineer Roles Framework

Check the original article by Mai-Lan Tomsen Bukovec at https://www.linkedin.com/pulse/principal-engineer-roles-framework-mai-lan-tomsen-bukovec-142df/.

The following is a copy paste of the above, for my own records.

I have worked on Amazon S3 for ~12 years and if there is one thing that I have learned, it is that when you run complex systems at scale, you must think deeply about how teams work. It’s not enough to be get into the details about what you build. You have to spend lots of time engineering, iterating, and improving how you and your team operate.

Human potential is everywhere and if you are a leader, it is on you to find talent and elevate it. I have amazing engineering talent across all of my teams (object and file storage, streaming and messaging, and the new Amazon Q Developer migration for Windows, VMWare, and mainframe capabilities). I do whatever I can to maximize the impact of engineers in general and Principal Engineers in particular.

In Amazon, a Principal Engineer is a very senior engineer. Your Principal Engineers set direction on the evolution of your code, shape the culture of your engineering and operations, and materially improve your economic model and product roadmap. Principal Engineers share their learning in the AWS Builders Library, so if you haven’t checked it out, I highly recommend you do. Amazing engineers like Clare Liguori (Senior Principal), David Yanacek (Senior Principal), Colm MacCárthaigh (Distinguished Engineer), Marc Brooker (Distinguished Engineer), Becky Weiss (Distinguished Engineer), Joe Magerramov (Distinguished Engineer), and many others share their observations, learning, and recommendations built over years of working at scale with AWS services.

As an organizational leader, I routinely seek input from Principal Engineers like Andrew Warfield (Distinguished Engineer) on different dimensions of the business because it makes the team and me better. It is also a fact that the time of our Principal Engineers is highly constrained. And if you are a Principal Engineer, it can be hard to know how to optimize your time to have the most impact for the business. That is why I started working on the Principal Engineer Roles framework back in 2016.

How The Roles Framework Got Started

In 2016, I was lucky enough to have one of the original Amazon S3 engineers (and former Amazon CTO) Allan Vermeulen rejoin the S3 team for a few months to help us think through some tough problems. Al is one of the most brilliant engineers that I have worked with in my career. He’s incisive on technical problems and amazing at working with groups of engineers at any level. He challenges, inspires, teaches, and explores a frontier of possibilities. He is also a do-er. He will step in and write code or make a hard decision if the team is stuck. As I watched Al in action, I noticed that he often switched up how he engaged with the team depending on the state of the project. He moved between roles seemingly effortlessly once he became aware of what the team needed. Sometimes he asked another engineer to step into a role for him. It was fascinating to watch how quickly the dynamics shifted and velocity picked up in the engineering teams. It also helped accelerate the development of engineers. By watching Al and the Principal Engineers in these roles, engineers who had recently joined Amazon from other companies or from university saw with clarity what skills they would need to build. I wrote up my observations on the different roles that Al played or asked others to play. I reviewed these role descriptions with him and other Principal Engineers like Senior Principals Brad Marshall and Seth Markle and then introduced the roles as a framework to the Amazon S3 and Glacier Principal Engineer community.

The framework was pretty helpful for us as a team to optimize (or “force multiply” as we say in the Amazon Principal Tenets) the impact of our super awesome Principal Engineer community. We used the role framework to assess Principal Engineer engagement for our most important projects across Amazon S3/Glacier. In some cases, we changed the role of the Principal Engineer. For example, we observed that a Principal Engineer playing a Catcher role had been playing a Catcher role for multiple projects back-to-back for over a year. So, we trained a talented engineer on the team to play Catcher, and then switched the Principal Engineer to a Guide role on another project (we’ll talk more about what these roles do in a bit). In another case, a Principal Engineer was spending so much time as a Participant across different projects that they had no time to focus on a single project. So, we worked with the Principal Engineer to narrow their Participant scope to a smaller set of projects and free up time for to be a Sponsor. These discussions with Principal Engineers and managers also illustrated where we were missing a role. For example, we realized that the reason that we weren’t making progress on some ideas is because we didn’t have anyone operating as a Catalyst. We picked our most interesting incubation concept and asked a Principal to be a Catalyst. When that happened, we saw more progress in four months than we had seen in a year on developing our ambitious but ambiguous idea.

In 2018, I shared the model with a larger group of Amazon Principal Engineers at an internal offsite and wrote an internal wiki on the framework. Since then, Seth Markle and other Principal Engineers have refined the framework, which is now a resource for engineering communities in Amazon.

Principal Engineer Roles

Here are the different Principal Engineer roles in the framework:

Sponsor: A Sponsor is a project/program lead, spanning multiple teams. Yes, this role can be played by a manager but it does not have to be (at least not at Amazon). If you are a Sponsor, you have to make sure decisions are made and that people aren’t stuck in analysis paralysis. This doesn’t mean that you yourself make those decisions (that’s often a Tie-breaker’s role which you may or may not be here). But you have to drive making sure decisions get made, which can mean owning those decisions, escalating to the right people, or whatever it takes to get it done.

A Sponsor is constantly clearing obstacles and getting things moving. It is a time-consuming role. You shouldn’t have time to act as Guide or a Sponsor on more than two projects combined, and you don’t have to be a Sponsor every year. But if a few years go by, and you haven’t been a Sponsor, it might be time to think about where you can step in and play that role. It tends to build new skills because you have to operate in different dimensions to land the right outcomes for the project.

Guide: Guides tend to be domain experts that are deeply involved in the architecture of a project. Guide will often drive the design but they’re not “The Architect.” A Guide often works through others to produce the designs, and themselves produce exemplary artifacts, like design docs or bodies of code. The code produced by a Guide is usually illustrative of a broader pattern or solving a difficult problem that the rest of the team will often run with afterwards. The difference between a Guide and a Sponsor is that the Guide focuses on the technical path for the project, and the Sponsor owns all aspects of project delivery, including product definition and organizational alignment.

Guides influence teams. If you are influencing individuals, you’re likely being a mentor and not a Guide. A Guide is a time-consuming role. You shouldn’t have time to Guide more than two projects, and that drops to one project if you are a Sponsor at the same time.

Catalyst: A Catalyst gets an idea off the ground, and it’s not always their idea. In my experience, the idea might not even come from the Catalyst—it can be something we’ve been talking about doing for years but never really got off the ground. Catalysts will create docs or prototypes and drive discussions with senior decision makers to think through the concept. Catalysts are not just “idea factories.” They take the time to develop the concept, drive buy-in for the idea, and work with the larger leadership team to assign engineers to deliver the project.

A Catalyst is a time-consuming role because of all the work that needs to be done. At Amazon, that involves prototypes, docs and discussions. It is hard to effectively Catalyze more than one or two things at once. It is important to note that Catalysts, like Tie-breakers, are not permanent roles. Once a project is catalyzed (e.g., in engineering with a dedicated team working on the project), a Catalyst moves out of the role. The Catalyst might take on a Guide or Sponsor role on the project, or not. Not every project needs a Catalyst. A Catalyst is a very helpful (arguably critical) role for your most ambitious, complex, and/or ambiguous problems to solve in the organization.

Tie Breaker: A Tie-Breaker makes a decision after a debate. At Amazon, that means deeply understanding the different positions, weighing in with a choice, and then formally closing it out with an email or a doc to the larger group. Not every project needs a Tie-Breaker. But if your project gets stuck in a consensus-seeking mode without making progress on hard decisions, a senior engineer might have to step in as a Tie-Breaker. Tie-Breakers own breaking a log-jam on direction in the team by making a decision. Obviously, a Tie-Breaker has to have great judgment. But, it is incredibly important that the Tie-Breaker listens well and understands all the nuances to the different positions as part of breaking the tie. When a Tie-Breaker drives a choice, they must bring other engineers into their thought process so that all the engineers in the debate understand the “why” behind the choice even if some are disappointed by the direction. A Tie-Breaker must have strong engineering and organizational acumen in this role.

Sometimes an organization will depend on a small set of senior engineers to play the role of Tie-Breaker because they are so good at it. As a successful Tie-Breaker, you want to be careful not to set a tone that every decision, no matter how small, must go through you. You’ll quickly transition from Tie-Breaker to a “decision bottleneck” at that point—and that is not a role any team needs. If a team finds itself frequently seeking out a Tie-Breaker, it could be a sign that the team needs help understanding how to make decisions. That’s a topic for a different time. The Tie-Breaker role is considered a “moment in time” role, versus Sponsor/Guide which are ongoing until you reach a milestone. Once the decision is made and closed out, you’re no longer the Tie-Breaker.

Catcher: A Catcher gets a project back on track, often from a technical perspective. It requires high judgement because a Catcher drives prioritization and formulating a pragmatic plan under tight deadlines. Catchers must quickly do their own detailed analysis to understand the nuances of the problem and come up with the path forward in the right timeframe. As a comparison, a Tie-breaker tends to step in when the pros/cons of the different approaches are well known and the team needs to make a hard decision. Once “caught” (i.e., the project is back on track and moving forward), a project doesn’t need the Catcher anymore.

Sometimes Principal Engineers can do too much catching. Don’t get me wrong, we are all Catchers sometimes—including me. Any fast-paced business needs Catchers in engineering and management. It teaches important skills about leadership in difficult moments and helps the business by landing deliverables. It also teaches you what not to do next time. However, it is better to generalize a Catcher skill set across more engineers and not depend on a small set of Principal Engineers to be Catchers. If a Principal Engineer plays Catcher all the time through a succession of projects, it leaves no time to develop skills in other roles.

Participant: A participant works on something without one of these explicitly assigned leadership roles. A Participant can be active or passive. Active participants are hands-on, and do things like spend a few days working through a design discussion or picking up a coding task occasionally on a project, etc. Passive participants offer up a few points in a meeting and move on. In general, if you’re going to participate it’s better to do so actively. Time-boxing some passive participation (e.g., office hours for engineers) can be a useful mechanism to stay connected to the team. However, keep in mind that it is easy for your time to get consumed by being a Participant in too many things.

How to Use the Roles Framework

The Roles Framework is an engagement model. It helps Principal Engineers intentionally plan how they work with teams to have the most impact. It is not a job description. So how do you do use this framework? The first thing is to assess what roles you play today. If you are a Principal Engineer, I recommend assigning a color to time blocks in your calendar for a particular role, like blue for Catcher and red for Participant. Look back across four to six weeks of work when you are color coding. Be thorough and honest with what role you ended up playing (versus want to play) in a meeting or work block. It will give you a sense of what roles you tend to fall into. Then, you can review with other Principal Engineers and managers to see if you are providing the most impact to the team in those roles, or if you should make some adjustments.

The framework provides a shared vocabulary between Principal Engineers, other engineers in the organization, and managers. For example, if you are a Principal Engineer and you get pulled into too many things as a Participant, you can share the framework with your teams and say you’d like to work in other engineers as Participants so you can move into a different role like a Guide for a specific project. The framework also helps communicate expectations to talented engineers to step into stretch roles. For example, a high potential engineer can be brought into a Catcher role as a learning experience. (There is nothing quite like playing Catcher to see what to avoid in your next project.) That might also free up your Principal Engineers will have more time for other roles.

Interpreting Signals About Roles

Often, there are signals from the team that will tell you how you need to adjust roles. For example, if you are a Principal Engineer, you might be in a meeting acting like a Participant, but the team keeps asking you to help them make a decision. In that case, what the team needs from you is to be a Tie-Breaker.

If you are an organizational leader, you can analyze what roles your Principal Engineers play across your projects and think about the signal it sends. Are all your Principal Engineers tied up in Catcher roles? If so, is it a moment in time (e.g., just this month) or a common pattern across the year? If your Principal Engineers are always playing Catcher, that is a signal that something is off in how your engineering projects execute. And those recurrent Catchers won’t have time to be Guides on critical projects or a Catalyst on the team’s next big idea.

Getting Started with the Roles Framework

If you are a leader that wants to pilot this engagement model, sit down with your Principal Engineers and do a full inventory of technical projects and/or decisions in the next six to eight months. Then, identify which of these projects need specific roles. Not all projects need all roles at all times. For example, every project doesn’t need a Sponsor, only the complex ones do. Sponsors aren’t always Principal Engineers either. They can be a talented engineer that has a natural talent for one of the roles or a software development manager. Don’t opt for “coverage” and auto-assign all your Principal Engineers to be a Sponsor. If you do that, your Principal Engineers won’t have time to do much else. The evolution of a project also impacts roles. For example, a new idea might need a Catalyst for a while and then move into needing a Guide.

The Anti-Entropy Effect of the Roles Framework

Entropy is a part of large-scale distributed systems. We see this with hard drives in massive storage fleets like Amazon S3. Drives fail and corrupt themselves, and our distributed systems need to counteract it. The same is true with our time and focus. Without intentional thought and effort, it is easy to any of us to devolve into Catchers and Participants in a fast-paced environment. That’s why I think frameworks like this are so useful. They give you a way to be intentional about your time and contribution.

If you are intentional about the role that a Principal Engineer plays on your team, you can elevate the voice and the talent of the Principal Engineer. It also helps other engineers by role-modeling situational leadership. And that will have long term benefits to the engineering community, the service you build and the customers who use it.