Building Empathy

It’s hard to develop empathy for another person, team, or discipline when you haven’t “walked a mile in their shoes.” Unfortunately, this is quite common for developers, operations specialists, and security specialists—how many former security professionals are working in development or vice versa, for example? To compound the issue, when empathy isn’t present, people have a difficult time understanding each other, which can create a lot of friction in professional relationships. At the beginning of the DevOps transformation, there was a lot of friction that existed between the developer and operations teams that impeded their work, which compounded the delays caused by siloed workstreams. Much later, streamlined workflows and common understanding has greatly improved their working relationships; however, security teams have frequently been left out of the DevOps transformation. As a result, these relationships haven’t flourished as much as they could.

DevSecOps is a new iteration of a familiar initiative, this time including security in the SDLC as opposed to finding themselves inundated with requests after all the code is written and likely needing to be changed. This initiative will also help developers and operators as well—just like DevOps required developers to be op-aware and operators to be dev-aware, DevSecOps requires all three groups to be aware of each other’s interdependencies and needs.

The foundation of this transformation is all the people involved. To that end, when you are looking to shift to a DevSecOps culture, you’ll need to look to the people first and the processes that support them second. As such, this guide starts by covering how to support developers, operators, and security professionals working together in a way they have not before—by building relationships, empathy, and trust. Then, we will proceed to discuss the beginning knowledge for how to shift security left within the SDLC.

Walk A Mile: Shadowing#

At PagerDuty, we shadow extensively for training purposes. In the context of DevSecOps, having each of the teams shadow with each other will similarly allow them to develop both empathy and understanding for what the workflow is like for the other teams.

For security, shadowing can mean sitting in with the development and/or operations team for a sprint. This could happen on a rotation—so for every sprint, one person from the security team would sit in with development or operations. Make sure to configure this on a rotation that makes sense for both teams. In the event of an incident, the shadowee should continue to shadow for as long as it makes sense, as there is a lot to be learned when a team is in “crisis mode.” That being said, if the incident evolves in such a way that the shadowee’s subject matter expertise is needed, then they should resume that role.

Similarly, developers and operations specialists can shadow security to learn more about their workflow. The important caveat here is that, typically, security team members have more access globally than most other teams, so it’s important when shadowing to ensure that the shadowee is not granted increased access or permissions unless it is appropriate.

Full-Service Ownership#

Here at PagerDuty, we talk a lot about Full-Service Ownership (FSO), and relatedly another way that security teams can learn about the production process to actually own a Tier 1 or 2 (critical) service that makes business sense. For example, if your business or organization uses a tool to manage secrets, tokens, etc., it might make sense for the security team to own that service. Under the FSO model, the security team would own the revision, upgrade/update, and release cycle for that service, as well as the on-call rotation for that service. Similarly, having a dev or ops team own a security service or process that will provide insight into how security works as well as have the team responsible for some of the same end goals as the security team.

It is important to choose services in either scenario that is both feasible and makes good business sense. For example, development and/or operations should not own a service that would require an escalation in their privileges and security should not own a service that requires support that is outside of their expertise. Luckily, most architectures require multiple services so in these cases another, more prudent, service can and should be chosen.

When choosing a service, make sure to avoid partial or shared service ownership as this can cause confusion or create gaps in process. For example, if Service A is owned by Teams 1 & 2, how are upgrades performed? Is approval needed from both? What happens if an update has a feature needed by one team, but has backwards breaking changes for the other? If both teams own the service, then it will be difficult to have a consensus. As a last example, which team is responsible for handling incidents and issues related to the service? If only one team is required for any of these tasks, then the other team may be caught off guard or resent not being included in the discussion and/or decision. When there is one owner, then while they still need to consider the needs of everyone using the service, there is still one team that needs to make the final decisions as well as handle issues.

Security Champions Program#

Bottlenecks frequently cause friction between the development, operations, and security teams. In order to improve cross-team communication, these teams need a regularly scheduled time to discuss important issues and updates with each other. A Security Champions Program does just that. The program would be facilitated by members of the security team, but participants would come from development, operations, and any other teams that make sense. The meeting will allow the various participants to update each other about different messaging, changes, updates, and other topics that they want brought back to each department.

Similar to other programs or resource groups that you might run in your company, you will need buy-in from different stakeholders to set the program up for success. In this case, you’ll be looking at not only the agreement for a few key people in various departments who will hopefully attend the meetings, but also executives and management. You’ll also need a good understanding of the security culture at your organization to facilitate the meetings, even if you are not a security specialist.

For who should attend the meeting, start with your knowledge of your current culture. Use this to define what and who should be in the initial meetings to gain traction as well as show value to other people and teams that you’d like to include. In the meeting, you’ll want to make sure that there is adequate time for everyone to give the updates they need to give, ask the questions of each other that they need to ask. This can initially start with development, operations, and security but should eventually extend beyond these groups. For example, someone who works with customers directly would benefit from being able to handle some starter security questions before deferring to an expert on the security team itself.

Meet Needs to Gain Momentum#

Depending on the existing company structure and culture, it is possible that security is viewed as a “necessary hindrance”. This could be for a variety of reasons, but some are at least partly due to the issues you’re here to solve: breaking down silos that can cause slowness and breakdowns in communication. Of course, the reality is that security, like development and operations, is an incredibly interesting and complex field that protects customers and users as well as gaining and maintaining their trust.

If this is where you’re starting from, never fear! The first step here is to start to reverse this mindset. To begin: identify both the stakeholders and the teams that you’ll be working with the most. Start having meetings with people in these groups and look to identify their needs. The needs may not, at first, be immediately relevant to security but still overlap with your security team’s expertise. Have them work on the issues that are being raised by the other stakeholders and teams. This will start to build the human connections between the humans of security, operations, and development.

While this work is ongoing, the security team can start to seed some better practices as they move forward. The idea here is to add value, especially where there are gaps. There is also an opportunity to “make it easy to do the right thing”. For example, if security is helping improve workflows while also making them more secure. The team that is implementing that change should see the benefit to themselves, but the new security layer(s) in the workflow should not add complexity. Building up these use cases, the team should be able to align their work with some combination of the company’s mission statement, goals, and/or KPIs. The caveat to this is to use good judgement: the security team can and should be empowered to take action with any current security incidents or any high risk vulnerabilities that need to be urgently addressed.