Software Development Lifecycle (SDLC)

This document describes our Software Development Lifecycle (SDLC), which is how we ship software to production.

It provides guidance, and is not prescriptive—not every step will be relevant to every project. This is because not all projects are equally security-critical, and therefore not all projects should be treated equally. When in doubt about what's needed to ship your project, ask your manager. If you deviate, note the deviation and rationale somewhere that isn't Slack (e.g. a design doc, github issue, etc.)

[!IMPORTANT] Guiding Principle: Always Be Release Ready

• At any time, the latest develop should be shippable via our deployment pipeline. Build confidence in correctness and security continually throughout the development lifecycle, not only at the end.
• Build in vertical slices: split fixes or features into end-to-end increments that can ship independently.
• Separate feature deployment from activation using feature flags.

Process

flowchart TD
  Start[[ ]] --> Step0[Step 0: Ideation and Planning]

  subgraph P[Steps 0-3 can be partly parallelized]
    direction TB
    Step0 --> Step1[Step 1: Design, Specs, and Risk Modeling]
    Step1 --> Step2[Step 2: Implementation]
    Step2 --> Step3a["Step 3a: Create Superchain Ops tasks<br>(L1 upgrades only)"]
    Step2 -->|"If needed"| Step3b[Step 3b: Audit execution]
  end

  Step3a --> Step4a[Step 4a: Alphanet and Optional Betanets]
  Step3b --> Step4a

  Step4a --> Step4b[Step 4b: Final Betanet]
  Step4b --> Step5[Step 5: Testnet rollout]

  Step5 --> Step7[Step 7: Mainnet rollout]
  Step5 -->|"If needed"| Step6[Step 6: Governance proposal]
  Step6 --> Step7

Step 0: Ideation and Planning

Make sure you know, and document, the answers to:

• What is the problem we need to solve?
• What requirements and constraints do we have?
• Who exactly is the customer?

Step 1: Design, Specs, and Risk Modeling

Design Doc

Now that the problem, requirements, and customers are known, create a design doc that describes the solution:

1. Create a design doc PR in design-docs or design-docs-private (using the templates in those repos).
2. Share this with customers and design partners to get feedback. Iterate on the design doc until customers are happy with it.
3. Announce the design doc PR in #protocol-general. Tag managers from impacted teams so they can choose a representative from their team to review the design doc:
   1. Product: Sam McIngvale
   2. DevRel: Matthew Cruz (soyboy)
   3. Ecosystem: Faina Shalts
   4. Protocol: Matt Slipper
   5. Proofs: Paul Dowman
   6. Platforms: Alfonso Munoz de Laborde
   7. EVM Safety: Matt Solomon
4. Ask Lewej Whitelow or Aaron Levin to schedule a design review. Include the team representatives and any other stakeholders/customers on the invite.

Specs

Once the design is finalized, if it modifies smart contracts, consensus, or protocol functionality, write/update specs in the specs repo. Good specs will clearly document assumptions and invariants, see the AnchorStateRegistry specs as an example of well-written specs.

Determine governance impact

As you develop a design for your change, you’ll need to determine if the change requires governance. Changes that affect consensus, touch smart contracts on L1, modify predeploys, or impact transaction ordering will generally require governance approval. If you’re unsure, consult Ben Jones.

For full criteria and examples for determining if governance is needed, refer to the Governance Criteria and the Law of Chains user protections.

Risk Modeling

This will typically, but not always, be in the form of threat modeling. Reach out to the EVM Safety team for guidance or training on threat modeling.

Your initial threat model will inform engineering planning by helping answer the questions of what tests are needed, what edge cases to cover, what new or updated monitors are needed, what runbooks need to be written or modified, and audit needs. For more info on determining audit needs, see our audit framework and audits.md.

Once the initial threat modeling is done, extract all answers into issues (or wherever you are tracking project tasks) for tracking purposes.

Be sure to revisit and update your threat model as the project evolves, and as mitigations are implemented.

Step 2: Implementation

At this stage, you can start writing your code. Make sure you follow these standards:

• All consensus code must be behind a feature flag, decoupled from the hardfork name (see https://github.com/ethereum-optimism/design-docs/blob/main/protocol/decoupled-features.md).
• All changes must go through code review, and have test automation. Use coverage tooling and reports to identify testing gaps.
• For new features, add acceptance tests.
• For smart contracts the specs must clearly define assumptions and invariants as described above, and you must have ~100% test coverage.
• Include any changes to OPCM and VerifyOPCM.

Step 3a: Create Superchain Ops tasks (L1 upgrades only)

If your change modifies L1 smart contracts, you’ll need a superchain-ops playbook to execute the multisig transactions. Contact Blaine Malone for help with this.

Step 3b: Audit execution

• See audits.md for more information on how to get approval for and execute an audit.
• Make sure to leave sufficient leave time for scheduling the audit.
• Only start the audit once code, specs, and tests are complete.
• The audit must be completed, with fixes implemented, before Sepolia rollout.
• If there are high severity issues, do NOT proceed to the next step after fixing them—instead, perform a retro to see how those issues got in, and what else may have been missed as a result.

Step 4: Alphanet/Betanet devnet rollout

[!WARNING] Prerequisites for the final betanet All of the steps above MUST be completed before the final betanet. That includes:

• Specs
• Completion of risk modeling
• Implementation of mitigations identified by risk modeling (tests, monitors, runbooks, etc.)
• Governance impact analysis
• Audit execution and required fixes

Next, it’s time to roll out to the Alphanet, then the Betanet. See the release process and acceptance testing docs for more details.

You may deploy to multiple betanets if needed, for example to rehearse an upgrade process, but the final betanet must have no known issues and must result in no new issues being discovered.

Step 5: Sepolia rollout

Sepolia is a production network, therefore has the same standards and security requirements as mainnet. This is why the betanet in the prior step must have no known issues and must result in no new issues being discovered.

Coordinate with DevRel and external partners that may be affected. Sepolia must match mainnet as much as possible, including for partner configurations.

Step 6: Governance proposal (if governance is needed)

1. Prepare Proposal:
   • Reference a stable release candidate.
   • Include all relevant info such as risk modeling results, audits, Sepolia performance, and activation schedule.
   • Use the standard governance template.
2. Review & Post:
   • Obtain Foundation and Legal approval.
     • Loop in Ben Jones and Bobby Dresser from the Foundation.
     • Loop in Eric Van Wart from legal.
   • Post the proposal on governance forums.
3. Approval & Veto:
   • Wait for the vote and veto period to complete.

Step 7: Mainnet rollout

1. Remove the rc suffixes from your releases.
2. Schedule the mainnet upgrade after the veto period expires.
3. Coordinate with EVM Safety to schedule the mainnet multisig upgrade.
4. The Product/Dev/PMO Leads to go through their checklist to ensure all stakeholder needs, documentation, and communications are in place. This includes working with Marketing, DevRel, Data, Finance, Foundation, etc.
5. Monitor post-release with on-call coverage, then collect customer feedback on the overall process and track it for future process improvements.