Transforming DevOps: The Shift from CI/CD Pipelines to Platform Engineering

“DevOps is a combination of software development (dev) and operations (ops). It is defined as a software engineering methodology which aims to integrate the work of development teams and operations teams by facilitating a culture of collaboration and shared responsibility.” (Gitlab)

Before DevOps, development and operations were separate disciplines. Developers have built the application, packaged it, and handed it over to the operations team. The operations team then installed the applications on the servers and maintained them. This separation led to inefficiencies, misunderstandings, and a slower pace of delivery.

Under a DevOps model, development and operations teams are no longer siloed. Each team is responsible for one or more services and contains people with different skills, such as development and operations skills.

DevOps helps a team handle the whole application lifecycle, including planning, developing, building, releasing, deploying, operating, monitoring, and testing.

Due to this model, the benefits of DevOps include:

• Faster innovation for customers and quicker adaptation to the market

• Increased reliability of software products

• Improved collaboration between teams

• Enhanced security

As organizations grow and adopt more complex architectures, the limitations of traditional DevOps practices become apparent. Managing microservices, multi-cloud environments, and diverse toolchains can overwhelm developers and lead to multiple problems:

• Complexity and Cognitive Overload:

  Too many tools and operational complexities lead to information overload.

• Toolchain fragmentation and inconsistent environments: Each team often builds its own pipelines and integrates its own tools, leading to inconsistencies.

• Developer inefficiencies and roadblocks: Developers face delays in accessing the right systems or waiting for required infrastructure or tools.

“By 2026, 80% of large software engineering organizations will establish platform engineering teams as internal providers of reusable services, components and tools for application delivery — up from 45% in 2022.” (Gartner)

Platform engineering is an emerging trend intended to modernize enterprise software delivery, particularly for digital transformation. A dedicated product team creates and maintains an Internal Developer Platform (IDP), which combines various technologies and tools to tackle the complexity of cloud-native applications and to benefit engineers in large digital organizations. Additionally, the growing complexity of software development is fueling a market of tools for improving developer productivity and experience. Tools and projects like Spotify's Backstage are enhancing developer productivity and experience, attracting significant interest and investment.

An Internal Developer Platform (IDP) is created by a platform team to build golden paths, enable developer self-service, and simplify infrastructure complexities. The platform team treats the platform as a product, with internal users e.g. developers as their customers, ensuring an exceptional user experience. They provide the platform's capabilities based on user needs and provide and curate tools, toolchains, and SaaS services. Product and application teams can use these capabilities and components through various interfaces like developer portals, APIs, and CLIs.

One key success factor for an internal developer platform is treating the platform as a product. The platform should be designed to meet its users' needs and evolve based on those needs. It should focus on providing capabilities that support common tasks for all teams, rather than specific ones for just one team, to deliver the most value. While every platform can have different capabilities, the CNCF has defined a list to consider when building platforms for cloud-native computing e.g.

• Web portals and APIs for observing and automatically provisioning products and capabilities

• “Golden path” templates and docs enabling optimal use of capabilities in products

• Automation for building and delivering services and products

• Observability for services and products using instrumentation and dashboards, including observation of functionality, performance and costs

• Infrastructure services including compute runtimes, programmable networks, and block and volume storage

• Security services including static analysis of code and artifacts, runtime analysis, and policy enforcement

CNCF also provides a table to help readers grasp each capability by loosely relating it to existing CNCF or CDF projects (below the capabilities)

While the primary users of an internal developer platform are developers, it is also valuable for project managers, QA teams, and IT support staff. The platform provides comprehensive insights into development processes and resources for all stakeholders.

The Cloud Native Computing Foundation defines a Golden Path as a "templated composition of well-integrated code and capabilities for rapid project development." Simply put, it is a self-service template for common tasks like:

• A crossplane CRD to create an infrastructure resource.

• A skeleton source code for a Java Spring Boot application.

• A CI/CD pipeline template

• Documentation or getting started guide

• Policy guardrails

E.g. A getting started guide helps new users onboard quickly by providing forkable repo templates, so they don't have to start from scratch.

Golden paths benefit not only developers but also other platform users e.g. Security admins can integrate best practices like security and compliance into templates, which developers then use automatically. Streamlined infrastructure and components also make monitoring easier for SREs.

A self-service platform helps its users remove frictions (like supporting infrastructure and build CI/CD) from the teams, so they can focus on the customer-facing product and bring more customer value.

According to VSM Consortium a platform is an investment in scale, standardization, and self-service, primarily serving internal platform users but also the customer. The platform helps its users to work more efficiently and effectively, making it a strong supporting value stream. They divide the values into two distinct types:

• Technical Value: It reduces cognitive overload, automates workflows, standardizes compliance policies, and improves the overall experience.

• Business Value: It allows teams to focus on delivering value to end customers by using self-service capabilities that eliminate dependencies and speed up software delivery.

If platform teams are aligned with a value stream to deliver products, services, or experiences to internal customers, we can define two types of product teams:

• Customer-Facing Application Teams (ASD): These teams focus on delivering value directly to end customers by developing and enhancing products and services.

• Platform Teams (SDP): These teams concentrate on building and maintaining the internal developer platform, which supports the customer-facing teams by providing essential tools, infrastructure, and automation to streamline their workflows and reduce cognitive load.

Important is to always focus on the platform’s customer outcome as goal and not on the engineering itself. For this, value streams can help keep the focus on flow and value realization.

An internal developer portal (IDP) is not to be confused with an Internal Developer Platform (also IDP). It is just a subset of the platform. An Internal Developer Portal serves as the user interface or the 'front-end' of the Internal Developer Platform. It allows developers to interact with the toolkit that the IDP provides via UI, APIs, CLIs, or even ChatOps. Popular examples of these portals are Spotify’s Backstage and Port.

The pillars of an Internal Developer Portal are:

• Software Catalog: A central place where developers can find information about all software and resources, like microservices, cloud resources, CI/CD data, etc. It helps developers see everything in one view (single pane of glass) and understand how they are connected.

• Self-Service: Developers can self-serve resources and environments, reducing reliance on IT or DevOps teams. This empowers developers to work independently and speeds up development.

• Workflow Automation: Enables operations to run workflows against the software catalog.

• Scorecards: These are Benchmarks for items in the software catalog, measuring quality, production readiness, and developer productivity. They set engineering standards and guide teams on improvements. Scorecards can also trigger alerts or be checked by CI/CD pipelines to prevent deployments if standards aren't met.

User experience is crucial for developers to use the portal effectively. The portal should feel like a well-designed product. It should offer flexibility, allowing users to self-serve resources through various interfaces like APIs, UI, or even ChatOps with AI.

Platform engineering is the next step in delivering software more reliably and efficiently, addressing the limitations of DevOps in complex enterprise structures. Internal developer platforms use self-service and golden paths to reduce developers' cognitive load, adding value for both end customers and internal users. As platform engineering grows and more companies establish platform engineering teams, it's crucial to keep the focus on meeting the needs of the platform's users for success.