The Evolving Landscape of CI/CD: From Skype to AI and Beyond

Continuous Integration and Continuous Delivery (CI/CD) remain some of the most challenging yet crucial aspects of modern software engineering. As the industry evolves, driven by new technologies like Kubernetes and AI, the practices and philosophies surrounding CI/CD are also shifting. This article delves into the journey of CI/CD, exploring its historical roots, the impact of new platforms, the rise of GitOps, the nuances of progressive delivery, and the future implications of AI.

From Weekly Releases to Continuous Delivery: A Skype Story

The journey into the world of CI/CD often begins with early experiences that highlight the limitations of traditional release processes. For Rob Erez, a CI/CD expert with over a decade of experience, this journey started at Skype in the early 2010s. At the time, deployments were a weekly affair, requiring sign-off from a Change Advisory Board (CAB). This process felt cumbersome for a web-based application running on Azure, where the team had the capability to deploy at any time.

"We'd make these changes through the week, but we'd kind of have to hold them back," Rob recalls. The team found a way to work around the system, building and shipping code as it became ready. This involved a robust process of committing code, running multiple layers of testing, and deploying to staging before production. This self-managed process was, in essence, a form of continuous delivery.

A particularly memorable aspect of their process was the use of "canary deployments," where new releases were rolled out to a small percentage of their customer base. New Zealand, being the first country to reach a new day and an English-speaking demographic, served as their consistent canary. This allowed them to test new features in a live environment with minimal risk, a practice that opened Rob's eyes to the power of progressive delivery.

The Rise of Octopus Deploy and the CI/CD Spectrum

After his time at Skype, Rob joined Octopus Deploy, a company specializing in deployment automation. This move solidified his deep involvement in the CI/CD space. He explains the distinction between Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment:

While Continuous Deployment offers significant benefits, Rob emphasizes that it's not a one-size-fits-all solution. Industries with stringent regulations and compliance requirements may need to retain manual gates for production releases. However, achieving Continuous Delivery, where changes are always deployable, is a valuable goal for most teams, as it significantly mitigates risk.

Kubernetes: The Cloud-Native Unifier, On-Premises Powerhouse

The advent of Kubernetes has dramatically reshaped the infrastructure landscape. Originally developed by Google, Kubernetes emerged as a powerful container orchestration platform. Its open-source nature and adoption by major cloud providers like AWS and Azure helped level the playing field, making it easier for organizations to move workloads between different cloud environments.

"Kubernetes came along at the time when there was a bunch of plays in the field for container orchestration," Rob notes. While other solutions existed, Kubernetes eventually emerged as the dominant force due to its robust features and broad ecosystem support.

Interestingly, despite being labeled "cloud-native," Kubernetes has found significant traction in on-premises deployments. Many organizations leverage Kubernetes to manage their own data centers or even run clusters on research vessels at sea. This allows them to benefit from Kubernetes' declarative configuration and self-healing capabilities while maintaining greater control over their infrastructure. The ability to define a desired state and have Kubernetes continuously reconcile reality with that state simplifies complex infrastructure management.

GitOps: Declarative Infrastructure and Version Control

GitOps has emerged as a prominent trend, particularly in conjunction with Kubernetes. At its core, GitOps is a methodology for managing infrastructure and application deployments using Git as the single source of truth. The key pillars of GitOps, as defined by Weaveworks, are:

  1. Declarative: The desired state of the system is declared.
  2. Versioned and Immutable: The desired state is stored in a version-controlled, immutable repository.
  3. Pull vs. Push: GitOps agents pull the desired state from the repository and apply it to the cluster.
  4. Continuous Reconciliation: GitOps agents continuously ensure that the actual state matches the desired state.

While the name "GitOps" suggests an absolute reliance on Git, Rob points out that the core principles don't strictly require it. The emphasis is on declarative state, versioning, and continuous reconciliation. The challenge arises when trying to apply GitOps dogma to all aspects of infrastructure, such as managing secrets, which are often better handled outside of a Git repository. The trend is towards adopting GitOps principles for managing infrastructure, especially within Kubernetes, as it provides a structured and auditable way to manage deployments.

Progressive Delivery: De-risking Releases with Feature Toggles

Progressive delivery represents an evolution beyond continuous delivery, focusing on releasing changes in a more controlled and gradual manner. This approach aims to minimize the impact of potential issues by releasing features to a subset of users before a full rollout. Common strategies include:

Rob advocates strongly for feature toggles, highlighting their advantages in terms of granular control, rapid rollback capabilities (often in seconds), and the ability to decouple deployment from release. This is particularly relevant in an era where AI-generated code might increase the velocity of deployments, making risk mitigation even more critical.

Rollbacks: The "Roll Forward" Philosophy

The concept of "rollback" is often discussed in CI/CD, but Rob argues for a "roll forward" philosophy. While stateless systems can often be rolled back by reverting to a previous version, stateful systems, particularly those involving database schema changes, present significant challenges. Attempting a rollback with schema drift can lead to complex inconsistencies.

Instead, Rob advises that if a bug is discovered, the focus should be on creating a hotfix and rolling forward to a new, corrected version. This approach, combined with the rapid rollback capabilities of feature toggles, provides a more robust strategy for managing issues.

The Rise of Platform Teams and Evolving Development Environments

The increasing complexity of software development and deployment has led to the rise of platform teams. These teams aim to provide standardized, self-service mechanisms for application teams, abstracting away much of the underlying infrastructure complexity. This allows application developers to focus on writing code rather than managing deployment scripts and infrastructure configurations.

The evolution of development environments also continues. While traditional Dev, Test, and Prod environments remain common, the concept of ephemeral environments is gaining traction. These temporary environments are spun up for specific features or pull requests, allowing developers to test and validate their work in isolation before merging. This accelerates feedback loops and improves the quality of code before it reaches more formal testing stages.

AI's Impact on CI/CD: Velocity and Risk Mitigation

The integration of AI into the software development lifecycle is undeniable. While the full impact on CI/CD is still unfolding, it's clear that AI will drive increased velocity in code generation and potentially influence how pipelines are managed.

Rob anticipates a shift in focus from the speed of the pipeline itself to risk mitigation. With AI generating more code, the emphasis will be on ensuring the quality and safety of these releases. This will likely lead to increased adoption of progressive delivery techniques, especially feature toggles, to manage the rollout of AI-generated features and to quickly disable them if issues arise.

The Enduring Importance of On-Premises and Hybrid Offerings

Despite the prevailing narrative of "everything in the cloud," Rob highlights the continued relevance of on-premises and hybrid solutions. Many organizations, particularly in finance, government, and regulated industries, require on-premises deployments for control, compliance, and security reasons.

Octopus Deploy's dual strategy of offering both SaaS and on-premises solutions exemplifies this reality. Managing both presents unique engineering challenges, especially concerning upgrade paths and backward compatibility for long-standing on-premises installations. However, the demand for these solutions remains strong, underscoring that a one-size-fits-all cloud-only approach is not universally applicable.

Key Takeaways