Introduction to CI/CD
(Based on GitLab’s and RedHat’s explanations)
Before moving on to the topic of GitHub Actions, we first need to understand a more general concept. Continuous Integration and Continuous Delivery (CI/CD) has become an essential practice in modern software development. It is a set of processes and principles that facilitate the development and deployment of software by automating repetitive tasks (e.g., building, testing and deploying), improving collaboration and ensuring the reliability and stability of the software.
Figure1: CI/CD visualized (Source: https://www.redhat.com/en/topics/devops/what-is-ci-cd)
Continuous Integration
The concept of CI/CD is driven by the motivation to minimize the time and effort required to deliver quality software while also maximizing the efficiency and productivity of development teams.
In this sense, CI involves regularly integrating code changes from multiple software engineers into the main branch of a shared code repository and then automatically building, testing and validating the code to ensure that it is working correctly. Frequent code integration not only reduces the possibility of having merge conflicts between developers, but can also help quickly identifying and fixing any issues and bugs with the help of automated tests (typically unit and integration tests). By ensuring that code changes are regularly integrated and tested, CI can help deliver high-quality software more quickly and efficiently.
Continuous Delivery
CD is most often referred to as Continuous Delivery. This practice involves uploading the tested changes to a repository such as GitHub or a container registry, and continuously delivering software changes to production or production-like environments (e.g., Staging, Production), where they can be tested and validated in real-world conditions. In this sense, Continuous Delivery automates the release of validated code to a repository with the goal of having always a codebase available for the deployment of production environment. The operations team (or the person responsible for deployments in small teams) is then able to quickly deploy new versions of the application to production.
Continuous Deployment
CD, on the other hand, is sometimes also referred to as Continuous Deployment, which can be viewed as an extension of Continuous Delivery and represents the final stage of a mature CI/CD pipeline. As the name implies, Continuous Deployment, as opposed to Continuous Delivery, is about automatically deploying software changes to production environments once they have passed all the required testing and validation steps defined in the pipeline. It requires a high degree of automation and testing, and cannot be done without having a proper CI in place. This means that software changes could go live within minutes, considering all tests defined in the pipeline passed. Therefore, Continuous Deployment makes it easier to continuously receive and incorporate feedback from users.
Github Actions
Github Actions is a powerful CI/CD solution provided by GitHub that allows developers to automate their workflows directly within their repositories. With GitHub Actions, it is possible to build, test and deploy the code right from GitHub, making it easy and convenient to ensure the quality and reliability of software.
In the following, the most important components and concepts are briefly addressed and explained. If you want to delve into specific topics in detail, the official GitHub Actions documentation is the perfect reference.
Components
Github Actions uses a workflow-based approach, where a workflow is a configurable, automated process triggered by specific events and consists of one or more jobs. Each job is executed within a virtual machine runner or container and contains a series of steps, where these steps represent the individual tasks to be executed. A sample workflow is visualized in Figure 2.
Figure 2: Sample Workflow visualized
Workflows
As stated earlier, a workflow is a configurable automated process that can run one or more jobs. They are defined using YAML files, which are then placed inside the .github/workflows directory of a repository. You can set up multiple workflows for building the code, testing the code against the current test suite, checking the formatting, deploying the application, and much more.
Events
Workflows are triggered by events. This can be events concerning pull requests (e.g., open, update or merge a pull request), opening an issue, or pushing a commit to a repository. You can specify the events that trigger a workflow in the YAML file using the on
keyword. Listing 1 shows an example of how such events can be defined. Specifically, this sample workflow is triggered whenever something is pushed to the main branch or when a pull request targeting the main branch is (re)opened or updated. An extensive list of possible events can be found here.
Listing 1: Sample events
Jobs
Jobs are a collection of steps that run sequentially on a specific runner. Jobs can run in parallel (which is the default setting) or depend on the success of previous jobs. This dependency can be established with the needs
keyword which ensures that a job is not executed until the depending job has successfully finished. Listing 2 gives an example of how a job can be defined and a dependency to another one created. More information about jobs can be found here.
Listing 2: Sample jobs
Steps
Steps are individual tasks within a job. Each step can either run a shell command or execute an action, which is a reusable piece of code that can be created by anyone. Actions usually perform complex but frequently repeated tasks and are used to reduce code duplication and the complexity of workflow files (more information about actions can be found here). It is important to highlight that steps are executed sequentially, which means that they are dependent on each other. Since they are all executed on the same runner, they can share data from one step to another. Listing 3 illustrates a job that performs linting on a project. The first step checks out the source code from the repository with the help of an already defined action. The second step then runs the linting process by installing the dependencies and executing the linting command.
Listing 3: Complete linting job
Runners
Runners are the environments where jobs execute. Each runner can run a single job at a time. GitHub provides hosted runners for different operating systems (e.g., Ubuntu Linux, Microsoft Windows or macOS) with each workflow running on a newly provisioned virtual machine (VM). There is also the possibility to set up self-hosted runners for greater control and customization. Listing 4 shows how to configure the previously defined job to run on the latest version of an Ubuntu Linux runner. In this sense, when the workflow is triggered the job with the jobid **_lint** will be executed on a new VM which is hosted by GitHub.
Listing 4: Configure runner
Contexts
Contexts are a set of predefined objects that provide information about the environment and the current state of the workflow. They contain meta-data and data that can be used in expressions within a workflow. There are many contexts available, such as the github
, job
, or runner
context. The data within a context can be accessed using expression syntax, i.e., ${{ <context> }}
. In the following, the env
and secrets
context is introduced. Further information about other contexts can be found here.
Environment variables
Environment variables are used to store configuration information and other data that can change depending on the environment in which the workflow runs. These variables can be accessed within the workflow steps and are helpful for customizing the behavior of scripts and actions. Environment variables can be defined at different levels within a workflow:
- Workflow-level: To define environment variables that are available to all jobs in a workflow, the
env
keyword can be used at the top level of the YAML file:
Listing 5: Workflow-level env variables
- Job-level: To define environment variables that are available only to specific jobs, the
env
keyword can be used within the respective job:
Listing 6: Job-level env variables
- Step-level: To define environment variables that are available only to a specific step, the
env
keyword can be used within the respective step:
Listing 7: Step-level env variables
Secrets
Secrets are used to store sensitive data, such as API keys, passwords or tokens, that you don’t want to expose in your repository or logs. They are encrypted and can be accessed only with the GitHub Actions workflow of the same repository. Secrets can be created by visiting the GitHub repository, clicking on “Settings” and then on the “Secrets” tab, where you can eventually add new secrets, provided you have the necessary permissions. Secrets can be accessed with the help of the secrets
context, as shown in Listing 8. This example makes use of the labeler action, which requires the GITHUB_TOKEN as the value for the repo-token
input parameter.
Listing 8: Access secrets context
If you use a secrets management platform like Doppler, you can easily integrate it with your GitHub repository, which then allows you to synchronize the secrets with GitHub Secrets. The beauty of this is that you can easily access these secrets in the same way as shown earlier in Listing 8.
Containerized services
Service containers are Docker containers that are used to provide additional services or dependencies required by the application during the execution of a workflow. They allow accessing the services they provide without having to install and configure them directly within the runner. They are particularly useful when the application relies on services like databases or memory caching, as it is the case for the Klicker app. For instance, if we want to run End-to-End tests, the workflow will require access to a database and a memory cache. By using these service containers, we can set up the necessary services for a successful execution of the workflow. This can be done with the help of the services
keyword as shown in Listing 9.
In this example, two service containers, PostgreSQL service container (i.e., database) and Redis service container (i.e., cache), are created using the official images from Docker Hub. In the case of the PostgreSQL service container, additional environment variables for the username, password and database are provided. The options
keyword is used for both service containers to define health check commands and further settings, to ensure that both services are ready before proceeding with the workflow.
As mentioned earlier, jobs can be executed on Runners that are hosted by GitHub (i.e., provisioned VMs) or on self-hosted runners where the jobs are executed in containers. Depending on the selection of the runner (VM or container), the communication between jobs and its service containers is different.
- Running jobs and services in containers makes network access easier. You don’t need to configure any ports because containers in the same Docker network automatically expose all ports to each other, but not outside the network. If you want to understand this magic better, feel free to delve into the details here.
- When running jobs directly on the runner machine, as it is the case in Listing 9, access to service containers is provided by using
localhost:<port>
or127.0.0.1:<port>
. GitHub configures the container network to allow the service containers to communicate with the Docker host (i.e., the VM that is running the Docker containers). We also learned earlier that Docker containers do not automatically expose their ports to the job on the runner. This means that we need to map ports on the service container to the Docker host to enable communication between the job and the containerized service. This can be observed in Listing 9 by theports
keyword. In this example, theports
keyword maps the port 6379 on the Redis container to port 6380 on the Docker host. This means that the job running on the runner machine can access the Redis service onlocalhost:6380
or127.0.0.1:6380
since any request to port 6380 is forwarded to the Redis container running on port 6379. Click here for more information.
Listing 9: Creating containerized services
Caching
GitHub Actions provide a caching feature that allows you to store and reuse data (e.g, dependencies or build artifacts) across multiple workflow runs. By caching data, the performance of workflows can be significantly improved and build times can be reduced, which in turn allows for faster feedback. This leads to reduced costs, which is all the more important considering that public repositories have 2000 minutes of compute time per month included.
The built-in caching mechanism of Github Actions uses the actions/cache
action, which enables you to cache data between workflow runs, based on a key that you specify. The cache is scoped to the repository and branch, ensuring that each cache is unique and isolated. Two common use-cases for caching are:
- Dependency caching: Caching dependencies downloaded or installed by package managers, such as npm or pip, can significantly reduce the time it takes to set up your environment and run, for example, your tests.
- Build artifact caching: If the build process generates intermediate artifacts that take a long time to create, you can cache those artifacts and reuse them in subsequent workflow runs and therefore reducing the overall build times.
Listing 10 shows an example of how caching could be implemented. After checking out the source code from the repository into the runner’s workspace, the next step, which uses the action/cache@v3
action, is triggered. It caches the npm dependencies and the build artifacts generated by the Next.js framework. The paths to the data we want to cache are specified by the path
key and include the global npm cache as well as the build caches for the three Next.js applications. Furthermore, the cache key
is unique and, in this case, generated based on the runner’s operating system, a fixed string “nextjs” and the hashes of package-lock.json and all .js, .jsx, .ts, and .tsx files in the repository. This ensures that a new cache is created whenever dependencies or source files change. An optional restore-keys
parameter is also used to specify a fallback cache in case an exact match (i.e., cache hit) for the cache key isn’t found (i.e., cache miss). In the case of a cache miss, the cache action sequentially searches for any caches that exactly match the restore-keys. If there are still no exact matches, it then looks for partial matches of the restore keys. If there is finally a match, the most recent cache is then restored, otherwise a completely new cache is created.
Listing 10: Caching dependencies and artifacts
Workflow Artifacts
Workflow artifacts are files or collection of files generated during the execution of a workflow. They are then stored, associated with a specific workflow run, and made available for download. These artifacts can include a variety of files, such as binaries, log files, test results, code coverage results, or any other output resulting from the build process. Storing artifacts is useful for various purposes, including sharing or auditing build process, deployment, further analysis, or simply for troubleshooting.
Other than build artifact caching, which is used to temporarily store intermediate files to improve build times (see section Caching), workflow artifacts are meant to save the final output of the workflow (e.g., build process, testing results) for later access, analysis, or deployment.
Upload artifacts
To upload and store artifacts, the actions/upload-artifact
action can be used within the workflow as illustrated in Listing 11. After installing, building, and testing the project, this action archives the produced artifacts by specifying the name of the artifacts with the name
keyword and also the directory path with the path
keyword. A custom retention period with the help of retention-days
can be defined as well (see Archive code coverage results step). However, this value cannot exceed the retention limit set by the repository or organization.
Listing 11: Upload build and test artifact
Download artifacts
To download artifacts, the actions/download-artifact
can be used. However, this action only allows downloading artifacts in subsequent jobs or steps within the same workflow run. Listing 12 shows how a previously uploaded artifact called my-artifact could be downloaded. If a name has not been defined for the download, the default name would be artifact. Leaving out the name
parameter results in downoading all artifacts that have been uploaded during the same workflow. Other than that, all uploaded artifacts can be downloaded outside a workflow with the GitHub UI for further analysis, for example.
Listing 12: Download single artifact
Job Matrices
Matrices provide a nice way to run a job with multiple combinations of environment variables, operating systems, or any other configurations. Such matrix configurations help you execute the same job with different parameters, enabling efficient testing across multiple environments, platforms, or configurations with minimal code duplication.
To define a matrix strategy, you need to use the matrix
keyword under the strategy
section of a job. This allows to specify the variation of the configurations you want to use in the matrix. Listing 13 shows an example of a matrix that runs a job on three different versions of Node.js across two different operating systems. The job test will run on the latest versions of Ubuntu and Windows, and it will use Node.js versions 14, 16, and 18. In this sense, the matrix creates a total of 6 combinations (2 operating systems * 3 Node.js versions), with the specified job running for each of them. The matrix variables, such as ${{ matrix.os }}
and ${{ matrix.node }}
, are used to reference current combination’s configuration when defining steps in the job.
Listing 13: Using a matrix strategy
There are even more configurations that can be applied to the matrices, such as customizing the maximum number of concurrent jobs or adding, expanding, or excluding configurations. To explore these kind of possibilities you can visit this page.
Generic Examples
In this section, you can find some initial examples of GitHub Action workflows that demonstrate many of the concepts covered on this site. Additional examples will be added over time to cover a wider range of common workflows.
Lint and publish a NPM package
Example 1 is triggered by a push
event to the repository and demonstrate best practices for building, testing and publishing a NPM package. A possible suggestion for improvement could be caching the npm dependencies to avoid re-downloading dependencies every time the workflow is run, therefore improving the performance as described in the Caching section.
Example 1: Lint, test, build, and publish a NPM package
Publish a docker image
Example 2 shows a workflow that is designed to build and push (only if it’s not a pull request) a Docker image for Klicker’s frontend-manage application. The purpose of this workflow is to ensure that the Docker image is built and pushed to the specified registry whenever there are changes to the application code or the workflow itself, using the appropriate environment configuration for the Staging (i.e., QA) environment.
Example 2: Build and publish a docker image
Share data between jobs
The workflow in Example 3 is triggered on a push
event. The workflow consists of three jobs that perform caluclations and share results using artifacts. It’s noteworthy to highlight that jobs that are dependent on a previous job’s artifact must wait for the dependent job to complete sucessfully. Hence, the needs
parameter has to be appropriately configured.
Example 3: Share data between jobs
The workflow run archives all artifacts that are generated. In this case, however, it is only one artifact, since we always use the same name and thus overwrite the uploaded artifacts twice. This is illustrated in Figure 3.
Figure 3: Workflow run captured in GitHub
SonarCloud analysis
Example 4 performs a SonarCloud analysis on the v3 branch and pull requests targeting the v3. It demonstrates how to integrate SonarCloud analysis into a workflow and can help ensure the code quality and maintainability of a project.
Example 4: Perform SonarCloud analysis