Table of contents
No headings in the article.
CI/CD, Continuous Integration, and Continuous Delivery is widely adopted and continue to gain traction as it facilitates faster delivery of products to the market before. As quick delivery is one of the main benefits of using a CI/CD pipeline, it is important that we focus on best practices for having a faster build. Depending on the tech stack there could be different ways to improve the speed but this article focuses on seven generic ways.
Optimize development process and adapt CI/CD pipeline to it
Optimize tests
Caching and artifacts
Parallelization
Microservices and Microfrontends
Containerization
Vertical scaling
Optimize the development process and adapt CI/CD pipeline to it
Build small, build what you need: Instead of building and deploying as much as possible in one go, it will be more optimized to split it into multiple independent builds. For example, A new payment feature supporting two new payment methods can be can be split into three deployments — One for each payment gateway integration and another for the final payment feature.
Modularize: Even if the architecture is monolithic, modularize as much as possible to allow for smaller commits thereby smaller and faster builds. Most programming languages and frameworks support building and integrating such modules within a single project. A shorter code-change-result cycle makes the code easier to fix and update also.
Avoid multiple features at once: Focussing on a single feature at a time will help speed up the build, improve testing and debugging, provides option to revert a single feature without impacting others, as well will be less confusing to end users.
Caching and artifacts
Caching: Downloading modules at build time takes a significant portion of a build. Caching can be used to speed things up, instead of starting from scratch on every build. Use cache for dependencies, like packages you download from the internet.
Artifacts: Jobs can output an archive of files and directories which are often referred to as artifacts or job artifacts. Use artifacts to pass intermediate build results between stages. Tools like Artifactory can be used for this.
It’s important to delete your cache and artifacts when you update them or when they are no longer necessary. You can also set an expiration for the same to manage it automatically.
Parallelization
Jobs can be run in parallel to decrease the overall build time. However, this might come up with an additional cost of resources required for running the jobs in parallel. A trade-off between cost and time needs to be done depending on the project priority before finalizing to go with parallelization. Almost all platforms like Jenkins, Bamboo, GitLab, CircleCI, and Bitbucket, etc. provide the option for parallelization within the box.
Apart from build jobs, tests can also be run in parallel providing significant improvement in build times. Some test suits provide the option for it and while others can be implemented using containers or multiple agents, etc.
Optimize tests
Regularly evaluate the tests: Over time we might have kept on adding new test cases and it is possible that we ended up creating duplicate or unused tests, so evaluate and remove redundant and obsolete tests.
Use mockups: Mockups allow to simulate connections to third-party services. As you need to test your code and not the services (especially external services), using real services is unnecessary and slows down the entire process.
Evaluate test tools: Some testing tools are faster than others, as well as some test settings can impact overall testing time.
Microservices and Microfrontends
Monolithic applications take a long time to compile and test, which can be overcome by using microservice architecture. Microservices can help build modules independent of each other and deploy them separately. This will reduce the overall deployment time as well as help improve the testing and debugging process.
In Micro-frontends, we can separate functionalities of the application and maintain separate build and release pipelines to constantly deliver updates and bug fixes. It is possible to integrate and deploy each app independently, allowing delivery of critical fixes more quickly making the CI/CD processes much faster.
Containerization
Containerization ensures consistency between different environments such as development, testing, and production environments, making it increasingly essential for continuous integration and development.
Scaling is easier and faster with containerized CI/CD pipelines. Scaling containers just require a container orchestrator, such as AWS ECS or Kubernetes, which has the added advantage of making container management easy.
“Build images” that include all the third-party libraries can be pre-built to reduce that time in the final pipeline.
Shared volume that includes the cache can be created and attached to the containers.
Like build automation, testing tools and scripts can be packaged in separate containers allowing test parallelization.
If different applications are using the same CI/CD infrastructure then probably there will be a different set of software and its versions. These can cause conflicts between software components which can be avoided with containers.
Vertical scaling
Though vertical scaling of CI/CD builds attracts additional costs, it might still be cost-effective in some situations. For example, rewriting a large monolithic application to microservices would incur additional costs in terms of manpower which would be much larger than deploying a powerful machine.
Processor-bound tasks, like building executables, or performing end-to-end tests, would run faster with vertical scaling. It will also allow the parallelization of these tasks.
In most cases, vertical scaling can be implemented much faster than horizontal scaling, especially in cloud environments. This would be helpful in time-bound situations where the priority is to run the pipelines run as early as possible.
Let me know in comments what are the other optimization techniques used in your organization.
#build #DevOps #Jenkins