Not everyone has the same idea about DevOps. For some it is a culture, others describe it as processes or practices, for others is a set of tools. Part of the problem is that the term itself is so broad and includes so many factors that it is not easy to clearly define it or set its boundaries.
The simplest—and probably the most popular—definition describes DevOps as the intersection of Development and Operations. The problem arises when we try to set the boundaries of where Development ends and Operations takes over and vice versa.
The Need for DevOps
To begin with, why do we even need something like DevOps? To understand this, we need to look at the release process.
When you develop an application, the end goal is always the same—no matter whether the team follows a waterfall strategy or agile or any other—and that is to deliver the application to the user.
Say, you have an idea for an application. You will bring it to life by first defining its functionality and setting out its features, then you will code the application, following which, you will test it, and finally host it on a server—probably a public server along with all the security it needs, viz. firewalls, access rules, etc. This is a stripped-down description of the steps any application deployment needs. But the story doesn’t end here.
You still need to monitor your application and check whether it is running as desired: are there any bugs, can the architecture handle high traffic loads, are users facing any problems? If any issues crop up, you, of course, need to fix them. Even if everything is fine, you may want to add new features, and further improve performance: maybe by provisioning better servers or making the application faster. But this isn’t the end… there’s still lots to do. And every time you make any changes to the application, be it the code or underlying infrastructure, you need to redeploy it to make it accessible to end users. Now there will be multiple iterations post the initial launch and you need to keep track of the updates, so you also need to establish a versioning system and keep track of them. The cycle of launch-update-relaunch is an endless cycle and DevOps helps streamline this by establishing a continuous delivery process. DevOps, in short, enables you to make endless improvements to your application, test the changes and deliver the updated application to your end users without interruption.
Challenges
There are several bottlenecks and roadblocks that slow down the deployment process and they can be listed as under:
- Lack of collaboration
Lack of collaboration between developers and Operations is the biggest challenge. These are actually separate aspects of the process. Developers are only concerned with creating (coding) the application and Operations personnel are responsible for running the application. Operations are running the application but often aren’t aware of how it works, so any attempt to tinker with the code is a recipe for disaster. Typically, developers will complete the coding but the documentation may be incomplete or just not good enough for the Operations team to deploy. If there are many usage issues, Operations simply throws it back to the Development team with improvement recommendations. All this means release takes longer; sometimes days and weeks and maybe even months.
- Checklists as bottlenecks
Sometimes handover from developer to Operations is governed by bureaucratic checklists that set out what needs to be completed, who needs to approve what, and so on. The problem with this is that it results in further delays as a lot of these tasks must be completed manually, checked, and verified before handing over.
- Differentiated goals
Developers are primarily focused on pushing out new features fast. This is their objective, whereas Operations want to ensure the application runs in a stable fashion. To ensure this, the Operations team, traditionally, tends to delay the release until all the i’s have been dotted and the t’s crossed. Since Operations usually isn’t familiar with code the process is a slow one. For instance, let’s say a new set of features has been developed but these features end up consuming so many resources in production that servers get overwhelmed and the application crashes. Now Operations needs to get it up and running again. Since fire-fighting is Operations’ responsibility, Developers are not are quality conscious as they should be, focusing more on the speed of churn than stability Even though it is the common goal of everyone in the organization to deliver high-quality applications to the end-users, quickly, in practice, there is conflict as both teams focus on their immediate goals.
- Security challenges
Just as the Operations team evaluates the changes to make sure it won’t break anything in a production environment, so too must the Security team evaluate changes to ensure they don’t impact security systems. The only problem with this is when done in a traditional setup, this process is another bottleneck as it involves the same kind of bureaucratic process as we saw in the Operations check. So it will take another couple of days or weeks, further slowing down the deployment process. When DevOps incorporates automation of security checks, it is known as DevSecOps.
- Manual processes
In a traditional setup, many of the tasks that are part of the deployment process are done manually and this slows down things considerably. For instance, Operations tasks are done manually by directly executing commands on the servers, patching, writing and running small scripts/programs, manually creating deployment environment and infrastructure, configuring access to servers, manually configuring Jenkins, and so on. All this manual effort is not just time-consuming, it is also prone to errors, and it is very hard to trace who executed what and when. So if something were to happen to the infrastructure, it would be very difficult to replicate the exact state quickly.
DevOps aims to resolve all these and focus on creating a fully automated streamlined process. We do this by removing the bottlenecks, one at a time, until we have a fully-optimized process that makes application releases easy and so fast, you can have multiple releases a day, instead of days or weeks, or months, like previously.
Defining DevOps
Per the official definition, DevOps is a combination of
- cultural philosophies
- Practices
- Tools
All of which aim to make the release process error-free and speedy.
The traditional definition of DevOps slowly evolved as the actual implementation of DevOps by individual companies was not uniform, but it eventually developed concrete patterns, and one of the changes that came from this was the creation of a specific role: the DevOps Engineer. The DevOps engineer could be either a developer doing DevOps or an Operations engineer only doing DevOps or a separate role. But the sole job responsibility would be to focus only on creating a streamlined process for release. And this process centers around the familiar process known as CI/CD, or continuous integration, continuous delivery/deployment.
What every DevOps Engineer Should Know
While DevOps engineers may not be working on the actual programming of the application, they need to understand versioning and how developers work, and how to prepare an application for release, this includes:
- Understanding how Git works, which Git workflow the development teams are using, how the application is configured, tested, etc.
- How the application is configured to interact with other services/databases
automated workflows
- Development of underlying Infrastructure, i.e. creation and configuration of servers, be it on-prem or in the cloud. This means proficiency with Linux servers, as these are the most commonly used ones.
- How to set up, and administer a Linux server, Linux commands, and Linux file system so the developer can use the Command Line Interface
- Basics of networking and security…configuring firewalls, how IPs address ports, opening some ports so the application can be accessed from outside; DNS and how it works
A DevOps engineer needn’t have advanced operating system skills or networking and security skills, or even how to administer servers from end-to-end. There are already specialists for these tasks. The DevOps engineer’s job is to understand all of the above only to the extent needed to prepare the server to run the application, rather than take over management of the servers and infrastructure.
- Containers: As containers have become the de facto infrastructure nowadays, the DevOps engineer needs to understand how to manage containerized applications. This means acquiring proficiency in popular container tools, like Docker. This means how to build docker images from your application, and save these in a Docker image repository, like Nexus or Docker hub.
- CI/CD pipelines and associated software
DevOps job is basically to release new application versions, fixes, etc, in a continuous and efficient, and automated way. To do this the DevOps engineer needs to know how to run tests, package the application as a jar or zip file, and deploy it. So they need to know software like Maven, Gradle (for Java), and NPM (for Javascript apps).
Now, this doesn’t have to be done manually, instead, the DevOps engineer needs to build an automated pipeline so all of the above is done in a sequential manner. This means knowledge of Jenkins, which is the most popular build automation tool. Next is connecting the pipeline to the git repository to get the code. This is the key part of the Continuous Integration process. Next is continuous testing and bug fixes if needed before deploying the new feature or bug fix. And this is Continuous Deployment. Along the way, it should also send notifications to the respective teams about pipeline state and failed deployments.
Continuous Integration/Continuous Deployment pipeline
In a Nutshell
Basically, DevOps connects Development and Operations teams. While there is no need to be a master in either of these areas, some basic knowledge of both is needed. Above all, DevOps engineers need to know all the tools and practices needed to build automated pipelines that can get the code for the new releases, test it, fix it—if there are bugs—and deploy it. This is a continuous process which is why the symbol for DevOps has become the infinity symbol, showing movement from Development to Operations in a continuous cycle.
