Tag Archive: devops

CFP: Things I love about SRE that I loathe about DevOps

January 28, 2020 8:32 pm Published by

DevOps & SRE – what is it?

Let’s define what those terms mean.

DevOps means, the same team building the software is responsible for running it. This can be easiest imagined for software that is operated by the vendor themselves, i.e. cloud services. The idea is, no one knows the software better than the developers, so no one could better operate and fix in-flight issues than them. On the other hand, developers have high interest in building software that is easiest to operate, if they operate it themselves. Issues found during operations can be addressed immediatly.

SRE is a concept where one team is responsible for running one or more services. Again, imagine a company building cloud services. They may all have similar requirements in operations, so why not create a dedicated team to run them all. This team contains software developers to automate common tasks that emerge when running the services to minimize manual efforts. SRE teams monitor the software to spot issues as soon as they appear and fix them in the best case before a customer notices them.

Which DevOps problems may SRE solve?

In DevOps, having a team that runs software and a separate team that builds it, is a well-known anti-pattern.

“If you have a DevOps team, you’re doing it wrong”

This quote can be found in many tweets, blog posts, and conference talks. But when it comes to SRE, this is a common pattern. One team is building software, another team is running it, and building software to improve running it.

In DevOps, you usually find a specific support role being rotated through the team. That means everybody is in that role for example for a week out of 10, given a team of 10 members.

Working in this support role usually has very different requirements than the usual development work.

The context can get completely lost during the transition into the support role (which often comes as surprise on Monday mornings). And once context is built up and the developer feels comfortable in that role, his shift ends and the support context is again lost.

This results in people hating to get into the support role, which also makes it less attractive to build things that improve the supportability of the software.

In SRE teams, people are on call as well every few weeks, often even 50% of their time. Why don’t they dislike it? The difference is the focus of the team. When not on call, developers in the SRE team work on improving the support. They are less involved in actual product code, but rather build software to ease the operations. For example, to automate updates and other maintenance tasks.

DevOps: supportability vs. new features
SRE: supportability = new features
DevOps: Dev team learning ops all together
SRE: Ops learning from devs and devs learning from Ops

Agile development in SRE teams

As SRE teams contain a fair portion of software development work, and get filled up by software developers, it is a natural move to also adapt agile software development practices. It depends heavily on the percentage of development work vs operations, which may be influenced by the team size, to find the right model to track and manage the project work. For example, in a small team where a high percentage of people is on call during the day, it might not make too much sense to plan sprints of 2 weeks if only a few backlog items are expected to get done in that timeframe.

Key Takeaways of this talk

By the end of this talk about the differences between SRE and DevOps working styles, attendees should have awareness (1) what the most significant differences between DevOps and SRE are, (2) A successful team running DevOps or SRE needs experienced Ops as well as Dev people, (3) If team members greatly dislike getting into the operating role, the team should work hard on improving the support experience.

Use Ansible to clone & update private git repositories via ssh

July 7, 2018 7:21 am Published by

One of the first things I wanted to do when I started using Ansible was to clone a git repository on a remote machine as I keep configuration, scripts, and source code in github or gitlab repositories. Things that are not meant for the public, I store in private repositories that I want to clone via ssh. Cloning and updating them I now want to automate with Ansible.

There are different ways to go for this task:

  • Checkout the repo locally and copy it to the server via a Ansible synchronize task
  • Generate an ssh key on the server and allow cloning the repo with that key manually
  • Copy a local ssh key to the server and allow cloning the repo with that key
  • use ssh-agent to load the local key and forward the agent to the server
While it might be tempting to just copy an ssh key via Ansible to the remote server, I find this quite risky,  as it means you copy a secret to a persistent storage on a remote server. Also, if you version your Ansible playbooks in a git repository as well to be able to execute the playbook from somewhere else, the private key has to be versioned along with it.

Using ssh-agent, you can easily load your ssh key prior to provisioning the git repo on the remote server without copying it over, and without allowing access to your repo for a different key than the one you have granted access for development.
Let’s go through this via a simple example. Let’s say you want to run the following playbook, which includes ensuring the git repository github.com/ntlx/my-private-repo is up-to-date.

1
2
3
4
5
6
7
 
---
- hosts: webserver
  tasks:
      - name: Ensure repo is up-to-date
        git:
            repo: git@github.com/ntlx/my-private-repo.git
            dest: repos/my-private-repo
I assume you added your public ssh key to your github.com repository so you are able to clone and work on the repository locally. To clone the repository on the remote machine, you need to load your ssh-key to ssh-agent with the following command.

ssh-add ~/.ssh/id_rsa

Now we need to enable the forwarding of the ssh agent to the remote machine so we can access the loaded key remotely. There are different ways to do so, but I find it most useful to do it in your ansible.cfg like this:

1
2
 
[ssh_connection]
ssh_args=-o ForwardAgent=yes

That way, you allow the forwarding for all your Ansible-managed hosts at once.

Now you can go on executing your playbook and should be able to clone the repository on the remote host.

To make it even easier, we can add a task to load the ssh-key before executing the other tasks in the playbook. For this, add the local host to your Ansible inventory:

1
2
 
[local]
local_machine ansible_connection=local ansible_host=localhost

Now we can add a small shell task to load the ssh-key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
 
---
- hosts: local
- name: load ssh key
  shell: |
      ssh-add ~/.ssh/id_rsa

- hosts: webserver
  tasks:
      - name: Ensure repo is up-to-date
        git:
            repo: git@github.com/ntlx/my-private-repo.git
            dest: repos/my-private-repo

When you now execute the playbook, you shouldn’t need to load the ssh-key before.