Integrations, code, and the oxford comma

Part Four: Drone pipelines – Let's make it do all the hard work for us

Note:

This is Part Four, in a multi-part series.
Here are the other parts that you should probably check out as well:

When I initially started getting into investigating CI pipelines in the first place, it was super confusing to understand exactly why I was supposed to be writing these tests. What was the point? Having only really seen basic steps illustrated in examples around the web (and super complicated unit tests in our BaM projects), I didn’t quite get it initially.

The pipeline and how it works (for dummies like me)

Here’s a look at a basic pipeline setup in Drone:

# This is a basic .drone.yml file

kind: pipeline
name: default

steps:
- name: frontend
  image: nginx
  commands:
  - echo hello world

Now let’s examine this a little bit so that we understand the basics. (I’m adding some quotes from the Drone documentation to further explain how they define the process.)

The top-level steps section defines sets of steps that are executed sequentially. Each step starts a new container that includes a clone of your repository, and then runs the contents of your commands section inside it.

So, we have three things to look at here.

  kind: pipeline
  • This is a top level definition that tells Drone that this is, in fact, a pipeline that it should run. You probably want to keep this in your .drone.yml file if you want it to work.
  name: default
  • This is just the overall name of your pipeline flow. For instance, if you wanted it to display How many jelly donuts can I fit inside of this Amazon box, you can do that. It’s really up to you. Name it whatever you want, I won’t judge.
  steps:
  • And lastly, the steps is where it all happens. This defines what you want Drone to actually do. Later, we’ll put all of the things that actually do something here.

The commands are executed inside the root directory of your git repository. The root of your git repository, also called the workspace, is a mounted volume shared by all steps in your pipeline. This allows filesystem changes to persist between steps.

This is important to understand. ALL of your tasks that you define inside of the steps section happens inside of a mounted volume that is shared across ALL of the steps that you define in your pipeline. These steps happen in the root directory of your git repo. At the beginning of the pipeline process, Drone will automatically mount the working volume and clone your repo into the root directory every time.

The container exit code is used to determine whether the step is passing or failing. If a step returns a non-zero exit code, the step is marked as failing. The overall pipeline status is also marked as failing, and remaining pipeline steps are skipped.

We’ll get more into this, but in a multi-stem pipeline, every step runs in sequence. So if you have a step that fails for whatever reason, your whole pipeline is marked as failing and everything else is skipped. Your build has effectively failed and you’ll get the big sad red X in the UI and on your build status on Github.

Steps, images, and commands

Inside of the steps portion of the pipeline is where you plop all of the good stuff. Heirarchy is important here, so mind your spacing. The suggested syntax for YAML files is to use 2 spaces for indentation, but YAML will follow whatever indentation system that your file uses. When setting up your files, I’d recommend that you use one set of indentation rules across your whole project. Currently, I use 2 spaces for all of my YAML files. This maintains enough tree readibility while not making the whole file messy and unorganized. Use whatever works for you.

Breaking down the rest of the example config, we have the following:

  - name: frontend
  • Simply the name of your step in the pipeline. This shows up in the sidebar of your Drone build as the step is running. Probably name this something that is short and descriptive.
    image: nginx
  • The Docker image that will be pulled down to use for the current build step. In this case, it is pulling the nginx image from the nginx registry on Docker Hub. This particular image is just using nginx with no specific tag, which pulls the standard image set by the repository owner.
    commands:
  • These are commands that are executed inside of your container. Similar to a traditional build script, these do things that you would typically do on the command line. Each command that you want to run is indented under commands and delimited with a - .
    - echo hello world
  • When this command is run, it’s essentialy the same as running echo hello world on the command line in your container. Output is visible in the Drone UI.

    All commands that you put here are converted to a simple shell script. For instance, the command listed in this step would be converted by Drone to:

    #!/bin/sh
    set -e
    
    echo hello world
    

    Once Drone converts the script at the beginning of the pipeline, it is then executed as the Docker entrypoint for your build container. Essentially, it’s adding this to the Docker command inside of your build:

    docker run --entrypoint=build.sh nginx ...
    

    You can put whatever you want your container to execute as a shell command inside of this step and it should just work. For example, if you had specified a Ubuntu image, you could run a shell command in bash like:

    # This is a .drone.yml file using Ubuntu
    
    kind: pipeline
    name: My Awesome Pipeline
    
    steps:
    - name: build
      image: ubuntu
      commands:
      - /bin/bash build.sh
    
      # Depending on the image you might not need to use the whole path for bash
      # - bash build.sh
    
      # Or if you had your script in a directory in your repo, etc:
      # - bash scripts/build.sh
    

Useful things for your pipeline

Secrets

I already started to explain how to enter your secrets back in Part Three, but that was only where you needed to enter your data. (Also see the Drone secrets documentation.) When you want to actually use your secrets IN your pipeline, you need to enter them like this:

# If you're passing a secret into your Docker container,
# put it under the environment tag:

environment:
  AWS_ACCESS:
+   from_secret: aws_access_key_id
  AWS_SECRET:
+   from_secret: aws_secret_access_key


# Or if you want to use it directly from something like
# a plugin, you can do it like this:

settings:
  host:
+   from_secret: system_ip_address
  username:
+   from_secret: ssh_username
  password:
+   from_secret: ssh_password

You definitely want to use secrets instead of putting usernames, passwords, etc. in your pipeline steps. In order to avoid putting any authentication information into a repo that will sit on the internet, and could possibly be publically exposed one day… Say, if you wanted to set your repo to public in the future (on purpose or by accident…), you’ll want all of your private info to not be in your configuration files. Even if you never plan on making your repo public, it’s still good practice to avoid putting this kind of stuff in there at all. USE THE SECRETS!

Plugins

Another great thing about Drone is the extensibility through plugins. You can toss a link to a plugin image into your pipeline and it will run the preconfigured plugin (a small Docker container) in your build without much manual configuration. Usually, this would be a specific task that you want to perform that doesn’t warrant a complete bash script or container to be pulled down, saving time in the build process. By design, most plugin images are tiny and run a minimized set of available resources. See the Drone plugin documentation for more info.

I use a few plugins in my build pipeline for this site:

  • drone-volume-cache - Used to preserve files and directories between my builds. It uses a mounted cache volume to save a persistent cache of files that I can mount and use in my other containers during builds.
  • drone-scp - Used to move my built jekyll _site folder over to my server. I have it setup for both development and production builds in seperate steps, currently. Since I like to develop locally and then push my changes up often, it’s good to see my files on a development site before I push them to the main production directory to be served to the public.

  • drone-ssh - Since I can’t use the drone-scp plugin to do everything on the remote server that I wanted to accomplish, drone-ssh performs a quick login and does them for me. (This includes changing file/dir permissions and restarting docker-compose on the server after each push). Probably not the most elegant solution, as the whole process could be cleaned up and simplified, but it works for now. In the future, I’ll likely just make a bash script that does all of this for me.

My Drone build pipeline

Through trial and error, I’ve setup my .drone.yml pipeline to automatically run various tasks that I would have had to do by hand before. It’s probably not the best or more optimized way to do any of these tasks, but it works for me right now… I’ll break each step down in Part Five. Prepare to be underwhelmed!

kind: pipeline
name: jasonpitman

steps:

- name: bundle-restore-cache
  image: drillster/drone-volume-cache
  volumes:
  - name: cache
    path: /cache
  settings:
    restore: true
    mount:
      - ./assets/images/instagram
      - ./vendor/bundle

- name: images to S3 + sed
  image: python:3.7.3-stretch
  environment:
    AWS_ACCESS:
      from_secret: aws_access_key_id
    AWS_SECRET:
      from_secret: aws_secret_access_key
  commands:
    - pip3 install awscli --upgrade --user
    - export PATH=/root/.local/bin:$PATH
    - bash _scripts/aws_creds.sh
    - bash _scripts/s3_upload.sh
    - bash _scripts/sed_findreplace.sh

- name: build the site
  image: ruby:latest
  commands:
    - gem update --system
    - gem install bundler
    - bundle install --path ./vendor/bundle
    - bundle exec jekyll build
    - bash _scripts/rm_imgs.sh

- name: bundle-rebuild-cache
  image: drillster/drone-volume-cache
  volumes:
  - name: cache
    path: /cache
  settings:
    rebuild: true
    mount:
      - ./assets/images/instagram
      - ./vendor/bundle

- name: dev-scp
  image: appleboy/drone-scp
  when:
    event:
      - push
    branch:
      - dev
  settings:
    host:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    target: /srv/www/dev/public
    rm: true
    source:
      - _site/*
    strip_components: 1

- name: prod-scp
  image: appleboy/drone-scp
  when:
    event:
      - push
    branch:
      - master
  settings:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    target: /srv/www/prod/public
    rm: true
    source:
      - _site/*
    strip_components: 1

- name: dev-ssh
  image: appleboy/drone-ssh
  when:
    event:
      - push
    branch:
      - dev
  environment:
    SSH_DEPLOY_KEY:
      from_secret: ssh_deploy_key
  settings:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    script:
      - cd /srv/www/dev/public
      - find . -type f -exec chmod 644 {} \;
      - find . -type d -exec chmod 755 {} \;
      - cd /srv/www/dev
      - docker-compose down
      - docker-compose up -d

- name: prod-ssh
  image: appleboy/drone-ssh
  when:
    event:
      - push
    branch:
      - master
  environment:
    SSH_DEPLOY_KEY:
      from_secret: ssh_deploy_key
  settings:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    script:
      - cd /srv/www/prod/public
      - find . -type f -exec chmod 644 {} \;
      - find . -type d -exec chmod 755 {} \;
      - cd /srv/www/prod
      - docker-compose down
      - docker-compose up -d
volumes:
- name: cache
  host:
    path: /tmp/cache

Continued in Part Five »