Integrations, code, and the oxford comma

Part Five: Breaking down MY Drone pipeline

Note:

This is Part Five, in a multi-part series.
Here are the other parts that you should probably check out as well:

As promised, I’m going to go through each of the steps in my Drone pipeline. Since several of the steps serve similar purposes, I’ll go over them together based on what they’re actually doing. Go back to Part Four if you want to see the whole .drone.yml file that I use.

Volume Cache

First off, you don’t need a cache in your build. The only reason that I added this into the pipeline in the first place is that I wanted to shave off some build time, as each change was occupying about two minutes of time for each build. I was hoping to get that number down a bit by caching some of the frequently reused bits. I definitely would recommend getting the rest of a regular build process nailed down before you even attempt to try to get this setup and integrated.

Restore cache

The purpose of the restore cache step should be relatively obvious. You want to restore your cache at the beginning of your pipeline (if it exists). This is the first step after the pipeline clones your git repo into the container, as you want your cache to be available in the steps early enough.

If the restore cache step finds an existing cache for the specified folders, it loads them up. Otherwise, it will skip the restore. This step is really quick and hardly noticible in the overall build time.

- name: bundle-restore-cache
  image: drillster/drone-volume-cache
  volumes:
  - name: cache
    path: /cache
  settings:
    restore: true
    mount:
      - ./assets/images/instagram
      - ./vendor/bundle

In the above step, I’m mounting the assets/images/instagram and vendor/bundle folders inside of the cache volume (setup below) and mounting each folder in their respective (original) locations, if the cache of those specific folders exist on the cache volume already.

You can mount any folder you want to be cached here and give it whatever path you need for your project. The restore: true tells the plugin that you want to restore your cache from the volume. This is required.

When this step is running, you’ll see output (in the Drone UI) that looks similar to this (if you have an existing cache):

1. Restoring cache for folder ./assets/images/instagram...
2. Restoring cache for folder ./vendor/bundle...

Your cache should now be ready to be used in your build.

Note: Your build has to have been run at least once to first generate the initial cache. If this is the first time you’re running the build, the plugin will tell you that the cache doesn’t exist and will move to the next step.

Rebuild cache

This step is almost identical to the bundle-restore-cache step, but it should be at the END of your pipeline, or at least where you want the cache to be generated inside of your steps. I have this inserted in after by _site build step and before any network operations.

- name: bundle-rebuild-cache
  image: drillster/drone-volume-cache
  volumes:
  - name: cache
    path: /cache
  settings:
    rebuild: true
    mount:
      - ./assets/images/instagram
      - ./vendor/bundle

When this step is running, you’ll see similar output generated in the UI:

Rebuilding cache for folder ./assets/images/instagram...
Rebuilding cache for folder ./vendor/bundle...

The cache volume

At the end of your pipeline, we specify that we want to use a volume (that already exists) for the cache. The path of the host (Drone container) has this volume mounted when it starts up, so it will already be available for our steps above to use in the pipeline.

volumes:
- name: cache
  host:
    path: /tmp/cache

Note: This volume needs to be setup on your Drone container when Docker is started, otherwise your step(s) in your pipeline won’t be able to mount it because the /cache volume won’t exist yet.

Adding the cache volume to your container

You’ll want to add --volume=/tmp/cache:/cache to your Docker command when you setup your Drone instance. If you already have an existing container, it’s recommended that you create a new one with the added volume specified at creation time. It’s a huge pain to add one to an existing container, I’ve found. It’s easier to just recreate your instance.

If you’re already running Portainer, however, it’s easy to add a volume to an existing container. It’s almost exactly the same as when we added a new ENV variable earlier. Here’s how you do that really quickly:

  1. Navigate to your container in the main list and click on the name.
  2. At the top, hot the STOP button to stop the running container
  3. Hit the Duplicate/Edit button
  4. Click the Volumes tab at the bottom of the page
  5. Click the map additional volume button above your existing volumes
  6. You want a Bind mount, so be sure to click to change from a Volume mount, which wants to use an existing image
  7. For the container field, enter /cache
  8. For the host field, enter /tmp/cache
  9. Hit the Deploy Container button in the Actions section above, click OK on the rename warning

Now, your new (edited) container should start up and will be running with your newly setup cache volume. Way easier than recreating your whole container, right?

Clearing your cache, or running a build with no cache

You have a few options if you don’t want to run a build with your existing cache, or want to clear out the cache. In my case, since I’m caching my images that I upload to S3, if I don’t clear out my cache directory before when I run the AWS step below, my build won’t see any new images that I’ve added to the folder in my repository. Adding an additional comment to a commit message is a quick way to clear it on the next build manually so that any new image will then be seen by the AWS step.

From the volume-cache documentation:

  • [CLEAR CACHE] - instruct the plugin to clear the entire cache. This only influences the restoring step, the plugin will still rebuild cache if instructed to do so.
  • [NO CACHE] - instruct the plugin not to restore or rebuild cache for this build.

If you add one of these to your commit, you should then see this in your Drone step:

Found [CLEAR CACHE] in commit message, clearing cache...

AWS

One of the more annoying parts of my pipeline is my Images step. It took a bit of trial and error before I found something that actually worked in processing this big dumb directory full of images. Since I have all of my Instagram images (thumbnails and full-sized images) processed and dumped into the assets/images/instagram folder, my initial builds took a decent amount of time.

Each time the site was built and uploaded, it had to process ALL of these images and put them onto the server. Every time. Since I wanted to keep this stupid folder of images in my repo for local development purposes (and I may revisit that decision in the future), I needed a way to move them with less overhead. So, why not put them on S3 and re-link everything in my codebase?

Easy enough, you’d think. But we’re talking about hundreds of images and hundreds of markdown files that needed to be edited. And because I’m a stubborn prick, I wanted everything to point to the original folder when I was developing locally. Whelp, I guess that I needed to write a script to make that easier. The trick was, I didn’t want to rename anything in the source of my codebase. I wanted everything renamed ONLY in the compiled build.

AWS Command Line Interface

Since I needed to put the images on S3, I needed to make a step to do this. Only problem is that I wanted to use the awscli so that I could run everything from a script. That meant that I needed to either find an image that already had it installed, or install it in my build.

First, I used a few different Docker images and installed everything via the pipeline commands. Success varied, depending on what image I was using, but I wasn’t happy with how long the whole process was taking. I also tried several existing Docker images that already had the awscli installed, but they were all slow and I always needed to install additional packages to run my other scripts.

Since the awscli runs in Python and installs via pip, running a current version of the Python3 image was the answer. In all other previous attempts, I had to manually install Python or pip, which was super slow. With the stock Python3 image, everything was already there, since I didn’t need to do those steps. Here’s what I ended up with:

- name: images to S3 + sed
  image: python:3.7.3-stretch
  environment:
    AWS_ACCESS:
      from_secret: aws_access_key_id
    AWS_SECRET:
      from_secret: aws_secret_access_key
  commands:
    - pip3 install awscli --upgrade --user
    - export PATH=/root/.local/bin:$PATH
    - bash _scripts/aws_creds.sh
    - bash _scripts/s3_upload.sh
    - bash _scripts/sed_findreplace.sh

Environment

I’m using Drone secrets here to pass the aws_access_key_id and aws_access_access_key values to my scripts. Nothing special here, other than putting them in the environment section makes them available to my commands. It basically passes the values like this:

Secret name Name passed to commands Value from Drone secret
aws_access_key_id AWS_ACCESS My AWS access key value
aws_secret_access_key AWS_SECRET My AWS secret key value

Commands

In the command section of my step, I’m basically doing three things:

  • pip3 install awscli --upgrade --user - install awscli with pip
  • export PATH=/root/.local/bin:$PATH - this inserts the path, ~/.local/bin at the front of the existing PATH variable in the shell profile (Now that I dig deeper into the documentation while writing this, this might not be necessary. Looks like I’d need to source this into the current bash profile with source ~/.bash_profile. Hah. I should really check that out.)
  • Run the three scripts below after awscli is installed

Scripts

I have three bash scripts that I run in this step to accomplish several tasks.

The AWS credentials script

bash _scripts/aws_creds.sh

This script sets up the credentials so that the awscli can talk to AWS and put things in my buckets. Looking at the file, I’m doing three things:

  • Make the .aws directory to put the credentials in
  • Create the file credentials in the .aws directory and pass the keys from our Drone secrets
  • Create the file config in the .aws directory and pass the region and output format that tells AWS where and how to do things
#!/bin/bash

#============================#
#  Make the S3 config files  #
#============================#

### Make the output look better

echo "----------------------------------------"
echo "Setting up AWS real quick..."

### This is the part that actually
### creates your files. Keep this.

mkdir ~/.aws

cat > ~/.aws/credentials << EOF
[default]
aws_access_key_id=${AWS_ACCESS}
aws_secret_access_key=${AWS_SECRET}
EOF

cat > ~/.aws/config << EOF
[default]
region=us-east-1
output=json
EOF

### Testing AWS - Will output the contents
### of the two files generated above

echo "----------------------------------------"
echo "AWS is configured. Here's your info:"
echo ""

echo "cat ~/.aws/credentials"
cat ~/.aws/credentials

echo "cat ~/.aws/config"
cat ~/.aws/config

echo "AWS is configured!"
echo "----------------------------------------"

echo ""

I’ve included some formatting that makes the output in the Drone UI look a little better. It also helps in debugging if you run it with all of the steps above enabled. That way, you can see that it’s actually making your files as it echos the contents of the actual file(s) that it just created. Not necessary at all, and you can safely remove any of the extra stuff once you’re ready for production.

The script that actually does the uploading to S3

bash _scripts/s3_upload.sh

This script runs two commands for the S3 image side of things:

  • The first simply changes the working directory to assets/images/instagram
  • The second runs the aws s3 sync command using the awscli:

    • The . signifies that you want to sync the current dir
    • The s3://jasonpitman/assets/images/instagram is the bucket/dir that the files are synced to
    • The --quiet tells the awscli that I don’t want it to garbage up the terminal output with EVERY file sync operation listed
    • The --acl public-read tells awscli to set the ACLs on all of the synced files to public
#!/bin/bash

#======================================================#
#  Upload anything in /assets/images/instagram/ to S3  #
#======================================================#

echo "----------------------------------------"
echo "Starting the S3 upload..."
echo "Syncing /assets/images/instagram/ to S3"

### This is the part that actually
### does the stuff. Keep this.

cd assets/images/instagram
aws s3 sync . s3://jasonpitman/assets/images/instagram --quiet --acl public-read

echo "S3 upload is done!"
echo "----------------------------------------"

echo ""

Again, prettied up the UI output with some echo commands. I don’t want this stuff to be too unreadable. I’ve found that if I add this kind of stuff as I’m writing my scripts, I’ll do it in ALL of the scripts that I write for a project. That way, when I go back to edit them or do documentation, I actually know what any given part is doing at a glance. Seriously… write your documentation as you go. It may be more work initially, but it’s invaluable in the long run.

bash _scripts/sed_findreplace.sh

This was seriously the most annoying part of the whole process… getting this script working correctly. You’d think that a simple rename of a specific string in a folder full of files wouldn’t be hard to quickly replace… and while that’s true, it took me awhile to get right syntax to do it. I need to brush up on my bash scripting.

So, this script does the following:

  • set -e stops the script execution on the first error. We don’t want this to run wild and try to change a bunch of files if it trips up on the first (or twentieth) one.
  • Since we’re not working in the _site directory that Jekyll builds quite yet, we use the _images directory, which holds all of the markdown files for each image/post that got pulled down from the repo. Since we’re not committing any changes here, there shouldn’t be any issue changing out local files before we process them for the _site build in the next step. This just sets the working directory to the _images folder in our root.
  • Use the for statement to create a loop of all files in the working directory (that we set in the last step). This essentially tells the script that for each file it encounters, we want it to do the following stuff to it. The /**/* just tells the loop to run through all files in the working directory.
  • strtorep (string to replace) lists what we want to hunt for and replace
  • newstr (new string) lists the string that we want to replace the original string with. In this case, we’re replacing the local images directory with our hardcoded S3 URL of the images directory in our bucket
  • I won’t get into the sed command yet (Just Google it. sed does A LOT of stuff and has a syntax that I have yet to wrap my brain around), but I have two versions of essentially the same command.

    • The original gsed command (that’s commented out) is in there for local testing. Using the homebrew command brew to install gnu-sed instead of using the local sed installed on a Mac. I had nothing but problems here and gnu-sed just works, so I’m keeping it in here for future reference.
    • The Linux version works just fine using the normal sed command, and is what I’m actually using in this script.

    Note that the normal forward slashes usually seen in sed commands are replaced with a |, because trying to replace URLs in a string that includes forward slashes doesn’t work and will error out. The pipe is just a substitute character that sed will use instead.

#!/bin/bash

set -e
workingdir="_images"

#===============================================#
#  Rename the local image paths to the S3 path  #
#===============================================#

echo "----------------------------------------"
echo "Renaming local paths in MD files..."

### This is the part that actually
### does the stuff. Keep this.

for file in $workingdir/**/*; do
    strtorep="/assets/images/instagram/"
    newstr="https://s3.amazonaws.com/jasonpitman/assets/images/instagram/"

    # -- Mac version --
    # Need to use gsed for this to work for some reason (on a mac)
    # brew install gnu-sed
    # gsed -i -e "s|${strtorep}|${newstr}|g" "$file"

    # -- Linux version --
    # Using pipes instead of forward slashes because
    # we're using a URL string that's getting replaced
    sed -i -e "s|${strtorep}|${newstr}|g" "$file"
done

echo "Renaming is done!"
echo "----------------------------------------"

Yep. More “pretty everything up” echos here too. I think this is actually becoming a habit? Time will tell.

Since the assets/images/instagram folder is cached already via the bundle-restore-cache and bundle-rebuild-cache, the aws s3 sync command in the script is pretty quick. It only has to read what’s in the bucket already and compare to our local folder in the cache. The whole command step takes less than 15 seconds, with the bulk of that 15 seconds being actually installing and building awscli itself. In my initial tests (when I tested images that had awscli pre-installed), the process actually took longer for some reason. I can deal with 15 seconds for now, but I might revisit it in the future and build my own custom image for these steps that runs even faster.

Building the site

Finally… we’ve actually made it to building the site. Which is the least interesting part in all of the steps, of course. Using a ruby container image, we’ll tell this step to do the following:

Commands

  • gem update --system - update the system gems in the image
  • gem install bundler - install bundler
  • bundle install --path ./vendor/bundle - run bundle install and set the install path to ./vendor/bundle, since that’s where I’m mounting the cache of things that bundler installs
  • bundle exec jekyll build - run the Jekyll command and build the site

Scripts

I only have one bash script in this step. It’s so simple, it should probably just be a command. Maybe someday.

Image deletions!

bash _scripts/rm_imgs.sh

This script does a single thing, but it’s such a time saver for the rest of the pipeline that it’s super important:

#!/bin/bash

set -e

#========================================================#
#  Remove all of the images in /assets/images/instagram  #
#========================================================#

echo "----------------------------------------"
echo "Dumping all of the instagram images"

rm -rf _site/assets/images/instagram

echo "Dumping is done, dude!"
echo "----------------------------------------"

Yep, it really only runs rm -rf _site/assets/images/instagram, which deletes the whole instagram images folder inside of the _site folder that Jekyll just built. Well that, and it puts the pretty echo output in the UI so that we see that it’s doing something, surrounded by a delightful presentation of dashes and status info.

Why am I deleting the instagram folder AFTER I just built it with Jekyll, you ask? Good question. I suppose that it could probably come before the build step, and I could just delete assets/images/instagram instead of _site/assets/images/instagram and that would make sense, right? Well, since I’m rebuilding the cache in the step AFTER the build command, I want the assets/images/instagram folder to still exist when that step happens in the pipeline. If it tried to rebuild the cache after I deleted the folder, there wouldn’t be anything in it to cache for the next build. It’s not that much overhead, as it’s just copying files, so I kept it here.

The whole step looks like this:

- name: build the site
  image: ruby:latest
  commands:
    - gem update --system
    - gem install bundler
    - bundle install --path ./vendor/bundle
    - bundle exec jekyll build
    - bash _scripts/rm_imgs.sh

SCP

Now that we have the cache setup, everything uploaded and replaced, and our site built… it’s time to upload it to our server. Yay!

I’m using the drone-scp plugin here to do all of the work for me, rather than scripting it myself. Since I have two different places that I want to push the site up to (depending on what git branch I’m pushing to), I have two different steps specified. One for the master branch and one for the dev branch. The only difference in each is the directory that they get pushed up to during the scp process. Drone secrets are heavily utilized here. Everything here should be fairly obvious if you’ve ever connected to an sFTP site, but there are a few things to note about this setup:

  • The when statement for the step tells Drone to run this step ONLY if the branch is the same as the one referenced. The event tells Drone that I want this step to run everytime I push to this branch. There are a lot of other options on how often you want to do this, which branches you want to use, what tags or release to check, etc. Check out the conditions documentation here to see all of it.
  • The rm: true tells the plugin that I want to remove the target directory before it copies anything to my target. This essentially nukes the current files on the server so we can put a fresh batch up.
  • Since I’m using _site/* as my source folder, we want to make sure to use strip_components: 1 which removes the specified leading path elements. In this case, it removes the _site folder and puts all of my uploaded files at the root of my target directory on the server.
- name: dev-scp
  image: appleboy/drone-scp
  when:
    event:
      - push
    branch:
      - dev
  settings:
    host:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    target: /srv/www/dev/public
    rm: true
    source:
      - _site/*
    strip_components: 1

- name: prod-scp
  image: appleboy/drone-scp
  when:
    event:
      - push
    branch:
      - master
  settings:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    target: /srv/www/prod/public
    rm: true
    source:
      - _site/*
    strip_components: 1

The drone-scp plugin takes my specified directory of files and makes it into a tarball for transfer to the target site. Once the tarball is there, it runs untar and deletes the tmp generated .tar file. Here’s some example output in my Drone UI for this step:

tar all files into /tmp/029241705/OOKkgAmis6.tar
scp file to server.
Remove target folder: /srv/www/dev/public
create folder /srv/www/dev/public
untar file OOKkgAmis6.tar
remove file OOKkgAmis6.tar
================================================
Successfully executed transfer data to all host.
================================================

SSH

SSH. Goddammit, SSH. I’m using the drone-ssh plugin here to do some things that I’m not entirely sure is necessary. Since the ssh plugin is extremely similar to the scp plugin (they both have the same author), I won’t get into the when, the env, or any secrets setup. I’ve got the same dev/prod setup here, dictated by which branch I’m pushing my files to. We’ve been over this stuff already. It’s not pretty, but it seems to work right now, so here it is:

- name: dev-ssh
  image: appleboy/drone-ssh
  when:
    event:
      - push
    branch:
      - dev
  environment:
    SSH_DEPLOY_KEY:
      from_secret: ssh_deploy_key
  settings:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    script:
      - cd /srv/www/dev/public
      - find . -type f -exec chmod 644 {} \;
      - find . -type d -exec chmod 755 {} \;
      - cd /srv/www/dev
      - docker-compose down
      - docker-compose up -d

- name: prod-ssh
  image: appleboy/drone-ssh
  when:
    event:
      - push
    branch:
      - master
  environment:
    SSH_DEPLOY_KEY:
      from_secret: ssh_deploy_key
  settings:
      from_secret: system_ip_address
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    script:
      - cd /srv/www/prod/public
      - find . -type f -exec chmod 644 {} \;
      - find . -type d -exec chmod 755 {} \;
      - cd /srv/www/prod
      - docker-compose down
      - docker-compose up -d

Here’s the explanation of what stupid things I’m doing here with the scripts, once it connects to my server, via SSH:

  • cd /srv/www/prod/public - Using prod as an example here, we’re changing the directory to the public dir inside of prod. This is where the prod-scp step uploaded all of my files already.
  • find . -type f -exec chmod 644 {} \; - Running your standard find command to change the permissions of all folders to 644.
  • find . -type d -exec chmod 755 {} \; - Running your standard find command to change the permissions of all files to 755.
  • cd /srv/www/prod - Change the working directory to one level up (cd ../ or cd .. doesn’t seem to work in this instance), as this is where our docker-compose.yml file for the server lives.
  • docker-compose down - Bring down the Docker container for the prod site
  • docker-compose up -d - Relaunch the Docker container for the prod site

Now, I say that I’m doing stupid things here because it seems like none of this is really necessary. I’m just replacing files, right? Well, as I’ve found (and I’m not sure if this is super specific to my server), but assuming that the simple thing is going to work was wrong.

If I simply ran docker-compose up -d to restart my existing container/site, Docker would really just completely freak out on me. It didn’t like something about that big mess of files changing (or more likely, me removing the entire public directory… while the container was running). To solve that, I just shut down the container before restarting it again.

And for good measure, changing all of the permissions of the files that we just uploaded helped with the 403/404/501/503 errors that nginx would occasionally serve up. I assume those errors were also related to my complete removal of the site source folder while the container was running. Oops.

In reality, running these six actions this way ON the server only takes at most, a few seconds. Everything comes back up normally now, even with the newly created public folder. If I had a high traffic site, I would find a better way to do the whole upload/deployment process, but for now this works for me.

Likely, at some point I’ll come back and optimize everything in my pipeline. There are definitely a lot of ways that I can make it more efficient. Introducting the drone-volume-cache alone made my build up to 50% faster every single time. A folder that’s 100+mb with a ton of small images takes longer than you’d think to process. Keeping these files (and the files that bundler installs) in a persistent cache was a no-brainer. I just wish that I had done it sooner.

Continued in Part Six »