Shell scripting for website build automation

Last year I started a new job and completed a switch over to Mac OS as my development environment. As part of the transition I am embracing working with the command line. Among other benefits, using the command line allows executing scripts which automate repetitive tasks. In this post I will go through a command line shell script that I wrote to automate the build process for a simple website.

While updating my personal website recently, I realised that I was performing the same menial tasks over and over. For instance, whenever I changed any CSS, I had to minify the file and alter the filename to prevent the cached version of the file from being loaded. Another task which was particularly tedious involved images: I serve each image in a different size depending on browser window width. So each time I added a new image to my site I used Gimp to create different sizes by resizing the original image! This felt extremely inefficient (and very “unprogrammer”-like), so I knew it was time for automation.

Why I decided not to rely on readily available tools

There are numerous tools available which aim to automate the build process in web development projects. These range from (somewhat) simpler ones such as Gulp to more complex and powerful systems like Webpack. However I ruled out using any of them since they all felt too complex for my needs. My website is a simple single page and ultimately there are just not many steps in the build process. Besides this was a perfect opportunity for me to dive into writing my own shell scripts :) By writing my own script I was able to keep the complexity down while relying on separate tools for each individual task.

Setup and shell script basics

To keep things clean I created two directories: src contains all source files that I actually edit. dist is the output directory into which the script places all processed files. I then created the script file itself called build (without a file extension) and placed it in the root directory of my project, outside of src and dist.

The first line of the script has the following text:

#!/bin/bash

This tells the command line interpreter to use Bash when the file is executed. This way I can navigate to my project directory in the terminal and run the script by entering:

$ build

I had to add my project directory to the $PATH variable for the command to be executable from my project directory.

And now I’ll go over the actual tasks!

Script tasks

Clean and move files

When I make a change to source files, I need these changes to show up in the output. So the first task before any processing is to copy all files over from src to dist. This is accomplished by deleting the dist directory and its contents, creating an empty one again and then copying all files from src to dist:

rm -rf dist && mkdir dist
cp -a src/. dist/

Now that all files are copied it’s time to process them.

Image sizes

For each image on the site I include 3 different sizes. Only one of these gets loaded depending on the user’s browser width - this is done using the srcset attribute of the img tag. If browser width is small (for example on a mobile device), then a smaller image is loaded which decreases site load time.

First I declare all required image sizes in a variable called sizes (the -a flag defines it as an array with multiple values):

declare -a sizes=('710' '1000' '1420')

I want to go through every image and for each one to create three different sizes. To access every size of every image I loop through all images (they are all found in the img directory) and then through the defined sizes (note the use of double quotes in "${sizes[@]}" to evaluate the variable):

for image in dist/img/*; do
  for size in "${sizes[@]}"; do
  # do something for each size of each image
  done
done

For each size I create a new file with a unique name. To do this I get the length of the original filename and save it as fnamelength. Using fnamelength I construct the new filename by inserting the size just before the file extension:

fnamelength=${#image}
outputimage=${image:0:(fnamelength-4)}-${size}${image:(fnamelength-4):fnamelength}

So if the original filename is image.png and size is 710 the output filename would be image-710.png. Now I create a file with the new name by duplicating the existing image:

cp $image $outputimage

At this point I have a new file, but it is just a copy of the original one, so now I need to actually resize it. To do this I use the sips image manipulation utility available on Mac OS. I specify filename of the file to be processed, desired output width and use the -Z flag, which instructs aspect ratio to be retained:

sips $outputimage -Z $size

Finally after I’ve created all sizes for an image and before I move onto the next one in the loop, I delete the original image as it is not required by the site:

rm $image

Putting it all together, this is how the script creates images of different sizes:

declare -a sizes=('710' '1000' '1420')

for image in dist/img/*; do

  for size in "${sizes[@]}"; do
    fnamelength=${#image}
    outputimage=${image:0:(fnamelength-4)}-${size}${image:(fnamelength-4):fnamelength}

    cp $image $outputimage

    sips $outputimage -Z $size
  done

  rm $image
done

Optimise images

In addition to creating images of different sizes I also want to optimise them to decrease site load time.

Image optimisation can be a slow process since it is a demanding task for the CPU. Sometimes when working on my site I want to quickly test things out locally before building a final version to be uploaded. In this situation I want the build script to run as quickly as possible. For this purpose I created a condition in the script which checks if the -d flag (which stands for “development”) has been set when running the script. To execute the script with the flag I need to run:

$ build -d

Image optimisation only takes place if the flag has not been set:

if [[ $1 != '-d' && '$1' != '' ]]; then
  # optimise images
fi

I use imagemin to optimise images. Like other external libraries I use in this script, I have imagemin installed as a Node.js module globally using npm. The npm package name for using imagemin from command line is imagemin-cli. This makes it available anywhere when using the terminal and therefore accessible to the build script.

When using imagemin from the command line it is not possible to directly optimise an image - a new output file or directory has to be specified instead. As a workaround I create a new temporary directory called imgmin. I then look for all existing files in the img directory and optimise them with imagemin, redirecting the output to the imgmin directory. I then delete the original img directory a create an empty one with the same name, copy all files from imgmin and finally delete the temporary directory:

mkdir dist/imgmin
imagemin dist/img/* --out-dir=dist/imgmin
rm -rf dist/img && mkdir dist/img
cp -a dist/imgmin/. dist/img
rm -rf dist/imgmin

Images are now optimised!

Autoprefixer

When writing CSS I want to ensure maximum browser compatibility. To achieve this, some CSS properties need to be specified with vendor prefixes which target older browsers. A great tool to automate this process is called Autoprefixer. To use it, I have installed the PostCSS npm package (the package name is postcss-cli). Autoprefixer processes my CSS file in the build script with the following command:

postcss --use autoprefixer -r dist/css/style.css

Autoprefixer needs to know which browsers I want to target. To do this I created a file called .browserlistrc in the root directory of the project with the following content:

Last 2 versions

This tells Autoprefixer to target the last 2 versions of every browser.

Minify CSS and clear cache

The last two tasks are to minify the CSS and to deliver the latest version of the CSS file to the user instead of the one cached by the browser.

To minify CSS I use clean-css which is available as clean-css on npm:

cleancss -o dist/css/style.min.css dist/css/style.css

To clear the cache I update the name of the CSS file by prepending the current timestamp to it:

timestamp=$(date +%s)
mv dist/css/style.min.css dist/css/$timestamp.style.min.css

References to the new CSS filename also need to be updated in all HTML files. For that I use the replace package (replace on npm) and search for the relevant string in HTML files to replace it with the new one:

replace 'css/style.css' 'css/'$timestamp'.style.min.css' dist/index.html
replace 'css/style.css' 'css/'$timestamp'.style.min.css' dist/404.html

Finally I remove the non-minified CSS file, since it is no longer required:

rm dist/css/style.css

And that’s it! All tasks are now complete and every time I alter the contents of my site in the src directory, I simply need to run build from the root project - the site will be automatically generated for me.

Conclusion

I wanted to automate repetitive tasks when updating my personal website and instead of using an existing tool I wrote a custom shell script to accomplish this. Although at times Bash syntax felt somewhat quirky compared to programming languages that I’m used to, I think that shell scripting can be valuable for simpler tasks.

Here is the complete script for reference.

It certainly feels like this script can be optimised and simplified, so I will be improving it as I continue learning my way around the command line.

Do you have any comments or questions? Let me know on Twitter!