Does Varnish Make Jekyll Sites Faster?

I started reading about Varnish a while ago and decided to give it a go on my server. I was curious to see if it would do anything to speed up delivery of my site even though it’s built using Jekyll, so is completely static. I only realised once I’d set it up (which took longer than expected do to the site using SSL) that I could notice a difference myself, and would have to jiggle some stuff around so that I could performance test it. I’ll explain how Varnish works and how I tested it’s performance.

How it works

Varnish sits on your server (or another server if you want) and handles all incoming HTTP requests. The requests are then forwarded to your web server of choice (Apache for me but others, such a Nginx, exist) and delivered to the person requesting the files, unless there is a copy available in Varnish’s cache. If so, the request to Apache is skipped and Varnish instead returns it’s own copy of the file. This can lead to some amazing performance benefits when the files are dynamically generated (via PHP scripts, for example). With static files it is less so.

One complication that arises is the use of HTTP/2 and SSL. Neither technology is supported by Varnish, so another service is required to handle incoming HTTPS requests. For this I had to install Nginx, and forward the requests through to Varnish, which then forwards through to Apache:

HTTPS Request > Nginx > Varnish > Apache

As you can imagine, it’s a complete pain to configure. I had to find several guides online to get it all working, but I struggled on and managed to get everything configured correctly. To be fair, part of the problem I had was that the guides I followed were written up for different Linux distros and, since I’m using Debian Testing, I ran into some pretty annoying discrepancies. But regardless, it works, so I can move on to testing.

How I tested

To test the performance of Varnish compared to Apache I had to update my Apache configuration. Rather than change the current settings I just copied the configuration file for www.matthewsimpson.net and changed the listening ports. To set up Varnish I set Apache to only listen to requests coming from 127.0.0.1 on port 8080, so no incoming requests would reach it. Varnish listens on port 80 and passes through to 8080, so external access is not necessary.

With the ports open I needed a good tool for testing. A quick search came up with ab, available from the apache2-utils package. Using ab I could request a page a given number of times and run as many of them concurrently as I needed. I decided to do 1000 requests and set the concurrency to 10. Hopefully this should be enough requests to weed out any unusual results and get a fairer average speed.

Here’s the complete output from ab, which I’ll summarise below:

HTTP Via Apache

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.matthewsimpson.net (be patient)


Server Software:        Apache
Server Hostname:        www.matthewsimpson.net
Server Port:            8444

Document Path:          /
Document Length:        10919 bytes

Concurrency Level:      10
Time taken for tests:   8.933 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      11215000 bytes
HTML transferred:       10919000 bytes
Requests per second:    111.95 [#/sec] (mean)
Time per request:       89.328 [ms] (mean)
Time per request:       8.933 [ms] (mean, across all concurrent requests)
Transfer rate:          1226.07 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       36   43   7.3     41     111
Processing:    39   46   4.9     45      81
Waiting:       36   43   4.9     42      80
Total:         77   89   9.5     87     160

Percentage of the requests served within a certain time (ms)
  50%     87
  66%     89
  75%     91
  80%     92
  90%     98
  95%    104
  98%    118
  99%    137
 100%    160 (longest request)

HTTP Via Varnish

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.matthewsimpson.net (be patient)


Server Software:        Apache
Server Hostname:        www.matthewsimpson.net
Server Port:            80

Document Path:          /
Document Length:        10919 bytes

Concurrency Level:      10
Time taken for tests:   9.049 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      11268690 bytes
HTML transferred:       10919000 bytes
Requests per second:    110.51 [#/sec] (mean)
Time per request:       90.490 [ms] (mean)
Time per request:       9.049 [ms] (mean, across all concurrent requests)
Transfer rate:          1216.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       35   44   7.4     42     127
Processing:    38   46   8.2     45     129
Waiting:       36   44   8.2     42     126
Total:         75   90  12.7     87     204

Percentage of the requests served within a certain time (ms)
  50%     87
  66%     91
  75%     93
  80%     95
  90%    100
  95%    105
  98%    128
  99%    154
 100%    204 (longest request)

HTTPS Via Apache

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.matthewsimpson.net (be patient)


Server Software:        Apache
Server Hostname:        www.matthewsimpson.net
Server Port:            8443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128

Document Path:          /
Document Length:        10919 bytes

Concurrency Level:      10
Time taken for tests:   17.205 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      11252000 bytes
HTML transferred:       10919000 bytes
Requests per second:    58.12 [#/sec] (mean)
Time per request:       172.050 [ms] (mean)
Time per request:       17.205 [ms] (mean, across all concurrent requests)
Transfer rate:          638.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      111  128  17.9    123     291
Processing:    37   43   8.2     42     133
Waiting:       34   41   8.2     39     131
Total:        149  172  22.8    166     379

Percentage of the requests served within a certain time (ms)
  50%    166
  66%    171
  75%    175
  80%    178
  90%    194
  95%    205
  98%    220
  99%    289
 100%    379 (longest request)

HTTPS Via Varnish/NGinx

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.matthewsimpson.net (be patient)


Server Software:        nginx/1.10.1
Server Hostname:        www.matthewsimpson.net
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128

Document Path:          /
Document Length:        10919 bytes

Concurrency Level:      10
Time taken for tests:   16.722 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      11321010 bytes
HTML transferred:       10919000 bytes
Requests per second:    59.80 [#/sec] (mean)
Time per request:       167.219 [ms] (mean)
Time per request:       16.722 [ms] (mean, across all concurrent requests)
Transfer rate:          661.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      109  124  13.5    121     211
Processing:    38   43   3.6     42      62
Waiting:       37   42   3.5     42      62
Total:        148  167  14.7    163     257

Percentage of the requests served within a certain time (ms)
  50%    163
  66%    167
  75%    170
  80%    172
  90%    178
  95%    184
  98%    234
  99%    240
 100%    257 (longest request)

That’s a lot of output, here’s a summary of the mean response times (in milliseconds) for each test:

HTTP

ApacheVarnish
89.32890.490

HTTPS

ApacheVarnish/Nginx
172.050167.219

As you can see, the difference in response times are negligible. The Varnish/Nginx setup for HTTPS requests is marginally faster but I’m inclined to think this is just good fortune on my part. I’ve compared the connection details for both HTTPS configurations on Webpagetest.org and noticed that Apache requests need to do an extra requests to an external Certificate Authority, so I think that I must not be serving the full chain correctly or something. I’m not too sure at this point in time.

Overall, the Varnish/Nginx configuration for serving these Jekyll sites isn’t really worth it. Configuring three services to work together is hard work, plus I had to reconfigure my Letsencrypt certificates to ensure that Apache didn’t get messed up every 90 days (maybe I didn’t have to do this but I wasn’t going to take any chances). Additionally, since Varnish doesn’t natively support HTTP/2 or SSL I’m not too enthralled with the setup. Even though Nginx supports HTTP/2 I’m not convinced that passing this through to Varnish will work as efficiently as it should. If I was using a CMS on a traditional LAMP stack (or anything that’s being built dynamically) then I’d definitely put Varnish in front of it all, but since I’m only serving static content on my sites it’s just not worth it. You can’t get much faster than direct file access!

UPDATE: It looks like Varnish is messing up my Piwik Analytics, all the visits were being reported as coming from 127.0.0.1! So I’ve removed Varnish and Nginx from my server and set Apache back to port 80/443.