Matthew Simpson.net Does Varnish Make Jekyll Sites Faster?
I started reading about Varnish a while ago and decided to give it a go on my server. I was curious to see if it would do anything to speed up delivery of my site even though it’s built using Jekyll, so is completely static. I only realised once I’d set it up (which took longer than expected do to the site using SSL) that I could notice a difference myself, and would have to jiggle some stuff around so that I could performance test it. I’ll explain how Varnish works and how I tested it’s performance.
How it works
Varnish sits on your server (or another server if you want) and handles all incoming HTTP requests. The requests are then forwarded to your web server of choice (Apache for me but others, such a Nginx, exist) and delivered to the person requesting the files, unless there is a copy available in Varnish’s cache. If so, the request to Apache is skipped and Varnish instead returns it’s own copy of the file. This can lead to some amazing performance benefits when the files are dynamically generated (via PHP scripts, for example). With static files it is less so.
One complication that arises is the use of HTTP/2 and SSL. Neither technology is supported by Varnish, so another service is required to handle incoming HTTPS requests. For this I had to install Nginx, and forward the requests through to Varnish, which then forwards through to Apache:
HTTPS Request > Nginx > Varnish > Apache
As you can imagine, it’s a complete pain to configure. I had to find several guides online to get it all working, but I struggled on and managed to get everything configured correctly. To be fair, part of the problem I had was that the guides I followed were written up for different Linux distros and, since I’m using Debian Testing, I ran into some pretty annoying discrepancies. But regardless, it works, so I can move on to testing.
How I tested
To test the performance of Varnish compared to Apache I had to update my Apache configuration. Rather than change the current settings I just copied the configuration file for www.matthewsimpson.net and changed the listening ports. To set up Varnish I set Apache to only listen to requests coming from 127.0.0.1 on port 8080, so no incoming requests would reach it. Varnish listens on port 80 and passes through to 8080, so external access is not necessary.
With the ports open I needed a good tool for testing. A quick search came up with ab, available from the apache2-utils package. Using ab I could request a page a given number of times and run as many of them concurrently as I needed. I decided to do 1000 requests and set the concurrency to 10. Hopefully this should be enough requests to weed out any unusual results and get a fairer average speed.
Here’s the complete output from ab, which I’ll summarise below:
HTTP Via Apache
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.matthewsimpson.net (be patient)
Server Software: Apache
Server Hostname: www.matthewsimpson.net
Server Port: 8444
Document Path: /
Document Length: 10919 bytes
Concurrency Level: 10
Time taken for tests: 8.933 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 11215000 bytes
HTML transferred: 10919000 bytes
Requests per second: 111.95 [#/sec] (mean)
Time per request: 89.328 [ms] (mean)
Time per request: 8.933 [ms] (mean, across all concurrent requests)
Transfer rate: 1226.07 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 36 43 7.3 41 111
Processing: 39 46 4.9 45 81
Waiting: 36 43 4.9 42 80
Total: 77 89 9.5 87 160
Percentage of the requests served within a certain time (ms)
50% 87
66% 89
75% 91
80% 92
90% 98
95% 104
98% 118
99% 137
100% 160 (longest request)
HTTP Via Varnish
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.matthewsimpson.net (be patient)
Server Software: Apache
Server Hostname: www.matthewsimpson.net
Server Port: 80
Document Path: /
Document Length: 10919 bytes
Concurrency Level: 10
Time taken for tests: 9.049 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 11268690 bytes
HTML transferred: 10919000 bytes
Requests per second: 110.51 [#/sec] (mean)
Time per request: 90.490 [ms] (mean)
Time per request: 9.049 [ms] (mean, across all concurrent requests)
Transfer rate: 1216.12 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 35 44 7.4 42 127
Processing: 38 46 8.2 45 129
Waiting: 36 44 8.2 42 126
Total: 75 90 12.7 87 204
Percentage of the requests served within a certain time (ms)
50% 87
66% 91
75% 93
80% 95
90% 100
95% 105
98% 128
99% 154
100% 204 (longest request)
HTTPS Via Apache
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.matthewsimpson.net (be patient)
Server Software: Apache
Server Hostname: www.matthewsimpson.net
Server Port: 8443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Document Path: /
Document Length: 10919 bytes
Concurrency Level: 10
Time taken for tests: 17.205 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 11252000 bytes
HTML transferred: 10919000 bytes
Requests per second: 58.12 [#/sec] (mean)
Time per request: 172.050 [ms] (mean)
Time per request: 17.205 [ms] (mean, across all concurrent requests)
Transfer rate: 638.67 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 111 128 17.9 123 291
Processing: 37 43 8.2 42 133
Waiting: 34 41 8.2 39 131
Total: 149 172 22.8 166 379
Percentage of the requests served within a certain time (ms)
50% 166
66% 171
75% 175
80% 178
90% 194
95% 205
98% 220
99% 289
100% 379 (longest request)
HTTPS Via Varnish/NGinx
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.matthewsimpson.net (be patient)
Server Software: nginx/1.10.1
Server Hostname: www.matthewsimpson.net
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Document Path: /
Document Length: 10919 bytes
Concurrency Level: 10
Time taken for tests: 16.722 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 11321010 bytes
HTML transferred: 10919000 bytes
Requests per second: 59.80 [#/sec] (mean)
Time per request: 167.219 [ms] (mean)
Time per request: 16.722 [ms] (mean, across all concurrent requests)
Transfer rate: 661.15 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 109 124 13.5 121 211
Processing: 38 43 3.6 42 62
Waiting: 37 42 3.5 42 62
Total: 148 167 14.7 163 257
Percentage of the requests served within a certain time (ms)
50% 163
66% 167
75% 170
80% 172
90% 178
95% 184
98% 234
99% 240
100% 257 (longest request)
That’s a lot of output, here’s a summary of the mean response times (in milliseconds) for each test:
HTTP
- Apache: 89.328
- Varnish: 90.490
HTTPS
- Apache: 172.050
- Varnish/Nginx: 167.219
As you can see, the difference in response times are negligible. The Varnish/Nginx setup for HTTPS requests is marginally faster but I’m inclined to think this is just good fortune on my part. I’ve compared the connection details for both HTTPS configurations on Webpagetest.org and noticed that Apache requests need to do an extra requests to an external Certificate Authority, so I think that I must not be serving the full chain correctly or something. I’m not too sure at this point in time.
Overall, the Varnish/Nginx configuration for serving these Jekyll sites isn’t really worth it. Configuring three services to work together is hard work, plus I had to reconfigure my Letsencrypt certificates to ensure that Apache didn’t get messed up every 90 days (maybe I didn’t have to do this but I wasn’t going to take any chances). Additionally, since Varnish doesn’t natively support HTTP/2 or SSL I’m not too enthralled with the setup. Even though Nginx supports HTTP/2 I’m not convinced that passing this through to Varnish will work as efficiently as it should. If I was using a CMS on a traditional LAMP stack (or anything that’s being built dynamically) then I’d definitely put Varnish in front of it all, but since I’m only serving static content on my sites it’s just not worth it. You can’t get much faster than direct file access!
UPDATE: It looks like Varnish is messing up my Piwik Analytics, all the visits were being reported as coming from 127.0.0.1! So I’ve removed Varnish and Nginx from my server and set Apache back to port 80/443.
