Nginx logs with GoAccess on a Digital Ocean Droplet with Ubuntu 16
I've recently gotten a Digital Ocean Droplet (aka virtual machine, just like AWS EC2) to mess around with doing all the magic stuff that usual hosting providers such as Heroku do for you. Automatic deploys, web servers, etc.
Right now I'm at a point where I'm hosting this blog as a Rails app, deployed automatically with Capybara. I also host jonandjess.studio as a static website. It's a Nextjs app (I wanted to test their export function) which gets deployed with a simple
yarn deploy which runs
next export then
scp the build folder in the correct folder on my Droplet (
It has an nginx config that allows to route calls to jonandjess.studio to its index.html file.
To be honest, just doing that was pretty eye opening to me on how things work.
I get amazed by this pretty quick, which is fun. For example, I'm working on a V2 for jonandjess.studio. I wanted Jess to be able to access the progress so she can give feedback, but it gets annoying when she randomly bugs me to see what I have so far. So now I'm just deploying the V2 at
/var/www/jonandjess/v2 and added an nginx config to route there when the url is jonandjess.studio/v2. Brilliant! (I know, I know...).
Anyways, now that we're at this point, I wanted to look into fun, simple monitoring solutions, because now that I can play around with the Droplet and nginx configs, I felt like understanding thing like Elastic Search would be easier (I'm not there yet, but soon!).
So the first solution I just installed, which was amazingly easy to do, is goaccess, an open source log visualizer.
By default Nginx logs every calls it gets at
/var/log/nginx/access.log (you can also override this path, and the format in the nginx config). I wanted to use GoAccess to have a visualisation of calls for jonandjess.studio.
setup goaccess on Ubuntu
First, you need to install goaccess.
sudo apt-get install goaccess
will do what it says (it is always recommended to run
sudo apt-get update first to make sure you install the latest version of the package).
Once you have that, goaccess has an easy CLI with a lot of options to run it (it can also have a config file, my case was simple enough for me not to use one for now).
To run the CLI version, which looks great, you can run
goaccess access.log -c
access.log is the path for your logs, so if you wanted to run this for your nginx logs from any directory, you'd pass it
That is nice, but this would imply that I have to ssh into my Droplet each time. I really wanted a web-based dashboard, which GoAccess offers, at no additional cost! (FREE!).
Web-based goaccess dashboard
You can pass a
-o /var/www/html/report.html option to the command line, which will output the dashboard as an html file.
Combine this with
--real-time-html option and your html file will get automatically updated + goaccess will use websockets to connect your browser and offer live reloads.
Speaking of the websocket, it uses port 7890, so you'll have to open this port with your firewall. Opening a port on Ubuntu is done with
sudo ufw allow 7890
Now that your port is open, you can run the goaccess command with all previously stated options.
I wanted to report to be accessed at jonandjess.studio/report.html (go check it out!) so my final command is
goaccess /var/log/nginx/access.log -o /var/www/jonandjess/html/report.html --log-format=COMBINED --real-time-html
This is nice, but then do I just CMD+Q my terminal hoping for this to keep running always? It doesn't seem too stable, so I decided to make it a service controlled by
systemctl so it can restart automatically etc.
Add goaccess to systemd
systemd allows you transform scripts into services, which it will then take care of, according to a list of rules you set (restart when it fails, start when the server boots, etc). You can also easily control your service with commands such as
systemctl start myservice, instead of using things such as
lsof to find which process it is running on and killing it, etc.
So first, we need to put our command into a file so it can be ran by systemd. I created my script in
#!/bin/bash goaccess /var/log/nginx/access.log -o /var/www/jonandjess/html/report.html --log-format=COMBINED --real-time-html
The script will get executed, so you need to make this file executable with
chmod +x /usr/local/bin/goaccessreport
Ok, now we have an executable which starts our goaccess live web dashboard (you can test by typing
/usr/local/bin/goaccessreport and it should start the web dashboard.
Last thing we need to do is create a service file for systemd, which tells systemd what to do with it. We can create the file at
[Unit] Description=Goaccess Web log Report For Site Example.com After=network.target [Service] Type=simple User=root Group=root Restart=always ExecStart=/usr/local/bin/goaccessreport StandardOutput=null StandardError=null [Install] WantedBy=multi-user.target
I'd invite you to learn more about systemd - it's a little confusing how it works, but when you get a hold of it, it's great and very useful!
And... that's it! Now we can run
systemctl enable goaccessreport.service systemctl start goaccessreport.service
to enable and start our service. jonandjess.studio/report.html is up and running. The
Restart=always directive in the service file will ensure that systemd will always keep the service up (either when the Droplet reboots, or the script fails, etc).