In my previous post about upgrading to Elastic 8, I signed off with the promise of sharing how I put Kibana behind an nginx proxy. Here’s the post on how I did that, and what I did to make it work after a few hours of messing around with various settings. If you want the TL;DR on the biggest lesson learned, it is “Delete your cookies!” See below for why that matters.
First, a brief reminder of the defaults for Kibana: if you don’t change anything, Kibana will listen on port 5601 on unencrypted HTTP. While that’s fine for testing, passing things like passwords over unencrypted HTTP isn’t a good idea security-wise. Plus, Edge browser won’t offer to save your username and password to log in automatically, nor should it because it is insecure! This was a hassle to me that I wanted to fix. To solve this, you can either enable TLS directly in Kibana, or you can put it behind a proxy like nginx and terminate TLS there.
If you do want to enable it directly in Kibana, there’s good documentation available. Basically, create a private key and certificate, configure it in kibana.yml, set “server.ssl.enabled” to “true” in the kibana.yml file, and you are good to go. This enables TLS, but still on the default port of 5601.
If you want to move ports though, you can’t just set Kibana to listen on port 443. In *nix systems, only root can bind to privileged ports, defined as ports below 1024. Yes, you can do things to make Kibana bind to port 443, but another option is to use a web server that’s already set up to listen on port 443, such as nginx, and use it as a reverse proxy.
With this setup, nginx will listen on port 443 and terminate the TLS connection, and pass traffic to port 5601 unencrypted so Kibana can respond. It’s important to make sure Kibana is only listening on localhost and not a public interface so traffic has to go through the proxy to ensure that there are no non-local unencrypted sessions being created.
Besides the benefit of not having to mess with Kibana to allow it to bind to port 443, another benefit of using a reverse proxy is that it can proxy to many apps. For example, you can direct “https://FDQN/kibana” to the Kibana app, and “https://FQDN/elastic” to the Elastic HTTP listener if you want. Fortunately, Kibana can handle this “server.basePath” configuration item, so it knows that URLs forwarded from nginx start with “/kibana” for example (although this is configurable).
Setting it up
My initial stab at getting this to work was to use the following nginx configuration:
server {
# SSL configuration
#
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
ssl on;
ssl_certificate /etc/ssl/certs/kibana.crt;
ssl_certificate_key /etc/ssl/private/kibana.key;
root /var/www/html;
location /kibana {
proxy_pass http://127.0.0.1:5601/kibana;
}
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
}This uses proxy_pass to forward any traffic sent to “/kibana*” to the Kibana server listening on localhost. The relevant part of kibana.yml is here:
server.port: 5601
server.publicBaseUrl: "http://localhost:5601/kibana"
server.host: "localhost"
server.basePath: "/kibana"
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
server.rewriteBasePath: true
This listens on port 5601 on localhost. The “/kibana” prefix is included in the publicBaseUrl and basePath to let Kibana know that URLs will be prefixed and ignore that part. The “server.rewriteBasePath” as stated will rewrite the requests to remove the “/kibana” prefix specified. As stated above, there are two options to configure this: have Kibana rewrite the URLs, or nginx. Here we are having Kibana do it.
After restarting both nginx and Kibana, and going to https://FQDN/kibana, we got the login page. An auspicious start!
However, when I tried to log in, I just got sent back to the login page in a loop. Time for some troubleshooting.
When in doubt, change everything
I first started by swapping the URL rewrites: instead of having Kibana do it, I had nginx do it. I flipped “server.rewriteBasePath” to false in Kibana, and added this line to the nginx config:
rewrite /kibana\/?(.*)$ /$1 break;I was able to see the login page again, but still had the loop. Since that didn’t work and I didn’t want nginx to rewrite things, I put things back to where they were before.
Getting nowhere, I needed more info. The next step was to set Kibana logging to DEBUG in the hopes that the error messages would be a bit more useful. I used the logging documentation to set everything to DEBUG, and found the following:
[2023-04-28T21:19:04.789+00:00][DEBUG][http.server.response] GET /login?next=%2Fkibana%2F 200 61ms - 89.9KB [2023-04-28T21:19:04.877+00:00][DEBUG][http.server.response] GET /node_modules/@kbn/ui-framework/dist/kui_light.min.css 304 29ms [2023-04-28T21:19:04.883+00:00][DEBUG][http.server.response] GET /ui/legacy_light_theme.min.css 304 28ms [2023-04-28T21:19:04.886+00:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized [2023-04-28T21:19:04.887+00:00][DEBUG][plugins.security.basic.basic] Trying to authenticate user request to /bootstrap-anonymous.js. [2023-04-28T21:19:04.889+00:00][DEBUG][plugins.security.http] Trying to authenticate user request to /bootstrap-anonymous.js. [2023-04-28T21:19:04.890+00:00][DEBUG][plugins.security.http] Authorization header is not presented. [2023-04-28T21:19:04.891+00:00][DEBUG][plugins.security.authentication] Could not handle authentication attempt
“Authorization header is not presented” seemed promising, and looking in the code indicated this error exists when the header isn’t present. Some more Googling led me to try adding the authorization headers to the nginx configuration explicitly, plus a few more in the “throw things at the wall” kind of approach:
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-User $http_authorization;
proxy_set_header Authorization $http_authorization;
proxy_pass_header Authorization;
This did not work, which should be no surprise if you remember the TL;DR from above. There I sat, stuck for a while, until I noticed that one of the error messages was coming from the “http.server.Kibana.cookie-session-storage” service. When reading the cookie documentation didn’t provide any obvious answers, I simply did what I should have done at the beginning: I deleted all of the cookies I had stored. Et voilà! I was able to get everything working again.
I’m not entirely sure what happened, but my guess is that since I had some cookies from before I started this whole mess, that was interfering with the login process. “Try deleting your cookies” is step one of website login troubleshooting, which is something I forgot!
Not done yet
While Kibana itself was working just fine, I soon discovered a new problem: Metricbeat was broken. “Of course”, I thought to myself, “the Kibana monitoring module needs updating”. And it did, to be pointed at the new base path, since Kibana was handling it, not nginx:
hosts: ["http://localhost:5601"]
basepath: "/kibana"
You may ask why I didn’t go through the nginx proxy to monitor this? Since Metricbeat is local to the server, there’s no benefit to doing so and it would add an unnecessary hop.
However, after restarting Metricbeat I was still getting errors about Kibana not being accessible. I had forgotten that in addition to the module, the metricbeat.yml config file itself has a Kibana configuration section for all of its dashboards, and it needed to be updated with a slightly different config:
setup.kibana:
host: "http://localhost:5601"
path: "/kibana"
Once that was updated, everything was working as expected and nginx, Kibana, and Metricbeat were all getting along nicely.
I glossed over the part about creating the certificate with my internal PKI. I’ll have the specifics about that in a future post, including a cameo by ChatGPT.