Tuesday, October 29, 2019

Secure-ish Workaround for Certbot http validation on Ubuntu

LetsEncrypt is a great tool for securing your non-production resources. I personally use it for securing my Unifi controller in AWS along with my NGINX server for my entertainment services. Certbot only has really two methods of validating the certificate renewal at the time of this writing, DNS and HTTP. DNS can be useful if you wish to update your TXT records frequently (API or manually) but clearly the more convenient is HTTP.

The problem I have with HTTP validation is, well it's HTTP and I do not want to keep an unsecured port open on my Ubuntu server perpetually. How can we keep our server secured in between certificate renewals? This posting will show you the required steps to make modifications to the systemd service for Certbot if you are using firewalld on your server.

If you are using Certbot's built-in systemd methods for renewing your certificate you've got a couple things you are in luck because systemd allows for 'ExecStartPre' and 'ExecStartPost' invocations which we can bookend around the 'ExecStart' of /lib/systemd/system/certbot.service . First we will tell systemd to add a new firewall rule to allow http to the appropriate firewall zone before we run the certbot renew command. After the renew command is finished we'll tell systemd to remove the temporary allowance. Simple right?

1) sudo vim /lib/systemd/system/certbot.service
[Service]
Type=oneshot
ExecStartPre=/usr/bin/firewall-cmd --add-service=http --zone=public ## add this line
ExecStart=/usr/bin/certbot -q renew
ExecStartPost=/usr/bin/firewall-cmd --remove-service=http --zone=public ## add this line

2) reload the daemon
sudo systemctl daemon-reload

3) profit

That's it. Those two lines in the systemd service file will open up http and then close it afterwards. You may test it out by commenting the ExecPostStart line to make sure it adds it in there, reloading the daemon, and then running the certbot.service. After you run the service you should see the public zone still has http listed as an available service
ubuntu@ubuntu-server:~$ sudo firewall-cmd --list-all --zone=public
public
  target: default
  icmp-block-inversion: no
  interfaces:
  sources:
  services: dhcpv6-client http https ssh
  ports: 6789/tcp 32400/tcp 33400/tcp 5050/tcp 8989/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
And if you repeat the process by commenting out the ExecPreStart and allowing the ExecPostStart to execute you'll see the service is gone.

I'm looking forward to getting no more notification emails about a certificate expiring in 20 days without leaving my server exposed in the meantime. Cheers!

Saturday, August 25, 2018

Adding Lets Encrypt to Unifi controller

Hey y'all.

I've had my EC2 t2.micro around for a long time and have always operated with a self-signed certificate. It hasn't bothered me much but it would be nice to get rid of the warning I always receive when logging in. With the advent of CAs such as LetsEncrypt there's really no reason to keep fumbling on the certificate. I had been casually searching on how to work the cert in and FINALLY found a good guide here, Larry Land's tutorial. I'm going to skip some of the information he wrote (he included how to install and setup the controller to begin with) and re-post what I used to get mine going.

Step 1: Generating the signed certificate with Lets Encrypt

Lets install Lets Encrypt now. Reminder that this needs to be done on this server, not your local machine. We’ll be using certbot and essentially the instructions there.
wget https://dl.eff.org/certbot-auto
chmod +x certbot-auto
./certbot-auto
That last line will configure certbot and also install some dependencies.
Now, using certbot, we generate the signed certificate. So lets run the wizard:
./certbot-auto certonly
Select option 2 (to use a temporary webserver), then enter your email (so you get alerts if things go wrong), agree to the agreement, then finally type in your domain name (along with the subdomain). If everything went well you should get a Congratulations message.

Part 2: Load the certs into the services

The Ubiquiti services are Java-based and they use the Java Keystore as a way of storing the private keys and certificates. We first need to generate a PKCS #12 certificate from the raw ones we just received:
sudo openssl pkcs12 -export -inkey /etc/letsencrypt/live/mysubdomain.mydomain.com/privkey.pem -in /etc/letsencrypt/live/mysubdomain.mydomain.com/fullchain.pem -out /home/ubuntu/cert.p12 -name ubnt -password pass:temppass
Again, don’t forget to replace mysubdomain.mydomain.com with your domain name. Everything else can remain as-is.
Now for each service you’ll need to load the PKCS #12 certificate into its own keystore.
  • Unifi:
    sudo keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /var/lib/unifi/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias ubnt -noprompt
Basically all that’s different is the keystore location of the service, and the password Ubiquiti uses to protect it.
Finally, delete the PKCS #12 files (since they’ve already been imported), and restart the services (as appropriate)
sudo rm /home/ubuntu/cert.p12
sudo /etc/init.d/unifi restart
Certificates must be renewed every (3) months. What else do you think crond is for :) I put the script into /etc/cron.monthly/ as that is often enough.

Part 3: Automating Lets Encrypt certificate renewal

As mentioned before, Lets Encrypt certificates only last 3 months. As such, we’ll need to get this machine to attempt to renew the certificates monthly and then place the new certs back into services. It’s essentially doing parts C and D on a scheduled job using cron. The renewal will fail fast if no renewal is necessary.
Create a new file /etc/cron.monthly/renew_lets_encrypt_cert.sh and customize it according to what you used in Parts C and D. No sudo needed since cron will run it automatically as a super user. Use full paths to files. Here’s an example:
# Get the certificate from LetsEncrypt
/home/ubuntu/certbot-auto renew --quiet --no-self-upgrade

# Convert cert to PKCS #12 format
openssl pkcs12 -export -inkey /etc/letsencrypt/live/mysubdomain.mydomain.com/privkey.pem -in /etc/letsencrypt/live/mysubdomain.mydomain.com/fullchain.pem -out /home/ubuntu/cert.p12 -name ubnt -password pass:temppass

# Load it into the java keystore that UBNT understands
keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /var/lib/unifi/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias ubnt -noprompt

# Clean up and use new cert
rm /home/ubuntu/cert.p12
/etc/init.d/unifi restart
Make sure the script is executable:
sudo chmod +x /etc/cron.monthly/renew_lets_encrypt_cert.sh
Let's check back next month and ensure everything renewed itself properly.

Wednesday, March 21, 2018

EdgeRouter recovery from start to finish on a Mac

The Ubiquity EdgeRouter Lite (ER3-Lite) router is a very capable router I have grown to lean on for most of my customer installs. It has a great value, good community support, and has a lot of features - even more if you dive into the CLI -  which business customers can take advantage of. I've covered a few other configuration walk throughs on here so I won't go into any detail on what it can do or why I like it.

The point of this post is to put together from start to finish how to rebuild your ER3-Lite router if the USB storage goes south and you have a Mac computer at your disposal. I will provide links back to the original instructions in the post so you can go find more information if there is a bit I either glaze over or you want to follow up on more of the solution. The reason I'm writing this one myself is to give credit to those who have come before me and to bring multiple posts into one spot for someone else.  Ubiquity community EdgeMax rescue thread: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

There are two pieces of extra equipment you're going to need if you follow these instructions plus one compatible USB storage drive: 1 serial console cable (sometimes called a rollover cable) and a USB-Serial adapter. For my use I had a couple console cables left over from Cisco switches and a Keyspan USB-Serial adapter already. I had good luck and good value/performance with Kingston Digital 16GB DataTraveler SE9 USB 2.0 Flash Drive Pack of 2 (KW-U4616Z02-8A). There are others out there which can be used but your success may not be guaranteed. This particular model works well and has been proven many times over. If this post is more than a few months old, the mentioned storage drive is no longer available, and you're looking for guidance on what to buy I suggest you reference back to the original Ubiquity community thread for what others are using. If I run out of stock of the drives and find another I'll update this post.

*** NOTE *** Previously unusable drives like the CruzerFit drives from Sandisk are confirmed to be usable if you follow the last bit of this post where you modify the bootcmd of the router to include a usb reset and a couple sleeps.

First, we need to install the USB-Serial drivers and then get the console to work via terminal. After you've installed the appropriate drivers for your USB-Serial adapter lets drop onto the CLI via Terminal and look at what device you have. The device is going to be listed under /dev/cu.something. Mine happens to have Keyspan in the name. The command we use to execute the connection will be:

screen /dev/cu.KeySerial1 115200

I suggest having the cable already inserted into your powered up EdgeMax and the new storage drive installed before executing the screen command. Once the screen command is showing you a prompt you should see something like this:

Octeon ubnt_e100#

You're ready to start the building process BUT we still need to fire up an FTP server on the Mac and serve up the image for tftpbooting. Download the latest available EMRK from here and place it into /private/tftpboot/.

open /private/tftpboot/ 

sudo chmod -R og+rX /private/tftpboot/
sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist
sudo launchctl start com.apple.tftpd


## To shut down tftp server
sudo launchctl unload -F /System/Library/LaunchDaemons/tftp.plist

Once you get your file downloaded and tftp server running keep following the instructions on the original thread, EdgeMax rescue kit (now you can reinstall EdgeOS from scratch). I preferred to plug my eth0 into my existing bench network (which allowed it to get out to the internet) and the console cable into the console port. This allowed me to also use dhcp from the bench network and to pull down the newest available EdgeRouter image from the internet. My commands were as follows:

dhcp
set bootfile emrk-0.9c.bin
set serverip <machost IP>
tftpboot;bootoctlinux $loadaddr

Once the file successfully pulled from my Mac's TFTP server and loaded up I was able to find myself looking at the EMRK loading away. I again used DHCP to configure eth0 which again made things easier for me. When at the EMRK> prompt I suggest deleting all existing partitions via the parted command. I wasted a little bit of time not doing this, expecting the EMRK to do it for me. It may work for others but what I'm giving you advise on will work every time.

parted -s /dev/sda rm 1
parted -s /dev/sda rm 2

Now that it's completely clean we can let the emrk-reinstall command do its magic. Before we continue lets get the URL from Ubquity to the newest available firmware release, Ubiquity Networks Downloads ER3-Lite. Save the URL in your clipboard as we will need it shortly. Upon execution of the emrk-reinstall script it will shortly ask for the URL to the firmware. The underlying execution uses a curl command to pull it in. Since the URL from Ubiquity is 'https' we must add '-k' to the beginning of the supplied URL so it will ignore the certificate error. I'm including an example below:

kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Enter EdgeOS image url: -https://dl.ubnt.com/firmwares/edgemax/v1.10.x/ER-e100.v1.10.1.5067571.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 68.3M  100 68.3M    0     0  8319k      0  0:00:08  0:00:08 --:--:-- 6318k

This should finally result in a finished image. If it does not please consult the original links for further troubleshooting. If you wish there are also tutorials on how you can expand the install script from the original 2GB partition to use the full size of your drive. I did this but it's outside the scope of this post.

You're ALMOST ready to say you're finished with this one but there's one more gotcha I'm going to post here to make this a 100% post. The USB storage available to us seems to be of different type than the ones used by the OEM. This can result in the router not correctly initilizing the storage upon boot reliably. You may find yourself sitting at a prompt as if all of your work was for nothing the first time the power is re-applied to the router. (It happened to me and is why I'm writing all of this for the community.) The last bit you need is here, List-of-Compatible-USB-drives post 24. Edit the bootcmd to allow the usb storage device to be reset before being accessed by the boot process. If this is done it'll fix the problem with dissimilar storage devices from a non-OEM. First set environment variable bootcmd with this command:

Octeon ubnt_e100# setenv bootcmd sleep 1\;usb reset\;sleep 1\;$(bootcmd)
Note all this must be on single line !
Next we must be sure that changes are accepted with this command:

Octeon ubnt_e100# printenv
Octeon ubnt_e100# saveenv

The router should now boot reliably. That's it! There are a few other shortcuts to all of this process made possible through other script writers but they are all doing the same thing as I've recounted here. The problem with always working through someone else's shortcuts is you never really know how to fix something on your own. Certain exceptions here are the creation of a tftpboot image and the tool itself we used, EMRK. I didn't think this should be something I should learn how to do on my own as it was going to be well outside the scope of anything I would be looking to accomplish. I hope this helps someone else. Not all of these links and procedures were in one place that I found over the course of a few days but now they are. Enjoy!

Friday, September 30, 2016

Update on using squidGuard Webproxy for Ubiquity

For those who have read my previous blog post about getting my feet wet with the webproxy package installed on our EdgeOS based routers I have some updated notes to help out with a few things I didn't find well documented and readily available on the internet.

1) how to un-disable the webproxy url-filtering.
   If you needed to disable the webproxy for one reason or another it's not that difficult. The command to do so is built into the auto-complete command set:
  configure
 set service webproxy url-filtering squidguard disable
 commit

What isn't documented very well in my searching was how to UN-disable it after you've made your changes and are ready to get the webproxy back online. It took me some thinking on my own, as I'll often do when stuck on a problem, but my logic was sound and I had nothing to lose.
  configure
 delete service webproxy url-filtering squidguard disable
 commit

Success! The service was back online and I didn't have to rebuild my configuration.

2) how to save your /var/log/ mount point after turning on DPI.
After enabling DPI, outlined in my other post, a lot of logging is turned on. Logging is good if you're wanting to do some checking on your traffic but this gets excessive when you have a busy router. Your /var/log/messages logs will overrun the mount point quickly of some housekeeping isn't done to curb this. Via the GUI you'll want to go to your rulesets and unchecked logging for all of your accept rules. If you are special and you have a server to accept all of your logs for you need you not worry.

3) Making a separate proxy profile for separate subnets
Guest Wi-Fi subnets rarely need the same allowances as an internal network may so if a guest complains about not being able to get to break.com is not a big deal. However, maybe someone internal needs it. What do you do?

You could make exceptions for the entire router for individual sites but over time the list could grow and then you'd be opening it up for everyone. Lucky for you source-groups are able to handle the task. With a source group you can leave the default allows and blocks on for all but make changes by exception source-groups.

set service webproxy url-filtering squidguard source-group guests address 192.168.2.2-192.168.2.254
set service webproxy url-filtering squidguard rule 10 source-group guests
set service webproxy url-filtering squidguard rule 10 allow-category adult

Once this has been committed and saved you now have allowed the adult category, which is what category break.com belongs to, sites for those IPs in the internal network. This same method can be utilized for other exceptions and restrictions.

If you have questions or comments please leave them below.

Ubiquity EdgeOS Deep Packet Inspection engine (DPI)

Recently I wrote a blog about how to use the built in package squidGuard to block content passing through EdgeOS routers. There is a more efficient way to block some categories though, DPI.

DPI is hardware offloaded and does not drag your CPU down as much on a busy router. The number of categories you can block are limited though:

Business                                  TopSites-Games
Bypass-Proxies-and-Tunnels  TopSites-Health
File-Transfer                           TopSites-Home
Games                                     TopSites-KidsnTeens
Instant-messaging                   TopSites-News
Mail-and-Collaboration          TopSites-Recreation
P2P                                         TopSites-Reference
Remote-Access-Terminals     TopSites-Regional
Security-Update                     TopSites-Science
Social-Network                     TopSites-Shopping
Stock-Market                        TopSites-Society
Streaming-Media                  TopSites-Sports
TopSites-Adult                      Voice-over-IP
TopSites-Arts                        Web
TopSites-Business                 Web-IM
TopSites-Computers

Most notable of the categories I like to use are the Bypass-Proxies-and-Tunnels and P2P. With making a separate firewall rule set for these two you can keep casual users from hopping on your wifi and downloading illegally data you would be left holding the bag for. By Turing on the DPI feature you can also see what protocols and categories are most used on your network by your top talkers.

With the below lines of configuration I can say it successfully has caught and stopped users from downloading torrents on a particular network I manage.

set system traffic-analysis dpi enable
set system traffic-analysis export enable
set firewall name DROP_SITES default-action accept
set firewall name DROP_SITES description "Drop Junk"
set firewall name DROP_SITES enable-default-log
set firewall name DROP_SITES rule 10 action drop
set firewall name DROP_SITES rule 10 application category P2P
set firewall name DROP_SITES rule 10 log enable
set firewall name DROP_SITES rule 10 description ‘P2P’
set firewall name DROP_SITES rule 15 action drop
set firewall name DROP_SITES rule 15 application category Bypass-Proxies-and-Tunnels
set firewall name DROP_SITES rule 15 log enable
set firewall name DROP_SITES rule 15 description Bypass-Proxies-and-Tunnels
set interfaces ethernet eth0 firewall in name DROP_SITES

After setting it all up you'll be able to see the rules stats counting up as traffic is passed. Logged drops are put into the default /var/log/messages log. As it is put into the general log I strongly suggest you turn compression on for  logrotated. Otherwise/var/log will fill up quickly lest you send all of your logs to a remote server.

Thursday, August 11, 2016

WSUS Server crapped the bed after installing KB3159706

I have several domain controllers I manage for local businesses and most of them I run WSUS on the hyper-v host. Lately I've had a run in with KB3159706 which was not pleasant. After it's installed I get the following error in Windows Event Viewer: 
"Login failed for user 'NT AUTHORITY\NETWORK SERVICE'. Reason: Failed to open the explicitly specified database 'SUSDB'. [CLIENT: <named pipe>]"

If you look on the internet for a bit you'll find several solutions saying to remove the referenced KB. I say nay to that.
Basically the steps boil down to:
  1. Open an elevated Command Prompt and run "C:\Program Files\Update Services\Tools\wsusutil.exe" postinstall /servicing
  2. Enable HTTP Activation under .NET Framework 4.5 Features in the Server Manager Add Roles and Features Wizard
  3. Restart the WSUS service
No need to uninstall KB3159706.
Source: https://social.technet.microsoft.com/Forums/windows/en-US/233a6b37-5919-4a9e-9538-aa0e7bd7111a/wsus-console-service-cant-start-reason-failed-to-open-the-explicitly-specified-database?forum=winserverwsus