Wednesday, March 21, 2018

EdgeRouter recovery from start to finish on a Mac

The Ubiquity EdgeRouter Lite (ER3-Lite) router is a very capable router I have grown to lean on for most of my customer installs. It has a great value, good community support, and has a lot of features - even more if you dive into the CLI -  which business customers can take advantage of. I've covered a few other configuration walk throughs on here so I won't go into any detail on what it can do or why I like it.

The point of this post is to put together from start to finish how to rebuild your ER3-Lite router if the USB storage goes south and you have a Mac computer at your disposal. I will provide links back to the original instructions in the post so you can go find more information if there is a bit I either glaze over or you want to follow up on more of the solution. The reason I'm writing this one myself is to give credit to those who have come before me and to bring multiple posts into one spot for someone else.  Ubiquity community EdgeMax rescue thread: EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

There are two pieces of extra equipment you're going to need if you follow these instructions plus one compatible USB storage drive: 1 serial console cable (sometimes called a rollover cable) and a USB-Serial adapter. For my use I had a couple console cables left over from Cisco switches and a Keyspan USB-Serial adapter already. I had good luck and good value/performance with Kingston Digital 16GB DataTraveler SE9 USB 2.0 Flash Drive Pack of 2 (KW-U4616Z02-8A). There are others out there which can be used but your success may not be guaranteed. This particular model works well and has been proven many times over. If this post is more than a few months old, the mentioned storage drive is no longer available, and you're looking for guidance on what to buy I suggest you reference back to the original Ubiquity community thread for what others are using. If I run out of stock of the drives and find another I'll update this post.

*** NOTE *** Previously unusable drives like the CruzerFit drives from Sandisk are confirmed to be usable if you follow the last bit of this post where you modify the bootcmd of the router to include a usb reset and a couple sleeps.

First, we need to install the USB-Serial drivers and then get the console to work via terminal. After you've installed the appropriate drivers for your USB-Serial adapter lets drop onto the CLI via Terminal and look at what device you have. The device is going to be listed under /dev/cu.something. Mine happens to have Keyspan in the name. The command we use to execute the connection will be:

screen /dev/cu.KeySerial1 115200

I suggest having the cable already inserted into your powered up EdgeMax and the new storage drive installed before executing the screen command. Once the screen command is showing you a prompt you should see something like this:

Octeon ubnt_e100#

You're ready to start the building process BUT we still need to fire up an FTP server on the Mac and serve up the image for tftpbooting. Download the latest available EMRK from here and place it into /private/tftpboot/.

open /private/tftpboot/ 

sudo chmod -R og+rX /private/tftpboot/
sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist
sudo launchctl start com.apple.tftpd


## To shut down tftp server
sudo launchctl unload -F /System/Library/LaunchDaemons/tftp.plist

Once you get your file downloaded and tftp server running keep following the instructions on the original thread, EdgeMax rescue kit (now you can reinstall EdgeOS from scratch). I preferred to plug my eth0 into my existing bench network (which allowed it to get out to the internet) and the console cable into the console port. This allowed me to also use dhcp from the bench network and to pull down the newest available EdgeRouter image from the internet. My commands were as follows:

dhcp
set bootfile emrk-0.9c.bin
set serverip <machost IP>
tftpboot;bootoctlinux $loadaddr

Once the file successfully pulled from my Mac's TFTP server and loaded up I was able to find myself looking at the EMRK loading away. I again used DHCP to configure eth0 which again made things easier for me. When at the EMRK> prompt I suggest deleting all existing partitions via the parted command. I wasted a little bit of time not doing this, expecting the EMRK to do it for me. It may work for others but what I'm giving you advise on will work every time.

parted -s /dev/sda rm 1
parted -s /dev/sda rm 2

Now that it's completely clean we can let the emrk-reinstall command do its magic. Before we continue lets get the URL from Ubquity to the newest available firmware release, Ubiquity Networks Downloads ER3-Lite. Save the URL in your clipboard as we will need it shortly. Upon execution of the emrk-reinstall script it will shortly ask for the URL to the firmware. The underlying execution uses a curl command to pull it in. Since the URL from Ubiquity is 'https' we must add '-k' to the beginning of the supplied URL so it will ignore the certificate error. I'm including an example below:

kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Enter EdgeOS image url: -https://dl.ubnt.com/firmwares/edgemax/v1.10.x/ER-e100.v1.10.1.5067571.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 68.3M  100 68.3M    0     0  8319k      0  0:00:08  0:00:08 --:--:-- 6318k

This should finally result in a finished image. If it does not please consult the original links for further troubleshooting. If you wish there are also tutorials on how you can expand the install script from the original 2GB partition to use the full size of your drive. I did this but it's outside the scope of this post.

You're ALMOST ready to say you're finished with this one but there's one more gotcha I'm going to post here to make this a 100% post. The USB storage available to us seems to be of different type than the ones used by the OEM. This can result in the router not correctly initilizing the storage upon boot reliably. You may find yourself sitting at a prompt as if all of your work was for nothing the first time the power is re-applied to the router. (It happened to me and is why I'm writing all of this for the community.) The last bit you need is here, List-of-Compatible-USB-drives post 24. Edit the bootcmd to allow the usb storage device to be reset before being accessed by the boot process. If this is done it'll fix the problem with dissimilar storage devices from a non-OEM. First set environment variable bootcmd with this command:

Octeon ubnt_e100# setenv bootcmd sleep 1\;usb reset\;sleep 1\;$(bootcmd)
Note all this must be on single line !
Next we must be sure that changes are accepted with this command:

Octeon ubnt_e100# printenv
Octeon ubnt_e100# saveenv

The router should now boot reliably. That's it! There are a few other shortcuts to all of this process made possible through other script writers but they are all doing the same thing as I've recounted here. The problem with always working through someone else's shortcuts is you never really know how to fix something on your own. Certain exceptions here are the creation of a tftpboot image and the tool itself we used, EMRK. I didn't think this should be something I should learn how to do on my own as it was going to be well outside the scope of anything I would be looking to accomplish. I hope this helps someone else. Not all of these links and procedures were in one place that I found over the course of a few days but now they are. Enjoy!