Home-Network

Home network schema

This series of posts describes the setup of a custom router for a home network. The router takes full advantage of multiple Internet connections, seamlessly manages Internet traffic when one of the connections drops or behaves poorly, and provides some additional functionality like firewalls, adblocking, VPN tunnels for specific clients, bandwidth shaping, network flow monitoring etc.

In Part 1, we covered the motivation to do this (desire to fully utilize multiple Internet connections without compromise), identified the enabling technology (a VPN service named Speedify), defined the needed functionality from the router as well as the hardware of choice. In this part, we’ll go through the software setup of the router for achieving all the must have pieces of functionality (DNS, DHCP, Firewall, Multi-WAN Load Balancing and Failover, Blocking Advertisements and other malicious activity.) Subsequent parts will cover more advanced functionality like policy routing and traffic shaping.

Operating system and network interfaces

I opted for Ubuntu server 20.04. Am familiar with Debian based distros and the Speedify VPN client as well as pi-hole (which is use for DNS, DHCP, and ad blocking) all work well with Ubuntu. Version 20.04 is an LTS release, which means it’ll be supported for five years – good enough.

OS installation was straightforward; setting up the network interfaces not so much. Ubuntu 20.04 (and possibly a few versions before it) use a configuration method called netplan for the network configuration. This is different from the /etc/network/interfaces method in Debian that I’m familiar with. So it took a bit of reading up but ultimately, the actual network configuration was trivially easy. In the file /etc/netplan/00-installer-config.yaml the following content needed to be entered.

network:
  ethernets:
    enp1s0:
      dhcp4: true
    enp2s0:
      dhcp4: true
    enp3s0:
      addresses:
      - 192.168.3.1/24
      nameservers: {}
  version: 2

It sets up the first two network interfaces enp1s0 and enp2s0 via dhcp and creates a static IP address of 192.168.3.1/24 for the third network interface, enp3s0. The reason for this configuration is that the home network (LAN) will be in the subnet 192.168.3.0/24 and will use the router as the gateway. Meanwhile, the router will use the first two interfaces for Internet access and they will receive IP addresses from the modem-devices provided by the ISPs, Comcast and AT&T. See the network schema in the Figure above. The Comcast modem is configured to use the subnet 192.168.1.0/24 while the AT&T modem uses 192.168.5.0/24. Apply the configuration and restart networking

sudo netplan apply
sudo systemctl restart systemd-networkd

DNS, DHCP, and ad blocking

My desire was to forego setting up each service individually (via bind, isc-dhcp-server, or dnsmasq) in preference for something simpler to install, configure, and maintain. As of this writing, there are really only two options: AdGuard Home and Pi-hole. Both are very similar to each other. Both require a single command to install, followed by easy graphical configuration. Both offer DNS, DHCP, and ad blocking and both are open source. I started with AdGuard Home because it looks like the easier one to install and maintain. It’s a single binary with everything included in it. To update, you can either press a button in the web interface – or if that fails – download the new binary and overwrite the old one. Easy peasy! The Getting Started guide for AdGuard Home is extremely well-written and easy to follow. I wish all guides were like that.

AdGuard Home works well. However, after a couple of days, I decided to switch to pi-hole. The reason for this is that I could not find a way to set a domain name for the home network with AdGuard Home’s configuration interface. Perhaps due to this, I was unable to ping other devices on the home network with their hostnames. To be fair, AdGuard Home makes it possible to set static IP addresses and set up custom DNS entries to make this possible, but I didn’t want to manually write entries for every device on the home network. Also, the DHCP configuration page is quite rudimentary. If you want to set up a static lease, you need to manually type the MAC address, IP address, and hostname for the device. There isn’t a single button you can click next to a current lease that transfers that device to a static address. Yes, this is a nitpick.

In the end, despite some reservations with pi-hole, I disabled AdGuard Home and installed pi-hole with its single line installation command. My reservations stemmed from earlier experiences with pi-hole wherein an upgrade broke the install and needed multiple forum thread searches and command-line tinkering and eventually a re-install (which was not straightforward) before it worked again. This time the installation was straightforward, but using the admin configuration screen immediately after the installation led to an error “Wrong token! Please re-login on the Pi-hole dashboard.” No configuration changes could be applied. Despite multiple Google and forum searches for over 1½ hour, I’ve still not managed to resolve it. Funnily enough, it works with a different browser. So I cleaned the cache and cookies of the browser where it does not work, but no luck ¯\_(ツ)_/¯ . Pi-hole’s DNS support seems more sophisticated than AdGuard Home and the DHCP configuration page enables setting a domain name. With Pi-hole, I am able to access other devices with their hostnames. Another positive is that one device on my home network apparently does not share a hostname..only an IP address is visible. With AdGuard Home, there was no way of knowing which device this was. With Pi-hole, under Tools > Network there’s a table showing all devices on the network, together with their MAC address and the company to which the MAC address apparently belongs. Here, I could see that the mystery device had ‘Nvidia’ listed as the manufacturer, which immediately cleared matters up. The only Nvidia device on the home network is an Nvidia Shield.

Multi-WAN load balancing and failover

I installed the Speedify command line client to take advantage of multiple Internet connections. Speedify uses each Internet connection to create a connection to one of its servers and that server becomes your jump off point to the Internet. The Speedify client will load balance between both Internet connections and seamlessly route traffic around a failed or lagging Internet connection. Moreover, since your jumping off point to the Internet is Speedify’s VPN server, any https sites being accessed see a single public IP for a session rather than two different ones (which is what happens with other load-balancing solutions e.g. pfsense). This keeps the https servers happy. With pfsense, my bank website, for example, would terminate the session abruptly because it didn’t like when packets for the same session came from two different public IP addresses.

Installation of the speedify_cli client was straightforward. Just use the command:
wget -qO- https://get.speedify.com | sudo -E bash -

It took some tinkering to get the client working well. Speedify provides a reference for the command line client, but it was not immediately obvious which commands to use and their sequence. Speedify documentation is geared towards “normal” users who are not running the client on a router. Eventually, I ended up with the following commands, starting by first disabling the usage of the enp3s0 interface, which is the LAN interface.

# Speedify should not use the enp3s0 interface, which connects to local LAN
/usr/share/speedify/speedify_cli adapter priority enp3s0 never
# log in. Remember to escape special characters in the passwd with\
/usr/share/speedify/speedify_cli login username password
# Tell speedify which server to connect to. This is optional. The fremont servers are not (yet) blocked by Netflix :)
/usr/share/speedify/speedify_cli connect us fremont 5
# Connect command
/usr/share/speedify/speedify_cli connect
# See state
/usr/share/speedify/speedify_cli state
# See which network adapters are being used
/usr/share/speedify/speedify_cli show adapters
# Show which server is being used
/sur/share/speedify/speedify_cli show currentserver
# enable speedify to connect on startup
/usr/share/speedify/speedify_cli startupconnect on

Speedify does have a way to share the Internet connection with other clients on the network, but their way to do so seems a bit limited. In the Firewall section below, we’ll see how to share the Internet connection without using Speedify settings. The routing table when all the interfaces and speedify are up is shown below. Speedify creates a network interface named connectify0 and it is used as the default route

sagar@router:~$ ip r
0.0.0.0/1 dev connectify0 scope link
default via 192.168.1.1 dev enp1s0 proto dhcp src 192.168.1.45 metric 100
default via 192.168.5.254 dev enp2s0 proto dhcp src 192.168.5.75 metric 100
10.202.0.0/24 dev connectify0 proto kernel scope link src 10.202.0.2
128.0.0.0/1 dev connectify0 scope link
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.45
192.168.1.1 dev enp1s0 proto dhcp scope link src 192.168.1.45 metric 100
192.168.3.0/24 dev enp3s0 proto kernel scope link src 192.168.3.1
192.168.5.0/24 dev enp2s0 proto kernel scope link src 192.168.5.75
192.168.5.254 dev enp2s0 proto dhcp scope link src 192.168.5.75 metric 100

sagar@router:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         128.0.0.0       U     0      0        0 connectify0
0.0.0.0         192.168.5.254   0.0.0.0         UG    100    0        0 enp2s0
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 enp1s0
0.0.0.0         192.168.1.1     0.0.0.0         UG    202    0        0 enp1s0
0.0.0.0         192.168.5.254   0.0.0.0         UG    203    0        0 enp2s0
10.202.0.0      0.0.0.0         255.255.255.0   U     0      0        0 connectify0
128.0.0.0       0.0.0.0         128.0.0.0       U     0      0        0 connectify0
192.168.1.0     0.0.0.0         255.255.255.0   U     202    0        0 enp1s0
192.168.1.1     0.0.0.0         255.255.255.255 UH    100    0        0 enp1s0
192.168.3.0     0.0.0.0         255.255.255.0   U     204    0        0 enp3s0
192.168.5.0     0.0.0.0         255.255.255.0   U     203    0        0 enp2s0
192.168.5.254   0.0.0.0         255.255.255.255 UH    100    0        0 enp2s0

Other book keeping commands for Speedify are

sudo systemctl enable speedify
sudo systemctl stop speedify
sudo systemctl disable speedify

Note: Do not set up Pi-hole/Adguard Home to use DNS-over-HTTPS. This will break Speedify’s ability to bypass sites (e.g. Netflix, Amazon Prime).

Firewall

As set up so far, only the router device has access to the Internet. To enable other devices on the home network to access the Internet, three configurations are necessary: 1. IP Forwarding needs to be turned on. 2. Masquerading (NAT) needs to be set up. 3. Firewall rules to prevent un-established inbound connections, forward traffic between LAN and WAN interfaces.

Initially, I considered simplifying tools like ufw, shorewall, and firewalld for configuring iptables. However, I ended up directly writing the iptables rules by hand, since my firewall needs are fairly simple (no port forwarding, single trusted zone, etc.). The complete firewall script is given below

#!/bin/bash

LAN_IFACE="enp3s0"
WAN1_IFACE="enp1s0"
WAN2_IFACE="enp2s0"
SPEEDIFY_IFACE="connectify0"

# Enable IPv4 forwarding
#echo 1 > /proc/sys/net/ipv4/ip_forward
sysctl -w net.ipv4.ip_forward=1

# Start from a blank slate. Flush all iptables rules
iptables -F
iptables -X
iptables -Z

# Default policy to drop all incoming and forwarded packets.
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Accept incoming packets from localhost and the LAN interface.
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i $LAN_IFACE -j ACCEPT

# Accept incoming packets from the WAN if the router initiated the connection.
iptables -A INPUT -i $WAN1_IFACE -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $WAN2_IFACE -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $SPEEDIFY_IFACE -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Accept forwarded packets from LAN to the WAN.
iptables -A FORWARD -i $LAN_IFACE -o $WAN1_IFACE -j ACCEPT
iptables -A FORWARD -i $LAN_IFACE -o $WAN2_IFACE -j ACCEPT
iptables -A FORWARD -i $LAN_IFACE -o $SPEEDIFY_IFACE -j ACCEPT

# Accept forwarded packets from WAN to the LAN if the LAN initiated the connection.
iptables -A FORWARD -i $WAN1_IFACE -o $LAN_IFACE -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i $WAN2_IFACE -o $LAN_IFACE -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i $SPEEDIFY_IFACE -o $LAN_IFACE -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# NAT traffic going out the WAN.
iptables -t nat -A POSTROUTING -o $WAN1_IFACE -j MASQUERADE
iptables -t nat -A POSTROUTING -o $WAN2_IFACE -j MASQUERADE
iptables -t nat -A POSTROUTING -o $SPEEDIFY_IFACE -j MASQUERADE

exit 0

An interesting observation is that even if a particular interface does not exist, the firewall rule can be specified. This surprised me somewhat because I’d assumed that if interface foo did not exist, then wouldn’t be possible to add an iptables rule containing -i foo and that the rule would need to be added in a post-hook after the interface is brought up. Apparently this is not the case, which means that the above firewall scrip will work even if, for example, the connectify0 interface isn’t brought up properly.

This script needs to be run at startup. It seems that the old ways of using /etc/rc.local are deprecated with the advent of systemd. So we create a systemd.service file /etc/systemd/system/firewall.service with contents

[Unit]
Description=Router Firewall by Sagar
ConditionFileIsExecutable=/home/sagar/firewall.sh
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/home/sagar/firewall.sh

[Install]
WantedBy=multi-user.target

and enable its execution at startup with sudo systemctl enable firewall

Testing

  1. If one of the two WAN links is disabled, speedify maintains seamless Internet connectivity. This can be tested with: sudo ip link set enp1s0 down or sudo ip link set enp2s0 down
  2. If speedify goes down (can be tested with sudo systemctl stop speedify), Internet traffic is seamlessly routed through one of the two WAN interfaces; either enp1s0 or enp2s0, whichever happens to be first in the routing table. I’m sure there is a way to set a preferred Internet connection but at this moment, I don’t need it, so have not set that up.
  3. Even if speedify goes down and then one of the WAN interfaces is disabled, Internet traffic is seamlessly routed through the other WAN interface.
  4. If one of the two Internet connections is not disabled, but merely stops dropping packets or becomes laggy, Speedify detects it and preferentially routes traffic through the other Internet connection. I have not tested this, but this is what Speedify is supposed to do, so I’m taking it as an article of faith (lol, wat?). I did see once that AT&T dropped the connection for several minutes and Speedify seamlessly routed traffic through Comcast.

Router health monitoring

I installed the open source software Netdata to monitor over 2000 health and state variables (incl. CPU/Memory usage) on the router with per second granularity. Netdata is installed with a single command
bash <(curl -Ss https://my-netdata.io/kickstart.sh) and this creates a really fancy graphical dashboard at port 19999 of the router’s local ip addr. Updating Netdata is fairly simple. The dashboard informs when a new update is available and you simply run the above installation command once more to update to the latest version.

Summary

The above setup leads to a router that robustly maintains Internet connectivity unless both Internet connections fail. There is a strong firewall on the router, which also provides DNS, DHCP, and network-wide adblocking to all devices on the home network. Health of the router is monitored via a graphical dashboard with per second granularity.

Future posts will cover setup of VPN clients, policy based routing, traffic shaping and QoS, and network flow monitoring.