Tag Archives: Networking

WireGuard “dynamic” routing on FreeBSD

I originally wrote about this on my Armenian blog when ISPs started blocking DNS queries during and after the war. I was forces to use either,, or any other major DNS resolver. For me this was a pain because I was not able to dig +trace, and I dig +trace a lot.

After some digging (as mentioned in the Armenian blog) I was able to figure out that this affects only the home users. Luckily, I also run servers at my home and the ISPs were not blocking anything on those “server” ranges, so I setup WireGuard.

This post is not about setting up WireGuard, there are plenty of posts and articles on the internet about that.

Over time my network became larger. I also started having servers outside of my network. One of the fast (and probably wrong) ways of restricting access to my servers was allowing traffic only from my own network.

I have a server that acts as WireGuard VPN Peer and does NAT-ing. That being said, the easiest way for me to start accessing my restricted servers is by doing route add restricted_server_addr -interface wg0.

Turns out I needed to write some code for that, which I love to do!

Anytime that I need to setup a WireGuard VPN client I go back to my Armenian post and read there, so now I’ll be blogging how to do dynamic routing with WireGuard so I read whenever I need to. I hope it becomes handy for you too!

Now, let’s assume you need to add a.b.c.d in your routes, usually you’d do route add a.b.c.d -interface wg0, but this would not work, since in your WireGuard configuration you have a line that says

AllowedIPs = w.x.y.z/24

Which means, even if you add the route, the WireGuard application/kernel module will not route those packets.

To achieve “dynamic” routing we could do

AllowedIPs =

This, however will route ALL your traffic via WireGuard, which is also something you don’t want, you want to add routes at runtime.

What we could do, however, is to ask WireGuard to NOT add the routes automatically. Here’s how.

PrivateKey      = your_private_key
Address         = w.x.y.z/32
Table           = off
PostUp          = /usr/local/etc/wireguard/add_routes.sh %i
DNS             = w.z.y.1

PublicKey       = their_public_key
PresharedKey    = pre_shared_key
AllowedIPs      =
Endpoint        = your_server_addr:wg_port

The two key points here are Table = off which asks WireGuard to not add the routes automatically and PostUp = /usr/local/etc/wireguard/add_routes.sh %i which is a script that does add the routes, where %i is expanded to the WireGuard interface name; could be wg0, could be home0, depends in your configuration.

Now for add_routes.sh we write the following.




for _n in ${networks};
  route -q -n add ${_n} -interface ${interface}

And we can finally do wg-quick up server0.conf

If you need to add another route while WireGuard is running, you can do

route add another_restricted_server -interface wg0

Okay, what if you need to route everything while WireGuard is running? Well, that’s easy too!

First, find your default gateway.

% route -n get default | grep gateway
    gateway: your_gateway

Next, add a route for your endpoint via your current default gateway.

route add you_server_addr your_gateway

Next, add TWO routes for WireGuard.

route add     -interface wg0
route add   -interface wg0

So it’s the two halves of the Internet 🙂

That’s all folks!

VNET Jail HowTo Part 2: Networking

As always, Dan has been tweeting about VNET Jail issues, which means it’s time for another VNET Jail post.

This post assumes that you’ve read the original post on VNET Jail HowTo.

In Part two we will discuss Networking.

We will use PF as a firewall to do things like NAT.

If you need more help please check the FreeBSD Handbook: Chapter – Firewalls or send me an email/tweet.

At this point (from the last post) we were able to ping from the Jail to the Host.

root@www:/ # ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.087 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.087/0.087/0.087/0.000 ms

Now we will setup PF on the host by adding the following to /etc/pf.conf


nat pass on $ext_if inet from $jailnet to any -> ($ext_if)

set   skip on { lo0, bridge0 }
pass  inet proto icmp
pass  out all keep state

We also need to enable IP Forwarding in the kernel

Add the following in /etc/sysctl.conf


And now execute

sysctl -f /etc/sysctl.conf
service pf restart

That should be it, now your Jail should be able to ping the outside world

root@zvartnots:~ # jexec -l www
You have mail.
root@www:~ # ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=61 time=2.566 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.566/2.566/2.566/0.000 ms
root@www:~ # 

If you setup a resolver, you should also be able to ping domain names as well.

root@www:~ # echo 'nameserver' > /etc/resolv.conf 
root@www:~ # ping -c 1 freebsd.org
PING freebsd.org ( 56 data bytes
64 bytes from icmp_seq=0 ttl=53 time=133.851 ms

--- freebsd.org ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 133.851/133.851/133.851/0.000 ms

Now, for a more complicated setup that assumes no firewalls and multiple IP addresses, where each Jail has its own IP address. I have a similar setup at home where my ZNC server Jail has its own IP address by connecting the physical NIC to the same bridge as the ZNC Jail.

In my rc.conf on the host

ifconfig_em0="inet netmask"

ifconfig_bridge0="addm em0"

Here’s an example with jail.conf

znc {
	$id		= "52";
	$addr		= "";
	$mask		= "";
	$gw		= "";
	vnet.interface	= "epair${id}b";

	exec.prestart	= "ifconfig epair${id} create up";
	exec.prestart	+= "ifconfig epair${id}a up descr vnet-${name}";
	exec.prestart	+= "ifconfig bridge0 addm epair${id}a up";

	exec.start	= "/sbin/ifconfig lo0 up";
	exec.start	+= "/sbin/ifconfig epair${id}b ${addr} netmask ${mask} up";
	exec.start	+= "/sbin/route add default ${gw}";
	exec.start	+= "/bin/sh /etc/rc";

	exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
	exec.poststop  += "ifconfig epair${id}a destroy";

	host.hostname = "${name}.bsd.am";
	path = "/usr/local/jails/${name}";
 	exec.consolelog = "/var/log/jail-${name}.log";

And that’s pretty much it!

That’s all folks.

FreeBSD USB Disk and ZVOL Encryption with GELI

Disk encryption is becoming more important in our day to day life, specially when you have access to some corporate servers or “top secret” files.

I love FreeBSD, it’s simple, rock-solid, easy to use, the handbook is amazing! It also has the option to encrypt the disks during installation. I use FreeBSD everywhere (and TrueOS on my laptop), but disk encryption takes a lot of power, so I chose instead of doing full disk encryption in my laptop, I’ll just have a small media like a USB drive or ZFS ZVOL and encrypt that.

Here’s how to do so 🙂

If you compiled your own kernel ensure it contains these options

options GEOM_ELI
device crypto

Now, make sure crypto and geom_eli is loaded and add these lines to /boot/loader.conf:


Let’s move on.

Now, we need the partition that we are going to encrypt it.
In case it’s a USB drive that you want to encrypt, here’s what to do. First, plug-in the USB drive into your computer. Now, let’s check it’s GEOM class name.

# geom disk list
Geom name: ada0
1. Name: ada0
   Mediasize: 480103981056 (447G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: SanDisk Ultra II 480GB
   lunid: 5001b444a4a40542
   ident: 162265428493
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Geom name: da0
1. Name: da0
   Mediasize: 4004511744 (3.7G)
   Sectorsize: 512
   Mode: r0w0e0
   descr: SanDisk Cruzer Fit
   lunname: SanDisk Cruzer Fit      4C532000030211123165
   lunid: SanDisk Cruzer Fit      4C532000030211123165
   ident: 4C532000030211123165
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Destroy and Create New Partitions

Okay, as we can see it’s da0. First, let’s destroy it and make a new partition on it! (Make sure you backup your data in case you have any important files).

# gpart destroy -F da0
da0 destroyed
# gpart create -s GPT da0
da0 created
# gpart add -t freebsd-ufs -i 1 da0
da0p1 added

In my case, I didn’t want to use a USB drive, I wanted to have an encrypted ZVOL, here’s how to do that as well.
First, create a ZVOL

# zfs create -V 1G zroot/private

Okay, so, in case of a USB drive, we have a partition waiting for us, in case of ZVOL, we have 1GB volume.

Let’s encrypt those!

Initiating Encryption

There are multiple ways to encrypt a disk, check geli(8) for detailed info. Here I’ll show you two options.

  • Encrypting with a master key that is protected with a passphrase.
  • Encrypting with a passphrase only.

For the first option first, generate a key!

# dd if=/dev/random of=/root/master.key bs=64 count=1

Now we initialize the provider which needs to be encrypted.

# geli init -s 4096 -K /root/master.key /dev/da0p1

or in case of ZVOL

# geli init -s 4096 -K /root/master.key /dev/zvol/zroot/private

You’ll be asked to enter your passphrase, twice.

For the second option, it’s exactly the same command without -K /root/master.key. So for the ZVOL it would be

# geli init -s 4096 /dev/zvol/zroot/private

Attaching Encrypted Disks

Now we can attach the provider with the generated key or without it, here’s an example.

# geli attach -k /root/master.key /dev/da0p1

You will be asked for your passphrase.
Or without the key, only the passphrase, here’s an example.

# geli attach /dev/zvol/zroot/private

This creates a new device with .eli extention:

# ls /dev/zvol/zroot/private
private.eli%    private%

Create New File System

First, let’s randomize whatever is on the device and then format it with UFS file system.

# dd if=/dev/random of=/dev/zvol/zroot/private.eli bs=1m
# newfs /dev/zvol/zroot/private.eli

Mount and Use

# mount /dev/zvol/zroot/private.eli /mnt/private
# echo 'some data' > /mnt/private/mytopsecretdata

Detaching Encrypted Volume

# umount /mnt/private
# geli detach /dev/zvol/zroot/private.eli

That’s all folks! 🙂