Tag Archives: HowTo

Meta-programming in Shell

Wikipedia defines meta-programming as:

programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running

Uncle Wiki

I had to write a “framework” at work where a shell program would run other shell programs “dynamically”. Let’s dig in!

As I mentioned in my earlier post Two Colons Equals Modules, you can “emulate” modules and functions in Shell (at least in FreeBSD’s /bin/sh) by using ::, so it would be module::function

Here we will do the same, however we will do hook::module.

The goal is to have a Shell program that would take a pid as an argument and do something with that PID, say print a group of information, maybe use DTrace to trace it, etc.

Let’s start by writing our main program.

#!/bin/sh
set -m

usage()
{
  echo "${0##*/} pid"
}

# print usage if argc < 1
[ "${#}" -lt "1" ] && usage && exit 1

# load scripts
load_scripts()
{
  for ctl in ./*.ctl.sh;
  do
    . "${ctl}"
  done
}

# stop the runner by killing the PIDs
runner_stop()
{
  IFS=":"
  for pid in $1;
  do
    kill $pid
  done
  exit
}

# Stop the runner if user sends an input
# This is useful if the runner is executed via a controller
wait_input()
{
  read command
  runner_stop ${PIDS}
}

# a.k.a. main()
runner_start()
{
  # make sure the process exists
  _pid="$1"
  ps -p "${_pid}" 1>/dev/null
  [ $? != 0 ] && exit 2

  # initiate scripts
  load_scripts

  # change IFS to :
  # loop over $SCRIPTS and execute the add hook
  IFS=":"
  for ctl in ${SCRIPTS};
  do
    add::${ctl} "${_pid}"
  done

  # now that we know the commands, loop over them too!
  # inside the loop set IFS to "," to set args
  for cmd in ${COMMAND};
  do
    IFS=","
    set -- "${cmd}"
    run::$1 $2
  done

  # Add trap for signals
  trap "runner_stop ${PIDS}" EXIT SIGINT SIGPIPE SIGHUP 0
  # reset IFS
  unset IFS
  wait_input
}

RUNNERDIR=$(dirname "$0")
(cd $RUNNERDIR && runner_start "$1")

Let’s digest a bit of that. First, we check if the number of arguments provided is less than 1

[ "${#}" -lt "1" ] && usage && exit 1

then we call usage and we exit with return code 1

The load_scripts function will load a bunch of scripts (from the same directory) as long as the scripts are suffixed .ctl.sh

Here’s an example script, say fds.ctl.sh, which will print File Descriptors used by the process, we will use procstat internally.

#!/bin/sh

add::fds()
{
  COMMAND="fds,$1:$COMMAND"
}

run::fds()
{
  procstat --libxo=xml -w 5 -f "$1" &
  PIDS="$!:$PIDS"
}

export SCRIPTS="fds:$SCRIPTS"

Here’s where meta-programming comes into use (I think), we have a variable named $SCRIPTS, which is modified to add the script name into it, $PATH-style, and two functions, add::fds and run::fds. As you have guessed add:: and run:: are the hook names.

I’ll add another script, it will use procstat as well, but this time we will print the resource usage

#!/bin/sh

add::resource()
{
  COMMAND="resource,$1:$COMMAND"
}

run::resource()
{
  procstat --libxo=xml -w 5 -r "$1" &
  PIDS="$!:$PIDS"
}

export SCRIPTS="resource:$SCRIPTS"

The same applies here, one variable, $SCRIPTS and two functions, add::resource and run::resource.

Which means, after loading our scripts all four functions will be loaded into our program and the environment variable $SCRIPTS will have the value resource:fds:

Good? Okay let’s continue.

Since we used : to separate the name of the scripts we must set IFS to :, and we start looping over $SCRIPTS. Now we just run add::${ctl}, which would be add::fds and add::resource. We also pass the ${_pid} variable, if we need to

These two functions would do more meta-programming by setting the $COMMAND variable to script_name,arguments:$COMMAND, again PATH-style.

Which means that the $COMMAND variable has the value fds,89913:resource,89913:

The next bit is a bit tricky, since we’ve set $COMMAND to prog0,arg1:prog1,arg1,arg2: (well, not really arg2, but we could’ve) then we need to

  1. Use “,” as IFS
  2. Tell sh to set the positional parameters, so prog0 becomes $1 and arg1 becomes $2, etc.

and now we execute run::$1 $2, which would be run::fds 89913 then run::resource 89913.

I think I can make this better by running run::$@, where $@ is basically all the parameters, but will test that later

– antranigv at 6am reading the code that he wrote drunk

In the end, we add some signal trapping, we reset IFS and we just wait for an input.

Okay, so we now have a piece of software that reads other programs and modifies itself while running. We have a meta-program!

Let’s give it a run.

# ./runner.sh 89913
<procstat version="1"><files><89913><procstat version="1"><rusage><89913><process_id>89913</process_id><command>miniflux</command><user time>01:37:54.339245</user time><system time>00:19:43.630210</system time><maximum RSS>61236</maximum RSS><integral shared memory>5917491656</integral shared memory><integral unshared data>1310633336</integral unshared data><integral unshared stack>114278656</integral unshared stack><process_id>89913</process_id><command>miniflux</command><files><fd>text</fd><fd_type>vnode</fd_type><vode_type>regular</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss/usr/local/bin/miniflux</path><page reclaims>16939</page reclaims><page faults>7</page faults><swaps>0</swaps><block reads>5</block reads><block writes>1</block writes><messages sent>12603917</messages sent></files><files><fd>cwd</fd><fd_type>vnode</fd_type><vode_type>directory</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss/root</path><messages received>14057863</messages received><signals received>807163</signals received><voluntary context switches>79530890</voluntary context switches><involuntary context switches>5489854</involuntary context switches></files><files><fd>root</fd><fd_type>vnode</fd_type><vode_type>directory</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss</path></89913></rusage></procstat></files><files><fd>jail</fd><fd_type>vnode</fd_type><vode_type>directory</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss</path></files><files><fd>0</fd><fd_type>vnode</fd_type><vode_type>character</vode_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>4</ref_count><offset>0</offset><protocol>-</protocol><path>/usr/local/jails/rss/dev/null</path></files><files><fd>1</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>2</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>3</fd><fd_type>kqueue</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>4</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>5</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>1</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>6</fd><fd_type>socket</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>3</ref_count><offset>0</offset><protocol>TCP</protocol><sendq>0</sendq><recvq>0</recvq><path>192.168.10.5:63835 192.168.10.3:5432</path></files><files><fd>7</fd><fd_type>socket</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>3</ref_count><offset>0</offset><protocol>TCP</protocol><sendq>0</sendq><recvq>0</recvq><path>::.8080 ::.0</path></files></89913></files></procstat>

Why XML? Because libxos JSON output is not “real” JSON when procstat‘s running in repeat mode, but that’s a story for another day.

All code examples can be found as a GitHub Gist.

That’s all folks…

Huginn on FreeBSD

Huginn is probably the best automation software that I’ve ever seen. It’s not only easy to use, but also easy to deploy and easy to extend. Unfortunately there’s no FreeBSD port for it, but looks like it’s something wanted by the community, at least according to WantedPorts.

I realized that I have at least 5 accounts on IFTTT, which is also an amazing automation service. However, 3/5 of these accounts were not “my own”. It belonged to our communities. You know, Meetups in Armenia and news listing websites similar to Lobste.rs. So if I get hit by a bus, it will be very hard for our community to operate these accounts, that’s why I wanted to deploy Huginn.

Like a sane person, I deploy in FreeBSD Jails (I recommend you do too!). Which meant there’s no official (or maybe even unofficial?) docs on how to deploy Huginn on FreeBSD.

It’s written in Ruby, which means it should work and should be very easy to ports. I’ll go over the deployment needs without the actual deployment, setup of Jails, or anything similar. Let’s go!

First thing first, you need Ruby thingies:

  • ruby
  • rubygem-bundler
  • rubygem-mimemagic
  • rubygem-rake
  • rubygem-mysql2

Here’s the full command for copy/paster:

pkg instal ruby rubygem-bundler rubygem-mimemagic rubygem-rake rubygem-mysql2

Next, you’ll need gmake for makefiles and node for assets:

pkg install gmake node

This should be enough. I’m also going to install git-tiny so I can follow their updates with ease.

pkg install git-tiny

Okay, let’s make a separate user for Huginn.

pw user add huginn -s /bin/tcsh -m -d /usr/local/huginn

Let’s switch our user

su - huginn

Okay, now I’m going to clone the repo 🙂

git clone --depth=1 https://github.com/huginn/huginn/

At this point you can go do installation.md#4-databases and configure your database.

You should also do cp .env.example .env and configure the environment, make sure to set RAILS_ENV=production

Next, as root, you should execute the following

cd /usr/local/huginn/huginn/ && bundle install

You might get an error saying

In Gemfile:
  mini_racer was resolved to 0.2.9, which depends on
    libv8

Don’t panic! That’s fine, unfortunately it’s trying to compile libv8 using Gems. Even if we installed the patched version of v8 using pkg, it still doesn’t work. I’ll try to workaround that later.
I an ideal world, all of these Ruby Gems should be ported to FreeBSD, I’m not sure which are ported, so I’ll just be using the bundle command to install them. And that’s why we use Jails 🙂

Anyways, the dependency Gem is mini_racer, comment its line in Gemfile

#gem 'mini_racer', '~> 0.2.4'      # JavaScriptAgent

Now let’s run Bundle again

cd /usr/local/huginn/huginn/ && bundle install

Okay! everything is good!

Let’s also build the assets, this one should be run as the user huginn

bundle exec rake assets:precompile RAILS_ENV=production

NOTE: If you get the following error ExecJS::RuntimeError: ld-elf.so.1: /lib/libcrypto.so.111: version OPENSSL_1_1_1e required by /usr/local/bin/node not found then you need to upgrade your FreeBSD version to the latest patch!

Aaand that’s it, everything is ready.

For the rest of the deployment process, such as the database, nginx, etc., please refer to installation.md

Currently, I’m running Huginn in a tmux session running bundle exec foreman start, but in the future, I’ll write an rc.d script and share it with you, too.

That’s all folks.

WireGuard “dynamic” routing on FreeBSD

I originally wrote about this on my Armenian blog when ISPs started blocking DNS queries during and after the war. I was forces to use either 9.9.9.9, 1.1.1.1, 8.8.8.8 or any other major DNS resolver. For me this was a pain because I was not able to dig +trace, and I dig +trace a lot.

After some digging (as mentioned in the Armenian blog) I was able to figure out that this affects only the home users. Luckily, I also run servers at my home and the ISPs were not blocking anything on those “server” ranges, so I setup WireGuard.

This post is not about setting up WireGuard, there are plenty of posts and articles on the internet about that.

Over time my network became larger. I also started having servers outside of my network. One of the fast (and probably wrong) ways of restricting access to my servers was allowing traffic only from my own network.

I have a server that acts as WireGuard VPN Peer and does NAT-ing. That being said, the easiest way for me to start accessing my restricted servers is by doing route add restricted_server_addr -interface wg0.

Turns out I needed to write some code for that, which I love to do!

Anytime that I need to setup a WireGuard VPN client I go back to my Armenian post and read there, so now I’ll be blogging how to do dynamic routing with WireGuard so I read whenever I need to. I hope it becomes handy for you too!

Now, let’s assume you need to add a.b.c.d in your routes, usually you’d do route add a.b.c.d -interface wg0, but this would not work, since in your WireGuard configuration you have a line that says

[Peer]
AllowedIPs = w.x.y.z/24

Which means, even if you add the route, the WireGuard application/kernel module will not route those packets.

To achieve “dynamic” routing we could do

[Peer]
AllowedIPs = 0.0.0.0/0

This, however will route ALL your traffic via WireGuard, which is also something you don’t want, you want to add routes at runtime.

What we could do, however, is to ask WireGuard to NOT add the routes automatically. Here’s how.

[Interface]
PrivateKey      = your_private_key
Address         = w.x.y.z/32
Table           = off
PostUp          = /usr/local/etc/wireguard/add_routes.sh %i
DNS             = w.z.y.1

[Peer]
PublicKey       = their_public_key
PresharedKey    = pre_shared_key
AllowedIPs      = 0.0.0.0/0
Endpoint        = your_server_addr:wg_port

The two key points here are Table = off which asks WireGuard to not add the routes automatically and PostUp = /usr/local/etc/wireguard/add_routes.sh %i which is a script that does add the routes, where %i is expanded to the WireGuard interface name; could be wg0, could be home0, depends in your configuration.

Now for add_routes.sh we write the following.

#!/bin/sh

interface=${1}

networks="""
w.x.y.0/24
restricted_server_addr/32
another_server/32
"""

for _n in ${networks};
do
  route -q -n add ${_n} -interface ${interface}
done

And we can finally do wg-quick up server0.conf

If you need to add another route while WireGuard is running, you can do

route add another_restricted_server -interface wg0

Okay, what if you need to route everything while WireGuard is running? Well, that’s easy too!

First, find your default gateway.

% route -n get default | grep gateway
    gateway: your_gateway

Next, add a route for your endpoint via your current default gateway.

route add you_server_addr your_gateway

Next, add TWO routes for WireGuard.

route add 0.0.0.0/1     -interface wg0
route add 128.0.0.0/1   -interface wg0

So it’s the two halves of the Internet 🙂

That’s all folks!

VoidLinux in FreeBSD Jail; with init

Two important things happened this week for me.

First, Faraz asked me if I can rename my Jail manager to something other than Jailio because he got that domain for his Jailer manager already. So I named it

Second, I was able to run a complete Linux system using Jailer. While the repo for Jailer is not released yet (we are auditing for possible security issues), I would like to share how I was able to run VoidLinux in a Jail.

Since Jailer is not announced yet, I will give the examples using jail.conf, as most people either are or should be familiar with its concepts.

I went with VoidLinux because I am able to run the init process without its need to be running as PID1.

Let’s start, shall we?

First, ZFS dataset for our jail!

zfs create zroot/jails/voidlinux

Next we need to fetch the base system of VoidLinux. Luckily they do provide it on their website.

fetch https://alpha.de.repo.voidlinux.org/live/current/void-x86_64-ROOTFS-20210218.tar.xz

Now we can extract this into our dataset

tar xf void-x86_64-ROOTFS-20210218.tar.xz -C /usr/local/jails/voidlinux/

You might get an error that ./usr/bin/iputils-ping: Cannot restore extended attributes: security.capability, which is fine, I think?

If you are on FreeBSD 12.2-RELEASE or later, now you need to enable the Linuxulator.

service linux enable; service linux start

Now you can at least chroot into the system.

chroot /usr/local/jails/voidlinux/ /bin/bash

If everything is fine until now, perfect.

Now we need to add a root user into the system.

root@host:~ # cd /usr/local/jails/voidlinux/etc/
root@host:/usr/local/jails/voidlinux/etc # echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd
root@host:/usr/local/jails/voidlinux/etc # pwd_mkdb -d ./ -p master.passwd
pwd_mkdb: warning, unknown root shell

Execute the rest of the commands in Void.

root@host:~ # chroot /usr/local/jails/voidlinux/ /bin/bash
bash-5.1# cd /etc/
bash-5.1# pwconv 
bash-5.1# grpconv 
bash-5.1# passwd 
New password: 
Retype new password: 
passwd: password updated successfully
bash-5.1# exit

If all went fine, then the system is ready to be run as a Jail!

First we need to make an fstab for the system.

Create a file at /usr/local/jails/voidlinux/etc/fstab.pre and insert the following inside

devfs       /usr/local/jails/voidlinux/dev      devfs           rw                      0   0
tmpfs       /usr/local/jails/voidlinux/dev/shm  tmpfs           rw,size=1g,mode=1777    0   0
fdescfs     /usr/local/jails/voidlinux/dev/fd   fdescfs         rw,linrdlnk             0   0
linprocfs   /usr/local/jails/voidlinux/proc     linprocfs       rw                      0   0
linsysfs    /usr/local/jails/voidlinux/sys      linsysfs        rw                      0   0
/tmp        /usr/local/jails/voidlinux/tmp      nullfs          rw                      0   0

Next, let’s create a loopback interface for networking. Oh yes, VNET is not supported yet, but I’m working on a patch 🙂

ifconfig lo1 create
ifconfig lo1 inet 10.10.0.1/24 up # sorry, 10.0.0.0/24 was unavailable :P

Okay, time to create our Jail conf!

exec.clean;
allow.raw_sockets;
mount.devfs;

voidlinux {
    $id     = "1";
    $ipaddr = "10.10.0.42";
    $mask   = "255.255.255.0";
    $domain = "srv0.bsd.am";
    devfs_ruleset  = 4;
    allow.mount;
    allow.mount.devfs;
    mount.fstab = "${path}/etc/fstab.pre";

    exec.start     = "/bin/sh /etc/runit/2 &";
    exec.stop      = "/bin/sh /etc/runit/3";


    ip4.addr      = "${ipaddr}";
    interface     = "lo1";
    host.hostname = "${name}.${domain}";
    path = "/usr/local/jails/voidlinux";
    exec.consolelog = "/var/log/jail-${name}.log";
    persist;
    allow.socket_af;
}

Let’s check?

# jls
   JID  IP Address      Hostname                      Path
     1  192.168.0.42    voidlinux.srv0.bsd.am         /usr/local/jails/voidlinux

And the process tree?

# ps auxd -J voidlinux
USER   PID %CPU %MEM  VSZ  RSS TT  STAT STARTED    TIME COMMAND
root 35182  0.0  0.1 2320 1428  -  SsJ  21:09   0:00.12 runsvdir -P /run/runit/runsvdir/current log: ot set SO_PASSCRED: Protocol not available\ncould not set SO_PASSCRED: Protocol
root 35190  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty6
root 35397  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty6 38400 linux
root 35191  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty1
root 35396  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty --noclear tty1 38400 linux
root 35192  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty5
root 35398  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.01 `-- agetty tty5 38400 linux
root 35193  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty2
root 35393  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty2 38400 linux
root 35194  0.0  0.1 2168 1396  -  RsJ  21:09   0:00.12 - runsv udevd
root 35195  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty3
root 35394  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty3 38400 linux
root 35196  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty4
root 35390  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty4 38400 linux

You may jexec now 🙂

# jexec voidlinux /bin/bash
bash-5.1# uname -a
Linux voidlinux.srv0.bsd.am 3.2.0 FreeBSD 12.2-RELEASE-p6 GENERIC x86_64 GNU/Linux

Let’s check networking?

bash-5.1# ping -c 1 10.10.0.1
ping: WARNING: setsockopt(ICMP_FILTER): Protocol not available
PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data.
64 bytes from 10.10.0.1: icmp_seq=1 ttl=64 time=0.069 ms

--- 10.10.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.069/0.069/0.069/0.000 ms

There you go! Well, things that are related to netlink might not work, but other than that it’s okay.

I did have some problems while installing packages, something about too many levels of symbolic links. Here’s the exact output when I was trying to install the curl package

[*] Unpacking packages
libev-4.33_1: unpacking ...
ERROR: libev-4.33_1: [unpack] failed to extract file `./usr/lib/libev.so.4': Too many levels of symbolic links
ERROR: libev-4.33_1: [unpack] failed to extract files: Too many levels of symbolic links
ERROR: libev-4.33_1: [unpack] failed to unpack files from archive: Too many levels of symbolic links
Transaction failed! see above for errors.

Now, I did not find the time to fix this yet, but if you have any idea, please let me know or comment below 🙂

So, what do we have here? A Linux Jail, running VoidLinux, with init, so you can also run services, and basic networking for it.

That’s all folks…

VNET Jail HowTo Part 2: Networking

As always, Dan has been tweeting about VNET Jail issues, which means it’s time for another VNET Jail post.

This post assumes that you’ve read the original post on VNET Jail HowTo.

In Part two we will discuss Networking.

We will use PF as a firewall to do things like NAT.

If you need more help please check the FreeBSD Handbook: Chapter – Firewalls or send me an email/tweet.

At this point (from the last post) we were able to ping from the Jail to the Host.

root@www:/ # ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.087 ms

--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.087/0.087/0.087/0.000 ms

Now we will setup PF on the host by adding the following to /etc/pf.conf

ext_if="em0"
jailnet="10.0.0.0/24"

nat pass on $ext_if inet from $jailnet to any -> ($ext_if)

set   skip on { lo0, bridge0 }
pass  inet proto icmp
pass  out all keep state

We also need to enable IP Forwarding in the kernel

Add the following in /etc/sysctl.conf

net.inet.ip.forwarding=1

And now execute

sysctl -f /etc/sysctl.conf
service pf restart

That should be it, now your Jail should be able to ping the outside world

root@zvartnots:~ # jexec -l www
You have mail.
root@www:~ # ping -c 1 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 56 data bytes
64 bytes from 9.9.9.9: icmp_seq=0 ttl=61 time=2.566 ms

--- 9.9.9.9 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.566/2.566/2.566/0.000 ms
root@www:~ # 

If you setup a resolver, you should also be able to ping domain names as well.

root@www:~ # echo 'nameserver 9.9.9.9' > /etc/resolv.conf 
root@www:~ # ping -c 1 freebsd.org
PING freebsd.org (96.47.72.84): 56 data bytes
64 bytes from 96.47.72.84: icmp_seq=0 ttl=53 time=133.851 ms

--- freebsd.org ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 133.851/133.851/133.851/0.000 ms

Now, for a more complicated setup that assumes no firewalls and multiple IP addresses, where each Jail has its own IP address. I have a similar setup at home where my ZNC server Jail has its own IP address by connecting the physical NIC to the same bridge as the ZNC Jail.

In my rc.conf on the host

ifconfig_em0="inet 192.168.0.34 netmask 255.255.255.0"
defaultrouter="192.168.0.1"

cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0"

Here’s an example with jail.conf

znc {
	$id		= "52";
	$addr		= "192.168.0.252";
	$mask		= "255.255.255.0";
	$gw		= "192.168.0.1";
	vnet;
	vnet.interface	= "epair${id}b";

	exec.prestart	= "ifconfig epair${id} create up";
	exec.prestart	+= "ifconfig epair${id}a up descr vnet-${name}";
	exec.prestart	+= "ifconfig bridge0 addm epair${id}a up";

	exec.start	= "/sbin/ifconfig lo0 127.0.0.1 up";
	exec.start	+= "/sbin/ifconfig epair${id}b ${addr} netmask ${mask} up";
	exec.start	+= "/sbin/route add default ${gw}";
	exec.start	+= "/bin/sh /etc/rc";

	exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
	exec.poststop  += "ifconfig epair${id}a destroy";

	host.hostname = "${name}.bsd.am";
	path = "/usr/local/jails/${name}";
 	exec.consolelog = "/var/log/jail-${name}.log";
	persist;
}

And that’s pretty much it!

That’s all folks.

Signal-cli with scli on FreeBSD

So couple of days ago I migrated from macOS on Macbook Pro to FreeBSD on ThinkPad T480s.

Unfortunately, since we are in war, I do not have the time to blog about the migration, although I’m taking notes every day about every change that I do so I can blog later on 🙂

However, one of the biggest concerns for me was running Signal on FreeBSD, as I understnad, Signal people are not interested in supporting the *BSDs.

As any sane person, I started searching the internet for possible solutions and turns out all I need is two pieces of software

The installation is as easy as running

pkg install signal-cli scli

Now for the simple part.

First, you need to link your phone by running

signal-cli link -n "FreeBSD"

It will give an output that says tsdevice:/?uuid=...&pub_key=....

Copy that output, and then in another terminal run

qrencode 'tsdevice:/?uuid=...&pub_key=...' -t ANSI256

You will be represented by a QR Code in the console (cool, aye?).

Using the phone app, link the device by scanning the QR Code.

To receive list of your contacts run

signal-cli -u +myphonenumber receive

Now try to run the TUI interface by running

scli

Side-note: In case you are not able to send or receive messages, you might need to do some DBUS magic.

First, find if you have DBUS running

antranigv@zvartnots:~ $ ps -x -o comm,pid | grep dbus
dbus-launch         53571
dbus-daemon         54064
dbus-daemon         54963

Then, you will need to find the DBUS_SESSION_BUS_ADDRESS environment variable, this is usually set in the DBUS child process, in our case, it’s 54963, so we can use procstat as root

root@zvartnots:~ # procstat -e 54963
  PID COMM             ENVIRONMENT                                          
54963 dbus-daemon      SHELL=/usr/local/bin/bash DBUS_STARTER_ADDRESS=unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c DISPLAY=:0.0 WMAKER_BIN_NAME=wmaker PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/antranigv/bin WINDOWPATH=9 MAIL=/var/mail/antranigv GTK_THEME=Adwaita:dark DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c USER=antranigv DBUS_STARTER_BUS_TYPE=session MM_CHARSET=UTF-8 WRASTER_COLOR_RESOLUTION0=4 PWD=/usr/home/antranigv BLOCKSIZE=K LANG=en_US.UTF-8 LOGNAME=antranigv HOME=/home/antranigv

Okay! we have our variable!

Now, we need to set the ENV and we are done, if you use (t)csh then execute

setenv DBUS_SESSION_BUS_ADDRESS unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c

If you are using bash, run the following

export DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c

Now, you can run scli again and it will work fine 🙂

Happy Chatting!@#$%

That’s all folks! 🙂

Erlang dbg Intro

If there’s one programming language that changed my life, that’s Erlang. After using Erlang for couple of years, I “moved” to Elixir, which is based on Erlang’s VM.

One the most important aspects of Erlang’s VM is that it’s a “real” VM, there’s a kernel, processes, messaging facilities and many more.

Lately I’ve been debugging a huge Erlang application whose architecture I was not very familiar with and I needed to find a way to see what kind of messages are being sent and received, which Modules and Functions are being called and what are they returning.

So I wanted to write a small How-To for me and you, in case we need it again in the future.

Okay, for this example I’ll be using Elixir TCP Server, a simple TCP server that gets data and sends it back to its origin.

First, let’s clone the repo.

antranigv@zvartnots:prj $ git clone https://github.com/SonaTigranyan/ElixirTcpServer

Okay, now let’s run the server

antranigv@zvartnots:ElixirTcpServer $ iex -S mix
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Compiling 3 files (.ex)
Generated tcp_server app
Interactive Elixir (1.10.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Good! By default, the TCP server listens on port 9000, as specified in the Application Tree.

Okay, we can send data now 🙂

antranigv@zvartnots:ElixirTcpServer $ echo test | nc localhost 9000
test

Or in an interactive way!

antranigv@zvartnots:ElixirTcpServer $ nc localhost 9000
First mesage!
First mesage!
Good TCP server!
Good TCP server!
bye
antranigv@zvartnots:ElixirTcpServer $

Good! As you can see the connection is closed when the server gets bye.

Okay, say we want to trace the do_send function, observe what does it get and return.

iex(2)> :dbg.start()
{:ok, #PID<0.191.0>}
iex(3)> :dbg.tracer()
{:ok, #PID<0.191.0>}
iex(4)> :dbg.tpl(TcpServer, :do_send, [{:_, [], [{:return_trace}]}])
{:ok, [{:matched, :nonode@nohost, 1}, {:saved, 1}]}
iex(5)> :dbg.p(:new_processes, :c)
{:ok, [{:matched, :nonode@nohost, 0}]}
iex(6)>
(<0.198.0>) call 'Elixir.TcpServer':do_send(#Port<0.545>,"Message from client!\n")

Okay, first we start the dbg facility, and then we start a tracing server on the local node.

After that, we use function tpl to specify which local calls we want to trace.

And in the end we use the p function to start tracing the calls (c) of all new_processes 🙂

Now, when the do_send function is called, we see what it gets.

And when we send bye, we see the following:

(<0.198.0>) returned from 'Elixir.TcpServer':do_send/2 -> ok

And all of this is happening when the software system is running. In production, we can do the same, by either attaching to the node or connecting to it!

That’s all folks! 🙂

autofs on macOS Catalina

One of the nice things (that used to be) about macOS is how much unix is underneath, but this has been less true each year.

Like any normal human being, I do my development on a real Unix system, FreeBSD. I ended up using FreeBSD VMs that have NFS exports, and I mount those on my macOS. However, there have been issues with Catalina, here’s the main problem:

mount | grep 'map auto_nfs_antranigv'  | wc -l
18

So for some reason in Catalina you can’t do nested mounts, because auto_nfs_antranigv was mounted in /Users/antranigv/nfs, which is inside /Users (also autofs’ed) it was duplicating and ended up eating a lot of CPU 🙂

Here’s the proper way of doing it

First, in /etc/auto_master add the following:

/System/Volumes/Data/netmount		auto_nfs_user

(you might want to change user to your username)

Then, create the following file → /etc/auto_nfs_user

Finally you can set your mounts, for example:

someoneelsecomputer	-fstype=nfs	cloudserver:/usr/home/sysadmin/cloud

here’s a complete example:

antranigv@zvartnots:~ $ cat /etc/auto_master
#
# Automounter master map
#
+auto_master			# Use directory service
#/net				-hosts		-nobrowse,hidefromfinder,nosuid
/home				auto_home	-nobrowse,hidefromfinder
/System/Volumes/Data/nfs	auto_nfs_antranigv
/Network/Servers		-fstab
/-				-static
antranigv@zvartnots:~ $ cat /etc/auto_nfs_antranigv
illuria-dev	-fstype=nfs illuria-dev:/usr/home/antranigv/illuria
devbsd-src	-fstype=nfs devbsd00:/usr/src

Now I’m happy! 🙂

VNET Jail HowTo

So Dan has been tweeting that there’s no good example to get started with VNET Jails with jail.conf, I thought it’s time to write one.

In this example I’ve used FreeBSD 12.1-RELEASE

root@jail-host:~ # freebsd-version
12.1-RELEASE
root@jail-host:~ # uname -a
FreeBSD jail-host 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@jail-host:~ #

First thing first, let’s setup a bridge on our host

root@jail-host:~ # sysrc cloned_interfaces="bridge0"
cloned_interfaces:  -> bridge0
root@jail-host:~ # sysrc ifconfig_bridge0="inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge"
ifconfig_bridge0:  -> inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge

Start the bridge0 interface without restarting the other interfaces

root@jail-host:~ # service netif start bridge0

Good! let’s setup a ZFS dataset for Jails 😉

root@jail-host:~ # zfs create -o mountpoint=/usr/local/jails zroot/jails

Good! now let’s fetch the base.txz file. I will be using my closest mirror, you should use yours.

root@jail-host:~ # mkdir /usr/local/jails/.dist-files
root@jail-host:~ # fetch -o /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz http://mirror.yandex.ru/freebsd/releases/amd64/12.1-RELEASE/base.txz

Perfect!

Now, we will extract the base into the jail.

root@jail-host:~ # zfs create zroot/jails/www
root@jail-host:~ # tar xf /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz -C /usr/local/jails/www/

Nicely done! Now let’s setup our /etc/jail.conf 🙂

Here’s my configuration.

# vim: set syntax=sh:
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
allow.raw_sockets;
allow.mount.tmpfs;
mount.devfs;


www {
    $id     = "10";
    $ipaddr = "10.0.0.${id}";
    $mask   = "255.255.255.0";
    $gw     = "10.0.0.1";
    vnet;
    vnet.interface = "epair${id}b";

    exec.prestart   = "ifconfig epair${id} create up";
    exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart  += "ifconfig bridge0 addm epair${id}a up";

    exec.start      = "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start     += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
    exec.start     += "/sbin/route add default ${gw}";
    exec.start     += "/bin/sh /etc/rc";

    exec.prestop    = "ifconfig epair${id}b -vnet ${name}";

    exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
    exec.poststop  += "ifconfig epair${id}a destroy";

    host.hostname = "${name}.jail-host";
    path = "/usr/local/jails/${name}";
    exec.consolelog = "/var/log/jail-${name}.log";
    persist;
}

Now let’s start our Jail!

root@jail-host:~ # service jail enable
jail enabled in /etc/rc.conf
root@jail-host:~ # service jail start www
Starting jails: www.
root@jail-host:~ # jls
   JID  IP Address      Hostname                      Path
     1                  www.jail-host                 /usr/local/jails/www

Let’s check the networking 🙂

root@jail-host:~ # ping -c 1 10.0.0.10
PING 10.0.0.10 (10.0.0.10): 56 data bytes
64 bytes from 10.0.0.10: icmp_seq=0 ttl=64 time=0.164 ms

--- 10.0.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.164/0.164/0.164/0.000 ms

We can do the same from the jail.

root@jail-host:~ # jexec www
root@www:/ # ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.087 ms

--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.087/0.087/0.087/0.000 ms

We can also stop all the jails.

root@jail-host:~ # service jail stop
Stopping jails: www.

Okay! Couple of notes 🙂

You can have jail.conf at /etc/jail.conf or /etc/something.jail.conf. The problem with the latter is that if you have jail_enable="YES" in rc.conf without defining jail_list then it will run only the jails in /etc/jail.conf

There are more ways to configure VNET Jails, either with jib or jng, an example is here.

Ideally, it would be nice to have /etc/jail.d/myjail.conf, and I wrote a patch for that (D24570), if you are a FreeBSD developer, please have a look 🙂

FreeBSD ZFS on Root Migration

My home server (running this blog) got old, it has only 2G of RAM and a very old hard drive.

I usually like to use laptops as home server, they get the job done! I know I can’t have fancy things like RAID1, but I usually keep my backups on a separate disk/machine anyway.

So, I got my Dell Latitude E5470 next to my 11 years old Dell Inspiron to start the migration process.

Here’s a simple how-to guide for migrating a FreeBSD system from one machine to another.


Snapshotting The Pool

Okay, on the old host let snapshot the pool.

zfs snap -r zroot@migrate

Let’s save that snapshot in a file.

zfs send -R zroot@migrate | gzip > /usr/local/zroot.zfs.gz

This might take a while, grab some coffee 🙂

Now let’s save the ZFS Boot Environment (bootfs) property

# zpool get bootfs zroot
NAME   PROPERTY  VALUE               SOURCE
zroot  bootfs    zroot/ROOT/default  local

Perfect! now let’s go to the new host! 🙂

Installing FreeBSD

Well, this is simple 🙂 first, install FreeBSD on the new machine. DON’T forget to use ZFS! 🙂

The reason why we are doing this so the FreeBSD installer will handle all the boot loader installation process. If you know how to do this manually, then be my guest (and blog about it).

“Reset” ZFS

Okay, now let’s “reset” that pool

zpool destroy zroot
zpool create -R /mnt zroot /dev/ada0p3 # please check the partition index thought
zpool export zroot # so we can import again with -N
zpool import -N -R /mnt zroot

Restore Old Snapshot

Here we go.

ssh antranigv@pingvinashen "gzcat /usr/local/zroot.zfs.gz" | zfs receive -v -u -F zroot

Aaand we wait again.

So, check you restored all the datasets with zfs list -r zroot

Set the bootfs property.

zpool set bootfs=zroot/ROOT/default zroot

Fingers Crossed and reboot

well, now it’s the time.

shutdown -r now

Done

Hope this helps 🙂