Tag Archives: FreeBSD

Meta-programming in Shell

Wikipedia defines meta-programming as:

programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running

Uncle Wiki

I had to write a “framework” at work where a shell program would run other shell programs “dynamically”. Let’s dig in!

As I mentioned in my earlier post Two Colons Equals Modules, you can “emulate” modules and functions in Shell (at least in FreeBSD’s /bin/sh) by using ::, so it would be module::function

Here we will do the same, however we will do hook::module.

The goal is to have a Shell program that would take a pid as an argument and do something with that PID, say print a group of information, maybe use DTrace to trace it, etc.

Let’s start by writing our main program.

set -m

  echo "${0##*/} pid"

# print usage if argc < 1
[ "${#}" -lt "1" ] && usage && exit 1

# load scripts
  for ctl in ./*.ctl.sh;
    . "${ctl}"

# stop the runner by killing the PIDs
  for pid in $1;
    kill $pid

# Stop the runner if user sends an input
# This is useful if the runner is executed via a controller
  read command
  runner_stop ${PIDS}

# a.k.a. main()
  # make sure the process exists
  ps -p "${_pid}" 1>/dev/null
  [ $? != 0 ] && exit 2

  # initiate scripts

  # change IFS to :
  # loop over $SCRIPTS and execute the add hook
  for ctl in ${SCRIPTS};
    add::${ctl} "${_pid}"

  # now that we know the commands, loop over them too!
  # inside the loop set IFS to "," to set args
  for cmd in ${COMMAND};
    set -- "${cmd}"
    run::$1 $2

  # Add trap for signals
  trap "runner_stop ${PIDS}" EXIT SIGINT SIGPIPE SIGHUP 0
  # reset IFS
  unset IFS

RUNNERDIR=$(dirname "$0")
(cd $RUNNERDIR && runner_start "$1")

Let’s digest a bit of that. First, we check if the number of arguments provided is less than 1

[ "${#}" -lt "1" ] && usage && exit 1

then we call usage and we exit with return code 1

The load_scripts function will load a bunch of scripts (from the same directory) as long as the scripts are suffixed .ctl.sh

Here’s an example script, say fds.ctl.sh, which will print File Descriptors used by the process, we will use procstat internally.



  procstat --libxo=xml -w 5 -f "$1" &

export SCRIPTS="fds:$SCRIPTS"

Here’s where meta-programming comes into use (I think), we have a variable named $SCRIPTS, which is modified to add the script name into it, $PATH-style, and two functions, add::fds and run::fds. As you have guessed add:: and run:: are the hook names.

I’ll add another script, it will use procstat as well, but this time we will print the resource usage



  procstat --libxo=xml -w 5 -r "$1" &

export SCRIPTS="resource:$SCRIPTS"

The same applies here, one variable, $SCRIPTS and two functions, add::resource and run::resource.

Which means, after loading our scripts all four functions will be loaded into our program and the environment variable $SCRIPTS will have the value resource:fds:

Good? Okay let’s continue.

Since we used : to separate the name of the scripts we must set IFS to :, and we start looping over $SCRIPTS. Now we just run add::${ctl}, which would be add::fds and add::resource. We also pass the ${_pid} variable, if we need to

These two functions would do more meta-programming by setting the $COMMAND variable to script_name,arguments:$COMMAND, again PATH-style.

Which means that the $COMMAND variable has the value fds,89913:resource,89913:

The next bit is a bit tricky, since we’ve set $COMMAND to prog0,arg1:prog1,arg1,arg2: (well, not really arg2, but we could’ve) then we need to

  1. Use “,” as IFS
  2. Tell sh to set the positional parameters, so prog0 becomes $1 and arg1 becomes $2, etc.

and now we execute run::$1 $2, which would be run::fds 89913 then run::resource 89913.

I think I can make this better by running run::$@, where $@ is basically all the parameters, but will test that later

– antranigv at 6am reading the code that he wrote drunk

In the end, we add some signal trapping, we reset IFS and we just wait for an input.

Okay, so we now have a piece of software that reads other programs and modifies itself while running. We have a meta-program!

Let’s give it a run.

# ./runner.sh 89913
<procstat version="1"><files><89913><procstat version="1"><rusage><89913><process_id>89913</process_id><command>miniflux</command><user time>01:37:54.339245</user time><system time>00:19:43.630210</system time><maximum RSS>61236</maximum RSS><integral shared memory>5917491656</integral shared memory><integral unshared data>1310633336</integral unshared data><integral unshared stack>114278656</integral unshared stack><process_id>89913</process_id><command>miniflux</command><files><fd>text</fd><fd_type>vnode</fd_type><vode_type>regular</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss/usr/local/bin/miniflux</path><page reclaims>16939</page reclaims><page faults>7</page faults><swaps>0</swaps><block reads>5</block reads><block writes>1</block writes><messages sent>12603917</messages sent></files><files><fd>cwd</fd><fd_type>vnode</fd_type><vode_type>directory</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss/root</path><messages received>14057863</messages received><signals received>807163</signals received><voluntary context switches>79530890</voluntary context switches><involuntary context switches>5489854</involuntary context switches></files><files><fd>root</fd><fd_type>vnode</fd_type><vode_type>directory</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss</path></89913></rusage></procstat></files><files><fd>jail</fd><fd_type>vnode</fd_type><vode_type>directory</vode_type><fd_flags>read</fd_flags><ref_count>-</ref_count><offset>-</offset><protocol>-</protocol><path>/usr/local/jails/rss</path></files><files><fd>0</fd><fd_type>vnode</fd_type><vode_type>character</vode_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>4</ref_count><offset>0</offset><protocol>-</protocol><path>/usr/local/jails/rss/dev/null</path></files><files><fd>1</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>2</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>3</fd><fd_type>kqueue</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>4</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>2</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>5</fd><fd_type>pipe</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>1</ref_count><offset>0</offset><protocol>-</protocol><path>-</path></files><files><fd>6</fd><fd_type>socket</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>3</ref_count><offset>0</offset><protocol>TCP</protocol><sendq>0</sendq><recvq>0</recvq><path></path></files><files><fd>7</fd><fd_type>socket</fd_type><fd_flags>read</fd_flags><fd_flags>write</fd_flags><fd_flags>nonblocking</fd_flags><ref_count>3</ref_count><offset>0</offset><protocol>TCP</protocol><sendq>0</sendq><recvq>0</recvq><path>::.8080 ::.0</path></files></89913></files></procstat>

Why XML? Because libxos JSON output is not “real” JSON when procstat‘s running in repeat mode, but that’s a story for another day.

All code examples can be found as a GitHub Gist.

That’s all folks…

Huginn on FreeBSD

Huginn is probably the best automation software that I’ve ever seen. It’s not only easy to use, but also easy to deploy and easy to extend. Unfortunately there’s no FreeBSD port for it, but looks like it’s something wanted by the community, at least according to WantedPorts.

I realized that I have at least 5 accounts on IFTTT, which is also an amazing automation service. However, 3/5 of these accounts were not “my own”. It belonged to our communities. You know, Meetups in Armenia and news listing websites similar to Lobste.rs. So if I get hit by a bus, it will be very hard for our community to operate these accounts, that’s why I wanted to deploy Huginn.

Like a sane person, I deploy in FreeBSD Jails (I recommend you do too!). Which meant there’s no official (or maybe even unofficial?) docs on how to deploy Huginn on FreeBSD.

It’s written in Ruby, which means it should work and should be very easy to ports. I’ll go over the deployment needs without the actual deployment, setup of Jails, or anything similar. Let’s go!

First thing first, you need Ruby thingies:

  • ruby
  • rubygem-bundler
  • rubygem-mimemagic
  • rubygem-rake
  • rubygem-mysql2

Here’s the full command for copy/paster:

pkg instal ruby rubygem-bundler rubygem-mimemagic rubygem-rake rubygem-mysql2

Next, you’ll need gmake for makefiles and node for assets:

pkg install gmake node

This should be enough. I’m also going to install git-tiny so I can follow their updates with ease.

pkg install git-tiny

Okay, let’s make a separate user for Huginn.

pw user add huginn -s /bin/tcsh -m -d /usr/local/huginn

Let’s switch our user

su - huginn

Okay, now I’m going to clone the repo 🙂

git clone --depth=1 https://github.com/huginn/huginn/

At this point you can go do installation.md#4-databases and configure your database.

You should also do cp .env.example .env and configure the environment, make sure to set RAILS_ENV=production

Next, as root, you should execute the following

cd /usr/local/huginn/huginn/ && bundle install

You might get an error saying

In Gemfile:
  mini_racer was resolved to 0.2.9, which depends on

Don’t panic! That’s fine, unfortunately it’s trying to compile libv8 using Gems. Even if we installed the patched version of v8 using pkg, it still doesn’t work. I’ll try to workaround that later.
I an ideal world, all of these Ruby Gems should be ported to FreeBSD, I’m not sure which are ported, so I’ll just be using the bundle command to install them. And that’s why we use Jails 🙂

Anyways, the dependency Gem is mini_racer, comment its line in Gemfile

#gem 'mini_racer', '~> 0.2.4'      # JavaScriptAgent

Now let’s run Bundle again

cd /usr/local/huginn/huginn/ && bundle install

Okay! everything is good!

Let’s also build the assets, this one should be run as the user huginn

bundle exec rake assets:precompile RAILS_ENV=production

NOTE: If you get the following error ExecJS::RuntimeError: ld-elf.so.1: /lib/libcrypto.so.111: version OPENSSL_1_1_1e required by /usr/local/bin/node not found then you need to upgrade your FreeBSD version to the latest patch!

Aaand that’s it, everything is ready.

For the rest of the deployment process, such as the database, nginx, etc., please refer to installation.md

Currently, I’m running Huginn in a tmux session running bundle exec foreman start, but in the future, I’ll write an rc.d script and share it with you, too.

That’s all folks.

Fortune in Times of Need

I was setting up Huginn on FreeBSD, I needed to do some manual testings of commands before I automate them, one of them was using twurl to Tweet. When I was trying to tweet in Armenian, the terminal prompt was giving me a bell. I realized that I needed to change the locale.

When I opened another shell to change the locale, FreeBSD’s fortune printed the following:

In order to support national characters for European languages in tools like
less without creating other nationalisation aspects, set the environment
variable LC_ALL to 'en_US.UTF-8'.

Ah, thank you!

By the way, if you ever saw a fortune that you liked and you needed it later, but didn’t remember the details, you can do fortune -m pattern freebsd-tips, here’s an example:

% fortune -m USB freebsd-tips
%% (freebsd-tips)
If you need to create a FAT32 formatted USB thumb drive, find out its devicename
running dmesg(8) after inserting it. Then create an MBR schema, a single slice and
format it:

# gpart create -s MBR ${devicename}
# gpart add -t fat32 ${devicename}
# newfs_msdos -F 32 -L thumbdrive ${devicename}s1

                -- Lars Engels <lme@FreeBSD.org>


ZFS compression is so good that it cost me 2 hours

So we have this build machine (build0) where we build FreeBSD in Jails and then we mount the src and obj dirs via NFS or we sync them using rsync to destinations so we can run make installworld on not-so-powerful servers.

Couple of days ago we had a network issue at the data center, the switches crashed and we had to reboot them. Turns out I was running rsync on one of our servers, so I decided to make sure that the files were copied.

Like a lazy sysadmin, I run the following commands on both the build0 server, as well as the remote host.

root@build0:~ # du -h -d 0 /usr/local/jails/f130/usr/obj/
 13G    /usr/local/jails/f130/usr/obj/

root@illuriasecurity:~ # du -h -d 0 /usr/obj/
5.5G    /usr/obj/

Hmm, maybe files were not copied properly? So I remove the obj dir and I rsync again.

Looks like the size is 5.5G AGAIN!

So I do a little bit of piping!

root@build0:/usr/local/jails/f130/usr/obj # find . | sort > /tmp/obj_build0.txt

root@illuriasecurity:/usr/obj # find . | sort > /tmp/obj.txt

zvartnots:~ $ scp illuria:/tmp/obj.txt  /tmp/
zvartnots:~ $ scp build0:/tmp/obj_build0.txt /tmp/

zvartnots:~ $ diff /tmp/obj.txt /tmp/obj_build0.txt

Um, no difference?

Looks like the size reported by du was… confusing?

Okay, let’s check the manual of du(1):

     -A		Display the apparent size instead of the disk usage.  This	can be
     		helpful when operating on compressed volumes or sparse files.

Oops, looks like ZFS compression is enabled on my machine…

Let’s try this again!

root@build0:~ # du -h -d 0 -A /usr/local/jails/f130/usr/obj/
 12G    /usr/local/jails/f130/usr/obj/

root@illuriasecurity:~ # du -h -d 0 -A /usr/obj/
 12G    /usr/obj/

Ok! This makes more sense 🙂

Let’s also check with ZFS.

root@illuriasecurity:~ # zfs get compression zroot/usr
zroot/usr  compression  lz4       inherited from zroot

I wonder what’s the build0 server is doing?

root@build0:~ # zfs get compression zroot/usr
cannot open 'zroot/usr': dataset does not exist

Hn o.O ? Oh yeah, I wonder.

root@build0:~ # mount | grep ' / '
/dev/ufs/rootfs on / (ufs, local, journaled soft-updates)

Okay, this makes much more sense now 🙂

That’s all folks!

Techlife Crisis

This is another migration story, like the one that I wrote back in 2020. Unlike the other story, the motivation of this migration is totally different. It’s emotional instead of technical.

Last year a friend of mine got a new job that I referred her to. She passed the interviews and I helped her to get on-boarded as the employer was a friend of mine and I was pretty familiar with their product. The job was remote and she didn’t have a good laptop. Since I have many laptops I ended up giving her my ThinkPad T480s where she ran Ubuntu. As you can tell the employer was a VERY close friend of mine 🙂

All of this meant that I moved back to my MacBook Pro running macOS. I used to like macOS, for me it was always a rock-solid UNIX system with a proper graphical interface.

Unfortunatly these years the UNIX part is not solid anymore and the graphical interface is more iOS-y eye candy than a proper desktop interface.

But I was okay with that, since I spent most of my time in a terminal running vim, ssh, etc. I’d run typical work apps like Mail.app with GPGSuite and a Slack browser client.

But then something snapped in me. I think it was after the car accident. I spent two weeks at home, not able to work. So I started coding on my open-source projects again, doing some patches in FreeBSD, improving code on software that I like and so on.

I realized that I’ve been an Open Source advocate for years, and yet I was in the Apple ecosystem. Not that I don’t like the Apple ecosystem, don’t get me wrong, but as someone who’s been telling the government to use open source, helping them migrate, giving lectures to students about the open source movement and its history, I felt… bad.

I had this MacBook Pro laptop and this iPhone that both control me more than I can control it.

I contacted my friend again, asking if we can swap the laptops and she told me yes. She actually ended up working at our company and now she has a fancy new MacBook Pro while I came back to my lovely ThinkPad T480s running FreeBSD like I wanted in the first place.

As I mentioned, this time it hit me hard, so I decided to escape non-OSS things completely and now I’m running a Pixel 2 with Lineage OS.

There’s a whole story on how I got that Pixel 2 at this day and age and that story is coming soon. And the funniest thing is, as soon as I completed my transaction/migration to Open Source, I got the news that Apple Pay will finally work in Armenia.

Open Source changed my life when I was a kid in Syria, I learned more about computers because of Open Source and while I got distracted with the cute and nice macOS for a while, it’s time to come back home.

Here’s a screenshot

That’s all folks!

WireGuard “dynamic” routing on FreeBSD

I originally wrote about this on my Armenian blog when ISPs started blocking DNS queries during and after the war. I was forces to use either,, or any other major DNS resolver. For me this was a pain because I was not able to dig +trace, and I dig +trace a lot.

After some digging (as mentioned in the Armenian blog) I was able to figure out that this affects only the home users. Luckily, I also run servers at my home and the ISPs were not blocking anything on those “server” ranges, so I setup WireGuard.

This post is not about setting up WireGuard, there are plenty of posts and articles on the internet about that.

Over time my network became larger. I also started having servers outside of my network. One of the fast (and probably wrong) ways of restricting access to my servers was allowing traffic only from my own network.

I have a server that acts as WireGuard VPN Peer and does NAT-ing. That being said, the easiest way for me to start accessing my restricted servers is by doing route add restricted_server_addr -interface wg0.

Turns out I needed to write some code for that, which I love to do!

Anytime that I need to setup a WireGuard VPN client I go back to my Armenian post and read there, so now I’ll be blogging how to do dynamic routing with WireGuard so I read whenever I need to. I hope it becomes handy for you too!

Now, let’s assume you need to add a.b.c.d in your routes, usually you’d do route add a.b.c.d -interface wg0, but this would not work, since in your WireGuard configuration you have a line that says

AllowedIPs = w.x.y.z/24

Which means, even if you add the route, the WireGuard application/kernel module will not route those packets.

To achieve “dynamic” routing we could do

AllowedIPs =

This, however will route ALL your traffic via WireGuard, which is also something you don’t want, you want to add routes at runtime.

What we could do, however, is to ask WireGuard to NOT add the routes automatically. Here’s how.

PrivateKey      = your_private_key
Address         = w.x.y.z/32
Table           = off
PostUp          = /usr/local/etc/wireguard/add_routes.sh %i
DNS             = w.z.y.1

PublicKey       = their_public_key
PresharedKey    = pre_shared_key
AllowedIPs      =
Endpoint        = your_server_addr:wg_port

The two key points here are Table = off which asks WireGuard to not add the routes automatically and PostUp = /usr/local/etc/wireguard/add_routes.sh %i which is a script that does add the routes, where %i is expanded to the WireGuard interface name; could be wg0, could be home0, depends in your configuration.

Now for add_routes.sh we write the following.




for _n in ${networks};
  route -q -n add ${_n} -interface ${interface}

And we can finally do wg-quick up server0.conf

If you need to add another route while WireGuard is running, you can do

route add another_restricted_server -interface wg0

Okay, what if you need to route everything while WireGuard is running? Well, that’s easy too!

First, find your default gateway.

% route -n get default | grep gateway
    gateway: your_gateway

Next, add a route for your endpoint via your current default gateway.

route add you_server_addr your_gateway

Next, add TWO routes for WireGuard.

route add     -interface wg0
route add   -interface wg0

So it’s the two halves of the Internet 🙂

That’s all folks!

VoidLinux in FreeBSD Jail; with init

Two important things happened this week for me.

First, Faraz asked me if I can rename my Jail manager to something other than Jailio because he got that domain for his Jailer manager already. So I named it

Second, I was able to run a complete Linux system using Jailer. While the repo for Jailer is not released yet (we are auditing for possible security issues), I would like to share how I was able to run VoidLinux in a Jail.

Since Jailer is not announced yet, I will give the examples using jail.conf, as most people either are or should be familiar with its concepts.

I went with VoidLinux because I am able to run the init process without its need to be running as PID1.

Let’s start, shall we?

First, ZFS dataset for our jail!

zfs create zroot/jails/voidlinux

Next we need to fetch the base system of VoidLinux. Luckily they do provide it on their website.

fetch https://alpha.de.repo.voidlinux.org/live/current/void-x86_64-ROOTFS-20210218.tar.xz

Now we can extract this into our dataset

tar xf void-x86_64-ROOTFS-20210218.tar.xz -C /usr/local/jails/voidlinux/

You might get an error that ./usr/bin/iputils-ping: Cannot restore extended attributes: security.capability, which is fine, I think?

If you are on FreeBSD 12.2-RELEASE or later, now you need to enable the Linuxulator.

service linux enable; service linux start

Now you can at least chroot into the system.

chroot /usr/local/jails/voidlinux/ /bin/bash

If everything is fine until now, perfect.

Now we need to add a root user into the system.

root@host:~ # cd /usr/local/jails/voidlinux/etc/
root@host:/usr/local/jails/voidlinux/etc # echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd
root@host:/usr/local/jails/voidlinux/etc # pwd_mkdb -d ./ -p master.passwd
pwd_mkdb: warning, unknown root shell

Execute the rest of the commands in Void.

root@host:~ # chroot /usr/local/jails/voidlinux/ /bin/bash
bash-5.1# cd /etc/
bash-5.1# pwconv 
bash-5.1# grpconv 
bash-5.1# passwd 
New password: 
Retype new password: 
passwd: password updated successfully
bash-5.1# exit

If all went fine, then the system is ready to be run as a Jail!

First we need to make an fstab for the system.

Create a file at /usr/local/jails/voidlinux/etc/fstab.pre and insert the following inside

devfs       /usr/local/jails/voidlinux/dev      devfs           rw                      0   0
tmpfs       /usr/local/jails/voidlinux/dev/shm  tmpfs           rw,size=1g,mode=1777    0   0
fdescfs     /usr/local/jails/voidlinux/dev/fd   fdescfs         rw,linrdlnk             0   0
linprocfs   /usr/local/jails/voidlinux/proc     linprocfs       rw                      0   0
linsysfs    /usr/local/jails/voidlinux/sys      linsysfs        rw                      0   0
/tmp        /usr/local/jails/voidlinux/tmp      nullfs          rw                      0   0

Next, let’s create a loopback interface for networking. Oh yes, VNET is not supported yet, but I’m working on a patch 🙂

ifconfig lo1 create
ifconfig lo1 inet up # sorry, was unavailable :P

Okay, time to create our Jail conf!


voidlinux {
    $id     = "1";
    $ipaddr = "";
    $mask   = "";
    $domain = "srv0.bsd.am";
    devfs_ruleset  = 4;
    mount.fstab = "${path}/etc/fstab.pre";

    exec.start     = "/bin/sh /etc/runit/2 &";
    exec.stop      = "/bin/sh /etc/runit/3";

    ip4.addr      = "${ipaddr}";
    interface     = "lo1";
    host.hostname = "${name}.${domain}";
    path = "/usr/local/jails/voidlinux";
    exec.consolelog = "/var/log/jail-${name}.log";

Let’s check?

# jls
   JID  IP Address      Hostname                      Path
     1    voidlinux.srv0.bsd.am         /usr/local/jails/voidlinux

And the process tree?

# ps auxd -J voidlinux
root 35182  0.0  0.1 2320 1428  -  SsJ  21:09   0:00.12 runsvdir -P /run/runit/runsvdir/current log: ot set SO_PASSCRED: Protocol not available\ncould not set SO_PASSCRED: Protocol
root 35190  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty6
root 35397  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty6 38400 linux
root 35191  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty1
root 35396  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty --noclear tty1 38400 linux
root 35192  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty5
root 35398  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.01 `-- agetty tty5 38400 linux
root 35193  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty2
root 35393  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty2 38400 linux
root 35194  0.0  0.1 2168 1396  -  RsJ  21:09   0:00.12 - runsv udevd
root 35195  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty3
root 35394  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty3 38400 linux
root 35196  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty4
root 35390  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty4 38400 linux

You may jexec now 🙂

# jexec voidlinux /bin/bash
bash-5.1# uname -a
Linux voidlinux.srv0.bsd.am 3.2.0 FreeBSD 12.2-RELEASE-p6 GENERIC x86_64 GNU/Linux

Let’s check networking?

bash-5.1# ping -c 1
ping: WARNING: setsockopt(ICMP_FILTER): Protocol not available
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.069 ms

--- ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.069/0.069/0.069/0.000 ms

There you go! Well, things that are related to netlink might not work, but other than that it’s okay.

I did have some problems while installing packages, something about too many levels of symbolic links. Here’s the exact output when I was trying to install the curl package

[*] Unpacking packages
libev-4.33_1: unpacking ...
ERROR: libev-4.33_1: [unpack] failed to extract file `./usr/lib/libev.so.4': Too many levels of symbolic links
ERROR: libev-4.33_1: [unpack] failed to extract files: Too many levels of symbolic links
ERROR: libev-4.33_1: [unpack] failed to unpack files from archive: Too many levels of symbolic links
Transaction failed! see above for errors.

Now, I did not find the time to fix this yet, but if you have any idea, please let me know or comment below 🙂

So, what do we have here? A Linux Jail, running VoidLinux, with init, so you can also run services, and basic networking for it.

That’s all folks…

Linux is dead, long-live Docker monoculture

Full Discloser: While reading this blog post, please put yourself in my shoes. You’ve been looking around for a simple monitoring solution, you found some. None of the some are working because you use an Operating System that is used by Apple, WhatsApp, Netflix and many more, but developers think that everyone, everywhere, runs either macOS or Linux. And they all use Docker.

A while back Rubenerd wrote that he’s not sure that UNIX won and how Linux created a monoculture of assuming everything is supposed to run on Linux.

For me, this was not much of a problem, I can run Linux binaries on FreeBSD, I even watch Netflix using Linuxulator.

But now things are on another level, WAY another level.

I have a simple monitoring setup using cron, Grafana, InfluxDB and ping. It basically pings my servers and sends me a telegram message if they are down.

I set that up years ago, but now I have more public facing infrastructure that other people use as well, such as an Armenian Lobsters instance, Jabber.am, a WriteFreely instance and more.

As a self-respecting Ops, I wanted to make a simple dashboard for my users to see the uptime status of these services as well. First, they won’t bug me asking if something is not working; they will SEE, that, SSL/TLS certificate is expired, or the network is an issue, or that the server is down.


So I started hunting on the internet for some software that do just that.

The first one that came to my mind was Gatus. I’ve used Gatus before for one of my clients, I like it a lot. It’s simple, it does what it’s supposed to do.

As a sane person, I fetched the code from GitHub using fetch, extracted the tarball and ran make. Nothing happens. Let’s see the Makefile, shall we?

Docker executed in Make

Oh boy, if only, only, I had Docker, all my problems would be solved. First of all, let’s talk about the fact that this Makefile is used as a… script. There’s no dependencies in the targets!

Okay, let’s read that Dockerfile. Executing the scripts inside it should help out, aye?

# Build the go application into a binary
FROM golang:alpine as builder
RUN apk --update add ca-certificates
COPY . ./
RUN CGO_ENABLED=0 GOOS=linux go build -mod vendor -a -installsuffix cgo -o gatus .

# Run Tests inside docker image if you don't have a configured go environment
#RUN apk update && apk add --virtual build-dependencies build-base gcc
#RUN go test ./... -mod vendor

# Run the binary on an empty container
FROM scratch
COPY --from=builder /app/gatus .
COPY --from=builder /app/config.yaml ./config/config.yaml
COPY --from=builder /app/web/static ./web/static
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

There are multiple things wrong in me this.

First, please stop putting your binaries in /app, please, pretty-please? We have /usr/local/bin/ for that.

Second, I thought that running go build without GOOS=linux would solve all of my problems. I was wrong, very wrong.

root@mon:~/gatus/gatus-2.8.1 # env CGO_ENABLED=0 go build -mod vendor -a -installsuffix cgo -o gatus .
package github.com/TwinProduction/gatus
        imports github.com/TwinProduction/gatus/config
        imports github.com/TwinProduction/gatus/storage
        imports github.com/TwinProduction/gatus/storage/store
        imports github.com/TwinProduction/gatus/storage/store/sqlite
        imports modernc.org/sqlite
        imports modernc.org/libc
        imports modernc.org/libc/errno: build constraints exclude all Go files in /root/gatus/gatus-2.8.1/vendor/modernc.org/libc/errno

Okay, check this out, the package is called modernc.org/sqlite and it says:

Package sqlite is a CGo-free port of SQLite.

SQLite is an in-process implementation of a self-contained, serverless, zero-configuration, transactional SQL database engine.

Of course it is. Looks like I have to port all of this to FreeBSD. Which, don’t get me wrong, I’m okay with doing that, but I thought that we have POSIX for a reason. notsomuch.

Okay, I’m an open-source guy, I’ll spend some time this weekend to port this to FreeBSD. Let’s look for another solution!

Here’s another one, it’s called statping, also written in Go, the readme is so promising.

No Requirements

Statping is built in Go Language so all you need is the precompile binary based on your operating system. You won’t need to install anything extra once you have the Statping binary installed. You can even run Statping on a Raspberry Pi.

Sounds good! Let’s try it out.

Again, I fetch the tarball, I extract and I bake make.

apt executed in Make

Of course it requires apt! Because not only we all run Linux, be we all run a specific distribution of Linux with a specific package manager.

While tweeting with anger, Daniel pointed out that I should tell them kindly and it’ll work out. I’m sure it will. Let’s hope I can make it work first. I don’t like just opening issues. I’d rather send a patch directly.


Overall, now I understand why most *BSD folks use, what’s the word here? ah, yes, old-school software on their systems, like Nagios and the rest.

The developers of the New World Order will assume, always, you are running Linux, as Ubuntu, and you always have Docker.

Hopefully this weekend I will be able to port these software to FreeBSD, otherwise I will just use the Linux layer.

Like Rubenerd said, I am thankful that the mainstream-ness of Linux helped other Unix systems as well, but monocultures are destroying what people have spent years to improve.

Hopefully, next week, I will write a blog post on how to fix these issues and how I got all of those up and running.

That’s all folks…

Two Colons Equals Modules

Days ago I tweeted a shell function which is part of jailio’s code base. Jailio is a project I’ve been working on for the last 6 months. As the name implies, it’s a container management software for FreeBSD Jails.

It has two unique things compared to other Jail management software. First of all, it has no dependencies, it’s written purely in Shell. You can say the same about BastilleBSD, however, Jailio’s second unique thing is that it uses base tools only and requires the base system only. For example, you need to have bastille_enable in BastilleBSD, it also uses its own config files, etc. In Jailio, you need to have jail_enable, because technically Jailio automates jail.conf files. It also uses my patch to automate the jail.confs in /etc/jail.conf.d.

Anyway, back to our topic about Colons and Modules.

I like modules, I got introduced to them when I started programming in school. In Syria, we learn programming at 7th grade but in our school we started a year early, so 6th grade. We always start with block diagrams and then Turbo Pascal!

Yes, 16-bit Turbo Pascal was my first programming language and it had the concept of modules which we called Units.

And then you have languages like C or Shell which don’t have modules. If you use modules you KNOW that it’s hard not to use modules after that.

While reading the source code of vm-bhyve I learned that you can use two colons (::) as part of the function name, which can give you an amazing new superpower to take over the world write cleaner code.

For me this was a life-changer. I write a LOT of Shell code. I ship them to production too. No, you don’t need to write everything in a fancy new language and run it on kubernetes, you can always use simple languages like Shell and run them in a FreeBSD Jail. Or in my case, write in Shell to automate FreeBSD Jails.

Here’s an example code with “modules” in Shell. Note, this works in FreeBSD’s shell, I have not tested other Shells yet.



. ./mod1.sh




  printf "Here I am, rock you like a hurricane\n"
antranigv@pingvinashen:~ % ./main.sh 
Here I am, Rock you like a hurricane

As you can see it all relies on the concept that the function name itself has two colons in its name.

Here’s the code from jailio that I tweeted.

  expr $(
    ( grep -s '$id' /etc/jail.conf.d/* || echo '$id = "0";' ) |
    awk -F '[="]' '{print $3}' |
    sort -h |
    tail -1
  ) + 1

After tweeting the code above Annatar replied that this should NOT work elsewhere and that’s how I got introduced to The Heirloom Project which provides traditional implementations of the original Unix tools from the original Unix source code.

Hopefully, I will see more people using “modules” in Shell scripts. Hopefully this trick works in other Shell implementations like Bash and zsh.

That’s all folks.

The OS App vs The Browser OS

I like listening to online radios like anonradio and DeepHouseRadio, instead of me trying to organize my local library or listening the same music over and over again on Deezer, I get lazy and just use their HTTP link.

Like a sane person, I would use a media player to “open” these HTTP radio links. On my FreeBSD machine, all I need to do is mplayer http://the.domain/path/to/content, but on macOS it would not be that simple.

The default media player on macOS is QuickTime. Here is where my problems start. I open QuickTime Player, I set the location to the HTTP link and it all works fine. Until it doesn’t. A small network lag and it stops playing completely.

I am usually connected to the internet via a cable in my office or the house, but when I go wireless, there’s a blind spot in one of the rooms. My FreeBSD laptop with mplayer handles it all fine, but QuickTime? Not so much.

So I decided to use the “other” “Operating System” in macOS, also known as a browser, in this case Firefox. I open the link and it all works fine. Even if there’s a network lag, Firefox would handle it fine.

It’s sad funny how browsers are handling things better than native desktop programs these days.

While writing this blog-post I realized that macOS has another media player known as Music.app, so will try with that as well, let’s see how it will handle it.

That’s all folks.