I started getting into retro computing again, but this time also for phones, so I’m blogging from a BlackBerry phone 🙂
Let’s see if it works!
Antranig Vartanian
October 9, 2025
Antranig Vartanian
September 21, 2025
There’s a lot of work that needs to be done on Jailer.
To start with, I have finally decided on the release model. The main development branch will be main, similar to FreeBSD, and then the releases will be tagged.
I’m not sure how I feel about branches, as I don’t want to have any kind of LTS versions, but I might need to understand if there’s anything missing in FreeBSD that I would need.
Another problem that I have right now is the fact that we don’t have a Port. I’m not happy with Jailer enough to make a Port for it, but maybe in the coming days.
- Fix the Git stuff that I don’t care about but I should
- Create a stable non-development version/tag
- Create more lists like this, because, let’s be honest, no one can work without lists.
I do wonder if I can pull an all nighter and finish everything that I hope to finish. Lets see.
Antranig Vartanian
July 19, 2025
Sometimes all I need to do is
sudo tmutil listlocalsnapshots / \
| grep TimeMachine \
| cut -d . -f 4 \
| xargs -I% sudo tmutil deletelocalsnapshots %
And suddenly I have like 70GB of more storage on my Mac.
I wish macOS was less… black magic-y.
Antranig Vartanian
July 7, 2025
Sometimes, when I have a long running process on my computer, like scp, rsync or curl I need to know when it’s done, so I don’t check the terminal every minute. That’s when I realized that this is a perfect fit for Apple Script and pwait.
Basically, it looks something like this:
pwait `pgrep xargs` \
&& osascript -e \
'display notification "Extraction Done" with title "large xargs"'
Now I can go back to work, and I don’t have to check if the job is done every minute. A notification will appear when it’s done.
For remote jobs, I have realized, that I can send a notification to myself, kind of like a self-hosted Pushover, using XMPP, I don’t want to go into the details, but that seems like to be a good solution as well. Maybe I should just use Pushover anyway.
Analyzing FreeBSD’s Package Mirror GeoDNS
One of the things that makes FreeBSD an amazing operating system is its package manager, pkg(8). Many people will keep arguing with me saying that apt or pacman is better, but pkg has some of the best features out there. Don’t even get me started on number of packages. A friend was telling me “but how can I use FreeBSD when you have 1/3rd number of packages that Debian has?”. Again, this is a lie. While Debian has 80K packages, each software is divided into -doc and -dev, while on FreeBSD a simple pkg install will install everything that a package needs.
But let’s not talk about how awesome FreeBSD’s pkg is, let’s talk about it’s issues.
Couple of years ago, I noticed that downloading from FreeBSD package mirror was veeeeeery slow. Turns out that based on GeoDNS Armenia was connecting to the South African package mirror pkg0.jinx.FreeBSD.org by default.
Lucky for me, there was a package to help you figure out which one is fastest for you, named fastest_pkg. I ran it and I learned that the Frankfurt package mirror was the fastest for me.
Initially, I changed the config in FreeBSD.conf as recommended by fastest_pkg, but I’m not a single server guy, I have dozens of hosts with hundreds of jails.
Solution? I asked the Cluster Admins to set Armenia’s GeoDNS to pkg0.fra.FreeBSD.org.
But servers come and go, and the Frankfurt server got decommissioned a while back. Luckily the project now has a server in… Sweden!
But my download speed is slow. Somehow Armenia defaulted back to South Africa.
Is this an issue for me, or is this an issue for everybody?
(Sidenote: turns out fastest_pkg has been broken for a while, ever since pkg moved to a new package format, so we also made a patch and submitted a PR)
During today’s FreeBSD “Ask the experts: AMA” we used RIPE Atlas to measure the speeds from all around the world, to all package mirror hosts.
All of the measurements can be found here.
So, here’s the goal
- Measure from around the world to all of the package mirror hosts (done).
- Measure from around the world to
pkg.FreeBSD.orgto get the default GeoDNS host for each area/country/network (in progress, recurring every 6 hours for the next 6 days) - Find mis-configurations between “fastest” and “default” (TODO).
While the big measurements is still running I tried to use my eyes to see if there’s anything GREEN (aka fast connection) in direct package mirror host measurements whilst being YELLOW/RED (aka bad connection) when connecting to pkg.FreeBSD.org.
Unsurprisingly, we found one! It was in Mombasa, Kenya.
Here’s what it looks like while connecting directly:
{
"fw": 5080,
"mver": "2.6.2",
"lts": 25,
"dst_name": "pkg0.jinx.freebsd.org",
"ttr": 10639.830509,
"af": 4,
"dst_addr": "196.10.53.168",
"src_addr": "160.119.216.205",
"proto": "ICMP",
"ttl": 59,
"size": 1000,
"result": [
{
"rtt": 47.434297
},
{
"rtt": 47.39957
},
{
"rtt": 47.412379
},
{
"rtt": 47.369089
},
{
"rtt": 47.851894
}
],
"dup": 0,
"rcvd": 5,
"sent": 5,
"min": 47.369089,
"max": 47.851894,
"avg": 47.493445799999996,
"msm_id": 110487154,
"prb_id": 22215,
"timestamp": 1750269643,
"msm_name": "Ping",
"from": "160.119.216.205",
"type": "ping",
"group_id": 110487154,
"step": null,
"stored_timestamp": 1750269644
}
And here it is via GeoDNS’s defaults:
{
"fw": 5080,
"mver": "2.6.2",
"lts": 10,
"dst_name": "pkg.freebsd.org",
"ttr": 10465.87807,
"af": 4,
"dst_addr": "173.228.147.98",
"src_addr": "160.119.216.205",
"proto": "ICMP",
"ttl": 52,
"size": 1000,
"result": [
{
"rtt": 240.19025
},
{
"rtt": 244.967071
},
{
"rtt": 236.466414
},
{
"rtt": 238.189775
},
{
"rtt": 233.570938
}
],
"dup": 0,
"rcvd": 5,
"sent": 5,
"min": 233.570938,
"max": 244.967071,
"avg": 238.6768896,
"msm_id": 110489920,
"prb_id": 22215,
"timestamp": 1750270943,
"msm_name": "Ping",
"from": "160.119.216.205",
"type": "ping",
"group_id": 110489920,
"step": null,
"stored_timestamp": 1750270944
}
Clearly, it’s set to pkg0.chi.freebsd.org (173.228.147.98) by default. That’s Chicago. That’s halfway around the world. It took it 238ms on average. While it took the same probe 47ms on average when connecting to pkg0.jinx.freebsd.org directly.
Clearly, I’m not the only one.
Hopefully after collecting data for 6 days, we can start analyzing and improving the GeoDNS setup that we have.
A good suggestion from crest was to have a pkg plugin that does the work of fastest_pkg on the fly. We’ll try that one day.
If anyone is interested in helping me out, feel free to contact me over email, IRC or Discord.
That’s all folks…
Migrating to FreeBSD -CURRENT with pkgbase using Boot Environments
Couple of months ago I got my sister a MacBook Air with the new M4 chip, and she was happier than ever. However, I was the happiest, as she gave me her old MacBook Pro 2015.
As always, I installed FreeBSD on it, it was FreeBSD 14.2-RELEASE at the time, installed the required drivers, attached a Thunderbolt 3 to Ethernet adapter to it (thank you to whoever made it work via the bge drivers) and… nothing. It’s been sitting on the floor for months now.
Last week I decided I’m going to do my FreeBSD work again. I have a todo list with 20+ action items in them.
But first thing first, I need a -CURRENT machine so I can do my development.
So, here’s a simple guid on how to migrate to FreeBSD -CURRENT using PkgBase.
First we need to setup the pkgbase repo, so pkg can use it. I will be using pkg.FreeBSD.am, which is my regional pkg cache, but other than that, everything is pretty standard.
root@mbp15srv0:~ # mkdir -p /usr/local/etc/pkg/repos
root@mbp15srv0:~ # cat - > /usr/local/etc/pkg/repos/FreeBSD-base.conf
FreeBSD-base {
url = "pkg+https://pkg.freebsd.am/FreeBSD:15:amd64/base_latest/";
mirror_type = "srv";
signature_type = "fingerprints";
fingerprints = "/usr/share/keys/pkg";
enabled = yes;
}
Next up, we create and mount a Boot Environment, that way, if we screw something up, we can always come back.
root@mbp15srv0:~ # bectl list
BE Active Mountpoint Space Created
default NR / 2.99G 2025-04-20 22:45
root@mbp15srv0:~ # bectl create 15.0-CURRENT-pkgbase
root@mbp15srv0:~ # bectl mount 15.0-CURRENT-pkgbase
/tmp/be_mount.6i35
But before we start, let’s make sure we’re on the latest version of pkg, since the devs keep adding cool features for PkgBase.
root@mbp15srv0:~ # pkg install pkg
Updating FreeBSD.am repository catalogue...
Fetching meta.conf: 0%
FreeBSD.am repository is up to date.
Updating FreeBSD-base repository catalogue...
Fetching meta.conf: 0%
FreeBSD-base repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be UPGRADED:
pkg: 2.1.2 -> 2.1.4 [FreeBSD.am]
Number of packages to be upgraded: 1
12 MiB to be downloaded.
Proceed with this action? [y/N]: y
[1/1] Fetching pkg-2.1.4.pkg: 100% 12 MiB 12.5MB/s 00:01
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 2.1.2 to 2.1.4...
[1/1] Extracting pkg-2.1.4: 100%
Updating FreeBSD.am repository catalogue...
Fetching meta.conf: 0%
FreeBSD.am repository is up to date.
Updating FreeBSD-base repository catalogue...
Fetching meta.conf: 0%
FreeBSD-base repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The most recent versions of packages are already installed
We need to do pkg update so we get the repo metadata. We’ll also set the ABI to 15, since we’ll be installing that.
root@mbp15srv0:~ # env ABI="FreeBSD:15:amd64" pkg update -f -r FreeBSD-base
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Updating FreeBSD-base repository catalogue...
pkg: Repository FreeBSD-base has a wrong packagesite, need to re-create database
Fetching meta.conf: 100% 179 B 0.2kB/s 00:01
Fetching data.pkg: 100% 52 KiB 53.2kB/s 00:01
Processing entries: 0%
Newer FreeBSD version for package FreeBSD-zoneinfo:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1500043
- running userland: 1500000
Ignore the mismatch and continue? [y/N]: y
Processing entries: 100%
FreeBSD-base repository update completed. 554 packages processed.
FreeBSD-base is up to date.
Now we can do the installation using -r, which will set the root directory. I’m also using -Fy which will fetch-only, and assume yes for everything.
root@mbp15srv0:~ # env ABI="FreeBSD:15:amd64" pkg -r /tmp/be_mount.6i35/ install -Fy -r FreeBSD-base -g 'FreeBSD-*'
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Updating FreeBSD-base repository catalogue...
Fetching meta.conf: 0%
FreeBSD-base repository is up to date.
FreeBSD-base is up to date.
The following 554 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
FreeBSD-acct: 15.snap20250612001522 [FreeBSD-base]
FreeBSD-acct-dbg: 15.snap20250612001522 [FreeBSD-base]
FreeBSD-acct-man: 15.snap20241026125659 [FreeBSD-base]
FreeBSD-acpi: 15.snap20250612160001 [FreeBSD-base]
FreeBSD-acpi-dbg: 15.snap20250612160001 [FreeBSD-base]
[...]
FreeBSD-zfs-dev-lib32: 15.snap20250612160001 [FreeBSD-base]
FreeBSD-zfs-lib32: 15.snap20250612160001 [FreeBSD-base]
FreeBSD-zfs-man: 15.snap20250531144220 [FreeBSD-base]
FreeBSD-zoneinfo: 15.snap20250521200023 [FreeBSD-base]
Number of packages to be installed: 554
The process will require 6 GiB more space.
1 GiB to be downloaded.
[1/507] Fetching FreeBSD-kernel-minimal-15.snap20250613001933.pkg: 100% 39 MiB 861.9kB/s 00:47
[...]
[371/484] Fetching FreeBSD-smbutils-dev-lib32-15.snap20250612160001.pkg: 100% 86 KiB 87.8kB/s 00:01
[372/484] Fetching FreeBSD-libsqlite3-dbg-lib32-15.snap20250612160001.pkg: 100% 1 MiB 1.2MB/s 00:01
Checking integrity... done (0 conflicting)
And finally, I can do the actual installation. Again, not forgetting about ABI, and BACKUP_LIBRARIES with BACKUP_LIBRARY_PATH.
root@mbp15srv0:~ # env ABI="FreeBSD:15:amd64" BACKUP_LIBRARIES=true BACKUP_LIBRARY_PATH=/tmp/be_mount.6i35/usr/local/lib/compat/pkg pkg -r /tmp/be_mount.6i35/ install
-Uy -r FreeBSD-base -g 'FreeBSD-*'
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Checking integrity... done (0 conflicting)
The following 554 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
FreeBSD-acct: 15.snap20250612001522 [FreeBSD-base]
FreeBSD-acct-dbg: 15.snap20250612001522 [FreeBSD-base]
FreeBSD-acct-man: 15.snap20241026125659 [FreeBSD-base]
[...]
Number of packages to be installed: 554
The process will require 6 GiB more space.
[1/554] Installing FreeBSD-acct-15.snap20250612001522...
[1/554] Extracting FreeBSD-acct-15.snap20250612001522: 100%
[2/554] Installing FreeBSD-acct-dbg-15.snap20250612001522...
[2/554] Extracting FreeBSD-acct-dbg-15.snap20250612001522: 100%
[3/554] Installing FreeBSD-acct-man-15.snap20241026125659...
[3/554] Extracting FreeBSD-acct-man-15.snap20241026125659: 100%
[4/554] Installing FreeBSD-acpi-15.snap20250612160001...
[4/554] Extracting FreeBSD-acpi-15.snap20250612160001: 100%
[5/554] Installing FreeBSD-acpi-dbg-15.snap20250612160001...
[...]
[553/554] Installing FreeBSD-zfs-man-15.snap20250531144220...
[553/554] Extracting FreeBSD-zfs-man-15.snap20250531144220: 100%
[554/554] Installing FreeBSD-zoneinfo-15.snap20250521200023...
[554/554] Extracting FreeBSD-zoneinfo-15.snap20250521200023: 100%
Done!
Now we can chroot in and do some changes.
chroot /tmp/be_mount.6i35/ /bin/sh
We will see a lot of files ending with .pkgsave, these are the old files. Some things are required for us, such as sshd_config, master.passwd, sysctl.conf. Now, this is a pretty fresh installation, so I don’t need to do other changes, but you might!
root@mbp15srv0:/ # cp /etc/ssh/sshd_config.pkgsave /etc/ssh/sshd_config
root@mbp15srv0:/ # cp /etc/master.passwd.pkgsave /etc/master.passwd
root@mbp15srv0:/ # cp /etc/group.pkgsave /etc/group
root@mbp15srv0:/ # pwd_mkdb -p /etc/master.passwd
root@mbp15srv0:/ # cp /etc/sysctl.conf.pkgsave /etc/sysctl.conf
I think this is a good time to delete the other .pkgsave files. The FreeBSD Wiki says that I also need to delete /boot/kernel/linker.hints
# find / -name '*.pkgsave' -delete
# rm /boot/kernel/linker.hints
Back on the host, outside the chroot, I can finally activate the BE, at least temporarily, so only for the next boot.
root@mbp15srv0:~ # bectl list
BE Active Mountpoint Space Created
15.0-CURRENT-pkgbase - /tmp/be_mount.6i35 4.62G 2025-06-13 13:05
default NR / 3.07G 2025-04-20 22:45
root@mbp15srv0:~ # bectl unmount 15.0-CURRENT-pkgbase
root@mbp15srv0:~ # bectl activate -t 15.0-CURRENT-pkgbase
Successfully activated boot environment 15.0-CURRENT-pkgbase
for next boot
root@mbp15srv0:~ # bectl list
BE Active Mountpoint Space Created
15.0-CURRENT-pkgbase T - 4.62G 2025-06-13 13:05
default NR / 3.07G 2025-04-20 22:45
A reboot… with fingers crossed!
shutdown -r now
The machine is up and running!
root@mbp15srv0:~ # uname -a
FreeBSD mbp15srv0.evn0.loc.illuriasecurity.com 15.0-CURRENT FreeBSD 15.0-CURRENT main-n277903-a14573de29de GENERIC amd64
Let’s make this BE the default now.
root@mbp15srv0:~ # bectl list
BE Active Mountpoint Space Created
15.0-CURRENT-pkgbase N / 4.62G 2025-06-13 13:05
default R - 3.08G 2025-04-20 22:45
root@mbp15srv0:~ # bectl activate 15.0-CURRENT-pkgbase
Successfully activated boot environment 15.0-CURRENT-pkgbase
root@mbp15srv0:~ # bectl list
BE Active Mountpoint Space Created
15.0-CURRENT-pkgbase NR / 7.54G 2025-06-13 13:05
default - - 666M 2025-04-20 22:45
Finally, I will re-install pkg (not sure if that’s needed) and update my packages.
pkg bootstrap -f
pkg upgrade
All good! Now I can get to work 🙂
Special thanks to everyone who wrote the Wiki article and to ivy on the FreeBSD IRC channel for answering my questions.
That’s all folks…
Antranig Vartanian
June 12, 2025
A while back I needed to get the input voltage from one of our UPSes, so I used bsnmpwalk(1) to get the information needed and ran it in a script with a loop and sleep. Running it in tmux(1), of course.
#!/bin/sh
lastoff="maybe"
while true;
do
inpvol=$(bsnmpwalk -o quiet -s public@172.20.42.101 1.3.6.1.2.1.33.1.3.3.1.3)
[ $? != 0 ] && \
curl -s -X POST \
https://api.telegram.org/botXXX:YYY/sendMessage \
-d chat_id=-ZZZ \
-d text="Something is wrong with the SNMP server"
[ "${inpvol}" -lt 200 ] && \
curl -s -X POST \
https://api.telegram.org/botXXX:YYY/sendMessage \
-d chat_id=-ZZZ \
-d text="Power seems to be off. I see Input Voltage as ${inpvol}" && \
lastoff="true"
[ "${inpvol}" -ge 200 ] && [ "${lastoff}" == "true" ] && \
curl -s -X POST \
https://api.telegram.org/botXXX:YYY/sendMessage \
-d chat_id=-ZZZ \
-d text="Power back on. I see Input Voltage as ${inpvol}" && \
lastoff="false"
printf "%s --- %s\n" "$(date)" "${inpvol}"
sleep 60
done
This got the job done, but I guess there’s place for improvement (leave a reply).
Anyways, I kept forgetting that I need to run the script in tmux after reboots, so I decided to use daemon(8).
touch /etc/rc.local
chmod +x /etc/rc.local
cat - >> /etc/rc.local
#!/bin/sh
daemon -u nobody -r -R 5 -f -t ups-notifier -o /var/log/ups_notifier.log /usr/local/bin/ups_notifier.sh
Again, there’s a place for improvement, specifically I can use a proper rc.d(8) script, yet again, this gets the job done.
Gotta say, I love the simplicity of FreeBSD.
Antranig Vartanian
February 27, 2025
For years I’ve been a night owl, staying up late until 5AM, hacking on code, chatting with friends and even drinking coffee at that late hour. I’ve been told that it’s a genetic thing, that people don’t choose what kind of a person they are, morning or evening. And yet, when I tried to find my “natural sleep cycle”, looks like I’ve been functioning at my best when I sleep around 10PM and wake up at 5AM. This has been a very weird experience, but what else do I not know about myself? I am not sure. The unknown unknown is a very weird place, unlike the known unknown.
In either case, I will keep a detailed log of my sleep for the coming months and see how if I actually function better, or is it just an illusion.
What’s your sleep cycle? when do you hack on code? or do you stay up late watching YouTube videos and doom-scroll on social media?
Syrian-Armenian Stories: My first hack, Arcades, Syrian Parties and Mortal Kombat
Couple of days ago my fiancée was watching an Armenian internet show named “Tany Layva” which roughly translates to “Live at home” that aired during COVID-19. The interesting part for me was not the show itself (frankly speaking I could not care less about Armenia’s pop culture) but the intro music triggered something in me… You can listen to it here.
Oh my god! It’s the theme song of Mortal Kombat! I think it was in the 1995 movie too, which is probably the first time I heard it.
However the trigger was not the song itself, but something that the song represents for me.
You see, until that moment I thought that the earliest hack that I remember was the PlayStation Swap Trick, which my father showed me in the year 2002. Growing up we did not have a lot of money, so we waited for the release of the PlayStation 2 in Syria, allowing us to buy the Original PlayStation at a discounted price. But games were expensive, and they are getting more expensive (that’s a topic for another day), however, the Syrian gaming community was very active in making bootleg/imported game CDs, selling them at a cheaper price. So how do you play these games?
Well, you insert an original disc (in my case, I kept using the Crash Team Racing disc), you wait for the PlayStation logo to appear, and you swap the disc with the one that you want to play.
This was my earliest memory of a “hack”, where you “bypass” a system check.
But why Mortal Kombat? Why was that important for my brain? Well, it reminded me of an event.
It was probably the year 2000 or 2001, we were at a family party. A wedding party maybe, an engagement party, or just a party overall? Not sure, but I remember two things: It was near a swimming pool and I was bored as hell.
Our family friends had a daughter, Natalie, she was roughly my age, and both of us were not just bored during that event, we were actively looking for something to do. Literally anything.
We were at the “Al Galaa Family Club Swimming Pool and Restaurant”, a place that still exists today. It was a pretty large area with multiple floors and a swimming pool, where people were dancing next to it.
At the basement we found our salvation. An Arcade. Multiple games. Oh my god, there’s even a multiplayer game.
We run to our dads, and we ask them to boot the multiplayer game. My father and Uncle Vartan connect the arcade to the socket, the game boots up. A very violent game. But we’re not Americanized are we? we don’t care about game age ratings, we care about having fun! We start playing the game. Very, very addictive. I play against my dad, my dad and uncle Vartan play against each other, and at the final round, I play against Natalie. It was indeed a Mortal Kombat.
You see, when Syrians party, they tend to stay up late, by late I mean until 6 in the morning. And just when I thought that this arcade is exactly what I need for the next 7 hours, the game asks for a coin.
I mean, sure, 10 Syrian Liras were not much back then, but for 7 hours? That would be very expensive.
And then my dad asked the question that would rewire my brain forever… “why didn’t it ask for money the first time?”
Ah yes. The free trial 🙂 First game was free, next game is not.
It’s the middle of the night, everyone’s at the pool, they are dancing and drinking, no one is watching us. What could possibly go wrong?
My dad disconnects and reconnects the game’s electricity socket, the boot menu shows up again and says “Press Start To Play”… FREE GAMES ALL NIGHT!
And just like that, we spend the next 6-7 hours, until the dawn of light, playing arcade games, from Mortal Kombat, Sonic The Hedgehog, to some random racing games that I don’t even remember their names.
And yes, Natalie did have fun as well!
I told my mom and dad this story couple of days ago, they remembered the event, but their reaction was “how can you possibly remember this? You were not even 6 years old”. Well, what can I say, how can someone forget their first hack? I mean, sure, I did, but I remembered it too, right?
I’d like to think that, where-ever Natalie is, she’s also doing hacks every once in a while, and I’d like to believe that, while I am mostly spending my days crafting emails to customers, writing code to solve problems, documenting my experience, I will, until the end of my life, be hacking on computers and teaching my children how to do hacks.
Unlocking a memory from a game music that was re-implemented by an Armenian TV show was not on my bingo card, but I’m happy that it happened.
Keep hacking y’all.
That’s all folks…
Enabling DTrace in Python on FreeBSD (again)
Last night I was running our usual Greybeard AMA on FreeBSD’s Discord server, when someone asked “I’ve been using Linux for years, but I also like FreeBSD. what can I do for FreeBSD, and what can FreeBSD do for me, as a Python Full Stack Developer?”
I started talking about FreeBSD Jails, ZFS, Boot Environments and more, but I also wanted to focus on DTrace. The ability to dynamically trace on production still sounds like magic to me. I know it isn’t, I’ve read the code and the papers, but the fact that I can ask the operating system questions on the fly, without recompiling, is amazing.
So when I was demoing DTrace, I also wanted to show the DTrace integrations with Python. Little did I know that the Python package on FreeBSD was not compiled with the --with-dtrace option.
As a sane person, I setup a Jail (using Jailer, of course), cloned the FreeBSD ports tree and added the --with-dtrace option back. It did not compile.
The first issue that we encountered was the following:
--- Include/pydtrace_probes.h --- dtrace: option requires an argument -- s
ah yes, looking over the Makefile we see the following
Include/pydtrace_probes.h: $(srcdir)/Include/pydtrace.d
$(MKDIR_P) Include
$(DTRACE) $(DFLAGS) -o $@ -h -s $<
: sed in-place edit with POSIX-only tools
sed 's/PYTHON_/PyDTrace_/' $@ > $@.tmp
mv $@.tmp $@
(you can also view the source here)
As far as I can tell, the $< thingie does not work with BSD Make, so how about if we try using gmake instead?
I did the following changes to the Makefile in the FreeBSD Ports tree. Basically adding gmake into USES=.
USES= compiler:c11 cpe ncurses pathfix pkgconfig \
- python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz
+ python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz gmake
Now let’s try compiling again.
We get the following errors now:
ld: error: undefined symbol: __dtraceenabled_python___function__entry
Ah yes, linking issues.
After an hour of digging we learned that the DTRACE_OBJS= variable is set to… nothing. But it needs to be set to Python/pydtrace.o. I was not able to fix this issue properly (in configure, configure.ac, or whatever madness that GNU Autotools use), so I just changed the line manually in the Makefile.
Now let’s try compiling again.
We get the following error:
./Python/sysmodule.c:223:24: warning: passing 'const char *' to parameter of type 'char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
Someone in the Discord chat recommended that I disable the LTO (Link-Time Optimization) option.
I did make clean, I patched that line again, and here we go, one more time.
It’s compiling, it’s compiling, it’s compiling… aaaaand we’re done! Let’s do make install
===> Installing for python311-3.11.11 ===> Checking if python311 is already installed ===> Registering installation for python311-3.11.11 [python.srv0.hackerspace.am] Installing python311-3.11.11...
good! now let’s try DTrace with Python!
In one terminal, I ran Python, and in another one I got the following
# dtrace -l -n 'python*:::' ID PROVIDER MODULE FUNCTION NAME 85302 python8737 libpython3.11.so.1.0 sys_audit audit 85303 python8737 libpython3.11.so.1.0 sys_audit_tstate audit 85304 python8737 libpython3.11.so.1.0 _PyEval_EvalFrameDefault function-entry 85305 python8737 libpython3.11.so.1.0 dtrace_function_return function-return 85306 python8737 libpython3.11.so.1.0 _PyEval_EvalFrameDefault function-return 85307 python8737 libpython3.11.so.1.0 gc_collect_main gc-done 85308 python8737 libpython3.11.so.1.0 gc_collect_main gc-start 85309 python8737 libpython3.11.so.1.0 PyImport_ImportModuleLevelObject import-find-load-done 85310 python8737 libpython3.11.so.1.0 PyImport_ImportModuleLevelObject import-find-load-start 85311 python8737 libpython3.11.so.1.0 _PyEval_EvalFrameDefault line
Woohoo! Now I’m happy!
Okay, so what did we learn?
- There’s a little bit of GNUMake-ism in Python’s Makefile. Either we have to use
gmakein FreeBSD when building Python or we need to submit an alternative to the upstream - Die GNU Autotools… I mean… the GNU Autotools usage in Python has an issue, where the generated Makefile does not set the
DTRACE_OBJScorrectly. Can anyone help with this? As a Pascal guy, reading GNU configure files makes me wanna die. - LTO is problematic. Solutions?
The real question is, how can we enable DTrace by default in FreeBSD packages for Python. DTrace is one of our market advantages and we should find a way to enable it everywhere we can.
This was a fun Greybeard AMA, where we touched multiple parts of the system. Looking forward for next week!
That’s all folks…