Category Archives: Tech

Incident Postmortem: BSD.am home server @ 3-4 July 2023

Incident Information

Between the hours of Mon Jul 3 03:05:59 2023 and Tue Jul 4 01:10:15 2023 the home server named BSD.am (also known as pingvinashen.am) was completely down.

The event was triggered by a battery issue due to high temperature at the apartment where the home server resides.

A battery swell caused the computer to shut down as it produced higher than normal heat into the system.

The event was detected by the monitoring system at mon.bsd.am which notified the operators using email and chat systems (XMPP).

This incident affected 100% of the users of the following services:

jabber.am public XMPP server
conference.jabber.am public XMPP MUC server
օրագիր.հայ public WriteFreely instance
սարեան.ցանցառներ.հայ public Lobste.rs instance
BIND.am public DNS server and its zones
Multiple hosted blogs, including this one you’re reading.
A private ZNC server for Armenian Hackers Community
git.bsd.am public Gitea server
A matterbridge instance connecting multiple communities
A Huginn instance automating tasks (such as RSS to Telegram, RSS to newsletter) for Armenian Hackers Communities
A newsletter instance running listmonk.app
A private Miniflux.app server for Armenian Hackers Community
FreeBSD Jail users’ meetup website

Multiple community members contacted the operator (yours truly) asking for an ETA.

Response

After receiving an email at Mon Jul 3 03:06:49 2023, the Chief Debugging Officer (yours truly) started analyzing the possible issue. According to Monit (mon.bsd.am) all the services were unavailable and the server was not reachable by IP (based on ICMP).

The usual possibility, network failure at the ISP level, was ruled out, as the second home server (arnet.am) was functioning properly.

The person closest to the server physically, was the operator’s sibling (lucy.vartanian.am), however she did not have the background in Unix system administration nor in hardware maintenance. Also, she was asleep.

Hours later the siblings (yours truly) organized a FaceTime call to debug the issues remotely.

The system did boot the kernel properly, however it would shutdown before the services could complete their startup.

Clearly, the machine needed to be shipped to the operator (yours truly) to be debugged at the spot.

So that’s what the team did.

Precise addresses are removed for privacy

Recovery

At the operator’s (yours truly) location, the BIOS logs have listed that the system suffered from a ASF2 Force Off. This usually means a thermal problem.

The operator (yours truly) disassembled the laptop, hoping the system needs a little dust clean-up and a thermal paste update.

Turns out the problem was actually a swollen battery.

After removing the battery, the system booted fine. Just to be sure that the swollen battery was the root cause, a complete system stress test was ran. No issues detected (Well, except “Missing Battery”).

The systems was returned to its residency, connected to the internet and all services were accessible again.

Precise addresses are removed for privacy

Next Steps

Install a new battery in the future, as the laptop is not connected to a UPS
Make sure to test the hardware during environmental changes (too cold, too hot, etc)
Run a simple status page with an RSS feed in a separate environment and notify users

If you’re new here, then first of all I’d like to thank you for reading this IR Postmortem article.

Yes, this was an IR Postmortem of a home server of a tiny community in a tiny country. This was not about Amazon, Google, Netflix, etc.

I wrote this for two reasons.

First, I wanted to show you how awesome the actual internet is. You see, when Amazon dies, everything dies with it. Your startup infra, your website, your hobby projects, everything.

When my server dies, only my server dies. And that’s the beauty of the internet. If you can, please, keep that beauty going.

Second, I run a small security company, illuria, Inc., where we help companies harden their environment and recover from incidents. It’s been years since I wrote an IR postmortem personally (my team members who do that are way smarter than me!), and I thought it would be a nice exercise to write it all by myself 🙂

I hope you liked this.

That’s all folks…

Reply via email.

Antranig Vartanian

July 1, 2023

A customer asked me to help them setup a tiny lab with many open-source tools. They are planning to move from corporate services to open-source alternatives such as NextCloud, Gitea, etc.

Unfortunately, they run only Linux, Ubuntu to be more specific, and as a UNIX gentlemen, I didn’t want to put everything into a single host, so I decided to use containers, in this case, LXC, a.k.a Linux Containers.

How hard could it be?

Oh god, layers of abstraction on within the system that have no idea about each other.

Like, who would assume that LXC would automatically download and install dnsmasq and assign IP addresses without my knowledge, or that it would push rules into the firewall?

The more I use Linux Container, the more I understand why FreeBSD Jails / illumos Zones didn’t win.

People don’t want automation or control, they want “please do this for me as I don’t wanna do it myself” tools.

I’d expect at least a message post-installation that says “We have installed and configured dnsmasq, reconfigured some systemd things, modified the following file (which is not mentioned in any man page, so you can use Google instead of man/apropos) and will use IP address ranges that you didn’t approve”

Is this why Docker won? Is it because people DIDN’T want to learn how to do software packaging? I hope not. I wanna believe its because developers wanted to “think operationally”

Oh, and from a FreeBSD perspective, what’s even more weird is that

there are no proper manual pages.
the documentation is weird. It talks about a utility named lxc but I’m using 20 utilities named lxc-*, and I still cannot find the proper documentation for that
it’s very much segmented. For example, on FreeBSD, we talk about which is better, jail.conf, BastilleBSD, pot, AppJail or Jailer. Here the same utility (lxc) that has multiple config files with no proper versioning, pretty complex manual pages and the not even examples or HowTos.

I’m looking at this and thinking ”oh well, if we build a proper tool, I bet we can win some of the market” until you realize, of course, that when people hear FreeBSD, they will be thinking ”it’s not Linux? maybe it’s not worth it, otherwise I would’ve heard about it”

I’m just angry here. Please ignore my rants.

Cheers y’all.

Reply via email.

Pen and Paper

For the last 6 to 10 months, I’ve been trying to find the proper digital tools to manage my life. Spoiler alert: I keep failing to do that.

In the last 5 years, my main and only job was to do one thing and one thing only, run illuria, Inc., a company that I co-founded with my friends. At some point, specifically when your team has more than three people, you need some kind of task management tool. And I’ll be honest here, I don’t care which one that is, most of them do the same thing anyway. We ended up using Notion, and we like it very much. I like the database feature and my team loves the Kanban boards. Half of the team does development and the other half does development-related things (release engineering, infra) and business-y stuff, such as sales, marketing, what have you, so we never had any issues with Notion.

(To be clear, while I like Notion and any other tool would do the job as well, I have to say that I never liked Jira’s UI/UX. That one is, indeed, enterprise-y, but that’s a story for another day).

But last year I started taking some more responsibilities (kind-of-)outside of work. Co-hosting and producing a podcast, running a community of Armenian hackers, teaching cybersecurity (I actually end up teaching Unix + Networking + how computers work, but turns out that’s what actually 80% of cybersecurity is anyway), contributing more to open-source (specially since we open-sourced our little utility, Jailer) to name a few.

Which meant that I needed a digital tool to manage the non-work part of my life as well.

The obvious choice was to use Notion, since I know it anyway. That ended up being a disaster for a very weird reason: It only works online. Even if you have the desktop app, it’s still just a wrapper around the website with some nice things like desktop notifications and such.

I know, this sounds strange to many people, but I don’t like being online all the time. Sometimes I enable iOS/macOS’s DnD, to get some work done, but sometimes I go completely offline with no distractions at all.

Unlike most other developers, I work completely locally. From my development environment to my infrastructure tools, everything is synced local/prod. This is actually a good reason to not use the fancy features of the cloud, but again, that’s a story for another day.

I have been told, by my friends, that my options are the following:

Go as basic as possible and use Notes.app. Well, I like this option, but I had two issues.

First, it’s Apple only. Yes, you can actually connect the Notes.app to your IMAP account and sync that with other Unix machines using clients like Evolution, but now the features are limited to text only. Not even tables :/

Second, the iCloud sync has some weird issues. not always, but from time to time, I was shouting “WHERE ARE MY NOTES???” just to see them appear minutes later.

Apple Notes.app? tested, liked it overall, but it’s not for me.

My friends’ second option? Go as deep as Obsidian!

I fired up Obsidian and I fell in love immediately. It was like love at first sight. Vi keybindings? it’s there. Plugins? it’s there. Run shell commands on your notes? it’s there!

After couple of days, I had everything ready. I had my folders (please, let’s call them directories!), my notes all migrated, all the plugins I needed for my weekly and daily notes (similar to what we had on Notion at work), etc etc.

And then days passed, and then weeks passed. What happened? I totally forgot that Obsidian even exists. I noticed that my wall had… sticky notes (FreeBSD branded!), my Mac had… sticky notes!

This made me so frustrated for multiple reasons.

Not that I only had two types of sticky notes (analog and digital), I also could not “search” in them!

I ended up turning the analog notes into digital, and tagging them at their title, so I could at least search using the macOS Window API.

And then I saw something awesome. Cortex Podcast released the Sidekick Notepad!

Wait wait wait, are you thinking that I bought the Sidekick Notepad? Nope, I did not 🙂

But what I ended up doing is putting all of our office’s legal pads next to me at home, we were not using them in the office anyway!

Two weeks later and I’m writing everything as needed. I take notes, I write my todo lists. I made my legal pads horizontal, similar to the Sidekick Notepad and woof is was awesome!

For a moment there I started using the Moleskine Classic Notebook, since it was more portable than yellow/white legal pads, but that didn’t work as well. I guess I needed something that can be teardown on the fly and no very-hard cover.

Why am I telling you about all of this? Well, uncle Dexter has asked on Mastodon “500 reMarkable ads later… Is anyone using one? Would you recommend it?”

I have used reMarkable (the first one), and I loved it. Not because it was an awesome technology or such, but because it made me think the same as if I was writing on paper with a pen.

So, if you, like me, have suffered for a long time to find the best “digital time/notes/todo management tool”, then you’re probably an analog person, like me.

Just take a sheet of paper, start writing on it with a pen.

That’s all folks…

P.S. I might actually end up buying the reMarkable 2 and check how that goes, or even the Sidekick Notepad. But with my writing speed, I’d need at least 4 Sidekicks every 3 months. Let’s wait and see 🙂

Reply via email.

Domains as Verification

Couple of days ago when I was browsing the internet I stumbled upon Jim Nielsen’s blog, where at the top it said

Verified ($10/year for the domain)

Luckily, his blog is so organized (unlike mine) where I found the post named Verified Personal Website in which he talked about this.

Personally, I don’t have enough CSS skills to do that, but I added a check mark next to my name on my blog (thank you Unicode!).

I think this is amazing and it should be used more by bloggers everywhere. If someone opens a blog they should see a check mark. Maybe a cute one in SVG, maybe a CSS trick, maybe it’s just an image, but it should be there.

Why? so we remind people that on the internet, whenever you have a domain, you are already verified.

Can scammers scam and criminals phish? yes, indeed. But unlike the not-very-social-media, it’s hard to do that.

Ironically, having a website on the internet costs less than having a “verified” social media account, say on Twitter.

Currently, Twitter Blue costs $8/month or $84/year.

Let’s see how much would it cost to have a blog on the internet.

First thing first, you need a domain, and it can be anything that you feel awesome with. Awesome-ness is the first and only rule.

Here’s an awesome domain that I found is available using NameCheap.

This is awesome!

Next, we need to host our website. Well, lemme check my favorite server hosting platform, Vultr.

A machine with a single CPU and a 1GB of RAM, that’s plenty!

I mean, with that much power, you can easily run WordPress (if you’re using caching).

Or, if you don’t want to get techy-techy at all, you can use a static site generator. You like Markdown and text files? There’s Hugo for you. Do you want to just click on buttons and BOOM, your website is ready? Have a look at Publii!

So, how much does it cost in the end? Here’s how it looks like if you pay annually or monthly, per year.

A/M	Twitter Blue	Website on the Internet
Monthly	$8×12 = $96	$8×12 + $10 = $70
Annually	$84	$8×12 + $10 = $70

So yes, it is cheaper to have a website on the internet.

Wait a second, annually vs monthly looks the same? OF COURSE IT DOES! THIS IS THE INTERNET! We want you to think “huh, 70 dollars? well that’s dope” and not about “well, if I pay annually now, I will save 12 dollars” and then completely forget about that service anyway.

Oh, and did I tell you about the features of having a website on the internet? Well we don’t have a list, but here’s some things from the top of my head.

You get to be verified, because welcome to the internet
You get to post whatever you want
you get to edit them! can you believe that?
You can upload photos and make it looks like a photo blog
Unlike other platforms, which seemed to be for photographers but not anymore, you can tag things, and make albums!
You can upload podcasts!
Hell, and if you ever want to leave, you can just redirect your domain to somewhere else 🙂

And I’m not even talking about the other awesome features of having a domain, like, custom emails! Be that person that does NOT have a @gmail.com, but @AwesomeIsHere.net!

And hey, Twitter Blue might die, Twitter might die, every other company might die, but the internet will not 🙂

That’s all folks…

Reply via email.

Downtime for the rest of us

If the homebrew server club had an official membership based on technicality, then I would be a very proud member, but it does not have a membership application. That being said, I am still a proud member of HBSC, as I’ve been running a home server for a decade now.

I can’t say that it’s been easy, but it has been evolving. When I tried setting up my first server, I had issues with an ISP that didn’t allow me to have more than a single public static IP address.

Over time, ISPs changed, servers have changed, but the only thing that remained the same is me running my server from my home.

Now, I do have multiple IPs, a VLAN with my ISP that we’ve agreed on the setup, an internal email where they answer my questions without me calling the general support line and finally a publicly available Looking Glass that anyone can use.

Unfortunately, it’s not all sunshine and roses. My biggest request for the last couple of years has been the same: a status page.

You know, that simple web page that tells you if a service is down?

Interestingly, when I was researching ISPs (that’s a post for another day) I noticed that most ISPs don’t provide a status page.

Some ISPs (like Google Fiber) ask for an address, while others ask you to log in.

I understand that an ISP is a complex beast, and it would not be an easy task to say “we have an issue”, but hey, someone has to start trying.

Oh, I forgot, the downtime mentioned in the title!

Well, my personal blogs don’t have a lot of traffic (unless if someone posts a link to the Orange Website, then I get 20K+ viewers per day), but many people use my services, such as my Jabber/XMPP chat server, a publicly available blogging system an Armenian tech forum and so on.

All of the local ISPs had issues this week and their first response was to fix the outbound traffic. So for most people in the country, they didn’t care, as long as they were able to use Telegram and log into their Meta-owned social media services.

But for me and my community, we had to wait almost 18 hours for them to fix the internal network issues.

However, I am still a proud member of HBSC, because unlike Big Tech companies, if I go down, only I go down. But if a cloud goes down, everyone goes down with them.

See you at the next downtime 😉

Reply via email.

Antranig Vartanian

May 16, 2023

A customer texted me saying they are having issues running our operating system on QEMU-KVM, so I opened a cloud provider’s portal, clicked on “Deploy a new bare metal”, and tried to SSH.

After waiting for a while, I see that I can ping the machine, but I can’t SSH into it…

I open the console, and this is what I see.

Screenshot 2023 05 16 at 6 37 28 PM

This. This is why I moved to FreeBSD.

Reply via email.

More macOS Display Resolutions

I assume that this feature has been around for a while, maybe it came with Ventura, but I noticed it just today.

Turns out, if you have a MacBook Air and you want more resolution in the expense of sharpness, you can go to System Settings → Displays, Click on Show all resolutions and get more options.

Screenshot 2023 05 15 at 1 04 07 PM

I just moved from 1680×1050 to 2048×1280. While I don’t recommend this for most people, it it useful if you do development in your terminal (like I do) and want to see more context.

Reply via email.

Antranig Vartanian

May 15, 2023

I like Windows because it’s the best for gaming!

Meanwhile on Windows: The simplest game that has been around for 30 years, Solitaire, has ads in it. You have to pay $2/month or $15/year to remove ads.

The design is ugly, it’s so ugly that the Windows XP version easily beats it and it takes at least 10 seconds to load the game, which used to take 1 second on Windows XP.

I’ve stopped using Windows 12 years ago, I’m not sure why everyone else hasn’t.

If I didn’t need to test network protocols every once in a while, I would be happier, because booting Windows feels like a pain already…

Reply via email.

FreeBSD package repo with specific versions

illuria’s ProfilerX runs on LureOS, which is our custom operating system based on FreeBSD.

To update the operating system we rely on two tools, pkg(8) for packages and freebsd-update for the base.

Initially, I’ve setup our poudriere and package repo in the FreeBSD way, so our URL looks like /FreeBSD:13:amd64/devel and /FreeBSD:13:amd64/prod. This is done by expanding the ${ABI} variable, similar to what FreeBSD does in FreeBSD.conf.

Initially, this worked fine, but now that there’s a new FreeBSD out there (13.2), I didn’t want to put the new packages in the old URL, but rather have a URL for each major.minor version. This is mostly for the enterprises who take their time to upgrade software.

Turns out the easiest way to do this is (after reading the pkg.conf(5) manual page) is to use the VERSION_MAJOR and VERSION_MINOR variables.

The new LureOS will use /${ABI}/${VERSION_MINOR}/repo, which will expand to /FreeBSD:13:amd64/1/devel, making it easier for us to extend life after a new release.

That’s all folks…

Reply via email.

Antranig Vartanian

May 12, 2023

The Pixelmator Team released Photomator a month ago (well, technically renamed), one of the things on their roadmap was Photomator for Mac.

I submitted a form to join the TestFlight Beta and I got in! I’ve been using it for couple of days now and I absolutely love it.

The Photomator Team sent an email saying “Photomator will be released on ███ ██th, so we ask all testers to hold back on sharing any screenshots of Photomator for Mac until this date.”

While I can’t share any screenshots, I will share a photo edited using Photomator for Mac.