dave eddy

You Suck at Programming

Dave Eddy — 2024-03-23T00:00:00-04:00

In late 2022 I started a new series on TikTok called You Suck at Programming. In it I mostly talk about really cool Bash and Unix tricks and pitfalls and just general advice.

I recently went through and cataloged all of the episodes and made them available on a new website that you can view in the browser, or on your terminal:

To pull the episodes run:

$ curl ysap.daveeddy.com
 _   _ ___  __ _ _ __
| | | / __|/ _` | '_ \   You suck at programming
| |_| \__ \ (_| | |_) |  Episodes and guide by dave eddy
 \__, |___/\__,_| .__/   https://www.daveeddy.com
  __/ |         | |
 |___/          |_|      Source: https://github.com/bahamas10/you-suck-at-programming

...

Void Linux Bhyve Image on OmniOS

Dave Eddy — 2022-09-19T00:00:00-04:00

This guide will show how to install Void Linux as a guest on OmniOS using hardware virtualization with bhyve.

Configure the system

To get started, first install zadm on OmniOS inside the global zone. This tool makes creating and managing zones simple and is reminiscent of the vmadm command on SmartOS.

Install zadm:

pkg install zadm
pkg install zones/brand/bhyve

I personally use this config for zadm (shoutouts to @papertigers for this), but feel free to customize this as you like.

/etc/opt/ooce/zadm/zadm.conf

{
    "CONFIG"  : {
        "format"          : "toml"
    },
    "CONSOLE"  : {
        "auto_connect"    : "off",
        "auto_disconnect" : "on",
        "escape_char"     : "_"
    },
    "SNAPSHOT" : {
        "prefix"          : "zadm__"
    }
}

Test that zadm was installed and works as expected by running:

root@datadyne:~# zadm list
NAME              STATUS     BRAND       RAM    CPUS  SHARES
global            running    ipkg       256G      64       1

Note that for this blog post I ran all of these commands on a fresh system so only 1 zone is running (the global zone).

Create the datasets needed. You can customize these if you’d like but for the sake of this post I’ll be using these datasets:

zfs create -o mountpoint=/zones rpool/zones
zfs create rpool/zones/config
zfs create rpool/zones/data
zfs create rpool/zones/zones
mkdir /zones/config/iso

Where:

/zones/config toml files for each zone used during zone creation.
/zones/data dataset to store persistent data for each zone that can be LOFS mounted into the zone.
/zones/zones confusingly named, but each zones root data will live in this directory.

For example, a zone named foo would have files like:

/zones/config/foo.toml
/zones/data/foo
/zones/zones/foo

Create the initial void VM

Pull the full ISO from the void Linux download page. I’m using the “Base Live Image x86_64 glibc” for this:

https://voidlinux.org/download/

Download this ISO (yours might be named differently) to the machine:

cd /zones/config/iso
wget https://alpha.de.repo.voidlinux.org/live/current/void-live-x86_64-20210930.iso

Create the zone toml file:

/zones/config/void-golden-image.toml

zonename="void-golden-image"
cdrom="/zones/config/iso/void-live-x86_64-20210930.iso"
zonepath="/zones/zones/void-golden-image"

ram="2G"
vcpus="4"

dns-domain="rapture.com"
resolvers=[
  "10.0.1.2",
  "10.0.1.3",
]

acpi="on"
xhci="on"
autoboot="false"
bootorder="cd"
brand="bhyve"
ip-type="exclusive"
limitpriv="default"

[capped-memory]
# should match global "ram" key above
physical="2G"
swap="2G"

[[net]]
# will use DHCP when creating the golden image
global-nic="e1000g0"
physical="voidgoldenimage0"

[bootdisk]
blocksize="8K"
path="rpool/zones/zones/void-golden-image/root"
size="50G"
sparse="false"

[vnc]
# needed for installation
enabled="on"

There are a lot of options in this toml file, but hopefully most of them are self-explanatory - just make sure to update options like the dns-domain and global-nic to match your environment.

Create the zone:

sudo zadm create create -b bhyve -t void-golden-image.toml void-golden-image
sudo zadm boot void-golden-image

Verify with:

$ sudo zadm list void-golden-image
NAME              STATUS     BRAND       RAM    CPUS  SHARES
void-golden-image running    bhyve        2G       4       1

At this point the zone should be running, and inside the zone is the bhyve process, which is responsible for booting the Void Linux image. We can’t quite connect to the console using zadm console or zlogin -C, so for now we must use VNC. Luckily, zadm comes with a simple VNC server built-in.

Create VNC server (leave this command running):

$ sudo zadm webvnc 0.0.0.0:8080 void-golden-image
Web application available at http://0.0.0.0:8080

If you are quick enough you may be able to see this grub menu!

If not don’t worry, grub will automatically pick the top option which is what we want.

You should end up with a screen that looks something like this:

As it says in the output, you can now login with the user root and the password voidlinux. At this point I prefer to run bash, so I’m using the bash shell for the installation process as opposed to just /bin/sh.

Install Void Linux

Now, run the command void-installer to bring up the graphical installer - it should look like:

void-installer

Setup

In the installer make sure to set these options:

RootPassword - set to something you’ll remember
BootLoader
- Set “use graphical terminal?” to no
Partition
- cfdisk
- gpt

Disk Partitions

Now, you should be inside cfdisk to set up the underlying hard drive (the 50G disk created in the toml file) for Void. Create the following:

3 partitions

/dev/vda1: 1G, EFI System
/dev/vda2: 2G, Linux swap
/dev/vda3: 47G (remaining), Linux filesystem

Your output should look like this:

NOTE: Make sure to write before quit - otherwise your changes won’t be saved.

File Systems

With the partitions created you can now create the file systems in the installer, they should be this:

FileSystems
- /dev/vda1: vfat (FAT32) - mounted at /boot/efi
- /dev/vda1: swap (Linux Swap)
- /dev/vda1: xfs (SGI’s XFS) - mounted at /

NOTE: xfs is optional - pick whatever filesystem you prefer.

Finally, click Install and everything should just work.

EFI Configuration

DON’T REBOOT - we’re not done yet!

Back in the live system and outside of the installer, mount the newly installed boot partition:

mount /dev/vda1 /mnt

Move the grub efi file into the proper location (from /boot/efi/EFI/void_grub/grubx64.efi -> /boot/efi/EFI/BOOT/bootx64.efi):

mv /mnt/EFI/void_grub /mnt/EFI/BOOT
mv /mnt/EFI/BOOT/grubx64.efi /mnt/EFI/BOOT/bootx64.efi

Unmount the filesystem and power off the machine:

umount /mnt
poweroff

Configure serial console

Inside the global zone, edit the zones config file with zadm to remove the CD device (I just commented the lines out for my system):

$ sudo zadm edit void-golden-image
...
# cdrom=[
#   "/zones/config/iso/void-live-x86_64-20210930.iso"
# ]
...
$ sudo zadm boot void-golden-image

This way, the machine will boot from the newly installed OS from the hard drive and not the CD ROM (where the installer ISO was mounted).

You may have to refresh the VNC webpage to access the zone, if it all works you should see a login prompt. You can use the user root with the root password you setup earlier:

Before going any further, we can make our lives easier by enabling the serial console:

ln -s /etc/sv/agetty-ttyS0 /var/service

Then, instruct grub to work over the serial console as well. Add these lines (or modify the existing lines) in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="loglevel=4 console=ttyS0,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

And run:

update-grub

Finally, power off the machine again:

poweroff

Now, from the global zone, boot the machine and connect to the console. We can do this all in one single command so we don’t miss anything on the console when the machine powers on:

$ sudo zadm boot -C void-golden-image
[Connected to zone 'void-golden-image' console]

[NOTICE: Zone booting up]
Loading Linux 5.13.19_1 ...
Loading initial ramdisk ...
- runit: $Id: 25da3b86f7bed4038b8a039d2f8e8c9bbcf0822b $: booting.
- runit: enter stage: /etc/runit/1
=> Welcome to Void!
...
_.
$

Use the escape sequence defined in the zadm config to break out of the console (_.).

If everything works, then you should see the kernel messages via the console on the terminal and not have to rely on VNC.

Setup the VM image

From here you are able to setup this machine however you’d like! I’d recommend running updates and installing some optional tools like:

xbps-install -Syu
xbps-install jq htop bwm-ng vim vsv vpm bat fd ripgrep

Check running services with vsv:

You can also remove the existing virtual terminals if you just plan on using the serial console. I personally leave one active in the event that I need to VNC back in for whatever reason.

rm /var/service/agetty-tty{2..6}

After you do whatever setup you’d like for the base image, we can get this prepared to be cloned for future VMs.

sshd is configured on Void Linux to generate new host keys if none exist when the service is started. These commands will make it so new keys will be generated on the next reboot:

sv down sshd
rm /etc/ssh/*_host_*

Next, optionally setup the skeleton for a static IP address (if you don’t want to use DHCP):

/etc/rc.local

#ip link set dev eth0 up
#ip addr add 10.0.1.53/24 brd + dev eth0
#ip route add default via 10.0.1.1

# mkdir -p /data
# mount /data

true

I leave these commented out in the golden image so I can simply uncomment them and modify them when I create a new machine.

The last thing I like to do is a put a text file like this in the root home directory - this is for future me so I can quickly remember how to setup a new machine that I clone from this image:

[root@void-golden-image ~]# cat getting-started.txt
run `hostname` and edit `/etc/hostname`
disable `dhcpcd`
put IP info in `/etc/rc.local`
put DNS info in `/etc/resolv.conf`
reboot

Break out of the console using _. or whatever you’ve defined.

Create the image

Snapshot the image root disk:

sudo zfs snapshot rpool/zones/zones/void-golden-image/root@golden-2022-05-23

Now, we have this image which is ready to be cloned for new VMs to make the installation process way easier for us!

Create a new instance of the VM

We’re going to create a new VM named void-test using this image. Create a new toml file:

/zones/config/void-test.toml

zonename="void-test"
zonepath="/zones/zones/void-test"

ram="2G"
vcpus="4"

dns-domain="rapture.com"
resolvers=[
  "10.0.1.2",
  "10.0.1.3",
]

acpi="on"
xhci="on"
autoboot="false"
bootorder="cd"
brand="bhyve"
ip-type="exclusive"
limitpriv="default"

[capped-memory]
# should match global "ram" key above
physical="2G"
swap="2G"

[[net]]
# using static IP for this instance
allowed-address="10.0.1.54/24"
defrouter="10.0.1.1"
global-nic="e1000g0"
physical="voidtest0"

[bootdisk]
blocksize="8K"
path="rpool/zones/zones/void-test/root"
size="50G"
sparse="false"

[vnc]
# needed for installation
enabled="on"

[[fs]]
dir="/data"
special="/zones/data/void-test"
type="lofs"
options=[]

[[virtfs]]
name="data"
path="/data"
ro="false"

There is a lot of new stuff in this toml - I’ll go through all of it below. Create a new dataset for the VMs persistent storage and then create the zone

sudo zfs create rpool/zones/data/void-test
sudo zadm create create -b bhyve -t void-test.toml void-test

Before booting the machine, we destroy the created root disk and clone the golden image into the same location:

sudo zfs destroy rpool/zones/zones/void-test/root
sudo zfs clone rpool/zones/zones/void-golden-image/root@golden-2022-05-23 rpool/zones/zones/void-test/root

This means that we have a brand new zone and VM created, but the root disk is a clone of the golden image we created above. Boot the VM with:

sudo zadm boot -C void-test

The username and password are the same as we set above for the image we created. To configure this new instance we have instructions waiting for us:

cat getting-started.txt

Following those instructions I simply ran:

hostname void-test
hostname > /etc/hostname
rm /var/service/dhcpcd
vim /etc/rc.local
vim /etc/resolv.conf
reboot

Persistent data

It may be confusing that I keep talking about persistent data in /zone/data/void-test when all of the data on the VM is technically persistent. The point of this is to a have a mount that exists in the zone that won’t be tied to the VM itself and won’t contain any OS installation files.

When just using zones, we accomplish this by LOFS mounting (analogous to bind mounts in Linux) files from the global zone into the zone. We are going to do that as well here (in fact, we do that in the toml file in the [[fs]] sections). But, we must go a step further because of bhyve and expose those mounts inside the zone into the hardware VM itself.

We accomplish this through 9pfs.

In the void VM you have newly created you must install 9mount:

xbps-install 9mount

Then, you can add this line to /etc/fstab:

data /data 9p rw,relatime,dirsync,uname=root,cache=mmap,access=client,trans=virtio,_netdev 0 0

And make sure the mount point exists:

mkdir /data

With all of this in place, you can finally run:

mount /data

And that directory inside the VM will now be the same as the directory in /zones/data/void-test on the rpool zpool! This allows for easy file sharing between the guest and the host, as well as persisting data in the event that you want to reprovision the VM from a new image. Make sure to uncomment the mkdir and mount commands in /etc/rc.local so the directory mounts at boot.

Conclusion

Running void under LX is great, but some software really doesn’t like running under LX virtualization and it’s just easier to use hardware virtualization at that point. bhyve makes it super nice and easy to accomplish this on OmniOS!

Programmatically Generated Piano Music

Dave Eddy — 2021-12-17T00:00:00-05:00

I wrote a computer program to generate piano music on the command line. The generation is done with a simple algorithm to create patterns in MIDI format, which is then imported into my DAW to process the data and turn it into music. There is no AI or Machine Learning in this algorithm - the music is programmatically generated and this video covers the algorithm that I created in detail.

I Made a Computer Program to Generate Piano Music

5 HOURS of ALGORITHMICALLY GENERATED Piano Music

The source code can be found on GitHub: https://github.com/bahamas10/music-generator

Plex Media Server on Void Linux

Dave Eddy — 2021-11-15T00:00:00-05:00

How to install Plex on Void

Download plex-install-manager from:

https://github.com/bahamas10/plex-install-manager

Install deps:

xbps-install -S binutils tar curl xz

Create user and directories needed:

useradd -s /bin/false -d /var/empty _plex
mkdir /opt/plex /opt/plex/src /opt/plex/data
chown _plex:_plex /opt/plex/data

Install Plex:

./plex-install-manager -a -d /opt/plex/src

Verify Plex installed:

$ ls -lh /opt/plex/src
total 2.0K
lrwxrwxrwx 1 root root 43 Nov 15 02:44 active -> plexmediaserver_1.24.5.5173-8dcc73a59_amd64/
drwxr-xr-x 4 root root  4 Nov 15 02:44 plexmediaserver_1.24.5.5173-8dcc73a59_amd64/

Create Plex service:

mkdir /etc/sv/plex
vim /etc/sv/plex/run

#!/usr/bin/env bash
# label=Plex Media Serever
exec 2>&1
cd '/opt/plex/src/active/root/usr/lib/plexmediaserver' || exit 1
export PLEX_MEDIA_SERVER_HOME=/opt/plex/src/active/root/usr/lib/plexmediaserver
export PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/opt/plex/data
export PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
exec chpst -u _plex:_plex ./Plex\ Media\ Server

ln -s /etc/sv/plex /var/service/plex

How to upgrade

To upgrade plex you simply need to run the following commands:

./plex-install-manager -a -d /opt/plex/src
sv restart plex

New Album - Inspired

Dave Eddy — 2021-06-01T00:00:00-04:00

I released my latest album on June 1st, 2021!

My 5th album (4th full-length). Inspired consists of songs mostly written during 2020 and 2021 - when I decided to quit my job and focus more on music creation. It’s comprised of piano and instrumental tracks.

Download or stream Inspired.

Tracklist

Journey Ahead 0:52
Morning Sun 3:23
Piano Night 4:08
In My Dream 3:33
Accordance 3:20
Sis Puella Magica (Piano Cover) 2:38
Hidden Away 3:18
I’m With You 4:58
Something Beautiful 3:48
Lost Corridors 3:57
Frozen 3:47
Secret Place 3:15
Undercurrents 4:00
Letting Go 3:41

sshp Rewrite from JavaScript to C

Dave Eddy — 2021-05-20T00:00:00-04:00

In 2013, I wrote a program in Node.js called sshp. This was right around the time I was investing heavily into learning node, and honestly having a blast doing it. Node is quick and fun to write, and with only a couple hundred lines of code, I was able to write node-sshp.

node-sshp is a command line utility that acts as a parallelizer for ssh. It works by taking in a file containing a list of hosts to connect to and looping over each host firing off an ssh command in parallel (with a configurable maximum number of concurrent processes). The tool’s description is:

sshp manages multiple ssh processes and handles coalescing the output to the terminal. By default, sshp will read a file of newline-separated hostnames or IPs and fork ssh subprocesses for them, redirecting the stdout and stderr streams of the child line-by-line to stdout of sshp itself.

Writing this tool in Node was an obvious choice at the time: the company I was working for was using Node heavily, and this tool was written specifically to be used at my job. Managing multiple concurrent child processes and IO streams also made Node the obvious choice.

Fast-forward eight years, and I decided it might be fun to revisit this code. I still use sshp at home and rely on it heavily; I don’t know if the company I worked for in 2013 is still using node-sshp, however. Looking at the existing code in Node, I thought it would be fun to try and rewrite it in C.

Jump to the finished program.

How sshp Works

node-sshp has 3 modes of execution:

line mode (line-by-line output, default).
group mode (grouped by hostname output, -g).
join mode (grouped by unique output, -j).

The first 2 modes, line and group, operate in largely the same way, differing only in how data is buffered from the child processes and printed to the screen. Line mode buffers the data line-by-line, whereas group mode does no buffering at all and prints the data once it is read from the child.

The last mode however, join, buffers all of the data from all of the child processes and outputs once all processes have finished. Instead of grouping the output by host, it is grouped by the output itself to show which hosts had the same output.

My goal in this rewrite was to keep sshp largely the same. Only the following options have changed, the rest of the user-facing interface remained exactly the same:

-b has been changed to -c off (disable color output).
-N has been removed in favor of -o StrictHostKeyChecking=no.
-o has been added to allow for any ssh option to be passed in.
-u has been removed (not applicable without npm).
-y has been removed in favor of -o RequestTTY=force.

The Decision to Rewrite

The truth is, I didn’t really have any reason for wanting to rewrite this code other than it looked “ugly” to me now. I found many places that I could “do better”, and a lot of places that had repeated lines of code with a single variable changed. Basically, there was a lot of sloppiness in the code I thought I could clean and make better, if for no other reason than to make myself feel better about the code.

I don’t think 2013 me would have ever attempted to this. I wasn’t at all confident in my ability in writing C (in fact, I was afraid of it and thought I would just have bugs and memory leaks everywhere). Now, in 2021, I’m still not super confident, but I think I’m just naive enough to start a rewrite and stubborn enough to finish it.

The Process of Rewriting

Rewriting from Node.js to C brought a lot of challenges. Like, a lot. Node has a lot of built-in facilities for handling multiple asynchronous processes and IO streams (thanks to libuv). On top of that, the JavaScript portions of Node are all single-threaded. When considering how to approach sshp.c, I had to decide if using threads made sense. Basically, with C I had a lot of options, which meant a lot decisions I had to make.

Modes of Execution

As mentioned above, sshp has 3 modes of execution. When I originally wrote node-sshp, this was something that I hadn’t considered explicitly - it sort of just so happened that there was a default mode, a -j mode, and a -g mode that popped up as I was iterating on it.

Because of this, it was easier to approach the rewrite since I had the blessing of hindsight to know how the program should look. Writing a new program that matches the interface of an existing program is similar to composing a cover or a remix of a song as opposed to writing one completely from scratch - it’s an easier process since a lot of the foundation is already laid.

File Descriptor Events

In C, it was up to me to figure out how I was going to approach this. I could just pull in libuv like Node.js does myself and have sshp depend on it, but I wanted to have as little dependencies as possible.

I made the choice to use epoll to simplify the process a bit. This would limit the number of operating systems that sshp will work on, but I mainly wanted to target the two operating systems I use the most at home: SmartOS and Void Linux, which both support the epoll interface natively. Other than the epoll dependency, sshp should work on any POSIX system with a C compiler.

Note: Since I originally wrote this blog post I have updated sshp to work with kqueue as well as epoll. You can see the update below for more information.

Threading

I originally imagined that I would need 2 threads. In fact, in the early commits of sshp, you can see how this was originally implemented with threads. It looked something like:

Child process execution manager
Child process stdio manager

At a certain point, I realized I had semaphores signaling each thread back and forth that 1. the child had executed and 2. the child needed to be reaped. Because they were mostly just blocking on each other to do work I eventually just pulled out the threading and just made it all a single thread with non-blocking IO from the child processes.

Final Thoughts

How many lines of code is sshp?

--------------------------------------------------------------------------------
 Language             Files        Lines        Blank      Comment         Code
--------------------------------------------------------------------------------
 C                        1         1966          239          494         1233
 JavaScript               1          393           39           33          321
--------------------------------------------------------------------------------

The C version of sshp is about 5x more lines than the Node version - it contains more lines of comments than the entire source of node-sshp.

You rewrote a program to have 5x more lines of code and less cross-platform compatibility?

Short answer: yes. I at least had fun during the process :).

Long answer: Node.js isn’t required to run sshp now. So, yay? :).

How long did it take to write originally? How long did the rewrite take?

The original program didn’t have all of the same functionality from the beginning of its life. For node-sshp, it took about 2-3 days to get the line mode and group mode working and another 2-3 days (months later) to get the join mode working.

For sshp in C however, it took about 2 full weeks of working to get it working. It was significantly faster to iterate in Node than C - but wow, am I way more proud of the code in C than I am of the code in JavaScript!

The Finished Program

sshp can be found on GitHub:

https://github.com/bahamas10/sshp

I had a ton of fun working on this, and am really happy with how it turned out. I hope you enjoy it! :).

sshp Examples

See the GitHub page linked above for more information and usage.

Given the following hosts file called hosts.txt:

# example hosts file
arbiter.rapture.com
cifs.rapture.com
decomp.rapture.com

Parallel ssh into hosts supplied by a file running uname -v:

Pass in -e to get the exit codes of the commands on the remote end. The local exit code will be 0 if all ssh processes exit successfully, or 1 if any of the ssh processes exit with a failure:

Also note that the hosts file can be passed in via stdin if -f is not supplied.

Run with -d to get debug information, making it clear what sshp is doing:

Run with -g (group mode) to group the output by hostname as it comes in. To illustrate this, -m is set to 1 to limit the maximum number of concurrent child processes to 1, effectively turning sshp into an ssh serializer:

Run with -j (join mode) to join the output by the output itself and not the hostname:

Send the sshp process a SIGSUR1 signal to print out process status information while it is running. In this example, a signal was sent twice to the process:

kqueue and epoll update

A little over a week after finishing up this blog post and sshp, I decided to actually look into kqueue and see how hard it would be to implement alongside epoll. This task turned out to be a lot easier than I initially planned since kqueue and epoll have very similar interfaces when it comes to watching for file descriptor read events.

This pull request contains all of the code to get kqueue supported. From a high level, I decided to start by abstracting the existing and working epoll logic in sshp into its own interface called FdWatcher. The interface works with the following functions:

FdWatcher *fdwatcher_create()
int fdwatcher_add(FdWatcher *fdw, int fd, void *ptr)
int fdwatcher_remove(FdWatcher *fdw, int fd)
int fdwatcher_wait(FdWatcher *fdw, void **events, int nevents, int timeout)
void fdwatcher_destroy(FdWatcher *fdw)

At first I was able to get this interface working for epoll only. Once that functioned properly, I started implementing kqueue support by using simple #ifdefs. After just an hour or so I had a working interface on my Mac using kqueue! This approach turned out to be really great in my opinion. Starting with a working implementation of epoll made it easier to decide what the FdWatcher interface should look like and fit kqueue into it.

The new code count is:

$ loc
--------------------------------------------------------------------------------
 Language             Files        Lines        Blank      Comment         Code
--------------------------------------------------------------------------------
 C                        2         2140          270          523         1347
 Markdown                 2          382          112            0          270
 Makefile                 1           57           11            5           41
 Plain Text               2           32            3            0           29
 C/C++ Header             1          132            8          111           13
--------------------------------------------------------------------------------
 Total                    8         2743          404          639         1700
--------------------------------------------------------------------------------

Now, sshp works on systems with kqueue - like MacOS and FreeBSD!

Super Mario 64 Decompilation on Void Linux

Dave Eddy — 2020-09-09T00:00:00-04:00

Super Mario 64 is one of my all-time favorite games. Last year, the source code for the decompilation project was made available to the public. I won’t go into detail on the decompilation project itself (see this post here for more information), but instead will focus on getting Super Mario 64 compiled on Void Linux.

Getting Started

A couple of dependencies will need to be installed that will be used throughout the installation process. Note that root privileges are only required in commands where sudo is used - the rest of the commands can be run as an unprivileged user.

sudo xbps-install git
sudo xbps-install make
sudo xbps-install wget

mkdir ~/dev ~/src

For this guide, the sm64 source code will be checked out into ~/dev, and the required dependencies that are installed manually will be in ~/src, of which there are 2:

qemu-irix
binutils-mips64

qemu-irix

To install this package, I found it easiest to take the qemu-irix release as a .deb file from the n64decomp GitHub organization and extract the deb myself.

In order to do this, you must first install a couple of dependencies:

sudo xbps-install binutils # installs 'ar'
sudo xbps-install libglib-devel
sudo xbps-install xz

mkdir ~/src/qemu-irix

Then, pull the package and “install” it (in this case, just extract it):

cd ~/src/qemu-irix
wget https://github.com/n64decomp/qemu-irix/releases/download/v2.11-deb/qemu-irix-2.11.0-2169-g32ab296eef_amd64.deb
ar x qemu-irix-2.11.0-2169-g32ab296eef_amd64.deb
tar xf data.tar.xz

Finally, test the installed qemu-irix program to ensure it runs:

$ ./usr/bin/qemu-irix -version
qemu-irix version 2.11.50 (v2.11.0-2169-g32ab296eef-dirty)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developer

binutils-mips64

This package will be installed manually, so a C compiler is required. Install gcc with:

xbps-install gcc

mkdir ~/src/binutils

Then, pull the source code and prepare it for compilation:

cd ~/src/binutils

wget ftp://ftp.gnu.org/gnu/binutils/binutils-2.35.tar.xz
tar xf binutils-2.35.tar.xz
cd binutils-2.35/

sed -i "/ac_cpp=/s/\$CPPFLAGS/\$CPPFLAGS -O2/" libiberty/configure

I don’t know if the sed command is strictly necessary - I based this process off of the mips64-elf-binutils PKGBUILD file used by the package in the AUR (for Arch Linux).

Next, compile the package:

./configure --target=mips64-elf --prefix=/usr --with-sysroot=/usr/mips64-elf --with-gnu-as --with-gnu-ld --enable-64-bit-bfd --enable-multilib --enable-plugins --disable-gold --disable-nls --disable-shared --disable-werror
make

Next, “install” the package into a local directory:

make "DESTDIR=$PWD/dest" install

Finally, test that the program works by executing mips64-elf-as:

$ ./dest/usr/bin/mips64-elf-as -version
GNU assembler (GNU Binutils) 2.35
Copyright (C) 2020 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `mips64-elf'.

Compile SM64

Finally, we are able to compile Super Mario 64!

Install the required dependencies:

sudo xbps-install audiofile-devel
sudo xbps-install python3

Clone the project from GitHub:

cd ~/dev
git clone git://github.com/n64decomp/sm64.git
cd sm64

In order to avoid any legal issues, the static assets from SM64 are NOT shipped with the decompiled source code. You must supply a rom to use as a “base”. In our case, we will need the following rom with the exact sha1 hash:

$ sha1sum 'Super Mario 64 (U) [!].z64'
9bef1128717f958171a4afac3ed78ee2bb4e86ce  Super Mario 64 (U) [!].z64

I can’t tell you where to get this rom so please don’t ask - but once you have the rom, move it into the decompilation repository and name it as follows:

mv ~/'Super Mario 64 (U) [!].z64' baserom.us.z64

Now we are almost ready to begin the compilation! The final step is to setup our PATH to use the programs we installed above. This step should be done once per session before you want to compile or recompile Mario.

PATH=$PATH:~/src/binutils/binutils-2.35/dest/usr/bin:~/src/qemu-irix/usr/bin

And finally, compile it!

$ make
...

build/us/sm64.us.z64: OK

Success! Part of the make process is that the sha sum being compared to the expected base rom, but we cany verify the output file manually with:

$ sha1sum build/us/sm64.us.z64
9bef1128717f958171a4afac3ed78ee2bb4e86ce  build/us/sm64.us.z64

Test The Rom

All of the above steps can be run on a headless Void Linux installation. The file ./build/us/sm64.us.z64 is created by the compilation process and can be used in any N64 emulator or even sent to a fashcart like the everdrive and ran on a real N64.

We can also test this on Void Linux with mupen64plus, assuming there is a windowing system installed and it is not a server/headless installation.

sudo xbps-install mupen64plus

Then, to test the rom:

mupen64plus --gfx mupen64plus-video-glide64mk2 --audio mupen64plus-audio-sdl --windowed ./build/us/sm64.us.z64

You can pass whatever options you like, but I find these create the most performant result. I also use a raphnet N64 to USB adapter to be able to use an original N64 controller on my computer.

This screenshot shows Mario is working as expected!

Rom Hacking

With all of this in place, we now have C source code that can be compiled perfectly into a Super Mario 64 rom file. As such, we are free to make any modifications to the source code we like.

As a simple example I had an idea for a (terrible) basic rom hack. When Mario is a certain distance away from the camera, for performance reasons, his high quality model is replaced with a low-polygon version of his model. This can become apparent during boss fights or when the camera is fixed, such as in the big house on Rainbow Ride. The idea is to make the low-poly version of Mario the default model regardless of camera distance.

In searching the source code to figure out how this logic is handled, I ended up actually learning something I didn’t know! This block comment in ./src/engine/graph_node.h explains what I didn’t know:

/** GraphNode that only renders its children if the current transformation matrix
 *  has a z-translation (in camera space) greater than minDistance and less than
 *  maxDistance.
 *  Usage examples: Mario has three level's of detail: Normal, low-poly arms only, and fully low-poly
 *  The tower in Whomp's fortress has two levels of detail.
 */

It turns out Mario has 3 levels of detail! I wasn’t aware of the “medium” level which is low-poly arms only. So already I’ve learned something new about this game!

Now, onto hacking - ./actors/mario/geo.inc.c controls the Mario model. The variables mario_geo_render_body and mario_geo are the important ones to note here. Starting with the first function:

// 0x17002D7C
const GeoLayout mario_geo_render_body[] = {
   GEO_NODE_START(),
   GEO_OPEN_NODE(),
      GEO_RENDER_RANGE(-2048, 600),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_body), // HIGH POLY (normal mario)
      GEO_CLOSE_NODE(),
      GEO_RENDER_RANGE(600, 1600),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_medium_poly_body), // MEDIUM POLY
      GEO_CLOSE_NODE(),
      GEO_RENDER_RANGE(1600, 32767),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_low_poly_body), // LOW POLY
      GEO_CLOSE_NODE(),
   GEO_CLOSE_NODE(),
   GEO_RETURN(),
};

I’ve added comments to better understand when each model is used. We can modify this to always load the medium poly Mario:

// 0x17002D7C
const GeoLayout mario_geo_render_body[] = {
   GEO_NODE_START(),
   GEO_OPEN_NODE(),
      GEO_RENDER_RANGE(-2048, 600),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_medium_poly_body), // HIGH POLY (normal mario)
      GEO_CLOSE_NODE(),
      GEO_RENDER_RANGE(600, 1600),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_medium_poly_body), // MEDIUM POLY
      GEO_CLOSE_NODE(),
      GEO_RENDER_RANGE(1600, 32767),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_medium_poly_body), // LOW POLY
      GEO_CLOSE_NODE(),
   GEO_CLOSE_NODE(),
   GEO_RETURN(),
};

Also, the variable below it needs to be modified as well:

// 0x17002DD4
const GeoLayout mario_geo[] = {
   GEO_SHADOW(SHADOW_CIRCLE_PLAYER, 0xB4, 100),
   GEO_OPEN_NODE(),
      GEO_SCALE(0x00, 16384),
      GEO_OPEN_NODE(),
         GEO_ASM(0, geo_mirror_mario_backface_culling),
         GEO_ASM(0, geo_mirror_mario_set_alpha),
         GEO_SWITCH_CASE(0, geo_switch_mario_stand_run),
         GEO_OPEN_NODE(),
            GEO_BRANCH(1, mario_geo_load_body), // THIS LINE
            GEO_BRANCH(1, mario_geo_render_body),
         GEO_CLOSE_NODE(),
         GEO_ASM(1, geo_mirror_mario_backface_culling),
      GEO_CLOSE_NODE(),
   GEO_CLOSE_NODE(),
   GEO_END(),
};

Changing the line that I commented earlier to match the polygon level we want:

GEO_BRANCH(1, mario_geo_load_medium_poly_body), // THIS LINE

With these two changes in place, we can now recompile our own personalized version of Super Mario 64 that will only use the medium-poly version of Mario.

$ make
...
tools/n64cksum build/us/sm64.us.bin build/us/sm64.us.z64
build/us/sm64.us.z64: FAILED
sha1sum: WARNING: 1 computed checksum did NOT match
The build succeeded, but did not match the official ROM. This is expected if you are making changes to the game.
To silence this message, use "make COMPARE=0".
make: *** [Makefile:339: all] Error 1

It broke?! Well not quite. The build was successful, but the sha1 hash check I mentioned above failed. This is ok - in fact this is expected - since we are modifying the source and intend to have a different version. We can instead run make with a variable set to not compare hashes after the build succeeds:

make COMPARE=0

Now, booting up the run with mupen64plus we can see what medium-poly Mario looks like.

The model looks mostly the same, but you can really see his arms just aren’t quite right.

This is cool, but I think we can make it more cursed. Let’s swap out the variables to only have low-poly Mario instead. The 2 variables inside ./actors/mario/geo.inc.c should look like:

// 0x17002D7C
const GeoLayout mario_geo_render_body[] = {
   GEO_NODE_START(),
   GEO_OPEN_NODE(),
      GEO_RENDER_RANGE(-2048, 600),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_low_poly_body),
      GEO_CLOSE_NODE(),
      GEO_RENDER_RANGE(600, 1600),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_low_poly_body),
      GEO_CLOSE_NODE(),
      GEO_RENDER_RANGE(1600, 32767),
      GEO_OPEN_NODE(),
         GEO_BRANCH(1, mario_geo_load_low_poly_body),
      GEO_CLOSE_NODE(),
   GEO_CLOSE_NODE(),
   GEO_RETURN(),
};

// This last geo is used to load all of Mario Geo in the Level Scripts

// 0x17002DD4
const GeoLayout mario_geo[] = {
   GEO_SHADOW(SHADOW_CIRCLE_PLAYER, 0xB4, 100),
   GEO_OPEN_NODE(),
      GEO_SCALE(0x00, 16384),
      GEO_OPEN_NODE(),
         GEO_ASM(0, geo_mirror_mario_backface_culling),
         GEO_ASM(0, geo_mirror_mario_set_alpha),
         GEO_SWITCH_CASE(0, geo_switch_mario_stand_run),
         GEO_OPEN_NODE(),
            GEO_BRANCH(1, mario_geo_load_low_poly_body),
            GEO_BRANCH(1, mario_geo_render_body),
         GEO_CLOSE_NODE(),
         GEO_ASM(1, geo_mirror_mario_backface_culling),
      GEO_CLOSE_NODE(),
   GEO_CLOSE_NODE(),
   GEO_END(),
};

Build the new rom with:

$ make COMPARE=0
...
mips64-elf-objcopy --pad-to=0x800000 --gap-fill=0xFF build/us/sm64.us.elf build/us/sm64.us.bin -O binary
tools/n64cksum build/us/sm64.us.bin build/us/sm64.us.z64

Run it with mupen64plus and:

Bam! the most cursed romhack of Super Mario 64 done by only modifying 4 lines of code.

Conclusion

This just scratches the surface of what could be done with the SM64 decompilation project. I hope to do more, and I hope this blog post also inspires you to do something with this :).

Some final thoughts I have are: how could this be made more efficient? If we are always using the same model, do we still need the logic to swap the model at a certain camera distance? Most likely not.

But for now the low-polygon version of Super Mario 64 that no one asked for is now here!

Looking Forward

I’m currently working on my first rom hack! The idea is very simple: toggle the water physics on and off completely with the L button. This means the whether will be completely dry, or completely flooded, with the push of a button. It’s already making for some really cool speedrunning strats:

thanks to the Super Mario 64 decomp project I present to you...
cursed sm64 decomp strat 1: ship clip 2.0 pic.twitter.com/Ue57mOkTiD
— dave eddy (@bahamas10_) September 9, 2020

Pump It Up Pad Refurbishment and Upgrade

Dave Eddy — 2020-02-06T00:00:00-05:00

In 2018 I created my own DIY Pump It Up Pad Platform With Bars - it was a combination of Precision Dance Pads and a platform with bars I created myself. This setup was… decent at best. I wanted the full arcade experience, and knew that nothing was going to satisfy that except for the real arcade pads.

Luckily for me, my friend Kate added me to a buy/sell/trade Facebook group for arcade rhythm games, and within the first week there was a post for used Pump It Up pads for sale about 8 hours away from my house. I jumped on that opportunity immediately!

Spoiler: Scroll down to see videos of the pads in action!

Getting the Pads

I woke up early on a Sunday, got in my (relatively small) car, and drove 8 hours from Buffalo, New York, to Maryland. I picked up the pads from the seller, barely got them loaded into my car, and drove 8 hours back home. It was a crazy day but I managed to stop once on the way there, and once on the way back at Sheetz so it was a good day :).

These pads originated from a PIU Zero machine in Ontario, Canada.

The pads inside my house ready to be connected to the computer with StepF2. In order to do this I had remove the top most panel to swap out the existing PCB with the BARR.io adapter.

Jess and Kate testing out the pads! They were not structurally sound at the time of testing :p.

I added the bar to the right pad. I also started doing some basic cleanup beneath the panels - the pads were really dirty.

The left pad with all of the panels removed. I went through and gutted the entire pad to clean it and wipe it all down.

Another angle of the pad close-up.

I went through a lot of paper towels and lysol wipes cleaning this pad. It was insane how much dirt and other stuff managed to get in there during its time at the arcade.

Moving the old set of pads out of the way.

Reassembling the pads now that they are cleaned up.

Pad Refurbishing

At this point, the pads have been cleaned and the USB adapters have been installed, and the pads all work! However, they have a lot of issues. Some of the panels are more responsive than others, and some just really need to be stomped on to register a click.

I followed the advice in this video. Using this I:

Replaced the metal bracket pads with Gorilla tape. The existing pads had absolutely no give to them.
Countersunk the metal corner brackets so the screws wouldn’t protrude and catch your feet.

Metal Bracket Padding

The existing metal brackets with the worn out pads.

Scraping off the existing padding.

It took a long time but I eventually got all of them clean.

Using Gorilla tape to add new life to the padding on the metal brackets. I would try to keep the brackets in groups of 5 to make them easier to cut.

I used an exacto knife to cut them into the groups of 5, and then I would put the groups of 5 back onto Gorilla tape until there were about 7 layers of tape. From there, I would use the knife to cut them one-by-one.

I used this paper cutter thing to cut off any excess tape from the brackets.

Reassembling the panels with the newly padded metal brackets.

Countersinking Corner Brackets

The next step in refurbishing was to countersink the metal corner brackets. This will make it so the screws sit flush with the pads.

The old style bracket on the left and the new style on the right.

Another before and after shot - the bottom hole is countersunk and the top is not.

A before picture of the metal bracket installed without the countersunk holes.

An after picture of the metal bracket installed with the countersunk holes.

One panel installed in all of its glory.

Slowly reassembling the pads.

After removing the old PCB I used Gorilla tape to cover the metal on the inside to ensure that there’s no chance of shorting the BARR.io adapter.

Pad almost reassembled.

Pad reassembled.

The left pad is complete - on to working on the right pad.

Both pads fully assembled!

Attaching the Pads

In order to connect the two pads to each other, almost all arcade pads have a metal bracket on the front and the back that screw into the pads and hold them in place. The pads I bought didn’t come with the bracket, so I had to improvise. I decided to save money, and make my own brackets out of plywood.

To do this, I used a piece of paper held over the pads, and stabbed holes where the screw holes are. Then, I used that as a template to cut pieces of plywood.

Two pieces of plywood cut.

Spray painted black.

I taped the 2 plywood pieces together and then used the template to scroll holes through them.

Screw holes cut!

Installed with some of the screws in place.

The pads held together with the new wooden brackets.

Customizing

Nothing too fancy, but I taped USB LED lights to the bottom of the pads.

Same pic in the dark.

A pic of the pads with the lights on.

Same pic in the dark.

The final thing is some of the caster wheels were broken on the original pads (or just missing), so I hit up Lowes and grabbed a couple of new wheels of the same size.

New wheels installed.

Bracketless Upgrade

Even though this is a single blog post, this project spanned over the duration of an entire year. I’ve used the pads with the metal corner brackets for about 6 months without issue and things have been good. However, playing some of the newer charts as well as still playing on the machine at the arcade, I decided that I wanted to upgrade my pads to be the newer style without the metal brackets.

On the same Facebook group where I found the seller of the pads, I found another seller that was selling a full “upgrade kit” complete with all new panels and frames. I bought the new kit and followed the below video.

The upgrade kit.

The YouTube video above outlines the screws needed to install the newer panels in the description. Below is the full list of screws I purchased from BoltDepot.

Part Number	Name	Quantity	Price
5187	Metric machine screws, Phillips pan head, Stainless steel 18-8 (A-2), 6mm x 1.0mm x 10mm	1 bag of 100	$13.31
5179	Metric machine screws, Phillips pan head, Stainless steel 18-8 (A-2), 5mm x 0.8mm x 20mm	1 bag of 100	$10.17
4824	Metric lock washers, Zinc plated steel, 5mm	1 bag of 100	$2.18
4526	Metric flat washers, Zinc plated steel, 5mm	1 bag of 100	$1.40

I had to Gorilla Glue the sensor holders onto the plastic frames.

The middle panel installed alongside the older panels.

One pad done! You can see how ridiculous it looks having an upgraded pad next to an older style pad ha.

Another angle of one completed pad.

The left pad with the new frames installed.

Almost all of the panels installed (without any lighting as the BARR.io currently doesn’t support lights).

9/10 new panels installed.

The Finished Pads

The fully upgraded pad!

The Pads In Action

I sometimes stream on twitch.tv!

void - Idealized Romance S16

Beltaine - Rockhill D12

DIY Wooden Long Desk

Dave Eddy — 2019-12-17T00:00:00-05:00

In 2016 I built an Overwatch Themed Desk for my bedroom to use with my gaming machine. Recently, I’ve been getting really into streaming on Twitch and I needed more desk space. I want to do more tech related streams, and gaming in general, and need more desk space for multiple computers.

I decided I should build a new longer desk to replace the desk I currently use. I wanted it to be the same height and depth, but this time make it 6 feet long.

The desk proportions are:

6’ wide
30” deep
30” height

I used the following pieces of lumber to make this desk:

6x 1x6x6 (trimmed to 5” wide for each board)
3x 2x4x8
3x 4x4x8

I cut the pieces down to the proper sizes first.

I used my table saw to trim off edges of the lumber to get rid of the rounded edges.

Lining up the 1x6 (now 1x5) pieces to get ready to glue for the desktop.

Pieces glued and lined up.

Glued and clamped.

I lined up the 4x4’s by eye only to figure out where I wanted the legs to be.

Using a pocket hole jig to get the 2x4’s ready to support the desktop.

I used an electric hand planer to get rid of the excess glue and prep the desktop for sanding.

Another shot of the desktop.

Random orbit sander for the desktop next.

I used my chop saw to cut the logs of the 4x4’s to look a little fancy.

I used my table saw to start cutting the cross halving joint on the support pieces.

Cross halving joints so far.

Beginning of assembling the desk.

The desk with the desktop (not assembled yet, just a teaser).

Another shot of the assembly.

Getting the support beams ready to glue.

Side and back supports have been glued and clamped.

Another shot of the glued and clamped supports.

Getting the frame and desktop ready to stain.

Espresso stain on the frame and Golden Oak stain on the desktop.

I’m an idiot and completely missed staining this part of the frame so I had to stain it separately.

The stained and polyurethaned frame in my house.

Signed my work the same way I did with my Overwatch desk :).

Overhead view of the table top (i love it!).

A shot of the finished desk.

The desk in its new home.

A wide-angle lens shot of the desk in its new home.

The desk with my streaming gear on it!

SmartOS COAL on Linux KVM with Virt Manager

Dave Eddy — 2019-02-12T00:00:00-05:00

COAL, or Cloud-on-a-Laptop, is an easy way to run and test SmartOS and the Triton Stack in a self-contained VM meant to be run on a laptop. The sdc-headnode repository makes a lot of assumptions when building a COAL image about using VMWare for virtualization on an OS X laptop. However, with some modifications it is possible to run COAL on Linux via KVM managed through virt-manager.

To get virt-manager setup and running on Void Linux, you can follow my guide KVM Virtualization with virt-manager on Void Linux.

Nested Virtualization (optional)

Before getting started with the COAL building process, you can choose to enable nested virtualization if your hardware supports it. This is required if you would like to run bhyve zones on your COAL setup. Note that this setup assumes an Intel CPU.

You can check if your CPU supports nested virtualization by running:

$ grep ^flags /proc/cpuinfo | grep -c ' vmx '
8

If the number returned is greater than 0, then nested virtualization is supported. Create the following file:

/etc/modprobe.d/kvm-nested.conf

options kvm-intel nested=1
options kvm-intel enable_shadow_vmcs=1
options kvm-intel enable_apicv=1
options kvm-intel ept=1

You can either reboot the machine here for good measure or remove and re-add the module to enable nested virtualization:

sudo modprobe -r kvm_intel
sudo modprobe -a kvm_intel

Finally, you can check to make sure it worked by running:

$ cat /sys/module/kvm_intel/parameters/nested
Y

It will return Y if nested virtualization is enabled.

Getting Started

You’ll need some external tools to build a COAL image. On Void Linux these can be installed with:

sudo vpm i git nodejs parted ksh

To get started, pull the sdc-headnode source code:

git clone git://github.com/joyent/sdc-headnode.git
cd sdc-headnode

I use a custom build.spec.local file, as well as an answers.json file. Feel free to modify these to fit your environment - I put what I consider the fields that can be easily modified without issue at the top of the files. The IP addresses and NIC mac addresses will be used later on.

build.spec.local

{
    "coal-memsize": 8192,

    "answer-file": "answers.json",
    "build-tgz": "false",
    "vmware_version": 7,
    "default-boot-option": 1,
    "clean-cache": true
}

answers.json

{
    "root_password": "root",
    "external_nic": "52:54:00:00:00:01",
    "external_ip": "10.88.88.200",
    "admin_nic": "52:54:00:00:00:00",
    "admin_ip": "10.99.99.7",

    "config_console": "vga",
    "skip_instructions": true,
    "simple_headers": true,
    "skip_final_confirm": true,
    "skip_edit_config": true,
    "datacenter_company_name": "Joyent",
    "region_name": "coal",
    "datacenter_name": "coal",
    "datacenter_location": "My Laptop",
    "admin_provisionable_start": "<default>",
    "admin_netmask": "255.255.255.0",
    "admin_gateway": "10.99.99.7",
    "setup_external_network": true,
    "external_vlan_id": "<default>",
    "external_provisionable_start": "<default>",
    "external_provisionable_end": "<default>",
    "external_netmask": "255.255.255.0",
    "external_gateway": "10.88.88.1",
    "headnode_default_gateway": "<default>",
    "dns_resolver1": "<default>",
    "dns_resolver2": "<default>",
    "dns_domain": "joyent.us",
    "dns_search": "joyent.us",
    "dhcp_range_end": "10.99.99.253",
    "ntp_host": "<default>",
    "admin_password": "joypass123",
    "api_password": "joypass123",
    "mail_to": "<default>",
    "mail_from": "<default>",
    "phonehome_automatic": "<default>",
    "update_channel": "release"
}

Again, take care when modifying any of these values because other values may have to be modified as a result (like an interfaces IP address and its gateway address).

Build COAL

With answers.json and build.spec.local in place, the next step is to build a COAL image. This can be done simply by running make usb - I have a small script that I use to do this.

The script will run make usb to build the USB image and then follow it up by creating a symlink ~/vms/coal.img that points to the latest COAL image built:

do-make

#!/usr/bin/env bash
#
# stupidly simple script to make COAL and link it to an img file

export MANTA_NO_AUTH=1

img=~/vms/coal.img

make usb || exit

latest=(usb-master-*/*.img)
latest=${latest[-1]}

mkdir -p "${img%/*}"
ln -svf "$PWD/$latest" "$img"

Note that this script can be run any time to build a new COAL image, but it should be run when the COAL VM is not running:

$ ./do-make
...
sent 2,984,703,731 bytes  received 5,321 bytes  192,561,874.32 bytes/sec
total size is 2,983,956,095  speedup is 1.00
WARNING: Copying in answers.json, remove answer-file from build.spec* to disable.
==> Cleaning up
==> Creating usb-master-20190210T041835Z-gafc88889/
'/home/dave/vms/coal.img' -> '/home/dave/dev/sdc-headnode/usb-master-20190210T041835Z-gafc88889/usb-master-20190210T041835Z-gafc88889-4gb.img'

Every time do-make is run, a new USB image will be created. I also have a small script called clean that can delete all old builds and keep the latest X (2 by default):

clean

#!/usr/bin/env bash
#
# clean builds from sdc-headnode

# keep this number of the latest builds
keep=2

for f in boot usb; do
	arr=("$f"-*)
	len=${#arr[@]}
	if ((len > keep)); then
		((len -= keep))
		arr=("${arr[@]:0:len}")
	else
		continue
	fi

	for d in "${arr[@]}"; do
		[[ -n $d && -d $d ]] || continue
		echo "removing $d"
		rm -rf "$d"
	done
done

Setup Virtual Networks

COAL requires 2 networks to be setup: admin and external. To create these networks, double-click the “QEMU/KV” section of the virt-manager GUI:

This will open an overview page. From there click the “Virtual Networks” tab and create 2 networks as follows.

Note that you may already have a default network created; this is ok and can be left alone as we won’t be using it for COAL.

Create the external network with the following settings:

Network: 10.88.88.0/24
DHCP range: disabled
Forwarding: NAT

Create the admin network with no IP configuration whatsoever:

Create the VM

With the 2 virtual networks, the VM can now be created. To create the VM, click the “new” button in virt-manager to open the wizard and start the process:

Step 1: Select “Local install media”. The architecture should be set to x86_64 (this should be default):

Step 2: Specify the location of the USB image created above, and set the OS to “Generic default”:

Step 3: Set the memory to 8192 (or the value specified in build.spec.local) and the number of CPUs to 4 (or whatever you’d like):

Step 4: Enable storage for the VM and create a disk (I personally use 100GiB just to make sure that running out of storage space isn’t an issue):

Step 5: Name the instance (I chose to call mine COAL), and tick “Customize configuration before install” - there are still some steps to finish before the VM is ready to run:

Customize the VM

Configure CPU

If you’d like to have nested virtualization (assuming your hardware supports it), set the CPU model to host-passthrough. This option will have to be written in manually as it will not come up automatically or with autocomplete:

Configure Network

Click the “Add Hardware” button to add a second network device.

Create the admin NIC by setting the network source to “Virtual Network: admin”, and set the MAC address to the value used in answers.json as admin_nic:

Create the external NIC by setting the network source to “Virtual Network: external”, and set the MAC address to the value used in answers.json as external_nic:

Configure USB

The USB image file will be setup as an “IDE CDROM” drive. Delete this device, and then click “Add Hardware” to add a new “Storage” device. Check the “Select or create custom storage” option and set the following parameters:

Path: ~/vms/coal.img (or path where do-make linked the latest image)
Device type: Disk Device
Bus type: USB

With the newly created USB device, go to the “Boot Options” section and make sure it is checked to be the only bootable device - this device contains the SmartOS image itself and will be used to boot the OS:

Configure IDE Disk

I’ve personally noticed that configuring the IDE disk to use the VirtIO Disk bus has resulted in better performance in the VM. Click the IDE disk in the configure window and change the “Disk bus” setting to VirtIO:

Boot the VM

Click the “Begin Installation” button at the top-left of the window to boot the VM and start the installation!

After the machine boots a lot of work will be done to finalize the install. The VM will reboot on its own during this process and then will provision every zone needed for COAL to function. A screen like this should appear on the console when it has finished installing:

Logging in

Once the setup has completed, the COAL instance is now ready to be used! To login to the machine, you can add a section your ~/.ssh/config file for COAL:

~/.ssh/config

Host coal
	ServerAliveInterval 120
	User root
	Hostname 10.88.88.200
	ForwardAgent yes
	StrictHostKeyChecking no
	UserKnownHostsFile /dev/null
	ControlMaster no

Then, with this in place, login simply with ssh coal (the password is root or whatever was set in answers.json):

$ ssh coal
Warning: Permanently added '10.88.88.200' (ECDSA) to the list of known hosts.
Password:
- SmartOS (build: 20190208T131006Z)
[root@headnode (coal) ~]#

You can get DC information with sdcadm:

[root@headnode (coal) ~]# sdcadm experimental info
Datacenter Company Name: Joyent
Datacenter Name: coal
Datacenter Location: My Laptop

Admin IP: 10.99.99.7
External IP: 10.88.88.200
DNS Domain: joyent.us

To check that nested virtualization worked as expected and that bhyve zones can be provisioned, sysinfo can be checked for the Bhyve Capable property:

[root@headnode (coal) ~]# sysinfo | json 'Bhyve Capable'
true

Health checks can be run to verify that everything is working as expected:

[root@headnode (coal) ~]# sdc-healthcheck
ZONE                                 STATE           AGENT               STATUS
global                               running         -                   online
assets                               running         -                   online
sapi                                 running         -                   online
binder                               running         -                   online
manatee                              running         -                   online
moray                                running         -                   online
amonredis                            running         -                   online
ufds                                 running         -                   online
workflow                             running         -                   online
amon                                 running         -                   online
sdc                                  running         -                   online
papi                                 running         -                   online
napi                                 running         -                   online
rabbitmq                             running         -                   online
imgapi                               running         -                   online
cnapi                                running         -                   online
dhcpd                                running         -                   online
fwapi                                running         -                   online
vmapi                                running         -                   online
ca                                   running         -                   online
mahi                                 running         -                   online
adminui                              running         -                   online
global                               running         ur                  online
global                               running         smartlogin          online

And:

[root@headnode (coal) ~]# sdcadm health
INSTANCE                              SERVICE          HOSTNAME  ALIAS       HEALTHY
e3512f4b-47d0-42b4-b181-510c44631086  adminui          headnode  adminui0    true
4ba89fe1-739b-4689-b5be-2b106c8bab92  amon             headnode  amon0       true
7ed7e5cf-8867-4e7b-a882-57c2662c0487  amonredis        headnode  amonredis0  true
a4f91949-8427-4926-9933-2f0ffe18c5a5  assets           headnode  assets0     true
bf778a2c-1b65-4899-8b67-1314582bb484  binder           headnode  binder0     true
96a42299-d4da-4423-a1cd-75b0245bb0ed  ca               headnode  ca0         true
98f20bf8-796c-406c-9924-c16e9a40703d  cnapi            headnode  cnapi0      true
e00d35b2-9018-478d-85db-bfe944319a0a  dhcpd            headnode  dhcpd0      true
afaff931-d8b2-4173-94e2-3306dff79179  fwapi            headnode  fwapi0      true
b3ce95a5-a255-4b00-93c9-f3055d8471e4  imgapi           headnode  imgapi0     true
114bd1d3-e174-4585-9ab7-977260237c9d  mahi             headnode  mahi0       true
0e40cd43-4556-401c-85d8-98f766996cf3  manatee          headnode  manatee0    true
465a7ca3-5e17-4740-a0cb-f3f55089a7b0  moray            headnode  moray0      true
142a8178-d17b-43a4-bcf6-36968212041d  napi             headnode  napi0       true
847f96a2-f943-458f-9d71-892d1f273b15  papi             headnode  papi0       true
3d8d8c0a-33c1-4c8c-b95b-4143d829b795  rabbitmq         headnode  rabbitmq0   true
5ea05603-e4f3-4618-83c5-16f876293c39  sapi             headnode  sapi0       true
8071a1a6-f056-4e6d-8eac-e6ac7715f7d6  sdc              headnode  sdc0        true
05924e1d-4948-49d1-8e26-ceef20dc5118  ufds             headnode  ufds0       true
69c8427c-42b0-4c3c-94e8-dcc4509c2fa6  vmapi            headnode  vmapi0      true
15c34920-c423-4f52-87f8-12b6079cf0fa  workflow         headnode  workflow0   true
9ad2f41c-13c4-b541-bd60-0c6230916ed7  global           headnode  global      true
4b04c9ce-8776-482f-8655-4aaa2f56f417  amon-agent       headnode  -           true
40ed0710-f5be-4948-b057-b9b7326790a1  amon-relay       headnode  -           true
13e16c93-2568-4368-ab82-ff2d306d2fb8  cmon-agent       headnode  -           true
6bafb5d9-a931-4dc3-a649-4c71a1e91823  cn-agent         headnode  -           true
07214449-0e9e-431b-951e-27d6c697f527  config-agent     headnode  -           true
732f2d92-e67a-40d6-bfe5-f0edf66b16b0  firewaller       headnode  -           true
e355af90-5bc7-4d2e-8eb4-dd3009090c3f  hagfish-watcher  headnode  -           true
ca2a413b-58fb-4a5f-85ef-25aa7c9e36ab  net-agent        headnode  -           true
8062f7c5-9afe-42fa-ab6a-af28d2d35522  smartlogin       headnode  -           true
f0e74f08-ac1e-4952-8b38-624825d48196  vm-agent         headnode  -           true

And finally, the test suite can be run against the platform (note this will take a long time, but is useful for testing changes to the platform itself):

[root@headnode (coal) ~]# touch /lib/sdc/.sdc-test-no-production-data
[root@headnode (coal) ~]# /usr/vm/test/runtests
# Running /usr/vm/test/tests/test-50-creates.js
Added "imgapi" image source "https://images.joyent.com/"
...

Staying up-to-date

With the COAL VM powered down, the do-make script can be run at any time to build a new COAL image with all of the latest agents and platform bits. After that finishes and the USB image is linked, simply turn the VM back on to run the latest version.

Also, I have the update-sdc script I use in the below section to keep an image updated without having to rebuild the entire image.

Development and Tools

Everyone seems to have their own methods for working with COAL and SmartOS in general, myself included. I have created a bunch of tools along the way that have helped when developing on/for SmartOS.

`update-sdc`

This script is useful for updating sdc/triton on a COAL instance without having to rebuild the image.

All steps are based on the update documentation in the sdcadm repository:

https://github.com/joyent/sdcadm/blob/master/docs/update.md

update-sdc

#!/usr/bin/env bash

set -o errexit

sdcadm() {
    echo "== sdcadm $* =="
    command sdcadm "$@"
}

sdcadm self-update --latest

sdcadm experimental update-gz-tools --latest --just-download
sdcadm experimental update-agents -y --latest --just-download --all

sdcadm dc-maint start

uuid=$(updates-imgadm list --latest name=agentsshar -o uuid -H)

[[ -n $uuid ]]

found=false
for f in /usbkey/extra/agents/*; do
	if [[ $f =~ $uuid ]]; then
		found=true
		break
	fi
done
if [[ $found == false ]]; then
	sdcadm experimental update-agents --latest --all --yes
fi

sdcadm experimental update-other
sdcadm experimental update-gz-tools --latest

sdcadm up -y --all --force-data-path -x sapi -x moray -x binder -x manatee
sdcadm up sapi -y
sdcadm up moray -y
sdcadm up binder -y
sdcadm up manatee -y

sdc-healthcheck
sdcadm health

sdcadm dc-maint stop

`pull-latest-platform`

This script will pull the latest compiled platform tarball from my build zone. I will iterate in the smartos-live repo and often make changes to the platform that need to be tested. This script, when ran on COAL, will reach out to my build zone and pull the latest compiled tarball and install it. It’s not the prettiest script, but for what it does it works and it makes development very easy.

Swap out the latest variable to fit your environment:

pull-latest-platform

#!/usr/bin/env bash

dir='/var/tmp/platforms'
stamp=$(TZ=UTC date '+%Y%m%dT%H%M%SZ')
file=$dir/$stamp.tgz
latest='dave@10.0.1.29:joyent/smartos-live/output/platform-latest.tgz'

mkdir -p "$dir"
cd "$dir" || exit

echo "-- pulling latest from build to $stamp --"
scp -o StrictHostKeyChecking=no "$latest" "$file" || exit

echo '-- cleaning key --'
/usbkey/scripts/cleanup-key.sh

echo '-- installing --'
sdcadm platform install "$file"

echo '-- assigning --'
sdcadm platform assign "$stamp" --all

Conclusion

Having COAL on my laptop makes it really easy to test and develop on SmartOS with the latest code. My personal development cycle is fairly straight-forward:

Build a COAL image using the above steps.
Optionally compile a SmartOS platform.
Boot the COAL image.
Optionally install the compiled platform.
Do whatever testing I need, then go back to step 1 or 3 depending on what I’m testing specifically.

Having COAL as easily on Linux as I do on OS X makes it simple to just focus on the actual development and iterative process involved with that, as opposed to worrying about getting the underlying system set up. virt-manager and the virsh command make it easy to manage the lifecycle of the VM as well, although normally I only need the ability to start and stop the VM.