I recently made a new storage server to replace my old one to keep up with my growing space requirements (I think 40T should hold me over for a while!). I store all of my movies, music, tv shows, etc. on it, as well as all of my backups. All of my laptops and desktop computers also backup to this server using rsync.

While it’s all stored on SmartOS using the ZFS filesystem in a raid setup that can handle 2 or more drive failures without data loss, it still worries me because it is all stored in one physical location: my closet. If there is a fire or some other disaster like that, all of my data could potential be lost.

To remedy this, I’ve repurposed the server I replaced (my old storage server) to be an off-site backup server that is used solely for ZFS receive. This server now runs FreeBSD, which you can read about in my blog post here

Automatic Snapshots

Before diving into my off-site backup solution, the first thing to talk about is how I handle automatic ZFS snapshots, and also removing snapshots as they get too old.

zfs-snapshot-all

https://github.com/bahamas10/zfs-snapshot-all

Recursively snapshot all zpools

I use this program to snapshot all zpools on my new storage server automatically in cron. My crontab looks something like this:

Read More...

I recently set up a server at my buddies house for remote, off-site, backups using ZFS send/recv. Since SmartOS is lacking any sort of encryption for ZFS I instead used FreeBSD with geli. This way, I get the benefits of ZFS incremental send and receive for doing backups, with the security of knowing my data is encrypted on disk.

To ensure maximum security, I made sure the passphrase for unlocking the drives was not stored anywhere on the server itself, and instead would require manual intervention on my part to decrypt the drives anytime the server is rebooted. To do this, I created 2 pools:

• zroot - the main pool, mounted at /, unencrypted
• paper - the pool used for backups, mounted at /paper, encrypted

The server will boot into the operating system automatically, and from there I must run a script to unlock the drives and import the paper pool.

PS: The zpool is called paper because the server is called paperdyne. That name was cleverly derived from dataDyne (my storage server) and @papertigerss (my buddy whose house this server now lives).

Setup

Install FreeBSD

The first step, of course, is to install FreeBSD. Go to the website, make a bootable USB (or DVD, or CD, or whatever) and install it.

Read More...

It’s been 5 years since I created my first storage server and was now approaching full capacity - I was at 90% used out of 9.5TB available. I built a new server with over 40TB usable to replace this server, and will be using the old server for offsite backups eventually.

The new server will run SmartOS like the server it is replacing, and will run almost all of the same zones as the original

Parts

Server parts

• 20x Hitachi Global Storage 0B26885 UltraStar 4TB SAS2 3.5in 7200RPM 64MB Bare
• 5x Cable Matters® Mini-SAS to 4 SATA Forward Breakout Cable - 1m
• 3x SAS9211-8I 8PORT Int 6GB Sata+sas Pcie 2.0
• 2x Crucial 16GB Kit (8GBx2) DDR3/DDR3L-1600MT/s (PC3-12800) DR x8 ECC UDIMM Server Memory CT2KIT102472BD160B/CT2CP102472BD160B
• 1x Thermaltake Cooling Fan for Intel Core i7/i5/i3 CL-P0556
• 1x SUPERMICRO MBD-X9SCL-F-O LGA 1155 Intel C202 Micro ATX Intel Xeon E3 Server Motherboard
• 1x Intel Xeon Quad-Core Processor E3-1230 v2 3.3GHz 8MB LGA 1155 CPU LGA BX80637E31230V2
• 1x Corsair CX Series 750 Watt ATX/EPS Modular 80 PLUS Bronze ATX12V/EPS12V 744 Power Supply CX750M

Rack and case parts

• 1x Tripp Lite SR4POST13 13U 4-Post Open Frame Rack Cabinet Square Holes 1000lb Capacity
• 1x NORCO 4U Rack Mount 20 x Hot-Swappable SATA/SAS 6G Drive Bays Server Rack mount RPC-4020
• 1x RL-26 26” 3-sections Ball Bearing Slide Rail
• 1x Norco 120mm fan wall bracket
• 3x Noctua NF-P12 120mm x 25mm Cooling Fan 3-Pin - 1300 RPM
• 2x Noctua NF-R8 80mm Fan (31 CFM)

Specs

The new build has 20x 4TB SAS drives, 32GB of ECC ram, and an Intel Xeon Quad Core 3.3GHz processor.

Memory

# prtconf | grep Memory
Memory size: 32740 Megabytes

Processor

# psrinfo -vp
The physical processor has 4 cores and 8 virtual processors (0-7)
  The core has 2 virtual processors (0 4)
  The core has 2 virtual processors (1 5)
  The core has 2 virtual processors (2 6)
  The core has 2 virtual processors (3 7)
    x86 (GenuineIntel 306A9 family 6 model 58 step 9 clock 3300 MHz)
      Intel(r) Xeon(r) CPU E3-1230 V2 @ 3.30GHz

Read More...

Graphite is an open source system for graphing data. It manages storing data (in a database format called “whisper”), as well as graphing data (with a project called “graphite-web”).

These instructions are fairly platform-independent, and should work on any Unix operating system - the only SmartOS specific bits will be at the end to create SMF services that ensure the services will start at boot.

The above graph is the temperature of my garage, as monitored by a Raspberry Pi, over a 1 month period.

Overview

There are 3 projects that will be used to setup the graphite server:

1. carbon - stats receiving daemon (called carbon-cache) that manages storage
2. whisper - database file format used by carbon to store data
3. graphite-web - web interface to compose graphite graphs

Prerequisites and Dependencies

To get started, create a user for the graphite services to run as:

groupadd graphite
useradd graphite

Read More...

Sending email on the command line is nothing new; In fact, it is not even all that exciting. When I got Nagios up and running at home, I needed an easy way to have it email me whenever there was an alert. When I worked at Voxer as an Operations Engineer I created a program to allow Nagios to generate HTML emails - now all I needed at home was to allow outbound email.

This, however, turned out to be more difficult than I thought, mostly because I was (and largely, still am) ignorant to the intricacies of the various email protocols in use today on the Internet. Joyent released a blog post when I was setting this all up that went over the various options for sending email from a SmartOS server. One of the options they mentioned but didn’t choose to implement really caught my eye: Google’s free SMTP server. Since my main email address is a gmail address (well, Google Apps), this route seemed perfect.

gmailx

gmailx was born! From the documentation:

Send email easily on the command line without running a server

This program is basically mailx but hardwired for Gmail (hence the name). Usage is simple:

Read More...

I have 2 SmartOS servers at home with a total of 17 zones running persistently - a majority of these provisioned with the latest LTS support release of pkgsrc: 2014Q4. All zones are setup to download a similar set of “bootstrap” packages to make them ready to be used. Packages like gcc, git, etc. I pull to all of my zones upon creation.

The problem, however, is my home internet is not the fastest - it usually can pull packages at around 100KB/s. git and gcc alone are responsible for over 100MB of compiled binary data, meaning these initial package downloads can take over 15 minutes per new zone.

A possible solution to speed up initial package downloads I investigated was to rsync the entire pkgsrc tree to a local server, and host it over an internal HTTP server. This, however, required a massive amount of storage dedicated to packages I am almost guaranteed to never use. I could have just pulled 2014Q4, but then I would have been required to add new releases whenever I wanted to upgrade, and keep around the old packages until all my old zones have been upgraded.

Instead, I created a Node.JS HTTP caching server to proxy GET and HEAD requests to Joyent’s pkgsrc server, and cache the data on the local filesystem. The requests will be proxied if the file does not exist locally, and will be dual-written to the requesting client, and the local filesystem. Subsequent requests for the same resource will be streamed from the local filesystem without every making an outbound request.

fs-caching-server

https://github.com/bahamas10/node-fs-caching-server

The program is written to be generic - it allows it to act as a caching proxy to any website, and also allows the user to specify a regex to test a URL when deciding if the request should be cached, or just proxied directly with no cache.

Read More...

For the last couple of years I’ve used PFSense as my home router. It’s been great - it’s easy to manage with the web interface, and really easy to add features like an OpenVPN server, bandwidth monitoring, etc.

But, I like to manage all of my servers and zones at home using Chef, and my router has always been left out as it required being configured manually through the web interface. So now, I’ve replaced PFSense with 2 SmartOS zones: 1 for NAT and the other for DHCP - both managed by Chef, and both monitored with Nagios. This also had the added effect of reducing two physical servers at home down to one, for a cheaper power bill.

There are 3 steps required to configure a SmartOS server as a home router replacement.

1. Add the External Interface
2. Create the NAT Zone
3. Create the DHCP Zone

1. Add the External Interface

The first thing is to ensure that you have 2 (or more) NICs on the SmartOS server - one for the external network (The Internet) and one for the internal network that will be behind NAT. To list the interfaces run

root - datadyne sunos ~ # dladm show-phys -m
LINK         SLOT     ADDRESS            INUSE CLIENT
rge0         primary  f4:6d:4:X:XX:XX    yes  rge0
e1000g0      primary  0:4:23:XX:XX:XX    yes  e1000g0

Cross-referencing this information with output from ifconfig(1M) reveals that the e1000g0 interface is not currently in use, and can be used as the external interface.

root - datadyne sunos ~ # ifconfig e1000g0
ifconfig: status: SIOCGLIFFLAGS: e1000g0: no such interface

Read More...

Sonos is a fantastic product that makes it easy to set up whole-house audio in a piecemeal fashion. I started with one speaker, then a couple weeks later was up to 4 speakers, and a couple months after that the whole house is equipped with enough speakers for music to be heard anywhere.

In order to stream your own music collection to Sonos, it must be available over the network using a supported protocol - sadly at the time of this writing NFS is unsupported by Sonos. In order to get Sonos up and running on my network, I setup a cifs/samba read-only share with my music collection inside a SmartOS zone.

This post is basically a modified version of Jonathan Perkin’s and Thomas Merkel’s posts covering mounting in a shared directory with music using lofs.

create the zone

To create the zone you have to first create a JSON manifest to be used by the vmadm(1M) command. This is the JSON manifest for the cifs zones called cifs.json I used. The filesystems array mounts in /goliath/entertainment from the Global Zone to /entertainment inside the zone. I do this because my entertainment directory is used by multiple zones - for example my plex zone uses it to share my plex library.

Read More...

djbdns is a software package for running a secure, fast, and simple DNS server.

djbdns is not actually a program itself, but instead is a collection of programs that can be used together to create a full caching, forwarding, and authoritative DNS system - this post will show how to setup all the necessary programs on SmartOS to mimic my home setup.

I use djbdns at home to give me:

1. DNS lookups for my internal network: rapture.com and 10.X.X.X
2. DNS caching daemon for quick look ups: it forwards to OpenDNS and caches the results

Installation

To install the suite of tools, run

pkgin in djbdns gmake

Note: the GNU version of make isn’t needed specifically, any implementation will suffice.

This will install a lot of programs, but the most important are:

• tinydns: a DNS server daemon
• dnscache: a recursive DNS caching daemon

The next step is to create the configuration directory which will be used later

mkdir -p /opt/local/etc/djbdns

Read More...

__dirname is a variable that is available to all scripts that are run by Node.JS - it contains a string that refers to the directory name where the currently executing script is contained. The variable is set on a per-file basis, so any script that is sourced using require will have its own __dirname variable that points to the directory where the script itself is contained.

Example In Node

Take the following node script located at /tmp/node-one

console.log(__dirname);

When executed, we can see

$ node /tmp/node-one
/tmp
$ cp /tmp/node-one /var/tmp/node-one
$ node /var/tmp/node-one
/var/tmp
$ cp /tmp/node-one ~/node-one
$ node ~/node-one
/home/dave

And, when using require, given the following script in ~/two.js

require('/tmp/node-one');

When executed, we can see

$ node ~/two.js
/tmp

Even though ~/two.js resides in /home/dave, it prints /tmp becasue the script being sourced resides in /tmp.

Because of this behavior, it is very easy and elegant for node scripts to require one another without needing to know an absolute path ahead of time. By using only relative require statements, all paths will be made relative to __dirname implicitly.

Problems

This behavior however, relies on very specific situations for the script to be executed. Imagine the case where node does not know where the JavaScript bytes are coming from. For example, using the original node-one script above:

Read More...