Build Script

Prerequisites

Connect to IPMI interface of the server by going to $ip in a web browser.

-	If this is first startup of system, then configure static IP for IPMI,
	and admin user and password.

Initial Configuration

Download the bootonly ISO of the current release of FreeBSD from reputable sources. (12.2 as of 11/07/2020)
Enable cd/dvd instance in IPMI under Settings -> Media Redirection Settings -> VMedia Instance Settings
Open up HTML5 viewer for IPMI, and attach ISO to virtual CD rom.
Start server, and boot to UEFI CD rom.
Let FreeBSD boot to the install menu. Don’t exit out of the autoboot menu.
Press enter to start the installation process.
Proceed through installation wizard. Press space to select options.

1.	Keymap = United States of America. Space will not work here, just
	scroll down and hit enter.

2.	Hostname = `the-vault`

3.	Disable kernel-dbg system component, and add the ports tree.

4.	Hit Ok to configure network for installation.

5.	Select gigabit interface 0, or whatever is actually connected.

6.	Configure IPv4 and use DHCP for now. We will configure static
	networking later.

7.	Confirm resolver config. (Defaults picked up from DHCP should be good
	to go.)

8.	Select mirror

9.	Select Auto (UFS) and then Entire Disk.

10.	Select GPT for partition scheme

11.	Review autogen partition layout, which should be good for our usages
	then hit finish and Commit.

12.	Wait for download, verification and extraction.

13.	Enter root password from password manager.

14.	Choose no at CMOS clock prompt. TODO: fix this.

15.	Choose timezone.

16.	Skip time and date setup if it is correct.

17.	Enable `local_unbound`, `sshd`, `ntpdate`, `ntpd`, `powerd` and `dumpdev` services
	to be started at boot

18.	Enable all security options except `disable_syslogd`, and `secure_console`.

19.	Add new admin user.

	1.	Username = `toxicsauce`
	2.	Full Name = `toxicsauce`
	3.	UID = (hit enter to accept default)
	4.	Login group = (hit enter to accept default)
	5.	Invite User to other groups = wheel
	6.	Login class = (hit enter to accept default)
	7.	Shell = sh (default)
	8.	home directory = (hit enter to accept default)
	9.	home directory permissions = (hit enter to accept defaults)
	10.	password auth = yes
	11.	empty password = no
	12.	random password = no
	13.	Enter password from password manager
	14.	lock account = no
	15.	check options and type in yes to confirm.
	16.	add additional users = no.

20.	hit exit, and no to exit installer.

21. Hit reboot.

22.	Stop CD media after system has rebooted.

Log in as the new user you created
Become root with su -. You will need root’s password.
Update system and install software

```sh
freebsd-update fetch
freebsd-update install

pkg update
# answer yes to install pkg system
pkg upgrade

pkg install nano sudo ipmitool openipmi
```

Configure sudo. visudo and uncomment line %wheel ALL=(ALL) ALL
Configure static IP addressing. First, run ifconfig to see what interface is actually connected, then add the following lines to /etc/rc.conf and modify existing lines so they read as follows.

```conf
ifconfig_igb0="inet $ipaddress netmask $netmask"
defaultrouter="$gateway"
```

Also, add the following line to `/etc/resolv.conf`

```conf
nameserver $gateway
```

Setup system logging and disable sendmail:

1.	Edit `/etc/rc.conf` and add the following lines to disable sendmail and enable syslog.

	```conf
	# disable sendmail and enable syslog
	syslogd_enable="YES"
	sendmail_enable="NO"
	sendmail_submit_enable="NO"
	sendmail_outbound_enable="NO"
	sendmail_msp_queue_enable="NO"
	newsyslog_enable="YES" # make sure this doesn't break in future defaults
	# fix cron trying to send mail
	cron_flags="-m ''"
	```

2.	Create log files needed for next step:

	```sh
	sudo touch /var/log/console.log
	sudo chmod 600 /var/log/console.log
	sudo touch /var/log/all.log
	sudo chmod 600 /var/log/all.log
	```

3.	Edit `/etc/syslog.conf` and uncomment the lines for `console.log` and
	`all.log`. TODO: setup remote logging.

4.	Check `/etc/newsyslog.conf` to confirm log rotation is setup correctly.

5.	Edit `/etc/periodic.conf` and add the following to the beginning of the
	file to disable sending root email and instead log to syslog. (This
	will override the defaults in `/etc/defaults/periodic.conf`)

	```conf
	# redirect periodic output to syslog files
	daily_output="/var/log/daily.log" # user or /file
	weekly_output="/var/log/weekly.log" # user or /file
	monthly_output="/var/log/monthly.log" # user or /file
	# turn off sendmail periodic functions
	daily_clean_hoststat_enable="NO"
	daily_status_mail_rejects_enable="NO"
	daily_status_include_submit_mailq="NO"
	daily_submit_queuerun="NO"
	```

6.	Fix crontab logging by editing `/etc/crontab` and appending `2>&1 |
	/usr/bin/logger -t cron_xxx` and replacing xxx with whatever the cron
	command was doing. Do not need to do this for periodic and periodic
	snapshot sections.

Random sysadmin tweaks:

1.	Edit `/etc/periodic.conf` and include the following lines:

	```conf
	# never going to use locate on a fileserver
	weekly_locate_enable="NO"	#Disable weekly locate run.
	```

Set up ssh-agent
Create Projects directory tree in ~ as follows:

```bash
mkdir -p ~/projects/src/github.com/sww1235
```

This mirrors the structure of how golang wants to set things up.

Make symlink in ~ as follows:

```bash
ln -s ~/projects/src/github.com/sww1235 myprojects
```

Clone dotfiles repo from GitHub using ssh and install vim and bash files using install.sh script.
Clone projects from github into Projects tree as desired.

UPS Configuration

Install Network UPS Tools from packages. sudo pkg install nut
Edit /usr/local/etc/nut/upsmon.conf and add in the following lines:

```conf
MONITOR cyberpower@$ipaddress 1 upsmonSlave $password slave
```

Edit /etc/rc.conf and add the following line:

```conf
nut_upsmon_enable="YES"
```

Start nut_upsmon service with sudo service nut_upsmon start and make sure it doesn’t error out.
Reboot system to make sure everything comes up normally.
Run a shutdown test to make sure nut is working properly.

1.	Login to `rack-monitor` system and run `upsdrvctl -t shutdown`. This
	will show you what will happen during a shutdown without actually
	shutting the systems down.

2.	Now run the command `upsmon -c fsd`. This will simulate a power failure
	and send the shutdown signal to all connected machines before shutting
	down itself.

3.	This should have worked, so now you need to manually remove power and
	apply power from all machines connected to the UPS to get them to come
	back up.

ZFS Configuration

Also see the ZFS chapter of the FreeBSD handbook

The current vdev design is 6 identical disks running in a RAIDZ2 configuration. When the storage is expanded, a new group of 6 disks will be added to a new vdev and added to the zpool.

vdev = group of drives. You cannot add more drives to a vdev once it is created.

zpool = group of vdevs. Redundancy is given by vdev configuration, not by multiple vdevs. if 1 vdev fails completely, all data in zpool is lost.

Install smartmontools and e2fsprogs packages.
If the hard drives are new, Test hard drives with sudo badblocks -b 4096 -wsv /dev/da* > da*.log, repla, replacing * with the number for each HDD. This will take a while depending on the size of the hard drives. TODO: add in instructions to import an existing pool.
Add the following line to /etc/rc.conf, then start the service manually sudo service zfs start so you don’t have to reboot.

```conf
zfs_enable="YES"
```

Add the following lines to /boot/loader.conf if they are not already there and reboot. This enables unique and static labels per disk, similar to /dev/disk/by-id/ on linux.

```conf
geom_label_load="YES"
kern.geom.label.disk_ident.enable="1"
```

Now create and configure the zpool and vdevs as follows:

```sh
# replace /dev/diskid/... with the names of the disks you are using in the zpool
sudo zpool create the-vault raidz2 /dev/diskid/...
# add any additional 6 disk sets
sudo zpool add the-vault raidz2 /dev/diskid/...
```

Now create ZFS datasets as follows:

```tags
sudo zfs create -o compress=lz4 the-vault/backups
sudo zfs create -o compress=lz4 the-vault/storage
sudo zfs create -o compress=lz4 the-vault/media
sudo zfs create -o compress=lz4 the-vault/archive
```

Install freebsd-snapshot from pkg.
Edit /etc/periodic.conf and add the following lines:

```conf
daily_status_zfs
```

Edit /etc/crontab and include the following:

```crontab

# Perform hourly/daily/weekly maintenance (FreeBSD UFS2/ZFS snapshots only).
0       *       *       *       *       root    /usr/local/sbin/periodic-snapshot hourly
0       0       *       *       *       root    /usr/local/sbin/periodic-snapshot daily
0       0       *       *       0       root    /usr/local/sbin/periodic-snapshot weekly
```

Edit /etc/periodic.conf and configure according to the datasets. Current config as below.

```conf
daily_scrub_zfs_enable="YES"
snapshot_enable="YES"
# :4:7:0 = 4 weekly snapshots (1 per week, and storing the last 4,
# 1 per day and storing the last 7, and 0 per hour, storing the last 0.
snapshot_schedule="/the-vault/backups:4:7:0 /the-vault/storage:4:7:0 /the-vault/media:2:3:0 /the-vault/archive:2:3:0"
```

Backups Configuration

Heavily borrowed from https://blog.alt255.com/post/restic/

Install restic from pkgs.
Create /root/.restic/restic-password.sh which exports RESTIC_PASSWORD set to the password of your repository. This is in password manager
Create /root/.restic/b2-credentials.sh which exports B2_ACCOUNT_ID and B2_ACCOUNT_KEY. These are set up on backblaze’s website.
su to root and then run the following commands to initialize the repository.

```sh
. /root/.restic/b2-credentials.sh
. /root/.restic/restic-password.sh
# multiple repos can be in one bucket.
# need to set up each repo individually
# RESTIC_REPOSITORY=b2:bucketname:path/to/repo
# path/to/repo is the path under the root of the bucket.
# it does not affect which files will actually be backed up.

export RESTIC_REPOSITORY=b2:the-vault-remote:archive
restic init

export RESTIC_REPOSITORY=b2:the-vault-remote:backups
restic init

export RESTIC_REPOSITORY=b2:the-vault-remote:media
restic init

export RESTIC_REPOSITORY=b2:the-vault-remote:storage
restic init
```

Initialize the first backups of the repositories. If you do not do this now, make sure to exit your root shell session to purge the environment variables.

```sh
# make sure to only run initial backups when we have enough bandwidth left
export RESTIC_REPOSITORY=b2:the-vault-remote:archive
restic backup --exclude-caches --exclude-if-present '.nobackup' /the-vault/archive

export RESTIC_REPOSITORY=b2:the-vault-remote:backups
restic backup --exclude-caches --exclude-if-present '.nobackup' /the-vault/backups

export RESTIC_REPOSITORY=b2:the-vault-remote:media
restic backup --exclude-caches --exclude-if-present '.nobackup' /the-vault/media

export RESTIC_REPOSITORY=b2:the-vault-remote:storage
restic backup --exclude-caches --exclude-if-present '.nobackup' /the-vault/storage
```

Create /etc/periodic/daily/601.restic-backblaze-backups with the following contents, and chmod it to 710, so only root can read and wheel can execute if need be.

```sh
#!/bin/sh

# modified from script at <https://blog.alt255.com/post/restic/>

# i18n, some files have non ASCII characters
export LC_ALL=en_US.UTF-8

# load credentials
. /root/.restic/b2-credentials.sh
. /root/.restic/restic-password.sh

QUIET="--quiet"
if [ -n "${NOQUIET}" ] || [ -n ${VERBOSE}" ]; then
	QUIET=""
fi

[ -z "${QUIET}" ] && echo "Starting backup set: archive"
export RESTIC_REPOSITORY=b2:the-vault-remote:archive
restic backup ${QUIET} \
	--exclude-caches \
	--exclude-if-present '.no-backup' \
	/the-vault/archive

[ -z "${QUIET}" ] && echo "Starting backup set: backups"
export RESTIC_REPOSITORY=b2:the-vault-remote:backups
restic backup ${QUIET} \
	--exclude-caches \
	--exclude-if-present '.no-backup' \
	/the-vault/backups

[ -z "${QUIET}" ] && echo "Starting backup set: media"
export RESTIC_REPOSITORY=b2:the-vault-remote:media
restic backup ${QUIET} \
	--exclude-caches \
	--exclude-if-present '.no-backup' \
	/the-vault/media

[ -z "${QUIET}" ] && echo "Starting backup set: storage"
export RESTIC_REPOSITORY=b2:the-vault-remote:storage
restic backup ${QUIET} \
	--exclude-caches \
	--exclude-if-present '.no-backup' \
	/the-vault/storage
```

Create /etc/periodic/daily/600.restic-check with the following contents, and chmod it to 710, so only root can read and wheel cna execute if need be.

```sh
#!/bin/sh
export RESTIC_REPOSITORY=b2:the-vault-remote:archive
restic check

export RESTIC_REPOSITORY=b2:the-vault-remote:backups
restic check

export RESTIC_REPOSITORY=b2:the-vault-remote:media
restic check

export RESTIC_REPOSITORY=b2:the-vault-remote:storage
restic check
```

Configure snapshot policies and pruning. TODO

Vault File Server Build Script

Notes