In this god-only-knows how many part series, I will be walking you through the steps I took to build a completely automated home lab/network/setup/etc.

The idea behind this was that I was tired of rebuilding/patching/etc and just wanted everything to work.  I realize there is an irony in the idea that I was going to spend years building a way to save minutes, but honestly, it was fun and good practice for my day job.

The tools/technologies used are as follows:

Networking – UniFi switches and firewalls. This part may or may not be outside the scope of this, not sure yet. There is not a lot to the networking side so…

Storage – FreeNAS 11.3 running on a 24-bay SuperMicro server. Populated with 24x 10Tb HDD, 16x 480Gb SSD, 256Gb of ECC Ram, 2x 6Core Xeons, and 2x 10G network cards.

Compute – 2x VMWare ESXi 7 running on a DL380p. Populated with 3x 147G drives, 384Gb of ECC Ram, 2x 12Core Xeons, and 10G networking.

Orchestration – Hashicorp Terraform 0.12.25

Automation – Ansible 2.9, Chef Infra 13.2

VM OS – RHEL 7, Ubuntu 20.04

The questions I commonly get asked when I discuss this with people:

Q: Why VMWare? Why not Proxmox/docker?
A: Most enterprises I have worked in over the years run VMWare, and you can use it non-prod for $200/yr. Plus most integrations for VMWare already exist. See further down the list for a complete reason as to why not docker, but net/net I just don’t like it.

Q: Why split your storage and compute? You know you can run FreeNAS under vmware right?
A: Yes, I am aware that you can do that. No, I have no desire to, nor should you. This is one of those things you see promoted a lot, and 100% of the time, is the wrong solution. You have introduced multiple single points of failure into your setup and a failure in either the storage or compute and your setup is 100% broken. Additionally, I use vMotion which requires shared storage, which is not possible/pointless if your running FreeNAS under vmware.

Q: Why not use vSAN for storage?
A: I have never set it up, and on a quick reading it seemed really hard. If you want to use that more power to you. I also use several NFS shares for backups in this, so if your following along at home, just spin up a VM on it to provide the same level of functionality.

Q: Why not chef Automate instead of Terraform + Chef Infra?
A: Have you seen the price on that? My complete software budget for this project was $200 and that was for the VMUG subscription.

Q: How can you say your budget was $200 when you have thousands in hardware?
A: Im a massive hypocrite? No, the actual answer is a lot more complicated. The compute nodes and the memory for the storage node was free, as they were being disposed of at my workplace. A friend no-longer needed the case for the storage so I got it for $50. The roughly $2500 in drives I had purchased for something completely different, and am using as the other project only used half the storage and I did not want to leave 80T idle. Again, if your playing the home version, feel free to use whatever you have on hand, this is just what IM using.

Q: Why the mix of ansible and chef?
A: I fundamentally dislike ansible. I need to use it in my day job, and find myself butting up against “limitation” in it constantly. I fully admit/recognize that these limitations are due to how I am trying to approach problems, but, it is what it is. That said, as of writing this, terraform does not have a chef-zero client, and the dependencies of getting all my cookbooks bundled/on a server to do a script based run is hella annoying. Ansible is great for getting the initial chef server stood up, after which everything shifts over to using it.

Q: You realize you have just re-invented docker in the least efficient way possible right?
A: And its user-unfriendly too! Yes. I realize that. I also don’t care, as this was done as a “can I do this?” type situation, and much like ansible, I have a deep dislike for docker. That is a diatribe in its own right, but it really boils down to a few keys facts.
  – I don’t fully understand it, and therefor dislike it.
  – I really don’t like the behavior it acclimates users to (running random things from the web)
  – Back when I started down this road, it did not have good resource constraints, and one misbehaving container would strangle everything else on the host. I am told this has since been fixed, but the bad taste doesn’t leave my mind so easily.

Q: Why RHEL? Why RHEL and Ubuntu? Why RHEL 7?
A: RHEL because its usable for free at home with a developer subscription and I use it in my professional life, so there is a comfort there. I know CentOS is basically the same thing, but ::shrug:: Just what I chose. If you want to use CentOS feel free, everything will still work. I use RHEL7 over RHEL8 due to the fact that as of writing this one of the core cookbooks I use (firewall) does not work on RHEL8. When it does I will update the template to be RHEL8 and just rebuild everything. Ubuntu was added to the mix as certain pieces of software (pi-hole, ruby > 2.${something}) do not play nicely with RHEL.

Q: Should I run any of this is production?
A: 100% no. You shouldn’t even do this at home. It’s provided for entertainment purposes only, I take no responsibility if it deletes you data, revives dead ancestors and violates them, burns down your house etc.

Q: Are you really running this at home?
A: The view in my living room.


Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>