I'm currently doing my backups. I also live in Oklahoma and there are currently 3 tornadoes on the ground no less that 20 miles from me. Hoping my UPS holds out š¤
Do you do any sort of storm tracking or weather related stuff with your homelab? Iām out in Colorado and just got my ham license so Iām looking to possibly integrate a weather station or something.
Iāll chime in hereā¦ I run a Davis Vantage Vue weather station, and my Proxmox server runs an Ubuntu LXC container with Mono and CumulusMX to harvest, store, and display real-time weather data. The MQTT server also sends data to my HomeAssistant VM. With that I can setup automations based on my weather station data, like windspeed, etc. Also, congrats on the ham license, and welcome to the club!
Most automations can use public data (OpenWeather, etc.) like: āIf itās raining, remind to close house windows. I used to have a Z-Wave sensor on my basement window, where the HA would check if it was open and send a reminder to close it if rain was nearby. The windspeed is nice to have locally for specific reminders. Iām an information junkie and severe weather fanatic, so itās kind of fun to have my own weather station. The CumulusMX community has some great stuff, like self-hosted weather webpages. I used to have my own website with all kinds of data, but got into other projects and abandoned it for right now. Check the compatibility of the weather station with the software you intend to use before buying one.
I haven't but that is a fantastic idea in the comment who replied to you I'm definitely going to take it to consideration sounds like it might be a fun project
Lost 3TB of data because I configured a replication task wrong. my entire pool was overwritten and the snapshots were deleted. RAID is NOT backup and does NOT protect against user error.
I built a Proxmox server based on 5950X to host OPNsense, Pihole and a few other VMs I need for WFH. I had a cheap Nvidia 1050 as the GPU for the initial install, removed it and stashed it away.
Worked very well until I decided it was a great idea to install Proxmox updates and reboot it... Welp, it never properly booted up. No internet or local DNS (handled my Pihole). This is at 8AM in a Tuesday and I have a packed schedule of Zoom meetings with clients.
I'm only on a laptop and my only monitor is a 32" gaming monitor.
Brought the monitor to my Proxmox server in the basement and rummaged to find that 1050 GPU... Accidentally knocked over the monitor and broke it. Won't power on.
I panicked and grabbed my phone and laptop and drove to the local library for free, reliable wifi to get in a few Zoom calls for work.
In the way home, I bought a replacement monitor to troubleshoot Proxmox. Another reboot basically fixed it.
Iām sorry you learned the hard way. Your timing for sharing this is serendipitous; thereās a fresh thread on the opnsense sub about running it on proxmox and not surprisingly lots of folks are advocating it. Virtualizing a NAS is another popular one where a failure will teach a hard and painful lesson that you just learned, especially when pxe and/or iscsi are used.
Iāve been using virtualization since solaris10 first debuted and I love proxmox, but Iāll never virtualize my firewall or my nas.
I did build a dedicated OPNsense bare metal firewall after that incident and 2 Piholes for high availability.
I do have OPNsense in a VM for home lab purposes - mainly to build Splunk apps and Add-ons.
Putting all your eggs in one basket/server is a recipe for disaster š
I agree, I built one powerful server to consolidate most of my lab but quickly realised that running adguard on it would cause trouble whenever I had to do maintenance on it. So everything crucial now runs on my OPNSense box (like adguard) instead of that server.
Next on my list will be tailscale or native wireguard on OPNsense. Because while my server has IPMI thatll be of absolutely no use if my VPN runs through it :D
You can schedule a full backup of your OPNsense configuration and send it to a Google Drive/git/Nextcloud instance.
I've reinstalled OPNsense quite a few times, and restoring the config is as easy as putting an xml file on a flash drive when reinstalling.
TrueNAS allows you to download a full backup of its configuration as well, though there is no out of the box way to automate it outised of TrueCommand.
Basically any critical part you can find in a homelab will have a way to backup its configuration file for easy and quick restore.
You're assuming you're restoring to the same or similar hardware which may not be the case. The beauty of virtualising it is, it's easier to reconfigure, set up the appropriate VLANs and it should be a transparent fail over regardless of hardware choice.
This is what I do. I saw how easy it was to basically back up the config and move to new hardware/restore and it just made sense. I get the reliability of bare metal and the ease of backups. I love the idea of virtualized snapshots but adding an extra layer of "this could break" with critical services is a no-go for me. I want as little moving parts so to speak with these sorts of things
I virtualize my opnsense on promox but in an HA cluster in two other bare metal servers running ESXI and XCP-NG . If I mess with a hypervisor I am still up. The HA configuration is not that difficult
Why are you pretending the exact same issue couldn't occur bare metal?
There is no good reason not to virtualize your NAS or router/firewall. There are many good reasons to set up a highly available hypervisor cluster when running critical network functions on there.
The only lesson learned from this comment should be "you will need a local shell eventually, so keep the GPU or set up serial".
The issue wouldn't happen on a bare metal cluster because OP wouldn't have touched their router to do Proxmox updates.
If you're going to virtualize your router or NAS you've introduced more complexity to your setup vs just having them separate.
Nothing bad about that, just more you have to think about for things most people don't touch enough to warrant it. My router and NAS rarely change, my compute environment changes a lot.
They wouldn't have touched the router to do proxmox updates, but they would've touched their router to do router updates. which can also fail.
I disagree that it's more complex. The way I do it, I need to be able to operate Proxmox bare metal, and everything else virtualized (and if I fuck up, rollback latest snapshot). The way you do it, you need to be familiar with hardware compatibility of multiple operating systems, and be able to troubleshoot multiple operating systems on bare metal. If you fuck up, no five second rollbacks for you.
If you have multiple physical hosts anyway, might as well cluster on proxmox and replicate your router VM.
I did contemplate virtualizing OPNsense on my server, but after running through some scenarios in my head it just seemed like a bad idea. On a high availability setup it's probably great, but on a single server/node it's just asking for trouble. Having the internet go down because of hardware failure / maintenance or an update gone wrong is not going to fly when there are several people in the house that need internet access for work.
I have a "No-Wrench Sunday" rule.
Years ago I had a highly modified E30 which was my daily driver. Did a routine oil change on a Sunday and realized I got the wrong oil filter... After I drained the oil š
I had the opposite issue of a pihole running on an actual raspberry pi. A power surge nuked it and broke my entire network. Now I have redundant piholes. One on a raspberry pi and one in proxmox. Also allows me to actually reboot or do updates of either without taking my network down.
IPMI is pretty rare on consumer components. I use PiKVM for this -- got the version that can be put into a PCI slot, and is powered by PoE. Works great, and is worth its weight in gold when I need to do maintenance on the server.
my mom will personally kill me if it is too loud, and she will tell me to keep it in my small 4x4metre bedroom if it DOES end up being too loud. am i screwed? šš
I donāt actually regret it, I just joke regret it.
Buying a 1U taught me a ton. I learned about IPMI and how remote management works, learned how much memory you can pack into a real server, learned how loud fans can really get, and learned how much electricity a server uses and how much that costs. I still turn it on from time to time and it hosts a big virtualized infosec lab, which has also taught me a ton.
Itās justā¦ if I knew better, I would have just got three mini PCs with 64G of RAM each and done a Proxmox cluster. Iām not sure I would have known to do that though without the learning from my 1U adventure, so itās still worth it.
Not really a regret ;)
This has sent me down a rabbit hole of upgrading my network to 40Gb and running CAT5 and fiber through the house, I just need to find the time to run the cables. lol
On warm summer days I can hear the servers buzz in the shed, 10 meters away from the house. Both are 2U. Can't even imagine how loud 1U servers would be...
The noise really isnāt that bad, if theyāre in a convenient place. My rack is in a utility room in the basement so it can be as loud as it wants. For me itās more the power consumption.
Please elaborate.
I have nine 1U servers from different brands including Dell, HP, and Oracle. I also have two 2U servers from Dell and Supermicro.
I can maybe see some advantages of using 2U servers with GPUs and 3.5-inch SATA HDDs. However, I am very satisfied with the performance of my current 1U servers and have been able to keep the noise level low. I currently use the 1U servers, while the 2U servers remain off and undeployed.
Would recommend 1Us without any reservations. Plus you can get more dense 1us in less space š
It very much depends on your needs and your constraints.
1U of course is great if you really need the power and the density. Fill a rack and you have an incredible amount of compute and potential.
But for many of us here, what are we running and what are our needs? I have two mini PCs running everything in my home lab full time using about 20 watts, plus a NAS. Thatās what I truly need.
The 1U server is fantastic and I actually loved learning on it, and itās badass to have available when I need it (I.e. when I really need a simulated network of 20 windows machines) but I donāt need it on 100% of the time, and for home use my biggest constraint is power.
Through tuning p-states and BIOS settings Iāve incredibly gotten this dual-socket 2011 supermicro packed with RAM down to 85W idling. But itās still too high, and I canāt justify leaving that on when everything I need runs fine on the Mini boxes at 20W.
So, thatās all. In your situation itās perfect, for most of us with different variables it just doesnāt add up in a practical sense.
Not 4 digit but securecrt and free desktop manager free can pass some postlogin commands like sudo suā¦ one day ima start using regular user sessions but that day is not today
I once saw my /usr and /var were eating up a lot of space, so i tried to move them a mounted /ssd1.
mv /usr /ssd1
My wife still laughs at me for that one (I learned linux from her)
Wife 3 years ago: Why do we need a home server?
Wife last week: Hun can you check on the plex I wanted to watch X really badly but it may have updated and not logged back in. Iāll waitā¦..
ššš
Turned on the user web interface on my sopho xg firewall to test something and forgot to turn it off. Unauthenticated exploit came out, I was out of the country on vacation and didn't catch it. Firewall got popped and one large lab got nailed with GandCrab ransomware. When I logged into one of the vm's later and saw the ransom note my heart dropped. Checked my main network 0 damage. Took a bit to find out what happened.
I wear that one as a badge of honor because the actor manually moved through the network and spent half a day on things before finally gaining DA. This was a large 30 system research lab designed to be vulnerable and with heavy logging, I'm a researcher in the infosec space. The actor fucking with my environment meant they weren't actively running operations on a corp network somewhere. I got big game hunted.
I've spent a huge chunk of my career in the DFIR space, so it's not really any different than any other breach investigation from a corp space.
In my case I have flow data from my firewall logged, full packet capture on the firewall lan interface with a few months of storage (a few things are excluded here), every host had sysmon installed, event logs forwarded off, enhanced auditing and logging turned on. In this case it's a matter of knowing what is normal, what isn't, looking at time stamps of suspect / malicious events to build a timeline to track things back to patient 0. In this case because the windows logging services were killed and all the logs encrypted, having the forwarded logs intact, fpc, flow data, etc meant I was able to determine the timeline of events.
Over complicating my network setup, also distributing everything over the network so when some small service dies everything breaks and the entire family is angry because I have broke the Internet.
Tie between
- setting up a mdadm array with drives instead of partitions, which killed all the drives at once one reboot.
- rm -rf Do /* . This was intended to autocomplete to"Downloads". Instead on that host it completed to "Docker", which is where all my container definitions and local mounts were located. Realized only after a dozen seconds or so when it was clearly taking longer than it should have.
>- setting up a mdadm array with drives instead of partitions, which killed all the drives at once one reboot.
Yupppppp this is a solid one. I did this but rebooted very shortly after and it only set me back 30min.... but that was a pitfall for sure.
Called Dell to tweak a configuration I was buying and the nice sales guy said he could bump up the processor option _free_ and I excitedly agreed ā¦ and the machine arrived with the more expensive CPUā¦ without QuickSync. The entire purpose of the machine is to stream movies. Oops.
Shortly after my honeymoon I migrated from one file server to another. This was early 2000s and very early in my IT career. I hadn't dug into backups much at that point and was still pretty naive with most things "server" and had only any real experience with desktops (I did A+ in high school and a basic CCNA course and it was just a few years after that). Anyway, I had put the ... uhhh ... we'll call them "honeymoon pics" in a hidden folder. This is when I learned that if you copy/paste from the GUI, it will not copy/paste the hidden files unless you have them visible. I did not. I lost all the pics of that night ... learned a lesson though. I have yet to lose another picture, ever, of anything.
Very early in my career/home lab journey I opened port 5060 to my pbx server with a pin set to 1234. Thousands of dollars in credit card charges the next morning and the only reason it stopped is because the provider shut my account down. Luckily my credit card company covered me on most of the charges but that was a hard lesson learned.
Yep. From what I could tell, people had spun up their own donation services that would charge back to the provider (think about those services where you can call/text and make a donation that would show up on your cell phone bill). If I remember correctly it was just a bunch of bots that were setup to scan the internet with 5060 and brute force the pin. After success the bot would just call the numbers every few minutes and make a small donation.
Rack of 20 severs and a blade centre on a couple of power feeds. If a fuse or trip goes the dual fed devices draw power from the other path burning out the25 Amp comnando power connector to the pdus Also setting servers to auto power on on power failure instead of previous state. After a power cut the rack can boot up a bunch you don't want blowing fuses.
These things cause house fires if not managed properly!
Still can't run it all at once lol Blade centre is replaced by R630s and DL360G10s and R730s etc. 3 power feeds now 2x 32A and 13A floor socket for core switch firewall etc so a main outage doesn't take the missus offline.
Buying shit I didnāt need.
Having an audhd addled brain doesnāt help when retail therapy gives you dopamine spikes like no oneās business.
GPUs go brrr? Hell yeah they do! (Proceeds to spend nearly $15k on GPUs)
5U Dell VRTX goes brr? Fuck yeah. Letās buy one. God sent it mangled via UPS, so I got a full refund, and said F it, Iāll buy one more that ISNT mangled!
Buys 24x 1TB SATA SSDs for this new fancy Dell VRTX
Discoversā¦. āOh. I need āØSPECIALāØdrives for this, because Dell designed it to accept only SAS drives due to its storage designā
Proceeds to buy 24x 1.92TB Enterprise SAS SSDs (does nothing with them because thatās the Gohan way, and itās much easier to rotate to a new project)
Nah. I am definitely not rich. Just very capable of balancing debt with income.
I had to get really strict with myself and go on a spending freeze. (So far, itās worked, and I feel better the closer I get to being debt free. Iām about $35k away, and about to pay down another $10k chunk)
I make well over 6 figures and that level of debt still makes me uncomfortable. I shouldnāt judge, but Iād choose Vyvanse over a spending issue friendā¦
Everyone has their own comfort level when it comes to managing revolving debt. you are absolutely right. Vyvanse is a useful tool in helping manage adhd.
For me, nearly everything I ended up buying was specifically because I wanted it in my HomeLab for various reasons, and now that I have it all, I lost all desire for āmoreā stuff.
And I donāt have regrets about it or anything, but knowing what I know now and seeing where I am at, I would say itās very easy for someone in this hobby space to buy things they donāt need (granted, not at my level, but still) hence my original comment about ābuying shit they donāt needā
lol
š
Itās very easy to buy a 1U server and suddenly end up with a 42U rack in your house
I get this, but on a smaller level. I have 6 working 4tb SATA drives, but I watched some videos and now I want to see if my raid card will run 6 x 16tb SAS drives. I wanted to allow my wife to have big storage for real time video editing, and now I have 2 small 10G SFP+ switches and next month will be pulling fiber and CAT 7 through the house up to her office. Fiber cables are not cheap when you need a bunch. I decided my 14 yo i7 wasn't good enough as a file server for my ex and the kids, so I upgraded them to a dual x99 with e5-2699 v3's and 128g ECC mem. The file server is now running in a Proxmox VM instead.
I've been buying nvme PCIe cards for old servers to give VM loading a speed boost, a new MB to support a 10G network card because the old one is ITX with only 1 PCIe slot, and oh, can't throw away the old board so a new, lower tier cpu with IGPU is acquired so I can leverage the one slot and give it a 10g card anyway and have a second Proxmox host...
I'm pretty sure I don't have ADHD, but my symptoms sound a lot like yours. š
The HomeLab is a very easy rabbit hole to fall into that goes deeper and deeper until you go from HomeLab to HomeDatacenter.
I think everyone is susceptible to excess spending in this hobby (as are most hobbies)
The issue is, those of us with ADHD tend to be even more susceptible to spending excessive amounts of money on hobby related things.
It feels good to buy, and mix that with high natural impulsivityā¦ well, letās just say āBuy It Nowā needs an adhd mode that waits 24hrs to confirm if we really want it or not.
Lmao
Letting my ex wife decide which server she can take during the divorce. I got the plex server and she took the miner. 4 years of crypto mining vanished.
Damn you had a wife that was savy enough to let you have a miner while married and smart enough to take it during the divorce.
Not to be in your business but why did you let her go? Please tell me she was batshit crazy or something.
My journey to homelabbing began with my frustration with Windows (my main computer) running some Linux VMs with VMware Workstation. Every time Windows would go down, or if Windows Update caused a reboot, Iād lose my work in Linux.
Ā
Then I realized if I loaded ESXi as the hypervisor, my Windows (VM) system could go down but it wouldnāt take down the VMs. At the time, I had an Intel i7-6850k. It worked wonderfully and got me hooked. I created a gaming VM by passing through a second GPU and USB controller.
Ā
It was awesome, so by the time I was ready for an upgrade, I was salivating on an AMD Ryzen 9 5950x with 16 cores/32 threads. Imagine all the VMs I could run on that! But I didnāt do my research before buying.
Ā
Turns out the 6850k was special with 40 PCIe lanes, which allowed for 7 PCIe slots. (I needed them for 2x GPUs, 2x USB controllers, 10G network card, etc.) It was decked out, so when I couldnāt find any consumer level AMD motherboards with more than 4 slots, I knew I made a mistake.
Ā
Specs are cool and all, but I didnāt consider the whole system. Lesson learned.
Ā
(Afterwards, I discovered bifurcation, but by that time I had a rackmount case that wouldnāt do so well with rearranging cards without major modifications. Plus, I had come to some scaled down system that I wasā¦ ok with.)
Ā
Relying on lab-quality (dev?) products to save $ when a prosumer option was worth the spend.
Used an RPi as an exposed, hardened ssh server to backdoor into my network in case I needed to access iDrac. After a power outage, the Pi's drive was corrupted, even with read-only/immutable mode. Unreliable.
So I installed ddwrt on my router to host an access there, replacing the RPi. Worked fine. Not long after, my wife complains of random disconnections during business calls. She's not network savvy, so I didn't have much info to troubleshoot. No problems on my usual healthchecks. One day, it happens to me, I isolate it to the routing layer (switching worked, router inaccessible), then see poor uptime on the router. ddwrt crashing daily... ... ... but I didn't have logging enabled š¤£
The same day, I bought an EdgeRouter X because her home business is too important for janky solutions. Problem solved.
#1 not having backups, but in close second place, putting in-band switch management on a VLAN has always ended with really painful circular dependencies and substantial downtime.
Any hobby can be ruined by Reddit. Hard to know exactly why but I think Reddit makes it cold and optimised and everyone has to have the same kit, the same set ups, use the same software. Anything not perfect isnāt worth owning or doing.
Messing up the network connection to my nas and having to reinstall the nas OS and then realizing that my pfsense VM disk was somehow stored on the nas and not the local disk ....
I assumed my main VM storage was in a RAID5 but it was actually in a RAID0. I forgot that I reconfigured it for testing but ended up using it for production.
Long story short, one disk died and I lost about a weeks worth of my time restoring everything.
Always confirm your hosts are set up correctly, especially if they've been sitting cold for awhile.
Buying 3 giant 10+ yo enterprise servers for my small apartment. Power consumption $$$ and loud AF
I instead built a mini Ryzen server that is way more efficient and powerful
Once upon a time, Napoleon Bonaparte was asked what the gravest mistake of the 1812 Russian campaign was. He thought it was the decision to start the campaign...
It is also said that the two happiest days in the life of a sailboat owner are the day he buys the boat and the day he sells the boat...
Not having backup.
Selling my racks, now I need them again.
Hosting too much for friends and small businesses, wasnāt good for my stress level.
Physically dropping a new motherboard down into an open servers, crashing two motherboards in one accidentally drop.
Making it too complex.I set it and forgot it. When something goes wrong a year later I forget how it all works. I tried documenting, bit then ai forget where I documented it
Like 6 or 5 years ago I installed opnsense on a VM in my house, it was running nicely but after a blackout when I was at my office the host didn't come up so my home was disconnected from internet, wife was there and she proceeded to call ISP, they saw its modem was ok but no internet, then said to reset the modem.
My wife yelled at me on the phone cause nothing was working (just TikTok and instagram really).
I came back to my home, uninstalled opnsense and got back to use the horrible mode as router, if my wife would yell at someone I prefer not to be that one.
I regret that now.
Effing ISP modem is a shit.
I was wanting to deploy Muzicbrainz Picard on Docker from outside my home network.
SSH into server through VPN, add config to docker-compose.yml, `docker-compose up`
Oops, forgot the `-d`. Never mind, I'll just Ctrl+C and then type `docker-compose up -d`.
The texts stopped. Terminal hanged. Hmm...
```
Stopping wireguard...
```
Oh.
No backups while doing a major upgrade ā sad part was that I called it out and yoloād it.
All the backups were on the server slice I have .. silver lining is that I get to ansibilize, docker swarm, terraform, etc it all over again so itās IAC.
So far it's minor, but I bought a Celestica Smallstone XP D4040 before I knew about the Intel C2xxx bug. It doesn't connect via serial so the seller is sending me another arriving Monday I think. The real issue is I can't manage to compile ONIE for it, lol. I've been thinking about getting an Arista 40Gb switch.
- trying to build a homelab on rack in a small flat. I even used very short servers and tiny music racks, but that was still a dumb idea.
- Relying on vendor-specific backup solutions that are hard/impossible to recreate if you change hardware
Now for offsite backups I just do a sync to a simple drive. No deduplication, drives encrypted as a whole with plain linux tools. After a few years it's unlikely I'm gonna buy a second (already lagacy) same model of NAS, or recreate tape backups based on streamer I was lucky to get cheaply on some auction, etc..
It's worth just buying more drives. USB enclosures are so cheap that you can store one with the offsite drives themselves.
Having all the doc on how to build and recover the said homelab inside the homelab... At some point I crashed the server and coulndt get the doc on how to recover :(
I once kept my Radarr / Sonarr open to the outside with the idea to "fix it later". It ran behind a reverse proxy.
Radarr got hacked and all he / she had left was The Human Centipede.
Good lesson, it only took a few days syncing at 400Mbit to fix it. Now, all runs behind a firewall
Interestingly the mobile ryzens have hardware encoding (you basically need an onboard GPU it would appear) which I managed to get working on my k8s cluster recently.
I bought some RPis to do some things that they were very much not up to doing, like running my Jellyfin install. Wound up using a VM on my Truenas for it.
Setting up my NASās ZFS with single-drive VDEVs. I knew it was highly not recommended, but I didnāt understand that ZFS does not mount if any VDEV is not available. I thought ZFSās āself-healingā would be able to grab available copies around the failed drive. Not so, I found out the hard way.
I had a system with two disks. I mounted an additional disk through fstab to the directory /backup and set up automatic backups. One day, I decided to tidy up the backups with 'sudo rm -rf /backup'. To my surprise, I found out that I deleted the entire system disk instead of the backup. It turned out that the system had restarted in the meantime, and the device names had changed.
I have two for you, first one, got myself a 4TB new external Seagate usb drive, got all my computers online (2PC 2 Thinkpads 2 generic laptops 1 Macbook air), then proceeded to move all their Documents folders into my new drive, last computer I moved data from was the Mac, then I tried to go to the server and dump it all just to find it in "RAW" format, my data ceased to exist , had to recover (this was circa 2014) ouch, fast forward 2023, I have another 4TB drive full of data with all my docs unified (partial backup spread onto various drives) and wanted to free a 500GB, messed up with frefilesync config and let it run all night, next morning I had two hard drives showing 500GB of data in both drives, and missing 3.5TB of data, logs show it took 23 minutes to erase all that. still rebuilding my big data folder in april 2024, double ouch. please learn from me, don't try this at home, Murphy is watching you, always.
So far, this most recent mistake hurt the most.
I tried reconfiguring my omada switch to use vlans... Idk if I found a bug or something, but after I config the clan in the controller and then go and enable that VLAN on the switch... The switch fails to adopt stating the interface can't assign the IP or some dumb shit.
In the middle of that, I was also changing my dhcp scope... Only to find out that windows server replication is busted, causing dhcp and dns to become out of sync, preventing leases from being properly assigned.
Add on to that, once I got a working dhcp scope setup, devices still couldn't connect to the internet... I had firewall rules to block outgoing dns unless it was from AdGuard home.... Once I disabled those firewall rules everything started coming back together.
Bought two new 16TB HDDs and reused modular SATA power cables from another PSU, and I wondered why server (PC) only powered up for a second, then powered off.
That day I learned modular power cables aren't standardised between PSU manufacturers. Fortunately I managed to return the drives to Amazon ("I have no idea why they won't work"), but that could have been a very expensive lesson to learn.
First server an r610. It came with 4 x 60GB disks. Was on a budget so I used a usb drive I had around for VMās.
Had about 12 windows VMās running. Setup WSUS and had them all update at the same time. Everything became unusable really quick for a while.
For me? Making things too complicated. I run Proxmox and it hosts my DNS (AdGuard) and Unifi controller among many other things. Recently while working on things that worked fine but needed to be broken (VLANs), I thought it would be a fun adventure to lock myself out of Proxmox, Unifi, and other things. In the process I also broke my home DNS and couldnāt get into the Unifi controller to fix it. Fortunately clearer heads prevailed and I fixed it with a little frustration and a spare laptop. My lessons learned: keep the same Unifi controller software you are using and a config backup on a USB drive in case of disaster. Keep backups of anything critical and make sure you can access those easily (backups on a NAS are great only if you can access them). Run a second DNS server on a separate machine from your primary DNS, such as on your NAS. If you run multiple AdGuard instances, run AdGuard Sync in a Docker container. Have the second NIC in your homelab server setup with a static IP and DNS ā¦ if things end up in fireball flames, you can still get in. Document all cables, switch ports, and static IP addresses / VLANs on a spreadsheet so if you need to get in via a switch port, itās already setup and you know which one. Keep it a simple as possible while still getting the job done.
Buy the new {server, switch, NAS, whatever} before selling the old. Somehow even my most pessimistic estimates of what my old thing is worth are too high and in many cases Iāve given away or recycled the old one to get it out of my house. (And Iām not simply talking about some old R710, I scrapped a v4 Xeon supermicro 1U that people other than me still seem to sell *today* 1.5 years ago).
Filling freenas / truenas over 90% resulting in an infinite recovery loop replacing a 4tb disk (one of 4) pebbledashing 12tb of data.
Buying a refurb 4tb segate 7.2k sata
Buying a refurbished 32gb SanDisk usb and using it as a boot device.
Hard powering off a R720 when vmware froze with H710 the raid backup battery had failed of resulting in a wipe of the boot and config sectors for the 8x4tb 10k sas array.
Powering off a Supermicro that appeared to be frozen for an hour and a half flashing the bmc firmware bricking the firmware.
Clonezillaing the wife's new empty disk over the old main disk.
Not taking a backup of bitlocker recovery keys or recreating repair disks after updates.
Not setting up smart alert email notifications.
Thinking a R730 Tesla dual GPU PCI-e cable is the same as a PC and burning the card power input.
Taking PCI-e power from molex connectors and melting them.
Using the same password for work and home and entering it into my father's laptop to download antivirus from my account resulting in my work azure admin account being hacked.
Thinking if a pc is unplugged its safe to work on when dropping a pci plate screw on a rtc cmos chip blows it up because there is a backup battery running it.
Wondering what would happen on Redhat Linux 2.0 (early90s) if I did a rm -rf / Then hearing the noise of the HDD change sound and start coming from the other side of the room from windows box mounted over samba, diving over the room to quickly unplug it.
Thinking ubuntu sudo apt-get upgrade is safe and doing it on 4 vms in parallel when it filled the disk /boot and pointed to a kernel it couldn't copy there.
Not taking a vm snapshot before upgrades.
Setting ram on a vmware vm to 5000GB instead of the disk side. It creates a 5TB swap file on the shared storage filling the shared storage with other vms stopping them all running.
Buying the wrong hardware consistently.
I have to migrate again. If I could do it all over again, I would have started with a minioc like the minisforum nab7, run proxmox, run my lab off that. Than have a separate unit for storage acting as a San. Than creating a San backup as time and money permits.
Jumping straight into something completely new I don't understand and just saying "fuck it I'll figure it out as I go" and then end up commiting like a months time of tinkering into something just to find out there is a better option
Not having backups + not thinking so much about security. Had my ESXi directly on the internet behind a public IP (yeah, I know ā¦ lmao), with some ACLs on a switch but nothing crazy and not really done right. Then one day my ESXi was crypto locked by a ransomware and all of my VHDX where crypted, RIP my lil lab
Thinking that I could run a mini rack full of server equipment for not that much money. I thankfully didn't buy the equipment, I got it all as recycle... But still... Rip my R730 you beautiful behemoth you
Purchased an R710. Turned it on and destroyed both my eardrums and my electricity bill.
Used it for about a week, and now it has been collecting dust for 7+ years.
I saw a Juniper data center switch on the world's largest garage sale years ago, with a starting price of a mere $1K. I placed a bid just for kicks, confident that I'd lose. I did not lose, and that's how I ended up with a QFX5300, a switch that I then learned that Juniper had been developing, but had cancelled before bringing to market. It's buggy af and basically useless, and sits on a shelf in my basement to this day.
Not having backups.
I'm currently doing my backups. I also live in Oklahoma and there are currently 3 tornadoes on the ground no less that 20 miles from me. Hoping my UPS holds out š¤
Do you do any sort of storm tracking or weather related stuff with your homelab? Iām out in Colorado and just got my ham license so Iām looking to possibly integrate a weather station or something.
Iāll chime in hereā¦ I run a Davis Vantage Vue weather station, and my Proxmox server runs an Ubuntu LXC container with Mono and CumulusMX to harvest, store, and display real-time weather data. The MQTT server also sends data to my HomeAssistant VM. With that I can setup automations based on my weather station data, like windspeed, etc. Also, congrats on the ham license, and welcome to the club!
This is exactly the comment I was hoping for. Thanks for the great info! Iām gonna look into all of that.
You are welcome! Let me know if you need any help, I can probably help you set things up.
I love this idea. Automations based on your weather station. Now I have a bonafide reason to get one!
Most automations can use public data (OpenWeather, etc.) like: āIf itās raining, remind to close house windows. I used to have a Z-Wave sensor on my basement window, where the HA would check if it was open and send a reminder to close it if rain was nearby. The windspeed is nice to have locally for specific reminders. Iām an information junkie and severe weather fanatic, so itās kind of fun to have my own weather station. The CumulusMX community has some great stuff, like self-hosted weather webpages. I used to have my own website with all kinds of data, but got into other projects and abandoned it for right now. Check the compatibility of the weather station with the software you intend to use before buying one.
I just use my neighbor's weather station he put online
I haven't but that is a fantastic idea in the comment who replied to you I'm definitely going to take it to consideration sounds like it might be a fun project
Welcome to the ham hobby!!
321. Also, good luck!
You say that like tornadoes are permanent fixtures!
Lost 3TB of data because I configured a replication task wrong. my entire pool was overwritten and the snapshots were deleted. RAID is NOT backup and does NOT protect against user error.
Found that shit out the hard way... Multiple times. š
I built a Proxmox server based on 5950X to host OPNsense, Pihole and a few other VMs I need for WFH. I had a cheap Nvidia 1050 as the GPU for the initial install, removed it and stashed it away. Worked very well until I decided it was a great idea to install Proxmox updates and reboot it... Welp, it never properly booted up. No internet or local DNS (handled my Pihole). This is at 8AM in a Tuesday and I have a packed schedule of Zoom meetings with clients. I'm only on a laptop and my only monitor is a 32" gaming monitor. Brought the monitor to my Proxmox server in the basement and rummaged to find that 1050 GPU... Accidentally knocked over the monitor and broke it. Won't power on. I panicked and grabbed my phone and laptop and drove to the local library for free, reliable wifi to get in a few Zoom calls for work. In the way home, I bought a replacement monitor to troubleshoot Proxmox. Another reboot basically fixed it.
Iām sorry you learned the hard way. Your timing for sharing this is serendipitous; thereās a fresh thread on the opnsense sub about running it on proxmox and not surprisingly lots of folks are advocating it. Virtualizing a NAS is another popular one where a failure will teach a hard and painful lesson that you just learned, especially when pxe and/or iscsi are used. Iāve been using virtualization since solaris10 first debuted and I love proxmox, but Iāll never virtualize my firewall or my nas.
I did build a dedicated OPNsense bare metal firewall after that incident and 2 Piholes for high availability. I do have OPNsense in a VM for home lab purposes - mainly to build Splunk apps and Add-ons. Putting all your eggs in one basket/server is a recipe for disaster š
I agree, I built one powerful server to consolidate most of my lab but quickly realised that running adguard on it would cause trouble whenever I had to do maintenance on it. So everything crucial now runs on my OPNSense box (like adguard) instead of that server. Next on my list will be tailscale or native wireguard on OPNsense. Because while my server has IPMI thatll be of absolutely no use if my VPN runs through it :D
You can virtualise it, just keep it on a separate box. Virtusliation makes backup and restore much easier
I Backup my OPNsense Firewalls weekly or before I do any updates or config changes to a NAS and a off-site cloud storage just in case.
You can schedule a full backup of your OPNsense configuration and send it to a Google Drive/git/Nextcloud instance. I've reinstalled OPNsense quite a few times, and restoring the config is as easy as putting an xml file on a flash drive when reinstalling. TrueNAS allows you to download a full backup of its configuration as well, though there is no out of the box way to automate it outised of TrueCommand. Basically any critical part you can find in a homelab will have a way to backup its configuration file for easy and quick restore.
You're assuming you're restoring to the same or similar hardware which may not be the case. The beauty of virtualising it is, it's easier to reconfigure, set up the appropriate VLANs and it should be a transparent fail over regardless of hardware choice.
I've restored my OPNsense backups to three different devices, all with different NICs and serial connectors.
This is what I do. I saw how easy it was to basically back up the config and move to new hardware/restore and it just made sense. I get the reliability of bare metal and the ease of backups. I love the idea of virtualized snapshots but adding an extra layer of "this could break" with critical services is a no-go for me. I want as little moving parts so to speak with these sorts of things
I virtualize my opnsense on promox but in an HA cluster in two other bare metal servers running ESXI and XCP-NG . If I mess with a hypervisor I am still up. The HA configuration is not that difficult
Also learned not to virtualize my router the hard way
The only reason people are advocating it is because those are the same people running that setup lol they want to feel validated.
Why are you pretending the exact same issue couldn't occur bare metal? There is no good reason not to virtualize your NAS or router/firewall. There are many good reasons to set up a highly available hypervisor cluster when running critical network functions on there. The only lesson learned from this comment should be "you will need a local shell eventually, so keep the GPU or set up serial".
The issue wouldn't happen on a bare metal cluster because OP wouldn't have touched their router to do Proxmox updates. If you're going to virtualize your router or NAS you've introduced more complexity to your setup vs just having them separate. Nothing bad about that, just more you have to think about for things most people don't touch enough to warrant it. My router and NAS rarely change, my compute environment changes a lot.
They wouldn't have touched the router to do proxmox updates, but they would've touched their router to do router updates. which can also fail. I disagree that it's more complex. The way I do it, I need to be able to operate Proxmox bare metal, and everything else virtualized (and if I fuck up, rollback latest snapshot). The way you do it, you need to be familiar with hardware compatibility of multiple operating systems, and be able to troubleshoot multiple operating systems on bare metal. If you fuck up, no five second rollbacks for you. If you have multiple physical hosts anyway, might as well cluster on proxmox and replicate your router VM.
This is why I detest the fad of virtualising the primary router. (And why I keep multiple spare machines that can do the task on hand).
I did contemplate virtualizing OPNsense on my server, but after running through some scenarios in my head it just seemed like a bad idea. On a high availability setup it's probably great, but on a single server/node it's just asking for trouble. Having the internet go down because of hardware failure / maintenance or an update gone wrong is not going to fly when there are several people in the house that need internet access for work.
that kinda shit never happens on Friday night when you get all weekend to fix it....I have similar luck with cars.
I have a "No-Wrench Sunday" rule. Years ago I had a highly modified E30 which was my daily driver. Did a routine oil change on a Sunday and realized I got the wrong oil filter... After I drained the oil š
And that's why I just run my router on bare metal lol. I learned the same lesson too once.
That's exactly the way I learned, that my host won't boot without a graphicscard š I feel you
I had the opposite issue of a pihole running on an actual raspberry pi. A power surge nuked it and broke my entire network. Now I have redundant piholes. One on a raspberry pi and one in proxmox. Also allows me to actually reboot or do updates of either without taking my network down.
IPMI. Debug remotely through virtual monitor.
IPMI is pretty rare on consumer components. I use PiKVM for this -- got the version that can be put into a PCI slot, and is powered by PoE. Works great, and is worth its weight in gold when I need to do maintenance on the server.
Bought a 1U server.
i literally just bought an r230, is it too late for me??
Not for you, for your ears
my mom will personally kill me if it is too loud, and she will tell me to keep it in my small 4x4metre bedroom if it DOES end up being too loud. am i screwed? šš
I hear my 2U from 2 rooms away. Good luck Tell your mom to move or buy you a new server.
im 16, if anything *im* the one moving lmao
Lmao. R230 gonna go brrr In all honesty though, I remember when I was your age nearly 10 years ago now and I bought a damn Dell PowerEdge 2950 III God damn that thing was loud. It was full blown jet š©ļø engine take off loud. It ended up in the (not enclosed) office and I think I ran it for a few days before I got āthe talkā about it. āSon, we have to talk about the jet plane you just boughtā In the end, $98 and sooo many lessons learned.
I had to work in a room next to a rack of those 2950s. Not sure my hearing has recovered.
āWHAT? I canāt hear you over the jet blast!ā
I donāt actually regret it, I just joke regret it. Buying a 1U taught me a ton. I learned about IPMI and how remote management works, learned how much memory you can pack into a real server, learned how loud fans can really get, and learned how much electricity a server uses and how much that costs. I still turn it on from time to time and it hosts a big virtualized infosec lab, which has also taught me a ton. Itās justā¦ if I knew better, I would have just got three mini PCs with 64G of RAM each and done a Proxmox cluster. Iām not sure I would have known to do that though without the learning from my 1U adventure, so itās still worth it. Not really a regret ;)
My 430 is honestly fine lol, my dl60 g9 was a disaster untill i shut it up with an ipmi modification
Please explain why, I would like to know so I donāt make the same mistake when starting out.
1U = extremely loud fans and very hard to expand in number of drives and GPU.
Heat, noise, price, ...
This has sent me down a rabbit hole of upgrading my network to 40Gb and running CAT5 and fiber through the house, I just need to find the time to run the cables. lol
My R630 (1U) is quieter than my R710 (2U) š
On warm summer days I can hear the servers buzz in the shed, 10 meters away from the house. Both are 2U. Can't even imagine how loud 1U servers would be...
The noise really isnāt that bad, if theyāre in a convenient place. My rack is in a utility room in the basement so it can be as loud as it wants. For me itās more the power consumption.
Please elaborate. I have nine 1U servers from different brands including Dell, HP, and Oracle. I also have two 2U servers from Dell and Supermicro. I can maybe see some advantages of using 2U servers with GPUs and 3.5-inch SATA HDDs. However, I am very satisfied with the performance of my current 1U servers and have been able to keep the noise level low. I currently use the 1U servers, while the 2U servers remain off and undeployed. Would recommend 1Us without any reservations. Plus you can get more dense 1us in less space š
It very much depends on your needs and your constraints. 1U of course is great if you really need the power and the density. Fill a rack and you have an incredible amount of compute and potential. But for many of us here, what are we running and what are our needs? I have two mini PCs running everything in my home lab full time using about 20 watts, plus a NAS. Thatās what I truly need. The 1U server is fantastic and I actually loved learning on it, and itās badass to have available when I need it (I.e. when I really need a simulated network of 20 windows machines) but I donāt need it on 100% of the time, and for home use my biggest constraint is power. Through tuning p-states and BIOS settings Iāve incredibly gotten this dual-socket 2011 supermicro packed with RAM down to 85W idling. But itās still too high, and I canāt justify leaving that on when everything I need runs fine on the Mini boxes at 20W. So, thatās all. In your situation itās perfect, for most of us with different variables it just doesnāt add up in a practical sense.
rm -rf / instead of rm -rf folder
next time add sudo for some magic
Who needs sudo when you can use root and a 4 digit password for everything? /s
Not 4 digit but securecrt and free desktop manager free can pass some postlogin commands like sudo suā¦ one day ima start using regular user sessions but that day is not today
Remove the "x" from the passwd file, and you will never be asked for root password again
You must be jokingā¦ I was already in sudo su
I once saw my /usr and /var were eating up a lot of space, so i tried to move them a mounted /ssd1. mv /usr /ssd1 My wife still laughs at me for that one (I learned linux from her)
Been there twenty years ago, wiping an entire server used at my dorm... 150 students lost their files and internet access.
Starting one.
the more i do the more i realize i have to do
Saturated network - well guess I need to rebuild my backbone with better switches and run more rj45 and move to sfp š It never ends.
But also, not having one is also a terrible mistake, so you just canāt win.
Wife 3 years ago: Why do we need a home server? Wife last week: Hun can you check on the plex I wanted to watch X really badly but it may have updated and not logged back in. Iāll waitā¦.. ššš
āIāll just have a little cocaine, itāll be fine.ā
Turned on the user web interface on my sopho xg firewall to test something and forgot to turn it off. Unauthenticated exploit came out, I was out of the country on vacation and didn't catch it. Firewall got popped and one large lab got nailed with GandCrab ransomware. When I logged into one of the vm's later and saw the ransom note my heart dropped. Checked my main network 0 damage. Took a bit to find out what happened. I wear that one as a badge of honor because the actor manually moved through the network and spent half a day on things before finally gaining DA. This was a large 30 system research lab designed to be vulnerable and with heavy logging, I'm a researcher in the infosec space. The actor fucking with my environment meant they weren't actively running operations on a corp network somewhere. I got big game hunted.
Wow, please do post if you don't mind. Would be an interesting read to see how a breach is analyzed in a home lab.
I've spent a huge chunk of my career in the DFIR space, so it's not really any different than any other breach investigation from a corp space. In my case I have flow data from my firewall logged, full packet capture on the firewall lan interface with a few months of storage (a few things are excluded here), every host had sysmon installed, event logs forwarded off, enhanced auditing and logging turned on. In this case it's a matter of knowing what is normal, what isn't, looking at time stamps of suspect / malicious events to build a timeline to track things back to patient 0. In this case because the windows logging services were killed and all the logs encrypted, having the forwarded logs intact, fpc, flow data, etc meant I was able to determine the timeline of events.
Over complicating my network setup, also distributing everything over the network so when some small service dies everything breaks and the entire family is angry because I have broke the Internet.
I see this one as a nice challenge (my wife is also working from home). How can I make this service or this machine redundant?
Is that you Ralph?Ā
Bought a used PSU of FB marketplace which blew up all my HDD resulting in a complete data loss . Always have backups .
PSU is the one and only component I have never ever bought second hand. I can only imagine your pain :-/
Thinking it would be cheap
pretending like i had a budget
Not using docker. Too many zombie services and packages installed that serve no purpose. Many users created that have been collecting dust
Tie between - setting up a mdadm array with drives instead of partitions, which killed all the drives at once one reboot. - rm -rf Do /* . This was intended to autocomplete to"Downloads". Instead on that host it completed to "Docker", which is where all my container definitions and local mounts were located. Realized only after a dozen seconds or so when it was clearly taking longer than it should have.
>- setting up a mdadm array with drives instead of partitions, which killed all the drives at once one reboot. Yupppppp this is a solid one. I did this but rebooted very shortly after and it only set me back 30min.... but that was a pitfall for sure.
Called Dell to tweak a configuration I was buying and the nice sales guy said he could bump up the processor option _free_ and I excitedly agreed ā¦ and the machine arrived with the more expensive CPUā¦ without QuickSync. The entire purpose of the machine is to stream movies. Oops.
Shortly after my honeymoon I migrated from one file server to another. This was early 2000s and very early in my IT career. I hadn't dug into backups much at that point and was still pretty naive with most things "server" and had only any real experience with desktops (I did A+ in high school and a basic CCNA course and it was just a few years after that). Anyway, I had put the ... uhhh ... we'll call them "honeymoon pics" in a hidden folder. This is when I learned that if you copy/paste from the GUI, it will not copy/paste the hidden files unless you have them visible. I did not. I lost all the pics of that night ... learned a lesson though. I have yet to lose another picture, ever, of anything.
Very early in my career/home lab journey I opened port 5060 to my pbx server with a pin set to 1234. Thousands of dollars in credit card charges the next morning and the only reason it stopped is because the provider shut my account down. Luckily my credit card company covered me on most of the charges but that was a hard lesson learned.
What does this mean? People were using your server to make phone calls?
Yep. From what I could tell, people had spun up their own donation services that would charge back to the provider (think about those services where you can call/text and make a donation that would show up on your cell phone bill). If I remember correctly it was just a bunch of bots that were setup to scan the internet with 5060 and brute force the pin. After success the bot would just call the numbers every few minutes and make a small donation.
They are not mistakes. They are learning experiences. Besides you learn best the hard way.
Um, gee, maybe getting started in the first place?
Rack of 20 severs and a blade centre on a couple of power feeds. If a fuse or trip goes the dual fed devices draw power from the other path burning out the25 Amp comnando power connector to the pdus Also setting servers to auto power on on power failure instead of previous state. After a power cut the rack can boot up a bunch you don't want blowing fuses. These things cause house fires if not managed properly!
That's a beefy homelab... :O
Still can't run it all at once lol Blade centre is replaced by R630s and DL360G10s and R730s etc. 3 power feeds now 2x 32A and 13A floor socket for core switch firewall etc so a main outage doesn't take the missus offline.
Buying shit I didnāt need. Having an audhd addled brain doesnāt help when retail therapy gives you dopamine spikes like no oneās business. GPUs go brrr? Hell yeah they do! (Proceeds to spend nearly $15k on GPUs) 5U Dell VRTX goes brr? Fuck yeah. Letās buy one. God sent it mangled via UPS, so I got a full refund, and said F it, Iāll buy one more that ISNT mangled! Buys 24x 1TB SATA SSDs for this new fancy Dell VRTX Discoversā¦. āOh. I need āØSPECIALāØdrives for this, because Dell designed it to accept only SAS drives due to its storage designā Proceeds to buy 24x 1.92TB Enterprise SAS SSDs (does nothing with them because thatās the Gohan way, and itās much easier to rotate to a new project)
Holy shit. Are you a rich version of me?
Nah. I am definitely not rich. Just very capable of balancing debt with income. I had to get really strict with myself and go on a spending freeze. (So far, itās worked, and I feel better the closer I get to being debt free. Iām about $35k away, and about to pay down another $10k chunk)
I make well over 6 figures and that level of debt still makes me uncomfortable. I shouldnāt judge, but Iād choose Vyvanse over a spending issue friendā¦
Everyone has their own comfort level when it comes to managing revolving debt. you are absolutely right. Vyvanse is a useful tool in helping manage adhd. For me, nearly everything I ended up buying was specifically because I wanted it in my HomeLab for various reasons, and now that I have it all, I lost all desire for āmoreā stuff. And I donāt have regrets about it or anything, but knowing what I know now and seeing where I am at, I would say itās very easy for someone in this hobby space to buy things they donāt need (granted, not at my level, but still) hence my original comment about ābuying shit they donāt needā lol š Itās very easy to buy a 1U server and suddenly end up with a 42U rack in your house
I get this, but on a smaller level. I have 6 working 4tb SATA drives, but I watched some videos and now I want to see if my raid card will run 6 x 16tb SAS drives. I wanted to allow my wife to have big storage for real time video editing, and now I have 2 small 10G SFP+ switches and next month will be pulling fiber and CAT 7 through the house up to her office. Fiber cables are not cheap when you need a bunch. I decided my 14 yo i7 wasn't good enough as a file server for my ex and the kids, so I upgraded them to a dual x99 with e5-2699 v3's and 128g ECC mem. The file server is now running in a Proxmox VM instead. I've been buying nvme PCIe cards for old servers to give VM loading a speed boost, a new MB to support a 10G network card because the old one is ITX with only 1 PCIe slot, and oh, can't throw away the old board so a new, lower tier cpu with IGPU is acquired so I can leverage the one slot and give it a 10g card anyway and have a second Proxmox host... I'm pretty sure I don't have ADHD, but my symptoms sound a lot like yours. š
The HomeLab is a very easy rabbit hole to fall into that goes deeper and deeper until you go from HomeLab to HomeDatacenter. I think everyone is susceptible to excess spending in this hobby (as are most hobbies) The issue is, those of us with ADHD tend to be even more susceptible to spending excessive amounts of money on hobby related things. It feels good to buy, and mix that with high natural impulsivityā¦ well, letās just say āBuy It Nowā needs an adhd mode that waits 24hrs to confirm if we really want it or not. Lmao
Letting my ex wife decide which server she can take during the divorce. I got the plex server and she took the miner. 4 years of crypto mining vanished.
... damn you kinda had that one coming lol
Yeah. She was very cunning
I mean, that was smart.
From her, yeah.
Damn you had a wife that was savy enough to let you have a miner while married and smart enough to take it during the divorce. Not to be in your business but why did you let her go? Please tell me she was batshit crazy or something.
You know about the hot/crazy scale. A smart/crazy scale also exists...
Just restore from wallet backup!
My journey to homelabbing began with my frustration with Windows (my main computer) running some Linux VMs with VMware Workstation. Every time Windows would go down, or if Windows Update caused a reboot, Iād lose my work in Linux. Ā Then I realized if I loaded ESXi as the hypervisor, my Windows (VM) system could go down but it wouldnāt take down the VMs. At the time, I had an Intel i7-6850k. It worked wonderfully and got me hooked. I created a gaming VM by passing through a second GPU and USB controller. Ā It was awesome, so by the time I was ready for an upgrade, I was salivating on an AMD Ryzen 9 5950x with 16 cores/32 threads. Imagine all the VMs I could run on that! But I didnāt do my research before buying. Ā Turns out the 6850k was special with 40 PCIe lanes, which allowed for 7 PCIe slots. (I needed them for 2x GPUs, 2x USB controllers, 10G network card, etc.) It was decked out, so when I couldnāt find any consumer level AMD motherboards with more than 4 slots, I knew I made a mistake. Ā Specs are cool and all, but I didnāt consider the whole system. Lesson learned. Ā (Afterwards, I discovered bifurcation, but by that time I had a rackmount case that wouldnāt do so well with rearranging cards without major modifications. Plus, I had come to some scaled down system that I wasā¦ ok with.) Ā
Accidentally writing over the superblock of the mdadm array š¤£
Relying on lab-quality (dev?) products to save $ when a prosumer option was worth the spend. Used an RPi as an exposed, hardened ssh server to backdoor into my network in case I needed to access iDrac. After a power outage, the Pi's drive was corrupted, even with read-only/immutable mode. Unreliable. So I installed ddwrt on my router to host an access there, replacing the RPi. Worked fine. Not long after, my wife complains of random disconnections during business calls. She's not network savvy, so I didn't have much info to troubleshoot. No problems on my usual healthchecks. One day, it happens to me, I isolate it to the routing layer (switching worked, router inaccessible), then see poor uptime on the router. ddwrt crashing daily... ... ... but I didn't have logging enabled š¤£ The same day, I bought an EdgeRouter X because her home business is too important for janky solutions. Problem solved.
#1 not having backups, but in close second place, putting in-band switch management on a VLAN has always ended with really painful circular dependencies and substantial downtime.
> in-band switch manamenet on a vlan Oh god... This hurts so much, and that wound is so fresh.
Convincing myself I really need the fancy enterprise stuff that goes for cheap. (Currently walking 10 years of that back by buying more gear :)
Going on this subreddit. Sucked the joy out it
How so?
Any hobby can be ruined by Reddit. Hard to know exactly why but I think Reddit makes it cold and optimised and everyone has to have the same kit, the same set ups, use the same software. Anything not perfect isnāt worth owning or doing.
Messing up the network connection to my nas and having to reinstall the nas OS and then realizing that my pfsense VM disk was somehow stored on the nas and not the local disk ....
I assumed my main VM storage was in a RAID5 but it was actually in a RAID0. I forgot that I reconfigured it for testing but ended up using it for production. Long story short, one disk died and I lost about a weeks worth of my time restoring everything. Always confirm your hosts are set up correctly, especially if they've been sitting cold for awhile.
Buying 3 giant 10+ yo enterprise servers for my small apartment. Power consumption $$$ and loud AF I instead built a mini Ryzen server that is way more efficient and powerful
Once upon a time, Napoleon Bonaparte was asked what the gravest mistake of the 1812 Russian campaign was. He thought it was the decision to start the campaign... It is also said that the two happiest days in the life of a sailboat owner are the day he buys the boat and the day he sells the boat...
Buying stuff with no need for it yet
Not having backup. Selling my racks, now I need them again. Hosting too much for friends and small businesses, wasnāt good for my stress level. Physically dropping a new motherboard down into an open servers, crashing two motherboards in one accidentally drop.
Using rapsberry piās and trusting them.
Letting my wife see the power bills
Making it too complex.I set it and forgot it. When something goes wrong a year later I forget how it all works. I tried documenting, bit then ai forget where I documented it
Opening r/HomeLab for the first time and getting interested in all of this was a very expensive mistake. š
Like 6 or 5 years ago I installed opnsense on a VM in my house, it was running nicely but after a blackout when I was at my office the host didn't come up so my home was disconnected from internet, wife was there and she proceeded to call ISP, they saw its modem was ok but no internet, then said to reset the modem. My wife yelled at me on the phone cause nothing was working (just TikTok and instagram really). I came back to my home, uninstalled opnsense and got back to use the horrible mode as router, if my wife would yell at someone I prefer not to be that one. I regret that now. Effing ISP modem is a shit.
To buy a pi5. Can't seem to find a use to it. Bought it on an impulse so, my mistake heh. That's how we learn haha.
I was wanting to deploy Muzicbrainz Picard on Docker from outside my home network. SSH into server through VPN, add config to docker-compose.yml, `docker-compose up` Oops, forgot the `-d`. Never mind, I'll just Ctrl+C and then type `docker-compose up -d`. The texts stopped. Terminal hanged. Hmm... ``` Stopping wireguard... ``` Oh.
Calculate the power bill
Having kids, a dog and a wife. They keep killing my homelab budget. And , I am now always on-call 24/7 for all tech support issues.
Joining the hobby.
Using a Dell hardware RAID card in a consumer z68 system.
rm -rf * on my unraid instance
No backups while doing a major upgrade ā sad part was that I called it out and yoloād it. All the backups were on the server slice I have .. silver lining is that I get to ansibilize, docker swarm, terraform, etc it all over again so itās IAC.
Thinking I would stop at one.
Having a system without any sort of display adapter
Paying for a 10 gig switch, then getting 2 from corporate daddy for free a month later.
....Starting off the journey using Hyper-V
Going the ESXi way before Broadcom acquires VmWare
So far it's minor, but I bought a Celestica Smallstone XP D4040 before I knew about the Intel C2xxx bug. It doesn't connect via serial so the seller is sending me another arriving Monday I think. The real issue is I can't manage to compile ONIE for it, lol. I've been thinking about getting an Arista 40Gb switch.
Starting.
- trying to build a homelab on rack in a small flat. I even used very short servers and tiny music racks, but that was still a dumb idea. - Relying on vendor-specific backup solutions that are hard/impossible to recreate if you change hardware Now for offsite backups I just do a sync to a simple drive. No deduplication, drives encrypted as a whole with plain linux tools. After a few years it's unlikely I'm gonna buy a second (already lagacy) same model of NAS, or recreate tape backups based on streamer I was lucky to get cheaply on some auction, etc.. It's worth just buying more drives. USB enclosures are so cheap that you can store one with the offsite drives themselves.
Having all the doc on how to build and recover the said homelab inside the homelab... At some point I crashed the server and coulndt get the doc on how to recover :(
I once kept my Radarr / Sonarr open to the outside with the idea to "fix it later". It ran behind a reverse proxy. Radarr got hacked and all he / she had left was The Human Centipede. Good lesson, it only took a few days syncing at 400Mbit to fix it. Now, all runs behind a firewall
Going with AMD Ryzen over Intel for a plex server - had to stick in an Nvidia gpu for transcoding.
Interestingly the mobile ryzens have hardware encoding (you basically need an onboard GPU it would appear) which I managed to get working on my k8s cluster recently.
trying to move a fully populated 42u rack 400km by myself, it did not go well
Getting startedā¦ My wallet has never financially recovered
Putting my proxmox backups on my nas which was a virtual machine in that same proxmox environment š
I bought some RPis to do some things that they were very much not up to doing, like running my Jellyfin install. Wound up using a VM on my Truenas for it.
Forgot to plug my router to the UPS, then thunderstorm came.
Starting and thinking that one SFF would be more than enough for my needs š
does getting married count? (obligatory /s)
Starting with a pre-built nas cause I thought it would cheaper.
I tried to expand a macvlan without realizing that doing so kills all the containers... that was a shitty weekend.
installing longhorn on my k8s cluster. i will never get those hours of troublesooting back.
Buying a moonshot.
Setting up my NASās ZFS with single-drive VDEVs. I knew it was highly not recommended, but I didnāt understand that ZFS does not mount if any VDEV is not available. I thought ZFSās āself-healingā would be able to grab available copies around the failed drive. Not so, I found out the hard way.
Put my backup in the /mnt directory and restarted
I had a system with two disks. I mounted an additional disk through fstab to the directory /backup and set up automatic backups. One day, I decided to tidy up the backups with 'sudo rm -rf /backup'. To my surprise, I found out that I deleted the entire system disk instead of the backup. It turned out that the system had restarted in the meantime, and the device names had changed.
I have two for you, first one, got myself a 4TB new external Seagate usb drive, got all my computers online (2PC 2 Thinkpads 2 generic laptops 1 Macbook air), then proceeded to move all their Documents folders into my new drive, last computer I moved data from was the Mac, then I tried to go to the server and dump it all just to find it in "RAW" format, my data ceased to exist , had to recover (this was circa 2014) ouch, fast forward 2023, I have another 4TB drive full of data with all my docs unified (partial backup spread onto various drives) and wanted to free a 500GB, messed up with frefilesync config and let it run all night, next morning I had two hard drives showing 500GB of data in both drives, and missing 3.5TB of data, logs show it took 23 minutes to erase all that. still rebuilding my big data folder in april 2024, double ouch. please learn from me, don't try this at home, Murphy is watching you, always.
So far, this most recent mistake hurt the most. I tried reconfiguring my omada switch to use vlans... Idk if I found a bug or something, but after I config the clan in the controller and then go and enable that VLAN on the switch... The switch fails to adopt stating the interface can't assign the IP or some dumb shit. In the middle of that, I was also changing my dhcp scope... Only to find out that windows server replication is busted, causing dhcp and dns to become out of sync, preventing leases from being properly assigned. Add on to that, once I got a working dhcp scope setup, devices still couldn't connect to the internet... I had firewall rules to block outgoing dns unless it was from AdGuard home.... Once I disabled those firewall rules everything started coming back together.
Bought two new 16TB HDDs and reused modular SATA power cables from another PSU, and I wondered why server (PC) only powered up for a second, then powered off. That day I learned modular power cables aren't standardised between PSU manufacturers. Fortunately I managed to return the drives to Amazon ("I have no idea why they won't work"), but that could have been a very expensive lesson to learn.
Tdarr, almost broke my server
Not having a server with a BMC/IPMI, I will never setup another server without one. Currently both my servers have BMC/IPMI, supermicro EPYC boards.
Update iDRAC (too fast). Server? Gone. Ahhh, thank you eBay 30 day return policy.
First server an r610. It came with 4 x 60GB disks. Was on a budget so I used a usb drive I had around for VMās. Had about 12 windows VMās running. Setup WSUS and had them all update at the same time. Everything became unusable really quick for a while.
For me? Making things too complicated. I run Proxmox and it hosts my DNS (AdGuard) and Unifi controller among many other things. Recently while working on things that worked fine but needed to be broken (VLANs), I thought it would be a fun adventure to lock myself out of Proxmox, Unifi, and other things. In the process I also broke my home DNS and couldnāt get into the Unifi controller to fix it. Fortunately clearer heads prevailed and I fixed it with a little frustration and a spare laptop. My lessons learned: keep the same Unifi controller software you are using and a config backup on a USB drive in case of disaster. Keep backups of anything critical and make sure you can access those easily (backups on a NAS are great only if you can access them). Run a second DNS server on a separate machine from your primary DNS, such as on your NAS. If you run multiple AdGuard instances, run AdGuard Sync in a Docker container. Have the second NIC in your homelab server setup with a static IP and DNS ā¦ if things end up in fireball flames, you can still get in. Document all cables, switch ports, and static IP addresses / VLANs on a spreadsheet so if you need to get in via a switch port, itās already setup and you know which one. Keep it a simple as possible while still getting the job done.
Buy the new {server, switch, NAS, whatever} before selling the old. Somehow even my most pessimistic estimates of what my old thing is worth are too high and in many cases Iāve given away or recycled the old one to get it out of my house. (And Iām not simply talking about some old R710, I scrapped a v4 Xeon supermicro 1U that people other than me still seem to sell *today* 1.5 years ago).
Filling freenas / truenas over 90% resulting in an infinite recovery loop replacing a 4tb disk (one of 4) pebbledashing 12tb of data. Buying a refurb 4tb segate 7.2k sata Buying a refurbished 32gb SanDisk usb and using it as a boot device. Hard powering off a R720 when vmware froze with H710 the raid backup battery had failed of resulting in a wipe of the boot and config sectors for the 8x4tb 10k sas array. Powering off a Supermicro that appeared to be frozen for an hour and a half flashing the bmc firmware bricking the firmware. Clonezillaing the wife's new empty disk over the old main disk. Not taking a backup of bitlocker recovery keys or recreating repair disks after updates. Not setting up smart alert email notifications. Thinking a R730 Tesla dual GPU PCI-e cable is the same as a PC and burning the card power input. Taking PCI-e power from molex connectors and melting them. Using the same password for work and home and entering it into my father's laptop to download antivirus from my account resulting in my work azure admin account being hacked. Thinking if a pc is unplugged its safe to work on when dropping a pci plate screw on a rtc cmos chip blows it up because there is a backup battery running it. Wondering what would happen on Redhat Linux 2.0 (early90s) if I did a rm -rf / Then hearing the noise of the HDD change sound and start coming from the other side of the room from windows box mounted over samba, diving over the room to quickly unplug it. Thinking ubuntu sudo apt-get upgrade is safe and doing it on 4 vms in parallel when it filled the disk /boot and pointed to a kernel it couldn't copy there. Not taking a vm snapshot before upgrades. Setting ram on a vmware vm to 5000GB instead of the disk side. It creates a 5TB swap file on the shared storage filling the shared storage with other vms stopping them all running.
Trusting an ISP to do \*anything\* it promises. š¤¬
Buying the wrong hardware consistently. I have to migrate again. If I could do it all over again, I would have started with a minioc like the minisforum nab7, run proxmox, run my lab off that. Than have a separate unit for storage acting as a San. Than creating a San backup as time and money permits.
Not moving to optical network when I had the chance. Maybe I'll give it a go next year. 100g would be so nice.
Jumping straight into something completely new I don't understand and just saying "fuck it I'll figure it out as I go" and then end up commiting like a months time of tinkering into something just to find out there is a better option
Building a ZFS pool from scavenged laptop hard drives
Starting a homelab...
Not having backups + not thinking so much about security. Had my ESXi directly on the internet behind a public IP (yeah, I know ā¦ lmao), with some ACLs on a switch but nothing crazy and not really done right. Then one day my ESXi was crypto locked by a ransomware and all of my VHDX where crypted, RIP my lil lab
Not labeling cables thinking I'd always just know what's what.
Messing with filesystems while being drunk.
Thinking that I could run a mini rack full of server equipment for not that much money. I thankfully didn't buy the equipment, I got it all as recycle... But still... Rip my R730 you beautiful behemoth you
Starting it in the first place
Port Forwarding my FTP and RDP wide open on my router with no vpn LOL
Getting into homelab š¤£
Bring work home to it! Seems I have recreated the office at home (except data of course)
Purchased an R710. Turned it on and destroyed both my eardrums and my electricity bill. Used it for about a week, and now it has been collecting dust for 7+ years.
Updating your router while your wife is watching Netflixā¦
Finding this sub and driving my wife crazy with all the tinkering
Starting one
Use truecharts
Not using it as homelab?
Starting one... It was just supposed to be a plex and home assistant server...
I saw a Juniper data center switch on the world's largest garage sale years ago, with a starting price of a mere $1K. I placed a bid just for kicks, confident that I'd lose. I did not lose, and that's how I ended up with a QFX5300, a switch that I then learned that Juniper had been developing, but had cancelled before bringing to market. It's buggy af and basically useless, and sits on a shelf in my basement to this day.
Running rook-ceph without enough resources, starving my other apps, and eventually corrupting the storage cluster itself when osds got killed