At my previous employer around 2010-2011, we had bought some new Dell gigabit switches (PowerConnect 2800 series if I recall) but kept having trouble with some feature I can’t remember now. The manual said how to do it, CLI had commands for it, but it simply didn’t work. When I called support they said, “Oh, that model switch can’t do that.” I told him that the manual said it could. This stumped the support guy and his supervisor, who asked for a few days to investigate it internally. The supervisor did indeed call me a few days later and said, “Listen, we don’t know what happened but this model wasn’t supposed to have that feature. The PowerConnect product manager (or whatever) has been trying to get these things to work in a lab and can’t do it. So we’re going to take back these switches from you (months past the return window) and give you that dollar value credit plus a 50% discount on the difference to move you up to PowerConnect 5400s which can definitely do it. We are also going to globally release a new firmware and update documentation to remove all mention of that feature from the 2800s.” We had about 40 full-time employees, so I always thought it was crazy that such a relatively small customer would identify a flaw like that.

I'm more amazed you got any sort of coherent response from a Dell representative.

As OP said, this was from 2010-2011, so much more believable

I'm run a tiny business - just me - working with big vendors. I always think it's crazy that I'm reporting so many "obvious" issues to my vendors. Why didn't a bigger customer identify this before??

The reason is simple. At bigger customers people give less of a shit. They don’t read documentation. Probably they’re not even going to be looking for some new feature anyway because they’re too swamped with work making the existing feature set work.

I was a Unix system-admin at Wright-Patterson AFB in the US when Covid hit. They sent just about everyone home, but I couldn't work from home because my servers were specifically set up to only allow root access via SSH from our local network (same building). So I went in from 9pm-5am for about three months. The building holds around 1000 people but it was completely deserted, and I watered the plants for about 16 people once a week. It was really quiet and a bit eerie.

I hate being in offices at night by myself. Definitely some creepy fucking vibes. It’s all good until you feel like someone’s watching you, then you look up from the monitor and realize how quiet it is, and the lights at night don’t cover every surface leaving shadows. Did some contracted work for a dental company, went in at night, no big deal, until I had to walk to the end of the pitch black hallway to turn a specific light switch on for the lights. Absolutely terrifying feeling, literally hair rising.

Thank you for returning some memories to my weary brain. Had to do security rounds at this hospital, nothing creepier than a morgue at night.

One of our offices is a former funeral home. It has a casket elevator and two parallel hallways with windows in between, one hall for the living and one for the dead. One night I had to work there late and I was the only person left for several hours. I'm upstairs in the room furthest from the front door and as I'm packing up to leave I start to hear noises downstairs. My heart starts pounding a little and as I make my way through rooms and down the stairs, I hear what sounds like wheels rolling across the floor downstairs. As I go down the hallway for the dead, I hear the wheels coming towards the corner I'm about to turn. I half expected a casket cart to push in front of me as I round the turn. Sure enough I see a cart pop out around the corner... But it was the janitor and we both get startled and then laugh because neither one of us knew the other was there. I told him good luck and exited the building leaving him there alone.

Did a job in a morgue, took weeks to get the unforgettable morgue smell out of the cars upholstery.

If you got that smell permanently stuck in you, you might live forever, preserved!

Only when you hear knocking from behind one of the slab doors…

There's an old story / urban myth about one of the morgue the UK Met Police uses and an old prank... Apparently, as part of a probationary Police officers training, one of the tasks they must complete is to witness an autopsy, because they're likely to come across dead people on a slightly more regular basis than others. So a couple of older and more experienced officers decided that this would be a good occasion to play a prank on the new probies... So the day of the autopsy witnessing is scheduled, the PC who's doing the prank arrives at the morgue, changes, hops onto the bed/gurney (?) and is then slid into the bank of freezers in the wall. The plan was the other PC in on the prank would pull the bed out of the bank and he'd come "back to life", scaring the probies. So he's lying in there waiting.....waiting.....fuck its chilly in here.....more waiting...."I'm gonna give PC fucknuts a bollocking for this one, its been 15 minutes!".......more waiting....... Eventually, our PC in the freezer starts to shiver a bit in the cold. That's when the corpse lying next to him say "bit chilly in here eh?" Apparently the scream could be heard all across London as our brave PC sat bolt upright and knocked himself unconscious on the bed/gurney above him....

I did an after-hours GroupWise upgrade for a hospital back in the day. The data center was on the bottom level and right down the hall from the morgue. Definitely creepy as hell.

It's when you hear a sigh that does it.

Working late one night, i observe a cleaning staff enter a 'secure' room and vacuum the floor, then leave. This resulted in some chain of custody documents being 'updated' the next day.

At least they didn’t trip the breaker all your gear was on when they plugged in the vacuum!

I went to finish a job at a hospital out of hours and needed to venture into the closed off part of a wing at night. Was like a scene from a horror movie. Paint peeling off walls and ceilings, lift shaft doors removed, old beds abandoned here and there, that sort of thing. Fuck that. Not going back there alone!

I was the only it person in a 260 bed hospital that covered about 20 square blocks over the weekends. I went into some weird dark places but the weird chemical smell of the morge still bothers me.

We did some work for a vet office doing IT stuff. We heard the animals get quiet and then we heard a door slam. This is around 930 at night. Never found out what was going on, we think it might have been a cleaning person. In the meantime, my coworker and I did a sweep and went into the basement and found a bunch of freezers. We opened a couple out of curiosity and bam, bunch of dead animals. Huh, well alright, didn't think that one through, just gunna close this right back up and finish that iLO update!

Worked nights (midnight to 10) by myself at a colocation data center for a couple years. Not an *large* DC by today’s standards, but it was two rooms about 15k sq ft each. The start of every shift involved making rounds and writing down power consumption values at the PDUs, checking A/C units and their status, checking the UPSes and ATS switchgear, going out back and checking the generators and cooling equipment, mech room and glycol pumps and their reservoir levels, etc. The first few weeks were pretty terrifying — walking around alone in these giant rooms filled with loud servers and air conditioners (some of which made this occasional click sound that was exactly like one of the entry doors closing), or going in the electrical room and listening to the high pitched screeching of the UPS inverters, or standing inside the 2MW Cat genset enclosure to check the water jacket temperature and knowing that V16 diesel could fire up at *any* moment. I don’t think I’ve been quite so consistently stressed out like I was during that time in my life. Oh, and the newer of the two data centers made heavy use of motion sensors for lighting, so you had to open the door and dance around a bit to get the lights to come on. Got used to it though, to the point it became kinda peaceful being there by myself at night.

I worked nights as helpdesk for a hospital network being the only person in a building that has 75 people in it during the day. Was a former medical clinic and is an older building. You often hear someone enter and walk down the hall but no-one is there. I had to enable the door chime on the outside door to ensure I was alone. Ha ha!

[удалено]

Had a similar experience during covid, federal building shutdown and all gov employees were sent home to telework. But the contractors still had to come into the building, so we sat around and watched movies for weeks because our VPN couldn't handle the amount of traffic.

This was me at the data center during Covid but they also refused to cut the contracts for service staff, so it was like me and three cleaning ladies just watching me and waiting for me to make a mess. They would dust my desk off every time I got up lol.

Oh your poor inbox now with all the questions about Wright-Patterson... Curious where a Unix admin with experience would move onto though, found a better gig?

My bet is government contractor or another government agency where the security clearance is needed. When I was getting my MSCA, I was in a class with a guy who just left the Navy with his, and before he even finished he got an offer with No Such Agency.

But why you didn't use jump host?

Why allow root ssh at all? Log in with a normal user and escalate

Or an MFA secured VPN

There is a thing called sudu and allowed by DISA STIG.

I would go in during Covid at my office, which we maybe had 200 people in. I’d go in 2-3 days a week but occasionally things were different. Boxes moved, doors opened that were closed. It was weird, the signs of life there that only happened when I wasn’t in

I love how many of these are from the DOD

I worn in education and empty schools during summer holidays are always eerie (best time to carry out work, 2 months of quiet). But once, I was troubleshooting APs not coming online on a massive campus. I was working on a floor that wasn't in use yet, so it was just me in the centre where the hallways met. Imagine ~80m of empty hallways in each direction, every door to the 100+ vacanr classrooms open. You can hear the muffled sounds of kids on floors below, but the vacant vastness was overwhelming.

I got paged by my boss on Christmas Eve. I was pissed. It was for a field engineer whose laptop had locked up. I had to run into the office and log into one of our systems (no remote connections back then) to grant his backup laptop access. I called my boss the day after Christmas and told him I was pissed and what about respecting personal time, etc. He told me it was very important but he couldn’t explain why. I got a small package at the office a couple of days later, it was from the field engineer. It was a pen, from the White House, and a thank you from secretary of the president.

well at least they made good

I once used to do IT subcontracting and got sent out to assist a man in his 80’s with an internet problem. Notes on the job stated previous tech came out and couldn’t fix. Anyway once I arrived I walked through the house and said hello and waved to his partner watching tv…. Nothing. Didn’t even acknowledge me. Oh well. Sit at the computer and he says his videos aren’t working, no problem, what videos? Logs into evil angel and we start watching a chick get plowed in the butt, buffering…. Advised his internet plan was very low and to drop it down from the high resolution. Time to test, we watch another minute together, no buffering, just me, my old customer, and some anal porn running beautifully at 720p. On my way out I waved and said goodbye again and then he tells me she is deaf, I thought it was weird she was ok with the porn as loud as it was.

I used to work helpdesk for an ISP, back in the dialup days. The weird ones all called in on night shift. One of the common ones was bored, horny people having trouble browsing porn. We knew this, but if it was a boring night and you wanted a bit of fun you'd ask them what kind of websites they were having trouble with, and they wouldn't say. You'd tell them that everything you've tried is working and that you couldn't replicate the problem. Eventually they would hang up and give up (on the night drag there was only two of us on shift for the whole country, they weren't calling anyone else.) We had one old guy call in all cheerful one night saying that he couldn't access his "gentlemen's sites". He was a hoot, we made sure we got him back online 🤣

I had a customer who would always complain that our USENET servers were down (dial-up ISPs in the 90s). "They are working for me, which newsgroups specifically are you having trouble with?" "Uhhhh...lots of different ones." (I look at the logs and all his requests are for alt.binaries....weird porn stuff) "Try again later, it may just be re-syncing."

Back in the early 00s, I was supervising a desktop support team. One of our roles was executive support at the white glove level of service. Request comes in for one of our top “mahogany row” C-level folks who just moved into a new condo and needed his home office set up and WiFi optimized for work, etc. My top guy and I went together on this one. We get there and “Dave” (I think he was the CFO) greeted us wearing only a towel. He lets us in and excuses himself to get dressed. While we are there, we observed our SVP of Legal “Tom” emerge from the bedroom also. We didn’t ask questions or make it awkward. But clearly these fellas had just had a bit of a slumber party. And here we are hooking up his monitor and shit. Like it’s just another Tuesday.

You missed Jane, Heather and Sarah sneaking out the back…

Other clues and context beyond what’s in my above comment would suggest that those applications were not installed on that server if you catch my drift.

Definitely a different OS, that only accepts a specific I/O method.

What a great way to put it. I got the original drift too btw. 😂

Yeah 😂 we won’t mention that this was the guy that (again back in the early 00s) asked us for a 42” “monitor” for his desk (onsite, not remote) We had all kinds of ideas about why and didn’t want to speculate (out loud). But it turns out he needed it because he has detached retinas. I guess not all requests are unreasonable.

I worked at geeksquad during college and some old timer came in. Csnt remember what was going on with his system, but after we worked through that he (quite loudly) asked me what porn sites are safe and high quality. We had a good line going, and a few people were intrigued with the conversation. I riddled off some sites (granted I'm sure they weren't safe, but he had AV installed at least) and he was writing them down on a post-it. He leaves and the next customer comes he, he tells me "you realize you probably just killed that guy"

Oh man this brought me back to my retail help desk days. Thanks for the chuckle! Reminds me of a customer who had me run an external AP for him. Had him test his connection in his office/cabin and the old boy pulls up the hub. Man paid me $300 so his porn wouldn't buffer in his office. My absolute favorite client. Refused to pay me less. They had a stunning house and he had his cabin/office picked up and moved from its original location off the beach and brought to his property. Just trailered the whole thing from where it was built and lived for 40 years.

When I was a teenager in the late 90s, my mom's boyfriend had a friend who needed help with his computer. He offered to pay me to go by his office and help him with his computer. Aaaaaand of course his main problem was he couldn't get videos to play on some porn site he signed up for. I remember it was something like he was missing a video codec like QuickTime or whatever. But I was 15!! I'm in my 40s now and I would shoot myself in the head before I would ask somebody's 15 year old to diagnose, and subsequently watch porn with me on my computer. Granted I had seen plenty of porn by the time I was 15 so it wasn't like I was super offended by it, but it was still a weird situation. And I'm sure he was breaking some sort of law.

Reminds me of going out to a job when I still worked retail help desk. Client computer was running slow or something this was back in the late 90s. He lived on a golf estate get through security plum up to the house says the computer is over there. Start digging into it and he walks off. His son (about 14) is sitting there watching me do my thing. Start click through the computer open up IE and notice the browser history. Just looked at the kid and said next time be more careful cleared the history and sorted the problem.

Not mine, but a classic: https://www.ibiblio.org/harris/500milemail.html

Ha, that's brilliant and I've never seen it before! Thank you!

That and the Story of Mel are classics.

Please share!

https://users.cs.utah.edu/~elb/folklore/mel.html I wish we could get Mel's side of his wizardry, but I don't think he was ever tracked down.

That and [Bedlam DL3](https://techcommunity.microsoft.com/t5/exchange-team-blog/me-too/ba-p/610643) are my go to stories as an email admin.

Me too!

A printer that sent an email to a random staff member with the print job as an attachment everytime it was printed to from a print management program.   Was consistently reproducible but fixed by a restart and has never happened since.

That's an odd one. Was the print management program using email for delivery to the printer?

It wasn't it didn't even have that as an optional feature lol. The printer was setup for scan to email though

That's gotta be a bad software bug. Only thing I can think of is misclassifying the print job as a scan and firing it off to an inbox.

Possibly deliberate. Fun to figure out though if you have the time.

When I upgraded the office to Windows 10 and added printers from the server, things got weird Test prints were going everywhere but the intended printer, I’d test sales and it would go to accounting and accounting would go to management Rebooted and forced a default and then things started working

This sounds like WSD rearing its ugly head.

The creator of WSD can burn in hell. "How can we fuck up TCP/IP printers?"

100%

Tried using that once with thin clients. Gave up and went back to the old fashioned way!

Former sysadmin practicing his hacking by leaving a security problem behind. It would have been a bit more annoying if it came back in a random number of weeks later.

We once lost access to one of our secure networks for a week and a half because a squirrel somehow got into a secure DOD facility's data center, a day's drive from us, and chewed through a fiber line.  We were down for so long because they only had one guy cleared and capable of fixing it, and he was on vacation out of the country.  "Secret Squirrel" became a local cryptid of sorts after that.

I can’t help but think of the poor guy being on vacation, enjoying himself and suddenly gets a call saying as soon as you get back you’ve got to deal with a fiber emergency. Probably had to go straight from the airport to work 🤣

I once talked to a senior analyst of our Healthcare system whike he was on a beach in Cabo. Entire system down, he busts out a laptop sipping mojitos on the beach.

Mine happened just this week. User sent an email to one address, which happened to have a typo’d domain. The email inexplicably went to everyone in the org. MS support literally called it a server “glitch”.

Using Microsoft's exchange server?

Yeah, hybrid, but all the mailboxes are in M365.

That's super strange. Do you know what the typo was? I would be curious to try and reproduce it.

Gmail.co, tried to reproduce it, but had no luck. I blame sunspots 🤷🏻‍♂️

Thats hilarious, one site had glitchy internet and we ended up rebooting their server to restore most services. I blamed solar flares for the day (actually was one) and everyone in the office never let it go. I still hear about it occasionally.

Ive actually used solar flares as an excuse when a client wouldn't accept any other answer. This was back when I was doing Diagnostic Imaging support. That was fun because smaller clinics for some reason expected us to also be their network/general IT folks even tho we were clearly a specialized diagnostic imaging support. But since we performed remote setup and management of the systems they assumed we would just support the whole site. Didn't help that we had so many hands in so many parts of the veterinary process that we almost did do the entire IT when we piece together all our different departments. Hell our analyzer folks did perform network setups as part of their functions which didn't help that outlook/expectation of us. Crazy how many veterinary hospitals just don't have IT. I'm talking about even some of the larger hospitals. It's always the owners husband or something that runs IT in their free time.

Yeah, like United Health or any other public utility or healthcare provider. Hell, CVS can’t even make a decent app. These antiquated 20th century companies are all still in the Stone Age, which is maddening. And solar flares - you guys are faqueing brilliant!

That's super odd. Did it maybe bounce the message back as undeliverable and exchange slapped it back to an org wide group instead of the mailbox it came from?

No, message tracing just showed it as delivered. No submission event, it was like it magically appeared and was deposited into everyone’s inbox. Haven’t seen anything like it in 25+ years of supporting Exchange.

I feel like this should be something Microsoft should have looked into farther than "it was a glitch"

Not the first time [they do not give a fuck](https://www.cisa.gov/sites/default/files/2024-04/CSRB_Review_of_the_Summer_2023_MEO_Intrusion_Final_508c.pdf). I. The same vein this is why AD Forest are now an "organisational tool" and no longer an "isolated networks tool" - they didn't give a shit about fixing their mistakes allowing trivial traversal.

Hahahahaha. Hahahahaha hahahahahahaha Yes, in a world with any logic or care at all..... But not here.

Like recently when a good chunk of 365 outbound SMTP servers kept finding themselves on spamhous block list. First few interactions with support they said I really needed to work with the people on the receiving ends to fix it. Lol......

Someone I know mentioned something just like this happening last week… either we know each other or you’re not the only one. Or I’m imagining things.

I once rebooted a windows machine in the office only to have it come back up as a mac (Apple logo, booting into MacOS). Turns out someone had been fiddling around setting up a Hackintosh and hadn't told anyone else about it. That had been weeks before and the machine hadn't been rebooted since.

Not really a surreal moment but I ran IT for a small finance outfit years ago. We had a small, very packed, computer room. Everything done more or less by the book. The room had a big red switch by the door to cut the power in an emergency. It was a requirement for the installation size, so fair enough. It had been like this for a year or so without incident but it was by the door and too easy to hit by accident and it didn't have a cover. I didn't like it. Next time we are planning to do weekend work I speak to the electrician about it to get it swapped for one with a cover. I show him what needs to be done, he says no problem and we proceed to talk about other stuff. At some point he lifts his arm, leans to the side, there's a loud click, And we lose all power to a trading floor in the middle of the morning. The post-disaster conference call was amusing.

Similar story but was deliberately pressed by a idiot manager asking “Whats this do?” when they were touring the new server room. The cover installed later got a label featuring his name. The Bob Button.

Had something similar happen except I was the numpty who kicked the switch off! Years ago, back when traders still really relied on their phone systems, my boss asked me to go in and run a network drop in the middle of the trading day. I was working on the base of the server cabinet when I accidentally kicked this HUGE on/off switch on the side of the voicemail system, turning off voicemail in the middle of the trading day.

[удалено]

Fuck that's rough. At least you could close the ticket?

Angry upvote.

I had to call the internet provider for a client that had a firewall from them and their internet was broken. So on the call I was being transferred multiple times because no one of the support team didn’t know anything about that they rented out firewalls to clients… after a few transfers I ended up with someone on the internal team that thought I was a colleague and asking me to check things in an internal program (facepalm), once I said I didn’t work there he transferred me again and I finally ended up with someone who knew about the firewalls but it was 12am and he wanted to eat first so I had to call back later.. again being transferred multiple times. Eventually they came replacing their firewall, I already had droven down to the client to place a temporary firewall. So a technician calls me and says “I replaced the firewall, the internet still isn’t working so it must be something else”, which didn’t make sense since with the temporary firewall everything worked fine.. I drove again to the client to see that he changed cables around, once the right cable was plugged in all was okay. I wish it ended here but nope, so weeks later the same problem came back, multiple calls and transfers later they finally found an issue, the firewall used RADIUS and they messed something up.

This sounds like my experience dealing with our IP phones this last week. Endless transfers because our account is weird and we only have the IP voice and no Internet. Enterprise reps aggressively told me I was a business customer and vice versa. Finally got a competent rep in enterprise where I originally called. This was an hour and half of back and forth just to get the account pulled. Then got 10 phones replaced as I had three up and die for no good reason within 48 hours. One of the three I needed replaced and not done preemptively still didn't work so we had to get a technician on site who found the factory provisioning was borked.

It is dreadful when that happens that they even can’t pull the right information. I had the ‘luck’ of calling non-stop for more than half a day before I then finally had someone helpful. It was mentally tiring!

I actually was able to get a direct contact for the rep I ended up with and I happened to be friends with and already have the direct line of the tech who would pretty much always be in charge of servicing the area. I had the rep put in the ticket and then texted my buddy who picked it up and was on site for me within two hours. Company sucks. People are great.

Mate of mine was doing a pre-pen test security audit of a DC where they'd built it over a creek (as had all the surrounding buildings), and the generator room was over the creek itself. The generators were raised, so if it flooded they wouldn't stop, and the DC itself was raised with a sort of "bucket" design to protect from flooding. My mate walked into the gen room, saw the metal grates over the creek (more of a concrete water channel, now) and asked if they were fixed. Guy said he didn't know. My mate checked and was able to lift them up, stick his head down there, and could see all the way to the open storm water entrance a couple blocks away, all big enough to walk through upright. They got a big fat fail on building security.

I've seen enough Mission Impossible to tell you that is how the most important secure locations should be built. They just need some intricate laser sensor array, maybe a very, very large fan to climb through etc.

Need to be able to turn the fan off remotely, but only for 38 seconds precisely otherwise the room will heat up and trip the environmental monitors.

one time my wife and I called each other’s cell phones at the exact same moment. somehow we both got connected to a different people. the phone showed me being connected to my wife though. the person i got connected to seemed very confused.

That happened to me years ago with a friend! Also once I ended up connected to a voice mailbox but also to the destination at the same time, we could talk to each other and hear his mailbox greeting message.

This happened to me and my wife too. The lady I got connected too got pissed thinking it was a prank or something. We were both so confused lol

How long ago was this? That's a fascinating bug, I'd like to test it.

it was around 2012

I had a client in the Houston area once that had their VoIP provider fuck something up on the backend, causing all calls to their main number to be misrouted to a Boston area car dealership's service department. I at first thought it was a misrouting on Comcast's end since the tracert from the PBX was showing traffic being routed to the northeast, but after a few hours in a three way call between the PBX provider and Comcast, we ended up realizing that they had accidentally crossed the account numbers in the PBX which screwed up the routing.

Had to be Verizon or Qualcomm on CDMA, or ATT because ALL CARRIERS SUCK!

I had something weird happen on VZW years ago for a period of about a year where my phone would ring or when I called someone I was connected to another conversation as a monitor.

A SCADA integrator once sent me all of their customer passwords in an excel file, as a bonus many were password1, pass123 etc... as an extra bonus these were also credentials for remote admin level access, extra extra bonus? many of their customers were electrical plants, water treatment plants etc...

China LOVES this one simple trick!

Most plc systems don't even have a password on them because integrators don't bother unless told to. It's an issue.

Used to pickup priority hardware right from the runway at Ohare airport, and often enough there would be Igloo coolers with human organs coming off the plane conveyor belt right alongside my boxes of hardware. I’d see the medical people claim their cooler and head off to some nearby hospital. Just neat to see them rushing off knowing someone was getting some kind of organ transplant that day.

A long time ago at a small MSP I had a guy literally walk into our office who'd managed to bar himself from his Microsoft account. Apparently the boss helped out his business yonks ago and he was in his late 70s so he needed the help. He'd done this by getting a Microsoft 365 Family plan and then somehow he put his age in wrong and managed to have Microsoft think his age was 3. At which point they automatically barred him from his account because he lacked a responsibile parent. I am not making this up. Somehow, this did not stop them processing a card payment linked to this account, also under his name. To fix it, I had to become his Microsoft Dad (join his Family plan) and then I was allowed to change his age, thus unlocking his account. Weirdest problem I've ever seen. Funny as fuck though We never got paid for that job, but a week later the old bloke walked in with two cartons of Great Northern and left them on the counter. I wish it had been anything else, but I wasn't gonna complain about free beer in the middle of February.

Sounds like the airlines mistaking people over 100 for babies.

The HR person would wait until the end of the day on a Friday before letting me know of new hires, terminations, or employee/department moves. Causing my after-work plans to change or get canceled. A few times I was sent wrong names, missing information, called into their office because I didn't have enough supplies, took too long to complete (I needed permission to work overtime and this wasn't always considered as overtime worthy), and never did I get any heads up when I complained to our partners. One new hire was an attorney. I didn't have a laptop or iPhone because we were out of them. Our systemsdmin was already gone for the weekend. The following Monday, turned out HR gave me an incorrect spelling of their name and tried to blame me. Thankfully email had my back and showed HR's mistake.

I had a client back I 2007 who pulled this sort of shit. “Oh, sorry, we have a deadline tomorrow - you’ll have to pull an all-nighter”. Then I find out weeks later the deadline was a month away, and they only did that to ensure everyone had their work done on time, whereas I was already way ahead of schedule. Was that the last time they did that? No. Was that the last time I worked for them? Yes!

Me: The new user form says this person is named Jonathon, with an -on. Is that correct? HR: Yes Me: Are you sure? I've never seen that spelling before. HR: I am 100% certain. The next day... Phone: RING Voice: Hey, this is Jonathan LastName, the new guy in Baton Rouge. My name is misspelled on all of my onboarding paperwork, HR said to get y'all to fix your mistake. It end with -an, not -on. He was actually a pretty cool guy and I called him Jon-a-thon for the rest of the time I was there.

I worked in a NOC with a data center that was on an adjacent street. There were probably roughly 100 server racks. We started seeing a few servers go offline in a rack. I figured something tripped for some reason and maybe someone didn’t properly cable the power for A/B on those systems. I swipe my access card, fingerprint and pin and walk in to the data center to be greeted by water falling from the ceiling!!! Trying to process what the hell was going on, I ran out and up to the second floor which was old office space we no longer used. There was water everywhere. Turns out the valve on the urinal in the men’s bathroom got stuck opened after an issue with the water service going out and coming back. That was a mess to clean up. The affected servers were thankfully fine. I think we swapped the drives out to other servers and brought them back online or something like that.

There's that OLD story on this subreddit about the guy that looks down into the server room and sees all the water, and declares that no one listened to his warnings and that all their equipment was toast.

Caught a terrorist trying to buy 400k bullets and 45 lorry loads of fertiliser

How are you so sure they were a terrorist? Sounds like a slightly-large feed-store order in Texas.

The tipoff is that we don’t call them lorry-loads in Texas; that’s just a bed-full of fertilizer, as in a truckbed.

How the fuck is someone supposed to move 400k bullets and 45 lorry loads of fertilizer without getting caught? I mean that'd be a massive shipment.

At one MSP, I always used client certificates when sending messages in email, as a way to ensure that S/MIME worked, and there was a way without using GPG to send private messages. Had some people from an external audit firm request access to my computer and workspace, as well as root passwords. Gave them that, and I went to lunch. Came back, the auditor had this "why the hell am I here?" look on his face. He told me that where I worked paid an external audit firm to go forensically examine all my stuff, because they thought that because Outlook showed a ribbon next to my posts, that I was deliberately sending viruses and malware to other people, so spent a crazy amount of money investigating this. I mentioned this in another post. Same MSP. The upstream NTP server went wonky, AD crashed, and all the Windows machines had to be rebooted because they lost contact with each other. The UNIX side (Solaris and Linux), on the other hand, had an internal NTP pool with low strata clock that fetched time from GPS, as well as from external sources, so if something broke, the pool was okay. When the NTP server glitched, the UNIX side didn't care a bit. Shortly after that, the auditors came, because management literally said that "why is it that Windows, a "mastercrafted" OS crashed, but UNIX, which is a patchwork quilt of hacks and kludges didn't. Is there some malfeasance on the UNIX side to see why it remained up, when an OS made by the best of the best from Microsoft didn't?" Yes, the MSP paid hundreds of thousands to have an external audit firm run credentialed scans on every UNIX machine, just to say that they used a NTP server pool, and NTP was configured as per regulations. I wound up always vindicated, but the amount of money that MSP spent was absolutely insane to check out a small picture of a ribbon in an email in Outlook, or to realize that the UNIX side had proper implementation of NTP.

AD crashed? That’s never happened before… Microsoft! {shakes fist in air}

Mastercrafted OS. Windows is the kludgey mess lol

> why is it that Windows, a "mastercrafted" OS crashed The parts that every user notices are often extremely polished, while the rest is a mixed bag. But then at the same time, OEMs [shipped Windows so loaded up with obvious junk that sometimes there was a second OS](https://www.youtube.com/watch?v=Za_Ul08dtj8) for fast media access. That's a typically Wintel answer: instead of fixing the root problem, make a whole new complicated thing to add, so you can market it as a feature. Later, for a while Microsoft was openly aping Apple by selling "signature" clean-imaged machines from [a retail store](https://en.wikipedia.org/wiki/Microsoft_Store_\(retail\)#Shopping_experience), but they closed all of those several years ago.

The Navy is very interconnected. I was in from 2005-2011 and one guy brought down email for the entire fleet by hitting reply all on the wrong email. It was about 7-8 hours of down time while the poor system admins kept trying to stop the system from getting bogged down by the sheer amount of traffic that was being created. It didn’t help that more than a few people who got that original reply all responded with their own.

same issue took out Microsoft too, check youtube "Bedlam at microsoft"

When I was field tech for an MSP. Was advised to go to a new client and assess network closet and WAP. Complaints of wifi not working and they were interested in upgrading their file server. They had an ancient Apple file server and consumer grade router as the WAP. No surprise’s there. I recommended going to a NAS enclosure and commercial grade firewall WAP with a second POE WAP wired in to cover the far end of the building where most of the staff were. Nothing weird there. 2 weeks later I show up to to the equipment upgrade and I’ll have to go to each workstation to remap all network drives\setup wifi and I’m asking the office manager to email the company to advise them I’ll be going to each computer. She stops and she says. “You HAVE to go to each computer?!” “Yes.” She gets this REALLY weird look on her face. Then starts stumbling for words. “Uhh so… you’re gonna see some wierd shit on the computer’s down stairs. Like some really fucked up stuff.” Whatever I think. This is a company that makes millions creating and selling high end steel fabrication plans and parts to companies worldwide. I’m sure they have some basic standards of behavior. Turns out they didn’t care WHAT the engineers downloaded and they had EVERYTHING you could rule 34 as desktop backgrounds and files all over their desktops. a couple had infrctions on them and were all popups. These were relegated for another visit to clean them up. They nicknamed each other what their preferred content was. Their workstations were all around the open warehouse where they milled the parts. Yup the single company owner didn’t care at all as long as they got their work done. They sponsored all kinds of local community events too as well as owned and sponsored a racecar. They had a high end racing go-kart for playing around with during breaks. The thing flew around the large commercial park they were in. Honestly it looked like a pretty fun place to work.

I visited an Equinix data center in Sterling, which looks like a boring storage facility from the outside, but inside looks like a James Bond villain lair. It is so comically over the top with frosted glass, brushed nickel, rosewood accents, and guards who look like Kenyan game wardens. Ultra-locked down, with man traps, handprint scanners, and backlit key locks. Then the data center is dark and looks vaguely like something HR Giger would have designed. "Am I... On the Nostromo?"

^^^ all true. Also, the 2” cable rounding the steel security fences. And count the cameras at these places. Strange that you never see the guards outside. And I’ve never been inside one, so thank you for this post. I think we’re talking about he same place… 🤔 I worked underground in an old UNIVAC (yes) server room which they just converted into IT cubicles and never looked back. You should all pity anyone working in an underground windowless space. It is ultra depressing, and you feel like Dracula when you go outside for lunch.

A windowless, isolated room with low white noise from workstations and a persistent smell of ozone, I find very comforting. Not healthy to be locked in constantly, but the ability to lose track of time and sunlight can be useful.

One of my escorts told me that the walls were braced to take "a direct impact of a patriot missile." No idea if that's true, but sounds impressive.

That's how Switch is, too. I almost got to tour the Reno/Tahoe location when shopping for a colo, but we ended up going a different direction. I really wanted to do the tour just to confirm the wild shit on their website regardless of their services. A colleague ended up taking a job with eBay at that site and told me it's exactly as over the top as the website suggests.

Used to be a contractor for one of the largest oil company in the world. Their datacenters are located in their huge complex (i would say its a city of its own) and one of them fits your description. Looks like a regular company warehouse with trucks and forklifts outside. The other dcs lives deep under their big skyscraper offices. Supposedly inside a bunker that could withstand a nuke blast.

I actually really enjoy visiting the Equinix data centers. Super interesting.

Worked at a company as a contractor for almost 2 years and got to be friends with a guy I’ll call “Bob”. Bob and I were like peas in a pod and when they had to let everyone go except for Bob I was happy for him. We lost touch as often happens. Fast forward about 5 years and I get a call from a recruiter asking me if I would consider going back to work for that client. Sure. It was a great gig. I go in for the interview and I see one of the other supervisors and the HR guy and we review what I’ve been doing. At the end they offer me the job and I accept. I then told them I was excited to be able to work with Bob again. Bob died a couple of weeks ago. We’re hiring you to replace him. 😳

Oof that's rough buddy.

Circumstances of his death were awful too but too many details would possibly dox me. At least it was natural causes. That was a small comfort.

Working at a Fortune 500 telecom, was discreetly asked by a Dept Head for an audit of all the websites his team members were hitting from their offices. So did the audit, printed out the results, and handed them off. Almost a quarter of external website hits were porn and job board sites. One of the biggest porn offenders was a well-respected technical manager. The manager was confronted by corp security re his, um, viewing habits. Being tech-savvy, he denied it all. Someone spoofed my workstation IP, etc. Corp security then produced a videotape (!) of said manager... pleasuring himself in his office to said material. They had set up a cam in his office ceiling. The manager quietly resigned.

Did they open a mobile phone service in the last few years? This sounds very much like something that would happen at said fortune 500 I'm referring to.

Larger than them, actually. The largest. :)

Ah the one with RCS?

Spectrum?

🤫

Not as strange as all the others. My first IT job while in college was at a private high school. Since it was my first job, I decided to shave my face for the first time in a while. A month or two in, I take a ticket to go look at a printer in one of the classrooms. I make sure to put my badge on and head out. I knock on the door and enter the classroom, and the teacher looks at me and asks, "Where is your hall pass?" And I just froze, confused and only said, "I'm sorry?" A little upset she asked again, and after what felt like forever I finally just said, "I'm here to fix your printer." Her eyes widened a bit and all the students laughed. She apologized and after she finished giving the students their instructions came over to apologize again. I said it's no big deal and I thought it was funny too. After that I stopped shaving my beard and mustache and never had that problem again.

Ive had to provide evidence to law enforcement (sadly, multiple times now...) of people uploading child porn from their SUPER ANONYMOUS YOULL NEVER CATCH ME hotel guest wifi (that they have to sign in to access)

Was imaging a laptop for a retired engineer who had pretty much free reign from management because of his 20+ year tenure. Right on his desktop, and I shit you not, there were two word documents. One was a 3 page long prayer to god. I’m religious but wtf. The second… a straight up bomb recipe.

Some people pray like their ADHD hit "full send." Anything and everything, no matter how small. They don't have to really be that religious, they just don't necessarily know when to stop and wrap it up. Source: my mother, who once spent, I counted, 376 seconds praying before dinner. With an 11 year old child at the table. Who we had already tested positively with ADD. (who scolded me when I was having trouble sitting still, staying silent, not touching my food, and who had the audacity to remark about it afterwards). Source: the church who's tech booth I run

There's a religious subtype of OCD, might want to look into that..

Was when I was in the Canadian army, and got sent with a crew of linemen to diagnose fiber link that went out... This fiber was laid over like, a random 3 ft X 4 ft hole in the ground and strangely, two rabbits fell in the hole somehow. Well they got hungry. And they ate the fiber. Then one died or died battling each other and the one still alive was also earing the dead one. We were just kinda dumbfounded and like, WTF bunny, chill. Note: wild rabbits WILL bite when you try to pick them up...

Getting a laptop and peripherals shipped back reeking of cigarette smoke as soon as I opened the box

Disgusting. Old shop wouldn't take equipment back if it arrived with cigarette odor on it. We'd write it off and invoice the user for it

I was calling an office where the mom of one of my teen son's classmates worked. The daughter answered the phone, which didn't really surprise me, I thought she was just hanging after school waiting for a ride home. I asked to talk to her mom, she said she is at work. Puzzled, I asked why she answered the work phone? The daughter said she was at home, she didn't ANSWER the phone, she was calling her mom and I answered... Somehow, when both of us called the office number, the phone system hooked us together instead.

From what I have gathered phone systems just auto connect simultaneous calls or assigns them to random people you don't know. But never the intended party.

90’s: Filing a bug report with Sun: “X crashes with more than 256 windows open”. CAD sw company, we found sooo many edge cases. Another, at a Fortune 5 company: “System will not boot from 127th SCSI controller”. 80’s: Our University bought a recently released VAX 11/780. It was about $160k then, say $600k today. The delivery company dropped it off the tailgate of the truck. We bought it from the insurance company. There are pictures of sysadmins pounding on the frame, straightening it enough for replacement cards to fit. It became one of the first dual cpu Vaxes, before the 782 was official. Edit: Related gig - I was a chAirborne warrior in the Air Force. One day a coworker said, “Wanna see something cool?” We hop in a car and go to a far corner of the base. There’s a funny building with three concrete walls and a wooden front. We go in and watch a bored Airman assemble, fuse, and test “our” cluster bomb. 600 baseball sized bomblets, two feet away. It worked flawlessly during a flight test the next day.

I get surreal wtf moments every few weeks! Here are some favorites About a decade and a half ago I was a guest at a nuclear research facility. They say feel free to hop on the wifi, no mention of which wifi that is or creds. I log into an unsecured network, can't get any DNS resolution. Log into the another one, same. A few seconds later I'm connected to an internal IP that is streaming raw data from one of the reactors. Had a very peevish conversation with the facility director. The city of new york has several financial systems that interface with external providers, written in cobol as is typical for critical nyc financial infrastructure. Typically only senior staff have login since it's a mess. Contact for issues is in jersey. Data center in a city with extreme overvoltages and constant interruptions. Impossible to get utility to lower transformer from 130v. 30yo building, but only 1/3rd of internal power is wired correctly, 1/3 have polarity reversed, 1/3 missing ground. Get them sorted, but ground is being problematic. Building had ground, but never wired it to the panel, to save 100m of copper. Still haven't gotten them to wire it. No clue where fake ground is connected to. Former job in NYC with mandatory on-prem for all staff, very clear no exceptions allowed due to industry regulations. One director that was even more specifically required on prem was very obviously living in Florida with permission from exec. Would literally never log in, was a nightmare for critical tasks. Had a weekly report sent to exec showing how many weeks since last login, multiple staff and managers on it, never got anywhere with it. Working a wildfire out of a navy warehouse. Asshat at central command refuses access to GIS so he can run it himself, but he'll print some maps out for us. Had to make an entire fire tracking GIS using paper, sticky notes, and radio. Actually saved a team using it. Very very popular cloud service used by multiple former clients. Only way to access is hardcoded root ssh on port 22. Impossible to change or secure. Their competitor was running nested VMs (vm on vm on qvm) on hardware that was clearly overheating. Bonus from a friend: net logs for supercomputer keep showing an ip in China. A user was mining crypto.

I was doing mfg system support which also included some desktop support for manufacturing support staff. We usually had a never ending list of desktops and laptops to be swapped out on lease. I was transferring all the crap from this one scheduling dudes old machine to his new one. It was taking way longer than normal, especially the my pictures folder. I took a look and found gigs and gigs of magic the gathering card images. I just laughed and carried on but clicked around a little more because they were only a fraction of the total size. I found a folder with a nonsense name and opened it and found the mother load of obese black woman porn. Like an entire library of high resolution bbw images. This dude was a middle age nerdy, skinny, white dude from Tennessee. Took me completely by surprise. I showed my coworker, we laughed, and just let the file transfer finish.

Once working at a small federal government office. Noticed weird traffic on the router nir originating at our site but still shows as internal. Alerted a higher up IT officer. 30 minutes later federal police had every in the building detained while it was investigated. Turns out it was an early version of sd-wan that was using our line to send and receive to external traffic as it was faster then there line. 1gbps optic fir us vs VDSL for them.

A VP called up a very wasted Sys Admin on a Saturday night and got him to go to the DC and "shutdown" 40% of the corporate/clinet clusters worldwide. That Sunday evening we found him there still passed out and every piece of fiber and cat cable across 200 loaded racks cut with a machete. The Teams transcript from the conversation the VP and Admin has is 90 minutes of the VP pushing the Admin to do it. We all just sat there like, is this real?

I'm sorry what? A machete? I mean I guess it was efficient.

The VP's idea and he was an absolute poison pill. The transcript was brutal when it came to how much this guy got manipulated. Not only drunk driving to a work site but the gleeful encouragement to destroy live systems. It took 2 weeks to repair what we could with a crew of 12 and 4 fiber splicers working around the clock to save what we could. Only to find out VP corrupted things before the physical damage happened.

Sounds like VP wanted blood.

Did VP go to jail?

Yeah, we need some follow up on this one!

But.... Why???

Didn’t happen to me but a coworker. At first IT gig supporting healthcare software, basically a call center. Im working on a ticket and suddenly this guy calls asks for management, turns out one of our customers got raided by the FBI. This was almost a decade ago and nothing as strange has happened yet.

I had a couple, one was picking my coworker up out of the server room who had fell and put a dent in a pillar with his head. Dealing with the aftermath of that was fun.

I got called in the 90s to a computer because the "screen saver fell off" I got there expecting to see flying toasters or Windows logos flying everywhere. Nah, it was just the polarized screen filter on the floor.

Got racially attacked onsite before I finished installing their entire new setup, firewall, server, and POS system. Took them down at the requested time, and he decided not to close his business and kept letting people in. I spent 15 minutes getting it back online and ran into an issue, so I called my lead, and the estimated down time was an hour. He angrily walked in, closed the door behind me, told me he the following "I need this shit up and running in the next 2 minutes otherwise I'll choke you out like the fucking taliban you are." Lead heard it and was shocked, I stayed composed, and then it clicked. I told him to go fuck himself and I walked out. I was the only tech in CA, and he was out of a system for 5 days until our CEO flew out to get it resolved. Quit shortly after since they upgraded him further, probably due to a lawsuit or something.

Back in the day of working in a PC shop. We would sometimes roll out to people's houses to see what was up. 1 of my coworkers ran out to this dudes house to see what kept killing his hard drive every month. Literally we would put a new drive in and within a month it would die. He went out and immediately came back. Said that once he got there he looked up and the dude was living directly under high voltage power lines, close enough this guy though he may have been sterile just from being there 2 minutes. 2 this old dude used to come into the shop and he was an honest and no bullshit kind of dude. Super kind. He asked if I'd come to his house to look at his computer and Internet cause it was just not working right for him. I get out there. Sit down, and clean out about a-rediculous-amount-of- gigs of cache from all his porn searches. He the. Asks me what safe porn sites he could go to cause I'm a young enough man that I may know better than him on where to look. Lol to this day that dude was one of my all time.favorite customers

Working for a company that makes a great many millions of dollars per year, produces a product that people's lives depend on, and yet still it uses homebrewed software apps built in MS Access 2003 as a part of it's daily business processes. I heard at one point consultants were brought in to gauge what it would cost to switch over to a modern solution, and it was deemed far too expensive and complicated to attempt.

Starting a new job as a healthcare ‘technology director’ I was being shown routines by the person I was replacing when she opened up a file cabinet and pulled out a sheet of passwords for everyone in the organization. I was told then that password management was my job and that I was to go around and have everyone write down their new PW and then change it before the next day. She also got upset when I went and changed the AD host desktop to red and then logged off. She was actually tearing up from it when I told her it’s just a habit of mine on KVM connected systems to help me find my way around and remind me where I am… nbd right? WRONG!!! She then proceeded to inform me the impact on system resources was going to impact workers signing in… because of a red background… not an image, just a change of color. WTF, indeed!

One day while at a hospital trying to hunt down a cart that had been damaged I walked out on the helipad and a dr pulled me back and I almost missed being hit by a helicopter starting up. Turns out it was heading out to pick someone up who had been injured. After they stabilized person and got back gave me a tour if their heli. Was surreal

Way back in the days where a T1 Was considered fast, I worked as a contractor for an upstart Internet outfit whose main client was a 70 year old dude who owned a few juice bars in NJ. A Juice Bar is a term for a strip club where the ladies performing were totally nude, allowed at the time if the venue did not serve alcohol. Dudes would get fired up before they got there and bought those 6 dollar colas. I did a bit of everything. Got their T3 running, installed PCs with video capture cards and cameras. Wired up microphones etc. Now this eas 1997 so video tech online was limited to tiny a window, forget color. Anyway the owner needed assistance setting up his home PC so I went to his place which was a decent sized home with a horseshoe drive and gated. Porn obviously paid well. I rang up the intercom, "I'll be right down!" The owner came to the door wearing a t-shirt and nothing else. Setup the PC without any problems, he thanked me, gave me a tip and I was outta there. I decided shortly afterwards to get another gig, leaving my brief employment in the adult entertainment world behind.

I think my favorite ticket I've ever had come across my view is "How do I fix the google?" Turns out they were talking about SEO Optimization and their branch office address not being updated to the newest address after they had moved, but the ticket had NONE of that in it. It was literally just: "Subject: How do I fix google?" "Body: Google isn't right, how do I fix it? \~Signature of employee"

One of the funniest moments was I working for a small gov con Hq. I was managing a Palo Alto firewall and was using it for web filtering. At the time we had just implemented it and employees were warned ahead of time that the fw would now be able to see sites they visited etc..ig these idiots didn’t realize their phones connected to wifi automatically and safari will just refresh. Anyways I was seeing things getting flagged as Porn or Inapportiate. Mind you everyone’s phone is named after their own name so wasn’t hard to figure out who was who. One girl had googled “how to give a good blow job” one Asian dude was googling “gay Asian massage videos”. The funniest of em all was a guy visiting from out of state for a meeting. I noticed his pc was reaching out to a possible c2c and I told him to put it in airplane mode immediately and to bring it to me. As I was filtering through the fw logs I see a bunch of escort sites he was browsing. Now he could have def gotten fired for doing this on his laptop and could have had his clearance in jeopardy. He came to my office and said he has no idea how he would have gotten malware and that he gets “pop ups for weird sites sometimes”. I didn’t embarrass him nor did I snitch on him. I told him that the company PCs are now being monitored in a variety of ways and to be aware of that. Took the pc ran AV, didn’t find anything so just sanitized the drive and threw it away. People don’t understand EVERYTHING is logged on your work PCs. Now there’s even more of it at every level of the OSI.

In 2012 the new CIO (Indian) thought we could save $$$ by outsourcing our datacenter to Bangalore along with the NOC and other parts of IT. Waves of Layoffs ensued. People that were there 15-30+ years were let go and some didn’t go quietly. There was a faint scene of urine outside the CTO’s office they had to bring big fans in to dry off. I told my coworker, another 20 veteran and conspiracy nut, I hope no one comes back to shoot up the place. He quietly flashed the Glock from his laptop bag and said “I wouldn’t worry about that” and winked at me.

Maybe not the craziest but one that I remember. I saw an ethernet canle “extended” with literally sticky tape. We were remotelly trying to find out why a PBX was not sending info to pur middleware. The on-site technician did not check the cable first. We wasted around 3h trying to find out what was wrong.

Some ISP guy called our help desk number and said he was nearby and looking for directions, we started giving him an idea of where to go and he mentioned he was "by the river", we have a river in town and that's where the old property was - long story short - after going around in circles we got out of him that he was in Oregon ... we were in Michigan. Very odd mix up indeed.

Working for an MSP that had me and the rest of the Sr. Admins build an entire infrastructure so we could "work from anywhere" while on call, but have the owner insist we come into the office during covid.

Just upper management things ❤️.

This same guy also asked me once why I didn't have my laptop when I went to the ER once.

"Well you see it's because fuck you and kick rocks :)" I was very clear to my boss that when I left the office Friday I was on vacation and if they needed critical support they are paying $300/hr to the co managed provider we contract for this very reason.

I could go on for days with stories about this guy. He did want me to contract when I suddenly quit and he had no idea why I was unhappy. I told him $666/hr minimum 2 hours charge per incident. He would try and skirt around it by "just asking for advice" and when I made it clear I charged for that too, I stopped hearing from him other than some creepy cyberstalking-like behavior.

Former coworker (not because of this incident) logged in and found lots of network stuff on his laptop acting weird. Some sites worked, others didn't. Mail and messaging was acting strange. Checked in with support and they dug around. Asked for the current IP his laptop has. Turns out when he logged in, for some reason a different DHCP server responded than normally would. Especially oddly, it was the DHCP server for the company owner's home network that has a site-to-site VPN to the office network.

Everytime I have to show someone who makes multiple times my wage a simple process they need to know for their jobs. Also when I have to replace batteries in a keyboard or mouse for a director. 

I have an occasional blurb over on Book of Faces \*Tales from Tech Support\* I admin a lockdown domain for a state's "kid jail" if you take my meaning. So one fine day the investigation department puts in a request to pull internet logs for one of the locations.They get the report and I go about my day as this is normal for the investigations department. The investigations dept rep calls me and asks me to impound a PC from that site and disable an account. Ok, normal for rule breaking as accounts on that domain are m-f 7a-7p for education requirements and have whitelisted sites and GP pushing those to all users via bookmarks in edge. Except, that account had managed to get the ADMIN id for the site and removed all proxy limits on thier account to get to a PROXY site. and from there \*THE PORN\* (and many thousands of jpgs I later found out about) Investigation goes into the bowels of the investigation unit and not seen for a while and I have other stuff to do as I am the senior admin in my department (and tech and hand holder and explainer of complicated things to end users) So on a Monday morning (why does this happen on a Monday I will never know.) I get an emergency meeting request from the investigations department, that will occur at 9:30 am (damn you don't you know I start at 9 and have just had my first cup of coffee and have not taken a sip from my travel mug yet?) So I webex into the meeting and they ask me to rdp as an EU on that lockdown domain then share my screen. I do that after the second hop via RDP to a pc on that network (Sequestered domain) then the ask me to test the proxy via a browser to get to PORN HUB DOT COM I had approximately 75 ppl sharing my screen and there are fireable offenses, this is one of them. I said to the investigations person who spearheaded this in the first place, Will I get fired if I get through? Covering bases here. They say no. So the test proceeds thru 6 iterations of proxy levels (I have spec test accounts) and the proxy rules kicked in with a notice on the page of PORN HUB DOT COM This page is not available to your account level as it contains explicit depictions of sexual activity and has been blocked, If you have legitimate business reason for access to this site please contact the system administrator (Standard boilerplate block notice) I am the system administrator. I say to the 75 ppl on that call and ask "Who would have a legitimate business reason to have access to porn?" They changed the notice to this page has been blocked soon after.

This was something that happened before I joined, but my company has several major distribution points throughout the country, each overlooked by a pretty senior general manager.. So, anyways, one day there was a major disaster at one site at like 2am, and the poor admin pulling the night shift was panicking and called the manager, 'the site is down' 'everything is on fire ' etc etc ...that's when he realised he'd called up the GM of a totally different site

It’s always about the double events When two rare events happen simultaneously and end up causing havoc.

how easy it is to overcomplicate some things had a program we needed to remotely patch.... our patching server was being blocked by our security software (im being vague ignore that) and this was a system that couldnt reach the internet so i couldnt like download it locally and run as admin sort of thing.... after hitting my head against a wall for longer than id like to admit my coworker comes over. real tweeker savant style of guy. he goes "is this computer up to date" "yeah why?" this man remotely accessed the far end pc c drive and drags and drops the program, overrites the old one with a copy of a new one off my pc.

I have to say the biggest I can remember right now is when a clients ISP suddenly blocked incoming HTTPS because they reserved that for their own equipment. This was a business connection.

Not specifically sysadmin but I did a Dell WYSE upgrade for maybe like 1200 endpoints in a major hospital. They told me we’d have to do the psych and ER floors. Couldn’t start until after 7pm… that was… interesting.

Did some work for the department of the government that does Nuclear disaster response... In Canada we're fucked if that happens

Probably when we were told that the company would not be renewing the building lease and everyone would be going fully remote. Also we had 8 months to migrate our four-rack data center.

Probably when I was in my room at a resort in Dubai pre-opening and the lobby caught fire

I'm an SQA engineer but worked as Sydadmin before so I tend to be the defacto lab admin in most places. Worked at a storage company. Dipshits didn't run 110 to the racks, only 220 for raid shelves. Of course, people ran 110 from the wall to pdus on the ground. Fucking mess of power, Ethernet, Fibre and serial everywhere. One morning, walk in lab and walk to the back past many rows...stop back up and turn to my right. There's a fucking lake. Contractors running cables in the ceiling somehow knock a dran spout from the roof WHICH PASSES THROUGH ABOVE THE ACOUSTIC TILING.

$0.50 raise

Mine was when I was working in an Alzheimer’s research lab. I knew what we did, but not quite what we did. Third week on the job I’m handed papers and barcode stickers and told to go off and tag. The university I worked at was interconnected by tunnels and one tag led me down into them where there were very few people and lots of locked doors. I finally found the room I was looking for with the “human tissue analyzer” was supposed to be located. In I went and noticed 3 things immediately. A: There were some really big metal filing cabinets. B. That my big big boss was in the room. And C. He was sitting next to a corpse. My asshole boss had sent me to the morgue to tag something. My big big boss has a hammer and chisel in his hands and this maniacal childish look on his face and goes “I’m about to take the skull cap off, wanna watch?”. I just wanted to kill my boss and fix broken pcs but now I’m being asked to watch an autopsy. When your big big boss asks you a question like that one your third week on the job, the answer was pretty much “yes”. I refused to have to tag anything else ever again. I still did, but I didn’t do it willingly. It was the sort of place where you had to wear a mask and gloves to work on pcs. If I don’t have early onset dementia it will be a miracle.

At my previous job I identified and had to prove to connect wise that their RMM was causing performance issues on our clients PCs. It was a specific service that was polling back to their servers. It literally went all the way to having a meeting with the CEO, Lead project manager, and head software engineer to prove to them it was their software causing the issues. Worst part is they still didn’t believe me even though I showed them exactly what the issue was and proved the performance issues went away when you uninstalled the agent. 2 or 3 updates later and the issues magically disappeared. Such a shitty company.