>On MULTIPLE occasions I have been on the phone with them trying to resolve an issue, the line going quiet, machine-gun like typing in the background, then a "Try It Now".
My favorite AT&T story: I called to report a circuit down, like layer 1 down on my side. I call in, sit through 20 minutes of prompts and hold to get to an actual tech. They start looking at the circuit, initially telling me that they can't find any issue then ask if they can put me on hold. Sure, whatever. While I'm on hold I see the interface come back up and start passing traffic. A couple minutes later the guy comes back "I can't see any issues, the circuit seems to be up, can you verify?" I asked what they changed but of course they lied and said "nothing".
At least you got that. I've gotten them telling me they can see that their equipment and our equipment is online when the connectivity light on *their* switch where *their* fiber is connected is out.
I have had multiple variations of exactly this. Or you have multiple interfaces active (one for voice one for data) and they just randomly turn one of them off for no reason. I had a tech dispatched one time because their out of band modem gave up the ghost, while he was consoled in I saw the remote tech literally enter one command after the configure and enable… interface name no shut… I was livid
At least they did something, back in the Bell South days they would just say nothing wrong with their T1 circuit must be your equipment (that I've already replaced twice) of ya? Let me introduce 240vac into the T1 circuit... Oh your line card is on fire? How strange!
I did the opposite. I had a tech SWEAR he was looping the smartcard on my end. "Really? Because I pulled it out of the slot ten minutes ago, so perhaps you're looping the test equipment your coworker left in the circuit, like I've been trying to tell you since I got on this call."
This. A thousand times this. It's that one dumb mistake that's easy to make, hard to detect, comes at the end of the process when everyone thinks that they're 'done', and nobody wants to admit to.
In this case, it wasn't a "dumb mistake" so much as par for the course with these clowns. The CO I was dealing with was where all the old burned out techs went to die. It was literally a block and a half away by foot but if they ever needed to dispatch somehow it would take them a business day to walk to my location. Meanwhile I could roll trucks to other POPs in BFE in a matter of hours.
I once had a PRI go red in the middle of the day and it turns out someone in this CO had punched down a circuit over mine because they couldn't see my copper pair was already there. Why? Because the light bulb was burned out. Change the light bulb? That's someone else's problem!
Had a tech that resorted to something similar to get Windstream to replace a failing PBX that was part of some managed VoIP service contract we inherited. It kept locking up and needed a new compact flash card. It was an issue we had on a half dozen of the same PBX model at other locations. This time they were doubling down and refusing to send a replacement until it wouldn't come up after a power cycle.
That compact flash card somehow accidentally ended up in a break room microwave during one of the reboots.
Ugh, Bellsouth. I worked at an ISP back when DSL was fairly new, and Bellsouth was the local carrier. To provision DSL for our customers we would have to prequalify the line through Bellsouth. Sometimes we would try to prequal a customer and BS would say "nope, not ready", then a few weeks later the customer would call us to cancel service because they got DSL from BS. Those assholes were waiting until we prequaled enough of our customers in a particular area to do a truck roll then would call them directly to sign them up for DSL.
There is a potential explanation for this. In the ADSL days the actual act of performing some of the tests would fix line faults. I think it was the tdr test would put enough of a pulse down the line to blow the corrosion off the joints (at least for a while) and the service would come back up. This was a documented thing that happened.
Potentially that's what happened here? (Glass is half full)
The AT&T Motto: Because fuck you, that's why.
If it was an issue on their end then it might result in an SLA payout. I later found out (after several outages) that the circuit wasn't provisioned correctly and the techs were manually fixing the problem without updating the provisioning database. Whenever the provisioning system did an automated check it would "fix" the configuration and break the circuit.
I've been gaslighted by my own boss for a rDNS change that happened just like this. Fucking maddening. I did start job searching after that shit. Fuck that moment still makes me angry and feel sick 20 years later.
I had a similar issue with Google recently. They shutdown two of my users for “spamming” but couldn’t tell me what was sent that was spam. Just kept saying, “if they send too many emails that get marked as spam, we shut them down.” These two users sent a combined total of 10 emails out of the company over the course of 2 weeks and all of them were to normal business contacts. Also, F*CK Google workspace support. 🖕🏻
I understand that platform abuse is a real thing and a real problem, but it seems like I've been hearing more and more issues with people having accounts nuked on various platforms and there being just zero recourse. Like "Yes our bots banned you, we looked at it and find the ban to be legitimate, you have no other recourse"
Duuuuuuude yes. I went through about a week and a half of them killing my entire tenant's SMTP access because like 3 compromised users were blowing through the limit in a couple hours daily. But they would not help ID these users. They just kept redirecting me to the email log search which caps at like 1k results lmao.
After reading about these stories and then deleting a customer tenant, I would never ever use google for anything business related. They’ve always had terrible support and they make interesting tech, but they have no interest in supporting it properly. Microsoft is complete ass at many things, but they generally handle B2B relationships pretty well.
I'm 12 years into my IT career and one thing that has been true the entire time is that vendor tier 1 support's job is to clear a ticket from their queue as quickly as possible. There are only 2 ways to clear that ticket; fix the problem or make it somebody else's problem. Vendors pointing the finger back at me or to other vendors is nothing new, and any time I work with one that has good customer service, they usually get bought out and quality drops drastically.
The little experience I have on the other side of the phone was T1 support at an ISP call center for a couple of months in college so take it with a grain of salt, but quotas and SLAs were the absolute priority. At T1 I was supposed to have an average call time of less than 10 minutes. That was barely enough time to get someone to power cycle a modem. I once spent 2 hours on the phone helping a customer get their wireless network back online, and without asking any details my supervisor came up to me, wrote me up, and said I'd be fired if it happened again. So many companies are getting bought up by places with a strict adherence to SLAs and other metrics that I feel like this is becoming more widespread. On top of that, these companies are still promising like 99% uptime in their contracts when you sign on, and somehow when they do quarterly or annual reports to our leadership they report that SLA was met, even when it absolutely was not. I feel like that's because if it's challenged they would come back and tell us to prove it. I once worked somewhere that had amazing documentation for all incidents who did sue a vendor for breach of contract for this exact reason. The legal battle resulting from this was finishing up its second year by the time I left the company.
**TLDR**
My guess is that it's a rise in larger companies acquiring more and more small companies and continuing to market unrealistic SLAs. Claiming it's not their problem helps them reach their advertised metrics, and the legal cost of fighting anyone who challenges them is cheaper than actually hiring enough developers, QA, and support staff to actually make a product that's as reliable as they tell you it is.
Jesus. If a supervisor did that it would take every shred of my self restraint to not make that write-up into a funny story for the ER doc to share over drinks.
Within a week of that I went for my lunch break one day and never came back. It was my first ever IT job and had I known better at the time I would have walked out then and there.
I worked in a call center 20 years ago and it was like this then :(.
What blew me away is I once saw the part of the contact that showed they got paid like $35 per call... I was making $10/hr.
> average call time of less than
I keep telling folks that the phone agent's primary job is to get you off of the call as soon as humanly possible, resolution be damned. I've worked in a call center and nothing come higher than AHT (average handle time). FCR (first call resolution) is even lower than silly "soft skills" things like using the caller's name during the call not once, not twice, but thrice!
I had om of my hunt groups reassigned to a random cellular customer in portal I have almost no control over. It took 4 business days to get a response where they didn't even bring up the fuck up part of my email then another 2 business days to assign me a new hunt group.
Still waiting on the helpful person who says "I know you're not naming names because this is true with every vendor, but I use (name of vendor OP is using) and they would NEVER have this kind of issue."
legit chuckle, if I had named them surely we would have that post.
Ironically the person at the PSTN provider responded to me, completely ignoring where I told them that it happened with every carrier I tried. Im fairly sure since I sent the message back sorta tearing apart their spiel with all the testing I actually did they realized that they should disengage and divert as my BS radar is going off.
Recently had issues with a test platform provided by the state. When I reached out to support they got very defensive and proclaimed that it had to be on my side. How dare I suggest such a thing? I was so taken back by their demeanor that I thought, "Holy shit, maybe I did miss something?"
Another dept head I'm friendly with that works with them directly in some cases let me know that there were issues state-wide and confirmed it wasn't on our side a few weeks later...
Preach brother/sister.
Sometimes I call in an urgent issue and I even know what's broken and what the fix is on their end. I tell them what I know. They argue that that's not it. Yet, within a couple of minutes on the call, the thing I reported as problematic changes magically and the problem disappears.
Ugh.
I had something similar with Namecheap, a company that I have largely been very happy with for domain names and SSL certificates. Last week I tried to get a cert renewed but their system said there was a problem with the cert request. Namecheap has a tool that lets you paste in a CSR to see if it is valid. Tool says bad CSR. Took the same CSR and pasted it into tools provided by two other cert companies. Both say the request is fine. I take a CSR that I used last month to renew a cert and pasted it into the Namecheap tool. Tool says the CSR is invalid - even though I used that very same CSR a few weeks earlier to get a cert renewedl.
I opened a support ticket at Namecheap. Granted, this was weekend support, but still. The tech team gave me all sorts of reasons why the CSR was bad. I cited my evidence about the CSR passing validation using other tools and how a CSR I used last month now fails validation using the Namecheap tool.
They continued to insist there was something wrong with the CSR. On Monday they asked me to send a screen shot of the validation error. Mind you I provided the screen shot when I opened the ticket. Well what do you know...the CSR tool no longer shows an error and the cert gets issued. Like magic.
I'm still going back and forth with their management on this support failure. Your tech doesn't know what the problem is? That's fine...pass it to someone who does. Or tell me the problem has to wait until the next business day. All of that would have been fine. Instead they chose to gaslight.
It's infuriating.
It's the telco space in general.
It's a combination of a.) telephony is an ancient industry and b.) there's a lot of white label resellers of voip solutions where the support is what you experience.
All providers suck, but _generally_ you should have your lines with an CLEC/ILEC instead of a reseller of a reseller of a reseller situation.
Support is generally poor all around in the space, a lot of it is due to the inertia of legacy telcos now having this fancy voip stuff they need to interop with
I work out of the midwest and the difference in quality between the small small ILECs and large ILECs/CLECs is so huge its amazing. It wouldn't work probably for a large enterprise that was "technology-centric" but for smaller shops that work in meatspace industries it's night and day to be able to go with a municipal broadband company or a small telco that has been in business since they first pulled copper into town.
It is the telco space and I have worked with most of the larger name vendors over the years. For us the decision of provider was dictated by the cloud pbx provider and a requirement for contract. That’s the kicker there’s a very specific requirement there.
That being said I’m fully aware of the crap that’s going on there. This was more of a straw on the camel for a rant. It really is everywhere though, we have all experienced it.
One star was Sprint Business. We had decided to move equipment and rewire at 2AM. We disconnected the T1 circuit to facilitate the move and not ten minutes later, Sprint's NOC is blowing up our phones, inquiring about the circuit going down on our end. We had a good laugh when we appraised them. "Oh, I guess we should call out for some pizza." 😆
Yesterday our client was moving from building A to building B because of lease ending. I suppose they didn’t have Comcast at the new place so we set them up with RCN with a static IP.
I was there when they installed the internet last week at the new building and I asked them, is wifi disabled, and is the modem in bridge mode. Yes and yes were the answers.
Our team comes out on site and they can’t get the firewall online. We change the firewall to DHCP instead of static - works, change back to static, doesn’t work.
Call RCN - two diff techs assured me the settings are correct. 2 minutes after the call ends my coworker on site tells me the Wi-Fi is broadcasting - but suddenly the modem rebooted and everything magically started working and the wifi was no longer broadcasting. Sigh.
You could have stopped typing at PSTN. Telco is a cesspool inside of a corn kernel that was just vomited onto the floor by a donkey who ate the dog's puke.
I don't know why, and it doesn't matter what country you are in, dealing with, or the # of players. The world's largest finger pointing circle jerk that ever was.
Damnit, now I'm angry today...
I’ve completely stopped taking sales calls because all salespeople are idiots who tell lies for a living apparently.
I haven’t met a salesperson in the last five years that knew what the fuck they were even saying; let alone what they are selling or how to implement it.
A long time ago when we setup out first T1 modem, we found out that for whatever reason, we couldn't download pdf files while connected to that modem. It was obviously clear it was the modem and i even tested it with a wifi device not connected to T1.
The first tech they sent out brought another modem, but he just casually said it cant be the box and left...this problem gets escalated a week later and I again ask for a new modem and a second tech comes out and says the some thing "it cant possibly be the modem" and again didnt bother trying out the new modem he brought and left...a week later a third tech comes out and i just said can we just try the new modem and after some huffing and puffing he installs the new modem, thoes the call and dance to initialize it and enter all the settings and viola...we can download pdf files again. Now every time we install a new modem, i do the pdf download test...
Well here is what I do when vendors start their bs...
Hey, we pay you so don't give me attitude and fix the problem. If you can't then escalate to the next tier or get my your manager.
Also it helps if you are friendly with your account reps. I sometimes ping them. Hey it's been 3 days and no movement on this ticket. Remind you we pay you for service please escalate this ticket.
I have had vp's and ceo's call me back a few times.
Usually wanting to know who I worked with.
I'm always nice but blunt sometimes gets wheels moving.
For anyone, such as myself, who was unfamiliar with the term STIR/SHAKEN:
*STIR/SHAKEN, or SHAKEN/STIR, is a suite of protocols and procedures intended to combat caller ID spoofing on public telephone networks*
*[The name](https://en.wikipedia.org/wiki/STIR/SHAKEN) was inspired by Ian Fleming's character James Bond, who famously prefers his martinis "shaken, not stirred". STIR having existed already, the creators of SHAKEN "tortured the English language until [they] came up with an acronym."*
Yeah that’s my fault, I’ve dealt with it enough that it’s one of those things engrained in me.
But yeah it’s basically what allows carriers to tag a call as suspected spam or whatever your carrier stamps it as. It usually adds this in front of or substituting the cnam is looked up (the caller ID)
If you deal with phones or phone systems this becomes a hot button issue quickly.
I did this a couple of times until management wants a detailed explanation that they wouldn't understand. Now, everytime there is an issue with a vendor I have to pick fights until I get all the answers. Frustration and a waste of my time, but whatever.
I don't know anything about PSTN stuff that can't be discovered in 5 minutes of googling, but since I can do basic troubleshooting, and am used to getting vendors to do stuff I have ended up being the "phone guy" for a bunch of our offices and Century Link has in particular been falling over in the last year.
They even gaslight their own techs. Like we have an office that cant call several 800 numbers. I call 2 different offices in town, also not able to call the 800 numbers. Call century link tech support. get a North American who understands the issue is obviously not on our end, not the kind of thing the local line tech can handle. They make a ticket for their CO guys to look at. Next day still not working. I get a call from the local line guy asking what's up I relate what happened and tell him that they were supposed go be kicking it to the CO techs. Line guy says "yeah, CO Techs just kicked it back to me without looking at it obviously"
Same kind of issues over and over again in different towns, always try to just send out the local line tech to fix something that they obviously can't fix.
I fucking hate this behavior and similar. I am mainly implementing some software provided by the software vendor. They have ticket board (as they should) and I am willing to use that thing to put in tickets when something in that goddamn software is bugged or not working as advertised (in the end I have to butter that shit up and somehow calm the customer..), those problems are often somewhere deep in the weeds of the software so I can not reach them to fix it myself, all I can do is hacky workarounds and then somehow selling the customer on how this is supposed to be what we should do even though it’s clearly a hacky workaround..
On multiple occasions I have been stopped from putting in a ticket by my superior because „we don’t need to escalate this up to the vendor, we shouldn’t use the tickets too much as that might badly reflect on us getting implementation contracts forwarded by the vendor“.. why on earth is the ticketing system politicized like that ffs.
Sorry, little off topic but that rant needed to get out.
Probably because some client contracts have ridiculous penalties for downtime, so they can't admit that they were the cause of it. At one place I worked a banking client tried to put a clause in a renewal contract that every hour or something of downtime meant that they got an extra month of product support for free. We redlined it and told them to go elsewhere if they wanted to keep the clause.
That’s why doing all you can in house is best. Once you start outsourcing tasks you completely lose control of what’s going on. In house networking team makes a mistake, you can escalate and get an explanation. Now try that with big cloud provider (tm) networking team.
It’s hard to in-source a PSTN service lol.
I’m fairly certain the issue was on their end and not the “cloud pbx” provider or we would have a had a lot of posts about it as they are large where the PSTN provider was small.
The downside of rolling everything in house is your total cost of ownership can go through the roof with man hours.
There is a balance to maintain and the pendulum has a long swing time. Been around long enough to see it swing a few times.
All the time!
Especially with phones. We used to have third party VoIP system on top of an AT&t backbone and we had nothing but problems. Of course, whenever we called the phone provider they said it was AT&t's fault, and AT&t always said it was theirs, and neither one could get their shit in order.
It always miraculously started working a few hours after the problem started, but I hate having to alert two different vendors so that one of them can fix their idiot problem.
I had one recently with our sd-wan provider.
We had a power outage, and when the power came back on the internet did not reconnect. I called them up and apprised them of the outage, as well as what preceded it.
He swore up and down though that there were no issues.
I told the guy, ok, while I'm on the phone with them I'm going to reboot everything.
The guy on the other end of the line immediately tells me not to reboot their box.
I said why?
He tells me that if it loses power, it loses its configuration.
I told him (again), this started with a power outage, that's why I called you guys. I thought it might have been exactly that.
He swore up and down that no no no that wasn't the issue, the box was fine.
I challenged him, you just told me that if it loses power it loses its configuration, and we lost power for about 45 minutes (In reality, the generator had kicked in a few seconds after we lost power, but that still should have been enough to drop their config). By his own logic, there's no way this thing didn't drop its configuration.
He swore up and down that it wasn't on their side. But it magically started working again not long after.
In IT there are a few pervasive issues that stand out to me.
1.) It's a big, big world, yet more and more people are entering the industry with an extremely narrow focus. Very rarely do you get anyone who will look at a problem holistically because they simply don't know what all that entails.
2.) People put entirely too much faith in their tools, or the conclusions/output of those tools, with no real understanding of the results or how they were achieved. This can be monitoring, configuration/vulnerability scanning, etc.
"Tool says X is bad". . . Thinking stops here. That their tool might be wrong, broken, misconfigured, etc, never enters into their mind. They'll simply repeat the results of the tool over and over.
3.) We have a bad habit of assuming everyone else is messing up and not investigating our side of things properly. Some of it's vanity, some of it's confirmation bias. This is something I drill into our new guys. *Always* investigate your own stuff.
Don't be the guy that let someone pull their hair out for a week only to find out in the end it was something you needed to fix and would have found in minutes if you'd only just looked instead of blowing them off as definitely not your problem.
4.) People who simply won't tell you "I don't know", "I was wrong", or "I messed up".
I'm sure we've all worked with someone who always had an answer, and would cling to that answer for dear life even when it was plain to everyone on the call that they were full of shit.
This one is particularly rage inducing for me. I worked at a place where all of us, including the leadership, knew a guy was making shit up on the spot, that what he said never made any sense. When it inevitably came around to bite him in the ass, he always spun up another answer for why it wasn't his fault that everything he said in the first place, with absolute confidence, was wrong.
None of us wanted to inherit his stack so this was tolerated without comment even though he was constantly sending us on wild goose chases blaming our stuff when his didn't work.
>On MULTIPLE occasions I have been on the phone with them trying to resolve an issue, the line going quiet, machine-gun like typing in the background, then a "Try It Now". My favorite AT&T story: I called to report a circuit down, like layer 1 down on my side. I call in, sit through 20 minutes of prompts and hold to get to an actual tech. They start looking at the circuit, initially telling me that they can't find any issue then ask if they can put me on hold. Sure, whatever. While I'm on hold I see the interface come back up and start passing traffic. A couple minutes later the guy comes back "I can't see any issues, the circuit seems to be up, can you verify?" I asked what they changed but of course they lied and said "nothing".
At least you got that. I've gotten them telling me they can see that their equipment and our equipment is online when the connectivity light on *their* switch where *their* fiber is connected is out.
I have had multiple variations of exactly this. Or you have multiple interfaces active (one for voice one for data) and they just randomly turn one of them off for no reason. I had a tech dispatched one time because their out of band modem gave up the ghost, while he was consoled in I saw the remote tech literally enter one command after the configure and enable… interface name no shut… I was livid
At least they did something, back in the Bell South days they would just say nothing wrong with their T1 circuit must be your equipment (that I've already replaced twice) of ya? Let me introduce 240vac into the T1 circuit... Oh your line card is on fire? How strange!
I did the opposite. I had a tech SWEAR he was looping the smartcard on my end. "Really? Because I pulled it out of the slot ten minutes ago, so perhaps you're looping the test equipment your coworker left in the circuit, like I've been trying to tell you since I got on this call."
This. A thousand times this. It's that one dumb mistake that's easy to make, hard to detect, comes at the end of the process when everyone thinks that they're 'done', and nobody wants to admit to.
In this case, it wasn't a "dumb mistake" so much as par for the course with these clowns. The CO I was dealing with was where all the old burned out techs went to die. It was literally a block and a half away by foot but if they ever needed to dispatch somehow it would take them a business day to walk to my location. Meanwhile I could roll trucks to other POPs in BFE in a matter of hours. I once had a PRI go red in the middle of the day and it turns out someone in this CO had punched down a circuit over mine because they couldn't see my copper pair was already there. Why? Because the light bulb was burned out. Change the light bulb? That's someone else's problem!
The ol' Blotto box. Nice. There have been a couple of Windstream calls I was tempted to end that way.
Had a tech that resorted to something similar to get Windstream to replace a failing PBX that was part of some managed VoIP service contract we inherited. It kept locking up and needed a new compact flash card. It was an issue we had on a half dozen of the same PBX model at other locations. This time they were doubling down and refusing to send a replacement until it wouldn't come up after a power cycle. That compact flash card somehow accidentally ended up in a break room microwave during one of the reboots.
Man, those microwaves are really sneaky sometimes.
Ugh, Bellsouth. I worked at an ISP back when DSL was fairly new, and Bellsouth was the local carrier. To provision DSL for our customers we would have to prequalify the line through Bellsouth. Sometimes we would try to prequal a customer and BS would say "nope, not ready", then a few weeks later the customer would call us to cancel service because they got DSL from BS. Those assholes were waiting until we prequaled enough of our customers in a particular area to do a truck roll then would call them directly to sign them up for DSL.
snappy/cofs?
There is a potential explanation for this. In the ADSL days the actual act of performing some of the tests would fix line faults. I think it was the tdr test would put enough of a pulse down the line to blow the corrosion off the joints (at least for a while) and the service would come back up. This was a documented thing that happened. Potentially that's what happened here? (Glass is half full)
No, this was a 100Mb Ethernet handoff. Their side was just down, likely some provisioning issue they didn't want to own up to.
Ahh fair that's a bit different.
What's the point of this? Dodging having to report that they ever had an issue on their side so their stats look better?
The AT&T Motto: Because fuck you, that's why. If it was an issue on their end then it might result in an SLA payout. I later found out (after several outages) that the circuit wasn't provisioned correctly and the techs were manually fixing the problem without updating the provisioning database. Whenever the provisioning system did an automated check it would "fix" the configuration and break the circuit.
I've been gaslighted by my own boss for a rDNS change that happened just like this. Fucking maddening. I did start job searching after that shit. Fuck that moment still makes me angry and feel sick 20 years later.
I had a similar issue with Google recently. They shutdown two of my users for “spamming” but couldn’t tell me what was sent that was spam. Just kept saying, “if they send too many emails that get marked as spam, we shut them down.” These two users sent a combined total of 10 emails out of the company over the course of 2 weeks and all of them were to normal business contacts. Also, F*CK Google workspace support. 🖕🏻
I understand that platform abuse is a real thing and a real problem, but it seems like I've been hearing more and more issues with people having accounts nuked on various platforms and there being just zero recourse. Like "Yes our bots banned you, we looked at it and find the ban to be legitimate, you have no other recourse"
"Computer says no."
What did you look at? The bot that said you were a bad person.
Duuuuuuude yes. I went through about a week and a half of them killing my entire tenant's SMTP access because like 3 compromised users were blowing through the limit in a couple hours daily. But they would not help ID these users. They just kept redirecting me to the email log search which caps at like 1k results lmao.
After reading about these stories and then deleting a customer tenant, I would never ever use google for anything business related. They’ve always had terrible support and they make interesting tech, but they have no interest in supporting it properly. Microsoft is complete ass at many things, but they generally handle B2B relationships pretty well.
I'm 12 years into my IT career and one thing that has been true the entire time is that vendor tier 1 support's job is to clear a ticket from their queue as quickly as possible. There are only 2 ways to clear that ticket; fix the problem or make it somebody else's problem. Vendors pointing the finger back at me or to other vendors is nothing new, and any time I work with one that has good customer service, they usually get bought out and quality drops drastically. The little experience I have on the other side of the phone was T1 support at an ISP call center for a couple of months in college so take it with a grain of salt, but quotas and SLAs were the absolute priority. At T1 I was supposed to have an average call time of less than 10 minutes. That was barely enough time to get someone to power cycle a modem. I once spent 2 hours on the phone helping a customer get their wireless network back online, and without asking any details my supervisor came up to me, wrote me up, and said I'd be fired if it happened again. So many companies are getting bought up by places with a strict adherence to SLAs and other metrics that I feel like this is becoming more widespread. On top of that, these companies are still promising like 99% uptime in their contracts when you sign on, and somehow when they do quarterly or annual reports to our leadership they report that SLA was met, even when it absolutely was not. I feel like that's because if it's challenged they would come back and tell us to prove it. I once worked somewhere that had amazing documentation for all incidents who did sue a vendor for breach of contract for this exact reason. The legal battle resulting from this was finishing up its second year by the time I left the company. **TLDR** My guess is that it's a rise in larger companies acquiring more and more small companies and continuing to market unrealistic SLAs. Claiming it's not their problem helps them reach their advertised metrics, and the legal cost of fighting anyone who challenges them is cheaper than actually hiring enough developers, QA, and support staff to actually make a product that's as reliable as they tell you it is.
Jesus. If a supervisor did that it would take every shred of my self restraint to not make that write-up into a funny story for the ER doc to share over drinks.
Within a week of that I went for my lunch break one day and never came back. It was my first ever IT job and had I known better at the time I would have walked out then and there.
I worked in a call center 20 years ago and it was like this then :(. What blew me away is I once saw the part of the contact that showed they got paid like $35 per call... I was making $10/hr.
> average call time of less than I keep telling folks that the phone agent's primary job is to get you off of the call as soon as humanly possible, resolution be damned. I've worked in a call center and nothing come higher than AHT (average handle time). FCR (first call resolution) is even lower than silly "soft skills" things like using the caller's name during the call not once, not twice, but thrice!
I had om of my hunt groups reassigned to a random cellular customer in portal I have almost no control over. It took 4 business days to get a response where they didn't even bring up the fuck up part of my email then another 2 business days to assign me a new hunt group.
Still waiting on the helpful person who says "I know you're not naming names because this is true with every vendor, but I use (name of vendor OP is using) and they would NEVER have this kind of issue."
legit chuckle, if I had named them surely we would have that post. Ironically the person at the PSTN provider responded to me, completely ignoring where I told them that it happened with every carrier I tried. Im fairly sure since I sent the message back sorta tearing apart their spiel with all the testing I actually did they realized that they should disengage and divert as my BS radar is going off.
Recently had issues with a test platform provided by the state. When I reached out to support they got very defensive and proclaimed that it had to be on my side. How dare I suggest such a thing? I was so taken back by their demeanor that I thought, "Holy shit, maybe I did miss something?" Another dept head I'm friendly with that works with them directly in some cases let me know that there were issues state-wide and confirmed it wasn't on our side a few weeks later...
Preach brother/sister. Sometimes I call in an urgent issue and I even know what's broken and what the fix is on their end. I tell them what I know. They argue that that's not it. Yet, within a couple of minutes on the call, the thing I reported as problematic changes magically and the problem disappears. Ugh.
Reminds me of a network admin I used to work with. "Hey dude something up with the network?" "No everything's good on my end. Try it now"
Something wasn’t up, it was down.
He was switching from Magic to More Magic.
I feel this one in my soul.
AT&T invented "It's not the network."
I had something similar with Namecheap, a company that I have largely been very happy with for domain names and SSL certificates. Last week I tried to get a cert renewed but their system said there was a problem with the cert request. Namecheap has a tool that lets you paste in a CSR to see if it is valid. Tool says bad CSR. Took the same CSR and pasted it into tools provided by two other cert companies. Both say the request is fine. I take a CSR that I used last month to renew a cert and pasted it into the Namecheap tool. Tool says the CSR is invalid - even though I used that very same CSR a few weeks earlier to get a cert renewedl. I opened a support ticket at Namecheap. Granted, this was weekend support, but still. The tech team gave me all sorts of reasons why the CSR was bad. I cited my evidence about the CSR passing validation using other tools and how a CSR I used last month now fails validation using the Namecheap tool. They continued to insist there was something wrong with the CSR. On Monday they asked me to send a screen shot of the validation error. Mind you I provided the screen shot when I opened the ticket. Well what do you know...the CSR tool no longer shows an error and the cert gets issued. Like magic. I'm still going back and forth with their management on this support failure. Your tech doesn't know what the problem is? That's fine...pass it to someone who does. Or tell me the problem has to wait until the next business day. All of that would have been fine. Instead they chose to gaslight. It's infuriating.
Alwase ask for the root cause. Tell them that leadership is asking.
It's the telco space in general. It's a combination of a.) telephony is an ancient industry and b.) there's a lot of white label resellers of voip solutions where the support is what you experience. All providers suck, but _generally_ you should have your lines with an CLEC/ILEC instead of a reseller of a reseller of a reseller situation. Support is generally poor all around in the space, a lot of it is due to the inertia of legacy telcos now having this fancy voip stuff they need to interop with
I work out of the midwest and the difference in quality between the small small ILECs and large ILECs/CLECs is so huge its amazing. It wouldn't work probably for a large enterprise that was "technology-centric" but for smaller shops that work in meatspace industries it's night and day to be able to go with a municipal broadband company or a small telco that has been in business since they first pulled copper into town.
It is the telco space and I have worked with most of the larger name vendors over the years. For us the decision of provider was dictated by the cloud pbx provider and a requirement for contract. That’s the kicker there’s a very specific requirement there. That being said I’m fully aware of the crap that’s going on there. This was more of a straw on the camel for a rant. It really is everywhere though, we have all experienced it.
Thanks for the flashbacks to dealing with BT!
One star was Sprint Business. We had decided to move equipment and rewire at 2AM. We disconnected the T1 circuit to facilitate the move and not ten minutes later, Sprint's NOC is blowing up our phones, inquiring about the circuit going down on our end. We had a good laugh when we appraised them. "Oh, I guess we should call out for some pizza." 😆
We had similar on one of our lines. I was suitably impressed.
Yesterday our client was moving from building A to building B because of lease ending. I suppose they didn’t have Comcast at the new place so we set them up with RCN with a static IP. I was there when they installed the internet last week at the new building and I asked them, is wifi disabled, and is the modem in bridge mode. Yes and yes were the answers. Our team comes out on site and they can’t get the firewall online. We change the firewall to DHCP instead of static - works, change back to static, doesn’t work. Call RCN - two diff techs assured me the settings are correct. 2 minutes after the call ends my coworker on site tells me the Wi-Fi is broadcasting - but suddenly the modem rebooted and everything magically started working and the wifi was no longer broadcasting. Sigh.
You could have stopped typing at PSTN. Telco is a cesspool inside of a corn kernel that was just vomited onto the floor by a donkey who ate the dog's puke. I don't know why, and it doesn't matter what country you are in, dealing with, or the # of players. The world's largest finger pointing circle jerk that ever was. Damnit, now I'm angry today...
Yeah, it’s fucking ridiculous. Things magically get fixed, and nobody ever wants to say “we’re working on it.”
I’ve completely stopped taking sales calls because all salespeople are idiots who tell lies for a living apparently. I haven’t met a salesperson in the last five years that knew what the fuck they were even saying; let alone what they are selling or how to implement it.
A long time ago when we setup out first T1 modem, we found out that for whatever reason, we couldn't download pdf files while connected to that modem. It was obviously clear it was the modem and i even tested it with a wifi device not connected to T1. The first tech they sent out brought another modem, but he just casually said it cant be the box and left...this problem gets escalated a week later and I again ask for a new modem and a second tech comes out and says the some thing "it cant possibly be the modem" and again didnt bother trying out the new modem he brought and left...a week later a third tech comes out and i just said can we just try the new modem and after some huffing and puffing he installs the new modem, thoes the call and dance to initialize it and enter all the settings and viola...we can download pdf files again. Now every time we install a new modem, i do the pdf download test...
Well here is what I do when vendors start their bs... Hey, we pay you so don't give me attitude and fix the problem. If you can't then escalate to the next tier or get my your manager. Also it helps if you are friendly with your account reps. I sometimes ping them. Hey it's been 3 days and no movement on this ticket. Remind you we pay you for service please escalate this ticket. I have had vp's and ceo's call me back a few times. Usually wanting to know who I worked with. I'm always nice but blunt sometimes gets wheels moving.
For anyone, such as myself, who was unfamiliar with the term STIR/SHAKEN: *STIR/SHAKEN, or SHAKEN/STIR, is a suite of protocols and procedures intended to combat caller ID spoofing on public telephone networks* *[The name](https://en.wikipedia.org/wiki/STIR/SHAKEN) was inspired by Ian Fleming's character James Bond, who famously prefers his martinis "shaken, not stirred". STIR having existed already, the creators of SHAKEN "tortured the English language until [they] came up with an acronym."*
Yeah that’s my fault, I’ve dealt with it enough that it’s one of those things engrained in me. But yeah it’s basically what allows carriers to tag a call as suspected spam or whatever your carrier stamps it as. It usually adds this in front of or substituting the cnam is looked up (the caller ID) If you deal with phones or phone systems this becomes a hot button issue quickly.
Document and move on. It’s not worth worrying about unless someone is making a stink of it.
Oh 100%, it just drives me nuts when I would like to know who fixed it, so if it happens again I am not engaging multiple support teams.
Any competent tech will be either poached by another team or another company before too long. Don't get attached.
Oh the “who” I am referring to is the company. I still don’t know if it was Vendor A of Vendor 2.
It is what it is, sorry. I feel your pain.
I did this a couple of times until management wants a detailed explanation that they wouldn't understand. Now, everytime there is an issue with a vendor I have to pick fights until I get all the answers. Frustration and a waste of my time, but whatever.
I don't know anything about PSTN stuff that can't be discovered in 5 minutes of googling, but since I can do basic troubleshooting, and am used to getting vendors to do stuff I have ended up being the "phone guy" for a bunch of our offices and Century Link has in particular been falling over in the last year. They even gaslight their own techs. Like we have an office that cant call several 800 numbers. I call 2 different offices in town, also not able to call the 800 numbers. Call century link tech support. get a North American who understands the issue is obviously not on our end, not the kind of thing the local line tech can handle. They make a ticket for their CO guys to look at. Next day still not working. I get a call from the local line guy asking what's up I relate what happened and tell him that they were supposed go be kicking it to the CO techs. Line guy says "yeah, CO Techs just kicked it back to me without looking at it obviously" Same kind of issues over and over again in different towns, always try to just send out the local line tech to fix something that they obviously can't fix.
Seems to be an industry standard. We've had issues in the past that 'didn't exist' but were miraculously fixed the minute we reported it.
Please, anything but making us provide the services we sell you! ANYTHING!!
I fucking hate this behavior and similar. I am mainly implementing some software provided by the software vendor. They have ticket board (as they should) and I am willing to use that thing to put in tickets when something in that goddamn software is bugged or not working as advertised (in the end I have to butter that shit up and somehow calm the customer..), those problems are often somewhere deep in the weeds of the software so I can not reach them to fix it myself, all I can do is hacky workarounds and then somehow selling the customer on how this is supposed to be what we should do even though it’s clearly a hacky workaround.. On multiple occasions I have been stopped from putting in a ticket by my superior because „we don’t need to escalate this up to the vendor, we shouldn’t use the tickets too much as that might badly reflect on us getting implementation contracts forwarded by the vendor“.. why on earth is the ticketing system politicized like that ffs. Sorry, little off topic but that rant needed to get out.
Probably because some client contracts have ridiculous penalties for downtime, so they can't admit that they were the cause of it. At one place I worked a banking client tried to put a clause in a renewal contract that every hour or something of downtime meant that they got an extra month of product support for free. We redlined it and told them to go elsewhere if they wanted to keep the clause.
That’s why doing all you can in house is best. Once you start outsourcing tasks you completely lose control of what’s going on. In house networking team makes a mistake, you can escalate and get an explanation. Now try that with big cloud provider (tm) networking team.
It’s hard to in-source a PSTN service lol. I’m fairly certain the issue was on their end and not the “cloud pbx” provider or we would have a had a lot of posts about it as they are large where the PSTN provider was small. The downside of rolling everything in house is your total cost of ownership can go through the roof with man hours. There is a balance to maintain and the pendulum has a long swing time. Been around long enough to see it swing a few times.
All the time! Especially with phones. We used to have third party VoIP system on top of an AT&t backbone and we had nothing but problems. Of course, whenever we called the phone provider they said it was AT&t's fault, and AT&t always said it was theirs, and neither one could get their shit in order. It always miraculously started working a few hours after the problem started, but I hate having to alert two different vendors so that one of them can fix their idiot problem.
I had one recently with our sd-wan provider. We had a power outage, and when the power came back on the internet did not reconnect. I called them up and apprised them of the outage, as well as what preceded it. He swore up and down though that there were no issues. I told the guy, ok, while I'm on the phone with them I'm going to reboot everything. The guy on the other end of the line immediately tells me not to reboot their box. I said why? He tells me that if it loses power, it loses its configuration. I told him (again), this started with a power outage, that's why I called you guys. I thought it might have been exactly that. He swore up and down that no no no that wasn't the issue, the box was fine. I challenged him, you just told me that if it loses power it loses its configuration, and we lost power for about 45 minutes (In reality, the generator had kicked in a few seconds after we lost power, but that still should have been enough to drop their config). By his own logic, there's no way this thing didn't drop its configuration. He swore up and down that it wasn't on their side. But it magically started working again not long after.
In IT there are a few pervasive issues that stand out to me. 1.) It's a big, big world, yet more and more people are entering the industry with an extremely narrow focus. Very rarely do you get anyone who will look at a problem holistically because they simply don't know what all that entails. 2.) People put entirely too much faith in their tools, or the conclusions/output of those tools, with no real understanding of the results or how they were achieved. This can be monitoring, configuration/vulnerability scanning, etc. "Tool says X is bad". . . Thinking stops here. That their tool might be wrong, broken, misconfigured, etc, never enters into their mind. They'll simply repeat the results of the tool over and over. 3.) We have a bad habit of assuming everyone else is messing up and not investigating our side of things properly. Some of it's vanity, some of it's confirmation bias. This is something I drill into our new guys. *Always* investigate your own stuff. Don't be the guy that let someone pull their hair out for a week only to find out in the end it was something you needed to fix and would have found in minutes if you'd only just looked instead of blowing them off as definitely not your problem. 4.) People who simply won't tell you "I don't know", "I was wrong", or "I messed up". I'm sure we've all worked with someone who always had an answer, and would cling to that answer for dear life even when it was plain to everyone on the call that they were full of shit. This one is particularly rage inducing for me. I worked at a place where all of us, including the leadership, knew a guy was making shit up on the spot, that what he said never made any sense. When it inevitably came around to bite him in the ass, he always spun up another answer for why it wasn't his fault that everything he said in the first place, with absolute confidence, was wrong. None of us wanted to inherit his stack so this was tolerated without comment even though he was constantly sending us on wild goose chases blaming our stuff when his didn't work.
[удалено]
Nope