This article was originally written by Dan Tynan for CIO on December 11, 2017.
Everybody makes mistakes. Most are harmless, some are embarrassing but forgivable, and some can take your career — or your company — down with them.
Some of the most common IT gaffes include becoming trapped in a relationship with a vendor you can't shake loose, hiring or promoting the wrong people, and hiding problems from top management until it's too late to recover.
When you're in charge of enterprise technology, the risks are much higher and the fallout from mistakes can be much worse. So we've ranked them by order of severity: Level 1 (an embarrassing story you'd tell over a beer, but maybe not right away); Level 2 (one you can recover from, but don't expect to be on the fast track for promotion); and Level 3 (you're fired).
Here are the biggest mistakes you're likely to make — and how to either avoid them or recover quickly.
IT management mistake No. 1: Vendor lock-in
Severity Level: 2
It's a form of seduction. Vendors lure you in with low prices and endless promises. But once they've got you in their grasp, they never let go.
"Almost every vendor that makes a product is trying to land and expand inside your environment," says Andrew Howard, CTO at Kudelski Security. "IT managers start out with good intentions, but before they know it, the vendor is not replaceable and has significant control of IT assets and tremendous pricing leverage. I have seen several IT managers lose their jobs over this type of vendor mismanagement."
There are some benefits to lock in, Howard acknowledges. Aside from volume discounts, getting multiple products from the same vendor should ensure smoother integration between them as well as tighter security (emphasis on should). And it means you have fewer vendors to deal with. This can be ideal for smaller organizations.
But when you decide it's time to move on, don't expect the vendor to help you. Howard recalls when he was working at a consulting firm, a workflow management vendor tried to keep his firm from transitioning to a different vendor by refusing to hand over its source code. That requirement, which was in the original software licensing agreement (SLA), somehow managed to evaporate in subsequent negotiations.
Going to the cloud doesn't make it any easier, he adds.
"A lot of our partners are having the same kind of problems with platform-as-a-service providers," he says. "Once you're invested with one, it's hard to transition that infrastructure to a competitor."
For that reason, Howard says he knows many CIOs who are hedging their bets by partnering with multiple cloud providers and developing strong technology management practices. He adds that IT managers need to work more closely with procurement to avoid becoming too dependent on any single vendor.
"I personally believe in diversification of technology," he says. "And the cheapest option is often not to your favor. Sometimes the short-term pain may benefit you in the long run."
IT management mistake No. 2: Treating the cloud like it's an extension of your data center
Severity: 2
In February 2016, Best Egg Personal Loans migrated from a VMware-based private cloud to a public cloud running on Amazon Web Services. The peer-to-peer lending service had spent months planning, configuring, and migrating lower-level services, and had done a 1:1 mapping between its servers and those in AWS. Everything was ready to rock. Then, two hours after going live, a critical AWS server died.
"That was a bit of a first-day wake-up call for us," says Brian Conneen, CIO/CTO of Best Egg parent, Marlette Funding. "Turns out the stability of a single cloud server is actually lower than a privately managed server or virtual machine. The 99.999 percent uptime of the cloud comes from the ability to dynamically provision new servers to replace those that have failed."
The crash happened over a weekend, and Best Egg was able to recover without any customer downtime, but Conneen had learned a valuable lesson: You can't treat a cloud server like it's just another machine in your data center.
After that, Best Egg's No. 1 priority was making sure it was optimized for a cloud environment. The No. 2 priority: keeping a close eye on cloud costs.
"You can provision a server whenever anyone wants one," he says. "Do that enough and pretty soon you're spending two or three times the amount you had planned."
Conneen also learned that servers are disposable: when they break you just throw them away. So Best Egg built in a lot more redundancy, assigned a pool of servers to systems that could not afford one moment of downtime, and created scripts that automatically spin up new servers when one misbehaves.
Now when Best Egg issues a new software release, it simply builds new servers, pushes the code to them, and spins down the old ones.
"The benefits of public cloud can only be realized when you design your infrastructure to the public cloud's strengths," he says. "Just migrating your servers to the cloud is not enough, you have to also migrate your thinking and approach."
IT management mistake No. 3: Over-engineering the business case
Severity: 1
It's been drummed into CIO's heads for so long it's like part of their cerebral cortex: to get approval for a big IT spend you need to build a solid business case. So managers can spend weeks researching options, crunching numbers, and assembling PowerPoints.
But unless you've got a business leader willing to take a bullet for your proposal, it's often all for naught, says Mark Settle, CIO for Okta, an identity-as-a-service provider.
"Several years ago I was interviewing for a CIO position with a Fortune 200 company and giving a mini-lecture to the CFO about the importance of business cases," he says. "The CFO told me he didn't believe any of the numbers in a business case, and only approves major IT initiatives when there's a business leader committed to leveraging the new capabilities."
Necessary spending on infrastructure or compliance are rare exceptions, Settle adds. But any kind of strategic IT initiative requires a passionate champion on the business side. Earning the trust of executives means not only doing your IT job seamlessly, but also partnering with other teams across the organization, taking their existing processes and improving them. Do that, and business leaders are far more likely to listen to your ideas when a strategic opportunity arises, he says.
"If you're bringing in a lot of over-engineered business cases without executives willing to stand up and fall on their sword for you, you're making things a lot harder for yourself."
IT management mistake No. 4: Hiring below your skill level
Severity: 2
It requires a team to build a successful enterprise, but it only takes one incompetent employee with a bad attitude to bring everyone and everything down.
"The biggest mistake IT managers make is hiring people who aren't smarter and better than they are," says Derek Johnson, VP of business development for recruiting firm Stride Search.
Unfortunately, managers' egos often keep them from picking the right person, he says. For example: Three years ago Stride Search had identified the perfect networking and software engineer for one of its clients, a SaaS startup. He was eloquent, charismatic, had a PhD in computer science and owned several patents. Everyone loved him — except the startup's CTO.
"The phone interview went well, but the in-person interview was an absolute disaster," Johnson says. "The CTO, who was both co-founder and hiring manager, spent the entire interview insulting the candidate and trying to one-up him. The rest of the management team wanted to extend an offer, but the CTO refused. The candidate ended up working for a competitor, which later crushed that startup. This happens so often it could be a parable."
Short of a performing an involuntary ego transplant, companies can combat this problem by requiring that no single person has the power to veto a hire, says Johnson. For top positions, a company's board of directors and that candidate's subordinates should also be involved.
"The famous saying, 'A players hire A players, while B players hire C players' really applies," he says. "There's nothing more catastrophic to an organization than hiring the wrong key person, or passing up the right one."
IT management mistake No. 5: Promoting the wrong internal candidate
Severity: 2
If failing to hire the right outside person is a mistake, so is promoting the wrong internal candidate.
Generally speaking, promoting from within is an excellent policy, notes Giancarlo Di Vece, president of Unosquare, an IT software company. But you need to do it for the right reasons.
The wrong reasons? Promoting someone to reward them for being a loyal employee, give them a career path, or make yourself feel like a good manager, says Di Vece. This can blow up in your face, especially if the employee isn't really suited for the new job.
"I have seen IT managers make a good developer a tech lead, and then that employee gets frustrated and eventually quits," he says. "You think you're being a great boss by giving people the opportunity to climb, and you end up losing them because you took them away from what they actually loved to do."
Di Vece says this happened to him about a year ago. He'd hired a rock-star developer for one of Unosquare's biggest clients and put him on a fast track for promotion. Soon the developer was managing a team of five.
Everything went well for three months, until the day he walked into Di Vece's office and resigned. Even though the team's work was excellent, the developer felt he had failed in his job, and could not be persuaded to stay on in his old role.
"I lost a fantastic programming resource because I thought I was providing upward mobility to his career," he says.
Since then, Di Vece says he's set up a framework where newly promoted candidates can provide and receive regular feedback, and supervisors can keep a close eye on how they're doing to help them be successful.
And while he still feels promoting from within is a good philosophy, Di Vece says it's not the right call in every case; managers need to choose their internal candidates wisely.
IT management mistake No. 6: Applying agile methodology to core systems
Severity: 3
With the explosion of cloud services and increasing demands for business velocity, CIOs understand that a lot of their organization's IT is out of their control.
But the same agile delivery mechanisms that allow companies to spin up Docker containers and micro-services in the cloud can have a disastrous impact on core IT systems that are the CIO's responsibility — like email, phone services, ERP, and back-office applications, says Kudelski's Howard.
"I've seen more CIOs lose their jobs because they couldn't keep email up than any other single issue," he says. "These agile methodologies often fly in the face of strong and rigorous change control that's necessary for core systems. If they go down, businesses can lose money quickly."
To mitigate this issue, CIOs need to draw strong boundaries, allowing agile change on business systems and enforcing more rigorous change control on core systems, says Howard. One size will not fit all.
"You cannot have things that are being touched by agile delivery mechanisms put your core services at risk," he says. "The challenge is that drawing and enforcing those lines is not always easy. Business demands velocity but prudence demands slow, methodical change. Those two things fly in the face of each other, and often the IT manager is caught in the middle."
IT management mistake No. 7: Saying yes too often
Severity: 2
Top IT managers are often accused of putting the “no” in innovation. But a bigger problem is when they don't know how to turn people down, and risk losing control over the security of their systems, says Richard Henderson, global IT security strategist at security firm Absolute.
"How many times have IT or security people gotten a call from someone in a high position demanding access to something risky?" he asks. "How often do business units inside the organization 'go rogue' and deploy a shiny new cloud-based tool or service without proper vetting or approval of the IT or security teams?"
Henderson says tools like cloud storage and SaaS solutions can offer huge benefits to teams. But when IT managers approve every single exception request, they create new holes and blind spots in their organization — and, potentially, new vulnerabilities.
It's hard to turn down the CEO when he makes a special request, Henderson acknowledges. But you need to have a plan in place to deal with the (hopefully) rare exceptions. A solid asset management scheme is essential, as is software that watches end-point devices and alerts you when users sign on to common cloud services.
Saying yes too often makes it impossible to keep everything patched and in compliance, especially if you're not in the loop when someone in marketing spins up a new AWS instance.
"The name of the game isn't to stop people from using these things, it's to empower teams who've decided to use these services to meet the minimum requirements for data protection," Henderson says. "CIOs need to say, 'Look, if you want to do this we'll help you, but here's the baseline of security you need to provide.'"
IT management mistake No. 8: Hiding problems
Severity: 3
When a big project starts to go south, a lot of IT managers try to bury the problem, hoping to fix it before the bosses notice, says Okta's Settle. Things usually just go downhill from there. By the time they finally get around to admitting that the new code release brought the whole system down for 48 hours, or they need another $4 million to complete the project, they've lost credibility.
"The sooner you expose bad news, the better," he says. "Because bad news never gets better by itself. And the sooner people start dealing with it, the more likely you can recover the project and get back on track."
Delivering bad news is never easy, but it will go down a lot smoother if you've established and maintained a good working relationship with business leaders.
"Rule No. 1 is that you never show up on an executive's doorstep for the first time when you're asking for money or seeking forgiveness," he adds. "That applies whether it's really the first time or simply an executive you haven't interacted with for six months."
Managers need to create opportunities to talk with the CFO and other business leaders when they're not in crisis mode. That's not always easy for tech-oriented people, but these are skills they need to develop, Settle says.
"Even if it's nothing more than a little social banter, and talking to the executive about the issues they're facing, that makes it easier to have the harder conversation when things go wrong and you need to put your hands in their wallet."