The solution to companies’ and organizations’ growing stale, stuffy, stodgy, and outmoded—as J.C. Penney’s undoubtedly was and may well still be—is to grow transformational leaders on the inside, forming a pipeline that reinvents and repopulates the senior ranks of the organization with a constant flow of talented and well- prepared insiders. But since, as I will be the first to admit, so few leaders and organizations know how to do that, all too frequently we get the opposite result from the vast majority of executive transitions and CEO selections: inertia, entropy, and decline.

J.C. Penney (JCP) was certainly sliding rapidly downhill—its revenues had been on the downswing for three years—when Pershing Square’s Bill Ackman offered Apple’s senior vice president of retail operations Ron Johnson the shot of a lifetime to turn a floundering retail icon around. At the time, Johnson’s reputation as a maverick genius of retail was riding high based on his extraordinary success over the previous decade at pioneering the concept of Apple’s retail stores and, in particular, the Genius Bar, under the leadership of the quintessential control-freak maverick CEO Steve Jobs.

Before Apple (AAPL), Johnson had been vice president of merchandising at Target, and the man deemed largely responsible for conceiving and implementing that chain’s innovative and extraordinarily successful “affordable chic” high-design, low-cost merchandising and marketing strategy that so many of its peers have since sought to emulate, with varying degrees of success.

In the wake of Steve Jobs’ death, precisely what role Johnson played in the success of Apple stores in the malls remained murky, at best. Whatever the truth, the ambitious Johnson became so enthralled with his purported power to work wonders at the troubled J.C. Penney that he put up $50 million of his own cash to invest in Penney’s stock—in the form of warrants—“as a demonstration of his confidence in J.C. Penney’s long-term potential.”

In classic outsider glass-breaking CEO style, Johnson employed a by-now predictable revolutionary rhetoric to position himself as truly transformative. His high-end A-team, recruited like himself from outside, would reinvent retail for the twenty-first century and create a new entity, to be known simply as JCP, that would be bolder, crisper, cleaner, simpler to understand, and chicly contemporary in a Target-esque sort of way.

The cost of those outsider recruitments, including his own, could be measured not just in dollars but in the incalculable cost of declining morale that blowing up the existing hierarchy in the name of transformation rang up on the chain’s internal registers. Johnson seemed to take a perverse pleasure in trashing the legacy locals by blowing an astounding $170 million in “executive transition costs” on the installation of himself and his top three lieutenants while simultaneously sending a message that he was too good to spend much time in down-market Plano, Texas, preferring to commute to Penney headquarters by corporate jet from his “real home” in cooler California.

Culturally, he also played all his cards wrong, failing to realize that he had to carefully bring his people along on the change path, not scare the bejesus out of them. As one commentator later put it:

On a deeper level, while Johnson transformed the look and feel of the stores, he failed to transform the company culture needed to support his vision. He tried to replicate Apple’s culture and practices, replacing executives with Apple veterans, for example, but those alums reportedly sequestered themselves away, working in an Apple-like bubble and failing to communicate with the rest of the company. Morale also dropped with each layoff, and not even sales floor staff—the front lines to the customer—could keep up with the changes in philosophies and policies.

After just a few months, when the vaunted revolution failed to deliver the goods, Johnson waved off the naysayers by citing his prior successes at confounding conventional wisdom with radical innovation on the floor. “What you can’t do is chicken out,” he informed Fortune. “If you had looked at the data on the Genius Bar after a year and a half, we should have taken it out of the store. But it was something I believed in with every bone in my body . . . There’s no reason to sell an idea short. The only risk would be to not fulfill the dream.”

By midsummer, however, ever widening cracks were creasing the sleek Johnson façade, prompting even former staunch supporter Bill Ackman to take a brief break from his ultimately successful campaign to unseat the CEO at P&G and dash off a quasi-contrite letter to the investors in his fund stating that while he had “complete confidence in Ron Johnson and the new management team he has assembled to execute a total transformation of this iconic U.S. retailer,” he could not deny the fact that “the execution has not … been flawless.”

After the company reported a $985 million annual loss for the year on revenues that had declined 25% from the previous year to $13 billion, prompting a 50% decline in the stock under Johnson’s erratic leadership, the JCP commercials began to admit that yes, Penney’s had “made a mistake” by not listening to its customers, while neglecting to mention the far greater mistake made by Ackman and Penney’s board for placing such blind faith in their superstar recruit in the first place.

In the ultimate rebuke of both Ackman and Johnson, a deeply fractured board voted to throw in the towel on Johnson’s activist-led revolution by making a much safer bet, and getting down on bended knee to beg Myron “I Told You So” Ullman, who had been brusquely pushed out to make way for Johnson, to return as the successor to his successor. After continuing to tussle with the board over the decision to bring Ullman back, Ackman capitulated and sold his remaining shares in Penney’s to Citigroup at a deep discount, reporting a $400 million loss on his run at the company.

So let’s take a minute to look back on where the Penney board and Ackman had gone wrong in throwing the Hail Mary pass to Johnson.

First and foremost, after dismissing Ullman, they apparently didn’t even consider drawing on Penney’s existing pipeline for a replacement. Instead, just as the HP board had done when recruiting Carly Fiorina a dozen years earlier, the board had fallen prey to the Halo Effect. Just as Fiorina’s halo had been due to the glamour that surrounded the Wall Street success of AT&T spin-off firm Lucent, Johnson’s genius had mainly been the genius of Jobs, and Johnson was subsequently revealed as Apple’s Wizard of Oz.

While the majority of postmortems focused on Johnson’s tragedy of errors, Columbia’s Mark Cohen correctly flagged the board’s cardinal errors as central to the catastrophe.

Pure and simple, the board had fatally misjudged the quality of Johnson’s credentials and the relevance of his prior experience. As Cohen correctly said, “He had never been a C.E.O., never mounted or managed a turnaround, had limited fashion-apparel experience, and had no experience in the middle- market space.”

But while all that was undoubtedly the case, the vast majority of technical, political, and cultural errors Johnson had gotten wrong at Penney’s could be chalked up to his outsider status.

Few subjects related to CEO succession arouse the intensity of interest or fuel such passionate and rancorous debate as the insider versus outsider dilemma. At the risk of repeating myself, the case for an insider selection cannot be overstated. The insider knows the company, knows the industry, knows the people, knows the networks, knows the subtle and shifting alliances, knows the existing strategy and—for better or worse, knows where many of the existing flaws and faults lie and where all the skeletons are buried and where the dirt has been carefully swept under the rug. Most important, to employ another common but useful cliché, the insider is likely to be more sensitive to the cost of throwing the baby out with the bath water.

The following is an adapted excerpt from Succession: Mastering the Make-Or-Break Process of Leadership Transition by Noel Tichy, in agreement with Portfolio, an imprint of Penguin Random House. Copyright (c) Noel Tichy, 2014.