Previous entries:

Evaluation is creation: hear it, you creators! Evaluating is itself the most valuable treasure of all that we value. It is only through evaluation that value exists: and without evaluation the nut of existence would be hollow. Hear it, you creators!

—Friedrich "Fat Freddie" Nietzsche

For something that was named for a cheesy, Eighties cough drops commercial and was inspired by an even cheesier Hal Bock article, I think this series has borne a good bit of fruit. After reviewing each decade and becoming ever increasingly more verbose and table-laden in the process—how could he have devoted two sections to 2003 alone?—I feel that it's finally time to wrap the whole enchilada up.

To that end I would like to take a stab at evaluating all relievers across all eras. First, though, I have to comment on the inherent limitations in just such a study given the quality of the historical data. This is due to the fact that until relatively recently the majority of pitchers started and relieved. It wasn't until 1978 that greater than half the pitchers in the majors were specialists, pure starters or pure relievers, as opposed to swingmen. Swingmen now comprise only about a quarter of all pitchers. Even so, the apogee for starter/relievers reached 81.52% of all pitchers in 1933. The nadir for pure starters was 1.53% in 1932 and for pure relievers was 4.42% in 1898 (that is, after free substitution of players was allowed in 1891). The average all-time is still 55.13% swingmen, 14.37% starters, and 30.49% relievers (In 2003, it was 27.94%, 21.90%, and 50.16%, respectively).

All this means that if you want to evaluate relief pitching since Bruce Sutter, then aside from an odd Byung-Hyun Kim being yo-yoed in and out of the pen, the data are pretty much available. Just compensate for era and go. Dating back to the Fifties, most star relievers rarely if ever started a game. Though, given that middle relief had its own evolution that followed the closer or ace reliever's, they were anything but pure relievers until even more recently.

However, if someone wants to study the history of relief pitching in toto as we are attempting here, he inevitably comes face to face with the limitations of the statistical record. Baseball has never officially separated a pitcher's statistics as a starter from those as a reliever. Sure, once can look on ESPN.com or MLB.com now and get statistical breakdowns for current players, but that does diddly for someone studying Hoyt Wilhelm, Firpo Marberry, or Doc Crandall. This is borne of the reliever role being less a position and more an ever-evolving strategic construct.

So until someone like Retrosheet has divided this Red Sea of data into starting and relieving statistics, we are left with guesswork for a good chunk of the data record. This is the sort of thing that Major League Baseball should be rectifying on its own dime, but Bud and his boys are more preoccupied with determining home field advantage in the World Series based on steroid use.

The only solution that made sense to me was to prorate the pitching stats based on the average innings pitched per game for the given league in the given year. I know that this is an approximation based on a small sample. Given that there has never been a full season since free substitution of players started in 1891 in which baseball had more pure starters then swingmen, this does induce a higher potential for error than I would like, but hey, what are you going to do. (By the way, the strike-shortened 1994 season did have more pure starters, 27.51%, than swingmen, 24.95%.)

Actually, it got a bit more complicated than that. I ended up prorating each swingman's stats based on a ratio of the prorated relief innings pitched divided by the sum of the prorated relief innings and the prorated starter innings all multiplied by the actual innings pitched. The prorated relief innings pitched are based on the league average in the given year of the innings pitched per relief appearance for all pure relievers multiplied by the given pitcher's relief appearances. The prorated starter innings pitched are based on the league average in the given year of the innings pitched per game started for all pure starters multiplied by the given pitcher's games started. My thought was that the numbers would be closer to the actual if I weighed the two against each other.

Finally, I made sure that the result was within the minimum and maximum asymptotes for the pitcher given his starting and relief stats. The max innings pitched as a relief are constrained by a) the total innings pitched (he couldn't exceed that as a reliever) and b) by the complete games he pitched. I assigned an eight-inning minimum to each complete game—I know it could be and probably was more. So the relief innings pitched could not exceed the total innings pitched minus eight innings per each complete game. As far as a min, I initially assigned a third of an inning per relief appearance but then realized that this was erroneous. There really is no minimum. A pitcher could make a dozen relief appearances and not get anyone out. He would then have zero innings pitched. I know that this would never happen in practice, but remember that we are talking about an absolute minimum. Anyway, I think only a handful butted up against the max (and even fewer hit the .1 IP min before I removed it).

Here are some formulae if you are into those things. Those who are uninterested, please skip ahead:

RIPAvg = Sum(IP)/Sum(G) (i.e., Relief IP Avg for yr and league. Note, this is just for pitchers who did not start a game (GS=0))

SIPAvg = Sum(IP)/Sum(G) (i.e., Starter IP Avg for yr and league. Note, this is just for pitchers who only started games (GS=GP))

EstRIP = (GP-GS) * RIPAvg (i.e., the Estimated Reliever IP)

EstSIP = GS * SIPAvg (i.e., the Estimated Starter IP)

BestEst = IP * EstRIP/(EstRIP+EstSIP) (i.e., the Best Estimate)

MaxIP = IP - CG * 8

FinalEst = MaxIP (If BestEst > MaxIP) otherwise FinalEst = BestEst (i.e., the Final Estimate)

For the sake of comparison, here the 2003 swingmen with the estimated relief innings (min. of 10 Est IP) and the actual (from ESPN.com):

Note that as the number of innings shrinks, the accuracy of the estimate becomes more of an issue. For pitchers with at least 50 estimated innings, the estimate is about 10%, or 5 innings, off; for 30 innings, about 20%; and for 10 innings, the estimate is more than a third off of the actual. This is due to a few especially long relief appearances or a few especially short starts being able to sway the smaller samples more easily. Given that the pitchers that we will be evaluating as among the best ever should have pitched over 50 innings in relief, I think a 10% error (5 innings) is the best we can do and therefore, has to be accepted. Of course, there is no way to know if historical data is as accurate as the 2003 data given that we don't have that breakdown to begin with.

OK, if we accept that these numbers are as accurate as we can get, let's move on. I then took the estimated relief innings pitched (i.e., FinalEst above) and prorated the pitcher's other stats by the estimated relief innings divided by total innings.

So now that we have all of the data for relievers, we need a way to compare one to another that crosses all eras. Well, I'm going to be lazy and rely on other people's research. I first started devising a statistical method to evaluate all relievers after reading Bill James' "Valuing Relievers" in his The New Historical Baseball Abstract. He has a section in which he researches these two questions:

1. What is the value of one run saved by a relief ace, as contrasted with one run saved by a starting pitcher?

2. Is the modern style of using the ace reliever, which involves using him almost exclusively in "save" situations, the optimal usage pattern?

James findings are that a run saved by a reliever has far more significance than one saved by a starter. He uses archetypes to answer the second question: Clint Brown for the mid-Thirties to mid-Fifties and an average of 58 games pitched, 106 innings, 10 saves, and 5.88 runs saved equal to one win. Elroy Face: mid-Fifties to 1962, 59 G, 96 IP, 15 Sv, 5.14 R equal to a win. Hoyt Wilhelm: 1963-78, 72 G, 128 IP, 24 Sv, 4.47 R to a win. Bruce Sutter: 1978-89, 61 G, 111 IP, 38 Sv, 4.73 R per win. Robb Nen: 1990 to present, 77 G, 91 IP, 41 Sv, 4.64 R per win. And one win per eight to nine runs saved for a generic starting pitcher. James found that the Wilhelm pattern was optimal. He expands on these findings to develop the theory of optimal reliever use that I outlined in the Gagne section. Oddly, James does not seem to use these findings to develop his Win Share formula for relief pitchers.

OK, so we can relate runs saved to wins, but how do we determine a relief pitcher's runs saved? For that I would like to use a formula from Total Baseball that has been modified by Baseball Prospectus. That formula is Pitching Runs (or, as BP calls it, Adjusted Pitching Runs). It calculates the (earned) runs prevented over the league average. The difference between the expected earned runs for the number of innings pitched minus the actual earned runs is the basis for PR. The original formula was innings pitched multiplied by the league average ERA divided by nine minus the pitcher's earned runs allowed (PR = IP * Lg avg ERA / 9 – ER). BP updated the formula to use runs instead of earned runs and the pitcher's homepark park factor (APR = Lg avg Run Avg * IP – R/Park factor). I am going to use a combination of the two: I will keep earned runs from TB but will add the park factor from BP (actually, the pitcher's park factor from BaseballReference.com via Sean Lahman's database). I am more concerned about the earned runs than the runs allowed. I added in the park factor to adjust the earned runs for the pitcher's home park. My final formula is innings pitched times the league average earned run average divided by nine minus earned runs divided by the pitcher ballpark factor.

Here are all the relievers who scored 25 or more Pitching Runs all-time. You'll note that there are a good number of middle relievers and early closers (though Gagne is 16th):

Now here are the worst all-time. There are some real old-timers in this mix:

Dick Welteroth was truly awful: 2-5, 52 GP (2nd in the AL), 2 GS, 25 GF (3rd in the AL), 2 saves, 95.1 IP, 107 H, 83 ER, 89 BB, 37 K (a 0.42 K:BB ratio), and 7.36 ERA (42% worse the park-adjusted league average). He was 21 and only pitched 6 more innings in the majors.

Just for fun, here are the top 100 relievers by career PR:

I had to include it because I thought it was quite a list. Just seeing names like John Hiller, Ron Perranoski, Ellis Kinder, Stu Miller, Lindy McDaniel, Roy Face, Frank Linzy, Bill Henry, Gerry Staley, Bobby Shantz, and Walter Johnson peppered among the modern closers in a statistical comparison whets my appetite for the adjusted comparison. Remember that James' findings show that modern closers' runs prevented are worth less in terms of wins than the ones for ace reliever of the past. So I expect the Wilhelmites to be buoyed even more by the final analysis.

Now here are the all-time worst in career PR:

Jerry Johnson pitched in the Seventies. Aside from a good year in 1971 with the Giants (12-9, 2.97 ERA—14 % better than the park-adjusted league average, 18 saves, 67 GP all in relief, 109 IP), he was pretty awful for 10 seasons mostly in relief.

Now, we have to divide them under Bill James' reliever archetypes and assign wins to the runs prevented. That sounds simple. Well, assigning individuals to roles turned out to be more difficult than I at first thought. The difficulty stems from the fact that the role definitions are based on use, which does not necessarily translate into the statistical record as it currently is configured. I had intended to use a statistical method called multiple (linear) regression. Those of you whose eyes just glazed over can feel free to skip to the results. The rest, follow me.

However, this method assumes that the result (dependent variable or here, pitching runs saved per win) has a linear relationship with its component variables (independent variables, which I'll define soon). Given that a reliever's role is defined best statistically by the innings he pitches per appearance (IP/G) and the number of saves per appearance (Sv/G), I chose those as the variables. But when you graph these variables against James' pitching runs saved per win, you get anything but a line. In fact the two curves defined by the variables double back on themselves as the Sutter and Nen roles result in a lower score from James. I could have assumed that there had been some error induced and forced a line on the data, but 1) that would be ignoring the results of James' study and 2) it would make the Nen type the de facto best.

Here is a table with the James roles and the ratios for those roles (Note that 1) the type will be used as a shortcut throughout the study and 2) the starter stats are based on all-time averages for pure starters that I derived myself):

So next I tried grouping relievers under the various types by similarity to the average type that James defined. I thought that using the season ranges that he provided would have been too facile and would have ignored the fact that reliever use does transcend era to a certain degree.

But grouping pitchers by how much their stats resemble Clint Brown's or Robb Nen's had its own inherent problems. What does "similar" mean after all? And could a pitcher be "similar" to two of our archetypes.

I also had another problem. What do I do with middle relievers, some of whom scored very high in pitching runs? They didn't even resemble Brown, and lumping them under starting pitchers would be unfair.

So I monkeyed around with the data some more. And then I found a way to divide post-Sutter type relievers from the earlier ones. That was saves per innings pitched. A Sutter type has a Sv/IP of .342. The highest value for the pre-Sutter style relievers was .188. I used .25 as the dividing line.

To divide the Sutter types from the Nen ones, the main criterion I used was innings pitched per appearance (IP/G). The Nen relievers were close to 1 IP/G and the Sutter ones approached the IP/G of the previous eras (1.82). I used 1.5 IP/G to separate the two groups. Initially, I set the upper limit for the Sutter group at 2.25, but found that only 100 pitcher years exceeded this value and that the group most resembled Sutter. So I ended up including them with the Sutter group.

The Sutter group ended up with 502 pitcher seasons, and the Nen with 814.

The earlier groups are differentiated by saves per game (Sv/G). I set the groups up like this: starters/middle relievers less than .1 Sv/G, Brown between .1 and .2 (inclusive) Sv/G, Face between .2 and .3 Sv/G, and Wilhelm greater than .3 Sv/G. (Again I limited the Wilhelm group to .4 Sv/G at the high end but found that the group that exceeded this figure best fit with the Wilhelmites.) The resulting pitcher season count per group was: middle reliever 20, 883, Brown 3481, Face 1362, and Wilhelm 1075.

Next, I tackled the conversion for middle reliever pitching runs saved to wins (By the way, the middle reliever type is "M"). They belonged in neither the starting pitcher camp nor the Clint Brown camp. Browns' denizens averaged 1.83 IP/G, and the starters averaged 6.87. My solution was to draw a line literally between the two and project each middle reliever along that line based on his innings pitched per appearance. The resulting formula is not for the faint of heart: R Win = PR/((8.5-5.88)/(6.87-1.83)*IP/G + (5.88-1.83)).

Here is a final breakdown by era of each of the various types:

Note that each type has outliers outside of their eras, but that each does peak in the appropriate era. Note especially how the Sutter type (D) has died out completely in favor of the Nen type (E). Lastly, middle relievers ("M") began to die out with each new reliever innovation but have been growing steadily since the Seventies as bullpens diversified and more pitchers per game were used.

To be completed…