Baseball Toaster Mike's Baseball Rants
This is my site with my opinions, but I hope that, like Irish Spring, you like it, too.
Frozen Toast
Google Search
Mike's Baseball Rants


10  09  07 
06  05  04  03 

12  11  10  09  08  07 
06  05  04  03  02  01 

12  11  10  09  08  07 
06  05  04  03  02  01 

12  11  10  09  08  07 
06  05  04  03  02  01 

12  11  10  09  08  07 
06  05  04  03  02  01 

12  11  10  09  08  07 
06  05  04  03  02  01 

12  11  10  09  08  07 
Links to MBBR
"Hall's of Relief"—Final Analysis (Really)
2004-02-08 01:24
by Mike Carminati

Previous entries:

The 1870s, '80s, and '90s
The 1900s and '10s
The 1920s, '30s, and '40s
The 1950s
The 1960s
The 1970s
The 1980s
The 1990s and 2000s
2003 Notes: Part I & II
Final Analysis: I, II, III, and IV.
Evaluation is creation: hear it, you creators! Evaluating is itself the most valuable treasure of all that we value. It is only through evaluation that value exists: and without evaluation the nut of existence would be hollow. Hear it, you creators!

—Friedrich "Fat Freddie" Nietzsche

For something that was named for a cheesy, Eighties cough drops commercial and was inspired by an even cheesier Hal Bock article, I think this series has borne a good bit of fruit. After reviewing each decade and becoming ever increasingly more verbose and table-laden in the process—how could he have devoted two sections to 2003 alone?—I feel that it's finally time to wrap the whole enchilada up.

To that end I would like to take a stab at evaluating all relievers across all eras. First, though, I have to comment on the inherent limitations in just such a study given the quality of the historical data. This is due to the fact that until relatively recently the majority of pitchers started and relieved. It wasn't until 1978 that greater than half the pitchers in the majors were specialists, pure starters or pure relievers, as opposed to swingmen. Swingmen now comprise only about a quarter of all pitchers. Even so, the apogee for starter/relievers reached 81.52% of all pitchers in 1933. The nadir for pure starters was 1.53% in 1932 and for pure relievers was 4.42% in 1898 (that is, after free substitution of players was allowed in 1891). The average all-time is still 55.13% swingmen, 14.37% starters, and 30.49% relievers (In 2003, it was 27.94%, 21.90%, and 50.16%, respectively).

All this means that if you want to evaluate relief pitching since Bruce Sutter, then aside from an odd Byung-Hyun Kim being yo-yoed in and out of the pen, the data are pretty much available. Just compensate for era and go. Dating back to the Fifties, most star relievers rarely if ever started a game. Though, given that middle relief had its own evolution that followed the closer or ace reliever's, they were anything but pure relievers until even more recently.

However, if someone wants to study the history of relief pitching in toto as we are attempting here, he inevitably comes face to face with the limitations of the statistical record. Baseball has never officially separated a pitcher's statistics as a starter from those as a reliever. Sure, once can look on or now and get statistical breakdowns for current players, but that does diddly for someone studying Hoyt Wilhelm, Firpo Marberry, or Doc Crandall. This is borne of the reliever role being less a position and more an ever-evolving strategic construct.

So until someone like Retrosheet has divided this Red Sea of data into starting and relieving statistics, we are left with guesswork for a good chunk of the data record. This is the sort of thing that Major League Baseball should be rectifying on its own dime, but Bud and his boys are more preoccupied with determining home field advantage in the World Series based on steroid use.

The only solution that made sense to me was to prorate the pitching stats based on the average innings pitched per game for the given league in the given year. I know that this is an approximation based on a small sample. Given that there has never been a full season since free substitution of players started in 1891 in which baseball had more pure starters then swingmen, this does induce a higher potential for error than I would like, but hey, what are you going to do. (By the way, the strike-shortened 1994 season did have more pure starters, 27.51%, than swingmen, 24.95%.)

Actually, it got a bit more complicated than that. I ended up prorating each swingman's stats based on a ratio of the prorated relief innings pitched divided by the sum of the prorated relief innings and the prorated starter innings all multiplied by the actual innings pitched. The prorated relief innings pitched are based on the league average in the given year of the innings pitched per relief appearance for all pure relievers multiplied by the given pitcher's relief appearances. The prorated starter innings pitched are based on the league average in the given year of the innings pitched per game started for all pure starters multiplied by the given pitcher's games started. My thought was that the numbers would be closer to the actual if I weighed the two against each other.

Finally, I made sure that the result was within the minimum and maximum asymptotes for the pitcher given his starting and relief stats. The max innings pitched as a relief are constrained by a) the total innings pitched (he couldn't exceed that as a reliever) and b) by the complete games he pitched. I assigned an eight-inning minimum to each complete game—I know it could be and probably was more. So the relief innings pitched could not exceed the total innings pitched minus eight innings per each complete game. As far as a min, I initially assigned a third of an inning per relief appearance but then realized that this was erroneous. There really is no minimum. A pitcher could make a dozen relief appearances and not get anyone out. He would then have zero innings pitched. I know that this would never happen in practice, but remember that we are talking about an absolute minimum. Anyway, I think only a handful butted up against the max (and even fewer hit the .1 IP min before I removed it).

Here are some formulae if you are into those things. Those who are uninterested, please skip ahead:

RIPAvg = Sum(IP)/Sum(G) (i.e., Relief IP Avg for yr and league. Note, this is just for pitchers who did not start a game (GS=0))

SIPAvg = Sum(IP)/Sum(G) (i.e., Starter IP Avg for yr and league. Note, this is just for pitchers who only started games (GS=GP))

EstRIP = (GP-GS) * RIPAvg (i.e., the Estimated Reliever IP)

EstSIP = GS * SIPAvg (i.e., the Estimated Starter IP)

BestEst = IP * EstRIP/(EstRIP+EstSIP) (i.e., the Best Estimate)

MaxIP = IP - CG * 8

FinalEst = MaxIP (If BestEst > MaxIP) otherwise FinalEst = BestEst (i.e., the Final Estimate)

For the sake of comparison, here the 2003 swingmen with the estimated relief innings (min. of 10 Est IP) and the actual (from

FirstLastTmFinalEst Actual% Diff
D.J.CarrascoKC 65.472.010.17%
JimBrowerSF 60.875.724.52%
50+ Est IP Avg10.83%
KrisWilsonKC 38.651.332.82%
30+ Est IP Avg20.58%
JeremyAffeldtKC 19.431.763.39%
10+ Est IP Avg33.97%

Note that as the number of innings shrinks, the accuracy of the estimate becomes more of an issue. For pitchers with at least 50 estimated innings, the estimate is about 10%, or 5 innings, off; for 30 innings, about 20%; and for 10 innings, the estimate is more than a third off of the actual. This is due to a few especially long relief appearances or a few especially short starts being able to sway the smaller samples more easily. Given that the pitchers that we will be evaluating as among the best ever should have pitched over 50 innings in relief, I think a 10% error (5 innings) is the best we can do and therefore, has to be accepted. Of course, there is no way to know if historical data is as accurate as the 2003 data given that we don't have that breakdown to begin with.

OK, if we accept that these numbers are as accurate as we can get, let's move on. I then took the estimated relief innings pitched (i.e., FinalEst above) and prorated the pitcher's other stats by the estimated relief innings divided by total innings.

So now that we have all of the data for relievers, we need a way to compare one to another that crosses all eras. Well, I'm going to be lazy and rely on other people's research. I first started devising a statistical method to evaluate all relievers after reading Bill James' "Valuing Relievers" in his The New Historical Baseball Abstract. He has a section in which he researches these two questions:

1. What is the value of one run saved by a relief ace, as contrasted with one run saved by a starting pitcher?

2. Is the modern style of using the ace reliever, which involves using him almost exclusively in "save" situations, the optimal usage pattern?

James findings are that a run saved by a reliever has far more significance than one saved by a starter. He uses archetypes to answer the second question: Clint Brown for the mid-Thirties to mid-Fifties and an average of 58 games pitched, 106 innings, 10 saves, and 5.88 runs saved equal to one win. Elroy Face: mid-Fifties to 1962, 59 G, 96 IP, 15 Sv, 5.14 R equal to a win. Hoyt Wilhelm: 1963-78, 72 G, 128 IP, 24 Sv, 4.47 R to a win. Bruce Sutter: 1978-89, 61 G, 111 IP, 38 Sv, 4.73 R per win. Robb Nen: 1990 to present, 77 G, 91 IP, 41 Sv, 4.64 R per win. And one win per eight to nine runs saved for a generic starting pitcher. James found that the Wilhelm pattern was optimal. He expands on these findings to develop the theory of optimal reliever use that I outlined in the Gagne section. Oddly, James does not seem to use these findings to develop his Win Share formula for relief pitchers.

OK, so we can relate runs saved to wins, but how do we determine a relief pitcher's runs saved? For that I would like to use a formula from Total Baseball that has been modified by Baseball Prospectus. That formula is Pitching Runs (or, as BP calls it, Adjusted Pitching Runs). It calculates the (earned) runs prevented over the league average. The difference between the expected earned runs for the number of innings pitched minus the actual earned runs is the basis for PR. The original formula was innings pitched multiplied by the league average ERA divided by nine minus the pitcher's earned runs allowed (PR = IP * Lg avg ERA / 9 – ER). BP updated the formula to use runs instead of earned runs and the pitcher's homepark park factor (APR = Lg avg Run Avg * IP – R/Park factor). I am going to use a combination of the two: I will keep earned runs from TB but will add the park factor from BP (actually, the pitcher's park factor from via Sean Lahman's database). I am more concerned about the earned runs than the runs allowed. I added in the park factor to adjust the earned runs for the pitcher's home park. My final formula is innings pitched times the league average earned run average divided by nine minus earned runs divided by the pitcher ballpark factor.

Here are all the relievers who scored 25 or more Pitching Runs all-time. You'll note that there are a good number of middle relievers and early closers (though Gagne is 16th):

Mark Eichhorn198643.13
Jim Kern197941.46
Mariano Rivera199635.07
John Hiller197334.46
Rich Gossage197734.23
Doug Corbett198033.32
Dan Quisenberry198332.70
Bruce Sutter197732.33
Willie Hernandez198431.64
Rich Gossage197531.30
Keith Foulke199930.33
Tim Burke198729.60
Derek Lowe199928.25
Roberto Hernandez199627.89
Sparky Lyle197727.77
Eric Gagne200327.44
Bruce Sutter198427.26
Mudcat Grant197026.85
Sid Monge197926.70
Chris Hammond200226.67
Dennis Eckersley199026.55
Ellis Kinder195326.53
Aurelio Lopez197926.47
Ted Abernathy196726.36
Jesse Orosco198326.32
Robb Nen199826.13
Damaso Marte200325.91
Paul Quantrill199725.87
Bob James198525.70
Gabe White200025.70
John Wetteland199325.69
Jeff Montgomery198925.56
Dick Radatz196425.52
Dick Radatz196325.48
Dan Quisenberry198525.46
Guillermo Mota200325.45
Jose Mesa199525.35
Mike Marshall197925.29
Donnie Moore198525.25
Octavio Dotel200225.17
Dick Radatz196225.16

Now here are the worst all-time. There are some real old-timers in this mix:

Dick Welteroth1949-29.32
Ron Davis1986-27.27
Connie Grob1956-26.57
Larry Benton1935-26.37
Lee Guetterman1992-25.74
Ben Wade1954-24.09
Jim Todd1979-23.52
Jaret Wright2003-23.11
Dave Hamilton1980-22.85
Art Decatur1927-22.36
Bobby Ayala1998-22.05
Norm Charlton1997-21.36
Jeff Kaiser1985-21.35
Carl Doyle1940-20.79
Ron Taylor1965-20.77
Mel Rojas1999-20.61
Les Lancaster1992-20.24
June Greene1929-20.12
Al Smith1938-20.05
Ken Burkhart1948-19.95
Johnny Klippstein1961-19.72
Ben Hayes1983-19.68
Jot Goar1896-19.59
Don McMahon1960-19.54
Glenn Liebhardt1936-19.50
Bryan Hickerson1995-19.48
Byron McLaughlin1980-19.43
Joe Heving1934-19.38
Mark Clear1983-19.31
Stu Flythe1936-19.29
Bob Miller1958-19.19
Rich Thompson1985-19.13
Alan Embree2001-19.04
Lloyd Allen1973-18.96
Bob Veale1971-18.90
Jim Brosnan1954-18.90
Mark Petkovsek2001-18.89
Bunky Stewart1954-18.76
Jess Dobernic1949-18.72
Vic Darensbourg1999-18.60
Darren Holmes2000-18.39
Rene Monteagudo1945-18.35
Omar Daal1997-18.34
George Frazier1985-18.34
Darold Knowles1975-18.24
Bill Fischer1958-18.12
Junior Walsh1951-18.01
Dave Danforth1919-17.63
Jesus Colome2002-17.52
Lee Guetterman1986-17.51

Dick Welteroth was truly awful: 2-5, 52 GP (2nd in the AL), 2 GS, 25 GF (3rd in the AL), 2 saves, 95.1 IP, 107 H, 83 ER, 89 BB, 37 K (a 0.42 K:BB ratio), and 7.36 ERA (42% worse the park-adjusted league average). He was 21 and only pitched 6 more innings in the majors.

Just for fun, here are the top 100 relievers by career PR:

Hoyt Wilhelm239.03
Rich Gossage177.62
Mariano Rivera162.88
John Franco155.44
Dan Quisenberry145.91
Kent Tekulve138.50
Tom Henke130.21
Lee Smith127.70
Doug Jones124.94
Mark Eichhorn124.16
Roberto Hernandez123.54
John Wetteland121.61
Sparky Lyle119.28
Paul Quantrill115.25
Jesse Orosco113.07
Keith Foulke112.12
Rollie Fingers110.55
Bruce Sutter109.34
Mike Jackson107.56
Trevor Hoffman107.24
Jeff Montgomery107.22
John Hiller106.70
Greg Harris101.51
Robb Nen101.48
Steve Reed101.11
Troy Percival100.68
Billy Wagner98.36
Dennis Eckersley98.24
Tug McGraw98.19
Jeff Nelson97.94
Mike Marshall96.14
Bob Stanley91.84
Clay Carroll90.97
Gary Lavelle90.09
Gene Garber89.65
Armando Benitez89.46
Mike Timlin87.02
Bob Wickman86.11
Ron Perranoski85.79
Rick Aguilera85.52
Don McMahon83.98
Jeff Reardon82.78
Tom Burgmeier81.33
Eric Plunk80.02
Ellis Kinder79.88
Mike Henneman77.52
Greg Minton77.42
Jeff Shaw77.15
Rod Beck76.07
Dave Veres75.26
Steve Farr72.52
Tim Burke72.46
Stu Miller72.08
Shigetoshi Hasegawa71.94
Willie Hernandez71.55
Dave Smith71.23
Jeff Russell71.11
Dan Plesac70.73
Larry Andersen70.53
Lindy McDaniel69.73
Jay Howell69.63
Randy Myers68.86
Ugueth Urbina66.12
Derek Lowe65.48
Bryan Harvey64.12
Ted Abernathy62.40
Roy Face61.45
Arthur Rhodes61.26
Jose Mesa60.92
Steve Howe59.71
Bob Locker59.47
Paul Shuey59.36
Mike Remlinger59.35
Frank Linzy58.06
Duane Ward57.47
Danny Graves57.41
Paul Assenmacher57.23
Ron Reed56.69
Terry Forster56.64
Gregg Olson56.15
Jim Corsi56.09
Octavio Dotel55.79
Greg McMichael55.71
Mike Fetters55.29
Curt Leskanic55.08
Bill Henry54.67
Jim Brewer53.80
Steve Bedrosian53.65
Gerry Staley53.18
Steve Karsay52.85
La Troy Hawkins52.40
Darold Knowles52.08
Todd Worrell51.82
Al Hrabosky51.71
Roger McDowell51.35
Walter Johnson50.79
Bobby Shantz50.26
Jeff Brantley49.97
Dick Radatz49.45
Steve Mingori48.33

I had to include it because I thought it was quite a list. Just seeing names like John Hiller, Ron Perranoski, Ellis Kinder, Stu Miller, Lindy McDaniel, Roy Face, Frank Linzy, Bill Henry, Gerry Staley, Bobby Shantz, and Walter Johnson peppered among the modern closers in a statistical comparison whets my appetite for the adjusted comparison. Remember that James' findings show that modern closers' runs prevented are worth less in terms of wins than the ones for ace reliever of the past. So I expect the Wilhelmites to be buoyed even more by the final analysis.

Now here are the all-time worst in career PR:

Jerry Johnson-41.51
John Montague-39.69
Larry Benton-39.68
Scott Bailes-36.89
Claude Willoughby-36.28
Frank LaCorte-36.00
George Grant-35.66
Dick Welteroth-35.09
George Smith-34.64
Paul Giel-34.02
Pat Mahomes-33.34
Rosy Ryan-33.18
Don Johnson-32.52
Mel Wright-32.48
Carl Scheib-32.04
Jeff Kaiser-31.54
Bill Zuber-31.51
Paul Foytack-31.14
Jack Hamilton-30.88
Phil Ortega-30.78
Jack Crimian-30.56
Marino Pieretti-30.46
Boom-Boom Beck-30.36
Dick Littlefield-30.25
Brian Williams-30.24
Dan McGinn-30.02
Bo McLaughlin-29.64
Alex Ferguson-29.18
Dave Wainhouse-28.79
Ferdie Schupp-28.74
Chuck Hartenstein-28.64
Omar Daal-28.59
Mike Kekich-28.55
Hal Elliott-28.15
Ray Burris-28.04
Bobby Coombs-27.63
Ron Taylor-27.63
Vic Frasier-27.57
Vic Darensbourg-27.08
Al Lyons-26.97
Tommie Sisk-26.92
Rene Monteagudo-26.92
Russ Van Atta-26.86
Al Gettel-26.78
Drew Hall-26.76
Connie Grob-26.57
Wes Gardner-26.57
Taylor Phillips-26.50
Leo Mangum-26.46
Dustin Hermanson-26.42
Hod Lisenbee-26.37
Frank Gabler-26.32
Bunky Stewart-26.11
Bill Fischer-26.05
Dallas Green-25.99
Lou McEvoy-25.86
Jim Panther-25.82
Larry Hardy-25.78
Sam Nahem-25.72
Matt Whiteside-25.64
Glenn Liebhardt-25.61
Ray Harrell-25.59
Ralph Branca-25.52
Karl Drews-25.41
Bill Laxton-25.39
Rube Fischer-25.11
Floyd Weaver-25.10
Junior Walsh-25.09

Jerry Johnson pitched in the Seventies. Aside from a good year in 1971 with the Giants (12-9, 2.97 ERA—14 % better than the park-adjusted league average, 18 saves, 67 GP all in relief, 109 IP), he was pretty awful for 10 seasons mostly in relief.

Now, we have to divide them under Bill James' reliever archetypes and assign wins to the runs prevented. That sounds simple. Well, assigning individuals to roles turned out to be more difficult than I at first thought. The difficulty stems from the fact that the role definitions are based on use, which does not necessarily translate into the statistical record as it currently is configured. I had intended to use a statistical method called multiple (linear) regression. Those of you whose eyes just glazed over can feel free to skip to the results. The rest, follow me.

However, this method assumes that the result (dependent variable or here, pitching runs saved per win) has a linear relationship with its component variables (independent variables, which I'll define soon). Given that a reliever's role is defined best statistically by the innings he pitches per appearance (IP/G) and the number of saves per appearance (Sv/G), I chose those as the variables. But when you graph these variables against James' pitching runs saved per win, you get anything but a line. In fact the two curves defined by the variables double back on themselves as the Sutter and Nen roles result in a lower score from James. I could have assumed that there had been some error induced and forced a line on the data, but 1) that would be ignoring the results of James' study and 2) it would make the Nen type the de facto best.

Here is a table with the James roles and the ratios for those roles (Note that 1) the type will be used as a shortcut throughout the study and 2) the starter stats are based on all-time averages for pure starters that I derived myself):

Clint Brown A58106105.881.830.170.094
Elroy FaceB5996155.141.630.250.156
Hoyt WilhelmC72128244.471.780.330.188
Bruce SutterD61111384.731.820.620.342
Robb NenE7791414.641.180.530.451

So next I tried grouping relievers under the various types by similarity to the average type that James defined. I thought that using the season ranges that he provided would have been too facile and would have ignored the fact that reliever use does transcend era to a certain degree.

But grouping pitchers by how much their stats resemble Clint Brown's or Robb Nen's had its own inherent problems. What does "similar" mean after all? And could a pitcher be "similar" to two of our archetypes.

I also had another problem. What do I do with middle relievers, some of whom scored very high in pitching runs? They didn't even resemble Brown, and lumping them under starting pitchers would be unfair.

So I monkeyed around with the data some more. And then I found a way to divide post-Sutter type relievers from the earlier ones. That was saves per innings pitched. A Sutter type has a Sv/IP of .342. The highest value for the pre-Sutter style relievers was .188. I used .25 as the dividing line.

To divide the Sutter types from the Nen ones, the main criterion I used was innings pitched per appearance (IP/G). The Nen relievers were close to 1 IP/G and the Sutter ones approached the IP/G of the previous eras (1.82). I used 1.5 IP/G to separate the two groups. Initially, I set the upper limit for the Sutter group at 2.25, but found that only 100 pitcher years exceeded this value and that the group most resembled Sutter. So I ended up including them with the Sutter group.

The Sutter group ended up with 502 pitcher seasons, and the Nen with 814.

The earlier groups are differentiated by saves per game (Sv/G). I set the groups up like this: starters/middle relievers less than .1 Sv/G, Brown between .1 and .2 (inclusive) Sv/G, Face between .2 and .3 Sv/G, and Wilhelm greater than .3 Sv/G. (Again I limited the Wilhelm group to .4 Sv/G at the high end but found that the group that exceeded this figure best fit with the Wilhelmites.) The resulting pitcher season count per group was: middle reliever 20, 883, Brown 3481, Face 1362, and Wilhelm 1075.

Next, I tackled the conversion for middle reliever pitching runs saved to wins (By the way, the middle reliever type is "M"). They belonged in neither the starting pitcher camp nor the Clint Brown camp. Browns' denizens averaged 1.83 IP/G, and the starters averaged 6.87. My solution was to draw a line literally between the two and project each middle reliever along that line based on his innings pitched per appearance. The resulting formula is not for the faint of heart: R Win = PR/((8.5-5.88)/(6.87-1.83)*IP/G + (5.88-1.83)).

Here is a final breakdown by era of each of the various types:


Note that each type has outliers outside of their eras, but that each does peak in the appropriate era. Note especially how the Sutter type (D) has died out completely in favor of the Nen type (E). Lastly, middle relievers ("M") began to die out with each new reliever innovation but have been growing steadily since the Seventies as bullpens diversified and more pitchers per game were used.

To be completed…

Comment status: comments have been closed. Baseball Toaster is now out of business.