Since 2001 MLB has been using QuesTec's UIS (Umpire Information System) to evaluate to its umpires' performance in calling balls and strikes, and on occasion it has been a source of controversy for both umpires and players. While still a Diamondback, dyed-in-the-wool(-sock) Red Sox pitcher Curt Schilling beat one QuesTec camera into submission for what he perceived as miscalls by the home plate umpire, who was supposedly intimidated by the Big Brotherian contraption.

The system has been rolled out to just ten of the thirty major-league stadia. However, a recent agreement between MLB and the umpires union gives baseball the green light to roll the system out universally.

I have come to an equilibrium of sorts with the UIS system. I initially dismissed the system as "inherently flawed" but started to come around as I have learned more about it (and have written extensively on the topic—a listing of articles is available in the article above). And I have been, hopefully, trying to keep an open mind on the matter.

After the agreement between the owners and the umps was inked, the opportunity came up to interview a major-league QuesTec/UIS operator, and I jumped at the chance.

Here's our exchange:

Q: First, can you give us an overview of what your job entails? That is, prior to, during, and after the game as well as from game to game.

A: As an Umpire Information System (UIS) operator, I arrive at the stadium about an hour before the game to do set-up. The system is made up of two different computers and a TV monitor. One of the computers is the “tracking computer” and has the tracking software installed on it. I’ll track some batting practice pitches to make sure that the system is working properly and that the cameras are aimed properly. Then I’ll make sure that our video/audio feed is coming in properly. After I’ve verified that, I’ll head off to the pressbox to get the lineups for the day. On my way, I usually check out the protective lenses on the dugout cameras to make sure that they aren’t obstructed or dirty. Very rarely, we’ll have to re-aim a camera because it’s been bumped out of position (this happened once last year). When I return to the system, I’ll enter the game information: lineups, umpires, teams, weather conditions, etc. This information goes into the second (“scoring”) computer. Once all of this is complete and batting practice is over, it’s time to calibrate the tracking system—something that is done for every game. The calibration files set baseline points from the back of home plate so that the tracking system knows what points it’s looking at. About 5-10 minutes before the game, I’ll start the video recorder, which will record the entire game in three (or more) digital video files—one file for every three innings.

During the game, my primary responsibility is to score the result of every pitch—foul, hit, swinging strike, called strike, called ball, etc. This information is recorded on a paper scoresheet and then entered into the scoring computer (using an Access DB entry form). I also have to make sure that the tracking system is operating properly and isn’t missing pitches, acting weird, etc. It’s important to note that no “calls” are made by the system during the game. There is a generic strike zone that makes estimates for each pitch, but during the game, that’s not adapted to each batter. The system can display location in real time, but cannot make judgements of ball/strike until after the game. When the game is over, the database (scoring computer) has information about each pitch, and the tracking system (tracking computer) has tracking information for each pitch. Corresponding pitch numbers allow this data to be correlated. Obviously, it’s important to make sure that these numbers match up during the course of the game. In addition to a track for each pitch, the system takes still photos (“snapshots”) of the batter as the pitch crosses the plate. Taken as a whole, then, the system gathers four elements of information about each pitch: the track, the still photo, the scoring information, and the digital video.

After the game is when the magic happens. First comes the most important step. The scoring computer outputs a list of all “called” pitches—balls and strikes—that excludes all other pitches: fouls, hits, and swinging strikes. That list is then correlated to the still photos producing a sequence of still photos, one for each called pitch. It is then my responsibility to set the strike zone (top and bottom) for each called pitch during the game. If a given batter takes four balls, I set the strike zone four times for that batter. Thus, it is possible for a given batter to have up to 6 still photos (and six different strike zones) per at bat. An average number of called pitches for a 9 inning game is about 150 out of a total of about 450 (including the warmup throws before each inning). I set the top and bottom of the strike zone at the hollow of the knee and the top of the belt on the photos. The system then adds in 2.5 ball widths to the mark I set at the top of the belt to move the top of the strike zone up to the regulation strike zone.

Once this process is complete, the system generates a series of reports that determine whether the umpire’s call was correct (C), incorrect (N), or close, but acceptable (A). The system has a built in margin of error (the .5 inch) to give umpires the benefit of the doubt. After the top/bot has been set for each pitch and the reports have been generated I then clip the game video. This produces a 10 second video clip for each called pitch so the umpire can see what happened on any given pitch. This video is taken from the centerfield camera. All of this information is then compiled into what is called the “Pitch table” which lists the pitch, the pitcher, the batter, the count, the speed of the pitch, the umpire’s call, and the system’s assessment. Also available for each pitch is the 10 second video clip, a diagram showing the location of the pitch, and the still photo that shows the strike zone as I set it. This information is used to throw out pitches that humans determine were incorrectly or unfairly assessed by the system or the operator. I don’t know exactly what standards are used to make those determinations; that work is done by a series of umpire supervisors in the employ of MLB.

Finally, it’s time for quality control. I go through each pitch on the pitch table and watch the video to make sure that I’ve scored it correctly. If I have made an error (say I scored a swinging strike as a called strike) I’ll make the necessary corrections and re-run the reports. I also make sure that the video clips have been produced correctly. Overall, there’s quite a bit of redundancy built in to eliminate operator error. If I discover, for example, that I’ve set a strike zone incorrectly on a particular pitch, that pitch is thrown out and will not count against that umpire.

The complete pitch table, along with graphics that show accuracy and consistency (color coded) are then burnt to a CD that I or another operator will give to the umpire the next day. Umpires have been provided with laptops by MLB so that they can look over the data. A copy of the CD goes to MLB, and a copy stays with QuesTec to serve as a backup in case something happens to the other CDs.

The post-game process usually takes about an hour, but that depends on how long the game was and how many called pitches there were. Needless to say, I’ve grown to appreciate quick workers who either generate a lot of swinging strikes and don’t walk many people or induce a lot of balls in play early in the count. This reduces the time of the game as well as the final number of called pitches.

Q: What kind of training and retraining were you provided and what kind of job requirements did they have? Who is your employer: the team, the stadium, or MLB?

A: I was trained by three different people who came into town to oversee my training process. I began by observing the system in operation, graduated to scoring on paper, and finally was placed in control of the system with an experienced operator there to answer questions and help me solve any problems that I might run into. In terms of setting the strike zone on the still photos, we used old data to practice. I’d set the strike zones, and the trainer would comment on my accuracy until both he and I felt confident that I was setting it properly. The whole system is surprisingly easy to learn, so long as one is comfortable using PCs. It’s a bit intense at first, but as you become more comfortable with what to look for, running it becomes relatively easy as well.

The listed job requirements were relatively simple, and by now infamous: live within 50 miles of an MLB stadium, be familiar with computers, and possess strong baseball knowledge. In addition, I had some background in audio and video production.

I am employed as an independent contractor by QuesTec, Inc., who owns and produces the Umpire Information System under contract to Major League Baseball. As to retraining, I’m assuming that we’ll have a visit or two from the manager of the UIS project, also an employee of QuesTec. Those details are still being worked out. We’ll do a few preseason games to get back in the swing of things and depending on schedules, we may do a few spring training games.

Q: How receptive have the umpires, players, managers, and other on-field personnel been of QuesTec and of the individuals involved with it? How much interaction do you have with them?

A: My interaction with personnel is relatively small. Umpires seem mostly to be ambivalent, though I have noticed a few rolled eyes when I deliver the CDs. Some umpires seem extremely interested in what we do; others seem to be uninterested. I run into players and radio/television announcers often enough, but they don’t know who I am, and I haven’t made it a point to tell them. There’s a clear pecking order to the pressbox area. The columnists and personalities are known by everyone, and the people with behind-the-scenes jobs (say, the scoreboard operator, the grounds crew, etc) are not known as well. I’ve talked the most with security people, who I see every day as I enter the stadium or use the staff elevator. I haven’t had occasion to talk with a player yet. I have had a few umpire supervisors stop by to see how the system is functioning, and to a man they have all been friendly, courteous and interested.

Q: Are you involved in the installation of the QuesTec system software? What about QuesTec upgrades or system hardware/software upgrades? What sort of equipment (PC, etc.) do you use?

A: I have not performed any installation. The system was in place when I was hired. QuesTec management comes around to each stadium to install upgrades, etc in the offseason so that all systems are running the same versions, etc. This offseason should see more action on that level since it appears that the green light has been given for expansion to new parks. I’m not sure yet what my involvement will be in that process, but I’m certainly interested in at least watching an installation. I think I covered the equipment we use above.

Q: How many cameras and where exactly on the field level are they mounted? Does it vary per field? I have heard many different configurations described. The one that I have heard most is that there are cameras mounted on the field level and the mezzanine, two per level. Is that correct?

A: There are four cameras that operate in sets of two. The Field level (mounted at first and third base dugouts) cameras take the still photos of the batter. The upper level (first and third base side) cameras do the tracking.

Q: If so, one concern I have is that the batter himself may block up to two of those camera angles (i.e, a field-level and a mezzanine level on the side from which he is batting). Is that the case? Doesn't that limit the system's accuracy?

A: To the best of my understanding, the batter blocking the tracking camera is not an issue since it is tracked from both sides. In addition, for the field level cameras, the important one is across from the batter, so he can’t block it. The biggest problem we have is when the field level cameras are blocked from the dugout, thus rendering the still photo useless. In that situation, I can’t set a strike zone, and thus the pitch is thrown out. This tends to happen in later innings when the game is on the line and players are standing on the top step of the dugout. If the third base camera is blocked, I can’t set the strike zone on a left-handed batter, and vice versa. This doesn’t happen very often, but is an obstacle to total accuracy.

Q: I still think that blocking the back camera(s) may interfere with the system's ability to triangulate a pitch given that its depth perception might be off. Have you seen anything like this in the games you've worked? Has it ever been the case that both sets of cameras have been obscured—say by the batter and an on-deck batter or fans—or have malfunctioned—say by inclement weather—at the same time on a given pitch? What would the system do and what would you do as an operator in that case?

A: The dugout level cameras are not involved in tracking. Hence, blocking it has no effect on the accuracy of the track. The only thing that can be hurt by blocking dugout level cameras is the operator’s ability to set the strike zone. On occasion, it is impossible. Those pitches are thrown out. The mezzanine cameras have an angle down to the plate that renders concerns about blocking mostly moot. Fans cannot obscure the upper set of cameras since in every park I’ve seen, they are hung out from the front of an upper deck, almost looking back at the batter, and down at the plate over the batter, if you can follow me. I have heard of cases where shadows have caused an issue with tracking either degrading the quality of a track or making the ball impossible to track altogether. Pitches that are not tracked obviously cannot be graded. This reduces the sample size in any game, but does not count against the umpire. There are adjustments that an operator can make to overcome some of these difficulties; clearly it depends on the magnitude of the problem. I have not yet had any equipment malfunction as regards the cameras. Regardless of the problem, it is my job as an operator to a) log what happened, b) pass that information on, and c) remedy the situation as soon as possible. It is possible that some problems could not be remedied until after the conclusion of a game. (if, say, Curt Schilling attacked another camera during the course of an actual game.)

One other point to understand in this is that not being able to see the ball for a portion of the time isn’t terribly important. The ball is tracked at a given number of points; between those points, there’s only so much that the ball can possibly do, especially given the minuscule amount of time it takes a ball traveling even as slow as 75 MPH to travel the less than 60 feet from the pitcher’s hand to the plate. Physically, only certain paths are possible, and those can be determined mathematically. The software that does all of this is outside my expertise, and is really, at its heart, what the UIS is all about. It is what QuesTec has to sell.

Q: Here's a long question with a lot of setup: A strike is defined as "a legal pitch when so called by the umpire, which (b) Is not struck at, if any part of the ball passes through any part of the strike zone" while the strike zone is defined as "that area over home plate". Robert K. Adair demonstrated in "The Physics of Baseball" that a 65-mile-per-hour curveball can drop out of the strike zone as it passes over home plate and by the same token a high, 95-mile-per-hour fastball can drop into the strike zone while traveling over home plate.

Does QuesTec track balls over the entire plate to take this into account? I have read otherwise, i.e., that the call is made just at the back of the plate. Harold Reynolds of "Baseball Tonight" claimed that the system loses the ball "3 feet from the plate", meaning at the dirt in front of home. Using one point to measure the location of the ball is inherently inaccurate.

A: The system produces graphical representations of the track that provide at least 4 points of data over the plate. These often show a pitch diving into the top or out of the bottom of the strike zone. While this may not be enough to provide a constant track, 4 different data points is clearly more than the umpire has while watching a 95 MPH fastball. I cannot verify if those are actual track points or extrapolations of the flight of the ball based on a smaller number of actual track points. Is it infallible? No. But there is information available to umpire supervisors to allow them to make those calls when they review the data.

Q: Here's another one with a bit of setup: The strike zone definition further states that "the upper limit of [the strike zone] is a horizontal line at the midpoint between the top of the shoulders and the top of the uniform pants, and the lower level is a line at the hallow beneath the knee cap. The Strike Zone shall be determined from the batter’s stance as the batter is prepared to swing at a pitched ball."

How can the system possibly measure these things while the batter is possibly moving (think Joe Morgan) while in his stance? The QuesTec system would have to be able to determine where the batter’s knee caps, belt, and shoulder are and to calculate the position of the horizontal line that defines the upper and lower bounds of the strike zone from these body parts.

Not only that, the system must be able to determine when the batter is prepared to swing, even one who has a lot of motion while he is in the box, to make the measurements of the zone.

A: There are occasions where this is very difficult--for example a called pitch on a bunt—that arise in situations when there are extreme variations in a batter’s stance. From the time a pitch is released to the time the ball crosses the plate there is usually very little alteration in the strike zone for a given batter. Sit in good seats behind the dugout some time and watch the batter (not the pitcher). You’ll see what I mean. As I mentioned, I make those determinations based on the still photos. There are possibly limitations to this system, but they are far less than have been described. I’d like to stress that a different strike zone is set for each and every called pitch during the course of a game.

Q: Specifically, regarding the "2.5 ball widths" above the belt that the system uses to demarcate the top of the strike zone, I see this as potentially problematic given the definition of a strike. This standard might fit the average batter, but what above extremely tall players like Richie Sexson or extremely short players and those who squat in their stance, like Pete Rose did? Is there an adjustment made in this instance?

A: I think you’re right. There are situations in which this probably would result in a somewhat large or small Questec set strike zone. This is why all of the information I described is contained in the Pitch Table that is given to the Ump and MLB. First of all, not every call is right or wrong based on the top or bottom of the strike zone. For example, if the pitch is a foot outside, it doesn’t matter about the top or the bottom of the zone. Secondly, the inclusion of the still photo from which the strike zone and the video clip of the pitch allows the operator AND MLB/the Ump to look over the data and form their own interpretations of a given call. Let’s say that an ump calls a given pitch a “ball.” The UIS calls it a strike. It is possible to double check that decision and account for any error or extenuating circumstances that may explain the discrepancy. If the system gives the Ump an “N” score based on what is clearly a faulty strike zone setting, I can throw it out, and MLB can throw it out in assessing umpire accuracy. Overall, I’d say that the top of the strike zone is probably the most error-free of the four areas. Inside and outside calls are wrong far more often.

Q: The umps state that a technician sets the coordinates of the system on the first pitch to the first batter. Given the size of that batter or the calibration method used, the strike zone may or may not be accurate. According to the New York Times, a computer technician sets the strike zone from a snapshot taken as the first pitch to a batter was on the way to the plate. Is either one of these scenarios true? Is the actual method more accurate and if so, how can we be sure of that?

A: Neither of these descriptions is accurate. The system is calibrated to home plate before each game; these (inside and outside edges of the plate) coordinates are fixed. The operator sets the strike zone from a still photo for EACH called pitch; as I mentioned before, this can result in up to six different strike zones in a given at bat for each player (2 called strikes, 4 balls; 3 called strikes, 3 balls). The strike zone is set for every called pitch to every batter. This method is clearly more accurate than what has been described. Again, there may be flaws in the method, but they are much different than what has been described and will result in a much lower degree of error. Given that there is a margin of error built into the system to give umpires the benefit of the doubt on their calls, I’d suggest that the amount of error is actually quite low.

Q: The umpires said in a 2003 statement that the computer is "heavily dependent upon decisions and actions by the QuesTec ballpark operators, almost none of whom have any experience in professional baseball" and that it "often incorrectly interprets the strike zone," producing "unacceptable inconsistencies between strike zones from ballpark to ballpark and from day-to-day in the same ballpark." Incidentally this contradicts their complaint in the previous question. Are the zone settings recalibrated throughout the game and do they remain accurate? If so, how can we be assured of their accuracy?

A: Aside from setting the top and bottom of the strike zone, there is nothing left to my judgment. I am simply recording what happened. There may be slight inconsistencies from operator to operator with regards to setting the strike zone. I don’t have any way to verify or refute those sorts of claims, as I’ve only seen my strike zones and those of the other operators I’ve worked with. We all worked together and I’d say that we were consistent with each other.

Further, while I respect the point about lacking experience in professional baseball, I don’t think that it’s particularly important to operating the system properly. Even someone who has never watched a baseball game can locate the hollow of the knee and the top of the belt buckle on a still photo. Other than that, there is almost no judgment exercised by the operators. The only other case of judgment is in deciding to throw out pitches where the track or the still photo is bad. Again, these judgments do not depend on baseball experience, but can be trained with relative ease.

Q: Tony Massarotti of "The Boston Herald" said that UIS may be causing more walks to be issued and also may not be achieving one of its proposed goals-to speed up games. What do you have to say about this?

A: The data on this score is inconclusive, and any differences between QuesTec and non-QuesTec parks seem to be so minimal as to be statistically insignificant. The obvious solution is to put it in all parks. It’s hard for me to say if the system has caused more walks, since watching a game in the stands and scoring a game on the system are very different experiences. There are so many variables involved: quality of pitcher, quality of opponent, etc, etc. My gut instinct is that there may be a few more outside pitches being called balls, but I have absolutely no way of proving or verifying that instinct. On the other hand, more high pitches are being called strikes. It’s important to remember that any of us watches only a small slice of the total number of games that are played; I only worked about 40 games last year in my park and attended about 15 others, mostly in other parks. That’s a very small percentage of the total number of games played, so observations that I make are limited by that small sample size and the fact that almost all of the games that I watched involved one particular team.

Q: Umpire crew chief Randy Marsh once made these comments regarding QuesTec: "In the past, there have been pitches that are a little off the plate that are hittable pitches that we’d call strikes. If we call them strikes now, we’re wrong. You have QuesTec looking over your shoulder every single pitch." Any comments?

A: If the rule book says those are balls, what difference does it make if they are hittable? Who defines “hittable”? What is hittable to Vlad Guerrero is not hittable to a lot of other players. These pitches (the outside ones especially) are ones that I’ve noticed produce the majority of mistakes as judged by the system. If they were called strikes before, then those calls were wrong. The point of the system is to help umpires call the strike zone as the rule book defines it. We should only judge it on how well it accomplishes that goal.

Q: Author of "The Diamond Appraised", Craig Wright said in a study on ESPN.com that "With the supposed pressure of their calls being tracked by QuesTec, the umpires are calling a slightly higher — not lower — percentage of strikes in 2003 than they did in 2002…[T]he emphasis on calling higher strikes has resulted in a higher percentage of strike calls." Do you think that having QuesTec monitoring games has affected the way umpires call games or at least how they call certain pitches?

A: First off, the UIS has been in operation since 2002, with some operation in 2001. So differences between 2002 and 2003 are somewhat inconclusive in showing the impact of the system on umpire calls. I would hope the monitoring has affected the way in which umpires call games. That’s the point of the system. There was a time when ball/strike calls were truly outrageous. Consistency is always more important than accuracy, and the system measures both. What I am most disheartened by are reports that the strike zones of various umpires vary from park to park (as in the Schilling quotation below). That’s not a flaw with the system, but with that umpire.

Q: Can you explain more fully how the system measures both consistency and accuracy?

A: What the consistency algorithm measures is if pitches in the same location are called the same thing throughout the game. Basically, the definition of the strike zone creates a universe of possible coordinates for the result “strike” on each pitch. Since each pitch has certain coordinates assigned to it, pitches can be compared, and the calls made on those pitches can be compared. I assume that the math takes into account distance from the strike zone, since that of course varies from pitch to pitch. What the system generates is a “consistency score” from 0-100, and a color keyed graphic that shows where the ump was accurate and where he was inaccurate. On average, low and outside are the least accurate and often the least consistent calls. Consistency scores tend to be higher than accuracy scores, but the accuracy scores are raw, unadjusted totals. Of course, there are variations from umpire to umpire. The final accuracy scores assigned to umpires (as I understand it) include all “C” and “A” grades, and also factor in any pitches that are removed due to problems with the data or extenuating circumstances.

Q: Are those consistency scores per pitcher or across all pitchers? I ask that since if a guy who those a 95-mile-per-hour fastball is followed by a junkballer, consistency may not necessarily be a good thing or even a possibility. If Umpire A officiates in a Randy Johnson (LHP)-Curt Schilling (RHP) matchup in which both last deep (eight innings?) in the game and are replaced by RHP Mariano Rivera and LHP Alan Embree for one inning. Umpire B calls a game between the Phils' Corey Lidle and the former Phil Eric Milton. Both are yanked by the third inning and both teams use five or six relievers, a few right-handers and a few lefties, with different types of pitches thrown at different speeds. Could it be unfair to the ump B given that finding consistency under those circumstances is more difficult?

A: The consistency scores are for the game as a whole. Consistency as judged by UIS is strictly about location; I’d think that that’s a good thing even across different pitcher types. However, I would think that as an umpire it would be particularly difficult to maintain that consistency given different pitcher types. The pitch table lists who the pitchers were, so there is an ability to adjust the raw data to reflect the situations you refer to. I would assume (though I do not know for sure) that MLB would take these sorts of things into account in interpreting and adjusting the raw scores. What this situation highlights is the reality of this (or any) system: it’s a tool to help humans, not a final answer.
As an aside, I have gotten the impression that many people think that the output of the system is the final word; that isn’t the case. When you run productivity reports at an office, adjustments are made based on project-load, difficulty of client, etc; the same applies here to UIS. What that means is that there is another level to the use of UIS that I as an operator am not part of—the interpretation of the data by MLB and umpire supervisors. I’d like to learn more about that myself. More than anything, the key to appropriate usage of the UIS lies in the interpretation and enforcement mechanisms at the MLB level; these are still uncharted territory as far as public knowledge goes. Like most assessment systems, this one is neither good nor bad in and of itself. It lies in the hands of MLB and the umpire supervisors to forge that direction. From my perspective, it appears that the compromise reached between the Umpires and MLB recognizes precisely that distinction, and allows for umpires judged substandard by the system to work closely with humans to improve and assess their performance. Thus, UIS is a tool to identify problems, not to make final judgments.

Q: Curt Schilling made these statements after his infamous run-in with a QuesTec camera: "I said something to one of the umpires about it and he said 'Do us a favor and break the other one'… The QuesTec system in this ballpark is a joke. The umpires have admitted it. They hate it. In the last three starts I’ve made here, multiple times umpires have said to the catcher, 'It’s a pitch I want to call a strike but the machine won’t let me.'" As an IT professional myself, I know that hearing this sort of feedback always warms my heart. What do you have to say about it?

A: Again, I think this is the point of the system—to enforce the strike zone as defined in the rule book. While I respect the experience of the umpires, I don’t think they have the right to “want” to call something a strike. They have a job to do, and most do it very well. Taken in the proper context, this system can only help them do that job better. Clearly Schilling hasn’t been hurt too much by the system; he had a pretty good season this year in a QuesTec park.

Q: After the Schilling incident, then-Diamondback manager Bob Brenly said, "They call balls and strikes differently in the ballparks where [QuesTec]’s set up. If the system is so good and the ball tracks so well, why do you need a ball-strike umpire? You could have a green light go on out on the scoreboard if it’s a ball and a red light if it’s a strike." I know that it's a backhanded compliment but do you think the players and on-field personnel would welcome ball-strike calls done in this manner?

A: I doubt if they would welcome it. Honestly, I’m not sure what I think of the idea. I like the idea of umpires making the calls. As an aside, the system as currently constructed can’t make real-time ball/strike calls. Determinations of accuracy are not made until after the game. It also has no way of recording swings or fouls, so we’d still need an umpire to do that. I’d hazard a guess, however, that we will someday see a system that can overcome those difficulties, but it’s a long way off.

Q: Brenly, as is his wont, went on to say, "The strike zone has always been very subjective, and the players know that going in. You put it [i.e., QuesTec] up in a ballpark, and the umpires are calling what they think they’re supposed to call. If you want a consistent strike zone, you’ve got to put QuesTec in all 30 ballparks." In a variant, do you think the players, etc. would be more welcoming of the system if it were used universally?

A: Aren’t umpires supposed to call “what they think they’re supposed to call?” I think expanding to all 30 parks would improve things dramatically. While I don’t know the details of umpire scheduling, it has occurred to me that some umpires could have more games in QuesTec-enabled parks than others. If true, that would mean that some umpires are being graded/assessed on a smaller sample size which is more likely to be influenced by an off-game or the like. Put it in every park, and some of the claims about unfairness simply disappear. On the other hand, it is disingenuous of umpire representatives to make this claim since it was their union that filed the grievance that held up installation in all 30 parks. I agree with Brenly on the universality comment 100%. When consistency is the goal, everyone must play on the same playing field.

Q. QuesTec claims that the system is accurate to one-half inch. Do you know under what conditions that benchmark was made? Do you see that sort of accuracy consistently when operating the system?

A: I do not know the details of that claim. I have read that the tracking technology was improved by a collaboration with Titan, a company that makes missile tracking devices. It’s difficult for me to say if I see that kind of accuracy. I don’t know what 1 inch vs. ½ inch accuracy would look like. In addition, almost everything I look at is either a graphical representation or televised, so nothing is its actual size. As a result, it’s hard for me to say that it’s accurate to this or that standard. What I can say is that watching the system has shown me the distorting effects of the off-center centerfield camera that is used to show almost every pitch on TV. It’s also gotten me to ignore the catcher’s mitt, which is the center of attention from the centerfield camera. My sense of what is and isn’t a strike has changed as a result—almost exclusively to the benefit of the umpires.

Q: Do you feel that any of the camera views one sees on TV is accurate? How about the view from the blimp? My favorite has to be the groundhog cam that Fox dubbed, I believe, Diamond View.

A: Sadly, I think the most accurate was the ESPN High Center camera. It really wasn’t any fun to watch, but it did give you a straight on view of the plate, which is good for inside-outside calls. Relatively useless, however, for top/bot judgements. I really liked the camera in the dirt that Fox used. I thought it really added something to the telecast, which is more than you can say for most gadgets. The best cameras would be (for top/bot) the ones we used, combined with one suspended just over the top of the plate, from above. The combination of those would probably work best for visuals. But since I’ve never seen the strictly overhead view zoomed in on the plate, I don’t know how well it would work. It seems like a good idea.

Q: Regarding calibration, are all these systems calibrated against some absolute to minimize variation across parks?

A: There is no universal standard that I am aware of; the system is calibrated to the specific park before each game. Those files are saved so they can be checked if the results seem suspicious for some reason. Given that the shape of the diamond and home plate are consistent across parks, I’d guess that there is almost no variation in this area, since those are the benchmarks that are used to calibrate the tracking cameras. What’s interesting is that I’ve found that after the system taught me about the distorting effect of the centerfield camera, I could generally predict what it would call pitches as I saw them. This predictability gives me confidence in the system at least as it operates in one park. It doesn’t say anything about differences from park to park. But it tells me that from day-to-day, umpires are being graded by a system that a) can teach even a relative amateur observer like me and b) is consistent across games.

Q: Peter Gammons stated on ESPN.com that he believes that "21-year-old [QuesTec] operators must be replaced by ex-umpires." What is your reaction to this statement? To Gammons election to the broadcasters' branch of the Hall of Fame? I withdraw the last question.

A: I don’t know any 21 year old QuesTec operators. For the most part, UIS operators, myself included, are at least in their mid-20s. Another operator I met was in his 30’s. As I explained above, I don’t think that bringing in ex-umpires would really change anything about how the system operates. It couldn’t hurt, so long as ex-umpires with the requisite computer skills could be found, but it’s really more of a computer job than a baseball job. What’s important is dedication to doing the job properly even when the work devolves into drudgery. (for example, setting the strike zone for 450 pitches in a 16 inning game). Every operator I’ve met has that dedication. Additionally, I’d be willing to bet that most former umpires aren’t willing to work for what they pay us. Gammons’ comment is typical talking head kind of stuff: it sounds like a strong stand, but it dissolves under any sort of examination. It’s like saying that all grocery store cashiers should have advanced degrees in mathematics. Sure, that’d be great, but do we really need PhDs running cash registers? And what would be the benefit? The place for those with experience is interpreting, not producing, the data.

Q: In a study on ESPN.com in 2003, Baseball Prospectus found that QuesTec does not bring consistency. Hitters' umps (i.e., ones that call more balls) are not reined in but rather overcompensate and call more strikes: "Our numbers reveal that QuesTec has made a difference for individual umpires, each of whom has adapted to the system differently, and unpredictably." Do you feel that QuesTec is an effective tool in training home plate umpires, especially after MLB redefined the strike zone in recent years?

A: I think it can be an effective tool, depending on how it’s used. I am not privy to the way in which MLB or umpire supervisors meet with individual umpires to discuss their performances. Like any other method of assessment, it’s only as effective as the people who are employing it. In any business, if employees know that they can ignore their reviews, there’s less of a tendency to work towards achieving the goals set out in their performance plan. The same applies here. The tool will work as well as the people using it let it work. The system, after all, is only a tool to aid the human eye. People using UIS effectively would find out which umpires were failing to improve their performance and sanction them in some mutually agreeable way. This, however, is subject to the collective bargaining process. I was pleased to see the recent agreement because it seems to strike the proper balance and understands that the UIS is a tool, and nothing more.

The Prospectus study was not some of their best work. There’s no attempt to control for different quality of teams involved in these games. The discussion of individual umpires does point out some trends, but the biggest effects that they can find are on the order of 2-3%. This means approximately 3 pitches a game. Without data on locations that are called differently, it is impossible to conclude what the effect on these umps is. If we took a random sample of games an arbitrarily labeled them group 1 and group 2, there is the possibility that the same discrepancies would show up. What I’m saying is that the science of umpire assessment is something new to the baseball community and the methods that have been publicly discussed seem a bit weak methodologically. I’d love to see the development of some more descriptive metrics using actual QuesTec data.

One last point: The UIS is a training tool. A study done in 2003 is only going to show the early stages of this change; it’s also done in the middle of a pitched fight between the WUA and MLB over the used of the system. Will results really be consistent in that context? I don’t want to sound like an apologist here, but that seems like jumping the gun to me.

Q: Robert K. Adair, author of "The Physics of Baseball" and a consultant hired by the umpires union to investigate QuesTec, found that "The umpire’s strike zone and the QuesTec strike zone are consistent, but in different ways… the QuesTec strike zone does closely mirror the strike zone defined in the rulebook." Do you feel that is a validation of the system?

A: It sounds like it. I suppose it depends on what he means by “different ways.” Umpires by and large have been consistent over the years to their own strike zones; if what Adair means is that the UIS more closely approximates the rulebook strike zone, then yes, I think that is a validation of the system, especially if viewed as a training tool. On the other hand, it could be suggesting that individual umpire consistency is more important than adherence to the rule book. In that case, I’d disagree. We’re all here to make sure that the rules are followed consistently.

Q: He went on to say that while a certain number of pitches do give incorrect readings, "Operators are given leave to kick those out, and typically they tend to throw out six or seven pitches per game." Is that the case?

A: We are given leave to remove pitches that are measured incorrectly for whatever reason. Some games none are removed; some games 8 are. I don’t know what an accurate average would be since I have experience in only one park. Each park generates its own unique circumstances (shadows, dugout design, lighting, open/dome configuration, etc) that might contribute to a pitch being thrown out. We also keep a log of each pitch that is thrown out and the information (but not the grade) is passed on to QT and MLB. The total number of pitches thus removed is kept somewhere (I don’t know what the number is or where it’s kept, but someone has that information to keep track of what’s happening with the system).

Q: My old friend Joe Morgan also wrote about QuesTec on ESPN.com: "Umpiring officials showed me the QuesTec system and explained why they felt it wasn't accurate. And after seeing their demonstration, I could see what they were talking about (from ballpark to ballpark, similar pitches to the same batter were called differently by the computer)…K-Zone, ESPN’s strike-zone innovation…is accurate to within four-tenths of an inch. I don’t use it to grade the umpires but to demonstrate how the pitcher uses different parts of the zone or misses just off the plate. I trust K-Zone because I know how it works and I know it’s accurate." Can you discuss K-Zone and compare it to QuesTec UIS? Morgan claims that it is the more accurate tool. What is your reaction to this?

A: I’ve read criticism of Morgan’s comments elsewhere that has not been charitable. I’d like to take a slightly different tack. Taken literally, what Morgan is saying is that the same batter received two similar pitches from two different pitchers in two different ballparks, and the results were called differently by the system. Why is that result surprising? The reason why a system like this was designed in the first place was precisely because of this situation: what appears to be true isn’t always. I don’t know precisely what examples Morgan looked at, but I’d submit that it’s at least possible that the reason the pitches were called differently is because they WERE different. What we see watching the game on TV is much different than what the system or the umpires see. If Morgan’s observations about K-Zone were accompanied by some specific examples, perhaps it’d be a more discussable point. For now, all we have is his word on it. As much as I love Joe as a player (I’m a Reds fan) I have no reason to believe that his observations about a computer system are particularly compelling. I find the assertion that he “trusts” the K-Zone system more because he “knows” that it’s accurate to be troubling. Is he doubting UIS simply because he doesn’t know how it works?

I don’t know anything about how the K-Zone system works or what its specifications are. If it is truly 1/10 of an inch more accurate than the UIS, then perhaps MLB should be using it instead. In addition, I don’t know what process is used to overlay a strike zone with the location data. That’s the real key to accuracy in judging ball/strike calls. I’d love to see an explanation of how that system operates.

In any case, the argument shouldn’t be about this system or that system. That should be subject to independent verification since it is a question that can be answered by testing. As near as I can tell, the main argument between MLB and the umpires has been about the use of any system whatsoever. Hopefully, that argument is over.

Q: Regarding the deal signed by MLB and the umpires, it seems that MLB is now free to install QuesTec in all major-league parks while the umps got a concession in how QuesTec will be used to evaluate their rank: "[U]mpires whose ball-and-strike calls are rated below standard by QuesTec will be evaluated by umpire supervisors based on videotape and in-game inspection." Do you see this as a victory for the QuesTec system and will it remain effective as an umpire evaluation tool in this more, shall we say, watered-down scenario?

A: UIS shouldn’t be about victory. What this agreement was really about was MLB getting umpires to agree that MLB had the right to use some sort of system to assess and improve their performance. This tool can be used in many ways. This sounds like it will be used to identify umpires who are less effective and turn them over to someone for remedial work. That sounds perfectly reasonable to me, and doesn’t seem to make UIS any less effective. The point is to identify umpires who aren’t doing their job correctly. One part of that job is calling balls and strikes as defined in the rule book. The system generates data that is subject to interpretation. If there is a place for former umpires, it’s in the interpretation of that data and the implementation of plans to improve umpire performance. I can run this system. I have no idea how to work with an umpire on improving his results since I’ve never stood behind the catcher while a 95 mph pitch comes rolling in.

Q: In your opinion, will the system eventually replace the umps in calling balls and strikes?

A: This system will not replace the umps, for technological reasons outlined above. It simply is not designed to take the place of human beings. A different system might. Something involving GPS locators in the balls and laser beams worn by the players, etc, etc. I’ve tried to imagine ways to replace the umpires and that’s the best I’ve come up with.

The issue is not accuracy, but about timing. The results of this system are produced after the game is over. Setting a strike zone for this system in real time will involve a different level of invasiveness, I think. Something like creating an “electric eye” beamed upwards from the plate and out from players. We’re just not there yet. Even our “smart bombs” aim for areas larger than the typical strike zone, and they only have to aim in two dimensions.

Q: What are QuesTec's plans, at least the ones for public consumption, regarding MLB and the UIS system?

A: I’m not aware of any public announcements on the subject. My guess is that they will be forthcoming as MLB and QuesTec assess what parks can easily be fitted with the system within the limited time left before the season begins. No action could be taken until the umpires’ grievance was settled.

Q: Are there plans to scorecard calls at first, fair/foul calls, catches against the wall, tag calls, balks, knowledge of the rules, or any of the other responsibilities of the umps that you are aware of?

A: I am not aware of any such plans. They would involve a completely different kind of system than the UIS.

Q: What about using QuesTec as replay tool in playoff games or the like?

A: The system as currently designed wouldn’t allow it. Even if I wanted to, I can’t go back and process an individual pitch after it happened. The system is designed only to handle the data in one batch; one pitch can’t be pulled out.

I’m not a big fan of replay in general. I think its use in football (along with other factors) made televised football almost mind-numbingly dull to watch. Can you imagine watching Game 7 of the Red Sox-Yankees series after Torre throws a red flag on the field to challenge a ball-strike call? Tim McCarver and Joe Buck would remind us about the need for indisputable visual evidence and argue with each other about what the Ques-play would show. And they’d have at least a minute and a half to do it. No thanks. That same Red Sox-Yankees series showed us that dedicated, properly trained umpires working as a crew can get calls right. I salute the umpires for those calls.

Q: In my opinion, QuesTec is caught in a power struggle over the strike zone that has been playing out since at least the mid-Eighties between the umps and the owners. I conducted a study that found in part that the power surge of the mid-Eighties led to the batter's box to be obliterated in an effort to get to outside pitches. As a result the umps widened and flattened the strike zone, leading to a rules redefinition by MLB to "restore" the zone. If the batter’s box were enforced in the first place, the umps wouldn't have had to give the pitchers an extra few inches on the outside of the plate. If they enforced the batter's box now, strike calling could go back to the old days. What's your opinion?

A: I have been infuriated by the disregard for the batter’s box rule ever since I moved to a big league city and got a chance to watch at least 15-20 games a season in person. In my mind, that’s one of the top five problems in the game today. In any game you watch, the back of the box is covered up by the second inning and you have guys setting up 4, 5, sometimes 6 inches back of the line. I hope MLB encourages stricter enforcement of this rule.

I think you’re right about outside pitches in general. The plate shifted several inches to the outside. I see this result in the UIS data; umpires consistently give some on the outside and call pitches balls that are on the inside corner. Enforce the batter’s box, and we’d see some differences. Unfortunately, that’s something that the UIS can have no impact on whatsoever.

I think the system has been caught in the middle of a power struggle between MLB and the umps that began some time ago, perhaps even with the infamous “enforce the balk rule” directive. The fact that each league ran its own umps led to the growth of fiefdoms, where you had the employees telling the employers how they would go about their business. Recently, that balance of power has changed, and the adoption of the system is just one part of that shift. I hope the recent agreement heralds a new era of cooperation between MLB and the umpires. After all, don’t we all just want to see the best baseball possible?

I love doing this job. It’s forced me to refine the way in which I watch and think about the game. I watch pitch-to-pitch now instead of at-bat to at-bat, and I think I understand more about how the game is played as a result. I don’t have a strong opinion about this particular system one way or the other, but I do firmly believe that the game can be improved by objective assessment of umpire performance. Thinking about that issue has given me a great deal of respect for the skill that umpires display on a regular basis, and gotten me more interested in the realities of what umpires do in every game. The best umps are invisible to fans, but they deserve our gratitude. They’re professionals who work hard to be the best at what they do. Hopefully, the UIS can help them achieve that goal.

Mike's Epilogue (A Quinn Martin Production): I still have some issues with UIS—how well the ball is tracked above the plate, using 2.5 ball widths above the belt uniformly to define the upper limit of the strike zone. However, I feel that the established procedures along with the dedication and professionalism of the operators (if this one is any indication), help overcome the limitations of the system itself. The ump is given the benefit of the doubt on extremely close calls (i.e., within one-half inch). The calls are photographed and reviewed. The operator can throw out calls on which he feels the system did not function optimally. All this says that the point is to get the call right if possible, to curb the poor calls but not to nitpick on borderline calls.

There has been a lot of inaccurate criticism published about UIS. I have asked every reasonable question, and some not so reasonable, that I've read elsewhere or that I've thought up myself in my wee brain, and I feel that they have been answered sufficiently.

I'm not saying that there cannot be improvements made to the system nor that we should not continue to raise questions about it. Also, QuesTec must continue not only to adhere to its established standards and procedures, but it should strive to improve them as the system is rolled out throughout the majors.

What I am saying is that I am finally a believer. Like E.G. Marshall in "12 Angry Men" I have finally set aside every misgiving, and can give it the thumbs-up. I think UIS is a step in the right direction for the sport. Rolling it out to the remaining twenty stadiums, as long as it is in a controlled manner and the new operators that will be employed are well trained, should be a good thing for baseball in general and us as fans. Now let's just hope they don't screw it up.