The Quest for QuesTec: What Is The Umpire Information System?

Much has been said about the umpire rating system that Major League Baseball is trying to put in place. The World Umpires Association says that it filed a grievance against owners last Friday over the system although I don't think they know what the suit is about. Their lawyer said the umpires think that the system "cannot properly evaluate pitches, especially breaking balls." On Sunday, the umpires demanded that Questec, the company who developed the system, remove a reference to the umps that said that they supported the system. The umpires wrote a letter to the company that we know about because they released it to the press-isn't that convenient-in which they said:

The technology has never been adequately explained to umpires or our legal and technical consultants. Questec Inc. has not answered questions from the WUA, and the office of the commissioner of baseball has refused to provide requested information, which refusal led to the WUA's latest grievance. ...
Even if the Questec system were more accurate, there remain legitimate questions as to whether this device belongs in major league baseball. Like last year's 'pitch count' gimmick, Questec interjects an extraneous element into the game and pressures umpires to compete with the machine, rather than giving their full attention to calling the game as they see it unfold before them.

The umpires bristle when anyone questions their authority to call a game in seemingly whatever fashion they see fit. The pitch count reference is to a misguided attempt made last year by the commissioner's office taking the average pitch count when each ump is behind the plate and ranking them. The umpires claimed that the commissioner was pressuring them to call more strike and reduce pitch counts. The WUA filed a grievance, and pitch counts were no longer used as an evaluation tool. At first blush, the comparison seems unfounded-pitch counts as an evaluation tool a) are dependent on the pitchers involved and the situations encountered in the game and b) don't measure whether the call was accurate or not. But having a tool that could accurately call balls and strikes would be a handy evaluation tool. Were the umpires being too possessive-"the balls and strikes are ours"-or is this a poor evaluation tool?

I visited the QuesTec, Inc. site to find out more about the umpire rating system from the horse's mouth (or at least its mouthpiece). They have an announcement on the site dated February 2001 of last year that states that MLB signed a five-year deal with Questec for a new version of PitchTrax, a "pitch measurement technology in support of MLB's previously announced strike zone initiatives." The agreement also included the use of the then new Umpire Information System (UIS) for five years. There were no further announcements as regards MLB.

The statement goes on to trumpet the PitchTrax product, how it has been used exclusively by Fox and, that it was modified to meet MLB's needs. It still contains the quote that so offended the umps to impel them to write their missive ("In general they {the umps} support it!''), including the excessive punctuation.

UIS was beta-tested during the Arizona Fall League's 2000 season. Major League umpires and officials were given access to the system. It adhered to the accuracy requirements established by the commissioner's office (though it does not specify what they were). It was then to be tested in the 2001 spring training games in Arizona. There are no follow-ups to indicate if that testing was in actuality conducted and if so, how the system fared.

Besides the usual self-aggrandizing typical of leading-edge companies (Wow, Scientific American did an article about you? Well, who owns them? Any relation?), the article further describes the technology employed:

. The ball tracking component uses cameras mounted in the stands off the first and third base lines to follow the ball as it leaves the pitcher's hand until it crosses the plate. Along the way, multiple track points are measured to precisely locate the ball in space and time. This information is then used to measure the speed, placement, and curvature of the pitch along its entire path. The entire process is fully automatic including detection of the start of the pitch, tracking of the ball, location computations, and identification of non-baseball objects such as birds or wind swept debris moving through the field of view. No changes are made to the ball, the field of play, or any other aspect of the game, to work with QuesTec technology. The tracking technology was originally developed for the US military and the company has adapted it to sports applications.

The U.S. Military? Well, it's assuring to know that are tax dollars are being put to good use. That's all fine and well, but on the product page for UIS, the state:

The UIS uses QuesTec's proprietary measurement technology that analyzes video from cameras mounted in the rafters of each ballpark to precisely locate the ball throughout the pitch corridor. Additional cameras are mounted at the field level to measure the strike zone for each individual batter, for each individual pitch, for each at bat. This information is compiled on a CD ROM disk and given to the home plate umpire immediately following each game. (Italics mine)

It's cool that the home plate ump gets a CD at the end of each game. But this system says it uses a camera in the rafters and cameras at field level whereas the PitchTrax system uses cameras mounted on the first base and third base lines. This raises many questions: How many cameras and where exactly on the field level are they mounted? At the same point as in the PitchTrax system? If UIS was ostensibly developed from PitchTrax, why are different camera angles used (at least the one in the rafters)? Were there issues with PitchTrax? What is meant by "the rafters"? Each stadium has a different infrastructure, some are domed, some have retractable roofs, and some are open. How are these differences accounted for? Later in the article, it states that the camera angles are the same as with PitchTrax, which is it?

They claim that the system is accurate to .5 inch. This sound good but I have some questions. How did they ascertain this given that it is extremely difficult to measure the trajectory of a ball in flight? Did they use a more advanced system to check the accuracy? If so, why not use the more accurate system as the UIS system? How do you know that the verification system is accurate? If there is no verification system, then how was the accuracy checking performed? By eye? Is that .5 inch for straight pitches or curving and breaking ones? If they tested curveballs and breaking balls, how did they do so? Were they simulated or did they call up Bert Blyleven one day and ask him to throw a bunch of curveballs? What speed is that up to? Is it more accurate at the point of release, in flight, over the plate, or when the catcher catches the ball? Over the plate is the only one that we are concerned with when calling balls and strikes. If it was over the plate was it the front of the plate, the middle, the back, or the entire length of the plate? Does .5 inch meet the commissioner's requirements? If so, did it exceed them? Is it better than the given umpire? Given that .5 inch at the corners is much more important than .5 inch a foot outside, does the system remain that accurate no matter what the ball's trajectory is: low, low, left right? What about balls that bounce before crossing home plate? If they bounce into the strike zone, are they considered strikes by the system? And so on. Try it. Make up your own list.

Then they do say that UIS differs from PitchTrax. UIS "uses different cameras, modified software, and a different calibration process to increase accuracy." Why is the software modified and how was it tested? How are the camera different, in their placement, the types of cameras, what they film, etc?

According to the site, UIS is only available in 14 stadiums as of 2002. Four parks had it last year, and ten more were scheduled for this year. Why the slow rollout if the tool is ready to go? Isn't it unfair not to use the system universally if it's to evaluate the umpires? If only 14 parks are involved aren't those umpires that frequently work in those parks singled out? Doesn't it skew the evaluation if the number of umpires evaluated is lower than possible? Why were those stadiums selected to have the system installed? Were they selected for a certain reason (e.g. they were more favorable to getting accurate results using the system than the others) or was it random?

QuesTec retains ownership of the product licensing it to MLB. Where else will it be employed? Will MLB be in charge of its development? Will the current stadiums be upgraded as the system develops or will some stadiums be in different releases, possibly tainting the accuracy?
Their last FAQ is telling:

Why is this deal important to QuesTec?

Gaining acceptance from both MLB and the umpires for the accuracy, reliability and value of our technology is like getting a Seal of Approval. We are not aware of any other measurement technology that has been accepted in this way by the governing body of a major US sport, or, in fact, any sport worldwide. This is the first real advance since the stop watch [sic] and the tape measure. We are a measurement company and now an independent organization has agreed that our technology works and is willing to use it in a very important capacity. We think that is pretty important.

The two devices that they cite, the stopwatch and the tape measure, differ greatly from the stated objectives for UIS. They do the actual measuring that determines a winner in a sporting event whereas, we are told, UIS will be a tool to evaluate the men whose decisions help determine the winner in a baseball game. It sound as if QuesTec envisions replacing the umpires or at least being the tool that umps use to call balls and strikes. Is that the case? And then does MLB envision this as well?

After reading the information, or dearth thereof, on QuesTec's site, I am torn on this issue. There are a number of questions that I have: What are the short-range and long-range goals for the system? Would the money be better spent instructing the umpires? Was that even investigated? Would this be a better tool for evaluating minor-league umpires as they advance to the majors? What are the stringent requirements that MLB demanded? How were they established? How will they change? How was the accuracy checked? How do we know that the standards are good enough? Was the use of the system negotiated with the WUA? What are the issues given that fewer than half the stadiums have the system installed? How will it be used as an overall evaluation tool? Is it the sole tool or one of many? Can umpires who score low be able to contest the results? Who owns the company? Are they publicly traded? How do they plan to sell the product in the future and to whom? Will the system eventually replace the umps in calling balls and strikes? If it's more accurate, why not? If it isn't how can it be used to evaluate the umps? Are there plans to scorecard calls at first, fair/foul calls, catches against the wall, tag calls, balks, knowledge of the rules, or any of the other responsibilities of the umps?

Believe me, the umps are a self-important lot who need to remember who the fans come to see. Umpires do not own the strike zone. It is established by the rules and should be called accordingly. They should not be allowed to have their own individual definition of the strike zone. But there may be no two groups as arrogant as than baseball owners and Information Technology professionals. Getting these two groups together unsupervised to implement this system may not have been a choice worthy of Solomon. The owners have to be more open and truthful about teh accuracy of this tool and their intended use.

. . .