indiemaps.com/blog

Here’s a map of yesterday’s perfect game pitched by Mark Buehrle — only the 18th in MLB history. Highlighted is defensive replacement Dewayne Wise’s perfect game-saving catch over the wall in the 9th inning.

Here the same, as displayed by MLB’s Gameday, for which the initial pixel coordinates were collected.

The conversion from the pixel-based coordinates used on the above to a geographic latitude-longitude space took a fair amount of work. More on that in a bit. So why even attempt it? What is the value added by presenting these data on a satellite photo?

At first, it was just to see if I could do it. From Wal-Marts to crimes to tweets, the slippy map has become the de facto platform for geovisualization. Hits — balls in play — are inherently spatial, though at a micro level that gets rare consideration from neocartographers. The conversion to geographic points would also let me take advantage of the various symbologies written for mapping APIs, from heatmaps to clusterers.

As I continued work, though, I became more interested in the issue of scale: the images produced reveal the scale of baseball, easily compared to the scale of urban life. For example, most MLB ballparks have fences as far as 400 feet from home plate. In New York City, the average N-S block is 264 feet long. Such a block may contain dozens of businesses or residences, all of which we’d expect Google Maps and others to accurately geolocate. So why not hits?

Here a Jake Fox homer at the intersection of Kenmore and Waveland outside Wrigley Field.

And here, a recent Washington Nationals game, with hits placed as though home plate was located at the center of Dupont Circle.

To get an idea for how field dimensions affect the game, we can display hits from a particular stadium as if they occurred at any other. Here a recent Pirates-Phillies game that took place at the relatively small Citizens Bank Park. In the second image, I’ve placed and oriented hits as though they took place at the Pirates’ PNC Park. As you can see, two, maybe three of the home runs would have been swallowed up by the visiting team’s more cavernous outfield.

What follows is a description of how I got the data, transformed it to lat-long coordinates, and displayed it on a slippy map. For the demo proof of concept application, go here.

Getting the data

MLB’s PITCHf/x system provides highly accurate real-time data on pitch speed, break, and location, all measured using high-speed video cameras. The data — freely available from MLB Gameday — are used in real-time by a number of media outlets and after the fact by many number-crunching sabermetricians. A similar system for batted balls does not exist, though is apparently in the works. Fairly accurate hit location data are available from the for-profit firms Baseball Info Solutions and STATS Inc, and data on home run distance/location is disseminated freely by HitTracker. But the only free source of locational information on all balls in play is the observational data collected by MLB Gameday for non-analytic purposes.

These data are purely observational, meaning hits are plotted by a MLB employee watching each game. For more on the accuracy of such data, see Is Seeing Believing? at The Hardball Times. For each ball in play, then, we have an X-Y coordinate. This coordinate is in an arbitrary pixel space and must be transformed quite a bit; more on that later.

Gameday’s XML data

MLB Gameday provides both the PITCHf/x and observational hit location data in online directories, starting here, that are organized chronologically and then by game. No API is provided, though, for querying this massive database of millions of pitches and hits — just directories and directories of XML files. Each game is described by dozens of XML files; the root game URL is accessed thusly:

http://gd2.mlb.com/components/game/mlb/ + year_{year}/month_{month}/day_{day}/ + gid_{year}_{month}_{day}_{away_team}mlb_{home_team}mlb_{game_number}/

Ex. gid_2009_07_21_balmlb_nyamlb_1, gid_2009_04_05_atlmlb_phimlb_1

In Baseball Hacks, Joseph Adler provides Perl scripts for traversing these directories, parsing the various XML files found in them, and storing the parsed data in a MySQL database. Mike Fast has updated the scripts, which can be used pretty quickly to create a pitch and hit database with this structure. I’ve done so, and the advantages of such a database are many, but the speed of queries is perhaps the greatest. For a small webapp, though, I didn’t want to go to the hassle of talking to a database, nor did I want the server-side dependencies. The Yahoo Query Language (YQL) provides a great solution in such cases, and allowed me to create an API where none existed before and use SQL-like syntax to query it.

My YQL tables

YQL lets you use a universal SQL-like syntax on any API or web data source. Since you can use subselects, YQL also lets you join disparate data sources. Typically it is used as a wrapper for APIs, so that developers may use the same syntax across many different data sources. They do so by writing a XML file that describes a YQL table. In a basic table, data are returned from the the API unmodified. Since I’ve no API to call, I rely on YQL’s execute element to run server-side Javascript for each query. With just a dozen or so lines of Javascript in the execute element of each table, I can quite easily return usable data from MLB’s directories of XML files.

I can also use the execute element to insert records into my own database (which can be MySQL or otherwise), allowing me to keep my database up-to-date more easily than with the Adler/Fast Perl update script. I should note, though, that the YQL queries are nowhere near as fast as querying a local database, and likely aren’t quick enough for actual production (though they’re great for the kind of experimentation and rapid prototyping described here).

I initially set out to recreate Mike Fast’s database environment with YQL tables: seven tables, one each for games, game types, pitches, pitch types, players, and umpires. For this exercise, I needed only the games and atbats tables. The latter includes the hit location information if a ball is sent into play. For both, I’ve added fields, grabbing more data from the Gameday XML files as needed. I also needed a stadiums table, though this draws from data I’ve personally collected, rather than the Gameday XML files. The three tables are hosted on github and available for use by anyone. If you’re logged into Yahoo, you can try em out right now by loading up the YQL console with the mlb.gd2.env environment file. You can then use SQL syntax on these tables. A few example queries (links are to REST queries that can be called from any web app):

• SELECT * FROM atbats WHERE gid=”gid_2009_04_11_nyamlb_kcamlb_1″ AND hit_x<>“NaN”
  
• SELECT stadium_id FROM games WHERE year=”2009″ AND month=”07″ AND day=”23″
• SELECT * FROM stadiums WHERE id=”3″

Transforming the data

Pixels to feet

The hit location data recorded by MLB is reported in pixels. The employees watching the games plot locations on a 250×250 image of the ballpark; the (0,0) origin is — not home plate — but simply the upper left corner of the image. The first step in making these data usable is to convert from this arbitrary pixel coordinate system to one in feet or meters. We need to know 1) the scale of the 250×250 image (feet per pixel) and 2) a known point on the field (preferably home plate) in the original pixel coordinates.

Initially, I assumed that these numbers would be the same or similar for all MLB ballparks. An article by Peter Jensen disabused me of this notion. Jensen undertook to determine the home plate X-Y location and distance multiplier for each MLB ballpark by assuming a uniform hit ball distribution and imposing physical constraints on hit balls. Unfortunately, when I plotted hits using Jensen’s numbers, the ball distribution diverged quite obviously from the Gameday-plotted version: outfield balls were plotted too far from home plate, infield balls too close. Jensen’s study used 2007-2008 hit locations, and MLB changes their images from season-to-season, so this may explain the divergence. Either way, I needed a new method.

I devised this simpler but less systematic approach. To determine the distance multiplier, I took a look at the stadium images provided by MLB Gameday. The stadium diagrams shown on Gameday’s real-time app are all 250×250 in expanded mode. Though I’m not positive that these are the exact diagrams upon which hits are initially plotted, they likely are as the resultant distance multipliers are quite accurate. Given a known distance (typically the straightaway center field fence distance), I can calculate the scale of the image in feet per pixel.

Determining home plate locations in Gameday’s coordinate system was less straightforward. In Baseball Hacks, Joseph Adler creates spray charts of the field using Gameday data. Unlike Jensen, Adler simply uses (125, 210) as the home plate location in pixels. I figured I could use this as a starting point, but to my surprise I found that I only needed to modify this origin for a half dozen stadiums. Of course, the only accuracy check I have is eyeballing back-and-forth between my rendering and the Gameday version, but given the observational nature of the data, I believe this is good enough.

Stadium locations and orientations

Using Google Earth, I recorded home plate latitude-longitude coordinates and eyeballed stadium orientations for all open MLB ballparks. Two stadiums are domes, so they’re out.

And three of the league’s six retractable-roofed stadiums were closed during their satellite shots.

LandShark Stadium was being used as a football field, and Citi Field hadn’t been built yet in Google’s satellite imagery, so home plate locations and stadium orientations weren’t collected for these.

For the other 23 stadiums, home plate latitude-longitude and X-Y coordinates, distance multipliers, and orientations are available in my YQL stadiums table.

Feet to latitude/longitude

Given the above, I can convert all pixel locations to feet with home plate as the origin. For stadiums oriented away from due north, these feet-based locations can be rotated about their home plate origins. The final step — converting from feet to geographic coordinates — is fairly simple, though I initially tried a couple of more complicated methods. First, I converted hit locations to meters, converted the home plate lat-long to UTM coordinates, added the hit coordinates to the latter, and converted back to lat-long. This worked fine, as did the second method I tried: a formula that calculates the hit’s geographic coordinate given the distance and bearing from home plate.

Given the huge scale and low accuracy required, I can utilize a much simpler method. Most mapping APIs include a distance method of some kind, taking as input two lat-long coordinates and returning the great circle (spherical, so perhaps off by as much as .3%) distance in meters or km. Here’s Yahoo’s, Google’s and Mapstraction’s. With this method, I can determine an approximate conversion factor between meters and degrees latitude-longitude:

Accuracy

A game was taking place when the satellite image of AT&T Park was taken; you can see the fans, players, and — importantly — the bases. Using the formulas above, I’ve placed white squares on the map where the bases should be, and this shows the typical accuracy achieved.

I’ve also plotted known outfield fence coordinates with similar accuracy.

Putting it all together

I created a demo application using Mapstraction, jQuery, and YUI. It’s very much a proof-of-concept, and the YQL queries can take quite a while to populate the map and table. Nonetheless, this is the only non-trivial web development I’ve done outside of Flash/Flex, and I’m quite impressed by these tools.

The app lets you visualize the hits in any individual game that took place in an open stadium in 2009. It’ll show games from earlier years, but I can’t vouch for the accuracy of the plotted points due to changes in Gameday’s observational data collection. By default, the app shows this year’s all-star game at Busch Stadium. To see another game, add a gid URL parameter. Here, for example, a few games from last night.

• the perfect game: gid_2009_07_23_tbamlb_chamlb_1
• gid_2009_07_23_sfnmlb_atlmlb_1
• gid_2009_07_23_seamlb_detmlb_1

The display is pretty basic: just pink dots for outs and blue ones for hits. And if you want to do any of the crazy stuff with plotting hits at different ballparks or arbitrary locations, you’ll have to dig around a bit. I can imagine many aggregation, filtering, and visualization options for these data. None are explored here. But all the code and data’s up on github if you want to have a go at it.

update Aug 08 2009: My proof-of-concept app was failing for a few days. Should be back now. My MLB Gameday YQL tables weren’t working as loaded from github, so I copied em over to indiemaps. Project’s still hosted on github. Any query links below utilizing the tables have been changed.