For a while now, I’ve wanted to build geographic topology in Flash. Topology, as described in an ESRI white paper, is

a set of rules and behaviors that model how points, lines, and polygons share geometry. For example, adjacent features, such as two counties, will share a common edge.

For the applications I’ve had in mind, polygonal or areal topology is all that’s required: I need to know which features share a common border with which features. As a bonus, it’d be nice to know how long a border they share (how contiguous are they?).

Described further down is a raster method to determine vector feature adjacency. First, though, I’ll cover why such information must be built in the first place and how it is useful for certain cartographic applications.

Application to cartography and visualization

Topology needs to be built because it is not encoded in the most popular vector GIS formats (KML and the shapefile). In these formats, features (states or counties perhaps) are all stored separately; redundant points (like the shared corner of the “four corners” states) are repeated. There are topological GIS formats, like Arc/Info Export (e00), but geospatial data are rarely distributed in such formats.

One potential use for topological information is graph decomposition. Dasgupta et al write in Algorithms:

A wide range of problems can be expressed with clarity and precision in the concise pictorial language of graphs. For instance, consider the task of coloring a political map. What is the minimum number of colors needed, with the obvious restriction that neighboring countries should have different colors? One of the difficulties in attacking this problem is that the map itself, even a stripped-down version like Figure 3.1(a), is usually cluttered with irrelevant information: intricate boundaries, border posts where three or more countries meet, open seas, and meandering rivers. Such distractions are absent from the mathematical object of Figure 3.1(b), a graph with one vertex for each country (1 is Brazil, 11 is Argentina) and edges between neighbors. It contains exactly the information needed for coloring, and nothing more.

The simple graph above is two steps away from a circular cartogram of the form popularized by Daniel Dorling. On such maps, 1) feature circles are sized according to some numeric attribute, and neighbors — instead of being connected by lines — are 2) iteratively moved together or apart so that adjacencies are maintained where possible while avoiding circle overlap. The circular cartogram below is a snapshot of a NY Times interactive app developed in part by Lee Byron.

In addition to knowing the neighbors of all features, Daniel Dorling’s original algorithm requires as input the shared border lengths of all contiguous features. These are used in the force-repulsion algorithm — while attempting to maintain all feature adjacencies, Dorling’s algorithm applies extra attraction forces to features that share relatively longer borders.

In cartography, topology has much value beyond that described above. Generalization immediately comes to mind; to generalize/simplify/smooth the outline of a polygonal feature, one must also consider the feature’s neighbors. If the cartographer fails to do so, gaps may be created where features previously neatly adjoined. That noted, I should say that the method described here — while quite accurate in evaluating adjacencies — can only determine adjacency and relative (pixel-based) feature overlap, whereas polygonal shapefile generalization necessitates a true (and computationally intensive) topological indexing of each and every coordinate in the dataset. More on this later.

Method

Features stored in KML and shapefiles are defined by a series of latitude-longitude coordinates. The brute force method of determining if two features are topological neighbors would be to loop through their coordinates searching for an exact match. Not only would this be quite computationally intensive, but it wouldn’t detect true neighbors that were digitized independently; a shared border does not necessarily mean that the two features are defined using the exact same control points.

Of course it can be done. Most GIS software today will accurately build topology from a non-topological data source. The NY Times’ Matt Bloch detailed a server-side method in his Wisconsin (natch) Cartography MS thesis (as well as a 2006 AUTOCARTO proceedings paper), used in his MapShaper app, for essentially converting a polygonal shapefile to a non-redundant polyline shapefile. Though more efficient than simple brute force, these applications still rely on algorithms that require a large number of computations, and this scales up quickly as features get more complex. This is why topology is always built server-side.

The method I settled on is similar to pixel-based collision detection (also called “pro pixel collision” or “pixel-perfect collision”) familiar to game developers. In Flash, hitTests are only possible with specific points. To test for overlap of complex polygonal features, game developers have often relied on a pixel-level bitmap test for overlap. As described in an article by Troy Gilbert,

It’s pretty straightforward: you render two display objects to separate color channels, combine the color channels, then search the resulting image for any overlapping color.

I first encountered the technique in a post by Grant Skinner describing a method for shape-based collision detection in Flash published in 2005. GSkinner’s method doesn’t work as is in detecting adjacency in geographic features drawn from shapefiles. But with a few additions, the method is quite accurate and efficient in detecting feature neighbors in real-time. Try it out:

Some code

This isn’t really fully developed. It’s just something I’ve been thinking about for a while and wanted to get out there. Here though is a class I wrote to perform this pixel-based adjacency testing, as used in the above demo. It contains two chief public static methods:

• getNeighborsForFeature(): Returns a Vector of DisplayObject features determined to be neighbors of the passed-in feature (from a passed-in Vector of all features)
• checkFeatureAdjacency(): Checks the two passed-in DisplayObject features for adjacency. Returns the number of overlapping pixels.

How it works

To test for adjacency, features are drawn to a bitmap. The second feature is overlain using the difference blend mode. Any pixels shared by both features will now be lighter than before; I can check for such pixels using the BitmapData.threshold() method.

Adjacent features, though, will only overlap if they’ve been drawn with an external stroke. The two methods above accept a strokeWidth parameter. If this is less than 2 (features with external strokes of 1 sometimes fail to detect overlap accurately), I apply a precise GlowFilter to each raster. This increases accuracy but affects performance; though it’s not necessary, the methods perform best when features are initially drawn with a 2px stroke (as in the demo above).

Limitations

Accuracy

This method is quite accurate for most realistic geographies. But of course it’s all dependent on 1) the scale and complexity of your data and 2) the size of the BitmapData instance used for testing. One particular area of concern is features that share a corner but no actual border line segments. In some applications, these should be considered neighbors; in others, not. With a large enough BitmapData, such adjacencies will be detected.

If such features should not be considered neighbors in your application, a threshold may be set. checkFeatureAdjacency() returns the number of overlapping pixels; if this number is sufficiently low, features can be considered non-adjacent.

Performance

I haven’t done any real performance tests, but the method works fine, in a test with the 3000+ U.S. counties drawn from a large-scale shapefile in Firefox on my MacBook; neighbor detection was detected in real-time and nothing was cached.

As noted above, the methods perform fastest when features are initially drawn with a 2px stroke. If the methods are to be called repeatedly, it’s best to pass in shared BitmapData instances to be used for testing; the larger the BitmapData instance, the greater the accuracy, though this is offset by increased processing time for the BitmapData instance methods. No matter what you pass in, though, this raster-based method is far faster than any true coordinate-based adjacency testing algorithm (important b/c it is being performed client-side).

Generalization

This method simply doesn’t work for generalization (simplification or smoothing of feature border linework). Such processes require knowledge of all shared line segments, necessitating true coordinate-based topology building; my method only returns the number of overlapping pixels, which only correlates with actual shared border length (this, though, is sufficient information for the Dorling circular cartogram algorithm).

But I hope it’s useful for at least a few visualization applications. True topological indexing is best; but this computationally intensive process cannot yet be performed client-side in Flash. I’ll be working with these methods in the future, most likely in an implementation of Dorling’s circular cartograms in indiemapper.

The six symbologies are the classic thematic cartography representation methods: choropleth, proportional symbol, dot density, flow, isarithmic, and cartogram. Borden Dent’s great cartography textbook, in the section titled “Techniques of Quantitative Thematic Mapping”, dedicates a chapter to each of these symbologies, and to no others. The classic Robinson textbook, as well as the modern Slocum et al one, also dedicate more space to these six representation methods than to any others.

isoline

The first isarithmic (representation of continuous phenomena using lines of equal value) map is often ascribed to Edwin Halley, with his 1701 isogonic contour maps of magnetic declination.

Not so, as the symbology has been traced back to as early as 1584. Robinson notes in Early Thematic Mapping:

The contour had tentative beginnings as early as the end of the sixteenth centruly in the form of lines of equal depth on a map of the River Spaarne made in 1584 by the Dutch surveyor Pieter Bruinsz. The next use of the symbol came more than a hundred years later in 1697 by Pierre Ancellin…

You can’t see too much in the above — a terrible scan I made from Robinson’s Early Thematic Mapping. This map isn’t listed on Milestones, nor could I find any reproductions of Bruinsz’s map on the internet. What I did find was a detail image of unknown provenance on the interesting site, Dutch thematic maps on the web.

On the above you can clearly see the dashed bathymetric line of equal depth.

choropleth

Frenchman Charles Dupin’s (1826) Carte figurative de l’instruction populaire de la France is the first known choropleth map.

Dupin’s map was and is well known, and had a great impact on cartographers and statisticians at the time. Interestingly, the particular form of Dupin’s choropleth map — the unclassed variety, in which each unique value gets a unique shade — didn’t stick. The classed variety quickly took hold, with the first such map appearing in an 1828 Prussian atlas, Administrativ-Statistischer Atlas vom Preussischen Staatae.

Notice not the subtle lean in my scan (from Robinson since I couldn’t find it elsewhere), but the class breaks shown along the bottom. The author of the thematic portion of the atlas is unknown.

dot density

Dupin’s fellow countryman Frère de Montizon is responsible for the first dot density map, Carte philosophique figurant la population de la France (1830).

Unlike Dupin, who is fairly well known, Robinson describes Montizon as a “mystery man”. He continues,

It is one of the accidents of history that Frère de Montizon’s invention of the thematic dot map should have gone completely unnoticed. In terms of cartographic innovation it ranks with the isothermal map, yet as far as can be ascertained no reference to him or to his dot map was made by anyone well into the twentieth century…this basically simple, logical idea had to wait some thirty years to be reinvented and much longer than that to become generally known.

The symbology was “reinvented” by Thure Alexander von Mentzer in his 1859 map of population distribution in the Scandinavian peninsula. I can’t find any reproductions of this map anywhere.

proportional symbols

The first known example of a proportional symbol map appeared in the Atlas to Accompany Second Report of the Railway Commissioners, Ireland (1837). Though the atlas was apparently quite innovative with regard to thematic cartography, it went largely unnoticed due to limited distribution.

Here, Henry Drury Harness’s map of Irish population density from the atlas.

They’re a bit hard to make out in this horrible scan (again from Robinson — listed as part of his personal collection), but there they are: circles scaled according to population centered at various Irish cities. This wasn’t the first time that proportional circles had been used to convey data, but it was the first time it had been done on a map. Notice the shading?– this is also one of the first dasymetric maps.

flow maps

The following map is more well known, and could share the title of first proportional symbol map, as it appeared in the same 1837 atlas as the above map. Here, the innovation is the use of line thickness to convey a quantitative value, in this case the traffic between Irish cities (again by Henry Drury Harness).

cartogram

I’ve written about this before, so below’s a quick summary.

Many sources, including Dent’s text, point to Émile Levasseur as the originator of the value-by-area cartogram. Levasseur included diagrammatic maps like the following 1868 map of Europe in his economic geography textbooks.

The above map was reprinted by Funkhouser in 1937 and by Waldo Tobler in his Thirty Five Years of Computer Cartograms. In the latter, Tobler notes:

This is a map of the countries of Europe in which each country is represented by a square whose size is proportional to the area of the country, and with countries in their approximately correct position and adjacency. Could this be called an equal area map? Or is it an equal area cartogram?

I believe the former: that this should be considered a diagrammatic equal area map, but not a value-by-area cartogram. Sara Fabrikant makes the case for the German provenance of the first cartogram:

Another notable and early European contribution to the thematic mapper’s toolbox is the value-by-area cartogram. Hermann Haack and H. Wiechel published a cartogram depicting election results from the German Reichstag in 1903 (cited in Eckert 1925).

Haack and Wiechel’s map is indeed cited in Eckert’s famous Die Kartenwissenschaft volumes, but it isn’t reprinted there. Nor is it clear that unit areas on their election maps represented something other than land area. The earliest true value-by-area cartogram that I’ve seen reproduced came by way of Professor John Krygier, who reprinted William Bailey’s 1911 population cartogram.

I doubt the above is truly the first cartogram, and welcome any scans or references to true value-by-area cartograms that precede the 1911 map.

conclusion

Four of the six classic thematic cartography symbologies — choropleth, dot density, proportional symbol, and flow — originated between 1826 and 1837. Two of them — proportional symbol and flow — were initially produced by one man (Harness), and appeared in the same obscure railway atlas. All were refined in the 19th century, and only one (isolines) predate the century.

Conspicuously absent above are the names William Playfair and Joseph Minard. Indeed, some sources still cite these well-known engineer-statisticians as the originators of the proportional symbol and flow symbologies. Playfair may have been the first to use proportional (value-by-area) symbols (specifically circles) to represent quantitative data, but it wasn’t on a map.

Funkhouser credited Minard with the invention of the flow symbology, ignoring Harness’s innovation eight years earlier:

The second, called cartograms with bands (cartogramme a bandes), was originated simultaneously and independently by the French engineer, Minard, and the Belgian engineer, Belpaire, in 1845. This consists of colored bands or ribbons which follow the watercourses and railways on a map, the width of the band being proportional to the amount of traffic or number of passengers carried.

It’s certainly the case, though, that Playfair paved the way for much thematic mapping experimentation in the early 19th C. and that Minard helped popularize the flow, proportional symbol, and choropleth symbologies.

]]> http://indiemaps.com/blog/2009/11/the-first-thematic-maps/feed/ http://indiemaps.com/blog/2009/10/classed-cartograms/ http://indiemaps.com/blog/2009/10/classed-cartograms/#comments Sat, 17 Oct 2009 04:15:36 +0000 zach'ry http://indiemaps.com/blog/?p=97

Classification is commonplace in thematic cartography. In choropleth mapping, classification is the norm. This was not always so. The first choropleth map (created by Baron Charles Dupin in 1826) was unclassed.

According to Arthur Robinson, in his Early Thematic Mapping in the History of Cartography:

So far as we now know, the first choropleth map to provide a legend and give class limits to the tones was the 1828 Prussian manuscript map of population density. After the early 1830s most choropleth maps employed classes of various sorts, some based on rankings of the districts, some based on percentage departures from a mean, and some based on categorizing the data itself.

It seems that classification in choropleth mapping was adopted for technical rather than theoretical reasons. Robinson notes that “engravers were unsuccessful in making such [unclassed] maps, since they did not have full control over the darkness of very many flat tones.” Nonetheless, empirical research, begun in the late 1970s and continuing into the 1990s, backed up the decision, at least as regards the acquisition of specific information. Data on the recall of specific information and the acquisition of general information was less conclusive; a good summary of this research is provided in Slocum et al’s cartography textbook.

(classed histogram legend from my first thematic map, produced for Mark Harrower’s Geography 370: Introduction to Cartography)

Range-graded proportional symbols

In proportional symbol mapping, the unclassed form is the norm. This, too, seems to be a technical rather than theoretically-based trend. As noted in the Slocum text:

Classed, or range-graded, maps can be created by classing the data and letting a single symbol size represent a range of data values, but unclassed proportional symbol maps are more common. This might seem surprising given the frequency with which classed choropleth maps are used. The difference stems, in part, from the ease with which unclassed proportional symbol maps could be created in manual cartography (either an ink compass or a circle template could be used to draw circles of numerous sizes).

The logical extension of classed choropleth mapping to proportional symbol mapping was first suggested by Hans Joachim Meihoefer in the late 1960s. Here, too, classification is purported to make the thematic map easier to comprehend; also from Slocum:

Range grading is considered advantageous because readers can easily discriminate symbol sizes and thus readily match map and legend symbols; another advantage is that the contrast in circle sizes might enhance the map pattern, in a fashion similar to the use of an arbitrary exponent…

This comes at the expense of precision, as map reader’s can never get exact values for enumeration units. Borden Dent notes, “In this scaling method, symbol-size discrimination is the design goal, rather than magnitude estimation.”

Rarely, though, is exact value retrieval (magnitude estimation) the goal of thematic cartography, at least of the static variety. Further, map readers are notoriously bad size estimators, as an array of psychophysical studies in and outside of academic cartography have established. The graph below (from John Krygier’s excellent blog post on perceptual scaling) sums up the average response to 1, 2, and 3D proportional symbols.

The above suggests that subjects uniformly underestimate the areas of proportional symbols. Perceptual scaling has been developed in response, in which symbol sizes are perceptually scaled according to a power function, rather than being scaled mathematically.

Perhaps more important, though, are the deviations among subjects hidden in the graph above. T.L.C. Griffin, in a 1985 study, found an average underestimation similar to previous studies, but stressed that “perceptual rescaling was shown to be inadequate to correct the estimates of poor judges, while seriously impairing the results of those who were more consistent.”

Both the Slocum et al and Dent cartography textbooks introduce range-grading of proportional symbols as a potential solution to the 1) average poor estimation of symbol size and 2) high deviation in this underestimation.

Cartograms then

This makes a case for classification in proportional symbol mapping. It also makes the case for classification on cartograms. Cartograms can be considered a variant of proportional symbol map in which the symbol shape used is that of the original enumeration unit in some projection. But in my cartogram research for the M.S. Cartography degree at the University of Wisconsin, I ran across no references to classifying the area of cartogram units. A more recent search also revealed no references. Classification seems particularly appropriate to cartograms because the limited research done on their perception suggests that users estimate cartogram feature area much less accurately than the simplified shapes of standard proportional symbol maps.

The above is a normal, unclassed cartogram showing U.S. population. Notice the continuous range of state areas, shown below in order of population (but smaller).

Despite the two legend chips, I think it’s quite difficult to estimate the population of states with any degree of precision. The following classed cartogram abandons precision in favor of a more easily and quickly interpreted map.

Exact values can never be retrieved, but each state can quickly be matched with one of four classes (grouped according to Jenks natural breaks classification).

The above maps were created with a beta version of indiemapper. No other mapping software allows the creation of classed cartograms. You can get around this when using tools such as ScapeToad or Frank Hardisty’s Cartogram Generator (both of which utilize the contiguous Gastner-Newman cartogram algorithm) by pre-classing your data, so that only 3-7 unique values are found in the dataset for a given attribute.

In addition to 1) increased discriminability of symbol sizes and 2) a potentially enhanced spatial pattern, classed cartograms may hold a third advantage over the unclassed variety, though only in bivariate mapping. Cartograms, though, are most typically and appropriately employed in bivariate mapping. When constructing a bivariate cartogram, the cartographer sizes enumeration units according to one variable (typically population) and colors the units, choropleth-style, according to some other variable (often election results). In unclassed cartograms, some features may shrink down so as to be nearly invisible, making the reading of the second (coloring) attribute impossible. In the classed variant, a small but still legible minimum size can be established for the first class of data, ensuring that the coloring attribute can be interpreted on all units. A minimum size can be established on an unclassed cartogram, but if mathematical/proportional scaling is employed it may result in absurdly large features at the higher end of the mapped variable.

Of course, classed cartograms won’t always have a larger minimum size; this must be a conscious design decision (in the U.S. cartograms above, the classed cartogram minimum size is smaller than the minimum size found on the unclassed version).

Inappropriate

Classification in cartogram scaling is not always appropriate. Indeed, the typical unclassed form is preferable in many cases. Borden Dent writes the following of classification and choropleth mapping; I believe it’s equally applicable to the proportional and cartogram symbologies:

The purpose of the choropleth map dictates its form. If the map’s main purpose is to simplify a complex geographical distribution for the purpose of imparting a particular message, theme, or concept, the conventional [classed] choropleth technique should be followed. On the other hand, if the goal is to provide an inventory for the reader in a form that the reader must simplify and generalize, then the unclassed form should be chosen.

This sounds a lot like the modern distinction between cartography and geovisualization, or like the older one between communication and represenation. These distinctions have been overblown; experts and amateurs are often looking at the same map. But there are certainly cases where the purpose and audience are more clear-cut. In such cases, classed cartograms may be an option.

While providing the cartographer with the “ability to direct the message of the communication” (Dent), this power comes with great responsibility — specifically the responsible choice of the 1) number of classes, 2) classification method, and 3) appropriate and easily discriminable symbol sizes for each class.

Research on classification in choropleth and proportional symbol mapping has settled on three and seven as the minimum and maximum number of classes that are effective. Classification methods have been much studied in academic cartography, statistics, and other fields. Jenks natural breaks classification is often employed; equal intervals and quantiles are also typical. Cartographers have less guidance on the third choice, that of an appropriate symbol size for each class.

The above shows a set of ten class sizes for range grading developed by Hans Joachim Meihoefer as the result of a visual experiment; the symbol sizes were developed for maximum discriminability, while maintaining a realistic size range (none too small, none too large). Ten classes weren’t recommended, but supposedly any five contiguous sizes could be chosen. Indiemapper uses a slight variation of Meihoefer’s sizing scheme for classed proportional symbol and cartogram sizing.

Despite the additional responsibilities placed on the cartographer, I believe a carefully-crafted classed cartogram can be a very effective representation of some datasets for certain audiences.

Here the same, as displayed by MLB’s Gameday, for which the initial pixel coordinates were collected.

The conversion from the pixel-based coordinates used on the above to a geographic latitude-longitude space took a fair amount of work. More on that in a bit. So why even attempt it? What is the value added by presenting these data on a satellite photo?

At first, it was just to see if I could do it. From Wal-Marts to crimes to tweets, the slippy map has become the de facto platform for geovisualization. Hits — balls in play — are inherently spatial, though at a micro level that gets rare consideration from neocartographers. The conversion to geographic points would also let me take advantage of the various symbologies written for mapping APIs, from heatmaps to clusterers.

As I continued work, though, I became more interested in the issue of scale: the images produced reveal the scale of baseball, easily compared to the scale of urban life. For example, most MLB ballparks have fences as far as 400 feet from home plate. In New York City, the average N-S block is 264 feet long. Such a block may contain dozens of businesses or residences, all of which we’d expect Google Maps and others to accurately geolocate. So why not hits?

Here a Jake Fox homer at the intersection of Kenmore and Waveland outside Wrigley Field.

And here, a recent Washington Nationals game, with hits placed as though home plate was located at the center of Dupont Circle.

To get an idea for how field dimensions affect the game, we can display hits from a particular stadium as if they occurred at any other. Here a recent Pirates-Phillies game that took place at the relatively small Citizens Bank Park. In the second image, I’ve placed and oriented hits as though they took place at the Pirates’ PNC Park. As you can see, two, maybe three of the home runs would have been swallowed up by the visiting team’s more cavernous outfield.

What follows is a description of how I got the data, transformed it to lat-long coordinates, and displayed it on a slippy map. For the demo proof of concept application, go here.

Getting the data

MLB’s PITCHf/x system provides highly accurate real-time data on pitch speed, break, and location, all measured using high-speed video cameras. The data — freely available from MLB Gameday — are used in real-time by a number of media outlets and after the fact by many number-crunching sabermetricians. A similar system for batted balls does not exist, though is apparently in the works. Fairly accurate hit location data are available from the for-profit firms Baseball Info Solutions and STATS Inc, and data on home run distance/location is disseminated freely by HitTracker. But the only free source of locational information on all balls in play is the observational data collected by MLB Gameday for non-analytic purposes.

These data are purely observational, meaning hits are plotted by a MLB employee watching each game. For more on the accuracy of such data, see Is Seeing Believing? at The Hardball Times. For each ball in play, then, we have an X-Y coordinate. This coordinate is in an arbitrary pixel space and must be transformed quite a bit; more on that later.

Gameday’s XML data

MLB Gameday provides both the PITCHf/x and observational hit location data in online directories, starting here, that are organized chronologically and then by game. No API is provided, though, for querying this massive database of millions of pitches and hits — just directories and directories of XML files. Each game is described by dozens of XML files; the root game URL is accessed thusly:

http://gd2.mlb.com/components/game/mlb/ + year_{year}/month_{month}/day_{day}/ + gid_{year}_{month}_{day}_{away_team}mlb_{home_team}mlb_{game_number}/

Ex. gid_2009_07_21_balmlb_nyamlb_1, gid_2009_04_05_atlmlb_phimlb_1

In Baseball Hacks, Joseph Adler provides Perl scripts for traversing these directories, parsing the various XML files found in them, and storing the parsed data in a MySQL database. Mike Fast has updated the scripts, which can be used pretty quickly to create a pitch and hit database with this structure. I’ve done so, and the advantages of such a database are many, but the speed of queries is perhaps the greatest. For a small webapp, though, I didn’t want to go to the hassle of talking to a database, nor did I want the server-side dependencies. The Yahoo Query Language (YQL) provides a great solution in such cases, and allowed me to create an API where none existed before and use SQL-like syntax to query it.

My YQL tables

YQL lets you use a universal SQL-like syntax on any API or web data source. Since you can use subselects, YQL also lets you join disparate data sources. Typically it is used as a wrapper for APIs, so that developers may use the same syntax across many different data sources. They do so by writing a XML file that describes a YQL table. In a basic table, data are returned from the the API unmodified. Since I’ve no API to call, I rely on YQL’s execute element to run server-side Javascript for each query. With just a dozen or so lines of Javascript in the execute element of each table, I can quite easily return usable data from MLB’s directories of XML files.

I can also use the execute element to insert records into my own database (which can be MySQL or otherwise), allowing me to keep my database up-to-date more easily than with the Adler/Fast Perl update script. I should note, though, that the YQL queries are nowhere near as fast as querying a local database, and likely aren’t quick enough for actual production (though they’re great for the kind of experimentation and rapid prototyping described here).

I initially set out to recreate Mike Fast’s database environment with YQL tables: seven tables, one each for games, game types, pitches, pitch types, players, and umpires. For this exercise, I needed only the games and atbats tables. The latter includes the hit location information if a ball is sent into play. For both, I’ve added fields, grabbing more data from the Gameday XML files as needed. I also needed a stadiums table, though this draws from data I’ve personally collected, rather than the Gameday XML files. The three tables are hosted on github and available for use by anyone. If you’re logged into Yahoo, you can try em out right now by loading up the YQL console with the mlb.gd2.env environment file. You can then use SQL syntax on these tables. A few example queries (links are to REST queries that can be called from any web app):

• SELECT * FROM atbats WHERE gid=”gid_2009_04_11_nyamlb_kcamlb_1″ AND hit_x<>“NaN”
  
• SELECT stadium_id FROM games WHERE year=”2009″ AND month=”07″ AND day=”23″
• SELECT * FROM stadiums WHERE id=”3″

Transforming the data

Pixels to feet

The hit location data recorded by MLB is reported in pixels. The employees watching the games plot locations on a 250×250 image of the ballpark; the (0,0) origin is — not home plate — but simply the upper left corner of the image. The first step in making these data usable is to convert from this arbitrary pixel coordinate system to one in feet or meters. We need to know 1) the scale of the 250×250 image (feet per pixel) and 2) a known point on the field (preferably home plate) in the original pixel coordinates.

Initially, I assumed that these numbers would be the same or similar for all MLB ballparks. An article by Peter Jensen disabused me of this notion. Jensen undertook to determine the home plate X-Y location and distance multiplier for each MLB ballpark by assuming a uniform hit ball distribution and imposing physical constraints on hit balls. Unfortunately, when I plotted hits using Jensen’s numbers, the ball distribution diverged quite obviously from the Gameday-plotted version: outfield balls were plotted too far from home plate, infield balls too close. Jensen’s study used 2007-2008 hit locations, and MLB changes their images from season-to-season, so this may explain the divergence. Either way, I needed a new method.

I devised this simpler but less systematic approach. To determine the distance multiplier, I took a look at the stadium images provided by MLB Gameday. The stadium diagrams shown on Gameday’s real-time app are all 250×250 in expanded mode. Though I’m not positive that these are the exact diagrams upon which hits are initially plotted, they likely are as the resultant distance multipliers are quite accurate. Given a known distance (typically the straightaway center field fence distance), I can calculate the scale of the image in feet per pixel.

Determining home plate locations in Gameday’s coordinate system was less straightforward. In Baseball Hacks, Joseph Adler creates spray charts of the field using Gameday data. Unlike Jensen, Adler simply uses (125, 210) as the home plate location in pixels. I figured I could use this as a starting point, but to my surprise I found that I only needed to modify this origin for a half dozen stadiums. Of course, the only accuracy check I have is eyeballing back-and-forth between my rendering and the Gameday version, but given the observational nature of the data, I believe this is good enough.

Stadium locations and orientations

Using Google Earth, I recorded home plate latitude-longitude coordinates and eyeballed stadium orientations for all open MLB ballparks. Two stadiums are domes, so they’re out.

And three of the league’s six retractable-roofed stadiums were closed during their satellite shots.

LandShark Stadium was being used as a football field, and Citi Field hadn’t been built yet in Google’s satellite imagery, so home plate locations and stadium orientations weren’t collected for these.

For the other 23 stadiums, home plate latitude-longitude and X-Y coordinates, distance multipliers, and orientations are available in my YQL stadiums table.

Feet to latitude/longitude

Given the above, I can convert all pixel locations to feet with home plate as the origin. For stadiums oriented away from due north, these feet-based locations can be rotated about their home plate origins. The final step — converting from feet to geographic coordinates — is fairly simple, though I initially tried a couple of more complicated methods. First, I converted hit locations to meters, converted the home plate lat-long to UTM coordinates, added the hit coordinates to the latter, and converted back to lat-long. This worked fine, as did the second method I tried: a formula that calculates the hit’s geographic coordinate given the distance and bearing from home plate.

Given the huge scale and low accuracy required, I can utilize a much simpler method. Most mapping APIs include a distance method of some kind, taking as input two lat-long coordinates and returning the great circle (spherical, so perhaps off by as much as .3%) distance in meters or km. Here’s Yahoo’s, Google’s and Mapstraction’s. With this method, I can determine an approximate conversion factor between meters and degrees latitude-longitude:

Accuracy

A game was taking place when the satellite image of AT&T Park was taken; you can see the fans, players, and — importantly — the bases. Using the formulas above, I’ve placed white squares on the map where the bases should be, and this shows the typical accuracy achieved.

I’ve also plotted known outfield fence coordinates with similar accuracy.

Putting it all together

I created a demo application using Mapstraction, jQuery, and YUI. It’s very much a proof-of-concept, and the YQL queries can take quite a while to populate the map and table. Nonetheless, this is the only non-trivial web development I’ve done outside of Flash/Flex, and I’m quite impressed by these tools.

The app lets you visualize the hits in any individual game that took place in an open stadium in 2009. It’ll show games from earlier years, but I can’t vouch for the accuracy of the plotted points due to changes in Gameday’s observational data collection. By default, the app shows this year’s all-star game at Busch Stadium. To see another game, add a gid URL parameter. Here, for example, a few games from last night.

• the perfect game: gid_2009_07_23_tbamlb_chamlb_1
• gid_2009_07_23_sfnmlb_atlmlb_1
• gid_2009_07_23_seamlb_detmlb_1

The display is pretty basic: just pink dots for outs and blue ones for hits. And if you want to do any of the crazy stuff with plotting hits at different ballparks or arbitrary locations, you’ll have to dig around a bit. I can imagine many aggregation, filtering, and visualization options for these data. None are explored here. But all the code and data’s up on github if you want to have a go at it.

update Aug 08 2009: My proof-of-concept app was failing for a few days. Should be back now. My MLB Gameday YQL tables weren’t working as loaded from github, so I copied em over to indiemaps. Project’s still hosted on github. Any query links below utilizing the tables have been changed.

I first saw this map browsing technique in a minimal browser Matt Bloch created for an older static project.

I implemented a lens tool in the final project version of the World Freedom Atlas. I also experimented in some of the early prototypes with a continuous fisheye effect for map browsing. The latter never really took off because of the distortion and pixelation inherent in the raster method.

And the same idea in a Google Maps mashup.

The originator of the fisheye/magnification method for multi-scale mapping is probably Edgar Kant, in a 1957 map he produced for a migration study of Asby, Sweden, by Torsten Hägerstrand. Here the “distance from the centre shrinks proportionally to the logarithm of the real distance.”

Much work proceeded on multi-scale map projections, with the touchstone article being Snyder’s 1987 “‘Magnifying-glass’ azimuthal map projections”. Good coverage on such projections (including parallels to cartogram distortion) can be found in Canter’s Small-scale Map Projection Design.

In non-mapping UIs, the magnification/fisheye effect is fairly common; the Mac dock does it and even Cover Flow can be considered somewhat of a variant. For browsing and selection from a “large linear list”, Ben Bederson at HCIL came up with fisheye menus.

So nothing new in general UI terms, but still pretty novel and perhaps especially applicable to online map browsing.

Map browsing

Axis Maps cartographer Andy Woodruff did a great post on a variety of map panning and zooming methods. The lens or magnifying glass tool performs both panning and zooming functions, and should be considered as an alternative to the nine methods outlined there.

In interactive applications, the approach’s major strengths are threefold:

• low mouse mileage for panning and making selections
• less disorientation or “getting lost” as general cues are always available
• the ability to see generals and specifics simultaneously

The last is particularly important in cartographic applications where the success of a good thematic map is often seen as its ability to present overall trends and specific values in the same map.

Semantic zoom lens tool

The Cartifact example above is particularly interesting because of the semantic zoom inherent in its lens tool. In normal, geometric zooming, a map (or other image) is simply blown up; more detail is shown by definition, because more pixels are dedicated to the image. In semantic zooming, different (typically more detailed) larger-scale renderings are shown at higher zoom levels; not only are features larger, but more details (and labels) are shown. Such semantic zooming is standard in slippy maps, which are produced and tiled at predefined zoom levels. Nonetheless, the application to a lens tool is noteworthy, especially in thematic cartography; generals and specifics can be presented simultaneously, and both can be tailored semantically to different zoom levels.

Here’s a quick example I threw together in Flex using Modest Maps (right-click to view source).

I like inverting the above, or perhaps more interestingly, showing Microsoft Aerial as the base and Microsoft Hybrid as the lens; the spotlight (zoomed in or otherwise) then serves to provide political/cultural details for the moused-over region.

Application to thematic cartography

In online thematic cartography, the practice of showing smaller enumeration units at higher zoom levels is somewhat common. The NY Times has done it a few times, including their Election 2008 results maps (the county-level choropleth is revealed by zooming in).

The same idea can be applied to a lens tool. Here the lens reveals the county vote results, and can be zoomed (again, a quick Flex job) to further investigate the local-level results. Click to launch the map (it’ll take a few secs to load, project, and draw the data).

The above is based on some projections and choropleth code I released last year. I think there’s more room for experimentation here: the size of the lens could be user-modifiable and semi-transparent (so you can still see where you’re mousing over the main map); I’d also like to create a less-pixellated fisheye lens and try out multiple lenses/fisheyes (for detailed comparisons of multiple areas while still providing context).

]]> http://indiemaps.com/blog/2009/04/lens-tools-and-fisheye-map-browsing/feed/ http://indiemaps.com/blog/2009/02/e00parser-an-actionscript-3-parser-for-the-arcinfo-export-topological-gis-format/ http://indiemaps.com/blog/2009/02/e00parser-an-actionscript-3-parser-for-the-arcinfo-export-topological-gis-format/#comments Sun, 22 Feb 2009 01:49:18 +0000 zach'ry http://indiemaps.com/blog/?p=83 First off, why mess with such a retro format as Arc/Info Export (.e00)?– any code written for this ASCII file type in the last few years has been on how to go from e00 to pretty much anything (especially to the non-topological data format, the shapefile).

Put simply, topological information makes a lot of things possible for the intrepid ActionScripter.

E00 files non-redundantly store all nodes, lines, and polygons that make up a geographic data layer. This geodata format is one of three currently distributed by the Census Bureau for boundary files (the others are the shapefile and the Ungenerate ASCII format). The GIS formats used in most web mapping applications (I’m thinking of shapefiles, GeoJSON, and KML) are non-topological, meaning features are stored independently, and topological information on shared borders and the like is quite difficult to extract. Like seriously hard. Something you don’t want to be doing in the browser. Matthew Bloch, of the NY Times, did his cartography master’s thesis (at Wisconsin, natch) on MapShaper, much of which involved a C++ server-side solution for building topology from a polygonal shapefile. Generalization requires non-redundant polylines so as not to create gaps between features when smoothing. Other visualization techniques, including cartogram construction and graph decomposition, also require knowing the shared borders of geographic features.

Ideally, such topological information could be created/extracted for any geography, regardless of the datasource. In reality, topology building is intensive and best suited to server-side processing. Using E00 files and my E00Parser lets you experiment with the visualization and cartographic techniques only possible when such topological information is known, without the expensive processing necessary to build it.

The code

I’ve gotten a ton of use out of Edwin van Rijkom’s SHP library. My noncontiguous cartogram, isolining, and political choropleth experiments relied on the code to load coordinate data in shapefile form at run-time, as did the early experiments that led to indiemapper. I’m hoping I’ll get just as much use out of this parser, for when adjacency information is critical to the visualization technique.

There are two main classes, E00Parser and E00Tools. E00Parser is based on the Perl extension Geo::E00 by Alessandro Zummo and Bert Tijhuis, with much aid from the (world famous) Arc/Info Export Format Analysis. There’s no way I would have attempted to write the AS3 E00 parser without Zummo and Tijhuis’ code, as theirs appears to be the only stand-alone open source code available for reading the format. Their Perl regular expressions were copied with few modifications, though I did fix an issue in some that was keeping their code from accurately reading certain sections of double-precision coverages. I wrote E00Tools to collect a handful of methods for working with the resultant data.

I setup a Google Code project for this work, as topology will likely form the basis for a decent amount of my cartographic experimentation in the near future.

• to browse the code, just go here
• the latest zip distribution is available here
• three examples are included in com.indiemaps.mapping.data.parsers.e00.examples

Oh, BTW:

ESRI considers the export/import file format to be proprietary. As a consequence, the identified format can only constitute a “best guess” and must always be considered as tentative and subject to revision, as more is learned.

(from the Arc/Info Export Format Analysis)

How to use

After loading your ASCII E00 file into a string, use something like the following to parse it.

var data : Object = E00Parser.parse( e00Text );

The returned data object includes all information contained in the file, and can have as many as nine sections. Of most use are the arc (non-redundant list of polylines), pal (list of all polygons and their associated lines), and ifo (attributes and labels) sections. The exact structure of the returned object is described on the wiki here.

There are three sweet examples to be found in com.indiemaps.mapping.data.parsers.e00.examples.

Tools

E00Tools contains some methods for working with the resultant data of E00Parser.parse(). Included are methods for:

• Drawing all features
• Drawing individual features
• Getting a list of polygon IDs for all features
• Getting the centroid of a feature
• Getting the shared border length of all features and their neighbors

Key above is the idea of the feature. Michigan is a feature. Features are not directly encoded in E00 files like they are in other formats. In a polygonal shapefile, for example, each feature is encoded as a multipolygon, constituted of one or more rings of points. In E00 files, only polygons are directly encoded; feature information (which polygons make up which features) can be ascertained from the INFO (ifo) section.

Experimentation

I created these AS3 classes for myself, because I wanted to experiment with topological geodata in visualization and cartographic applications. This typically boils down to knowing which features are neighbors and how much of a border they share. The E00Tools methods getAllFeatureNeighbors and getAllFeatureSharedBorderLengths gives you easy access to this information.

Daniel Dorling popularized the circular cartogram form among academic cartographers, outlining the symbology most notably in his 1991 PhD thesis and 1996’s Area Cartograms: Their Use and Creation (available here in PDF form along with many other gems of quantitative geography). Dr. Dorling made Pascal and C code available. I ported it to Python, and began experimenting, mostly in vain, on a method that worked with a shapefile as input, but without the expense of building topology. It produced at best a pale imitation. Dorling describes the gravity model used to produce the cartograms in his dissertation:

The algorithm which was developed to create the area cartograms worked by repeatedly applying a series of forces to the circles representing the places. Circles attract those they are topologically adjacent to; the strength of this attraction being greater the larger the distance is between them and the longer their common boundary.

The algorithm thus requires the shared border lengths of all features and their neighbors. Producing this info is easy with E00Tools, but it seems kind of backward to parse my geodata in ActionScript only to produce the rendering in Python. I’m working on porting Dorling’s algorithm to AS3 so I can go directly from geodata to cartogram without switching platforms.

Lee Byron mentions another technique, used to generate the Olympic Medal Count cartograms he helped produce for the Times. Byron didn’t release any code, but notes that a soft body force directed layout algorithm written in ActionScript was used. I haven’t been able to reproduce his method, but I’ve included an example that drops the topological information gathered from an E00 file into a Flare visualization using a force directed layout. The example is minimal, but shows how the E00 classes can be integrated with the Flare visualization API, and may point the way to a slightly different method for producing circular cartograms client-side.

]]> http://indiemaps.com/blog/2009/02/e00parser-an-actionscript-3-parser-for-the-arcinfo-export-topological-gis-format/feed/ http://indiemaps.com/blog/2009/01/political-cartography-voting-with-our-pocketbooks/ http://indiemaps.com/blog/2009/01/political-cartography-voting-with-our-pocketbooks/#comments Mon, 19 Jan 2009 10:46:11 +0000 zach'ry http://indiemaps.com/blog/?p=80 These election maps are kinda late. Here I’m interested in comparing how we, as a country, voted with our ballots versus how we voted with our dollars. Obama received about 70% of the money donated to the major candidates in 2008, but only 53% of the votes, so I expected a bluer map. But I wasn’t sure what the spatial distribution of the difference would be.

As a first blush, the state level is alright (sorry Alaska, Hawaii). Here I’m showing the proportion of the dollars donated to the major candidates that went to Barack Obama.

Compare that blue and purple beauty, with only Mississippi to be embarrassed of, with this — the results of the popular vote.

Some of those states were obviously more consequential to the candidates’ finances. Here’s an interactive cartogram, sized by either votes or total dollars. Both cartograms use the 10th densest state as the “anchor unit” (in both cases, New Jersey), so comparisons between the two are meaningful. I talk more about that in my post on noncontiguous cartograms.

The state view is too coarse. The obvious choice is the county level, but such aggregated data is not available from the FEC, nor from the NY Times Campaign Finance API, where I retrieved all the finance data for this post. The data are available as individual records, or as summaries requestable by state or ZIP code.*

So I wrote some Python scripts to retrieve and process all 32,800 ZIP codes available from the Times API. There are more ZIP codes out there, but perhaps they had no donations in 2008. This had to be spread over a few days, because the Times limits requests to 5000 per day per API key.

Thanks to the shapefiles available from the Census here I was able to map the proportion of donations to Obama from those 32,800 ZIP codes. But too many ZIP codes lacked donations, leading to an unsightly choropleth characterized by radical change and data-less regions. Best to aggregate to larger units (but smaller than the states above). The NY Times made some nice interactive campaign finance maps, candidate-by-candidate, and aggregated to sub-state regions (ex. “Southern Wisconsin”, “Eastern Shore and northern Maryland”). I’ve settled on a finer unit, the three-digit ZIP Code Tabulation Areas (ZCTAs) of the Census Bureau (an aggregation of ZIP codes based on their first three digits). These first three digits correspond to the sectional center facility of the USPS that serves the area. Though that sounds rather arbitrary, the Census Bureau has aggregated to such units in some of their data since 2000. The following shows donations originating in the 877 three-digit ZIP code regions of the U.S.A., using the same color scheme as the maps above.

As above, compared to the outcome of the popular vote (but by county):

I’ll spare you the ZIP code regions noncontiguous cartogram. Cartograms rely on the recognizability of features on the distorted image, and 3-digit ZIP code regions lack familiarity save when they happen to line up with county boundaries. A better technique in such cases is described by Andy Woodruff of Axis Maps:

It’s a standard red-blue map indicating the winner of each county in the lower 48 states, where the transparency indicates the population of a county. The many counties with low population fade into the background, diminishing their visual prominence. This is meant to accomplish something similar to a cartogram, where sizes are distorted to show the actual distribution of votes.

Their election maps adapt the technique of encoding uncertainty information in transparency initially suggested by Alan MacEachren in 1992 and refined by Igor Drecki in 1999.

Andy tells me they grouped counties into 16 opacity classes using the natural breaks (Jenks optimization) method. I do the same here for my ZIP code regions. This method minimizes the sum of deviations from class means, thus producing an optimal classification. Sixteen classes ensures the appearance of a smooth gradient of transparencies. I used R and the add-on package classInt to create the classification. Here then: finance compared to votes, with both opacitized by consequentiality (total dollars donated in one case, total votes cast in the other).

And here the same over a white background (thus switching the visual variable representing consequentiality to saturation).

I’ve said very little about what these maps actually show. I’ll let the maps do the talking on that, though please do contact me if you’d like the data used in these maps for your own experiments.

One thing I’ve neglected to mention thus far: all of the above graphics were produced with ActionScript 3, using just a text editor and the latest free Flex SDK. I used Python to retrieve and process the campaign finance data, OpenOffice to paste the processed data into the DBF files of the shapefiles retrieved from the Census Bureau, and R to classify the data. It’s pretty sweet that such visualizations can be created using only free tools and data.

update: As I toiled on my ZIP code detour, it turns out GeoCommons Finder was accumulating the data I craved. As described there: “The monthly individual donor data was downloaded from FEC (Federal Election Commission), geocoded and then aggregated to county level for the lower 48 states.” The data provided there by county will still require some processing and doesn’t cover the full range of the data presented by ZIP code region above, but the common county aggregation makes further comparisons with voting data possible, and I’ll show some bivariate maps utilizing this new data in the near future.

]]> http://indiemaps.com/blog/2009/01/political-cartography-voting-with-our-pocketbooks/feed/ http://indiemaps.com/blog/2008/12/early-cartograms/ http://indiemaps.com/blog/2008/12/early-cartograms/#comments Mon, 08 Dec 2008 08:40:26 +0000 zach'ry http://indiemaps.com/blog/?p=78 I’m kind of on a cartogram kick lately. I’m interested in the pioneers of the form, those who first thought to distort borders and explode topologies in order to convey the distribution of some thematic variable. When was the first cartogram produced, where, and by whom? I ran into a lot of material while researching my thesis; this post only begins the discussion.

1868

The honor typically goes to Émile Levasseur for the diagrammatic maps contained in his 1868 and 1875 economic geography textbooks.

H. Gray Funkhouser (1937) wrote of these “colored bar graphs”,

squares proportional to the extent of surfaces, population, budget, commerce, merchant marine of the countries of Europe, the squares being grouped about each other in such a manner as to correspond to their geographical position

Interestingly, Waldo Tobler (2004) points out that the example printed by Funkhouser (above) was sized by land area and thus not a true value-by-area cartogram. I don’t have access to Levasseur’s texts, and it’s odd that the only available scan of Levasseur’s first cartogram shows a diagrammatic map, not a true cartogram.

1897

On the other hand is the image below, whose units are definitely sized to the data, but whose geographic arrangement is questionable. I first saw this page from an 1897 Rand McNally Atlas of the World in a SpatialCollective post; a high res version is available from the David Rumsey Map Collection.

Circles on the left are sized proportional to population, those on the right to debt. Though the arrangement seems haphazard, geography is not ignored as the circles are grouped together by continent. I don’t really buy these as cartograms, but they’re certainly a predecessor to the circular cartogram form popularized by Danny Dorling nearly 100 years later.

1903

Others refer to the election maps of Hermann Haack and Hans Wiechel as the first cartograms (presumably because the popular example of Levasseur’s earlier technique was scaled by land area). These cartograms, which show election results in the Reichstag, were sized to population and of the same rectangular form as the earlier Levasseur diagrams.

Though these maps are mentioned in the second volume of Eckert’s touchstone history of cartography, Die Kartenwissenschaft (1925), they’re not reprinted there, and I’ve no access to the original sources.

1911

Professor John Krygier dug this one up — perhaps the first American cartogram, and an interesting example of the form.

This is the earliest non-rectangular cartogram I’ve seen of any provenance, and it is unique in maintaining the exact outer shape of the United States, while abandoning unit shape and position.

1929

Another early American example, the above map by Joseph Grundy was published by the Washington Post in 1929. States are scaled “on the basis of population and Federal taxes”. Though somewhat rough, I consider this the first modern cartogram, as it maintains topology without abstracting to rectangles.

1934

Erwin Raisz was the first to give cartograms academic attention, describing their production in “The rectangular statistical cartogram” (1934) and devoting significant coverage to the form in his popular cartography textbooks.

1961

A later example, but still a first. In 1961, Waldo Tobler arrived at the University of Michigan as an assistant professor and began working on the first computer programs for cartogram production. His “pseudo cartograms” were created by expanding or compressing the lat/long grid until the minimum root mean squared error of unit densities resulted.

Errors were typically quite high, though likely no higher than those on the manually-produced ones that came before. All subsequent cartogram algorithms can be considered variants of Tobler’s method.

]]> http://indiemaps.com/blog/2008/12/early-cartograms/feed/ http://indiemaps.com/blog/2008/12/noncontiguous-area-cartograms/ http://indiemaps.com/blog/2008/12/noncontiguous-area-cartograms/#comments Thu, 04 Dec 2008 07:13:26 +0000 zach'ry http://indiemaps.com/blog/?p=75 notes on noncontiguous cartograms and ActionScript 3 classes for producing them

Fully contiguous cartograms have stretched and distorted borders but perfectly maintained topologies. Like the Gastner-Newman diffusion-based cartograms we see all over the place. Though all sorts of cartogram designs have been produced, those with perfect topology preservation (fully contiguous cartograms) receive the majority of academic and popular press attention.

Some notable exceptions are the well done animated ones by Mapping Worlds and a recent NY Times example showing electors per voter that I’ll return to later. These fully noncontiguous cartograms preserve the shapes of enumeration units perfectly, but don’t even attempt to preserve any borders or adjacencies from the original map.

Judy Olson (Wisconsin Geography alum natch) wrote the only academic article to focus specifically on this cartogram symbology in 1976. She believed noncontiguous cartograms held three potential advantages over contiguous cartograms (I’ve three more below):

1. “the empty areas, or gaps, between observation units are meaningful representations of discrepancies of values, these discrepancies generally being a major reason for constructing a cartogram”
2. production of noncontiguous cartograms involves “only the discrete units for which information is available and only the lines which can be accurately relocated on the original map appear on the noncontiguous cartogram”
3. because of perfect shape preservation, “recognition of the units represented is relatively uncomplicated for the reader”

Despite these inherent advantages (along with ease of production), all the early value-by-area cartograms I’ve seen maintain contiguity. Some took the radical step of abstracting features to geometric primitives, like Levasseur’s early French examples (which may not have been cartograms) and Erwin Raisz’s early American “rectangular statistical cartograms”. But in many ways the noncontiguous design is the more radical cartogram, as it actually breaks the basemap apart — rather than skewing shared borders it abandons them.

my AS3 classes

Olson outlines a technique — the projector method — for manually producing such cartograms. A projector capable of precise numeric reduction/enlargement was required, but not much else, and accurate cartograms could be produced in minutes. A scaling factor was calculated for each enumeration unit, the projector was set to this value, and the projected borders were traced, keeping units centered on their original centers.

My AS3 NoncontiguousCartogram class works similarly. It takes an array of objects containing geometry and attribute properties and creates a noncontiguous cartogram. I include methods for creating the input array from a shapefile/dbf combo, but using KML, WKT, or geoJSON representations wouldn’t be too hard. Methods are included for projecting this lat/long linework (to Lambert’s Conformal Conic projection at least). The NoncontiguousCartogram class draws the input geography, figures the area of each feature, and scales figures according to their density in the chosen thematic variable.

It’s all good/in ActionScript 3, so can be used in Flash or Flex. The zip distribution includes the following:

• the main NoncontiguousCartogram.as class
• two example applications and the data needed to run them
• utility classes, including some that make creating cartograms from shp/dbf input quite easy
• Edwin Van Rijkom’s SHP and DBF libraries, which are used to load the shapefiles in both of the included examples
• Keith Peters’ MinimalComps AS3 component library, for the components used in one of the examples
• Grant Skinner’s gTween class, which is required by the NoncontiguousCartogram class for tween transitions

Browse all the above or download the zip.

Flash examples

I created a few examples, all started from just shapefile input. The following are screenshots, but there are interactive examples further down. First, an example out of Olson’s article, presumably produced using the “projector method” described above:

Here I’ve updated Olson’s example with more recent data and a snazzy red outline:

This one took a few secs to load the shapefile, project to Winkel Tripel, draw, and scale the cartogram by population:

The above is great as an artistic rendering, but is probably a bad depiction of the data as 10% of the counties have increased in size and there is a fair bit of overlap. Decreasing the alpha of the counties reveals the overlap, and creates a really interesting rendering of the data.

If you’ve read this far, you probably at least want to see some shit move around or something. Here’s an example I put together with election data; it shows off the basic updating and tweening capabilities built into the class. Notice how all features are drawn around their centroid and therefore remain centered as they change scale. Tweening is via Grant Skinner’s gTween and interface components are from Keith Peters’ MinimalComps library, which is great for prototyping in only AS3.

In the above, the “electors per voter” option is based on the same data as the well-made NY Times example I mentioned earlier, but something’s fishy about one of our scalings, because they don’t line up.

design considerations

Position and size. Both affect the one thing you’re trying to avoid on a noncontiguous cartogram: feature overlap. If features are left at their original centroids, overlap is likely if any units are enlarged. When sizing features on a noncontiguous cartogram, one feature is chosen as the “anchor unit”. This feature will stay the same size, and all others will be scaled up or down depending on whether their density is higher or lower than the anchor. If the unit with the highest density is chosen, overlap will be avoided because all other units will be reduced. This can often produce large areas of white space and units too small to recognize.

To address this, both the anchor unit and the feature centers are configurable. The “anchor percentile” is set in the class constructor, and can range from zero to one. If set to one, all but one feature will be reduced; if set to zero, all but one feature will be enlarged.

By default, features scale around their original centroids. You can pass a Dictionary of feature centers (using the objects contained in the combined array as keys) to the constructor (or via the settable property, featureCenters), which allows you to manually position the features. I include the following example to give you some ideas for how to use this feature. I create an interactive cartogram on which features can be moved around. Well Known Text representations of the feature centers are generated and converted into the required Dictionary via some included utility methods. Check out the source of this example to see what’s going on.

projections

In her 1976 article, Judy Olson used linework projected with an equal area projection. This was a requirement of her method, as she was using known land area measurements in her calculation of densities and scaling factors. My class calculates areas of features — projected or otherwise — before scaling, so any projection can be utilized and an accurately-scaled cartogram will result. In the examples above, I use a conformal projection (included in the source) on the points before passing to the cartogram class. Conformal projections preserve local angles (shape, sort of) and may result in features that are easier to recognize.

That said, for most applications it’s probably a good idea to use equivalent (equal area) projections before constructing these cartograms, as the divergence of the thematic variable from land area is often the point of creating the cartogram in the first place.

more advantages

In my thesis research last spring, noncontiguous cartograms performed quite well: subjects rated them highly on aesthetics and could locate and estimate the areas of features with relatively high accuracy. I would add the following to Olson’s list of noncontiguous cartogram advantages.

1. Olson concentrates on the perfect shape preservation of noncontiguous cartograms. The form (well, those with units centered on the original enumeration unit centroids, as in Olson’s projector method) also perfectly preserves the location of the features on the resultant transformed cartogram. Not only are features easier to recognize, but locations within the transformed units can be accurately located as well (cities or mountain ranges from the original geography can be accurately plotted on the transformed cartogram).
2. Because units are separate on the transformed cartogram, their figure-ground is increased and areas of features can therefore be more accurately estimated.
3. Many cartogram designs (including most manual cartograms and the Gastner-Newman-produced cartograms) sacrifice some accuracy for shape recognition. This is a defensible tradeoff, especially as area estimation is notoriously inaccurate and nonlinear. Yet it’s a tradeoff that noncontigous cartograms need not make, as they can always perfectly represent the data with relative areas without sacrificing shape preservation.

Thus, noncontiguous cartograms seem to excel at the cartogram’s two main map-reading tasks: shape recognition and area estimation. This is mediated somewhat by the chief advantage of contiguous cartograms: compactness. Because no space is created between enumeration units, contiguous cartogram enumeration units can be larger than those on noncontiguous cartograms, all other things equal. The increased size on contiguous cartograms may improves their legibility.

]]> http://indiemaps.com/blog/2008/12/noncontiguous-area-cartograms/feed/ http://indiemaps.com/blog/2008/10/nightingales-roses-in-actionscript-3/ http://indiemaps.com/blog/2008/10/nightingales-roses-in-actionscript-3/#comments Tue, 21 Oct 2008 08:30:00 +0000 zach'ry http://indiemaps.com/blog/?p=70 I’ve long been a sucker for the polar area/coxcomb/rose charts popularized by Florence Nightingale. These multivariate charts can show ordered or unordered categorical data. As noted in an Economist piece on influential information graphics,

As with today’s pie charts, the area of each wedge is proportional to the figure it stands for, but it is the radius of each slice (the distance from the common centre to the outer edge) rather than the angle that is altered to achieve this.

I wanted to produce some just for kicks, so looked around for a script in AS3. No dice. OK, any language? Didn’t see anything. So I sat on the idea for a while and then finally thought up the technique that made producing them in AS3 quite easy. With the resultant classes, producing graphics like the following small multiples of U.S. soldier deaths in Iraq is a snap. The classes are written in AS3, so can be used with Flash, Flex, or mxmlc. All the example screenshots below are PNGs captured from SWFs produced with only AS3 (extended Sprites). To see the code (which includes a lot of ugly annotation), click ‘view source‘ below any image. All source code is included in the ZIP distribution linked below.

U.S. Soldier Deaths in Iraq, March 2003 to October 2008

view source

The above chart series utilizes the coxcomb in the same manner as Nightingale’s original — as a month-by-month temporal chart. The following is also a small multiples presentation, but utilizes categorical coxcombs to show car color popularity by percent manufactured.

Cars manufactured by color, 2003 to 2006

view source

And as the ultimate test for the classes, I decided to try my hand at reproducing Nightingale’s original graphic. I relied on Hugh Small’s beautiful reproduction of Nightingale’s graphic. I quickly gave up on reproducing the fonts, and indeed a few of Nightingale’s embellishments/idiosyncrasies are not reproduced in my graphic. But one of these nights I’ll extend the class to more faithfully reproduce Nightingale’s original. Nevertheless, not bad eh?

view source

I’m open to the criticism that these coxcombs (and other area-based symbologies) are rarely preferable to a simple bar chart (a length-based symbology). But I still believe they ought to be a part of the information designer’s toolkit, especially as they hold definite advantages over their more popular offspring, the pie chart. For animated or small multiples applications in particular, the coxcomb is preferable to the pie chart because of the constant position/angle of slices. It’s easy to glance across a series of small multiples coxcombs, or watch an animating coxcomb, and follow individual slices. I’ve built basic interactivity and tweening (using Grant Skinner’s lightweight GTween Engine) into the class. See this discussion over at Edward Tufte’s site for some of the advantages and disadvantages of coxcomb charts.

Further, though bar charts may be easier to read, they may not work in all contexts, including interactive mapping, in which compactness may outweigh other concerns.

Here’s an example of some of the basic tweening and interactive capabilities of the class.

view source

When I initially thought of coding these charts, I balked at the idea of constructing slices whose relative areas would faithfully represent the data. But I eventually realized that I could just use masked circles. The circles could be accurately scaled to the data, and since each segment would mask the same percentage of the circle, the resultant pie slices would faithfully represent the data. The technique (demonstrated in the diagram below) has the added advantage of allowing tweening of slices in animated displays.

the main technique behind CoxcombChart.as — the masking and rotation of CoxcombSlices

A bit easier said than done, since the strokes on each slice need to be drawn. But that boils down to geometry. Each slice need only be rotated into place to form the final rose chart.

I’ll create a few more examples in the future — I’m particularly interested in the cartographic applications of this multivariate chart. I’ll likely post a Modest Maps layer showing coxcomb charts of average precipitation-by-month for major cities, or something, in the near future. For now, here’s all the code used to produce the graphics above. It should be pretty easy to produce a coxcomb chart of any data using this code, and my interface is flexible enough to modify the appearance and interactivity of the charts for most applications. But feel free to extend the class and override methods to alter these properties.

• A ZIP of the full distribution. This also includes Grant Skinner’s lightweight GTween Engine (a single .as class) which is required for the class. Andy Woodruff’s DashedLine class is also included, as it is used in the Nightingale example.
• For you browsers, view all the source here

Creating the coxcombs should be pretty easy. Here’s an example instantiation of a non-interactive coxcomb, 350 pixels wide/tall, with 3 slices, each a different color.

var myCoxcomb : CoxcombChart = new CoxcombChart(
    [
      { label : 'red', value : 15 },
      { label : 'white', value : 20 },
      { label : 'blue', value : 25 }
    ],
    350, [ 0xff0000, 0xffffff, 0x0000ff ], 0x888888
);

]]> http://indiemaps.com/blog/2008/10/nightingales-roses-in-actionscript-3/feed/