Washington Maps and GIS Data

One of the best sources of Washington map data is (surprisingly) Washington State Office of Financial Management.

WA OFM Geospatial Data
Some useful Census data (like water features), but as Windows self-extracting zip exe’s(!). I get the same md5sum’s “running” it under wine as I do by renaming the .exe to a .zip and unziping it.

has data for all of Washington State–satellite photos, .tiff topo maps, and DEMs.
Digital Ortho Quads (registerd gray-scale photos with 1-meter pixel size)
Digital Raster Graphics (scanned 7.5′ topo maps)
clipped Digital Raster Graphics (scanned 7.5′ topo maps without borders)
10-meter DEMs (by 7.5′ quad; look here for other formats)

KC GIS Center
Don’t bother looking here; these datasets are only available to County employees or on a $200+ DVD. The datasets are browseable through iMAP, but all you can get out of it is a screenshot.

GIS File Formats

Of course there’s no Open, widely adopted standard for map data. Here are dome of the file formats I’ve encountered:

Digital Elevation Models. “The USGS DEM standard is a geospatial file format developed by the United States Geological Survey for storing a digital elevation model (or surface elevation).”1

Digital Orthophoto Quads

Digital Raster Graphs. These can be viewed using the free version of ESRI’s ArcExplorer GIS.

an (ESRI) ArcInfo (aka ARC/INFO) interchange/export file.

.img files register as Garmin MapSource Map.

.mp files are for some reason called Polish Format, at least by the folks at GPSMapEdit and MapCenter.

.shp files register as ESRI shapefiles. This seems to be one of the more common formats.

TIGER/Line File
Topologically Integrated Geographic Encoding and Referencing, or TIGER, or TIGER/Line is a format used by the United States Census Bureau to describe land attributes such as roads, buildings, rivers, and lakes.2

Florida Maps and GIS Data

Florida has all kinds of great data available. The two sites I’ve been working with are

FDOT has many different Road and Road Data datasets, but unfortunately most of them do not contain street names, only road ID numbers.

The “Roads with Local Names” dataset includes street names for 14599 roads statewide, but does not contain neighborhood streets.

FGLD has USGS 1:24k road maps, but they too forgot to include the street names.

FGLD also has TIGER maps at 1:100k; they’re newer, and include the street names. Both the TIGER and USGS road maps must be downloaded on a county-by-county basis.

FGLD even has numerous line and polygon hydrographic maps. I got to these by clicking ‘download’ on the TIGER road map and then changing the Data Layer.

Unfortuneatly, all the FGLD maps use the Albers Conical Equal Area projection, which GPSMapEdit can’t read. MapWindow GIS can–but I haven’t figured out how to export anything from that program yet.

Utah Maps and GIS Data

The Utah Department of Transportation’s GIS site is pretty bare.

The Utah Automated Geographic Reference Center has a Geographic Information Database which has all kinds of old data, broken links, and, buried in it’s depths:

From http://agrc.utah.gov/agrc_faqs/dsdlgisdata.html:

Note: Most data contained within the SGID uses the Universal Transverse Mercator projected coordinate system (UTM Zone 12 North) using the 1927 North American Datum (NAD83). UTM Zone 12N, NAD 27 is the current state standard for projection, coordinate system, and datum.

GIS Coordinate Systems

From just playing around with GPSMapEdit I was able to figure out most of the shape file import options. Sometimes, however, I’d open a shape file and be confronted with a big red square, a mass of tangled lines at such a scale that it was impossible to view the entire mess at once.

I never did figure out what the “Coordinate system” and “datum” dropdowns on the 3rd step of the file import were for. I used to just choose different values at random until my maps seemed to line up. Turns out they are what determine not only whether your maps line up but also whether you get anything that resembles a map at all.

The Florida Department of Transportation GIS Data Directory has a wealth of data, and below each of the section headings their site says: Projection: UTM 17; Datum: NAD 83. Sure enough, there’s a UTM coordinate system defined in GPSMapEdit, and a Zone dropdown that lets you choose 17, and NAD 83 just so happens to be one of the available Datum’s.

This inspired me to find out what these things really mean.

“Projection is a fundamental component of map making. A projection is a mathematical means of transferring information from a model of the Earth, which represents a three-dimensional curved surface, to a two-dimensional medium–paper or a computer screen. Different projections are used for different types of maps because each projection particularly suits certain uses. For example, a projection that accurately represents the shapes of the continents will distort their relative sizes.”1 Wikipedia, not surprisingly, has an entire article devoted to different types of map projections.

The National Geodetic Glossary defines datum in general as “Any quantity or set of such quantities that may serve as a referent or basis for calculation of other quantities” and specifically relating to things geodetic as “A set of constants specifying the coordinate system used for geodetic control, i.e., for calculating coordinates of points on the Earth.”2

Specifically, “The North American Datum of 1983 (NAD 83) is ‘The horizontal control datum for the United States, Canada, Mexico, and Central America, based on a geocentric origin and the Geodetic Reference System 1980. This datum, designated as NAD 83, is the new geodetic reference system. … NAD 83 is based on the adjustment of 250,000 points including 600 satellite Doppler stations which constrain the system to a geocentric origin.’ (Geodetic Glossary, p 57)”3

WGS 84 is the World Geodetic System frame of reference.

So depending on what part of the world you’re mapping, and how you expect the map to be used, you may choose to encode the map using different projections and datums. When using maps made by other people, it helps to know what coordinate systems they used. To turn the red blob into a map, I just had to change coordinate systems. The map projection and coordinate system information should be in the map metadata file.

Backups with rsync

Now for another incredibly interesting post, this time about the rsync.

The rsync man page includes a couple examples for using rsync to back up directory trees, but what do all those options do, really? And just as important, which options do I like best?

rsync -Cavz . arvidsjaur:backup

  • -C, –cvs-exclude auto-ignore files in the same way CVS does [includes .svn/ subtrees]
  • -a, –archive archive mode; same as -rlptgoD (no -H)
    • -r, –recursive recurse into directories
    • -l, –links copy symlinks as symlinks
    • -p, –perms preserve permissions
    • -t, –times preserve times
    • -g, –group preserve group
    • -o, –owner preserve owner (root only)
    • -D, –devices preserve devices (root only)
    • -H, –hard-links preserve hard links [“This option can be quite slow, so only use it if you need it.”]
  • -v, –verbose increase verbosity
  • -z, –compress compress file data during the transfer

Other options of note:

  • –delete deletes files on the receiving side which aren’t on the sending side, but only if -r or -d is also included, and only for those directories which are being sync’d; always use --dry-run first to see what would be affected
  • –exclude any path which matches this simple filter isn’t sync’d
  • –progress show progress during transfer [don’t use inside cron]

Compare rsync -avz foo:src/bar /data/tmp and rsync -avz foo:src/bar/ /data/tmp: the first will create /data/tmp/bar, the second will copy the contents of bar into /data/tmp. Without the tailing slash an additional directory level is created at the destination, with the tailing slash just the contents of the directory are copied. Thus, these are identical:

  • rsync -av /src/foo /dest
  • rsync -av /src/foo/ /dest/foo

So for me the command would look something like:

time rsync -Cavz --exclude 'research/gutenberg' --exclude 'research/output' --delete --dry-run /home/steve/data /media/usbdisk/steve/backup

rsync -Cavz --progress --delete --dry-run ~/music/ ideaharbor.org:~/music/

(Make sure you check du -hT on your destination before you begin.)

There are two different ways rsync can connect to a remote system. Which method is used is controlled by the number of colons (:’s) included in the source or destination paths. When using a single colon, as in the examples above, rsync will use ssh to connect to the remote machine as the specified user. Hence any commands (such as creating new directories or writing to files) will be run as the specified user. When two colons are used rsync will connect to the rsync daemon (rsyncd) running on the remote machine, and any commands executed will be limited by the permissions of the user rsyncd is running as. rsyncd is configured using the rsyncd.conf file (see `man rsyncd.conf` for details).

An Introduction to Screen

What is screen?

(4:01 PM) sharmahd: /me is amazed at how many people don't know about screen
(4:04 PM) hulet: I don't know about screen   
I've looked at the man page, but didn't really grok it. It looks like a fancy nohup?
(4:04 PM) sharmahd: how do you survive?
(4:04 PM) hulet: What problem does screen solve?
(4:05 PM) sharmahd: well, hmmm... how do i explain it
(4:05 PM) hulet: exactly

GNU Screen is a free terminal multiplexer developed by the GNU Project. It allows a user to access multiple separate terminal sessions inside a single terminal window or remote terminal session. It is useful for dealing with multiple programs from the command line, and for separating programs from the shell that started the program.1

As I’m learning screen here’s what I’ve found useful to know:

Starting screen:

screen to start
screen -r to reattach

screen -ls to list the currently active screen sessions and their <pid>.<tty>.<host> session names
screen -r <session name> to reattach to a specific session where session name has the format <pid>.<tty>.<host>

Within screen:

C-a c create window
C-a n and C-a p to switch to the next or previous window in the list, by number.
C-a N where N is a number from 0 to 9, to switch to the corresponding window.
C-a C-a switch to the most recently visited window
C-a “ to get a full-screen list of windows.
C-a A to name a window

C-a d detach from session (or just close the terminal window)

C-a [ or C-a <esc> enter copy mode (necessary for scrolling); enter or space delimits selections
C-a ] paste
C-a H toggle logging

C-a s Send a control-s to the current window, which usually has the result of suspending (freezing) the terminal; use C-a q to resume normal operation
C-a S Split the current region into two new ones; use C-a Q to close all regions but the currently active one

C-a ? to get help

I also made a .screenrc file.

powered by WordPress     themed by Mukkamu     presented by ideaharbor.org     everything else by steve hulet