Clip polygon layer by another polygon layer

Ok, so I did this once, didn't write it down and then couldn't figure out how to do it again.

I continue to lament on the 'improvements' to arcGIS. Version 10 is not my idea of good software design.
Anyway, I have a number of polygon shape files, that design buffers around existing agricultural areas in the landscape. However, I want to remove the existing urban read from these files to ensure that agricultural expansion cannot override existing landscape features - roads, protected areas, urban centres etc. I don't have the 'erase' tool seeing as this requires the advanced version of arcGIS10.

So what I figured out I need to do is use the 'clip' function, but in reverse; i.e. create a shape file that identifies all the areas outside of protected areas, urban centres, etc and then clip the agricultural files by this 'non' file.

Analysis Tools - Extract - Clip

Finally!

A bit of a workaround, but this seems the best that arcGIS10 has to offer.

To create the 'non' file, use the Union function:

Analysis Tools - Overlay - Union
select Urban polygon file
select corridor outline shapefile

output file identifies urban areas on top of corridor file
then simply remove the urban areas via the attribute table - select urban polygon and delete from the file. done!

then need to do this again with the corridor designs 1 and 2.

repeat with the urban area - removing existing agricultural areas as well as protected reserves and corridor designs (in separate files).

Arc GIS 10

I am completely frustrated by this newest version of Arc GIS. My previous version of 9.3 was easy to use, straight-forward and functional. This new version has a complicated menu arrangement and features that were easy-to use and 'available' in version 9 are now not available in the new version!

I have to say, the look and feel of the programme has not been improved either. Floating menu boxes make it easy to lose menus, and the map does not adjust to fit the screen when a new menu is added: for instance, when I add Arc Toolbox, which now seems to be hidden within the Geoprocessing option in the toolbar, I have to adjust the underlying map and the toolbox sits awkwardly at the side of the screen.

I have also been attempting to use a simple feature of GIS all day - all I want to do is clip a shape file by another shape file so that polygons in one file do not overlap with polygons in the second file and it seems impossible! Apparently, the erase tool (which online sources seems to suggest will perform this function) is only available under the advanced software download. Ridiculous!

I am NOT impressed.

Distance to urban areas in GIS

I've just finished the last updates to my GIS data for using in my ABM and I generated such a cool picture that I thought I would share:

This is a combination of the distance to the main urban areas and distance to tarmac roads. I removed all of the tracks and trails from my road file as this layer is supposed to represent a general cost layer where proximity to humans is detrimental to jaguars and they would rather be in the low cost areas to the far left and upper regions of the map.

Some things to remember where trying to generate these types of maps:
1. the euclidean distance method seems to be the most appropriate:

                 Spatial Analyst - Distance - Euclidean Distance

2. this method can be used on shape file data or on raster data making it a pretty flexible function
3. the output will only be generate for the exact extent of the data you are performing the function on. For instance, if you have polyline data, then the distance will extend only until the smallest rectangle that includes all the lines, and NOT the surrounding NoData areas. Therefore you need to do the distance function on a larger dataset and then clip it to the area you want if this is a problem.
4. In my case, this was not possible so I converted my polyline shape file (roads) to a raster first, then extended the extent to the area required by my model and then performed the distance function.
5. To get the map above, I combined distance maps for urban areas and roads.
6. My urban areas were actually polygons and the distance function worked just fine with this.
7. As I said earlier, I decided to remove the small roads and tracks from my roads dataset before calculating the distances as these do not impact negatively on the movement of my agents (jaguars).
8. Once I had the two datasets that were each of the correct size and extent, I simply added the layers together:

                  Spatial Analyst - Math - Plus

9. Adding the two layers together creates a new layer that matches the size of the smallest input raster, so I actually added together the roadProximity file (purely for the model area) together with the urbanProximity file for the whole of Belize. 

GIS update - transforming coordinate systems

Following on from a previous post about getting my data files ready to use in my model, I need to update the transform coordinates instructions.

I previously stated that this could be done effectively from within ArcMap. However I have been having problems getting this to work correctly through the transform command within ArcMap (View - Data Frame Properties - Coordinate System - Transformations).

Some of the data I have been provided with has an underlying coordinate system of WGS_1984_UTM_Zone_16N, but camera data I have has been supplied in NAD_1927_UTM_Zone_16N and so my data files need converting to the later coordinate system.

There is a difference between the Geographic Coordinate System and the Projected Coordinate System. The supplied WGS_1984 is from the Geographic Coordinate System and transforming this only allows you access to other Geographic Coordinate Systems. The correct NAD1927 coordinate system is not geographic but projected and so the transformation function within ArcMap does not adequately transform the data....

so... this can be done more effectively from within ArcCatalog...

Navigate to the appropriate file in ArcCatalog, i'll run through this for my data layer of rivers

1. Select modelRivers.shp
2. Click on File - Properties
3. Select XY Coordinate
4. Choose Select
5. Chose Projected Coordinate Systems - UTM - NAD 1927 - NAD 1927 UTM Zone 16N.prj
6. Click Add
7. Click Apply - OK

Note, you cannot change the coordinate system if the file is already open in ArcMap, so you need to remove the layer from ArcMap before attempting to change the coordinate system.

Some GIS analysis

Ok, so I'm fed up of doing something in GIS, forgetting to write it down and then having to repeat the process due to an update of GIS data, or a change in my model needs etc that I am now trying to make a note of the main GIS analysis to get the data into some format that I can then use in my models, so here goes.

Data I need at the moment:
Habitat Map in raster format, cells of size 100m x 100m
Roads and trails, in raster format, cells of size 100x100
Cockscomb, raster - a simple map of where the protected forest is
CameraTraps - a shape file of the location of all 47 camera traps (in suitable projection to match the raster data
Sample area - a raster dataset of a buffer around the camera traps that represents the sampling area of the traps
Rivers, in raster formation, cells of size 100x100
Model Area, a shape file created to represent the area to be used in the model

Each of these datasets also needs to be of the same area (an annoying problem in some cases)

So, all data was provided in shape file format which meant it all had to be converted or processed and then converted to raster.

All information is relevant to ArcGIS version 9.3. (ArcMap unless otherwise stated).

HabitatMap
1. A simple conversion from 'feature to raster', using Ecosystem field as the field of interest and stating a cell size of 100.

[Conversion Tools - To Raster - Feature To Raster]

2. This raster then covered the entire area of Belize, so it needed to be clipped to meet the model area of interest. (I'll discuss how to create a shape file of the required area next), but otherwise use the 'clip' command. Some commands from the various toolboxes only work on specific types of dataset, such as the clip command within the Analysis Tools toolbox only works with feature/shapefile data and NOT with rasters. For Rasters, use the following, putting the shape file to use as the 'clipping' feature in the 'output extent' box and selecting the 'use input features for clipping geometry box', otherwise you can designate your own coordinates to create a rectangle to use to clip...

 [Data Management Tools - Raster - Raster Processing - Clip]

3. Now we should have a raster of cell size 100x100, but only having data for the required region/area of Belize. This now needs to be converted to an ASCII file for me to use and import into RePast.

[Conversion Tools - From Raster - Raster to ASCII]

Model Area

For this we need to create a new shape file and then amend it to create a new polygon for the area we want to use as our model area.

1. Open Arc Catalog, locate the folder you want the file to be created in.

2. Right click in the main box (showing contents of the folder you are looking at), go to New, and shape file. Then give it an appropriate name.

3. Ok, now go back into ArcMap and add the new shape file to the display.

4. Now you need to edit the shape file

[View (from the top bar) - Toolbars - Editor - Editor - Start Editing]

5. At this point, you may need to select the fold or database which you would like to edit. Select the folder which houses the newly created and added shape file, click OK.

6. Make sure the correct shape file is displayed in the Target box on the Editor Toolbar

7. Now you need to create a polygon of the area you want to be the model area

[Editor -More Editing Tools - Advanced Editing]

8. A new toolbox should have popped up called Advanced Editing. Select the rectangle icon (last but one on the right) and now you should be able to draw a rectangle within the correct shape file layer. You can have other layers open so you can see where to draw the rectangle. If the rectangle is selected it can also be deleted so it doesn't matter if it isn't correct first time. Once you have the area you are happy with, go back to the editor toolbar

[Editor - Save Edits- Stop Editing]

9. There should now be a shape file with a single polygon representing the outline of the model area you want to use. This shape file can now be used to 'clip' any dataset required (as in step 2 above).

Cockscomb

1. Ok, so I was given a dataset with all the protected areas outlined on it, as well as a single dataset with the outline of Cockscomb Basin Wildlife Sanctuary. This simply needed to be clipped to the model area (my model area only covers a portion of the sanctuary) and then converted to a raster and then ask ASCII.

So first, clip the polygon of Cockscomb, and use the newly created ModelArea file (above) as the 'clip features'

[Analysis Tools - Extract - Clip]

2. Now convert it to raster (you could also convert it to a raster first and then use the raster clip function (outlined above) to clip the raster).

[Conversion Tools - To Raster - Feature To Raster]

3. Now convert the raster to an ASCII file

[Conversion Tools - From Raster - Raster To ASCII]

Roads/Trails and Rivers

1. These are both done in the same manner so can be dealt with together. They came in polyline format (shape file) so they need processing and then converting to a raster. Firstly, they needed transforming form they current WGS-1984 projection to the required NAD1927-16N projection (all GPS and camera data information is in the latter format). This is relatively simple. If you have other data open when you add the roads/rivers data it will tell you it is in a different coordinate system and you can change it from that pop-up box, otherwise do the following:

[View (top bar) - Data Frame Properties - Coordinate System]

2. You should now see what coordinate system your current data is in, if it needs to be change:

[Transformations - Select the correct coordinate system in the 'Convert from' box and make sure the correct new coordinate system is in the 'Into' box, now select a transformation technique from the 'Using' box, (they should all do the same job), then press OK - Apply - OK] 

Data should now match.

3. Now the data is in the correct coordinate system, it needs to be processed. I had several different files that needed merging together to create a single file of the entire road/river network. This can be done several ways, but the easiest (and most effective) way I found was via the editor toolbox. I also needed to create some additional trails from point data supplied.

4. Creating new trails, need to create a new shape file layer in Arc Catalog (as above for the Model Area). Add the new empty layer into ArcMap. Display the data of points that represents the information you need. 

[View - Toolbars - Edtor - Editor - Start Editing]

5. Now all you need to do is draw over the points to create a line. The trick is to make lots of joins to make sure the line follows the points. Selecting the pencil icon on the editor toolbar allows you to draw onto the empty shape file layer. A single click make a join and a double click ends the current line. You can connect separate lines together at the end, so I zoomed in and make several lines that I then joined up at the end to create the entire trail network. Once all the lines have been drawn, you can then merge them.

 [Click on the arrow head on the editor toolbar - draw a large square around your newly created lines to select them all - Editor - Merge]

6. Now all the lines should be joined up but still selected. Selection(from top bar) - Clear Selected Features, will clear a selection at any time. Now you need to join the newly created trails with the several existing datasets. A similar process can work successfully, or you can do it via the Data Management Tools 

[Data Management Tools - General - Merge]

7. Now you should have one layer with all the relevant roads or rivers in. Now this needs to be clipped using the Model Area shape file.

[Analysis Tools - Extract - Clip]

8. Again now convert it to a raster and then to an ASCII file.

9. Some problem I encountered - the original roads data was given to me, not of the whole of Belize, but of a small area within the area I was using as my model area. Therefore the dataset was SMALLER than the model area I needed. Problem!

10. Much investigation didn't come up with a good process for extending the display of the dataset (it only needed additional NoData information to make up the entire area). You can extend the display of the layers 

[right-click on the layer - Properties - Extent - choose any layer in the 'set the extent to' drop down box]

11. Whilst this seemed to work when I converted the layer to a raster it lost this additional extent. So do this instead, you need to convert the ModelArea shape file to a raster and then add the two raster layers together:

[ModelArea = Conversion Tools - To Raster - Feature to Raster]

[View(top tool bar) - Toolbars - Spatial Analyst - Options - Extent - Union of Inputs] then

[Spatial Analyst - Raster Calculator - select the new raster of the modelArea then select the + and then select the roads raster file]

12. This should create a raster layer of the roads that same size as the modelArea. The raster layer of the ModelArea needs to have a non-negative, non-zero value (mine went to 1 automatically) so that this value is added to the value of the roads (I have 6 classes of roads all with a value from 1 to 6 representing the class). The values of my new raster went from 2 to 7 so I used the reclass function to put them back to 1 - 6, with the NoData values sticking at -9999. I had some problems converting these values to anything else in the reclass function so I left them at the default value.

[Spatial Analyst Tools - Reclass - Reclassify - choose the new roads raster and enter in the new values in the NewValues column in the box in the centre]

Camera Traps

1. These were fairly straightforward. I had the data as x and y coordinates in an excel spreadsheet. So add the spreadsheet to the ArcMap display:

[right click on the spreadsheet - Display XY Data - and choose the correct columns that represent the x and y data - choose the correct coordinate system, via - Edit - Select - Projected Coordinate Systems - UTM - NAD 1927 - NAD 1927 UTM Zone 16N.prj - Add]

2. You might then need to make the layer permanent, to do this you need to export it as a shape file:

[right-click on the new 'events' - Data - Export Data - then choose All Features in the 'Export' box and the folder you want in the 'Output shape file or feature class' box and give it a name you want - OK]

Sample Area

1. So this is the effective sampling area of my dataset - basically just a circle around each camera Trap of a specified radius - calculated from the average home range of a jaguar - 1784m in this case (half of the full estimated diameter of 3568m - taken from thesis if Rebecca Foster (currently working for Panthera in Belize and collaborating on this work)). So to do this you use the buffer function: 

[Analysis Tools - Proximity - Buffer - new cameraTrap file as the 'Input Features', choose an appropriate name and location for the output file, put the radius in the 'Linear Unit' box and make sure the next box is in the correct units]

2. This now needs to be converted to a raster and made the same size as the modelArea, then converted to an ASCII.

All these files need to be converted to .pgm files (apart from the cameraTraps) for importing into RePast. I'll cover this in a later post.

GRASS GIS

I'm now mainly working on Mac OS so using ArcGIS is becoming a bit of a problem. I have it on my Del laptop, but this is getting a bit slow now and it takes a while to do anything with the big datasets that I need to use of Belize. I also have VMWare on my iMac, but its linked to my university desktop environment, rather than being standalone and so the whole windows experience is quite a painful one.

Roll in - GRASS. An open source GIS package, find more information here.
GRASS works on any platform which makes it appear a good choice. I've yet to test its capabilities but I'll be updating my progress with it in the near future.

To get GRASS working, it requires three frameworks (GDAL Complete, Freetype and Cairo) to be installed prior to installing the GRASS package. Find them, and more info at www.kyngchaos.com/software/grass.