GeoServer: Past, Present, Future Justin Deoliveira Welcome everybody. My name is Justin Deoliveira. I am a GeoServer developer and I work for an organization called OpenGeo. I am hear to talk to you today about GeoServer. This presentation has actually been given at the last 2 foss4g’s, this being the third. And its really about a look back over the past year to see what was accomplished. A sort of “state of GeoServer address” if you will.

What is GeoServer? Geospatial Web Server Open standards vector and raster data Open standards WFS WMS WCS Gateway to the “GeoWeb” Before delving into what was accomplished this year just a brief introduction to what GeoServer is for those of us who have never seen or heard of GeoServer before. If you asked three different people what is GeoServer you would get three different answers. But basically GeoServer is a web server for geo content. It reads spatial data in a variety for formats, and pushes that data onto the web. Just like a web server reads other types of content and pushes it out onto the web. You will often hear GeoServer described in terms of the protocols and standards its supports. I am sure most people here are familiar with the terms WMS, WFS, and many of the other Geospatial web service standards that are defined by the OGC. GeoServer supports a number of these standards including WMS, or the Web Map Service. Which means GeoServer is a map rendering engine, it takes some data, applies some pre-defined styling rules, and spits out a map. GeoServer is also a WFS, or Web Feature Service, which does not something that produces a map or something to look at, but delivers the data behind the map. Similar in nature to WCS, Web Coverage Service, which serves the same purpose as WFS, but for raster data rather than vector data. GeoServer has also been tied closely to the open standards movement. The original goal of GeoServer back in 2001 was to be an open source implementation of the WFS specification. It did so and actually became the reference implementation of wfs 1.0. And from there on GeoServer evolved to adopt new standards such as WMS, which was added around 2003. WCS was added in 2005. And currently being worked on is an implementation of WPS (Web Processing Service) which is something geoserver developers are very excited about. As of today GeoServer Is the reference implementation of WFS 1.0 and 1.1, as well as WCS 1.1. And GeoServer is fully cite compliant with the WMS 1.1 and WCS 1.0 specifications. And past open standards is the idea of GeoServer as a gateway to make geo information available to a wider “geoweb”. Supporting other not necessarily OGC defined formats and protocols, in an effort to make geospatial information as widely available as possible.

What is GeoServer? So again to sum up, GeoServer is an engine that reads in spatial data in a wide variety of formats from file based formats like Shapefile, to spatial databases like PostGIS and Oracle, to raster formats like GeoTiff, etc… and pushes that data onto the web via well defined protocols, to a wide variety of clients which consume those services, from desktop clients like udig, gvSig, qgis to web based clients like openlayers, etc…

The Past So in preparation for this presentation I went through release notes in order to come up with a feature timeline for the past year. But then I found myself going back further in time so I decided to keep going back to the very beginning. This diagram shows GeoServer back from the very start. The project was started in 2001, and as you can see the first few releases were quite basic in terms of what GeoServer is today. It started as just a basic WFS with support for a few formats like Shapefile and PostGIS. Full transactional capabilities were added for the 1.0 release, so from 1.0 GeoServer has been a complete WFS implementation. However, the utility of a WFS all by its self was somewhat limited. More and more people realized that they needed a way to visualize data and at the time there was no WFS client to do the visualization. So after getting tired of staring at GML in the browser all day developers added a WMS component to GeoServer whose purpose was not necessarily to be a full WMS, but really just a way to visualize WFS. So am integrated WMS was added for the 1.1 release by Gabriel Roldan working on contract for the Spanish government. 1.2 saw a heap of functionality added by Refractions Research in Victoria, British Columbia Canada, who added the web administration interface and a feature validation engine. The 1.3 branch of GeoServer began to focus more on the WMS component. With the uptake of WMS being much greater than the uptake of WFS, and the continued lack of a good WFS client, it was becoming more and more evident that there was a lot of utility to be had in a general WMS. So the 1.3 release really focused on the aspects of WMS like rendering, and support for SLD allowing users to control styling.

The Past Compared to other branches the 1.4 release was relatively short lived and was really not much of a feature oriented release, but was more centered around work to the core of GeoServer. In 1.4 GeoServer moved to the spring framework and was broken apart into a modular architecture. The 1.5 branch marked the inclusion of the great work from GeoSolutions in Italy to bring raster capabilities to GeoServer. Which was really what has made GeoServer a complete solution for serving spatial data. Support for raster formats came with the addition of the Web Coverage Service as well. The 1.6 branch really began to see a feature explosion. WFS 1.1 support was added and with it the ability to do reprojection with WFS. A security subsystem as added enabling the ability to secure data and services. Much work was done to better the performance of the GeoSever WMS. Much of the 1.6 branch focused on supporting more “geoweb” friendly formats like GeoRSS, and GeoJSON, as well as improving existing KML support adding templates. This feature push continued onto into the early stages of the 1.7 branch, which saw further improvements to the security system enabling finer grained access control at the dataset level, rather than at the service level. Support for a much wider array of raster formats such as ECW and MrSID was added again courtesy of GeoSolutions, who leverage the existing functionality of the GDAL library.

The Present Which brings us to the past year of development. As you can see it has been quite a busy one.

KML There has been a lot of work further continuing to improve KML support. Improvements to super overlay output which allow KML produced by geoserver to be broken up into tiles or regions and allows for the nice streaming affect that you get when you view native layers on google earth. I will be showing an example of this in a few minutes. KML extrudes were added giving the ability to produce KML with extrudes or height information attached giving a 3D effect. This works with the existing GeoServer template system. So if you have a dataset that has some notion of height associated it you can write a simple template file that specifies the attribute and that information will be passed through to google earth. The example I have actually shows extrudes in conjunctions with super overlays. Supporting Googles GeoSearch has been something focused on and partly funded by Google. Much of the work has centered around working with data partners to start pushing some of their datasets out via GeoServer KML and getting that data indexed by the Google Crawler.

REST Past KML, a RESTful configuration api has been added. This allows clients to modify GeoServer configuration programmatically in REST-like, without having to use the user interface or without having to hack configuration files. I say REST-like because I don’t want to aggravate the REST purists in the audience today. While for the most part it does follow REST best practices there are places where it deviates and performs some no nos. One of the clients to note consuming this new API is the “Styler” extension, which is based on the GeoExt library. It allows one to configure styles directly in the browser with an integrated map panel which immediately renders the style changes. I will be showing this shortly.

Rendering There has been much work to improve rendering quality in GeoServer, and really key features when it comes to good cartography. Things like labeling along curves for linear features like roads, and wrapping labels for polygonal features. I hope to show a few of these features off. Also on the rendering front has been the ability to have more control over the resources that get used during a rendering job. Unfortunately there are various ways to really grind GeoServer to a halt with a single request. For example requesting an image that is huge in size and forcing the server to allocate a ton of memory to render it. There are also cases when a connection to a database goes haywire in the middle of a rendering job leaving the job hanging and holding own to server resources. Thankfully now there are some sensible limitations in place that can limit the amount of time a rendering job takes, as well as the amount of memory it can use up. And these settings are configurable so that the server admin can tweak them on a site by site basis.

ArcSDE ArcSDE support is something I am happy to say has gotten a lot better over this past year. Both on the vector side and the raster side of the fence. We see a much wider range of raster support including pretty much all flavors. And scalability across the board has been improved drastically. I will talk a bit more about the specifics in a later slide.

Databases Much work has gone into the database support over the year, utilizing some key work at the geotools level to improve the architecture and the implementation of database datastore drivers. With it we have seen improved datastore drivers for oracle, db2, and postgis, as well as the addition of a new sql server driver. The new architecture has also allowed us to include things like JNDI defined connections, which allow datastores to share global connection pools, or connection pools defined by the application server etc…

Output Formats And as usually there has been a number of new output formats added to GeoServer. An output for WFS based on OGR was developed that allows one to configure Geoserver to point at an OGR installation on the system. This then allows GeoServer to output vector data in any of the formats the OGR installation is configured for. Also contributed as a WFS output format is the ability to CSV or directly as an Excel spreadsheet. On the WMS side we saw the addition of the HTML ImageMap format, currently available as a GeoServer extension. When activates data is output as an HTML image map, and allows one to easily attach events to various features or geometries in a dataset. For example it would allow one to implement google maps like popups quite easily with the imagemap output and a bit of javascript.

Extensions And there have been various other interesting extensions over the year. The charting extension allows one to dynamically render charts and diagrams directly onto a map using a google charts like api facility. The JDBC mosaic extension allows one to store tiles and pyramids for large raster files in a database. The vector pyramid extension allows one to set up datasets with multiple level of generalization in a pyramid with appropriate levels of generalization attached to appropriate scales. This can drastically improve the rendering performance of vector data. And more… I apologize to those whose contributions I have overlooked… too many to keep track of.

2.0

The Year in Review 489 bug fixes 1682 commits 7 releases So looking back at a year in Review. About 1100 bugs fixed, 1700 commits made, and 7 major releases. I should also point out that this is just three quarters of the year and missing the months of October, November and December. When I gathered these numbers I thought it would be interesting to see how these stacked up against previous years of the project. So for fun I decided to gather some stats from previous years as well. * Without October, November, December

Bug Fixes The next stat I gathered was number of bugs fixed per year going back to 2004. No real surprises here but I am not sure how to interpret the graph. Either we are being more productive and closing more issues as time goes by or just introducing more bugs as time goes by. Maybe a bit of both.

Commits Now this one is a bit more interesting. It’s the number of commits made into version control. The big spike in 2004 is the work done by Refractions for the 1.2 release adding the new user interface and the validation engine. I actually kind of surprised when I analyzed the number of commits made. Equally as interesting is the sudden drop off in 2005. This one actually baffles me umm… because their should be more commits logged. My guess is that some of the logs were lost during the changover in subversion repositories to the codehaus repository we use now. But from there on we see a general upward trend yet still not able to achieve the huge numbers of 2004.

Downloads And the rest of the graphs are pretty straight forward. Here we see downloads over time. I should also mention that the numbers gathered in these stats are not exactly accurate. For instance these numbers come from the sourceforge download stats. I suspect the tracker is a bit on the generous side when it comes to the numbers but I could be wrong. That said the interesting part of the analysis are the relative differences between subsequent years.

Mailing List Traffic And finally one last stat is traffic on the users mailing list. Thankfully we see this going upward as well which means the user base is broadening. That or the users we have are becoming vocal.

GeoServer International English – 987 members Brazilian – 300 members Chinese – 282 members Italian – 54 members Spanish – 53 members While on the topic of mailing lists one very exiting occurrence of this year has been a burst in the wider international GeoServer community. The Chinese, Italian and Spanish mailing lists are all new this year, and continuing to grow. The Brazlian geoserver users list reached a milestone of 300 members this month. And the original English list continues to grow as well.

… While on the topic of mailing lists one very exiting occurrence of this year has been a burst in the wider international GeoServer community. The Chinese, Italian and Spanish mailing lists are all new this year, and continuing to grow. The Brazlian geoserver users list reached a milestone of 300 members this month. And the original English list continues to grow as well.

2.0!

New UI Ajax Submit-Apply-Save Extensible So with a brief overview of the year out of the way I want to go into a bit more detail into some of the major features developed over the year. Starting with the major focus of GeoServer 2.0, the new web user interface. Basically the web administration console has been totally rewritten from scratch with a newer web development framework called Wicket. With Wicket one can build rich ajaxified web applications, with out actually having to do Ajax or Javascript. This sort of makes it a very natural fit for GeoServer since most of us old Java developers aren’t as hip to that new web stuff. And with it we have been able to create a new interface that is much more dynamic and convenient than the old version, may it rest in peace. However, while we have rewritten the user interface from scratch we have tried to maintain the same basic workflows and functionality of the old user interface. The first milestone release of 2.0 had the goal to have the new user interface cover all the features of the old user interface. While this perhaps did not allow us to work on exciting new features, it did allow us to create something that would not be totally foreign to users of the old interface. And also to ensure that users switching did not feel like they were losing any functionality by upgrading. However, one thing was intentionally lost with the new design. The Submit – Apply – Save workflow. Anyone who has used the old web interface has probably found it weird and tedious to have to click three buttons for every single change one wants to make. So this has been done away with. Now when you submit a form on a page, that change is persisted, no need to explicitly apply or save. We are hoping that this is something that users wont miss. Time permitting I will show off a bit of the new interface a little later. And one last thing to mention is that the new interface has been developed in an extensible manner. Which means that people wanting to develop extensions for GeoServer can now build a user interface for their extensions in a modular way. This was also one of the driving decisions behind going with Wicket as the web framework, as it allows you do this.

Security UI Users and roles Data security Service security Which has already lead to the first major web extensions, the security user interface which allows you to configure the GeoServer security sub system. Before 2.0 the only way to add users, roles, and set up access control rules was to hack on property files. But now with the security ui this can all be done from the user interface. The extension was contributed by the geoSDI group, working for the Italian Department of Civil Protection. Special thanks goes out to them and Francesco Izzi, who is the project lead for geoSDI, and the maintainer of the extension.

Application Schemas Complex feature schemas Feature chaining

KML Super Overlays Geo Search Extrudes Partial 3D Moving on… once again this year KML has been a primary focus. The main bit of work being around super overlays and regionation. Historically the way GeoServer has served KML has been to send a single kml stream that matches the current camera viewport. And to continually update so that when the camera moves the kml stream updates. The downside of this is that there is continually a split second in which the view is updating. This leads to a bit of an undesirable visualization affect. However, KML supports the notion of regions. A region being more or less a tile, a square section of the screen. By grouping KML into regions allows google earth to dynamically turn certain regions on and off as a region comes into and out of view. Also regions can be grouped into a hierarchy or pyramid the same way tiles are. And the same rule applies, as the camera zooms in regions or tiles lower in the hierarchy get activated and come into view. Now.. The upshot of all of this is the nice streaming affect that occurs as one zooms and pans around and as different regions come into view they get phased in and as regions come out of view they get phased out. A much more visually pleasing affect than the constant updating of the entire kml stream that results in a lot of flicker. We are all used to seeing this with the regular layers in google earth. But now one can achieve this same affect with layers coming from GeoServer. Also of much focus this past year was the GeoSearch. This feature allows GeoServer to publish KML in such a way that it can be crawled and index by Google. The end result being the ability to search for your geospatial data right in google maps. And I will try to show this in action if time permits. KML extrudes allow one to attach height or extrude information to a KML feature. Now GeoServer allows one to specify an attribute or some expression based on an attribute, which will be used as the value of an extrude. This allows 3Dish visualization in google earth. Again I will try to show this off if time permits.

Rendering Labels Dynamic symbolizers Resource limits curved labeling repetition wrapping displacement Dynamic symbolizers hatched fills dynamic glyphs Resource limits Moving past KML but sticking with the visualization theme, much work has been done to improve cartographic output in GeoServer, with various rendering features implemented. And pretty much all of this is courtesy of the hard work of Andrea Aime, one of the lead developers on the project. Improvements to GeoServer’s labeling capabilities were implemented. This work was funded by Trimet, the transport authority for the city of Portland, in Oregon USA. But GeoServer is now able to label along linear features such as roads or streams. This is something that other map servers have been able to do for some time, and something fairly fundamental so it is nice to have this feature finally implemented. Other improvements having to do with the labeling of linear features are the ability to repeat labels on a single curve, and specify the distance between repeating labels. Along with curved labels for linear features is the ability wrap labels for polygonal features, so that labels stay contained within the bounds of a containing building for park for instance. Also implemented this year were the notion of dynamic symbolizers. The idea behind dynamic symbolizers Is to be able to do advanced symbology with SLD, advanced meaning going on what is strictly specified in the SLD standard, but still staying within the bounds of the SLD language itself. With this feature one can do things like hatched fills, or cross hatches along lines, and various other interesting effects. And finally, while not much a visual feature but a very important one nonetheless is the ability to set resource limits on GeoServer rendering jobs. Unfortunately before this feature was present it was quite easy to bring a GeoServer to a grinding halt with a single request. For instance requesting a huge image causing GeoServer to allocate a ton of memory. Or requesting that millions of features be rendered to a small image causing GeoServer to start a very expensive rendering job. And then there is always the problem of misbehaved clients. That is clients that start a rendering job but then close the connection before the job is finished. Unfortunately GeoServer can not know that it should stop rendering so this can lead to a lot of pointless rendering jobs piling up and bogging down the server. So basically what this allows for is the ability to set limiits on such things. For instance setting the maximum amount of memory any single job can have access to. And setting a timeout on a rendering job so that after a certain time the job is canceled instead of continuing on and on consuming resources pointlessly. Things very very important when running GeoServer in a production system.

REST RESTful configuration Styler A feature that has become very popular among users is the RESTful configuration api which allows for the configuration of a GeoServer instance via a RESTful interface. This gives people the ability to configure GEoServer from a variety of scripting and programming environments. And more importantly it allows for the easy automation of configuring a GeoServer instance. Before the hacks involved in a programmatically configuring GeoServer were quite ugly. This feature is very important for organizations that need to batch configure GeoServer with many layers of data, or that need to add new layers to a GeoServer at frequent intervals. For instance think about continually capturing imagery from a satellite and needing to publish that imagery on the fly. With the REST api it is possible to add new Shapefiles, PostGIS tables, GeoTIFF file, and more. It is also possible to add and configure styles with the interface. Which is where the Styler application comes in. Styler is a web application that provides a style editor directly in the browser. It uses the REST api to edit and publish GeoServer styles. So at this point I am going to cut out to a quick demo to showcase the last few features I have discussed.

ArcSDE Raster Arbitrary number of bands and pixel depth Color maps Raster catalogs Ok, so back the year in review, next up is ArcSDE. A whole lot of work has been done around SDE support in GeoServer. First and foremost is SDE raster support. Until recently GeoServer supported a very limited flavor of SDE rasters. Basically of the top of my head I think 3 and 4 band 8 bit unsigned without any color map support. But now Geoserver should support basically all pixel depths signed and unsigned, with color maps. And as well raster catalogs of any type. This work was funded by MassGIS, the office of Geographical and Environmental information for the state of massachussets. MassGIS is one of the biggest and earliest users of GEoServer in production. Most of the development on sde support in GEoServer has been carried out directly or indirectly by MassGIS. So special thanks to them and also special thanks to Gabriel Roldan for actually carying out the raster work.

ArcSDE Miscellaneous Scalability with sessions Multi-version tables Geometryless JNDI However massgis is not the only ones to thank. A number of ArcSDe features were implemented this year on behalf of, and now I know I am going to butcher this name, but Rijkswaterstaat (RWS). The Ministry of Transport, Public Work, and Water Management in the Netherlands. They funded support for multi version tables, and basically publishing a snapshot of a version table. Also support for tables without a geometric value. And finally support for connecting to an SDE instance via a JNDI defined connection. So special thanks to Thijs van Menen who is the Coordinate GeoServices for setting that funding up. And also to Bart van den Eijnden for helping with the implementation and testing.

Raster Improvements GDAL support Mosaics Coverage rendering more formats robust bindings Mosaics automated index creation transparency Coverage rendering which formats? * mosaics: - making missing tiles transparent - specifying pixel values for transparency * coverage rendering?

Databases “NG” architecture Improved PostGIS and Oracle SQL Server security performance Improved PostGIS and Oracle SQL Server JNDI connections

Extensions Tiles and pyramids in the database Vector pyramids Charts HTML image maps

On the Horizon Database configuration storage Multi-dimensional coverages Service profiles Web Processing Service Scripting

Thanks! http://geoserver.org Questions?