LEAD Data Catalog

Introduction

Data catalog supports registration of data collections available as THREDDS catalogs or provided as LEAD Metadata documents conforming to the XMLschema. Once registered, the THREDDS catalogs are crawled using a metadata crosswalk to convert them to the LEAD Metadata schema(LMS), and indexed in the Lucene index. LEAD Metadata documents sumbitted in LEAD metadata schema are directly indexed without requiring crawling.

The THREDDS data server is a middleware often used by the earth sciences community as a repository for data from various sensors and instruments. Data, coming in periodically, is organized hierarchically as files and directories according to predefined configurations, and an XML document describing the contents of each directory is created and updated based on a manually defined template for each directory. These XML documents are called THREDDS catalogs and describe spatio-temporal attributes, data format and access protocols, publisher, and so on. These catalogs are accessed from the TDS through a HTTP URL. The hierarchical structure of the directories means that the catalogs themselves can point to child catalogs in addition to listing data products. These catalogs form one of the main sources of public data for the Data Catalog.

The catalog can be searched for using the web-service API or the commandline client. Geo-Gui, a graphical portlet interface for querying the default LEAD data catalog, is also available here.

Data products can also be published from the myLEAD personal catalog to be shared among the LEAD community. These data products are already described using LMS and can be directly added to the Data Catalog .

Architecture

New Release

• Data Catalog v1.5 (2008-09-14) [Upgraded to new version of UniData crosswalk(v0.7), Upgraded Lucene library(v2.3.1), Reduced crawling time from 30minutes to 5minutes, etc.]

Old Releases

• Data Catalog v1.4 (2007-06-29) [Added support for starting service using properties configuration file in addition to command-line params, etc.]
• Data Catalog v1.3 (2007-04-13) [Upgraded to new version of UniData crosswalk(v0.6) that supports theme keys for DatasetTypes, etc.]
• Data Catalog v1.2 (2007-04-13) [ Moved to new version of crosswalk that has the DatasetTypes.lead.org theme key, etc.]
• Data Catalog v1.1.7 (2007-03-28) [Added security to the service, Chaged dataetime formatting in Lucene index, etc.]
• Data Catalog v1.0 (2007-01-02) [Added support for multi threaded crawlers (THREDDS -> LMS) & metadata processors (LMS -> Lucene Document), Added recovery mechanism for unresponsive crawling, etc.]

Documentation

• Data Catalog manual v1.0 with description of architecture, functionality, and service API, WSDL definition & XSD Schema, and LMS attributes used in Data Catalog search queries.