XML Metadata Concept Catalog
Client Jars, WSDL, and Schemas:
Current Version: Version 1.2.6
Prior Version:
Version 1.2.4 July 24, 2009
Version 1.2.0 January 20, 2009
Version 1.0.15 July 17, 2008
Version 1.0.10 June 6, 2008
Version 1.0.8 May 14, 2008
Version 1.0.7 April 20, 2008
Version 1.0.4 April 15, 2008
Version 1.0.3 April 07, 2008
Version 1.0.1 March 13, 2008
Version 0.0.1 February 11, 2008
Related Topics:
Metadata Post-Processing
Workflow Metadata Schema
LEAD-Specific Related Topics:
Metadata Concept Definitions Used in LEAD
Graphical View of the LEAD Metadata Schema (LMS)
Adding Metadata for Data Products in LEAD
The LEAD Metadata Schema, an XML Bean implementation,
and documentation is available at the following site:
http://www.extreme.indiana.edu/rescat/metadata/index.htm
A graphic depiction of the schema is also available as a PDF
|
|
Status Summary:
The returned status codes for errors are standardized at the top level of the status returned. The
status can contain nested errors with detailed explanations, detailed error status messages, and stack dumps.
For each of the operation types, the following table shows the top-level status message when there is
a fatal error or a partial error. There are three cases that are standard across all operations:
No error - returns a status of: OPERATION_SUCCESSFUL.
The XML of the request is not schema-valid - returns a status of: NOT_SCHEMA_VALID.
An unhandled exception such as thrown by Axis2 is caught - returns a status of: UNHANDLED_EXCEPTION.
| Operation | Fatal Error | Partial Error |
| Status | FATAL_STATUS_ERROR | NON_FATAL_STATUS_ERROR |
| Create User | FATAL_CREATE_ERROR | NON_FATAL_CREATE_ERROR |
| Delete User | FATAL_DELETE_ERROR | NON_FATAL_DELETE_ERROR |
| User Exists | FATAL_QUERY_ERROR | NON_FATAL_QUERY_ERROR |
| Query Whiteboard | FATAL_QUERY_ERROR | NON_FATAL_QUERY_ERROR |
| Create Object | FATAL_CREATE_ERROR | NON_FATAL_CREATE_ERROR |
| Delete Object | FATAL_DELETE_ERROR | NON_FATAL_DELETE_ERROR |
| Move Object | FATAL_UPDATE_ERROR | OPERATION_SUCCESSFUL (1) |
| Object Exists | FATAL_QUERY_ERROR | NON_FATAL_QUERY_ERROR |
| Add Property | FATAL_UPDATE_ERROR | NON_FATAL_UPDATE_ERROR |
| Update Property | FATAL_UPDATE_ERROR | NON_FATAL_UPDATE_ERROR |
| Delete Property | FATAL_DELETE_ERROR | NON_FATAL_DELETE_ERROR |
| Context Query | FATAL_QUERY_ERROR | NON_FATAL_QUERY_ERROR |
| Query By ID | FATAL_QUERY_ERROR | NON_FATAL_QUERY_ERROR |
| Workspace Query | FATAL_QUERY_ERROR | NON_FATAL_QUERY_ERROR |
| Property Def Query | FATAL_QUERY_ERROR | NON_FATAL_QUERY_ERROR |
| | |
| (1) | The only non-fatal error that can occur in the move object operation is after the changes
have been successfully committed, so the non-fatal error is a successful operation. |
| Note: | The "Update User" and "Query User" methods have not been programmed yet because of
anticipated changes to make the additional metadata about users more flexible. However,
the error status settings for these methods will be the same as the other update and query
methods respectively. |
LEAD Vocabulary:
The LEAD Metadata Schema (LMS) contains elements for tracking keywords used to describe the data products.
Although the schema allows for four types of keywords: theme, place, stratum, and temporal, mostly the metadata
contains theme keywords. For data products available in the LEAD portal from Unidata, the CF-1.0 vocabulary has been
used to describe the data products. These are included in the metadata as theme keywords.
Each keyword has a thesaurus element and one or more keywords. The thesaurus element identifies the source of
the keyword's definition (such as CF-1.0 for the keywords from that vocabulary). In addition to the CF-1.0
vocabulary, we have defined additional LEAD-specific vocabularies. Following are the thesaurus element values we use in LEAD:
| CF-1.0 | These are terms from the CF-1.0 controlled vocabulary. |
| leadproject.org | This is used to identify the cross-walk version for files from UNIDATA
and to identify the version of the LEAD minimal metadata utility used. This also is used with the value "LEAD"
to provide a required default theme keyword. |
| DatasetTypes.lead.org | When data files are downloaded from the Data Catalog in the
LEAD Portal, this identifies the type of data file (e.g., NAM versus ADAS). |
| NAMELIST | This is used for namelist files to identify the type and version of the namelist
(e.g., WRF versus Terrain). |
| VisTools.lead.org | This is used to identify viewable files.
The only current theme key value for this is IDV. |
Examples:
Following are some small test programs that provide an example of using the catalog service. In the
programs for adding objects (e.g., experiments or files), as well as the program for adding properties
to an existing object, the XML for the LEAD Schema valid metadata is provided as a file. This is because
for test purposes, we need to populate an XML bean. in an actual program using the catalog, other code would
have already generated the XML bean with the LEAD metadata. All of these tests are also included in
this jar file. All of the programs are in the package
edu.indiana.dde.catalog.catalogtester. For each program, if run from the command line, including only
-help as a parameter will show the usage.
These tests all require the client jar (catalog-client-1.0.0.jar) listed above and all of the jars in the
Axis2 lib folder. You will also need to swap out the xbean-2.2.0.jar file from that lib folder and replace
it with the xbean.jar file from the current XML Beans 2.3.0 installation.
All of the necessary Axis2 jars are in
this tarball.
The additional LEAD schema and catalog files are in
this tarball.
- TesterBase - This is the abstract base file all of the test programs now extend. This was added
so the adding of the user's DN as a header could be reused for all of the tests.
- TestCheckStatus - checkStatus operation which provides some basic stats about the service.
- TestObjectExists - objectExists operation which returns a boolean as to whether the global ID passed
as a parameter represents a valid object for the user making the request.
- TestUserExists - doesUserExist operation which returns a boolean as to whether the DN passed as a parameter
represents a valid user.
- TestWhiteboardId - whiteboardId operation that returns the global ID of the whiteboard project for
the user who's DN is passed as a parameter.
- TestObjectIdQuery - objectIdQuery operation. Run with -help to see the options for this query based
on the object's global ID.
- TestContextQuery - query operation. Run with -help to see the options for this query. The target
for the query is passed to this example program as a file containing a queryTarget element from the types schema.
This sample target file can be used as an example.
Another example of a context query is "get me all files where the file name (title) starts with "wrfout" and the experiment
the file belongs to starts within certain temporal bounds and the spatial bounding box of the experiment intersects certain
spatial bounds". These are the target
and context files for this example. As in the other examples, the query
would generally be built using the XML Bean for the types schema and not loaded from a file.
- TestCreateObject - createObject operation. Run with -help to see the options for this insert. The
LMS-valid document is passed as a file. If you need data, run the query by ID
and use the metadata returned (using the FULL_SCHEMA content filter). A new substitute
GUID can be passed as a parameter - just make sure it is unique.
- TestAddApplicationMetadata - createObject operation but also takes parameters for adding
application metadata. This sample contains a static method named addApplication which along with some static variables declared at the
class level can be dropped into java code to add application metadata to a LEADresourceDocument instance (the document XML Bean for an
instance of the LMS schema). The parameters are similar to the TestCreateObject.java sample. Run with -help to see the options available.
- TestAddProperty - addProperty operation. Adds metadata to an existing object. Run with -help to see the
options for this operation. This test program takes text files with the XML for the
properties to be added. Following are some samples for theme keywords:
1
2
3
4
5 .
- TestAllWorkspaceQuery - allWorkspaceQuery operation. This query is used to get all of a
user's workspace. By default it gets only global ID and the title within the citation property for each object. Different content filters
can be applied and the results can be returned either directly in the response or in streamed.
- TestCreateUser - createUser operation. This defines a new user in the catalog and creates
a default project for the user. This test program takes a file that contains the XML defining a new user. This is the catalogUser element
from the types schema. This example user file can be used for the file parameter. So that the
file can be reused without editing, the test program takes additional parameters for a new user DN and/or user name that should be substituted
for the user in the file. Each user must have a unique DN.
- TestPropertyDefQuery - queryPropertyDefinitions operation. This query returns all of the
property definitions currently in the catalog in their catalog catagories and including all sub-properties and metadata elements. This query
is used to dynamically build the query interface so it is automatically tailored to the domain the catalog is serving. Results can be returned
directly in the response or streamed back.
- TestUpdateProperty - updateProperty operation. This operation is used to update an existing operation.
This test program takes a file parameter that includes the XML passed as a parameter to the operation for the property being updated (the
catalogPropertyUpdate element in the domain properties schema). This example theme update file can be
used to update an existing theme property in a user's workspace.
- TestDeleteProperty - deleteProperty operation. This operation is used to remove a metadata
property from an existing object's metadata. Removing the property must not result in a schema-invalid document - for example a required
schema element with cardinality of one. If an element has cardinality greater than 1, the position of the instance to be deleted must also
be specified (including possibly multiple positions). As with the update property operation, the position specified starts at 1 (as is normally
the case outside the world of computer science). If deleting multiple instances of a property, a range can also be specified (inclusive of the
first and last positions in the range).
- TestDeleteObject - deleteObject operation. This operation is used to remove all of the metadata for
an object (e.g., a file or experiment). If the object being removed has any child objects (such as collections or files within an experiment) then
those child objects are also deleted.
- TestDeleteUser - deleteUser operation. This operation is used to remove the definition of a user and all
of the objects in their workspace.
- TestMoveObject - moveObject operation. This operation is used to move an object (e.g., a file,
experiment, etc.) from one part of the user's workspace to another. For example, if the user wanted to aggregate a set of experiments under a
single project, or alternately wanted to move a file they had imported into their workspace (say from the LEAD GEO-GUI) into an experiment
or collection. The moveObject operation takes the global ID's of the object to be moved and the new parent object it should be moved under.
If the object being moved contains a hierarchy (such as an experiment containing collections and files) the hierarchy moves with the
object being moved. Circular ownership (e.g., moving a collection into a collection it owns) is prevented. Likewise, an object cannot be moved
under a lower level in the hierarchy (e.g., an experiment cannot be made a child of a collection since experiments are a higher level in the
LEAD hierarchy - different domains can specify different hierachies).
NOTE: Examples are designed to work with the current version and may not work with earlier versions.
Metadata Catalog Types Jar Version 1.2.6 November 12, 2009
XMC Cat Schemas and Jars
xmccat_types_metadata.xsd
xmccat_status.xsd
xmccat-types-metadata-1.2.6.jar
Client Jars:
xmccat-client-1.2.6.jar
xmccat-client-xbeans-packaged-1.2.6.jar
LEAD-Specific Jar
lead-xmccat-domain-metadata-1.2.1.jar
Changes in This Version:
Changes in this version are mainly to put XMC Cat into the OGCE incubator, but there was a change in the
query result format (used for all XMC Cat queries) that adds two additional optional boolean elements:
1) showCreateDate
2) showModifyDate
If these elements are set to true, then the query response for each entity will contain attributes
named creationDate and modificationDate respectively with the date value as a long (number of millisenconds
similar to a Java Date or Calendar instance).
Metadata Catalog Types Jar Version 1.2.4 July 24, 2009
XMC Cat Schemas and Jars Javadocs:
online
tarball
xmccat_types_metadata.xsd
xmccat_status.xsd
xmccat-types-metadata-1.2.4.jar
LEAD-Specific Domain Schema and Jar
The domain-specific schema jar for LEAD has not changed since version 1.2.0 (
available here), but version 1.8.1 of the
LEAD schema XML Bean jar should be used (
available here).
Client Jars:
xmccat-client-1.2.4.jar
xmccat-client-xbeans-packaged-1.2.4.jar
Changes in This Version:
Most of the changes in this version were on the server side and the existing version 1.2.0 client jars would continue to work
except for the two new operations (that were not in version 1.2.0). The two new operations are:
- LastUpdate - this operation takes an object's global ID (which in LEAD is the resourceID) as a parameter and returns a long with
a timestamp (in UTC) of when the last change was made to that object or and child object in the hierarchy under that object. For
example, if the ID of an experiment is passed as the parameter, and there was a file recently updated within a collection in that
experiment, the timestamp for that change (expressed in UTC) would be the timestamp returned.
If the ID passed as a parameter is not valid (or not valid for that user) then the current time is returned and a non-fatal error is returned
with an explanation. This same appraoch is used for any error that is encountered since if an error occurs, XMC Cat does not know when
the last change was for the specified object, and the current time is the most conservative answer (so users will not rely on out-of-date
metadata).
- AssignObject - this operation was added for a project in a different domain and is the first step in allowing an object to be assigned as
a child of multiple parents.
Metadata Catalog Types Jar Version 1.2.0 January 20, 2009
XMC Cat Schemas and Jars Javadocs:
online
tarball
xmccat_types_metadata.xsd
xmccat_status.xsd
xmccat-types-metadata-1.2.0.jar
LEAD-Specific Domain Schema and Jar Javadocs:
online
tarball
lead-xmccat-domain-metadata.xsd
lead-xmccat-domain-metadata-1.2.0.jar
Client Jars:
xmccat-client-1.2.0.jar
xmccat-client-xbeans-packaged-1.2.0.jar
Changes in This Version:
Executive Summary: If you are using the client API, replace the types jar (mylead-catalog-types-1.0.15.jar)
with the types and domain-specific jars (
xmccat-types-metadata-1.2.0.jar
and
lead-xmccat-domain-metadata-1.2.0.jar). Then, if you
use the * to import all of the classes in the types jar, just add the following import for the domain-specific jar:
import edu.indiana.dde.metadata.catalog.domain.*;
If instead you import specific classes, look at the list in the 4th bullet point below and change the package name of the
import to the extent you use those classes
Use the new client jars listed above and you are all done.
More Details About These Changes:
- The build process has been moved from ANT to Maven2.
- Most of the changes were on the server side and relate to making XMC Cat adaptable to
different scientific domains (different schema), additional error handling, query processing
and allowing users to store query definitions from the GUI search interface. These changes
do not impact code that uses the client API.
- The schema used in the WSDL for XMC Cat is composed of three separate schemas:
- catalog_types (revised name is xmccat_types_metadata)
- catalog_status (revised name is xmccat_status)
- domain_properties (revised name is xmccat_doamin_metadata)
For the LEAD Metadata Schema, the implementation of the xmccat_doamin_metadata schema is named lead_xmccat_doamin_metadata.
In prior versions of XMC Cat, both the catalog_types and domain_properties had the same target namespace:
http://www.cs.indiana.edu/dde/namespaces/2008/02/catalog/types and were compiled into the same XML Beans jar.
In this version, the domain_properties schema has been renamed xmccat_domain_metadata and has its own target
namespace: http://www.cs.indiana.edu/dde/namespaces/2008/02/catalog/domain and is compiled into a separate
XML Bean. This allows us to switch domains by replacing the LEAD domain-specific jar with an XML Bean
based on a different metadata schema, but we do not need to replace the types jar.
- When XML Beans are compiled for the XMC Cat schemas, prior version used the same package name for all three
of the XMC Cat schemas (edu.indiana.dde.metadata.catalog.types). The domain-specific portion of the
schema is now in a separate namespace and jar using the package name edu.indiana.dde.metadata.catalog.domain.
In programs using this jar, if the import previously included all classes
(import edu.indiana.dde.metadata.catalog.types.*) then adding an import for the new domains package is all
that is needed. If individual classes were imported from the types jar, then imports for the following classes
(to the extent used in your code) need to change from the types to the domain package:
- CatalogDataType (catalogDataType complexType)
- ObjectResponseDocument (objectResponse element)
- ObjectResponseType (objectResponseType complexType)
- PropertyResponseDocument (propertyResponse element)
- PropertyResponsType (propertyResponseType complexType)
- CatalogAggregationType and the enum inner class (catalogAggregationType element)
- CatalogPropertyAddDocument (catalogPropertyAdd element)
- CatalogPropertyAddType (catalogPropertyAddType complexType)
- CatalogPropertyUpdateDocument (catalogPropertyUpdate element)
- CatalogPropertyUpdateType (catalogPropertyUpdateType complexType)
In addition, the following two groups were also moved to the domain namespace, but these are not
used directly by clients:
- catalogObjectGroup
- propertyFilterGroup
- When the xmccat_domain_metadata schema is compiled with XML Beans, the XML Bean jar for the domain-specific
metadata schema is included as a dependency (lead-metadata-1.8.jar for the LEAD Metadata Schema), so the
location is not needed in the import element in the schema and was removed. This change has no effect on the
code of clients using XMC Cat.
Metadata Catalog Types Jar Version 1.0.15 July 17, 2008 ** Beta - Development **
mylead-catalog-types-1.0.15.jar
catalog_types.xsd
catalog_status.xsd
catalog-client-1.0.15.jar
catalog-client-XBeans-packaged-1.0.15.jar
Please Note: This change was only implemented on the development server as of 2008-07-17. It will be added to the
production server after the weather camp completes.
Changes in This Version:
- catalog_status was changed to fix a problem with the stack trace messages not appearing in the operation status returned
when an exception occurs.
- catalog_types.xsd was changed to add an optional boolean element named showHasChildren to the query result format type. When
this element is set to true, the query response will contain an optional attribute in the tag for each object above the
File level that indicates whether the object has any children (e.g., whether a collection contains any files). This
attribute is named hasChildren and will contain either true or false. This addition was needed for lazy loading of the
workspace when a user's workspace contains many experiments and files. Since the element in the query request is optional
and the attribute is optional in the response, there should be no changes required for existing code using the types jar unless
you need this new attribute in the query response.
- The catalog client and associated XBeans-packaged jar are needed to use the new features in version 1.0.15.
Metadata Catalog Types Jar Version 1.0.10 June 06, 2008
mylead-catalog-types-1.0.10.jar
catalog-client-1.0.3.jar
catalog-client-XBeans-packaged-1.0.3.jar
Metadata Catalog Types Jar Version 1.0.8 May 14, 2008 ** Beta **
mylead-catalog-types-1.0.8.jar
Javadocs:
online
download
catalog-client-1.0.0.jar
XBeans-packaged.jar
catalog.wsdl
catalog_types.xsd
catalog_status.xsd
domain_properties.xsd
Changes in This Version:
- The checkStatus operation was updated to reflect the changes in the database connection pooling in version 1.0.7. The
response for this operation is now more flexible in that it contains an unbounded set of catalogStat elements which are
text elements that have a name attribute which is the name of the statistic regarding the XMC Cat and contain a simple
text value. The most important statistics at this time are:
- Insert_Connection_Time_Success - this indicates the number of milliseconds needed to acquire a database connection
for an insert operation.
- Insert_Connection_Time_Failure - if a database connection could not be acquired from the pool of connections for
insert operations, then this is the number of milliseconds until the request failed.
- Query_Connection_Time_Success - the number of milliseconds needed to acquire a connection for a query operation.
- Query_Connection_Time_Failure - if a database connection could not be acquired for a query operation.
A status response will only have a success or failure stat for each type of connection.
- In the response detail for errors, the field for a java exception and the associated stack trace now have an XML type of anyType
instead of string. This was changed because an error could contain characters that are not valid in an XML string.
Metadata Catalog Types Jar Version 1.0.7 April 20, 2008 ** Beta **
mylead-catalog-types-1.0.7.jar
Javadocs:
download
catalog-client-1.0.0.jar
XBeans-packaged.jar
catalog.wsdl
catalog_types.xsd
catalog_status.xsd
domain_properties.xsd
Changes in This Version:
- The user's DN is now passed in the header of the request, so it has been removed as an element in the requests
- Elements in the property definition that needed to be optional were fixed
- The domain_properties schema was cleaned up for elements no longer used
- Additional status types were added in the catalog_status schema
- Program changes and fixes in the catalog service for the change in handling
the user's DN, fixes in the query responses for both direct and streaming,
and fixes related to testing on the create user, update property, workspace query, and property definition
query operations
- New examples for create user, update property, workspace query, and property definition query operations
- Updated versions of existing examples for changes in version 1.0.7 - particularly the handling
of the user's DN
Metadata Catalog Types Jar Version 1.0.4 April 15, 2008 ** Beta **
mylead-catalog-types-1.0.4.jar
Javadocs:
download
catalog-client-1.0.0.jar
XBeans-packaged.jar
catalog.wsdl
catalog_types.xsd
catalog_status.xsd
domain_properties.xsd
Changes in This version:
Global element declarations were added for three globally declared types. Whithin the schema
elements based on these types are by reference (ref) to the global elements instead of the types.
- catalogProperty using catalogPropertyType
- updateProperty using updatePropertyType
- deleteProperty using deletePropertyType
Metadata Catalog Types Jar Version 1.0.3 April 07, 2008 ** Beta **
mylead-catalog-types-1.0.3.jar
Javadocs:
download
Changes in This Version:
- The client has been recompiled to use the types jar instead of using the namespace of the schema. When
using the client jar, the types jar must also be on the class path.
- In catalog_types.xsd, metadataDefinition was made nillable in case there is an error and
the definitions are not populated.
- In catalog.wsdl, the operations, port types, and messages for the query attribute (Attr) definition
was changed to query property definitions.
- The returned status codes for errors were standardized at the top level of the status returned. The
status can contain nested errors with detailed explanations, detailed error status messages, and stack dumps.
For each of the operation types, the following table shows the top-level status message when there is
a fatal error or a partial error. If there is no error, the returned status is: OPERATION_SUCCESSFUL.
Metadata Catalog Types Jar Version 1.0.1 March 13, 2008 ** Alpha **
mylead-catalog-types-1.0.1.jar
Javadocs: online - see version 1.0.3
download
catalog.wsdl
catalog_types.xsd
catalog_status.xsd
domain_properties.xsd
Changes from version 1.0 to version 1.0.1:
The package name used for the java classes from the catalog schema namespace:
http://www.cs.indiana.edu/dde/namespaces/2008/02/catalog/types
now use the package name:
edu.indiana.dde.metadata.catalog.types
This prevents the need for updating all of the imports when there is a change in the schema jar.
Changes:
In the prior version, the WSDL consisted of 3 files (plus the LEAD Metadata Schema). In this version
the schema types (types section of the catalog.wsdl) is broken out as a separate schema and the
mylead-catalog-types-1.0.jar is compiled from that schema as well as the catalog_status.xsd and
domain_properties.xsd that are included in the catalog_types.xsd.
There are also some minor fixes to the schema.
A new version of the client and examples that use this types jar will be posted soon.
Metadata Catalog WSDL and Client Jars Version 0.0.1 February 11, 2008 ** Pre-Alpha **
Catalog-client-0.0.1.jar
Catalog-XBeans-packaged-0.0.1.jar
catalog.wsdl
catalog_status.xsd
domain_properties.xsd
** THIS VERSION IS PRE-ALPHA **
Requires:
- lead-metadata-1.8.jar (production)
- XML Beans: jsr173_1.0_api.jar
- XML Beans: xbean.jar
- AXIS2: All of the files in the axis2-1.2/lib directory of the Axis2 installation
- JDK 1.5
Description:
The WSDL consists of three files: catalog.wsdl, catalog_status.xsd, and domain_properties.xsd. The WSDL file defines the
operations the catalog will perform. The catalog_status is the schema for the portion of the response that includes the
status of the operation. the domain_properties.xsd schema file is the domain specific (LEAD) part of the schema. This is
separate so later it can be swapped out for other domains without changing the other two files.
Expected Further Changes:
The catalog_status.xsd will change as far as some of the catalogStatusType enums that indicate the status returned - these are
somewhat similar to the ReturnType in myLEAD but more extensive. The domain_properties.xsd will change as the list of properties
(formerly attributes) that can be deleted is added to the schema.
Documentation
The link below is to the javadocs for the Client jar and the types used in the Catalog methods.
Client Javadocs
WSDL Types Javadocs
Example Program
Coming Soon ....