Page tree
Contents

Harvester Settings

These settings control all aspects of harvesting metadata from your data source.

URI

The 'URI' is the  location of the Data Source, from which XML documents are harvested by ANDS. Also known as the harvest point.

Harvest Method

The 'Harvest Method' specifies the means by which your metadata will be retrieved from the harvest point 'URI'.

  • GET Harvester: harvester connects directly to the Data Source URI using a HTTP Get and downloads XML in format specified in the Provider Type.
  • OAI-PMH Harvester: harvester connects to your OAI-PMH feed using the OAI-PMH protocol and downloads in format specified in the Provider Type. Includes the support for resumption tokens in the case of large harvests, and error and exception reporting.
  • CSW Harvester: (Custom method) harvester connects to a Catalog Service for the Web (CSW) instance and downloads XML in format specified in the Provider Type.
  • CKAN Harvester: (Custom method) harvester connects to CKAN instance and downloads JSON in format specified in the Provider Type.

Please note that custom harvest methods require additional configuration to take place before they can be used. Additional harvest methods can also be supported. For more information please contact services@ands.org.au

OAI-PMH  Harvester

Sample OAI-PMH harvester settings

Troubleshooting: Make sure you don't accidentally specify leading or trailing spaces when you enter your harvest URI-this is one of the causes of harvests failing. 

OAI Set

The 'OAI Set' field is available with a 'Harvest Method' of OAI-PMH. It allows you to instruct the ANDS harvester to retrieve records from a specific set in your OAI-PMH feed. This is especially useful when you only wish ANDS to harvest a subset of the records available through your OAI-PMH instance.

Metadata Prefix

The 'Metadata Prefix' drop down is available with a 'Harvest Method' of OAI-PMH. The drop down allows you to tell the harvester what metadata prefix should be requested from your OAI-PMH service. ‘rif’ (i.e. RIF-CS) is the default value. Additional prefixes can be added to your data source by using the ‘Add Crosswalk’ button, however if the format of the data is not RIF-CS a crosswalk will need to be uploaded.

To specify a crosswalk and add supporting files, please see Add Crosswalk and Supporting File section below.

CSW Harvester


Sample CSW harvester settings  

Harvest Params

The 'Harvest Params' fields are only displayed where a 'Harvest Method' of 'CSW Harvester' is selected. The fields are displayed as Name-Value pairs, and allow you to customise the parameters passed in a CSW harvest request.

By default the system provides the minimum set of CSW parameters required. You may change the value for these default parameters but should not delete them. If removed the system will automatically reinsert them with the default value upon save. To add additional parameters to the request, simply click the 'Add Parameters' button displayed at the bottom of the table and enter the parameter name and value in the added fields.

Output Schema

The 'Output Schema' drop down is available with a 'Harvest Method' of CSW. The drop down allows you to specify the ‘outputSchema’ parameter which is required by CSW service requests. The parameter specifies the return format for the CSW request and must be set to the URI of the output metadata schema (e.g. http://www.isotc211.org/2005/gmd).

To specify a crosswalk and add supporting files, please see Add Crosswalk and Supporting File section below.

GET Harvester or CKAN Harvester

Sample CKAN harvester settings  

Provider Type

The 'Provider Type' drop down is available with Harvest Methods GET and CKAN. The Provider Type value describes the kind of metadata document you are making available for harvest and is used to identify a matching crosswalk in your Data Source. ‘rif’ (i.e. RIF-CS) is the default value and the type expected by the registry for ingest. Other provider types (e.g. Dublin Core, ISO19139, CSV ,etc) can configured for your data source by adding a crosswalk from the type to RIF-CS.

To specify a crosswalk and add supporting files, please see Add Crosswalk and Supporting File section below.

Add Crosswalk & Supporting File

  •   The  'Add Crosswalk'  button allows users to upload an XLST crosswalk and configure it to be run during a harvest. Multiple crosswalks can be uploaded but each needs to be uniquely identified by specifying a Crosswalk Provider Type, Output Format or Metadata Prefix. Once uploaded and identified the crosswalk will be available for selection from the Provider Type, Output Format or Metadata Prefix drop down (dependent on harvest configuration). When selected, a green 'active' label will display next to the crosswalk.
  • The 'Add Supporting File' button can be used to upload a supporting file which can be referenced and used from within an XSLT crosswalk. Supporting files must be in a format XML or XSL

Visit the Crosswalks: Transform your metadata page on the ANDS website for the list of available crosswalks to RIF-CS (e.g. ISO19115, CKAN, Dublin Core, etc.)

Advanced Harvest Mode

One of three harvest modes can be selected for ingesting records, these modes are as follows:

  • Standard Mode: Standard mode will ingest all records from the data source feed (Standard is the default mode)
  • Incremental Mode: The Incremental Mode will use OAI-PMH to support harvesting on a "from and until" basis. This means that only records that have been created or modified "from" the last harvest date, "until" the date of the new harvest will be included within the Record providers existing dataset.



    The above example shows that after ingest record 8 & 9 will be included into the data source, while the original record 4 will be replaced with the newly ingested record 4. 

    This mode will only function if the harvest method is set to "OAI-PMH Harvester", and if the OAI-PMH provider correctly supports the "from and until" parameters as per the OAI-PMH Specification.

  • Full Refresh Mode: The Full Refresh Mode will remove all previously harvested records and update the dataset with the latest ingested records. Records created manually using the Add Registry Object screens will not be removed when using this harvest mode.



    The above example shows that records 1, 2, 5 & 6 are removed, new records 8 & 9 are placed in the data source and record 4 is replaced. Also record 3 was not removed as it was manually created using the Registry Add Registry Object screen.

Harvest Date

Use the 'Harvest Date' field to specify a future date and time when a harvest will occur or when regular reoccurring harvests begin. The calendar widget allows nomination of a time zone.

Harvest Frequency

Use the 'Harvest Frequency' drop down to specify the frequency for your harvest.

 

More information