org.apache.oodt.cas.crawl

Class AutoDetectProductCrawler

java.lang.Object

org.apache.oodt.cas.crawl.config.ProductCrawlerBean

org.apache.oodt.cas.crawl.ProductCrawler

org.apache.oodt.cas.crawl.AutoDetectProductCrawler

All Implemented Interfaces:

CoreMetKeys, SpringSetIdInjectionType

public class AutoDetectProductCrawler
extends ProductCrawler
implements CoreMetKeys

A ProductCrawler that uses a suite of files to define its crawling and ingestion policy:

• actions-map.xml - This file is an XML specification for actions that the crawler should take in response to its 3 lifecycle phases: preIngest, postIngestSuccess, and postIngestFail.
• met-extr-preconditions.xml - This file defines preconditions that MetExtractors must pass before being called by the AutoDetectCrawler.
• mime-extractor-map.xml - This file maps MimeType names to names of MetExtractors to call for a particular Product File as it is encountered during a crawl (e.g., assuming that Metadata needs to be generated, as oppossed to being available apriori). See ./src/resources/examples/mime-extractor-map.xml for an example of the structure of this file.
• mimetypes.xml - An Apache Tika style mimetypes file, augmented with the ability to have arbitrary regular expressions that define a particular Product MimeType. This MimeType is then mapped to an extractor vai the mime-extractor-map.xml file, described above.
.

Author:

mattmann (Chris Mattmann), bfoster (Brian Foster)

Field Summary

Fields

Modifier and Type	Field and Description
static String	MIME_TYPES_HIERARCHY

Fields inherited from class org.apache.oodt.cas.crawl.ProductCrawler

actionRepo, DIR_FILTER, FILE_FILTER, ingester, ingestStatus, LOG

Fields inherited from interface org.apache.oodt.cas.filemgr.metadata.CoreMetKeys

FILE_LOCATION, FILE_SIZE, FILENAME, MIME_TYPE, PRODUCT_ID, PRODUCT_NAME, PRODUCT_RECEVIED_TIME, PRODUCT_STRUCTURE, PRODUCT_TYPE

Constructor Summary

Constructors

Constructor and Description
AutoDetectProductCrawler()

Method Summary

Modifier and Type	Method and Description
protected Metadata	getMetadataForProduct(File product)
protected boolean	passesPreconditions(File product)
protected File	renameProduct(File product, Metadata productMetadata)
void	setMimeExtractorRepo(String mimeExtractorRepo)

Methods inherited from class org.apache.oodt.cas.crawl.ProductCrawler

crawl, crawl, getIngestStatus, handleFile

Methods inherited from class org.apache.oodt.cas.crawl.config.ProductCrawlerBean

getActionIds, getApplicationContext, getClientTransferer, getDaemonPort, getDaemonWait, getFilemgrUrl, getGlobalMetadata, getId, getProductPath, getRequiredMetadata, isCrawlForDirs, isNoRecur, isSkipIngest, setActionIds, setApplicationContext, setClientTransferer, setCrawlForDirs, setDaemonPort, setDaemonWait, setFilemgrUrl, setGlobalMetadata, setId, setNoRecur, setProductPath, setRequiredMetadata, setSkipIngest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MIME_TYPES_HIERARCHY

public static final String MIME_TYPES_HIERARCHY

See Also:

Constant Field Values

Constructor Detail

AutoDetectProductCrawler

public AutoDetectProductCrawler()

Method Detail

getMetadataForProduct

protected Metadata getMetadataForProduct(File product)
                                  throws IOException,
                                         MetExtractionException

Specified by:

getMetadataForProduct in class ProductCrawler

Throws:

IOException

MetExtractionException

passesPreconditions

protected boolean passesPreconditions(File product)

Specified by:

passesPreconditions in class ProductCrawler

renameProduct

protected File renameProduct(File product,
                             Metadata productMetadata)
                      throws NamingConventionException

Specified by:

renameProduct in class ProductCrawler

Throws:

NamingConventionException

setMimeExtractorRepo

public void setMimeExtractorRepo(String mimeExtractorRepo)
                          throws IllegalAccessException,
                                 CrawlerActionException,
                                 MetExtractionException,
                                 InstantiationException,
                                 FileNotFoundException,
                                 ClassNotFoundException

Throws:

IllegalAccessException

CrawlerActionException

MetExtractionException

InstantiationException

FileNotFoundException

ClassNotFoundException