Import and Export Plug-ins

Import and Export Plug-ins are all about taking an object, de-serialising it into an eprint (or other object in the EPrints system) and re-serialising it back into an object to be supported.

We would typically advocate that every import plug-in should have a corresponding export plug-in and visa-versa.

Input Data

We are going to attempt to import the following XML record into our system. If you know the format of this record all the better else we'll leave it as a surprise for now.

 <?xml version="1.0" encoding="UTF-8"?>
 
 <package xmlns="http://www.idpf.org/2007/opf" unique-identifier="EPB-UUID" version="2.0">
  <metadata xmlns:opf="http://www.idpf.org/2007/opf"
            xmlns:dc="http://purl.org/dc/elements/1.1/">
     <dc:title>The Adventures of Sherlock Holmes</dc:title>
     <dc:creator opf:role="aut" opf:file-as="Doyle, Sir Arthur Conan">Sir Arthur Conan Doyle</dc:creator>
     <dc:publisher>epubBooks (www.epubbooks.com)</dc:publisher>
     <dc:date opf:event="epub-publication">2010-06-15</dc:date>
     <dc:subject>Crime/Detective</dc:subject>
     <dc:subject>Short Stories</dc:subject>
     <dc:source>Project Gutenberg</dc:source>
     <dc:rights>
           Provided for free by epubBooks.com. Not for commercial use.
           This EPUB eBook is released under a Creative Commons (BY-NC-ND/3.0) Licence.
           Source text and images are in the Public Domain.
     </dc:rights>
     <dc:identifier id="EPB-UUID">urn:uuid:FF946905-6C08-1014-90EA-E81F8523F0DC</dc:identifier>
     <dc:language>en-gb</dc:language>
  </metadata>
  <manifest>
  </manifest>
  <spine>
  </spine>
 </package>

Importing

To import this record we need a fully featured import plug-in which is capable of handle both file and raw XML input, we are also going to use an XSLT plug-ins to make the process even easier.

Stage 1: Create the files

We need the following:

• cfg/plugins/EPrints/Plugin/Import/OPFXML.pm
• cfg/plugins/EPrints/Plugin/Import/XSLT/DC.xsl

OPFXML.pm

This will be our master import plug-in to handle the entire chunk of XML. We will then use the DC plug-in to map the metadata into an eprint we can use later.

Since this is the master plug-in this requires quiet a sizeable amount of code:

 package EPrints::Plugin::Import::OPFXML;
 
 use strict;
 
 # Declare Namespaces
 our $DC_NS = "http://purl.org/dc/elements/1.1/";
 our $DCTERMS_NS = "http://purl.org/dc/terms/";
 our $OPF_NS = "http://www.idpf.org/2007/opf";
 
 # This is just a variant of the DefaultXML plug-in
 use EPrints::Plugin::Import::DefaultXML;
 
 our @ISA = qw/ EPrints::Plugin::Import::DefaultXML /;
 
 sub new
 {
       my( $class, %params ) = @_;
 
       my $self = $class->SUPER::new(%params);
 
       $self->{name} = "OPF Resource";
       # Make it visible on the import menu and elsewhere
       $self->{visible} = "all";
       $self->{produce} = [ 'list/eprint', 'dataobj/eprint' ];
 
       # Functionality to recognise XML types on import by recognising the base namespace, works with the sword packaging format, dc:conformsTo or similar.
       $self->{xmlns} = "http://www.idpf.org/2007/opf"
 
       my $rc = EPrints::Utils::require_if_exists("MIME::Types");
 
       unless( $rc )
       {
               $self->{visible} = "";
               $self->{error} = "Failed to load required module MIME::Types";
       }
 
       return $self;
 }
 
 # Input File Handle Method, for when files are uploaded
 sub input_fh
 {
       my( $plugin, %opts ) = @_;
 
       my $fh = $opts{"fh"};
 
       my $xml = join "", <$fh>;
 
       my $list;
 
       if( $xml =~ /^<\?xml/ )
       {
               $list = $plugin->input_fh_xml( $xml, %opts );
       }
       else
       {
               $list = $plugin->input_fh_list( $xml, %opts );
       }
 
       $list ||= EPrints::List->new(
                       dataset => $opts{dataset},
                       session => $plugin->{session},
                       ids => [] );
 
       return $list;
 }
 
 # Handle direct XML input
 sub input_fh_xml
 {
       my( $plugin, $xml, %opts ) = @_;
 
       my $doc = EPrints::XML::parse_xml_string( $xml );
 
       my $dataobj = $plugin->xml_to_dataobj( $opts{dataset}, $doc->documentElement );
 
       EPrints::XML::dispose( $doc );
 
       return EPrints::List->new(
                       dataset => $opts{dataset},
                       session => $plugin->{session},
                       ids => [$dataobj->get_id] );
 }
 
 # Go grab input from a URL
 sub input_fh_list
 {
       my( $plugin, $url, %opts ) = @_;
 
       $url =~ s/\s+//g;
 
       my $tmpfile = File::Temp->new;
 
       my $r = EPrints::Utils::wget( $plugin->{session}, $url, $tmpfile );
       seek($tmpfile,0,0);
 
       if( $r->is_error )
       {
               $plugin->error( "Error reading resource from $url: ".$r->code." ".$r->message );
               return;
       }
 
       my @ids;
 
       while(my $url = <$tmpfile>)
       {
               $url =~ s/\s+//g;
               next unless $url =~ /^http/;
 
               my $doc;
               eval { $doc = EPrints::XML::parse_url( $url ) };
               if( $@ )
               {
                       $plugin->warning( "Error parsing: $url\n" );
               }
 
               my $dataobj = $plugin->xml_to_dataobj( $opts{dataset}, $doc->documentElement );
 
               EPrints::XML::dispose( $doc );
 
               if( defined $dataobj )
               {
                       push @ids, $dataobj->get_id;
               }
       }
         
       return EPrints::List->new(
                       dataset => $opts{dataset},
                       session => $plugin->{session},
                       ids => \@ids );
 }
 # Translate this XML into an EPrint
 sub xml_to_dataobj
 {
       # $xml is the PubmedArticle element
       my( $plugin, $dataset, $xml ) = @_;
 
       my $session = $plugin->{session};
 
       # Locate the metadata element
       my $metadata = $xml->getElementsByTagNameNS( $OPF_NS, "metadata" )->[0];
 
       # Load the DC plugin
       my $dc_plugin = $session->plugin( "Import::XSLT::DC",
               processor => $plugin->{processor},
               dataset => $dataset,
       );
 
       # Spew the metadata element to a temp file
       my $tmpfile2 = File::Temp->new;
       print $tmpfile2 $metadata->toString();
       seek($tmpfile2,0,0);
 
       # Parse the file using the plug-in to get back a list of eprints
       my $list = $dc_plugin->input_fh( fh => $tmpfile2, dataset => $dataset );
 
       my( $eprint ) = $list->get_records( 0, 1 );
 
       return $eprint;
 }
 
 1;

XSLT/DC.xsl

Using XSLT is really quiet a nice way of parsing data and well documented so I shouldn't have to say much about how this works here.

 <?xml version="1.0"?>
 
 
 <xsl:stylesheet
       version="1.0"
       xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
       ept:name="DC"
       ept:visible="all"
       ept:advertise="1"
       ept:sourceNamespace="http://purl.org/dc/elements/1.1/"
       ept:targetNamespace="http://eprints.org/ep2/data/2.0"
       ept:produce="list/eprint"
       xmlns:ept="http://eprints.org/ep2/xslt/1.0"
       xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"
       xmlns:dc="http://purl.org/dc/elements/1.1/"
       xmlns:opf="http://www.idpf.org/2007/opf"
       xmlns="http://eprints.org/ep2/data/2.0"
 >
 
 <xsl:output method="xml" indent="yes" encoding="utf-8"/>
 
 <xsl:template match="/">
       <eprints>
       <eprint>
       <xsl:apply-templates select="./*" />
       <eprint_status>inbox</eprint_status>
       <creators>
       <xsl:for-each select="//dc:creator">
               <item><name><family/><given><xsl:value-of select="."/></given></name></item>
       </xsl:for-each>
       </creators>
       <corp_creators>
       <xsl:for-each select="//dc:source">
               <item>
                       <xsl:value-of select="." />
               </item>
       </xsl:for-each>
       </corp_creators>
       <keywords>
       <xsl:for-each select="//dc:subject">
               <xsl:value-of select="." />,
       </xsl:for-each>
       </keywords>
       </eprint>
       </eprints>
 </xsl:template>
 
 <xsl:template match="dc:title">
       <title><xsl:value-of select="." /></title>
 </xsl:template>
 
 <xsl:template match="dc:publisher">
       <publisher><xsl:value-of select="." /></publisher>
 </xsl:template>
 
 <xsl:template match="dc:rights">
       <copyright_holders><xsl:value-of select="." /></copyright_holders>
 </xsl:template>
 
 <xsl:template match="dc:date">
       <date><xsl:value-of select="." /></date>
 </xsl:template>
 
 <xsl:template match="dc:identifier">
       <id_number><xsl:value-of select="." /></id_number>
 </xsl:template>
 
 <xsl:template match="dc:language|dc:subject|dc:source|dc:creator"/>
 
 </xsl:stylesheet>

Testing

You can test your plug-ins by simply clicking on the "manage deposits" button in your toolbar and selecting "OPF Resource" from the drop down list of import plug-ins. If it doesn't appear restart you web server and check the error log!

Reminder:

 apache2ctl restart && tail -f /var/log/apache2/error.log

Once you have the import screen, copy and paste the XML above into it and see if it creates you an EPrint (which you can see back at the manage deposits screen)

Exporting

So now we have imported our EPrint from this OPF format we need to be able to put it back into this format for exporting, this has the advantage that you can also export ALL other eprints in this format.