Difference between revisions of "Contribute: Plugins/ImportPluginsCSV"

From EPrints Documentation
Jump to: navigation, search
m (User:Tom/Import Plugins/Csv moved to User:Tom/Import Plugins/CSV)
(input_fh)
Line 202: Line 202:
 
This method takes a filehandle, processes it, tries to import DataObjs in to the repository and then returns a List of the DataObjs imported.
 
This method takes a filehandle, processes it, tries to import DataObjs in to the repository and then returns a List of the DataObjs imported.
  
 +
This array will be used to create a List of DataObjs later.
 
<pre>
 
<pre>
 
         my @ids;
 
         my @ids;
 
</pre>
 
</pre>
 +
 +
Here we open the filehandle passed, and read the lines into an array.
 
<pre>
 
<pre>
 
         my $fh = $opts{fh};
 
         my $fh = $opts{fh};
 
         my @records = <$fh>;
 
         my @records = <$fh>;
 
</pre>
 
</pre>
 +
 +
We create a Text::CSV object to handle the input. Using a dedicated CSV handling package is preferable to using Perl's split function as it handles a number of more complicated scenarios such as commas within records using double quotes.
 
<pre>
 
<pre>
 
         my $csv = Text::CSV->new();
 
         my $csv = Text::CSV->new();
 
</pre>
 
</pre>
 +
 +
After setting up an array for metadata field names, we attempt to parse the first line of our file. The parse method does not return an array of fields, but reports success or failure. In the event of success we use the fields method to return the last fields parsed. In the event of failure we use the error_input method to get the last error, and return undef.
 
<pre>
 
<pre>
 
         my @fields;
 
         my @fields;

Revision as of 10:02, 17 September 2007

Import Plugin Tutorial 1: CSV

In this tutorial we will look at creating a relatively simple plugin to import eprints into our repository by reading files containing comma separated variables. We won't be dealing with documents and files, but will be focusing on importing eprint metadata.

Import plugins are inherently more complicated than export plugins because of the error checking that must be done, however in this example error checking has been kept to a minimum to simplify the example. In a "real" plugin you should check that the appropriate metadata fields are set for a given type of eprint, and unfortunately there appears to be no quick way to do this.

Before You Start

It is sensible to separate the plugins you create for EPrints from those included with it. Create a directory for your import plugins in the main plugin directory (usually /opt/eprints3/perl_lib/EPrints/Plugin/import) for example /opt/eprints3/perl_lib/EPrints/Plugin/import/MyPlugins.

To prepare for this tutorial you should install the Text::CSV module. The following command as root, or using sudo should work.

cpan Text::CSV

CSV.pm

package EPrints::Plugin::Import::MyPlugins::CSV;

use EPrints::Plugin::Import::TextFile;
use strict;

our @ISA = ('EPrints::Plugin::Import::TextFile');

sub new
{
        my( $class, %params ) = @_;

        my $self = $class->SUPER::new( %params );

        $self->{name} = 'CSV';
        $self->{visible} = 'all';
        $self->{produce} = [ 'list/eprint' ];

        my $rc = EPrints::Utils::require_if_exists('Text::CSV');
        unless( $rc )
        {
                $self->{visible} = '';
                $self->{error} = 'Failed to load required module Text::CSV';
        }

        return $self;
}

sub input_fh
{
        my( $plugin, %opts ) = @_;
        my @ids;
        my $fh = $opts{fh};
        my @records = <$fh>;
        my $csv = Text::CSV->new();
        my @fields;

        if ($csv->parse(shift @records))
        {
                @fields = $csv->fields();
        }
        else
        {
                $plugin->error($csv->error_input);
                return undef;
        }

        foreach my $row (@records)
        {
                my @input_data = (join(',',@fields),$row);

                my $epdata = $plugin->convert_input(\@input_data);
                next unless defined $epdata;

                my $dataobj = $plugin->epdata_to_dataobj($opts{dataset},$epdata);
                if( defined $dataobj )
                {
                        push @ids, $dataobj->get_id;
                }
        }

        return EPrints::List->new(
                        dataset => $opts{dataset},
                        session => $plugin->{session},
                        ids=>\@ids );
}

sub convert_input
{
        my $plugin = shift;
        my @input = @{shift @_};
        my $csv = Text::CSV->new();

        my @record;
        if ($csv->parse($input[1]))
        {
                @record = $csv->fields();
        }
        else
        {
                $plugin->error($csv->error_input);
                return undef;
        }

        my @fields = split(',',$input[0]);
        #Check length of row
        if (scalar @fields != scalar @record)
        {
                $plugin->warning('Row length mismatch');
                return undef;
        }

        my %output = ();

        my $dataset = $plugin->{session}->{repository}->get_dataset('archive');

        my $i = 0;
        foreach my $field (@fields)
        {
                #Check field exists
                unless ($dataset->has_field($field))
                {
                        $i++;
                        next;
                }

                my $metafield = $dataset->get_field($field);
                #Check for multiple
                if ($metafield->get_property('multiple'))
                {
                        #Split
                        my @values = split(';',$record[$i]);

                        #Check for name
                        if ($metafield->{type} eq 'name')
                        {
                                my @names = ();

                                foreach my $value (@values)
                                {
                                        my $name = $value;

                                        next unless ($value =~ /^(.*?),(.*?)(,(.*?))?$/);
                                        push @names, { family => $1, given => $2, lineage => $4 };
                                }

                                $output{$field} = \@names;
                        }
                        else
                        {
                                $output{$field} = \@values;
                        }
                }
                else
                {
                        $output{$field} = $record[$i];
                }
                $i++;
        }
        return \%output;
}

1;

In More Detail

Modules

Here we import the superclass for our plugin.

use EPrints::Plugin::Import::TextFile;

Inheritance

Our plugin will not inherit from the Import class directly, but from the TextFile subclass. This contains some extra file handling code that means we can ignore certain differences in text file formats. If you are creating an import plugin which imports non-text files you should subclass the EPrints::Plugin::Import class directly.

our @ISA = ('EPrints::Plugin::Import::TextFile');

Constructor

For import plugins we must set a 'produce' property, to tell the repository what kinds of objects the plugin can import. This plugin only supports importing lists of eprints, but if it supported importing individual eprints we could add 'dataobj/eprint' to this property. We would then have to implement the "input_dataobj" method. Most plugins implement this method, but it is rarely used in practice. Most imports are done in lists (even if that list only contains one member), via the import items screen.

        $self->{produce} = [ 'list/eprint' ];

Here we use a module that is not included with EPrints, Text::CSV, so we import it in a different way. First we check that it is installed, and load it if it is with "EPrints::Utils::require_if_exists".If it isn't we make the plugin invisible and produce an error message. It is good practice to import non-standard modules in this way rather than with "use".

        my $rc = EPrints::Utils::require_if_exists('Text::CSV');
        unless( $rc )
        {
                $self->{visible} = '';
                $self->{error} = 'Failed to load required module Text::CSV';
        }

Input

Import plugins have to implement a couple of methods to read data from a file or string, manipulate it and turn it into a form which can be imported into the repository. That process will be described below.

input_fh

This method takes a filehandle, processes it, tries to import DataObjs in to the repository and then returns a List of the DataObjs imported.

This array will be used to create a List of DataObjs later.

        my @ids;

Here we open the filehandle passed, and read the lines into an array.

        my $fh = $opts{fh};
        my @records = <$fh>;

We create a Text::CSV object to handle the input. Using a dedicated CSV handling package is preferable to using Perl's split function as it handles a number of more complicated scenarios such as commas within records using double quotes.

        my $csv = Text::CSV->new();

After setting up an array for metadata field names, we attempt to parse the first line of our file. The parse method does not return an array of fields, but reports success or failure. In the event of success we use the fields method to return the last fields parsed. In the event of failure we use the error_input method to get the last error, and return undef.

        my @fields;
        if ($csv->parse(shift @records))
        {
                @fields = $csv->fields();
        }
        else
        {
                $plugin->error($csv->error_input);
                return undef;
        }
        foreach my $row (@records)
        {
                my @input_data = (join(',',@fields),$row);

                my $epdata = $plugin->convert_input(\@input_data);
                next unless defined $epdata;

                my $dataobj = $plugin->epdata_to_dataobj($opts{dataset},$epdata);
                if( defined $dataobj )
                {
                        push @ids, $dataobj->get_id;
                }
        }
        return EPrints::List->new(
                        dataset => $opts{dataset},
                        session => $plugin->{session},
                        ids=>\@ids );

convert_input

sub convert_input
{
        my $plugin = shift;
        my @input = @{shift @_};
        my $csv = Text::CSV->new();

        my @record;
        if ($csv->parse($input[1]))
        {
                @record = $csv->fields();
        }
        else
        {
                $plugin->error($csv->error_input);
                return undef;
        }

        my @fields = split(',',$input[0]);
        #Check length of row
        if (scalar @fields != scalar @record)
        {
                $plugin->warning('Row length mismatch');
                return undef;
        }

        my %output = ();

        my $dataset = $plugin->{session}->{repository}->get_dataset('archive');

        my $i = 0;
        foreach my $field (@fields)
        {
                #Check field exists
                unless ($dataset->has_field($field))
                {
                        $i++;
                        next;
                }

                my $metafield = $dataset->get_field($field);
                #Check for multiple
                if ($metafield->get_property('multiple'))
                {
                        #Split
                        my @values = split(';',$record[$i]);

                        #Check for name
                        if ($metafield->{type} eq 'name')
                        {
                                my @names = ();

                                foreach my $value (@values)
                                {
                                        my $name = $value;

                                        next unless ($value =~ /^(.*?),(.*?)(,(.*?))?$/);
                                        push @names, { family => $1, given => $2, lineage => $4 };
                                }

                                $output{$field} = \@names;
                        }
                        else
                        {
                                $output{$field} = \@values;
                        }
                }
                else
                {
                        $output{$field} = $record[$i];
                }
                $i++;
        }
        return \%output;
}