Difference between revisions of "API:EPrints/Storage"

From EPrints Documentation
Jump to: navigation, search
m (→‎close_write: Reminder that this is an alternative to store.)
Line 8: Line 8:
 
<!-- Pod2Wiki=head_name -->
 
<!-- Pod2Wiki=head_name -->
 
==NAME==
 
==NAME==
−
'''EPrints::Storage''' - store and retrieve objects in the storage engine
+
EPrints::Storage - manage data streams in the storage devices
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
Line 30: Line 30:
 
<!-- Pod2Wiki=head_description -->
 
<!-- Pod2Wiki=head_description -->
 
==DESCRIPTION==
 
==DESCRIPTION==
 +
Unless you are directly manipulating how data streams are stored you should use the methods for retrieving and writing data available from [[API:EPrints/DataObj/File|EPrints::DataObj::File]].
 +
 
This module is the storage control layer which uses [[API:EPrints/Plugin/Storage|EPrints::Plugin::Storage]] plugins to support various storage back-ends. It enables the storage, retrieval and deletion of data streams. The maximum size of a stream is dependent on the back-end storage mechanism.
 
This module is the storage control layer which uses [[API:EPrints/Plugin/Storage|EPrints::Plugin::Storage]] plugins to support various storage back-ends. It enables the storage, retrieval and deletion of data streams. The maximum size of a stream is dependent on the back-end storage mechanism.
 +
 +
Storage works on [[API:EPrints/DataObj/File|EPrints::DataObj::File]] objects, which record the size of the data stream and where it has been stored. Unless a specific storage back-end is used where a file is stored depends on the storage policy, by default located in ''lib/storage/default.xml'':
 +
 +
<source lang="xml"><store xmlns="http://eprints.org/ep3/storage" xmlns:epc="http://eprints.org/ep3/control">
 +
<epc:choose>
 +
<epc:when test="datasetid = 'document'">
 +
<plugin name="Local"/>
 +
</epc:when>
 +
<epc:otherwise>
 +
<plugin name="Local"/>
 +
</epc:otherwise>
 +
</epc:choose>
 +
</store></source>
 +
 +
The storage policy is a EP-script file that resolves to a list of one or more "plugin" elements. When storing, all matching plugins will be given the data stream to store (i.e. multiple copies). When retrieving, the first plugin to be successfully opened for reading will be used. The item context used within the storage policy is the [[API:EPrints/DataObj/File|EPrints::DataObj::File]], which may only contain partial information.
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
Line 56: Line 73:
  
 
</source>
 
</source>
−
Read from and store all data from CODEREF for $fileobj.
+
Read from and store all data from CODEREF for $fileobj. If $offset is given starts writing from that point onwards.
  
 
Behaviour is undefined if an attempt is made to write beyond $fileobj's '''filesize'''.
 
Behaviour is undefined if an attempt is made to write beyond $fileobj's '''filesize'''.
  
−
Returns undef if the file couldn't be stored, otherwise the number of bytes actually written.
+
Returns the number of bytes written or undef if an error occurred.
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
 +
 
The combination of methods open_write, write, and close_write can be used as an alternative to this method.
 
The combination of methods open_write, write, and close_write can be used as an alternative to this method.
−
 
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki=head_retrieve -->
 
<!-- Pod2Wiki=head_retrieve -->
−
 
 
===retrieve===
 
===retrieve===
  
−
<source lang="perl">$success = $store->retrieve( $fileobj, $sourceid, $offset, $n, CALLBACK )
+
<source lang="perl">$success = $store->retrieve( $fileobj, $offset, $n, CALLBACK )
  
 
</source>
 
</source>
Line 82: Line 98:
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki=head_delete -->
 
<!-- Pod2Wiki=head_delete -->
−
 
 
===delete===
 
===delete===
  
−
<source lang="perl">$success = $store->delete( $fileobj, $sourceid )
+
<source lang="perl">$success = $store->delete( $fileobj )
  
 
</source>
 
</source>
Line 95: Line 110:
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki=head_delete_copy -->
 
<!-- Pod2Wiki=head_delete_copy -->
−
 
 
===delete_copy===
 
===delete_copy===
  
Line 101: Line 115:
  
 
</source>
 
</source>
−
Delete the copy of this file stored in $plugin.
+
Delete the copy of this file stored in the storage layer [[API:EPrints/Plugin/Storage|$plugin]].
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
Line 110: Line 124:
 
===get_local_copy===
 
===get_local_copy===
  
−
<source lang="perl">$fh = $store->get_local_copy( $fileobj, $sourceid )
+
<source lang="perl">$fh = $store->get_local_copy( $fileobj )
  
 
</source>
 
</source>
Line 124: Line 138:
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki=head_get_remote_copy -->
 
<!-- Pod2Wiki=head_get_remote_copy -->
−
 
 
===get_remote_copy===
 
===get_remote_copy===
  
Line 130: Line 143:
  
 
</source>
 
</source>
 +
Some storage back-ends may provide direct Web access to the data stream (e.g. Amazon S3). If the back-end supports this, a user can be redirected to the storage back-end rather than downloading the file from EPrints. Internally, EPrints will always use the [[API:EPrints/Storage#retrieve|retrieve]] method.
 +
 
Returns a URL from which this file can be accessed.
 
Returns a URL from which this file can be accessed.
  
Line 144: Line 159:
  
 
</source>
 
</source>
−
Returns the [[API:EPrints/Plugin/Storage|EPrints::Plugin::Storage]] plugin(s) to use for $fileobj. If more than one plugin is returned they should be used in turn until one succeeds.
+
Returns the [[API:EPrints/Plugin/Storage|EPrints::Plugin::Storage]] plugin(s) available to use for $fileobj, based on the storage policy. If more than one plugin is returned they should be used in turn until one succeeds.
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
Line 156: Line 171:
  
 
</source>
 
</source>
−
Copy the contents of $fileobj into another storage $plugin.
+
Copy the contents of $fileobj into another storage $plugin. This uses [[API:EPrints/Storage#retrieve|retrieve]] and [[API:EPrints/Storage#store|store]].
  
 
Returns 1 on success, 0 on failure and -1 if a copy already exists in $plugin.
 
Returns 1 on success, 0 on failure and -1 if a copy already exists in $plugin.
Line 171: Line 186:
 
</source>
 
</source>
 
Start a write session for $fileobj. $fileobj must have at least the "filesize" property set (which is the total number of bytes that will be written).
 
Start a write session for $fileobj. $fileobj must have at least the "filesize" property set (which is the total number of bytes that will be written).
 +
 +
Don't forget to [[API:EPrints/Storage#close_write|close_write]] otherwise the handle to the write will be left open.
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
 +
 
This, along with write and close_write, is an alternative to the store method.
 
This, along with write and close_write, is an alternative to the store method.
−
 
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki=head_write -->
 
<!-- Pod2Wiki=head_write -->
−
 
 
===write===
 
===write===
  
Line 186: Line 202:
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
 +
 
This, along with open_write and close_write, is an alternative to the store method.
 
This, along with open_write and close_write, is an alternative to the store method.
−
 
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki=head_close_write -->
 
<!-- Pod2Wiki=head_close_write -->
−
 
 
===close_write===
 
===close_write===
  
Line 199: Line 214:
  
 
<!-- Edit below this comment -->
 
<!-- Edit below this comment -->
−
This, along with open_write and write, is an alternative to the store method.
+
 
  
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki= -->
 
<!-- Pod2Wiki=head_copyright -->
 
<!-- Pod2Wiki=head_copyright -->
−
 
 
==COPYRIGHT==
 
==COPYRIGHT==
 
Copyright 2000-2011 University of Southampton.
 
Copyright 2000-2011 University of Southampton.

Revision as of 09:51, 25 July 2013

EPrints 3 Reference: Directory Structure - Metadata Fields - Repository Configuration - XML Config Files - XML Export Format - EPrints data structure - Core API - Data Objects

API: Core API

Latest Source Code (3.4, 3.3) | Revision Log | Before editing this page please read Pod2Wiki


Warning The interface for this module has not been finalised and may change in the future.

NAME

EPrints::Storage - manage data streams in the storage devices


SYNOPSIS

my $store = $repository->storage();

$store->store(
	$fileobj,		# file object
	"diddle.pdf",	# filename
	$fh				# file handle
);


DESCRIPTION

Unless you are directly manipulating how data streams are stored you should use the methods for retrieving and writing data available from EPrints::DataObj::File.

This module is the storage control layer which uses EPrints::Plugin::Storage plugins to support various storage back-ends. It enables the storage, retrieval and deletion of data streams. The maximum size of a stream is dependent on the back-end storage mechanism.

Storage works on EPrints::DataObj::File objects, which record the size of the data stream and where it has been stored. Unless a specific storage back-end is used where a file is stored depends on the storage policy, by default located in lib/storage/default.xml: 

<store xmlns="http://eprints.org/ep3/storage" xmlns:epc="http://eprints.org/ep3/control">
<epc:choose>
	<epc:when test="datasetid = 'document'">
		<plugin name="Local"/>
	</epc:when>
	<epc:otherwise>
		<plugin name="Local"/>
	</epc:otherwise>
</epc:choose>
</store>

The storage policy is a EP-script file that resolves to a list of one or more "plugin" elements. When storing, all matching plugins will be given the data stream to store (i.e. multiple copies). When retrieving, the first plugin to be successfully opened for reading will be used. The item context used within the storage policy is the EPrints::DataObj::File, which may only contain partial information.


METHODS

new

$store = EPrints::Storage->new( $repository )

Create a new storage object for $repository. Should not be used directly, see EPrints::Session.


store

$len = $store->store( $fileobj, CODEREF [, $offset ] )

Read from and store all data from CODEREF for $fileobj. If $offset is given starts writing from that point onwards.

Behaviour is undefined if an attempt is made to write beyond $fileobj's filesize.

Returns the number of bytes written or undef if an error occurred.


The combination of methods open_write, write, and close_write can be used as an alternative to this method.

retrieve

$success = $store->retrieve( $fileobj, $offset, $n, CALLBACK )

Retrieve the contents of the $fileobj starting at $offset for $n bytes.

CALLBACK = $rc = &f( BUFFER )


delete

$success = $store->delete( $fileobj )

Delete all object copies stored for $fileobj.


delete_copy

$ok = $store->delete_copy( $plugin, $fileobj )

Delete the copy of this file stored in the storage layer $plugin.


get_local_copy

$fh = $store->get_local_copy( $fileobj )

Return a local copy of the file. Potentially expensive if the file has to be retrieved.

Stringifying $fh will give the full path to the local file, which may be useful for calling external tools (see File::Temp).

Returns undef if retrieval failed.


get_remote_copy

$url = $store->get_remote_copy( $fileobj )

Some storage back-ends may provide direct Web access to the data stream (e.g. Amazon S3). If the back-end supports this, a user can be redirected to the storage back-end rather than downloading the file from EPrints. Internally, EPrints will always use the retrieve method.

Returns a URL from which this file can be accessed.

Returns undef if this file is not available via another service.


get_plugins

@plugins = $store->get_plugins( $fileobj )

Returns the EPrints::Plugin::Storage plugin(s) available to use for $fileobj, based on the storage policy. If more than one plugin is returned they should be used in turn until one succeeds.


copy

$ok = $store->copy( $plugin, $fileobj )

Copy the contents of $fileobj into another storage $plugin. This uses retrieve and store.

Returns 1 on success, 0 on failure and -1 if a copy already exists in $plugin.


open_write

$ok = $storage->open_write( $fileobj [, $offset ] )

Start a write session for $fileobj. $fileobj must have at least the "filesize" property set (which is the total number of bytes that will be written).

Don't forget to close_write otherwise the handle to the write will be left open.


This, along with write and close_write, is an alternative to the store method.

write

$ok = $storage->write( $fileobj, $buffer )

Write $buffer to the storage plugin(s), starting from the previously written data.


This, along with open_write and close_write, is an alternative to the store method.

close_write

$ok = $storage->close_write( $fileobj );

Finish writing to the storage plugin(s) for $fileobj.


COPYRIGHT

Copyright 2000-2011 University of Southampton.

This file is part of EPrints http://www.eprints.org/.

EPrints is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

EPrints is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with EPrints. If not, see http://www.gnu.org/licenses/.



Retrieved from ‘https://wiki.eprints.org/w/index.php?title=API:EPrints/Storage&oldid=10817’