EPrints Documentation - User contributions [en-gb]

Talk:GDPR

2018-06-01T11:18:42Z

Cjg:

I'm kind of torn between the two methods - John's has the advantage that it isn't dependent upon the mechanism by which someone creates a login ticket so should be more universal (e.g. for us), but Will's has the advantage that it's perhaps easier to understand for the less specialist eprints dev / admin. I.e. it doesn't use the magic of triggers, and it uses a date format that can be easily read rather than epoch timestamp (which apparently not everyone tracks on a daily basis) - Alan (2018-04-25 14:12:30z+01:00)

== Code queries ==

John, Are you deliberately storing just the expiry date of the login ticket rather than the specific login time? Grand scheme I don't suppose it makes a lot of difference if there's an extra 7 days in the delete check.

- Alan

----

No real reason - I was thinking that the retention period would be e.g. a year or so - and was trying to make things as least impactful as possible - so reusing a value that was already set.
The problem with this is that if a repository has set a long expiry time, it could be bad (so maybe Will's suggestion is better).

I initially looked at using a commit trigger on the login ticket - but this changes every time you do anything - so introduces a lot of 'churn' on the system.

Thinking about it again, a simple `time` call might be better (storing it as seconds).
In the cleanup scripts, you can then calculate the '1-year-ago' seconds and do a 'last_login < 1_YEAR_AGO'.

- JLRS (John)

----

I prefer the trigger to using user_login.pl was going to suggest using the trigger + localtime (like Will's done), but int timestamps do make date maths less onerous as John points out

- RM (Rory)

----

<blockquote> >date maths less onerous</blockquote>
And 1 DB field rather than 3 or 6 - simpler queries, more efficient

Also, rather than using the 'expiry' field, there is a 'time' field - which makes more sense (I think)

- JLRS

The logic of the weird time fields in EPrints is to distinguish between vague dates eg. year only. In retrospect using a timestamp and a "resolution indicator" would have been better. Sigh. [[User:Cjg|Cjg]] ([[User talk:Cjg|talk]]) 11:18, 1 June 2018 (UTC)
----

GDPR

2018-06-01T10:34:17Z

Cjg: /* Request a copy Dataset */

This page has been created to gather information and share code snippets to help EPrints repositories handle GDPR responsibilities. Some of the issues around GDPR will be specific to a repository contents, but others are quite generic. nb. We're not lawyers!

As a general rule of thumb, this is about people's rights and privacy. Imagine a worst case scenario where the entire filesystem and database is made entirely public -- it's not a great situation, but consider what steps you could take to reduce the damage if that did happen - don't store any information about people that you don't actually need.

=== Areas of GDPR that may be relevant to EPrints repositories ===
* Giving people a clear privacy policy about how you will use data you gather about them
* Gaining consent for storing and processing data you gather about them
* Defining and enforcing a retention policy to remove information about people that is no longer required
* "Subject Access Requests" where someone wants to know everything you know about them
* "Right to Erasure" where people may request you remove information about them, but there are exceptions to this.

=== Information about people ===
EPrints potentially stores information about people in a number of places.
* Users Dataset
* EPrints Metadata
** In the revision XML files storing old versions of metadata.
* In actual documents
* In the "request a copy" Dataset
* Web access logs
** In the History Dataset
** In the Apache logs in the operating system (this will not be dealt with on this page, but should not be forgotten)

== Users Dataset ==
This dataset is either populated with a web based sign up form, or in many cases, it is automatically built from an institutional accounts system. These cases need to be addressed in different ways.

=== Imported users Dataset ===

First of all, stop importing any fields you don't actually need and remove those fields from EPrints. nb. Removing the field in the config may not remove it from the database (citation required.. how the hell does that work?)

Next stop importing any users you just don't need -- if they can't do anything while logged in they should never exist in the Users dataset.

Next implement a data retention policy. This will be a script that removes or blanks the data of users that match a criteria. eg. have not appeared in the import script for more than 90 days and never made a deposit.

=== Web signup users Dataset ===

You should ensure that the privacy policy is up to date and in line with GDPR best practice.

Stop collecting any information you don't actually need, and remove those fields from the database.

Next implement a data retention policy. This will be a script that removes or blanks the data of users that match a criteria. eg. accounts more than a year old that have never made a deposit.

=== Removing vs Blanking ===

In some cases it may be more desirable to remove all the personal metadata from a record in the Users dataset rather than delete it completely. This has pros and cons.

If you remove a user entirely who has deposited EPrints you may wish to reassign those deposits to an admin user, or a fake user called "left".

=== Last Login Time===
Storing the last login time of a user can be useful to identify which users are active and which are not to help ensure data is not being stored longer than is necessary.

See [[GDPR/Last Login Time]]

===Non-Active Users Report ===
TODO
worth having a count of items linked with the user on the report, along with their statuses?

===Delete User Action===
*Suggestion* Cron job to select all user accounts which have been inactive over a certain time threshold, check those users for affiliation to a publication (creator, editor, depositor etc.) which is either archive or buffer status and, where no connection exists, remove those user accounts.

Admin will be able to select user accounts from the report to delete (within the report or individually?)

Could be similar to: <tt>EPrints::DataObj::LoginTicket::expire_all</tt> - but with an <tt>$c->{'account_retention_time'}</tt> param taken away from the <tt>time()</tt> call?

Need to check logintickets too?
Someone may not have actually logged in for ages, but maintained a browser session/cookies for a long time?
(currently our longest 'active' login is ~20 days).

==Add 'Agree to privacy and data statement' checkbox on registration or request forms==
TODO - should be 'versioned' - so if the statement changes, the version that was agreed to can be shown.
Possibly a set/namedset, with the available options limited to only one when rendered to a user?

Chris Gutteridge has written a blog post on GDPR with a few comments about the EPrints Request a Copy feature: http://blog.soton.ac.uk/webteam/2018/05/10/gdpr-preperations/#post-content-1793;char=5429-7155

==EPrints Metadata==
TODO
==In actual documents==
TODO
==Request a copy Dataset==
This collects an email address and a reason for requesting the document. Without intervention this can be stored indefinitely.

You should add a privacy statement to the form to say what you'll do with the information, add a reminder to the email sent to the authors that they shouldn't misuse the contact info, define a retention policy, automate the enforcement of the retention policy.

===Adding a Privacy Statement===
TODO
===Altering the email to authors===
TODO
===Automating the removal of old requests===
TODO

==History dataset (used for access stats)==
TODO

2010-05-19T12:04:28Z

Cjg:

EPrints is written in the [http://www.perl.org/ Perl language]. Sometimes you may need to write your own Perl code using the EPrints Application Programming Interface (API).

Reasons you may need to write your own EPrints code:

* customising the way the eprints summary pages are rendered
* writing your own command-line script to control EPrints in some way
* writing a new CGI script (dynamic web page)
* writing a plugin

== Key Sections ==
* [[Core API]] provides a basic introduction to using the EPrints API.
* [[StyleGuide]] gives guidance on how to code in a compatible style.
* [[Modules]] discusses the framework for Plugins you write for the community.

==Principle Modules==

This is a list of the principle modules in the EPrints API. These modules are the primary means of connecting to and manipulating objects in the repository.

===[[API:EPrints/Repository|EPrints::Repository]]===

An Repository is a eprints archive with it's own website configuration and data. One install of the eprints software can run serveral seperate repositories. Sharing code but with totally different configurations. Before EPrints 2.4 this was known as EPrints::Archive. This was changed to avoid confusion with the eprint status of "archive".

===[[API:EPrints/Database|EPrints::Database]]===

The connection to the MySQL back end. ''datasets'' are stored in the MySQL system, but you do not have to address it directly.

===[[API:EPrints/DataSet|EPrints::DataSet]]===

A dataset is a collection of ''items'' of the same type. It can be searched.

Some datasets all have the same "config id". The "config id" is used to get information about the dataset from the archive config - inbox, buffer, archive and deletion all have the same metadata fields and types.

Core datasets are:

{|
| DATASET ID
| COMMENT
|-
| eprint
| EPrint records are the core of the system.
|-
| user
| Users registered with the system.
|-
| subject
| The subject tree.
|-
| document
| Documents belonging to EPrints. Every document is part of an EPrint record.
|-
| subscription
| Subscriptions made by users. Every subscription is a part of a Subscription record.
|-
| history
| Stores actions performed on records.
|}

In addtion to these datasets are four virtual datasets: inbox, buffer, archive and deletion. These act just like "eprint" except that they are filtered to only contain records with those status.

Note that prior to 2.4 the "eprint" dataset was virtual, rather than "inbox", "buffer" etc. The History dataset was introduced in 2.4.

===[[API:EPrints/MetaField|EPrints::MetaField]]===

A single field in a dataset. Each dataset has a few "system" fields which eprints uses to manage the system and then any number of ''archive'' specific fields which you may configure.

===[[API:EPrints/DataObj|EPrints::DataObj]]===

The "super class" of ''subjects'', ''users'', ''eprints'' and ''documents'' etc. In the very core of the system these are all treated identically and much of the configuration and methods of these classes of "thing" are identical. We use the term ''item'' to speak about the general case.

; '''type''' or '''user-type''' or '''eprint-type''' : ''users'', ''eprints'' and ''documents'' all have a "type". This controls how they are "cited" and also for ''users'' and ''eprints'' it controls what ''fields'' may be edited, and which are required.

====[[API:EPrints/DataObj/Document|EPrints::DataObj::Document]]====

A document is a single format of an ''eprint'', eg. HTML, PDF, PS etc. It can contain more than one file, for example HTML may contain more than one html page + image files. The actual files are stored in the filesystem. Pre 2.4 this was known as EPrints::Document.

====[[API:EPrints/DataObj/EPrint|EPrints::DataObj::EPrint]]====

An eprint is a record in the system which has one or more ''documents'' and some ''metadata''. Usually, more than one ''document'' is to provide the same information in multiple formats, although this is not compulsory. Pre 2.4 this was known as EPrints::EPrint.

====[[API:EPrints/DataObj/SavedSearch|EPrints::DataObj::SavedSearch]]====

Some software refers to this concept as "alerts".

A stored search which is performed every day/week/month and any new results are the mailed to the user who owns the subscription.

This diagram does not show "Subscription". Subscription is a subclass of DataObj (like EPrint, User etc.). A Subscription is associated with one User. A User is associated with 0..n Subscription's.

====[[API:EPrints/DataObj/Subject|EPrints::DataObj::Subject]]====

A ''subject'' has an id and a list of who it's parents are. There is a build in ''subject'' with the id "ROOT" to act as the top level. A subject can have more than one parent to allow you to create a rich lattice, rather than just a tree, but loops are not allowed.

====[[API:EPrints/DataObj/User|EPrints::DataObj::User]]====

A user registered with the system. (NOT necesarily the author of the ''eprints'' they deposit). Pre 2.4 this was known as EPrints::User

2010-05-19T12:00:26Z

Cjg:

2010-05-18T17:21:52Z

Cjg: /* limit */

EPrints Control Format

2010-05-18T17:21:43Z

Cjg: /* $index (3.2.0+) */

{{formats}}

= EPrints Control =

== Output Tags ==

=== epc:print ===

<epc:print expr='expression' opt='key=value;key2=value2'/>

<cite:citation xmlns="http://www.w3.org/1999/xhtml"
xmlns:cite="http://eprints.org/ep3/citation"
xmlns:epc="http://eprints.org/ep3/control">
<epc:print expr="$item.citation('default')" />
</cite:citation>

Outputs the result of the expression in the 'expr' attribute.

Rendering options may be passed in the opt attribute using a key=value form, with semicolons separating multiple options.

Any [[EPScript]] expression can be used. The most simple is just 'eprintid','title','creators' etc. You can also use values from the configuration files by using '$config{base_url}'.

=== Print in attributes using {} ===

Sometimes you may wish to insert a value into an XML attribute. eg. the "href" part of an anchor. Rather than a complicated system, but correct XML, we decide to go for something a bit more readable. Any {} pair in an XML attribute will be treated as a epc:print.

So you can do something like

<nowiki><a href="http://eprints.badger.edu/cgi/myscript.pl?eprintid={eprintid}">run myscript on this eprint</a></nowiki>

=== epc:phrase ===

simple:
<epc:phrase ref='phraseid' />

with pins:
<epc:phrase ref='phraseid'>
<epc:param name='somepin'>Content</epc:param>
</epc:phrase>

Outputs the content of the phrase refered to by 'ref'. Any necessary parameters may be set using the <epc:param> tag, with the contents inserted into the pin with the name corresponding to the name attribute.

== Conditional Tags ==

=== epc:if ===

<epc:if test='expr'>
...
</epc:if>

Outputs any XHTML content and evaluates any EPrints Control structures within the epc:if block if the expression in the test attribute is true.

=== epc:choose, epc:when, epc:otherwise ===

<epc:choose>
<epc:when test='expr'>
...
</epc:when>
<epc:when test='expr2'>
...
</epc:when>
<epc:otherwise>
...
</epc:otherwise>
</epc:choose>

epc:choose allows for the construction of a complex conditional. Each epc:when block's 'test' attribute is evaluated, and the content of the block is returned if the result is true. If no expressions return true, the optional epc:otherwise block is returned. Note that no subsequent epc:when blocks are evaluated once the first block to return true is reached.

== Loops ==

Multiple values can be iterated over. For example.

<epc:foreach expr="creators_name" iterator="name">
( <epc:print expr="$name" /> )
</epc:foreach>

=== $index (3.2.0+) ===

Within a foreach block you can use $index to find what iteration of the loop is on. The first value is zero.

This can be used with the modulus "%" to add a strip effect, if wanted, eg.

<nowiki><table>
<epc:foreach expr="creators_name" iterator="name">
<epc:if expr="$index % 2">
<tr class='odd'><td><epc:print expr="$name" /></td></tr>
</epc:if>
<epc:if expr="!($index % 2)">
<tr class='even'><td><epc:print expr="$name" /></td></tr>
</epc:if>
</epc:foreach>
</table></nowiki>

=== limit ===

You can set a limit of how many iterations to process using a "limit" attribute on the epc:foreach element. For example,

<epc:foreach expr="creators_name" iterator="name" limit="3">

2010-03-11T11:51:31Z

Cjg: /* Semantic Web Support */

[[Category:Releases]]
{{releasenotes}}
This page is now describing actual features, rather than planned.

The [http://wiki.eprints.org/index.php?title=New_Features_in_EPrints_3.2&oldid=6394 Planned Features] may be useful to improve the quality of descriptions on this page.

Unsorted items:
* ISIWok?
* Added support for "input_boxes" property to the workflow, so you can now specify the number of input boxes to show for multiple fields, fixes #3675 [Alexander 'Leo' Bergolth <leo@strike.wu.ac.at>]
* Added --clear option to indexer to clear 'inprogress' events (in case things go wrong)
* epc: no longer crashes eprints on bad scripts, just reports an error

==NOTE==
* We now recommend LibXML in preference to GDOME. It's less buggy, and easier to install.
* Upgrade may take several hours as it cleans up the unicode issues in the database.

==Database==
* In addition to MySQL, EPrints 3.2 now supports Oracle and Postgres

==API==
* This release features a formal API. Not all functionality is yet available via the API, but will be added slowly and carefully in future releases.
* The bugbear of EPrints internals, EPrints::Session has been merged into EPrints::Repository. All old code will still work.

==Documents==
* Thumbnails are now documents in their own right
* Built in document-format icons, as well as those you configure yourself
* Thumbnailing now happens in the background as part of the indexer process

==Deposit Interface==
* Edit Locking locks records reduces risk of 2 people editing a record at the same time.
* Option to extract metadata and images from OpenXML files (.docx and .pptx)
* Offers options to users and editors on the deposit screen if there are problems
* Document upload screen has been redesgined to be clearer.
* Split document uploading into adding a new document and editing existing documents
* The documents inside an EPrint may now be re-ordered
* Progress bar on file upload
* Document upload methods (file, url, zip etc.) are now plugin-based and can be extended
* When attempting to deposit an eprint with problems show Save button
* Made it an option to provide action buttons top and bottom in workflow

==Search & Indexing==
* The search library has been entirely re-written to reduce use of cache tables and to improve performance. Simple searches are now over ten times faster.
* The indexer now uses plugins, so you can schedule other tasks, like thumbnail conversion, to be done in the background.
* Added config option "cache_max" to limit the cachemap tables used

==Unicode==
* EPrints use of unicode has been significantly improved.

==REST==
* A "REST" style interface to objects, via /rest/eprint/23/title.txt, for example. This can also support "PUT" to alter fields!

==WebDav==
* ???
* (tdb: Too buggy for official release - no decent clients available on standard platforms)
* [[User:Cjg|Cjg]]: (remove this section if it is wrong, I guess?)

==SWORD2==
* SWORD2 (1.3 Specification) is supported.

==Linked Data Support==
* Ability to establish arbitrary relations between objects or provide additional metadata in triple form.

==Collections Support==
* Collections can be built via use of linked data, object ids and relationships.

== Semantic Web / Linked Data (RDF) ==

We have made a (difficult) decision to move these features to 3.2.1 (due out soon after 3.2.0) because testing showed it caused a significant slow down.

We're rewriting it to do the same thing but with much less overhead!

==Storage Layer==
* Now uses plugins to store files
** Local Filesystem
** Amazon S3
** Sun Cloud Storage

==Speed==
* Search & Indexing much faster
* Import is faster
* Other parts of the code have been audited for speed, and optimised.

==Import==
* Modified Import UI to allow a per-plugin/single/bulk workflow

==EPC & EPrints Script==
* New EPC tag: epc:debug, which is like print but sends the XML to STDERR for debugging purposes.
* New EPC tag: epc:set which defines a variable inside it's scope.
* Improvements to the epc:foreach processing (better handling of multiple object types in lists)
* Added "limit" option to epc:foreach to limit the number
* Inside <epc:foreach> blocks an $index variable is set, allowing you to test which interation it's on.
* New EPScript methods: citation_link, dataset, related_objects, url, doc_size, is_public, thumbnail_url, preview_link, icon, human_filesize, control_url, contact_email, property, substr, filter_compound_list, to_data_array, pretty_list, array_concat, action_list, action_button, action_title, action_description, action_icon
* New Script methods:
** $data.property($key) which takes a string and returns a property from a hash or dataobj.
** $eprint.documents() which returns all the "real" (non-volatile) documents.
* New Script inline math functions: + - / * %
* New EPrints Script datatype: DATA_ARRAY: Represents a list of tuples of [$value, $epscript_type]

==OAI==
* Stateless OAI Interface means no timing-out
* Support for multiple constraints in custom OAI sets

==Unit Tests==
* We have introduced unit-tests to improve both the short and long term quality of our code.
==Metadata Types==
* Counter (incrementing value)
* Timestamp (defaults to the current time)
* UUID
* MetaField::Search now has two properties:
** "namedfields" which is an array ref of field names to search OR
** "namedfields_config" which is the name of a config variable
* MetaField::Search can now be used in any workflow (not hard-coded to editpermfields)
* A captcha pseudo-field based on http://recaptcha.net/
* added "repeat_secret" property to secret fields that will render a confirmation box which is checked with validate()
* Storable (store arbitrary Perl structures - internal use)

==Administration Interface==
* Converted Admin screen into several tabs.
* Improved the BatchEdit interface
* Show a progress bar while records are updated during batch edit

==Editorial Interface==
* Improved "Review" Screen
* The "Review buffer" can now be filtered for better management of large review buffers.
* When an editor provide the "Move to Review" button if there are problems

==User Defined Datasets==
* Allows 3rd party tools to create their own additional datasets
* Suite of interface screens to work with these new datasets

==Command Line Tools==
* Allow eprint ids to be specified for redo_thumbnails

==Export ==
* Added support for OAI-ORE
* Added support for JSONP
* Added support for an 'n' argument to search exports
* Added arguments support to export plugins. Passed by CGI arguments on abstract search or by the --arg option in bin/export

==Abstract Page==
* Now generated with a citation
* Shows an "action list", so plugins can register to appear on this page

==Phrases==
* Primary method of editing phrases is now the web interface
* Added "ref" option to phrases, which will cause the referenced phrase to be used instead - Equivalent to calling the referenced phrase directly

==Views==
* Entire rendering of item lists and menus can be over-ridden by a function

==Misc. Changes ==
* Can now disable a repository through a system configuration setting
* Refactoured DataObj::get_defaults so that you can now specify default values through a "default_value" property
* Most of get_defaults() can now be specified through the metafield spec.
* Can now apply multiple changes to the same field (???? I assume this means metafield?)
* Preference field for users (to store k/v pairs in)

==Key Bugfixes==
* Fixed login page not using phrase for title
* Fixed spurious history objects being created on document upload
* Fixed an HTML insertion bug in the <title> element [Brian D. Gregg]
* Fixed schema errors in uketd_dc and METS/MODS export plugins
* Fixed bug in Compound creation of Set types that squashed the set options
* Fixed order static directories are searched to: repository->theme->system
* Support long values in browse views by using the MD5 of the value,
* Subject inputform component can now be used with singular values
* Fixed bug that is_advertised property on export plugins was being ignored.
* Fixed bug in indexer which meant it didn't index in a round-robin fashion.

= Changes to repository configuration =
We've made some changes to the configuration of a new repository. These will not be automatically applied to your current repositories when upgrading.

== Suggested Changes ==

If upgrading from 3.1 to 3.2, the following changes to your own configuration are suggested to gain the features described above.

* cp lib/defaultcfg/cfg.d/rdf* archives/YOURID/cfg/cfg.d/
* run epadmin recommit
* edit cfg/lang/en/static/index.xpage and add the following to the <xpage:head> section.
<xpage:head>
<link rel="alternate" type="application/rss+xml" title="Items in {phrase('archive_name')}" href="{$config{http_cgiurl}}/latest_tool?output=RSS2"></link>
<link rel="alternate" type="application/atom+xml" title="Items in {phrase('archive_name')}" href="{$config{http_cgiurl}}/latest_tool?output=Atom"></link>
<link rel="alternate" type="application/rdf+xml" title="Repository Summary RDF+XML" href="{$config{http_cgiurl}}/repositoryinfo/RDFXML/devel.rdf"></link>
<link rel="alternate" type="text/n3" title="Repository Summary RDF+N3" href="{$config{http_cgiurl}}/repositoryinfo/RDFN3/devel.n3"></link>
</xpage:head>
(you may already have the rss & atom bits)

* This list has NOT been completed yet, we're working on it!

==New and Altered Config options==

These need documenting and noting which ones we recommend setting/altering when upgrading.

* Set "hide_document_conversion" to hide the Convert link on the document workflow
* Broke up SystemSettings into logically named files
* Can now disable a repository through a system configuration setting
* Moved most of eprint_render.pl into a citation file: summary_page.xml
* Updated defaults views.pl to show current configuration style
* Improved document_upload.pl layout to make it easier to add/remove suffix to mimetype mappings.
* Added URI to EPrint Summary Page
* Added RDF+XML and N3+NT Document formats
* New metafield option: $defaults{render_max_search_values} = 5;
* Added "show_help" option to workflow component to disable collapsing Usage: show_help={always,toggle,never}
* Added config option "cache_max" to limit the cachemap tables used
* user defined datasets
* Made it an option to provide action buttons top and bottom in workflow
**$c->{locking}->{eprint}->{enable} = 1;
**$c->{locking}->{eprint}->{timeout} = 600;
*REST privs
*check registation email callback
*epc:debug, epc:set, changes to epc:foreach
*lots of eprints script functions (see list above)
*views.pl
** "DEFAULT;render_fn=render_view_items_3col_boxes",
** render_menu => "render_view_menu_3col_boxes"
** ranges & variations were introduced in 3.1.? but need documenting.
* Storage plugins
* adding actions to abstract page
* select targz,zip,plain etc. in workflow/upload
# Added support for "input_boxes" property to the workflow, so you can now specify the number of input boxes to show for multiple fields, fixes #3675 [Alexander 'Leo' Bergolth <leo@strike.wu.ac.at>]
# Added --clear option to indexer to clear 'inprogress' events (in case things go wrong)