Difference between revisions of "Autocompletion"

From EPrints Documentation
Jump to: navigation, search
(Anatomy of an input ID)
m (typos corrected, links added, '== romeo ==' removed)
 
(29 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 
{{reference}}
 
{{reference}}
  
(link to how to?)
+
For a how-to, please see [[Autocompletion and Authority Files (Romeo Autocomplete)]] or [[Adding an Auto-Completer to a non-workflow page]]
  
Autocompletion in EPrints 3 consists of serveral stages.
+
Autocompletion in EPrints 3 consists of several stages.
  
* A field in the workflow is configured to say what autocompletion URL to use, plus any additional parameters to pass to the script. This URL must be on the same server (eg. foo.eprints.org) but does not have to be part of the EPrints system.  
+
* A field in the workflow is configured to say what autocompletion URL to use, plus any additional parameters to pass to the script. This URL must be on the same server (e.g. foo.eprints.org) but does not have to be part of the EPrints system.  
 
* The autocomplete script takes the text typed so far (and maybe the additional parameters) and returns a chunk of XML describing possible autocomplete options. This XML consists of a number of rows (how many is up to the script).  
 
* The autocomplete script takes the text typed so far (and maybe the additional parameters) and returns a chunk of XML describing possible autocomplete options. This XML consists of a number of rows (how many is up to the script).  
* Each row contains some HTML to show the person viewing plus a magic <nowiki><ul></nowiki> block which is hidden from display, but is used by the autocomplete javascript to autocomplete the page.
+
* Each row contains some HTML to show the person viewing plus a magic <nowiki><ul></nowiki> block which is hidden from display, but is used by the autocomplete JavaScript to autocomplete the page.
  
 
== Autocomplete Scripts ==
 
== Autocomplete Scripts ==
Line 14: Line 14:
  
 
There are several kinds of autocomplete scripts:
 
There are several kinds of autocomplete scripts:
* thoses that just use the existing data in your repository (these are dead easy as they work out of the box)
+
* those that just use the existing data in your repository (these are dead easy as they work out of the box).
* ones which use a file which you place in your repositories cfg/autocomplete/ directory.
+
* ones which use a file which you place in your repository's cfg/autocomplete/ directory.
 
* more clever ones.
 
* more clever ones.
  
 
You may be able to find new autocomplete scripts and authority files on http://files.eprints.org/
 
You may be able to find new autocomplete scripts and authority files on http://files.eprints.org/
  
Scripts are in (rough) order of complexity to use...
+
Scripts are in (rough) order of complexity to use:
  
 
=== journal_by_name ===
 
=== journal_by_name ===
Line 30: Line 30:
 
As above, but attached to the ISSN field.
 
As above, but attached to the ISSN field.
  
== event_by_name ===
+
=== event_by_name ===
  
 
Similar to journal_by_name. Is attached to the event_title field and autocompletes from existing repository data. If they are in the same (multi) input component it will also try and autocomplete event_location, event_dates and event_type.
 
Similar to journal_by_name. Is attached to the event_title field and autocompletes from existing repository data. If they are in the same (multi) input component it will also try and autocomplete event_location, event_dates and event_type.
  
== name ==
+
=== name ===
  
Attached to a multiple compound name/id field (eg. creators) looks up the name in the existing list in the repository. Can match on any id or given or family. Populates all parts of the current row it can.  
+
Attached to a multiple compound name/id field (e.g. creators) looks up the name in the existing list in the repository. Can match on any id or given or family name. Populates all parts of the current row it can.  
  
== title_duplicates ==
+
=== title_duplicates ===
  
 
This is a slightly odd script as it doesn't actually provide any autocomplete data. What it does is search the list of existing titles to see if there is a match. It only searches if there are 5 or more characters entered so far.
 
This is a slightly odd script as it doesn't actually provide any autocomplete data. What it does is search the list of existing titles to see if there is a match. It only searches if there are 5 or more characters entered so far.
  
If it finds any matches it lists them with a warning that they might be a problem, but does not assist autocompletion. If many matches are made then a short title only is shown, if the list is only 4 or lest then a full citation is shown.
+
If it finds any matches it lists them with a warning that they might be a problem, but does not assist autocompletion. If many matches are made then a short title only is shown, if the list is only 4 or fewer then a full citation is shown.
  
 
This is set to "on" by default in the hope that it will reduce duplicate submissions.
 
This is set to "on" by default in the hope that it will reduce duplicate submissions.
  
== simple_file ==
+
=== simple_file ===
  
File needs an additional parameter to be passed to it. This is configured in the workflow. This parameter is the name of a file in the cfg/autocompete directory. This file contains a list of values which are searched (case insensitively) and matches returned. A second parameter of "mode=prefix" can be set to only match values which start with the text being typed, rather than contain it.
+
File needs an additional parameter to be passed to it. This is configured in the workflow. This parameter is the name of a file in the cfg/autocomplete directory. This file contains a list of values which are searched (case insensitively) and matches returned. A second parameter of "mode=prefix" can be set to only match values which start with the text being typed, rather than contain it. Add a second and subsequent parameter by using a semi-colon ( ; ) as delimiter (tested in 3.3.10)
  
== simple_sql ==
+
=== simple_sql ===
  
 
Similar to simple_file but gets its values from a database table.
 
Similar to simple_file but gets its values from a database table.
  
The table must be in the eprints database used by this repository and start with "ac_". The script needs a param. passed from workflow to indicate the name of the table WITHOUT the ac_ prefix. Eg. if the table was "ac_badgers" the parameter would be "table=badgers". The only field used is "value" which works like the lines in the text file. If you want this to be blindingly fast you can make sure "value" is indexed, and set mode=prefix. With those set autocompleting from a dictionary of half a million words worked cheerfully.
+
The table must be in the eprints database used by this repository and start with "ac_". The script needs a param. passed from workflow to indicate the name of the table WITHOUT the ac_ prefix. E.g. if the table was "ac_badgers" the parameter would be "table=badgers". The only field used is "value" which works like the lines in the text file. If you want this to be blindingly fast you can make sure "value" is indexed, and set mode=prefix. With those set autocompleting from a dictionary of half a million words worked cheerfully.
  
== romeo ==
+
=== url_name_value ===
  
(not included in 3.0, expected in 3.1) This script uses the EPrints/Romeo data to provide journal autocomplete data. Should be attached to the publication field. This is almost identical to file, but inserts the required Powered by Sherpa note.
+
This works like simple_sql except for the fact it uses three columns. url, name and value. It searches and autocompletes using value, but the human-readable description is supplied by "name" and if url is set then a (more info) link is shown. The link opens a new window to avoid mid-form trauma.
 +
 
 +
=== file ===
 +
 
 +
This is for more complex autocompletion authority files. It works like simple_file except that the file format is more complicated. For a how-to, please see [https://wiki.eprints.org/w/Autocompletion_and_Authority_Files_(Romeo_Autocomplete)#Creating_Your_Own_Authority_File_.28complex.29 Autocompletion and Authority Files (Romeo Autocomplete)]
 +
 
 +
The file consists of lines which contain:
 +
* a value to search, (e.g. "African Journal of Agricultural Research")
 +
* a tab
 +
* a <nowiki><li></nowiki> autocomplete chunk. (with no line breaks) e.g.  
 +
<nowiki> <li style='border-right: solid 50px #30FF30' ></nowiki>
 +
 
 +
=== external source ===
  
== url_name_value ==
+
This takes all the ideas above, and extends them to make an API call to an external data source.  This has the advantage that you are always referring to the authoritative source, but the disadvantage that you are reliant on both the network being up and the external source being available.
  
This works like simple_sql except for the fact it uses three columns. url, name and value. It searches and autocompletes using value, but the human-readable description is supplied by "name" and if url is set then a (more info) link is shown. The link opens a new window to avoid mid-form trauma.
+
It breaks down into two parts:
 +
* the autocompleter call in the web page
 +
* the script being called
  
== file ==
+
For an example, here is one way to query the RoMEO data directly:
  
This is for more complex autocompletion authority files. It works like simple_file except that the file format is more complicated.
+
First, set the autocompleter in the eprints workflow:
 +
<pre>
 +
    <component type="Field::Multi">
 +
      <title>Article Publication Details</title>
 +
        <field ref="publication" input_lookup_url="{$config{perl_url}}/get_journals" />
 +
        <field ref="publisher" />
 +
        <field ref="issn" />
 +
      </component>
 +
</pre>
  
The file constists of lines which contan:  
+
Next have the script:
* a value to search, (eg. "African Journal of Agricultural Research")
+
<pre>
* a tab
+
use strict;
* a <li> autocomplete chunk. (with no line breaks) eg.
+
use HTTP::Request;
<nowiki> <li style='border-right: solid 50px #30FF30' >&quot;African Journal of Agricultural
+
use LWP::UserAgent;
Research&quot; published by &quot;Academic Publishers&quot;<br /><small>(a Green
+
use XML::Twig;
publisher)</small>ISSN: 1991-637X<ul><li id="for:value:component:_publication">African
 
Journal of Agricultural Research</li><li id="for:value:component:_publisher">Academic
 
Publishers</li><li id="for:value:component:_issn">1991-637X</li></ul></li></nowiki>
 
  
See below for more information on the meaning of this arcane chunk!
+
use Data::Dumper;
 +
use EPrints;
  
== sql ==
+
my $journal_data = {};
  
As for simple_sql except that a second column named "xml" is used to provide the actual results returned (value is still searched).
+
sub urldecode{
 +
  my ($url) = @_;
 +
  $url =~ s/%([0-9a-f][0-9a-f])/pack("C",hex($1))/egi;
 +
  $url =~ s/\x2B/ /; # swap '+' for ' '
 +
  return $url;
 +
}
  
The xml column contains data in the autocomplete <nowiki><li></nowiki> format described below.
+
# XML::Twig's routine for dealing with a journal entry
 +
sub process_journal {
 +
  my ( $twig, $journal ) = @_;
  
= Making a custom script =
+
  # get the components
 +
  my $title = urldecode( $journal->first_child('jtitle')->text );
  
Autocompletion scripts are configured to eprint fields within the workflow. If the field is multiple then the same script is attached to each input row.
+
  my $zetoc = urldecode( $journal->first_child('zetocpub')->text )
 +
                  if $journal->first_child('zetocpub');
 +
  my $romeo = urldecode( $journal->first_child('romeopub')->text )
 +
                  if $journal->first_child('romeopub');
 +
  my $issn  = urldecode( $journal->first_child('issn')->text )
 +
                  if $journal->first_child('issn');
  
The only parameter you need to look at is "q" which contains the text being autocompleted. For simple fields (eg. text), that is ones which only have one input-box per value. Not names, compound or pagerange etc.. the response should be of the format:
+
  my $publisher = $romeo;
<nowiki>
+
  $publisher = $zetoc if (not $publisher && $zetoc);
<?xml version="1.0" encoding="UTF-8" ?>
+
 
<ul>
+
  # build a lub of html based on the components
   <li>Human Friendly Text <ul><li id='for:value:relative:'>text-to-insert</li></ul><li>
+
  my $html .= "<li>$title";
   <li>Human Friendly Text <ul><li id='for:value:relative:'>text-to-insert</li></ul><li>
+
  $html .= "<br />published by $publisher" if $publisher;
   <li>Human Friendly Text <ul><li id='for:value:relative:'>text-to-insert</li></ul><li>
+
 
   <li>Human Friendly Text <ul><li id='for:value:relative:'>text-to-insert</li></ul><li>
+
  $html .= "<ul>";
   etc.
+
   if ($title) {
</ul></nowiki>
+
      $html .= "<li id='for:value:component:_publication'>$title</li>";
 +
  }
 +
   if ($publisher) {
 +
      $html .= "<li id='for:value:component:_publisher'>$publisher</li>";
 +
  }
 +
   if ($issn) {
 +
    $html .= "<li id='for:value:component:_issn'>$issn</li>";
 +
  }
 +
   $html .= "</ul></li>\n";
 +
  warn "\n\n$html\n\n";
 +
   # save the html
 +
  $journal_data->{$title} = $html;
  
== Other useful CGI parameters ==
+
  1;
 +
} ## end process_journal
  
All parts of the field (or field row in multiple fields) get sent as CGI parameters. The name of these parameters is the ID of the HTML input element itself, but with the relative prefix removed (phew!).
+
# get a list of journals that match the query
 +
sub get_journals {
 +
  my $journal = shift;
 +
  my @html = ();
  
Simple example: title field. One single value. It's not relevant, just use "q".
+
  if ($journal) {
 +
    return ("<ul><li>keep typing....</li></ul>") if (length($journal) < 3);
  
More complex example: pagerange field. While you were typing in the "to" box (the second one). It would send "?q=45&_from=12&_to=45". Obviously the numbers are made up. q= will always be the same as one of the values.
+
    $journal =~ s/\s+/\+/;
 +
    my $query = "http://www.sherpa.ac.uk/romeoapi.php?qtype=starts&jtitle=$journal";
  
Even more complex example: creators field. Which is a multiple compound field. Parts sent would be q, _id, _name_given and _name_family.
+
    my $request = HTTP::Request->new( GET => "$query" );
  
== Anatomy of an input ID ==
+
    my $ua = LWP::UserAgent->new();
 +
    my $response = $ua->request($request);
 +
    my $content = $response->content();
  
This is an explanation of how the id="" tags are generated for individual input elements. (ie. a &lt;input&gt;) in a HTML page generated by the EPrints workflow system.
+
    my $twig = XML::Twig->new(
 +
                      'keep_encoding' => 1,
 +
                      'TwigRoots' => { 'journals' => 1 },
 +
                      'TwigHandlers' => { 'journal' => \&process_journal, }
 +
                      );
 +
    $twig->parse($content);
 +
    if (scalar keys %{$journal_data}) {
 +
      push @html, "<ul class='journals'>\n";
 +
      foreach my $title (sort keys %{$journal_data}) {
 +
        push @html, "$journal_data->{$title}\n";
 +
      } ## end of  foreach my $title (sort keys %{$journal_data})
 +
      push @html, "</ul>\n";
 +
    } ## end of if (scalar keys %{$journal_data}) ...
 +
  } else {
 +
    push @html, "<!-- No journal name supplied -->\n";
 +
  }
  
This is useful to understand if you're doing really complex stuff.
+
  return (join "\n", @html)
  
First of all, some jargon:
+
} ## end get_journals
  
* The '''workflow''' contains '''pages'''
+
my $session = EPrints::Session->new();
* A '''page''' is rendered as a web page. It is displayed containing one or more '''input components'''
 
* ''input components'' allow the inputing of zero or more ''eprint fields''.
 
* '''eprint fields''' may be multiple. If so they contain multiple '''rows''' of input. Otherwise they contain a single '''row'''
 
* a '''row''' in a compound field consists of all the '''input elements''' of all the sub-fields.
 
* a '''row''' in a non-compound field consists of the fields own '''input elements'''
 
* the '''input elements''' are the actual HTML bits needed to input the data (eg. 3 for date, 2 for page range, one for a title).
 
  
The id is constructed thus...
+
# we need the send an initial content-type
 +
print <<END;
 +
<?xml version="1.0" encoding="UTF-8" ?>
  
* EPrints assigns a prefix to the component. This is just a string eprints assigns when generating the page, and should not be relied on.
+
END
** eg. id3
 
** This is the '''component prefix'''
 
* For a field in a coponent, add "_" and the name of the field.
 
** eg. id3_creators
 
** This is the '''field prefix'''
 
* If this is a multiple field, add "_" and the number of the row. Otherwise add nothing.
 
** eg. id3_creators_2
 
** eg. id3_creators_3
 
** eg. id4_title (not multiple)
 
** eg. id5_pagerange
 
** This is the '''row prefix'''
 
* If this is a compound field, then add "_" and the sub-field id we're inputing.
 
** eg. id3_creators_2_name
 
** eg. id4_title  (not compound)
 
* If this (sub)field has more than one input component, add the suffix for that input component (see below).
 
* eg. id3_creators_2_name_given
 
* eg. id4_title
 
* eg. id5_pagerange_from
 
  
=== Field types with more than one input component ===
+
# then we send the fragment of html for the autocompleter
 +
print get_journals( lc $session->param( "q" ) );
  
date: _day, _month, _year
+
$session->terminate;
name: _given, _family (and possibly _lineage and _honourific)
+
</pre>
pagerange: _from, _to
 
time: _day, _month, _year, _second, _minute, _hour
 

Latest revision as of 17:29, 17 September 2019

EPrints 3 Reference: Directory Structure - Metadata Fields - Repository Configuration - XML Config Files - XML Export Format - EPrints data structure - Core API - Data Objects

For a how-to, please see Autocompletion and Authority Files (Romeo Autocomplete) or Adding an Auto-Completer to a non-workflow page

Autocompletion in EPrints 3 consists of several stages.

  • A field in the workflow is configured to say what autocompletion URL to use, plus any additional parameters to pass to the script. This URL must be on the same server (e.g. foo.eprints.org) but does not have to be part of the EPrints system.
  • The autocomplete script takes the text typed so far (and maybe the additional parameters) and returns a chunk of XML describing possible autocomplete options. This XML consists of a number of rows (how many is up to the script).
  • Each row contains some HTML to show the person viewing plus a magic <ul> block which is hidden from display, but is used by the autocomplete JavaScript to autocomplete the page.

Autocomplete Scripts

EPrints autocomplete scripts live in /opt/eprints3/cgi/users/lookup/ you can add your own here, or maybe elsewhere if, for example, you needed to use PHP.

There are several kinds of autocomplete scripts:

  • those that just use the existing data in your repository (these are dead easy as they work out of the box).
  • ones which use a file which you place in your repository's cfg/autocomplete/ directory.
  • more clever ones.

You may be able to find new autocomplete scripts and authority files on http://files.eprints.org/

Scripts are in (rough) order of complexity to use:

journal_by_name

Can only be used on the "publication" field. Looks up the publication in the existing publications in the repository and autocompletes the publication. If ISSN and/or publisher exist in the same input component as the journal field they will also be completed if data is available.

journal_by_issn

As above, but attached to the ISSN field.

event_by_name

Similar to journal_by_name. Is attached to the event_title field and autocompletes from existing repository data. If they are in the same (multi) input component it will also try and autocomplete event_location, event_dates and event_type.

name

Attached to a multiple compound name/id field (e.g. creators) looks up the name in the existing list in the repository. Can match on any id or given or family name. Populates all parts of the current row it can.

title_duplicates

This is a slightly odd script as it doesn't actually provide any autocomplete data. What it does is search the list of existing titles to see if there is a match. It only searches if there are 5 or more characters entered so far.

If it finds any matches it lists them with a warning that they might be a problem, but does not assist autocompletion. If many matches are made then a short title only is shown, if the list is only 4 or fewer then a full citation is shown.

This is set to "on" by default in the hope that it will reduce duplicate submissions.

simple_file

File needs an additional parameter to be passed to it. This is configured in the workflow. This parameter is the name of a file in the cfg/autocomplete directory. This file contains a list of values which are searched (case insensitively) and matches returned. A second parameter of "mode=prefix" can be set to only match values which start with the text being typed, rather than contain it. Add a second and subsequent parameter by using a semi-colon ( ; ) as delimiter (tested in 3.3.10)

simple_sql

Similar to simple_file but gets its values from a database table.

The table must be in the eprints database used by this repository and start with "ac_". The script needs a param. passed from workflow to indicate the name of the table WITHOUT the ac_ prefix. E.g. if the table was "ac_badgers" the parameter would be "table=badgers". The only field used is "value" which works like the lines in the text file. If you want this to be blindingly fast you can make sure "value" is indexed, and set mode=prefix. With those set autocompleting from a dictionary of half a million words worked cheerfully.

url_name_value

This works like simple_sql except for the fact it uses three columns. url, name and value. It searches and autocompletes using value, but the human-readable description is supplied by "name" and if url is set then a (more info) link is shown. The link opens a new window to avoid mid-form trauma.

file

This is for more complex autocompletion authority files. It works like simple_file except that the file format is more complicated. For a how-to, please see Autocompletion and Authority Files (Romeo Autocomplete)

The file consists of lines which contain:

  • a value to search, (e.g. "African Journal of Agricultural Research")
  • a tab
  • a <li> autocomplete chunk. (with no line breaks) e.g.
 <li style='border-right: solid 50px #30FF30' >

external source

This takes all the ideas above, and extends them to make an API call to an external data source. This has the advantage that you are always referring to the authoritative source, but the disadvantage that you are reliant on both the network being up and the external source being available.

It breaks down into two parts:

  • the autocompleter call in the web page
  • the script being called

For an example, here is one way to query the RoMEO data directly:

First, set the autocompleter in the eprints workflow:

     <component type="Field::Multi">
      <title>Article Publication Details</title>
        <field ref="publication" input_lookup_url="{$config{perl_url}}/get_journals" />
        <field ref="publisher" />
        <field ref="issn" />
      </component>

Next have the script:

use strict;
use HTTP::Request;
use LWP::UserAgent;
use XML::Twig;

use Data::Dumper;
use EPrints;

my $journal_data = {};

sub urldecode{
  my ($url) = @_;
  $url =~ s/%([0-9a-f][0-9a-f])/pack("C",hex($1))/egi;
  $url =~ s/\x2B/ /; # swap '+' for ' '
  return $url;
}

# XML::Twig's routine for dealing with a journal entry
sub process_journal {
  my ( $twig, $journal ) = @_;

  # get the components
  my $title = urldecode( $journal->first_child('jtitle')->text );

  my $zetoc = urldecode( $journal->first_child('zetocpub')->text ) 
                  if $journal->first_child('zetocpub');
  my $romeo = urldecode( $journal->first_child('romeopub')->text )
                  if $journal->first_child('romeopub');
  my $issn  = urldecode( $journal->first_child('issn')->text )
                  if $journal->first_child('issn');

  my $publisher = $romeo;
  $publisher = $zetoc if (not $publisher && $zetoc);
  
  # build a lub of html based on the components
  my $html .= "<li>$title";
  $html .= "<br />published by $publisher" if $publisher;
  
  $html .= "<ul>";
  if ($title) {
      $html .= "<li id='for:value:component:_publication'>$title</li>";
  }
  if ($publisher) {
      $html .= "<li id='for:value:component:_publisher'>$publisher</li>";
  }
  if ($issn) {
    $html .= "<li id='for:value:component:_issn'>$issn</li>";
  }
  $html .= "</ul></li>\n";
  warn "\n\n$html\n\n";
  # save the html
  $journal_data->{$title} = $html;

  1; 
} ## end process_journal

# get a list of journals that match the query
sub get_journals {
  my $journal = shift;
  my @html = ();

  if ($journal) {
    return ("<ul><li>keep typing....</li></ul>") if (length($journal) < 3);

    $journal =~ s/\s+/\+/;
    my $query = "http://www.sherpa.ac.uk/romeoapi.php?qtype=starts&jtitle=$journal";

    my $request = HTTP::Request->new( GET => "$query" );

    my $ua = LWP::UserAgent->new();
    my $response = $ua->request($request);
    my $content = $response->content();

    my $twig = XML::Twig->new(
                       'keep_encoding' => 1,
                       'TwigRoots' => { 'journals' => 1 },
                       'TwigHandlers' => { 'journal' => \&process_journal, }
                      );
    $twig->parse($content);
    if (scalar keys %{$journal_data}) {
      push @html, "<ul class='journals'>\n";
      foreach my $title (sort keys %{$journal_data}) {
        push @html, "$journal_data->{$title}\n";
      } ## end of  foreach my $title (sort keys %{$journal_data})
      push @html, "</ul>\n";
    } ## end of if (scalar keys %{$journal_data}) ...
  } else {
    push @html, "<!-- No journal name supplied -->\n";
  }

  return (join "\n", @html)

} ## end get_journals

my $session = EPrints::Session->new();

# we need the send an initial content-type
print <<END;
<?xml version="1.0" encoding="UTF-8" ?>

END

# then we send the fragment of html for the autocompleter
print get_journals( lc $session->param( "q" ) );

$session->terminate;