Editing IRStats2 (section)

=API=

This section presents a few examples on how to get data out of IRStats2 for embedding data on pages or for re-use in analysis scripts (for instance).

There are two ways to get data out:

#From a script: this is the real API, using PERL
#From an Ajax request: this is usually to embed data on pages

==Core concepts==

===Datatype===
Datatype refers to which data to provide with IRStats2 able to process any of data on your repository. The typical use of IRStats2 is however for usage statistics so this is the main dataset, but data on deposits, open access, full text (etc) are also processed. Some repositories even include data from Scopus (citation counts).

Main datatypes:

*downloads: good old download statistics - downloads of full-text documents
*views: number of hits on the summary page (of a publication)
*deposits: number of publications deposited
*doc_access: provides 4 metrics (full_text, no_full_text, open_access and no_open_access) used for computing percentages of Open Access and Full-Text *documents in the repository
*doc_format: MIME type of full-texts
*history: analysis of the "history" dataset - this provides information on when publications were created, edited, made live, deleted etc.
*referrer: information on how site visitors got to the repository (e.g. from Google, internal uni pages, etc)
*search_terms: if coming from a search site (or the internal EPrints search) which words were used to get to the publication
*browser: which browser visitors used on the repository

===Sets===
By default, IRStats2 returns data over the entire repository, i.e. the entire set of eprints is assumed. You can however restrict which "set" to use: the publications of an author, of a university division, of a subject, etc.

===Dates and ranges===
You can also restrict by dates or by a range. By default, all the stats are returned without any dates restrictions.

Dates can be set as YYYYMMDD or YYYY-MM-DD or YYYY/MM/DD (e.g. 20170101, 2017-11-04 etc). Dates is a hash containing two keys: from and to (either may be omitted to say: from that particular date, or up to that particular date).

Ranges follow a %d%c format and the upper limit is "now" or "today", for instance:

*6m: over the past 6 months
*12d: over the past 12 days
*3y: over the past 3 years

Only "m" (months), "d" (days) or "y" (years) may be used. 12m is the same as 1y.

===Groupings===
This tells IRStats2 how to group data and is generally only used for things like "give me the TOP eprints", "give me the TOP authors".

So having a "grouping" set to "eprint" means the top eprints. If set to "authors", the top authors etc. The grouping must be a valid set except for when it equals to "eprint".

===Misc===
It is possible to limit the amount of records being returned (for when this is relevant: if you want the top downloads, since the beginning of time, then you'd only get one data row back, which is that count). But for queries which ask for, say, the top authors, it is then interesting to be able to get only the first 10 authors. 10 here is the limit.

It is also possible to ask IRStats2 to return certain data field in queries. For top eprints, you generally want the "eprintid" field. To draw timeline graphs (eg. evolution of downloads over-time), you'd want the "datestamp" field. More examples are illustrated below.

==Data from scripts==

===Main API===

<pre>
# get the IRStats2 handler, required to query IRStats2
my $handler = $repo->plugin( "Stats::Handler" );

# ask IRStats2 to show debug statements (SQL queries)
$handler->debug(1);

# Create a Context object
my $ctx = $handler->context( { datatype: "downloads" } );

# Retrieve data rows
my $data = $handler->data( $ctx )->select();

# How many rows returned:
printf "I got %d data rows back\n", $data->count;

# Get stats for divisions "uos-ecs":
$ctx->set( { set_name => 'divisions', set_value => 'uos-ecs' } );

# Get stats over the last 6 months:
$ctx->dates( { range => '6m' } );

# Get stats between 1st January 2012 and 31st March 2012:
$ctx->dates( { from => '20120101', to => '20120331' } );

# Data may be exported (see Stats/Export/ for a list of currently supported plug-ins):
my $export = $repo->plugin( "Stats::Export::CSV" );
$data->export( { export_plugin => $export } );
</pre>

===Full Examples===
Actually those are not really full examples. They assume you can write the beginning of a PERL script and that you have already instantiated the Stats Handler (cf. above) as $handler.

<pre>
# How many downloads in total over the entire repository
my $ctx = $handler->context( { datatype => "downloads" } );
printf "I got %d downloads\n", $handler->data( $ctx )->select->sum_all;
</pre>

<pre>
# How many downloads in 2013 over the entire repository
my $ctx = $handler->context( { datatype => "downloads", range => "2013" } );
printf "I got %d downloads\n", $handler->data( $ctx )->select->sum_all;
</pre>

<pre>
# The top 5 EPrints over the entire repository
my $ctx = $handler->context( { grouping => "eprint", datatype => "downloads" } );

my $stats = $handler->data( $ctx )->select( fields => ["eprintid"], limit => 5 );

foreach( @{ $stats->data } )
{
        printf "EPrint %d got %d downloads\n", $_->{eprintid}, $_->{count};
}
</pre>

<pre>
# The top 10 Subjects (let's assume LoC) for deposits (not downloads!!)
my $ctx = $handler->context( { set_name => "subjects", datatype => "deposits" } );

my $stats = $handler->data( $ctx )->select( fields => ["set_value"], limit => 10 );

my $i = 1;
foreach( @{ $stats->data } )
{
        printf "%d) %s with %d items deposited\n", $i++, $_->{set_value}, $_->{count};
}
</pre>

<pre>
# The top 5 downloaded EPrints for LoC Subject "D1"
my $ctx = $handler->context( { set_name => "subjects", set_value => 'D1', datatype => "downloads" } );

my $stats = $handler->data( $ctx )->select( fields => ["eprintid"], limit => 5 );

my $i = 1;
foreach( @{ $stats->data } )
{
        printf "%d) EPrintd %d with %d downloads\n", $i++, $_->{eprintid}, $_->{count};
}
</pre>

==Embedding data==
This is similar to retrieving data from scripts (cf. section above) but with a few extra options:

*"view": the name of the Stats::View plug-in which will draw the requested stuff (a Table? a Graph? etc.)
*"container_id": the DOM element "id", where the drawn stuff will be inserted on the page (if the Ajax callback is successful)

Then there exists a number of options proper to each View plug-in. See the provided examples below.

===Graphs===
The typical example is to embed the global downloads graph. This is usually the first displayed item on the IRStats2 main report page (/cgi/stats/report).

<pre>
<!--
This will basically insert the downloads graph into the "mygraph" div element. Note that it's using the supplied "irstats2_googlegraph" CSS class.

      Context options:
      - irs2report: What type of report/data do you want to display (e.g. "main", "deposits", "requests")
      - range: What date range show this report cover (e.g. "_ALL_" is across all dates, "1y" is the last year "1m" is the last month, etc.) 
      - datatype: What type of data is being displayed (e.g. "downloads", "views", "deposits", "countries")
      - datafilter: What attribute on which to filter the data (e.g. "archive")
      - set_name: A set on which to filter (e.g. "authors", "type", "divisions", "subjects", "eprintid")
      - set_value: A value in the set on which to filterm (e.g for an author the hash of their name such as "c6edf1ce30d927a9a005e4b272ead4b4" for a division or subject, their subject ID, for eprint type, its type ID)
      
      Graph options:
      - graph_type: either "column" or "area"
      - show_average: either 1 or 0 - displays the average graph'
      - date_resolution: either "year", "month" or "day" - groups data by year, month or day (be careful: selecting day may generate LOTS of data points)
-->

<div id="mygraph" class="irstats2_googlegraph"/>

<script type="text/javascript">
document.observe("dom:loaded",function(){
         new EPJS_Stats_GoogleGraph( { 
                'context': {'irs2report': 'main', 'range': '_ALL_', 'datatype': 'downloads'},
                'options': { 'graph_type': 'column', 'container_id': 'mygraph', 'view': 'Google::Graph', 'show_average': '1', 'date_resolution': 'month' } 
        });
});
</script>
</pre>

===Tables===
The example below displays the top 10 downloaded eprints in the repository.

<pre>
<!--
This will insert the top table into the "mytable" div element. Note that it's using the supplied
     "irstats2_table" CSS class.
      
      Context options:
      - irs2report: What type of report/data do you want to display (e.g. "most_popular_eprints", "most_popular_authors")
      - range: What date range show this report cover (e.g. "_ALL_" is across all dates, "1y" is the last year "1m" is the last month, etc.) 
      - datatype: What type of data is being displayed (e.g. "downloads", "views")
      - set_name: A set on which to filter (e.g. "authors", "type", "divisions", "subjects", "eprintid")
      - set_value: A value in the set on which to filterm (e.g for an author the hash of their name such as "c6edf1ce30d927a9a005e4b272ead4b4" for a division or subject, their subject ID, for eprint type, its type ID)

      Table options:
      - top: the top "thing" to display - similar to the "grouping" parameter when using scripts
      - limit: the max number of items to retrieve
      - show_count: 1 or 0 - display the counts or not
      - show_order: 1 or 0 - display the ordering (1,2,3 ...) or not
      - show_more: 1 or 0 - shows the "show more" options or not (to retrieve more results)
      - human_display: 1 or 0 - separate 1000 with a comma (as done in English): 10000 becomes 10,000
-->
<div id="mytable" class="irstats_table"/>

<script type="text/javascript">
document.observe( "dom:loaded", function() {

        new EPJS_Stats_Table( {
                'context': { 'irs2report': 'most_popular_eprints', 'range': '_ALL_', 'datatype': 'downloads'},
                'options': { 'container_id': 'mytable', 'top': 'eprint', 'view': 'Table', 'limit': '5' }   
        } );

});
</script>
</pre>

===Misc===
Graphs and Tables are the most common displays, but there are a few other ones to explore. The javascript classes are in 90_irstats2.js and the associated PERL Class in Stats/View/

*GoogleSpark: similar to GoogleGraph but shows a sparkline instead (which is essentially a tiny graph).
*GoogleGeoChart: country map
*GooglePieChart: a pie chart
*Counter: a simple counter (for instance to show the download count for your repository).

The View prefixed by "Google" means that they are rendered by the Google Chart Javascript library. Important note: no data is sent to Google! The data is, instead, drawn by the browser client using SVG.

== Example: Creating a processor for citation statistics ==

IRStats2 can evaluate any field or combination of fields in the above mentioned datasets. For this, new datatypes and associated Stats::Processor modules must be created.

A simple case is statistics on citation count of publications, since the datatype of the associated field is a scalar.

Citation counts can be harvested using the [http://files.eprints.org/815/  Citation count dataset and import plugins] from Queensland University of Technology. Upon following the installation procedure, the fields scopus_impact, wos_impact and gscholar_impact are created in the EPrint table.

=== Modifying an existing Processor module ===

We create now a Processor module that evaluates the scopus_impact field and provides a scopus_citations datatype. 

First, we inspect the Stats::Processor::EPrint Processor modules to find out which Processor module is best suited for adapting. A good candidate is lib/plugins/EPrints/Plugin/Stats/Processor/EPrint/DocumentAccess.pm. We copy it to archives/{repo}/cfg/plugins/EPrints/Plugin/Stats/Process/EPrint/ScopusCitations.pm and modify it as follows:

<pre>
package EPrints::Plugin::Stats::Processor::EPrint::ScopusCitations;

our @ISA = qw/ EPrints::Plugin::Stats::Processor /;

use strict;

sub new
{
	my( $class, %params ) = @_;
	my $self = $class->SUPER::new( %params );

#  provide the name of the datatype
	$self->{provides} = [ "scopus_citations" ];

	$self->{disable} = 0;

	return $self;
}


sub process_record
{
	my ($self, $eprint ) = @_;

	my $epid = $eprint->get_id;
	return unless( defined $epid );

	my $status = $eprint->get_value( "eprint_status" );
	unless( defined $status ) 
	{
##		print STDERR "IRStats2: warning - status not set for eprint=".$eprint->get_id."\n";
		return;
	}

	return unless( $status eq 'archive' );

	my $datestamp = $eprint->get_value( "datestamp" ) || $eprint->get_value( "lastmod" );

	my $date = $self->parse_datestamp( $self->{session}, $datestamp );

	my $year = $date->{year};
	my $month = $date->{month};
	my $day = $date->{day};

# get the citation count
	my $scopus_citation_count = $eprint->get_value( "scopus_impact" );

# store the citation count per eprint id
	if (defined $scopus_citation_count)
	{
		$self->{cache}->{"$year$month$day"}->{$epid}->{scopus} = $scopus_citation_count;
	}
}

1;
</pre>

Next, we need to enable the new Processor plugin in our IRStats2 configuration file. In the Bazaar config section, add the following line:

<pre>
$c->{plugins}{"Stats::Processor::EPrint::ScopusCitations"}{params}{disable} = 0;
</pre>

In the Reports section, we can add now a new citation report that uses the scopus_citations datatype:

<pre>
        citations => {
                items => [
                { plugin => 'ReportHeader' },
                {
                        plugin => 'Grid',
                        options => {
                                items => [
                                {
                                        plugin => 'Table',
                                        datatype => 'scopus_citations',
                                        options => {
                                                limit => 10,
                                                top => 'eprint',
                                                title_phrase => 'top_scopus_citations',
                                        },
                                },]
                        },
                },
                {
                        plugin => 'Grid',
                        options => {
                                items => [
                                {
                                        plugin => 'Table',
                                        datatype => 'scopus_citations',
                                        options => {
                                                limit => 10,
                                                top => 'authors',
                                                title_phrase => 'top_scopus_citations_authors'
                                        }
                                },]
                        },
                },
                ],
                category => 'advanced',
        },
</pre>

Also, the phrases in irstats2.xml must be completed accordingly.


=== Generalization ===

For each citation datum, a processor module is required. We see, that these will differ only slightly,
namely in the datatype declaration in the new() method, and in the processing of the citation datum field 
in the last part of the process_record() method. Therefore, we can generalize the processor by providing 
an abstract value tracker module. It does the following:

In the new() method, it provides an abstract value tracker datatype. Note that the plugin is disabled.
In the process_record() method, it makes reference to the field that is processed and a value_id to where the value will be stored.

<pre>
package EPrints::Plugin::Stats::Processor::EPrint::AbstractValueTracker;

our @ISA = qw/ EPrints::Plugin::Stats::Processor /;

use strict;

sub new
{
	my( $class, %params ) = @_;
	my $self = $class->SUPER::new( %params );
	
	$self->{provides} = [ "abstract_value_tracker" ];
	$self->{disable} = 1;
	
	$self->{field_id} = 'DEFINE IN SUBCLASS';
	$self->{value_id} = 'DEFINE IN SUBCLASS';
	
	return $self;
}


sub process_record
{
	my ($self, $eprint ) = @_;
	
	my $epid = $eprint->get_id;
	return unless( defined $epid );

	my $status = $eprint->get_value( "eprint_status" );
	unless( defined $status )
	{
#		print STDERR "IRStats2: warning - status not set for eprint=".$eprint->get_id."\n";
		return;
	}

	return unless( $status eq 'archive' );

	my $datestamp = $eprint->get_value( "datestamp" ) || $eprint->get_value( "lastmod" );

	my $date = $self->parse_datestamp( $self->{session}, $datestamp );

	my $year = $date->{year};
	my $month = $date->{month};
	my $day = $date->{day};

	my $value = $eprint->value($self->{field_id});

	if (defined $value)
	{
		$self->{cache}->{"$year$month$day"}->{$epid}->{$self->{value_id}} = $value;
    }
}

1;
</pre>


Now, the declaration of new scalar datatypes reduces to the task to defining a subclass for each datatype, 
and to declaring the assocation between the datatype and the field to be processed in the IRStats2 configuration:

<pre>
package EPrints::Plugin::Stats::Processor::EPrint::ScopusCitations;

use EPrints::Plugin::Stats::Processor::EPrint::AbstractValueTracker;
our @ISA = qw/ EPrints::Plugin::Stats::Processor::EPrint::AbstractValueTracker /;

use strict;

sub new
{
	my( $class, %params ) = @_;
	my $self = $class->SUPER::new( %params );
	my $repo = $self->repository;

	$self->{provides} = [ "scopus_citations" ];

	$self->{disable} = 0;
	
	$self->{field_id} = $repo->config('irstats2','scopus_citations','field_id');
	$self->{value_id} = 'scopus';

	return $self;
}

1;
</pre>

<pre>
############
# Datatypes
############
# Declare as $c->{irstats2}->{datatypename}->{field_id} = 'fieldname';

$c->{irstats2}->{scopus_citations}->{field_id} = 'scopus_impact';
</pre>

=== Specific requirements by Scopus and Web of Science ===

If you have a valid API key to the Scopus and Web of Science APIs, it is allowed by both database producers to show aggregate citation counts, given that a link back to the record in the Scopus and Web of Science database and attribution to their brand is provided. I leave that to you to figure out how this can be achieved in the Table view.

Google Scholar does not allow to harvest citation counts.