Generic Reporting Framework

Introduction

The Generic Reporting Framework Bazaar plugin facilitates the creations of reports on records held within an EPrints repository that can be configured to return records based on certain criteria and to validate these records based on given test criteria.

For Users

The Generic Reporting Framework adds a new link to the key tools bar (alongside links such as 'Manage Deposits' and 'Admin') called 'Reports', that is only shown to administrator users.

Generating a Report

This link displays the Reports splash page from which any available reports may be selected.

The Reports splash page shows both 'Preset Reports' and 'Custom Reports' depending on the reports available. 'Preset Reports' display records that match predefined criteria and are typically added to the repository through the installation of other plugins such as REF CC or ORCID Support. 'Custom Reports' displays a dropdown of available reports, and a search form for the selected report. This can used to generate a report comprised of records which match the given search criteria. Generic Reporting Framework v3.0+ offers two custom reports by default, one for EPrints and another for Users.

Viewing a Report

Once a report has been selected the page will reload to display the results of that report. This process may take a short while if there are lots of records to process.

Next to each record is a coloured bar. For reports which indicate some form of validation (e.g. the REF CC compliance reports) this bar will indicate the status of the record, e.g. green for compliant, red for non-compliant. For reports which do not make use of compliance, this bar will be set to a default colour (which can be customised - see For Developers section below.

Each record may also be accompanied by some bullet points offering further information about the record (e.g. why it might not meet compliance criteria).

Reports may also be exported using the 'Export' button at the top of the page. Some reports support custom exports and where available a number of fields will be listed alongside checkboxes. Each checked field will be included in the exported report.

Reports may also be grouped and sorted by different fields using the 'Group by' and 'Sort by' links. Grouping recors has the affect of breaking down the page in to a number of different sections with a header to represent each group. Sorting records results in the records being reordered based on the selected field. If the records have already been grouped the records will be sorted within each group. The fields that are available for a given report's 'Group by' and 'Sort by' links may be set in the report's configuration - see For Developers for more details.

Exporting a Report

HTML Export

The HTML report reproduces the report as an HTML page, with each record's citation displaying followed by a list of fields, as chosen from the export options. A 'Print' button at the top of the screen allows you to print the report, which may also be used to turn the report into a PDF document depending upon browser functionality.

Custom templates can be applied to each report's configuration to alter the look and feel of the HTML export (see For Developers for more details).

CSV Export

The CSV export displays the fields selected from the export options in a multi-line CSV format, i.e. each field is separated by a comma, with fields that contain multiple values, such as the Creators, field being displayed one per row when viewed as a spreadsheet.

The value in the first column of the CSV is repeated for every row. For this reason it is often useful to use an ID field as the first column (e.g. EPrint ID).

JSON Export

The JSON export represents the fields selected from the export options, in JSON format.

For Developers

GitHub: https://github.com/eprintsug/reports

Returning Report Bullet Points (v1.1+ only)

By default, any problems that are identified by the report's 'validate_dataobj' function are displayed as bullet points for that item in the report. However, if you wish to display bullet points for an item which does not report any problems when validating, a new 'bullet_points' function may be defined, e.g.

sub bullet_points
{
        my( $self, $eprint ) = @_;

        my $repo = $self->{repository};

        my @bullets;

        foreach my $creator( @{ $eprint->value( "creators" ) } )
        {
                if( EPrints::Utils::is_set( $creator->{orcid} ) )
                {
                        push @bullets, 
                                EPrints::XML::to_string( $repo->html_phrase( 
                                       "creator_with_orcid", 
                                       creator => $repo->xml->create_text_node(EPrints::Utils::make_name_string( $creator->{name} ) ),
                                       orcid => $repo->xml->create_text_node( $creator->{orcid} ) 
                                ) );
                }
                else
                {
                        push @bullets, 
                                EPrints::XML::to_string( $repo->html_phrase( 
                                       "creator_no_orcid", 
                                       creator => $repo->xml->create_text_node(EPrints::Utils::make_name_string( $creator->{name} ) ) 
                                 ) );
                }
        }

        return @bullets;
}

The 'bullet_points' function should then be called from ajax_eprint when generating the report's JSON data, e.g.

sub ajax_eprint
{
        my( $self ) = @_;

        my $repo = $self->repository;

        my $json = { data => [] };

        $repo->dataset( "eprint" )
        ->list( [$repo->param( "eprint" )] )
        ->map(sub {
                (undef, undef, my $eprint) = @_;

                my $frag = $eprint->render_citation_link;
                push @{$json->{data}}, {
                        datasetid => $eprint->dataset->base_id,
                        dataobjid => $eprint->id,
                        summary => EPrints::XML::to_string( $frag ),
                        problems => [ $self->validate_dataobj( $eprint ) ],
                        bullets => [ $self->bullet_points( $eprint ) ],
                };
        });
        print $self->to_json( $json );
}

Returning Report Colours (v1.1+ only)

Version 1.0 of the Generic Reporting Framework plugin, can only display two colours for records displayed in the report. When validating a record using the a report plugin's 'validate_dataobj' function, if an array of problems is returned the item is displayed with a red bar, and if none are returned the item is displayed with a green bar.

To return other colours, define a 'get_state' function which should return a string representing a hex encoded colour. E.g.

sub get_state
{
        my( $self, $eprint ) = @_;

        my $repo = $self->{repository};

        if( $eprint->hasProblem() )
        {
            return "#E19141"; #orange
        }
        else
        {
            return undef;
        }
}

The 'get_state' function should then be called from ajax_eprint when generating the report's JSON data, e.g.

sub ajax_eprint
{
        my( $self ) = @_;

        my $repo = $self->repository;

        my $json = { data => [] };

        $repo->dataset( "eprint" )
        ->list( [$repo->param( "eprint" )] )
        ->map(sub {
                (undef, undef, my $eprint) = @_;
                
                my $frag = $eprint->render_citation_link_staff;
                push @{$json->{data}}, {
                        datasetid => $eprint->dataset->base_id,
                        dataobjid => $eprint->id,
                        summary => EPrints::XML::to_string( $frag ),
                        problems => [ $self->validate_dataobj( $eprint ) ],
                        state => $self->get_state( $eprint ),
                };
        });

        print $self->to_json( $json );
}

Custom Reports (v1.1+ only)

For a report to display in the 'Custom Reports' tab of the splash page, its custom property must be set. E.g.

#from within the local archive level
$c->{plugins}{"Screen::Report::REF_CC"}{params}{custom} = 1;

Custom Export Fields (v2.0+ only)

To specify which fields should appear in the custom export box for a specific report plugin, include an 'export_conf' parameter in the plugin's configuration that defines an ID for locating configuration relating to exports. E.g.

#in the report plugin's 'sub new' constructor
$self->{export_conf} = 'hefce_report';

The export fields themselevs should then be defined at the local archive configuration level, using this 'export_conf' ID.

Export fields are described using a hash of 1 or more arrays, allowing fields to be grouped under a particular heading. Note, that as per a search config, fields of other datasets may be defined also, providing the dataset has a connection with the parent dataset (e.g. document fields can be defined within an eprint context as documents.fieldname). E.g.

 
$c->{hefce_report}->{exportfields} = {
        ref_core => [ qw(
                eprintid
                documents.content
                type
                title
                abstract
                creators
                publisher
                divisions
                dates
                id_number
                isbn
                issn
                official_url                                            
        )],
        ref_rioxx => [ qw(
                rioxx2_free_to_read
                rioxx2_license_ref
                rioxx2_coverage
                rioxx2_source
                rioxx2_subject
                rioxx2_dateAccepted
                rioxx2_publication_date
                rioxx2_apc
                rioxx2_project
                rioxx2_version
                rioxx2_version_of_record
        )],
        ref_exceptions => [ qw(
                hoa_compliant
                hoa_problems
                hoa_ref_pan
                hoa_ex_dep
                hoa_ex_dep_txt
                hoa_ex_acc
                hoa_ex_acc_txt
                hoa_ex_tec
                hoa_ex_tec_txt
                hoa_ex_oth
                hoa_ex_oth_txt
        )],
};

For each of the fields defined above a checkbox will be displayed in the Report's 'Export Options' box. Some of these may be selected by default, by defining an "exportfield_defaults" value, also at the local archive configuration level, E.g.

$c->{hefce_report}->{exportfield_defaults} = [ qw( eprintid documents.content type title creators dvisisions dates hoa_compliant hoa_problems ) ];

Finally, functions may be defined that allow fields to be displayed in a bespoke manner. These functions are also defined at the local archive level via use of the 'export_conf' ID, with each field mapping on to a function. The functions take the DataObj and Report plugin as parameters. E.g.

$c->{hefce_report}->{custom_export} = {
        hoa_compliant => sub {
                my( $dataobj, $plugin ) = @_;

                my $compliance = "Compliant";

                my @problems = $plugin->validate_dataobj( $dataobj );
                if( scalar( @problems ) > 0 )
                {
                        $compliance = "Not Compliant";
                }
                $compliance = "Compliant pending open access" if defined $plugin->get_state( $dataobj );
                return $compliance;
        },
        creators => sub {
                my( $dataobj ) = @_;

                my @creator_names;
                my $creators = $dataobj->get_value( "creators" );
                foreach my $c ( @{$creators} )
                {
                        push @creator_names, EPrints::Utils::make_name_string( $c->{name} );
                }
                return join( ";", @creator_names );
        },

Custom Report Search Configuration (v2.0+ only)

Custom reports allow users to generate reports via a search config. This is stored in a report plugin parameter call "sconf" and is by default defined as "report":

$self->{sconf} = "report";

With default report config is defined as follows, matching the format used by search configs such as the Advanced Search:

$c->{search}->{report} =
{
        search_fields => [
                { meta_fields => [ "title" ] },
                { meta_fields => [ "creators_name" ] },
                { meta_fields => [ "date" ] },
                { meta_fields => [ "subjects" ] },
                { meta_fields => [ "type" ] },
                { meta_fields => [ "divisions" ] },
                { meta_fields => [ "publication" ] },
        ],
        order_methods => {
                "byyear"         => "-date/creators_name/title",
                "byyearoldest"   => "date/creators_name/title",
                "byname"         => "creators_name/-date/title",
                "bytitle"        => "title/creators_name/-date"
        },
        default_order => "byyear",
        show_zero_results => 1,
};

Bespoke search configurations can be defined, by overriding the 'sconf' parameter. E.g.

#in the report plugin's 'sub new' constructor
$self->{sconf} = 'hefce_report';

And then a corresponding search configuration may be defined at the local archive level. E.g.

$c->{search}->{hefce_report} =
{
        search_fields => [
                { meta_fields => [ "title" ] },
                { meta_fields => [ "creators_name" ] },
                { meta_fields => [ "date" ] },
                { meta_fields => [ "hoa_date_acc" ] },
                { meta_fields => [ "subjects" ] },
                { meta_fields => [ "type" ] },
                { meta_fields => [ "ispublished" ] },
                { meta_fields => [ "divisions" ] },
                { meta_fields => [ "publication" ] },
        ],
        order_methods => {
                "byyear"         => "-date/creators_name/title",
                "byyearoldest"   => "date/creators_name/title",
                "byname"         => "creators_name/-date/title",
                "bytitle"        => "title/creators_name/-date"
        },
        default_order => "byyear",
        show_zero_results => 1,
};

Custom Report Header (v2.1+ only)

The compliance information in the report header (the compliance bar visualisation and the percentage) can be removed by setting the 'show_compliance' value for a report. E.g.

#in the report plugin's 'sub new' constructor
$self->{show_compliance} = 0;

The word 'outputs' in the report header to show how many items are included in the report, may be customised by setting the 'labels' value. E.g.

#in the report plugin's 'sub new' constructor
$self->{labels} = {
                outputs => "users"
        };

Only the 'outputs' value can be changed at present, with the value stored in a 'labels' hash for any future changes that may be applied.

Grouping and Sorting (v3.0+ only)

To specify grouping and sorting parameters, first a key must be defined as a report plugin parameter, that allows the grouping and sorting configuration to be uniquely identified:

$self->{sort_conf} = 'eprint_report';
$self->{group_conf} = 'eprint_report';

At the local archive level sorting configurations may be defined like so:

$c->{eprint_report}->{sortfields} = {
        "byname" => "creators_name/-date/title",
        "byyear" => "-date/creators_name/title",
        "bytitle" => "title/creators_name/-date",
        "bydivision" => "divisions/creators_name/-date",
};

Sorting configurations can be defined using the same format as used by any search configuration in the repository.

Grouping configurations may be defined as follows:

$c->{eprint_report}->{groupfields} = [ qw(
        divisions
        subjects
        type
        date;res=year;reverse_order=1
)];

Much like the 'menus' and 'variations' when configuring Browse Views, further options may be passed for each group, with separated by a semi-colon.

'reverse_order=1': reverses the ordering of the groups, e.g. dates will be presented from newest to oldest, rather than oldest to newest.
'truncate': truncates the field value by the number of characters specified (e.g. 'creators_name;truncate=1' will only use the creator's first initial).
'res': This only applies to 'date' fields, applying a resolution for the date, e.g. 'date;res=year' (see Date_field for more information).

HTML Export Templates (v3.0+ only)

To customise the look and feel of an HTML report export, a number of parameters may be defined at the local archive configuration level:

$c->{plugins}{"Export::Report::HTML"}{params}{title_phrase} = "eprint_report_title";
$c->{plugins}{"Export::Report::HTML"}{params}{custom_class} = "eprint_report";
$c->{plugins}{"Export::Report::HTML"}{params}{template} = "eprint_report";

Setting the 'title_phrase' will apply a custom phrase as the title of the HTML report.

Setting the 'custom_class' parameter, will result in the value being added as a class attribute to the various HTML elements that make up the report, allowing for easier CSS customisation.

The 'template' parameter is used to specify the name of a template XML file that the report should use instead of default.xml.

Export Virtual Report Fields (v3.0+ only)

The list of fields to be included in a report export, may now also include virtual report fields. These fields do not need to be defined anywhere else in the dataset configuration.

To define a virtual report field, first include it in the list of the 'exportfields', e.g.

$c->{eprint_report}->{exportfields} = {
        eprint_report_core => [ qw(
                eprintid
                title
                virtual_field_foo
        )]
};

For the field to be exported successfully, we must define a function that generates a value to be included in the export. E.g.

$c->{eprint_report}->{custom_export} = {
        virtual_field_foo => sub {
                my( $dataobj, $plugin ) = @_;
                return "My exported value";       
        }
};

To return values over multiple rows in the multiline CSV export, we can simply return an array of the values we want to return.

The export functions (which may also be used to override existing fields in the dataset), receive both the dataobject and the report plugin as parameters, which may be used when generating the value that is to be exported.

Set Export Field Checkbox Ordering (v3.1+ only)

The fields which may be included in an export are displayed as columns of checkboxes, as defined by the 'exportfields' value, e.g.:

$c->{my_report}->{exportfields} = {
        eprint_core => [ qw(
                eprintid
                title
                abstract
        )],
        eprint_metadata => [ qw(
                publication
                divisions               
                subjects
        )]
};

The order in which these columns are displayed may be set by providing a list of the keys of this hash, e.g.:

$c->{my_report}->{exportfield_order} = [ qw( eprint_core eprint_metadata )];

In both these examples the value 'my_report' used as a key to find the right bit of configuration can be defined in the report's 'export_conf' parameter, e.g.:

$self->{export_conf} = 'my_report';

Set Available Export Plugins (v3.1+ only)

The ordering and presence of export plugins presented in the 'Export Options' box at the top of a report screen, may be configured for each report, e.g.

$c->{eprint_report}->{export_plugins} = [ qw( Export::Report::CSV Export::Report::HTML Export::Report::JSON )]; #set the CSV export to appear at the top of the select element

In the above example, the value 'eprint_report' used as a key to find the configuration can be defined in the report's 'export_conf' parameter, e.g.:

$self->{export_conf} = 'eprint_report';

Change Log

1.0 (11 March 2015)

Initial version of the Generic Reporting Framework plugin.

1.1 (14 June 2017)

Added support for custom exports, where users can choose which fields to include in CSV exports from a predefined list in the report's plugin configuration.
Added support for flexible reporting. Report plugins may be specified as permitting custom reports. Existing defined reports are displayed on the splash page in the 'Preset Reports' tab, a search form for custom reports is displayed in the 'Custom Reports' tab.
Added support for allowing reports to return different colours, beyond the green=compliant, red=non-compliant states.
Added support for bullet points to be added to any item in the report, not just non-compliant items.

1.1.1 (15 June 2017)

Changed export interface to present field options in one div element, rather than as tabs.
Removed custom report tab from the report splash page if none of the report plugins support custom exports.

1.1.2 (16 June 2017)

Updates to export plugins. When custom export fields are available, the generic CSV exporter is used. When custom export fields are not available, any bespoke export plugins that come with the report are available instead. (In the event that custom export fields are not available and there are no bespoke export plugins, the export option is now shown, as per version 1.0.)

2.0 (8 September 2017)

Custom export fields maybe defined at local archive level.
Custom search configurations for reports may be defined at local archive level.
Fixes to rendering of search forms and report item states (coloured bars) for IE.
Allow default export fields to be defined.
Allow export field custom rendering via functions defined at local archive level.

2.1 (3 August 2018)

Allow reports to hide the compliance percentage where not relevant, and customise the word "outputs" so that it can represent different data objects, e.g. "users".
Update Bazaar icon

3.0 (21 December 2018)

Provides reports on EPrint and User datasets, allowing for searches to be conducted over these datasets with the user able to choose whihc fields should be included in the export.
Provides an HTML report export, which can be used to generate PDF reports using a browser's HTML to PDF functionality.
Grouping of items within a report based on fields specified in report config
Sorting of items within a report based on fields specified in report config

3.1 (01 March 2019)

Add 'New search' and 'Refine search' links when viewing a report generated by a search
Fix bug with 'Reset the form' button not working
Add "Select/Deselect All" button to export field interface
Allow ordering and appearance of export plugin options to be configurable at a report level
Allow ordering of export field checkboxes to be configurable at a report level
Update CSV export to render subject names rather than values
Update CSV export to render virtual fields
Remove 'counter' from list of fields included in the user reports
Fix bug with JSON export not working when no custom export functions defined