Unicode

To support non-ASCII characters (a to z) EPrints uses Unicode, which is a database of characters (glyphs) used by languages of the world. To store Unicode strings EPrints uses Unicode::String. Unicode::String stores strings internally in UCS-2 (2 bytes per character) but converts that to utf-8 when stringified (variable-length bytes per character).

Perl's native encoding for Unicode is utf-8.

EPrints 3.2 replaces Unicode::String with Perl's native Unicode support.

Unicode and MySQL

Before 5.0 MySQL only supported iso-8859-1 (aka latin-1) character encodings.

Determining the current character set

To determine the character set MySQL is currently using execute this from the MySQL client:

mysql> show create table version;
 *************************** 1. row ***************************
        Table: version
 Create Table: CREATE TABLE `version` (
   `version` varchar(255) default NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
 1 row in set (0.00 sec)

In this case the database is set to latin1 (aka iso-8859-1).

What's actually stored

When EPrints stores utf-8 data in MySQL it is simply stored as bytes. For characters below code point 128 this doesn't make any difference (because the Unicode code points are the same as the iso-8859-1 equivalent). For code points above 128 multiple bytes will be stored in MySQL which, if viewed from a MySQL client, will look like this (in this case é):

é

When the data is read by EPrints it will use the raw bits as utf-8. So long as MySQL doesn't change the data coming in and out everything works correctly.

MySQL 5.0 and above support two Unicode encodings: utf-8 and ucs-2.

If MySQL is set to use utf-8 it will re-encode the utf-8 data sent from EPrints. For the "é" example above the data, when viewed in MySQL, will still look like:

é

But what MySQL has actually stored is:

é

MySQL and Collation

MySQL (in common with other databases) will perform some language-specific collation. Collation determines how strings should be sorted and matched. For instance MySQL will match "âge" to "age" if the data is correctly stored.

Because MySQL does not realise the data it is passed by EPrints is utf-8 it will not store the correct data and hence can not correctly perform collation for non-ASCII characters.

Making EPrints and MySQL talk the same language

To enable MySQL to understand the data EPrints is giving it (utf-8) you need to call the following SQL statement at the start of an EPrints session:

SET NAMES utf8;

You can do this by modifying archives/[repoid]/cfg/cfg.d/session.pl as follows:

$c->{session_init} = sub
{
   my( $session, $offline ) = @_;

   $session->get_database->do("SET NAMES utf8");
};

Warning! You can't do this to an existing database because it's storing utf-8 in an iso-8859-1 encoded columns.

Migrating individual columns from utf-8 in latin-1 to utf-8:

ALTER [table] MODIFY [column] VARBINARY(255);
ALTER [table] MODIFY [column] VARCHAR(255) CHARACTER SET utf8;

ALTER [table] MODIFY [column] BLOB;
ALTER [table] MODIFY [column] TEXT CHARACTER SET utf8;