Jump to content
Wiki Home
EPrints Home
Bazaar
Files
Main menu
Main menu
move to sidebar
hide
Navigation
Front Page
EPrints 3 Manual
EPrints Training Course
EPrints Glossary
Installing EPrints
How-to Configure EPrints
EPrints 3.2+ API
EPrints 3 Reference
EPrints Plugins
Wiki
All categories
Recent changes
Main Page
Help
EPrints Documentation
Search
Search
Appearance
Log in
Request account
Personal tools
Log in
Request account
Pages for logged out editors
learn more
Contributions
Talk
Editing
Unicode
Page
Discussion
British English
Read
Edit
View history
Tools
Tools
move to sidebar
hide
Actions
Read
Edit
View history
General
What links here
Related changes
Special pages
Page information
Appearance
move to sidebar
hide
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
[[Category:Languages]] To support non-ASCII characters (a to z) EPrints uses [http://www.unicode.org/ Unicode], which is a database of characters (glyphs) used by languages of the world. To store Unicode strings EPrints uses [http://search.cpan.org/~gaas/Unicode-String-2.09/ Unicode::String]. Unicode::String stores strings internally in UCS-2 (2 bytes per character) but converts that to utf-8 when stringified (variable-length bytes per character). Perl's native encoding for Unicode is utf-8. EPrints 3.2 replaces Unicode::String with [http://perldoc.perl.org/perlunicode.html Perl's native Unicode] support. == Unicode and MySQL == Before 5.0 MySQL only supported [http://en.wikipedia.org/wiki/Iso-8859-1 iso-8859-1] (aka latin-1) character encodings. === Determining the current character set === To determine the character set MySQL is currently using execute this from the MySQL client: mysql> show create table version; *************************** 1. row *************************** Table: version Create Table: CREATE TABLE `version` ( `version` varchar(255) default NULL ) ENGINE=MyISAM DEFAULT CHARSET=latin1 1 row in set (0.00 sec) In this case the database is set to latin1 (aka iso-8859-1). === What's actually stored === When EPrints stores utf-8 data in MySQL it is simply stored as bytes. For characters below code point 128 this doesn't make any difference (because the Unicode code points are the same as the iso-8859-1 equivalent). For code points above 128 multiple bytes will be stored in MySQL which, if viewed from a MySQL client, will look like this (in this case é): é When the data is read by EPrints it will use the raw bits as utf-8. So long as MySQL doesn't change the data coming in and out everything works correctly. MySQL 5.0 and above support two Unicode encodings: utf-8 and ucs-2. If MySQL is set to use utf-8 it will re-encode the utf-8 data sent from EPrints. For the "é" example above the data, when viewed in MySQL, will still look like: é But what MySQL has actually stored is: é == MySQL and Collation == MySQL (in common with other databases) will perform some language-specific collation. Collation determines how strings should be sorted and matched. For instance MySQL will match "âge" to "age" if the data is correctly stored. Because MySQL does not realise the data it is passed by EPrints is utf-8 it will not store the correct data and hence can not correctly perform collation for non-ASCII characters. == Making EPrints and MySQL talk the same language == From EPrints 3.3 onwards MySQL connections are configured to understand the data stored as UTF-8. This is acheived by the following MySQL command being used each time a MySQL connectionn is established. SET NAMES utf8; If you have a very old MySQL database you may need to migrate individual columns from Latin-1 to UTF-8: ALTER [table] MODIFY [column] VARBINARY(255); ALTER [table] MODIFY [column] VARCHAR(255) CHARACTER SET utf8; ALTER [table] MODIFY [column] BLOB; ALTER [table] MODIFY [column] TEXT CHARACTER SET utf8; == Managing 32-Bit Unicode Characters == In recent years there has been a massive expansion to the number of Unicode characters, to the extent that there is not enough bits to represent each character in UTF-8. Therefore there was an augmentation called UTF-8 MB4, which allows this extra Unicode characters, such as 𝒪 (mathematical script capital O). If you need support for such characters there are two things you need to do: 1. Modify '''perl_lib/EPrints/Database/mysql.pm''' and chnage the line: $self->do("SET NAMES 'utf8'"); to: $self->do("SET NAMES 'utf8mb4'"); 2. Modify the column in the table. You cannot do this for VARCHAR columns unless they are 191 characters or fewer. Due to the maximum number of bytes allowed for a VARCHAR column. ALTER [table] MODIFY [column] TEXT CHARACTER SET utf8mb4; ALTER [table] MODIFY [column] LONGTEXT CHARACTER SET utf8mb4; ALTER [table] MODIFY [column] VARCHAR(191) CHARACTER SET utf8mb4; There are plans to implement these changes in [[EPrints 3.4.4]], so that newly created repository will be able to handle 32-bit Unicode characters. If this is implemented upgrading to 3.4.4 will likely still require manually updating existing fields. For more information see https://www.eversql.com/mysql-utf8-vs-utf8mb4-whats-the-difference-between-utf8-and-utf8mb4/
Summary:
Please note that all contributions to EPrints Documentation may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see
EPrints Documentation:Copyrights
for details).
Do not submit copyrighted work without permission!
Cancel
Editing help
(opens in new window)