https://wiki.eprints.org/w/index.php?action=history&feed=atom&title=API%3AEPrints%2FIndex%2FTokenizer
API:EPrints/Index/Tokenizer - Revision history
2026-04-12T09:32:54Z
Revision history for this page on the wiki
MediaWiki 1.31.8
https://wiki.eprints.org/w/index.php?title=API:EPrints/Index/Tokenizer&diff=14126&oldid=prev
Pod2wiki: Created page with "<!-- Pod2Wiki=_preamble_ This page has been automatically generated from the EPrints 3.4 source. Any wiki changes made between the 'Pod2Wiki=*' and 'Edit below this comment'..."
2022-01-10T01:00:05Z
<p>Created page with "<!-- Pod2Wiki=_preamble_ This page has been automatically generated from the EPrints 3.4 source. Any wiki changes made between the 'Pod2Wiki=*' and 'Edit below this comment'..."</p>
<p><b>New page</b></p><div><!-- Pod2Wiki=_preamble_ <br />
This page has been automatically generated from the EPrints 3.4 source. Any wiki changes made between the 'Pod2Wiki=*' and 'Edit below this comment' comments will be lost.<br />
-->{{API}}{{Pod2Wiki}}{{API:Source|file=EPrints/Index/Tokenizer.pm|package_name=EPrints::Index::Tokenizer}}[[Category:API|TOKENIZER]][[Category:API:EPrints/Index|TOKENIZER]]<div><!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki=_private_ --><!-- Pod2Wiki=head_name --><br />
==NAME==<br />
'''EPrints::Index::Tokenizer''' - text indexing utility methods.<br />
<br />
<div style='background-color: #e8e8f; margin: 0.5em 0em 1em 0em; border: solid 1px #cce; padding: 0em 1em 0em 1em; font-size: 80%; '><br />
<span style='display:none'>User Comments</span><br />
<!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki= --><br />
</div><br />
<!-- Pod2Wiki=head_description --><br />
==DESCRIPTION==<br />
This module provides utility methods for processing free text into indexable things.<br />
<br />
<div style='background-color: #e8e8f; margin: 0.5em 0em 1em 0em; border: solid 1px #cce; padding: 0em 1em 0em 1em; font-size: 80%; '><br />
<span style='display:none'>User Comments</span><br />
<!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki= --><br />
</div><br />
<!-- Pod2Wiki=head_methods --><br />
==METHODS==<br />
<div style='background-color: #e8e8f; margin: 0.5em 0em 1em 0em; border: solid 1px #cce; padding: 0em 1em 0em 1em; font-size: 80%; '><br />
<span style='display:none'>User Comments</span><br />
<!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki= --><br />
</div><br />
<!-- Pod2Wiki=item_split_words --><br />
===split_words===<br />
<br />
@words = EPrints::Index::Tokenizer::split_words( $session, $utext )<br />
Splits a utf8 string <tt>$utext</tt> into individual words. <br />
<br />
<div style='background-color: #e8e8f; margin: 0.5em 0em 1em 0em; border: solid 1px #cce; padding: 0em 1em 0em 1em; font-size: 80%; '><br />
<span style='display:none'>User Comments</span><br />
<!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki= --><br />
</div><br />
<!-- Pod2Wiki=item_split_search_value --><br />
===split_search_value===<br />
<br />
@terms = EPrints::Index::Tokenizer::split_search_value( $session, $value )<br />
Splits and returns <tt>$value</tt> into search terms.<br />
<br />
<div style='background-color: #e8e8f; margin: 0.5em 0em 1em 0em; border: solid 1px #cce; padding: 0em 1em 0em 1em; font-size: 80%; '><br />
<span style='display:none'>User Comments</span><br />
<!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki= --><br />
</div><br />
<!-- Pod2Wiki=item_apply_mapping --><br />
===apply_mapping===<br />
<br />
$utext2 = EPrints::Index::Tokenizer::apply_mapping( $session, $utext )<br />
Replaces certain unicode characters in <tt>$utext</tt> with ASCII equivalents and returns the new string.<br />
<br />
This is used before indexing words so that things like umlauts will be ignored when searching.<br />
<br />
<div style='background-color: #e8e8f; margin: 0.5em 0em 1em 0em; border: solid 1px #cce; padding: 0em 1em 0em 1em; font-size: 80%; '><br />
<span style='display:none'>User Comments</span><br />
<!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki= --><br />
</div><br />
<!-- Pod2Wiki=head_copyright --><br />
==COPYRIGHT==<br />
{{API:Copyright}}<br />
<div style='background-color: #e8e8f; margin: 0.5em 0em 1em 0em; border: solid 1px #cce; padding: 0em 1em 0em 1em; font-size: 80%; '><br />
<span style='display:none'>User Comments</span><br />
<!-- Edit below this comment --><br />
<br />
<br />
<!-- Pod2Wiki= --><br />
</div><br />
<!-- Pod2Wiki=_postamble_ --><!-- Edit below this comment --></div>
Pod2wiki