perl-String-Tokenizer - A simple string tokenizer

Distribution: OpenMandriva Cooker
Repository: OpenMandriva Contrib x86_64
Package name: perl-String-Tokenizer
Package version: 0.50.0
Package release: 4-omv2015.0
Package architecture: noarch
Package type: rpm
Installed size: 24.38 KB
Download size: 16.58 KB
Official Mirror:
A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways. This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple 'split / /, $string' and the other options that involve much larger and complex modules. Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.



  • perl(String::Tokenizer) == 0.50.0
  • perl(String::Tokenizer::Iterator)
  • perl-String-Tokenizer == 0.50.0-4:2015.0

    Install Howto

    1. Enable OpenMandriva Contrib repository on Install and Remove Software"
    2. Update packages list:
      # urpmi.update -a
    3. Install perl-String-Tokenizer rpm package:
      # urpmi perl-String-Tokenizer


    • /usr/lib/perl5/vendor_perl/5.20.1/String/
    • /usr/share/doc/perl-String-Tokenizer/Changes
    • /usr/share/doc/perl-String-Tokenizer/README
    • /usr/share/man/man3/String::Tokenizer.3pm.xz