Tokenizes a string according to a pattern.

Related Syntax Constructs

The following construct(s) refer to this construct:

---

Syntax

fn:tokenize(string $string, string $pattern) => string*

fn:tokenize(string $string, string $pattern, string $flags) => string*

Description

This function breaks the input $string into a sequence of strings, treating any substring that matches $pattern as a separator. The separators themselves are not returned. The $flag argument is interpreted in the same way as for the fn:matches() function.

Argument

$string	a string to be tokenized
$pattern	a separator pattern
$flags	modifiers for patternmatching

Example

• fn:tokenize("Some unparsed <br> HTML <br> text", "\s*<br>\s*", "i") 
  

  The query returns ("Some unparsed", "HTML", "text") the pattern "\s*" denoting any number of whitespace characters.