tokenize

Tokenize a character string.

Available in:

Apps (win)	Apps (char)	Reportwriter	RPC	Standalone PL
X	X	X	X	X

Syntax

list tokenize(data[,separator[,quote[,nulls]]])
string        data,separator,quote
int           nulls

Description

Breaks up the tokens in data. The default token separator is a blank character. The separator string overrides this by providing a string of characters to be used as token delimiters. There is no default quote character. If quote is provided, then all characters within a pair of quote characters are returned as one token. The nulls flag controls how tokenize treats leading, trailing and consecutive separators. The default is false which means that leading, trailing and consecutive separators will just be ignored. Setting nulls to true means that an empty list row will be inserted wherever there are leading or consecutive separators and appended if there is a trailing separator. The tokens are returned in a single column list.

data	specifies the string to tokenize.
separator	(optional) specifies the character(s) to use as token delimiter(s).
quote	(optional) specifies the character(s) to use as quote(s).
nulls	(optional) specifies how consecutive separators are treated. The default is false which does not insert an empty list row for consecutive separators.

Notes

Tokenize does not accept NULL as a delimiter. Use translate to first convert any NULL characters to something else before calling tokenize.

Example

Simple blank delimited string with no quotes:

LL = tokenize("Trifox is located in California");

         LL will contain: Trifox
                          is
                          located
                          in
                          California

Use "/" and "." as the token delimiters:

LL = tokenize("/etc/mail.log","/.");
    
         LL will contain: etc
                          mail
                          log

Use " " and "," as token delimiters and "'" as the quote character:

LL = tokenize("what, might 'this be'"," ,","'"); 

         LL will contain: what
                          might
                          this be

Use "," as the token delimiters:

LL = tokenize("what,might,,be",",","'",true); 

         LL will contain: what
                          might

                          be