token — A unit of information.



A token identifies a unit of information. Usually, “tokens” are the result of some processing pass that has performed lexical analysis and divided a data set into the smallest units of information used for subsequent processing.

Exactly what constitutes a token varies by context.

Processing expectations

Formatted inline.

Parents [+]

This element occurs in 51 elements.

Children [+]

This element contains 16 elements.


  1 <article xmlns=''>
  2 <title>Example token</title>
  4 <para>In parsing, line ends are turned into the <token>CRLF</token>, all other
    whitespace becomes <token>WHITESP</token>.
  6 </para>
  8 </article>

In parsing, line ends are turned into the CRLF, all other whitespace becomes WHITESP.

Last revised by Norman Walsh on 6 Jun 2011 (git hash: 6ffcc7640bbc5f852a318e452c9f210f03292cb9)