|
|
|
EmitToken (integer, integer, integer, integer, character, ListProToken)
|
Emits one token in the requested representation
Notes : Called only for modes 1 and 2 (mode 0 counts inline).
@param piMode The scan mode (1 = object, 2 = temp-table)
@param piType The token type
@param piSeq The token sequence number
@param piLine The source line
@param pcText The token text
@param poList The token list (mode 1)
*/
|
|
|
|
Consultingwerk.Studio.ProparseApi.Implementation.ProparseAbl.ProToken GetToken (integer)
|
Materializes a ProToken for a stored token on demand
Notes : Demonstrates lazy materialization from the columnar store.
@param piSeq The 1-based token sequence number
@return The ProToken, or the unknown value if out of range
*/
|
|
|
|
INTEGER LookupKeyword (character)
|
Resolves an identifier to a token type, with caching
Notes :
@param pcText The identifier text
@return The resolved token type
*/
|
|
|
|
INTEGER Scan (longchar, integer, ListProToken)
|
Single-pass block-scanning tokenizer over a byte buffer
Notes :
@param plcSource The source code
@param piMode The representation mode (0 count, 1 objects, 2 temp-table)
@param poList The token list (mode 1)
@return The number of tokens produced
*/
|
|
|
|
Consultingwerk.Studio.ProparseApi.Implementation.ProparseAbl.ListProToken Tokenize (longchar)
|
Tokenizes source and returns the tokens as a ProToken list
Notes :
@param plcSource The source code
@return The token list
*/
|
|
|
|
INTEGER TokenizeCount (longchar)
|
Tokenizes source and returns only the token count
Notes : Isolates raw scanning throughput from token materialization.
@param plcSource The source code
@return The number of tokens
*/
|
|
|
|
INTEGER TokenizeToTable (longchar)
|
Tokenizes source into a columnar temp-table store
Notes : The lazy / non-object token representation. ProToken objects
are materialized only on demand via GetToken.
@param plcSource The source code
@return The number of tokens stored
*/
|
