# Class: Tokenizer

## ProtoBuf.DotProto. Tokenizer

A ProtoBuf .proto Tokenizer.

#### new Tokenizer(proto)

  Constructs a new Tokenizer.

  ##### Parameters:

  Name | Type | Description
  --- | --- | ---
  `proto` | string | Proto to tokenize

### Members

#### index :number

  Current index.

#### line :number

  Current line.

#### readingString :boolean

  Whether currently reading a string or not.

#### source :string

  Source to parse.

#### stack :Array

  Stacked values.

### Methods

#### next() → {string}

  Gets the next token and advances by one.

  ##### Throws:

  If it's not a valid proto file
  Type
    Error

  ##### Returns:

  Token or `null` on EOF

  Type
    string

#### peek() → {string}

  Peeks for the next token.

  ##### Throws:

  If it's not a valid proto file
  Type
    Error

  ##### Returns:

  Token or `null` on EOF

  Type
    string

#### toString() → {string}

  Returns a string representation of this object.

  ##### Returns:

  String representation as of "Tokenizer(index/length)"

  Type
    string