# Tokenizer

A small library for converting tokenized PHP source code into XML.

[![Test](https://github.com/theseer/tokenizer/actions/workflows/ci.yml/badge.svg)](https://github.com/theseer/tokenizer/actions/workflows/ci.yml)

## Installation

You can add this library as a local, per-project dependency to your project using [Composer](https://getcomposer.org/):

    composer require theseer/tokenizer

If you only need this library during development, for instance to run your project's test suite, then you should add it as a development-time dependency:

    composer require --dev theseer/tokenizer

## Usage examples

```php
$tokenizer = new TheSeer\Tokenizer\Tokenizer();
$tokens = $tokenizer->parse(file_get_contents(__DIR__ . '/src/XMLSerializer.php'));

$serializer = new TheSeer\Tokenizer\XMLSerializer();
$xml = $serializer->toXML($tokens);

echo $xml;
```

The generated XML structure looks something like this:

```xml
<?xml version="1.0"?>
<source xmlns="https://github.com/theseer/tokenizer">
 <line no="1">
  <token name="T_OPEN_TAG">&lt;?php </token>
  <token name="T_DECLARE">declare</token>
  <token name="T_OPEN_BRACKET">(</token>
  <token name="T_STRING">strict_types</token>
  <token name="T_WHITESPACE"> </token>
  <token name="T_EQUAL">=</token>
  <token name="T_WHITESPACE"> </token>
  <token name="T_LNUMBER">1</token>
  <token name="T_CLOSE_BRACKET">)</token>
  <token name="T_SEMICOLON">;</token>
 </line>
</source>
```

• [D] src
• [F] CHANGELOG.md
  Delete
• [F] composer.json
  Delete
• [F] composer.lock
  Delete
• [F] LICENSE
  Delete
• [F] README.md
  Delete

CUSTOMER COMPLAINT FORM

Please use this form to give us suggestions, compliments or complaints.
Click here to check complaint status.
Click here to show Term of Business Agreement

• Config option 1
• Config option 2

Salutation *

-- Please Choose -- Mr. Mrs. Ms. PT. CV. UD.

Name *

E-mail *

Account Type *

-- Please Choose -- Individual Company

Company Name *

Reference Number of Type Document *

-- Please Choose -- Policy Contract Invoice Number

*

About *

-- Please Choose -- Claim Service System

Risk *

-- Please Choose --

Subject *

Details *

Attachment

Add Attachment

Notes: *.png, *.jpg, *.jpeg, *.pdf, *.doc, *.docx, *.xls, *.xlsx, *.ppt, *.pptx, *.eml are allowed, and size must be smaller than 5Mb.

   Disclaimer :

• The information filled in form is true and valid. PT Howden Insurance Brokers Indonesia is not liable to any lawful obligations arising from any false or invalid information in this application form whatsoever, all the false or invalid information will not be processed.
• The client hereby agree to render PT Howden Insurance Brokers Indonesia a right to handle complaint including but not limited to grab data from records, emails, documents, confidential data, and others, liaise with relatable Parties e.g. (Insurer, Adjuster, Legal Advisor, etc) to settled complaint.
• The client hereby released PT Howden Insurance Brokers Indonesia from any material or immaterial damages if eventually, there is no evidence of damages caused by PT Howden Insurance Brokers Indonesia.

* is required

Clear Submit

Copyright © 2026 PT. Howden Insurance Brokers Indonesia. All rights reserved.
Authorised and regulated by Otoritas Jasa Keuangan (OJK).
Member of The Association of Indonesian Insurance & Reinsurance Brokers (APPARINDO).