Skip to content

Commit 7250f21

Browse files
Update README.md with goals (#2)
1 parent 24ce2b7 commit 7250f21

1 file changed

Lines changed: 50 additions & 17 deletions

File tree

README.md

Lines changed: 50 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -8,63 +8,96 @@
88
[<img alt="support chat" src="https://img.shields.io/badge/support-chat-blue.svg" height="20">](https://gitter.im/postcss/postcss)
99

1010
This tools lets you tokenize CSS according to the [CSS Syntax Specification](https://drafts.csswg.org/css-syntax/).
11-
It works by separating a string of CSS into its smallest, semantic parts.
12-
This is intended for use by other tools, in either the backend or the frontend, and it will contribute roughly 1kB of code.
11+
Tokenizing CSS is separating a string of CSS into its smallest, semantic parts — otherwise known as tokens.
12+
13+
This tool is intended to be used in other tools on the front and back end. It seeks to maintain:
14+
15+
- 100% compliance with the CSS syntax specification. ✨
16+
- 100% code coverage. 🦺
17+
- 100% static typing. 💪
18+
- 1kB maximum contribution size. 📦
19+
- Superior quality over Shark P. 🦈
1320

1421
## Usage
1522

16-
Add this [CSS tokenizer](https://github.com/csstools/tokenizer) to your project:
23+
Add the [CSS tokenizer](https://github.com/csstools/tokenizer) to your project:
1724

1825
```sh
1926
npm install @csstools/tokenizer
2027
```
2128

22-
Use this tool to tokenizer your CSS in JavaScript:
29+
Tokenize CSS in JavaScript:
2330

2431
```js
2532
import cssTokenizer from '@csstools/tokenizer'
2633

27-
const tokens = Array.from(cssTokenizer(cssText))
34+
const tokens = Array.from(cssTokenizer(cssText)) // an array of css tokens
2835
```
2936

30-
Use this tool in classical NodeJS:
37+
Tokenize CSS in _classical_ NodeJS:
3138

3239
```js
3340
const cssTokenizer = require('@csstools/tokenizer')
3441

3542
let iterator = cssTokenizer(cssText), iteration
3643

3744
while (!(iteration = iterator()).done) {
38-
console.log(iteration.value) // logs the token
45+
console.log(iteration.value) // logs an individual css token
3946
}
4047
```
4148

42-
Use this tool in scripts:
49+
Tokenize CSS in client-side scripts:
4350

4451
```html
4552
<script type="module">
46-
import cssTokenizer from "https://unpkg.com/@csstools/tokenizer?module"
47-
cssTokenizer(cssText)
53+
54+
import cssTokenizer from 'https://unpkg.com/@csstools/tokenizer?module'
55+
56+
const tokens = Array.from(cssTokenizer(cssText)) // an array of css tokens
57+
4858
</script>
4959
```
5060

51-
Use this tool in classical scripts:
61+
Tokenize CSS in _classical_ client-side scripts:
5262

5363
```html
5464
<script src="http://unpkg.com/@csstools/tokenizer"></script>
5565
<script>
56-
cssTokenizer(cssText)
66+
67+
const tokens = Array.from(cssTokenizer(cssText)) // an array of css tokens
68+
5769
</script>
5870
```
5971

6072
## How it works
6173

74+
The CSS tokenizer separates a string of CSS into tokens represented as an array.
75+
76+
```ts
77+
[
78+
/** Position in the string at which the token was retrieved. */
79+
number,
80+
81+
/** Negative number that identifies the kind of token. */
82+
number,
83+
84+
/** Opening token content, like the opening of a comment or the quotation mark of a string. */
85+
string,
86+
87+
/** Main token content, like the numbers before a unit, or the letters after an at-sign. */
88+
string,
89+
90+
/** Closing token content, like the unit of a number, or the closing of a comment. */
91+
string,
92+
]
93+
```
94+
6295
As an example, the string `@media` would become a **Name** token where `@` and `media` are recognized as distinct parts of that token. As another example, the string `5px` would become a **Number** token where `5` and `px` are recognized as distinct parts of that token. As a final example, the string `5px 10px` would become 3 tokens; the **Number** as mentioned before (`5px`), a **Space** token that represents a single space (` `), and then another **Number** token (`10px`).
6396

64-
An actual token is represented in a series of 5 items;
97+
An actual token is represented in a series of 5 items;
6598

6699
```js
67-
[ 0, -9, '', '100', '%' ] // CSS with a value of "100%"
100+
[0, -9, '', '100', '%'] // CSS with a value of "100%"
68101
```
69102

70103
The **first** number represents the position at which the token was read. The **second** number represents the type id of the token. The **third**, **fourth**, and **fifth** strings represent the text prefix, value, and suffix of the token.
@@ -90,9 +123,9 @@ Benchmark: Bootstrap
90123
└────────────────────────────────────────────────┴──────┴────────┴────────┘
91124
```
92125

93-
## Contributing: Local Scripts
126+
## Development
94127

95-
You want to take a deeper dive? Awesome! Here are a few useful development commands.
128+
You wanna take a deeper dive? Awesome! Here are a few useful development commands.
96129

97130
### npm run build
98131

@@ -121,6 +154,6 @@ As of September 26, 2020, this tokenizer has 100% test coverage:
121154
npm run test
122155
```
123156

124-
[Bootstrap]: https://getbootstrap.com
157+
[Boostrap]: https://getbootstrap.com
125158
[PostCSS]: https://postcss.org
126159
[Tailwind CSS]: https://tailwindcss.com

0 commit comments

Comments
 (0)