tokenization

Author: libjohn

00_tokenization.Rmd

@@ -0,0 +1,42 @@
+---
+title: "unnest tokens"
+author: "John Little"
+date: "`r Sys.Date()`"
+abstract: "<br> This document derived from Chapter 1 of Silge's and Robinson's Text Minning with R<br>https://www.tidytextmining.com/tidytext.html"
+output: html_notebook
+---
+
+
+```{r message=FALSE, warning=FALSE}
+library(tidyverse)
+library(tidytext)
+```
+
+
+## Text
+
+```{r}
+text <- c("Because I could not stop for Death -",
+          "He kindly stopped for me -",
+          "The Carriage held but just Ourselves -",
+          "and Immortality")
+
+text
+```
+
+## A tidy table
+
+```{r}
+text_df <- tibble(line = 1:4, text = text)
+
+text_df
+```
+
+
+## Todenization
+
+```{r}
+text_df %>%
+  unnest_tokens(word, text)
+```
+