Update README.md

Author: chrisseto

whitepapers/README.md

@@ -1,4 +1,7 @@
-# The SHARE Pipeline
+# The SHARE System Spec (Code name: Tamandua)
+
+This set of documents exist to both formalize and restructure how information is processed by SHARE.
+Over time the "pipline" has changed significantly from its original inception. The vocabulary to describe the process has not, leading to a decent amount of confusion.
 
 ## Vocabulary
 
@@ -7,21 +10,40 @@
 * State -- Source specific version of a given SHARE Object.
 * Final -- The representation of an object in the publicly accessibly SHARE dataset.
 * Harvest -- Collecting data from 
-* Transform --
-* [Data] Cleaning --
 * Normalize -- Defunct; see Transform.
+* Harvester -- Code responsible for aquiring data to be processed by SHARE
+* Transformer -- Code responsible for reformatting harvested data into a SHARE compliant format
+* Regulators -- Code responsible for cleaning and validating values expelled by a transformer
+* Deduplicator -- The job/code responsible for matching states together to be assembled into the final data set
+* Assembler -- The code responsible for selecting the best attributes from individual states
+
+### Why did you pick X word?
+* Harvest -- The meaning of this word never actually changed. Used for historic reasons.
+* [Transform](http://www.dictionary.com/browse/transform) -- To change in form, appearance, or structure; metamorphose.
+* [Regulate](http://www.dictionary.com/browse/regulate) -- To control or direct by a rule, principle, method, etc.
+* [Deduplicate]() --
+* [Assemble](http://www.dictionary.com/browse/assemble) -- To put or fit together; put together the parts of.
 
 ## Overview
 
-### Main pipeline:
+### There is no pipeline
+
+```
+   +--------------+      +--------------------------------------------+
+   | Harvest Task | ---> | Transform  --->  Regulate ---> Consolidate |
+   +--------------+      +--------------------------------------------+
 
-* Data Ingest Task -- Collect and store data
-* Data Process Task -- Parse and clean data
+   +---------------+
+   | Deduplication |
+   +---------------+
 
-### Auxillary Tasks
+   +----------+
+   | Assemble |
+   +----------+
+```
 
-* Data Disambiguate Task -- Link together individual records
-* Data Build Task -- Intellgently aggregate records describing the same object
+At first glance, the SHARE workflow may appear to be a pipeline. It is not. Data only addressed a single enity, briefly, in the begining of the workflow.
+Once data is full processed, it is treated as part of a whole. All jobs, aside from the first two boxes, operate on the dataset as a whole.
 
 ### Janitor
 
@@ -54,11 +76,21 @@ The rebuilds will heal any issues in the final dataset
 
 
 
-
 Single global diambigation task
 Store tags and subjects as arrays
 SUIDS as another table. Lock to prevent racing on rawdata updating
 
 # Ideas
 Counters for started, finished, etc on log jobs
 Janitor actually deletes states marked as `is_deleted`
+
+Move sources to their own table
+
+
+Harvesters
+Transformers
+Regulators
+
+
+Deduplicator
+Assembler