Merge pull request #9 from clue-labs/streaming

Add streaming API

Author: clue

README.md

@@ -73,6 +73,8 @@ The `Client` is responsible for assembling and sending HTTP requests to the Dock
 It requires a `Browser` object bound to the main `EventLoop` in order to handle async requests and a base URL.
 The recommended way to create a `Client` is using the `Factory` (see above).
 
+#### Commands
+
 All public methods on the `Client` resemble the API described in the [Remote API documentation](https://docs.docker.com/reference/api/docker_remote_api_v1.15/) like this:
 
 ```php
@@ -92,6 +94,8 @@ $client->version();
 
 Listing all available commands is out of scope here, please refer to the [Remote API documentation](https://docs.docker.com/reference/api/docker_remote_api_v1.15/) or the class outline.
 
+#### Promises
+
 Sending requests is async (non-blocking), so you can actually send multiple requests in parallel.
 Docker will respond to each request with a response message, the order is not guaranteed.
 Sending requests uses a [Promise](https://github.com/reactphp/promise)-based interface that makes it easy to react to when a request is fulfilled (i.e. either successfully resolved or rejected with an error):
@@ -107,6 +111,128 @@ $client->version()->then(
 });
 ```
 
+#### TAR streaming
+
+The following API endpoints resolve with a string in the [TAR file format](https://en.wikipedia.org/wiki/Tar_%28computing%29):
+
+```php
+$client->containerExport($container);
+$client->containerCopy($container, $config);
+```
+
+Keep in mind that this means the whole string has to be kept in memory.
+This is easy to get started and works reasonably well for smaller files/containers.
+
+For bigger containers it's usually a better idea to use a streaming approach,
+where only small chunks have to be kept in memory.
+This works for (any number of) files of arbitrary sizes.
+The following API endpoints complement the default Promise-based API and return
+a [`Stream`](https://github.com/reactphp/stream) instance instead:
+
+```php
+$stream = $client->containerExportStream($image);
+$stream = $client->containerCopyStream($image, $config);
+```
+
+Accessing individual files in the TAR file format string or stream is out of scope
+for this library.
+Several libraries are available, one that is known to work is [clue/tar-react](https://github.com/clue/php-tar-react).
+
+See also the [copy example](examples/copy.php) and the [export example](examples/export.php).
+
+#### JSON streaming
+
+The following API endpoints take advantage of [JSON streaming](https://en.wikipedia.org/wiki/JSON_Streaming):
+
+```php
+$client->imageCreate();
+$client->imagePush();
+```
+
+What this means is that these endpoints actually emit any number of progress
+events (individual JSON objects).
+At the HTTP level, a common response message could look like this:
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json
+
+{"status":"loading","current":1,"total":10}
+{"status":"loading","current":2,"total":10}
+…
+{"status":"loading","current":10,"total":10}
+{"status":"done","total":10}
+```
+
+The user-facing API hides this fact by resolving with an array of all individual
+progress events once the stream ends:
+
+```php
+$client->imageCreate('clue/streamripper')->then(
+    function ($data) {
+        // $data is an array of *all* elements in the JSON stream
+    },
+    function ($error) {
+        // an error occurred (possibly after receiving *some* elements)
+        
+        if ($error instanceof Io\JsonProgressException) {
+            // a progress message (usually the last) contains an error message
+        } else {
+            // any other error, like invalid request etc.
+        }
+    }
+);
+```
+
+Keep in mind that due to resolving with an array of all progress events,
+this API has to keep all event objects in memory until the Promise resolves.
+This is easy to get started and usually works reasonably well for the above
+API endpoints.
+
+If you're dealing with lots of concurrent requests (100+) or 
+if you want to access the individual progress events as they happen, you
+should consider using a streaming approach instead,
+where only individual progress event objects have to be kept in memory.
+The following API endpoints complement the default Promise-based API and return
+a [`Stream`](https://github.com/reactphp/stream) instance instead:
+     
+```php
+$stream = $client->imageCreateStream();
+$stream = $client->imagePushStream();
+```
+
+The resulting stream will emit the following events:
+
+* `progress`: for *each* element in the update stream
+* `error`:    once if an error occurs, will close() stream then
+  * Will emit an [`Io\JsonProgressException`](#jsonprogressexception) if an individual progress message contains an error message
+  * Any other `Exception` in case of an transport error, like invalid request etc.
+* `close`:    once the stream ends (either finished or after "error")
+
+Please note that the resulting stream does not emit any "data" events, so
+you will not be able to pipe() its events into another `WritableStream`.
+
+```php
+$stream = $client->imageCreateStream('clue/redis-benchmark');
+$stream->on('progress', function ($data) {
+    // data will be emitted for *each* complete element in the JSON stream
+    echo $data['status'] . PHP_EOL;
+});
+$stream->on('close', function () {
+    // the JSON stream just ended, this could(?) be a good thing
+    echo 'Ended' . PHP_EOL;
+});
+```
+
+See also the [pull example](examples/pull.php) and the [push example](examples/push.php).
+
+### JsonProgressException
+
+The `Io\JsonProgressException` will be thrown by [JSON streaming](#json-streaming)
+endpoints if an individual progress message contains an error message.
+
+The `getData()` method can be used to obtain the progress message.
+
 ## Install
 
 The recommended way to install this library is [through composer](http://getcomposer.org). [New to composer?](http://getcomposer.org/doc/00-intro.md)

composer.json

@@ -16,7 +16,12 @@
     "require": {
         "php": ">=5.3",
         "react/event-loop": "~0.3.0|~0.4.0",
-        "clue/buzz-react": "~0.2.0",
-        "react/promise": "~1.0|~2.0"
+        "clue/buzz-react": "~0.3.0",
+        "react/promise": "~1.0|~2.0",
+        "clue/json-stream": "~0.1.0"
+    },
+    "require-dev": {
+        "clue/tar-react": "~0.1.0",
+        "clue/caret-notation": "~0.2.0"
     }
 }

examples/copy.php

@@ -0,0 +1,49 @@
+<?php
+// this example shows how the containerCopy() call returns a TAR stream,
+// how it can be passed to a TAR decoder and how we can then pipe each
+// individual file to the console output.
+
+require __DIR__ . '/../vendor/autoload.php';
+
+use React\EventLoop\Factory as LoopFactory;
+use Clue\React\Docker\Factory;
+use Clue\React\Tar\Decoder;
+use React\Stream\ReadableStreamInterface;
+use Clue\CaretNotation\Encoder;
+
+$container = isset($argv[1]) ? $argv[1] : 'asd';
+$file = isset($argv[2]) ? $argv[2] : '/etc/passwd';
+echo 'Container "' . $container . '" dumping "' . $file . '" (pass as arguments to this example)' . PHP_EOL;
+
+$loop = LoopFactory::create();
+
+$factory = new Factory($loop);
+$client = $factory->createClient();
+
+$stream = $client->containerCopyStream($container, array('Resource' => $file));
+
+$tar = new Decoder();
+
+// use caret notation for any control characters expect \t, \r and \n
+$caret = new Encoder("\t\r\n");
+
+$tar->on('entry', function ($header, ReadableStreamInterface $file) use ($caret) {
+    // write each entry to the console output
+    echo '########## ' . $caret->encode($header['filename']) . ' ##########' . PHP_EOL;
+    $file->on('data', function ($chunk) use ($caret) {
+        echo $caret->encode($chunk);
+    });
+});
+
+$tar->on('error', function ($e = null) {
+    // should not be invoked, unless the stream is somehow interrupted
+    echo 'ERROR processing tar stream' . PHP_EOL . $e;
+});
+$stream->on('error', function ($e = null) {
+    // will be called if either parameter is invalid
+    echo 'ERROR requesting stream' . PHP_EOL . $e;
+});
+
+$stream->pipe($tar);
+
+$loop->run();

examples/export.php

@@ -0,0 +1,30 @@
+<?php
+// this example shows how the containerExport() call returns a TAR stream
+// and how we it can be piped into a output tar file.
+
+require __DIR__ . '/../vendor/autoload.php';
+
+use React\EventLoop\StreamSelectLoop;
+use Clue\React\Docker\Factory;
+use React\Stream\Stream;
+
+$container = isset($argv[1]) ? $argv[1] : 'asd';
+$target = isset($argv[2]) ? $argv[2] : ($container . '.tar');
+echo 'Exporting whole container "' . $container . '" to "' . $target .'" (pass as arguments to this example)' . PHP_EOL;
+
+$loop = new StreamSelectLoop();
+
+$factory = new Factory($loop);
+$client = $factory->createClient();
+
+$stream = $client->containerExportStream($container);
+
+$stream->on('error', function ($e = null) {
+    // will be called if the container is invalid/does not exist
+    echo 'ERROR requesting stream' . PHP_EOL . $e;
+});
+
+$out = new Stream(fopen($target, 'w'), $loop);
+$stream->pipe($out);
+
+$loop->run();

examples/pull.php

@@ -0,0 +1,28 @@
+<?php
+// this example shows how the imageCreateStream() call can be used to pull a given image.
+// demonstrates the JSON streaming API, individual progress events will be printed as they happen.
+
+require __DIR__ . '/../vendor/autoload.php';
+
+use React\EventLoop\Factory as LoopFactory;
+use Clue\React\Docker\Factory;
+
+$image = isset($argv[1]) ? $argv[1] : 'clue/redis-benchmark';
+echo 'Pulling image "' . $image . '" (pass as argument to this example)' . PHP_EOL;
+
+$loop = LoopFactory::create();
+
+$factory = new Factory($loop);
+$client = $factory->createClient();
+
+$stream = $client->imageCreateStream($image);
+
+$stream->on('progress', function ($progress) {
+    echo 'progress: '. json_encode($progress) . PHP_EOL;
+});
+
+$stream->on('close', function () {
+    echo 'stream closed' . PHP_EOL;
+});
+
+$loop->run();

examples/push.php

@@ -0,0 +1,26 @@
+<?php
+// this example shows how the imagePush() call can be used to publish a given image.
+// this requires authorization and this example includes some invalid defaults.
+
+require __DIR__ . '/../vendor/autoload.php';
+
+use React\EventLoop\Factory as LoopFactory;
+use Clue\React\Docker\Factory;
+
+$image = isset($argv[1]) ? $argv[1] : 'asd';
+$auth = json_decode('{"username": "string", "password": "string", "email": "string", "serveraddress" : "string", "auth": ""}');
+echo 'Pushing image "' . $image . '" (pass as argument to this example)' . PHP_EOL;
+
+$loop = LoopFactory::create();
+
+$factory = new Factory($loop);
+$client = $factory->createClient();
+
+$client->imagePush($image, null, null, $auth)->then(
+    function ($response) {
+        echo 'response: ' . json_encode($response) . PHP_EOL;
+    },
+    'var_dump'
+);
+
+$loop->run();