Documentation for streaming API endpoints

Author: clue

README.md

@@ -73,6 +73,8 @@ The `Client` is responsible for assembling and sending HTTP requests to the Dock
 It requires a `Browser` object bound to the main `EventLoop` in order to handle async requests and a base URL.
 The recommended way to create a `Client` is using the `Factory` (see above).
 
+#### Commands
+
 All public methods on the `Client` resemble the API described in the [Remote API documentation](https://docs.docker.com/reference/api/docker_remote_api_v1.15/) like this:
 
 ```php
@@ -92,6 +94,8 @@ $client->version();
 
 Listing all available commands is out of scope here, please refer to the [Remote API documentation](https://docs.docker.com/reference/api/docker_remote_api_v1.15/) or the class outline.
 
+#### Promises
+
 Sending requests is async (non-blocking), so you can actually send multiple requests in parallel.
 Docker will respond to each request with a response message, the order is not guaranteed.
 Sending requests uses a [Promise](https://github.com/reactphp/promise)-based interface that makes it easy to react to when a request is fulfilled (i.e. either successfully resolved or rejected with an error):
@@ -107,6 +111,106 @@ $client->version()->then(
 });
 ```
 
+#### TAR streaming
+
+The following API endpoints resolve with a string in the [TAR file format](https://en.wikipedia.org/wiki/Tar_%28computing%29):
+
+```php
+$client->containerExport($container);
+$client->containerCopy($container, $config);
+```
+
+Keep in mind that this means the whole string has to be kept in memory.
+This is easy to get started and works reasonably well for smaller files/containers.
+
+For bigger containers it's usually a better idea to use a streaming approach,
+where only small chunks have to be kept in memory.
+This works for (any number of) files of arbitrary sizes.
+The following API endpoints complement the default Promise-based API and return
+a [`Stream`](https://github.com/reactphp/stream) instance instead:
+
+```php
+$stream = $client->containerExportStream($image);
+$stream = $client->containerCopyStream($image, $config);
+```
+
+Accessing individual files in the TAR file format string or stream is out of scope
+for this library.
+Several libraries are available, one that is known to work is [clue/tar-react](https://github.com/clue/php-tar-react).
+
+See also the [copy example](examples/copy.php) and the [export example](examples/export.php).
+
+#### JSON streaming
+
+The following API endpoints take advantage of [JSON streaming](https://en.wikipedia.org/wiki/JSON_Streaming):
+
+```php
+$client->imageCreate();
+$client->imagePush();
+```
+
+What this means is that these endpoints actually emit any number of progress
+events (individual JSON objects).
+
+The user-facing API hides this fact by resolving with an array of all individual
+progress events once the stream ends.
+
+These progress events can also be accessed individually via the Promise
+progress handler like this:
+
+```php
+$client->imageCreate('clue/streamripper')->then(
+    function ($data) {
+        // $data is an array of *all* elements in the JSON stream
+    },
+    function ($error) {
+        // an error occurred (possibly after receiving *some* elements)
+    },
+    function ($element) {
+        // will be invoked for *each* complete $element in the JSON stream
+    }
+);
+```
+
+Keep in mind that due to resolving with an array of all progress events,
+this API has to keep all event objects in memory until the Promise resolves.
+This is easy to get started and usually works reasonably well for the above
+API endpoints.
+
+If you're dealing with lots of concurrent requests (100+),
+it could be a better idea to use a streaming approach,
+where only individual progress event objects have to be kept in memory.
+The following API endpoints complement the default Promise-based API and return
+a [`Stream`](https://github.com/reactphp/stream) instance instead:
+     
+```php
+$stream = $client->imageCreateStream();
+$stream = $client->imagePushStream();
+```
+
+The resulting stream will emit the following events:
+
+* `progress`: for *each* element in the update stream
+* `error`:    once if an error occurs, will close() stream then
+* `close`:    once the stream ends (either finished or after "error")
+
+Please note that the resulting stream does not emit any "data" events, so
+you will not be able to pipe() its events into another `WritableStream`.
+
+```php
+$stream = $client->imageCreateStream('clue/redis-benchmark');
+$stream->on('progress', function ($data) {
+    // data will be emitted for *each* complete element in the JSON stream
+    echo $data['status'] . PHP_EOL;
+});
+$stream->on('close', function () {
+    // the JSON stream just ended, this could(?) be a good thing
+    echo 'Ended' . PHP_EOL;
+});
+```
+
+See also the [pull example](examples/pull.php) and the [push example](examples/push.php).
+
 ## Install
 
 The recommended way to install this library is [through composer](http://getcomposer.org). [New to composer?](http://getcomposer.org/doc/00-intro.md)

src/Client.php

@@ -149,9 +149,21 @@ public function containerChanges($container)
     /**
      * Export the contents of container id
      *
+     * This resolves with a string in the TAR file format containing all files
+     * in the container.
+     *
+     * Keep in mind that this means the whole string has to be kept in memory.
+     * For bigger containers it's usually a better idea to use a streaming
+     * approach, see containerExportStream() for more details.
+     *
+     * Accessing individual files in the TAR file format string is out of scope
+     * for this library. Several libraries are available, one that is known to
+     * work is clue/tar-react (see links).
+     *
      * @param string $container container ID
      * @return Promise Promise<string> tar stream
      * @link https://docs.docker.com/reference/api/docker_remote_api_v1.15/#export-a-container
+     * @link https://github.com/clue/php-tar-react library clue/tar-react
      * @see self::containerExportStream()
      */
     public function containerExport($container)
@@ -162,12 +174,23 @@ public function containerExport($container)
     /**
      * Export the contents of container id
      *
+     * This returns a stream in the TAR file format containing all files
+     * in the container.
+     *
+     * This works for containers of arbitrary sizes as only small chunks have to
+     * be kept in memory.
+     *
+     * Accessing individual files in the TAR file format stream is out of scope
+     * for this library. Several libraries are available, one that is known to
+     * work is clue/tar-react (see links).
+     *
      * The resulting stream is a well-behaving readable stream that will emit
      * the normal stream events.
      *
      * @param string $container container ID
      * @return ReadableStreamInterface tar stream
      * @link https://docs.docker.com/reference/api/docker_remote_api_v1.15/#export-a-container
+     * @link https://github.com/clue/php-tar-react library clue/tar-react
      * @see self::containerExport()
      */
     public function containerExportStream($container)
@@ -294,10 +317,22 @@ public function containerRemove($container, $v = false, $force = false)
     /**
      * Copy files or folders of container id
      *
+     * This resolves with a string in the TAR file format containing all files
+     * specified in the $config array.
+     *
+     * Keep in mind that this means the whole string has to be kept in memory.
+     * For bigger containers it's usually a better idea to use a streaming approach,
+     * see containerCopyStream() for more details.
+     *
+     * Accessing individual files in the TAR file format string is out of scope
+     * for this library. Several libraries are available, one that is known to
+     * work is clue/tar-react (see links).
+     *
      * @param string $container container ID
      * @param array  $config    resources to copy `array('Resource' => 'file.txt')` (see link)
      * @return Promise Promise<string> tar stream
      * @link https://docs.docker.com/reference/api/docker_remote_api_v1.15/#copy-files-or-folders-from-a-container
+     * @link https://github.com/clue/php-tar-react library clue/tar-react
      * @see self::containerCopyStream()
      */
     public function containerCopy($container, $config)
@@ -308,13 +343,24 @@ public function containerCopy($container, $config)
     /**
      * Copy files or folders of container id
      *
+     * This returns a stream in the TAR file format containing all files
+     * specified in the $config array.
+     *
+     * This works for (any number of) files of arbitrary sizes as only small chunks have to
+     * be kept in memory.
+     *
+     * Accessing individual files in the TAR file format stream is out of scope
+     * for this library. Several libraries are available, one that is known to
+     * work is clue/tar-react (see links).
+     *
      * The resulting stream is a well-behaving readable stream that will emit
      * the normal stream events.
      *
      * @param string $container container ID
      * @param array  $config    resources to copy `array('Resource' => 'file.txt')` (see link)
      * @return ReadableStreamInterface tar stream
      * @link https://docs.docker.com/reference/api/docker_remote_api_v1.15/#copy-files-or-folders-from-a-container
+     * @link https://github.com/clue/php-tar-react library clue/tar-react
      * @see self::containerCopy()
      */
     public function containerCopyStream($container, $config)
@@ -338,8 +384,11 @@ public function imageList($all = false)
     /**
      * Create an image, either by pulling it from the registry or by importing it
      *
-     * Will resolve with an array of all progress events. These can also be
-     * accessed via the Promise progress handler.
+     * This is a JSON streaming API endpoint that resolves with an array of all
+     * individual progress events.
+     *
+     * These progress events can also be accessed individually via the Promise
+     * progress handler.
      *
      * Pulling a private image from a remote registry will likely require authorization, so make
      * sure to pass the $registryAuth parameter, see `self::authHeaders()` for
@@ -365,6 +414,8 @@ public function imageCreate($fromImage = null, $fromSrc = null, $repo = null, $t
     /**
      * Create an image, either by pulling it from the registry or by importing it
      *
+     * This is a JSON streaming API endpoint that returns a stream instance.
+     *
      * The resulting stream will emit the following events:
      * - progress: for *each* element in the update stream
      * - error:    once if an error occurs, will close() stream then
@@ -423,8 +474,11 @@ public function imageHistory($image)
     /**
      * Push the image name on the registry
      *
-     * Will resolve with an array of all progress events. These can also be
-     * accessed via the Promise progress handler.
+     * This is a JSON streaming API endpoint that resolves with an array of all
+     * individual progress events.
+     *
+     * These progress events can also be accessed individually via the Promise
+     * progress handler.
      *
      * Pushing to a remote registry will likely require authorization, so make
      * sure to pass the $registryAuth parameter, see `self::authHeaders()` for
@@ -448,6 +502,8 @@ public function imagePush($image, $tag = null, $registry = null, $registryAuth =
     /**
      * Push the image name on the registry
      *
+     * This is a JSON streaming API endpoint that returns a stream instance.
+     *
      * The resulting stream will emit the following events:
      * - progress: for *each* element in the update stream
      * - error:    once if an error occurs, will close() stream then
@@ -465,7 +521,7 @@ public function imagePush($image, $tag = null, $registry = null, $registryAuth =
      * @param string|null $registry     (optional) the registry to push to (e.g. `registry.acme.com:5000`)
      * @param array|null  $registryAuth (optional) AuthConfig object (to send as X-Registry-Auth header)
      * @return ReadableStreamInterface stream of image push messages
-     * @uses authHeaders()
+     * @uses self::authHeaders()
      * @link https://docs.docker.com/reference/api/docker_remote_api_v1.15/#push-an-image-on-the-registry
      */
     public function imagePushStream($image, $tag = null, $registry = null, $registryAuth = null)