BackgroundProducer adaptive sleep

Author: mlin

README.md

@@ -1,6 +1,6 @@
 # Genomics Extension for SQLite
 
-**("GenomicSQLite")**
+## ("GenomicSQLite")
 
 This [SQLite3 loadable extension](https://www.sqlite.org/loadext.html) adds features to the [ubiquitous](https://www.sqlite.org/mostdeployed.html) embedded RDBMS supporting applications in genome bioinformatics:
 
@@ -10,8 +10,32 @@ This [SQLite3 loadable extension](https://www.sqlite.org/loadext.html) adds feat
 
 Notice: this project is not associated with the SQLite developers.
 
+### Use cases
+
+The extension makes SQLite an efficient foundation for:
+
+1. Integrative genomics data warehouse
+
+    * BED, GFF/GTF, FASTA, FASTQ, SAM, VCF, ...
+
+    * One file, zero administration, portable between platforms and languages
+
+2. Slicing & basic analysis with indexed SQL queries, joins, & aggregations
+
+3. Transactional storage engine for API services, incremental reanalysis, real-time basecalling & metagenomics, ...
+
+4. Experimental new data models, before dedicated storage format & tooling are warranted, if ever.
+
+### Contraindications
+
+1. Huge numerical arrays: see [HDF5](https://www.hdfgroup.org/solutions/hdf5/), [Zarr](https://zarr.readthedocs.io/en/stable/), [Parquet](https://parquet.apache.org/), [Arrow](https://arrow.apache.org/). <small>SQLite's [BLOB I/O](https://www.sqlite.org/c3ref/blob_open.html) leaves the door open for mash-ups!</small>
+
+2. Parallel SQL analytics / OLAP: see [Spark](https://spark.apache.org/), [DuckDB](https://duckdb.org/), many commercial products. <small>Some bases can be covered with a sharding harness for a pool of threads with their own SQLite connections...</small>
+
+3. Streaming: SQLite I/O, while often highly sequential in practice, relies on randomly seeking throughout the database file.
+
 ## Under construction
 
 ![build](https://github.com/mlin/GenomicSQLite/workflows/build/badge.svg?branch=main)
 
-The extension isn't quite ready for general use. The repo is public to facilitate "soft launch" preparations.
+The extension isn't quite ready for general use. The repo is public while we work on packaging and documentation.

loaders/common.hpp

@@ -169,11 +169,15 @@ template <class Item> class BackgroundProducer {
             if (ok) {
                 p_.store(++p, memory_order_release);
                 if (p - max(c_.load(memory_order_acquire), 1LL) == R_ - 1) {
+                    // Ring is full; enter semi-busy wait with adaptive sleep. Goal is to keep the
+                    // consumer well-fed, without excessively wasteful busy-loop, using atomics for
+                    // coordination. Underlying assumption: producer is usually faster than the
+                    // consumer, while the ring provides a buffer if it has the occasional hiccup.
                     auto t_spin = chrono::high_resolution_clock::now();
+                    long long spin = 0;
                     do {
-                        // assumption -- producer is usually faster than the consumer, and ringsize
-                        // provides a buffer if that's occasionally not the case
-                        this_thread::sleep_for(chrono::nanoseconds(10000));
+                        this_thread::sleep_for(
+                            chrono::nanoseconds(10000 + 990000 * min(spin++, 100LL) / 100));
                     } while (!stop_.load(memory_order_relaxed) &&
                              (p - max(c_.load(memory_order_acquire), 1LL) == R_ - 1));
                     p_blocked_ += chrono::high_resolution_clock::now() - t_spin;
@@ -194,8 +198,8 @@ template <class Item> class BackgroundProducer {
 
     virtual ~BackgroundProducer() { abort(); }
 
-    // advance to next item for consumption, return false when item stream has ended successfully,
-    // or throw an exception.
+    // advance to next item for consumption & return true, return false when item stream has ended
+    // with success, or throw an exception.
     bool next() {
         if (!worker_) {
             while (ring_.size() < R_) {
@@ -205,6 +209,7 @@ template <class Item> class BackgroundProducer {
         }
         long long p = p_.load(memory_order_acquire), c = c_.load(memory_order_relaxed);
         if (c == p) {
+            // Ring is empty; enter busy wait
             auto t_spin = chrono::high_resolution_clock::now();
             while (c == p && !stop_.load(memory_order_relaxed)) {
                 this_thread::yield();