You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**Splitgraph** is a tool for building, versioning and querying reproducible datasets. It's inspired
11
-
by Docker and Git, so it feels familiar. And it's powered by [PostgreSQL](https://postgresql.org), so it [works seamlessly with existing tools](https://www.splitgraph.com/connect) in the Postgres ecosystem. Use Splitgraph to package your data into self-contained **data images** that you can [share with other Splitgraph instances](https://www.splitgraph.com/docs/getting-started/decentralized-demo).
12
-
13
-
[**Splitgraph.com**](https://www.splitgraph.com), or **Splitgraph Cloud**, is a public Splitgraph instance where you can share and discover data. It's a Splitgraph peer powered by the **Splitgraph Core** code in this repository, adding proprietary features like a data catalog, multitenancy, and a distributed SQL proxy.
11
+
**`sgr`** is the CLI for [**Splitgraph**](https://www.splitgraph.com), a
12
+
serverless API for data-driven Web applications.
14
13
15
-
You can explore [40k+ open datasets](https://www.splitgraph.com/explore) in the catalog. You can also connect directly to the [Data Delivery Network](https://www.splitgraph.com/connect) and query any of the datasets, without installing anything.
14
+
With addition of the optional [`sgr` Engine](engine/README.md) component, `sgr`
15
+
can become a stand-alone tool for building, versioning and querying reproducible
16
+
datasets. We use it as the storage engine for Splitgraph. It's inspired by
17
+
Docker and Git, so it feels familiar. And it's powered by
18
+
[PostgreSQL](https://postgresql.org), so it works seamlessly with existing tools
19
+
in the Postgres ecosystem. Use `sgr` to package your data into self-contained
20
+
**Splitgraph data images** that you can
21
+
[share with other `sgr` instances](https://www.splitgraph.com/docs/getting-started/decentralized-demo).
16
22
17
-
To install `sgr` (the command line client) or a local Splitgraph Engine, see the [Installation](#installation) section of this readme.
23
+
To install the `sgr` CLI or a local `sgr` Engine, see the
24
+
[Installation](#installation) section of this readme.
18
25
19
26
### Build and Query Versioned, Reproducible Datasets
20
27
21
-
[**Splitfiles**](https://www.splitgraph.com/docs/concepts/splitfiles) give you a declarative language, inspired by Dockerfiles, for expressing data transformations in ordinary SQL familiar to any researcher or business analyst. You can reference other images, or even other databases, with a simple JOIN.
28
+
[**Splitfiles**](https://www.splitgraph.com/docs/concepts/splitfiles) give you a
29
+
declarative language, inspired by Dockerfiles, for expressing data
30
+
transformations in ordinary SQL familiar to any researcher or business analyst.
31
+
You can reference other images, or even other databases, with a simple JOIN.
22
32
23
33
24
34
25
-
When you build data with Splitfiles, you get provenance tracking of the resulting data: it's possible to find out what sources went into every dataset and know when to rebuild it if the sources ever change. You can easily integrate Splitgraph into your existing CI pipelines, to keep your data up-to-date and stay on top of changes to upstream sources.
26
-
27
-
Splitgraph images are also version-controlled, and you can manipulate them with Git-like operations through a CLI. You can check out any image into a PostgreSQL schema and interact with it using any PostgreSQL client. Splitgraph will capture your changes to the data, and then you can commit them as delta-compressed changesets that you can package into new images.
28
-
29
-
Splitgraph supports PostgreSQL [foreign data wrappers](https://wiki.postgresql.org/wiki/Foreign_data_wrappers). We call this feature [mounting](https://www.splitgraph.com/docs/concepts/mounting). With mounting, you can query other databases (like PostgreSQL/MongoDB/MySQL) or open data providers (like [Socrata](https://www.splitgraph.com/docs/ingesting-data/socrata)) from your Splitgraph instance with plain SQL. You can even snapshot the results or use them in Splitfiles.
30
-
31
-
### Why Splitgraph?
32
-
33
-
Splitgraph isn't opinionated and doesn't break existing abstractions. To any existing PostgreSQL application, Splitgraph images are just another database. We have carefully designed Splitgraph to not break the abstraction of a PostgreSQL table and wire protocol, because doing otherwise would mean throwing away a vast existing ecosystem of applications, users, libraries and extensions. This means that a lot of tools that work with PostgreSQL work with Splitgraph out of the box.
35
+
When you build data images with Splitfiles, you get provenance tracking of the
36
+
resulting data: it's possible to find out what sources went into every dataset
37
+
and know when to rebuild it if the sources ever change. You can easily integrate
38
+
`sgr` your existing CI pipelines, to keep your data up-to-date and stay on top
39
+
of changes to upstream sources.
40
+
41
+
Splitgraph images are also version-controlled, and you can manipulate them with
42
+
Git-like operations through a CLI. You can check out any image into a PostgreSQL
43
+
schema and interact with it using any PostgreSQL client. `sgr` will capture your
44
+
changes to the data, and then you can commit them as delta-compressed changesets
45
+
that you can package into new images.
46
+
47
+
`sgr` supports PostgreSQL
48
+
[foreign data wrappers](https://wiki.postgresql.org/wiki/Foreign_data_wrappers).
49
+
We call this feature
50
+
[mounting](https://www.splitgraph.com/docs/concepts/mounting). With mounting,
51
+
you can query other databases (like PostgreSQL/MongoDB/MySQL) or open data
52
+
providers (like
53
+
[Socrata](https://www.splitgraph.com/docs/ingesting-data/socrata)) from your
54
+
`sgr` instance with plain SQL. You can even snapshot the results or use them in
55
+
Splitfiles.
34
56
35
57
36
58
37
59
## Components
38
60
39
-
The code in this repository, known as **Splitgraph Core**, contains:
61
+
The code in this repository contains:
40
62
41
-
-**[`sgr` command line client](https://www.splitgraph.com/docs/architecture/sgr-client)**: `sgr` is the main command line tool used to work with Splitgraph "images" (data snapshots). Use it to ingest data, work with splitfiles, and push data to Splitgraph.com.
42
-
-**[Splitgraph Engine](engine/README.md)**: a [Docker image](https://hub.docker.com/r/splitgraph/engine) of the latest Postgres with Splitgraph and other required extensions pre-installed.
43
-
-**[Splitgraph Python library](https://www.splitgraph.com/docs/python-api/splitgraph.core)**: All Splitgraph functionality is available in the Python API, offering first-class support for data science workflows including Jupyter notebooks and Pandas dataframes.
We also recommend reading our Blog, including some of our favorite posts:
53
83
54
-
-[Supercharging `dbt` with Splitgraph: versioning, sharing, cross-DB joins](https://www.splitgraph.com/blog/dbt)
84
+
-[Supercharging `dbt` with `sgr`: versioning, sharing, cross-DB joins](https://www.splitgraph.com/blog/dbt)
55
85
-[Querying 40,000+ datasets with SQL](https://www.splitgraph.com/blog/40k-sql-datasets)
56
86
-[Foreign data wrappers: PostgreSQL's secret weapon?](https://www.splitgraph.com/blog/foreign-data-wrappers)
57
87
58
88
## Installation
59
89
60
90
Pre-requisites:
61
91
62
-
- Docker is required to run the Splitgraph Engine. `sgr` must have access to Docker. You either need to [install Docker locally](https://docs.docker.com/install/) or have access to a remote Docker socket.
92
+
- Docker is required to run the `sgr` Engine. `sgr` must have access to Docker.
93
+
You either need to [install Docker locally](https://docs.docker.com/install/)
94
+
or have access to a remote Docker socket.
63
95
64
-
For Linux and OSX, once Docker is running, install Splitgraph with a single script:
This will download the `sgr` binary and set up the Splitgraph Engine Docker container.
71
-
72
-
Alternatively, you can get the `sgr` single binary from [the releases page](https://github.com/splitgraph/splitgraph/releases) and run [`sgr engine add`](https://www.splitgraph.com/docs/sgr/engine-management/engine-add) to create an engine.
108
+
This will download the `sgr` binary and set up the `sgr` Engine Docker
109
+
container.
73
110
74
-
See the [installation guide](https://www.splitgraph.com/docs/getting-started/installation) for more installation methods.
111
+
See the
112
+
[installation guide](https://www.splitgraph.com/docs/sgr-cli/installation) for
113
+
more installation methods.
75
114
76
115
## Quick start guide
77
116
78
-
You can follow the [quick start guide](https://www.splitgraph.com/docs/getting-started/five-minute-demo) that will guide you through the basics of using Splitgraph with public and private data.
that will guide you through the basics of using `sgr` with Splitgraph or
120
+
standalone.
79
121
80
-
Alternatively, Splitgraph comes with plenty of [examples](examples) to get you started.
122
+
Alternatively, `sgr` comes with plenty of [examples](examples) to get you
123
+
started.
81
124
82
-
If you're stuck or have any questions, check out the [documentation](https://www.splitgraph.com/docs/) or join our [Discord channel](https://discord.gg/4Qe2fYA)!
125
+
If you're stuck or have any questions, check out the
0 commit comments