You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+34-14Lines changed: 34 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,9 +2,13 @@
2
2
3
3
by Abishek Ramdas and Ghislain Fourny
4
4
5
-
This is the Python edition of [RumbleDB](https://rumbledb.org/), which brings [JSONiq](https://www.jsoniq.org) to the world of Spark and DataFrames. JSONiq is a language considerably more powerful than SQL as it can process [messy, heterogeneous datasets](https://arxiv.org/abs/1910.11582), from kilobytes to Petabytes, with very little coding effort.
5
+
This is the Python edition of [RumbleDB](https://rumbledb.org/), which brings [JSONiq](https://www.jsoniq.org) to the world of Python.
6
6
7
-
The Python edition of RumbleDB is currently only a prototype (alpha) and probably unstable. We welcome bug reports.
7
+
JSONiq is a language considerably more powerful than SQL as it can process [messy, heterogeneous datasets](https://arxiv.org/abs/1910.11582), from kilobytes to Petabytes, with very little coding effort.
8
+
9
+
Spark aficionados can also pass DataFrames to JSONiq queries and take back DataFrames. This gives them an environment in which both Spark SQL and JSONiq co-exist to manipulate the data.
10
+
11
+
The Python edition of RumbleDB is currently a prototype (alpha) and probably unstable. We welcome bug reports and feedback.
8
12
9
13
## About RumbleDB
10
14
@@ -24,20 +28,33 @@ A RumbleSession is a wrapper around a SparkSession that additionally makes sure
24
28
25
29
JSONiq queries are invoked with rumble.jsoniq() in a way similar to the way Spark SQL queries are invoked with spark.sql().
26
30
27
-
Any number of Python DataFrames can be attached to external JSONiq variables used in the query. It will later also be possible to read tables registered in the Hive metastore, similar to spark.sql(). Alternatively, the JSONiq query can also read many files of many different formats from many places (local drive, HTTP, S3, HDFS, ...) directly with simple builtin function calls such as json-lines(), text-file(), parquet-file(), csv-file(), etc. See [RumbleDB's documentation](https://rumble.readthedocs.io/en/latest/).
28
-
29
-
The resulting sequence of items can be retrieved as DataFrame, as an RDD, as a Python list, or with a streaming iteration over the items.
31
+
JSONiq variables can be bound to lists of JSON values (str, int, float, True, False, None, dict, list) or to Pyspark DataFrames. A JSONiq query can use as many variables as needed (for example, it can join between different collections).
30
32
31
-
The individual items can be processed using the RumbleDB [Item API](https://github.com/RumbleDB/rumble/blob/master/src/main/java/org/rumbledb/api/Item.java).
33
+
It will later also be possible to read tables registered in the Hive metastore, similar to spark.sql(). Alternatively, the JSONiq query can also read many files of many different formats from many places (local drive, HTTP, S3, HDFS, ...) directly with simple builtin function calls such as json-lines(), text-file(), parquet-file(), csv-file(), etc. See [RumbleDB's documentation](https://rumble.readthedocs.io/en/latest/).
32
34
33
-
Alternatively, it is possible to directly get a Python list of JSON values, or a streaming iteration of JSON values. This is a convenience that makes it unnecessary to use the Item API, especially for a first-time user.
35
+
The resulting sequence of items can be retrieved as a list of JSON values, as a Pyspark DataFrame, or, for advanced users, as an RDD or with a streaming iteration over the items using the [RumbleDB Item API](https://github.com/RumbleDB/rumble/blob/master/src/main/java/org/rumbledb/api/Item.java).
34
36
35
-
It is also possible to write the sequence of items to the local disk, to HDFS, to S3, etc in a way similar to how DataFrames are written back by Spark.
37
+
It is also possible to write the sequence of items to the local disk, to HDFS, to S3, etc in a way similar to how DataFrames are written back by Pyspark.
36
38
37
-
The design goal is that it should be possible to chain DataFrames between JSONiq and Spark SQL queries seamlessly. For example, JSONiq can be used to clean up very messy data and turn it into a clean DataFrame, which can then be processed with Spark SQL, spark.ml, etc.
39
+
The design goal is that it is possible to chain DataFrames between JSONiq and Spark SQL queries seamlessly. For example, JSONiq can be used to clean up very messy data and turn it into a clean DataFrame, which can then be processed with Spark SQL, spark.ml, etc.
38
40
39
41
Any feedback or error reports are very welcome.
40
42
43
+
## Type mapping
44
+
45
+
When passing Python values to JSONiq or getting them from a JSONiq queries, the mapping is as follows:
46
+
47
+
| Python | JSONiq |
48
+
|-------|-------|
49
+
|dict|object|
50
+
|list|array|
51
+
|str|string|
52
+
|int|integer|
53
+
|bool|boolean|
54
+
|None|null|
55
+
56
+
Furthermore, other JSONiq types will be mapped to string literals. Users who want to preserve JSONiq types can use the Item API instead.
57
+
41
58
## Installation
42
59
43
60
Install with
@@ -49,7 +66,7 @@ pip install jsoniq
49
66
50
67
## Sample code
51
68
52
-
We will make more documentation available as we go. In the meantime, you will find a sample code below that should just run
69
+
We will make more documentation available as we go. In the meantime, you will find a sample, commented code below that should just run
53
70
after installing the library.
54
71
55
72
You can directly copy paste the code below to a Python file and execute it with Python.
@@ -60,9 +77,11 @@ from jsoniq import RumbleSession
60
77
# The syntax to start a session is similar to that of Spark.
61
78
# A RumbleSession is a SparkSession that additionally knows about RumbleDB.
62
79
# All attributes and methods of SparkSession are also available on RumbleSession.
80
+
63
81
rumble = RumbleSession.builder.getOrCreate();
64
82
65
83
# Just to improve readability when invoking Spark methods
84
+
# (such as spark.sql() or spark.createDataFrame()).
66
85
spark = rumble
67
86
68
87
##############################
@@ -75,10 +94,11 @@ spark = rumble
75
94
# of items, here the sequence with just the integer item 2.
76
95
items = rumble.jsoniq('1+1')
77
96
78
-
# A sequence of items can simply be converted to a list of Python values with json().
79
-
# Since there is only one value in the sequence output by this query, we get a singleton list with the integer 2.
97
+
# A sequence of items can simply be converted to a list of Python/JSON values with json().
98
+
# Since there is only one value in the sequence output by this query,
99
+
# we get a singleton list with the integer 2.
100
+
# Generally though, the results may contain zero, one, two, or more items.
80
101
python_list = items.json()
81
-
82
102
print(python_list)
83
103
84
104
############################################
@@ -141,7 +161,7 @@ print(seq.json());
141
161
# queries are sequence of items.
142
162
# A Python list will be seamlessly converted to a sequence of items by the library.
143
163
# Currently we only support strs, ints, floats, booleans, None, lists, and dicts.
144
-
# But if you need more (like date, bytes, etc) we can add them without any problem.
164
+
# But if you need more (like date, bytes, etc) we will add them without any problem.
0 commit comments