Example for study and suite (#810)

* fill in simple studies tutorial

* add benchmark suites tutorial

* add study tutorial

* Take Pieter's feedback into account

* [skip ci] missed on remark

* Take into account Arlind's comments

Author: mfeurer

examples/20_basic/simple_studies_tutorial.py

@@ -0,0 +1,66 @@
+"""
+================
+Benchmark suites
+================
+
+This is a brief showcase of OpenML benchmark suites, which were introduced by
+`Bischl et al. (2019) <https://arxiv.org/abs/1708.03731v2>`_. Benchmark suites standardize the
+datasets and splits to be used in an experiment or paper. They are fully integrated into OpenML
+and simplify both the sharing of the setup and the results.
+"""
+
+import openml
+
+####################################################################################################
+# OpenML-CC18
+# ===========
+#
+# As an example we have a look at the OpenML-CC18, which is a suite of 72 classification datasets
+# from OpenML which were carefully selected to be usable by many algorithms and also represent
+# datasets commonly used in machine learning research. These are all datasets from mid-2018 that
+# satisfy a large set of clear requirements for thorough yet practical benchmarking:
+#
+# 1. the number of observations are between 500 and 100,000 to focus on medium-sized datasets,
+# 2. the number of features does not exceed 5,000 features to keep the runtime of the algorithms
+#    low
+# 3. the target attribute has at least two classes with no class having less than 20 observations
+# 4. the ratio of the minority class and the majority class is above 0.05 (to eliminate highly
+#    imbalanced datasets which require special treatment for both algorithms and evaluation
+#    measures).
+#
+# A full description can be found in the `OpenML benchmarking docs
+# <https://docs.openml.org/benchmark/#openml-cc18>`_.
+#
+# In this example we'll focus on how to use benchmark suites in practice.
+
+####################################################################################################
+# Downloading benchmark suites
+# ============================
+
+suite = openml.study.get_suite(99)
+print(suite)
+
+####################################################################################################
+# The benchmark suite does not download the included tasks and datasets itself, but only contains
+# a list of which tasks constitute the study.
+#
+# Tasks can then be accessed via
+
+tasks = suite.tasks
+print(tasks)
+
+####################################################################################################
+# and iterated for benchmarking. For speed reasons we'll only iterate over the first three tasks:
+
+for task_id in tasks[:3]:
+    task = openml.tasks.get_task(task_id)
+    print(task)
+
+####################################################################################################
+# Further examples
+# ================
+#
+# * `Advanced benchmarking suites tutorial <../30_extended/suites_tutorial.html>`_
+# * `Benchmarking studies tutorial <../30_extended/study_tutorial.html>`_
+# * `Using studies to compare linear and non-linear classifiers
+#   <../40_paper/2018_ida_strang_example.html>`_

examples/20_basic/simple_suites_tutorial.py

@@ -0,0 +1,112 @@
+"""
+=================
+Benchmark studies
+=================
+
+How to list, download and upload benchmark studies.
+
+In contrast to `benchmark suites <https://docs.openml.org/benchmark/#benchmarking-suites>`_ which
+hold a list of tasks, studies hold a list of runs. As runs contain all information on flows and
+tasks, all required information about a study can be retrieved.
+"""
+############################################################################
+import uuid
+
+import numpy as np
+import sklearn.tree
+import sklearn.pipeline
+import sklearn.impute
+
+import openml
+
+
+############################################################################
+# .. warning:: This example uploads data. For that reason, this example
+#   connects to the test server at test.openml.org before doing so.
+#   This prevents the crowding of the main server with example datasets,
+#   tasks, runs, and so on.
+############################################################################
+
+
+############################################################################
+# Listing studies
+# ***************
+#
+# * Use the output_format parameter to select output type
+# * Default gives ``dict``, but we'll use ``dataframe`` to obtain an
+#   easier-to-work-with data structure
+
+studies = openml.study.list_studies(output_format='dataframe', status='all')
+print(studies.head(n=10))
+
+
+############################################################################
+# Downloading studies
+# ===================
+
+############################################################################
+# This is done based on the study ID.
+study = openml.study.get_study(123)
+print(study)
+
+############################################################################
+# Studies also features a description:
+print(study.description)
+
+############################################################################
+# Studies are a container for runs:
+print(study.runs)
+
+############################################################################
+# And we can use the evaluation listing functionality to learn more about
+# the evaluations available for the conducted runs:
+evaluations = openml.evaluations.list_evaluations(
+    function='predictive_accuracy',
+    output_format='dataframe',
+    study=study.study_id,
+)
+print(evaluations.head())
+
+############################################################################
+# Uploading studies
+# =================
+#
+# Creating a study is as simple as creating any kind of other OpenML entity.
+# In this examples we'll create a few runs for the OpenML-100 benchmark
+# suite which is available on the OpenML test server.
+
+openml.config.start_using_configuration_for_example()
+
+# Very simple classifier which ignores the feature type
+clf = sklearn.pipeline.Pipeline(steps=[
+    ('imputer', sklearn.impute.SimpleImputer()),
+    ('estimator', sklearn.tree.DecisionTreeClassifier(max_depth=5)),
+])
+
+suite = openml.study.get_suite(1)
+# We'll create a study with one run on three random datasets each
+tasks = np.random.choice(suite.tasks, size=3, replace=False)
+run_ids = []
+for task_id in tasks:
+    task = openml.tasks.get_task(task_id)
+    run = openml.runs.run_model_on_task(clf, task)
+    run.publish()
+    run_ids.append(run.run_id)
+
+# The study needs a machine-readable and unique alias. To obtain this,
+# we simply generate a random uuid.
+alias = uuid.uuid4().hex
+
+new_study = openml.study.create_study(
+    name='Test-Study',
+    description='Test study for the Python tutorial on studies',
+    run_ids=run_ids,
+    alias=alias,
+    benchmark_suite=suite.study_id,
+)
+new_study.publish()
+print(new_study)
+
+
+############################################################################
+openml.config.stop_using_configuration_for_example()

examples/30_extended/study_tutorial.py

@@ -0,0 +1,97 @@
+"""
+================
+Benchmark suites
+================
+
+How to list, download and upload benchmark suites.
+
+If you want to learn more about benchmark suites, check out our
+`brief introductory tutorial <../20_basic/simple_suites_tutorial.html>`_ or the
+`OpenML benchmark docs <https://docs.openml.org/benchmark/#benchmarking-suites>`_.
+"""
+############################################################################
+import uuid
+
+import numpy as np
+
+import openml
+
+
+############################################################################
+# .. warning:: This example uploads data. For that reason, this example
+#   connects to the test server at test.openml.org before doing so.
+#   This prevents the main server from crowding with example datasets,
+#   tasks, runs, and so on.
+############################################################################
+
+
+############################################################################
+# Listing suites
+# **************
+#
+# * Use the output_format parameter to select output type
+# * Default gives ``dict``, but we'll use ``dataframe`` to obtain an
+#   easier-to-work-with data structure
+
+suites = openml.study.list_suites(output_format='dataframe', status='all')
+print(suites.head(n=10))
+
+############################################################################
+# Downloading suites
+# ==================
+
+############################################################################
+# This is done based on the dataset ID.
+suite = openml.study.get_suite(99)
+print(suite)
+
+############################################################################
+# Suites also feature a description:
+print(suite.description)
+
+############################################################################
+# Suites are a container for tasks:
+print(suite.tasks)
+
+############################################################################
+# And we can use the task listing functionality to learn more about them:
+tasks = openml.tasks.list_tasks(output_format='dataframe')
+
+# Using ``@`` in `pd.DataFrame.query <
+# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html>`_
+# accesses variables outside of the current dataframe.
+tasks = tasks.query('tid in @suite.tasks')
+print(tasks.describe().transpose())
+
+############################################################################
+# Uploading suites
+# ================
+#
+# Uploading suites is as simple as uploading any kind of other OpenML
+# entity - the only reason why we need so much code in this example is
+# because we upload some random data.
+
+openml.config.start_using_configuration_for_example()
+
+# We'll take a random subset of at least ten tasks of all available tasks on
+# the test server:
+all_tasks = list(openml.tasks.list_tasks().keys())
+task_ids_for_suite = sorted(np.random.choice(all_tasks, replace=False, size=20))
+
+# The study needs a machine-readable and unique alias. To obtain this,
+# we simply generate a random uuid.
+
+alias = uuid.uuid4().hex
+
+new_suite = openml.study.create_benchmark_suite(
+    name='Test-Suite',
+    description='Test suite for the Python tutorial on benchmark suites',
+    task_ids=task_ids_for_suite,
+    alias=alias,
+)
+new_suite.publish()
+print(new_suite)
+
+
+############################################################################
+openml.config.stop_using_configuration_for_example()