Initial data formats for tagging data quality (#5554)

* First version for flagging time ranges for analysis

Conflicts:
	DataFormats/CMakeLists.txt

* Add detector ranges

* Reworked FlagReasons and Time-Ranges

* Add codeowners

* typo in codeowners

* Use Bracket from MathUtils for the time interval

* Allow for persistent storage of FlagReasons

Unfortunately, it implies larger memory footprint and no constexpr construction,
 but i don't see other way to achieve it ATM.

* Delete unneeded ClassImp and fix typo

Co-authored-by: wiechula <Jens.Wiechula@cern.ch>

Author: knopers8

CODEOWNERS

@@ -50,6 +50,7 @@
 #/DataFormats/Legacy
 #/DataFormats/MemoryResources
 /DataFormats/Parameters        @shahor02
+/DataFormats/QualityControl    @knopers8 @Barthelemy @chiarazampolli
 /DataFormats/Reconstruction    @shahor02
 #/DataFormats/TimeFrame
 /DataFormats/common            @shahor02

DataFormats/CMakeLists.txt

@@ -7,4 +7,4 @@ add_subdirectory(simulation)
 add_subdirectory(TimeFrame)
 add_subdirectory(Parameters)
 add_subdirectory(Calibration)
-
+add_subdirectory(QualityControl)

DataFormats/QualityControl/CMakeLists.txt

@@ -0,0 +1,37 @@
+# Copyright CERN and copyright holders of ALICE O2. This software is distributed
+# under the terms of the GNU General Public License v3 (GPL Version 3), copied
+# verbatim in the file "COPYING".
+#
+# See http://alice-o2.web.cern.ch/license for full licensing information.
+#
+# In applying this license CERN does not waive the privileges and immunities
+# granted to it by virtue of its status as an Intergovernmental Organization or
+# submit itself to any jurisdiction.
+
+o2_add_library(DataFormatsQualityControl
+               SOURCES src/FlagReasons.cxx
+                       src/TimeRangeFlag.cxx
+                       src/TimeRangeFlagCollection.cxx
+  PUBLIC_LINK_LIBRARIES O2::Headers
+                        O2::FrameworkLogger
+                        O2::DetectorsCommonDataFormats)
+
+o2_target_root_dictionary(DataFormatsQualityControl
+                          HEADERS include/DataFormatsQualityControl/FlagReasons.h
+                                  include/DataFormatsQualityControl/TimeRangeFlag.h
+                                  include/DataFormatsQualityControl/TimeRangeFlagCollection.h)
+
+o2_add_test(FlagReasons
+  SOURCES test/testFlagReasons.cxx
+  COMPONENT_NAME DataFormatsQualityControl
+  PUBLIC_LINK_LIBRARIES O2::DataFormatsQualityControl)
+
+o2_add_test(TimeRangeFlag
+            SOURCES test/testTimeRangeFlag.cxx
+            COMPONENT_NAME DataFormatsQualityControl
+            PUBLIC_LINK_LIBRARIES O2::DataFormatsQualityControl)
+
+o2_add_test(TimeRangeFlagCollection
+            SOURCES test/testTimeRangeFlagCollection.cxx
+            COMPONENT_NAME DataFormatsQualityControl
+            PUBLIC_LINK_LIBRARIES O2::DataFormatsQualityControl)

DataFormats/QualityControl/README.md

@@ -0,0 +1,100 @@
+\page refDataFormatsQualityControl Data Formats Quality Control
+
+Data formats for tagging good quality data for analysis.
+
+# Flagging time ranges
+## General idea
+* Each detector has its own CCDB (QCDB) entry - TimeRangeFlagCollection
+* The CCDB Entry validity will be run or fill, depending on the final data taking granularity
+* Each entry can define sub ranges (TimeRangeFlags) of certain data characteristics (FlagReasons) inside the CCDB Entry validity
+* Flags are defined in a common store, they are used to derive the Data Tags - the final filters for good quality data during analysis.
+* Data Tag are stored as CCDB entries. They might require different detectors and may suppress different flags dependent on the analysis type.
+
+## Implementation
+
+[Flag Reason](include/DataFormatsQualityControl/FlagReasons.h) is defined with an identifier number, a name and a 'bad quality' determinant.
+The latter decides if such a flag should mark the data quality as bad by default.
+FlagReasons can be created only with FlagReasonFactory, so that the list of available reasons is common and centralised. 
+For example:
+```
+id: 10
+name: Limited Acceptance
+bad: true
+```
+
+With [TimeRangeFlags](include/DataFormatsQualityControl/TimeRangeFlag.h) we can define the time range of a chosen FlagReason, add an additional comment and specify the source of this flag.
+For example:
+```
+start: 1612707603626 
+end: 1613999652000
+flag: Limited Acceptance
+comment: Sector B in TPC inactive
+source: o2::quality_control_modules::tpc::ClustersCheck
+```
+
+[TimeRangeFlagCollection](include/DataFormatsQualityControl/TimeRangeFlagCollection.h) contains all TimeRangeFlags for the validity range (run or fill).
+TimeRangeFlags may overlap, e.g. if they use different FlagReasons and they are sorted by their start time.
+If certain period does not include any TimeRangeFlags with *bad* FlagReasons, then the data quality can be considered as good.
+The [TimeRangeFlagCollection test](test/testTimeRangeFlagCollection.cxx) shows the usage example.
+
+TimeRangeFlagCollections are supposed to be created automatically with QC Post-processing Tasks based on Quality Objects created by QC Checks.
+However, they might be created manually by QA experts as well.
+The procedure to do that has to be defined.
+
+## TODO
+* Define the complete list of available Flag Reasons
+* implement CCDB storage and access
+  - define CCDB storage place e.g.
+    * `<Detector>/QC/QualityFlags`
+    * `Analysis/QualityFlags/<Detector>`
+* Data Tags Definitions and Data Tags
+
+### Notes on plans for Data Tags
+
+Data Tag definition has:
+* name
+* global suppression list for bad flag reasons (applies to all detectors)
+* global requirement list for not bad flag reasons
+* list of needed detectors,
+* list of suppression list and requirement list specific to detectors
+
+Example configurations
+```json
+{
+  "name": "CentralBarrelTrackingLimitedAcceptance",
+  "globalRequiredDetectors" : [ "TPC", "ITS" ],
+  "globalIgnoredFlags" : ["Limited acceptance", "asdf"],
+  "globalRequiredFlags" : [],
+  "detectorSpecificFlags" : [
+    {
+      "name" : "TPC",
+      "ignoredFlags" : [ "Reason X" ]
+    }
+  ]
+},
+{
+  "name": "CentralBarrelTracking",
+  "globalRequiredDetectors" : [ "TPC", "ITS" ],
+  "globalIgnoredFlags" : [],
+  "globalRequiredFlags" : [],
+  "detectorSpecificFlags" : []
+},
+{
+  "name": "CentralBarrelTrackingOnlyLimitedAcceptanceinTPC",
+  "globalRequiredDetectors" : [ "TPC", "ITS" ],
+  "globalIgnoredFlags" : [],
+  "detectorSpecificFlags" : [
+   {
+      "name" : "TPC",
+      "requiredFlags" : [ "Limited acceptance" ]
+   }
+}
+]
+```
+
+## Wishlist / Ideas
+* executable to add flags to the flag store
+* executable to extrags the flag store
+* summary of all masks for one TimeRangeFlagCollection
+* functionality to extract flags for a specific detector (from CCDB)
+* cut class to specify detectors and flags which to exclude

DataFormats/QualityControl/include/DataFormatsQualityControl/FlagReasons.h

@@ -0,0 +1,86 @@
+// Copyright CERN and copyright holders of ALICE O2. This software is
+// distributed under the terms of the GNU General Public License v3 (GPL
+// Version 3), copied verbatim in the file "COPYING".
+//
+// See http://alice-o2.web.cern.ch/license for full licensing information.
+//
+// In applying this license CERN does not waive the privileges and immunities
+// granted to it by virtue of its status as an Intergovernmental Organization
+// or submit itself to any jurisdiction.
+
+#ifndef ALICEO2_ANALYSIS_FLAGREASONS
+#define ALICEO2_ANALYSIS_FLAGREASONS
+
+/// \file FlagReasons.h
+/// \brief classes keeping reasons for flagging time ranges
+/// \author Jens Wiechula, jens.wiechula@ikf.uni-frankfurt.de
+/// \author Piotr Konopka, piotr.jan.konopka@cern.ch
+
+// STL
+#include <iosfwd>
+#include <cstdint>
+
+// ROOT includes
+#include <Rtypes.h>
+
+namespace o2
+{
+namespace quality_control
+{
+
+struct FlagReasonFactory;
+
+class FlagReason
+{
+ private:
+  uint16_t mId;
+  std::string mName;
+  bool mBad; // if true, data should become bad by default
+
+  // By making the constructor private and FlagReasons available only in the FlagReasonFactory
+  // we forbid to declare any flags in the user code. If you need a new FlagReason, please add it FlagReasonFactory.
+ private:
+  FlagReason(uint16_t id, const char* name, bool bad) : mId(id), mName(name), mBad(bad) {}
+
+ public:
+  FlagReason& operator=(const FlagReason&) = default;
+  FlagReason(const FlagReason&) = default;
+  bool operator==(const FlagReason& rhs) const;
+  bool operator!=(const FlagReason& rhs) const;
+  bool operator<(const FlagReason& rhs) const;
+  bool operator>(const FlagReason& rhs) const;
+
+  uint16_t getID() { return mId; }
+  const std::string& getName() { return mName; }
+  bool getBad() { return mBad; }
+
+  friend std::ostream& operator<<(std::ostream& os, FlagReason const& me);
+  friend FlagReasonFactory;
+
+  ClassDefNV(FlagReason, 1);
+};
+
+struct FlagReasonFactory {
+  FlagReasonFactory() = delete;
+
+  // TODO: migrate the flag list from RCT
+  // TODO: find a way to have a nicely formatted list of reasons.
+
+  // !!! NEVER MODIFY OR DELETE EXISTING FLAGS AFTER RUN 3 STARTS !!!
+  static FlagReason Invalid() { return {static_cast<uint16_t>(-1), "Invalid", true}; }
+
+  static FlagReason Unknown() { return {1, "Unknown", true}; }
+  static FlagReason ProcessingError() { return {2, "Processing error", true}; }
+  // it can be used when there are no required Quality Objects in QCDB in certain time range.
+  static FlagReason MissingQualityObject() { return {3, "Missing Quality Object", true}; }
+  // Quality Object is there, but it has Quality::Null
+  static FlagReason MissingQuality() { return {4, "Missing Quality", true}; }
+
+  // TODO: to be seen if we should actively do anything when a detector was off.
+  static FlagReason DetectorOff() { return {10, "Detector off", true}; }
+  static FlagReason LimitedAcceptance() { return {11, "Limited acceptance", true}; }
+};
+
+} // namespace quality_control
+} // namespace o2
+#endif

DataFormats/QualityControl/include/DataFormatsQualityControl/TimeRangeFlag.h

@@ -0,0 +1,86 @@
+// Copyright CERN and copyright holders of ALICE O2. This software is
+// distributed under the terms of the GNU General Public License v3 (GPL
+// Version 3), copied verbatim in the file "COPYING".
+//
+// See http://alice-o2.web.cern.ch/license for full licensing information.
+//
+// In applying this license CERN does not waive the privileges and immunities
+// granted to it by virtue of its status as an Intergovernmental Organization
+// or submit itself to any jurisdiction.
+
+#ifndef ALICEO2_ANALYSIS_TIMERANGEFLAGS
+#define ALICEO2_ANALYSIS_TIMERANGEFLAGS
+
+/// \file TimeRangeFlag.h
+/// \brief Class to define a time range of a flag type
+/// \author Jens Wiechula, jens.wiechula@ikf.uni-frankfurt.de
+/// \author Piotr Konopka, piotr.jan.konopka@cern.ch
+
+// System includes
+#include <iosfwd>
+#include <string>
+
+// ROOT includes
+#include <Rtypes.h>
+
+#include <MathUtils/detail/Bracket.h>
+
+#include "DataFormatsQualityControl/FlagReasons.h"
+
+namespace o2
+{
+namespace quality_control
+{
+
+/// \class TimeRangeFlag
+/// A Class for associating a bit mask with a time range
+class TimeRangeFlag
+{
+ public:
+  using time_type = uint64_t;
+  using flag_type = FlagReason;
+  using RangeInterval = o2::math_utils::detail::Bracket<time_type>;
+
+  TimeRangeFlag() = default;
+  TimeRangeFlag(TimeRangeFlag const&) = default;
+  TimeRangeFlag(time_type start, time_type end, flag_type flag, std::string comment = "", std::string source = "Unknown");
+
+  time_type getStart() const { return mInterval.getMin(); }
+  time_type getEnd() const { return mInterval.getMax(); }
+  RangeInterval& getInterval() { return mInterval; }
+  flag_type getFlag() const { return mFlag; }
+  const std::string& getComment() const { return mComment; }
+  const std::string& getSource() const { return mSource; }
+
+  void setStart(time_type start) { mInterval.setMin(start); }
+  void setEnd(time_type end) { mInterval.setMax(end); }
+  void setInterval(RangeInterval interval) { mInterval = interval; }
+  void setFlag(flag_type flag) { mFlag = flag; }
+  void setComment(const std::string& comment) { mComment = comment; }
+  void setSource(const std::string& source) { mSource = source; }
+
+  /// equal operator
+  bool operator==(const TimeRangeFlag& rhs) const;
+
+  /// comparison operators
+  bool operator<(const TimeRangeFlag& rhs) const;
+  bool operator>(const TimeRangeFlag& rhs) const;
+
+  /// write data to ostream
+  void streamTo(std::ostream& output) const;
+
+  /// overloading output stream operator
+  friend std::ostream& operator<<(std::ostream& output, const TimeRangeFlag& data);
+
+ private:
+  RangeInterval mInterval = {};                   ///< time interval of the masked range
+  flag_type mFlag = FlagReasonFactory::Invalid(); ///< flag reason
+  std::string mComment = "";                      ///< optional comment, which may extend the reason
+  std::string mSource = "Unknown";                ///< optional (but encouraged) source of the flag (e.g. Qc Check name)
+
+  ClassDefNV(TimeRangeFlag, 1);
+};
+
+} // namespace quality_control
+} // namespace o2
+#endif