Merge pull request #1 from APIOpsCycles/codex/add-json-schemas-and-package.json-for-module

Add JSON schemas and validation for method data

Author: mniinio

.gitignore

@@ -0,0 +1,5 @@
+node_modules
+
+# translation cache and temporary translation folder
+.json-autotranslate-cache
+src/data/locales/

README.md

@@ -4,38 +4,48 @@ Read more about the method at (https://www.apiopscycles.com/)
 
 This repository contains the source for the APIOps Cycles method, including resources such as canvases. It can be used for multiple purposes, such as the method website (https://www.apiopscycles.com/), which is generated from these files, and the Canvas Creator tool that allows creating, importing and exporting the canvases, and provides a UI. (https://canvascreator.apiopscycles.com/). 
 
-## Requirements
+## License
 
-You can use the JSON files as is and download a .zip file or clone the repository. You can also install them using `npm install apiops-cycles-method-data`. 
+All content in this repository is provided under Apache 2 license for programmable use, the content and the design of the canvases is under the CC BY-SA 4.0 license unless noted otherwise.
+APIOps and APIOps Cycles are trademarks owned by Osaango Oy (https://www.osaango.com). Permission is given to use them in connection with the method, for community purposes. Contact Osaango for partnerships, or sponsoring to support the method development. See method website for current partners (https://www.apiopscycles.com/)
 
-- **Node.js 22** or newer
-- npm
 
-Install dependencies once with:
+## Repository structure
 
-```bash
-npm install
+```
+├── src/
+│   ├── assets/             # APIOps Cycles logos and other core assets which you might need in your tooling or product
+│   ├── data/method/        # The Method JSON files (structure, relationship, and guideline content)
+│   ├── data/method/canvas/ # The Canvases included in the method as JSON files (also used by tools like Canvas Creator)
+│   ├── snippets/           # Raw markdown files used for long content for resource docs (only essential extensions for the json files)
+├── scripts/                # Utility scripts
+└── package.json
 ```
 
-### Canvas editor
+## Requirements
+
+You can use the JSON files as is and download a .zip file or clone the repository. You can also install them using `npm install apiops-cycles-method-data`.
 
-Interactive canvases on the site are powered by the [Canvas Creator](https://canvascreator.apiopscycles.com) tool. The component is installed from npm so make sure to run `npm install` before building or running the dev server.
+The module exposes top-level exports so you can import the data files directly, for example:
 
-## Local development
+```js
+import stations from "apiops-cycles-method-data/method/stations.json";
+import canvasData from "apiops-cycles-method-data/canvasData";
+```
 
-Run a local dev server:
+Validate the files locally with:
 
 ```bash
-npm run dev
-```
+npm test
+``` 
 
-This command copies the resource files to `public/` so that links to canvas JSON
-files work during development.
+- **Node.js 22** or newer
+- npm
 
-Build the production site:
+Install dependencies once with:
 
 ```bash
-npm run build
+npm install
 ```
 
 ## Contributing
@@ -46,17 +56,13 @@ If you spot a problem in the documentation or have an idea for new content, plea
 
 ### Editing or adding content
 
-Most pages are generated from the JSON files in `src/data/method/`. These base files (`lines.json`, `stations.json`, `resources.json`, `criteria.json` and `station-criteria.json`) are not localized and live at the root of the folder. Textual values in them reference label keys. English labels are in `src/data/method/en-US` and translations are provided in `labels.lines.json`, `labels.stations.json`, `labels.resources.json` and `labels.criteria.json` under each locale folder. Some longer or more complex resource pages like the API Audit Checklist also use markdown snippets `src/snippets/` linked to the `resources.json`. Do not use any frontmatter in the snippet files. Any supported markdown markup is ok. See references from [Starlight markdown reference](https://starlight.astro.build/guides/authoring-content/) and [Extended markdown reference](https://www.markdownguide.org/extended-syntax/).
+The main method files (instructions, guidelines, method structure) is located in the the JSON files at `src/data/method/`. These base files (`lines.json`, `stations.json`, `resources.json`, `criteria.json` and `station-criteria.json`) are not localized and live at the root of the folder. Textual values in them reference label keys. English labels are in `src/data/method/en-US` and translations are provided in `labels.lines.json`, `labels.stations.json`, `labels.resources.json` and `labels.criteria.json` under each locale folder. Some longer or more complex resource pages like the API Audit Checklist also use markdown snippets `src/snippets/` linked to the `resources.json`. Do not use any frontmatter in the snippet files. Any supported markdown markup is ok. See references from [Starlight markdown reference](https://starlight.astro.build/guides/authoring-content/) and [Extended markdown reference](https://www.markdownguide.org/extended-syntax/).
 
-Each station page lists the station's entry criteria followed by the next core station's criteria as exit criteria.
-
-After editing these files, run `npm run generate:method` to update the Markdown files in `src/content/docs/`. Note that the generated markdown files are not inlcuded in version control to avoid confusion on where editing should happen.
-
-For fixes in the Markdown files included in version control under `src/content/docs/`, edit them directly.
+Each station links to specific entry criteria followed by the next core station's criteria as exit criteria.`criteria.json`, `station-criteria.json` and `labels.criteria.json` 
 
 #### Editing existing content of Method pages (metrolines, core- and substations, resources). 
 1. Go to `src/data/method/en` and edit the content in English (English is considered the master langauge, and for the translations to work for other languages, it must always be the first to be edited). 
-2. Validate that your changes work TODO
+2. Validate that your changes work by running the schema validations (`npm test`)
 3. Follow the translation guide if you are able to translate the content to other languages manually or automatically.
 4. Commit your code and make a pull request.
 5. If you were not able to create the translations of your changes to all languages, create an issue for in the repository for the translations.
@@ -71,6 +77,8 @@ Content under `src/snippets/` and `src/assets/resource/` can be localized by pla
 
 Create/Edit the translaton files and put them in to the `src/data/method/` with correct language subfolder manually, or...
 
+Install json-autotranslate with `npm install json-autotranslate`. It is not included as a dev-dependency at all, because most people would never need it and it's installing over 200+ dependencies.
+
 1. Create a copy of the `src/data/method/` folder as `.json-autotranslate-cache` (in the root of the project). The cache is not under version control.
 2. Create a new folder under the `src/data/locale` (this is a temporary folder, not versioned ) using the 2-letter ISO-code for the new language you are about to translate, e.g. `src/data/locale/fi` if creating translations for Finnish language.
 3. Make sure the target language folder is *empty*, e.g. `src/data/locale/fi`. The folder really needs to be empty, even if there are already translations done for this language.
@@ -93,26 +101,8 @@ Create/Edit the translaton files and put them in to the `src/data/method/` with
 9. If the translated files look ok, go to the translation cache folder `.json-autotranslate-cache` and look for the folder in the target language, i.e. `.json-autotranslate-cache/fi`. 
  - (Explanation: For some strange reason the plugin writes the contents in different (nested) format in the source folder (i.e. `locales`) than what it can actually read during the next translation round. The flat format it uses in the cache is perfect for it and also the most efficient for our page generation scripts).
 10. These files are in the correct flat format for our page generation scripts, so copy these files and paste them to `src/data/method/` folder for the target language, in this case `src/data/method/fi`
-11. Validate that your changes work TODO
+11. Validate that your changes work with `npm test`
 12. Commit your code and make a pull request. 
 13. After your pull request has been merged, remember to pull changes so that your local branch is deleted, too. (Make sure you don't leave locally any old files under `.json-autotranslate-cache`.)
 
-If you are planning to use automated translation services: Note that there are some languages that are supported by some of the translation services with local varieties. Check the documentation (https://github.com/leolabs/json-autotranslate) of supported automated translation services and how to see which languages they support.
-  
-
-## Repository structure
-
-```
-├── src/
-│   ├── assets/             # APIOps Cycles logos and other core assets which you might need in your tooling or product
-│   ├── data/method/        # The Method JSON files (structure, relationship, and guideline content)
-│   ├── data/method/canvas/ # The Canvases included in the method as JSON files (also used by tools like Canvas Creator)
-│   ├── snippets/           # Raw markdown files used for long content for resource docs (only essential extensions for the json files)
-├── scripts/                # Utility scripts
-└── package.json
-```
-
-## License
-
-All content in this repository is provided under Apache 2 license for programmable use, the content and the design of the canvases is under the CC BY-SA 4.0 license unless noted otherwise.
-APIOps and APIOps Cycles are trademarks owned by Osaango Oy (https://www.osaango.com). Permission is given to use them in connection with the method, for community purposes. Contact Osaango for partnerships, or sponsoring to support the method development. See method website for current partners (https://www.apiopscycles.com/)
+If you are planning to use automated translation services: Note that there are some languages that are supported by some of the translation services with local varieties. Check the documentation (https://github.com/leolabs/json-autotranslate) of supported automated translation services and how to see which languages they support. 

package-lock.json

@@ -0,0 +1,23 @@
+{
+  "name": "apiops-cycles-method-data",
+  "version": "3.0.0",
+  "description": "APIOps Cycles Method data and canvases",
+  "license": "Apache-2.0",
+  "type": "module",
+  "files": ["src/data","src/snippets", "src/assets/", "schemas/"],
+  "exports": {
+    "./method/": "./src/data/method/",
+    "./canvasData": "./src/data/canvas/canvasData.json",
+    "./localizedData": "./src/data/canvas/localizedData.json",
+    "./canvas/": "./src/data/canvas/import-export-templates/"
+  },
+  "scripts": {
+    "test": "node scripts/validate.mjs"
+  },
+  "devDependencies": {
+    "ajv": "^8.17.1"
+  },
+  "engines": {
+    "node": ">=22"
+  }
+}

package.json

@@ -0,0 +1,60 @@
+import Ajv from 'ajv';
+import fs from 'fs/promises';
+import path from 'path';
+
+const ajv = new Ajv({ allErrors: true });
+
+async function loadJSON(file) {
+  return JSON.parse(await fs.readFile(file, 'utf8'));
+}
+
+async function validate(schemaPath, files) {
+  const schema = await loadJSON(schemaPath);
+  const validate = ajv.compile(schema);
+  for (const file of files) {
+    const data = await loadJSON(file);
+    const valid = validate(data);
+    if (!valid) {
+      console.error(`Validation failed for ${file}`);
+      console.error(validate.errors);
+      process.exitCode = 1;
+    } else {
+      console.log(`Validated ${file}`);
+    }
+  }
+}
+
+async function gatherLabelFiles() {
+  const base = 'src/data/method';
+  const dirs = await fs.readdir(base, { withFileTypes: true });
+  const files = [];
+  for (const dir of dirs) {
+    if (dir.isDirectory()) {
+      const subfiles = await fs.readdir(path.join(base, dir.name));
+      for (const f of subfiles) {
+        if (f.startsWith('labels') && f.endsWith('.json')) {
+          files.push(path.join(base, dir.name, f));
+        }
+      }
+    }
+  }
+  return files;
+}
+
+async function gatherTemplateFiles() {
+  const dir = 'src/data/canvas/import-export-templates';
+  const files = await fs.readdir(dir);
+  return files.filter(f => f.endsWith('.json')).map(f => path.join(dir, f));
+}
+
+(async () => {
+  await validate('src/schemas/criteria.schema.json', ['src/data/method/criteria.json']);
+  await validate('src/schemas/lines.schema.json', ['src/data/method/lines.json']);
+  await validate('src/schemas/resources.schema.json', ['src/data/method/resources.json']);
+  await validate('src/schemas/station-criteria.schema.json', ['src/data/method/station-criteria.json']);
+  await validate('src/schemas/stations.schema.json', ['src/data/method/stations.json']);
+  await validate('src/schemas/canvasData.schema.json', ['src/data/canvas/canvasData.json']);
+  await validate('src/schemas/canvas-localized.schema.json', ['src/data/canvas/localizedData.json']);
+  await validate('src/schemas/import-export-template.schema.json', await gatherTemplateFiles());
+  await validate('src/schemas/labels.schema.json', await gatherLabelFiles());
+})();