Validation_goldens modfication

[niveditasing]

Summary
This PR fixes issues in our validation pipeline, including data filtering mismatches, malformed CSV output formatting, file path resolution errors, and CSV/MCF parsing crashes.

1. runner.py — Dynamic Path Resolution

• Config Directory Injection:

  • Saved the validation_config_path during runner initialization.
  • Extracted the absolute directory of the configuration file (config_dir) and injected it into rule_params before executing rule validation.

  Path in validation_config.json -
  "params": { "golden_files": "../../../../../golden_data/golden_summary_report.csv" }

2. validator_goldens.py — Safe CSV Serialization and Namespace Stripping

• Path Resolution Support:
  • Added a helper utility _resolve_paths(path, config_dir) to resolve relative file paths to absolute paths relative to the config
    directory.
  • Integrated path resolution inside validate_goldens to resolve both inputs and golden_files paths.
• Namespace Stripping for CSV Comparisons:
  • In load_nodes_from_file, stripped namespace prefixes (such as dcs:, dcid:, schema:) from cell values when loading nodes from CSV
    files, ensuring reliable string and ID comparisons.

3. file_util.py

• Problem:
  when writing a dictionary of dictionaries (such as validator golden nodes) with key_column_name=None, file_write_csv_dict would scan the inner dictionaries and identify the single property (e.g., ['GeoId']). Because len(columns) == 1, it blindly assumed the values were primitives and appended a default 'value' column. This resulted in an extra empty column in the generated CSV (e.g., "GeoId","value")

• Fix:
  Added a check using any(isinstance(value, dict) for value in py_dict.values()) to determine if the dictionary values are nested dictionaries: If the values are dictionaries, the extracted single key is preserved as-is, and no extra 'value' column is added. If the values are primitives/simple types, the function still falls back to appending 'value' to keep existing functionality working perfectly.

  Before:
  "GeoId","value"
  "","{'GeoId': 'country/ALB'}"
  "","{'GeoId': 'country/AUT'}"
  "","{'GeoId': 'country/BEL'}"
  "","{'GeoId': 'country/BGR'}"
  "","{'GeoId': 'country/CHE'}"
  After (Correct):
  "GeoId"
  "Earth"
  "country/AGO"

---

Verification and Testing Performed:

tested imports to determine if the code is working. Below are the validation_output results, one showing showing success.
2. EurostatData_Fertility: https://storage.mtls.cloud.google.com/datcom-import-test/scripts/eurostat/regional_statistics_by_nuts/fertility_rate_mother_age/EurostatData_Fertility/2026_06_23T23_40_38_697535_07_00/input0/validation/validation_output.csv

[gemini-code-assist]

Code Review

This pull request updates validator_goldens.py to support namespace-stripped matching when generating goldens and replaces the helper function for writing CSVs with custom csv.DictWriter logic. A review comment points out that extracting CSV column headers from only the first node in golden_nodes assumes all nodes share the same keys, which can lead to errors or missing columns if keys vary. It suggests collecting the union of all keys across all nodes instead.

[balit-raibot]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request updates tools/import_validation/validator_goldens.py to import the csv module, strip namespaces when matching nodes against must_include_values, and replace the helper file_write_csv_dict with a custom csv.DictWriter implementation. The feedback suggests adding newline='' when opening the CSV file to prevent platform-specific carriage return issues and simplifying the logic for extracting and sorting unique keys.

[niveditasing]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces path resolution logic for summary report rules in the validation runner, cleans up CSV node keys during loading, and updates golden file generation to use Python's standard csv.DictWriter. Feedback focuses on adhering to PEP 8 line length limits, replacing debug print statements with proper logging.debug calls, and opening CSV files with newline='' to ensure cross-platform compatibility on Windows.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[niveditasing]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces relative path resolution for the GOLDENS_CHECK validator and improves CSV parsing robustness when handling values containing colons (such as DCIDs). Feedback on these changes suggests using repository marker files (like .git or WORKSPACE) for more robust root path resolution, replacing debug print statements with proper logging.debug calls, and streaming CSV files directly to csv.DictReader to avoid reading entire files into memory.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[niveditasing]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces path resolution improvements for the GOLDENS_CHECK validator, cleans up "dcid:" prefixes when loading golden nodes, and refactors CSV writing logic to handle dictionary values correctly. The code reviewer provided several valuable suggestions, including fixing a bug where remote paths (like GCS URIs) are incorrectly treated as relative local paths, correcting a documentation mismatch in Validations.md, removing a redundant logging import, and handling file paths robustly in the directory search helper. Additionally, the reviewer recommended replacing debug print statements with proper logging, using the modern removeprefix method for string cleanup, and optimizing a dictionary check in file_util.py to run in O(1) time.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[pulkeet-wq]

Thanks for editing the code as discussed @niveditasing. LGTM