Command line tools for XML sync testing between languages (#222)

* Command line tool for XML sync testing between languages: tags, revtag, PI, ws
* Modernize array() usage.
* Review

Author: alfsb

scripts/translation/README.md

@@ -16,15 +16,19 @@ Because of the above, it's possible to silence each alert indempendly. These
 scripts will output `--add-ignore` commands that, if executed, will omit the
 specific alerts in future executions.
 
-## First execution
+## broken.php
 
-The first execution of these scripts may generate an inordinate amount of
-alerts. It's advised to initially run each command separately, and work the
-alerts on a case by case basis. After all interesting cases are fixed,
-it's possible to rerun the command and `grep` the output for `--add-ignore`
-lines, run these commands, and by so, mass ignore the residual alerts.
+`doc-base/scripts/broken.php` will test if individual XML files are
+ill-formed. That is, if a file contains Unicode BOM, carriage returns (CR),
+or if XML contents are not
+[well-balanced](https://www.w3.org/TR/xml-fragment/#defn-well-balanced).
+
+Unbalanced XML contents are invalid XML and will result in a broken build.
+BOM and CR marks may not result in broken builds, but *will* cause several
+tools below to misbehave, as `libxml` behaviour changes if XML text contains
+these bytes.
 
-## qaxml-attributes.php (structural)
+## qaxml-attributes.php
 
 `doc-base/scripts/translation/qaxml-attributes.php` checks if all translated
 files have the same tag-attribute-value triplets. Tag's attributes are
@@ -35,7 +39,7 @@ This script accepts an `--urgent` option, to filter alerts related to `xml:id`
 attributes. This will help translators on languages that are failing to build,
 to focus on mismatches that are probably most related with build fails.
 
-## qaxml-entities.php (structural)
+## qaxml-entities.php
 
 `doc-base/scripts/translation/qaxml-entities.php` checks if all translated
 files contain the same XML Entities References as the original files.
@@ -55,15 +59,99 @@ entities when generating alerts. This is handy in languages that use some
 `&zb;` and `&dh;` entities, and could run with `-zb -dh` to avoid generating
 alerts for these entities' differences.
 
-## Old tools (below)
+## qaxml-pi.php
+
+`doc-base/scripts/translation/qaxml-pi.php` checks if all translated files have
+the same processing instructions (PI) as the original files. Unbalanced PIs may
+cause compilation errors, as they are utilized in the manual build process.
+
+## qaxml-tags.php
+
+`doc-base/scripts/translation/qaxml-tags.php` checks if all translated files
+have the same tags as the original files. Different number of tags between
+source texts and translations indicated mismatched translated texts, and may
+cause compilation errors
+
+This script accepts an `--detail` option, that will print lines of each
+mismatched tag, to facilitate the work on big files.
+
+This script also accepts an `--content=` option, that will check the
+*contents* of tags, to inspect tags where the contents are expected *not* to
+be translated. Example below.
+
+## qaxml-ws.php
+
+`doc-base/scripts/translation/qaxml-ws.php` inspect whitespace usage inside
+some known tags. Spurious whitespace may break manual linking or generate
+visible artifacts.
+
+## qaxml-revtag.php
+
+`doc-base/scripts/translation/qaxml-revtag.php` checks if all translated
+files have valid [revision tags](https://doc.php.net/guide/translating.md).
+Files without revision tags in expected format will fail to generate pretty
+diffs on [Translation status](https://doc.php.net/revcheck.php) website or
+locally generated `revcheck.php` status pages.
+
+## Suggested execution
+
+The first execution of these scripts may generate an inordinate amount of
+alerts. It's advised to initially run each command separately, and work the
+alerts on a case by case basis. After all interesting cases are fixed,
+it's possible to rerun the command and `grep` the output for `--add-ignore`
+lines, run these commands, and by so, mass ignore the residual alerts.
+
+Structural checks:
+
+```
+php doc-base/scripts/broken.php
+php doc-base/scripts/translation/qaxml-revtag.php
+
+php doc-base/scripts/translation/qaxml-attributes.php
+php doc-base/scripts/translation/qaxml-entities.php
+php doc-base/scripts/translation/qaxml-pi.php
+php doc-base/scripts/translation/qaxml-tags.php --detail
+php doc-base/scripts/translation/qaxml-ws.php
+```
 
-The tools on `doc-base/scripts/translation/` are slowly being rewritten. While
-this effort is not complete, the previous tools, document below, could be used
-to supply for features yet not completed.
+Tags where is expected no translations:
+
+```
+php doc-base/scripts/translation/qaxml-tags.php --content=acronym
+php doc-base/scripts/translation/qaxml-tags.php --content=classname
+php doc-base/scripts/translation/qaxml-tags.php --content=constant
+php doc-base/scripts/translation/qaxml-tags.php --content=envar
+php doc-base/scripts/translation/qaxml-tags.php --content=function
+php doc-base/scripts/translation/qaxml-tags.php --content=interfacename
+php doc-base/scripts/translation/qaxml-tags.php --content=parameter
+php doc-base/scripts/translation/qaxml-tags.php --content=type
+php doc-base/scripts/translation/qaxml-tags.php --content=classsynopsis
+php doc-base/scripts/translation/qaxml-tags.php --content=constructorsynopsis
+php doc-base/scripts/translation/qaxml-tags.php --content=destructorsynopsis
+php doc-base/scripts/translation/qaxml-tags.php --content=fieldsynopsis
+php doc-base/scripts/translation/qaxml-tags.php --content=funcsynopsis
+php doc-base/scripts/translation/qaxml-tags.php --content=methodsynopsis
+```
+
+Tags where is expected few translations:
+
+```
+php doc-base/scripts/translation/qaxml-tags.php --content=code
+php doc-base/scripts/translation/qaxml-tags.php --content=computeroutput
+php doc-base/scripts/translation/qaxml-tags.php --content=filename
+php doc-base/scripts/translation/qaxml-tags.php --content=literal
+php doc-base/scripts/translation/qaxml-tags.php --content=varname
+```
 
 ---
 
-Before using the old scripts, they need be configured:
+## Old tools (below)
+
+Document below is the previous version of these tools. These tools are
+deprecated, and scheduled for remotion very soon.
+
+
+These old tools needed to be separated configured, before use:
 ```
 php doc-base/scripts/translation/configure.php $LANG_DIR
 ```
@@ -107,44 +195,3 @@ contents, as some tag contents are expected *not* be translated.
 
 `--detail` will also print line definitions of each mismatched tag,
 to facilitate bitsecting.
-
-## Suggested execution
-
-Structural checks:
-
-```
-php doc-base/scripts/translation/configure.php $LANG_DIR
-
-php doc-base/scripts/translation/qarvt.php
-
-php doc-base/scripts/translation/qaxml.a.php
-php doc-base/scripts/translation/qaxml.e.php
-php doc-base/scripts/translation/qaxml.p.php
-php doc-base/scripts/translation/qaxml.t.php
-php doc-base/scripts/translation/qaxml.w.php
-```
-Tags where is expected no translations:
-```
-php doc-base/scripts/translation/qaxml.t.php acronym
-php doc-base/scripts/translation/qaxml.t.php classname
-php doc-base/scripts/translation/qaxml.t.php constant
-php doc-base/scripts/translation/qaxml.t.php envar
-php doc-base/scripts/translation/qaxml.t.php function
-php doc-base/scripts/translation/qaxml.t.php interfacename
-php doc-base/scripts/translation/qaxml.t.php parameter
-php doc-base/scripts/translation/qaxml.t.php type
-php doc-base/scripts/translation/qaxml.t.php classsynopsis
-php doc-base/scripts/translation/qaxml.t.php constructorsynopsis
-php doc-base/scripts/translation/qaxml.t.php destructorsynopsis
-php doc-base/scripts/translation/qaxml.t.php fieldsynopsis
-php doc-base/scripts/translation/qaxml.t.php funcsynopsis
-php doc-base/scripts/translation/qaxml.t.php methodsynopsis
-```
-Tags where is expected few translations:
-```
-php doc-base/scripts/translation/qaxml.t.php code
-php doc-base/scripts/translation/qaxml.t.php computeroutput
-php doc-base/scripts/translation/qaxml.t.php filename
-php doc-base/scripts/translation/qaxml.t.php literal
-php doc-base/scripts/translation/qaxml.t.php varname
-```

scripts/translation/lib/OutputIgnoreArgv.php

@@ -34,7 +34,7 @@ function __construct( array & $argv )
             if ( str_starts_with( $arg , "--add-ignore=" ) )
             {
                 $file = OutputIgnoreArgv::cacheFile();
-                $list = $file->load( array() );
+                $list = $file->load( [] );
                 $line = substr( $arg , 13 );
                 if ( ! in_array( $line , $list ) )
                 {
@@ -47,7 +47,7 @@ function __construct( array & $argv )
             if ( str_starts_with( $arg , "--del-ignore=" ) )
             {
                 $file = OutputIgnoreArgv::cacheFile();
-                $list = $file->load( array() );
+                $list = $file->load( [] );
                 $line = substr( $arg , 13 );
                 $dels = 0;
                 while ( in_array( $line , $list ) )

scripts/translation/lib/OutputIgnoreBuffer.php

@@ -25,8 +25,8 @@ class OutputIgnoreBuffer
 
     private string $filename = "";
     private string $header = "";
-    private array  $matter = array();
-    private array  $footer = array();
+    private array  $matter = [];
+    private array  $footer = [];
 
     private OutputIgnoreArgv $args;
 
@@ -82,14 +82,14 @@ function print( bool $useAlternatePrinting = false )
 
         $markhead = $this->filename . ':' . $this->hash( false ) . ':';
         $markfull = $markhead . $this->hash( true );
-        $marks = OutputIgnoreArgv::cacheFile()->load( array() );
+        $marks = OutputIgnoreArgv::cacheFile()->load( [] );
 
         if ( $this->args->showIgnore )
         {
             // --add-ignore
 
             if ( in_array( $markfull , $marks ) )
-                $this->matter = array();
+                $this->matter = [];
             else
                 $this->args->pushAddIgnore( $this , $markfull );
 
@@ -132,9 +132,9 @@ function print( bool $useAlternatePrinting = false )
 
     private function printMatterAlternate() : void
     {
-        $add = array();
-        $del = array();
-        $rst = array();
+        $add = [];
+        $del = [];
+        $rst = [];
 
         foreach( $this->matter as $text )
         {

scripts/translation/lib/RevcheckData.php

@@ -32,9 +32,9 @@ class RevcheckData
     public string $lang = "";
     public string $date = "";
     public string $intro = "";
-    public $translators  = array(); // nick => RevcheckDataTranslator
-    public $fileSummary  = array(); // RevcheckStatus => int
-    public $fileDetail   = array(); // filename => RevcheckDataFile
+    public $translators  = []; // nick => RevcheckDataTranslator
+    public $fileSummary  = []; // RevcheckStatus => int
+    public $fileDetail   = []; // filename => RevcheckDataFile
 
     public function __construct()
     {

scripts/translation/lib/RevcheckFileList.php

@@ -21,7 +21,7 @@
 
 class RevcheckFileList
 {
-    private $list = array();
+    private $list = [];
 
     function __construct( $lang )
     {

scripts/translation/lib/RevtagParser.php

@@ -70,7 +70,7 @@ public static function parseComment( DOMNode $node , RevtagInfo $ret , $filename
             // /EN-Revision:\s*(\S+)\s*Maintainer:\s*(\S+)\s*Status:\s*(\S+)/       // restrict maintainer without spaces
             // /EN-Revision:\s*(\S+)\s*Maintainer:\s(.*?)\sStatus:\s*(\S+)/         // accepts maintainer with spaces
 
-            $match = array();
+            $match = [];
             $regex = "/EN-Revision:\s*(\S+)\s*Maintainer:\s(.*?)\sStatus:\s*(\S+)/";
             if ( preg_match( $regex , $text , $match ) )
             {
@@ -91,7 +91,7 @@ public static function parseComment( DOMNode $node , RevtagInfo $ret , $filename
 
         if ( str_starts_with( $text , "CREDITS:" ) )
         {
-            $match = array();
+            $match = [];
             $regex = "/CREDITS:(.*)/";
             if ( preg_match( $regex , $text , $match ) )
             {

scripts/translation/lib/XmlUtil.php

@@ -34,7 +34,7 @@ public static function extractEntities( $filename )
         libxml_clear_errors();
         libxml_use_internal_errors( $was );
 
-        $ret = array();
+        $ret = [];
         foreach ($errors as $error)
         {
             if ( preg_match( "/Entity '(\S+)' not defined/" , $error->message , $matches ) )
@@ -45,7 +45,7 @@ public static function extractEntities( $filename )
 
     public static function listNodeType( DOMNode $node , int $type )
     {
-        $ret = array();
+        $ret = [];
         XmlUtil::listNodeTypeRecurse( $node , $type , $ret );
         return $ret;
     }

scripts/translation/libqa/ArgvParser.php

@@ -26,7 +26,6 @@ class ArgvParser
     public function __construct( array $argv )
     {
         $this->argv = array_values( array_filter( $argv ) );
-        $this->used = [];
         $this->used = array_fill( 0 , count( $argv ) , false );
     }
 
@@ -58,6 +57,9 @@ public function consume( string $equals = null , string $prefix = null , int $po
                 $this->argv[ $pos ] = null;
                 $this->used[ $pos ] = true;
 
+                if ( $foundByPrefix )
+                    return substr( $arg , strlen( $prefix ) );
+
                 return $arg;
             }
         }

scripts/translation/libqa/OutputBuffer.php

@@ -28,6 +28,8 @@ class OutputBuffer
     private OutputIgnore $ignore;
     private string $options;
 
+    public int $printCount = 0;
+
     public function __construct( string $header , string $filename , OutputIgnore $ignore )
     {
         $filename = str_replace( "/./" , "/" , $filename );
@@ -81,7 +83,7 @@ public function contains( string $text ) : bool
         return false;
     }
 
-    public function print( bool $useAlternatePrinting = false )
+    public function print( bool $alternatePrinting = false )
     {
         if ( count( $this->matter ) == 0 && count( $this->footer ) == 0 )
             return;
@@ -93,9 +95,11 @@ public function print( bool $useAlternatePrinting = false )
         if ( $this->ignore->shouldIgnore( $this , $hashFile , $hashHead , $hashFull ) )
             return;
 
+        $this->printCount++;
+
         print $this->header;
 
-        if ( $useAlternatePrinting )
+        if ( $alternatePrinting )
             $this->printMatterAlternate();
         else
             foreach( $this->matter as $text )
@@ -115,9 +119,9 @@ public function print( bool $useAlternatePrinting = false )
 
     private function printMatterAlternate() : void
     {
-        $add = array();
-        $del = array();
-        $rst = array();
+        $add = [];
+        $del = [];
+        $rst = [];
 
         foreach( $this->matter as $text )
         {
@@ -128,8 +132,8 @@ private function printMatterAlternate() : void
 
         for ( $idx = 0 ; $idx < count( $this->matter ) ; $idx++ )
         {
-            if ( isset( $add[ $idx ] ) ) print $add[ $idx ];
             if ( isset( $del[ $idx ] ) ) print $del[ $idx ];
+            if ( isset( $add[ $idx ] ) ) print $add[ $idx ];
         }
 
         foreach( $rst as $text )