Strip invalid chars without raising encoding exception regardless of encoding

On upgrade to faraday 2, HTTP responses can come back in specific encodings per the content-type of response. This effected one of our tests, that is using a live query to google.com (probably not a great idea), that comes back ISO-8859-1. But it actually would have failed if UTF-8 too, because we're trying to gsub some intentionally illegal bytes.   I am not sure if the approach to encoding is totally sound in this gem in general -- XML does have to be Unicode legally, but can be UTF-8 or -16, this strip routine may be assuming UTF8? But not actually properly making it with the ruby UTF8 encoding?

But this is just an attempt to change as little as possible leaving as backward compat as possible while passing tests in faraday 2; and resolving the specific problem of the strip_invalid_utf_8_chars failing when response comes back with ruby encoding tag from faraday 2.

Author: jrochkind

lib/oai/client.rb

@@ -331,14 +331,27 @@ def parse_date(value)
     # Regex is from WebCollab:
     #   http://webcollab.sourceforge.net/unicode.html
     def strip_invalid_utf_8_chars(xml)
-      xml && xml.gsub(/[\x00-\x08\x10\x0B\x0C\x0E-\x19\x7F]
+      return xml unless xml
+
+      # If it's in a specific encoding other than BINARY, it may trigger
+      # an exception to try to gsub these illegal bytes. Temporarily
+      # put it in BINARY. NOTE: We're not totally sure what's going on
+      # with encodings in this gem in general, it might not be totally reasonable.
+      orig_encoding = xml.encoding
+      xml.force_encoding("BINARY")
+
+      xml = xml.gsub(/[\x00-\x08\x10\x0B\x0C\x0E-\x19\x7F]
                              | [\x00-\x7F][\x80-\xBF]+
                              | ([\xC0\xC1]|[\xF0-\xFF])[\x80-\xBF]*
                              | [\xC2-\xDF]((?![\x80-\xBF])|[\x80-\xBF]{2,})
                              | [\xE0-\xEF](([\x80-\xBF](?![\x80-\xBF]))
                              | (?![\x80-\xBF]{2})|[\x80-\xBF]{3,})/x, '?')\
                 .gsub(/\xE0[\x80-\x9F][\x80-\xBF]
                        | \xED[\xA0-\xBF][\x80-\xBF]/,'?')
+
+      xml.force_encoding(orig_encoding)
+
+      xml
     end
 
   end