add missing docs

Author: gildesmarais

ruby-gem/how-to/advanced-features.md

@@ -0,0 +1,122 @@
+---
+layout: default
+title: Advanced Features
+nav_order: 9
+parent: How-To Guides
+grand_parent: Ruby Gem
+---
+
+# Advanced Features
+
+This guide covers advanced features and performance optimizations for html2rss.
+
+## Parallel Processing
+
+html2rss uses parallel processing to improve performance when scraping multiple items. This happens automatically and doesn't require any configuration.
+
+### How It Works
+
+- **Auto-source scraping:** Multiple scrapers run in parallel to analyze the page
+- **Item processing:** Each scraped item is processed in parallel
+- **Performance benefit:** Significantly faster when dealing with many items
+
+### Performance Tips
+
+1. **Use appropriate selectors:** More specific selectors reduce processing time
+2. **Limit items when possible:** Use CSS selectors that target only the content you need
+3. **Cache responses:** The web application caches responses automatically
+4. **Choose the right strategy:** Use `faraday` for static content, `browserless` only when JavaScript is required
+
+## Memory Optimization
+
+html2rss is designed to be memory-efficient:
+
+- **Frozen objects:** Parsed content is frozen to prevent accidental modifications
+- **Efficient data structures:** Uses `Set` instead of `Array` for lookups
+- **Minimal allocations:** Prefers bang methods to avoid unnecessary memory allocations
+
+## Large Feed Handling
+
+For websites with many items:
+
+```yaml
+# Use specific selectors to limit items
+selectors:
+  items:
+    selector: ".article:not(.advertisement)" # Exclude ads
+  title:
+    selector: "h2" # More specific than generic selectors
+```
+
+## Error Recovery
+
+html2rss includes built-in error handling:
+
+- **Graceful degradation:** If one scraper fails, others continue
+- **Detailed logging:** Set `LOG_LEVEL=debug` for detailed information
+- **Validation:** Configuration is validated before processing
+
+## Custom Headers for Performance
+
+Optimize requests with appropriate headers:
+
+```yaml
+headers:
+  Accept: "text/html,application/xhtml+xml" # Avoid JSON if not needed
+  Accept-Encoding: "gzip, deflate" # Enable compression
+  User-Agent: "html2rss/1.0" # Identify your requests
+```
+
+## Monitoring and Debugging
+
+### Enable Debug Logging
+
+```bash
+LOG_LEVEL=debug html2rss feed config.yml
+```
+
+### Web Application Health Checks
+
+Use the health check endpoint to monitor feed generation:
+
+```bash
+curl -u username:password http://localhost:3000/health_check.txt
+```
+
+## Article Validation
+
+html2rss includes built-in validation for articles to ensure feed quality:
+
+### Validation Rules
+
+Articles are considered valid if they have:
+- A non-empty URL
+- Either a title OR description (or both)
+- A unique ID
+
+### Invalid Articles
+
+Invalid articles are automatically filtered out to prevent empty or broken feed items.
+
+### Custom Validation
+
+You can add custom validation by using post-processors:
+
+```yaml
+selectors:
+  title:
+    selector: "h2"
+    post_process:
+      - name: "gsub"
+        pattern: "^\\s*$"
+        replacement: "Untitled"
+```
+
+## Best Practices
+
+1. **Test configurations:** Always test your configurations before deploying
+2. **Monitor performance:** Use health checks to detect issues early
+3. **Keep selectors simple:** Complex selectors are harder to maintain
+4. **Use auto-source when possible:** It's often more reliable than manual selectors
+5. **Handle errors gracefully:** Implement proper error handling in your applications
+6. **Validate your data:** Ensure your selectors return valid content

ruby-gem/how-to/backward-compatibility.md

@@ -0,0 +1,54 @@
+---
+layout: default
+title: Backward Compatibility
+nav_order: 11
+parent: How-To Guides
+grand_parent: Ruby Gem
+---
+
+# Backward Compatibility
+
+html2rss maintains backward compatibility with older configuration formats and attribute names.
+
+## Renamed Attributes
+
+Some attribute names have been renamed for clarity, but the old names still work:
+
+| Current Name    | Legacy Names        | Description                    |
+| --------------- | ------------------- | ------------------------------ |
+| `published_at`  | `updated`, `pubDate` | Publication date of the item  |
+
+### Example
+
+Both of these configurations work identically:
+
+```yaml
+# Current format (recommended)
+selectors:
+  published_at:
+    selector: ".date"
+
+# Legacy format (still supported)
+selectors:
+  updated:
+    selector: ".date"
+```
+
+## Migration Guide
+
+If you're upgrading from an older version of html2rss:
+
+1. **Update attribute names**: Replace `updated` with `published_at` in your configurations
+2. **Test your feeds**: Verify that all feeds still work correctly after the update
+
+## Deprecated Features
+
+The following features are deprecated but still supported:
+
+- **Legacy attribute names**: While still supported, use the current names for new configurations
+
+## Getting Help
+
+If you encounter issues with backward compatibility:
+
+- **Report issues**: Open an issue if you find compatibility problems

ruby-gem/reference/auto-source.md

@@ -18,13 +18,22 @@ channel:
 auto_source: {}
 ```
 
+## Default Configuration
+
+When you use `auto_source: {}`, html2rss uses these default settings:
+
+- **All scrapers enabled:** `schema`, `semantic_html`, `html`, and `rss_feed_detector`
+- **HTML scraper settings:** `minimum_selector_frequency: 2`, `use_top_selectors: 5`
+- **Cleanup settings:** `keep_different_domain: true`, `min_words_title: 3`
+
 ## How It Works
 
 `auto_source` uses the following strategies to find content:
 
 1.  **`schema`:** Parses `<script type="json/ld">` tags containing structured data (e.g., [Schema.org](https://schema.org/)).
 2.  **`semantic_html`:** Searches for semantic HTML5 tags like `<article>`, `<main>`, and `<section>`.
 3.  **`html`:** Analyzes the HTML structure to find frequently occurring selectors that are likely to contain the main content.
+4.  **`rss_feed_detector`:** Automatically detects and uses existing RSS feeds on the target website. This is particularly useful when a site already has an RSS feed that can be consumed directly.
 
 ## Fine-Tuning
 
@@ -45,6 +54,8 @@ auto_source:
       enabled: true
       minimum_selector_frequency: 3 # default: 2
       use_top_selectors: 3 # default: 5
+    rss_feed_detector:
+      enabled: false # default: true
 ```
 
 ### Cleanup Options

ruby-gem/reference/selectors.md

@@ -37,6 +37,21 @@ selectors:
     enhance: true # default: true
 ```
 
+## Item Ordering
+
+You can control the order of items in your feed:
+
+```yml
+selectors:
+  items:
+    selector: ".article"
+    order: "reverse" # Reverse the order of items (newest first)
+```
+
+Available options:
+- `"reverse"`: Reverses the order of items (useful when the website shows oldest items first)
+- Default: Items appear in the order they are found on the page
+
 ## RSS 2.0 Selectors
 
 While you can define any named selector, only the following are used in the final RSS feed:

support/troubleshooting.md

@@ -23,6 +23,17 @@ If your feed is empty, check the following:
 - **URL:** Ensure the `url` in your configuration is correct and accessible.
 - **`items.selector`:** Verify that the `items.selector` matches the elements on the page.
 - **Website Changes:** Websites change their HTML structure frequently. Your selectors may be outdated.
+- **JavaScript Content:** If the content is loaded via JavaScript, use the `browserless` strategy instead of `faraday`.
+- **Authentication:** Some sites require authentication - check if you need to add headers or use a different strategy.
+
+### Configuration Errors
+
+Common configuration-related errors:
+
+- **`UnsupportedResponseContentType`:** The website returned content that html2rss can't parse (not HTML or JSON).
+- **`UnsupportedStrategy`:** The specified strategy is not available. Use `faraday` or `browserless`.
+- **`Configuration must include at least 'selectors' or 'auto_source'`:** You need to specify either manual selectors or enable auto-source.
+- **`stylesheet.type invalid`:** Only `text/css` and `text/xsl` are supported for stylesheets.
 
 ### Missing Item Parts
 
@@ -46,6 +57,15 @@ If you are getting a "command not found" error, try the following:
 - **Re-install:** Re-install `html2rss` to ensure it is installed correctly: `gem install html2rss`.
 - **Check `PATH`:** Ensure that the directory where Ruby gems are installed is in your system's `PATH`.
 
+### Web Application Errors
+
+For html2rss-web specific issues:
+
+- **`401 Unauthorized`:** Check your `AUTO_SOURCE_USERNAME` and `AUTO_SOURCE_PASSWORD` environment variables.
+- **`403 Forbidden`:** The URL is not in the `AUTO_SOURCE_ALLOWED_URLS` list, or the origin is not in `AUTO_SOURCE_ALLOWED_ORIGINS`.
+- **`500 Internal Server Error`:** Check the application logs for detailed error information.
+- **Health check failures:** Use the `/health_check.txt` endpoint to identify which specific feed configurations are broken.
+
 ## Tips & Tricks
 
 - **Mobile Redirects:** Check that the channel URL does not redirect to a mobile page with a different markup structure.

web-application/reference/env-variables.md

@@ -12,7 +12,6 @@ grand_parent: Web Application
 
 | Name                           | Description                        |
 | ------------------------------ | ---------------------------------- |
-| `BASE_URL`                     | default: '<http://localhost:3000>' |
 | `LOG_LEVEL`                    | default: 'warn'                    |
 | `HEALTH_CHECK_USERNAME`        | default: auto-generated on start   |
 | `HEALTH_CHECK_PASSWORD`        | default: auto-generated on start   |
@@ -21,6 +20,7 @@ grand_parent: Web Application
 | `AUTO_SOURCE_USERNAME`         | no default.                        |
 | `AUTO_SOURCE_PASSWORD`         | no default.                        |
 | `AUTO_SOURCE_ALLOWED_ORIGINS`  | no default.                        |
+| `AUTO_SOURCE_ALLOWED_URLS`     | no default. Wildcard patterns supported. |
 |                                |                                    |
 | `PORT`                         | default: 3000                      |
 | `RACK_ENV`                     | default: 'development'             |
@@ -29,3 +29,6 @@ grand_parent: Web Application
 | `WEB_MAX_THREADS`              | default: 5                         |
 |                                |                                    |
 | `SENTRY_DSN`                   | no default.                        |
+|                                |                                    |
+| `RUBY_PATH`                    | default: 'ruby'                    |
+| `APP_ROOT`                     | default: '.'                       |