feat: update strategy, request and cli

Author: gildesmarais

src/content/docs/ruby-gem/how-to/custom-http-requests.mdx

@@ -5,6 +5,12 @@ description: "Learn how to customize HTTP requests with custom headers, authenti
 
 Some websites require custom HTTP headers, authentication, or other request settings to access their content. `html2rss` lets you customize requests for those cases.
 
+Keep this structure in mind:
+
+- `headers` stays top-level
+- `strategy` stays top-level
+- request-specific controls such as budgets and Browserless options live under `request`
+
 ## When You Need Custom Headers
 
 You might need custom HTTP requests when:
@@ -35,6 +41,32 @@ selectors:
     selector: "url"
 ```
 
+## Request Controls
+
+Request budgets are configured under `request`, not as top-level keys:
+
+```yaml
+headers:
+  User-Agent: "Mozilla/5.0 (compatible; html2rss/1.0)"
+request:
+  max_redirects: 5
+  max_requests: 6
+channel:
+  url: https://example.com/articles
+selectors:
+  items:
+    selector: article
+  title:
+    selector: h2
+  url:
+    selector: a
+    extractor: href
+```
+
+- `request.max_redirects` limits redirect hops
+- `request.max_requests` limits the total request budget for the feed build
+- `request.browserless.*` is reserved for Browserless-only behavior such as preload actions
+
 ## Common Use Cases
 
 ### API Authentication

src/content/docs/ruby-gem/how-to/handling-dynamic-content.mdx

@@ -9,6 +9,29 @@ Some websites load their content dynamically using JavaScript. The default `html
 
 Use the [`browserless` strategy](/ruby-gem/reference/strategy) to render JavaScript-heavy websites with a headless browser.
 
+Keep the strategy at the top level and put request-specific options under `request`:
+
+```yaml
+strategy: browserless
+request:
+  max_redirects: 5
+  max_requests: 6
+  browserless:
+    preload:
+      wait_for_network_idle:
+        timeout_ms: 5000
+channel:
+  url: https://example.com/app
+selectors:
+  items:
+    selector: .article
+  title:
+    selector: h2
+  url:
+    selector: a
+    extractor: href
+```
+
 ## When to Use Browserless
 
 The `browserless` strategy is necessary when:
@@ -18,6 +41,56 @@ The `browserless` strategy is necessary when:
 - **Infinite scroll** - Content loads as you scroll
 - **Dynamic forms** - Content changes based on user interaction
 
+## Preload Actions
+
+For dynamic sites, rendering once is often not enough. Use `request.browserless.preload` to wait, click, or scroll before the
+HTML snapshot is taken.
+
+### Wait for JavaScript Requests
+
+```yaml
+strategy: browserless
+request:
+  browserless:
+    preload:
+      wait_for_network_idle:
+        timeout_ms: 4000
+```
+
+### Click "Load More" Buttons
+
+```yaml
+strategy: browserless
+request:
+  browserless:
+    preload:
+      click_selectors:
+        - selector: ".load-more"
+          max_clicks: 3
+          delay_ms: 250
+          wait_for_network_idle:
+            timeout_ms: 3000
+```
+
+### Scroll Infinite Lists
+
+```yaml
+strategy: browserless
+request:
+  browserless:
+    preload:
+      scroll_down:
+        iterations: 5
+        delay_ms: 200
+        wait_for_network_idle:
+          timeout_ms: 2500
+```
+
+These preload steps can be combined in a single config when a site needs several interactions before all items appear.
+
+If a click or scroll step causes a real navigation, html2rss returns the final document metadata, not the original page-load
+metadata. That keeps extracted relative links anchored to the rendered page.
+
 ## Performance Considerations
 
 The `browserless` strategy is slower than the default `faraday` strategy because it:

src/content/docs/ruby-gem/reference/cli-reference.mdx

@@ -24,6 +24,9 @@ html2rss auto https://example.com/articles
 # Force browserless for JavaScript-heavy pages
 html2rss auto https://example.com/app --strategy browserless
 
+# Override request budgets at runtime
+html2rss auto https://example.com/app --strategy browserless --max-redirects 5 --max-requests 6
+
 # Hint the item selector while keeping auto enhancement
 html2rss auto https://example.com/articles --items_selector ".post-card"
 ```
@@ -44,12 +47,17 @@ html2rss feed feeds.yml my-first-feed
 # Override the request strategy at runtime
 html2rss feed single.yml --strategy browserless
 
+# Override request budgets at runtime
+html2rss feed single.yml --max-redirects 5 --max-requests 6
+
 # Pass dynamic parameters into %<param>s placeholders
 html2rss feed single.yml --params id:42 foo:bar
 ```
 
 Command: `html2rss feed YAML_FILE [feed_name]`
 
+The CLI keeps `strategy` as a top-level override and writes runtime request limits into the generated config under `request`.
+
 ### Schema
 
 Prints the exported JSON Schema for the current gem version.

src/content/docs/ruby-gem/reference/strategy.mdx

@@ -8,6 +8,8 @@ The `strategy` key defines how `html2rss` fetches a website's content.
 - **`faraday`** (default): Makes a direct HTTP request. It is fast but does not execute JavaScript.
 - **`browserless`**: Renders the website in a headless Chrome browser, which is necessary for JavaScript-heavy sites.
 
+`strategy` stays a top-level config key. Request-specific controls now live under `request`.
+
 ## `browserless`
 
 To use the `browserless` strategy, you need a running instance of [Browserless.io](https://www.browserless.io/).
@@ -27,10 +29,48 @@ docker run \
 
 ### Configuration
 
-Set the `strategy` at the top level of your feed configuration:
+Set the `strategy` at the top level of your feed configuration and put request controls under `request`:
+
+```yml
+strategy: browserless
+request:
+  max_redirects: 5
+  max_requests: 6
+channel:
+  url: "https://example.com/app"
+selectors:
+  items:
+    selector: ".article"
+  title:
+    selector: "h2"
+  url:
+    selector: "a"
+    extractor: "href"
+```
+
+### Request Structure
+
+Use this split consistently:
+
+- `strategy`: selects `faraday` or `browserless`
+- `headers`: top-level headers shared by all strategies
+- `request.max_redirects`: redirect limit for the request session
+- `request.max_requests`: total request budget for the whole feed build
+- `request.browserless.*`: Browserless-only options
+
+Example:
 
 ```yml
 strategy: browserless
+headers:
+  User-Agent: "Mozilla/5.0 (compatible; html2rss/1.0)"
+request:
+  max_redirects: 5
+  max_requests: 6
+  browserless:
+    preload:
+      wait_for_network_idle:
+        timeout_ms: 5000
 channel:
   url: "https://example.com/app"
 selectors:
@@ -43,6 +83,38 @@ selectors:
     extractor: "href"
 ```
 
+### Browserless Preload
+
+Browserless can interact with the page before html2rss captures the final HTML. Configure preload steps under
+`request.browserless.preload`.
+
+```yml
+strategy: browserless
+request:
+  browserless:
+    preload:
+      wait_for_network_idle:
+        timeout_ms: 5000
+      click_selectors:
+        - selector: ".load-more"
+          max_clicks: 3
+          delay_ms: 250
+          wait_for_network_idle:
+            timeout_ms: 4000
+      scroll_down:
+        iterations: 5
+        delay_ms: 200
+        wait_for_network_idle:
+          timeout_ms: 3000
+```
+
+- `wait_for_network_idle`: pauses before and after preload steps
+- `click_selectors`: clicks matching elements until they disappear or `max_clicks` is reached
+- `scroll_down`: scrolls until the page height stops growing or `iterations` is reached
+
+If preload triggers a real navigation or redirect, html2rss keeps the final document metadata. Relative links and follow-up
+pagination therefore resolve against the page that was actually rendered after preload completed.
+
 ### Command-Line Usage
 
 You can also specify the strategy on the command line:
@@ -53,6 +125,9 @@ BROWSERLESS_IO_WEBSOCKET_URL="ws://127.0.0.1:3000" \
 BROWSERLESS_IO_API_TOKEN="6R0W53R135510" \
   html2rss feed my_config.yml --strategy browserless
 
+# Override request budgets at runtime
+html2rss feed my_config.yml --max-redirects 5 --max-requests 6
+
 # Or rely on the strategy stored in the YAML config
 html2rss feed my_config.yml
 ```