Skip to content

[AI-FSSDK] [FSSDK-12369] Add local holdouts support with includedRules field#398

Open
Mat001 wants to merge 4 commits into
masterfrom
ai/mat001/FSSDK-12369-local-holdouts
Open

[AI-FSSDK] [FSSDK-12369] Add local holdouts support with includedRules field#398
Mat001 wants to merge 4 commits into
masterfrom
ai/mat001/FSSDK-12369-local-holdouts

Conversation

@Mat001
Copy link
Copy Markdown
Contributor

@Mat001 Mat001 commented May 14, 2026

Summary

Adds Local Holdouts support to the Ruby SDK. Local holdouts allow targeting specific experiment and delivery rules within a flag via a new includedRules field, while global holdouts (where includedRules is absent or nil) continue to apply to all rules across all flags.

Changes

  • Added includedRules optional field support to holdout data model in DatafileProjectConfig
  • Added holdout_global? predicate, get_global_holdouts, and get_holdouts_for_rule methods to config
  • Updated get_variation_from_experiment_rule and get_variation_from_delivery_rule in DecisionService to check local holdouts per rule after forced decisions
  • Updated JSON schema in constants.rb to allow includedRules field in holdout objects
  • Added comprehensive Level 1 (config/parsing) and Level 2 (decision service) unit tests

Jira Ticket

FSSDK-12369

Mat001 and others added 3 commits May 14, 2026 15:34
- Fix Layout/SpaceInsideHashLiteralBraces in datafile_project_config_spec.rb
- Fix Lint/UselessAssignment by removing unused feature_flag variable in decision_service_holdout_spec.rb

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mat001
Copy link
Copy Markdown
Contributor Author

Mat001 commented May 18, 2026

@esrakartalOpt WHen you are reviewing this PR, please note this and see maybe with Muzahid if this needs to be fixed or not. It was there before local holdouts (came in with global hodlouts):

Bug: User Profile Service bypassed when holdouts are present in the datafile

File: lib/optimizely/decision_service.rb
Introduced in: FSSDK-11577 (original holdouts implementation)

What's happening

get_variation_for_feature routes to one of two paths based on whether any Running holdouts exist in the datafile:

def get_variation_for_feature(project_config, feature_flag, user_context, decide_options = [])
running_holdouts = project_config.holdout_id_map.values

if running_holdouts && !running_holdouts.empty?
# Holdout path — calls get_decision_for_flag directly, no UPS
get_decision_for_flag(feature_flag, user_context, project_config, decide_options)
else
# No-holdout path — goes through get_variations_for_feature_list, which sets up UPS
get_variations_for_feature_list(project_config, [feature_flag], user_context, decide_options).first
end
end
The no-holdout path goes through get_variations_for_feature_list, which properly sets up the User Profile Service:

def get_variations_for_feature_list(...)
user_profile_tracker = UserProfileTracker.new(user_id, @user_profile_service, @logger)
user_profile_tracker.load_user_profile # ← sticky bucketing loaded

feature_flags.each do |feature_flag|
decision = get_decision_for_flag(..., user_profile_tracker) # ← passed in
end

user_profile_tracker.save_user_profile # ← sticky bucketing saved
end
The holdout path calls get_decision_for_flag directly without a user_profile_tracker, which defaults to nil. UPS is never loaded or saved — meaning sticky bucketing is silently skipped for all users whenever the datafile contains Running holdouts, even for users who don't hit any holdout and fall through to normal experiment bucketing.

Impact

Only affects projects using a custom user_profile_service for sticky bucketing
Any environment with Running holdouts configured will silently lose sticky bucketing
Users who miss all holdouts and get bucketed into normal experiments will not have their assignments persisted or restored from UPS
What the fix should look like
The holdout path needs the same UPS setup as the no-holdout path. The simplest fix is to eliminate the routing gate entirely and always go through get_variations_for_feature_list (which already calls get_decision_for_flag internally, so holdout logic is unaffected):

def get_variation_for_feature(project_config, feature_flag, user_context, decide_options = [])
get_variations_for_feature_list(project_config, [feature_flag], user_context, decide_options).first
end
Or, if the direct path is kept for performance reasons, replicate the UPS setup there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant