Scaler fails only when failing to get counts from all the interceptor endpoints

[Mizhentaotuo]

Provide a description of what has been changed
We observe behavior that the scaler fails and exit the loop when failing to get counts from any of the interceptor replica.
Not sure this is the intended behavior but sometimes one interceptor replica is down only because it is on a spot node. When the node is down and the endpoints of the interceptor service is not updated yet, the scaler still try to get from and endpoint which does not exist. And most of the time the killed interceptor pod will heal itself.

Checklist

• Commits are signed with Developer Certificate of Origin (DCO)
• Changelog has been updated and is aligned with our changelog requirements
• Any necessary documentation is added, such as:
  • README.md
  • The docs/ directory
  • The docs repo

Fixes #
Change so that the scaler fails only when fetching all the counts failed.

Comment:
I am new to this, not sure the existing version is the intended behavior. Please let me know if there is a better way or it can be handled by any config value that I am not aware of. Appreciated.

[JorTurFer]

Hey!
Thanks for the PR, but I don't get the point (but probably I'm missing something), the interceptor cache is updated every second, so I'd expect that the Scaler status isn't really affected because when you drain a node, the endpoints should get removed quite fast from the list of endpoints. Am I missing something?
We decided to restart the scaler if the metrics aren't available as getting all the endpoints it's the only way to ensure that the traffic is measured well, otherwise the scaler could respond with a wrong value which triggers the scaling in, impacting the users.

[JorTurFer]

Said that, I guess that we could try to figure out a better way to hit the interceptors, something like getting the ready pods and going through calculating the endpoints in scaler side instead of using k8s endpoints.

[Mizhentaotuo]

Hey! Thanks for the PR, but I don't get the point (but probably I'm missing something), the interceptor cache is updated every second, so I'd expect that the Scaler status isn't really affected because when you drain a node, the endpoints should get removed quite fast from the list of endpoints. Am I missing something? We decided to restart the scaler if the metrics aren't available as getting all the endpoints it's the only way to ensure that the traffic is measured well, otherwise the scaler could respond with a wrong value which triggers the scaling in, impacting the users.

Hey! Thanks a lot for the quick reply.

so I'd expect that the Scaler status isn't really affected because when you drain a node, the endpoints should get removed quite fast from the list of endpoints. Am I missing something?

No, no I agree that is the expected behavior. But the behavior on our cluster is that the scaler failed because one node is removed by gcp, and it spin up another node which takes some time (could be 1 min). could be that the endpoints list is updated, but just because the node is not ready yet, so the pod is not ready either? This part I do not know much about.
in any case, I think your idea of getting the ready pod first is nice, I will try to update the PR.
Thanks again!