Debug fastpath

[aagbsn]

add some metrics to monitor fastpath behavior more closely

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.62%. Comparing base (dbf3cca) to head (337e845).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1163      +/-   ##
==========================================
+ Coverage   91.84%   98.62%   +6.78%     
==========================================
  Files          85       29      -56     
  Lines        7907     2552    -5355     
  Branches      476       95     -381     
==========================================
- Hits         7262     2517    -4745     
+ Misses        539       32     -507     
+ Partials      106        3     -103     

Flag	Coverage Δ
ooniauth	100.00% <ø> (ø)
oonifindings	97.31% <ø> (ø)
oonimeasurements	?
ooniprobe	?
oonirun	98.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LDiazN]

Is it possible to also log the queue size from here?

backend/fastpath/fastpath/localhttpfeeder.py

Line 40 in 54dec8d

queue.put(msm_tup, block=True, timeout=QUEUE_PUT_TIMEOUT)

And additionally maybe a "queue percent" metric to check how close to full the queue usually is

[hellais]

Is it possible to also log the queue size from here?

backend/fastpath/fastpath/localhttpfeeder.py

Line 40 in 54dec8d

queue.put(msm_tup, block=True, timeout=QUEUE_PUT_TIMEOUT)

And additionally maybe a "queue percent" metric to check how close to full the queue usually is

I'm not sure we can log metrics from inside of the gunicorn worker, or what impact it would have to add logging there.

What we can do however is add a metric inside of the worker that looks up queue.qsize()(https://docs.python.org/3/library/queue.html#queue.Queue.qsize). I think we can also record the timing for queue.get() operations since those are blocking and we can check how much the delay in getting items from the queue are.