Completely re-write the Gurobi MOI wrapper. by odow · Pull Request #216 · jump-dev/Gurobi.jl

odow · 2019-06-10T00:26:19Z

All tests pass locally

Most recent benchmarks compared to current Gurobi master:

========== Results ==========

delete_constraint
BenchmarkTools.TrialJudgement: 
  time:   -96.03% => improvement (5.00% tolerance)
  memory: -98.63% => improvement (1.00% tolerance)

add_variable_constraints
BenchmarkTools.TrialJudgement: 
  time:   -32.44% => improvement (5.00% tolerance)
  memory: -36.64% => improvement (1.00% tolerance)

add_variables
BenchmarkTools.TrialJudgement: 
  time:   -20.63% => improvement (5.00% tolerance)
  memory: -31.15% => improvement (1.00% tolerance)

add_constraint
BenchmarkTools.TrialJudgement: 
  time:   -48.83% => improvement (5.00% tolerance)
  memory: -46.73% => improvement (1.00% tolerance)

add_variable
BenchmarkTools.TrialJudgement: 
  time:   -29.04% => improvement (5.00% tolerance)
  memory: -82.91% => improvement (1.00% tolerance)

delete_variable_constraint
BenchmarkTools.TrialJudgement: 
  time:   -52.91% => improvement (5.00% tolerance)
  memory: -76.74% => improvement (1.00% tolerance)

add_variable_constraint
BenchmarkTools.TrialJudgement: 
  time:   -52.85% => improvement (5.00% tolerance)
  memory: -54.30% => improvement (1.00% tolerance)

add_constraints
BenchmarkTools.TrialJudgement: 
  time:   -29.20% => improvement (5.00% tolerance)
  memory: -16.59% => improvement (1.00% tolerance)

delete_variable
BenchmarkTools.TrialJudgement: 
  time:   -35.54% => improvement (5.00% tolerance)
  memory: +6315.52% => regression (1.00% tolerance)

!!! The massive increase in memory in delete_variable is due to the rebuild of the OrderedDict !!!

Notable changes include:

removing the dependence on LQOI
more comprehensive coverage of MOI tests
removing VectorAffine support

~~I found a couple of bugs in MOIT, will push a PR soon.~~

Todos

Closes #217
Closes #215
Closes #213
Closes #203
Closes #201
Closes #144

odow · 2019-06-10T00:55:38Z

+
+    @testset "start_values_test" begin
+        # We don't support ConstraintDualStart or ConstraintPrimalStart yet.
+        # @test_broken MOIT.start_values_test(Gurobi.Optimizer(GUROBI_ENV), OPTIMIZER)


TODO: consider implementing ConstraintDualStart and ConstraintPrimalStart

chriscoey · 2019-06-10T01:51:58Z

Are we saying bye bye to LQOI?

odow · 2019-06-10T14:55:37Z

Are we saying bye bye to LQOI?

Not definitely, but I would like to. I think this wrapper is considerably more readable than previously (it has a lot less re-direction) at the cost of some duplication (2000 lines compared to 500 + LQOI).

Because of all the tests, it was pretty easy to implement, if a little tedious. It also enables Gurobi-specific work-arounds like the needs_update thing.

chriscoey · 2019-06-10T16:33:24Z

Sounds reasonable. Hopefully it makes it easier to maintain and extend.

mlubin

Nice! Closes 5 issues :)
This really lowers the barrier to understanding the wrapper.

odow · 2019-06-11T20:07:22Z

Okay, now with an implementation of CleverDict™. Vector-backed storage so long as no elements are deleted. Keeps track of an offset so MOI.empty! works. Switches to Dict on delete!, and then back to vector on empty!.

odow · 2019-06-12T16:35:11Z

EDIT: latest benchmarks are at the top of the PR.

Now with a benchmarking script, that yields the following compared to Gurobi#master + LQOI#master.

add_constraints is slower because we haven't implemented that yet.

========== Results ==========

delete_constraint
BenchmarkTools.TrialJudgement: 
  time:   -53.10% => improvement (5.00% tolerance)
  memory: -76.74% => improvement (1.00% tolerance)

add_variable_constraint
BenchmarkTools.TrialJudgement: 
  time:   -47.47% => improvement (5.00% tolerance)
  memory: -85.41% => improvement (1.00% tolerance)

add_constraints
BenchmarkTools.TrialJudgement: 
  time:   +18.24% => regression (5.00% tolerance)
  memory: -55.91% => improvement (1.00% tolerance)

odow · 2019-06-13T19:15:05Z

Okay, things could do with a review again. I can't believe how much speed we left on the table with LQOI (see first post). The benchmarks definitely call for re-writing individual wrappers. It's also now much easier to test improvements via benchmarking. I wonder if we should roll these out via MOI.

mlubin · 2019-06-14T01:17:16Z

It's also now much easier to test improvements via benchmarking. I wonder if we should roll these out via MOI.

Makes sense. The CleverDict could also go to MOIU if it's going to be used in multiple wrappers.

joaquimg · 2019-06-21T03:16:41Z

+    # constraints cannot be distinguished from previously created ones.
+    last_constraint_index::Int
+    # ScalarAffineFunction{Float64}-in-Set storage.
+    affine_constraint_info::Dict{Int, ConstraintInfo}


Why these are not Clever ones?

Because the last_constraint_index counter is shared over the types of constraints, so that there would be large holes in the CleverDict Array automatically.

Note, that this is not necessary, since the i in ConstraintIndex{F,S}(i) must only be unique per F, S type (see docstring). It would probably be better to switch to last_{affine,sos,quadratic}_constraint_index and CleverDict for each of them.

Mainly because it's not performance critical. The only time we really need it is querying duals/ConstraintFunctions/etc. The extra hash doesn't matter. In contrast, we get column indices all the time, so it makes sense to make these fast.

odow · 2019-07-11T16:49:02Z

blegat · 2019-07-11T17:10:20Z

What is the 0printed at the end ?

remi-garcia · 2019-08-08T16:24:29Z

I'm reading this to learn and I found a typo, not exactly in the PR though
https://github.com/JuliaOpt/Gurobi.jl/blob/bf90918ab72943ffb06713d72db39a63ef65a362/src/MOI_wrapper.jl#L197-L208
_require_update -> _update_if_necessary

odow · 2019-08-08T16:27:34Z

Thanks @remi-garcia. I've corrected.

joaquimg · 2019-08-13T10:15:37Z

MOI 0.9 almost ready.
@odow do you have plans to take a last look in this PR before we can do the same for xpress and cplex?

odow · 2019-08-13T13:04:08Z

I'll take a look today. You should be able take a start with Xpress. It should be sufficient to copy paste the whole thing, and
then change out the Xpress specific calls.

odow · 2019-08-13T13:43:50Z

~~Currently getting these errors:~~

Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Test Failed at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:68 Expression: MOI.get(model, MOI.VariableIndex, "Var1") Expected: Exception No exception thrown Stacktrace: [1] macro expansion at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:68 [inlined] [2] macro expansion at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [inlined] [3] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:22 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Test Failed at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:120 Expression: MOI.get(model, MOI.ConstraintIndex, "Con2") Expected: Exception No exception thrown Stacktrace: [1] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:120 [2] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [3] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [4] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [5] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [6] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:48 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Test Failed at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:121 Expression: MOI.get(model, typeof(c2), "Con2") Expected: Exception No exception thrown Stacktrace: [1] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:121 [2] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [3] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [4] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [5] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [6] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:48 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Test Failed at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:122 Expression: MOI.get(model, typeof(cx), "Con2") Expected: Exception No exception thrown Stacktrace: [1] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:122 [2] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [3] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [4] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [5] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [6] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:48 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Test Failed at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:154 Expression: MOI.get(model, MOI.ConstraintIndex, "Con2") Expected: Exception No exception thrown Stacktrace: [1] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:154 [2] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [3] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [4] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [5] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [6] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:48 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Test Failed at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:155 Expression: MOI.get(model, typeof(c2), "Con2") Expected: Exception No exception thrown Stacktrace: [1] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:155 [2] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [3] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [4] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [5] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [6] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:48 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Test Failed at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:156 Expression: MOI.get(model, typeof(cx), "Con2") Expected: Exception No exception thrown Stacktrace: [1] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:156 [2] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [3] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [4] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [5] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [6] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:48 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}: Error During Test at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:21 Got exception outside of a @test MethodError: Cannot `convert` an object of type MathOptInterface.ConstraintIndex{MathOptInterface.ScalarAffineFunction{Float64},MathOptInterface.EqualTo{Float64}} to an object of type String Closest candidates are: convert(::Type{T<:AbstractString}, !Matched::T<:AbstractString) where T<:AbstractString at strings/basic.jl:207 convert(::Type{T<:AbstractString}, !Matched::AbstractString) where T<:AbstractString at strings/basic.jl:208 convert(::Type{T}, !Matched::T) where T at essentials.jl:154 Stacktrace: [1] setindex!(::Dict{String,MathOptInterface.ConstraintIndex}, ::String, ::MathOptInterface.ConstraintIndex{MathOptInterface.ScalarAffineFunction{Float64},MathOptInterface.EqualTo{Float64}}) at ./dict.jl:373 [2] set(::Gurobi.Optimizer, ::MathOptInterface.ConstraintName, ::MathOptInterface.ConstraintIndex{MathOptInterface.ScalarAffineFunction{Float64},MathOptInterface.EqualTo{Float64}}, ::String) at /Users/oscar/.julia/dev/Gurobi/src/MOI_wrapper.jl:1348 [3] set at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Bridges/bridge_optimizer.jl:619 [inlined] [4] macro expansion at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:171 [inlined] [5] macro expansion at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [inlined] [6] nametest(::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}) at /Users/oscar/.julia/packages/MathOptInterface/mvlGI/src/Test/modellike.jl:22 [7] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [8] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [9] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:59 [10] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [11] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/MOI_wrapper.jl:48 [12] include at ./boot.jl:326 [inlined] [13] include_relative(::Module, ::String) at ./loading.jl:1038 [14] include(::Module, ::String) at ./sysimg.jl:29 [15] include(::String) at ./client.jl:403 [16] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/runtests.jl:17 [17] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083 [18] top-level scope at /Users/oscar/.julia/dev/Gurobi/test/runtests.jl:17 [19] include at ./boot.jl:326 [inlined] [20] include_relative(::Module, ::String) at ./loading.jl:1038 [21] include(::Module, ::String) at ./sysimg.jl:29 [22] include(::String) at ./client.jl:403 [23] top-level scope at none:0 [24] eval(::Module, ::Any) at ./boot.jl:328 [25] exec_options(::Base.JLOptions) at ./client.jl:243 [26] _start() at ./client.jl:436 Academic license - for non-commercial use only Academic license - for non-commercial use only Academic license - for non-commercial use only Academic license - for non-commercial use only Academic license - for non-commercial use only Academic license - for non-commercial use only Test Summary: | Pass Fail Error Broken Total MathOptInterface Tests | 2085 7 1 1 2094 Unit Tests | 807 807 Linear tests | 517 517 Quadratic tests | 190 190 Linear Conic tests | 135 135 Integer Linear tests | 139 139 ModelLike tests | 213 7 1 1 222 default_objective_test | No tests default_status_test | 3 3 nametest | 103 7 1 111 Name test with MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer} | 103 7 1 111 validtest | 11 11 emptytest | 12 12 orderedindicestest | 13 13 copytest | 37 37 scalar_function_constant_not_zero | 2 2 start_values_test | 3 3 supports_constrainttest | 8 1 9 set_lower_bound_twice | 10 10 set_upper_bound_twice | 10 10 Gurobi Callback | 10 10 LQOI Issue #38 | No tests User limit handling (issue #140) | 3 3 Constant objective (issue #111) | 4 4 Env | 11 11 Conflict refiner | 41 41 RawParameter | 2 2 QCPDuals without needing to pass QCPDual=1 | 12 12 Add constraints | 1 1 ERROR: LoadError: Some tests did not pass: 2085 passed, 7 failed, 1 errored, 1 broken. in expression starting at /Users/oscar/.julia/dev/Gurobi/test/runtests.jl:16 ERROR: Package Gurobi errored during testing

~~cc @blegat~~ Fix incoming.

odow · 2019-08-13T15:24:27Z

odow · 2019-08-13T16:10:40Z

odow mentioned this pull request Jun 10, 2019

Fix tests from Gurobi #216 jump-dev/MathOptInterface.jl#763

Merged

odow changed the title ~~WIP: Completely re-write the Gurobi MOI wrapper.~~ Completely re-write the Gurobi MOI wrapper. Jun 10, 2019

odow commented Jun 10, 2019

View reviewed changes

Comment thread test/MOI_wrapper.jl Outdated

odow commented Jun 10, 2019

View reviewed changes

Comment thread test/MOI_wrapper.jl Outdated

odow commented Jun 10, 2019

View reviewed changes

blegat reviewed Jun 10, 2019

View reviewed changes

Comment thread src/MOI_wrapper.jl Outdated

blegat reviewed Jun 10, 2019

View reviewed changes

Comment thread src/MOI_wrapper.jl Outdated

blegat reviewed Jun 10, 2019

View reviewed changes

Comment thread src/MOI_wrapper.jl Outdated

mlubin reviewed Jun 11, 2019

View reviewed changes

mlubin reviewed Jun 12, 2019

View reviewed changes

Comment thread Project.toml

Comment thread perf/perf.jl Outdated

Comment thread src/CleverDict.jl Outdated

Comment thread src/CleverDict.jl Outdated

Comment thread src/CleverDict.jl Outdated

coroa mentioned this pull request Jun 13, 2019

Benchmark MOI PyPSA/EnergyModels.jl#16

Closed

odow mentioned this pull request Jun 13, 2019

[WIP] Implementation of write_to_file that use MathOptFormat. jump-dev/JuMP.jl#1982

Closed

mlubin approved these changes Jun 14, 2019

View reviewed changes

Comment thread src/CleverDict.jl Outdated

blegat mentioned this pull request Jun 14, 2019

Update solvers to MOI v0.9 jump-dev/MathOptInterface.jl#736

Closed

20 tasks

blegat reviewed Jun 14, 2019

View reviewed changes

Comment thread src/MOI_wrapper.jl Outdated

blegat approved these changes Jun 14, 2019

View reviewed changes

odow mentioned this pull request Jun 14, 2019

Add CleverDicts as submodule to MOI.Utilities jump-dev/MathOptInterface.jl#767

Merged

blegat reviewed Jun 15, 2019

View reviewed changes

Comment thread src/MOI_wrapper.jl Outdated

Comment thread src/MOI_wrapper.jl Outdated

joaquimg reviewed Jun 21, 2019

View reviewed changes

odow added 4 commits July 11, 2019 11:00

Remove benchmarking in favour of MOI.Benchmarks

8676f49

Add test for VariablePrimalStart

32831bc

Enable QCPDual by default

c523cfb

Update merge with IIS conflits

e6c869d

odow force-pushed the od/moi9 branch from f164589 to e6c869d Compare July 11, 2019 16:28

odow added 3 commits July 11, 2019 11:34

Fix bug in add_constraints

b6431c4

Remove unneeded updates

8158648

MOI.ModelLike -> MOI.AbstractOptimizer

2061916

Remove print from tests

bf90918

Update MOI_wrapper.jl

74f7526

Add TimeLimitSec

3b577d4

Fix handling of duplicate constraint names

f55a419

Fix updating of names and add extra tests

e2f9b15

This was referenced Aug 13, 2019

Add extra tests to nametest jump-dev/MathOptInterface.jl#833

Merged

[MOI] Set Gurobi-specific attributes #211

Closed

Change version back to 0.7.0

d42ff2e

odow merged commit ffdc849 into master Aug 13, 2019

odow deleted the od/moi9 branch August 13, 2019 23:49

odow mentioned this pull request Aug 13, 2019

VariablePrimalStart doesn't support incremental setting #225

Closed

This was referenced Aug 18, 2019

New MOI Wrapper jump-dev/Xpress.jl#40

Closed

Update wrapper to MOI 0.9 jump-dev/CPLEX.jl#249

Closed

Conversation

odow commented Jun 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

odow Jun 10, 2019

Choose a reason for hiding this comment

Uh oh!

chriscoey commented Jun 10, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

odow commented Jun 10, 2019

Uh oh!

chriscoey commented Jun 10, 2019

Uh oh!

mlubin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

odow commented Jun 11, 2019

Uh oh!

odow commented Jun 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

odow commented Jun 13, 2019

Uh oh!

Uh oh!

mlubin commented Jun 14, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joaquimg Jun 21, 2019

Choose a reason for hiding this comment

Uh oh!

coroa Jul 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

odow Jul 11, 2019

Choose a reason for hiding this comment

Uh oh!

odow commented Jul 11, 2019

Uh oh!

blegat commented Jul 11, 2019

Uh oh!

remi-garcia commented Aug 8, 2019

Uh oh!

odow commented Aug 8, 2019

Uh oh!

joaquimg commented Aug 13, 2019

Uh oh!

odow commented Aug 13, 2019

Uh oh!

odow commented Aug 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odow commented Aug 13, 2019

Uh oh!

odow commented Aug 13, 2019

Uh oh!

Reviewers

Assignees

Labels

odow commented Jun 10, 2019 •

edited

Loading

odow commented Jun 12, 2019 •

edited

Loading

coroa Jul 11, 2019 •

edited

Loading

odow commented Aug 13, 2019 •

edited

Loading