Generating several N+1 queries when adding multiple resources to an export thus slowing down query and timing out Heroku

[subratrout]

When I am trying to export a resource which has two or three other related models data to be exported along with it, it is generating several N+1 queries and with couple of thousand rows I am facing server time out issues.
Upon installing bullet gem I get following on my bullet.log

USE eager loading detected
  Resource => [:sponsorships]
  Add to your query: .includes([:sponsorships])

GET /admin/resources/export.csv
USE eager loading detected
  Resource => [:organizations]
  Add to your query: .includes([:organizations])

Just for example my resource_dashboard contains following: 


```ATTRIBUTE_TYPES = {
    id: Field::Number,
    name: Field::String,
    resource_type: Field::Enumerate,
    organizations: Field::HasMany.with_options(
      transform_on_export: -> (field) { field.data&.map(&:name)&.join(", ")  if field.data }
    ),
    event_detail: Field::HasOne,
    questions: Field::NestedHasMany.with_options(
      skip: :resource,
      transform_on_export: -> (field) { field.data&.map(&:text)&.join(", ") if field.data }
    ),


Is there a way to speed up query or prevent eager loading?

[andreibondarev]

@subratrout A couple of thoughts-- it looks like within the transform_on_export you may be select-ing all fields from associations. Replacing field.data&.map(&:name) with field.data&.pluck(:name) might pull a smaller ActiveRecord object.

If you're on Heroku, how many records are you attempting to export and how much RAM do you have configured on the dyno?

[subratrout]

The table contains only 2100 rows. I am able to export if exporting from single table, however when I join one or more tables that's when it times out. The app using two standard-2x-dyno with 1GB Memory each.

[MichalJaneda]

You may preload them manualy by overriding controller method.
Here is my example:

def scoped_resource
  return super.preload(:operational_region, :logistic_region) if action_name == 'export'

  super
end

[DavidGeismarLtd]

Is there any way to configure the gem to run the export method in a sidekiq job ?

[andreibondarev]

@DavidGeismarLtd Not currently. It's rather complex. What issue are you having? Server timing out?

[mattcarey]

We are getting a H12 error on Heroku when exporting to CSV using this gem. Our export isn't big but has a few pieces of data from other tables. Any idea of a solution please?

[andreibondarev]

We are getting a H12 error on Heroku when exporting to CSV using this gem. Our export isn't big but has a few pieces of data from other tables. Any idea of a solution please?

If you copied your production DB to local -- are you able to successfully generate the export?