Generalized paramset insert?

[CBroz1]

Currently, multiple elements (and sometimes multiple schemas within each) have a paramset insert function

• ephys
• calcium imaging
• deeplabcut (under revision)

Should element-interface host a generalized version of this process?

[kabilar]

Thanks @CBroz1. Yes, if possible, let's centralize. The attributes are not exactly the same in all cases so it will require refactoring.

Also, @dimitri-yatsenko had suggested the following refactor for ClusteringParamSet.insert_new_params. I would suggest keeping the function name as insert_new_params to ensure backwards compatibility.

@classmethod
def insert_params(cls, clustering_method: str, paramset_idx: int, paramset_desc: str, params: dict):
    param_hash = {'param_set_hash': dict_to_uuid(params)}
    try:
        existing_idx = (cls & param_hash).fetch1('paramset_idx')
    except dj.DataJointError:
        cls.insert1(dict(locals(), **param_hash), ignore_extra_fields=True)
    else:
        if existing_idx != paramset_idx: 
            raise dj.DataJointError(
                f'The specified param-set already exists - paramset_idx: {existing_idx}')

[dimitri-yatsenko]

Aha!

If the idea is to skip duplicates if the secondary parameters match, then we can consider the following type of implementation:

def insert1_skip_full_duplicates(table, entry):
    """
    This function inserts one entry into the table. 
    It ignores duplicates on if all the other entries also match.
    Duplicates on secondary keys are not ignored.
    After validation, this functionality will be integrating into core DataJoint.
    """
    try:
        table.insert1(entry)
    except dj.errors.DuplicateError as err:
        if "PRIMARY" not in err.args[0]:
            # secondary indexes are checked only after confirming primary uniqueness
            raise  
        key = {k: v for k, v in entry.items() if k in table.primary_key}
        if (table & key).fetch1() != entry:
            raise err.suggest(
                'Duplicate primary key with different secondary attributes from existing value.')
            
    
                                  
@schema
class PreprocessParamSet(dj.Lookup):
    definition = """  #  Parameter set used for pre-processing of calcium imaging data
    paramset_idx:  smallint
    ---
    -> PreprocessMethod
    paramset_desc: varchar(128)
    param_set_hash: uuid
    unique index (param_set_hash)
    params: longblob  # dictionary of all applicable parameters
    """

    @classmethod
    def insert_params(cls, paramset_idx: int, *, preprocess_method: str, paramset_desc: str, params: dict):
        """
        Insert a set of parameters. Ignore duplicates unless attempting to enter 
        the same parameters under a different primary key
        """
        # construct entry from input arguments
        param_set_hash = dict_to_uuid(params)
        entry = dict({k: v for k, v in locals().items() if k in cls.heading.names})
        insert1_skip_full_duplicates(cls(), entry)

[dimitri-yatsenko]

I would put the primary key as the first argument. To enforce backward validity, I would put * after the first argument as in the example above.

[dimitri-yatsenko]

Let's have a quick code review on this and propagate throughout.

[ttngu207]

I'd replace

entry = dict({k: v for k, v in locals().items() if k in cls.heading.names})

with explicitly creating the dictionary

entry = dict(paramset_idx=paramset_idx, paramset_desc=paramset_desc, param_set_hash=param_set_hash, params=params)

It's a bit longer but more readable compared to using locals()

[ttngu207]

@schema
class PreprocessParamSet(dj.Lookup):
    definition = """  #  Parameter set used for pre-processing of calcium imaging data
    paramset_idx:  smallint
    ---
    -> PreprocessMethod
    paramset_desc: varchar(128)
    param_set_hash: uuid
    unique index (param_set_hash)
    params: longblob  # dictionary of all applicable parameters
    """

    @classmethod
    def insert_params(cls, paramset_idx: int, *, preprocess_method: str, paramset_desc: str, params: dict):
        """
        Insert a set of parameters. Ignore duplicates unless attempting to enter 
        the same parameters under a different primary key
        """
        # construct entry from input arguments
        param_set_hash = dict_to_uuid({**params, 'preprocess_method': preprocess_method})
        entry = dict(paramset_idx=paramset_idx, 
          paramset_desc=paramset_desc, 
          param_set_hash=param_set_hash, 
          params=params)
        insert1_skip_full_duplicates(cls(), entry)

note that the param_set_hash needs to include also the preprocess_method

[dimitri-yatsenko]

note that the param_set_hash needs to include also the preprocess_method

But not paramset_desc?

[tdincer]

Let's not complicate it. process_method (or preprocess_method) is to indicate the suite2p vs caiman like information. These cases would have completely different set of parameters.