make tools easily installable as modules

[JessicaS11]

I'm trying to use IceFlow in another workflow, but it's non-trivial to install in its current configuration. You can't install the module using a full path because of the dashes in the repo name (NSIDC-Data_Tutorials), and if you add the repo to your path and try to import there are all sorts of relative path/module errors. @betolink are there any plans to set this up? Currently I'm modifying iceflow/__init__.py to import each module, and then debugging the relative path calls for each module.

[betolink]

This is on the radar, IceFlow as such is not really the notebook widget but the back-end that does the data harmonization. Having an installable library will simplify how to access the service for sure!

With that said a payload for a POST request to IceFlow will look like:

{
	"selection_criteria": {
		"filters": {
			"dataset_short_name": "ATM1B",
			"dataset_version": "1",
			"bounding_box": "-52.9,68.9,-47.3,69.7",
			"time_start": "1994-05-26 00:00:00",
			"time_end": "1994-05-27 00:00:00",
			"valkyrie_itrf": "ITRF2014",
			"valkyrie_epoch": "2018.5"
		}
	},
	"fulfillment": "valkyrie",
	"delivery": "valkyrie",
	"uid": "betolink"
}

This after we get authenticated with NSIDC's data ordering endpoint. So yeah having an easy to use library like IcePyx (or even incorporating it to IcePyx) would make so much sense. There are other shortcomings with IceFlow IMO, i.e. it delivers a single HDF5 file that does not use any compression, with point cloud data that's far from optimal. I think a V2 should create a Zarr file with chunking and compression or even use that point cloud format that Howard Butler was talking about at the last cloud native summit.

[JessicaS11]

Thanks for the clarifications! I was hoping to show the widget (I'm showcasing open-source tools for getting and integrating ICESat-2 data), but from the icepyx side I'd also love to have more widget-based front-end tools, so this could be a good jumping off point for that too. With all the dev time in the world... 😃

[betolink]

Another thing to be aware of, NSIDC is migrating its website and seems like recent work with the global proxy is causing issues with the authentication endpoint. I'll take a look at what's going on on Monday.

[asteiker]

CRYO-197

[JessicaS11]

@asteiker I don't use Jira so I don't have a login - is there a way to share the link so it's viewable without an account?

[asteiker]

@JessicaS11 I'm playing around with a synchronization from our GH repos to Jira where we currently manage our team's projects at NSIDC. I don't think this link is public, but it's simply pointing to a copy of this Issue. We will continue to work and communicate within GH so nothing's changing there. I apologize for leaving this Issue hanging for some time. Following on from your and the other UWG member's input last fall, we will be further assessing where we go with IceFlow and will communicate those plans with you all.

[JessicaS11]

Thanks @asteiker for the context. I thought it might be a response to the issue itself (which I'd forgotten about so no worries on it hanging). Happy to chat more about the future of IceFlow at some point if it's helpful.