Skip to content

Commit 3a6cbb4

Browse files
mvdocmvdoc
authored
FIX update setup colab notebook to download data with datalad (#13)
* FIX use datalad to download data on colab * FIX path and dev null * Update merged notebook for colab * FIX path for colab * Update merged notebooks * FIX modify python files
1 parent 12f7e77 commit 3a6cbb4

3 files changed

Lines changed: 305 additions & 173 deletions

File tree

‎tutorials/notebooks/shortclips/00_setup_colab.ipynb‎

Lines changed: 193 additions & 120 deletions
Original file line numberDiff line numberDiff line change
@@ -1,122 +1,195 @@
11
{
2-
"cells": [
3-
{
4-
"cell_type": "code",
5-
"execution_count": null,
6-
"metadata": {
7-
"collapsed": false
8-
},
9-
"outputs": [],
10-
"source": [
11-
"%matplotlib inline"
12-
]
13-
},
14-
{
15-
"cell_type": "markdown",
16-
"metadata": {},
17-
"source": [
18-
"\n# Setup Google Colab\n\nIn this script, we setup a Google Colab environment. This script will only work\nwhen run from `Google Colab <https://colab.research.google.com/>`_). You can\nskip it if you run the tutorials on your machine.\n"
19-
]
20-
},
21-
{
22-
"cell_type": "code",
23-
"execution_count": null,
24-
"metadata": {
25-
"collapsed": false
26-
},
27-
"outputs": [],
28-
"source": [
29-
""
30-
]
31-
},
32-
{
33-
"cell_type": "markdown",
34-
"metadata": {},
35-
"source": [
36-
"## Change runtime to use a GPU\n\nThis tutorial is much faster when a GPU is available to run the computations.\nIn Google Colab you can request access to a GPU by changing the runtime type.\nTo do so, click the following menu options in Google Colab:\n\n(Menu) \"Runtime\" -> \"Change runtime type\" -> \"Hardware accelerator\" -> \"GPU\".\n\n"
37-
]
38-
},
39-
{
40-
"cell_type": "markdown",
41-
"metadata": {},
42-
"source": [
43-
"## Download the data and install all required dependencies\n\nUncomment and run the following cell to download the tutorial data and\ninstall the required dependencies.\n\n"
44-
]
45-
},
46-
{
47-
"cell_type": "code",
48-
"execution_count": null,
49-
"metadata": {
50-
"collapsed": false
51-
},
52-
"outputs": [],
53-
"source": [
54-
"# !pip install -U --no-cache-dir gdown --pre\n# ![ -f \"vim-5-for-ccn.tar.gz\" ] || gdown --id 1b0I0Ytj06m6GCmfxfNrZuyF97fDo3NZb\n# ![ -d \"vim-5\" ] || tar xzf vim-5-for-ccn.tar.gz\n# !ln -s vim-5 shortclips\n# !apt-get install -qq inkscape > /dev/null\n# !pip install -q voxelwise_tutorials==0.1.3\n# ![ -f \"ngrok-stable-linux-amd64.zip\" ] || wget -q https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n# ![ -f \"ngrok\" ] || unzip ngrok-stable-linux-amd64.zip"
55-
]
56-
},
57-
{
58-
"cell_type": "markdown",
59-
"metadata": {},
60-
"source": [
61-
"For the record, here is what each command does:\n\n"
62-
]
63-
},
64-
{
65-
"cell_type": "code",
66-
"execution_count": null,
67-
"metadata": {
68-
"collapsed": false
69-
},
70-
"outputs": [],
71-
"source": [
72-
"# - update gdown to get the latest fixes\n# - Download the dataset archive\n# - Extract the dataset archive\n# - Create a symlink to the extracted dataset\n# - Install Inkscape, to use more features from Pycortex\n# - Install the tutorial helper package, and all the required dependencies\n# - Download ngrok to create a tunnel for pycortex 3D brain viewer\n# - Extract the ngrok archive"
73-
]
74-
},
75-
{
76-
"cell_type": "markdown",
77-
"metadata": {},
78-
"source": [
79-
"Now run the following cell to set up the environment variables for the\ntutorials and pycortex.\n\n"
80-
]
81-
},
82-
{
83-
"cell_type": "code",
84-
"execution_count": null,
85-
"metadata": {
86-
"collapsed": false
87-
},
88-
"outputs": [],
89-
"source": [
90-
"import os\nos.environ['VOXELWISE_TUTORIALS_DATA'] = \"/content\"\n\nimport sklearn\nsklearn.set_config(assume_finite=True)"
91-
]
92-
},
93-
{
94-
"cell_type": "markdown",
95-
"metadata": {},
96-
"source": [
97-
"Your Google Colab environment is now set up for the voxelwise tutorials.\n\n"
98-
]
99-
}
100-
],
101-
"metadata": {
102-
"kernelspec": {
103-
"display_name": "Python 3",
104-
"language": "python",
105-
"name": "python3"
106-
},
107-
"language_info": {
108-
"codemirror_mode": {
109-
"name": "ipython",
110-
"version": 3
111-
},
112-
"file_extension": ".py",
113-
"mimetype": "text/x-python",
114-
"name": "python",
115-
"nbconvert_exporter": "python",
116-
"pygments_lexer": "ipython3",
117-
"version": "3.9.0"
118-
}
2+
"cells": [
3+
{
4+
"cell_type": "code",
5+
"execution_count": null,
6+
"metadata": {},
7+
"outputs": [],
8+
"source": [
9+
"%matplotlib inline"
10+
]
11911
},
120-
"nbformat": 4,
121-
"nbformat_minor": 0
122-
}
12+
{
13+
"cell_type": "markdown",
14+
"metadata": {},
15+
"source": [
16+
"\n",
17+
"# Setup Google Colab\n",
18+
"\n",
19+
"In this script, we setup a Google Colab environment. This script will only work\n",
20+
"when run from `Google Colab <https://colab.research.google.com/>`_). You can\n",
21+
"skip it if you run the tutorials on your machine.\n"
22+
]
23+
},
24+
{
25+
"cell_type": "markdown",
26+
"metadata": {},
27+
"source": [
28+
"## Change runtime to use a GPU\n",
29+
"\n",
30+
"This tutorial is much faster when a GPU is available to run the computations.\n",
31+
"In Google Colab you can request access to a GPU by changing the runtime type.\n",
32+
"To do so, click the following menu options in Google Colab:\n",
33+
"\n",
34+
"(Menu) \"Runtime\" -> \"Change runtime type\" -> \"Hardware accelerator\" -> \"GPU\".\n",
35+
"\n"
36+
]
37+
},
38+
{
39+
"cell_type": "markdown",
40+
"metadata": {},
41+
"source": [
42+
"## Download the data and install all required dependencies\n",
43+
"\n",
44+
"Uncomment and run the following cell to download the required packages.\n",
45+
"\n"
46+
]
47+
},
48+
{
49+
"cell_type": "code",
50+
"execution_count": null,
51+
"metadata": {},
52+
"outputs": [],
53+
"source": [
54+
"# !git config --global user.email \"you@example.com\" && git config --global user.name \"Your Name\"\n",
55+
"# !wget -O- http://neuro.debian.net/lists/impish.us-ca.libre | sudo tee /etc/apt/sources.list.d/neurodebian.sources.list\n",
56+
"# !apt-key adv --recv-keys --keyserver hkps://keyserver.ubuntu.com 0xA5D32F012649A5A9 > /dev/null\n",
57+
"# !apt-get -qq update > /dev/null\n",
58+
"# !apt-get install -qq inkscape git-annex-standalone > /dev/null\n",
59+
"# !pip install -q voxelwise_tutorials \n",
60+
"# ![ -f \"ngrok-stable-linux-amd64.zip\" ] || wget -q https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
61+
"# ![ -f \"ngrok\" ] || unzip ngrok-stable-linux-amd64.zip"
62+
]
63+
},
64+
{
65+
"cell_type": "markdown",
66+
"metadata": {},
67+
"source": [
68+
"For the record, here is what each command does:\n",
69+
"\n"
70+
]
71+
},
72+
{
73+
"cell_type": "code",
74+
"execution_count": null,
75+
"metadata": {},
76+
"outputs": [],
77+
"source": [
78+
"# - Set up an email and user name to use git, git-annex, and datalad (required to download the data)\n",
79+
"# - Add NeuroDebian to the package sources\n",
80+
"# - Update the gpg keys to use NeuroDebian\n",
81+
"# - Update the list of available packages\n",
82+
"# - Install Inkscape to use more features from Pycortex, and install git-annex to download the data\n",
83+
"# - Install the tutorial helper package, and all the required dependencies\n",
84+
"# - Download ngrok to create a tunnel for pycortex 3D brain viewer\n",
85+
"# - Extract the ngrok archive"
86+
]
87+
},
88+
{
89+
"cell_type": "markdown",
90+
"metadata": {},
91+
"source": [
92+
"Now run the following cell to install the data for the tutorials."
93+
]
94+
},
95+
{
96+
"cell_type": "code",
97+
"execution_count": null,
98+
"metadata": {},
99+
"outputs": [],
100+
"source": [
101+
"colab_data_directory = \"/content/shortclips\"\n",
102+
"\n",
103+
"from voxelwise_tutorials.io import download_datalad\n",
104+
"\n",
105+
"DATAFILES = [\n",
106+
" \"features/motion_energy.hdf\",\n",
107+
" \"features/wordnet.hdf\",\n",
108+
" \"mappers/S01_mappers.hdf\",\n",
109+
" # \"mappers/S02_mappers.hdf\",\n",
110+
" # \"mappers/S03_mappers.hdf\",\n",
111+
" # \"mappers/S04_mappers.hdf\",\n",
112+
" # \"mappers/S05_mappers.hdf\",\n",
113+
" \"responses/S01_responses.hdf\",\n",
114+
" # \"responses/S02_responses.hdf\",\n",
115+
" # \"responses/S03_responses.hdf\",\n",
116+
" # \"responses/S04_responses.hdf\",\n",
117+
" # \"responses/S05_responses.hdf\",\n",
118+
" # \"stimuli/test.hdf\",\n",
119+
" # \"stimuli/train_00.hdf\",\n",
120+
" # \"stimuli/train_01.hdf\",\n",
121+
" # \"stimuli/train_02.hdf\",\n",
122+
" # \"stimuli/train_03.hdf\",\n",
123+
" # \"stimuli/train_04.hdf\",\n",
124+
" # \"stimuli/train_05.hdf\",\n",
125+
" # \"stimuli/train_06.hdf\",\n",
126+
" # \"stimuli/train_07.hdf\",\n",
127+
" # \"stimuli/train_08.hdf\",\n",
128+
" # \"stimuli/train_09.hdf\",\n",
129+
" # \"stimuli/train_10.hdf\",\n",
130+
" # \"stimuli/train_11.hdf\",\n",
131+
"]\n",
132+
"\n",
133+
"source = \"https://gin.g-node.org/gallantlab/shortclips\"\n",
134+
"\n",
135+
"for datafile in DATAFILES:\n",
136+
" local_filename = download_datalad(\n",
137+
" datafile, \n",
138+
" destination=colab_data_directory, \n",
139+
" source=source\n",
140+
" )"
141+
]
142+
},
143+
{
144+
"cell_type": "markdown",
145+
"metadata": {},
146+
"source": [
147+
"Now run the following cell to set up the environment variables for the\n",
148+
"tutorials and pycortex.\n",
149+
"\n"
150+
]
151+
},
152+
{
153+
"cell_type": "code",
154+
"execution_count": null,
155+
"metadata": {},
156+
"outputs": [],
157+
"source": [
158+
"import os\n",
159+
"os.environ['VOXELWISE_TUTORIALS_DATA'] = \"/content\"\n",
160+
"\n",
161+
"import sklearn\n",
162+
"sklearn.set_config(assume_finite=True)"
163+
]
164+
},
165+
{
166+
"cell_type": "markdown",
167+
"metadata": {},
168+
"source": [
169+
"Your Google Colab environment is now set up for the voxelwise tutorials.\n",
170+
"\n"
171+
]
172+
}
173+
],
174+
"metadata": {
175+
"kernelspec": {
176+
"display_name": "Python 3 (ipykernel)",
177+
"language": "python",
178+
"name": "python3"
179+
},
180+
"language_info": {
181+
"codemirror_mode": {
182+
"name": "ipython",
183+
"version": 3
184+
},
185+
"file_extension": ".py",
186+
"mimetype": "text/x-python",
187+
"name": "python",
188+
"nbconvert_exporter": "python",
189+
"pygments_lexer": "ipython3",
190+
"version": "3.9.12"
191+
}
192+
},
193+
"nbformat": 4,
194+
"nbformat_minor": 1
195+
}

0 commit comments

Comments
 (0)