You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"\n# Speeding up PARAFAC2 with SVD compression\n\nPARAFAC2 can be very time-consuming to fit. However, if the number of rows greatly\nexceeds the number of columns or the data matrices are approximately low-rank, we can\ncompress the data before fitting the PARAFAC2 model to considerably speed up the fitting\nprocedure.\n\nThe compression works by first computing the SVD of the tensor slices and fitting the\nPARAFAC2 model to the right singular vectors multiplied by the singular values. Then,\nafter we fit the model, we left-multiply the $B_i$-matrices with the left singular\nvectors to recover the decompressed model. Fitting to compressed data and then\ndecompressing is mathematically equivalent to fitting to the original uncompressed data.\n\nFor more information about why this works, see the documentation of\n:py:meth:`tensorly.decomposition.preprocessing.svd_compress_tensor_slices`.\n"
8
+
]
9
+
},
10
+
{
11
+
"cell_type": "code",
12
+
"execution_count": null,
13
+
"metadata": {
14
+
"collapsed": false
15
+
},
16
+
"outputs": [],
17
+
"source": [
18
+
"from time import monotonic\nimport tensorly as tl\nfrom tensorly.decomposition import parafac2\nimport tensorly.preprocessing as preprocessing"
19
+
]
20
+
},
21
+
{
22
+
"cell_type": "markdown",
23
+
"metadata": {},
24
+
"source": [
25
+
"## Function to create synthetic data\n\nHere, we create a function that constructs a random tensor from a PARAFAC2\ndecomposition with noise\n\n"
26
+
]
27
+
},
28
+
{
29
+
"cell_type": "code",
30
+
"execution_count": null,
31
+
"metadata": {
32
+
"collapsed": false
33
+
},
34
+
"outputs": [],
35
+
"source": [
36
+
"rng = tl.check_random_state(0)\n\n\ndef create_random_data(shape, rank, noise_level):\n I, J, K = shape # noqa: E741\n pf2 = tl.random.random_parafac2(\n [(J, K) for i in range(I)], rank=rank, random_state=rng\n )\n\n X = pf2.to_tensor()\n X_norm = [tl.norm(Xi) for Xi in X]\n\n noise = [rng.standard_normal((J, K)) for i in range(I)]\n noise = [noise_level * X_norm[i] / tl.norm(E_i) for i, E_i in enumerate(noise)]\n return [X_i + E_i for X_i, E_i in zip(X, noise)]"
37
+
]
38
+
},
39
+
{
40
+
"cell_type": "markdown",
41
+
"metadata": {},
42
+
"source": [
43
+
"## Compressing data with many rows and few columns\n\nHere, we set up for a case where we have many rows compared to columns\n\n"
44
+
]
45
+
},
46
+
{
47
+
"cell_type": "code",
48
+
"execution_count": null,
49
+
"metadata": {
50
+
"collapsed": false
51
+
},
52
+
"outputs": [],
53
+
"source": [
54
+
"n_inits = 5\nrank = 3\nshape = (10, 10_000, 15) # 10 matrices/tensor slices, each of size 10_000 x 15.\nnoise_level = 0.33\n\nuncompressed_data = create_random_data(shape, rank=rank, noise_level=noise_level)"
55
+
]
56
+
},
57
+
{
58
+
"cell_type": "markdown",
59
+
"metadata": {},
60
+
"source": [
61
+
"### Fitting without compression\n\nAs a baseline, we see how long time it takes to fit models without compression.\nSince PARAFAC2 is very prone to local minima, we fit five models and select the model\nwith the lowest reconstruction error.\n\n"
62
+
]
63
+
},
64
+
{
65
+
"cell_type": "code",
66
+
"execution_count": null,
67
+
"metadata": {
68
+
"collapsed": false
69
+
},
70
+
"outputs": [],
71
+
"source": [
72
+
"print(\"Fitting PARAFAC2 model without compression...\")\nt1 = monotonic()\nlowest_error = float(\"inf\")\nfor i in range(n_inits):\n pf2, errs = parafac2(\n uncompressed_data,\n rank,\n n_iter_max=1000,\n nn_modes=[0],\n random_state=rng,\n return_errors=True,\n )\n if errs[-1] < lowest_error:\n pf2_full, errs_full = pf2, errs\nt2 = monotonic()\nprint(\n f\"It took {t2 - t1:.1f}s to fit a PARAFAC2 model a tensor of shape {shape} \"\n + \"without compression\"\n)"
73
+
]
74
+
},
75
+
{
76
+
"cell_type": "markdown",
77
+
"metadata": {},
78
+
"source": [
79
+
"### Fitting with lossless compression\n\nSince the tensor slices have many rows compared to columns, we should be able to save\na lot of time by compressing the data. By compressing the matrices, we only need to\nfit the PARAFAC2 model to a set of 10 matrices, each of size 15 x 15, not 10_000 x 15.\n\nThe main bottleneck here is the SVD computation at the beginning of the fitting\nprocedure, but luckily, this is independent of the initialisations, so we only need\nto compute this once. Also, if we are performing a grid search for the rank, then\nwe just need to perform the compression once for the whole grid search as well.\n\n"
80
+
]
81
+
},
82
+
{
83
+
"cell_type": "code",
84
+
"execution_count": null,
85
+
"metadata": {
86
+
"collapsed": false
87
+
},
88
+
"outputs": [],
89
+
"source": [
90
+
"print(\"Fitting PARAFAC2 model with SVD compression...\")\nt1 = monotonic()\nlowest_error = float(\"inf\")\nscores, loadings = preprocessing.svd_compress_tensor_slices(uncompressed_data)\nt2 = monotonic()\nfor i in range(n_inits):\n pf2, errs = parafac2(\n scores,\n rank,\n n_iter_max=1000,\n nn_modes=[0],\n random_state=rng,\n return_errors=True,\n )\n if errs[-1] < lowest_error:\n pf2_compressed, errs_compressed = pf2, errs\npf2_decompressed = preprocessing.svd_decompress_parafac2_tensor(\n pf2_compressed, loadings\n)\nt3 = monotonic()\nprint(\n f\"It took {t3 - t1:.1f}s to fit a PARAFAC2 model a tensor of shape {shape} \"\n + \"with lossless SVD compression\"\n)\nprint(f\"The compression took {t2 - t1:.1f}s and the fitting took {t3 - t2:.1f}s\")"
91
+
]
92
+
},
93
+
{
94
+
"cell_type": "markdown",
95
+
"metadata": {},
96
+
"source": [
97
+
"We see that we saved a lot of time by compressing the data before fitting the model.\n\n"
98
+
]
99
+
},
100
+
{
101
+
"cell_type": "markdown",
102
+
"metadata": {},
103
+
"source": [
104
+
"### Fitting with lossy compression\n\nWe can try to speed the process up even further by accepting a slight discrepancy\nbetween the model obtained from compressed data and a model obtained from uncompressed\ndata. Specifically, we can truncate the singular values at some threshold, essentially\nremoving the parts of the data matrices that have a very low \"signal strength\".\n\n"
105
+
]
106
+
},
107
+
{
108
+
"cell_type": "code",
109
+
"execution_count": null,
110
+
"metadata": {
111
+
"collapsed": false
112
+
},
113
+
"outputs": [],
114
+
"source": [
115
+
"print(\"Fitting PARAFAC2 model with lossy SVD compression...\")\nt1 = monotonic()\nlowest_error = float(\"inf\")\nscores, loadings = preprocessing.svd_compress_tensor_slices(uncompressed_data, 1e-5)\nt2 = monotonic()\nfor i in range(n_inits):\n pf2, errs = parafac2(\n scores,\n rank,\n n_iter_max=1000,\n nn_modes=[0],\n random_state=rng,\n return_errors=True,\n )\n if errs[-1] < lowest_error:\n pf2_compressed_lossy, errs_compressed_lossy = pf2, errs\npf2_decompressed_lossy = preprocessing.svd_decompress_parafac2_tensor(\n pf2_compressed_lossy, loadings\n)\nt3 = monotonic()\nprint(\n f\"It took {t3 - t1:.1f}s to fit a PARAFAC2 model a tensor of shape {shape} \"\n + \"with lossy SVD compression\"\n)\nprint(\n f\"Of which the compression took {t2 - t1:.1f}s and the fitting took {t3 - t2:.1f}s\"\n)"
116
+
]
117
+
},
118
+
{
119
+
"cell_type": "markdown",
120
+
"metadata": {},
121
+
"source": [
122
+
"We see that we didn't save much, if any, time in this case (compared to using\nlossless compression). This is because the main bottleneck now is the CP-part of\nthe PARAFAC2 procedure, so reducing the tensor size from 10 x 15 x 15 to 10 x 4 x 15\n(which is typically what we would get here) will have a negligible effect.\n\n"
123
+
]
124
+
},
125
+
{
126
+
"cell_type": "markdown",
127
+
"metadata": {},
128
+
"source": [
129
+
"## Compressing data that is approximately low-rank\n\nHere, we simulate data with many rows and columns but an approximately low rank.\n\n"
"### Fitting without compression\n\nAgain, we start by fitting without compression as a baseline.\n\n"
148
+
]
149
+
},
150
+
{
151
+
"cell_type": "code",
152
+
"execution_count": null,
153
+
"metadata": {
154
+
"collapsed": false
155
+
},
156
+
"outputs": [],
157
+
"source": [
158
+
"print(\"Fitting PARAFAC2 model without compression...\")\nt1 = monotonic()\nlowest_error = float(\"inf\")\nfor i in range(n_inits):\n pf2, errs = parafac2(\n uncompressed_data,\n rank,\n n_iter_max=1000,\n nn_modes=[0],\n random_state=rng,\n return_errors=True,\n )\n if errs[-1] < lowest_error:\n pf2_full, errs_full = pf2, errs\nt2 = monotonic()\nprint(\n f\"It took {t2 - t1:.1f}s to fit a PARAFAC2 model a tensor of shape {shape} \"\n + \"without compression\"\n)"
159
+
]
160
+
},
161
+
{
162
+
"cell_type": "markdown",
163
+
"metadata": {},
164
+
"source": [
165
+
"### Fitting with lossless compression\n\nNext, we fit with lossless compression.\n\n"
166
+
]
167
+
},
168
+
{
169
+
"cell_type": "code",
170
+
"execution_count": null,
171
+
"metadata": {
172
+
"collapsed": false
173
+
},
174
+
"outputs": [],
175
+
"source": [
176
+
"print(\"Fitting PARAFAC2 model with SVD compression...\")\nt1 = monotonic()\nlowest_error = float(\"inf\")\nscores, loadings = preprocessing.svd_compress_tensor_slices(uncompressed_data)\nt2 = monotonic()\nfor i in range(n_inits):\n pf2, errs = parafac2(\n scores,\n rank,\n n_iter_max=1000,\n nn_modes=[0],\n random_state=rng,\n return_errors=True,\n )\n if errs[-1] < lowest_error:\n pf2_compressed, errs_compressed = pf2, errs\npf2_decompressed = preprocessing.svd_decompress_parafac2_tensor(\n pf2_compressed, loadings\n)\nt3 = monotonic()\nprint(\n f\"It took {t3 - t1:.1f}s to fit a PARAFAC2 model a tensor of shape {shape} \"\n + \"with lossless SVD compression\"\n)\nprint(\n f\"Of which the compression took {t2 - t1:.1f}s and the fitting took {t3 - t2:.1f}s\"\n)"
177
+
]
178
+
},
179
+
{
180
+
"cell_type": "markdown",
181
+
"metadata": {},
182
+
"source": [
183
+
"We see that the lossless compression no effect for this data. This is because the\nnumber ofrows is equal to the number of columns, so we cannot compress the data\nlosslessly with the SVD.\n\n"
184
+
]
185
+
},
186
+
{
187
+
"cell_type": "markdown",
188
+
"metadata": {},
189
+
"source": [
190
+
"### Fitting with lossy compression\n\nFinally, we fit with lossy SVD compression.\n\n"
191
+
]
192
+
},
193
+
{
194
+
"cell_type": "code",
195
+
"execution_count": null,
196
+
"metadata": {
197
+
"collapsed": false
198
+
},
199
+
"outputs": [],
200
+
"source": [
201
+
"print(\"Fitting PARAFAC2 model with lossy SVD compression...\")\nt1 = monotonic()\nlowest_error = float(\"inf\")\nscores, loadings = preprocessing.svd_compress_tensor_slices(uncompressed_data, 1e-5)\nt2 = monotonic()\nfor i in range(n_inits):\n pf2, errs = parafac2(\n scores,\n rank,\n n_iter_max=1000,\n nn_modes=[0],\n random_state=rng,\n return_errors=True,\n )\n if errs[-1] < lowest_error:\n pf2_compressed_lossy, errs_compressed_lossy = pf2, errs\npf2_decompressed_lossy = preprocessing.svd_decompress_parafac2_tensor(\n pf2_compressed_lossy, loadings\n)\nt3 = monotonic()\nprint(\n f\"It took {t3 - t1:.1f}s to fit a PARAFAC2 model a tensor of shape {shape} \"\n + \"with lossy SVD compression\"\n)\nprint(\n f\"Of which the compression took {t2 - t1:.1f}s and the fitting took {t3 - t2:.1f}s\"\n)"
202
+
]
203
+
},
204
+
{
205
+
"cell_type": "markdown",
206
+
"metadata": {},
207
+
"source": [
208
+
"Here we see a large speedup. This is because the data is approximately low rank so\nthe compressed tensor slices will have shape R x 2_000, where R is typically below 10\nin this example. If your tensor slices are large in both modes, you might want to plot\nthe singular values of your dataset to see if lossy compression could speed up\nPARAFAC2.\n\n"
# The same regularization coefficient l1_reg is used for all the modes. Here the l1 penalization induces sparsity given that the regularization coefficient is large enough.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4&size=40){kind=avatar}
0 commit comments