ValueError: No performance file performance_all_0.json found: your network has failed.

[ZhaooooooYUE]

following information:

• OS: Ubuntu
• Python version: 3.7
• WORC version : 3.6.3
• pyradiomics version: 2.2.0

When I run the code, I receive the following error:

ValueError: No performance file performance_all_0.json found: your network has failed.

I have also noticed that the following jobs have failed or been cancelled:

-1-: calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0___MDD_0012_0 with status JobState.failed

-2-: featureconverter_train_pyradiomics_Pyradiomics_1_0_MRI_0___MDD_0012_0 with status JobState.cancelled

I am not sure what is causing these errors and would be grateful for any assistance you could provide. I have checked the documentation but have not been able to find a solution.

Would you be able to suggest any steps I can take to resolve this issue? I would be happy to provide any additional information that may be helpful.

Thank you for your time and consideration.

[ZhaooooooYUE]

[ZhaooooooYUE]

[MStarmans91]

Hi @ZhaooooooYUE ,

WORC makes use of fastr to execute all jobs in an experiment, which gives the reports you showed on which jobs have successfully executed and failed. You have to debug the failed experiment to figure out which specific jobs have failed for which reasons, e.g., the Python error of a specific script. In this case, your classification results has failed because your feature extraction jobs have failed, and we need to figure out the reason that happened. See this part of the WORC manual on debugging a failed network: https://worc.readthedocs.io/en/latest/static/user_manual.html#debugging. If you follow that, which means you end up with running a command similar to fastr trace "C:\Users\Martijn Starmans\Documents\GitHub\WORCTutorial\WORC_Example_STWStrategyHN\__sink_data__.json" --sinks features_train_CT_0_predict --samples HN1004 (see the link), you should be able to find out the exact errors why your experiment failed.

Hope that clears things up: feel free to share the exact error / output of the fastr trace here if you need help.

Kind regards,

Martijn

[ZhaooooooYUE]

Dear Martijn,

Thank you for taking the time to respond to my inquiry. Your reply has been very helpful in allowing me to better understand the issue.

Based on the information you provided regarding fastr trace, I inspected the code and found that it is able to generate "feature" files but not "performance" files.

I attempted to run the fastr trace command:

fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_0526/__sink_data__.json --sinks classification -v

However, the error message provided is not specific enough to determine the exact cause of the error.

I have attached a screenshot of the error message for your reference.

I would be very grateful if you could assist me once again in identifying the specific cause of the error.

Thank you again for your time and support.

Sincerely,
Yue Zhao

[MStarmans91]

You are almost there, you should trace back the error to a single sample in a sink, which corresponds to single job, to get exact errors. Hence in this case that would be

fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_0526/__sink_data__.json --sinks classification -samples all

From the looks of it however, it seems also some of the feature computations failed, since you get 23 failed samples in the features_train_MRI_0_pyradiomics sink. Hence could you also run such a fastr trace on a failed sample from that sink, something like

fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_0526/__sink_data__.json --sinks features_train_MRI_0_pyradiomics-samples [failed_sample_ID]

You can check which samples failed with

fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_0526/__sink_data__.json --samples

Some samples appear in multiple sinks, so you will see multiple failed jobs, but based on the IDs of your samples you should be able to figure out which are the ones that failed in the features sink.

If you could share the output of these, ideally in text instead of screenshots, we will see the error and I can help you further!

[Bin-Jing0908]

Dear Martijn,
I met a similar problem as the abovementioned one, when I used 'fastr trace' to track the error:
Input command: fastr trace /home/bjing/Desktop/MDD/WORC_1_MDD_05293/sink_data.json --sinks features_train_MRI_0_pyradiomics --sample HC_0012_0
I got the following output:
[WARNING] init:0084 >> Not running in a production installation (branch "unknown" from installed package)
Tracing errors for sample HC_0012_0 from sink features_train_MRI_0_pyradiomics
Located result pickle: /home/bjing/Desktop/MDD/WORC_1_MDD_05293/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0012_0/fastr_result.yaml

===== JOB WORC_MDD_05293___calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0___HC_0012_0 =====
Network: WORC_MDD_05293
Run: WORC_MDD_05293_2024-05-29T09-00-55
Node: calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0
Sample index: (11)
Sample id: HC_0012_0
Status: JobState.execution_failed
Timestamp: 2024-05-29 01:01:04.172060
Job file: /home/bjing/Desktop/MDD/WORC_1_MDD_05293/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0012_0/fastr_result.yaml

----- ERRORS -----

• FastrSubprocessNotFinished: There is no information that the subprocess finished properly: appears the job crashed before the subprocess registered as finished. (/home/bjing/PycharmProjects/PY37/lib/python3.7/site-packages/fastr/execution/executionscript.py:117)
• FastrExecutableNotFoundError: (/home/bjing/PycharmProjects/PY37/lib/python3.7/site-packages/fastr/core/target.py:194)

---

No process information:
Cannot find process information in Job information, processing probably got killed.
If there are no other errors, this is often a result of too high memory use or
exceeding some other type of resource limit.

Output data:
{}

Status history:
2024-05-29 01:01:04.172074: JobState.created
2024-05-29 01:01:04.191316: JobState.hold
2024-05-29 01:02:24.434559: JobState.queued
2024-05-29 01:04:34.722684: JobState.running
2024-05-29 01:04:37.399254: JobState.execution_failed

It seemed that Pyradiomics didn't calculate the features, but the provided information are not useful to solve it. I also checked that all subjects didn't have the calculated features from Pyradimics, but succedd in PREDICT.
Operating System: Ubantu 20.04
WORC: 3.6
Python: 3.7
Is there any method to detailedly track the error? Thanks.

[ZhaooooooYUE]

Dear Martijn,

Thank you for your prompt response, which has helped me to better troubleshoot the current issue.

Following your guidance, I have gathered the following information.

I have attached the relevant code and error logs to this email for your reference:

(venv) (base) zhaoyue@zhaoyue-virtual-machine:/PycharmProjects/pythonProject$ fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_0526/sink_data.json --sinks
[WARNING] init:0084 >> Not running in a production installation (branch "unknown" from installed package)
Barchart_PNG -- 1 failed -- 0 succeeded
Barchart_Tex -- 1 failed -- 0 succeeded
BoxplotsFeatures_Zip -- 1 failed -- 0 succeeded
Decomposition_PNG -- 1 failed -- 0 succeeded
Hyperparameters_CSV -- 1 failed -- 0 succeeded
PRC_CSV -- 1 failed -- 0 succeeded
PRC_PNG -- 1 failed -- 0 succeeded
PRC_Tex -- 1 failed -- 0 succeeded
ROC_CSV -- 1 failed -- 0 succeeded
ROC_PNG -- 1 failed -- 0 succeeded
ROC_Tex -- 1 failed -- 0 succeeded
RankedPercentages_CSV -- 1 failed -- 0 succeeded
RankedPercentages_Zip -- 1 failed -- 0 succeeded
RankedPosteriors_CSV -- 1 failed -- 0 succeeded
RankedPosteriors_Zip -- 1 failed -- 0 succeeded
StatisticalTestFeatures_CSV -- 1 failed -- 0 succeeded
StatisticalTestFeatures_PNG -- 1 failed -- 0 succeeded
StatisticalTestFeatures_Tex -- 1 failed -- 0 succeeded
classification -- 1 failed -- 0 succeeded
config_MRI_0_sink -- 0 failed -- 1 succeeded
config_classification_sink -- 0 failed -- 1 succeeded
features_train_MRI_0_pyradiomics -- 23 failed -- 1 succeeded
performance -- 1 failed -- 0 succeeded
(venv) (base) zhaoyue@zhaoyue-virtual-machine:/PycharmProjects/pythonProject$ fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_0526/sink_data.json --sinks features_train_MRI_0_pyradiomics
[WARNING] init:0084 >> Not running in a production installation (branch "unknown" from installed package)
features_train_MRI_0_pyradiomics -- 23 failed -- 1 succeeded
(venv) (base) zhaoyue@zhaoyue-virtual-machine:/PycharmProjects/pythonProject$ fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_0526/sink_data.json --samples
[WARNING] init:0084 >> Not running in a production installation (branch "unknown" from installed package)
HC_0001_0 -- 1 failed -- 0 succeeded
HC_0002_0 -- 1 failed -- 0 succeeded
HC_0003_0 -- 1 failed -- 0 succeeded
HC_0004_0 -- 1 failed -- 0 succeeded
HC_0005_0 -- 1 failed -- 0 succeeded
HC_0006_0 -- 1 failed -- 0 succeeded
HC_0007_0 -- 1 failed -- 0 succeeded
HC_0008_0 -- 1 failed -- 0 succeeded
HC_0009_0 -- 1 failed -- 0 succeeded
HC_0010_0 -- 1 failed -- 0 succeeded
HC_0011_0 -- 1 failed -- 0 succeeded
HC_0012_0 -- 1 failed -- 0 succeeded
MDD_0001_0 -- 1 failed -- 0 succeeded
MDD_0002_0 -- 0 failed -- 1 succeeded
MDD_0003_0 -- 1 failed -- 0 succeeded
MDD_0004_0 -- 1 failed -- 0 succeeded
MDD_0005_0 -- 1 failed -- 0 succeeded
MDD_0006_0 -- 1 failed -- 0 succeeded
MDD_0007_0 -- 1 failed -- 0 succeeded
MDD_0008_0 -- 1 failed -- 0 succeeded
MDD_0009_0 -- 1 failed -- 0 succeeded
MDD_0010_0 -- 1 failed -- 0 succeeded
MDD_0011_0 -- 1 failed -- 0 succeeded
MDD_0012_0 -- 1 failed -- 0 succeeded
all -- 20 failed -- 0 succeeded
id_0 -- 0 failed -- 2 succeeded
(venv) (base) zhaoyue@zhaoyue-virtual-machine:/PycharmProjects/pythonProject$ fastr trace /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/sink_data.json --sinks features_train_MRI_0_pyradiomics --sample HC_0001_0
[WARNING] init:0084 >> Not running in a production installation (branch "unknown" from installed package)
Tracing errors for sample HC_0001_0 from sink features_train_MRI_0_pyradiomics
Located result pickle: /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0001_0/fastr_result.yaml

===== JOB WORC_MDD_05266___calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0___HC_0001_0 =====
Network: WORC_MDD_05266
Run: WORC_MDD_05266_2024-05-29T15-53-02
Node: calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0
Sample index: (0)
Sample id: HC_0001_0
Status: JobState.execution_failed
Timestamp: 2024-05-29 07:53:23.532734
Job file: /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0001_0/fastr_result.yaml

----- ERRORS -----

FastrOutputValidationError: Output value [PyRadiomicsCSVFile] "vfs://home/MDD_HC_10/WORC_1_MDD_05266/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0001_0/features_0.csv" not valid for datatype "'PyRadiomicsCSVFile'" (/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/fastr/execution/job.py:1155)
FastrOutputValidationError: The output "features" is invalid! (/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/fastr/execution/job.py:1103)
FastrErrorInSubprocess: [2024-05-29 15:59:30] W: radiomics.featureextractor: Fixed bin Count enabled! However, we recommend using a fixed bin Width. See http://pyradiomics.readthedocs.io/en/latest/faq.html#radiomics-fixed-bin-width for more details
[2024-05-29 15:59:31] E: radiomics.script: Feature extraction failed!
Traceback (most recent call last):
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/scripts/segment.py", line 68, in _extractFeatures
feature_vector.update(extractor.execute(imageFilepath, maskFilepath, label, label_channel))
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/featureextractor.py", line 325, in execute
featureVector.update(self.computeFeatures(inputImage, inputMask, imageTypeName, **inputKwargs))
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/featureextractor.py", line 501, in computeFeatures
for (featureName, featureValue) in six.iteritems(featureClass.execute()):
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/base.py", line 185, in execute
self._calculateSegment()
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/base.py", line 227, in _calculateSegment
for success, featureName, featureValue in self._calculateFeatures():
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/base.py", line 234, in _calculateFeatures
self._initCalculation(voxelCoordinates)
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/gldm.py", line 75, in _initCalculation
self.P_gldm = self._calculateMatrix(voxelCoordinates)
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/gldm.py", line 103, in _calculateMatrix
P_gldm = numpy.delete(P_gldm, emptyGrayLevels - 1, 1)
File "<array_function internals>", line 6, in delete
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/numpy/lib/function_base.py", line 4552, in delete
keep[obj,] = False
IndexError: arrays used as indices must be of integer (or boolean) type
(/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/fastr/execution/executionscript.py:111)
FastrValueError: Output values are not valid! (/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/fastr/execution/job.py:834)
Command:
List representation: ['pyradiomics', '/home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/preprocessing_train_MRI_0/HC_0001_0/image_0.nii.gz', '/home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/convert_seg_train_MRI_0/HC_0001_0/image_0.nii.gz', '--param', '/home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/config_pyradiomics_MRI_0/id_0/result/config_pyradiomics_WORC_MDD_05266_0.yaml', '--format', 'csv', '--out', '/home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0001_0/features_0.csv']

String representation: pyradiomics /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/preprocessing_train_MRI_0/HC_0001_0/image_0.nii.gz /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/convert_seg_train_MRI_0/HC_0001_0/image_0.nii.gz --param /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/config_pyradiomics_MRI_0/id_0/result/config_pyradiomics_WORC_MDD_05266_0.yaml --format csv --out /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0001_0/features_0.csv

Output data:
{'features': [<PyRadiomicsCSVFile: 'vfs://home/MDD_HC_10/WORC_1_MDD_05266/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0001_0/features_0.csv'>]}

Status history:
2024-05-29 07:53:23.532753: JobState.created
2024-05-29 07:53:23.556486: JobState.hold
2024-05-29 07:56:47.280691: JobState.queued
2024-05-29 07:59:29.366004: JobState.running
2024-05-29 07:59:31.528699: JobState.execution_failed

----- STDOUT -----

----- STDERR -----
[2024-05-29 15:59:30] W: radiomics.featureextractor: Fixed bin Count enabled! However, we recommend using a fixed bin Width. See http://pyradiomics.readthedocs.io/en/latest/faq.html#radiomics-fixed-bin-width for more details
[2024-05-29 15:59:31] E: radiomics.script: Feature extraction failed!
Traceback (most recent call last):
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/scripts/segment.py", line 68, in _extractFeatures
feature_vector.update(extractor.execute(imageFilepath, maskFilepath, label, label_channel))
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/featureextractor.py", line 325, in execute
featureVector.update(self.computeFeatures(inputImage, inputMask, imageTypeName, **inputKwargs))
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/featureextractor.py", line 501, in computeFeatures
for (featureName, featureValue) in six.iteritems(featureClass.execute()):
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/base.py", line 185, in execute
self._calculateSegment()
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/base.py", line 227, in _calculateSegment
for success, featureName, featureValue in self._calculateFeatures():
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/base.py", line 234, in _calculateFeatures
self._initCalculation(voxelCoordinates)
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/gldm.py", line 75, in _initCalculation
self.P_gldm = self._calculateMatrix(voxelCoordinates)
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/radiomics/gldm.py", line 103, in _calculateMatrix
P_gldm = numpy.delete(P_gldm, emptyGrayLevels - 1, 1)
File "<array_function internals>", line 6, in delete
File "/home/zhaoyue/PycharmProjects/pythonProject/venv/lib/python3.7/site-packages/numpy/lib/function_base.py", line 4552, in delete
keep[obj,] = False
IndexError: arrays used as indices must be of integer (or boolean) type

Your help would be greatly appreciated. Please let me know if you require any additional information.

Sincerely,
Yue Zhao

[MStarmans91]

Great, now we see the actual error in the PyRadiomics Python job. For your info, the command you see in this report is exactly what was being executed on the command line, hence you can run that to replicate the error, or debugging is you change something.

The error you got very much reminds me of a previous error in PyRadiomics 3.0.0, see AIM-Harvard/pyradiomics#592 and my fix in https://worc.readthedocs.io/en/latest/static/faq.html#error-file-lib-site-packages-numpy-lib-function-base-py-line-4406-in-delete-keep-obj-false-indexerror-arrays-used-as-indices-must-be-of-integer-or-boolean-type. That issue was fixed in PyRadiomics version 3.0.1, which I personally would highly recommend over both 3.0.0 and 3.1.0 which is also sometimes buggy.

So I would recommend: check the PyRadiomics version to see if that could be the issue, if it's not 3.0.1. go to that version, and rerun the calling command of this job instead of the whole WORC experiment to see if that helped:

pyradiomics /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/preprocessing_train_MRI_0/HC_0001_0/image_0.nii.gz /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/convert_seg_train_MRI_0/HC_0001_0/image_0.nii.gz --param /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/config_pyradiomics_MRI_0/id_0/result/config_pyradiomics_WORC_MDD_05266_0.yaml --format csv --out /home/zhaoyue/MDD_HC_10/WORC_1_MDD_05266/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0001_0/features_0.csv

@Jing-Bin-Lab your issue seems slightly different as you see from the fastr traces. Fully agreed that the provided information is not descriptive enough to solve it, I've send a message to the fastr developer to ask if he can help and improve these kind of messages. So it's also a bit guessing for me. The FastrExecutableNotFoundError seems to hint that maybe pyradiomics cannot be called as an executable: can you check that? You could see if that specific job has actually run, and copy the calling command to execute it on the command line to hopefully get a more descriptive error. Unfortunately for you it's not in the fastr trace you did, but should be in the file /home/bjing/Desktop/MDD/WORC_1_MDD_05293/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0012_0/fastr_stdout.txt. If that's not it: when running the experiment, do you see earlier jobs failing, e.g., images, segmentations, segmentix, that may cause this job to have been cancelled?