Spectra in CSV and TXT (and SPC) format?

[furutaka]

Hi,

CSV and TXT as histogram formats are rather vague.

How should I organize values in these formats to feed to SpecUtils/Cambio?
(Did I overlook their description in the documents?)

Thanks in advance.
Kazuyoshi

[furutaka]

Hi,

I found the explanation of these formats when "Save" tab of cambio was selected.

By the way, the attached flat.Spc file is a test file I made in SPC format; there're 256 bins all have the same value, 1.e+9 (if my understanding of the format is correct).
flat.Spc.gz

When fed to cambio, the contents of the first and the last bins seem to be lost;

Indeed, the values of the first and last bins in the file saved in CSV format (flat.csv) vanish.
flat.csv

After changing the first/last values to the same ones in other bins and then fed again to cambio, the result
was the following...

Conversion to TXT format yields the same values for all bins.

But when the same data was saved in SPC format and the resultant file was fed again to cambio, the contents of the first and last bin vanish (flatMod_1.csv)...
flatMod_1.spc.gz
flatMod_1.csv

Weird like a Candidate...

In what did I make mistakes? Original SPC-format file? Interpretation of CSV-format file?
Any comments are welcome.

Thanks in advance...
Kazuyoshi

[wcjohns]

Hello Kazuyoshi,

The Measurement::set_info_from_txt_or_csv(...) function is what reads spectra from a CSV or TXT file.

It's not a pretty function, so may not be very useful to look at.
The function accepts a number of formats and information fields, primarily for compatibility from other software.
But the basic format is similar to

Measurement start: 20241121T09:32:14
LiveTime: 60.2s
RealTime: 65 seconds
calibcoeff: a=0, b =0, c=3, d=0
Title: First Measurement
Remark: Some Remark
Channel, Counts
1, 0
2, 3
3, 9
4, 10
...

# A few blank lines to signal a new spectrum
LiveTime: 59.2s
RealTime: 60 s
Title: Second Measurement
Energy, Counts
0, 0
3, 3
6, 9
9, 10
...

Where in this example there were two spectra in the file; the first one used a polynomial energy calibration (where order of coefficients are reverse their normal meaning), while the second one instead specified the lower energy of each channel.

The function tries to be reasonably flexible on input it allows in. So you could provide both Energy and Channel column, or if you could just provide Channel with no polynomial (in which case a default range of 3 MeV will be used).

I found the explanation of these formats when "Save" tab of cambio was selected.
In InterSpec v1.0.12, when you export a spectrum file, there is also a little question mark next to each format that you can hover over, and may contain slightly better descriptions than cambio:

SPC Issues

You are correct in your interpretation of the SPC format.
It looks like the first and last channels are explicitly set to zero, here.
If I recall right, this is because one source of SPC files put garbage values in these channels.
Do you think it would be okay if instead of always setting to zero, we check that they are not Inf/NaN, are positive, and not the highest count channel in the spectrum? But if any of these tests fail, then set to 0?

And just to make sure, do you think there is an issue with the CSV/TXT input/output? Or are the issues there likely related to the SPC issue?

Thank you for reporting this - and I am sorry this issue has cost you so much investigation.
-will

[furutaka]

Dear Dr. Johnson,

Thanks for the info on spectral formats! Indeed, at first I converted a spectrum to SPC format and then imported to InterSpec, and found that (at first I thought it was just shifted due to some offset issues of my conversion script from ROOT but in reality) the first channel vanished . Then I fed the SPC spectrum to Cambio and found the same behavior and suspect it's a SpecUtils issue (but never tried to export from InterSpec)...

Until now I didn't see any flaws in converting to CSV/TXT format.

As for the SPC spectrum with strange values such as Inf/NaN...

• Considering the use of FP in SPC (as a FP version of SPC spectrum), it may be a normalized and/or subtracted (= net) spectrum and is possible to have negative values,
• (the lowest part of spectra obtained in real measurements tend to be cut off by noise discrimination but) spectra obtained in some simulation calculation frequently has the highest value at the lowest channels.

Considering the above, for a spectrum-reading routine, I think it's better just to stop processing it if some of the values are Inf/NaN, but otherwise leave them as they are. (Isn't it better to tell the user that a part of the spectral data is invalid instead of quietly converting it to a proper value?)

Your thoughts?

Thanks,
Kazuyoshi