read_sav(): UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 49: invalid start byte

[zenelba]

Hi!

Have an issue with reading an SPSS .sav file.

I have already deleted all the "string" variables but the problem persists :(

Do you have any idea how to solve this?

This is the first time I encounter such an issue in more than 5 years of using this library.

z

[MDS-JAnthony]

Hello,
This could be caused by number of different things.
Are you able to open your file in SPSS and check what encoding has been used to save the file?

[ofajardo]

that's a good idea! once you know the encoding, you can pass it to read_sav:

df, meta = pyreadstat.read_sav('/path/to/a/file.sav', encoding="LATIN1")

If you do not know the encoding I am afraid you will have to iterate through a list of encodings to see if one of them works

[zenelba]

Good idea. SPSS says it is windows-1250.

I have used: df, metadata = pyreadstat.read_sav(f,encoding="CP1250")

and still get:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 49: invalid start byte

Meanwhile I used this in SPSS:
SET UNICODE = ON.

and saved SAV file one more time. In this case it works.

[zenelba]

I have a feeling merging in SPSS does something weird to encodings.

BTW I can share the SAV file, maybe someone can find how to deal with it in python.

[ofajardo]

you can share the file if you wish, to take a look

[ofajardo]

Thanks for asking! Moved this to discussions. To create an issue there has to be a way to reproduce the issue and all the information in the template has to be provided.

[zenelba]

The SAV was produced by my colleagues. after that they were not able to "reproduce" it.
I will try to get the "non working" one and share it with you. It is a sensitive topic so I can not share it publically.

[ofajardo]

Please do not share sensitive data publically or orivately with me. You would need to replicate the issue with dummy data

[ofajardo]

BTW, what is the full stacktrace? Is by chance handle_note mentioned in the error? A reproducible bug was just reported, apparently Readstat is not correctly translating characters to utf-8 for notes ... (see here)

[zenelba]

Interesting... notes, do they exist in SPSS?

[ofajardo]

They do, yes, but maybe it is something else not translated to utf-8. If the file is very old, it has also happened before that it was using annencoding not supoorted by iconv.