Foreign characters and XML entries

@rpedde wrote:

We talked about this before right? This was a ntfs drive that got moved to a unix box and is now accessed via samba? Is that right?

Yes, we discussed this in another thread which disappeared after the recent change to the forum structure/pages. It’s a ntfs that got moved to a NSLU2 unlsung with the latest 6.8 and support for Western Europe/Latin 1(850).

@rpedde wrote:

The files are stored as utf-16 on disk, but iTunes is obviously storing the file names as utf-8. So up-promote utf-8 to utf-16? This might take some work.

Well, I tried to change one entry in the XML file where iTunes had Bj%C3%B6rk (C3 B6) is the UTF-8 code for the letter ö, opened the file with an xml editor and changed %C3%B6 to 00F6 – the code for the same character in UTF-16, but mt-daapd still couldn’t find it.

Also, when I Telnet into the slug I can access the folder named Björk by just typing cd Björk. So the Slug certainly understands ö.

@rpedde wrote:

I think I have some bjork around, I’ll see if I can’t replicate it. I might not be able to — at least not on windows. I’m pretty sure I was playing with some Bjork music when I was working through playlist stuff, and it worked okay locally, so it’s probably a conversion issue on samba.

If you can tell me how to edit the xml file so that mt-daapd can process the path info for something like …/Björk/…, which iTunes writes as ../Bj%C3%B6rk/…, I can do it for the other cases myself, even if that means having to change the xml entries manually.

Cheers

@schiers wrote:

Hi,

we can be lucky that Björk Guðmundsdóttir uses only her first name, don’t we? 8)

BR,
Carsten.

Good point mate 😀

But if you think the set of characters in her last name is bad, check out her early jazz album “Gling-Gló” and take a look at the name of the tracks… 😯

Ok, as Ron suggested I tried to replace Bj%C3%B6rk with Bj%94rk in the file names and this time it worked.

So since I couldn’t find a conversion table online (even though there must be one somewhere), I went to Mathematica and wrote a small conversion tool from UTF8 to Code Page CP850 for Western European languages, which consists of 128 additional “special” characters. Here is the result for the ones which are likely to appear in a file name.

The table format is as follows: A %BB%CC %DD

“A” is the special character
“%BB%CC” is the way iTunes writes it in the XML file
“%DD” is what it should be in the XML file instead of “%BB%CC”

UTF-8 CP850
Ç %C3%87 %80

ü %C3%BC %81

é %C3%A9 %82

â %C3%A2 %83

ä %C3%A4 %84

à %C3%A0 %85

å %C3%A5 %86

ç %C3%A7 %87

ê %C3%AA %88

ë %C3%AB %89

è %C3%A8 %8A

ï %C3%AF %8B

î %C3%AE %8C

ì %C3%AC %8D

Ä %C3%84 %8E

Å %C3%85 %8F

É %C3%89 %90

æ %C3%A6 %91

Æ %C3%86 %92

ô %C3%B4 %93

ö %C3%B6 %94

ò %C3%B2 %95

û %C3%BB %96

ù %C3%B9 %97

ÿ %C3%BF %98

Ö %C3%96 %99

Ü %C3%9C %9A

ø %C3%B8 %9B

£ %C2%A3 %9C

Ø %C3%98 %9D

× %C3%97 %9E

ƒ %C6%92 %9F

á %C3%A1 %A0

í %C3%AD %A1

ó %C3%B3 %A2

ú %C3%BA %A3

ñ %C3%B1 %A4

Ñ %C3%91 %A5

ª %C2%AA %A6

º %C2%BA %A7

¿ %C2%BF %A8

® %C2%AE %A9

¬ %C2%AC %AA

½ %C2%BD %AB

¼ %C2%BC %AC

¡ %C2%A1 %AD

« %C2%AB %AE

» %C2%BB %AF

Á %C3%81 %B5

 %C3%82 %B6

À %C3%80 %B7

© %C2%A9 %B8

¢ %C2%A2 %BD

¥ %C2%A5 %BE

ã %C3%A3 %C6

à %C3%83 %C7

¤ %C2%A4 %CF

ð %C3%B0 %D0

Ð %C3%90 %D1

Ê %C3%8A %D2

Ë %C3%8B %D3

È %C3%88 %D4

Í %C3%8D %D6

Î %C3%8E %D7

Ï %C3%8F %D8

¦ %C2%A6 %DD

Ì %C3%8C %DE

Ó %C3%93 %E0

ß %C3%9F %E1

Ô %C3%94 %E2

Ò %C3%92 %E3

õ %C3%B5 %E4

Õ %C3%95 %E5

µ %C2%B5 %E6

þ %C3%BE %E7

Þ %C3%9E %E8

Ú %C3%9A %E9

Û %C3%9B %EA

Ù %C3%99 %EB

ý %C3%BD %EC

Ý %C3%9D %ED

¯ %C2%AF %EE

´ %C2%B4 %EF

­ %C2%AD %F0

± %C2%B1 %F1

¾ %C2%BE %F3

¶ %C2%B6 %F4

§ %C2%A7 %F5

÷ %C3%B7 %F6

¸ %C2%B8 %F7

° %C2%B0 %F8

¨ %C2%A8 %F9

· %C2%B7 %FA

¹ %C2%B9 %FB

³ %C2%B3 %FC

² %C2%B2 %FD

Ron, if you want I can send you the table in a txt file formatted in a way which is easier for you to implement the transfomation rules in mt-daapd.

Perhaps there could be an option in the mt-daapd.conf file where people with ntfs drives attached to a slug could turn the option value to 1 thereby telling mt-daapd how to interpret/translate the UTF-8 paths in the XML correctly?