Firefly Media Server doesn’t scan for Chinese filename

[ckyang]

I am happy to have firefly server run on my Windows XP. It’s a lightweight program. Previously I use iTunes. It uses too much resource.

The first time I install RC3. It works fine. But I notice that some Chinese songs disappear.

My configuration is:
Windows XP Home, Traditional Chinese Edition
All songs in Applelosses format and MP3

After try RC4 and two Nightlies build (svn-1328 and svn-1348), all Chinese songs disappear.

-D9 debug information is activated
some logs are extracted as followings:
1. ‘D:MyMusicXmasXmas13 Silent Nightt.m4a’ is OK
2. When scan “D:MyMusic?? & China Blue”
It seems not to work. I notice that original “?? & China Blue” becomes “??? & China Blue” in firefly logs.
3. The error message “Error statting: ???????????” means “Error statting: system cannot find specific file”. It shows Chinese error message. I guess it’s related to Windows Chinese edition.

2006-08-21 00:06:55 (8e8aa07c): Executing: select count(*) from songs where path=’D:MyMusicXmasXmas13 Silent Nightt.m4a’ and idx=0

2006-08-21 00:06:55 (8e8aa07c): Rows: 1
2006-08-21 00:06:55 (8e8aa07c): Executing: insert into updated (id) select id from songs where path=’D:MyMusicXmasXmas13 Silent Nightt.m4a’ and idx=0
2006-08-21 00:06:55 (8e8aa07c): Rows: 1
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic/?????
2006-08-21 00:06:55 (8e8aa07c): Error statting: ???????????
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic/??? & China Blue
2006-08-21 00:06:55 (8e8aa07c): Found ??? & China Blue.. recursing
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic??? & China Blue/???
2006-08-21 00:06:55 (8e8aa07c): Error statting: ???????????
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic/?????
2006-08-21 00:06:55 (8e8aa07c): Error statting: ???????????

[rpedde]

@ckyang wrote:

I am happy to have firefly server run on my Windows XP. It’s a lightweight program. Previously I use iTunes. It uses too much resource.

The first time I install RC3. It works fine. But I notice that some Chinese songs disappear.

My configuration is:
Windows XP Home, Traditional Chinese Edition
All songs in Applelosses format and MP3

After try RC4 and two Nightlies build (svn-1328 and svn-1348), all Chinese songs disappear.

-D9 debug information is activated
some logs are extracted as followings:
1. ‘D:MyMusicXmasXmas13 Silent Nightt.m4a’ is OK
2. When scan “D:MyMusic?? & China Blue”
It seems not to work. I notice that original “?? & China Blue” becomes “??? & China Blue” in firefly logs.
3. The error message “Error statting: ???????????” means “Error statting: system cannot find specific file”. It shows Chinese error message. I guess it’s related to Windows Chinese edition.

2006-08-21 00:06:55 (8e8aa07c): Executing: select count(*) from songs where path=’D:MyMusicXmasXmas13 Silent Nightt.m4a’ and idx=0
2006-08-21 00:06:55 (8e8aa07c): Rows: 1
2006-08-21 00:06:55 (8e8aa07c): Executing: insert into updated (id) select id from songs where path=’D:MyMusicXmasXmas13 Silent Nightt.m4a’ and idx=0
2006-08-21 00:06:55 (8e8aa07c): Rows: 1
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic/?????
2006-08-21 00:06:55 (8e8aa07c): Error statting: ???????????
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic/??? & China Blue
2006-08-21 00:06:55 (8e8aa07c): Found ??? & China Blue.. recursing
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic??? & China Blue/???
2006-08-21 00:06:55 (8e8aa07c): Error statting: ???????????
2006-08-21 00:06:55 (8e8aa07c): Found D:MyMusic/?????
2006-08-21 00:06:55 (8e8aa07c): Error statting: ???????????

I have reproduced this, and have a traditional chinese xp install for testing. I’m not entirely sure why this happens, because if you put that drive in an english xp and run firefly and point it to the traditional chinese hard drive, it works fine. So there is something strange going on there, and I’m not entirely sure what it is.

But I have a system to debug on (although it’s hard for me to get around on it, not being able to read chinese!), and fixing it is a high priority.

I’ll post back to this thread when I have something to test.

— Ron

[rpedde]

I’ll post back to this thread when I have something to test.

— Ron

Okay. I spent the evening trying to isolate this, and I think I have something, but it needs testing. Are you willing to test?

If so, install the latest nightly versions a http://nightlies.mt-daapd.org. Once you have this installed, download a replacement .exe file here:
http://nightlies.mt-daapd.org/tchinese-firefly.exe.zip.

Stop the firefly server, copy the firefly.exe from the zip file over the top of the existing firefly.exe (in c:program filesfirefly media server) and start the server.

I think that will probably work, although there still might be metadata problems if your metadata is in big5 rather than utf8, but you can test and see.

I would appreciate any feedback you could give me.

Thanks.

— Ron

[ckyang]

After replace firefly.exe:

2006-08-21 15:03:53 (ad2cf70a): Starting with debuglevel 0
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/rsp.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/w32-event.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/ssc-ffmpeg.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/ssc-wma.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Starting rendezvous daemon
2006-08-21 15:03:53 (ad2cf70a): db_sqlite2_open: unable to open database: C/songs.db (C/songs.db)
2006-08-21 15:03:53 (ad2cf70a): Error opening db: Misc SQL Error: unable to open database: C/songs.db

So I copy C:Program FilesFirefly Media Serverplugins*.* to C:Prog
But still cannot scan songs.

2006-08-21 15:06:05 (ad2cf70a): Starting with debuglevel 0
2006-08-21 15:06:05 (ad2cf70a): Starting rendezvous daemon
2006-08-21 15:06:05 (ad2cf70a): db_sqlite2_open: unable to open database: C/songs.db (C/songs.db)
2006-08-21 15:06:05 (ad2cf70a): Error opening db: Misc SQL Error: unable to open database: C/songs.db

One of my friends review the source code and give some suggestion.

1. Firefly is built with multibyte character set. So when TChinese XP read file with Chinese filename, it will get MBCS (Multi-byte character set) encoded filename.
2. In this function os_readdir_r(), it uses UTF8 to UTF16 conversion. It may cause wrong conversion.
3. His suggestion is to use Windows API “MultiByteToWideChar” instead of UTF8-to-UTF16.

Thank you for prompt response.

[rpedde]

@ckyang wrote:

After replace firefly.exe:

2006-08-21 15:03:53 (ad2cf70a): Starting with debuglevel 0
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/rsp.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/w32-event.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/ssc-ffmpeg.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Error loading plugin C:Prog/ssc-wma.dll: ?????????w???????C
2006-08-21 15:03:53 (ad2cf70a): Starting rendezvous daemon
2006-08-21 15:03:53 (ad2cf70a): db_sqlite2_open: unable to open database: C/songs.db (C/songs.db)
2006-08-21 15:03:53 (ad2cf70a): Error opening db: Misc SQL Error: unable to open database: C/songs.db

Was this from a working installation of current nightlies? Because it doesn’t look right. Did it start and run properly (aside from not finding some files) when you ran it before you copied over the .exe?

So I copy C:Program FilesFirefly Media Serverplugins*.* to C:Prog
But still cannot scan songs.

Again, this doesn’t look like a multi-byte issue, this looks like a configuration issue.

Try installing the nightly as normal. Start firefly, make sure it works. Once you have that working, stop the server, drop the .exe from the zip file on top of the existing firefly.exe (not in a new directory — overwrite the executable that you just installed from the nightly). See if that works.

I’m almost certain it will.

1. Firefly is built with multibyte character set. So when TChinese XP read file with Chinese filename, it will get MBCS (Multi-byte character set) encoded filename.
2. In this function os_readdir_r(), it uses UTF8 to UTF16 conversion. It may cause wrong conversion.
3. His suggestion is to use Windows API “MultiByteToWideChar” instead of UTF8-to-UTF16.

Hrm. I tend to not want to, as it is not portable to non-windows systems. Also, I’m not sure that this conversion is wrong.

The os_readdir_r functions are all all in the os-win32-u.c file, with is compled with UNICODE support turned on. Which means I should get filenames etc in UTF-16, not multibyte. A UTF-16 to UTF-8 conversion (and back again) shouldn’t lose any data, as both utf-8 and utf-16 can represent the whole unicode set, including those characters above the BMP.

It still might be the case that the log file prints out crap, since thats all utf-8, and your text editor might assume either codepage or utf-16, but that’s just the log file. Internally, all the file open, stat, and other operations are done with utf-16 filenames, using the windows W functions.

It’s possible that there is still a function somewhere that is mangling filenames, but I thought I got the last one last night.

In fact, I was able for the first time last night to index songs with traditional chinese file names using the same binary I uploaded to you.

So I’m thinking maybe it’s a configuration issue.

[ckyang]

Procedure:
1. Install Firefly Media Server (svn-1359)
2. Aftern scanning all music files, it show 5119 songs.

2006-08-22 13:42:33 (ad2cf70a): Starting with debuglevel 0
2006-08-22 13:42:33 (ad2cf70a): Starting rendezvous daemon
2006-08-22 13:42:33 (ad2cf70a): Initializing database
2006-08-22 13:42:39 (ad2cf70a): Starting web server from C:Program FilesFirefly Media Serveradmin-root on port 9999
2006-08-22 13:42:39 (ad2cf70a): Registering rendezvous names
2006-08-22 13:42:39 (ad2cf70a): Serving 5119 songs. Startup complete in 6 seconds
2006-08-22 13:42:39 (ad2cf70a): Rescanning database

3. Stop firefly server
4. Replace firefly.exe in C:Program FilesFirefly Media Server
5. Restart firefly server
6. Firefly server stops after 5 seconds. (Button becomes “start Server”)
The error message:

2006-08-22 13:44:33 (ad2cf70a): Starting with debuglevel 0
2006-08-22 13:44:33 (ad2cf70a): Error loading plugin C:Prog/rsp.dll: ?????????

2006-08-22 13:44:33 (ad2cf70a): Error loading plugin C:Prog/w32-event.dll: ?????????

2006-08-22 13:44:33 (ad2cf70a): Error loading plugin C:Prog/ssc-ffmpeg.dll: ?????????

2006-08-22 13:44:33 (ad2cf70a): Error loading plugin C:Prog/ssc-wma.dll: ?????????

2006-08-22 13:44:33 (ad2cf70a): Starting rendezvous daemon
2006-08-22 13:44:33 (ad2cf70a): db_sqlite2_open: unable to open database: C/songs.db (C/songs.db)
2006-08-22 13:44:33 (ad2cf70a): Error opening db: Misc SQL Error: unable to open database: C/songs.db

????????: means “cannot find specific module”

[rpedde]

@ckyang wrote:

2006-08-22 13:44:33 (ad2cf70a): Starting rendezvous daemon
2006-08-22 13:44:33 (ad2cf70a): db_sqlite2_open: unable to open database: C/songs.db (C/songs.db)
2006-08-22 13:44:33 (ad2cf70a): Error opening db: Misc SQL Error: unable to open database: C/songs.db

Okay… sorry… clearly I misunderstood before. Let me look at this more.

Thank you for your help. I appreciate your patience.

— Ron

[ckyang]

Yes, it’s a configuration problem.
Finally I copy song.db to C:Program FilesFirefly Media ServerC
copy *.dll to C:Prog

It works very fine. 9959 out of 9975 songs are scanned.
It seems that allmost all Chinese/Japanese/Korean songs work fine.

Two problem left
1.) 16 songs are not found. I will find which of kind of songs if possible. It may take a lot of time.
2.) Information of hundreds songs is wrong. It may relate to MP3 ID3 tag encoding (BIG5 or Unicode issue). I will test it and reply.

Thank you very much. I am very happy with this progress.

[rpedde]

@ckyang wrote:

Yes, it’s a configuration problem.
Finally I copy song.db to C:Program FilesFirefly Media ServerC
copy *.dll to C:Prog

I think there is still something not right there.. I think that “c:prog” should be “c:program files…”, but something is truncating it. I’ll have to look at that.

It works very fine. 9959 out of 9975 songs are scanned.
It seems that all Chinese/Japanese songs work fine.

So *something* at least is working. Seems like maybe it’s just reading the configuration file that is bad, somehow.

Two problem left
1.) 16 songs are not found. I will find if possible. It may take a lot of time.
2.) Information of 20~30 songs is wrong. It may relate to MP3 ID3 tag version (BIG5 or Unicode issue). I will test it and reply.

Some songs have id3v2 tags that say they are latin1, when they are really in codepage (big5). I know how to fix this, and it is on my list of things to fix, but I wanted to get the path stuff working first, then I’ll worry about the codepage tags.

Thank you very much. I am very happy with this progress.

And thanks again for your help. I don’t have a lot of chinese music or access to a real chinese system. Mostly I’m kind of guessing in the dark.

It’s nice to be making progress, though. 🙂

— Ron

[ckyang]

After convert all MP3 ID3tag to unicode. All information is correct for Chinese/Japanese/Korean.

iTunes shows 9975 songs. Format is AAC, MP3 and Applelossless.
Firefly server shows 9959 songs.

Firefly test load works fine with Chinese filename in Windows XP Chinese Edition.

[Author]

Posts