Handling UTF/ISO 8859-1 characters in SQLite3

[Anonymous]

Present cfgn: running svn-1696 on Windows XP with iTunes as a client.

I’ve built my music library in FLAC files, encoding through EAC and using it to generate the M3U playlist files. I use MP3TAG v2.40 to clean up tags and add artwork.

I have a number of albums that use accented characters in song titles, file names and playlists, e.g.:
The Köln Concert
Thíós Fá’n Chósta

Where these characters are encountered in a song title/file the playlist as displayed by iTunes omits the corresponding song. MP3TAG reports the tags fine.

If I look at songs.db database with SQL Manager (Firefox plugin) the songs table has the right number of songs with the correct text (accented chars OK). However, the playlists table has an incorrect number of items (total songs less number of songs with accented chars).

It looks like the playlist file scan is separate from the song file scan and there’s an error occuring when the playlist file is being read while scanning the M3U file.

Any help appreciated in identifying where the problem lies and working towards a fix.

Thanks

[Anonymous]

I’ve just done a little bit of work on this – I took a copy of songs.db & manually edited it for a playlist where 2 tracks were missing because their titles included ISO-8859-1 legal accented chars.

1) Songs are in songs table, no problem there
2) In the playlist table I corrected the items to 10 from 8 for the specific playlist
3) I added two playlistitems table entries for the errant songs

I changed FF config to disable scanning, stopped FF, copied in the amended songs.db, restarted….

All worked – iTunes shows all the songs with the correct characters in the titles.

So.. where are the .m3u playlist files parsed into the database?

[LeoD]

In order to have the playlists processed correctly you have to save them in UTF-8 format:

Notepad : File->Save As->Change coding from ANSI to UTF-8

EMACS : Options->Mule->Set Codings Systems->
For Saving This Buffer->utf-8

Note that the playlist entrys with special characters won’t be recognised by Winamp any more unless you rename it to *.m3u8. (But then Firefly won’t find it. I once tried to patch the source for that, but I did not succeed due to my lack of C programming skills.)

[Anonymous]

Hey Leo,

Thanks for that point on Winamp – my problem is with Firefly, I’ve debugged it down to how the playlist info is parsed into SQLite3 database. The filenames are loaded into the DB fine but when Firefly’s code tries to parse the playlist files that’s when the cockup happens.

[fizze]

There is an option in firefly that specifies the format. It’s in the advanced preferences, under “Scanning”. Set that to UTF-8 and you’re good to go with UTF-8 playlists, that is. 😉

[Anonymous]

here is an option in firefly that specifies the format. It’s in the advanced preferences, under “Scanning”.

True, but the default is ISO-8859-1 anyway (andI assume that’s Latin-1 too).

I went digging around the trac site and there’s a bug open about this but no progress toward a resolution. The SQLite3 database is getting loaded with the tag information correctly, full character set and all, from the individual media files. I think the problem is coming when Firefly then scans the M3U files, it’s failing to parse the entries with extended set chars: it ends up counting X less items per playlist (X being songs/files with extended set chars). Accordingly, the playlistitems table is then missing the matches between files and playlist item IDs. It’s easy to hack it back together for a dozen or so files.

[fizze]

This might be limited to Windows platforms…?

I know for a fact that I do have playlists with extended chars and they work. I’m running svn-1696 on a nslu2 (POSIX)

I know this might not be productive, but it might help to locate the problem before fixing it 😉

[Author]

Posts