Firefly for Mac and a big iTunes Library (300GB)

[eldino]

Hi there,
I got the Mac version of Firefly from Rokulabs page and I decided to test it on my main, big library (300GB, mostly mp3s, 45000+ tracks), that resides on a nice, dedicated, HFS+ formatted, Firewire400, 500GB external drive.

My only Mac is my iBook G4 1,33 512MB (the last one before Macbooks birth) . I installed Firefly and I gave it the music folder residing on the external hard drive, then It started scanning.. after an hour my laptop was blocked.. I started (after maaaany minutes) the System Monitor and WOW.. “Firefly takes 1,67Gb of virtual memory (swap!) and 300mb of real memory! I tried to kill the server and restart it several times and after some days of tests I decided to post my experience here..

It depends by my hardware or my library is too big?
Anybody had similar experience?
Since I have many projects in mind regarding Firefly, DAAP and Creative Commons music, I’d like to know if it’s capable to manage big libraries with my or older hardware.
Ah it seems that the Songs.db file stops growing up at 49 MB (ca 32000 songs)..

My point is: if a crappy NAS (with a small MIPS cpu in it) can handle mp-daapd and hundreds of GBs of music, why can’t I do it with a real CPU?

I’m gonna install md-daapd on my linux laptop (that is old than my Mac) and try to do the trick with that.. could I get advantages of this in terms of speed?

Thanx for replies

[rpedde]

@eldino wrote:

Hi there,
I got the Mac version of Firefly from Rokulabs page and I decided to test it on my main, big library (300GB, mostly mp3s, 45000+ tracks), that resides on a nice, dedicated, HFS+ formatted, Firewire400, 500GB external drive.

My only Mac is my iBook G4 1,33 512MB (the last one before Macbooks birth) . I installed Firefly and I gave it the music folder residing on the external hard drive, then It started scanning.. after an hour my laptop was blocked.. I started (after maaaany minutes) the System Monitor and WOW.. “Firefly takes 1,67Gb of virtual memory (swap!) and 300mb of real memory! I tried to kill the server and restart it several times and after some days of tests I decided to post my experience here..

It depends by my hardware or my library is too big?
Anybody had similar experience?
Since I have many projects in mind regarding Firefly, DAAP and Creative Commons music, I’d like to know if it’s capable to manage big libraries with my or older hardware.
Ah it seems that the Songs.db file stops growing up at 49 MB (ca 32000 songs)..

My point is: if a crappy NAS (with a small MIPS cpu in it) can handle mp-daapd and hundreds of GBs of music, why can’t I do it with a real CPU?

I’m gonna install md-daapd on my linux laptop (that is old than my Mac) and try to do the trick with that.. could I get advantages of this in terms of speed?

Thanx for replies

Try nightlies at http://nightlies.mt-daapd.org. Specifically, I’d look at the nightly version 1489. I think there may have been some memory leaks in the xml parser, depending on what roku version you have.

But it ought to scale okay, so I think it’s probably a version-dependent bug.

See where 1489 gets you. I think it should work okay for you. There are some folks at some universities that are serving terabyte scale libraries with it (live performance stuff from college of music), so I know it scales that high.

— Ron

[eldino]

Hi thx for repyl 🙂 .. terabytes of stuff?? wow!! I will try your suggest 🙂
also If I doubt it depends by the XML parser, since my “ExternalDisk/Music” folder doesn’t not contain the iTunes library.xml but just the music.. btw i will check thx 🙂

ah i have any roku product (also if they look so nice!!) but i’m just trying to share music via DAAP inside my wlan (and later in some other places, see “DAAP Party” thing =)

[rpedde]

@eldino wrote:

Hi thx for repyl 🙂 .. terabytes of stuff?? wow!! I will try your suggest 🙂
also If I doubt it depends by the XML parser, since my “ExternalDisk/Music” folder doesn’t not contain the iTunes library.xml but just the music.. btw i will check thx 🙂

ah i have any roku product (also if they look so nice!!) but i’m just trying to share music via DAAP inside my wlan (and later in some other places, see “DAAP Party” thing =)

There were likely others, but the big one I remeber was xml. It’s possible you are hitting a code path that’s generating more unfreed memory, or perhaps the item counts you have are doing it. In any case, I bet the version mentioned takes care of it.

If not, I have memory debugging stuff in there I might ask you to run, if you would have the time and inclination.

Thanks.

— Ron

[eldino]

Hey Mr. Ron, as I describe with more details on a post on my blog (http://eldino.wordpress.com/2007/07/16/en-firefly-media-server-how-does-it-act-with-a-big-music-collection/),
i kinda solved my problem.

It seems that, when Firefly, both Mac and Windows version, scans some mp3s that it doesnt like, It hangs your computer eating every kb of real and swap memory, and cpu too, and the only solution is to reboot the machine. I isolated some folders of mp3s it doesn’t like and I will do more testings about that soon, so If you are interested, i will post here more infos about what kind of mp3s they are (bitrate, encoder etc).

Actually I run the 98% of my music collection inside Firefly and everything works so great!! I stream music to myself because loading all those 45000+ tracks inside iTunes “natively” takes ca 400mb of ram (!!), so I serve songs via Firefly and I listen to them via iTunes sharing thing =)
cool hmhm? =)

[rpedde]

@eldino wrote:

Hey Mr. Ron, as I describe with more details on a post on my blog (http://eldino.wordpress.com/2007/07/16/en-firefly-media-server-how-does-it-act-with-a-big-music-collection/),
i kinda solved my problem.

It seems that, when Firefly, both Mac and Windows version, scans some mp3s that it doesnt like, It hangs your computer eating every kb of real and swap memory, and cpu too, and the only solution is to reboot the machine. I isolated some folders of mp3s it doesn’t like and I will do more testings about that soon, so If you are interested, i will post here more infos about what kind of mp3s they are (bitrate, encoder etc).

Actually I run the 98% of my music collection inside Firefly and everything works so great!! I stream music to myself because loading all those 45000+ tracks inside iTunes “natively” takes ca 400mb of ram (!!), so I serve songs via Firefly and I listen to them via iTunes sharing thing =)
cool hmhm? =)

If you find some pathological mp3 files, I’d like to see them at [email protected].

— Ron

Edit: Hey! The localization works! 🙂 The performance thing hopefully will be fixed after the Great DB Reorg, that’s been pending for a long time, but may soon happen. 🙂

[eldino]

Hey! The localization works! Smile The performance thing hopefully will be fixed after the Great DB Reorg, that’s been pending for a long time, but may soon happen.

Sure it works =) just the menu that comes down from the menubar is not traslated but the rest is fine =) yay for db reorg..

[rpedde]

@eldino wrote:

Hey! The localization works! Smile The performance thing hopefully will be fixed after the Great DB Reorg, that’s been pending for a long time, but may soon happen.

Sure it works =) just the menu that comes down from the menubar is not traslated but the rest is fine =) yay for db reorg..

Aaaah… I’ll add that to the translation files.

Thanks.

— Ron

[Author]

Posts