Boomzilla's Journey (ongoing)

[KeithL]

It's not as puzzling as it may seem and, as the saying goes, "they have their reasons".

Roon is essentially a big database that stores metadata... and an engine for manipulating and displaying that data.

According to the information I've seen Roon usually does incremental backups.

This means that, each time you do a backup, only the database records that have been added or changed since the last backup are saved.

Whenever a record is added or saved a flag is set indicating that it is new.

Then, the next time you do a backup, the program reads through the database, and picks out the records whose flag is set.

These records are included in the current backup and their flag is cleared so they won't be included in the next one.

This method can be very efficient in terms of space - especially if you don't make a lot of changes.

However it does mean that the entire database must be read, item by item, and each item considered separately.

Because of this it can become very slow if you make a lot of changes.

(And, if you use this method to back up the entire database, you essentially set ALL the flags, so ALL records are treated as if they were changed.)

(In most software this is actually done in the inverse... an "archive" flag is set when a record is backed up.)

However, if you want to back up the entire database, you DO NOT want to do it this way.

In that case what you want to do is to back up the entire database file as a single big file.

(Since you're going to save every single record anyway there's no compelling purpose to look each one individually.)

This is normally referred to as a full backup rather than an incremental backup.

However not all programs offer that option.

(And that's why using an "outside" program to save or migrate the entire database as a file may be a LOT faster.)

Note that there are (arguably) a few reasons why some folks may still prefer to do it that way anyway....

For one thing, it allows them to examine each record, and discard records that are no longer being used, but may not have been removed yet.

(This is known as "compacting the database".)

And, for another, it offers them an opportunity to "rebuild the database", at which point they can repair any errors they might notice.

It analogous to dumping out a drawer, and sorting the contents into a new drawer, compared to just moving the drawer to a new location.

(You get to decide what to keep, you get to rearrange things neatly, and you get to dump out any lint that has collected in the corners.)

However it CAN be VERY slow...

And, because so many individual operations are being carried out, it offers more opportunities for things to go wrong.

Dec 7, 2020 11:17:47 GMT -5 Boomzilla said:

The "Roon backup" does not actually backup any music. Instead, it backs up the state of the Roon metadata and file organization. A typical Roon backup is but a few gigabytes, which is why it is so puzzling that it takes so long.

[KeithL]

For a project like that, assuming you have separate files, your best bet is to use a separate converter program.

For that I would recommend dBPowerAmp; they have a "Batch Converter" that is quite fast and does an excellent job.

(You get to choose your output format and quality settings and it has check boxes for which files and folders to include.)

Dec 7, 2020 10:48:41 GMT -5 ttocs said:

I recently needed to convert a couple hundred gigs of music files to a crummier, smaller format not to exceed 64 gigs. There was a requirement (out of state special school issued MP3 player (no cell phones allowed)) for the music files to be on a USB stick, so I was "stuck" with that form factor. The initial estimate said it would take 16 hours - which I didn't believe, I thought it would take less time, but it took 21 hours to do the conversion/load onto the stick.

For a sanity check I performed the same task using an external USB SSD drive. The estimated time showed it would take about 20 minutes (it always takes a little longer), so I stopped, just wanted to make sure I wasn't crazy, yet.

[klinemj]

KeithL - It might be a good time for you to learn a bit more about what Roon is doing before commenting much more. Note boom's comments about what Roon is doing - and not doing.

Boomzilla - it is odd that it is taking yours so long. When I first set mine up, it was very fast. Of course, I did have very little in the way of customized metadata even though I had over 1000 CD's in my collection. From what I recall, you have a lot for which you entered custom data. I do look forward to what you learn about the pops and clicks.

(And, oddly enough...when putting my system back together after tearing it apart to install the new TV, I had a few Roon issues. Mine were all self-created. First, I had unplugged the ethernet cable that connected my core to wall jack, and when I put the system back together - I forgot I had unplugged it. Roon couldn't find my music library located on a drive attached to my router. It took me about 5 minutes to find where the end of the cable that plugs into the wall had gone! And, I couldn't "see" my zone in my great room. My wife had requested we move the TV and stand it's on - which includes that Roon zone. Somehow, in the process of the move, the power strip that Roon endpoint is connected to got turned off. So, luckily, all issues of my or my wife's creation and easily solved).

Mark

[ttocs]

Dec 7, 2020 13:16:42 GMT -5 KeithL said:

For a project like that, assuming you have separate files, your best bet is to use a separate converter program.

For that I would recommend dBPowerAmp; they have a "Batch Converter" that is quite fast and does an excellent job.

(You get to choose your output format and quality settings and it has check boxes for which files and folders to include.)

Dec 7, 2020 10:48:41 GMT -5 ttocs said:

I recently needed to convert a couple hundred gigs of music files to a crummier, smaller format not to exceed 64 gigs. There was a requirement (out of state special school issued MP3 player (no cell phones allowed)) for the music files to be on a USB stick, so I was "stuck" with that form factor. The initial estimate said it would take 16 hours - which I didn't believe, I thought it would take less time, but it took 21 hours to do the conversion/load onto the stick.

For a sanity check I performed the same task using an external USB SSD drive. The estimated time showed it would take about 20 minutes (it always takes a little longer), so I stopped, just wanted to make sure I wasn't crazy, yet.

I used JRiver, like always. It was only the USB stick that slowed things down. So going from drive to drive was quick, as I proved to myself.

[DYohn]

Roon Backup backs up your local index so the system does not have to re-scan every time it updates. If you wish to back up your library you have to use software in your server, it is not part of Roon.

One of the reasons I like using Vortexbox as my server OS is it has a very efficient library backup routine built into it. All I have to do is click a button.

Read this (everything ROCK applies to Nucleus) help.roonlabs.com/portal/en/kb/articles/rock-storage-basics so you know where to point your backup software

[Boomzilla]

Dec 7, 2020 13:21:16 GMT -5 klinemj said:

Boomzilla - it is odd that it is taking yours so long. When I first set mine up, it was very fast. Of course, I did have very little in the way of customized metadata even though I had over 1000 CD's in my collection. From what I recall, you have a lot for which you entered custom data. I do look forward to what you learn about the pops and clicks...Mark

Hi klinemj -

From what I read on the Roon groups, my backup time isn't unusual for a library my size. The best way I see to get back what I lost is to do the following:

Hook the library drive back up to the Mac Mini - That drive is where the year-old backup that Roon accepts as "good" is located.

Do an incremental backup to that 2019 backup file - This will catch any changes since that backup

Restore the now-up-to-date backup on the nucleus

This should get back all the changes since the previous backup.

And now a question - based on your experiences, would it be prudent to power the nucleus through a UPS?

Thanks - Boomzilla

[klinemj]

Dec 7, 2020 14:50:06 GMT -5 Boomzilla said:

Dec 7, 2020 13:21:16 GMT -5 klinemj said:

Boomzilla - it is odd that it is taking yours so long. When I first set mine up, it was very fast. Of course, I did have very little in the way of customized metadata even though I had over 1000 CD's in my collection. From what I recall, you have a lot for which you entered custom data. I do look forward to what you learn about the pops and clicks...Mark

Hi klinemj -

From what I read on the Roon groups, my backup time isn't unusual for a library my size. The best way I see to get back what I lost is to do the following:

Hook the library drive back up to the Mac Mini - That drive is where the year-old backup that Roon accepts as "good" is located.

Do an incremental backup to that 2019 backup file - This will catch any changes since that backup

Restore the now-up-to-date backup on the nucleus

This should get back all the changes since the previous backup.

And now a question - based on your experiences, would it be prudent to power the nucleus through a UPS?

Thanks - Boomzilla

Your plan makes sense to me.

On using a UPS with the nuke, I have never considered it myself. My PC is connected to an ordinary surge protector, but other than that - nothing special.

Mark

[Boomzilla]

I have several drives that I'd like to convert to NAS. Is this as simple as using a USB to Ethernet adapter, or are there other electronics involved?

[klinemj]

Dec 7, 2020 15:27:33 GMT -5 Boomzilla said:

I have several drives that I'd like to convert to NAS. Is this as simple as using a USB to Ethernet adapter, or are there other electronics involved?

If your router has USB ports, you can attach a USB hard drive to it and have anything on your intranet access it. That's what I do. I was thinking of doing an actual NAS but I already had other hard drives sitting around and I didn't need to access the data remotely.

My Roon Core quickly found the drive and mapped it (I looked back to my review of Roon and see that it just took 15 minutes and I was up and running.) And, I also access that same drive from my desktop and my Surface tablet to get to pix, documents, etc. Of course, as Keith recommends, I do keep that drive backed up...redundantly (and the backup drives are not connected to power or anything except for during backups...avoiding risk of lightning strikes...).

Mark

[Boomzilla]

Hi klinemj -

First, my router DOES have USB ports. So let me up the ante. If I use a four-port USB pigtail, can I hook up multiple drives to the router's USB port and have all of them available via Ethernet? The drives would all have unique names.

Second, does it matter what format the drives use? Roon recommends EXFAT, if I remember right, but says it can also read any other MS-compatible format (NTFS, etc.). I believe that the Roon will also read OS-X format (APFS or Mac OS-X Extended). The question is, will it read multiple formats off the same network as independent drives.

Third, OS-X has the ability to combine multiple drives into a single logical RAID. Can I use multiple network drives, different formats, combined into a RAID by OS-X as a single logical drive from the Roon nucleus? This would allow me to have a HUGE logical drive without having to buy new (expensive and larger) physical drives.

Fourth, assuming I opt for a NUC with ROCK (or buy the nucleus+), is there any advantage to using a 4TB or larger SSD as an internal drive in the Roon Core?

Fifth, can Roon use multiple library drives simultaneously. For example, if I wanted to assign a single drive to all my classical recordings, and another drive to all my pop & jazz, can Roon Core simultaneously consider both drive locations part of a unified Roon library?

Thanks - Boomzilla

And before someone asks "what's the purpose of all this?" I'll go ahead and answer the question. Currently, my audio rack (in the listening room) contains a lot of "not really audio" components, currently including:

A Mac Mini computer
A DVDRW for ripping new CDs to the USB HDD
A USB HDD that contains all the Roon Library music
An Ethernet switch
An Apple AirPort Express Wi-Fi router
Power cords and connectors for all the above

To the extent possible, I'd like to clear this clutter from the audio rack and locate it elsewhere in the house. If I can get the HDD away from the Roon device, the rest will be easy. So the main goal that I'm trying to accomplish in the near-term is to get that USB HDD that is the Roon Library drive onto the Ethernet network and away from its native USB connector. I'm going to try using the "USB socket in the router" gambit tomorrow (and THANKS again for the info, Mark!).

[LuisV]

Yes, you can have separate drives or files shares with different libraries and Roon will consolidate them; exactly how I have it configured.  The drives in my NAS are formatted with XFS and Roon is perfectly fine with it.  

[audiobill]

As Boom has pointed out, much discussion on the Roon forum; @keithl could gain some background in Roon there.

[klinemj]

Boomzilla - good questions! Re. the 1st, I think that would work. I have not tried it, but it's worth a shot. Re. the 2nd...I have never looked into that. I don't have any OS-X format in my system...just the standard MS-compatible formats. And, as I am not familiar with OS-X at all, I'll defer to others on that.

Re. #4, if I were doing an NUC w/ROCK, I would not personally put a large internal drive in it UNLESS it were for the purpose you state - of de-cluttering. In my case, that would not apply as my router if far away from my AV rack and therefore - so is my HD that's attached to the router. Both are tucked away on the stand my family room TV is in - which is on a different floor than my listening room. The reason I would not put an internal drive in is that I tend to like to have small HD's on computers that only hold the OS and minimal other stuff, then have each access the central drive for true storage. I have 4 PC's around the house for various needs (my office, my Surface tablet, my PC that runs DIRAC and is my Roon Core, and one my wife uses). Having all data stored on 1 drive makes the backup process very easy...just 1 drive to worry about backing up. But - for decluttering your rack - makes sense.

On #5...I am not 100% sure, but I think the answer is yes. If you go into "settings, storage" you will see the option to add a folder for Roon to watch. I think you can add a folder on a different drive, but I have never tried as I put all music onto 1 drive. That said, as you search for a particular selection, I don't know if you can steer Roon to one folder over another - so it may be academic.

Mark

[klinemj]

One other thing I forgot to mention about having the HD on the router...all my PC's will find the drive UNLESS I happen to be using another network. In my house, I have a cable modem and router from my ISP, and it has it's own wifi network but I typically don't use it. I use one from my Asus router as it's very strong, very fast, and reaches far distances outside my house (important when I am floating in the pool and wanting to control Sonos from my phone via the Asus's wifi). (I keep the ISP's network intact in case something happens to my Asus router...I had a router die from a lightning strike once, and having the ISP's as backup was handy while I waited for the new router to arrive.)

Anyway - occasionally, my Surface will disconnect from the Asus wifi and lock into the ISP's wifi if I am downstairs in a spare room. If it does, my Surface won't see the HD on the Asus because the Asus is a separate network. If you don't have this situation - you won't have that problem. When I encounter it (which is rare), I just tell the Surface - hey dummy - stop using the ISP's wifi, and normal conditions return.

Mark

[audiobill]

Dec 8, 2020 9:00:33 GMT -5 klinemj said:

One other thing I forgot to mention about having the HD on the router...all my PC's will find the drive UNLESS I happen to be using another network. In my house, I have a cable modem and router from my ISP, and it has it's own wifi network but I typically don't use it. I use one from my Asus router as it's very strong, very fast, and reaches far distances outside my house (important when I am floating in the pool and wanting to control Sonos from my phone via the Asus's wifi). (I keep the ISP's network intact in case something happens to my Asus router...I had a router die from a lightning strike once, and having the ISP's as backup was handy while I waited for the new router to arrive.)

Anyway - occasionally, my Surface will disconnect from the Asus wifi and lock into the ISP's wifi if I am downstairs in a spare room. If it does, my Surface won't see the HD on the Asus because the Asus is a separate network. If you don't have this situation - you won't have that problem. When I encounter it (which is rare), I just tell the Surface - hey dummy - stop using the ISP's wifi, and normal conditions return.

Mark

So much progress from "the old days" when you were challenged by modem speed!!

[klinemj]

Dec 8, 2020 9:19:32 GMT -5 audiobill said:

So much progress from "the old days" when you were challenged by modem speed!!

You bet! I don't even have the fastest internet my ISP offers, and it's faster than my brain is.

Mark

[DYohn]

Yes, you can add multiple folders to the "Watched Folders" list in the Roon storage setting. I do not think there is a limit but I have read that some users experience crashes if one of the listed folders becomes unavailable. As far as adding a network path that is not a problem. If it's on the same network and subnet you can add it - it must be shared.

[DYohn]

And to add to the above: Roon watches the listed folders in real time 100% of the time. You cannot add a path that is not on line, for example, nor can you expect the system to remember a path if it becomes unavailable. Leave the system on all the time is the key to things working properly. If you can't do this, then expect issues.

[klinemj]

Dec 8, 2020 11:47:13 GMT -5 DYohn said:

And to add to the above: Roon watches the listed folders in real time 100% of the time. You cannot add a path that is not on line, for example, nor can you expect the system to remember a path if it becomes unavailable. Leave the system on all the time is the key to things working properly. If you can't do this, then expect issues.

Interesting...I've never had Roon lose a path when power is off either to the Core or the router. I should say..."knock on wood"! But, it's good to know that's a possibility.

I do like how Roon watched the listed folders in real time. It's nice to rip some CD's, put them onto my HD, and immediately be able to listen to them. With Sonos they can take a while to show up unless I force an update of the library.

Mark

[Boomzilla]

I (sadly) packed up both the Black Ice Audio Fusion F11 vacuum tube integrated amplifier and the full set of Kimber Kable interconnects and speaker wires for return to their respective manufacturers. I'm sorry to see them go.

Next, I need to dig out the documentation on my RAID box and figure out how to configure it as NAS. I think that getting from an external USB drive to NAS (without a host computer somewhere in the middle) is a battle that may not be worth fighting

Later this week, I need to retrieve my Heathkit amps and try them out on the Double Impact speakers.

BZ