Migrating a Synology NAS to a UniFi UNAS Pro 8 with Robocopy, SMB Multichannel, and Surprising Performance Traps

I’ve had a Synology NAS for a very long time, and recently I started moving its contents to a new Ubiquiti UniFi UNAS Pro 8. This seemed like it ought to be a fairly boring operation. Both devices speak SMB, I have a fast network (recently upgraded to 10 gigabit internally), and Windows has had tools for copying files reliably between machines for decades. Naturally, it turned into a whole evening of learning things I thought I already knew, which is why I started a blog lol.

There was a nice bit of history here for me because back in 2007 (good lord!) I wrote a post called “XCopy considered harmful – Robocopy or XXCopy or SyncBack.” My argument at the time was basically that once you are moving enough files, Explorer stops being the move and Robocopy starts looking pretty good. I even used /Z, Robocopy’s restartable mode, because being able to resume a partially transferred file was useful on unreliable connections.

Almost twenty years later, /Z turned out to be one of the most important things I needed to remove because it made everything hella slow.

The migration

The basic job was straightforward. I had shares on the Synology such as:

\\server\music

and matching shares on the UNAS:

\\UNAS-Pro-8\music

I initially used Explorer, mostly because it was there and because sometimes the easy thing really is the easy thing. That lasted until Explorer started producing errors on individual files:

The requested operation could not be completed due to a file system limitation

My first thought was filenames. NAS migrations are full of opportunities to discover that one filesystem is more permissive than another, and there were filenames with parentheses and other punctuation in them.

Then this failed:

\\server\music\Athlete\Tourist\05 Wires.m4p

There is nothing especially exotic about 05 Wires.m4p, so I moved over to Robocopy to get a little more information. It consistently got to 92% and returned Windows error 665:

92%        New File               4.3 m        05 Wires.m4p

ERROR 665 (0x00000299) Copying File
The requested operation could not be completed due to a file system limitation

At this point the useful question was no longer “what is wrong with that filename?” but “which part of the path is refusing this file?”

I copied the file from the Synology to my local Windows desktop. That worked. I then copied the local file from Windows to the UNAS, and that failed with the same filesystem limitation.

That isolated the problem so the Synology could read the file, Windows could store it, and something about writing this particular file to the UNAS was causing trouble.

Alternate Data Streams, again

NTFS files can contain named Alternate Data Streams in addition to the ordinary unnamed stream that we usually think of as the contents of a file. This is an old Windows filesystem feature, and it happens to be one I wrote about in 2007 when discussing Zone.Identifier, which Windows can use to record where a downloaded file came from. I even blogged about Alternate Data Streams in 2003!!! Windows can expose these streams with DIR /R.

So I ran:

dir /r "%USERPROFILE%\Desktop\05 Wires.m4p"

and got:

11/30/2011  02:17 PM         4,576,368 05 Wires.m4p
                               360,456 05 Wires.m4p:01APIC_03.jpg:$DATA

There it is. Alongside the normal 4.5 MB music file was a roughly 360 KB named data stream called 01APIC_03.jpg.

That also explained the strange 92% failure. Robocopy was successfully getting through the main contents of the file and then encountering the additional stream. What had looked like a failure somewhere in the middle of an ordinary .m4p file was actually occurring when Windows attempted to deal with the additional filesystem data.

Robocopy has support for exactly this situation. Microsoft documents X as one of the /COPY flags, meaning “skip alternate data streams.” So:

/COPY:DATX

means copy the file’s data, attributes, and timestamps, but do not copy the alternate streams. /DCOPY:DATX applies the corresponding behavior to directories. I retried the same file:

robocopy "\\server\music\Athlete\Tourist" "\\UNAS-Pro-8\music\Athlete\Tourist" "05 Wires.m4p" /R:0 /W:0 /COPY:DATX /DCOPY:DATX /V

and it completed successfully. The important distinction here is that DATX does not remove metadata stored inside an MP3, M4A, M4P, JPEG, or other file format. It tells Robocopy not to reproduce separate filesystem streams associated with the file. In my case those extra streams were not something I needed to preserve on the new NAS.

The copy worked, but it was slow

Once the ADS issue was understood, I started the larger migration with a fairly conventional-looking Robocopy command:

robocopy "\\server\music" "\\UNAS-Pro-8\music" /E /Z /MT:16 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILE%\Desktop\synology-to-unas.log"

It ran, but performance was all over the place. Sometimes I would see a few hundred megabits per second, then it would drop dramatically. A small file could appear to sit there for a long time. I started wondering whether I was looking at buffering, slow disks, parity calculations, SMB behavior on the UNAS, or maybe my Synology had finally reached its limits.

So now it’s “just try random stuff (bisect)” time. I reduced the number of threads. I tried single threaded. None of that helped. Then I removed /Z.

Microsoft’s Robocopy documentation describes /Z as restartable mode, which lets an interrupted file resume rather than starting again from byte zero. What I had forgotten is that Microsoft’s current migration guidance specifically warns that /Z should be used cautiously because the extra logging required for restartability can significantly reduce copy performance.

My successful music run ended up using:

robocopy "\\server\music" "\\UNAS-Pro-8\music" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILE%\Desktop\synology-to-unas-DATX.log"

The summary from that run was:

               Total    Copied   Skipped  Mismatch    FAILED
    Files :     15940      6991      8949         0         0
    Bytes :  70.907 g  47.917 g  22.989 g         0         0

   Speed :           187,185,171 Bytes/sec.

So the run that copied almost 48 GB of remaining data averaged about 187 MB/sec, with no failed files.

This was not a controlled benchmark where I changed exactly one variable while everything else remained identical, so I’m not going to pretend the number proves that /Z accounted for every bit of the earlier slowdown. The practical difference was large enough, however, that /Z is no longer something I will automatically put in a LAN migration command just because restartability sounds desirable. On a stable local network I would start without it and add it only when I actually need its semantics.

/MT is useful, but it helps a particular kind of problem

Robocopy’s /MT:n option runs copies using multiple threads. It supports values from 1 through 128, with eight threads as the default if /MT is supplied without a number. Microsoft’s own migration guidance also points out that more threads do not automatically translate into a faster migration and recommends measuring thread counts against the actual workload.

This made more sense once I stopped thinking of /MT:4 as “make one file four times faster.”

Imagine a music collection with thousands of files of questionable provenance (I ripped them, just kidding). There is work associated with opening a file, creating the destination file, reading and writing its contents, dealing with metadata, and closing it again. A single-threaded copy has periods where the network or storage can be waiting while one of those operations completes. Having several files in progress at once gives Robocopy opportunities to overlap that work.

For this particular collection, four threads turned out to be a good fit. Sixteen wasn’t obviously helping more, and one thread wasn’t an improvement. I would resist turning /MT into a magic value that belongs in every command line, because a directory containing 50,000 photographs presents a different workload from four 900 GB disk images.

There is also a logging cost worth remembering. Microsoft recommends redirecting Robocopy output to a log when using multithreaded copies, and its migration guidance uses switches such as /NP, /NFL, and /NDL when the objective is throughput rather than watching every filename scroll by.

For a migration I am not actively watching, I would probably use something like:

robocopy "\\server\share" "\\UNAS-Pro-8\share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILE%\Desktop\nas-migration.log"

Then came the large files

Later I started copying some files that were hundreds of gigabytes each. Microsoft describes /J as unbuffered I/O and recommends it for large files, so it seemed like the obvious option to try.

With /J enabled, however, NAS-to-NAS transfer slowed dramatically. The useful thing about having a Windows machine in the middle is that I could test each half of the trip separately. I took one of the exact same large files and copied it directly from the Synology to my local machine. That ran at roughly 250 MB/sec, so the Synology was perfectly capable of reading the file at high speed.

I then removed /J from the direct Synology-to-UNAS Robocopy command:

robocopy "\\server\share" "\\UNAS-Pro-8\share" "huge-file.ext" /R:0 /W:0 /COPY:DATX /NP

and the speed came back.

I don’t think the useful conclusion is that /J is bad. Microsoft recommends it for large-file copies for a reason, and it is entirely possible that it is exactly what you want when copying from local disk to local disk or in another network configuration. What mattered here was that Windows was simultaneously reading from one SMB server and writing to another SMB server, and on this particular path buffered I/O performed much better.

That is a good reminder that command-line switches describe behavior, not guaranteed performance improvements. /J changes the I/O model. /MT changes concurrency. /Z adds restartability. Whether those changes improve a migration depends on the rest of the system.

The Synology was faster than I gave it credit for

At several points I blamed the aging Synology. It is an old machine with spinning disks, so it was easy to assume that a few hundred megabits per second was simply all it had left. Then I remembered that the Synology has four 1 GbE interfaces and that SMB 3 supports Multichannel. I am still surprised this worked so well.

SMB Multichannel allows an SMB session to use multiple network paths simultaneously. Microsoft documents this specifically as a way to aggregate available network bandwidth, and Synology supports SMB3 Multichannel for the same reason.

Windows makes the active channels easy to inspect:

Get-SmbMultichannelConnection -ServerName server |
    Format-Table ServerName,Selected,ClientIpAddress,ServerIpAddress,ClientLinkSpeed,ServerLinkSpeed,CurrentChannels

My machine reported:

ServerName Selected ClientIpAddress ServerIpAddress ClientLinkSpeed ServerLinkSpeed
---------- -------- --------------- --------------- --------------- ---------------
server         True 192.168.1.45    192.168.1.210        1000000000      1000000000
server         True 192.168.1.45    192.168.1.198        1000000000      1000000000
server         True 192.168.1.45    192.168.1.197        1000000000      1000000000
server         True 192.168.1.45    192.168.1.26         1000000000      1000000000

All four 1 GbE interfaces on the Synology were participating in the SMB connection.

My Windows machine currently has a 2.5 GbE adapter (10 gig coming soon), and during the fast copy I was seeing approximately 250 MB/sec arriving from the Synology. That suddenly made the behavior of the system much less mysterious. The Synology was not limited to the throughput of one gigabit Ethernet connection because SMB Multichannel was allowing Windows to use the four available server-side paths, while the 2.5 GbE link on the PC was becoming the smaller network pipe.

Synology’s documentation makes an important distinction here between SMB Multichannel and ordinary link aggregation. Multichannel can increase SMB performance for one client by using multiple network connections, while conventional link aggregation is generally about aggregate throughput across multiple clients and services.

Like I said, I have a 10 GbE adapter on the way for the Windows machine, so there is another experiment available after the migration. The Synology still only has four 1 GbE interfaces, which gives it 4 Gb/sec of network links in aggregate, but removing the current 2.5 GbE client bottleneck should show how much farther the disks and the Synology itself can go. For an older NAS that I had already mentally demoted to “the slow backup NAS,” it performed surprisingly well.

I also tried rsync

Yes, I know, what about rsync? You are saying Windows as a middleman is unnecessary. The Synology can provide rsync, and UniFi Drive can pull from an rsync server using daemon mode. Ubiquiti documents the rsync path under Drive’s Backup Tasks and requires daemon mode for this type of source.

I tried a separate movie share with rsync while the other experiments were going on. It worked, and I saw about 67 MB/sec. It also gave me a destination layout with some additional directory nesting that I would need to clean up afterward.

That is not an argument that rsync is slow in general, nor that its directory behavior cannot be configured correctly. It is just what happened in this particular Synology-to-UNAS test. Once the Robocopy path was reaching roughly 187 MB/sec and preserving exactly the UNC share layout I wanted, there wasn’t much incentive for me to make rsync the primary migration mechanism.

The slightly amusing result was that the apparently indirect path:

Synology -> SMB -> Windows -> SMB -> UNAS

was considerably faster in my environment than asking the two NAS devices to transfer the test share directly with the rsync implementation exposed by UniFi Drive. My guess is because rsync wasn’t using the 4 1gig connections linked. Let me know what you think in the comments.

The Robocopy command I ended up with

For the normal shares containing lots of files, this is the version I would start with now:

robocopy "\\server\share" "\\UNAS-Pro-8\share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILE%\Desktop\nas-migration.log"

The switches have fairly specific jobs:

/E           Copy subdirectories, including empty ones
/MT:4        Allow four file-copy threads
/R:2         Retry a failed copy twice
/W:2         Wait two seconds between retries
/COPY:DATX   Copy data, attributes and timestamps, but skip ADS
/DCOPY:DATX  Apply the corresponding directory copy flags
/XJ          Exclude junction points
/NP          Don't print percentage progress
/NFL         Don't log every filename
/NDL         Don't log every directory
/LOG         Write the useful output to a file

I deliberately do not have /Z in there. I also would not automatically add /J; for a workload dominated by very large files I would test the same transfer both with and without it before committing to a multi-terabyte run.

The /MT value is similarly empirical. Four worked extremely well for this Synology and this mix of files, but I would try 1, 4, 8, or another sensible number against a representative slice of the real data rather than assuming that the largest available thread count is best.

Checking the result

For the files where I care enough to prove that the destination contains exactly the same bytes as the source, PowerShell’s Get-FileHash is a convenient final check:

Get-FileHash "\\server\share\huge-file.ext" -Algorithm SHA256
Get-FileHash "\\UNAS-Pro-8\share\huge-file.ext" -Algorithm SHA256

Get-FileHash uses SHA-256 by default, and if the SHA-256 values match then the two inputs produced the same digest. For enormous files this requires reading the entire file again at both ends, so I am unlikely to hash every song in a music collection, but it is an easy way to verify the particularly valuable multi-hundred-gigabyte files after a migration.

A few things I’d check before blaming the NAS

What made this migration interesting was that the symptoms could have supported several plausible explanations. The UNAS has a new RAID array, the Synology is old, Windows is acting as an SMB client in both directions, the files came from years of different applications, and Robocopy has enough switches to make almost any command line look authoritative.

When a file failed, I copied it Synology-to-local and then local-to-UNAS, which exposed the destination-side ADS problem. When large files were slow, I copied the same file Synology-to-local, which proved the old NAS could still deliver about 250 MB/sec. When the network suddenly became much faster, Get-SmbMultichannelConnection showed that all four Synology Ethernet interfaces were participating. When /J looked slow, removing just that behavior restored the throughput I was expecting.

The final performance was not the result of finding a secret “fast Robocopy” command from a forum. It came from thinking about which features I actually wanted for this migration and removing a few that were useful in other circumstances but expensive in mine.

That is probably the part I will want to remember the next time I do this. /Z, /J, and /MT are not levels on a performance slider. They change restartability, buffering, and concurrency. Alternate Data Streams are real data even when Explorer normally hides them. SMB Multichannel can make an old NAS with several gigabit interfaces much more capable than one might assume from looking at any single Ethernet port.

Robocopy ended up being the fastest thing I tried. As with all advice, this worked for me. Ideally you’ll gind more value in the comments as Hacker News folks and Windows experts will drop in with better tools and strategies. Just remember, there’s more than one way to saturate a network and I completely saturated this one, so I’m pretty happy with the result of my migration.