[image]

Tech Notes: PC-Phone USB Sync

This page hosts technical notes for technical readers. It's a random collection of more-advanced topics related to using the PC-Phone USB Sync app. This page by nature is also a work in progress; please check back occasionally for new usage notes here. Current contents:

The Fate of Symlinks in Sync Roundtrips

Symlinks are a special type of file that redirect accesses to other files or folders. They're not commonly used, but when they are, they allow content to be shared instead of duplicated. They're also not what some platforms call aliases or shortcuts (which are really features of the GUI, not the filesystem), and if you don't know what they are, you probably don't need to read the rest of this note.

But if you use symlinks, you need to know that they are a special case in this app. As mentioned briefly in the Usage Overview, symlinks are not portable across all devices and platforms, and your best option is to omit them in content folders managed by this app. The full story on this both subtle and platform specific, and merits extra coverage here.

Symlinks on macOS

Short story: symlinks stored on macOS will survive roundtrips to platforms that don't support them, as long as you don't change the stub files macOS makes to smuggle them across device boundaries. No workarounds are required in this limited macOS-only role, but you shouldn't expect your macOS symlinks to work on other platforms, and other use cases are more gray.

Details: macOS automatically and internally forges symlinks both to and from stub text files, whenever they are written to removable drives that don't naturally support them—including those using the portable exFAT and FAT32 filesystems. When these stubs are processed on Windows or Linux, they are seen as simple text files instead of links, and recorded there that way.

For example, here's what a stub file written for a symlink on macOS looks like on an exFAT drive when viewed on Windows; macOS converts this back to a symlink when it's later accessed, but Windows treats it as a simple text file:

XSym
0015
18eeefc3e8a24afd5f6beaf5f1e0b3ec
folder/file.txt

Like many acts of covert hackery, the macOS symlink forgery is a mixed bag. Among its upsides:

The downside to macOS's symlink forgery is that it's only skin-deep: if you ever try to use the stub files outside of macOS's scope, they will be simple text files, and this may have consequences that are both unwanted and unexpected.

For instance, serving a folder with macOS stubs from a network server (e.g., WebDAV or Samba) may serve stubs as simple text files—which will likely yield mixed-type mismatches with the corresponding symlinks back on macOS. The net effect will generally replace your macOS symlinks with simple files during a cross-network sync, thereby destroying them. Yes, surprise!

On the whole, macOS's stubs files mean that you can use its symlinks in your content folders on macOS, without losing them in sync roundtrips with this app. But you should be aware of the potential for unforeseen and dire consequences in other contexts, and shouldn't expect your symlinks to have any meaning outside the macOS realm.

Symlinks on Linux

Short story: symlinks stored on Linux don't work in this app, unless you use removable drives formatted with filesystems that won't generally work on non-Linux devices. For portable usage, your options in this app are to omit symlinks from your content folders; use a provided script to restore them after roundtrip syncs; or zip and unzip folders containing them for transit. Omitting is easiest.

Details: Linux throws errors whenever you try to copy symlinks to drives that do not support them (e.g., exFAT and FAT32). During syncs, these errors are reported in this app's action logfiles, and the failing symlinks are skipped. Because this omits symlinks in content copies, your Linux symlinks won't survive sync roundtrips: syncing back from a symlink-less drive to Linux will classify symlinks on Linux as unique-TO items, and delete them as part of making TO match the FROM drive.

For reference, here's what a symlink error looks like in logfiles, when Python-coded folder syncs and copies attempt to create a symlink on an exFAT drive on Linux—and ultimately skip it altogether (look for an error indicator in logfile summaries):

propagating symlink ./file-link
**Error copying FROM file: skipped ./file-link
 [Errno 38] Function not implemented: 'file' -> '/media/me/T7/_temp/test-symlinks-cpall-linux/file-link'

Unfortunately, creating stub files on Linux as on macOS won't help. They would be seen as simple text files when synced back to Linux, producing mixed-type mismatches that would overwrite and replace your Linux symlinks.

The stub trick works on macOS only because its low-level system libraries map stubs back to links. No such mapping happens on Linux, and there's no reliable way to distinguish manual stub files from other use cases. Users may really want to replace a link with a plain file, and no special file content is guaranteed to uniquely identify a stub. (To be fair, macOS's stub trick isn't foolproof either, but we'll leave the judgement aside here.)

And while this app might skip symlinks in both FROM and TO much like it handles platform-unique cruft, this could discard part of your content; preclude using symlinks in contexts that support them; or convolute the UI needlessly for a type of file too rare to have become a universal across platforms and devices. There is no tradeoff-free automatic answer to the symlink conundrum.

Workarounds: all that being said, if your content folders contain symlinks on Linux, there are three procedural ways you can address them when using this app:

Restore script
As a partial concession to symlink portability limitations, this app provides a free utility script available online here, which restores deleted symlinks on Linux immediately after a SYNC removes them. This script works by scanning the TO folder's __bkp__ change backups to find all removed symlinks to put back. It's run from a Terminal command line with the content folder's path as a sole argument; but if you use symlinks, you probably already know how to run a Python script from a command line.
Zip and unzip
If that script won't suffice, you can also retain symlinks in your content folder by zipping them for transit with the free and open-source ziptools system. That system protects symlinks by encoding them in zipfiles in a way that supports recreating them on unzips. To use this to keep your symlinks intact across this app's syncs, simply zip any subfolders that harbor symlinks in FROM, and unzip them on TO when you need to access the subfolders' content. This requires more manual steps than the restore script, unless you rarely need to access zipped subfolders. Keep in mind that, even with ziptools adjustments, symlinks are portable only if they use all-relative paths; symlinks that use absolute or platform-specific links paths will never work on some receiving ends under any workaround.
Omit symlinks
Finally, if the restore script and zips are too onerous, your only recourse in this app is to avoid storing symlinks in content folders destined for drives and platforms that don't support them. This is a regrettable but unavoidable interoperability dead end. For help locating symlinks in your content folder that must be relocated, either watch for error message in logfiles, or run this free and open-source script.

In the end, omitting symlinks from content folders destined for use across devices and platforms is your safest and simplest option. Restores and unzips can help if you must sync symlinks, but they add more manual steps than most content collections should impose.

Symlinks on Windows

Short story: Windows' symlinks-portability story is the same as on Linux. Reread the preceding section replacing every "Linux" with "Windows" and you're mostly set. Just as on Linux, your options in this app are to omit symlinks; use a script to restore them after roundtrip syncs; or zip and unzip folders containing them for transit. Omitting is easiest.

Details: symlinks are even rarer on Windows than Unix (i.e., macOS and Linux) due to historically limited support, but Windows 11 allows symlinks to be used more freely than its predecessors (in short, admin permission is now largely optional). Even so, symlink writes to portable removable drives will fail with error messages on Windows as on Linux, leaving holes that trigger symlinks removals if you ever sync back from removable drive to Windows.

Technically, the fate of your Windows symlinks depends on the tools you use to try exporting them. Explorer copies will simply follow symlinks and wrongly copy what they point to, while Python code like that used by this app will try to copy the link itself and generate a Windows error message; in copies and syncs, the net result skips the symlink, which means it will be unique and hence deleted on Windows in later back syncs. Here's a logfile error message triggered by a symlink copy to exFAT on Windows:

propagating symlink test-symlinks\link-file.txt
**Error copying FROM file: skipped test-symlinks\link-file.txt
 [WinError 1] Incorrect function: 'file.txt' -> 'd:\\_temp\\test-symlinks-cpall\\link-file.txt'

Workarounds: as also on Linux, you can use the same provided restore script to put back your symlinks on Windows after a sync removes them; zip and unzip folders containing symlinks for transit as needed with ziptools; or simply keep symlinks out of your synced content folders. See this script for help with the latter and simpler option.

Symlinks on Android

Short story: you cannot create symlinks in either the shared (PHONE) or app-specific (APP) on-phone storage folders made available in this app, so they are a moot point on Android: there is nothing to copy or preserve. Moreover, removable drives on Android have the same limitations as on straight Linux: exFAT and FAT32 don't support symlinks at all.

Details: for the record, it is possible to make symlinks on Android in app-private storage (Android is Linux, though a locked down flavor of it), but this storage cannot be accessed directly by any other app than its owner, and it's not exposed or discussed by this app for that reason. Sans a rooted phone, symlinks are a nonstarter on Android for content folders.

Symlink Wrap-Up

In sum, this app fully supports symlinks on platforms and devices which do too. But as a rule, your content-management life will be much less stressful if you simply omit symlinks from cross-platform content. If you don't, there's no guarantee that your symlinks will survive all the journeys on which they embark, and the rules that govern their fate are convoluted. Symlinks probably should have been standardized early on, but the world we inhabit is more proprietary.

All that said, this app could be changed to skip symlinks in both FROM and TO as an option, if there is demand for this mod. This is not implemented today, because it would discard part of your content; complicate the app's user experience for a very-rarely-used tool; throw off content counts used to validate syncs; and either preclude or convolute symlinks in same-platform use cases that would otherwise support them. If you use symlinks and have an opinion on this, please register it with the hamburger menu's Contact; this app's goal is to satisfy customer needs, not theoretical whims.

Related point: filenames in your content may also be nonportable, but the app's NAME action can be used to make them work across all platforms automatically; see the GUI Guide for more info. There's no such tool for symlinks, because they cannot be fixed without deleting them from your content altogether—which seems dangerous enough to warrant manual intervention.

More on Logfiles and Errors

New: as of 1.1.0, action-exit popups now extract and present logfile summaries. This often makes it unnecessary to view logfiles (the popups' info usually suffices when no problems occur), but this note presents error-handling details useful to all users. For more on the popups, see the News post.

Logfiles are not covered much in this app's other docs because their formats are self-explanatory, and you generally need to inspect just their brief summary reports displayed by TAIL. But they are a key part of using this system, and play a major role in recommended usage patterns.

For example, you should generally inspect the logfile of a SHOW command before running SYNC, especially when first starting out with this app. This logfile shows you in detail what a subsequent SYNC will do, and allows you to verify and approve the changes. While you can also roll back changes with UNDO, it's usually easier to eyeball a SHOW's logfile ahead of time.

As another guideline, you should also normally run a SHOW and inspect the difference counts at the end of its logfile after running a SYNC, to verify that FROM and TO are the same. These counts should all be zero if the two content copies were made the same by the SYNC. You can instead trust SYNC and skip SHOWs, but you may miss errors if you do.

A DIFF verifies more deeply, but taxes your drive, and is not required after every SYNC. Run one occasionally to confirm that SYNCs are working as expected, and to catch any content differences not noticed by SHOW's timestamp comparisons. Many such differences are normal and harmless as we'll see ahead, but some can reflect drive failures—along with the summary indicators described in the next section.

As a broadest rule of thumb: most errors reported in logfiles can be resolved by simply rerunning the SYNC or COPY action which reported errors, after resolving any issues requiring user attention. These actions always make two content copies the same, no matter what happened in prior runs. For more info on handling specific errors, though, read on.

Using Logfile Summaries

Because logfiles' closing summaries will be your main focus when using this app, this section calls out a handful of summary details which may warrant extra attention.

Notably, the logfiles made by SYNC include a notification line at the end whenever anything abnormal is reported earlier in the log. You should always check for this line after a SYNC, with the Logs tab's TAIL, WATCH, or OPEN. Though rare, SYNCs can go awry for reasons you may not expect or control, including drive failures. Here's the end of a logfile from one such event:

*Summary
Compared    => files: 3265, folders: 560, symlinks: 0
Differences => samefile: 256, uniqueto: 1528, uniquefrom: 2658, mixedmode: 2
Changed:
files   => created: 1890, deleted: 461, replaced: 257
folders => created: 69, deleted: 65, replaced: 1
**There are error messages in the log file above: see "**Error"

A corrupted drive, as in this example, will often register errors in a logfile, and may require fixing, reformatting, or replacing. In the case that produced the logfile snippet above, a power surge ruined the drive's index tables, triggering bogus SYNC updates that could be repaired by an UNDO, SYNC, or full COPY (the latter is required only if there is major damage).

As noted earlier, drive failures may also manifest as inexplicable file differences in DIFF (e.g., for bad sectors). While SSDs tend to be more reliable than flash drives, every drive has a limited life span—which is, of course, a primary reason for using this app to create multiple copies of your content. But logfile errors may be your first warning of drive demise.

That said, keep in mind that you can expect years of error-free service from almost any storage device, and most logfile indicators are not drive failures. Some may reflect easily repaired conditions—including permission errors for locked files (simply unlock and resync, per the extended coverage here), and unexpected drive unmounts (just rerun the sync to update files skipped)—and some indicators may stem from the known symlink limitations covered in the prior section.

Still other summary indicator lines are not errors at all. The end of a SYNC, SHOW, UNDO, or COPY logfile may also display a note indicator like this:

*Summary
...
**There are non-error notes in this logfile: see "**Note" above

These notes are informational messages that don't necessarily imply errors, but should be inspected nonetheless (e.g., the macOS Unicode skew covered ahead may crop up in logfiles as notes). DIFF's logfile tail doesn't list errors or notes, but simply summarizes mismatches found, or tells you there were none:

Diffs found: 84
- items UNIQUE at [/Users/me/MY-STUFF/Websites] - [/Volumes/T7/MY-STUFF/Websites]
...
Search for "*" above for more info

As a rule: if you see any of these indicator lines in summaries, open logfiles in full with OPEN and search them for more details. A search for "**" will locate both errors and notes after SYNC, SHOW, UNDO, and COPY; and a search for "*" will find more details on differences after DIFF. Some DIFF items may warrant single-file comparisons (e.g., with Windows fc, Unix diff, or this script), but others are expected and ignorable—per the next section.

Logfile Special Cases

Finally, you should also be aware of three special cases that may impact the results you'll see in logfiles:

DIFF: ignore backup-folder uniques
On all platforms, the UNDO backups stored by SYNC in top-level __bkp__ subfolders are skipped by SYNC and SHOW because they are not part of your content, but are not skipped by DIFF because it runs a complete compare. Hence, they will generally be reported as unique items in DIFF summaries alone, and can be safely ignored as normal differences.
DIFF: ignore older Excel-file differences
On both Windows and macOS, simply viewing an older Microsoft Excel file may change its content in trivial ways without updating its modified timestamp. The net result is that these files will not register as different in the timestamp-based SYNC and SHOW, but will register as different in the content-based DIFF. There's more about this rare phenomenon here.

These files do not differ in any user-visible way, and you can safely ignore their differences reported by DIFF alone. The covert content change behind them probably qualifies as an Excel bug, but there is no known fix. If you don't want to see the differences, delete these files in TO so they will be recopied on the next SYNC, or convert them to newer Excel formats.

SYNC: ignore covert Android mods to .nomedia files
By longstanding convention on Android, placing a file named .nomedia in a folder prevents the media system from crawling that folder. This is often used to stop image galleries from vacuuming up images you prefer not to contribute. Samsung's Gallery, for instance, will ignore images in a folder tree whose root level has a file like this, and not add them to automatically generated photo collections.

Unfortunately, recent Androids have begun silently changing the content of .nomedia files in shared storage—oddly inserting the file's folder path as a new line. This in turn causes these files to register as differences in SYNC, SHOW, and DIFF. This is mostly harmless and can be largely ignored: the file will be simply copied over anew on each sync. But this can throw sync summary counts off by one for each such file in your content tree.

Really, your files in your folders should never be silently changed by Android this way, but .nomedia are at least still respected by crawlers when they appear in your content. In the roots of special folders like Downloads, however, these files suffer a harsher fate: Android will now silently delete them with neither your permission nor your consent. (Insert outrage here.)

SYNC: ignore or avoid email-file recopies on Windows
On Windows only, files whose names end in a .eml extension are always erroneously copied with the date and time of the copy operation—not the date and time of the original file being copied. This happens even though the files' content has not been changed in any way, and reflects a glitch deep in Windows' system libraries.

The upshot here is that .eml files will always register as different in SYNC and SHOW (for timestamps), though not in DIFF (for content). Worse, this causes all such files to be recopied on every SYNC due to timestamp differences: because they are always recopied with the current date and time, this guarantees a spurious recopy on each later SYNC.

This reflects a longstanding Windows bug, which oddly special-cases .eml files in system libraries, and has not been addressed from Vista or 7 (depending on the accuracy of the web) through 11. The best explanation seems to be that Windows' NTFS filesystem stores non-content extra data used to identify files' origination points for security purposes; setting or changing this extra data wrongly changes files' content-modification time, and the default rules for .eml files—among others—invoke this behavior in low-level system calls used for copies.

You can read more about this Windows anomaly widely on the web; see this search. It arguably reflects an engineering kludge either naive or rude, but we'll leave the outrage to the web. Per results of the search link, you can generally avoid differences and recopies for these files by disabling indexing options for the .eml extension with this sequence or similar:

  1. Control Panel
  2. Indexing Options
  3. Advanced (button)
  4. File Types (tab)
  5. Uncheck eml file type (and click OK)

You can, of course, simply ignore the Windows effect instead. Since these files are by default recopied on each SYNC, they will be always current—if perpetually annoying!

For more info on handling errors, see the alternative take in the docs of the embedded Mergeall system.

Nested-Subfolder Syncs and Rollbacks

As suggested in the Usage Overview, you can sync your content folder in its entirety and/or sync nested subfolder located inside it. For example, you may wish to sync just a DCIM photos subfolder nested in your complete content folder; this can save time if you've only added or edited photos and haven't changed content in any other subfolders.

If you opt to do this, though, keep in mind that if you both sync subfolders and sync the whole content folder containing it, a sync of the whole content folder will either change or delete the __bkp__ backups folder in the nested subfolders you've synced separately. The net effect removes UNDO rollbacks as an option for subfolder, at least in terms of their former separately-synced content.

This stems from the fact that a SYNC's basic function is to make TO a mirror-image copy of FROM. When you sync the whole folder, it will make subfolders in TO mirror-image copies of the corresponding subfolders in FROM. This invalidates the subfolders' __bkp__ folders: they no longer apply to the subfolders' new mirror copies—and in fact will not work to restore the subfolders' content to what it was before the whole-folder sync.

Specifically, when a TO subfolder has its own __bkp__, a whole-folder sync will either delete it if there is none in FROM, or replace it with the same-named folder at the same location in FROM. If instead a FROM subfolder has a __bkp__ but TO does not, it will be copied to TO verbatim.

This effect makes sense if you remember that whole-folder syncs make TO's subfolders mirror-image copies of the same subfolders in FROM: in all cases, __bkp__ folders are propagated or removed so that they apply to the mirror copy created in TO by the sync. The former separately synced content is no longer relevant.

In sum: feel free to sync subfolders as you wish, but know that a sync to the content folder's root will make subfolders in TO match those in FROM, which may make it impossible to roll back recent subfolder changes separately. You can, of course roll back all the changes at the root level, which will also restore subfolder __bkp__ folders, but this is unlikely to be useful after you've synced at the content's root.

Also keep in mind that syncing subfolders syncs just those subfolders; if you forget that you've changed content in other subfolders too, their changes won't be propagated. Generally speaking, you're better off syncing your content folders as a whole.

USB-Drive Support on Android Phones

This app requires a removable drive in most roles, to store and transfer content and changes. As alternatives, network shares work in this app on PCs, but not on phones, and are much slower and less secure; microSD drives embedded in phones work too, but have grown too rare for emphasis in this app's docs; and MTP direct connect is right out. Especially for Android, a removable drive attached by USB port is the most common usage mode.

Most phones produced in the last few years support both removeable drives on a USB port, as well as the portable exFAT filesystem recommended for this app. Most USB components available today also support the required OTG protocol, which allows phones to switch host/peripheral roles (for drive/PC connection). Given the wide variability in the Android world, however, these are not universals.

Case in point: the 2020 budget-level Pixel4a phone does not support removable USB drives with its stock Android 11, even if you grant the app the requisite All Files Access permission. This reflects a bug in the underlying Android 11 shipped on this phone.

The good news is that upgrading the Pixel4a to either Android 12 or Android 13 (as is essentially forced by Google) enables USB access in full in this app. Still, it's not impossible that some other phones may have similar constraints, especially if they are older phones that have not been upgraded to newer Android releases.

As a rule, be sure to vet this app to verify that your phone supports USB access. This is especially true for older and functionally limited phones. It's impossible to test every vendor's phone in use, but newer is generally better when it comes to USB support—as well as most other aspects of the Android user experience.

Even with Android 12 or 13 upgrades, the Pixel4a supports the FAT32 filesystem on removable drives, but not the recommended exFAT. While FAT32 works in this app and is widely available on nearly all devices with a USB port, it has substantial limitations, especially when it comes to timestamp-based syncs like that use in this app. For more details, read on to the next note.

Why You Should Avoid FAT32—or Not

exFAT is now part of the Linux kernal embedded in Android, and hence should be supported on recent Android phones (inluding Pixels). If you can use drives formatted with this filesysem on your phone, you are encouraged to do so, and exFAT users can safely skip the rest of this note.

However, if you must use FAT32-formatted removable drives on your phone (as in the prior note's Pixel4a example), be sure to see the Mergeall coverage here of issues that can arise when your devices' time changes due to Daylight-Saving-Time rollovers or time-zone changes.

In short, because FAT32 records time as absolute "local" time, its timestamps can trigger spurious differences on syncs. If not addressed, these in turn can invoke full and pointless recopies of your content folders, because every file's modification time erroneously differs. To avoid this, either use exFAT where available, or run the simple and free command-line script available here to adjust content timestamps when needed; see the script for more info.

The recommended exFAT filesystem uses timestamps that are relative to a fixed date, and so are immune to issues when device times change. FAT32 also suffers from an archaic 4G filesize limitation not present in exFAT. In fact, FAT32 is so limited by today's standards, that you may want to consider upgrading your phone or its Android to lift the FAT32 constraint.

Exception: on older Samsung devices running Androids 8 and 9, there was a known timestamp bug which made it impossible to use exFAT for sync programs like this app. In short, the bug botched modification times on exFAT drives; which caused all files to be wrongly classified as changed; which triggered spurious copies of the entire content folder. This bug was fixed in Samsung's Android 10, so users of 10 and later are immune and free to use exFAT. If you're still using some older phones, however, you may need to use FAT32 for removable drives in this app if its SYNC or SHOW oddly classify all files as differing; full details.

When Good Phones Go Bad: App Kills

This app's About tab includes a caution about battery-optimization settings when run on Android. In brief, some phones aggressively kill running apps to conserve battery power, and this might have consequences for this app's long-running actions. This kill may occur due to excessive CPU usage, or more arbitrary culls, but it's a fact of life on some phones.

It's important to note that this app by default runs actions as officially sanctioned Android foreground services, which should be immune to most reaping. It also offers a thread-mode alternative that is likely immune to process-killing logic added in Android 12. Moreover, no actions kills have been seen or reported for this app to date, despite testing and usage that has included multi-hour action runs which process 200G content folders having nearly 200K files. While that probably makes this a non-issue for this app, Android's variability again makes absolute statements perilous.

Though unlikely, if this app is ever killed inexplicably on your phone, your first recourse is to disable all battery optimizations for this app; they're generally in Settings=>Apps=>this app, and go by various names on different phones. It may also help to run this app's actions as threads instead of services to avoid 12+ process killers (see the Config tab), though CPU usage may be classified similarly either way.

Again, there are no known kill issues for this app, but some vendors more ruthlessly limit programs than others. In the end, optimizing battery longevity at the expense of basic phone functionality is a very slippery slope, and you should probably consider a vendor's track record in this department before purchasing a phone. Devices that reliably support only short-lived tasks aren't as useful as their marketing may imply.

Related: in addition to kills, some Android devices may also temporarily stop apps to conserve battery power. On Android, this app also uses a partial wake lock during Main-tab actions, to keep the CPU (and hence the app) running when the screen turns off due to a user-specified timeout or power-button press. Even so, disabling battery optimization for this app in Settings may also prevent power-related app stops. While this story is convoluted by Android history and variability, the disable is a generally recommended antidote for both kills and stops. See also: notes about app speed and throttling here.

A macOS Filename Portability Pitfall

This note applies only to users who will sync content to a macOS PC, and use non-ASCII filenames with characters that have multiple representations in the Unicode standard. If that set doesn't include you, you can happily skip this subtle story. Else, either read on for more info now, or file this for future reference in case your Unicode names ever fail on macOS, in this app or otherwise.

The Short Version

If you propagate content from and to macOS, you may run into a rare portability peril for filenames that contain non-ASCII Unicode variant characters. This reflects a Unicode hurdle imposed by macOS, and seems likely to be a problem only in roundtrips from and to macOS through exFAT drives. Moreover, most transfers seem immune to the drama: this may crop up only if filenames formerly exported by macOS are recopied on Windows from scratch.

That's an atypical usage context to be sure, but if non-ASCII filenames on exFAT drives fail and are skipped with logfile errors when exporting to macOS, you can fix them by simply running a provided script on Windows available here and covered further ahead.

Okay, but What's a Unicode Variant?

To truly understand the peril, we first need to take a short side trip to the land of Unicode. One of the goals of the Unicode standard is to uniquely identify characters with sequences of one or more integers called code points. Oddly, though, this standard allows a small set of characters, especially those with accents or other augmentations, to be represented by multiple, different code-point sequences. These different-but-same representations as generally known as Unicode variants.

For example, the character can be represented as either the single code-point 196, or the two code-point sequence 65 and 776 (when encoded as UTF-8, these are two and three bytes, respectively: \xc3\x84 and A\xcc\x88). The first of these is known as the composed form, which goes by acronym NFC; the second is called decomposed (i.e., accents separate), and abbreviated NFD. Though they differ, both forms are defined to stand for the same character per standard: . For programmers, here's the story in a Python 3.X shell on macOS:

>>> x = chr(196)                # NFC
>>> y = chr(65) + chr(776)      # NFD
>>> x, y
('Ä', 'Ä')
>>> len(x), len(y)              # same-but-different code points
(1, 2)
>>> x == y
False

# encoded (stored) forms differ too
>>> x.encode('utf8'), y.encode('utf8')
(b'\xc3\x84', b'A\xcc\x88')
>>> len(x.encode('utf8')), len(y.encode('utf8'))
(2, 3)

Naturally, a standard that uniquely identifies characters with identifiers that are not unique can create interoperability problems. If a variant character stored in one form is transferred to a system that mandates another, you have a built-in portability issue (example impacts here). Most platforms in common use prefer and create NFC, but leniently allow NFD too. macOS, however, is the exception: it mostly mandates NFD, and even refuses to recognize NFD on exFAT in one context described next.

Why It May (and May Not) Matter

To see how all this can cause problems, you also need to understand the convoluted behavior of exFAT on macOS. When you store filenames with Unicode variant characters on exFAT, they adhere to the following rules—at least as tested on macOS Catalina and Windows 11 (Android likely mirrors Windows, though this remains to be verified and may vary by storage type):

The last point is what can lead to interoperability crises. Both NFD files created on macOS, and NFC files created on Windows can be propagated between macOS and Windows without problems: Windows allows macOS's NFD, and macOS allows Windows' NFC.

NFD files created anew on Windows, however, won't be recognized by macOS on exFAT drives, for reasons known only to that platform's exFAT driver. Since Windows won't generally create NFD names itself, this normally arises only when Windows recopies an NFD file passed to it previously from macOS. That is, a recopy on Windows of content that originated on macOS may trigger errors.

macOS Unicode Workarounds

This is an unlikely use case (and arguably qualifies as a macOS bug), but may arise if you restore macOS content from a Windows copy. If you ever run into it, the solution is to convert any NFD filenames to NFC on Windows prior to propagation, so they will be recognized on macOS.

There are a number of ways to convert, but to make this simple, this app supports Unicode filename conversions with a free utility script available online here. You must run it on Windows (macOS can't process the offending NFD files, after all), but it's a one-time command line. See the script for more details; it includes a usage example at its end.

Finally, if you really want to ensure pain-free portability for your content, the simplest route is to stick to ASCII characters for your filenames to fully avoid macOS issues. While you're at it, be sure to also avoid characters in the set rendered nonportable by Windows, a similar but less subtle and more widespread pitfall addressed by this app's NAME.

The all-ASCII solution may not be ideal if your native language includes non-ASCII characters which the Unicode standard failed to standardize. But it will spare you from the drama generated by computer vendors that seem to view proprietary lock-in as valid business plan—and customers like us as collateral damage.

Linux Library-Skew Workaround

This app's Linux package is a simple executable that was built on Ubuntu 20 (originally) with the PyInstaller system. As noted in the package guide, it is expected that this package may have issues on some Linux installs due to the variability of Linux libraries and the policies of the build tool.

One such issue has now emerged: like other programs based on the Kivy GUI toolkit or OpenGL and built with PyInstaller, this app's Linux executable fails to run out of the box on the latest Ubuntu 22 (specifically, on a newly installed or upgraded Ubuntu 22.04.3 LTS). The app's tkinter-based splash screen appears, but its full Kivy-based GUI does not.

The thorny technical cause for this is described near the end of the PyInstaller docs here. In brief, PyInstaller bundles some system libraries but not others, and both sets are prone to vary among Linux installs.

To fix this, simply copy/paste the following into the end of your ~/.profile or similar start-up file on Linux:

# fix PPUS on Ubuntu 22 (and others?)
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6

This forces the library linker to use the correct tools. With it, the PC-Phone USB Sync executable works well on Ubuntu 22, and possibly other Linux installs. It's not required (and may be ill-advised) on Ubuntu 20, the executable's build platform. You can also add the Bash code above to a custom launcher script, if you don't want it to apply to every program.

For more background info, search the web like this for the following error message—which you'll be able to see if you run the Linux executable directly from a command line in the Terminal shell, before the fix:

libGL error: MESA-LOADER: failed to open iris: /usr/lib/dri/iris_dri.so

A build for Ubuntu 22 may appear here soon, if it resolves the Kivy library issue without requiring the workaround above. Still, library issues like this are typical for Linux, and even expected for a simple executable like that of this app, and a single build can't accommodate every install. If the app's executable won't work for you, search the web for shell error messages, view the related Linux tip, and see the App Packages guide for the Mergeall GUI's source-code alternative you can use to sync on Linux.

Update: The Linux version of this app was later rebuilt on Ubuntu 22, and made available on this site. This build does not require the export workaround covered above when run on Ubuntu 22. Because it may not support older Linux installs, however, the new build is provided as an alternative to the original Ubuntu 20 build, and the workaround above may still be useful on some Linux installs.

Related: There is also a minor issue with Tcl/Tk library skew for some text editors launched with the app's OPEN. This does not apply to text editors not based on Tcl/Tk, including Ubuntu's default TextEditor, and can be addressed by changing Tcl/Tk libraries, associating a different editor to open text files, or using the app's EXPLORE instead of OPEN. The current theory is that the app's PyInstaller splash screen, based on Tcl/Tk, botches library search paths; more info as it emerges.

Windows WSL2 Linux: Marginal but Hopeful

Version 2 of the Windows Subsystem for Linux—WSL2—recently added the potentially game-changing ability to run Linux GUI applications via X-Windows/Wayland support. This extension is formally known as WSLg, and is documented by Microsoft here.

In principle, this means that the Linux version of this app, PC-Phone USB Sync, could be run from the WSL2 environment on Windows, enabling a more Linux-focused experience than the app's Windows version. For instance, folder choosers would show common Linux (not Windows) roots, file views would open Linux viewers, and content syncs could be run to and from Linux's fast native filesystems.

In practice, though, this works so poorly at this writing that it is not generally recommended. WSL2 is hobbled by both quality and performance issues that can make it an exercise in frustration for users of GUI apps. Most users of Windows computers would be better served by running this app's Windows version instead, and syncing WSL2 folders by accessing them as mapped drives within the Windows app.

That advice is based on vetting of WSL2 in February 2024. WSL2 is naturally prone to improve over time, and even in its present form may suffice for strictly command-line tasks (e.g., app builds sans GUI tests). Moreover, none of this note applies to either Android or stand-alone Linux outside WSL2. Today, though, WSL2 seems beta-level software that's not viable as a host for this app, and is undoubtedly subpar for other Linux GUI programs too.

If you want to jump right into visuals, you can find screenshots of WSL2 usage on both Windows and WSL2 itself in this note's extra galleries. The rest of this note lays out the WSL2 story on both platforms.

Using the Windows App for WSL2 Content

Before digging into this app's Linux version on WSL2, it is important to note that this mode is optional, and may even be superfluous. For most roles, Windows users are much better off running the Windows version of this app, and using it to sync content stored on WSL2's filesystem.

To make this work, simply install this app's Windows version available here, and mount WSL2's network location to a drive letter in Windows Explorer. Use \\wsl.localhost (or \\wsl$) for the WSL2 network location when mapping in Explorer; this makes it visible to the app as well as every other program on Windows.

Once so mapped, WSL2 content will show up as storage-type root (Remote) in the app's Main-tab FROM and TO folder choosers on Windows, like this. When selected, this root will resolve to path \\wsl.localhost, which provides access to your entire WSL2 Linux distribution (technically, this root corresponds to / at the top of WSL2's Linux-native file hierarchy).

Hence, you'll easily be able to use the Windows app to sync content both to and from any location nested in your:

In the Windows app, your Linux home folder is located in home within the (Remote) Linux root, and unlabeled dives are labeled (Removable) in the app's chooser. There's generally no reason to run the app in WSL2, when its tried-and-true Windows version offers all this functionality without seams (and usually with better speed: watch for details ahead). For screenshots of this recommended mode in action, see the full Windows WSL2 gallery.

Using the Linux App in WSL2 Itself

If you're satisfied with the foregoing recommendation, you can happily ignore the rest of this note. But if you're a Linux user interested in WSL2 GUIs and up for a challenge, the rest of this note describes current setup and usage for the app in WSL2. This mode may be useful today for users who can tolerate a set of quirks and wish to sync content on Linux filesystems alone; as we'll see, such syncs will run quicker in WSL2 itself. For other roles, this mode is neither practical nor suggested, but this coverage may help you decide whether WSL2 is worth the effort.

Install Requirements

Getting the Linux app to run in WSL2 at all is the usual slog, but no worse than you'd probably expect for a mixed-platform environment that includes Linux. Here are the salient points of the setup process's one-time steps:

WSL2 and Linux
First off you must install (or upgrade to) the latest WSL2 in Windows, and install (or upgrade to) an Ubuntu Linux distribution within WSL2. Ubuntu 22 is recommended today because that's the most recent app target, but later versions may work too. For pointers on this process, see the web; this generally entails running commands in both Windows and WSL2-Linux terminals, but is beyond this note's scope.
Graphics driver
At this point, the app will launch, but its display will probably be unusable. To fix, you must install a new graphics driver with vGPU support, per Microsoft's docs and links here. This is risky, because it replaces your vendor-installed driver, and is global to your computer (i.e., Windows programs may be impacted too); it seems safe in usage thus far, but the standard "installer beware" applies.
Tcl and Tk
After that, you must manually install Tcl and Tk in Ubuntu, because they are required by the app's splash screen, and are not present by default in WSL2. To install, simply open a WSL2 terminal and run sudo apt install Tcl, and likewise for Tk.

This step stems from the fact that the PyInstaller tool used to build the Linux app uses Tk for its splash screen, and assumes standard Linux libraries are present instead of bundling them with the app; WSL2 is an exception on this and other fronts. The app could forego the splash screen for WSL2's sake, but this would require a custom build for WSL2, and incur a silent wait of up to 10 seconds on each launch while the app unzips itself.

xdg-utils for xdg-open, plus viewers
Next, you must manually install the xdg-utils package in Ubuntu. This package's xdg-open command is required to launch text editors and file explorers for the app's OPEN and EXPLORE actions, and this standard Linux command is not present by default in WSL2 (yes, a pattern is emerging). To install, open a WSL2 terminal and run sudo apt install xdg-utils.

Once installed, OPEN and EXPLORE will launch Linux viewers. By default, OPEN spawns vi (in the console) and EXPLORE launches Firefox (if installed in WSL2), which is arguably subpar. For a more GUI experience, also install gedit, nautilus, or similar editors and explorers; like xdg-open, these are not preinstalled on WSL2. While the app could instead run Windows Explorer for views in WSL2, Linux users likely expect to use Linux viewers.

PC-Phone USB Sync app
Finally and naturally, you must install the Linux version of this app available here. You can unzip it on either the Windows or Linux filesystem; the former is available in WSL2 as /mnt/c, but the latter is faster within WSL2 (a claim we'll prove ahead). Be sure to get the Ubuntu 22 version, for its mods coming up in the next section.

General Issues

Once you've jumped through the preceding install hoops, the Linux app does run if launched from WSL2 (e.g., by command-line invocation of the unzipped executable), and can be used to sync content both to and from any location nested in your:

In the Linux app on WSL2, your Windows system drive is automatically mounted by WSL2 as /mnt/c, and displayed by the app's folder choosers as label c; your Windows home folder is nested in this root's Users, like this. Other drives are displayed in choosers if they are mounted in /media or /mnt. For screenshots of what this looks like, see the complete Linux WSL2 gallery.

Despite the pictures, though, the app is marginal in this mode at best. It suffers from multiple glitches and quirks, some of which may be fatal to common app use cases in WSL2, and none of which are present in either the Windows app running on Windows, or the Linux app running on stand-alone Linux. Here's a rundown of the general issues uncovered so far:

The GUI may be too slow to use
First and perhaps foremost, the app's GUI is often painfully slow in WSL2. On a relatively recent Dell laptop with the required Intel graphics driver, tabs-bar scrolls can noticeably stutter, and text displays are so sluggish that intermediate rendering stages for new text are often visible. Worse, resizing the app's window invokes a snail-paced process, which shifts the window a few millimeters further each second—and renders the app unresponsive during the agonizingly long wait. Watch the excruciating effect in this video, if you have a moment to spare. Your machine's speed (and your patience) may vary, but this obviously does not qualify as production-level GUI support, and makes the app tedious to use. See also GUI-speed update ahead for more on this front.
A visual glitch may render the GUI useless
Though more intermittent than its speed problems, a click on the app's GUI in WSL2 sometimes triggers a major visual glitch that makes the app fully unusable. See the effect in the screenshots starting here; this state can persist through multiple clicks, and might just pass as modern art if it wasn't so crippling. This happened on multiple app runs spanning WSL2 restarts, and on the same recent Dell laptop with the prescribed Intel graphics driver. Though some machines may be immune, it's a widespread WSL2 bug; see the formal report. A future WSL2 may fix this, but it's arguably lethal when it appears today; your only recourse is to shut down and restart WSL2 and hope for better fortune.
Normal text scrolls are iffy: try grab-and-drag
For reasons unknown, scrolling text in the GUI with a touchpad swipe or mouse scroll-wheel spin almost never works in WSL2. This impacts logfile text in the Logs tab, as well as info text in Help and About: scrolls go nowhere in these if run with normal gestures. Fortunately, grab-and-drag (click on the text and move it while holding down on the selection) does still work to scroll, but is less convenient and intuitive, and can sometimes become ineffectual after initial scrolls. Exact causes remain to be isolated, but this is likely outside the scope of the app itself. For now, text does scroll in the app on WSL2, but only by grab-and-drag, and often too poorly to count. To skip some of the drama, use Logs' OPEN to view logfiles.
Web links work but can be very slow
Happily, web links in the app's GUI correctly open linked pages without extra installs, because they use Python's webbrowser instead of Linux's xdg-open. Less happily, web links can take up to 10 seconds to do so in the installed WSL2 Firefox; this may reflect inherent sloth in WSL2, and has no workaround if so (apart, of course, from using the Windows app or stand-alone Linux).
Some glitches were fixed by Linux-app mods
To make the app at least marginally useful on WSL2, the Linux version of this app (Ubuntu-22 build only) was rereleased in late February 2024 with the following mods. They fix problems unique to WSL2 and are harmless elsewhere, but may warrant attention in other GUIs targeting WSL2:

The good news is that the Linux app is somewhat usable on WSL2 with the mods listed above, if you're willing to tolerate GUI sloth and occasional GUI glitches, and use grab-and-drag (-and-hope) for scrolls. The bad news is that there's more bad news coming up in the next section.

GUI-speed update: after further vetting, other GUIs on WSL2 do not seem to be impacted by the GUI plodding that hampers this app. Notably, window resizes in both the gedit text editor and nautilus file explorer run quickly, and without lags. Hence, the slowdowns are likely triggered by code in the Kivy GUI toolkit which this app uses, the SDL library which Kivy uses, or the OpenGL system at the bottom.

That said, this remains a WSL2-only GUI defect, and can be fairly classified as a WSL2 bug. This app has no such issues on multiple flavors of Windows, macOS, Android, and stand-alone Linux. While this bug may eventually be worked around by changes in a component of the app's software stack, it's just one of multiple GUI issues that impede usage today on WSL2 alone.

Storage Issues

While the prior section's list may be enough to disqualify WSL2 to some users, its storage issues may be worse. In short, removable drives attached after WSL2 starts up are not available; some removable drives may never be available in WSL2; and removable drives will likely be too slow to use in WSL2 even when they appear. None of these can be remedied by changing WSL2 config-file settings, and obscure manual workarounds, if any may be, are not acceptable for a system promoted as a replacement for stand-alone desktop Linux. Here are the sordid details:

Drives are noticed only at start-up time
Attached removable drives are recognized and mounted by WSL2 only at its start-up time. Attaching one after WSL2 is already running goes wholly unnoticed, and the drive won't appear (or disappear) in the app's folder choosers. There are potential workarounds on the web, but most seem iffy hacks. Barring a fix, you'll have to wsl --shutdown in Windows and restart both WSL2's console and this app every time you attach or detach a removable drive—and burn about a minute each time you perform this ritual. This feels onerous and archaic; drives really should be automatically mounted in 2024, shouldn't they?
Some drives are not fully mounted
Even after shutting down and restarting WSL2, some drives may not appear in the app's chooser, because they are not fully mounted. In one case, an SSD attached by USB and mounted in Windows did not appear in the Linux mount command's output after restarting. Strangely, it did show up as a subfolder in /mnt named for its Window's drive letter (alongside a drive mounted correctly), but this folder was empty, so navigating to it from ROOT (/) didn't help. An additional WSL2 shut-down and restart seems to fix this limbo state, but is far too much to ask of users.
Some drives never show up at all
No matter how many shut-downs and restarts are run, some drives will never show up the app's chooser—or anywhere else in WSL2. For example, a camera attached by USB, which Windows and stand-alone Linux both happily mount, has no presence in Linux's mount or filesystem, and neither navigation from the app's ROOT folder nor additional restarts will help. This camera also has no label, but does appear in both the Windows app, at path D: and labeled (Removable); and the Linux app outside WSL2, at path /media/user/serial and labeled by serial number (more here). Whatever the cause, this camera is a permanent no-show in WSL2 alone.
Drives lose their human-friendly labels
For those removable drives that are noticed by WSL2 at start-up time, any meaningful drive labels are absent in choosers: removables are labeled only by Windows mapped letter (e.g., by label d for a drive mounted at path /mnt/d on start-up). This reflects the automatic mount of such drives in WSL2, and there is no simple workaround (and no, manual remounts don't qualify as a reasonable fix for a modern desktop system).
Non-Linux storage may be unusably slow
The speed of some types of storage is abysmal in WSL2. As an example, the same SHOW action run on the same removable-drive folder took just 1 second in the Windows version of the app, but a whopping 84 seconds in the Linux version in WSL2 (see the evidence). While WSL2 may fare better for Linux-native folders (this was its marquee improvement in version 2, after all), most use cases require the app to process content on removable drives too, and will thus incur similar speed hits. See also Storage-speed update ahead for more-comprehensive testing results.
Drives used in WSL2 show suspicious corruption
Disturbingly, an exFAT removable drive read (but not written) by the app in WSL2 suffered unexplainable corruption, and required a sync to restore. While random drive failures can and do happen on their own, they are quite rare, and WSL2 seems the more likely culprit. Moreover, WSL2 exhibits other system-state anomalies that make its role in disk errors more plausible (e.g., file locks may outlive WSL2 sessions, phantom WSL2 sessions may hoard memory, and Windows sleep sans WSL2 shutdown is perilous). See related reports on the web here and here. This remains to be verified, but trashing drives would obviously kick WSL2 out of the practical club.

Caveat: all of this section's issues reflect a single encounter with WSL2, and additional problems seem likely to be uncovered with further use (e.g., storage speed is more subtle than initially deemed per the following update, and the psychedelic graphics distortions inexplicably returned on more than one revisit). That said, the results listed above seem already bad enough to render a verdict.

Storage-speed update: per later testing in February 2024, using the native Linux filesystem is indeed very fast in the app on WSL2, but this applies only to atypical all-Linux use cases. Syncing native Linux folders with either Windows or removable-drive folders incurs massive speed hits, and this is required by most of this app's roles. Specifically, for a SHOW action that compared the same folder on the host's native filesystem to identical copies elsewhere, the two apps' average times over multiple runs were measured as follows:

Takeaway: if you're going to sync WSL2 Linux content to or from Windows or removable-drive folders, using the Windows app with mapped Linux drives is likely to be your fastest option today. In the test, for example, using WSL2 folders in Windows took 21 seconds, but using Windows folders in WSL2 took 44, and using removable drives was radically faster in Windows (3 seconds in Windows versus 44 in WSL2).

For detail-minded readers, the 1-versus-84 second finding noted earlier reflects a worst-case combination of speed hits for accessing both Windows and removable drives in WSL2: WSL2 is roughly 40 + 40 seconds slower than Windows in this mode. More-typical syncs in WSL2 pairing Linux and non-Linux filesystems will post a gain from the former that's easily outstripped by a loss from the latter (40 seconds in the test, and worse on larger content).

Also note that SHOW actions simply compare folders by structure, timestamps, and size. Though untimed, SYNC's content updates probably perform at least as poorly in WSL2 for all non-Linux storage, and will incur SHOW's comparison times as an initial step in any event.

These findings reflect WSL2's implementation (stand-alone Linux's speed naturally differs), and may not be a concern for some users. Especially for substantial content collections, though, WSL2's noble mixed-platform vision unfortunately seems yet to be realized.

WSL2 Wrap-Up

To be fair, some of the preceding issues might be addressed by additional mods to this app's Linux version for WSL2 usage. For instance, the app's splash screen could be dropped on Linux to remove Tcl/Tk dependencies that also cause issues elsewhere, and text scrolling's breakage feels reminiscent of other Kivy GUI glitches that were worked around in app code.

Even so, some WSL2 issues are outright showstoppers that are beyond the app's power to address, and make further app mods difficult to justify. In sum:

Indeed, WSL2's GUI support feels like it probably shouldn't have been released in its present form, and its storage issues generally limit its scope to Linux-only use cases. We're left with a command-line Linux that doesn't play well with the broader world, and falls far short of rendering stand-alone Linux alternatives obsolete.

The upside is that WSL2 is still new, and will likely improve with time. Its cross-platform file sloth, for example, is a widely known defect that's getting focus like this. WSL2's GUI support is even newer, but its problems are already being escalated, per the graphics glitch and speed reports here and here. And despite the warts, some tenacious Linux users might judge WSL2's problems an even trade for the sake of faster Linux filesystems.

But unless and until this story changes, you're probably better off either sticking with stand-alone Linux, or using the well-worn Windows version of this app to sync content stored on a mix of Windows, WSL2, and removable-drive filesystems. Your syncs will run faster, and your GUI will work better.

For More Info

For additional info on installing and using this app, see the main sections of its User Guide.