![[image]](usbsync.png)
This page hosts technical notes for technical readers. It's a random collection of more-advanced topics related to using the PC-Phone USB Sync app. This page by nature is also a work in progress; please check back occasionally for new usage notes here. Current contents:
Symlinks are a special type of file that redirect accesses to other files or folders. They're not commonly used, but when they are, they allow content to be shared instead of duplicated. They're also not what some platforms call aliases or shortcuts (which are really features of the GUI, not the filesystem), and if you don't know what they are, you probably don't need to read the rest of this note.
But if you use symlinks, you need to know that they are a special case in this app. As mentioned briefly in the Usage Overview, symlinks are not portable across all devices and platforms, and your best option is to omit them in content folders managed by this app. The full story on this both subtle and platform specific, and merits extra coverage here.
Short story: symlinks stored on macOS will survive roundtrips to platforms that don't support them, as long as you don't change the stub files macOS makes to smuggle them across device boundaries. No workarounds are required in this limited macOS-only role, but you and shouldn't expect your macOS symlinks to work on other platforms, and other use cases are more gray.
Details: macOS automatically and internally forges symlinks both to and from stub text files, whenever they are written to removable drives that don't naturally support them—including those using the portable exFAT and FAT32 filesystems. When these stubs are processed on Windows or Linux, they are seen as simple text files instead of links, and recorded there that way.
For example, here's what a stub file written for a symlink on macOS looks like on an exFAT drive when viewed on Windows; macOS converts this back to a symlink when it's later accessed, but Windows treats it as a simple text file:
XSym 0015 18eeefc3e8a24afd5f6beaf5f1e0b3ec folder/file.txt
Like many acts of covert hackery, the macOS symlink forgery is a mixed bag. Among its upsides:
The downside to macOS's symlink forgery is that it's only skin-deep: if you ever try to use the stub files outside of macOS's scope, they will be simple text files, and this may have consequences that are both unwanted and unexpected.
For instance, serving a folder with macOS stubs from a network server (e.g., WebDAV or Samba) may serve stubs as simple text files—which will likely yield mixed-type mismatches with the corresponding symlinks back on macOS. The net effect will generally replace your macOS symlinks with simple files during a cross-network sync, thereby destroying them. Yes, surprise!
On the whole, macOS's stubs files mean that you can use its symlinks in your content folders on macOS, without losing them in sync roundtrips with this app. But you should be aware of the potential for unforeseen and dire consequences in other contexts, and shouldn't expect your symlinks to have any meaning outside the macOS realm.
Short story: symlinks stored on Linux don't work in this app, unless you use removable drives formatted with filesystems that won't generally work on non-Linux devices. For portable usage, your options in this app are to omit symlinks from your content folders; use a provided script to restore them after roundtrip syncs; or zip and unzip folders containing them for transit. Omitting is easiest.
Details: Linux throws errors whenever you try to copy symlinks to drives that do not support them (e.g., exFAT and FAT32). During syncs, these errors are reported in this app's action logfiles, and the failing symlinks are skipped. Because this omits symlinks in content copies, your Linux symlinks won't survive sync roundtrips: syncing back from a symlink-less drive to Linux will classify symlinks on Linux as unique-TO items, and delete them as part of making TO match the FROM drive.
For reference, here's what a symlink error looks like in logfiles, when Python-coded folder syncs and copies attempt to create a symlink on an exFAT drive on Linux—and ultimately skip it altogether (look for an error indicator in logfile summaries):
propagating symlink ./file-link **Error copying FROM file: skipped ./file-link[Errno 38] Function not implemented: 'file' -> '/media/me/T7/_temp/test-symlinks-cpall-linux/file-link'
Unfortunately, creating stub files on Linux as on macOS won't help. They would be seen as simple text files when synced back to Linux, producing mixed-type mismatches that would overwrite and replace your Linux symlinks.
The stub trick works on macOS only because its low-level system libraries map stubs back to links. No such mapping happens on Linux, and there's no reliable way to distinguish manual stub files from other use cases. Users may really want to replace a link with a plain file, and no special file contents is guaranteed to uniquely identify a stub. (To be fair, macOS's stub trick isn't foolproof either, but we'll leave the judgement aside here.) And while this app could skip symlinks both in FROM and TO much like cruft, this would discard part of your content, and preclude using them in contexts that support them.
Workarounds: if your content folders contain symlinks on Linux, there are three ways you can address them when using this app:
__bkp__ change
backups to find all removed symlinks to put back. It's run from a
Terminal command line with the content folder's path as a sole argument;
but if you use symlinks, you probably already know how to run a Python
script from a command line.
In the end, omitting symlinks from content folders destined for use across devices and platforms is your safest and simplest option. Restores and unzips can help if you must sync symlinks, but they add more manual steps than most content collections should impose.
Short story: Windows' symlinks-portability story is the same as on Linux. Reread the preceding section replacing every "Linux" with "Windows" and you're mostly set. Just as on Linux, your options in this app are to omit symlinks; use a script to restore them after roundtrip syncs; or zip and unzip folders containing them for transit. Omitting is easiest.
Details: symlinks are even rarer on Windows than Unix (i.e., macOS and Linux) due to historically limited support, but Windows 11 allows symlinks to be used more freely than its predecessors (in short, admin permission is now largely optional). Even so, symlink writes to portable removable drives will fail with error messages on Windows as on Linux, leaving holes that trigger symlinks removals if you ever sync back from removable drive to Windows.
Technically, the fate of your Windows symlinks depends on the tools you use to try exporting them. Explorer copies will simply follow symlinks and wrongly copy what they point to, while Python code like that used by this app will try to copy the link itself and generate a Windows error message; in copies and syncs, the net result skips the symlink, which means it will be unique and hence deleted on Windows in later back syncs. Here's a logfile error message triggered by a symlink copy to exFAT on Windows:
propagating symlink test-symlinks\link-file.txt **Error copying FROM file: skipped test-symlinks\link-file.txt[WinError 1] Incorrect function: 'file.txt' -> 'd:\\_temp\\test-symlinks-cpall\\link-file.txt'
Workarounds: as also on Linux, you can use the same provided restore script to put back your symlinks on Windows after a sync removes them; zip and unzip folders containing symlinks for transit as needed with ziptools; or simply keep symlinks out of your synced content folders. See this script for help with the latter and simpler option.
Short story: you cannot create symlinks in either the shared (PHONE) or app-specific (APP) storage-type folders made available in this app, so they are a moot point on Android: there is nothing to copy or preserve.
Details: for the record, it is possible to make symlinks on Android in app-private storage (Android is Linux, though a locked down flavor of it), but this storage cannot be accessed directly by any other app than its owner, and it's not exposed or discussed by this app for that reason. Sans a rooted phone, symlinks are a nonstarter on Android for content folders.
All that said, this app could be changed to skip symlinks in both FROM and TO as an option, if there is demand for this mod. This is not implemented today, because it would discard part of your content; complicate the app's user experience for a very-rarely-used tool; throw off content counts used to validate syncs; and either preclude or convolute symlinks in same-platform use cases that would otherwise support them. If you use symlinks and have an opinion on this, please register it with the hamburger menu's Contact; this app's goal is to satisfy customer needs, not theoretical whims.
Related point: filenames in your content may also be nonportable, but the app's NAME action can be used to make them work across all platforms automatically; see the GUI Guide for more info. There's no such tool for symlinks, because they cannot be fixed without deleting them from your content altogether—which seems dangerous enough to warrant manual intervention.
Logfiles are not covered much in this app's other docs because their formats are self-explanatory, and you generally need to inspect just their brief summary reports displayed by TAIL. But they are a key part of using this system, and play a major role in recommended usage patterns.
For example, you should generally inspect the logfile of a SHOW command before running SYNC, especially when first starting out with this app. This logfile shows you in detail what a subsequent SYNC will do, and allows you to verify and approve the changes. While you can also roll back changes with UNDO, it's usually easier to eyeball a SHOW's logfile ahead of time.
As another guideline, you should also normally run a SHOW and inspect the difference counts at the end of its logfile after running a SYNC, to verify that FROM and TO are the same. These counts should all be zero if the two content copies were made the same by the SYNC. You can instead trust SYNC and skip SHOWs, but you may miss errors if you do.
A DIFF verifies more deeply, but taxes your drive, and is not required after every SYNC. Run one occasionally to confirm that SYNCs are working as expected, and to catch any content differences not noticed by SHOW's timestamp comparisons. Many such differences are normal and harmless as we'll see ahead, but some can reflect drive failures—along with the summary indicators described in the next section.
Because logfiles' closing summaries will be your main focus when using this app, this section calls out a handful of summary details which may warrant extra attention.
Notably, the logfiles made by SYNC include a notification line at the end whenever anything abnormal is reported earlier in the log. You should always check for this line after a SYNC, with the Logs tab's TAIL, WATCH, or OPEN. Though rare, SYNCs can go awry for reasons you may not expect or control, including drive failures. Here's the end of a logfile from one such event:
*Summary Compared => files: 3265, folders: 560, symlinks: 0 Differences => samefile: 256, uniqueto: 1528, uniquefrom: 2658, mixedmode: 2 Changed: files => created: 1890, deleted: 461, replaced: 257 folders => created: 69, deleted: 65, replaced: 1 **There are error messages in the log file above: see "**Error"
A corrupted drive, as in this example, will often register errors in a logfile, and may require fixing, reformatting, or replacing. In the case that produced the logfile snippet above, a power surge ruined the drive's index tables, triggering bogus SYNC updates that could be repaired by an UNDO, SYNC, or full COPY (the latter is required only if there is major damage).
As noted earlier, drive failures may also manifest as inexplicable file differences in DIFF (e.g., for bad sectors). While SSDs tend to be more reliable than flash drives, every drive has a limited life span—which is, of course, a primary reason for using this app to create multiple copies of your content. But logfile errors may be your first warning of drive demise.
That said, keep in mind that you can expect years of error-free service from almost any storage device, and most logfile indicators are not drive failures. Some may reflect easily repaired conditions—including permission errors for locked files (simply unlock and resync, per the extended coverage here), and unexpected drive unmounts (just rerun the sync to update files skipped)—and some indicators may stem from the known symlink limitations covered in the prior section.
Still other summary indicator lines are not errors at all. The end of a SYNC, SHOW, UNDO, or COPY logfile may also display a note indicator like this:
*Summary ... **There are non-error notes in this logfile: see "**Note" above
These notes are informational messages that don't necessarily imply errors, but should be inspected nonetheless (e.g., the macOS Unicode skew covered ahead may crop up in logfiles as notes). DIFF's logfile tail doesn't list errors or notes, but simply summarizes mismatches found, or tells you there were none:
Diffs found: 84 - items UNIQUE at [/Users/me/MY-STUFF/Websites] - [/Volumes/T7/MY-STUFF/Websites] ... Search for "*" above for more info
As a rule: if you see any of these indicator lines in summaries, open logfiles
in full with OPEN and search them for more details. A search for "**" will
locate both errors and notes after SYNC, SHOW, UNDO, and COPY; and a search
for "*" will find more details on differences after DIFF. Some DIFF
items may warrant single-file comparisons (e.g., with Windows fc,
Unix diff, or
this script),
but others are expected and ignorable—per the next section.
Finally, you should also be aware of three special cases that may impact the results you'll see in logfiles:
__bkp__ subfolders are skipped by SYNC and SHOW because they are
not part of your content, but are not skipped by DIFF because it runs a
complete compare. Hence, they will generally be reported as unique items in
DIFF summaries alone, and can be safely ignored as normal differences.
These files do not differ in any user-visible way, and you can safely ignore their differences reported by DIFF alone. The covert content change behind them probably qualifies as an Excel bug, but there is no known fix. If you don't want to see the differences, delete these files in TO so they will be recopied on the next SYNC, or convert them to newer Excel formats.
.eml extension are always
erroneously copied with the date and time of the copy operation—not
the date and time of the original file being copied. This happens even though
the files' content has not been changed in any way, and reflects a glitch
deep in Windows' system libraries.
The upshot here is that .eml files will always register as different in
SYNC and SHOW (for timestamps), though not in DIFF (for content). Worse, this
causes all such files to be recopied on every SYNC due to timestamp
differences: because they are always recopied with the current date and time,
this guarantees a spurious recopy on each later SYNC.
This reflects a longstanding Windows bug, which oddly special-cases
.eml files in system libraries, and has not been addressed from
Vista or 7 (depending on the accuracy of the web) through 11. The best
explanation seems to be that Windows' NTFS filesystem stores non-content extra
data used to identify files' origination points for security purposes; setting
or changing this extra data wrongly changes files' content-modification time,
and the default rules for .eml files—among others—invoke
this behavior in low-level system calls used for copies.
You can read more about this Windows anomaly widely on the web; see
this search.
It arguably reflects an engineering kludge either naive or rude,
but we'll leave the outrage to the web.
Per results of the search link, you can generally avoid differences
and recopies for these files by disabling indexing options for the .eml
extension with this sequence or similar:
You can, of course, simply ignore the Windows effect instead. Since these files are by default recopied on each SYNC, they will be always current—if perpetually annoying!
For more info on handling errors, see the alternative take in the docs of the embedded Mergeall system.
As suggested in the Usage Overview, you can sync your content folder in its entirety and/or sync nested subfolder located inside it. For example, you may wish to sync just a DCIM photos subfolder nested in your complete content folder; this can save time if you've only added or edited photos and haven't changed content in any other subfolders.
If you opt to do this, though, keep in mind that if you both sync
subfolders and sync the whole content folder containing it, a sync of the
whole content folder will either change or delete the __bkp__
backups folder in the nested subfolders you've synced separately.
The net effect removes UNDO rollbacks as an option for subfolder, at least
in terms of their former separately-synced content.
This stems from the fact that a SYNC's basic function is to make TO a
mirror-image copy of FROM. When you sync the whole folder, it will
make subfolders in TO mirror-image copies of the corresponding subfolders
in FROM. This invalidates the subfolders' __bkp__ folders:
they no longer apply to the subfolders' new mirror copies—and in
fact will not work to restore the subfolders' content to what it was
before the whole-folder sync.
Specifically, when a TO subfolder has its own __bkp__,
a whole-folder sync will either delete it if there is none in FROM,
or replace it with the same-named folder at the same location in FROM.
If instead a FROM subfolder has a __bkp__ but TO does not,
it will be copied to TO verbatim.
This effect makes sense if you remember that whole-folder syncs make
TO's subfolders mirror-image copies of the same subfolders
in FROM: in all cases, __bkp__ folders are propagated
or removed so that they apply to the mirror copy created in TO by the sync.
The former separately synced content is no longer relevant.
In sum: feel free to sync subfolders as you wish, but know that
a sync to the content folder's root will make subfolders in TO
match those in FROM, which may make it impossible to roll back
recent subfolder changes separately. You can, of course roll back
all the changes at the root level, which will also restore
subfolder __bkp__ folders, but this is unlikely
to be useful after you've synced at the content's root.
Also keep in mind that syncing subfolders syncs just those subfolders; if you forget that you've changed content in other subfolders too, their changes won't be propagated. Generally speaking, you're better off syncing your content folders as a whole.
Most phones produced in the last few years support both removeable drives on a USB port, as well as the portable exFAT filesystem recommended for this app. Given the wide variability in the Android world, however, this is not universal.
Case in point: the 2020 budget-level Pixel4a phone does not support removable USB drives with its stock Android 11, even if you grant the app the requisite All Files Access permission. This reflects a bug in the underlying Android 11 shipped on this phone.
The good news is that upgrading the Pixel4a to either Android 12 or Android 13 (as is essentially forced by Google) enables USB access in full in this app. Still, it's not impossible that some other phones may have similar constraints, especially if they are older phones that have not been upgraded to newer Android releases.
As a rule, be sure to fetch the trial version of this app and verify that your phone supports USB access, before purchasing the full version. This is especially true for older and functionally limited phones. It's impossible to test every vendor's phone in use, but newer is generally better when it comes to USB support—as well as most other aspects of the Android user experience.
Even with Android 12 or 13 upgrades, the Pixel4a supports the FAT32 filesystem on removable drives, but not the recommended exFAT. While FAT32 works in this app and is widely available on nearly all devices with a USB port, it has substantial limitations, especially when it comes to timestamp-based syncs like that use in this app. For more details, read on to the next note.
If you must use FAT32-formatted removable drives on your phone (as on the prior note's Pixel4a example), be sure to see the Mergeall coverage here of issues that can arise when your devices' time changes due to Daylight-Saving-Time rollovers or time-zone changes.
In short, because FAT32 records time as absolute "local" time, its timestamps can trigger spurious differences on syncs. If not addressed, these in turn can invoke full and pointless recopies of your content folders, because every file's modification time erroneously differs. To avoid this, either use exFAT where available, or run the simple and free command-line script available here to adjust content timestamps when needed; see the script for more info.
The recommended exFAT filesystem uses timestamps that are relative to a fixed date, and so are immune to issues when device times change. FAT32 also suffers from an archaic 4G filesize limitation. In fact, it's so limited by today's standards, that you may want to consider upgrading your phone or its Android to lift the FAT32 constraint.
Exception: on older Samsung devices running Androids 8 and 9, there was a known timestamp bug which made it impossible to use exFAT for sync programs like this app. In short, the bug botched modification times on exFAT drives; which caused all files to be wrongly classified as changed; which triggered spurious copies of the entire content folder. This bug was fixed in Samsung's Android 10, so users of 10 and later are immune and free to use exFAT. If you're still using some older phones, however, you may need to use FAT32 for removable drives in this app if its SYNC or SHOW oddly classify all files as differing; full details.
This app's About tab includes a caution about battery-optimization settings when run on Android. In brief, some phones aggressively kill running apps to conserve battery power, and this might have consequences for this app's long-running actions. This kill may occur due to excessive CPU usage, or more arbitrary culls, but it's a fact of life on some phones.
It's important to note that this app by default runs actions as officially sanctioned Android foreground services, which should be immune to most reaping. It also offers a thread-mode alternative that is likely immune to process-killing logic added in Android 12. Moreover, no actions kills have been seen or reported for this app to date, despite testing and usage that has included multi-hour action runs which process 200G content folders. While that probably makes this a non-issue for this app, Android's variability again makes absolute statements perilous.
Though unlikely, if this app is ever killed inexplicably on your phone, your first recourse is to disable all battery optimizations for this app; they're generally in Settings=>Apps=>this app, and go by various names on different phones. It may also help to run this app's actions as threads instead of services to avoid 12+ process killers (see the Config tab), though CPU usage may be classified similarly either way.
Again, there are no known kill issues for this app, but some vendors more ruthlessly limit programs than others. In the end, optimizing battery longevity at the expense of basic phone functionality is a very slippery slope, and you should probably consider a vendor's track record in this department before purchasing a phone. Devices that reliably support only short-lived tasks aren't as useful as their marketing may imply.
This note applies only to users who will sync content to a macOS PC, and use non-ASCII filenames with characters that have multiple representations in the Unicode standard. If that set doesn't include you, you can happily skip this subtle story. Else, either read on for more info now, or file this for future reference in case your Unicode names ever fail on macOS, in this app or otherwise.
If you propagate content from and to macOS, you may run into a rare portability peril for filenames that contain non-ASCII Unicode variant characters. This reflects a Unicode hurdle imposed by macOS, and seems likely to be a problem only in roundtrips from and to macOS through exFAT drives. Moreover, most transfers seem immune to the drama: this may crop up only if filenames formerly exported by macOS are recopied on Windows from scratch.
That's an atypical usage context to be sure, but if non-ASCII filenames on exFAT drives fail and are skipped with logfile errors when exporting to macOS, you can fix them by simply running a provided script on Windows available here and covered further ahead.
To truly understand the peril, we first need to take a short side trip to the land of Unicode. One of the goals of the Unicode standard is to uniquely identify characters with sequences of one or more integers called code points. Oddly, though, this standard allows a small set of characters, especially those with accents or other augmentations, to be represented by multiple, different code-point sequences. These different-but-same representations as generally known as Unicode variants.
For example, the character Ä can be represented as either
the single code-point 196, or the two code-point sequence 65 and 776
(when encoded as UTF-8, there are two and three bytes, respectively:
\xc3\x84 and A\xcc\x88). The first
of these is known as the composed form, which goes by acronym NFC; the second
is called decomposed (i.e., accents separate), and abbreviated NFD. Though they
differ, both forms are defined to stand for the same character per standard:
Ä.
Naturally, a standard that uniquely identifies characters with identifiers that are not unique can create interoperability problems. If a variant character stored in one form is transferred to a system that mandates another, you have a built-in portability issue. Most platforms in common use prefer and create NFC, but leniently allow NFD too. macOS, however, is the exception: it mostly mandates NFD, and even refuses to recognize NFD on exFAT in one context described next.
To see how all this can cause problems, you also need to understand the convoluted behavior of exFAT on macOS. When you store filenames with Unicode variant characters on exFAT, they adhere to the following rules—at least as tested on macOS Catalina and Windows 11 (that latter of which likely mirrors Android, though this remains to be verified and may vary by storage type):
ls.
The last point is what can lead to interoperability crises. Both NFD files created on macOS, and NFC files created on Windows can be propagated between macOS and Windows without problems: Windows allows macOS's NFD, and macOS allows Windows' NFC.
NFD files created anew on Windows, however, won't be recognized by macOS on exFAT drives, for reasons known only to that platform's exFAT driver. Since Windows won't generally create NFD names itself, this normally arises only when Windows recopies an NFD file passed to it previously from macOS. That is, a recopy on Windows of content that originated on macOS may trigger errors.
This is an unlikely use case (and arguably qualifies as a macOS bug), but may arise if you restore macOS content from a Windows copy. If you ever run into it, the solution is to convert any NFD filenames to NFC on Windows prior to propagation, so they will be recognized on macOS.
There are a number of ways to convert, but to make this simple, this app supports Unicode filename conversions with a free utility script available online here. You must run it on Windows (macOS can't process the offending NFD files, after all), but it's a one-time command line. See the script for more details; it includes a usage example at its end.
Finally, if you really want to ensure pain-free portability for your content, the simplest route is to stick to ASCII characters for your filenames to fully avoid macOS issues. While you're at it, be sure to also avoid characters in the set rendered nonportable by Windows, a similar but less subtle and more widespread pitfall addressed by this app's NAME.
The all-ASCII solution may not be ideal if your native language includes non-ASCII characters which the Unicode standard failed to standardize. But it will spare you from the drama generated by computer vendors that seem to view proprietary lock-in as valid business plan—and customers like us as collateral damage.
For additional info on installing and using this app, see the main sections of its User Guide.