#!/usr/bin/env python3 """ ========================================================================================= PC-Phone USB Sync A standalone and Python-coded Android (and PC) app for Mergeall. [Please note: this code was not meant to be published. If you stumble onto it anyhow, ignore the rough edges and snarky bits.] Copyright © 2024 quixotely.com. All rights reserved. License: this file is provided without warranties of any kind, and only for online viewing and vetting. It may not be copied, posted to GitHub, modified, repackaged, or sold. It is a view-only resource. This Python 3.X script file is run as an Android app; a macOS app; Windows and Linux executables; and source code. Only the latter requires Python and Kivy installs. This script's GUI uses the portable Kivy library; the Android app is built with buildozer (which uses p4a); its macOS app and Windows and Linux executables are built with PyInstaller; and Python 3.9 (3.8 on some PCs) is used internally for both GUI and sync code. buildozer builds use the command lines below, and PyInstaller builds use pc-build/build.py on all three PC platforms. On Android, this app runs on Android 8.0 (api 26) and later (tested to A13), and targets Android 12 (api 31) due to Play store rules for new apps. On PCs, this runs as source code broadly, and its 64-bit exes are built on macOS Catalina (universal2), Windows 11, and Ubuntu Linux. Mergeall's former tkinter GUI still works on earlier Androids and other PCs, though their market share is rapidly approaching zero. This app is a free executable on PCs, and runs on Windows, macOS, and Linux; see https://quixotely.com for downloads. The PC versions were added later, but also use the Kivy GUI, and this same code. The Android app is for pay on the Google Play store at a modest cost, with a free trial that expires after a preset number of app runs (more or less; see handle_trial_counter() for subversions+caveats). Dependency: Kivy currently requires Open GL ES 2.0, which has been marked as deprecated by Apple in favor of its proprietary alternative (yes, rudely!). Support for Open GL may also be murky on Windows. The Kivy project has already begun moving to alternatives on macOS (using a compatability layer developed by Google), but in the worst case, Mergeall's own tkinter GUI can be used to run syncs on PCs too. See https://github.com/kivy/kivy/issues/5789. The underlying Mergeall system run by both PC executables and Android apps is open source, and available at learning-python.com/mergeall.html. This app/GUI is not open source, but may become so eventually; it was first released as a premium app, to set it apart from free Android fare. ----------------------------------------------------------------------------------------- ANDROID BUILD CHEAT SHEET [see also ../_HOW.txt for the build saga] [see also pc-build/build.py for PC builds: totally different, PyInstaller] Start code must be main.py (this), can config folder in .spec There is a .buildozer/ in both ~ and ., plus code installed in python /Frameworks buildozer.spec is in . after running "buildozer init" => mod for app Built apps show up in ./bin Kivy ui def: "class Xxx" => file "xxx.kv" MANAGE_EXTERNAL_STORAGE works if in manifest via ./bd.spec file: needs runtime perm foreground services and file providers require extra steps: see notes here + bd.spec MAIN BUILD COMMANDS: # caution: folder name in below's .sh code changed for 1.2 (and 1.3) # set vars here and in buildozer.spec to make 4 apps: full/trial & debug/release # now set just REL* here/ahead for release version: no more trial app [1.3.0] cd ~/Desktop/DEV-BD/apps/PC-Phone-USB-Sync-1.3 (find {., ~/.buildozer} -name ".DS_Store" -print -exec rm {} \;) source _dev-misc/UPLOAD-KEY/set-env-vars.sh (set upload-key vars for signing 'release') buildozer android debug (make apk: can sideload or stream install) (mod REL* here/below for release=True) buildozer android release (make aab uploadable instead of apk) PLUS: # stream install, logcat seperately (mind the version # in the filename!) export ADB=~/.buildozer/android/platform/android-sdk/platform-tools/adb $ADB install bin/usbsync-1.3.0-arm64-v8a_armeabi-v7a-debug.apk $ADB install bin/usbsynctrial-1.3.0-arm64-v8a_armeabi-v7a-debug.apk $ADB logcat | grep python DEV: # build+install+run+logcat: buildozer android debug deploy run logcat | grep python (add "clean" to reset app components: runs long) DEV: # build only (and to see more error msgs): ~/Desktop/DEV-BD/apps/PC-Phone-USB-Sync-1.3$ buildozer -v android debug ETC: buildozer -v android debug deploy run logcat buildozer -v android debug deploy run logcat > my_log.txt buildozer -v android deploy run logcat | grep python ETC: buildozer --help (for all options) Android-SDK/cmdline-tools/latest/bin/apkanalyzer files list bin/usbsync-1.3.0-*-debug.apk Android-SDK/cmdline-tools/latest/bin/apkanalyzer manifest print bin/usbsync-*-debug.apk Android-SDK/build-tools/33.0.1/aapt dump badging bin/usbsync-1.3.0-*-debug.apk ----------------------------------------------------------------------------------------- MORE DEV NOTES **See ../PC-Phone-USB-Demo/main.py for more early notes trimmed here** **And ../HOW.txt plus the ../../{DEV, DEV2}/HOW older files it names** SUMMARY As of rc2, a "clean" build requires 2 manual steps, + 1 if view files in Finder: - [1.2.0] to install a newly targeted android API level (sdk): 1) change target-api setting in ./buildozer.spec 2) get new sdk in ~/.buildozer (buildozer doesn't (!)) ~/.buildozer/android/platform/android-sdk/tools/bin$ export ANDROID_HOME=/Users/me/.buildozer/android/platform/android-sdk ./sdkmanager --sdk_root=${ANDROID_HOME} --install "platforms;android-33" ls ../../platforms => android-31 android-33 3) mod the following TWO build files for new target-api level: ./.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/dists/usbsync/project.properties and likewise for ../usbsynctrial else buildozer uses new api to compile, but doesn't reset it in manifest (!) (avoids "clean" or "appclean" build that refetches new versions of everything) (https://stackoverflow.com/questions/72746288/buildozer-android-api-doesnt-change) - [1.2.0] to force python38: "source ~/use38" and mod ~/.bash_profile to be sure and "py3 main.py" to test app as source - must "find {., ~/.buildozer} -name ".DS_Store" -print -exec rm {} \;" else obscure for version# errs - must edit jnius/env.py file, TWICE - in both arch builds (post make clean) else JNIUS undef error early in build log see ../HOW.txt - must insert fileprovider code, TWICE - in TWO dists manifest templates (post make clean) else Logs OPEN won't workin the app see ./manifest-manual-*/xml MORE BUILD/DEBUG HURDLES - had to declare properties in Main() ref'd in .kv file - had to add .txt to source.include_exts in .spec for help text files in apk folder - had to set android.no-compile-pyo=True in .spec else mergeall config not editable - no-compile-pyo => no-byte-compile-python Nov22 (but still in old spec files!!) - adding '.' to sys.path fails, but os.getcwd() works - the usual 2 JAVA_HOME edits after build clean (see _dev*/demo-app/main.py) - [android:largeHeap="true"] in manifest doesn't help for file loads (java, not ndk) - BIG: had to patch buildozer manifest template for : see do_logs_open() - had to disable Back close: detecting service running on restarts deprecated+errorprone - maxnumbackups was a mess: 'from' statement, function-arg default, del sys.modules - BIG: mod mergeall.cpall to use forked shutil--skip chmod errs on Android for exFAT+FAT32 - two apps reuires two ids everywhere: file providers, foreground service, etc. - cancel failing update actions for removable folders outside app folder on android 10- - BIG: fix fatal kivy TextInput memory blowup by clearing its cache manually in TAIL+WATCH - aab signing for play store: upload key in _dev-misc, poorly documented and onerous - BIG: fix fatal kivy race condition for ToggleButton weakref dels in on_tab_switch() - fix kivy black TextInput lines, loss of scrolling, slow loading, orange dots, and more - plus lots of other kivy, buildozer, and pyinstaller workarounds not listed here PC PORTS macOS largely worked unchanged x usb drive names => list /Volumes x permissions (incl. full-disk-access) is weakly documented: downplay Windows problems resolved: x usb drive reports as drive_fixed (below) x opens too small (set window size) x Help takes 8secs to open (defer set to next cycle) x weird persistent red/orange dot post btn touch, on macos too (obscure hack) x checkboxes too bright (allow to vary at startup) x emdash and copyright symbols fail in text (use utf encoding) x unicode nfd variants propagated from mac=>win=>mac can fail on mac Linux mostly worked post windows x macos open => xdg-open x join => osjoin (see gotcha below) x usb drive names => scrape mount x RunningOnLinux was also True for android (win size) => fix GOTCHA: 3 "join()" calls in this file worked in the Android app, but NOT when run as source code on PCs! The p4a wrapper code used to launch this file in Android app imported "join" from os, but didn't delete it from __main__'s scope before launching this code. Changed to use "osjoin()" everywhere here, but the wrapper is error-prone and should be improved. The p4a startup code (in github, builds): python-for-android/pythonforandroid/bootstraps/common/build/jni/application/src/start.c PC BUILDS macOS buildozer was a fail use pynstaller --onedir so opens fast sans splash screen pyinstaller worked after add .kv to datas use '.' for file --add-datas, else a subdir need to set app title/icon and window icon here, else titles drops ' _', kivy icon app seems to lose Document folder permissions on reinstall; see Settings later: required .spec file for verion# BUNDLE arg available in spec file only later: add prompt for full-disk-access perm on run1, similar to android a-f-a later: built universal2 binary exe, using py 3.10 and pillow wheels combo on macbook Windows add special ctypes code for dpi scaling, else large/blurry (worse than tkinter) need to use spec file with Tree()s for kivy-deps, else no window provider add special code to close the splash screen, else never goes away need to restore builtins.print in thread wrapper: mergeall.py resets for frozen exes exes are suddenly >2G, killing build: exclude pc-build/ from tempsrc, copy its files up add special code for kivy logging error: writes to newly None sys.std* if --windowed Linux does NOT use spec files and kivy-deps like Windows; Kivy docs are deceptive on this! the .kv file is looked up ONLY in _MEI* temp unzip folder: must --add-data windows looks up .ks in _MEI* too, but seems to add to exe automatically lots of giant modules are pointless: excludes shrunk size form 200M to 40M running in virtualenv to reduce size doesn't help: shot up with undef installs plus all the Windows WSL2 Linux shenanigans; see Tech-Notes.html, feb24+. ----------------------------------------------------------------------------------------- TODO (rough, now dated: see _dev-misc/todo2.txt for laters) See also _dev-misc/todo.txt, Note20 memo n run ma as imported pkg to avoid os.cdir()? [kivy impact unknown]? x allow #logfiles-saved and #bkp-saved to be configs? x help-message.txt x icon x recode gui... x persist paths... x threads in gui (exec or import/recode)... ? disable file selection only (what?)... n animate disabled-image for Main buttons... Button().background_disabled_normal = image n bkgrnd and power-management issues, kills, and throttling power setting? Foreground Service? WorkManager?... thread & service run if left/pause and screen off states x is wakelock really held? yes: scripts run with screen off x is thread really immune to process kills? probably - it's not a child process x does thread run if app paused? yes: and at same speed as fg and termux x resize gui for keyboard?... https://stackoverflow.com/questions/36770050/properly-resize-main-kivy-window-when-soft-keyboard-appears-on-android#61037979 x why does icon/list swap work?.... => because kivy keeps both views, swaps, and updates redundantly x why doesn't ActionButton@Button work?... => because it clashes with a Kivy class name (!) - disable app-spec content backup to cloud?... - punt on app-specific altogether? (it's complicated, requires filename fixer for android) - reconsider cruft-pattern edits? (complex: skip+keep lists + case, pattern syntax) x fileprovider for open x foreground service for action runs x consider supporting older android with prior permissions scheme; usb differs x config tab, fontsize etc x catch/disable app exit if thread running? back button? subprocess differs? did for back button if script (thread|process) running; recent swipe can't catch x are mergeall modules compiled to bytecode? YES - in __pycache__, with .cpython-39.opt-2.pyc suffix x scrolled run output: \n dropped, ascii() quotes, mono font, scroll to end, speed but now moot: abandned for animated gif + Log TAIL/WATCH on demand x configs: color chooser, etc x port to macos, windows, linux x build pc exes x fonts in icon view in filechooser: FIXED in .kv n consider more responsive layout for landscape phones; PUNT (for now) x userguide x domain name, website x signing/store x trial version (see _dev-misc/todo2.txt for laters) ========================================================================================= """ #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # IMPORTS1 + PC PLATFORM TWEAKS #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # Python (see also GOTCHA above) import sys, os import pickle, glob, time, re, queue, webbrowser, json, datetime, threading, platform osjoin, osexists = os.path.join, os.path.exists # Python 3.X+ only assert(int(sys.version[0]) >= 3) """ ----------------------------------------------------------- Platform: we mostly care about the app/exe being run here, because a Python is bundled, and source is not distributed. Per Python's sys.platform: Linux means native, WSL on Windows, or Android don't care about WSL: must run Windows or Linux exe Cygwin means Windows, but using its own Python don't care about Cygwin: must run Windows exe, not code Linux must be reset to mean non-Android Linux only, else some code would erroneously fire on Android too (e.g., window size). Android detection prior to Python 3.7 is partly heuristic: any(key for key in os.environ if key.startswith('ANDROID_')). Python 3.7+'s sys.getandroidapilevel() is set at build time. 3.7+ req ok: app uses 3.9+ for Android (3.8+ among all PCs). ----------------------------------------------------------- """ RunningOnAndroid = hasattr(sys, 'getandroidapilevel') # py 3.7+, but Android uses 3.9+ RunningOnMacOS = sys.platform.startswith('darwin') # intel and apple m (rosetta?) RunningOnWindows = sys.platform.startswith('win') # Windows py, may be run by Cygwin RunningOnCygwin = sys.platform.startswith('cygwin') # Cygwin's own py, run on Windows RunningOnLinux = sys.platform.startswith('linux') # native, Windows WSL, Android RunningOnLinux = RunningOnLinux and not RunningOnAndroid # Linux ONLY: else true on Android too # [1.1.0]+ (feb24) Windows WSL2 Linux only wsl2sig = 'microsoft-standard-WSL2' RunningOnWSL2 = RunningOnLinux and platform.uname().release.endswith(wsl2sig) if RunningOnWindows: """ ----------------------------------------------------------- [1.1.0]+ (nov23) Move win32 imports to top of script so they will run once at script startup, instead of later in drive-name method storage_names_and_paths_removables(). This avoids import aborts if/when Windows 10+'s Storage Sense wrongly deletes the PytInstaller --onedir _MEI* temp folder prematurely (it was seen to do so once). That abort reflects a nasty bug in Storage Sense: temp files of still-running programs should NEVER be silently removed! The abort was also exceedingly rare: it was observed just once, when an odd system state caused the abort in an instance that had not yet opened a folder chooser before a restart. Still, the workaround here is easier than explaining all this to users... Note: mergeall/ scripts and imported modules are immune to this, because they are loaded from the install unzip folder (.exe's host), not the _MEI* per-run unzip folder. Code ahead cds to the install folder during startup, and mergeall/ is accessed from '.' therafter. Also note: win32 modules are included in the exe despite no --hidden-import. ----------------------------------------------------------- """ # get drive roots, network paths [after 'pip3 install pywin32' on windows] import win32api, win32file, win32wnet if RunningOnWindows: """ ----------------------------------------------------------- Like tkinter, Kivy does not do DPI saling well on Windows, which makes GUIs open large and blurry! This code must be run before importing kivy and addresses the issue, but is provisional, pending a fix in SDL2 layers in the Kivy stack. Note that this applies to python.exe for source-code runs, but to the frozen executable for its run: per-process call. Used for tkinter: windll.shcore.SetProcessDpiAwareness(1) https://github.com/kivy/kivy/pull/7299 https://github.cwindll.user32.SetProcessDpiAwarenessContext(c_int64(-4))om/kivy/kivy/issues/3705 https://learning-python.com/post-release-updates.html#win10blurryguis ----------------------------------------------------------- """ from ctypes import windll, c_int64 windll.user32.SetProcessDpiAwarenessContext(c_int64(-4)) if hasattr(sys, 'frozen') and (RunningOnWindows or RunningOnLinux): """ --------------------------------------------------------------- Yet another Kivy open bug workaround! In --windowed mode, Kivy logging writes to sys.stdout/err that are now set to None by PyInstaller instead of dummy objects, thereby triggerring a recursion-limit error. Doesn't happen if --windowed not used. This must be called this early in the script, else logs happen. Likely to be fixed very soon; but how wasn't this tested?... https://github.com/kivy/kivy/issues/8074 https://github.com/pyinstaller/pyinstaller/issues/7329 --------------------------------------------------------------- """ os.environ['KIVY_NO_CONSOLELOG'] = '1' # this is call-stack depth, and defaults to 1000 on Windows # NO: pyinstaller code was stuck in a loop, this didn't help # sys.setrecursionlimit(2000) if hasattr(sys, 'frozen') and (RunningOnWindows or RunningOnLinux): """ --------------------------------------------------------------- Tell the PyInstaller splash screen to close (else it doesn't!). The splash screen isn't supported on macOS, but the --onedir build for macOS opens quickly, unlike Windows/Linux --onefile. Android apps get splash screens from buildozer that just work. Nice feature, but why in the world doesn't this auto close?... [1.1.0]+ (feb24) doc: the splash screen (only) uses tkinter, which requires Tcl/Tk to be present. This causes problems on Windows WSL2 Linux - users must manually install Tcl/Tk to use this app there, because PyInstaller assumes they're present, and they're not in WSL2 Ubuntu. There was also another obscure issue on desktop Linux with Tcl/Tk version numbers. Plus all the weirdness of not auto closing; is this really worth it? --------------------------------------------------------------- """ import pyi_splash # Update the text on the splash screen? pyi_splash.update_text('Loading program...') # currently does nothing... # Close the splash screen. It does not matter when the call # to this function is made; the splash screen remains open until # this function is called or the Python program is terminated. # # pyi_splash.close() # see next function, called later def close_pc_splash_screen(): # run this in App.on_start() instead, to wait for window if hasattr(sys, 'frozen') and (RunningOnWindows or RunningOnLinux): import pyi_splash pyi_splash.close() if hasattr(sys, 'frozen') and (RunningOnMacOS or RunningOnWindows or RunningOnLinux): """ --------------------------------------------------------------- Set up runtime cwd context for frozen PC executables. PyInstaller unzips --onefile exes and their added data to a temporary folder, _MEI*, which is deleted after app exit. This differs from the app's install folder, where the download is unzipped. In this context, this app must arrange access to both persistent and non-persistent data items by cwd or other: - shipped mergeall/ subdir - modified mergeall/mergeall_configs.py module - shipped help and about message files - created run-counter and configs-pickle files - shipped usbsync-anim.gif used for image in gui - created logfiles and UNDO backups - shipped .kv GUI-def file, not handled auto! - shipped usbsync-pc/, PCs must set as icon in .py Some of these these may not work in _MEI* temporary unzip dir, because they are changed and must be retained between app runs: - mergeall/ scripts are loaded as source code from its folder, so cannot be .pycs; never change, so can be in either install or _MEI*; and are already open source, so need not be hidden. Some modules in mergeall/ are also imported here (backup, cpall). - mergeall/mergeall_configs.py is auto-changed by this script for GUI settings. It can be in _MEI* iff it is changed on _every_ Main-tab action run; else it must be in install (or elsewhere). Because this must be accessible, all of mergeall/ must be too. - Help and about files are read-only. They could be in install or _MEI*, but might as well follow the same policy as mergeall/. - Run-counter and configs-pickle files cannot be in _MEI*, because they must be changed and retained between runs. They can be in the install dir, except on macOS which maps them to ~/Library because apps may not be able to write own folders (tbd). These might also be mapped to ~/Documents everywhere, but are not (yet?). - The gif is read only and can be in any cwd (install or _MEI*); and the .kv is read only (and optional, if its code is in .py). - usbsync.png is used at build time (buildozer, pyinstaller), but also for .py icon calls required on PCs (else kivy icon). UPDATE: PCs now use prebuilt icons in usbsync-pc/, not pyinstaller. - logfiles are mapped to ~/Documents everywhere and so are moot. Other: SYNC backups are in TO/, and so are also moot here. To handle these cases, this app uses these policies for PC builds: For macOS: Since PyInstaller builds an app folder anyhow, use --onedir; use --add-data to store all data items alongside the executable in the app folder; and cd to sys.executable's dir (the install unzip's /Contents/MacOS) on startup for cwd-relative access to mergeall/, help, about, gif, and .kv file. The app dir persists between runs, and is assumed changeable by app. This gives persistence for mergeall/mergeall_configs.py, as well as access for help, about, and gif loads. The run-count and configs-pickle files are mapped to ~/Library/appname. This also avoids lags for startup unzips sans splash sceeen on macos. For Windows and Linux: Store data items alongside the --onefile single-file executable in a manually created zipfile, and cd to sys.executable's dir (the install's unzip) on startup for cwd-relative paths (like Mergeall). These items will not unzip to _MEI*, and hence will persist between runs as needed. There is a lag for the unzip at app startup, but these platforms now post a splash screen. Exception: Linux, ONLY, does not look for the .kv file in cwd, but ONLY in the _MEI* temp unzip folder. On Linux alone, must add it via --add-data and can skip copying it to install folder. UPDATE: per more tests, Windows exes run if the .kv is not present in cwd (install folder) and is not added via --add-data, so it must be adding it to the exe at build time. macOS requires that the .kv be added by --add-data on --onedir mode, else the app won't open (and --onefile on macOS takes ~12 seconds to open with no support for a splash screen, vs 1~2 for --onedir mode). Kivy's folder-selection code is way too whack to plumb for answers, and its handling of .kv files in frozen executables is an undocumented, convoluted mess. PC builds work; punt! macOS alt: there's a chance that the app folder won't be writeable by the app. If so, use --onefile to build and unzip to _MEI*; _always_ mod mergeall/ mergeall_configs.py for numbackups on each Main action run in this context ONLY, to make its persistence moot; and cd to __file__ (_MEI*) on startup for cwd-relative access to mergeall/, help, about, and gif. The run-counter and configs-pkl files are already mapped to ~/Libraries/appname for macOS only. UPDATE: not required - app can mod mergeall/mergeall_configs.py. The chdir here makes an empty __file__ dir map to install dir. It precludes any relative paths in cmdline args to frozen exes, but this is moot, as exes here don't access items in users' cwds. Mods here aren't run for either source code or the Android app, which both keep working normally and as is. For more background and another PyInstaller example, see Mergeall's fixfrozenpaths.py. SEE ALSO: pc-builds/build.py applies these policies in builds. --------------------------------------------------------------- """ # sys.executable is the frozen exe for --onefile and --onedir # sys.argv[0] is the path used in the launch command, poss rel # sys._MEIPASS is the TEMPORARY uzip folder for --onefile # __file__ is sys._MEIPASS+script for --onefile, else install folder exepath = sys.executable exedir = os.path.dirname(os.path.abspath(exepath)) # install folder os.chdir(exedir) # '.' for all extras # not required: this + launcher mod path based on abs(cwd) # sys.path.append(exedir) # for ma/configs .py import #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # IMPORTS2 #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # app from run_script_as_thread import run_script_thread_wrapper, Queuer, Flusher from run_script_as_process import run_script_process_wrapper # auto-run as a script by p4a service support: import for error check only if RunningOnAndroid: from run_script_as_service import run_script_service_wrapper # kivy: GUI from kivy.app import App from kivy.uix.floatlayout import FloatLayout from kivy.uix.popup import Popup from kivy.uix.button import Button from kivy.uix.label import Label from kivy.uix.textinput import TextInput from kivy.uix.togglebutton import ToggleButton from kivy.uix.checkbox import CheckBox from kivy.uix.boxlayout import BoxLayout from kivy.properties import ObjectProperty, StringProperty, NumericProperty from kivy.clock import mainthread, Clock from kivy.core.window import Window # instance is app.root_window import kivy.metrics from kivy.config import Config from kivy.cache import Cache from kivy.uix.filechooser import FileSystemLocal # hangware [1.1.0]+ (feb24) # pyjnius: Java interface if RunningOnAndroid: from jnius import autoclass, cast # p4a: helpers if RunningOnAndroid: import android.storage #from android.storage import app_storage_path #from android.storage import primary_external_storage_path from android.runnable import run_on_ui_thread from android.config import ACTIVITY_CLASS_NAME from android.broadcast import BroadcastReceiver else: run_on_ui_thread = lambda func: func # no-op decorator for PCs # old-style storage perms if RunningOnAndroid: from android.permissions import request_permission, check_permission, Permission #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # GLOBALS #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # version also redundantly in buldozer.spec + pc-build/{_macOS*.spec, _winver*}; # version string for Play can be anything, x.y.z allows simple patches to x.y; # buildozer (really, p4a) builds versionCode from it: '10'+sdkmin+shiftedVERSION; # Search on [VERSION] to find mods in code; [VERSION]+ is select platforms only; # History: [1.0.0] May 2023; [1.1.0] June 2023; [1.1.0]+ later; [1.2.0] March 2024; # [1.3.0] September 2024: make free and drop trial version, update docs accordingly. APPNAME = 'PC-Phone USB Sync' # used for assorted folder names, etc. VERSION = '1.3.0' # versionName in manifest, inserted in About PUBDATE = 'September 2024' # inserted in About by on_start() too [1.1.0] # Android trial app: mod TRIAL_VERSION + buildozer.spec [title, package, presplash, icon] # per handle_trial_counter(), app data clears and installs can subvert run counter; # use 'buildozer android release' to make release aab of both, use '...debug' for apk; # this works because this file's code is copied into app as it was at build time here; TRIAL_VERSION = False # True for trial: limited #opens TRIAL_APP_OPENS = 10 # 5 seems too few, but reinstalls should be a bit onerous # distinguish android trail app where it matters (only) [1.1.0] APPNAME_FULL_OR_TRIAL = APPNAME if not TRIAL_VERSION else APPNAME + ': Trial' # release aab on Android: mod to disable trace print()s to logcat (per Play); # change manually - can have release versions of _both_ full and trial app; # if `buildozer android release` ensures this, it's a very well-kept secret; # print()s in nested mergeall/ code are routed to logfile in all run modes; # [1.3.0] trial version is gone, so this is the only per-build mod now required RELEASE_VERSION = False # True for release: don't trace # global font size: it's complicated; see set_font_size() # [1.1.0]+ (oct23) add a fudge factor for linux - init fontsize is _very_ # small [caveat: this is risky - tested on just one linux device so far]; # # also changed for linux in [1.1.0]+: # - (sep23) scale initial window size and add fudge factor too, ahead # - (oct23) executable rebuilt on ubuntu22 for lib-skew issue on 20 # # per kivy docs: sp() for fonts (uses user font settings), else dp(); # note: RunningOnLinux constant means linux only, and not android; lx = 5 if RunningOnLinux else 0 # [1.1.0]+, +3 still seems too small FONT_SIZE_DEFAULT = int(kivy.metrics.sp(15+lx)) # kv default=15sp, scaled for device+user # items in app-install folder (not app-private) (no user access on android) HELP_FILE = 'help-message.txt' ABOUT_FILE = 'about-template.txt' TERMS_OF_USE_FILE = 'terms-of-use.txt' MERGEALL_PATH = 'mergeall' SERVICE_BREADCRUMB = 'latestservice.txt' # no longer used # items in app-private folder (not app-install) (no user access on android) SETTINGS_FILE = 'settings.pkl' RUNCOUNT_FILE = 'runcounter.txt' # logfiles in Documents folder (android: shared-storage, pcs: ~, user can access; !trial) LOGS_SUBFOLDER = APPNAME # spaces ok+common; allow in OPEN/EXPLORE; .replace(' ', '_') # [1.2.] now used for multiple greps (globs) in logfiles folders: ensure same LOGFILE_NAME_PATTERN = 'date*-time*--*.txt' # macos app-private folder in ~/Library (avoid app-bundle folder, user can access, !trial) MACOS_APPPRIV_SUBFOLDER = APPNAME # items in TO/__bkp__ sync backups folder (user can access) UNDOABLE_FILE = '__undoable__' # see also '--UNDONE' renames # values of root.script_running (0 is False, else True) NOTHING_RUNNING = 0 # running as thread or service or process?: for app close THREAD_RUNNING = 1 # as thread only: for trace() - don't print() to streams SERVICE_RUNNING = 2 # as service only: for on_pause/resume() - mod msg receiver? PROCESS_RUNNING = 3 # as service only: for giggles? # thread coding choices (temp, probably) STATUS_ANIMATION, STATUS_SCROLL = True, False # save Main() instance in global (because self.trace is too much to type...) guiroot = None def trace(*args, **kargs): """ ----------------------------------------------------------- Don't print if thread running: streams sent to action's logfile (formerly, line queue then logfile; thread mode now uses thread Flusher which writes to logfile directly). Prints go to adb logcat on Android, console on macOS?, etc - for developer viewing only. Alt: logcat direct? In addition: frozen Windows/Linux exes mod builtin print, which raises excs for flush*2. Avoided by THREAD_RUNNING: builtin print is changed by in-process mergeall.py code run in a thread, but restored by thread wrapper on mergeall exit before script_running is cleared (and before the clearing code is scheduled to run!). See run_script_as_thread.py. UPDATE: mergeall/mergeall.py now omits flush=True if one is sent here, because excs still happened during tab-switch callbacks run at odd times (thread/GUI timing is complex). Moot if not THREAD_RUNNING: standard streams in different process for foreground service (and unused simple process), so app can print freely here and print reset has no impact. Don't print trace messages in release builds for Android. That means there's nothing to go on for errors, but most users won't be able to run logcat anyhow, and Google Play chides about trace messages (and is douchefully dogmatic). ----------------------------------------------------------- """ if RELEASE_VERSION: return if not guiroot.script_running == THREAD_RUNNING: # don't print to logfile print('(PPUS)', *args, **kargs, flush=True) # don't flush*2 if frozen def trace2(*args, **kargs): """ ----------------------------------------------------------- NOT CURRENTLY USED, but can be used for app debugging. To trace while action thread runs; also pass trace=True to run_script_thread_wrapper() so stderr is not rerouted to line-queue Pipe. Don't double flush in frozen Windows and Linux exes: they already reset builtin print to do so. Don't print trace messages in release builds for Android. [1.1.0]+ "RunningOnWindows or RunningOnLinux and hasattr(..)" didn't look right and was corrected (py 'and' binds tighter than 'or'), but this is moot and harmless - function unused, and the need to omit flush=True was lifted (see trace() above; was this why there were still unexplained print exceptions??); ----------------------------------------------------------- """ if RELEASE_VERSION: return if (RunningOnWindows or RunningOnLinux) and hasattr(sys, 'frozen'): # [1.1.0]+ (oct23) print('(PPUS)', *args, **kargs, file=sys.stderr) else: print('(PPUS)', *args, **kargs, file=sys.stderr, flush=True) class _PopupStack: """ =========================================================== A simple stack for popup dialogs that may nest. Kivy coding structure makes it easiest to save Popups for later closes, but Popups may nest here (e.g., an info_message while a folderchooser is open, though this may have been impossible in earlier code). UPDATE: Button double-tap nonsense: 1) Pops of _empty_ stack triggered excs and app aborts when dialog close buttons were double tapped on all platforms. This specific issue is fixed by checking for empty here, but double taps create other issues. 2) For one, double taps can close _two_ dialogs stacked atop each other at once. To avoid this, all dismiss calls now pass the Popup's content, and dismiss here skips the call if passed contents do not match that of the stacked Popup. This avoids closing overlaid Popup on double taps. 3) Now also uses auto_dismiss=False on all Popup() so Popup not dismissed without a pop() here. Else, bogus auto-dismissed entries remain stacked and are dismissed on later dbl taps (though this seems to be harmless). Could keep auto-dismiss by catching on_dismiss and doing pop() in catcher, but taps outside multi-button popups are ambiguous and seem error-prone in practice anyhow. More generally, a double tap on _any_ button may trigger its callback twice... This seems a Kivy misfeature (bug!), but hasn't been an issue in testing. If can only matter if event is fired because the widget still active; the Popup case is probably timing - Button event beats the dismiss() event. Alt: disable Button double-tap instead? =========================================================== """ def __init__(self): self.popups = [] def push(self, popup): self.popups.append(popup) # stack Popup object def empty(self): return not self.popups def pop(self): if not self.empty(): return self.popups.pop(-1) # remove+return top def top(self): if not self.empty(): # avoid dbl-tap excs return self.popups[-1] def open(self): if not self.empty(): self.top().open() # display topmost Popup def dismiss(self, content): if not self.empty(): if self.top().content == content: # avoid dbl-tap closes self.pop().dismiss() # close topmost Popup class FileSystemLocalNoHang(FileSystemLocal): """ =========================================================== [1.1.0]+ (feb24) Wrap methods of the Kivy FileChooser's FileSystemLocal interface class (os module calls, mostly) in thread timouts. Else the chooser can stall (block and possibly hang) as users navigate by tapping its UI. Network drives, cameras, etc., may go offline, especially on Windows, causing raw, unwrapped calls to block indefinitely. This happens deep in the Kivy FileChooser code, but its file_system hook supports mods. Code here fixes the chooser widget; filesystem calls in the app itself were wrapped in thread timeouts earlier (the 1.1.0 release in Jun23). Coding notes: this doesn't call os functions directly, because Kivy's FileSystemLocal.is_hidden() uses tight magic to access and use a private win32 call on Windows. guiroot is the global Main instance: see PCPhoneUSBSync.on_start(). This adds a thread per chooser nav tap, but PCs and phones are fast. TBD: is this useful only, or at least primarily, on Windows? If so, could avoid one thread per os-module call elsewhere. To date, this is the only platform where nav hangs have been seen, though PCs and phones are both awfully fast... =========================================================== """ def listdir(self, filepath, warn=True): """ called for both test and result: warn user """ trace('NoHang.listdir') listed = guiroot.try_any_nohang(super().listdir, filepath, onfail=None, log=False) if listed != None: return listed # empty or not (no exc or timeout) else: trace('Skipping dir: listdir') if warn: guiroot.info_message('Cannot access folder.', usetoast=True) raise OSError # kill dir in filechooser (run in try) def getsize(self, filepath, warn=False): """ may be used if is_dir fails, but rare """ trace('NoHang.getsize') getsize = guiroot.try_any_nohang(super().getsize, filepath, onfail=None, log=False) if getsize != None: return getsize # dir or not (no exc or timeout) else: trace('Skipping file: getsize') if warn: guiroot.info_message('Cannot access file.', usetoast=True) raise OSError # kill dir in filechooser (run in try) def is_hidden(self, filepath, warn=False): """ win32 privates used in super call numerous times: don't warn user """ #trace('NoHang.is_hidden') ishidden = guiroot.try_any_nohang(super().is_hidden, filepath, onfail=None, log=False) if ishidden != None: return ishidden # hidden or not (no exc or timeout) else: trace('Skipping dir: is_hidden') if warn: guiroot.info_message('Cannot access folder.', usetoast=True) raise False # kill dir in filechooser (run in try) def is_dir(self, filepath, warn=False): """ called an insane number of times: don't warn user """ #trace('NoHang.is_dir') isdir = guiroot.try_any_nohang(super().is_dir, filepath, onfail=None, log=False) if isdir != None: return isdir # dir or not (no exc or timeout) else: trace('Skipping dir: is_dir') if warn: guiroot.info_message('Cannot access folder.', usetoast=True) return False # kill dir in filechooser fileSystemLocalNoHang = FileSystemLocalNoHang() # set in chooser on create #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # KIVY INTERFACE #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ class MainPathPickDialog(FloatLayout): """ ======================================================= The file-chooser dialog's callbacks. Also defined with <> instance rule in .kv. self argument implied in .kv file: root=instance. Class and its properties _must_ by defined in .py. ======================================================= """ # these become self attrs onpick = ObjectProperty(None) # refs callback in Main, run in .kv oncancel = ObjectProperty(None) # all args to class _must_ be declared here pickstart = StringProperty() # set here, used in nested fc: root.pickstart rootpath = StringProperty() # same as dialog.ids.filechooser.rootpath onviewmode = ObjectProperty() # icon|list from/to the Config tab (persistent) class InfoDialog(FloatLayout): """ ======================================================= The info-message dialog. Also defined with <> instance rule in .kv. self argument implied in .kv file: root=instance. Class and its properties _must_ by defined in .py. ======================================================= """ message = StringProperty() # passed here as args => root properties oncancel = ObjectProperty(None) # refs bound callback in Main, run in .kv class ConfirmDialog(FloatLayout): """ ======================================================= All in .kv, but must declare here too ======================================================= """ message = StringProperty() onyes = ObjectProperty(None) onno = ObjectProperty(None) class ColorPickDialog(FloatLayout): """ ======================================================= All in .kv, but must declare here too ======================================================= """ onpick = ObjectProperty(None) oncancel = ObjectProperty(None) class TrialEndedDialog(FloatLayout): """ ======================================================= All in .kv, but must declare here too ======================================================= """ message = StringProperty() oncancel = ObjectProperty(None) class ConfirmNameDialog(FloatLayout): """ ======================================================= All in .kv, but must declare here too ======================================================= """ message = StringProperty() onrunreport = ObjectProperty(None) onrunupdate = ObjectProperty(None) oncancel = ObjectProperty(None) class ConfigCheckbox(CheckBox): """ ======================================================= Ref in .kv, def in .py: need to set color per patform [r, g, b, a] for image tinting - too dim on macos only Default is [1, 1, 1, 1], which works well elsewhere ======================================================= """ color = [1, 3, 5, 4] if RunningOnMacOS else [1, 1, 1, 1] class TextInputNoSelect(TextInput): """ ======================================================================== Workaround for a Kivy text selection bug on Android. Or: The struggles to disable persistent handles and pointless selections. Kivy currently has a nasty bug which makes selection handles linger on in the GUI on Android - both after popup dismiss, and after tab switch. This is an issue only for _readonly_ text, as handles go away on either keyboard minimize or tab switch; chooser paths and log text are readonly. To workaround, first disabled copy, handles, and buttons in .kv file. This works, but user actions can still select the text pointlessly. Selection was first addressed by cancel_selection on tab switch (popup is auto by recreate), but it's still subpar to allow useless selections. The final workaround overrides selection methods in the TextInput class here (it didn't work in .kv) to disale selections in full. This makes the earlier fixes moot, and brings an end to all the troubles... ======================================================================== """ def __init__(self, **kwargs): super().__init__(**kwargs) # try 1: prevent changes, and selection handles and bubbles (.kv) #self.readonly = True # else keyboard still covers (iff font?) #self.allow_copy = False # else select handles on other tabs... #self.use_handles = False # prevents handles hell... #self.use_bubble = False # prevents action-bubble popups... # nuclear option: prevent selections altogether (here) def on_double_tap(self): pass # trace('on_double_tap\n') def on_triple_tap(self): pass # trace('on_triple_tap\n') def on_quad_touch(self): pass # trace('on_quad_touch\n') def long_touch(self, dt): pass # trace('long_touch\n') """ NO LONGER USED: in tab class LogfilePickDialog(FloatLayout): "" ======================================================= All in .kv, but must declare here too ======================================================= "" pickstart = ObjectProperty(None) onpick = ObjectProperty(None) oncancel = ObjectProperty(None) """ """ # TBD class SectionLabel(Label): def __init__(self, **args): super().__init__(**args) self.padding = [8, 8] valign = 'center' self.text_size = root.width, None self.size = self.texture_size class InfoLabel(Label): def __init__(self, **args): super().__init__(**args) #self.padding = [8, 8] valign = 'left' self.text_size = self.texture_size class LeftButton(Button): def __init__(self, **args): super().__init__(**args) """ #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # MAIN (ROOT) GUI CLASS #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ class Main(FloatLayout): """ ======================================================= The main/start window's callbacks. Corresponds by name to root rule in .kv. self argument implied in .kv file: root=instance. ======================================================= """ # kivy properties from_path = ObjectProperty(None) # from/to path display+edit fields in GUI to_path = ObjectProperty(None) logfile_path = StringProperty() # referenced in kv file during build() about_message = StringProperty() # default initial value is '' (a False) help_message = StringProperty() if STATUS_SCROLL: run_output = ObjectProperty(None) # bound to same name in .kv via TextInput's id #======================================================================= # MAIN-TAB FILE CHOOSER #======================================================================= def do_main_path(self, kind, kindid): """ ---------------------------------------------------------------------- Run from Main's FROM|TO buttons to pick a content path. kind = 'FROM'|'TO' for use in popup dialog label only. kindid = 'frompath'|'topath', which gives access via self.ids[] to pathfield = text display for the selected folder kind in Main tab. The last of these is used to initialize the selection-area display: the picker opens at the last Main pathfield setting for kindid, unless it's not a valid dir, in which case the picker opens on shared (here). Dynamically adds drive-name buttons after the standard storages; on taps, reset_main_picker_path() changes picker content dynamically. This just works in the folder chooser (but it also has multiple bugs). ROOT is complicated. It's unclear if the app can simply access '/' folders sans su commands on rooted phones (Termux "ls" required su for USB), and there's no root support in the kivy FileChooser. Enable iff Config, show '/' if access else api's /system, and avoid permission excs by setting dialog.rootpath to avoid '..' navigation. Illegal folders (e/g. '/', '/storage') throw excs if reached by '..'. Changes in the chooser's viewmode in the popup modify the popup's display, but also the Config tab's associated settings. The Config tab's settings are used for later restarts and persistence saves. ---- UPDATE: fix icon-view clipping for bigger fonts by copying template from kivy.style to .kv and modifying the label; this is awful, but there are no ids or other hooks for doing this; improve me, Kivy! UPDATE: make red storage-name buttons radio toggles, so it's clearer to users what is being displayed. This is probably moot once users get used to path names on their devices, but seems ambiguous at first. UPDATE: use child-list crawls here to nuke the "Size" label in list view, which is bogus for a folder chooser, and gets truncated to just "e" on some phones and font sizes (!). The fix simply sets the label to '' which may waste its space, but there are no builtin ids or hooks, and a core-code copy+mod seems worse (see the icon-view font fix in .kv). Children are reverse order of kivy-lang defs: see FileChooserListLayout in https://github.com/kivy/kivy/blob/master/kivy/data/style.kv (builtin) UPDATE [1.1.0]+ (feb24): subclass the chooser's file-system interface class, to wrap its calls in thread timeouts to avoid blocks and hangs. Else navigation taps can stall deep in the chooser widget itself. ---------------------------------------------------------------------- """ #--------------------------------------------------------------------- # prelims: get storage name/paths, make dialog #--------------------------------------------------------------------- # user may have passed on open or later if not self.get_storage_permission(): # confirm permission or ask again return # had to reask user: user must retry # get storages; shared is primary, appspec is for android only # CAUTION: label_path_string() mimics some of this logic [1.1.0] shared = self.storage_path_shared() appspec = self.storage_path_app_specific() drives = self.storage_names_and_paths_removables() # do before root for Windows root = self.storage_path_root() docs = self.storage_path_documents() # TBD: not curently shown by chooser # view-style utils def initstyle(): """ use Config tab style on first open (used ahead) the popup sets config buttons, but not vice versa """ return 'icon' if self.ids.mainchoosericon.state == 'down' else 'list' def changestyle(viewmode): """ mod Config tab style on chooser style button press yes, both buttons must be set, despite radio group note: these are Config-tab's buttons, not chooser's """ style = str(viewmode) # now moot self.ids.mainchoosericon.state = 'down' if style == 'icon' else 'normal' self.ids.mainchooserlist.state = 'down' if style == 'list' else 'normal' # init display to from|to path in Main tab (or shared, if run1 or drive absent) pathfield = self.ids[kindid] # naive: pickstart = pathfield.text if os.path.isdir(pathfield.text) else shared # avoid chooser exc: prior pick may have been dir but no permission; # probably should check on Pick, but actions already do before runs """ try: os.listdir(pathfield.text) # might hang... except: pickstart = shared # unmounted, permissions, edited else: pickstart = pathfield.text # prior pick still present and valid """ # same, but use timeouts to avoid lags for inaccessible drives [1.1.0] listed = self.try_listdir_nohang(pathfield.text) if listed: pickstart = pathfield.text # accessible: prior pick still present and valid else: pickstart = shared # hung, or fail (unmounted, permission, manual edit) # new chooser: see also .kv picker = MainPathPickDialog( pickstart=pickstart, # start navs here rootpath=pickstart, # stop up-navs here onviewmode=changestyle, onpick=lambda path, popupcontent: self.do_main_path_pick(kindid, path, popupcontent), oncancel=self.dismiss_popup) #----------------------------------------------------------------------- # set up-navigation cutoff for initial path in chooser # # allow '..' nav from last-run start path copied from Main, but only # up to its storage-type root, if any (+1 '..' for paths in app-spec); # else '..' navs above roots trigger silent exceptions in chooser; #----------------------------------------------------------------------- # use absolute path comparisons: user may enter anything manually [1.1.0] abspickstart = os.path.abspath(pickstart) if pickstart == pathfield.text: # copied over? topmost = None # no cutoff #------------------------------------------------------------------------ # android appspec first: it's nested in shared, and ../.. throws exc; # also sort drives by decr path len: usb nested in os on linux [1.1.0]; # # [1.1.0]+ (feb24) what?; this may reflect an erroneous mount of C: to # /media/me, which hid USB drives that ubuntu auto-mounts to /media/me/XXX: # the effect nested USB in C:'s mount (!), and only one might be usable; # # either way, match prefix by longest = most specific/nested first, # to handle nesting on all platforms as well as possible; user can # always tap a root button to restart (and trigger similar code ahead); # # assumes no drive will be nested in shared=home on Unix (macos+linux): # unikely, imposs on android+windows, and just a few extra ..s if wrong; # # a bit gray here: can't tell which storage was used to select prior path; # here the effect may be to limit .. navs to a lower nested storage root, # but this roughly corresponds to button preselect ahead: most specific; # # NIT: in hindsight, it seems that calling out appspec before shared is # not needed here, because the decreasing-path-length sort would order # these too that way anyhow (and is relied on to do so in similar code # at tab preselect and popup label fetch ahead, which could be merged); #------------------------------------------------------------------------ drives_sorted = sorted(drives, key=(lambda np: len(np[1])), reverse=True) for storage in [appspec, shared] + [path for (name, path) in drives_sorted]: if storage != None and abspickstart.startswith(storage): topmost = storage if storage != appspec else storage + '/..' break picker.ids.filechooser.rootpath = topmost # same as picker.rootpath # left align paths on open, can be scrolled after def leftalign(dt): picker.ids.pathname.cursor = (0, 0) Clock.schedule_once(leftalign) # Worst Kivy Hack Ever (or so far): nuke list-view "Size" label per above listlayoutobj = picker.ids.filechooserlistlayout try: listsizelabel = listlayoutobj.children[0].children[1].children[1] if listsizelabel.text == 'Size': listsizelabel.text = '' # to test: 'NOPE' except: pass # it's not worth dying for... # [1.1.0]+ (feb24) wrap chooser's os calls in timeouts to avoid nav-tap stalls; # this sets a Kivy hook to an instance of the file-system subclass (see above) picker.ids.filechooser.file_system = fileSystemLocalNoHang #--------------------------------------------------------------------- # add scrollable device/storage buttons to top of picker #--------------------------------------------------------------------- class StorageToggleButton(ToggleButton): """ ================================================================== red button storage-root toggles at top of Main-tab folder chooser: it's possible to mod 'down' color via on_state(), but not worth it; originally just buttons (not toggles) sans scrolling; they've grown; the 'down' preselect uses simple '==' because app.startswith(shared) on android, and everything.startwith(root) on pcs: broadest wins; update: preselect now uses startswith() + decreasing len() sort ahead; reset picker to button's storage root, even if already toggled down; this is complicated, and basically reimplements toogles behavior, but there seems no other way to catch a tab on selected toggles?; the usual on_release callback doesn't get triggerred in this case; on_touch events weirdly bubble through _all_ widgets that register for them: must check touch position for intersection with self, else this appears to always trigger on the same/wrong button (but it's really the first button in .children - which is added last!); note that this would work as a simple func or lambda assigned to togglebtn.on_touch_down (like on_release), but was coded as a class along the way due to a state-skew theory (before found bubbling); most self attrs here would be found in enclosing method's scope, but 'self' of enclosing method would be hidden: use all explicits; this was first inline code, then nested func, then nested class; ================================================================== """ numtouch = 0 def __init__(self, togglelabel, togglepath, picker, devbar, root, navup=False): super().__init__(text=togglelabel) # this button's state self.togglepath = togglepath # used by sort for preselect, etc self.picker = picker # rest used on touch callbacks self.devbar = devbar # picker = popup, devbar = buttons self.root = root # root = self of enclosing method self.navup = navup # used to allow 1 '..' for appspec # stadard toggle button bits self.background_color = 'red' self.group = 'pickstorage' self.allow_no_selection = False # set ToggleButton size, so BoxLayout has size, so ScrollView scrolls # self.padding_x = 144 self.padding_x = kivy.metrics.dp( # scale to density [1.1.0] 72 if not RunningOnAndroid else 55) # see docs at top of .kv self.size_hint = (None, 1) self.bind(texture_size=self.setter('size')) def on_touch_down(self, coords): self.__class__.numtouch += 1 if not self.collide_point(*coords.pos): # it wasn't on self, but we get it anyhow # trace('touchdown %d ignored=>' % self.numtouch, id(self)) return False # keep bubbling, you magnificent bastard else: # it's in this widget's screen area # trace('touchdown %d hit==>' % self.numtouch, id(self)) if self.state == 'normal': # was up: clear all buttons sans self, change picker for child in self.devbar.children: child.state = 'normal' self.state = 'down' self.root.reset_main_picker_path(self.picker, self.togglepath, self.navup) else: # was down: keep self down (only), change picker # oddly, need to reset ~elsewhere first, else no-op elsewhere = osjoin(self.picker.ids.pathname.text, '.') self.root.reset_main_picker_path(self.picker, elsewhere) self.root.reset_main_picker_path(self.picker, self.togglepath, self.navup) return True # end processing: did all toggle behavior manually #--------------------------------------------------------------------- # back to storage-buttons setup #--------------------------------------------------------------------- # or .children[:], .remove_widget() devscr = picker.ids.devicebuttonsscroll devbar = picker.ids.devicebuttons devbar.clear_widgets() # likely superfluous - it's a new popup # shared: really primary on-device (formerly 'SHARED') (never None) lshared = 'PHONE' if RunningOnAndroid else 'PC' bshared = StorageToggleButton(lshared, shared, picker, devbar, self) devbar.add_widget(bshared) #-------------------------------------------------------------------------- # drives: usb + microsd + cd/bdr (and other usb; hubs work on android too!) # should drivename be truncated for button width? no: devbar now scrolled # note the drivepath=drivepath, else same path for all in gui (see lp5e!) # prior point now moot: lambda here replaced with call to external class #-------------------------------------------------------------------------- for (drivename, drivepath) in drives: bdrive = StorageToggleButton(drivename, drivepath, picker, devbar, self) devbar.add_widget(bdrive) # app: optional, never on PCs; after usb, because less used if appspec != None and self.ids.appfolderselections.active: bappspec = StorageToggleButton('APP', appspec, picker, devbar, self, navup=True) # allow 1 up-nav devbar.add_widget(bappspec) # root: optional, weak|None on android, usable on pcs (system drive on Widows) if root != None and self.ids.rootfolderselections.active: # iff Config, show '/' if access else / broot = StorageToggleButton('ROOT', root, picker, devbar, self) devbar.add_widget(broot) # test: to verify storage-button scrolling sans devices teststoragescroll = False if teststoragescroll: for i in range(5): ltest = 'BREAKME' + str(i) btest = StorageToggleButton(ltest, shared, picker, devbar, self) devbar.add_widget(btest) ##bshared.size_hint = (None, 1) #bshared.text_size = devscr.width, None #bshared.size = bshared.texture_size #'120px' #bshared.texture_size[1] ##bshared.bind(texture_size=bshared.setter('size')) # tbd: show documents folder too? - it's a shared sub, and seems too much #-------------------------------------------------------------------------- # preselect: set the button for the storage in which start path is located; # # some storages nest in others (e.g., APP in PHONE on Android, PC in ROOT # on PCs), so match prefix by longest first = most specific/nested; this # works because longer means more specific in hierarchical filesystems; # no button is selected if no match: e.g., in '/' and root disabled on pcs; # # the sort also handles linux 'os' at /media/me and drives at /media/me/xxx # [but see (feb24) note at similar code above: the 'os' mount os C: was bad], # as well as the fact that linux mounts might be anywhere (a mount in home # (PC) will be longer than home itself - though curr only greps in /media # and /mnt: there will be no storage-type root for mounts in /home in the # chooser, but users can manually enter anything (preselect may be anything); # # see also kin: label_path_string() [1.1.0], rootpath nav above [1.1.0] # (some of these could probably have merged or reused code, in retrospect); #--------------------------------------------------------------------------- # sort bar buttons by decreasing path length = longest (most specific) first btnsbypathlen = sorted(devbar.children, key=(lambda c: len(c.togglepath)), reverse=True) for btn in btnsbypathlen: if abspickstart.startswith(btn.togglepath): # absolute path [1.1.0] btn.state = 'down' break # size the bar for its buttons, so it scrolls: => now in .kv """DEFUNCT ------------------------------------------------------------- # output: 0 0, 100 ##trace(devbar.minimum_height, devbar.minimum_width) ##trace(picker.ids.devicebuttonsscroll.width) def setdevbarwidth(dt): # after widgets are drawn: have sizes devscr = picker.ids.devicebuttonsscroll devbar = picker.ids.devicebuttons # output: 0 692, 13920 ##trace(devbar.minimum_height, devbar.minimum_width) ##trace(picker.ids.devicebuttonsscroll.width) devbar.size_hint = (None, None) devbar.height = devscr.height ##devbar.width = max(devbar.minimum_width, devscr.width) devbar.width = devbar.minimum_width ##trace('devbar.width=>', devbar.width) # it doesn't work yet here Clock.schedule_once(setdevbarwidth) # more bind()s might help too ##picker.do_layout() ##devscr.bind(width=lambda i, v: setdevwidth(0)) ------------------------------------------------------------- DEFUNCT""" #------------------------------------------------------------ # update: binds work better, though kivy scrolling is far too subtle; # max() in defunct above triggered scroll long before buttons clipped; # devbar.height = devscr.height <= needed iff size_hint=(None, None); # # THE KICKER: the next two lines can be done entirely in the .kv, and # even before any buttons are added to the BoxLayout (though buttons' # size still must be arranged as they are built above); .kv is easier # way to setup bindings that will fire later; alas, Kivy is so tersely # documented, that you have to do things wrong a few times first... # # now in the .kv file ##devbar.size_hint = (None, 1) ##devbar.bind(minimum_width = devbar.setter('width')) #------------------------------------------------------------ #--------------------------------------------------------------------- # post that dialog #--------------------------------------------------------------------- self._popups.push(Popup(title='Choose %s Folder' % kind, content=picker, auto_dismiss=False, # ignore taps outside size_hint=(0.9, 0.9))) # need max space self._popups.open() #self._popups.top().border = [0, 0, 0, 0] # works iff background image # this can't be set in the .kv spec: see the note there # and can't be done till after return to kivy loop (omg) def do_this_later(dt): picker.ids.filechooser.view_mode = initstyle() Clock.schedule_once(do_this_later) # asap after open def try_any_nohang(self, func, *args, waitsecs=0.25, onfail=None, log=True): # [1.1.0]+ (feb24) """ -------------------------------------------------------------------- [1.1.0] Run any system call with a timeout to avoid GUI hangs. Returns onfail for exception or timeout, else func(*args)'s result. For portability, runs func(*args) in a thread with a timed join. Also wraps the call in a try handler to catch any/all exceptions. Initially coded for os.listdir() only, then updated for any func. UPDATE: os.isdir() hangs on Windows (but not macOS) too: generalize this to call any folder-access function with a timeout. This now returns onfail if func(*args) either raised an exc before timeout or hung; else returns func result if it ended without exc or hang. Still not universal (no **kargs), but handles this app's use cases. Windows drive enumeration now also uses this, not a custom version. Timeouts can also be had with signals on Unix, but not on Windows, and they are also available in multiprocessing, but not on Android. Threads always finish even if timeout: no simple way to kill, but a shared result object per thread ensures that new results aren't overwritten. Hung thread is left to finish (exit on target return) and modify a result object that is ignored by the calling thread. But set daemon=True so thread auto-killed at app exit if hung that long (unlikely); else inherits False from app process and may delay app exit (which _may_ have triggered a kivy logger loop on exit). INITIAL: run an os.listdir() to verify folder access before ops that rely on it, but run it in a thread with a timeout to avoid lags/hangs for drives that are inaccessible - temporarily or permanently. This includes mounted but disconnected network drives, but also others (e.g., USB drives not yet fully mounted). Returns False iff os.listdir() either failed immediately or hung. Used on all platforms and all contexts - chooser start folder, chooser storage tap/reset, and Main action folder prechecks. Also used for an Android hang in storage_path_app_specific(). [1.1.0]+ (feb24) UPDATE: change waitsecs default timer duration from 0.50 to 0.25 seconds; half a second is a bit jarring when a Windows mapped network drive is offline (as they regularly are), and a quarter of a second should be plenty to notice a ghost. Also added logs=True param to allow callers to turn off traces (traces can be too much, but never happen in released Android apps). -------------------------------------------------------------------- """ def try_any(func, args, result, onfail): try: got = func(*args) # run folder-access call: hangware except: result.append(onfail) # failure, immediate, or later if hung else: result.append(got) # success, now or later; got may = onfail result = [] # new shared object for each thread nohang = threading.Thread( target=try_any, args=(func, args, result, onfail), daemon=True) nohang.start() nohang.join(timeout=waitsecs) # wait till thread finished or timeout if nohang.is_alive(): if log: trace('try thread timeout') sendback = onfail # let thread finish and set its ignored result else: if log: trace('try thread exit') sendback = result[0] # thead finished and set result: ok or exc return sendback # can be onfail for exc, timeout, or got #---------------------------------------------------------------------------- # No-hang helpers # # try_listdir_nohang() => True IFF listable, empty or not: no list returned; # use a try_any_nohang() instead if list is needed too (see examples ahead); # try_isdir_nohang() => False if exc, timeout, or false (callers don't care); #---------------------------------------------------------------------------- def try_listdir_nohang(self, path): return self.try_any_nohang(os.listdir, path, onfail=None) != None def try_isdir_nohang(self, path): return self.try_any_nohang(os.path.isdir, path, onfail=False) # Other try_any() clients: Windows drive enumeration, Android drive name, file-system def reset_main_picker_path(self, pickdialog, pickpath, navup=False): """ -------------------------------------------------------------------- On device/storage name tap in Main folder chooser: reset its path + rootpath. Changing the arg-based property changes the chooser's content (magically). (Update: usually; toggle-down clicks resets have to change ~ elsewhere first.) If navup, rootpath includes '..' after path, to allow nav 1 level up (only). Run os.listdir here to make sure it's still there - USB may be removed, though USB won't be in the devices list if user skipped All files access. Storage permission is checked both before we get here for the folder chooser, and before user confirmation for Main-tab actions. Folder access is checked both here, and just after Main-tab action perm check. ---- UPDATE: storage-type-in-list-but-inaccessible is an error state common on newer Androids for the first run (or later, if A-F-A perm is granted later). This now auto-Cancels the chooser dialog on info-message Okay in this state. Else, the dialog remains open with the failed button down; the display is that of a prior type; and it's unclear which button to auto-select for calls. Users must restart the app as instructed if permission is pending, though the dialog could remain for mount pending with prior button still set if recode callers to check listdir() _before_ this is called or the fail button is set. Leaving the dialog open may be better for pending mounts, but seems worse for pending permissions that require a full app restart - an immediate Back no longer kills the app if popups are open, so there are two steps to close. Closing the dialog is a compromise, with a trivial reopen for pending mounts. ---- NIT: chooser open has been seen to hang briefly (2~7 seconds) while a USB drive finishes mounting on some devices - notably, a Fold4 which had just been granted A-F-A without a restart, and a Note20U in the same state. This might trigger ANR, but has not yet, and it's unknown where the hang occurs (may be here or in the chooser's code). On the Fold4, later pending-mount opens do not hang, and do not return the pending drive in the removables list till it's fully mounted; the next note docs Note20U's odd behavior. PUNT - rare, one time, harmless, very complex to address (thread?). ---- ALSO NIT: a Note20U (Android 10) adds a drive in pending-mount state to the chooser list with inaccessible path '/dev/null' till mounted; and, worse, leaves it in the list after drive unmount as inaccessible (to listdir)... UNTIL the user physically removes the drive's cable from the phone. Both cases yield error popups in this app, and now dialog auto-Cancels. By contrast, a Fold4 (Android 13) correctly does NOT add drives to the list till they are fully mounted; and correctly does NOT leave them in list after they are unmounted - regardless of the cable's status. (The dialog refetches the list on each dialog open, so all new states may require a new dialog.) This is too convoluted to address - and likely bugs in older Androids or Samsung add-ons. The only known fix for Note20U oddness is to catch USB mount/unmount events, and this would be required only for the now-fading Android 10 and earlier (presumably) user base. PUNT; users will have to use some common sense when the now-augmented error dialog here appears... ---- UPDATE [1.1.0] hangs for not-yet-mounted drives on Android or elsewhere will no longer happen here, because the listing call is run with a timeout. This (plus similar timeouts elsewhere) addresses the fold4 case NIT above, as well as network drives that have become inaccessible since dialog open. UPDATE [1.1.0]: the Android run1 hang was eventually isolated, and resolved with a timeout in storage_path_app_specific(). UPDATE [1.1.0]+ on macOS only, due to the elimination of hangs, listdir (et al) now returns immediately if the system permission-verify popup is opened (this formerly paused/hung until the popup was closed). Tweak the message to clarify that an app restart is not needed (hopefully). Also, qualify the opening as "on Android" because it won't apply anywhere else: Windows+Linux have no such popups, macOS popups differ. The opener might be omitted on PCs, but its second sentence applies. For 1.1.0, Android+Windows_Winux don't need to care: it's moot or not. This model is also used at verify_paths() and init_logfile_folder(). -------------------------------------------------------------------- """ """ try: os.listdir(pickpath) except: # nested modal dialog else: # pickdialog.path bound to pickstart via creation arg """ # same, but use timeouts to avoids lags for inaccessible drives [1.1.0] listed = self.try_listdir_nohang(pickpath) if not listed: # hung (inaccessible), or immediate fail (unmounted?) # nested modal dialog noaccessmsg = ( 'Cannot access %s.' # [1.1.0] add dots to all line1 '\n\n' 'If you just granted storage permission on Android, ' # [1.1.0]+ 'please try restarting this app now.' '\n\n' 'Otherwise, retry after ensuring that drives ' 'have been fully mounted and recognized.' ) if RunningOnMacOS: noaccessmsg += ( '\n\n' 'On macOS, you may be able to simply rerun the last ' 'request if you just handled the permission popup.' # [1.1.0]+ ) if RunningOnAndroid and self.android_version() <= 10: noaccessmsg += ( '\n\n' 'On older Androids, you may need to wait for ' 'mounts to finish, or detach after unmounts.' ) if not RunningOnAndroid: # [1.1.0] added noaccessmsg += ( '\n\n' 'On PCs, also check folder permissions and locks, ' 'and network-drive status.' ) closechooser = lambda: self.dismiss_popup(pickdialog) self.info_message(noaccessmsg % pickpath, usetoast=False, nextop=closechooser) # close chooser too else: # storage root accessible now # pickdialog.path bound to pickstart via creation arg up = '/..' if navup else '' # android app-specific only pickdialog.pickstart = pickpath # reset chooser's content (ids.fc.path) pickdialog.rootpath = pickpath+up # remove '..', avoid excs for bad dirs # left align on reset, can be scrolled after def leftalign(dt): pickdialog.ids.pathname.cursor = (0, 0) Clock.schedule_once(leftalign) def do_main_path_pick(self, kindid, popuppath, popupcontent): """ On Pick, copy chosen path to Main tab, and close the picker dialog (which shows the same path next time). Don't save from/to paths until a Main-tab action is run (else may be a bogus path, and would need to catch any manual changes too). No need to clear selection: TextInputNoSelection disables it. (Formerly: clear_selection() here doesn't fix lingering selection handles, and prior selection auto disappears with new popup.) """ # now passed in from .kv file # popuppath = self._popups.top().ids.pathname.text # brittle, that mainpath = self.ids[kindid] # from|to, Main tab mainpath.text = popuppath # from or to path self.dismiss_popup(popupcontent) def dismiss_popup(self, popupcontent): """ Used for all modal popups; this coding avoids setting popup content's close handler after the popup is made (popup needs content, but content close needs popup...). Now just a convenience interface to the popups stack. Kivy Popups also have an auto_dismiss, which runs dismiss() on taps outside the dialog, and is enabled by default. UPDATE: pass Popup's content to avoid closing an overlaid dialog on double taps (see popups stack for more info). Also set Kivy auto_dismiss=False to prevent taps and closes outside scope of popups stack (plus ambiguous+twitchy). """ self._popups.dismiss(popupcontent) # topmost: no longer assumes only 1 modal """ UNUSED def filter_dirs_by_permission(self, itemslist): "" In Main filechooser, rule out folders that can't be accessed. These don't open in the GUI, but trigger uncaught exceptions. UNUSED: only folders accessed by ".." generate uncaught (harmless) exceptions: avoid by setting chooser's rootpath. "" def accessible(folder): try: os.listdir(folder) # caution: may hang except: return False return True return [item for item in itemslist if not os.path.isdir(item) or accessible(item)] """ #======================================================================= # MAIN-TAB ACTION - OPTIONS (DOC) #======================================================================= """ This program runs tasks that are long running, and potentially _very_ long running. These tasks cannot block the GUI, and so must be run in parallel with it. To do so, there are a number of options whose tradeoffs vary per plaform: 1) Simple child processes created by subprocess.Popen() Continue to run if the app or program is left for another, (which pauses the app on Android), and are a portable tool. This is an option on PCs, but not Android. On Android, child processes may be arbitrarily terminated by the Android 12+ 'phantom' process killer, which makes this a nonstarter. On PCs, there is neither pause equivalent nor processes killer, and this is a standard option. This is what was used for scripts and GUIs on Android in the Termux and Pydroid 3 apps. Neither ever saw a process kill, but this varies by phone usage patterns. This mode requires an IPC scheme to signal script-process exit to the GUI process, unless the GUI is polling for EOF on the logfile read by a threas (see Mergeall's GUI launcher) or the appearance of a sentinel file (see Frigcal's launcher). The tkinter GUI did EOF polling with unbuffered output streams + a consumer thread's blocking-read loop + a timer-polled thread queue. This costs cpu and power on phones, but a similar scheme can be used here on PCs to skip IPC altogether. TBD: the next option may work just as well on PCs. YES: thread scheme coded for Android adopted for all PCs too. 2) Threads spawned by the GUI process Continue to run if the program or app is left/paused; are fully portable to PCs and Android; are immune to Android process kills; and require no IPC mechanism to signal their exit (an in-process callback can be run directly in the thread-wrapper's exit code). This is available on Android as an option, and was the first coding. It's only downside seems to be that script exit cannot be reported to users until the app is resumed. Threads may incur a minor speed hit for thread-switching performed within the Python virtual machine (see its global interpreter lock). However, this is likely to be minor, and negligable in multi-minute runs. Per benchmarks, there is _no_ noticable speed difference for the same tasks run as process in Termux and thread in the app (and later, service in the app). Indeed, Android pseudo-nondeterminism makes processes as likely to lose the race. 3) Android-only Foreground services, which run scripts in auto processes Continue to run if the app is left, and are unlikely to be killed by Android, but are fully proprietary and nonportable, and require a simlarly proprietary IPC mechanism to catch exits while paused. This is used by default on Android, for its notifications. The only advantage over threads is that notifications are posted for tasks, which sufficces to report script status and exit to the user while the app is paused. The app's toast is posted on script exit but doesn't appear except in the app; global overlays are possible, but intrusive (and possibly rude). Service processes may require an IPC mechanism for exits, because the script runs in a separate process (callbacks are out); EOF or sentinel-file polling in the GUI may not detect a script exit while the app is left/paused; and EOF polling requires setting stdout to be unbuffered for the script's run. Unix IPC and pipes are unusable, because the process is auto-spawned by Android. As coded, script exit is coded as a proprietary BroadcastReceiver. In hindsight, given that toasts don't overlay other apps and can't appear intil the app is revisited, signal-file polling may suffice. Complication: foreground services keep running if app killed (as supported in kivy/p4a today), so must check if still running on app restart, and reset in-progress state, as well as any postscr callback. This is handled with a service breadcrumb file as coded here. The alternative is killing the service with the app, and seems as hard. UPDATE: punt - Back-button app kills now prohibited if any Main action is in progress, as service-running detection is inaccurate and iffy. Android services can also run a thread, which may avoid IPC, but also requires from-scratch service coding, and has no clear benefit. 4) The multiprocessing module's processes Cannot be used on Android, due to a missing semaphore call, and seems overkill and seems likely to be slower in any event. """ #======================================================================= # MAIN-TAB ACTION: PROCESS #======================================================================= def launch_script_process(self, opname, script, cmdargs, logfilepath, stdin=None, postop=None): """ -------------------------------------------------------------------- On PCs, launch the script as a simple subprocess.Popen() child process, with sentinel-file or EOF polling to detect script exit. DEV: it's not clear that this would be any better than using the portable thread scheme developed for Android intially... UPDATE: now using threads on all PCs. This has not discernable performance negative, but the future remains to be written... -------------------------------------------------------------------- """ return # TBD (sort of) """ # catch app close (prints okay: new process) self.script_running = PROCESS_RUNNING self.on_script_start(opname) # label, gif, buttons self.on_script_exit(opname, logfilepath, logfile=None) # updates gui """ #======================================================================= # MAIN-TAB ACTION: SERVICE+PROCESS #======================================================================= def launch_script_service(self, opname, script, cmdargs, logfilepath, stdin=None, postop=None): """ -------------------------------------------------------------------- Run the Main -tab action's script as an Android ForegroundService, in an automatically spawned process (but not 'sticky': restarts). This allows the action to keep running when users leave the app (like threads), and reports script status and exit with a normal notification while paused (unlike threads). Much of the java interface is automated by p4a (unlike FileProvider!), including both a java service class and pyjnius broadcast-receiver code for process-exit dispatch, though there are glaring seams (see the hurdles list in the service runner). This scheme also requires: 1) buildozer.spec service=arg setting (below), which maps service name to python file, and triggers creation of java service class. Also: +FOREGROUND_SERVICE in permission settings (crucial, undoc!). 2) run_script_as_service.py, which has the main+wrapper code run in the service process that unpacks args and runs the target script's code. This file is auto run as a script on service start below. 3) Broadcast send in the closing code of #2, using an Intent and the same message ID used to create the BroadcastReceiver here. Docs (such as they are): https://python-for-android.readthedocs.io/en/latest/services/ (ditto)/en/latest/apis/?highlight=broadcast#module-android.broadcast https://github.com/kivy/python-for-android/... (for code spelunking) Generated java service class(ES): ~/.../PC-Phone-USB-Sync/.buildozer/android/platform/ build-arm64-v8a_armeabi-v7a/dists/usbsync(TRIAL?)/ src/main/java/com/quixotely/usbsync(TRIAL?)/ServiceRunscript.java Min: service.start(mActivity, argument) Max: service.start(mActivity, 'small_icon', 'title', 'content' , argument) Empty strings on three args pick reasonable defaults in generated Java code -------------------------------------------------------------------- """ # java dir varies by app: here (start + msgid), service wrapper (exit msg) appsuffix = 'usbsync' if not TRIAL_VERSION else 'usbsynctrial' # config gui and msg receiver in this process msgid = self.service_in_progress_state(opname, logfilepath, postop, appsuffix) # pass args as json string argsdict = dict(mainfilepath=script, argslist=cmdargs, stdouterr=logfilepath, onexitid=msgid, appname=appsuffix, stdin=stdin) argumentstr = json.dumps(argsdict) trace('json=>', argumentstr) # start service+process (via generated java class: dirname differs per app!) service = autoclass('com.quixotely.%s.ServiceRunscript' % appsuffix) mActivity = autoclass(ACTIVITY_CLASS_NAME).mActivity service.start(mActivity, '', APPNAME_FULL_OR_TRIAL, 'Running %s action' % opname , argumentstr) # and in buildozer.spec (not + ':sticky' = restart): # services = runscript:run_script_as_service.py:foreground # save breadcrumb file for app restarts while service still running # no longer used: worked, but service-running test is deprecated # and error prone; see check_for_running_service() for more docs """ svcrecord = open(SERVICE_BREADCRUMB, 'wb') pickle.dump([opname, cmdargs], svcrecord) svcrecord.close() """ def service_in_progress_state(self, opname, logfilepath, postop, appname): """ -------------------------------------------------------------------- A method, because used both at service launch, and by app start-up code if service in progress post app kill+restart. We either need to kill a service on app exit (kivy/p4a don't), or detect and reset in-progress state to avoid allowing a new action run. Threads don't need to care: they die with the Android app. On restarts: opname unknown, and postop is MIA. Nit: should msgid vary for full/trail apps? This may or may not matter, users seem very unlikely to run both at once, and maybe we don't need to care. App-name diffs are _forced_ for both the foreground service itself (by the names of generated java-code folders), and OPEN's file providers (by installer which checks that provider name is not already used by an installed app). UPDATE: appname (suffix) now used in broadcast message id here too. It's unknown if this is required for message ids, but was adopted here after installer mandated it for file providers... -------------------------------------------------------------------- """ # catch app close (prints okay: new process) self.script_running = SERVICE_RUNNING self.on_script_start(opname) # label, gif, buttons # start script-exit broadcast receiver msgid = 'com.quixotely.%s.SCRIPT_EXIT' % appname def onexit(context, intent): # always 2 args! self.on_service_process_exit(opname, logfilepath, postop) self.breceiver = BroadcastReceiver( callback=onexit, # run in this process, not service actions=[msgid]) # intent action name, arbitrary self.breceiver.start() return msgid def on_service_process_exit(self, opname, logfilepath, postop): """ -------------------------------------------------------------------- On receipt of the script-exit broadcast message from service/process in the main app/gui process. Same as for thread, but run postop in main app/gui process here and now (the thread variant runs it in the thread runner itself), and the service process closes the logfile. This will be triggered whether the app is foreground or paused, though the toast it generates doens't appear unless and until the app is in the foreground (visible). There are ways to force app overlays, but this is too intrusive. The service does, however, post a notification while running. This notification usually shows up after ~8 seconds on first run (and faster thereafter); is not noticable for very short runs; and is auto-removed on service exit. It also qualifies as useful and ample user notice, and warrants using a service. This callback apparently has 5 seconds to finish; bizarro! UPDATE [1.2.0] there is a rumor (well, just 2 unrecreatable aborts among a gazillion runs) that pyjnius code somewhere in breceiver.stop() is prone to raise an uncaught exception that closes the app. Try to address here, but this may be due to holding Java objects too long (or not long enough), and is too low in the libs to be sure (it's a hard crash, with Java stack details that mean zit here). Such is life with a glitchy stack... -------------------------------------------------------------------- """ if postop: postop() self.on_script_exit(opname, logfilepath, logfile=None) # updates gui try: self.breceiver.stop() except: pass # [1.2.0] maybe recover? trace('exiting broadcast receipt callback') #======================================================================= # MAIN-TAB ACTION: THREAD #======================================================================= def launch_script_thread(self, opname, script, cmdargs, logfilepath, logfile, stdin=None, postop=None, timersecs=0.10): """ -------------------------------------------------------------------- Create a cross-thread object queue, start the script thread, and start a timer-event queue consumer in the main GUI thread. All stdout and stderr in script thread goes to Pipe's queue. Unicode: logs always UTF-8, GUI always run through ascii(). Threads keep running when the app is left/paused, but script exit cannot be reported to the user until the app is resumed on Android (unlike services, they do not post a notifcation). Threads are used on Android as an option (services are default, except on Android 8), and also on macOS, Windows, and Linux PCs. ---- UPDATE: scrolled run output proved too slow, and was replaced with an animated GIF plus on-demand TAIL/WATCH on Logs tab. For this new mode, STATUS_ANIMATION, there is no need for either a timer loop or queue. Instead, use a simple stream flusher, and pass a callback object to be invoked when the thread runner closes the streams flusher after the script's code finishes. This works because it's all one process: the callback is in-process, and the script runner in the thread knows when the script exits. The callback uses kivy's @mainthread to ensure GUI-thread execution for GUI upates triggered by the callback. There is also no need for running display text through ascii(): kivy displays unknown glyphs as rectangles, which is as good/bad. -------------------------------------------------------------------- """ # catch app close, disallow prints self.script_running = THREAD_RUNNING self.on_script_start(opname) # mod label, gif, buttons if STATUS_SCROLL: self.text_size = 0 # reset output dislay/size self.text_line = 0 # num lines for scrolling self.pending_lines = '' # initialize lines batch self.pending_count = 0 self.run_output.text = '' # clear text area in gui if STATUS_ANIMATION: pipe = Flusher(logfile, on_eof=lambda: self.on_script_exit(opname, logfilepath, logfile)) if STATUS_SCROLL: pipe = Queuer() thread = run_script_thread_wrapper( # spawn the script thread script, # path to script file cmdargs, # sys.argv[1:] for script pipe, # logfile proxy + exit signal postop, # postop run by wrapper stdin, # canned input iff used chdir=True, # goto script's folder trace=False) # trace=True for trace2() if STATUS_ANIMATION: pass # no timer loop needed if STATUS_SCROLL: def callback(dt): self.scroll_on_action_timer_callback(pipe, logfile) self.timer_callback = callback # keep a ref, ignore dt Clock.schedule_interval(callback, timersecs) # auto-rescheculed timer @mainthread def on_script_start(self, opname): """ Common script-start GUI code: thread, service, process. Assumes animation coding; former scroll scheme won't work. [1.1.0] Update: set partial wakelock for cpu if screen off. """ self.ids.statuslabel.text = opname # action label self.ids.statusimg.anim_delay = 0.20 # animate gif: 5x/sec # disable action buttons for actionbutton in self.action_buttons: self.ids[actionbutton].disabled = True # acquire partial wake lock on action start [1.1.0] if RunningOnAndroid and not self.wakelock.isHeld(): trace('acquiring wakelock') self.wakelock.acquire() @mainthread def on_script_exit(self, opname, logfilepath, logfile=None): """ -------------------------------------------------------------------- When action exits in animation coding, reenable app closes and prints, stop animation, close logfile, reenable Main buttons. This is now used by thread and service+process/broadcast modes. It's likely thread and broadcast safe: scheduled by @mainthread. Update: also clear arrowheads with .reload() - marginally neater. Nit: could re-deable NAME on Windows here, but punting on this. [1.1.0] Also scrape logifile summary info, and release Android partial wake lock (supports screen timeouts: see App.on_start()). -------------------------------------------------------------------- """ # reenable app close, prints (thread or not), stop WATCH self.script_running = NOTHING_RUNNING self.ids.statuslabel.text = 'Action Status' # back to generic label self.ids.statusimg.anim_delay = -1 # stop Main-tab animation self.ids.statusimg.reload() # reset: clear arrows if logfile: logfile.close() # if thread, not service+process # reenable action buttons (maybe undo) for actionbutton in self.action_buttons: self.ids[actionbutton].disabled = False if self.undo_verboten: self.ids.undobutton.disabled = True # and Windows NAME? - punt # release partial wake lock on action exit [1.1.0] if RunningOnAndroid and self.wakelock.isHeld(): trace('releasing wakelock') self.wakelock.release() # try to scrape and display crucial info from logfile [1.1.0] summary = self.scrape_action_summary_info(opname, logfilepath) self.info_message('%s action finished.%s' % (opname, summary), usetoast=(not ('\n' in summary))) def scrape_action_summary_info(self, opname, logfilepath, tailsize=(10 * 1024)): """ -------------------------------------------------------------------- [1.1.0]: For convenience, try to parse out and present the crucial logfile info in the exit popup. It's arguably hard to view logs on a narrow phone sans landscape rotate, and summmary bits extracted here normaly suffice, especially when verifying counts against syncs on PCs. This must avoid hangs/lags by reading just the end of file (like TAIL and WATCH), and the parse can easily fail and must recover (logs can end with errors sans summaries). Also changed to propagate logfilepath from spawners (service +thread); it originates in Main's confirmed(). -------------------------------------------------------------------- """ def commify(obj): # add commas to str|int number for display if isinstance(obj, str): obj = int(obj) # str=>int return '{:,}'.format(obj) # always returns str try: # read tail only: don't hang gui logsize = os.path.getsize(logfilepath) logfile = open(logfilepath, 'r', encoding='utf8', errors='replace') logfile.seek(max(0, logsize - tailsize)) logtext = logfile.read() logfile.close() if opname in ['SYNC', 'UNDO']: scrape = ( r'^Compared => files: (\d+), folders: (\d+), symlinks: (\d+)\n' r'^Differences => .*?\n' r'^Changed:\n' r'^files => created: (\d+), deleted: (\d+), replaced: (\d+)\n' r'^folders => created: (\d+), deleted: (\d+), replaced: (\d+)\n' ) matched = re.search(scrape, logtext, re.MULTILINE) if matched: groups = matched.groups() summary = ( 'Items compared:\n' ' Files: %s\n' ' Folders: %s\n' ' Symlinks: %s\n\n' ) totals = [int(text) for text in groups] nummods = sum(totals[3:]) if nummods == 0: groups = groups[:3] summary += 'No changes were made.' else: summary += 'File changes: %s\n' % commify(sum(totals[3:6])) summary += ( ' Created: %s\n' ' Deleted: %s\n' ' Replaced: %s\n\n') summary += 'Folder changes: %s\n' % commify(sum(totals[6:9])) summary += ( ' Created: %s\n' ' Deleted: %s\n' ' Replaced: %s') elif opname == 'SHOW': scrape = ( r'^Compared => files: (\d+), folders: (\d+), symlinks: (\d+)\n' r'^Differences => ' r'samefile: (\d+), uniqueto: (\d+), uniquefrom: (\d+), mixedmode: (\d+)' ) matched = re.search(scrape, logtext, re.MULTILINE) if matched: groups = matched.groups() summary = ( 'Items compared:\n' ' Files: %s\n' ' Folders: %s\n' ' Symlinks: %s\n\n' ) totals = [int(text) for text in groups] numdiffs = sum(totals[3:]) if numdiffs == 0: groups = groups[:3] summary += 'No differences were found.' else: summary += 'Differences found: %s\n' % commify(numdiffs) summary += ( ' Same file: %s\n' ' Unique in TO: %s\n' ' Unique in FROM: %s\n' ' Mixed kind: %s') elif opname == 'COPY': scrape = ( r'^Copied (\d+) files, (\d+) folders, (\d+) symlinks' ) matched = re.search(scrape, logtext, re.MULTILINE) if matched: groups = matched.groups() totals = [int(text) for text in groups] summary = 'Copied items: %s\n' % commify(sum(totals)) summary += ( ' Files: %s\n' ' Folders: %s\n' ' Symlinks: %s') elif opname == 'DIFF': scrape0 = ( r'^Files checked: (\d+), Folders checked: (\d+), ' r'Symlinks checked: (\d+), Files skipped: (\d+)\n' r'(?:.*\n)?' ) scrape1 = scrape0 + '^No diffs found\.' scrape2 = scrape0 + r'^Diffs found: (\d+)' summary = ( 'Items checked:\n' ' Files: %s\n' ' Folders: %s\n' ' Symlinks: %s\n' ' Skipped: %s\n\n') matched = re.search(scrape1, logtext, re.MULTILINE) if matched: groups = matched.groups() summary += 'No differences were found.' else: matched = re.search(scrape2, logtext, re.MULTILINE) if matched: groups = matched.groups() summary += 'Differences found: %s\n' uniques = logtext.count('\n- items UNIQUE at [') differs = logtext.count('\n- files DIFFER at [') differs += logtext.count('\n- links DIFFER at [') misses = logtext.count('\n- items MISSED at [') summary += ( ' Same file: %s\n' ' Has uniques: %s\n' ' Mixed kind: %s' ) % tuple(commify(tot) for tot in (differs, uniques, misses)) elif opname == 'NAME': scrape0 = r'^Visited (\d+) files and (\d+) folders\n' scrape1 = scrape0 + r'^Total nonportable names found but unchanged: (\d+)' scrape2 = scrape0 + r'^Total nonportable names found and changed: (\d+)' summary = ( 'Items visited:\n' ' Files: %s\n' ' Folders: %s\n\n') matched = re.search(scrape1, logtext, re.MULTILINE) if matched: groups = matched.groups() summary += 'Names found but unchanged: %s.' else: matched = re.search(scrape2, logtext, re.MULTILINE) if matched: groups = matched.groups() summary += 'Names found and changed: %s.' if not matched: summary = '\n\nCould not extract summary.\n\n' else: commags = tuple(commify(text) for text in groups) summary = summary % commags summary = '\n\n' + summary + '\n\n' # spurious for name and diff - today indicator = r'^\*\*There are (error messages|non-error notes)' if re.search(indicator, logtext, re.MULTILINE): summary += '**This run has errors or notes to inspect.\n\n' summary += 'See Logs for more info.' except: trace('scrape exc:', sys.exc_info()) summary = '\n\nCould not extract summary: see Logs.\n\n' pass # bail on all file or parse errors return summary def init_mergeall_globals(self): """ -------------------------------------------------------------------- UPDATE: this is not required for foreground service+process or process run options, as the script's global state is its own. See also update_mergeall_configs_maxbackups() for related mods. A downside of running Mergeall scripts as strings in a thread: any globals in imported modules aren't reinitialized per run, because the script is rerun in the same process. For example, this causes problems in backup.py, because it uses globals to trigger creation of a backup-name timestamp once per run; if not reset, the same backups folder is created for each mergeall.py run in the app; ouch. The options. 1) Going back to subprocesses instead of threads fixes the global issue, but makes app actions subject to the phantom-process killer for child processes on Android 12+; this is a non-starter. 2) Importing modules with global state and resetting globals before mergeall.py runs (something similar is already done for changing mergell_configs.MAX_BACKUPS changes). 3) Removing the offending modules from sys.modules to force reimports on each run. Of these, 2 seems least risky, given the many changes to imports in python 3.Xs. Note that this pertains only to loaded modules; the top-level script is recompiled from a string each time, and hence starts from scratch. This is also required for mergeall.py only: on SYNC (for backups and copies), and UNDO (for copies), but not on SHOW (which does neither). NIT: this could avoid imports if modules are already in sys.modules, but the import is required on first thread run, and trivial after. NOTE: none of the attibutes reset here show up as function argument defaults, which could make the resets useless - see maxbackups... Running as a thread with exec also makes output tricky (all prints go to the script's output pipe), and requires that the cwd is moot. OTOH, random child-process kills are a Very Bad Thing in syncs... -------------------------------------------------------------------- """ # ./mergeall/ modules are loaded after a run - in just one form # for mod in sys.modules.values(): # trace('$>', mod) # __name__, __file__ # no-op if not running a thread: process imports anew # TBD: also no-op for simple process, iff used on pcs # as coded, this assumes either service else thread, # probably better to call from launch_script_thread() if RunningOnAndroid and self.ids.runactionasservice.active: return else: # for thread runs of mergeall.py only: android or pc trace('ma=> globals reset: backup, cpall') os.chdir('mergeall') # in app-install (unzip) folder sys.path.append(os.getcwd()) # .pyc not text, '.' fails here import backup, cpall # identical to mergeall.py runs backup.datetimestamp = None # run in subfolder, same process backup.pruned = False # reset globals: as if new process backup.firstbkpmsg = True cpall.anyErrorsReported = False os.chdir('..') # back to app's cwd, and import path sys.path.pop() # NO LONGER USED """UNUSED def scroll_on_action_timer_callback(self, pipe, logfile, linesbatch=8): "" On timer event: consume next batch of output lines queued by action thread, if any, and route each line to the line handler. Returning False here means don't reschedule callback in Kivy. This structure avoids blocking main GUI thread while script runs. "" try: line = pipe.linequeue.get(block=False) except queue.Empty: return True # reschedule timer loop # process a batch of lines batchnum = 0 while True: if line == pipe.eofsignal: self.scroll_on_output_line(None, logfile) return False # end of consuming loop else: batchnum += 1 self.scroll_on_output_line(line, logfile) # add to log and gui if batchnum == linesbatch: break # back to top of timer loop elif pipe.linequeue.empty(): break # back to top of timer loop else: line = pipe.linequeue.get(block=False) # back to top of batch loop return True # reschedule next callback UNUSED""" # NO LONGER USED """UNUSED def scroll_on_output_line(self, line, logfile, guimaxchars=(256 * 1024), guibatch=100): "" Handle next output line from action thread. Run from on_action_timer_callback() for lines in queue. This avoids overflflow in the text widget by halving text (TBD). This avoids Unicode issues in the GUI with ASCII mapping (TBD). UPDATE: despite tweaking timer and queue batch settings and halving of text size, scrolling line by line was too slow on phones - thread runtimes > doubled, and runs occasionally hung or terminated. Tried changing to send text to GUI only in guibatch chunks here, This helped with speed, but a few random crashes were still seen. PUNT: on realtime scrolling, and use an animated GIF for run status, along with tail/watch on-demand options in the Logs tab. NIT: the logfile write() here should really catch Unicode errors and use ascii() like the Flusher class does, but we're punting anyhow... "" if line == None: # eof at thread exit # same as animation endthread (plus image) self.on_script_exit(?, logfile) # has changed... # show final lines batch in gui self.run_output.text += self.pending_lines self.ids.runoutputscroll.scroll_y = 0 self.pending_lines = '' self.pending_count = 0 else: # next output line from thread try: #trace2('>>', repr(line)) logfile.write(line) # <= should catch Unicode errors #logfile.flush() # to tail file (slows slightly) except: self.info_message('Cannot write line to logfile.', usetoast=True) line = ascii(line[:-1]) # avoid Unicode probs in GUI, keep \n line = line[1:-1] # strip quotes added by ascii() line = line + '\n' # lines always have a \n at end linelen = len(line) # len after any ascii() expansion if self.text_size + linelen >= guimaxchars: # too big: chop in half trace2('Halving text', self.text_size) fulltext = self.run_output.text # else text widget hangs fulllen = len(fulltext) halflen = fulllen // 2 halftext = fulltext[halflen:] self.run_output.text = halftext self.text_size = fulllen - halflen self.text_line = halftext.count('\n') self.text_size += linelen self.text_line += 1 self.pending_lines += line self.pending_count += 1 if self.pending_count == guibatch: # show batch of text in gui now self.run_output.text += self.pending_lines # insert_text(s, from_undo=False) #self.run_output.cursor = (0, self.text_line) # scroll to line added? - no # textinput.cursor doesn't hscroll (kivy bug): use scrollview wrapping it self.ids.runoutputscroll.scroll_y = 0 self.pending_lines = '' self.pending_count = 0 # gui redrawn auto now or on return to timer loop # alt: force redraws with self.canvas.ask_update()) UNUSED""" #======================================================================= # MAIN-TAB ACTIONS #======================================================================= # All Main-tab action button ids in .kv file action_buttons = [ 'syncbutton', 'showbutton', 'undobutton', 'copybutton', 'diffbutton', 'namebutton' ] def confirm_action(self, message, # text for confirm popup, None=preconfirmed opname, # text of button pressed script, # path to main script file cmdargs, # list of args for script postcon=None, # run iff confirm, before script, in gui thread postscr=None, # run iff confirm, after script, in gui|script thread stdin=None, # string to pipe into thread's stdin popupcontent=None # propagated from callback by preconfirmed callers ): """ -------------------------------------------------------------------- Confirm action with message. Launch script thread/service/process iff confirmed. Also save settings now: configs and from/to paths. Split into steps/continuations to force modal dialogs. Formerly started a GUI timer loop; now just status animation, and wait for thread exit or broadcast message. Simple process mode may start a polling loop for pipe reads; tbd. postcon: called iff confirmed, before script, in the GUI thread. postscr: called iff confirmed, after script, in script thread for thread mode, and in GUI thread in service mode on message receipt. postscr is complicated by fact that app may be restarted while a service is still running. To support app restarts, save opname to a file, and handle UNDO's postscr callback specially on restarts. UPDATE: punt - service-running detection is deprecated and iffy. UPDATE: if message is None, it's assumed that the action has been preconfimed with a still-open popup - skip the normal confirmation dialog, and run the action as if Yes had been pressed. Added for the NAME report|update rewrite ahead; a bit of a kludge, but it's simpler than refactoring confirmed() at this point in the project. UPDATE [1.2.0]: this is now 3 steps instead of 2, to allow for up to 2 modal dialogs before action run: new Android 13+ notification permission infobox + dialog, and/or prior user-confirmation dialog. -------------------------------------------------------------------- """ def confirm_confirmed(popupcontent): """ ---------------------------------------------------------- [confirm_action step #3] On user confirmation of any Main-tab action. Uses multiple variables in the enclosing scope. Storage perms now checked in verify, before this. Notification perm resolved before this, if needed. Pruning + new logfile here will cause the Logs tab to reload its logfile list when switch to Logs tab. [1.2.0] Pruner now filters out new logfile by name passed here, else may prune newly made logfile if host's date/time is old. See make_new_logfile(). ---------------------------------------------------------- """ # close run confirmation self.dismiss_popup(popupcontent) # always log, but auto-clean folder (Logs tab will reload) logfileandpath = self.make_new_logfile(opname) if not logfileandpath: return logfile, logfilepath = logfileandpath # bailed if failed # prune for new log on each run (Logs tab will reload) self.auto_clean_logfiles(logfilepath) # [1.2.0] save impending run's valid logfile path for WATCH self.current_run_logfile_path = logfilepath # save Main paths on each run self.save_persisted_settings_Main() # if-confirmed op in this thread and process, pre-run if postcon: postcon() # allow non-update actions to be killed by Back button? # no: requires detecting others in-progress on restart # self.update_action = (opname not in ('SHOW', 'DIFF')) doservice = self.ids.runactionasservice.active if RunningOnAndroid and doservice: # Android only: foreground service + process (default) # broadcast receiver for script-exit signal logfile.close() self.launch_script_service( opname, script, cmdargs, logfilepath, stdin, postscr) elif RunningOnAndroid or not RunningOnAndroid: # Shakespearean, that # Android and PCs: posix thread in the app's process (original) # run if paused, script-exit via thread wrapper self.launch_script_thread( opname, script, cmdargs, logfilepath, logfile, stdin, postscr) else: # PCs only: child process via subprocess.Popen? (TBD) # sentinel-file polling for script-exit signal self.launch_script_process( opname, script, cmdargs, logfilepath, logfile, stdin, postscr) def confirm_askuser(): """ ---------------------------------------------------------- [confirm_action step #2] [1.2.0] On dismissal (or no-show) of notifications info. Uses multiple variables in the enclosing scope. Appears under system perm dialog on first service action, but this seem better than both convolutinf first open, and pre-andr13 api perm dialog while first action runs. Exception: NAME will run while perm dialog is open; rare. ---------------------------------------------------------- """ if message == None: # preconfirmed, propagate content confirm_confirmed(popupcontent) # uses args in enclosing scope else: confirmer = ConfirmDialog(message=message, onyes=confirm_confirmed, onno=self.dismiss_popup) self._popups.push(Popup(title='Confirm Action Run', content=confirmer, auto_dismiss=False, # ignore taps outside size_hint=(0.9, 0.8))) # wide for phones self._popups.open() #---------------------------------------------------------- # [confirm_action, step #1] # # On all Main-tab actions, post checks and preps. #---------------------------------------------------------- # [1.2.0] verify or ask first, on android 13+ for services self.check_notification_permission(nextop=confirm_askuser) def check_notification_permission(self, nextop): """ -------------------------------------------------------------------- [1,2,0] Android 13 (API level 33) mods notifications permission: it must be called out in the manifest (via buildozer.spec), and apps themselves are now responsible for asking the user to enable the permission with code below (it used to be fully automatic, and asked once at first foreground-service start). Because Play requires API level <= 1 year old for app changes, supporting this was required to release a new version with fixes. Yes, grrr. This is _almost_ automated by p4a's permissions lib, but it should really ask just once, and bail if the user said no (or yes) in the past; unlike storage permission, this is not a core req, and applies iff an action is run as a foreground service. Also: the Android request call simply posts an old-style runtime permission dialog, instead of opening Settings' notification toggles page. This is far less custom than All Files Access. Ref: https://stackoverflow.com/questions/72310162/ how-do-i-request-push-notification-permissions-for-android-13 ---- Coding/timing dilemma: 1) If this is posted on first service run, the infobox will be overlaid by the end-run summary for a fast action, and the system dialog doesn't appear till the infobox is dismissed (a four-level interaction: confirm, summary, infobox, dialog; yikes). The perm also won't apply to the first action run (as before). 2) If this is posted at app first open, it won't be lost under first action run, but there's no reason to ask unless and until an action is run as a service (users may select threads instead, especially if future Androids deprecate fgservice for bkp/sync). #1 is basically how this auto worked before Andr13, except that the infobox keeps the system dialog from appearing while covered. It was confusing before, is probably more confusing with the new infobox, and means no notification for first action. Should either: A- Ask at first open always (like/after a-f-a) B- Drop the infobox to emulate pre-13 behavior C- Make infobox modal with another continuation, like confirm dialog #A is coherent, but overkill if just threads, and convolutes startup UI #B is still confusing when a first action finishes before dialog closed #C is best but complicates code: requires another continuation in confirm Either way, should ask on first _service_ run (not threads) and only if haven't already asked (granted or not). #A doesn't have to check if already asked (and may be able to skip checking if granted with the bizarre+iffy android api method, though unclear for upgrades), but it requests the permission pointlessly if user will run threads. Went with #C. Made infobox here a first modal step of confirm, so that system perm dialog overlays action confirm popup, and must be handled+dismissed before action starts to run. This makes the permission active for the first action too, and defers action run until perm is handled. Caveat: NAME still runs before perm dialog is handled, but it's better than pre-13 (the infobox here gets focus first); the odds of NAME running first are very slim; and this is just a one-time event, after all (users probably won't care). -------------------------------------------------------------------- """ # not for pcs or threads doservice = self.ids.runactionasservice.active if not (RunningOnAndroid and doservice): trace('np=> not relevant platform or mode') nextop(); return # not for androids 8..12 if self.android_version() < 13: # a.k.a. API level 33 trace('np=> not relevant android') nextop(); return # work the way it used # monkey patch: kivy android.permissions grew this later if not hasattr(Permission, 'POST_NOTIFICATIONS'): Permission.POST_NOTIFICATIONS = "android.permission.POST_NOTIFICATIONS" # already got one? if check_permission(Permission.POST_NOTIFICATIONS): trace('np=> Has post_notification permission') nextop(); return # welcome to trial-and-error theater #actcompat = autoclass('android.ActivityCompat') #Manifest = autoclass('android.Manifest') activity = self.get_android_activity() permname = Permission.POST_NOTIFICATIONS # don't ask again if user previously denied; yes, True=denied (!) if activity.shouldShowRequestPermissionRationale(permname): trace('np=> User denied post_notification') nextop(); return # explain the system dialog first, then open it msg = ('This app uses Android\'s notifications permission for services.' '\n\n' 'In the following dialog, please grant this app this ' 'permission. This will allow this app to post a notification ' 'that lets you know when a foreground service is running ' 'to process a Main-tab action.' '\n\n' 'The service will work whether you grant this permission or not, ' 'and you can change this permission option in your phone\'s ' 'Settings panels at any time.') self.info_message(msg, nextop=lambda: (request_permission(Permission.POST_NOTIFICATIONS), nextop()) ) def verify_paths(self, frompath, topath): """ -------------------------------------------------------------------- Paths are editable and drives can be removed: verify that both name existing directories, and are accessible to the app. Either can be None: NAME uses only FROM, UNDO uses just TO. Storage permission is checked both here by Main-tab actions, and prior to opening popup by folder chooser. Folder access is checked both here (post permission) and on folder-chooser storage buttons. Subtle: permission is now checked here instead of post confirmation, else the listdir here may fail due to no permission without reasking for permission, and the restart prompt must follow granting perms. Though should happen only for MANUALLY typed paths: the chooser asks for perms before opening its popup too... UPDATE [1.1.0] avoid lags/hangs for inaccessible paths prior to Main action runs. This may arise for network or other drives unmounted since a chooser pick, as well as arbitrary paths entered manually. UPDATE [1.1.0]+ tweak the message to clarify that an app restart is not needed on macOS (probably), and qualify the opening as "on Android" because it won't apply anywhere else. See reset_main_picker_path(). ---- UPDATE [1.2.0] On Android only, avoid exposing the app's source code for a manually entered path in FROM. Do likewise to avoid harming app source code for manual entries in TO. Else, for example, a SYNC/COPY with a manual FROM='..' copies the app's code in full, and the same actions with TO='..' destroys the app. Formally, both absolute and relative paths may refer to the app's code when input manually (i.e., not by chooser pick). Because the app first chdir()s into the ./mergeall subfolder when actions are run in both thread and service modes on Android, the following relative paths apply: -=works, *=prevented by simple abspath==abspath, x=inaccessible - name is a subfolder in mergeall/ (okay to expose) - '.' is mergeall/ (okay to expose, THOUGH a mod version) * '..' is the app's full code (bad) * '../..' is a folder with app/ (the code: bad), runcounter, and settings x '../../..' is inaccessible, and issues an error popup And the following absolute paths apply both post- and pre-chdir() (checked pre-chdir() here, when '.' is still the app's code folder): - (mergeall) /data/data/com.quixotely.usbsync/files/app/mergeall * (. = cwd) /data/data/com.quixotely.usbsync/files/app [works] * (..) /data/data/com.quixotely.usbsync/files [works] - (../..) /data/data/com.quixotely.usbsync [works! =rel ../../..] x (../../..) /data/data [inaccessible] BUT: all the abs paths also work with /data/user/0 instead of /data/data, and this form is not detected by a simple comparison to abspath('.'), and Python's os.path.realpath() does NOT map either form to the other, and os.path.relpath() does NOT return the same string for each form. THOUGH: os.path.samefile(form1, form2) is True on Unix (incl Android) per inode testing (via stat()) that is immune to all path-form diffs. HENCE => must either: (1) Use [hangable] samefile(input-path, *code-folders) instead of ip == cf (2) Punt on '..'s and fail if split(abspath))[1] == split(appprivate)[1] GO WITH #1, because #2 is hampered by the fact that different filesystem roots may lead to the same place independently (split[1] isn't enough). #1 requires try-any_nohang() to avoid unlikely hangs for the stat call, as well as except catches for inaccessible roots above code (may vary); it may also burn time on paths that don't make sense, but phones are fast. Caveat: try_any_nohhang() will return False if exc, timeout, or false. The exc and false cases are what we want, but a timeout may occur for timeout when the path refers to a private folder (a false negative). Punt; this check seems hardly worth the effort already put into it... See also storage_path_app_private() for more on the folder structure; per logcat: (PPUS) app-priv: /data/user/0/com.quixotely.usbsync/files. Defunct note: this also assumes that os.path.abspath() won't hang: it runs only os.getcwd() plus purely syntactic path transforms. Alt: disallow _all_ relative paths, but this seems too extreme, and doesn't handle absolute-path refs to the code folder or parents. This is only an issue on Android, because Buildozer apps don't do anything about hiding source automatically: .py files are in the run folder, which is app private, but completely open to the app itself (and thus exposed to users via unchecked pathname entries!). PC apps are built with PyInstaller, which normally bundles just bytecode in formatted exes or files, and ".." is just the app unzip folder (Windows+Linux) or application folder (macOS)... with the following glaring caveat. Also in [1.2.0], avoided exposing the app's source-code files in the Windows PyInstaler exe, per this from pc-build/(windows-spec-file): # # [ML] [1.2.0] Tighten up source-code visibility with excludes=[...] # in added Tree(), so raw .py source-code files are not exposed in # PyInstaller _MEI* Temp unzip folders (an undocumented side-effect # of a Tree() spec-file insert required+prescribed by Kivy!). # # This applies only to the GUI's code (Mergeall's code used for # file ops is fully open source); and only to Windows (Linux and # macOS exe/dir don't add a Tree('.') and so don't include .pys, # and Andoid's Buildozer has different source-code policies). # # PyInstaller used to have a '--key' encrypted-bytecode option, # but dropped it in 6.0 because it was too weak: keys were shipped # in PI exes, and Github has tools to extract+decrypt+decompyle. # Encryption or not, PyInstaller packages code, but hides it from # casual eyes only; this might encourage open source and free. # (Ref: https://github.com/pyinstaller/pyinstaller/pull/6999). # # Also closed up a code leak for manually input paths on Android. # In the end, this all may be a moot point: it's probaby easier # to make a workalike app than steal+mod+rebuild this one's vast # Python code, and Play payola and audience naiveté remain hurdles. # UPDATE [1.2.0]: also changed to catch invalid relative paths in the GUI, instead of letting them reach and fail in mergeall.py. This can happen for paths like '../app' which are valid in the cwd here, but not after chdir to mergeall/ to run the action: must check in the context if the mergeall/ forlder for relative paths. Such paths are not kicked out as app-private, because they are invalid in mergeall/, and hence skipped by app-private tests. Current coding runs os.chdir() > once, but this is too fast to matter. -------------------------------------------------------------------- """ # confirm permission or ask again before listdir, force retry if ask if not self.get_storage_permission(): return False # verify access next checks = [('FROM', frompath), ('TO', topath)] # check from and to for (kind, path) in checks: if path == None: continue # skip if unused by action # [1.2.0] checks vary is_abs_path = os.path.isabs(path) # check cwd does not matter is_rel_path = not is_abs_path # check relative to mergeall/ # check isdir # os.path.isdir(path) can also hang on Windows [1.1.0] if is_rel_path: os.chdir('mergeall') # [1.2.0] isdir = self.try_isdir_nohang(path) if is_rel_path: os.chdir('..') # [1.2.0] if not isdir: self.info_message('%s is not a valid folder path.' % kind, usetoast=True) # [1.1.0] expanded return False # cancel action # check app-private # [1.2.0] disallow app's code folder as FROM or TO if RunningOnAndroid: exposed = False samefile = os.path.samefile if is_abs_path: # chdir won't matter for p in ['mergeall', '.', '..', '../..']: if self.try_any_nohang(samefile, path, p, onfail=False): exposed = True break else: # chdir will matter os.chdir('mergeall') for p in ['.', '..', '../..', '../../..']: if self.try_any_nohang(samefile, path, p, onfail=False): exposed = True break os.chdir('..') if exposed: self.info_message('%s cannot be app private.' % kind, usetoast=True) return False # cancel action # check listdir """ try: os.listdir(path) except: """ # same, but use timeouts to avoids lags for inaccessible drives [1.1.0] if is_rel_path: os.chdir('mergeall') # [1.2.0] listed = self.try_listdir_nohang(path) if is_rel_path: os.chdir('..') # [1.2.0] if not listed: # hung (inaccessible), or immediate fail (unmounted?) # differs from chooser: about folder, not drives list noaccessmsg = ( '%s is not accessible.' '\n\n' 'If you just granted storage permission on Android, ' # [1.1.0]+ 'please try restarting this app now.' '\n\n' 'Otherwise, retry after ensuring that your ' 'device has finished mounting your drive in full.' ) if RunningOnMacOS: noaccessmsg += ( '\n\n' 'On macOS, you may be able to simply rerun the last ' 'request if you just handled the permission popup.' # [1.1.0]+ ) if not RunningOnAndroid: # [1.1.0] added noaccessmsg += ( '\n\n' 'On PCs, also check folder permissions and locks, ' 'and network-drive status.' ) self.info_message(noaccessmsg % kind, usetoast=False) return False # cancel action return True # all passed: proceed with action, if user confirms def check_older_android_removeable_updates(self, updatepath): """ -------------------------------------------------------------------- Because Android's SAF sucks in full. (And the Android 10- audience is shrinking rapidly.) ((And most users will sync only PC-to-phone anyhow.)) Prohibit non-appfolder removable-drive updates in Android 10- for SYNC, UNDO, COPY, and NAME, which would all fail with error messages in the logfile. Called on action picks: this can't be addressed in the chooser, because it's okay to read non-app folders on removables on all Androids, the action selection is not known until the popup is closed, and users can manually type/paste paths into Main. The chooser also doesn't preset paths to app folder paths instead of drive roots for removables on 10-, because it's valid and normal to choose other folders on these drives for reads (e.g., PC=>phone syncs). -------------------------------------------------------------------- """ message = ('CAUTION: you have selected a removable-drive folder that ' 'does not support updates on your version of Android, ' 'and a Main-tab action that may modify the folder.' '\n\n' 'Per this app\'s docs, Androids 11 and later allow reads and ' 'updates anywhere on removable drives. Androids 10 an earlier ' 'allow reads in any removable-drive folder, but writes are ' 'supported only in the drive\'s Android/data/com.quixotely.%s.' '\n\n' 'Because any updates outside this folder will fail with a ' 'logfile error message, this action must be cancelled. ' 'Please relocate your content on the drive as needed.') if RunningOnAndroid and self.android_version() <= 10: # through 10/Q, api 29 appsuffix = 'usbsync' if not TRIAL_VERSION else 'usbsynctrial' appfolder = '/Android/data/com.quixotely.' + appsuffix removables = self.storage_names_and_paths_removables() for (rname, rpath) in removables: if updatepath.startswith(rpath): # removable if updatepath.startswith(rpath + appfolder): return True # proceed: in app folder else: self.info_message(message % appsuffix, usetoast=False) return False # cancel: outside app folder return True # proceed: does not apply or not removable def label_path_string(self, path): """ -------------------------------------------------------------------- [1.1.0] Format path with its volume label. Now shown in action confimation popups, both to remind users of paths, and associate logical label with physical path. This was a compromise: it's important to remind users of the FROM and TO subjects in the popup, and useful to show the label with the path. But displaying the label in Main tab FROM and TO fields is problematic, because these are user-editable fields, and there are few delimeters that would work for all platforms in path strings. E.g., "