""" =============================================================================== CALENDAR-FOLDER STORAGE (part of Frigcal) About the SAF... This app's code is bifurcated for Android's permissions rules: • On all PCs, it uses the usual, portable, and interoperable filepaths and POSIX calls for the calendars folder. No permissions are required. • On Android, it uses Android's proprietary SAF picker for the calendars folder, which grants required permission but also forces the app to use content URIs and the arguably tortuous SAF API for storage processing (except where file descriptors can be had at the bottom of the code). Using SAF for the calendars folder on Android roughly _doubles_ the amount of storage code just for the sake of Android and obscures this app's implementation substantially. Moreover, SAF's API is wildly convoluted compared to POSIX/filepath equivalents (see ahead), and the SAF is widely known to be very slow, which matters even more on phones. By contrast, quixotely.com's PPUS sync app uses Android's All Files Access (a.k.a. MANAGE_EXTERNAL_FILES) permission because it requires both broad storage access and optimal storage speed. This permission also allows a single version of code based on POSIX and filepaths to be used on all platforms, thereby avoiding code bifurcation and its inherent costs. There is some extra code for Android storage (e.g., asks) but it's minor. In the end, All Files Access was deemed overkill here because, unlike PPUS, this app uses only one folder in storage, and its speed requirements are negligible (loads and saves are rare events, not core behavior). Also unlike PPUS, using All Files Access would likely preclude distributing this app on the Play store, which is officially a downside, though the rules that Play regularly imposes on developers may qualify as a negative too. The SAF itself may have been an interesting idea in theory - accessing both physical and non-physical storage the same way - but it doesn't seem to survive contact with reality. It's bloated, functionally limited, and almost non-sensical at times (again, see the code ahead); there is no compelling reason to support non-physical storage in some apps (clouds and networks can be a Bad Thing, especially for private calendar data); and POSIX/filepath code is both much simpler and works on PCs too. As it stands, developers must choose between supporting Android or PCs, or budgeting for double the effort to support both. The former is a choice that naturally benefits monopolistic agendas and the latter is an added cost that Google seems unlikely to reimburse. Can't we do better? Related: main.py's storage_path_app_private() is a platform-specific path for storage of admin files, not calendars, which also varies by PC kind. =============================================================================== """ from common import * # shared collector module from hangless import try_any_nohang, try_listdir_nohang # hang-free IO calls # ICS-file parser/generator (3rd-party lib, uses pytz, both shipped with app) try: import icalendar except: # should never happen, but... print('Required and included icalendar or pytz packages not present') sys.exit(1) from icsfiletools import * # global tables, icalendar objects from icalendar.parser import unescape_char # currently unused from icalendar.parser import escape_char, foldline # for Unicode file text #============================================================================== # 🗓 ICS events data structures: internal format of calendars and events #============================================================================== """ ------------------------------------------------------------------------------- Global calendar and event data structures ICS calendar files: - Are loaded into the following in-memory data structures on calendar loads, which occur on app startup and calendar-folder switches - Are converted back to iCalendar text from this form and written to ICS files on main-menu Save, which is prompted on app exit and folder switches The GUI is drawn from the in-memory form, and event updates are applied to it until user Save requests. The in-memory form is three global tables: - CalendarsTable embeds icalendar-library Calendar objects created by ICS-file parses and used for ICS-file generations. Calendar objects may also be created by main-menu New. - CalendarsDirty designates user changes and is separate to minimize code changes in earlier Frigcal vesions. - EventsTable is an index that combines all ICS files' events and uses two-level dictionary table nesting: [date][uid]. All three are largely built in storage.py's calendar parsing. Structure: CalendarsTable = # Parsed file's raw calendar data (all data) {icsfilename: # key: basename of .ics file parsed (with '.ics') icalendar.cal # icalendar.cal Calendar object, from parser or New } CalendarsDirty = {icsfilename: # key: basename of .ics file parsed Boolean # True=calendar changed since startup or latest save } EventsTable = # Union of all events in all ics files (indexed data) {(m, d, y): # key: date object for event start (and display) date {uid: # key: event's required unique id, for fast deletes (.calendar, # ics file basename, link to icalendar in CalendarsTable .summary, # event summary text (aka Title in GUI) .description, # event description text (aka Note in GUI) .category, # category (just 1/first used) .uid, # event's required unique id, for easy access .orderby, # creation (or else mod) date: display order sort .veventobj) # reference link to vevent in icalendar.cal object } } More about EventsTable: Events from all calendars are stored in a 2-level table, indexed by date and unique id (UID) for quick access to both displayed and parsed data. Because events in the table are stamped with their calendar name, there is no need to segregate by a 3rd calendar dimension (an initial design) - the GUI displays the _union_ of all calendar files' events, and a single combined table still supports display grouping by calendar, via sorts. Events in the table also have by-name links back to their parsed icalendar object for fast in-place updates in memory (files are regenerated only on Save), and GUI month windows keep UID-indexed tables of displayed events for quick deletion. This model assumes events have UIDs, but the iCalendar standard requires these. Subtlety: IN-PLACE CHANGE to mutable objects shared by multiple references is crucial in this model. EventsTable-index objects are referenced from registered GUI event handlers, and there is just one copy of each CalendarsTable parsed icalendar array, referenced from index objects and used at file regeneration time. ------------------------------------------------------------------------------- """ # loaded calendar/event tables, per above (singletons) # changed in memory, transferred to/from external ICS files on demand # reset in-place before loads for Settings-screen folder changes CalendarsTable = {} # {icsfilebasename: icalendar.cal.Calendar()} CalendarsDirty = {} # {icsfilebasename: Boolean} [1.1] EventsTable = {} # {Edate(): {uid: EventData()} ] class Edate: """ ------------------------------------------------------- Encapsulate and name event date numbers. != ViewDate, used to manage GUI day in monthgui.py. ------------------------------------------------------- """ def __init__(self, month, day, year): self.month = month # a namedtuple may do some of this, but not all self.day = day self.year = year @staticmethod # copy datetime.{date, datetime} object attrs def from_datetime(dateobject): # staticmethod optional if 3.X class calls only return Edate(dateobject.month, dateobject.day, dateobject.year) def to_datetime_date(self): return datetime.date(**self.__dict__) # copy attrs to a datetime.date object def as_tuple(self): return (self.month, self.day, self.year) def as_string(self): return '%02d/%02d/%4d' % self.as_tuple() # formatted display, 'MM/DD/YYYY' def as_nice_string(self): # [4.0] month-labeled and locale-neutral 'mmm D, YYYY' # don't zero-pad day with %d: users aren't computers return self.to_datetime_date().strftime(f'%b {self.day}, %Y') # 'Nov 8, 2025' # support use in dictionary key, comparisons, sort key def __hash__(self): return hash(self.as_tuple()) # dictionary key (else by addr) def __eq__(self, other): return self.as_tuple() == other.as_tuple() # comparisons (else by addr) def __lt__(self, other): # sort keys (else fails) """ [1.4] order events on dates by year first (y-m-d), not month; but keep as_tuple() format (m-d-y): it's also used for displays; was originally: return self.as_tuple() < other.as_tuple() """ return ((self.year, self.month, self.day) < (other.year, other.month, other.day)) class EventData: """ ------------------------------------------------------- Maximal attributes set for an event. Used for both EventsTable index and GUI widget values. ------------------------------------------------------- """ def __init__(self, uid='', calendar='', summary='', description='', category='', orderby='', veventobj=None): self.uid = uid # event unique id, required by iCalendar std self.calendar = calendar # calendar file's basename (not path) self.summary = summary # for event summary in month, elsewhere self.description = description # for extra text in footer, edit dialog self.category = category # for colorizing event summary entries self.orderby = orderby # for ordering events entries initially self.veventobj = veventobj # a reference to icalender object: updates, deletes # or: for (name, value) in kargs.update(defaults): setattr(self, name, value) def copy(self): return EventData(**self.__dict___) # use attrs dict for keyword args def __eq__(self, other): return self.__dict__ == other.__dict__ # comparisons (else by addr) #============================================================================== # 💾 BASE CLASS -- common code for both PCs and Android #============================================================================== class CalendarsStorage: """ ------------------------------------------------------------------ Implement file IO ops per the hosting platform's permission rules. One instance of one sublcass, created by and stored in App instance. Base class with common/shared methods for Android and PCs subclasses. Example usage in App: self.storage = \ CalendarsStorage_Android(self) if RunningOnAndroid else CalendarsStorage_PCs(self) self.storage.run_with_calendars_folder( self.storage.load_all_calendar_files, forcedask=?) self.storage.run_with_calendars_folder( self.storage.backup_andsave_all_changed_calendar_files) ------------------------------------------------------------------ """ def __init__(self, app, maxbackups): self.app = app # link to state+ops in main.py's App() self.maxbackups = maxbackups # from App.settings, changed by Settings self.folder_details_shown = False # show dialog extra info once pre run self.any_file_ops_since_empty = False # warn about saves sans folder access def clear_calendar_and_event_tables(self): """ ------------------------------------------------------------------ Clear global tables of loaded calendars and events to be empty prior to loads, for Settings-screen folder changes. This must be an in-place change. It cannot simply assign {}s to these names, because they are imported with 'from' by other modules, which share objects and would not notice name changes here (see LP6E!). ------------------------------------------------------------------ """ for table in (CalendarsTable, CalendarsDirty, EventsTable): table.clear() # same object in this module and others def ask_calendars_folder_info(self, nextop, forcedask, rawnextop): """ ------------------------------------------------------------------ Folder ask step 1: called on all platforms before file ops when there is no access to a calendars folder. Popup info about the ask, then run a platform-specific asker, which later runs the nextop platform-specific load or save (three levels). Context: - forcedask is True only for Settings screen's calendar-folder mod. - rawnextop is the true next step after ask, for save+chdir caution. This step is generic enough to be the same for Android and PCs and multiple ops, but the message must be tailored for Android: see default_calendar_folder_path(). FORMATTING: Caveat: on wsl2, [size=20sp] seems not to scale to font size as it does in Help/About, likely due to default font size diffs and a wsl2 glitch; it _should_ scale per Kivy's 2.3.1 code, which calls dpi2px() for 'Nsp' to scale (despite the fact the Google's AI said it is not supported...). Update: [size=20sp] was dropped because it also doesn't change if the user changes globalfontsize in the Settings screen (there's no way to bind it to a property). Instead, headings now use color, here and in Help and About text. This is subtle: color must work in both Dark and Light mode, alt fonts (e.g, DejaVuSans) makes text look weak, and italic and underscores are already used much in non-header text. This oddball need not match Help/About style per se, but it does. DETAILS EXTRA: The details part of this message might be limited to first-run asks, when self.app.runnum == 1, because it's mostly just for new users setting the folder initially; after that, there is similar info in Help's calendars coverage. Update: as a compromise, show details just once per run, instead of either always or on first run only. Probably a useful tip on future Settings folder mods, but they may be annoying if > once in a run. Nit: the spacing for details is less than in first-run terms of use and Help/About, but it should match the caution up next and we don't want the caution to be missed by users. Trivial but significant. SAVES EXTRA: rawnextop is used to detect calls from run_with* for a main-menu Save, when there are calendar changes and the prior calendar folder cannot be accessed due to a delete, rename, or app-permission removal after a load while app is running. This is a dicey UX boundary case. There are three contexts that invoke run_with_calendars_folder() and its asks: - Initial calendar loads, where nextop is load_all_ics_files() - On-demand calendar-folder changes, where nextop is load_all_ics_files() - On-demand calendar saves, where nextop is backup_and_save_all_changed_ics_files() The first two already load all calendars next by design. Saves do not reload calendars, because doing so would wipe out the very changes that the user is trying to save, but not reloading means that the GUI won't reflect the folder after Save. Reloading post saves would load only calendars that were saved in the new folder, and that's not right either. The best Save can do in this case is to save changes to a newly picked folder, and leave it to the user to sort out the aftermath. This is also so unlikely as to be either impossible or attributable to user error. But to minimize risk, run_with* asks now rely on this method to detect this rare use case and augment the pre-select info_message so the user is aware of the consequences. The save proceeds in the new calendar folder if the user picks one, however, to save the calendar changes. See main.py's on_menu_save for parallel notes and context. Subtlety: we're interested only in Save sans folder access when the in-memory calendars reflect a real folder, such that saves will create skew. To detect this, we must check rawnextop for the save method but also whether there were prior calendar loads since either startup or the last folder change. In either case, it's okay to not yet have folder access on Save if calendars have been added but not loaded or saved to files (e.g., after adding new calendars on first run or switching to a new empty folder in Settings). This cannot check CalendarsTable, because new calendars are added there pre load/save. Instead, any_file_ops_since_empty flags file ops since the last empty-folder state, so we can know if in-memory calendars reflect a real folder's content and thus may yield skew on Save to a new folder. Update: this moved from separate popup to augmentation here, to avoid popups overload (three is too much). Postscript: the save-sans-folder state might have been avoided if Save saved _all_ calendars, instead of just changed calendars. The new folder would then always be complete. But partial saves are by design because saving all calendars on every save might be noticeably slow, especially on slower phones. In the end, speed is more important than avoiding a wildly unlikely UI state. ------------------------------------------------------------------ """ first_run_extra = save_caution_extra = '' # provide extra details once per run only, info in Help later if not self.folder_details_shown: self.folder_details_shown = True first_run_extra = ( '\n\n\n' '[H2]Details[/H2]' '\n\n' 'This app creates and uses calendars formatted per the ' 'iCalendar standard and stores them in a folder as portable ICS ' 'files with extension .ics. Calendar [i]loads[/i] combine all the ' '.ics files in this folder, and calendar [i]saves[/i] update all ' 'changed .ics files there after creating backup copies.' '\n\n' 'To share your calendars with other programs and devices, simply ' 'sync or manually copy .ics files to or from your chosen calendars ' 'folder. Some other calendar apps may require imports or exports ' 'for ICS files, but this app uses them directly on all platforms.' ) # alert user about picking a new folder for saves, not a separate popup if (self.any_file_ops_since_empty and rawnextop == self.app.storage.backup_and_save_all_changed_ics_files): save_caution_extra = ( '\n\n\n' '[u][H2]Caution[/H2][/u]: save sans prior-folder access[/color][/b]' '\n\n' 'The external calendars folder to which your changed calendars ' 'correspond is no longer accessible.' '\n\n' '[i]Proceeding[/i] with this save by choosing a new folder in the ' 'following chooser will result in only your changed calendars being ' 'saved to the new folder, and the app\'s GUI will not be reloaded ' 'to reflect the new folder\'s contents. You may have to reconcile ' 'prior/new folder differences manually after the save.' '\n\n' '[i]Cancelling[/i] this save in the following chooser will avoid folder ' 'and GUI skew but will not save your calendar changes. If you removed ' 'app permissions or deleted or renamed your folder since this app was ' 'started, you may wish to undo your mods now and retry the save.' ) reason = ( 'You have asked to change the calendars folder' if forcedask else 'This app does not have access to a calendars-storage folder' ) suggested = ( '' if RunningOnAndroid else 'For convenience, a Frigcal subfolder in your home ' 'folder is available as a suggested default.' ) self.app.info_message( '[b]Calendar folder selection[/b]' '\n\n' f'{reason}. ' 'In the next screen, please select the folder to be used for ' 'storing your calendar files.' '\n\n' 'You can select any folder that you can access on your device, ' 'including one that is part of content you normally sync with ' f'other devices. {suggested}' f'{first_run_extra}' f'{save_caution_extra}' , usetoast=False, nextop=nextop ) # and the folder chooser comes up next, followed by nextop def no_calendar_files_message(self, folderPathOrUri): """ ------------------------------------------------------------------ Shared by POSIX/filepath and SAF/content-uri calendar loaders. folderPathOrUri is a py str: POSIX pathname or SAF content URI. Called only before load if have calendars-folder access by a prior op or a newly provided chooser reply, and folder is empty. This can happen on startup and on later Settings folder changes. Clears any_file_ops_since_empty because this is a known state in which internal calendars do not correspond to an external folder, and a Save cannot write partial folder contents. All later loads and saves set this flag because internal reflects external. TBD: automatically add a default calendar file if none in folder, or expect the user to add one with New? Either policy may be seen as onerous by some users: manually deleting an unwanted calendar versus manually adding a new one. CHOICE: assume user must run New run next, and show this info_message with tips. Further assume that user will run a Save to write the calendar's ICS file if and when desired. No default cal required. UPDATE: but must not skip calendars load if no calendars in folder, because this may occur after a change-folder in Settings, and the GUI should reflect the new empty folder. Else, a Save would save the prior folder's calendars in the new and unrelated folder. This is acheived by coding in the load methods, not here. ------------------------------------------------------------------ """ self.any_file_ops_since_empty = False self.app.info_message( '[b]No calendars in folder[/b]' '\n\n' f'....{folderPathOrUri}' '\n\n' 'If you are using this folder for the first time, either:' '\n\n' '1) Add a [i]first calendar[/i] to this folder now using the ' 'main menu\'s New Calendar, and use its Save Calendars to ' 'create the new calendar\'s file when desired.' '\n\n' '2) Copy [i]existing calendar files[/i] to this folder, and ' 'load them by either restarting this app or changing to the ' 'same folder again in the Settings screen.' '\n\n' 'To use a different folder altogether, see Settings.') # and end UI interaction for calendar loads (after reload!) def parse_one_ics_file(self, pyFileObject, icsfilebasename): r""" ------------------------------------------------------------------ ICS events file parser: creates/returns in-memory data from a .ics file. Load events at startup from ICS file whose open file object is passed to pyFileObject. icsfilebasename is the ICS file's basename (a py str), regardless if it's from a pathname or URI, and this is run within a non-GUI thread by its caller so it cannot block the GUI. Run by a platform-specific version of loading code, which uses POSIX/paths on all PCs and SAF on Android. Backups and saves instead have a platform- neutral driver that calls out to platform-specifc parts; which is better? The caller finds, opens, and closes pyFileObject in platform-specific code. Don't reinitialize tables here: they are already imported by object via 'from'! Any errors here are caught and displayed by the caller: the data may be bad. VTEXT is a str subclass, and works as is: no extra str() conversion is needed. icalendar uses the UTF8 iCalendar default Unicode encoding scheme throughout. from_ical parse auto-unescapes any "\X" icalendar sequences and does Unicode decoding. to_ical generate auto adds "\X" escapes if needed and does Unicode encoding. See also the note about reading all at once here, in generate_ics_files() ahead. Unicode NB: this formerly used icalendar.parser.DEFAULT_ENCODING for calendar files' Unicode encodings, with a note about not hardcoding UTF-8. In FC 4.0, both callers hardcode UTF-8 because it's used by iCalendar per spec. This may be an issue for non-conforming calendar tools that use other encodings, but there is no source code to be changed in frozen apps/executables, and making this a user setting may not help (encoding might vary per ICS file). Bytes works in icalendar API too, but it's not clear how this would help. ------------------------------------------------------------------ """ # TBD: make calendar/event tables attrs of this self, not globals? global CalendarsTable, CalendarsDirty, EventsTable # all filled here trace('Parsing', icsfilebasename) # read ics text, caller handles open, close, exceptions icstext = pyFileObject.read() # all at once: small files # parse ics text: 3rd party package(s), huge dependency icsarray = icalendar.cal.Calendar.from_ical(icstext) # .cal optional but explicit CalendarsTable[icsfilebasename] = icsarray # add to global calendars table CalendarsDirty[icsfilebasename] = False # no changes yet at startup self.any_file_ops_since_empty = True # extract data used here, index by start date and uid for event in icsarray.walk('VEVENT'): # for all events in parsed calendar # icalendar.prop.vDDDTypes: .dt is datetime.{date or datetime} with .m/d/y startdate = event['DTSTART'] # required start date datestamp = event['DTSTAMP'] # required modtime: display order if no created createdon = event.get('CREATED', None) # optional createtime: for display order # icalendar.prop.vText (VText): .to_ical() is bytes, str() decodes auto uniqueid = event['UID'] # required id, for update searches labeltext = event.get('SUMMARY', '') # for main label and view/edit dialog extratext = event.get('DESCRIPTION', '') # for view/edit dialog, main text category = event.get('CATEGORIES', '') # for coloring and view/edit dialog if isinstance(category, list): category = category[0] # if multiple, use just one (first) for coloring # vdates: .dt is a datetime.date/datetime object # transfer to custom date type for better control here eventdate = Edate.from_datetime(startdate.dt) if createdon: orderdate = Edate.from_datetime(createdon.dt) else: orderdate = Edate.from_datetime(datestamp.dt) # vtext: see above--ok as is, but undo odd '\X' escapes in some existing data labeltext = labeltext.replace('\\"', '"') extratext = extratext.replace('\\"', '"') # '\"' -> '"' # [1.4] undo =5C's from file, restoring '\'; see unescape function ahead labeltext = backslash_unescape(labeltext) extratext = backslash_unescape(extratext) # '=5C' -> '\' # index this file's event data for date by start m/d/y icsdata = EventData(uid=uniqueid, calendar=icsfilebasename, summary=labeltext, description=extratext, category=category, orderby=orderdate, veventobj=event) if eventdate not in EventsTable: # .keys() optional, eitbti? EventsTable[eventdate] = {} # first event for this date EventsTable[eventdate][uniqueid] = icsdata # add to global events union table def backup_and_save_all_changed_ics_files(self, folderPathOrUri): """ ------------------------------------------------------------------ Backup and save the ICS files of all calendars that have been changed since loaded or last saved. This is a two-step combo that is run in a thread as one unit to avoid blocking the GUI. Never called if no calendars have changes: caller ensures. folderPathOrUri is a py str on PCs or a java Uri on Android. Called by run_with_calendars_folder() only if the app already has calendar-folder access or it's given in an popup precursor. Runs platform-specific code in subclasses for actual file IO. Subtle: if the user was asked for a new folder before this ran due to permission/folder changes, this does _not_ load calendars from the new folder, because doing so could lose any unsaved changes in the in-memory calendars. This will simply save changed calendars to the new folder. For more info on this, see SAVES EXTRA in ask_calendars_folder_info(). ------------------------------------------------------------------ """ # backup+save files in new thread to avoid blocking gui def backup_and_save_ics_files_thread(): # two-step combo backupfails = self.backup_all_changed_ics_files(folderPathOrUri) if not backupfails: savefails = self.save_all_changed_ics_files(folderPathOrUri) # on thread exit, update gui in gui thread def on_backup_and_save_thread_exit(dt): # blocker is in enclosing scope, #self.app.dismiss_popup(blocker) # despite the different threads self.app.finalizeBusyPopup(blocker) # else too fast?, blocker in scope if backupfails: self.app.info_message( '[b]Backups failed - all saves skipped[/b]' '\n\n' 'Errors occurred while backing up these ICS files:\n\n' f' {backupfails}\n\n' 'Due to the errors, no saves were performed. Other files ' 'were backed up, and the prior versions of all ICS files ' 'were retained.') elif savefails: self.app.info_message( '[b]Saves failed - some files not saved[/b]' '\n\n' 'Errors occurred while saving these ICS files:\n\n' f' {savefails}\n\n' 'Due to the errors, these files\' saves were aborted, ' 'but all other files\' saves were successful and all ' 'ICS files were backed up.') Clock.schedule_once(on_backup_and_save_thread_exit) # same as @mainthread # post modal popup, start function in thread blocker = self.app.postBusyPopup(kindtext='saving') threading.Thread( target=backup_and_save_ics_files_thread, # don't block gui thread args=(), # uses locals in scopes daemon=True).start() # DON'T close thread if app closed: writes # and continue in gui thread with popup def backup_all_changed_ics_files(self, folderPathOrUri): r""" ------------------------------------------------------------------ Backup ICS file(s) that have changes on main-menu Save, as a precursor to file saves. folderPathOrUri is a py str on PCs or a java Uri on Android. A true result denotes errors as a list of failed filenames that cancels saves. Unlike loads, this runs a platform-specific method in subclasses along the way to minimize some redundancy. It's run within its caller's thread, so it cannot block the GUI. ------------------------------------------------------------------ """ global CalendarsTable # used here fails = [] icsfoldername = self.app.settings.calendarsfolder # str: POSIX path or SAF uri assert icsfoldername == \ (folderPathOrUri.toString() if RunningOnAndroid else folderPathOrUri) trace('Backing up to', icsfoldername) # get|make bkp subdir: (str or Uri) or None backupsSubfolder = self.backups_get_subfolder(folderPathOrUri) if not backupsSubfolder: fails.append('') # too rare to do more else: for icsfilename in CalendarsTable: if not CalendarsDirty[icsfilename]: # no changes since startup or save: don't backup/write this file continue try: # trim backups folder if needed # keeps most recent N-1 of each, based on sort order of bkp names # glob returns pathname strs or DocumentFile objs backuppatt = 'date*-time*--' + icsfilename currbackups = self.backups_glob_dir(backupsSubfolder, backuppatt) currbackups.sort(reverse=True, key=lambda x: x if not RunningOnAndroid else x.getName()) prunes = currbackups[(self.maxbackups - 1):] # earliest last for prunee in prunes: # globs have paths trace('pruning', prunee) try: # file deletes can fail: self.backups_remove_file(prunee) # but don't cancel bkp/save except: trace(f'Error while pruning: skipped {prunee}') # too rare to do more traceback.print_exc() # transfer current file to backup folder/name # note: move is quick and may be less error prone, # but copy retains original file in case save fails datetimestamp = time.strftime('date%y%m%d-time%H%M%S') # add date/time prefix backupname = f'{datetimestamp}--{icsfilename}' self.backups_backup_file( folderPathOrUri, icsfilename, backupsSubfolder, backupname) except: # on any and first failure for this file fails.append(icsfilename) trace(f'Error while backing up: skipped {icsfilename}') traceback.print_exc() # tbd: break to stop now since saves will be cancelled? return fails # [failed filenames] or []=False:OK def save_all_changed_ics_files(self, folderPathOrUri): r""" ------------------------------------------------------------------ Store events in ICS file(s) on main-menu Save, but only after successful backups. folderPathOrUri is a py str on PCs or a java Uri on Android. A true result denotes errors as a list of failed filenames. Unlike loads, this runs a platform-specific method in subclasses along the way to minimize some redundancy. It's run within its caller's thread, so it cannot block the GUI. Coding note: calendar files are written here, and read in the parse function, all at once with .read() and .write(all), not in chunks via .read(N) and .write(part). These files are small (app proprietor's largest is just 2M for 20 years), all usable devices have gigabytes of memory today, and arbitrary buffer sizes should work on all platforms under Python 3.X today. An old Windows issue precluded writing buffers > 64M on network drives but is fixed in recent libs (and 64M equates to 640 comparable years of events for the app proprietor's usage). ------------------------------------------------------------------ """ global CalendarsTable # used here fails = [] icsfoldername = self.app.settings.calendarsfolder # str: POSIX path or SAF uri assert icsfoldername == \ (folderPathOrUri.toString() if RunningOnAndroid else folderPathOrUri) trace('Generating to', icsfoldername) for icsfilename in CalendarsTable: if not CalendarsDirty[icsfilename]: # no mods to this file since startup or Save: don't backup/write continue try: icscalendar = CalendarsTable[icsfilename] if not icscalendar.walk('VEVENT'): # a list # if no events after New or deletes, add a required one now (rare but true) vevent = new_vevent( # fix name error [1.5] dtstart=None, uid=icalendar_unique_id() + '-' + icsfilename, # make unique if > one [1.5] summary=f'{APPNAME}-generated event', description='Required sole event generated.\n\n' # add tips text [1.5] 'You can delete this event at any time ' 'by tapping it within the app.\n\n' 'To completely remove this event\'s calendar, ' 'delete its ".ics" file from your calendars folder.', category='') # color? (else white) [2.0] icscalendar.add_component(vevent) # was 'system internal' icstext = icscalendar.to_ical() # bytes, major dependency self.save_one_ics_file(icsfilename, folderPathOrUri, icstext) # platform-specific subs CalendarsDirty[icsfilename] = False # no changes after save self.any_file_ops_since_empty = True except: # skip file but keep going fails.append(icsfilename) traceback.print_exc() trace(f'Error while generating ICS file: {icsfilename}') return fails # [failed filenames] or []=False:OK #============================================================================== # 💻 PC SUBCLASS -- factored-out code for PCs only: POSIX/filepaths #============================================================================== class CalendarsStorage_PCs(CalendarsStorage): """ ------------------------------------------------------------------ Subclass to factor out PC-specific calendars behavior. On all PCs, use the portable POSIX toolset and file pathnames. Don't use these on Android to avoid All Files Access permission. ------------------------------------------------------------------ """ def pprintFolder(self, pathnameStr): """ ------------------------------------------------------------------ Nothing to see here (show POSIX pathnames as they are). ------------------------------------------------------------------ """ return pathnameStr def default_calendar_folder_path(self): r""" ------------------------------------------------------------------ The user's home folder on all PCs: app always has access. Path: macOS=/Users/me, Windows=C:\Users\me, Linux=/home/me For suggested Frigcal calendars-folder default on PCs - only. ------------------------------------------------------------------ """ return os.path.expanduser('~') def run_with_calendars_folder(self, nextop, forcedask=False): r""" ------------------------------------------------------------------ PCs version: verify in filesystem or use folder chooser. forcedask is True for folder changes in Settings. nextop callable is run with pick after folder selection. [See the Android subclass's version for common overview] ABOUT WINDOWS/WSL2 PATH CONVERSIONS FOR OPENS: This does not attempt to manually convert the priorPath from Windows to WSL (Linux in Windows) format on WSL, nor from WSL to Windows format on Windows. It's possible to do so with a "wslpath" command lines in both platforms (e.g., /home/me to \\wsl.localhost\Ubuntu\home\me on Windows and C:\Users\me to /mnt/c/users/me on WSL2. See main.py's launch_file_explorer() for code that does this because it's required for file explorers. For file opens here, this conversion happens automatically in Windows but not in WSL2, for reasons with no known documentation. Applying manual conversions here are more dubious. Detecting platform-specific paths is iffy, and this seems a use case with no users: it's possible only if Windows and WSL2 share a common settings file, and this can only occur when running a common source-code package on both. It's unclear that a full source-code package will be released, and even if it is, users will normally instead use separate Windows and Linux executable packages that have their own settings files. ------------------------------------------------------------------ """ haveaccess = False priorPath = self.app.settings.calendarsfolder # None or prior ask trace(f'{priorPath=}') if priorPath != None: try: listed = try_listdir_nohang(priorPath) if not listed: # hung (inaccessible), or immediate fail (unmounted?) haveaccess = False else: haveaccess = True except: pass if haveaccess and not forcedask: nextop(priorPath) # pass py str else: self.ask_calendars_folder_info( # step 1: info popup nextop=lambda: self.ask_calendars_folder(nextop), # modal asks forcedask=forcedask, # if Settings chdir rawnextop=nextop) # for save caution def ask_calendars_folder(self, nextop): """ ------------------------------------------------------------------ Folder ask step 2: on all PCs, ask user to pick a calendars folder in the Kivy folder chooser. This varies completely from the Android version, which must use the Android-only SAF chooser for permissions. PPUS's chooser has far too much functionality to reuse here (e.g., removable-drives), and Kivy's base chooser is really just a chooser-construction kit. Instead, use the simple KivyMD chooser. Downsides: non-native and limited (e.g., no new-folder op, though the app makes one in ~). The chooser is inline code here, not .kv code (a past lesson). Alternative: bundle tkinter and use its folder choosers from Kivy? Per a web search ("bundle tkinter with a kivy app [on android]"), this seems feasible on PCs but impossible on Android. We don't care here for PCs, but there is a showstopper: tkinter's chooser blocks the Kivy GUI: it won't redraw, resize, minimize... It might work in a thread, but PyInstaller packaging is tricky. tkinter's chooser also has known issues on some macOS (see PyEdit). Alternative: Kivy's plyer also has a file chooser that tries to be platform native, but it's unusable on Android (the SAF chooser is required to get and persist permissions), and its folder chooser is lousy on Windows (a shell hack that posts a blurry 1990s-ish box that's so bad that plyer itself recommends alternatives). The weak Windows support is a full showstopper for PCs here. ------------------------------------------------------------------ """ # open in ~/Frigcal suggested default created by App.on_start() trace('ask_calendars_folder') def on_exit_folderchooser(*args): trace('fm exit') folderchooser.close() self.app.folderchooser_open = False def on_select_folderchooser(path): trace('fm select:', path) on_exit_folderchooser() self.on_selection(path, nextop) dohiddens = self.app.settings.chooserShowHiddenFiles # unchangeable (tbd) startpath = self.app.defcalspath # chooser silently refuses to open if the default path has been renamed # we have limited access on Android, but this is run on PCs only (else SAF) if not os.path.isdir(startpath): startpath = self.default_calendar_folder_path() # ~ sans subfolder folderchooser = MDFileManager( search='dirs', # folders only show_hidden_files=dohiddens, # tbd: Settings? exit_manager= on_exit_folderchooser, select_path=on_select_folderchooser ) folderchooser.show(startpath) self.app.folderchooser = folderchooser # for routing Back taps self.app.folderchooser_open = True # back=>up in folderchooser def on_selection(self, pickedPath, nextop): """ ------------------------------------------------------------------ Folder ask step 3: after user has interacted with Kivy calendar-folder chooser on PCs, run the nextop with the selected URI, if any. ------------------------------------------------------------------ """ trace('on_selection', pickedPath) if pickedPath: # update folder setting (only) for pick self.app.settings.update_and_save_settings(calendarsfolder=pickedPath) # update Settings screen for pick self.app.settingsgui.screen.ids.calendarsfolder.text = pickedPath nextop(pickedPath) # pass selected path's py pathname str # TBD: reload calendar files and redraw gui, unless nextop is load_all_calendars? # NO: see ask_calendars_folder_info() - reload would delete changes def load_all_ics_files(self, folderPath): """ ------------------------------------------------------------------ thread(load calendars from folderPath), draw gui Uses chosen POSIX pathname str, app has folder permission+access. Called by run_with_calendars_folder() only if the app already has calendar-folder access or it's given in an popup precursor. Runs files-load loop in a thread to avoid blocking the GUI. Runs superclass's generic/common parsing code in the thread. See no_calendar_files_message() for TBD on policy used here. ------------------------------------------------------------------ """ global CalendarsTable, CalendarsDirty, EventsTable # all reset in-place here # TBD: add default calendar file if none in folder? # NO: expect user to run New, plus Save for file/changes trace('loading from', folderPath) if not folderPath: return listed = try_any_nohang(os.listdir, folderPath, onfail=None) if not listed: # hung (inaccessible), or immediate fail (unmounted?) calslist = [] else: calslist = [filename for filename in listed if filename.endswith(ICS_FILE_EXTENSION)] # i.e., glob.gob trace(f'{calslist=}') # list of .ics filenames or [] if not calslist: # alert user but load anyhow: clear prior folder's events, if any self.no_calendar_files_message(folderPath) # clear global tables for Settings-screen folder change, in-place self.clear_calendar_and_event_tables() # load files in new thread to avoid blocking gui def load_all_ics_files_thread(): fails = [] for calname in calslist: calpath = osjoin(folderPath, calname) try: fileobj = open(calpath, 'r', encoding='utf8') # open can fail except: fails.append(calname) trace(f'{calname=} skipped:\n{sys.exc_info()=}') else: try: self.parse_one_ics_file(fileobj, calname) # *common ops super trace(f'{calname=} parsed') # read/parse can fail except: fails.append(calname) trace(f'{calname=} skipped:\n{sys.exc_info()=}') finally: fileobj.close() # always close object # (load thread) on thread exit, update gui in gui thread def on_load_thread_exit(dt): #self.app.dismiss_popup(blocker) # may be too fast: button self.app.refill_month_widgets() # update gui self.app.finalizeBusyPopup(blocker) # blocker in scope if fails: self.app.info_message( '[b]Some files not loaded[/b]' '\n\n' 'The following ICS files had loading errors ' 'and were skipped:\n\n' f' {fails}\n\n' 'These files\'s events will not be shown.') # (load thread) schedule gui update Clock.schedule_once(on_load_thread_exit) # == @mainthread # show modal popup, start function in thread blocker = self.app.postBusyPopup(kindtext='loading') threading.Thread( target=load_all_ics_files_thread, # don't block gui thread args=(), # uses locals in scopes daemon=False).start() # close thread if app closed: reads # and continue in gui thread with popup def save_one_ics_file(self, icsfilename, icsfolder, icstext): """ ------------------------------------------------------------------ Save one (changed) calendar to its ICS file, by POSIX + pathname. Run as part of a thread by super's common ICS-file maker loop. - icsfilename is the 'xxx.ics' name of the calendar being saved. - icsfolder is the POSIX folder filepath str chosen by the user. - icstext is the bytes object created for ICS text, made by the icalendar lib for the in-memory calendar. ------------------------------------------------------------------ """ icsfilepath = osjoin(icsfolder, icsfilename) trace('writing', icsfilepath, '...') if not osexists(icsfilepath): trace(f'creating new ics file: {icsfilename}') try: # write or overwrite fileobj = open(icsfilepath, 'wb') # text is bytes, not str except: trace(f'open fail: {sys.exc_info()=}') else: try: fileobj.write(icstext) # all at once: small files except: trace(f'write fail: {sys.exc_info()=}') finally: fileobj.close() # flush, exc or not (!with: eibti) def backups_get_subfolder(self, icsfolderPath): """ ------------------------------------------------------------------ This and its 3 followers are called by super's common backups loop and implement the POSIX/filepath versions of the required ops. All are run from the caller's thread, so they cannot block GUI. This method returns pathname str or falsy None. ------------------------------------------------------------------ """ backupdir = osjoin(icsfolderPath, BACKUPS_SUBDIR_NAME) try: if not os.path.exists(backupdir): os.mkdir(backupdir) assert os.path.exists(backupdir) and os.path.isdir(backupdir) return backupdir except: trace('backupDir make fail or already a non-dir') return None def backups_glob_dir(self, backupsSubfolderPath, backuppatt): return glob.glob(osjoin(backupsSubfolderPath, backuppatt)) # result has paths def backups_remove_file(self, pruneePath): os.remove(pruneePath) def backups_backup_file(self, folderPath, icsfilename, backupsSubfolder, backupname): # any exception caught by caller frompath = os.path.join(folderPath, icsfilename) topath = os.path.join(backupsSubfolder, backupname) # skip: new in-memory calendar that doesn't yet have a file if not osexists(frompath): trace(f'no file to backup: {icsfilename}') return else: trace(f'saving "{frompath}" to "{topath}"') shutil.copy2(frompath, topath) # copy content+info (write) #============================================================================== # 📱 ANDROID SUBCLASS -- factored-out code for Android only: SAF/URIs #============================================================================== class CalendarsStorage_Android(CalendarsStorage): """ ------------------------------------------------------------------ Subclass to factor out Android-specific calendars behavior. On Android, use the SAF toolset and content URIs for permission. We can't use POSIX/pathnames on Android without All Files Access, and the SAF chooser returns content URIs that provide permission but are not filepaths and tightly bind file ops to the SAF API. ------------------------------------------------------------------ """ # till loaded from config or asked calendarsFolderURI = None def pprintFolder(self, escapedUriStr): """ ------------------------------------------------------------------ Content URIs received from the SAF chooser may have a few embedded %3A escapes (for ':') and %2F (for '/') that app users shouldn't have to grok (content URIs are iffy as is). ------------------------------------------------------------------ """ # no: java, via pyjnius #URLDecoder = autoclass('java.net.URLDecoder') #return URLDecoder.decode(encodedUriStr, 'UTF-8') # yes: python, direct from urllib.parse import unquote_plus return unquote_plus(escapedUriStr, encoding='utf-8', errors='ignore') def default_calendar_folder_path(self): """ ------------------------------------------------------------------ The internal-storage root on Android: requires user permission. Path: /storage/emulated/0 or its /sdcard synonym on some devices. This is general shared storage, not app-specific nor app-private. For creating a suggested-default calendars folder on first run. UNUSED: shared storage cannot be used for a suggested calendars- folder default on Android because it requires a user permission granted by either All Files Access or single-folder choice: - All Files Access would allow POSIX APIs but is unused by this app because it would likely preclude Play store distribution. - Single-folder choice, used for the calendar folder in general, would require opening the SAF folder chooser just to access a known calendar default folder, and the chooser returns a content URI which essentially bind ops to the SAF API. The SAF chooser also comes with a poorly thought-out catch-22 for the initial folder path: a content URI is required to point users to a suggested folder in the chooser, but apps cannot get a suggested folder's URI without using the chooser. IOW, apps must use the SAF chooser to get a URI to pass to the SAF chooser! More broadly, there is no good path to use for either calendars or settings out of the box. Besides shared storage: - A Documents/ URI might be forged, but it's iffy, and still requires user permission as for all of shared storage. - An app-specific (aka "external") folder located at: /storage/emulated/0/Android/data/com.quixotely.frigcal/files is no longer accesisble to most file-explorer apps as of 2025. App-specific is still available in Cx (oddly), but it generally requires the Files app (which runs AOSP's explorer) or root emulation with the shizuku app (which may not be available). - An app-private (aka "internal") folder in /data is never accessible to other apps, including explorers and syncs. Both app-specific and app-private can be served to other apps by coding a file provider, but explorer-app support is limited, and both app-specific and app-private are deleted on app uninstalls. Hence, PUNT: there is no good option for this default in Android scoped storage, so users must make their own folder. See also App.storage_path_app_private()'s similar dilemma for settings; Android's lockdowns convolute development and limit users - badly. ------------------------------------------------------------------ """ return android.storage.primary_external_storage_path() # arbitrary number that ids a specific Android intent (how 1980s...) REQUEST_ID_OPEN_DOCUMENT_TREE = 987654 def run_with_calendars_folder(self, nextop, forcedask=False): """ ------------------------------------------------------------------ Android version: verify by SAF or use SAF chooser Overview: Called on all calendar loads and saves to verify or ask for the calendars folder. Run by loads for app startup, saves for main-menu's Save Calendars (including after change alerts), and loads for calendar-folder changes in the Settings screen. forcedask is True only for the latter, and nextop is a callable that's run with the pick after a folder is selected. Note that users can remove either a folder or the app's permission to a folder at any time, so the app must check before each action, even if already asked and persisted. On Android, there is a fundamental schism between POSIX filepaths and SAF content URIs - they use very different APIs, and converting between the two is iffy ('file://' URIs are no longer supported). Fetch POSIX file descriptors from the SAF API where possible, and assume that the SAF API calls used cannot hang (see hangless.py). ------------------------------------------------------------------ """ haveaccess = False Uri = autoclass('android.net.Uri') DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile') priorUriString = self.app.settings.calendarsfolder # None or prior ask trace(f'{priorUriString=}') if priorUriString != None: priorUri = Uri.parse(priorUriString) # py str => java Uri # check uri in persisted permissions activity = self.app.get_android_activity() context = self.app.get_android_context() persistedUriPermissions = \ activity.getContentResolver().getPersistedUriPermissions() hasUriPermission = any( priorUriString == uritry.getUri().toString() for uritry in persistedUriPermissions) trace('perms:') trace([uritry.toString() for uritry in persistedUriPermissions]) trace(f'{hasUriPermission=}') if hasUriPermission: # try listing: it may not be accessible now try: documentsTree = DocumentFile.fromTreeUri(context, priorUri) if documentsTree: for documentFile in documentsTree.listFiles(): break haveaccess = True except: pass if haveaccess and not forcedask: nextop(priorUri) # pass java Uri else: self.ask_calendars_folder_info( # step 1: info popup nextop=lambda: self.ask_calendars_folder(nextop), # modal asks forcedask=forcedask, # if Settings chdir rawnextop=nextop) # for save caution def ask_calendars_folder(self, nextop): """ ------------------------------------------------------------------ Folder ask step 2: on Android, ask user to pick a calendars folder in Android's SAF UI so permissions are granted. The content URI returned requires later processing to use SAF too, unless file descriptors can be fetched/used (not for folders). On Android, the SAF picker is effectively modal (blocking) because the app is paused until the user closes the picker (App on_pause() and on_resume() run on picker open/close): ''' While it is technically a separate system activity and not a modal dialog in the traditional programming sense, it has the practical effect of being a modal experience from the user's perspective, as it blocks interaction with your app until the task is complete. ''' Hence, the app can chain actions assuming the picker will send a result to its callback before anything else happens in the app. The missing magic bits: "p4a_activity" uses pyjnius to make a Java class with a callback method named onActivityResult() in Python, which is registered to Java through the app's Android activity and simply calls the Python callable that we pass to p4a_activity.bind() here when the Intent finishes: https://github.com/kivy/python-for-android/blob/develop/ pythonforandroid/recipes/android/src/android/activity.py SUBTLE: this must also _unbind_ any prior binds, here or in the results handler, or arrange to bind just once (e.g., in __init__). Else, N prior runs and binds here means N+1 calls to the result handler instead of 1 (yes, really!). Barring user folder or permission changes, this crops up only for the Settings screen's calendar-folder change but is an arguably perilous API choice. See the above activity.py; its code seems the only docs on this... Tbd: Intents sent to and from Android have putExtra()/get*Extra() to pass data on to the result handler, but it's Java strongly typed, and it's unclear how to send a Python callback object to/from this API. PUNT: the current coding works as is. TBD: folder-ask intent also has an initial folder to open the popup at, but converting the default filepath to a URI is iffy~impossible [intent.putExtra(DocumentsContract.EXTRA_INITIAL_URI, uriToLoad)] _May_ work, possibly with file:// prefix: if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) fileUri = Uri.parse(self.app.defcalspath); else fileUri = Uri.fromFile(new File(self.app.defcalspath)); Uri.parse(new File("/sdcard/Documents/Frigcal").toString()) PUNT: this is just how the AOSP/SAF icker works. ------------------------------------------------------------------ """ self.nextop = nextop # post-user-choice action # python-for-android's activity class: for on_activity_result listener from android import activity as p4a_activity # use SAF chooser for Android permissions, not kivy widget # must use mActivty, not the casted app activity mActivity = self.app.get_android_activity(docast=False) # ask user, run action after UI result appears later Intent = autoclass('android.content.Intent') intent = Intent() intent.setAction(Intent.ACTION_OPEN_DOCUMENT_TREE) # required: unbind prior bind if any, else N callbacks for N binds! p4a_activity.unbind(on_activity_result=self.on_ask_calendars_folder_result) # bind for results callback via p4a activity p4a_activity.bind(on_activity_result=self.on_ask_calendars_folder_result) mActivity.startActivityForResult(intent, self.REQUEST_ID_OPEN_DOCUMENT_TREE) # now the SAF picker popup appears and the app waits for the user's response def on_ask_calendars_folder_result(self, requestCode, resultCode, intent): """ ------------------------------------------------------------------ Folder ask step 3: after user has interacted with SAF calendar-folder chooser on Android, run the nextop with the selected URI, if any. This is called by the Java API (by way of the Kivy Activity class) when the user dismisses the folder picker. The app is blocked until the user selects a folder or cancels the picker, so it's effectively modal. The selected URI is in the intent and is a java Uri, not a py str. This could convert URI to filepath (sort of; see web), but that would limit the app to physical storage. We may or may not need to care, but using content URIs allows for virtual storage like clouds, which may be useful but are also innately perilous from a privacy/safety perspective. ------------------------------------------------------------------ """ trace('saf picker callback') activity = self.app.get_android_activity() Activity = autoclass('android.app.Activity') Intent = autoclass('android.content.Intent') if requestCode != self.REQUEST_ID_OPEN_DOCUMENT_TREE: trace('folder picker - irrelevant intent result') else: if resultCode == Activity.RESULT_CANCELED: trace('folder picker - cancelled') elif resultCode != Activity.RESULT_OK: trace('folder picker - unknown resultCode:', resultCode) else: # OK: persist permission, save its uri, run nextop # chosen uri is in passed intent pickedUri = intent.getData() # content uri, not posix pathname if not pickedUri: # assume it's a folder & accessible trace('folder picker - no URI') return # possible? # save to settings as a str, Uri.parse(str) undoes pickedUriString = pickedUri.toString() # java Uri => py str trace('folder picker results:', pickedUriString) # update setting in app-private storage, auto backed up self.app.settings.update_and_save_settings( calendarsfolder=pickedUriString) # update Settings screen for pick: must be run in gui thread def setum(dt): guifield = self.app.settingsgui.screen.ids.calendarsfolder guifield.text = pickedUriString Clock.schedule_once(setum) # save persistable URI permission for long-term access activity.getContentResolver().takePersistableUriPermission( pickedUri, Intent.FLAG_GRANT_READ_URI_PERMISSION | Intent.FLAG_GRANT_WRITE_URI_PERMISSION ) # run dependent action: ensure run on main gui thread if needed; # same as calling a @mainthread decorated callback func now/here; # passes the picked folder as the java Uri, not the py str; Clock.schedule_once(lambda dt: self.nextop(pickedUri)) # TBD: reload calendar files and redraw gui, unless nextop is load_all_calendars? # NO: see ask_calendars_folder_inf() - reload would delete changes def load_all_ics_files(self, folderUri): """ ------------------------------------------------------------------ thread(load calendars from folderUri), draw gui. Uses chosen SAF folder URI, app has folder permission+access. folderUri is the java Uri from asker, not the py str. Called by run_with_calendars_folder() only if the app already has calendar-folder access or it's given in an popup precursor. Runs files-load loop in a thread to avoid blocking the GUI. Runs superclass's generic/common parsing code in the thread. See no_calendar_files_message() for TBD on policy used here. About DocumentFile: it's a wrapper around a URI that represents a document or directory managed by a SAF ContentProvider, which provides file and directory access sans ContentResolver queries. Requires androidx.documentfile.documentfile:1.1.0 in bd.spec. About fds: this reads clendars in Python via os file descriptor fetched from SAF, coded per Java examples + Android docs (yuck!). Do not also run afd.close() (parcel file descriptor): closefd=True by default. Can also get file uri from its name this way: Uri = autoclass('android.net.Uri') baseuri = Uri.parse(folderUri) itemUri = Uri.withAppendedPath(baseUri, calname) ------------------------------------------------------------------ """ global CalendarsTable, CalendarsDirty, EventsTable # all reset in-place here trace('loading from', folderUri and folderUri.toString() or '?') if not folderUri: return # fetch a list of the *.ics files in the calendars folder activity = self.app.get_android_activity() context = self.app.get_android_context() DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile') # get DocumentFile objects for the selected directory calslist = [] documentsTree = DocumentFile.fromTreeUri(context, folderUri) if documentsTree: for documentFile in documentsTree.listFiles(): if documentFile.isFile(): documentName = documentFile.getName() if documentName and documentName.lower().endswith(ICS_FILE_EXTENSION): calslist.append((documentName, documentFile.getUri())) trace(f'{calslist=}') # list of .ics filenames or [] if not calslist: # alert user but load anyhow: clear prior folder's events, if any self.no_calendar_files_message(self.pprintFolder(folderUri.toString())) # clear global tables for Settings-screen folder change, in-place self.clear_calendar_and_event_tables() # read calendars via os file descriptor fetched from SAF contentResolver = activity.getApplicationContext().getContentResolver() # load files in new thread to avoid blocking gui def load_all_ics_files_thread(): fails = [] for (calname, caluri) in calslist: afd = contentResolver.openFileDescriptor(caluri, 'r') if not afd: fails.append(calname) trace(f'{calname=} skipped: no fd') else: try: fd = afd.detachFd() # back in py (ahh) fileobj = os.fdopen(fd, 'r', encoding='utf8') # open can fail except: fails.append(calname) trace(f'{calname=} skipped: open {sys.exc_info()=}') else: try: self.parse_one_ics_file(fileobj, calname) # *common ops super trace(f'{calname=} parsed') # read/parse can fail except: fails.append(calname) trace(f'{calname=} skipped: parse {sys.exc_info()=}') finally: fileobj.close() # always close object # (load thread) on thread exit, update gui in gui thread def on_load_thread_exit(dt): #self.app.dismiss_popup(blocker) # may be too fast: button self.app.refill_month_widgets() # update gui self.app.finalizeBusyPopup(blocker) # blocker in scope if fails: self.app.info_message( '[b]Some files not loaded[/b]' '\n\n' 'The following ICS files had loading errors ' 'and were skipped:\n\n' f' {fails}\n\n' 'These files\'s events will not be shown.') # (load thread) schedule gui update Clock.schedule_once(on_load_thread_exit) # == @mainthread # show modal popup, start function in thread blocker = self.app.postBusyPopup(kindtext='loading') threading.Thread( target=load_all_ics_files_thread, # don't block gui thread args=(), # uses locals in scopes daemon=False).start() # DO close thread if app closed: just reads # and continue in gui thread with popop def save_one_ics_file(self, icsfilename, icsfolder, icstext): """ ------------------------------------------------------------------ Save one (changed) calendar to its ICS file, by Android SAF. Run as part of a thread by super's common ICS-file maker loop. - icsfilename is the 'xxx.ics' str name of the calendar being saved. - icsfolder is the SAF folder content URI chosen by the user: a java Uri here, not a py str. - icstext is the bytes object created for ICS text, made by the icalendar lib for the in-memory calendar. TBD: documentDir.createFile is supposed to be called sans extension, on the assumption that the provider adds one per MIME type. This seems gray in practice, but try type/calendar for a .ics instead of text/plain which may add .txt (alt: Application/octet-stream?). UPDATE: this works as coded. ABOUT 'stupid_saf_mode': Android 10 bizarrely changed the open mode for contentResolver.open* such that the standard 'w' mode no longer truncates the file to zero bytes. The net effect can leave garbage bytes at the end of the file if its new content is shorter in the new write. This is wildly proprietary and was woefully undocumented (and comes off as naive and arrogant). THE FIX: Mode 'wt' can force truncation but apparently failed for Google Drive's SAF provider. Mode 'rwt' used here, and recommeded as a workaround by Google/Android, truncates too and works in GD. The garbage bytes seemed sporadic so it's not certain that this fixes the problem, but all signs point to yes. FOR MORE on this Android 10 breakage (and a weak Google reply): https://issuetracker.google.com/issues/180526528 or search: "android saf open for writing does not truncate file" Python's file.truncate(0) also truncates, and writing the bytes in smaller chunks may help (fails were seen on a 2.3M file), but a single write() works outside Android, and neither of these should be required. An earlier flush() fix attempt is now moot. ------------------------------------------------------------------ """ trace(f'writing {icsfilename} in {icsfolder.toString()} ...') activity = self.app.get_android_activity() context = self.app.get_android_context() DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile') documentDir = DocumentFile.fromTreeUri(context, icsfolder) if not documentDir: trace('No documentDir') else: documentFile = documentDir.findFile(icsfilename) if not documentFile: # may be first save for a new calendar trace(f'creating new ics file: {icsfilename}') documentFile = documentDir.createFile('text/calendar', icsfilename) if not documentFile: trace('No documentFile') else: # write as bytes via file descriptor stupid_saf_mode = 'rwt' # see above icsfileuri = documentFile.getUri() contentResolver = activity.getApplicationContext().getContentResolver() afd = contentResolver.openFileDescriptor(icsfileuri, stupid_saf_mode) if afd: try: fd = afd.detachFd() # back in py (ahh) fileobj = os.fdopen(fd, 'wb') # open+write can fail except: trace(f'{icsfilename=} skipped: {sys.exc_info()=}') else: try: fileobj.write(icstext) # actually, it's bytes except: trace(f'{icsfilename=} skipped: {sys.exc_info()=}') finally: #fileobj.flush() # might matter for fd fileobj.close() # always close object def backups_get_subfolder(self, icsfolderUri): """ ------------------------------------------------------------------ Verify or create backups subfolder. -icsfolderUri here is a java Uri: not python str or DocumentFile. This and its 3 followers are called by super's common backups loop and implement the Android SAF/URI versions of the required ops. Returns a DocumentFile wrapping a SAF content URI or falsy py None, not a Uri or py str. Could convert result to Uri with DF.getUri() but that would require extra reconversions later and Uri not used. ------------------------------------------------------------------ """ trace(f'{icsfolderUri=}') context = self.app.get_android_context() DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile') documentDir = DocumentFile.fromTreeUri(context, icsfolderUri) if not documentDir: trace('No documentDir') return None else: try: backupDir = documentDir.findFile(BACKUPS_SUBDIR_NAME) if not backupDir: trace('making backup dir') backupDir = documentDir.createDirectory(BACKUPS_SUBDIR_NAME) trace('have backup dir') assert backupDir and backupDir.exists() and backupDir.isDirectory() return backupDir except: trace('backupDir make fail or already a non-dir') return None def backups_glob_dir(self, backupsSubfolderUri, backuppatt): """ ------------------------------------------------------------------ Returns a list of DocumentFile objects, for "*.ics" in Backups. - backupsSubfolderUri is a DocumentFile, from backups_get_subfolder. It wraps a Uri but isn't one itself, despite the early name. ------------------------------------------------------------------ """ trace(f'{backupsSubfolderUri=}') context = self.app.get_android_context() DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile') # vs glob.glob: SAF Java API is gross calslist = [] ##documentsTree = DocumentFile.fromTreeUri(context, backupsSubfolderUri) documentsTree = backupsSubfolderUri if documentsTree: for documentFile in documentsTree.listFiles(): if documentFile.isFile(): documentName = documentFile.getName() if documentName and fnmatch.fnmatch(str(documentName), backuppatt): calslist.append(documentFile) # path implied, or .getUri() return calslist def backups_remove_file(self, pruneeDoc): """ ------------------------------------------------------------------ Remove a file, the SAF way. - pruneeDoc is a DocumentFile, from backups_glob_dir. Or use a list of documentFile.getUri() results and: assert context.getContentResolver().delete(pruneeUri, None, None) > 0 ------------------------------------------------------------------ """ assert pruneeDoc.delete() # a DocumentFile def backups_backup_file(self, folderUri, icsfilename, backupsSubfolderUri, backupname): """ ------------------------------------------------------------------ Read+write calendars via os file descriptors fetched from SAF. - backupsSubfolderUri is a DocumentFile, from backups_get_subfolder. - folderUri is a java Uri (not py str) and requires a fromTreeUri. Vs joins + [shutil.copy2 or .read() + .write()]: SAF is horrific! TBD: copying file metadata like timestamps seem impossible in SAF? ------------------------------------------------------------------ """ activity = self.app.get_android_activity() context = self.app.get_android_context() DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile') # read FROM any exception caught by caller documentDir = DocumentFile.fromTreeUri(context, folderUri) assert documentDir documentFileFrom = documentDir.findFile(icsfilename) # skip: new in-memory calendar that doesn't yet have a file if not documentFileFrom: trace(f'no file to backup: {icsfilename}') return else: trace(f'saving "{icsfilename}" to "{backupname}"') contentResolver = activity.getApplicationContext().getContentResolver() # read as text in py native code afd = contentResolver.openFileDescriptor(documentFileFrom.getUri(), 'r') assert afd fd = afd.detachFd() fileobj = os.fdopen(fd, 'r', encoding='utf8') fromtext = fileobj.read() fileobj.close() # create+write TO, any exception caught by caller ##documentDir = DocumentFile.fromTreeUri(context, backupsSubfolderUri) documentDir = backupsSubfolderUri assert documentDir documentFileTo = documentDir.createFile('text/calendar', backupname) assert documentFileTo # write as text in py native code ('rwt' not needed here) afd = contentResolver.openFileDescriptor(documentFileTo.getUri(), 'w') assert afd fd = afd.detachFd() fileobj = os.fdopen(fd, 'w', encoding='utf8') fileobj.write(fromtext) fileobj.flush() fileobj.close()