File: Frigcal/code/Frigcal--source/storage.py
"""
===============================================================================
CALENDAR-FOLDER STORAGE (part of Frigcal)
About the SAF...
This app's code is bifurcated for Android's permissions rules:
• On all PCs, it uses the usual, portable, and interoperable filepaths
and POSIX calls for the calendars folder. No permissions are required.
• On Android, it uses Android's proprietary SAF picker for the calendars
folder, which grants required permission but also forces the app to use
content URIs and the arguably tortuous SAF API for storage processing
(except where file descriptors can be had at the bottom of the code).
Using SAF for the calendars folder on Android roughly _doubles_ the amount
of storage code just for the sake of Android and obscures this app's
implementation substantially. Moreover, SAF's API is wildly convoluted
compared to POSIX/filepath equivalents (see ahead), and the SAF is widely
known to be very slow, which matters even more on phones.
By contrast, quixotely.com's PPUS sync app uses Android's All Files Access
(a.k.a. MANAGE_EXTERNAL_FILES) permission because it requires both broad
storage access and optimal storage speed. This permission also allows a
single version of code based on POSIX and filepaths to be used on all
platforms, thereby avoiding code bifurcation and its inherent costs.
There is some extra code for Android storage (e.g., asks) but it's minor.
In the end, All Files Access was deemed overkill here because, unlike PPUS,
this app uses only one folder in storage, and its speed requirements are
negligible (loads and saves are rare events, not core behavior). Also
unlike PPUS, using All Files Access would likely preclude distributing this
app on the Play store, which is officially a downside, though the rules
that Play regularly imposes on developers may qualify as a negative too.
The SAF itself may have been an interesting idea in theory - accessing
both physical and non-physical storage the same way - but it doesn't
seem to survive contact with reality. It's bloated, functionally limited,
and almost non-sensical at times (again, see the code ahead); there is
no compelling reason to support non-physical storage in some apps (clouds
and networks can be a Bad Thing, especially for private calendar data);
and POSIX/filepath code is both much simpler and works on PCs too.
As it stands, developers must choose between supporting Android or PCs,
or budgeting for double the effort to support both. The former is a
choice that naturally benefits monopolistic agendas and the latter is
an added cost that Google seems unlikely to reimburse. Can't we do better?
Related: main.py's storage_path_app_private() is a platform-specific path
for storage of admin files, not calendars, which also varies by PC kind.
===============================================================================
"""
from common import * # shared collector module
from hangless import try_any_nohang, try_listdir_nohang # hang-free IO calls
# ICS-file parser/generator (3rd-party lib, uses pytz, both shipped with app)
try:
import icalendar
except:
# should never happen, but...
print('Required and included icalendar or pytz packages not present')
sys.exit(1)
from icsfiletools import * # global tables, icalendar objects
from icalendar.parser import unescape_char # currently unused
from icalendar.parser import escape_char, foldline # for Unicode file text
#==============================================================================
# 🗓 ICS events data structures: internal format of calendars and events
#==============================================================================
"""
-------------------------------------------------------------------------------
Global calendar and event data structures
ICS calendar files:
- Are loaded into the following in-memory data structures on calendar loads,
which occur on app startup and calendar-folder switches
- Are converted back to iCalendar text from this form and written to ICS
files on main-menu Save, which is prompted on app exit and folder switches
The GUI is drawn from the in-memory form, and event updates are applied to
it until user Save requests. The in-memory form is three global tables:
- CalendarsTable embeds icalendar-library Calendar objects created by
ICS-file parses and used for ICS-file generations. Calendar objects
may also be created by main-menu New.
- CalendarsDirty designates user changes and is separate to minimize code
changes in earlier Frigcal vesions.
- EventsTable is an index that combines all ICS files' events and uses
two-level dictionary table nesting: [date][uid].
All three are largely built in storage.py's calendar parsing. Structure:
CalendarsTable = # Parsed file's raw calendar data (all data)
{icsfilename: # key: basename of .ics file parsed (with '.ics')
icalendar.cal # icalendar.cal Calendar object, from parser or New
}
CalendarsDirty =
{icsfilename: # key: basename of .ics file parsed
Boolean # True=calendar changed since startup or latest save
}
EventsTable = # Union of all events in all ics files (indexed data)
{(m, d, y): # key: date object for event start (and display) date
{uid: # key: event's required unique id, for fast deletes
(.calendar, # ics file basename, link to icalendar in CalendarsTable
.summary, # event summary text (aka Title in GUI)
.description, # event description text (aka Note in GUI)
.category, # category (just 1/first used)
.uid, # event's required unique id, for easy access
.orderby, # creation (or else mod) date: display order sort
.veventobj) # reference link to vevent in icalendar.cal object
}
}
More about EventsTable:
Events from all calendars are stored in a 2-level table, indexed by date and
unique id (UID) for quick access to both displayed and parsed data. Because
events in the table are stamped with their calendar name, there is no need to
segregate by a 3rd calendar dimension (an initial design) - the GUI displays
the _union_ of all calendar files' events, and a single combined table still
supports display grouping by calendar, via sorts.
Events in the table also have by-name links back to their parsed icalendar object
for fast in-place updates in memory (files are regenerated only on Save), and GUI
month windows keep UID-indexed tables of displayed events for quick deletion.
This model assumes events have UIDs, but the iCalendar standard requires these.
Subtlety: IN-PLACE CHANGE to mutable objects shared by multiple references is
crucial in this model. EventsTable-index objects are referenced from registered
GUI event handlers, and there is just one copy of each CalendarsTable parsed
icalendar array, referenced from index objects and used at file regeneration time.
-------------------------------------------------------------------------------
"""
# loaded calendar/event tables, per above (singletons)
# changed in memory, transferred to/from external ICS files on demand
# reset in-place before loads for Settings-screen folder changes
CalendarsTable = {} # {icsfilebasename: icalendar.cal.Calendar()}
CalendarsDirty = {} # {icsfilebasename: Boolean} [1.1]
EventsTable = {} # {Edate(): {uid: EventData()} ]
class Edate:
"""
-------------------------------------------------------
Encapsulate and name event date numbers.
!= ViewDate, used to manage GUI day in monthgui.py.
-------------------------------------------------------
"""
def __init__(self, month, day, year):
self.month = month # a namedtuple may do some of this, but not all
self.day = day
self.year = year
@staticmethod # copy datetime.{date, datetime} object attrs
def from_datetime(dateobject): # staticmethod optional if 3.X class calls only
return Edate(dateobject.month, dateobject.day, dateobject.year)
def to_datetime_date(self):
return datetime.date(**self.__dict__) # copy attrs to a datetime.date object
def as_tuple(self):
return (self.month, self.day, self.year)
def as_string(self):
return '%02d/%02d/%4d' % self.as_tuple() # formatted display, 'MM/DD/YYYY'
def as_nice_string(self):
# [4.0] month-labeled and locale-neutral 'mmm D, YYYY'
# don't zero-pad day with %d: users aren't computers
return self.to_datetime_date().strftime(f'%b {self.day}, %Y') # 'Nov 8, 2025'
# support use in dictionary key, comparisons, sort key
def __hash__(self):
return hash(self.as_tuple()) # dictionary key (else by addr)
def __eq__(self, other):
return self.as_tuple() == other.as_tuple() # comparisons (else by addr)
def __lt__(self, other): # sort keys (else fails)
"""
[1.4] order events on dates by year first (y-m-d), not month;
but keep as_tuple() format (m-d-y): it's also used for displays;
was originally: return self.as_tuple() < other.as_tuple()
"""
return ((self.year, self.month, self.day) <
(other.year, other.month, other.day))
class EventData:
"""
-------------------------------------------------------
Maximal attributes set for an event.
Used for both EventsTable index and GUI widget values.
-------------------------------------------------------
"""
def __init__(self,
uid='',
calendar='', summary='', description='', category='',
orderby='', veventobj=None):
self.uid = uid # event unique id, required by iCalendar std
self.calendar = calendar # calendar file's basename (not path)
self.summary = summary # for event summary in month, elsewhere
self.description = description # for extra text in footer, edit dialog
self.category = category # for colorizing event summary entries
self.orderby = orderby # for ordering events entries initially
self.veventobj = veventobj # a reference to icalender object: updates, deletes
# or: for (name, value) in kargs.update(defaults): setattr(self, name, value)
def copy(self):
return EventData(**self.__dict___) # use attrs dict for keyword args
def __eq__(self, other):
return self.__dict__ == other.__dict__ # comparisons (else by addr)
#==============================================================================
# 💾 BASE CLASS -- common code for both PCs and Android
#==============================================================================
class CalendarsStorage:
"""
------------------------------------------------------------------
Implement file IO ops per the hosting platform's permission rules.
One instance of one sublcass, created by and stored in App instance.
Base class with common/shared methods for Android and PCs subclasses.
Example usage in App:
self.storage = \
CalendarsStorage_Android(self) if RunningOnAndroid else
CalendarsStorage_PCs(self)
self.storage.run_with_calendars_folder(
self.storage.load_all_calendar_files, forcedask=?)
self.storage.run_with_calendars_folder(
self.storage.backup_andsave_all_changed_calendar_files)
------------------------------------------------------------------
"""
def __init__(self, app, maxbackups):
self.app = app # link to state+ops in main.py's App()
self.maxbackups = maxbackups # from App.settings, changed by Settings
self.folder_details_shown = False # show dialog extra info once pre run
self.any_file_ops_since_empty = False # warn about saves sans folder access
def clear_calendar_and_event_tables(self):
"""
------------------------------------------------------------------
Clear global tables of loaded calendars and events to be empty
prior to loads, for Settings-screen folder changes. This must
be an in-place change. It cannot simply assign {}s to these names,
because they are imported with 'from' by other modules, which
share objects and would not notice name changes here (see LP6E!).
------------------------------------------------------------------
"""
for table in (CalendarsTable, CalendarsDirty, EventsTable):
table.clear() # same object in this module and others
def ask_calendars_folder_info(self, nextop, forcedask, rawnextop):
"""
------------------------------------------------------------------
Folder ask step 1: called on all platforms before file ops when
there is no access to a calendars folder. Popup info about the
ask, then run a platform-specific asker, which later runs the
nextop platform-specific load or save (three levels). Context:
- forcedask is True only for Settings screen's calendar-folder mod.
- rawnextop is the true next step after ask, for save+chdir caution.
This step is generic enough to be the same for Android and PCs
and multiple ops, but the message must be tailored for Android:
see default_calendar_folder_path().
FORMATTING:
Caveat: on wsl2, [size=20sp] seems not to scale to font size as it does
in Help/About, likely due to default font size diffs and a wsl2 glitch;
it _should_ scale per Kivy's 2.3.1 code, which calls dpi2px() for 'Nsp'
to scale (despite the fact the Google's AI said it is not supported...).
Update: [size=20sp] was dropped because it also doesn't change if the
user changes globalfontsize in the Settings screen (there's no way to
bind it to a property). Instead, headings now use color, here and in
Help and About text. This is subtle: color must work in both Dark and
Light mode, alt fonts (e.g, DejaVuSans) makes text look weak, and
italic and underscores are already used much in non-header text.
This oddball need not match Help/About style per se, but it does.
DETAILS EXTRA:
The details part of this message might be limited to first-run asks,
when self.app.runnum == 1, because it's mostly just for new users
setting the folder initially; after that, there is similar info in
Help's calendars coverage.
Update: as a compromise, show details just once per run, instead of
either always or on first run only. Probably a useful tip on future
Settings folder mods, but they may be annoying if > once in a run.
Nit: the spacing for details is less than in first-run terms of use
and Help/About, but it should match the caution up next and we don't
want the caution to be missed by users. Trivial but significant.
SAVES EXTRA:
rawnextop is used to detect calls from run_with* for a main-menu
Save, when there are calendar changes and the prior calendar folder
cannot be accessed due to a delete, rename, or app-permission removal
after a load while app is running. This is a dicey UX boundary case.
There are three contexts that invoke run_with_calendars_folder() and
its asks:
- Initial calendar loads, where nextop is load_all_ics_files()
- On-demand calendar-folder changes, where nextop is load_all_ics_files()
- On-demand calendar saves, where nextop is backup_and_save_all_changed_ics_files()
The first two already load all calendars next by design. Saves do not
reload calendars, because doing so would wipe out the very changes that
the user is trying to save, but not reloading means that the GUI won't
reflect the folder after Save. Reloading post saves would load only
calendars that were saved in the new folder, and that's not right either.
The best Save can do in this case is to save changes to a newly picked
folder, and leave it to the user to sort out the aftermath. This is also
so unlikely as to be either impossible or attributable to user error.
But to minimize risk, run_with* asks now rely on this method to detect
this rare use case and augment the pre-select info_message so the user
is aware of the consequences. The save proceeds in the new calendar
folder if the user picks one, however, to save the calendar changes.
See main.py's on_menu_save for parallel notes and context.
Subtlety: we're interested only in Save sans folder access when the
in-memory calendars reflect a real folder, such that saves will create
skew. To detect this, we must check rawnextop for the save method but
also whether there were prior calendar loads since either startup or
the last folder change. In either case, it's okay to not yet have
folder access on Save if calendars have been added but not loaded or
saved to files (e.g., after adding new calendars on first run or
switching to a new empty folder in Settings). This cannot check
CalendarsTable, because new calendars are added there pre load/save.
Instead, any_file_ops_since_empty flags file ops since the last
empty-folder state, so we can know if in-memory calendars reflect a
real folder's content and thus may yield skew on Save to a new folder.
Update: this moved from separate popup to augmentation here, to
avoid popups overload (three is too much).
Postscript: the save-sans-folder state might have been avoided
if Save saved _all_ calendars, instead of just changed calendars.
The new folder would then always be complete. But partial saves
are by design because saving all calendars on every save might
be noticeably slow, especially on slower phones. In the end,
speed is more important than avoiding a wildly unlikely UI state.
------------------------------------------------------------------
"""
first_run_extra = save_caution_extra = ''
# provide extra details once per run only, info in Help later
if not self.folder_details_shown:
self.folder_details_shown = True
first_run_extra = (
'\n\n\n'
'[H2]Details[/H2]'
'\n\n'
'This app creates and uses calendars formatted per the '
'iCalendar standard and stores them in a folder as portable ICS '
'files with extension .ics. Calendar [i]loads[/i] combine all the '
'.ics files in this folder, and calendar [i]saves[/i] update all '
'changed .ics files there after creating backup copies.'
'\n\n'
'To share your calendars with other programs and devices, simply '
'sync or manually copy .ics files to or from your chosen calendars '
'folder. Some other calendar apps may require imports or exports '
'for ICS files, but this app uses them directly on all platforms.'
)
# alert user about picking a new folder for saves, not a separate popup
if (self.any_file_ops_since_empty and
rawnextop == self.app.storage.backup_and_save_all_changed_ics_files):
save_caution_extra = (
'\n\n\n'
'[u][H2]Caution[/H2][/u]: save sans prior-folder access[/color][/b]'
'\n\n'
'The external calendars folder to which your changed calendars '
'correspond is no longer accessible.'
'\n\n'
'[i]Proceeding[/i] with this save by choosing a new folder in the '
'following chooser will result in only your changed calendars being '
'saved to the new folder, and the app\'s GUI will not be reloaded '
'to reflect the new folder\'s contents. You may have to reconcile '
'prior/new folder differences manually after the save.'
'\n\n'
'[i]Cancelling[/i] this save in the following chooser will avoid folder '
'and GUI skew but will not save your calendar changes. If you removed '
'app permissions or deleted or renamed your folder since this app was '
'started, you may wish to undo your mods now and retry the save.'
)
reason = (
'You have asked to change the calendars folder'
if forcedask else
'This app does not have access to a calendars-storage folder'
)
suggested = (
'' if RunningOnAndroid else
'For convenience, a Frigcal subfolder in your home '
'folder is available as a suggested default.'
)
self.app.info_message(
'[b]Calendar folder selection[/b]'
'\n\n'
f'{reason}. '
'In the next screen, please select the folder to be used for '
'storing your calendar files.'
'\n\n'
'You can select any folder that you can access on your device, '
'including one that is part of content you normally sync with '
f'other devices. {suggested}'
f'{first_run_extra}'
f'{save_caution_extra}'
, usetoast=False, nextop=nextop
)
# and the folder chooser comes up next, followed by nextop
def no_calendar_files_message(self, folderPathOrUri):
"""
------------------------------------------------------------------
Shared by POSIX/filepath and SAF/content-uri calendar loaders.
folderPathOrUri is a py str: POSIX pathname or SAF content URI.
Called only before load if have calendars-folder access by a
prior op or a newly provided chooser reply, and folder is empty.
This can happen on startup and on later Settings folder changes.
Clears any_file_ops_since_empty because this is a known state in
which internal calendars do not correspond to an external folder,
and a Save cannot write partial folder contents. All later loads
and saves set this flag because internal reflects external.
TBD: automatically add a default calendar file if none in folder,
or expect the user to add one with New? Either policy may be seen
as onerous by some users: manually deleting an unwanted calendar
versus manually adding a new one.
CHOICE: assume user must run New run next, and show this info_message
with tips. Further assume that user will run a Save to write the
calendar's ICS file if and when desired. No default cal required.
UPDATE: but must not skip calendars load if no calendars in folder,
because this may occur after a change-folder in Settings, and the
GUI should reflect the new empty folder. Else, a Save would save
the prior folder's calendars in the new and unrelated folder.
This is acheived by coding in the load methods, not here.
------------------------------------------------------------------
"""
self.any_file_ops_since_empty = False
self.app.info_message(
'[b]No calendars in folder[/b]'
'\n\n'
f'....{folderPathOrUri}'
'\n\n'
'If you are using this folder for the first time, either:'
'\n\n'
'1) Add a [i]first calendar[/i] to this folder now using the '
'main menu\'s New Calendar, and use its Save Calendars to '
'create the new calendar\'s file when desired.'
'\n\n'
'2) Copy [i]existing calendar files[/i] to this folder, and '
'load them by either restarting this app or changing to the '
'same folder again in the Settings screen.'
'\n\n'
'To use a different folder altogether, see Settings.')
# and end UI interaction for calendar loads (after reload!)
def parse_one_ics_file(self, pyFileObject, icsfilebasename):
r"""
------------------------------------------------------------------
ICS events file parser: creates/returns in-memory data from a .ics file.
Load events at startup from ICS file whose open file object is passed to
pyFileObject. icsfilebasename is the ICS file's basename (a py str),
regardless if it's from a pathname or URI, and this is run within a
non-GUI thread by its caller so it cannot block the GUI.
Run by a platform-specific version of loading code, which uses POSIX/paths
on all PCs and SAF on Android. Backups and saves instead have a platform-
neutral driver that calls out to platform-specifc parts; which is better?
The caller finds, opens, and closes pyFileObject in platform-specific code.
Don't reinitialize tables here: they are already imported by object via 'from'!
Any errors here are caught and displayed by the caller: the data may be bad.
VTEXT is a str subclass, and works as is: no extra str() conversion is needed.
icalendar uses the UTF8 iCalendar default Unicode encoding scheme throughout.
from_ical parse auto-unescapes any "\X" icalendar sequences and does Unicode decoding.
to_ical generate auto adds "\X" escapes if needed and does Unicode encoding.
See also the note about reading all at once here, in generate_ics_files() ahead.
Unicode NB: this formerly used icalendar.parser.DEFAULT_ENCODING for calendar
files' Unicode encodings, with a note about not hardcoding UTF-8. In FC 4.0,
both callers hardcode UTF-8 because it's used by iCalendar per spec. This
may be an issue for non-conforming calendar tools that use other encodings,
but there is no source code to be changed in frozen apps/executables, and
making this a user setting may not help (encoding might vary per ICS file).
Bytes works in icalendar API too, but it's not clear how this would help.
------------------------------------------------------------------
"""
# TBD: make calendar/event tables attrs of this self, not globals?
global CalendarsTable, CalendarsDirty, EventsTable # all filled here
trace('Parsing', icsfilebasename)
# read ics text, caller handles open, close, exceptions
icstext = pyFileObject.read() # all at once: small files
# parse ics text: 3rd party package(s), huge dependency
icsarray = icalendar.cal.Calendar.from_ical(icstext) # .cal optional but explicit
CalendarsTable[icsfilebasename] = icsarray # add to global calendars table
CalendarsDirty[icsfilebasename] = False # no changes yet at startup
self.any_file_ops_since_empty = True
# extract data used here, index by start date and uid
for event in icsarray.walk('VEVENT'): # for all events in parsed calendar
# icalendar.prop.vDDDTypes: .dt is datetime.{date or datetime} with .m/d/y
startdate = event['DTSTART'] # required start date
datestamp = event['DTSTAMP'] # required modtime: display order if no created
createdon = event.get('CREATED', None) # optional createtime: for display order
# icalendar.prop.vText (VText): .to_ical() is bytes, str() decodes auto
uniqueid = event['UID'] # required id, for update searches
labeltext = event.get('SUMMARY', '') # for main label and view/edit dialog
extratext = event.get('DESCRIPTION', '') # for view/edit dialog, main text
category = event.get('CATEGORIES', '') # for coloring and view/edit dialog
if isinstance(category, list):
category = category[0] # if multiple, use just one (first) for coloring
# vdates: .dt is a datetime.date/datetime object
# transfer to custom date type for better control here
eventdate = Edate.from_datetime(startdate.dt)
if createdon:
orderdate = Edate.from_datetime(createdon.dt)
else:
orderdate = Edate.from_datetime(datestamp.dt)
# vtext: see above--ok as is, but undo odd '\X' escapes in some existing data
labeltext = labeltext.replace('\\"', '"')
extratext = extratext.replace('\\"', '"') # '\"' -> '"'
# [1.4] undo =5C's from file, restoring '\'; see unescape function ahead
labeltext = backslash_unescape(labeltext)
extratext = backslash_unescape(extratext) # '=5C' -> '\'
# index this file's event data for date by start m/d/y
icsdata = EventData(uid=uniqueid,
calendar=icsfilebasename,
summary=labeltext,
description=extratext,
category=category,
orderby=orderdate,
veventobj=event)
if eventdate not in EventsTable: # .keys() optional, eitbti?
EventsTable[eventdate] = {} # first event for this date
EventsTable[eventdate][uniqueid] = icsdata # add to global events union table
def backup_and_save_all_changed_ics_files(self, folderPathOrUri):
"""
------------------------------------------------------------------
Backup and save the ICS files of all calendars that have been
changed since loaded or last saved. This is a two-step combo
that is run in a thread as one unit to avoid blocking the GUI.
Never called if no calendars have changes: caller ensures.
folderPathOrUri is a py str on PCs or a java Uri on Android.
Called by run_with_calendars_folder() only if the app already
has calendar-folder access or it's given in an popup precursor.
Runs platform-specific code in subclasses for actual file IO.
Subtle: if the user was asked for a new folder before this ran
due to permission/folder changes, this does _not_ load calendars
from the new folder, because doing so could lose any unsaved
changes in the in-memory calendars. This will simply save
changed calendars to the new folder. For more info on this,
see SAVES EXTRA in ask_calendars_folder_info().
------------------------------------------------------------------
"""
# backup+save files in new thread to avoid blocking gui
def backup_and_save_ics_files_thread():
# two-step combo
backupfails = self.backup_all_changed_ics_files(folderPathOrUri)
if not backupfails:
savefails = self.save_all_changed_ics_files(folderPathOrUri)
# on thread exit, update gui in gui thread
def on_backup_and_save_thread_exit(dt): # blocker is in enclosing scope,
#self.app.dismiss_popup(blocker) # despite the different threads
self.app.finalizeBusyPopup(blocker) # else too fast?, blocker in scope
if backupfails:
self.app.info_message(
'[b]Backups failed - all saves skipped[/b]'
'\n\n'
'Errors occurred while backing up these ICS files:\n\n'
f' {backupfails}\n\n'
'Due to the errors, no saves were performed. Other files '
'were backed up, and the prior versions of all ICS files '
'were retained.')
elif savefails:
self.app.info_message(
'[b]Saves failed - some files not saved[/b]'
'\n\n'
'Errors occurred while saving these ICS files:\n\n'
f' {savefails}\n\n'
'Due to the errors, these files\' saves were aborted, '
'but all other files\' saves were successful and all '
'ICS files were backed up.')
Clock.schedule_once(on_backup_and_save_thread_exit) # same as @mainthread
# post modal popup, start function in thread
blocker = self.app.postBusyPopup(kindtext='saving')
threading.Thread(
target=backup_and_save_ics_files_thread, # don't block gui thread
args=(), # uses locals in scopes
daemon=True).start() # DON'T close thread if app closed: writes
# and continue in gui thread with popup
def backup_all_changed_ics_files(self, folderPathOrUri):
r"""
------------------------------------------------------------------
Backup ICS file(s) that have changes on main-menu Save, as
a precursor to file saves.
folderPathOrUri is a py str on PCs or a java Uri on Android.
A true result denotes errors as a list of failed filenames
that cancels saves. Unlike loads, this runs a platform-specific
method in subclasses along the way to minimize some redundancy.
It's run within its caller's thread, so it cannot block the GUI.
------------------------------------------------------------------
"""
global CalendarsTable # used here
fails = []
icsfoldername = self.app.settings.calendarsfolder # str: POSIX path or SAF uri
assert icsfoldername == \
(folderPathOrUri.toString() if RunningOnAndroid else folderPathOrUri)
trace('Backing up to', icsfoldername)
# get|make bkp subdir: (str or Uri) or None
backupsSubfolder = self.backups_get_subfolder(folderPathOrUri)
if not backupsSubfolder:
fails.append('<Could not make backups folder>') # too rare to do more
else:
for icsfilename in CalendarsTable:
if not CalendarsDirty[icsfilename]:
# no changes since startup or save: don't backup/write this file
continue
try:
# trim backups folder if needed
# keeps most recent N-1 of each, based on sort order of bkp names
# glob returns pathname strs or DocumentFile objs
backuppatt = 'date*-time*--' + icsfilename
currbackups = self.backups_glob_dir(backupsSubfolder, backuppatt)
currbackups.sort(reverse=True,
key=lambda x: x if not RunningOnAndroid else x.getName())
prunes = currbackups[(self.maxbackups - 1):] # earliest last
for prunee in prunes: # globs have paths
trace('pruning', prunee)
try: # file deletes can fail:
self.backups_remove_file(prunee) # but don't cancel bkp/save
except:
trace(f'Error while pruning: skipped {prunee}') # too rare to do more
traceback.print_exc()
# transfer current file to backup folder/name
# note: move is quick and may be less error prone,
# but copy retains original file in case save fails
datetimestamp = time.strftime('date%y%m%d-time%H%M%S') # add date/time prefix
backupname = f'{datetimestamp}--{icsfilename}'
self.backups_backup_file(
folderPathOrUri, icsfilename, backupsSubfolder, backupname)
except:
# on any and first failure for this file
fails.append(icsfilename)
trace(f'Error while backing up: skipped {icsfilename}')
traceback.print_exc()
# tbd: break to stop now since saves will be cancelled?
return fails # [failed filenames] or []=False:OK
def save_all_changed_ics_files(self, folderPathOrUri):
r"""
------------------------------------------------------------------
Store events in ICS file(s) on main-menu Save, but only after
successful backups.
folderPathOrUri is a py str on PCs or a java Uri on Android.
A true result denotes errors as a list of failed filenames.
Unlike loads, this runs a platform-specific method in subclasses
along the way to minimize some redundancy. It's run within its
caller's thread, so it cannot block the GUI.
Coding note: calendar files are written here, and read in the
parse function, all at once with .read() and .write(all), not
in chunks via .read(N) and .write(part). These files are small
(app proprietor's largest is just 2M for 20 years), all usable
devices have gigabytes of memory today, and arbitrary buffer
sizes should work on all platforms under Python 3.X today. An
old Windows issue precluded writing buffers > 64M on network
drives but is fixed in recent libs (and 64M equates to 640
comparable years of events for the app proprietor's usage).
------------------------------------------------------------------
"""
global CalendarsTable # used here
fails = []
icsfoldername = self.app.settings.calendarsfolder # str: POSIX path or SAF uri
assert icsfoldername == \
(folderPathOrUri.toString() if RunningOnAndroid else folderPathOrUri)
trace('Generating to', icsfoldername)
for icsfilename in CalendarsTable:
if not CalendarsDirty[icsfilename]:
# no mods to this file since startup or Save: don't backup/write
continue
try:
icscalendar = CalendarsTable[icsfilename]
if not icscalendar.walk('VEVENT'): # a list
# if no events after New or deletes, add a required one now (rare but true)
vevent = new_vevent( # fix name error [1.5]
dtstart=None,
uid=icalendar_unique_id() + '-' + icsfilename, # make unique if > one [1.5]
summary=f'{APPNAME}-generated event',
description='Required sole event generated.\n\n' # add tips text [1.5]
'You can delete this event at any time '
'by tapping it within the app.\n\n'
'To completely remove this event\'s calendar, '
'delete its ".ics" file from your calendars folder.',
category='') # color? (else white) [2.0]
icscalendar.add_component(vevent) # was 'system internal'
icstext = icscalendar.to_ical() # bytes, major dependency
self.save_one_ics_file(icsfilename, folderPathOrUri, icstext) # platform-specific subs
CalendarsDirty[icsfilename] = False # no changes after save
self.any_file_ops_since_empty = True
except:
# skip file but keep going
fails.append(icsfilename)
traceback.print_exc()
trace(f'Error while generating ICS file: {icsfilename}')
return fails # [failed filenames] or []=False:OK
#==============================================================================
# 💻 PC SUBCLASS -- factored-out code for PCs only: POSIX/filepaths
#==============================================================================
class CalendarsStorage_PCs(CalendarsStorage):
"""
------------------------------------------------------------------
Subclass to factor out PC-specific calendars behavior.
On all PCs, use the portable POSIX toolset and file pathnames.
Don't use these on Android to avoid All Files Access permission.
------------------------------------------------------------------
"""
def pprintFolder(self, pathnameStr):
"""
------------------------------------------------------------------
Nothing to see here (show POSIX pathnames as they are).
------------------------------------------------------------------
"""
return pathnameStr
def default_calendar_folder_path(self):
r"""
------------------------------------------------------------------
The user's home folder on all PCs: app always has access.
Path: macOS=/Users/me, Windows=C:\Users\me, Linux=/home/me
For suggested Frigcal calendars-folder default on PCs - only.
------------------------------------------------------------------
"""
return os.path.expanduser('~')
def run_with_calendars_folder(self, nextop, forcedask=False):
r"""
------------------------------------------------------------------
PCs version: verify in filesystem or use folder chooser.
forcedask is True for folder changes in Settings.
nextop callable is run with pick after folder selection.
[See the Android subclass's version for common overview]
ABOUT WINDOWS/WSL2 PATH CONVERSIONS FOR OPENS:
This does not attempt to manually convert the priorPath
from Windows to WSL (Linux in Windows) format on WSL,
nor from WSL to Windows format on Windows.
It's possible to do so with a "wslpath" command lines in both
platforms (e.g., /home/me to \\wsl.localhost\Ubuntu\home\me on
Windows and C:\Users\me to /mnt/c/users/me on WSL2. See
main.py's launch_file_explorer() for code that does this
because it's required for file explorers. For file opens
here, this conversion happens automatically in Windows but
not in WSL2, for reasons with no known documentation.
Applying manual conversions here are more dubious. Detecting
platform-specific paths is iffy, and this seems a use case
with no users: it's possible only if Windows and WSL2 share
a common settings file, and this can only occur when running
a common source-code package on both. It's unclear that a
full source-code package will be released, and even if it is,
users will normally instead use separate Windows and Linux
executable packages that have their own settings files.
------------------------------------------------------------------
"""
haveaccess = False
priorPath = self.app.settings.calendarsfolder # None or prior ask
trace(f'{priorPath=}')
if priorPath != None:
try:
listed = try_listdir_nohang(priorPath)
if not listed:
# hung (inaccessible), or immediate fail (unmounted?)
haveaccess = False
else:
haveaccess = True
except:
pass
if haveaccess and not forcedask:
nextop(priorPath) # pass py str
else:
self.ask_calendars_folder_info( # step 1: info popup
nextop=lambda: self.ask_calendars_folder(nextop), # modal asks
forcedask=forcedask, # if Settings chdir
rawnextop=nextop) # for save caution
def ask_calendars_folder(self, nextop):
"""
------------------------------------------------------------------
Folder ask step 2: on all PCs, ask user to pick a calendars
folder in the Kivy folder chooser.
This varies completely from the Android version, which must use
the Android-only SAF chooser for permissions. PPUS's chooser has
far too much functionality to reuse here (e.g., removable-drives),
and Kivy's base chooser is really just a chooser-construction kit.
Instead, use the simple KivyMD chooser. Downsides: non-native and
limited (e.g., no new-folder op, though the app makes one in ~).
The chooser is inline code here, not .kv code (a past lesson).
Alternative: bundle tkinter and use its folder choosers from Kivy?
Per a web search ("bundle tkinter with a kivy app [on android]"),
this seems feasible on PCs but impossible on Android. We
don't care here for PCs, but there is a showstopper: tkinter's
chooser blocks the Kivy GUI: it won't redraw, resize, minimize...
It might work in a thread, but PyInstaller packaging is tricky.
tkinter's chooser also has known issues on some macOS (see PyEdit).
Alternative: Kivy's plyer also has a file chooser that tries to be
platform native, but it's unusable on Android (the SAF chooser is
required to get and persist permissions), and its folder chooser
is lousy on Windows (a shell hack that posts a blurry 1990s-ish
box that's so bad that plyer itself recommends alternatives).
The weak Windows support is a full showstopper for PCs here.
------------------------------------------------------------------
"""
# open in ~/Frigcal suggested default created by App.on_start()
trace('ask_calendars_folder')
def on_exit_folderchooser(*args):
trace('fm exit')
folderchooser.close()
self.app.folderchooser_open = False
def on_select_folderchooser(path):
trace('fm select:', path)
on_exit_folderchooser()
self.on_selection(path, nextop)
dohiddens = self.app.settings.chooserShowHiddenFiles # unchangeable (tbd)
startpath = self.app.defcalspath
# chooser silently refuses to open if the default path has been renamed
# we have limited access on Android, but this is run on PCs only (else SAF)
if not os.path.isdir(startpath):
startpath = self.default_calendar_folder_path() # ~ sans subfolder
folderchooser = MDFileManager(
search='dirs', # folders only
show_hidden_files=dohiddens, # tbd: Settings?
exit_manager= on_exit_folderchooser,
select_path=on_select_folderchooser
)
folderchooser.show(startpath)
self.app.folderchooser = folderchooser # for routing Back taps
self.app.folderchooser_open = True # back=>up in folderchooser
def on_selection(self, pickedPath, nextop):
"""
------------------------------------------------------------------
Folder ask step 3: after user has interacted with Kivy calendar-folder
chooser on PCs, run the nextop with the selected URI, if any.
------------------------------------------------------------------
"""
trace('on_selection', pickedPath)
if pickedPath:
# update folder setting (only) for pick
self.app.settings.update_and_save_settings(calendarsfolder=pickedPath)
# update Settings screen for pick
self.app.settingsgui.screen.ids.calendarsfolder.text = pickedPath
nextop(pickedPath) # pass selected path's py pathname str
# TBD: reload calendar files and redraw gui, unless nextop is load_all_calendars?
# NO: see ask_calendars_folder_info() - reload would delete changes
def load_all_ics_files(self, folderPath):
"""
------------------------------------------------------------------
thread(load calendars from folderPath), draw gui
Uses chosen POSIX pathname str, app has folder permission+access.
Called by run_with_calendars_folder() only if the app already
has calendar-folder access or it's given in an popup precursor.
Runs files-load loop in a thread to avoid blocking the GUI.
Runs superclass's generic/common parsing code in the thread.
See no_calendar_files_message() for TBD on policy used here.
------------------------------------------------------------------
"""
global CalendarsTable, CalendarsDirty, EventsTable # all reset in-place here
# TBD: add default calendar file if none in folder?
# NO: expect user to run New, plus Save for file/changes
trace('loading from', folderPath)
if not folderPath:
return
listed = try_any_nohang(os.listdir, folderPath, onfail=None)
if not listed:
# hung (inaccessible), or immediate fail (unmounted?)
calslist = []
else:
calslist = [filename for filename in listed
if filename.endswith(ICS_FILE_EXTENSION)] # i.e., glob.gob
trace(f'{calslist=}') # list of .ics filenames or []
if not calslist:
# alert user but load anyhow: clear prior folder's events, if any
self.no_calendar_files_message(folderPath)
# clear global tables for Settings-screen folder change, in-place
self.clear_calendar_and_event_tables()
# load files in new thread to avoid blocking gui
def load_all_ics_files_thread():
fails = []
for calname in calslist:
calpath = osjoin(folderPath, calname)
try:
fileobj = open(calpath, 'r', encoding='utf8') # open can fail
except:
fails.append(calname)
trace(f'{calname=} skipped:\n{sys.exc_info()=}')
else:
try:
self.parse_one_ics_file(fileobj, calname) # *common ops super
trace(f'{calname=} parsed') # read/parse can fail
except:
fails.append(calname)
trace(f'{calname=} skipped:\n{sys.exc_info()=}')
finally:
fileobj.close() # always close object
# (load thread) on thread exit, update gui in gui thread
def on_load_thread_exit(dt):
#self.app.dismiss_popup(blocker) # may be too fast: button
self.app.refill_month_widgets() # update gui
self.app.finalizeBusyPopup(blocker) # blocker in scope
if fails:
self.app.info_message(
'[b]Some files not loaded[/b]'
'\n\n'
'The following ICS files had loading errors '
'and were skipped:\n\n'
f' {fails}\n\n'
'These files\'s events will not be shown.')
# (load thread) schedule gui update
Clock.schedule_once(on_load_thread_exit) # == @mainthread
# show modal popup, start function in thread
blocker = self.app.postBusyPopup(kindtext='loading')
threading.Thread(
target=load_all_ics_files_thread, # don't block gui thread
args=(), # uses locals in scopes
daemon=False).start() # close thread if app closed: reads
# and continue in gui thread with popup
def save_one_ics_file(self, icsfilename, icsfolder, icstext):
"""
------------------------------------------------------------------
Save one (changed) calendar to its ICS file, by POSIX + pathname.
Run as part of a thread by super's common ICS-file maker loop.
- icsfilename is the 'xxx.ics' name of the calendar being saved.
- icsfolder is the POSIX folder filepath str chosen by the user.
- icstext is the bytes object created for ICS text, made by the
icalendar lib for the in-memory calendar.
------------------------------------------------------------------
"""
icsfilepath = osjoin(icsfolder, icsfilename)
trace('writing', icsfilepath, '...')
if not osexists(icsfilepath):
trace(f'creating new ics file: {icsfilename}')
try: # write or overwrite
fileobj = open(icsfilepath, 'wb') # text is bytes, not str
except:
trace(f'open fail: {sys.exc_info()=}')
else:
try:
fileobj.write(icstext) # all at once: small files
except:
trace(f'write fail: {sys.exc_info()=}')
finally:
fileobj.close() # flush, exc or not (!with: eibti)
def backups_get_subfolder(self, icsfolderPath):
"""
------------------------------------------------------------------
This and its 3 followers are called by super's common backups loop
and implement the POSIX/filepath versions of the required ops.
All are run from the caller's thread, so they cannot block GUI.
This method returns pathname str or falsy None.
------------------------------------------------------------------
"""
backupdir = osjoin(icsfolderPath, BACKUPS_SUBDIR_NAME)
try:
if not os.path.exists(backupdir):
os.mkdir(backupdir)
assert os.path.exists(backupdir) and os.path.isdir(backupdir)
return backupdir
except:
trace('backupDir make fail or already a non-dir')
return None
def backups_glob_dir(self, backupsSubfolderPath, backuppatt):
return glob.glob(osjoin(backupsSubfolderPath, backuppatt)) # result has paths
def backups_remove_file(self, pruneePath):
os.remove(pruneePath)
def backups_backup_file(self, folderPath, icsfilename, backupsSubfolder, backupname):
# any exception caught by caller
frompath = os.path.join(folderPath, icsfilename)
topath = os.path.join(backupsSubfolder, backupname)
# skip: new in-memory calendar that doesn't yet have a file
if not osexists(frompath):
trace(f'no file to backup: {icsfilename}')
return
else:
trace(f'saving "{frompath}" to "{topath}"')
shutil.copy2(frompath, topath) # copy content+info (write)
#==============================================================================
# 📱 ANDROID SUBCLASS -- factored-out code for Android only: SAF/URIs
#==============================================================================
class CalendarsStorage_Android(CalendarsStorage):
"""
------------------------------------------------------------------
Subclass to factor out Android-specific calendars behavior.
On Android, use the SAF toolset and content URIs for permission.
We can't use POSIX/pathnames on Android without All Files Access,
and the SAF chooser returns content URIs that provide permission
but are not filepaths and tightly bind file ops to the SAF API.
------------------------------------------------------------------
"""
# till loaded from config or asked
calendarsFolderURI = None
def pprintFolder(self, escapedUriStr):
"""
------------------------------------------------------------------
Content URIs received from the SAF chooser may have a few
embedded %3A escapes (for ':') and %2F (for '/') that app
users shouldn't have to grok (content URIs are iffy as is).
------------------------------------------------------------------
"""
# no: java, via pyjnius
#URLDecoder = autoclass('java.net.URLDecoder')
#return URLDecoder.decode(encodedUriStr, 'UTF-8')
# yes: python, direct
from urllib.parse import unquote_plus
return unquote_plus(escapedUriStr, encoding='utf-8', errors='ignore')
def default_calendar_folder_path(self):
"""
------------------------------------------------------------------
The internal-storage root on Android: requires user permission.
Path: /storage/emulated/0 or its /sdcard synonym on some devices.
This is general shared storage, not app-specific nor app-private.
For creating a suggested-default calendars folder on first run.
UNUSED: shared storage cannot be used for a suggested calendars-
folder default on Android because it requires a user permission
granted by either All Files Access or single-folder choice:
- All Files Access would allow POSIX APIs but is unused by this
app because it would likely preclude Play store distribution.
- Single-folder choice, used for the calendar folder in general,
would require opening the SAF folder chooser just to access a
known calendar default folder, and the chooser returns a
content URI which essentially bind ops to the SAF API.
The SAF chooser also comes with a poorly thought-out catch-22 for
the initial folder path: a content URI is required to point users
to a suggested folder in the chooser, but apps cannot get a
suggested folder's URI without using the chooser. IOW, apps
must use the SAF chooser to get a URI to pass to the SAF chooser!
More broadly, there is no good path to use for either calendars
or settings out of the box. Besides shared storage:
- A Documents/ URI might be forged, but it's iffy, and still
requires user permission as for all of shared storage.
- An app-specific (aka "external") folder located at:
/storage/emulated/0/Android/data/com.quixotely.frigcal/files
is no longer accesisble to most file-explorer apps as of 2025.
App-specific is still available in Cx (oddly), but it generally
requires the Files app (which runs AOSP's explorer) or root
emulation with the shizuku app (which may not be available).
- An app-private (aka "internal") folder in /data is never
accessible to other apps, including explorers and syncs.
Both app-specific and app-private can be served to other apps by
coding a file provider, but explorer-app support is limited, and
both app-specific and app-private are deleted on app uninstalls.
Hence, PUNT: there is no good option for this default in Android
scoped storage, so users must make their own folder. See also
App.storage_path_app_private()'s similar dilemma for settings;
Android's lockdowns convolute development and limit users - badly.
------------------------------------------------------------------
"""
return android.storage.primary_external_storage_path()
# arbitrary number that ids a specific Android intent (how 1980s...)
REQUEST_ID_OPEN_DOCUMENT_TREE = 987654
def run_with_calendars_folder(self, nextop, forcedask=False):
"""
------------------------------------------------------------------
Android version: verify by SAF or use SAF chooser
Overview: Called on all calendar loads and saves to verify or
ask for the calendars folder. Run by loads for app startup,
saves for main-menu's Save Calendars (including after change
alerts), and loads for calendar-folder changes in the Settings
screen. forcedask is True only for the latter, and nextop is
a callable that's run with the pick after a folder is selected.
Note that users can remove either a folder or the app's permission
to a folder at any time, so the app must check before each action,
even if already asked and persisted.
On Android, there is a fundamental schism between POSIX filepaths
and SAF content URIs - they use very different APIs, and converting
between the two is iffy ('file://' URIs are no longer supported).
Fetch POSIX file descriptors from the SAF API where possible, and
assume that the SAF API calls used cannot hang (see hangless.py).
------------------------------------------------------------------
"""
haveaccess = False
Uri = autoclass('android.net.Uri')
DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile')
priorUriString = self.app.settings.calendarsfolder # None or prior ask
trace(f'{priorUriString=}')
if priorUriString != None:
priorUri = Uri.parse(priorUriString) # py str => java Uri
# check uri in persisted permissions
activity = self.app.get_android_activity()
context = self.app.get_android_context()
persistedUriPermissions = \
activity.getContentResolver().getPersistedUriPermissions()
hasUriPermission = any(
priorUriString == uritry.getUri().toString()
for uritry in persistedUriPermissions)
trace('perms:')
trace([uritry.toString() for uritry in persistedUriPermissions])
trace(f'{hasUriPermission=}')
if hasUriPermission:
# try listing: it may not be accessible now
try:
documentsTree = DocumentFile.fromTreeUri(context, priorUri)
if documentsTree:
for documentFile in documentsTree.listFiles():
break
haveaccess = True
except:
pass
if haveaccess and not forcedask:
nextop(priorUri) # pass java Uri
else:
self.ask_calendars_folder_info( # step 1: info popup
nextop=lambda: self.ask_calendars_folder(nextop), # modal asks
forcedask=forcedask, # if Settings chdir
rawnextop=nextop) # for save caution
def ask_calendars_folder(self, nextop):
"""
------------------------------------------------------------------
Folder ask step 2: on Android, ask user to pick a calendars
folder in Android's SAF UI so permissions are granted. The
content URI returned requires later processing to use SAF too,
unless file descriptors can be fetched/used (not for folders).
On Android, the SAF picker is effectively modal (blocking)
because the app is paused until the user closes the picker
(App on_pause() and on_resume() run on picker open/close):
'''
While it is technically a separate system activity and not a
modal dialog in the traditional programming sense, it has the
practical effect of being a modal experience from the user's
perspective, as it blocks interaction with your app until the
task is complete.
'''
Hence, the app can chain actions assuming the picker will send
a result to its callback before anything else happens in the app.
The missing magic bits: "p4a_activity" uses pyjnius to make a
Java class with a callback method named onActivityResult() in
Python, which is registered to Java through the app's Android
activity and simply calls the Python callable that we pass to
p4a_activity.bind() here when the Intent finishes:
https://github.com/kivy/python-for-android/blob/develop/
pythonforandroid/recipes/android/src/android/activity.py
SUBTLE: this must also _unbind_ any prior binds, here or in the
results handler, or arrange to bind just once (e.g., in __init__).
Else, N prior runs and binds here means N+1 calls to the result
handler instead of 1 (yes, really!). Barring user folder or
permission changes, this crops up only for the Settings screen's
calendar-folder change but is an arguably perilous API choice.
See the above activity.py; its code seems the only docs on this...
Tbd: Intents sent to and from Android have putExtra()/get*Extra() to
pass data on to the result handler, but it's Java strongly typed, and
it's unclear how to send a Python callback object to/from this API.
PUNT: the current coding works as is.
TBD: folder-ask intent also has an initial folder to open the popup
at, but converting the default filepath to a URI is iffy~impossible
[intent.putExtra(DocumentsContract.EXTRA_INITIAL_URI, uriToLoad)]
_May_ work, possibly with file:// prefix:
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N)
fileUri = Uri.parse(self.app.defcalspath);
else
fileUri = Uri.fromFile(new File(self.app.defcalspath));
Uri.parse(new File("/sdcard/Documents/Frigcal").toString())
PUNT: this is just how the AOSP/SAF icker works.
------------------------------------------------------------------
"""
self.nextop = nextop # post-user-choice action
# python-for-android's activity class: for on_activity_result listener
from android import activity as p4a_activity
# use SAF chooser for Android permissions, not kivy widget
# must use mActivty, not the casted app activity
mActivity = self.app.get_android_activity(docast=False)
# ask user, run action after UI result appears later
Intent = autoclass('android.content.Intent')
intent = Intent()
intent.setAction(Intent.ACTION_OPEN_DOCUMENT_TREE)
# required: unbind prior bind if any, else N callbacks for N binds!
p4a_activity.unbind(on_activity_result=self.on_ask_calendars_folder_result)
# bind for results callback via p4a activity
p4a_activity.bind(on_activity_result=self.on_ask_calendars_folder_result)
mActivity.startActivityForResult(intent, self.REQUEST_ID_OPEN_DOCUMENT_TREE)
# now the SAF picker popup appears and the app waits for the user's response
def on_ask_calendars_folder_result(self, requestCode, resultCode, intent):
"""
------------------------------------------------------------------
Folder ask step 3: after user has interacted with SAF calendar-folder
chooser on Android, run the nextop with the selected URI, if any.
This is called by the Java API (by way of the Kivy Activity class) when
the user dismisses the folder picker. The app is blocked until the
user selects a folder or cancels the picker, so it's effectively modal.
The selected URI is in the intent and is a java Uri, not a py str.
This could convert URI to filepath (sort of; see web), but that would
limit the app to physical storage. We may or may not need to care, but
using content URIs allows for virtual storage like clouds, which may be
useful but are also innately perilous from a privacy/safety perspective.
------------------------------------------------------------------
"""
trace('saf picker callback')
activity = self.app.get_android_activity()
Activity = autoclass('android.app.Activity')
Intent = autoclass('android.content.Intent')
if requestCode != self.REQUEST_ID_OPEN_DOCUMENT_TREE:
trace('folder picker - irrelevant intent result')
else:
if resultCode == Activity.RESULT_CANCELED:
trace('folder picker - cancelled')
elif resultCode != Activity.RESULT_OK:
trace('folder picker - unknown resultCode:', resultCode)
else:
# OK: persist permission, save its uri, run nextop
# chosen uri is in passed intent
pickedUri = intent.getData() # content uri, not posix pathname
if not pickedUri: # assume it's a folder & accessible
trace('folder picker - no URI')
return # possible?
# save to settings as a str, Uri.parse(str) undoes
pickedUriString = pickedUri.toString() # java Uri => py str
trace('folder picker results:', pickedUriString)
# update setting in app-private storage, auto backed up
self.app.settings.update_and_save_settings(
calendarsfolder=pickedUriString)
# update Settings screen for pick: must be run in gui thread
def setum(dt):
guifield = self.app.settingsgui.screen.ids.calendarsfolder
guifield.text = pickedUriString
Clock.schedule_once(setum)
# save persistable URI permission for long-term access
activity.getContentResolver().takePersistableUriPermission(
pickedUri,
Intent.FLAG_GRANT_READ_URI_PERMISSION |
Intent.FLAG_GRANT_WRITE_URI_PERMISSION
)
# run dependent action: ensure run on main gui thread if needed;
# same as calling a @mainthread decorated callback func now/here;
# passes the picked folder as the java Uri, not the py str;
Clock.schedule_once(lambda dt: self.nextop(pickedUri))
# TBD: reload calendar files and redraw gui, unless nextop is load_all_calendars?
# NO: see ask_calendars_folder_inf() - reload would delete changes
def load_all_ics_files(self, folderUri):
"""
------------------------------------------------------------------
thread(load calendars from folderUri), draw gui.
Uses chosen SAF folder URI, app has folder permission+access.
folderUri is the java Uri from asker, not the py str.
Called by run_with_calendars_folder() only if the app already
has calendar-folder access or it's given in an popup precursor.
Runs files-load loop in a thread to avoid blocking the GUI.
Runs superclass's generic/common parsing code in the thread.
See no_calendar_files_message() for TBD on policy used here.
About DocumentFile: it's a wrapper around a URI that represents
a document or directory managed by a SAF ContentProvider, which
provides file and directory access sans ContentResolver queries.
Requires androidx.documentfile.documentfile:1.1.0 in bd.spec.
About fds: this reads clendars in Python via os file descriptor
fetched from SAF, coded per Java examples + Android docs (yuck!).
Do not also run afd.close() (parcel file descriptor): closefd=True
by default. Can also get file uri from its name this way:
Uri = autoclass('android.net.Uri')
baseuri = Uri.parse(folderUri)
itemUri = Uri.withAppendedPath(baseUri, calname)
------------------------------------------------------------------
"""
global CalendarsTable, CalendarsDirty, EventsTable # all reset in-place here
trace('loading from', folderUri and folderUri.toString() or '?')
if not folderUri:
return
# fetch a list of the *.ics files in the calendars folder
activity = self.app.get_android_activity()
context = self.app.get_android_context()
DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile')
# get DocumentFile objects for the selected directory
calslist = []
documentsTree = DocumentFile.fromTreeUri(context, folderUri)
if documentsTree:
for documentFile in documentsTree.listFiles():
if documentFile.isFile():
documentName = documentFile.getName()
if documentName and documentName.lower().endswith(ICS_FILE_EXTENSION):
calslist.append((documentName, documentFile.getUri()))
trace(f'{calslist=}') # list of .ics filenames or []
if not calslist:
# alert user but load anyhow: clear prior folder's events, if any
self.no_calendar_files_message(self.pprintFolder(folderUri.toString()))
# clear global tables for Settings-screen folder change, in-place
self.clear_calendar_and_event_tables()
# read calendars via os file descriptor fetched from SAF
contentResolver = activity.getApplicationContext().getContentResolver()
# load files in new thread to avoid blocking gui
def load_all_ics_files_thread():
fails = []
for (calname, caluri) in calslist:
afd = contentResolver.openFileDescriptor(caluri, 'r')
if not afd:
fails.append(calname)
trace(f'{calname=} skipped: no fd')
else:
try:
fd = afd.detachFd() # back in py (ahh)
fileobj = os.fdopen(fd, 'r', encoding='utf8') # open can fail
except:
fails.append(calname)
trace(f'{calname=} skipped: open {sys.exc_info()=}')
else:
try:
self.parse_one_ics_file(fileobj, calname) # *common ops super
trace(f'{calname=} parsed') # read/parse can fail
except:
fails.append(calname)
trace(f'{calname=} skipped: parse {sys.exc_info()=}')
finally:
fileobj.close() # always close object
# (load thread) on thread exit, update gui in gui thread
def on_load_thread_exit(dt):
#self.app.dismiss_popup(blocker) # may be too fast: button
self.app.refill_month_widgets() # update gui
self.app.finalizeBusyPopup(blocker) # blocker in scope
if fails:
self.app.info_message(
'[b]Some files not loaded[/b]'
'\n\n'
'The following ICS files had loading errors '
'and were skipped:\n\n'
f' {fails}\n\n'
'These files\'s events will not be shown.')
# (load thread) schedule gui update
Clock.schedule_once(on_load_thread_exit) # == @mainthread
# show modal popup, start function in thread
blocker = self.app.postBusyPopup(kindtext='loading')
threading.Thread(
target=load_all_ics_files_thread, # don't block gui thread
args=(), # uses locals in scopes
daemon=False).start() # DO close thread if app closed: just reads
# and continue in gui thread with popop
def save_one_ics_file(self, icsfilename, icsfolder, icstext):
"""
------------------------------------------------------------------
Save one (changed) calendar to its ICS file, by Android SAF.
Run as part of a thread by super's common ICS-file maker loop.
- icsfilename is the 'xxx.ics' str name of the calendar being saved.
- icsfolder is the SAF folder content URI chosen by the user:
a java Uri here, not a py str.
- icstext is the bytes object created for ICS text, made by the
icalendar lib for the in-memory calendar.
TBD: documentDir.createFile is supposed to be called sans extension,
on the assumption that the provider adds one per MIME type. This
seems gray in practice, but try type/calendar for a .ics instead
of text/plain which may add .txt (alt: Application/octet-stream?).
UPDATE: this works as coded.
ABOUT 'stupid_saf_mode': Android 10 bizarrely changed the open
mode for contentResolver.open* such that the standard 'w' mode
no longer truncates the file to zero bytes. The net effect can
leave garbage bytes at the end of the file if its new content is
shorter in the new write. This is wildly proprietary and was
woefully undocumented (and comes off as naive and arrogant).
THE FIX: Mode 'wt' can force truncation but apparently failed for
Google Drive's SAF provider. Mode 'rwt' used here, and recommeded
as a workaround by Google/Android, truncates too and works in GD.
The garbage bytes seemed sporadic so it's not certain that this
fixes the problem, but all signs point to yes.
FOR MORE on this Android 10 breakage (and a weak Google reply):
https://issuetracker.google.com/issues/180526528
or search: "android saf open for writing does not truncate file"
Python's file.truncate(0) also truncates, and writing the bytes
in smaller chunks may help (fails were seen on a 2.3M file), but
a single write() works outside Android, and neither of these
should be required. An earlier flush() fix attempt is now moot.
------------------------------------------------------------------
"""
trace(f'writing {icsfilename} in {icsfolder.toString()} ...')
activity = self.app.get_android_activity()
context = self.app.get_android_context()
DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile')
documentDir = DocumentFile.fromTreeUri(context, icsfolder)
if not documentDir:
trace('No documentDir')
else:
documentFile = documentDir.findFile(icsfilename)
if not documentFile:
# may be first save for a new calendar
trace(f'creating new ics file: {icsfilename}')
documentFile = documentDir.createFile('text/calendar', icsfilename)
if not documentFile:
trace('No documentFile')
else:
# write as bytes via file descriptor
stupid_saf_mode = 'rwt' # see above
icsfileuri = documentFile.getUri()
contentResolver = activity.getApplicationContext().getContentResolver()
afd = contentResolver.openFileDescriptor(icsfileuri, stupid_saf_mode)
if afd:
try:
fd = afd.detachFd() # back in py (ahh)
fileobj = os.fdopen(fd, 'wb') # open+write can fail
except:
trace(f'{icsfilename=} skipped: {sys.exc_info()=}')
else:
try:
fileobj.write(icstext) # actually, it's bytes
except:
trace(f'{icsfilename=} skipped: {sys.exc_info()=}')
finally:
#fileobj.flush() # might matter for fd
fileobj.close() # always close object
def backups_get_subfolder(self, icsfolderUri):
"""
------------------------------------------------------------------
Verify or create backups subfolder.
-icsfolderUri here is a java Uri: not python str or DocumentFile.
This and its 3 followers are called by super's common backups loop
and implement the Android SAF/URI versions of the required ops.
Returns a DocumentFile wrapping a SAF content URI or falsy py None,
not a Uri or py str. Could convert result to Uri with DF.getUri()
but that would require extra reconversions later and Uri not used.
------------------------------------------------------------------
"""
trace(f'{icsfolderUri=}')
context = self.app.get_android_context()
DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile')
documentDir = DocumentFile.fromTreeUri(context, icsfolderUri)
if not documentDir:
trace('No documentDir')
return None
else:
try:
backupDir = documentDir.findFile(BACKUPS_SUBDIR_NAME)
if not backupDir:
trace('making backup dir')
backupDir = documentDir.createDirectory(BACKUPS_SUBDIR_NAME)
trace('have backup dir')
assert backupDir and backupDir.exists() and backupDir.isDirectory()
return backupDir
except:
trace('backupDir make fail or already a non-dir')
return None
def backups_glob_dir(self, backupsSubfolderUri, backuppatt):
"""
------------------------------------------------------------------
Returns a list of DocumentFile objects, for "*.ics" in Backups.
- backupsSubfolderUri is a DocumentFile, from backups_get_subfolder.
It wraps a Uri but isn't one itself, despite the early name.
------------------------------------------------------------------
"""
trace(f'{backupsSubfolderUri=}')
context = self.app.get_android_context()
DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile')
# vs glob.glob: SAF Java API is gross
calslist = []
##documentsTree = DocumentFile.fromTreeUri(context, backupsSubfolderUri)
documentsTree = backupsSubfolderUri
if documentsTree:
for documentFile in documentsTree.listFiles():
if documentFile.isFile():
documentName = documentFile.getName()
if documentName and fnmatch.fnmatch(str(documentName), backuppatt):
calslist.append(documentFile) # path implied, or .getUri()
return calslist
def backups_remove_file(self, pruneeDoc):
"""
------------------------------------------------------------------
Remove a file, the SAF way.
- pruneeDoc is a DocumentFile, from backups_glob_dir.
Or use a list of documentFile.getUri() results and:
assert context.getContentResolver().delete(pruneeUri, None, None) > 0
------------------------------------------------------------------
"""
assert pruneeDoc.delete() # a DocumentFile
def backups_backup_file(self, folderUri, icsfilename, backupsSubfolderUri, backupname):
"""
------------------------------------------------------------------
Read+write calendars via os file descriptors fetched from SAF.
- backupsSubfolderUri is a DocumentFile, from backups_get_subfolder.
- folderUri is a java Uri (not py str) and requires a fromTreeUri.
Vs joins + [shutil.copy2 or .read() + .write()]: SAF is horrific!
TBD: copying file metadata like timestamps seem impossible in SAF?
------------------------------------------------------------------
"""
activity = self.app.get_android_activity()
context = self.app.get_android_context()
DocumentFile = autoclass('androidx.documentfile.provider.DocumentFile')
# read FROM any exception caught by caller
documentDir = DocumentFile.fromTreeUri(context, folderUri)
assert documentDir
documentFileFrom = documentDir.findFile(icsfilename)
# skip: new in-memory calendar that doesn't yet have a file
if not documentFileFrom:
trace(f'no file to backup: {icsfilename}')
return
else:
trace(f'saving "{icsfilename}" to "{backupname}"')
contentResolver = activity.getApplicationContext().getContentResolver()
# read as text in py native code
afd = contentResolver.openFileDescriptor(documentFileFrom.getUri(), 'r')
assert afd
fd = afd.detachFd()
fileobj = os.fdopen(fd, 'r', encoding='utf8')
fromtext = fileobj.read()
fileobj.close()
# create+write TO, any exception caught by caller
##documentDir = DocumentFile.fromTreeUri(context, backupsSubfolderUri)
documentDir = backupsSubfolderUri
assert documentDir
documentFileTo = documentDir.createFile('text/calendar', backupname)
assert documentFileTo
# write as text in py native code ('rwt' not needed here)
afd = contentResolver.openFileDescriptor(documentFileTo.getUri(), 'w')
assert afd
fd = afd.detachFd()
fileobj = os.fdopen(fd, 'w', encoding='utf8')
fileobj.write(fromtext)
fileobj.flush()
fileobj.close()