I think my challenge is two-fold.
- First issue
I do initially import all emails from third-party email providers, and providers like Gmail make the process challenging. For Gmail, I have to perform multiple API calls (ListMessages, GetMessage, GetAttachments, etc.) to get each email resulting in thousands of API calls for the average mailbox which can take hours. They are processed in batches of 500 at the moment and I have duplicated the flow across every page in my app so if a user navigates away from a page, the next page will kick off a new flow.
This can lead to recalling the same nextpage token and running through emails that it has already processed, but I went with a large page size to reduce the amount calls slightly and I just ignore the email if it's already inserted.
In terms of Wappler performance during this processing, it appears stellar. There are no UI issues, even when performing other actions like sending emails while it's processing the imports.
Once the full import is complete, then I start using historyIds to check for updates to the mailbox and process those.
I broke these two processes into separate App Flows. I think it would be ideal if the initial import could run in a shared worker that lived across the pages, so it didn't have to restart the import process and go back through emails that may have already been imported if shared workers somehow stayed alive across page changes. This isn't a single page app. Each page is its own HTML page.
- Second issue
I think I've narrowed down the UI blocking to the querying of the SQLite Email table. Because it's not reactive and there are tens of thousands of emails that it has to go through before loading.
I have added some indexes that do help keep the app from fully freezing, but it still takes time to load as seen in this video and this causes some issues.
Once it's loaded, all is well, until I perform an action that requires it to reload all of the emails again. For example, opening an email and navigating back causes an API to fire to gmail to mark the email as read. This also updates the SQLite db, which I then need to reload. Since it is not reloading just the one email it again blocks the UI which makes clicking on another email or scrolling not work during that time.
The issue also occurs when the system adds a new email and needs to reload the list.
Here's a video showing that. When I navigate back to the list, there is about a second where I am unable to scroll or click into another email while it is reloading the entire list.
I believe a solution to this post would help resolve it.