I’m using AppFlows to get email records in different folders/labels when the user clicks on a folder/label. Sometimes it takes up to a minute for the flow to finish. I want the user to be able to click on another folder/label and for it to kick off another AppFlow while the previous one is still running.
Right now, I get the follow message in Console.
The app flow component is comparable to a server connect component, it fetches some data from a local action instead of a remote action. The server connect component does abort the previous action when it was run again before the previous finished. With app flow it doesn't abort the previous action because it could lead to data loss and you can only run again when previous was finished. When you don't depend on data you can use a flow action directly on the event which doesn't depend on the data output and can be run multiple times while previous is still running.
Hey @patrick, thanks for the response. It took me a minute to recall what this was for, but I'm guessing this was because I had a link to this post in the Web Workers request.
For this specific request I separated the logic so the App flow updates the SQLite db and the switching between labels only queries the SQLite db.
But, the desire for web workers/shared workers to run app flows/api actions outside of the main process is still desired for various use cases.
The web workers will not fix the issue with running multiple flows at the same times in app flows but will indeed help with offloading the main process by doing heavy task is a separate thread.
Do you process a large amount of data in the flow that it takes so long?
When doing small tasks it is better not to use web workers since it also causes some extra overhead every time you start them up. Also sending large amount of data to a web worker can lockup your UI since it first has to be serialized it within the main process and when receiving it is first being parsed blocking the event loop.
I think my challenge is two-fold.
I do initially import all emails from third-party email providers, and providers like Gmail make the process challenging. For Gmail, I have to perform multiple API calls (ListMessages, GetMessage, GetAttachments, etc.) to get each email resulting in thousands of API calls for the average mailbox which can take hours. They are processed in batches of 500 at the moment and I have duplicated the flow across every page in my app so if a user navigates away from a page, the next page will kick off a new flow.
This can lead to recalling the same nextpage token and running through emails that it has already processed, but I went with a large page size to reduce the amount calls slightly and I just ignore the email if it's already inserted.
In terms of Wappler performance during this processing, it appears stellar. There are no UI issues, even when performing other actions like sending emails while it's processing the imports.
Once the full import is complete, then I start using historyIds to check for updates to the mailbox and process those.
I broke these two processes into separate App Flows. I think it would be ideal if the initial import could run in a shared worker that lived across the pages, so it didn't have to restart the import process and go back through emails that may have already been imported if shared workers somehow stayed alive across page changes. This isn't a single page app. Each page is its own HTML page.
I think I've narrowed down the UI blocking to the querying of the SQLite Email table. Because it's not reactive and there are tens of thousands of emails that it has to go through before loading.
I have added some indexes that do help keep the app from fully freezing, but it still takes time to load as seen in this video and this causes some issues.
Once it's loaded, all is well, until I perform an action that requires it to reload all of the emails again. For example, opening an email and navigating back causes an API to fire to gmail to mark the email as read. This also updates the SQLite db, which I then need to reload. Since it is not reloading just the one email it again blocks the UI which makes clicking on another email or scrolling not work during that time.
The issue also occurs when the system adds a new email and needs to reload the list.
Here's a video showing that. When I navigate back to the list, there is about a second where I am unable to scroll or click into another email while it is reloading the entire list.
I believe a solution to this post would help resolve it.
Have you considered using IndexedDB Keith?
IndexedDB is a database that is built into a browser, much more powerful than localStorage.
localStorage.I was initially using IndexedDB via PouchDB, but I started to experience memory leaks, so I switched to SQLite.
I also plan to encrypt the SQLite database and store the keys in OS credential vaults, once I can talk the Wappler team into adding some of my feature requests, which, AFAIK, isn't possible in IndexedDB. 
Not sure if you already do this but on the initial fetch why not just fetch the subject, sender, and time stamp of the message rather than the full message contents? Then store this in cache and only update from the last cached record (historyId) when the user returns back rather than fetching the lot again? Twenty Four thousand plus records is quite a lot... Is this not an extreme circumstance fetching so many messages? I know some hoarders but daaaammmmnn that takes the biscuit! 
Maybe limit the initial fetch to the most recent 100 messages and then do it in batches based on pagination..?
All the data already exists in the SQLite db. It's just querying through 40K records looking for messages where Labels contains "INBOX". I don't believe it would make much difference if I return a subset of fields or all.
The way I structured the Labels in a json field versus a related table may cause some performance issues in the querying, but I think it would be ideal for the SQLite db to be reactive and only provide the one record that was updated and not require a full pull of the 100 records.
It's not fetching all records, only 100.
Why are you not using Googles own Gmail API to do the work for you?
I am using their APIs to populate the SQLite DB. Long-term I want the email client to work when the user is temporarily offline, so storing the emails locally is needed.
I'd put a limit on the number they can store locally but then that is me. Surely the User must expect some limit? I'd assume some 80% of those stored messages outside of the most recent 30 days are the items they are interested in, been interacting with, responding to? If a User wants to store 24k of messages locally then schedule it and inform the User the task is complete. But then I see this is Capacitor... And offline. Sorry if I wasted your time Keith!
Most non-mobile email clients (Outlook, Thunderbird, etc.) download all messages. Mobile clients like gmail use sync windows (e.g. 90 days), but there's more complexity involved in building similar functionality. At some point, I'll probably move that way for the mobile clients, but I really really really need to gain some traction first. I've spent way too many years building and I'm trying to get something out the door to validate if I should continue.
I know the feeling very well indeed Keith and highly admire what you are doing.
The only suggestion I could give would be to show a little Toast in the bottom corner letting the User know that syncing with the provider/database is taking place and maybe have a little spinner show while the sync is occurring (within the Toast, not a Preloader). For me personally I'd accept this as a User quite happily (and expect a short delay especially if I had tens of thousands of messages stored and being retrieved). Maybe fade the Container slightly while this sync takes place as well..?
This particular issue isn't related to the syncing of emails from Gmail (we've gotten a little off in this post from what it was originally created for).
This is related to the emails that already exist in the SQLite DB (i.e. dmx.app.data.Emails) and the UI blocking that is happening while it reruns the database SQL query due to some change to the data (i.e. removing the UNREAD label after a user clicks into the email from the list view).
When a user clicks on an email in list view, I trigger a call to gmail to remove the UNREAD label, which then updates the SQLite DB as well. Once the user is done with the email, and clicks the back button, I trigger the email list to reload (i.e. rerun the SQL query to refresh the list) so it shows the email as being read.
What would be ideal is if the reload only changed the one threadMessage in the array, not refresh the entire array via the SQL query.
