The Whys and Hows of Data Stores

I’ve been working with Data Stores a lot this week and have had a few questions about my work, so I thought it would be helpful to have a thread about how we are using them.

Here is my experience

Why Use Data Stores

I am using them for two reasons:

  1. To create a local database that mirrors part of my actual database.
    The reason for this is to minimise database hits when people are searching for items. So for example a user of my system may have 1000 contact records, each of which has 20 fields. If you use the “normal” method of doing search by name for example, Wappler will hit the database literally each time you type another letter in the search box and bring back loads of data. When my app has 1000s of users this will be very expensive (£,$,€) in terms of database usage. If I read the basic details of all the contacts into a data store, all the searching can go on locally on the user’s machine and then I can just go to the database once for the one record they want to edit.

Of course, in doing this you need to consider the maximum you can store locally… and data security… I’d be interested to hear people’s knowledge and experience about this!

  1. Booking Form Implementation
    I will create a large booking form, and I will use data stores to record a user’s responses, then when they press [Book Now] I will hit the database once with all their final responses.

How I am Using Them

I’m creating a different data store for each database table I want to mirror, so one for contacts, one for events, etc.

Each data store just has simple variables in it to store the associated column values, so one for first_name, last_name etc.

I have extra variables in the data store for local actions taken within the app. So for example if someone searches all contacts with a gender of male then I have a flow which sets an is_searched variable to true for all males, and a repeat which lists all contacts where is_searched==1. I also have an is_selected variable to select a working list of contacts to apply a single action to like sending an email.

In the process, I have learned that you have to be a bit careful with data types… in particular:

  • You may need .toNumber() to convert text to numbers in the Which Record search part of the Data Store update.

  • I define fields which represent yes/no values as type number rather than boolean as it is consistent with how the database tends to work and how we tend to represent yes/no values in the app construction.

  • There is no date type, so use types text or number depending on if you have a date of type datetime or timestamp.

Also, be careful with the use of session storage. Remember that if you don’t tick the use session storage box then the data will remain on the computer after the user closes the tab of the browser running your app.

Oh, and once you have named a data store, it is best not to re-name it right now.

In order to fill data stores from the database, I am using flows… there is an article below about how I do that.

I’m really loving working with Data Stores.

Be great to hear your experiences too!

Best wishes,
Antony.

8 Likes

I’ve never used them. However, reading your post I believe I need to learn the process. If you don’t mind, I would like to ask more questions as they arise. Thanks for what you have shared so far. I really appreciate it!

2 Likes

It doesn’t have to do this, and usually it’s something you would want to avoid - particularly in the situation you describe, where there are lots of users. The most expensive queries will be initial ones, when the first few letters are typed - and these queries will probably be of little or no use to the person searching. There are several ways you could improve this - eg by using debounce (simplest method), by only searching after a specified number of characters is typed or by searching only when the Enter key is pressed. With a little tweaking you could dramatically decrease the number of queries and amount of redundant data returned, and increase usability. Obviously the best approach is determined by a number of factors, but in any case, you are in control.

3 Likes

Agree with Tom :slight_smile: also in addition - you can just cache the server action results and not reload them every time.

1 Like

I appreciate your ideas @TomD!

I’ve almost finished the Data Store solution now, so I’ll bear all that in mind for the next high search volume data type!

Antony,
so, 2 years down the road.
Have you realized you could have had a performance improvement by using API Caching?.. or are you still glad you took the data store approach back then?

In more recent projects what are your thoughts on those two components?

1 Like

@Antony

id love to know your thoughts as well with what rebert had asked
if ya have the time

Hey folks, thanks for your posts!

My app is developed and I’m on marketing now, so I’m sorry, I have no opinion on this!

What is API Caching… the term isn’t familiar to me!