Right now on one of our main apps, we query a significant amount of data via ElasticSearch and return this in a repeating format.
The interface has over 29 different filters currently, including boolean and keyword search.
Would this need to be custom built, or does Wappler have the capabilities to handle the initial outbound query with such complexity? E.g. all 29 filters being used at once, and each filter change then resubmits the query to ES.
I’d appreciate being pointed in the correct direction for any docs/reading about Wappler handling this so I can learn ahead of us testing this out.
I’m not sure if this is just about the length of the query or some specific type of query. I have a query with about 85 parameters. This uses query builder. There are not 85 URL params - some are split up before being passed to the query. After the query is run, the script checks for corresponding images and PDFs in two folders, one of which contains thousands of files. It all runs very quicky and without any problem (probably less than second), but only a proportion of the 85 possible parts of the query will be executed - there are lots of conditions.
I haven’t any experience of such large databases; it may be that special considerations would have to be made in such cases. Typically, the number of records I deal with is not more than 10’s of thousands in a table, never more than 200k. I’ve tested the search I mentioned above with a table containing about 120k records. It probably wouldn’t make much difference if there were many more records, but I don’t know about 100s of millions. The fields included in the queries often contain quite lot of text - hundreds, occasionally thousands of words.
I use various types of searches in the same query, eg search by category/subcategory, advanced search, keyword search, search by selected IDs - so there is only one results page. This is made easy using the great feature in query builder to make each rule or set of rules conditional.
Keyword searches make up quite a few of the parameters in the query, as I split the words in the query string into a series of LIKEs, and search up to about 10 fields with these. (Using FULLTEXT searches might be a good alternative to this approach.)
I’m not sure if any of this is very helpful, and as I say, I haven’t any experience with datasets of the size you’re dealing with, but let me know if I there is anything specific I might be able to help with.
Thanks Tom. ES handles the size essentially - and so as long as it has enough resources I’d very fast.
What I’d be really appreciative of is guidance on how to setup such a large query - and do I use the query manager? I have 29 inputs That users can use some, all and also delete sum or all and the query should change each time - that’s the part I’m a little stuck on and how this would be done within Wappler
If you only need one type of search, create the form with all relevant inputs (perhaps in a modal) and create the server connect file, first creating all the GET parameters by selecting the file/search form. Then create your query, using query manager, adding a condition for each criterion/parameter - so any combination of fields can be used for the query. I store the form in an include file which is called from another include file (a header which appears on all pages).
I don’t think there is anything unusual here, so am not sure if this is helpful.
This looks like a good way to tackle it - the only variation is that I need to use JSON for the query, however, when you mention create your query using the query manager - do you mean the Front End component ‘Query Manager’ or build a quiery on the SC Action using all the GET parameters?
And sorry, one more, what do you mean/what condition would you add to each criterion/parameter?
If neither input has a value, this fields are ignored in the search. In the above screenshot, there is also an example of checking if two words are used in the Place field; if there are two words, the query will use both of them.
