Hi all,
As we work with a large amount of JSON data via api I’m looking into document dB storage as a better way to save and work with this data.
I understand Wappler doesn’t support such in dB manager so wondering if there are any forum recommendations for a NoSql DB that can easily be worked with via API for simple read/write/delete actions.
MongoDB atlas perhaps? AWS DynamoDB looks a bit more complicated to work with.
What are you going to do with this large json data? And how is it structured? Records or a lot of nested objects?
We currently is ElasticSearch to store and query over 300m very nested records.
I’m looking at a dB that we can store the full json records from the ES instance for both faster query in users project folders but also to edit/add data per user per record.
We could setup another ES instance for this - just exploring options
Depending on the data, how often it change and your query needs - it might be better to design an import process and normalize the data to relational database.
Dealing with highly nested data - even in document based, NoSQL databases, is easy to retrieve as whole but difficult to query in deep parts.
While once normalized - it will be well indexed and very fast to query.
I guess I find it challenging to convert so much data to MySQL DB. Right now I’m converting json to string, inputting in text dB field and then parsing that to be processed fir the user.
As the json we receive changes from record to record some arrays will have one value, others 6 - I’m not sure how best to normalise that.
A NoSql dB would avoid that part of the process.
I don’t need to deep query the NoSql dB but mainly just store and return records based on a single UID field, eg project_id and user_id.
@George I spent some more time researching this, and feel that certainly MySql8 could do a very nice job or us here using the JSON field type, especially with the new partial update function.
My question - I tried to use the JSON field type previously, but constantly had errors with this - and ended up having to just stringify the json and then input into a TEXT field.
Can you confirm if the current Wappler implementation supports full MySql8 functionality for the JSON field type, please before I go too far down this road?
I assume also that to use the JSON field, it will all need to be custom SQL query builder?
Support for JSON (mysql and postgresql) natively within Wappler would be awesome to have!
Agreed. We have the JSON field, but full blown native support - would be so very useful, for so many use cases and very poweful.