@sid and is this what you are advising for within the single repeat?
Workflow:
API Call (this returns full schema). Output OFF.
Internal DB Query (checks API Call ID against internal db ID’s)
Repeat API Call Data. Output ON.
Like this?
Pretty much.
But set value have to be same names.
With output on.
If api does not give you the list of columns in repeat step, its a bug. Have encountered before. @patrick
As a workround you can try to type those out in the json itself.
if I open up the server connect in editor?
I’ve tried this, but i’m only returning 3 results, from the 15 that should display. Can you advise what condition you’d use in the repeat?
Really appreciate your help here - i’m feeling optimistic
@sid I thought I’d cracked it, I manually wrote in the final object in the API response (_source) - I suddenly had all the available fields to choose from in the repeat.
However, it errored when running the query telling me the repeat does not have array or object data.
Then I tried as you mention simply editing the JSON schema, with one nesting back (which avoided the above error).
I can only return the next object, which then returns ALL data. If I manually try to add in nested json in the output fields then it doesn’t return any data…
as I have _source - it returns the entire response. if I change _source to say _source.id or _source.first_name then I get no response data.
Changing to:
Then gives me all fields:
But I get a console error, repeat has no array or object, which appears to be a bug?
Any ideas?
Not sure how well Wappler can handle nested data.
Can you check if all _source keys have an array value? A null or non-array in between could be the cause of error on runtime.
How would I best define this?
Here is a screen of the response in network tab:
if you are referring ‘keys’ as the values within the _source object, then yep - lots do have null values, and this varies between record.
By key I mean _source itself.
So the last hits object is an array. It has the key _source.
Can you check if the value for _source is null in any of the rows in hits array?
none of the array items within hits are empty. These are keyed 0 to 25 and all hold data:
hits itself (one nest above _source) has no null values:
When I open _source this does indeed have many null values.
Ok. That looks fine.
Can you please also check if any of the _source keys in the 26 items inside hits are null?
Not values inside _source, but _source itself.
Thanks @sid. Just rechecked. _source itself never returns a null value. Within _source there are as always a large number of data points, and a few arrays so it always has something.
Ok this is good.
I rechecked the code you posted earlier… and there seems to be an issue there.
PDTSearchMain.data.hits.hits._source does not look right to be in a repeat.
It should be something like PDTSearchMain.data.hits.hits[0]._source, since second hits itself is an array.
No sure how Wappler is reading this - because its able to show the _source fields.
Using in repeat PDTSearchMain.data.hits.hits does give you data right?
Its only the _source inside the each row output that you have a problem with here?
Thanks @sid
If I change it to hits[0]._source then I get no field outputs in the repeat.
If I change it back to hits._source - then I get the full list of fields outputs in the repeat.
If i use data.hits.hits - the only outputs I see are at that level, so all I can see is _source for example in the output options…
But, if you try to set any of the expected keys in a set value step, do you see the value in output when running the server action?
Eg: setvalue > name: x1, globalname: x1, value: {{_source.full_name}}
Eg: setvalue > name: x2, globalname: x2, value: {{_source.twitter_url}}
Ok, so we are getting somewhere it seems.
If I leave it as hits.hits and then set value like you suggest with the name and global name hte same, and then manually write {{_source.full_name}} and {{_source.id}} then I’m returning those values.
Progress.!
So, how would you suggest I set a condition up within the repeat, and importantly - show the full schema in each Set Value (x1 showing contact info, x2 not showing contact info).
The minute I change the set value to just _source - then it shows the entire API response again, even if I define the schema of the set value myself. I can’t seeing it being very performant to have to set value for every 26+ data points I want to display?
26+ set values will not be a performance issue from what I understand. The repeats are usually very fast too.
But, it will definitely be a maintainability issue.
I would suggest to use a custom server side formatter here.
Something that would take list of fields to remove from supplied json object.
Thanks @sid.
@patrick any chance you can advise what this would look like as a custom formatter please?
I will take a look at it tomorrow at work, can you tell me what server model you are using (asp/php/node) and what the expected output should be.
I think you are almost there, with the set value steps in the repeater it should work, but can be lot of work if you have a lot of data to output.
You have your api call and query. Repeat PDTSearchMail.data.hits.hits and in the repeat use Set Value steps to output the desired values like {{_source.fullname}}. You can use conditional output in the set value like {{condition ? _source.price : null}}.
Alternative is a custom module/formatter, but for that I need more information about the data and the expected output.
Hey @patrick thank you so much!
Server model is NodeJS.
Our API Call returns over 26 data points, with about 6/7 of these are nested. Some are nested with values and some are nested with objects and values.
We want to control the output schema. One schema will have all data returned to the front end, and the other will have about 5 fields NOT sent to the front end.
We determine which schema should be sent to the front end per record based on a check in ou internal DB.
Each document returned by ES has a unique id, this is source.id . We store the id in our DB.
When a source.id matched the id in our DB we want to show the full schema. If a record does not have a match in the DB we do not want to show the limited schema.
The issue I have found here, is that I need to display the data in a repeat, so I need a parent array to display. If I use the set value steps, then I cannot select them all at once, so I need to chose the core repeat array. BUT, when I use pdtseach.hit.hit - the repeat holds the entire array schema output.
If I use pdtsearch.hit.hit._source then I can choose the output fields, but it shows the aforementioned error.
I was really optimistic with this, it seems to be the perfect solution. But if I use 26 SetValue’s for the data, then that doesn’t work in my repeat group as I need to select the one array as the expression for the repeat on the FE. Then the Repeat in SC actually holds the full schema. This would work perfectly if I could choose what fields the repeat will output and then use a condition and set the values of the information that should be shown in the true condition. But it’s not quite there due to not being able to choose what appears in the repeat.
I have sent you two messages, one with the preview response (how it is returned to us by ElasticSearch) and also the full JSON response of the actual data itself. I sent it in a private message as there is come confidential data within the schema.
Essentially, if you look at the data, IF a _source.id from the ES response matches the internal DB query then we send the full schema to the FE (ideally we can still choose which fields, to save resources).
IF a _source.id does NOT match the internal DB query then we send the full schema EXCLUDING: work_email
phone_numbers (array).
emails (array)
mobile_phone
profiles
linkedin_url
twitter_url
github_url
facebook_url
Noting that we are displaying at least 15 records per page, so some records will match and some won’t - so we have to complete these actions per record.
What then happens, is the user can choose to reveal contact information. This then creates a new DB entry, and we re-run SC to go back through the process.
I have this all working with a FE condition, but the confidential info still appears, so we need to handle and process the data on the backend.
This is literally 2nd last task before we complete this complex project, and we can go to production in 1 to 2 weeks from completing this, so I REALLY REALLY appreciate your help on this @patrick
This is where I got to with sid’s advice, but came across the repeat issue which might stop this from working in it’s current form due to the above mentioned reasons.
This is all very useful… I think I’m having some similar issues with using nested repeats and trying to access nested arrays returned from an API.
@patrick if you’re still looking to this i had some info here and am still unable to properly access the value v which is nested in two arrays as per the below post.
