Complex Server Connect - help! - Process data on backend to only send certain data to repeat items on FE

[mgaussie]

Hey all,

So as we get closer and closer to production, I’m replicating one critical item from our existing non Wappler project.

We call a third party API to return nested JSON schema. Within this schema there is confidential information I want to process on the backend, not the front end - and only send it to rows in a repeat that it should appear.

Currently, this is all handled on the backend. We make one call to the API, check an internal DB of which record should, and should not display certain data - and then only send the data to those records, without needing to make a second API call.

So our current system essentially hides/sends data based on this.

I’m trying to figure out how I would do this in Wappler. I thought a set value could work, but regardless of the schema I build within the Set Value, it displays everything.

Does anyone have any ideas how / what the best approach would be to handle the sending of certain data to the FE based on a db query?

@patrick would really be grateful for some expertise here in regards to this, and how to achieve it in Wappler. Do you think we’d need another custom module to handle this in the Server Connect?

[sid]

@mgaussie

thought a set value could work, but regardless of the schema I build within the Set Value, it displays everything.

Repeat the data from API.
Do not select any columns in the repeat step. Just enable output.
Check condition inside repeat, if row needs to be shown, use set value.
If row should not be shown, make another set value with same names and set it as null.

Outside of repeat, filter the response where repeat’s set value is not null.

Combine API data with Database data

That is correct. Repeat creates an entry for every iteration. Best solution here would be from the API end. If you can filter the result via API request, your repeat would not require any changes. If not, here’s what I have done at one place where I had a similar requirement. Inside the repeat, filter the single query. For unmatched category groups, you will see null entries in the report output. Next, remove output from repeat. Just repeat. And create a separate set value step with output …

[mgaussie]

Thanks @sid - this looks promising.

I do have an issue however, that all rows must be displayed, it’s just certain data within the response that must not.

E.g. lets say we have 1000 responses with car information, that we want to display all 1000 to a user. But some cars we don’t want to display a price from the response data, and some we do.

That’s what I’m struggling with - displaying all results from the search, but do not send certain information to the front end from each response…

[sid]

The logic becomes easier for you in that case.
Select all common fields in the repeat step itself.
For car price, use set value conditional value logic explained above.

No need for post repeat filter also.

[mgaussie]

hmm, I don’t seem to have the luxury of choosing all repeat columns, it’s quite nested JSON and it’s not appearing as an option?

[mgaussie]

@sid and is this what you are advising for within the single repeat?

Workflow:
API Call (this returns full schema). Output OFF.
Internal DB Query (checks API Call ID against internal db ID’s)
Repeat API Call Data. Output ON.

• Condition Inside Repeat. Does Internal DB Query match API Call ID.
• Yes - Show Schema /w price
• No: Show schema /wout price

Like this?

[sid]

Pretty much.
But set value have to be same names.
With output on.

If api does not give you the list of columns in repeat step, its a bug. Have encountered before. @patrick
As a workround you can try to type those out in the json itself.

[mgaussie]

@sid

As a workround you can try to type those out in the json itself.

if I open up the server connect in editor?

@sid

Pretty much.
But set value have to be same names.
With output on.

I’ve tried this, but i’m only returning 3 results, from the 15 that should display. Can you advise what condition you’d use in the repeat?

Really appreciate your help here - i’m feeling optimistic

[mgaussie]

@sid I thought I’d cracked it, I manually wrote in the final object in the API response (_source) - I suddenly had all the available fields to choose from in the repeat.

However, it errored when running the query telling me the repeat does not have array or object data.

Then I tried as you mention simply editing the JSON schema, with one nesting back (which avoided the above error).

I can only return the next object, which then returns ALL data. If I manually try to add in nested json in the output fields then it doesn’t return any data…

as I have _source - it returns the entire response. if I change _source to say _source.id or _source.first_name then I get no response data.

Changing to:

Then gives me all fields:

But I get a console error, repeat has no array or object, which appears to be a bug?

Any ideas?

[sid]

Not sure how well Wappler can handle nested data.
Can you check if all _source keys have an array value? A null or non-array in between could be the cause of error on runtime.

[mgaussie]

@sid

Can you check if all _source keys have an array value? A null or non-array in between could be the cause of error on runtime.

How would I best define this?

Here is a screen of the response in network tab:

if you are referring ‘keys’ as the values within the _source object, then yep - lots do have null values, and this varies between record.

[sid]

By key I mean _source itself.
So the last hits object is an array. It has the key _source.
Can you check if the value for _source is null in any of the rows in hits array?

[mgaussie]

none of the array items within hits are empty. These are keyed 0 to 25 and all hold data:

hits itself (one nest above _source) has no null values:

When I open _source this does indeed have many null values.

[sid]

Ok. That looks fine.
Can you please also check if any of the _source keys in the 26 items inside hits are null?
Not values inside _source, but _source itself.

[mgaussie]

Thanks @sid. Just rechecked. _source itself never returns a null value. Within _source there are as always a large number of data points, and a few arrays so it always has something.

[sid]

Ok this is good.
I rechecked the code you posted earlier… and there seems to be an issue there.
PDTSearchMain.data.hits.hits._source does not look right to be in a repeat.
It should be something like PDTSearchMain.data.hits.hits[0]._source, since second hits itself is an array.
No sure how Wappler is reading this - because its able to show the _source fields.

Using in repeat PDTSearchMain.data.hits.hits does give you data right?
Its only the _source inside the each row output that you have a problem with here?

[mgaussie]

Thanks @sid

If I change it to hits[0]._source then I get no field outputs in the repeat.

If I change it back to hits._source - then I get the full list of fields outputs in the repeat.

If i use data.hits.hits - the only outputs I see are at that level, so all I can see is _source for example in the output options…

[sid]

@mgaussie

If i use data.hits.hits - the only outputs I see are at that level, so all I can see is _source for example in the output options…

But, if you try to set any of the expected keys in a set value step, do you see the value in output when running the server action?
Eg: setvalue > name: x1, globalname: x1, value: {{_source.full_name}}
Eg: setvalue > name: x2, globalname: x2, value: {{_source.twitter_url}}

[mgaussie]

Ok, so we are getting somewhere it seems.

If I leave it as hits.hits and then set value like you suggest with the name and global name hte same, and then manually write {{_source.full_name}} and {{_source.id}} then I’m returning those values.

Progress.!

So, how would you suggest I set a condition up within the repeat, and importantly - show the full schema in each Set Value (x1 showing contact info, x2 not showing contact info).

[mgaussie]

The minute I change the set value to just _source - then it shows the entire API response again, even if I define the schema of the set value myself. I can’t seeing it being very performant to have to set value for every 26+ data points I want to display?