Is it possible to pass a full JSON query (with data bindings within) from a server action form input to the Server Action for use in API Action call?

Wappler General How To
#1

Hi all,

Working on our ElasticSearch query a little more this week. We’ve been playing with custom JS but it just feels like there should be a way to do this within Wappler UI.

Based on some advice in the forum, I’ve put all 29 input fields into a single form - that I could then trigger based on value changes.

The Server Action itself is an API Call that must be made with JSON.

So what I am attempting to do, is for each input it would have (this example for searching a job title):
{
“query”: {
“match_phrase”: {
“primary.job.title.name”: “Software Engineer”
}
}
}

The ‘Current Title’ input would pass this JSON and the “Software Engineer” is the data binding coming from the form input. We’d need to be able to add multiple, so “Software Engineer”, “Software Developer”, “Developer” etc.

In the Server Action side, I’d then think it might be possble to construct the full JSON call with a number of listed {{$_GET}} variables each tied to the form input.

So in theory, if the form input is empty, nothing gets constructed in the JSON before being sent to the API, but as that input has values - this then gets added to the full JSON query.

The big question is - is this even possible? Can I pass a JSON ‘section’ (probably wrong terminology) via a form input somehow to the Server Action to then be used in the API Call?

Alternatively, I could potentially construct each JSON section (aligned to each form input query) in the Query Parameters - and add the data binding in the value part there?

Ideally I don’t try this for days if it’s not possible within the existing Wappler UI?

Expert needed ASAP - convert form inputs to dynamic JSON output (not static template) for API query
#2

You could generate json using the querystring by using names like ?query[match_phrase][primary.job.title.name]=Software%20Engeneer

1 Like
#3

Hmm, how do you think I would make this dynamic? (add / remove as the user interacts with filters)

For example, I have to send a JSON query to the ES API Endpoint.

Our current implementation, by default sends a basic query with some ordering built in to server recent results as soon as the page is opened.

Then, as each field is used - we either add a new section of JSON to the query, or if a field is removed, then we remove that section of JSON.

I feel confident that Wappler should be able to handle such, but we are getting a bit stuck with how to get the filters/inputs to add/subtract the section of JSON to the JSON output to the API endpoint.

#4

You can use these naming in a serverconnect form, it also works with post. By disabling or hiding inputs it will be removed from the form post, that way you can make it dynamic.

Usage is like https://www.w3.org/TR/html-json-forms/ but without using the enctype='application/json'.

There are more ways of doing it, but using a form is probably the easiest way within Wappler.

#5

Thanks for sharing that link Patrick, this seems interesting for some additional functionality, e.g. passing multiple values from a single form field with an array - so I appreciate this link.

However, in the case of hidden/disabled fields, that’s not an option as all fields need to remain active.

The issue I have is ES is failing if I use a $POST field as a data binding in the JSON output - we don’t have null_value as an option in the ES index, so ideally I can find a way to add / subtract entire sections of JSON output

#6

@patrick - spent time looking into this, and what I have concluded:

  1. I currently use a JSON Data Type, so I’m outputting JSON, rather than building the query with inputs or query parameters.
  2. If I use you propsed method, I would infact be using inputs - and constructing the $POST in a similar structure to the JSON output e.g as highlighted here by @George:
    image
    image733×769 40.8 KB

This is an interesting implementation.

Two issues I think I’ll come across I’d appreciate your input, and a potential bug I might have identified:

  1. The full JSON, has each section starting with the same object which is
    {
              "bool": {

So if I put together two queries in the same JSON output (noting I can have 29+ queries in the same output to ES API:

 {
              "bool": {
                "should": [
                  {
                    "bool": {
                      "must": [
                        {
                          "match_phrase": {
                            "primary.job.title.name": {
                              "query": "engineer"
                            }
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "should": [
                  {
                    "bool": {
                      "must": [
                        {
                          "match_phrase": {
                            "gender": {
                              "query": "male"
                            }
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            }
          ]
        }
      },

So, this poses a potential issue, I’d then have to name each input starting with bool as the starting object. This I imagine, is not a possible implementation due to conflicts?

  1. I still end up with the same problem of empty values, or in this case, empty JSON sections being sent through the API to ElasticSearch. IF, a form input is empty will the JSON output of that input still be sent via the SC Form and thus through to the API? Or will nothing be sent (and thus potentially solving my problem?).

I don’t have the option to hide / disable the inputs as a way of not passing the data through, as all fields need to remain active and useable.

I’d really appreciate some suggestions here - I’m so close to a full ElasticSearch implementation querying over 400 million documents in our dataset.

Potential bug:
I implemented the same naming structure as you propsed, and also the one George used in this pos here.
When I go to global on the SC side to pull in the inputs, the input with this naming setup does not pull through at all, not even incorrectly - it just doesn’t recognize it.

#7
#8

These bool objects are probably within an array, naming would then be query[][bool][should].... Notice the extra [], this will make an array from query and output will look like:

{
  "query": [
    {
      "bool": {
        "should": ...
      }
    },
    {
      "bool": {
        "should": ...
      }
    },
    ...
  ]
}

If the inputs doesn’t do the work for you then you could write your own custom action or formatter to generate the needed json structure. I can help you with creating them.

#9

Great, thank you @patrick let me see if I can get this working with your guidance here before asking for additional custom help.

I already see the issue with conflicts, however - I named two fields like this, both with different ending parameters - and the global either only pulls in one, or it pulls in one with a wierd ‘mulitple’ conflict on the final nest.

How would you recommend I work around that considering all fields will start with query.

#10

Hey @patrick - I spoke with ElasticSearch and they highly recommend creating the JSON query dynamically, and not using a static template.

So this would mean, if using the form inputs as suggested:

I can construct the JSON query as you propose, using the links provided and your guidance.

I have two issues, I’d appreciate some additional guidance on - then I think I can complete this part of the project.

  1. How do I get around conflicts. If each form input starts with either Query or Bool - then in my Global $POST parameters, each nested structure will start the same - I can imagine lots of conflicts and issues here.
  2. How within Wappler UI - would I ONLY output the nested structure of a form input when it’s used. E.g. if the end user only uses two inputs, then the JSON output would only include two query parameters in the JSON structure. If they use 20, then it’ll create a long JSON output of 20 nested queries.

These are the two final parts so really appreciate any assistance here.

Matt

#11

Depending on how you name it and use the brackets in the name correctly it will automatically create arrays or object from them. Having multiple fields with a name like query[] will create an array with all the values of each field. data[prop]=123 will create an object {data:{prop: 123 }}. You can create complex structured data this way.

Do you want to know how to create the form and make it dynamically, so that you can add/remove inputs to the form?

#12

Hey @patrick

Thank you - I think i understand this part, the issue is - it’s failing when I use multiple form fields with the same starting structure, it just combines them all in the $POST section, it doesn’t separate each field out.

No, so I have 29 inputs in one form.

A user might use 1, 5 or all 29.

The JSON output needs to be dynamic, so if they use 5 inputs, then we only add the 5 Query sections to the JSON output, if they use 1, then we only add 1, if they use 29 then we add all 29.

What ElasticSearch advised against was using a statis JSON template, so I need to figure out how to create the JSON output dynamically based on the inputs ‘used’. Can you help with this please?

#13

Friendly bump @patrick - really need this working, and not sure how I can do it with existing Wappler UI.

#14

Hi @Teodor perhaps you would be able to help instead?

To re-summarize:

I have 29+ inputs, that the user adds values to - I then use these inputs as part of a JSON query to our ElasticSearch API Call.

I have this working on one field input right now, just to get all the output data bindings working. I now need to create the dynamic JSON output.

I’ve been advised against using a static JSON template.

So what I need to create, and I am REALLY trying to do this purely in Wappler native and without any custom coding.

If a user uses 1 input then the JSON output would have the output of the one input in the correct JSON structure, if they use 5, then it will have 5 structured JSON outputs in the JSON file, if they use all 29 then it will have 29 and so on.

Patricks suggestion results in duplication/conflict issue.

I’m a bit at a loss as to how to do this natively in Wappler, so would appreciate some expert help!

Matt

#15

Hey @patrick just raising this to the top of your inbox again!

I’ve tried all day on this, and I believe the likely solution is perhaps an initial repeat on $POST inputs and then in using set values. I was able to construct the correct schema of a single query input with a set value, and then use this attribute as a value in the normal input fields section of the API connect, setting the header to application/json.

But I’m still struggling with how I would dynamically create the full JSON output based the various inputs a user chooses to use including if multiple fields are used. In some cases the end user might use 1 field, in others 5, on others 29 - so the JSON output needs to be able to build to suit which inputs are being used, even if the input has multiple values.

To me anyway, it’s seeming super challenging to achieve this in Wappler, but critical to our project and now very time sensistive so I would be really appreciative if you can help with this final piece of the puzzle for us. It doesn’t ‘have’ to be via a form - however this is how I currently have it working, open to any high-performance solutions to generate this output file.

#16

Could you send me the page so that I can have a look at what you currently have and give you further advice.

#17

Hi @mgaussie,
What you could do is have a ‘visible’ form that the user uses to change the search options. Each field would have a corresponding field in a ‘hidden’ form (you could either use hidden fields or set the whole form to ‘hide’)
Each of the hidden fields would be given a dynamic value of the corresponding visible input but also a dynamic attribute for disabled corresponding to ! the same, visible input value (e.g. !input1.value). That way, when the hidden form is submitted, only the completed inputs will be sent.
If you use naming, as suggested above, you could make the form submit in the exact json structure you want to send to ES.

2 Likes
Expert needed ASAP - convert form inputs to dynamic JSON output (not static template) for API query
#18

Hi @patrick - how would you like me to send the page? Just the full code copy / paste?

#19

You can send me the file in a private message, just drag and drop the file in the text editor to add it as attachment.

1 Like
#20

Thank you Ben, I think this will work this way!

I’m still having some issues with the schema itself in NodeJS outputting with the form inputs, but hopefully @patrick can help with that part today, and then I can fully test out your suggestion!

1 Like