Best way to bypass API limitations

[psweb]

I seem to run into this issue all the time with so many APIs, and would love some feedback if there is a “Best Way” to do this.

Many APIs only allow 100 results at a time and to get to the next 100 results I have to use a query parameter to offset to the next set of results, however there are situations where I need to return a full list of all 350 results in a single output.

I have found many ways to return all 350 results by either:
Using While Loops and set value steps to write each 100 results to a temporary database table, and then query the full database table created.
Or if I do not want to use a temporary database I can also use a custom merge array server connect module based on what Sid created a while ago.

When Wappler 4.9.0 released I saw the new Group and Parallel server actions though and was wondering if now there is a better way to do this.
If anyone has figured this out please could you share your idea.

[Teodor]

With the parallel action you just can run several api calls at the same time without waiting the previous one to finish. Not sure if that helps in your case, where you need to repeat the data X times, put offset, call the action again to get the new data with the new offset set.

[psweb]

The parallel action helps only because it puts all the output in a single array, but then inside that array are many others, so kind of wondering if i can then flatten them down.

I suppose it also depends on how I go about “future proofing” the system I am creating, I mean right now there are only 350 results, but in a matter of time there could be 450, in which case manually adding all the API calls would be a bad idea just to avoid the temporary database table.
I think for now I will stick to the while loop and temp table solution because its dynamic and future proof, and maybe at some point Wappler may come up with a data transform for merge array that could do this easier.

[psweb]

Best implementation I have come up with so far as it does not require the temporary database table, however it is a bit slow as the API calls can not run in parallel, and you need the custom Merge Array module added to Wappler.

The test API Provider is a bit of a horrible one because it does not give you all the information some API’s do, such as how many records there are in total, and it also does not give a link to a next or previous page as many other API Providers do.

Steps

1. API Action to fetch the first batch of records needed, in my case API Providers limit is 100 results, and my offset is 0 to start at the first record.
   – ID: api1

2. SetValue — Name: firstRecords, Global Name: firstRecords, Value: api.data.products (This is an Array)

3. SetValue — Name: lastRun, Global Name: lastRun, Value: 1

4. While Loop – Condition: lastRun==1
   – API Action to fetch second batch of records, ID: api2, offset: {{firstRecords.count()}}
   – SetValue — Name: set_firstRecords, Global Name: firstRecords, Value: {{firstRecords.merge_array(api2.data.products)}}
   – SetValue — Name: set_lastRun, Global Name: lastRun, Value: {{(api2.data.products.count()==100) ? 1 : 0}}

5. Out of While Loop
   – SetValue — Name: 'products_holder, Global Name: NOT SET, Value: {{firstRecords}}`

On run I get a single array with 350 results, which means the API is being called 4 times in total, making the results take about 6-8 seconds.

@Teodor is there any way you can think of to improve this?

[Teodor]

That’s already possible using the Server Connect Array List component:

Working with Server Connect Array Lists

Intro The Server Connect Array List component allows you create and manipulate Array Lists easily with Server Connect actions. You can add items to an array, join multiple sources, remove and set arbitrary items, keep simple or complex objects or collections in an Array List. It’s also easy to use it as a string builder, when you need multi lines strings. Basics In order to be able to add items to an array you first need to define the array. The Array List actions are located in the Data Man…