Advice on Scheduled Server Actions

Here are the need to knows.

  1. I have an API I am pulling data from as regularly as possible because it changes constantly every day.
  2. I have to query at least a years worth of data as it is for holiday availability, and people book holidays even 2 years in advance.
  3. Some holiday products are way more active than others

So knowing the above facts, I will split up the timing results of how long the script takes per option.

Fetching a Full year of All vs Active Data
a. 7m 44s - 1 Year of All Data (188 Products)
b. 5m 39s - 1 Year of Active Data (50 Products)

Fetching 1 Month of All vs Active Data
c. 2m 24s - 1 Month of All Data (188 Products)
d. 1m 08s - 1 Month of Active Data (50 Products)

So my question is I would like to use a combination of schedules, every 5 minutes it runs ( c ), and every 4 hours it runs ( a )

My concern is overlap of times, I can not have script ( a ) and ( c ) running at the same time, so is there a way to only run a script if the other one has completed.

I hope this makes sense.

Here’s how I manage it.

  1. Create a DB table for sync status. Whichever job starts first, will make an entry saying its processing.
  2. Whichever job comes next, need to check in the DB - if there is something processing already, reutrn with a response step.
  3. You can reduce the scheduled time of the different jobs, so there is less buffer waiting period between the two.
  4. Also, if a job is called again before the previous run completes, it will not do anything because of the processing condition in place.

You might have to keep an eye on the table to ensure there is no stuck processing entry by also recording start/end/processing times & many other fields that you want to log to track it.

1 Like

Thanks Sid, what a brilliant brilliant plan, love it, that way I can have almost no wait times between calls, shame, this poor API provider is going to moan at me when they see activity 24 hours a day, lol.

2 Likes

You already got the scheduling problem answer.

You could also check with @mebeingken about this as he implemented Bull for queue handling.

Additionally I am leaving this library here because it might help you with the overall process.

I am assuming the response is huge given the times you are giving and you will have to compare for differences.

This library might be helpful to return the diff between two JSONs.

1 Like

Thanks Jon, looks pretty interesting, you do know if it includes installing modules or git repository stuff you best keep on eye on those PMs from me, that whole side still confuses me to distraction.

1 Like

Just as a side note, I work with some API providers that have parts of the JSON response that really are useful, like some show how many records there are in total even if they limit me to 100 results per query, so knowing how many there are in total helps creating accurate repeats, some show if i am on the first page of results or the last page of results, and I have found a few that even have a separate endpoint for only results which have been modified in a certain time period.

All these extra bits of data really help them reduce the load on their servers as people only need to get what they need instead of having to pull 8000 records when only 2 have actually altered.

I really just wish API providers would have some standard they have to meet so we can have a bit of an easier time when linking so many APIs together all day long.

Currently the project I am working on
3 different Activity APIs
1 Accounting API
1 GPS API so I can know where each person is and each boat, car etc.
1 People Management Schedule calendar API
1 resource API holding all data on things like Life Jackets, all the way to company stationary.
a few APIs for accommodation availability

So when a person books a holiday for a week, with 2 hotels, a horse ride, and a ski trip included
It checks all the availability, books it, sends the payment through, reserves a boat, makes sure there are enough life jackets, and flare guns, and oars, and cameras, makes sure there are enough beers on board, tells the hotel Joe Soap is coming, pays the hotel, tells the skipper Joe Soap is coming, makes sure the skipper and boat are geographically in the right location for the booking, then creates an invoice for the entire thing.

The moral of the story is, its quite common for me to have to link data from a few APIs together and they all work differently, it drives me slightly nutty.

1 Like

makes sure there are enough beers on board

Business critical.

4 Likes

Yup, next plan Jon, link Facebook profile, so we can see if Teo is on the boat, triple the beer stock.

2 Likes

I am the worst customer when it comes to supplying enough beer :rofl:
All inclusive resorts don’t accept me as a guest hahaha

3 Likes

Lol, Lol, Lol, you need to go to lots of weddings, beer is normally free, just hard liquor is to pay for, you could be happy for free every weekend.

2 Likes

There is an all-you-can eat chain here that includes beer.

And its not a bad brand. It could be considered like a Heineken equivalent here.

It is full always. Crazy people.

2 Likes

We need an all you can drink place :rofl:

2 Likes