JavaScript heap out of memory questions

Wappler General
1

I keep getting these JavaScript heap out of memory errors.

In many scripts I have been able to get rid of them by turning off the output for all the steps, or at least the big ones, and have been able to add large amounts of data to my database using bulk inserts.

My confusion is, I have a script that needs to insert about 600k records, over multiple relational tables.
I have turned off all outputs, although the Create Array List, Add in Array List, and Get Array List Value steps seem to cause major output in my Web Server Logs regardless.

When the script runs, it manages to insert 340k records in about an hour, and then the script stops and returns the JavaScript heap out of memory error.
I added in a way for the script to continue from where it last left off, however the second time I run the script, it runs out of memory after only 25k inserts, i run it again and manage another 10k.
My question is, why would if do so many in the first run, and not in the second or third runs.
Does the Javascript heap not reset or something.

Secondly is there a way to turn off the output for the Array steps so they do not fill up the web server logs? I did read somewhere that Array, Function, an Object are all stored in the heap, while Number, String, Bool are all stored in the stack, so possibly no way around that part.

Still confusing though how with really similar data, run 1 can do 340k and subsequent runs only 10k to 25k before the memory is depleted.

Not sure if it helps at all but here is a screenshot of my server action.

2024-09-09_10-08-03
2024-09-09_10-08-031592×2098 198 KB

The logs

2024-09-09_10-27-11
2024-09-09_10-27-111920×1614 242 KB

2

Follow up question, Would having debug mode ON or OFF impact this at all?

3

Looks like turning off Debug Mode was a winner, my next run is at 50k records inserted so far, best second run i have had yet, doubled my last best.

4

It seems you're on the road of becoming a mini Patrick :guide_dog:

The JavaScript interpreter has a garbage collection mechanism, it deletes data that's no longer needed from memory. If you don't give it enough time, the data isn't deleted from memory yet. This is different from other certain programming languages (such as C) where the programmer has control of when to delete the data, there's no garbage collection and the developer is responsible for explicitly deleting the data from memory.

Wappler also has its own significant overhead, Wappler steps don't run as fast and as memory-efficient as raw JavaScript, though such isn't noticeable to 99% of Wapplers.

Wappler features such as debug mode might have influence on the performance and memory usage due to additional overhead of printing to console (and what comes prior).

2 Likes
5

Lol, I think if my brain became anything like Patricks, my head would run out of stack and heap space.

Thanks Apple for the extra info, it all helps me figure out whats going on.

1 Like
6

Hi Paul,
Not really an answer to your question, but might it be possible to import in a different way? I don't know what database you're using, so it may not be available, but at least with MySQL, you can import 10s of thousands of rows per second using LOAD DATA INFILE. There was a discussion about it here.

1 Like
7

By default I believe 1gb of heap space is allocated on 64 bit systems.

Have you looked at increasing that?

Not sure of syntax of terminal command but it is something like:

node --max-old-space-size=8192 index.js

where 8192 would allocate 8gb

1 Like
8

It seems, you're looping over the whole file and just split up 10k lines of it.
Did i saw that correctly? :sweat_smile:
I'm not 100% sure what wappler does on a SC under the hood, but if you are just chunking the main records into 10k sublists and process those, then the garbage collector of node does nearly nothing.
The already done chunks are not freed up, because inside of the function (your whole SC) it creates a memory-scope and this scope stays open as long as the function runs, even if they are not actively used anymore.
if you want to work like this, it would be better to split your initial files into 10k chunk files and read them seperated from each other. This way the memory gets cleared after running one file.

As a workaround if you don't run the function that often. Do you run your node server on local docker dev or anywhere else? you could check the assigned memory limit to the nodejs container. Doesn't solve the problem but could work :wink:

1 Like
9

It a strange one to explain, but basically i have a read table and a write table.

I read the last record from the read table, and get a single number returned like 3 million
I then read the last record that was written from the write table, if it has never run it will return 0, and if it is partly complete it will return say 340k

For this explanation pretend i have a start position of 0 and an end position of 3 million, and I am splitting everything into 10k chunks to work with.

I take the 3 million and divide by 10k, and hand that to the repeater, so i know i need to repeat 10k 300 times.
in the repeat, i write a custom query with LIMIT :P1, 10000
As the parameter I use {{($index * 10000)}}

So on repeat 1 it would be LIMIT 0, 10000 and on repeat 2 it would be LIMIT 10000, 10000 and so on.

I then stick it all in a transaction, so i know all tables in the write set have been written to completely, before it moves onto my second repeat. I am kind of hoping that on repeat 2 it clears everything in its garbage collector before starting fresh with repeat 2.

Unfortunately, once a week I will have to take 3 files, 1 x 3 milliion record file, and one 11 million record file, and 1 x 1 million record file, and overwrite my database files to get all the hotels under the supported API as well as all the region data and review data.

The JSONL they hand me is pretty intense too, and requires quite a job to get to everything, here is a sample of a single line, and I have 3 million more just like it.

{
  "address": "Al Marsa Street, Dubai",
  "amenity_groups": [
    {
      "amenities": [
        "Air conditioning",
        "24-hour reception",
        "Elevator/lift",
        "Currency exchange",
        "Smoke-free property",
        "Newspapers",
        "Ticket assistance",
        "Washing machine",
        "Garden",
        "Terrace",
        "Express check-out",
        "All Spaces Non-Smoking (public and private)",
        "Electric car charging"
      ],
      "group_name": "General"
    },
    {
      "amenities": [
        "Non-smoking rooms",
        "Soundproof rooms",
        "Fridge",
        "Family room",
        "Cable TV",
        "TV",
        "Minibar",
        "DVD Player",
        "Hairdryer",
        "Shower/Bathtub",
        "Bathtub",
        "Shower",
        "Wardrobe/Closet",
        "Toiletries"
      ],
      "group_name": "Rooms"
    },
    {
      "amenities": [
        "Accessibility features"
      ],
      "group_name": "Accessibility"
    },
    {
      "amenities": [
        "Iron and board",
        "Ironing",
        "Luggage storage",
        "Laundry",
        "Safe-deposit box",
        "Concierge services",
        "Dry-cleaning",
        "Shoe shine",
        "Luggage storage",
        "Wake-up service",
        "Telephone",
        "Iron"
      ],
      "group_name": "Services and amenities"
    },
    {
      "amenities": [
        "Coffee/tea for guests",
        "Breakfast",
        "Cafe",
        "Restaurant",
        "Breakfast/lunch to go",
        "Microwave oven",
        "Kettle"
      ],
      "group_name": "Meals"
    },
    {
      "amenities": [
        "Free Wi-Fi"
      ],
      "group_name": "Internet"
    },
    {
      "amenities": [
        "Shuttle",
        "Car rental",
        "Transfer services"
      ],
      "group_name": "Transfer"
    },
    {
      "amenities": [
        "German",
        "English",
        "French",
        "Multi-language staff",
        "Arabic"
      ],
      "group_name": "Languages Spoken"
    },
    {
      "amenities": [
        "Tour assistance"
      ],
      "group_name": "Tourist services"
    },
    {
      "amenities": [
        "Library",
        "Hiking",
        "Sun Deck"
      ],
      "group_name": "Recreation"
    },
    {
      "amenities": [
        "Parking"
      ],
      "group_name": "Parking"
    },
    {
      "amenities": [
        "Swimming pool",
        "Outdoor pool",
        "Pool facilities",
        "Next to the beach"
      ],
      "group_name": "Pool and beach"
    },
    {
      "amenities": [
        "Conference Hall"
      ],
      "group_name": "Business"
    },
    {
      "amenities": [
        "Fitness facilities"
      ],
      "group_name": "Sports"
    },
    {
      "amenities": [
        "Sauna"
      ],
      "group_name": "Beauty and wellness"
    },
    {
      "amenities": [
        "Babysitting and childcare",
        "Children's playground",
        "Children swimming pool",
        "Family/Kid Friendly"
      ],
      "group_name": "Kids"
    },
    {
      "amenities": [
        "Pets Not Allowed"
      ],
      "group_name": "Pets"
    },
    {
      "amenities": [
        "Temperature control for staff",
        "Personal protection equipment for staff",
        "Extra decontamination measures",
        "Temperature control for guests",
        "Personal protection equipment for guests",
        "Contactless check-in and/or check-out",
        "Individually packaged food",
        "Additional measures against COVID-19"
      ],
      "group_name": "Health and Safety Measures"
    }
  ],
  "check_in_time": "15:00:00",
  "check_out_time": "12:00:00",
  "description_struct": [
    {
      "paragraphs": [
        "<em>The Radisson Blu Residence, Dubai Marina</em> is located in Dubai. This apartment hotel is located in 26 km from the city center. You can take a walk and explore the neighbourhood area of the apartment hotel. Places nearby: Dubai Marina Beach, Nasimi Beach and Aquaventure."
      ],
      "title": "Location"
    },
    {
      "paragraphs": [
        "You can stop by the bar. Taste the local cuisine and have a rest in the restaurant. If you can’t live without coffee, drop by the cafe. Free Wi-Fi on the territory will help you to stay on-line.",
        "Specially for tourists who travel by car, there’s a parking zone. Also, the following services are available for guests at the apartment hotel: a sauna and a spa center. Sports fans will be able to enjoy a fitness center and a gym. You won’t be bored as at the apartment hotel you will find a library.",
        "Here, you can treat yourself with water procedures as there will be a pool, an indoor pool and an outdoor pool. For business meetings, there is: a business center, a conference hall, event facilities. The tour assistance desk of the apartment hotel will help you book an excursion. The staff of the apartment hotel will order a transfer for you.",
        "Accessibility: there is an elevator/lift. Additional services that the apartament hotel offers to its guests: a laundry, dry cleaning, private check-in and check-out, ironing, press, car rental, a safe-deposit box and a concierge. The staff of the apartment hotel speaks English, German and French."
      ],
      "title": "At the apartament hotel"
    },
    {
      "paragraphs": [
        "Here’s what you’ll find in the room to have a rest after a long day: a DVD player, an alarm clock, a shower, a TV, a mini-bar, a bathrobe and slippers. The room equipment depends on its category."
      ],
      "title": "Room amenities"
    }
  ],
  "id": "radisson_blu_residence_dubai_marina",
  "hid": 7499177,
  "images": [
    "https://cdn.worldota.net/t/{size}/content/cf/78/cf786d0b39bb5cdf56ff16815fc3ce55d26f071e.jpeg",
    "https://cdn.worldota.net/t/{size}/content/4f/9d/4f9d943175458b01513847294d5ae1b7e8a75030.jpeg",
    "https://cdn.worldota.net/t/{size}/content/51/83/5183b301d39c594a62e0529d0a11ef48330efa7a.jpeg",
    "https://cdn.worldota.net/t/{size}/content/eb/c8/ebc88f4ac338b8b41628df3f304ca07b6203d7ce.jpeg",
    "https://cdn.worldota.net/t/{size}/content/c1/71/c1710f7192b3135ffbf9ad9a5b1b7c9d860a3b34.jpeg",
    "https://cdn.worldota.net/t/{size}/content/5a/5e/5a5eeb274805682fdbc0c9a5d7954fde3dfff8b2.jpeg",
    "https://cdn.worldota.net/t/{size}/content/73/78/7378394ebb6e18f33a165d1e3056b4da3c40659c.jpeg",
    "https://cdn.worldota.net/t/{size}/content/03/d0/03d03e17c5a8e933eda0d8b20447756fddb4bf43.jpeg",
    "https://cdn.worldota.net/t/{size}/content/06/9d/069db88f82e327fe917d34860e68b6a91dd83bc8.jpeg",
    "https://cdn.worldota.net/t/{size}/content/0c/38/0c381c77713a7fd31fc6d592a6ccee1729e166c5.jpeg",
    "https://cdn.worldota.net/t/{size}/content/ec/ff/ecffc4f5e5af8741a1e0b4b7f2ecb0bee7b1d05a.jpeg",
    "https://cdn.worldota.net/t/{size}/content/b7/b9/b7b942ca6eef2f61224b6130476c7b22499f650f.jpeg",
    "https://cdn.worldota.net/t/{size}/content/3b/0d/3b0def94f5e0416ae7dd993ed8af89f8fdde40ab.jpeg",
    "https://cdn.worldota.net/t/{size}/content/02/ff/02ffa50a2f6a96eebbb3e9964349492b016a030c.jpeg",
    "https://cdn.worldota.net/t/{size}/content/e4/b1/e4b1c65030739d1eb84a3da85f11198ed497b650.jpeg",
    "https://cdn.worldota.net/t/{size}/content/4f/3a/4f3a20ed4c6bd492ae09ea609d2daf222dc2cbda.jpeg",
    "https://cdn.worldota.net/t/{size}/content/3f/51/3f516cce795046efb884c9b1338391452c459b4f.jpeg",
    "https://cdn.worldota.net/t/{size}/content/8c/e8/8ce8e59eca69b497da94dc7e99d17d4fef9e13d8.jpeg",
    "https://cdn.worldota.net/t/{size}/content/aa/e7/aae70714ceb3dd63a6773ad69271ab1ab1c057a1.jpeg",
    "https://cdn.worldota.net/t/{size}/content/ef/d2/efd26ea3343a929c2d233908ab7f00017344f7bc.jpeg",
    "https://cdn.worldota.net/t/{size}/content/a2/0a/a20a0f71ea081dd06b7f47010fa3cf3f74e867f7.jpeg",
    "https://cdn.worldota.net/t/{size}/content/6a/4a/6a4ae3676e15120e73841b59ca5d08f85e31a656.jpeg",
    "https://cdn.worldota.net/t/{size}/content/e0/01/e0012a59bc2a485725fdabae16125b2e6ac74146.jpeg",
    "https://cdn.worldota.net/t/{size}/content/05/9d/059d616cccbfbb9d81640b5f7bc9ac3578ced598.jpeg",
    "https://cdn.worldota.net/t/{size}/content/2c/a0/2ca06e2908898328f5bcc6d904b5b8cf4c89a4ea.jpeg",
    "https://cdn.worldota.net/t/{size}/content/c4/a7/c4a7ef1cc34c2992d3162a8a67a9a840c88ae32c.jpeg",
    "https://cdn.worldota.net/t/{size}/content/20/4e/204e75c3b7349f76957100e9f9cb3be43c0af200.jpeg",
    "https://cdn.worldota.net/t/{size}/content/1f/f6/1ff6bb77c96ad43c6cefd3b23a17e1c25102aafd.jpeg",
    "https://cdn.worldota.net/t/{size}/content/05/72/0572d4f3d84e89ae474eca6b11b8af61bcfe3b5d.jpeg",
    "https://cdn.worldota.net/t/{size}/content/8c/fd/8cfdd115f93afaa27640a1dd573750ad4368043b.jpeg",
    "https://cdn.worldota.net/t/{size}/content/a1/13/a1134751e2bdcf0afa581025926cafb4516c80ad.jpeg",
    "https://cdn.worldota.net/t/{size}/content/6c/35/6c3576c56eaba210f26b939508dbaae5a5ebb2fb.jpeg",
    "https://cdn.worldota.net/t/{size}/content/a0/4b/a04ba597a805a50c831b8cffae12050f32179393.jpeg",
    "https://cdn.worldota.net/t/{size}/content/f3/33/f3333441e48502df9a828ded112c338a337c62a7.jpeg",
    "https://cdn.worldota.net/t/{size}/content/5d/09/5d09f9afc887bc979772e206fbb39333bd9a81b1.jpeg",
    "https://cdn.worldota.net/t/{size}/content/ea/4a/ea4a996d9bf98b3adac8b784d24fa2460d8d04c9.jpeg",
    "https://cdn.worldota.net/t/{size}/content/d3/a3/d3a3968e6249e83025d77e2cab10a32395c049c4.jpeg",
    "https://cdn.worldota.net/t/{size}/content/19/38/193810e965f3682f27355d3b9c1d269c2861c44d.jpeg",
    "https://cdn.worldota.net/t/{size}/content/b6/99/b699ce2c41de7ce56b3582091d7ca96e2a7d5243.jpeg",
    "https://cdn.worldota.net/t/{size}/content/98/65/9865623e82bb413aa2c2148351b934a86fe583a2.jpeg",
    "https://cdn.worldota.net/t/{size}/content/d2/f1/d2f140e3308d25ad512c7ab37d5e1169761f5eab.jpeg",
    "https://cdn.worldota.net/t/{size}/content/67/c6/67c63b0a23a2095760586dd3a27bcde73beaddc6.jpeg",
    "https://cdn.worldota.net/t/{size}/content/49/87/4987b81e92fb06c7e2e9d529c17f276e8f0be75d.jpeg",
    "https://cdn.worldota.net/t/{size}/content/32/f1/32f12a36737797e39bee01c633337ed231e38a4e.jpeg",
    "https://cdn.worldota.net/t/{size}/content/be/72/be72612638401d7efaca3e52da57e2bf4281dca3.jpeg",
    "https://cdn.worldota.net/t/{size}/content/a3/fb/a3fb8209ce4bec8b6df3604cd8cc75688d2e3404.jpeg",
    "https://cdn.worldota.net/t/{size}/content/77/59/7759e1622bae50bb814a95ec92b266610332252e.jpeg",
    "https://cdn.worldota.net/t/{size}/content/53/de/53debd0e7a49b3f219ed179083109ef9d3530582.jpeg",
    "https://cdn.worldota.net/t/{size}/content/df/6b/df6b709e64792f1eb90fda2225e76fbca9d45b02.jpeg",
    "https://cdn.worldota.net/t/{size}/content/7e/86/7e86036fbd11104440b6a965e4b484aaf7c6f2fc.jpeg"
  ],
  "kind": "Apart-hotel",
  "latitude": 25.078308,
  "longitude": 55.143173,
  "name": "The Radisson Blu Residence, Dubai Marina",
  "phone": "97144355000",
  "policy_struct": [
    {
      "paragraphs": [
        "Price of an additional breakfast: 75.00 AED per person. Information about the type of meals included in the price is indicated in the rate details.",
        "Meals may be limited during Ramadan."
      ],
      "title": "Meals"
    },
    {
      "paragraphs": [
        "Fee for an extra bed: 125.00 AED per night.",
        "The number of extra beds depends on the room category. You must take a look at the information about the size of the selected room."
      ],
      "title": "Children and information about extra beds"
    },
    {
      "paragraphs": [
        "Transfer from and to airport available. Cost: 400.00 AED both ways."
      ],
      "title": "Transfer"
    },
    {
      "paragraphs": [
        "Please note that a surcharge may apply for UAE residents upon arrival.\r",
        "\r",
        "Mandatory gala dinner fees may apply at UAE hotels. Guests should contact the hotel directly for more information.",
        "For an Apartment with one room the city tax is AED 10.00, with two rooms AED 20.00."
      ],
      "title": "Extra info"
    }
  ],
  "postal_code": "73029",
  "room_groups": [
    {
      "room_group_id": 20591127,
      "images": [
        "https://cdn.worldota.net/t/{size}/content/ab/f7/abf7934e8661340a764b92bdb9cf4caa8978313d.jpeg",
        "https://cdn.worldota.net/t/{size}/content/4d/19/4d196f12983979cbef958f0c3ab2824484fa60fb.jpeg",
        "https://cdn.worldota.net/t/{size}/content/14/13/14131f257fd061278e5117564ec3f7d36afb0d4f.jpeg",
        "https://cdn.worldota.net/t/{size}/content/80/db/80dbce5c16e3ce3a329241a8057a691fb2a4d7f7.jpeg"
      ],
      "name": "1 Bedroom Double Apartment full double bed",
      "room_amenities": [
        "air-conditioning",
        "hairdryer",
        "private-bathroom",
        "tv"
      ],
      "rg_ext": {
        "class": 6,
        "quality": 0,
        "sex": 0,
        "bathroom": 2,
        "bedding": 3,
        "family": 0,
        "capacity": 2,
        "club": 0,
        "bedrooms": 1,
        "balcony": 0,
        "floor": 0,
        "view": 0
      },
      "name_struct": {
        "bathroom": null,
        "bedding_type": "full double bed",
        "main_name": "1 Bedroom Double Apartment"
      }
    },
    {
      "room_group_id": 20591127,
      "images": [],
      "name": "2 Bedrooms Double Apartment full double bed",
      "room_amenities": [
        "private-bathroom"
      ],
      "rg_ext": {
        "class": 6,
        "quality": 0,
        "sex": 0,
        "bathroom": 2,
        "bedding": 3,
        "family": 0,
        "capacity": 2,
        "club": 0,
        "bedrooms": 2,
        "balcony": 0,
        "floor": 0,
        "view": 0
      },
      "name_struct": {
        "bathroom": null,
        "bedding_type": "full double bed",
        "main_name": "2 Bedrooms Double Apartment"
      }
    },
    {
      "room_group_id": 20591207,
      "images": [
        "https://cdn.worldota.net/t/{size}/content/b5/25/b525869de3c3268062f58fad9098c88c72e4ab17.jpeg",
        "https://cdn.worldota.net/t/{size}/content/ab/f7/abf7934e8661340a764b92bdb9cf4caa8978313d.jpeg",
        "https://cdn.worldota.net/t/{size}/content/80/db/80dbce5c16e3ce3a329241a8057a691fb2a4d7f7.jpeg",
        "https://cdn.worldota.net/t/{size}/content/be/21/be2128b55b6d2bfa80cfd812133122d5c8994960.jpeg",
        "https://cdn.worldota.net/t/{size}/content/e3/d7/e3d7d840ae0a91533d8a76ed047bcbfd341306c7.jpeg"
      ],
      "name": "1 Bedroom Superior Double Apartment full double bed",
      "room_amenities": [
        "air-conditioning",
        "hairdryer",
        "private-bathroom",
        "tv"
      ],
      "rg_ext": {
        "class": 6,
        "quality": 5,
        "sex": 0,
        "bathroom": 2,
        "bedding": 3,
        "family": 0,
        "capacity": 2,
        "club": 0,
        "bedrooms": 1,
        "balcony": 0,
        "floor": 0,
        "view": 0
      },
      "name_struct": {
        "bathroom": null,
        "bedding_type": "full double bed",
        "main_name": "1 Bedroom Superior Double Apartment"
      }
    },
    {
      "room_group_id": 20591207,
      "images": [],
      "name": "2 Bedrooms Superior Double Apartment full double bed",
      "room_amenities": [
        "private-bathroom"
      ],
      "rg_ext": {
        "class": 6,
        "quality": 5,
        "sex": 0,
        "bathroom": 2,
        "bedding": 3,
        "family": 0,
        "capacity": 2,
        "club": 0,
        "bedrooms": 2,
        "balcony": 0,
        "floor": 0,
        "view": 0
      },
      "name_struct": {
        "bathroom": null,
        "bedding_type": "full double bed",
        "main_name": "2 Bedrooms Superior Double Apartment"
      }
    },
    {
      "room_group_id": 20591399,
      "images": [],
      "name": "2 Bedrooms Premium Double Apartment with Marina view full double bed",
      "room_amenities": [
        "private-bathroom"
      ],
      "rg_ext": {
        "class": 6,
        "quality": 17,
        "sex": 0,
        "bathroom": 2,
        "bedding": 3,
        "family": 0,
        "capacity": 2,
        "club": 0,
        "bedrooms": 2,
        "balcony": 0,
        "floor": 0,
        "view": 17
      },
      "name_struct": {
        "bathroom": null,
        "bedding_type": "full double bed",
        "main_name": "2 Bedrooms Premium Double Apartment with Marina view"
      }
    },
    {
      "room_group_id": 250,
      "images": [
        "https://cdn.worldota.net/t/{size}/ostrovok/31/2e/312e5d8b720f234a9534fda93b4823ca79fa43b9.jpeg",
        "https://cdn.worldota.net/t/{size}/content/62/dc/62dc7fc98365d52d8c91e8d106ccb46483de49bb.jpeg",
        "https://cdn.worldota.net/t/{size}/ostrovok/df/0f/df0f8a3c1d27e3c21cd08b0ec04c1cfc73bdaeba.png",
        "https://cdn.worldota.net/t/{size}/ostrovok/dd/ce/ddcecfc432db9e0940792b6ec5c8d1bebbb35ff4.jpeg"
      ],
      "name": "Double Studio full double bed",
      "room_amenities": [
        "private-bathroom"
      ],
      "rg_ext": {
        "class": 7,
        "quality": 0,
        "sex": 0,
        "bathroom": 2,
        "bedding": 3,
        "family": 0,
        "capacity": 2,
        "club": 0,
        "bedrooms": 0,
        "balcony": 0,
        "floor": 0,
        "view": 0
      },
      "name_struct": {
        "bathroom": null,
        "bedding_type": "full double bed",
        "main_name": "Double Studio"
      }
    },
    {
      "room_group_id": 20591128,
      "images": [
        "https://cdn.worldota.net/t/{size}/ostrovok/7f/fd/7ffd25b8510077866a9ffab7ce2bd3cf42454874.jpeg",
        "https://cdn.worldota.net/t/{size}/ostrovok/31/2e/312e5d8b720f234a9534fda93b4823ca79fa43b9.jpeg",
        "https://cdn.worldota.net/t/{size}/ostrovok/39/b4/39b4804974b2811183964e80d259cdce8bb0e665.jpeg",
        "https://cdn.worldota.net/t/{size}/ostrovok/dd/ce/ddcecfc432db9e0940792b6ec5c8d1bebbb35ff4.jpeg"
      ],
      "name": "Double Studio with balcony full double bed",
      "room_amenities": [
        "air-conditioning",
        "balcony",
        "bathrobe",
        "desk",
        "dining-area",
        "fireplace",
        "fridge",
        "hairdryer",
        "iron",
        "kitchen",
        "microwave",
        "mini-bar",
        "private-bathroom",
        "safe",
        "slippers",
        "soundproofing",
        "tea",
        "tea-or-coffee",
        "telephone",
        "toiletries",
        "towels",
        "tv",
        "wardrobe",
        "water",
        "wi-fi"
      ],
      "rg_ext": {
        "class": 7,
        "quality": 0,
        "sex": 0,
        "bathroom": 2,
        "bedding": 3,
        "family": 0,
        "capacity": 2,
        "club": 0,
        "bedrooms": 0,
        "balcony": 1,
        "floor": 0,
        "view": 0
      },
      "name_struct": {
        "bathroom": null,
        "bedding_type": "full double bed",
        "main_name": "Double Studio with balcony"
      }
    }
  ],
  "region": {
    "id": 6053839,
    "country_code": "AE",
    "iata": "DXB",
    "name": "Dubai",
    "type": "City"
  },
  "star_rating": 5,
  "email": "info.marina.dubai@radissonblu.com",
  "serp_filters": [
    "has_internet",
    "has_airport_transfer",
    "has_parking",
    "has_kids",
    "has_pool",
    "has_fitness",
    "has_meal",
    "has_disabled_support",
    "has_business",
    "has_spa",
    "beach",
    "air_conditioning",
    "has_ecar_charger"
  ],
  "deleted": false,
  "is_closed": false,
  "is_gender_specification_required": false,
  "metapolicy_struct": {
    "internet": [],
    "meal": [
      {
        "inclusion": "not_included",
        "price": "75.00",
        "currency": "AED",
        "meal_type": "breakfast"
      }
    ],
    "children_meal": [],
    "extra_bed": [
      {
        "inclusion": "not_included",
        "price": "125.00",
        "currency": "AED",
        "price_unit": "per_guest_per_night",
        "amount": 1
      }
    ],
    "cot": [],
    "pets": [],
    "shuttle": [
      {
        "inclusion": "not_included",
        "price": "400.00",
        "currency": "AED",
        "shuttle_type": "two_ways",
        "destination_type": "airport"
      }
    ],
    "parking": [],
    "children": [],
    "visa": {
      "visa_support": "unspecified"
    },
    "deposit": [],
    "no_show": {
      "availability": "unspecified",
      "time": null,
      "day_period": "unspecified"
    },
    "add_fee": [],
    "check_in_check_out": []
  },
  "metapolicy_extra_info": "Please note that a surcharge may apply for UAE residents upon arrival.\r\n\r\nMandatory gala dinner fees may apply at UAE hotels. Guests should contact the hotel directly for more information.\nFor an Apartment with one room the city tax is AED 10.00, with two rooms AED 20.00.",
  "star_certificate": null,
  "facts": {
    "floors_number": 20,
    "rooms_number": 152,
    "year_built": 2008,
    "year_renovated": null,
    "electricity": {
      "frequency": [
        50
      ],
      "voltage": [
        240
      ],
      "sockets": [
        "g"
      ]
    }
  },
  "payment_methods": [
    "visa",
    "maestro",
    "jcb",
    "master_card",
    "american_express",
    "visa_electron",
    "diners_club"
  ],
  "hotel_chain": "Radisson Blu",
  "front_desk_time_start": null,
  "front_desk_time_end": null
}
10

So, quick update on this, i turned off all outputs, and turned of debug option and managed to do the entire 3 million records in one go without running out of memory for the full run.

Secondly, when running this previously I was keeping an eye on my digital ocean CPU, Load, Memory graphs to see how hard the server was being hit, and it was jumping from 4% CPU usage to 90% when the script ran, however for the full run with Debug off, it never went over 15% CPU usage, which I found very interesting.

Here are some stats over 24 hours of my Application Container, the 100% is before turning off Debug and the right hand side is after turning it off, considering the server action never changed at all.

2024-09-10_08-43-37
2024-09-10_08-43-37984×1648 125 KB

This is the database container, with 50% CPU going down to 10%

2024-09-10_08-46-13
2024-09-10_08-46-13987×1650 149 KB

I am kind of wondering if, in light of this, it might be a good feature request to see if Wappler could give a debug on/off swith on a per server action basis.

2 Likes
11
2 Likes
12

Thanks Apple, back in 2022 i was not smart enough to have known I even needed that, takes me a few years to catch up, but i do get there eventually.