Generating a site map for dynamic data

Wappler General How To
#1

So my site is now done and ready to submit to Google, however I am having issues creating a sitemap for the content (which is dynamic)
with over 7K products I can not get the dynamic product links to show when I generate a site map online. Pages that have static links are no issues and they index on the sitemap.

I know with magento it will create a site map of the products, but how can I get this done using Wappler. Getting away from Magento is why I created this site.
In the old days DMX had an extensions to allow access in to the database, does anyone know the code it added to allow this?

Automatic Sitemap Creation (and Submission)
How could a keyword cloud be done?
#2

@gunnery have you figured out how to do it using Wappler?

#3

I ended up using a program called Screaming Frog, which generated a site map for me, had some pretty good results from it up to now.

#4

Does it easily generate sitemaps for dynamic pages? I’d give it a try.

I envision something like dynamic action to update / generate the sitemap on success. Would be good to have it built in Wappler.

#5

I agree would be good to have in Wappler. in answer to your question, yes its very easy to do and I’m actually using the program a lot now for SEO and ALL my websites are ranking well within weeks.

#6

I create XML sitemaps using PHP. It would be good if this were possible using Wappler, but I think it would be best if more general functionality were added in relation to this: the ability to read/write files.

If we could write a mixture of dynamic and static content (to include headers etc.) it should be possible to generate any type of text file (csv, xml etc.) in a much more comprehensive way than using the existing export facility, which is easy to use but very limited.

I think this is one of the most important features missing from Wappler. Please consider voting for it here.

1 Like
#7

Hi @gunnery. I’m needing to create an XML sitemap for a dynamic site so have downloaded and installed Screaming Frog but it seems to only be finding the static pages. All the dynamic pages use routing so have friendly URLs and are linked to from a search page so I would have expected them to all be found but it’s not finding any.

Is there anything I need to be aware of to get this to work?

#8

are you using the trial version ?

From memory I had a problem with that and ending up buying it because it would not work. If you want send me over the url and I’l scan it for you and send over the file. You’ll have to leave it with me for a couple of days as we are flat out at the min. Plus I’ve got to renew my licence as it has run out in May lol

1 Like
#9

Yes, I’m using the trial version. Happy to pay for it so long as it works otherwise I was thinking I’d code it to build the XML file dynamically. I did this for another site a few years ago but it would still take some time to do. The site is aircrewaccommodation.com if you’re OK running it through your copy, that would be great. Thanks for that. No hurry, though. Much appreciated.

#10

Hi @sitestreet have been looking at the site with Spider.
You have an issues that you need to be corrected before it gets indexed, otherwise it will be pointless.

At the moment your accomodation listing is coming up as
https://www.aircrewaccommodation.com/accommodation/12585
in the url

For people to be able to find it through a search engine, you need to change it to something like
https://www.aircrewaccommodation.com/area/accommodation type/ Large-double-ensuite-to-rent-near-Stansted Airport

This way the search engine has something to find when it searches. (at the moment all it will find is /accommodation/12585)

It’s pointless you having the number there as it will never rank or be found when someone is searching.
Have a think about how you would search for accomodation:
ie, Google Search: room to rent stansted airport

Here is an example from one of my websites:
https://www.thegunnery.com.au/4031/guns-and-ammo/centerfire-rifles/axis-ii-xp-blued-package-65-creedmoor-22-inch-4-shot-dm-/

So my layout is as follows:
https://www.thegunnery.com.au/4031/Main-Category-Name/Sub-category-Name/Product-Title/

Golden Rules for good ranking are:

  1. Search term in URL
  2. H1 with Search Items as high on the page as you can get
  3. Internal Links so on you search page make the links to: https://www.aircrewaccommodation.com/area/accommodation type/accomodation-title

SEO is a pain in the backside, but when you get the right formula it works bloody well.

Hope this make sense to you.
Let me know when you have made the changes and I will crawl through it again.

#11

Thanks @gunnery. The URLs will be addressed later, I just want to test the sitemap building so would you be able to run your version of Screaming Frog to see how it does?

#12

It works, but obviously at the moment there is nothing but numbers for it to pick up.
When I did mine for the shop it picked up all 78k products

#13

Numbers is fine. It should still pick up all the products so long as it can crawl all the links to them? Changing the URLs won’t affect the crawling. I’m not looking for SEO improvements, just a solution to build a sitemap which I can be confident includes all the results. Otherwise I’ll be building it with code querying the database but this sounded like a much faster solution so long as it’s comprehensive.

#14

ok no worries, I’m crawling it now and I’ll message the file over to you.
I’m doing a full index so it will have everything in it :slight_smile:

1 Like
#15

Top man @gunnery. Much appreciated.

1 Like
#16

just sent you the file, but I’ve just double checked it and it does not seem to have all accomodation in there. Running it again for you :slight_smile:

1 Like
#17

@gunnery

Hello I am in the same point, the dymanic URL are not listed on XML.

Can you help me check if this work with the screaming frog licence prior buying it ?

https://www.cbmhuatulco.com/

My site is very small.

#18

Does screaming frog solve the problem with Dynamic URL??

#19

I solved the problem with this great little extension:

And a NodeJS version here:

#20

I tried screaming frog but Google wouldn’t accept the XML file on my nodejs setup