Hi - I'm trying to set up a new project on Heroku. All OK but I now I need to connect to the database and it's not working / I'm lost.
I've created a local database as per the docs, adding it in resource manager under local services and adding it to the development target as a Cloud Database. So far so good.
The problems start when I switch to production target and want to connect to the database in the Heroku app. It's a Heroku postgresql database.
If I add the details here in production target settings, it doesn't work:
If I try to create a new connection in the database manager, it says I can only make database changes on local targets.
If I create the new connection on the local target it works and testing the connection works. But as soon as I switch to the production target it refuses to connect and any options to edit anything are greyed out.
I've used Heroku with Node.js and their Postgres cluster. It worked well for me, so I'm curious what's up.
What kind of local/development database are you using on your Development target?
If you can, make a development DB on the same kind of server you'll be using in production, especially for Postgres - that way, your Development environment will give you a good indication of how the site will perform when running in production.
If you have a ~$50 cluster from Heroku, you can often make a development DBs at no extra cost and have a few for staging as well. If you are doing the Basic DB option, you can buy a development DB add on for about the same cost to avoid having to jump to a $50 cluster.
The Database Manager connection is for the database editor built into Wappler, which is only intended for use on your development targets.
The built-in Database Manager lets you view/edit tables and data, etc. - but you don't generally want to edit the live site in Database Manager.
Changes to your production database are migrated from Dev to Production when you deploy. I prefer to deploy via git, see below.
To be clear, what you enter in Database Manager is not what your web app uses to connect to your production database server when it is running live on a web server. The DB manager info is to let you edit and update the database schema and data itself in your development environment.
This great video by Hyperbytes video doesn't cover Heroku specifically, but is worth a watch to understand how to change and understand the various Database settings and options
How are you deploying to Heroku? I used the github option where you click with a deploy within the Heroku admin interface or CLI and highly recommend this method.
Thanks nomad. I missed setting up the database connection in globals under workflows.
I had everything working fine on digital ocean but wanted to try heroku. I might end up leaving the database on DO and pointing at that since this is still not quite working properly.
My local db is postgres set up via Wappler. I understand how the database manager is supposed to work, but if I can't see the db in there = db connection doesn't work = none of the scripts will work on the live app. I was just trying different things as I couldn't find all the right places to edit the settings!
I'm deploying to Heroku using the deploy button in Wappler which seems to work fine.
The weird thing now is the db seems to have connected, but if I create a table in development it's not deploying to production though it appears to in Wappler. If I create a table with pgAdmin I see it in Heroku but not in Wappler.
The production target in Wappler is definitely the Heroku db so I have no idea how that's happening (I do refresh full schema) but clearly it's not connecting to the right place.
If I don't get it to work I'll keep the db on digital ocean and point it there.
Separate question - what's best practice for a staging server with Wappler + Heroku. Do you use a branch in the same Heroku app or create a second Heroku app to use as a staging target?
Perhaps it's me but I think there's a bug. It's seriously confusing to have database connection settings in so many places and it's not at all clear which is trumping the other when they conflict, so it might just be I'm doing something wrong but I can't see what.
I have 2 databases - one local (set as a cloud server in the resource manager) and one on Heroku.
In project settings I have the database targets as local for development and Heroku for production. These seem to be routinely ignored by Wappler.
In the server connect globals I have a database connection. Whatever I put here, the database manager shows me the local database for both development and production. I've tried changing the db.json settings for both targets and it does nothing.
In the development target, I have the option to click database connection. Using that I can connect to the Heroku database, but the production one stays as local. I noticed that when I add the Heroku database details as 'custom' and save it, it works but when I open it again I see a mix of 'cloud server' and the local one selected + the Heroku details below. It still connects to Heroku though.
When I switch to production, I can't connect to Heroku whatever I do. I also get an error so perhaps something was corrupted in the project set up?
File C:\Users\pauls\AppData\Local\Wappler\resources\app\Shared\DMXzone\Databases\Connections\node\dmxDatabase.json does not exists!
For what it's worth, changing the settings to connect to my Digital Ocean database cluster works perfectly and I can point at local db or any individual db on the cluster. So I'm going to stick to that and give up on Heroku databases!
I was also confused and frustrated by the lack of clear difference between the various database connection settings in Wappler on my first ~10 projects. It really took a while for me to understand.
Just one person's opinion, but I think database & deployment are the roughest areas to navigate in Wappler right now while learning, especially given the lack of organized documentation or updated guidance.
Here's a webinar from 2020 that explains some of the complexities that I think are causing confusion, if you want to dive into it... it is the only place I've heard/seen a lot of the DB Manager stuff explained, but like many docs about 1/2 of it is old info and that can be hard to parse.
I also understand just going with what works Good luck!
Thanks. Yes totally agree that deployment and databases are really hard to navigate in Wappler and any docs that do exist are really out of date. I understand there are a lot of different possible server and database set-ups so it's hard to cover them all, but there should be something explaining what is doing what within Wappler.
My new problem is that I can't point to more than one Heroku project within one Wappler project but i'll start a different thread for that.
So what do you feel is missing from the video i created last week or the "using the resource manager" series from july last year covering cloud databases (which remains unchanged in Wappler since then).
If people can be specific i may be able to plug the holes. Obviously I cant (and won't) produce a tutorial for every provider but the process is pretty generic, people either use hosted database, cloud database or docker.
Hi Hyperbytes - firstly I'd just say my comment was related to Wappler docs, rather than contributor videos! Also, when I first started with Wappler your intro series helped me loads on getting up to speed quickly, so thank you for producing these videos. I know it's time-consuming.
The video from last week I assume is the one Nomad posted above? Again it's a really helpful video, and helped me realise that I needed to be looking at the database connection settings in globals, but if you want feedback on areas I think could be improved or are missing:
you start with a really basic description of the data flows, which is really helpful for the beginners the video is aimed at, but then you don't relate how target settings, global db connections etc fit with that model
you assume people will already know what a db connection is and suggest the name doesn't matter. I think you'll find beginners setting connections with different names - one for dev and one for production and not understanding what the purpose is of the connection name when building api workflows etc. You show how to change the database connection for each target but assume the viewer will know why do that vs create a second connection
I still don't really understand what Wappler is doing with the database connection info in the project settings (targets) vs the globals (db.json). I think it's the db.json that dictates the actual connection, and the target settings is a shortcut way of filling that in? If that's correct, it's only one way so when you change the db.json, the target settings don't follow. What happens then? The target settings are ignored?
Similarly, the database connection button on the database manager. What is it doing? Is it bypassing the globals setting or updating it? If I use that can I ignore the project target settings? If I connect using it, is it for everything or only the database manager? If I'm using it, I can only edit in development so how do I connect a different production database? Why is it sometimes greyed out in development?
Those are some of the areas I think people would struggle to understand straight away, and while a step by step example is really useful, it's hard to apply to your own setup if the understanding isn't there. The other issues most people have I think will be around security access and restricted IP access or similar on their database settings, which I agree are not really for a Wappler video.
Sorry if that's a bit of a stream of consciousness but hope that's somewhat helpful. I haven't seen the resource manager series so can't comment on that one.
Thanks, feedback is really useful.
Perhaps, having worked in database services for 40+ years, what i consider obvious may not be to learners.
I will give your feedback some thought and maybe create an intro to the one i released last week where i really "dumb things down" (no offence meant).
I am sure your are not alone in this thinking.
I have heard of this referred to as the curse of knowledge and it is a tough one!
You and Ben's videos (and this forum) are the only reason I was able to find my way thorugh Wappler. I really appreciate your willingness to share 40+ years of experience with so us all.
Things I think aren't immediately obvious to learners include:
The difference between Database Manager settings and what ends up in Server Connect Globals - Database Connections really didn't make sense to me for a long time. Wappler doesn't seem to have user interface cues or guardrails for things you must do ... like closing db.json before swiching between and editing environments. Sure, it makes sense why you have to close these things, but the lack of error messaging makes it one of those things you just "have to know"
I learned a lot watching the 2020 Zoom meeting I linked above where George intro'd the first version of Database Manager. It is outdated now, but contains so much foundational knowledge that helped me grasp how the Database Manager is expected to work and be used. I don't know if that same info is in Docs somewhere.
I've also been using Wappler long enough to remember when every API step needed its own DB connection step in a repetitive way. So I understand why things are the way they are now, but if you didn't use the "old way" the new way really seems counterintuitive to me. Not that it can't be learned or doesn't make sense, it is just difficult to understand why the Target settings and Database Manager settings and Globals - Database Connections can sometimes by sync'd, sometimes not, sometimes overwritten, etc.
Thanks again for all the knowledge you share on these topics!
Basically you may not have fully understood the function of the database connection in workflow -> Globals and the Database Manager.
In some cases they will be the same, in others different but always should be considered as separate entities.
Each connection (both types) can have settings unique to each target.
You can also also have more than one database connection which allows you to manage multiple databases.
Should you wish to watch my series on couchDB you can see how i use two database connections to manage a couch and a mysql database side by side and move data back and forth.
Many thanks and if you do create a video please link to it here so we don't miss it.
I think there's 2 general issues for people:
Understanding the concepts and how to connect to databases etc
Understanding the Wappler terminology and where to find everything
I've found the naming of things in Wappler can be non-obvious. As one example - if I set up a local database in the resource manager under services, when I point to it from the project settings I need to select 'cloud database' which is definitely counter-intuitive.
I don't think people are deliberately trying to complicate or overthink things. When things work first time it's easy. When it doesn't and there's no error message or explanation you look into all the possible settings to see what the problem could be. I don't know the history, but coming in cold to Wappler I would have expected one place to define the db connection and then reference that as required. There are at least 4 places in Wappler you can enter connection details (resource manager, server connect globals, project targets, database manager). I would bet good money that if you put an experienced database engineer in front of Wappler, without any instruction, they would struggle to get connections to multiple databases up and running straight away. It's really interesting when you read these forums, how many people are saying 'I can connect to my database easily using workbench or similar, but I can't do it in Wappler'. And given the point of Wappler is to make coding easier and more accessible, that suggests it might not be so simple without clear guidance.
I really like Wappler overall and I'm building more complicated apps faster than I could without it for sure, but like nomad said, a lot of the learning curve comes from realising that Wappler will sometimes crash/ not save/ overwrite something without warning or error message. Also, I like to understand how things work and then apply that. A lot of the Wappler written docs are step by step examples to do something but there's very little 'this is what XYZ is in Wappler, now you go and apply it'. One example off the top of my head is dmx functions in javascript. There's a tutorial somewhere about how to pass a variable - that's great. But there's nothing that says you can use DMX functions in javascript and here's a list of them including dmx.parse(). Patrick wrote a list of them in a comment on this forum somewhere, but that's the kind of thing that should be in the docs.
Yes there is but they all perform a different and specific function.
I will try to clarify as best i can.
In general terms "cloud database" normally refers to a database hosted via docker.
Obviously I'm not suggesting Wappler's design is your doing, and the time you spend educating others on it is commendable and much appreciated. Really, I feel Wappler relies too much on this forum and the goodwill of people like you to produce videos. I know there are also Wappler employees on the forums who are really helpful and I'm not suggesting the docs are non-existent, but they are less comprehensive and less well maintained than other subscription services.
Re the multiple connection details, the fact they do different things is irrelevant. It's clearly a hangover from how Wappler has developed, rather than the logical way to build it if you started now. If you had a blank canvas, you would set up connections once in one place, with an ability to test the connection (not available in all of the current ones) and simply reference the right connection in each application of it. That's what happens if I add a digital ocean database in the resource centre - I can simply refer to it in the target settings etc. No reason why that shouldn't be possible for any manual connection settings. Or perhaps it is and I can't find it
Just pop in here to say thank you @Hyperbytes the diagram and explanation at the start of the video made It abundantly clear what the difference between the two are.
I'm still fuzzy on the importance of setting the database in the target tab. Is this merely a convenience? I was not able to get that to take, I ran into the naughty wappler as you mentioned in the video I guess. But no amount of closing the tab or hitting the refresh button like it's going out of style would get Wappler to inject the credentials from the target tab.
Instead I manually entered them into the globals, and have left the database in the target tab empty. This seems to have worked, as I'm able to see the data on the live site. But I was wondering if there would be any issues with doing it that way, or at the end of the day, it is the same thing?
@PaulS As it happens I just managed to get this working with Heroku. I'm not sure it would help you or if you've already given up on it, but I'd be happy to write down the steps I took with some screen grabs?
Spoiler alert! it's pretty much what is In the video above, but manually entering the heroku credentials as I wasn't able to get Wappler to do it of its own volition.
Database Manager connection is for the database editor built into Wappler, which is only intended for use on your development targets.
Good luck!