Wappler 4 RFC: Rethinking of the deployment workflow and target settings

[George]

As we are getting some great discussions and ideas about the upcoming features of Wappler 4, there is another topic I wanted to gather as much feedback as possible and that is the deployment and target management.

Currently we define in the Project options, various targets that you want to work on and deploy to.

However working on other targets than the local development target, has always caused confusion and errors.

The flow we want to follow is to always work on a local development target and only when you are done, choose a target to deploy to and hit the deploy to complete a full controlled deployment.

To achieve this ultimate goal, we want to rework the workflow and improve the UI’s to make this more clear.

Our goals:

1. Remove the target choice drop down below - you are always working on local development target
2. Add more options to the deploy action, like choose target and and maybe directly git commit description tag version, like with the publishing to Heroku now.
3. Improve project options - targets editing. Maybe make a whole new panel for targets only where you can manage each target and their options
4. Consolidate the other target specific settings like database connections, server connect globals - with the global target settings (preferred) or offer a target dropdown to choose where you edit them on (less preferred)

So as you can see we have tons of target specific settings that are now pretty much scattered all over the place.

So what will be your ideal organizations of target settings and deployment workflow?

[sitestreet]

This sounds good.

Up until now I’ve not really used a local development target, I have a live server which I save to as I go. This means I can share the progress with others and also know everything will be fine when ready to go live.

Switching to this method is fine, though.

My main issue to flag is when working on multiple computers. I switch between desktop and laptop and, occasionally another desktop (when I’m in the office which I haven’t been since Covid began!) so I would need to be sure that updating between them would be seemless. I currently use GIT to keep everything in sync.

Hope that helps.

I’ve not yet tried Docker or NodeJS - I’m still a good old fashioned PHP with the standard MAMP and LAMP stack.

[Sorry_Duh]

I think making most things tabs is the way to go especially at the rate Wappler is growing I think a good way to implement tabs fully would be as I mentioned in the other post

Wappler 4 RFC: Editor Tabs for Server Connect actions

Is there any reason the entire left panel couldn’t be tab control my idea is let’s use actions as a example one click it opens without the tabs showing at the top, when you edit the action a blue dot appears in the left navigation area when saving this is back to normal this would be for everything git databases pages everything if possible. Then have some save and close options on actions pages etc etc save and close, save, close without saving Also for easy access to the open tabs a new activ…

this gives us a nice easy way to work with tabs and have lots of extra space great for when things like PWA caches come into play as we could have cache reset options or something in this new targets tab or other target specific things

[karh]

We’re struggling with a few things with the current setup, it would be great if:

1. We can deploy on production based on git versions
2. We can set up the connection to the ‘managed db’ from digitalocean in the project settings for the production environment. And keep the docker db for the development environment. And make sure that the ‘db’ connection in server connect will target the right database depending on the selected target
3. It’s easy to deploy remote staging servers of git commits / pull requests

Furthermore my workflow is similar to what you described: always working on local dev and deploying to live when ready.

[George]

@Sorry_Duh

I think making most things tabs is the way to go especially at the rate Wappler is growing I think a good way to implement tabs fully would be as I mentioned in the other post

this gives us a nice easy way to work with tabs and have lots of extra space great for when things like PWA caches come into play as we could have cache reset options or something in this new targets tab or other target specific things

Yes we want to move the project options in a tab editor indeed.

Just not sure if the target settings should stay there and be extended (those are tabs as well), or should we make a whole new targets management/configuration settings panel.

[George]

@karh

We can deploy on production based on git versions

Yes that is a main power we want to integrate more closely indeed. So that everything is versioned and well bound with git.

So by doing a deploy - you are actually creating a git version tag and committing everything first - and then deploying it.

So you can see all your deploys as version tags in git manager as well.

The one challenge with that is to have the database migrations (changes) also running next to the deploy

[JonL]

@George

Just not sure if the target settings should stay there and be extended (those are tabs as well), or should we make a whole new targets management/configuration settings panel.

Whole new. The targets stuff is very confusing for old and new users.

That being said. I do not use Wappler deployment because I haven’t found a way to make it work for me as I use CI/CD elements.

I do use migrations extensively though so there has to be a way to still be able to control database migrations between targets even if you are only developing on local(which is a good and standard approach).

This is my workflow if it helps understand.

• I have 3 environments that match targets in Wappler: local, QA(remote server) and PRD(remote server).

If no database schema changes:

• I develop on local.
• Whenever I am happy with a code change I commit and push to my GIT remote.
• I have a git hook for commits that syncs the code in my remote with my QA server and builds it.
• I test in QA and when satisfied I promote to PRD manually.

If there are database schema changes:

• I develop on local.
• Whenever I am happy with a code change I commit and push to my GIT remote.
• I switch wappler target to QA and apply database changes there.
• I have a git hook for commits that syncs the code in my remote with my QA server and builds it.
• I test in QA and when satisfied I promote to PRD manually.
• I switch wappler target to PRD and apply database changes there.

[Sorry_Duh]

@George

Just not sure if the target settings should stay there and be extended (those are tabs as well), or should we make a whole new targets management/configuration settings panel.

I think redesigning the git area would be a great approach or just extending it and having targets next to it then we would be able to push from git in the same area

[TomD]

@George

choose a target to deploy to and hit the deploy to complete a full controlled deployment

That would be fine, as long as it’s still possible to upload individual files too.

An issue I have with the current setup is that I right-click a file and choose ‘upload’ and nothing happens - because I’ve forgotten to change to the remote target. This is since the change where you can’t upload if your target is local (which was certainly a great improvement). I think ‘upload’ should only appear when relevant.

Asssuming it will still be possible to upload individual files, it would save a lot of time (at least for me) if I didn’t have to find the file in File Manager to upload files. I may not have File Manager open. Eg I may be doing a global file search and making some minor changes to several files. It’s very time consuming to switch to File Manager, locate the file and choose upload, and then go back to where I was. If there were an upload option on the tab it would be extremely convenient. This will apply even more with the proposed switch to tabs in Server Connect etc.

[NewMedia]

Local deploy, for me, is the most natural because I always set up a localhost environment with a mysql database first.

Some errors I made were the result of inadvertently changing and saving the json values for a local connection into the remote target or vice-versa. The back and forth with the Targets as a little selectable dropdown at the bottom tool bar is, to me, a tool that should NOT be located there.
On my MacPro 15-inch it is One-quarter the size of this screenshot.

I think the Targets tool should be MUCH more Prominent with a large icon on the Left-side vertical menu. Furthermore, when a page is in Edit mode or Code Edit mode, there should be a Live Notification over the page that THIS page is presently going to be Targeted to the localhost, remote host, git account, node server, etc IF it is Uploaded after its Page save.

This dynamic Target notification should appear for any Page as a quick Visual Confirmation on Save & Upload.

It signals that either I DO want to be in that Target Mode as I intended, or that when I last saved this page I was working in the Remote Target “GIT-NewApp” and I must orient myself, etc.

One last request:
TABS – where are ADDED Tabs going?

My screen is laptop 15" – I only get 6 tabs on open files before they disappear behind the Left Side Vertical Navigation panel.
It would be a great help if this column could be Disappeared when I choose.

If I could slide this panel out of the way I would get 1/4 more column width for my Edit View of a page. A keystroke to Restore it.

Thanks for the Consideration!

[mebeingken]

@George

The flow we want to follow is to always work on a local development target and only when you are done, choose a target to deploy to and hit the deploy to complete a full controlled deployment.

I welcome the overall change here, except for the mandate of the “complete a full controlled deployment” requirement. I believe what you propose should be the normal workflow, and thus exposed prominently, but having the flexibility to upload individual files through a less visible right mouse in file manager (as it is now) is very valuable.

@George

1. Remove the target choice drop down below - you are always working on local development target

Yes

@George

2. Add more options to the deploy action, like choose target and and maybe directly git commit description tag version, like with the publishing to Heroku now.

I do not like the thought of an auto-version tag. I only create a tag for a production push, not staging, and not remote development. I work with clients who only can see staging (for them to preview everything) and production. As the developer though, I maintain a remote dev target as the test environment (So Dev, Test, Staging, Production). So I would like flexibility in what happens when I push to the various targets.

@George

3. Improve project options - targets editing. Maybe make a whole new panel for targets only where you can manage each target and their options
4. Consolidate the other target specific settings like database connections, server connect globals - with the global target settings (preferred) or offer a target dropdown to choose where you edit them on (less preferred)

Yes…the new panel should have ANYTHING that is target specific.

@George

Yes that is a main power we want to integrate more closely indeed. So that everything is versioned and well bound with git.

So by doing a deploy - you are actually creating a git version tag and committing everything first - and then deploying it.

So you can see all your deploys as version tags in git manager as well.

This is not appealing to me. Deployments are not also final, so my workflow is to only push to master AFTER I’m happy with the production push. And having a target that for me is Test, I would hate to have the git panel cluttered with a bunch of tags that I really only want on Production.

My vote is to keep deployment separate from git. Or, make sure we have plenty of flexibility. I understand the desire to help novices with a “standard flow” but there are plenty of us that need to maintain control and a flexible system.

[JonL]

This is going to be a though and critical one. One thing is thinking you are going to be more or less productive having a SC in a tab and complete different thing is having to change deployment workflows.

My suggestion is:
a) Start from scratch. New files for this.
b) Mark it as experimental for as long as needed. Even if it takes 6 months.
c) Leave current functionality as-is. UI and code wise.
d) All new panel in a tab separated from the old UI.
e) Iterate progressively until the current deployment workflows being used are covered and your new deployment framework(and servers/services reimagining) and UI is flexible enough to keep adding/changing in the future.
f) Switch off old UI/logic.

[sid]

I have not had much issues with current Wappler target settings - apart from the fact that DB connections gets messed up when setting up or switching between targets - even more so with Docker.
Usually, I set it up once, and forget about it.

The integration with Git and versioning and all sounds like an “advanced” type of workflow which would require docs even for things which are out of Wappler’s purview - like Git & Docker setup - and support too.
The docs right now are not very extensive on this.

As informed by Patrick in a post yesterday, the Git workflow right now is built with the idea of keeping things simple with some advanced stuff. It will NOT replace powerful tools like SourceTree etc.
So, I am not entirely sure how the idea here is going to take shape.

The main concern I have is there are as many deployment workflows as there are developers, which then gets complicated by number of server models.
I like the idea of a more mature target settings & deployment workflow management from Wappler, as it matures with more features.

I do like Jonas’ suggestions to take things slow in this regard, but not sure how slow would be good enough.

[sid]

@JonL

I do use migrations extensively

Do you use Wappler’s DB manager for this?

[JonL]

Yes. I do modify the migrations file by hand for some advance schema features that are not available from the UI.

But the act of applying and reverting schema changes I do it from wappler’s db manager.

Ideally I would run automatically those migration files from my CI/CD pipeline but I haven’t got to that yet.

Maybe I won’t have to worry about that if Wappler offers a powerful and flexible CI/CD solution in this new iteration of targets and deployment.

[sid]

@JonL

Yes. I do modify the migrations file by hand for some advance schema features that are not available from the UI.

Will probably have to trouble you someday to understand this. In my experience, the DB manager has been unusable. DB migration is something I learned about when trying out Laravel, but haven’t been able to use it yet.

[brad]

As long as I can still use my MAMP Pro as my development target and I can still choose a single file to upload to production without having to do a full deploy. Quite often I am working on my dashboard on MAMP Pro development target but make a small change to a page I need to be able to upload that single file without deploying unfinished work.

[TomD]

@brad

I can still choose a single file to upload to production without having to do a full deploy

I think this is essential and needed frequently as you say. Eg today, I had to search a new site for the word ‘delivery’ and replace it with ‘shipping’. This affected several files which I uploaded in a minute or so. Later, while on the phone to the client, I made a few similar changes. Redeploying a whole site in such cases would be extremely inefficient, and potentially cause all sorts of problems.

[NewMedia]

Which is why I usually keep an FTP client like FileZilla running while working on changes in one file or more. I always need to know what I’m targeting on uploads for small but significant changes.

Sometimes I run the same application referring to the same database in another “private” folder which I want to check in the online browser without “panicking” the client if the main application is in use online.

I’d rather not have to “bother Wappler” with a whole new target setup for this reason.

[TomD]

I also use FileZilla in parallel with Wappler, but less so now that Wappler’s upload features have improved and I’m more confident that Wappler won’t upload files I don’t want uploaded. I also use it because Wappler’s File Manager is rather basic. Eg I might want to upload a set of recently edited files - but Wappler has no way of sorting files.