Revamp Targets, Database Manager, Database Connections and deploy features

I am having a hard time putting the pieces together. I think you guys need to take a step back and rethink how all these pieces should fit together because, in the current state, it is very messy and lacking.

Right now we have Targets and Database Manager.

Target

Hosting -> Own server
Usage: Development, Staging, Production
Access type: OS Folder, FTP, SFTP

Hosting -> Firebase
Usage: Development, Staging, Production
Web Server URL

Hosting -> Docker Engine
Usage: Development, Staging, Production
Connection -> Local
Connection -> Remote -> Docker Machine

Database Manager

Database Connections -> Server Connect or Direct
Database
Migrations

These features I can make work but don’t make sense to me. Specially since the introduction of the Database Manager.

Probably we just need renaming some old wappler terminology, tweaking some and then adding a bit more.

For me it should be something like this:

Top level we have:

Projects

We store configuration that is shared through out the whole project.

  • Server model
  • Add-ons (bootstrap, app connect, font awesome, etc)
  • Version control (remotes and credentials)

Next level we have:

Environments

Similar to targets in concept but with a few key differences.
Users can create whatever number of environments that you want.
Each environment has a user defined color, a name and possibly an icon. We do this because It would be nice to have visual indicators of which environment we are in.
Users can attach to this environment nodes/features/options/plugins that suit their needs. The list of nodes will keep growing as you add goodies to Wappler :slight_smile:

Nodes you can attach:

Docker Engine

  • Web servers: Apache, nginx, lighttpd, IIS
  • DB engine: Mysql, Mariadb, Postgresql, sqlite
  • Custom dockerfile(you could duplicate the ones created by Wappler)

Local (User will provide configuration like installation folders, run parameters, etc)

  • Web servers: Apache, nginx, lighttpd, IIS
  • DB engine: Mysql, Mariadb, Postgresql, sqlite

You can mix nodes from docker and local. I might want to run my DB on docker but want to use lighttpd on my desktop or via dockerfile. This would have to be handled by Wappler as there needs to be configuration when mixing both.

For the webserver you will have to provide the folder where the files will be. This is handled internally in the case of Docker.

For each DB engine node you add you will have to provide the root user and password. Pre-populated in the case of Docker.

In the database manager you can create databases and for each database you will have to define the connection details specific to that database. They would be pre-populated as we already have root access from the environment setup.

For some environments you may have access to certain nodes or not. For instance, I might not have access to my files in QA and PRD when hosting with Heroku. But I still want to be able to run my migrations there in a streamlined fashion. That will depend on our own pipeline which brings me to my next concept.

On top of the environments you have:

Pipelines

These are optional for the user. It will depend on the complexity of the app.

A pipeline defines the flow of steps your app follows from your development system to production and how you move your files in between. In the future you can add nodes in each step like testing, CI and bundling. But right now it could handle database migrations, GIT pushing, FTP uploads and Docker machines which are the current methods of moving files. So you add environments to the pipeline and define how you are moving files between them.

This structure or similar would allow Wappler to be more flexible as to how users develop and deploy. It would also streamline database provisioning and management. Because right now it’s all quite messy.

Sure this post are just some ideas. I can clearly identify a gap already in my approach. But I just wanted to let you know that the current implementation could be improved and make it more 2020ish. I believe what we currently have was OK when FTP was the only way of deploying from Wappler but now that we are starting to have more options it needs to be re-imagined.

What do you all think?

Your over extensive brain power has flown entirely over my head, haha, sorry Jon, you lost me. I think the flow makes perfect sense as is, but I assume I just haven’t hit the same hurdle as you thus far.

I understand. I drafted this quite quickly so there might be some chaos in my post.

Indeed I have some specific deployment needs that don’t adjust to what Wappler provides. But these needs are actually nothing out of the ordinary nowadays.

So I have one single target in Wappler. This target uses a docker apache+PHP+PostgresSQL. Although I would love to use nginx as that is what I use in PRD.

Here I do all my development and write my test spec files. Quite common for any developer. My project is in sync with two GIT remote repositories. One is used as part of my deployment pipeline and the other I just use as a backup.

Once I send a pull request(PR) with the latest change to the master branch in the main repository Heroku will build automatically a new server with just the changes contained in this PR. It will create as many apps as open PRs. This way repository reviewers can access the full app with the changes contained in a specific PR and see them live. Once the reviewers approve the PR they will merge the changes into master but not before TravisCI does his thing.

TravisCI will run all the test specs that have been written for the app. The new tests I wrote for the latest changes and old tests to make sure I didn’t break anything else. If the tests are passed Travis will also bundle and compile all static assets automatically.

Once this is done the app is deployed automatically to a Heroku staging server. Here a subset of users can view new features, confirm that a reported bug no longer exists or report regressions.

Once we know everything is ok we give the order to Heroku that the app can be promoted to Production which are a set of servers with a load balancer in front of them.

It may seem complex at first, but that is actual quite a normal approach nowadays.

Wappler currently can only help on the first part: coding.

But if you could manage Wappler projects with environments and pipelines it could be more flexible and cover simple cases like copying from OS filesystem to FTP or handling Databases appropriately so I can creat my staging environment and run db migrations there for instance in a streamlined manner. That is without working around the strict limitations of the current targets approach.

So not saying that what’s available works bad. It is just that it’s not focused on how apps are deployed nowadays. That’s why it needs a revamp by setting strong foundations that allow flexibility. As each modern app follows more or less the same approach but might be using CircleCI instead of TravisCI or might work with different GIT branches for Staging and Production. Or might have a pre-production environment.

2 Likes

Thanks for the extended explanation, that is now clear as mud.
I would not personally hit such an issue at this stage as currently I do not use docker nor git as I work as a total one man band and probably talk more with you guys than real life people. Its just me and my 4 walls and a few projects I work on, so because of this I haven’t had a need as my projects are all self contained with no reviewers and nobody else adding code branches etc.
In your situation it does however make perfect sense why you need the extended workflow.

1 Like

Actually it’s just me and the end-users at the moment. This workflow works even with one developer.

It takes nothing(literally) to set up something like this and if I have to hand over the project or add more people to the development I already know that I don’t have to worry about the deployment pipeline as it’s already there.

It also embeds in your brain very good devops practices for the future.

People may read this post and ask themselves: how many servers do you have? What does your bill look like? I only pay for Wappler’s license and a domain. The rest are all free resources. Adhering to best practices and creating enterprise grade apps and ass tight proof deployment pipelines doesn’t have to be expensive. It can actually be very cheap until you start getting paying users.

1 Like

Hey Jon, I like your creative thinking.

I don’t quite have time to follow all your detail at the moment, but what I do know is that there several concepts in Wappler which aren’t clear and don’t flow, so I trust you have hit on one here.

I’d love the whole target/database thing to move towards and beyond the simplicity of Bubble’s “Just publish this thing with one click”… the beyond being with more possibilities and flexibility.

So yes, you get my vote…

to follow your 2020s crusade! :unicorn:

1 Like

Hi Jon,

Good thinking! We are indeed struggling a bit with the current targets to find the optimal user experience and flexibility.

I like you idea with environments - but they do sound to me like the targets we have now - but just a bit more customizable.

How do you propose to handle the publishing? Is it there where the pipelines comes in place?
If you have complex deploy with CI indeed, i see a whole pipeline, but what about regular sites with databases?

How do you see you ideal simple publishing/deploy?

Thanks! I am glad I am not alone in this. I know this may seem far too complicated for some users but that is because they are probably used to the old way of working with FTP targets.

As for your question of simple websites with a database the user would be able to create a pipeline of just two environments.

Dev -> Prod

Dev is already created from the environment step as a project should have a minimum of one environment(where you build the app).

He would need to create an additional environment called ”MyProdEnvironment” for instance and add two nodes: SFTP and a database connection. He would then provide connection details to the remote SFTP and his remote database.

Once he has these two environments he could create a pipeline with just Dev and Prd. This pipeline could have additional options like backup FTP folder contents or run migrations automatically which are not environment related but deployment related. That’s why they go in the pipeline.

Now Wappler knows that when pressing deploy, the code should be uploaded via SFTP and migrations, if present, will be ran for him if selected. If not selected he could go to the database manager and run them there manually. He would just need to change the environment to PRD and run them. Same as today.

All in all I see all this as a one single big JSON file. Right now you have several dispersed throughout the app filesystem. With a single file you could create templates for users that don’t want to spend time configuring a complicated pipeline and just want to handle via SFTP a simple website with a mysql DB.
By using templates users could share theirs with others.

The ability to duplicate tables in the Database Manager would be great addition as well!

I’m actually thinking more of a “services” that you use per project. and then each target has its set of configuration for those services.

So for example you just define per project which services you want to use.

  • A database server based on MySQL
  • A mail service
  • A security provider services
    (you can have as many as you wish)

Then per target you just define the exact connections/services.
Like do you want to use docker for this services or connect to a ready to go online service

Then the project components will just know about the used services and allow to pick them.
Like the Database manager will show you just the connections to the defined services and that’s it.

You do have to use exact the same services on all targets - that is why you define them on a global project level. So you have to have them all running on the different targets.

Does this makes more sense?

Love the topic, but buried with work right now…only to say, Production should be read only, and the process to push to production (pipeline, etc.) should be a clearly differentiated process that looks completely different.

5 Likes

Yeay, Ken, nice to see you on here in your busy-ness.

I totally agree… It is soooo easy to leave your target set as Production in Wappler and update things you didn’t mean to.

Part of being unsure about using Database Manager is that I want to go somewhere completely different to update my production world.

How about a confirmation box when you save or publish for the first time after changing target to remind you where it’s going to? You don’t want to see it every time you save but after changing target, or the first time after opening a project, or the first time after launching Wappler?

I avoid today by not having a production target at all. I just do a git pull directly from the production server. This is all fine with me, and works quite well…but if changes to Wappler are to be made, this needs to be considered.

1 Like

Actually that is what the “usage” option in the project target is where you can choose between development and production.

So marking a target as “production” will effectively disable all live editing to it and allow only controlled “deploy”

At least that is the idea :slight_smile: -is not implemented yet.

4 Likes

I think these two settings should be per environment. Not per project. I could be using sqlite and Mailhog (or similar) for my development environment. PostgreSQL and mailhog for my QA environment and PostgreSQL and sendgrid for production.

Also I still think the concept of target should be changed to environment. A local environment will never be a target. It is an origin. Target sounds like a better term for the pipeline concept I mentioned for deployment.

And I agree with Ken. The deployment functionality whichever is needs to be a separated process from the environment. Because there are dozens of combinations possible. This will be more evident as Wappler grows(which it will) and starts attracting more complex apps with original landscapes.

I do think that a convention over configuration is good when coding. But not with infrastructure and deployment. There you want configuration over convention.

So the idea of the target’s usage could be another potential issue in the future as it constrains us into preconceptions.

Instead of making me add a target as production. Let me mark an environment as read-only. Or even better. Let me mark the web files of an environment as read-only but not the database. Because I might need/want to make changes to my production database.

I still stand strong with my original request about Projects, environments and deployment pipelines and maximum configuration for them.

But also to keep it simple for other users lets provide templates.

1 Like

Do you really think it is wise to use different types of databases for development and production? This will make it very difficult to guarantee that what works on development will work on production as well. That is why I was thinking to require to use the same types but just different configurations for development and production. But I’m open for more arguments. Knex does make it more abstract so different database types might work… but again I think the best practice is to have exactly same type of servers - docker makes this all possible.

For mail servers is indeed clear that you will be using a simple test service for development and full blown mail service or SaaS for production. The functionality should be the same though.

It is quite a common setup to use sqllite in development or pre-development(yeah I’ve seen this setup :D)

It’s easy to setup up and it doesn’t have to consume 2 GB of RAM like Docker. Not bashing Docker! It’s still one of the most important features Wappler has. But in some contexts(like having a low-end spec computer) it might make more sense to develop on sqlite and let knex handle the creation of a PostgreSQL database and the migrations on a remote server. Another context would be…let me spin up a development system to draft an idea. I will use sqlite and an integrated light webserver like the PHP internal web server( php -S localhost:8000). No docker, no WAMP -> Happy 2010 laptop :slight_smile:

I never actually mentioned this before because knex(or any other query builder/ORM) was not available to Wappler. But now that knex is available it wouldn’t surprise me the usage of different database engines per environment.

This applies not only for databases engines, but even different web servers or different configurations for the same web server. I may want to cat logs from the database in development and have a more verbose output from the webserver. But not in QA or PRD.

Again this may sound very complex to some and unneeded and for that we would have templates so they don’t have to worry about this. But when Wappler starts attracting more builders with a coding background it will become necessary. That’s why it is important that if you are thinking of revamping this whole section some good foundations are set based on flexibility and the paradigm that each project has it’s own landscape. And you will never be able to anticipate how creative some landscapes are.

In App Connect if you can’t get something out of the box you can always build it or hack around it. But if a company uses sqlite and php internal web server for development and postgreSQL for production and Wappler doesn’t facilitate that setup it could be a potential deal breaker. I’ve worked with companies that provided very low-end spec laptops to work with. So that setup was common.

What I see now is that with every great new feature you add the concept of targets gets entangled because they are a reminiscence of FTP deployment. That’s why I asked to take a step back and rethinking the foundations. Obviously I’m not asking to implement all the things I said like being able to add a docker feature/node/plugin based on my own docker file. Just to implement a new system with solid foundations that cover what we have today and what we will have tomorrow.

1 Like

To clarify more. Not saying you have to implement the option of having different database engines per environment because that is not just changing the UI. It means more knex configuration and I know that is not a priority now. And probably it shouldn’t be for the time being.

Just saying do not add the database engine to the project level because if some day there is a need to have different db engines per environment we will be back in a topic called “Revamp Targets, Database Manager and Database Connections 2”.

I think when unclear if a feature should be part of a project or part of the environments it should belong to the environments.

Maybe just implement a flag called “share configuration” between all environments. That would be a nice middle ground solution that doesn’t constrain future flexibility. Some of these flags would be predefined by system and not editable like db engines because the feature of different engines is not implemented. And other flags would be user editable because the functionality is there.

You convinced me totally :slight_smile: will see what we can do. It all makes sense now :slight_smile:

1 Like